Chapter 13

web scraping agent

Notebookagents-towards-production40 cells

Building an Intelligent Web Scraping Agent with LangGraph and Bright Data + MCP

Overview

This comprehensive tutorial demonstrates how to build a production-ready web scraping agent that combines LangGraph's ReAct (Reasoning and Acting) framework with Bright Data's advanced scraping infrastructure. The resulting system can intelligently navigate websites, extract structured data, and conduct complex research workflows with minimal human intervention.

The agent you'll build represents a significant advancement over traditional web scraping approaches. Instead of writing custom scrapers for each website, you'll create an intelligent system that can reason about different scraping strategies, select appropriate tools, and adapt to various web structures automatically.

Learning Objectives

By completing this tutorial, you will understand how to:

Implement LangGraph ReAct agents with external tool integration for autonomous decision-making
Configure Bright Data's Model Context Protocol (MCP) server for enterprise-grade web scraping
Design intelligent agents that dynamically select optimal scraping strategies based on target websites
Extract structured data from major platforms including e-commerce sites, social media, and news sources
Implement browser automation workflows for complex user interactions
Build comprehensive research pipelines that synthesize information from multiple sources

Architecture Overview

The system architecture consists of three primary components working in harmony:

code

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────────┐
│   User Query    │───▶│   ReAct Agent    │───▶│   Bright Data MCP   │
│                 │    │   (LangGraph)    │    │      Server         │
└─────────────────┘    │                  │    │                     │
                       │ ┌──────────────┐ │    │ ┌─────────────────┐ │
                       │ │ Reasoning    │ │    │ │ Search Engines  │ │
                       │ │ Engine       │ │    │ │ Web Scrapers    │ │
                       │ └──────────────┘ │    │ │ Platform APIs   │ │
                       │ ┌──────────────┐ │    │ │ Browser Tools   │ │
                       │ │ Tool         │ │    │ └─────────────────┘ │
                       │ │ Selection    │ │    └─────────────────────┘
                       │ └──────────────┘ │
                       └──────────────────┘

The ReAct agent serves as the intelligent coordinator, analyzing user requests and selecting appropriate tools from Bright Data's comprehensive suite of web scraping capabilities. This design ensures both flexibility and reliability in handling diverse web scraping scenarios.

Prerequisites and Environment Setup

Before beginning the implementation, you'll need to establish accounts and configure your development environment. This section guides you through the essential setup steps.

Required Accounts and API Keys

Bright Data Account Setup

Create a Bright Data account at This link. New accounts receive 5,000 unlocker requests monthly at no cost, providing substantial resources for development and testing.
Navigate to your account settings and locate your API key. This credential will authenticate your agent with Bright Data's infrastructure.

Language Model Access

Register for an OpenRouter account to obtain API access for language models. While this tutorial uses Gemini through OpenRouter for optimal performance and cost efficiency, the architecture supports any compatible language model.

Development Environment

Ensure your development environment includes Python 3.8 or higher with pip package management capabilities.

Dependency Installation and Configuration

The following section installs the required Python packages and establishes the foundational imports for our web scraping agent. Each dependency serves a specific role in the overall architecture.

API Key Configuration

This cell creates environment variables for your API credentials. Replace the placeholder values with your actual API keys before execution.

In [1]python · cell 5

python

# To export your API key into a .env file, run the following cell (replace with your actual keys):
!echo "BRIGHT_DATA_API_TOKEN=<your-brightdataa-api-key>" >> .env
!echo "OPENROUTER_API_KEY=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" >> .env

Package Installation

This cell installs the core dependencies required for the web scraping agent. The packages include LangGraph for agent orchestration, OpenAI client for language model interaction, MCP client for Bright Data integration, and supporting utilities.

In [5]python · cell 7

python

# Install required packages
%pip install langgraph langchain-openai mcp-use python-dotenv asyncio --quiet
%load_ext autoreload
%autoreload 2

Output

Note: you may need to restart the kernel to use updated packages.

Core Imports and Environment Loading

This cell imports the essential libraries and loads environment variables from the configuration file. The imports include async handling capabilities, language model clients, agent frameworks, and MCP integration components.

In [1]python · cell 9

python

# Import all necessary libraries
import asyncio
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from mcp_use.client import MCPClient
from mcp_use.adapters.langchain_adapter import LangChainAdapter
from dotenv import load_dotenv
import os

# Load environment variables from .env
load_dotenv()

Output

True

Environment Verification

Before proceeding with agent configuration, it's essential to verify that all required environment variables are properly loaded. This verification step prevents runtime errors and ensures smooth operation.

In [2]python · cell 11

python

# Verify environment setup
print("✅ Environment setup complete!")
print(f"OpenRouter API Key loaded: {'Yes' if os.getenv('OPENROUTER_API_KEY') else 'No'}")

Output

✅ Environment setup complete!
OpenRouter API Key loaded: Yes

Understanding the ReAct Agent Framework

The ReAct (Reasoning and Acting) framework represents a significant advancement in autonomous agent design. Unlike traditional rule-based systems or simple chain-of-thought approaches, ReAct agents integrate reasoning capabilities with action execution in a unified framework.

Core Principles of ReAct Architecture

ReAct agents operate on a fundamental principle that combines deliberative reasoning with interactive tool usage. This approach enables agents to:

Dynamic Reasoning: The agent analyzes each situation and develops a reasoning strategy tailored to the specific task requirements. Rather than following predetermined scripts, it evaluates available information and determines the most effective approach.

Tool Selection: Based on its reasoning, the agent selects appropriate tools from its available repertoire. This selection process considers factors such as data requirements, website characteristics, and desired output formats.

Iterative Refinement: The agent can execute multiple actions in sequence, using the results of previous actions to inform subsequent decisions. This capability is particularly valuable for complex web scraping scenarios that require multi-step workflows.

Application to Web Scraping

In the context of web scraping, ReAct agents excel because they can adapt their approach based on the target website's characteristics. For instance, when encountering an e-commerce site, the agent might recognize the need for structured product data extraction. Conversely, when analyzing news articles, it might prioritize content extraction and summarization techniques.

This adaptability eliminates the need for maintaining separate scraping scripts for different websites, significantly reducing development overhead while improving reliability and maintainability.

Bright Data MCP Integration

The Model Context Protocol (MCP) represents a standardized approach for integrating external services with language model applications. Bright Data's MCP server provides access to a comprehensive suite of web scraping tools through a unified interface.

MCP Architecture Benefits

The MCP integration offers several advantages over direct API integration:

Standardized Interface: All Bright Data tools are accessible through consistent function signatures, simplifying agent development and reducing integration complexity.

Automatic Tool Discovery: The MCP server dynamically exposes available tools, allowing the agent to discover and utilize new capabilities without code modifications.

Error Handling: Built-in error handling and retry mechanisms improve reliability when dealing with network issues or temporary service unavailability.

Performance Optimization: The MCP server includes caching and request optimization features that enhance overall system performance.

Bright Data MCP Server Configuration

This function establishes the connection to Bright Data's MCP server and converts the available tools into LangChain-compatible format. The configuration process involves setting up the server connection parameters and authenticating with your API credentials.

In [3]python · cell 15

python

async def setup_bright_data_tools():
    """
    Configure Bright Data MCP client and create LangChain-compatible tools
    """
    
    # Configure the MCP server connection
    bright_data_config = {
        "mcpServers": {
                    "Bright Data": {
            "command": "npx",
            "args": ["@brightdata/mcp"],
            "env": {
                "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
            }
            }
        }
    }
    
    # Create MCP client and adapter
    client = MCPClient.from_dict(bright_data_config)
    adapter = LangChainAdapter()
    
    # Convert MCP tools to LangChain-compatible format
    tools = await adapter.create_tools(client)
    
    print(f"✅ Connected to Bright Data MCP server")
    print(f"📊 Available tools: {len(tools)}")
    
    return tools

# Test the connection
tools = await setup_bright_data_tools()

Output

2025-07-22 14:19:26,387 - mcp_use - INFO - No active sessions found, creating new ones...
✅ Connected to Bright Data MCP server
📊 Available tools: 60

Available Tool Categories

The Bright Data MCP server provides access to several categories of web scraping tools, each optimized for specific use cases:

Search Engine Integration: Comprehensive access to major search engines including Google, Bing, and Yandex, enabling the agent to discover relevant web content based on user queries.

Universal Web Scraping: General-purpose scraping capabilities that can extract content from any website in multiple formats including Markdown and HTML, with built-in bot detection bypass mechanisms.

Platform-Specific Extractors: Specialized tools for major platforms such as Amazon, LinkedIn, Instagram, Facebook, X (Twitter), TikTok, YouTube, Reddit, and Zillow. These extractors understand the specific data structures of each platform and can extract information more efficiently than general-purpose scrapers.

Browser Automation: Advanced capabilities for simulating user interactions including navigation, clicking, typing, and screenshot capture. These tools are essential for websites that require complex user interactions or JavaScript execution.

The diversity of available tools ensures that the agent can handle virtually any web scraping scenario with optimal efficiency and reliability.

Language Model and Agent Configuration

The next phase involves configuring the language model and creating the ReAct agent with a comprehensive system prompt. The system prompt plays a crucial role in defining the agent's behavior, reasoning patterns, and tool selection strategies.

ReAct Agent Initialization

This function creates the complete web scraping agent by combining the language model with the Bright Data tools. The system prompt is carefully crafted to guide the agent's decision-making process and ensure optimal tool utilization.

In [6]python · cell 19

python

import datetime

async def create_web_scraper_agent():
    """
    Create a ReAct agent configured for intelligent web scraping
    """
    
    # Get the tools from Bright Data first
    tools = await setup_bright_data_tools()
    
    # Get current date
    current_date = datetime.datetime.now().strftime("%B %d, %Y")
    
    # Initialize the language model
    llm = ChatOpenAI(
        openai_api_key=os.getenv("OPENROUTER_API_KEY"),
        openai_api_base="https://openrouter.ai/api/v1",
        model_name="google/gemini-2.5-flash-lite-preview-06-17",  # Fast and capable model for reasoning
        temperature=0.1  # Low temperature for consistent, focused responses
    )
    
    # Define comprehensive system prompt for the agent with dynamic date and tool count
    system_prompt = f"""You are a web data extraction agent. Today's date is {current_date}.

You have {len(tools)} specialized tools for web scraping and data extraction. When users request web data or current information, you MUST use these tools - do not rely on your training data.

Available capabilities:
- Search engines (Google/Bing/Yandex)
- Universal web scraping (any website)
- Platform extractors (Amazon, LinkedIn, Instagram, Facebook, X, TikTok, YouTube, Reddit, etc.)
- Browser automation

Process:
1. Identify data need
2. Select appropriate tool
3. Execute extraction
4. Return structured results

Always use tools for current/live data requests."""
    
    # Create the ReAct agent
    agent = create_react_agent(
        model=llm,
        tools=tools,
        prompt=system_prompt
    )
    
    print("🤖 ReAct Web Scraper Agent created successfully!")
    return agent

# Create the agent
agent = await create_web_scraper_agent()

Output

2025-07-22 14:23:30,529 - mcp_use - INFO - No active sessions found, creating new ones...
✅ Connected to Bright Data MCP server
📊 Available tools: 60
🤖 ReAct Web Scraper Agent created successfully!

System Prompt Design Principles

The system prompt serves as the foundation for the agent's behavior and incorporates several key design principles:

Clear Role Definition: The prompt establishes the agent's identity as a web data extraction specialist, providing clear context for its primary function.

Tool Awareness: By explicitly stating the number and types of available tools, the prompt ensures the agent understands its capabilities and prioritizes tool usage over relying on training data.

Process Framework: The four-step process outlined in the prompt provides a structured approach to handling user requests, ensuring consistent and methodical responses.

Temporal Context: Including the current date helps the agent understand the temporal context of requests and prioritize recent information when relevant.

Mandatory Tool Usage: The instruction to always use tools for current data requests prevents the agent from providing outdated information from its training data.

Basic Search Functionality Testing

With the agent configured, we can now test its fundamental capabilities. The first test demonstrates the agent's ability to search for current information and synthesize results from multiple sources.

Understanding Agent Decision-Making

When presented with a search query, the agent follows a systematic decision-making process. It first analyzes the query to understand the information requirements, then selects the most appropriate search tool based on the query characteristics. The agent then executes the search, analyzes the results, and presents a structured summary.

Basic Search Query Execution

This test demonstrates the agent's ability to search for current information and process multiple search results into a coherent summary. The agent will automatically select appropriate search engines and present the findings in a structured format.

In [11]python · cell 23

python

async def test_basic_search():
    """
    Test the agent's ability to search for current information
    """
    
    print("Testing Basic Search Functionality...")
    print("="*50)
    
    # Simple search query
    search_result = await agent.ainvoke({
        "messages": [("human", "Give me the latest AI news from this week, Include full URLs to source.")],
    })
    
    print("\n🔍 Search Results:")
    print(search_result["messages"][-1].content)
    
    return search_result

# Execute the test
basic_search_result = await test_basic_search()

Output

Testing Basic Search Functionality...
==================================================

🔍 Search Results:
Here's the latest AI news from this week:

*   **Why Apple is playing it slow with AI** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Tech giants split on EU AI code as compliance deadline looms** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Can speed and safety truly coexist in the AI race?** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Mistral** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Latent Labs launches web-based AI model to democratize protein design** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **Instead of selling to Meta, AI chip startup FuriosaAI signed a huge customer** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **OpenAI and** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **Nvidia CEO Draws Rock-Star Reception in China After Chip Sales OK'd** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **Trump Touts Billions in AI Investments from Blackstone, Google** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **The Real Energy Cost of AI** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **Subscribe to a weekly collection of AI News and resources on Artificial Intelligence and Machine Learning. For free.** - AI Weekly: https://aiweekly.co/
*   **Microsoft server hack hit about 100 organizations, experts say** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Researchers say AI-powered medical imaging tech could cut radiation exposure** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Meta investors,** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Leaked Memo: Anthropic CEO Says the Company Will Pursue Gulf State Investments After All** - WIRED: https://www.wired.com/tag/artificial-intelligence/
*   **OpenAI's New CEO of Applications Strikes** - WIRED: https://www.wired.com/tag/artificial-intelligence/
*   **Netflix uses AI effects for first time to cut costs** - BBC News: https://www.bbc.com/news/topics/ce1qrvleleqt
*   **The streaming firm says AI allowed The Eternaut to complete a sequence faster and cheaper.** - BBC News: https://www.bbc.com/news/topics/ce1qrvleleqt
*   **How OpenAI is Seeing the UK to AI Glory via Data Centres** - AI Magazine: https://aimagazine.com/
*   **Intelliscale: Meeting AI's Rising Demands in Data Centres** - AI Magazine: https://aimagazine.com/
*   **Hexagon: The AI Robotics Transformation in** - AI Magazine: https://aimagazine.com/
*   **European Union Unveils Rules for Powerful A.I. Systems** - The New York Times: https://www.nytimes.com/spotlight/artificial-intelligence
*   **A.I.-Generated Images of Child Sexual Abuse Are Flooding the Internet. Organizations that track the** - The New York Times: https://www.nytimes.com/spotlight/artificial-intelligence
*   **The latest news and top stories on artificial intelligence, including AI chatbots like Microsoft's ChatGPT, Apple's AI Chatbot and Google's Bard.** - NBC News: https://www.nbcnews.com/artificial-intelligence

Search Result Analysis

The search functionality demonstrates several key capabilities of the ReAct agent:

Query Understanding: The agent correctly interprets the request for recent AI news and understands the temporal requirement ("this week").

Tool Selection: Based on the query type, the agent selects the appropriate search engine tool from its available options.

Result Processing: The agent aggregates results from multiple sources and presents them in a structured, readable format with source URLs.

Content Prioritization: The results show the agent's ability to identify and prioritize relevant, current information from authoritative sources.

Advanced Platform-Specific Data Extraction

Beyond general web searching, the agent's true power lies in its ability to extract structured data from specific platforms. Each platform presents unique challenges in terms of data structure, access methods, and content organization.

E-commerce Platform Analysis

E-commerce platforms like Amazon contain rich structured data including product specifications, pricing, reviews, and availability information. The agent's platform-specific tools can navigate these complex data structures and extract relevant information efficiently.

When analyzing e-commerce data, the agent considers multiple factors including product features, pricing trends, customer feedback, and comparative analysis across similar products. This comprehensive approach provides users with actionable insights for purchase decisions or market research.

E-commerce Data Extraction and Analysis

This test demonstrates the agent's ability to research and compare products on e-commerce platforms. The agent will search for products, extract structured data, and provide comparative analysis with pricing and feature information.

In [12]python · cell 27

python

async def test_ecommerce_scraping():
    """
    Test structured data extraction from e-commerce platforms
    """
    
    print("Testing E-commerce Data Extraction...")
    print("="*50)
    
    # Ask the agent to find and analyze a product
    ecommerce_result = await agent.ainvoke({
        "messages": [("human", "Find information about the top-rated wireless headphones on Amazon and compare their features and prices")]
    })
    
    print("\n🛒 E-commerce Analysis:")
    print(ecommerce_result["messages"][-1].content)
    
    return ecommerce_result

# Execute the test
ecommerce_result = await test_ecommerce_scraping()

Output

Testing E-commerce Data Extraction...
==================================================

🛒 E-commerce Analysis:
I found several top-rated wireless headphones on Amazon. Here's a comparison of some of the most popular options:

**1. Sony WH-1000XM4**
*   **Features:** Industry-leading noise cancellation, excellent sound quality, comfortable design, long battery life (up to 30 hours), speak-to-chat functionality, multipoint connection.
*   **Price:** Typically around $350, but often on sale.

**2. Bose QuietComfort 45**
*   **Features:** Renowned noise cancellation, comfortable for long wear, balanced sound, good battery life (up to 24 hours), Aware Mode for hearing surroundings.
*   **Price:** Around $330, with occasional discounts.

**3. Apple AirPods Max**
*   **Features:** Premium build quality, excellent active noise cancellation, immersive spatial audio, seamless integration with Apple devices, comfortable fit.
*   **Price:** Premium pricing, usually around $550.

**4. Sennheiser Momentum 4 Wireless**
*   **Features:** Exceptional sound quality, very long battery life (up to 60 hours), effective noise cancellation, comfortable design, customizable EQ.
*   **Price:** Around $380.

**5. JBL Tune 510BT**
*   **Features:** Affordable option, decent sound quality, up to 40 hours of battery life, lightweight and foldable design, built-in microphone.
*   **Price:** Typically under $50.

**6. Anker Soundcore Life Q20**
*   **Features:** Hybrid active noise cancellation, Hi-Fi stereo sound, deep bass, up to 40 hours of playtime, comfortable earcups.
*   **Price:** Usually around $60.

**To help you choose, consider these factors:**

*   **Budget:** The JBL Tune 510BT and Anker Soundcore Life Q20 are great budget-friendly options. The Sony, Bose, Apple, and Sennheiser models are in the premium price range.
*   **Noise Cancellation:** Sony WH-1000XM4 and Bose QuietComfort 45 are top contenders for the best noise cancellation.
*   **Sound Quality:** Sony, Bose, Sennheiser, and Apple are generally praised for their superior audio performance.
*   **Battery Life:** Sennheiser Momentum 4 Wireless leads with up to 60 hours, while Sony and JBL offer very competitive battery life as well.
*   **Ecosystem:** If you're heavily invested in the Apple ecosystem, AirPods Max offer the most seamless integration.

I recommend checking the specific product pages on Amazon for the most up-to-date pricing and detailed feature comparisons. Would you like me to look up any of these models in more detail?

E-commerce Analysis Capabilities

The e-commerce analysis demonstrates several sophisticated capabilities:

Product Discovery: The agent can search for products within specific categories and identify top-rated options based on customer reviews and ratings.

Feature Extraction: Detailed product specifications, features, and technical details are systematically extracted and organized for easy comparison.

Price Analysis: Current pricing information is gathered and presented alongside historical pricing trends when available.

Comparative Framework: The agent structures its analysis to facilitate decision-making by highlighting key differentiators between products.

Actionable Recommendations: Rather than simply presenting data, the agent provides guidance based on different use cases and budget considerations.

Social media platforms represent a unique challenge for data extraction due to their dynamic nature, complex user interfaces, and diverse content formats. The agent's social media capabilities enable analysis of discussions, sentiment, and trending topics across various platforms.

Reddit Discussion Analysis

Reddit's structure of communities (subreddits) and threaded discussions provides rich opportunities for understanding public opinion and expert insights on specific topics. The agent can navigate Reddit's interface, extract relevant discussions, and analyze the content for key themes and insights.

Reddit Content Extraction and Analysis

This test demonstrates the agent's ability to search for and analyze social media content. The agent will search for relevant Reddit discussions and extract key conversation topics and community insights.

In [23]python · cell 31

python

async def test_social_media_simple():
    """
    Test Reddit extraction with a specific approach
    """
    print("Testing Reddit Extraction...")
    print("="*50)
    
    result = await agent.ainvoke({
        "messages": [("human", "Search for 'electric vehicles reddit' and then scrape one of the Reddit discussion pages you find. Show me what people are discussing.")]
    })
    
    print("\n📱 Reddit Analysis:")
    print(result["messages"][-1].content)
    return result

# Test this version
social_simple = await test_social_media_simple()

Output

Testing Reddit Extraction...
==================================================

📱 Reddit Analysis:
Here are some recent discussions from the r/electricvehicles subreddit:

*   **"This $11,000 Chinese EV with semi-solid state batteries is about to shake up the industry"**: This post discusses a new, affordable EV from China that features semi-solid state batteries, potentially disrupting the market.
*   **"Anyone else want an EV just so you no longer have to deal with the bullshit that comes with owning an ICE engine?"**: This user expresses their desire for an EV to avoid the common maintenance and repair issues associated with internal combustion engine (ICE) vehicles, highlighting the peace of mind EVs offer.
*   **"These cars are losing value fast — that's GREAT news for used EV buyers!"**: This article suggests that the rapid depreciation of some EVs is creating good opportunities for those looking to buy used electric vehicles.

It seems the community is actively discussing new EV technology, the benefits of EV ownership over traditional cars, and the used EV market.

The social media analysis showcases several key capabilities:

Platform-Specific Navigation: The agent understands Reddit's structure and can navigate to specific subreddits and discussion threads.

Content Aggregation: Multiple posts and comments are analyzed to identify common themes and discussion topics.

Sentiment Analysis: The agent can interpret the tone and sentiment of discussions, identifying positive, negative, or neutral attitudes toward topics.

Trend Identification: By analyzing multiple discussions, the agent can identify emerging trends and popular topics within communities.

Context Understanding: The agent maintains context about the specific community culture and discussion norms when interpreting content.

Complex Multi-Step Research Workflows

The agent's most sophisticated capability lies in conducting complex research that requires multiple tools, reasoning steps, and information synthesis. These workflows demonstrate the full potential of the ReAct framework in handling real-world research scenarios.

Research Workflow Architecture

Complex research workflows involve several interconnected phases:

Task Decomposition: The agent analyzes complex queries and breaks them down into manageable sub-tasks that can be addressed individually.

Tool Orchestration: Multiple tools are employed in sequence, with each tool's output informing the selection and configuration of subsequent tools.

Information Synthesis: Data from various sources is analyzed, compared, and synthesized into coherent insights that address the original research question.

Quality Validation: The agent evaluates the quality and relevance of gathered information, identifying gaps that require additional research.

Multi-Step Research Execution

This test challenges the agent with a complex research task requiring multiple information sources and analysis steps. The agent must demonstrate its ability to plan research steps, execute them systematically, and synthesize findings into actionable insights.

In [25]python · cell 35

python

async def test_complex_research():
    """
    Test the agent's ability to conduct multi-step research
    """
    
    print("Testing Complex Multi-Step Research...")
    print("="*50)
    
    # Complex research query
    research_result = await agent.ainvoke({
        "messages": [("human", """
        I need to research the current state of the renewable energy market. Please:
        1. Find recent news about renewable energy developments
        2. Look up major renewable energy companies and their stock performance
        3. Analyze social media sentiment about renewable energy
        4. Provide a comprehensive market overview with key insights
        """)]
    })
    
    print("\n🔬 Complex Research Results:")
    print(research_result["messages"][-1].content)
    
    return research_result

# Execute the test
research_result = await test_complex_research()

Output

Testing Complex Multi-Step Research...
==================================================

🔬 Complex Research Results:
I can help you research the renewable energy market. However, I need some clarification to proceed effectively.

For step 1, could you please specify what you mean by "recent news"? For example, are you interested in news from the past week, month, or a specific event?

For step 2, which major renewable energy companies are you interested in? Knowing specific company names will help me find their stock performance more accurately.

For step 3, analyzing social media sentiment requires access to specific platforms and potentially specialized tools. Could you clarify which social media platforms you'd like me to analyze (e.g., X, Reddit, etc.)?

Once I have this information, I can use my tools to gather the data you need for a comprehensive market overview.

Research Methodology and Planning

The agent's response to complex research requests demonstrates sophisticated planning capabilities:

Requirement Analysis: The agent evaluates the research request and identifies areas where additional specification would improve research quality.

Scope Definition: Rather than proceeding with assumptions, the agent seeks clarification to ensure research efforts are focused and relevant.

Tool Mapping: The agent understands which tools are appropriate for different types of information gathering and outlines its approach.

Quality Assurance: By requesting specific parameters, the agent ensures that the final research output will meet professional standards and user expectations.

This planning approach reflects best practices in professional research methodology and demonstrates the agent's capability to handle enterprise-level research requirements.

Comprehensive Research Assistant Implementation

The final component of our tutorial involves creating a comprehensive research assistant function that demonstrates the full integration of all the agent's capabilities. This function serves as a template for building production-ready research workflows.

Research Assistant Architecture

The research assistant function incorporates several design principles that make it suitable for production use:

Parameterized Configuration: Users can specify research scope, source limits, and other parameters to control the research process.

Structured Output: Research results are organized in a consistent format that facilitates further analysis or reporting.

Progress Tracking: The function provides feedback on research progress, helping users understand the complexity and scope of the work being performed.

Quality Controls: Built-in mechanisms ensure that research focuses on high-quality, relevant sources rather than simply maximizing the quantity of information gathered.

Research Assistant Function Implementation

This comprehensive function demonstrates how to structure complex research queries and configure the agent for optimal performance. The function includes parameterization for research scope and provides structured output formatting.

In [28]python · cell 39

python

async def research_assistant(query: str, max_sources: int = 5):
    """
    A comprehensive research assistant using our web scraping agent
    
    Args:
        query (str): The research question or topic
        max_sources (int): Maximum number of sources to analyze
    """
    
    print(f"🔍 Starting research on: {query}")
    print("="*60)
    
    # Enhanced research prompt
    research_prompt = f"""
    Please conduct comprehensive research on: "{query}"
    
    Your research should include:
    1. Current news and developments (last 30 days)
    2. Expert opinions and analysis
    3. Statistical data and trends
    4. Social media sentiment (if relevant)
    5. Key players and companies involved
    
    Use multiple sources and provide a well-structured summary with:
    - Executive summary
    - Key findings
    - Supporting data
    - Sources used
    
    Limit your research to {max_sources} high-quality sources.
    """
    
    result = await agent.ainvoke({
        "messages": [("human", research_prompt)]
    })
    
    print(f"\n📊 Research Complete!")
    print(result["messages"][-1].content)
    
    return result

# Test the research assistant
research_result = await research_assistant("Impact of artificial intelligence on job markets in 2025",5)

Output

🔍 Starting research on: Impact of artificial intelligence on job markets in 2025
============================================================

📊 Research Complete!

Research Assistant Benefits and Applications

The comprehensive research assistant function demonstrates several key benefits:

Scalable Research Framework: The parameterized approach allows the same function to handle research projects of varying scope and complexity.

Consistent Output Structure: The structured prompt ensures that research results follow a predictable format, facilitating integration with other systems or processes.

Quality Focus: By limiting the number of sources and emphasizing quality, the function prioritizes depth over breadth in research coverage.

Professional Standards: The research methodology incorporates best practices from professional research organizations, ensuring outputs meet enterprise requirements.

Customization Capability: The function can be easily modified to address specific industry requirements or research methodologies.

Chapter 13

web scraping agent

Notebookagents-towards-production40 cells

Building an Intelligent Web Scraping Agent with LangGraph and Bright Data + MCP

Overview

Learning Objectives

By completing this tutorial, you will understand how to:

Implement LangGraph ReAct agents with external tool integration for autonomous decision-making
Configure Bright Data's Model Context Protocol (MCP) server for enterprise-grade web scraping
Design intelligent agents that dynamically select optimal scraping strategies based on target websites
Extract structured data from major platforms including e-commerce sites, social media, and news sources
Implement browser automation workflows for complex user interactions
Build comprehensive research pipelines that synthesize information from multiple sources

Architecture Overview

The system architecture consists of three primary components working in harmony:

code

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────────┐
│   User Query    │───▶│   ReAct Agent    │───▶│   Bright Data MCP   │
│                 │    │   (LangGraph)    │    │      Server         │
└─────────────────┘    │                  │    │                     │
                       │ ┌──────────────┐ │    │ ┌─────────────────┐ │
                       │ │ Reasoning    │ │    │ │ Search Engines  │ │
                       │ │ Engine       │ │    │ │ Web Scrapers    │ │
                       │ └──────────────┘ │    │ │ Platform APIs   │ │
                       │ ┌──────────────┐ │    │ │ Browser Tools   │ │
                       │ │ Tool         │ │    │ └─────────────────┘ │
                       │ │ Selection    │ │    └─────────────────────┘
                       │ └──────────────┘ │
                       └──────────────────┘

Prerequisites and Environment Setup

Before beginning the implementation, you'll need to establish accounts and configure your development environment. This section guides you through the essential setup steps.

Required Accounts and API Keys

Bright Data Account Setup

Create a Bright Data account at This link. New accounts receive 5,000 unlocker requests monthly at no cost, providing substantial resources for development and testing.
Navigate to your account settings and locate your API key. This credential will authenticate your agent with Bright Data's infrastructure.

Language Model Access

Register for an OpenRouter account to obtain API access for language models. While this tutorial uses Gemini through OpenRouter for optimal performance and cost efficiency, the architecture supports any compatible language model.

Development Environment

Ensure your development environment includes Python 3.8 or higher with pip package management capabilities.

Dependency Installation and Configuration

The following section installs the required Python packages and establishes the foundational imports for our web scraping agent. Each dependency serves a specific role in the overall architecture.

API Key Configuration

This cell creates environment variables for your API credentials. Replace the placeholder values with your actual API keys before execution.

In [1]python · cell 5

python

# To export your API key into a .env file, run the following cell (replace with your actual keys):
!echo "BRIGHT_DATA_API_TOKEN=<your-brightdataa-api-key>" >> .env
!echo "OPENROUTER_API_KEY=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" >> .env

Package Installation

In [5]python · cell 7

python

# Install required packages
%pip install langgraph langchain-openai mcp-use python-dotenv asyncio --quiet
%load_ext autoreload
%autoreload 2

Output

Note: you may need to restart the kernel to use updated packages.

Core Imports and Environment Loading

In [1]python · cell 9

python

# Import all necessary libraries
import asyncio
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from mcp_use.client import MCPClient
from mcp_use.adapters.langchain_adapter import LangChainAdapter
from dotenv import load_dotenv
import os

# Load environment variables from .env
load_dotenv()

Output

True

Environment Verification

In [2]python · cell 11

python

# Verify environment setup
print("✅ Environment setup complete!")
print(f"OpenRouter API Key loaded: {'Yes' if os.getenv('OPENROUTER_API_KEY') else 'No'}")

Output

✅ Environment setup complete!
OpenRouter API Key loaded: Yes

Understanding the ReAct Agent Framework

Core Principles of ReAct Architecture

ReAct agents operate on a fundamental principle that combines deliberative reasoning with interactive tool usage. This approach enables agents to:

Application to Web Scraping

This adaptability eliminates the need for maintaining separate scraping scripts for different websites, significantly reducing development overhead while improving reliability and maintainability.

Bright Data MCP Integration

MCP Architecture Benefits

The MCP integration offers several advantages over direct API integration:

Standardized Interface: All Bright Data tools are accessible through consistent function signatures, simplifying agent development and reducing integration complexity.

Automatic Tool Discovery: The MCP server dynamically exposes available tools, allowing the agent to discover and utilize new capabilities without code modifications.

Error Handling: Built-in error handling and retry mechanisms improve reliability when dealing with network issues or temporary service unavailability.

Performance Optimization: The MCP server includes caching and request optimization features that enhance overall system performance.

Bright Data MCP Server Configuration

In [3]python · cell 15

python

async def setup_bright_data_tools():
    """
    Configure Bright Data MCP client and create LangChain-compatible tools
    """
    
    # Configure the MCP server connection
    bright_data_config = {
        "mcpServers": {
                    "Bright Data": {
            "command": "npx",
            "args": ["@brightdata/mcp"],
            "env": {
                "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"),
            }
            }
        }
    }
    
    # Create MCP client and adapter
    client = MCPClient.from_dict(bright_data_config)
    adapter = LangChainAdapter()
    
    # Convert MCP tools to LangChain-compatible format
    tools = await adapter.create_tools(client)
    
    print(f"✅ Connected to Bright Data MCP server")
    print(f"📊 Available tools: {len(tools)}")
    
    return tools

# Test the connection
tools = await setup_bright_data_tools()

Output

2025-07-22 14:19:26,387 - mcp_use - INFO - No active sessions found, creating new ones...
✅ Connected to Bright Data MCP server
📊 Available tools: 60

Available Tool Categories

The Bright Data MCP server provides access to several categories of web scraping tools, each optimized for specific use cases:

Search Engine Integration: Comprehensive access to major search engines including Google, Bing, and Yandex, enabling the agent to discover relevant web content based on user queries.

The diversity of available tools ensures that the agent can handle virtually any web scraping scenario with optimal efficiency and reliability.

Language Model and Agent Configuration

ReAct Agent Initialization

In [6]python · cell 19

python

import datetime

async def create_web_scraper_agent():
    """
    Create a ReAct agent configured for intelligent web scraping
    """
    
    # Get the tools from Bright Data first
    tools = await setup_bright_data_tools()
    
    # Get current date
    current_date = datetime.datetime.now().strftime("%B %d, %Y")
    
    # Initialize the language model
    llm = ChatOpenAI(
        openai_api_key=os.getenv("OPENROUTER_API_KEY"),
        openai_api_base="https://openrouter.ai/api/v1",
        model_name="google/gemini-2.5-flash-lite-preview-06-17",  # Fast and capable model for reasoning
        temperature=0.1  # Low temperature for consistent, focused responses
    )
    
    # Define comprehensive system prompt for the agent with dynamic date and tool count
    system_prompt = f"""You are a web data extraction agent. Today's date is {current_date}.

You have {len(tools)} specialized tools for web scraping and data extraction. When users request web data or current information, you MUST use these tools - do not rely on your training data.

Available capabilities:
- Search engines (Google/Bing/Yandex)
- Universal web scraping (any website)
- Platform extractors (Amazon, LinkedIn, Instagram, Facebook, X, TikTok, YouTube, Reddit, etc.)
- Browser automation

Process:
1. Identify data need
2. Select appropriate tool
3. Execute extraction
4. Return structured results

Always use tools for current/live data requests."""
    
    # Create the ReAct agent
    agent = create_react_agent(
        model=llm,
        tools=tools,
        prompt=system_prompt
    )
    
    print("🤖 ReAct Web Scraper Agent created successfully!")
    return agent

# Create the agent
agent = await create_web_scraper_agent()

Output

2025-07-22 14:23:30,529 - mcp_use - INFO - No active sessions found, creating new ones...
✅ Connected to Bright Data MCP server
📊 Available tools: 60
🤖 ReAct Web Scraper Agent created successfully!

System Prompt Design Principles

The system prompt serves as the foundation for the agent's behavior and incorporates several key design principles:

Clear Role Definition: The prompt establishes the agent's identity as a web data extraction specialist, providing clear context for its primary function.

Tool Awareness: By explicitly stating the number and types of available tools, the prompt ensures the agent understands its capabilities and prioritizes tool usage over relying on training data.

Process Framework: The four-step process outlined in the prompt provides a structured approach to handling user requests, ensuring consistent and methodical responses.

Temporal Context: Including the current date helps the agent understand the temporal context of requests and prioritize recent information when relevant.

Mandatory Tool Usage: The instruction to always use tools for current data requests prevents the agent from providing outdated information from its training data.

Basic Search Functionality Testing

Understanding Agent Decision-Making

Basic Search Query Execution

In [11]python · cell 23

python

async def test_basic_search():
    """
    Test the agent's ability to search for current information
    """
    
    print("Testing Basic Search Functionality...")
    print("="*50)
    
    # Simple search query
    search_result = await agent.ainvoke({
        "messages": [("human", "Give me the latest AI news from this week, Include full URLs to source.")],
    })
    
    print("\n🔍 Search Results:")
    print(search_result["messages"][-1].content)
    
    return search_result

# Execute the test
basic_search_result = await test_basic_search()

Output

Testing Basic Search Functionality...
==================================================

🔍 Search Results:
Here's the latest AI news from this week:

*   **Why Apple is playing it slow with AI** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Tech giants split on EU AI code as compliance deadline looms** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Can speed and safety truly coexist in the AI race?** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Mistral** - Artificial Intelligence News: https://www.artificialintelligence-news.com/
*   **Latent Labs launches web-based AI model to democratize protein design** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **Instead of selling to Meta, AI chip startup FuriosaAI signed a huge customer** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **OpenAI and** - TechCrunch: https://techcrunch.com/category/artificial-intelligence/
*   **Nvidia CEO Draws Rock-Star Reception in China After Chip Sales OK'd** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **Trump Touts Billions in AI Investments from Blackstone, Google** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **The Real Energy Cost of AI** - The Wall Street Journal: https://www.wsj.com/tech/ai
*   **Subscribe to a weekly collection of AI News and resources on Artificial Intelligence and Machine Learning. For free.** - AI Weekly: https://aiweekly.co/
*   **Microsoft server hack hit about 100 organizations, experts say** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Researchers say AI-powered medical imaging tech could cut radiation exposure** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Meta investors,** - Reuters: https://www.reuters.com/technology/artificial-intelligence/
*   **Leaked Memo: Anthropic CEO Says the Company Will Pursue Gulf State Investments After All** - WIRED: https://www.wired.com/tag/artificial-intelligence/
*   **OpenAI's New CEO of Applications Strikes** - WIRED: https://www.wired.com/tag/artificial-intelligence/
*   **Netflix uses AI effects for first time to cut costs** - BBC News: https://www.bbc.com/news/topics/ce1qrvleleqt
*   **The streaming firm says AI allowed The Eternaut to complete a sequence faster and cheaper.** - BBC News: https://www.bbc.com/news/topics/ce1qrvleleqt
*   **How OpenAI is Seeing the UK to AI Glory via Data Centres** - AI Magazine: https://aimagazine.com/
*   **Intelliscale: Meeting AI's Rising Demands in Data Centres** - AI Magazine: https://aimagazine.com/
*   **Hexagon: The AI Robotics Transformation in** - AI Magazine: https://aimagazine.com/
*   **European Union Unveils Rules for Powerful A.I. Systems** - The New York Times: https://www.nytimes.com/spotlight/artificial-intelligence
*   **A.I.-Generated Images of Child Sexual Abuse Are Flooding the Internet. Organizations that track the** - The New York Times: https://www.nytimes.com/spotlight/artificial-intelligence
*   **The latest news and top stories on artificial intelligence, including AI chatbots like Microsoft's ChatGPT, Apple's AI Chatbot and Google's Bard.** - NBC News: https://www.nbcnews.com/artificial-intelligence

Search Result Analysis

The search functionality demonstrates several key capabilities of the ReAct agent:

Query Understanding: The agent correctly interprets the request for recent AI news and understands the temporal requirement ("this week").

Tool Selection: Based on the query type, the agent selects the appropriate search engine tool from its available options.

Result Processing: The agent aggregates results from multiple sources and presents them in a structured, readable format with source URLs.

Content Prioritization: The results show the agent's ability to identify and prioritize relevant, current information from authoritative sources.

Advanced Platform-Specific Data Extraction

E-commerce Platform Analysis

E-commerce Data Extraction and Analysis

In [12]python · cell 27

python

async def test_ecommerce_scraping():
    """
    Test structured data extraction from e-commerce platforms
    """
    
    print("Testing E-commerce Data Extraction...")
    print("="*50)
    
    # Ask the agent to find and analyze a product
    ecommerce_result = await agent.ainvoke({
        "messages": [("human", "Find information about the top-rated wireless headphones on Amazon and compare their features and prices")]
    })
    
    print("\n🛒 E-commerce Analysis:")
    print(ecommerce_result["messages"][-1].content)
    
    return ecommerce_result

# Execute the test
ecommerce_result = await test_ecommerce_scraping()

Output

Testing E-commerce Data Extraction...
==================================================

🛒 E-commerce Analysis:
I found several top-rated wireless headphones on Amazon. Here's a comparison of some of the most popular options:

**1. Sony WH-1000XM4**
*   **Features:** Industry-leading noise cancellation, excellent sound quality, comfortable design, long battery life (up to 30 hours), speak-to-chat functionality, multipoint connection.
*   **Price:** Typically around $350, but often on sale.

**2. Bose QuietComfort 45**
*   **Features:** Renowned noise cancellation, comfortable for long wear, balanced sound, good battery life (up to 24 hours), Aware Mode for hearing surroundings.
*   **Price:** Around $330, with occasional discounts.

**3. Apple AirPods Max**
*   **Features:** Premium build quality, excellent active noise cancellation, immersive spatial audio, seamless integration with Apple devices, comfortable fit.
*   **Price:** Premium pricing, usually around $550.

**4. Sennheiser Momentum 4 Wireless**
*   **Features:** Exceptional sound quality, very long battery life (up to 60 hours), effective noise cancellation, comfortable design, customizable EQ.
*   **Price:** Around $380.

**5. JBL Tune 510BT**
*   **Features:** Affordable option, decent sound quality, up to 40 hours of battery life, lightweight and foldable design, built-in microphone.
*   **Price:** Typically under $50.

**6. Anker Soundcore Life Q20**
*   **Features:** Hybrid active noise cancellation, Hi-Fi stereo sound, deep bass, up to 40 hours of playtime, comfortable earcups.
*   **Price:** Usually around $60.

**To help you choose, consider these factors:**

*   **Budget:** The JBL Tune 510BT and Anker Soundcore Life Q20 are great budget-friendly options. The Sony, Bose, Apple, and Sennheiser models are in the premium price range.
*   **Noise Cancellation:** Sony WH-1000XM4 and Bose QuietComfort 45 are top contenders for the best noise cancellation.
*   **Sound Quality:** Sony, Bose, Sennheiser, and Apple are generally praised for their superior audio performance.
*   **Battery Life:** Sennheiser Momentum 4 Wireless leads with up to 60 hours, while Sony and JBL offer very competitive battery life as well.
*   **Ecosystem:** If you're heavily invested in the Apple ecosystem, AirPods Max offer the most seamless integration.

I recommend checking the specific product pages on Amazon for the most up-to-date pricing and detailed feature comparisons. Would you like me to look up any of these models in more detail?

E-commerce Analysis Capabilities

The e-commerce analysis demonstrates several sophisticated capabilities:

Product Discovery: The agent can search for products within specific categories and identify top-rated options based on customer reviews and ratings.

Feature Extraction: Detailed product specifications, features, and technical details are systematically extracted and organized for easy comparison.

Price Analysis: Current pricing information is gathered and presented alongside historical pricing trends when available.

Comparative Framework: The agent structures its analysis to facilitate decision-making by highlighting key differentiators between products.

Actionable Recommendations: Rather than simply presenting data, the agent provides guidance based on different use cases and budget considerations.

Reddit Discussion Analysis

Reddit Content Extraction and Analysis

In [23]python · cell 31

python

async def test_social_media_simple():
    """
    Test Reddit extraction with a specific approach
    """
    print("Testing Reddit Extraction...")
    print("="*50)
    
    result = await agent.ainvoke({
        "messages": [("human", "Search for 'electric vehicles reddit' and then scrape one of the Reddit discussion pages you find. Show me what people are discussing.")]
    })
    
    print("\n📱 Reddit Analysis:")
    print(result["messages"][-1].content)
    return result

# Test this version
social_simple = await test_social_media_simple()

Output

Testing Reddit Extraction...
==================================================

📱 Reddit Analysis:
Here are some recent discussions from the r/electricvehicles subreddit:

*   **"This $11,000 Chinese EV with semi-solid state batteries is about to shake up the industry"**: This post discusses a new, affordable EV from China that features semi-solid state batteries, potentially disrupting the market.
*   **"Anyone else want an EV just so you no longer have to deal with the bullshit that comes with owning an ICE engine?"**: This user expresses their desire for an EV to avoid the common maintenance and repair issues associated with internal combustion engine (ICE) vehicles, highlighting the peace of mind EVs offer.
*   **"These cars are losing value fast — that's GREAT news for used EV buyers!"**: This article suggests that the rapid depreciation of some EVs is creating good opportunities for those looking to buy used electric vehicles.

It seems the community is actively discussing new EV technology, the benefits of EV ownership over traditional cars, and the used EV market.

The social media analysis showcases several key capabilities:

Platform-Specific Navigation: The agent understands Reddit's structure and can navigate to specific subreddits and discussion threads.

Content Aggregation: Multiple posts and comments are analyzed to identify common themes and discussion topics.

Sentiment Analysis: The agent can interpret the tone and sentiment of discussions, identifying positive, negative, or neutral attitudes toward topics.

Trend Identification: By analyzing multiple discussions, the agent can identify emerging trends and popular topics within communities.

Context Understanding: The agent maintains context about the specific community culture and discussion norms when interpreting content.

Complex Multi-Step Research Workflows

Research Workflow Architecture

Complex research workflows involve several interconnected phases:

Task Decomposition: The agent analyzes complex queries and breaks them down into manageable sub-tasks that can be addressed individually.

Tool Orchestration: Multiple tools are employed in sequence, with each tool's output informing the selection and configuration of subsequent tools.

Information Synthesis: Data from various sources is analyzed, compared, and synthesized into coherent insights that address the original research question.

Quality Validation: The agent evaluates the quality and relevance of gathered information, identifying gaps that require additional research.

Multi-Step Research Execution

In [25]python · cell 35

python

async def test_complex_research():
    """
    Test the agent's ability to conduct multi-step research
    """
    
    print("Testing Complex Multi-Step Research...")
    print("="*50)
    
    # Complex research query
    research_result = await agent.ainvoke({
        "messages": [("human", """
        I need to research the current state of the renewable energy market. Please:
        1. Find recent news about renewable energy developments
        2. Look up major renewable energy companies and their stock performance
        3. Analyze social media sentiment about renewable energy
        4. Provide a comprehensive market overview with key insights
        """)]
    })
    
    print("\n🔬 Complex Research Results:")
    print(research_result["messages"][-1].content)
    
    return research_result

# Execute the test
research_result = await test_complex_research()

Output

Testing Complex Multi-Step Research...
==================================================

🔬 Complex Research Results:
I can help you research the renewable energy market. However, I need some clarification to proceed effectively.

For step 1, could you please specify what you mean by "recent news"? For example, are you interested in news from the past week, month, or a specific event?

For step 2, which major renewable energy companies are you interested in? Knowing specific company names will help me find their stock performance more accurately.

For step 3, analyzing social media sentiment requires access to specific platforms and potentially specialized tools. Could you clarify which social media platforms you'd like me to analyze (e.g., X, Reddit, etc.)?

Once I have this information, I can use my tools to gather the data you need for a comprehensive market overview.

Research Methodology and Planning

The agent's response to complex research requests demonstrates sophisticated planning capabilities:

Requirement Analysis: The agent evaluates the research request and identifies areas where additional specification would improve research quality.

Scope Definition: Rather than proceeding with assumptions, the agent seeks clarification to ensure research efforts are focused and relevant.

Tool Mapping: The agent understands which tools are appropriate for different types of information gathering and outlines its approach.

Quality Assurance: By requesting specific parameters, the agent ensures that the final research output will meet professional standards and user expectations.

This planning approach reflects best practices in professional research methodology and demonstrates the agent's capability to handle enterprise-level research requirements.

Comprehensive Research Assistant Implementation

Research Assistant Architecture

The research assistant function incorporates several design principles that make it suitable for production use:

Parameterized Configuration: Users can specify research scope, source limits, and other parameters to control the research process.

Structured Output: Research results are organized in a consistent format that facilitates further analysis or reporting.

Progress Tracking: The function provides feedback on research progress, helping users understand the complexity and scope of the work being performed.

Quality Controls: Built-in mechanisms ensure that research focuses on high-quality, relevant sources rather than simply maximizing the quantity of information gathered.

Research Assistant Function Implementation

In [28]python · cell 39

python

async def research_assistant(query: str, max_sources: int = 5):
    """
    A comprehensive research assistant using our web scraping agent
    
    Args:
        query (str): The research question or topic
        max_sources (int): Maximum number of sources to analyze
    """
    
    print(f"🔍 Starting research on: {query}")
    print("="*60)
    
    # Enhanced research prompt
    research_prompt = f"""
    Please conduct comprehensive research on: "{query}"
    
    Your research should include:
    1. Current news and developments (last 30 days)
    2. Expert opinions and analysis
    3. Statistical data and trends
    4. Social media sentiment (if relevant)
    5. Key players and companies involved
    
    Use multiple sources and provide a well-structured summary with:
    - Executive summary
    - Key findings
    - Supporting data
    - Sources used
    
    Limit your research to {max_sources} high-quality sources.
    """
    
    result = await agent.ainvoke({
        "messages": [("human", research_prompt)]
    })
    
    print(f"\n📊 Research Complete!")
    print(result["messages"][-1].content)
    
    return result

# Test the research assistant
research_result = await research_assistant("Impact of artificial intelligence on job markets in 2025",5)

Output

🔍 Starting research on: Impact of artificial intelligence on job markets in 2025
============================================================

📊 Research Complete!

Research Assistant Benefits and Applications

The comprehensive research assistant function demonstrates several key benefits:

Scalable Research Framework: The parameterized approach allows the same function to handle research projects of varying scope and complexity.

Consistent Output Structure: The structured prompt ensures that research results follow a predictable format, facilitating integration with other systems or processes.

Quality Focus: By limiting the number of sources and emphasizing quality, the function prioritizes depth over breadth in research coverage.

Professional Standards: The research methodology incorporates best practices from professional research organizations, ensuring outputs meet enterprise requirements.

Customization Capability: The function can be easily modified to address specific industry requirements or research methodologies.

Building an Intelligent Web Scraping Agent with LangGraph and Bright Data + MCP

Overview

Learning Objectives

Architecture Overview

Prerequisites and Environment Setup

Required Accounts and API Keys

Dependency Installation and Configuration

API Key Configuration

Package Installation

Core Imports and Environment Loading

Environment Verification

Understanding the ReAct Agent Framework

Core Principles of ReAct Architecture

Application to Web Scraping

Bright Data MCP Integration

MCP Architecture Benefits

Bright Data MCP Server Configuration

Available Tool Categories

Language Model and Agent Configuration

ReAct Agent Initialization

System Prompt Design Principles

Basic Search Functionality Testing

Understanding Agent Decision-Making

Basic Search Query Execution

Search Result Analysis

Advanced Platform-Specific Data Extraction

E-commerce Platform Analysis

E-commerce Data Extraction and Analysis

E-commerce Analysis Capabilities

Social Media Content Analysis

Reddit Discussion Analysis

Reddit Content Extraction and Analysis

Social Media Analysis Insights

Complex Multi-Step Research Workflows

Research Workflow Architecture

Multi-Step Research Execution

Research Methodology and Planning

Comprehensive Research Assistant Implementation

Research Assistant Architecture

Research Assistant Function Implementation

Research Assistant Benefits and Applications

Building an Intelligent Web Scraping Agent with LangGraph and Bright Data + MCP

Overview

Learning Objectives

Architecture Overview

Prerequisites and Environment Setup

Required Accounts and API Keys

Dependency Installation and Configuration

API Key Configuration

Package Installation

Core Imports and Environment Loading

Environment Verification

Understanding the ReAct Agent Framework

Core Principles of ReAct Architecture

Application to Web Scraping

Bright Data MCP Integration

MCP Architecture Benefits

Bright Data MCP Server Configuration

Available Tool Categories

Language Model and Agent Configuration

ReAct Agent Initialization

System Prompt Design Principles

Basic Search Functionality Testing

Understanding Agent Decision-Making

Basic Search Query Execution

Search Result Analysis

Advanced Platform-Specific Data Extraction

E-commerce Platform Analysis

E-commerce Data Extraction and Analysis

E-commerce Analysis Capabilities

Social Media Content Analysis

Reddit Discussion Analysis

Reddit Content Extraction and Analysis

Social Media Analysis Insights

Complex Multi-Step Research Workflows

Research Workflow Architecture

Multi-Step Research Execution

Research Methodology and Planning

Comprehensive Research Assistant Implementation

Research Assistant Architecture