How Do You Use Web Scraping With LangChain?

TL;DR

You use web scraping with LangChain by wrapping a scraping call as a LangChain tool and giving it to an agent, or by using a document loader to pull page content into a chain. The agent then decides when to scrape, and the returned text or structured data flows into its reasoning or a retrieval index.

Two Integration Patterns

As an agent tool. For agents that decide when to fetch, wrap a scraping function as a @tool. The agent calls it during reasoning, the same pattern as adding web search:

from langchain.tools import tool
 
@tool
def scrape_page(url: str) -> str:
    """Fetch a web page and return its clean markdown content."""
    return scrape_api(url, format="markdown")   # your scraping provider
 
agent = create_agent(model, tools=[scrape_page])

As a document loader. For RAG pipelines, use scraping to load pages as documents, then split, embed, and index them. Here scraping is an ingestion step, not an agent decision:

docs = load_pages(urls)                 # scrape each URL to clean text
chunks = text_splitter.split_documents(docs)
vectorstore.add_documents(chunks)       # now retrievable in a chain

Which Pattern to Use

Use the tool pattern when the agent should decide dynamically what to fetch (research, verification, open-ended tasks). Use the loader pattern when you are building a fixed knowledge base ahead of time. Many applications use both: a loader for the static corpus, a tool for live lookups the corpus cannot cover.

Getting the Output Right

LangChain passes whatever the tool returns straight into the model's context, so return clean markdown or structured JSON, not raw HTML. Raw markup burns tokens and degrades reasoning. A scraping layer that returns model-ready text is the difference between an agent that reasons well and one that chokes on <div> soup.

Key Takeaways

Wrap scraping as a LangChain tool for agents, or a document loader for RAG.
Tool pattern for dynamic fetching; loader pattern for prebuilt knowledge bases.
Return clean markdown or JSON, since LangChain feeds tool output directly to the model.

How ScrapeGraphAI Handles This

ScrapeGraphAI ships a LangChain integration exposing scrape, extract, and search as ready-made tools that return clean markdown or typed JSON, so you register them with an agent or chain without writing wrapper code.