How to Create a Multi-Agent System with LangGraph, CrewAI and LlamaIndex

If you're building autonomous agents for data extraction, reasoning, or task automation, and you're looking to scale them intelligently — this tutorial is for you.

In this post, you'll learn how to integrate LangGraph, LlamaIndex, and CrewAI into a seamless multi-agent system that’s modular, memory-aware, and built for complex workflows. This integration unlocks dynamic agent orchestration, tool usage, and semantic memory at scale.

---

🧠 Why Multi-Agent Systems?

Multi-agent systems are AI architectures composed of independent agents that:

• Specialize in different tasks (e.g., reasoning, data extraction, summarization)
• Communicate with each other in a structured flow
• Act autonomously or cooperatively to solve complex problems

When combined with LangGraph’s graph-based orchestration, LlamaIndex’s semantic memory and retrieval, and CrewAI’s task delegation and personas, you get a production-ready AI system that can:

✅ Ingest knowledge
✅ Route intelligently between agents
✅ Act on structured/unstructured data
✅ Generate human-like outputs

---

🧰 Stack Overview

Tool	Role
LangGraph	Graph-based control flow for agents
LlamaIndex	Long-term memory & retrieval-augmented generation
CrewAI	Define agent roles, tasks, tools, and collaboration
LangChain	LLM wrappers, tools, and memory integration
ScrapeGraphAI	(Optional) Agent-powered web data pipelines

---

🔧 Prerequisites

bash
pip install langgraph crewai llama-index langchain openai scrapegraph-py

Make sure you have an LLM set up (like OpenAI or Ollama).

---

1. 🔗 Create CrewAI Agents

python
from crewai import Agent, Task, Crew

researcher = Agent(
    role="Researcher",
    goal="Collect and summarize the latest market trends",
    backstory="An expert at scraping and summarizing complex data",
)

analyst = Agent(
    role="Analyst",
    goal="Draw insights from summarized content",
    backstory="A strategic thinker who uses structured data to generate recommendations",
)

---

2. 🧠 Load LlamaIndex for Context Memory

python
from llama_index import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

You can plug this query engine into your agents to give them RAG capabilities.

---

3. 🕸️ Design the LangGraph Flow

python
from langgraph.graph import StateGraph

def collect_data(state):
    content = researcher.run("Scrape AI trends from the web")
    return {**state, "content": content}

def analyze_data(state):
    summary = query_engine.query(state["content"])
    insights = analyst.run(f"Analyze the summary: {summary}")
    return {"output": insights}

workflow = StateGraph(dict)
workflow.add_node("Scrape", collect_data)
workflow.add_node("Analyze", analyze_data)
workflow.set_entry_point("Scrape")
workflow.add_edge("Scrape", "Analyze")
workflow.set_finish_point("Analyze")

graph = workflow.compile()
result = graph.invoke({})
print(result["output"])

---

⚡ Bonus: Add Dynamic Agent Routing

python
def router(state):
    if "financial" in state.get("input", "").lower():
        return "FinanceAgent"
    return "GeneralAgent"

workflow.add_node("Router", router)

workflow.add_node("FinanceAgent", analyze_data)
workflow.add_node("GeneralAgent", collect_data)

workflow.add_conditional_edges("Router", {
    "FinanceAgent": "Analyze",
    "GeneralAgent": "Scrape"
})

---

🔄 Real-Time Use Case: Web Data Extraction with ScrapeGraphAI

ScrapeGraphAI is the perfect pairing here — you can use it as a tool for your agents:

python
from scrapegraphai.graphs import SmartScraperGraph

def scrape_with_scrapegraph(url):
    sg = SmartScraperGraph(config={"llm": "gpt-4"})
    return sg.run({"url": url, "question": "What are the main product categories?"})

You can expose this as a tool in CrewAI, or plug into LangGraph directly.

---

🛠️ How to Add ScrapeGraphAI as a Tool in Your Python Agents

You can also use the official ScrapeGraphAI Python client to interact with the ScrapeGraph API directly:

python
from scrapegraph_py import Client
from scrapegraph_py.logger import sgai_logger

# Set logging level to INFO
sgai_logger.set_logging(level="INFO")

# Initialize the client with your API key (keep your key secret!)
sgai_client = Client(api_key="sgai-XXXX-XXXX-XXXX-XXXXXXXXXXXX")

try:
    # Make a SmartScraper request
    response = sgai_client.smartscraper(
        website_url="https://example.com",
        user_prompt="Extract webpage information"
    )

    # Print the response data
    print(f"Request ID: {response['request_id']}")
    print(f"Result: {response['result']}")

    # Optional: print reference URLs if available
    if response.get('reference_urls'):
        print("Reference URLs:")
        for url in response['reference_urls']:
            print(f" - {url}")

finally:
    # Close the client session
    sgai_client.close()

Note: Replace

text

"sgai-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

with your actual ScrapeGraphAI API key, and never expose your API key publicly.

---

📊 Applications

• Financial trend analysis
• Research assistants
• Autonomous agents for scraping and summarization
• Business intelligence pipelines
• Scientific Q&A systems

---

🎯 SEO Takeaways

• Use LangGraph for orchestration
• Use CrewAI for multi-agent collaboration
• Use LlamaIndex for contextual memory and document search
• Use ScrapeGraphAI for autonomous data extraction

---

🔚 Conclusion

By integrating LangGraph + CrewAI + LlamaIndex, you're no longer building simple chatbots — you're engineering intelligent, collaborative, task-driven agents.

This is the future of AI infrastructure.

---

🔗 Ready to build?

Start your agent system today at scrapegraph.ai
→ or ask me below to generate a GitHub-ready template 🚀

---

❓ Frequently Asked Questions (FAQs)

What is LangGraph used for?

LangGraph is a framework for defining stateful AI workflows as graphs. It enables complex reasoning paths, branching logic, and persistent memory flows between agents.

---

What is the difference between CrewAI and LangGraph?

• CrewAI manages agent definitions, tasks, and interactions.
• LangGraph handles the orchestration and flow between those agents, allowing dynamic routing and state transitions.

Using both together provides a flexible and powerful system for building AI agents that collaborate and reason over multiple steps.

---

How does LlamaIndex improve my agents?

LlamaIndex provides long-term memory and context-aware retrieval, allowing agents to access large corpora of documents and use relevant context in real-time. Perfect for RAG-based systems.

---

Can I use this for web scraping?

Yes! You can integrate ScrapeGraphAI as an agent tool or standalone service to autonomously scrape, structure, and query data from the web.

---

Is this production ready?

Yes — with proper error handling and LLM token/latency optimization, this stack can be used in production to power intelligent assistants, dashboards, search agents, and more.

---