ScrapeGraphAIScrapeGraphAI
MCP Server: Connect Claude and AI Assistants to the Web
//ScrapeGraphAI\\

MCP Server: Connect Claude and AI Assistants to the Web

Author 1

Marco Vinciguerra

AI assistants like Claude are powerful. But they have a critical weakness: they cannot see the live web. No current prices. No fresh news. No real-time data extraction. That limitation ends here.

The Model Context Protocol (MCP) bridges this gap. ScrapeGraphAI's MCP Server connects Claude, Cursor IDE, and other AI tools directly to web scraping capabilities.

What is MCP?

The Model Context Protocol is an open standard enabling AI models to interact with external tools and data sources. Consider it a universal adapter that extends what AI assistants can do.

With ScrapeGraphAI's MCP Server, AI assistants gain the power to:

  • Scrape any website on demand
  • Extract structured data from web pages
  • Convert pages to markdown for processing
  • Crawl entire sites for comprehensive data
  • Search the web and extract results

Why MCP + ScrapeGraphAI?

Claude and other LLMs offer built-in web browsing. It falls short:

Feature Built-in Browser ScrapeGraphAI MCP
JavaScript rendering Limited Full support
Structured extraction No Yes
Anti-bot bypass No Yes
Multi-page crawling No Yes
Custom prompts No Yes
Rate limiting Strict Flexible

ScrapeGraphAI transforms Claude into a proper web scraping tool.

Setting Up ScrapeGraphAI MCP Server

Step 1: Install the MCP Server

The fastest method uses Smithery:

npx -y @smithery/cli install @anthropic/claude-desktop @anthropic/mcp-server-scrapegraph
     --config ./mcp-config.json

Or add this to your Claude Desktop configuration manually:

{
  "mcpServers": {
    "scrapegraph": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-scrapegraph"],
      "env": {
        "SCRAPEGRAPH_API_KEY": "your-api-key-here"
      }
    }
  }
}

Step 2: Get Your API Key

  1. Go to dashboard.scrapegraphai.com
  2. Sign up or log in
  3. Navigate to API Keys
  4. Create a new key

Step 3: Restart Claude Desktop

Restart Claude Desktop after configuration. ScrapeGraphAI tools will appear in the tools menu.

What You Can Do

Ask Claude to Scrape Websites

Once configured, make natural requests:

"Scrape the pricing page of Notion and extract all plan names, prices, and features"

Claude uses ScrapeGraphAI to fetch the page and return structured data.

Real-Time Research

"What are the top 10 trending repositories on GitHub right now? Extract the repo name, description, stars, and language."

Claude pulls live data rather than relying on stale training data.

Competitive Analysis

"Go to Shopify's pricing page and Squarespace's pricing page. Extract all pricing tiers and create a comparison table."

Claude visits both sites and synthesizes the findings.

Data Collection

"Crawl the blog section of techcrunch.com and extract all article titles and summaries from the last 7 days."

Claude crawls multiple pages to gather comprehensive data.

Available MCP Tools

ScrapeGraphAI's MCP Server exposes several tools. Here is how to use them programmatically with Python:

smartscraper

Extract structured data from any URL using AI.

from scrapegraph_py import Client
 
# Initialize the client
client = Client(api_key="your-api-key-here")
 
# SmartScraper request
response = client.smartscraper(
    website_url="https://www.notion.so/pricing",
    user_prompt="Extract all plan names, prices, and key features"
)
 
print("Result:", response)

markdownify

Convert any webpage to clean markdown for processing.

from scrapegraph_py import Client
 
# Initialize the client
client = Client(api_key="your-api-key-here")
 
# Markdownify request
response = client.markdownify(
    website_url="https://docs.python.org/3/tutorial/index.html"
)
 
print("Result:", response)

searchscraper

Search the web and extract structured data from results.

from scrapegraph_py import Client
 
# Initialize the client
client = Client(api_key="your-api-key-here")
 
# SearchScraper request
response = client.searchscraper(
    user_prompt="Find the best project management tools in 2025 with pricing and
        features",
    num_results=5
)
 
print("Result:", response)

Structured Output with Schemas

When using ScrapeGraphAI programmatically, Pydantic (Python) or Zod (JavaScript) schemas guarantee typed, consistent responses:

from scrapegraph_py import Client
from pydantic import BaseModel, Field
from typing import List
 
class PricingTier(BaseModel):
    name: str = Field(description="Plan name")
    price: str = Field(description="Monthly price")
    features: List[str] = Field(description="Included features")
    cta: str = Field(description="Call to action text")
 
class PricingPage(BaseModel):
    company: str = Field(description="Company name")
    tiers: List[PricingTier] = Field(description="All pricing tiers")
    currency: str = Field(description="Currency used")
 
client = Client(api_key="your-api-key-here")
 
response = client.smartscraper(
    website_url="https://www.notion.so/pricing",
    user_prompt="Extract all pricing tiers with features",
    output_schema=PricingPage
)
 
pricing = PricingPage(**response["result"])
for tier in pricing.tiers:
    print(f"{tier.name}: {tier.price}")

scrape

Get the raw HTML content of any webpage.

from scrapegraph_py import Client
 
# Initialize the client
client = Client(api_key="your-api-key-here")
 
# Scrape request
response = client.scrape(
    website_url="https://news.ycombinator.com"
)
 
print("Result:", response)

Use Cases in Cursor IDE

Developers using Cursor can leverage MCP for:

Documentation Research

"Scrape the React documentation and summarize the hooks API"

Package Comparison

"Compare the npm pages for axios, fetch, and got. Which has more downloads and better maintenance?"

API Discovery

"Extract all API endpoints from Stripe's API reference page"

Bug Research

"Search for issues related to 'memory leak' in the Next.js GitHub issues and summarize the common causes"

Use Cases for Claude Desktop

Personal Research

"What are the current mortgage rates from major banks? Scrape Chase, Wells Fargo, and Bank of America mortgage pages."

Shopping Assistance

"Find the best price for a Sony WH-1000XM5 across Amazon, Best Buy, and Walmart"

For automated price tracking, see our price monitoring bot guide.

Travel Planning

"Scrape the top 10 restaurants in Rome from TripAdvisor with their ratings and cuisine types"

News Monitoring

"What are the top stories on Hacker News right now?"

Best Practices

Be Specific in Requests

Vague requests produce vague results. Tell Claude exactly what you need:

"Extract the product name, price, rating, and number of reviews"

Not:

"Get the product info"

Use for Real-Time Data

MCP delivers the most value when you need current information beyond Claude's training data.

Combine with Claude's Analysis

Scrape first. Then ask Claude to analyze, compare, or summarize the results.

Respect Rate Limits

ScrapeGraphAI handles most complexity, but avoid hammering endpoints with excessive requests.

Supported AI Clients

ScrapeGraphAI MCP works with:

  • Claude Desktop - Full MCP support
  • Cursor IDE - Built-in MCP integration
  • Continue.dev - VSCode AI assistant
  • Any MCP-compatible client - Growing ecosystem

Get Started Today

Turn your AI assistant from a static knowledge base into a live web intelligence tool. ScrapeGraphAI MCP Server makes Claude your personal web research assistant.

Ready to unlock real-time web access? Sign up for ScrapeGraphAI and follow our MCP setup guide to get started in minutes.

Give your AI Agent superpowers with lightning-fast web data!

Get Started!