ScrapeGraphAIScrapeGraphAI
...
Claude x ScrapeGraphAI

Claude x ScrapeGraphAI

Marco Vinciguerra

Marco Vinciguerra

@Marco Vinciguerra

Anthropic has just released new versions of their Frontier models Claude 4 Sonnet and Claude 4 Opus. They come with better context handling and they are developer friendly and powerful than ever. They come with enhanced tool alling abilities which makes them great for agentic workflows and it is considered the best coding model available in the market right now.

Enhanced Agentic Capabilities

Both Claude Opus 4 and Sonnet 4 reduce shortcut-taking by 65% compared to Sonnet 3.7, making them reliable for mission-critical tasks like multi-step workflows and cross-department coordination

Here are a few reasons why Claude 4 stands out in tool calling:

  • Accurate Interpretation: Claude 4 can accurately interpret user requests and determine which tools or functions are necessary to fulfill those requests.
  • Seamless Integration: It integrates smoothly with a variety of external tools and APIs, allowing for a wide range of functionalities.
  • Efficient Execution: Claude 4 executes tool calls efficiently, minimizing delays and ensuring quick response times.

What is Tool Calling?
Tool calling refers to an AI model's ability to understand when and how to use external functions or APIs to fulfill a user's request. It's a pivotal feature that enables AI to move beyond simple information retrieval to performing actual tasks in the real world.

Potential Applications

Claude 4's proficiency in tool calling opens up a plethora of potential applications, including but not limited to:

  • Complex Tool Calls: Claude 4 can handle complex tool calls with type safe arguments.
  • Web Scraping: Claude 4 can use web scraping tools like ScrapeGraphAI to scrape the websites and use those results for data analysis.
  • Data Analysis: Interacting with data analysis tools to generate reports.

Using Claude 4 to use ScrapeGraphAI tool

This example demonstrates how to construct a basic autonomous agent leveraging the capabilities of Claude 4 in conjunction with the ScrapeGraphAI library. The primary function of this agent will be to automatically extract information from websites. We will walk through the essential steps involved in setting up this agent, including configuring Claude 4, integrating ScrapeGraphAI for intelligent website navigation and data extraction, and defining the agent's core task of web scraping. This will provide a clear, step-by-step illustration of how these powerful tools can be combined to automate the process of gathering data from the internet.

What is ScrapeGraphAI?

ScrapeGraphAI is an API for extracting data from the web with the use of AI.
So it will help you with the data collection part which is focussed on scraping and aggregating information from various sources to gain insights from. This service will fit in your data pipeline perfectly because of the easy to use apis that we provide which are fast and accurate.And it's all AI powered.

1.Lets Import the Packages

import os  
from dotenv import load_dotenv  
import logging  
from scrapegraph_py import Client  
from langchain_anthropic import ChatAnthropic  
from langgraph.graph import MessagesState, StateGraph, START  
from langgraph.prebuilt import tools_condition, ToolNode  
from langchain_core.messages import HumanMessage, SystemMessage

2.Load the api keys

# Load environment variables from .env file  
load_dotenv()
# Access environment variables  
scrape_graph_api_key = os.getenv('SCRAPEGRAPH_API_KEY')  
anthropic_api_key = os.getenv('ANTHROPIC_API_KEY')
 
# Set up logging  
logging.basicConfig(level=logging.INFO)

3.Define a Tool to use ScrapeGraphAI

# Define the scrape_website function  
def scrape_website(website_url, user_prompt) -> dict:  
    """  
    Perform a scraping request on a website using ScrapeGraphAI.
    Parameters:  
    - website_url (str): The URL of the website to scrape  
    - user_prompt (str): The data extraction prompt
    Returns:  
    - A dictionary containing the scraped data  
    """  
    try:  
        sgai_client = Client(api_key=scrape_graph_api_key)  
        logging.info("Creating ScrapeGraphAI client")  
          
        response = sgai_client.smartscraper(  
            website_url=website_url,  
            user_prompt=user_prompt  
        )  
          
        sgai_client.close()  
        result = response.values()  
        # Extract the data dictionary from the response (usually at index 4)  
        data_dict = list(result)[4]  
          
        # Ensure the return is a dictionary  
        if not isinstance(data_dict, dict):  
            data_dict = {"data": data_dict}  
          
        return data_dict  
      
    except Exception as e:  
        logging.error(f"Error in scrape_website: {str(e)}")  
        raise

4.Create the agent:

# Create the agent  
tools = [scrape_website]  
llm = ChatAnthtropic(model="claude-opus-4-20250514", api_key=anthropic_api_key)  
llm_with_tools = llm.bind_tools(tools)
 
# System message for the assistant  
sys_msg = SystemMessage(content="""You are a helpful assistant tasked with performing   
                        Web scraping.You have access to the following tools:  
- scrape_website: to scrape a website  
""")
 
# Assistant function  
def assistant(state: MessagesState):  
    return {"messages": [llm_with_tools.invoke([sys_msg] + state["messages"])]}
# Build the graph  
builder = StateGraph(MessagesState)  
builder.add_node("assistant", assistant)  
builder.add_node("tools", ToolNode(tools))  
builder.add_edge(START, "assistant")  
builder.add_conditional_edges(  
    "assistant",  
    tools_condition,  
)  
builder.add_edge("tools", "assistant")  
react_graph = builder.compile()

Conclusion

Claude 4's enhanced tool calling capabilities, especially when combined with specialized tools like ScrapeGraphAI, open up new possibilities for automating complex tasks such as web scraping and data analysis.

By accurately interpreting requests and integrating seamlessly with external APIs, Claude 4 empowers users to build efficient and reliable agentic workflows.

This combination significantly streamlines data extraction and processing, making it an invaluable asset for developers and researchers needing robust automated solutions.

FAQ

How to obtain an API key for ScrapeGraphAI?

To obtain an API key: Visit the https://dashboard.scrapegraphai.com/ . Create an account or log in if you already have one. Generate a new API key from your user profile.

How to obtain an API key for Claude 4?

To obtain an API key: Visit the https://www.anthropic.com/api Create an account or log in if you already have one. Generate a new API key.

What services does ScrapeGraphAI offer?

ScrapeGraphAI offers 3 services searchscraper , smartscraper and markdownify.
Checkout https://docs.scrapegraphai.com/introduction

Does ScrapeGraphAI have integration with No code platforms?

Yes ScrapeGraphAI has integrations with many no code platforms like n8n, zapier, bubble etc.

Related Resources

Want to learn more about AI model integration and web scraping? Explore these guides:

These resources will help you understand how to effectively integrate AI models with web scraping tools and build powerful data extraction systems.

Give your AI Agent superpowers with lightning-fast web data!

Get Started