How to Scrape Real Estate websites with ScrapeGraphAI and LangChain

In today's data-driven world, having high-quality datasets is essential. However, collecting data can often be challenging, expensive, and time-consuming. With tools like ScrapeGraphAI and 🔗 LangChain, creating datasets becomes much simpler and faster. In this post, we'll show you how to use these tools to collect data intelligently.

Why Use ScrapeGraphAI and LangChain?

ScrapeGraphAI and 🔗 LangChain are a powerful combination for collecting and organizing data efficiently. ScrapeGraphAI uses advanced AI to handle even the most complex websites, structuring data clearly while ensuring scalability and cleanliness. 🔗 LangChain complements this by enabling the creation of intelligent AI-powered programs that can maximize the value of collected data. Together, they simplify building robust software solutions that depend on high-quality datasets.

Setting Up Your Environment

Before you begin, make sure you have:

• Python (Version 3.8 or later)
• LangChain-ScrapeGraph: Install with pip install langchain-scrapegraph

Building Your Dataset: A Step-by-Step Guide

Step 1: Define Your Data Requirements

Think about the websites or APIs that contain the data you're looking for. For example, you might want to collect information about houses for sale from sites like Zillow. Consider what specific data points you need:

• Property details (price, bedrooms, bathrooms)
• Location information
• Agent details
• Additional features and amenities

Step 2: Configure ScrapeGraphAI

First, you'll need to set up your environment and authenticate:

python
import os
from getpass import getpass

# Check if the API key is already set
sgai_api_key = os.getenv("SGAI_API_KEY")

if sgai_api_key:
    print("SGAI_API_KEY found.")
else:
    print("SGAI_API_KEY not found.")
    sgai_api_key = getpass("Enter your SGAI_API_KEY: ").strip()
    if sgai_api_key:
        os.environ["SGAI_API_KEY"] = sgai_api_key
        print("SGAI_API_KEY set.")

Next, define your data schema to ensure structured output:

python
from pydantic import BaseModel, Field
from typing import List, Optional

class HouseListingSchema(BaseModel):
    price: int = Field(description="Price of the house in USD")
    bedrooms: int = Field(description="Number of bedrooms")
    bathrooms: int = Field(description="Number of bathrooms")
    square_feet: int = Field(description="Total square footage")
    address: str = Field(description="Address of the house")
    city: str = Field(description="City where the house is located")
    state: str = Field(description="State where the house is located")
    zip_code: str = Field(description="ZIP code of the house")
    tags: List[str] = Field(description="Tags like 'New construction' or 'Large garage'")
    agent_name: str = Field(description="Name of the agent")
    agency: str = Field(description="Real estate agency")

class HousesListingsSchema(BaseModel):
    houses: List[HouseListingSchema] = Field(description="List of house listings")

Step 3: Implement the Scraper

Now you can create your scraper instance and start collecting data:

python
from langchain_scrapegraph.tools import SmartScraperTool

tool = SmartScraperTool()

def extract_house_listings(url: str, api_key: str):
    response = tool.scrapegraph_smartscraper(
        prompt="Extract information about the houses visible on the page",
        url=url,
        api_key=api_key,
        schema=HousesListingsSchema,
    )
    return response["result"]

url = "https://www.homes.com/san-francisco-ca/"
house_listings = extract_house_listings(url, sgai_api_key)

Step 4: Process and Store the Data

Finally, save your data in a format that suits your needs:

python
import json
import pandas as pd

# Save as JSON for flexibility
with open("houses.json", "w") as f:
    json.dump(house_listings, f, indent=2)

# Create a DataFrame for analysis
df = pd.DataFrame(house_listings["houses"])
df.to_csv("houses_for_sale.csv", index=False)

Key Applications and Use Cases

1. Real Estate Market Analysis

   • Track property prices and trends
   • Analyze market dynamics
   • Monitor competitor listings

2. E-Commerce Intelligence

   • Price optimization
   • Competitor monitoring
   • Product trend analysis

3. Healthcare Data Collection

   • Medical research aggregation
   • Healthcare provider information
   • Treatment cost analysis

4. Financial Market Research

   • Investment opportunities
   • Market sentiment analysis
   • Economic indicators tracking

For more examples and detailed use cases, check out our 📚 ScrapeGraphAI Cookbook which contains ready-to-use recipes for various scraping scenarios.

Best Practices for Efficient Scraping

When using ScrapeGraphAI for data collection, keep these tips in mind:

• Respect Rate Limits: Always implement appropriate delays between requests
• Validate Data: Always verify the quality and completeness of collected data

Conclusion

The combination of ScrapeGraphAI and 🔗 LangChain provides a powerful solution for modern data collection needs. Whether you're analyzing real estate markets, tracking e-commerce trends, or gathering research data, these tools make the process efficient and reliable.