X (Twitter) Data Extraction with ScrapeGraphAI

TL;DR

Extract X (Twitter) profile data and post metrics with ScrapeGraphAI extract using natural language prompts instead of selectors.

• Profiles and posts: bios, follower counts, tweet text, hashtags, media URLs, and engagement metrics
• A few lines of Python: pass a URL and describe what you want; add a Pydantic schema when you need typed, consistent output

X (formerly Twitter) is one of the highest-signal sources for real-time sentiment, competitor activity, and public reaction to events. The hard part has never been finding the data, it is getting it out of a JavaScript-heavy, anti-bot-protected page in a structured form. ScrapeGraphAI extract turns a public X URL and a plain-English prompt into structured JSON, and handles rendering and extraction behind the scenes.

Why X Data Matters

What makes X distinct from other social platforms is timing. Reactions land within minutes of an event, and engagement on a post is a fast, public proxy for how a message is being received. That immediacy is exactly why teams pull X data on a schedule rather than as a one-off. X data is useful across a few concrete workflows:

✅ Real-time market insights - Track trending topics and sentiment in your industry

✅ Competitive analysis - Monitor competitor engagement and content cadence

✅ Public opinion research - Analyze reactions to events, products, or campaigns

✅ Influencer research - Evaluate potential partnerships through real engagement metrics

✅ Content strategy - Understand what content resonates with your audience

If your goal is broader than a single platform, pair this with market research scraping to combine X signals with other sources.

Available X Data

ScrapeGraphAI extract reads both profile and post data from a public X page. Here is what you can pull:

Profile Information

• Basic details

  • X ID and profile URL
  • Profile name and display name
  • Biography and location
  • Profile image and banner image
  • Date joined
  • External links

• Account status

  • Verification status
  • Business/Government account flags
  • Category name (for business accounts)

• Engagement metrics

  • Follower count
  • Following count
  • Posts count
  • Subscription count

Post Data

• Content

  • Post text and description
  • Hashtags
  • Photo and video URLs
  • Post URL and ID
  • Posting date

• Engagement metrics

  • Likes
  • Replies
  • Reposts
  • View count

Extracting Profile Data

The simplest case is a plain prompt against a profile URL. The AI reads the rendered page and returns whatever your prompt asks for as JSON.

import json
from scrapegraph_py import ScrapeGraphAI
 
sgai = ScrapeGraphAI()  # uses SGAI_API_KEY env var
 
url_list = [
    "https://x.com/elonmusk",
    "https://x.com/SenatorBaldwin",
]
 
for url in url_list:
    result = sgai.extract(
        "Extract profile details: display name, bio, follower count, following count, and verification status",
        url=url,
    )
 
    if result.status == "success":
        print(json.dumps(result.data.json_data, indent=2))

Typed Output With a Schema

For anything that feeds a database or dashboard, define a Pydantic schema so every response has the same shape and types. This is the difference between exploratory scraping and something you can run in production.

from pydantic import BaseModel, Field
from scrapegraph_py import ScrapeGraphAI
 
sgai = ScrapeGraphAI()  # uses SGAI_API_KEY env var
 
class XProfile(BaseModel):
    display_name: str = Field(description="Profile display name")
    handle: str = Field(description="X handle, including the @")
    bio: str = Field(description="Profile biography")
    followers: int = Field(description="Follower count")
    following: int = Field(description="Following count")
    verified: bool = Field(description="Whether the account is verified")
 
result = sgai.extract(
    "Extract the profile details",
    url="https://x.com/elonmusk",
    schema=XProfile.model_json_schema(),
)
 
if result.status == "success":
    print(result.data.json_data)

For a deeper walkthrough of prompts, schemas, and response handling, see Mastering ScrapeGraphAI.

Extracting Posts and Engagement Metrics

Profiles tell you who someone is; posts tell you what is working. To analyze content performance you usually want the post text alongside its engagement numbers, and a nested schema keeps that structured even when a profile has many posts.

from typing import List
from pydantic import BaseModel, Field
from scrapegraph_py import ScrapeGraphAI
 
sgai = ScrapeGraphAI()  # uses SGAI_API_KEY env var
 
class Post(BaseModel):
    text: str = Field(description="Post text")
    hashtags: List[str] = Field(description="Hashtags used in the post")
    likes: int = Field(description="Like count")
    replies: int = Field(description="Reply count")
    reposts: int = Field(description="Repost count")
    views: int = Field(description="View count")
    posted_at: str = Field(description="Posting date")
 
class ProfileWithPosts(BaseModel):
    display_name: str = Field(description="Profile display name")
    followers: int = Field(description="Follower count")
    posts: List[Post] = Field(description="Recent posts with engagement metrics")
 
result = sgai.extract(
    "Extract the profile name, follower count, and the recent posts with their engagement metrics",
    url="https://x.com/elonmusk",
    schema=ProfileWithPosts.model_json_schema(),
)
 
if result.status == "success":
    data = result.data.json_data
    for post in data["posts"]:
        engagement = post["likes"] + post["replies"] + post["reposts"]
        print(f"{engagement:>7} engagement | {post['text'][:60]}")

Because the schema is nested, each post comes back as a typed record. That makes it straightforward to rank posts by engagement, filter by hashtag, or push the whole set into a dataframe for analysis.

Handling Scale and Rate Limits

A single profile is one request. Real social-listening jobs run across hundreds of accounts, and that is where a few habits keep you out of trouble:

• Batch deliberately. Iterate over your URL list and process results as they return rather than firing everything at once. This keeps memory flat and makes partial failures easy to retry.
• Retry transient failures only. Check result.status on every call. Retry network errors and timeouts with backoff; do not retry a page that is simply private or removed.
• Cache what you already have. Profiles change slowly. If you refresh a cohort daily, store the last result and skip accounts that have not changed if your use case tolerates it.
• Narrow the prompt. The fewer fields you request, the faster and cheaper each call is. Pull the pinned post separately from a full timeline sweep instead of always asking for both.

import time
 
def extract_profiles(urls, schema, max_retries=2):
    results = []
    for url in urls:
        for attempt in range(max_retries + 1):
            result = sgai.extract("Extract the profile details", url=url, schema=schema)
            if result.status == "success":
                results.append(result.data.json_data)
                break
            if attempt < max_retries:
                time.sleep(2 ** attempt)  # exponential backoff
    return results

Worked Example: Social Listening on a Topic

Say you want to gauge sentiment around a product launch. Collect the profiles and recent posts of the accounts driving the conversation, then aggregate the engagement to see which messages actually landed.

accounts = [
    "https://x.com/handle_one",
    "https://x.com/handle_two",
    "https://x.com/handle_three",
]
 
profiles = extract_profiles(accounts, ProfileWithPosts.model_json_schema())
 
# Rank every collected post by total engagement
all_posts = [
    {"author": p["display_name"], **post}
    for p in profiles
    for post in p.get("posts", [])
]
top = sorted(all_posts, key=lambda x: x["likes"] + x["reposts"], reverse=True)
 
for post in top[:5]:
    print(f"{post['author']}: {post['likes']} likes / {post['reposts']} reposts")
    print(f"  {post['text'][:80]}\n")

From here the structured output feeds directly into a sentiment model, a dashboard, or a weekly report. The point is that once the data is typed and consistent, the analysis is ordinary Python rather than fragile HTML parsing.

Best Practices for X Data Extraction

1. Be specific in your prompt. "Extract bio, follower count, and the text of the pinned post" returns cleaner data than "extract everything." Name the fields you actually need.
2. Use a schema for repeatable jobs. A Pydantic model gives you typed, validated output and makes downstream parsing trivial.
3. Extract only the fields you use. Narrower prompts are faster and cheaper than pulling the full page every time.
4. Respect platform terms and rate limits. Scrape public data, stagger requests, and handle sensitive information responsibly. See our guide on scraping social media legally.

Why Traditional Scraping Struggles With X

X is one of the harder targets for a hand-written scraper, and it is worth understanding why before you invest in one:

• Everything renders client-side. The initial HTML is nearly empty; posts, counts, and profile fields are injected by JavaScript after load. A plain requests + BeautifulSoup script sees almost nothing. extract renders the page first, so the data is actually present when extraction runs.
• The markup changes often. Class names are obfuscated and rotate frequently. CSS selectors that work today break next week. Natural language prompts describe the data by meaning, not by DOM position, so they survive layout changes that would break a selector-based scraper.
• Anti-bot measures are aggressive. Raw requests are throttled or blocked quickly. Extraction handles the fetch layer for you, so you are not maintaining proxy pools and header rotation by hand.
• Engagement numbers are formatted for humans. Counts show up as "1.2M" or "34K" in the DOM. A schema field typed as int lets the model normalize those into real numbers instead of you writing brittle parsing code.

This is the core argument for AI extraction over classic scraping on a site like X: you spend your time on the data, not on keeping a fragile scraper alive. For the broader tradeoff, see AI Agent Web Scraping.

Scraping vs the Official X API

The official X API is the right tool when you have authorized access and need firehose-scale, real-time streams or write access. It comes with paid tiers, quota tiers, and an application process. Scraping public pages with extract is a better fit when you need a quick, read-only snapshot of public profiles and posts without managing API credentials or committing to a subscription tier.

A practical rule: if you need authenticated, guaranteed, high-volume streaming, use the API. If you need flexible, on-demand extraction of public data across many accounts with a prompt you can change on the fly, scraping is faster to stand up and cheaper to iterate on. Many teams use both, the API for accounts they own and scraping for public competitor and market research.

Beyond X: Other Social Platforms

The same extract pattern works across social platforms. If your project spans more than X, these companion guides use the identical approach:

• Facebook data extraction
• Instagram data extraction
• LinkedIn data extraction

Frequently Asked Questions

Can I extract data from a private X account?

No. ScrapeGraphAI extracts publicly visible data only. If a profile or post is private or requires login to view, it is out of scope, and you should rely on the official X API for authorized access to that data.

Do I need to handle JavaScript rendering myself?

No. X is a JavaScript-heavy single-page app, and extract renders the page before extraction. You pass a URL and a prompt; the rendering, anti-bot handling, and parsing happen server-side.

How do I get consistent output across many profiles?

Pass a Pydantic schema to extract. The schema enforces field names and types on every response, so a batch of profiles returns uniform records instead of prompt-dependent shapes. See the typed-output example above.

Is scraping X legal?

Extracting public data is generally permitted, but the specifics depend on your jurisdiction, the data involved, and X's terms of service. Read Is Web Scraping Legal? and confirm your use case before running at scale.

Conclusion

X data is valuable for market research, social listening, and competitor tracking, but the platform's structure makes it hard to scrape with traditional tools. ScrapeGraphAI extract removes that friction: describe the fields you want in plain English, add a schema when you need typed output, and get structured, ready-to-use data back.

Related Articles

• Mastering ScrapeGraphAI - Deep dive into the extraction platform
• AI Agent Web Scraping - Build AI-powered scraping workflows
• Building Intelligent Agents - Create automation agents with ScrapeGraphAI
• Market Research Scraping - Combine social signals into research pipelines
• Is Web Scraping Legal? - Understand the legal considerations