TL;DR
Selector-based scraping is not dead, but it is no longer the default for every job. Hardcoded CSS and XPath selectors are still fast and cheap on stable pages, yet they break whenever a site changes its markup. AI extraction that targets meaning instead of structure has taken over the cases where sites change often or vary widely.
What "Selector-Based" Means
Traditional scraping locates data by structure: div.product-card > span.price. You write CSS selectors or XPath that point at exact positions in the HTML, and the parser pulls the text there. It works well when the structure is known and stable, and it is cheap: no model calls, just string matching.
Why It Started to Break Down
Two pressures exposed the limits of selectors:
- Fragility. A selector encodes assumptions about markup. The day a site renames
pricetoproduct-priceor reorders its divs, the scraper returns nothing or the wrong field, often silently. Maintaining selectors across dozens of sites becomes a running cost. - Scale across layouts. Scraping 200 different sites means writing and maintaining 200 selector sets. There is no reuse, because every site's markup differs.
What Replaced It Where It Broke
AI extraction reads a page the way a person would: you describe the fields you want ("product name, price, availability") and the model finds them regardless of the exact tags or class names. See AI-powered extraction vs traditional parsing. This survives markup changes and generalizes across sites, at the cost of a model call per page.
The Honest Verdict
Selectors remain the right tool when you scrape one stable site at high volume and want minimal cost per page. AI extraction wins when sites change often, when you scrape many different layouts, or when you value resilience over the lowest possible per-page cost. Most mature stacks use both: selectors for the stable high-volume targets, AI for the long tail and the shifting ones.
Key Takeaways
- Selector scraping is alive for stable, high-volume, single-site jobs.
- It breaks silently on markup changes and does not generalize across layouts.
- AI extraction covers the volatile and multi-site cases; many stacks use both.
How ScrapeGraphAI Handles This
ScrapeGraphAI is built on the AI-extraction approach: the extract endpoint targets data by meaning against a schema you define, so it holds up when a site changes its HTML and works across many different layouts without per-site selector maintenance.