How Do Web Scraping APIs Convert HTML to JSON?

TL;DR

Web scraping APIs convert HTML to JSON by fetching the page, cleaning the markup, and mapping its content to fields. Selector-based tools read fixed CSS or XPath paths into keys; AI-based tools read the page against a schema you define and return typed JSON, holding up even when the HTML structure changes.

The Two Conversion Methods

Selector-based mapping. The classic approach: the API (or your code) applies CSS selectors or XPath to pull values from known positions and assembles them into a JSON object. It is fast and cheap but brittle, because each key depends on a fixed path that breaks when the site changes its markup.

# selector-based: each field is a hardcoded path
{"price": soup.select_one(".price").text,
 "title": soup.select_one("h1").text}

AI-based extraction. The modern approach: you define the shape of the JSON you want (a schema), and a model reads the cleaned page and fills it in by meaning rather than position. It survives markup changes and generalizes across different layouts. See AI-powered extraction vs traditional parsing.

The Role of the Schema

A schema is what turns a vague request into reliable JSON. Instead of "get the product info", you declare the exact fields and types:

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool
 
# the API returns JSON validated against this shape
data = extract(url, schema=Product)

The schema does two jobs: it tells the extractor precisely what to find, and it validates the output so downstream code gets typed, predictable data. See schema-based extraction.

Why AI Extraction Won for This

Converting HTML to JSON at scale across many sites is exactly where selectors struggle: every site needs its own selector set, and every redesign breaks them. Schema-driven AI extraction reuses one schema across layouts and does not break on cosmetic markup changes, which is why most modern scraping APIs offer it as the primary path to JSON.

Key Takeaways

Selector-based conversion maps fixed CSS/XPath paths to keys; brittle but cheap.
AI-based conversion fills a schema by meaning; resilient across layouts and changes.
A schema both directs extraction and validates the resulting JSON.

How ScrapeGraphAI Handles This

ScrapeGraphAI's extract endpoint takes a schema (a Pydantic model or JSON schema) and returns validated JSON read from the page by meaning, so HTML-to-JSON conversion works across sites without per-site selectors.