How Do You Get Structured JSON Data From URLs?

TL;DR

You get structured JSON from URLs by defining the fields you want as a schema, then fetching each URL and mapping its content onto that schema. Selector-based code assigns fixed CSS or XPath paths to keys; schema-based AI extraction fills the JSON by meaning, returning typed, validated records that hold up across different pages.

Start From the Schema

The reliable way to structured JSON begins with deciding the shape, not the scraping. Declare the fields and types you want, and everything downstream targets that shape:

from pydantic import BaseModel
 
class Product(BaseModel):
    name: str
    price: float
    in_stock: bool
    rating: float | None = None

A schema does two things at once: it tells the extractor exactly what to find, and it validates the result so your pipeline receives typed, predictable data instead of guesses.

Two Ways to Fill It

Selector mapping. Fetch each URL, then assign values from fixed CSS or XPath paths into the schema fields. Cheap and deterministic, but you write a selector set per site and it breaks when markup changes.

Schema-based AI extraction. Fetch each URL and pass the page plus the schema to a model that fills the fields by meaning. One schema works across different layouts, and it survives redesigns, as covered in HTML to JSON conversion.

Doing It Across Many URLs

For a list of URLs, the loop is the same regardless of method: fetch, map to schema, collect. Keep a few things in mind at scale: handle failures per URL so one bad page does not stop the batch, deduplicate URLs first, and validate each record against the schema so malformed results are caught rather than stored.

results = []
for url in urls:
    try:
        results.append(extract(url, schema=Product))   # validated per URL
    except ValidationError:
        log_bad_page(url)

Key Takeaways

Define a schema first; it directs extraction and validates the output.
Selector mapping is cheap but per-site and brittle; AI extraction generalizes.
At scale, isolate per-URL failures, deduplicate, and validate every record.

How ScrapeGraphAI Handles This

ScrapeGraphAI's extract endpoint takes a URL and a schema and returns validated JSON, and search can extract the same schema across many result pages, so you go from URLs to structured records without per-site selector code.