How Do You Extract Data From Nested HTML Structures?

TL;DR

You extract data from nested HTML by walking the structure relationship by relationship: select each repeating container, then read the fields inside it relative to that container. Scoped, relative selectors keep records aligned, and AI extraction handles deeply nested or inconsistent structures by reading them against a schema instead.

The Alignment Problem

Nested data is where naive scraping breaks. Say a page has ten product cards, each with a name, price, and rating. If you select all names, all prices, and all ratings into three flat lists and zip them, one missing price shifts every later record by one, silently corrupting the data. The fix is to iterate containers, not fields.

Scoped, Relative Selection

Select the repeating container first, then read each field relative to that container so every record stays self-contained:

from bs4 import BeautifulSoup
 
soup = BeautifulSoup(html, "lxml")
products = []
for card in soup.select("div.product-card"):        # one container per record
    products.append({
        "name": card.select_one("h2").get_text(strip=True),
        # relative to THIS card, so a missing price cannot shift other records
        "price": (p := card.select_one(".price")) and p.get_text(strip=True),
        "rating": (r := card.select_one(".rating")) and r.get("data-value"),
    })

The pattern generalizes to deeper nesting: for a list of articles each with a list of tags, loop articles, then loop tags within each article. XPath helps when you must navigate upward to a parent to find the container.

Tables and Lists

Tables are a common nested case: iterate rows, then cells within each row, mapping cells to columns by header. Nested lists (a definition list, or list items containing sub-lists) follow the same container-then-children logic.

When Structure Is Too Messy

Real pages nest inconsistently: optional wrappers, varying depth, fields that move between layouts. Selector code for this becomes a tangle of special cases. AI extraction sidesteps it by reading the page against a schema and returning aligned records by meaning, which holds up far better on deeply nested or irregular structures.

Key Takeaways

Iterate the repeating container, then read fields relative to it, never flat-zip lists.
The same container-then-children pattern scales to deep nesting, tables, and lists.
For inconsistent nesting, AI extraction against a schema beats special-case selectors.

How ScrapeGraphAI Handles This

ScrapeGraphAI's extract endpoint returns aligned nested records from a schema (lists of objects, objects containing lists) by reading the page's meaning, so you get correctly grouped data without writing container-walking selector code.