TL;DR
You extract only the main content by removing the boilerplate (navigation, headers, footers, sidebars, ads) and keeping the primary article or product body. Readability-style algorithms score text density and link ratios to find the main block, and AI extraction targets the meaningful content directly against a schema.
Why Boilerplate Is the Enemy
A web page is mostly not its content. Around the article sits navigation, a header, a footer, related-links sidebars, cookie banners, newsletter prompts, and ads. For a reader the eye ignores this; for a scraper it is noise that pollutes output, inflates token counts for LLM input, and drags irrelevant text into search indexes. Isolating the main content is a core step for clean data and for RAG.
The Established Approaches
Readability algorithms. Tools in the lineage of Mozilla's Readability score each block of the DOM by text length, link density, tag type, and position. High-text, low-link blocks are likely content; nav and footers are link-dense and score low. This is fast and works well on article-shaped pages.
from readability import Document
doc = Document(html)
main_html = doc.summary() # the main content block, boilerplate removed
title = doc.title()Heuristic stripping. Remove known boilerplate tags and roles (nav, header, footer, aside, elements with ad-related classes) before parsing. Cruder but effective as a first pass.
AI extraction. Ask a model for the article body or the specific fields you want; it separates content from chrome by meaning, which handles unusual layouts that trip up density heuristics.
Choosing a Method
Readability-style extraction is the reliable default for articles and blog posts. For structured targets (a product's fields, not its prose), skip main-content detection and extract the fields directly against a schema. For pages where content and boilerplate are visually tangled, AI extraction handles it most reliably.
Key Takeaways
- Main-content extraction removes nav, headers, footers, sidebars, and ads.
- Readability algorithms score text density and link ratios to find the content block.
- Use readability for articles, schema extraction for fields, AI for tangled layouts.
How ScrapeGraphAI Handles This
ScrapeGraphAI returns main content by default: scrape strips boilerplate and gives clean markdown of the primary content, and extract pulls the specific fields you define, so you never post-process nav and footers out of your data.