Definition
Schema generation is the process of using AI to automatically analyze a web page and produce a structured data schema describing the extractable information. Instead of manually defining what fields exist on a page and what types they hold, the AI examines the content and proposes a schema that captures the available data points and their relationships.
How Schema Generation Works
- Page analysis — the AI processes the page content, identifying distinct pieces of information: product names, prices, dates, descriptions, ratings, and other data points
- Type inference — each identified field is assigned an appropriate data type: string, number, boolean, array, or nested object
- Structure proposal — the AI organizes fields into a logical schema, grouping related data and identifying repeating patterns (like lists of products or reviews)
- Schema output — the result is a formal schema definition (JSON Schema, Pydantic model, or similar format) ready for use in extraction
Use Cases
Exploratory Scraping
When you encounter a new data source and need to understand what is available before deciding what to extract. Schema generation surveys the page and tells you what data exists.
Rapid Prototyping
Instead of manually inspecting HTML and drafting schemas, generate a starting schema automatically and refine it for your specific needs.
Multi-Site Normalization
When scraping similar content from multiple sites (e.g., product data from different e-commerce platforms), generated schemas help identify common fields and site-specific variations.
Benefits
- Speed — generates schemas in seconds that might take minutes or hours to write manually
- Completeness — may identify extractable data points you would have overlooked
- Correctness — type inference reduces errors from mismatched field types
Schema Generation in ScrapeGraphAI
ScrapeGraphAI offers automatic schema generation as a core feature. Submit a URL and the platform analyzes the page to produce a comprehensive extraction schema. You can use this schema as-is or refine it to focus on the specific fields your application needs, significantly accelerating the setup of new extraction pipelines.