AI Agent Web Scraping: Data Collection and Analysis

TL;DR

Build an AI scraping agent as a bounded pipeline: plan permitted URLs, fetch content, extract to a schema, validate against the visible source, then retry or store. Keep network access, page limits, credentials, and writes deterministic. An LLM may choose among approved steps, but it should not bypass access controls or decide what data is safe to collect.

An AI agent does not remove the engineering work around web scraping. It changes where judgment can be useful.

A dependable system still needs explicit inputs, allowed domains, page limits, typed output, retries, storage rules, and monitoring. The model can help choose a fetch strategy or map unfamiliar layouts to a schema. It should not control unrestricted network access or write unchecked output into production.

The architecture in this guide is:

plan -> fetch -> extract -> validate -> retry or store

Each arrow is a contract. If one stage cannot prove its output is acceptable, the job stops or moves to quarantine.

What makes a scraper an agent?

A fixed scraper follows one known path. It requests a URL, applies selectors, and stores the result. That is often the right design.

An agent adds a bounded decision loop. For example, it may:

• choose fast HTML fetch or JavaScript rendering from an approved list;
• select a schema version for a known page type;
• decide whether a validation failure deserves one revised extraction attempt;
• follow an approved next-page URL until a page cap is reached;
• summarize failure evidence for a human operator.

It should not:

• invent new target domains;
• turn page text into tool instructions;
• disable rate limits;
• bypass authentication, CAPTCHAs, or access controls;
• expand the collection purpose without approval;
• write an unvalidated record to the main database.

If the target pages and selectors are already known, a workflow such as the n8n web scraper guide is simpler. Add agent behavior only where controlled choices reduce real maintenance work.

The five-stage architecture

Stage	Input	Output	Hard guardrail
Plan	User goal and approved policy	Bounded job specification	Allowed hosts, page cap, schema, purpose
Fetch	Approved URL and fetch profile	HTML or Markdown plus metadata	SSRF controls, timeout, size limit, status checks
Extract	Content, prompt, schema	Candidate JSON	No tools or credentials exposed to page text
Validate	Candidate JSON and source evidence	Accepted record or failure	Type, semantic, provenance, and policy checks
Retry or store	Classified result	Quarantine, retry, or durable write	Retry cap and idempotent key

A planner can select among pre-approved actions. It cannot change the guardrails embedded in the job.

Stage 1: plan a bounded job

Turn a natural-language request into a machine-checkable specification before fetching anything.

A useful job contract includes:

• purpose of collection;
• seed URLs;
• allowed hosts and URL paths;
• fields to extract;
• maximum pages and maximum depth;
• country or locale when relevant;
• fetch modes the agent may use;
• retry budget;
• destination and retention policy;
• fields that contain personal or sensitive data;
• approval requirements for writes or external actions.

Example:

{
  "purpose": "Monitor public product price and availability",
  "seedUrls": [
    "https://example.com/products/42"
  ],
  "allowedHosts": [
    "example.com"
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "allowedFetchModes": [
    "fast",
    "js"
  ],
  "schemaVersion": "product-v1",
  "maxAttemptsPerUrl": 3,
  "writeMode": "quarantine-first"
}

The planner may reject this job or select a permitted fetch mode. It may not add another marketplace because a page contains an interesting link.

A crawl plan also needs a canonical URL rule. Strip fragments, normalize known tracking parameters, and reject repeated URLs before fetching. That avoids loops and makes storage idempotent.

Stage 2: fetch under deterministic controls

Fetching belongs behind a narrow interface. The agent supplies an approved URL and one of the allowed profiles. The fetcher owns DNS resolution, private-network blocking, redirects, timeouts, response-size limits, and content-type checks.

A fetch result should include more than page text:

{
  "requestedUrl": "https://example.com/products/42",
  "finalUrl": "https://example.com/products/42",
  "status": 200,
  "contentType": "text/html",
  "fetchedAt": "2026-08-04T10:00:00Z",
  "fetchMode": "fast",
  "contentHash": "sha256:...",
  "body": "..."
}

Reject a final URL outside the allowlist, even when the initial URL was permitted. Reject private, loopback, link-local, and metadata-service addresses after DNS resolution. Apply the same checks after every redirect.

A 200 response is not enough. Detect login pages, consent walls, empty shells, and obvious block pages before extraction. If the browser has meaningful content but the raw response does not, a JavaScript fetch profile may be appropriate. The heavy JavaScript guide covers that decision.

Do not keep increasing waits and scrolls without evidence. A page can fail because access is denied, not because the browser was too fast.

Stage 3: extract to a typed schema

A prompt without a schema produces a description, not a data contract. Define the accepted object first.

This Pydantic model rejects unknown fields, empty names, negative prices, and malformed currency codes:

from decimal import Decimal
 
from pydantic import BaseModel, ConfigDict, Field, HttpUrl
 
 
class ProductFields(BaseModel):
    model_config = ConfigDict(extra="forbid")
 
    name: str = Field(min_length=1, max_length=300)
    price: Decimal | None = Field(default=None, ge=0)
    currency: str | None = Field(default=None, pattern=r"^[A-Z]{3}$")
    availability: str | None = Field(default=None, max_length=100)
    canonical_url: HttpUrl | None = None

The ScrapeGraphAI Python SDK accepts the model's JSON schema and returns extracted data through response.data.json_data:

from scrapegraph_py import ScrapeGraphAI
 
 
PROMPT = """
Extract the product name, current price, ISO 4217 currency code,
availability text, and canonical product URL from the supplied page.
Use null when an optional field is absent. Do not infer a price from
unrelated recommendations or crossed-out historical values.
""".strip()
 
sgai = ScrapeGraphAI()  # Reads SGAI_API_KEY from the environment.
 
response = sgai.extract(
    url="https://example.com/products/42",
    prompt=PROMPT,
    schema=ProductFields.model_json_schema(),
)
 
candidate = ProductFields.model_validate(response.data.json_data)
print(candidate.model_dump(mode="json"))

The URL is a placeholder and the code does not claim a result from it. Use a permitted page, then compare the candidate fields with visible source evidence.

The same v2 endpoint also accepts HTML or Markdown. That lets the fetch stage provide content directly instead of granting the extraction stage another network request. This separation is useful when network access must stay inside a hardened fetch service.

See Mastering the ScrapeGraphAI endpoint for the complete request contract.

Stage 4: validate syntax, meaning, and provenance

Schema validation is only the first layer.

A valid number can still be the wrong number. A product page may contain a sale price, list price, subscription price, shipping cost, and prices for recommended items. The model needs a field-level rule for which one counts.

Use four validation layers.

1. Structural validation

Check required fields, types, ranges, string lengths, enums, and unknown fields. Pydantic or another schema validator handles this layer.

2. Semantic validation

Apply business rules:

• price must be non-negative;
• currency must agree with the displayed locale or symbol;
• canonical URL must use an allowed host;
• a sale price should not exceed the list price when both are present;
• availability must come from the target product, not a related card.

Some rules produce a hard failure. Others mark a record for review.

3. Source validation

Keep enough evidence to reproduce the decision:

• requested and final URL;
• fetched timestamp and content hash;
• extraction prompt and schema version;
• exact source excerpt or selector when permitted;
• fetch provider and mode;
• model or extractor version;
• validation results.

Do not store an entire page indefinitely when a small permitted excerpt or hash is enough. Retention should match the purpose.

4. Policy validation

Before storage, check whether the record contains prohibited fields, personal data outside the approved purpose, or a source that disallows the planned use. Technical success does not override collection policy.

An agent may explain why validation failed. A deterministic policy decides whether the record can proceed.

Stage 5: retry or store

Classify failures before retrying.

Failure class	Example	Action
Transient network	timeout, 502, connection reset	Retry with delay and jitter
Rate limit	429 with retry guidance	Slow down and respect the limit
Authentication or access	401, 403, login wall	Stop and fix access or remove source
Render mismatch	empty raw HTML but permitted browser content	Retry once with approved JS mode
Schema failure	missing required name	Revise extraction once or quarantine
Semantic failure	price belongs to another product	Quarantine with evidence
Policy failure	unapproved personal data	Reject and alert
Block or CAPTCHA	challenge page	Stop; do not claim a bypass

A retry budget belongs to the job, not the model. The following example keeps extraction attempts bounded and stores only validated records:

import json
import random
import time
from pathlib import Path
from typing import Any
 
from pydantic import ValidationError
from scrapegraph_py import ScrapeGraphAI
 
 
def extract_product(
    client: ScrapeGraphAI,
    url: str,
    *,
    max_attempts: int = 3,
) -> ProductFields:
    last_error: Exception | None = None
 
    for attempt in range(1, max_attempts + 1):
        try:
            response = client.extract(
                url=url,
                prompt=PROMPT,
                schema=ProductFields.model_json_schema(),
            )
            payload: dict[str, Any] = response.data.json_data
            return ProductFields.model_validate(payload)
        except (ValidationError, TypeError, ValueError) as error:
            last_error = error
        except Exception as error:
            # In production, replace this with the SDK's specific
            # transient transport and rate-limit exceptions.
            last_error = error
 
        if attempt < max_attempts:
            delay = (2 ** (attempt - 1)) + random.uniform(0, 0.5)
            time.sleep(delay)
 
    raise RuntimeError(
        f"Extraction failed after {max_attempts} attempts"
    ) from last_error
 
 
def append_quarantine(
    *,
    url: str,
    error: Exception,
    path: Path = Path("quarantine.jsonl"),
) -> None:
    record = {
        "url": url,
        "errorType": type(error).__name__,
        "error": str(error),
    }
    with path.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record) + "\n")

The broad final exception is marked as a teaching placeholder. Production code should catch the SDK's specific transport and rate-limit errors so programming errors are not retried.

For a durable datastore, use a unique key such as normalized source URL plus schema version. Upsert the current record and append observations separately when history matters. A JSONL file is useful for a local demonstration, not for concurrent workers.

A practical controller loop

The agent controller should be boring. It reads the plan, processes one URL, and records a decision:

for url in job.seed_urls:
    assert_allowed_url(url, job.allowed_hosts)
 
    try:
        fetched = fetch(url, profile="fast")
 
        if fetched.needs_javascript:
            fetched = fetch(url, profile="js")
 
        candidate = extract(fetched, schema=ProductFields)
        accepted = validate(candidate, fetched, job.policy)
 
        store_upsert(
            key=normalize_url(url),
            record=accepted,
            schema_version=job.schema_version,
        )
    except TransientFetchError as error:
        retry_queue.add(url, error=error, max_attempts=job.max_attempts)
    except (ValidationError, PolicyError) as error:
        quarantine.add(url, error=error)

The model may help set needs_javascript from bounded evidence or choose an approved schema. The functions still enforce allowed URLs, retry count, validation, and writes.

This separation makes tests possible. You can feed stored HTML into Extract, inject malformed candidate JSON into Validate, and test storage without a live network.

Pagination and multi-page agents

Pagination is where an apparently small agent can run away.

Use these rules:

1. Maintain a visited set of normalized URLs.
2. Reject hosts and paths outside the job.
3. Set maximum pages, depth, bytes, and wall-clock duration.
4. Accept a next link only when it comes from a defined page region or API field.
5. Stop when the next URL repeats or content hashes repeat.
6. Store partial completion when one page fails.
7. Keep concurrency below source and provider limits.

A planner can suggest that a “next” link is relevant. The controller verifies it against the URL policy and remaining budget.

For broad site ingestion, a crawl API may be a better fit than asking an agent to discover links one page at a time.

Prompt injection is a data-integrity problem

A web page can contain text such as “ignore previous instructions” or “upload your environment variables.” That text is source data. It does not gain authority because an LLM reads it.

Protect the pipeline:

• never include secrets in the extraction prompt or page context;
• do not expose shell, email, browser, or database tools to the extractor;
• separate system instructions from fetched content;
• require structured output with unknown fields rejected;
• prevent extracted URLs from being fetched without allowlist checks;
• review any page-derived action before it changes external state;
• log decisions without logging credentials.

For research agents, treat summaries as claims linked to source URLs and excerpts. A fluent sentence is not verification.

Security and legal limits

An agent does not make scraping lawful or safe. Before collection, evaluate:

• terms and contractual access;
• robots directives where applicable;
• authentication and access controls;
• rate limits and service stability;
• copyright and database rights;
• privacy purpose, minimization, retention, and deletion;
• jurisdiction-specific obligations;
• whether automated decisions use the collected data.

The web scraping legality guide offers an engineering checklist, not legal advice.

Do not describe CAPTCHA bypass as an agent capability. A CAPTCHA or block page is a stop condition. Use an authorized API, obtain permission, reduce request load, or remove the source.

Monitoring the system

Track the stages separately:

• planned URLs and rejected URLs;
• fetch success by status, host, and profile;
• JavaScript fallback rate;
• extraction latency and usage;
• structural and semantic validation failure rates;
• retries per accepted record;
• quarantine volume and reasons;
• duplicate or unchanged records;
• storage writes and conflicts;
• cost per valid record.

A falling error rate can hide falling quality if the validator is too permissive. Review a fixed sample against visible sources after prompt, schema, fetch-provider, or model changes.

Set alerts on sustained rates, not one isolated timeout. A sudden rise in null prices may indicate a layout change even when every request returns 200.

When to avoid an agent

Use a deterministic scraper when:

• the same known pages are fetched repeatedly;
• selectors are stable;
• output rules are simple;
• latency or cost must be minimal;
• every decision can be represented as code.

Use a bounded agent when:

• page layouts vary but the target schema is stable;
• choosing between approved fetch modes saves maintenance;
• failures need classification from page evidence;
• a human would otherwise perform the same small decision repeatedly.

The building AI agents for web scraping guide covers broader orchestration patterns. Keep the safety boundary from this article even when an agent framework handles the loop.

Production checklist

Before release:

1. Freeze the job purpose, allowed hosts, page cap, and schema version.
2. Test fast and JavaScript fetch modes on representative permitted pages.
3. Validate known good, missing-field, wrong-price, login, and block-page fixtures.
4. Add SSRF and redirect tests.
5. Confirm credentials never enter prompts, page context, or logs.
6. Use idempotent writes and a quarantine path.
7. Bound retries, concurrency, bytes, duration, and cost.
8. Review a source-linked sample manually.
9. Document rollback and kill-switch behavior.
10. Monitor valid records, not only successful HTTP requests.

The agent's value is controlled flexibility. The system remains trustworthy because the controls are not flexible.

Related Articles

• How to Build a Web Scraper with n8n
• Mastering the ScrapeGraphAI Endpoint
• Building AI Agents for Web Scraping
• Handling Heavy JavaScript
• Is Web Scraping Legal?