How to Build a Web Scraper with n8n: Complete Guide for 2026

TL;DR

Use n8n's Schedule Trigger, HTTP Request, HTML, and Code nodes when a page returns useful HTML. Use a schema-based ScrapeGraphAI request when JavaScript rendering or layout changes make selectors expensive to maintain. In both cases, cap pagination, validate every record, retry bounded failures, and upsert by a stable key.

An n8n web scraper is a workflow, not one magic node. The useful part is the control plane around the fetch: credentials, pagination, validation, retries, scheduling, and storage.

This guide builds two versions:

1. a native HTTP and HTML workflow for predictable server-rendered pages;
2. a ScrapeGraphAI workflow for JavaScript-heavy pages or structured extraction from changing layouts.

The examples use placeholder URLs because you should test against a site you are allowed to access. The node settings and ScrapeGraphAI body match the current n8n documentation and the v2 API contract checked on August 4, 2026.

Choose the simpler workflow first

Use native n8n nodes when the response already contains the fields you need. A CSS selector is faster and cheaper than an AI extraction call when the markup is stable.

Use schema-based extraction when the target depends on client-side rendering, the same fields appear across different layouts, or the output must conform to a typed contract. The AI agent web scraping architecture explains where that extraction step belongs in a larger system.

Requirement	Native HTTP and HTML	ScrapeGraphAI request
Server-rendered HTML	Good fit	Works, but may be unnecessary
Stable CSS selectors	Good fit	Optional
JavaScript rendering	Usually needs another fetch layer	Set fetch mode to JS
Typed JSON	Requires mapping and validation	Prompt plus JSON schema
Layout variation	Selector maintenance	Semantic extraction
Lowest request cost	Usually better	Depends on extraction work
Auditability	Exact selectors and raw response	Prompt, schema, raw input, and response metadata

Start with native nodes. Move only the difficult fetch or extraction step to a managed API.

Workflow A: scrape predictable HTML with native n8n nodes

The core path is:

Schedule Trigger -> Build page list -> Loop Over Items -> HTTP Request -> HTML -> Code validation -> Storage

The names below follow current n8n terminology. The HTTP Request node fetches the page, while the HTML node extracts values with CSS selectors.

1. Configure the Schedule Trigger

Add a Schedule Trigger and choose an interval that matches how often the source changes. A daily inventory job does not need to run every five minutes.

Before activating it:

• set the workflow timezone explicitly;
• estimate the maximum execution duration;
• prevent overlapping runs if one crawl can outlast the schedule;
• keep a manual trigger while developing;
• record the run timestamp with the stored rows.

A schedule starts the workflow. It does not make the job idempotent. Storage logic must still prevent duplicates.

2. Build a bounded page list

Pagination should have a known stop condition. If the target accepts a query parameter such as ?page=2, create one item per allowed page in a Code node:

const firstPage = 1;
const lastPage = 5;
 
return Array.from(
  { length: lastPage - firstPage + 1 },
  (_, index) => ({
    json: {
      page: firstPage + index,
      url: "https://example.com/catalogue?page=" + (firstPage + index),
    },
  }),
);

Send those items through Loop Over Items before the HTTP request. Five is an intentional safety cap, not a claim that the source has five pages.

For API-style endpoints, the HTTP Request node also supports pagination modes based on a next URL or an updated query/body parameter. The current n8n pagination documentation uses expressions such as {{ $pageCount + 1 }}. Prefer that mode only when the response exposes an unambiguous next-page rule. For ordinary HTML, an explicit item list is easier to inspect and stop.

Never use “continue until empty” without a maximum page count. A markup change can turn a small job into an unbounded crawl.

3. Fetch the HTML

Configure HTTP Request with these baseline settings:

Setting	Value
Method	GET
URL	{{ $json.url }}
Response format	Text
Timeout	A bounded value appropriate for the source
Redirects	Follow only when expected
Retry On Fail	Enabled for transient failures
On Error	Stop Workflow, or Continue using error output when you have an error branch

Do not paste cookies or tokens into the URL or node body. Store them in an n8n credential. Add a custom User-Agent only when the site permits the automation and the value truthfully identifies your client. Faking browser headers does not grant permission or make a blocked source reliable.

Log the final URL and HTTP status. A 200 response can still be a login page, consent screen, or soft block, so status alone is not validation.

4. Extract fields with the HTML node

Set the HTML node to read the HTTP response field that contains the page text. Add one extraction value for every required field.

Suppose a permitted catalogue uses product card markup like this:

<article class="product-card">
  <a class="product-link" href="https://example.com/products/42">Mechanical Keyboard</a>
  <span class="price" data-currency="USD">129.00</span>
  <span class="stock">In stock</span>
</article>

use selectors like these:

Output key	CSS selector	Return value
name	.product-card .product-link	Text
url	.product-card .product-link	Attribute: href
price	.product-card .price	Text
currency	.product-card .price	Attribute: data-currency
availability	.product-card .stock	Text

Inspect the actual permitted page before copying selectors. A selector that returns nothing is not proof that the product is unavailable. It may mean the response was rendered differently, the page is blocked, or the markup changed.

5. Normalize and validate every row

The Code node should turn extracted strings into a stable record and reject incomplete output before it reaches storage:

return items.map(({ json }) => {
  const name = String(json.name ?? "").trim();
  const sourceUrl = new URL(String(json.url), "https://example.com").toString();
  const price = Number(String(json.price ?? "").replace(/[^0-9.]/g, ""));
  const currency = String(json.currency ?? "").trim().toUpperCase();
  const availability = String(json.availability ?? "").trim();
 
  if (!name) throw new Error("Missing product name");
  if (!Number.isFinite(price) || price < 0) throw new Error("Invalid price");
  if (!/^[A-Z]{3}$/.test(currency)) throw new Error("Invalid currency");
 
  return {
    json: {
      sourceUrl,
      name,
      price,
      currency,
      availability,
      scrapedAt: new Date().toISOString(),
    },
  };
});

This check catches empty selectors, localized price formats that your parser does not understand, and partial block pages. If commas and periods have locale-specific meanings, parse them with a locale-aware rule instead of deleting characters blindly.

In production, store invalid rows in a quarantine table with the source URL, run ID, and reason. Do not mix them into the main dataset.

6. Store with an idempotent key

Use a stable source identifier such as the normalized product URL. An upsert should update the latest values without creating a new product on every run.

A useful storage record contains:

• normalized source URL;
• extracted business fields;
• source HTTP status;
• scrape timestamp and workflow execution ID;
• validation status;
• content hash when change detection matters.

Google Sheets is acceptable for a small manual workflow. A database is safer for concurrency, unique constraints, and historical observations. If you need price history, keep the current product table and a separate observation table instead of overwriting the only prior value.

Workflow B: use ScrapeGraphAI for JavaScript and schemas

A page can return almost empty HTML to the native HTTP node while the browser shows a complete product card. That is a fetch problem, not an n8n parsing problem.

Add another HTTP Request node for managed rendering and extraction:

• method: POST;
• URL: https://v2-api.scrapegraphai.com/api/extract;
• authentication: Generic Credential Type, Header Auth;
• header name: SGAI-APIKEY;
• body content type: JSON;
• response format: JSON.

Store the key in the credential. Do not put it in the workflow JSON, screenshots, or execution logs.

A contract-valid extraction body

The following body matches the repository's current v2 extract request schema. The URL, prompt, schema, and fetch settings are explicit:

{
  "url": "{{ $json.url }}",
  "prompt": "Extract the product name, current price, currency, availability, and canonical product URL. Return null for a missing optional field.",
  "schema": {
    "type": "object",
    "properties": {
      "name": { "type": "string", "minLength": 1 },
      "price": { "type": ["number", "null"], "minimum": 0 },
      "currency": {
        "type": ["string", "null"],
        "pattern": "^[A-Z]{3}$"
      },
      "availability": { "type": ["string", "null"] },
      "canonicalUrl": {
        "type": ["string", "null"],
        "format": "uri"
      }
    },
    "required": ["name", "price", "currency", "availability", "canonicalUrl"],
    "additionalProperties": false
  },
  "fetchConfig": {
    "mode": "js",
    "timeout": 60000,
    "wait": 2000,
    "scrolls": 2
  }
}

The v2 response exposes extracted data under the top-level json field, with raw, usage, and metadata alongside it. In the next n8n node, validate {{ $json.json }}. Do not build downstream expressions around an older SDK response shape.

Use mode: auto first when JavaScript is not required. JavaScript mode, extra waiting, and scrolling add work. Raise them only after inspecting a real failure.

A Python implementation of the same endpoint appears in Mastering the ScrapeGraphAI endpoint.

Validate the API response again

A JSON schema guides extraction, but your workflow still owns the final contract. Check:

• the response has a non-null json object;
• required values have the correct types;
• price and currency agree with the visible source;
• the canonical URL belongs to an allowed host;
• the response is not stale compared with the run timestamp.

Keep the prompt and schema under version control. When the contract changes, increment a schema version in the stored row.

Retries and failure handling

Retries should target transient failures, not every bad result.

Failure	Retry?	Response
Timeout or 502/503	Yes, bounded	Retry with delay and jitter
429	Yes, respect rate limits	Reduce concurrency and delay
401/403	Usually no	Fix credentials or access policy
Missing selector	Not immediately	Inspect HTML and extraction rule
Schema validation failure	Once at most	Quarantine with raw evidence
Login or consent page	No blind retry	Add permitted session handling or stop
CAPTCHA or block page	No bypass claim	Stop, respect access controls, choose another source

In Settings, n8n nodes provide Retry On Fail plus On Error options for stopping, continuing, or sending the error through a separate output. For workflow-level alerts, assign an error workflow that starts with Error Trigger. Include the failed URL, execution ID, node, status, and final error. Exclude credentials and raw personal data.

A practical retry policy is two or three attempts with increasing delay. Ten immediate retries usually increase load without fixing the cause.

Scheduling, pagination, and concurrency checklist

Before activating the workflow:

1. Set a maximum page count and maximum items per run.
2. Keep concurrency below the source and API rate limits.
3. Add a delay only when the source policy or rate limit requires it.
4. Prevent overlapping schedules or make every write idempotent.
5. Stop pagination when the next URL repeats.
6. Record partial-run status when one page fails.
7. Alert on sustained failure rate, not one isolated timeout.
8. Recheck a sample against the visible source after selector or prompt changes.

For JavaScript-heavy sources, read Handling heavy JavaScript before increasing waits and scroll counts. More browser work is not automatically more accurate.

Legal and security limits

Only collect data you are permitted to access. Review terms, robots directives where applicable, rate limits, copyright, database rights, and privacy obligations. The web scraping legality guide provides a practical checklist, but it is not legal advice.

Treat fetched pages as untrusted input:

• allowlist target schemes and hosts to reduce SSRF risk;
• block private and link-local network ranges;
• never let page text change credentials, tools, or storage policy;
• remove secrets from execution logs;
• minimize personal data;
• set retention and deletion rules;
• review prompt-injection risk before page content reaches another agent.

n8n coordinates the job. It does not make access lawful, defeat a CAPTCHA, or guarantee that a page's claims are true.

Which version should you ship?

Ship the native version when selectors remain stable across your sample and the response already contains the data. It is easier to debug because you can point to the exact HTTP response and selector.

Ship the ScrapeGraphAI version when rendering or layout variation is the expensive part. Keep deterministic pagination, validation, storage, and monitoring in n8n. The API should replace the fragile extraction step, not the workflow controls around it.

Larger autonomous pipelines can reuse the controls in the AI agent web scraping guide.

Related Articles

• AI Agent Web Scraping: Data Collection and Analysis
• Mastering the ScrapeGraphAI Endpoint
• Handling Heavy JavaScript
• Is Web Scraping Legal?