TL;DR
A 404 Not Found error means the server has no resource at the requested URL. In web scraping, 404s come from dead links in your URL list, changed site structures, or geo and bot rules that hide existing pages, and crawlers must record them instead of retrying them.
Where Scraping 404s Come From
Stale URL lists. Product catalogs churn constantly. A URL list scraped last month can easily lose several percent of its entries to delisted products, expired job posts, or removed listings. These 404s are data, not errors: the disappearance of a page is often the signal you are being paid to detect.
Site restructures. Sites migrate URL schemes and do not always add redirects. If a whole section of your list starts returning 404 at once, look for the new URL pattern instead of assuming the content is gone.
Selective 404s. Some sites answer suspected bots with 404 instead of 403 to leak less information. If a URL opens in your browser but 404s in your scraper, treat it as a block, not a missing page.
Soft 404s: the Inverted Problem
The opposite failure also exists: the server returns 200 with a "page not found" template. Status-code checks miss these entirely, so pair them with body checks:
response = requests.get(url, timeout=(5, 30))
is_gone = response.status_code == 404 or (
response.status_code == 200
and any(marker in response.text.lower()
for marker in ("page not found", "no longer available"))
)Handling 404s in a Pipeline
Three rules keep 404s cheap:
- Do not retry. A genuine 404 is stable. Retrying wastes requests and looks like bot behavior. Retry only if you suspect a selective 404, and then with a different identity.
- Record, don't discard. Store the URL and timestamp. Price monitors, job boards, and real estate trackers all treat page disappearance as an event worth reporting.
- Watch the rate. A stable background rate of 404s is normal. A sudden spike means a restructure, an expired sitemap, or a block pattern, and it should page a human.
Key Takeaways
- 404 means gone, not blocked; but sites sometimes lie in both directions.
- Never retry plain 404s; log them as signals about the target site.
- Catch soft 404s with body validation, not status codes alone.
How ScrapeGraphAI Handles This
ScrapeGraphAI returns explicit, typed errors for missing pages so pipelines can distinguish gone from blocked programmatically. For change detection, the monitor service watches URLs on a schedule and reports when a page disappears or its content changes, turning 404s into structured events.