TL;DR
A 200 status code means the server accepted the request and returned a response body. In web scraping it confirms delivery, not data quality: many sites return 200 with a CAPTCHA page, an empty JavaScript shell, or a soft block instead of the content you asked for, so scrapers must validate the body too.
What a 200 Response Actually Guarantees
The HTTP spec defines 200 OK as "the request has succeeded". That guarantee stops at the transport layer. The server processed your request and sent something back. It says nothing about whether that something is the product page, the article, or the price table you wanted.
For browsers this distinction rarely matters, because a human immediately sees what loaded. A scraper only sees bytes, which is why 200 is the most misleading code in scraping logs.
The Three 200 Responses That Contain No Data
Soft blocks. Anti-bot systems such as Cloudflare, DataDome, and Akamai often serve their challenge or "access denied" page with a 200 instead of a 403. Your scraper records a success while collecting the same block page a thousand times.
JavaScript shells. Single-page applications return a valid HTML skeleton with 200, then load the real content through API calls in the browser. Without JavaScript rendering, the body contains a root div and a bundle reference, nothing else.
Soft 404s. Some sites answer removed URLs with a 200 and a "page not found" template, which also poisons crawls that trust the status code alone.
How to Validate a 200 Response
Status-code checks alone miss all three cases, so validate the body as well. A cheap validation layer catches most silent failures:
import requests
response = requests.get(url, timeout=(5, 30))
body = response.text
is_valid = (
response.status_code == 200
and len(body) > 2000
and "captcha" not in body.lower()
and "cf-challenge" not in body
and "product-title" in body # a selector you expect on real pages
)Useful signals: minimum body length, presence of an element that only exists on real pages, absence of challenge markers, and a stable text-to-HTML ratio. If any check fails, treat the request as blocked and retry through a different proxy or with rendering enabled.
Key Takeaways
- 200 confirms the server responded, not that you received the target content.
- Soft blocks, JS shells, and soft 404s all arrive as 200 responses.
- Always pair status-code checks with body validation before storing data.
How ScrapeGraphAI Handles This
ScrapeGraphAI validates content, not status codes. Every response goes through content checks that detect challenge pages, empty shells, and truncated documents, and failed fetches are retried through alternative providers automatically. The scrape endpoint returns clean markdown or HTML only when the page actually contains content.