Definition
HTTP status codes are standardized three-digit numbers returned by web servers in response to client requests. They indicate whether the request was successful, redirected, resulted in a client error, or caused a server error. Understanding these codes is essential for building reliable web scrapers.
Status Code Categories
HTTP status codes are grouped into five classes:
2xx — Success
- 200 OK — the request succeeded and the response contains the requested data (though not necessarily the data you wanted)
- 204 No Content — the request succeeded but there is no response body
3xx — Redirection
- 301 Moved Permanently — the resource has a new URL; update your references
- 302 Found — temporary redirect; follow it but keep the original URL
- 304 Not Modified — the cached version is still valid
4xx — Client Errors
- 402 Payment Required — an API or metered service ran out of credits or quota
- 403 Forbidden — the server refuses the request, often due to anti-bot detection
- 404 Not Found — the requested page does not exist
- 407 Proxy Authentication Required — the proxy rejected your credentials before the target ever saw the request
- 429 Too Many Requests — rate limit exceeded; back off and retry later
5xx — Server Errors
- 500 Internal Server Error — something went wrong on the server side
- 503 Service Unavailable — the server is temporarily overloaded or under maintenance
- 499 Client Closed Request — nginx's code for a client that gave up before the response arrived
- 520 and 521/522/525 — Cloudflare's codes for an origin that failed behind the edge
Transient failures in this family are exactly what exponential backoff exists for, and slow responses that never complete become request timeouts on your side.
Why Status Codes Matter in Web Scraping
Scrapers must interpret status codes correctly to handle failures gracefully. A 429 response means you should slow down, not retry immediately. A 403 might indicate your request headers or IP have been flagged. A 503 suggests waiting and retrying after a delay.
Ignoring status codes leads to incomplete data, wasted resources on repeated failed requests, and potential IP bans from aggressive retry behavior.
How ScrapeGraphAI Handles Status Codes
ScrapeGraphAI automatically interprets HTTP status codes and applies appropriate strategies — retrying with backoff on transient errors, rotating proxies on 403 responses, and respecting rate limit headers on 429s. This built-in error handling ensures more reliable data collection without requiring manual retry logic.