TL;DR
A 429 Too Many Requests error means the server is rate limiting you: your client sent more requests in a time window than the site allows. In web scraping it is the most common block, and the fix is to slow down, honor the Retry-After header, and retry with exponential backoff instead of hammering the endpoint.
Why 429 Exists
Every site defends its infrastructure with rate limits, typically enforced per IP address, per session, or per API key. Cross the threshold and the server answers 429 instead of content. Unlike a 403, a 429 is not a judgment that you are a bot. It is a speed limit, and it usually clears on its own once you back off.
The response often carries the recovery time explicitly. Retry-After: 60 means wait sixty seconds. API-style targets may add X-RateLimit-Remaining and X-RateLimit-Reset headers that let you pace requests before ever hitting the wall.
How to Handle 429: Backoff, Not Retries
Immediate retries make 429s worse, and on stricter sites they escalate a temporary limit into an IP ban. The standard pattern is exponential backoff with jitter, honoring Retry-After when present:
import random
import time
import requests
def get_with_backoff(url: str, max_retries: int = 5) -> requests.Response:
for attempt in range(max_retries):
response = requests.get(url, timeout=(5, 30))
if response.status_code != 429:
return response
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(2 ** attempt, 60)
time.sleep(delay + random.uniform(0, 1))
raise RuntimeError(f"still rate limited after {max_retries} retries")Preventing 429s in the First Place
- Pace requests to the target's tolerance; one request per second is a safe starting point for most sites.
- Spread load across IPs with proxy rotation, since most limits are per IP.
- Randomize timing. Perfectly regular intervals are easy to fingerprint; jitter makes traffic look organic.
- Scrape off-peak for the target's timezone, when thresholds are less likely to be tightened by load.
Key Takeaways
- 429 is a speed limit, not a bot verdict; patience fixes it.
- Honor Retry-After when present, otherwise back off exponentially with jitter.
- Sustained 429s at low rates mean the limit is per IP: rotate or slow down further.
How ScrapeGraphAI Handles This
ScrapeGraphAI paces requests, reads rate-limit headers, and retries with backoff across a rotating proxy pool automatically. Send your full URL batch to the scrape endpoint or use crawl with concurrency controls, and the platform absorbs the rate limiting for you.