What Is Exponential Backoff in Web Scraping?

TL;DR

Exponential backoff is a retry strategy where the wait time doubles after each failed attempt, for example 1, 2, 4, then 8 seconds, usually with random jitter added. In web scraping it is the standard response to 429 and transient 5xx errors because it gives struggling servers room to recover instead of hammering them.

Why Doubling Beats Fixed Delays

A fixed retry delay keeps pressure on the server constant: if it was overloaded at attempt one, it is probably still overloaded three seconds later. Doubling the delay makes pressure decay geometrically, so short outages resolve within a few attempts and longer ones stop consuming your request budget. It also protects you: aggressive retry loops are one of the fastest ways to convert a temporary 429 into a permanent IP ban.

The Reference Implementation

Three details separate production-grade backoff from a naive loop: a delay cap, random jitter, and respect for the server's own Retry-After header when it sends one.

import random
import time
import requests
 
RETRYABLE = {429, 500, 502, 503, 504, 520, 522}
 
def fetch(url: str, max_retries: int = 5) -> requests.Response:
    for attempt in range(max_retries):
        response = requests.get(url, timeout=(5, 30))
        if response.status_code not in RETRYABLE:
            return response
        retry_after = response.headers.get("Retry-After")
        if retry_after:
            delay = float(retry_after)
        else:
            delay = min(2 ** attempt, 60)          # 1, 2, 4, 8... capped at 60s
        time.sleep(delay + random.uniform(0, delay / 2))  # jitter
    raise RuntimeError(f"gave up on {url} after {max_retries} attempts")

Jitter matters more than it looks. A fleet of workers that all fail together and all retry after exactly 4 seconds produces synchronized traffic spikes (the thundering herd) that re-trigger the failure. Randomizing each worker's delay spreads the retries out.

What Not to Retry

Backoff only helps when time fixes the problem. Retrying a 403 with the same identity, a 404, or a 402 wastes attempts on errors that need a different identity, a different URL, or a bigger budget. Classify first, then retry only the transient class.

Key Takeaways

• Double the delay per attempt, cap it, and always add jitter.
• Honor Retry-After when the server provides it.
• Retry only transient errors (429, most 5xx); everything else needs a different fix.

How ScrapeGraphAI Handles This

ScrapeGraphAI applies capped, jittered backoff with per-error-class retry policies on every request, across multiple fetch providers. Calling the scrape endpoint gets you production retry behavior without writing a retry loop at all.