TL;DR
Request timeouts in web scraping happen when the target server, a proxy in the chain, or a slow-rendering page takes longer than the client is willing to wait. The usual causes are overloaded origins, JavaScript-heavy pages, slow residential proxies, and connect or read timeout values set too low for the target.
Connect Timeout vs Read Timeout
A single "timeout" setting hides two different clocks. The connect timeout bounds how long you wait for the TCP handshake to complete; failures here mean network or proxy trouble. The read timeout bounds the wait for response data after connecting; failures here mean the server accepted you and then stalled. Python's requests accepts them as a tuple, and setting them separately is the single highest-value timeout practice:
# 5s to connect, 30s to receive the response
response = requests.get(url, timeout=(5, 30))Leaving timeout unset in requests means waiting forever. Every production scraper sets one, or a hung connection eventually freezes the whole worker pool.
The Usual Suspects
Slow origins. Small sites under load answer in seconds instead of milliseconds. If p95 latency is 12 seconds, a 10-second read timeout guarantees a steady failure rate.
Residential proxy latency. Traffic through residential proxies rides real consumer connections, adding seconds per request compared to datacenter routes. Timeouts tuned on direct connections break the moment you switch.
JavaScript rendering. With a headless browser, the clock covers scripts, XHR calls, and rendering. Page-load timeouts of 30 to 60 seconds are normal where raw HTTP needs 5.
Tarpitting. Some anti-bot systems deliberately answer suspected bots at a crawl. If only your scraper times out while a browser is fast, you are being slow-walked, and the fix is identity, not patience.
Setting Timeouts That Work
Measure the target's real latency distribution, set the read timeout around p95 plus margin, and keep the connect timeout short (3 to 5 seconds) so dead routes fail fast. Retry timeouts once or twice with exponential backoff, because transient slowness is common; but treat systematic timeouts as a signal to investigate, not a number to raise forever.
Key Takeaways
- Split connect and read timeouts; they diagnose different failures.
- Never scrape with no timeout: one hung connection can stall a whole worker.
- Tune against measured latency per target, and re-tune when adding proxies or rendering.
How ScrapeGraphAI Handles This
ScrapeGraphAI sizes timeouts per fetch provider, from fast direct requests to full browser rendering, and retries transient stalls automatically. The scrape endpoint accepts JavaScript-heavy targets without you tuning a single timeout value.