TL;DR
Crawl delay is the minimum time a crawler waits between requests to the same site, so it does not overload the server. It can be set by a Crawl-delay directive in robots.txt or chosen by the crawler, and it is the main lever for polite crawling that avoids rate-limit blocks.
Where the Value Comes From
Two sources set the delay:
robots.txt. Some sites publish a Crawl-delay directive telling crawlers how many seconds to wait between requests:
User-agent: *
Crawl-delay: 5Not every crawler honors it (Google ignores it in favor of its own signals), but respecting a stated delay is basic courtesy and keeps you off block lists.
Your own choice. When no directive exists, you pick a delay based on the target's size and tolerance. A large site absorbs fast crawling; a small one does not. One request every one to two seconds is a safe default for unknown targets.
Why It Matters
Crawl delay is the difference between a crawl a site tolerates and one it blocks. Fire requests with no delay and you cross per-IP rate limits, collect 429 errors, and can even push a small origin into 520 failures. A sensible delay keeps traffic under the radar and the server healthy, which serves you too: a crashed origin returns no data.
Delay Is Not the Only Lever
Crawl delay controls rate per IP. To crawl faster without hammering one server, combine a polite per-host delay with proxy rotation so requests spread across many IPs, and add jitter so intervals are not perfectly regular. This is the core of polite crawling.
Key Takeaways
- Crawl delay is the wait between requests to one site; it prevents overload.
- Honor a
Crawl-delayin robots.txt; otherwise default to 1-2 seconds. - Combine delay with proxy rotation and jitter to scale politely.
How ScrapeGraphAI Handles This
ScrapeGraphAI paces requests per target and spreads them across a proxy pool automatically, so the crawl endpoint stays polite and unblocked without you tuning a delay value by hand.