TL;DR
Polite crawling is crawling a site in a way that does not harm it: respecting robots.txt, pacing requests with a crawl delay, limiting concurrency, and identifying your crawler honestly. The goal is to collect data without degrading the site's performance or triggering blocks.
The Rules of Polite Crawling
Respect robots.txt. Read the site's robots.txt and stay out of disallowed paths. It is the site's stated preference, and ignoring it is both rude and a fast way to get blocked.
Pace your requests. Apply a crawl delay between requests to the same host so you do not spike its load. Add jitter so the pattern is not robotically regular.
Limit concurrency. Do not open dozens of parallel connections to one origin. A small origin can be pushed into timeouts or 520 errors by aggressive concurrency, and that hurts your crawl as much as the site.
Identify honestly where appropriate. Many well-behaved crawlers use a descriptive user agent with a contact URL, so site owners can reach out instead of blocking blindly.
Crawl off-peak. Where you know the site's timezone, crawling during low-traffic hours reduces the impact of your requests.
Why Politeness Is Self-Interested
Politeness is also self-interest. An overloaded server returns errors, serves slower, or blocks your IP, all of which reduce the data you collect. A crawl that respects a site's limits runs longer, gets blocked less, and produces cleaner results. The fastest crawl is rarely the most aggressive one.
Scaling Without Being Rude
To crawl a lot of pages quickly while staying polite, spread load across many sites rather than hammering one, and use proxy rotation so per-host request rates stay low even as your total throughput rises. Politeness is per-origin; total speed comes from parallelism across origins.
Key Takeaways
- Polite crawling respects robots.txt, paces requests, and limits concurrency per host.
- It is self-interested: overloaded sites block you and return worse data.
- Scale by spreading load across hosts and IPs, not by hammering one origin.
How ScrapeGraphAI Handles This
ScrapeGraphAI's crawl endpoint enforces per-host pacing and concurrency limits and spreads requests across managed proxies, so crawls stay polite and unblocked by default.