TL;DR
Crawl scope is the set of rules that decide which discovered links a crawler is allowed to follow: which domains, paths, and URL patterns are in bounds. It keeps a crawl focused on the pages you want instead of wandering across the whole domain or off-site, and it is the main defense against a crawl that never ends.
Why Scope Is Non-Negotiable
Without scope rules, a crawler follows every link it finds, which quickly leads off the target site (an outbound link to a social network, then that network's entire graph) or into infinite low-value spaces like faceted-search URLs and calendar pages. Scope bounds the crawl so it collects the section you care about and stops, instead of expanding forever.
The Common Scope Rules
- Domain scope. Stay on the seed host, or an allowlist of hosts. The most basic and important rule:
same host only. - Path scope. Restrict to a subtree, for example only URLs under
/blog/or/products/. - Pattern include/exclude. Follow URLs matching a pattern and skip others, such as excluding
?sort=and?page=faceted variants that multiply pages without adding content. - Depth limit. Cap how far from the seed the crawl travels, via crawl depth.
def in_scope(url, host):
p = urlparse(url)
return (
p.netloc == host # domain scope
and p.path.startswith("/products/") # path scope
and "sort=" not in p.query # pattern exclude
)Scope vs Depth vs Budget
These three limits work together but bound different things. Scope decides which links are eligible at all. Crawl depth caps distance from the seed. Crawl budget caps total pages or resources spent. A well-configured crawl usually sets all three, so it stays on-topic, shallow enough, and bounded in cost.
Key Takeaways
- Crawl scope defines which links a crawler may follow, by domain, path, and pattern.
- Without it, crawls wander off-site or into infinite faceted spaces.
- Pair scope with depth and budget limits for a fully bounded crawl.
How ScrapeGraphAI Handles This
ScrapeGraphAI's crawl endpoint accepts scope controls so you constrain the crawl to the domain, paths, and patterns you want, and it enforces them alongside depth and page limits automatically.