TL;DR
Focused crawling is crawling that prioritizes pages relevant to a specific topic or goal instead of traversing a site indiscriminately. The crawler scores links by how likely they are to lead to relevant content and follows the promising ones first, so it collects the pages you want with far fewer requests.
Broad vs Focused
A broad crawl treats every in-scope link equally and tries to visit everything. A focused crawl adds a relevance judgment: before following a link, it estimates whether that link leads toward the target topic, and it orders the frontier so high-relevance links go first. On a large site, that difference is the gap between fetching thousands of pages and fetching the few hundred that actually matter.
How Relevance Gets Scored
Focused crawlers use signals available before or just after fetching:
- Anchor text and surrounding context. A link labeled "Pricing" or "API reference" is a strong hint about the destination.
- URL patterns. Paths like
/docs/or/products/predict relevant content. - Parent page relevance. Links on an already-relevant page tend to lead to more relevant pages.
- Content classification. After fetching, the page's own text confirms relevance and adjusts how its outbound links are scored.
This makes focused crawling a prioritization strategy layered on top of the normal crawl loop, not a different loop.
Why It Matters
Focused crawling saves the two things crawls are short on: time and request budget. Every irrelevant page fetched is crawl budget spent for nothing and one more request the target has to serve. By steering toward relevant content, a focused crawl finishes faster, stays politer, and returns a cleaner set of pages to process downstream.
Key Takeaways
- Focused crawling prioritizes topically relevant links instead of crawling everything.
- Relevance is scored from anchor text, URL patterns, and page content.
- It saves crawl budget and returns a cleaner, more targeted page set.
How ScrapeGraphAI Handles This
ScrapeGraphAI's AI-driven crawl endpoint lets you describe the content you want in natural language, so the crawl concentrates on relevant pages rather than traversing a whole site blindly.