What Is a Seed URL in Web Crawling?

TL;DR

A seed URL is the starting point a crawler begins from. The crawler fetches the seed, extracts its links, and follows them outward, so the seed determines which part of the web the crawl can reach. Good seeds are high-connectivity pages like a homepage, sitemap, or category index.

Why the Seed Matters

A crawler can only reach pages that are linked, directly or indirectly, from its seeds. Pick a dead-end page as your seed and the crawl stops almost immediately. Pick a hub that links to everything you care about and the crawl fans out to cover the whole section. The seed is the single most important input to coverage, ahead of any traversal setting.

Choosing Good Seeds

The best seeds are pages with high outbound connectivity into your target area:

• Homepage. Links to every top-level section; a safe default.
• Sitemap. A sitemap lists canonical URLs directly, often the fastest way to seed a complete crawl.
• Category or index pages. For a product or article crawl, the relevant listing page reaches every item with minimal depth.

Multiple seeds are common. Seeding with a homepage plus several category pages shortens the path to deep content and reduces the crawl depth needed to reach it.

seeds = [
    "https://example.com/",
    "https://example.com/sitemap.xml",
    "https://example.com/products",
]
frontier = deque(seeds)   # the crawl starts from all seeds at once

Seeds and Scope

The seed pairs with crawl scope rules. A seed sets where the crawl starts; scope rules decide which discovered links it is allowed to follow. Together they keep a crawl focused on the right section instead of drifting across the whole domain or off-site.

Key Takeaways

• The seed is the crawler's starting URL and bounds what it can reach.
• Prefer high-connectivity seeds: homepage, sitemap, or category index.
• Use multiple seeds to reach deep content with less depth.

How ScrapeGraphAI Handles This

With the crawl endpoint you provide the starting URL and rules for what to collect, and the platform manages the frontier, depth, and deduplication from there, so seeding is a single clear input rather than a tuning exercise.