How Do You Get All URLs From a Website?

TL;DR

The fastest way to get all URLs from a website is to read its sitemap.xml, which lists canonical pages directly. When no sitemap exists or it is incomplete, crawl from the homepage: fetch each page, extract its links, follow the in-scope ones, and track visited URLs until no new pages appear.

Start With the Sitemap

Most sites publish a sitemap at /sitemap.xml (or list one in /robots.txt). It is the site owner's own list of canonical URLs, so it is the cleanest source when it exists:

import requests
from xml.etree import ElementTree
 
xml = requests.get("https://example.com/sitemap.xml", timeout=(5, 30)).text
ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
urls = [loc.text for loc in ElementTree.fromstring(xml).iterfind(".//s:loc", ns)]

Sitemaps can be nested: a sitemap index points to child sitemaps (products, blog, categories). Follow each child to get the full set. See sitemap crawling.

Crawl When There Is No Sitemap

If the sitemap is missing or stale, discover URLs by crawling from a seed and collecting links:

from collections import deque
from urllib.parse import urljoin, urlparse
 
def all_urls(seed):
    host = urlparse(seed).netloc
    queue, seen = deque([seed]), {seed}
    while queue:
        html = fetch(queue.popleft())
        for href in extract_links(html):
            link = urljoin(seed, href).split("#")[0]
            if urlparse(link).netloc == host and link not in seen:
                seen.add(link)
                queue.append(link)
    return seen

Two details prevent noise: strip fragments (#section) and query junk so the same page is not counted twice (URL normalization), and keep the crawl in scope so it does not wander off-site.

Combine Both for Completeness

The thorough approach uses both: seed the crawl with the sitemap URLs, then crawl outward to catch pages the sitemap missed. Sitemaps are often incomplete, and crawling alone can miss orphan pages with no inbound links, so together they give the fullest coverage.

Key Takeaways

Read sitemap.xml first; it is the owner's own canonical URL list.
Follow nested sitemap indexes to get every child sitemap.
With no sitemap, crawl from a seed, normalizing URLs and staying in scope.

How ScrapeGraphAI Handles This

ScrapeGraphAI's crawl endpoint discovers a site's URLs for you by traversing links from a starting URL, with depth limits and include/exclude URL patterns to keep the crawl in scope, so you get the page set without writing the discovery logic.