Introducing ScrapeGraphAI V2 — better, faster, cheaper APIs. Read the blog →
ScrapeGraphAIScrapeGraphAI
Dark

// Glossary

Web Crawling

Web crawling strategies, robots.txt protocol, sitemap usage, and crawl management.

10 terms

What is Breadth-First Crawling?What is Crawl Budget?What is Crawl Depth?What is Crawl Scheduling?What is Depth-First Crawling?What is Distributed Crawling?What is JavaScript Rendering?What is Link Discovery?What is Sitemap Crawling?What is URL Normalization?