// Glossary
Web Crawling
Web crawling strategies, robots.txt protocol, sitemap usage, and crawl management.
21 terms
How Do You Get All URLs From a Website?How Does a Web Crawler Work?What Is a Seed URL in Web Crawling?What Is a URL Frontier in Web Crawling?What Is a Web Crawling API?What is Breadth-First Crawling?What is Crawl Budget?What Is Crawl Delay?What is Crawl Depth?What is Crawl Scheduling?What Is Crawl Scope?What is Distributed Crawling?What Is Focused Crawling?What Is Incremental Crawling?What is JavaScript Rendering?What is Link Discovery?What Is Polite Crawling?What is Sitemap Crawling?What Is the Difference Between a Web Crawler and a Web Spider?What Is the Difference Between Web Crawling and Web Scraping?What is URL Normalization?