Definition
Pagination handling refers to the techniques used in web scraping to systematically navigate through paginated content — data that is split across multiple pages or loaded incrementally. Most websites displaying lists of items (products, search results, articles) divide content into pages to manage load times and server resources.
Common Pagination Patterns
Page Number Pagination
The most straightforward pattern, where URLs follow a predictable structure like /products?page=1, /products?page=2, etc. Scrapers iterate through page numbers until they reach an empty result or a known last page.
Offset-Based Pagination
Similar to page numbers but using item offsets: /api/items?offset=0&limit=20, then offset=20, offset=40, and so on. Common in API endpoints.
Cursor-Based Pagination
Each response includes a cursor token pointing to the next batch: /api/items?cursor=abc123. The scraper must extract the cursor from each response to request the next page. This pattern is increasingly popular in modern APIs because it handles real-time data changes more gracefully.
Infinite Scroll
Content loads as the user scrolls down, triggered by JavaScript intersection observers or scroll events. There are no explicit page links — new items appear dynamically. Scraping this requires a headless browser that simulates scrolling and waits for new content to render.
Load More Buttons
A variation of infinite scroll where additional content loads only when a button is clicked. Requires either simulating the click in a headless browser or intercepting the underlying API request.
Challenges
- Detecting the last page without explicit total counts
- Handling inconsistent page sizes
- Avoiding duplicate items across pages
- Managing rate limits across many sequential requests
Pagination in ScrapeGraphAI
ScrapeGraphAI's crawling capabilities handle pagination automatically when crawling a site. The platform identifies pagination patterns and follows them to collect complete datasets, managing the sequencing, deduplication, and rate limiting internally.