Python Scrapy Pagination and Multi-Page Crawling
Learn how to implement pagination in Python Scrapy to crawl multiple pages efficiently. Covers CrawlSpider rules, manual request loops, offset-based URLs, rate limiting, and error handling.

When a website splits its content across multiple pages, a Scrapy spider must follow the pagination chain to collect every item. Pagination is a core pattern in most scraping projects, and Scrapy offers several ways to handle it. The right approach depends on the site's URL structure, whether the next-page link is a real HTML anchor, and how much control you need over the crawl flow.
Understanding Pagination in Scrapy
Pagination means the site presents content in numbered or sequentially linked pages. A typical example is an e-commerce category page that shows 24 products per page and provides a "Next" button or a page-numbered URL like ?page=2. For a Scrapy spider to collect all products, it must issue requests for each page and parse the same item extraction logic on every response.
Scrapy does not automatically follow pagination. You must explicitly tell the spider how to find the next page URL and when to stop. The two most common strategies are using CrawlSpider with LinkExtractor rules, or manually constructing Request objects inside a parse method. Both work, but they differ in how much control you have and how they handle edge cases.
Using CrawlSpider with LinkExtractor for Automatic Pagination
CrawlSpider is a Scrapy spider class designed to crawl links that match a set of rules. It is a good fit when the next-page link is a standard <a> tag with a predictable pattern. You define a Rule that tells the spider which links to follow and a callback that parses the item data.
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class ProductSpider(CrawlSpider):
name = "products"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/products"]
rules = (
Rule(
LinkExtractor(restrict_xpaths="//a[@rel='next']"),
callback="parse_item",
follow=True,
),
)
def parse_start_url(self, response):
return self.parse_item(response)
def parse_item(self, response):
for product in response.xpath("//div[@class='product']"):
yield {
"name": product.xpath(".//h2/text()").get(),
"price": product.xpath(".//span[@class='price']/text()").get(),
}In this example, LinkExtractor looks for an anchor with rel="next", a common semantic marker for pagination. When CrawlSpider follows one of those links, it calls parse_item on the new response. Because follow=True, it also continues extracting links from that response, so the spider walks through the pagination chain. The parse_start_url method delegates to parse_item so the product data on the first page is not skipped.
CrawlSpider handles request scheduling and deduplication for you. However, it is less flexible when the next-page URL is not a simple anchor, or when you need to pass data between pages.
Manual Pagination with parse and Request Callbacks
When you need more control, you can write a regular Spider and manually yield Request objects for the next page. This is useful when the pagination uses query parameters, when you need to modify headers or cookies between pages, or when you want to stop based on a condition other than a missing link.
import scrapy
class ProductSpider(scrapy.Spider):
name = "products_manual"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/products?page=1"]
def parse(self, response):
for product in response.xpath("//div[@class='product']"):
yield {
"name": product.xpath(".//h2/text()").get(),
"price": product.xpath(".//span[@class='price']/text()").get(),
}
next_page = response.xpath("//a[@rel='next']/@href").get()
if next_page:
yield scrapy.Request(response.urljoin(next_page), callback=self.parse)Here, response.urljoin resolves relative links. The callback is the same parse method, so the spider continues until no next link is found. This approach gives you full control over the request. You can add headers, metadata, or change the callback based on the page number.
A common variation is to construct the next URL manually when the site uses a numeric parameter but does not expose a next link. Include a stopping condition to avoid an endless crawl:
MAX_PAGES = 10
class ProductSpider(scrapy.Spider):
name = "products_manual"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/products?page=1"]
def parse(self, response):
for product in response.xpath("//div[@class='product']"):
yield {
"name": product.xpath(".//h2/text()").get(),
"price": product.xpath(".//span[@class='price']/text()").get(),
}
page_number = response.meta.get("page_number", 1)
if page_number < MAX_PAGES:
yield scrapy.Request(
f"https://example.com/products?page={page_number + 1}",
callback=self.parse,
meta={"page_number": page_number + 1},
)This pattern is useful when the site does not provide a next link but uses a predictable URL scheme. Define a stopping condition, such as a maximum page number, an empty response, or a page that returns no new items.
Handling Pagination Parameters and Dynamic URLs
Some sites use offset-based pagination (?offset=24), cursor-based pagination, or even POST requests for the next page. In those cases, you cannot rely on a simple LinkExtractor. You need to parse the response to extract the next page identifier, or compute it from the current state.
For offset-based pagination, you can read the number of items on the page and increment the offset. For example, if each page shows 24 items, the next offset is current_offset + 24. Store the current offset in the request meta.
class OffsetSpider(scrapy.Spider):
name = "offset"
start_urls = ["https://example.com/api/items?offset=0"]
def parse(self, response):
items = response.json().get("items", [])
for item in items:
yield item
if len(items) == 24: # assume full page means more
offset = response.meta.get("offset", 0) + 24
yield scrapy.Request(
f"https://example.com/api/items?offset={offset}",
callback=self.parse,
meta={"offset": offset}
)Dynamic URLs that require JavaScript rendering cannot be handled by Scrapy alone. You may need to use a headless browser like Splash or Selenium to execute the JavaScript and extract the next page link. Scrapy integrates with Splash via scrapy-splash, but that adds a dependency and operational overhead. Before going down that path, check whether the site exposes a JSON API or a hidden HTML link that Scrapy can use directly.
Performance and Rate Limiting Considerations
Multi-page crawling multiplies the number of requests your spider sends. Without throttling, you risk overwhelming the target server and getting your IP blocked. Scrapy provides built-in settings to control concurrency and delay.
CONCURRENT_REQUESTScontrols how many requests are processed in parallel.DOWNLOAD_DELAYsets a fixed minimum delay between requests.AUTOTHROTTLE_ENABLEDlets Scrapy adjust the delay based on server response times.
For a polite crawler, a common setup is to enable autothrottle:
# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0With autothrottle enabled, Scrapy adjusts the delay between AUTOTHROTTLE_START_DELAY and AUTOTHROTTLE_MAX_DELAY. If you prefer a fixed delay, leave autothrottle disabled and set DOWNLOAD_DELAY; RANDOMIZE_DOWNLOAD_DELAY can add slight variation. You can also pass per-request meta keys such as download_timeout or dont_retry for finer control. For long crawls, JOBDIR lets you pause and resume, and LOG_LEVEL can reduce log noise.
Error Handling and Robustness in Pagination
Pagination chains can break. The next link may disappear temporarily, the site may return a 404, or the page structure may change. A robust spider should handle these failures gracefully.
Scrapy's retry middleware retries requests based on RETRY_TIMES and RETRY_HTTP_CODES. You can also implement your own stop conditions. For example, if the next page returns an empty item list, you might stop rather than continue.
def parse(self, response):
items = response.xpath("//div[@class='product']")
if not items:
self.logger.info("No items found, stopping pagination")
return
for item in items:
yield {...}
next_link = response.xpath("//a[@rel='next']/@href").get()
if next_link:
yield scrapy.Request(response.urljoin(next_link), callback=self.parse)Another concern is duplicate pages. Scrapy's default request deduplication prevents the same URL from being requested twice. If URLs contain tracking parameters or session tokens, normalize them when you build or extract links so the same page is not requested under different URLs. If you intentionally need to request the same URL again, pass dont_filter=True to Request.
When using CrawlSpider, the Rule can include a process_request function to modify the request before it is sent. This is useful for adding headers or changing the callback based on the URL. For manual pagination, you have full control over the request lifecycle, so you can implement custom retry logic or stop conditions.
A common edge case is an infinite pagination loop caused by a next link that points to the current page. In manual pagination, compare the next URL with response.url and stop if they match. In CrawlSpider, the link extractor may follow the same link repeatedly, but Scrapy's deduplication will eventually skip it; even so, restrict the LinkExtractor with allow or deny patterns to avoid unwanted links.
For sites that require a login or session, you must handle authentication before starting the pagination chain. Use cookies in the request or a DownloaderMiddleware to attach session cookies. The pagination logic itself remains the same, but the initial request must be authenticated.
Finally, avoid accumulating every item in memory. Yield items as you parse them so the item pipeline can process them incrementally. If you need to aggregate data across pages, store the results in a database or file rather than a Python list that grows for the whole crawl.
Pagination is a fundamental part of web scraping with Scrapy. Whether you choose CrawlSpider for simplicity or manual requests for control, the key is to understand the site's pagination mechanism and to build a spider that can handle failures and respect the server's resources. The patterns shown here cover the majority of pagination scenarios you will encounter, from simple next links to offset-based APIs and dynamic rendering workarounds.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.