Python Scrapy Spider: Requests, Responses, and Selectors Explained

Scrapy spiders interact with three core objects: Request, Response, and Selector. The Request describes what to fetch, the Response carries the fetched content, and the Selector extracts the data you need. This article shows how these objects work together in a Python Scrapy spider, from issuing a request to parsing the response with CSS and XPath selectors.

The Minimal Spider That Uses Requests and Responses

A basic spider defines a start_urls list and a parse method. Scrapy turns each URL into a Request, sends it, and calls parse with the resulting Response. Here is a complete spider that scrapes quotes from a demo site:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small.author::text').get(),
            }

The response.css() method returns a SelectorList. Each quote is a Selector, and calling .get() on it returns the first matching value. This pattern is the foundation of most Scrapy spiders.

Creating Request Objects Explicitly

While start_urls is convenient, you often need to generate requests dynamically. Use scrapy.Request to specify a URL, a callback, headers, and metadata. The meta argument lets you pass arbitrary data to the callback. The data is internal to the crawler and is not sent to the target server.

def parse(self, response):
    for href in response.css('li.next a::attr(href)').getall():
        yield scrapy.Request(
            url=response.urljoin(href),
            callback=self.parse,
            meta={'page': response.meta.get('page', 1) + 1},
        )

response.urljoin() converts a relative URL to an absolute one. The meta dictionary is shallow-copied onto the new request, so you can read values from response.meta in the callback. This is how you pass state like page numbers or session tokens between requests.

Working with Response Objects

The Response object exposes the result of a request. Key attributes include response.status, response.headers, response.url, response.body (bytes), and response.text (decoded string). You can check the status code to handle errors or redirects:

def parse(self, response):
    if response.status != 200:
        self.logger.warning('Got status %s for %s', response.status, response.url)
        return
    # proceed with parsing

response.headers is a case-insensitive dict-like object. For example, response.headers.get('Content-Type') returns the Content-Type header value. If you need the raw HTML, response.text is usually what you want; Scrapy decodes it based on the response's encoding headers and page metadata.

Using Selectors to Extract Data

Selectors are the primary tool for extracting structured data from HTML or XML. Scrapy supports both CSS and XPath expressions. The .css() and .xpath() methods return SelectorList objects, and you can chain them.

# CSS selector
title = response.css('h1::text').get()

# XPath selector
price = response.xpath("//span[@class='price']/text()").get()

# Extract all matching elements
items = response.css('ul.items li').getall()

Use .get() for the first match and .getall() for a list of all matches. For attributes, use ::attr(name) in CSS or @name in XPath. Selectors can also be reused: store a Selector object and call .css() or .xpath() on it again when you need to extract multiple fields from the same element.

Passing Data Between Requests with meta

The meta parameter on Request is a dictionary that travels with the request. It is accessible in the callback via response.meta. This is essential for carrying context like pagination counters, unique IDs, or authentication tokens.

def parse(self, response):
    for product in response.css('div.product'):
        yield scrapy.Request(
            url=product.css('a::attr(href)').get(),
            callback=self.parse_product,
            meta={'product_id': product.css('::attr(data-id)').get()},
        )

def parse_product(self, response):
    product_id = response.meta['product_id']
    yield {
        'id': product_id,
        'name': response.css('h1::text').get(),
    }

Note that meta is shallow-copied for each request. If you put a mutable object inside, changes in the callback will affect the original object. For simple scalar values this is not an issue. For complex state, consider using copy.deepcopy or passing immutable values.

Following Relative URLs with response.follow

response.follow() is a convenience method that creates a Request from a relative or absolute URL. It automatically resolves the URL against the current response, so you do not need response.urljoin().

def parse(self, response):
    for next_page in response.css('li.next a'):
        yield response.follow(next_page, callback=self.parse)

You can pass a Selector object directly, and Scrapy will extract the href attribute automatically. This is cleaner than manually extracting the attribute and calling urljoin. response.follow() also supports passing meta, headers, and other Request parameters.

Error Handling and Retry Behavior

Scrapy has a built-in retry middleware that automatically retries failed requests. By default, it retries requests that receive certain HTTP status codes, such as 500, 502, or 503, or that raise connection errors. You can control this via settings such as RETRY_TIMES and RETRY_HTTP_CODES.

For more control, pass an errback to Request. The errback is called when the request fails, allowing you to log the error or schedule a fallback.

def parse(self, response):
    yield scrapy.Request(
        url='https://example.com/api/data',
        callback=self.parse_api,
        errback=self.handle_error,
        meta={'retry_count': 0},
    )

def handle_error(self, failure):
    self.logger.error('Request failed: %s', failure.request.url)
    retries = failure.request.meta.get('retry_count', 0)
    if retries < 3:
        return scrapy.Request(
            url=failure.request.url,
            callback=self.parse_api,
            errback=self.handle_error,
            dont_filter=True,
            meta={'retry_count': retries + 1},
        )
    return None

When rescheduling the same URL yourself, include dont_filter=True; otherwise the default duplicate filter may suppress the retry. This pattern gives you fine-grained control over retry decisions.

Performance and Runtime Considerations

Scrapy is asynchronous and can handle many concurrent requests. The CONCURRENT_REQUESTS setting controls how many requests are processed in parallel. For large crawls, you may need to adjust DOWNLOAD_DELAY to be polite to the target server.

Be mindful of memory usage when handling very large responses. response.body is a bytes object, and Scrapy loads the full body into memory for each response. If you expect very large responses, use settings such as DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE to limit or warn about the size before the full download completes.

Another consideration is duplicate request detection. Scrapy's default filter removes requests with the same fingerprint, which prevents repeated crawls of the same URL. This is useful, but it can also suppress intentional repeat requests. You can customize the duplicate filter with the DUPEFILTER_CLASS setting if that behavior gets in the way.

When you use meta to pass data, remember that it is copied for each request. If you pass large objects, this can increase memory overhead. Keep meta small and focused on scalar values or small identifiers.

Finally, Scrapy infers response encoding from HTTP headers and page metadata. If a site declares an incorrect charset, you may see garbled text. In that case, parse the page with an explicit encoding, such as by supplying encoding when you construct a TextResponse or by normalizing the response in a downloader middleware.