Python Scrapy Middleware for Cookies, Proxy, and User Agent
python scrapy middleware cookies proxy and user agent: Learn how to build Scrapy middleware to manage cookies, rotate user agents, and route requests through proxies,...

python scrapy middleware cookies proxy and user agent requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.
When a Scrapy spider needs to handle cookies, rotate user agents, or route traffic through proxies, the cleanest place to implement that logic is in downloader middleware. These components sit between the engine and the downloader, allowing you to modify each request before it leaves the process and each response before it reaches the spider. This article explains how to write and configure middleware for cookies, proxy, and user agent handling in a Python Scrapy project.
Why Middleware Is the Right Place for Request Modification
Scrapy's downloader middleware is a framework for intercepting requests and responses globally. Each request passes through a chain of middleware objects, and each can modify the request, return a response directly, or raise an exception. This is ideal for cross-cutting concerns like setting a User-Agent header, attaching a proxy address, or managing cookies—concerns that apply to many requests rather than a single spider.
Spider middleware, in contrast, operates on the spider's input and output. For request-level attributes like headers and meta, downloader middleware is the correct layer. You can also use the process_request method to alter the request object before it is sent.
Handling Cookies with Scrapy Middleware
Scrapy ships with a built-in CookiesMiddleware that maintains a cookie jar per domain. It is enabled by default and works transparently for most sessions. However, you may need to customize behavior, such as using a specific cookie for authentication or persisting cookies across runs.
To override the default behavior, you can create a custom middleware that modifies the Cookie header directly. For example, to attach a fixed session cookie:
class FixedCookieMiddleware:
def __init__(self, cookie_value):
self.cookie_value = cookie_value
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings.get('FIXED_COOKIE'))
def process_request(self, request, spider):
if self.cookie_value:
request.headers['Cookie'] = self.cookie_value
return NoneThis middleware sets the Cookie header for every request. If you need to manage multiple cookies or handle expiration, you can store them in a dictionary and update the header based on the request URL.
The built-in CookiesMiddleware uses the cookiejar meta key to isolate sessions. If you want to use a specific jar for a request, set request.meta['cookiejar'] = 'session1'. This is useful when a spider handles multiple login sessions simultaneously.
Rotating User Agents in Middleware
Websites often block requests that use a default or repeated User-Agent string. Rotating the User-Agent per request reduces the chance of detection. You can implement a middleware that cycles through a list of common browser strings.
import random
class RandomUserAgentMiddleware:
def __init__(self, user_agents):
self.user_agents = user_agents
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings.getlist('USER_AGENTS'))
def process_request(self, request, spider):
if self.user_agents:
request.headers['User-Agent'] = random.choice(self.user_agents)
return NoneSet USER_AGENTS in settings.py as a list of strings. The middleware randomly picks one for each request. If you need a more deterministic rotation, you can use a counter based on the request index.
Note that Scrapy's default UserAgentMiddleware sets a single User-Agent from the USER_AGENT setting. Your custom middleware should be placed after the default in the middleware order to override it, or you can disable the default and rely entirely on your own.
Managing Proxies with Middleware
Proxies are often required to avoid IP bans or to access geo-restricted content. Scrapy does not include a built-in proxy middleware, so you must implement one. The standard approach is to set the proxy key in request.meta to the proxy URL. The downloader then uses that address for the connection.
class ProxyMiddleware:
def __init__(self, proxies):
self.proxies = proxies
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings.getlist('PROXIES'))
def process_request(self, request, spider):
if self.proxies:
request.meta['proxy'] = random.choice(self.proxies)
return NoneFor authenticated proxies, include credentials in the URL, such as http://user:pass@host:port. Be careful with sensitive credentials in logs; avoid printing the proxy URL.
You can also assign proxies per request based on the domain or spider. For example, use a different proxy for each request to avoid reusing the same IP for consecutive requests.
Combining Cookies, Proxy, and User Agent in One Middleware
While separate middleware classes are easier to maintain, you can combine all three concerns into a single middleware if the logic is tightly coupled. For instance, when a proxy requires a specific cookie or when the User-Agent should be consistent with the proxy's region.
class UnifiedRequestMiddleware:
def __init__(self, user_agents, proxies, cookie):
self.user_agents = user_agents
self.proxies = proxies
self.cookie = cookie
@classmethod
def from_crawler(cls, crawler):
return cls(
crawler.settings.getlist('USER_AGENTS'),
crawler.settings.getlist('PROXIES'),
crawler.settings.get('FIXED_COOKIE')
)
def process_request(self, request, spider):
if self.user_agents:
request.headers['User-Agent'] = random.choice(self.user_agents)
if self.proxies:
request.meta['proxy'] = random.choice(self.proxies)
if self.cookie:
request.headers['Cookie'] = self.cookie
return NoneThis approach reduces the number of middleware classes but makes the code less modular. Use it only when the settings are always applied together.
Configuring Middleware Order in Settings
Scrapy processes downloader middleware in the order defined by the DOWNLOADER_MIDDLEWARES setting. The order matters because each middleware can modify the request before the next one sees it. For example, if you want your proxy middleware to run before the user-agent middleware, list it earlier.
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.ProxyMiddleware': 543,
'myproject.middlewares.RandomUserAgentMiddleware': 544,
'myproject.middlewares.FixedCookieMiddleware': 545,
}The numeric values determine the order: lower numbers run first. The built-in middleware have default orders; you can override them by setting the same class with a different number. To disable a built-in middleware, set its value to None.
| Middleware Class | Default Order | Purpose |
|---|---|---|
CookiesMiddleware | 700 | Maintains cookie jars |
UserAgentMiddleware | 400 | Sets default User-Agent |
RetryMiddleware | 550 | Retries failed requests |
ProxyMiddleware (custom) | 543 | Assigns proxy per request |
This table shows typical orders. Your custom middleware should be placed before the built-in ones if you need to override their behavior. For instance, to override the default User-Agent, your middleware must run before UserAgentMiddleware (order < 400) or disable it entirely.
Common Pitfalls and Production Considerations
One common mistake is forgetting that process_request must return None or a Request/Response object. Returning None lets the request continue to the next middleware. Returning a Request object stops the chain and sends that new request instead. Returning a Response object bypasses the downloader entirely.
When rotating proxies, be aware that not all proxies support HTTPS or specific HTTP methods. Test your proxy list before deploying. Also, some proxies leak your original IP if the X-Forwarded-For header is not set correctly. You can set this header in the middleware, but be aware that many sites ignore it.
Cookie management with multiple sessions requires careful handling of the cookiejar meta key. If you reuse the same jar across requests, you may inadvertently share session state between different spiders or domains. Use unique jar names per logical session.
Performance is another consideration. Randomly selecting a proxy from a list is cheap, but if you need to check proxy availability or maintain a pool of healthy proxies, you may need to integrate a proxy service or a health-check mechanism. Avoid doing I/O in process_request unless necessary, because it runs for every request and can slow down the crawl.
Security matters when handling proxy credentials. Never log the full proxy URL, and consider using environment variables or Scrapy's settings with sensitive values redacted. Also, be cautious when setting cookies from external sources; validate the cookie domain and path to prevent session fixation attacks.
Finally, test your middleware in a staging environment before running a large crawl. Use Scrapy's scrapy shell to verify that headers and meta are set correctly on individual requests. This helps catch configuration errors early without burning through your proxy quota or triggering anti-bot measures.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.