How to Manage Headers, Authentication, and Cookies in Python aiohttp

When building HTTP clients with aiohttp, you often need to send custom headers, authenticate with tokens, and maintain cookies across requests. The ClientSession object is the core of this behavior, and understanding how headers, authentication, and cookies interact will save you from subtle bugs in production. This article covers practical patterns for these cases on both the client and server side.

Setting Headers on aiohttp Requests

Every aiohttp request accepts a headers parameter that overrides or adds to the session's default headers. The simplest way to send a custom header is to pass a dictionary directly to the request method:

import aiohttp
import asyncio

async def fetch_with_header():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            'https://api.example.com/data',
            headers={'X-API-Key': 'my-secret-key'}
        ) as resp:
            return await resp.text()

For repeated requests, define headers at the session level. This avoids repeating the same dictionary on every call and keeps your code DRY:

async def session_with_default_headers():
    headers = {'User-Agent': 'MyApp/1.0', 'Accept': 'application/json'}
    async with aiohttp.ClientSession(headers=headers) as session:
        async with session.get('https://api.example.com/endpoint') as resp:
            print(resp.status)

When you pass headers both to the session and to an individual request, the request-level headers take precedence for keys that overlap. This is useful for overriding a default header for a specific call without mutating the session.

Authentication Patterns: Basic, Bearer, and Custom

Authentication typically involves setting an Authorization header. aiohttp provides a helper for Basic auth, but you can also construct the header manually for Bearer tokens or custom schemes.

Basic Authentication

Use aiohttp.BasicAuth to encode credentials and set the header automatically:

async def basic_auth_example():
    auth = aiohttp.BasicAuth('username', 'password')
    async with aiohttp.ClientSession(auth=auth) as session:
        async with session.get('https://api.example.com/secure') as resp:
            print(resp.status)

The auth parameter is passed to the session, and aiohttp applies it to every request unless a request supplies its own auth. If you need different credentials for specific requests, pass auth to the request method instead.

Bearer Token Authentication

For token-based APIs, set the Authorization header directly:

async def bearer_auth_example(token):
    headers = {'Authorization': f'Bearer {token}'}
    async with aiohttp.ClientSession(headers=headers) as session:
        async with session.get('https://api.example.com/profile') as resp:
            return await resp.json()

If the token changes frequently, avoid storing it in the session headers. Instead, pass it per request or use a TraceConfig hook to inject the current token before each request.

Custom Authentication Schemes

Some APIs use non-standard authentication headers such as X-API-Key, or expect custom schemes in the Authorization header. These are just headers, so you can set them like any other header. The key is to keep the authentication logic centralized, perhaps in a helper function that returns the headers for a given request.

Managing Cookies with aiohttp

Cookies are essential for maintaining state across requests, especially for login-based flows. aiohttp's ClientSession automatically stores cookies from responses and sends them on subsequent requests to the same domain.

async def cookie_flow():
    async with aiohttp.ClientSession() as session:
        # First request logs in and receives a session cookie
        async with session.post('https://example.com/login', data={'user': 'a', 'pass': 'b'}) as resp:
            await resp.text()  # Read/release the response so the session can reuse the connection
        # Subsequent requests automatically include the cookie
        async with session.get('https://example.com/dashboard') as resp:
            print(resp.status)

You can also manually set cookies using the cookies parameter of ClientSession:

async def manual_cookie():
    async with aiohttp.ClientSession(cookies={'session_id': 'abc123'}) as session:
        async with session.get('https://example.com/') as resp:
            print(resp.status)

If you use a custom CookieJar, call update_cookies(cookies, response_url=...) so the cookie is associated with a domain.

To inspect cookies after a response, access session.cookie_jar. This is useful for debugging or for extracting tokens that the server sets.

Server-Side: Reading Headers and Cookies

On the server side, aiohttp handlers receive a Request object with .headers and .cookies properties. You can access authentication information from the headers and validate it.

from aiohttp import web

async def handler(request):
    auth_header = request.headers.get('Authorization', '')
    if auth_header.startswith('Bearer '):
        token = auth_header[7:]
        # Validate token
    user_cookie = request.cookies.get('session_id')
    return web.json_response({'status': 'ok'})

When building a server, you often need to set cookies in the response. Use web.Response and its set_cookie method:

async def login_handler(request):
    resp = web.Response(text='Logged in')
    resp.set_cookie('session_id', 'new-session-token', max_age=3600, httponly=True)
    return resp

Setting httponly=True prevents JavaScript from accessing the cookie, which is a basic security measure.

Handling Authentication Errors and Redirects

Authentication often fails with 401 Unauthorized or 403 Forbidden. Your client should handle these statuses explicitly rather than assuming success. A common pattern is to retry once with fresh credentials if the token has expired.

async def request_with_retry(session, url, token):
    headers = {'Authorization': f'Bearer {token}'}
    async with session.get(url, headers=headers) as resp:
        if resp.status == 401:
            # Refresh token and retry
            new_token = await refresh_token()
            headers['Authorization'] = f'Bearer {new_token}'
            async with session.get(url, headers=headers) as retry_resp:
                return retry_resp
        return resp

Redirects can also affect authentication. By default, aiohttp follows redirects, and the Authorization header is stripped when redirecting to a different host. This is a security feature, but it can break flows that rely on cross-domain redirects. Use allow_redirects=False if you need to handle redirects manually.

Security Considerations: TLS, Cookie Flags, and Session Reuse

When dealing with authentication and cookies, security is not optional. Always use HTTPS in production to prevent credentials from being intercepted. aiohttp verifies TLS certificates by default; disable verification only for local testing with ssl=False, and never in production.

Cookie flags matter. When setting cookies server-side, use secure=True to send them only over HTTPS, and httponly=True to prevent client-side script access. The samesite attribute can mitigate CSRF attacks. On the client side, be careful when storing cookies from untrusted domains; aiohttp's CookieJar has an unsafe flag that controls whether it accepts cookies from IP addresses, and the default is conservative.

Session reuse is another concern. If you are using a ClientSession for a long-lived process, periodically rotate the authentication token if the server supports it. Also, avoid logging headers that contain sensitive tokens. Use repr() carefully and redact Authorization and Cookie headers in logs.

Common Pitfalls and Debugging

Several issues commonly trip up developers when working with aiohttp headers, authentication, and cookies.

Header Case and Duplicate Headers

HTTP header names are case-insensitive. aiohttp stores headers in a case-insensitive multidict, so request.headers['authorization'] and request.headers.get('Authorization') both work. If a header can appear more than once, use request.headers.getall(name) when you need every value. Set-Cookie is handled separately by aiohttp's cookie jar.

Cookies Not Being Sent

If cookies are not sent on subsequent requests, check that the domain and path of the cookie match the request URL. Also read and release the response before the session context exits; aiohttp updates its cookie jar from response headers, and releasing the response lets the session reuse the connection.

Authentication Header Overwritten

If you set Authorization at the session level and then pass a different Authorization in a request, the request-level value wins. If you need to omit a session-level header for a specific request, pass skip_headers={'Authorization'} to the request method.

Debugging with Trace

For complex issues, use a TraceConfig hook to inspect the actual headers being sent. The advanced example below shows how to attach one. Enabling aiohttp's debug logging can surface low-level client messages, but it is not a substitute for inspecting the request yourself.

Advanced Usage: Custom Cookie Jar and Header Injection

For advanced scenarios, you can subclass aiohttp.CookieJar to implement custom cookie policies, such as ignoring cookies from certain domains. Similarly, you can create a custom TraceConfig to inject headers into every request before it is sent. This is useful for adding correlation IDs or dynamic tokens without repeating code.

from aiohttp import ClientSession, TraceConfig

async def on_request_start(session, trace_config_ctx, params):
    params.headers['X-Correlation-ID'] = 'generated-id'

trace_config = TraceConfig()
trace_config.on_request_start.append(on_request_start)

async def main():
    async with ClientSession(trace_configs=[trace_config]) as session:
        async with session.get('https://example.com') as resp:
            print(resp.status)

This pattern keeps your request logic clean and centralizes cross-cutting concerns like authentication and tracing. Just be careful not to override headers that are set explicitly on the request method, as the trace callback runs before the request is sent and may conflict with per-request headers.