Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Requests Redirects and Response Encoding

Learn how Python requests handles redirects, why response encoding comes from the final response, and how to set resp.encoding reliably.

~4 min readPython
#requests#encoding#redirects#http
Illustration of a Python HTTP request following a redirect, with a magnifying glass inspecting the response headers for charset detection.

When you use the requests library to fetch a URL that redirects, the library follows the redirect chain by default for normal GET requests and returns the final Response object. That final response determines the encoding you see, not any intermediate redirect. If the final URL is on a different domain or path, the server may send different headers or content, so the encoding can change unexpectedly. This article explains how redirects interact with response encoding in Python requests and how to control encoding reliably.

Redirect Behavior and the Final Response Object

requests handles redirects automatically when allow_redirects is True (the default for GET requests). After the redirect chain completes, the returned Response object corresponds to the last request in the chain. The intermediate responses are preserved in the history attribute as a list of Response objects in the order they were received.

python
import requests resp = requests.get('https://example.com/redirect') print(resp.url) # final URL after redirects print(resp.history) # list of intermediate responses

Because the final response is what you interact with, its encoding attribute and text property reflect the final response's headers and body. If the final URL is a different domain or path, the server may send a different Content-Type header, including a different charset. That is the first place where redirects can affect encoding.

How Requests Determines Response Encoding

The Response.encoding attribute is normally initialized from the final response's Content-Type header. If that header includes a charset parameter, requests uses it directly:

python
resp = requests.get('https://example.com/page') print(resp.encoding) # e.g., 'utf-8' if the header says so

If the header has no charset, the result depends on the Content-Type value. For text/* responses such as text/html, requests commonly defaults to ISO-8859-1. That fallback is often wrong for modern pages, which is why text can appear garbled when a server omits the charset. If no Content-Type header is present, resp.encoding is None, and accessing resp.text will make requests use apparent_encoding, a body-based detection step, before decoding.

Header Encoding, Meta Tags, and Body Detection

The Content-Type header is the authoritative source for encoding when it includes a charset. However, many servers send Content-Type: text/html without a charset parameter. In that case, requests does not parse HTML <meta charset> tags or XML encoding declarations when setting resp.encoding. You may need to set resp.encoding manually based on the actual content, or use apparent_encoding:

python
resp = requests.get('https://example.com/no-charset') # If Content-Type: text/html has no charset, resp.encoding may be 'ISO-8859-1', # even if the HTML contains a <meta charset> tag.

For application/json responses without an explicit charset, requests commonly treats the body as UTF-8. If a server returns JSON using a different encoding, set resp.encoding manually. For XML, check both the HTTP header and the XML declaration for the declared encoding.

Using response.encoding and apparent_encoding

To override the detected encoding, assign a new value to resp.encoding before accessing resp.text:

python
resp.encoding = 'utf-8' text = resp.text

If you are unsure of the correct encoding, resp.apparent_encoding uses a charset detection library to guess from the raw response bytes. This can help when the server sends no encoding information:

python
resp.encoding = resp.apparent_encoding text = resp.text

However, apparent_encoding reads the entire response body to make its guess, which adds overhead and can be slow for large responses. It is also not always accurate, especially for short or mixed-language content. Treat it as a fallback, not the default.

Handling Encoding After Redirects

When a redirect occurs, the Response object you work with is the final one. The history list contains the intermediate responses, each with its own encoding and content. If you need to inspect the encoding at redirect steps, iterate over history:

python
resp = requests.get('https://example.com/redirect') for r in resp.history: print(r.url, r.encoding) print('Final:', resp.url, resp.encoding)

If you need the first response's encoding instead of the final one, set allow_redirects=False and inspect the initial response:

python
resp = requests.get('https://example.com/redirect', allow_redirects=False) print(resp.url, resp.encoding)

This is rarely necessary, but it can help debug cases where the redirect chain changes the encoding. For example, a redirect from a legacy server sending ISO-8859-1 to a modern server sending UTF-8 produces a final response with UTF-8 encoding. If your code expected the original encoding, inspect the final response's headers rather than assuming the first request's header was preserved.

Common Pitfalls and Production Considerations

Set resp.encoding before reading resp.text. The text property decodes resp.content using the current encoding value, so changing encoding after accessing text will not re-decode the content.

Avoid using apparent_encoding for every request. It forces requests to read the entire body into memory and run a charset detection algorithm, which can be expensive for large files. In production, set the encoding explicitly when you know the API or content type. If you must use apparent_encoding, reuse the result so you do not run detection repeatedly on the same response.

Redirects can also introduce security concerns. If a redirect goes to a different domain, the final response may have different trust properties. Validate the final URL if your application depends on the origin. For encoding, verify that the final response's Content-Type header matches your expectations, especially when processing user-supplied URLs.

Finally, resp.content gives the raw bytes without decoding. If you need exact bytes, use resp.content instead of resp.text. For text, set resp.encoding explicitly when the server's default is unreliable, and remember that redirects can change the encoding chain.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.