Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Detect File and Byte Encoding with Python chardet

Use Python chardet to detect file and byte encoding from raw bytes, interpret confidence scores, and handle short or ambiguous inputs.

~4 min readPython
#chardet#encoding detection#file encoding#Python bytes
A document page with a magnifying glass revealing its character encoding label, representing encoding detection.

Byte sequences are not self-describing. When a Python program reads a file that was not written as UTF-8, the bytes alone do not carry a reliable label saying which encoding produced them. The chardet library addresses this by examining raw bytes and returning a best-guess encoding. This article shows how to use chardet to detect the encoding of raw bytes and files, interpret the confidence score, and handle ambiguous results.

What chardet Returns

chardet.detect() accepts a bytes object and returns a dictionary with three keys: encoding, confidence, and language. The encoding value is a string such as 'utf-8', 'ISO-8859-1', or 'windows-1252'. The confidence value is a float between 0 and 1 that expresses how sure chardet is about its prediction. The language value is often empty or None; it is only populated for encodings that are strongly tied to a specific language, such as koi8-r for Russian.

The function never raises an error for valid bytes input. It always returns a dictionary with those three keys, even when it cannot make a prediction.

Detecting Encoding from Raw Bytes

The simplest usage is passing a bytes object directly to chardet.detect():

python
import chardet raw = b"\xc3\xa9criture" result = chardet.detect(raw) print(result)

The output is similar to:

python
{'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}

The input must be raw bytes. Passing a str object raises a TypeError because chardet needs the byte-level pattern to make a prediction. If your data is already a string, encoding it back to bytes requires choosing an encoding, so detection is not useful for data that has already been decoded.

Reading a File and Detecting Its Encoding

To detect the encoding of a file, open it in binary mode and pass the read bytes to chardet.detect():

python
import chardet with open("data.txt", "rb") as f: raw = f.read() result = chardet.detect(raw) print(result["encoding"])

Reading in binary mode is essential. If you open the file in text mode, Python decodes it using the default locale encoding, which is usually UTF-8 on modern systems. That decoding step changes the bytes and destroys the evidence chardet needs.

Once you have the detected encoding, you can decode the same bytes correctly:

python
text = raw.decode(result["encoding"])

If the confidence is low, decoding with the detected encoding may still produce mojibake. The next section explains how to handle that.

Understanding the Confidence Score

The confidence score is the most useful part of the result for making decisions. A confidence near 1.0 means chardet found a strong statistical match. A confidence below 0.5 should be treated as a guess rather than a fact.

A common pattern is to only trust the result above a threshold:

python
result = chardet.detect(raw) if result["confidence"] > 0.7: text = raw.decode(result["encoding"]) else: text = raw.decode("utf-8", errors="replace")

The threshold depends on your application. A log parser that tolerates occasional wrong output can accept a lower threshold. A data pipeline where corrupted output is costly should require a higher one, or should fall back to asking the caller for an explicit encoding.

Handling Short Inputs and Ambiguous Results

chardet builds a statistical profile from the bytes it sees. With very short inputs, such as a single word, the profile is too small and the prediction may be wrong. Some encodings overlap heavily; for example, Latin-1 and Windows-1252 share most of their byte ranges, so short strings in either encoding are often misclassified.

When the input is too short to analyze, chardet may return None as the encoding. For example, an empty bytes object gives chardet no data to examine:

python
result = chardet.detect(b"") # encoding is None, confidence is 0.0

In that case you need a fallback strategy. For files, reading more of the file usually fixes the problem. For a single short string, you may need to assume UTF-8 or let the caller specify the encoding explicitly.

Performance and Memory Considerations for Large Files

Reading an entire file into memory just to detect its encoding is wasteful for large files. chardet's accuracy improves with more data, but it does not need the whole file. A common approach is to read a sample:

python
with open("large.log", "rb") as f: sample = f.read(100_000) result = chardet.detect(sample)

A sample of 100 KB is usually enough for a reliable prediction. For small files, reading the whole file is simpler and avoids depending on a sample. For large files, sample the beginning and, if possible, a second chunk from the middle to catch BOMs or sections whose byte patterns differ from the start.

The tradeoff is between memory usage and detection accuracy. If the file is small enough to fit in memory comfortably, read it entirely. If it is large, use one or more samples instead.

When chardet Is Not the Right Tool

chardet is a statistical detector, not a validator. It cannot tell you with certainty what encoding a file uses; it gives a probability. If you control the data, you should record the encoding at write time instead of guessing at read time.

For files with a byte-order mark, Python's built-in utf-8-sig, utf-16, and utf-32 codecs handle the BOM automatically when you open the file in text mode. chardet also recognizes BOMs, but the built-in codecs are more reliable for that specific case because they do not rely on statistical guessing.

For UTF-8 validation without detection, raw.decode("utf-8") with an error handler is simpler and faster than running chardet. Use chardet when the encoding is genuinely unknown and you need a best guess to move forward.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.