Detect File and Byte Encoding with Python chardet
Use Python chardet to detect file and byte encoding from raw bytes, interpret confidence scores, and handle short or ambiguous inputs.

Byte sequences are not self-describing. When a Python program reads a file that was not written as UTF-8, the bytes alone do not carry a reliable label saying which encoding produced them. The chardet library addresses this by examining raw bytes and returning a best-guess encoding. This article shows how to use chardet to detect the encoding of raw bytes and files, interpret the confidence score, and handle ambiguous results.
What chardet Returns
chardet.detect() accepts a bytes object and returns a dictionary with three keys: encoding, confidence, and language. The encoding value is a string such as 'utf-8', 'ISO-8859-1', or 'windows-1252'. The confidence value is a float between 0 and 1 that expresses how sure chardet is about its prediction. The language value is often empty or None; it is only populated for encodings that are strongly tied to a specific language, such as koi8-r for Russian.
The function never raises an error for valid bytes input. It always returns a dictionary with those three keys, even when it cannot make a prediction.
Detecting Encoding from Raw Bytes
The simplest usage is passing a bytes object directly to chardet.detect():
import chardet
raw = b"\xc3\xa9criture"
result = chardet.detect(raw)
print(result)The output is similar to:
{'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}The input must be raw bytes. Passing a str object raises a TypeError because chardet needs the byte-level pattern to make a prediction. If your data is already a string, encoding it back to bytes requires choosing an encoding, so detection is not useful for data that has already been decoded.
Reading a File and Detecting Its Encoding
To detect the encoding of a file, open it in binary mode and pass the read bytes to chardet.detect():
import chardet
with open("data.txt", "rb") as f:
raw = f.read()
result = chardet.detect(raw)
print(result["encoding"])Reading in binary mode is essential. If you open the file in text mode, Python decodes it using the default locale encoding, which is usually UTF-8 on modern systems. That decoding step changes the bytes and destroys the evidence chardet needs.
Once you have the detected encoding, you can decode the same bytes correctly:
text = raw.decode(result["encoding"])If the confidence is low, decoding with the detected encoding may still produce mojibake. The next section explains how to handle that.
Understanding the Confidence Score
The confidence score is the most useful part of the result for making decisions. A confidence near 1.0 means chardet found a strong statistical match. A confidence below 0.5 should be treated as a guess rather than a fact.
A common pattern is to only trust the result above a threshold:
result = chardet.detect(raw)
if result["confidence"] > 0.7:
text = raw.decode(result["encoding"])
else:
text = raw.decode("utf-8", errors="replace")The threshold depends on your application. A log parser that tolerates occasional wrong output can accept a lower threshold. A data pipeline where corrupted output is costly should require a higher one, or should fall back to asking the caller for an explicit encoding.
Handling Short Inputs and Ambiguous Results
chardet builds a statistical profile from the bytes it sees. With very short inputs, such as a single word, the profile is too small and the prediction may be wrong. Some encodings overlap heavily; for example, Latin-1 and Windows-1252 share most of their byte ranges, so short strings in either encoding are often misclassified.
When the input is too short to analyze, chardet may return None as the encoding. For example, an empty bytes object gives chardet no data to examine:
result = chardet.detect(b"")
# encoding is None, confidence is 0.0In that case you need a fallback strategy. For files, reading more of the file usually fixes the problem. For a single short string, you may need to assume UTF-8 or let the caller specify the encoding explicitly.
Performance and Memory Considerations for Large Files
Reading an entire file into memory just to detect its encoding is wasteful for large files. chardet's accuracy improves with more data, but it does not need the whole file. A common approach is to read a sample:
with open("large.log", "rb") as f:
sample = f.read(100_000)
result = chardet.detect(sample)A sample of 100 KB is usually enough for a reliable prediction. For small files, reading the whole file is simpler and avoids depending on a sample. For large files, sample the beginning and, if possible, a second chunk from the middle to catch BOMs or sections whose byte patterns differ from the start.
The tradeoff is between memory usage and detection accuracy. If the file is small enough to fit in memory comfortably, read it entirely. If it is large, use one or more samples instead.
When chardet Is Not the Right Tool
chardet is a statistical detector, not a validator. It cannot tell you with certainty what encoding a file uses; it gives a probability. If you control the data, you should record the encoding at write time instead of guessing at read time.
For files with a byte-order mark, Python's built-in utf-8-sig, utf-16, and utf-32 codecs handle the BOM automatically when you open the file in text mode. chardet also recognizes BOMs, but the built-in codecs are more reliable for that specific case because they do not rely on statistical guessing.
For UTF-8 validation without detection, raw.decode("utf-8") with an error handler is simpler and faster than running chardet. Use chardet when the encoding is genuinely unknown and you need a best guess to move forward.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.