Working with Python Bytes: Creation, Encoding, and Binary I/O
Practical examples for creating, manipulating, encoding, and decoding Python bytes, including binary I/O and performance tips.

When working with binary data, network protocols, or file formats, you need to understand Python's bytes type. A bytes object represents a sequence of integers in the range 0–255, and it behaves differently from str in ways that affect encoding, indexing, and I/O. This article covers the essential operations for creating, manipulating, and converting bytes, along with performance considerations for real-world applications.
bytes vs str: The Core Difference
In Python 3, bytes and str are distinct types. A str object stores Unicode code points, while a bytes object stores raw binary data as a sequence of small integers. This distinction is fundamental: you cannot mix them implicitly. For example, concatenating a str with a bytes raises a TypeError.
Indexing a bytes object returns an integer, not a one-byte bytes object. This surprises many developers coming from Python 2, where str was a byte string. In Python 3, the following behavior is standard:
data = b'hello'
print(data[0]) # 104 (the ASCII value of 'h')To get a one-byte bytes slice, use a slice instead of an index:
print(data[0:1]) # b'h'This distinction matters when you iterate over binary data or compare individual bytes.
Creating bytes Objects
There are several ways to create a bytes object, each suited to different scenarios.
Bytes Literals
The simplest way is a literal with a b prefix:
literal = b'hello'Bytes literals only allow ASCII characters. Non-ASCII characters must be represented as escape sequences, such as \x00 or \xff.
From an Iterable of Integers
You can create a bytes object from any iterable of integers in the range 0–255:
byte_seq = bytes([104, 101, 108, 108, 111])
print(byte_seq) # b'hello'This is useful when constructing binary data from computed values.
From a Hexadecimal String
To parse a hex string into bytes, use bytes.fromhex():
hex_data = bytes.fromhex('48656c6c6f')
print(hex_data) # b'Hello'This is common when reading hex-encoded configuration or network payloads.
From a String via Encoding
To convert a str to bytes, call .encode() on the string with a specific encoding:
text = 'Hello, world'
data = text.encode('utf-8')This is the standard way to prepare text for transmission or storage.
Common Operations on bytes
bytes supports many operations similar to str and list, but with binary semantics.
Slicing and Concatenation
Slicing returns a new bytes object, and concatenation uses the + operator:
b1 = b'foo'
b2 = b'bar'
b3 = b1 + b2 # b'foobar'
sliced = b3[1:4] # b'oob'Repetition and Membership
You can repeat a bytes object with * and test membership with in:
b = b'ab'
print(b * 3) # b'ababab'
print(b'ab' in b) # TrueMethods for Searching and Replacing
bytes has methods like find, startswith, endswith, and replace:
b = b'one two one'
print(b.find(b'two')) # 4
print(b.replace(b'one', b'1')) # b'1 two 1'These methods expect bytes arguments, not str. Passing a str raises a TypeError.
Comparison and Ordering
Bytes compare lexicographically, which is useful for sorting binary data:
print(b'a' < b'b') # TrueEncoding and Decoding Between bytes and str
Converting between text and binary is a frequent task. Use .encode() on str and .decode() on bytes.
text = 'café'
encoded = text.encode('utf-8')
print(encoded) # b'caf\xc3\xa9'
decoded = encoded.decode('utf-8')
print(decoded) # 'café'Choosing an Encoding
UTF-8 is the default and works for most text. For legacy systems, you might use latin-1 or utf-16. The encoding must match what the receiver expects; otherwise, you get garbled data or a UnicodeDecodeError.
Handling Encoding Errors
When decoding, you can control error handling with the errors parameter. Common options are 'strict' (default), 'ignore', 'replace', and 'backslashreplace':
raw = b'\xff\xfe'
try:
raw.decode('utf-8')
except UnicodeDecodeError:
print('Strict mode fails')
print(raw.decode('utf-8', errors='replace')) # '��'
print(raw.decode('utf-8', errors='ignore')) # ''Use 'replace' when invalid bytes should not stop decoding and you want to preserve their positions in the output. Use 'ignore' only when you are sure those invalid bytes are safe to drop.
Working with Binary Data: struct and memoryview
Binary file formats and network protocols often require packing and unpacking structured data. The struct module provides functions to convert between Python values and bytes.
Packing and Unpacking with struct
import struct
# Pack an integer and a float as big-endian
packed = struct.pack('>if', 42, 3.14)
print(packed) # b'\x00\x00\x00*@H\xf5\xc3'
# Unpack back
value_int, value_float = struct.unpack('>if', packed)
print(value_int, value_float) # 42 3.140000104904175The format string '>if' specifies big-endian, a 4-byte integer, and a 4-byte float. Always match the format to the data layout.
Zero-Copy Views with memoryview
memoryview lets you access the underlying buffer of a bytes (or bytearray) without copying. This is valuable for large binary data where slicing would create a new object.
data = b'0123456789'
view = memoryview(data)
slice_view = view[2:5]
print(bytes(slice_view)) # b'234'The slice is a view, not a copy. Modifying the original bytes is impossible (it is immutable), but if you use a bytearray, you can modify the view in place.
Performance and Memory Considerations
bytes objects are immutable, so every operation that creates a new bytes object (like concatenation or slicing) allocates new memory. In tight loops, this can be inefficient.
Use bytearray for Mutable Binary Data
If you need to modify binary data, use bytearray instead of repeatedly creating new bytes objects:
buf = bytearray(b'hello')
buf[0] = 0x48 # 'H'
print(bytes(buf)) # b'Hello'bytearray supports in-place modification and is a drop-in replacement in many contexts.
Avoid Unnecessary Copies with Memoryview
When passing slices of large binary data to functions, use memoryview to avoid copying the entire slice. This is especially important in high-performance network servers or when processing large files.
Concatenation Efficiency
Building a large bytes object by repeated + operations is O(n²). Instead, accumulate parts in a list and join once:
parts = [b'foo', b'bar', b'baz']
result = b''.join(parts)This is both faster and more memory-efficient.
Handling Bytes in I/O and Network Operations
Binary file I/O and socket communication require explicit handling of bytes. Use binary mode ('rb' or 'wb') when opening files so that reads return bytes and writes accept bytes.
with open('data.bin', 'rb') as f:
chunk = f.read(1024)Network sockets return bytes from recv() and expect bytes in sendall():
import socket
s = socket.socket()
s.connect(('example.com', 80))
s.sendall(b'GET / HTTP/1.0\r\n\r\n')
parts = []
while True:
part = s.recv(4096)
if not part:
break
parts.append(part)
response = b''.join(parts)A common pitfall is assuming recv() returns all the data at once. It may return partial data, so you need to loop until the expected length is reached or the connection closes. For fixed-size messages, use a helper to read exactly N bytes:
def recv_exact(sock, n):
data = bytearray()
while len(data) < n:
chunk = sock.recv(n - len(data))
if not chunk:
raise ConnectionError('Connection closed')
data.extend(chunk)
return bytes(data)This pattern is essential for protocols that frame messages by length.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.