Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.

When you need reliable message delivery between services, RabbitMQ is a common choice, and pika is the Python client that most developers reach for. Understanding how queues, exchanges, routing keys, and acknowledgements fit together is essential for building a system that doesn't silently drop messages or stall on failures. This article walks through the core mechanics with practical code examples, focusing on the decisions that matter in real applications.
Setting Up a Pika Connection to RabbitMQ
Before any queue or exchange can be used, you need a connection to the broker. Pika provides two connection styles: BlockingConnection for simple, synchronous scripts, and SelectConnection for asynchronous, event-driven applications. Most services start with BlockingConnection because it's easier to reason about, and it's sufficient for the majority of use cases.
import pika
connection = pika.BlockingConnection(
pika.ConnectionParameters(host='localhost')
)
channel = connection.channel()The ConnectionParameters object accepts more than just the host. You can specify port, virtual host, credentials, and socket timeouts. For a production setup, you'll likely pass a PlainCredentials object:
credentials = pika.PlainCredentials('user', 'password')
parameters = pika.ConnectionParameters(
host='rabbitmq.example.com',
port=5672,
virtual_host='/myvhost',
credentials=credentials,
heartbeat=600
)
connection = pika.BlockingConnection(parameters)The heartbeat keeps the connection alive when there's no traffic. If you don't set it, the broker may drop idle connections. Also, note that BlockingConnection is not thread-safe; each thread that needs to communicate with RabbitMQ should have its own connection.
Declaring Queues and Exchanges
Queues hold messages until a consumer picks them up. Exchanges receive messages from producers and route them to queues based on routing keys and exchange type. Declaring both is idempotent when the settings match: calling queue_declare or exchange_declare multiple times is safe if the object already exists. If you redeclare with conflicting properties—for example, a durable queue as non-durable—RabbitMQ raises a precondition failure.
channel.queue_declare(queue='task_queue', durable=True)
channel.exchange_declare(exchange='logs', exchange_type='fanout')Setting durable=True on a queue means it survives a broker restart. However, durability also requires that messages be published with delivery_mode=2 to persist them. For a queue that must not lose messages, you need both.
Exchanges come in four types: direct, fanout, topic, and header. The routing key behavior differs by type. A fanout exchange ignores routing keys entirely and broadcasts to every bound queue. A direct exchange routes messages to queues whose binding key exactly matches the routing key. A topic exchange allows pattern matching with * and # wildcards. header exchanges use message headers instead of routing keys.
Publishing Messages with Routing Keys
To send a message, you publish to an exchange with a routing key. The broker decides which queues should receive it based on the exchange type and the bindings.
channel.basic_publish(
exchange='logs',
routing_key='',
body='Hello World!',
properties=pika.BasicProperties(delivery_mode=2)
)For a fanout exchange, the routing key is often empty because it's ignored. For a direct exchange, the routing key must match a binding key exactly:
channel.exchange_declare(exchange='direct_logs', exchange_type='direct')
channel.queue_declare(queue='error_queue', durable=True)
channel.queue_bind(queue='error_queue', exchange='direct_logs', routing_key='error')
channel.basic_publish(
exchange='direct_logs',
routing_key='error',
body='Critical error message'
)Notice that the queue is bound to the exchange with a specific routing key. When you publish with that routing_key='error', the message lands in error_queue. If no binding matches, the broker drops the message, so make sure a matching binding exists for the routing keys you publish.
Consuming Messages with Callbacks
Consumers subscribe to a queue and receive messages asynchronously. Pika uses a callback function that gets invoked for each message. The callback receives four arguments: the channel, the method frame (which contains delivery information), the message properties, and the message body.
def on_message(ch, method, properties, body):
print(f'Received {body}')
ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(queue='task_queue', on_message_callback=on_message)
print('Waiting for messages. To exit press Ctrl+C')
channel.start_consuming()start_consuming() blocks and processes messages until you stop it. The callback must acknowledge the message (or reject it) to tell RabbitMQ it was handled. If you don't call basic_ack, the message will be redelivered after the consumer disconnects or the channel is closed.
Manual Acknowledgements and Prefetch
By default, Pika uses manual acknowledgement mode: auto_ack=False, so a message is not acknowledged until your callback explicitly acknowledges it. If you set auto_ack=True, RabbitMQ considers the message delivered as soon as it is sent to the consumer, even if the callback crashes before processing. For critical workloads, keep auto_ack=False and acknowledge explicitly.
channel.basic_consume(
queue='task_queue',
on_message_callback=on_message,
auto_ack=False
)Manual ack gives you control. You can basic_ack after successful processing, or basic_nack with requeue=True if you want the message to go back to the queue. This prevents message loss if your consumer crashes mid-processing.
Prefetch controls how many messages are sent to a consumer before it finishes acknowledging the previous ones. Without a prefetch limit, RabbitMQ can send a large backlog to a slow consumer, causing memory spikes. Set basic_qos to limit unacknowledged messages:
channel.basic_qos(prefetch_count=1)This tells RabbitMQ to send at most one message at a time until the consumer acks it. For tasks that take variable time, a prefetch of 1 ensures even distribution across workers.
Handling Connection Failures and Reconnection
Network issues happen. BlockingConnection will raise an exception when the connection drops, and your script will exit unless you catch it. For a long-running consumer, you need to handle reconnection gracefully.
A common pattern is to wrap the connection and channel setup in a function and retry on failure with a backoff delay:
import time
import pika
def connect_with_retry():
while True:
try:
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
return connection
except pika.exceptions.AMQPConnectionError:
print('Connection failed, retrying in 5 seconds...')
time.sleep(5)But this only handles the initial connection. If the connection drops during start_consuming(), you need to catch the exception and reconnect. For a worker process, a retry loop around the whole consume block is usually enough:
def run_consumer():
while True:
connection = connect_with_retry()
try:
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=on_message)
channel.start_consuming()
return
except pika.exceptions.AMQPConnectionError:
print('Connection lost; reconnecting in 5 seconds...')
time.sleep(5)
finally:
if connection.is_open:
connection.close()This avoids recursive retries and closes the old connection before looping, so the worker does not leak sockets.
Choosing Between Blocking and Select Connection
The BlockingConnection is synchronous and blocks the main thread while consuming. It is fine for scripts and simple workers. For high-throughput or event-driven applications, SelectConnection integrates with an I/O loop and allows you to run other tasks while waiting for messages. However, SelectConnection requires a more complex callback structure and manual management of the I/O loop.
If you're building a web service that also consumes messages, you might be better off using a separate worker process with BlockingConnection rather than trying to multiplex with SelectConnection. The decision depends on your concurrency model. If you need to handle many channels or connections in one thread, SelectConnection is the way. If you can scale by running more processes, BlockingConnection is simpler and less error-prone.
A common production setup is to run a pool of worker processes, each with its own BlockingConnection and a prefetch count of 1. This gives you horizontal scaling without the complexity of async code. Only when you need to share a connection across many consumers in a single thread does SelectConnection become necessary.
Routing Key Patterns and Exchange Types in Practice
Routing keys are not just literal strings. With a topic exchange, you can use wildcards to match multiple queues. For example, a routing key logs.* matches logs.info and logs.error but not logs.error.detail. The # wildcard matches zero or more words.
channel.exchange_declare(exchange='topic_logs', exchange_type='topic')
channel.queue_declare(queue='info_queue')
channel.queue_bind(queue='info_queue', exchange='topic_logs', routing_key='logs.*')
channel.basic_publish(exchange='topic_logs', routing_key='logs.info', body='Info message')This pattern is powerful for building flexible routing. But it also introduces a failure mode: if you publish with a routing key that doesn't match any binding, the message is silently dropped. For critical messages, use a direct exchange with a guaranteed binding or enable publisher confirms and mandatory=True so unroutable messages are returned to the client.
Publisher confirms are the recommended way to ensure messages actually reach the broker. In BlockingConnection, enable them with channel.confirm_delivery(). Keep in mind they confirm broker acceptance, not consumer processing—you still need consumer acknowledgements for the full path.
channel.confirm_delivery()
try:
channel.basic_publish(
exchange='',
routing_key='task_queue',
body='important',
mandatory=True
)
except pika.exceptions.UnroutableError:
print('Message could not be routed')Without confirms, you can't be sure the message was accepted. For a production system, enable confirms when you can tolerate the small latency cost.
Final Considerations for Reliable Message Handling
In a manual acknowledgement setup, a message is only truly processed when the consumer acknowledges it. If your consumer crashes after receiving a message but before ack, the message will be redelivered to another consumer. This means your processing logic must be idempotent—handling the same message twice should not cause duplicate side effects. Design your handlers to be safe under redelivery.
Also, consider the tradeoff between basic_ack and basic_reject. If a message is malformed and can never be processed, repeatedly requeueing it will cause an infinite loop. Use basic_reject with requeue=False to discard the message, or, if the queue is configured with a dead-letter exchange, route it to a separate queue for inspection.
Finally, monitor your queue lengths and consumer health. A growing queue with no consumers is a sign of a bottleneck. Use RabbitMQ's management UI or API to track metrics. Pika itself doesn't provide observability, so integrate with your existing monitoring stack to alert on queue depth or unacked messages.
By understanding how queues, exchanges, routing keys, and acknowledgements interact, you can build a messaging layer that is both flexible and reliable. The key is to make deliberate choices about durability, prefetch, and acknowledgement strategy based on your workload's tolerance for loss and latency.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
Python APScheduler Persistent Jobs and SQL Job Stores
python apscheduler persistent jobs and sql job stores: Configure APScheduler SQL job stores so jobs survive restarts. Understand serialization, locking, and how to cho...
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.