Python Celery: Redis and RabbitMQ Brokers, Result Backends
Configure Python Celery with Redis and RabbitMQ as brokers and result backends. Compare delivery guarantees, visibility timeouts, and failure modes.

A common Celery deployment uses RabbitMQ as the broker and Redis as the result backend. The broker transports task messages from your application to workers; the result backend stores the state and return value of each executed task. Redis can also handle both roles, but the delivery guarantees and failure modes differ.
How the Broker and Result Backend Fit Together
A task is published to the broker as a message. A worker consumes that message, executes the task, and writes the outcome to the result backend. The application that called the task can then read the result from the backend.
This separation means you are not forced to use the same technology for both roles. The choice depends on reliability requirements, feature needs, and operational constraints.
The two relevant settings in your Celery configuration are broker_url and result_backend. Both accept a URL string that identifies the transport and its connection details.
Configuring RabbitMQ as the Broker
RabbitMQ is a common choice for the broker. It implements the AMQP protocol and provides durable queues, message acknowledgments, and priority support. For production workloads where message loss is unacceptable, RabbitMQ is the safer default.
A minimal configuration using RabbitMQ as the broker looks like this:
broker_url = "amqp://user:password@localhost:5672//"
result_backend = "redis://localhost:6379/0"The amqp:// scheme points at the RabbitMQ server. The URL includes the credentials, host, port, and virtual host. The trailing // selects the default virtual host. If you use a dedicated vhost, replace it with the vhost name.
RabbitMQ's behavior matters for task delivery. Workers acknowledge messages after processing, and Celery can be configured to acknowledge late or early. With task_acks_late = True, a worker acknowledges a message only after the task completes, so a crashed worker causes the message to be redelivered rather than lost.
Configuring Redis as the Broker
Redis can also serve as the broker. Celery implements the Redis broker using Redis lists and blocking list operations rather than pub/sub. Messages are pushed to a list, and workers pop them with BRPOPLPUSH, which moves each message to a separate unacknowledged list while it is being handled.
broker_url = "redis://localhost:6379/0"
result_backend = "redis://localhost:6379/1"The URL selects a database number after the port. Using a separate database for the broker and the result backend avoids one role interfering with the other's keyspace.
The Redis broker has a notable operational parameter: broker_transport_options with visibility_timeout. Because Redis does not track message acknowledgments the way AMQP does, Celery uses a visibility timeout to decide when a message that was popped but never acknowledged should be made visible to other workers again. The default is one hour. If a task runs longer than the visibility timeout, another worker may pick up the same message and execute the task twice. For long-running tasks, increase the timeout:
broker_transport_options = {"visibility_timeout": 7200}This is the main reliability difference between the two brokers. RabbitMQ tracks acknowledgment state explicitly; Redis relies on a timeout heuristic.
Using Redis as the Result Backend
Redis is the most widely used result backend for Celery. It stores task state and results as keys with a configurable expiration. Results are written when the task finishes, and the calling application reads them by task ID.
result_backend = "redis://localhost:6379/1"
result_expires = 86400result_expires controls how long results remain in Redis before Celery removes them. The default is one day. In a high-throughput system, results accumulate quickly, so setting a sensible expiration prevents Redis memory from growing without bound.
Redis as a result backend stores everything in memory. A task that returns a multi-megabyte structure will consume memory for the duration of result_expires. Keep result payloads small and set an expiration that matches how long callers actually need the data.
Mixing RabbitMQ Broker with Redis Result Backend
The combination of RabbitMQ as the broker and Redis as the result backend is common in production. It gives you reliable message delivery from RabbitMQ while keeping result storage fast and simple in Redis.
broker_url = "amqp://user:password@localhost:5672//"
result_backend = "redis://localhost:6379/1"
task_acks_late = True
worker_prefetch_multiplier = 1worker_prefetch_multiplier controls how many messages a worker reserves at once. A value of one keeps tasks distributed evenly across workers, which matters when task durations vary. This setting applies regardless of which broker you use, but it is especially relevant when you want to avoid one slow worker holding many queued messages.
This mixed setup decouples the failure modes of the two systems. If Redis is temporarily unavailable, workers can still consume and execute tasks from RabbitMQ; only result storage fails. If RabbitMQ is unavailable, new tasks cannot be published, but the result backend remains reachable for reading previously stored results.
Failure Modes and Operational Behavior
Understanding what happens when each component fails helps you choose the right combination.
If the Redis broker loses connectivity, workers stop consuming and the application cannot publish tasks. Because Redis is an in-memory store, messages that were not yet consumed are lost if the Redis process restarts without persistence. Redis persistence options such as RDB snapshots or AOF can reduce this risk, but they do not provide the same delivery guarantees as AMQP.
If RabbitMQ is the broker, messages published to a durable queue survive a broker restart. Workers that crash before acknowledging a message cause redelivery when task_acks_late is enabled. This makes RabbitMQ the stronger choice when task loss is unacceptable.
The result backend fails differently. A task that completes successfully but cannot write its result to Redis will be marked as failed from the caller's perspective, even though the task logic ran. This is why result storage should be monitored separately from the broker. If results are not critical, you can disable the result backend entirely by leaving result_backend unset, and Celery will not store results at all.
Choosing Between Redis and RabbitMQ
The decision depends on what you are optimizing for.
| Criterion | RabbitMQ broker | Redis broker |
|---|---|---|
| Delivery guarantee | Durable queues, explicit acks | Visibility timeout heuristic |
| Message loss on restart | Survives with durable queues | Requires persistence config |
| Priority handling | AMQP queue priorities | Multiple priority lists, bounded |
| Operational complexity | Separate server, vhosts | Simple, often already deployed |
| Typical pairing | With Redis result backend | With Redis result backend |
Use RabbitMQ when tasks must not be lost and when you need priority queues or strict delivery semantics. Use Redis as the broker when you already run Redis, when the deployment must stay simple, and when occasional duplicate execution under a worker crash is acceptable.
The result backend choice is more one-sided. Redis is the standard choice because it is fast, simple to configure, and supports expiration. RabbitMQ can act as a result backend through the rpc:// transport, but that mode stores results in per-worker reply queues and does not persist them after the worker stops. Redis is the more practical result backend for multi-worker deployments.
A practical starting point for a new service is RabbitMQ for the broker and Redis for the result backend, with task_acks_late enabled and result_expires set to match how long callers need results.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.