Python Celery Rate Limits: Syntax and Worker Behavior

Python Celery rate limits control how many times a task can run within a given time window. The limit is declared on the task itself, either in the task decorator or through task annotations, and the worker enforces it as tasks are consumed from the broker. Rate limits are useful for tasks that call external APIs with quota limits, write to rate-limited services, or must not flood a shared resource.

How Celery Rate Limits Are Enforced

The rate limit is not applied by the broker. The broker simply delivers messages to workers. Each worker instance maintains its own rate limiter and applies the limit independently. This means the effective throughput for a task is the per-worker rate multiplied by the number of worker instances running that task.

Celery implements rate limiting with a token bucket. The bucket refills at a fixed rate, and each task execution consumes a token. When the bucket is empty, the worker holds the task until a token becomes available. The task stays in the worker's memory during this wait; it is not returned to the broker.

Setting a Rate Limit on a Task

The simplest way to set a rate limit is the rate_limit argument in the task decorator:

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task(rate_limit="10/m")
def send_notification(user_id):
    # ...
    pass

Here 10/m means the worker will start at most ten executions of send_notification per minute. The rate limit applies to that task only. Other tasks in the same worker are unaffected.

You can also set the rate limit after the task is defined:

send_notification.rate_limit = "30/m"

Changing the attribute affects the task definition in the current process. For a change to apply across all workers, it must be part of the task definition or configuration that every worker loads.

Rate Limit String Syntax

The rate limit string normally consists of a number and a time unit.

The time unit is a single letter: s for seconds, m for minutes, h for hours, d for days. The number is an integer; a positive integer sets the allowed executions per unit, while 0 and None disable the limit.

Runtime Configuration with task_annotations

Rate limits can also be applied globally through task_annotations in the Celery configuration. This keeps rate limits out of the task code and lets you change them without editing the task module:

app.conf.task_annotations = {
    "send_notification": {"rate_limit": "30/m"},
    "tasks.export_report": {"rate_limit": "2/m"},
}

Annotations can match by task name, and the values are merged into the task options at worker startup. This is useful when the same task is shared across projects and each deployment needs a different limit.

Why Rate Limits Are Per Worker Instance

The most common misunderstanding is that a rate limit of 10/m means ten executions per minute across the entire cluster. It does not. Each worker instance enforces its own limit.

If you run two celery worker instances for the same task with rate_limit="10/m", the cluster can execute up to 20 times per minute, because each instance maintains its own token bucket. The same applies when you run multiple worker instances on different machines. A single worker with --concurrency=4 is still one worker instance; --concurrency controls how many child processes execute tasks, not how many independent rate-limit buckets exist.

This behavior matters when the limit exists to protect an external service. If the external API allows 100 requests per minute, a single worker with rate_limit="100/m" is safe. Scaling to two worker instances with the same limit doubles the request rate to 200 per minute, which may exceed the quota.

To enforce a cluster-wide limit, you need an external mechanism such as a shared token bucket in Redis or a distributed rate limiter. Celery's built-in rate limit does not coordinate across workers.

Edge Cases and Common Mistakes

Rate limits interact with task retries. When a task is retried, the retry is a separate execution and consumes a token. A task that fails repeatedly can exhaust the bucket and delay all other executions of that task.

Long-running tasks also affect the observed rate. The rate limit controls how many tasks the worker starts per time window, not how long each task takes. If tasks take longer than the interval between allowed starts, the actual throughput will be lower than the configured rate.

The rate limit applies per task, not per queue. Two different tasks that both call the same external API each have their own limit. If you need a combined limit across multiple tasks, you need a shared limiter outside Celery.

Observing Rate Limit Behavior in Production

Depending on your logging configuration and Celery version, the worker may log when a task is rate limited. Monitoring the queue depth and the worker's active and reserved task counts helps distinguish rate limiting from a slow consumer.

If a rate-limited task is also part of a chain or group, the downstream tasks wait until the limited task completes. This can create backpressure that appears as an idle worker even though the queue is full. Inspecting the worker's reserved tasks shows whether tasks are held by the rate limiter.