Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Celery Beat and Periodic Tasks

Set up Celery Beat for periodic tasks in Python: interval and crontab schedules, timezone and duplicate-execution pitfalls, and running the scheduler reliably.

~4 min readPython
#Celery#Task Scheduling#Periodic Tasks#Python
A diagram showing a Celery Beat scheduler sending periodic task messages to a broker, which workers consume.

What Celery Beat Does and How It Fits Into Celery

Celery Beat is the scheduler component that sends periodic task messages to Celery workers. It does not execute tasks itself; instead, it enqueues task messages on the broker at the times you define. Workers then consume those messages and run the tasks. This separation lets you scale workers independently of the scheduler, and Beat only needs to know the schedule and the broker connection.

Configuring the Beat Scheduler

To use Beat, add a beat_schedule entry to your Celery app configuration. This dictionary maps a schedule entry name to a definition containing the task name, schedule, and optional arguments. The schedule can be an interval or a crontab object. You also need to set the timezone so schedules are interpreted correctly.

python
from celery import Celery from celery.schedules import crontab app = Celery('tasks', broker='redis://localhost:6379/0') app.conf.timezone = 'UTC' app.conf.beat_schedule = { 'add-every-30-seconds': { 'task': 'tasks.add', 'schedule': 30.0, 'args': (16, 12) }, 'add-every-monday-morning': { 'task': 'tasks.add', 'schedule': crontab(hour=8, minute=30, day_of_week=1), 'args': (16, 12) } }

The schedule key can be a number of seconds, a timedelta, a celery.schedules.schedule object, or a crontab object. The args and kwargs values are passed to the task when it runs.

Defining Periodic Tasks with Interval Schedules

Interval schedules are the simplest way to run a task at a fixed frequency. You can specify the interval as a number of seconds, a timedelta, or a celery.schedules.schedule object. For example, to run a task every five minutes:

python
from celery.schedules import schedule app.conf.beat_schedule = { 'run-every-5-minutes': { 'task': 'tasks.cleanup', 'schedule': schedule(run_every=300), } }

The schedule class accepts a run_every argument that can be a timedelta or a number of seconds. Using timedelta is more readable for longer intervals:

python
from datetime import timedelta app.conf.beat_schedule = { 'run-every-hour': { 'task': 'tasks.report', 'schedule': timedelta(hours=1), } }

Interval schedules are appropriate when the exact start time is not important, only the frequency.

Using Crontab Schedules for Calendar-Based Tasks

When you need a task to run at a specific time of day, day of week, or month, use crontab. The crontab class accepts the same fields as a Unix cron expression: minute, hour, day_of_month, month_of_year, and day_of_week. You can also use expressions like */15 for every 15 minutes.

python
from celery.schedules import crontab app.conf.beat_schedule = { 'daily-backup': { 'task': 'tasks.backup', 'schedule': crontab(hour=2, minute=0), }, 'weekday-morning': { 'task': 'tasks.send_digest', 'schedule': crontab(hour=7, minute=30, day_of_week='mon-fri'), } }

The day_of_week field accepts 0-6 where 0 is Sunday, or names like mon, tue, etc. You can also use ranges and lists.

Running the Beat Service and Ensuring It Starts Once

Beat runs as a separate process from your workers. You start it with:

bash
celery -A tasks beat

This reads the beat_schedule from your app configuration and begins sending tasks. In production, run Beat as a daemon or in a container with a restart policy. A common mistake is running multiple Beat instances, which can cause duplicate task sends. Use a single instance or a leader election mechanism if you need high availability.

Beat stores the last time each task was run in a scheduler backend. By default, it uses a local file (celerybeat-schedule). The directory must be writable, and the file should be kept between restarts. You can configure the filename explicitly:

python
app.conf.beat_scheduler = 'celery.beat:PersistentScheduler' app.conf.beat_schedule_filename = '/var/run/celery/beat-schedule'

If the file is deleted or stored on ephemeral disk, Beat may lose its last-run times and send duplicate task messages after a restart.

Common Pitfalls: Timezones, Duplicate Execution, and Task Idempotency

Timezone handling is the most common source of confusion. Celery Beat uses the timezone set in app.conf.timezone. If you set timezone='UTC', all crontab expressions are interpreted in UTC. If you want local time, set it to your local timezone, but be aware of daylight saving changes. Always test with a schedule that is easy to observe.

Duplicate execution can happen if Beat is restarted and the scheduler backend loses the last-run timestamp, or if you accidentally run two Beat processes. Make your tasks idempotent so that running them twice has no harmful side effects. For example, use a unique key in a database or check the current state before performing an action.

Another pitfall is forgetting to include the args or kwargs in the schedule definition. If your task requires arguments, they must be provided in the schedule entry.

Production Considerations: Persistent Backend, Monitoring, and Failure Handling

In production, monitor Beat and the workers. If Beat crashes, no new scheduled tasks will be sent. Use a process supervisor like systemd or a container orchestration tool to restart it. You should also monitor the broker for backlog and the workers for task failures.

The scheduler backend should be persistent so Beat can recover its last-run times. If you use Django, django-celery-beat provides a DatabaseScheduler backed by your database and an admin interface:

python
app.conf.beat_scheduler = 'django_celery_beat.schedulers:DatabaseScheduler'

For non-Django projects, choose a scheduler implementation that stores state in a durable location you control. A local file is acceptable if it survives restarts, but a database or other persistent store is easier to manage across hosts. Installing a broker or serializer extra does not automatically make the schedule data persistent.

Finally, consider the impact of long-running tasks on the schedule. Beat sends the task at the scheduled time, but if the worker is busy, the task may wait in the queue. Use appropriate worker concurrency and queue priorities to avoid delays.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.