Python Celery Beat and Periodic Tasks
Set up Celery Beat for periodic tasks in Python: interval and crontab schedules, timezone and duplicate-execution pitfalls, and running the scheduler reliably.

What Celery Beat Does and How It Fits Into Celery
Celery Beat is the scheduler component that sends periodic task messages to Celery workers. It does not execute tasks itself; instead, it enqueues task messages on the broker at the times you define. Workers then consume those messages and run the tasks. This separation lets you scale workers independently of the scheduler, and Beat only needs to know the schedule and the broker connection.
Configuring the Beat Scheduler
To use Beat, add a beat_schedule entry to your Celery app configuration. This dictionary maps a schedule entry name to a definition containing the task name, schedule, and optional arguments. The schedule can be an interval or a crontab object. You also need to set the timezone so schedules are interpreted correctly.
from celery import Celery
from celery.schedules import crontab
app = Celery('tasks', broker='redis://localhost:6379/0')
app.conf.timezone = 'UTC'
app.conf.beat_schedule = {
'add-every-30-seconds': {
'task': 'tasks.add',
'schedule': 30.0,
'args': (16, 12)
},
'add-every-monday-morning': {
'task': 'tasks.add',
'schedule': crontab(hour=8, minute=30, day_of_week=1),
'args': (16, 12)
}
}The schedule key can be a number of seconds, a timedelta, a celery.schedules.schedule object, or a crontab object. The args and kwargs values are passed to the task when it runs.
Defining Periodic Tasks with Interval Schedules
Interval schedules are the simplest way to run a task at a fixed frequency. You can specify the interval as a number of seconds, a timedelta, or a celery.schedules.schedule object. For example, to run a task every five minutes:
from celery.schedules import schedule
app.conf.beat_schedule = {
'run-every-5-minutes': {
'task': 'tasks.cleanup',
'schedule': schedule(run_every=300),
}
}The schedule class accepts a run_every argument that can be a timedelta or a number of seconds. Using timedelta is more readable for longer intervals:
from datetime import timedelta
app.conf.beat_schedule = {
'run-every-hour': {
'task': 'tasks.report',
'schedule': timedelta(hours=1),
}
}Interval schedules are appropriate when the exact start time is not important, only the frequency.
Using Crontab Schedules for Calendar-Based Tasks
When you need a task to run at a specific time of day, day of week, or month, use crontab. The crontab class accepts the same fields as a Unix cron expression: minute, hour, day_of_month, month_of_year, and day_of_week. You can also use expressions like */15 for every 15 minutes.
from celery.schedules import crontab
app.conf.beat_schedule = {
'daily-backup': {
'task': 'tasks.backup',
'schedule': crontab(hour=2, minute=0),
},
'weekday-morning': {
'task': 'tasks.send_digest',
'schedule': crontab(hour=7, minute=30, day_of_week='mon-fri'),
}
}The day_of_week field accepts 0-6 where 0 is Sunday, or names like mon, tue, etc. You can also use ranges and lists.
Running the Beat Service and Ensuring It Starts Once
Beat runs as a separate process from your workers. You start it with:
celery -A tasks beatThis reads the beat_schedule from your app configuration and begins sending tasks. In production, run Beat as a daemon or in a container with a restart policy. A common mistake is running multiple Beat instances, which can cause duplicate task sends. Use a single instance or a leader election mechanism if you need high availability.
Beat stores the last time each task was run in a scheduler backend. By default, it uses a local file (celerybeat-schedule). The directory must be writable, and the file should be kept between restarts. You can configure the filename explicitly:
app.conf.beat_scheduler = 'celery.beat:PersistentScheduler'
app.conf.beat_schedule_filename = '/var/run/celery/beat-schedule'If the file is deleted or stored on ephemeral disk, Beat may lose its last-run times and send duplicate task messages after a restart.
Common Pitfalls: Timezones, Duplicate Execution, and Task Idempotency
Timezone handling is the most common source of confusion. Celery Beat uses the timezone set in app.conf.timezone. If you set timezone='UTC', all crontab expressions are interpreted in UTC. If you want local time, set it to your local timezone, but be aware of daylight saving changes. Always test with a schedule that is easy to observe.
Duplicate execution can happen if Beat is restarted and the scheduler backend loses the last-run timestamp, or if you accidentally run two Beat processes. Make your tasks idempotent so that running them twice has no harmful side effects. For example, use a unique key in a database or check the current state before performing an action.
Another pitfall is forgetting to include the args or kwargs in the schedule definition. If your task requires arguments, they must be provided in the schedule entry.
Production Considerations: Persistent Backend, Monitoring, and Failure Handling
In production, monitor Beat and the workers. If Beat crashes, no new scheduled tasks will be sent. Use a process supervisor like systemd or a container orchestration tool to restart it. You should also monitor the broker for backlog and the workers for task failures.
The scheduler backend should be persistent so Beat can recover its last-run times. If you use Django, django-celery-beat provides a DatabaseScheduler backed by your database and an admin interface:
app.conf.beat_scheduler = 'django_celery_beat.schedulers:DatabaseScheduler'For non-Django projects, choose a scheduler implementation that stores state in a durable location you control. A local file is acceptable if it survives restarts, but a database or other persistent store is easier to manage across hosts. Installing a broker or serializer extra does not automatically make the schedule data persistent.
Finally, consider the impact of long-running tasks on the schedule. Beat sends the task at the scheduled time, but if the worker is busy, the task may wait in the queue. Use appropriate worker concurrency and queue priorities to avoid delays.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.