Python FastAPI Background Tasks: Built-in vs Celery
Compare FastAPI's built-in BackgroundTasks with Celery for background work. See code examples, execution-model details, retry behavior, and production tradeoffs.

When a FastAPI request handler needs to perform work that does not need to block the response—such as sending an email, updating a cache, or processing a file—the work can be moved to a background task. Python FastAPI background tasks provide two main approaches: the built-in BackgroundTasks class from Starlette, and a full task queue like Celery. This article explains both, when each makes sense, and how to handle common production concerns.
Using the Built-in BackgroundTasks Class
FastAPI includes BackgroundTasks directly from Starlette. It lets you register callable functions that run after the response is sent. The simplest usage is to declare a parameter of type BackgroundTasks in your endpoint and add a task with add_task.
from fastapi import FastAPI, BackgroundTasks
app = FastAPI()
def write_log(message: str):
with open("log.txt", "a") as f:
f.write(f"{message}\n")
@app.post("/send-notification")
async def send_notification(background_tasks: BackgroundTasks):
background_tasks.add_task(write_log, "notification sent")
return {"message": "Notification is being processed"}Here, write_log runs after the response is sent to the client. The function is executed in the same process, so it is not handled by a separate worker or queue. The response is sent before the task starts, but the task still occupies application capacity until it finishes.
For synchronous functions, FastAPI runs them in a thread pool so they do not block the event loop. For async functions, they run directly on the event loop. This distinction matters when your task performs blocking I/O, such as file writes or network calls. If you have a blocking operation and your task is defined as async, it will block the event loop. Prefer defining background tasks as regular def functions unless they are truly non-blocking.
What Happens When the Response Is Sent
The execution model of BackgroundTasks is tied to the response lifecycle. When an endpoint returns a response, FastAPI sends it to the client, then runs the registered background tasks. If the client disconnects before the response is fully sent, the background tasks may still run. However, if the server process crashes or is restarted before the tasks complete, they are lost.
This behavior makes BackgroundTasks suitable for short, non-critical operations like writing a log entry, sending a simple email, or invalidating a cache. It is not suitable for tasks that must survive a crash, require retries, or take a long time to complete. The task runs in the same process, so a memory leak or a long-running task can degrade the health of the entire application.
Adding Multiple Tasks and Passing Arguments
You can add multiple tasks to the same BackgroundTasks object. They run sequentially in the order they were added. Each task can receive positional and keyword arguments.
def send_email(email: str, subject: str):
# send email logic
pass
def update_analytics(user_id: int):
# update analytics
pass
@app.post("/register")
async def register(user_id: int, background_tasks: BackgroundTasks):
background_tasks.add_task(send_email, "user@example.com", "Welcome")
background_tasks.add_task(update_analytics, user_id)
return {"message": "User registered"}There is no built-in retry mechanism. Each task should handle expected errors internally. If a task raises an exception, the remaining tasks registered for that request are skipped. The error is not sent to the client, so catch expected failures inside the task and implement retries or compensation logic yourself if needed.
When the Built-in Approach Is Not Enough
For tasks that are long-running, require durable execution, need to be distributed across multiple workers, or must survive application restarts, BackgroundTasks is insufficient. Consider a video transcoding job that takes several minutes, or a nightly report generation that must be retried on failure. These scenarios require a dedicated task queue.
Celery is the most common choice for Python applications. It runs tasks in separate worker processes, communicates via a message broker (like Redis or RabbitMQ), and provides features such as retries, scheduling, and result storage. FastAPI integrates with Celery by simply calling Celery tasks from within endpoints; the HTTP response is returned immediately, and the worker handles the task asynchronously.
Setting Up Celery with FastAPI
To use Celery, you need a broker and a worker. The broker stores the task messages; the worker consumes them. Here is a minimal setup.
# tasks.py
from celery import Celery
celery_app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/0")
@celery_app.task
def process_file(file_path: str):
# long-running processing
return f"Processed {file_path}"In your FastAPI application, you import the Celery task and call it with .delay().
from fastapi import FastAPI
from tasks import process_file
app = FastAPI()
@app.post("/process")
async def process(file_path: str):
task = process_file.delay(file_path)
return {"task_id": task.id, "status": "queued"}The endpoint returns immediately with a task ID. The worker picks up the task and runs it independently. This decouples the HTTP request from the actual work, allowing the web server to handle more requests without being blocked by long operations.
Celery tasks can be monitored via the backend, which stores task results. You can query the task status from another endpoint or a separate service.
Comparing BackgroundTasks and Celery
| Aspect | BackgroundTasks | Celery |
|---|---|---|
| Execution location | Same process as the web server | Separate worker processes |
| Durability | Lost on crash or restart | Tasks remain queued while workers restart; broker is still a dependency |
| Retry mechanism | None built-in | Built-in retries with configurable policy |
| Task scheduling | Not supported | Supported via celery beat |
| Scalability | Limited by server resources | Can scale workers independently |
| Best for | Short, non-critical tasks | Long-running, critical, or distributed tasks |
Use BackgroundTasks when the task is quick, failure is acceptable, and you want minimal infrastructure. Use Celery when you need reliability, retries, or the ability to process tasks outside the web process.
Error Handling and Retries in Celery
Celery provides a robust retry mechanism. You can configure max_retries, retry_backoff, and retry_backoff_max on the task. Here is an example that retries up to three times with exponential backoff.
@celery_app.task(bind=True, max_retries=3, retry_backoff=True)
def send_email_task(self, email: str):
try:
# send email
pass
except Exception as exc:
raise self.retry(exc=exc)The bind=True argument gives the task access to self, which is required to call self.retry(). This pattern is useful for transient failures like network timeouts.
For BackgroundTasks, you would need to implement your own try/except and retry logic, which is error-prone and couples the task to the request lifecycle.
Production Considerations
Monitor how long each BackgroundTasks function takes. A synchronous task that hangs can exhaust the thread pool; an async task that blocks can stall the event loop. Move long-running work to Celery.
With Celery, monitor the queue length and worker utilization. A growing queue indicates that workers cannot keep up with the incoming tasks. Scale workers horizontally or adjust concurrency settings. Also ensure that the broker is highly available; if Redis or RabbitMQ goes down, tasks are not accepted.
Both approaches require careful attention to error logging. BackgroundTasks errors are not exposed to the client, so log them inside the task if you need visibility. Celery tasks can be tracked by task ID, and results can be inspected. Use structured logging to correlate task IDs with request IDs for easier debugging.
Choosing the Right Approach for Your Use Case
The decision between BackgroundTasks and Celery comes down to the nature of the work. If the task is a fire-and-forget operation that must complete within a few seconds and losing it occasionally is acceptable, the built-in class is sufficient. If the task is long-running, must be retried on failure, or needs to be processed outside the web server's lifecycle, Celery is the appropriate tool.
For tasks that are short but require retries, you could also consider a lightweight queue like RQ or Dramatiq. The key is to avoid using BackgroundTasks for anything that is critical to your business logic, because it does not provide durability or retries. Instead, reserve it for auxiliary operations like logging, cache invalidation, or sending non-critical notifications.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.