Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python FastAPI Background Tasks: Built-in vs Celery

Compare FastAPI's built-in BackgroundTasks with Celery for background work. See code examples, execution-model details, retry behavior, and production tradeoffs.

~6 min readPython
#FastAPI#Background Tasks#Celery#Async
Illustration of a FastAPI request triggering a background task that runs asynchronously after the response.

When a FastAPI request handler needs to perform work that does not need to block the response—such as sending an email, updating a cache, or processing a file—the work can be moved to a background task. Python FastAPI background tasks provide two main approaches: the built-in BackgroundTasks class from Starlette, and a full task queue like Celery. This article explains both, when each makes sense, and how to handle common production concerns.

Using the Built-in BackgroundTasks Class

FastAPI includes BackgroundTasks directly from Starlette. It lets you register callable functions that run after the response is sent. The simplest usage is to declare a parameter of type BackgroundTasks in your endpoint and add a task with add_task.

python
from fastapi import FastAPI, BackgroundTasks app = FastAPI() def write_log(message: str): with open("log.txt", "a") as f: f.write(f"{message}\n") @app.post("/send-notification") async def send_notification(background_tasks: BackgroundTasks): background_tasks.add_task(write_log, "notification sent") return {"message": "Notification is being processed"}

Here, write_log runs after the response is sent to the client. The function is executed in the same process, so it is not handled by a separate worker or queue. The response is sent before the task starts, but the task still occupies application capacity until it finishes.

For synchronous functions, FastAPI runs them in a thread pool so they do not block the event loop. For async functions, they run directly on the event loop. This distinction matters when your task performs blocking I/O, such as file writes or network calls. If you have a blocking operation and your task is defined as async, it will block the event loop. Prefer defining background tasks as regular def functions unless they are truly non-blocking.

What Happens When the Response Is Sent

The execution model of BackgroundTasks is tied to the response lifecycle. When an endpoint returns a response, FastAPI sends it to the client, then runs the registered background tasks. If the client disconnects before the response is fully sent, the background tasks may still run. However, if the server process crashes or is restarted before the tasks complete, they are lost.

This behavior makes BackgroundTasks suitable for short, non-critical operations like writing a log entry, sending a simple email, or invalidating a cache. It is not suitable for tasks that must survive a crash, require retries, or take a long time to complete. The task runs in the same process, so a memory leak or a long-running task can degrade the health of the entire application.

Adding Multiple Tasks and Passing Arguments

You can add multiple tasks to the same BackgroundTasks object. They run sequentially in the order they were added. Each task can receive positional and keyword arguments.

python
def send_email(email: str, subject: str): # send email logic pass def update_analytics(user_id: int): # update analytics pass @app.post("/register") async def register(user_id: int, background_tasks: BackgroundTasks): background_tasks.add_task(send_email, "user@example.com", "Welcome") background_tasks.add_task(update_analytics, user_id) return {"message": "User registered"}

There is no built-in retry mechanism. Each task should handle expected errors internally. If a task raises an exception, the remaining tasks registered for that request are skipped. The error is not sent to the client, so catch expected failures inside the task and implement retries or compensation logic yourself if needed.

When the Built-in Approach Is Not Enough

For tasks that are long-running, require durable execution, need to be distributed across multiple workers, or must survive application restarts, BackgroundTasks is insufficient. Consider a video transcoding job that takes several minutes, or a nightly report generation that must be retried on failure. These scenarios require a dedicated task queue.

Celery is the most common choice for Python applications. It runs tasks in separate worker processes, communicates via a message broker (like Redis or RabbitMQ), and provides features such as retries, scheduling, and result storage. FastAPI integrates with Celery by simply calling Celery tasks from within endpoints; the HTTP response is returned immediately, and the worker handles the task asynchronously.

Setting Up Celery with FastAPI

To use Celery, you need a broker and a worker. The broker stores the task messages; the worker consumes them. Here is a minimal setup.

python
# tasks.py from celery import Celery celery_app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/0") @celery_app.task def process_file(file_path: str): # long-running processing return f"Processed {file_path}"

In your FastAPI application, you import the Celery task and call it with .delay().

python
from fastapi import FastAPI from tasks import process_file app = FastAPI() @app.post("/process") async def process(file_path: str): task = process_file.delay(file_path) return {"task_id": task.id, "status": "queued"}

The endpoint returns immediately with a task ID. The worker picks up the task and runs it independently. This decouples the HTTP request from the actual work, allowing the web server to handle more requests without being blocked by long operations.

Celery tasks can be monitored via the backend, which stores task results. You can query the task status from another endpoint or a separate service.

Comparing BackgroundTasks and Celery

AspectBackgroundTasksCelery
Execution locationSame process as the web serverSeparate worker processes
DurabilityLost on crash or restartTasks remain queued while workers restart; broker is still a dependency
Retry mechanismNone built-inBuilt-in retries with configurable policy
Task schedulingNot supportedSupported via celery beat
ScalabilityLimited by server resourcesCan scale workers independently
Best forShort, non-critical tasksLong-running, critical, or distributed tasks

Use BackgroundTasks when the task is quick, failure is acceptable, and you want minimal infrastructure. Use Celery when you need reliability, retries, or the ability to process tasks outside the web process.

Error Handling and Retries in Celery

Celery provides a robust retry mechanism. You can configure max_retries, retry_backoff, and retry_backoff_max on the task. Here is an example that retries up to three times with exponential backoff.

python
@celery_app.task(bind=True, max_retries=3, retry_backoff=True) def send_email_task(self, email: str): try: # send email pass except Exception as exc: raise self.retry(exc=exc)

The bind=True argument gives the task access to self, which is required to call self.retry(). This pattern is useful for transient failures like network timeouts.

For BackgroundTasks, you would need to implement your own try/except and retry logic, which is error-prone and couples the task to the request lifecycle.

Production Considerations

Monitor how long each BackgroundTasks function takes. A synchronous task that hangs can exhaust the thread pool; an async task that blocks can stall the event loop. Move long-running work to Celery.

With Celery, monitor the queue length and worker utilization. A growing queue indicates that workers cannot keep up with the incoming tasks. Scale workers horizontally or adjust concurrency settings. Also ensure that the broker is highly available; if Redis or RabbitMQ goes down, tasks are not accepted.

Both approaches require careful attention to error logging. BackgroundTasks errors are not exposed to the client, so log them inside the task if you need visibility. Celery tasks can be tracked by task ID, and results can be inspected. Use structured logging to correlate task IDs with request IDs for easier debugging.

Choosing the Right Approach for Your Use Case

The decision between BackgroundTasks and Celery comes down to the nature of the work. If the task is a fire-and-forget operation that must complete within a few seconds and losing it occasionally is acceptable, the built-in class is sufficient. If the task is long-running, must be retried on failure, or needs to be processed outside the web server's lifecycle, Celery is the appropriate tool.

For tasks that are short but require retries, you could also consider a lightweight queue like RQ or Dramatiq. The key is to avoid using BackgroundTasks for anything that is critical to your business logic, because it does not provide durability or retries. Instead, reserve it for auxiliary operations like logging, cache invalidation, or sending non-critical notifications.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.