Python asyncio.wait: Managing Concurrent Tasks

When you need to run several coroutines concurrently and want control over completion conditions, asyncio.wait gives you more control than asyncio.gather. It returns two sets of tasks—done and pending—and lets you specify a completion condition with return_when. That makes it useful for partial results, timeouts, and deliberate exception handling.

What asyncio.wait Does and When to Use It

The asyncio.wait coroutine takes an iterable of awaitables (normally Task objects) and waits until a condition defined by return_when is met. Unlike asyncio.gather, which waits for all tasks and then returns their results in input order, asyncio.wait returns two sets: done and pending. You can inspect these sets to see which tasks finished and which are still running.

import asyncio

async def worker(name, delay):
    await asyncio.sleep(delay)
    return f"{name} finished"

async def main():
    tasks = [
        asyncio.create_task(worker("A", 2)),
        asyncio.create_task(worker("B", 1)),
        asyncio.create_task(worker("C", 3)),
    ]
    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    print("Done:", [t.result() for t in done])
    print("Pending count:", len(pending))
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)

asyncio.run(main())

In this example, asyncio.wait returns as soon as the first task (B) completes. The pending set still contains the other two tasks, which you can continue to await or cancel. To keep processing results as tasks finish, pass the pending set back into another asyncio.wait call, often with return_when=asyncio.FIRST_COMPLETED.

Understanding return_when

The return_when parameter accepts three constants from the asyncio module:

FIRST_EXCEPTION is useful for fail-fast workflows. If you have independent requests and one fails, you can stop waiting and handle the error immediately. If you do not want the remaining tasks to keep running after that, cancel them from the pending set.

With ALL_COMPLETED, asyncio.wait waits for all tasks, like gather, but it still returns done and pending sets, which can help with cleanup.

Working with the Done and Pending Sets

After asyncio.wait returns, you can iterate over done to retrieve results or exceptions. The pending set contains tasks that have not finished yet. You can await them later, cancel them, or ignore them. If you ignore them, they keep running; when the loop shuts down, they may be cancelled abruptly, and their results or exceptions may never be retrieved. Explicitly cancel pending tasks when you no longer need them.

The done and pending sets are sets, so their iteration order is not guaranteed.

async def main():
    tasks = [asyncio.create_task(worker(f"Task {i}", i)) for i in range(1, 4)]
    done, pending = await asyncio.wait(tasks, timeout=2.0)
    for task in done:
        try:
            print(task.result())
        except Exception as exc:
            print(f"Task raised: {exc}")
    # Cancel remaining tasks to avoid background execution
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)

This pattern is common when you want to enforce a deadline. If a task does not finish within the timeout, it remains in pending, and you can cancel it explicitly. Without cancellation, the event loop may complain about unretrieved exceptions or unfinished tasks.

Handling Exceptions and Cancellation

Tasks in the done set may have completed normally, raised an exception, or been cancelled. To check which happened, use task.cancelled() and task.exception(). Calling task.result() on a task that raised an exception re-raises it, so wrap it in a try/except or call task.exception() first. Calling task.result() on a cancelled task raises CancelledError, so check task.cancelled() first.

async def failing_task():
    raise ValueError("boom")

async def main():
    task = asyncio.create_task(failing_task())
    done, _ = await asyncio.wait({task})
    if task.cancelled():
        print("Task was cancelled")
    elif task.exception() is not None:
        print(f"Task failed: {task.exception()}")
    else:
        print(task.result())

Cancellation is a separate state. If you cancel a task before it completes, it appears in done with cancelled() returning True. When you use asyncio.wait with FIRST_COMPLETED, a cancelled task can trigger the return, so check for cancellation explicitly rather than assuming every task in done has a result.

asyncio.wait vs asyncio.gather

asyncio.gather is the simpler choice when you want all results in the original order and are prepared to wait for everything. By default it raises the first exception produced by any task, and it returns a list of results. asyncio.wait is better when you need partial results, timeouts, or the ability to act on tasks as they finish.

Use gather when you need all results and can tolerate waiting for the slowest task. Use asyncio.wait when you want to implement a timeout, process results incrementally, or inspect failures without letting one exception abort the whole operation.

Timeouts and Partial Completion

The timeout parameter is a number of seconds. When the timeout expires, asyncio.wait returns whatever tasks have completed so far and leaves the rest in pending. It does not raise TimeoutError and it does not cancel the pending tasks; they continue to run unless you cancel them.

async def main():
    tasks = [asyncio.create_task(worker(f"Task {i}", i)) for i in range(1, 5)]
    done, pending = await asyncio.wait(tasks, timeout=2.5)
    print(f"Completed {len(done)} tasks, {len(pending)} still pending")
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)

This pattern gives you a soft deadline: collect what finished within the time limit, then decide whether to cancel the rest or let them run. For a hard deadline, prefer applying asyncio.wait_for to individual tasks so the timeout cancels the work you still care about. If you instead wrap the asyncio.wait call in asyncio.wait_for, the timeout cancels the wait operation but not the tasks in its sets; you still need to cancel anything left in pending.

Common Pitfalls and Runtime Behavior

One common mistake is passing bare coroutine objects to asyncio.wait instead of Task objects. In older Python releases, asyncio.wait scheduled coroutine objects as tasks internally, but in Python 3.11 and later that pattern is deprecated. When you are working with coroutines, create Task objects explicitly with asyncio.create_task so you can cancel and inspect them before and after the wait.

Another issue is forgetting to consume exceptions from tasks in the done set. If a task raises an exception and you never call task.result() or task.exception(), the event loop logs an "exception was never retrieved" warning. Always inspect the outcome of each task in done.

Finally, asyncio.wait is itself a coroutine, so it must be awaited from inside another async def coroutine; it is not a synchronous function. asyncio.create_task was added in Python 3.7; for Python 3.6 and earlier, use asyncio.ensure_future instead.

When you need to coordinate multiple concurrent operations and want fine-grained control over completion, timeouts, and partial results, asyncio.wait is a useful tool. Its done and pending sets show what has finished and what is still running, and return_when lets you choose the condition that ends the wait.