Python Async Generators: Syntax and Real-World Use

When you need to produce a sequence of values that arrive asynchronously—from a network, a database cursor, or a long-running computation—a regular generator cannot await anything. A Python async generator fills that gap: it is a function defined with async def that contains yield, so it can suspend not only on yield but also on await expressions. This makes it the natural tool for streaming data in asyncio-based applications.

The Core Syntax of an Async Generator

An async generator is defined like a normal generator: the function is declared with async def and the body contains at least one yield. The presence of yield makes it an async generator, not a coroutine.

import asyncio

async def countdown(n: int):
    while n > 0:
        await asyncio.sleep(1)
        yield n
        n -= 1

async def main():
    async for number in countdown(3):
        print(number)

asyncio.run(main())

Calling countdown(3) does not execute the body. It returns an asynchronous generator object, which implements the asynchronous iterator protocol. The function starts running only when you begin iterating with async for or by calling __anext__() directly.

Each time the generator reaches a yield, it suspends and returns a value. The next iteration resumes execution from that point, allowing await expressions to run between yields. This is the key difference from a regular generator, which cannot await inside its body.

How Async Iteration Works

An async generator is an async iterator, meaning it implements __anext__ and __aiter__. The async for loop calls __anext__ repeatedly and awaits each result. The loop terminates when __anext__ raises StopAsyncIteration.

Inside an async function, you can manually advance the generator like this:

async def advance_manually():
    ag = countdown(2)
    print(await ag.__anext__())  # 2
    print(await ag.__anext__())  # 1

You rarely call __anext__ directly, but understanding the protocol helps when you need to manually advance a stream or integrate with code that expects an async iterator.

If you stop iterating early and need deterministic cleanup, call await ag.aclose() from a finally block rather than relying only on garbage collection. This is important for releasing resources like open sockets or file handles.

Practical Use Cases for Async Generators

Async generators shine when data arrives incrementally and you want to process it without buffering the entire result in memory. Common scenarios include:

• Streaming lines from a large file read asynchronously.
• Paginating through an API where each page requires an HTTP request.
• Reading from a message queue or WebSocket.
• Producing chunks of data from a slow computation.

Here is a pattern for paginating a REST API:

async def fetch_pages(client, url):
    page = 1
    while True:
        data = await client.get(f"{url}?page={page}")
        if not data:
            return
        yield data
        page += 1

The consumer can iterate over pages with async for and stop early if needed, without downloading every page. The generator holds no extra state beyond its local variables, and the caller controls when the next page is fetched.

Handling Exceptions and Cleanup

Like a regular generator, an async generator can wrap its body in try/finally to ensure cleanup runs when the generator is closed, whether by normal exhaustion, an explicit aclose() call, or an exception. The async version can also await inside that cleanup.

async def open_stream():
    try:
        async for chunk in network_source():
            yield chunk
    finally:
        await close_connection()

The finally block runs when the generator is closed or runs to completion. If you need deterministic cleanup after an early exit, call await ag.aclose() from the consumer.

If an exception is raised inside the generator body, it propagates to the consumer. The consumer can catch it in the async for loop. The generator is then closed, and the finally block executes.

Performance and Resource Considerations

Async generators are not faster than regular generators; they are a different tool for a different problem. Their value is that they avoid blocking the event loop while waiting for I/O. When you await inside an async generator, the event loop can run other tasks. This is the core performance benefit.

Memory usage is also a key advantage. Because values are produced lazily, you never need to hold the full sequence in memory. For example, streaming a 10 GB file line by line uses only enough memory for one line, provided the file is read asynchronously in chunks.

There is a subtle backpressure consideration. The consumer controls the pace: the generator does not run ahead unless the consumer requests the next value. This is natural flow control. However, if you wrap an async generator in a task and buffer values manually, you can consume memory quickly. Let the consumer drive the iteration unless you have a specific reason to prefetch.

When to Use an Async Generator vs. Other Patterns

A regular generator is sufficient when you do not need to await inside the producer. If your data is already in memory or can be produced synchronously, a regular generator is simpler and faster.

An async iterator class is an alternative when you need to maintain state across iterations in a more structured way. Async generators are usually more concise, but a class gives you explicit control over __anext__ and __aiter__, which can be useful for complex state machines.

A list comprehension or list() call is appropriate only when the data set is small and you need random access. For streaming, an async generator is almost always the better choice because it avoids buffering the entire collection.

Here is a quick decision guide:

One common mistake is to write an async generator and then collect all values with an async list comprehension such as [value async for value in ag]. That defeats the lazy streaming benefit and can exhaust memory. If you need all values, consider whether you actually need them at once.

Async generators integrate cleanly with asyncio code. You can pass them to functions that accept async iterables, consume them with async for, or chain them by iterating over one inside another. They are a core building block for writing non-blocking, incremental data processing in Python.