Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Implementing Streaming and Async in Python LangChain

Learn how to combine streaming and async in Python LangChain to build responsive LLM applications. Includes token streaming, async calls, error handling, and concurrency.

~4 min readPython
#LangChain#Python#Async#Streaming
Diagram showing streaming and async flow in a Python LangChain application

To combine streaming and async in LangChain, use the model's astream() method. It returns an async iterator that yields response chunks as they become available, so you can push tokens to a client while the event loop continues serving other work. This article covers the synchronous and asynchronous streaming methods, async single calls, and practical concerns like error handling, cancellation, and concurrency.

Understanding Streaming and Async in LangChain

Streaming and async solve different problems. Streaming delivers partial results as they become available, so the user sees tokens appear in real time. Async allows multiple operations to overlap without blocking the event loop, which is essential when you are making many LLM calls or waiting on external services.

In LangChain, streaming and async are independent but often used together. A model can support streaming, async, or both. The API exposes separate methods for each combination: stream for synchronous streaming, astream for asynchronous streaming, ainvoke for asynchronous single calls, and invoke for synchronous calls. The exact method names have changed across versions, but the pattern remains consistent.

Streaming Tokens from an LLM

To stream tokens synchronously, call stream on a model and iterate over the result. Each chunk contains a piece of the output, typically a token or a small group of tokens.

python
from langchain_openai import ChatOpenAI # Assumes OPENAI_API_KEY is set in the environment model = ChatOpenAI(model="gpt-4o-mini") for chunk in model.stream("Write a short poem about async programming"): print(chunk.content, end="", flush=True)

The stream method returns a generator that yields chunks as they arrive. This works well in a script or a synchronous web framework where you can send each chunk to the client immediately.

Making Async Calls to LangChain Models

For a single async call, use ainvoke. This returns a full response once the model finishes, but it does not block the event loop while waiting.

python
import asyncio from langchain_openai import ChatOpenAI async def main(): model = ChatOpenAI(model="gpt-4o-mini") response = await model.ainvoke("Explain async/await in Python") print(response.content) asyncio.run(main())

ainvoke is the preferred method in recent LangChain versions. Older code may use arun, but that method is deprecated and should be replaced with ainvoke where possible.

Combining Streaming and Async

When you need both streaming and non-blocking behavior, use astream. It returns an async iterator, which you consume with async for. The loop yields chunks as they are produced, while the event loop remains free to handle other tasks.

python
import asyncio from langchain_openai import ChatOpenAI async def main(): model = ChatOpenAI(model="gpt-4o-mini") async for chunk in model.astream("Write a short story about a robot learning to code"): print(chunk.content, end="", flush=True) asyncio.run(main())

This pattern is ideal for web applications that serve responses over WebSocket or Server-Sent Events. The server can push each chunk to the client as soon as it arrives, without holding a thread or blocking other requests.

Handling Errors and Cancellation

Async streaming introduces two failure modes: errors from the model and cancellation by the client. Both need explicit handling.

Provider errors can surface as exceptions from the model SDK, such as OpenAI's rate-limit or network errors, or as LangChain-specific exceptions. Catch the exceptions you expect from your provider, or catch a broad Exception in the stream loop so you can stop the stream cleanly. Because asyncio.CancelledError inherits from BaseException, use a separate except asyncio.CancelledError block if you need to handle cancellation.

python
import asyncio from langchain_openai import ChatOpenAI async def stream_with_error_handling(): model = ChatOpenAI(model="gpt-4o-mini") try: async for chunk in model.astream("Generate a long response"): print(chunk.content, end="", flush=True) except Exception as exc: # Log provider failures, timeouts, or rate limits print(f"\nStream failed: {exc}") except asyncio.CancelledError: # Client disconnected or task was cancelled print("\nStream cancelled") raise

When a client disconnects, the async stream may be cancelled with CancelledError. Clean up any resources, such as closing a database connection or releasing a semaphore, but re-raise the exception so cancellation propagates correctly.

Performance and Concurrency Considerations

Using asyncio.gather lets you run multiple async LLM calls concurrently. This is useful when you need to summarize several documents or generate multiple suggestions in parallel.

python
import asyncio from langchain_openai import ChatOpenAI async def generate_summary(model, text): response = await model.ainvoke(f"Summarize: {text}") return response.content async def main(): model = ChatOpenAI(model="gpt-4o-mini") texts = ["First document...", "Second document...", "Third document..."] results = await asyncio.gather(*(generate_summary(model, t) for t in texts)) print(results) asyncio.run(main())

Be aware of rate limits. Running many concurrent calls may hit provider limits. Wrap the per-call work in a function that acquires an asyncio.Semaphore if you need to cap simultaneous requests. The exact limit depends on your provider and plan, so tune it based on your usage.

Streaming and async also affect resource usage. Streaming can reduce memory pressure when you process chunks incrementally instead of accumulating the entire response. Async reduces thread usage by avoiding a blocked thread per request, which is important in high-concurrency web servers.

Choosing Between Streaming and Async

Streaming and async are not mutually exclusive, but you may need to choose one depending on your use case.

Use synchronous streaming (stream) when you are in a simple script or a synchronous framework like Flask and you only need to show progress to the user. Use async streaming (astream) when you are in an async framework like FastAPI or when you need to handle multiple clients concurrently. Use ainvoke when you only need the final result and want to avoid streaming overhead, but still want non-blocking behavior.

For a web endpoint that returns a complete JSON response, ainvoke is sufficient. For an endpoint that streams tokens to a chat interface, astream is the right choice. The decision depends on whether the client expects incremental updates or a single final payload.

Cancellation and Resource Cleanup in Async Streaming

Cleanup code inside an asynchronous stream consumer should go in a finally block so it runs on both normal completion and cancellation. If you exit a stream loop early, call aclose() on the async iterator when the object supports it, especially if you hold file handles, database connections, or rate-limit slots.

python
async def consume_stream(): model = ChatOpenAI(model="gpt-4o-mini") stream = model.astream("Long text") try: async for chunk in stream: yield chunk.content finally: await stream.aclose()

In a FastAPI endpoint that streams responses, the framework may cancel the generator when the client disconnects. The finally block runs, allowing you to log the cancellation or clean up state. This prevents resource leaks in long-running applications.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.