Implementing Streaming and Async in Python LangChain
Learn how to combine streaming and async in Python LangChain to build responsive LLM applications. Includes token streaming, async calls, error handling, and concurrency.

To combine streaming and async in LangChain, use the model's astream() method. It returns an async iterator that yields response chunks as they become available, so you can push tokens to a client while the event loop continues serving other work. This article covers the synchronous and asynchronous streaming methods, async single calls, and practical concerns like error handling, cancellation, and concurrency.
Understanding Streaming and Async in LangChain
Streaming and async solve different problems. Streaming delivers partial results as they become available, so the user sees tokens appear in real time. Async allows multiple operations to overlap without blocking the event loop, which is essential when you are making many LLM calls or waiting on external services.
In LangChain, streaming and async are independent but often used together. A model can support streaming, async, or both. The API exposes separate methods for each combination: stream for synchronous streaming, astream for asynchronous streaming, ainvoke for asynchronous single calls, and invoke for synchronous calls. The exact method names have changed across versions, but the pattern remains consistent.
Streaming Tokens from an LLM
To stream tokens synchronously, call stream on a model and iterate over the result. Each chunk contains a piece of the output, typically a token or a small group of tokens.
from langchain_openai import ChatOpenAI
# Assumes OPENAI_API_KEY is set in the environment
model = ChatOpenAI(model="gpt-4o-mini")
for chunk in model.stream("Write a short poem about async programming"):
print(chunk.content, end="", flush=True)The stream method returns a generator that yields chunks as they arrive. This works well in a script or a synchronous web framework where you can send each chunk to the client immediately.
Making Async Calls to LangChain Models
For a single async call, use ainvoke. This returns a full response once the model finishes, but it does not block the event loop while waiting.
import asyncio
from langchain_openai import ChatOpenAI
async def main():
model = ChatOpenAI(model="gpt-4o-mini")
response = await model.ainvoke("Explain async/await in Python")
print(response.content)
asyncio.run(main())ainvoke is the preferred method in recent LangChain versions. Older code may use arun, but that method is deprecated and should be replaced with ainvoke where possible.
Combining Streaming and Async
When you need both streaming and non-blocking behavior, use astream. It returns an async iterator, which you consume with async for. The loop yields chunks as they are produced, while the event loop remains free to handle other tasks.
import asyncio
from langchain_openai import ChatOpenAI
async def main():
model = ChatOpenAI(model="gpt-4o-mini")
async for chunk in model.astream("Write a short story about a robot learning to code"):
print(chunk.content, end="", flush=True)
asyncio.run(main())This pattern is ideal for web applications that serve responses over WebSocket or Server-Sent Events. The server can push each chunk to the client as soon as it arrives, without holding a thread or blocking other requests.
Handling Errors and Cancellation
Async streaming introduces two failure modes: errors from the model and cancellation by the client. Both need explicit handling.
Provider errors can surface as exceptions from the model SDK, such as OpenAI's rate-limit or network errors, or as LangChain-specific exceptions. Catch the exceptions you expect from your provider, or catch a broad Exception in the stream loop so you can stop the stream cleanly. Because asyncio.CancelledError inherits from BaseException, use a separate except asyncio.CancelledError block if you need to handle cancellation.
import asyncio
from langchain_openai import ChatOpenAI
async def stream_with_error_handling():
model = ChatOpenAI(model="gpt-4o-mini")
try:
async for chunk in model.astream("Generate a long response"):
print(chunk.content, end="", flush=True)
except Exception as exc:
# Log provider failures, timeouts, or rate limits
print(f"\nStream failed: {exc}")
except asyncio.CancelledError:
# Client disconnected or task was cancelled
print("\nStream cancelled")
raiseWhen a client disconnects, the async stream may be cancelled with CancelledError. Clean up any resources, such as closing a database connection or releasing a semaphore, but re-raise the exception so cancellation propagates correctly.
Performance and Concurrency Considerations
Using asyncio.gather lets you run multiple async LLM calls concurrently. This is useful when you need to summarize several documents or generate multiple suggestions in parallel.
import asyncio
from langchain_openai import ChatOpenAI
async def generate_summary(model, text):
response = await model.ainvoke(f"Summarize: {text}")
return response.content
async def main():
model = ChatOpenAI(model="gpt-4o-mini")
texts = ["First document...", "Second document...", "Third document..."]
results = await asyncio.gather(*(generate_summary(model, t) for t in texts))
print(results)
asyncio.run(main())Be aware of rate limits. Running many concurrent calls may hit provider limits. Wrap the per-call work in a function that acquires an asyncio.Semaphore if you need to cap simultaneous requests. The exact limit depends on your provider and plan, so tune it based on your usage.
Streaming and async also affect resource usage. Streaming can reduce memory pressure when you process chunks incrementally instead of accumulating the entire response. Async reduces thread usage by avoiding a blocked thread per request, which is important in high-concurrency web servers.
Choosing Between Streaming and Async
Streaming and async are not mutually exclusive, but you may need to choose one depending on your use case.
Use synchronous streaming (stream) when you are in a simple script or a synchronous framework like Flask and you only need to show progress to the user. Use async streaming (astream) when you are in an async framework like FastAPI or when you need to handle multiple clients concurrently. Use ainvoke when you only need the final result and want to avoid streaming overhead, but still want non-blocking behavior.
For a web endpoint that returns a complete JSON response, ainvoke is sufficient. For an endpoint that streams tokens to a chat interface, astream is the right choice. The decision depends on whether the client expects incremental updates or a single final payload.
Cancellation and Resource Cleanup in Async Streaming
Cleanup code inside an asynchronous stream consumer should go in a finally block so it runs on both normal completion and cancellation. If you exit a stream loop early, call aclose() on the async iterator when the object supports it, especially if you hold file handles, database connections, or rate-limit slots.
async def consume_stream():
model = ChatOpenAI(model="gpt-4o-mini")
stream = model.astream("Long text")
try:
async for chunk in stream:
yield chunk.content
finally:
await stream.aclose()In a FastAPI endpoint that streams responses, the framework may cancel the generator when the client disconnects. The finally block runs, allowing you to log the cancellation or clean up state. This prevents resource leaks in long-running applications.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.