Python aiohttp Streaming File Download
python aiohttp streaming file download: Learn how to stream file downloads with aiohttp to keep memory usage low, handle large files, and manage errors in async Python.

The Problem with Loading a Whole File into Memory
When you download a file with aiohttp, the default behavior loads the entire response body into memory. For large files, that can exhaust memory or cause the process to stall. Streaming the download reads the response in chunks, letting you write each chunk to disk as it arrives.
This article shows how to implement python aiohttp streaming file download patterns that keep memory usage flat and handle large files reliably.
A Minimal aiohttp Streaming Download
The core of streaming a file download with aiohttp is the ClientResponse.content attribute. Instead of calling read() to pull the whole body, you iterate over chunks.
import aiohttp
import asyncio
async def download_file(url: str, dest: str) -> None:
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
resp.raise_for_status()
with open(dest, "wb") as f:
async for chunk in resp.content.iter_chunked(1024):
f.write(chunk)
asyncio.run(download_file("https://example.com/large-file.zip", "file.zip"))iter_chunked(1024) yields bytes objects of up to 1024 bytes. The loop writes each chunk to disk immediately, so the memory footprint stays bounded regardless of file size.
Choosing a Chunk Size
The chunk size you pass to iter_chunked affects both memory usage and I/O efficiency. A smaller chunk size (e.g., 256 bytes) uses less memory but makes more write calls. A larger chunk size (e.g., 64 KB) reduces write overhead but increases memory per chunk. The right value depends on your file size and system.
| Chunk Size | Memory per Chunk | Write Frequency |
|---|---|---|
| 256 B | Very low | High |
| 1 KB | Low | Moderate |
| 64 KB | Moderate | Low |
| 1 MB | High | Very low |
For most downloads, a chunk size between 1 KB and 64 KB works well. If you are downloading many files concurrently, a smaller chunk size can help keep total memory down.
Controlling Timeouts and Connection Limits
Streaming a large file can take a long time. The default aiohttp timeout applies to the whole operation, so you may need to adjust it. Use ClientTimeout to set separate read and connect timeouts.
import aiohttp
import asyncio
async def download_with_timeout(url: str, dest: str) -> None:
timeout = aiohttp.ClientTimeout(total=3600, connect=10)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(url) as resp:
resp.raise_for_status()
with open(dest, "wb") as f:
async for chunk in resp.content.iter_chunked(64 * 1024):
f.write(chunk)The total parameter sets the maximum time for the entire request, including reading the body. If you expect a very large download, set it high enough or use None to disable the total timeout and rely on the read timeout instead.
Handling Errors and Partial Downloads
Network failures can interrupt a stream. raise_for_status() catches HTTP errors, but connection resets and timeouts raise exceptions. Wrap the download in a try/except block to clean up partial files.
import aiohttp
import asyncio
import os
async def safe_download(url: str, dest: str) -> bool:
try:
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
resp.raise_for_status()
with open(dest, "wb") as f:
async for chunk in resp.content.iter_chunked(64 * 1024):
f.write(chunk)
return True
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
if os.path.exists(dest):
os.remove(dest)
print(f"Download failed: {e}")
return FalseRemoving the partial file prevents a corrupted file from being used later. If you want to resume a download, you would need to send a Range header and open the file in append mode, which is more complex.
Streaming Server-Side Responses
aiohttp is not only a client; it also provides a server framework. If you need to serve a file as a stream, use StreamResponse and write chunks to the response. This is useful for large files that should not be loaded into memory.
from aiohttp import web
async def handle_download(request):
resp = web.StreamResponse()
resp.content_type = "application/octet-stream"
resp.headers["Content-Disposition"] = 'attachment; filename="file.zip"'
await resp.prepare(request)
with open("file.zip", "rb") as f:
while chunk := f.read(64 * 1024):
await resp.write(chunk)
await resp.write_eof()
return respThis server-side pattern keeps memory usage flat on the server as well. The StreamResponse sends headers first, then writes chunks as they are read from disk.
Memory and Performance Considerations
Streaming a download avoids loading the entire file into RAM. The memory used is roughly the chunk size plus the aiohttp internal buffers. This is critical when downloading files that are hundreds of megabytes or gigabytes.
The main performance cost of streaming is the number of write calls. Each chunk triggers a disk write, so a very small chunk size can slow the download. On the other hand, a very large chunk size increases memory usage. The optimal chunk size balances these factors and depends on your storage and network speed.
For concurrent downloads, each stream holds its own chunk buffer. If you download many files at once, use a smaller chunk size to keep total memory under control.
Concurrent Downloads with asyncio
One advantage of aiohttp is the ability to download multiple files concurrently without threads. You can create a list of tasks and run them with asyncio.gather.
import aiohttp
import asyncio
async def download_one(session, url, dest):
async with session.get(url) as resp:
resp.raise_for_status()
with open(dest, "wb") as f:
async for chunk in resp.content.iter_chunked(32 * 1024):
f.write(chunk)
async def download_many(urls_and_dests):
async with aiohttp.ClientSession() as session:
tasks = [download_one(session, url, dest) for url, dest in urls_and_dests]
await asyncio.gather(*tasks)The connection pool in ClientSession limits the number of simultaneous connections. By default, it allows 100 connections. If you start more tasks, they queue until a connection is free. Adjust limit in TCPConnector to control concurrency.
connector = aiohttp.TCPConnector(limit=10)
async with aiohttp.ClientSession(connector=connector) as session:
...This keeps the number of active downloads bounded, which is important when downloading from a server that might throttle or when you want to avoid overwhelming your own network.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.