Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python httpx File Upload, Download, and Streaming

Learn how to upload, download, and stream files with Python's httpx library, including memory-safe large file handling, retries, resumable downloads, and sync vs async choices.

~6 min readPython
#httpx#file upload#file download#streaming
Illustration of Python httpx library handling file upload, download, and streaming operations.

Moving files over HTTP in Python is a common task. httpx offers a modern synchronous and asynchronous client for uploads, downloads, and streaming. This article covers the practical mechanics of using it for file transfer, with attention to memory usage, error handling, and concurrency.

Uploading Files with httpx

The simplest way to upload a file with httpx is to pass a file handle to the files parameter of a request. The library sends it as multipart/form-data.

python
import httpx with open("report.pdf", "rb") as f: response = httpx.post("https://example.com/upload", files={"file": f}) print(response.status_code)

The files parameter accepts a dictionary where the key is the field name expected by the server. The value can be a file object, a tuple of (filename, file_object), or a tuple of (filename, file_object, content_type). If you need to control the filename, use the tuple form:

python
files = {"file": ("quarterly-report.pdf", f, "application/pdf")}

For small files this is fine. However, when httpx encodes a multipart body from a file object, the file data may be read into memory, so the files parameter is not the best choice for multi-gigabyte files. For those, you need streaming.

Downloading Files with httpx

Downloading a file with httpx is equally direct. The get method returns a response object whose .content attribute holds the full body in memory. Write it to disk with a binary write.

python
import httpx response = httpx.get("https://example.com/data.zip") response.raise_for_status() with open("data.zip", "wb") as f: f.write(response.content)

This approach loads the entire file into RAM. For a 2 GB archive, that is a problem. The stream context manager avoids this by exposing the response body as a byte iterator.

python
with httpx.stream("GET", "https://example.com/data.zip") as response: response.raise_for_status() with open("data.zip", "wb") as f: for chunk in response.iter_bytes(): f.write(chunk)

iter_bytes() yields chunks of configurable size (default 65536 bytes). This keeps memory usage bounded by the chunk size, not the file size.

Streaming File Uploads

To upload a large file without loading it into memory, use httpx's streaming request body. The content parameter accepts a generator that yields bytes. Combined with the headers parameter to set the content type, this sends the file as a raw stream rather than multipart.

python
import httpx def file_chunks(file_path, chunk_size=65536): with open(file_path, "rb") as f: while True: chunk = f.read(chunk_size) if not chunk: break yield chunk response = httpx.post( "https://example.com/upload-raw", content=file_chunks("large.iso"), headers={"Content-Type": "application/octet-stream"} ) print(response.status_code)

This is a true streaming upload. The generator is consumed by the underlying HTTP connection, so memory usage stays low. However, the server must accept raw body uploads; many APIs expect multipart form data. For multipart streaming, you need a different approach, such as building the multipart body yourself with a generator. httpx does not provide a built-in streaming multipart upload, so if the server requires multipart/form-data, you must either buffer the file or implement the multipart encoding manually.

Streaming File Downloads

Streaming downloads are more common and better supported. The stream context manager gives you fine-grained control over the response body. Beyond iter_bytes, you can use iter_text for text files, iter_lines for line-delimited data, and iter_raw for the raw undecoded bytes.

python
with httpx.stream("GET", "https://example.com/large.jsonl") as response: for line in response.iter_lines(): process_line(line)

When you use iter_lines, httpx decodes the bytes according to the response encoding. For binary data, iter_bytes is the right choice. The stream is closed automatically when you exit the context manager, releasing the connection back to the pool.

One caveat: if you read only part of the stream and then exit, the remaining bytes are discarded. This is fine for a one-off request, but if you reuse a client with a connection pool, the connection may be closed instead of reused. To force a clean close, call response.close() explicitly.

Handling Large Files and Memory Usage

The main reason to use streaming is to keep memory usage flat. The files parameter and .content attribute can both buffer the entire payload. For files that exceed a few hundred megabytes, that can cause the process to run out of memory or trigger swap. Streaming with generators and iter_bytes keeps the working set small.

There is also a practical limit on the chunk size. A larger chunk reduces the number of iterations but increases memory per chunk. A chunk size of 64 KB is a reasonable default; you can tune it based on your network and disk speed. For extremely high-throughput transfers, you might increase it to 1 MB, but test it in your environment first.

When downloading, you also need to consider disk I/O. Writing each chunk with a separate write call can be slow. Using a buffered writer or accumulating chunks into a larger buffer before writing can improve throughput. For example:

python
with open("output.bin", "wb") as f: for chunk in response.iter_bytes(): f.write(chunk)

The file object itself is buffered by default, so this is usually sufficient. Avoid calling flush on every iteration.

Error Handling and Retries for File Transfers

Network failures are common during large transfers. httpx raises httpx.RequestError for connection problems and httpx.HTTPStatusError for 4xx/5xx responses when you call raise_for_status(). You should catch these and decide whether to retry.

python
import httpx import time max_retries = 3 for attempt in range(max_retries): try: with httpx.stream("GET", "https://example.com/bigfile") as response: response.raise_for_status() with open("bigfile", "wb") as f: for chunk in response.iter_bytes(): f.write(chunk) break except httpx.TransportError as exc: if attempt == max_retries - 1: raise time.sleep(2 ** attempt)

Retrying a streaming download is tricky because you cannot resume from the middle of a response. The simplest approach is to restart the request from the beginning. If the server supports range requests, you can implement resumable downloads by sending the Range header and opening the file in append mode.

python
headers = {"Range": f"bytes={existing_size}-"} with httpx.stream("GET", url, headers=headers) as response: if response.status_code == 206: # Partial Content with open(file, "ab") as f: for chunk in response.iter_bytes(): f.write(chunk) else: response.raise_for_status() with open(file, "wb") as f: for chunk in response.iter_bytes(): f.write(chunk)

This pattern works only if the server supports Accept-Ranges. Not all do, so check the response headers before relying on it.

Concurrency and Performance Considerations

httpx supports asynchronous requests with AsyncClient, which allows you to upload or download multiple files concurrently. For I/O-bound tasks, this can significantly improve throughput.

python
import asyncio import httpx async def download_one(client, url, dest): async with client.stream("GET", url) as response: response.raise_for_status() with open(dest, "wb") as f: async for chunk in response.aiter_bytes(): f.write(chunk) async def main(): async with httpx.AsyncClient() as client: tasks = [ download_one(client, f"https://example.com/file{i}.zip", f"file{i}.zip") for i in range(5) ] await asyncio.gather(*tasks) asyncio.run(main())

When using concurrency, be aware of the connection pool limit. httpx defaults to a maximum of 100 connections per client. If you have more concurrent transfers, you may need to increase that limit or use multiple clients. Also, each stream holds a connection open, so you should not open an unbounded number of streams. Use a semaphore to limit concurrency.

For uploads, streaming a generator with AsyncClient requires an async generator. Define a generator that yields chunks and pass it to content.

python
import asyncio import httpx async def async_file_chunks(file_path): with open(file_path, "rb") as f: while True: chunk = f.read(65536) if not chunk: break yield chunk async def upload_file(file_path): async with httpx.AsyncClient() as client: response = await client.post( "https://example.com/upload", content=async_file_chunks(file_path), headers={"Content-Type": "application/octet-stream"} ) return response asyncio.run(upload_file("large.bin"))

Note that the async generator runs in the same event loop, so it will block the loop during file reads. For truly non-blocking file I/O, you would need a library like aiofiles, but for many cases the blocking read is acceptable because the disk is faster than the network.

Choosing Between Sync and Async for File Transfers

Deciding whether to use the synchronous or asynchronous API depends on your application's concurrency model. If you are building a CLI tool or a script that transfers one file at a time, the synchronous API is simpler and easier to debug. If you are building a web service or a crawler that must handle many files simultaneously, the async API is the right fit.

The streaming mechanisms are equivalent in both APIs: iter_bytes and aiter_bytes, stream and async with client.stream. The error-handling patterns are also similar. The main difference is the event loop. In an async application, you should never call the synchronous httpx methods inside a coroutine, because they block the event loop. Use AsyncClient consistently.

One practical consideration is timeout configuration. Large file transfers can take minutes. The default timeout in httpx is 5 seconds per operation, which is too low if a read or write can take longer than that. Set a higher timeout or disable it entirely for streaming operations.

python
client = httpx.Client(timeout=httpx.Timeout(connect=10, read=300, write=300, pool=10))

For streaming, the read timeout applies to each chunk read, not the entire transfer. So if the server sends a chunk every 30 seconds, a read timeout of 60 seconds is safe. If you set it too low, the transfer will abort even though the connection is healthy.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.