Using Python tqdm with Multiprocessing and Asyncio
Integrate tqdm progress bars with multiprocessing and asyncio in Python. Learn practical patterns for multiprocessing.Pool, asyncio.gather, and progress bar overhead control.

Using tqdm with multiprocessing and asyncio requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns shown in the examples below.
When you run a loop in Python, tqdm gives you a simple progress bar. But when you distribute work across processes or run tasks concurrently with asyncio, the same straightforward usage breaks. The progress bar either never updates, updates erratically, or the output becomes interleaved with worker logs. This article shows how to use Python tqdm with multiprocessing and asyncio so the progress bar reflects real work.
Why tqdm Needs Special Handling in Concurrent Code
A standard tqdm loop works because the loop body runs sequentially in the same process. When you use multiprocessing or asyncio, the work happens outside the main loop. With multiprocessing, each worker has its own memory space, so calling pbar.update() inside a worker does not affect the progress bar in the parent process. With asyncio, tasks run on the same thread but yield control, so a naive for loop that awaits tasks one by one will not show progress until all tasks finish unless you update the bar between awaits.
The core issue is that tqdm needs to be updated from the main thread or process, at the point where you collect results. The patterns below reflect that.
Using tqdm with multiprocessing.Pool
The most reliable way to use tqdm with a multiprocessing.Pool is to iterate over the results as they arrive, using imap or imap_unordered. This keeps the update in the parent process.
import multiprocessing
from tqdm import tqdm
def worker(x):
# simulate work
return x * x
if __name__ == "__main__":
data = range(100)
with multiprocessing.Pool() as pool:
results = []
for result in tqdm(pool.imap_unordered(worker, data), total=len(data)):
results.append(result)imap_unordered yields results as soon as they are ready, so the progress bar advances as each task completes. The total argument tells tqdm how many items to expect. Without it, tqdm would show an indeterminate bar.
If you need results in the original order, use imap instead. The progress bar will still update, but it may pause if a slow task blocks the order.
For a simpler API, tqdm provides tqdm.contrib.concurrent.process_map, which wraps a ProcessPoolExecutor and returns results in order:
from tqdm.contrib.concurrent import process_map
def worker(x):
return x * x
if __name__ == "__main__":
results = process_map(worker, range(100), max_workers=4)This function handles the progress bar internally and is useful when you do not need fine-grained control over the pool.
Using tqdm with asyncio Tasks
With asyncio, you typically gather tasks with asyncio.gather. To show progress, you can use tqdm.asyncio.tqdm, which provides an async-compatible progress bar. The simplest way is to replace asyncio.gather with tqdm.gather:
import asyncio
from tqdm.asyncio import tqdm
async def worker(x):
await asyncio.sleep(0.1)
return x * x
async def main():
data = range(100)
tasks = [worker(x) for x in data]
results = await tqdm.gather(*tasks, total=len(tasks))
return results
if __name__ == "__main__":
asyncio.run(main())tqdm.gather behaves like asyncio.gather but updates the progress bar as each task completes. It returns results in the same order as the input tasks.
If you need more control, you can use asyncio.as_completed with a standard tqdm object:
import asyncio
from tqdm import tqdm
async def main():
tasks = [worker(x) for x in range(100)]
pbar = tqdm(total=len(tasks))
for coro in asyncio.as_completed(tasks):
await coro
pbar.update()
pbar.close()This pattern lets you process each result as it arrives, but it requires manual progress bar management. The tqdm.gather approach is usually cleaner.
Choosing Between multiprocessing and asyncio for Progress Tracking
The choice of concurrency model depends on the workload, not on tqdm. Use multiprocessing when the work is CPU-bound and benefits from multiple cores. Use asyncio when the work is I/O-bound, such as network requests or file reads, where most time is spent waiting.
tqdm works with both, but the integration differs. With multiprocessing, you must avoid updating the bar from worker processes. With asyncio, you must avoid blocking the event loop with synchronous tqdm writes. The tqdm.asyncio module handles that by using async-compatible methods.
For a thread-based approach that suits blocking I/O or mixed workloads, tqdm also provides tqdm.contrib.concurrent.thread_map. It works like process_map but uses a thread pool, which is lighter than processes and suitable for I/O-bound tasks.
Controlling Refresh Rate and Overhead
tqdm writes to stderr and refreshes at a rate controlled by mininterval (default 0.1 seconds). In concurrent workloads, frequent updates from many tasks can cause excessive output and slow down the program. Set mininterval higher, for example 0.5 or 1.0, to reduce writes:
with tqdm(total=len(data), mininterval=0.5) as pbar:
for result in pool.imap_unordered(worker, data):
pbar.update()The overhead of tqdm itself is small, but it can become noticeable if you update the bar millions of times. The mininterval limit ensures that tqdm does not write on every update; it only writes when the interval has elapsed. This is especially important when using imap_unordered with a large dataset.
Common Pitfalls and How to Avoid Them
One frequent mistake is calling tqdm.update() inside a worker function. As mentioned, this does not affect the parent process's bar. Always update from the main process.
Another issue is forgetting the if __name__ == "__main__": guard when using multiprocessing on Windows or macOS with spawn. Without it, the pool creation can recurse and cause errors.
When using process_map, note that it uses concurrent.futures.ProcessPoolExecutor. The worker function must be picklable, so define it at the module level, not inside a closure or a local function.
With asyncio, a common error is mixing synchronous tqdm with async code. The standard tqdm object's update method is synchronous, not awaitable, and frequent calls can block the event loop while writing output. Use tqdm.asyncio.tqdm to get an async-compatible version.
Finally, if you see overlapping progress bars or garbled output, it usually means multiple processes are writing to the same terminal. The patterns in this article avoid that by keeping all tqdm output in the main process.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.