Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python concurrent.futures: Thread and Process Pools

Learn to use Python's concurrent.futures module to run tasks in thread and process pools, collect results, handle failures, and choose the right executor for CPU- and I/O-bound work.

~6 min readPython
#concurrency#ThreadPoolExecutor#ProcessPoolExecutor#as_completed
A clean editorial illustration showing two parallel execution pools routing task cards to worker threads and processes, with results emerging on the right side.

Python's concurrent.futures module lets you run callables asynchronously in a pool of worker threads or processes and collect their results through Future objects. This article explains the shared executor API, how to handle results and failures, and how to choose between ThreadPoolExecutor and ProcessPoolExecutor.

What concurrent.futures Provides

concurrent.futures is a high-level asynchronous-execution API. Instead of starting threads or processes directly, you submit callables to an executor and receive Future objects that represent pending results.

The module ships with two concrete executors:

  • ThreadPoolExecutor runs tasks in a pool of worker threads.
  • ProcessPoolExecutor runs tasks in a pool of worker processes.

Both executors share the same API, so switching between them often means changing one import and one constructor call. That shared interface is the main reason developers choose concurrent.futures over raw threading or multiprocessing code: the orchestration logic stays the same regardless of the executor.

Submitting Tasks and Collecting Results

The simplest way to use an executor is the submit method, which schedules a callable and returns a Future immediately:

python
from concurrent.futures import ThreadPoolExecutor def square(n): return n * n with ThreadPoolExecutor(max_workers=4) as executor: future = executor.submit(square, 10) result = future.result() print(result) # 100

The result() call blocks until the task completes. If the callable raises an exception, result() re-raises it in the caller, so error handling feels similar to synchronous code.

For bulk operations, map is more convenient. It applies a function to every item in an iterable and returns results in input order:

python
from concurrent.futures import ThreadPoolExecutor def square(n): return n * n with ThreadPoolExecutor(max_workers=4) as executor: results = list(executor.map(square, range(10))) print(results) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

map preserves input order, which is useful when result position matters. The trade-off is that later results are blocked by a slow earlier task: if the task for index 5 finishes before the task for index 3, the iterator still waits until index 3 is ready before yielding index 5.

Processing Results as They Complete

When tasks have uneven durations, waiting for results in submission order wastes time. The as_completed function yields futures in the order they finish:

python
from concurrent.futures import ThreadPoolExecutor, as_completed import random import time def fetch(url): # Simulate variable network latency delay = random.uniform(0.1, 1.0) time.sleep(delay) return url urls = ["https://example.com/a", "https://example.com/b", "https://example.com/c"] with ThreadPoolExecutor(max_workers=3) as executor: futures = {executor.submit(fetch, url): url for url in urls} for future in as_completed(futures): url = futures[future] print(f"Finished {url}: {future.result()}")

The dictionary mapping futures back to their inputs is a common pattern here, because as_completed only gives you the Future object, not the original argument.

Choosing Between Threads and Processes

The most important decision when using concurrent.futures is which executor to instantiate. The choice depends on what your tasks actually do.

CPU-bound tasks

Python threads cannot execute Python bytecode in parallel because of the Global Interpreter Lock (GIL) in CPython. For CPU-bound pure-Python work, ThreadPoolExecutor will not give you parallelism; it will only add thread scheduling overhead. Some native libraries release the GIL during long operations, but that behavior is library-specific.

Use ProcessPoolExecutor for CPU-bound tasks. Each worker runs in a separate process with its own interpreter, so the GIL does not constrain them.

python
from concurrent.futures import ProcessPoolExecutor import hashlib def compute_checksum(data): # CPU-heavy work return hashlib.sha256(data).hexdigest() chunks = [b"alpha", b"beta", b"gamma"] if __name__ == "__main__": with ProcessPoolExecutor(max_workers=4) as executor: results = list(executor.map(compute_checksum, chunks))

The if __name__ == "__main__": guard is especially important on Windows, where the default process start method is spawn; it prevents each worker process from recursively re-running the script.

I/O-bound tasks

For network requests, file reads, or database queries, the thread spends most of its time waiting on I/O. During that wait, the GIL is released, so other threads can run. ThreadPoolExecutor is the right choice here because threads are cheaper to create than processes, and because they live in the same process, arguments and return values do not need to be pickled.

Task typeRecommended executorReason
CPU-boundProcessPoolExecutorAvoids GIL, gives true parallelism
I/O-boundThreadPoolExecutorGIL released during I/O waits, threads are lightweight
MixedEvaluate per taskSeparate the workload or use both executors

Process overhead

Processes do not share memory. Arguments and return values must be pickled and sent over a pipe. If your tasks pass large objects, the serialization cost can dominate the actual computation. In that case, a thread-based approach may be faster even for CPU-heavy work, because no pickling is needed.

Handling Exceptions and Timeouts

A Future captures the outcome of a task, including failures. When a task raises an exception, the exception is stored in the future and re-raised when you call result():

python
from concurrent.futures import ThreadPoolExecutor def risky_task(value): if value < 0: raise ValueError("negative values not allowed") return value * 2 with ThreadPoolExecutor(max_workers=2) as executor: future = executor.submit(risky_task, -5) try: result = future.result() except ValueError as exc: print(f"Task failed: {exc}")

The exception is raised in the caller's thread, not in the worker, so you can handle it with normal try/except blocks. This is a significant advantage over raw threading, where an unhandled exception in a worker thread would print a traceback and terminate that thread without any clean way to propagate the error.

result() also accepts a timeout argument. If the task does not finish within the given seconds, it raises TimeoutError:

python
import time from concurrent.futures import ThreadPoolExecutor def slow_task(): time.sleep(5) return "done" with ThreadPoolExecutor(max_workers=2) as executor: future = executor.submit(slow_task) try: result = future.result(timeout=2.0) except TimeoutError: print("Task did not finish in time")

Note that a timeout does not cancel the task. The worker keeps running in the background; only the wait is abandoned. If you need to cancel, use future.cancel(), but cancellation only works while the task is still queued and has not started running.

Cancellation and Cleanup

The Future.cancel() method attempts to cancel a task. It returns True if the task was cancelled before any worker picked it up, and False if the task is already running or finished. Once a task is running, there is no safe way to stop it from the outside; the executor will wait for it to complete.

Using the executor as a context manager ensures that shutdown() is called automatically:

python
with ThreadPoolExecutor(max_workers=4) as executor: # Submit tasks and use their futures here. pass # executor.shutdown(wait=True) runs on exit

By default, shutdown(wait=True) blocks until all pending futures finish. If you pass wait=False, the method returns immediately, but the interpreter will not exit until the worker threads are done. In practice, the context manager form with default settings is the safest choice for most scripts.

A Practical Example: Parallel HTTP Fetching

Putting the pieces together, here is a realistic pattern for fetching multiple URLs concurrently with ThreadPoolExecutor:

python
from concurrent.futures import ThreadPoolExecutor, as_completed import urllib.request def fetch_url(url): with urllib.request.urlopen(url, timeout=5) as response: return url, response.status, len(response.read()) urls = [ "https://example.com", "https://httpbin.org/status/200", "https://httpbin.org/status/404", ] with ThreadPoolExecutor(max_workers=5) as executor: futures = {executor.submit(fetch_url, url): url for url in urls} for future in as_completed(futures): try: url, status, size = future.result() print(f"{url}: {status} ({size} bytes)") except Exception as exc: print(f"Request failed: {exc}")

The max_workers value controls how many requests run concurrently. Setting it too high can overwhelm the remote server or exhaust local file descriptors; setting it too low means the pool underutilizes available bandwidth. For I/O-bound workloads, a good starting point is often several times the number of CPU cores, but the right value depends on the external service's limits.

Where This Approach Breaks Down

concurrent.futures is not a general-purpose parallel programming toolkit. It works well for independent tasks, but it has limitations you should know about before building a large system on it.

Tasks must be independent

The executor model assumes tasks do not need to coordinate with each other. If one task depends on the result of another, you either chain them inside a single callable or fall back to lower-level synchronization primitives. The module does not provide a way to express dependencies between futures.

Shared state is problematic

With ThreadPoolExecutor, worker threads share memory, so you need locks or other synchronization if multiple tasks mutate the same object. With ProcessPoolExecutor, there is no shared memory at all; each task gets a copy of its arguments. Mutating a shared object inside a process pool task has no effect on the parent process.

Pickling constraints

Process pool arguments and return values must be picklable. Lambdas, local functions, and some objects cannot be pickled, which limits what you can pass to a process pool task. A common workaround is to define the task function at module level.

Task granularity matters

The overhead of submitting a task and collecting a result is small but not zero. If each task takes microseconds, the orchestration overhead can exceed the task duration. In that case, batching work into larger chunks or using a different approach such as multiprocessing.Pool with chunksize may be more effective.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.