Python ProcessPoolExecutor: Parallel Execution for CPU-Bound Tasks
Learn how to use Python's ProcessPoolExecutor for CPU-bound parallelism, including map(), submit(), error handling, and performance tradeoffs.

When a Python program is CPU-bound rather than I/O-bound, the Global Interpreter Lock (GIL) prevents threads from running Python bytecode in parallel. ProcessPoolExecutor, from the standard library's concurrent.futures module, sidesteps that limitation by running callables in separate processes. This article covers how to use ProcessPoolExecutor for CPU-bound work, its main API methods, error handling, and the tradeoffs involved.
How ProcessPoolExecutor Distributes Work
ProcessPoolExecutor creates a pool of worker processes, each with its own Python interpreter and memory space. When you submit a callable, the executor serializes the callable and its arguments, sends the work item to an available worker, and later returns the result to the parent process. Because each worker process runs independently, the GIL does not block parallel execution of CPU-intensive code.
The executor is part of the standard library, so you can import it directly:
from concurrent.futures import ProcessPoolExecutorThe most common way to use it is as a context manager, which ensures the pool is cleaned up when the block exits:
with ProcessPoolExecutor() as executor:
future = executor.submit(pow, 2, 10)
result = future.result()
print(result)This example shows the core pattern: submit() schedules a callable and returns a Future object. Calling result() blocks until the computation finishes and returns the value.
Submitting Tasks with submit() and as_completed()
For workloads where tasks are not known in advance, or where results should be processed as they complete, use submit() together with as_completed().
from concurrent.futures import ProcessPoolExecutor, as_completed
def compute_square(n):
return n * n
numbers = [1, 2, 3, 4, 5]
with ProcessPoolExecutor(max_workers=3) as executor:
futures = {executor.submit(compute_square, n): n for n in numbers}
for future in as_completed(futures):
n = futures[future]
try:
result = future.result()
print(f'Square of {n} is {result}')
except Exception as exc:
print(f'Task {n} raised {exc}')as_completed() yields futures in the order they finish, not in submission order. This is useful when tasks have varying durations and you want to start processing results as soon as they are available. The dictionary maps each future back to its original input so you can identify which task produced which result.
Mapping Tasks with map() for Ordered Results
The map() method is a simpler alternative when you have a sequence of inputs and want results in the same order. It behaves like the built-in map(), but distributes the work across the pool.
with ProcessPoolExecutor(max_workers=4) as executor:
results = executor.map(compute_square, range(10))
for value in results:
print(value)map() returns an iterator that yields results in the order of the input iterable, regardless of completion order. If any task raises an exception, the exception is raised when iteration reaches that position. You can also pass multiple iterables if the callable accepts multiple arguments:
def add(a, b):
return a + b
with ProcessPoolExecutor() as executor:
sums = executor.map(add, [1, 2, 3], [4, 5, 6])
print(list(sums)) # [5, 7, 9]Unlike a manual batch-by-batch approach, map() eagerly submits tasks for the entire input before the first result is yielded and keeps internal future objects until results are consumed. It is therefore not a way to bound memory for extremely large inputs; process such inputs in smaller batches if memory is a concern. It also does not provide per-task control over cancellation or early exit.
Handling Exceptions and Timeouts
Exceptions raised inside a worker process are pickled and re-raised in the parent process when result() is called. This means you can catch them like normal Python exceptions, but the exception object must be picklable.
def divide(a, b):
return a / b
with ProcessPoolExecutor() as executor:
future = executor.submit(divide, 1, 0)
try:
future.result()
except ZeroDivisionError as exc:
print(f'Caught: {exc}')If a task runs longer than expected, set a timeout on result():
try:
result = future.result(timeout=5)
except TimeoutError:
print('Task did not finish in time')
future.cancel()Note that cancel() only works if the task has not yet started running. If the task is already executing in a worker process, cancellation is not possible; the worker will continue until the task completes or the process is terminated.
ProcessPoolExecutor vs ThreadPoolExecutor
The choice between ProcessPoolExecutor and ThreadPoolExecutor depends on the nature of the workload. The table below summarizes the key differences:
| Criterion | ProcessPoolExecutor | ThreadPoolExecutor |
|---|---|---|
| GIL impact | Bypasses the GIL | Limited by the GIL for Python code |
| Memory model | Separate memory space per process | Shared memory within one process |
| Overhead | Higher (process creation, pickling) | Lower (thread creation) |
| Best suited for | CPU-bound tasks | I/O-bound tasks |
| Data sharing | Requires pickling and IPC | Direct access to shared objects |
For CPU-bound pure Python code, ProcessPoolExecutor is usually the right choice because it allows true parallel execution across multiple CPU cores. For I/O-bound tasks such as network requests or file reads, threads are generally more efficient because they avoid process creation and serialization overhead.
Performance Overhead and When It Pays Off
Using ProcessPoolExecutor introduces two main sources of overhead: starting worker processes and serializing data. Starting a process is more expensive than starting a thread. In addition, callables, arguments, and results are typically pickled so they can cross the process boundary.
For tiny tasks that finish in microseconds, serialization and IPC cost can dominate and make the pool slower than sequential execution. The pool is most likely to pay off when each task involves enough independent CPU work to amortize process startup and data transfer. If the tradeoff is not obvious, benchmark the actual workload.
You can control the number of workers with max_workers. Setting it to the number of CPU cores is a common starting point, but the optimal value depends on the workload and on how much of each task is CPU-bound versus waiting.
import os
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
# distribute workCommon Pitfalls: Pickling, Global State, and Startup Cost
Because work items are sent through an inter-process queue, the callable, its arguments, and its return value must be picklable. This excludes lambdas, nested functions, and many dynamically created objects. Define the callable at module level and pass simple picklable data.
Start methods vary by platform and Python version. On Windows, spawn is the default; on Unix-like systems, the default may be fork, forkserver, or spawn depending on the Python version. Use multiprocessing.get_start_method() if you need to know. With spawn, each worker re-imports the main module; with forkserver, a server process imports it before forking workers. In both cases, guard the entry point with if __name__ == '__main__': to avoid recursive process creation.
A fork-based start method can reduce startup cost because a worker inherits the parent process's memory, including module-level objects, at the time the worker is created. It does not remove the need to pickle values passed through the executor; callable arguments still cross process boundaries and must be serializable. Forking can also be unsafe when the parent process is multithreaded.
Global state is not shared between processes. If a task modifies a global variable, the change is not visible to the parent or to other workers. With fork, workers initially inherit the parent's module-level state, but later changes are not propagated. This is a common source of confusion when porting threaded code to a process pool.
Finally, be mindful of startup cost. Creating a pool with many workers takes time, so create the executor once and reuse it for multiple batches of tasks rather than creating a new pool for each small set of calls. The context manager pattern ensures cleanup, but you can also call shutdown() explicitly when you need more control over the pool's lifetime.
When used correctly, ProcessPoolExecutor is a reliable way to parallelize CPU-bound work in Python. Understanding its serialization requirements and process overhead helps you choose when it is the right tool and structure code to avoid common failure modes.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.