Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python ProcessPoolExecutor: Parallel Execution for CPU-Bound Tasks

Learn how to use Python's ProcessPoolExecutor for CPU-bound parallelism, including map(), submit(), error handling, and performance tradeoffs.

~6 min readPython
#concurrency#parallelism#multiprocessing#concurrent.futures
Illustration of Python ProcessPoolExecutor distributing CPU-bound tasks across multiple processes.

When a Python program is CPU-bound rather than I/O-bound, the Global Interpreter Lock (GIL) prevents threads from running Python bytecode in parallel. ProcessPoolExecutor, from the standard library's concurrent.futures module, sidesteps that limitation by running callables in separate processes. This article covers how to use ProcessPoolExecutor for CPU-bound work, its main API methods, error handling, and the tradeoffs involved.

How ProcessPoolExecutor Distributes Work

ProcessPoolExecutor creates a pool of worker processes, each with its own Python interpreter and memory space. When you submit a callable, the executor serializes the callable and its arguments, sends the work item to an available worker, and later returns the result to the parent process. Because each worker process runs independently, the GIL does not block parallel execution of CPU-intensive code.

The executor is part of the standard library, so you can import it directly:

python
from concurrent.futures import ProcessPoolExecutor

The most common way to use it is as a context manager, which ensures the pool is cleaned up when the block exits:

python
with ProcessPoolExecutor() as executor: future = executor.submit(pow, 2, 10) result = future.result() print(result)

This example shows the core pattern: submit() schedules a callable and returns a Future object. Calling result() blocks until the computation finishes and returns the value.

Submitting Tasks with submit() and as_completed()

For workloads where tasks are not known in advance, or where results should be processed as they complete, use submit() together with as_completed().

python
from concurrent.futures import ProcessPoolExecutor, as_completed def compute_square(n): return n * n numbers = [1, 2, 3, 4, 5] with ProcessPoolExecutor(max_workers=3) as executor: futures = {executor.submit(compute_square, n): n for n in numbers} for future in as_completed(futures): n = futures[future] try: result = future.result() print(f'Square of {n} is {result}') except Exception as exc: print(f'Task {n} raised {exc}')

as_completed() yields futures in the order they finish, not in submission order. This is useful when tasks have varying durations and you want to start processing results as soon as they are available. The dictionary maps each future back to its original input so you can identify which task produced which result.

Mapping Tasks with map() for Ordered Results

The map() method is a simpler alternative when you have a sequence of inputs and want results in the same order. It behaves like the built-in map(), but distributes the work across the pool.

python
with ProcessPoolExecutor(max_workers=4) as executor: results = executor.map(compute_square, range(10)) for value in results: print(value)

map() returns an iterator that yields results in the order of the input iterable, regardless of completion order. If any task raises an exception, the exception is raised when iteration reaches that position. You can also pass multiple iterables if the callable accepts multiple arguments:

python
def add(a, b): return a + b with ProcessPoolExecutor() as executor: sums = executor.map(add, [1, 2, 3], [4, 5, 6]) print(list(sums)) # [5, 7, 9]

Unlike a manual batch-by-batch approach, map() eagerly submits tasks for the entire input before the first result is yielded and keeps internal future objects until results are consumed. It is therefore not a way to bound memory for extremely large inputs; process such inputs in smaller batches if memory is a concern. It also does not provide per-task control over cancellation or early exit.

Handling Exceptions and Timeouts

Exceptions raised inside a worker process are pickled and re-raised in the parent process when result() is called. This means you can catch them like normal Python exceptions, but the exception object must be picklable.

python
def divide(a, b): return a / b with ProcessPoolExecutor() as executor: future = executor.submit(divide, 1, 0) try: future.result() except ZeroDivisionError as exc: print(f'Caught: {exc}')

If a task runs longer than expected, set a timeout on result():

python
try: result = future.result(timeout=5) except TimeoutError: print('Task did not finish in time') future.cancel()

Note that cancel() only works if the task has not yet started running. If the task is already executing in a worker process, cancellation is not possible; the worker will continue until the task completes or the process is terminated.

ProcessPoolExecutor vs ThreadPoolExecutor

The choice between ProcessPoolExecutor and ThreadPoolExecutor depends on the nature of the workload. The table below summarizes the key differences:

CriterionProcessPoolExecutorThreadPoolExecutor
GIL impactBypasses the GILLimited by the GIL for Python code
Memory modelSeparate memory space per processShared memory within one process
OverheadHigher (process creation, pickling)Lower (thread creation)
Best suited forCPU-bound tasksI/O-bound tasks
Data sharingRequires pickling and IPCDirect access to shared objects

For CPU-bound pure Python code, ProcessPoolExecutor is usually the right choice because it allows true parallel execution across multiple CPU cores. For I/O-bound tasks such as network requests or file reads, threads are generally more efficient because they avoid process creation and serialization overhead.

Performance Overhead and When It Pays Off

Using ProcessPoolExecutor introduces two main sources of overhead: starting worker processes and serializing data. Starting a process is more expensive than starting a thread. In addition, callables, arguments, and results are typically pickled so they can cross the process boundary.

For tiny tasks that finish in microseconds, serialization and IPC cost can dominate and make the pool slower than sequential execution. The pool is most likely to pay off when each task involves enough independent CPU work to amortize process startup and data transfer. If the tradeoff is not obvious, benchmark the actual workload.

You can control the number of workers with max_workers. Setting it to the number of CPU cores is a common starting point, but the optimal value depends on the workload and on how much of each task is CPU-bound versus waiting.

python
import os from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor: # distribute work

Common Pitfalls: Pickling, Global State, and Startup Cost

Because work items are sent through an inter-process queue, the callable, its arguments, and its return value must be picklable. This excludes lambdas, nested functions, and many dynamically created objects. Define the callable at module level and pass simple picklable data.

Start methods vary by platform and Python version. On Windows, spawn is the default; on Unix-like systems, the default may be fork, forkserver, or spawn depending on the Python version. Use multiprocessing.get_start_method() if you need to know. With spawn, each worker re-imports the main module; with forkserver, a server process imports it before forking workers. In both cases, guard the entry point with if __name__ == '__main__': to avoid recursive process creation.

A fork-based start method can reduce startup cost because a worker inherits the parent process's memory, including module-level objects, at the time the worker is created. It does not remove the need to pickle values passed through the executor; callable arguments still cross process boundaries and must be serializable. Forking can also be unsafe when the parent process is multithreaded.

Global state is not shared between processes. If a task modifies a global variable, the change is not visible to the parent or to other workers. With fork, workers initially inherit the parent's module-level state, but later changes are not propagated. This is a common source of confusion when porting threaded code to a process pool.

Finally, be mindful of startup cost. Creating a pool with many workers takes time, so create the executor once and reuse it for multiple batches of tasks rather than creating a new pool for each small set of calls. The context manager pattern ensures cleanup, but you can also call shutdown() explicitly when you need more control over the pool's lifetime.

When used correctly, ProcessPoolExecutor is a reliable way to parallelize CPU-bound work in Python. Understanding its serialization requirements and process overhead helps you choose when it is the right tool and structure code to avoid common failure modes.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.