Python Multiprocessing: Process and Pool Explained
Learn how to run CPU-bound tasks in parallel with Python multiprocessing: Process, Pool, shared memory, locks, and common pitfalls.

Python's multiprocessing module lets you run multiple processes to execute CPU-bound work in parallel, bypassing the Global Interpreter Lock (GIL) that limits CPython threads. This article explains the core APIs—Process, Pool, and synchronization primitives—and shows how to share data safely between processes.
Why Multiprocessing Instead of Threads
CPython threads are limited by the GIL, a mutex that prevents multiple threads from executing Python bytecode simultaneously. For CPU-bound tasks, threads often provide no speedup because only one thread runs at a time. Multiprocessing sidesteps this limitation by spawning separate interpreter processes, each with its own memory space and GIL. This allows true parallel execution on multi-core hardware.
For I/O-bound tasks, threads or asyncio are usually a better fit because they can spend most of their time waiting on external resources without the overhead of separate processes. Multiprocessing is the right tool when the bottleneck is the CPU.
Creating and Starting Processes with Process
The Process class provides a direct way to start a new process. You instantiate it with a target function and arguments, then call start() and join().
import multiprocessing
def worker(n):
print(f"Working on {n}")
return n * 2
if __name__ == "__main__":
processes = []
for i in range(4):
p = multiprocessing.Process(target=worker, args=(i,))
processes.append(p)
p.start()
for p in processes:
p.join()Each process runs independently, and join() blocks until the process terminates. The if __name__ == "__main__" guard is required on Windows and recommended elsewhere to prevent infinite process spawning when the module is imported.
Using Pool for Parallel Mapping
For tasks that can be expressed as a function applied to many inputs, Pool provides a higher-level interface. Pool.map distributes the inputs across worker processes and collects results in order.
from multiprocessing import Pool
def square(x):
return x * x
if __name__ == "__main__":
with Pool(4) as pool:
results = pool.map(square, range(10))
print(results)The with block terminates the pool when the block exits. Pool.map blocks until all results are ready. For asynchronous execution, apply_async returns a result object that you can wait on later.
Sharing Data Between Processes
Processes do not share memory by default. To exchange data, you can use Queue, Pipe, or shared memory objects like Value and Array. Queues are thread- and process-safe and are the simplest way to pass messages.
from multiprocessing import Process, Queue
def producer(q):
q.put("data")
def consumer(q):
print(q.get())
if __name__ == "__main__":
q = Queue()
p1 = Process(target=producer, args=(q,))
p2 = Process(target=consumer, args=(q,))
p1.start(); p2.start()
p1.join(); p2.join()Shared memory (Value and Array) is faster but requires careful synchronization because multiple processes can read and write the same memory region concurrently.
Synchronizing Concurrent Access
When multiple processes access shared resources, you need locks to prevent race conditions. multiprocessing.Lock provides mutual exclusion.
from multiprocessing import Process, Lock, Value
def increment(lock, counter):
for _ in range(1000):
with lock:
counter.value += 1
if __name__ == "__main__":
lock = Lock()
counter = Value("i", 0)
processes = [Process(target=increment, args=(lock, counter)) for _ in range(4)]
for p in processes: p.start()
for p in processes: p.join()
print(counter.value)Without the lock, the increments would interleave unpredictably. Event and Semaphore are also available for more complex coordination.
Common Pitfalls and How to Avoid Them
Multiprocessing introduces several pitfalls that can break your code in production.
Pickling Limitations
Arguments and return values must be picklable. Lambdas, local functions, and some objects (like open sockets) cannot be pickled. Define target functions at module level and avoid passing unpicklable state.
Fork vs Spawn Start Methods
The default start method varies by platform. Windows uses spawn; macOS has used spawn since Python 3.8; Linux commonly uses fork. fork copies the parent memory, while spawn re-imports the module. Code that relies on global state initialized in the parent may behave differently across platforms. Use multiprocessing.get_context('spawn') (or another explicit method) to make the behavior consistent.
Global State Isolation
Each process has its own copy of global variables. Changes in one process do not affect others. This is often surprising but is a fundamental design choice. If you need shared state, use explicit shared memory or a manager.
Performance Considerations and When to Use Multiprocessing
Multiprocessing adds overhead: process creation, inter-process communication, and memory duplication for fork. For small tasks, the overhead can exceed the speed benefit. Use multiprocessing when the work is CPU-intensive and the input size is large enough to justify the cost.
A rough rule is to use multiprocessing when each unit of work is CPU-heavy enough that process creation and communication overhead are small relative to the computation. For tasks that are very short or I/O-bound, threading or asyncio is often a better fit.
Also consider the memory footprint. Each process has its own Python interpreter and memory space, which can be significant for large data structures. Shared memory or Array can reduce duplication, but they add complexity.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.