Using joblib Parallel and delayed with multiprocessing
Learn how to parallelize Python loops with joblib's Parallel and delayed, how they relate to multiprocessing, and how to avoid common pitfalls.

Parallelizing a loop with joblib usually means creating a sequence of delayed calls and passing it to Parallel. This article explains the core pattern, describes how joblib's default process-based backend relates to Python's multiprocessing, and covers practical decisions about backends, memory, and debugging.
When you need to parallelize a loop in Python, joblib's Parallel and delayed are often the quickest way to get started. They hide the details of process pools and task scheduling, but you still need to understand how they relate to multiprocessing when you move from a toy example to a real workload.
The Core Pattern: Parallel and delayed
The typical usage is a two-step process: wrap a function call with delayed, then pass a list of these delayed calls to Parallel. Here's a minimal example that computes the square of each number in a list:
from joblib import Parallel, delayed
def square(x):
return x * x
results = Parallel(n_jobs=2)(delayed(square)(i) for i in range(10))
print(results)The generator expression (delayed(square)(i) for i in range(10)) creates a series of delayed tasks. Parallel then distributes these tasks across the specified number of jobs. The result is a list containing the return values in the same order as the input, regardless of which worker finished first.
This pattern works because delayed captures the function and its arguments, and Parallel decides how to execute them. You can also use it with functions that take multiple arguments or keyword arguments:
def add(a, b=0):
return a + b
results = Parallel(n_jobs=2)(delayed(add)(i, b=10) for i in range(5))How joblib Uses Multiprocessing Under the Hood
By default, joblib uses the loky backend, a process-based executor that manages a pool of worker processes. Each job runs in a separate Python process, not a thread. Worker processes receive the function and its arguments through cloudpickle, an extended pickling mechanism, and send results back the same way.
Because each worker is a separate process, CPU-bound Python code can run in parallel without being limited by the Global Interpreter Lock (GIL). However, the serialization step adds overhead. For simple functions with small arguments, the cost of pickling can dominate the benefit of parallel execution, especially when the function itself is fast.
The loky backend also handles the re-importation of the main module in each worker. This matters on platforms that use the spawn process start method, such as Windows. You should still guard the executable part of a script with if __name__ == '__main__': to avoid side effects and infinite recursion.
Choosing the Backend: multiprocessing vs threading
joblib lets you switch between process-based and thread-based execution with the backend parameter. The two most common choices are loky (process-based) and threading. Here's how to use threading:
from joblib import Parallel, delayed
results = Parallel(n_jobs=2, backend='threading')(delayed(square)(i) for i in range(10))Threading avoids the pickling overhead and allows shared memory, but it does not bypass the GIL for normal Python code. Therefore, threading is only beneficial when the function releases the GIL, such as I/O operations, C extensions, or NumPy operations that release it. For pure Python CPU-bound code, processes are necessary to achieve speedup.
The following table summarizes the key differences:
| Criterion | Process-based (loky) | Thread-based |
|---|---|---|
| GIL bypass | Yes | No |
| Memory sharing | No (unless using memmap) | Yes |
| Overhead | Higher (pickling) | Lower |
| Best for | CPU-bound Python code | I/O-bound or C extensions |
You can also use the multiprocessing backend explicitly, which uses the standard library's multiprocessing.Pool. This can be more predictable if you need to control the process creation method, but loky is generally more robust and reuses processes across calls.
Managing Memory and Large Data
One of the most common problems with process-based parallelism is memory duplication. Each worker process receives a copy of the data it needs, and large arrays can be copied multiple times. joblib provides a memmap feature to share read-only data across processes without copying by using a file-backed array.
For example, if every task needs to read from a large NumPy array, you can save the array to disk and open it with mmap_mode='r' inside each task. This avoids pickling the entire array for each task. Here's a simplified pattern:
import numpy as np
from joblib import Parallel, delayed, dump, load
# Save the array to a memmap file
large_array = np.random.rand(1000, 1000)
dump(large_array, 'array.mmap')
# In each task, load the memmap and use it
def process_row(i):
data = load('array.mmap', mmap_mode='r')
return data[i].sum()
results = Parallel(n_jobs=4)(delayed(process_row)(i) for i in range(1000))This approach works because the memmap file is shared between processes, and the operating system handles memory mapping efficiently. It introduces I/O overhead, so test whether it helps. For smaller data, the pickling overhead is usually acceptable. The key is to measure your workload: if each task takes milliseconds and the arguments are a few hundred bytes, parallelization may not help. If each task takes seconds and the arguments are small, the overhead is negligible.
Error Handling and Debugging in Parallel Tasks
When a task raises an exception, joblib propagates it to the main process. By default, the exception is re-raised when the batch is computed; with return_as='generator' (available in newer versions), exceptions can be raised as you consume each result. The exact behavior can vary by version, so check the joblib documentation for your release.
A common issue is that tracebacks from worker processes are less informative because they occur in a different process. To debug, set n_jobs=1 to run sequentially and see the full traceback.
Another consideration is that functions defined inside a script or a Jupyter notebook may not be picklable by standard pickle. The loky backend uses cloudpickle, which can serialize most functions, but it is still safer to define your target function at the module level. If you encounter pickling errors, move the function to a separate module or use functools.partial with a module-level function.
If you need to handle partial failures, catch the exception in the task and return a value that carries the error, or wrap the whole Parallel call in try/except if you are willing to abandon the batch. Catching every exception inside the task without recording it can hide real errors.
When to Use joblib Instead of multiprocessing Directly
joblib's Parallel and delayed are a higher-level API for executing functions in parallel. The default loky backend is not simply a wrapper around multiprocessing.Pool, but both are process-based tools. joblib provides automatic batching, progress monitoring with the verbose parameter, and a simpler API. For many map-style use cases, this is all you need.
However, there are situations where you might want to use multiprocessing directly. If you need fine-grained control over the pool lifecycle, such as imap for lazy iteration, custom pool initializers, or an executor with concurrent.futures.as_completed, the lower-level API may be more appropriate.
The decision often comes down to the complexity of your task. For a straightforward map operation, joblib is cleaner. For complex task dependencies or dynamic scheduling, you may need a more flexible framework like concurrent.futures or asyncio.
Common Pitfalls and How to Avoid Them
One frequent mistake is creating and calling a new Parallel object inside a loop. Each Parallel(...)(...) call can start a new pool, which is expensive. Instead, reuse the same Parallel instance across multiple calls if you have many batches:
from joblib import Parallel, delayed
parallel = Parallel(n_jobs=4)
for batch in batches:
results = parallel(delayed(process)(item) for item in batch)Another pitfall is using n_jobs=-1 without considering memory. -1 uses all available CPU cores, which can lead to excessive memory consumption if each task holds a large dataset. It's often better to set n_jobs to a value that fits your memory budget, or use n_jobs=2 when the tasks are memory-intensive.
Also, be aware that the order of results is preserved by default, but if you use return_as='generator', the order is not guaranteed. If you need results in the original order, stick with the default list behavior.
Finally, when running on Windows, the process creation method is spawn, which means the main module is imported in each worker. This can cause issues if you have side effects at module level. Always guard your main execution with if __name__ == '__main__': to avoid infinite recursion and unexpected behavior.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.