Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Using joblib Parallel and delayed with multiprocessing

Learn how to parallelize Python loops with joblib's Parallel and delayed, how they relate to multiprocessing, and how to avoid common pitfalls.

~6 min readPython
#joblib#parallel processing#multiprocessing#Python performance
Diagram showing a Python loop being split into parallel tasks across multiple processes using joblib.

Parallelizing a loop with joblib usually means creating a sequence of delayed calls and passing it to Parallel. This article explains the core pattern, describes how joblib's default process-based backend relates to Python's multiprocessing, and covers practical decisions about backends, memory, and debugging.

When you need to parallelize a loop in Python, joblib's Parallel and delayed are often the quickest way to get started. They hide the details of process pools and task scheduling, but you still need to understand how they relate to multiprocessing when you move from a toy example to a real workload.

The Core Pattern: Parallel and delayed

The typical usage is a two-step process: wrap a function call with delayed, then pass a list of these delayed calls to Parallel. Here's a minimal example that computes the square of each number in a list:

python
from joblib import Parallel, delayed def square(x): return x * x results = Parallel(n_jobs=2)(delayed(square)(i) for i in range(10)) print(results)

The generator expression (delayed(square)(i) for i in range(10)) creates a series of delayed tasks. Parallel then distributes these tasks across the specified number of jobs. The result is a list containing the return values in the same order as the input, regardless of which worker finished first.

This pattern works because delayed captures the function and its arguments, and Parallel decides how to execute them. You can also use it with functions that take multiple arguments or keyword arguments:

python
def add(a, b=0): return a + b results = Parallel(n_jobs=2)(delayed(add)(i, b=10) for i in range(5))

How joblib Uses Multiprocessing Under the Hood

By default, joblib uses the loky backend, a process-based executor that manages a pool of worker processes. Each job runs in a separate Python process, not a thread. Worker processes receive the function and its arguments through cloudpickle, an extended pickling mechanism, and send results back the same way.

Because each worker is a separate process, CPU-bound Python code can run in parallel without being limited by the Global Interpreter Lock (GIL). However, the serialization step adds overhead. For simple functions with small arguments, the cost of pickling can dominate the benefit of parallel execution, especially when the function itself is fast.

The loky backend also handles the re-importation of the main module in each worker. This matters on platforms that use the spawn process start method, such as Windows. You should still guard the executable part of a script with if __name__ == '__main__': to avoid side effects and infinite recursion.

Choosing the Backend: multiprocessing vs threading

joblib lets you switch between process-based and thread-based execution with the backend parameter. The two most common choices are loky (process-based) and threading. Here's how to use threading:

python
from joblib import Parallel, delayed results = Parallel(n_jobs=2, backend='threading')(delayed(square)(i) for i in range(10))

Threading avoids the pickling overhead and allows shared memory, but it does not bypass the GIL for normal Python code. Therefore, threading is only beneficial when the function releases the GIL, such as I/O operations, C extensions, or NumPy operations that release it. For pure Python CPU-bound code, processes are necessary to achieve speedup.

The following table summarizes the key differences:

CriterionProcess-based (loky)Thread-based
GIL bypassYesNo
Memory sharingNo (unless using memmap)Yes
OverheadHigher (pickling)Lower
Best forCPU-bound Python codeI/O-bound or C extensions

You can also use the multiprocessing backend explicitly, which uses the standard library's multiprocessing.Pool. This can be more predictable if you need to control the process creation method, but loky is generally more robust and reuses processes across calls.

Managing Memory and Large Data

One of the most common problems with process-based parallelism is memory duplication. Each worker process receives a copy of the data it needs, and large arrays can be copied multiple times. joblib provides a memmap feature to share read-only data across processes without copying by using a file-backed array.

For example, if every task needs to read from a large NumPy array, you can save the array to disk and open it with mmap_mode='r' inside each task. This avoids pickling the entire array for each task. Here's a simplified pattern:

python
import numpy as np from joblib import Parallel, delayed, dump, load # Save the array to a memmap file large_array = np.random.rand(1000, 1000) dump(large_array, 'array.mmap') # In each task, load the memmap and use it def process_row(i): data = load('array.mmap', mmap_mode='r') return data[i].sum() results = Parallel(n_jobs=4)(delayed(process_row)(i) for i in range(1000))

This approach works because the memmap file is shared between processes, and the operating system handles memory mapping efficiently. It introduces I/O overhead, so test whether it helps. For smaller data, the pickling overhead is usually acceptable. The key is to measure your workload: if each task takes milliseconds and the arguments are a few hundred bytes, parallelization may not help. If each task takes seconds and the arguments are small, the overhead is negligible.

Error Handling and Debugging in Parallel Tasks

When a task raises an exception, joblib propagates it to the main process. By default, the exception is re-raised when the batch is computed; with return_as='generator' (available in newer versions), exceptions can be raised as you consume each result. The exact behavior can vary by version, so check the joblib documentation for your release.

A common issue is that tracebacks from worker processes are less informative because they occur in a different process. To debug, set n_jobs=1 to run sequentially and see the full traceback.

Another consideration is that functions defined inside a script or a Jupyter notebook may not be picklable by standard pickle. The loky backend uses cloudpickle, which can serialize most functions, but it is still safer to define your target function at the module level. If you encounter pickling errors, move the function to a separate module or use functools.partial with a module-level function.

If you need to handle partial failures, catch the exception in the task and return a value that carries the error, or wrap the whole Parallel call in try/except if you are willing to abandon the batch. Catching every exception inside the task without recording it can hide real errors.

When to Use joblib Instead of multiprocessing Directly

joblib's Parallel and delayed are a higher-level API for executing functions in parallel. The default loky backend is not simply a wrapper around multiprocessing.Pool, but both are process-based tools. joblib provides automatic batching, progress monitoring with the verbose parameter, and a simpler API. For many map-style use cases, this is all you need.

However, there are situations where you might want to use multiprocessing directly. If you need fine-grained control over the pool lifecycle, such as imap for lazy iteration, custom pool initializers, or an executor with concurrent.futures.as_completed, the lower-level API may be more appropriate.

The decision often comes down to the complexity of your task. For a straightforward map operation, joblib is cleaner. For complex task dependencies or dynamic scheduling, you may need a more flexible framework like concurrent.futures or asyncio.

Common Pitfalls and How to Avoid Them

One frequent mistake is creating and calling a new Parallel object inside a loop. Each Parallel(...)(...) call can start a new pool, which is expensive. Instead, reuse the same Parallel instance across multiple calls if you have many batches:

python
from joblib import Parallel, delayed parallel = Parallel(n_jobs=4) for batch in batches: results = parallel(delayed(process)(item) for item in batch)

Another pitfall is using n_jobs=-1 without considering memory. -1 uses all available CPU cores, which can lead to excessive memory consumption if each task holds a large dataset. It's often better to set n_jobs to a value that fits your memory budget, or use n_jobs=2 when the tasks are memory-intensive.

Also, be aware that the order of results is preserved by default, but if you use return_as='generator', the order is not guaranteed. If you need results in the original order, stick with the default list behavior.

Finally, when running on Windows, the process creation method is spawn, which means the main module is imported in each worker. This can cause issues if you have side effects at module level. Always guard your main execution with if __name__ == '__main__': to avoid infinite recursion and unexpected behavior.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.