Python Executor.map: Parallel Mapping Explained

Python's concurrent.futures.Executor.map provides an ordered, iterator-based way to apply a function to many inputs in parallel. It submits calls to a ThreadPoolExecutor or ProcessPoolExecutor and returns results in the same order as the input iterable. This article covers its syntax, behavior, exception handling, and performance tradeoffs.

How Executor.map Differs from Built-in map

The built-in map applies a function to each element of an iterable synchronously and returns an iterator. Executor.map keeps that ordered, iterator-based interface but schedules each call on a pool of workers. The signature is:

Executor.map(func, *iterables, timeout=None, chunksize=1)

For each element in the iterable, func is called, and the results are yielded in the same order as the input. The timeout parameter controls how long to wait for each result. If a call exceeds the timeout, a TimeoutError is raised. The chunksize parameter is only used by ProcessPoolExecutor; it controls the size of the batches sent to each worker, reducing pickling and communication overhead.

Here is a minimal example using ThreadPoolExecutor:

from concurrent.futures import ThreadPoolExecutor

def square(x):
    return x * x

with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(square, range(10)))

print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The with block ensures the executor is shut down after the work completes. The map call returns an iterator, so converting it to a list materializes all results.

Choosing Between ThreadPoolExecutor and ProcessPoolExecutor

The two concrete executor classes in concurrent.futures are ThreadPoolExecutor and ProcessPoolExecutor. The choice depends on the nature of the workload.

If your function spends most of its time waiting on network calls, file reads, or database queries, ThreadPoolExecutor is usually sufficient and lighter. For pure computation that uses the CPU heavily, ProcessPoolExecutor can achieve true parallelism across multiple cores, but it incurs serialization overhead because inputs and outputs must be pickled between processes.

Controlling Concurrency and Ordering

Executor.map guarantees that results are yielded in the same order as the input iterable. This is convenient when you need to associate results with their original inputs. However, it also means that if one item takes much longer than others, the iterator will block until that result is ready, even if later items have already completed.

To get results as soon as they are available, use executor.submit with as_completed instead. That approach returns futures that can be iterated as they complete, but you lose the input-order guarantee.

The max_workers parameter controls the number of threads or processes in the pool. Setting it too high can cause resource contention, while setting it too low may underutilize available cores. A common heuristic for ProcessPoolExecutor is to set max_workers to the number of CPU cores, but this depends on the workload and memory constraints.

For ProcessPoolExecutor, the chunksize parameter is important for large iterables. Instead of sending each item individually, the executor groups items into chunks and sends them to workers in batches. This reduces pickling overhead and can significantly speed up processing. The default is 1, which sends one item at a time. For CPU-bound tasks, a larger chunksize can improve throughput by reducing pickling overhead, but it also increases memory usage and can make work distribution less even.

Handling Exceptions in Worker Functions

When a function passed to Executor.map raises an exception, the exception is not raised immediately. Instead, it is raised when you iterate over the results. This means you can catch exceptions at the point where you consume the iterator.

from concurrent.futures import ThreadPoolExecutor

def risky(x):
    if x == 3:
        raise ValueError('bad input')
    return x * 2

with ThreadPoolExecutor(max_workers=2) as executor:
    try:
        results = list(executor.map(risky, range(5)))
    except ValueError as e:
        print(f'Caught: {e}')

If you need to handle exceptions per item without stopping the entire iteration, you can wrap the function in a try/except inside the worker, or use executor.submit and inspect each future individually. The map method is not designed for fine-grained error recovery; it stops at the first exception by default.

Performance Considerations and Overhead

The primary performance benefit of Executor.map is that it parallelizes independent function calls. However, the overhead of managing a pool and, for processes, serializing data can outweigh the benefits for very small tasks. For example, calling a trivial function on a list of 10 integers may be slower with a process pool than with a simple loop because the cost of spawning processes and pickling data dominates.

For I/O-bound tasks, threads are cheap, and ThreadPoolExecutor can dramatically reduce wall-clock time when many operations are waiting on external resources. For CPU-bound tasks, ProcessPoolExecutor can use multiple cores in CPython, but you must be mindful of the chunksize and the amount of data passed between processes.

Memory usage is another concern. Each process in a ProcessPoolExecutor has its own Python interpreter and memory space. If your input iterable is large, passing it to the pool requires pickling, which can be expensive. In contrast, threads share memory, so no serialization is needed, but the GIL prevents true parallelism for CPU-bound code.

Practical Example: Parallel Web Scraping

A common use case for Executor.map is fetching multiple URLs concurrently. Here is an example using ThreadPoolExecutor and the requests library:

import requests
from concurrent.futures import ThreadPoolExecutor

def fetch_url(url):
    response = requests.get(url)
    return response.status_code, len(response.content)

urls = [
    'https://example.com',
    'https://httpbin.org/get',
    'https://jsonplaceholder.typicode.com/posts',
]

with ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(fetch_url, urls))

for url, (status, size) in zip(urls, results):
    print(f'{url}: {status}, {size} bytes')

The results are paired with the original URLs because map preserves order. This pattern works well for I/O-bound tasks where the GIL is not a bottleneck.

Avoiding Common Pitfalls with Executor.map

One common point of confusion is that Executor.map consumes the input iterable in the calling thread while building the list of submitted futures. If you pass a generator, it is exhausted up front, not advanced by the workers. As a result, you do not need to protect the generator from concurrent access, but you should be aware that map is not a streaming API: for very large or infinite lazy iterables, you may want to process batches manually or switch to executor.submit in a loop.

Another pitfall is relying on side effects inside the mapped function. With ThreadPoolExecutor, shared state must be protected with locks because multiple threads mutate the same objects. With ProcessPoolExecutor, each worker has its own memory, so updates to global variables are not shared between processes and cannot be used to communicate results.

Finally, be careful with the timeout parameter. It applies to each result retrieval, not to the entire map call. If a single function call takes longer than the timeout, a TimeoutError is raised when you try to fetch that result. This can be useful for bounding the execution time of each item, but it does not cancel the underlying work; the function continues running in the background.

When you need to cancel tasks or handle results out of order, executor.submit with as_completed gives you more control. For many batch-processing scenarios, however, Executor.map provides a clean, readable way to parallelize a function across an iterable with minimal code.