Python itertools: Efficient Iteration and Combinations
Learn how to use python itertools to create efficient iterators, generate combinations, and process data streams without loading everything into memory.

The Python itertools module is a standard library collection of functions that create and manipulate iterators. It is designed to be fast, memory-efficient, and composable, making it a practical choice for data processing pipelines that would otherwise require manual loop logic or eager list construction.
What Is itertools and Why It Matters
Iterators in Python are objects that yield values one at a time. The itertools module builds on this by providing building blocks that consume and produce iterators lazily. Instead of building a list of results in memory, each function returns an iterator that computes values on demand. This lazy evaluation is the core reason itertools is often used in scenarios where input data is large, infinite, or generated on the fly.
For example, consider the difference between range(1000000) and list(range(1000000)). The first is a lazy iterable that produces numbers as needed; the second allocates a list of one million integers. itertools functions behave like range in that respect, which is why they are essential for writing scalable Python code.
Core Iterator Functions: count, cycle, repeat
Three functions in itertools cover common cases for counters, cycles, and repeated values: count, cycle, and repeat. They are useful for creating counters, cycling through a finite set, or repeating a value a fixed number of times.
from itertools import count, cycle, repeat
# count(start, step) yields an infinite arithmetic sequence
for i in count(10, 2):
if i > 20:
break
print(i) # 10, 12, 14, 16, 18, 20
# cycle(iterable) repeats the iterable forever
colors = cycle(['red', 'green', 'blue'])
for _ in range(6):
print(next(colors)) # red, green, blue, red, green, blue
# repeat(object, times) yields the object a fixed number of times
for value in repeat('A', 3):
print(value) # A, A, Acount is often paired with zip to add an index to an iterable, similar to enumerate but with more control over the starting value and step. cycle is handy for round-robin scheduling or assigning tasks to a fixed set of workers. repeat is useful when you need a constant value repeated a specific number of times without constructing a list.
Combining Iterators: chain, zip_longest, product
When you need to treat multiple iterables as a single sequence, chain is the tool. It concatenates iterables without creating a new list. zip_longest combines iterables element-wise, filling missing values with a specified default. product computes the Cartesian product of input iterables.
from itertools import chain, zip_longest, product
# chain: flatten multiple iterables
combined = chain([1, 2], ['a', 'b'], [3])
print(list(combined)) # [1, 2, 'a', 'b', 3]
# zip_longest: zip with padding
for a, b in zip_longest([1, 2], ['x'], fillvalue='-'):
print(a, b) # (1, 'x'), (2, '-')
# product: Cartesian product
for pair in product([1, 2], ['a', 'b']):
print(pair) # (1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')chain is often used to process flattened data from multiple sources without copying. zip_longest is valuable when aligning data of unequal lengths, such as merging time series with different timestamps. product is the basis for nested loops when the number of dimensions is not known in advance.
Combinations and Permutations
The itertools module provides three functions for generating combinations and permutations: combinations, combinations_with_replacement, and permutations. These are essential for solving problems in combinatorics, testing, and brute-force search.
from itertools import combinations, permutations, combinations_with_replacement
team = ['A', 'B', 'C']
# combinations: order does not matter, no repetition
print(list(combinations(team, 2))) # [('A', 'B'), ('A', 'C'), ('B', 'C')]
# permutations: order matters, no repetition
print(list(permutations(team, 2))) # [('A', 'B'), ('A', 'C'), ('B', 'A'), ('B', 'C'), ('C', 'A'), ('C', 'B')]
# combinations_with_replacement: order does not matter, repetition allowed
print(list(combinations_with_replacement(team, 2))) # [('A', 'A'), ('A', 'B'), ('A', 'C'), ('B', 'B'), ('B', 'C'), ('C', 'C')]These functions return iterators, so they do not precompute all results. This is critical when dealing with large input sets; the number of possible selections grows quickly, and generating them eagerly would exhaust memory. Use them when you need to iterate over all possible selections without materializing the full list.
Grouping and Accumulating: groupby, accumulate
groupby groups consecutive elements of an iterable based on a key function. It is important to note that groupby only groups adjacent items; the input must be sorted by the same key for meaningful grouping. accumulate applies a binary function cumulatively to the elements, producing running totals or other cumulative results.
from itertools import groupby, accumulate
data = [('apple', 3), ('banana', 2), ('apple', 1), ('banana', 5)]
# Sort by the first element to group all apples together
sorted_data = sorted(data, key=lambda x: x[0])
for key, group in groupby(sorted_data, key=lambda x: x[0]):
print(key, [item[1] for item in group])
# Output:
# apple [3, 1]
# banana [2, 5]
# accumulate: running sum by default
print(list(accumulate([1, 2, 3, 4]))) # [1, 3, 6, 10]
# accumulate with a custom function
print(list(accumulate([1, 2, 3, 4], lambda a, b: a * b))) # [1, 2, 6, 24]groupby is a powerful alternative to manual grouping in loops, but it requires sorted input. accumulate is useful for computing running statistics, prefix sums, or factorial-like sequences without a loop.
Performance and Memory Considerations
The primary performance benefit of itertools is its lazy evaluation. Each function returns an iterator that computes values only when next() is called. This avoids the memory overhead of building intermediate lists and can reduce the time spent on allocation and garbage collection.
For example, list(combinations(range(100), 3)) creates a list of 161,700 tuples, consuming memory proportional to the output. In contrast, iterating over combinations(range(100), 3) directly processes each tuple and discards it, keeping memory usage constant. This distinction matters when the input size is large or when the results are consumed by a streaming process.
Another performance consideration is that itertools functions are implemented in C, so they are often faster than equivalent Python loops. However, the actual speedup depends on the specific operation and the size of the data. The main advantage is that you avoid the overhead of Python-level iteration for the combinatorial logic itself.
When combining multiple itertools functions, lazy evaluation lets you build pipelines without intermediate storage. For instance, zip(count(), data) adds an index as data is consumed, instead of building a list of indexed values first. One exception is sorting before groupby: sorted() returns a list, so that step necessarily materializes the input before grouping.
Common Pitfalls and How to Avoid Them
One frequent mistake is assuming that groupby works on unsorted data. Because it groups only consecutive equal keys, you must sort the input by the same key first. Otherwise, you will get multiple groups for the same key, which is rarely the intended behavior.
Another pitfall is consuming an iterator multiple times. Iterators are single-use; after you iterate through it, it is exhausted. If you need to reuse the sequence, either convert it to a list or re-create the iterator. This is especially relevant when using itertools functions in a loop or passing them to multiple consumers.
A third issue is forgetting that product, permutations, and combinations produce tuples, not lists. Tuples are hashable, so they can be used as dictionary keys, but they are immutable. If you need mutable sequences, you must convert each tuple to a list, which adds overhead.
Finally, be cautious with infinite iterators like count and cycle. They never stop on their own, so you must provide a termination condition, such as a break statement or a takewhile from itertools. Without that, your program will hang or consume unbounded CPU.
Choosing Between itertools and Custom Generators
While itertools covers many common patterns, there are cases where a custom generator function is more readable or flexible. For example, if you need to implement a complex stateful iteration that does not map to a single itertools function, a generator with yield allows you to express the logic directly.
Use itertools when the operation is a standard combination, permutation, grouping, or accumulation, and when the input is large enough that lazy evaluation matters. Use a custom generator when you need to combine multiple operations in a way that itertools does not directly support, or when the logic is clearer with explicit yield statements.
A good rule of thumb: if you find yourself writing nested loops to generate combinations or to process adjacent pairs, itertools likely has a function that expresses the intent more clearly and with better performance. If you are implementing a state machine or a streaming algorithm that does not fit a standard pattern, a custom generator gives you full control.
Both approaches are valid, and they can be combined. You can use itertools inside a custom generator to handle a sub-step, or wrap an itertools iterator with additional logic in a generator. The key is to choose the tool that makes the code maintainable and efficient for the specific task.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.