Python Iterator Exhaustion: Why Iterators Are Single-Use
Learn why Python iterators become exhausted, how to detect and handle exhaustion, and practical patterns for reusing or re-creating iterators.

When you iterate over a Python iterator a second time, you may be surprised to get no results. This is not a bug; it is the iterator protocol working as designed. Python iterator exhaustion means that once an iterator has yielded all its values, it cannot be restarted. Understanding this behavior is essential for writing correct code, especially when passing iterators to functions or looping over the same data source multiple times.
What Happens When an Iterator Is Exhausted?
Every iterator in Python follows the iterator protocol: it implements __iter__() and __next__(). Calling next() on an iterator repeatedly returns successive values until the iterator is exhausted. At that point, next() raises StopIteration. A for loop catches this exception internally and exits cleanly, but any direct call to next() will propagate it.
numbers = iter([1, 2, 3])
print(next(numbers)) # 1
print(next(numbers)) # 2
print(next(numbers)) # 3
# The next call raises StopIterationOnce StopIteration has been raised, the iterator is permanently exhausted. Calling next() again will raise StopIteration again. This is true for all iterators, including list iterators, generator expressions, and generator objects.
Why Iterators Are Single-Pass by Design
The single-pass nature is a deliberate tradeoff. Iterators are lazy: they produce values on demand and do not store the entire sequence in memory. This is what allows processing of infinite streams or files larger than RAM. For example, a generator can yield lines from a huge log file without loading the whole file into memory.
def read_lines(path):
with open(path) as f:
for line in f:
yield lineOnce a generator has been consumed, it cannot be rewound. The state is gone. This design keeps memory usage low but requires the developer to plan for multiple passes explicitly.
Detecting Exhaustion Without Catching StopIteration
Sometimes you need to know whether an iterator still has values without triggering an exception. The next() function accepts a default argument that is returned instead of raising StopIteration.
it = iter([1, 2])
print(next(it, None)) # 1
print(next(it, None)) # 2
print(next(it, None)) # NoneThis pattern is useful when you want to peek at the first element or check if an iterator is empty. However, it consumes the value you peek at. If you need to keep that value, you must store it separately.
Another approach uses the two-argument form of iter() with a sentinel. This is handy for building an iterator that stops when a callable returns a specific value.
with open('data.txt') as f:
for block in iter(lambda: f.read(64), ''):
process(block)Here the iterator calls the callable until it returns the sentinel '', which marks the end. This is a clean way to build an iterator that stops at a custom condition.
Reusing an Iterator: Recreating vs. Copying
Because iterators cannot be reset, the only reliable way to iterate twice is to create a fresh iterator from the original data source. If the source is a list, you can call iter() again. If it is a generator function, you can call the function again. But if you have a generator object that has already been consumed, you cannot recover its values.
def generate():
yield 1
yield 2
gen = generate()
print(list(gen)) # [1, 2]
print(list(gen)) # [] because gen is exhausted
# To iterate twice, create a new generator:
gen2 = generate()
print(list(gen2)) # [1, 2]For iterators created from a list, you can create a fresh iterator from the original list if you still have it. But if you only have an iterator object (e.g., from map() or filter()), you cannot re-create it without the original arguments.
Practical Patterns for Multiple Passes
When you need to iterate over the same sequence more than once, you have several options. The simplest is to materialize the iterator into a list, but that defeats the memory advantage. For large or infinite sequences, itertools.tee() is a standard tool when you need multiple independent iterators.
from itertools import tee
def process(iterable):
it1, it2 = tee(iterable, 2)
# Now it1 and it2 are independent iterators
for x in it1:
pass
for y in it2:
passtee() clones the iterator into multiple independent iterators. However, it does so by buffering values that have been consumed from one clone but not the other. If you advance one clone far ahead, the buffer grows. This can be memory-intensive, so tee() is best used when the clones are consumed roughly in parallel.
Another pattern is to wrap the data source in an iterable class whose __iter__() returns a fresh iterator on demand. This is useful when the underlying data can be re-read, such as a file or a database cursor.
class RepeatableIterable:
def __init__(self, data_factory):
self.data_factory = data_factory
def __iter__(self):
return self.data_factory()This lets you create a new iterator on demand by calling the factory function each time __iter__ is invoked.
Common Pitfalls with Exhausted Iterators
A frequent mistake is passing the same iterator to multiple functions that each iterate over it. The first function consumes the iterator, and the second receives an empty iterator.
def sum_and_count(numbers):
total = sum(numbers)
count = len(list(numbers)) # numbers is already exhausted
return total, countThe count will always be zero because sum() consumed the iterator. To fix this, either pass a sequence like a list, or use tee() to create two independent iterators.
Another pitfall is assuming that a generator can be reused after a for loop. Once the loop completes, the generator is exhausted. If you need to loop twice, create a new generator object or use itertools.tee().
Performance and Memory Tradeoffs
Choosing between an iterator and a list affects memory and latency. An iterator computes values lazily, so it uses minimal memory and can start producing results immediately. A list stores all values, which may be necessary if you need random access or multiple passes without re-computation.
itertools.tee() sits between these extremes. It avoids storing the entire sequence, but it does store the difference between the fastest and slowest clone. If one clone is consumed completely before the other starts, tee() will buffer everything, effectively becoming a list in memory. Understanding this behavior helps you decide when to use tee() versus a simple list.
For most small datasets, converting to a list is the simplest and most readable approach. For large or infinite streams, you must design your code to avoid multiple passes, or use tee() with careful consumption patterns. The key is to know whether your data source can produce a fresh iterator; if it cannot, you need to buffer or re-architect.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.