Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Iterator One-Time Use: Why Iterators Can't Be Reused

Python iterators are one-time use objects. Learn why that matters, how to create fresh iterators, and when to use lists or itertools.tee.

~4 min readPython
#iterators#generators#itertools#memory-management
Illustration of a Python iterator being exhausted after one pass, with a fresh iterator created from an iterable.

Python iterators are one-time use objects: once you consume an iterator, it is exhausted and cannot be reused. This behavior is by design. It follows directly from how the iterator protocol works, and it affects how you structure loops, functions, and data pipelines.

What Makes an Iterator One-Time Use

An iterator in Python is an object with a __next__() method that returns the next item or raises StopIteration when there are no more items. It also has an __iter__() method that returns itself, so it can be used in a for loop. A for loop calls iter() on the object and then calls next() internally until it catches StopIteration. Once that exception is raised, the iterator is permanently exhausted. There is no built-in "rewind" mechanism.

Consider a simple list iterator:

python
numbers = [1, 2, 3] it = iter(numbers) print(next(it)) # 1 print(next(it)) # 2 print(next(it)) # 3 # next(it) would raise StopIteration

After the third next(), it is done. Calling next(it) again raises StopIteration. The iterator does not reset.

The Problem with Reusing an Iterator

The one-time nature becomes a problem when you try to iterate over the same iterator twice. For example:

python
it = iter([1, 2, 3]) for x in it: print(x) # Second loop produces nothing for x in it: print(x)

The first loop consumes all items. The second loop sees an exhausted iterator and immediately stops. This is a common source of bugs, especially when an iterator is passed to a function that consumes it and then the caller tries to use it again.

Creating a Fresh Iterator for Each Pass

The solution is to create a new iterator for each pass. For iterable objects like lists, tuples, and strings, you can call iter() again. For generators, you call the generator function again to get a fresh generator.

python
def generate_numbers(): yield 1 yield 2 yield 3 for x in generate_numbers(): print(x) # Works again because generate_numbers() returns a new generator for x in generate_numbers(): print(x)

The key is to distinguish between an iterable (something you can call iter() on) and an iterator (the result of that call). Lists are iterables, not iterators. Generators are both iterable and iterator, but calling the generator function creates a new generator each time.

When to Materialize an Iterator into a List

If you need to iterate over the same data multiple times and the data set is small, converting the iterator to a list is the simplest approach.

python
data = list(generate_numbers()) for x in data: print(x) for x in data: print(x)

This stores all items in memory, which is fine for small data. For large data, materializing can be expensive in terms of memory. Use this approach when you know the data fits comfortably in memory and you need random access or multiple passes.

Using itertools.tee to Duplicate an Iterator

When you want to avoid materializing the entire sequence but still need multiple iterators, itertools.tee can help. It creates independent iterators from a single source iterator.

python
from itertools import tee it = iter([1, 2, 3, 4]) it1, it2 = tee(it, 2) print(next(it1)) # 1 print(next(it2)) # 1 print(next(it1)) # 2

tee buffers items as they are consumed from the original iterator. The memory cost is proportional to the difference in consumption between the derived iterators. If one iterator is consumed far ahead of the other, the buffer grows. This is a tradeoff between memory and avoiding materializing the entire sequence.

After calling tee, do not continue consuming the original iterator it; the derived iterators will not see items consumed directly from the original source. Use the derived iterators instead.

Practical Patterns for One-Time Iterators

A common design pattern is to have functions accept iterables rather than iterators. This allows the caller to decide whether to pass a list, a generator, or any other iterable, and the function can call iter() on it each time it needs to iterate.

python
def process(data): for item in data: print(item) # Passing a list works, and the list can be reused later my_list = [1, 2, 3] process(my_list) process(my_list) # works because my_list is an iterable, not an iterator

If you must pass an iterator, be aware that it will be consumed. If the function needs to iterate multiple times, it should accept an iterable and create its own iterator internally.

Performance and Memory Considerations

The choice between re-creating iterators, materializing to a list, or using tee depends on your data size and access pattern.

  • Re-creating an iterator from an iterable is cheap if the underlying data is already in memory (e.g., a list). For generators, re-calling the function may involve re-computing values, which could be expensive if the generator does significant work.
  • Materializing to a list uses O(n) memory but gives you random access and the ability to iterate any number of times without recomputation.
  • tee uses memory proportional to the consumption difference, which is often much less than O(n) if the iterators are consumed roughly in sync. However, it does not give random access, and it can still be memory-heavy if one iterator lags far behind.

For large data streams, prefer re-creating iterators when possible, or use tee with careful consumption order. Avoid materializing huge datasets into lists unless you have no alternative.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.