Python Iterator vs Generator: Key Differences

When developers compare iterators and generators in Python, the distinction often comes down to how each implements lazy iteration. Both produce sequences of values on demand, but they differ in syntax, state handling, and the amount of code required. This article explains the practical differences and the conditions that should drive your choice.

The Iterator Protocol: What Makes an Object Iterable

An iterator in Python is any object that implements the iterator protocol, which consists of two methods: __iter__() and __next__(). The __iter__() method returns the iterator object itself, and __next__() returns the next value in the sequence. When no more values are available, __next__() raises StopIteration.

class Counter:
    def __init__(self, limit):
        self.limit = limit
        self.value = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.value >= self.limit:
            raise StopIteration
        current = self.value
        self.value += 1
        return current

for number in Counter(3):
    print(number)

This custom iterator keeps its state in instance attributes. Each call to next() advances the internal counter until the limit is reached. The for loop implicitly calls iter() on the object and then repeatedly calls next() until StopIteration is raised.

Generator Functions: Iteration Built on yield

A generator function is a function that uses the yield keyword instead of return. When called, it returns a generator object without executing the function body immediately. Execution starts on the first call to next() and pauses at each yield, preserving the local state.

def counter(limit):
    value = 0
    while value < limit:
        yield value
        value += 1

for number in counter(3):
    print(number)

The generator above produces the same sequence as the custom iterator, but the code is shorter and the state is stored in local variables rather than instance attributes. The generator object itself is an iterator, so it implements __iter__() and __next__() automatically.

Key Differences Between an Iterator and a Generator

The most direct way to understand the relationship is to compare them side by side.

A generator is a convenient way to create an iterator, but not every iterator is a generator. If you need complex state transitions or multiple methods beyond iteration, a custom iterator class gives you more control. If you only need to produce a sequence of values, a generator is usually simpler.

Lazy Evaluation and Memory Behavior

The iterator protocol and generator functions both support lazy evaluation: they produce one value at a time and do not store the entire sequence in memory. This is the primary reason to use either approach when working with large datasets.

def read_lines(file_path):
    with open(file_path) as file:
        for line in file:
            yield line.strip()

This generator reads a file line by line, so the memory footprint stays constant regardless of file size. A custom iterator could achieve the same behavior, but the generator version is more readable and less error-prone because the with block manages the file lifecycle.

A generator expression offers the same laziness in a compact form:

squares = (x * x for x in range(1000000))

This does not create a list of a million squares. It creates a generator that computes each square only when requested. The equivalent list comprehension would allocate memory for all values at once.

When to Write a Custom Iterator Instead of a Generator

Generators are not always the best fit. A custom iterator is preferable when you need to implement additional methods or maintain complex internal state that is easier to express as instance attributes. For example, an iterator that supports resetting its position or exposing metadata about the iteration may be clearer as a class.

class ResettableCounter:
    def __init__(self, limit):
        self.limit = limit
        self.value = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.value >= self.limit:
            raise StopIteration
        current = self.value
        self.value += 1
        return current

    def reset(self):
        self.value = 0

Here the reset() method is part of the iterator's public interface. A generator cannot expose extra methods without wrapping it in a class, which defeats the purpose. If you need to pass the iterator to code that expects the full iterator protocol, a custom class is also more explicit.

Common Pitfalls With Iterators and Generators

One frequent mistake is assuming that an iterator can be reused. Both iterators and generators are single-use. Once exhausted, calling next() raises StopIteration permanently. If you need to iterate multiple times, you must create a new iterator or generator.

gen = (x for x in range(3))
print(list(gen))  # [0, 1, 2]
print(list(gen))  # []

Another pitfall is mixing up iterables and iterators. A list is iterable but not an iterator; it does not have a __next__() method. Calling next() on a list raises TypeError. You must call iter() first to obtain an iterator.

Generators also have a subtle behavior: they are single-pass and cannot be indexed. If you need random access to a sequence, convert it to a list first, but be aware that this defeats the memory advantage.

Maintainability and Performance Considerations

Generators avoid the boilerplate of defining a class and managing instance attributes. The interpreter handles state suspension and resumption internally, which makes simple sequence generation concise and less error-prone. Custom iterators are useful when the iteration logic benefits from splitting it into methods, such as exposing additional state or behavior.

Performance depends on the Python implementation, version, and workload, so avoid choosing between iterators and generators based on micro-benchmarks. In most application code, the main consideration is whether a generator expresses the behavior clearly. If iteration is the bottleneck, profile that specific path before optimizing.