Python Generator Expression: Lazy Iteration Explained
Learn how python generator expressions create memory-efficient iterators, when to use them over list comprehensions, and how to avoid common pitfalls.

When you need to process a sequence of values without holding all of them in memory at once, a Python generator expression is a compact way to create an iterator. The syntax looks like a list comprehension, but with parentheses instead of square brackets. For example, (x * x for x in range(10)) produces a generator that yields squares one at a time. The key difference is that a list comprehension builds the entire list immediately, while a generator expression computes each value only when requested.
Syntax and Basic Behavior
A generator expression is defined with round brackets and a for clause, optionally followed by if conditions. The result is a generator object that implements the iterator protocol. You can consume it with a for loop, or pass it to functions that accept iterables.
squares = (x * x for x in range(5))
for value in squares:
print(value)This prints 0, 1, 4, 9, 16. The generator object squares is not a list; it does not contain the values. Instead, it holds the expression and the iteration state. Each call to next() computes the next value and advances the internal counter.
Lazy Evaluation and the Iterator Protocol
Generator expressions rely on lazy evaluation. The expression that produces each value is not evaluated until next() is called. This behavior is what makes them memory-efficient for large or infinite sequences. The iterator protocol requires two methods: __iter__ returning the iterator itself, and __next__ returning the next value or raising StopIteration. Generator expressions implement both automatically.
Consider a generator over a large range:
def process_values(iterable):
for item in iterable:
# do something
pass
large_gen = (i for i in range(10**9))
process_values(large_gen)The generator does not allocate a billion integers. It produces one integer at a time, keeping memory usage constant regardless of the range size. This is the primary advantage over a list comprehension, which would attempt to create a list of one billion items and likely exhaust memory.
Generator Expression vs List Comprehension
The choice between a generator expression and a list comprehension depends on whether you need the entire sequence at once. A list comprehension returns a list, which supports indexing, slicing, and repeated iteration. A generator expression returns an iterator that can only be traversed once.
| Feature | List Comprehension | Generator Expression |
|---|---|---|
| Syntax | [expr for ...] | (expr for ...) |
| Result | List | Generator object |
| Memory usage | Stores all elements | Produces one element at a time |
| Reusable | Yes | No (one pass) |
| Indexing / slicing | Supported | Not supported |
| Creation time | Eager | Lazy |
If you only need to iterate once and the sequence is large, a generator expression is usually the better choice. If you need random access or must iterate multiple times, a list comprehension is more appropriate. For small sequences, the difference is negligible, and readability should guide the decision.
Using Generator Expressions in Function Calls
Generator expressions are often passed directly to functions that consume iterables, such as sum, max, min, any, and all. This avoids creating an intermediate list and can reduce memory overhead.
total = sum(x * x for x in range(1000))
has_even = any(x % 2 == 0 for x in numbers)In these calls, the generator expression is evaluated lazily, and the function consumes it. For sum, each value is added to the running total without storing the entire sequence. This pattern is idiomatic and often clearer than building a list first.
One subtlety is that a generator expression can omit its extra parentheses only when it is the sole argument to a function. If the function call needs other arguments, wrap the generator expression in its own parentheses.
Chaining and Combining Generators
Generator expressions can be chained, meaning you can use one generator as the input to another. This allows you to build pipelines that process data in stages without creating intermediate collections.
values = (int(line.strip()) for line in open('data.txt'))
filtered = (v for v in values if v > 0)
squared = (v * v for v in filtered)Each generator here is lazy. Reading a line from the file, converting it to an integer, filtering, and squaring all happen one item at a time. This is memory-efficient even for large files. However, note that each generator is a separate object, and they are consumed in order. If you need to reuse the data, you must recreate the chain or materialize it into a list.
Common Pitfalls and Edge Cases
One common mistake is assuming a generator expression can be reused. After you iterate over a generator, it is exhausted. Calling next() again raises StopIteration. If you need to iterate multiple times, convert it to a list or recreate the generator.
Another pitfall involves variable scope. The iteration variable in a generator expression is local to the expression and does not leak into the surrounding scope. Names from the enclosing scope, however, are looked up when the generator runs, not when it is created. If the value of such a name changes before iteration begins, the generator reflects the new value:
n = 3
gen = (x * n for x in range(3))
n = 5
print(list(gen)) # [0, 5, 10]This late-binding behavior follows the normal rules of Python closures and can be surprising when you build several generators in a loop.
When to Choose a Generator Expression
Use a generator expression when you need to iterate over a large or potentially infinite sequence and you do not need to keep the data around. This includes reading files, processing streaming data, or generating values on the fly. If you need to access elements by index, slice the data, or iterate multiple times, use a list comprehension or a regular list.
Generator expressions also shine when combined with functions that short-circuit, like any or all. Because they are lazy, these functions can stop early without processing the entire sequence. For example, any(x > 10 for x in huge_list) stops as soon as it finds a value greater than 10, saving time on large inputs.
Generator Expression vs Generator Function
A generator function uses yield to produce values and can contain multiple statements, loops, and complex logic. A generator expression is limited to a single expression. If you need more than a simple expression, write a generator function.
def read_and_filter(file_path):
with open(file_path) as f:
for line in f:
value = int(line.strip())
if value > 0:
yield value * valueThis function does the same as a chained generator expression but is more readable when the logic grows. For simple transformations, a generator expression is concise and direct. For anything requiring statements, use a function.
Generator expressions are a powerful tool for writing memory-efficient, lazy Python code. They are not a replacement for list comprehensions but a complementary option that should be chosen based on the specific needs of your algorithm.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.