Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python itertools tee: Splitting Iterables into Multiple Streams

python itertools tee: Learn how itertools.tee creates independent iterators from a single iterable, its memory tradeoffs, and when to use it instead of list conversion.

~6 min readPython
#itertools#iterators#generators#lazy evaluation
Illustration of a single data stream splitting into two independent streams, representing Python's itertools.tee function.

When you have a generator or any one-shot iterable, you cannot iterate over it twice. The first loop exhausts it, and the second loop sees nothing. Python's itertools.tee provides a way to create multiple independent iterators from a single source, allowing you to traverse the same data multiple times without converting everything into a list. This article explains how itertools.tee works, its memory behavior, and the practical tradeoffs you need to consider before using it.

What itertools.tee Does

itertools.tee takes an iterable and an optional integer n (default 2) and returns a tuple of n independent iterators. Each iterator yields the same sequence of items as the original iterable. The key point is that the iterators are independent: you can consume them at different paces, and each one starts from the beginning.

Under the hood, tee does not copy the data. It also does not consume the source when you call it. Instead, it lazily pulls items from the original iterable and stores references to them in an internal buffer (a deque) until every returned iterator has consumed them. Once all iterators have passed an item, that item is discarded from the buffer. This design allows tee to work with infinite or large iterables without loading everything into memory at once, provided the iterators are consumed at roughly the same rate.

Basic Usage: Creating Independent Iterators

Here is a minimal example that duplicates a generator into two iterators:

python
from itertools import tee def generate_numbers(): for i in range(5): yield i numbers = generate_numbers() it1, it2 = tee(numbers) print(list(it1)) # [0, 1, 2, 3, 4] print(list(it2)) # [0, 1, 2, 3, 4]

Both it1 and it2 produce the same sequence. tee(numbers) does not pull from numbers at the moment it is called; the source is consumed only as the returned iterators are advanced. After the two list() calls above, the original generator is exhausted. In general, you should not use the original iterable after calling tee: if it is advanced independently, the tee iterators cannot see those values.

The second argument to tee controls how many iterators you get. For example, tee(data, 3) returns three independent iterators. This is useful when you need to process the same stream in multiple passes, such as computing several aggregations in one run.

How tee Buffers Items in Memory

Because tee is lazy, it does not precompute all items. Instead, it stores only the items that have been produced from the source but not yet consumed by every iterator. The buffer size depends on the consumption pattern of the iterators.

Consider this scenario:

python
from itertools import tee data = iter(range(1000)) it1, it2 = tee(data) # Consume it1 completely list(it1) # it2 still hasn't consumed anything list(it2)

When it1 is fully consumed, tee has pulled all 1000 items from data and stored them in the buffer because it2 hasn't consumed any yet. Only after it2 starts does the buffer shrink. If you consume one iterator far ahead of the other, memory usage grows proportionally to the difference. If both iterators are consumed in lockstep, the buffer stays very small.

This behavior is the central tradeoff of tee. It gives you the flexibility of multiple passes without duplicating the entire dataset, but it can still consume significant memory if the iterators diverge. For infinite iterables, this can become a serious problem if you never consume one of the iterators.

Comparing tee with Converting to a List

The most common alternative to tee is converting the iterable to a list and then iterating over the list multiple times. Both approaches allow multiple passes, but they differ in memory and laziness.

Aspectteelist()
MemoryBuffers only un-consumed itemsStores all items at once
LazyYes, items produced on demandNo, all items produced immediately
Random accessNo, only sequentialYes, supports indexing
Best forLarge or infinite iterables, limited passesSmall data, need indexing, repeated random access

Use tee when the source is a generator or a stream that you cannot re-create, and you need to iterate over it only a few times. Use list() when the dataset is small enough to fit in memory and you need random access or want to avoid the complexity of managing multiple iterator states.

Common Mistakes When Using tee

Several pitfalls can trip up developers new to tee.

Using the original iterable after calling tee. tee does not consume the source just by being called, but the original iterable can be advanced while the returned iterators are consumed. If you then iterate over it directly, you may see an exhausted iterator or an inconsistent subset of the data. Always use the iterators returned by tee.

Assuming tee copies data. tee does not copy the underlying objects. It creates references to the same objects. If the items are mutable and you modify them through one iterator, the change will be visible through the other. This is usually what you want, but it can cause unexpected side effects if you are not careful.

Keeping an unused iterator alive. If you create three iterators but hold on to one that you never consume, that iterator keeps the shared buffer alive. Memory will not be reclaimed until all iterator objects that share the buffer are garbage-collected. In long-running processes, avoid keeping unused tee iterators around, and consume or release the iterators you do not need.

Using tee with infinite iterables without a consumption plan. If you have an infinite stream and you consume only one iterator, the other will buffer every item produced, causing memory to grow without bound. You must ensure that all iterators are consumed at a similar pace, or drop the extra iterators so their references are released.

Using tee with Infinite or Large Iterables

tee can be useful when you need to process an infinite or very large stream in multiple ways. For example, suppose you have a stream of sensor readings and you want to compute both the running average and the maximum value over the first 1000 readings without storing all of them.

python
from itertools import tee, islice, count def sensor_stream(): for i in count(1): yield i * 0.5 # simulated reading readings = sensor_stream() avg_it, max_it = tee(readings) running_sum = 0 count_readings = 0 max_value = float('-inf') for avg_value, max_candidate in islice(zip(avg_it, max_it), 1000): running_sum += avg_value max_value = max(max_value, max_candidate) count_readings += 1 print(running_sum / count_readings, max_value)

The zip advances both avg_it and max_it at the same rate, so the internal buffer stays small. If you instead consumed avg_it for 1000 items and then called max(max_it) on an infinite stream, the call would never terminate because max has no way to know when the stream ends. It is important to design the consumption pattern before splitting a stream with tee.

A practical pattern is to use zip to consume multiple iterators together:

python
from itertools import tee stream = iter(range(1000)) it1, it2 = tee(stream) for a, b in zip(it1, it2): # process a and b simultaneously pass

This keeps the buffer size at one item because both iterators advance at the same rate.

When to Choose tee Over Other Approaches

tee is not always the right tool. If you can re-create the iterable cheaply, iterating twice directly is simpler and avoids any buffer overhead. For example, if your data comes from a function that generates a sequence deterministically, you can call that function twice instead of using tee.

Use tee when:

  • The source is a generator that cannot be re-created without side effects or expensive computation.
  • You need only a small number of independent passes (typically 2 or 3).
  • The iterators will be consumed at roughly the same rate, keeping memory bounded.

Avoid tee when:

  • You need random access to the data; use list() instead.
  • The data is small and you need to iterate many times; a list is simpler and faster.
  • You have an infinite stream and cannot guarantee that all iterators will be consumed; you risk unbounded memory growth.

In practice, tee is a specialized tool that solves a specific problem: splitting a one-shot iterable into multiple independent streams without materializing the entire dataset. Understanding its buffering behavior and memory implications helps you decide when it is the right choice and when a simpler alternative is better.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.