Python Iterators and Generators Explained

An iterable is anything you can loop over, like a list or a string. An iterator is the object that hands out the items one at a time. A generator is the easy way to make your own iterator: you write a function that uses yield, and Python produces its values lazily, one per request, remembering where it left off.

The payoff is memory and flexibility. A generator can produce a million values, or an endless stream, without ever holding them all at once. Below is how it works step by step, with real output.

Iterables and iterators

Calling iter() on an iterable gives you an iterator. Calling next() on the iterator returns the items one by one. When there is nothing left, it raises StopIteration:

nums = [10, 20, 30]
it = iter(nums)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("done")

Output

10
20
30
done

An iterable and an iterator are not quite the same thing. A list is iterable but is not itself an iterator: it has no __next__. The iterator you get from it does, and an iterator is also iterable (calling iter() on it returns itself):

nums = [1, 2, 3]
print(iter(nums) is nums)
it = iter(nums)
print(iter(it) is it)
print(hasattr(nums, "__next__"), hasattr(it, "__next__"))

Output

False
True
False True

What a for loop really does

A for loop is a tidy wrapper around exactly this. It calls iter() once, then calls next() repeatedly, and stops quietly when it sees StopIteration. This while loop does the same job as for item in nums:

nums = [10, 20, 30]
it = iter(nums)
while True:
    try:
        item = next(it)
    except StopIteration:
        break
    print(item)

Output

10
20
30

That is why anything with the iterator protocol works in a for loop, in list(), in sum() and in comprehensions. See Python Loops for the loops themselves.

Building an iterator by hand

To make your own iterator you write a class with two methods: __iter__ (returns the iterator, usually self) and __next__ (returns the next value, or raises StopIteration):

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

print(list(Countdown(3)))
for n in Countdown(2):
    print(n)

Output

[3, 2, 1]
2
1

This works, but it is a lot of ceremony. Notice that all the bookkeeping (self.current) has to be stored by hand. Generators remove that.

Generators: the easy way with yield

A function that contains yield is a generator function. Calling it does not run the body. It returns a generator object, which is an iterator. Each next() runs the function until the next yield, hands that value back, and pauses:

def countdown(start):
    while start > 0:
        yield start
        start -= 1

g = countdown(3)
print(type(g).__name__)
print(next(g))
print(next(g))
print(list(g))

Output

generator
3
2
[1]

The same countdown took four lines instead of twelve, and Python kept track of start for us. The last line, list(g), simply drained what was left.

yield vs return

return ends a function and gives back one value. yield hands back a value and keeps the function alive, ready to continue from that exact line. A function can yield many times.

Watching a generator pause and resume

The easiest way to understand it is to put print calls inside. Nothing inside the body runs when the generator is created. Each next() runs to the next yield:

def steps():
    print("start")
    yield 1
    print("middle")
    yield 2
    print("end")

g = steps()
print("created")
print(next(g))
print(next(g))
try:
    next(g)
except StopIteration:
    print("finished")

Output

created
start
1
middle
2
end
finished

Read the output in order. “created” appears first with no “start”, proving the body had not begun. The first next() printed “start” and stopped at yield 1. The second resumed, printed “middle”, and stopped at yield 2. The third resumed, printed “end”, ran off the end of the function, and that raised StopIteration.

Why generators save memory

A list holds every item in memory at once. A generator produces each item only when asked, and forgets it afterwards. So this sums the squares of a million numbers without ever building a million-item list:

total = sum(n * n for n in range(1_000_000))
print(total)

Output

333332833333500000

The shorter form, (n * n for n in range(...)), is a generator expression, the round-bracket cousin of a list comprehension. We compare them in Python List Comprehension Explained Simply. Use a generator when you only need to go through the values once, especially with big files or streams of data.

Patterns: infinite streams, pipelines and yield from

Infinite streams

Because a generator only produces values on demand, it can describe a sequence that never ends. You just take as many as you want, for example with itertools.islice:

from itertools import islice

def naturals():
    n = 1
    while True:
        yield n
        n += 1

print(list(islice(naturals(), 5)))

Output

[1, 2, 3, 4, 5]

Pipelines

Generators chain nicely. Each stage takes one item at a time from the previous stage, so the data flows through without any large intermediate lists. This is how you would process a huge log file line by line:

def read_lines():
    for line in ["  apple ", "", "banana", "  ", "cherry"]:
        yield line

def clean(lines):
    for line in lines:
        line = line.strip()
        if line:
            yield line

def shout(lines):
    for line in lines:
        yield line.upper()

print(list(shout(clean(read_lines()))))

Output

['APPLE', 'BANANA', 'CHERRY']

yield from

yield from hands over to another iterable and yields all of its items, which is a tidy way to combine sources:

def chain(a, b):
    yield from a
    yield from b

print(list(chain([1, 2], "xy")))

Output

[1, 2, 'x', 'y']

Common mistakes

Mistake 1: using a generator twice

A generator is used up once it has been consumed. The second pass finds nothing:

g = (n * n for n in range(3))
print(list(g))
print(list(g))

Output

[0, 1, 4]
[]

If you need the values again, store them in a list, or create a new generator.

Mistake 2: expecting len() or indexing

A generator does not know how many items it will produce, and it cannot jump to item 5. Convert it with list() first if you need either:

g = (n for n in range(3))
try:
    len(g)
except TypeError as e:
    print(type(e).__name__)
print(len(list(g)))

Output

TypeError
3

Mistake 3: forgetting that nothing runs until you ask

Calling a generator function does not run any of its code. If you call it and never iterate, nothing happens, and any errors inside it appear only later, when the values are first requested.

Mistake 4: using a generator when you need the list

If you will loop over the data several times, or need its length, a plain list is simpler. Generators shine for one-pass, large or endless data.

Try it yourself

Work out each answer first, then open the solution.

1. Write a generator evens(limit) that yields the even numbers from 0 up to and including limit.

def evens(limit):
    for n in range(0, limit + 1, 2):
        yield n

print(list(evens(10)))

[0, 2, 4, 6, 8, 10]

2. What does this print?

it = iter([1, 2])
next(it)
print(list(it))

[2]

3. Write a generator for the Fibonacci numbers and take the first eight.

from itertools import islice

def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

print(list(islice(fib(), 8)))

[0, 1, 1, 2, 3, 5, 8, 13]

4. What does this print?

g = (x for x in [1, 2, 3])
next(g)
print(sum(g))

5

Run these in our free Python compiler.

Frequently asked questions

What is the difference between an iterator and an iterable?

An iterable is something you can loop over (a list, string, dictionary). An iterator is the object that produces the items one by one through next(). Calling iter() on an iterable gives you an iterator.

What is a generator in Python?

A generator is an iterator created by a function that contains yield (or by a generator expression). It produces values one at a time and remembers its position between them.

What is the difference between yield and return?

return ends the function and gives back a single value. yield gives back a value and pauses the function so it can continue from the same place on the next request.

When should I use a generator instead of a list?

When you only need to go through the values once, when the data is large, or when it is endless. If you need the length, indexing, or several passes, use a list.

Why can I only use a generator once?

It produces each value and moves on, so once it is exhausted there is nothing left to give. Create a new generator, or store the values in a list, to go through them again.

What is StopIteration?

It is the exception an iterator raises when it has no more items. Loops catch it for you and end quietly, so you rarely see it unless you call next() yourself.

Related reading

• Python List Comprehension Explained Simply – including generator expressions.
• Python Loops: for, while, break and continue – the loops that use iterators behind the scenes.
• Python Classes and Objects for Beginners – how the __iter__ and __next__ methods fit in.
• Python Context Managers and the with Statement – another use of generators.