After this lesson you will understand what a for loop does behind the scenes, and you will be able to write functions that produce a sequence of values one at a time, without building the whole sequence first.
Iterables and iterators
Lists, strings, dictionaries and range objects are all iterables: things you can loop over. When a for loop starts, Python asks the iterable for an iterator using iter(). The iterator is the object that remembers where you are, and each call to next() gives the following value.
books = ["Dune", "Emma", "Ulysses"]
it = iter(books)
print(next(it))
print(next(it))
print(next(it))Dune
Emma
UlyssesWhen the values run out, next() raises StopIteration:
next(it)StopIterationThat is the whole secret of the for loop. It calls iter() once, calls next() repeatedly, and quietly stops when StopIteration appears. You can write the loop by hand to see it:
it = iter(["pen", "ink"])
while True:
try:
item = next(it)
except StopIteration:
break
print("Got", item)Got pen
Got inkAn iterator remembers its place, so it can be used only once. A list is different: each for loop over it creates a new iterator, so you can loop over a list as many times as you like.
Why we want lazy values
Suppose you need the first million square numbers but you will only use a few. A list would build all of them in memory first. A generator produces each value only when asked. That saves memory and time, and it even allows sequences with no end.
Generator functions and yield
A function containing yield becomes a generator function. Calling it does not run the body. It returns a generator object. Each next() runs the body until the next yield, hands over that value, and pauses with all its local variables intact.
def countdown(start):
print("starting")
while start > 0:
yield start
start -= 1
print("finished")
gen = countdown(3)
print("generator created")
print(next(gen))
print(next(gen))
print(next(gen))generator created
starting
3
2
1Notice that starting appeared only at the first next(), not when the generator was created. After the last yield, one more next() would run to the end of the function, print finished, and raise StopIteration. In a for loop all of this is handled for you:
for n in countdown(2):
print("number", n)starting
number 2
number 1
finishedA practical example: a pipeline
Generators suit pipelines where data passes through stages. The first generator below picks the items that cost more than a limit, and the second applies a discount to each price it receives.
def prices_over(prices, limit):
for name, price in prices.items():
if price > limit:
yield name, price
def discounted(items, percent):
for name, price in items:
yield name, round(price * (100 - percent) / 100, 2)
shop = {"pen": 2.5, "bag": 40.0, "lamp": 25.0, "mug": 8.0}
for name, price in discounted(prices_over(shop, 5), 10):
print(name, price)bag 36.0
lamp 22.5
mug 7.2No list of intermediate results was ever built. Each item travels through both stages one at a time.
Generator expressions
A list comprehension with round brackets instead of square ones is a generator expression. It is lazy.
numbers = [1, 2, 3, 4, 5]
squares_list = [n * n for n in numbers]
squares_gen = (n * n for n in numbers)
print(squares_list)
print(type(squares_gen).__name__)
print(sum(squares_gen))[1, 4, 9, 16, 25]
generator
55Printing a generator object itself shows only its type and a memory address, never its values, so the example prints the type name instead. When a generator expression is the only argument to a function, you can drop the extra brackets, as in sum(n * n for n in numbers).
Generators are used up
A generator remembers its place, so a second pass finds nothing left.
gen = (n * 10 for n in range(3))
print(list(gen))
print(list(gen))[0, 10, 20]
[]If you need the values twice, store them in a list, or call the generator function again to get a fresh generator.
Infinite generators
Because values are produced on demand, a generator can run forever. You take as many as you need, for instance with itertools.islice.
from itertools import islice
def ticket_numbers(start):
number = start
while True:
yield number
number += 1
print(list(islice(ticket_numbers(100), 5)))[100, 101, 102, 103, 104]Never call list() on an infinite generator directly, because it would never finish.
Delegating with yield from
When a generator wants to hand over every value of another iterable, yield from does it in one line.
def all_names(*groups):
for group in groups:
yield from group
print(list(all_names(["Asha", "Ben"], ["Chloe"], ("Daniel",))))['Asha', 'Ben', 'Chloe', 'Daniel']Common mistakes
- Expecting a generator to be reusable. Once it is exhausted it stays empty. Create it again.
- Printing a generator and expecting values. Printing shows only that it is a generator object. Loop over it or wrap it in
list(). - Using
returnwhere you meantyield. Areturnends the function and gives back one value. Ayieldpauses and gives one value at a time. - Calling
list()on an infinite generator. The program never ends. Limit it withisliceor abreak. - Forgetting that code runs lazily. Errors inside a generator appear when you ask for the value, not when you create it.
Exercise
Write a generator function evens(limit) that yields the even numbers from 2 up to and including limit. Print list(evens(10)), then use a generator expression with sum to add up the even numbers up to 10.
Show one possible answer
def evens(limit):
n = 2
while n <= limit:
yield n
n += 2
print(list(evens(10)))
print(sum(n for n in range(1, 11) if n % 2 == 0))[2, 4, 6, 8, 10]
30Next: decorators build on functions that take and return functions. A video version of this topic is in the video course at /learn/python/.
Key points
- An iterable can be looped over. An iterator is the object that hands out the values one at a time with next().
- A for loop calls iter() on the iterable, then next() until StopIteration is raised.
- A function that contains yield is a generator function. Calling it returns a generator that produces values lazily, pausing at each yield.
- A generator expression such as (x * 2 for x in data) is a compact lazy version of a list comprehension.
- Generators are used up after one pass, but they can be infinite and use very little memory.