The iterator protocol
In plain words
A for loop is three polite questions: "May I have a bookmark?", "What's next?" and "Are we done?" When the bookmark says there's nothing left, the loop stops. Anything you can for over answers those questions.
Those questions are iter, next, and
StopIteration. That trio is the iterator protocol.
Get a cursor
iter(nums) returns an iterator. The list stays a list. The iterator remembers position.
Ask for one object
next(it) returns a reference to the next item and advances the cursor.
Signal the end
When nothing is left, the iterator raises StopIteration. for catches it and exits.
list, str, dict). An iterator produces values and holds the walk state. Every iterator is also iterable, and this is the part that catches people: iter(it) hands back the same object, not a fresh one. Ask a list for a bookmark twice and you get two bookmarks. Ask a generator and you get the same spent bookmark back. That's why looping over a generator a second time gets you nothing.
yield pauses a frame instead of destroying it
In plain words A normal function finishes the whole job, then throws its notes away. A generator says "here's one answer" and takes a nap with the notes still out. The next time you ask, it wakes up on the same line.
Calling a generator function returns a generator object straight away. It doesn't run the body yet. Each next() resumes the suspended frame until the
next yield. Locals such as n stay alive on that object.
Frame dies on return
Locals are gone. To remember state you need an object, a closure, or a global.
Frame sleeps on yield
n is still there on the next next(). The generator object is the handle to that sleeping frame.
count_up() doesn't compute a list of numbers. It allocates a generator. The work happens later, one next at a time, which is the same idea as lazy evaluation.
When does the heap pay for the result?
In plain words Baking every cookie before anyone asks is eager. Baking one cookie when someone reaches for the jar is lazy. The first way lets you count the cookies and eat them twice. The second way saves the oven if you only wanted the first two.
A list comprehension is eager: every value exists before you use the list. A generator expression is lazy: values appear as you pull them. Eager gives you length, indexing and reuse. Lazy saves work when you only need the first few.
| Form | What exists immediately | Can you reuse it? |
|---|---|---|
[x * x for x in xs] |
A full list of results | Yes. Index it, loop twice, pass it around. |
(x * x for x in xs) |
A generator object | Once. After exhaustion, another loop sees nothing. |
range(1_000_000) |
A small range object, not a million ints | Yes. It is iterable and can produce a fresh iterator. |
g = (x for x in [1, 2]) then list(g) twice.
The second call is [] because the generator was exhausted.
Compare that with r = range(2) and list(r) twice, where both calls succeed.
for loop, the list is still there. The hidden iterator is spent.
A second for nums creates a new iterator. A second loop over the same generator object doesn't, because generators are their own iterator and don't rewind.
The bugs this model prevents
In plain words When a loop behaves strangely, ask three everyday questions. Did I move the books around while someone was reading them? Did two people try to share one bookmark? Am I treating a bookmark as if it were the shelf?
Those are: mutating a sequence during iteration, sharing one exhausted iterator
between consumers, and expecting len() or indexing from an object
that only knows how to hand you the next item.
Removing items from a list while you loop over it
The iterator holds a plain integer index into a list it does not own. Delete an element and everything after it slides down one slot, but the cursor still moves forward, so it steps straight over the item that took the vacant place. With xs = [1, 2, 3, 4], removing 1 on the first pass visits [1, 3, 4]. The 2 is skipped and no error is raised. Loop over a copy (for x in xs[:]) or build a new list.
Handing the same generator to two consumers
g = (x for x in [1, 2, 3]), then sum(g) gives 6
and list(g) gives []. The first consumer drained it. Nothing was deleted and nothing raised. There's just nothing left to yield. If two pieces
of code need the values, materialise them once with list(...), or build
a fresh generator for each.
Asking a generator for its length or its third item
len(g) raises TypeError: object of type 'generator' has no len()
and g[0] raises 'generator' object is not subscriptable. That's honest rather than stingy. The generator really doesn't know how many values are coming, because it hasn't computed them yet. Length and indexing are what you buy when you pay up front for a list.
A collection (the shelf), a separate iterator (the bookmark) holding a link and an index, and, for generators, a paused frame whose locals survive between yields.
If you can draw those three, for, yield, and lazy
expressions all fit the same picture.