The iterator protocol
In plain words
A for loop is three polite questions: “May I have a bookmark?”
“What’s next?” and “Are we done?” When the bookmark says there is nothing
left, the loop stops. Anything you can for over answers those questions.
Those questions are iter, next, and
StopIteration. That trio is the iterator protocol.
Get a cursor
iter(nums) returns an iterator. The list stays a list. The iterator remembers position.
Ask for one object
next(it) returns a reference to the next item and advances the cursor.
Signal the end
When nothing is left, the iterator raises StopIteration. for catches it and exits.
list, str, dict).
An iterator produces values and holds the walk state. Every iterator is also
iterable, and this is the part that catches people: iter(it) hands back
the same object, not a fresh one. Ask a list for a bookmark twice and you get two
bookmarks; ask a generator and you get the same spent bookmark back. That is why looping
over a generator a second time gets you nothing.
yield pauses a frame instead of destroying it
In plain words A normal function finishes the whole job, then throws its notes away. A generator says “here is one answer” and takes a nap with the notes still out. The next time you ask, it wakes up on the same line.
Calling a generator function returns a generator object immediately — it does not
run the body yet. Each next() resumes the suspended frame until the
next yield. Locals such as n stay alive on that object.
Frame dies on return
Locals are gone. To remember state you need an object, a closure, or a global.
Frame sleeps on yield
n is still there on the next next(). The generator object is the handle to that sleeping frame.
count_up() does not compute a list of numbers. It allocates a generator.
Work happens later, one next at a time. That is the same idea as lazy evaluation.
When does the heap pay for the result?
In plain words Baking every cookie before anyone asks is eager. Baking one cookie when someone reaches for the jar is lazy. The first way lets you count the cookies and eat them twice. The second way saves the oven if you only wanted the first two.
A list comprehension is eager: every value exists before you use the list. A generator expression is lazy: values appear as you pull them. Eager gives length, indexing, and reuse. Lazy saves work when you only need a prefix.
| Form | What exists immediately | Can you reuse it? |
|---|---|---|
[x * x for x in xs] |
A full list of results | Yes. Index it, loop twice, pass it around. |
(x * x for x in xs) |
A generator object | Once. After exhaustion, another loop sees nothing. |
range(1_000_000) |
A small range object, not a million ints | Yes. It is iterable and can produce a fresh iterator. |
g = (x for x in [1, 2]) then list(g) twice.
The second call is [] because the generator was exhausted.
Compare with r = range(2) and list(r) twice — both succeed.
for loop, the list is still there. The hidden iterator is spent.
A second for nums creates a new iterator. A second loop over the
same generator object does not — generators are their own iterator and do not rewind.
The bugs this model prevents
In plain words When a loop behaves strangely, ask three everyday questions: did I move the books around while someone was reading them? Did two people try to share one bookmark? And am I treating a bookmark as if it were the shelf?
Those are: mutating a sequence during iteration, sharing one exhausted iterator
between consumers, and expecting len() or indexing from an object
that only knows how to hand you the next item.
Removing items from a list while you loop over it
The iterator holds a plain integer index into a list it does not own. Delete an
element and everything after it slides down one slot — but the cursor still moves
forward, so it steps straight over the item that took the vacant place.
xs = [1, 2, 3, 4], then removing 1 on the first pass,
visits [1, 3, 4]: the 2 is silently skipped and no error
is raised. Loop over a copy (for x in xs[:]) or build a new list.
Handing the same generator to two consumers
g = (x for x in [1, 2, 3]), then sum(g) gives 6
and list(g) gives []. The first consumer drained it. Nothing
was deleted and nothing raised — there is simply nothing left to yield. If two pieces
of code need the values, materialise them once with list(...), or build
a fresh generator for each.
Asking a generator for its length or its third item
len(g) raises TypeError: object of type 'generator' has no len()
and g[0] raises 'generator' object is not subscriptable. That is
honest, not stingy: the generator genuinely does not know how many values are coming,
because it has not computed them. Length and indexing are exactly what you buy when
you pay eagerly for a list.
A collection (the shelf), a separate iterator (the bookmark) holding a link and an
index, and — for generators — a paused frame whose locals survive between yields.
If you can draw those three, for, yield, and lazy
expressions all fit the same picture.