# Py Internals: full text
Source: https://inboxpraveen.github.io/Py-Internals/ (https://github.com/inboxpraveen/Py-Internals). Licence: course content CC BY-SA 4.0, code Apache-2.0.
This file is the complete text of every session, the demo code with its step-by-step narration, the quizzes, and the glossary, in Markdown.
---
# Session 01: Variables & Mutability
URL: https://inboxpraveen.github.io/Py-Internals/sessions/01-variables/
Summary: See what x = 42 really does in memory. Names, objects, id(), aliasing, mutation versus rebinding and reference counting, animated one step at a time.
`x = 42` isn't putting a number in a box. It's sticking a nametag
on something that already lives in memory, and two nametags can share one thing.
We'll start there, then get precise about identity, mutation and references.
**Learning goal**
## A variable is a nametag, not a box.
Start there. `x` doesn't hold 42, it points at an object that is 42. Once that feels natural, the technical words (identity, mutation, aliasing, refcount) are just names for things you can already see.
**Recommended path**
**Watch the stage.** Step through each line and notice which object appears or changes.
**Name the operation.** Ask: did Python rebind a name, mutate an object, or add another reference?
**Read the guide.** Use the narrative below to connect the animation to everyday Python code.
## What is a variable in Python?
**In plain words**
Imagine a sticky note that says `x`. The note isn't the number. It's stuck on a real thing sitting in memory. Two notes can sit on the same thing, and moving a note doesn't change the thing. It just points the note somewhere else.
The precise model has three parts: the **name** (the sticky note),
the **object** (the thing in memory), and the **reference**
(the stickiness connecting them).
### 1. Name
The label you type, such as `x`. It lives in a namespace.
It is not a storage box.
### 2. Object
The actual value: `42`, `"hello"`, `[1, 2, 3]`.
These live on the heap and each has its own identity.
### 3. Reference
The connection. Assignment sticks a name onto an object.
More than one name can share one object.
Many languages teach a variable as a named box that contains a value.
Python is different. When you write `x = 42`, Python creates or finds
an integer object, then binds the name `x` to it.
The name does not contain the number; it points at the object that represents it.
**Assignment binds a name**
In Python, **everything is an object**: numbers, strings, functions, classes, all of them. Assignment with `=` binds a name to an object. It doesn't copy the object unless you ask Python for a copy.
## What is `id()`?
**In plain words**
`id()` answers one question: are these two names looking at the *same* thing, or at two things that happen to look alike? The same id means one object with two nametags.
Every object has a unique identity, which in CPython is a stand-in for its memory address. `id(x)` returns that as an integer. Two names with the
**same `id()` point to the exact same object**. No copy was made.
```python
x = 42
y = x
print(id(x) == id(y)) # True, same object
print(x is y) # True, 'is' checks identity
```
**`is` vs `==`**
`==` checks **value equality**: do the objects look the same?
`is` checks **identity**: are they literally the same object in memory?
Use `is None`, `is True`, `is False` for singletons.
For everything else, prefer `==`.
---
Mutable or Immutable
## Can the object itself change?
**In plain words**
Some things you can edit (a shopping list). Some things you can only replace
(a printed number). If you "change" a number, Python throws the old one away and points your name at a new one.
That's mutability. Once a name points at an object, ask whether the object can change in place, or whether a "change" has to create a different object.
**Immutable**
The object's value cannot change. A "change" creates or finds another object and rebinds
the name to it.
`int` `float` `str` `tuple`
**Mutable**
The object's contents can change while its identity stays the same. All aliases see the
updated contents.
`list` `dict` `set`
**Most surprises come from sharing mutable objects**
`a = [1, 2, 3]` followed by `b = a` does not copy the list.
Both names point to one object. If `b.append(4)` runs, `a`
sees the new item too.
---
Reference Counting
## How Python tracks when to free memory
**In plain words**
Python counts how many nametags are stuck on an object. When you add a name,
the count goes up. When you peel one off, it goes down. When the last nametag
comes off, Python can throw the object away.
That hidden counter is the **reference count**. CPython updates it
as names and containers are bound and unbound. At zero the object is freed immediately, unless it's stuck in a cycle, which needs a separate collector.
Try this on a list, not on a number. A few values (small integers, `True`, `False`, `None`) are **immortal** in CPython 3.12 and later. Their count is pinned and they're never freed, so `sys.getrefcount(42)` returns a huge fixed number rather than a count you can watch move. There's more on that further down.
Refcount **increases** when a new name is bound to the object
(`x = obj`, `b = a`, or passing it as a function argument).
Refcount **decreases** when a name is rebound (`x = other`),
deleted (`del x`), or goes out of scope (function returns).
When the refcount reaches **0**, CPython frees the object's memory right away. No waiting and no delay, it's collected on the spot.
```python
import sys
x = [1, 2, 3]
print(sys.getrefcount(x)) # 2 (x + getrefcount's argument)
y = x # y also points to the list
print(sys.getrefcount(x)) # 3
del y # y removed; refcount drops
print(sys.getrefcount(x)) # 2 again
```
**Reference cycles**
If two objects reference each other (e.g., a list containing itself), their refcounts
never reach 0 even when nothing else points to them.
CPython has a **cyclic garbage collector** that handles these cases, but it runs now and then rather than immediately. That's why CPython doesn't *only* use reference counting.
---
Mutation vs Rebinding
## The distinction that matters most
**In plain words**
Did you move the nametag, or did you edit the thing it was stuck to? Moving the nametag leaves the other nametags alone. Editing the thing is visible to everyone still pointing at it.
Those two operations are **rebinding** and **mutation**.
Almost every "why did my list change?" bug comes from mixing them up.
**Rebinding**
`x = new_value` makes the name point to a different object.
The old object is unchanged; it simply loses one reference.
**Mutation**
`x.append()` or `x[0] = ...` changes the existing object.
Every name pointing at that object observes the change.
```python
# ── REBINDING (immutable types) ─────────────────────────────
s = "hello"
t = s # both point to the same "hello" object
s = "world" # s rebound to a NEW object; t still "hello"
print(t) # hello, unaffected
```
# ── MUTATION (mutable types) ─────────────────────────────────
a = [1, 2, 3]
b = a # both point to the same list
a.append(4) # mutates the shared object
print(b) # [1, 2, 3, 4], b sees the change
Immutable types (`int`, `str`, `float`, `tuple`) can *only* be rebound. You can never change the object itself.
Mutable types (`list`, `dict`, `set`) can be both mutated
and rebound.
| Operation | Example | Creates new object? | id() changes? | Other aliases see change? |
| --- | --- | --- | --- | --- |
| **Rebind** | `x = 99` | Yes | Yes | No |
| **Augmented rebind** | `x += 1` (int) | Yes | Yes | No |
| **Mutate (append)** | `lst.append(x)` | No | No | Yes |
| **Mutate (index)** | `lst[0] = x` | No | No | Yes |
| **Augmented rebind** | `lst += [x]` (list) | No* | No* | Yes* |
* For lists, `+=` calls `__iadd__` which mutates in-place (unlike with immutable types).
---
Python's Built-in Types
## All the Variable Types
**In plain words**
You don't need to memorise every type today. For each one, ask one question: can I edit this thing, or can I only replace it?
Python's built-in types differ mainly by **mutability**, and that decides whether a "change" is a mutation or a rebind.
int
**immutable**
x = 42
x = -7
x = 0
Whole numbers, any size. Immutable: `x += 1` creates a new int object, doesn't change the old one.
float
**immutable**
x = 3.14
x = 2.0
x = -0.5
Decimal numbers (64-bit IEEE 754). Immutable. Watch out for floating point precision.
str
**immutable**
s = "hello"
s = 'world'
s = """multi
line"""
Sequences of Unicode characters. They can't be changed after creation: `s[0] = 'H'` raises a TypeError.
bool
**immutable**
done = True
active = False
A subclass of `int`! True == 1, False == 0. Immutable singleton objects.
NoneType
**immutable**
result = None
The absence of a value. It's a singleton: there is exactly **one** None object in the whole Python process.
tuple
**immutable**
t = (1, 2, 3)
t = ("a", True)
Ordered, fixed-length sequence. You can't add/remove/change elements after creation. Great for function return values.
list
**mutable**
nums = [1, 2, 3]
nums.append(4)
nums[0] = 99
Ordered, changeable sequence. You can add, remove, and update items in place. The object's identity stays the same.
dict
**mutable**
d = {"a": 1}
d["b"] = 2
del d["a"]
Key-value store. Mutable and ordered (Python 3.7+). Keys must be immutable (hashable).
set
**mutable**
s = {1, 2, 3}
s.add(4)
s.discard(1)
An unordered collection of unique items. Mutable, so you can add and remove, but the elements themselves must be immutable (hashable).
### Quick Reference: Mutability at a glance
| Type | Literal | Mutable? | Why it matters |
| --- | --- | --- | --- |
| int | `42` | No | `x += 1` rebinds `x` to a new object |
| float | `3.14` | No | Any arithmetic produces a new float |
| str | `"hello"` | No | String methods return new strings; you can't change characters in-place |
| bool | `True` | No | Only two boolean objects exist; they're singletons |
| NoneType | `None` | No | One None object exists; always use `is None` to check |
| tuple | `(1, 2)` | No | Can't add/remove/change; safe to use as dict keys |
| list | `[1, 2]` | Yes | A function can modify the list you passed in, so be careful with aliases |
| dict | `{"k": v}` | Yes | A function gets a second name for the same dict, so changes made inside it are visible to the caller |
| set | `{1, 2}` | Yes | Can grow/shrink; elements must be immutable/hashable |
**The aliasing trap**
`a = [1, 2, 3]` then `b = a` does NOT copy the list.
Both `a` and `b` point to the same list object.
Doing `b.append(4)` will also affect `a`!
Use `b = a.copy()` or `b = list(a)` to get an independent copy.
**Some objects are shared, and a few live forever**
CPython builds the small integers once, at start-up, and hands out the same object every time, typically for `-5` through `256`. So
`a = 5; b = 5` gives `a is b` → `True`: one object,
two names. It does the same for short identifier-like strings, which is why
`s = "hello"; t = "hello"` also gives `s is t` → `True`.
Outside that cached range there is no promise either way. Two large numbers *computed* separately are usually different objects (`n = 999; c = n + 1; d = 1000; c is d` → `False`), but two identical literals written in the same file often still share one object, because the compiler stores the constant once. Don't test interning with
`id(5) == id(5)`: that is `True` for every number, interned or
not, since both halves are the same literal.
One more consequence: in CPython 3.12 and later these shared values (small ints, `True`, `False`, `None`) are **immortal**. They're never freed, and their reference count is pinned at a huge fixed number.
That is why `sys.getrefcount(42)` prints something like
`4294967295` instead of a small count, and why the memory panel shows
`∞` for `42` rather than a number that ticks up and down.
All of this is an implementation detail, not a language guarantee, so never write code
whose correctness depends on it.
**The golden rule**
**Immutable objects** are safe to share, because no one can change them through another reference. **Mutable objects** need care when passed around, because anyone with a reference can change the object.
**Try this in a REPL**
`a = [1, 2]; b = a; print(a is b)` then `b = a.copy(); print(a is b)`.
First `True` (alias), then `False` (new list).
**What you should be able to draw**
A namespace of names, heap objects with identities, and arrows for references.
Rebinding moves an arrow. Mutation changes the object the arrows already point at.
---
Common mistakes
## Four ways this goes wrong
**In plain words**
Every one of these is the same slip: treating a nametag as if it were the thing it's stuck to. When code surprises you, find the object first, then count the names on it.
### Expecting `b = a` to make a copy
It adds a second nametag to the same object. If the object is a list or a dict,
a change through either name is a change both names see. When you want a separate
object, ask for one: `b = a.copy()` or `b = list(a)`.
(Session 03 shows why even that copies only one level.)
### Thinking `del x` deletes the object
`del` removes the *name*. The object loses one reference, and it
is freed only if that was the last one. After `a = [1, 2]; b = a; del a`,
the list is alive and well under `b`. Nothing you write ever destroys an object directly. You only ever let go of it.
### Reading `+=` as "edit in place" for every type
On an `int` or a `str`, `x += 1` builds a new object
and moves the name; other names keep the old value. On a `list`,
`x += [1]` edits the object, and every alias sees it. The same two
characters, two different operations. The famous trap: `t = ([1], 2)` then `t[0] += [3]` raises `TypeError`, *and* the inner list has already grown to `[1, 3]`, because the mutation happened before the tuple refused the assignment.
### Testing values with `is`
`a is b` asks "same object?", and for numbers and strings the answer depends on how the objects were made. A cached small int says `True`, two computed `1000`s say `False`, and two literals in one file often say `True`. None of that is a language promise.
Compare values with `==`; keep `is` for `None`,
`True` and `False`, and for checking aliasing while you learn.
**One question answers most of these**
"How many names are on this object right now?" If the answer is more than one, a mutation through any of them is visible through all of them, and a rebinding
of one of them changes nothing for the rest.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · Assigning a variable
Watch for: A name appears, then an object. The name doesn't hold the value, it points at it.
```python
x = 42
y = "hello"
z = 3.14
print(hex(id(x))) # 0x7f10a0c0 (yours will differ)
print(hex(id(y))) # 0x7f20b810 (a different object)
print(hex(id(z))) # 0x7f30c454 (and another one again)
```
1. **Nothing has run yet** No names, no objects. The namespace is empty and so is the heap. Press Next to run one line at a time.
2. **x = 42: the nametag goes on an object** Python finds or makes the object 42, then sticks the nametag x on it. The name doesn't contain the number. It points at the object that holds it. The reference count shows ∞ because small whole numbers are built once when Python starts and shared for the rest of the run. Demo 2 comes back to that.
3. **y = "hello": a second object, somewhere else** A string object appears at a different address and y is stuck on it. Different address, different thing, so x and y have nothing to do with each other. Strings are immutable like numbers: you can replace one, but you can't edit it in place.
4. **z = 3.14: a third object** Anything you write as a literal (a number, a string, a list, even a function) becomes a real object in memory, with its own address, type and value. Three lines have run, so there are three separate objects with one nametag each.
5. **id() tells you which object a name is on** id() answers one question: are these two names on the same thing? It returns an integer that identifies the object, which in CPython is its memory address, and hex() prints that integer the way the memory panel does. Three different numbers here, because these really are three different objects.
6. **Summary: three names, three objects, no boxes** None of these lines put a value inside a name. Each one found or made an object and stuck a nametag on it. That's the rule to keep: the same address means one object with several names on it, and different addresses mean different objects, however alike they look on screen.
### Demo 2 · Rebinding
Watch for: The name moves. Watch which object gets freed, and which one Python keeps for good.
```python
x = 42
print(hex(id(x))) # 0x7f10a0c0 (yours will differ)
x = [1, 2] # rebind: x now points at a brand-new list
print(hex(id(x))) # 0x7f40d0f8, a different address
x = 42 # back to the very same 42 object
print(hex(id(x))) # 0x7f10a0c0, same address as line 1
```
1. **Initial state: an empty namespace** This demo is about rebinding: pointing the name x at three different objects in turn. Keep one question in mind as you go. Does the object change, or does the name change?
2. **x = 42, and Python already had a 42** Small whole numbers come up so often that CPython builds them once at start-up and hands out the same object every time. Nothing is created here. The name x is pointed at the 42 that already exists at address 0x7f10a0c0.
3. **x = [1, 2]: this object really is new** Lists aren't cached. This one is built fresh, gets its own address, and its refcount starts at 1 because one name points at it. x has let go of 42, but 42 isn't freed. It was never yours to free.
4. **x = 42 again: the list loses its last name** Now it goes the other way. x moves back to the same 42 object, so no name points at the list any more. Its refcount drops to 0.
5. **The list is freed at once. The 42 isn't** CPython counts references. The moment a count reaches zero the object is freed, on that very line, with no waiting and no sweep. The list is gone. The 42 is still at 0x7f10a0c0, the same address as on line 1, because the whole program shares it.
6. **Summary: the name moved three times and no object was edited** Rebinding never changes an object. It points a name somewhere else. What happens next depends on who else is holding on. The list had one holder and was freed the moment it lost it. The 42 is cached by CPython and outlives your program either way. That cache covers -5 to 256 and it's an implementation detail, not a language rule, so never write code that depends on it.
### Demo 3 · Aliasing
Watch for: Two names, one list, one address. append through either name changes the same object.
```python
a = [1, 2, 3]
b = a # b is an alias, not a copy
b.append(4) # mutates the shared list
print(a) # [1, 2, 3, 4] a sees it too
print(a is b) # True, same identity
```
1. **Nothing has run yet** Two nametags on one list. That's all aliasing is, and it causes more surprise bugs than almost anything else in Python. Watch what happens when we change the list through one name and then look at it through the other.
2. **a = [1, 2, 3]: a list you're allowed to edit** A list object is built at address 0x7f50e880 and a is stuck on it. Notice the mutable badge. Unlike an int or a str, this object's contents can change without it becoming a different object.
3. **b = a: b is an alias, not a copy** Assignment never copies. b = a puts a second nametag on the list that's already there, at the same address 0x7f50e880, so the count of names on it goes up to 2. If you wanted a separate list you have to ask for one: b = a.copy() or b = list(a).
4. **b.append(4) mutates the shared list** There's only one list, so appending through b changes the object a is on. The panel marks it mutated: the contents changed, the address 0x7f50e880 didn't. Same object, new contents.
5. **print(a) → [1, 2, 3, 4]** a was never assigned to again, yet it now shows four items. Nothing odd happened. a and b were always on one list, and that list changed. This is how Python is meant to work, and it's also where a shared list catches people out.
6. **a is b → True: one object, two names** is asks "same object?", not "same value?". Both nametags are on one list, so the answer is True. a == b asks the other question, whether the two look the same, and that can be true of objects at completely different addresses.
7. **Summary: one list, two names, one change** Assignment never copies. b = a only added a second nametag, so a change made through either name is a change both names see. There's nothing special about lists here. It happens with every mutable object, and with every function you hand one to.
### Demo 4 · Mutation vs Rebinding
Watch for: append keeps the same address. nums = ... makes a new list and moves the name.
```python
nums = [10, 20, 30]
print(hex(id(nums))) # 0x7f60f010 (yours will differ)
# Mutation: change the object in place
nums.append(40)
print(hex(id(nums))) # still 0x7f60f010, same object
# Rebinding: point nums at a brand-new object
nums = [10, 20, 30, 40, 50]
print(hex(id(nums))) # 0x7f70a448, a different object
```
1. **Nothing has run yet** Two things that look almost the same in code and are very different in memory: editing a list, and pointing a name at a different list. The address is what tells them apart, so keep an eye on it all the way through.
2. **nums = [10, 20, 30]: one list, one nametag** A list is built at address 0x7f60f010 and line 2 prints that address. Treat it as the baseline for the rest of the demo. After each operation, ask whether it has stayed the same.
3. **nums.append(40): the list is edited in place** Appending changes what's inside the list. No second list is made and the nametag doesn't move. The object at 0x7f60f010 now holds four items. That's mutation, and the address doesn't budge.
4. **The address printed is the same one as on line 2** Still 0x7f60f010. Editing an object never changes which object it is. That's why a function that appends to a list you passed in changes your list: there was only ever one list, and the function had another name for it.
5. **nums = [10, 20, 30, 40, 50]: a brand-new list** This line doesn't touch the old list at all. Python builds a second list at 0x7f70a448 and moves the nametag onto it. The old list now has no names on it, its count hits 0, and CPython frees it on this line, not later.
6. **This time the printed address is different** It now reads 0x7f70a448. Same variable name, near-identical contents on screen, but a different object. The nametag was moved and nothing was edited.
7. **Summary: editing keeps the object, assigning makes a new one** nums.append(40) kept one object and changed it, so every name on that object saw the change. nums = [...] made a second object and moved one name to it, leaving any other name on the old one. Both lines end with nums holding five numbers, and that's the trap. The address is the only thing that tells you which of the two just happened.
## Check yourself (quiz)
**Q: 1. After x = 42 then y = x, what is true?**
- y holds a copy of 42 in a separate box
- x and y are two names pointing at the same integer object (correct)
- y points at x, which points at 42
Why: Assignment binds a name to an object. y = x copies the reference, not the integer.
**Q: 2. a = [1, 2] then b = a then b.append(3). What is a?**
- [1, 2, 3], because both names are on one list (correct)
- [1, 2], because append only changes b
- An error, because lists cannot be shared
Why: Mutation changes the object. Every alias sees it. Use b = a.copy() if you need a separate list.
**Q: 3. When does CPython free an object that is not in a reference cycle?**
- At the end of the program only
- The next time you call gc.collect()
- As soon as its reference count hits zero (correct)
Why: Reference counting frees the object straight away. The cyclic collector is only needed when objects point at each other.
**Q: 4. a = [1, 2], b = a, then del a. What happens to the list?**
- It's freed immediately, because del destroys the object
- Nothing. Only the name a is gone, and b still points at the list (correct)
- Python raises an error because b still uses it
Why: del removes a name and drops one reference. The list still has one name on it, so its count is 1, not 0, and it stays.
**Q: 5. s = "hi", t = s, s += "!". What is t?**
- "hi", because strings are immutable, so += built a new string and moved only s (correct)
- "hi!", because t is an alias and sees the change
- An error, because you cannot change a string
Why: No string object was edited. s += "!" made a new object "hi!" and rebound s; t still wears its nametag on the old "hi".
**Q: 6. x = [1, 2], y = x, then x = x + [3]. What is y?**
- [1, 2, 3], because y is an alias of x
- An error, because x was reassigned while y used it
- [1, 2], because x + [3] built a new list and rebound x, while y stayed on the old one (correct)
Why: x + [3] is a rebinding: a third list appears and only x moves. If the line had been x += [3], the shared list would have been mutated and y would show [1, 2, 3].
---
# Session 02: Functions, Scope & the Call Stack
URL: https://inboxpraveen.github.io/Py-Internals/sessions/02-functions/
Summary: Watch a call frame appear and disappear. Local scope, LEGB lookup, arguments passed by reference, closures with real cells and the mutable default trap.
Calling a function is like opening a temporary desk, doing the work, then packing it away. We'll watch that desk appear, then get precise about function objects, frames, arguments, returns and closures.
**Learning goal**
## A call opens a temporary desk, then clears it away.
That's the everyday picture. The technical steps are: create a function object, create a call frame, bind the arguments to local names, run the body, return an object, then remove the frame.
**Recommended path**
**Watch stack frames.** New function calls appear as frames above global scope.
**Track names separately.** Global and local names may have the same spelling but live in different frames.
**Follow object identity.** Arguments and returns pass references to objects, not boxes of copied values.
## What happens when Python calls a function?
**In plain words**
You write `add(2, 3)` and expect an answer. Behind the scenes Python pulls up a short-lived desk, does the maths there, hands the answer back, and clears the desk. Local names live on that desk, and only for as long as it exists.
That desk is a **frame**. Python creates it on the call, runs the body inside it, then removes it on return. Every time, without exception. The objects that were on it can outlive it, but the frame itself never does.
**Step 1**
### Function object
Running `def` creates a function object and binds a name to it. The body is stored for later. It doesn't run at definition time.
**Step 2**
### Call frame
Calling the function creates a new frame. Parameters become local names in that frame,
each bound to the argument objects.
**Step 3**
### Return path
`return` hands an object back to the caller, and then the frame is removed, always. If an inner function captured a name, that one name lives on in a separate little box the inner function carries. The desk is still cleared.
**A frame is Python's desk for one call**
A frame is where Python tracks the local names of one call, the line it is currently on,
and the way back to the caller. The exact low-level layout belongs to the runtime, but the mental model is stable. Each active call has its own frame, calls stack up, and the top frame is the one running right now.
---
Scope
## Local names are separate from global names
**In plain words**
A name inside a function isn't automatically the same as a name outside, even if they're spelled the same. Think of two sticky notes that both say `message`, stuck on two different desks.
A **scope** is where Python looks up a name. Reading works outward
through four rings, and the first one that has the name wins:
**L**ocal, **E**nclosing functions, **G**lobal,
**B**uilt-ins, or **LEGB** for short. Assignment is the opposite: it always
writes to the innermost ring unless you declare `global` or `nonlocal`.
Here are all four rings in one program. There's nothing special about `len`. It's just a name, and Python has to walk out to the last ring to find it:
```python
TAX = 0.2 # G: global
def outer():
rate = 0.1 # E: enclosing, as seen from inner
def inner():
total = 100 # L: local to inner
return len(str(total)) * rate * TAX
return inner
```
**L · inner**
its own locals, where only `total` lives
miss
**E · outer**
the enclosing function, which holds `rate` (that one *is* found this way)
miss
**G · module**
the global frame, which holds `TAX` and `outer`
miss
**B · builtins**
the last ring, always there: `len`, `print`, `range`, `str`
hit
`rate` stops at ring E and `TAX` stops at ring G. That E ring isn't a footnote. It's the machinery behind closures, which demo 4 takes apart. And because the last ring is a real namespace you can shadow, writing `list = [1, 2]` at the top of a file hides the built-in `list` for the rest of that module.
**Reading**
### Name lookup searches outward
If a name is not found in the local frame, Python looks in enclosing scopes, then globals,
then built-ins. That is why a function can read a global constant.
**Assigning**
### Assignment creates a local name by default
If you assign to `message` inside a function, Python treats it as local unless
you declare `global message` or `nonlocal message`.
**Same spelling does not mean same variable**
A global `message` and a local `message` can both exist at the same time.
They are different names in different frames. The animation above shows both names pointing
to different string objects.
---
Arguments
## Arguments bind parameter names to existing objects
**In plain words**
Passing a list into a function is like handing someone your notebook rather than a photocopy. If they write on a page, you'll see it. If they put the notebook down and pick up a new one, yours is unchanged.
When you call `add_item(bag)`, Python doesn't copy the list. It binds the local name `items` to the same object `bag` already refers to. Mutation is visible to the caller. Rebinding the parameter isn't.
| Operation inside function | What changes? | Caller sees it? |
| --- | --- | --- |
| `items.append("x")` | The list object is mutated in-place. | Yes, because caller and parameter point to the same list. |
| `items = ["x"]` | The local name is rebound to a new list. | No, the caller's name still points to the original list. |
| `return items` | The referenced object is handed back to the caller. | Only if the caller stores or uses the returned object. |
### How Python decides which name gets which object
The binding above is the same whatever the call looks like, but which parameter name each object lands on depends on how you wrote the call. Python fills the desk in a fixed order: positional arguments first, left to right, then anything passed by keyword, then defaults for whatever is still unfilled. A `*name` parameter sweeps up the
leftover positionals into a tuple, and `**name` sweeps up the leftover keywords
into a dict.
```python
def order(item, qty=1, *extras, note=None, **tags):
...
| The call | What lands on the desk | Why |
| --- | --- | --- |
| `order("tea")` | `item` → `"tea"`, `qty` → `1`, `note` → `None`, `extras` → `()`, `tags` → `{}` | One positional fills `item`; the rest fall back to defaults, and the two sweepers come up empty. |
| `order("tea", 2)` | `item` → `"tea"`, `qty` → `2` | Positionals are matched strictly left to right, so the second one goes to the second parameter. |
| `order(qty=2, item="tea")` | Identical to the row above | Keywords are matched by name, so their order does not matter. Written the other way round, `order(qty=2, "tea")`, it's a syntax error, because positionals may never follow keywords. |
| `order("tea", 2, "hot", "large", note="rush", size="L")` | `extras` → `("hot", "large")`, `note` → `"rush"`, `tags` → `{"size": "L"}` | Leftover positionals collect into the `*extras` tuple; `note` is named so it binds directly, and the unrecognised `size` collects into the `**tags` dict. |
| `order(*pair, **opts)` | Same as spelling the items out by hand | At the call site the stars mean "unpack": `*` spreads a sequence into positionals, `**` spreads a dict into keywords. Same two stars, opposite direction from the `def`. |
```
Nothing here changes the model. However an object arrives, the result is the same:
a local name on the call's desk, pointing at an object the caller already had.
`*extras` and `**tags` are the one exception worth noting. Those two objects, a fresh tuple and a fresh dict, really are built for that one call.
---
Closures
## Closures let a function remember an outer name
**In plain words**
Usually, when a function finishes, the desk is cleared and its notes go with it. A closure is the exception. Before the outer desk is cleared, the inner function takes one note along and keeps it in its pocket.
That pocket is real and it has a name. When Python compiles an outer function
and sees that an inner one uses one of its names, it doesn't store that name on the desk at all. It puts it in a **cell**, a one-slot box that the inner function object holds a reference to. You can look at them:
`inc.__closure__` is a tuple of cells, one per captured name, and
`inc.__closure__[0].cell_contents` is the value in the first one.
The frame is still removed on return. Only the cells survive.
**`nonlocal` means "write through to the cell"**
Reading an outer name needs no declaration. That's the **E** in LEGB, the enclosing ring. `nonlocal count` is only needed to *assign*:
without it, `count = count + 1` would make a brand-new local name inside
`inc` (and fail, because it would be read before assignment). With it,
both the read and the write go straight to the shared cell. Step demo 4 and watch
`inc()`'s desk: it stays empty, because `inc` has no locals at all.
**Closures capture the cell, not the value: the loop trap**
`fs = [lambda: i for i in range(3)]` then `[f() for f in fs]`
gives `[2, 2, 2]`, not `[0, 1, 2]`. All three functions share
*one* cell for `i`, and by the time you call them the loop has
finished and that cell holds `2`. This is the closure bug people actually hit: callbacks registered in a loop all firing with the last value. The fix is to
copy the value at definition time with a default argument,
`lambda i=i: i`, or with `functools.partial(lambda i: i, i)`.
Both give `[0, 1, 2]`.
### What you should understand by the end
| Concept | Plain-English meaning |
| --- | --- |
| **Function object** | The object created by `def`; a name can point to it like any other object. |
| **Frame** | The temporary desk (one namespace) belonging to one active call. |
| **Local scope** | The names that belong to a particular call frame. |
| **Argument passing** | Parameter names are bound to the objects supplied by the caller. |
| **Closure** | An inner function object plus the **cells** holding the outer names it captured. Reachable at `fn.__closure__`. |
---
Default arguments
## Mutable defaults are created once
**In plain words**
The empty list in `def add_item(item, bag=[])` is packed when you *write* the function, not each time you call it. Every call that skips
`bag` reuses that one list, so it keeps growing.
Default values are evaluated at definition time and stored on the function object. Immutable defaults (numbers, strings) are safe to share. Mutable ones are the classic surprise.
**The safe pattern**
Use `None` as the default and create a new list inside the function:
`def add_item(item, bag=None):` then `if bag is None: bag = []`.
Immutable defaults such as numbers and strings are safe to share.
**Try this in a REPL**
Define `def add_item(item, bag=[]): bag.append(item); return bag`,
then call it twice with `"a"` and `"b"`. Print both results.
Then inspect `add_item.__defaults__[0]`, which is the shared list. (`__defaults__` itself is a *tuple* of every default value, so you'll see `(['a', 'b'],)` with a trailing comma. That's a one-item tuple, not an error.) Then do the same with the `bag=None` version and check `one is two`.
**What you should be able to draw**
A function object on the heap, a call frame that appears and disappears,
parameters as extra names for the caller's objects, and, for closures, the cells that outlive the outer frame and hold the names it captured.
---
Common mistakes
## Five ways this goes wrong
**In plain words**
Each of these comes from forgetting one of the three desk rules: a call gets its own desk, assignment writes on the desk you're at, and the desk is cleared on the way out.
### Assigning to a global inside a function
`count = 0` at the top of the file, then `count += 1` inside a
function, fails with `UnboundLocalError: cannot access local variable 'count'
where it is not associated with a value` (Python 3.10 words it
*referenced before assignment*). Because the function *assigns*
to `count`, Python decided at compile time that `count` is local to that desk, and the read on the right-hand side happens before anything was written there. Declare `global count` if you really mean the module name, or, more
often, return the new value instead of reaching outside.
### Forgetting `return`
A function that only `print`s hands back `None`. The error shows up one line later, as `TypeError: 'NoneType' object is not subscriptable` or `unsupported operand type(s) for +: 'NoneType' and 'int'`, when the caller tries to use the result. If a variable is unexpectedly `None`, look at
the last line of the function that produced it.
### A mutable default argument
`def add_item(item, bag=[])` creates one list when the `def` runs
and hands it to every call that omits `bag`. Demo 5 shows it growing across
calls. The fix is the pattern above: default to `None` and build the list
inside the call.
### Shadowing a built-in
`list = [1, 2]` at the top of a file, and every later `list(...)`
in that module raises `TypeError: 'list' object is not callable`. The
built-ins ring is the last one LEGB checks, so any module-level name with the same
spelling wins first. Names like `list`, `dict`, `id`,
`type` and `input` are the usual casualties.
### Calling the function when you meant to pass it
`Thread(target=work())` runs `work` right now, on the current
desk, and passes its *return value*, usually `None`, as the target.
The thread then starts and does nothing. The same slip hides in
`button.on_click(save())` and `sorted(items, key=priority())`.
A function object is a value like any other. The parentheses are what run it.
**Two questions catch nearly all of it**
**Which desk is this name on?** That finds the shadowed global and the `UnboundLocalError`. **What object came back from the call?** That finds the missing `return`, the shared default list and the function that was called by accident.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · Call and return
Watch for: A new frame appears on top, then disappears on return. The locals go with it.
```python
def add(a, b):
total = a + b
return total
answer = add(2, 3)
```
1. **Initial state: only the global frame exists** Before this code runs there is one namespace, global. No function object exists yet and no call frame has been created.
2. **def add(a, b): creates a function object** A def statement is executable code. Python builds a function object and binds the name add to it in the global namespace. The function body is not running yet.
3. **add(2, 3) creates a new call frame** Calling the function opens a fresh desk for that call. The parameters a and b are local names on it, bound to the objects 2 and 3. Those two objects aren't built here. CPython made them at startup, along with every int from -5 to 256.
4. **total = a + b creates a local name** Python evaluates a + b and gets back the same cached 5 the rest of your program uses, so nothing new is built. It then binds the local name total to that object. The name lives only on this desk. The object doesn't.
5. **return total hands an object back to the caller** The return statement doesn't return the name total. It returns the object total points to, and that object is handed back to the global line that was waiting on the call.
6. **Summary: the desk is cleared, the objects on it are not** The frame is gone, so the names a, b and total are gone with it. Every object they pointed at is still where it was: 2, 3 and 5 are cached ints that outlive any call. The only change in global is one new name, answer, pointing at the object the function handed back.
### Demo 2 · Local scope
Watch for: Assignment creates a local name, even if a global has the same spelling.
```python
message = "global"
def show():
message = "local"
return message
result = show()
print(message)
```
1. **Initial state: no names yet** This demo shows that assigning to a name inside a function creates a local binding unless you say otherwise.
2. **message = "global" binds a global name** The global frame now has a name message pointing to the string object "global".
3. **def show() creates another function object** Python binds show in the global namespace. The body still has not run, so the local message does not exist yet.
4. **show() creates a local frame** The call creates a show() frame. Inside it, message = "local" creates a new local name. It shadows the global name. It doesn't overwrite it.
5. **return message returns the local object** Name lookup starts in the local frame, so message means the local string "local". The global message is still untouched.
6. **Summary: two names, two frames, two objects** The desk is cleared, so the local message is gone as a name. The object it pointed at survives, because result now points at it too. The global message was never touched, so print(message) prints global. Same spelling, different frames, different names.
### Demo 3 · Mutable arguments
Watch for: The parameter points at the caller's list. append is visible outside. items = ... is not.
```python
def add_item(items):
items.append("notebook")
return items
bag = []
same_bag = add_item(bag)
print(bag)
print(bag is same_bag)
```
1. **Initial state: about to pass a list** Arguments are passed by object reference. The parameter name receives a reference to the same object the caller passed in.
2. **def add_item(items) creates the function** The name add_item points to a function object. No list exists yet, and the parameter items doesn't exist until a call begins.
3. **bag = [] creates one mutable list** The global name bag points to an empty list object. Lists are mutable, so this object can be changed in place.
4. **add_item(bag) binds the parameter items** No list is copied. The local name items points to the same list object as the global name bag. The refcount rises because two names now reference one object.
5. **items.append("notebook") mutates the shared list** The append changes the list object itself. bag and items point to the same object, so the caller will see the new item.
6. **return items returns the same list object** The function returns the object items refers to. That object is the original bag list, not a new one.
7. **same_bag and bag are the same object** The function frame is gone. The global names bag and same_bag both point to the same list, so print(bag) shows the item and bag is same_bag is True.
### Demo 4 · Closures
Watch for: Look at inc()'s desk: it has no locals at all. The remembered count lives in a cell that the function object carries around.
```python
def make_counter():
count = 0
def inc():
nonlocal count
count = count + 1
return count
return inc
counter = make_counter()
value = counter()
```
1. **Initial state: nothing defined yet** A closure is what you get when an inner function still needs a name from an outer function after that outer call has finished. Nothing exists yet, just the global frame.
2. **def make_counter() stores the whole body for later** Running the def builds one function object and binds make_counter to it. Everything indented under it, including the inner def inc(), is stored rather than run. No count exists yet.
3. **make_counter() puts count in a cell, not on the desk** The call opens a desk for make_counter. But Python read the body at compile time and saw that inc uses count, so count is never stored on the desk. count = 0 writes into a cell, a one-slot box that can outlive the desk. That's why make_counter's desk looks empty here.
4. **def inc() creates a function that carries the cell** The inner function object is built during the outer call. Because its body mentions count, Python hands it a reference to that same cell, not a copy of the value. From here on, one box has two users.
5. **return inc hands back the function object** The outer function returns the inner function itself, the object rather than a call to it. Nothing is copied. The same inc object on the heap is handed to the caller, cell and all.
6. **The desk is cleared. The cell is not** The outer call is over, so make_counter's frame is removed completely. Nothing about it is kept alive. What survives is a separate object: the cell, held by the inc function object, which the global name counter now points at.
7. **counter() writes through to the cell** Look at inc()'s desk: it's empty. inc has no local count, and that is what nonlocal count buys you. Reading and writing count both go straight through to the cell, so count + 1 points the cell's one slot at 1.
8. **Summary: the state lives in the cell, not in a frame** The inner desk is cleared too, and value is bound to the object inc returned. Every frame this demo opened has been removed, and the counter still works because its memory was never in a frame. That's a closure: a function object plus the cells it carries.
### Demo 5 · Mutable defaults
Watch for: One shared list for bag=[]. Then watch bag=None build a fresh list on every call.
```python
def add_item(item, bag=[]):
bag.append(item)
return bag
first = add_item("a")
second = add_item("b")
def safe_add(item, bag=None):
if bag is None:
bag = []
bag.append(item)
return bag
one = safe_add("x")
two = safe_add("y")
```
1. **Initial state: nothing defined yet** Two versions of the same function are coming up. The first has the classic bug, the second is the habit to keep. Count how many list objects each version creates.
2. **The default list is built once, at def time** Python evaluates bag=[] while it is building the function object, not on each call. That one empty list is stored on the function and handed out again and again.
3. **The first call binds bag to that same list** add_item("a") doesn't build a new list. The local name bag is bound to the list already sitting on the function, and append mutates it in place. Check the address: it's the one from the previous step.
4. **first is another name for the default list** The desk is cleared, but the function returned the default list itself, so first now points at it. The list the function will reach for next time is no longer empty.
5. **The second call appends to the same list** add_item("b") doesn't start from []. It reaches for the same stored list, which already holds "a". That's why second is ["a", "b"], and why first is too. There's only one list here, wearing three names.
6. **The cure: def safe_add(item, bag=None)** Same idea, one change. The default is now None, a single immutable object with nothing to mutate. Nothing gets stored on this function that a later call could grow.
7. **safe_add("x") builds a list inside the call** Here's the difference, on screen. bag arrives as None, the if is true, and line 10 runs bag = []: a brand-new list object at a brand-new address, belonging to this call alone.
8. **Summary: one shared list, or a fresh one per call** safe_add("y") runs line 10 again and gets a second, different list. Count the addresses. first and second are one object that keeps growing, while one and two are two separate objects. The rule behind both halves is the same: def runs its defaults once, so never let a mutable object be one.
## Check yourself (quiz)
**Q: 1. You assign message = "hi" inside a function. There is also a global message. What happens?**
- The global is overwritten
- A new local name is created in the call frame, unless you declared global (correct)
- Python raises an error because the name already exists
Why: Assignment makes a local by default. Same spelling, different frames, different names.
**Q: 2. A function does items.append("x") on a list the caller passed. Does the caller see it?**
- Yes, because the parameter is another name for the same list (correct)
- No, because arguments are always copied
- Only if the function also returns the list
Why: Mutation is visible through every alias. Rebinding items = [] would not change the caller.
**Q: 3. Why can a mutable default argument grow across calls?**
- Because Python caches return values
- Because lists are global by default
- Because the default object is created once at def time and reused (correct)
Why: The empty list lives on the function object. Each call that omits bag mutates that same list.
**Q: 4. count = 0 at module level, then def bump(): count += 1. What does bump() do?**
- Increments the global count to 1
- Creates a local count equal to 1 and leaves the global alone
- Raises UnboundLocalError, because the assignment made count local and it's read before it's written (correct)
Why: Assignment anywhere in the body makes the name local to that desk for the whole call. The count + 1 on the right then reads a local that has no value yet. global count changes the rule, and returning a value avoids the question altogether.
**Q: 5. def greet(): print("hi"), then x = greet(). What is x?**
- "hi", because the printed text is returned
- None, because a function with no return hands back None (correct)
- The function object greet itself
Why: print writes to the screen and returns None, and so does a function that falls off its last line. The value you see printed and the value handed back to the caller are two different things.
**Q: 6. fs = [lambda: i for i in range(3)], then [f() for f in fs]. What comes out?**
- [2, 2, 2], because all three closures share one cell for i, and the loop finished on 2 (correct)
- [0, 1, 2], because each lambda captured the value of i when it was created
- A NameError, because i no longer exists after the loop
Why: A closure captures the cell, not the value. There is one i cell for the whole comprehension, and by the time any lambda is called it holds 2. lambda i=i: i copies the value at definition time and gives [0, 1, 2].
---
# Session 03: Lists, Dicts & References
URL: https://inboxpraveen.github.io/Py-Internals/sessions/03-lists-dicts/
Summary: Why b = a shares a list, what a shallow copy really copies, how deepcopy differs, and why passing a dict to a function can change the caller's data.
If two variables share a list, changing one changes the other. That's not a bug. They're looking at the same list. This session makes that picture precise: containers, aliases, copies, and mutations that travel.
**Learning goal**
## A list is a tray of arrows, not a box of values.
Each slot points at some other object. Two names can share one tray. A cheap copy makes a new tray but keeps the same arrows. Once that feels true, aliases, nested mutation and function side effects stop being surprises.
**Recommended path**
**Track the container first.** Ask which names point to the list or dict object.
**Track the contents next.** List slots and dict values are references to other objects.
**Name the operation.** Is Python mutating an object, rebinding a name, or copying only one container level?
## A list or dict is a container object
**In plain words**
Putting `99` in a list doesn't swallow the number into the list. The list keeps a pointer that says "slot 0 is over there". Another name can point at the same list, and then both of you are looking at the same tray.
A container doesn't absorb its contents. It stores **references** to other objects. That Python-level model is enough to predict everyday code. You don't need CPython's internal storage layout for it.
nums
->
list @ 0x7f530010
[0] -> 99
[1] -> 20
[2] -> 30
same
->
same list object @ 0x7f530010
**Step 1**
### Name
A name such as `nums` lives in a namespace or frame. It is a label, not the list itself.
**Step 2**
### Container object
The list or dict object lives on the heap. It has identity, mutability, contents, and a refcount.
**Step 3**
### Element references
List slots and dict values point to other objects. For nested containers, those objects may be lists or dicts too.
**Containers hold references at the Python level**
The runtime uses efficient internal storage to make lists and dictionaries fast. You do not need those
low-level details to reason correctly. For everyday Python, use this model: the container has slots or
key-value entries, and those entries refer to objects.
---
Copying
## Shallow copy copies one container level
**In plain words**
A cheap copy is like getting a new folder and putting the *same papers* in it. The folder is yours, but scribbling on a paper still shows up in the other folder. A deep copy photocopies the papers too.
A **shallow copy** creates a new outer container filled with the same inner references.
The outer list or dict is independent. Nested objects are shared.
**Before copy**
### One outer list
`original = [[1], [2]]` creates an outer list. Its two slots point to two inner list objects.
**After shallow copy**
### Two outer lists, shared inner lists
`shallow = original.copy()` creates a second outer list, but both outer lists point to the same
inner lists.
| Operation | New outer container? | Nested objects shared? | Common use |
| --- | --- | --- | --- |
| `b = a` | No | Yes, because everything is the same object | Create another name for the same container |
| `b = a.copy()` | Yes | Yes | Copy a flat list or dict |
| `b = a[:]` | Yes | Yes | The classic slice copy, with the same result as `list(a)` |
| `b = list(a)` | Yes | Yes | Make a shallow list copy from any iterable |
| `b = copy.deepcopy(a)` | Yes | No, mutable nested objects are cloned | Independent nested data, used deliberately |
**Deep copy is powerful, but not always the right default**
A deep copy walks through nested objects and copies them too. That can be expensive and can be wrong
for objects that should stay shared, such as open files, database connections, or intentionally shared
configuration. Use it when you specifically need nested independence.
Immutable nested values such as ints, strings and tuples of immutables are still shared after a deep copy. That's safe, because nothing can change them.
### The multiplication trap
**In plain words**
`[[0] * 3] * 3` doesn't make three rows. It makes one row and then points at it three times. Write into any cell and the whole column seems to change, because there was only ever one list to write into.
List multiplication repeats *references*, exactly like a shallow copy. The list comprehension
runs `[0] * 3` once per row, so each row is a separate object.
```python
grid = [[0] * 3] * 3 # three names for ONE row
grid[0][0] = 1
print(grid) # [[1, 0, 0], [1, 0, 0], [1, 0, 0]]
print(grid[0] is grid[1]) # True
grid = [[0] * 3 for _ in range(3)] # a fresh row each time
grid[0][0] = 1
print(grid) # [[1, 0, 0], [0, 0, 0], [0, 0, 0]]
```
---
Functions
## Passing a container passes the object reference
**In plain words**
Hand a function your scorebook and it can write a new score on the page you already have. If it tosses that book aside and starts a blank one, your original book is still on your desk.
The parameter is a local name pointing at the same mutable object as the caller.
Mutation is visible. Rebinding the parameter is not.
**Mutating**
### Caller sees it
`scores["math"].append(95)` follows references to the original list and changes it in-place.
The caller sees the new score.
**Rebinding**
### Caller does not see it
`scores = {"math": []}` changes only the local name `scores`. The caller's
`book` name still points to the old dict.
```python
def add_score(scores):
scores["math"].append(95) # mutates caller-visible list
scores = {"math": []} # rebinds only the local name
return scores
```
### The same split shows up in `+=` versus `+`
These two lines read almost the same and do different things. On a list, `+=` is an in-place mutation, while `a = a + [...]` builds a new list and rebinds the name.
**Mutating**
### `a += [2]` edits the list
Start with `a = [1]; b = a`. After `a += [2]` you get
`a = [1, 2]` and `b = [1, 2]`, and `a is b` is still
`True`. One list, edited where it stands.
**Rebinding**
### `a = a + [2]` makes a new list
Same start. After `a = a + [2]` you get `a = [1, 2]` but
`b = [1]`, and `a is b` is `False`. The old list is
untouched; `a` simply points somewhere else now.
---
Dictionaries
## Dict keys and values are references too
**In plain words**
A dict is a labelled tray: each label points at some value. The value can be a list
you later append to. The label itself has to stay stable. If the key could change after you filed it, Python couldn't find it again.
Keys must be **hashable**, which for built-ins usually means immutable
(strings, ints, tuples of immutables). Values can be anything, including mutable lists.
| Question | Plain answer | Why it matters |
| --- | --- | --- |
| Can two names point to one dict? | Yes. `alias = profile` creates another reference. | Mutating through either name changes one shared dict. |
| Can a dict value be a list? | Yes. Values can be mutable objects. | `profile["skills"].append(...)` mutates the nested list. |
| Can a list be a dict key? | No. Lists are mutable and unhashable. | If a key could change after insertion, lookup would become unreliable. |
| Does `dict.copy()` copy nested values? | No. It is shallow. | Nested lists and dicts remain shared unless you copy them deliberately. |
**The practical rule**
Before changing a list or dict, ask who else has a reference to the same object. If another name, object or function parameter points there too, it will see the mutation.
### What you should understand by the end
**Identity**
### `is` answers "same object?"
Use identity checks while you're learning, to confirm whether two names are aliases. In normal application code, prefer `==` unless you're checking a singleton like `None`.
**Mutation**
### Object changes, names stay bound
`append`, item assignment, and dict updates change existing objects. The address stays the same.
**Rebinding**
### Name changes, object may not
`x = ...` makes a name point somewhere else. Other aliases keep pointing to the original object.
**Copying**
### Choose the copy depth intentionally
Use assignment for sharing, a shallow copy for a new outer container, and a deep copy only when the nested objects really need to be independent.
**Hashability**
### A key's hash has to hold still
`{[1, 2]: "x"}` raises `TypeError: unhashable type: 'list'`. Swap the list for a tuple and `{(1, 2): "x"}[(1, 2)]` returns `"x"`, matching on hash and `==`, not on `is`.
**Repetition**
### `* 3` repeats the arrow, not the object
`[[0] * 3] * 3` gives one row referenced three times, so `grid[0] is grid[1]` is `True`. Build rows with a comprehension when you want three separate lists.
**Try this in a REPL**
Paste this whole line: `import copy; a = [[1]]; b = a.copy(); b[0].append(2); print(a)`. It prints `[[1, 2]]`, because `a` and `b` share the inner list. Now, in the same session, run `c = copy.deepcopy(a); c[0].append(3); print(a, c)`. The `3` lands only in `c`. Shallow shares the inner list, deep doesn't.
**What you should be able to draw**
A tray of arrows, not a box of values. Two nametags on one tray.
A cheap copy as a new tray holding the same inner arrows.
---
Common mistakes
## Four ways this goes wrong
**In plain words**
Containers hold arrows, and arrows are cheap to copy. Almost every container bug is two arrows pointing at one tray when you pictured two trays, or a method returning nothing when you expected a value.
### `x = y = []`: one list, two names
Chained assignment builds the list *once* and sticks both nametags on it.
`x.append(1)` then shows up in `y`, and `x is y` is
`True`. Write two statements, `x = []` and `y = []`,
when you want two trays. The same trap wears a different costume in
`[[0] * 3] * 3`, which repeats one row three times.
### `nums = nums.sort()`
`sort()` sorts the list in place and returns `None`. It's a mutation, not a value. After that line, `nums` is `None` and the
sorted list has lost its only name. The same goes for `append`,
`reverse`, `extend` and `update`. When you want a new
object handed back, use the function form: `sorted(nums)`,
`reversed(nums)`.
### Deleting keys from a dict while looping over it
`for k in d: del d[k]` raises `RuntimeError: dictionary changed size
during iteration`. The loop is walking the dict's own slots and you are moving
them underneath it. Loop over a snapshot instead, with `for k in list(d):`, or build a new dict with a comprehension and rebind the name.
### Trusting a copy to go all the way down
`a.copy()`, `a[:]`, `list(a)` and `dict(d)`
all copy one level. Nested lists and dicts inside are still shared, so `b[0].append(99)` changes `a[0]` too, which is demo 2 all over again. Reach for `copy.deepcopy` when the nested objects really have to be yours, and only then, because it walks the whole tree.
**One question answers most container bugs**
"Is this a new tray, or the same tray with another nametag?" Compare the addresses with `id(a) == id(b)`, or just `a is b`, and the surprise usually disappears on the spot.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · List aliases
Watch for: Both names show the same list address. Mutation through one is visible through the other.
```python
nums = [10, 20]
same = nums
nums.append(30)
same[0] = 99
print(nums)
print(nums is same)
```
1. **Initial state: no names, no list yet** This demo is about one question: when two names point to one list, what changes when the list is edited?
2. **nums = [10, 20] creates one mutable list object** Python creates a list object on the heap and binds the global name nums to it. The list object owns slots for its elements. At the Python level, think of each slot as holding a reference to an element object.
3. **same = nums adds another name, not another list** Assignment copies the reference. It doesn't duplicate the list. Now nums and same point to the same heap object, so the list's refcount rises to 2.
4. **nums.append(30) mutates the shared object** append changes the existing list in place. The list's address stays 0x7f530010, and because both names point there, both names see the new element.
5. **same[0] = 99 mutates through the other name** Index assignment changes a slot inside the same list object. It doesn't matter which name you used for the mutation. same and nums are two paths to one object.
6. **nums is same is True** The final print shows [99, 20, 30], and the identity check confirms the picture: one list object, two names, and every mutation visible through both.
### Demo 2 · Shallow copy
Watch for: The outer list is new. The nested objects keep the same address, and that is what "shallow" means.
```python
original = [[1], [2]]
shallow = original.copy()
shallow.append([3])
shallow[0].append(99)
print(original)
print(shallow)
```
1. **Initial state: a shallow copy goes one level deep** A shallow copy creates a new outer container but keeps references to the same inner objects. That one sentence explains most nested-list surprises.
2. **original = [[1], [2]] creates an outer list and two inner lists** The outer list has two slots. Each slot points to an inner list object. In the diagram, the outer list cells show labels for those inner-list references.
3. **shallow = original.copy() creates a new outer list** shallow points to a different outer list object. But the inner references are reused, so both outer lists still point to the same two inner lists.
4. **shallow.append([3]) changes only the copied outer list** Appending to shallow mutates the new outer list. The original outer list is separate, so it does not grow a third slot.
5. **shallow[0].append(99) changes a shared inner list** This time Python follows shallow[0] to the inner list shared with original[0]. That inner object mutates, so both outer lists now show [1, 99] in their first slot.
6. **Summary: a shallow copy separates the container, not the contents** original prints as [[1, 99], [2]] and shallow as [[1, 99], [2], [3]]. Two outer lists, one shared inner list. When you really need independent nested objects, use the copy module: copy.deepcopy(original). Demo 5 walks through it.
### Demo 3 · Function mutation
Watch for: Mutation through the parameter is visible to the caller. Rebinding the parameter is not.
```python
def add_score(scores):
scores["math"].append(95)
scores = {"math": []}
return scores
book = {"math": []}
new_book = add_score(book)
print(book)
print(new_book)
```
1. **Initial state: arguments bind names to objects** This demo joins Session 02 up with containers. Passing a dict to a function binds the parameter name to the same dict object the caller passed.
2. **def add_score(scores) creates the function object** The function body is stored for later. Nothing inside the body runs until add_score(book) is called.
3. **book = {"math": []} creates a dict containing a list** The dict maps key "math" to a list object. The dict is mutable, and the list stored as its value is mutable too.
4. **add_score(book) creates a frame, and scores points at book** No dictionary is copied. The global name book and the local parameter scores point to the same dict object.
5. **scores["math"].append(95) mutates the caller-visible list** Python looks up "math", finds the list object, then appends to that list. The list sits inside the same dict object book uses, so the caller will see the score.
6. **scores = {"math": []} rebinds only the local name** This line doesn't replace book. It makes the local name scores point to a new dict. The original book still points to the old dict, which now contains [95].
7. **return scores hands back the new dict** The frame is gone, and the local name scores is gone with it. What travelled out is the object it pointed to at return time: the new, empty dict, which the caller now calls new_book.
8. **Summary: the mutation travelled, the rebinding did not** Two dicts, two outcomes. print(book) shows {'math': [95]} because the append changed the object the caller still holds. print(new_book) shows {'math': []} because scores = {...} only moved a local name onto a brand-new dict.
### Demo 4 · Dict aliases
Watch for: A dict value can be a list. That list is still one object, reachable from every alias.
```python
profile = {"name": "Ada", "skills": ["Python"]}
alias = profile
profile["skills"].append("debugging")
alias["city"] = "London"
print(profile)
print(alias is profile)
```
1. **Initial state: a dict is a mutable container of references** A dictionary stores key-value pairs. At the Python level the keys and values are objects, and the dict keeps references to them.
2. **profile = {...} creates a dict with a nested list value** The dict object points from key "name" to a string and from key "skills" to a list object. The list can be mutated separately from the dict itself.
3. **alias = profile shares the dict** Like lists, dictionaries aren't copied by assignment. alias becomes a second name for the same dict object.
4. **profile["skills"].append("debugging") mutates the nested list** Python first fetches the list stored under "skills", then mutates it. The dict still points to the same list object, but the list's contents have changed.
5. **alias["city"] = "London" mutates the shared dict** Adding a key-value pair changes the dict object itself. alias and profile are the same dict, so both names see the new "city" key.
6. **alias is profile is True** The final state confirms the rule: a dictionary mutation through any alias updates the one shared dictionary. To avoid sharing, make a copy on purpose, and remember that dict.copy() is shallow.
### Demo 5 · Deep copy
Watch for: The nested objects get new addresses. Independence costs a full walk of the tree.
```python
import copy
original = [[1]]
shallow = original.copy()
deep = copy.deepcopy(original)
original[0].append(2)
print(original, shallow, deep)
```
1. **Initial state: nothing allocated yet** One question drives this demo: which of the three names survives a change to the original? Nothing exists yet, so we start from an empty namespace and an empty heap.
2. **original = [[1]] creates two list objects, not one** Line 1 pulls in the standard-library copy module for later. Line 2 builds the data: an outer list whose single slot points at an inner list. Two containers, two addresses, and that's why copying has a depth at all.
3. **original.copy() shares the inner list** A new outer list appears. Its slot still points at the same inner list. This is the shallow-copy rule from demo 2, and the inner list's refcount rising to 2 is the proof.
4. **copy.deepcopy clones the inner list too** deep is a new outer list whose slot points at a new inner list holding the same value. Deepcopy walked the whole tree instead of stopping at the first level, so no nested object is shared any more.
5. **Mutating the original inner list leaves deep alone** original[0].append(2) changes the shared inner list at 0x7f5a1010, so shallow sees the 2 as well. It was pointing there all along. deep still holds [1], because its inner list is a different object at 0x7f5a4010.
6. **Summary: only the deep copy is independent** The print gives [[1, 2]] [[1, 2]] [[1]]. Three outer lists but only two inner lists: original and shallow still share one, and deep owns its own. How deep a copy goes is a choice, and this line is where you find out which one you made.
## Check yourself (quiz)
**Q: 1. After b = a.copy() on a nested list, what is shared?**
- The inner objects. Only the outer list is new (correct)
- Nothing. Copy always clones every level
- The outer list only; inner lists become integers
Why: Shallow copy fills the new container with the same references. Use deepcopy when nested independence is required.
**Q: 2. Why can a list not be a dict key?**
- Lists are too large
- Dicts can only store strings
- Lists are mutable, so they're unhashable. A key's hash must not change while it sits in the dict (correct)
Why: If a key could change after insertion, the dict could not find it again. Tuples of immutables are allowed. Identity is not required: two equal tuples work as one key even though they are different objects, because lookup goes by hash and ==.
**Q: 3. You pass a dict into a function that does scores["math"].append(95) and then scores = {}. What does the caller see?**
- An empty dict, because both operations affect the caller
- The appended score, but not the empty dict. Mutation is shared, rebinding is local (correct)
- Nothing. Function arguments are copies
Why: The parameter starts as an alias. append mutates the nested list. scores = {} only rebinds the local name.
**Q: 4. x = y = [], then x.append(1). What is y?**
- [1], because chained assignment made one list and gave it two names (correct)
- [], because each name got its own empty list
- An error, because a list cannot have two names
Why: The [] is evaluated once. Both x and y point at that single object, so a mutation through one is visible through the other. x is y is True.
**Q: 5. nums = [3, 1, 2], then result = nums.sort(). What are result and nums?**
- result is [1, 2, 3] and nums is unchanged
- Both are [1, 2, 3], because the method returns the list it sorted
- result is None and nums is [1, 2, 3], because sort mutates in place and returns nothing (correct)
Why: In-place methods return None on purpose, so you cannot mistake them for the copying kind. sorted(nums) is the one that builds and returns a new list.
**Q: 6. d = {"a": 1, "b": 2}, then for k in d: del d[k]. What happens?**
- The dict ends up empty
- RuntimeError: dictionary changed size during iteration (correct)
- Only "a" is deleted, then the loop stops quietly
Why: The loop walks the dict's own storage, and deleting a key changes that storage under it. Iterate over a copy of the keys, for k in list(d):, or build the dict you want and rebind the name.
---
# Session 04: Classes & Objects
URL: https://inboxpraveen.github.io/Py-Internals/sessions/04-classes/
Summary: What self really is, where attributes live, how bound methods work, why class attributes are shared and how inheritance is just a longer lookup.
Think of a class as a recipe and an instance as one cake. `self` just means "this cake". Once you see where the ingredients live and where the instructions live, object-oriented Python stops feeling like magic.
**Learning goal**
## Data on the cake. Instructions on the recipe.
Each instance keeps its own attributes. Methods are shared on the class and aren't copied onto every object. `self` is just that instance being passed in. Reading a name walks from the instance to the class to the parent classes.
**Recommended path**
**Watch construction.** Class first, then instance, then `__init__` writing attributes.
**Ask where the name lives.** Instance dict, class dict, or a bound method object?
**Reuse earlier rules.** Storing a list on `self` is still aliasing. Inheritance is just a longer lookup.
## What is `self`?
**In plain words**
`self` isn't a magic keyword. It means "the object I'm talking about right now". When you write `p = Point(3, 4)`, Python hands that new point into `__init__` as `self` so the function can label it.
It's the conventional name for **the instance this method is running on**. Python passes it for you on `p = Point(3, 4)` or `g.hello("Ada")`. You could spell the first parameter anything you like. The runtime only cares that it's bound to the instance.
**Step 1**
### Class object
The indented body runs first and makes the function objects. Then Python builds the class from that namespace and binds the name `Point` to it.
**Step 2**
### Instance + `__init__`
Calling the class allocates an instance, then calls `__init__(self, ...)` with that instance as `self`.
**Step 3**
### Instance `__dict__`
`self.x = 3` writes into the instance's attribute mapping. The class is unchanged.
**You could name it anything**
The first parameter of an instance method is just a parameter. Python programmers write `self` by convention, but the runtime doesn't require that spelling. What matters is that it's bound to the instance.
---
Attribute lookup
## Instance first, then class
**In plain words**
Reading a name is like asking "does this cake have that label?" If not, ask the recipe. If the recipe inherited from another recipe, keep asking. Writing a name usually just sticks a new label on *this* cake. It doesn't edit the shared recipe.
Reading `a.lives` searches `a.__dict__`, then the class, then the
method resolution order (MRO).
Writing `a.score = 10` doesn't search at all. It writes into the instance dict.
**Reading**
### Walk outward
Instance dict → class dict → bases. That is why every `Player()` can read `lives` even when the instance dict is empty.
**Writing**
### Usually stay local
`a.score = 10` creates an instance attribute. It does not change `Player.score` or `b.score`.
**`a.lives = 1` is not the same as `Player.lives = 1`**
The first shadows `lives` only on `a`. The second changes the shared class attribute.
If `lives` were a mutable list stored on the class, `a.lives.append(1)` would mutate the shared list. Lookup finds the class object, then the mutation changes it.
**Two places the simple rule bends**
"Instance first" is the right default and it covers almost everything you'll write. Two exceptions are worth knowing about. If the class defines the name as a `@property`, the class wins and the instance dict is never consulted. That's why `a.lives = 1` would raise `AttributeError: property 'lives' of 'Player' object has no setter` instead of shadowing it. Properties are **data descriptors**, and those outrank the instance. Plain methods are non-data descriptors and don't, so a value forced into `a.__dict__['speak']` would still win over the class function. The other exception is a class that declares `__slots__`, which can have no instance `__dict__` at all. Its attributes sit in fixed slots. The search order is unchanged, the box we keep drawing just isn't there.
---
Methods
## Accessing a function on an instance binds `self`
**In plain words**
The greeting function lives on the recipe, shared by every greeter. When you say `g.hello`, Python staples `g` onto that function so you don't have to pass it yourself. That staple is why the definition has two parameters and the call looks like it passes one.
`g.hello` is a **bound method**: a small object that remembers
the function and the instance. Calling it supplies `self` automatically.
| Expression | What you get | Who fills `self`? |
| --- | --- | --- |
| `Greeter.hello` | The raw function on the class | You must pass the instance: `Greeter.hello(g, "Ada")` |
| `g.hello` | A bound method | The method object already holds `g` |
| `g.hello("Ada")` | A call through that bound method | Python, automatically |
**Try this in a REPL**
After `g = Greeter()`, print `g.hello` and `Greeter.hello`.
Then compare `g.hello.__self__ is g`. You should see `True`.
---
type and __class__
## Every instance knows its class
**In plain words**
A cake doesn't carry a photocopy of the whole recipe. It carries a note that says "I was made from Point". When you ask it to do something it doesn't have on itself, it follows that note back to the recipe.
That note is `type(p)` / `p.__class__`. Methods stay on the class.
Lookup starts on the instance, then follows this pointer.
p
->
Point() @ 0x7f720010
__class__ -> Point
__dict__ { x, y }
Point
->
class Point @ 0x7f7100a0
__init__ -> function
total -> function
**In plain words**
Making a point is two jobs: first get a blank cake tin, then write the ingredients on it. You almost never handle the tin yourself. `Point(3, 4)` does both, then hands you the finished cake.
Those two jobs are `__new__` (allocate) and `__init__` (initialise).
The instance already exists by the time `__init__` gets it as `self`,
and `Point(3, 4)` hands you that instance, not whatever `__init__` returned.
**Allocate**
### When you would override `__new__`
Almost never. The cases that exist are real, though: subclassing an immutable type such as
`int`, `str` or `tuple`, where the value has to be decided at
allocation time, and singleton-style classes that return an existing object instead of a fresh one.
**Initialise**
### `__init__` must return `None`
This is enforced, not just a convention. Slip a `return self` or `return 5` into `__init__` and Python raises `TypeError: __init__() should return None, not 'int'`. Its job is to fill the object in, not to produce one.
---
Inheritance
## A subclass does not copy methods
**In plain words**
A puppy doesn't get a photocopied list of every parent trick. If it doesn't know `speak`, it asks its parent recipe. The trick still runs on *this* puppy, not on some generic animal.
`class Dog(Animal)` creates a new class and records `Animal`
in `__bases__`. The MRO is the search list:
`Dog → Animal → object`. `d.speak` walks that list, then binds
`self` to `d`.
**1 · d**
instance `__dict__`
miss
**2 · Dog**
class namespace, where `pass` copied nothing
miss
**3 · Animal**
finds `speak`, then binds `self` to `d`
hit
**`self` is still the subclass instance**
The function was found on `Animal`, but the bound method remembers `d`. That's how a base method can read subclass attributes: it receives the actual instance. If `Dog` later defines its own `speak`, the lookup stops there.
**Try this in a REPL**
After the inheritance demo, print `Dog.__mro__` and `d.speak.__func__ is Animal.speak`.
You should see the search order and `True`.
---
Pitfalls
## The bugs this model prevents
**In plain words**
If something odd happens with a class, ask three everyday questions. Did I write on this one object or on the shared recipe? Did I share a list by accident? Did I assume each object got its own copy of the methods?
Those map to lookup versus assignment, mutable class attributes, and the fact that methods live on the class, plus aliasing through `self`.
### A list on the class is one list for every instance
`class Team: members = []` then `a.members.append("Ada")` mutates
the class attribute. Lookup found the shared list; mutation changed it.
Put `self.members = []` inside `__init__` instead.
### `self.items = items` does not copy
Demo 4 is Session 03 in a new costume. If the caller still holds the list, an `append` on the instance is visible outside. Copy when you need a private container.
### Methods are not copied onto each instance
Creating a thousand `Point` objects doesn't create a thousand `__init__` functions. Instances share the class. That's why `__class__` matters.
**What you should be able to draw**
A class (the recipe) with shared methods. Instances (the cakes) with their own
`__dict__` and a `__class__` arrow back. A bound method
that staples the instance onto the function. A subclass with `__bases__`
that asks the parent instead of copying methods. And `self` in a
call frame pointing at the same instance the caller used.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · Create an instance
Watch for: Find self and p. The same address means the same instance. Both functions stay on the class, x and y land in the instance __dict__, and total reads them straight back out.
```python
class Point:
def __init__(self, x, y):
self.x = x
self.y = y
def total(self):
return self.x + self.y
p = Point(3, 4)
t = p.total()
```
1. **Initial state: only the global frame** No class exists yet. A class statement is executable code, just like def. Press Next to watch Python build the class, then the instance.
2. **The class body runs, then Point gets its name** The order matters, and it is the opposite of what most people guess. Python runs the indented body first, which creates the two function objects and collects them in a fresh namespace. Only when the body finishes does Python build the class object and bind the name Point to it. There's no moment where you can see a half-built Point.
3. **Point(3, 4) allocates an instance, then calls __init__** Calling a class does two jobs: create a new instance (with __class__ pointing at Point), then call __init__ with that instance as self. The instance dict is still empty. x and y bind to the cached 3 and 4 from Session 01, so nothing new is built for them.
4. **self.x = x writes into the instance __dict__** This isn't a local that stays on the instance. It stores a reference in the instance's attribute mapping. self still points at the same object, and the class is unchanged.
5. **self.y = y: data on the instance, behaviour on the class** The instance now holds x and y. The methods stay on the class. That split is the mental model in one line. type(p) is still Point, through __class__.
6. **p is bound to the finished instance** __init__ returns None (implicitly). The call frame disappears, and the global name p now refers to the instance. self was just another name for the same object.
7. **p.total(): the method reads back out of the instance** This is why x and y were stored on the instance. The function lives on Point, but its self is this point, so self.x and self.y reach into this instance dict and find 3 and 4. Change the point and the same shared function returns a different answer.
8. **Final state: data on the instance, functions on the class** The frame is gone and t holds 7. Look at the finished picture: one class object holding two functions, and one instance holding x and y plus a __class__ link back. Make a thousand points and you still have two function objects.
### Demo 2 · Class vs instance
Watch for: Every print here is a lookup. Watch it miss on the instance and land on the class, until a gets a lives of its own.
```python
class Player:
lives = 3
a = Player()
b = Player()
a.score = 10
print(a.lives, b.lives)
Player.lives = 2
print(a.lives, b.lives)
a.lives = 1
print(a.lives, b.lives)
```
1. **Initial state: no class yet** A class attribute is a name stored on the class object. Watch where lives lives as instances appear, and watch each print go looking for it.
2. **Class attributes live on the class object** lives = 3 is stored on Player, not copied into each instance. The 3 is one of the pre-created small integers from Session 01, shared by everything that needs a 3 and never freed.
3. **Two instances, both with empty dicts** Neither a nor b has a lives of its own. Nothing was copied down from the class. Each instance carries only a __class__ link back to Player.
4. **a.score = 10 writes only on a** Assignment on an instance doesn't go looking anywhere. It writes straight into that instance's __dict__. b has no score, and neither does the class.
5. **The first print: both reads miss, then hit the class** This is the lookup the demo exists for. a.lives checks a's own dict (only score is there, so it misses), follows __class__ to Player, and hits. b.lives takes the same walk. Output: 3 3. Both arrive at the one 3 object, not at a copy each.
6. **Player.lives = 2 repoints the shared class attribute** The class dict now refers to the 2 object instead of the 3. Notice what does not happen: the 3 isn't destroyed. CPython pre-creates every small integer from -5 to 256 and keeps them alive for the whole run. Nothing points at this one right now, and it stays anyway.
7. **The second print: one edit, both instances see it** Same walk as before, a miss on the instance and a hit on the class. But the class now points at 2, so the output is 2 2. Neither instance was touched. That's what "shared" really means. They were never holding a value, only a route to one.
8. **Final state: a shadows the name, b still walks** a.lives = 1 wrote into a's own dict, so the last print gives 1 2. a hits on itself and stops, and b walks to the class as it always did. That's the rule in one picture: reading walks outward, writing stays home, and an instance assignment hides a class attribute instead of changing it.
### Demo 3 · Bound methods
Watch for: The function lives on the class. g.hello is a new object that already remembers g as self.
```python
class Greeter:
def hello(self, name):
return "hi " + name
g = Greeter()
fn = g.hello
msg = fn("Ada")
```
1. **Initial state: no class yet** A method defined in a class body is a function object. The binding happens later, when you access it on an instance.
2. **hello lives on the class as a function** Nothing is bound to an instance yet. Greeter.hello is the raw function, and you would have to pass the instance yourself.
3. **g = Greeter(): an instance with no attributes** The instance is empty. It can still find hello by following __class__ to Greeter.
4. **fn = g.hello creates a bound method** Attribute access on the instance doesn't return the raw function. Python builds a bound method that already remembers g as self. That's why you write g.hello("Ada") with one argument, not two.
5. **fn("Ada") calls the function with self already filled in** The call frame receives self (the instance) and name ("Ada"). You didn't pass self yourself. The bound method did. Compare the addresses: self is g.
6. **msg receives the returned string** The method frame returns and disappears. The bound method can be called again later, and it still remembers g. Method binding in one sentence: a function plus a remembered instance.
### Demo 4 · Mutable on self
Watch for: The list address on self.items should match data. append mutates that one list.
```python
class Box:
def __init__(self, items):
self.items = items
data = [1]
b = Box(data)
b.items.append(2)
```
1. **Initial state: no class yet** Same rule as Session 03: storing a list on self is still aliasing. We start from an empty global frame, then build the class.
2. **class Box stores __init__ on the class** No instance yet. The function that will write self.items lives on Box.
3. **data is a list object** The name data points at a mutable list. Passing it into Box won't copy it.
4. **Box(data): self.items aliases the same list** Inside __init__, items and data are already the same object. self.items = items stores that reference on the instance. Three names reach this one list right now: the global data, the local items, and self.items. When the frame returns the local goes away and the count drops to 2.
5. **After __init__, two names still share one list** The call frame is gone. data and b.items still point at 0x7f780010. The instance didn't get a private copy.
6. **b.items.append(2) mutates the shared list** data sees [1, 2] as well. Storing a mutable object on self is the same aliasing rule you already know, reached through an attribute this time. Copy in __init__ if you need independence: self.items = list(items).
7. **Final state: one list, reached by two different routes** Count the list objects on the heap: there is one, with a refcount of 2. data reaches it by name and b.items reaches it through an attribute. An attribute isn't a container, it's another reference, so everything Session 03 taught about aliasing applies here unchanged.
### Demo 5 · Inheritance
Watch for: Dog has no speak. Follow d.__class__ to Dog, then __bases__ to Animal.
```python
class Animal:
def speak(self):
return "hi"
class Dog(Animal):
pass
d = Dog()
msg = d.speak()
```
1. **Initial state: two classes will appear** Inheritance is attribute lookup with an extra stop. The subclass doesn't copy methods. It remembers its bases and walks them.
2. **Animal.speak lives on the base class** A normal class with one function in its namespace. Dog doesn't exist yet.
3. **class Dog(Animal): a subclass with an empty namespace** Dog is a different class object. It doesn't copy speak. It records Animal in __bases__. The method resolution order (MRO) is Dog, then Animal, then object.
4. **d = Dog(): the instance's type is Dog, not Animal** d.__class__ is Dog. The instance dict is empty, and type(d) is Dog is true. Inheritance only shows up when we look up a name.
5. **d.speak misses twice, then binds self** The lookup goes instance dict (miss), Dog (miss), Animal (hit). Python then builds a bound method whose __self__ is d, not some generic Animal. The function still lives on Animal.
6. **The call runs with self bound to the Dog instance** Even though the function was found on Animal, self is d. That's how a base method can use subclass state: it receives the actual instance.
7. **msg is the returned string, and the classes didn't change** No method was copied onto d or Dog. Inheritance is a search path. If Dog later defined its own speak, the lookup would stop there and shadow the base.
## Check yourself (quiz)
**Q: 1. After p = Point(3, 4), where does the name x live?**
- As a local variable that stays in __init__ forever
- As an attribute on the instance: p.__dict__['x'] points at the 3 object (correct)
- On the class, shared by every Point
Why: self.x = x copies a reference into the instance dict. The __init__ locals disappear when the call returns.
**Q: 2. a and b are two Player() instances. You run Player.lives = 2. What do they see?**
- Only a changes, because it was created first
- Neither changes, because class attributes are copies
- Both see 2, unless one of them already assigned lives on itself (correct)
Why: Lookup misses on the instance and finds lives on the class. An instance assignment would shadow that name for that instance only.
**Q: 3. Why can you call g.hello("Ada") when hello is defined as hello(self, name)?**
- Because g.hello is a bound method that already remembers g as self (correct)
- Because Python ignores the first parameter
- Because "Ada" is assigned to self
Why: Attribute access on the instance creates a bound method. The call only needs the remaining arguments.
**Q: 4. class Team: members = [] then a = Team(); b = Team(); a.members.append("Ada"). What is b.members?**
- [], because each instance got its own list
- ["Ada"], because lookup found the shared class list and the mutation changed it (correct)
- An error, because instances cannot see class attributes
Why: Reading a.members misses on the instance and finds the class list. append mutates that one object. Put self.members = [] in __init__.
**Q: 5. d is a Dog() and speak is defined only on Animal. Where does d.speak come from?**
- Python copied speak onto d at construction
- It lives on Dog because subclasses inherit by copying
- Lookup misses on d and Dog, finds the function on Animal, then binds self to d (correct)
Why: Inheritance is a search path (__class__, then __bases__ / the MRO). The bound method still remembers the subclass instance.
---
# Session 05: Iterators & Generators
URL: https://inboxpraveen.github.io/Py-Internals/sessions/05-iterators/
Summary: How a for loop uses iter and next, what StopIteration does, how yield pauses a frame and what lazy evaluation costs in memory compared with a list.
A `for` loop doesn't walk the list itself. It asks for a bookmark, and the bookmark is the thing that remembers "I'm on item 2". The list never moves. A generator is a function that takes a nap mid-sentence and wakes up where it left off, with its notes still on the desk.
**Learning goal**
## The shelf stays. The bookmark moves.
The collection is the shelf of books. The iterator is the bookmark. A generator is a reader who dozes off mid-page and keeps a finger on the line. Once that feels true, `iter`, `next` and `yield` are just names for those two objects.
**Recommended path**
**Watch the extra object.** The shelf stays put; a new bookmark walks it.
**Name the protocol.** `iter` → `next` → `StopIteration`.
**See the paused frame.** `yield` keeps locals alive inside the generator.
## The iterator protocol
**In plain words**
A `for` loop is three polite questions: "May I have a bookmark?", "What's next?" and "Are we done?" When the bookmark says there's nothing left, the loop stops. Anything you can `for` over answers those questions.
Those questions are `iter`, `next`, and
`StopIteration`. That trio is the **iterator protocol**.
**1 · iter**
### Get a cursor
`iter(nums)` returns an iterator. The list stays a list. The iterator remembers position.
**2 · next**
### Ask for one object
`next(it)` returns a reference to the next item and advances the cursor.
**3 · StopIteration**
### Signal the end
When nothing is left, the iterator raises `StopIteration`. `for` catches it and exits.
**Iterable vs iterator**
An **iterable** can produce an iterator (`list`, `str`, `dict`). An **iterator** produces values and holds the walk state. Every iterator is also iterable, and this is the part that catches people: `iter(it)` hands back *the same object*, not a fresh one. Ask a list for a bookmark twice and you get two bookmarks. Ask a generator and you get the same spent bookmark back. That's why looping over a generator a second time gets you nothing.
---
Generators
## `yield` pauses a frame instead of destroying it
**In plain words**
A normal function finishes the whole job, then throws its notes away. A generator says "here's one answer" and takes a nap with the notes still out. The next time you ask, it wakes up on the same line.
Calling a generator function returns a generator object straight away. It doesn't run the body yet. Each `next()` resumes the suspended frame until the
next `yield`. Locals such as `n` stay alive on that object.
**Ordinary function**
### Frame dies on return
Locals are gone. To remember state you need an object, a closure, or a global.
**Generator**
### Frame sleeps on yield
`n` is still there on the next `next()`. The generator object is the handle to that sleeping frame.
**Calling a generator function is cheap**
`count_up()` doesn't compute a list of numbers. It allocates a generator. The work happens later, one `next` at a time, which is the same idea as lazy evaluation.
---
Lazy vs eager
## When does the heap pay for the result?
**In plain words**
Baking every cookie before anyone asks is *eager*. Baking one cookie
when someone reaches for the jar is *lazy*. The first way lets you
count the cookies and eat them twice. The second way saves the oven
if you only wanted the first two.
A list comprehension is eager: every value exists before you use the list. A generator expression is lazy: values appear as you pull them. Eager gives you length, indexing and reuse. Lazy saves work when you only need the first few.
| Form | What exists immediately | Can you reuse it? |
| --- | --- | --- |
| `[x * x for x in xs]` | A full list of results | Yes. Index it, loop twice, pass it around. |
| `(x * x for x in xs)` | A generator object | Once. After exhaustion, another loop sees nothing. |
| `range(1_000_000)` | A small range object, not a million ints | Yes. It is iterable and can produce a fresh iterator. |
**Try this in a REPL**
`g = (x for x in [1, 2])` then `list(g)` twice.
The second call is `[]` because the generator was exhausted.
Compare that with `r = range(2)` and `list(r)` twice, where both calls succeed.
**Exhausted is a state on the iterator, not the collection**
After a `for` loop, the list is still there. The hidden iterator is spent.
A second `for nums` creates a *new* iterator. A second loop over the same generator object doesn't, because generators are their own iterator and don't rewind.
---
Pitfalls
## The bugs this model prevents
**In plain words**
When a loop behaves strangely, ask three everyday questions. Did I move the books around while someone was reading them? Did two people try to share one bookmark? Am I treating a bookmark as if it were the shelf?
Those are: mutating a sequence during iteration, sharing one exhausted iterator
between consumers, and expecting `len()` or indexing from an object
that only knows how to hand you the next item.
### Removing items from a list while you loop over it
The iterator holds a plain integer index into a list it does not own. Delete an element and everything after it slides down one slot, but the cursor still moves forward, so it steps straight over the item that took the vacant place. With `xs = [1, 2, 3, 4]`, removing `1` on the first pass visits `[1, 3, 4]`. The `2` is skipped and no error is raised. Loop over a copy (`for x in xs[:]`) or build a new list.
### Handing the same generator to two consumers
`g = (x for x in [1, 2, 3])`, then `sum(g)` gives `6`
and `list(g)` gives `[]`. The first consumer drained it. Nothing was deleted and nothing raised. There's just nothing left to yield. If two pieces
of code need the values, materialise them once with `list(...)`, or build
a fresh generator for each.
### Asking a generator for its length or its third item
`len(g)` raises `TypeError: object of type 'generator' has no len()`
and `g[0]` raises `'generator' object is not subscriptable`. That's honest rather than stingy. The generator really doesn't know how many values are coming, because it hasn't computed them yet. Length and indexing are what you buy when you pay up front for a list.
**What you should be able to draw**
A collection (the shelf), a separate iterator (the bookmark) holding a link and an index, and, for generators, a paused frame whose locals survive between yields.
If you can draw those three, `for`, `yield`, and lazy
expressions all fit the same picture.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · for loop
Watch for: A second object appears, the iterator, with a cursor of its own. The list stays put.
```python
nums = [10, 20]
for n in nums:
print(n)
```
1. **Initial state: nothing to loop over yet** A for loop needs two objects: the collection, and a separate iterator that remembers the position.
2. **A list is iterable, but it isn't the iterator** The list object holds the values. The for loop will ask the list for a separate iterator that remembers how far it has walked.
3. **for n in nums calls iter(nums)** Python creates a list iterator. It has its own identity and an internal cursor (shown here as index). The list is unchanged.
4. **The first next() binds n to 10** The loop body runs with n pointing at 10, and the iterator advances its cursor. This is what for hides from you: n = next(it), over and over.
5. **The second next() rebinds n to 20** n is rebound: same name, different object. The list still holds both values, so 10 is alive and well. The iterator now sits past the last item.
6. **StopIteration ends the loop** The next next() raises StopIteration. The for loop catches it and stops, and the iterator, which nothing else named, is freed. Look at what survived: the list, both of its values, and n. Only the bookmark was thrown away. A second for nums would make a new one.
### Demo 2 · iter / next
Watch for: You hold the iterator. next() only moves that cursor. The list doesn't walk itself.
```python
nums = [10, 20]
it = iter(nums)
a = next(it)
b = next(it)
next(it) # raises StopIteration
```
1. **Initial state: the protocol isn't running yet** This demo spells out what for hides: ask for a bookmark, ask for a value, ask again, and finally hear "there's nothing left".
2. **One object so far: the list, and nothing walking it** This is the moment for never lets you see. There's a collection on the heap and no cursor anywhere. Nothing in this picture knows about a position yet.
3. **it = iter(nums): the bookmark appears** A second object, at its own address. It doesn't hold copies of the values. It holds a link back to the list (over) plus an index. That link is a reference, so the list's refcount goes up by one.
4. **a = next(it): one step, one object** next returns the current item and advances the cursor. a is just a name bound to 10. It has no memory of where it came from. The iterator is the thing with state.
5. **b = next(it): the same iterator, further along** Two names, two values, one shared cursor. Nothing was copied and nothing was taken out of the list. The only thing that changed in the whole picture is the number in index.
6. **The third next(it) raises StopIteration** The cursor is past the end, so the iterator says "nothing left" the only way it can, by raising. That's the signal a for loop catches for you without a word. CPython even drops the iterator's link to the list at this point, which is why nums is back down to one reference. Final tally: the list and both values survived, it is spent and will never rewind, and iter(nums) would hand you a brand-new bookmark at index 0.
### Demo 3 · yield
Watch for: The frame sleeps inside the generator. The local n survives between next calls, until the generator runs out.
```python
def count_up():
n = 1
yield n
n = 2
yield n
g = count_up()
a = next(g)
b = next(g)
next(g) # raises StopIteration
```
1. **Initial state: no generator yet** A function that contains yield is special. Calling it returns a generator object instead of running the body.
2. **def count_up creates a generator function** Because the body contains yield, calling this function will not run the body. It will return a generator object.
3. **g = count_up(): a paused call, not a result** The generator object exists, but not one line of the body has run. Python calls this state GEN_CREATED. There's no n yet, because the frame hasn't even been started.
4. **next(g) runs until the first yield** Now the body runs: n = 1, then yield n. The value 1 is handed to the caller. The frame does not disappear. It stays paused inside the generator, remembering n.
5. **The second next(g) resumes the same frame** Execution continues after the first yield: n = 2, then a yield again. Same generator object, updated local state. This is how generators remember work without globals.
6. **The third next(g): the generator runs out** There's no code after the second yield, so the generator raises StopIteration and drops its frame. n goes with it, and the state row now reads closed. This is the difference the whole session builds up to. A second for nums gets a fresh bookmark, because a list is not its own bookmark. A generator is its own bookmark, so looping over g again yields nothing at all. Notice that g itself is still here. It isn't gone, it's empty.
### Demo 4 · Lazy vs eager
Watch for: The list holds both squares before anyone asks. The generator computes one only when you call next, at a brand-new address.
```python
eager = [x * x for x in [20, 30]]
lazy = (x * x for x in [20, 30])
first = next(lazy)
```
1. **Initial state: two recipes, two different heap costs** Eager evaluation pays for every result up front. Lazy evaluation pays per next. Same squares, very different bills.
2. **A list comprehension builds every value now** Line 1 computed 400 and 900 and put both on the heap, each in its own box. eager is a list of references to them. The work is finished before you have used a single result.
3. **A generator expression stores the recipe, not the answers** One new box appears, and it isn't a number. lazy is a generator holding the plan and the inputs it hasn't reached yet. No multiplication has happened. That's lazy evaluation: pay for a value only when someone asks for it.
4. **next(lazy) computes only the first square** first is 400. Now look at the address. It is not the 400 already sitting in eager. The generator just multiplied and allocated a fresh object. Try it in a REPL: first == eager[0] is True, but first is eager[0] is False. And the 900 from this generator still doesn't exist. It's the one item still marked pending.
5. **Same picture, two bills: this is the trade** Nothing new happens here, so just read the heap. The list paid for every square on line 1 and will keep paying storage for all of them. The generator paid for one and is holding one plan. At two items that's a rounding error. At a million, [x * x for x in range(1_000_000)] is about 8.4 MB of slots plus a million int objects, while the generator expression is 200 bytes however long the run gets. Eager buys you len(), indexing, and a second loop. Lazy buys you the freedom to stop early.
## Check yourself (quiz)
**Q: 1. In for n in nums, what object remembers how far the loop has gone?**
- The list nums itself
- A separate iterator created by iter(nums) (correct)
- The name n
Why: The list is the iterable. The iterator is a different object with a cursor. A second for creates a new iterator.
**Q: 2. What happens when you call a function that contains yield?**
- You get a generator object; the body has not run yet (correct)
- The body runs to completion and returns a list
- You get the first yielded value immediately
Why: The first next() (or the first loop step) is what runs the body up to the first yield.
**Q: 3. You loop over a generator twice. The second loop prints nothing. Why?**
- Generators delete the original data
- Python forbids looping over generators
- The generator is an iterator that exhausted itself; it does not rewind (correct)
Why: A generator's iter() returns itself. After StopIteration, there is nothing left to yield unless you create a new generator.
**Q: 4. Right after eager = [x*x for x in [20, 30]] and lazy = (x*x for x in [20, 30]), what exists on the heap?**
- A full list of two squares, and a generator that has not computed them yet (correct)
- Nothing, because both wait until you loop
- Two identical lists; the parentheses are only style
Why: Square brackets are eager. Parentheses make a generator expression. Memory cost is the difference.
**Q: 5. xs = [1, 2, 3, 4], and inside for x in xs: you call xs.remove(x) when x == 1. Which values does the loop visit?**
- 1, 2, 3, 4, because the iterator took a snapshot of the list first
- It raises an error as soon as the list changes size
- 1, 3, 4, because removing 1 slides 2 into the slot the cursor has already passed (correct)
Why: The list iterator holds an index into a list it doesn't own. After the removal, index 1 holds 3, so 2 is never visited and nothing complains. Loop over a copy, for x in xs[:], when you need to remove while iterating.
**Q: 6. g = (x for x in [1, 2, 3]), then sum(g), then list(g). What does the second call return?**
- [1, 2, 3], because the generator restarts for each consumer
- [], because sum drained the generator, and a generator doesn't rewind (correct)
- A StopIteration error escapes to your code
Why: A generator is its own iterator. sum pulled every value until StopIteration, which it handled for you, and list then asks the same exhausted object and gets nothing. Build a fresh generator, or materialise the values once with list(...).
---
# Session 06: Decorators
URL: https://inboxpraveen.github.io/Py-Internals/sessions/06-decorators/
Summary: A decorator is one line of assignment. See the wrapper, the closure cell that keeps your function alive, functools.wraps, decorator factories and stacking.
A decorator looks like magic and is actually one line of assignment. You hand your function to another function, it hands back a replacement, and your old name gets pointed at the replacement. The original doesn't vanish. It's tucked inside, still doing the real work.
**Learning goal**
## `@deco` means `f = deco(f)`.
That's the whole feature. Session 01 taught that assignment moves a nametag, and Session 02 taught that a function can remember an outer name. Put those two together and you have decorators. Everything else (arguments, stacking, the lost docstring) follows from that one line.
**Recommended path**
**Do it by hand first.** Demo 1 has no `@` at all, just an assignment.
**Then add the `@`.** Demo 2 ends in the same picture. Only the order is worth watching.
**Ask where the original went.** It is alive inside a closure cell, with no name pointing at it.
## What the `@` line actually means
**In plain words**
You wrote a function. Someone hands it to a helper and says "give me back a better version". The better version goes under the old name, and your original goes in the helper's pocket. That's a decorator. The `@` symbol is just a shorter way to write the handover.
To be precise, `@deco` above `def f` compiles to `f = deco(f)`. The `def` runs first and produces a function object, that object is passed to `deco`, and the name `f` is bound to whatever comes back.
**1 · Build**
### The `def` runs
A function object appears on the heap. It has no name yet, because the decorator line hasn't finished.
**2 · Hand over**
### The decorator is called
Your function object goes in as an argument. Whatever comes out is usually a new function.
**3 · Rebind**
### The name is assigned
The name in your module now points at the returned object. Session 01's rule, applied to a function.
**A decorator is not a keyword**
There's no `decorator` type in Python. Anything callable that accepts one argument works: a function, a class, an object with `__call__`. The `@` is punctuation, not machinery.
**Try this in a REPL**
Define `def shout(f): ...` and a plain `def greet(name): ...`,
then run `greet = shout(greet)` by hand. Compare `id(greet)`
before and after. You just decorated a function without typing `@` once.
---
Inside the wrapper
## Where your original function goes
**In plain words**
Nothing is thrown away. Your function is still there, at the same address it always had. What changed is that no *name* points at it any more. It's reachable only through the wrapper that swallowed it.
The wrapper uses `func`, a name from the enclosing call. That makes `func` a free variable, so Python attaches a **closure cell** to the wrapper holding a reference to your function. The decorator's frame is discarded when it returns. The cell survives because it belongs to the wrapper.
greet
->
function wrapper @ 0x7f9101f0
closure cell -> function greet @ 0x7f9100c8
**What moved**
### The name
`greet` was pointing at your function. Now it points at the wrapper. Nothing was copied or edited.
**What stayed**
### The object
Your function still exists at the same address, with its refcount held up by the closure cell alone.
**This is Session 02's closure, doing a job**
In Session 02 a closure remembered a counter. Here it remembers a function. Same mechanism, same diagram. Only the thing being remembered has changed.
---
Identity
## The wrapper wears the wrong name tag
**In plain words**
You asked for a coat and got the coat back with someone else's name sewn in.
Your function still runs, but every tool that reads its label (`help()`, a traceback, a log line) now reports the wrapper's name instead of yours.
`functools.wraps(original)` is a decorator you apply to the wrapper.
It copies `__name__`, `__qualname__`, `__doc__`,
`__module__` and `__annotations__` from the original onto
the wrapper, updates `__dict__`, and sets `__wrapped__` to
the original object so tools can unwrap it.
| You call | Without `wraps` | With `wraps` |
| --- | --- | --- |
| `greet.__name__` | `'wrapper'` | `'greet'` |
| `greet.__doc__` | `None` | Your docstring |
| `greet.__wrapped__` | `AttributeError` | The undecorated function |
| `greet("ada")` | `'HI ADA'` | `'HI ADA'`, the behaviour is identical |
**Skipping `wraps` is a debugging tax you pay later**
It costs one line and never changes behaviour. Without it, a stack trace three months from now points at `wrapper` in every decorated function in your codebase, and they all look the same.
**Try this in a REPL**
Decorate a function twice, once with `@functools.wraps(func)` on the wrapper and once without, then compare `help()` on each. The difference is what your future self will be reading.
---
Arguments
## Why `@repeat(2)` needs one more `def`
**In plain words**
The `@` line only ever passes your function to one thing. If you also want to pass a number, you need something that takes the number first and gives back the thing that will take your function. That's the third layer.
`@repeat(2)` evaluates `repeat(2)` immediately, before any decorating happens, and the result is what gets applied to your function. So `repeat` is a **decorator factory**, not a decorator. Two calls happen at import time, and your function body still runs zero times.
**Call 1**
`repeat(2)` runs. Returns `decorator`, which remembers `times = 2`.
decorator
**Call 2**
`decorator(ping)` runs. Returns `wrapper`, which remembers `func` and `times`.
wrapper
**Bind**
The name `ping` is pointed at `wrapper`.
ping
**Parentheses tell you which one you have**
`@deco` means `deco` receives your function.
`@deco()` means `deco()` is called first and *its result*
receives your function. Mixing these up is the most common decorator bug there is, and the error message is usually a baffling `TypeError` about the wrong number of arguments.
---
Stacking
## Two decorators, two different orders
**In plain words**
Think of wrapping a parcel. The layer you put on first ends up innermost, and the last layer you add is the one someone opens first. Decorators are applied from the bottom up and entered from the top down, for the same reason.
`@bold` above `@italic` above `def text` compiles to
`text = bold(italic(text))`. The inner call happens first, so
`italic` is applied first and ends up nested inside `bold`'s
wrapper. Calling `text()` then enters `bold`'s wrapper first.
### Applied bottom-up
`italic(text)` runs, then `bold(...)` runs on its result.
The decorator nearest the `def` gets your real function.
### Executed top-down
`text()` enters bold's wrapper, which calls italic's wrapper,
which calls your function. Result: `hi`.
**What you should be able to draw**
A name, an outermost wrapper, a closure cell inside it holding the next wrapper,
and your original function at the bottom of the chain. If you can draw that, you can predict what any stack of decorators does, and where to look when one of them breaks.
---
Common mistakes
## Four ways this goes wrong
### Forgetting to return the wrapper
A decorator that defines `wrapper` but never returns it returns
`None`. Your name is then bound to `None`, and the first
call gives `TypeError: 'NoneType' object is not callable`. If you see
that error right after adding a decorator, check its last line.
### A wrapper that only accepts the arguments you tested with
`def wrapper(text)` works until someone decorates a two-argument
function. Write `def wrapper(*args, **kwargs)` and pass them through
with `func(*args, **kwargs)` unless you have a reason not to.
### Leaving out `functools.wraps`
Behaviour is fine; everything that reads a function's labels is not.
Tracebacks, `help()`, API docs and log lines all start reporting
`wrapper`. One line fixes it permanently.
### Writing `@deco()` when the decorator takes no arguments
That calls `deco` with nothing, then applies whatever it returned
to your function. If `deco` expected a function, you get a `TypeError` at import time, before any of your code runs. The reverse, `@repeat` without `(2)`, binds your function to the wrong layer without a word.
**One question answers most decorator bugs**
"After this line runs, what object does the name point at?" If the answer is `None`, you forgot a `return`. If it's your original function, the decorator didn't wrap. If it's a wrapper with the wrong signature, the `*args` are missing.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · f = deco(f)
Watch for: Watch the name greet move. It ends up pointing at a different function from the one you wrote.
```python
def shout(func):
def wrapper(text):
return func(text).upper()
return wrapper
def greet(name):
return "hi " + name
greet = shout(greet)
```
1. **Nothing has run yet** A decorator isn't new machinery. It's one line you already know how to read: take a function, hand it to another function, and give the result the old name.
2. **def shout makes a function object** def always does two things: build a function object on the heap, and bind a name to it. Nothing inside shout runs yet.
3. **def greet makes a second function object** Two ordinary functions, two objects, two names. Nothing special so far. shout just happens to expect a function as its argument.
4. **shout(greet) opens a frame, and func is a second name for your function** Calling shout creates a frame. The parameter func binds to the same object that greet points at. Same address, so the reference count goes from 1 to 2. This is Session 02's rule, unchanged.
5. **def wrapper builds a third function that remembers func** This def runs during the call. Because wrapper uses func, Python attaches a closure cell to it, the same remembered-state box from Session 02. The cell holds your original function.
6. **return wrapper: the frame goes, the wrapper stays** The shout frame is thrown away, so the local name func disappears. The closure cell doesn't. It's part of the wrapper object, and the wrapper is being returned.
7. **The name greet is rebound, and that's the trick** Assignment moves a nametag, just as in Session 01. greet now points at the wrapper. No name points at your original function any more. The only thing keeping it alive is the closure cell inside the wrapper. Call greet("ada") and you get "HI ADA".
### Demo 2 · The @ shorthand
Watch for: Same ending as demo 1, so watch the order instead. The function is built first, decorated second, named last.
```python
def shout(func):
def wrapper(text):
return func(text).upper()
return wrapper
@shout
def greet(name):
return "hi " + name
```
1. **This ends in the same place as demo 1** Nothing new happens here. The only difference is that Python writes the greet = shout(greet) line for you. Watch the order of the three moments.
2. **def shout: a decorator is just a function** Anything callable that takes one function and returns something can sit after an @. There's no special "decorator type" in Python.
3. **Moment 1: the body under @ is built first** Python reads the def and creates the function object. Notice what's not on the screen: a name. The object exists before anything is called with it.
4. **Moment 2: that object is passed straight to shout** The @shout line means "call shout with whatever this def just made". Same frame, same closure cell, same wrapper as in demo 1.
5. **Moment 3: the name greet is bound to the result** Only now does the name appear, and it points at the wrapper. Compare this picture with the last step of demo 1. They're identical. @shout saved you one line and changed nothing else.
6. **Say it in one sentence** @deco above a def means name = deco(name). Every later question (arguments, stacking, lost docstrings) is a question about that one line.
### Demo 3 · functools.wraps
Watch for: Read the __name__ row on the function object. It's wrong until functools.wraps fixes it.
```python
import functools
def shout(func):
@functools.wraps(func)
def wrapper(text):
return func(text).upper()
return wrapper
@shout
def greet(name):
"Say hello."
return "hi " + name
```
1. **A wrapper is a different object, so it has different labels** Your original function carries a name, a docstring and a module. The wrapper is a brand new object and carries its own, which say wrapper and nothing else.
2. **Without help, the wrapper introduces itself as "wrapper"** Here is the plain wrapper before wraps runs. help(), tracebacks, log lines and debuggers all read these labels, so all of them would now say the wrong thing.
3. **@functools.wraps(func) is itself a decorator** It's applied to wrapper the same way @shout is applied to greet. wraps(func) is a factory: it returns a small decorator that already knows which function to copy the labels from.
4. **It copies the labels across, and leaves a trail back** Five labels are copied: __name__, __qualname__, __doc__, __module__ and __annotations__. It also sets __wrapped__, a direct reference to the function underneath, so tools can still find the real thing.
5. **Same wrapper, honest name tag** greet still points at the wrapper, so you still get "HI ADA". But greet.__name__ is now 'greet', help(greet) shows the real docstring, and greet.__wrapped__ hands you the undecorated function back.
### Demo 4 · Decorator with an argument
Watch for: Count the calls. repeat(2) runs first and returns a decorator. Only then is anything decorated.
```python
def repeat(times):
def decorator(func):
def wrapper():
return [func() for _ in range(times)]
return wrapper
return decorator
@repeat(2)
def ping():
return "ping"
```
1. **Why three levels of def?** Because @ only ever calls one thing with your function. If you also want to pass a 2, something has to accept the 2 first and hand back a decorator. That's the extra layer, and nothing more.
2. **def repeat: this isn't the decorator yet** repeat is a decorator factory, a function whose job is to produce a decorator. What goes after @ is whatever repeat(...) returns.
3. **Call 1: repeat(2) runs and remembers the 2** A frame opens with times bound to 2. Inside it, def decorator creates a function that uses times, so it gets a closure cell holding that 2.
4. **The repeat frame closes, and decorator is what @ will use** The line @repeat(2) has now finished its first half. What sits above def ping is this decorator object, and it already knows the number 2.
5. **Call 2: def ping builds the function, then hands it over** The same three moments as in demo 2. The ping object is created, passed to decorator, and decorator's frame opens with func pointing at it.
6. **wrapper gets two remembered names** It uses func (from this frame) and times (from the frame above), so its closure holds both. This is what the extra layer was for: the argument you wrote in @repeat(2) travelled down two levels and is still there.
7. **The name ping lands on the wrapper** Now ping() returns ['ping', 'ping']. Two calls happened while the file was being read, repeat(2) and then decorator(ping), and neither of them ran your function body.
### Demo 5 · Stacking two
Watch for: Applied bottom-up, executed top-down. Two different orders, and both are visible in the diagram.
```python
def bold(func):
def wrapper():
return "" + func() + ""
return wrapper
def italic(func):
def wrapper():
return "" + func() + ""
return wrapper
@bold
@italic
def text():
return "hi"
```
1. **Two decorators, stacked** The question everyone asks is which one runs first. The answer depends on whether you mean "applied" or "called", and both answers fall out of name = deco(name).
2. **Both decorators are just functions on the heap** Neither has done anything yet. They only act when something is handed to them.
3. **def text builds your function** As always, the body under the decorators is created first. It isn't named text yet.
4. **The nearest decorator goes first: @italic** Decorators are applied from the def outwards, so the bottom line goes first. italic(text) returns a wrapper whose cell holds your original function.
5. **Then @bold wraps the wrapper** bold never sees your function. It receives the italic wrapper, and its own cell holds that. You now have a chain three objects deep.
6. **The name text lands on the outermost wrapper** The decorator closest to def is applied first, so the one furthest away ends up outermost. That's why the top line is the last one applied and the first one entered.
7. **Calling text() walks the chain outside-in** Bold runs, calls italic, which calls your function. The result comes back through the same chain in reverse: hi. Applied bottom-up, executed top-down, in one picture.
## Check yourself (quiz)
**Q: 1. What does @shout above def greet compile to?**
- It edits the body of greet in place
- greet = shout(greet), so the name is rebound to whatever shout returns (correct)
- It registers greet so shout runs before every call, without changing any name
Why: A decorator never modifies your function. It calls something with it and rebinds the name to the result. If that result isn't callable, the next call fails.
**Q: 2. After decorating, what keeps the original undecorated function alive in memory?**
- Nothing. It's garbage collected, and the wrapper holds a copy of its code
- The module keeps a hidden second name for it
- The wrapper's closure cell holds a reference to it (correct)
Why: No name points at it any more, but its reference count isn't zero, because the wrapper's closure cell still refers to it. That's the only reason calling the wrapper still works.
**Q: 3. A decorated function reports __name__ == 'wrapper'. What fixes it?**
- Apply @functools.wraps(func) to the wrapper (correct)
- Return the original function instead of the wrapper
- Rename the inner function to match the decorated one
Why: wraps copies __name__, __doc__ and friends onto the wrapper and sets __wrapped__. Renaming the inner def would only work for one function; wraps works for all of them.
**Q: 4. In @repeat(2), when does repeat(2) actually run?**
- Every time the decorated function is called
- Once, while the file is being read, before the function is decorated (correct)
- Never; @ passes the 2 straight through to the wrapper
Why: repeat(2) is evaluated first and must return a decorator. That decorator is then called with your function. Two calls at import time, and your function body runs in neither.
**Q: 5. With @bold written above @italic, which decorator receives your original function?**
- bold, because it is written first
- Both receive it, and the results are combined
- italic, because it is closest to the def (correct)
Why: It compiles to text = bold(italic(text)). The inner call runs first, so italic gets your function and bold only ever sees italic's wrapper.
**Q: 6. A decorator defines wrapper but its last line is missing. What happens?**
- The name is bound to None, so the next call raises TypeError: 'NoneType' object is not callable (correct)
- Python falls back to the undecorated function
- A SyntaxError at the @ line
Why: A function with no return returns None, and f = deco(f) binds f to that. The failure shows up at the first call, not at the decorator.
---
# Session 07: The GIL & Concurrency
URL: https://inboxpraveen.github.io/Py-Internals/sessions/07-gil/
Summary: Why CPython has a global interpreter lock, why it doesn't make your code thread-safe, and when threads, multiprocessing or asyncio actually help.
Python can start as many threads as you like, and will run one of them at a time. That single sentence explains why your threaded program got slower, why your counter came out one short, and why the fix is sometimes a lock, sometimes a second process, and sometimes neither. Here's the lock itself, and what it's actually protecting.
**Learning goal**
## The GIL exists because of `refcount`.
Session 01 showed that every object carries a count of how many names
point at it. That count is a plain number in memory. Let two threads change it at the same instant and it goes wrong without a sound. Too high and the object never gets freed, too low and it gets freed while you're still using it. CPython's answer is one lock around the whole interpreter. Everything else in this session follows from that.
**Recommended path**
**Watch one row.** Demo 1 adds a grey box at the bottom of the memory panel. The `GIL` line in it names the only thread allowed to run.
**Lose a deposit.** Demo 2 shows a correct-looking line of code dropping an update, with the GIL working perfectly the whole time.
**Then pick a tool.** Demos 4 to 6 are three different escapes: release the lock, own a second one, or never contend for it at all.
## Why there is a lock at all
**In plain words**
Imagine a room full of people and one microphone. Anyone can be in the room,
but only the person holding the microphone may speak. It looks like a strange
rule until you hear the room with four people talking at once. Python's microphone is called the **Global Interpreter Lock**, and the noise it prevents isn't people talking. It's threads editing the same numbers.
Every object in CPython carries a reference count, and Session 01 showed you it moving. Incrementing that count isn't one indivisible machine operation. It's a read, an add and a write. Two threads doing that at the same
instant can both read `4` and both write `5`, so one
increment is lost. A count that ends up too high leaks the object forever.
A count that ends up too low frees it while your code is still holding it,
and the next line reads memory that no longer belongs to Python.
**1 · The problem**
### Shared bookkeeping
Refcounts, the small-integer cache, the free lists, the import machinery.
Every thread touches all of it, constantly, on almost every line.
**2 · The obvious fix**
### A lock on every object
Correct, and expensive. You'd pay for locking on every operation in every program, including the single-threaded ones, which is most of them.
**3 · The choice made**
### One lock, held by one thread
Nearly free when one thread runs, and it makes the whole interpreter safe in one go. The bill only arrives when you want two threads computing.
**The GIL is a property of CPython, not of Python**
The language doesn't require it. Jython and IronPython never had one, PyPy has one for the same reasons CPython does, and CPython itself can now be built without it (see A future without it). "Python is single-threaded" is wrong twice over. The threads are real OS threads, and the restriction belongs to one implementation.
**Try this in a REPL**
`import sys`, then `a = [1, 2, 3]` and
`sys.getrefcount(a)`. Bind a second name, `b = a`, and
look again. That number is the thing the GIL protects, and it changes on almost every line you write.
---
The handover
## Five milliseconds, then pass the microphone
**In plain words**
Nobody gets to hold the microphone forever. A timer runs, and when it goes
off the current speaker is asked to hand it over. They don't stop mid-syllable. They finish the word they're on. And if they walk off to fetch something, they hand it over straight away without being asked.
The interpreter sets a flag when a thread has held the GIL for `sys.getswitchinterval()` seconds, which is **5 ms** by default. The running thread only notices that flag at certain points in the bytecode, chiefly a jump back to the top of a loop and a call into a function. It's never checked in the middle of a short straight run of bytecodes. A thread that instead calls something that waits (`time.sleep`, a socket read, a disk write) releases the GIL itself, before it starts waiting.
**Tick**
5 ms have passed. Another thread is waiting, so the interpreter raises a "please drop it" flag.
requested
**Safe point**
The running thread reaches a loop jump or a call and notices the flag. The current bytecode always finishes.
noticed
**Release**
It drops the GIL and queues up again behind everybody else. Its frame, its locals and its half-finished line all stay exactly as they were.
suspended
**Reacquire**
Some other thread takes it and runs. Eventually the first one gets it back and carries on from the same bytecode.
resumed
**Nothing can force a handover that never reaches a safe point**
A C extension that goes away for two seconds without releasing the GIL
freezes every other Python thread for two seconds, and there is no timer
that can help. This is why a well-behaved extension releases the lock around its own long computations, and why a badly-behaved one makes a whole web server stutter.
**Try this in a REPL**
`sys.getswitchinterval()` prints `0.005`.
`sys.setswitchinterval(0.05)` makes threads swap ten times less
often, and `sys.setswitchinterval(0.0001)` ten times more. Time a
two-thread CPU loop at each setting. Faster swapping doesn't buy you throughput. It buys you overhead.
---
The dangerous half-truth
## "The GIL makes Python thread-safe" is false
**In plain words**
The microphone rule stops two people speaking over each other. It does not
stop one person reading a number off the whiteboard, being interrupted, and
writing their answer back after somebody else has already changed it. The
number they wrote is arithmetically correct and completely wrong.
The GIL guarantees that one *bytecode* completes before another begins. It makes no promise about one *line* of your source, which is almost always several bytecodes. `acct.set(acct.get() + 1)` compiles to a load, a call, an add and another call, and a call is one of the places the interpreter checks whether to switch threads.
| What the GIL promises | What it does not promise |
| --- | --- |
| One bytecode finishes before the next starts | One line of your code finishes before another thread runs |
| Reference counts stay correct | *Your* counters, totals and caches stay correct |
| The interpreter never corrupts its own structures | Your dictionary is in a sensible state halfway through an update |
| You will not get a segfault from plain Python code | You will not get a wrong answer |
**Why it is easy to miss**
### It usually works
Run `count += 1` in a tight loop across four threads on CPython 3.13 and you'll very likely get the right total. It did six runs out of six while I was writing this session. The whole read-add-write sits between two safe points, so nothing gets in.
**Why that is worse**
### Until it does not
Move the same logic into a getter and a setter and the calls open a gap.
The same script, on the same machine, lost **139,004** of a million increments on one run, and lost none on the next.
**A bug that appears one run in three is worse than one that appears every time
It passes your tests. It passes code review. It shows up as a support ticket six weeks later, on a busier machine, and it won't reproduce on yours. Don't reason about whether a switch *will* land in your gap. Remove the gap.
### Removing the gap
A `threading.Lock` is a plain object with two states. One thread at a time may hold it, and anybody else asking is parked until it's released. Always take it with `with`, so it's released even when the body raises.
```python
# the gap is open: another thread can run between get() and set()
acct.set(acct.get() + 1)
```
# the gap is closed: nobody else can be inside these lines
with lock:
acct.set(acct.get() + 1)
Thread-1
->
holds the GIL and holds the Lock
running, and safe from interference
Thread-2
->
waiting on the Lock @ 0x7fa300c0
not in the GIL queue at all, a different wait
Thread-3
->
waiting for the GIL
ready to run, just not chosen yet
Those are two different queues, and demo 3 shows both at once. Getting them confused is how people conclude that "locks do nothing in Python because there's already a lock".
**Usually you should not be writing locks at all**
`queue.Queue` is already correct: producers `put`,
consumers `get`, and no shared variable exists for anyone to
race on. `concurrent.futures` hands results back as return
values. Shared mutable state plus hand-written locking is the last resort, not the default.
**Try this in a REPL**
Write the `Account` class from Demo 2, run it across four
threads doing 100,000 deposits each, and print the total. Run it ten times.
Then wrap the line in `with lock:` and run it ten more times. The first set of numbers is the lesson.
---
Choosing a tool
## Is your code waiting, or is it thinking?
**In plain words**
If your program is slow because it keeps waiting for other people (a website, a database, a disk), then doing all the waiting at once helps enormously, and that's what threads do. If it's slow because it's thinking hard, threads can't help, because thinking needs the microphone.
A thread that blocks on I/O has released the GIL, so every other
thread is free to run. Four one-second sleeps across four threads finished in
**0.50 s** instead of **2.01 s**. A thread
that computes never releases it voluntarily, so two of them simply take turns
and pay the swapping cost on top: the same work took **1.81 s**
serially and **2.49 s** across two threads.
| Your bottleneck | Reach for | Why it works | What it costs |
| --- | --- | --- | --- |
| Waiting on network, disk, database, a subprocess | `threading`, or `ThreadPoolExecutor` | Every blocking call releases the GIL, so the waits overlap | Shared objects mean you now own the race conditions |
| Thousands of simultaneous network waits | `asyncio` | One thread, one loop, no OS thread per connection | Async all the way down, since one blocking call stalls everything |
| Heavy computation in pure Python | `multiprocessing`, or `ProcessPoolExecutor` | A second interpreter has a second GIL, so a second core is usable | Startup time, and every argument and result is copied |
| Heavy computation inside NumPy, hashing, compression | `threading` is fine here | Those libraries release the GIL while they work in C | Only applies while you are inside the library, not around it |
**You can test the answer by changing one word**
`concurrent.futures` gives you `ThreadPoolExecutor`
and `ProcessPoolExecutor` with an identical API. Write the code
against one, swap the class name, and time both. That measurement beats any argument about what your bottleneck "should" be.
**Waiting** overlaps: threads and `asyncio` both turn
*n* waits into roughly one wait.
**Thinking** does not overlap inside one interpreter: threads
interleave it and add overhead. Only a second process adds a second core.
**Neither** fixes an algorithm. Four processes on an
*O(n²)* loop is still an *O(n²)* loop.
---
Separate rooms
## A second process brings its own lock, and its own memory
**In plain words**
If the trouble is that everyone is sharing one microphone, build a second room with its own. The catch is immediate and total: nothing said in one room is heard in the other. Anything you want the other room to know has to be written down, carried across, and read out again.
`multiprocessing.Process` starts a second operating-system process running a second CPython interpreter, with its own heap and its own GIL. The two really do execute Python at the same instant. The benchmark that took **1.81 s** serially and **2.49 s** across two threads took **1.08 s** across two processes.
Arguments and results are **pickled** to bytes and rebuilt on the
other side, so the child gets equal values at different addresses, never the
same objects.
### A thread shares your objects
Same heap, same addresses, same `id()`. Mutate a list in one thread and every other thread sees it, which is convenient, and is also where every bug in the section above comes from.
### A process shares your values
Different heap, different addresses. Mutate the copy in the child and the parent never finds out. No locking needed, because there's nothing shared left to lock.
### The guard is not optional
To reach your `child` function, the new process has to import your module, which re-runs every line at the top level of it. Without the
guard, that includes the line that starts a process, so the child starts a
grandchild, and so on until something gives out.
```python
import multiprocessing as mp
def work(n):
return n * n
if __name__ == "__main__": # required, not a style choice
print(mp.get_start_method()) # 'spawn' on Windows and macOS
with mp.Pool(4) as pool:
print(pool.map(work, range(10)))
```
**Run this from a file, never from the REPL**
Where the start method is `spawn`, the child imports `__main__`, and an interactive session isn't importable.
Every `multiprocessing` example in this session belongs in a
`.py` file. Check what your platform does with
`mp.get_start_method()` before you assume.
### Getting something back
Since mutation does not travel, the child has to hand results over explicitly.
In rough order of how often you should reach for them:
**Best**
Return a value from the function and let `Pool.map` or `ProcessPoolExecutor` collect it.
return
**Streaming**
A `multiprocessing.Queue` or `Pipe`, for results that arrive as they are produced.
Queue
**Large data**
`shared_memory` or a `Manager`, when copying is what is actually costing you.
shared
**Never**
Mutating a global and hoping. It works in your fork-based test on Linux and stops working elsewhere without telling you.
no
**Everything you send has to be picklable, and pays for the trip
Lambdas, nested functions, open file handles, database connections and
locks cannot be pickled, and you find out when you submit the job. Even
when it works, a process pool over a large array can spend more time copying than computing, which is why the benchmark above used a task big enough to be worth the postage.
---
The third option
## One thread, one to-do list, no lock to fight over
**In plain words**
Think of one person cooking dinner. They put the kettle on and, instead of
standing there, go and chop onions. They aren't two people and they never do two things at once. They just never stand still while something else is boiling. `asyncio` is that, and the `await` keyword is
where they put a job down.
An `async def` function returns a **coroutine object** when called and runs none of its body, just like a generator in Session 05. The **event loop** drives those objects,
running one until it hits an `await` that cannot complete
immediately. At that point the coroutine suspends, keeping its frame, and the
loop picks up the next ready one. Two one-second waits took
**0.50 s** under `gather`, against **1.01 s** awaited one after the other, on a single thread, with `threading.enumerate()` reporting only `MainThread`.
### Threads are pre-emptive
Your code can be interrupted anywhere the interpreter allows it, and you
cannot see where by reading the source. That is why you need locks, and
why Demo 2 lost a deposit.
### Coroutines are cooperative
A coroutine is only ever suspended at an `await`, a word you typed. Between two `await`s nothing else can run, so plain
read-modify-write of a local structure is safe without a lock.
**Session 05, with a scheduler attached**
A generator suspends at `yield` and someone calls
`next()` to resume it. A coroutine suspends at
`await` and the event loop resumes it. Same paused frame, same kept locals. The only new idea is *who* decides when to continue, and what they were waiting for.
### The one mistake that undoes all of it
There is one thread. A synchronous call that blocks it blocks the loop, and every other coroutine stops, including the ones that were just about to finish. This isn't a subtle performance issue. It turns a concurrent program back into a sequential one.
| Blocks the whole loop | Use instead |
| --- | --- |
| `time.sleep(1)` | `await asyncio.sleep(1)` |
| `requests.get(url)` | An async client, awaited |
| `open(path).read()` on a slow disk | `await asyncio.to_thread(read_it, path)` |
| A long pure-Python computation | A process pool, awaited through `run_in_executor` |
**A forgotten `await` fails quietly**
Calling `fetch(url)` without awaiting it builds a coroutine
object and throws it away. Nothing runs, no exception is raised, and you
get `RuntimeWarning: coroutine 'fetch' was never awaited` on stderr, which is easy to miss in a busy log. If an async function
seems to do nothing at all, look for the missing keyword first.
**What you should be able to draw**
One thread. One event loop object holding a list of ready tasks and a list of
timers. Several coroutine objects on the heap, each with a paused frame and a
line number it stopped at. No second stack, no GIL contention, and not one lock, because nothing ever ran at the same time as anything else.
---
What happens next
## A future without it, and why your bugs survive
**In plain words**
The microphone rule is finally being lifted. There is now a build of CPython
with no GIL at all, where threads really can compute at the same time. It's a big change, and it makes the mistakes in this session *more* likely to bite, not less.
PEP 703
added an optional free-threaded build. Python 3.13 shipped it as
experimental and Python 3.14 made it officially supported, alongside the
ordinary build, which still has the GIL and remains the default. On 3.13 or
newer, ask the interpreter which one you are in with
`sys._is_gil_enabled()`.
**What changes**
### CPU-bound threads finally scale
Four threads on four cores computing at once, sharing objects directly,
with none of the pickling that `multiprocessing` charges you.
That is what all the work was for.
**What does not**
### Every race in this session
The lost deposit in demo 2 isn't caused by the GIL. It happens *despite* it. Remove the lock and the two threads no longer take
turns at all, so the gap you were relying on getting away with is wide open.
**Code that was already correct stays correct**
If your shared state is behind a `Lock`, or lives in a `queue.Queue`, or doesn't exist at all because every task returns a value, then free-threaded Python is pure upside. The work you did to make threads safe is the same work that makes them fast later.
**Try this in a REPL**
`import sys; sys._is_gil_enabled()` exists from Python 3.13 onwards, and raises `AttributeError` before that. On a normal build it prints `True`. Free-threaded builds are installed separately (on most platforms the executable has a `t` suffix, such as `python3.14t`) and print `False`. The same
script, two answers, and only one of them runs your threads in parallel.
---
Common mistakes
## Six ways this goes wrong
### Adding threads to speed up a computation
The classic. The work is pure Python, so every thread needs the GIL for
every bytecode, so they take turns and pay for the swapping on top. Measure before and after, and you'll usually find it got slower. Use processes,
or a library that releases the GIL, or a better algorithm.
### Trusting the GIL to protect your data
"Only one thread runs at a time, so I don't need a lock." Only one thread runs *one bytecode* at a time. Anything you write as one line and think of as one action (read-add-write, check-then-act, append-if-missing) is several, and a switch can land in the middle.
### Blocking the event loop
One `time.sleep`, one synchronous HTTP call or one long
computation inside a coroutine stops every other coroutine dead. The
symptom is an async program that is exactly as fast as the sequential one
it replaced. Push the blocking part out with
`asyncio.to_thread` or an executor.
### Omitting `if __name__ == "__main__":`
Wherever the start method is `spawn`, the child re-imports your
module, re-runs the line that started it, and spawns another child. The
error is usually a wall of `RuntimeError` about the bootstrapping
phase. The guard is part of the API, not a convention.
### Never joining, or never awaiting
A thread you don't `join()` may not have finished when you read its results. A coroutine you don't `await` never runs at all and only whispers a `RuntimeWarning`. In both cases the program produces an answer, which is why the bug survives so long.
### Taking two locks in two different orders
One thread holds A and wants B; another holds B and wants A; neither will
ever move again, and your program simply stops with no error at all. Fix it
by always taking locks in one fixed order, by holding one lock at a time, or
by not sharing the state in the first place.
**Two questions catch nearly all of it**
**Is this code waiting or thinking?** That picks threads, processes or `asyncio`. **Between which two lines could another thread run?** That finds the race. If the answer to the second is "anywhere", you need a lock, a queue, or no sharing at all.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · One lock, two threads
Watch for: Watch one row: GIL, in the grey box at the bottom. Every thread above it is ready to run. Only the one named there actually is.
```python
import threading
def work(name):
for i in range(3):
print(name, i)
t1 = threading.Thread(target=work, args=("A",))
t2 = threading.Thread(target=work, args=("B",))
t1.start()
t2.start()
t1.join()
t2.join()
```
1. **One thread, and it already holds the lock** Even a program with no threads in it is running in a thread, the one Python calls MainThread. The grey box at the bottom isn't a call frame. It's the interpreter, and it shows who currently has permission to execute Python bytecode.
2. **def work builds a function object** Nothing concurrent yet. This is Session 02 unchanged: def makes an object on the heap and binds a name to it. The body won't run until somebody calls it.
3. **Two Thread objects, and still one thread** threading.Thread(...) is a constructor call like any other. It produces an instance with a __dict__ that holds a reference to your function. Look at ident: it's None. The operating system hasn't been told about anything yet.
4. **t1.start() asks the operating system for a real thread** Now ident has a number. An actual OS thread exists and is ready to run. But start() returns straight away to the main thread, which still holds the GIL, so Thread-1 is ready and not running. That gap is what this session is about.
5. **t2.start(): three threads, still one GIL** Three OS threads now exist and all three are perfectly capable of running. The interpreter will let one of them execute Python at a time. The other two aren't paused by your code. They're parked by the runtime.
6. **t1.join(): the main thread steps aside and the GIL moves** join() means "wait here until that thread ends". Waiting isn't computing, so the main thread hands the GIL back before it parks. The interpreter gives it to Thread-1, which finally starts running your loop.
7. **Five milliseconds later, the GIL is taken away mid-loop** Thread-1 didn't finish and didn't ask to stop. The interpreter asked it to drop the lock at the next safe point, the jump back to the top of the for loop. Its frame is still there, with i frozen at 1. Thread-2 starts from the beginning.
8. **The lock keeps changing hands until both loops are done** That is the mechanism, all of it. The two loops interleave, so their output lines mix, but they never overlap: at no instant are two threads executing Python bytecode. The work took as long as doing it one after the other, plus the cost of all that swapping.
9. **Both threads end, and the main thread gets the lock back** The two call frames are gone. The Thread objects are still in memory, because they're ordinary objects and t1 and t2 still name them, but the OS threads behind them have exited. join() returns and the main thread carries on.
### Demo 2 · The lost deposit
Watch for: Watch balance on the heap and value read inside each thread. Two deposits go in. Only one comes out.
```python
import threading
class Account:
def get(self): return self.balance
def set(self, v): self.balance = v
acct = Account()
acct.balance = 0
def deposit():
acct.set(acct.get() + 1)
t1 = threading.Thread(target=deposit)
t2 = threading.Thread(target=deposit)
t1.start(); t2.start()
t1.join(); t2.join()
```
1. **Two threads, one number, one expected answer** Each thread will add 1 to the same balance once. Nobody would argue about what the answer should be: 2. Keep that number in mind.
2. **One account object, one attribute, value 0** Session 04's picture: an instance with a __dict__, and balance living inside it. There's one of these, and both threads are about to reach for it.
3. **Both threads start, and Thread-1 gets the lock first** Same picture as demo 1. Thread-2 is ready and waiting. Nothing has gone wrong yet, and nothing is about to go wrong with the GIL. What goes wrong is the shape of line 11.
4. **Thread-1 reads 0 into its own private stack** Line 11 isn't one action. Its first part, acct.get(), copies the value 0 out of the object and onto Thread-1's evaluation stack, shown here as value read. From this instant on, Thread-1 is working from a snapshot.
5. **The handover lands between the read and the write** Calling a method is one of the moments the interpreter checks whether to switch threads. It switches here. Thread-1 is frozen holding a 0 that is about to stop being true, and it has no way to notice.
6. **Thread-2 reads the same 0, adds 1, and stores 1** Thread-2 runs the whole line without interruption. It reads 0, computes 1, writes it back. From its point of view everything is fine: the balance went from 0 to 1, one deposit.
7. **Thread-1 wakes up and writes its stale answer over the top** Thread-1 resumes where it stopped, still holding 0. It computes 0 + 1 and stores 1. The value it overwrites is also 1, so nothing looks broken, but Thread-2's deposit has just been erased.
8. **Two deposits went in. The balance says 1.** No exception, no warning, no corrupted memory. The interpreter did its job perfectly. It promised that one bytecode would finish before another began. It never promised your line would. The GIL protects the interpreter's own data structures, not yours.
### Demo 3 · Fixing it with a Lock
Watch for: Two different queues appear. One thread waits for the GIL, the other waits for the Lock. They aren't the same queue.
```python
import threading
class Account:
def get(self): return self.balance
def set(self, v): self.balance = v
acct = Account()
acct.balance = 0
lock = threading.Lock()
def deposit():
with lock:
acct.set(acct.get() + 1)
t1 = threading.Thread(target=deposit)
t2 = threading.Thread(target=deposit)
t1.start(); t2.start()
t1.join(); t2.join()
```
1. **Same program, one extra object** The same Account as in demo 2, and the same two threads. Nothing about the GIL changes. What changes is that you add a lock of your own, covering the three operations that must not be split.
2. **A Lock is an ordinary object, and it starts unlocked** threading.Lock() allocates an object like any other. It knows one thing: whether somebody is currently holding it. Right now, nobody is.
3. **Thread-1 enters the with block and takes the lock** with lock: calls lock.acquire(). It was free, so Thread-1 walks straight in and locked() flips to True. Nothing has slowed down yet.
4. **Thread-2 reaches the same line and is turned away at the door** The GIL moved to Thread-2, which immediately hit with lock:. The lock is taken, so acquire() blocks. Blocking is waiting, so Thread-2 hands the GIL straight back. It's now in a different queue.
5. **Thread-1 reads 0, and the 5 ms tick no longer matters** The interpreter is still free to take the GIL away from Thread-1. It just has nobody useful to give it to. Thread-2 isn't asking for the GIL, it's asking for the lock. So Thread-1 keeps going.
6. **Thread-1 writes 1 and leaves the block** The read, the add and the write all happened with the lock held, so no other thread could slip in between them. Leaving the with block calls release(), and because it's a with block, that happens even if the body raises.
7. **Thread-2 takes the lock and does its whole deposit** It reads 1, not the stale 0 from the racy version, because it wasn't allowed to read until Thread-1 had finished writing. Same three operations, this time one after the other.
8. **The balance is 2, and it will be 2 every single run** That last part is what you were really buying. The racy version isn't always wrong. It's sometimes wrong, which is far harder to find. A lock makes the outcome the same every time, on every machine, at every thread count.
### Demo 4 · Waiting vs computing
Watch for: Watch the GIL row go free. That only ever happens because a thread stepped away to wait for something.
```python
import threading, time
def nap():
time.sleep(1)
def burn():
total = 0
for i in range(5000000):
total += i
naps = [threading.Thread(target=nap) for _ in range(4)]
for t in naps: t.start()
for t in naps: t.join()
```
1. **Two shapes of work, and only one of them shares well** nap spends its whole life waiting. burn spends its whole life computing. Threads will transform the first and do nothing at all for the second, and the reason is visible in one row of the diagram.
2. **time.sleep gives the GIL back before it waits** This is the deal every blocking call in the standard library keeps. The moment it's going to sit and wait, for a timer, a socket, a file or a database, it releases the GIL first, and takes it back when the answer arrives.
3. **burn never waits, so it never volunteers the lock** There's no point in this loop where the thread has nothing to do. It only ever loses the GIL because the interpreter takes it, after 5 ms, at the jump back to the top of the loop. Then it has to queue up and get it again.
4. **Four nap threads start, and the first one reaches sleep** Thread-1 gets the GIL, calls time.sleep(1), and immediately hands the lock back. Its frame stays alive, because it's waiting rather than finished, but it's out of the interpreter's way.
5. **The second thread doesn't have to wait its turn** The GIL was free, so Thread-2 took it, called sleep, and gave it up as well. Two threads are now waiting at the same time. Their two one-second waits overlap, and that is the benefit threads give you.
6. **All four are asleep, and nobody holds the GIL at all** Look at the bottom box. The lock everybody was fighting over is free, because there's no Python left to run. Four threads, four waits, all happening at once. Nothing is being computed, so nothing is contended.
7. **One second later they all wake up. Total elapsed: one second.** Four sleeps of one second finished in one second, not four. Measured on the machine this session was written on: 2.01 s when run one after the other, 0.50 s across four threads.
8. **Now swap in burn, and the GIL is never free** Two compute threads never step aside, so the interpreter has to prise the lock away from each in turn. Only one runs at any instant, and all that swapping costs real time. On the same machine, 1.81 s serial became 2.49 s across two threads. Threads made the CPU work slower.
9. **The rule that decides everything: threads overlap waiting, never computing** Ask one question about your slow code: is it waiting, or is it thinking? Waiting on a network, a disk, a database, a subprocess: threads help, a lot. Thinking in Python: threads can't help, because thinking needs the lock.
### Demo 5 · A second process
Watch for: Two grey interpreter boxes appear, each with its own GIL. The memory panel is showing two separate address spaces side by side.
```python
import multiprocessing as mp
def child(data):
data.append("from child")
print("child sees:", data)
if __name__ == "__main__":
data = [1, 2, 3]
p = mp.Process(target=child, args=(data,))
p.start()
p.join()
print("parent sees:", data)
```
1. **If the lock is the problem, stop sharing the lock** A thread shares everything with its siblings, including the one GIL. A process shares nothing. It has its own interpreter, its own GIL, its own heap. Two processes really do run Python at the same instant.
2. **One list, in the parent's memory** An ordinary list at an ordinary address. Everything so far is the single-process world you already know.
3. **mp.Process(...) builds an object, and it's still one process** Just like Thread, this is a constructor, not a fork. It records what to run and what to run it with. The operating system hasn't been asked for anything yet.
4. **p.start() launches a whole second Python** Two interpreter boxes now, and two GILs. The child re-imports your module to get at child, which is why the if __name__ == "__main__": guard on line 7 is not optional. Without it the child re-runs p.start() and spawns forever.
5. **The argument is pickled across: a copy, not a link** The list can't be handed over, because the child can't reach the parent's memory. So it's pickled to bytes, sent, and rebuilt. Same three values. Different object, at an address in a different address space.
6. **The child appends to its list** Session 01 said mutation changes the object in place and every name pointing at it sees the change. That's still true, but only for names in this process. No name in the parent points here.
7. **The child exits and takes everything with it** When a process ends, its whole address space goes with it: its heap, its objects, its GIL. Nothing is written back. p.join() returns, and the parent has no idea what the child changed.
8. **The parent prints [1, 2, 3]** This catches almost everybody once. The model that makes it obvious: a Thread shares your objects, a Process shares your values. To get something back, it has to travel, as a return value from a Pool, through a Queue or a Pipe, or in shared memory.
9. **And this is why processes fix CPU-bound work** Two interpreters means two GILs, which means two cores really running Python at once. The same benchmark as in demo 4: 1.81 s serial, 2.49 s on two threads, 1.08 s on two processes. You pay for it in startup time and in copying everything you send.
### Demo 6 · One event loop
Watch for: Count the threads: there's one, the whole way through. The overlapping happens between objects, not between threads.
```python
import asyncio
async def fetch(name):
await asyncio.sleep(1)
return name.upper()
async def main():
a, b = await asyncio.gather(fetch("one"), fetch("two"))
print(a, b)
asyncio.run(main())
```
1. **A third answer, and it never leaves the main thread** Threads let the operating system interleave your waits. asyncio does the interleaving itself, inside one thread, using objects that know how to pause. If you understood generators in Session 05, you already understand most of this.
2. **async def doesn't make a function that runs** It makes a function that, when called, hands you a coroutine object and runs nothing. Session 05's generators behaved the same way: calling the function built an object, and the body waited to be driven.
3. **asyncio.run starts one event loop** The loop is an ordinary object living on the heap of the thread you're already in. Its job is small and boring: keep a list of things that are ready, run one of them until it pauses, repeat.
4. **Two coroutine objects appear, and not one line of fetch has run** The two calls inside gather(...) are evaluated first, as arguments always are. Each produces a coroutine in the state CORO_CREATED. Forget to await one and Python will tell you: RuntimeWarning: coroutine 'fetch' was never awaited.
5. **gather wraps each coroutine in a Task and gives it to the loop** A coroutine on its own does nothing. Something has to drive it. A Task is that driver, an object that holds a coroutine and asks the loop to keep stepping it. Both tasks are now in the loop's ready list.
6. **Task-1 runs until its first await, then suspends** await asyncio.sleep(1) is a yield point, marked in your own source. The coroutine stops there, keeps its half-finished frame, and registers a timer with the loop. It moves to CORO_SUSPENDED.
7. **The loop doesn't wait for it. It starts Task-2 straight away** This is the trick, and it's nothing like a thread switch. Nobody was pre-empted. Task-1 chose to pause at a point you can see in the source. Now two coroutines are suspended, both timers are running, and there's still one thread.
8. **One second later both timers fire and each coroutine resumes where it stopped** The loop hands control back into the middle of fetch, on line 4, with name still bound. Line 5 runs and each coroutine returns its string, reaching CORO_CLOSED.
9. **Two one-second waits took one second, on one thread** Measured: 1.01 s awaiting them one after the other, 0.50 s through gather. The same win as threads on I/O, with no GIL hand-off and no locking, because nothing ever ran at the same time as anything else.
## Check yourself (quiz)
**Q: 1. Why does CPython have a Global Interpreter Lock at all?**
- Because Python threads are not real operating-system threads
- Because the language specification requires programs to be deterministic
- Because interpreter bookkeeping such as reference counts would be corrupted by unsynchronised threads (correct)
Why: The threads are genuine OS threads, and the language says nothing about this. The GIL is CPython's way of keeping its own shared structures, refcounts first among them, correct without putting a lock on every object.
**Q: 2. Four threads each run a tight loop of pure-Python arithmetic on a four-core machine. What happens?**
- Roughly four times faster, one thread per core
- No faster, and usually slower, because only one thread can execute bytecode at a time (correct)
- Exactly the same speed, since the interpreter runs them in turn at no cost
Why: They interleave rather than overlap, and every handover costs something. Measured while writing this session, 1.81 s serially became 2.49 s across two threads. Processes are the fix.
**Q: 3. Two threads run acct.set(acct.get() + 1) once each, starting from 0. What can the balance be?**
- Either 1 or 2, depending on where the thread switch lands (correct)
- Always 2, because the GIL makes the line atomic
- Always 1, because the second write always overwrites the first
Why: The GIL keeps one bytecode indivisible, not one line. A switch at the call boundary lets both threads read 0, and the second write lands on top of the first. Sometimes it does, sometimes it doesn't, and that's what makes it dangerous.
**Q: 4. Four threads each call time.sleep(1). Roughly how long does the whole thing take?**
- About four seconds, because the GIL serialises them
- About two seconds, because they overlap in pairs
- About one second, because sleep releases the GIL, so the four waits overlap (correct)
Why: Blocking calls hand the GIL back before they wait. All four threads sit outside the interpreter at the same time and nobody holds the lock at all. Measured: 2.01 s serially, 0.50 s across four threads.
**Q: 5. A child process appends to a list it was given as an argument. What does the parent see afterwards?**
- The appended item, because lists are mutable and mutation is shared
- The original list, unchanged, because the child mutated a pickled copy in its own memory (correct)
- An error, because mutable objects cannot be passed to a process
Why: Processes do not share a heap. The argument was pickled and rebuilt at a different address in a different address space. To get data back, return it, or send it through a Queue or Pipe.
**Q: 6. Inside a coroutine you call time.sleep(2) instead of await asyncio.sleep(2). What happens?**
- The whole event loop stops for two seconds, and every other coroutine is frozen (correct)
- Only this coroutine pauses; the loop carries on with the others
- asyncio detects it and moves the call to a worker thread
Why: There is one thread and the loop is running on it. A synchronous block holds that thread, so nothing else can be resumed. await is what hands control back, and a blocking call never does.
**Q: 7. On a free-threaded build (Python 3.13+, where sys._is_gil_enabled() returns False), what happens to the lost-deposit bug from Demo 2?**
- It disappears, because without a GIL there are no thread switches to interrupt the line
- It is unchanged, because it was caused by the GIL in the first place
- It gets easier to hit, because the threads now really run at once, so the gap is wide open (correct)
Why: The race was never caused by the GIL. It survived despite it. Removing the lock removes the accidental protection of "only one thread at a time", so code that happened to get away with it stops getting away with it. Your own Lock still works, and is still the answer.
---
# Session 08: Exceptions, Tracebacks & Context Managers
URL: https://inboxpraveen.github.io/Py-Internals/sessions/08-exceptions/
Summary: What raise does to the call stack, how to read a traceback, why finally runs past a return, what with calls for you and how exceptions chain with from.
A traceback isn't Python shouting at you. It's a receipt: the list of desks an exception cleared on its way up the stack, looking for someone to deal with it. Once you can see that walk, `try`, `finally` and `with` stop being syntax to memorise and become three ways of standing in its path.
**Learning goal**
## An exception is an object that climbs the stack.
`raise` builds an ordinary instance, then clears one frame after another
until a frame with a matching `except` says stop. Every feature in this session (the traceback, `finally`, `with`, chaining) is a detail of that climb: what it records, what it runs on the way, and who gets to intercept it.
**Recommended path**
**Let one escape.** Demo 1 raises with no handler anywhere. Watch the desks go and the `__traceback__` row grow.
**Then catch it.** Demo 2 stops the climb one desk up, and shows why `err` vanishes before the function returns.
**Guarantee the cleanup.** Demos 3 to 5 are the same `try/finally` idea three times over, ending with `with`.
## What `raise` actually does
**In plain words**
Picture an office where every call opens a desk on top of the last. Someone at the top desk hits a problem they can't solve, writes it on a card, clears their desk and hands the card down. The next person either deals with it or clears their own desk
and passes it on. If it reaches the front door unanswered, the card is pinned to the wall (that's the traceback) and the office closes for the day.
`raise X` first evaluates `X`, which is normally a constructor call, so an instance of an exception class appears on the heap. The interpreter then attaches a traceback entry for the current frame and looks for a `try` block covering the current line. If there's none, the frame is popped, locals and all, just as on `return`, and the search continues in the caller at the line that made the call, which gets a traceback entry of its own.
Only a matching `except` stops the walk. If nothing matches before the
module level, the traceback is printed to `stderr` and the process exits
with status `1`.
**1 · Build**
### An ordinary instance
`ValueError("bad data")` is a Session 04 constructor call: a
`__class__` link, an `args` tuple, and room for a traceback.
**2 · Unwind**
### Frames are popped
Each frame without a handler is cleared and adds one entry to
`__traceback__`. The locals die with the frame. The exception object doesn't.
**3 · Stop, or crash**
### A handler, or the wall
The first frame whose `except` matches keeps its desk and runs the handler. Otherwise the traceback is printed, the exit status is 1, and nothing else runs.
**Read a traceback from the bottom up**
The last line is the error and its message. The entry directly above it is where
it was raised. Everything above that is the chain of calls that led there, outermost first, which is why the heading says *most recent call last*. The printed text is just the `__traceback__` row from the memory panel, one frame per entry.
**Try this in a REPL**
`try: 1 / 0` then `except ZeroDivisionError as e: tb = e.__traceback__`.
Now inspect it: `tb.tb_frame.f_code.co_name` and `tb.tb_lineno`
tell you where, and `tb.tb_next` walks toward the raise site. It's the same chain the panel draws, and the same one `traceback.print_exc()` prints.
---
Catching
## `try`, `except`, `else`, `finally`
**In plain words**
A `try` block is a desk with a note on it: "if a card arrives while I'm on one of these lines, check it against this list". The list is the `except` clauses, read top to bottom. The first one that fits handles it, and the rest are ignored.
The match is `isinstance(exc, ExceptionClass)`, so a clause catches the named class and every subclass of it. `except LookupError` catches a `KeyError`, and `except Exception` catches nearly everything you'll ever raise yourself. `else` runs only if the `try` body finished without raising, and `finally` runs no matter what. The next section is about that.
**try**
The body runs. If it finishes cleanly, skip every `except`.
always
**except**
Only if the body raised: clauses are tested in order, and the first `isinstance` match runs. If nothing matches, the frame is cleared and the climb continues.
first match
**else**
Only if the body did *not* raise. It keeps the protected code small, so a bug in the `else` block isn't swallowed by the `except` above it.
no error
**finally**
Runs last on every path: after a clean body, after a handler, after an unmatched exception, and even on the way out through a `return`.
always
**`as err` is a normal name that gets deleted for you**
At the end of an `except ... as err` block Python executes
`del err`. Reading `err` afterwards gives
`NameError`. The reason is a cycle: the exception references its
traceback, the traceback references the frames it walked through, and the frame you're in would reference `err`. That loop would keep every local in every one of those frames alive until the cyclic collector got round to it. If you need the exception later, bind it to another name inside the block.
### The family tree decides what a clause catches
Exceptions are classes, and `except` matches by inheritance. A handful of parents cover most of what you'll meet:
| You raise or meet | Its parents | So this clause catches it |
| --- | --- | --- |
| `KeyError`, `IndexError` | `LookupError` → `Exception` | `except LookupError` |
| `ZeroDivisionError`, `OverflowError` | `ArithmeticError` → `Exception` | `except ArithmeticError` |
| `FileNotFoundError`, `PermissionError` | `OSError` → `Exception` | `except OSError` |
| `ValueError`, `TypeError`, `StopIteration` | `Exception` | `except Exception` |
| `KeyboardInterrupt`, `SystemExit` | `BaseException` only | a bare `except:`, and almost never on purpose |
**A bare `except:` catches Ctrl-C**
`KeyboardInterrupt` and `SystemExit` deliberately sit outside
`Exception`, so that `except Exception:` lets the user quit
and lets `sys.exit()` work. A bare `except:` catches them
too, which is how a script ends up impossible to interrupt. Name the class you mean. If you really do mean "anything my code could raise", write `except Exception`.
**Try this in a REPL**
`KeyError.__mro__` prints the whole chain up to `object`.
Then `try: {}["x"]` with `except LookupError as e: print(type(e).__name__)`. It prints `KeyError`, because the clause matched a subclass.
Finally, after any `except ... as e` block, type `e`: `NameError`.
---
The way out
## `finally` runs before the frame is left, whatever the reason
**In plain words**
"Lock the door on your way out" applies whether you leave because you finished, because you were called away, or because you're running from a fire.
`finally` is that rule. Demo 3 shows the version people find hardest to
believe: a `return` inside the `try` still runs the
`finally` block first.
When control leaves a `try` that has a `finally`, whether by falling off the end, by `return`, `break` or `continue`, or by an exception, the interpreter parks the pending result (the return value, or the exception in flight), runs the `finally` block, and then resumes whatever was pending. A `return` *inside* `finally` replaces the pending result, which throws away an exception that was on its way out.
**What it guarantees**
### The block runs on every exit
Files get closed, locks get released, temporary state gets restored, even when the line that failed is one you didn't think could fail.
**What it doesn't**
### It doesn't cancel the exception
After the block, the parked exception carries on climbing. The only thing that stops it is a `return`, `break` or `continue` inside `finally`, and Python 3.14 warns about all three
(PEP 765)
because they almost always hide a bug.
```python
def risky():
try:
raise ValueError("lost")
finally:
return "fine" # the ValueError is thrown away, nobody will ever see it
print(risky()) # fine
```
**Try this in a REPL**
Define `def f(): try: return "from try" finally: print("cleanup")` on
separate lines and call it. `cleanup` prints *before* the value
comes back. Then change the `finally` to `return "from finally"`
and see which string wins.
---
Context managers
## `with` is `try/finally` with the cleanup moved into the object
**In plain words**
Instead of every caller having to remember to lock the door, the door locks itself. An object that knows how to set itself up and tear itself down is a **context manager**, and `with` is the two-line way of saying "set up, do this, tear down, no matter what".
`with EXPR as name:` evaluates `EXPR` to an object, looks up `__enter__` and `__exit__` on its *class*, calls `__enter__()` and binds its return value to `name`, runs the body, and then calls `__exit__(exc_type, exc, tb)`, with three `None`s if the body finished, or with the exception and its traceback if it didn't. If `__exit__` returns a true value the exception is suppressed. Otherwise it continues as if the `with` had never been there.
### What you write
`with open(path) as f:` then a body that uses `f`. Two lines,
no cleanup in sight. Demo 4 is the success path; demo 5 is the body raising.
### What runs (roughly)
`mgr = open(path)`; `f = type(mgr).__enter__(mgr)`;
a `try` around the body; on an exception,
`__exit__(type, exc, tb)` and re-raise unless it returned true;
on success, `__exit__(None, None, None)`.
```python
# with EXPR as name: BODY is close to this:
mgr = EXPR
exit_ = type(mgr).__exit__
name = type(mgr).__enter__(mgr)
try:
BODY
except BaseException as e:
if not exit_(mgr, type(e), e, e.__traceback__):
raise # __exit__ said "not handled": keep climbing
else:
exit_(mgr, None, None, None)
```
### Managers you already use
| Expression | `as` gives you | `__exit__` does |
| --- | --- | --- |
| `open(path)` | the file object itself | closes it, which is why `with open(...)` is the idiom rather than `f = open(...)` |
| `threading.Lock()` | `True`, so nobody writes `as` | releases the lock, which is why Session 07 said to always take it with `with` |
| `tempfile.TemporaryDirectory()` | the path, as a string | deletes the whole directory |
| `sqlite3.connect(...)` | the connection | commits on success, rolls back on an exception, and does *not* close the connection |
| `contextlib.suppress(FileNotFoundError)` | `None` | returns `True` for the named classes, swallowing them |
**Session 05's generator, cut at the `yield`**
`@contextlib.contextmanager` turns a generator function into a context
manager: everything before the `yield` is `__enter__`, the
yielded value is what `as` binds, and everything after it is
`__exit__`. If the body raises, the exception is thrown *into* the paused generator at the `yield`, so wrap the `yield` in `try/finally` if the cleanup must happen either way. It's the same suspended frame as in Session 05, with a different caller pulling the strings.
**Try this in a REPL**
`with 42: pass` gives `TypeError: 'int' object does not support the context manager protocol`, raised before the body ever runs. Then
`with open("x.txt", "w") as f: pass` followed by `f.closed`: the
name survived the block and reports `True`.
---
Chaining
## One error causes another
**In plain words**
Sometimes the person who receives the card can't fix the problem, but can describe it better. "The file is corrupt" is more useful to the caller than "invalid literal for int()". Python lets you raise the better card while keeping the original stapled to it, and the traceback shows both.
Any exception raised while another is being handled gets `__context__` set to the one being handled, automatically. `raise New from old` also sets `__cause__` and `__suppress_context__ = True`. The printer uses those attributes to choose its sentence, and `from None` hides the first exception from the output while leaving it reachable through `__context__`.
| Inside an `except` block | Attributes on the new exception | The traceback says |
| --- | --- | --- |
| `raise RuntimeError("bad input")` | `__context__` → the original | *During handling of the above exception, another exception occurred:* |
| `raise RuntimeError("bad input") from err` | `__cause__` and `__context__` → the original | *The above exception was the direct cause of the following exception:* |
| `raise RuntimeError("bad input") from None` | `__context__` → the original, but suppressed | only the new exception is printed |
| `raise` (bare) | the *same* object, unchanged | the original traceback, continued from here |
**`raise`, not `raise err`, to re-raise**
Both continue the same exception, but `raise err` adds a fresh traceback
entry for the line it is on, so the same function appears twice in the printed traceback, once for the original failure and once for the re-raise. A bare `raise` continues the traceback where it was.
**What you should be able to draw**
A stack of frames with one drawn red, an exception instance on the heap whose
`__traceback__` lists every frame it has cleared, a handler frame that keeps its desk, and, for chaining, a second exception whose `__cause__` arrow points back at the first. If you can draw that, every
traceback you meet is a picture you have already seen.
---
Common mistakes
## Six ways this goes wrong
### Catching too much
`except Exception: pass` turns every bug in the block (a typo, a `None` where a list should be, a missing key) into silence. Catch
the class you expect, at the place you can actually recover. If you only want to
log and move on, log the traceback (`logging.exception(...)`) so the bug
still leaves a trail.
### A bare `except:`
It catches `KeyboardInterrupt` and `SystemExit` as well, so
Ctrl-C stops working and `sys.exit()` gets swallowed. The widest net
that still lets the program be stopped is `except Exception`.
### Using `err` after the block
`except ValueError as err:` then `print(err)` two lines later gives `NameError: name 'err' is not defined`. Python deleted it to break the cycle between the frame and the traceback. Bind it to another name inside the block if you
need it afterwards.
### `return` inside `finally`
It overrides the pending return value *and* discards any exception in
flight. The failure isn't an error you see. It's an error you never see. Python 3.14 emits a `SyntaxWarning` for it. Put the `return` after the `try` statement instead.
### Opening a file without `with`
`data = open(path).read()` works on CPython because the file object is freed, and therefore closed, the moment its reference count hits zero.
Keep the object alive by accident, or run under PyPy, and the handle stays open.
`with open(path) as f:` closes it on a line you can point at.
### Several classes without the parentheses
`except ValueError, TypeError:` is a `SyntaxError` in every
Python up to 3.13 (*multiple exception types must be parenthesized*). Write
the tuple: `except (ValueError, TypeError) as err:`. Python 3.14 finally allows the bare form when there's no `as`, but the tuple works everywhere.
**Two questions catch nearly all of it**
**Which frame should deal with this?** That decides where the `try` goes, and it's usually further up than the line that fails. **What must happen on the way out, even if it fails?** That's a `finally`, or better, a `with`.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · raise, uncaught
Watch for: Watch the __traceback__ row on the exception. It grows by one frame every time a desk is cleared, and nobody ever catches it.
```python
def load():
raise ValueError("bad data")
def parse():
return load()
def main():
return parse()
main()
```
1. **Nothing has run yet** An exception isn't a crash. It's an object that travels up the call stack, clearing one desk at a time, until some frame says it will deal with it. In this demo no frame does, so you get to see the whole journey.
2. **Three ordinary function objects** Session 02, unchanged: each def builds a function object and binds a name. The raise on line 2 is just a line of code inside load. Nothing has been raised yet.
3. **main(), parse(), load(): three desks on the stack** Each call opens a frame on top of the last. Every frame below the top one is paused on the line that made the call, waiting for a value to come back. Keep that picture in mind. It's what the exception is about to walk through.
4. **raise ValueError("bad data") builds an object first** ValueError("bad data") is a constructor call like Point(3, 4) in Session 04. It makes an instance with a __class__ link and an args tuple. Only then does raise act on it: it attaches a traceback entry for this frame and asks whether load has a handler around line 2. It doesn't.
5. **No handler in load, so its desk is cleared and the exception moves up** The frame is removed just as it would be on return, except that no value goes back. Instead the exception object arrives in parse, at the line that made the call, and the same question is asked there. Look at __traceback__: a second entry, for parse, was added on the way in.
6. **Same again in main** parse had no try around line 5, so its frame goes too. Now main is asked, at line 8. Three entries in the traceback, three desks walked through. Nothing in your code has been consulted yet. This is bookkeeping the interpreter does between lines.
7. **It reaches the module level, and there's no handler anywhere** The last desk is the module itself, paused on line 10. No try there either. When an exception runs out of frames, the interpreter prints the traceback to stderr and the process exits with status 1. Nothing after line 10 will ever run.
8. **The printed traceback is that chain, written top to bottom** Traceback (most recent call last):, then one entry per frame, outermost first: line 10, main line 8, parse line 5, load line 2, and finally ValueError: bad data. Read it from the bottom. The last line is what went wrong, the entry above it is where, and everything above that is how the program got there.
### Demo 2 · try / except
Watch for: Watch main's desk. It survives, err appears on it, and then err is deleted before the function returns.
```python
def load():
raise ValueError("bad data")
def main():
try:
return load()
except ValueError as err:
print("recovered:", err)
return "fallback"
result = main()
```
1. **A handler is a frame that says "I'll deal with that"** Same load, same raise. The one difference is on lines 5 to 7: main wraps its call in a try. That's enough to stop the unwinding one desk up, and to show you what as err really binds.
2. **Two functions, and main has a try around its call** Defining a function with a try in it costs nothing at definition time. The except clause isn't registered anywhere. It's a region of main's code that the interpreter will look up only if an exception arrives while line 6 is executing.
3. **main() calls load() from inside the try** Two desks. main is paused on line 6, which sits between try and except. That position is what will matter in a moment. Not the word try itself, but which line the frame was on when the exception came back.
4. **raise builds the exception, and load has no handler** Just as in demo 1: an instance appears, a traceback entry for load is attached, and load's desk is cleared because nothing in it can handle a ValueError.
5. **It arrives in main at line 6, inside a try** This desk is not cleared. Because line 6 is covered by a try, Python walks the except clauses in order and asks isinstance(exc, ValueError) for each one. The first match wins. A subclass would match too (except LookupError catches a KeyError), and if none matched, the desk would go and the search would continue upward.
6. **as err binds an ordinary local name to the exception object** Nothing magical here. err is a name on main's desk pointing at the instance, at the same address as before. The handler body runs and prints recovered: bad data. While it runs, the interpreter also holds a reference of its own, so sys.exception() can hand the object to a bare raise.
7. **The handler ends, and err is deleted on purpose** This catches everyone out once. At the end of an except ... as err block Python runs del err for you, so print(err) on line 9 would raise NameError. The reason is a cycle: the exception holds the traceback, the traceback holds the frames it passed through, and this frame would hold err, so nothing would ever be freed. With err gone and the handler finished, the exception's count hits zero and it takes the traceback with it. Bind it to another name if you need it later.
8. **Summary: the frame that caught it finishes like any other** Line 9 runs, main returns "fallback", and the global name result is bound to it. Nothing below the handler ever knew an exception existed. A caught exception is an object that was built, used for a few lines, and discarded. The stack only looks dramatic while it's in flight.
### Demo 3 · finally and return
Watch for: The return on line 4 doesn't leave the frame. Watch closed flip to True first, and only then the desk go.
```python
def read(path):
f = open(path)
try:
return f.read()
finally:
f.close()
text = read("notes.txt")
```
1. **finally is a promise about the way out** Whatever happens inside the try (a return, an exception, a break, or just reaching the end), the finally block runs before the frame is left. This demo takes the most common case: a file that must be closed.
2. **read("notes.txt") opens a file object** open returns an ordinary instance (Session 04) whose class is TextIOWrapper. Behind it sits something the diagram can't draw: a real operating-system file handle, a limited resource that Python doesn't get back until this object is closed.
3. **return f.read(): the value is computed and set aside** Here's the part people don't expect. return evaluates its expression, so the file is read and a string is built, but the frame is not left yet. Because the return sits inside a try that has a finally, the result is parked and control jumps to line 5.
4. **finally runs: f.close()** The cleanup happens with the return value still parked. closed flips to True and the OS handle is released. Had f.read() raised instead, this block would have run just the same, with the exception parked instead of the value, and the exception would carry on afterwards.
5. **Now the parked value is returned and the desk is cleared** Only after finally finishes does the pending return complete. text is bound to the string, the frame goes, and the file object loses its last name and is freed. It was closed by your code, on purpose, not by the luck of garbage collection, which on other Python implementations can be a long time coming.
6. **Summary: what finally guarantees, and what it doesn't** It guarantees the block runs on every exit path. It does not cancel an exception in flight. The exception resumes afterwards, unless the finally block itself does a return, which quietly swallows it. That bug is common enough that Python 3.14 warns about return inside finally. Demo 4 shows the idiom that packages this whole shape into two lines: with.
### Demo 4 · with
Watch for: Two method calls you never wrote: __enter__ before the body and __exit__ after it. Watch what as r is bound to.
```python
class Resource:
def __enter__(self):
print("open")
return self
def __exit__(self, exc_type, exc, tb):
print("close")
return False
with Resource() as r:
print("working")
```
1. **Two methods with reserved names** A context manager is any object whose class defines __enter__ and __exit__. The with statement is the code that calls them for you, in a fixed order, with a try/finally around the body, just like demo 3.
2. **The class holds __enter__ and __exit__** Session 04's picture: a class object with two function objects in its namespace. Nothing has been entered. These names are only special because with knows to look them up. There's no registration, no decorator, and no base class to inherit from.
3. **with Resource() builds the instance, then calls __enter__** The expression after with is evaluated first. It's a constructor call, so an instance appears. Before running anything else, Python looks up __exit__ and __enter__ on its class. If either is missing you get TypeError: 'Resource' object does not support the context manager protocol right here. Then __enter__ is called with the instance as self.
4. **Whatever __enter__ returns is what as r binds** It returned self, so r is the same instance, at the same address. It didn't have to. open() returns the file, threading.Lock().__enter__ returns True, and plenty of managers return None, which is why with lock: without as is so common. The name lands on the enclosing desk, not on a desk of its own.
5. **The body runs, inside an invisible try** Line 11 prints working. Nothing in memory changes, but something is armed. The interpreter has wrapped this body in the same kind of try/finally as demo 3, with __exit__ as the cleanup. Whichever way the body ends, that call is coming.
6. **Leaving the block calls __exit__(None, None, None)** Three Nones mean "the body finished without an exception". This call is the finally block of demo 3, given a name and moved into the class, so every user of Resource gets the cleanup without writing it. Its return value, False, is ignored on the success path.
7. **After the block, r is still there** The as name is an ordinary variable of the enclosing frame, and nothing deletes it when the block ends. What ended is the guarantee. __exit__ has run, so from here on r is a finished resource, and using it (reading a closed file, say) is your mistake to make. The with statement has let go of its own reference, so the count is back to 1.
### Demo 5 · with, when the body raises
Watch for: This time __exit__ receives three real arguments. Watch what its return False does to the exception afterwards.
```python
class Resource:
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
print("close")
return False
with Resource() as r:
raise ValueError("boom")
print("never printed")
```
1. **What happens to the cleanup when the body blows up?** This case is the whole reason context managers exist. The body raises, the resource still has to be released, and then the exception has to keep going so that somebody up the stack finds out. Watch all three happen in order.
2. **Enter, just as in demo 4** Instance built, __enter__ called, r bound to what it returned. The with statement holds its own reference to the instance so it can call __exit__ later, even if the body rebinds r to something else.
3. **raise inside the body builds the exception** Demo 1's first step, at module level: an instance of ValueError, with a traceback entry for the frame it was raised in. The line after the body, print("never printed"), is now unreachable. The only question is whether the exception leaves this frame.
4. **__exit__ is called with the exception, not with three Nones** The with statement catches the exception before it can go anywhere and calls __exit__(type(exc), exc, exc.__traceback__). Compare the desk with demo 4: exc_type points at the class, exc at the instance, tb at its traceback. The cleanup runs, close is printed, and it knows what went wrong.
5. **return False means "I didn't handle it", so the exception carries on** The with statement re-raises the same object, as if the with had never been there, except that the cleanup has happened. A bare function returns None, which counts as False, so this is the default. Returning True would swallow the exception and line 12 would print. contextlib.suppress is an __exit__ that returns True for the classes you name.
6. **Uncaught at module level: the traceback is printed and line 12 never runs** Output, in order: close, then Traceback (most recent call last): File "app.py", line 10, in ... ValueError: boom. Notice what is not in that traceback: __exit__. It ran and returned normally, so it left no frame behind. The cleanup was invisible and guaranteed, which is the promise with makes.
### Demo 6 · raise ... from
Watch for: Two exception objects end up on the heap. Watch __cause__ on the second one point at the first. That link is what the two-part traceback is printing.
```python
def parse(text):
try:
return int(text)
except ValueError as err:
raise RuntimeError("bad input") from err
parse("x1")
```
1. **One error causes another** Real handlers often can't fix the problem, so they translate it. A low-level ValueError from int() becomes a RuntimeError that means something to the caller. Python keeps both objects, links them, and prints both, as long as you tell it how they're related.
2. **parse("x1"): int() raises inside the try** int is written in C, so there's no Python frame for it. The first desk the exception lands on is parse's, at line 3. Its message is the one you've seen a hundred times: invalid literal for int() with base 10: 'x1'.
3. **Matched by except ValueError as err** Demo 2's step: the desk survives and err is bound to the instance. The handler could log it and return a default. Instead it's going to raise something new, while this exception is still the "current" one.
4. **raise RuntimeError("bad input") from err builds a second object and links it** A second instance appears. Two attributes on it now point back at the first: __cause__, set explicitly by from err, and __context__, which Python sets on any exception raised while another is being handled. from also sets __suppress_context__ to True, which is how the traceback printer knows to say "direct cause" rather than "during handling".
5. **The new exception unwinds. err is gone, but the first exception survives through __cause__** Leaving the handler deletes err and clears the current-exception slot, as in demo 2. The ValueError would normally be freed here, but the RuntimeError holds it, so it travels along inside the second exception, traceback and all. parse's desk is cleared, and the module level has no handler.
6. **The traceback prints both, oldest first** First the ValueError with its own traceback (parse, line 3), then the sentence The above exception was the direct cause of the following exception:, then the RuntimeError with its traceback ( line 7, parse line 5). Leave out from err and the link is still there through __context__, but the sentence becomes During handling of the above exception, another exception occurred:, with Python assuming the second error was an accident. from None hides the first one entirely.
## Check yourself (quiz)
**Q: 1. What is the first thing raise ValueError("bad data") does?**
- Prints the traceback and stops the program
- Calls ValueError("bad data"), an ordinary constructor, and gets an instance back (correct)
- Searches the whole stack for a matching except before building anything
Why: The expression after raise is evaluated first, like any other expression. Only once the instance exists does the interpreter attach a traceback entry and start looking for a handler, one frame at a time.
**Q: 2. An exception reaches the module level and no try covers it. What happens?**
- Python skips the failing line and carries on with the next one
- The exception is stored in a variable for you to inspect later
- The traceback is printed to stderr, the process exits with status 1, and nothing after that line runs (correct)
Why: An exception that runs out of frames ends the program. The traceback you see is the __traceback__ chain, one entry per frame it cleared, outermost first.
**Q: 3. {}["x"] raises a KeyError inside a try whose only clause is except LookupError:. Is it caught?**
- Yes, because KeyError is a subclass of LookupError, and matching uses isinstance (correct)
- No, because the class names have to be identical
- Only if KeyError is listed as well, in a tuple
Why: Exceptions are classes with a family tree. A clause catches the class it names and every subclass. That's also why except Exception catches nearly everything, and why a bare except: catches too much.
**Q: 4. After an except ValueError as err: block ends, you write print(err). What happens?**
- It prints the message, because err is a normal local variable
- It prints None, because the exception has been handled
- NameError, because Python deletes err at the end of the block to break a reference cycle (correct)
Why: The exception holds the traceback, the traceback holds the frames, and this frame would hold err. To stop that loop keeping every local alive, the name is unbound for you. Copy it to another name if you need it.
**Q: 5. Inside a try that has a finally, the body executes return f.read(). In what order do things happen?**
- The frame returns immediately; finally runs later, when the file object is garbage collected
- The value is computed and parked, finally runs, and only then is the parked value returned (correct)
- finally is skipped, because return leaves the block before it is reached
Why: finally runs on every way out of the try, and return is one of them. The return value is evaluated first, held, and delivered after the cleanup, unless the finally block returns something of its own.
**Q: 6. The body of a with block raises. The manager's __exit__ runs and returns False. What happens next?**
- The same exception continues climbing the stack, with the cleanup already done (correct)
- The exception is suppressed; execution continues after the with block
- A second exception is raised, saying __exit__ failed to handle the first
Why: __exit__ receives the exception and votes: a true return value swallows it, a false one (including the None a bare function returns) lets it carry on. Either way the cleanup ran, and that's the point of with.
**Q: 7. Inside an except ValueError as err: block you write raise RuntimeError("bad input") from err. What does the printed traceback show?**
- Only the RuntimeError, because the ValueError was handled and is gone
- Only the ValueError, because the original always wins
- Both, oldest first, joined by The above exception was the direct cause of the following exception (correct)
Why: from err sets __cause__ on the new exception, which keeps the old one alive and tells the printer how the two are related. Without from, __context__ is still set and the sentence becomes During handling of the above exception, another exception occurred.
---
# Session 09: Modules & Imports
URL: https://inboxpraveen.github.io/Py-Internals/sessions/09-modules/
Summary: What import really does: find a file, run it once, cache it in sys.modules and bind a name. Why from-imports go stale, what __name__ is, and how circular imports break.
Nearly every Python file you'll ever write starts with `import`, and most people never see what that line does. It finds a file, runs it once, keeps the result in a cache and binds a name to it. Once you can see those four moves, the odd cases (a setting that won't update, a circular import, a file called `random.py` that breaks everything) stop being odd.
**Learning goal**
## A module is an object, built once and kept in `sys.modules`.
`import` checks a cache, runs the file into a new module object if it has to, and binds a name in your file. Everything else in this session (from-imports, the main guard, circular imports, a file that hides a library) is one of those moves happening at a moment you didn't expect.
**Recommended path**
**Watch one import.** Demo 1 runs a file once and then finds it in the cache. Keep an eye on `sys.modules`.
**Then the two surprises.** Demo 2 is a from-import that goes stale, and demo 3 is a file that behaves differently depending on who loads it.
**Then the two classic bugs.** Demos 4 and 5 are a circular import and a file that hides a standard module. In both, the cache is only doing its job.
## What `import` actually does
**In plain words**
Think of a workshop with a tool wall. The first time anyone asks for a drill, somebody fetches it from the store room and hangs it on the wall. After that, everyone who asks for a drill gets the one on the wall. Nobody fetches a second drill. `sys.modules` is the wall, and `import` is the asking.
`import config` does four things. It looks up `"config"` in `sys.modules`, and if it's there, it skips straight to the last step. Otherwise it searches `sys.path` for `config.py` (or a `config/` package), creates an empty module object and stores it in `sys.modules`, then runs the file's code with the module's `__dict__` as its global namespace. Last, it binds the name `config` in the namespace that ran the `import`. The module is cached before its body runs, not after, and that's what makes circular imports possible at all.
**1 · Check**
### Already in the cache?
`sys.modules` is a plain dict from names to module objects. A hit costs one lookup, so the second `import` of a module, anywhere in the program, is nearly free.
**2 · Load**
### Run the file once
A new module object is registered, then the file runs top to bottom in a frame whose globals are the module's `__dict__`. Every top-level `def`, class and assignment becomes an attribute.
**3 · Bind**
### A name in your file
`import config` binds `config`. `import config as cfg` binds `cfg`. `from config import retries` binds `retries`, and doesn't bind `config` at all.
**A module's attributes are its globals**
`config.TIMEOUT` from outside and the name `TIMEOUT` inside `config.py` are the same dict entry, seen from two sides. That's also why `global` in Session 02 means *module level*, not program-wide. A function looks up its globals in the module where it was defined (its `__globals__`), whichever file happens to call it.
**Runs once**
### Top-level code, on the first import
Prints, opened connections, files read, settings parsed. All of it runs the first time the module is imported and never again in this process, however many files import it afterwards.
**Doesn't rerun**
### Edits to the file while it's running
Changing `config.py` under a running program changes nothing. `importlib.reload(config)` runs the body again into the *same* module object, but names that other files copied with `from config import ...` still point at the old objects.
**Keep import-time work small**
A module's body runs the moment someone imports it, so slow or risky work at the top level (a network call, loading a large file, connecting to a database) happens before the importer's next line runs. That includes a test that only wanted one small function. Put that work in a function and call it when you actually need it.
**Try this in a REPL**
`import sys, json`, then `sys.modules["json"] is json` gives `True`. `len(sys.modules)` shows how many modules were loaded before you typed anything, and `json.__file__` shows which file this one came from. Finally `type(json)`: ``, an ordinary class with ordinary instances.
---
Import forms
## `from x import y` copies a name
**In plain words**
`import config` gives you the office's phone number. `from config import retries` is writing down what the office told you this morning. If the office changes its answer at lunchtime, the person with the phone number hears the new one. The person with the note doesn't.
`from config import retries` imports `config` as usual, then does the equivalent of `retries = config.retries` in your namespace and stops. It doesn't create an alias or a link. If code later rebinds `config.retries` to a new object, your `retries` still points at the old one. If code *mutates* the object instead, say by appending to a list both names share, both names see it. That's Session 01's rule again, with the two names living in different files.
| You write | Names bound in your file | Sees the module rebind it later? |
| --- | --- | --- |
| `import config` | `config` | Yes, `config.retries` is looked up every time |
| `import config as cfg` | `cfg` only | Yes, the same module under another name |
| `from config import retries` | `retries` only, not `config` | No, it's a copy of one binding |
| `from config import *` | every public name, or just the ones listed in `__all__` | No, and a reader can't tell where each name came from |
| `import os.path` | `os` | Yes. The package is bound, and `os.path` is reached through it |
**`mock.patch` aims at a name, not at a module**
This is why tests sometimes patch the wrong thing. If `user.py` does `from config import retries`, then `patch("config.retries", 99)` changes the module's slot, and `user.py` never looks there again. Patch `"user.retries"`, the name your code actually reads. The `unittest.mock` docs put it as: patch where an object is looked up, not where it's defined.
**Try this in a REPL**
Save `retries = 3` as `config.py`. Then `from config import retries`, then `import config`, then `config.retries = 5`. `retries` is still `3`, and `retries is config.retries` is `False`. Before the `import config` line, typing `config` would have given `NameError`, even though `"config"` was already in `sys.modules`.
---
The main guard
## `__name__` and `if __name__ == "__main__":`
**In plain words**
A file can't tell from its own name whether it's the whole program or just a part of one. Python tells it, by setting `__name__` before the first line runs: `"__main__"` if you started it, and its module name if something imported it.
The script you run is executed as a module registered in `sys.modules` under the key `"__main__"`, with `__name__` set to `"__main__"`. The same goes for `python -m pkg.tool`. When it's imported instead, the same file gets `__name__ == "pkg.tool"`. The guard is an ordinary `if` comparing two strings, and it separates what a file *defines* from what it *does* when you start it.
```python
import sys
def main(argv):
print("hello", *argv)
if __name__ == "__main__":
main(sys.argv[1:]) # python greet.py Sam prints: hello Sam
# import greet defines main and runs nothing
```
**One file can turn into two modules**
If `app.py` is the script you run *and* some other module does `import app`, the file runs twice: once as `__main__` and once as `app`. Those are two separate module objects with two separate sets of globals, so a counter or a cache at the top of `app.py` now exists twice, and changes to one don't show in the other. Keep the file you run thin, and import the real code from somewhere else.
**Try this in a REPL**
Type `__name__` in a fresh REPL: `'__main__'`. The REPL is a module too. Then `import json; json.__name__` gives `'json'`, and `import sys; sys.modules["__main__"]` shows the module you're typing into.
---
Circular imports
## Two modules that need each other
**In plain words**
Two people each wait for the other to finish a sentence before they finish their own. Python breaks the deadlock by letting the second person carry on with whatever the first had said so far. That works fine, as long as the second person doesn't need the part that hasn't been said yet.
Because a module goes into `sys.modules` before its body runs, an import that comes back round to it finds the partly run module and returns it without running anything. Names defined above the import line are there. Names below it don't exist yet, and reading one of them during the import raises `AttributeError` (for `a.NAME`) or `ImportError: cannot import name` (for `from a import NAME`). When an import fails, every module whose body didn't finish is removed from `sys.modules` again, so nothing is left half built in the cache.
**Read names later.** Keep `import a` at the top, but use `a.VALUE` inside functions. By the time a function is called, both modules have finished loading.
**Move the shared part out.** If both modules need `VALUE`, put it in a third module that imports neither of them. This is usually the real fix, because the cycle was a sign the code wanted splitting.
**Import inside the function.** `def f(): from a import VALUE` waits until the call. It's a fair last resort, and after the first call it costs one dict lookup.
**Reading the error on Python 3.13**
Python 3.13 ends the message with a hint in brackets. For modules inside a package it usually says *partially initialized module ... (most likely due to a circular import)*, which is right. For files sitting next to the script you ran, as in demo 4, it says *consider renaming ... if it has the same name as a library you intended to import*, which sends you the wrong way. If the traceback shows two of your own modules importing each other, it's circular, whatever the hint says.
---
The search
## Where Python looks: `sys.path`
**In plain words**
Finding `random.py` is like calling a name down a corridor of rooms, nearest room first. The first room that answers wins, even if the person you wanted was further down.
On a cache miss, Python asks the modules built into the interpreter first (`sys`, for one), then walks `sys.path`, a plain list of folders, in order, and takes the first match: a `random.py` file, a `random/` package folder, or a compiled extension. Entry 0 is the folder of the script you ran (the current folder for `python -m`, and an empty string meaning the current folder in the REPL). After it come the standard library and then `site-packages`, where `pip` installs things.
**sys.modules**
Already loaded? Then you get that object, and the search never starts.
cache
**built in**
Modules compiled into the interpreter, such as `sys`, and a few frozen copies of standard modules it needs at startup.
interpreter
**sys.path[0]**
The script's own folder. Your files live here, which is why one of them can hide anything further down.
your code
**stdlib**
The standard library folders, such as `lib/python3.13` and `lib-dynload`.
batteries
**site-packages**
Everything `pip` installed, inside your virtual environment if one is active.
third party
**Don't name your files after modules you use**
`random.py`, `json.py`, `email.py`, `test.py`, `requests.py`: each one hides the real module from every import in the program, including imports made inside the standard library itself. Python 3.13 notices the standard-library case and says so in the error. For a `pip` package like `requests` you only get the plain `AttributeError`.
### Packages are modules with a `__path__`
A folder with an `__init__.py` is a package. `import shop.orders` runs `shop/__init__.py` first, then `shop/orders.py`, registers both `"shop"` and `"shop.orders"` in `sys.modules`, and binds only `shop` in your file. The package's `__path__` is the list of folders its submodules are searched in, a small `sys.path` of its own.
### What `__pycache__` is
The first import of `config.py` compiles it to bytecode and saves that as `__pycache__/config.cpython-313.pyc`. Next time, if the source hasn't changed, Python loads the saved bytecode and skips compiling. It speeds up start-up, not running time, and it's safe to delete. The script you run directly isn't saved this way.
**Try this in a REPL**
`import sys; print(*sys.path, sep="\n")` prints the search order on your own machine. Then `import random; random.__file__` shows where the real `random` lives. If that path is inside your project, you've found a shadow.
**What you should be able to draw**
A `sys.modules` dict with one entry per loaded module, each pointing at a module object whose `__dict__` holds that file's top-level names. A frame that exists only while a file's body runs. Two names in two files pointing at one object, which is all a from-import leaves behind. And a module that sits in the cache half built while its own imports are still running. Draw those four and you can predict every case in this session.
---
Common mistakes
## Six ways this goes wrong
### Naming a file after a module you import
A `random.py` or `requests.py` in your project folder wins the search, and the real module never loads. The error shows up somewhere else entirely, as a missing attribute. `print(module.__file__)` settles it in one line.
### Expecting a from-imported name to update
`from config import retries` takes a copy of one binding at import time. If the module rebinds `retries` later, your copy doesn't move. Use `config.retries`, or a function that returns the current value.
### Doing real work at import time
Top-level code runs for everyone who imports the module, including tests and tools that only wanted one function. Connections, big file loads and prints belong in a function, called from the main guard or by whoever needs them.
### Running a file inside a package directly
`python shop/orders.py` with a relative import such as `from . import users` fails with `ImportError: attempted relative import with no known parent package`, because a file run directly is `__main__`, not part of `shop`. Run it from the project folder as `python -m shop.orders`.
### Re-importing to pick up an edit
In a REPL or a notebook, a second `import config` after you edit the file does nothing, because the old module is still in the cache. `importlib.reload(config)` runs it again, but from-imported names and instances of old classes keep pointing at the old objects. Restarting is the honest fix.
### Two modules that need each other at import time
A circular import is fine until one side reads the other's names during loading. Read them inside functions instead, or move the shared names into a third module. The traceback's tell is two of your modules importing each other.
**Two questions catch nearly all of it**
**Is it already in `sys.modules`?** If it is, nothing runs and you get that object. **Which namespace does this name live in?** A from-import put a copy in yours, and `module.name` reads theirs.
## Interactive lab (code and step-by-step narration)
Each demo is a short Python program. The steps below are the text shown as the memory diagram animates.
### Demo 1 · import runs it once
Watch for: Keep one eye on sys.modules. The new module lands in it before any of config.py runs, and the second import never gets past it.
```python
# main.py
import config
import config
print(config.TIMEOUT)
# config.py
print("loading config")
TIMEOUT = 30
```
1. **Nothing has run yet** Two files in one folder. You run python main.py, and config.py sits next to it. The dict on the heap is sys.modules, where the interpreter keeps every module it has loaded so far. It already has a few dozen entries before your first line runs. Two are drawn: sys, and __main__, which is main.py itself. Its namespace is the frame above.
2. **import config looks in sys.modules first** Before it touches the disk, Python asks one question: is there a module called config in sys.modules already? There isn't, so it goes looking for a file (demo 5 shows where it looks), finds config.py, and creates an empty module object. Look at the order. The new module goes into sys.modules before a single line of config.py has run.
3. **The body of config.py runs in a frame of its own** Importing a module means running it, top to bottom, like a script. It gets a frame, much like a function call in Session 02, and that frame's global namespace is the module's __dict__. Line 7 prints loading config. main.py waits on line 2 until the body is done.
4. **TIMEOUT = 30 lands in the frame and in the module at once** An assignment at the top level of a file binds a global name, and this frame's globals are the module's __dict__. So the new row shows up on the config.py frame and inside the module object at the same moment. It's one dict, drawn twice.
5. **The body finishes. The frame goes, the module stays** After the last line of config.py, its frame is cleared like any other. The names survive, because they belong to the module object, and sys.modules is holding on to that. Only now does import do the part you can see: it binds the name config in main.py. Two references, the cache and your name.
6. **The second import config is a dict lookup** config is already in sys.modules, so Python takes that object and binds the name to it again. No file is opened and the body doesn't run, so loading config is printed once, not twice. That holds for the whole process. Every file that imports config gets this same object.
7. **config.TIMEOUT reads the module's __dict__** A dotted name on a module is ordinary attribute lookup (Session 04), and a module keeps its attributes in its __dict__. config.TIMEOUT finds the TIMEOUT entry there, so line 4 prints 30.
8. **Summary: one file, one module object, run once per process** The output is loading config, then 30. An import is four moves: check the cache, create and register a module, run the file into its __dict__, bind a name. If you edit config.py while the program is running, nothing changes until the process restarts, or until you call importlib.reload(config), which runs the body again into the same object.
### Demo 2 · from x import y
Watch for: Compare two addresses: the one your retries points at, and the one inside the module's __dict__. They start out the same. (sys.modules is still there, it's just not drawn this time.)
```python
# main.py
import config
from config import retries
config.set_retries(5)
print(config.retries, retries)
# config.py
retries = 3
def set_retries(n):
global retries
retries = n
```
1. **from config import retries looks like a shortcut** It reads like "let me write retries instead of config.retries". It isn't that. It makes a new name in your file, pointing at whatever object config.retries pointed at when the line ran. Watch what happens when config changes its mind later.
2. **import config builds the module, as in demo 1** Running config.py fills the module's __dict__ with two names: retries, pointing at the int 3, and set_retries, a function object. The function remembers where it was defined: its __globals__ is this module's __dict__. main.py has one name so far, config.
3. **from config import retries copies the binding, not the name** Python imports config (cached, so nothing runs), reads config.retries, and binds a new name retries in main.py to that same object. Two names in two namespaces, one int, one address. Nothing links the two names to each other. They just point at the same thing, like b = a in Session 01.
4. **config.set_retries(5): global means this module** Inside set_retries, global retries tells Python to bind retries in the function's global namespace. A function's globals belong to the module where it was defined, not to the file that called it. So line 12 rebinds the retries slot in config's __dict__ to the int 5.
5. **The call returns. config.retries is 5, your retries is still 3** Line 5 prints 5 3. The module's slot was rebound to a new object. Your name was never touched, because nothing ever told it to follow. It's Session 01's rule stretched across two files: rebinding one name never moves another.
6. **Summary: config.retries follows changes, a from-imported name doesn't** Write config.retries when you want whatever the module holds now. That lookup happens every time the line runs. A from-imported name is a snapshot of one binding. Mutation is different: if retries were a list and the function appended to it, both names would see the change, because they share one object. The same rule decides where unittest.mock.patch has to aim, which the notes below cover.
### Demo 3 · __name__ and the guard
Watch for: Watch the __name__ row in the tools.py frame. Same file, same line, two different values, depending on which command started the program.
```python
# tools.py
def double(x):
return x * 2
if __name__ == "__main__":
print(double(21))
# app.py
import tools
print(tools.double(4))
```
1. **One file, two ways to load it** Every module has a __name__. Python sets it before the body runs, and its value depends on how the file was loaded, not on what the file is called. This demo runs two separate programs: first python tools.py, then python app.py.
2. **python tools.py: the file you run is called __main__** The script you start is loaded as a module too, but it's registered in sys.modules under the key "__main__", and that's also the __name__ in its namespace. The file name plays no part. Then def builds double as usual.
3. **The guard is true, so line 6 runs** __name__ == "__main__" compares two ordinary strings. They match, so the block runs and prints 42. Read the guard as "only when this file is the program".
4. **A new run, python app.py: now tools is imported** A fresh process, with nothing carried over. This time app.py is __main__, and line 9 imports tools the way demo 1 imported config: a module object registered as "tools", and its body run in a frame of its own. The first entry in that namespace is __name__, set to "tools".
5. **This time the guard is false, so line 6 is skipped** __name__ is "tools", which isn't "__main__". The test print is skipped and the body ends with nothing but double defined. That's what the guard is for: the code under it runs when you start the file, and stays quiet when someone imports it.
6. **Back in app.py: tools.double(4)** The tools.py frame is gone and the module stays, just as in demo 1. app.py now has the name tools and calls a function through it, so 8 is printed. 42 never is.
7. **Summary: __name__ is set by whoever loads the file** python tools.py and python -m tools both load it as "__main__". import tools loads it as "tools". Put the code that starts your program under the guard, usually as one call to a main() function, and keep definitions above it. Session 07 showed a place where the guard is required: with the spawn start method, multiprocessing imports your main file again in every child process.
### Demo 4 · circular import
Watch for: Look inside module a when b asks for it. It's in the cache, so b gets it, but line 6 of a.py hasn't run yet.
```python
# main.py
import a
# a.py
import b
VALUE = 1
# b.py
import a
print(a.VALUE)
```
1. **Two files that import each other** a imports b, and b imports a. Python allows that, and often it works. It breaks when one of them reads a name from the other while the imports are still going on. Watch sys.modules. It's what stops this from looping forever, and it's also what hands out a half-built module.
2. **import a: module a is registered, then its body starts** The same moves as demo 1. a isn't in the cache, so a module object is created and stored in sys.modules, and a.py starts running in a new frame. Its __dict__ holds nothing but __name__ so far.
3. **Line 5, import b: module b is registered and a waits** a.py stops on its very first line, before it ever reaches VALUE = 1, and the same moves start for b. Three frames are open now, each one paused on an import line, stacked up like nested function calls.
4. **Line 9, import a: found in the cache, half built** a is already in sys.modules, so Python doesn't run a.py again. If it did, a would import b, which would import a, forever. Instead b gets the module object that exists, and its name a is bound to it. Look at what's inside: still only __name__, because line 6 hasn't run.
5. **Line 10, a.VALUE: the name isn't there yet** Looking up an attribute on a module means looking in its __dict__, and VALUE isn't in it. Python raises AttributeError: module 'a' has no attribute 'VALUE', and the frame starts to unwind, as in Session 08. The exception keeps the module it was looking in as its obj attribute, which is the extra reference on a. Python 3.13 adds a hint in brackets after the message. The notes below explain why it can point you the wrong way.
6. **A failed import is taken back out of sys.modules** b.py has no handler, so its frame is cleared. Its body never finished, so the import system deletes "b" from sys.modules. A module that blew up halfway isn't left in the cache for the next import b to find. The exception arrives in a.py at line 5, which has no handler either.
7. **Then a goes too, and main.py has no handler** a is removed from the cache the same way, and the exception reaches main.py at line 2 with no try around it. The traceback lists main.py line 2, a.py line 5 and b.py line 10, outermost first: three frames and no function in sight. The import system's own frames are left out of tracebacks on purpose.
8. **Summary: read the other module's names later, not at import time** Nothing was wrong with import a itself. The bug was using a.VALUE while a was still half built. Move that line into a function that runs after both imports have finished, and it works, because by then the lookup finds a full module. The other cures are to move the shared name into a third module that imports neither, or to import inside the function that needs it. from a import VALUE on line 9 fails the same way, with ImportError: cannot import name.
### Demo 5 · a file named random.py
Watch for: Read sys.path from the left. The search stops at the first folder that has a random.py in it, and the standard library comes later.
```python
# /home/sam/game/random.py
def pick():
return 4
# /home/sam/game/game.py
import random
print(random.randint(1, 6))
```
1. **A file with an unlucky name** Someone writing a dice game saved a helper as random.py, in the same folder as game.py. The standard library has a random module too. Which one does import random find? The answer is in a list you can print yourself, sys.path, drawn here as it would be for python game.py on a Linux machine with a virtual environment.
2. **import random checks the cache first** random isn't loaded at startup, so it's not in sys.modules, and Python has to search for it. It walks sys.path in order and takes the first match. Entry 0 is the folder of the script you ran.
3. **Entry 0 is the script's own folder, and there's a random.py in it** The search stops at the first folder that has random.py (or a random/ package). That's /home/sam/game. The real one in /usr/lib/python3.13 is never even looked at. Your file is registered under the name random, and its body runs.
4. **random now means your file, for the whole process** The name random in game.py points at your module, and so does sys.modules["random"]. Any other code in this process that runs import random, including modules in the standard library, now gets your two-line file instead.
5. **random.randint doesn't exist in your file** Your module has pick and nothing else. Line 7 raises AttributeError: module 'random' has no attribute 'randint'. Python 3.13 spots this case and adds a hint after the message: consider renaming '/home/sam/game/random.py' since it has the same name as the standard library module named 'random'.
6. **Summary: the first match on sys.path wins** Rename the file (dice.py, say) and the real module comes back. When an import behaves strangely, print(random.__file__) tells you which file you actually got. The usual suspects are files named after modules you also use: random.py, json.py, email.py, test.py, or requests.py for a library from pip.
## Check yourself (quiz)
**Q: 1. config.py starts with print("loading"). Three different files in one program each run import config. How many times is loading printed?**
- Three times, once for each import
- Once, because the later imports find the module in sys.modules and don't run the file (correct)
- Never, because importing a module only reads its definitions
Why: The first import runs the whole file into a new module object and caches it. Every import after that, in any file, is a lookup in sys.modules plus a name binding. Nothing runs again unless you call importlib.reload.
**Q: 2. When is a new module added to sys.modules?**
- Before its body runs, as soon as the empty module object exists (correct)
- After its body finishes, so only complete modules are ever cached
- Only when the program exits
Why: Registering first is what lets a circular import find the module instead of loading it forever. It's also why the other module can see it half built. If the body then fails, the entry is removed again.
**Q: 3. main.py runs from config import retries, and later config.set_retries(5) rebinds the module's retries from 3 to 5. What does retries in main.py hold now?**
- 5, because retries is an alias for config.retries
- It raises NameError, because the original binding was replaced
- 3, because the from-import made a separate name pointing at the object config.retries held at the time (correct)
Why: from config import retries works like retries = config.retries. Rebinding the module's name moves that one name only. If the object had been mutated instead (a list appended to, say), both names would see the change, because they share it.
**Q: 4. After from config import retries and nothing else, you type config. What happens?**
- You get the module, because importing anything from it binds its name too
- NameError. The module is in sys.modules, but the only name bound in your file is retries (correct)
- None, because the module was only partly imported
Why: The whole module was loaded and cached, but a from-import binds only the names you list. import config is what binds config. Being in the cache and being a name in your namespace are two different things.
**Q: 5. You start a program with python tools.py. Inside tools.py, what is __name__?**
- "__main__", because it's the file you ran (correct)
- "tools", because that's the file's name
- "tools.py"
Why: Python sets __name__ from how the file was loaded. The script you start, or the one you run with python -m, is "__main__". The same file imported by another module gets "tools", and then its guarded block doesn't run.
**Q: 6. a.py is import b then VALUE = 1. b.py is import a then print(a.VALUE). You run import a. What happens?**
- It prints 1, because Python loads the two files in the right order for you
- It loops forever, each module importing the other
- AttributeError: b gets the half-built a from the cache, and VALUE hasn't been defined yet (correct)
Why: a is registered, starts running, and pauses on import b. b's import a finds a in sys.modules and gets it as it is, before line 2 of a.py has run. The cache stops the loop, and it's also what hands over an unfinished module.
**Q: 7. Your project folder has game.py and a helper you saved as random.py. game.py runs import random. Which file is loaded?**
- The standard library's random, because built-in names always win
- Yours, because sys.path[0] is the script's folder and the first match wins (correct)
- Python raises an error, because two modules have the same name
Why: The search walks sys.path in order and stops at the first match, and your script's folder comes before the standard library. From then on sys.modules["random"] is your file, for every import in the process. Rename the helper.
---
# Glossary
URL: https://inboxpraveen.github.io/Py-Internals/glossary.html
## Objects and names
- **Name**: A label in a namespace, such as x. It does not store a value. It points at an object. See Session 01.
- **Object**: Everything in Python is an object: numbers, strings, lists, functions, classes, instances. An object has identity, a type, and a value. Objects live on the heap.
- **Reference**: The connection from a name (or a container slot) to an object. Assignment creates or changes a reference. It does not copy the object unless you ask for a copy.
- **Identity**: Which object something is, not what it looks like. id(x) and x is y test identity. Two names with the same identity are aliases.
- **Equality**: == asks whether two objects have the same value. They can be equal and still be different objects.
- **Aliasing**: Two or more names (or slots) pointing at the same object. Mutations through any alias are visible to all of them.
- **Rebinding**: x = something_else makes the name point at a different object. The old object is unchanged; it just loses one reference.
- **Mutation**: Changing an object in place without creating a new one. lst.append(1) mutates. The object's id() stays the same.
- **Mutable / immutable**: A mutable object can change its contents (list, dict, set, instances). An immutable object cannot (int, str, tuple). "Changing" an immutable object always rebinds the name instead.
- **Reference count**: How many names or containers currently point at an object. In CPython, when this hits zero the object is freed immediately. Cycles need a separate collector.
- **Heap**: The region of memory where objects live. Every Python object (an int, a list, a function, a class) is allocated on the heap. Names live in namespaces and point into it.
- **Call stack**: The pile of frames for calls currently in progress. Each call pushes a frame; each return pops one. A traceback is a printout of this stack. See Session 02.
- **Garbage collection**: Freeing objects nothing refers to any more. CPython does this mainly by reference counting: the instant a count hits zero the object is freed. A separate cyclic collector handles objects that only refer to each other.
- **Small-integer cache**: CPython builds the integers -5 to 256 once at start-up and hands out the same object every time, so a = 42; b = 42 gives two names for one object. Those objects are never freed. It's an implementation detail, not a language guarantee, so never write code that depends on it.
- **Interning**: CPython reuses some immutable objects, especially small integers (typically -5 to 256) and some strings. That is an optimisation, not a language rule. Do not write code that depends on it.
## Functions and scope
- **Function object**: def creates a function object and binds a name to it. The body does not run at definition time. See Session 02.
- **Frame**: The temporary workspace for one function call: local names, the current line, and the path back to the caller. Frames stack; the top one is running.
- **LEGB**: Name lookup order: Local, Enclosing function scopes, Global, Built-ins. Assignment creates a local name unless you declare global or nonlocal.
- **Argument binding**: A parameter is a local name bound to the object the caller passed. Python does not copy that object. Mutating it is visible to the caller; rebinding the parameter is not.
- **Closure**: An inner function plus the outer names it still needs after the outer call returns. nonlocal updates a remembered outer name instead of creating a new local.
- **Default argument**: Default values are created once, when the function is defined, and stored on the function object. A mutable default is shared across calls, which is a classic surprise.
## Containers
- **Container**: An object whose contents are references to other objects. Lists, dicts, tuples, and sets are containers. See Session 03.
- **Shallow copy**: A new outer container filled with the same inner references. list.copy() and dict.copy() are shallow. Nested objects stay shared.
- **Deep copy**: A recursive copy of nested objects. Use it only when you need nested independence. Some objects should stay shared on purpose.
- **Hashable**: An object with a stable hash, required for dict keys and set elements. Built-in immutables are usually hashable. Lists and dicts are not.
## Classes
- **Class object**: class creates a class object. It holds methods and class attributes. Calling the class creates an instance. See Session 04.
- **Instance**: An object created by calling a class. It has its own identity and usually its own __dict__ for instance attributes.
- **self**: The conventional name for the instance a method is operating on. Python passes it automatically when you write obj.method().
- **__dict__**: The mapping that stores an object's writable attributes. Instance attributes live on the instance dict. Methods usually live on the class dict.
- **Attribute lookup**: Python looks on the instance first, then the class, then base classes (the MRO). That is why a class attribute is visible on every instance until one instance shadows it.
- **Bound method**: Accessing a function on an instance creates a method object that already remembers self. g.hello is not the raw function; it is hello bound to g.
- **__init__**: The initialiser. After the instance exists, Python calls __init__(self, ...) to write attributes. It returns None. The caller receives the instance, not that return value. See Session 04.
- **__new__**: The allocator. Calling a class first creates the instance, then initialises it. You almost never override __new__. The important picture: the object exists before __init__ runs.
- **Class attribute**: A name stored on the class object. Every instance can read it until it assigns the same name on itself. A mutable class attribute (a list on the class) is one shared object.
- **Instance attribute**: A name stored in the instance __dict__. Assignment on an instance usually creates one of these. It shadows a class attribute of the same name for that instance only.
- **MRO**: Method resolution order: the list of classes Python searches for an attribute. For class Dog(Animal) it is Dog → Animal → object. Subclasses do not copy methods; they walk this list.
- **__class__ / type()**: The instance's link to its class. type(p) is Point and p.__class__ is Point are the same idea. Lookup follows this pointer after the instance dict misses.
## Iteration
- **Iterable**: An object you can loop over. It can produce an iterator via iter(obj). Lists, strings, dicts, and generators are iterable. See Session 05.
- **Iterator**: An object with state that yields one item at a time through next(). When it is exhausted it raises StopIteration. A for loop hides this protocol.
- **Generator**: A function that uses yield. Calling it does not run the body; it returns a generator object. Each next() resumes a suspended frame until the next yield.
- **Lazy evaluation**: Work is done only when the next value is requested. A generator expression keeps one pending computation; a list comprehension builds every value immediately.
- **StopIteration**: The signal that an iterator is exhausted. A for loop catches it and exits. The collection is unchanged; the iterator is spent. A new for on a list creates a new iterator. A generator does not rewind.
## Decorators
- **Decorator**: A callable that takes a function and returns a replacement. @deco above a def is shorthand for f = deco(f): your function is built, handed to deco, and the name is bound to whatever comes back. See Session 06.
- **Wrapper**: The function a decorator returns. It usually calls the original and adds something around it. After decorating, your name points at the wrapper, not at the function you wrote.
- **Closure cell**: The small box that keeps an outer name alive for an inner function. It is why a wrapper can still reach your original function after the decorator's frame is gone, and why that original is not garbage collected even though no name points at it. See Session 02.
- **Decorator factory**: A function you call to get a decorator, which is how a decorator takes arguments. In @repeat(2), repeat(2) runs first and returns the decorator that is then applied to your function.
- **functools.wraps**: A decorator applied to a wrapper. It copies __name__, __qualname__, __doc__, __module__ and __annotations__ from the original function onto the wrapper, so tracebacks and help() keep telling the truth.
- **__wrapped__**: The attribute functools.wraps sets on a wrapper, pointing at the function underneath. It is how tools reach past a decorator to the real thing.
## Concurrency
- **GIL (Global Interpreter Lock)**: The single lock a thread must hold to execute Python bytecode in CPython. It exists because interpreter bookkeeping, reference counts above all, would be corrupted by unsynchronised threads. It's a property of CPython, not of the Python language. See Session 07.
- **Thread**: A real operating-system thread, sharing the whole heap with its siblings. threading.Thread(...) only creates an object; the OS thread appears at .start(), which is when ident stops being None.
- **Switch interval**: How long a thread may hold the GIL before the interpreter asks it to hand over: sys.getswitchinterval(), 5 ms by default. The request is only honoured at a safe point, chiefly a jump back to the top of a loop or a call into a function.
- **Race condition**: A bug whose outcome depends on the order in which threads happen to run. The classic shape is read-modify-write: two threads read the same value, both compute from it, and the second write erases the first. The GIL doesn't prevent this. It only keeps single bytecodes indivisible.
- **Atomic**: An operation that cannot be interrupted partway. In CPython one bytecode is atomic; one line of your source usually is not, because it compiles to several. If a switch can land in the middle, the operation is not atomic.
- **Lock**: threading.Lock, an object only one thread may hold at a time. Wrapping a read-modify-write in with lock: makes it indivisible with respect to every other thread that uses the same lock. A thread waiting for a lock is in a different queue from a thread waiting for the GIL.
- **Deadlock**: Two threads each holding a lock the other needs. Neither can ever proceed and no error is raised. The program just stops. Prevented by always acquiring locks in the same order, or by holding only one at a time.
- **CPU-bound / I/O-bound**: The question that picks your tool. I/O-bound code is slow because it waits, and threads and asyncio overlap the waiting. CPU-bound code is slow because it computes, and only a second process (or a library that releases the GIL) adds a second core.
- **Process**: A second operating-system process running a second interpreter, with its own heap and its own GIL. Two processes really do execute Python at the same instant. They share no objects at all, so anything passed between them is copied.
- **Pickling**: Turning an object into bytes so it can cross a process boundary and be rebuilt on the other side. The result is an equal object at a different address, never the same object. Lambdas, open files, connections and locks cannot be pickled.
- **Start method**: How multiprocessing creates a child: spawn, fork or forkserver. Check yours with mp.get_start_method(). Where the child re-imports your module (spawn is the default on Windows and macOS), the if __name__ == "__main__": guard is required or the child spawns its own child.
- **Event loop**: The object asyncio uses to drive coroutines on a single thread. It keeps a list of tasks that are ready and a list of things being waited on, runs one task until it suspends, then picks the next.
- **Coroutine**: What calling an async def function gives you: an object that has run none of its body, exactly like a generator object. It suspends at await, keeping its frame and locals, and the event loop resumes it later. Never awaiting one produces RuntimeWarning: coroutine ... was never awaited.
- **await (yield point)**: The only place a coroutine can be suspended, and a word you typed, visible in the source. This is what makes asyncio cooperative: between two awaits nothing else runs, so no lock is needed. Threads, by contrast, are pre-empted wherever the interpreter allows.
- **Task**: An object that drives one coroutine on the event loop. A coroutine on its own does nothing; asyncio.gather or create_task wraps it in a Task and hands it to the loop.
- **Free-threaded build**: An optional CPython build with no GIL, added by PEP 703. It was experimental in 3.13, is officially supported in 3.14, and is separate from the default build, which still has one. Check with sys._is_gil_enabled(), which exists from 3.13 onwards. It makes CPU-bound threads scale. It doesn't fix any race condition, and it makes the ones you have easier to hit.
## Exceptions
- **Exception**: An ordinary instance of a class that inherits from BaseException. raise X evaluates X (usually a constructor call) and then starts the object climbing the call stack. It carries args, a __traceback__, and links to related exceptions. See Session 08.
- **Unwinding**: What happens between raise and the handler: each frame without a matching except around the current line is popped, locals and all, just as on return, and the exception arrives in the caller at the line that made the call. If it runs out of frames, the traceback is printed and the process exits with status 1.
- **Traceback**: The record of frames an exception has passed through, outermost first, reachable as exc.__traceback__ and walked with tb_next. The printed Traceback (most recent call last) is that chain written out one frame per entry; read it from the bottom, where the error and the raise site are.
- **Handler (try / except)**: An except clause covering the line an exception arrived on. Clauses are tested in order with isinstance, so a clause catches the class it names and every subclass. The frame that catches keeps its desk; the name bound by as err is deleted automatically when the block ends, to break the cycle between the exception, its traceback and the frame.
- **finally**: A block that runs on every way out of a try: a clean finish, a handled exception, an unhandled one, or a return. The pending result is parked while it runs and resumed afterwards. A return inside finally replaces that result and throws away an exception in flight. Python 3.14 warns about it.
- **Context manager**: Any object whose class defines __enter__ and __exit__. with EXPR as name: calls __enter__ and binds its return value to name, runs the body inside an invisible try, and then calls __exit__(exc_type, exc, tb), with three Nones on success. If __exit__ returns a true value the exception is suppressed. @contextlib.contextmanager builds one from a generator: the code before yield is the enter half, after it the exit half.
- **Exception chaining**: An exception raised while another is being handled gets __context__ set to the one being handled; the traceback then says During handling of the above exception, another exception occurred. raise New from old also sets __cause__, and the sentence becomes The above exception was the direct cause of the following exception. from None hides the earlier one.
- **Exception hierarchy**: Exceptions are classes with parents: KeyError and IndexError are LookupErrors, most things you raise yourself are Exceptions, and KeyboardInterrupt and SystemExit sit directly under BaseException so that except Exception lets a program be interrupted. A bare except: catches those too, which is why it is almost always a mistake.
## Modules
- **Module**: An ordinary object, an instance of the class module, made the first time a file is imported. Its __dict__ holds the file's top-level names, so config.TIMEOUT is a lookup in that dict, and a function defined in the file uses it as its globals. See Session 09.
- **import**: Four moves. Look the name up in sys.modules. On a miss, find the file on sys.path, create a module object, register it, and run the file with the module's __dict__ as its globals. Then bind a name in the namespace that ran the import. A module's body runs once per process, however many files import it.
- **sys.modules**: The import cache: a dict from module names to module objects. A module goes in before its body runs, which is how a circular import can find it half built, and comes out again if the body fails. A second import of the same name is one lookup here and runs nothing.
- **from ... import**: from config import retries imports config, then binds retries in your namespace to whatever config.retries points at right then, much like retries = config.retries. It doesn't bind config, and it doesn't follow later rebinding inside the module. Mutating a shared object is still visible through both names.
- **__name__ and the main guard**: __name__ is set before a module's body runs: "__main__" for the script you started (or ran with python -m), and the module's own name when it is imported. if __name__ == "__main__": runs a block only when the file is the program.
- **Circular import**: Two modules that import each other. The second import finds the first module in sys.modules before its body has finished, so reading a name it hasn't defined yet raises AttributeError, or ImportError: cannot import name for a from-import. Read the names inside functions instead, or move them into a third module.
- **sys.path**: The list of folders import searches, in order, after the modules built into the interpreter. Entry 0 is the folder of the script you ran, so a file there with a library's name (random.py, requests.py) hides the real one from the whole program. module.__file__ tells you which file you got.
- **Package**: A module that can hold submodules, usually a folder with an __init__.py. import shop.orders runs shop/__init__.py first, then shop/orders.py, caches both, and binds the name shop. The package's __path__ lists the folders its submodules are searched in.
- **__pycache__**: The folder where Python saves compiled bytecode for imported modules, as files like config.cpython-313.pyc, so the next start can skip compiling when the source hasn't changed. It speeds up start-up, not running time, and it is safe to delete.
- **importlib.reload**: Runs a module's file again into the same module object, so code that reads config.name sees the new values. Names copied out earlier with from-imports, and instances of the old classes, still point at the old objects, which is why restarting is often simpler.