From 87b5dad48689d267225b5dbf6bc9a6f2cc3cba90 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sun, 13 Sep 2026 08:07:44 +0700 Subject: [PATCH] Say what the spec now disagrees with, and that item 6 is closed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BUILT.md gets the section: the 32/32 split, live-is-odd and the two things that fall out of it, wrapping retiring the slot, why resolve answers (Option (Ptr T)) and where the spec already said so, why len is the slot high-water and not the live count, and why a slot is released through the pool rather than through free. Two amendments to a frozen spec, both deferrals: .field and at do not auto-deref a handle, and deref is not overloaded on one. Neither can answer "gone", which is the whole job, and the spec's own worked example resolves first and matches. The pointer hole is written down rather than implied: a (Ptr T) from resolve dies on any insert that grows, which is the slice contract one level down. NEXT.md swept, not just struck — five places beyond item 6 were still asserting that Handle did not exist. --- BUILT.md | 149 +++++++++++++++++++++++++++++++++++++ NEXT.md | 62 +++++++++------ test/programs/handles.flan | 12 +-- 3 files changed, 197 insertions(+), 26 deletions(-) diff --git a/BUILT.md b/BUILT.md index b6b7d8d..efe3ffb 100644 --- a/BUILT.md +++ b/BUILT.md @@ -2510,6 +2510,155 @@ there and matching it from a program. `dev.ml`'s inspector still says "union val frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates. +## `(Handle T)` and the pool, which is what a stale reference answers with + +A handle is a reference to something that can die, which reports that it died rather than silently resolving to +whatever reused its slot. `check.ml` refused `(Handle T)` by name as milestone 6; this is what it stood for, and the +pool came with it because the pool is what makes the report possible. + +The problem is concrete and is not about memory safety. Entities live in a pool; something holds a reference to one — +a projectile chasing it, the UI showing its health. The entity dies, the slot is reused, and a raw index now names a +different entity. Nothing crashes. The projectile chases the wrong thing, at full speed, for the rest of the game. + +It was built for two reasons, both already recorded. On its own terms, for entities referred to across frames. And +because **it is the real gate on managed classes**: plan.org's rule is that nothing starts on `defclass` until +ordinary `struct`, `Handle` and reload semantics work, and `Handle` was the only one of the three missing. That is not +an incidental precondition — `migrate-instances` has to *enumerate* live instances, and a pool behind generational +handles gives that by construction where a world arena and an owned region do not. plan.org presents the three storage +strategies as a free choice and they are not. + +`PORTING.md` found no customer for handles in the author's real game today, so this is deliberately the smallest +correct thing rather than a rich API: eight names, no iteration protocol, no cursor type, no `clone`. + +### A handle is one i64, and the halves are 32 and 32 + +Slot index in the low 32 bits, that slot's generation counter in the high 32. One machine word, so it copies, zeroes +and compares like the integer it is, and it **owns nothing** — the pool is the single owner. That is what lets a +handle sit in a struct field and in a global where a `Vec` may not, and it is the reason the ownership rules needed no +new case: `Types.is_move_only` says yes to `Pool` and no to `Handle`, and the three existing refusals (a struct field, +a union case, a global) picked the pool up unchanged with the messages they already had. + +The index is 32 bits because a `Vec`'s index is an `i32` here and widening indices is one change across every +container, not a pool question. + +### Live is odd, and two things fall out of it + +A slot's generation starts at 0 and is bumped on every allocation *and* on every release, so an odd generation means +live and an even one means dead. Both consequences are load-bearing: + +- **A zeroed handle resolves to nothing.** Generation 0 is even, so ZII gives a `(Handle T)` field the right meaning +for free instead of pointing it at slot 0. `handles.flan` prints one: ``, and resolving it answers `None`. +- **Iteration needs no second array and no spare bit.** Asking whether a slot is live is asking whether its generation +is odd. + +### The generation wraps by retiring the slot + +32 bits is 2^31 allocate/release pairs on one slot — every frame at 60fps for a year and a bit. "Rare" is not an +answer when the failure it produces is the silent wrong one this type exists to prevent, so a release from generation +`0xFFFFFFFF` bumps to 0 and does **not** put the slot back on the free list. The slot is retired: dead forever, its +payload leaked, and no future handle can collide with an old one. Leaking is defined behaviour here, and one slot is a +bounded price for making the collision unrepresentable rather than unlikely. + +### `resolve` answers `(Option (Ptr T))`, and the spec settled that, not this lane + +The task asked whether a lookup should answer `(Option T)`, matching `(get m k)`. It answers `(Option (Ptr T))`, and +`spec-memory.md` already writes it out — its worked example under "Mutating something you matched" is annotated +`(Option (Ptr Enemy))` — for the reason given one line above it: *pattern bindings bind values, so a matched struct is +a copy*. A copy cannot be written back, and writing to the pooled entity in place is what a pool is for. `(Option T)` +would answer a question nobody asked. + +The `Option` half is `get`'s shape and for `get`'s reason: absence is an answer, not a failure. A trap would be wrong +here — the entity dying is the *expected* case, not a bug. + +**The hole, said plainly.** A `(Ptr T)` from `resolve` is invalidated by any `insert` that grows the pool, exactly as +a slice is invalidated by a `push`. The handle survives that and the pointer does not. This is `spec-memory.md`'s +explicit Zig/Odin borrowing contract one level down, and it is worth naming rather than implying, because it +reintroduces the silent-wrong-answer mode the handle just removed for anyone who keeps a resolved pointer across an +insert. Chunked never-moving storage is the fix and it costs code; taking the contract is the smaller correct thing, +given `as-slice` already established it. + +### `len` is the slot high-water and `live` is the count, in that direction + +`(len p)` is how many slots have ever been handed out. `(live p)` is how many of them are live now. It had to be that +way round: `0..(len p)` are the indices `(pool-handle p i)` accepts, so a loop bounded by `len` visits every live +entry. Bounded by the live count instead, it would silently skip entries the moment anything had been released — +which is exactly the quiet wrong answer the whole type exists to remove. + +`(pool-handle p i)` answers `(Option (Handle T))`: the handle of slot `i`, or `None` if that slot is dead. That plus +`len` is the whole of iteration. An index outside `0..(len p)` **traps**, exactly as `(at v i)` traps: an index is an +index here, and answering `None` for one would hide a bug rather than a death. + +### A slot is released through the pool, and that is not a third release point + +`free` consumes its argument as a move, and a handle is a copyable number that owns nothing — consuming one copy would +say nothing about the others. So `(free h)` is refused by name and the release operation is on the owner: +`(release p h)`. `spec-memory.md`'s two release points are untouched: `(free p)` is release point 1 applied to the +pool, and a `free-all` of the region takes the pool with everything else. `release` recycles a slot inside storage the +pool still owns, which is not a release of storage at all. + +It answers `bool` rather than `()`: true if this call released it, false if the handle was already gone. The +generational scheme makes a double release **detectable**, and that is worth handing to the caller — this is the one +place in the language where freeing something twice is an answer instead of a refusal. + +### Growth is transactional, because `retry` re-attempts the same call + +`StorageExhausted`'s restart re-attempts the *same* request, so a failed grow has to leave the pool byte for byte as +it was — including a `cap` that still agrees with the real block sizes, since the next attempt passes `cap` as the +allocator's `old_size`. A pool grows two blocks together (payloads and slot headers), so resizing the first in place +and then failing on the second would leave `cap` describing neither. So the runtime allocates both, copies, and only +then releases the old pair: nothing is mutated after the last thing that can fail. An allocator without `can-free` +leaks the first block when the second fails, which is the defined outcome and not a new one — the request failed +because the region is exhausted, and the region is about to be released whole or its ceiling raised. + +### Two failures, kept apart + +A stale handle is an **answer**: `resolve` says `None` and the program carries on. A pool whose allocator was released +**traps**, through the same epoch check a `Vec` gets — the slot array went with the storage and there is nothing left +to ask. `test/programs/pool-stale-region.flan` is that case, and keeping the two apart is the same rule that keeps a +`Vec`'s generation word and its epoch word apart: they answer different questions and must not be conflated. + +### Two amendments to a frozen spec + +Both are places where `spec-memory.md` describes a handle doing something that cannot answer "gone", which is the one +thing the type exists to do. **This amends it: both are deferred, not built.** + +**1. `.field` and `at` do not auto-deref a handle.** The Places table says `x` may be a struct, a `(Ptr S)` or a +`(Handle S)`, and that the two forms auto-deref exactly one pointer *or handle* level. They auto-deref one pointer +level and nothing else. A `(set (.hp h) ...)` through a handle has two possible meanings when the entity is dead — trap, +or do nothing — and both are worse than the third option, which is the spec's own worked example: resolve first, match, +and the compiler makes you handle the `None`. The spec contradicts itself here and the example is the half that is +right. + +**2. `deref` is not overloaded on `(Handle a)`.** The Generics section says "`deref` yields a value; `resolve` yields a +pointer. Both are overloaded on `(Ptr a)` and `(Handle a)`." `deref` is `(Ptr a)` only. Same reason: `deref` returns a +value and has nowhere to put "gone". + +### What this does not have, and one of the gaps is not a pool question + +- **No `clone`.** Refused by name. A copied pool would carry the same slot generations, so one handle would resolve in +both copies and name two different things — the exact confusion the type removes. A program that wants a second world +builds one and inserts into it, and the new handles say they are new. +- **No pool of an owning element.** `(Pool (Vec i32))` is refused where `(Vec (Vec i32))` is refused and for the same +reason: the type-erased runtime copies and releases slots bytewise. Recursive teardown arrives with `drop`. +- **A handle is not a map key.** For the reason a `Ptr` is not: hashing an identity is a different operation from +hashing what it names, and a stale handle hashes the same as it always did while naming nothing. +- **Handles compare with `=` and not with `<`.** `Types` grew a second predicate, `is_equatable`, beside +`is_comparable`. Two handles are equal exactly when they name the same slot at the same generation, so a stale handle +is never equal to the live one that replaced it — that is worth one integer compare. Ordering them would order a slot +index, which is a free-list artefact and means nothing. +- **A pool passed to a helper is consumed**, because a pool is move-only exactly as a `Vec` is and there is no +borrowing parameter in the language. `test/programs/handles.flan` is one long function for that reason, and it does +not work around it. This is a pre-existing gap and not a pool question: the same sentence is true of every `Vec` in +the tree. + +### What classes still need + +The enumeration primitive is the piece migration was blocked on, and it exists now. What is left is `defclass` itself +and its runtime shape metadata; `migrate-instances`, which is a walk over `(len p)` and `(pool-handle p i)`; generic +functions and method dispatch, whose expensive half is already built and tested (a generic function is an indirection +cell whose body is a dispatch table, and a reload extends the table); and the rule that `Enemy@1` stays resolvable for +as long as any instance holds it — the same rule as "nothing is ever `dlclose`d". + ## Macros: the compiler dlopens the program "Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing diff --git a/NEXT.md b/NEXT.md index 70ec092..107f381 100644 --- a/NEXT.md +++ b/NEXT.md @@ -18,7 +18,8 @@ of a dead session. Note this interacts with the merged one-process build: killin **What `PORTING.md` says NOT to build, with evidence:** escaping closures (one capture site, fixed by one parameter), `Handle`/pools, `Result`/`try`, `handler-case`, `loop`/`recur` and tail calls, user allocators, structural typing — -**none has a customer in that code**. And **generics is not the blocker** there either: the element-changing maps are +**none has a customer in that code**. (`Handle` and the pool were built anyway, and on the other reason: they are the +gate on classes. The finding stands and is why they were built small — see [`BUILT.md`](BUILT.md).) And **generics is not the blocker** there either: the element-changing maps are five-line load-time loops. That last one hangs on a design decision the report states flatly — whether the game's state holds fixed arrays or `Vec`s. @@ -289,9 +290,10 @@ whatever the flag says. plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its -own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. It is here so -that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of -them in plan.org yet: +own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. **All three +now do**, `Handle` and the pool having landed, so this section is no longer "not yet" but "next, and deliberately not +started here". It is here so that a session reading plan.org cold does not take it as the next task. Three things +found while reviewing it, none of them in plan.org yet: - **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection @@ -299,7 +301,8 @@ cells already solve. A generic function is a cell whose body is a dispatch table expensive half of classes is therefore already built and tested. - **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously. -plan.org presents the three as a free choice and they are not. +plan.org presents the three as a free choice and they are not. **The pool is built**, and `(len p)` with +`(pool-handle p i)` is that enumeration. - **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one. @@ -502,18 +505,16 @@ Three things to settle while building it: whether it takes focus or only display the window, being a deliberate stop rather than a failure; and what it does when the program stops while point is mid-edit in another buffer. -**`Handle` and the pool are the real gate on classes, and they are buildable now.** plan.org's rule is that nothing -starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". Checked against the -tree: structs work fully; reload works with one known hole (a changed signature is refused rather than versioned); -**`Handle` does not exist at all** — `check.ml:218` still refuses `(Handle T)` by name. +**~~`Handle` and the pool are the real gate on classes, and they are buildable now.~~ Built.** plan.org's rule is +that nothing starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". All +three now hold: structs work fully; reload works with one known hole (a changed signature is refused rather than +versioned); and `(Handle T)` and `(Pool T)` exist, with the enumeration primitive `migrate-instances` was blocked on. +See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which is what a stale reference answers with". -It is not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a +It was not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a generational `(Handle T)` gives that by construction while a world arena and an owned region do not. plan.org presents the three storage strategies as a free choice and they are not: handles are the one that makes migration possible. - -**The allocator and arena landing today are what unblock it** — a pool is built on them, so `Handle` is buildable now -where it was not this morning. Build `Handle` and the pool next and treat *that* as the gate. It earns its place -independently of classes: stable references to things that move or die is something any game wants. +`(len p)` plus `(pool-handle p i)` is that enumeration, and it is two entry points rather than an iteration protocol. Already banked, and it means classes are less work than plan.org implies: **a generic function is an indirection cell** whose body is a dispatch table, which a reload extends. That is the expensive half of method dispatch, and it is built @@ -526,7 +527,7 @@ working the case through rather than by preference, so the reasoning is worth ke decided against it. It runs code somewhere the reader is not looking, which is the C++ behaviour the author explicitly does not want. It would not even cover the motivating case — `Image` and `Texture2D` are *raylib's* types, and attaching a hook to a foreign type is its own unsolved design question. And its one real advantage, cascading through a container, -is the case `Handle` is about to make rare: entities holding handles hold numbers, not resources. +is the case `Handle` makes rare: entities holding handles hold numbers, not resources. `with-cleanup` / `unwind-protect` was also put and rejected: awkward with several resources, and it reads worse than what already exists. The raylib begin/end pairs that seemed to motivate it are a macro problem, not a primitive one — @@ -836,9 +837,27 @@ run one lane at a time; item 4 is disjoint and runs alongside any of them. `[f32]` would be one copy per ordered pair. A user-written allocator is *not* on this list any more — it wants a C-shaped callback and somewhere to put a `flan_allocator`, neither of which is a type parameter. -6. **`Handle` and the pool.** A reference to something that can die, that reports that it died rather than silently - resolving to whatever reused the slot. Wanted on its own terms for entities referred to across frames, and it is the - real gate on classes. Buildable now that the allocator exists. +6. ~~**`Handle` and the pool.**~~ **Built.** A reference to something that can die, that reports that it died rather + than silently resolving to whatever reused the slot. See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which + is what a stale reference answers with". A handle is one `i64` — slot index low, generation high — so it copies, + zeroes and compares like an integer and owns nothing; a live slot's generation is odd, which makes a zeroed handle + resolve to nothing rather than to slot 0; and a generation that would wrap retires its slot instead, because "rare" + is not an answer when the failure is the silent wrong one the type exists to prevent. + + What the spec did not settle and this lane did, beyond those: `resolve` answers `(Option (Ptr T))` and not + `(Option T)` — the spec's own worked example is annotated that way, for the reason written a line above it, that a + pattern binding binds a value and a copy cannot be written back. `(len p)` is the *slot high-water* and `(live p)` + is the live count, in that direction, so a loop bounded by `len` cannot silently skip a live entry. A slot is + recycled by `(release p h)` on the owner and never by `free`, because a handle owns nothing and consuming one copy + would say nothing about the others — so `spec-memory.md`'s two release points are untouched. + + **Two amendments to the frozen spec, both deferrals**: `.field` and `at` do not auto-deref a handle, and `deref` is + not overloaded on one. Neither can answer "gone", which is the whole job; the spec's own example resolves first and + matches, and that is the half that is right. + + **Still open**: a `(Ptr T)` from `resolve` dies on any `insert` that grows the pool, the same explicit contract a + slice has against `push`. Chunked never-moving storage is the fix and it costs code. And a pool passed to a helper + is consumed, because there is no borrowing parameter — a pre-existing `Vec` gap, not a pool one. 7. ~~**`break` and `continue`, with loop labels.**~~ **Built.** Labels are Odin's in the head position, both blockers are answered, and the refusals name the construct they refuse for. See BUILT.md, "`break` and `continue`, and the @@ -1013,7 +1032,7 @@ natural anyway. **Flexible field order waits for classes, deliberately.** A class has an implementation-defined representation, so the compiler owns the layout and field order stops being observable — any order can match. That is the right place to pay for flexibility, because a class already carries identity and metadata, and a `Vector2` should pay for neither. See the -`defclass` entry: `Handle` is the gate. +`defclass` entry: `Handle` was the gate, and it is built now. Note what this settles from the earlier discussion: writability was the question that decided layout, and requiring identical layout answers it — fields are writable on the ordinary terms, by value a copy and through a `(Ptr T)` the @@ -1273,8 +1292,9 @@ rest on, and because the escape it describes was tested rather than assumed — The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`), a written `fn` literal (`Ast.Fn`), a -`defn`'s name used as a value, and calling anything other than a named function. `(Map K V)`, `(Result T E)` and -`(Handle T)` are still refused beside those as milestone 6; `(Vec T)` is not, any more. Read straight off those lines, +`defn`'s name used as a value, and calling anything other than a named function. `(Result T E)` was still refused +beside those as milestone 6, as `(Map K V)`, `(Handle T)` and `(Vec T)` were when this was written; only `Result` is +now. Read straight off those lines, milestone 6's allocators need milestone 5's function values and the work doubles. **The escape is real and the work did not double** — this is the claim the built thing confirms. All four refusals are about *surface syntax*, and a value the diff --git a/test/programs/handles.flan b/test/programs/handles.flan index 49f8187..99e2189 100644 --- a/test/programs/handles.flan +++ b/test/programs/handles.flan @@ -92,10 +92,12 @@ (Some e) (println (.hp e)) ; 30 None (println -1)) - ;; spec-memory.md's first release point, applied to the owner. Every - ;; handle into it is stale afterwards and says so — which is a - ;; better afterlife than a freed Vec's binding gets, that one being - ;; a compile error the checker can see and this one an answer it - ;; cannot. + ;; spec-memory.md's first release point, applied to the owner. The + ;; runtime leaves the pool empty, so a handle into it would resolve + ;; to None rather than into released storage — but that is not + ;; demonstrable from here and this program does not pretend it is: + ;; free consumes pool, so a resolve on the next line is a compile + ;; error. The runtime property is real and the checker makes it + ;; unreachable. (free pool) 0)))))