diff --git a/BUILT.md b/BUILT.md index b6b7d8d..efe3ffb 100644 --- a/BUILT.md +++ b/BUILT.md @@ -2510,6 +2510,155 @@ there and matching it from a program. `dev.ml`'s inspector still says "union val frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates. +## `(Handle T)` and the pool, which is what a stale reference answers with + +A handle is a reference to something that can die, which reports that it died rather than silently resolving to +whatever reused its slot. `check.ml` refused `(Handle T)` by name as milestone 6; this is what it stood for, and the +pool came with it because the pool is what makes the report possible. + +The problem is concrete and is not about memory safety. Entities live in a pool; something holds a reference to one — +a projectile chasing it, the UI showing its health. The entity dies, the slot is reused, and a raw index now names a +different entity. Nothing crashes. The projectile chases the wrong thing, at full speed, for the rest of the game. + +It was built for two reasons, both already recorded. On its own terms, for entities referred to across frames. And +because **it is the real gate on managed classes**: plan.org's rule is that nothing starts on `defclass` until +ordinary `struct`, `Handle` and reload semantics work, and `Handle` was the only one of the three missing. That is not +an incidental precondition — `migrate-instances` has to *enumerate* live instances, and a pool behind generational +handles gives that by construction where a world arena and an owned region do not. plan.org presents the three storage +strategies as a free choice and they are not. + +`PORTING.md` found no customer for handles in the author's real game today, so this is deliberately the smallest +correct thing rather than a rich API: eight names, no iteration protocol, no cursor type, no `clone`. + +### A handle is one i64, and the halves are 32 and 32 + +Slot index in the low 32 bits, that slot's generation counter in the high 32. One machine word, so it copies, zeroes +and compares like the integer it is, and it **owns nothing** — the pool is the single owner. That is what lets a +handle sit in a struct field and in a global where a `Vec` may not, and it is the reason the ownership rules needed no +new case: `Types.is_move_only` says yes to `Pool` and no to `Handle`, and the three existing refusals (a struct field, +a union case, a global) picked the pool up unchanged with the messages they already had. + +The index is 32 bits because a `Vec`'s index is an `i32` here and widening indices is one change across every +container, not a pool question. + +### Live is odd, and two things fall out of it + +A slot's generation starts at 0 and is bumped on every allocation *and* on every release, so an odd generation means +live and an even one means dead. Both consequences are load-bearing: + +- **A zeroed handle resolves to nothing.** Generation 0 is even, so ZII gives a `(Handle T)` field the right meaning +for free instead of pointing it at slot 0. `handles.flan` prints one: ``, and resolving it answers `None`. +- **Iteration needs no second array and no spare bit.** Asking whether a slot is live is asking whether its generation +is odd. + +### The generation wraps by retiring the slot + +32 bits is 2^31 allocate/release pairs on one slot — every frame at 60fps for a year and a bit. "Rare" is not an +answer when the failure it produces is the silent wrong one this type exists to prevent, so a release from generation +`0xFFFFFFFF` bumps to 0 and does **not** put the slot back on the free list. The slot is retired: dead forever, its +payload leaked, and no future handle can collide with an old one. Leaking is defined behaviour here, and one slot is a +bounded price for making the collision unrepresentable rather than unlikely. + +### `resolve` answers `(Option (Ptr T))`, and the spec settled that, not this lane + +The task asked whether a lookup should answer `(Option T)`, matching `(get m k)`. It answers `(Option (Ptr T))`, and +`spec-memory.md` already writes it out — its worked example under "Mutating something you matched" is annotated +`(Option (Ptr Enemy))` — for the reason given one line above it: *pattern bindings bind values, so a matched struct is +a copy*. A copy cannot be written back, and writing to the pooled entity in place is what a pool is for. `(Option T)` +would answer a question nobody asked. + +The `Option` half is `get`'s shape and for `get`'s reason: absence is an answer, not a failure. A trap would be wrong +here — the entity dying is the *expected* case, not a bug. + +**The hole, said plainly.** A `(Ptr T)` from `resolve` is invalidated by any `insert` that grows the pool, exactly as +a slice is invalidated by a `push`. The handle survives that and the pointer does not. This is `spec-memory.md`'s +explicit Zig/Odin borrowing contract one level down, and it is worth naming rather than implying, because it +reintroduces the silent-wrong-answer mode the handle just removed for anyone who keeps a resolved pointer across an +insert. Chunked never-moving storage is the fix and it costs code; taking the contract is the smaller correct thing, +given `as-slice` already established it. + +### `len` is the slot high-water and `live` is the count, in that direction + +`(len p)` is how many slots have ever been handed out. `(live p)` is how many of them are live now. It had to be that +way round: `0..(len p)` are the indices `(pool-handle p i)` accepts, so a loop bounded by `len` visits every live +entry. Bounded by the live count instead, it would silently skip entries the moment anything had been released — +which is exactly the quiet wrong answer the whole type exists to remove. + +`(pool-handle p i)` answers `(Option (Handle T))`: the handle of slot `i`, or `None` if that slot is dead. That plus +`len` is the whole of iteration. An index outside `0..(len p)` **traps**, exactly as `(at v i)` traps: an index is an +index here, and answering `None` for one would hide a bug rather than a death. + +### A slot is released through the pool, and that is not a third release point + +`free` consumes its argument as a move, and a handle is a copyable number that owns nothing — consuming one copy would +say nothing about the others. So `(free h)` is refused by name and the release operation is on the owner: +`(release p h)`. `spec-memory.md`'s two release points are untouched: `(free p)` is release point 1 applied to the +pool, and a `free-all` of the region takes the pool with everything else. `release` recycles a slot inside storage the +pool still owns, which is not a release of storage at all. + +It answers `bool` rather than `()`: true if this call released it, false if the handle was already gone. The +generational scheme makes a double release **detectable**, and that is worth handing to the caller — this is the one +place in the language where freeing something twice is an answer instead of a refusal. + +### Growth is transactional, because `retry` re-attempts the same call + +`StorageExhausted`'s restart re-attempts the *same* request, so a failed grow has to leave the pool byte for byte as +it was — including a `cap` that still agrees with the real block sizes, since the next attempt passes `cap` as the +allocator's `old_size`. A pool grows two blocks together (payloads and slot headers), so resizing the first in place +and then failing on the second would leave `cap` describing neither. So the runtime allocates both, copies, and only +then releases the old pair: nothing is mutated after the last thing that can fail. An allocator without `can-free` +leaks the first block when the second fails, which is the defined outcome and not a new one — the request failed +because the region is exhausted, and the region is about to be released whole or its ceiling raised. + +### Two failures, kept apart + +A stale handle is an **answer**: `resolve` says `None` and the program carries on. A pool whose allocator was released +**traps**, through the same epoch check a `Vec` gets — the slot array went with the storage and there is nothing left +to ask. `test/programs/pool-stale-region.flan` is that case, and keeping the two apart is the same rule that keeps a +`Vec`'s generation word and its epoch word apart: they answer different questions and must not be conflated. + +### Two amendments to a frozen spec + +Both are places where `spec-memory.md` describes a handle doing something that cannot answer "gone", which is the one +thing the type exists to do. **This amends it: both are deferred, not built.** + +**1. `.field` and `at` do not auto-deref a handle.** The Places table says `x` may be a struct, a `(Ptr S)` or a +`(Handle S)`, and that the two forms auto-deref exactly one pointer *or handle* level. They auto-deref one pointer +level and nothing else. A `(set (.hp h) ...)` through a handle has two possible meanings when the entity is dead — trap, +or do nothing — and both are worse than the third option, which is the spec's own worked example: resolve first, match, +and the compiler makes you handle the `None`. The spec contradicts itself here and the example is the half that is +right. + +**2. `deref` is not overloaded on `(Handle a)`.** The Generics section says "`deref` yields a value; `resolve` yields a +pointer. Both are overloaded on `(Ptr a)` and `(Handle a)`." `deref` is `(Ptr a)` only. Same reason: `deref` returns a +value and has nowhere to put "gone". + +### What this does not have, and one of the gaps is not a pool question + +- **No `clone`.** Refused by name. A copied pool would carry the same slot generations, so one handle would resolve in +both copies and name two different things — the exact confusion the type removes. A program that wants a second world +builds one and inserts into it, and the new handles say they are new. +- **No pool of an owning element.** `(Pool (Vec i32))` is refused where `(Vec (Vec i32))` is refused and for the same +reason: the type-erased runtime copies and releases slots bytewise. Recursive teardown arrives with `drop`. +- **A handle is not a map key.** For the reason a `Ptr` is not: hashing an identity is a different operation from +hashing what it names, and a stale handle hashes the same as it always did while naming nothing. +- **Handles compare with `=` and not with `<`.** `Types` grew a second predicate, `is_equatable`, beside +`is_comparable`. Two handles are equal exactly when they name the same slot at the same generation, so a stale handle +is never equal to the live one that replaced it — that is worth one integer compare. Ordering them would order a slot +index, which is a free-list artefact and means nothing. +- **A pool passed to a helper is consumed**, because a pool is move-only exactly as a `Vec` is and there is no +borrowing parameter in the language. `test/programs/handles.flan` is one long function for that reason, and it does +not work around it. This is a pre-existing gap and not a pool question: the same sentence is true of every `Vec` in +the tree. + +### What classes still need + +The enumeration primitive is the piece migration was blocked on, and it exists now. What is left is `defclass` itself +and its runtime shape metadata; `migrate-instances`, which is a walk over `(len p)` and `(pool-handle p i)`; generic +functions and method dispatch, whose expensive half is already built and tested (a generic function is an indirection +cell whose body is a dispatch table, and a reload extends the table); and the rule that `Enemy@1` stays resolvable for +as long as any instance holds it — the same rule as "nothing is ever `dlclose`d". + ## Macros: the compiler dlopens the program "Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing diff --git a/NEXT.md b/NEXT.md index 70ec092..107f381 100644 --- a/NEXT.md +++ b/NEXT.md @@ -18,7 +18,8 @@ of a dead session. Note this interacts with the merged one-process build: killin **What `PORTING.md` says NOT to build, with evidence:** escaping closures (one capture site, fixed by one parameter), `Handle`/pools, `Result`/`try`, `handler-case`, `loop`/`recur` and tail calls, user allocators, structural typing — -**none has a customer in that code**. And **generics is not the blocker** there either: the element-changing maps are +**none has a customer in that code**. (`Handle` and the pool were built anyway, and on the other reason: they are the +gate on classes. The finding stands and is why they were built small — see [`BUILT.md`](BUILT.md).) And **generics is not the blocker** there either: the element-changing maps are five-line load-time loops. That last one hangs on a design decision the report states flatly — whether the game's state holds fixed arrays or `Vec`s. @@ -289,9 +290,10 @@ whatever the flag says. plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its -own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. It is here so -that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of -them in plan.org yet: +own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. **All three +now do**, `Handle` and the pool having landed, so this section is no longer "not yet" but "next, and deliberately not +started here". It is here so that a session reading plan.org cold does not take it as the next task. Three things +found while reviewing it, none of them in plan.org yet: - **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection @@ -299,7 +301,8 @@ cells already solve. A generic function is a cell whose body is a dispatch table expensive half of classes is therefore already built and tested. - **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously. -plan.org presents the three as a free choice and they are not. +plan.org presents the three as a free choice and they are not. **The pool is built**, and `(len p)` with +`(pool-handle p i)` is that enumeration. - **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one. @@ -502,18 +505,16 @@ Three things to settle while building it: whether it takes focus or only display the window, being a deliberate stop rather than a failure; and what it does when the program stops while point is mid-edit in another buffer. -**`Handle` and the pool are the real gate on classes, and they are buildable now.** plan.org's rule is that nothing -starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". Checked against the -tree: structs work fully; reload works with one known hole (a changed signature is refused rather than versioned); -**`Handle` does not exist at all** — `check.ml:218` still refuses `(Handle T)` by name. +**~~`Handle` and the pool are the real gate on classes, and they are buildable now.~~ Built.** plan.org's rule is +that nothing starts on managed classes "until ordinary `struct`, `Handle`, and reload semantics are working". All +three now hold: structs work fully; reload works with one known hole (a changed signature is refused rather than +versioned); and `(Handle T)` and `(Pool T)` exist, with the enumeration primitive `migrate-instances` was blocked on. +See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which is what a stale reference answers with". -It is not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a +It was not an incidental precondition. `migrate-instances` has to *enumerate* live instances, and a pool behind a generational `(Handle T)` gives that by construction while a world arena and an owned region do not. plan.org presents the three storage strategies as a free choice and they are not: handles are the one that makes migration possible. - -**The allocator and arena landing today are what unblock it** — a pool is built on them, so `Handle` is buildable now -where it was not this morning. Build `Handle` and the pool next and treat *that* as the gate. It earns its place -independently of classes: stable references to things that move or die is something any game wants. +`(len p)` plus `(pool-handle p i)` is that enumeration, and it is two entry points rather than an iteration protocol. Already banked, and it means classes are less work than plan.org implies: **a generic function is an indirection cell** whose body is a dispatch table, which a reload extends. That is the expensive half of method dispatch, and it is built @@ -526,7 +527,7 @@ working the case through rather than by preference, so the reasoning is worth ke decided against it. It runs code somewhere the reader is not looking, which is the C++ behaviour the author explicitly does not want. It would not even cover the motivating case — `Image` and `Texture2D` are *raylib's* types, and attaching a hook to a foreign type is its own unsolved design question. And its one real advantage, cascading through a container, -is the case `Handle` is about to make rare: entities holding handles hold numbers, not resources. +is the case `Handle` makes rare: entities holding handles hold numbers, not resources. `with-cleanup` / `unwind-protect` was also put and rejected: awkward with several resources, and it reads worse than what already exists. The raylib begin/end pairs that seemed to motivate it are a macro problem, not a primitive one — @@ -836,9 +837,27 @@ run one lane at a time; item 4 is disjoint and runs alongside any of them. `[f32]` would be one copy per ordered pair. A user-written allocator is *not* on this list any more — it wants a C-shaped callback and somewhere to put a `flan_allocator`, neither of which is a type parameter. -6. **`Handle` and the pool.** A reference to something that can die, that reports that it died rather than silently - resolving to whatever reused the slot. Wanted on its own terms for entities referred to across frames, and it is the - real gate on classes. Buildable now that the allocator exists. +6. ~~**`Handle` and the pool.**~~ **Built.** A reference to something that can die, that reports that it died rather + than silently resolving to whatever reused the slot. See [`BUILT.md`](BUILT.md), "`(Handle T)` and the pool, which + is what a stale reference answers with". A handle is one `i64` — slot index low, generation high — so it copies, + zeroes and compares like an integer and owns nothing; a live slot's generation is odd, which makes a zeroed handle + resolve to nothing rather than to slot 0; and a generation that would wrap retires its slot instead, because "rare" + is not an answer when the failure is the silent wrong one the type exists to prevent. + + What the spec did not settle and this lane did, beyond those: `resolve` answers `(Option (Ptr T))` and not + `(Option T)` — the spec's own worked example is annotated that way, for the reason written a line above it, that a + pattern binding binds a value and a copy cannot be written back. `(len p)` is the *slot high-water* and `(live p)` + is the live count, in that direction, so a loop bounded by `len` cannot silently skip a live entry. A slot is + recycled by `(release p h)` on the owner and never by `free`, because a handle owns nothing and consuming one copy + would say nothing about the others — so `spec-memory.md`'s two release points are untouched. + + **Two amendments to the frozen spec, both deferrals**: `.field` and `at` do not auto-deref a handle, and `deref` is + not overloaded on one. Neither can answer "gone", which is the whole job; the spec's own example resolves first and + matches, and that is the half that is right. + + **Still open**: a `(Ptr T)` from `resolve` dies on any `insert` that grows the pool, the same explicit contract a + slice has against `push`. Chunked never-moving storage is the fix and it costs code. And a pool passed to a helper + is consumed, because there is no borrowing parameter — a pre-existing `Vec` gap, not a pool one. 7. ~~**`break` and `continue`, with loop labels.**~~ **Built.** Labels are Odin's in the head position, both blockers are answered, and the refusals name the construct they refuse for. See BUILT.md, "`break` and `continue`, and the @@ -1013,7 +1032,7 @@ natural anyway. **Flexible field order waits for classes, deliberately.** A class has an implementation-defined representation, so the compiler owns the layout and field order stops being observable — any order can match. That is the right place to pay for flexibility, because a class already carries identity and metadata, and a `Vector2` should pay for neither. See the -`defclass` entry: `Handle` is the gate. +`defclass` entry: `Handle` was the gate, and it is built now. Note what this settles from the earlier discussion: writability was the question that decided layout, and requiring identical layout answers it — fields are writable on the ordinary terms, by value a copy and through a `(Ptr T)` the @@ -1273,8 +1292,9 @@ rest on, and because the escape it describes was tested rather than assumed — The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`), a written `fn` literal (`Ast.Fn`), a -`defn`'s name used as a value, and calling anything other than a named function. `(Map K V)`, `(Result T E)` and -`(Handle T)` are still refused beside those as milestone 6; `(Vec T)` is not, any more. Read straight off those lines, +`defn`'s name used as a value, and calling anything other than a named function. `(Result T E)` was still refused +beside those as milestone 6, as `(Map K V)`, `(Handle T)` and `(Vec T)` were when this was written; only `Result` is +now. Read straight off those lines, milestone 6's allocators need milestone 5's function values and the work doubles. **The escape is real and the work did not double** — this is the claim the built thing confirms. All four refusals are about *surface syntax*, and a value the diff --git a/test/programs/handles.flan b/test/programs/handles.flan index 49f8187..99e2189 100644 --- a/test/programs/handles.flan +++ b/test/programs/handles.flan @@ -92,10 +92,12 @@ (Some e) (println (.hp e)) ; 30 None (println -1)) - ;; spec-memory.md's first release point, applied to the owner. Every - ;; handle into it is stale afterwards and says so — which is a - ;; better afterlife than a freed Vec's binding gets, that one being - ;; a compile error the checker can see and this one an answer it - ;; cannot. + ;; spec-memory.md's first release point, applied to the owner. The + ;; runtime leaves the pool empty, so a handle into it would resolve + ;; to None rather than into released storage — but that is not + ;; demonstrable from here and this program does not pretend it is: + ;; free consumes pool, so a resolve on the next line is a compile + ;; error. The runtime property is real and the checker makes it + ;; unreachable. (free pool) 0)))))