From c2116a5e71b430d5ead8dbe3fc9fc39616af26cf Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sat, 12 Sep 2026 10:37:37 +0700 Subject: [PATCH 1/2] The four allocator questions are answered; say so, and say what to build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spec-memory.md settled them and NEXT.md kept asking. Ranked item 1 now states the decisions and points at the section that holds them. The build order is new. An allocator is a procedure plus a data pointer, and check.ml refuses function values four ways as milestone 5, which reads as milestone 6 depending on milestone 5. It does not: every one of those refusals is about surface syntax, and the compiler already builds function values no Flan type names — a handler-bind clause reaching flan_handler's fn pointer, and a dev build's call through an indirection cell. So Allocator is a builtin opaque type and the built-in allocators need nothing from milestone 5. A user-written one does, because it needs a defn's name in value position. Also recorded: 5 and 6 interleave, since the macro expander is blocked on union values; the operation table has free-all and no retain-capacity, which Zig splits and a frame arena wants; and two line citations in the Allocators section point at the wrong place while the claims they support are true. --- NEXT.md | 124 +++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 110 insertions(+), 14 deletions(-) diff --git a/NEXT.md b/NEXT.md index 4f2b851..40a4c1e 100644 --- a/NEXT.md +++ b/NEXT.md @@ -325,20 +325,18 @@ plan.org's single line on it (831) names a `for` the language does not have and game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted - procedures passed as a runtime argument (`src/llvm_backend.cpp`, `Map_Info`). That is exactly what - `spec-memory.md` already specifies. Before writing any of it, settle the four things the Odin and Carp studies - converged on, because all four are cheap now and expensive after: - - **When is storage released?** `spec-memory.md` never says. Odin's answer is `defer delete`, which Flan cannot - express — `defer` is function-scoped and refused in a `let`, a loop or a branch. Carp's answer is scope-end - frees, which it then could not reconcile with arenas and so has no allocator at all. - - **A `drop` hook** for a struct owning something that is not memory — a `Texture2D`, a socket, a file handle. Carp - shipped `delete` and then had to add a separate `drop` interface (`docs/Drop.md`). The hard part is ours alone: - what runs `drop` when the arena resets underneath the value. - - **`alignment`** appears nowhere in the design. Every Odin allocation carries it, and `#soa` and component-wise - fixed arrays want 16-byte alignment. - - **Allocation failure.** Unspecified. Odin returns an ignorable error, so a failed `append` silently appends - nothing. Flan has a better answer available for free: a `StorageExhausted` condition with a `retry` restart. - Decide which, because `push`/`put`/`clone`'s signatures depend on it. + procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what + `spec-memory.md` specifies. + + **The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, frozen along + with the rest of that file. Storage is released at exactly two points, `(free v)` and `(free-all a)`, and nothing is + released at scope exit. `drop` takes a pointer, fires only inside `free`, and a value carrying one may not be + constructed against an allocator lacking `can-free` — which is what stops an arena reset from having to run + destructors. Alignment is a property of the type, computed at the call site, passed as a parameter, and not stored in + the header. And an allocator that cannot satisfy a request signals `StorageExhausted` inside a `restart-case` + offering `retry`, so `push`, `put` and `clone` keep the result types they would otherwise have and no allocating + operation returns a `Result`. One question is left open there on purpose; it does not block the build. The order to + build the thing in is below. 2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard @@ -352,8 +350,106 @@ plan.org's single line on it (831) names a `for` the language does not have and establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind` plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists. +### `Vec` and `Map` — the order to build them in + +The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an +allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four +ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`, line 209), a written `fn` literal +(`Ast.Fn`, 458), a `defn`'s name used as a value (571), and calling anything other than a named function (1023). +`(Vec T)`, `(Map K V)`, `(Result T E)` and `(Handle T)` are refused at 215–218 as milestone 6. Read straight off those +lines, milestone 6's allocators need milestone 5's function values and the work doubles. + +**The escape is real and the work does not double.** All four refusals are about *surface syntax*, and a value the +compiler builds that no surface form names trips none of them. The compiler already does exactly this, twice: + +- A `handler-bind` clause is lowered to a function whose address goes into a `flan_handler` and is called back through + `h->fn(condition, xfer)` (`runtime/flan_rt.c:38` and `:66`). `check.ml` builds that body as its own `Tast.fn` + (`:619`, `:654`), not as an `Ast.Fn`, so line 458 never sees it, and no Flan type names the result. +- In a dev build, `emit.ml`'s `call` loads a pointer out of an indirection cell and calls through it + (`lib/emit.ml:781`–`793`). That is the indirect call line 1023 refuses in source, emitted routinely. + +It is also what `spec-memory.md` already assumes for `Map`: the hash and equality pair is compiler-emitted and passed +as a runtime argument. Odin's `Map_Info` is two contextless `proc` fields (`base/runtime/core.odin:369`), and Odin's +`Allocator` is a `procedure` plus a `data: rawptr` (`:422`) — the same shape, reached the same way. If the hash pair is +expressible with no function type in the surface language, so is the allocator's procedure. + +So: **`Allocator` is a builtin opaque type, the way `string` is a builtin ptr+len.** It is a `Types.t` case with no +user-writable constructor. Its procedure is an ordinary top-level function resolved to a symbol at the emit site, and +`vec-new`, `push`, `put`, `clone`, `free` and `free-all` are named calls, which `check_call` already routes through +`named_call` (`check.ml:1021`). **The built-in allocators need nothing from milestone 5.** + +What *does* need milestone 5 is a **user-written** allocator: the moment a program says "here is my proc, make an +`Allocator` from it", it needs a `defn`'s name in value position, which is `check.ml:571` verbatim. That is a real +limit and not a fatal one — Odin ships arena, general-purpose, stack, pool and scratch in its own std, and most +programs write none. Ship the built-in set; user allocators arrive with function values. + +**5 and 6 interleave rather than nest.** plan.org orders generics and macros (5) before allocators and containers (6), +and that order cannot hold: the macro expander is blocked on `Form` being a Flan union and union *values* are milestone +6 (see "Macros" below). Conditions and restarts, also listed under 6, are already three steps of four. The milestone +numbers are a topological hint, not a sequence. Take 6's container half first, 5's generics half second, and 5's +expander last, on 6's unions. + +1. **`Allocator` and the arena.** The builtin type, the four operations (`alloc`, `resize`, `free`, `free-all`) with + `size` and `align` as parameters, the capability set with `can-free` in it, `with-allocator`, and the dev-build + epoch counter. No container yet. Odin reads its capability set back through the same procedure — `Query_Features` + returning an `Allocator_Mode_Set` (`core/mem/allocators.odin:315`) — and a field on the allocator value is the same + information without the round trip. +2. **`(Vec T)`**, over the type-erased runtime: `ptr + len + cap + allocator` in release, plus the generation word and + the epoch in dev. `push`, `reserve`, `at`, `len`, `as-slice`, `free`, `clone`. `size_of`/`align_of` are produced at + the call site, which here is simply the concrete call site, there being no generics yet. This is what unblocks the + accumulation pattern `handler-bind` was built for — `(fn [c] (push errors c) ...)` — which conditions want today and + cannot have. +3. **`StorageExhausted` and `retry`, with step 2 and not after.** `restart-case` and the transfer channel exist, so + this is a compiler-emitted restart at each allocating site and little else. It goes in at the same time because the + signatures depend on it: retrofitting it later adds a transfer check to every call site of every allocating + operation, which is the whole point of having decided it now. +4. **`(Map K V)`** — flat open-addressed key and value arrays, with a compiler-emitted hash and equality pair per key + type passed as arguments. `spec-memory.md`'s structural-key restriction holds this to the built-in key set, so there + is no dispatch to design. +5. **`drop`.** The hook, the transitive move-only and non-`clone`able rules, and the refusal to construct a + `drop`-carrying value against an allocator without `can-free`. It is additive — no type in the repo has a hook today + — but the `can-free` refusal has to land with the construction path it guards, before any arena-allocated container + of a user struct is trusted. +6. **`(Result T E)` and `try`, then the rest of union values.** Unions are what `Form` needs, and `Form` is what the + macro expander needs. +7. **Generics and monomorphisation**, then function values. User-written allocators and escaping closures both fall out + of the second. +8. **The macro expander**, last, on 6's unions. + +**What is genuinely unsettled, and none of it blocks step 1.** + +- `spec-memory.md`'s "Open: catching a use-after-release statically" is open by decision, not by omission. It says the + static rule needs to know which allocator a construction used and that `with-allocator` plus `context/allocator` are + exactly what deny that knowledge; the shipping answer is the dev-build epoch trap, which is specified and buildable. + It also says what would settle it — real Flan programs using arenas, to show whether the escapes that occur are + lexical — and that is evidence this repo cannot produce until after step 2. **Build against the epoch trap.** +- **The operation table may be one operation short.** It has `free-all` and nothing else. Zig's `ArenaAllocator.reset` + takes a `ResetMode` of `free_all`, `retain_capacity` or `retain_with_limit` + (`lib/std/heap/arena_allocator.zig:57`–`69`), and for a frame arena reset every frame, retain-capacity is the normal + case and free-all is the unusual one — handing the pages back to the backing allocator only to ask for them again. + Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing buffer and the call only + sets `offset = 0` (`core/mem/allocators.odin:287`). Flan should decide whether `free-all` means either of these or + takes the mode. It is an amendment to `spec-memory.md`, it is small, and it is better made before the arena is + written than after. +- Escaping closures are still deferred (`spec-memory.md`, "Function values", case 3), and a user-written allocator is + not one — its procedure is a top-level `defn` with no captured environment. The two should not be conflated when + function values arrive. + ### Bugs found and not yet fixed +- **Two citations in `spec-memory.md`'s Allocators section do not land where they say.** Checked against Odin + `819fdc7a8` and Carp `ea121b5a`, every other one is exact — `Map_Info` at `base/runtime/core.odin:369`, + `Allocator_Proc` at `:422`, the arena answering `.Free` with `.Mode_Not_Implemented` at + `core/mem/allocators.odin:307`–`308`, `#optional_allocator_error` on `append_elem` at + `base/runtime/core_builtin.odin:767`, and `// TODO(bill): Better error handling for failed reservation` at + `base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`, + not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for, + that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn` + at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited. Also worth a word when that + section is next edited: Carp's `getDropFunc` is at `Memory.hs:804` and the drop-before-delete emit at `Emit.hs:1044`, + both within a couple of lines of what is written, and `docs/Drop.md` confirms the sentence they support — "`A.drop` + will be run ... when the `let` scope ends". + - ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end: `check.sh` already builds this one `--dev` and runs it under `timeout 5`, keeping what it printed before it From 651d04496fc77a698e8ebb7080b7906aced25068 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sat, 12 Sep 2026 10:39:44 +0700 Subject: [PATCH 2/2] Vec does not buy the accumulation pattern; capture does not exist yet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 2 claimed (Vec T) unblocks (fn [c] (push errors c) ...) and it does not. check.ml's captured reads ctx.outer only to raise a better refusal and lookup never reads it at all, so closure capture — spec-memory.md's case 2 — is unbuilt. And the same spec captures a Vec by pointer rather than by value, a split no capture path has needed because every capturable type today is a value type. Two separate items, named as such under step 2. Item 1 stops restating the four decisions and points at the section holding them, keeping only the consequence the build order turns on. The Carp line numbers were checked and are right, so they come out of the bug entry. --- NEXT.md | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/NEXT.md b/NEXT.md index 40a4c1e..1fc9a6e 100644 --- a/NEXT.md +++ b/NEXT.md @@ -328,15 +328,12 @@ plan.org's single line on it (831) names a `for` the language does not have and procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what `spec-memory.md` specifies. - **The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, frozen along - with the rest of that file. Storage is released at exactly two points, `(free v)` and `(free-all a)`, and nothing is - released at scope exit. `drop` takes a pointer, fires only inside `free`, and a value carrying one may not be - constructed against an allocator lacking `can-free` — which is what stops an arena reset from having to run - destructors. Alignment is a property of the type, computed at the call site, passed as a parameter, and not stored in - the header. And an allocator that cannot satisfy a request signals `StorageExhausted` inside a `restart-case` - offering `retry`, so `push`, `put` and `clone` keep the result types they would otherwise have and no allocating - operation returns a `Result`. One question is left open there on purpose; it does not block the build. The order to - build the thing in is below. + **The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, which is + frozen along with the rest of that file: when storage is released, the `drop` hook, alignment, and allocation + failure. Read them there rather than in a second copy here. The one consequence the build order below turns on is + that no allocating operation returns an error — a failure signals `StorageExhausted` under a `retry` restart — so + `push` and `put` are `Unit`, `clone` returns the container, and no signature grows a `Result`. One question is left + open in that section on purpose; it does not block the build. 2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard @@ -396,9 +393,17 @@ expander last, on 6's unions. information without the round trip. 2. **`(Vec T)`**, over the type-erased runtime: `ptr + len + cap + allocator` in release, plus the generation word and the epoch in dev. `push`, `reserve`, `at`, `len`, `as-slice`, `free`, `clone`. `size_of`/`align_of` are produced at - the call site, which here is simply the concrete call site, there being no generics yet. This is what unblocks the - accumulation pattern `handler-bind` was built for — `(fn [c] (push errors c) ...)` — which conditions want today and - cannot have. + the call site, which here is simply the concrete call site, there being no generics yet. + + **`Vec` alone does not buy the accumulation pattern, and it is worth knowing that before step 2 is scoped.** + `(fn [c] (push errors c) ...)` over an enclosing `(Vec ParseError)` is the pattern `handler-bind` exists for, and it + needs two further things. Capture does not exist at all: `check.ml`'s `captured` (`:176`) consults `ctx.outer` only + to raise a better refusal — "a handler cannot see %s ... Use a global, or pass it on the condition" — and `lookup` + (`:167`) never reads `outer`. That is `spec-memory.md`'s case 2, a non-escaping `fn` capturing by value into a stack + environment, and it is unbuilt. On top of it, the same spec says a captured `Vec` or `Map` is captured **by pointer, + not moved**, and every capturable type today is a value type, so the by-value/by-pointer split has never had to + exist in a capture path. Both are their own work and neither falls out of `Vec`. Step 2 delivers a container; + accumulating into one from a handler is a separate item and should be planned as one. 3. **`StorageExhausted` and `retry`, with step 2 and not after.** `restart-case` and the transfer channel exist, so this is a compiler-emitted restart at each allocating site and little else. It goes in at the same time because the signatures depend on it: retrofitting it later adds a transfer check to every call site of every allocating @@ -445,10 +450,9 @@ expander last, on 6's unions. `base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`, not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for, that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn` - at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited. Also worth a word when that - section is next edited: Carp's `getDropFunc` is at `Memory.hs:804` and the drop-before-delete emit at `Emit.hs:1044`, - both within a couple of lines of what is written, and `docs/Drop.md` confirms the sentence they support — "`A.drop` - will be run ... when the `let` scope ends". + at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited, and the Carp citations are right: + `getDropFunc` is `Memory.hs:804`, the drop-before-delete emit is `Emit.hs:1044`, and `docs/Drop.md` says outright + that `A.drop` "will be run ... when the `let` scope ends". - ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end: