diff --git a/NEXT.md b/NEXT.md index 4f2b851..40a4c1e 100644 --- a/NEXT.md +++ b/NEXT.md @@ -325,20 +325,18 @@ plan.org's single line on it (831) names a `for` the language does not have and game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted - procedures passed as a runtime argument (`src/llvm_backend.cpp`, `Map_Info`). That is exactly what - `spec-memory.md` already specifies. Before writing any of it, settle the four things the Odin and Carp studies - converged on, because all four are cheap now and expensive after: - - **When is storage released?** `spec-memory.md` never says. Odin's answer is `defer delete`, which Flan cannot - express — `defer` is function-scoped and refused in a `let`, a loop or a branch. Carp's answer is scope-end - frees, which it then could not reconcile with arenas and so has no allocator at all. - - **A `drop` hook** for a struct owning something that is not memory — a `Texture2D`, a socket, a file handle. Carp - shipped `delete` and then had to add a separate `drop` interface (`docs/Drop.md`). The hard part is ours alone: - what runs `drop` when the arena resets underneath the value. - - **`alignment`** appears nowhere in the design. Every Odin allocation carries it, and `#soa` and component-wise - fixed arrays want 16-byte alignment. - - **Allocation failure.** Unspecified. Odin returns an ignorable error, so a failed `append` silently appends - nothing. Flan has a better answer available for free: a `StorageExhausted` condition with a `retry` restart. - Decide which, because `push`/`put`/`clone`'s signatures depend on it. + procedures passed as a runtime argument (`Map_Info`, `base/runtime/core.odin:369`). That runtime is what + `spec-memory.md` specifies. + + **The four questions that used to sit here are answered**, in `spec-memory.md`'s "Allocators" section, frozen along + with the rest of that file. Storage is released at exactly two points, `(free v)` and `(free-all a)`, and nothing is + released at scope exit. `drop` takes a pointer, fires only inside `free`, and a value carrying one may not be + constructed against an allocator lacking `can-free` — which is what stops an arena reset from having to run + destructors. Alignment is a property of the type, computed at the call site, passed as a parameter, and not stored in + the header. And an allocator that cannot satisfy a request signals `StorageExhausted` inside a `restart-case` + offering `retry`, so `push`, `put` and `clone` keep the result types they would otherwise have and no allocating + operation returns a `Result`. One question is left open there on purpose; it does not block the build. The order to + build the thing in is below. 2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard @@ -352,8 +350,106 @@ plan.org's single line on it (831) names a `for` the language does not have and establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind` plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists. +### `Vec` and `Map` — the order to build them in + +The dependency nobody had written down, and the reason it looked worse than it is. `spec-memory.md` defines an +allocator as "a procedure plus an opaque data pointer" — a function value. `check.ml` refuses function values four +ways, and all four say milestone 5: a written `(Fn ...)` annotation (`Ast.Tfn`, line 209), a written `fn` literal +(`Ast.Fn`, 458), a `defn`'s name used as a value (571), and calling anything other than a named function (1023). +`(Vec T)`, `(Map K V)`, `(Result T E)` and `(Handle T)` are refused at 215–218 as milestone 6. Read straight off those +lines, milestone 6's allocators need milestone 5's function values and the work doubles. + +**The escape is real and the work does not double.** All four refusals are about *surface syntax*, and a value the +compiler builds that no surface form names trips none of them. The compiler already does exactly this, twice: + +- A `handler-bind` clause is lowered to a function whose address goes into a `flan_handler` and is called back through + `h->fn(condition, xfer)` (`runtime/flan_rt.c:38` and `:66`). `check.ml` builds that body as its own `Tast.fn` + (`:619`, `:654`), not as an `Ast.Fn`, so line 458 never sees it, and no Flan type names the result. +- In a dev build, `emit.ml`'s `call` loads a pointer out of an indirection cell and calls through it + (`lib/emit.ml:781`–`793`). That is the indirect call line 1023 refuses in source, emitted routinely. + +It is also what `spec-memory.md` already assumes for `Map`: the hash and equality pair is compiler-emitted and passed +as a runtime argument. Odin's `Map_Info` is two contextless `proc` fields (`base/runtime/core.odin:369`), and Odin's +`Allocator` is a `procedure` plus a `data: rawptr` (`:422`) — the same shape, reached the same way. If the hash pair is +expressible with no function type in the surface language, so is the allocator's procedure. + +So: **`Allocator` is a builtin opaque type, the way `string` is a builtin ptr+len.** It is a `Types.t` case with no +user-writable constructor. Its procedure is an ordinary top-level function resolved to a symbol at the emit site, and +`vec-new`, `push`, `put`, `clone`, `free` and `free-all` are named calls, which `check_call` already routes through +`named_call` (`check.ml:1021`). **The built-in allocators need nothing from milestone 5.** + +What *does* need milestone 5 is a **user-written** allocator: the moment a program says "here is my proc, make an +`Allocator` from it", it needs a `defn`'s name in value position, which is `check.ml:571` verbatim. That is a real +limit and not a fatal one — Odin ships arena, general-purpose, stack, pool and scratch in its own std, and most +programs write none. Ship the built-in set; user allocators arrive with function values. + +**5 and 6 interleave rather than nest.** plan.org orders generics and macros (5) before allocators and containers (6), +and that order cannot hold: the macro expander is blocked on `Form` being a Flan union and union *values* are milestone +6 (see "Macros" below). Conditions and restarts, also listed under 6, are already three steps of four. The milestone +numbers are a topological hint, not a sequence. Take 6's container half first, 5's generics half second, and 5's +expander last, on 6's unions. + +1. **`Allocator` and the arena.** The builtin type, the four operations (`alloc`, `resize`, `free`, `free-all`) with + `size` and `align` as parameters, the capability set with `can-free` in it, `with-allocator`, and the dev-build + epoch counter. No container yet. Odin reads its capability set back through the same procedure — `Query_Features` + returning an `Allocator_Mode_Set` (`core/mem/allocators.odin:315`) — and a field on the allocator value is the same + information without the round trip. +2. **`(Vec T)`**, over the type-erased runtime: `ptr + len + cap + allocator` in release, plus the generation word and + the epoch in dev. `push`, `reserve`, `at`, `len`, `as-slice`, `free`, `clone`. `size_of`/`align_of` are produced at + the call site, which here is simply the concrete call site, there being no generics yet. This is what unblocks the + accumulation pattern `handler-bind` was built for — `(fn [c] (push errors c) ...)` — which conditions want today and + cannot have. +3. **`StorageExhausted` and `retry`, with step 2 and not after.** `restart-case` and the transfer channel exist, so + this is a compiler-emitted restart at each allocating site and little else. It goes in at the same time because the + signatures depend on it: retrofitting it later adds a transfer check to every call site of every allocating + operation, which is the whole point of having decided it now. +4. **`(Map K V)`** — flat open-addressed key and value arrays, with a compiler-emitted hash and equality pair per key + type passed as arguments. `spec-memory.md`'s structural-key restriction holds this to the built-in key set, so there + is no dispatch to design. +5. **`drop`.** The hook, the transitive move-only and non-`clone`able rules, and the refusal to construct a + `drop`-carrying value against an allocator without `can-free`. It is additive — no type in the repo has a hook today + — but the `can-free` refusal has to land with the construction path it guards, before any arena-allocated container + of a user struct is trusted. +6. **`(Result T E)` and `try`, then the rest of union values.** Unions are what `Form` needs, and `Form` is what the + macro expander needs. +7. **Generics and monomorphisation**, then function values. User-written allocators and escaping closures both fall out + of the second. +8. **The macro expander**, last, on 6's unions. + +**What is genuinely unsettled, and none of it blocks step 1.** + +- `spec-memory.md`'s "Open: catching a use-after-release statically" is open by decision, not by omission. It says the + static rule needs to know which allocator a construction used and that `with-allocator` plus `context/allocator` are + exactly what deny that knowledge; the shipping answer is the dev-build epoch trap, which is specified and buildable. + It also says what would settle it — real Flan programs using arenas, to show whether the escapes that occur are + lexical — and that is evidence this repo cannot produce until after step 2. **Build against the epoch trap.** +- **The operation table may be one operation short.** It has `free-all` and nothing else. Zig's `ArenaAllocator.reset` + takes a `ResetMode` of `free_all`, `retain_capacity` or `retain_with_limit` + (`lib/std/heap/arena_allocator.zig:57`–`69`), and for a frame arena reset every frame, retain-capacity is the normal + case and free-all is the unusual one — handing the pages back to the backing allocator only to ask for them again. + Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing buffer and the call only + sets `offset = 0` (`core/mem/allocators.odin:287`). Flan should decide whether `free-all` means either of these or + takes the mode. It is an amendment to `spec-memory.md`, it is small, and it is better made before the arena is + written than after. +- Escaping closures are still deferred (`spec-memory.md`, "Function values", case 3), and a user-written allocator is + not one — its procedure is a top-level `defn` with no captured environment. The two should not be conflated when + function values arrive. + ### Bugs found and not yet fixed +- **Two citations in `spec-memory.md`'s Allocators section do not land where they say.** Checked against Odin + `819fdc7a8` and Carp `ea121b5a`, every other one is exact — `Map_Info` at `base/runtime/core.odin:369`, + `Allocator_Proc` at `:422`, the arena answering `.Free` with `.Mode_Not_Implemented` at + `core/mem/allocators.odin:307`–`308`, `#optional_allocator_error` on `append_elem` at + `base/runtime/core_builtin.odin:767`, and `// TODO(bill): Better error handling for failed reservation` at + `base/runtime/dynamic_array_internal.odin:107` and `:128`. The two that miss: `Map_Cell_Info` is at `core.odin:351`, + not `:350`; and `check.ml:1670` is the FFI `Declare` arm, not the `defer` registration — the claim it is offered for, + that a top-level `defer` is checked in a scope holding only parameters and globals, is true and lives in `check_fn` + at `check.ml:1800`–`1812`. `check.ml:505` is the `defer` refusal exactly as cited. Also worth a word when that + section is next edited: Carp's `getDropFunc` is at `Memory.hs:804` and the drop-before-delete emit at `Emit.hs:1044`, + both within a couple of lines of what is written, and `docs/Drop.md` confirms the sentence they support — "`A.drop` + will be run ... when the `let` scope ends". + - ~~`web/examples/breakdemo.out` is stale and `check.sh` fails on it.~~ **Fixed.** Commit `4a6a8fa` made the break banner number its restarts and the `.out` was never repinned. Nothing had to drive the socket in the end: `check.sh` already builds this one `--dev` and runs it under `timeout 5`, keeping what it printed before it