From c95f11ff18e411e21bb0371abafea4701adf9e7b Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sun, 13 Sep 2026 09:25:51 +0700 Subject: [PATCH] What the registry answers, and the two places a release build is not free MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BUILT.md on the split that is the design — the type is emitted because only the checker knows it, the death is not because an address needs no type — and on the part that reads backwards: (Ptr Enemy) already said Enemy, so the registry supplies permission rather than identification. The honesty is in the same section rather than a footnote. A release build pays a load and a not-taken branch per free, because flan_dev.c is in every build and a second allocator selected by a build flag is worse than a branch. The table is calloc'd when armed rather than declared, so nothing else is carried. The arena free-all is answerable now and still invisible to memcheck, and those are two different claims. NEXT.md keeps the entry open for what was not built: an op that points at an arbitrary address, the breakdown by type, leak attribution, the memcheck half, and the test_dev.ml case that would drive dev-ptr.flan. --- BUILT.md | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ NEXT.md | 56 +++++++++++++------------ 2 files changed, 154 insertions(+), 26 deletions(-) diff --git a/BUILT.md b/BUILT.md index 8b7cad6..8783fb2 100644 --- a/BUILT.md +++ b/BUILT.md @@ -4445,3 +4445,127 @@ everything else there is arithmetic over a container header. failure — the region a container lived in was released, and there is no frame to go back to that would not read freed memory — and the map path was left alone rather than converted half-way. Written down here so it is a known edge rather than a discovery. + + +## An address answers with a type, and nothing grew a tag word + +A Flan struct is exactly its C layout. No header, no tag word — deliberately, and it is what makes a struct free and +what makes the FFI work. The consequence is stated elsewhere in this file more than once: *a Flan value carries no +header, so nothing at run time can say what it is*, which is why a rendering is a compile-time walk over a type and +why `(watch "v" v)` cannot reach a composite without an arm in `check.ml`. + +The allocation registry does not answer that question. It sidesteps it. **The allocator's caller knows the type at the +moment it asks for memory**, and the compiler is standing right there, so a dev build writes it down: base address, +extent, element size, and the type's printed spelling. Nothing about any value's layout changes, and raylib never +finds out. + +### The split, which is the whole design + +Two halves, and they are in different files because they need different things. + +- **Recording the type is emitted.** `check.ml` builds one `flan_dev_reg_note_*` call after every operation that may + have allocated — `vec-new`, `push`, `reserve`, `clone`, `pool-new`, `insert`, `map-new`, `put`, `slurp`. Here is the + only place the concrete element type exists, so here is the only place that can name it. It is built in **every** + build, because a tree that differed by build flag would make every pass between the checker and the backend ask + which one it was looking at; `Emit`'s `Rt` arm drops the family when `dev` is off. +- **Recording that a block died is not emitted.** An address needs no type, so `flan_rt.c` calls into the table + directly from `flan_heap_proc`'s free, from a heap resize for the block it moved away from, from the arena's + `free-all`, and from `arena-destroy`. No signature in the allocator grew a type name and no ABI moved. + +The drop in `Emit` happens **before** the arguments are walked, not after. A note takes the address of the container +it describes; emitting that address and then discarding the call would leave an escaped `alloca` behind, and an +escaped `alloca` is one mem2reg will not promote. So a release build's IR is the same IR it always was. + +### It is armed by a constructor, and that is not fastidiousness + +`@llvm.global_ctors` in the dev module, not a line at the top of `main`. A `defvar` initialiser can allocate, and it +runs before `main` does; a note that arrived before the flag was set would be a block the table never heard of, which +is a live pointer the inspector would call dead. That is the one failure mode worse than no registry at all. + +### Lookup is containment, and that is not an optimisation + +An entry is a **block**, not a value. Every pointer a program can hold into heap storage is interior: `(at v i)` is +`v->ptr + i*size`, and `(resolve p h)` is an item in the middle of a pool's items array. Neither is ever a base +address. A table that answered only exact hits would answer nothing anyone can ask it. + +The two sides of the table therefore look different, and the difference is which thread is asking: + +- `flan_dev_reg_note` and `flan_dev_reg_dead` are **probes** — a hash slot and a linear walk from it. Both are on the + writer's side, and a free hands back the same base address the allocator gave out, so equality is the whole + question there. +- `flan_dev_reg_live` and the epitaph are a **scan** of all 4096 slots. The reader is a person pressing a key, so the + cost belongs there. +- `flan_dev_reg_dead_range` is a scan too, and it is the one that runs per frame: an arena's `free-all` has no list of + what it handed out, so the region is matched against the table rather than the other way round. That is a real + per-frame cost in a dev build and it is named here rather than discovered later. + +Dead entries are kept. That is the second thing the registry buys — an address that was freed still names what died — +and an entry is dropped only when the allocator hands the same address out again, which is exactly when the old answer +stopped being true. When the table fills it is compacted, dropping the dead and re-inserting the live; in a +long-running program the dead are the bulk of it. + +### What it is for: permission, not identification + +This is the part worth stating plainly, because the obvious reading is wrong. + +`(Ptr Enemy)` **already says Enemy**, at compile time, in `Render`'s walk. The type at the far end was never the +difficulty. What was missing is *permission*: whether it is still true to read the storage there. `render.ml`'s old +comment said a pointer is never followed because "dereferencing one a REPL was handed is not a safe thing to do on +someone's behalf", and that sentence is still correct — the registry just makes the safety checkable. So the arm +became a branch: + +``` +("live" "(Ptr Enemy)" "") +("dead" "(Ptr Enemy)" "") +``` + +The recorded type name is therefore not what selects the renderer. It is what the *epitaph* says, and a cross-check +available to anything that wants one. + +An address the registry never saw renders ``, unchanged. That is a stack local, a global, or a pointer from C, +and the shadow stack and the static type table already answer for the first two by name. + +**`println` does not follow a pointer, and will not.** `spec-memory.md` fixes what a printed `Ptr` prints, a printed +line belongs to the program and has to read the same in a release build, and a release build has no registry to ask. +The two callers of `Render` already differ in an emitter record; they now differ in a `pointers` record too, and +`check.ml` passes `None`. This is also why the epitaph carries **no address**: an address is not stable across two +runs, so printing one would make a rendering — and any test that reads one — depend on where the heap landed. It is +the rule `Render` already follows for an allocator. + +### The arena hole: answerable, not reported + +`test_valgrind.ml` measures a hole and this does not close it. `free-all` is retain-capacity: the offset goes to zero, +the pages stay mapped, and from `malloc`'s point of view nothing died — so memcheck is never told, and a later read of +stale bytes is a read of memory that is, as far as it knows, perfectly alive. + +The registry **is** told. What that buys is that the same read is *answerable*: a pointer into a released region comes +back dead and names what used to be there. Memcheck still says nothing, and closing that half is +`VALGRIND_MAKE_MEM_UNDEFINED` in `flan_arena_proc`, which `test_valgrind.ml` names as the fix and which is still not +written. **The two must not be blurred into one claim.** + +### What a release build actually carries, said honestly + +"Release builds carry none of it" is the goal and it is not quite true, in exactly two places: + +- **A load and a not-taken branch per free, and per `free-all`.** `flan_dev.c` is compiled into every build (see + `Build`, which says why), so `flan_rt.c` calls the dead-marking hooks unconditionally and each begins by testing a + flag only a dev build sets. The alternative is a second version of the allocator selected by a build flag, which is + worse than a branch for the reason the `Vec` header already carries its two dev words in every build: a layout or a + code path that changes with a flag is one that can disagree across the redefinition boundary silently. +- **Nothing else.** The table is `calloc`'d when it is armed, not declared as an array — a fixed 4096-entry table + would have been a quarter of a megabyte of BSS in a shipped game for something that build never writes. A release + binary carries a null pointer, a zero flag, and the declarations, which cost nothing. + +The `@llvm.global_ctors` entry, the notes, and everything that reads them are dev-only, which is the same arrangement +the indirection cells and the shadow stack have. + +### Tested twice, and one thing tested by hand + +`programs/registry.flan` is **one program read twice** in the acceptance table. A dev build answers for an address at +the heap tier, the arena tier and the pool tier; a release build answers 0 to every question. The difference between +the two expectations *is* the assertion, and writing it as one program means nobody can change what a dev build does +without the release row noticing. + +The inspector's pointer arm is **not** covered. `test/programs/dev-ptr.flan` is the program, its header carries the +two lines above, and they were read off a running session by hand — the `test_dev.ml` case that would drive it belongs +to another lane's file. `NEXT.md` says so rather than letting the verification read as automated. diff --git a/NEXT.md b/NEXT.md index 963e2b1..ddac519 100644 --- a/NEXT.md +++ b/NEXT.md @@ -20,42 +20,46 @@ This matters more here than in most Lisps because the intended use is a **game l skip a frame and carry on rather than die — exactly the case where a non-idempotent mutation bites. Write it into `conditions.org` and `spec-conditions.md`'s prose, and into `web/index.html` beside the restart documentation. -## Queued: a dev-build allocation registry — address to type +## ~~Queued: a dev-build allocation registry~~ — **landed, in part; three of the six remain** -Raised in conversation and wanted. **What it is:** in a dev build only, every allocation records what type it was -made for. Address → type becomes a lookup. +Built. The table is in `runtime/flan_dev.c`, the note is emitted by `check.ml` and dropped by `emit.ml` in a release +build, and the inspector reads it. `BUILT.md`'s *"An address answers with a type"* is the account of it; what follows +is only what is **not** there, so that the gap is a queue entry rather than a discovery. -**Why it is possible without tagging anything.** A Flan struct is exactly its C layout with no header and no tag word -— deliberately, and it is what makes structs free and the FFI work. So "what is at this address" is normally -unanswerable at run time, and a tag cannot be added without breaking raylib. The registry sidesteps that: the -*allocator* knows the type at the moment it hands out memory, so nothing about the value's layout has to change. -Costs nothing in a release build. +**Landed:** items 1 and 2 — the inspector follows a live `(Ptr T)` and renders the pointee, and names what died at a +dead one (``). The dead-marking covers the heap `free`, a heap resize's old +block, an arena `free-all` and `arena-destroy`. -**What it buys, in rough order of value:** +**Left, and in this order:** -- **Following a pointer.** `(Ptr T)` renders as `` today and stops there. With a registry the inspector follows - it and renders what is actually at the other end. -- **Use-after-free that names what died.** The address is in the registry and marked dead: *"this was an Enemy, freed - at frame 412"* instead of garbage, a crash, or silence. -- Point at any heap address and get a typed rendering rather than bytes. -- **A memory breakdown by type** — how many bytes are Enemies, how many are strings. A real profiling tool for a game. -- **Leak attribution at exit**, which generalises the debug tracking allocator already queued for counting forgotten - textures. -- It closes the **arena hole Valgrind found**: releasing a region could mark everything in it dead, where today the - bytes stay quietly readable because `free-all` is retain-capacity and memcheck is never told. +- **Item 3, point at any heap address.** The lookup is there and answers for any address; nothing exposes it as an + editor op. It wants a verb beside `inspect` that takes an address and a type rather than a frame and a slot, and the + registry's own answer for the type when none is given — which needs the recorded name resolved back to a + `Types.t`, and the table records a string. +- **Item 4, a breakdown by type**, and **item 5, leak attribution at exit.** Both are a walk over the table and a + group-by; `flan_dev_reg_count` is the whole of what exists. Neither is hard and neither has a reader yet, which is + why they were left rather than half-built. +- **The memcheck half of item 6.** The registry now knows an arena's `free-all` killed everything in the region, so a + later read through a pointer into it is *answerable*. Memcheck still says nothing, because nothing told it: the + pages stay mapped and `free-all` is an integer going to zero inside one allocation. Closing that is + `VALGRIND_MAKE_MEM_UNDEFINED` in `flan_arena_proc`, which `test/test_valgrind.ml` already names. **The two must not + be blurred** — the registry answer and the memcheck answer are different tools reaching different people. +- **A test that drives the inspector's pointer arm.** `test/programs/dev-ptr.flan` is the program and its header has + the two lines a session answers with; they were read off a running session **by hand**. The case belongs beside the + other `locals`/`inspect` cases in `test_dev.ml`, which was another lane's file. `programs/registry.flan` covers the + table itself from the acceptance table, in a dev build and a release one. **What it does not cover, and does not need to:** stack locals and globals, which the shadow stack and the static type -table already answer by name. - -**On cost — settled by the author: keep it simple.** One registry insert per allocation, always on in a dev build, -no opt-out. An arena allocation is a bump pointer and an insert may well cost more than the allocation itself in a -per-frame loop, but a dev build already carries indirection cells and a shadow stack, and the author's instruction is -to build the straightforward thing and revisit only if a real program shows a problem. Do **not** build per-region -recording, range recording, or a per-allocator opt-out on speculation. +table already answer by name. A stack address is deliberately not in the table, and a pointer to one still renders +``. **Note on classes:** `defclass` instances will carry shape metadata by design, so they get identification for free and do not need the registry. This is for plain structs, `Vec`, `Map` and pool storage. +**On cost, as built.** One insert per allocation, always on in a dev build, no opt-out — the author's instruction, +followed literally. Nothing was built per-region, no range recording, no per-allocator opt-out. Revisit only if a real +program shows a problem, and `BUILT.md` names the two places a release build is not quite free. + ## Picked up first, 2026-09-13 Three things, in order. The first two are one line each and unblock a real game.