diff --git a/BUILT.md b/BUILT.md new file mode 100644 index 0000000..0953989 --- /dev/null +++ b/BUILT.md @@ -0,0 +1,1183 @@ +# How the parts that exist work + +The reasons behind the code, kept separate from `NEXT.md` so that what to do next is not buried under what was already +done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever `dlclose`d, why a call +bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather +than a function in the runtime. Deleting it would mean deriving it again. + +`NEXT.md` is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that +first. Come here when you need to know why something is the shape it is. + +## What milestone 4 added + +**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a +hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not +assignable, which makes the generated step its only writer. + +**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top +level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it +stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer +written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the +bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision. + +`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at +function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. +Block-scoped defer is real work and is not done. + +**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot, +so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed +type and logical on an unsigned one), and `rand-f32`. + +**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is +only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets +the state. This is what the bitwise operators were added for. + +**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type +in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather +than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on. + +## Why the FFI goes through a C shim + +The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64: + +``` +Vector2 {float,float} → declare <2 x float> @GetMousePosition() +Color {u8,u8,u8,u8} → declare void @ClearBackground(i32) +Rectangle {4 × int} → declare { i64, i64 } @mkrect() +``` + +None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a +per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that +in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning +garbage rather than as a link error. + +So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an +out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a +copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare` +signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host +ABI, implemented twice", and `flan_rt.c` is the same pattern. + +### The wrappers are generated now — `declare-c`, `lib/shim.ml` + +The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to +generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory +holds `raylib.flan` and `link` and no C at all. + +One binding is now one line: + +``` +(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture") +``` + +`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like +any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the +function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an +ordinary Flan `defn` above it when the signature has a struct in it. `flan shim ` prints the whole file. + +**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path +string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h +i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural +rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited. + +**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same +`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what +makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted: +that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real +signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was +considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard: +for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct +— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves. +Everything where they could diverge is refused at the field. + +**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a +scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the +hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against +the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree. + +**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path, +512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a +string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on +malloc failure, where the alternative is handing C a null pointer. + +**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice +and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is +refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not +recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes +`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned +string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with +the reason, and the acceptance table asserts on the reasons. + +**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc + +`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function +that calls an existing binding is unaffected. + +`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is +`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17 +34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity +would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed. + +The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write. + +Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows. + +The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance +case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a +GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins +`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all — +storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`, +`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four +ways for that reason. + +No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared +library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora +ships the runtime library without the `.so` symlink. + +### What a headless FFI test can and cannot pin + +Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing. + +- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C +writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and +getting identical output. +- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the +way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, +passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from. +Distances are worse: the reflection does not even reach them. +- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does +not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision +cases look like they cover it. +- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different +field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged. + +- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available. +`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values +in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely, +including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU +reads its width, height or mipmaps at all. +- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image +`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and +answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more. +- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the +package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the +`x`/`y` permutation above. + +The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail +by permuting the fields and watching it go red. A case not verified that way is decoration. + +One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It +measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no +window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime` +and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are +exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them. + +## Packages + +`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is +a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no +collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares +becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is +rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of +declarations whose names contain a slash. + +A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file +named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below. + +**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also +a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import +all four. A file carries no `.c` and no `link` file; those belong to a directory. + +**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that +is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under +one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which +is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time, +and the namespace being flat, mutually dependent packages simply work. The same directory under two *different* aliases +is refused. + +**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment +anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would +keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the +checker it would be "unknown name", which is true and useless. + +Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable. + +## The link follows the program + +`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the +importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that +link cannot succeed. + +`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it, +follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing +else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument. + +Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and +`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs** +from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a +dropped one is a silently different program. + +**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far. + +The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and +`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker +arguments together, so a caller cannot take one without the other. + +## sand.flan is one program + +It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a +program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both +drivers imported it. + +Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports +`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches +raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both +targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation. + +What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no +input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and +`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still +168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost. + +Two claims that got run together in an earlier note, for the record: + +- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the +**wasi** path, which is what the headless table targets, and which has no GL and no browser. +- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with +`emscripten_set_main_loop` instead of a `while`. That is a different `main`, not a different program. + +**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes: + +- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`, +and there is no implicit widening. +- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`. +- `(defn main [])` is unchanged — the short form, as plan.org says. + +Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what +cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press. + +## Bounds checks — done at milestone 3 + +`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three +check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds +never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo +<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. +`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the +same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`. + +The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor +that split them would break the write check silently. The test covers both. + +Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s +unchecked. Indistinguishable. + +## Why there is no interpreter + +Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had +expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, +perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is +hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second" +criterion is dropped, and the host ABI moved onto the critical path. + +## The layout, which is the whole backend design + +``` +i8..i64 / u8..u64 i8..i64 signedness lives in the ops +f32 f64 float double +bool i1 +an enum i32 +[T] and string { ptr, i64 } ptr+len, non-owning +[n T] [n x T] inline, a value +(Ptr T) ptr opaque pointers +(Option T) { i8, T } tag 0 None, 1 Some +a struct a literal struct, declaration order +Unit and Never {} +``` + +No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the +semantics: + +- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is* +the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody +addressed. `test/programs/values.flan` pins this down. +- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a +`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong. + +Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding, +so wasm32 needs no exception proposal. + +## The reload primitive — dev loop steps 1 and 2, measured + +`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled +into its own object and called inside a process that is already running, twice, with a changed body the second time. + +| Step | Cost | +|---|---| +| `Emit.redefinition` | below the timer (<0.1ms) | +| `llc -O2 -filetype=obj` | 15–17ms | +| `ld -shared` | 3ms | +| `dlopen` + `dlsym` | **0.04ms** | + +**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the +number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost, +which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since +the driver absorbs IR the bare tools reject. + +`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that +*is* the mechanism. What the new module does **not** define is the whole design: + +- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a +redefinition can change a function's body and can never re-initialise the program's data. Define the global and the +loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the +thesis. +- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a +private copy of it. +- **no `main`.** This module is loaded, not started. + +Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss +because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is +spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a +slice parameter. + +**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the +compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the +agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release +builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_get`, which reads as a compiler +bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and +the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason. + +**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a +loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property +of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol. + +Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**, +since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since +`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And +`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares +`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the +host's copy. With the two bodies identical nothing at run time would notice a module that grew its own. + +String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's — +worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would +fail at link time. The fixtures each print a literal so that path is actually exercised. + +### Cells — how a call site follows a redefinition + +Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes +every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current. + +``` +@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it +%p = load ptr, ptr @"flan.cell.bump" ; every call site +%r = call i64 %p() +``` + +Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and +exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what +makes a frame-boundary swap a non-event. + +The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of +one. + +Four things about this that are not free choices: + +- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on +whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are +separate on purpose. +- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to +*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would +publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, +because the failure is silent. +- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined +function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture +recurses on purpose, and would print the old body's text if it did not. +- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is +the first time `opts` means something semantic rather than an optimisation level. + +LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46 +indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well; +the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function. + +### Names that did not exist when the process started + +Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF +cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else: + +``` +void **flan_dev_cell(const char *name); /* a new function's cell */ +void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */ +``` + +Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A +new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores +on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not +reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the +rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a +call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the +common case pays nothing for the general one. + +**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c +C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a +redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet. + +Four rules, each of which is a silent failure if broken: + +- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are +still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`. +- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the +old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. +This is the layout-drift rule's first enforcement point. Retyping a var needs a restart. +- **Nothing is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it leaves every call site +pointing at unmapped memory. That is a constraint on the agent too. +- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed +capacity with a loud failure rather than growable. + +The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at +run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell* +both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still +pass and the transcript would read 246 instead of 432. + +Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout +calculator in OCaml that would have to agree with LLVM's on every target. + +### The agent — dev loop step 3 + +`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link` +asks for `-lpthread`. + +``` +(agent/start path) listen on a unix socket; once, at startup +(agent/poll) install whatever has arrived; returns how many +(agent/wait ms) the same, but waits for something first +``` + +The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded — +so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a +redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks. +The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the +loader. + +`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits +for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first +introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if +it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that. + +Two details found by running it: + +- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a +program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at +all, which is how this was found. +- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program +can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`. +- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows +when it is between frames. + +**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under +Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body +that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is +refused on the listener thread and the loop never notices. + +`flan reload ... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the +names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what +the session will track automatically. + +### The session + +`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change +accepted since. + +**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a +declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not +replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that +follows. There is a test for the case that actually matters: a typo, then a good form, in the same session. + +Two things the session knows that no single evaluation could: + +- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST, +because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `print-line` reads +as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. +- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded: + +| Change | What it would have broken | +|---|---| +| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below | +| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve | +| a struct's fields | the values the process is holding have the old layout | +| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it | +| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too | + +**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should +make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while +existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked +caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error. +Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines +(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because +the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message +should not be read as the final answer. + +A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past +long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global` +record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than +the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is +the same fact that makes them unreloadable. + +Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first, +loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program +uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for +that reason. + +A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A +dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a +module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That +is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the +folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could. + +**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in +`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s +own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a +brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it +already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently +did nothing. + +It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a +Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written +nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to +either. + +The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise +re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name +pass rejects it. `C-c C-k` on sand.flan's own text is the test. + +`flan reload ` is that path from the command line: a session over the program the process was +built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw` +and 910 consecutive frames drew it. + +Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on +every invocation, so if the program file has been edited since the process launched, its idea of which names the host +has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to +``, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass +the real buffer path, which is the same key CIDER's `eval` carries. + +### The daemon — `flan dev` + +`flan dev ` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside +the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part +of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe +the process that is actually running rather than a guess about it. + +**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client +became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form, +from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the +editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of +Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains +newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor. + +``` +(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t) +(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0) + → (:status "error" :message "…" :loc "/buf.flan:1:19") +(:op "defs") → (:status "ok" :defs ((name kind signature loc) …)) +(:op "close") +``` + +`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the +program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer. +It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing +else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none +to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself, +which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step +[i64 f32] i64`: types only. + +The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan +dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by +guessing which directory it was relative to. + +An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage +— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one +anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need. + +`:file` is not decoration: `Session.eval`'s origin defaults to ``, so without it every error an editor shows +points into a file that does not exist. + +Two things the daemon must not paper over, both of which would look like a successful evaluation: + +- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan +says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead +fails silently: the module compiles, is built, and nobody receives it. +- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed +connect or a reply that is not `ok` becomes an error the editor sees. + +It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads +like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather +than leaving an editor waiting on a socket nobody is serving. + +### The Emacs client + +`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is +s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is +Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is +written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords. + +`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a +request and `read` reads a reply. + +| | | +|---|---| +| `C-c C-c` | the top-level form at point, recompiled and installed | +| `C-c C-k` | the whole buffer, as **one** module | +| `C-x C-e` | the expression before point, evaluated *in the running program* | +| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect | +| `C-c C-o` | the running program's own output, in `*flan-output*` | +| `C-c C-r` | a prompt on the running program (`*flan-repl*`) | +| `C-c C-b` | what a **stopped** program is offering, and which to take | +| `C-c C-d` | what the running program currently defines | +| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR | +| `M-.` / `M-,` | where a name is written, through an `xref` backend | + +eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on +an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at +the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just +installed completes at once. + +The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away, +which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the +socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the +daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice. + +`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to +arrive in the same load, or the first refers to storage that does not exist yet. + +**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary, +or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload +has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives +the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a +mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml` +and fails here. + +An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the +next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte +offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and +`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the +daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1 +and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the +reader skips, so the reply's line numbers are the buffer's own. + +An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent +success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the +one point looked like it was in. + +**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to +into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form +"asm"|"ir")` is the op. + +What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d +--disassemble=flan.` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared` +deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was, +so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else +on the machine still has that text. A table from function name to the last module accepted for it is the whole of the +bookkeeping. + +**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket +takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address +space, and `C-x C-e` renders a pointer as `` on purpose — so nothing the daemon can ask would tell it what a cell +holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first +instruction: + +- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*; +- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed; +- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed, +since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further +installs until it resumes. + +From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch +targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this +build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When +the debug build lands, that is the line to delete. + +**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an +annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything. +`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the +overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added +where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke +in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the +same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it +is Emacs' contract and a test claiming to check it would be checking nothing. + +**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the +next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the +output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and +not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the +accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot +grow the daemon without limit. + +### `C-x C-e` — evaluating an expression + +A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body +into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by +exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an +expression reading the program's state sees a point the program agrees is consistent. + +**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say +what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout +decision's bill, and it is why the printer set is small rather than universal. + +It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a +dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's +socket. The read is safe without a handshake because `flan_dev_result` bumps a generation counter last; the daemon waits +for it to move rather than assuming the program has reached a frame boundary. + +**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a +compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints +structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth +and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three +bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a +builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time +type to walk. Not the walk itself. + +The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise +because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an +allocator the language does not have. + +``` +big 18446744073709551615 +col :blue +(.pos b) (V {:x 1.5 :y 0}) +b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]}) +(slice (.tags b) 0 3) [ 0 42 0] +(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68}) +sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...] +``` + +Details that are decisions rather than formatting: + +- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come +back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point +instead. +- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather +than as the value it is. +- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are +erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is +exactly what you would want to see. +- **A pointer is never followed** — ``. It is the only thing that could make the walk cycle, and dereferencing one +a REPL was handed is not a safe thing to do on someone's behalf. +- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does +not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K +and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it +means no renderer carries a budget. +- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is +not evaluated once per element. + +What still refuses by name: `Map`, `Fn`, a type variable. + +A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer. +`flan run calc-me.flan "1.5 * 2.0"` has always said `3`. + +An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe` +does not fill up with `eval/N` for every expression ever typed. + +**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by +`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points +into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the +agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition* +retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind, +and can never claim this. + +Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop +would exhaust it. + +The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must +disagree. A value computed in the compiler, or read out of a copy of the program's state, would not. + +### The REPL buffer + +`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No +new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as +deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess +behind it — the "process" is a stub comint needs in order to have a prompt at all. + +Three things about it that are decisions: + +- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in +sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it +because the file says which package it belongs to; a prompt has no file and nothing to derive one from. +- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a +paren inside a string does not count. +- **A value and the program's output are different things and arrive by different routes.** The value is the result of +the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply +and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the +separation. + +That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so +`(print-line "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A Unit +expression is almost always a call made for its effect, and is now evaluated and *then* reported. + +### Conditions — step 1: `handler-bind` and `signal` + +`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters +control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling +function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists +yet, and no signature changed. + +``` +(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))] + (load-all)) +``` + +The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing +function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame +rather than by count, so restoring what this one displaced is right even if something below it left the stack out of +step. + +Three decisions worth keeping: + +- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and +every handler a running program had already pushed would then match the wrong type. FNV-1a over the name. +- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is +nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the +name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address +instead. +- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a +branch in the function that wrote it. +- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from +plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a +particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the +duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still +valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the +stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When +`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them. + +Which gives the two refusals, both by the house rule rather than by accident: + +- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a +reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in +scope, which is what the accumulation case needs. + +What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not +outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by +value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete +use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the +biggest usability limit in conditions is not behind the thing that was just deferred. +- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would +leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block. + +### The break loop — conditions step 3 + +Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the +frame that erred with nothing unwound, so the condition and every restart between there and the top are still live. + +``` +flan: unhandled Missing — stopped, not dead. + restart: retry + restart: use-placeholder +``` + +Four decisions, each of which is the reason something is where it is: + +- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the +release runtime and must not depend on something a program may never import. A program with no agent leaves the hook +null and dies exactly as it did before. +- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes, +reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same +act, lowered once. Nothing about §6 needed changing to support it. +- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a +convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop +that did not drain that queue would hang `C-x C-e` exactly when it is most wanted. +- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must +not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new — +and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the +cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of +stopping. + +**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but +a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also +`compute-restarts`' data, whenever that arrives. + +**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused +there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that +cannot happen. + +The socket verbs are `restarts`, `restart-at [name]`, `restart ` and `abort`, and all four are refused with +the reason when the program is not stopped — there is no restart stack to walk from a running one. `restarts` answers +` <+|-> ` per line: the index is the identity, and the flag says whether a transfer to that frame has +anywhere to land. `test/programs/break.flan` errors three times — two restarts taken by name, then a shadowed pair +where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved +by name could pass. + +### The break loop in the editor — conditions step 3, the other half + +The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it +follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a +question an editor chose to ask. + +So the state is learned **twice, deliberately**: + +- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`, +or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now +errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding +out *after* the echo area had said the evaluation was fine. +- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is +drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies +that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left +behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the +`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs +timers, so a poll firing inside a read would eat the reply that read was waiting for. + +Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is +stopped, so the poll and the prompt cannot disagree. + +``` +(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3) + :stopped t :condition "Missing") +(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …") +(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …") +(:op "abort") → (:status "ok" :note "the program is exiting; …") +``` + +`:restarts` is *positional* — innermost first, duplicates kept — because the position is what `restart-at` takes. +`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is +not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a +prompt cannot take a different restart than the one it showed. + +`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by +someone about to choose from it. + +**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against +the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would +poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and +lets the next poll settle it. + +The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running +one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly +right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key — +it is the thing you pick when none of the restarts is the answer. + +**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A +`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop +cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the +nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of +breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices +each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an +expression that errors and resumes it, which fails against the old shape. + +The agent grew one verb, `status`, answered in **both** states — `running` or `stopped `. Everything else the +break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had +to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the +hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know. + +`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the +state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way +round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e` +answering while the program sits in the break loop is checked there against the real client, not only in OCaml. + +### Conditions — step 2: `restart-case` and `invoke-restart` + +`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a +`restart-case` further out. + +``` +(defn fetch [n i32] i32 + (restart-case (middle n) ; its value if nothing transfers + (use-placeholder [] -1) + (retry [] 7))) + +(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))] + (fetch 2)) ; -1 +``` + +**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an +`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the +release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's +slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers. +`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather +than a hunt. + +**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare +pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and +plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight +through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform +also means redefinition acquires no new refusal class. + +**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous +note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only +probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The +frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is +how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was. +Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's +"innermost offering the name" is just the order of the walk. + +**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever +frames it established and either catches the transfer or forwards it outward: + +- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts +the target back and goes on out; +- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, +and without it the handler stack would be left pointing into a frame that has gone; +- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is +still refused by name. + +A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would +jump straight past the very form that was supposed to catch it. + +**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is +guarded; with the channel still set the first of them would branch straight back into the landing block it came from. +Same reason the clause body starts with it null. + +`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes +the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot +cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and +there are no function values yet, so nothing can call *back* into Flan across one. + +Scope, each piece refused by name with its reason and a test on the reason: + +- **restarts take no parameters.** That covers §1's own `load-texture` example and skips argument marshalling and §3's +runtime arity check. +- **TODO — CL-style interactive recovery.** A stopped program should be able to offer a typed restart such as +`(use-value [value T] value)` or `(use-function [replacement (Fn ...)] ...)`, and the editor should show its signature +and ask for the replacement before invoking it. That is the missing "this variable is None; what should I use instead?" +path: named, hard-coded branches are useful for `retry` and `skip`, but not a substitute for an interactive value or +alternate implementation. It needs restart argument marshalling and validation in the runtime, plus an editor protocol +for entering and checking the supplied value/expression. +- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops. +- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a +choice nothing in the source shows. +- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting +there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a +defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see +it. +- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past +everything. There is nowhere to resume, so there is nothing else to do. + +**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal` +returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past +it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked +`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case` +body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled +case: it cannot be an `outputs` row, because it does not exit 0. + +`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart` +through a call traps either way, and the message names the rule rather than the path, since the rule is the same. + +**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a +redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a +hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable +nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value. +A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot. + +`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca` +in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and +crosses `probe`, which the host was compiled with. Those three do not meet anywhere else. + +Two things found by writing it: + +- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates +the body's slots into a record the function never sees again and the indices collide. It was harmless only because no +`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now. +- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not +fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere +else entirely. + +`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant +here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each +of those calls goes through a cell. + +### What is left +- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays. + +**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the +memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. +Attaching to a process someone else built is not a thing to support by default. + +## Where build time goes + +`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms: + +| Step | Cost | +|---|---| +| frontend: read → parse → load → check → emit | <10ms, below the timer | +| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** | +| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** | +| link | 20ms | + +Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which +compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler +(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build), +`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0` +and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d +into place, so two concurrent builds cannot see a half-written one. + +Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never +the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever +a disk-space decision. + +The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It +is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted +IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse +errors. + + +## The order it was built in, and why that order + +Every item here is done; it is kept because the *sequencing* is the part worth remembering. Decided in conversation: +wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and **the dev loop is the +thesis of the project**, so it came first. Staged so each step was runnable on its own — the failure mode being to +build a daemon and a protocol before knowing whether the reload primitive worked. + +1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms. +See the section above. +2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`, +`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a +run-time-added function is itself redefined. See the section above. +3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs, +and sand.flan polling at the top of its frame. See the section above. +4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than +nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if +something else ever needs to talk to it. + +**One decision left to settle before step 2**, because both change codegen and are painful to retrofit: + +- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module +declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep +the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload, +and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet. +- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what +`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from. + +Deferred until after the dev loop: + +6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and +`test/programs/sand-headless.flan` prints `-2851001042534928384` under it — the same hash as native, byte for byte, at +`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared +across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer +would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and +runs it — rather than by looking for a binary on PATH. + +Three things this cost that were not in the old note: + +- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main` +to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a +signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")` +label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite +self-call. +- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes ``; without `--sysroot` +it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the +triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`. +- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both +shapes count. + +**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to +come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it +`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple +(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk +is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's +beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is +root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named +by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in. + +**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of +`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes +an `ExperimentalWarning` to stderr on every run and the harness compares combined output. + +**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with +one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime +at it). + +Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. wasi-sdk is right for the +headless table; it is not necessarily right for the eventual game build. diff --git a/NEXT.md b/NEXT.md index efa8d73..3c53ec8 100644 --- a/NEXT.md +++ b/NEXT.md @@ -2,111 +2,61 @@ ## Start here — next session -**Branch `dev-loop`, 154 commits, working tree clean, `dune test` green.** +**Branch `dev-loop`, 199 commits, working tree clean, `dune test` green.** + +**`NEXT.md` is what is left. [`BUILT.md`](BUILT.md) is why the existing parts are the shape they are** — the reload +primitive, cells, the agent, the session, the daemon, the Emacs client, conditions, the FFI shim, the layout, and the +order it all got built in. This file was half build log until it was split; do not let it become one again. When a +track here finishes, its explanation moves there and its entry here goes away. The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the -running process. Conditions are two steps in of four. +running process. -`(error c)` is in — §2's diverging variant, same lookup, type `Never`. A handler that returns normally has not answered -it, so only a transfer gets past; with nothing transferring the program stops and names the condition. `flan_error` is -where the break loop goes. +**Conditions are three steps of four.** `(error c)` is the diverging variant — a handler that returns normally has not +answered it, so only a transfer gets past. The break loop is in, editor half included: an unhandled `error` stops the +program on the frame that erred, the daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` lists the +restarts and resumes into the choice. A restart is chosen **by position**, off a snapshot taken when the break was +entered, because a name resolves to the innermost frame offering it and the stopped thread's stack does not hold still. +Restarts below the evaluation a break is inside are listed, marked, and refused with the reason. -**The break loop is in, editor half included.** An unhandled `error` stops the program on the frame that erred, the -daemon annotates every reply with `:stopped`/`:condition`, and `C-c C-b` in Emacs lists the restarts and resumes into -the choice. See "The break loop" and "The break loop in the editor" below — both describe what is built. - -**A restart is chosen by position now, not by name**, which is the fix for the shadowing bug and the reason `:restarts` -is a positional list with a `:unreachable` set beside it. Restarts below the evaluation a break is inside are listed, -marked, and refused with the reason; the list itself is a snapshot taken when the break was entered, because the -stopped thread's stack does not hold still. `spec-conditions.md` §4 now says all of this. - -**The old next task, now done, was the dev-build break loop, `spec-conditions.md` §2** — where an unhandled `error` -stops and talks to the daemon instead of `rt_die()`, and where **"a crash kills the program"** finally gets fixed. The -transfer it needs exists now: §6's channel is in every signature and `restart-case` catches on it, so a break loop is a -place to *stand* while the program is stopped, not a new way to move. - -Then restarts offered in the Emacs minibuffer, which wants `compute-restarts` plus two protocol ops. SBCL's restart -struct carries `report-function` and `interactive-function` for exactly that prompt and `spec-conditions.md` mentions -neither — worth adding before that step. `find-restart` and `compute-restarts` are both named in §4 and neither exists -yet; the runtime already has the stack they would walk. - -Still open from §3, and each refused by name today: **restarts with parameters** (argument marshalling plus the runtime -arity check), and `handler-case`, which §"What this does not settle" leaves open as possibly a macro over `handler-bind` -plus a transfer. `error`, `find-restart` and `compute-restarts` now refuse by name too — they used to fall through to a -call and come back as *unknown name*, which is the house rule's own class of bug. +Still open from §3, each refused by name today: **restarts with parameters** (argument marshalling plus a runtime arity +check), and **`handler-case`**, which §"What this does not settle" leaves open as possibly a macro over `handler-bind` +plus a transfer. **`find-restart` and `compute-restarts` are blocked on a type, not on effort** — §4 gives them +`(Option Restart)` and a list, and there is no `Restart` type and no list to return one in. The minibuffer prompt never +needed them; it reads the snapshot over the agent's socket. And a `restart-case` clause should carry a **report +string** before any of this: `use-placeholder` is what `invoke-restart` needs, not what a person reading a list needs. Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`, which §6 rules out. -### In flight — four tracks, one session, 2026-09-12 +### Landed 2026-09-12 — five tracks, one session -Four agents, parallel worktrees. Listed so a session reading this cold does not start one of them again. 1 and 2 are -merged; 3 and 4 are still running. +Five agents in parallel worktrees. Kept short on purpose; the reasoning that outlives the change is in `BUILT.md` or in +the commit that made it. -1. **Done.** `nth` removed, and `println` built. `nth` was an *asymmetric* alias of `at`: `check.ml` aliased them, but - `parse.ml` and `place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while - the `at` forms worked. `println` lifted `Session.render` into `lib/render.ml`, parameterised on an emitter and a - slot allocator, so the REPL and stdout share one walk. Found on the way: `field_addr` in `emit.ml` accepted only - `Types.Named`, so a field of an `Option` threw at emit time and **the walk's Option arm had never run** — the - inspector would have failed on the first `(Option T)` pointed at it. -2. **Done.** `restart-at`, by position. The editor half already existed — this session's brief was written off a - stale "Start here" line and is corrected below. The real work was taking a restart the innermost frame shadows: - `restart ` resolves to the inner one, so a visible outer restart could not be taken at all. An immutable - snapshot per break makes an index mean something, `:name` rides along as a receipt checked against it, and a - generation stamp keeps a nested break from claiming the outer one's choice. -3. **The first ten raylib core examples**, plus the window and input bindings they need. The deliverable is the - *language gap list* as much as the ported files — sand.flan is the only real raylib program today, so this is the - first time the language is pushed by code it was not designed around. -4. **Names in DWARF.** *Done.* `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under - lldb instead of `s0`; a slot the compiler invented keeps `s`, because a synthesized slot has no source name - and inventing one puts a variable in the debugger that is not in the file. Shadowing had to be decided rather than - assumed: every `!DILocalVariable` is scoped to the subprogram — the typed IR has no block structure to build a - `!DILexicalBlock` from — so two slots called `v` left lldb answering `p v` with the outer one while the body computed - with the inner, and not listing the inner at all. A repeat now gets a `~2` suffix, which is unambiguous because `~` - cannot occur in a source symbol. That is a way of not lying, not a way of being right; the real fix is a lexical - block per `Let` and the `llvm.dbg.declare`s moved out of the entry block, and it is the one thing left here. +1. **`nth` removed**, an alias of `at` that was asymmetric — `check.ml` aliased them but `parse.ml` and + `place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while the `at` forms + worked. +2. **`println` and `print`**, compiler-provided and structural. `Session.render` was already the compile-time walk + plan.org asks for; it moved to `lib/render.ml` parameterised on an emitter and a slot allocator, so the REPL and + stdout share one copy. Found doing it: `field_addr` in `emit.ml` accepted only `Types.Named`, so a field of an + `Option` threw at emit time and **the walk's Option arm had never run** — the inspector would have failed on the + first `(Option T)` pointed at it. prelude.ml's claim that this had to wait for milestone 5 and generics was wrong, + and is gone: a printer selected per *concrete* type has nothing to dispatch on and no type variable in it. +3. **`restart-at`** — a restart is taken by position now. See "Start here". +4. **Names in DWARF.** `Tast.fn` carries `snames` beside `slots`, so a let-bound local is its own name under lldb + instead of `s0`; a slot the compiler invented keeps `s`, because inventing a name puts a variable in the + debugger that is not in the file. Shadowing had to be decided rather than assumed: every `!DILocalVariable` is + scoped to the subprogram — the typed IR has no block structure to build a `!DILexicalBlock` from — so two slots + called `v` left lldb answering `p v` with the outer one while the body computed with the inner, and not listing the + inner at all. A repeat gets a `~2` suffix, unspellable in source. That is a way of not lying rather than a way of + being right; see "One line away". Also `flan dev --debug`, one flag for host and every redefinition module, off by + default because a debug build is an `-O0` build. +5. **Ten raylib core examples** in `examples/`, plus seven bindings and the colour palette. The gap list they produced + is under "Unblocked now, and ranked"; the top item, that no number could reach `draw-text`, is fixed — `(string b)` + reinterprets a `[u8]` as a `string`, which costs no instructions because they are already the same 16 bytes. - And `flan dev --debug` now builds the host *and* every redefinition module with DWARF. One flag, because it is one - decision — measured, not reasoned: a line breakpoint needs a line table on the host to fire before the first - `C-c C-c` and one in each module to still be firing after. Off by default, because a debug build is an `-O0` build - and silently making every reloaded body `-O0` changes the frame time of the function being iterated on. - -**What a dlopen'd redefinition module does to a breakpoint, measured.** lldb picks the new module's DWARF up on the -`dlopen` and says so — "1 location added to breakpoint 3". A breakpoint set by *name* gains a second location either -way, so dlopen was never the difficulty; what the module's line table buys is that it stops with **source** rather than -disassembly, and that a *file and line* breakpoint on the new body resolves at all — it sits at `locations = 0 -(pending)` forever without one. A file-and-line breakpoint on the **host's** copy stays pinned at `locations = 1`, which -is correct rather than stale: the old body is still mapped and every call site that has not gone through its cell again -still reaches it. The stack crosses the boundary intact — a frame in the reloaded `.so` and the frame below it in the -host each name their own `.flan` file. The transcripts are in `emacs/flan-dape.el`, under "Reloading and breakpoints". - -**Source interleaving in the disassembly buffer is unblocked, and not done.** `Dev.asm_of` runs `objdump -d`; with a -`--debug` module `objdump -dS` interleaves the Flan source correctly (verified). What it needs is the `-S` and a -`parse_listing` that tolerates source lines among the instructions. - -**`flan-cnr.el`'s stack pane was refusing for the wrong reason** and now names the real one. DWARF was never the gap: -what is missing is anything *attached* to the stopped program. That buffer reaches it over the daemon's socket, and a -socket cannot read another process's frames — the break loop stopped itself, it is not being debugged. It wants either -an unwinder in the agent or lldb on the same pid. - -### Queued — split this file - -About half of it is a build log. Of 1,682 lines, roughly 850 narrate work that is finished: lines 727-1507, from "The -reload primitive" through the two break-loop sections, plus "What milestone 4 added" and "Where build time goes". - -Do not simply delete them. That narrative is where the load-bearing reasons live — why nothing is ever `dlclose`d, why -a call bound at link time cannot be made to notice a redefinition, why the renderer is a compile-time walk over a type -rather than a function in the runtime. Losing those means deriving them again. - -- **`NEXT.md` keeps** what is forward-looking: Start here, In flight, the Queued sections, Managed classes, Open, - Blocked and unfinished with its six sublists, Sharp edges, Macros, Watch for, Untracked on purpose. -- **`BUILT.md` takes** how the parts that exist actually work: the reload primitive, cells, the agent, the session, - the daemon, the Emacs client, `C-x C-e`, the REPL buffer, conditions and the break loop — plus "The layout, which is - the whole backend design" and "Why the FFI goes through a C shim", which are rationale rather than plans. -- **Delete outright:** "Next — the REPL is the priority". Every item in it is struck through and marked Done, and it - says what "Start here" already says. - -`overview.md` is not the home for any of it; it is 54 lines and marks itself superseded. +**In flight now:** retiring the `print-*` family, below. ### Queued — the runtime under a sanitizer @@ -160,9 +110,11 @@ About 470 call sites across ~60 files. Two places want reading rather than `sed` - `web/index.html` (55 sites) and `web/examples/` carry pinned output, so the prose changes with them. That is also the moment to document `println` at all: it currently has **no mention on the language's own doc page**. -**DWARF is not missing, and `flan-cnr.el`'s stack pane reads as though it were.** `flan build --debug` emits it and -`flan-dape.el` drives lldb with it. What the break loop lacks is DWARF *in a dev build* — a `flan dev` process is not a -`--debug` one. Worth rewording that refusal so it names the real gap. +**`flan-cnr.el`'s stack pane is not blocked on DWARF**, and said so for two revisions. DWARF exists; `flan dev +--debug` now puts it in the host and every redefinition module. The real gap is that **nothing is attached to the +stopped program**: that buffer reaches it over the daemon's socket, and a socket cannot read another process's frames — +the break loop stopped itself, it is not being debugged. It wants an unwinder in the agent, or lldb on the same pid. +The refusal text now names that. ### Managed classes are planned. Do not start them. @@ -210,7 +162,7 @@ Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240 consecutive frames drew it. -Steps 1, 2 and 3 are done — see *The reload primitive* below. A list of top-level forms can be recompiled and installed +Steps 1, 2 and 3 are done — see *The reload primitive* in `BUILT.md`. A list of top-level forms can be recompiled and installed into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes a redefinition over a socket and installs it between frames. @@ -220,7 +172,7 @@ running process was built with, and speaks a protocol an editor can talk to. Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the -interpreter was dropped — open decision #7, settled, see below). +interpreter was dropped — open decision #7, settled — see "Why there is no interpreter" in `BUILT.md`). ``` reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅ @@ -273,302 +225,6 @@ $ flan run test/programs/sand-headless.flan $ flan run sand.flan # a window, 120 fps, hold space ``` -## What milestone 4 added - -**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a -hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not -assignable, which makes the generated step its only writer. - -**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top -level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it -stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer -written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the -bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision. - -`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at -function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. -Block-scoped defer is real work and is not done. - -**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot, -so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed -type and logical on an unsigned one), and `rand-f32`. - -**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is -only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets -the state. This is what the bitwise operators were added for. - -**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type -in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather -than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on. - -## Why the FFI goes through a C shim - -The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64: - -``` -Vector2 {float,float} → declare <2 x float> @GetMousePosition() -Color {u8,u8,u8,u8} → declare void @ClearBackground(i32) -Rectangle {4 × int} → declare { i64, i64 } @mkrect() -``` - -None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a -per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that -in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning -garbage rather than as a link error. - -So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an -out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a -copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare` -signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host -ABI, implemented twice", and `flan_rt.c` is the same pattern. - -### The wrappers are generated now — `declare-c`, `lib/shim.ml` - -The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to -generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory -holds `raylib.flan` and `link` and no C at all. - -One binding is now one line: - -``` -(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture") -``` - -`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like -any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the -function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an -ordinary Flan `defn` above it when the signature has a struct in it. `flan shim ` prints the whole file. - -**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path -string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h -i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural -rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited. - -**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same -`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what -makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted: -that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real -signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was -considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard: -for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct -— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves. -Everything where they could diverge is refused at the field. - -**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a -scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the -hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against -the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree. - -**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path, -512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a -string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on -malloc failure, where the alternative is handing C a null pointer. - -**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice -and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is -refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not -recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes -`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned -string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with -the reason, and the acceptance table asserts on the reasons. - -**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc + -`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function -that calls an existing binding is unaffected. - -`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is -`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17 -34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity -would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed. - -The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write. - -Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows. - -The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance -case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a -GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins -`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all — -storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`, -`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four -ways for that reason. - -No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared -library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora -ships the runtime library without the `.so` symlink. - -### What a headless FFI test can and cannot pin - -Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing. - -- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C -writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and -getting identical output. -- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the -way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, -passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from. -Distances are worse: the reflection does not even reach them. -- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does -not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision -cases look like they cover it. -- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different -field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged. - -- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available. -`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values -in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely, -including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU -reads its width, height or mipmaps at all. -- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image -`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and -answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more. -- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the -package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the -`x`/`y` permutation above. - -The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail -by permuting the fields and watching it go red. A case not verified that way is decoration. - -One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It -measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no -window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime` -and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are -exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them. - -## Packages - -`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is -a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no -collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares -becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is -rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of -declarations whose names contain a slash. - -A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file -named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below. - -**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also -a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import -all four. A file carries no `.c` and no `link` file; those belong to a directory. - -**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that -is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under -one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which -is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time, -and the namespace being flat, mutually dependent packages simply work. The same directory under two *different* aliases -is refused. - -**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment -anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would -keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the -checker it would be "unknown name", which is true and useless. - -Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable. - -## The link follows the program - -`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the -importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that -link cannot succeed. - -`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it, -follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing -else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument. - -Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and -`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs** -from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a -dropped one is a silently different program. - -**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far. - -The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and -`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker -arguments together, so a caller cannot take one without the other. - -## sand.flan is one program - -It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a -program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both -drivers imported it. - -Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports -`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches -raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both -targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation. - -What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no -input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and -`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still -168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost. - -Two claims that got run together in an earlier note, for the record: - -- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the -**wasi** path, which is what the headless table targets, and which has no GL and no browser. -- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with -`emscripten_set_main_loop` instead of a `while`. That is a different `main`, not a different program. - -**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes: - -- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`, -and there is no implicit widening. -- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`. -- `(defn main [])` is unchanged — the short form, as plan.org says. - -Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what -cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press. - -## Bounds checks — done at milestone 3 - -`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three -check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds -never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo -<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. -`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the -same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`. - -The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor -that split them would break the write check silently. The test covers both. - -Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s -unchecked. Indistinguishable. - -## Why there is no interpreter - -Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had -expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, -perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is -hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second" -criterion is dropped, and the host ABI moved onto the critical path. - -## The layout, which is the whole backend design - -``` -i8..i64 / u8..u64 i8..i64 signedness lives in the ops -f32 f64 float double -bool i1 -an enum i32 -[T] and string { ptr, i64 } ptr+len, non-owning -[n T] [n x T] inline, a value -(Ptr T) ptr opaque pointers -(Option T) { i8, T } tag 0 None, 1 Some -a struct a literal struct, declaration order -Unit and Never {} -``` - -No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the -semantics: - -- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is* -the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody -addressed. `test/programs/values.flan` pins this down. -- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a -`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong. - -Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding, -so wasm32 needs no exception proposal. - ## Blocked and unfinished Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a @@ -576,6 +232,59 @@ vague intention — if it is listed, someone has already established it is real. ### Unblocked now, and ranked +**From porting ten raylib examples** — the first code the language was pushed by that it was not designed around. +Ranked by how often they were hit, top two first because they are walls rather than conveniences: + +1. ~~No number reaches `draw-text`.~~ **Fixed** by `(string b)`. +2. **An enum parameter cannot be driven by a loop variable.** `(rl/get-gamepad-axis-movement pad i)` with an `i32` + index: *expected rl/GamepadAxis, found i32*, and the other direction refuses too — *i32 converts a number, found + rl/Gesture*. The obvious escape is closed as well: a second `declare-c` of the same symbol with an `i32` face gives + *one declare-c per C function, and another Flan name for it is a defn* — and a `defn` renames without retyping. + Two individually-correct rules composing into a wall; the caller writes a `cond` over the members instead. Cost: + every gamepad-axis loop, and four comparisons on the raw gesture bitfield. +3. **`break` is not implemented.** Declined deliberately rather than built — see below. +4. **A `let` binding takes no type annotation**, so a fixed array is either a top-level `defvar` or a literal with + every element spelled out. `(let [pts [4 rl/Vector2]] …)` parses as a two-element array literal and fails with + *unknown name rl/Vector2*. Cost: 32 hand-written `Vector2`s in one example. +5. **Arithmetic is strictly binary** — *+ takes 2 arguments, given 5*. +6. **No `sin`/`cos`/`abs` for floats.** The prelude has `sqrt-f32` and nothing else transcendental. Two `declare` + lines over libm, and `(max x (- 0.0 x))`; the examples each declare their own, which is a copy per file. + +**A `string` cannot be returned from C at all**, which is what makes `GetGamepadName` unbindable: *a string only +crosses as a parameter — a C function that returns one returns something Flan has no owner for*. Same rule refuses +`TextFormat`, which is also variadic and so has no honest signature. + +**The negative result is worth as much.** None of the gaps expected blocked anything — no generics, no allocator, no +`Vec`/`Map`, no escaping closures, and function-scoped `defer` never came up. Input-and-draw over fixed-size state is +the shape the language already has. Three constructs unexercised anywhere else in the repo worked first try: a fixed +array with a struct element, a 2-D struct array, and `[N string]` as both `defconst` and mutable `defvar`. + +### `break`, and why it was not built + +Settled, so the next attempt is cheap rather than a rediscovery: + +- **`dotimes` gets it free** — it desugars to `Tast.While`, so one implementation covers both loop forms. +- **`defer` is a non-question.** It is function-scoped, `break` does not leave the function, nothing fires. No refusal + needed and no interaction to design. +- **Type it `Never`**, as `exit` and `return` already are. +- **`pads` is the structural model.** `emit_while` already makes an `endloop` label; break is a push/pop of that around + the body plus a `br`. `return` is a direct terminator with no context threading, so there is nothing else to mirror. + +What stopped it, and neither is small: + +- **`check.ml`'s `in_frames` rule does not extend.** It refuses `return` inside `handler-bind`/`restart-case` because + those frames are popped on the way out, and that refusal is blanket because `return` *always* crosses. `break` + crosses only sometimes — a loop wholly inside a `restart-case` body has a legitimate local break — so the precedent + has to be replaced by a loop-depth-relative-to-frame-entry rule nobody has ruled on. +- **`continue` forces a `Tast.While` signature change.** `check_dotimes` folds the step into the body as + `While (cond, body @ [step])`, so a `continue` branching to the header skips the increment and hangs. It needs a + latch — `While of expr * expr list * expr list` — across `check.ml` and `emit.ml`. plan.org settles break and + continue as one item and `parse.ml` refuses them in one case, so building break against today's `While` is exactly + the thing that would have to be undone. + +plan.org's single line on it (831) names a `for` the language does not have and gives no mechanism. + + 1. **Allocators, then `Vec` and `Map`.** The critical path, and the only thing standing between this and writing a game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin @@ -689,6 +398,10 @@ Sixty mutations, nineteen left the whole suite green. The severe cluster is clos - **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which `load.ml` matches exhaustively. - **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`. +- **A `!DILexicalBlock` per `Let`.** Not one line, but the one thing left in the DWARF work: every `!DILocalVariable` + is currently scoped to the subprogram, so inside `(let [v 22] …)` nested in `(let [v 11] …)` lldb still answers + `p v` with **11**. The `~2` suffix makes both *visible*, which is not the same as making the answer right. It needs + block structure the typed IR does not carry, and the `llvm.dbg.declare`s moved out of the entry block. ### Deferred with a reason @@ -780,885 +493,10 @@ inline. - `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous. -## The reload primitive — dev loop steps 1 and 2, measured +## Loose ends from milestone 4 -`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled -into its own object and called inside a process that is already running, twice, with a changed body the second time. - -| Step | Cost | -|---|---| -| `Emit.redefinition` | below the timer (<0.1ms) | -| `llc -O2 -filetype=obj` | 15–17ms | -| `ld -shared` | 3ms | -| `dlopen` + `dlsym` | **0.04ms** | - -**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the -number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost, -which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since -the driver absorbs IR the bare tools reject. - -`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that -*is* the mechanism. What the new module does **not** define is the whole design: - -- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a -redefinition can change a function's body and can never re-initialise the program's data. Define the global and the -loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the -thesis. -- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a -private copy of it. -- **no `main`.** This module is loaded, not started. - -Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss -because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is -spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a -slice parameter. - -**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the -compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the -agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release -builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_get`, which reads as a compiler -bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and -the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason. - -**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a -loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property -of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol. - -Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**, -since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since -`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And -`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares -`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the -host's copy. With the two bodies identical nothing at run time would notice a module that grew its own. - -String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's — -worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would -fail at link time. The fixtures each print a literal so that path is actually exercised. - -### Cells — how a call site follows a redefinition - -Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes -every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current. - -``` -@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it -%p = load ptr, ptr @"flan.cell.bump" ; every call site -%r = call i64 %p() -``` - -Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and -exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what -makes a frame-boundary swap a non-event. - -The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of -one. - -Four things about this that are not free choices: - -- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on -whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are -separate on purpose. -- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to -*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would -publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, -because the failure is silent. -- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined -function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture -recurses on purpose, and would print the old body's text if it did not. -- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is -the first time `opts` means something semantic rather than an optimisation level. - -LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46 -indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well; -the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function. - -### Names that did not exist when the process started - -Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF -cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else: - -``` -void **flan_dev_cell(const char *name); /* a new function's cell */ -void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */ -``` - -Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A -new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores -on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not -reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the -rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a -call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the -common case pays nothing for the general one. - -**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c -C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a -redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet. - -Four rules, each of which is a silent failure if broken: - -- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are -still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`. -- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the -old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. -This is the layout-drift rule's first enforcement point. Retyping a var needs a restart. -- **Nothing is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it leaves every call site -pointing at unmapped memory. That is a constraint on the agent too. -- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed -capacity with a loud failure rather than growable. - -The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at -run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell* -both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still -pass and the transcript would read 246 instead of 432. - -Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout -calculator in OCaml that would have to agree with LLVM's on every target. - -### The agent — dev loop step 3 - -`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link` -asks for `-lpthread`. - -``` -(agent/start path) listen on a unix socket; once, at startup -(agent/poll) install whatever has arrived; returns how many -(agent/wait ms) the same, but waits for something first -``` - -The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded — -so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a -redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks. -The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the -loader. - -`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits -for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first -introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if -it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that. - -Two details found by running it: - -- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a -program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at -all, which is how this was found. -- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program -can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`. -- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows -when it is between frames. - -**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under -Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body -that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is -refused on the listener thread and the loop never notices. - -`flan reload ... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the -names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what -the session will track automatically. - -### The session - -`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change -accepted since. - -**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a -declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not -replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that -follows. There is a test for the case that actually matters: a typo, then a good form, in the same session. - -Two things the session knows that no single evaluation could: - -- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST, -because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `print-line` reads -as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. -- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded: - -| Change | What it would have broken | -|---|---| -| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below | -| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve | -| a struct's fields | the values the process is holding have the old layout | -| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it | -| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too | - -**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should -make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while -existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked -caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error. -Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines -(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because -the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message -should not be read as the final answer. - -A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past -long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global` -record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than -the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is -the same fact that makes them unreloadable. - -Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first, -loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program -uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for -that reason. - -A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A -dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a -module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That -is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the -folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could. - -**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in -`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s -own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a -brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it -already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently -did nothing. - -It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a -Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written -nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to -either. - -The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise -re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name -pass rejects it. `C-c C-k` on sand.flan's own text is the test. - -`flan reload ` is that path from the command line: a session over the program the process was -built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw` -and 910 consecutive frames drew it. - -Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on -every invocation, so if the program file has been edited since the process launched, its idea of which names the host -has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to -``, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass -the real buffer path, which is the same key CIDER's `eval` carries. - -### The daemon — `flan dev` - -`flan dev ` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside -the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part -of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe -the process that is actually running rather than a guess about it. - -**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client -became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form, -from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the -editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of -Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains -newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor. - -``` -(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t) -(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0) - → (:status "error" :message "…" :loc "/buf.flan:1:19") -(:op "defs") → (:status "ok" :defs ((name kind signature loc) …)) -(:op "close") -``` - -`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the -program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer. -It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing -else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none -to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself, -which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step -[i64 f32] i64`: types only. - -The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan -dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by -guessing which directory it was relative to. - -An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage -— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one -anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need. - -`:file` is not decoration: `Session.eval`'s origin defaults to ``, so without it every error an editor shows -points into a file that does not exist. - -Two things the daemon must not paper over, both of which would look like a successful evaluation: - -- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan -says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead -fails silently: the module compiles, is built, and nobody receives it. -- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed -connect or a reply that is not `ok` becomes an error the editor sees. - -It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads -like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather -than leaving an editor waiting on a socket nobody is serving. - -### The Emacs client - -`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is -s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is -Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is -written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords. - -`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a -request and `read` reads a reply. - -| | | -|---|---| -| `C-c C-c` | the top-level form at point, recompiled and installed | -| `C-c C-k` | the whole buffer, as **one** module | -| `C-x C-e` | the expression before point, evaluated *in the running program* | -| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect | -| `C-c C-o` | the running program's own output, in `*flan-output*` | -| `C-c C-r` | a prompt on the running program (`*flan-repl*`) | -| `C-c C-b` | what a **stopped** program is offering, and which to take | -| `C-c C-d` | what the running program currently defines | -| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR | -| `M-.` / `M-,` | where a name is written, through an `xref` backend | - -eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on -an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at -the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just -installed completes at once. - -The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away, -which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the -socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the -daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice. - -`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to -arrive in the same load, or the first refers to storage that does not exist yet. - -**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary, -or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload -has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives -the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a -mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml` -and fails here. - -An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the -next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte -offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and -`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the -daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1 -and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the -reader skips, so the reply's line numbers are the buffer's own. - -An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent -success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the -one point looked like it was in. - -**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to -into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form -"asm"|"ir")` is the op. - -What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d ---disassemble=flan.` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared` -deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was, -so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else -on the machine still has that text. A table from function name to the last module accepted for it is the whole of the -bookkeeping. - -**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket -takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address -space, and `C-x C-e` renders a pointer as `` on purpose — so nothing the daemon can ask would tell it what a cell -holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first -instruction: - -- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*; -- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed; -- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed, -since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further -installs until it resumes. - -From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch -targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this -build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When -the debug build lands, that is the line to delete. - -**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an -annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything. -`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the -overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added -where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke -in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the -same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it -is Emacs' contract and a test claiming to check it would be checking nothing. - -**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the -next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the -output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and -not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the -accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot -grow the daemon without limit. - -### `C-x C-e` — evaluating an expression - -A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body -into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by -exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an -expression reading the program's state sees a point the program agrees is consistent. - -**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say -what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout -decision's bill, and it is why the printer set is small rather than universal. - -It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a -dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's -socket. The read is safe without a handshake because `flan_dev_result` bumps a generation counter last; the daemon waits -for it to move rather than assuming the program has reached a frame boundary. - -**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a -compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints -structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth -and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three -bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a -builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time -type to walk. Not the walk itself. - -The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise -because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an -allocator the language does not have. - -``` -big 18446744073709551615 -col :blue -(.pos b) (V {:x 1.5 :y 0}) -b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]}) -(slice (.tags b) 0 3) [ 0 42 0] -(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68}) -sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...] -``` - -Details that are decisions rather than formatting: - -- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come -back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point -instead. -- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather -than as the value it is. -- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are -erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is -exactly what you would want to see. -- **A pointer is never followed** — ``. It is the only thing that could make the walk cycle, and dereferencing one -a REPL was handed is not a safe thing to do on someone's behalf. -- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does -not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K -and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it -means no renderer carries a budget. -- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is -not evaluated once per element. - -What still refuses by name: `Map`, `Fn`, a type variable. - -A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer. -`flan run calc-me.flan "1.5 * 2.0"` has always said `3`. - -An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe` -does not fill up with `eval/N` for every expression ever typed. - -**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by -`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points -into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the -agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition* -retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind, -and can never claim this. - -Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop -would exhaust it. - -The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must -disagree. A value computed in the compiler, or read out of a copy of the program's state, would not. - -### The REPL buffer - -`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No -new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as -deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess -behind it — the "process" is a stub comint needs in order to have a prompt at all. - -Three things about it that are decisions: - -- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in -sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it -because the file says which package it belongs to; a prompt has no file and nothing to derive one from. -- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a -paren inside a string does not count. -- **A value and the program's output are different things and arrive by different routes.** The value is the result of -the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply -and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the -separation. - -That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so -`(print-line "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A Unit -expression is almost always a call made for its effect, and is now evaluated and *then* reported. - -### Conditions — step 1: `handler-bind` and `signal` - -`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters -control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling -function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists -yet, and no signature changed. - -``` -(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))] - (load-all)) -``` - -The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing -function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame -rather than by count, so restoring what this one displaced is right even if something below it left the stack out of -step. - -Three decisions worth keeping: - -- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and -every handler a running program had already pushed would then match the wrong type. FNV-1a over the name. -- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is -nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the -name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address -instead. -- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a -branch in the function that wrote it. -- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from -plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a -particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the -duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still -valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the -stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When -`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them. - -Which gives the two refusals, both by the house rule rather than by accident: - -- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a -reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in -scope, which is what the accumulation case needs. - -What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not -outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by -value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete -use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the -biggest usability limit in conditions is not behind the thing that was just deferred. -- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would -leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block. - -### The break loop — conditions step 3 - -Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the -frame that erred with nothing unwound, so the condition and every restart between there and the top are still live. - -``` -flan: unhandled Missing — stopped, not dead. - restart: retry - restart: use-placeholder -``` - -Four decisions, each of which is the reason something is where it is: - -- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the -release runtime and must not depend on something a program may never import. A program with no agent leaves the hook -null and dies exactly as it did before. -- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes, -reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same -act, lowered once. Nothing about §6 needed changing to support it. -- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a -convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop -that did not drain that queue would hang `C-x C-e` exactly when it is most wanted. -- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must -not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new — -and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the -cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of -stopping. - -**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but -a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also -`compute-restarts`' data, whenever that arrives. - -**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused -there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that -cannot happen. - -The socket verbs are `restarts`, `restart-at [name]`, `restart ` and `abort`, and all four are refused with -the reason when the program is not stopped — there is no restart stack to walk from a running one. `restarts` answers -` <+|-> ` per line: the index is the identity, and the flag says whether a transfer to that frame has -anywhere to land. `test/programs/break.flan` errors three times — two restarts taken by name, then a shadowed pair -where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved -by name could pass. - -### The break loop in the editor — conditions step 3, the other half - -The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it -follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a -question an editor chose to ask. - -So the state is learned **twice, deliberately**: - -- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`, -or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now -errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding -out *after* the echo area had said the evaluation was fine. -- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is -drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies -that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left -behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the -`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs -timers, so a poll firing inside a read would eat the reply that read was waiting for. - -Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is -stopped, so the poll and the prompt cannot disagree. - -``` -(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3) - :stopped t :condition "Missing") -(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …") -(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …") -(:op "abort") → (:status "ok" :note "the program is exiting; …") -``` - -`:restarts` is *positional* — innermost first, duplicates kept — because the position is what `restart-at` takes. -`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is -not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a -prompt cannot take a different restart than the one it showed. - -`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by -someone about to choose from it. - -**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against -the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would -poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and -lets the next poll settle it. - -The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running -one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly -right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key — -it is the thing you pick when none of the restarts is the answer. - -**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A -`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop -cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the -nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of -breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices -each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an -expression that errors and resumes it, which fails against the old shape. - -The agent grew one verb, `status`, answered in **both** states — `running` or `stopped `. Everything else the -break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had -to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the -hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know. - -`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the -state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way -round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e` -answering while the program sits in the break loop is checked there against the real client, not only in OCaml. - -### Conditions — step 2: `restart-case` and `invoke-restart` - -`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a -`restart-case` further out. - -``` -(defn fetch [n i32] i32 - (restart-case (middle n) ; its value if nothing transfers - (use-placeholder [] -1) - (retry [] 7))) - -(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))] - (fetch 2)) ; -1 -``` - -**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an -`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the -release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's -slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers. -`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather -than a hunt. - -**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare -pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and -plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight -through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform -also means redefinition acquires no new refusal class. - -**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous -note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only -probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The -frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is -how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was. -Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's -"innermost offering the name" is just the order of the walk. - -**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever -frames it established and either catches the transfer or forwards it outward: - -- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts -the target back and goes on out; -- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, -and without it the handler stack would be left pointing into a frame that has gone; -- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is -still refused by name. - -A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would -jump straight past the very form that was supposed to catch it. - -**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is -guarded; with the channel still set the first of them would branch straight back into the landing block it came from. -Same reason the clause body starts with it null. - -`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes -the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot -cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and -there are no function values yet, so nothing can call *back* into Flan across one. - -Scope, each piece refused by name with its reason and a test on the reason: - -- **restarts take no parameters.** That covers §1's own `load-texture` example and skips argument marshalling and §3's -runtime arity check. -- **TODO — CL-style interactive recovery.** A stopped program should be able to offer a typed restart such as -`(use-value [value T] value)` or `(use-function [replacement (Fn ...)] ...)`, and the editor should show its signature -and ask for the replacement before invoking it. That is the missing "this variable is None; what should I use instead?" -path: named, hard-coded branches are useful for `retry` and `skip`, but not a substitute for an interactive value or -alternate implementation. It needs restart argument marshalling and validation in the runtime, plus an editor protocol -for entering and checking the supplied value/expression. -- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops. -- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a -choice nothing in the source shows. -- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting -there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a -defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see -it. -- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past -everything. There is nowhere to resume, so there is nothing else to do. - -**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal` -returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past -it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked -`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case` -body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled -case: it cannot be an `outputs` row, because it does not exit 0. - -`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart` -through a call traps either way, and the message names the rule rather than the path, since the rule is the same. - -**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a -redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a -hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable -nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value. -A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot. - -`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca` -in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and -crosses `probe`, which the host was compiled with. Those three do not meet anywhere else. - -Two things found by writing it: - -- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates -the body's slots into a record the function never sees again and the indices collide. It was harmless only because no -`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now. -- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not -fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere -else entirely. - -`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant -here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each -of those calls goes through a cell. - -### What is left -- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays. - -**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the -memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. -Attaching to a process someone else built is not a thing to support by default. - -## Where build time goes - -`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms: - -| Step | Cost | -|---|---| -| frontend: read → parse → load → check → emit | <10ms, below the timer | -| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** | -| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** | -| link | 20ms | - -Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which -compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler -(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build), -`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0` -and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d -into place, so two concurrent builds cannot see a half-written one. - -Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never -the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever -a disk-space decision. - -The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It -is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted -IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse -errors. - - -## Next — the REPL is the priority - -Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place), -and **the dev loop is the thesis of the project**, so it comes first. Staged so each step is runnable on its own — the -failure mode is building a daemon and a protocol before knowing the reload primitive works. - -1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms. -See the section above. -2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`, -`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a -run-time-added function is itself redefined. See the section above. -3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs, -and sand.flan polling at the top of its frame. See the section above. -4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than -nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if -something else ever needs to talk to it. - -**One decision left to settle before step 2**, because both change codegen and are painful to retrofit: - -- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module -declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep -the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload, -and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet. -- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what -`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from. - -Deferred until after the dev loop: - -6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and -`test/programs/sand-headless.flan` prints `-2851001042534928384` under it — the same hash as native, byte for byte, at -`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared -across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer -would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and -runs it — rather than by looking for a binary on PATH. - -Three things this cost that were not in the old note: - -- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main` -to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a -signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")` -label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite -self-call. -- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes ``; without `--sysroot` -it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the -triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`. -- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both -shapes count. - -**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to -come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it -`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple -(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk -is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's -beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is -root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named -by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in. - -**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of -`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes -an `ExperimentalWarning` to stderr on every run and the harness compares combined output. - -**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with -one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime -at it). - -Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. wasi-sdk is right for the -headless table; it is not necessarily right for the eventual game build. - -7. **Loose ends from milestone 4**, none of them blocking: block-scoped `defer`; package visibility, so -`rl/get-color-raw` is not callable; a package importing a package; imported unions. +None of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; a package +importing a package; imported unions. ## Macros — the reader and the declaration are in, the expander is not @@ -1679,7 +517,7 @@ for a design to rot, and the storage shape is the expander author's first decisi ### How the expander should work -**There is no interpreter** (see "Why there is no interpreter") and there is not going to be one, so running a macro at +**There is no interpreter** (see "Why there is no interpreter" in `BUILT.md`) and there is not going to be one, so running a macro at compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured: `Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.