flan/BUILT.md
Joseph Ferano 65a62bbb97 A socket that looped back into its own address space, deleted
flan dev is one process, so the unix socket between the compiler and the
program was a connect, a write and a read that never left the machine's own
memory. A delivery is now a call into the agent's verb table.

flan_agent.c's handlers write into a sink that is either an fd or a growing
buffer, so serve() and the new flan_agent_request() share one verb table rather
than two that would drift -- every answer in there is assembled from several
pieces, so the seam had to be the writing. Dev.deliver, Dev.result and Dev.ask
were three copies of the same socket dance and are now three lines over one
Dev.request.

Which path is taken is the linker's answer, not a flag: flan_agent_request is
weak in dynload_stubs.c, null in every binary that links no flan_agent.o, and
Agent.request is None there. So --two-process, flan reload and every test go on
using the socket with nothing to configure.

Three things the direct path must not quietly change. The ring's room check is
separate from its store, which was safe only while the accept loop was the sole
producer, so handle_line runs under a mutex. A delivery is still only published
-- the install is one store per function on the game thread at a frame
boundary, and doing it on the spot would be a frame running half in the old
code and half in the new. And the OCaml runtime system is released across the
call, because a delivery is a dlopen.

Measured, and the number is the point: the transport was ~50us of a 23ms
redefinition, so end to end did not move. Code generation is 19 of the 22
milliseconds. The merge's prize was never latency.

The test is a deletion, because a reply cannot say which way it came:
test_dev.ml unlinks the agent's socket file once the merged program has bound
it, and every evaluation after that still installs.
2026-09-12 22:22:46 +07:00

249 KiB
Raw Blame History

How the parts that exist work

The reasons behind the code, kept separate from NEXT.md so that what to do next is not buried under what was already done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever dlclosed, why a call bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather than a function in the runtime. Deleting it would mean deriving it again.

NEXT.md is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that first. Come here when you need to know why something is the shape it is.

What milestone 4 added

dotimes desugars in check.ml to a Let plus a While — no new IR node. The bound is evaluated once into a hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not assignable, which makes the generated step its only writer.

defer is recognised in check_fn and nowhere else, because that is the only place that knows a form is at the top level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it stands. Function exit runs them innermost-first, and an explicit return runs the ones registered above it — a defer written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the bounds-check shape (noreturn then unreachable) rather than being a separate decision.

defer inside a let, a loop or a branch is rejected, not accepted with function scope. It would run once at function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. Block-scoped defer is real work and is not done.

New builtins: zeroed (takes its type from the place it is stored into), min/max (each operand through a slot, so neither is evaluated twice), bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a signed type and logical on an unsigned one), and rand-f32.

rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state. It is not libc's, because a grid hash is only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed sets the state. This is what the bitwise operators were added for.

Enums and keywords. (defenum Name [member value ...]) gives a type that is an i32 at run time and its own type in the checker, so :space at a call site resolves against the parameter's enum and a typo is an error there rather than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.

Why the FFI goes through a C shim

The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:

Vector2 {float,float}   →  declare <2 x float> @GetMousePosition()
Color   {u8,u8,u8,u8}   →  declare void @ClearBackground(i32)
Rectangle {4 × int}     →  declare { i64, i64 } @mkrect()

None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a per-target classification the caller has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that in emit.ml is three classifiers to write and then keep correct forever, and a mistake shows up as (.y m) returning garbage rather than as a link error.

So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a copy. clang classifies all of it, per target, for free. check.ml enforces the rule — an aggregate in a declare signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host ABI, implemented twice", and flan_rt.c is the same pattern.

The wrappers are generated now — declare-c, lib/shim.ml

The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to generate "if that ever becomes the bottleneck" came true at 84 of them. vendor/raylib/shim.c is gone; the directory holds raylib.flan and link and no C at all.

One binding is now one line:

(declare-c draw-texture [t Texture2D  x i32  y i32  tint Color] "DrawTexture")

declare-c names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like any other: the typedefs for the structs involved, made from the Flan defstructs; the extern prototype in the function's true signature; the wrapper that flattens it; and the flattened declare the Flan side calls, with an ordinary Flan defn above it when the signature has a struct in it. flan shim <file> prints the whole file.

It is a second form and not a change to declare, for one reason worth remembering: (declare start-raw [path string] i32 "flan_agent_start") in vendor/agent means the symbol takes ptr+len, and (declare-c init-window [w i32 h i32 title string] "InitWindow") means it takes a NUL-terminated char *. Same shape, opposite claims, so no structural rule can separate them. declare is untouched, and sqrtf and the agent still work unedited.

What the generator guarantees, and what it trusts. Guaranteed: the C typedef and the Flan struct come from the same defstruct, so they cannot disagree — permute the defstruct and the typedef permutes with it, which is exactly what makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted: that the defstruct matches the library's real struct, and that the declare-c signature is the function's real signature — no header is read, deliberately, so nothing can check either. A _Static_assert on sizeof/offsetof was considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard: for every field type the generator admits — the machine integers, the two floats, bool, a pointer and a nested struct — LLVM's layout is C's, and emit.ml writes no datalayout, so clang applies the target's rules to both halves. Everything where they could diverge is refused at the field.

One thing got sharper and should be said plainly: the prototype is now generated from the declaration, so a scalar's width carries ABI weight it did not before. f64 where raylib says float used to be narrowed by clang at the hand-written call site; now it emits double and raylib reads garbage. All 84 migrated prototypes were diffed against the deleted shim.c's — which was the ground truth for the true signatures — and agree.

Strings. The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, PATH_MAX for a path, 512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on malloc failure, where the alternative is handing C a null pointer.

Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C. collision-point-poly? takes a slice and collision-lines answers with an Option; neither is raylib's own signature. A slice parameter in a declare-c is refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not recoverable from [T] — so that one declares (Ptr Vector2) with an explicit count i32 and the Flan wrapper passes (addr (at points 0)) and (len points). Every other refusal — an Option, a union, a fixed array, a map, a returned string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with the reason, and the acceptance table asserts on the reasons.

Known edge, not fixed: a REPL redefinition that introduces a new declare-c cannot work. Build.shared is llc + ld -shared and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function that calls an existing binding is unaffected.

raylib.flan carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is (rl/get-mouse-position) returning a Vector2. Verified end to end, headless: GetColor(0x11223344) comes back as 17 34 51 68, four separate bytes — a Color is not the little-endian reading of the packed integer, so an identity would have passed a weaker test. That case is in the acceptance table, skipped if libraylib is not installed.

The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven collision-* predicates; textures; the Image family; Camera2D; RenderTexture2D; the whole audio surface (device, Wave, Sound, Music); fonts and glyphs; and gamepads, touch and gestures — plus the Key, MouseButton, TraceLogLevel, GamepadButton, GamepadAxis and Gesture enums, raylib's own named colour palette, and the FLAG_ window hints. Adding one is a single declare-c line; there is no C to write.

Two things the ported examples in examples/ wanted and could not have, both refused for reasons that are right. GetGamepadName returns a char * into raylib's static storage: the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that returns one returns something Flan has no owner for. And an enum parameter cannot be indexed — GetGamepadAxisMovement takes a GamepadAxis, a loop variable is an i32, expected rl/GamepadAxis, found i32, and a second declare-c of the same symbol with an i32 face is refused too: one declare-c per C function, and another Flan name for it is a defn — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a cond over the members it knows.

The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a GPU: GetCollisionRec, which pins Rectangle completely, and SetShapesTexture's default substitution, which pins Texture2D's id and format and nothing else. Handing a struct over and reading it back proves nothing at all — storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. width, height and mipmaps are therefore checked only by looking at sand.flan running, which draws the brush sprite four ways for that reason.

No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared library being linkable and not on raylib-devel. vendor/raylib/link carries -l:libraylib.so.550 because Fedora ships the runtime library without the .so symlink.

The header is read now — headers, lib/cimport.ml

The section above ends by naming what the generator trusts: that the defstruct matches the library's real struct, and that the declare-c signature is the function's real signature. "No header is read, deliberately, so nothing can check either." A header is read now, and both are checked.

The dependency, which is the crux, and which this project already answered once. Zig's old @cImport ran clang as a library. That is exactly the dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++ library breaks routinely on upgrade, a binary on PATH does not. So this shells out for clang -Xclang -ast-dump=json -fsyntax-only, which is the same binary every build already runs and adds nothing that is not already being paid for. lib/cjson.ml is enough JSON to read that dump and no more, so it adds no opam package either.

A note worth recording, because it strengthens the argument rather than weakening it: Zig has since abandoned clang here too. translate_c.zig is gone; lib/compiler/translate-c/ is built on Aro, a C frontend written in Zig. Their reason was to ship a compiler containing no clang at all — the opposite premise to this one, where clang on PATH is the toolchain assumption. Both projects walked away from linking libclang; only the destination differs.

What is imported: functions, and only functions. Not structs, not enums, not macros. The bound on how much is not a curated list but the package's own defstructs — a function whose signature mentions a struct the package has not described is refused with that reason, so vendor/raylib describing thirteen structs is what makes the import thirteen structs wide, and describing a fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are refused, and 172 are left alone because the package already binds them by hand.

Not generating defstructs is what makes the check possible at all. Generate them and the header becomes the authority on layout, and comparing the package's layouts against the header's would be comparing the header with itself. A _Static_assert on sizeof/offsetof was rejected in the section above as circular for exactly that reason; this is not circular, because the two sides have different authors. It is the cheapest real closure of the trusted-not-guaranteed gap.

Refusing by demotion, which is the one thing taken wholesale from Zig. Zig's translator never drops a declaration it cannot handle: failDecl binds the name to a @compileError carrying the reason, so the name still exists, the program still compiles, and asking for that one name fails at the use site with the reason. A wholesale import has a hundred and fifty refusals and a caller cares about the one they typed. Flan already had that mechanism — Load.refuse_hidden, built for main — so rl/get-gamepad-name is not a name, and a program that writes it is told the return type is a string, and a string only crosses as a parameter rather than "unknown name".

That is also the split shim.ml needed and did not have. It refuses through Loc.fail, which is right when a human named one function and wrong for a wholesale import, where one returned const char * would kill the header. Same judgement, different disposition; a hand-written declare-c still hard-fails and shim.ml is untouched.

Two C spellings mean something in a parameter that they mean nowhere else. const char * is a string going in, and the generator already knows how to hand one over. char * without the const is very often a buffer the callee writes, and handing it a NUL-terminated temporary would lose the writes with no diagnostic anywhere — const is the only thing in the header that separates the two, so it is what decides, and an out-buffer keeps a hand-written binding saying (Ptr u8). long, size_t and the rest are refused rather than guessed, and for a reason specific to this project: it builds for x86-64, for wasm32-wasi and for the browser, and long is 64 bits on the first and 32 on the others, so a guess would be right for the target that gets tested and silently wrong for two that do not.

Naming. rl/InitWindow is rl/init-window. Reversibility is not a property of the rule — the C symbol is stored verbatim in the declaration, so the wrapper reads the library's spelling rather than reconstructing it. What the rule must be is injective over one header, since two C functions arriving under one Flan name would surface as a duplicate declaration about a name nobody wrote. A boundary goes before an uppercase letter after a lowercase one, before an uppercase letter between an uppercase and a lowercase, and before a digit after a lowercase; nowhere else. So SetTargetFPS is set-target-fps and not set-target-f-p-s, BeginMode2D is begin-mode-2d, UnloadUTF8 is unload-utf8. raylib's 581 names are injective under it. When two do collide, neither takes the name — resolving by order would mean that moving two lines in somebody else's header silently rebinds a name a program is already calling. Both are refused, both say why, and the author binds the one they want with a declare-c.

Where it runs. headers beside link in the package directory, read the same way: a path, any clang flags it needs, ${NAME} expanded from the environment. What comes back is ordinary declare-c declarations, generated before the package's names are qualified, so they arrive as rl/… exactly like the hand-written ones and nothing downstream can tell which is which. No new form, no new decl_kind, no reader or parser change. A C symbol the package already binds by hand is left alone, so declare-c stays the escape hatch and stays the thing that wins.

A leading ? makes a line optional, and vendor/raylib uses it. "No raylib headers are needed" is a real property — a build needs libraylib linkable, not raylib-devel installed — and requiring a header would take it from everyone in order to give the check to whoever has one. Unset FLAN_RAYLIB_H and the build is exactly what it was; set it and every signature is checked. A path that is set and wrong is an error naming it, because silently behaving as though nobody had opted in is the difference between an opt-in and a trap.

What the diff found

The evidence the whole lane exists for. Against raylib 5.5 — the version whose .so link names — all 16 defstructs and all 172 hand-written declare-c agree exactly. The half BUILT.md called trusted is now checked, and it was right.

That is only worth stating because the check has teeth. Against the 5.1-dev header installed in /usr/local it reports ten differences: nine functions that version does not have (CheckCollisionCircleLine, the six Is*Valid renames, DrawRectangleRoundedLinesEx) and DrawRectangleRoundedLines, which gained a parameter. Picking the wrong header is therefore loud, which matters, because the two headers are on the same machine and only one matches the linked library.

Both comparisons run at build time and stop the build, not just in the tool. Verified by breaking them: a permuted Texture2D fails naming the field that moved, and f64 where raylib says float fails naming the parameter — which is the hazard the section above calls out by name and says only a test can catch. The message points at the line in raylib.flan, not at the header.

flan import-c <header> [package.flan…] prints what it would produce, what it refused and why, and both comparisons, without building anything. That also makes "generate once and commit the result" available for the cost of a printer — explicit in the source, checked against reality, no header read at build time.

What it costs, measured

The number that decides how much to import, because reach.ml was the reason to think a wholesale import could be free.

today (172 by hand) + 256 imported
release build, cold 0.298s 0.312s
release build, warm 0.078s 0.082s
redefinition (flan reload) 31.0ms 46.5ms
dev build, cold 0.649s 0.982s

Reach.link already drops a generated wrapper whose declaration nothing reachable calls, and that is what makes the release column nearly flat. Confirmed on the case it exists for: a wasm32-wasi build of a program that imports raylib and calls none of it still links without libraylib, with 256 extra declarations in play. Dev builds are not pruned, on purpose, so one compiles all 428 wrappers — once, at session start, since Build.shared is llc + ld -shared and compiles no C.

Reading the header is cached, and the cache earned itself against a measurement rather than a guess: 64ms of a 72ms check, against 8ms for the whole program without it. What is cached is the extracted signatures and not clang's JSON, because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest is reading it. Keyed the way the object cache is keyed, on everything that could change the answer: the header's path, size and mtime, the full flag list, and a format version, since the value is marshalled. That takes the delta to 17ms.

The 15.5ms on redefinition is the real cost and it is the argument against importing at build time, on the branch where the dev loop is the priority. It is the strongest case for the third option — generate from the header, commit the result, regenerate when the library moves — and that decision is open.

Where that 15.5ms actually is — and it is not the header

Written above as though a redefinition were re-reading the header on every reload. It is not, and the number was never taken apart. Taken apart now, with a timer around each stage of the same flan reload (raylib 5.5, sand.flan, one defn in the forms file, warm caches):

no header header imported delta
Session.create 7.0ms 21.5ms +14.5
Session.eval 4.8ms 8.4ms +3.6
llc + ld 21ms 21ms 0

And inside that create, the header's own share: reading the cached dump 0.33ms, turning it into declarations 3.3ms, checking the package's layouts and its hand-written signatures against it 0.55ms. Roughly 4ms of the 14.5, once. The other ~10ms is Load and Check doing their ordinary work over 256 more declarations, and so is the +3.6ms on eval.

So a C-c C-c in a daemon pays no header cost at all, and never did. Session.eval puts the evaluated forms through Load.program, and a form list with no (import …) in it reads no package and therefore no header. The import runs when the session is created, and again on a C-c C-k whose buffer carries the package's own import line. flan reload is a fresh process per redefinition, which is why its number carries a session's startup cost — it is the CLI's shape, not the dev loop's.

What a redefinition does pay for an imported package is the +3.6ms of checking and emitting against a program with 256 more declarations in it. That is in Check and in Emit.redefinition declaring every sibling; it is real, it is the number to attack next, and no cache touches it.

Two caches, and what a header change mid-session means

Both levels exist now. On disk, keyed on the header's realpath, size, mtime, the full flag list and a format version — the object cache's own convention, and it now lives in the object cache directory rather than a second one beside it. Cold that read is 50ms of clang and parse; served from the file it is 0.33ms.

In the session, two tables in Cimport: the extracted dump by header, and the generated declarations by header and by what the package already declares (names taken, structs and enums known, C symbols bound by hand), so a package an evaluation has added a decl to is worked out again rather than served a stale answer. A repeat import in one process goes from 3.65ms to nothing — measured on a whole-buffer evaluation, which imports twice.

A header edited mid-session does not take effect until the session is restarted, and that is deliberate rather than incidental: the in-memory key is the header's path and flags, with no mtime and no stat. It is the rule a changed .c file already follows — the daemon compiled the package's C at startup and a redefinition does not recompile it — and the rule the running program follows, since its struct layouts are the ones it was built with. Taking a new header mid-session would mean type-checking new signatures against a process still running the old layouts, which is the silent disagreement the whole header check exists to catch. The disk cache does key on mtime, so the next session reads the new header rather than the old dump.

One attribution to not repeat: the disk cache is not what the dev build's +333ms cold is about. Measured on this machine, a cold dev build of sand.flan is 1.37s without the header and 1.83s with it and both caches cold; clear only the header cache and the same build is 0.92s against 0.83s. So the header read is ~6090ms of a fresh session and the rest of the cold delta is the object cache compiling a shim with 428 wrappers in it, which is the object cache's business and already warm after one build.

What a headless FFI test can and cannot pin

Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.

  • A struct round trip is worthless. Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C writes and reads the same wrong slots and the test passes for any field order. Found by permuting two fields and getting identical output.

  • Axis-aligned geometry cannot pin Vector2. Exchanging x and y is a reflection, applied to the inputs on the way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, passes with the fields swapped — verified by swapping the defstruct, which is what the typedef is now made from. Distances are worse: the reflection does not even reach them.

  • What does pin Vector2 is the rotated camera, because a 90-degree rotation is not axis-aligned and therefore does not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision cases look like they cover it.

  • What pins Rectangle is arithmetic on its fieldsGetCollisionRec computes four numbers from four different field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.

  • Scalars in, fields out is the strongest shape there is, and the Image family is where it was finally available. GenImageColor(4, 2, colour) is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values in four adjacent i32 slots, with no input struct for a permutation to cancel against. That pins Image completely, including that data is present and first; Texture2D could never be pinned that way because nothing without a GPU reads its width, height or mipmaps at all.

  • A non-square image is an axis discriminator. GetImageColor indexes y*width + x, so on a 4-wide, 2-tall image (3,0) exists and its transpose does not: exchange the wrapper's x and y and the read goes out of bounds and answers transparent black. ImageFlipHorizontal against ImageFlipVertical says the same thing twice more.

  • A file is external ground truth, so ExportImage then LoadImage is not the symmetric round trip the rest of the package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the x/y permutation above.

The rule that falls out: make raylib compute something whose answer differs per axis, then verify the test can fail by permuting the fields and watching it go red. A case not verified that way is decoration.

One correction to an assumption that has now cost two lanes a guess: MeasureText is not headless material. It measures with the default font, LoadFontDefault is not exported, and nothing but InitWindow loads it — so with no window it answers 0 for every string. Measured against libraylib.so.550, not reasoned about. GetFrameTime, GetTime and GetScreenWidth/Height are all 0 headless for the same kind of reason. All five are bound, and all five are exercised by running sand.flan and looking, which is the whole of what can be claimed for them.

Packages

lib/load.ml resolves (import rl "vendor:raylib") before the checker runs. The directory is the package; vendor: is a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no collection is relative to the importing file. Importing is a rename: every top-level name the package declares becomes alias/name, and every use of one — in a type, in a body, in a struct literal, in an array length — is rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of declarations whose names contain a slash.

A package may also carry the C it binds to: every .c file in the directory is compiled into the build, and a file named link lists extra linker arguments. Whether those reach the build at all is decided after checking — see below.

A package may be a single .flan file named outright, rather than a directory. That is for the program that is also a library: sand.flan shares the repository root with three other loose programs, so naming its directory would import all four. A file carries no .c and no link file; those belong to a directory.

A package may import a package. The qualification flattens to the inner alias — raylib imported by a package that is itself imported is still rl/…, never sand/rl/… — because a directory reached along two routes has to arrive under one set of names or the checker sees every declaration twice. It is the one decision here that would be hardest to change later, because a qualified name is an identity the layout op and the break loop both resolve by.

A directory is keyed by its real path and read once, so a diamond shares one copy of its bottom package. programs/pkg-diamond.flan is the case that proves it at the level that matters: area/ and draw/ both import shape/, a shape/Box is built inside area/ and passed to a function declared inside draw/. Read shape/ twice and there are two structs both named shape/Box which do not unify, so what proves the dedupe is that the program prints 3 6 20 and not that it compiles.

The same directory under two different aliases is refused, including when one of the two is a package's own import and the other is pages away in the entry file — programs/pkg-alias-clash.flan.

A ring is refused and named. Earlier it was not: the read-once table swallowed the second arrival, so mutually dependent packages appeared to work, and BUILT.md said so. What that cost is a definite package order, which is what the macro expander needs — every defmacro has to be compiled before anything that calls it — so it is now a refusal that names the ring, a -> b -> c -> a, rather than the single import that happened to close it. The two questions are kept apart by two pieces of state: the chain currently being read, and the set already finished. Found in the first is a cycle; found only in the second is the diamond's second route and still a no-op.

Load.t.pkgs comes back dependencies first — the topological order the acyclic rule buys. The declaration list is deliberately unsorted: check.ml collects every top-level name in one pass before it checks any body, so top-level names are order-independent by construction. The order exists for the expander, which cannot work that way.

Visibility is one rule: main is not exported. A package carrying one would collide with the importer's the moment anything imported it, so a program could never be a package; and main is a reachability root, so an imported one would keep everything it calls alive. Writing sand/main is refused at the line that wrote it, with the reason — left to the checker it would be "unknown name", which is true and useless.

Still missing: a package-private marker for anything other than main, which is why rl/get-color-raw is callable. The gap is surface syntax and not load.mlexported is one predicate and the refusal machinery that points at the line which tried already exists, so a second rule is a line. What does not exist is any way for a package to mark a name private, and inventing one is a reader and parser change.

Load used to hand a package's .c files and link arguments to the build the moment it was imported, whatever the importing program did with them. So anything naming vendor:raylib linked libraylib on every target, and on wasm32 that link cannot succeed.

lib/reach.ml answers it from the checked program instead. Start at main and at the globals that run before it, follow every call — including the Handled frames, where a lifted handler clause is reached by address and by nothing else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.

Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and wasm-ld would fail on the symbols rather than on the argument list. So the same walk prunes functions and externs from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a dropped one is a silently different program.

Dev builds are not pruned. What a REPL may redefine next is not a function of what has been called so far.

The filtering happens at the call sites — bin/main.ml, the tests — because Build.executable receives csrcs and lflags from its caller and never sees the import list. Reach.link returns the pruned program and its C and linker arguments together, so a caller cannot take one without the other.

sand.flan is one program

It was two files, and only ever for the reason above: the headless run is the one CI does on native and wasm32, and a program that imported raylib linked libraylib whatever its main did. So the simulation lived in sand-sim/ and both drivers imported it.

Now sand.flan holds the simulation and the raylib front-end, and test/programs/sand-headless.flan imports sand.flan itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches raylib; sand.flan's main is not exported, so the only main is the headless one; and the hash is unchanged on both targets at -O2 and -O0, which is the point. A refactor that moved that number would have moved the simulation.

What still justifies two entry points is smaller and stands on its own: the headless test needs no window and no input on any target. sand.flan could not be that test even natively — with no mouse the grid stays empty and settle and move-grain never run on real data. Measured through the probe: 168 grains painted around row 48, still 168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.

Two claims that got run together in an earlier note, for the record:

  • raylib does not work on wasm — false. It works through emscripten. What is true is that it does not work on the wasi path, which is what the headless table targets, and which has no GL and no browser.
  • a game loop cannot be expressed on wasm — false. The browser cannot be blocked, so a web build drives the loop with emscripten_set_main_loop instead of a while. That is a different main, not a different program. The premise held and the conclusion did not. It is the same main and the same program: -sASYNCIFY answers the same browser fact without cutting anything in half. See "The browser is the third target" below.

Three edits were made to sand.flan's own text when it was ported, and they are language decisions rather than fixes:

  • (defconst gravity 0.05)(defconst gravity f32 0.05). An untyped float constant is f64, velocity is [f32], and there is no implicit widening.
  • (defvar current-color u32)i32. It is an index into colors, and (len colors) is an i32.
  • (defn main []) is unchanged — the short form, as plan.org says.

Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.

Bounds checks — done at milestone 3

at and slice emit icmpbr → cold block → callunreachable; a failure names the source location. Three check sites: at on [n T] (static bound, folded by LLVM for a literal index — and a literal that is out of bounds never reaches emit, check.ml rejects it), at on a slice or string (runtime len), and slice (two comparisons — lo <= hi is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. Build.opts.checks is on by default and not tied to opts.opt, which is what lets the acceptance table run the same programs at -O0 and -O2 with identical checks. The flag is --no-bounds-checks.

The write path is its own case: (set (at arr n) …) lowers through place/Pindex, not through At, so a refactor that split them would break the write check silently. The test covers both.

Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.110.12s checked against 0.120.13s unchecked. Indistinguishable.

Sanitizers, and why hand-written IR does not get them for free

--sanitize builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last of those is not what passing -fsanitize=address to the clang run over the .ll gets you, and the gap is silent.

AddressSanitizer is an LLVM pass, but it instruments only functions carrying the sanitize_address attribute. That attribute is put there by clang's C frontend. Emit writes .ll by hand, so it wrote none, so the pass walked past every Flan function and instrumented flan_rt.c. The measurement that settled it: an out-of-bounds read of a defvar array in a --no-bounds-checks build printed its garbage and exited 0; with an attributes #0 = { sanitize_address } group named on every define, the same program reports global-buffer-overflow in flan.main. Globals are the exception — the module pass redzones them whether or not any function is attributed — which is why the shape of a sanitized build looked right long before it worked.

UndefinedBehaviorSanitizer has no equivalent lever. Its checks are not a pass: the C frontend emits branches to __ubsan_handle_* inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and (<< 1 32) is still unremarked under -fsanitize=undefined. Shift UB, alignment and the f32→i32 cast on NaN are therefore a compiler feature if they are wanted — checks emitted from Emit behind the flag, the same shape the bounds checks already have — and not a flag away. test_sanitize pins both halves with controls: one program that must report and one that must not, so either fact changing is a test failure rather than a discovery.

-fno-sanitize=signed-integer-overflow is the only exclusion, because wrapping is what this language's arithmetic means and without it every program trips on its first +.

The flag deliberately does not force -O0, unlike --debug, whose reason (mem2reg deletes the alloca a llvm.dbg.declare describes) does not apply. The optimiser is half of what is being measured, and bounds.flan proves it: with checks off, its read past the end of a string constant is reported at -O0 and silent at -O2, because an out-of-bounds inbounds getelementptr into a constant is poison and LLVM folds the load away. The program then prints a wrong answer instead of touching memory. Same family as (<< 1 32) compiling to a bare retq.

What ASan covers of the bounds checks' job, since --sanitize --no-bounds-checks is the run that asks. Three of bounds.flan's six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is layout rather than anything about the access: ASan lays a global out as {data, redzone}, so reading before one lands in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is. Measured both ways — silent in bounds.flan, reported as soon as another defvar is declared in front of arr. A reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees out-of-object access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a second net, not a replacement.

The sweep lives on its own dune alias rather than on dune test: a sanitized program links to a statically linked 1.8MB binary, and twenty-eight of them twice over is minutes against the suite's seconds.

Valgrind, and the two questions a sanitizer cannot be asked

dune build --root . @valgrind runs the headless corpus under memcheck: forty-seven programs checked, twelve of them a second time with --no-bounds-checks, in 88 seconds against @sanitize's nine minutes — and that figure includes the compiles, because the programs are built once here rather than twice. Memcheck is 2050x on execution and the corpus is small; the sanitized build's static link was always the expensive half.

It needs nothing from Emit, and that is the entire reason it was reachable. The sanitize_address attribute story above is a story about an LLVM pass that only instruments what the C frontend marked. Memcheck instruments the binary: it never sees IR, never sees an attribute, and cannot tell Emit's output from clang's. Hand-written IR, the runtime's C and libc arrive on the same footing. This is also why MSan was ruled out and memcheck was not — MSan needs every dependency instrumented and raylib settles it, and memcheck needs none.

The two tools are complementary and neither is a superset. Measured on bounds.flan's six deliberate out-of-bounds cases: ASan catches three, memcheck catches zero. Every one of them is a global or a stack array, and memcheck's addressability checking covers heap blocks only — it has no redzone concept for anything else. What memcheck has instead is definedness, per byte, which ASan does not have at all. Believe neither sweep alone.

The control that justifies the sweep. Index 3 of a Vec with len 2 and cap 4 is inside the allocation — every addressability check in existence says it is fine, ASan among them — and was never written. Memcheck reports it, and --track-origins=yes names flan_vec_push's aligned_alloc as where the undefined bytes came from. That is NEXT.md's "ASan does not see uninitialised reads" turned into a test. test_valgrind.ml also asserts a heap overrun that must report, and a Map key with two seven-byte holes that must not — the last being direct evidence for the emitted per-key hash and equality pair walking fields rather than bytes, where maps.flan could only show the consequence.

What --no-bounds-checks actually removes, which is less than its name suggests. check_at and check_slice in emit.ml are behind the flag. A Vec's and a Map's bounds checks are not: they live inside flan_vec_at and the map probe in flan_rt.c, are ordinary C, and run in every build. So the flag lowers the guard on fixed arrays and slices only, and the single way to reach unguarded heap storage from Flan is a slice taken over a Vec — which is what both positive controls do. This retro-explains why unchecked_controls in test_sanitize.ml only ever found anything through fixed arrays.

What a clean run does not prove

The sweep is clean. Enumerated, in the spirit of "three of six is a ceiling, not a measurement of the risk":

  • Globals and stack are outside it. Measured: 0 of 6 against ASan's 3 of 6.
  • Arena storage reused after free-all is not re-poisoned. Measured, and it is the sharpest hole. Round one writes four elements; free-all resets the offset and keeps the pages; round two allocates the same bytes back and reads one it never wrote — and prints round one's 44, with memcheck silent. Nothing told memcheck the storage died, because from malloc's point of view it did not: an arena is one malloc(cap) and free-all is an integer going to zero inside it. So the uninitialised-read coverage the control demonstrates holds for the heap allocator and not for the per-frame pattern the arena exists for. Closing it means VALGRIND_MAKE_MEM_UNDEFINED in flan_arena_proc, which is a runtime/ change.
  • Interior overruns inside a single allocation are invisible by construction. The arena's alignment padding and the gap between its offset and its capacity are one block to memcheck, so a read across a sub-object boundary crosses nothing. The same holds for the Map, whose data is one allocation laid out keys | values | hashes | scratch: a probe walking off the end of the hashes array into the values region is not an error memcheck can see. "Probe overrun at high load" is therefore not clean — it is not observable by this tool.
  • The union aliasing case is not exercised. unions.flan reassigns a payload across cases and copies a union through a struct field, but match is tag-dispatched and the checker enforces it, so reading case A's bytes after writing case B cannot be written in the language. The sweep says nothing about that getelementptr because no program can reach it.
  • raylib and the windowed examples are excluded, so nothing is claimed about them. Under memcheck this matters more than under ASan, not less: memcheck reports on uninstrumented code too, so including them would bury the signal rather than lose it.
  • Both positive controls had to be synthesized. No program in the corpus reaches a state where an uninitialised read is observable. That is what a corpus passing its own acceptance table should look like, but it means the sweep's value is as a regression net from here on, not as evidence that the current runtime was audited and found sound.

test/valgrind.supp exists and contains no suppressions, which is a finding rather than an oversight: the sweep was run with --gen-suppressions=all before the file existed and memcheck produced nothing to suppress — no false positives from the hand-written IR, the arena or zeroed, and no true ones either. The file is the four expected complaints with the reason each failed to appear, and the rule for adding to it: paste valgrind's own generated text, and write above it why the report is not a bug.

One thing the sweep found that is not a memory defect: slurp.flan is not idempotent. Its last section expects a missing file, and its handler barfs that file into existence and invokes retry; run twice, the second run finds the file already there and prints a handler count of 0 where the first printed 1. Running each program plain and then under memcheck is exactly two runs, so this presented as "diverges under memcheck" and was nothing of the kind — reproduced with no valgrind anywhere near it. test_acceptance.ml already cleared the same two filenames for the same reason; test_valgrind.ml now does too.

Why there is no interpreter

Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.

It stopped being an absence when macros landed. A macro has to run at compile time and there is nothing to interpret it with, so the compiler compiles it into a shared object and dlopens it into its own process — see "Macros: the compiler dlopens the program". The decision's cost and its mechanism are the same thing.

The layout, which is the whole backend design

i8..i64 / u8..u64   i8..i64          signedness lives in the ops
f32 f64             float double
bool                i1
an enum             i32
[T] and string      { ptr, i64 }     ptr+len, non-owning
[n T]               [n x T]          inline, a value
(Ptr T)             ptr              opaque pointers
(Option T)          { i8, T }        tag 0 None, 1 Some
a struct            a literal struct, declaration order
Unit and Never      {}

No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:

  • Every slot is an alloca. Reading a local is a load, assigning is a store, and a store of an aggregate is the copy spec-memory.md requires. addr of a local is then just the alloca, and mem2reg removes the ones nobody addressed. test/programs/values.flan pins this down.
  • A place is a pointer, a value is a load from it. (set (.pos c) …) through a (Ptr Cursor) becomes a getelementptr on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.

Non-local exit is lowered explicitly: return, some and a failed bounds check are branches, never platform unwinding, so wasm32 needs no exception proposal.

The reload primitive — dev loop steps 1 and 2, measured

llcld -shareddlopen → call, with no protocol and no daemon. dune test runs it: one function is recompiled into its own object and called inside a process that is already running, twice, with a changed body the second time.

Step Cost
Emit.redefinition below the timer (<0.1ms)
llc -O2 -filetype=obj 1517ms
ld -shared 3ms
dlopen + dlsym 0.04ms

~19ms end to end, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the number from this codebase. For contrast, clang -shared on the same IR is 50ms — the driver is again most of the cost, which is why the dev path skips it. llc and clang are both 20.1.8 here; check that before trusting the .ll, since the driver absorbs IR the bare tools reject.

ld -shared rather than clang -shared for a second reason: a shared object is allowed undefined symbols, and that is the mechanism. What the new module does not define is the whole design:

  • a global is external. This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand's grid would reset on every reload, and "edit the code, keep the sand" is the thesis.
  • every other function is a declare, so a redefined settle calls the host's move-grain rather than freezing a private copy of it.
  • no main. This module is loaded, not started.

Its string constants still come along; omitting them is an undefined @.str.N at link time, and it is easy to miss because a one-function module usually has none. Emit.signature is now the single place a function's LLVM signature is spelled, because a define here and a declare there drift the moment one of them grows a case for Unit or for a slice parameter.

flan_dev.c is compiled into every build, not only a dev one. Nothing in a release build calls into it — the compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the agent package's C refers to it, and a package's C sources are collected whatever main does. Leaving it out of release builds made flan build sand.flan fail at the link with an undefined flan_dev_result_read, which reads as a compiler bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; -rdynamic and the cells are still what --dev means. test_agent.ml links the agent program both ways for this reason.

-rdynamic is load-bearing. A normal executable exports nothing: nm -D calc-me | grep 'flan\.' is empty, so a loaded module's declares would have nothing to bind to. The test passes it through lflags, which keeps it a property of the dev build rather than of every build. dlsym on "flan.bump" works — a dot is legal in an ELF symbol.

Two things about the test are deliberate and are what make it prove anything: both loads happen in one process, since two runs would pass while saying nothing about an in-process swap; and the versions are two paths, since dlopen caches by path and re-opening one would hand back the handle it already had, so the check would lie. And helper is (* x 2) in one fixture and (* x 3) in the other: the second body is dead text, since the module declares helper rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.

String constants are emitted private unnamed_addr, so the module's own @.str.N cannot be interposed by the host's — worth knowing, because with external linkage a redefined function would silently print the old text and nothing would fail at link time. The fixtures each print a literal so that path is actually exercised.

Cells — how a call site follows a redefinition

Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so a dev build routes every Flan-to-Flan call through a cell: a mutable global holding the address of the function that is current.

@"flan.cell.bump" = global ptr @"flan.bump"        ; the host defines it
%p = load ptr, ptr @"flan.cell.bump"               ; every call site
%r = call i64 %p()

Redefinition is then one store. A redefinition module declares the cells external, exactly like the globals, and exposes flan_reload_install() that stores its own body into its own cell — cost below a microsecond, which is what makes a frame-boundary swap a non-event.

The cell load is emitted after the arguments, so a redefinition landing between two calls cannot land in the middle of one.

Four things about this that are not free choices:

  • flan_reload_install is a named function and not an ELF constructor. A constructor runs during dlopen, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose.
  • A redefinition's own body is hidden. Default visibility in a shared object is interposable, and that applies to taking the address too: plain @"flan.bump" inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent.
  • This also fixes the self-call edge, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not.
  • -rdynamic is what exports the cells, so it and cells are one flag: Build.opts.dev, flan build --dev. This is the first time opts means something semantic rather than an optimisation level.

LLVM cannot fold the indirection away — the cell is an external mutable global — and a --dev build of calc-me keeps 46 indirect calls at -O2. The acceptance table now runs values, machine and sand-headless as dev builds as well; the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.

Names that did not exist when the process started

Editing a defvar or a defn is a symbol the host exports. Adding one is not: there is no symbol to bind to and ELF cannot grow one. Those go through runtime/flan_dev.c, which is two lookups and nothing else:

void **flan_dev_cell(const char *name);              /* a new function's cell */
void  *flan_dev_global(const char *name, uint64_t);  /* a new global's storage */

Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A new global's declared initial value travels with it, as a constant the runtime copies on the allocation and ignores on every call after: calloc alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the common case pays nothing for the general one.

The unit is a list of top-level forms, not one function — Emit.redefinition ~fns. C-c C-c passes one name, C-c C-k passes a file's worth, one code path either way. It has to be: v3 of the fixture adds extra and uses it from a redefined bump, and splitting that into two loads would leave a module referring to storage that does not exist yet.

Four rules, each of which is a silent failure if broken:

  • Every lookup resolves before any body is published. Publish first and a caller reaches a function whose slots are still null. Not race-testable, so it is asserted on the emitted flan_reload_install.
  • flan_dev_global refuses a size change. The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
  • Nothing that published anything is ever dlclosed. A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being pointed into, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer, or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the handle value rather than the mapping: dlopen refcounts by path, so re-sending the same bad file raised a count nothing could lower, and the one reference that could was dropped on the floor.
  • The registry never moves. A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable.

The test that separates this from a plausible wrong version is v4, which redefines added — a name v3 introduced at run time. v3's bump is already installed and is not rebuilt, so it picks v4 up only if its call goes through a cell both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still pass and the transcript would read 246 instead of 432.

Sizes are spelled LLVM's way — ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64) — rather than by a layout calculator in OCaml that would have to agree with LLVM's on every target.

The agent — dev loop step 3

vendor/agent/ is a package like any other: agent.flan declares three calls, flan_agent.c implements them, link asks for -lpthread.

(agent/start path)   listen on a unix socket; once, at startup
(agent/poll)         install whatever has arrived; returns how many
(agent/wait ms)      the same, but waits for something first

The split between them is the design. dlopen relocates a module and takes the loader lock — milliseconds, unbounded — so it happens on the listener thread. flan_reload_install is one store per function and must not land while a redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks. The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the loader.

A full ring is refused, at the sender, before the dlopen. Of the three honest answers this is the only one that reaches the person who asked: dropping loses a reload the sender was told was ok, which is the same lie more quietly, and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also stop answering status and abort, leaving the dev loop with no way to reach a program that had stopped listening to it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away, since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says what to do about it — call agent/poll.

The way out of the break loop is _exit, not exit. exit runs the atexit chain and the ELF destructors, which want the loader lock the listener thread may be holding inside dlopen; a program asked to abort would hang instead of dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134 stays because that is what a trap exits with.

wait exists for tests. A test that races the frame rate fails on a loaded machine, so test/programs/agent.flan waits for the reload instead of sleeping past it. It takes two reloads, which is the daemon's actual loop: the first introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.

Two details found by running it:

  • stdout is line buffered, set in flan_rt_init. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found.
  • The reply goes out before the module is queued. The other way round, the game thread can install and the program can exit between the two, and the answer reaches the sender as a connection reset rather than as ok.
  • ok means queued, not installed. The sender does not get to know when the swap happened; only the program knows when it is between frames.

sand.flan calls agent/poll at the top of its loop, which is what step 3 was for. Verified: with sand running under Xvfb, flan reload sand-probe.flan game-draw and one line on the socket, and 455 consecutive frames drew from a body that did not exist when the process started. Building without --dev is fine — there are no cells, so a module is refused on the listener thread and the loop never notices.

flan reload <file.flan> <fn>... [-o out.so] [--new name,...] builds one module the way the daemon will. --new is the names the host was not built with; it is the one thing the command cannot work out for itself, and it is exactly what the session will track automatically.

The session

lib/session.ml is the program as a live thing: the declarations the running process was built from, plus every change accepted since.

Transactionality came for free and needed no machinery. Check.program builds a fresh environment from a declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the llc that follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.

Two things the session knows that no single evaluation could:

  • Which names the running process was built with. It comes from the checked program, not from any accumulated AST, because Check.program prepends the prelude and no AST contains it. Derive it from declarations and println reads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
  • What that process's memory looks like. Three changes are refused with a reason rather than loaded:
Change What it would have broken
a function's signature a cell is a bare ptr; every call site compiled before the change still passes the old arguments through it — and this is now a stopgap, see below
a global's type the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve
a struct's fields the values the process is holding have the old layout
a defconst's value, when the checker consumed it it is in the shape of the program — (defconst rows (/ h c)) decides grid's type before anything else resolves — so no store can reach it
a defenum member :space is erased to an i32 literal in the caller, so it is folded there too

The signature row is the one the plan has moved past. plan.org now says a signature-changing redefinition should make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while existing callers and stored Fn values keep the old version and stay safe, and the session warns at every tracked caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error. Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines (a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message should not be read as the final answer.

A defvar's initial value is deliberately not in that table. Its storage holds live state the program moved past long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same Tast.global record as a defconst, opposite answers, told apart by gconst. The enum comparison runs over declarations rather than the checked program, because Tast.program carries no enums at all — they are erased to i32 in the checker, which is the same fact that makes them unreloadable.

Note what the checker catches on its own: change helper's parameter type and the caller fails to type check first, loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program uses, which is exactly where the silent version lives. The fixtures carry an unused defvar and a C-called defn for that reason.

A defconst the checker never consumed is a different matter and can be changed: it is only ever bytes in memory. A dev build emits every defconst as a mutable global rather than a constant — so LLVM cannot fold a read of it and a module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That is how sand's colors gets tuned live while rows stays refused. Release builds emit constant and get all the folding back; Tast.global.gfolded is what tells the two apart, because nothing downstream of the checker could.

A form typed into a file that is imported as a package is qualified the way the import qualified it. poll in vendor/agent/agent.flan becomes agent/poll, and its call to poll-raw becomes agent/poll-raw — through Load's own qualify_decl, so the rule cannot drift from the one used at import time. Without this the form spliced as a brand-new unrelated name: the evaluation answered ok, and the running program went on calling the sim/settle it already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently did nothing.

It is derived from the file's path and not sent by the editor, which is where this departs from CIDER's ns key: a Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to either.

The accumulated list is the post-Load one, so an evaluated (import …) is spliced as its expansion. Otherwise re-evaluating a file that imports something appends a second import, Load expands it again, and the duplicate-name pass rejects it. C-c C-k on sand.flan's own text is the test.

flan reload <program.flan> <forms.flan> is that path from the command line: a session over the program the process was built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form game-draw and 910 consecutive frames drew it.

Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on every invocation, so if the program file has been edited since the process launched, its idea of which names the host has and what its memory looks like describes a binary that is not running. And Session.eval's origin defaults to <eval>, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass the real buffer path, which is the same key CIDER's eval carries.

The daemon — flan dev

flan dev <program.flan> holds one Session, builds the program, launches it, and listens on .flan-dev.sock beside the source. What it adds over flan reload is that the session persists — a defvar added by one evaluation is part of what the next one is checked against — and that it owns the build, which is what makes its layout rules describe the process that is actually running rather than a guess about it.

The protocol is s-expressions, not bencode. nREPL was the plan and the argument for it evaporated once the client became ours too: there is no CIDER to be compatible with, eval is string-in/string-out with no slot for which form, from which file, and Emacs already has read and prin1. So it is one sexp per message — no parsing code on the editor side, and on this side the parser is the language's own reader, where :op is already a keyword and a payload of Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains newlines. An nREPL front end can sit on the same Session later; it should not have gated the editor.

(:op "describe")                        → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
                                        → (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "defs")                            → (:status "ok" :defs ((name kind signature loc) …))
(:op "close")

defs is its own op rather than more fields on describe, because describe is what an editor polls — it is how the program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer. It is asked once on connect and again after each accepted install. Four strings an editor reads with read and nothing else: eldoc, completion and find-definition want the same three facts about a name. loc is empty where there is none to give, because only Tast.fn carries one — an editor must refuse rather than go looking for the definition itself, which in a program of several files finds the wrong one. Parameter names are not in the Tast, so a signature is step [i64 f32] i64: types only.

The daemon makes its own source path absolute before building, because every location it reports derives from it. flan dev src/game.flan run from a project root otherwise answered src/game.flan:12:7, which an editor can only resolve by guessing which directory it was relative to.

An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage — is accepted and answered with :note "nothing to install" rather than by shipping an empty module. Building one anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.

:file is not decoration: Session.eval's origin defaults to <eval>, so without it every error an editor shows points into a file that does not exist.

Two things the daemon must not paper over, both of which would look like a successful evaluation:

  • The agent socket is chosen by the daemon, not by the program. A program's source has to name some path — sand.flan says /tmp/flan-sand.sock — and the daemon overrides it through FLAN_AGENT_SOCKET before spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it.
  • Delivery is checked. agent/start returning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is not ok becomes an error the editor sees.

It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving.

One process — flan dev is the program, and the compiler is a thread in it

flan dev builds one binary that is the compiled Flan program and the whole OCaml compiler, and execs it. There is no child: /proc/<pid>/exe on the pid an editor launched points at the program the daemon built, and test/test_dev.ml checks exactly that rather than taking the claim on trust.

Who owns what:

thread what runs there
main the program. C's main holds it — macOS needs a window on the main thread
a pthread caml_startup, then Flan.Dev.merged_setup and merged_serve: the compiler and the editor socket
a pthread flan_agent.c's accept loop, as it always was

The program keeps main() and the compiler comes up beside it, rather than the other way round, because that is the shape DISCUSS.md §11 landed on: the agent is already a server inside the program, so the compiler moves into the program's process. It is SLIME's model — you start the image, it serves, the editor connects.

--two-process is the escape hatch, for a machine where the compiler object cannot be built (no ocamlfind, no flan.cmxa beside the binary). It has its own test and it stays.

The editor socket and its wire protocol did not move. Emacs cannot tell the difference, which is what made the merge testable: the whole existing suite is the check.

Two rules hold in one address space and did not before:

  • The game thread must never call into OCaml. A native thread has no safe points, so the collector can never stop it — which is precisely why a frame is never paused, and precisely what one convenient direct call would undo. Requests reach the compiler by being left somewhere and picked up, never by a call out of the frame loop.
  • No OCaml value goes into Flan storage without caml_register_global_root. That is how "the GC does not touch the arenas" stops being true.

A Flan main does not return — Emit ends it with flan_exit and an unreachable — so in one process that call would take the compiler down with a program that merely finished. flan_rt.c has a hook, null in every other build, that the merged entry point uses to flush, close stdout and park. The compiler learns the program is done the way the daemon did: the pipe reads EOF.

The internal socket is gone — the transport, measured

The merge deliberately deleted nothing, so that it could be tested against the shape it replaced. This is the first thing to go: the unix socket between the compiler and the program.

It was a connect, a write and a read that looped back into the same address space. vendor/agent/flan_agent.c's verb table is now handle_line(line, sink *), where a sink is either an fd or a growing buffer, and there are two callers: serve(), which reads a line off a connection as before, and flan_agent_request(), which the compiler thread calls directly. One verb table, not two — every answer is assembled from several pieces (a header, a name, a newline), so the seam had to be the writing rather than the handlers, or they would have been duplicated and would drift. Dev.deliver, Dev.result and Dev.ask were three copies of the same socket dance; they are now three lines over one Dev.request.

Which path is taken is decided by the linker, not by a flag. flan_agent_request is declared weak in lib/dynload_stubs.c, so it is null in every binary that links no flan_agent.o — the flan launcher, flan reload, the two-process daemon, every test — and Agent.request answers None there, which is what makes Dev fall back to the socket. A flag could disagree with reality; this cannot.

Three things had to be right:

  • The ring still has one producer at a time. queue_room checks for space separately from publish's store, which was safe only because the accept loop was the sole producer and was single-threaded. The compiler thread can now ask while the accept loop is serving, so handle_line runs under a mutex. It is held across the dlopen, which is what the accept loop already did to itself by serving connections inline.
  • A delivery is still only queued. The direct call publishes to the ring exactly as the listener did; the install is one store per function on the game thread at a frame boundary. Installing on the spot would be a frame running half in the old code and half in the new, and it would be an easy thing to do by accident here.
  • The OCaml runtime system is released across the call. A delivery is a dlopen — milliseconds of relocation and the loader lock — and holding OCaml's lock through it stalls every other OCaml thread. DISCUSS.md §14's third cost, in the one place this change creates it.

What it is worth, measured (this machine, warm caches, a one-defn redefinition over the editor socket, median of 12):

before after
redefinition, end to end 23.2ms 22.0ms
...of which the build (:ms) 20.4ms 19.0ms
a break round trip (editor socket + one agent ask) 0.069ms 0.019ms

The transport was about 50µs of a 23ms redefinition, and removing it does not move that number. That is the finding, and it is worth more than a speedup would have been: the end-to-end column moved by less than its own run-to-run spread, and --two-process measures 21.6ms — marginally faster than the merged build did before this change. The merge's prize was never latency. It is that the compiler and the program now share an address space, which is what makes the items below deletable at all and what unblocks reading the stopped frame's memory directly. Anyone reaching for an in-process JIT on the strength of "transport is slow" should read this row first: code generation is 19 of the 22 milliseconds.

The test that pins it is a deletion, because a reply cannot say which way it came: test_dev.ml unlinks the agent's socket file once the merged program has bound it, and then runs every evaluation in the file. Unlinking a bound unix socket does not disturb the listener, it makes new connects fail — so if the deliveries still install, nothing connected. The agent still binds it, for --two-process and for a person at a raw socket.

The Emacs client

emacs/flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is most of the work: Flan is s-expressions, so sexp motion, paren matching, beginning-of-defun and indentation are already right. What it adds is Flan's own bracket syntax ([ and { are brackets, not symbol characters — every binding list and every type is written with them), the characters a Flan name may contain (-, ?, /, .), and its keywords.

emacs/flan-dev.el is the client. There is no parser in it, which is the point of the protocol choice: prin1 writes a request and read reads a reply.

C-c C-c the top-level form at point, recompiled and installed
C-c C-k the whole buffer, as one module
C-x C-e the expression before point, evaluated in the running program
C-c C-z / C-c C-q connect (finds .flan-dev.sock upward) / disconnect
C-c C-o the running program's own output, in *flan-output*
C-c C-r a prompt on the running program (*flan-repl*)
C-c C-b what a stopped program is offering, and which to take
C-c C-d what the running program currently defines
C-c C-a the code a name compiled to — amd64, or C-u for the LLVM IR
M-. / M-, where a name is written, through an xref backend

eldoc, completion-at-point and M-. all read one cached defs reply rather than asking per keystroke: eldoc fires on an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a defn just installed completes at once.

The modeline says whether there is a program on the other end, in three states. lost is a daemon that has gone away, which is ordinary rather than an error — flan dev ends when its program does — so the next request reconnects on the socket it was on. Strictly before a send, never after one: a connection that died mid-request may have died after the daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.

C-c C-k sends one module rather than a form at a time on purpose: a defvar and the function that uses it have to arrive in the same load, or the first refers to storage that does not exist yet.

Framing is in bytes and Emacs counts characters. Every length goes through string-bytes and the process is binary, or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload has of them — a bug that would look like a corrupt protocol and appear only for some users. test/test_emacs.ml drives the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a mistake in the framing, in beginning-of-defun over Flan's syntax table, or in the reply reader passes test_dev.ml and fails here.

An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the next time that buffer's evaluation is accepted. Two things had to be right first. The column in a :loc is a byte offset, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and forward-char with it put the marker as many columns right as the line had non-ASCII characters before it. And the daemon numbers lines from the start of what it was sent, so C-c C-c on a defn halfway down a buffer answered line 1 and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the reader skips, so the reply's line numbers are the buffer's own.

An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent success is indistinguishable from silent failure, and beginning-of-defun may well have found a different form from the one point looked like it was in.

Live disassembly — done. C-c C-a on a name writes the amd64 the running program's copy of it was assembled to into *flan-disassembly*; C-u C-c C-a writes the LLVM IR that body was built from. (:op "disassemble" :name … :form "asm"|"ir") is the op.

What makes it possible is that the daemon owns the build: it compiled every module it sent, so objdump -d --disassemble=flan.<name> on the right object is the disassembly and the retained .ll is the IR. Build.shared deletes its own .ll and Build.executable leaves the host's under a name that says nothing about which module it was, so the daemon now writes its own copy beside each .so and keeps the host's as host.ll — ten reloads in, nothing else on the machine still has that text. A table from function name to the last module accepted for it is the whole of the bookkeeping.

What it will not claim is that the code shown is installed, and this is the interesting half. The agent's socket takes a module path and five verbs; none of them reports an address, flan_dev_cell lives in the program's address space, and C-x C-e renders a pointer as <ptr> on purpose — so nothing the daemon can ask would tell it what a cell holds. The reply carries :basis saying which of three things is true, and the buffer prints it above the first instruction:

  • nothing has been delivered for this name, so the cell still holds the host's body — the one case that is certain;
  • a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
  • a module was delivered and the program is stopped — which says nothing either way about whether it installed, since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further installs until it resumes.

From SBCL: offsets from the function's own start rather than addresses into an object, and L0: labels on branch targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When the debug build lands, that is the line to delete.

Transient error overlays — done. The diagnostic ghost text is feedback about the evaluation that just failed, not an annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything. pre-command-hook and not post-command-hook, which fires at the end of the failing command and would clear the overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke in all of them. execute-kbd-macro runs no pre-command-hook under --batch, so the test drives run-hooks — the same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it is Emacs' contract and a test claiming to check it would be checking nothing.

The program's stdout is a pipe into the daemon, and whatever it printed since the last reply rides along with the next one into *flan-output*. Having it arrive with a reply rather than by a separate request is the point: the output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the accept loop's select, not only when an editor asks, and the buffer is capped so a program printing every frame cannot grow the daemon without limit.

C-x C-e — evaluating an expression

A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body into, so the expression is wrapped in a function with nowhere to be called from; the module says run this once by exporting flan_reload_call, and the agent calls it after the install — on the game thread, at a frame boundary, so an expression reading the program's state sees a point the program agrees is consistent.

Nothing is marshalled back, because nothing could be. A Flan value carries no header, so no code at run time can say what it is. The compiler knows the type and renders it there, in the thunk, into flan_dev_result. That is the layout decision's bill, and it is why the printer set is small rather than universal.

It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a dev-only feature must not put a branch in it — so flan_rt.c is untouched and the value is read back over the agent's socket. The read is safe without a handshake because the counter is a seqlock, and it had to be made into a real one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the bare pointer was the bug rather than the ordering. flan_dev_result_read copies into the caller's buffer and checks the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader that loses the race reports the last complete generation and no bytes, so a daemon polling for a new value keeps polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's "has it moved" still means what it meant. Marking the counter odd needs a release fence and not a release store — a release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which is the original bug with more ceremony. And the odd mark is set rather than incremented, because a begin with no end is reachable: the thunk calls begin before it evaluates anything, so an expression that signals is stopped inside that window, and a restart taken from the break transfers past the thunk and end never runs. Incrementing would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has reached a frame boundary.

This renderer is most of println, which is worth knowing before anyone schedules it. plan.org describes a compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints structs, fixed arrays and options structurally, prints a Ptr as its address rather than following it, and bounds depth and length. That is a description of what Session already does for C-x C-e — same walk, same refusals, same three bounds — aimed at flan_dev_emit and the wire instead of at stdout. What println needs on top is a stdout sink, a builtin that takes its printer from the argument's type, and the any/Error dynamic cases, which have no compile-time type to walk. Not the walk itself.

The renderer is a compile-time walk over the type, emitting a piece at a time through flan_dev_emit. Piecewise because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an allocator the language does not have.

big                       18446744073709551615
col                       :blue
(.pos b)                  (V {:x 1.5 :y 0})
b                         (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3)     [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid                  [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]

Details that are decisions rather than formatting:

  • u64 renders in C, with %llu. The language's own i64->bytes is signed, so it used to refuse rather than come back as -1 — but refusing a whole struct because one field is a u64 is much worse, so the runtime got an entry point instead.
  • Strings are quoted and escaped, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is.
  • An enum renders as :name, recovered from the checker's table as a chain of comparisons, because members are erased to i32 before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see.
  • A pointer is never followed<ptr>. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf.
  • Three separate bounds, easy to conflate. depth (4) and span (8) bound the walk, so [100 [100 u32]] does not become ten thousand render sites in one module. The output is bounded once in the runtime — emit truncates at 4K and end appends ... — because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget.
  • A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element.

What still refuses by name: Map, Fn, a type variable.

A caveat inherited from the language, not introduced here: 3.0 renders as 3, indistinguishable from the integer. flan run calc-me.flan "1.5 * 2.0" has always said 3.

An evaluation is not a declaration: the thunk is built against the program and never spliced into it, so describe does not fill up with eval/N for every expression ever typed.

The module is unloaded afterwards, which is the one case where that is safe. The thunk is called directly by flan_reload_call rather than through a cell, and it takes no registry slot — so once it has returned, nothing points into its text and the value it produced has been copied out. It declares that with @flan_reload_transient and the agent dlcloses it. Measured: sixteen expression evaluations retain zero mappings, where each redefinition retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind, and can never claim this.

Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it.

The test that matters is the same expression twice: the fixture increments ticks every frame, so two evaluations must disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.

The REPL buffer

flan-repl.el is a comint-mode buffer whose every line goes through the same eval-expr request C-x C-e uses. No new protocol and no compiler support. Deriving from comint rather than hand-rolling a prompt is the same call as deriving flan-mode from lisp-mode: history, the input ring and kill/yank already exist. There is no subprocess behind it — the "process" is a stub comint needs in order to have a prompt at all.

Three things about it that are decisions:

  • It is program-scoped. A name typed at the prompt resolves against the running program's top-level namespace, so in sand you write sim/settle and not settle. A buffer visiting a package's own file gets the alias applied for it because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
  • RET on a half-typed form opens a line instead of sending it. Balance is checked with the Flan syntax table, so a paren inside a string does not count.
  • A value and the program's output are different things and arrive by different routes. The value is the result of the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply and goes to *flan-output*. Showing them in one place would be convenient and wrong, so there is a test for the separation.

That test is what caught a real bug: the renderer's Unit case emitted () without evaluating the expression, so (println "x") — the most ordinary thing anyone types at a prompt — answered () while nothing happened. A Unit expression is almost always a call made for its effect, and is now evaluated and then reported.

Conditions — step 1: handler-bind and signal

spec-conditions.md §1 and §2, and nothing else yet. They are worth having on their own because neither alters control flow: signal returns Unit whatever it finds, a handler that returns normally leaves the signalling function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists yet, and no signature changed.

(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
  (load-all))

The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing function's own stack, and signal with an empty stack is a null check — which is what §2 asks for. Popping is by frame rather than by count, so restoring what this one displaced is right even if something below it left the stack out of step.

Three decisions worth keeping:

  • A condition's type is a hash of its name, not an index. An index would shift the moment a struct were added, and every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
  • The condition crosses as a pointer, because a handler runs while the signalling frame is still alive and there is nothing to copy. What the clause binds is the condition itself, though — the pointer is a hidden parameter and the name is a slot loaded from it, so a handler passing c to something expecting the struct is not handed an address instead.
  • A clause is lifted into a function of its own. A handler runs from wherever the signal was, so it cannot be a branch in the function that wrote it.
  • A pushed handler frame holds the clause's body address, not a cell. This is a deliberate divergence from plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a particular body. A handler frame is not a Fn value — nothing in the language can name it — and it is live only for the duration of the handler-bind body, so a reload landing while it is on the stack finds the clause it pushed still valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the stack does not observe a redefinition of its own clause; the next entry to the handler-bind pushes the new one. When Fn values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.

Which gives the two refusals, both by the house rule rather than by accident:

  • A handler cannot see the establishing function's locals. That is a closure with an explicit environment, so a reference to one is refused for that reason rather than reported as an unknown name. Globals and the condition are in scope, which is what the accumulation case needs.

What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not outlive the function that established it, so this is spec-memory.md's case 2 — a non-escaping fn capturing by value into a stack environment — and not the escaping closure that plan.org's open decision #5 defers until a concrete use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the biggest usability limit in conditions is not behind the thing that was just deferred.

  • return inside a handler-bind body is refused. The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape as defer inside a block.

The break loop — conditions step 3

Where "a crash kills the program" stops being true. An unhandled error runs a hook instead of rt_die(), on the frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.

flan: unhandled Missing — stopped, not dead.
   0. restart: retry
   1. restart: use-placeholder

Four decisions, each of which is the reason something is where it is:

  • It is a hook, not a call. The loop lives in vendor/agent/, which is an optional package; flan_rt.c is the release runtime and must not depend on something a program may never import. A program with no agent leaves the hook null and dies exactly as it did before.
  • The hook resumes by writing a restart into the transfer channel — the same channel an invoke-restart writes, reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same act, lowered once. Nothing about §6 needed changing to support it.
  • The break loop is the poll loop, run from the error instead of from the frame boundary. That is not a convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop that did not drain that queue would hang C-x C-e exactly when it is most wanted.
  • Installing while stopped is allowed, which contradicts a rule stated above and should. "A redefined function must not be swapped while it is on the stack" is about mid-frame consistency — half a frame of old code and half of new — and there is no frame in progress here. The old body on the stack keeps running; a retry restart calls through the cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of stopping.

A restart frame carries its name now, beside the hash. Matching never needs it — that is what the hash is for — but a break loop has to show someone their choices and nothing at run time can turn a hash back into a name. It is also compute-restarts' data, whenever that arrives.

A choice is validated on the listener thread, against a stack the stopped game thread is holding still, and refused there. Accepting it and discovering on the game thread that no frame offers it would answer ok for something that cannot happen.

The socket verbs are restarts, restart-at <n> [name], restart <name> and abort, and all four are refused with the reason when the program is not stopped — there is no restart stack to walk from a running one. restarts answers <index> <+|-> <name> per line: the index is the identity, and the flag says whether a transfer to that frame has anywhere to land. test/programs/break.flan errors three times — two restarts taken by name, then a shadowed pair where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved by name could pass.

The break loop in the editor — conditions step 3, the other half

The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it follows from one fact: a program stops at a moment nobody asked about. Every other op in the protocol answers a question an editor chose to ask.

So the state is learned twice, deliberately:

  • It rides on every reply, beside the program's output and for the same reason. :stopped t :condition "Missing", or :stopped nil. The likeliest instant for a program to stop is the one just after an evaluation — a body that now errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding out after the echo area had said the evaluation was fine.
  • And a timer asks anyway, once a second, with describe — the cheap op, which is also how the output pipe is drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies that never come says nothing. The timer never reconnects: flan-dev--request reopens a socket a restarted daemon left behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the lost state that exists to be seen. It also skips while another request is in flight — accept-process-output runs timers, so a poll firing inside a read would eat the reply that read was waiting for.

Three ops, and the annotation owns :stopped, not the ops — one place in the daemon decides whether the program is stopped, so the poll and the prompt cannot disagree.

(:op "break")                       → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
                                         :stopped t :condition "Missing")
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
(:op "restart" :name "retry")       → (:status "ok" :restart "retry" :note "accepted; …")
(:op "abort")                       → (:status "ok" :note "the program is exiting; …")

:restarts is positional — innermost first, duplicates kept — because the position is what restart-at takes. :unreachable names the positions that are on the list and cannot be chosen. restart-at's :name is optional and is not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a prompt cannot take a different restart than the one it showed.

break carries only the restart list, because it costs a second round trip to the program and is wanted only by someone about to choose from it.

ok from restart means accepted, not resumed. The choice is validated on the program's listener thread against the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and lets the next poll settle it.

The modeline is a fourth state, flan:stopped(Missing), before live: a stopped program looks exactly like a running one from anywhere else in Emacs. The prompt is a completing-read over the names with require-match, which is exactly right for a closed set the program computed, and abort is the last entry on that same list rather than a second key — it is the thing you pick when none of the restarts is the answer.

flan_agent_poll had to become re-entrant, and that is the one thing here that was a bug rather than an addition. A C-x C-e thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop cached head and tail and wrote tail back at the end, so the outer call rewound the index over everything the nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of breaks rather than a stumble. It now claims each job by advancing tail before running it, and re-reads both indices each time round. Still single-consumer: only the game thread writes tail, nesting included. test_dev.ml evaluates an expression that errors and resumes it, which fails against the old shape.

The agent grew one verb, status, answered in both states — running or stopped <condition>. Everything else the break loop offers is refused while running, and rightly; but the question an editor asks without already knowing had to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.

test/programs/dev-break.flan stops on its first frame, so the daemon meets a program that is already stopped — the state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way round, by installing a step that errors into a loop that calls it, fixes it while stopped, and then resumes: C-x C-e answering while the program sits in the break loop is checked there against the real client, not only in OCaml.

layout — a type's fields, with no program involved

(:op "layout" :type "sim/Cell")  → (:status "ok" :type "sim/Cell"
                                       :fields (("heat" "f32") ("next" "(Option sim/Cell)")))
                                 → (:status "error" :message "Missing is not a qualified name; …"
                                       :candidates ("a/Missing" "b/Missing"))

The daemon can answer this with nothing running. A layout is a fact about the build, and the daemon owns the build — Tast.structs is sitting in the session it compiled the process from. That is why the conditions buffer can name and type a condition's fields while every one of their values stays refused: the shape is knowable and the contents are not, and drawing them apart says more than drawing neither.

The type is a name, and the name is the qualified one. This was the open question — a class name is not an identity, and two packages each declaring Missing would leave the daemon unable to pick. It turned out to need no new machinery: Load.qualify_decl rewrites Defstruct (n, …) to Defstruct (alias/n, …) at import, so by the time anything reaches Tast.structs the names are one flat namespace in which a collision cannot exist. The name is the type id, with no table to keep in step across a reload, and the existing spelling of a type — Types.to_string — already prints it.

And the break loop was already speaking it. Emit.struct_name_of takes Types.Named n — the qualified name — and passes it to flan_error; the agent holds it in condition_name; break answers it as :condition. So the string the conditions buffer already had in hand resolves as :type by construction, and test_dev.ml round-trips exactly that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat worth writing down — condition_name is a char[128], so the round trip holds for names up to 127 bytes and a longer one is truncated and will not resolve.

A bare name is refused, not resolved, even when only one struct's last segment matches it. Resolving a unique suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot rely on. The refusal carries :candidates, so a person is one copy-paste from the answer and a client has its completion list — the same shape as package_of refusing a directory imported under two aliases rather than picking one. An enum is refused by kind (X is an enum, not a struct): its members are erased to i32 before Tast.program exists, which is the same fact that makes a defenum unreloadable. A union is refused the same way and for its own reason — it is declared, and union values are milestone 6. Both are Types.Named at a use site, so falling through to "no struct is named X" would say a type does not exist about one that plainly does.

render.ml is not reused, and that is not a second walk. It walks a value and emits the code that prints it; this describes a type and emits text. What is shared is the spelling: field types go through Types.to_string, which is what defs spells a signature with, so (Option T), [T], [n T] and (Ptr T) read the same in a layout, in a signature and in the source. A field that is itself a struct shows its qualified name — which is a :type this op accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude structs are answered like any other, because Render resolves against the same list and an editor that could see a Split printed but not ask about it would be the two disagreeing.

On the Emacs side flan-cnr-layout makes the request and flan-cnr-show passes the result into flan-cnr-state-from-reply, which stays a function from data to data so the fixture-driven tests keep working without a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and turning C-c C-b into an error would take away the restarts — the decision the buffer exists for — over a missing annotation.

The shadow stack, and backtrace

plan.org's "Dev vs release builds" table has had Frames: shadow stack in the dev column since the project began and nothing had ever built it. This is it. It is what (:op "backtrace") is made of, and it is dev-only, so a shipped game pays nothing — the same bargain the indirection cells already make.

Chosen over the DWARF route deliberately, and the author's reason is the one that decides it: the more a break loop can show, the less often a real debugger is needed. DWARF buys frames in lldb; this buys them in the break loop, where someone already is when they want them.

A frame is four words on the calling function's own stack — the frame it displaced, a pointer to a static description of the function, and two words reserved for locals. The description is per function and not per call, because nothing about a function changes between two calls to it: the qualified name, file:line:col as Loc spells it, and how many slots the frame has. So the name in a backtrace comes off the frame itself and needs no debug information, no symbol table and no agreement with the optimiser about what a stack frame looks like. It is the same mechanism on wasm32, which is the other reason it is not .eh_frame: plan.org lowers every non-local exit explicitly rather than through platform unwinding, so there is no unwinder here to borrow.

The compiler emits the push and the pop inline rather than calling into the runtime. This is on every call in a dev build, and a call made to record a call would be most of what it costs.

The pop is at every ret, and the transfer path is the one that matters. There are five: an explicit return with a value and without, the none arm of (some x), the tail of the body, and the landing block a transfer leaves through. emit.ml funnels all five through one ret, because the way to get this wrong is to write the pop on the normal path and not on the other one — and then every handled error leaves a dead frame behind, and the backtrace after the fifth one is five frames of fiction. test_dev.ml takes five breaks and resumes all of them by condition transfer before asking for a backtrace, and the answer has to be two frames. Same lesson with-allocator learned about restoring at the pad.

A plain global, not a thread-local. It matches what the handler stack and the restart stack in flan_rt.c already assume: one thread runs Flan, and the listener thread runs C and the loader and never enters a Flan body. If the language grows threads this becomes thread-local and the compiler's two stores become TLS-relative, which is the whole of the change.

The chain is snapshotted on the stopped thread, into the same snapshot the restarts are copied into, at the same moment and for exactly the same reason: the break loop polls, a poll runs Flan, and a chain read by the listener thread is a chain that can be popped underneath the reader. Names are copied as bytes rather than kept as pointers, because a transient C-x C-e module does get dlclosed and its rodata with it. Frame addresses are kept beside the text, because reading a frame's locals means going back to that frame and not to whatever is at index 2 by then.

(:op "backtrace")  → (:status "ok" :frames (("fetch" "/game.flan:15:7" "program" 0)
                                            ("main"  "/game.flan:27:7" "program" 0))
                                   :more 0 :stopped t :condition "Missing")

Innermost first. :more is how many frames deep recursion left off the end — the innermost ones are what the question is about. origin is "program" or "eval": a break inside a C-x C-e thunk has that thunk's frames on top of the program's, and "where is my program" answered with eval/7 is true and not the question. The boundary is recorded at the call, in flan_agent_poll, exactly where restart_floor is and for the same reason — except that its out-of-a-thunk value is -1 rather than 0, because zero restarts on the stack is a real answer and zero frames belonging to the program is not.

Refused while the program is running, like every other break verb. The chain is the game thread's and it is pushed and popped on every call; a walk of it from the daemon would have the shape of a backtrace and the contents of a race.

What it costs, measured rather than assumed. Two benchmarks, one compiler built per variant, every binary kept and then run alternately. The figure is the minimum of nine runs, because this machine is shared and a mean measures whatever else was running; the whole table reproduces to three digits on a second pass.

Dev build, -O2 no frames frames frames + slots
2000 sweeps of a 100×100 grid — sand's inner loop, in miniature 30.0 ms 40.0 ms (+33%) 48.3 ms (+61%)
fib(30) plus 20M calls in a loop — nothing but calls 28.7 ms 31.1 ms (+8%) 35.1 ms (+22%)

Per call that is about 0.1 ns for the frame and 0.3 ns for the frame and the slot table together; per sweep of the grid, 5 µs and 10 µs, which is 0.03% and 0.06% of a 16.6 ms frame at 60fps. The dev loop's premise is that redefinition does not stutter a running game, and a sixteenth of a percent of a frame does not. A dev build is already deliberately slower than a release one — every call goes through a cell, every index is checked, no defconst folds — and this joins that list rather than starting a new one.

The chain's shape was chosen by measurement and not before it. An array with a stack pointer — no alloca, no address escaping — was built and timed as the obvious alternative and is worse on both benchmarks: the frame record on the calling function's own stack is already hot, and an indexed store into a megabyte of BSS is not. It also has a fixed depth, which a chain of stack records does not.

Locals of a stopped frame

The half the shadow stack was built for. (:op "locals" :frame N) answers what a stopped frame's named locals hold.

(:op "locals" :frame 0) → (:status "ok" :frame "look"
                              :locals (("n" "i64" "3") ("label" "string" "\"hello\"")
                                       ("p" "Point" "(Point {:x 1.5 :y 2.5})")
                                       ("xs" "[3 i32]" "[ 10 20 30]") ("flag" "bool" "true"))
                              :refused (("after" "not bound yet at the point the program stopped")))

Nothing is copied out of the program, and nothing could be. A Flan value carries no header, so bytes read from another process would be bytes with no meaning. What the daemon has instead is the typeTast.fn.slots, from the build it owns — and the name beside it in snames, which was already there for the DWARF work. So it compiles a thunk that renders those types at those addresses, in the program, on the stopped thread, and reads the text back through the same seqlocked result buffer C-x C-e uses. The only fact that comes from the running program is where the frame is. That is render.ml's existing walk with its root changed: Render.render over (Deref (Ptr T) (flan/dev-slot f i)) instead of over an expression — the pointer-rooted render thunk NEXT.md said locals needed, and it turned out to need one new arm in the whole backend (a pointer-to-pointer cast, which under opaque pointers emits nothing).

A slot's entry is its address, and null until it is bound. That is the whole of the liveness answer: there is no analysis, no PC-to-scope map and no bitmap. The store that binds a slot stores the address, so a slot the program has not reached yet reads as null and is refused by name. Without it, (let [after 99] …) sitting past an error would render whatever the stack held, and a slice or a struct of garbage does not misprint — it faults, on the game thread of a program that is already stopped, which is the worst moment this project has to offer. The daemon asks the program which slots are bound before it builds the thunk, so the set it emits code for is the set the thunk will resolve.

Only named slots are recorded, and this is where most of the cost went. A slot whose address is stored anywhere escapes, and an escaped alloca is one mem2reg cannot promote — so recording a slot is paying for it in every call to that function, for ever. The slots that would hurt most are exactly the ones with nothing to show: dotimes's hidden bound, the temporaries (min) and (max) evaluate their operands into, the render walk's own scratch. They keep their promotion and are refused by name (s4, "a slot the compiler made up") rather than shown under an invented one. Recording every slot instead was built and timed and came out inside the noise on both benchmarks, so the rule stands on what it shows rather than on what it saves.

Shadowing is right here, and that is not an accident of this design — it is the thing the DWARF route still owes. check.ml's fresh_slot only ever allocates, so (let [v 22] …) inside (let [v 11] …) is two slots, both named v, and both appear with their own values. lldb answers p v with 11 in that program and will until a !DILexicalBlock per Let exists.

Four refusals, each by name and with its reason. Three per slot — invented, not yet bound, and no printer for the type (a map, a function value, a type variable; the arm exists, no program the checker accepts has reached it yet, so it is written and untested) — and two whole frames: one belonging to a C-x C-e thunk, whose Tast the session does not keep, and one whose body is not the body this session holds.

The superseded frame, and why a count could not find it. Installing while stopped is deliberately allowed — it is the fix-it-and-retry loop — so the frame on the stack and the body the session holds can be two bodies of one function. A slot count catches a body that gained or lost a binding and nothing else; the case that matters is a rename, which changes neither the count nor the types, and which would otherwise show every new name against the old body's storage with nothing saying so. That is the confident wrong answer, in the one place someone is working out what went wrong.

So each %fninfo carries a slot fingerprint: Emit.slot_fingerprint over every slot's name together with the spelling of its type, since either can change on its own. The frame carries the value for the body it was compiled from, flan_dev_frame_slotsig reads it, the agent snapshots it with the rest of the frame and puts it on the backtrace line, and Dev.locals recomputes it from the body it holds and compares. Different means refused by name — this frame's body was redefined since it was entered, so its names no longer describe its values — rather than answered. The count check stays in front of it because its message is the more specific one.

Computed in emit.ml and read from there by dev.ml, so there is one definition of it and the two ends cannot drift. It stays off the wire: (:op "backtrace") still answers four fields per frame, because a hash is not something an editor can act on and the refusal says the fact in words instead. The same is true of the globals fingerprint that now travels beside it. The bound is worth stating: it is a 30-bit hash, so a collision is possible in principle, and it would reproduce exactly the silent wrong answer this catches — but only between two differing bodies of a function whose qualified name has already matched, since find_fn is what gates the comparison at all.

The first build of this shipped the fingerprint into %fninfo and stopped there — no accessor, no wire field, no comparison — and was left in the tree with its own test red. Four of five hand-offs missing looks exactly like one hand-off dropping a number, which is what the note left behind said it was.

A (Vec T) shows as <vec> and a (Ptr T) as <ptr>, because that is what render.ml already does for them everywhere else: following a pointer a REPL was handed is not a safe thing to do on someone's behalf, and walking a Vec structurally is a walk over storage the frame does not own — (print (as-slice v)) is how that is asked for, and it says at the call site that it borrowed.

Each slot is rendered from its address rather than copied into the thunk first. A copy would be one alloca the size of the slot — 40KB for sand's grid — and the walk only ever shows eight elements of it. The cost is one call to flan/dev-slot per leaf the walk reaches rather than one per slot, which the depth and span caps already bound.

Globals of a stopped stack

The other half, and in this language arguably the more useful one: a game keeps most of its state in top-level defvars and sand.flan holds its entire grid that way, so "what is this program's state right now" was a question with nowhere to ask it.

(:op "globals") → (:status "ok"
                      :globals (("grid" "[4 i32]" "[ 7 5 0 0]" (0 1))
                                ("pressure" "i64" "12" (0))
                                ("label" "string" "\"running\"" (1)))
                      :refused () :skipped ())

Not per frame, and that is the design rather than a layout preference. A global is not part of a frame — it is program state the frame happened to touch — so nesting it under one implies an ownership that is not there and repeats the name once per frame that reads it. So: one section, whose contents are the union of the globals every frame on the current stack references.

The compiler does the choosing. Reach.expr_refs is the walk that already computes what a function refers to — it is how the link drops a package nothing calls — and pointed at one body it answers that body's reference set. Listing all of a program's globals instead would bury the one that matters under the prelude's PRNG state; taking only what the stack reaches is a filter the compiler can apply and a person cannot. Direct references only, with no transitive closure through calls: a callee that reads a global is either on this stack, contributing its own references already, or it is not on it and is not part of where the program stopped.

Each entry carries the frames that touch it, by index, which is what the conditions buffer already numbers frames by. That recovers what per-frame nesting would have told you — "the whole chain is reading this" reads differently from "only the innermost does" — at no cost in duplication. Ordered by the innermost frame that touches it, because a deep stack makes the union large and proximity to the error is what puts the likely culprit on top; ties keep declaration order.

The mechanism is one step simpler than locals. A local is reached by address, and only the stopped program knows where its frame is — hence flan/dev-slot, hence the bound_slots round trip, hence the null-until-bound refusal. A global is reached by name: Emit.redefinition writes a global the host already has as external, so the dlopened thunk binds to the program's own storage and the dynamic linker does the work. Nothing is asked of the stopped thread at all, and there is no not-yet-bound case, because a global's storage exists from the moment the process started. It is Render.render over (Global n) — the same root C-x C-e uses when you type a global's name at it.

A frame that cannot be attributed contributes nothing and says so. An eval frame has no declaration in this session; a lifted handler clause has none of its own; a frame whose body was redefined since it was entered holds a body whose reference set is a claim about different code. All three go into :skipped with the reason, because "the union is incomplete and here is why" and "these are all of them" are different answers and the second one is the lie.

The hole this had, and the second fingerprint that closes it. Emit.slot_fingerprint hashes a body's slots. For locals that is exactly the right cut: identical slots means the names still describe the storage, so the answer is still true. Here it was not, because a body can change which globals it names without touching a single slot — and then this section showed the new body's reference set attributed to the old frame. The values were never at risk; they are read from the program's storage by name. What was, is one frame's membership in the union and the frame numbers beside an entry.

So %fninfo carries a second number beside the slot fingerprint: Reach.ref_fingerprint, over the set of globals the body names — sorted and deduplicated, because a reference set is not ordered and a body that mentions the same two globals the other way round is the same body, where slot indices make the slot fingerprint order-sensitive on purpose. It travels the same path the first one does: flan_dev_frame_refsig, the agent's snapshot, the backtrace line, and Dev.globals_op recomputing it from the body it holds. Different means that frame is in :skipped with its own reason — this frame's body names different globals than the one this session holds — and the rest of the stack still contributes.

Two numbers and not one combined, which is the whole reason this is a second fingerprint rather than a wider first one. They are different facts: a frame whose slots match and whose globals do not has locals that are perfectly readable and attribution that is not, and one hash over both would make locals refuse a frame with nothing wrong with it. So locals still checks the slot fingerprint alone.

It lives in reach.ml rather than emit.ml because Reach.expr_refs is already the walk that answers "what does this body refer to" and is the same walk the union above is built from — the two ends cannot disagree about what counts as a reference. Which names are globals is the caller's to say: the emitter knows the program's globals, and so does the session. test_dev.ml now drives the exact case — a redefinition that binds identical locals and names untouched where the stopped frame's body names pressure — and with the check disabled it fails twice: once on the missing refusal, once on untouched appearing in the union under frame 0.

Conditions — step 2: restart-case and invoke-restart

spec-conditions.md §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a restart-case further out.

(defn fetch [n i32] i32
  (restart-case (middle n)          ; its value if nothing transfers
    (use-placeholder [] -1)
    (retry           [] 7)))

(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
  (fetch 2))                        ; -1

The channel is an out-parameter, as §6 now says: one ptr appended to every Flan signature, written by an invoke-restart and checked after every call. The return type stays what the source says, so the disassembly is the release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's slot and each frame only has to check and return early, which reuses the existing return path and with it §5's defers. Emit.signature was already the one place a signature is spelled, which is what made this a three-line change rather than a hunt.

Every function is transfer-transparent, release included — and that is the ABI, not a stopgap. A cell holds a bare pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and plan.org both now say the later optimisation may stop a function checking the channel, or pass the pointer straight through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform also means redefinition acquires no new refusal class.

The transfer target is the restart frame's own address, not a clause id. This is a correction to what the previous note settled. A static id has to be unique against every module a running program may later load, and a hash is only probably unique — two restart-cases colliding means the inner one silently catches a transfer aimed at the outer. The frame is an alloca in the function that offers it, so its address is exact, and it also says which clause, which is how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was. Re-entering a restart-case then works with nothing extra: each activation allocates its own frames, and §4's "innermost offering the name" is just the order of the walk.

Cleanup happens in landing blocks, one per region. A guard branches to the innermost open one, which pops whatever frames it established and either catches the transfer or forwards it outward:

  • a restart-case's pops its restart frames, compares the target against its own, and either runs that clause or puts the target back and goes on out;
  • a handler-bind's pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, and without it the handler stack would be left pointing into a frame that has gone;
  • the function's own runs its defers (§5) and returns early. errdefer does not run and never could: try/Result is still refused by name.

A single function-wide unwind block would have been wrong for the first two: a call inside a restart-case body would jump straight past the very form that was supposed to catch it.

The channel is cleared before any cleanup runs and put back after. A defer makes ordinary calls and each one is guarded; with the channel still set the first of them would branch straight back into the landing block it came from. Same reason the clause body starts with it null.

flan_signal takes the channel and passes it to each handler, and stops walking once one has written to it. That makes the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot cross a foreign frame" would make restart-case useless. It is also the only such frame: extern is Flan-to-C only and there are no function values yet, so nothing can call back into Flan across one.

Scope, each piece refused by name with its reason and a test on the reason:

  • restarts took no parameters. That covered §1's own load-texture example and skipped argument marshalling and §3's runtime check. Both are in now; see "Conditions — step 3" below.
  • return inside a restart-case body, exactly as inside handler-bind: a bare ret skips the pops.
  • one restart-case offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a choice nothing in the source shows.
  • invoke-restart inside a defer. A defer is the cleanup a transfer runs on its way out, so a transfer starting there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a defer that reaches one through a call is trapped at run time by flan_transfer_fail, because nothing static could see it.
  • no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past everything. There is nowhere to resume, so there is nothing else to do.

(error c), §2. The same walk as signal, and the difference is entirely what happens when the walk ends: signal returns Unit and the signalling function carries on, error has type Never and stops. So only a transfer gets past it, which is why emit puts a guard after the call and then unreachable — and why flan_error cannot be marked noreturn, since it does return, on exactly one path. Being Never is also what lets it stand as a restart-case body's fall-through, which is the shape §1's load-texture example needs. test/programs/error.flan is the unhandled case: it cannot be an outputs row, because it does not exit 0.

flan_transfer_fail covers the ordinary return path as well as the unwind one: a defer that reaches an invoke-restart through a call traps either way, and the message names the rule rather than the path, since the rule is the same.

A lifted handler clause is named after the function it came out ofhandler/step/0/Missing — and is emitted by a redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable nor attributable, and a redefinition of a function containing a handler-bind failed in llc with an undefined value. A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.

test/test_dev.ml drives that path: a third evaluation redefines step to a restart-case whose frame is an alloca in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and crosses probe, which the host was compiled with. Those three do not meet anywhere else.

Two things found by writing it:

  • { ctx with in_handler = true } was a latent bug. ctx.slots and ctx.slot_tys are mutable, so a copy allocates the body's slots into a record the function never sees again and the indices collide. It was harmless only because no handler-bind body in the tests had a let in it. The flags are set on ctx and restored now.
  • test/reload_host.c had to learn the parameter. It calls flan.outer through an __asm__ label, which does not fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere else entirely.

test/programs/restarts.flan runs in the acceptance table at -O2, at -O0 and as a dev build. -O0 is not redundant here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each of those calls goes through a cell.

Conditions — step 3: restarts take parameters

§3's other half, and the half the comparative studies all ask for: use-value and store-value are the two restarts whose answer is not in the program.

(restart-case (middle n)
  (use-value [v i32] (* v 2))
  (retry     []      7))

(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7))   ; 42

The parameters live in a buffer the restart-case owns. The obvious place is the invoker's frame — it is where the values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so the invoking side is gone by then. The invoker stores into the target's buffer while both are still alive, which is the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any other.

What a restart takes is checked at run time, and it has to be. §4 finds a restart by name on a dynamic stack: the invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and invoke-restart compares both against its own before it stores anything. Every frame carries them, parameterless ones included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not redundant with the hash: it is what keeps a hash collision between two different signatures harmless, since a collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well, because the message has to say what was wanted and what was given and neither end knows both:

restarts.flan:79:36: restart use-value takes (i32), given ()
restarts.flan:87:36: restart retry takes (), given (i32)

The arguments are evaluated into slots before the invoke node, not hung off it. Two reasons and both are real. An argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside an argument has to be on the walk Reach and Load already do — InvokeRestart was a leaf to both, and a leaf that grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker. restarts.flan has exactly that function, half, to keep the claim tested.

The break loop can take a restart it cannot fill in, so it is refused. A transfer has two sources: an invoke-restart, which writes the arguments first, and the break loop, which aims the channel at a frame by position and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked unfilled, invoke-restart marks it filled, and a clause with parameters checks the mark before reading. Choosing use-value from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the top of NEXT.md: the answer is a Flan expression, and there is already something that compiles one against the live program.

Lookup stayed by name, and the signature is checked against what it found. flan_find_restart matches the name hash and nothing else, so an inner (use-value [s string] ...) shadows an outer (use-value [v i32] ...) and an i32 is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first frame offering the name") and it is the thing a reader will assume works the other way, so restarts.flan has a case for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments, which is overload resolution on a dynamic stack.

Both of these guards go through fail_block unconditionally, unlike the bounds checks: --no-bounds-checks does not remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or written to a different layout, is memory corruption.

runtime/flan_rt.c gained two message functions and nothing else. The restart frame's first four fields are the ones C declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.

§3's other open point, a report string per clause, is still open and was not settled first as §3 asks. The field and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing that listing anyway.

What is left

  • Editor comforts: completion, eldoc, jump-to-definition, error overlays.

Session identity is the daemon that owns the build. A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default.

Allocators, (Vec T) and StorageExhausted

spec-memory.md's Allocators section is frozen and settles what to build. This is why the built thing is the shape it is, and — separately and loudly — the three places it amends that section plus the one thing it adds to it.

Allocator is a builtin opaque type, so none of milestone 5 was needed

The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and check.ml refuses function values four ways — a written (Fn ...) annotation, a written fn literal, a defn's name in value position, and calling anything other than a named function. Read straight off those lines, containers need function values and the work doubles.

None of the four is anywhere near this. All four are about surface syntax, and a value the compiler builds that no surface form names trips none of them:

  • Allocator is a Types.t case with no user-writable constructor, the way string is a builtin ptr+len. No Flan type names its procedure.
  • The procedure is a C symbol the emitter names.
  • vec-new, push, free, free-all and the rest are ordinary named calls, which check_call already routes through named_call.

The precedent was already in the repo twice: a handler-bind clause is lowered to a function called back through h->fn(condition, xfer) and built as its own Tast.fn, never an Ast.Fn; and a dev build's call loads a pointer out of a cell and calls through it, which is the indirect call the surface language refuses.

What does need milestone 5 is a user-written allocator: "here is my proc, make an Allocator from it" wants a defn's name in value position. make-allocator, allocator-from and allocator are refused by name with that reason, rather than coming back as unknown functions.

An Allocator value is a pointer to the runtime's struct, never a copy of one. That is forced, not chosen. The capability set has to be readable from wherever a container landed, and free-all bumps an epoch every container made from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never fires.

The surface

Name What
(heap-allocator) the general-purpose tier: malloc, aligned, with free
(arena-new bytes) a bump arena over one fixed backing buffer
(arena-destroy a) hands the pages back — see the amendment below
(free-all a) releases everything the allocator holds, retain-capacity
(can-free? a) / (can-free-all? a) the capability set, read at run time
(alloc-epoch a) / (alloc-id a) the counter free-all bumps; the allocator's identity
(alloc-budget a) / (set-alloc-budget a n) a ceiling on live bytes — an addition, see below
(alloc-live-blocks a) "did you forget to free", answered at the tier that can answer it
context/allocator / context/temp the current implicit allocator, and the per-frame arena
(with-allocator a body...) rebinds for a dynamic extent and releases nothing
(vec-new T) / (vec-new T a) / (vec-new) a Vec, against the context or a named allocator
(push v x) / (reserve v n) Unit, both
(at v i) / (len v) the array names, extended — not a parallel pair
(as-slice v) / (as-slice v lo hi) a non-owning [T] view
(clone v) / (clone v a) the only copy; assignment moves
(free v) consumes its argument

Three amendments to a frozen spec, and one addition

1. free-all is retain-capacity, and arena-destroy is the operation that hands pages back. The spec's table has free-all and nothing else. Zig's ArenaAllocator.reset takes a ResetMode of free_all / retain_capacity / retain_with_limit; Odin's arena_free_all is retain-capacity in effect, because its arena is one fixed backing buffer and the call only sets offset = 0. For a frame arena reset every frame, retain-capacity is the normal case and handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being the same does not make a container built before the reset valid, which is the whole point of the trap.

2. context/allocator and context/temp are dynamic variables with save and restore, not extra parameters. The spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other lane working in emit.ml. The dynamic variable is the implementation; the literal reading is deferred and is a performance question (a parameter avoids a global load), not a semantic one.

3. The Vec header is six words in every build, not four in release. The spec fixes the release layout at ptr + len + cap + allocator with the generation word and the epoch dev-only. A layout that changes with a build flag is a layout that can disagree silently across the reload boundary: a redefinition module is built by llc and ld against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag before it can land.

The addition: a budget. flan_allocator grew a ceiling on live bytes, 0 for none. The spec's retry restart is answerable only by a handler that can make the same request succeed, and for a fixed backing store the handler that works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which is exactly what the epoch check catches. The spec names "grows the arena and then invokes retry" as the handler that works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on purpose.

with-allocator is its own IR node because of the transfer path

Save, run, restore — and restore again at the pad. That second restore is the whole reason it is a node rather than a let and two calls. A body that errors, or one a handler transfers out of, leaves through current_pad, and a context allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is precisely where something is about to allocate in order to render a condition. test/programs/allocators.flan asserts that path by taking a restart out of a with-allocator body.

It releases nothing, per the spec: not at the end of a let, not at the end of a function, not at the end of the body. The program proves it by reading the epoch either side.

(Vec T): one type-erased runtime, and the backend learned almost nothing

The element type appears nowhere below the call site. size_of and align_of are produced where the concrete type is known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement (base/runtime/dynamic_array_internal.odin). The backend grew four prims in total:

  • Rt of string — a call into the runtime's C named by symbol, with argument and result LLVM types read off the expression nodes. The container runtime is type-erased and therefore is a list of C entry points, so one arm covers all of them. A Vec argument crosses as its address, which is also what lets an operation mutate the caller's Vec.
  • SizeOf / AlignOf of a type, filled in from the same layout calculator DWARF uses — the one the acceptance test already checks against LLVM's own getelementptr answers.
  • AddrOf of any expression, place or not, because the element a push copies may be computed. The backend already spilled a non-place to a temporary for exactly this.

at and len were already the names for a fixed array and a slice, so a Vec extends them rather than adding a parallel pair — the asymmetry nth was removed for. The value form (at v i) and the place form (set (at v i) x) go through one helper, so they cannot drift apart the way nth did.

A Vec's length and index are i32, like every other length here. Widening indices is one change across every container and not a Vec question.

let has no type annotation — parse.ml settles that a triple binding is ambiguous and that types are inferred — so a local Vec has nowhere to say what it holds and the element type is written at the call: (vec-new i32). This is not the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an ordinary type rather than binding a type variable. Where the context does say — a defvar's type, a return type, an argument — it may be left out.

A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a different reason (below), but the adopt rule is what makes Zero of a Vec a usable value rather than a null deref.

Move-only is a dead set, and it is flow-sensitive at a join

Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing one to a function, binding it, returning it and freeing it are all moves and all reach one place, and the handful of operations that only look at a container (at, len, as-slice, push, reserve, clone) say so. Only a syntactically simple target counts as a borrow — in (len (f v)) the call still moves v.

At an if and at a match, every arm is checked from the state before the form and the union of what they moved survives the join. A flat set is wrong in both directions: it refuses (if c (free v) (free v)), which is legal, and it accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union says.

The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move, not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a move of a binding declared outside the loop is refused, with that as the reason.

What ownership is not transitive through yet, and why each is refused

The spec says ownership is structural — a struct containing a Vec is itself move-only, free recurses into owning fields, and a field cannot be freed on its own. That machinery is the recursive teardown drop brings. Until it lands, three shapes are refused where they are declared, each naming drop:

  • a struct field of Vec type, because the struct copies its header on assignment and nothing records a move;
  • a global of Vec type, because the dead set is per function — two functions each freeing it is a double free nothing could see, and a global read does not go through the move path at all, so even the one-function case would be accepted. Half a rule is worse than none. A global Allocator is a different thing and stays legal: an allocator is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot see the locals of the function that established it;
  • a Vec of a Vec, because the type-erased runtime copies and releases elements bytewise: clone would duplicate inner headers instead of copying what they own and free would drop their buffers.

And a Vec does not cross to C: handing a header that owns storage to C hands out an owner. (as-slice v) as (Ptr T) plus (len v) is the shape that does cross, and the refusal says so.

StorageExhausted went in with Vec, not after it

No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the operation signals (StorageExhausted {.bytes n .align a .allocator id}) with error — whose type is Never — inside a restart-case offering retry. One rule over every allocating operation, which is what keeps push and reserve at Unit, clone at the container, and no signature anywhere growing a Result.

It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every allocating operation, which is the point of having decided it first. Odin's append returns an ignorable Allocator_Error, and its type-erased path returns the old length on a failed reserve; an append that appends nothing and says nothing is the outcome this rule exists to make impossible.

The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:

(let [ok false]
  (while (not ok)
    (restart-case
      (do (set ok ATTEMPT)
          (if (not ok) (error (StorageExhausted {...}))))
      (retry []))))

A handler that frees something, releases a scratch region or raises the ceiling and then invokes retry lands in the clause, the clause falls through, and the while re-attempts the same request. Every argument to the attempt is bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a push was given. With nothing handling it, error stops the program on the frame that erred.

This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site can. The condition is a value struct with fixed numeric fields and no rendered message, because formatting would allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.

The epoch trap, which is the shipping answer to an open question

spec-memory.md leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which allocator a construction used, and with-allocator plus context/allocator are exactly the mechanisms that deny that knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, free-all bumps that counter, and any operation on a container whose recorded epoch has moved traps naming the site. test/programs/stale-region.flan is the case, and the point of it is that v is still in scope, still looks fine, and nothing marked it — which is precisely what a static rule cannot see.

The generation word has no reader. It is bumped on every reallocation, as specified, and the stale-slice trap it exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said plainly here rather than implied by the word's presence in the header.

What this leaves for steps 5 to 7

(Map K V) is built — see below. What is left: drop and with it the transitive move-only rule, recursive teardown, and the refusal to construct a drop-carrying container against an allocator without can-free; (Result T E) and try; generics; the macro expander. And the accumulation pattern(fn [c] (push errors c) ...) over an enclosing Vec — which Vec does not buy: capture does not exist at all, and the spec's captured-Vec-by-pointer rule has never had to exist because every capturable type today is a value type. It is its own item and should be planned as one.

(Map K V), which is Odin's map

Step 4 of the container build order, over the runtime Vec already established. spec-memory.md is followed as written; the two places this departs from Odin are named below, and there is one amendment to the spec and one restriction under it.

The three properties that were the point

Odin's header states them and they are why it was the thing to follow (base/runtime/dynamic_map_internal.odin).

  • Open-addressed Robin Hood hashing at a 75% load factor. No buckets and no per-entry allocation: one block holds every key, every value and every hash. Robin Hood is the part that earns its keep — on insert, an element that is further from the slot its hash wanted than the occupant it is looking at takes the slot and sends the occupant on. Probe distances even out, and a lookup may stop the moment it is further from home than the occupant it is looking at, because no element is ever further from home than one it passed. That early exit is why a miss costs about what a hit does.
  • Cache-line cell packing. A flat [capacity]K array lets one key straddle two cache lines, so a probe walking four slots can touch five lines. A cell packs as many K as fit in 64 bytes and pads the rest, so no key ever straddles a line. Keys, values and hashes are three separate runs, so a probe — which reads hashes, and only then one key — touches hash lines and nothing else until it has a candidate.
  • Pointer-width integers throughout, so no sign extension or masking gets into the probe loop.

Two departures from Odin, both deliberate

There are no tombstones, because spec-memory.md defers removal ("Move-aware lookup, removal, and owned entries are deferred"). A slot is empty or occupied and nothing else, which deletes Odin's backward-shift loop entirely — it is the single largest reason this file is shorter than the original. When removal arrives, that loop is what it costs.

The header does not tag the capacity into the data pointer. Odin stuffs log2cap into the low six bits because its Raw_Map must be three words. This header already carries an allocator, a generation and an epoch, so the tagging would buy nothing, cost a mask on every access, and — the part that actually matters — make correctness depend on the block being 64-byte aligned. Alignment is requested; an arena whose base is not cache-aligned now gives a slower map rather than a wrong one.

The header is six words, 48 bytes, the same as a Vec's and for the same reason: a layout that changes with a build flag can disagree across the reload boundary.

data  len  log2cap  allocator  gen  epoch

The hash and equality pair, and why most key types do not get one

spec-memory.md restricts keys to built-in structural types and makes hashing and equality compiler-provided structural operations. So there is no dispatch to design: a key type resolves to a pair of symbols, passed to the type-erased runtime the way Odin hangs two contextless procs off a Map_Info.

Most key types need nothing emitted. A key whose equality is bytewise and whose bytes are all present — every integer, enum, bool, and fixed array of those — is served by one runtime pair over (pointer, size). Two kinds are not, and the reasons are worth keeping because they are what a bytewise shortcut would have got wrong:

  • a string is ptr+len and its bytes are elsewhere, so two equal strings at different addresses must hash alike;
  • a struct may have padding, whose bytes are indeterminate — two structs equal field by field can differ bytewise — and it may hold a string, which brings the first problem inside it.

A struct therefore gets a pair emitted for it, walking its fields in declaration order and addressing nothing but fields. Two maps with the same key type share one pair, and a struct reached twice through two fields emits one.

Tast.FnAddr is what carries the symbol, and it is not a function value: nothing in the surface language can produce one, name its type, or call through it. It is the same escape the allocator used — the one NEXT.md predicted would work — and reach.ml learned the edge, because a function reached only by address is invisible to the reachability walk otherwise. That was already true of handler clauses.

A pair emitted for a struct is an ordinary Flan function, so its signature ends with the transfer channel like every other. The runtime's typedef spells that pointer out rather than hoping nothing writes through it, and each built-in hasher exists in two spellings — the pointer form that matches the typedef, and the direct form an emitted hasher calls per field, which has no channel to hand on.

The surface

Name What
(map-new) (map-new K V) (map-new a) (map-new K V a) a new map; the pair may be omitted where the context says
(put m k v) upsert, Unit
(get m k) (Option V) — absence is None
(has-key? m k) bool, copying no value — an addition; the spec does not name it
(len m) (reserve m n) (clone m) (clone m a) (free m) extended, not duplicated

has-key? is not in spec-memory.md and is an addition, flagged because everything else here is the spec's. (get m k) answers the same question, but through an Option the caller then has to match, and the common use is a condition. It copies no value, which is also why it is not just get with the result thrown away. The ? suffix follows can-free?.

len, reserve, clone and free were extended rather than given map-shaped names of their own, which is what at and len already did for Vec: one question, one word. reserve's n is entries, not slots — the runtime sizes the block so n still sits under the load factor, which is the only reading of "room for n" that does not reallocate on the nth put.

get builds the Option in the checker, not the runtime, which has no idea what an Option's layout is; keeping it that way is what lets one entry point serve every value type. put and get bind their arguments to slots before the allocation guard, so a retry re-attempts the allocation and not the expressions that produced the key and value.

{K V} is the type spelling and there is no map literal. A bare map form in expression position is a struct literal's field list, and giving the same braces two meanings is what the colon-to-dot change was for. A map is built with map-new and filled with put.

The refusals, each by name

  • A float key — not a milestone question, which is why it is said separately. NaN is not equal to itself, and 0.0 and -0.0 are equal while differing bytewise. There is no equality there for a map to hash.
  • A Ptr, slice, Vec or Map key — would hash an address rather than what it points at, which is a different operation.
  • A fixed array whose elements are not compared bytewise — an array of structs or of strings needs the per-element walk a struct key gets, driven by a loop rather than a field list. Nothing has wanted one, so it is refused rather than written untested, and the shape that does work (a struct holding the array) is named beside it. This is narrower than the spec, which lists fixed arrays without qualification.
  • A move-only value — the refusal (Vec (Vec T)) already carries, for the identical reason: the runtime copies entries bytewise, so clone would duplicate headers and free would leak what they own. Owned entries arrive with drop.
  • Unit as a value — there is nothing to store, and the cell geometry divides a cache line by the element size. Named rather than left to divide by zero, because it is the natural spelling of a set.

StorageExhausted under retry holds over map-new, put, reserve and clone, reusing the machinery that landed with Vec; no operation returns an error. A map is the harder of the two containers for that rule, and map-exhausted.flan is why it gets its own program: a Vec's failing allocation leaves the Vec untouched, whereas a map's growth allocates a new block, rehashes into it, and only then releases the old one — so a failure partway must leave the map exactly as it was, or the retry re-attempts against a half-moved map.

The epoch trap covers the map too, and map-stale-region.flan is separate from stale-region.flan because the two reach the check by different routes: a Vec's operations check on the way in and stop there, while a map's get goes on to call a hash and an equality function through pointers into the block. A missing check there is not a wrong number, it is a probe loop walking released memory.

Is this another Python dict? — measured

The question was asked directly and the answer is that Python's algorithm is fine. What makes CPython's dict slow is that every key and every value is a separately allocated, reference-counted object and hashing goes through __hash__ and __eq__ calls that cannot be inlined. This map stores raw bytes in the block and compiles hashing and comparison concretely per key type. That is most of the gap before any cleverness.

Measured on this machine, i64 to i64, against CPython 3.13's dict on the same workload:

Working set Flan CPython dict
10k entries, 10M lookups (cache-resident) 21 ns/lookup 132 ns/lookup
1M entries, 10M puts + 10M lookups 1.41 s 1.16 s

Six times quicker cache-resident, and slower at a million entries — and the second row is written down rather than left out. Both are waiting on memory there, and this layout waits longer: keys, values and hashes are three separate runs, so a lookup that misses everything takes three cache misses where a compact dict takes two, and the hash run is a full eight bytes a slot. Cell packing buys probe locality, which is a win while the hash run is resident and a loss once nothing is. One byte of metadata a slot — the Swiss-table arrangement — is the known answer and is not built.

The path from 35 ns to 18 ns (the cache-resident floor, at 500 entries) was profiled, and the first two guesses were both wrong: the per-slot cell division and the block-size divisions were each replaced first and neither moved the number. What did: FNV one byte at a time became eight, and a key that is one machine word became one load and one mix with no loop; equality on eight bytes stopped being a call into libc's vectorised memcmp, and copying a value out stopped being a call into memmove; the block geometry stopped being recomputed five times over in a function that ran twice per lookup; and the seed stopped being a five-multiply avalanche on the critical path for mixing the hasher does again immediately after. 64/size is a table, which is Odin's Map_Cell_Info by another route — Odin precomputes it per type because the probe loop must not divide, and here the sizes arrive as ordinary arguments.

What remains at 18 ns is the type erasure itself: a non-inlinable call into the runtime and two non-inlinable indirect calls to the pair. That is the trade spec-memory.md chose deliberately — "It is type-erased on purpose… No generics are involved, and none are needed" — and monomorphisation is what would buy it back, at the cost the spec declined.

Unions, and the tag they carry

defunion parsed and its shape was checked long before this; naming the type (check.ml:312) and constructing a value (:1075) were both refused as milestone 6. They are not any more.

It is closer than the milestone number suggested, and the reason is Option

Tast.arm already carried acase (a case name) and binds (the slots a payload binds to). That is union shape, built and exercised, because Option is a two-case union wearing a special coat and match over it worked already. So check_match grew a second subject rather than a second path: one function decides which case each arm names and what type each name it binds has, and everything after it — the dead-set join across alternatives, the result type, the exhaustiveness check — is the code that was there.

Option was not desugared into a declared union, and that is deliberate: Option is generic and no declared union is. The coat is the part that cannot be taken off until generics exist.

A union is Types.Named exactly as a struct is. One case in Types.t covers both, and which table the name is in is the only thing that tells them apart. That is what let a union be a field, a parameter, a return type, a slot and a copy without a single one of those paths learning that unions exist.

The layout, which is the part that has to be exactly right

%"U"       = type { i32, [k x iA] }
%"U.Case"  = type { the case's fields, in declaration order }

A tag, then room for the largest case, with A the alignment the widest member of any case needs and k the size rounded up to it. Writing the payload as an array of iA rather than of i8 is what makes LLVM align it without an explicit align on a type — and it is what makes the whole thing

struct { int tag; union { ... } u; }

byte for byte. That agreement is the point. A macro is [Form] -> Form, so Form has to be the same bytes in the compiler and in the dlopened macro, and nothing at run time would notice a disagreement.

The tag is an i32 and not an i8. With an 8-byte payload alignment the two cost the same, and i32 is what a C enum field is — the spelling the macro lane will have to write by hand on the other side of the boundary.

Both the union and one struct per case are emitted as named LLVM types, so every reader geps rather than computing byte offsets of its own. Construction is a store of the tag and a store of the case struct through a gep into the payload; match is a load of the tag and a gep the other way. There is one function that knows how a payload is read — case_field_addr — because there are two readers: a match arm's binds and the structural printer.

The layout goes through the same oracle the DWARF section uses: LLVM's own answer for the emitted type, read back as a constant-folded ptrtoint. Two unions are checked, one whose widest case is a pair of f64 and one whose cases are all i32, so the payload size and alignment cannot be constants the test agreed with by accident. The Shape in test/programs/unions.flan was checked against clang's answer for the same declaration in C: 32 bytes aligned 8 with the payload at offset 8, and 40/8 for a struct holding one.

The surface

(defunion Shape
  [Empty
   (Dot  [x f64  y f64])
   (Rect [w i32  h i32])
   (Tag  [name string  n u8])])

(Shape.Rect {.w 3 .h 3})     ; a value: the type and the case, then the fields
(Shape.Rect {.w 3})          ; ZII, exactly as in a struct literal — .h is 0
Shape.Empty                  ; a case with no fields is a whole value, not a call

(match s
  Empty        "empty"
  (Dot x y)    "a dot"       ; positional, in declaration order
  (Rect w h)   "a rect"
  (Tag name n) name)

Construction is qualified and a pattern is bare. The two are not inconsistent: at a constructor nothing says which union is meant, and at a pattern the scrutinee's type already does. The qualified spelling is accepted in a pattern too, since that is how the value was written and writing it again should not be an error. Two unions may therefore share a case name, and that is not refused — refusing it would be a restriction with no mechanism behind it.

Construction needed no change to parse.ml. . is a symbol constituent, so Shape.Rect reads as one name; the struct-literal arm fires on a symbol followed by a map; and the field-access arm needs a leading dot, so the two cannot collide. (Name {.field value}) is one syntax for both, and which it is, is decided against the tables.

Tags are declaration order, so case order is part of the contract

Tag n is the nth declared case, from zero. The consequence is that an all-bytes-zero union is the first declared case with a zeroed payload — which is exactly the rule that makes an Option's zero a None, and it is what makes (defvar u U) and an omitted union-typed struct field mean something rather than nothing. Reordering a union's cases is a layout change, the same way reordering a struct's fields is.

What is refused, and why each

  • A non-exhaustive match. Refused, not defaulted, and the message names the cases with no arm. A match that fell through would have to produce a value of the match's type out of nothing, and the case a union grows tomorrow is exactly the one a reader wants to be told about today. _ is how to say "the rest", written where it can be seen.
  • A case pattern binding some of a case's fields. All of them or none, positionally — binding a prefix reads the wrong field the moment one is inserted above it.
  • Two arms for one case, which is a mistake and never an intent.
  • A union with no cases. No value of it can exist, so a parameter of that type is a function nothing can call.
  • A case field that is move-only, in the same words a struct field already gets and for the same reason: recursive teardown arrives with drop.
  • A union as a map key. The payload past the case in hand is indeterminate, so hashing the blob would make two equal values hash differently. Hashing one properly is a per-case walk driven by a switch — a different shape from the field list struct_key_pair emits, and nothing has wanted it.
  • uninit on a union. This is the one refusal that is not the struct rule. Everywhere else uninit is an opt-out from ZII and the bytes are whatever they were: a garbage f64 is a garbage number. A union's tag steers control flow — a tag no case names falls past every comparison in a match into a block LLVM is entitled to assume cannot be reached. That is the one place where garbage becomes "the optimiser may do anything".
  • (.x u). A union's fields belong to a case, and which case is being held is what the tag says. match is how one is opened, and its arms bind the fields of the case they matched.
  • A global initialised with a case. (defvar g U (U.B {.x 1})) would mean serialising the fields into the payload blob at link time, which is a byte-level encoder this compiler does not have and which could not express a string field at all — that is a pointer the linker has to relocate and a byte array has nowhere to put a relocation. A zeroed global is fine and needs none of it: it is the first declared case.

Recursion, and the shape Form will have

A union that contains itself by value has no finite size, and payload_lay would recurse forever laying one out rather than failing. It does not get the chance: check_finite already walked a union's cases, so (defunion T [Leaf (Node [l T r T])]) is refused with "T contains itself by value, so it has no size — go through (Ptr T)", and so is a pair of unions that contain each other. Through a pointer it works, and that is the shape a Form has:

(defunion Tree [Leaf (Node [l (Ptr Tree)  n i32])])

(defn depth [t (Ptr Tree)] i32
  (match (deref t)
    Leaf 0
    (Node l n) (+ n (depth l))))

Why match's fall-through is still unreachable

The block after the last case comparison is unreachable, kept from the Option path. That is only sound if no reachable program can hold a tag no case names — and none can: Zero is tag 0, which is a real case; every construction writes a tag the checker resolved; and uninit, the one way to get bytes nobody wrote, is refused on a union for exactly this reason. The refusal is what pays for the unreachable.

The diagnostics bug, fixed

(A {.x 1}) on a case of a union said "unknown struct A", because env had no table of case names and could not tell a case from a misspelling. It has one now, keyed both by the full spelling U.C and by the bare C. The full spelling is a key rather than something split out of a dotted name at the use site, because a union's own name can contain a slash — an imported rl/U — and string surgery would own an edge this does not have to.

Printing

render.ml walks a concrete type for print, the REPL inspector and the break buffer's locals, and a union fell through its Named arm to <Shape>. It recovers the case from the tag by a chain of comparisons — the same shape the enum arm already had, and for the same reason: the name is erased before any backend sees it — and reads the fields of that case only. It prints (Shape.Dot {.x 1.5 .y -2.5}), which is what the source would write.

What is left

An imported union is still refused by name at load.ml:312, so a union is file-local. That is not a blocker for the macro expander: the prelude is parsed and prepended into the same flat namespace before collect runs, so a defunion Form in prelude.ml is an ordinary same-file declaration needing no import — verified by declaring one there and matching it from a program. dev.ml's inspector still says "union values are milestone 6" for a stopped frame's locals, and shim.ml's "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates.

Macros: the compiler dlopens the program

"Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing that turns that decision into a mechanism rather than an absence: running a macro at compile time means compiling it and loading it into the compiler's own process. There is nothing to interpret it with and there is not going to be, so Emit.redefinitionBuild.shareddlopen, the reload primitive the dev loop already runs, is pointed at the compiler instead of at a running program.

(defmacro name [args] body ...) is one function, [Form] -> Form. One parameter, the slice of forms written at the call site, which is where variadics come from in a language with no &rest: (len args) is how many were written.

A defmacro is a defn, and there is no Ast.Defmacro

Parse turns (defmacro m [args] body) into (defn m [args [Form]] Form body) and nothing below the parser knows the word exists. The checker checks it like any function, the backend emits it like any function, Reach.link drops it from a program that does not call it like any function. The only thing that makes it a macro is that Macro calls it at compile time instead of the program calling it at run time.

This is also why there is no macro table. Storage was the question the front half deliberately left open, and the answer is that there is none: the macro set is recomputed by scanning the top level for the word defmacro, which is the only place it survives, and the compiled artefact is a .so keyed by a digest.

Form, and the three numbers

A macro's parameter and its result are Form, so Form has to exist on the Flan side: a defunion in prelude.ml mirroring lib/form.ml. It mirrors Form.value and not Form.t — there is no loc field, deliberately. A macro cannot invent a source location, so the unmarshaller stamps the call site's Loc.t onto every node of what a macro returns. That is the structural answer to "keep the source location of the call site attached to what a macro produces", and it is what the queued structured-error work reads.

Case order is tag order, so the list in the prelude is a layout contract and says so. The widest cases are (Str [s string]) and (List [xs [Form]]); a string and a slice are both %slice = {ptr, i64}, 16 bytes at align 8. So the image is { i32 tag, [2 x i64] payload }: 24 bytes, align 8, payload at offset 8, and every case holds its one member at the payload's start, so there is no third offset anywhere in the marshaller.

Those three numbers are asserted, not assumed. test_acceptance.ml's "Form's image format" asks LLVM for each of them through the same ptrtoint-of-getelementptr-through-null oracle the DWARF offsets go through. Alignment needed a probe the oracle did not have: the offset of field 1 in { i8, %"Form" } is alignof(Form), because a struct member sits at the first offset its own alignment allows. Reading [2 x i64] out of the emitted type and concluding 8 would be asserting the layout against itself, which is the circularity that got a _Static_assert rejected for the FFI.

Nothing aggregate crosses to C

The boundary is void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer) — one thunk per macro, written by Emit.macro_thunk. The thunk builds the %slice from (args, n) on the LLVM side, calls the macro, and stores the result through %out.

The correction that matters here is not obvious from the diff. The unions work verified a union's memory layout against clang; that is a different claim from LLVM's calling convention for an aggregate passed or returned by value in hand-written IR, which is not promised to be clang's C ABI for the equivalent struct. Memory is the agreement that actually exists, so pointers and scalars are all that cross. %xfer is the transfer channel every Flan signature carries; flan_macro_call supplies a zeroed one, because a macro that signals with nothing above it to handle it aborts inside the compiler, and the channel still has to be a real slot.

Build.macro_module produces a self-contained .so: the runtime linked in, no undefined Flan symbols, -fPIC on every object including the .ll. Self-contained is what keeps -rdynamic off the compiler's own link. It goes through clang rather than llc + ld -shared, unlike Build.shared, because there are C objects and a libc to find — exactly the part of the driver the dev path skips.

OCaml has no dlopen for ELF (Dynlink loads OCaml), so lib/dynload_stubs.c is the whole boundary: dlopen, dlsym, dlclose, the four-argument call, calloc/free, and a peek/poke family, because OCaml cannot address raw memory and a Form image is written into it one field at a time.

Quasiquote runs before the walk, and that is not a preference

Quasiquote is a desugaring over Form and nothing more: it becomes form-nil, form-cons per item and form-append per splice — the prelude's three form-building functions and no fourth. It is pure, it needs nothing loaded, and Parse.program runs it on the way in, which is what lets the prelude's own macros parse in a process that has not built a macro module yet.

Running it before the expander's walk is load-bearing. A recursive conditional macro's body contains a quasiquoted call to itself; with the quasiquote still standing, the walk would see that head and expand it then and there, against the wrong arguments. Desugared first, that subform is a (Form.Sym {.s "cond"}) and there is no head left to mistake — so the walk needs no idea that quoting exists.

Nesting levels are counted nowhere: not by the reader, which was written that way deliberately, and not by the desugaring. A quasiquote inside a quasiquote is refused by name. Only a macro that writes a macro wants one.

A call inside a quasiquote is output, not a dependency

This is the distinction that is easy to get wrong, and the first cycle test written for this work got it wrong: it quasiquoted, and it was not a cycle at all.

A macro body that calls another macro outside a quasiquote needs that macro compiled and loaded first, because until then the call is a name nothing defines and the body will not compile. That is a compile-order dependency and it is what makes the pre-pass a fixpoint. A macro body that quasiquotes a call to another macro needs nothing: the call is part of what the macro answers, and the answer is expanded again after it returns.

So there are two different ways expansion fails to terminate, and they are different failures:

  • A ring — two macros whose bodies call each other for real. There is no order to compile them in, so it is refused, naming both. test/programs/macro-cycle.flan.
  • A macro that expands into a call to a macro and does not get smaller. That is an ordinary loop, not an ordering problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site. test/programs/macro-spin.flan.

The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a macro never sees a call to another macro in what it is handed.

Hygiene is an escape hatch, not a system

Deliberately non-hygienic, Common Lisp's rule and Clojure's, settled in plan.org's open decision 2. A macro that needs a name of its own calls gensym, which is a prelude function the loaded module runs while it runs. The name is ~g<n>, and ~ is a delimiter now — it opens an unquote — so no symbol the reader can produce contains one and a gensym cannot collide with a name someone wrote. The counter lives in the loaded module rather than in the compiler, which is the one place this departs from the original sketch; a module is dlopened once per compiler process, so it is process-wide in practice.

-linkall, and why the hook could not be installed by hand

Expanding a macro means compiling it, so Macro needs Check, Build and Emit and therefore sits above the parser it feeds. The join is Parse.expander, a ref that Macro fills in at module initialisation.

Nothing references Macro, so without -linkall the linker drops it from every executable that does not name the module — bin/main.exe among them — and a program calling a macro fails with an unknown name. Installing by hand at every entry point was the alternative and it is not viable: lib/session.ml calls Parse.program for C-c C-c and C-c C-k, and test_session.ml drives the session library in-process rather than through the CLI, so the set of places that would need an install call is open-ended and a missed one is silent. (library_flags (-linkall)) in lib/dune is the guarantee instead.

Macro.building is the re-entrancy guard. Build.macro_module goes through Check.program, which parses the prelude, which calls back into Parse.program — and that would re-enter the expander forever. Nothing is lost by refusing to expand there: a macro compiled in round n calls only macros compiled in earlier rounds, and those calls were already expanded before the build was entered.

What it costs

  • A build of a program that names no macro: 50ms, unchanged. The pass scans the top level, finds nothing, and no compiler runs. This is nearly every program, and it is the reason the prelude can grow a defmacro without every build paying a clang driver.
  • A program that calls one: 310ms the first time, 70ms after. The 240ms is the clang driver; the module is cached under the object cache and keyed by a digest of the prelude's source plus the file's defmacro forms, so it is paid once per change rather than once per build. Every flan build is a fresh process, which is what makes the on-disk cache rather than a memo the right shape.
  • A hello-world's binary carries exactly one symbol out of all of this: flan.gensym-n, eight bytes. Reach.link drops unless, form-cons, form-nil, form-append, form-rest and gensym, because nothing reachable calls them.

unless is the proof

plan.org milestone 5 says when, unless, until, cond and dotimes are special forms only until macros land. unless is the first to stop being one — it is now a defmacro in prelude.ml and parse.ml has nothing to say about it — and it was chosen because it is the one the prelude itself does not use. That matters: the prelude is compiled into the macro module, so a prelude macro that the prelude's own functions call would need the expander to compile the thing the expander needs in order to run.

Its coverage is sand.flan, seven calls, compiled through Session in test_session — the in-process path, and the reason -linkall is not optional. Say plainly what that coverage is not: nothing in test/programs used unless before this landed, so macro-unless.flan is a test written after the feature. The corpus written before it is sand.flan and web/examples/control.flan, and both compile unchanged.

break and continue, and the rule that replaced a blanket refusal

Declined once deliberately; NEXT.md's break, and why it was not built records what was settled then and still holds — dotimes gets it free because it desugars to a While, defer is a non-question because it is function-scoped and a break does not leave the function, and both are typed Never as exit and return already are. What stopped it was two things, and both are answered here.

Labels are Odin's, in the head position. (while :outer (< i n) ...), and (break :outer). A keyword there is unambiguous because a loop condition is never one, so one small label function in parse.ml serves while, until, dotimes, break and continue and no form has to count its arguments. until and dotimes were not asked for and cost a line each: until is a While by the time the parser is done with it, and a nested dotimes scanning a grid is the case the label exists for.

It is not a goto. A label names one of the loops the form is lexically inside — the checker resolves it against exactly those and refuses anything else by name — so control can only leave a loop it is already in. That is Odin's restriction and it is what keeps the feature small: there is no arbitrary target, no forward jump, and nothing to say about scopes being entered.

The in_frames question, ruled on

return is refused inside a handler-bind or a restart-case because those forms push frames and pop them on the way out, and a return that leaves would strand them on the stack pointing into a frame that is gone. That refusal is blanket, and correctly so: a return always crosses.

A break crosses only sometimes. A loop written wholly inside a restart-case body has a perfectly good local break, and refusing it would be refusing the common case for the sake of the uncommon one. So the rule here is relative, and it is one list rather than a flag: ctx.loops holds the loops this form is inside, innermost first, with a barrier entry pushed by every construct a jump may not cross. A break resolves by walking outwards; a barrier reached before the target loop is a refusal that names the constructbreak :outer would leave a restart-case, which it may not. A loop nested inside the construct sits below the barrier and is never affected.

The barriers, and why each is one:

  • handler-bind and restart-case bodies — the frames they pushed are popped on the way out, and a br past the pop leaves a dangling frame. The same reason return is refused, scoped to crossings instead of to everything.
  • A restart-case clause body — it runs after a transfer landed, with the form's frames still to be popped.
  • A defer's forms — they are copied into the function's exit paths, where the loop they were written beside is not running. (Unreachable today, because defer is already refused in a loop body for its own reason. Written anyway: the rule is about what the forms mean, not about which other refusal happens to fire first.)

A handler clause is not on the list at all. It is lifted into a function of its own with a fresh ctx, so its loop stack starts empty and nothing in it can name a loop outside it — the refusal falls out of the lifting.

The two rules agree where they overlap, which is the check that the relative one is not weaker: a return is a jump whose target is always outside every barrier, so the blanket refusal is the special case of this one. in_frames is left exactly as it was.

continue and the latch

Tast.While is now expr * expr list * expr list — condition, body, and a latch that runs after the body and before the test. check_dotimes folded its increment onto the end of the body, and a continue branching at the header would have jumped straight past it: the counter would never advance and the loop would hang. So the step is the latch, continue branches to the latch block rather than to the header, and a while has an empty latch that every optimiser folds away. emit_while emits four blocks instead of three.

A labelled continue is the half worth stating outright: (continue :rows) branches to the named loop's latch, so that loop's counter advances and the rest of its body is skipped along with the rest of every loop inside it. It is "start the next iteration of :rows", not "skip the rest of this innermost body". loops.flan asserts exactly that — an outer dotimes whose tail never prints while its counter still runs out.

Tast.Break and Tast.Continue carry a relative depth — how many loops out the target is, innermost first — rather than a name or an id, because that is exactly what a backend already has. emit keeps one entry per While it is inside, the same shape and for the same reason as pads, and indexes it. The invariant this rests on: the checker mints a depth only from its own loop stack, and the two stacks are pushed once per While each. A While the checker inventsalloc_guard's retry and the file-failure retry — is built directly and contains no jump, so its emit entry matches nothing.

Nothing in lib/prelude.ml wants either. Every early exit there is a return from the function — bytes=?, index-of-byte, index-of-bytes, valid-utf8?, bytes->i64 — and a break cannot replace one: it leaves the loop and the function still has to answer. The loop-with-a-sentinel-flag shape that break exists to remove does not appear in the prelude. The two compiler-emitted retry loops in check.ml are that shape, and they are the one place it cannot help: their sentinel is set inside a restart-case body, which is a barrier.

(array 4 rl/Vector2), and the one position with no type slot

[4 T] is the ordinary type spelling and is unchanged. It already works everywhere a type is expected — (defvar points [4 rl/Vector2]), (defn draw [pts [4 rl/Vector2]] ...), a defstruct field, a return. A let binding is the single position with no type slot, and there the brackets are read as what they are in expression position: an array literal of two elements, whose second element is a type name nothing declares. So (let [pts [4 rl/Vector2]] ...) failed with unknown name rl/Vector2, which describes the symptom and not the mistake. It cost 32 hand-written Vector2s in one raylib example.

(array COUNT TYPE) is the answer: a zeroed fixed array, told its count and its element type as plain arguments. (array 4 rl/Vector2) and the type [4 rl/Vector2] denote the same type, so the constructor is assignable to a declaration written the other way and either spelling can be the parameter — array-ctor.flan asserts exactly that.

It is a parser form and not a builtin call, because the second argument is a type and there are no types in the parser's callers. Parse assembles the whole Tarray (len COUNT, TYPE) itself, which is why the count takes a constant's name for free — len is the same function [n T] goes through — and why a non-type second argument is refused by the type reader's own message rather than as an unknown name. The checker resolves it and hands back Tast.Zero, the same node a declaration with no initialiser gets. There is no new backend node and no new type.

(zeroed) was the first proposal and was rejected on how it reads. (zeroed [4 rl/Vector2]) is unambiguous to the parser — a bracket in argument position could be a type there — but to a person it still looks like a two-element vector, which is the exact confusion being fixed. zeroed keeps its existing job: the empty value of whatever type the destination wants, inferred and never written. array is the one that is told.

The prelude's second tier: the functions that return new storage

Everything in the prelude before this was slice-based and allocation-free, and NEXT.md's diagnosis of why was exact: there was nothing to allocate from when it was written. Vec, Map, an arena and StorageExhausted changed that, and this is the tier that follows — 24 additions, of which the twelve that matter most return new things instead of writing into a buffer the caller supplies. The rest fill in the slice family at the element types that were missing, and one of them is a macro.

Three rules hold across all of it, and they are stated once at the head of the section rather than repeated:

  1. The result is owned and the caller frees it. Nothing is released at scope exit — not at the end of a let, not at the end of a function (spec-memory.md). A caller writes (free v), or lets a (free-all a) take the whole region.
  2. The allocator is the context's, and with-allocator is the override. This is the one design decision the spec did not settle by itself. (vec-new) and (map-new) take an optional trailing allocator because the checker builds them and can vary their arity; a Flan defn cannot, so the choice was an allocator parameter on every signature or none. None — (with-allocator a (join parts sep)) is the override, the Vec records the arena, and free and clone never need it named again.
  3. No Result anywhere. Allocation failure signals StorageExhausted under retry, and no allocating operation returns an error, so every signature says what it produces and nothing about how it might fail.

The builder is not a type

Odin's strings.Builder wraps a [dynamic]u8. Here the (Vec u8) already is that and already has push, so the struct would be a move-only wrapper whose only method is the one it wraps. What was actually missing is appending a run of bytes, and append! is that.

It takes a (Ptr (Vec u8)) and not a (Vec u8), and that is not style: a Vec parameter moves, so a by-value builder would be consumed by its first append and refused on the second.

append-i64! and append-f64! are the argument for the whole shape. NEXT.md's "Sharp edges" records that flan_i64_to_bytes and its neighbours render into one static char scratch[64], so two formatted numbers cannot be held at once; these copy out of that buffer before returning, so the hazard ends at the call and a builder holds as many numbers as it likes. strings.flan puts two integers and a float on one line, which is the case that could not be written before.

split answers a (Vec [u8]), and the owning shape is unrepresentable

The fields are slices of the input. That is not a performance choice — (Vec (Vec u8)) is refused outright (programs/vec-of-vec.flan, "copies and releases elements bytewise"), so there is no owning shape to have chosen instead. It follows that the result dies with whatever the input pointed at, which is the same contract trim and split-next! already have.

The rule is split-on-byte's, unchanged: n separators always yield n+1 fields, so an empty input yields one empty field and a trailing separator yields a trailing empty one. That is Odin's allocating strings.split and not Odin's split_by_byte_iterator, which disagree with each other on exactly that input.

Constructing it needed a one-line (defn slices-new [] (Vec [u8]) (vec-new)), because check.ml's vec_new_elem takes the element type as a single bare symbol and [u8] is not one — so a (Vec [u8]) can only be made where the context names the type, and a return type is a context while a let is not. Written down in NEXT.md as a compiler gap rather than worked around silently.

format-f64, and the rounding rule it does not share with printf

f64->bytes is snprintf "%g": six significant digits, exponent notation of its own accord, no precision to pass it. A frame time of 1/60 comes back 0.0166667 and a score past a million 1.23457e+06.

format-f64 returns a Vec, so it inherits neither that nor the shared scratch buffer, and it renders the integer part and the fraction through that buffer in strict sequence — the discipline append-i64! exists to make automatic.

It rounds half away from zero at the last digit kept, which is round-f32's rule and the rest of the prelude's. printf rounds the binary value to nearest-even at the decimal digit, so 0.125 at two places is 0.13 here and 0.12 there. Matching printf would mean pinning a particular libc's answer, and that answer is not the same on every target anyway.

Three lines in it are the ones a plausible version ships without, and each is a separate test case:

  • The carry. 0.999995 at five places scales to exactly 100000, which is not a fraction — it is the next integer. Without the carry it prints 0.100000.
  • The zero padding. The fraction of 1.005 at three places is 5, and 5 is not 005; without the pad it prints 1.5.
  • The sign. It belongs to the number, not to its integer part: -0.5 has an integer part of 0, and i64->bytes of 0 carries no sign.

-0.0 prints as 0.00, because the sign test is (< x 0.0), which -0.0 fails. Past 9e18 the integer part does not fit in an i64 and there are no fractional bits left anyway, so it falls back to %g rather than approximating.

clamp is a macro, and atan2/pow are declares

clamp is the second prelude defmacro after unless, and the reason is the prelude's own objection to wrapping (min hi (max lo x)) turned around rather than dropped. min and max are builtins at every numeric type and there are no generics, so a clamp function is one copy per type — clamp-i32, clamp-f32, clamp-i64. A macro is type-agnostic for free and emits nothing at all. math2.flan makes the same three-word call at i32, i64, u8 and f32 to show it, and counts evaluations to show each argument appears once.

atan2-f32 and pow-f32 inherit sin-f32/cos-f32's caveat in full and not sqrt-f32's: IEEE-754 requires nothing of atan2f or powf either, so they are the third and fourth places in the prelude where native and wasm32 may differ in the last bit. Every case in math2.flan is therefore a value exact in binary — a quadrant boundary, a power of two, a perfect square — and the -O0 run is the one that proves the symbols resolve, since at -O2 LLVM constant-folds a powf of two literals and leaves nothing to link.

What could not be built, and why it is not "no generics"

Four things on NEXT.md's list did not land, and the interesting part is that the reason differs in each case.

  • Map keys and values need a map iterator, and there is none. flan_map_len, _get, _put, _has, _clone, _reserve, _free is the runtime's entire map surface; nothing walks the open-addressed block. One runtime function taking a cursor and one builtin in check.ml to emit the key and value sizes is the whole job, and none of it is a generics question.
  • map, filter, reduce and a comparator sort are blocked on function values, which is sharper than "no generics" and matters because generics alone would not fix it. Types.Fn exists; check.ml refuses it with "a function type is not implemented yet — milestone 5"; there is nothing in the language to pass. The concrete answer is the one that shipped: sort-f32! and sort-bytes! are the second and third sorts in the language, and sum-i32/sum-f32 already are reduce with the + written in.
  • The prelude is never macro-expanded, so a prelude function may not call a prelude macro. Macro.program runs over the file being compiled; the prelude reaches the checker through Check.program's prepend. The call resolves to the macro's underlying defn and reports an arity error, which is why format-f64 writes (min 9 (max 0 prec)).
  • A returned Vec is a move and the dead set spans the function, so an early (return v) on one branch kills the binding at the foot of another. replace-bytes guards its empty needle with an if rather than a when/return for that reason.

The refusal block is down from eight reasons to four

The list at the foot of prelude.ml used to be one sentence — every entry needed to produce bytes that did not exist in its input, and there was no allocator. join, concat, split, to-lower, to-upper, repeat and replace have moved up into the code; string-from-bytes turned out to be the string builtin all along, and (string (as-slice v)) is the round trip, free precisely because the layouts are identical.

What remains is refused for four different reasons, and is now written that way: pad/center for nothing at all except that no caller has asked; format/sprintf for variadics of mixed type; map/filter/reduce/sort-by for function values; map-keys/map-values for the missing iterator.

Tests: programs/strings.flan, programs/format.flan, programs/algorithms.flan, programs/math2.flan, each at -O2 and -O0, and strings.flan also in a dev build — the one that checks a container's recorded allocator epoch, so it is what would catch one of these Vecs being used after the arena under it was released.

defer may be written in a let

The whole of this project's resource-cleanup answer, and NEXT.md records drop and a with-cleanup form as both considered and rejected before it.

defer is a compile-time construct: the cleanup is copied into every exit path of the function. That is why a loop body and a branch stay refused — a loop body's would fire once at function exit rather than once per iteration, and a branch would have to express "maybe registered", which a form copied into every exit path or into none cannot say.

A let is neither. It is not a frame here: its bindings are function slots like any other and nothing is released at scope exit, so a let at the top level of a function body has exactly the function's extent and a defer written in it always registers. It was refused for a reason that does not apply to it.

The rule, stated precisely, because "top level of a function body" is easy to get wrong: a form at the top level of the body may carry one, and so may a form in the body of a let that itself may — to any depth. A let inside a while or an if has the loop's extent or the arm's, and inherits the refusal rather than the permission.

The implementation detail worth keeping: the permission is granted again before every form of a body, never once around the body. check withdraws it as it starts, so granting it once would let the first defer through and refuse the second — and two resources acquired in one let is the case this exists for. defer-let.flan covers exactly that, along with nesting, interleaved registration order across the let boundary, and an early return.

This amends spec-memory.md, which states under "When storage is released" that (defer (free v)) for a let-bound v is "not expressible today" and that no idiom in the spec may depend on it. It is expressible now.

do at the top level of a body has the same extent argument and is deliberately not included: nothing asked for it, and the rule is easier to state and to trust with one construct in it.

Assets are baked in, and the reason it is a compiler feature

NEXT.md decision 1. (embed "brush.png") is a [u8], (embed "brush.png" string) is a string, and (embed-dir "assets") is a [n EmbedFile] sorted by name. Odin's #load and #load_directory are the model (src/parser.cpp, and check_load_directive / check_load_directory_directive in src/check_builtin.cpp); Odin's # is not imported, because an s-expression language already has a head position for a name and these resolve as ordinary named calls exactly the way vec-new and heap-allocator do.

The reason for this shape rather than a build flag is the one that decided it. It is a compiler feature, so it needs no linker arguments and no per-target packaging, and it works identically on desktop and web. That matters more here than it does for Odin, because Load hands out lflags only to a directory package and main is not exported, so a program can never be a package: the single file doing (rl/load-texture "brush.png") is structurally the one file with no link channel at all. The web lane found that hole and did not invent a flag for it. Embedding has no such hole, because there is nothing to tell the linker.

It costs nothing at run time. The bytes reach the program as a Tast.Str node typed [u8], which emit.ml turns into the same private unnamed_addr constant every string literal already becomes, and its escape is byte-exact across the whole 0255 range, so a PNG survives the round trip through the .ll. Bound with defconst at top level an embed-dir is an LLVM constant outright, through emit.ml's const.

A Str node typed [u8], not a Bytes prim over a string. This is the one non-obvious choice. Bytes is identity — emit.ml lowers Types.String and Types.Slice _ to the same %slice — but wrapping the literal in a prim makes the node non-constant, and const then refuses an embed-dir in a defconst with a global's value must be a compile-time constant. Both of emit.ml's string emitters take the bytes and ignore the node's type, so it is the same constant either way and this one is a constant a global can hold.

Two spellings, not one form that changes type with its context. Odin threads a type_hint everywhere and can afford #load("p") to mean a string here and a []u8 there. With structural equality, no implicit widening and no coercion anywhere, the same text meaning two types would be a wart, so string is written down when it is wanted. The site's expectation is a fallback only and nothing depends on it.

The path is a literal and resolves relative to the file the form is written in. Both are Odin's rules and for Odin's reasons: the bytes must be in hand before any value exists, which is what makes the result free; and a path relative to the compiler's working directory would make a package's assets depend on where flan was invoked from, which cannot be right. A missing file is a compile error naming it, never an empty embed — an asset silently absent is exactly the quiet wrongness this removes. An empty directory is not that case and embeds cleanly as [0 EmbedFile].

The directory lookup is a linear scan, and that is the chosen answer rather than the fallback one. embed-find is an ordinary prelude function over a [EmbedFile]. A directory embed is tens of entries whose names sit in cache-warm .rodata; a compile-time perfect hash would be a build-time map with its own failure modes that nothing has asked for, and sort-and-bisect is the next step if a program ever embeds thousands of files — it would not change the type. It takes a slice rather than the array, because an array's length is part of its type and there are no generics, so the call reads (embed-find (slice assets 0 (len assets)) "brush.png"). Entries are sorted by name because readdir order is filesystem-dependent and an unsorted embed would make two builds of identical sources emit different .ll. Non-recursive, files only — Odin again.

The sharp edge, inherited and not widened. The slice points into .rodata, so a store through it segfaults at -O0 and is deleted as undefined behaviour at -O2 — the same trap the prelude's ASCII-case note measures for (bytes "Hi"), and the same one NEXT.md tracks as "writing through a string literal". Nothing here makes it worse and nothing here fixes it; provenance is what would. To get a mutable copy, clone the bytes into a Vec. It is worth saying loudly because an embedded asset is precisely the thing someone will try to decode in place.

What this does not do. sand.flan still calls (rl/load-texture "brush.png"), which hands raylib a path for raylib to open. Pointing raylib at embedded bytes needs LoadImageFromMemory and LoadTextureFromImage in place of LoadTexture — a raylib binding question, not an embedding one — so the flagship program is not yet asset-free on the web. The mechanism it needs is in.

slurp, barf, and the two ways they fail

NEXT.md decisions 2 and 5. (slurp path) and (slurp path allocator) read a whole file into a (Vec u8); (barf path bytes) writes one.

slurp waited for Vec because its result has no length until the file is read, and it obeys spec-memory.md's rule without an exception: no allocating operation returns an error. There is no Result here, no out-parameter and no error code — slurp's type is (Vec u8) and barf's is Unit.

Two failures, two conditions, and the guards nest rather than merge. Allocation failure is StorageExhausted under retry, unchanged and reused. File failure is FileError {.path .op .reason} under retry and use-value. They stay apart because they ask two different answerable questions: the handler that grows an arena is not the handler that supplies another path, and collapsing them would make one handler guess which it was looking at. file_guard in check.ml is alloc_guard's shape built from the same nodes — a while, a restart-case and an error — so the backend learns nothing new.

The restarts are Common Lisp's pair for a file-error. retry for "the file may be there now"; use-value [p string] for "try this other path". use-value is the first restart clause the compiler itself emits with a parameter — typed restarts landed the same session — and its parameter is the path slot the attempt reads, so the clause body is empty. emit.ml's bind_params stores the invoker's argument into the slot, the clause falls through, and the loop re-attempts against the new path. Everything is inside that loop, so a use-value naming a different file re-measures it and re-allocates for its size; the Vec is freed at the top of each turn, which is why a retry does not leak, and freeing a Vec that never allocated is a no-op.

The two forms resolve paths by opposite rules, and it is worth saying in one place. An embed path is resolved at compile time relative to the file the form is written in. A slurp or barf path is resolved at run time by the host, against the process's working directory — these are ordinary values, and one can arrive from argv or from a use-value restart. test/programs/slurp.flan reads "programs/assets/a.txt" only because the suite runs from _build/default/test. Two forms in one section with opposite rules is exactly where someone gets bitten.

The break loop can be offered this restart and cannot fill it in. That is not new behaviour, only a new way to reach it: a break loop chooses a restart by position and has nothing to supply a parameter with, and emit.ml already emits a flan_restart_unarmed guard on every clause that takes one, so taking it refuses with the reason rather than running the clause on a zeroed buffer. slurp's use-value is simply the first such clause the compiler emits — alloc_guard's retry takes no parameters — and it rides the same path a hand-written one does. The acceptance suite asserts the guard is on the emitted IR for both.

On the web, barf signals — every time, with the path in the condition. This is the decision worth restating, because two more obvious answers are both wrong here. A build-time refusal is unusable: Flan has no conditional compilation, nothing in parse.ml or check.ml reads the target, so "isolate this to desktop" is not expressible in source and the refusal would have nowhere to be silenced. A silent no-op is worse than either, because that is how a save file disappears with nothing said. So the program gets a condition and decides — which is the language having something Odin does not. Odin stubs its whole file API on js/wasm to .Unsupported (core/os/file_js.odin) so that importing core:os "panics cleanly", and a panic is not a decision. The restriction was taken; the mechanism was not.

Nothing in the compiler reads the target to do this. The refusal is one #ifdef __EMSCRIPTEN__ in flan_rt.c, which is where the host ABI is already implemented twice. slurp keeps working on the web — it compiles, runs, and reports a missing file honestly — and the bytes a web program actually wants come from an embed.

test/programs/web-files.flan is one source built for both targets, and test/test_web.ml runs it under node rather than asserting the artifact's shape: an artifact-shape assertion would say nothing about the thing the decision bought. The test checks that the refusal is printed with its path and reason, and that the desktop's success line is absent — a silent no-op would have taken that branch.

What the host ABI grew by, and why that much

plan.org names the filesystem as the #1 portability risk — "pack assets, one abstraction, never touch paths" — so the widening is written down rather than assumed. It is three calls and one reader:

Call What it does
flan_file_size(path, n, &out) how many bytes are there
flan_file_read(path, n, buf, cap, &got) fill a buffer the caller owns
flan_file_write(path, n, buf, len) write a whole file
flan_file_fail_reason() which of the four reasons it was

They are POSIX-shaped and Vec-ignorant: no handle crosses the boundary, nothing is held between calls, and each takes a path and answers 1/0 the way every allocator entry point already does. flan_slurp_into — the part that knows what a Vec is — is runtime glue on this side of the ABI, not a fourth host call, so a second target implements three functions and inherits the rest. The reason is a global rather than an out-parameter for the same reason flan_alloc_fail_bytes is: the condition is a value struct on the failing frame's stack with fixed numeric fields and no rendered message.

These do touch paths, which is the widening plan.org warned about, and decision 2 took it knowingly. embed is the half that does not: it needs no host ABI at all, so "pack assets" remains the answer for anything known at build time and slurp is for bytes that genuinely are not.

FileError is one type with a reason field rather than a family, because conditions have no hierarchy today (spec-conditions.md §1) and a family would need one handler clause per member to say "any file error". NEXT.md decision 4's parent link is the answer to that and is not built; when it is, these reasons can become types without any call site changing.

One thing slurp and barf reveal that is not theirs to fix. A handler that wants "try to save, and carry on if you cannot" has nowhere to go: error is diverging (spec-conditions.md §2), a handler returning normally has not answered it, and neither retry nor use-value means give up. web-files.flan calls exit for that reason. A continue-style restart — or handler-case, which unwinds — is what the case wants, and neither exists.

Where build time goes

flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object cache is in, and it is now ~110ms:

Step Cost
frontend: read → parse → load → check → emit <10ms, below the timer
clang on the .ll 60ms — llc does the same codegen in 20ms
clang on flan_rt.c 40ms — now cached, paid once
link 20ms

Every C translation unit a build needs — the host shim and each package's shim — goes through Build.compile_c, which compiles to a .o under $TMPDIR/flan-objcache and reuses it. The key is a digest of the source text, the compiler (its path, size and mtime, so an upgrade invalidates without paying a clang --version subprocess per build), opts.opt and opts.target. The opt level has to be in there: the acceptance table builds the same programs at -O0 and -O2, and an -O2 object must not serve an -O0 build. The object is written to a temporary name and renamed into place, so two concurrent builds cannot see a half-written one.

Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never the cost. The cache is keyed by content, so it never needs invalidating by hand — rm -rf on the directory is only ever a disk-space decision.

The other cheap win is still open: skip the clang driver for the .ll (llc + link directly), worth another ~40ms. It is a subset of the dev path's machinery. Check llc's major version against clang's before relying on it — the emitted IR text is currently absorbed by the driver behind -Wno-override-module, and a version mismatch surfaces as IR parse errors.

The order it was built in, and why that order

Every item here is done; it is kept because the sequencing is the part worth remembering. Decided in conversation: wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and the dev loop is the thesis of the project, so it came first. Staged so each step was runnable on its own — the failure mode being to build a daemon and a protocol before knowing whether the reload primitive worked.

  1. The reload primitive, measured. DoneEmit.redefinition, Build.shared, test/reload_host.c, ~19ms. See the section above.
  2. Indirection cells. DoneBuild.opts.dev / flan build --dev, flan_reload_install, runtime/flan_dev.c for names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above.
  3. The agent, in C. Donevendor/agent/, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above.
  4. The daemon and 5. the Emacs clientboth done, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same Session if something else ever needs to talk to it.

One decision left to settle before step 2, because both change codegen and are painful to retrofit:

  • Do cells cover globals, or only functions? Settled by step 1: functions only. A redefinition module declares every global external, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding a defvar to a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
  • What is a redefinition unit — one function, or a file? A file is much easier to make correct and is what load-file wants anyway; one function is what C-c C-c wants and is where the 16ms number comes from.

Deferred until after the dev loop:

  1. wasm32. Done, with one glued joint. flan build --target=wasm32-wasi produces a module, and test/programs/sand-headless.flan prints 15595743031174623232 under it — the same hash as native, byte for byte, at -O2 and at -O0. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared across targets, and it compares equal. values.flan and machine.flan run there too, which is where a 32-bit pointer would have shown. The acceptance table runs all four, and skips them by probing — it builds the smallest program and runs it — rather than by looking for a binary on PATH.

Three things this cost that were not in the old note:

  • The entry point is not main. wasi-libc's start code calls __main_argc_argv; clang renames C's argc/argv main to that, and the .ll Emit writes says @main literally. The link succeeds and the program traps on a signature-mismatched weak stub. Build.wasm_main_source is a two-line C shim that bridges it, and the __asm__("main") label in it is load-bearing: spelling the callee main makes clang rename that too and the shim becomes an infinite self-call.
  • The target has to reach the C compiles, not just the link. flan_rt.c includes <stdio.h>; without --sysroot it never gets that far. target_flags is computed once and passed to both, and the whole flag list — not just the triple — is in the object cache key, so repointing a sysroot cannot serve a stale .o.
  • Fedora's sysroot is one level deeper than wasi-sdk's: include/wasm32-wasi/stdio.h, not include/stdio.h. Both shapes count.

The glued joint, and the one thing this contradicts in the old note. The old note said the builtins archive has to come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it libcompiler_rt.a, and dropping that in as libclang_rt.builtins.a links and runs. It is a different triple (wasm32-unknown-emscripten) built by a different clang (22 against Fedora's 20), so it is substituting, and wasi-sdk is still the proper article. build.ml looks for FLAN_WASM_BUILTINS, then /opt/wasi-sdk/..., then emscripten's beside emcc on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real include symlinked in.

The runtime is Node. No wasmtime and no wasmer on this machine; test/wasm-run.mjs is twenty lines of node:wasi and the table prefers wasmtime or wasmer if either appears. --no-warnings, because node:wasi writes an ExperimentalWarning to stderr on every run and the harness compares combined output.

Refused by name, not half-supported: --dev with a wasm target (the reload path is dlopen), Build.shared with one (same reason), and flan run --target= (a .wasm is not something this host execs — build it and point a runtime at it).

Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. Done — and it is a third target, not a mode of this one. wasi-sdk is right for the headless table and was never going to be right for the game build. See the next section.

The browser is the third target

flan build --target=web produces a page, its JS and a .wasm, and a raylib example opens in a browser from source that was not touched. The two wasm targets share the word and almost nothing else, so is_wasi and is_web are separate predicates and is_wasm is their union — the union is exactly the set of facts about the machine (32-bit pointers, no dlopen), which is what the refusals are about, and nothing else is shared.

The compiler is emcc, not clang, and that is the whole of the sysroot story. Everything the wasi target has to find by hand — a sysroot, a builtins archive, a shadow resource directory, the __main_argc_argv shim — is what emcc is. target_flags for web is the empty list; the only thing checked is that emcc exists, refused by name where the reason can say so. The one fact that had to be true for any of this: emcc takes a .ll on its command line, which it does, so Emit's output needs no change and the IR stays target-independent. The object cache keys on the compiler binary's path, size and mtime as it always did — now of whichever compiler the target uses, so an emcc flan_rt.o and a clang one cannot collide.

The main loop: -sASYNCIFY, not emscripten_set_main_loop. The older note above had the browser fact right — it cannot be blocked — and drew the wrong conclusion from it. emscripten_set_main_loop wants the loop body as a callback, so every one of the eleven examples that writes

(until (rl/window-should-close?) ...)

would have to be split by hand into an init and a tick, and the web program would stop being the native program. Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is built for precisely that: WindowShouldClose() on PLATFORM_WEB is an emscripten_sleep(16) that then returns false (raylib 5.5, platforms/rcore_web.c, read rather than assumed). So the loop yields once a frame at a call it already makes, and no example changed a character. The price is real and is paid by every web build: asyncify instruments the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a question Build can answer, and a flag set that varies per program is a cache key that varies per program.

link lines can be addressed to a target. vendor/raylib/link named libraylib.so.550, which exists on the host and nowhere else. A line may now carry @native, @wasi or @web, an untagged line applies everywhere — which is what every existing link file already is — and ${NAME} expands from the environment. The selection happens in Build and not in Load, which is where the file is read, because Load resolves imports before a target is chosen: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings. Load's part in this is to pass the lines through untouched, which it already did.

raylib for the browser is built, not installed. No emscripten port provides it (emcc --show-ports: there is contrib.glfw3 and no raylib), so vendor/raylib/build-web.sh clones raylib at the 5.5 tag — the one whose .so.550 the host links, because raylib.flan carries raylib's struct layouts and enum values and two targets built from different raylibs would disagree about them in silence — and compiles the seven modules with -DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2 into one archive under vendor/raylib/web/ (gitignored). rglfw.c is not among them: the web platform uses emscripten's own GLFW port, which is why link carries @web -sUSE_GLFW=3. No headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.

The HTML shell is a string in Build, not a file in the tree, for the same reason Runtime_src is: it has to be wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas and a Module.print that puts stdout on the page; FLAN_WEB_SHELL replaces it. --shell-file is passed only when the output is a .html, because emcc accepts and ignores it otherwise.

Refused by name, inherited whole: --dev, --debug, Build.shared and flan run --target= are refused for web exactly as for wasm32, each naming web rather than wasm32 in the message. --sanitize is refused too, but the web half of that refusal is weaker than the wasi half and says so: emscripten does ship an ASan, and nothing here has ever run it. A sanitizer that has never been run is one whose silence means nothing.

What the test can honestly check. test/test_web.ml is headless and permanently so. It probes — emscripten may not be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it asserts: the three files exist, the module starts with \0asm, the page references its own JS and carries the canvas, and node runs the emitted JS and gets ok. For raylib it builds core-basic-window.flan unchanged and then reads the module for the two things that would be false if the mechanism were wrong: an asyncify_start_unwind export, and a glViewport import that can only have come from raylib's web platform. Import and export names are plain strings in the binary, so this needs no wasm reader.

sand.flan in a browser

The flagship program builds for the browser and the artifact opens. Three things were between it and the target, and none of them was the #include the earlier note named.

The brush was a path, and a path is what cannot work. (rl/load-texture "brush.png") hands raylib a filename to open; a bare relative path has no meaning where there is no filesystem, so raylib would have opened nothing and the cursor would simply have been missing. It is (embed "brush.png") now, decoded by a new binding — LoadImageFromMemory, which completes the chain embedded bytes -> Image -> Texture2D that LoadTextureFromImage already had the other half of. The declaration is (Ptr u8) plus an explicit i32 count, because shim.ml refuses a slice parameter and its refusal says exactly that; the Flan wrapper beside it takes the slice apart, which is the idiom collision-point-poly? and load-font-ex already established. One decode serves both textures now: the unflipped upload first, then ImageFlipHorizontal in place, then the mirrored upload — the order is load-bearing, and if the two badges look the same it was swapped.

load-texture and load-image lose their only call site in this repository by this change. That is deliberate rather than an accident of editing: a path-based load is the one shape the browser cannot have, and the bindings stay for the desktop programs that will want them.

The embedded path resolves relative to the file the form is written in, which is why test/programs/sand-headless.flan still works: it reaches sand.flan through ../../ out of a sandboxed _build, and the PNG is found beside sand.flan and not beside the working directory. test/dune therefore lists brush.png as a dependency of every stanza that builds sand — an embed is read by the checker, so it is a build input and not a run-time one. The same fact reached test_session's C-c C-k case, which re-evaluates sand.flan's whole text: it now passes ~origin, which is the buffer path both editor paths already send, because the default <eval> origin would resolve the embed against the working directory instead.

A package's C may be addressed to one target, the way a link line already could. Load collects a package's .c files by listing the directory, and there is nowhere in a directory listing to put a tag except the name, so the tag goes there, before the extension:

vendor/agent/flan_agent.c        compiled everywhere, unless displaced
vendor/agent/flan_agent.web.c    compiled for the browser, and displaces the above

One rule: a tagged file is compiled only on its own target, and there it replaces the untagged file of the same base name. Untagged is the default and every existing package is untagged, so nothing that did not opt in changed. Replacement rather than the pure tagging Go's _windows.go and Odin's file_js.odin use, and the difference is the point — pure tagging would mean renaming flan_agent.c to flan_agent.native.c to teach the package about a target it had never heard of, and this way a package gains a target by gaining a file. The selection is in Build and not in Load, for the reason select_lflags gives: Load resolves imports before a target is chosen.

The dev agent on the web is a no-op, and it is not the barf decision being contradicted. The compile error that led here — struct timeval incomplete, because emscripten's headers do not pull <sys/time.h> in transitively — is the surface. Underneath: the agent is a socket server and a browser has no sockets. Adding the include produces an agent that compiles, links, starts and can never accept a connection.

Refusing vendor:agent on a web target was the other candidate and is ruled out by arithmetic, not taste. Flan has no conditional compilation, sand.flan calls (agent/start ...) unconditionally, and Reach cannot prune a package something reachable calls into — so a build-time refusal means the flagship program does not build for the browser at all without being edited into a second program. A refusal is only honest when the caller has a way to not ask.

The two decisions look contradictory and are not, and the difference is what the caller loses. barf is asked to make something durable; a no-op returns success to a program that now believes the bytes are on disk, and the loss is real, is the user's, and is discovered later or never. The agent is asked to accept redefinitions from an editor; on the web there is no editor, no socket and no session — --dev is refused by name on every wasm target, so a web build has no cells to install a redefinition into even if one arrived. Nothing is lost because there was never anything there. sand.flan already says the same about a native release build, at the call site: "Building without --dev is fine — nothing has cells to install into, so a module is refused on the listener thread and the loop never notices." A web build reaches that outcome by a shorter route. start returns -1, which is what flan_agent.c returns for a path it cannot bind; poll and wait return 0, which is what the native build returns on every frame nothing arrived on. The whole argument is written at the top of vendor/agent/flan_agent.web.c, where the next reader will meet it.

Building it and opening it

The raylib archive is built once and is not in the tree. From the repository root:

sh vendor/raylib/build-web.sh          # clones raylib 5.5 and compiles it with emcc
export FLAN_RAYLIB_WEB=$PWD/vendor/raylib/web/libraylib-5.5.a

build-web.sh prints that export line itself. Then:

flan build sand.flan --target=web -o sand.html

The output must be named .html. --shell-file is passed only when it is, because emcc accepts and ignores it otherwise — so -o sand produces a module with no shell, no canvas, and a page that looks like it built fine and paints nothing. Three files land beside it, in whatever directory -o names: sand.html, sand.js, sand.wasm.

A file:// URL will not work. The page fetches sand.wasm, and a browser refuses that from the filesystem. Serve the directory holding the three files:

python3 -m http.server 8000

and open http://localhost:8000/sand.html. Left mouse paints; the keys are the ones the native build has.

What only a human opening it can settle

Verified headlessly, by test/test_web.ml: the three files exist, the module carries asyncify_start_unwind and glViewport, and brush.png's own bytes are in the module, whole — the assertion that keeps the embed from rotting. Not IHDR: stb_image, linked in from raylib, carries that string itself, so an IHDR check would pass on a build where the embed emitted nothing.

Not verified, and not verifiable here. node sand.js instantiates the module, runs main, and dies inside glfwInit on window is not defined — which says the module is live and says nothing about the canvas.

  • Whether it paints at all. Nothing in CI has ever seen a pixel of this.
  • Audio. start-audio generates a tone, ExportWaves it to /tmp/flan-sand-tone.wav and loads it back as a music stream. That is raylib's own fopen, not Flan's barf, so the #ifdef in flan_rt.c does not cover it and emscripten's MEMFS may well give it a writable /tmp. Every use is behind music-ok/tone-ok, so a failure is silence and not a crash. Separately, browsers suspend the audio context until a user gesture, so audio-ok may be true while nothing is heard until the first click.
  • The loop never exits. WindowShouldClose() on PLATFORM_WEB is an emscripten_sleep that returns false, so until never terminates and none of main's defers ever run — no CloseWindow, no UnloadTexture. That is correct for a page, which is torn down by the tab closing, and it is worth knowing before reading anything into it.
  • Canvas size against screen-width/screen-height. The shell is a string in Build and its canvas is not sized from the program, so 900x600 may be letterboxed or cropped.

The colon belongs to keys, so a field label is a dot

{.x 1.0 .y 2.0} is how a struct is constructed, and {inner .field} is how a pattern names one. The colon is gone from both, and what is left of it is one job: keys — map keys and enum members.

What was wrong with the colon. Nothing, taken alone. The problem was that it had two jobs and the dot had one. (.x v) already read a field; {:x 1.0} also named a field, while :space named an enum member. So the dot meant "field" and the colon meant "field, or member, depending". Moving the label to the dot leaves each mark with one meaning, and it costs nothing to read because the delimiter already disambiguates: (.x v) is a list and therefore a call and therefore an access; {.x 1.0} is a brace form and therefore a construction. There is no position where the two could be confused, which is why the same spelling can serve both.

Why it had to land before Map, and this is the real reason. A map literal wants to be {:key value}. While struct construction owned that exact spelling, a map literal and a struct literal were the same syntax, and the only thing that could tell them apart was what the checker expected at that position. That is a context-sensitive grammar for no gain. Reserving the colon for keys keeps the two visibly distinct at the reader, before any type is known. Doing it after Map landed would have meant changing both; doing it first meant changing one.

:keys kept its colon, and that is the point rather than an inconsistency. {:keys [x y]} is the one thing in a brace that is not a field name — it is an instruction to the compiler that happens to sit there, and it takes a vector rather than a value. Giving it a dot would have made the dot mean "a field, or the word keys". Leaving it a colon lets the dot mean exactly one thing, this names a field, which is the whole reason the colon was given up. Everything else Clojure puts in that position — :as, :or, :strs, :syms — is still refused by its own name.

The old spelling is refused, not accepted quietly, and the refusal names the new one: a field label is written .x, not :x — the colon is for keys. Two accepted spellings is how two spellings become permanent, and the standing rule here is that what is not supported is rejected explicitly with the reason. Both refusals are tested by their reason, on the construction side and on the destructuring side, which is what stops the colon drifting back.

The sweep is a tool, not a one-off. tools/colon-to-dot.py converted 681 labels across 45 .flan files, vendor/ included, and 94 more in the Flan embedded in lib/prelude.ml and the tests. It works on forms, not on text: a keyword becomes a dot only where it sits in a field-label position inside a brace, so an enum member in value position ({.k :hi}), a genuine EDN map inside a string (test/programs/edn.flan), and a type-position {K V} are all left alone. It was kept in the tree because several lanes branched before it and their Flan needs the same pass at merge.

What it deliberately did not change: the printed form. render.ml still prints (V {:x 1.5 :y 0}). That string is a wire format — emacs/flan-inspect.el parses it back and hard-codes the colon when it reads a field out — so moving the printer alone would break struct inspection in the dev loop without breaking any test that says so. The printer moves when its reader does, in the Emacs lane. It is the one place the old spelling is still correct, and the reason is worth keeping: a format with two ends only changes at both.

Since written: the printer moved, and both ends moved together. render.ml prints (V {.x 1.5 .y 0}) now and emacs/flan-inspect.el reads the dot, which is the "moves when its reader does" the paragraph above was waiting on. The reader tells ... from a field label by one character of lookahead, because both begin with a dot and a field name never starts with a second one.

Two ways to root a walk, and why neither subsumes the other

i in the break buffer sent a local's name to be evaluated. An expression is evaluated where the evaluator stands, so on the innermost frame that lands in the right frame by luck; on any other it may resolve to a global, to another binding of the same name, or to nothing — with the locals listing right above it showing the frame's own storage and nothing saying the two disagree. The display was right and the inspector was not, which is the worst arrangement of the two.

The obvious fix was tried and rejected, and the rejection was half wrong. Rooting the walk at the slot's address does not work on its own: an address is not an expression, so the first RET has nothing to build the next expression from and navigation dies at step one. What that argument assumed is that the step has to be an expression too, and the shadow stack is what stopped that being true. The daemon holds the frame's address and every slot's type, so stepping into a field is an address plus an offset with that field's type — which is exactly the arithmetic Render.render already does for the locals listing. So Session.render_slot is render_locals with a path applied to the root before the walk and one line out instead of one per slot. No second walk was written and no backend change was needed.

The verb is (:op "inspect" :frame N :slot I :path (...)). A path step is a string for a struct field, an integer for an array or slice element, and the symbol some for an option's payload; a union case's field is spelled Union.case.field, because the payload's offset depends on which case the value is in and only the renderer knows which case it currently holds — it wrote the head (Union.case {…}). Guessing the case from a field name two cases share would read one case's layout over another's payload. Every step that does not fit the type in hand is refused by name with its reason. A pointer is still never followed; that is the renderer's rule and not this mode's.

The slot travels by index, not by name. check.ml's fresh_slot only ever allocates, so (let [v 22] …) inside (let [v 11] …) is two slots both called v and both are in the listing; and a refused slot is not in the listing at all, so its position there is not an identifier either. locals therefore puts the slot index on each entry as a fourth element, and that is what the break buffer hands back.

The frame checks are the listing's, by construction. Dev.stopped_frame is one function and locals, globals and inspect all go through it: alive, stopped, the frame exists, it is the program's and not a C-x C-e thunk's, its body is one this session holds, the slot count matches, and Emit.slot_fingerprint matches. An inspector with its own copy of those conditions would be free to read a frame whose body was redefined since it was entered, which is precisely the stale-slot answer the listing refuses. inspect adds one refusal of its own, for the listing's reason: an unbound slot is a null address and a thunk that read it would fault on the game thread of a program that is already stopped.

What each root cannot do that the other can

Both are wanted and the buffer says which it is on.

The expression root works on a running program and starts from anything you can write, a call included. It cannot name a frame — that is the bug — and it cannot reach an option's payload, because the compiler gets at that as field 1 and nothing in the surface language does.

The slot root is exact to one frame and one slot, and it reaches an option's payload and a union case's fields, which have offsets but no accessor form to write. It needs a stopped program, it is refused when the frame's body was redefined since it was entered — the same fingerprint the listing is refused by — and it cannot root at an expression at all, so g after the program resumes is refused rather than quietly answered from somewhere else.

A refusal someone can read is the point of the second one existing. The failure being fixed was not "no answer", it was a confident answer from the wrong place.

l does not cross between them, structurally

A stack entry in flan-inspect.el is (ROOT PATH . POINT). RET only ever appends a step to the path under the root the buffer already has, and every new root — flan-inspect, flan-inspect-slot — starts with an empty stack. A stack with two kinds of root in it therefore cannot be constructed, so the question of what l should do when it crosses one does not arise. That stays true if a third rooting mode is added, which is why it is worth having as structure rather than as a rule in a comment.

The Emacs state is a root plus a path rather than a retained value for the same reason the expression stack was: nothing on this side can hold a Flan value. A value has no header, the thunk that rendered it is dlclosed the moment it returns, and there is no heap to retain it in. So every step and every g is a fresh request, which is what keeps the view from ever being stale — and it is also why g is a key someone presses rather than a timer, since an expression root with an effect in it would fire once a second for ever.

One wire detail worth recording. An empty :path is sent by omission. Emacs prints an empty list as nil, which is a symbol on the wire and would be read as a step, so there is no way for a client in that language to spell (). The daemon reads a missing :path — and nil — as the slot itself.