flan/NEXT.md

125 KiB
Raw Blame History

Where this is

Start here — next session

Branch dev-loop, 154 commits, working tree clean, dune test green.

The dev loop works end to end: flan dev program.flan, then C-c C-c, C-x C-e and C-c C-r in Emacs against the running process. Conditions are two steps in of four.

(error c) is in — §2's diverging variant, same lookup, type Never. A handler that returns normally has not answered it, so only a transfer gets past; with nothing transferring the program stops and names the condition. flan_error is where the break loop goes.

The break loop is in, editor half included. An unhandled error stops the program on the frame that erred, the daemon annotates every reply with :stopped/:condition, and C-c C-b in Emacs lists the restarts and resumes into the choice. See "The break loop" and "The break loop in the editor" below — both describe what is built.

A restart is chosen by position now, not by name, which is the fix for the shadowing bug and the reason :restarts is a positional list with a :unreachable set beside it. Restarts below the evaluation a break is inside are listed, marked, and refused with the reason; the list itself is a snapshot taken when the break was entered, because the stopped thread's stack does not hold still. spec-conditions.md §4 now says all of this.

The old next task, now done, was the dev-build break loop, spec-conditions.md §2 — where an unhandled error stops and talks to the daemon instead of rt_die(), and where "a crash kills the program" finally gets fixed. The transfer it needs exists now: §6's channel is in every signature and restart-case catches on it, so a break loop is a place to stand while the program is stopped, not a new way to move.

Then restarts offered in the Emacs minibuffer, which wants compute-restarts plus two protocol ops. SBCL's restart struct carries report-function and interactive-function for exactly that prompt and spec-conditions.md mentions neither — worth adding before that step. find-restart and compute-restarts are both named in §4 and neither exists yet; the runtime already has the stack they would walk.

Still open from §3, and each refused by name today: restarts with parameters (argument marshalling plus the runtime arity check), and handler-case, which §"What this does not settle" leaves open as possibly a macro over handler-bind plus a transfer. error, find-restart and compute-restarts now refuse by name too — they used to fall through to a call and come back as unknown name, which is the house rule's own class of bug.

Read SBCL for what restarts should mean and ignore how it moves control: it transfers with block/return-from, which §6 rules out.

In flight — four tracks, one session, 2026-09-12

Four agents, parallel worktrees. Listed so a session reading this cold does not start one of them again. 1 and 2 are merged; 3 and 4 are still running.

  1. Done. nth removed, and println built. nth was an asymmetric alias of at: check.ml aliased them, but parse.ml and place_of_expr matched only at, so (set (nth a i) x) and (addr (nth a i)) were refused while the at forms worked. println lifted Session.render into lib/render.ml, parameterised on an emitter and a slot allocator, so the REPL and stdout share one walk. Found on the way: field_addr in emit.ml accepted only Types.Named, so a field of an Option threw at emit time and the walk's Option arm had never run — the inspector would have failed on the first (Option T) pointed at it.
  2. Done. restart-at, by position. The editor half already existed — this session's brief was written off a stale "Start here" line and is corrected below. The real work was taking a restart the innermost frame shadows: restart <name> resolves to the inner one, so a visible outer restart could not be taken at all. An immutable snapshot per break makes an index mean something, :name rides along as a receipt checked against it, and a generation stamp keeps a nested break from claiming the outer one's choice.
  3. The first ten raylib core examples, plus the window and input bindings they need. The deliverable is the language gap list as much as the ported files — sand.flan is the only real raylib program today, so this is the first time the language is pushed by code it was not designed around.
  4. Names in DWARF. The two items under "One line away" below, promoted here because the inspector and the conditions buffer are only as good as what lldb can say: let-bound locals print as s0, s2 because Tast refers to them by slot index and Check drops the names, and Session.eval never passes the ~debug that Emit.redefinition already takes and is already tested for. The second also unblocks source interleaving in the disassembly buffer.

Queued — the runtime under a sanitizer

Nothing has ever run under ASan or UBSan on the test path. grep -i 'sanitize\|asan\|valgrind' over lib/ bin/ test/ runtime/ vendor/ returns nothing at all. This is not a big project — the whole C surface is about 1,260 lines (flan_rt.c 394, flan_dev.c 219, flan_agent.c 647) plus what shim.ml generates.

The bugs are not in allocation. There are four malloc/calloc/strdup sites in the entire runtime and every one is allocate-once-never-free by design — rt_args says so in its own comment — so LeakSanitizer would mostly produce suppressions. The risk is fixed static buffers with bounds arithmetic, and the ones with no coverage are already written down: scratch[SCRATCH] and escaped[ESCAPE_MAX] in flan_rt.c, the 4K result cap, the registry overflow guard, and SNAP_MAX/SNAP_NAMES from the restart snapshot.

There is evidence the sweep pays. Checking flan_escape_bytes by hand against lengths 01300 under ASan with a red zone found the guard correct but its comment understating its own reserve by four bytes — worst output 1021 into 1024. That was one buffer, found by looking.

The shape:

  1. A --sanitize flag beside --debug in lib/build.ml, reaching both the clang run over the .ll and the runtime's own C.
  2. Run the existing corpus under it. test/programs/ is about forty programs with pinned output — a second pass over them is the cheapest coverage available here, and needs no new test written.
  3. UBSan is worth more than ASan, with one exclusion that is not optional: arithmetic wraps by design, so -fno-sanitize=signed-integer-overflow or every program trips on the first +. What is left is real — shift UB ((<< 1 32) compiled to a bare retq at -O2, see Sharp edges), alignment, and the f32→i32 cast on NaN or an infinity that floor-f32 guards by hand and nothing else does.
  4. The variant worth its own run: ASan with --no-bounds-checks. That asks whether the bounds checks are the only thing between the language and corruption, which the checked build cannot ask.

Two limits, so nobody is surprised. raylib and libm are not instrumented, so the windowed examples are noise and the headless corpus is the target — sand-headless, values, machine, virtual-controls-headless. And ASan does not see uninitialised reads, which is where zeroed and struct padding live; that wants Valgrind as a slower second pass, because MSan needs every dependency instrumented and raylib settles that.

Queued — the print-* family goes

print and println are the whole printing surface. print-str, print-i64, print-f64, print-bytes, print-line and newline are kept only until tracks 3 and 4 land, because an agent working in a tree that still has them will write new fixtures that use them and the sweep would run twice.

print subsumes all of them, and is better: there is no implicit widening here, so print-i64 forces an explicit (i64 x) at every call site and print takes the i32 as it is. (print-line s) is (println s); (newline) is (println "").

About 470 call sites across ~60 files. Two places want reading rather than sed:

  • (print-i64 (i64 x)) where x is a u64 changes its answer, not just its spelling — the old form widened through the signed printer and printed -1 for 0xFFFF…F, and print routes a u64 through its own shim. The pinned .out files move, and each move is a thing to look at.
  • web/index.html (55 sites) and web/examples/ carry pinned output, so the prose changes with them. That is also the moment to document println at all: it currently has no mention on the language's own doc page.

DWARF is not missing, and flan-cnr.el's stack pane reads as though it were. flan build --debug emits it and flan-dape.el drives lldb with it. What the break loop lacks is DWARF in a dev build — a flan dev process is not a --debug one. Worth rewording that refusal so it names the real gap.

Managed classes are planned. Do not start them.

plan.org grew a class facility beside struct: identity, runtime shape metadata, an implementation-defined representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its own last line is the rule — nothing until ordinary struct, Handle and reload semantics are working. It is here so that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of them in plan.org yet:

  • A generic function is a cell. "A later module can add (defmethod draw ((e Enemy)) ...) without editing the original" means every compiled call site of draw has to find the new method — which is the problem the indirection cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The expensive half of classes is therefore already built and tested.
  • The pool is not one storage option among three. migrate-instances has to enumerate live instances. A pool behind generational (Handle T) gives that by construction; a world arena and an owned region do not obviously. plan.org presents the three as a free choice and they are not.
  • Enemy@1 has to stay resolvable for migrate to dispatch on it, so the session retains every layout version's metadata for as long as any instance holds it. Same rule as "nothing is ever dlclosed", and worth stating as one.

Open: can a condition be a class?

Unanswered, and it wants answering before handler-case, because it decides whether handler matching has one path or two.

It would buy the thing conditions most lack: a hierarchy. §1 says flatly there is none, which is why nothing can say "any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.

Three costs, one serious:

  • Signalling would allocate. A struct condition is a stack value and signal takes its address; a class instance needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
  • §5's lifetime inverts. Today the condition dies with the signalling frame and a handler that keeps it copies, which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
  • Layout versions meet handler frames. A struct condition cannot change layout; it is refused. A class can, and then a frame pushed against MyError@1 is on the stack while the signaller builds MyError@2.

The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash, dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost. Either way it is an amendment to a frozen spec-conditions.md, not a gap in it.

The dev loop is closed. C-c C-c in Emacs recompiles the top-level form at point and installs it in a running program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to game-draw, and 240 consecutive frames drew it.

Steps 1, 2 and 3 are done — see The reload primitive below. A list of top-level forms can be recompiled and installed into a running process; call sites compiled before they existed follow them, and a defn or defvar the process was never built with can be added and then redefined again. That is the whole of C-c C-c, minus an editor: sand.flan takes a redefinition over a socket and installs it between frames.

What is left is the session — something that holds the checker environment between evaluations, tracks which names the running process was built with, and speaks a protocol an editor can talk to.

Milestone 4 is done: sand.flan builds, links raylib and runs, and its simulation has a headless acceptance case that runs on the dune test path at -O0 and -O2. Milestones 2 and 3 are behind it (calc-me.flan compiles and runs; the interpreter was dropped — open decision #7, settled, see below).

reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
File What it does
lib/loc.ml source locations + Loc.Error, the frontend's one exception
lib/form.ml reader output: Sym Kw Int Float Str Byte List Vec Map
lib/reader.ml hand-written S-expression reader, no menhir/ocamllex
lib/ast.ml AST: texpr, expr, place, pattern, decl
lib/parse.ml forms → AST; special forms, desugaring, declarations
lib/load.ml imports: a package directory → qualified declarations
lib/types.ml resolved types; structural equality, Never fits anywhere
lib/tast.ml the typed IR the backend consumes
lib/check.ml AST → typed IR; two passes, bidirectional
lib/session.ml a live program: what the process was built from, plus every change since
lib/wire.ml the editor protocol: one s-expression per message, length framed
lib/dev.ml flan dev: a session, the program running beside it, and a socket
lib/prelude.ml printers + rand-f32, written in Flan
lib/emit.ml typed IR → LLVM IR text
lib/build.ml .ll + the shim + the packages' C → clang → executable
runtime/flan_rt.c the host ABI: argv, stdout, exit, 4 conversions
runtime/flan_dev.c dev only: the by-name registry a run-time-new name needs
lib/shim.ml declare-c -> the generated C that flattens a struct crossing
vendor/raylib/ the raylib package: raylib.flan and link, and no C at all
vendor/agent/ the dev agent: a socket, a loader thread, install at a frame boundary
emacs/ flan-mode.el, flan-dev.el, flan-repl.el: the editor half of the dev loop
bin/main.ml flan read | parse | check | emit | shim | build | run | reload | dev
test/test_flan.ml reader, parser and checker
test/test_acceptance.ml expression/result pairs + whole programs + the traps
test/test_reload.ml the reload primitive: recompile one function, load it, call it
test/test_agent.ml a running program taking a redefinition over a socket
test/test_session.ml what a running process cannot be told, and recovering from a typo
test/test_dev.ml the daemon, driven the way an editor drives it
test/test_repl.ml C-x C-e: an expression evaluated inside a running program
test/programs/conditions.flan handler-bind and signal, the accumulation case
conditions.org a cheatsheet for driving conditions: what works, the exact refusals, the gotchas
conditions-play.flan a program to poke at them with, built to be attached to by flan dev
test/programs/restarts.flan restart-case and invoke-restart: the transfer, across two frames
test/test_emacs.ml the client, driven against a real daemon and a real program
test/reload_host.c the C host that loads and installs two rebuilds, in one process
test/wasm-run.mjs a WASI host in twenty lines of node:wasi, so the table can run a wasm32 build
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
3.5
$ flan run test/programs/sand-headless.flan
-2851001042534928384
$ flan run sand.flan                      # a window, 120 fps, hold space

What milestone 4 added

dotimes desugars in check.ml to a Let plus a While — no new IR node. The bound is evaluated once into a hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not assignable, which makes the generated step its only writer.

defer is recognised in check_fn and nowhere else, because that is the only place that knows a form is at the top level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it stands. Function exit runs them innermost-first, and an explicit return runs the ones registered above it — a defer written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the bounds-check shape (noreturn then unreachable) rather than being a separate decision.

defer inside a let, a loop or a branch is rejected, not accepted with function scope. It would run once at function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. Block-scoped defer is real work and is not done.

New builtins: zeroed (takes its type from the place it is stored into), min/max (each operand through a slot, so neither is evaluated twice), bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a signed type and logical on an unsigned one), and rand-f32.

rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state. It is not libc's, because a grid hash is only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed sets the state. This is what the bitwise operators were added for.

Enums and keywords. (defenum Name [member value ...]) gives a type that is an i32 at run time and its own type in the checker, so :space at a call site resolves against the parameter's enum and a typo is an error there rather than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.

Why the FFI goes through a C shim

The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:

Vector2 {float,float}   →  declare <2 x float> @GetMousePosition()
Color   {u8,u8,u8,u8}   →  declare void @ClearBackground(i32)
Rectangle {4 × int}     →  declare { i64, i64 } @mkrect()

None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a per-target classification the caller has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that in emit.ml is three classifiers to write and then keep correct forever, and a mistake shows up as (.y m) returning garbage rather than as a link error.

So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a copy. clang classifies all of it, per target, for free. check.ml enforces the rule — an aggregate in a declare signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host ABI, implemented twice", and flan_rt.c is the same pattern.

The wrappers are generated now — declare-c, lib/shim.ml

The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to generate "if that ever becomes the bottleneck" came true at 84 of them. vendor/raylib/shim.c is gone; the directory holds raylib.flan and link and no C at all.

One binding is now one line:

(declare-c draw-texture [t Texture2D  x i32  y i32  tint Color] "DrawTexture")

declare-c names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like any other: the typedefs for the structs involved, made from the Flan defstructs; the extern prototype in the function's true signature; the wrapper that flattens it; and the flattened declare the Flan side calls, with an ordinary Flan defn above it when the signature has a struct in it. flan shim <file> prints the whole file.

It is a second form and not a change to declare, for one reason worth remembering: (declare start-raw [path string] i32 "flan_agent_start") in vendor/agent means the symbol takes ptr+len, and (declare-c init-window [w i32 h i32 title string] "InitWindow") means it takes a NUL-terminated char *. Same shape, opposite claims, so no structural rule can separate them. declare is untouched, and sqrtf and the agent still work unedited.

What the generator guarantees, and what it trusts. Guaranteed: the C typedef and the Flan struct come from the same defstruct, so they cannot disagree — permute the defstruct and the typedef permutes with it, which is exactly what makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted: that the defstruct matches the library's real struct, and that the declare-c signature is the function's real signature — no header is read, deliberately, so nothing can check either. A _Static_assert on sizeof/offsetof was considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard: for every field type the generator admits — the machine integers, the two floats, bool, a pointer and a nested struct — LLVM's layout is C's, and emit.ml writes no datalayout, so clang applies the target's rules to both halves. Everything where they could diverge is refused at the field.

One thing got sharper and should be said plainly: the prototype is now generated from the declaration, so a scalar's width carries ABI weight it did not before. f64 where raylib says float used to be narrowed by clang at the hand-written call site; now it emits double and raylib reads garbage. All 84 migrated prototypes were diffed against the deleted shim.c's — which was the ground truth for the true signatures — and agree.

Strings. The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, PATH_MAX for a path, 512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on malloc failure, where the alternative is handing C a null pointer.

Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C. collision-point-poly? takes a slice and collision-lines answers with an Option; neither is raylib's own signature. A slice parameter in a declare-c is refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not recoverable from [T] — so that one declares (Ptr Vector2) with an explicit count i32 and the Flan wrapper passes (addr (at points 0)) and (len points). Every other refusal — an Option, a union, a fixed array, a map, a returned string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with the reason, and the acceptance table asserts on the reasons.

Known edge, not fixed: a REPL redefinition that introduces a new declare-c cannot work. Build.shared is llc + ld -shared and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function that calls an existing binding is unaffected.

raylib.flan carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is (rl/get-mouse-position) returning a Vector2. Verified end to end, headless: GetColor(0x11223344) comes back as 17 34 51 68, four separate bytes — a Color is not the little-endian reading of the packed integer, so an identity would have passed a weaker test. That case is in the acceptance table, skipped if libraylib is not installed.

The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven collision-* predicates; textures; the Image family; Camera2D; RenderTexture2D; the whole audio surface (device, Wave, Sound, Music); fonts and glyphs; and gamepads, touch and gestures — plus the Key, MouseButton, TraceLogLevel, GamepadButton, GamepadAxis and Gesture enums, raylib's own named colour palette, and the FLAG_ window hints. Adding one is a single declare-c line; there is no C to write.

Two things the ported examples in examples/ wanted and could not have, both refused for reasons that are right. GetGamepadName returns a char * into raylib's static storage: the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that returns one returns something Flan has no owner for. And an enum parameter cannot be indexed — GetGamepadAxisMovement takes a GamepadAxis, a loop variable is an i32, expected rl/GamepadAxis, found i32, and a second declare-c of the same symbol with an i32 face is refused too: one declare-c per C function, and another Flan name for it is a defn — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a cond over the members it knows.

The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a GPU: GetCollisionRec, which pins Rectangle completely, and SetShapesTexture's default substitution, which pins Texture2D's id and format and nothing else. Handing a struct over and reading it back proves nothing at all — storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. width, height and mipmaps are therefore checked only by looking at sand.flan running, which draws the brush sprite four ways for that reason.

No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared library being linkable and not on raylib-devel. vendor/raylib/link carries -l:libraylib.so.550 because Fedora ships the runtime library without the .so symlink.

What a headless FFI test can and cannot pin

Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.

  • A struct round trip is worthless. Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C writes and reads the same wrong slots and the test passes for any field order. Found by permuting two fields and getting identical output.

  • Axis-aligned geometry cannot pin Vector2. Exchanging x and y is a reflection, applied to the inputs on the way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, passes with the fields swapped — verified by swapping the defstruct, which is what the typedef is now made from. Distances are worse: the reflection does not even reach them.

  • What does pin Vector2 is the rotated camera, because a 90-degree rotation is not axis-aligned and therefore does not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision cases look like they cover it.

  • What pins Rectangle is arithmetic on its fieldsGetCollisionRec computes four numbers from four different field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.

  • Scalars in, fields out is the strongest shape there is, and the Image family is where it was finally available. GenImageColor(4, 2, colour) is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values in four adjacent i32 slots, with no input struct for a permutation to cancel against. That pins Image completely, including that data is present and first; Texture2D could never be pinned that way because nothing without a GPU reads its width, height or mipmaps at all.

  • A non-square image is an axis discriminator. GetImageColor indexes y*width + x, so on a 4-wide, 2-tall image (3,0) exists and its transpose does not: exchange the wrapper's x and y and the read goes out of bounds and answers transparent black. ImageFlipHorizontal against ImageFlipVertical says the same thing twice more.

  • A file is external ground truth, so ExportImage then LoadImage is not the symmetric round trip the rest of the package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the x/y permutation above.

The rule that falls out: make raylib compute something whose answer differs per axis, then verify the test can fail by permuting the fields and watching it go red. A case not verified that way is decoration.

One correction to an assumption that has now cost two lanes a guess: MeasureText is not headless material. It measures with the default font, LoadFontDefault is not exported, and nothing but InitWindow loads it — so with no window it answers 0 for every string. Measured against libraylib.so.550, not reasoned about. GetFrameTime, GetTime and GetScreenWidth/Height are all 0 headless for the same kind of reason. All five are bound, and all five are exercised by running sand.flan and looking, which is the whole of what can be claimed for them.

Packages

lib/load.ml resolves (import rl "vendor:raylib") before the checker runs. The directory is the package; vendor: is a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no collection is relative to the importing file. Importing is a rename: every top-level name the package declares becomes alias/name, and every use of one — in a type, in a body, in a struct literal, in an array length — is rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of declarations whose names contain a slash.

A package may also carry the C it binds to: every .c file in the directory is compiled into the build, and a file named link lists extra linker arguments. Whether those reach the build at all is decided after checking — see below.

A package may be a single .flan file named outright, rather than a directory. That is for the program that is also a library: sand.flan shares the repository root with three other loose programs, so naming its directory would import all four. A file carries no .c and no link file; those belong to a directory.

A package may import a package. The qualification flattens to the inner alias — raylib imported by a package that is itself imported is still rl/…, never sand/rl/… — because a directory reached along two routes has to arrive under one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time, and the namespace being flat, mutually dependent packages simply work. The same directory under two different aliases is refused.

Visibility is one rule: main is not exported. A package carrying one would collide with the importer's the moment anything imported it, so a program could never be a package; and main is a reachability root, so an imported one would keep everything it calls alive. Writing sand/main is refused at the line that wrote it, with the reason — left to the checker it would be "unknown name", which is true and useless.

Still missing: a package-private marker for anything other than main, which is why rl/get-color-raw is callable.

Load used to hand a package's .c files and link arguments to the build the moment it was imported, whatever the importing program did with them. So anything naming vendor:raylib linked libraylib on every target, and on wasm32 that link cannot succeed.

lib/reach.ml answers it from the checked program instead. Start at main and at the globals that run before it, follow every call — including the Handled frames, where a lifted handler clause is reached by address and by nothing else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.

Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and wasm-ld would fail on the symbols rather than on the argument list. So the same walk prunes functions and externs from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a dropped one is a silently different program.

Dev builds are not pruned. What a REPL may redefine next is not a function of what has been called so far.

The filtering happens at the call sites — bin/main.ml, the tests — because Build.executable receives csrcs and lflags from its caller and never sees the import list. Reach.link returns the pruned program and its C and linker arguments together, so a caller cannot take one without the other.

sand.flan is one program

It was two files, and only ever for the reason above: the headless run is the one CI does on native and wasm32, and a program that imported raylib linked libraylib whatever its main did. So the simulation lived in sand-sim/ and both drivers imported it.

Now sand.flan holds the simulation and the raylib front-end, and test/programs/sand-headless.flan imports sand.flan itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches raylib; sand.flan's main is not exported, so the only main is the headless one; and the hash is unchanged on both targets at -O2 and -O0, which is the point. A refactor that moved that number would have moved the simulation.

What still justifies two entry points is smaller and stands on its own: the headless test needs no window and no input on any target. sand.flan could not be that test even natively — with no mouse the grid stays empty and settle and move-grain never run on real data. Measured through the probe: 168 grains painted around row 48, still 168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.

Two claims that got run together in an earlier note, for the record:

  • raylib does not work on wasm — false. It works through emscripten. What is true is that it does not work on the wasi path, which is what the headless table targets, and which has no GL and no browser.
  • a game loop cannot be expressed on wasm — false. The browser cannot be blocked, so a web build drives the loop with emscripten_set_main_loop instead of a while. That is a different main, not a different program.

Three edits were made to sand.flan's own text when it was ported, and they are language decisions rather than fixes:

  • (defconst gravity 0.05)(defconst gravity f32 0.05). An untyped float constant is f64, velocity is [f32], and there is no implicit widening.
  • (defvar current-color u32)i32. It is an index into colors, and (len colors) is an i32.
  • (defn main []) is unchanged — the short form, as plan.org says.

Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.

Bounds checks — done at milestone 3

at and slice emit icmpbr → cold block → callunreachable; a failure names the source location. Three check sites: at on [n T] (static bound, folded by LLVM for a literal index — and a literal that is out of bounds never reaches emit, check.ml rejects it), at on a slice or string (runtime len), and slice (two comparisons — lo <= hi is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. Build.opts.checks is on by default and not tied to opts.opt, which is what lets the acceptance table run the same programs at -O0 and -O2 with identical checks. The flag is --no-bounds-checks.

The write path is its own case: (set (at arr n) …) lowers through place/Pindex, not through At, so a refactor that split them would break the write check silently. The test covers both.

Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.110.12s checked against 0.120.13s unchecked. Indistinguishable.

Why there is no interpreter

Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.

The layout, which is the whole backend design

i8..i64 / u8..u64   i8..i64          signedness lives in the ops
f32 f64             float double
bool                i1
an enum             i32
[T] and string      { ptr, i64 }     ptr+len, non-owning
[n T]               [n x T]          inline, a value
(Ptr T)             ptr              opaque pointers
(Option T)          { i8, T }        tag 0 None, 1 Some
a struct            a literal struct, declaration order
Unit and Never      {}

No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:

  • Every slot is an alloca. Reading a local is a load, assigning is a store, and a store of an aggregate is the copy spec-memory.md requires. addr of a local is then just the alloca, and mem2reg removes the ones nobody addressed. test/programs/values.flan pins this down.
  • A place is a pointer, a value is a load from it. (set (.pos c) …) through a (Ptr Cursor) becomes a getelementptr on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.

Non-local exit is lowered explicitly: return, some and a failed bounds check are branches, never platform unwinding, so wasm32 needs no exception proposal.

Blocked and unfinished

Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a vague intention — if it is listed, someone has already established it is real.

Unblocked now, and ranked

  1. Allocators, then Vec and Map. The critical path, and the only thing standing between this and writing a game. Vec does not need generics — that was wrong and is worth un-learning: Odin's containers are compiler builtins over a type-erased runtime (base/runtime/dynamic_array_internal.odin), where $T appears only in thin wrappers producing size_of/align_of at the call site, and per-key hash and equality are compiler-emitted procedures passed as a runtime argument (src/llvm_backend.cpp, Map_Info). That is exactly what spec-memory.md already specifies. Before writing any of it, settle the four things the Odin and Carp studies converged on, because all four are cheap now and expensive after:

    • When is storage released? spec-memory.md never says. Odin's answer is defer delete, which Flan cannot express — defer is function-scoped and refused in a let, a loop or a branch. Carp's answer is scope-end frees, which it then could not reconcile with arenas and so has no allocator at all.
    • A drop hook for a struct owning something that is not memory — a Texture2D, a socket, a file handle. Carp shipped delete and then had to add a separate drop interface (docs/Drop.md). The hard part is ours alone: what runs drop when the arena resets underneath the value.
    • alignment appears nowhere in the design. Every Odin allocation carries it, and #soa and component-wise fixed arrays want 16-byte alignment.
    • Allocation failure. Unspecified. Odin returns an ignorable error, so a failed append silently appends nothing. Flan has a better answer available for free: a StorageExhausted condition with a retry restart. Decide which, because push/put/clone's signatures depend on it.
  2. Typed restarts — (use-value [v T] v). The author's third TODO, and the most-wanted thing across every comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard vocabulary", because use-value and store-value are the only two whose answer comes from outside the program. Needs argument marshalling in emit.ml and §3's arity check in check.ml; both files are free now. The leverage SBCL lacks: eval already compiles and runs an expression inside the live program, and the daemon already holds the struct layouts, so "ask the human, type-check the answer, hand it over" is a short hop.

  3. handler-case. Not a convenience — it is the fix for the loudest gotcha in conditions.org. A handler closes over nothing only because a handler-bind clause runs at the signal point; a handler-case clause runs in the establishing frame, which is ordinary in-frame code exactly like a restart-case clause. SBCL's is handler-bind plus a transfer and nothing more (src/code/error.lisp:196-268). Every piece exists.

Bugs found and not yet fixed

  • A shadowed restart is offered and cannot be taken. Fixed. A restart is taken by position now: (:op "restart-at" :index N :name NAME) on the daemon, restart-at N NAME on the agent, and a numbered completing-read in C-c C-b. :name is a receipt, not the lookup — it is checked against the name the snapshot holds at that index and refused if the two have drifted, so a bare integer can be wrong out loud. restart <name> survives for a raw socket and is now defined as restart-at on the first index offering the name, so the two verbs cannot disagree. break.flan grew the shadowed pair and asserts 900, which is the only value in that file no by-name lookup can produce. The C&R buffer still marks the shadowed row by name and could now offer it instead — small, and not done here.

  • A restart chosen at a break inside a thunk is accepted, announced, and silently not taken. Fixed by refusing it, with the reason. Not by the depth NEXT.md proposed: recording the restart-stack depth on entering the break loop counts the frames a restart-case inside the thunk pushed before it erred, and those are above the boundary and work. The boundary is where it is made — restart_floor is set to flan_restart_count() around j.call() in flan_agent_poll, saved and restored so thunks nest — and the outermost floor entries of the snapshot are marked unreachable. They are listed and marked rather than hidden, refused by the listener before the reply, and carried to the editor as :unreachable (2 3). test_dev.ml breaks a stopped program a second time from inside C-x C-e and asserts both halves: index 2 refused, index 0 taken.

  • Restart names are served from a stack that is being mutated. Fixed, and it was a precondition rather than a separate bug. Index-based resume is wrong by construction against a moving stack: unlike a name, an index carries no evidence of what it meant. The agent copies the list on entering break_loop — names into its own buffer, frames as the addresses a transfer carries — one snapshot per nested break, and every verb answers from it. Caps are SNAP_MAX 64 restarts and SNAP_NAMES 4096 bytes; past either, the listing says how many it did not show. Neither cap has a test, same blind spot as the 4K result cap below.

  • A snapshot generation has no test, and the window is a race. A choice is validated against the snapshot on top when the request arrives and resolved against the snapshot on top when the game thread next looks. Between those, an evaluation the break loop is running can error and push a break of its own, whose loop would otherwise reach [chosen_ready] first and take its index 2 for the one someone chose from the outer list. Each snapshot now carries a generation, a choice is stamped with the one it was validated against, and a loop claims only what is addressed to it — a mismatch is left set rather than discarded, because the listener already answered ok for it. Depth would not do: an outer break resuming and a new one starting reuses the number. None of this is tested, because arranging the window means landing a request inside a two-millisecond poll from outside the process. It wants a hook the test can drive, not a sleep.

  • The job ring has no fullness check, and the comment describing its overflow is wrong. publish never consults tail; past QUEUE entries it overwrites the slot the consumer is reading, and job is 24 non-atomic bytes. Reachable from a program that goes a long time between agent/poll calls.

  • flan_dev_result_get is not the seqlock its comment claims — it reads the generation first, then a non-atomic length, then returns a bare pointer the caller sends later.

  • Smaller: exit(134) from the break loop with the listener inside dlopen; a dlopen handle leaked when a module has no installer.

  • (A {:x 1}) on a union variant says "unknown struct A" rather than the union refusal check_struct plainly intends — env has no table of variant names. A diagnostics bug, not a backend death.

Test blind spots, from a mutation pass

Sixty mutations, nineteen left the whole suite green. The severe cluster is closed (cleanup.flan, signedness.flan); these are not:

  • Reach's walk of index expressions, addr places and restart-case clause bodies — each confirmed to prune a function a valid program calls, so the build fails to link.
  • flan_dev_global's size-change guard — the layout-drift check, with no test that retypes a global across a reload.
  • A local shadowing an imported name is qualified anyway.
  • The 4K result cap and the registry overflow guard have no coverage at all, rather than a missing assertion.
  • The reader accepts an unknown string escape; +5 stops being a number.
  • And a warning: a reader mutation makes the suite hang rather than fail. A green run is not the only outcome to plan for in CI.

Asked for by the editor lanes

  • (:op "layout" :type T) → the struct's fields and their types. Tast.structs is held by the daemon at all times because it owns the build, and no running program is involved — this is the cheapest real win on the list, and the C&R buffer already draws its result.
  • (:op "condition") → the stopped program's condition, rendered. Two steps: break_loop currently does (void)condition; and discards the pointer, so stash it beside condition_name; then the daemon builds a render thunk aimed at that address, which is Session.render rooted at a Ptr instead of an expression.
  • One thing to get right for both: the type must be an identity the daemon can resolve to a Tast type, not a bare class name. The hook is handed a string, and two packages both declaring Missing leave the daemon unable to pick a layout. A qualified name or a type id. The same wrinkle bites locals later, because DWARF also gives a name.
  • (:op "backtrace") is blocked on frame metadata — unlocked by the DWARF work, then a new agent verb. Locals are blocked twice: DWARF for the frame layout, and the pointer-rooted render thunk. Restart source locations and arity are blocked too — flan_restart carries prev, name_id, name and namelen, so both need a new field in the frame, which means the compiler emitting it.

One line away

  • match over enums. Fully desugarable, wanted, and blocked only by Ast.pattern needing a keyword case, which load.ml matches exhaustively.
  • [in flight] DWARF for a redefinition module. Emit.redefinition takes ~debug and is tested; Session.eval does not pass it. That also unblocks source interleaving in the disassembly buffer.
  • [in flight] Let-bound locals print as s0, s2 under lldb. Parameters get their real names; Tast refers to the rest by slot index and Check drops the names.
  • Build.executable returns only out, so the daemon recovers the host .ll by recomputing Build.workdir ().

Deferred with a reason

  • Writing through a string literal — see Sharp edges. Needs provenance, which is open decision #3.
  • cstring as a type. Odin has no string → cstring conversion at all; it pays the same copy our shim already makes. The one thing it buys is the return direction, and nothing in vendor/raylib returns a string.
  • rune. Odin's is a 4-byte integer distinguished by a flag, so i32 is the same thing. Non-ASCII text is blocked on font loading, not on the string layer — and fonts are now bound.
  • Macro expansion. The reader and the declaration are in. Running a macro means compiling it and dlopening it into the compiler, which is the reload primitive pointed at ourselves — but a macro is [Form] -> Form, so Form has to be a Flan union whose layout the compiler and the loaded macro agree on, and union values are milestone 6.

Documents that contradict the code

  • plan.org's jank #947 citation is wrong in its mechanism. jank does not relink (it calls through vars, which are already indirection cells) and never unloads (remove_symbol has no callers). The real cause was a process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A normative document citing the wrong mechanism protects the wrong invariant.
  • plan.org still lists open decision #7 as open and the interpreter as a backend. It was settled the other way; NEXT.md records the consequences as "already applied" to plan.org, and they never were.
  • nREPL's eval does carry file, line and column — jank reads all three. The choice of s-expressions still stands on its other grounds; the stated reason does not.

Sharp edges

  • Writing through a string literal is undefined, and the two build modes disagree about how. (let [s (bytes "Hi")] (set (at s 0) \h)) stores into a private unnamed_addr constant. At -O0 that is a store to read-only memory and the program takes SIGSEGV; at -O2 LLVM deletes it as undefined and the program prints Hi and exits 0. Same source, and which way it fails depends on a flag — the worst shape available, and worse than either outcome alone.

    Nothing refuses it. bytes turns a string into a [u8], the language lets you write through a slice, and by then nothing records that the bytes came from a constant. The honest fix is provenance — knowing a slice's origin — which is plan.org open decision #3 and deliberately deferred. A cheaper one that is not a fix: emitting literals as mutable globals only moves which flag misbehaves, and costs their read-only placement.

    Found by the string lane while deciding whether lower-ascii should mutate in place. It ships the copying version for exactly this reason, and that is the rule to follow until provenance exists: a function over a string must not write through it.

Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count, both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are fixed; each says so. They stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.

  • An index converts from a narrower integer and never from a wider one. (at colors current-color) with a u32 index works — anything above 2³¹ truncates to a negative i32 and the unsigned bounds check rejects it. An i64 index is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
  • There is one top-level namespace, and check.ml now enforces it. The environment's tables are per-kind — structs, unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a duplicate. (defn item …) beside (defvar item …) type checked and then died in LLVM as redefinition of function '@flan.item', a message about an emitted symbol with no source location left, and two colliding type declarations were not caught anywhere. One pass over Ast.declared_name now runs before every other collection pass and rejects the second declaration of a name whatever kind either one is. declared_name lives in ast.ml because Load needs exactly the same set — the names an import renames — and two copies of that list would drift.
  • A shift count is bounded, two different ways. A shift by the operand's own width or more is poison in LLVM, not a wrong number: (defn main [] i32 (<< 1 32)) compiled at -O2 to a bare retq, returning an undefined value. A literal count out of range is now rejected in check.ml — that is the typo case — and emit.ml masks a computed count to width - 1, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The prelude's rotate masks its own count; that is now redundant but harmless.
  • A u64 literal is its 64-bit pattern, so 0xcbf29ce484222325 is a real u64 and not an error. The cost is that a negative decimal literal is accepted as a u64 too, because the reader records the value and not how it was written. Narrower unsigned types keep the strict check, which is where a typo like 300 for a u8 actually shows up.
  • A folded constant skips check. (defconst rows (/ h c)) is emitted from the folding pass's value, because a global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is therefore its own call to in_range; there is a regression test.
  • A let binding takes no type annotation, which is why sand.flan names its FNV constants instead of writing them inline.
  • (defn f [] f65 0.0) still says unknown name rather than did you mean f64: with a single body form the parser cannot tell a return type from the first expression. Only the parameter position and (Option …) are unambiguous.

The reload primitive — dev loop steps 1 and 2, measured

llcld -shareddlopen → call, with no protocol and no daemon. dune test runs it: one function is recompiled into its own object and called inside a process that is already running, twice, with a changed body the second time.

Step Cost
Emit.redefinition below the timer (<0.1ms)
llc -O2 -filetype=obj 1517ms
ld -shared 3ms
dlopen + dlsym 0.04ms

~19ms end to end, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the number from this codebase. For contrast, clang -shared on the same IR is 50ms — the driver is again most of the cost, which is why the dev path skips it. llc and clang are both 20.1.8 here; check that before trusting the .ll, since the driver absorbs IR the bare tools reject.

ld -shared rather than clang -shared for a second reason: a shared object is allowed undefined symbols, and that is the mechanism. What the new module does not define is the whole design:

  • a global is external. This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand's grid would reset on every reload, and "edit the code, keep the sand" is the thesis.
  • every other function is a declare, so a redefined settle calls the host's move-grain rather than freezing a private copy of it.
  • no main. This module is loaded, not started.

Its string constants still come along; omitting them is an undefined @.str.N at link time, and it is easy to miss because a one-function module usually has none. Emit.signature is now the single place a function's LLVM signature is spelled, because a define here and a declare there drift the moment one of them grows a case for Unit or for a slice parameter.

flan_dev.c is compiled into every build, not only a dev one. Nothing in a release build calls into it — the compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the agent package's C refers to it, and a package's C sources are collected whatever main does. Leaving it out of release builds made flan build sand.flan fail at the link with an undefined flan_dev_result_get, which reads as a compiler bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; -rdynamic and the cells are still what --dev means. test_agent.ml links the agent program both ways for this reason.

-rdynamic is load-bearing. A normal executable exports nothing: nm -D calc-me | grep 'flan\.' is empty, so a loaded module's declares would have nothing to bind to. The test passes it through lflags, which keeps it a property of the dev build rather than of every build. dlsym on "flan.bump" works — a dot is legal in an ELF symbol.

Two things about the test are deliberate and are what make it prove anything: both loads happen in one process, since two runs would pass while saying nothing about an in-process swap; and the versions are two paths, since dlopen caches by path and re-opening one would hand back the handle it already had, so the check would lie. And helper is (* x 2) in one fixture and (* x 3) in the other: the second body is dead text, since the module declares helper rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.

String constants are emitted private unnamed_addr, so the module's own @.str.N cannot be interposed by the host's — worth knowing, because with external linkage a redefined function would silently print the old text and nothing would fail at link time. The fixtures each print a literal so that path is actually exercised.

Cells — how a call site follows a redefinition

Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so a dev build routes every Flan-to-Flan call through a cell: a mutable global holding the address of the function that is current.

@"flan.cell.bump" = global ptr @"flan.bump"        ; the host defines it
%p = load ptr, ptr @"flan.cell.bump"               ; every call site
%r = call i64 %p()

Redefinition is then one store. A redefinition module declares the cells external, exactly like the globals, and exposes flan_reload_install() that stores its own body into its own cell — cost below a microsecond, which is what makes a frame-boundary swap a non-event.

The cell load is emitted after the arguments, so a redefinition landing between two calls cannot land in the middle of one.

Four things about this that are not free choices:

  • flan_reload_install is a named function and not an ELF constructor. A constructor runs during dlopen, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose.
  • A redefinition's own body is hidden. Default visibility in a shared object is interposable, and that applies to taking the address too: plain @"flan.bump" inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent.
  • This also fixes the self-call edge, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not.
  • -rdynamic is what exports the cells, so it and cells are one flag: Build.opts.dev, flan build --dev. This is the first time opts means something semantic rather than an optimisation level.

LLVM cannot fold the indirection away — the cell is an external mutable global — and a --dev build of calc-me keeps 46 indirect calls at -O2. The acceptance table now runs values, machine and sand-headless as dev builds as well; the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.

Names that did not exist when the process started

Editing a defvar or a defn is a symbol the host exports. Adding one is not: there is no symbol to bind to and ELF cannot grow one. Those go through runtime/flan_dev.c, which is two lookups and nothing else:

void **flan_dev_cell(const char *name);              /* a new function's cell */
void  *flan_dev_global(const char *name, uint64_t);  /* a new global's storage */

Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A new global's declared initial value travels with it, as a constant the runtime copies on the allocation and ignores on every call after: calloc alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the common case pays nothing for the general one.

The unit is a list of top-level forms, not one function — Emit.redefinition ~fns. C-c C-c passes one name, C-c C-k passes a file's worth, one code path either way. It has to be: v3 of the fixture adds extra and uses it from a redefined bump, and splitting that into two loads would leave a module referring to storage that does not exist yet.

Four rules, each of which is a silent failure if broken:

  • Every lookup resolves before any body is published. Publish first and a caller reaches a function whose slots are still null. Not race-testable, so it is asserted on the emitted flan_reload_install.
  • flan_dev_global refuses a size change. The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
  • Nothing is ever dlclosed. A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too.
  • The registry never moves. A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable.

The test that separates this from a plausible wrong version is v4, which redefines added — a name v3 introduced at run time. v3's bump is already installed and is not rebuilt, so it picks v4 up only if its call goes through a cell both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still pass and the transcript would read 246 instead of 432.

Sizes are spelled LLVM's way — ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64) — rather than by a layout calculator in OCaml that would have to agree with LLVM's on every target.

The agent — dev loop step 3

vendor/agent/ is a package like any other: agent.flan declares three calls, flan_agent.c implements them, link asks for -lpthread.

(agent/start path)   listen on a unix socket; once, at startup
(agent/poll)         install whatever has arrived; returns how many
(agent/wait ms)      the same, but waits for something first

The split between them is the design. dlopen relocates a module and takes the loader lock — milliseconds, unbounded — so it happens on the listener thread. flan_reload_install is one store per function and must not land while a redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks. The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the loader.

wait exists for tests. A test that races the frame rate fails on a loaded machine, so test/programs/agent.flan waits for the reload instead of sleeping past it. It takes two reloads, which is the daemon's actual loop: the first introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.

Two details found by running it:

  • stdout is line buffered, set in flan_rt_init. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found.
  • The reply goes out before the module is queued. The other way round, the game thread can install and the program can exit between the two, and the answer reaches the sender as a connection reset rather than as ok.
  • ok means queued, not installed. The sender does not get to know when the swap happened; only the program knows when it is between frames.

sand.flan calls agent/poll at the top of its loop, which is what step 3 was for. Verified: with sand running under Xvfb, flan reload sand-probe.flan game-draw and one line on the socket, and 455 consecutive frames drew from a body that did not exist when the process started. Building without --dev is fine — there are no cells, so a module is refused on the listener thread and the loop never notices.

flan reload <file.flan> <fn>... [-o out.so] [--new name,...] builds one module the way the daemon will. --new is the names the host was not built with; it is the one thing the command cannot work out for itself, and it is exactly what the session will track automatically.

The session

lib/session.ml is the program as a live thing: the declarations the running process was built from, plus every change accepted since.

Transactionality came for free and needed no machinery. Check.program builds a fresh environment from a declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the llc that follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.

Two things the session knows that no single evaluation could:

  • Which names the running process was built with. It comes from the checked program, not from any accumulated AST, because Check.program prepends the prelude and no AST contains it. Derive it from declarations and print-line reads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
  • What that process's memory looks like. Three changes are refused with a reason rather than loaded:
Change What it would have broken
a function's signature a cell is a bare ptr; every call site compiled before the change still passes the old arguments through it — and this is now a stopgap, see below
a global's type the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve
a struct's fields the values the process is holding have the old layout
a defconst's value, when the checker consumed it it is in the shape of the program — (defconst rows (/ h c)) decides grid's type before anything else resolves — so no store can reach it
a defenum member :space is erased to an i32 literal in the caller, so it is folded there too

The signature row is the one the plan has moved past. plan.org now says a signature-changing redefinition should make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while existing callers and stored Fn values keep the old version and stay safe, and the session warns at every tracked caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error. Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines (a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message should not be read as the final answer.

A defvar's initial value is deliberately not in that table. Its storage holds live state the program moved past long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same Tast.global record as a defconst, opposite answers, told apart by gconst. The enum comparison runs over declarations rather than the checked program, because Tast.program carries no enums at all — they are erased to i32 in the checker, which is the same fact that makes them unreloadable.

Note what the checker catches on its own: change helper's parameter type and the caller fails to type check first, loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program uses, which is exactly where the silent version lives. The fixtures carry an unused defvar and a C-called defn for that reason.

A defconst the checker never consumed is a different matter and can be changed: it is only ever bytes in memory. A dev build emits every defconst as a mutable global rather than a constant — so LLVM cannot fold a read of it and a module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That is how sand's colors gets tuned live while rows stays refused. Release builds emit constant and get all the folding back; Tast.global.gfolded is what tells the two apart, because nothing downstream of the checker could.

A form typed into a file that is imported as a package is qualified the way the import qualified it. poll in vendor/agent/agent.flan becomes agent/poll, and its call to poll-raw becomes agent/poll-raw — through Load's own qualify_decl, so the rule cannot drift from the one used at import time. Without this the form spliced as a brand-new unrelated name: the evaluation answered ok, and the running program went on calling the sim/settle it already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently did nothing.

It is derived from the file's path and not sent by the editor, which is where this departs from CIDER's ns key: a Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to either.

The accumulated list is the post-Load one, so an evaluated (import …) is spliced as its expansion. Otherwise re-evaluating a file that imports something appends a second import, Load expands it again, and the duplicate-name pass rejects it. C-c C-k on sand.flan's own text is the test.

flan reload <program.flan> <forms.flan> is that path from the command line: a session over the program the process was built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form game-draw and 910 consecutive frames drew it.

Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on every invocation, so if the program file has been edited since the process launched, its idea of which names the host has and what its memory looks like describes a binary that is not running. And Session.eval's origin defaults to <eval>, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass the real buffer path, which is the same key CIDER's eval carries.

The daemon — flan dev

flan dev <program.flan> holds one Session, builds the program, launches it, and listens on .flan-dev.sock beside the source. What it adds over flan reload is that the session persists — a defvar added by one evaluation is part of what the next one is checked against — and that it owns the build, which is what makes its layout rules describe the process that is actually running rather than a guess about it.

The protocol is s-expressions, not bencode. nREPL was the plan and the argument for it evaporated once the client became ours too: there is no CIDER to be compatible with, eval is string-in/string-out with no slot for which form, from which file, and Emacs already has read and prin1. So it is one sexp per message — no parsing code on the editor side, and on this side the parser is the language's own reader, where :op is already a keyword and a payload of Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains newlines. An nREPL front end can sit on the same Session later; it should not have gated the editor.

(:op "describe")                        → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
                                        → (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "defs")                            → (:status "ok" :defs ((name kind signature loc) …))
(:op "close")

defs is its own op rather than more fields on describe, because describe is what an editor polls — it is how the program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer. It is asked once on connect and again after each accepted install. Four strings an editor reads with read and nothing else: eldoc, completion and find-definition want the same three facts about a name. loc is empty where there is none to give, because only Tast.fn carries one — an editor must refuse rather than go looking for the definition itself, which in a program of several files finds the wrong one. Parameter names are not in the Tast, so a signature is step [i64 f32] i64: types only.

The daemon makes its own source path absolute before building, because every location it reports derives from it. flan dev src/game.flan run from a project root otherwise answered src/game.flan:12:7, which an editor can only resolve by guessing which directory it was relative to.

An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage — is accepted and answered with :note "nothing to install" rather than by shipping an empty module. Building one anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.

:file is not decoration: Session.eval's origin defaults to <eval>, so without it every error an editor shows points into a file that does not exist.

Two things the daemon must not paper over, both of which would look like a successful evaluation:

  • The agent socket is chosen by the daemon, not by the program. A program's source has to name some path — sand.flan says /tmp/flan-sand.sock — and the daemon overrides it through FLAN_AGENT_SOCKET before spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it.
  • Delivery is checked. agent/start returning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is not ok becomes an error the editor sees.

It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving.

The Emacs client

emacs/flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is most of the work: Flan is s-expressions, so sexp motion, paren matching, beginning-of-defun and indentation are already right. What it adds is Flan's own bracket syntax ([ and { are brackets, not symbol characters — every binding list and every type is written with them), the characters a Flan name may contain (-, ?, /, .), and its keywords.

emacs/flan-dev.el is the client. There is no parser in it, which is the point of the protocol choice: prin1 writes a request and read reads a reply.

C-c C-c the top-level form at point, recompiled and installed
C-c C-k the whole buffer, as one module
C-x C-e the expression before point, evaluated in the running program
C-c C-z / C-c C-q connect (finds .flan-dev.sock upward) / disconnect
C-c C-o the running program's own output, in *flan-output*
C-c C-r a prompt on the running program (*flan-repl*)
C-c C-b what a stopped program is offering, and which to take
C-c C-d what the running program currently defines
C-c C-a the code a name compiled to — amd64, or C-u for the LLVM IR
M-. / M-, where a name is written, through an xref backend

eldoc, completion-at-point and M-. all read one cached defs reply rather than asking per keystroke: eldoc fires on an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a defn just installed completes at once.

The modeline says whether there is a program on the other end, in three states. lost is a daemon that has gone away, which is ordinary rather than an error — flan dev ends when its program does — so the next request reconnects on the socket it was on. Strictly before a send, never after one: a connection that died mid-request may have died after the daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.

C-c C-k sends one module rather than a form at a time on purpose: a defvar and the function that uses it have to arrive in the same load, or the first refers to storage that does not exist yet.

Framing is in bytes and Emacs counts characters. Every length goes through string-bytes and the process is binary, or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload has of them — a bug that would look like a corrupt protocol and appear only for some users. test/test_emacs.ml drives the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a mistake in the framing, in beginning-of-defun over Flan's syntax table, or in the reply reader passes test_dev.ml and fails here.

An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the next time that buffer's evaluation is accepted. Two things had to be right first. The column in a :loc is a byte offset, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and forward-char with it put the marker as many columns right as the line had non-ASCII characters before it. And the daemon numbers lines from the start of what it was sent, so C-c C-c on a defn halfway down a buffer answered line 1 and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the reader skips, so the reply's line numbers are the buffer's own.

An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent success is indistinguishable from silent failure, and beginning-of-defun may well have found a different form from the one point looked like it was in.

Live disassembly — done. C-c C-a on a name writes the amd64 the running program's copy of it was assembled to into *flan-disassembly*; C-u C-c C-a writes the LLVM IR that body was built from. (:op "disassemble" :name … :form "asm"|"ir") is the op.

What makes it possible is that the daemon owns the build: it compiled every module it sent, so objdump -d --disassemble=flan.<name> on the right object is the disassembly and the retained .ll is the IR. Build.shared deletes its own .ll and Build.executable leaves the host's under a name that says nothing about which module it was, so the daemon now writes its own copy beside each .so and keeps the host's as host.ll — ten reloads in, nothing else on the machine still has that text. A table from function name to the last module accepted for it is the whole of the bookkeeping.

What it will not claim is that the code shown is installed, and this is the interesting half. The agent's socket takes a module path and five verbs; none of them reports an address, flan_dev_cell lives in the program's address space, and C-x C-e renders a pointer as <ptr> on purpose — so nothing the daemon can ask would tell it what a cell holds. The reply carries :basis saying which of three things is true, and the buffer prints it above the first instruction:

  • nothing has been delivered for this name, so the cell still holds the host's body — the one case that is certain;
  • a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
  • a module was delivered and the program is stopped — which says nothing either way about whether it installed, since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further installs until it resumes.

From SBCL: offsets from the function's own start rather than addresses into an object, and L0: labels on branch targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When the debug build lands, that is the line to delete.

Transient error overlays — done. The diagnostic ghost text is feedback about the evaluation that just failed, not an annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything. pre-command-hook and not post-command-hook, which fires at the end of the failing command and would clear the overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke in all of them. execute-kbd-macro runs no pre-command-hook under --batch, so the test drives run-hooks — the same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it is Emacs' contract and a test claiming to check it would be checking nothing.

The program's stdout is a pipe into the daemon, and whatever it printed since the last reply rides along with the next one into *flan-output*. Having it arrive with a reply rather than by a separate request is the point: the output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the accept loop's select, not only when an editor asks, and the buffer is capped so a program printing every frame cannot grow the daemon without limit.

C-x C-e — evaluating an expression

A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body into, so the expression is wrapped in a function with nowhere to be called from; the module says run this once by exporting flan_reload_call, and the agent calls it after the install — on the game thread, at a frame boundary, so an expression reading the program's state sees a point the program agrees is consistent.

Nothing is marshalled back, because nothing could be. A Flan value carries no header, so no code at run time can say what it is. The compiler knows the type and renders it there, in the thunk, into flan_dev_result. That is the layout decision's bill, and it is why the printer set is small rather than universal.

It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a dev-only feature must not put a branch in it — so flan_rt.c is untouched and the value is read back over the agent's socket. The read is safe without a handshake because flan_dev_result bumps a generation counter last; the daemon waits for it to move rather than assuming the program has reached a frame boundary.

This renderer is most of println, which is worth knowing before anyone schedules it. plan.org describes a compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints structs, fixed arrays and options structurally, prints a Ptr as its address rather than following it, and bounds depth and length. That is a description of what Session already does for C-x C-e — same walk, same refusals, same three bounds — aimed at flan_dev_emit and the wire instead of at stdout. What println needs on top is a stdout sink, a builtin that takes its printer from the argument's type, and the any/Error dynamic cases, which have no compile-time type to walk. Not the walk itself.

The renderer is a compile-time walk over the type, emitting a piece at a time through flan_dev_emit. Piecewise because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an allocator the language does not have.

big                       18446744073709551615
col                       :blue
(.pos b)                  (V {:x 1.5 :y 0})
b                         (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3)     [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid                  [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]

Details that are decisions rather than formatting:

  • u64 renders in C, with %llu. The language's own i64->bytes is signed, so it used to refuse rather than come back as -1 — but refusing a whole struct because one field is a u64 is much worse, so the runtime got an entry point instead.
  • Strings are quoted and escaped, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is.
  • An enum renders as :name, recovered from the checker's table as a chain of comparisons, because members are erased to i32 before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see.
  • A pointer is never followed<ptr>. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf.
  • Three separate bounds, easy to conflate. depth (4) and span (8) bound the walk, so [100 [100 u32]] does not become ten thousand render sites in one module. The output is bounded once in the runtime — emit truncates at 4K and end appends ... — because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget.
  • A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element.

What still refuses by name: Map, Fn, a type variable.

A caveat inherited from the language, not introduced here: 3.0 renders as 3, indistinguishable from the integer. flan run calc-me.flan "1.5 * 2.0" has always said 3.

An evaluation is not a declaration: the thunk is built against the program and never spliced into it, so describe does not fill up with eval/N for every expression ever typed.

The module is unloaded afterwards, which is the one case where that is safe. The thunk is called directly by flan_reload_call rather than through a cell, and it takes no registry slot — so once it has returned, nothing points into its text and the value it produced has been copied out. It declares that with @flan_reload_transient and the agent dlcloses it. Measured: sixteen expression evaluations retain zero mappings, where each redefinition retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind, and can never claim this.

Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it.

The test that matters is the same expression twice: the fixture increments ticks every frame, so two evaluations must disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.

The REPL buffer

flan-repl.el is a comint-mode buffer whose every line goes through the same eval-expr request C-x C-e uses. No new protocol and no compiler support. Deriving from comint rather than hand-rolling a prompt is the same call as deriving flan-mode from lisp-mode: history, the input ring and kill/yank already exist. There is no subprocess behind it — the "process" is a stub comint needs in order to have a prompt at all.

Three things about it that are decisions:

  • It is program-scoped. A name typed at the prompt resolves against the running program's top-level namespace, so in sand you write sim/settle and not settle. A buffer visiting a package's own file gets the alias applied for it because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
  • RET on a half-typed form opens a line instead of sending it. Balance is checked with the Flan syntax table, so a paren inside a string does not count.
  • A value and the program's output are different things and arrive by different routes. The value is the result of the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply and goes to *flan-output*. Showing them in one place would be convenient and wrong, so there is a test for the separation.

That test is what caught a real bug: the renderer's Unit case emitted () without evaluating the expression, so (print-line "x") — the most ordinary thing anyone types at a prompt — answered () while nothing happened. A Unit expression is almost always a call made for its effect, and is now evaluated and then reported.

Conditions — step 1: handler-bind and signal

spec-conditions.md §1 and §2, and nothing else yet. They are worth having on their own because neither alters control flow: signal returns Unit whatever it finds, a handler that returns normally leaves the signalling function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists yet, and no signature changed.

(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
  (load-all))

The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing function's own stack, and signal with an empty stack is a null check — which is what §2 asks for. Popping is by frame rather than by count, so restoring what this one displaced is right even if something below it left the stack out of step.

Three decisions worth keeping:

  • A condition's type is a hash of its name, not an index. An index would shift the moment a struct were added, and every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
  • The condition crosses as a pointer, because a handler runs while the signalling frame is still alive and there is nothing to copy. What the clause binds is the condition itself, though — the pointer is a hidden parameter and the name is a slot loaded from it, so a handler passing c to something expecting the struct is not handed an address instead.
  • A clause is lifted into a function of its own. A handler runs from wherever the signal was, so it cannot be a branch in the function that wrote it.
  • A pushed handler frame holds the clause's body address, not a cell. This is a deliberate divergence from plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a particular body. A handler frame is not a Fn value — nothing in the language can name it — and it is live only for the duration of the handler-bind body, so a reload landing while it is on the stack finds the clause it pushed still valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the stack does not observe a redefinition of its own clause; the next entry to the handler-bind pushes the new one. When Fn values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.

Which gives the two refusals, both by the house rule rather than by accident:

  • A handler cannot see the establishing function's locals. That is a closure with an explicit environment, so a reference to one is refused for that reason rather than reported as an unknown name. Globals and the condition are in scope, which is what the accumulation case needs.

What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not outlive the function that established it, so this is spec-memory.md's case 2 — a non-escaping fn capturing by value into a stack environment — and not the escaping closure that plan.org's open decision #5 defers until a concrete use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the biggest usability limit in conditions is not behind the thing that was just deferred.

  • return inside a handler-bind body is refused. The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape as defer inside a block.

The break loop — conditions step 3

Where "a crash kills the program" stops being true. An unhandled error runs a hook instead of rt_die(), on the frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.

flan: unhandled Missing — stopped, not dead.
  restart: retry
  restart: use-placeholder

Four decisions, each of which is the reason something is where it is:

  • It is a hook, not a call. The loop lives in vendor/agent/, which is an optional package; flan_rt.c is the release runtime and must not depend on something a program may never import. A program with no agent leaves the hook null and dies exactly as it did before.
  • The hook resumes by writing a restart into the transfer channel — the same channel an invoke-restart writes, reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same act, lowered once. Nothing about §6 needed changing to support it.
  • The break loop is the poll loop, run from the error instead of from the frame boundary. That is not a convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop that did not drain that queue would hang C-x C-e exactly when it is most wanted.
  • Installing while stopped is allowed, which contradicts a rule stated above and should. "A redefined function must not be swapped while it is on the stack" is about mid-frame consistency — half a frame of old code and half of new — and there is no frame in progress here. The old body on the stack keeps running; a retry restart calls through the cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of stopping.

A restart frame carries its name now, beside the hash. Matching never needs it — that is what the hash is for — but a break loop has to show someone their choices and nothing at run time can turn a hash back into a name. It is also compute-restarts' data, whenever that arrives.

A choice is validated on the listener thread, against a stack the stopped game thread is holding still, and refused there. Accepting it and discovering on the game thread that no frame offers it would answer ok for something that cannot happen.

The socket verbs are restarts, restart-at <n> [name], restart <name> and abort, and all four are refused with the reason when the program is not stopped — there is no restart stack to walk from a running one. restarts answers <index> <+|-> <name> per line: the index is the identity, and the flag says whether a transfer to that frame has anywhere to land. test/programs/break.flan errors three times — two restarts taken by name, then a shadowed pair where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved by name could pass.

The break loop in the editor — conditions step 3, the other half

The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it follows from one fact: a program stops at a moment nobody asked about. Every other op in the protocol answers a question an editor chose to ask.

So the state is learned twice, deliberately:

  • It rides on every reply, beside the program's output and for the same reason. :stopped t :condition "Missing", or :stopped nil. The likeliest instant for a program to stop is the one just after an evaluation — a body that now errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding out after the echo area had said the evaluation was fine.
  • And a timer asks anyway, once a second, with describe — the cheap op, which is also how the output pipe is drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies that never come says nothing. The timer never reconnects: flan-dev--request reopens a socket a restarted daemon left behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the lost state that exists to be seen. It also skips while another request is in flight — accept-process-output runs timers, so a poll firing inside a read would eat the reply that read was waiting for.

Three ops, and the annotation owns :stopped, not the ops — one place in the daemon decides whether the program is stopped, so the poll and the prompt cannot disagree.

(:op "break")                       → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
                                         :stopped t :condition "Missing")
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
(:op "restart" :name "retry")       → (:status "ok" :restart "retry" :note "accepted; …")
(:op "abort")                       → (:status "ok" :note "the program is exiting; …")

:restarts is positional — innermost first, duplicates kept — because the position is what restart-at takes. :unreachable names the positions that are on the list and cannot be chosen. restart-at's :name is optional and is not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a prompt cannot take a different restart than the one it showed.

break carries only the restart list, because it costs a second round trip to the program and is wanted only by someone about to choose from it.

ok from restart means accepted, not resumed. The choice is validated on the program's listener thread against the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and lets the next poll settle it.

The modeline is a fourth state, flan:stopped(Missing), before live: a stopped program looks exactly like a running one from anywhere else in Emacs. The prompt is a completing-read over the names with require-match, which is exactly right for a closed set the program computed, and abort is the last entry on that same list rather than a second key — it is the thing you pick when none of the restarts is the answer.

flan_agent_poll had to become re-entrant, and that is the one thing here that was a bug rather than an addition. A C-x C-e thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop cached head and tail and wrote tail back at the end, so the outer call rewound the index over everything the nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of breaks rather than a stumble. It now claims each job by advancing tail before running it, and re-reads both indices each time round. Still single-consumer: only the game thread writes tail, nesting included. test_dev.ml evaluates an expression that errors and resumes it, which fails against the old shape.

The agent grew one verb, status, answered in both states — running or stopped <condition>. Everything else the break loop offers is refused while running, and rightly; but the question an editor asks without already knowing had to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.

test/programs/dev-break.flan stops on its first frame, so the daemon meets a program that is already stopped — the state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way round, by installing a step that errors into a loop that calls it, fixes it while stopped, and then resumes: C-x C-e answering while the program sits in the break loop is checked there against the real client, not only in OCaml.

Conditions — step 2: restart-case and invoke-restart

spec-conditions.md §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a restart-case further out.

(defn fetch [n i32] i32
  (restart-case (middle n)          ; its value if nothing transfers
    (use-placeholder [] -1)
    (retry           [] 7)))

(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
  (fetch 2))                        ; -1

The channel is an out-parameter, as §6 now says: one ptr appended to every Flan signature, written by an invoke-restart and checked after every call. The return type stays what the source says, so the disassembly is the release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's slot and each frame only has to check and return early, which reuses the existing return path and with it §5's defers. Emit.signature was already the one place a signature is spelled, which is what made this a three-line change rather than a hunt.

Every function is transfer-transparent, release included — and that is the ABI, not a stopgap. A cell holds a bare pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and plan.org both now say the later optimisation may stop a function checking the channel, or pass the pointer straight through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform also means redefinition acquires no new refusal class.

The transfer target is the restart frame's own address, not a clause id. This is a correction to what the previous note settled. A static id has to be unique against every module a running program may later load, and a hash is only probably unique — two restart-cases colliding means the inner one silently catches a transfer aimed at the outer. The frame is an alloca in the function that offers it, so its address is exact, and it also says which clause, which is how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was. Re-entering a restart-case then works with nothing extra: each activation allocates its own frames, and §4's "innermost offering the name" is just the order of the walk.

Cleanup happens in landing blocks, one per region. A guard branches to the innermost open one, which pops whatever frames it established and either catches the transfer or forwards it outward:

  • a restart-case's pops its restart frames, compares the target against its own, and either runs that clause or puts the target back and goes on out;
  • a handler-bind's pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, and without it the handler stack would be left pointing into a frame that has gone;
  • the function's own runs its defers (§5) and returns early. errdefer does not run and never could: try/Result is still refused by name.

A single function-wide unwind block would have been wrong for the first two: a call inside a restart-case body would jump straight past the very form that was supposed to catch it.

The channel is cleared before any cleanup runs and put back after. A defer makes ordinary calls and each one is guarded; with the channel still set the first of them would branch straight back into the landing block it came from. Same reason the clause body starts with it null.

flan_signal takes the channel and passes it to each handler, and stops walking once one has written to it. That makes the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot cross a foreign frame" would make restart-case useless. It is also the only such frame: extern is Flan-to-C only and there are no function values yet, so nothing can call back into Flan across one.

Scope, each piece refused by name with its reason and a test on the reason:

  • restarts take no parameters. That covers §1's own load-texture example and skips argument marshalling and §3's runtime arity check.
  • TODO — CL-style interactive recovery. A stopped program should be able to offer a typed restart such as (use-value [value T] value) or (use-function [replacement (Fn ...)] ...), and the editor should show its signature and ask for the replacement before invoking it. That is the missing "this variable is None; what should I use instead?" path: named, hard-coded branches are useful for retry and skip, but not a substitute for an interactive value or alternate implementation. It needs restart argument marshalling and validation in the runtime, plus an editor protocol for entering and checking the supplied value/expression.
  • return inside a restart-case body, exactly as inside handler-bind: a bare ret skips the pops.
  • one restart-case offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a choice nothing in the source shows.
  • invoke-restart inside a defer. A defer is the cleanup a transfer runs on its way out, so a transfer starting there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a defer that reaches one through a call is trapped at run time by flan_transfer_fail, because nothing static could see it.
  • no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past everything. There is nowhere to resume, so there is nothing else to do.

(error c), §2. The same walk as signal, and the difference is entirely what happens when the walk ends: signal returns Unit and the signalling function carries on, error has type Never and stops. So only a transfer gets past it, which is why emit puts a guard after the call and then unreachable — and why flan_error cannot be marked noreturn, since it does return, on exactly one path. Being Never is also what lets it stand as a restart-case body's fall-through, which is the shape §1's load-texture example needs. test/programs/error.flan is the unhandled case: it cannot be an outputs row, because it does not exit 0.

flan_transfer_fail covers the ordinary return path as well as the unwind one: a defer that reaches an invoke-restart through a call traps either way, and the message names the rule rather than the path, since the rule is the same.

A lifted handler clause is named after the function it came out ofhandler/step/0/Missing — and is emitted by a redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable nor attributable, and a redefinition of a function containing a handler-bind failed in llc with an undefined value. A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.

test/test_dev.ml drives that path: a third evaluation redefines step to a restart-case whose frame is an alloca in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and crosses probe, which the host was compiled with. Those three do not meet anywhere else.

Two things found by writing it:

  • { ctx with in_handler = true } was a latent bug. ctx.slots and ctx.slot_tys are mutable, so a copy allocates the body's slots into a record the function never sees again and the indices collide. It was harmless only because no handler-bind body in the tests had a let in it. The flags are set on ctx and restored now.
  • test/reload_host.c had to learn the parameter. It calls flan.outer through an __asm__ label, which does not fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere else entirely.

test/programs/restarts.flan runs in the acceptance table at -O2, at -O0 and as a dev build. -O0 is not redundant here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each of those calls goes through a cell.

What is left

  • Editor comforts: completion, eldoc, jump-to-definition, error overlays.

Session identity is the daemon that owns the build. A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default.

Where build time goes

flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object cache is in, and it is now ~110ms:

Step Cost
frontend: read → parse → load → check → emit <10ms, below the timer
clang on the .ll 60ms — llc does the same codegen in 20ms
clang on flan_rt.c 40ms — now cached, paid once
link 20ms

Every C translation unit a build needs — the host shim and each package's shim — goes through Build.compile_c, which compiles to a .o under $TMPDIR/flan-objcache and reuses it. The key is a digest of the source text, the compiler (its path, size and mtime, so an upgrade invalidates without paying a clang --version subprocess per build), opts.opt and opts.target. The opt level has to be in there: the acceptance table builds the same programs at -O0 and -O2, and an -O2 object must not serve an -O0 build. The object is written to a temporary name and renamed into place, so two concurrent builds cannot see a half-written one.

Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never the cost. The cache is keyed by content, so it never needs invalidating by hand — rm -rf on the directory is only ever a disk-space decision.

The other cheap win is still open: skip the clang driver for the .ll (llc + link directly), worth another ~40ms. It is a subset of the dev path's machinery. Check llc's major version against clang's before relying on it — the emitted IR text is currently absorbed by the driver behind -Wno-override-module, and a version mismatch surfaces as IR parse errors.

Next — the REPL is the priority

Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place), and the dev loop is the thesis of the project, so it comes first. Staged so each step is runnable on its own — the failure mode is building a daemon and a protocol before knowing the reload primitive works.

  1. The reload primitive, measured. DoneEmit.redefinition, Build.shared, test/reload_host.c, ~19ms. See the section above.
  2. Indirection cells. DoneBuild.opts.dev / flan build --dev, flan_reload_install, runtime/flan_dev.c for names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above.
  3. The agent, in C. Donevendor/agent/, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above.
  4. The daemon and 5. the Emacs clientboth done, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same Session if something else ever needs to talk to it.

One decision left to settle before step 2, because both change codegen and are painful to retrofit:

  • Do cells cover globals, or only functions? Settled by step 1: functions only. A redefinition module declares every global external, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding a defvar to a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
  • What is a redefinition unit — one function, or a file? A file is much easier to make correct and is what load-file wants anyway; one function is what C-c C-c wants and is where the 16ms number comes from.

Deferred until after the dev loop:

  1. wasm32. Done, with one glued joint. flan build --target=wasm32-wasi produces a module, and test/programs/sand-headless.flan prints -2851001042534928384 under it — the same hash as native, byte for byte, at -O2 and at -O0. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared across targets, and it compares equal. values.flan and machine.flan run there too, which is where a 32-bit pointer would have shown. The acceptance table runs all four, and skips them by probing — it builds the smallest program and runs it — rather than by looking for a binary on PATH.

Three things this cost that were not in the old note:

  • The entry point is not main. wasi-libc's start code calls __main_argc_argv; clang renames C's argc/argv main to that, and the .ll Emit writes says @main literally. The link succeeds and the program traps on a signature-mismatched weak stub. Build.wasm_main_source is a two-line C shim that bridges it, and the __asm__("main") label in it is load-bearing: spelling the callee main makes clang rename that too and the shim becomes an infinite self-call.
  • The target has to reach the C compiles, not just the link. flan_rt.c includes <stdio.h>; without --sysroot it never gets that far. target_flags is computed once and passed to both, and the whole flag list — not just the triple — is in the object cache key, so repointing a sysroot cannot serve a stale .o.
  • Fedora's sysroot is one level deeper than wasi-sdk's: include/wasm32-wasi/stdio.h, not include/stdio.h. Both shapes count.

The glued joint, and the one thing this contradicts in the old note. The old note said the builtins archive has to come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it libcompiler_rt.a, and dropping that in as libclang_rt.builtins.a links and runs. It is a different triple (wasm32-unknown-emscripten) built by a different clang (22 against Fedora's 20), so it is substituting, and wasi-sdk is still the proper article. build.ml looks for FLAN_WASM_BUILTINS, then /opt/wasi-sdk/..., then emscripten's beside emcc on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real include symlinked in.

The runtime is Node. No wasmtime and no wasmer on this machine; test/wasm-run.mjs is twenty lines of node:wasi and the table prefers wasmtime or wasmer if either appears. --no-warnings, because node:wasi writes an ExperimentalWarning to stderr on every run and the harness compares combined output.

Refused by name, not half-supported: --dev with a wasm target (the reload path is dlopen), Build.shared with one (same reason), and flan run --target= (a .wasm is not something this host execs — build it and point a runtime at it).

Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. wasi-sdk is right for the headless table; it is not necessarily right for the eventual game build.

  1. Loose ends from milestone 4, none of them blocking: block-scoped defer; package visibility, so rl/get-color-raw is not callable; a package importing a package; imported unions.

Macros — the reader and the declaration are in, the expander is not

The front half landed. What exists:

  • The reader reads `x, ~x and ~@x as (quasiquote x), (unquote x) and (unquote-splicing x), exactly as 'x reads as (quote x). It stays dumb: it does not count nesting levels, does not know whether an unquote is inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma is whitespace in is_delimiter and every binding vector in the corpus relies on that. Backtick and tilde are delimiters now, so a~b is two things.
  • parse.ml refuses all four by name. quasiquote and gensym say expansion is not wired up; unquote and unquote-splicing say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature. (defmacro name [params] body ...) at the top level is checked for shape and then refused — a malformed defmacro and an unimplemented one get different reasons, so the shape rule is enforced before the feature exists.

Nothing is stored. There is deliberately no macro table and no Ast.Defmacro, because a table nothing reads is a place for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit.

How the expander should work

There is no interpreter (see "Why there is no interpreter") and there is not going to be one, so running a macro at compile time means compiling it and loading it into the compiler. That machinery already exists and is measured: Emit.redefinitionBuild.shareddlopen is ~19ms end to end, with the load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.

The shape it wants:

  1. A macro is a function [Form] -> Form. Its parameters are forms and its result is a form, which means Form.t has to exist on the Flan side — a defunion mirroring lib/form.ml, in the prelude, plus constructors and accessors. That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on the layout of a Form, not merely its shape, so whatever the checker does for unions has to be exact here. Until unions are values this cannot start — check.ml puts union values and match on a union at milestone 6, so that is milestone 6 work landing before milestone 5's.
  2. Expansion runs over Form, before Parse. Not a pass over Ast: there is no Ast.Defmacro and Parse refuses defmacro outright, so an Ast-level pass would have nothing to work with. That refusal is not a dead end, it is the ordering — the expander runs first and Parse never sees a macro call at all. It is also the Clojure ordering, and the reason a macro expanding to a special form is ordinary rather than a special case.
  3. Order matters and files do not have one. Top-level names in a package are order-independent everywhere else (declared_types, the constant fixpoint in check.ml). Macros cannot be: a macro must be compiled and loaded before a call to it is expanded. Either collect every defmacro in a pre-pass and compile them as one module, or require definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest of the frontend already behaves.
  4. A macro's own body may call macros, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be detected and named rather than looping.
  5. gensym is a runtime function of the compiler, called by the loaded macro while it runs. It needs a counter that lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org, open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit gensym, no macrolet until a concrete use case appears.
  6. Quasiquote itself is a macro-shaped desugaring, not a compiler feature: `(a ~b) becomes list-construction over quoted pieces, with ~@ splicing. Written once, in the expander, over Form.

The four files this touches — build.ml, check.ml, emit.ml, load.ml — were owned by other lanes when the front half landed, which is the only reason the expander is not here too.

What would tell you it works

when, unless, until, cond and dotimes are special forms in parse.ml today, and plan.org milestone 5 says they are special forms only until macros land. Moving one of them out of the compiler and into the prelude as a defmacro, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote, gensym and the ordering pre-pass at once, against a test suite written before any of them existed.

Watch for

The rule that caught the two misparse bugs applies unchanged: anything that binds a name, alters control flow, or is not yet implemented must be recognised explicitly and rejected if unsupported. check.ml rejects Vec, Map, Result/try, union values, closures, quoted symbols, generics and function values by name, each with the milestone it belongs to; load.ml rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The tests assert on the reason, not just on the failure.

Untracked on purpose

calc-me and sand, the executables flan build drops beside their sources, are now in .gitignore — anchored (/calc-me, /sand) so the patterns cannot match anything nested.

old-ocaml/ — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root dune file. Its contents are also in git history at 2c232dd.