BUILT.md gains "The header is read now", directly under the section whose last paragraph promised that reading a header was what would convert the trusted half into a checked one and that it was not built. That sentence is replaced by a pointer to the one below it, in BUILT.md and in shim.ml's docstring both. It records the things worth not re-deriving: why the dump and not libclang (and that Zig left libclang too, which strengthens the argument rather than weakening it), why the import is bounded by the package's own defstructs, why generating defstructs would make the check circular in exactly the way a _Static_assert was rejected for, refusal-by-demotion from Zig's failDecl, the naming rule and what it must actually guarantee, and both const-vs-non-const char * and the target-varying widths. The diff and the costs are stated as measurements, with the table: 16 of 16 defstructs and 172 of 172 declare-c agree against 5.5, ten real differences against 5.1-dev, release +4ms warm, redefinition 31.0 -> 46.5ms. DISCUSS.md item 6 is rewritten rather than removed. The mechanism question is settled and is now in BUILT.md; what is left is narrower and is two decisions that are the author's — whether the header stays a build-time read or becomes a committed generator, and whether the 172 hand-written lines migrate. Both have the argument on each side written out, including what migration would lose: key-pressed? is a better name than is-key-pressed, and an enum parameter imports as i32 because nothing tells the importer the package calls KeyboardKey "Key".
169 KiB
How the parts that exist work
The reasons behind the code, kept separate from NEXT.md so that what to do next is not buried under what was already
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever dlclosed, why a call
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
than a function in the runtime. Deleting it would mean deriving it again.
NEXT.md is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
first. Come here when you need to know why something is the shape it is.
What milestone 4 added
dotimes desugars in check.ml to a Let plus a While — no new IR node. The bound is evaluated once into a
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
assignable, which makes the generated step its only writer.
defer is recognised in check_fn and nowhere else, because that is the only place that knows a form is at the top
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
stands. Function exit runs them innermost-first, and an explicit return runs the ones registered above it — a defer
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
bounds-check shape (noreturn then unreachable) rather than being a separate decision.
defer inside a let, a loop or a branch is rejected, not accepted with function scope. It would run once at
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
Block-scoped defer is real work and is not done.
New builtins: zeroed (takes its type from the place it is stored into), min/max (each operand through a slot,
so neither is evaluated twice), bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a signed
type and logical on an unsigned one), and rand-f32.
rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state. It is not libc's, because a grid hash is
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed sets
the state. This is what the bitwise operators were added for.
Enums and keywords. (defenum Name [member value ...]) gives a type that is an i32 at run time and its own type
in the checker, so :space at a call site resolves against the parameter's enum and a typo is an error there rather
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
Why the FFI goes through a C shim
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
per-target classification the caller has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
in emit.ml is three classifiers to write and then keep correct forever, and a mistake shows up as (.y m) returning
garbage rather than as a link error.
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
copy. clang classifies all of it, per target, for free. check.ml enforces the rule — an aggregate in a declare
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
ABI, implemented twice", and flan_rt.c is the same pattern.
The wrappers are generated now — declare-c, lib/shim.ml
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
generate "if that ever becomes the bottleneck" came true at 84 of them. vendor/raylib/shim.c is gone; the directory
holds raylib.flan and link and no C at all.
One binding is now one line:
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
declare-c names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
any other: the typedefs for the structs involved, made from the Flan defstructs; the extern prototype in the
function's true signature; the wrapper that flattens it; and the flattened declare the Flan side calls, with an
ordinary Flan defn above it when the signature has a struct in it. flan shim <file> prints the whole file.
It is a second form and not a change to declare, for one reason worth remembering: (declare start-raw [path string] i32 "flan_agent_start") in vendor/agent means the symbol takes ptr+len, and (declare-c init-window [w i32 h i32 title string] "InitWindow") means it takes a NUL-terminated char *. Same shape, opposite claims, so no structural
rule can separate them. declare is untouched, and sqrtf and the agent still work unedited.
What the generator guarantees, and what it trusts. Guaranteed: the C typedef and the Flan struct come from the same
defstruct, so they cannot disagree — permute the defstruct and the typedef permutes with it, which is exactly what
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
that the defstruct matches the library's real struct, and that the declare-c signature is the function's real
signature — no header is read, deliberately, so nothing can check either. A _Static_assert on sizeof/offsetof was
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
for every field type the generator admits — the machine integers, the two floats, bool, a pointer and a nested struct
— LLVM's layout is C's, and emit.ml writes no datalayout, so clang applies the target's rules to both halves.
Everything where they could diverge is refused at the field.
One thing got sharper and should be said plainly: the prototype is now generated from the declaration, so a
scalar's width carries ABI weight it did not before. f64 where raylib says float used to be narrowed by clang at the
hand-written call site; now it emits double and raylib reads garbage. All 84 migrated prototypes were diffed against
the deleted shim.c's — which was the ground truth for the true signatures — and agree.
Strings. The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, PATH_MAX for a path,
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
malloc failure, where the alternative is handing C a null pointer.
Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C. collision-point-poly? takes a slice
and collision-lines answers with an Option; neither is raylib's own signature. A slice parameter in a declare-c is
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
recoverable from [T] — so that one declares (Ptr Vector2) with an explicit count i32 and the Flan wrapper passes
(addr (at points 0)) and (len points). Every other refusal — an Option, a union, a fixed array, a map, a returned
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
the reason, and the acceptance table asserts on the reasons.
Known edge, not fixed: a REPL redefinition that introduces a new declare-c cannot work. Build.shared is llc +
ld -shared and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
that calls an existing binding is unaffected.
raylib.flan carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
(rl/get-mouse-position) returning a Vector2. Verified end to end, headless: GetColor(0x11223344) comes back as 17 34 51 68, four separate bytes — a Color is not the little-endian reading of the packed integer, so an identity
would have passed a weaker test. That case is in the acceptance table, skipped if libraylib is not installed.
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven collision-* predicates; textures; the Image family; Camera2D; RenderTexture2D; the whole audio surface (device, Wave, Sound, Music); fonts and glyphs; and gamepads, touch and gestures — plus the Key, MouseButton, TraceLogLevel, GamepadButton, GamepadAxis and Gesture enums, raylib's own named colour palette, and the FLAG_ window hints. Adding one is a single declare-c line; there is no C to write.
Two things the ported examples in examples/ wanted and could not have, both refused for reasons that are right. GetGamepadName returns a char * into raylib's static storage: the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that returns one returns something Flan has no owner for. And an enum parameter cannot be indexed — GetGamepadAxisMovement takes a GamepadAxis, a loop variable is an i32, expected rl/GamepadAxis, found i32, and a second declare-c of the same symbol with an i32 face is refused too: one declare-c per C function, and another Flan name for it is a defn — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a cond over the members it knows.
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
GPU: GetCollisionRec, which pins Rectangle completely, and SetShapesTexture's default substitution, which pins
Texture2D's id and format and nothing else. Handing a struct over and reading it back proves nothing at all —
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. width,
height and mipmaps are therefore checked only by looking at sand.flan running, which draws the brush sprite four
ways for that reason.
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
library being linkable and not on raylib-devel. vendor/raylib/link carries -l:libraylib.so.550 because Fedora
ships the runtime library without the .so symlink.
The header is read now — headers, lib/cimport.ml
The section above ends by naming what the generator trusts: that the
defstruct matches the library's real struct, and that the declare-c
signature is the function's real signature. "No header is read, deliberately,
so nothing can check either." A header is read now, and both are checked.
The dependency, which is the crux, and which this project already answered
once. Zig's old @cImport ran clang as a library. That is exactly the
dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++
library breaks routinely on upgrade, a binary on PATH does not. So this shells
out for clang -Xclang -ast-dump=json -fsyntax-only, which is the same binary
every build already runs and adds nothing that is not already being paid for.
lib/cjson.ml is enough JSON to read that dump and no more, so it adds no opam
package either.
A note worth recording, because it strengthens the argument rather than
weakening it: Zig has since abandoned clang here too. translate_c.zig is
gone; lib/compiler/translate-c/ is built on Aro, a C frontend written in Zig.
Their reason was to ship a compiler containing no clang at all — the opposite
premise to this one, where clang on PATH is the toolchain assumption. Both
projects walked away from linking libclang; only the destination differs.
What is imported: functions, and only functions. Not structs, not enums,
not macros. The bound on how much is not a curated list but the package's own
defstructs — a function whose signature mentions a struct the package has not
described is refused with that reason, so vendor/raylib describing thirteen
structs is what makes the import thirteen structs wide, and describing a
fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are
refused, and 172 are left alone because the package already binds them by hand.
Not generating defstructs is what makes the check possible at all. Generate
them and the header becomes the authority on layout, and comparing the
package's layouts against the header's would be comparing the header with
itself. A _Static_assert on sizeof/offsetof was rejected in the section
above as circular for exactly that reason; this is not circular, because the
two sides have different authors. It is the cheapest real closure of the
trusted-not-guaranteed gap.
Refusing by demotion, which is the one thing taken wholesale from Zig. Zig's
translator never drops a declaration it cannot handle: failDecl binds the name
to a @compileError carrying the reason, so the name still exists, the program
still compiles, and asking for that one name fails at the use site with the
reason. A wholesale import has a hundred and fifty refusals and a caller cares
about the one they typed. Flan already had that mechanism — Load.refuse_hidden,
built for main — so rl/get-gamepad-name is not a name, and a program that
writes it is told the return type is a string, and a string only crosses as a
parameter rather than "unknown name".
That is also the split shim.ml needed and did not have. It refuses through
Loc.fail, which is right when a human named one function and wrong for a
wholesale import, where one returned const char * would kill the header. Same
judgement, different disposition; a hand-written declare-c still hard-fails
and shim.ml is untouched.
Two C spellings mean something in a parameter that they mean nowhere else.
const char * is a string going in, and the generator already knows how to hand
one over. char * without the const is very often a buffer the callee writes,
and handing it a NUL-terminated temporary would lose the writes with no
diagnostic anywhere — const is the only thing in the header that separates the
two, so it is what decides, and an out-buffer keeps a hand-written binding
saying (Ptr u8). long, size_t and the rest are refused rather than
guessed, and for a reason specific to this project: it builds for x86-64, for
wasm32-wasi and for the browser, and long is 64 bits on the first and 32 on
the others, so a guess would be right for the target that gets tested and
silently wrong for two that do not.
Naming. rl/InitWindow is rl/init-window. Reversibility is not a property
of the rule — the C symbol is stored verbatim in the declaration, so the wrapper
reads the library's spelling rather than reconstructing it. What the rule must
be is injective over one header, since two C functions arriving under one Flan
name would surface as a duplicate declaration about a name nobody wrote. A
boundary goes before an uppercase letter after a lowercase one, before an
uppercase letter between an uppercase and a lowercase, and before a digit after
a lowercase; nowhere else. So SetTargetFPS is set-target-fps and not
set-target-f-p-s, BeginMode2D is begin-mode-2d, UnloadUTF8 is
unload-utf8. raylib's 581 names are injective under it. When two do collide,
neither takes the name — resolving by order would mean that moving two lines
in somebody else's header silently rebinds a name a program is already calling.
Both are refused, both say why, and the author binds the one they want with a
declare-c.
Where it runs. headers beside link in the package directory, read the
same way: a path, any clang flags it needs, ${NAME} expanded from the
environment. What comes back is ordinary declare-c declarations, generated
before the package's names are qualified, so they arrive as rl/… exactly like
the hand-written ones and nothing downstream can tell which is which. No new
form, no new decl_kind, no reader or parser change. A C symbol the package
already binds by hand is left alone, so declare-c stays the escape hatch and
stays the thing that wins.
A leading ? makes a line optional, and vendor/raylib uses it. "No raylib
headers are needed" is a real property — a build needs libraylib linkable, not
raylib-devel installed — and requiring a header would take it from everyone in
order to give the check to whoever has one. Unset FLAN_RAYLIB_H and the build
is exactly what it was; set it and every signature is checked. A path that is
set and wrong is an error naming it, because silently behaving as though
nobody had opted in is the difference between an opt-in and a trap.
What the diff found
The evidence the whole lane exists for. Against raylib 5.5 — the version
whose .so link names — all 16 defstructs and all 172 hand-written
declare-c agree exactly. The half BUILT.md called trusted is now checked,
and it was right.
That is only worth stating because the check has teeth. Against the 5.1-dev
header installed in /usr/local it reports ten differences: nine functions that
version does not have (CheckCollisionCircleLine, the six Is*Valid renames,
DrawRectangleRoundedLinesEx) and DrawRectangleRoundedLines, which gained a
parameter. Picking the wrong header is therefore loud, which matters, because
the two headers are on the same machine and only one matches the linked library.
Both comparisons run at build time and stop the build, not just in the tool.
Verified by breaking them: a permuted Texture2D fails naming the field that
moved, and f64 where raylib says float fails naming the parameter — which is
the hazard the section above calls out by name and says only a test can catch.
The message points at the line in raylib.flan, not at the header.
flan import-c <header> [package.flan…] prints what it would produce, what it
refused and why, and both comparisons, without building anything. That also
makes "generate once and commit the result" available for the cost of a
printer — explicit in the source, checked against reality, no header read at
build time.
What it costs, measured
The number that decides how much to import, because reach.ml was the reason to
think a wholesale import could be free.
| today (172 by hand) | + 256 imported | |
|---|---|---|
| release build, cold | 0.298s | 0.312s |
| release build, warm | 0.078s | 0.082s |
redefinition (flan reload) |
31.0ms | 46.5ms |
| dev build, cold | 0.649s | 0.982s |
Reach.link already drops a generated wrapper whose declaration nothing
reachable calls, and that is what makes the release column nearly flat.
Confirmed on the case it exists for: a wasm32-wasi build of a program that
imports raylib and calls none of it still links without libraylib, with 256
extra declarations in play. Dev builds are not pruned, on purpose, so one
compiles all 428 wrappers — once, at session start, since Build.shared is
llc + ld -shared and compiles no C.
Reading the header is cached, and the cache earned itself against a measurement rather than a guess: 64ms of a 72ms check, against 8ms for the whole program without it. What is cached is the extracted signatures and not clang's JSON, because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest is reading it. Keyed the way the object cache is keyed, on everything that could change the answer: the header's path, size and mtime, the full flag list, and a format version, since the value is marshalled. That takes the delta to 17ms.
The 15.5ms on redefinition is the real cost and it is the argument against importing at build time, on the branch where the dev loop is the priority. It is the strongest case for the third option — generate from the header, commit the result, regenerate when the library moves — and that decision is open.
What a headless FFI test can and cannot pin
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
-
A struct round trip is worthless. Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C writes and reads the same wrong slots and the test passes for any field order. Found by permuting two fields and getting identical output.
-
Axis-aligned geometry cannot pin
Vector2. Exchangingxandyis a reflection, applied to the inputs on the way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, passes with the fields swapped — verified by swapping thedefstruct, which is what the typedef is now made from. Distances are worse: the reflection does not even reach them. -
What does pin
Vector2is the rotated camera, because a 90-degree rotation is not axis-aligned and therefore does not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision cases look like they cover it. -
What pins
Rectangleis arithmetic on its fields —GetCollisionReccomputes four numbers from four different field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged. -
Scalars in, fields out is the strongest shape there is, and the Image family is where it was finally available.
GenImageColor(4, 2, colour)is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values in four adjacenti32slots, with no input struct for a permutation to cancel against. That pinsImagecompletely, including thatdatais present and first;Texture2Dcould never be pinned that way because nothing without a GPU reads its width, height or mipmaps at all. -
A non-square image is an axis discriminator.
GetImageColorindexesy*width + x, so on a 4-wide, 2-tall image(3,0)exists and its transpose does not: exchange the wrapper'sxandyand the read goes out of bounds and answers transparent black.ImageFlipHorizontalagainstImageFlipVerticalsays the same thing twice more. -
A file is external ground truth, so
ExportImagethenLoadImageis not the symmetric round trip the rest of the package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by thex/ypermutation above.
The rule that falls out: make raylib compute something whose answer differs per axis, then verify the test can fail by permuting the fields and watching it go red. A case not verified that way is decoration.
One correction to an assumption that has now cost two lanes a guess: MeasureText is not headless material. It
measures with the default font, LoadFontDefault is not exported, and nothing but InitWindow loads it — so with no
window it answers 0 for every string. Measured against libraylib.so.550, not reasoned about. GetFrameTime, GetTime
and GetScreenWidth/Height are all 0 headless for the same kind of reason. All five are bound, and all five are
exercised by running sand.flan and looking, which is the whole of what can be claimed for them.
Packages
lib/load.ml resolves (import rl "vendor:raylib") before the checker runs. The directory is the package; vendor: is
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
collection is relative to the importing file. Importing is a rename: every top-level name the package declares
becomes alias/name, and every use of one — in a type, in a body, in a struct literal, in an array length — is
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
declarations whose names contain a slash.
A package may also carry the C it binds to: every .c file in the directory is compiled into the build, and a file
named link lists extra linker arguments. Whether those reach the build at all is decided after checking — see below.
A package may be a single .flan file named outright, rather than a directory. That is for the program that is also
a library: sand.flan shares the repository root with three other loose programs, so naming its directory would import
all four. A file carries no .c and no link file; those belong to a directory.
A package may import a package. The qualification flattens to the inner alias — raylib imported by a package that
is itself imported is still rl/…, never sand/rl/… — because a directory reached along two routes has to arrive under
one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which
is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time,
and the namespace being flat, mutually dependent packages simply work. The same directory under two different aliases
is refused.
Visibility is one rule: main is not exported. A package carrying one would collide with the importer's the moment
anything imported it, so a program could never be a package; and main is a reachability root, so an imported one would
keep everything it calls alive. Writing sand/main is refused at the line that wrote it, with the reason — left to the
checker it would be "unknown name", which is true and useless.
Still missing: a package-private marker for anything other than main, which is why rl/get-color-raw is callable.
The link follows the program
Load used to hand a package's .c files and link arguments to the build the moment it was imported, whatever the
importing program did with them. So anything naming vendor:raylib linked libraylib on every target, and on wasm32 that
link cannot succeed.
lib/reach.ml answers it from the checked program instead. Start at main and at the globals that run before it,
follow every call — including the Handled frames, where a lifted handler clause is reached by address and by nothing
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
wasm-ld would fail on the symbols rather than on the argument list. So the same walk prunes functions and externs
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
dropped one is a silently different program.
Dev builds are not pruned. What a REPL may redefine next is not a function of what has been called so far.
The filtering happens at the call sites — bin/main.ml, the tests — because Build.executable receives csrcs and
lflags from its caller and never sees the import list. Reach.link returns the pruned program and its C and linker
arguments together, so a caller cannot take one without the other.
sand.flan is one program
It was two files, and only ever for the reason above: the headless run is the one CI does on native and wasm32, and a
program that imported raylib linked libraylib whatever its main did. So the simulation lived in sand-sim/ and both
drivers imported it.
Now sand.flan holds the simulation and the raylib front-end, and test/programs/sand-headless.flan imports
sand.flan itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
raylib; sand.flan's main is not exported, so the only main is the headless one; and the hash is unchanged on both
targets at -O2 and -O0, which is the point. A refactor that moved that number would have moved the simulation.
What still justifies two entry points is smaller and stands on its own: the headless test needs no window and no
input on any target. sand.flan could not be that test even natively — with no mouse the grid stays empty and
settle and move-grain never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
Two claims that got run together in an earlier note, for the record:
- raylib does not work on wasm — false. It works through emscripten. What is true is that it does not work on the wasi path, which is what the headless table targets, and which has no GL and no browser.
- a game loop cannot be expressed on wasm — false. The browser cannot be blocked, so a web build drives the loop with
emscripten_set_main_loopinstead of awhile.That is a differentThe premise held and the conclusion did not. It is the samemain, not a different program.mainand the same program:-sASYNCIFYanswers the same browser fact without cutting anything in half. See "The browser is the third target" below.
Three edits were made to sand.flan's own text when it was ported, and they are language decisions rather than fixes:
(defconst gravity 0.05)→(defconst gravity f32 0.05). An untyped float constant isf64,velocityis[f32], and there is no implicit widening.(defvar current-color u32)→i32. It is an index intocolors, and(len colors)is ani32.(defn main [])is unchanged — the short form, as plan.org says.
Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
Bounds checks — done at milestone 3
at and slice emit icmp → br → cold block → call → unreachable; a failure names the source location. Three
check sites: at on [n T] (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
never reaches emit, check.ml rejects it), at on a slice or string (runtime len), and slice (two comparisons — lo <= hi is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
Build.opts.checks is on by default and not tied to opts.opt, which is what lets the acceptance table run the
same programs at -O0 and -O2 with identical checks. The flag is --no-bounds-checks.
The write path is its own case: (set (at arr n) …) lowers through place/Pindex, not through At, so a refactor
that split them would break the write check silently. The test covers both.
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.
Sanitizers, and why hand-written IR does not get them for free
--sanitize builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last
of those is not what passing -fsanitize=address to the clang run over the .ll gets you, and the gap is silent.
AddressSanitizer is an LLVM pass, but it instruments only functions carrying the sanitize_address attribute. That
attribute is put there by clang's C frontend. Emit writes .ll by hand, so it wrote none, so the pass walked past
every Flan function and instrumented flan_rt.c. The measurement that settled it: an out-of-bounds read of a defvar
array in a --no-bounds-checks build printed its garbage and exited 0; with an attributes #0 = { sanitize_address }
group named on every define, the same program reports global-buffer-overflow in flan.main. Globals are the exception
— the module pass redzones them whether or not any function is attributed — which is why the shape of a sanitized
build looked right long before it worked.
UndefinedBehaviorSanitizer has no equivalent lever. Its checks are not a pass: the C frontend emits branches to
__ubsan_handle_* inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and
(<< 1 32) is still unremarked under -fsanitize=undefined. Shift UB, alignment and the f32→i32 cast on NaN are
therefore a compiler feature if they are wanted — checks emitted from Emit behind the flag, the same shape the bounds
checks already have — and not a flag away. test_sanitize pins both halves with controls: one program that must report
and one that must not, so either fact changing is a test failure rather than a discovery.
-fno-sanitize=signed-integer-overflow is the only exclusion, because wrapping is what this language's arithmetic
means and without it every program trips on its first +.
The flag deliberately does not force -O0, unlike --debug, whose reason (mem2reg deletes the alloca a
llvm.dbg.declare describes) does not apply. The optimiser is half of what is being measured, and bounds.flan proves
it: with checks off, its read past the end of a string constant is reported at -O0 and silent at -O2, because an
out-of-bounds inbounds getelementptr into a constant is poison and LLVM folds the load away. The program then prints a
wrong answer instead of touching memory. Same family as (<< 1 32) compiling to a bare retq.
What ASan covers of the bounds checks' job, since --sanitize --no-bounds-checks is the run that asks. Three of
bounds.flan's six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is
layout rather than anything about the access: ASan lays a global out as {data, redzone}, so reading before one lands
in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is.
Measured both ways — silent in bounds.flan, reported as soon as another defvar is declared in front of arr. A
reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees
out-of-object access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical
bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a
second net, not a replacement.
The sweep lives on its own dune alias rather than on dune test: a sanitized program links to a statically linked 1.8MB
binary, and twenty-eight of them twice over is minutes against the suite's seconds.
Why there is no interpreter
Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.
The layout, which is the whole backend design
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
f32 f64 float double
bool i1
an enum i32
[T] and string { ptr, i64 } ptr+len, non-owning
[n T] [n x T] inline, a value
(Ptr T) ptr opaque pointers
(Option T) { i8, T } tag 0 None, 1 Some
a struct a literal struct, declaration order
Unit and Never {}
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:
- Every slot is an
alloca. Reading a local is aload, assigning is astore, and astoreof an aggregate is the copyspec-memory.mdrequires.addrof a local is then just the alloca, andmem2regremoves the ones nobody addressed.test/programs/values.flanpins this down. - A place is a pointer, a value is a load from it.
(set (.pos c) …)through a(Ptr Cursor)becomes agetelementptron the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
Non-local exit is lowered explicitly: return, some and a failed bounds check are branches, never platform unwinding,
so wasm32 needs no exception proposal.
The reload primitive — dev loop steps 1 and 2, measured
llc → ld -shared → dlopen → call, with no protocol and no daemon. dune test runs it: one function is recompiled
into its own object and called inside a process that is already running, twice, with a changed body the second time.
| Step | Cost |
|---|---|
Emit.redefinition |
below the timer (<0.1ms) |
llc -O2 -filetype=obj |
15–17ms |
ld -shared |
3ms |
dlopen + dlsym |
0.04ms |
~19ms end to end, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
number from this codebase. For contrast, clang -shared on the same IR is 50ms — the driver is again most of the cost,
which is why the dev path skips it. llc and clang are both 20.1.8 here; check that before trusting the .ll, since
the driver absorbs IR the bare tools reject.
ld -shared rather than clang -shared for a second reason: a shared object is allowed undefined symbols, and that
is the mechanism. What the new module does not define is the whole design:
- a global is
external. This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand'sgridwould reset on every reload, and "edit the code, keep the sand" is the thesis. - every other function is a
declare, so a redefinedsettlecalls the host'smove-grainrather than freezing a private copy of it. - no
main. This module is loaded, not started.
Its string constants still come along; omitting them is an undefined @.str.N at link time, and it is easy to miss
because a one-function module usually has none. Emit.signature is now the single place a function's LLVM signature is
spelled, because a define here and a declare there drift the moment one of them grows a case for Unit or for a
slice parameter.
flan_dev.c is compiled into every build, not only a dev one. Nothing in a release build calls into it — the
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
agent package's C refers to it, and a package's C sources are collected whatever main does. Leaving it out of release
builds made flan build sand.flan fail at the link with an undefined flan_dev_result_read, which reads as a compiler
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; -rdynamic and
the cells are still what --dev means. test_agent.ml links the agent program both ways for this reason.
-rdynamic is load-bearing. A normal executable exports nothing: nm -D calc-me | grep 'flan\.' is empty, so a
loaded module's declares would have nothing to bind to. The test passes it through lflags, which keeps it a property
of the dev build rather than of every build. dlsym on "flan.bump" works — a dot is legal in an ELF symbol.
Two things about the test are deliberate and are what make it prove anything: both loads happen in one process,
since two runs would pass while saying nothing about an in-process swap; and the versions are two paths, since
dlopen caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
helper is (* x 2) in one fixture and (* x 3) in the other: the second body is dead text, since the module declares
helper rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
String constants are emitted private unnamed_addr, so the module's own @.str.N cannot be interposed by the host's —
worth knowing, because with external linkage a redefined function would silently print the old text and nothing would
fail at link time. The fixtures each print a literal so that path is actually exercised.
Cells — how a call site follows a redefinition
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so a dev build routes every Flan-to-Flan call through a cell: a mutable global holding the address of the function that is current.
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
%p = load ptr, ptr @"flan.cell.bump" ; every call site
%r = call i64 %p()
Redefinition is then one store. A redefinition module declares the cells external, exactly like the globals, and
exposes flan_reload_install() that stores its own body into its own cell — cost below a microsecond, which is what
makes a frame-boundary swap a non-event.
The cell load is emitted after the arguments, so a redefinition landing between two calls cannot land in the middle of one.
Four things about this that are not free choices:
flan_reload_installis a named function and not an ELF constructor. A constructor runs duringdlopen, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose.- A redefinition's own body is
hidden. Default visibility in a shared object is interposable, and that applies to taking the address too: plain@"flan.bump"inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent. - This also fixes the self-call edge, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not.
-rdynamicis what exports the cells, so it and cells are one flag:Build.opts.dev,flan build --dev. This is the first timeoptsmeans something semantic rather than an optimisation level.
LLVM cannot fold the indirection away — the cell is an external mutable global — and a --dev build of calc-me keeps 46
indirect calls at -O2. The acceptance table now runs values, machine and sand-headless as dev builds as well;
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
Names that did not exist when the process started
Editing a defvar or a defn is a symbol the host exports. Adding one is not: there is no symbol to bind to and ELF
cannot grow one. Those go through runtime/flan_dev.c, which is two lookups and nothing else:
void **flan_dev_cell(const char *name); /* a new function's cell */
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
new global's declared initial value travels with it, as a constant the runtime copies on the allocation and ignores
on every call after: calloc alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
common case pays nothing for the general one.
The unit is a list of top-level forms, not one function — Emit.redefinition ~fns. C-c C-c passes one name, C-c C-k passes a file's worth, one code path either way. It has to be: v3 of the fixture adds extra and uses it from a
redefined bump, and splitting that into two loads would leave a module referring to storage that does not exist yet.
Four rules, each of which is a silent failure if broken:
- Every lookup resolves before any body is published. Publish first and a caller reaches a function whose slots are
still null. Not race-testable, so it is asserted on the emitted
flan_reload_install. flan_dev_globalrefuses a size change. The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.- Nothing that published anything is ever
dlclosed. A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being pointed into, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer, or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the handle value rather than the mapping:dlopenrefcounts by path, so re-sending the same bad file raised a count nothing could lower, and the one reference that could was dropped on the floor. - The registry never moves. A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable.
The test that separates this from a plausible wrong version is v4, which redefines added — a name v3 introduced at
run time. v3's bump is already installed and is not rebuilt, so it picks v4 up only if its call goes through a cell
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
pass and the transcript would read 246 instead of 432.
Sizes are spelled LLVM's way — ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64) — rather than by a layout
calculator in OCaml that would have to agree with LLVM's on every target.
The agent — dev loop step 3
vendor/agent/ is a package like any other: agent.flan declares three calls, flan_agent.c implements them, link
asks for -lpthread.
(agent/start path) listen on a unix socket; once, at startup
(agent/poll) install whatever has arrived; returns how many
(agent/wait ms) the same, but waits for something first
The split between them is the design. dlopen relocates a module and takes the loader lock — milliseconds, unbounded —
so it happens on the listener thread. flan_reload_install is one store per function and must not land while a
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
loader.
A full ring is refused, at the sender, before the dlopen. Of the three honest answers this is the only one that
reaches the person who asked: dropping loses a reload the sender was told was ok, which is the same lie more quietly,
and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also
stop answering status and abort, leaving the dev loop with no way to reach a program that had stopped listening to
it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away,
since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says
what to do about it — call agent/poll.
The way out of the break loop is _exit, not exit. exit runs the atexit chain and the ELF destructors, which
want the loader lock the listener thread may be holding inside dlopen; a program asked to abort would hang instead of
dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134
stays because that is what a trap exits with.
wait exists for tests. A test that races the frame rate fails on a loaded machine, so test/programs/agent.flan waits
for the reload instead of sleeping past it. It takes two reloads, which is the daemon's actual loop: the first
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
Two details found by running it:
- stdout is line buffered, set in
flan_rt_init. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found. - The reply goes out before the module is queued. The other way round, the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection reset rather than as
ok. okmeans queued, not installed. The sender does not get to know when the swap happened; only the program knows when it is between frames.
sand.flan calls agent/poll at the top of its loop, which is what step 3 was for. Verified: with sand running under
Xvfb, flan reload sand-probe.flan game-draw and one line on the socket, and 455 consecutive frames drew from a body
that did not exist when the process started. Building without --dev is fine — there are no cells, so a module is
refused on the listener thread and the loop never notices.
flan reload <file.flan> <fn>... [-o out.so] [--new name,...] builds one module the way the daemon will. --new is the
names the host was not built with; it is the one thing the command cannot work out for itself, and it is exactly what
the session will track automatically.
The session
lib/session.ml is the program as a live thing: the declarations the running process was built from, plus every change
accepted since.
Transactionality came for free and needed no machinery. Check.program builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the llc that
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
Two things the session knows that no single evaluation could:
- Which names the running process was built with. It comes from the checked program, not from any accumulated AST,
because
Check.programprepends the prelude and no AST contains it. Derive it from declarations andprintlnreads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. - What that process's memory looks like. Three changes are refused with a reason rather than loaded:
| Change | What it would have broken |
|---|---|
| a function's signature | a cell is a bare ptr; every call site compiled before the change still passes the old arguments through it — and this is now a stopgap, see below |
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
| a struct's fields | the values the process is holding have the old layout |
a defconst's value, when the checker consumed it |
it is in the shape of the program — (defconst rows (/ h c)) decides grid's type before anything else resolves — so no store can reach it |
a defenum member |
:space is erased to an i32 literal in the caller, so it is folded there too |
The signature row is the one the plan has moved past. plan.org now says a signature-changing redefinition should
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
existing callers and stored Fn values keep the old version and stay safe, and the session warns at every tracked
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
should not be read as the final answer.
A defvar's initial value is deliberately not in that table. Its storage holds live state the program moved past
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same Tast.global
record as a defconst, opposite answers, told apart by gconst. The enum comparison runs over declarations rather than
the checked program, because Tast.program carries no enums at all — they are erased to i32 in the checker, which is
the same fact that makes them unreloadable.
Note what the checker catches on its own: change helper's parameter type and the caller fails to type check first,
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
uses, which is exactly where the silent version lives. The fixtures carry an unused defvar and a C-called defn for
that reason.
A defconst the checker never consumed is a different matter and can be changed: it is only ever bytes in memory. A
dev build emits every defconst as a mutable global rather than a constant — so LLVM cannot fold a read of it and a
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
is how sand's colors gets tuned live while rows stays refused. Release builds emit constant and get all the
folding back; Tast.global.gfolded is what tells the two apart, because nothing downstream of the checker could.
A form typed into a file that is imported as a package is qualified the way the import qualified it. poll in
vendor/agent/agent.flan becomes agent/poll, and its call to poll-raw becomes agent/poll-raw — through Load's
own qualify_decl, so the rule cannot drift from the one used at import time. Without this the form spliced as a
brand-new unrelated name: the evaluation answered ok, and the running program went on calling the sim/settle it
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
did nothing.
It is derived from the file's path and not sent by the editor, which is where this departs from CIDER's ns key: a
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
either.
The accumulated list is the post-Load one, so an evaluated (import …) is spliced as its expansion. Otherwise
re-evaluating a file that imports something appends a second import, Load expands it again, and the duplicate-name
pass rejects it. C-c C-k on sand.flan's own text is the test.
flan reload <program.flan> <forms.flan> is that path from the command line: a session over the program the process was
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form game-draw
and 910 consecutive frames drew it.
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
has and what its memory looks like describes a binary that is not running. And Session.eval's origin defaults to
<eval>, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
the real buffer path, which is the same key CIDER's eval carries.
The daemon — flan dev
flan dev <program.flan> holds one Session, builds the program, launches it, and listens on .flan-dev.sock beside
the source. What it adds over flan reload is that the session persists — a defvar added by one evaluation is part
of what the next one is checked against — and that it owns the build, which is what makes its layout rules describe
the process that is actually running rather than a guess about it.
The protocol is s-expressions, not bencode. nREPL was the plan and the argument for it evaporated once the client
became ours too: there is no CIDER to be compatible with, eval is string-in/string-out with no slot for which form,
from which file, and Emacs already has read and prin1. So it is one sexp per message — no parsing code on the
editor side, and on this side the parser is the language's own reader, where :op is already a keyword and a payload of
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
newlines. An nREPL front end can sit on the same Session later; it should not have gated the editor.
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
(:op "close")
defs is its own op rather than more fields on describe, because describe is what an editor polls — it is how the
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
It is asked once on connect and again after each accepted install. Four strings an editor reads with read and nothing
else: eldoc, completion and find-definition want the same three facts about a name. loc is empty where there is none
to give, because only Tast.fn carries one — an editor must refuse rather than go looking for the definition itself,
which in a program of several files finds the wrong one. Parameter names are not in the Tast, so a signature is step [i64 f32] i64: types only.
The daemon makes its own source path absolute before building, because every location it reports derives from it. flan dev src/game.flan run from a project root otherwise answered src/game.flan:12:7, which an editor can only resolve by
guessing which directory it was relative to.
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
— is accepted and answered with :note "nothing to install" rather than by shipping an empty module. Building one
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
:file is not decoration: Session.eval's origin defaults to <eval>, so without it every error an editor shows
points into a file that does not exist.
Two things the daemon must not paper over, both of which would look like a successful evaluation:
- The agent socket is chosen by the daemon, not by the program. A program's source has to name some path — sand.flan
says
/tmp/flan-sand.sock— and the daemon overrides it throughFLAN_AGENT_SOCKETbefore spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it. - Delivery is checked.
agent/startreturning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is notokbecomes an error the editor sees.
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving.
The Emacs client
emacs/flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is most of the work: Flan is
s-expressions, so sexp motion, paren matching, beginning-of-defun and indentation are already right. What it adds is
Flan's own bracket syntax ([ and { are brackets, not symbol characters — every binding list and every type is
written with them), the characters a Flan name may contain (-, ?, /, .), and its keywords.
emacs/flan-dev.el is the client. There is no parser in it, which is the point of the protocol choice: prin1 writes a
request and read reads a reply.
C-c C-c |
the top-level form at point, recompiled and installed |
C-c C-k |
the whole buffer, as one module |
C-x C-e |
the expression before point, evaluated in the running program |
C-c C-z / C-c C-q |
connect (finds .flan-dev.sock upward) / disconnect |
C-c C-o |
the running program's own output, in *flan-output* |
C-c C-r |
a prompt on the running program (*flan-repl*) |
C-c C-b |
what a stopped program is offering, and which to take |
C-c C-d |
what the running program currently defines |
C-c C-a |
the code a name compiled to — amd64, or C-u for the LLVM IR |
M-. / M-, |
where a name is written, through an xref backend |
eldoc, completion-at-point and M-. all read one cached defs reply rather than asking per keystroke: eldoc fires on
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a defn just
installed completes at once.
The modeline says whether there is a program on the other end, in three states. lost is a daemon that has gone away,
which is ordinary rather than an error — flan dev ends when its program does — so the next request reconnects on the
socket it was on. Strictly before a send, never after one: a connection that died mid-request may have died after the
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
C-c C-k sends one module rather than a form at a time on purpose: a defvar and the function that uses it have to
arrive in the same load, or the first refers to storage that does not exist yet.
Framing is in bytes and Emacs counts characters. Every length goes through string-bytes and the process is binary,
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
has of them — a bug that would look like a corrupt protocol and appear only for some users. test/test_emacs.ml drives
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
mistake in the framing, in beginning-of-defun over Flan's syntax table, or in the reply reader passes test_dev.ml
and fails here.
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
next time that buffer's evaluation is accepted. Two things had to be right first. The column in a :loc is a byte
offset, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
forward-char with it put the marker as many columns right as the line had non-ASCII characters before it. And the
daemon numbers lines from the start of what it was sent, so C-c C-c on a defn halfway down a buffer answered line 1
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
reader skips, so the reply's line numbers are the buffer's own.
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
success is indistinguishable from silent failure, and beginning-of-defun may well have found a different form from the
one point looked like it was in.
Live disassembly — done. C-c C-a on a name writes the amd64 the running program's copy of it was assembled to
into *flan-disassembly*; C-u C-c C-a writes the LLVM IR that body was built from. (:op "disassemble" :name … :form "asm"|"ir") is the op.
What makes it possible is that the daemon owns the build: it compiled every module it sent, so objdump -d --disassemble=flan.<name> on the right object is the disassembly and the retained .ll is the IR. Build.shared
deletes its own .ll and Build.executable leaves the host's under a name that says nothing about which module it was,
so the daemon now writes its own copy beside each .so and keeps the host's as host.ll — ten reloads in, nothing else
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
bookkeeping.
What it will not claim is that the code shown is installed, and this is the interesting half. The agent's socket
takes a module path and five verbs; none of them reports an address, flan_dev_cell lives in the program's address
space, and C-x C-e renders a pointer as <ptr> on purpose — so nothing the daemon can ask would tell it what a cell
holds. The reply carries :basis saying which of three things is true, and the buffer prints it above the first
instruction:
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is certain;
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
- a module was delivered and the program is stopped — which says nothing either way about whether it installed, since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further installs until it resumes.
From SBCL: offsets from the function's own start rather than addresses into an object, and L0: labels on branch
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
the debug build lands, that is the line to delete.
Transient error overlays — done. The diagnostic ghost text is feedback about the evaluation that just failed, not an
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
pre-command-hook and not post-command-hook, which fires at the end of the failing command and would clear the
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
in all of them. execute-kbd-macro runs no pre-command-hook under --batch, so the test drives run-hooks — the
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
is Emacs' contract and a test claiming to check it would be checking nothing.
The program's stdout is a pipe into the daemon, and whatever it printed since the last reply rides along with the
next one into *flan-output*. Having it arrive with a reply rather than by a separate request is the point: the
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
accept loop's select, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
grow the daemon without limit.
C-x C-e — evaluating an expression
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
into, so the expression is wrapped in a function with nowhere to be called from; the module says run this once by
exporting flan_reload_call, and the agent calls it after the install — on the game thread, at a frame boundary, so an
expression reading the program's state sees a point the program agrees is consistent.
Nothing is marshalled back, because nothing could be. A Flan value carries no header, so no code at run time can say
what it is. The compiler knows the type and renders it there, in the thunk, into flan_dev_result. That is the layout
decision's bill, and it is why the printer set is small rather than universal.
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
dev-only feature must not put a branch in it — so flan_rt.c is untouched and the value is read back over the agent's
socket. The read is safe without a handshake because the counter is a seqlock, and it had to be made into a real
one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived
and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to
be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the
bare pointer was the bug rather than the ordering. flan_dev_result_read copies into the caller's buffer and checks
the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader
that loses the race reports the last complete generation and no bytes, so a daemon polling for a new value keeps
polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's
"has it moved" still means what it meant. Marking the counter odd needs a release fence and not a release store — a
release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which
is the original bug with more ceremony. And the odd mark is set rather than incremented, because a begin with no
end is reachable: the thunk calls begin before it evaluates anything, so an expression that signals is stopped
inside that window, and a restart taken from the break transfers past the thunk and end never runs. Incrementing
would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit
means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has
reached a frame boundary.
This renderer is most of println, which is worth knowing before anyone schedules it. plan.org describes a
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
structs, fixed arrays and options structurally, prints a Ptr as its address rather than following it, and bounds depth
and length. That is a description of what Session already does for C-x C-e — same walk, same refusals, same three
bounds — aimed at flan_dev_emit and the wire instead of at stdout. What println needs on top is a stdout sink, a
builtin that takes its printer from the argument's type, and the any/Error dynamic cases, which have no compile-time
type to walk. Not the walk itself.
The renderer is a compile-time walk over the type, emitting a piece at a time through flan_dev_emit. Piecewise
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
allocator the language does not have.
big 18446744073709551615
col :blue
(.pos b) (V {:x 1.5 :y 0})
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3) [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
Details that are decisions rather than formatting:
u64renders in C, with%llu. The language's owni64->bytesis signed, so it used to refuse rather than come back as-1— but refusing a whole struct because one field is au64is much worse, so the runtime got an entry point instead.- Strings are quoted and escaped, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is.
- An enum renders as
:name, recovered from the checker's table as a chain of comparisons, because members are erased toi32before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see. - A pointer is never followed —
<ptr>. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf. - Three separate bounds, easy to conflate.
depth(4) andspan(8) bound the walk, so[100 [100 u32]]does not become ten thousand render sites in one module. The output is bounded once in the runtime —emittruncates at 4K andendappends...— because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget. - A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element.
What still refuses by name: Map, Fn, a type variable.
A caveat inherited from the language, not introduced here: 3.0 renders as 3, indistinguishable from the integer.
flan run calc-me.flan "1.5 * 2.0" has always said 3.
An evaluation is not a declaration: the thunk is built against the program and never spliced into it, so describe
does not fill up with eval/N for every expression ever typed.
The module is unloaded afterwards, which is the one case where that is safe. The thunk is called directly by
flan_reload_call rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
into its text and the value it produced has been copied out. It declares that with @flan_reload_transient and the
agent dlcloses it. Measured: sixteen expression evaluations retain zero mappings, where each redefinition
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
and can never claim this.
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it.
The test that matters is the same expression twice: the fixture increments ticks every frame, so two evaluations must
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
The REPL buffer
flan-repl.el is a comint-mode buffer whose every line goes through the same eval-expr request C-x C-e uses. No
new protocol and no compiler support. Deriving from comint rather than hand-rolling a prompt is the same call as
deriving flan-mode from lisp-mode: history, the input ring and kill/yank already exist. There is no subprocess
behind it — the "process" is a stub comint needs in order to have a prompt at all.
Three things about it that are decisions:
- It is program-scoped. A name typed at the prompt resolves against the running program's top-level namespace, so in
sand you write
sim/settleand notsettle. A buffer visiting a package's own file gets the alias applied for it because the file says which package it belongs to; a prompt has no file and nothing to derive one from. - RET on a half-typed form opens a line instead of sending it. Balance is checked with the Flan syntax table, so a paren inside a string does not count.
- A value and the program's output are different things and arrive by different routes. The value is the result of
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
and goes to
*flan-output*. Showing them in one place would be convenient and wrong, so there is a test for the separation.
That test is what caught a real bug: the renderer's Unit case emitted () without evaluating the expression, so
(println "x") — the most ordinary thing anyone types at a prompt — answered () while nothing happened. A Unit
expression is almost always a call made for its effect, and is now evaluated and then reported.
Conditions — step 1: handler-bind and signal
spec-conditions.md §1 and §2, and nothing else yet. They are worth having on their own because neither alters
control flow: signal returns Unit whatever it finds, a handler that returns normally leaves the signalling
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
yet, and no signature changed.
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
(load-all))
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
function's own stack, and signal with an empty stack is a null check — which is what §2 asks for. Popping is by frame
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
step.
Three decisions worth keeping:
- A condition's type is a hash of its name, not an index. An index would shift the moment a struct were added, and every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
- The condition crosses as a pointer, because a handler runs while the signalling frame is still alive and there is
nothing to copy. What the clause binds is the condition itself, though — the pointer is a hidden parameter and the
name is a slot loaded from it, so a handler passing
cto something expecting the struct is not handed an address instead. - A clause is lifted into a function of its own. A handler runs from wherever the signal was, so it cannot be a branch in the function that wrote it.
- A pushed handler frame holds the clause's body address, not a cell. This is a deliberate divergence from
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
particular body. A handler frame is not a
Fnvalue — nothing in the language can name it — and it is live only for the duration of thehandler-bindbody, so a reload landing while it is on the stack finds the clause it pushed still valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the stack does not observe a redefinition of its own clause; the next entry to thehandler-bindpushes the new one. WhenFnvalues arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
Which gives the two refusals, both by the house rule rather than by accident:
- A handler cannot see the establishing function's locals. That is a closure with an explicit environment, so a reference to one is refused for that reason rather than reported as an unknown name. Globals and the condition are in scope, which is what the accumulation case needs.
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
outlive the function that established it, so this is spec-memory.md's case 2 — a non-escaping fn capturing by
value into a stack environment — and not the escaping closure that plan.org's open decision #5 defers until a concrete
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
biggest usability limit in conditions is not behind the thing that was just deferred.
returninside ahandler-bindbody is refused. The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape asdeferinside a block.
The break loop — conditions step 3
Where "a crash kills the program" stops being true. An unhandled error runs a hook instead of rt_die(), on the
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
flan: unhandled Missing — stopped, not dead.
0. restart: retry
1. restart: use-placeholder
Four decisions, each of which is the reason something is where it is:
- It is a hook, not a call. The loop lives in
vendor/agent/, which is an optional package;flan_rt.cis the release runtime and must not depend on something a program may never import. A program with no agent leaves the hook null and dies exactly as it did before. - The hook resumes by writing a restart into the transfer channel — the same channel an
invoke-restartwrites, reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same act, lowered once. Nothing about §6 needed changing to support it. - The break loop is the poll loop, run from the error instead of from the frame boundary. That is not a
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
that did not drain that queue would hang
C-x C-eexactly when it is most wanted. - Installing while stopped is allowed, which contradicts a rule stated above and should. "A redefined function must
not be swapped while it is on the stack" is about mid-frame consistency — half a frame of old code and half of new —
and there is no frame in progress here. The old body on the stack keeps running; a
retryrestart calls through the cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of stopping.
A restart frame carries its name now, beside the hash. Matching never needs it — that is what the hash is for — but
a break loop has to show someone their choices and nothing at run time can turn a hash back into a name. It is also
compute-restarts' data, whenever that arrives.
A choice is validated on the listener thread, against a stack the stopped game thread is holding still, and refused
there. Accepting it and discovering on the game thread that no frame offers it would answer ok for something that
cannot happen.
The socket verbs are restarts, restart-at <n> [name], restart <name> and abort, and all four are refused with
the reason when the program is not stopped — there is no restart stack to walk from a running one. restarts answers
<index> <+|-> <name> per line: the index is the identity, and the flag says whether a transfer to that frame has
anywhere to land. test/programs/break.flan errors three times — two restarts taken by name, then a shadowed pair
where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved
by name could pass.
The break loop in the editor — conditions step 3, the other half
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it follows from one fact: a program stops at a moment nobody asked about. Every other op in the protocol answers a question an editor chose to ask.
So the state is learned twice, deliberately:
- It rides on every reply, beside the program's output and for the same reason.
:stopped t :condition "Missing", or:stopped nil. The likeliest instant for a program to stop is the one just after an evaluation — a body that now errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding out after the echo area had said the evaluation was fine. - And a timer asks anyway, once a second, with
describe— the cheap op, which is also how the output pipe is drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies that never come says nothing. The timer never reconnects:flan-dev--requestreopens a socket a restarted daemon left behind, which is right for something a person did and wrong for a background poll, because it would quietly erase theloststate that exists to be seen. It also skips while another request is in flight —accept-process-outputruns timers, so a poll firing inside a read would eat the reply that read was waiting for.
Three ops, and the annotation owns :stopped, not the ops — one place in the daemon decides whether the program is
stopped, so the poll and the prompt cannot disagree.
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
:stopped t :condition "Missing")
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
(:op "abort") → (:status "ok" :note "the program is exiting; …")
:restarts is positional — innermost first, duplicates kept — because the position is what restart-at takes.
:unreachable names the positions that are on the list and cannot be chosen. restart-at's :name is optional and is
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
prompt cannot take a different restart than the one it showed.
break carries only the restart list, because it costs a second round trip to the program and is wanted only by
someone about to choose from it.
ok from restart means accepted, not resumed. The choice is validated on the program's listener thread against
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
lets the next poll settle it.
The modeline is a fourth state, flan:stopped(Missing), before live: a stopped program looks exactly like a running
one from anywhere else in Emacs. The prompt is a completing-read over the names with require-match, which is exactly
right for a closed set the program computed, and abort is the last entry on that same list rather than a second key —
it is the thing you pick when none of the restarts is the answer.
flan_agent_poll had to become re-entrant, and that is the one thing here that was a bug rather than an addition. A
C-x C-e thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
cached head and tail and wrote tail back at the end, so the outer call rewound the index over everything the
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
breaks rather than a stumble. It now claims each job by advancing tail before running it, and re-reads both indices
each time round. Still single-consumer: only the game thread writes tail, nesting included. test_dev.ml evaluates an
expression that errors and resumes it, which fails against the old shape.
The agent grew one verb, status, answered in both states — running or stopped <condition>. Everything else the
break loop offers is refused while running, and rightly; but the question an editor asks without already knowing had
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
test/programs/dev-break.flan stops on its first frame, so the daemon meets a program that is already stopped — the
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
round, by installing a step that errors into a loop that calls it, fixes it while stopped, and then resumes: C-x C-e
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
layout — a type's fields, with no program involved
(:op "layout" :type "sim/Cell") → (:status "ok" :type "sim/Cell"
:fields (("heat" "f32") ("next" "(Option sim/Cell)")))
→ (:status "error" :message "Missing is not a qualified name; …"
:candidates ("a/Missing" "b/Missing"))
The daemon can answer this with nothing running. A layout is a fact about the build, and the daemon owns the build —
Tast.structs is sitting in the session it compiled the process from. That is why the conditions buffer can name and
type a condition's fields while every one of their values stays refused: the shape is knowable and the contents are
not, and drawing them apart says more than drawing neither.
The type is a name, and the name is the qualified one. This was the open question — a class name is not an identity,
and two packages each declaring Missing would leave the daemon unable to pick. It turned out to need no new
machinery: Load.qualify_decl rewrites Defstruct (n, …) to Defstruct (alias/n, …) at import, so by the time
anything reaches Tast.structs the names are one flat namespace in which a collision cannot exist. The name is the
type id, with no table to keep in step across a reload, and the existing spelling of a type — Types.to_string —
already prints it.
And the break loop was already speaking it. Emit.struct_name_of takes Types.Named n — the qualified name — and
passes it to flan_error; the agent holds it in condition_name; break answers it as :condition. So the string
the conditions buffer already had in hand resolves as :type by construction, and test_dev.ml round-trips exactly
that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat
worth writing down — condition_name is a char[128], so the round trip holds for names up to 127 bytes and a longer
one is truncated and will not resolve.
A bare name is refused, not resolved, even when only one struct's last segment matches it. Resolving a unique
suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot
rely on. The refusal carries :candidates, so a person is one copy-paste from the answer and a client has its
completion list — the same shape as package_of refusing a directory imported under two aliases rather than picking
one. An enum is refused by kind (X is an enum, not a struct): its members are erased to i32 before Tast.program
exists, which is the same fact that makes a defenum unreloadable. A union is refused the same way and for its own
reason — it is declared, and union values are milestone 6. Both are Types.Named at a use site, so falling through
to "no struct is named X" would say a type does not exist about one that plainly does.
render.ml is not reused, and that is not a second walk. It walks a value and emits the code that prints it;
this describes a type and emits text. What is shared is the spelling: field types go through Types.to_string, which
is what defs spells a signature with, so (Option T), [T], [n T] and (Ptr T) read the same in a layout, in a
signature and in the source. A field that is itself a struct shows its qualified name — which is a :type this op
accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude
structs are answered like any other, because Render resolves against the same list and an editor that could see a
Split printed but not ask about it would be the two disagreeing.
On the Emacs side flan-cnr-layout makes the request and flan-cnr-show passes the result into
flan-cnr-state-from-reply, which stays a function from data to data so the fixture-driven tests keep working without
a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and
turning C-c C-b into an error would take away the restarts — the decision the buffer exists for — over a missing
annotation.
The shadow stack, and backtrace
plan.org's "Dev vs release builds" table has had Frames: shadow stack in the dev column since the project began and
nothing had ever built it. This is it. It is what (:op "backtrace") is made of, and it is dev-only, so a shipped game
pays nothing — the same bargain the indirection cells already make.
Chosen over the DWARF route deliberately, and the author's reason is the one that decides it: the more a break loop can show, the less often a real debugger is needed. DWARF buys frames in lldb; this buys them in the break loop, where someone already is when they want them.
A frame is four words on the calling function's own stack — the frame it displaced, a pointer to a static description
of the function, and two words reserved for locals. The description is per function and not per call, because nothing
about a function changes between two calls to it: the qualified name, file:line:col as Loc spells it, and how many
slots the frame has. So the name in a backtrace comes off the frame itself and needs no debug information, no symbol
table and no agreement with the optimiser about what a stack frame looks like. It is the same mechanism on wasm32,
which is the other reason it is not .eh_frame: plan.org lowers every non-local exit explicitly rather than through
platform unwinding, so there is no unwinder here to borrow.
The compiler emits the push and the pop inline rather than calling into the runtime. This is on every call in a dev build, and a call made to record a call would be most of what it costs.
The pop is at every ret, and the transfer path is the one that matters. There are five: an explicit return with
a value and without, the none arm of (some x), the tail of the body, and the landing block a transfer leaves
through. emit.ml funnels all five through one ret, because the way to get this wrong is to write the pop on the
normal path and not on the other one — and then every handled error leaves a dead frame behind, and the backtrace
after the fifth one is five frames of fiction. test_dev.ml takes five breaks and resumes all of them by condition
transfer before asking for a backtrace, and the answer has to be two frames. Same lesson with-allocator learned about
restoring at the pad.
A plain global, not a thread-local. It matches what the handler stack and the restart stack in flan_rt.c already
assume: one thread runs Flan, and the listener thread runs C and the loader and never enters a Flan body. If the
language grows threads this becomes thread-local and the compiler's two stores become TLS-relative, which is the whole
of the change.
The chain is snapshotted on the stopped thread, into the same snapshot the restarts are copied into, at the same
moment and for exactly the same reason: the break loop polls, a poll runs Flan, and a chain read by the listener
thread is a chain that can be popped underneath the reader. Names are copied as bytes rather than kept as pointers,
because a transient C-x C-e module does get dlclosed and its rodata with it. Frame addresses are kept beside the
text, because reading a frame's locals means going back to that frame and not to whatever is at index 2 by then.
(:op "backtrace") → (:status "ok" :frames (("fetch" "/game.flan:15:7" "program" 0)
("main" "/game.flan:27:7" "program" 0))
:more 0 :stopped t :condition "Missing")
Innermost first. :more is how many frames deep recursion left off the end — the innermost ones are what the question
is about. origin is "program" or "eval": a break inside a C-x C-e thunk has that thunk's frames on top of
the program's, and "where is my program" answered with eval/7 is true and not the question. The boundary is recorded
at the call, in flan_agent_poll, exactly where restart_floor is and for the same reason — except that its
out-of-a-thunk value is -1 rather than 0, because zero restarts on the stack is a real answer and zero frames
belonging to the program is not.
Refused while the program is running, like every other break verb. The chain is the game thread's and it is pushed and popped on every call; a walk of it from the daemon would have the shape of a backtrace and the contents of a race.
What it costs, measured rather than assumed. Two benchmarks, one compiler built per variant, every binary kept and then run alternately. The figure is the minimum of nine runs, because this machine is shared and a mean measures whatever else was running; the whole table reproduces to three digits on a second pass.
| Dev build, -O2 | no frames | frames | frames + slots |
|---|---|---|---|
| 2000 sweeps of a 100×100 grid — sand's inner loop, in miniature | 30.0 ms | 40.0 ms (+33%) | 48.3 ms (+61%) |
| fib(30) plus 20M calls in a loop — nothing but calls | 28.7 ms | 31.1 ms (+8%) | 35.1 ms (+22%) |
Per call that is about 0.1 ns for the frame and 0.3 ns for the frame and the slot table together; per sweep of
the grid, 5 µs and 10 µs, which is 0.03% and 0.06% of a 16.6 ms frame at 60fps. The dev loop's premise is that
redefinition does not stutter a running game, and a sixteenth of a percent of a frame does not. A dev build is already
deliberately slower than a release one — every call goes through a cell, every index is checked, no defconst folds —
and this joins that list rather than starting a new one.
The chain's shape was chosen by measurement and not before it. An array with a stack pointer — no alloca, no address escaping — was built and timed as the obvious alternative and is worse on both benchmarks: the frame record on the calling function's own stack is already hot, and an indexed store into a megabyte of BSS is not. It also has a fixed depth, which a chain of stack records does not.
Locals of a stopped frame
The half the shadow stack was built for. (:op "locals" :frame N) answers what a stopped frame's named locals hold.
(:op "locals" :frame 0) → (:status "ok" :frame "look"
:locals (("n" "i64" "3") ("label" "string" "\"hello\"")
("p" "Point" "(Point {:x 1.5 :y 2.5})")
("xs" "[3 i32]" "[ 10 20 30]") ("flag" "bool" "true"))
:refused (("after" "not bound yet at the point the program stopped")))
Nothing is copied out of the program, and nothing could be. A Flan value carries no header, so bytes read from
another process would be bytes with no meaning. What the daemon has instead is the type — Tast.fn.slots, from the
build it owns — and the name beside it in snames, which was already there for the DWARF work. So it compiles a thunk
that renders those types at those addresses, in the program, on the stopped thread, and reads the text back through
the same seqlocked result buffer C-x C-e uses. The only fact that comes from the running program is where the frame
is. That is render.ml's existing walk with its root changed: Render.render over (Deref (Ptr T) (flan/dev-slot f i)) instead of over an expression — the pointer-rooted render thunk NEXT.md said locals needed, and it turned out
to need one new arm in the whole backend (a pointer-to-pointer cast, which under opaque pointers emits nothing).
A slot's entry is its address, and null until it is bound. That is the whole of the liveness answer: there is no
analysis, no PC-to-scope map and no bitmap. The store that binds a slot stores the address, so a slot the program has
not reached yet reads as null and is refused by name. Without it, (let [after 99] …) sitting past an error would
render whatever the stack held, and a slice or a struct of garbage does not misprint — it faults, on the game thread of
a program that is already stopped, which is the worst moment this project has to offer. The daemon asks the program
which slots are bound before it builds the thunk, so the set it emits code for is the set the thunk will resolve.
Only named slots are recorded, and this is where most of the cost went. A slot whose address is stored anywhere
escapes, and an escaped alloca is one mem2reg cannot promote — so recording a slot is paying for it in every call to
that function, for ever. The slots that would hurt most are exactly the ones with nothing to show: dotimes's hidden
bound, the temporaries (min) and (max) evaluate their operands into, the render walk's own scratch. They keep their
promotion and are refused by name (s4, "a slot the compiler made up") rather than shown under an invented one.
Recording every slot instead was built and timed and came out inside the noise on both benchmarks, so the rule stands
on what it shows rather than on what it saves.
Shadowing is right here, and that is not an accident of this design — it is the thing the DWARF route still owes.
check.ml's fresh_slot only ever allocates, so (let [v 22] …) inside (let [v 11] …) is two slots, both named
v, and both appear with their own values. lldb answers p v with 11 in that program and will until a
!DILexicalBlock per Let exists.
Four refusals, each by name and with its reason. Three per slot — invented, not yet bound, and no printer for the
type (a map, a function value, a type variable; the arm exists, no program the checker accepts has reached it yet, so
it is written and untested) — and two whole frames: one belonging to a C-x C-e thunk, whose Tast the session does
not keep, and one whose body is not the body this session holds.
The superseded frame, and why a count could not find it. Installing while stopped is deliberately allowed — it is the fix-it-and-retry loop — so the frame on the stack and the body the session holds can be two bodies of one function. A slot count catches a body that gained or lost a binding and nothing else; the case that matters is a rename, which changes neither the count nor the types, and which would otherwise show every new name against the old body's storage with nothing saying so. That is the confident wrong answer, in the one place someone is working out what went wrong.
So each %fninfo carries a slot fingerprint: Emit.slot_fingerprint over every slot's name together with the
spelling of its type, since either can change on its own. The frame carries the value for the body it was compiled from,
flan_dev_frame_slotsig reads it, the agent snapshots it with the rest of the frame and puts it on the backtrace line,
and Dev.locals recomputes it from the body it holds and compares. Different means refused by name — this frame's body
was redefined since it was entered, so its names no longer describe its values — rather than answered. The count check
stays in front of it because its message is the more specific one.
Computed in emit.ml and read from there by dev.ml, so there is one definition of it and the two ends cannot drift.
It stays off the wire: (:op "backtrace") still answers four fields per frame, because a hash is not something an
editor can act on and the refusal says the fact in words instead. The bound is worth stating: it is a 30-bit hash, so a
collision is possible in principle, and it would reproduce exactly the silent wrong answer this catches — but only
between two differing bodies of a function whose qualified name has already matched, since find_fn is what gates the
comparison at all.
The first build of this shipped the fingerprint into %fninfo and stopped there — no accessor, no wire field, no
comparison — and was left in the tree with its own test red. Four of five hand-offs missing looks exactly like one
hand-off dropping a number, which is what the note left behind said it was.
A (Vec T) shows as <vec> and a (Ptr T) as <ptr>, because that is what render.ml already does for them
everywhere else: following a pointer a REPL was handed is not a safe thing to do on someone's behalf, and walking a
Vec structurally is a walk over storage the frame does not own — (print (as-slice v)) is how that is asked for,
and it says at the call site that it borrowed.
Each slot is rendered from its address rather than copied into the thunk first. A copy would be one alloca the
size of the slot — 40KB for sand's grid — and the walk only ever shows eight elements of it. The cost is one call to
flan/dev-slot per leaf the walk reaches rather than one per slot, which the depth and span caps already bound.
Conditions — step 2: restart-case and invoke-restart
spec-conditions.md §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
restart-case further out.
(defn fetch [n i32] i32
(restart-case (middle n) ; its value if nothing transfers
(use-placeholder [] -1)
(retry [] 7)))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
(fetch 2)) ; -1
The channel is an out-parameter, as §6 now says: one ptr appended to every Flan signature, written by an
invoke-restart and checked after every call. The return type stays what the source says, so the disassembly is the
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
slot and each frame only has to check and return early, which reuses the existing return path and with it §5's defers.
Emit.signature was already the one place a signature is spelled, which is what made this a three-line change rather
than a hunt.
Every function is transfer-transparent, release included — and that is the ABI, not a stopgap. A cell holds a bare pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and plan.org both now say the later optimisation may stop a function checking the channel, or pass the pointer straight through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform also means redefinition acquires no new refusal class.
The transfer target is the restart frame's own address, not a clause id. This is a correction to what the previous
note settled. A static id has to be unique against every module a running program may later load, and a hash is only
probably unique — two restart-cases colliding means the inner one silently catches a transfer aimed at the outer. The
frame is an alloca in the function that offers it, so its address is exact, and it also says which clause, which is
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
Re-entering a restart-case then works with nothing extra: each activation allocates its own frames, and §4's
"innermost offering the name" is just the order of the walk.
Cleanup happens in landing blocks, one per region. A guard branches to the innermost open one, which pops whatever frames it established and either catches the transfer or forwards it outward:
- a
restart-case's pops its restart frames, compares the target against its own, and either runs that clause or puts the target back and goes on out; - a
handler-bind's pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, and without it the handler stack would be left pointing into a frame that has gone; - the function's own runs its defers (§5) and returns early.
errdeferdoes not run and never could:try/Resultis still refused by name.
A single function-wide unwind block would have been wrong for the first two: a call inside a restart-case body would
jump straight past the very form that was supposed to catch it.
The channel is cleared before any cleanup runs and put back after. A defer makes ordinary calls and each one is guarded; with the channel still set the first of them would branch straight back into the landing block it came from. Same reason the clause body starts with it null.
flan_signal takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
cross a foreign frame" would make restart-case useless. It is also the only such frame: extern is Flan-to-C only and
there are no function values yet, so nothing can call back into Flan across one.
Scope, each piece refused by name with its reason and a test on the reason:
- restarts took no parameters. That covered §1's own
load-textureexample and skipped argument marshalling and §3's runtime check. Both are in now; see "Conditions — step 3" below. returninside arestart-casebody, exactly as insidehandler-bind: a bareretskips the pops.- one
restart-caseoffering a name twice — §4 finds the first frame offering it, and two in one frame makes that a choice nothing in the source shows. invoke-restartinside adefer. A defer is the cleanup a transfer runs on its way out, so a transfer starting there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a defer that reaches one through a call is trapped at run time byflan_transfer_fail, because nothing static could see it.- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past everything. There is nowhere to resume, so there is nothing else to do.
(error c), §2. The same walk as signal, and the difference is entirely what happens when the walk ends: signal
returns Unit and the signalling function carries on, error has type Never and stops. So only a transfer gets past
it, which is why emit puts a guard after the call and then unreachable — and why flan_error cannot be marked
noreturn, since it does return, on exactly one path. Being Never is also what lets it stand as a restart-case
body's fall-through, which is the shape §1's load-texture example needs. test/programs/error.flan is the unhandled
case: it cannot be an outputs row, because it does not exit 0.
flan_transfer_fail covers the ordinary return path as well as the unwind one: a defer that reaches an invoke-restart
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
A lifted handler clause is named after the function it came out of — handler/step/0/Missing — and is emitted by a
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
nor attributable, and a redefinition of a function containing a handler-bind failed in llc with an undefined value.
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
test/test_dev.ml drives that path: a third evaluation redefines step to a restart-case whose frame is an alloca
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
crosses probe, which the host was compiled with. Those three do not meet anywhere else.
Two things found by writing it:
{ ctx with in_handler = true }was a latent bug.ctx.slotsandctx.slot_tysare mutable, so a copy allocates the body's slots into a record the function never sees again and the indices collide. It was harmless only because nohandler-bindbody in the tests had aletin it. The flags are set onctxand restored now.test/reload_host.chad to learn the parameter. It callsflan.outerthrough an__asm__label, which does not fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere else entirely.
test/programs/restarts.flan runs in the acceptance table at -O2, at -O0 and as a dev build. -O0 is not redundant
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
of those calls goes through a cell.
Conditions — step 3: restarts take parameters
§3's other half, and the half the comparative studies all ask for: use-value and store-value are the two restarts
whose answer is not in the program.
(restart-case (middle n)
(use-value [v i32] (* v 2))
(retry [] 7))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7)) ; 42
The parameters live in a buffer the restart-case owns. The obvious place is the invoker's frame — it is where the values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so the invoking side is gone by then. The invoker stores into the target's buffer while both are still alive, which is the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any other.
What a restart takes is checked at run time, and it has to be. §4 finds a restart by name on a dynamic stack: the
invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker
to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and
invoke-restart compares both against its own before it stores anything. Every frame carries them, parameterless ones
included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not
redundant with the hash: it is what keeps a hash collision between two different signatures harmless, since a
collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well,
because the message has to say what was wanted and what was given and neither end knows both:
restarts.flan:79:36: restart use-value takes (i32), given ()
restarts.flan:87:36: restart retry takes (), given (i32)
The arguments are evaluated into slots before the invoke node, not hung off it. Two reasons and both are real. An
argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside
an argument has to be on the walk Reach and Load already do — InvokeRestart was a leaf to both, and a leaf that
grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker.
restarts.flan has exactly that function, half, to keep the claim tested.
The break loop can take a restart it cannot fill in, so it is refused. A transfer has two sources: an
invoke-restart, which writes the arguments first, and the break loop, which aims the channel at a frame by position
and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them
apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked
unfilled, invoke-restart marks it filled, and a clause with parameters checks the mark before reading. Choosing
use-value from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the
top of NEXT.md: the answer is a Flan expression, and there is already something that compiles one against the live
program.
Lookup stayed by name, and the signature is checked against what it found. flan_find_restart matches the name
hash and nothing else, so an inner (use-value [s string] ...) shadows an outer (use-value [v i32] ...) and an i32
is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first
frame offering the name") and it is the thing a reader will assume works the other way, so restarts.flan has a case
for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments,
which is overload resolution on a dynamic stack.
Both of these guards go through fail_block unconditionally, unlike the bounds checks: --no-bounds-checks does not
remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or
written to a different layout, is memory corruption.
runtime/flan_rt.c gained two message functions and nothing else. The restart frame's first four fields are the ones C
declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.
§3's other open point, a report string per clause, is still open and was not settled first as §3 asks. The field and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing that listing anyway.
What is left
- Editor comforts: completion, eldoc, jump-to-definition, error overlays.
Session identity is the daemon that owns the build. A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default.
Allocators, (Vec T) and StorageExhausted
spec-memory.md's Allocators section is frozen and settles what to build. This is why the built thing is the shape it
is, and — separately and loudly — the three places it amends that section plus the one thing it adds to it.
Allocator is a builtin opaque type, so none of milestone 5 was needed
The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and
check.ml refuses function values four ways — a written (Fn ...) annotation, a written fn literal, a defn's name
in value position, and calling anything other than a named function. Read straight off those lines, containers need
function values and the work doubles.
None of the four is anywhere near this. All four are about surface syntax, and a value the compiler builds that no surface form names trips none of them:
Allocatoris aTypes.tcase with no user-writable constructor, the waystringis a builtin ptr+len. No Flan type names its procedure.- The procedure is a C symbol the emitter names.
vec-new,push,free,free-alland the rest are ordinary named calls, whichcheck_callalready routes throughnamed_call.
The precedent was already in the repo twice: a handler-bind clause is lowered to a function called back through
h->fn(condition, xfer) and built as its own Tast.fn, never an Ast.Fn; and a dev build's call loads a pointer
out of a cell and calls through it, which is the indirect call the surface language refuses.
What does need milestone 5 is a user-written allocator: "here is my proc, make an Allocator from it" wants a
defn's name in value position. make-allocator, allocator-from and allocator are refused by name with that
reason, rather than coming back as unknown functions.
An Allocator value is a pointer to the runtime's struct, never a copy of one. That is forced, not chosen. The
capability set has to be readable from wherever a container landed, and free-all bumps an epoch every container made
from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never
fires.
The surface
| Name | What |
|---|---|
(heap-allocator) |
the general-purpose tier: malloc, aligned, with free |
(arena-new bytes) |
a bump arena over one fixed backing buffer |
(arena-destroy a) |
hands the pages back — see the amendment below |
(free-all a) |
releases everything the allocator holds, retain-capacity |
(can-free? a) / (can-free-all? a) |
the capability set, read at run time |
(alloc-epoch a) / (alloc-id a) |
the counter free-all bumps; the allocator's identity |
(alloc-budget a) / (set-alloc-budget a n) |
a ceiling on live bytes — an addition, see below |
(alloc-live-blocks a) |
"did you forget to free", answered at the tier that can answer it |
context/allocator / context/temp |
the current implicit allocator, and the per-frame arena |
(with-allocator a body...) |
rebinds for a dynamic extent and releases nothing |
(vec-new T) / (vec-new T a) / (vec-new) |
a Vec, against the context or a named allocator |
(push v x) / (reserve v n) |
Unit, both |
(at v i) / (len v) |
the array names, extended — not a parallel pair |
(as-slice v) / (as-slice v lo hi) |
a non-owning [T] view |
(clone v) / (clone v a) |
the only copy; assignment moves |
(free v) |
consumes its argument |
Three amendments to a frozen spec, and one addition
1. free-all is retain-capacity, and arena-destroy is the operation that hands pages back. The spec's table has
free-all and nothing else. Zig's ArenaAllocator.reset takes a ResetMode of free_all / retain_capacity /
retain_with_limit; Odin's arena_free_all is retain-capacity in effect, because its arena is one fixed backing
buffer and the call only sets offset = 0. For a frame arena reset every frame, retain-capacity is the normal case and
handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown
the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being
the same does not make a container built before the reset valid, which is the whole point of the trap.
2. context/allocator and context/temp are dynamic variables with save and restore, not extra parameters. The
spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the
FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other
lane working in emit.ml. The dynamic variable is the implementation; the literal reading is deferred and is a
performance question (a parameter avoids a global load), not a semantic one.
3. The Vec header is six words in every build, not four in release. The spec fixes the release layout at
ptr + len + cap + allocator with the generation word and the epoch dev-only. A layout that changes with a build flag
is a layout that can disagree silently across the reload boundary: a redefinition module is built by llc and ld
against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and
so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag
before it can land.
The addition: a budget. flan_allocator grew a ceiling on live bytes, 0 for none. The spec's retry restart is
answerable only by a handler that can make the same request succeed, and for a fixed backing store the handler that
works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which
is exactly what the epoch check catches. The spec names "grows the arena and then invokes retry" as the handler that
works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on
purpose.
with-allocator is its own IR node because of the transfer path
Save, run, restore — and restore again at the pad. That second restore is the whole reason it is a node rather than a
let and two calls. A body that errors, or one a handler transfers out of, leaves through current_pad, and a context
allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is
precisely where something is about to allocate in order to render a condition. test/programs/allocators.flan asserts
that path by taking a restart out of a with-allocator body.
It releases nothing, per the spec: not at the end of a let, not at the end of a function, not at the end of the body.
The program proves it by reading the epoch either side.
(Vec T): one type-erased runtime, and the backend learned almost nothing
The element type appears nowhere below the call site. size_of and align_of are produced where the concrete type is
known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement
(base/runtime/dynamic_array_internal.odin). The backend grew four prims in total:
Rt of string— a call into the runtime's C named by symbol, with argument and result LLVM types read off the expression nodes. The container runtime is type-erased and therefore is a list of C entry points, so one arm covers all of them. AVecargument crosses as its address, which is also what lets an operation mutate the caller's Vec.SizeOf/AlignOfof a type, filled in from the same layout calculator DWARF uses — the one the acceptance test already checks against LLVM's owngetelementptranswers.AddrOfof any expression, place or not, because the element apushcopies may be computed. The backend already spilled a non-place to a temporary for exactly this.
at and len were already the names for a fixed array and a slice, so a Vec extends them rather than adding a
parallel pair — the asymmetry nth was removed for. The value form (at v i) and the place form (set (at v i) x) go
through one helper, so they cannot drift apart the way nth did.
A Vec's length and index are i32, like every other length here. Widening indices is one change across every
container and not a Vec question.
let has no type annotation — parse.ml settles that a triple binding is ambiguous and that types are inferred — so a
local Vec has nowhere to say what it holds and the element type is written at the call: (vec-new i32). This is not
the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an
ordinary type rather than binding a type variable. Where the context does say — a defvar's type, a return type, an
argument — it may be left out.
A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is
Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a
different reason (below), but the adopt rule is what makes Zero of a Vec a usable value rather than a null deref.
Move-only is a dead set, and it is flow-sensitive at a join
Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing
one to a function, binding it, returning it and freeing it are all moves and all reach one place, and the handful of
operations that only look at a container (at, len, as-slice, push, reserve, clone) say so. Only a
syntactically simple target counts as a borrow — in (len (f v)) the call still moves v.
At an if and at a match, every arm is checked from the state before the form and the union of what they moved
survives the join. A flat set is wrong in both directions: it refuses (if c (free v) (free v)), which is legal, and it
accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union
says.
The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move, not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a move of a binding declared outside the loop is refused, with that as the reason.
What ownership is not transitive through yet, and why each is refused
The spec says ownership is structural — a struct containing a Vec is itself move-only, free recurses into owning
fields, and a field cannot be freed on its own. That machinery is the recursive teardown drop brings. Until it lands,
three shapes are refused where they are declared, each naming drop:
- a struct field of
Vectype, because the struct copies its header on assignment and nothing records a move; - a global of
Vectype, because the dead set is per function — two functions each freeing it is a double free nothing could see, and a global read does not go through the move path at all, so even the one-function case would be accepted. Half a rule is worse than none. A globalAllocatoris a different thing and stays legal: an allocator is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot see the locals of the function that established it; - a
Vecof aVec, because the type-erased runtime copies and releases elements bytewise:clonewould duplicate inner headers instead of copying what they own andfreewould drop their buffers.
And a Vec does not cross to C: handing a header that owns storage to C hands out an owner. (as-slice v) as
(Ptr T) plus (len v) is the shape that does cross, and the refusal says so.
StorageExhausted went in with Vec, not after it
No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the
operation signals (StorageExhausted {.bytes n .align a .allocator id}) with error — whose type is Never — inside a
restart-case offering retry. One rule over every allocating operation, which is what keeps push and reserve at
Unit, clone at the container, and no signature anywhere growing a Result.
It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every
allocating operation, which is the point of having decided it first. Odin's append returns an ignorable
Allocator_Error, and its type-erased path returns the old length on a failed reserve; an append that appends nothing
and says nothing is the outcome this rule exists to make impossible.
The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:
(let [ok false]
(while (not ok)
(restart-case
(do (set ok ATTEMPT)
(if (not ok) (error (StorageExhausted {...}))))
(retry []))))
A handler that frees something, releases a scratch region or raises the ceiling and then invokes retry lands in the
clause, the clause falls through, and the while re-attempts the same request. Every argument to the attempt is
bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a
push was given. With nothing handling it, error stops the program on the frame that erred.
This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site can. The condition is a value struct with fixed numeric fields and no rendered message, because formatting would allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.
The epoch trap, which is the shipping answer to an open question
spec-memory.md leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which
allocator a construction used, and with-allocator plus context/allocator are exactly the mechanisms that deny that
knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, free-all bumps
that counter, and any operation on a container whose recorded epoch has moved traps naming the site.
test/programs/stale-region.flan is the case, and the point of it is that v is still in scope, still looks fine, and
nothing marked it — which is precisely what a static rule cannot see.
The generation word has no reader. It is bumped on every reallocation, as specified, and the stale-slice trap it exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said plainly here rather than implied by the word's presence in the header.
What this leaves for steps 4 to 7
(Map K V); drop and with it the transitive move-only rule, recursive teardown, and the refusal to construct a
drop-carrying container against an allocator without can-free; (Result T E) and try; generics; the macro expander.
And the accumulation pattern — (fn [c] (push errors c) ...) over an enclosing Vec — which Vec does not buy:
capture does not exist at all, and the spec's captured-Vec-by-pointer rule has never had to exist because every
capturable type today is a value type. It is its own item and should be planned as one.
Assets are baked in, and the reason it is a compiler feature
NEXT.md decision 1. (embed "brush.png") is a [u8], (embed "brush.png" string) is a string, and
(embed-dir "assets") is a [n EmbedFile] sorted by name. Odin's #load and #load_directory are the model
(src/parser.cpp, and check_load_directive / check_load_directory_directive in src/check_builtin.cpp); Odin's
# is not imported, because an s-expression language already has a head position for a name and these resolve as
ordinary named calls exactly the way vec-new and heap-allocator do.
The reason for this shape rather than a build flag is the one that decided it. It is a compiler feature, so it
needs no linker arguments and no per-target packaging, and it works identically on desktop and web. That matters more
here than it does for Odin, because Load hands out lflags only to a directory package and main is not exported,
so a program can never be a package: the single file doing (rl/load-texture "brush.png") is structurally the one file
with no link channel at all. The web lane found that hole and did not invent a flag for it. Embedding has no such
hole, because there is nothing to tell the linker.
It costs nothing at run time. The bytes reach the program as a Tast.Str node typed [u8], which emit.ml turns
into the same private unnamed_addr constant every string literal already becomes, and its escape is byte-exact
across the whole 0–255 range, so a PNG survives the round trip through the .ll. Bound with defconst at top level an
embed-dir is an LLVM constant outright, through emit.ml's const.
A Str node typed [u8], not a Bytes prim over a string. This is the one non-obvious choice. Bytes is
identity — emit.ml lowers Types.String and Types.Slice _ to the same %slice — but wrapping the literal in a prim
makes the node non-constant, and const then refuses an embed-dir in a defconst with a global's value must be a
compile-time constant. Both of emit.ml's string emitters take the bytes and ignore the node's type, so it is the same
constant either way and this one is a constant a global can hold.
Two spellings, not one form that changes type with its context. Odin threads a type_hint everywhere and can
afford #load("p") to mean a string here and a []u8 there. With structural equality, no implicit widening and no
coercion anywhere, the same text meaning two types would be a wart, so string is written down when it is wanted. The
site's expectation is a fallback only and nothing depends on it.
The path is a literal and resolves relative to the file the form is written in. Both are Odin's rules and for
Odin's reasons: the bytes must be in hand before any value exists, which is what makes the result free; and a path
relative to the compiler's working directory would make a package's assets depend on where flan was invoked from,
which cannot be right. A missing file is a compile error naming it, never an empty embed — an asset silently absent is
exactly the quiet wrongness this removes. An empty directory is not that case and embeds cleanly as [0 EmbedFile].
The directory lookup is a linear scan, and that is the chosen answer rather than the fallback one. embed-find is
an ordinary prelude function over a [EmbedFile]. A directory embed is tens of entries whose names sit in cache-warm
.rodata; a compile-time perfect hash would be a build-time map with its own failure modes that nothing has asked for,
and sort-and-bisect is the next step if a program ever embeds thousands of files — it would not change the type. It
takes a slice rather than the array, because an array's length is part of its type and there are no generics, so
the call reads (embed-find (slice assets 0 (len assets)) "brush.png"). Entries are sorted by name because readdir
order is filesystem-dependent and an unsorted embed would make two builds of identical sources emit different .ll.
Non-recursive, files only — Odin again.
The sharp edge, inherited and not widened. The slice points into .rodata, so a store through it segfaults at
-O0 and is deleted as undefined behaviour at -O2 — the same trap the prelude's ASCII-case note measures for
(bytes "Hi"), and the same one NEXT.md tracks as "writing through a string literal". Nothing here makes it worse and
nothing here fixes it; provenance is what would. To get a mutable copy, clone the bytes into a Vec. It is worth
saying loudly because an embedded asset is precisely the thing someone will try to decode in place.
What this does not do. sand.flan still calls (rl/load-texture "brush.png"), which hands raylib a path for
raylib to open. Pointing raylib at embedded bytes needs LoadImageFromMemory and LoadTextureFromImage in place of
LoadTexture — a raylib binding question, not an embedding one — so the flagship program is not yet asset-free on the
web. The mechanism it needs is in.
slurp, barf, and the two ways they fail
NEXT.md decisions 2 and 5. (slurp path) and (slurp path allocator) read a whole file into a (Vec u8);
(barf path bytes) writes one.
slurp waited for Vec because its result has no length until the file is read, and it obeys spec-memory.md's
rule without an exception: no allocating operation returns an error. There is no Result here, no out-parameter and
no error code — slurp's type is (Vec u8) and barf's is Unit.
Two failures, two conditions, and the guards nest rather than merge. Allocation failure is StorageExhausted under
retry, unchanged and reused. File failure is FileError {.path .op .reason} under retry and use-value. They stay
apart because they ask two different answerable questions: the handler that grows an arena is not the handler that
supplies another path, and collapsing them would make one handler guess which it was looking at. file_guard in
check.ml is alloc_guard's shape built from the same nodes — a while, a restart-case and an error — so the
backend learns nothing new.
The restarts are Common Lisp's pair for a file-error. retry for "the file may be there now"; use-value [p string] for "try this other path". use-value is the first restart clause the compiler itself emits with a
parameter — typed restarts landed the same session — and its parameter is the path slot the attempt reads, so the
clause body is empty. emit.ml's bind_params stores the invoker's argument into the slot, the clause falls through,
and the loop re-attempts against the new path. Everything is inside that loop, so a use-value naming a different file
re-measures it and re-allocates for its size; the Vec is freed at the top of each turn, which is why a retry does
not leak, and freeing a Vec that never allocated is a no-op.
The two forms resolve paths by opposite rules, and it is worth saying in one place. An embed path is resolved at
compile time relative to the file the form is written in. A slurp or barf path is resolved at run time by the
host, against the process's working directory — these are ordinary values, and one can arrive from argv or from a
use-value restart. test/programs/slurp.flan reads "programs/assets/a.txt" only because the suite runs from
_build/default/test. Two forms in one section with opposite rules is exactly where someone gets bitten.
The break loop can be offered this restart and cannot fill it in. That is not new behaviour, only a new way to
reach it: a break loop chooses a restart by position and has nothing to supply a parameter with, and emit.ml already
emits a flan_restart_unarmed guard on every clause that takes one, so taking it refuses with the reason rather than
running the clause on a zeroed buffer. slurp's use-value is simply the first such clause the compiler emits —
alloc_guard's retry takes no parameters — and it rides the same path a hand-written one does. The acceptance suite
asserts the guard is on the emitted IR for both.
On the web, barf signals — every time, with the path in the condition. This is the decision worth restating,
because two more obvious answers are both wrong here. A build-time refusal is unusable: Flan has no conditional
compilation, nothing in parse.ml or check.ml reads the target, so "isolate this to desktop" is not expressible in
source and the refusal would have nowhere to be silenced. A silent no-op is worse than either, because that is how
a save file disappears with nothing said. So the program gets a condition and decides — which is the language having
something Odin does not. Odin stubs its whole file API on js/wasm to .Unsupported (core/os/file_js.odin) so that
importing core:os "panics cleanly", and a panic is not a decision. The restriction was taken; the mechanism was
not.
Nothing in the compiler reads the target to do this. The refusal is one #ifdef __EMSCRIPTEN__ in flan_rt.c, which
is where the host ABI is already implemented twice. slurp keeps working on the web — it compiles, runs, and reports
a missing file honestly — and the bytes a web program actually wants come from an embed.
test/programs/web-files.flan is one source built for both targets, and test/test_web.ml runs it under node
rather than asserting the artifact's shape: an artifact-shape assertion would say nothing about the thing the decision
bought. The test checks that the refusal is printed with its path and reason, and that the desktop's success line is
absent — a silent no-op would have taken that branch.
What the host ABI grew by, and why that much
plan.org names the filesystem as the #1 portability risk — "pack assets, one abstraction, never touch paths" — so the widening is written down rather than assumed. It is three calls and one reader:
| Call | What it does |
|---|---|
flan_file_size(path, n, &out) |
how many bytes are there |
flan_file_read(path, n, buf, cap, &got) |
fill a buffer the caller owns |
flan_file_write(path, n, buf, len) |
write a whole file |
flan_file_fail_reason() |
which of the four reasons it was |
They are POSIX-shaped and Vec-ignorant: no handle crosses the boundary, nothing is held between calls, and each
takes a path and answers 1/0 the way every allocator entry point already does. flan_slurp_into — the part that knows
what a Vec is — is runtime glue on this side of the ABI, not a fourth host call, so a second target implements
three functions and inherits the rest. The reason is a global rather than an out-parameter for the same reason
flan_alloc_fail_bytes is: the condition is a value struct on the failing frame's stack with fixed numeric fields and
no rendered message.
These do touch paths, which is the widening plan.org warned about, and decision 2 took it knowingly. embed is the
half that does not: it needs no host ABI at all, so "pack assets" remains the answer for anything known at build time
and slurp is for bytes that genuinely are not.
FileError is one type with a reason field rather than a family, because conditions have no hierarchy today
(spec-conditions.md §1) and a family would need one handler clause per member to say "any file error". NEXT.md decision
4's parent link is the answer to that and is not built; when it is, these reasons can become types without any call
site changing.
One thing slurp and barf reveal that is not theirs to fix. A handler that wants "try to save, and carry on if
you cannot" has nowhere to go: error is diverging (spec-conditions.md §2), a handler returning normally has not
answered it, and neither retry nor use-value means give up. web-files.flan calls exit for that reason. A
continue-style restart — or handler-case, which unwinds — is what the case wants, and neither exists.
Where build time goes
flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object cache is in, and it is now ~110ms:
| Step | Cost |
|---|---|
| frontend: read → parse → load → check → emit | <10ms, below the timer |
clang on the .ll |
60ms — llc does the same codegen in 20ms |
clang on flan_rt.c |
40ms — now cached, paid once |
| link | 20ms |
Every C translation unit a build needs — the host shim and each package's shim — goes through Build.compile_c, which
compiles to a .o under $TMPDIR/flan-objcache and reuses it. The key is a digest of the source text, the compiler
(its path, size and mtime, so an upgrade invalidates without paying a clang --version subprocess per build),
opts.opt and opts.target. The opt level has to be in there: the acceptance table builds the same programs at -O0
and -O2, and an -O2 object must not serve an -O0 build. The object is written to a temporary name and renamed
into place, so two concurrent builds cannot see a half-written one.
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
the cost. The cache is keyed by content, so it never needs invalidating by hand — rm -rf on the directory is only ever
a disk-space decision.
The other cheap win is still open: skip the clang driver for the .ll (llc + link directly), worth another ~40ms. It
is a subset of the dev path's machinery. Check llc's major version against clang's before relying on it — the emitted
IR text is currently absorbed by the driver behind -Wno-override-module, and a version mismatch surfaces as IR parse
errors.
The order it was built in, and why that order
Every item here is done; it is kept because the sequencing is the part worth remembering. Decided in conversation: wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and the dev loop is the thesis of the project, so it came first. Staged so each step was runnable on its own — the failure mode being to build a daemon and a protocol before knowing whether the reload primitive worked.
The reload primitive, measured.Done —Emit.redefinition,Build.shared,test/reload_host.c, ~19ms. See the section above.Indirection cells.Done —Build.opts.dev/flan build --dev,flan_reload_install,runtime/flan_dev.cfor names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above.The agent, in C.Done —vendor/agent/, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above.The daemonand5. the Emacs client— both done, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the sameSessionif something else ever needs to talk to it.
One decision left to settle before step 2, because both change codegen and are painful to retrofit:
Do cells cover globals, or only functions?Settled by step 1: functions only. A redefinition module declares every globalexternal, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding adefvarto a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.- What is a redefinition unit — one function, or a file? A file is much easier to make correct and is what
load-filewants anyway; one function is whatC-c C-cwants and is where the 16ms number comes from.
Deferred until after the dev loop:
wasm32.Done, with one glued joint.flan build --target=wasm32-wasiproduces a module, andtest/programs/sand-headless.flanprints15595743031174623232under it — the same hash as native, byte for byte, at-O2and at-O0. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared across targets, and it compares equal.values.flanandmachine.flanrun there too, which is where a 32-bit pointer would have shown. The acceptance table runs all four, and skips them by probing — it builds the smallest program and runs it — rather than by looking for a binary on PATH.
Three things this cost that were not in the old note:
- The entry point is not
main. wasi-libc's start code calls__main_argc_argv; clang renames C's argc/argvmainto that, and the.llEmitwrites says@mainliterally. The link succeeds and the program traps on a signature-mismatched weak stub.Build.wasm_main_sourceis a two-line C shim that bridges it, and the__asm__("main")label in it is load-bearing: spelling the calleemainmakes clang rename that too and the shim becomes an infinite self-call. - The target has to reach the C compiles, not just the link.
flan_rt.cincludes<stdio.h>; without--sysrootit never gets that far.target_flagsis computed once and passed to both, and the whole flag list — not just the triple — is in the object cache key, so repointing a sysroot cannot serve a stale.o. - Fedora's sysroot is one level deeper than wasi-sdk's:
include/wasm32-wasi/stdio.h, notinclude/stdio.h. Both shapes count.
The glued joint, and the one thing this contradicts in the old note. The old note said the builtins archive has to
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
libcompiler_rt.a, and dropping that in as libclang_rt.builtins.a links and runs. It is a different triple
(wasm32-unknown-emscripten) built by a different clang (22 against Fedora's 20), so it is substituting, and wasi-sdk
is still the proper article. build.ml looks for FLAN_WASM_BUILTINS, then /opt/wasi-sdk/..., then emscripten's
beside emcc on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real include symlinked in.
The runtime is Node. No wasmtime and no wasmer on this machine; test/wasm-run.mjs is twenty lines of
node:wasi and the table prefers wasmtime or wasmer if either appears. --no-warnings, because node:wasi writes
an ExperimentalWarning to stderr on every run and the harness compares combined output.
Refused by name, not half-supported: --dev with a wasm target (the reload path is dlopen), Build.shared with
one (same reason), and flan run --target= (a .wasm is not something this host execs — build it and point a runtime
at it).
Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. Done — and it is a third
target, not a mode of this one. wasi-sdk is right for the headless table and was never going to be right for the game
build. See the next section.
The browser is the third target
flan build --target=web produces a page, its JS and a .wasm, and a raylib example opens in a browser from source
that was not touched. The two wasm targets share the word and almost nothing else, so is_wasi and is_web are
separate predicates and is_wasm is their union — the union is exactly the set of facts about the machine (32-bit
pointers, no dlopen), which is what the refusals are about, and nothing else is shared.
The compiler is emcc, not clang, and that is the whole of the sysroot story. Everything the wasi target has to
find by hand — a sysroot, a builtins archive, a shadow resource directory, the __main_argc_argv shim — is what emcc
is. target_flags for web is the empty list; the only thing checked is that emcc exists, refused by name where the
reason can say so. The one fact that had to be true for any of this: emcc takes a .ll on its command line, which
it does, so Emit's output needs no change and the IR stays target-independent. The object cache keys on the compiler
binary's path, size and mtime as it always did — now of whichever compiler the target uses, so an emcc flan_rt.o
and a clang one cannot collide.
The main loop: -sASYNCIFY, not emscripten_set_main_loop. The older note above had the browser fact right —
it cannot be blocked — and drew the wrong conclusion from it. emscripten_set_main_loop wants the loop body as a
callback, so every one of the eleven examples that writes
(until (rl/window-should-close?) ...)
would have to be split by hand into an init and a tick, and the web program would stop being the native program.
Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is
built for precisely that: WindowShouldClose() on PLATFORM_WEB is an emscripten_sleep(16) that then returns false
(raylib 5.5, platforms/rcore_web.c, read rather than assumed). So the loop yields once a frame at a call it already
makes, and no example changed a character. The price is real and is paid by every web build: asyncify instruments
the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a
question Build can answer, and a flag set that varies per program is a cache key that varies per program.
link lines can be addressed to a target. vendor/raylib/link named libraylib.so.550, which exists on the host
and nowhere else. A line may now carry @native, @wasi or @web, an untagged line applies everywhere — which is
what every existing link file already is — and ${NAME} expands from the environment. The selection happens in
Build and not in Load, which is where the file is read, because Load resolves imports before a target is
chosen: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings.
Load's part in this is to pass the lines through untouched, which it already did.
raylib for the browser is built, not installed. No emscripten port provides it (emcc --show-ports: there is
contrib.glfw3 and no raylib), so vendor/raylib/build-web.sh clones raylib at the 5.5 tag — the one whose
.so.550 the host links, because raylib.flan carries raylib's struct layouts and enum values and two targets built
from different raylibs would disagree about them in silence — and compiles the seven modules with
-DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2 into one archive under vendor/raylib/web/ (gitignored). rglfw.c is not
among them: the web platform uses emscripten's own GLFW port, which is why link carries @web -sUSE_GLFW=3. No
headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.
The HTML shell is a string in Build, not a file in the tree, for the same reason Runtime_src is: it has to be
wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas
and a Module.print that puts stdout on the page; FLAN_WEB_SHELL replaces it. --shell-file is passed only when the
output is a .html, because emcc accepts and ignores it otherwise.
Refused by name, inherited whole: --dev, --debug, Build.shared and flan run --target= are refused for
web exactly as for wasm32, each naming web rather than wasm32 in the message. --sanitize is refused too, but
the web half of that refusal is weaker than the wasi half and says so: emscripten does ship an ASan, and nothing here
has ever run it. A sanitizer that has never been run is one whose silence means nothing.
What the test can honestly check. test/test_web.ml is headless and permanently so. It probes — emscripten may not
be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it
asserts: the three files exist, the module starts with \0asm, the page references its own JS and carries the canvas,
and node runs the emitted JS and gets ok. For raylib it builds core-basic-window.flan unchanged and then reads the
module for the two things that would be false if the mechanism were wrong: an asyncify_start_unwind export, and a
glViewport import that can only have come from raylib's web platform. Import and export names are plain strings in
the binary, so this needs no wasm reader.
sand.flan in a browser
The flagship program builds for the browser and the artifact opens. Three things were between it and the target,
and none of them was the #include the earlier note named.
The brush was a path, and a path is what cannot work. (rl/load-texture "brush.png") hands raylib a filename to
open; a bare relative path has no meaning where there is no filesystem, so raylib would have opened nothing and the
cursor would simply have been missing. It is (embed "brush.png") now, decoded by a new binding —
LoadImageFromMemory, which completes the chain embedded bytes -> Image -> Texture2D that LoadTextureFromImage
already had the other half of. The declaration is (Ptr u8) plus an explicit i32 count, because shim.ml refuses a
slice parameter and its refusal says exactly that; the Flan wrapper beside it takes the slice apart, which is the
idiom collision-point-poly? and load-font-ex already established. One decode serves both textures now: the
unflipped upload first, then ImageFlipHorizontal in place, then the mirrored upload — the order is load-bearing,
and if the two badges look the same it was swapped.
load-texture and load-image lose their only call site in this repository by this change. That is deliberate rather
than an accident of editing: a path-based load is the one shape the browser cannot have, and the bindings stay for the
desktop programs that will want them.
The embedded path resolves relative to the file the form is written in, which is why
test/programs/sand-headless.flan still works: it reaches sand.flan through ../../ out of a sandboxed _build,
and the PNG is found beside sand.flan and not beside the working directory. test/dune therefore lists brush.png
as a dependency of every stanza that builds sand — an embed is read by the checker, so it is a build input and not a
run-time one. The same fact reached test_session's C-c C-k case, which re-evaluates sand.flan's whole text: it now
passes ~origin, which is the buffer path both editor paths already send, because the default <eval> origin would
resolve the embed against the working directory instead.
A package's C may be addressed to one target, the way a link line already could. Load collects a package's
.c files by listing the directory, and there is nowhere in a directory listing to put a tag except the name, so the
tag goes there, before the extension:
vendor/agent/flan_agent.c compiled everywhere, unless displaced
vendor/agent/flan_agent.web.c compiled for the browser, and displaces the above
One rule: a tagged file is compiled only on its own target, and there it replaces the untagged file of the same base
name. Untagged is the default and every existing package is untagged, so nothing that did not opt in changed.
Replacement rather than the pure tagging Go's _windows.go and Odin's file_js.odin use, and the difference is the
point — pure tagging would mean renaming flan_agent.c to flan_agent.native.c to teach the package about a target
it had never heard of, and this way a package gains a target by gaining a file. The selection is in Build and not in
Load, for the reason select_lflags gives: Load resolves imports before a target is chosen.
The dev agent on the web is a no-op, and it is not the barf decision being contradicted. The compile error that
led here — struct timeval incomplete, because emscripten's headers do not pull <sys/time.h> in transitively — is
the surface. Underneath: the agent is a socket server and a browser has no sockets. Adding the include produces an
agent that compiles, links, starts and can never accept a connection.
Refusing vendor:agent on a web target was the other candidate and is ruled out by arithmetic, not taste. Flan has no
conditional compilation, sand.flan calls (agent/start ...) unconditionally, and Reach cannot prune a package
something reachable calls into — so a build-time refusal means the flagship program does not build for the browser at
all without being edited into a second program. A refusal is only honest when the caller has a way to not ask.
The two decisions look contradictory and are not, and the difference is what the caller loses. barf is asked to
make something durable; a no-op returns success to a program that now believes the bytes are on disk, and the loss is
real, is the user's, and is discovered later or never. The agent is asked to accept redefinitions from an editor; on
the web there is no editor, no socket and no session — --dev is refused by name on every wasm target, so a web build
has no cells to install a redefinition into even if one arrived. Nothing is lost because there was never anything
there. sand.flan already says the same about a native release build, at the call site: "Building without --dev
is fine — nothing has cells to install into, so a module is refused on the listener thread and the loop never
notices." A web build reaches that outcome by a shorter route. start returns -1, which is what flan_agent.c
returns for a path it cannot bind; poll and wait return 0, which is what the native build returns on every frame
nothing arrived on. The whole argument is written at the top of vendor/agent/flan_agent.web.c, where the next reader
will meet it.
Building it and opening it
The raylib archive is built once and is not in the tree. From the repository root:
sh vendor/raylib/build-web.sh # clones raylib 5.5 and compiles it with emcc
export FLAN_RAYLIB_WEB=$PWD/vendor/raylib/web/libraylib-5.5.a
build-web.sh prints that export line itself. Then:
flan build sand.flan --target=web -o sand.html
The output must be named .html. --shell-file is passed only when it is, because emcc accepts and ignores it
otherwise — so -o sand produces a module with no shell, no canvas, and a page that looks like it built fine and
paints nothing. Three files land beside it, in whatever directory -o names: sand.html, sand.js, sand.wasm.
A file:// URL will not work. The page fetches sand.wasm, and a browser refuses that from the filesystem. Serve
the directory holding the three files:
python3 -m http.server 8000
and open http://localhost:8000/sand.html. Left mouse paints; the keys are the ones the native build has.
What only a human opening it can settle
Verified headlessly, by test/test_web.ml: the three files exist, the module carries asyncify_start_unwind and
glViewport, and brush.png's own bytes are in the module, whole — the assertion that keeps the embed from
rotting. Not IHDR: stb_image, linked in from raylib, carries that string itself, so an IHDR check would pass on a
build where the embed emitted nothing.
Not verified, and not verifiable here. node sand.js instantiates the module, runs main, and dies inside glfwInit
on window is not defined — which says the module is live and says nothing about the canvas.
- Whether it paints at all. Nothing in CI has ever seen a pixel of this.
- Audio.
start-audiogenerates a tone,ExportWaves it to/tmp/flan-sand-tone.wavand loads it back as a music stream. That is raylib's ownfopen, not Flan'sbarf, so the#ifdefinflan_rt.cdoes not cover it and emscripten's MEMFS may well give it a writable/tmp. Every use is behindmusic-ok/tone-ok, so a failure is silence and not a crash. Separately, browsers suspend the audio context until a user gesture, soaudio-okmay be true while nothing is heard until the first click. - The loop never exits.
WindowShouldClose()onPLATFORM_WEBis anemscripten_sleepthat returns false, sountilnever terminates and none ofmain'sdefers ever run — noCloseWindow, noUnloadTexture. That is correct for a page, which is torn down by the tab closing, and it is worth knowing before reading anything into it. - Canvas size against
screen-width/screen-height. The shell is a string inBuildand its canvas is not sized from the program, so 900x600 may be letterboxed or cropped.
The colon belongs to keys, so a field label is a dot
{.x 1.0 .y 2.0} is how a struct is constructed, and {inner .field} is how a pattern names one. The colon is gone
from both, and what is left of it is one job: keys — map keys and enum members.
What was wrong with the colon. Nothing, taken alone. The problem was that it had two jobs and the dot had one.
(.x v) already read a field; {:x 1.0} also named a field, while :space named an enum member. So the dot meant
"field" and the colon meant "field, or member, depending". Moving the label to the dot leaves each mark with one
meaning, and it costs nothing to read because the delimiter already disambiguates: (.x v) is a list and
therefore a call and therefore an access; {.x 1.0} is a brace form and therefore a construction. There is no
position where the two could be confused, which is why the same spelling can serve both.
Why it had to land before Map, and this is the real reason. A map literal wants to be {:key value}. While
struct construction owned that exact spelling, a map literal and a struct literal were the same syntax, and the
only thing that could tell them apart was what the checker expected at that position. That is a context-sensitive
grammar for no gain. Reserving the colon for keys keeps the two visibly distinct at the reader, before any type is
known. Doing it after Map landed would have meant changing both; doing it first meant changing one.
:keys kept its colon, and that is the point rather than an inconsistency. {:keys [x y]} is the one thing in a
brace that is not a field name — it is an instruction to the compiler that happens to sit there, and it takes a
vector rather than a value. Giving it a dot would have made the dot mean "a field, or the word keys". Leaving it a
colon lets the dot mean exactly one thing, this names a field, which is the whole reason the colon was given up.
Everything else Clojure puts in that position — :as, :or, :strs, :syms — is still refused by its own name.
The old spelling is refused, not accepted quietly, and the refusal names the new one: a field label is written .x, not :x — the colon is for keys. Two accepted spellings is how two spellings become permanent, and the standing
rule here is that what is not supported is rejected explicitly with the reason. Both refusals are tested by their
reason, on the construction side and on the destructuring side, which is what stops the colon drifting back.
The sweep is a tool, not a one-off. tools/colon-to-dot.py converted 681 labels across 45 .flan files,
vendor/ included, and 94 more in the Flan embedded in lib/prelude.ml and the tests. It works on forms, not on text: a keyword
becomes a dot only where it sits in a field-label position inside a brace, so an enum member in value position
({.k :hi}), a genuine EDN map inside a string (test/programs/edn.flan), and a type-position {K V} are all left
alone. It was kept in the tree because several lanes branched before it and their Flan needs the same pass at merge.
What it deliberately did not change: the printed form. render.ml still prints (V {:x 1.5 :y 0}). That string
is a wire format — emacs/flan-inspect.el parses it back and hard-codes the colon when it reads a field out — so
moving the printer alone would break struct inspection in the dev loop without breaking any test that says so. The
printer moves when its reader does, in the Emacs lane. It is the one place the old spelling is still correct, and
the reason is worth keeping: a format with two ends only changes at both.