Two follow-ups to the marking commit. `Dev.eval_expr`'s new wait matched `Stopped _`, which fires on the first iteration when the program is already parked on something else — the break loop allows evaluating, so that is reachable — and answers for a thunk that has not run yet, on a reply whose own `:condition` names the other condition. It now waits for `Stopped "Pause"`, which the agent reports under a nested break because `condition_name` is overwritten on the way in and restored on the way out. `dev-pause.flan` grows a `Missing` and a `boom` so the test can park the program on something else first and tell the two apart. And the flake NEXT.md had as "seen once and unexplained": `the daemon never listened` is not a race, it is an llc-and-link of the whole program before `flan dev` binds — ~600ms idle, measured at 6.6s and 6.8s with the rest of the suite beside it, against a 5s and 8s await. All three test binaries now wait a minute; the watchdog is what bounds the run. Two consecutive full runs green.
364 KiB
How the parts that exist work
The reasons behind the code, kept separate from NEXT.md so that what to do next is not buried under what was already
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever dlclosed, why a call
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
than a function in the runtime. Deleting it would mean deriving it again.
NEXT.md is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
first. Come here when you need to know why something is the shape it is.
What milestone 4 added
dotimes desugars in check.ml to a Let plus a While — no new IR node. The bound is evaluated once into a
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
assignable, which makes the generated step its only writer.
defer is recognised in check_fn and nowhere else, because that is the only place that knows a form is at the top
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
stands. Function exit runs them innermost-first, and an explicit return runs the ones registered above it — a defer
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
bounds-check shape (noreturn then unreachable) rather than being a separate decision. Amended once a bounds
failure became a signal: an answered one leaves through the unwind path and runs them like any other transfer, an
unanswered one still runs none. See "An index out of range is a condition" at the foot of this file.
defer inside a let, a loop or a branch is rejected, not accepted with function scope. It would run once at
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
Block-scoped defer is real work and is not done.
New builtins: zeroed (takes its type from the place it is stored into), min/max (each operand through a slot,
so neither is evaluated twice), bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a signed
type and logical on an unsigned one), and rand-f32.
rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state. It is not libc's, because a grid hash is
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed sets
the state. This is what the bitwise operators were added for.
Enums and keywords. (defenum Name [member value ...]) gives a type that is an i32 at run time and its own type
in the checker, so :space at a call site resolves against the parameter's enum and a typo is an error there rather
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
Why the FFI goes through a C shim
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
per-target classification the caller has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
in emit.ml is three classifiers to write and then keep correct forever, and a mistake shows up as (.y m) returning
garbage rather than as a link error.
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
copy. clang classifies all of it, per target, for free. check.ml enforces the rule — an aggregate in a declare
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
ABI, implemented twice", and flan_rt.c is the same pattern.
The wrappers are generated now — declare-c, lib/shim.ml
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
generate "if that ever becomes the bottleneck" came true at 84 of them. vendor/raylib/shim.c is gone; the directory
holds raylib.flan and link and no C at all.
One binding is now one line:
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
declare-c names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
any other: the typedefs for the structs involved, made from the Flan defstructs; the extern prototype in the
function's true signature; the wrapper that flattens it; and the flattened declare the Flan side calls, with an
ordinary Flan defn above it when the signature has a struct in it. flan shim <file> prints the whole file.
It is a second form and not a change to declare, for one reason worth remembering: (declare start-raw [path string] i32 "flan_agent_start") in vendor/agent means the symbol takes ptr+len, and (declare-c init-window [w i32 h i32 title string] "InitWindow") means it takes a NUL-terminated char *. Same shape, opposite claims, so no structural
rule can separate them. declare is untouched, and sqrtf and the agent still work unedited.
What the generator guarantees, and what it trusts. Guaranteed: the C typedef and the Flan struct come from the same
defstruct, so they cannot disagree — permute the defstruct and the typedef permutes with it, which is exactly what
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
that the defstruct matches the library's real struct, and that the declare-c signature is the function's real
signature — no header is read, deliberately, so nothing can check either. A _Static_assert on sizeof/offsetof was
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
for every field type the generator admits — the machine integers, the two floats, bool, a pointer and a nested struct
— LLVM's layout is C's, and emit.ml writes no datalayout, so clang applies the target's rules to both halves.
Everything where they could diverge is refused at the field.
One thing got sharper and should be said plainly: the prototype is now generated from the declaration, so a
scalar's width carries ABI weight it did not before. f64 where raylib says float used to be narrowed by clang at the
hand-written call site; now it emits double and raylib reads garbage. All 84 migrated prototypes were diffed against
the deleted shim.c's — which was the ground truth for the true signatures — and agree.
Strings. The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, PATH_MAX for a path,
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
malloc failure, where the alternative is handing C a null pointer.
Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C. collision-point-poly? takes a slice
and collision-lines answers with an Option; neither is raylib's own signature. A slice parameter in a declare-c is
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
recoverable from [T] — so that one declares (Ptr Vector2) with an explicit count i32 and the Flan wrapper passes
(addr (at points 0)) and (len points). Every other refusal — an Option, a union, a fixed array, a map, a returned
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
the reason, and the acceptance table asserts on the reasons.
Known edge, not fixed: a REPL redefinition that introduces a new declare-c cannot work. Build.shared is llc +
ld -shared and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
that calls an existing binding is unaffected.
raylib.flan carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
(rl/get-mouse-position) returning a Vector2. Verified end to end, headless: GetColor(0x11223344) comes back as 17 34 51 68, four separate bytes — a Color is not the little-endian reading of the packed integer, so an identity
would have passed a weaker test. That case is in the acceptance table, skipped if libraylib is not installed.
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven collision-* predicates; textures; the Image family; Camera2D; RenderTexture2D; the whole audio surface (device, Wave, Sound, Music); fonts and glyphs; and gamepads, touch and gestures — plus the Key, MouseButton, TraceLogLevel, GamepadButton, GamepadAxis and Gesture enums, raylib's own named colour palette, and the FLAG_ window hints. Adding one is a single declare-c line; there is no C to write.
Two things the ported examples in examples/ wanted and could not have, both refused for reasons that are right. GetGamepadName returns a char * into raylib's static storage: the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that returns one returns something Flan has no owner for. And an enum parameter cannot be indexed — GetGamepadAxisMovement takes a GamepadAxis, a loop variable is an i32, expected rl/GamepadAxis, found i32, and a second declare-c of the same symbol with an i32 face is refused too: one declare-c per C function, and another Flan name for it is a defn — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a cond over the members it knows.
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
GPU: GetCollisionRec, which pins Rectangle completely, and SetShapesTexture's default substitution, which pins
Texture2D's id and format and nothing else. Handing a struct over and reading it back proves nothing at all —
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. width,
height and mipmaps are therefore checked only by looking at sand.flan running, which draws the brush sprite four
ways for that reason.
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
library being linkable and not on raylib-devel. vendor/raylib/link carries -l:libraylib.so.550 because Fedora
ships the runtime library without the .so symlink.
The header is read now — headers, lib/cimport.ml
The section above ends by naming what the generator trusts: that the
defstruct matches the library's real struct, and that the declare-c
signature is the function's real signature. "No header is read, deliberately,
so nothing can check either." A header is read now, and both are checked.
The dependency, which is the crux, and which this project already answered
once. Zig's old @cImport ran clang as a library. That is exactly the
dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++
library breaks routinely on upgrade, a binary on PATH does not. So this shells
out for clang -Xclang -ast-dump=json -fsyntax-only, which is the same binary
every build already runs and adds nothing that is not already being paid for.
lib/cjson.ml is enough JSON to read that dump and no more, so it adds no opam
package either.
A note worth recording, because it strengthens the argument rather than
weakening it: Zig has since abandoned clang here too. translate_c.zig is
gone; lib/compiler/translate-c/ is built on Aro, a C frontend written in Zig.
Their reason was to ship a compiler containing no clang at all — the opposite
premise to this one, where clang on PATH is the toolchain assumption. Both
projects walked away from linking libclang; only the destination differs.
What is imported: functions, and only functions. Not structs, not enums,
not macros. The bound on how much is not a curated list but the package's own
defstructs — a function whose signature mentions a struct the package has not
described is refused with that reason, so vendor/raylib describing thirteen
structs is what makes the import thirteen structs wide, and describing a
fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are
refused, and 172 are left alone because the package already binds them by hand.
(253 and 156 once bindings excludes raylib's three allocator entry points —
see the next section.)
Not generating defstructs is what makes the check possible at all. Generate
them and the header becomes the authority on layout, and comparing the
package's layouts against the header's would be comparing the header with
itself. A _Static_assert on sizeof/offsetof was rejected in the section
above as circular for exactly that reason; this is not circular, because the
two sides have different authors. It is the cheapest real closure of the
trusted-not-guaranteed gap.
Refusing by demotion, which is the one thing taken wholesale from Zig. Zig's
translator never drops a declaration it cannot handle: failDecl binds the name
to a @compileError carrying the reason, so the name still exists, the program
still compiles, and asking for that one name fails at the use site with the
reason. A wholesale import has a hundred and fifty refusals and a caller cares
about the one they typed. Flan already had that mechanism — Load.refuse_hidden,
built for main — so rl/get-gamepad-name is not a name, and a program that
writes it is told the return type is a string, and a string only crosses as a
parameter rather than "unknown name".
That is also the split shim.ml needed and did not have. It refuses through
Loc.fail, which is right when a human named one function and wrong for a
wholesale import, where one returned const char * would kill the header. Same
judgement, different disposition; a hand-written declare-c still hard-fails
and shim.ml is untouched.
Two C spellings mean something in a parameter that they mean nowhere else.
const char * is a string going in, and the generator already knows how to hand
one over. char * without the const is very often a buffer the callee writes,
and handing it a NUL-terminated temporary would lose the writes with no
diagnostic anywhere — const is the only thing in the header that separates the
two, so it is what decides, and an out-buffer keeps a hand-written binding
saying (Ptr u8). long, size_t and the rest are refused rather than
guessed, and for a reason specific to this project: it builds for x86-64, for
wasm32-wasi and for the browser, and long is 64 bits on the first and 32 on
the others, so a guess would be right for the target that gets tested and
silently wrong for two that do not.
Naming. rl/InitWindow is rl/init-window. Reversibility is not a property
of the rule — the C symbol is stored verbatim in the declaration, so the wrapper
reads the library's spelling rather than reconstructing it. What the rule must
be is injective over one header, since two C functions arriving under one Flan
name would surface as a duplicate declaration about a name nobody wrote. A
boundary goes before an uppercase letter after a lowercase one, before an
uppercase letter between an uppercase and a lowercase, and before a digit after
a lowercase; nowhere else. So SetTargetFPS is set-target-fps and not
set-target-f-p-s, BeginMode2D is begin-mode-2d, UnloadUTF8 is
unload-utf8. raylib's 581 names are injective under it. When two do collide,
neither takes the name — resolving by order would mean that moving two lines
in somebody else's header silently rebinds a name a program is already calling.
Both are refused, both say why, and the author binds the one they want with a
declare-c.
Where it runs. headers beside link in the package directory, read the
same way: a path, any clang flags it needs, ${NAME} expanded from the
environment. What comes back is ordinary declare-c declarations, generated
before the package's names are qualified, so they arrive as rl/… exactly like
the hand-written ones and nothing downstream can tell which is which. No new
form, no new decl_kind, no reader or parser change. A C symbol the package
already binds by hand is left alone, so declare-c stays the escape hatch and
stays the thing that wins.
A leading ? makes a line optional, and vendor/raylib uses it. "No raylib
headers are needed" is a real property — a build needs libraylib linkable, not
raylib-devel installed — and requiring a header would take it from everyone in
order to give the check to whoever has one. Unset FLAN_RAYLIB_H and the build
is exactly what it was; set it and every signature is checked. A path that is
set and wrong is an error naming it, because silently behaving as though
nobody had opted in is the difference between an opt-in and a trap.
What the diff found
The evidence the whole lane exists for. Against raylib 5.5 — the version
whose .so link names — all 16 defstructs and all 172 hand-written
declare-c agree exactly. The half BUILT.md called trusted is now checked,
and it was right.
That is only worth stating because the check has teeth. Against the 5.1-dev
header installed in /usr/local it reports ten differences: nine functions that
version does not have (CheckCollisionCircleLine, the six Is*Valid renames,
DrawRectangleRoundedLinesEx) and DrawRectangleRoundedLines, which gained a
parameter. Picking the wrong header is therefore loud, which matters, because
the two headers are on the same machine and only one matches the linked library.
Both comparisons run at build time and stop the build, not just in the tool.
Verified by breaking them: a permuted Texture2D fails naming the field that
moved, and f64 where raylib says float fails naming the parameter — which is
the hazard the section above calls out by name and says only a test can catch.
The message points at the line in raylib.flan, not at the header.
flan import-c <header> [package.flan…] prints what it would produce, what it
refused and why, and both comparisons, without building anything. That also
makes "generate once and commit the result" available for the cost of a
printer — explicit in the source, checked against reality, no header read at
build time.
What it costs, measured
The number that decides how much to import, because reach.ml was the reason to
think a wholesale import could be free.
| today (172 by hand) | + 256 imported | |
|---|---|---|
| release build, cold | 0.298s | 0.312s |
| release build, warm | 0.078s | 0.082s |
redefinition (flan reload) |
31.0ms | 46.5ms |
| dev build, cold | 0.649s | 0.982s |
Reach.link already drops a generated wrapper whose declaration nothing
reachable calls, and that is what makes the release column nearly flat.
Confirmed on the case it exists for: a wasm32-wasi build of a program that
imports raylib and calls none of it still links without libraylib, with 256
extra declarations in play. Dev builds are not pruned, on purpose, so one
compiles all 428 wrappers — once, at session start, since Build.shared is
llc + ld -shared and compiles no C.
Reading the header is cached, and the cache earned itself against a measurement rather than a guess: 64ms of a 72ms check, against 8ms for the whole program without it. What is cached is the extracted signatures and not clang's JSON, because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest is reading it. Keyed the way the object cache is keyed, on everything that could change the answer: the header's path, size and mtime, the full flag list, and a format version, since the value is marshalled. That takes the delta to 17ms.
The 15.5ms on redefinition is the real cost and it is the argument against importing at build time, on the branch where the dev loop is the priority. It is the strongest case for the third option — generate from the header, commit the result, regenerate when the library moves — and that decision is open.
Where that 15.5ms actually is — and it is not the header
Written above as though a redefinition were re-reading the header on every
reload. It is not, and the number was never taken apart. Taken apart now, with a
timer around each stage of the same flan reload (raylib 5.5, sand.flan, one
defn in the forms file, warm caches):
| no header | header imported | delta | |
|---|---|---|---|
Session.create |
7.0ms | 21.5ms | +14.5 |
Session.eval |
4.8ms | 8.4ms | +3.6 |
llc + ld |
21ms | 21ms | 0 |
And inside that create, the header's own share: reading the cached dump
0.33ms, turning it into declarations 3.3ms, checking the package's
layouts and its hand-written signatures against it 0.55ms. Roughly 4ms of
the 14.5, once. The other ~10ms is Load and Check doing their ordinary work
over 256 more declarations, and so is the +3.6ms on eval.
So a C-c C-c in a daemon pays no header cost at all, and never did.
Session.eval puts the evaluated forms through Load.program, and a form list
with no (import …) in it reads no package and therefore no header. The import
runs when the session is created, and again on a C-c C-k whose buffer carries
the package's own import line. flan reload is a fresh process per
redefinition, which is why its number carries a session's startup cost — it is
the CLI's shape, not the dev loop's.
What a redefinition does pay for an imported package is the +3.6ms of checking
and emitting against a program with 256 more declarations in it. That is in
Check and in Emit.redefinition declaring every sibling; it is real, it is
the number to attack next, and no cache touches it.
Two caches, and what a header change mid-session means
Both levels exist now. On disk, keyed on the header's realpath, size, mtime, the full flag list and a format version — the object cache's own convention, and it now lives in the object cache directory rather than a second one beside it. Cold that read is 50ms of clang and parse; served from the file it is 0.33ms.
In the session, two tables in Cimport: the extracted dump by header, and
the generated declarations by header and by what the package already declares
(names taken, structs and enums known, C symbols bound by hand), so a package an
evaluation has added a decl to is worked out again rather than served a stale
answer. A repeat import in one process goes from 3.65ms to nothing — measured on
a whole-buffer evaluation, which imports twice.
A header edited mid-session does not take effect until the session is
restarted, and that is deliberate rather than incidental: the in-memory key is
the header's path and flags, with no mtime and no stat. It is the rule a
changed .c file already follows — the daemon compiled the package's C at
startup and a redefinition does not recompile it — and the rule the running
program follows, since its struct layouts are the ones it was built with. Taking
a new header mid-session would mean type-checking new signatures against a
process still running the old layouts, which is the silent disagreement the
whole header check exists to catch. The disk cache does key on mtime, so the
next session reads the new header rather than the old dump.
One attribution to not repeat: the disk cache is not what the dev build's +333ms cold is about. Measured on this machine, a cold dev build of sand.flan is 1.37s without the header and 1.83s with it and both caches cold; clear only the header cache and the same build is 0.92s against 0.83s. So the header read is ~60–90ms of a fresh session and the rest of the cold delta is the object cache compiling a shim with 428 wrappers in it, which is the object cache's business and already warm after one build.
The bindings are committed now — generated.flan, bindings, flan generate-c
The section above reads the header at build time, behind an opt-in
?${FLAN_RAYLIB_H} in headers, because a build should need libraylib
linkable and not raylib-devel installed. That opt-in was doing two jobs and
only one of them was defensible: it decided whether a check ran, which is
fine to make optional, and it decided whether a package had 172 bindings or
428, which is not. DrawTexturePro being reachable only by exporting an
environment variable is the shape of that second job, and it blocked a real
game — see PORTING.md.
Generate once, commit the result, regenerate when raylib moves.
flan generate-c vendor/raylib reads the header named by headers, writes
vendor/raylib/generated.flan, and that file is checked in. No header is
needed by anybody: every build gets all 425 declarations, they are greppable,
and they show up in a diff when the library moves. Caching is not the
argument — the dump is already cached on disk and in memory, so a build that
reads a header pays for it once either way. The argument is the dependency and
the diff.
What it costs is the check, so regeneration runs it and the check gates the
write. A file on disk has no second opinion, so nothing compares the bindings
against the library on an ordinary build any more. The one function that writes
generated.flan therefore compares first — every defstruct against the
header's record, every hand-written declare-c against the header's signature
— and writes nothing when they disagree. It is not possible to regenerate
without comparing, because there is no other way to write the file. Pointed at
the 5.1-dev header on this machine while the package is written for 5.5, it
reports the same ten real differences the diff above found and writes nothing.
The 172 hand-written lines stay, and the reason is not caution. It was tempting to delete them: 136 of the 172 are exactly what the kebab rule would have produced, and the other 36 could be spelled as name overrides, so the generated set really is a superset. The argument against is the one this section is about. Everything the generator emits agrees with the header by construction — the declaration and the prototype come from one dump — so diffing generated output against the header it came from is a tautology, and replacing the hand-written set would quietly reduce the signature half of the check to nothing. The hand-written lines were transcribed from raylib's documentation by a person; they are the only declarations in the package a header can actually contradict. All ten of the 5.1-dev differences came from them. They are not a parallel set to maintain — they are the second opinion, and the check is what maintains them.
The opt-in did not go away, it stopped deciding anything important. With
FLAN_RAYLIB_H set, an ordinary build still reads the header, and since every
C symbol is now bound — by hand or by generation — the importer generates
nothing and the read is purely the check. It is a better check than before:
425 declarations rather than 172, because generated.flan is a package file
like any other and is checked like one.
bindings, beside headers, is what survives regeneration. A committed
generated file cannot be hand-corrected — the next run overwrites it and the
edit is destroyed without anybody being told, which is the worst shape an edit
can have — so the corrections have to live somewhere regeneration reads. Two
directives, which are the two things the header cannot decide: exclude <symbol or pattern> and name <symbol> <flan-name>. A postprocessing transform pass
was considered and rejected: a second program to understand, run over text the
generator had already committed to.
Both are applied while the declarations are made, which is not a detail. The
kebab rule is consulted in exactly one place, so collision groups are computed
on the name a function will really take — which means renaming one of two
colliding symbols dissolves the collision instead of leaving both refused, and
Spin2D/spin2d gains a way out that is not a hand-written line. An excluded
symbol still reports that it was excluded rather than going quiet: "there is no
such binding" and "the package decided against this binding" are different
answers.
What is actually in raylib's: exclude Mem*, because raylib exports
malloc/realloc/free under its own names and binding them would put a second
untracked heap behind three innocuous-looking Flan names, against plan.org's
rule that an operation never falls back to a hidden allocator. And 19 name
lines giving the generated predicates the ? spelling the hand-written ones
already use — window-ready? rather than is-window-ready, because
key-pressed? and is-window-ready living in one package is precisely the
split this change exists to remove. The C symbol is kept verbatim in
Ast.DeclareC either way, so a rename costs nothing: it is still what is
called and still what the check compares against.
What committing 253 more declarations costs, measured
The section above says the cost is the check. That is the cost that mattered, but it is not the only one, and this file does not omit measured numbers.
Every build now carries 425 declare-c where a default build carried 172, and
the obvious worry is the shim: BUILT.md's own cold-build attribution above
blames "the object cache compiling a shim with 428 wrappers in it", and that
was the opt-in path. It is not what happens, because Reach.link drops the
bindings nothing reachable calls and the shim comes back in parts for exactly
that purpose. sand.flan links 110 wrappers, not 425 — nm sand | grep -c flan_shim_. A program that never draws still compiles no drawing wrapper.
(flan shim <file> prints the unpruned view, so it says 427 and is not the
number a build pays.)
What is left is frontend work on 253 more declarations, and it is small:
cold build of sand.flan, object cache cleared |
best of 3 |
|---|---|
| 425 declarations (committed bindings) | 2.16s |
172 declarations (generated.flan moved aside) |
2.10s |
+65ms, about 3%, and cold only — the object cache serves the shim after one build, and the redefinition path never recompiles C at all. Against it: the header read this removes was 60–90ms of a fresh session by the measurement above, and DISCUSS.md 6a measured it at 15.5ms on every redefinition, which is a 50% increase on the number the dev-loop lane exists to keep small. So for anyone who had the opt-in switched on this is a straight win, and for everyone else it is 65ms once per cold build in exchange for 253 bindings that were previously unreachable.
Two bindings the config cannot express stay hand-written, and they are the
reason declare-c remains the escape hatch: LoadFontEx and
LoadImageFromMemory are bound -raw and wrapped by a Flan function of the
same name without the suffix, one taking a slice and one answering with an
Option. The importer refuses them by name collision with those wrappers,
which is the correct answer.
What a headless FFI test can and cannot pin
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
-
A struct round trip is worthless. Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C writes and reads the same wrong slots and the test passes for any field order. Found by permuting two fields and getting identical output.
-
Axis-aligned geometry cannot pin
Vector2. Exchangingxandyis a reflection, applied to the inputs on the way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance, passes with the fields swapped — verified by swapping thedefstruct, which is what the typedef is now made from. Distances are worse: the reflection does not even reach them. -
What does pin
Vector2is the rotated camera, because a 90-degree rotation is not axis-aligned and therefore does not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision cases look like they cover it. -
What pins
Rectangleis arithmetic on its fields —GetCollisionReccomputes four numbers from four different field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged. -
Scalars in, fields out is the strongest shape there is, and the Image family is where it was finally available.
GenImageColor(4, 2, colour)is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values in four adjacenti32slots, with no input struct for a permutation to cancel against. That pinsImagecompletely, including thatdatais present and first;Texture2Dcould never be pinned that way because nothing without a GPU reads its width, height or mipmaps at all. -
A non-square image is an axis discriminator.
GetImageColorindexesy*width + x, so on a 4-wide, 2-tall image(3,0)exists and its transpose does not: exchange the wrapper'sxandyand the read goes out of bounds and answers transparent black.ImageFlipHorizontalagainstImageFlipVerticalsays the same thing twice more. -
A file is external ground truth, so
ExportImagethenLoadImageis not the symmetric round trip the rest of the package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by thex/ypermutation above.
The rule that falls out: make raylib compute something whose answer differs per axis, then verify the test can fail by permuting the fields and watching it go red. A case not verified that way is decoration.
One correction to an assumption that has now cost two lanes a guess: MeasureText is not headless material. It
measures with the default font, LoadFontDefault is not exported, and nothing but InitWindow loads it — so with no
window it answers 0 for every string. Measured against libraylib.so.550, not reasoned about. GetFrameTime, GetTime
and GetScreenWidth/Height are all 0 headless for the same kind of reason. All five are bound, and all five are
exercised by running sand.flan and looking, which is the whole of what can be claimed for them.
Packages
lib/load.ml resolves (import rl "vendor:raylib") before the checker runs. The directory is the package; vendor: is
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
collection is relative to the importing file. Importing is a rename: every top-level name the package declares
becomes alias/name, and every use of one — in a type, in a body, in a struct literal, in an array length — is
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
declarations whose names contain a slash.
A package may also carry the C it binds to: every .c file in the directory is compiled into the build, and a file
named link lists extra linker arguments. Whether those reach the build at all is decided after checking — see below.
A package may be a single .flan file named outright, rather than a directory. That is for the program that is also
a library: sand.flan shares the repository root with three other loose programs, so naming its directory would import
all four. A file carries no .c and no link file; those belong to a directory.
A package may import a package. The qualification flattens to the inner alias — raylib imported by a package that
is itself imported is still rl/…, never sand/rl/… — because a directory reached along two routes has to arrive under
one set of names or the checker sees every declaration twice. It is the one decision here that would be hardest to
change later, because a qualified name is an identity the layout op and the break loop both resolve by.
A directory is keyed by its real path and read once, so a diamond shares one copy of its bottom package.
programs/pkg-diamond.flan is the case that proves it at the level that matters: area/ and draw/ both import
shape/, a shape/Box is built inside area/ and passed to a function declared inside draw/. Read shape/ twice
and there are two structs both named shape/Box which do not unify, so what proves the dedupe is that the program
prints 3 6 20 and not that it compiles.
The same directory under two different aliases is refused, including when one of the two is a package's own import
and the other is pages away in the entry file — programs/pkg-alias-clash.flan.
A ring is refused and named. Earlier it was not: the read-once table swallowed the second arrival, so mutually
dependent packages appeared to work, and BUILT.md said so. What that cost is a definite package order, which is what
the macro expander needs — every defmacro has to be compiled before anything that calls it — so it is now a refusal
that names the ring, a -> b -> c -> a, rather than the single import that happened to close it. The two questions are
kept apart by two pieces of state: the chain currently being read, and the set already finished. Found in the first is
a cycle; found only in the second is the diamond's second route and still a no-op.
Load.t.pkgs comes back dependencies first — the topological order the acyclic rule buys. The declaration list is
deliberately unsorted: check.ml collects every top-level name in one pass before it checks any body, so top-level
names are order-independent by construction. The order exists for the expander, which cannot work that way.
Visibility is one rule: main is not exported. A package carrying one would collide with the importer's the moment
anything imported it, so a program could never be a package; and main is a reachability root, so an imported one would
keep everything it calls alive. Writing sand/main is refused at the line that wrote it, with the reason — left to the
checker it would be "unknown name", which is true and useless.
Still missing: a package-private marker for anything other than main, which is why rl/get-color-raw is callable.
The gap is surface syntax and not load.ml — exported is one predicate and the refusal machinery that points at the
line which tried already exists, so a second rule is a line. What does not exist is any way for a package to mark a
name private, and inventing one is a reader and parser change.
The link follows the program
Load used to hand a package's .c files and link arguments to the build the moment it was imported, whatever the
importing program did with them. So anything naming vendor:raylib linked libraylib on every target, and on wasm32 that
link cannot succeed.
lib/reach.ml answers it from the checked program instead. Start at main and at the globals that run before it,
follow every call — including the Handled frames, where a lifted handler clause is reached by address and by nothing
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
wasm-ld would fail on the symbols rather than on the argument list. So the same walk prunes functions and externs
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
dropped one is a silently different program.
Dev builds are not pruned. What a REPL may redefine next is not a function of what has been called so far.
The filtering happens at the call sites — bin/main.ml, the tests — because Build.executable receives csrcs and
lflags from its caller and never sees the import list. Reach.link returns the pruned program and its C and linker
arguments together, so a caller cannot take one without the other.
sand.flan is one program
It was two files, and only ever for the reason above: the headless run is the one CI does on native and wasm32, and a
program that imported raylib linked libraylib whatever its main did. So the simulation lived in sand-sim/ and both
drivers imported it.
Now sand.flan holds the simulation and the raylib front-end, and test/programs/sand-headless.flan imports
sand.flan itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
raylib; sand.flan's main is not exported, so the only main is the headless one; and the hash is unchanged on both
targets at -O2 and -O0, which is the point. A refactor that moved that number would have moved the simulation.
What still justifies two entry points is smaller and stands on its own: the headless test needs no window and no
input on any target. sand.flan could not be that test even natively — with no mouse the grid stays empty and
settle and move-grain never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
Two claims that got run together in an earlier note, for the record:
- raylib does not work on wasm — false. It works through emscripten. What is true is that it does not work on the wasi path, which is what the headless table targets, and which has no GL and no browser.
- a game loop cannot be expressed on wasm — false. The browser cannot be blocked, so a web build drives the loop with
emscripten_set_main_loopinstead of awhile.That is a differentThe premise held and the conclusion did not. It is the samemain, not a different program.mainand the same program:-sASYNCIFYanswers the same browser fact without cutting anything in half. See "The browser is the third target" below.
Three edits were made to sand.flan's own text when it was ported, and they are language decisions rather than fixes:
(defconst gravity 0.05)→(defconst gravity f32 0.05). An untyped float constant isf64,velocityis[f32], and there is no implicit widening.(defvar current-color u32)→i32. It is an index intocolors, and(len colors)is ani32.(defn main [])is unchanged — the short form, as plan.org says.
Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
Bounds checks — done at milestone 3
at and slice emit icmp → br → cold block → call → unreachable; a failure names the source location. Three
check sites: at on [n T] (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
never reaches emit, check.ml rejects it), at on a slice or string (runtime len), and slice (two comparisons — lo <= hi is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
Build.opts.checks is on by default and not tied to opts.opt, which is what lets the acceptance table run the
same programs at -O0 and -O2 with identical checks. The flag is --no-bounds-checks.
The write path is its own case: (set (at arr n) …) lowers through place/Pindex, not through At, so a refactor
that split them would break the write check silently. The test covers both.
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.
Sanitizers, and why hand-written IR does not get them for free
--sanitize builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last
of those is not what passing -fsanitize=address to the clang run over the .ll gets you, and the gap is silent.
AddressSanitizer is an LLVM pass, but it instruments only functions carrying the sanitize_address attribute. That
attribute is put there by clang's C frontend. Emit writes .ll by hand, so it wrote none, so the pass walked past
every Flan function and instrumented flan_rt.c. The measurement that settled it: an out-of-bounds read of a defvar
array in a --no-bounds-checks build printed its garbage and exited 0; with an attributes #0 = { sanitize_address }
group named on every define, the same program reports global-buffer-overflow in flan.main. Globals are the exception
— the module pass redzones them whether or not any function is attributed — which is why the shape of a sanitized
build looked right long before it worked.
UndefinedBehaviorSanitizer has no equivalent lever. Its checks are not a pass: the C frontend emits branches to
__ubsan_handle_* inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and
(<< 1 32) is still unremarked under -fsanitize=undefined. Shift UB, alignment and the f32→i32 cast on NaN are
therefore a compiler feature if they are wanted — checks emitted from Emit behind the flag, the same shape the bounds
checks already have — and not a flag away. test_sanitize pins both halves with controls: one program that must report
and one that must not, so either fact changing is a test failure rather than a discovery.
-fno-sanitize=signed-integer-overflow is the only exclusion, because wrapping is what this language's arithmetic
means and without it every program trips on its first +.
The flag deliberately does not force -O0, unlike --debug, whose reason (mem2reg deletes the alloca a
llvm.dbg.declare describes) does not apply. The optimiser is half of what is being measured, and bounds.flan proves
it: with checks off, its read past the end of a string constant is reported at -O0 and silent at -O2, because an
out-of-bounds inbounds getelementptr into a constant is poison and LLVM folds the load away. The program then prints a
wrong answer instead of touching memory. Same family as (<< 1 32) compiling to a bare retq.
What ASan covers of the bounds checks' job, since --sanitize --no-bounds-checks is the run that asks. Three of
bounds.flan's six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is
layout rather than anything about the access: ASan lays a global out as {data, redzone}, so reading before one lands
in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is.
Measured both ways — silent in bounds.flan, reported as soon as another defvar is declared in front of arr. A
reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees
out-of-object access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical
bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a
second net, not a replacement.
The sweep lives on its own dune alias rather than on dune test: a sanitized program links to a statically linked 1.8MB
binary, and twenty-eight of them twice over is minutes against the suite's seconds.
Valgrind, and the two questions a sanitizer cannot be asked
dune build --root . @valgrind runs the headless corpus under memcheck: forty-seven programs checked, twelve of them a
second time with --no-bounds-checks, in 88 seconds against @sanitize's nine minutes — and that figure includes
the compiles, because the programs are built once here rather than twice. Memcheck is 20–50x on execution and the
corpus is small; the sanitized build's static link was always the expensive half.
It needs nothing from Emit, and that is the entire reason it was reachable. The sanitize_address attribute
story above is a story about an LLVM pass that only instruments what the C frontend marked. Memcheck instruments the
binary: it never sees IR, never sees an attribute, and cannot tell Emit's output from clang's. Hand-written IR,
the runtime's C and libc arrive on the same footing. This is also why MSan was ruled out and memcheck was not —
MSan needs every dependency instrumented and raylib settles it, and memcheck needs none.
The two tools are complementary and neither is a superset. Measured on bounds.flan's six deliberate
out-of-bounds cases: ASan catches three, memcheck catches zero. Every one of them is a global or a stack array,
and memcheck's addressability checking covers heap blocks only — it has no redzone concept for anything else. What
memcheck has instead is definedness, per byte, which ASan does not have at all. Believe neither sweep alone.
The control that justifies the sweep. Index 3 of a Vec with len 2 and cap 4 is inside the allocation — every
addressability check in existence says it is fine, ASan among them — and was never written. Memcheck reports it, and
--track-origins=yes names flan_vec_push's aligned_alloc as where the undefined bytes came from. That is
NEXT.md's "ASan does not see uninitialised reads" turned into a test. test_valgrind.ml also asserts a heap overrun
that must report, and a Map key with two seven-byte holes that must not — the last being direct evidence for the
emitted per-key hash and equality pair walking fields rather than bytes, where maps.flan could only show the
consequence.
What --no-bounds-checks actually removes, which is less than its name suggests. check_at and check_slice in
emit.ml are behind the flag. A Vec's and a Map's bounds checks are not: they live inside flan_vec_at and the
map probe in flan_rt.c, are ordinary C, and run in every build. So the flag lowers the guard on fixed arrays and
slices only, and the single way to reach unguarded heap storage from Flan is a slice taken over a Vec — which is
what both positive controls do. This retro-explains why unchecked_controls in test_sanitize.ml only ever found
anything through fixed arrays.
What a clean run does not prove
The sweep is clean. Enumerated, in the spirit of "three of six is a ceiling, not a measurement of the risk":
- Globals and stack are outside it. Measured: 0 of 6 against ASan's 3 of 6.
- Arena storage reused after
free-allis not re-poisoned. Measured, and it is the sharpest hole. Round one writes four elements;free-allresets the offset and keeps the pages; round two allocates the same bytes back and reads one it never wrote — and prints round one's44, with memcheck silent. Nothing told memcheck the storage died, because frommalloc's point of view it did not: an arena is onemalloc(cap)andfree-allis an integer going to zero inside it. So the uninitialised-read coverage the control demonstrates holds for the heap allocator and not for the per-frame pattern the arena exists for. Closing it meansVALGRIND_MAKE_MEM_UNDEFINEDinflan_arena_proc, which is aruntime/change. - Interior overruns inside a single allocation are invisible by construction. The arena's alignment padding and
the gap between its offset and its capacity are one block to memcheck, so a read across a sub-object boundary
crosses nothing. The same holds for the
Map, whosedatais one allocation laid outkeys | values | hashes | scratch: a probe walking off the end of the hashes array into the values region is not an error memcheck can see. "Probe overrun at high load" is therefore not clean — it is not observable by this tool. - The union aliasing case is not exercised.
unions.flanreassigns a payload across cases and copies a union through a struct field, butmatchis tag-dispatched and the checker enforces it, so reading case A's bytes after writing case B cannot be written in the language. The sweep says nothing about thatgetelementptrbecause no program can reach it. - raylib and the windowed examples are excluded, so nothing is claimed about them. Under memcheck this matters more than under ASan, not less: memcheck reports on uninstrumented code too, so including them would bury the signal rather than lose it.
- Both positive controls had to be synthesized. No program in the corpus reaches a state where an uninitialised read is observable. That is what a corpus passing its own acceptance table should look like, but it means the sweep's value is as a regression net from here on, not as evidence that the current runtime was audited and found sound.
test/valgrind.supp exists and contains no suppressions, which is a finding rather than an oversight: the sweep
was run with --gen-suppressions=all before the file existed and memcheck produced nothing to suppress — no false
positives from the hand-written IR, the arena or zeroed, and no true ones either. The file is the four expected
complaints with the reason each failed to appear, and the rule for adding to it: paste valgrind's own generated text,
and write above it why the report is not a bug.
One thing the sweep found that is not a memory defect: slurp.flan is not idempotent. Its last section expects a
missing file, and its handler barfs that file into existence and invokes retry; run twice, the second run finds
the file already there and prints a handler count of 0 where the first printed 1. Running each program plain and then
under memcheck is exactly two runs, so this presented as "diverges under memcheck" and was nothing of the kind —
reproduced with no valgrind anywhere near it. test_acceptance.ml already cleared the same two filenames for the same
reason; test_valgrind.ml now does too.
Why there is no interpreter
Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.
It stopped being an absence when macros landed. A macro has to run at compile time and there is nothing to interpret
it with, so the compiler compiles it into a shared object and dlopens it into its own process — see "Macros: the
compiler dlopens the program". The decision's cost and its mechanism are the same thing.
The layout, which is the whole backend design
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
f32 f64 float double
bool i1
an enum i32
[T] and string { ptr, i64 } ptr+len, non-owning
[n T] [n x T] inline, a value
(Ptr T) ptr opaque pointers
(Option T) { i8, T } tag 0 None, 1 Some
a struct a literal struct, declaration order
() and Never {}
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:
- Every slot is an
alloca. Reading a local is aload, assigning is astore, and astoreof an aggregate is the copyspec-memory.mdrequires.addrof a local is then just the alloca, andmem2regremoves the ones nobody addressed.test/programs/values.flanpins this down. - A place is a pointer, a value is a load from it.
(set (.pos c) …)through a(Ptr Cursor)becomes agetelementptron the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
Non-local exit is lowered explicitly: return, some and a failed bounds check are branches, never platform unwinding,
so wasm32 needs no exception proposal.
The reload primitive — dev loop steps 1 and 2, measured
llc → ld -shared → dlopen → call, with no protocol and no daemon. dune test runs it: one function is recompiled
into its own object and called inside a process that is already running, twice, with a changed body the second time.
| Step | Cost |
|---|---|
Emit.redefinition |
below the timer (<0.1ms) |
llc -O2 -filetype=obj |
15–17ms |
ld -shared |
3ms |
dlopen + dlsym |
0.04ms |
~19ms end to end, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
number from this codebase. For contrast, clang -shared on the same IR is 50ms — the driver is again most of the cost,
which is why the dev path skips it. llc and clang are both 20.1.8 here; check that before trusting the .ll, since
the driver absorbs IR the bare tools reject.
ld -shared rather than clang -shared for a second reason: a shared object is allowed undefined symbols, and that
is the mechanism. What the new module does not define is the whole design:
- a global is
external. This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand'sgridwould reset on every reload, and "edit the code, keep the sand" is the thesis. - every other function is a
declare, so a redefinedsettlecalls the host'smove-grainrather than freezing a private copy of it. - no
main. This module is loaded, not started.
Its string constants still come along; omitting them is an undefined @.str.N at link time, and it is easy to miss
because a one-function module usually has none. Emit.signature is now the single place a function's LLVM signature is
spelled, because a define here and a declare there drift the moment one of them grows a case for () or for a
slice parameter.
flan_dev.c is compiled into every build, not only a dev one. Nothing in a release build calls into it — the
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
agent package's C refers to it, and a package's C sources are collected whatever main does. Leaving it out of release
builds made flan build sand.flan fail at the link with an undefined flan_dev_result_read, which reads as a compiler
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; -rdynamic and
the cells are still what --dev means. test_agent.ml links the agent program both ways for this reason.
-rdynamic is load-bearing. A normal executable exports nothing: nm -D calc-me | grep 'flan\.' is empty, so a
loaded module's declares would have nothing to bind to. The test passes it through lflags, which keeps it a property
of the dev build rather than of every build. dlsym on "flan.bump" works — a dot is legal in an ELF symbol.
Two things about the test are deliberate and are what make it prove anything: both loads happen in one process,
since two runs would pass while saying nothing about an in-process swap; and the versions are two paths, since
dlopen caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
helper is (* x 2) in one fixture and (* x 3) in the other: the second body is dead text, since the module declares
helper rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
String constants are emitted private unnamed_addr, so the module's own @.str.N cannot be interposed by the host's —
worth knowing, because with external linkage a redefined function would silently print the old text and nothing would
fail at link time. The fixtures each print a literal so that path is actually exercised.
Cells — how a call site follows a redefinition
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so a dev build routes every Flan-to-Flan call through a cell: a mutable global holding the address of the function that is current.
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
%p = load ptr, ptr @"flan.cell.bump" ; every call site
%r = call i64 %p()
Redefinition is then one store. A redefinition module declares the cells external, exactly like the globals, and
exposes flan_reload_install() that stores its own body into its own cell — cost below a microsecond, which is what
makes a frame-boundary swap a non-event.
The cell load is emitted after the arguments, so a redefinition landing between two calls cannot land in the middle of one.
Four things about this that are not free choices:
flan_reload_installis a named function and not an ELF constructor. A constructor runs duringdlopen, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose.- A redefinition's own body is
hidden. Default visibility in a shared object is interposable, and that applies to taking the address too: plain@"flan.bump"inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent. - This also fixes the self-call edge, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not.
-rdynamicis what exports the cells, so it and cells are one flag:Build.opts.dev,flan build --dev. This is the first timeoptsmeans something semantic rather than an optimisation level.
LLVM cannot fold the indirection away — the cell is an external mutable global — and a --dev build of calc-me keeps 46
indirect calls at -O2. The acceptance table now runs values, machine and sand-headless as dev builds as well;
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
Names that did not exist when the process started
Editing a defvar or a defn is a symbol the host exports. Adding one is not: there is no symbol to bind to and ELF
cannot grow one. Those go through runtime/flan_dev.c, which is two lookups and nothing else:
void **flan_dev_cell(const char *name); /* a new function's cell */
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
new global's declared initial value travels with it, as a constant the runtime copies on the allocation and ignores
on every call after: calloc alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
common case pays nothing for the general one.
The unit is a list of top-level forms, not one function — Emit.redefinition ~fns. C-c C-c passes one name, C-c C-k passes a file's worth, one code path either way. It has to be: v3 of the fixture adds extra and uses it from a
redefined bump, and splitting that into two loads would leave a module referring to storage that does not exist yet.
Four rules, each of which is a silent failure if broken:
- Every lookup resolves before any body is published. Publish first and a caller reaches a function whose slots are
still null. Not race-testable, so it is asserted on the emitted
flan_reload_install. flan_dev_globalrefuses a size change. The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.- Nothing that published anything is ever
dlclosed. A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being pointed into, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer, or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the handle value rather than the mapping:dlopenrefcounts by path, so re-sending the same bad file raised a count nothing could lower, and the one reference that could was dropped on the floor. - The registry never moves. A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable.
The test that separates this from a plausible wrong version is v4, which redefines added — a name v3 introduced at
run time. v3's bump is already installed and is not rebuilt, so it picks v4 up only if its call goes through a cell
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
pass and the transcript would read 246 instead of 432.
Sizes are spelled LLVM's way — ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64) — rather than by a layout
calculator in OCaml that would have to agree with LLVM's on every target.
The agent — dev loop step 3
vendor/agent/ is a package like any other: agent.flan declares three calls, flan_agent.c implements them, link
asks for -lpthread.
(agent/start path) listen on a unix socket; once, at startup
(agent/poll) install whatever has arrived; returns how many
(agent/wait ms) the same, but waits for something first
The split between them is the design. dlopen relocates a module and takes the loader lock — milliseconds, unbounded —
so it happens on the listener thread. flan_reload_install is one store per function and must not land while a
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
loader.
A full ring is refused, at the sender, before the dlopen. Of the three honest answers this is the only one that
reaches the person who asked: dropping loses a reload the sender was told was ok, which is the same lie more quietly,
and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also
stop answering status and abort, leaving the dev loop with no way to reach a program that had stopped listening to
it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away,
since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says
what to do about it — call agent/poll.
The way out of the break loop is _exit, not exit. exit runs the atexit chain and the ELF destructors, which
want the loader lock the listener thread may be holding inside dlopen; a program asked to abort would hang instead of
dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134
stays because that is what a trap exits with.
wait exists for tests. A test that races the frame rate fails on a loaded machine, so test/programs/agent.flan waits
for the reload instead of sleeping past it. It takes two reloads, which is the daemon's actual loop: the first
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
Two details found by running it:
- stdout is line buffered, set in
flan_rt_init. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found. - The reply goes out before the module is queued. The other way round, the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection reset rather than as
ok. okmeans queued, not installed. The sender does not get to know when the swap happened; only the program knows when it is between frames.
sand.flan calls agent/poll at the top of its loop, which is what step 3 was for. Verified: with sand running under
Xvfb, flan reload sand-probe.flan game-draw and one line on the socket, and 455 consecutive frames drew from a body
that did not exist when the process started. Building without --dev is fine — there are no cells, so a module is
refused on the listener thread and the loop never notices.
flan reload <file.flan> <fn>... [-o out.so] [--new name,...] builds one module the way the daemon will. --new is the
names the host was not built with; it is the one thing the command cannot work out for itself, and it is exactly what
the session will track automatically.
The session
lib/session.ml is the program as a live thing: the declarations the running process was built from, plus every change
accepted since.
Transactionality came for free and needed no machinery. Check.program builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the llc that
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
Two things the session knows that no single evaluation could:
- Which names the running process was built with. It comes from the checked program, not from any accumulated AST,
because
Check.programprepends the prelude and no AST contains it. Derive it from declarations andprintlnreads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. - What that process's memory looks like. Three changes are refused with a reason rather than loaded:
| Change | What it would have broken |
|---|---|
| a function's signature | a cell is a bare ptr; every call site compiled before the change still passes the old arguments through it — and this is now a stopgap, see below |
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
| a struct's fields | the values the process is holding have the old layout |
a defconst's value, when the checker consumed it |
it is in the shape of the program — (defconst rows (/ h c)) decides grid's type before anything else resolves — so no store can reach it |
a defenum member |
:space is erased to an i32 literal in the caller, so it is folded there too |
The signature row is the one the plan has moved past. plan.org now says a signature-changing redefinition should
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
existing callers and stored Fn values keep the old version and stay safe, and the session warns at every tracked
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
should not be read as the final answer.
A defvar's initial value is deliberately not in that table. Its storage holds live state the program moved past
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same Tast.global
record as a defconst, opposite answers, told apart by gconst. The enum comparison runs over declarations rather than
the checked program, because Tast.program carries no enums at all — they are erased to i32 in the checker, which is
the same fact that makes them unreloadable.
Note what the checker catches on its own: change helper's parameter type and the caller fails to type check first,
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
uses, which is exactly where the silent version lives. The fixtures carry an unused defvar and a C-called defn for
that reason.
A defconst the checker never consumed is a different matter and can be changed: it is only ever bytes in memory. A
dev build emits every defconst as a mutable global rather than a constant — so LLVM cannot fold a read of it and a
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
is how sand's colors gets tuned live while rows stays refused. Release builds emit constant and get all the
folding back; Tast.global.gfolded is what tells the two apart, because nothing downstream of the checker could.
A form typed into a file that is imported as a package is qualified the way the import qualified it. poll in
vendor/agent/agent.flan becomes agent/poll, and its call to poll-raw becomes agent/poll-raw — through Load's
own qualify_decl, so the rule cannot drift from the one used at import time. Without this the form spliced as a
brand-new unrelated name: the evaluation answered ok, and the running program went on calling the sim/settle it
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
did nothing.
It is derived from the file's path and not sent by the editor, which is where this departs from CIDER's ns key: a
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
either.
The accumulated list is the post-Load one, so an evaluated (import …) is spliced as its expansion. Otherwise
re-evaluating a file that imports something appends a second import, Load expands it again, and the duplicate-name
pass rejects it. C-c C-k on sand.flan's own text is the test.
flan reload <program.flan> <forms.flan> is that path from the command line: a session over the program the process was
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form game-draw
and 910 consecutive frames drew it.
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
has and what its memory looks like describes a binary that is not running. And Session.eval's origin defaults to
<eval>, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
the real buffer path, which is the same key CIDER's eval carries.
The daemon — flan dev
flan dev <program.flan> holds one Session, builds the program, launches it, and listens on .flan-dev.sock beside
the source. What it adds over flan reload is that the session persists — a defvar added by one evaluation is part
of what the next one is checked against — and that it owns the build, which is what makes its layout rules describe
the process that is actually running rather than a guess about it.
The protocol is s-expressions, not bencode. nREPL was the plan and the argument for it evaporated once the client
became ours too: there is no CIDER to be compatible with, eval is string-in/string-out with no slot for which form,
from which file, and Emacs already has read and prin1. So it is one sexp per message — no parsing code on the
editor side, and on this side the parser is the language's own reader, where :op is already a keyword and a payload of
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
newlines. An nREPL front end can sit on the same Session later; it should not have gated the editor.
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
(:op "close")
defs is its own op rather than more fields on describe, because describe is what an editor polls — it is how the
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
It is asked once on connect and again after each accepted install. Four strings an editor reads with read and nothing
else: eldoc, completion and find-definition want the same three facts about a name. loc is empty where there is none
to give, because only Tast.fn carries one — an editor must refuse rather than go looking for the definition itself,
which in a program of several files finds the wrong one. Parameter names are not in the Tast, so a signature is step [i64 f32] i64: types only.
The daemon makes its own source path absolute before building, because every location it reports derives from it. flan dev src/game.flan run from a project root otherwise answered src/game.flan:12:7, which an editor can only resolve by
guessing which directory it was relative to.
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
— is accepted and answered with :note "nothing to install" rather than by shipping an empty module. Building one
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
:file is not decoration: Session.eval's origin defaults to <eval>, so without it every error an editor shows
points into a file that does not exist.
Two things the daemon must not paper over, both of which would look like a successful evaluation:
- The agent socket is chosen by the daemon, not by the program. A program's source has to name some path — sand.flan
says
/tmp/flan-sand.sock— and the daemon overrides it throughFLAN_AGENT_SOCKETbefore spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it. - Delivery is checked.
agent/startreturning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is notokbecomes an error the editor sees.
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving.
One process — flan dev is the program, and the compiler is a thread in it
flan dev builds one binary that is the compiled Flan program and the whole OCaml compiler, and execs it.
There is no child: /proc/<pid>/exe on the pid an editor launched points at the program the daemon built, and
test/test_dev.ml checks exactly that rather than taking the claim on trust.
Who owns what:
| thread | what runs there |
|---|---|
| main | the program. C's main holds it — macOS needs a window on the main thread |
| a pthread | caml_startup, then Flan.Dev.merged_setup and merged_serve: the compiler and the editor socket |
| a pthread | flan_agent.c's accept loop, as it always was |
The program keeps main() and the compiler comes up beside it, rather than the other way round, because that is the
shape DISCUSS.md §11 landed on: the agent is already a server inside the program, so the compiler moves into the
program's process. It is SLIME's model — you start the image, it serves, the editor connects.
--two-process is the escape hatch, for a machine where the compiler object cannot be built (no ocamlfind, no
flan.cmxa beside the binary). It has its own test and it stays.
The editor socket and its wire protocol did not move. Emacs cannot tell the difference, which is what made the merge testable: the whole existing suite is the check.
Two rules hold in one address space and did not before:
- The game thread must never call into OCaml. A native thread has no safe points, so the collector can never stop it — which is precisely why a frame is never paused, and precisely what one convenient direct call would undo. Requests reach the compiler by being left somewhere and picked up, never by a call out of the frame loop.
- No OCaml
valuegoes into Flan storage withoutcaml_register_global_root. That is how "the GC does not touch the arenas" stops being true.
A Flan main does not return — Emit ends it with flan_exit and an unreachable — so in one process that call would
take the compiler down with a program that merely finished. flan_rt.c has a hook, null in every other build, that
the merged entry point uses to flush, close stdout and park. The compiler learns the program is done the way the daemon
did: the pipe reads EOF.
The internal socket is gone — the transport, measured
The merge deliberately deleted nothing, so that it could be tested against the shape it replaced. This is the first thing to go: the unix socket between the compiler and the program.
It was a connect, a write and a read that looped back into the same address space. vendor/agent/flan_agent.c's
verb table is now handle_line(line, sink *), where a sink is either an fd or a growing buffer, and there are two
callers: serve(), which reads a line off a connection as before, and flan_agent_request(), which the compiler
thread calls directly. One verb table, not two — every answer is assembled from several pieces (a header, a name, a
newline), so the seam had to be the writing rather than the handlers, or they would have been duplicated and would
drift. Dev.deliver, Dev.result and Dev.ask were three copies of the same socket dance; they are now three lines
over one Dev.request.
Which path is taken is decided by the linker, not by a flag. flan_agent_request is declared weak in
lib/dynload_stubs.c, so it is null in every binary that links no flan_agent.o — the flan launcher, flan reload,
the two-process daemon, every test — and Agent.request answers None there, which is what makes Dev fall back to
the socket. A flag could disagree with reality; this cannot.
Three things had to be right:
- The ring still has one producer at a time.
queue_roomchecks for space separately frompublish's store, which was safe only because the accept loop was the sole producer and was single-threaded. The compiler thread can now ask while the accept loop is serving, sohandle_lineruns under a mutex. It is held across thedlopen, which is what the accept loop already did to itself by serving connections inline. - A delivery is still only queued. The direct call publishes to the ring exactly as the listener did; the install is one store per function on the game thread at a frame boundary. Installing on the spot would be a frame running half in the old code and half in the new, and it would be an easy thing to do by accident here.
- The OCaml runtime system is released across the call. A delivery is a
dlopen— milliseconds of relocation and the loader lock — and holding OCaml's lock through it stalls every other OCaml thread. DISCUSS.md §14's third cost, in the one place this change creates it.
What it is worth, measured. This machine, warm caches, a one-defn redefinition driven over the editor socket,
median of 12; the four columns taken back to back in one sitting, because the run-to-run drift on this machine is
larger than what is being measured:
| merged, before | merged, after | --two-process, before |
--two-process, after |
|
|---|---|---|---|---|
| redefinition, end to end | 22.1ms | 21.2ms | 20.8ms | 22.1ms |
...of which the build (:ms) |
19.5ms | 18.8ms | 18.4ms | 19.5ms |
a break round trip |
0.061ms | 0.020ms | 0.060ms | 0.064ms |
The break row is the editor socket plus one question to the agent, so the ~41µs it lost is the whole of the
internal socket. Everything else is noise: the redefinition column moves by less than its own spread and moves in both
directions.
The transport was about 40µs of a 21ms redefinition, and removing it does not move that number. That is the finding, and it is worth more than a speedup would have been. The sharpest evidence for it is that the merged and two-process columns change places between runs — two-process ahead by 1.3ms in the before pair, merged ahead by 0.9ms in the after pair — which is what a difference made of noise looks like. The merge's prize was never latency. It is that the compiler and the program share an address space, which is what makes the items below deletable at all and what unblocks reading the stopped frame's memory directly. Anyone reaching for an in-process JIT on the strength of "transport is slow" should read the build row first: code generation is 19 of the 21 milliseconds, and the socket was 0.2% of it.
The test that pins it is a deletion, because a reply cannot say which way it came: test_dev.ml unlinks the agent's
socket file once the merged program has bound it, and then runs every evaluation in the file. Unlinking a bound unix
socket does not disturb the listener, it makes new connects fail — so if the deliveries still install, nothing
connected. The agent still binds it, for --two-process and for a person at a raw socket.
The 4K result cap is not a transport buffer, and it stays
Next on the list, and the answer is no — with half of it deleted anyway, which is the useful part.
RESULT_MAX reads like a wire size: 4096 bytes, a cap on a value the compiler reads back after C-x C-e. It was
written down twice, once in runtime/flan_dev.c and once in vendor/agent/flan_agent.c, with a run-time check
that the two had not drifted. That second copy was transport — a buffer sized to be sent through a socket — and it
is gone: the agent asks flan_dev_result_cap() and allocates, so the bound is one file's decision now and the
drift check has nothing left to check.
The bound itself cannot go, and the reason is the rule the merge was built around. result is the buffer the
game thread writes into, from a render thunk at a frame boundary. A growable one means the frame thread calling
realloc — an allocation in the one place this design exists to keep allocation out of. And it would break the
seqlock, which the premise for this work correctly says must stay: a seqlock is a protocol about torn contents, and
it assumes the address it memcpys from neither moves nor goes away underneath the reader. Growing on the writer's
side is a use-after-free the counter cannot see.
So it is a render budget, not a wire size, and it was only ever mistaken for one because the agent had a copy of it. Removing the bound is a redesign of the read — probe the length, allocate, re-read, validate the generation, retry — and it belongs with moving the read to a frame boundary, which is the seqlock's own decision and its own lane.
The break snapshot is not marshalling either, and all of it stays
Third on the list, and the answer is no, with nothing left over. "The compiler can read the stopped frame's memory directly, so copying it is now ceremony" is the right instinct and the wrong diagnosis: the snapshot was never about two address spaces. It is about two threads, and there are still two.
A stopped program is not holding still. The break loop polls, flan_agent_poll runs whatever the compiler delivered,
and a C-x C-e thunk is arbitrary Flan — it pushes and pops the one global restart list and the shadow stack while it
runs. The compiler is a thread beside it either way. So every copy in snap_push has the same justification it had
before:
- Restart names are copied because serving them off the live list hands the reader a pointer into a frame the break loop's own poll may already have popped. A pointer is meaningful to the compiler now; the frame it points into is no more alive for that.
- Frame names and locations are copied from the held-still stack for the same reason, and the fingerprints have a sharper one already written down: the module a frame's description lives in can be unloaded once a replacement is installed, and the comparison happens after that.
- The generation stamp (
snap_gen,chosen_gen) is about nested breaks, not about processes. A thunk this loop runs can error, push a break of its own, and reachchosen_readyfirst — claiming an index someone chose from the outer list. Depth cannot tell those apart, because an outer break resuming and a new one starting reuse the number. A generation can. One process changes nothing about that.
The fixed caps go with it: the snapshot is taken on the game thread, so it cannot allocate, which is why
SNAP_MAX, FRAME_MAX and FRAME_TEXT are literals and why truncation is reported rather than avoided.
What the merge does unlock here is one thing and it is the next item: flan_agent_frame_slot already hands back the
address of a slot, and in one process the compiler could read the value at that address instead of compiling a render
thunk to print it. That is the render-thunk-per-inspection redesign — a different mechanism rather than a deletion, and
what makes "the inspector can retain a value" reachable. It is deliberately not done here.
So of the three things the merge was expected to make deletable, one was. The socket was transport and is gone; the result cap and the snapshot are both concurrency, and they were only ever mistaken for transport because the socket was the thing in front of them.
The Emacs client
emacs/flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is most of the work: Flan is
s-expressions, so sexp motion, paren matching, beginning-of-defun and indentation are already right. What it adds is
Flan's own bracket syntax ([ and { are brackets, not symbol characters — every binding list and every type is
written with them), the characters a Flan name may contain (-, ?, /, .), and its keywords.
emacs/flan-dev.el is the client. There is no parser in it, which is the point of the protocol choice: prin1 writes a
request and read reads a reply.
C-c C-c |
the top-level form at point, recompiled and installed |
C-c C-k |
the whole buffer, as one module |
C-x C-e |
the expression before point, evaluated in the running program |
C-c C-z / C-c C-q |
connect (finds .flan-dev.sock upward) / disconnect |
C-c C-o |
the running program's own output, in *flan-output* |
C-c C-r |
a prompt on the running program (*flan-repl*) |
C-c C-b |
what a stopped program is offering, and which to take |
C-c C-d |
what the running program currently defines |
C-c C-a |
the code a name compiled to — amd64, or C-u for the LLVM IR |
M-. / M-, |
where a name is written, through an xref backend |
eldoc, completion-at-point and M-. all read one cached defs reply rather than asking per keystroke: eldoc fires on
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a defn just
installed completes at once.
The modeline says whether there is a program on the other end, in three states. lost is a daemon that has gone away,
which is ordinary rather than an error — flan dev ends when its program does — so the next request reconnects on the
socket it was on. Strictly before a send, never after one: a connection that died mid-request may have died after the
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
C-c C-k sends one module rather than a form at a time on purpose: a defvar and the function that uses it have to
arrive in the same load, or the first refers to storage that does not exist yet.
Framing is in bytes and Emacs counts characters. Every length goes through string-bytes and the process is binary,
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
has of them — a bug that would look like a corrupt protocol and appear only for some users. test/test_emacs.ml drives
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
mistake in the framing, in beginning-of-defun over Flan's syntax table, or in the reply reader passes test_dev.ml
and fails here.
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
next time that buffer's evaluation is accepted. Two things had to be right first. The column in a :loc is a byte
offset, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
forward-char with it put the marker as many columns right as the line had non-ASCII characters before it. And the
daemon numbers lines from the start of what it was sent, so C-c C-c on a defn halfway down a buffer answered line 1
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
reader skips, so the reply's line numbers are the buffer's own.
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
success is indistinguishable from silent failure, and beginning-of-defun may well have found a different form from the
one point looked like it was in.
Live disassembly — done. C-c C-a on a name writes the amd64 the running program's copy of it was assembled to
into *flan-disassembly*; C-u C-c C-a writes the LLVM IR that body was built from. (:op "disassemble" :name … :form "asm"|"ir") is the op.
What makes it possible is that the daemon owns the build: it compiled every module it sent, so objdump -d --disassemble=flan.<name> on the right object is the disassembly and the retained .ll is the IR. Build.shared
deletes its own .ll and Build.executable leaves the host's under a name that says nothing about which module it was,
so the daemon now writes its own copy beside each .so and keeps the host's as host.ll — ten reloads in, nothing else
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
bookkeeping.
What it will not claim is that the code shown is installed, and this is the interesting half. The agent's socket
takes a module path and five verbs; none of them reports an address, flan_dev_cell lives in the program's address
space, and C-x C-e renders a pointer as <ptr> on purpose — so nothing the daemon can ask would tell it what a cell
holds. The reply carries :basis saying which of three things is true, and the buffer prints it above the first
instruction:
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is certain;
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
- a module was delivered and the program is stopped — which says nothing either way about whether it installed, since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further installs until it resumes.
From SBCL: offsets from the function's own start rather than addresses into an object, and L0: labels on branch
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
the debug build lands, that is the line to delete.
Transient error overlays — done. The diagnostic ghost text is feedback about the evaluation that just failed, not an
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
pre-command-hook and not post-command-hook, which fires at the end of the failing command and would clear the
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
in all of them. execute-kbd-macro runs no pre-command-hook under --batch, so the test drives run-hooks — the
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
is Emacs' contract and a test claiming to check it would be checking nothing.
The program's stdout is a pipe into the daemon, and whatever it printed since the last reply rides along with the
next one into *flan-output*. Having it arrive with a reply rather than by a separate request is the point: the
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
accept loop's select, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
grow the daemon without limit.
C-x C-e — evaluating an expression
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
into, so the expression is wrapped in a function with nowhere to be called from; the module says run this once by
exporting flan_reload_call, and the agent calls it after the install — on the game thread, at a frame boundary, so an
expression reading the program's state sees a point the program agrees is consistent.
Nothing is marshalled back, because nothing could be. A Flan value carries no header, so no code at run time can say
what it is. The compiler knows the type and renders it there, in the thunk, into flan_dev_result. That is the layout
decision's bill, and it is why the printer set is small rather than universal.
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
dev-only feature must not put a branch in it — so flan_rt.c is untouched and the value is read back over the agent's
socket. The read is safe without a handshake because the counter is a seqlock, and it had to be made into a real
one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived
and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to
be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the
bare pointer was the bug rather than the ordering. flan_dev_result_read copies into the caller's buffer and checks
the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader
that loses the race reports the last complete generation and no bytes, so a daemon polling for a new value keeps
polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's
"has it moved" still means what it meant. Marking the counter odd needs a release fence and not a release store — a
release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which
is the original bug with more ceremony. And the odd mark is set rather than incremented, because a begin with no
end is reachable: the thunk calls begin before it evaluates anything, so an expression that signals is stopped
inside that window, and a restart taken from the break transfers past the thunk and end never runs. Incrementing
would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit
means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has
reached a frame boundary.
This renderer is most of println, which is worth knowing before anyone schedules it. plan.org describes a
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
structs, fixed arrays and options structurally, prints a Ptr as its address rather than following it, and bounds depth
and length. That is a description of what Session already does for C-x C-e — same walk, same refusals, same three
bounds — aimed at flan_dev_emit and the wire instead of at stdout. What println needs on top is a stdout sink, a
builtin that takes its printer from the argument's type, and the any/Error dynamic cases, which have no compile-time
type to walk. Not the walk itself.
The renderer is a compile-time walk over the type, emitting a piece at a time through flan_dev_emit. Piecewise
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
allocator the language does not have.
big 18446744073709551615
col :blue
(.pos b) (V {:x 1.5 :y 0})
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3) [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
Details that are decisions rather than formatting:
u64renders in C, with%llu. The language's owni64->bytesis signed, so it used to refuse rather than come back as-1— but refusing a whole struct because one field is au64is much worse, so the runtime got an entry point instead.- Strings are quoted and escaped, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is.
- An enum renders as
:name, recovered from the checker's table as a chain of comparisons, because members are erased toi32before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see. - A pointer is never followed —
<ptr>. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf. - Three separate bounds, easy to conflate.
depth(4) andspan(8) bound the walk, so[100 [100 u32]]does not become ten thousand render sites in one module. The output is bounded once in the runtime —emittruncates at 4K andendappends...— because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget. - A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element.
What still refuses by name: Map, Fn, a type variable.
A caveat inherited from the language, not introduced here: 3.0 renders as 3, indistinguishable from the integer.
flan run calc-me.flan "1.5 * 2.0" has always said 3.
An evaluation is not a declaration: the thunk is built against the program and never spliced into it, so describe
does not fill up with eval/N for every expression ever typed.
The module is unloaded afterwards, which is the one case where that is safe. The thunk is called directly by
flan_reload_call rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
into its text and the value it produced has been copied out. It declares that with @flan_reload_transient and the
agent dlcloses it. Measured: sixteen expression evaluations retain zero mappings, where each redefinition
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
and can never claim this.
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it.
The test that matters is the same expression twice: the fixture increments ticks every frame, so two evaluations must
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
The REPL buffer
flan-repl.el is a comint-mode buffer whose every line goes through the same eval-expr request C-x C-e uses. No
new protocol and no compiler support. Deriving from comint rather than hand-rolling a prompt is the same call as
deriving flan-mode from lisp-mode: history, the input ring and kill/yank already exist. There is no subprocess
behind it — the "process" is a stub comint needs in order to have a prompt at all.
Three things about it that are decisions:
- It is program-scoped. A name typed at the prompt resolves against the running program's top-level namespace, so in
sand you write
sim/settleand notsettle. A buffer visiting a package's own file gets the alias applied for it because the file says which package it belongs to; a prompt has no file and nothing to derive one from. - RET on a half-typed form opens a line instead of sending it. Balance is checked with the Flan syntax table, so a paren inside a string does not count.
- A value and the program's output are different things and arrive by different routes. The value is the result of
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
and goes to
*flan-output*. Showing them in one place would be convenient and wrong, so there is a test for the separation.
That test is what caught a real bug: the renderer's unit case emitted () without evaluating the expression, so
(println "x") — the most ordinary thing anyone types at a prompt — answered () while nothing happened. A ()
expression is almost always a call made for its effect, and is now evaluated and then reported.
Conditions — step 1: handler-bind and signal
spec-conditions.md §1 and §2, and nothing else yet. They are worth having on their own because neither alters
control flow: signal returns () whatever it finds, a handler that returns normally leaves the signalling
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
yet, and no signature changed.
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
(load-all))
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
function's own stack, and signal with an empty stack is a null check — which is what §2 asks for. Popping is by frame
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
step.
Three decisions worth keeping:
- A condition's type is a hash of its name, not an index. An index would shift the moment a struct were added, and every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
- The condition crosses as a pointer, because a handler runs while the signalling frame is still alive and there is
nothing to copy. What the clause binds is the condition itself, though — the pointer is a hidden parameter and the
name is a slot loaded from it, so a handler passing
cto something expecting the struct is not handed an address instead. - A clause is lifted into a function of its own. A handler runs from wherever the signal was, so it cannot be a branch in the function that wrote it.
- A pushed handler frame holds the clause's body address, not a cell. This is a deliberate divergence from
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
particular body. A handler frame is not a
Fnvalue — nothing in the language can name it — and it is live only for the duration of thehandler-bindbody, so a reload landing while it is on the stack finds the clause it pushed still valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the stack does not observe a redefinition of its own clause; the next entry to thehandler-bindpushes the new one. WhenFnvalues arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
Which gives the two refusals, both by the house rule rather than by accident:
- A handler cannot see the establishing function's locals. That is a closure with an explicit environment, so a reference to one is refused for that reason rather than reported as an unknown name. Globals and the condition are in scope, which is what the accumulation case needs.
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
outlive the function that established it, so this is spec-memory.md's case 2 — a non-escaping fn capturing by
value into a stack environment — and not the escaping closure that plan.org's open decision #5 defers until a concrete
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
biggest usability limit in conditions is not behind the thing that was just deferred.
returninside ahandler-bindbody is refused. The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape asdeferinside a block.
The break loop — conditions step 3
Where "a crash kills the program" stops being true. An unhandled error runs a hook instead of rt_die(), on the
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
flan: unhandled Missing — stopped, not dead.
0. restart: retry
1. restart: use-placeholder
Four decisions, each of which is the reason something is where it is:
- It is a hook, not a call. The loop lives in
vendor/agent/, which is an optional package;flan_rt.cis the release runtime and must not depend on something a program may never import. A program with no agent leaves the hook null and dies exactly as it did before. - The hook resumes by writing a restart into the transfer channel — the same channel an
invoke-restartwrites, reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same act, lowered once. Nothing about §6 needed changing to support it. - The break loop is the poll loop, run from the error instead of from the frame boundary. That is not a
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
that did not drain that queue would hang
C-x C-eexactly when it is most wanted. - Installing while stopped is allowed, which contradicts a rule stated above and should. "A redefined function must
not be swapped while it is on the stack" is about mid-frame consistency — half a frame of old code and half of new —
and there is no frame in progress here. The old body on the stack keeps running; a
retryrestart calls through the cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of stopping.
A restart frame carries its name now, beside the hash. Matching never needs it — that is what the hash is for — but
a break loop has to show someone their choices and nothing at run time can turn a hash back into a name. It is also
compute-restarts' data, whenever that arrives.
A choice is validated on the listener thread, against a stack the stopped game thread is holding still, and refused
there. Accepting it and discovering on the game thread that no frame offers it would answer ok for something that
cannot happen.
The socket verbs are restarts, restart-at <n> [name], restart <name> and abort, and all four are refused with
the reason when the program is not stopped — there is no restart stack to walk from a running one. restarts answers
<index> <+|-> <name> per line: the index is the identity, and the flag says whether a transfer to that frame has
anywhere to land. test/programs/break.flan errors three times — two restarts taken by name, then a shadowed pair
where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved
by name could pass.
The break loop in the editor — conditions step 3, the other half
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it follows from one fact: a program stops at a moment nobody asked about. Every other op in the protocol answers a question an editor chose to ask.
So the state is learned twice, deliberately:
- It rides on every reply, beside the program's output and for the same reason.
:stopped t :condition "Missing", or:stopped nil. The likeliest instant for a program to stop is the one just after an evaluation — a body that now errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding out after the echo area had said the evaluation was fine. - And a timer asks anyway, once a second, with
describe— the cheap op, which is also how the output pipe is drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies that never come says nothing. The timer never reconnects:flan-dev--requestreopens a socket a restarted daemon left behind, which is right for something a person did and wrong for a background poll, because it would quietly erase theloststate that exists to be seen. It also skips while another request is in flight —accept-process-outputruns timers, so a poll firing inside a read would eat the reply that read was waiting for.
Three ops, and the annotation owns :stopped, not the ops — one place in the daemon decides whether the program is
stopped, so the poll and the prompt cannot disagree.
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
:stopped t :condition "Missing")
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
(:op "abort") → (:status "ok" :note "the program is exiting; …")
:restarts is positional — innermost first, duplicates kept — because the position is what restart-at takes.
:unreachable names the positions that are on the list and cannot be chosen. restart-at's :name is optional and is
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
prompt cannot take a different restart than the one it showed.
break carries only the restart list, because it costs a second round trip to the program and is wanted only by
someone about to choose from it.
ok from restart means accepted, not resumed. The choice is validated on the program's listener thread against
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
lets the next poll settle it.
The modeline is a fourth state, flan:stopped(Missing), before live: a stopped program looks exactly like a running
one from anywhere else in Emacs. The prompt is a completing-read over the names with require-match, which is exactly
right for a closed set the program computed, and abort is the last entry on that same list rather than a second key —
it is the thing you pick when none of the restarts is the answer.
flan_agent_poll had to become re-entrant, and that is the one thing here that was a bug rather than an addition. A
C-x C-e thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
cached head and tail and wrote tail back at the end, so the outer call rewound the index over everything the
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
breaks rather than a stumble. It now claims each job by advancing tail before running it, and re-reads both indices
each time round. Still single-consumer: only the game thread writes tail, nesting included. test_dev.ml evaluates an
expression that errors and resumes it, which fails against the old shape.
The agent grew one verb, status, answered in both states — running or stopped <condition>. Everything else the
break loop offers is refused while running, and rightly; but the question an editor asks without already knowing had
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
test/programs/dev-break.flan stops on its first frame, so the daemon meets a program that is already stopped — the
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
round, by installing a step that errors into a loop that calls it, fixes it while stopped, and then resumes: C-x C-e
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
layout — a type's fields, with no program involved
(:op "layout" :type "sim/Cell") → (:status "ok" :type "sim/Cell"
:fields (("heat" "f32") ("next" "(Option sim/Cell)")))
→ (:status "error" :message "Missing is not a qualified name; …"
:candidates ("a/Missing" "b/Missing"))
The daemon can answer this with nothing running. A layout is a fact about the build, and the daemon owns the build —
Tast.structs is sitting in the session it compiled the process from. That is why the conditions buffer can name and
type a condition's fields while every one of their values stays refused: the shape is knowable and the contents are
not, and drawing them apart says more than drawing neither.
The type is a name, and the name is the qualified one. This was the open question — a class name is not an identity,
and two packages each declaring Missing would leave the daemon unable to pick. It turned out to need no new
machinery: Load.qualify_decl rewrites Defstruct (n, …) to Defstruct (alias/n, …) at import, so by the time
anything reaches Tast.structs the names are one flat namespace in which a collision cannot exist. The name is the
type id, with no table to keep in step across a reload, and the existing spelling of a type — Types.to_string —
already prints it.
And the break loop was already speaking it. Emit.struct_name_of takes Types.Named n — the qualified name — and
passes it to flan_error; the agent holds it in condition_name; break answers it as :condition. So the string
the conditions buffer already had in hand resolves as :type by construction, and test_dev.ml round-trips exactly
that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat
worth writing down — condition_name is a char[128], so the round trip holds for names up to 127 bytes and a longer
one is truncated and will not resolve.
A bare name is refused, not resolved, even when only one struct's last segment matches it. Resolving a unique
suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot
rely on. The refusal carries :candidates, so a person is one copy-paste from the answer and a client has its
completion list — the same shape as package_of refusing a directory imported under two aliases rather than picking
one. An enum is refused by kind (X is an enum, not a struct): its members are erased to i32 before Tast.program
exists, which is the same fact that makes a defenum unreloadable. A union is refused the same way and for its own
reason — it is declared, and union values are milestone 6. Both are Types.Named at a use site, so falling through
to "no struct is named X" would say a type does not exist about one that plainly does.
render.ml is not reused, and that is not a second walk. It walks a value and emits the code that prints it;
this describes a type and emits text. What is shared is the spelling: field types go through Types.to_string, which
is what defs spells a signature with, so (Option T), [T], [n T] and (Ptr T) read the same in a layout, in a
signature and in the source. A field that is itself a struct shows its qualified name — which is a :type this op
accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude
structs are answered like any other, because Render resolves against the same list and an editor that could see a
Split printed but not ask about it would be the two disagreeing.
On the Emacs side flan-cnr-layout makes the request and flan-cnr-show passes the result into
flan-cnr-state-from-reply, which stays a function from data to data so the fixture-driven tests keep working without
a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and
turning C-c C-b into an error would take away the restarts — the decision the buffer exists for — over a missing
annotation.
The shadow stack, and backtrace
plan.org's "Dev vs release builds" table has had Frames: shadow stack in the dev column since the project began and
nothing had ever built it. This is it. It is what (:op "backtrace") is made of, and it is dev-only, so a shipped game
pays nothing — the same bargain the indirection cells already make.
Chosen over the DWARF route deliberately, and the author's reason is the one that decides it: the more a break loop can show, the less often a real debugger is needed. DWARF buys frames in lldb; this buys them in the break loop, where someone already is when they want them.
A frame is four words on the calling function's own stack — the frame it displaced, a pointer to a static description
of the function, and two words reserved for locals. The description is per function and not per call, because nothing
about a function changes between two calls to it: the qualified name, file:line:col as Loc spells it, and how many
slots the frame has. So the name in a backtrace comes off the frame itself and needs no debug information, no symbol
table and no agreement with the optimiser about what a stack frame looks like. It is the same mechanism on wasm32,
which is the other reason it is not .eh_frame: plan.org lowers every non-local exit explicitly rather than through
platform unwinding, so there is no unwinder here to borrow.
The compiler emits the push and the pop inline rather than calling into the runtime. This is on every call in a dev build, and a call made to record a call would be most of what it costs.
The pop is at every ret, and the transfer path is the one that matters. There are five: an explicit return with
a value and without, the none arm of (some x), the tail of the body, and the landing block a transfer leaves
through. emit.ml funnels all five through one ret, because the way to get this wrong is to write the pop on the
normal path and not on the other one — and then every handled error leaves a dead frame behind, and the backtrace
after the fifth one is five frames of fiction. test_dev.ml takes five breaks and resumes all of them by condition
transfer before asking for a backtrace, and the answer has to be two frames. Same lesson with-allocator learned about
restoring at the pad.
A plain global, not a thread-local. It matches what the handler stack and the restart stack in flan_rt.c already
assume: one thread runs Flan, and the listener thread runs C and the loader and never enters a Flan body. If the
language grows threads this becomes thread-local and the compiler's two stores become TLS-relative, which is the whole
of the change.
The chain is snapshotted on the stopped thread, into the same snapshot the restarts are copied into, at the same
moment and for exactly the same reason: the break loop polls, a poll runs Flan, and a chain read by the listener
thread is a chain that can be popped underneath the reader. Names are copied as bytes rather than kept as pointers,
because a transient C-x C-e module does get dlclosed and its rodata with it. Frame addresses are kept beside the
text, because reading a frame's locals means going back to that frame and not to whatever is at index 2 by then.
(:op "backtrace") → (:status "ok" :frames (("fetch" "/game.flan:15:7" "program" 0)
("main" "/game.flan:27:7" "program" 0))
:more 0 :stopped t :condition "Missing")
Innermost first. :more is how many frames deep recursion left off the end — the innermost ones are what the question
is about. origin is "program" or "eval": a break inside a C-x C-e thunk has that thunk's frames on top of
the program's, and "where is my program" answered with eval/7 is true and not the question. The boundary is recorded
at the call, in flan_agent_poll, exactly where restart_floor is and for the same reason — except that its
out-of-a-thunk value is -1 rather than 0, because zero restarts on the stack is a real answer and zero frames
belonging to the program is not.
Refused while the program is running, like every other break verb. The chain is the game thread's and it is pushed and popped on every call; a walk of it from the daemon would have the shape of a backtrace and the contents of a race.
What it costs, measured rather than assumed. Two benchmarks, one compiler built per variant, every binary kept and then run alternately. The figure is the minimum of nine runs, because this machine is shared and a mean measures whatever else was running; the whole table reproduces to three digits on a second pass.
| Dev build, -O2 | no frames | frames | frames + slots |
|---|---|---|---|
| 2000 sweeps of a 100×100 grid — sand's inner loop, in miniature | 30.0 ms | 40.0 ms (+33%) | 48.3 ms (+61%) |
| fib(30) plus 20M calls in a loop — nothing but calls | 28.7 ms | 31.1 ms (+8%) | 35.1 ms (+22%) |
Per call that is about 0.1 ns for the frame and 0.3 ns for the frame and the slot table together; per sweep of
the grid, 5 µs and 10 µs, which is 0.03% and 0.06% of a 16.6 ms frame at 60fps. The dev loop's premise is that
redefinition does not stutter a running game, and a sixteenth of a percent of a frame does not. A dev build is already
deliberately slower than a release one — every call goes through a cell, every index is checked, no defconst folds —
and this joins that list rather than starting a new one.
The chain's shape was chosen by measurement and not before it. An array with a stack pointer — no alloca, no address escaping — was built and timed as the obvious alternative and is worse on both benchmarks: the frame record on the calling function's own stack is already hot, and an indexed store into a megabyte of BSS is not. It also has a fixed depth, which a chain of stack records does not.
Locals of a stopped frame
The half the shadow stack was built for. (:op "locals" :frame N) answers what a stopped frame's named locals hold.
(:op "locals" :frame 0) → (:status "ok" :frame "look"
:locals (("n" "i64" "3") ("label" "string" "\"hello\"")
("p" "Point" "(Point {:x 1.5 :y 2.5})")
("xs" "[3 i32]" "[ 10 20 30]") ("flag" "bool" "true"))
:refused (("after" "not bound yet at the point the program stopped")))
Nothing is copied out of the program, and nothing could be. A Flan value carries no header, so bytes read from
another process would be bytes with no meaning. What the daemon has instead is the type — Tast.fn.slots, from the
build it owns — and the name beside it in snames, which was already there for the DWARF work. So it compiles a thunk
that renders those types at those addresses, in the program, on the stopped thread, and reads the text back through
the same seqlocked result buffer C-x C-e uses. The only fact that comes from the running program is where the frame
is. That is render.ml's existing walk with its root changed: Render.render over (Deref (Ptr T) (flan/dev-slot f i)) instead of over an expression — the pointer-rooted render thunk NEXT.md said locals needed, and it turned out
to need one new arm in the whole backend (a pointer-to-pointer cast, which under opaque pointers emits nothing).
A slot's entry is its address, and null until it is bound. That is the whole of the liveness answer: there is no
analysis, no PC-to-scope map and no bitmap. The store that binds a slot stores the address, so a slot the program has
not reached yet reads as null and is refused by name. Without it, (let [after 99] …) sitting past an error would
render whatever the stack held, and a slice or a struct of garbage does not misprint — it faults, on the game thread of
a program that is already stopped, which is the worst moment this project has to offer. The daemon asks the program
which slots are bound before it builds the thunk, so the set it emits code for is the set the thunk will resolve.
Only named slots are recorded, and this is where most of the cost went. A slot whose address is stored anywhere
escapes, and an escaped alloca is one mem2reg cannot promote — so recording a slot is paying for it in every call to
that function, for ever. The slots that would hurt most are exactly the ones with nothing to show: dotimes's hidden
bound, the temporaries (min) and (max) evaluate their operands into, the render walk's own scratch. They keep their
promotion and are refused by name (s4, "a slot the compiler made up") rather than shown under an invented one.
Recording every slot instead was built and timed and came out inside the noise on both benchmarks, so the rule stands
on what it shows rather than on what it saves.
Shadowing is right here, and that is not an accident of this design — it is the thing the DWARF route still owes.
check.ml's fresh_slot only ever allocates, so (let [v 22] …) inside (let [v 11] …) is two slots, both named
v, and both appear with their own values. lldb answers p v with 11 in that program and will until a
!DILexicalBlock per Let exists.
Four refusals, each by name and with its reason. Three per slot — invented, not yet bound, and no printer for the
type (a map, a function value, a type variable; the arm exists, no program the checker accepts has reached it yet, so
it is written and untested) — and two whole frames: one belonging to a C-x C-e thunk, whose Tast the session does
not keep, and one whose body is not the body this session holds.
The superseded frame, and why a count could not find it. Installing while stopped is deliberately allowed — it is the fix-it-and-retry loop — so the frame on the stack and the body the session holds can be two bodies of one function. A slot count catches a body that gained or lost a binding and nothing else; the case that matters is a rename, which changes neither the count nor the types, and which would otherwise show every new name against the old body's storage with nothing saying so. That is the confident wrong answer, in the one place someone is working out what went wrong.
So each %fninfo carries a slot fingerprint: Emit.slot_fingerprint over every slot's name together with the
spelling of its type, since either can change on its own. The frame carries the value for the body it was compiled from,
flan_dev_frame_slotsig reads it, the agent snapshots it with the rest of the frame and puts it on the backtrace line,
and Dev.locals recomputes it from the body it holds and compares. Different means refused by name — this frame's body
was redefined since it was entered, so its names no longer describe its values — rather than answered. The count check
stays in front of it because its message is the more specific one.
Computed in emit.ml and read from there by dev.ml, so there is one definition of it and the two ends cannot drift.
It stays off the wire: (:op "backtrace") still answers four fields per frame, because a hash is not something an
editor can act on and the refusal says the fact in words instead. The same is true of the globals fingerprint that
now travels beside it. The bound is worth stating: it is a 30-bit hash, so a
collision is possible in principle, and it would reproduce exactly the silent wrong answer this catches — but only
between two differing bodies of a function whose qualified name has already matched, since find_fn is what gates the
comparison at all.
The first build of this shipped the fingerprint into %fninfo and stopped there — no accessor, no wire field, no
comparison — and was left in the tree with its own test red. Four of five hand-offs missing looks exactly like one
hand-off dropping a number, which is what the note left behind said it was.
A (Vec T) shows as <vec> and a (Ptr T) as <ptr>, because that is what render.ml already does for them
everywhere else: following a pointer a REPL was handed is not a safe thing to do on someone's behalf, and walking a
Vec structurally is a walk over storage the frame does not own — (print (as-slice v)) is how that is asked for,
and it says at the call site that it borrowed.
Each slot is rendered from its address rather than copied into the thunk first. A copy would be one alloca the
size of the slot — 40KB for sand's grid — and the walk only ever shows eight elements of it. The cost is one call to
flan/dev-slot per leaf the walk reaches rather than one per slot, which the depth and span caps already bound.
Globals of a stopped stack
The other half, and in this language arguably the more useful one: a game keeps most of its state in top-level
defvars and sand.flan holds its entire grid that way, so "what is this program's state right now" was a question
with nowhere to ask it.
(:op "globals") → (:status "ok"
:globals (("grid" "[4 i32]" "[ 7 5 0 0]" (0 1))
("pressure" "i64" "12" (0))
("label" "string" "\"running\"" (1)))
:refused () :skipped ())
Not per frame, and that is the design rather than a layout preference. A global is not part of a frame — it is program state the frame happened to touch — so nesting it under one implies an ownership that is not there and repeats the name once per frame that reads it. So: one section, whose contents are the union of the globals every frame on the current stack references.
The compiler does the choosing. Reach.expr_refs is the walk that already computes what a function refers to — it
is how the link drops a package nothing calls — and pointed at one body it answers that body's reference set. Listing
all of a program's globals instead would bury the one that matters under the prelude's PRNG state; taking only what
the stack reaches is a filter the compiler can apply and a person cannot. Direct references only, with no transitive
closure through calls: a callee that reads a global is either on this stack, contributing its own references already,
or it is not on it and is not part of where the program stopped.
Each entry carries the frames that touch it, by index, which is what the conditions buffer already numbers frames by. That recovers what per-frame nesting would have told you — "the whole chain is reading this" reads differently from "only the innermost does" — at no cost in duplication. Ordered by the innermost frame that touches it, because a deep stack makes the union large and proximity to the error is what puts the likely culprit on top; ties keep declaration order.
The mechanism is one step simpler than locals. A local is reached by address, and only the stopped program
knows where its frame is — hence flan/dev-slot, hence the bound_slots round trip, hence the null-until-bound
refusal. A global is reached by name: Emit.redefinition writes a global the host already has as external, so the
dlopened thunk binds to the program's own storage and the dynamic linker does the work. Nothing is asked of the stopped
thread at all, and there is no not-yet-bound case, because a global's storage exists from the moment the process
started. It is Render.render over (Global n) — the same root C-x C-e uses when you type a global's name at it.
A frame that cannot be attributed contributes nothing and says so. An eval frame has no declaration in this
session; a lifted handler clause has none of its own; a frame whose body was redefined since it was entered holds a
body whose reference set is a claim about different code. All three go into :skipped with the reason, because "the
union is incomplete and here is why" and "these are all of them" are different answers and the second one is the lie.
The hole this had, and the second fingerprint that closes it. Emit.slot_fingerprint hashes a body's slots. For
locals that is exactly the right cut: identical slots means the names still describe the storage, so the answer is
still true. Here it was not, because a body can change which globals it names without touching a single slot — and then
this section showed the new body's reference set attributed to the old frame. The values were never at risk; they
are read from the program's storage by name. What was, is one frame's membership in the union and the frame numbers
beside an entry.
So %fninfo carries a second number beside the slot fingerprint: Reach.ref_fingerprint, over the set of globals
the body names — sorted and deduplicated, because a reference set is not ordered and a body that mentions the same two
globals the other way round is the same body, where slot indices make the slot fingerprint order-sensitive on purpose.
It travels the same path the first one does: flan_dev_frame_refsig, the agent's snapshot, the backtrace line, and
Dev.globals_op recomputing it from the body it holds. Different means that frame is in :skipped with its own
reason — this frame's body names different globals than the one this session holds — and the rest of the stack still
contributes.
Two numbers and not one combined, which is the whole reason this is a second fingerprint rather than a wider first
one. They are different facts: a frame whose slots match and whose globals do not has locals that are perfectly
readable and attribution that is not, and one hash over both would make locals refuse a frame with nothing wrong
with it. So locals still checks the slot fingerprint alone.
It lives in reach.ml rather than emit.ml because Reach.expr_refs is already the walk that answers "what does this
body refer to" and is the same walk the union above is built from — the two ends cannot disagree about what counts as a
reference. Which names are globals is the caller's to say: the emitter knows the program's globals, and so does the
session. test_dev.ml now drives the exact case — a redefinition that binds identical locals and names untouched
where the stopped frame's body names pressure — and with the check disabled it fails twice: once on the missing
refusal, once on untouched appearing in the union under frame 0.
Conditions — step 2: restart-case and invoke-restart
spec-conditions.md §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
restart-case further out.
(defn fetch [n i32] i32
(restart-case (middle n) ; its value if nothing transfers
(use-placeholder [] -1)
(retry [] 7)))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
(fetch 2)) ; -1
The channel is an out-parameter, as §6 now says: one ptr appended to every Flan signature, written by an
invoke-restart and checked after every call. The return type stays what the source says, so the disassembly is the
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
slot and each frame only has to check and return early, which reuses the existing return path and with it §5's defers.
Emit.signature was already the one place a signature is spelled, which is what made this a three-line change rather
than a hunt.
Every function is transfer-transparent, release included — and that is the ABI, not a stopgap. A cell holds a bare pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and plan.org both now say the later optimisation may stop a function checking the channel, or pass the pointer straight through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform also means redefinition acquires no new refusal class.
The transfer target is the restart frame's own address, not a clause id. This is a correction to what the previous
note settled. A static id has to be unique against every module a running program may later load, and a hash is only
probably unique — two restart-cases colliding means the inner one silently catches a transfer aimed at the outer. The
frame is an alloca in the function that offers it, so its address is exact, and it also says which clause, which is
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
Re-entering a restart-case then works with nothing extra: each activation allocates its own frames, and §4's
"innermost offering the name" is just the order of the walk.
Cleanup happens in landing blocks, one per region. A guard branches to the innermost open one, which pops whatever frames it established and either catches the transfer or forwards it outward:
- a
restart-case's pops its restart frames, compares the target against its own, and either runs that clause or puts the target back and goes on out; - a
handler-bind's pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, and without it the handler stack would be left pointing into a frame that has gone; - the function's own runs its defers (§5) and returns early.
errdeferdoes not run and never could:try/Resultis still refused by name.
A single function-wide unwind block would have been wrong for the first two: a call inside a restart-case body would
jump straight past the very form that was supposed to catch it.
The channel is cleared before any cleanup runs and put back after. A defer makes ordinary calls and each one is guarded; with the channel still set the first of them would branch straight back into the landing block it came from. Same reason the clause body starts with it null.
flan_signal takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
cross a foreign frame" would make restart-case useless. It is also the only such frame: extern is Flan-to-C only and
there are no function values yet, so nothing can call back into Flan across one.
Scope, each piece refused by name with its reason and a test on the reason:
- restarts took no parameters. That covered §1's own
load-textureexample and skipped argument marshalling and §3's runtime check. Both are in now; see "Conditions — step 3" below. returninside arestart-casebody, exactly as insidehandler-bind: a bareretskips the pops.- one
restart-caseoffering a name twice — §4 finds the first frame offering it, and two in one frame makes that a choice nothing in the source shows. invoke-restartinside adefer. A defer is the cleanup a transfer runs on its way out, so a transfer starting there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a defer that reaches one through a call is trapped at run time byflan_transfer_fail, because nothing static could see it.- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past everything. There is nowhere to resume, so there is nothing else to do.
(error c), §2. The same walk as signal, and the difference is entirely what happens when the walk ends: signal
returns () and the signalling function carries on, error has type Never and stops. So only a transfer gets past
it, which is why emit puts a guard after the call and then unreachable — and why flan_error cannot be marked
noreturn, since it does return, on exactly one path. Being Never is also what lets it stand as a restart-case
body's fall-through, which is the shape §1's load-texture example needs. test/programs/error.flan is the unhandled
case: it cannot be an outputs row, because it does not exit 0.
flan_transfer_fail covers the ordinary return path as well as the unwind one: a defer that reaches an invoke-restart
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
A lifted handler clause is named after the function it came out of — handler/step/0/Missing — and is emitted by a
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
nor attributable, and a redefinition of a function containing a handler-bind failed in llc with an undefined value.
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
test/test_dev.ml drives that path: a third evaluation redefines step to a restart-case whose frame is an alloca
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
crosses probe, which the host was compiled with. Those three do not meet anywhere else.
Two things found by writing it:
{ ctx with in_handler = true }was a latent bug.ctx.slotsandctx.slot_tysare mutable, so a copy allocates the body's slots into a record the function never sees again and the indices collide. It was harmless only because nohandler-bindbody in the tests had aletin it. The flags are set onctxand restored now.test/reload_host.chad to learn the parameter. It callsflan.outerthrough an__asm__label, which does not fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere else entirely.
test/programs/restarts.flan runs in the acceptance table at -O2, at -O0 and as a dev build. -O0 is not redundant
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
of those calls goes through a cell.
Conditions — step 3: restarts take parameters
§3's other half, and the half the comparative studies all ask for: use-value and store-value are the two restarts
whose answer is not in the program.
(restart-case (middle n)
(use-value [v i32] (* v 2))
(retry [] 7))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7)) ; 42
The parameters live in a buffer the restart-case owns. The obvious place is the invoker's frame — it is where the values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so the invoking side is gone by then. The invoker stores into the target's buffer while both are still alive, which is the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any other.
What a restart takes is checked at run time, and it has to be. §4 finds a restart by name on a dynamic stack: the
invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker
to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and
invoke-restart compares both against its own before it stores anything. Every frame carries them, parameterless ones
included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not
redundant with the hash: it is what keeps a hash collision between two different signatures harmless, since a
collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well,
because the message has to say what was wanted and what was given and neither end knows both:
restarts.flan:79:36: restart use-value takes (i32), given ()
restarts.flan:87:36: restart retry takes (), given (i32)
The arguments are evaluated into slots before the invoke node, not hung off it. Two reasons and both are real. An
argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside
an argument has to be on the walk Reach and Load already do — InvokeRestart was a leaf to both, and a leaf that
grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker.
restarts.flan has exactly that function, half, to keep the claim tested.
The break loop can take a restart it cannot fill in, so it is refused. A transfer has two sources: an
invoke-restart, which writes the arguments first, and the break loop, which aims the channel at a frame by position
and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them
apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked
unfilled, invoke-restart marks it filled, and a clause with parameters checks the mark before reading. Choosing
use-value from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the
top of NEXT.md: the answer is a Flan expression, and there is already something that compiles one against the live
program.
Lookup stayed by name, and the signature is checked against what it found. flan_find_restart matches the name
hash and nothing else, so an inner (use-value [s string] ...) shadows an outer (use-value [v i32] ...) and an i32
is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first
frame offering the name") and it is the thing a reader will assume works the other way, so restarts.flan has a case
for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments,
which is overload resolution on a dynamic stack.
Both of these guards go through fail_block unconditionally, unlike the bounds checks: --no-bounds-checks does not
remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or
written to a different layout, is memory corruption.
runtime/flan_rt.c gained two message functions and nothing else. The restart frame's first four fields are the ones C
declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.
§3's other open point, a report string per clause, is still open and was not settled first as §3 asks. The field and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing that listing anyway.
What is left
- Editor comforts: completion, eldoc, jump-to-definition, error overlays.
Session identity is the daemon that owns the build. A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default.
Allocators, (Vec T) and StorageExhausted
spec-memory.md's Allocators section is frozen and settles what to build. This is why the built thing is the shape it
is, and — separately and loudly — the three places it amends that section plus the one thing it adds to it.
Allocator is a builtin opaque type, so none of milestone 5 was needed
The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and
check.ml refuses function values four ways — a written (Fn ...) annotation, a written fn literal, a defn's name
in value position, and calling anything other than a named function. Read straight off those lines, containers need
function values and the work doubles.
None of the four is anywhere near this. All four are about surface syntax, and a value the compiler builds that no surface form names trips none of them:
Allocatoris aTypes.tcase with no user-writable constructor, the waystringis a builtin ptr+len. No Flan type names its procedure.- The procedure is a C symbol the emitter names.
vec-new,push,free,free-alland the rest are ordinary named calls, whichcheck_callalready routes throughnamed_call.
The precedent was already in the repo twice: a handler-bind clause is lowered to a function called back through
h->fn(condition, xfer) and built as its own Tast.fn, never an Ast.Fn; and a dev build's call loads a pointer
out of a cell and calls through it, which is the indirect call the surface language refuses.
What does need milestone 5 is a user-written allocator: "here is my proc, make an Allocator from it" wants a
defn's name in value position. make-allocator, allocator-from and allocator are refused by name with that
reason, rather than coming back as unknown functions.
An Allocator value is a pointer to the runtime's struct, never a copy of one. That is forced, not chosen. The
capability set has to be readable from wherever a container landed, and free-all bumps an epoch every container made
from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never
fires.
The surface
| Name | What |
|---|---|
(heap-allocator) |
the general-purpose tier: malloc, aligned, with free |
(arena-new bytes) |
a bump arena over one fixed backing buffer |
(arena-destroy a) |
hands the pages back — see the amendment below |
(free-all a) |
releases everything the allocator holds, retain-capacity |
(can-free? a) / (can-free-all? a) |
the capability set, read at run time |
(alloc-epoch a) / (alloc-id a) |
the counter free-all bumps; the allocator's identity |
(alloc-budget a) / (set-alloc-budget a n) |
a ceiling on live bytes — an addition, see below |
(alloc-live-blocks a) |
"did you forget to free", answered at the tier that can answer it |
context/allocator / context/temp |
the current implicit allocator, and the per-frame arena |
(with-allocator a body...) |
rebinds for a dynamic extent and releases nothing |
(vec-new T) / (vec-new T a) / (vec-new) |
a Vec, against the context or a named allocator |
(push v x) / (reserve v n) |
(), both |
(at v i) / (len v) |
the array names, extended — not a parallel pair |
(as-slice v) / (as-slice v lo hi) |
a non-owning [T] view |
(clone v) / (clone v a) |
the only copy; assignment moves |
(free v) |
consumes its argument |
Three amendments to a frozen spec, and one addition
1. free-all is retain-capacity, and arena-destroy is the operation that hands pages back. The spec's table has
free-all and nothing else. Zig's ArenaAllocator.reset takes a ResetMode of free_all / retain_capacity /
retain_with_limit; Odin's arena_free_all is retain-capacity in effect, because its arena is one fixed backing
buffer and the call only sets offset = 0. For a frame arena reset every frame, retain-capacity is the normal case and
handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown
the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being
the same does not make a container built before the reset valid, which is the whole point of the trap.
2. context/allocator and context/temp are dynamic variables with save and restore, not extra parameters. The
spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the
FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other
lane working in emit.ml. The dynamic variable is the implementation; the literal reading is deferred and is a
performance question (a parameter avoids a global load), not a semantic one.
3. The Vec header is six words in every build, not four in release. The spec fixes the release layout at
ptr + len + cap + allocator with the generation word and the epoch dev-only. A layout that changes with a build flag
is a layout that can disagree silently across the reload boundary: a redefinition module is built by llc and ld
against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and
so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag
before it can land.
The addition: a budget. flan_allocator grew a ceiling on live bytes, 0 for none. The spec's retry restart is
answerable only by a handler that can make the same request succeed, and for a fixed backing store the handler that
works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which
is exactly what the epoch check catches. The spec names "grows the arena and then invokes retry" as the handler that
works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on
purpose.
with-allocator is its own IR node because of the transfer path
Save, run, restore — and restore again at the pad. That second restore is the whole reason it is a node rather than a
let and two calls. A body that errors, or one a handler transfers out of, leaves through current_pad, and a context
allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is
precisely where something is about to allocate in order to render a condition. test/programs/allocators.flan asserts
that path by taking a restart out of a with-allocator body.
It releases nothing, per the spec: not at the end of a let, not at the end of a function, not at the end of the body.
The program proves it by reading the epoch either side.
(Vec T): one type-erased runtime, and the backend learned almost nothing
The element type appears nowhere below the call site. size_of and align_of are produced where the concrete type is
known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement
(base/runtime/dynamic_array_internal.odin). The backend grew four prims in total:
Rt of string— a call into the runtime's C named by symbol, with argument and result LLVM types read off the expression nodes. The container runtime is type-erased and therefore is a list of C entry points, so one arm covers all of them. AVecargument crosses as its address, which is also what lets an operation mutate the caller's Vec.SizeOf/AlignOfof a type, filled in from the same layout calculator DWARF uses — the one the acceptance test already checks against LLVM's owngetelementptranswers.AddrOfof any expression, place or not, because the element apushcopies may be computed. The backend already spilled a non-place to a temporary for exactly this.
at and len were already the names for a fixed array and a slice, so a Vec extends them rather than adding a
parallel pair — the asymmetry nth was removed for. The value form (at v i) and the place form (set (at v i) x) go
through one helper, so they cannot drift apart the way nth did.
A Vec's length and index are i32, like every other length here. Widening indices is one change across every
container and not a Vec question.
let has no type annotation — parse.ml settles that a triple binding is ambiguous and that types are inferred — so a
local Vec has nowhere to say what it holds and the element type is written at the call: (vec-new i32). This is not
the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an
ordinary type rather than binding a type variable. Where the context does say — a defvar's type, a return type, an
argument — it may be left out.
A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is
Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a
different reason (below), but the adopt rule is what makes Zero of a Vec a usable value rather than a null deref.
Move-only is a dead set, and it is flow-sensitive at a join
Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing
one to a function, binding it, returning it and freeing it are all moves and all reach one place, and the handful of
operations that only look at a container (at, len, as-slice, push, reserve, clone) say so. Only a
syntactically simple target counts as a borrow — in (len (f v)) the call still moves v.
At an if and at a match, every arm is checked from the state before the form and the union of what they moved
survives the join. A flat set is wrong in both directions: it refuses (if c (free v) (free v)), which is legal, and it
accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union
says.
The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move, not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a move of a binding declared outside the loop is refused, with that as the reason.
What ownership is not transitive through yet, and why each is refused
The spec says ownership is structural — a struct containing a Vec is itself move-only, free recurses into owning
fields, and a field cannot be freed on its own. That machinery is the recursive teardown drop brings. Until it lands,
three shapes are refused where they are declared, each naming drop:
- a struct field of
Vectype, because the struct copies its header on assignment and nothing records a move; - a global of
Vectype, because the dead set is per function — two functions each freeing it is a double free nothing could see, and a global read does not go through the move path at all, so even the one-function case would be accepted. Half a rule is worse than none. A globalAllocatoris a different thing and stays legal: an allocator is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot see the locals of the function that established it; - a
Vecof aVec, because the type-erased runtime copies and releases elements bytewise:clonewould duplicate inner headers instead of copying what they own andfreewould drop their buffers.
And a Vec does not cross to C: handing a header that owns storage to C hands out an owner. (as-slice v) as
(Ptr T) plus (len v) is the shape that does cross, and the refusal says so.
StorageExhausted went in with Vec, not after it
No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the
operation signals (StorageExhausted {.bytes n .align a .allocator id}) with error — whose type is Never — inside a
restart-case offering retry. One rule over every allocating operation, which is what keeps push and reserve at
(), clone at the container, and no signature anywhere growing a Result.
It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every
allocating operation, which is the point of having decided it first. Odin's append returns an ignorable
Allocator_Error, and its type-erased path returns the old length on a failed reserve; an append that appends nothing
and says nothing is the outcome this rule exists to make impossible.
The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:
(let [ok false]
(while (not ok)
(restart-case
(do (set ok ATTEMPT)
(if (not ok) (error (StorageExhausted {...}))))
(retry []))))
A handler that frees something, releases a scratch region or raises the ceiling and then invokes retry lands in the
clause, the clause falls through, and the while re-attempts the same request. Every argument to the attempt is
bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a
push was given. With nothing handling it, error stops the program on the frame that erred.
This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site can. The condition is a value struct with fixed numeric fields and no rendered message, because formatting would allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.
The epoch trap, which is the shipping answer to an open question
spec-memory.md leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which
allocator a construction used, and with-allocator plus context/allocator are exactly the mechanisms that deny that
knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, free-all bumps
that counter, and any operation on a container whose recorded epoch has moved traps naming the site.
test/programs/stale-region.flan is the case, and the point of it is that v is still in scope, still looks fine, and
nothing marked it — which is precisely what a static rule cannot see.
The generation word has no reader. It is bumped on every reallocation, as specified, and the stale-slice trap it exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said plainly here rather than implied by the word's presence in the header.
What this leaves for steps 5 to 7
(Map K V) is built — see below. What is left: drop and with it the transitive move-only rule, recursive teardown,
and the refusal to construct a drop-carrying container against an allocator without can-free; (Result T E) and
try; generics; the macro expander.
And the accumulation pattern — (fn [c] (push errors c) ...) over an enclosing Vec — which Vec does not buy:
capture does not exist at all, and the spec's captured-Vec-by-pointer rule has never had to exist because every
capturable type today is a value type. It is its own item and should be planned as one.
(Map K V), which is Odin's map
Step 4 of the container build order, over the runtime Vec already established. spec-memory.md is followed as
written; the two places this departs from Odin are named below, and there is one amendment to the spec and one
restriction under it.
The three properties that were the point
Odin's header states them and they are why it was the thing to follow (base/runtime/dynamic_map_internal.odin).
- Open-addressed Robin Hood hashing at a 75% load factor. No buckets and no per-entry allocation: one block holds every key, every value and every hash. Robin Hood is the part that earns its keep — on insert, an element that is further from the slot its hash wanted than the occupant it is looking at takes the slot and sends the occupant on. Probe distances even out, and a lookup may stop the moment it is further from home than the occupant it is looking at, because no element is ever further from home than one it passed. That early exit is why a miss costs about what a hit does.
- Cache-line cell packing. A flat
[capacity]Karray lets one key straddle two cache lines, so a probe walking four slots can touch five lines. A cell packs as manyKas fit in 64 bytes and pads the rest, so no key ever straddles a line. Keys, values and hashes are three separate runs, so a probe — which reads hashes, and only then one key — touches hash lines and nothing else until it has a candidate. - Pointer-width integers throughout, so no sign extension or masking gets into the probe loop.
Two departures from Odin, both deliberate
There are no tombstones, because spec-memory.md defers removal ("Move-aware lookup, removal, and owned entries
are deferred"). A slot is empty or occupied and nothing else, which deletes Odin's backward-shift loop entirely — it
is the single largest reason this file is shorter than the original. When removal arrives, that loop is what it costs.
The header does not tag the capacity into the data pointer. Odin stuffs log2cap into the low six bits because
its Raw_Map must be three words. This header already carries an allocator, a generation and an epoch, so the tagging
would buy nothing, cost a mask on every access, and — the part that actually matters — make correctness depend on the
block being 64-byte aligned. Alignment is requested; an arena whose base is not cache-aligned now gives a slower map
rather than a wrong one.
The header is six words, 48 bytes, the same as a Vec's and for the same reason: a layout that changes with a build
flag can disagree across the reload boundary.
data len log2cap allocator gen epoch
The hash and equality pair, and why most key types do not get one
spec-memory.md restricts keys to built-in structural types and makes hashing and equality compiler-provided
structural operations. So there is no dispatch to design: a key type resolves to a pair of symbols, passed to the
type-erased runtime the way Odin hangs two contextless procs off a Map_Info.
Most key types need nothing emitted. A key whose equality is bytewise and whose bytes are all present — every integer,
enum, bool, and fixed array of those — is served by one runtime pair over (pointer, size). Two kinds are not, and
the reasons are worth keeping because they are what a bytewise shortcut would have got wrong:
- a string is ptr+len and its bytes are elsewhere, so two equal strings at different addresses must hash alike;
- a struct may have padding, whose bytes are indeterminate — two structs equal field by field can differ bytewise — and it may hold a string, which brings the first problem inside it.
A struct therefore gets a pair emitted for it, walking its fields in declaration order and addressing nothing but fields. Two maps with the same key type share one pair, and a struct reached twice through two fields emits one.
Tast.FnAddr is what carries the symbol, and it is not a function value: nothing in the surface language can
produce one, name its type, or call through it. It is the same escape the allocator used — the one NEXT.md predicted
would work — and reach.ml learned the edge, because a function reached only by address is invisible to the
reachability walk otherwise. That was already true of handler clauses.
A pair emitted for a struct is an ordinary Flan function, so its signature ends with the transfer channel like every other. The runtime's typedef spells that pointer out rather than hoping nothing writes through it, and each built-in hasher exists in two spellings — the pointer form that matches the typedef, and the direct form an emitted hasher calls per field, which has no channel to hand on.
The surface
| Name | What |
|---|---|
(map-new) (map-new K V) (map-new a) (map-new K V a) |
a new map; the pair may be omitted where the context says |
(put m k v) |
upsert, () |
(get m k) |
(Option V) — absence is None |
(has-key? m k) |
bool, copying no value — an addition; the spec does not name it |
(len m) (reserve m n) (clone m) (clone m a) (free m) |
extended, not duplicated |
has-key? is not in spec-memory.md and is an addition, flagged because everything else here is the spec's.
(get m k) answers the same question, but through an Option the caller then has to match, and the common use is a
condition. It copies no value, which is also why it is not just get with the result thrown away. The ? suffix
follows can-free?.
len, reserve, clone and free were extended rather than given map-shaped names of their own, which is what
at and len already did for Vec: one question, one word. reserve's n is entries, not slots — the runtime sizes
the block so n still sits under the load factor, which is the only reading of "room for n" that does not reallocate
on the nth put.
get builds the Option in the checker, not the runtime, which has no idea what an Option's layout is; keeping it
that way is what lets one entry point serve every value type. put and get bind their arguments to slots before the
allocation guard, so a retry re-attempts the allocation and not the expressions that produced the key and value.
{K V} is the type spelling and there is no map literal. A bare map form in expression position is a struct
literal's field list, and giving the same braces two meanings is what the colon-to-dot change was for. A map is built
with map-new and filled with put.
The refusals, each by name
- A float key — not a milestone question, which is why it is said separately. NaN is not equal to itself, and
0.0and-0.0are equal while differing bytewise. There is no equality there for a map to hash. - A
Ptr, slice,VecorMapkey — would hash an address rather than what it points at, which is a different operation. - A fixed array whose elements are not compared bytewise — an array of structs or of strings needs the per-element walk a struct key gets, driven by a loop rather than a field list. Nothing has wanted one, so it is refused rather than written untested, and the shape that does work (a struct holding the array) is named beside it. This is narrower than the spec, which lists fixed arrays without qualification.
- A move-only value — the refusal
(Vec (Vec T))already carries, for the identical reason: the runtime copies entries bytewise, soclonewould duplicate headers andfreewould leak what they own. Owned entries arrive withdrop. ()as a value — there is nothing to store, and the cell geometry divides a cache line by the element size. Named rather than left to divide by zero, because it is the natural spelling of a set.
StorageExhausted under retry holds over map-new, put, reserve and clone, reusing the machinery that landed
with Vec; no operation returns an error. A map is the harder of the two containers for that rule, and
map-exhausted.flan is why it gets its own program: a Vec's failing allocation leaves the Vec untouched, whereas a
map's growth allocates a new block, rehashes into it, and only then releases the old one — so a failure partway must
leave the map exactly as it was, or the retry re-attempts against a half-moved map.
The epoch trap covers the map too, and map-stale-region.flan is separate from stale-region.flan because the two
reach the check by different routes: a Vec's operations check on the way in and stop there, while a map's get goes
on to call a hash and an equality function through pointers into the block. A missing check there is not a wrong
number, it is a probe loop walking released memory.
Is this another Python dict? — measured
The question was asked directly and the answer is that Python's algorithm is fine. What makes CPython's dict slow
is that every key and every value is a separately allocated, reference-counted object and hashing goes through
__hash__ and __eq__ calls that cannot be inlined. This map stores raw bytes in the block and compiles hashing and
comparison concretely per key type. That is most of the gap before any cleverness.
Measured on this machine, i64 to i64, against CPython 3.13's dict on the same workload:
| Working set | Flan | CPython dict |
|---|---|---|
| 10k entries, 10M lookups (cache-resident) | 21 ns/lookup | 132 ns/lookup |
| 1M entries, 10M puts + 10M lookups | 1.41 s | 1.16 s |
Six times quicker cache-resident, and slower at a million entries — and the second row is written down rather than left out. Both are waiting on memory there, and this layout waits longer: keys, values and hashes are three separate runs, so a lookup that misses everything takes three cache misses where a compact dict takes two, and the hash run is a full eight bytes a slot. Cell packing buys probe locality, which is a win while the hash run is resident and a loss once nothing is. One byte of metadata a slot — the Swiss-table arrangement — is the known answer and is not built.
The path from 35 ns to 18 ns (the cache-resident floor, at 500 entries) was profiled, and the first two guesses were
both wrong: the per-slot cell division and the block-size divisions were each replaced first and neither moved the
number. What did: FNV one byte at a time became eight, and a key that is one machine word became one load and one mix
with no loop; equality on eight bytes stopped being a call into libc's vectorised memcmp, and copying a value out
stopped being a call into memmove; the block geometry stopped being recomputed five times over in a function that ran
twice per lookup; and the seed stopped being a five-multiply avalanche on the critical path for mixing the hasher does
again immediately after. 64/size is a table, which is Odin's Map_Cell_Info by another route — Odin precomputes it
per type because the probe loop must not divide, and here the sizes arrive as ordinary arguments.
What remains at 18 ns is the type erasure itself: a non-inlinable call into the runtime and two non-inlinable indirect
calls to the pair. That is the trade spec-memory.md chose deliberately — "It is type-erased on purpose… No generics
are involved, and none are needed" — and monomorphisation is what would buy it back, at the cost the spec declined.
Unions, and the tag they carry
defunion parsed and its shape was checked long before this; naming the type (check.ml:312) and constructing a
value (:1075) were both refused as milestone 6. They are not any more.
It is closer than the milestone number suggested, and the reason is Option
Tast.arm already carried acase (a case name) and binds (the slots a payload binds to). That is union shape,
built and exercised, because Option is a two-case union wearing a special coat and match over it worked
already. So check_match grew a second subject rather than a second path: one function decides which case each arm
names and what type each name it binds has, and everything after it — the dead-set join across alternatives, the
result type, the exhaustiveness check — is the code that was there.
Option was not desugared into a declared union, and that is deliberate: Option is generic and no declared
union is. The coat is the part that cannot be taken off until generics exist.
A union is Types.Named exactly as a struct is. One case in Types.t covers both, and which table the name is in
is the only thing that tells them apart. That is what let a union be a field, a parameter, a return type, a slot and a
copy without a single one of those paths learning that unions exist.
The layout, which is the part that has to be exactly right
%"U" = type { i32, [k x iA] }
%"U.Case" = type { the case's fields, in declaration order }
A tag, then room for the largest case, with A the alignment the widest member of any case needs and k the size
rounded up to it. Writing the payload as an array of iA rather than of i8 is what makes LLVM align it without an
explicit align on a type — and it is what makes the whole thing
struct { int tag; union { ... } u; }
byte for byte. That agreement is the point. A macro is [Form] -> Form, so Form has to be the same bytes in
the compiler and in the dlopened macro, and nothing at run time would notice a disagreement.
The tag is an i32 and not an i8. With an 8-byte payload alignment the two cost the same, and i32 is what a C
enum field is — the spelling the macro lane will have to write by hand on the other side of the boundary.
Both the union and one struct per case are emitted as named LLVM types, so every reader geps rather than computing
byte offsets of its own. Construction is a store of the tag and a store of the case struct through a gep into the
payload; match is a load of the tag and a gep the other way. There is one function that knows how a payload is
read — case_field_addr — because there are two readers: a match arm's binds and the structural printer.
The layout goes through the same oracle the DWARF section uses: LLVM's own answer for the emitted type, read back as a
constant-folded ptrtoint. Two unions are checked, one whose widest case is a pair of f64 and one whose cases are
all i32, so the payload size and alignment cannot be constants the test agreed with by accident. The Shape in
test/programs/unions.flan was checked against clang's answer for the same declaration in C: 32 bytes aligned 8 with
the payload at offset 8, and 40/8 for a struct holding one.
The surface
(defunion Shape
[Empty
(Dot [x f64 y f64])
(Rect [w i32 h i32])
(Tag [name string n u8])])
(Shape.Rect {.w 3 .h 3}) ; a value: the type and the case, then the fields
(Shape.Rect {.w 3}) ; ZII, exactly as in a struct literal — .h is 0
Shape.Empty ; a case with no fields is a whole value, not a call
(match s
Empty "empty"
(Dot x y) "a dot" ; positional, in declaration order
(Rect w h) "a rect"
(Tag name n) name)
Construction is qualified and a pattern is bare. The two are not inconsistent: at a constructor nothing says which union is meant, and at a pattern the scrutinee's type already does. The qualified spelling is accepted in a pattern too, since that is how the value was written and writing it again should not be an error. Two unions may therefore share a case name, and that is not refused — refusing it would be a restriction with no mechanism behind it.
Construction needed no change to parse.ml. . is a symbol constituent, so Shape.Rect reads as one name; the
struct-literal arm fires on a symbol followed by a map; and the field-access arm needs a leading dot, so the two
cannot collide. (Name {.field value}) is one syntax for both, and which it is, is decided against the tables.
Tags are declaration order, so case order is part of the contract
Tag n is the nth declared case, from zero. The consequence is that an all-bytes-zero union is the first declared
case with a zeroed payload — which is exactly the rule that makes an Option's zero a None, and it is what makes
(defvar u U) and an omitted union-typed struct field mean something rather than nothing. Reordering a union's cases
is a layout change, the same way reordering a struct's fields is.
What is refused, and why each
- A non-exhaustive
match. Refused, not defaulted, and the message names the cases with no arm. A match that fell through would have to produce a value of the match's type out of nothing, and the case a union grows tomorrow is exactly the one a reader wants to be told about today._is how to say "the rest", written where it can be seen. - A case pattern binding some of a case's fields. All of them or none, positionally — binding a prefix reads the wrong field the moment one is inserted above it.
- Two arms for one case, which is a mistake and never an intent.
- A union with no cases. No value of it can exist, so a parameter of that type is a function nothing can call.
- A case field that is move-only, in the same words a struct field already gets and for the same reason:
recursive teardown arrives with
drop. - A union as a map key. The payload past the case in hand is indeterminate, so hashing the blob would make two
equal values hash differently. Hashing one properly is a per-case walk driven by a switch — a different shape from
the field list
struct_key_pairemits, and nothing has wanted it. uniniton a union. This is the one refusal that is not the struct rule. Everywhere elseuninitis an opt-out from ZII and the bytes are whatever they were: a garbagef64is a garbage number. A union's tag steers control flow — a tag no case names falls past every comparison in amatchinto a block LLVM is entitled to assume cannot be reached. That is the one place where garbage becomes "the optimiser may do anything".(.x u). A union's fields belong to a case, and which case is being held is what the tag says.matchis how one is opened, and its arms bind the fields of the case they matched.- A global initialised with a case.
(defvar g U (U.B {.x 1}))would mean serialising the fields into the payload blob at link time, which is a byte-level encoder this compiler does not have and which could not express astringfield at all — that is a pointer the linker has to relocate and a byte array has nowhere to put a relocation. A zeroed global is fine and needs none of it: it is the first declared case.
Recursion, and the shape Form will have
A union that contains itself by value has no finite size, and payload_lay
would recurse forever laying one out rather than failing. It does not get the
chance: check_finite already walked a union's cases, so (defunion T [Leaf (Node [l T r T])]) is refused with "T contains itself by value, so it has no
size — go through (Ptr T)", and so is a pair of unions that contain each
other. Through a pointer it works, and that is the shape a Form has:
(defunion Tree [Leaf (Node [l (Ptr Tree) n i32])])
(defn depth [t (Ptr Tree)] i32
(match (deref t)
Leaf 0
(Node l n) (+ n (depth l))))
Why match's fall-through is still unreachable
The block after the last case comparison is unreachable, kept from the
Option path. That is only sound if no reachable program can hold a tag no
case names — and none can: Zero is tag 0, which is a real case; every
construction writes a tag the checker resolved; and uninit, the one way to
get bytes nobody wrote, is refused on a union for exactly this reason. The
refusal is what pays for the unreachable.
The diagnostics bug, fixed
(A {.x 1}) on a case of a union said "unknown struct A", because env had no table of case names and could not
tell a case from a misspelling. It has one now, keyed both by the full spelling U.C and by the bare C. The full
spelling is a key rather than something split out of a dotted name at the use site, because a union's own name can
contain a slash — an imported rl/U — and string surgery would own an edge this does not have to.
Printing
render.ml walks a concrete type for print, the REPL inspector and the break buffer's locals, and a union fell
through its Named arm to <Shape>. It recovers the case from the tag by a chain of comparisons — the same shape the
enum arm already had, and for the same reason: the name is erased before any backend sees it — and reads the fields of
that case only. It prints (Shape.Dot {.x 1.5 .y -2.5}), which is what the source would write.
What is left
An imported union is still refused by name at load.ml:312, so a union is file-local. That is not a blocker for
the macro expander: the prelude is parsed and prepended into the same flat namespace before collect runs, so a
defunion Form in prelude.ml is an ordinary same-file declaration needing no import — verified by declaring one
there and matching it from a program. dev.ml's inspector still says "union values are milestone 6" for a stopped
frame's locals, and shim.ml's "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards
is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates.
(Handle T) and the pool, which is what a stale reference answers with
A handle is a reference to something that can die, which reports that it died rather than silently resolving to
whatever reused its slot. check.ml refused (Handle T) by name as milestone 6; this is what it stood for, and the
pool came with it because the pool is what makes the report possible.
The problem is concrete and is not about memory safety. Entities live in a pool; something holds a reference to one — a projectile chasing it, the UI showing its health. The entity dies, the slot is reused, and a raw index now names a different entity. Nothing crashes. The projectile chases the wrong thing, at full speed, for the rest of the game.
It was built for two reasons, both already recorded. On its own terms, for entities referred to across frames. And
because it is the real gate on managed classes: plan.org's rule is that nothing starts on defclass until
ordinary struct, Handle and reload semantics work, and Handle was the only one of the three missing. That is not
an incidental precondition — migrate-instances has to enumerate live instances, and a pool behind generational
handles gives that by construction where a world arena and an owned region do not. plan.org presents the three storage
strategies as a free choice and they are not.
PORTING.md found no customer for handles in the author's real game today, so this is deliberately the smallest
correct thing rather than a rich API: eight names, no iteration protocol, no cursor type, no clone.
A handle is one i64, and the halves are 32 and 32
Slot index in the low 32 bits, that slot's generation counter in the high 32. One machine word, so it copies, zeroes
and compares like the integer it is, and it owns nothing — the pool is the single owner. That is what lets a
handle sit in a struct field and in a global where a Vec may not, and it is the reason the ownership rules needed no
new case: Types.is_move_only says yes to Pool and no to Handle, and the three existing refusals (a struct field,
a union case, a global) picked the pool up unchanged with the messages they already had.
The index is 32 bits because a Vec's index is an i32 here and widening indices is one change across every
container, not a pool question.
Live is odd, and two things fall out of it
A slot's generation starts at 0 and is bumped on every allocation and on every release, so an odd generation means live and an even one means dead. Both consequences are load-bearing:
- A zeroed handle resolves to nothing. Generation 0 is even, so ZII gives a
(Handle T)field the right meaning for free instead of pointing it at slot 0.handles.flanprints one:<handle 0:0>, and resolving it answersNone. - Iteration needs no second array and no spare bit. Asking whether a slot is live is asking whether its generation is odd.
The generation wraps by retiring the slot
32 bits is 2^31 allocate/release pairs on one slot — every frame at 60fps for a year and a bit. "Rare" is not an
answer when the failure it produces is the silent wrong one this type exists to prevent, so a release from generation
0xFFFFFFFF bumps to 0 and does not put the slot back on the free list. The slot is retired: dead forever, its
payload leaked, and no future handle can collide with an old one. Leaking is defined behaviour here, and one slot is a
bounded price for making the collision unrepresentable rather than unlikely.
resolve answers (Option (Ptr T)), and the spec settled that, not this lane
The task asked whether a lookup should answer (Option T), matching (get m k). It answers (Option (Ptr T)), and
spec-memory.md already writes it out — its worked example under "Mutating something you matched" is annotated
(Option (Ptr Enemy)) — for the reason given one line above it: pattern bindings bind values, so a matched struct is
a copy. A copy cannot be written back, and writing to the pooled entity in place is what a pool is for. (Option T)
would answer a question nobody asked.
The Option half is get's shape and for get's reason: absence is an answer, not a failure. A trap would be wrong
here — the entity dying is the expected case, not a bug.
The hole, said plainly. A (Ptr T) from resolve is invalidated by any insert that grows the pool, exactly as
a slice is invalidated by a push. The handle survives that and the pointer does not. This is spec-memory.md's
explicit Zig/Odin borrowing contract one level down, and it is worth naming rather than implying, because it
reintroduces the silent-wrong-answer mode the handle just removed for anyone who keeps a resolved pointer across an
insert. Chunked never-moving storage is the fix and it costs code; taking the contract is the smaller correct thing,
given as-slice already established it.
len is the slot high-water and live is the count, in that direction
(len p) is how many slots have ever been handed out. (live p) is how many of them are live now. It had to be that
way round: 0..(len p) are the indices (pool-handle p i) accepts, so a loop bounded by len visits every live
entry. Bounded by the live count instead, it would silently skip entries the moment anything had been released —
which is exactly the quiet wrong answer the whole type exists to remove.
(pool-handle p i) answers (Option (Handle T)): the handle of slot i, or None if that slot is dead. That plus
len is the whole of iteration. An index outside 0..(len p) traps, exactly as (at v i) traps: an index is an
index here, and answering None for one would hide a bug rather than a death.
A slot is released through the pool, and that is not a third release point
free consumes its argument as a move, and a handle is a copyable number that owns nothing — consuming one copy would
say nothing about the others. So (free h) is refused by name and the release operation is on the owner:
(release p h). spec-memory.md's two release points are untouched: (free p) is release point 1 applied to the
pool, and a free-all of the region takes the pool with everything else. release recycles a slot inside storage the
pool still owns, which is not a release of storage at all.
It answers bool rather than (): true if this call released it, false if the handle was already gone. The
generational scheme makes a double release detectable, and that is worth handing to the caller — this is the one
place in the language where freeing something twice is an answer instead of a refusal.
Growth is transactional, because retry re-attempts the same call
StorageExhausted's restart re-attempts the same request, so a failed grow has to leave the pool byte for byte as
it was — including a cap that still agrees with the real block sizes, since the next attempt passes cap as the
allocator's old_size. A pool grows two blocks together (payloads and slot headers), so resizing the first in place
and then failing on the second would leave cap describing neither. So the runtime allocates both, copies, and only
then releases the old pair: nothing is mutated after the last thing that can fail. An allocator without can-free
leaks the first block when the second fails, which is the defined outcome and not a new one — the request failed
because the region is exhausted, and the region is about to be released whole or its ceiling raised.
Two failures, kept apart
A stale handle is an answer: resolve says None and the program carries on. A pool whose allocator was released
traps, through the same epoch check a Vec gets — the slot array went with the storage and there is nothing left
to ask. test/programs/pool-stale-region.flan is that case, and keeping the two apart is the same rule that keeps a
Vec's generation word and its epoch word apart: they answer different questions and must not be conflated.
Two amendments to a frozen spec
Both are places where spec-memory.md describes a handle doing something that cannot answer "gone", which is the one
thing the type exists to do. This amends it: both are deferred, not built.
1. .field and at do not auto-deref a handle. The Places table says x may be a struct, a (Ptr S) or a
(Handle S), and that the two forms auto-deref exactly one pointer or handle level. They auto-deref one pointer
level and nothing else. A (set (.hp h) ...) through a handle has two possible meanings when the entity is dead — trap,
or do nothing — and both are worse than the third option, which is the spec's own worked example: resolve first, match,
and the compiler makes you handle the None. The spec contradicts itself here and the example is the half that is
right.
2. deref is not overloaded on (Handle a). The Generics section says "deref yields a value; resolve yields a
pointer. Both are overloaded on (Ptr a) and (Handle a)." deref is (Ptr a) only. Same reason: deref returns a
value and has nowhere to put "gone".
What this does not have, and one of the gaps is not a pool question
- No
clone. Refused by name. A copied pool would carry the same slot generations, so one handle would resolve in both copies and name two different things — the exact confusion the type removes. A program that wants a second world builds one and inserts into it, and the new handles say they are new. - No pool of an owning element.
(Pool (Vec i32))is refused where(Vec (Vec i32))is refused and for the same reason: the type-erased runtime copies and releases slots bytewise. Recursive teardown arrives withdrop. - A handle is not a map key. For the reason a
Ptris not: hashing an identity is a different operation from hashing what it names, and a stale handle hashes the same as it always did while naming nothing. - Handles compare with
=and not with<.Typesgrew a second predicate,is_equatable, besideis_comparable. Two handles are equal exactly when they name the same slot at the same generation, so a stale handle is never equal to the live one that replaced it — that is worth one integer compare. Ordering them would order a slot index, which is a free-list artefact and means nothing. - A pool passed to a helper is consumed, because a pool is move-only exactly as a
Vecis and there is no borrowing parameter in the language.test/programs/handles.flanis one long function for that reason, and it does not work around it. This is a pre-existing gap and not a pool question: the same sentence is true of everyVecin the tree.
What classes still need
The enumeration primitive is the piece migration was blocked on, and it exists now. What is left is defclass itself
and its runtime shape metadata; migrate-instances, which is a walk over (len p) and (pool-handle p i); generic
functions and method dispatch, whose expensive half is already built and tested (a generic function is an indirection
cell whose body is a dispatch table, and a reload extends the table); and the rule that Enemy@1 stays resolvable for
as long as any instance holds it — the same rule as "nothing is ever dlclosed".
Macros: the compiler dlopens the program
"Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing
that turns that decision into a mechanism rather than an absence: running a macro at compile time means compiling
it and loading it into the compiler's own process. There is nothing to interpret it with and there is not going to
be, so Emit.redefinition → Build.shared → dlopen, the reload primitive the dev loop already runs, is pointed at
the compiler instead of at a running program.
(defmacro name [args] body ...) is one function, [Form] -> Form. One parameter, the slice of forms written at the
call site, which is where variadics come from in a language with no &rest: (len args) is how many were written.
A defmacro is a defn, and there is no Ast.Defmacro
Parse turns (defmacro m [args] body) into (defn m [args [Form]] Form body) and nothing below the parser knows
the word exists. The checker checks it like any function, the backend emits it like any function, Reach.link drops
it from a program that does not call it like any function. The only thing that makes it a macro is that Macro calls
it at compile time instead of the program calling it at run time.
This is also why there is no macro table. Storage was the question the front half deliberately left open, and the
answer is that there is none: the macro set is recomputed by scanning the top level for the word defmacro, which is
the only place it survives, and the compiled artefact is a .so keyed by a digest.
Form, and the three numbers
A macro's parameter and its result are Form, so Form has to exist on the Flan side: a defunion in prelude.ml
mirroring lib/form.ml. It mirrors Form.value and not Form.t — there is no loc field, deliberately. A macro
cannot invent a source location, so the unmarshaller stamps the call site's Loc.t onto every node of what a
macro returns. That is the structural answer to "keep the source location of the call site attached to what a macro
produces", and it is what the queued structured-error work reads.
Case order is tag order, so the list in the prelude is a layout contract and says so. The widest cases are
(Str [s string]) and (List [xs [Form]]); a string and a slice are both %slice = {ptr, i64}, 16 bytes at
align 8. So the image is { i32 tag, [2 x i64] payload }: 24 bytes, align 8, payload at offset 8, and every case
holds its one member at the payload's start, so there is no third offset anywhere in the marshaller.
Those three numbers are asserted, not assumed. test_acceptance.ml's "Form's image format" asks LLVM for each of them
through the same ptrtoint-of-getelementptr-through-null oracle the DWARF offsets go through. Alignment needed a
probe the oracle did not have: the offset of field 1 in { i8, %"Form" } is alignof(Form), because a struct member
sits at the first offset its own alignment allows. Reading [2 x i64] out of the emitted type and concluding 8 would
be asserting the layout against itself, which is the circularity that got a _Static_assert rejected for the FFI.
Nothing aggregate crosses to C
The boundary is void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer) — one thunk per macro, written by
Emit.macro_thunk. The thunk builds the %slice from (args, n) on the LLVM side, calls the macro, and stores the
result through %out.
The correction that matters here is not obvious from the diff. The unions work verified a union's memory layout
against clang; that is a different claim from LLVM's calling convention for an aggregate passed or returned by
value in hand-written IR, which is not promised to be clang's C ABI for the equivalent struct. Memory is the
agreement that actually exists, so pointers and scalars are all that cross. %xfer is the transfer channel every Flan
signature carries; flan_macro_call supplies a zeroed one, because a macro that signals with nothing above it to
handle it aborts inside the compiler, and the channel still has to be a real slot.
Build.macro_module produces a self-contained .so: the runtime linked in, no undefined Flan symbols, -fPIC on
every object including the .ll. Self-contained is what keeps -rdynamic off the compiler's own link. It goes
through clang rather than llc + ld -shared, unlike Build.shared, because there are C objects and a libc to find
— exactly the part of the driver the dev path skips.
OCaml has no dlopen for ELF (Dynlink loads OCaml), so lib/dynload_stubs.c is the whole boundary: dlopen,
dlsym, dlclose, the four-argument call, calloc/free, and a peek/poke family, because OCaml cannot address raw
memory and a Form image is written into it one field at a time.
Quasiquote runs before the walk, and that is not a preference
Quasiquote is a desugaring over Form and nothing more: it becomes form-nil, form-cons per item and
form-append per splice — the prelude's three form-building functions and no fourth. It is pure, it needs nothing
loaded, and Parse.program runs it on the way in, which is what lets the prelude's own macros parse in a process
that has not built a macro module yet.
Running it before the expander's walk is load-bearing. A recursive conditional macro's body contains a
quasiquoted call to itself; with the quasiquote still standing, the walk would see that head and expand it then and
there, against the wrong arguments. Desugared first, that subform is a (Form.Sym {.s "cond"}) and there is no head
left to mistake — so the walk needs no idea that quoting exists.
Nesting levels are counted nowhere: not by the reader, which was written that way deliberately, and not by the desugaring. A quasiquote inside a quasiquote is refused by name. Only a macro that writes a macro wants one.
A call inside a quasiquote is output, not a dependency
This is the distinction that is easy to get wrong, and the first cycle test written for this work got it wrong: it quasiquoted, and it was not a cycle at all.
A macro body that calls another macro outside a quasiquote needs that macro compiled and loaded first, because until then the call is a name nothing defines and the body will not compile. That is a compile-order dependency and it is what makes the pre-pass a fixpoint. A macro body that quasiquotes a call to another macro needs nothing: the call is part of what the macro answers, and the answer is expanded again after it returns.
So there are two different ways expansion fails to terminate, and they are different failures:
- A ring — two macros whose bodies call each other for real. There is no order to compile them in, so it is
refused, naming both.
test/programs/macro-cycle.flan. - A macro that expands into a call to a macro and does not get smaller. That is an ordinary loop, not an ordering
problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site.
test/programs/macro-spin.flan.
The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a macro never sees a call to another macro in what it is handed.
Hygiene is an escape hatch, not a system
Deliberately non-hygienic, Common Lisp's rule and Clojure's, settled in plan.org's open decision 2. A macro that needs
a name of its own calls gensym, which is a prelude function the loaded module runs while it runs. The name is
~g<n>, and ~ is a delimiter now — it opens an unquote — so no symbol the reader can produce contains one and a
gensym cannot collide with a name someone wrote. The counter lives in the loaded module rather than in the compiler,
which is the one place this departs from the original sketch; a module is dlopened once per compiler process, so it is
process-wide in practice.
-linkall, and why the hook could not be installed by hand
Expanding a macro means compiling it, so Macro needs Check, Build and Emit and therefore sits above the
parser it feeds. The join is Parse.expander, a ref that Macro fills in at module initialisation.
Nothing references Macro, so without -linkall the linker drops it from every executable that does not name the
module — bin/main.exe among them — and a program calling a macro fails with an unknown name. Installing by hand at
every entry point was the alternative and it is not viable: lib/session.ml calls Parse.program for C-c C-c and
C-c C-k, and test_session.ml drives the session library in-process rather than through the CLI, so the set of
places that would need an install call is open-ended and a missed one is silent. (library_flags (-linkall)) in
lib/dune is the guarantee instead.
Macro.building is the re-entrancy guard. Build.macro_module goes through Check.program, which parses the
prelude, which calls back into Parse.program — and that would re-enter the expander forever. Nothing is lost by
refusing to expand there: a macro compiled in round n calls only macros compiled in earlier rounds, and those calls
were already expanded before the build was entered.
What it costs
- A build of a program that names no macro: 50ms, unchanged. The pass scans the top level, finds nothing, and no
compiler runs. This is nearly every program, and it is the reason the prelude can grow a
defmacrowithout every build paying a clang driver. - A program that calls one: 310ms the first time, 70ms after. The 240ms is the clang driver; the module is cached
under the object cache and keyed by a digest of the prelude's source plus the file's
defmacroforms, so it is paid once per change rather than once per build. Everyflan buildis a fresh process, which is what makes the on-disk cache rather than a memo the right shape. - A hello-world's binary carries exactly one symbol out of all of this:
flan.gensym-n, eight bytes.Reach.linkdropsunless,form-cons,form-nil,form-append,form-restandgensym, because nothing reachable calls them.
unless is the proof
plan.org milestone 5 says when, unless, until, cond and dotimes are special forms only until macros land.
unless is the first to stop being one — it is now a defmacro in prelude.ml and parse.ml has nothing to say
about it — and it was chosen because it is the one the prelude itself does not use. That matters: the prelude is
compiled into the macro module, so a prelude macro that the prelude's own functions call would need the expander to
compile the thing the expander needs in order to run.
Its coverage is sand.flan, seven calls, compiled through Session in test_session — the in-process path, and the
reason -linkall is not optional. Say plainly what that coverage is not: nothing in test/programs used unless
before this landed, so macro-unless.flan is a test written after the feature. The corpus written before it is
sand.flan and web/examples/control.flan, and both compile unchanged.
break and continue, and the rule that replaced a blanket refusal
Declined once deliberately; NEXT.md's break, and why it was not built records what was settled then and still
holds — dotimes gets it free because it desugars to a While, defer is a non-question because it is
function-scoped and a break does not leave the function, and both are typed Never as exit and return already
are. What stopped it was two things, and both are answered here.
Labels are Odin's, in the head position. (while :outer (< i n) ...), and (break :outer). A keyword there is
unambiguous because a loop condition is never one, so one small label function in parse.ml serves while,
until, dotimes, break and continue and no form has to count its arguments. until and dotimes were not
asked for and cost a line each: until is a While by the time the parser is done with it, and a nested dotimes
scanning a grid is the case the label exists for.
It is not a goto. A label names one of the loops the form is lexically inside — the checker resolves it against exactly those and refuses anything else by name — so control can only leave a loop it is already in. That is Odin's restriction and it is what keeps the feature small: there is no arbitrary target, no forward jump, and nothing to say about scopes being entered.
The in_frames question, ruled on
return is refused inside a handler-bind or a restart-case because those forms push frames and pop them on the
way out, and a return that leaves would strand them on the stack pointing into a frame that is gone. That refusal is
blanket, and correctly so: a return always crosses.
A break crosses only sometimes. A loop written wholly inside a restart-case body has a perfectly good local break,
and refusing it would be refusing the common case for the sake of the uncommon one. So the rule here is relative,
and it is one list rather than a flag: ctx.loops holds the loops this form is inside, innermost first, with a
barrier entry pushed by every construct a jump may not cross. A break resolves by walking outwards; a barrier
reached before the target loop is a refusal that names the construct — break :outer would leave a restart-case,
which it may not. A loop nested inside the construct sits below the barrier and is never affected.
The barriers, and why each is one:
handler-bindandrestart-casebodies — the frames they pushed are popped on the way out, and abrpast the pop leaves a dangling frame. The same reasonreturnis refused, scoped to crossings instead of to everything.- A
restart-caseclause body — it runs after a transfer landed, with the form's frames still to be popped. - A
defer's forms — they are copied into the function's exit paths, where the loop they were written beside is not running. (Unreachable today, becausedeferis already refused in a loop body for its own reason. Written anyway: the rule is about what the forms mean, not about which other refusal happens to fire first.)
A handler clause is not on the list at all. It is lifted into a function of its own with a fresh ctx, so its
loop stack starts empty and nothing in it can name a loop outside it — the refusal falls out of the lifting.
The two rules agree where they overlap, which is the check that the relative one is not weaker: a return is a jump
whose target is always outside every barrier, so the blanket refusal is the special case of this one. in_frames is
left exactly as it was.
continue and the latch
Tast.While is now expr * expr list * expr list — condition, body, and a latch that runs after the body and
before the test. check_dotimes folded its increment onto the end of the body, and a continue branching at the
header would have jumped straight past it: the counter would never advance and the loop would hang. So the step is
the latch, continue branches to the latch block rather than to the header, and a while has an empty latch that
every optimiser folds away. emit_while emits four blocks instead of three.
A labelled continue is the half worth stating outright: (continue :rows) branches to the named loop's latch,
so that loop's counter advances and the rest of its body is skipped along with the rest of every loop inside it. It is
"start the next iteration of :rows", not "skip the rest of this innermost body". loops.flan asserts exactly that —
an outer dotimes whose tail never prints while its counter still runs out.
Tast.Break and Tast.Continue carry a relative depth — how many loops out the target is, innermost first —
rather than a name or an id, because that is exactly what a backend already has. emit keeps one entry per While
it is inside, the same shape and for the same reason as pads, and indexes it. The invariant this rests on: the
checker mints a depth only from its own loop stack, and the two stacks are pushed once per While each. A While
the checker invents — alloc_guard's retry and the file-failure retry — is built directly and contains no jump, so
its emit entry matches nothing.
Nothing in lib/prelude.ml wants either. Every early exit there is a return from the function — bytes=?,
index-of-byte, index-of-bytes, valid-utf8?, bytes->i64 — and a break cannot replace one: it leaves the loop
and the function still has to answer. The loop-with-a-sentinel-flag shape that break exists to remove does not appear
in the prelude. The two compiler-emitted retry loops in check.ml are that shape, and they are the one place it
cannot help: their sentinel is set inside a restart-case body, which is a barrier.
into, which fuses at compile time because it is a macro
(into xs (vec-new i32) (map double) (filter even?))
Source, destination, then any number of transforms — the shape of the into-> macro the author already uses in
Clojure. It reads as a sentence (take this, put it there, doing these) and the variadic tail has to trail anyway,
which is the mechanical reason the transforms cannot sit in the middle.
A macro, and therefore not transducers and not iterators. Transducers compose at run time: function values,
closures, an allocation, and a chain of indirect calls per element. Rust has no transducers either — it has iterators,
which fuse into one loop at compile time through monomorphisation, and that needs generics. A macro reaches the same
place with neither. (map double) expands to (double x) written straight into the loop body, so the function name
is syntax and never a value: no intermediate collection at any step, no closure, no generics, and nothing to
inline. into.flan counts every call to the transform functions, which is the assertion a unit test cannot make — a
chain that built a Vec per stage would pull a different number.
What it gives up is building a transformation at run time and passing it around. That is transducers' actual selling
point, it is the one part that would need run-time machinery, and it is close to useless in a game. Clojure's
:eduction branch is dropped for exactly that reason.
Why the destination is in the form. Every collecting operation here allocates from an explicit allocator, which
is a frozen rule in spec-memory.md. A ->> chain hides where the result goes; naming the destination means the
macro knows its type, emits the right loop, and the rule is honoured by construction. (vec-new i32 a) names an
allocator here as it does anywhere else, because the destination form is written out untouched. This is what ->>
threading over slices in plan.org is replaced by for the collecting cases.
Reductions do not share the form, and that was the open question
(into xs 0 (map cost) (sum)) reads oddly because zero is not a collection, and the oddness is the tell rather than a
matter of taste. The whole reason the destination sits in the form is that the destination is the allocation — it
is what makes the explicit-allocator rule checkable by construction. A seed is not an allocation, so a form that
accepted one would be two forms sharing a spelling, and the destination would have stopped being honest about what it
is. So into collects, and a reducing macro of the same shape is a separate form the day something wants one.
The parts that took a decision
- The destination is a
Vec, becausepushis what fills it. AMapdestination is refused bypushitself, which says push takes a (Vec T) and names the real problem. There is no second lowering and no reason to invent one before something asks. - A source that is already a name is used as it is; anything else is bound to a gensym. Both halves are needed and
neither is cosmetic. Binding is what a source that is a call needs —
(len s)and(at s i)have to be the sames, or the call is made once per element. Not binding a name is what everything else needs: a(Vec T)is move-only, so(let [s v] ...)would hand the caller'svto a binding it cannot see andvwould be dead after anintothat only read it; and a fixed array would be copied into the binding, once perinto.lenandatborrow, so used directly the source is only read. - An owning temporary as the source leaks, and this is the wart.
(into (make-a-vec) ...)binds the result to a name the caller cannot reach and therefore cannot free. The macro cannot know whether the type owns anything. A call in that position should borrow —into.flan's does — and the daydropexists this stops being a question. - One element name throughout, shadowed by each
(map f)stage:(let [x (f x)] ...). Aletbinding's value is checked before its name is bound, so the initialiser reads the outerx— that is the language's rule, not an accident, andbind'sx~2debug suffix exists precisely so a debugger does not lie about which is which. A type-changingmapis the case this most plausibly breaks and it does not: each stage is a fresh slot at the stage's own type, andinto.flanruns ani32source into a(Vec f32)to say so.
What the prelude's macro limits cost, exactly
All four bit, and none blocked anything.
- A macro has no error facility, so the three refusals are calls to names nothing defines:
into-takes-a-source-a-destination-and-transforms,into-transform-is-map-or-filterand…-of-one-function. The report is unknown function at the call site with a expanded from the macro into note under it, which is the right place and the wrong sentence. The bad transform is passed along as an argument so that at least it is named. - A prelude macro may not call another macro, so
into-wrapis a plaindefnand uses only special forms —loop,cond,when,let,if. Aclampor anunlessin there would have put it in the setMacro.reducedrops. - Nested quasiquote is refused, and it was not needed: each wrapper is a single-level quasiquote over a
bodyalready built. - Macros are not importable, which is why
intois in the prelude rather than a library.
into-wrap walks the transforms in reverse, because the chain is built from the inside out: the innermost form is
the push, and each transform wraps what the ones after it produced. That reverse walk with two accumulators is the
first thing in the prelude written as a loop/recur, which landed in the commit before this one.
loop and recur, and why recur is better than tail calls and not only cheaper
There is no TCO anywhere in this compiler — nothing emits a tail call, and plan.org mentions them only as something
the backend choice could control. recur is the answer, and the reason is not that it is cheap. It is checked.
The compiler verifies the call is in the loop body's tail position and turns it into a jump, so writing it in the
wrong place is a compile error at the place it was written. Under silent TCO the same mistake compiles and is a stack
overflow at run time, with a backtrace pointing at whatever ran out of stack rather than at what was wrong. Clojure
adopted recur because the JVM lacks TCO; it turned out to be the better design, and it is the better design here for
the same reason.
What it does not give is mutual recursion between two functions. That needs real tail calls and is out of scope,
and the refusal for a recur outside any loop says so in as many words.
Nothing new reaches the backend
(loop [x 0 acc 1] body ...) is a let over the names, a While whose condition is true, and two jumps. emit.ml
is untouched. That is the whole argument for building this on the labelled break/continue that landed just before
it: the machinery was already there, and the question recur asks — may this jump cross that — is the question
break already answers.
- A loop answers with the value of its body. The result goes into a slot of its own on the way out and is read after the loop, so an accumulator comes back without a mutable local and without a sentinel flag.
- A
Unitbody needs no slot, and aNeverbody needs neither slot nor break — a body every path of which recurs or returns never falls off the end, so there is nothing to break to and nothing to store. Setof aNeverbody is safe, which is the one thing that had to be checked rather than assumed.emitcloses a block at its terminator and drops what follows (instestsf.live), andTast.Setresolves the place before the value, where a local's place is an address with no instruction behind it. So when the body ends in a jump, the store is simply never written.recurrebinds every name at once. The new values go into temporaries and are written afterwards, so(recur y x)swaps. Interleaved writes would givey y, andrecur.flanasserts the swap for exactly that reason.- A move-only accumulator goes round.
(loop [acc (vec-new i32) i 0] ... (recur acc (+ i 1)))is the shape the form exists for, and it is the one the move tracker had to be taught about (below).recur.flancarries aVecround three iterations and answers with it.
tast.ml said a While the checker invents contains no jumps, because the depths would be minted against a stack
it is not on. check_loop's While is the exception, and it is the exception because it is pushed on ctx.loops
like any other: being invented was never the property that mattered, being on the stack is. The comment now says so.
Tail position, as a permission that is withdrawn
The alternative was a pre-pass over the Ast marking tail positions, which would have to enumerate every constructor
and stay in step with the type forever. Instead ctx.tail is read and withdrawn at the top of check, the same
read-and-withdraw defer_ok already does and for the same reason: nothing reached from here inherits it. Three forms
hand it back on, and they are the only three that pass a tail through — the last form of a block, both arms of an
if (including the one-armed when shape, which is how nearly every loop is written), and a match arm. Everything
else is non-tail by construction, and no walk has to list the cases that are not.
The bodies of restart-case and handler-bind are tails semantically and are deliberately not given the permission
here — the barrier below refuses them anyway, and with a better sentence.
loop is a barrier for break and continue
lentry gains Lrecur, carrying the slot and type of each of the loop's names. It is the target a recur resolves
to, by the same walk over the same stack break makes, refusing on the same barriers — handler-bind, restart-case
and a defer's forms — rather than by a second mechanism.
It is also a barrier itself, and that is a restriction added here rather than one inherited. A loop answers with
the value of its body; a break out of one would have to produce that value from somewhere and there is nowhere, and
a continue would re-run the body without rebinding anything. So both are refused, and the message names the fix
(answer with the value, or use a while). A while written inside a loop sits below the entry and keeps its own
perfectly good break, which is the relative rule doing the job it was built for.
Two consequences fall out of this and are worth stating:
looptakes no label, because there is nothing for a label to name. A leading keyword is caught inparse.mlrather than left tobindings, which would have complained that:outerhas no value.- A
recurcan only ever be at depth 0 in practice. A loop body is not a tail position, so norecuris ever written inside a nested loop.recur_targetcounts the depth anyway rather than assuming it, because the count is whatemitindexes.
Two small things the shape forced
A loop binding is a plain name. let's bindings expands a destructuring pattern into several bindings from one
form, and then recur's argument count would no longer be readable off the binding vector. loop_bindings is the
pairs without the patterns, and it refuses a duplicate name.
The move tracker had to be told. in_loop refuses a body that moves a binding declared outside the loop, because
the second iteration would use what the first gave away. A loop's own names are bound before the entry is pushed —
their initial values are evaluated once, outside — so they would have landed in that set, and
(loop [v (vec-new i32)] ...) would have been refused for doing the ordinary thing. recur writes every one of them
on the way round, so the rule is not about them; in_loop takes the loop's own slots and excludes them.
(array 4 rl/Vector2), and the one position with no type slot
[4 T] is the ordinary type spelling and is unchanged. It already works everywhere a type is expected — (defvar points [4 rl/Vector2]), (defn draw [pts [4 rl/Vector2]] ...), a defstruct field, a return. A let binding is
the single position with no type slot, and there the brackets are read as what they are in expression position: an
array literal of two elements, whose second element is a type name nothing declares. So (let [pts [4 rl/Vector2]] ...) failed with unknown name rl/Vector2, which describes the symptom and not the mistake. It cost 32 hand-written
Vector2s in one raylib example.
(array COUNT TYPE) is the answer: a zeroed fixed array, told its count and its element type as plain arguments.
(array 4 rl/Vector2) and the type [4 rl/Vector2] denote the same type, so the constructor is assignable to a
declaration written the other way and either spelling can be the parameter — array-ctor.flan asserts exactly that.
It is a parser form and not a builtin call, because the second argument is a type and there are no types in the
parser's callers. Parse assembles the whole Tarray (len COUNT, TYPE) itself, which is why the count takes a
constant's name for free — len is the same function [n T] goes through — and why a non-type second argument is
refused by the type reader's own message rather than as an unknown name. The checker resolves it and hands back
Tast.Zero, the same node a declaration with no initialiser gets. There is no new backend node and no new type.
(zeroed) was the first proposal and was rejected on how it reads. (zeroed [4 rl/Vector2]) is unambiguous to the
parser — a bracket in argument position could be a type there — but to a person it still looks like a two-element
vector, which is the exact confusion being fixed. zeroed keeps its existing job: the empty value of whatever type the
destination wants, inferred and never written. array is the one that is told.
The prelude's second tier: the functions that return new storage
Everything in the prelude before this was slice-based and allocation-free, and NEXT.md's diagnosis of why was exact:
there was nothing to allocate from when it was written. Vec, Map, an arena and StorageExhausted changed that,
and this is the tier that follows — 24 additions, of which the twelve that matter most return new things
instead of writing into a buffer the caller supplies. The rest fill in the slice family at the element types that
were missing, and one of them is a macro.
Three rules hold across all of it, and they are stated once at the head of the section rather than repeated:
- The result is owned and the caller frees it. Nothing is released at scope exit — not at the end of a
let, not at the end of a function (spec-memory.md). A caller writes(free v), or lets a(free-all a)take the whole region. - The allocator is the context's, and
with-allocatoris the override. This is the one design decision the spec did not settle by itself.(vec-new)and(map-new)take an optional trailing allocator because the checker builds them and can vary their arity; a Flandefncannot, so the choice was an allocator parameter on every signature or none. None —(with-allocator a (join parts sep))is the override, theVecrecords the arena, andfreeandclonenever need it named again. - No
Resultanywhere. Allocation failure signalsStorageExhaustedunderretry, and no allocating operation returns an error, so every signature says what it produces and nothing about how it might fail.
The builder is not a type
Odin's strings.Builder wraps a [dynamic]u8. Here the (Vec u8) already is that and already has push, so
the struct would be a move-only wrapper whose only method is the one it wraps. What was actually missing is appending
a run of bytes, and append! is that.
It takes a (Ptr (Vec u8)) and not a (Vec u8), and that is not style: a Vec parameter moves, so a by-value
builder would be consumed by its first append and refused on the second.
append-i64! and append-f64! are the argument for the whole shape. NEXT.md's "Sharp edges" records that
flan_i64_to_bytes and its neighbours render into one static char scratch[64], so two formatted numbers cannot be
held at once; these copy out of that buffer before returning, so the hazard ends at the call and a builder holds as
many numbers as it likes. strings.flan puts two integers and a float on one line, which is the case that could not
be written before.
split answers a (Vec [u8]), and the owning shape is unrepresentable
The fields are slices of the input. That is not a performance choice — (Vec (Vec u8)) is refused outright
(programs/vec-of-vec.flan, "copies and releases elements bytewise"), so there is no owning shape to have chosen
instead. It follows that the result dies with whatever the input pointed at, which is the same contract trim and
split-next! already have.
The rule is split-on-byte's, unchanged: n separators always yield n+1 fields, so an empty input yields one empty
field and a trailing separator yields a trailing empty one. That is Odin's allocating strings.split and not Odin's
split_by_byte_iterator, which disagree with each other on exactly that input.
Constructing it needed a one-line (defn slices-new [] (Vec [u8]) (vec-new)), because check.ml's vec_new_elem
takes the element type as a single bare symbol and [u8] is not one — so a (Vec [u8]) can only be made where the
context names the type, and a return type is a context while a let is not. Written down in NEXT.md as a compiler
gap rather than worked around silently.
format-f64, and the rounding rule it does not share with printf
f64->bytes is snprintf "%g": six significant digits, exponent notation of its own accord, no precision to pass
it. A frame time of 1/60 comes back 0.0166667 and a score past a million 1.23457e+06.
format-f64 returns a Vec, so it inherits neither that nor the shared scratch buffer, and it renders the integer
part and the fraction through that buffer in strict sequence — the discipline append-i64! exists to make automatic.
It rounds half away from zero at the last digit kept, which is round-f32's rule and the rest of the prelude's.
printf rounds the binary value to nearest-even at the decimal digit, so 0.125 at two places is 0.13 here and
0.12 there. Matching printf would mean pinning a particular libc's answer, and that answer is not the same on every
target anyway.
Three lines in it are the ones a plausible version ships without, and each is a separate test case:
- The carry.
0.999995at five places scales to exactly100000, which is not a fraction — it is the next integer. Without the carry it prints0.100000. - The zero padding. The fraction of
1.005at three places is5, and5is not005; without the pad it prints1.5. - The sign. It belongs to the number, not to its integer part:
-0.5has an integer part of0, andi64->bytesof0carries no sign.
-0.0 prints as 0.00, because the sign test is (< x 0.0), which -0.0 fails. Past 9e18 the integer part does
not fit in an i64 and there are no fractional bits left anyway, so it falls back to %g rather than approximating.
clamp is a macro, and atan2/pow are declares
clamp is the second prelude defmacro after unless, and the reason is the prelude's own objection to wrapping
(min hi (max lo x)) turned around rather than dropped. min and max are builtins at every numeric type and
there are no generics, so a clamp function is one copy per type — clamp-i32, clamp-f32, clamp-i64. A macro
is type-agnostic for free and emits nothing at all. math2.flan makes the same three-word call at i32, i64, u8
and f32 to show it, and counts evaluations to show each argument appears once.
atan2-f32 and pow-f32 inherit sin-f32/cos-f32's caveat in full and not sqrt-f32's: IEEE-754 requires
nothing of atan2f or powf either, so they are the third and fourth places in the prelude where native and wasm32
may differ in the last bit. Every case in math2.flan is therefore a value exact in binary — a quadrant boundary, a
power of two, a perfect square — and the -O0 run is the one that proves the symbols resolve, since at -O2 LLVM
constant-folds a powf of two literals and leaves nothing to link.
What could not be built, and why it is not "no generics"
Four things on NEXT.md's list did not land, and the interesting part is that the reason differs in each case.
Three of the four have since landed — see "map-next!, the one thing a Map could not do", "Function values, with
no capture" and "A prelude function may call a prelude macro" below — and each was fixed by the thing named here
rather than by generics, which is the argument this list was making. The fourth, the path-insensitive dead set, is
still open. Kept as written because the diagnoses are what the later lanes worked from, and one of them turned out to
be wrong in a way worth being able to see: the prelude was reaching the expander.
Mapkeys and values need a map iterator, and there is none.flan_map_len,_get,_put,_has,_clone,_reserve,_freeis the runtime's entire map surface; nothing walks the open-addressed block. One runtime function taking a cursor and one builtin incheck.mlto emit the key and value sizes is the whole job, and none of it is a generics question.map,filter,reduceand a comparator sort are blocked on function values, which is sharper than "no generics" and matters because generics alone would not fix it.Types.Fnexists;check.mlrefuses it with "a function type is not implemented yet — milestone 5"; there is nothing in the language to pass. The concrete answer is the one that shipped:sort-f32!andsort-bytes!are the second and third sorts in the language, andsum-i32/sum-f32already arereducewith the+written in.- The prelude is never macro-expanded, so a prelude function may not call a prelude macro.
Macro.programruns over the file being compiled; the prelude reaches the checker throughCheck.program's prepend. The call resolves to the macro's underlyingdefnand reports an arity error, which is whyformat-f64writes(min 9 (max 0 prec)). - A returned
Vecis a move and the dead set spans the function, so an early(return v)on one branch kills the binding at the foot of another.replace-bytesguards its empty needle with anifrather than awhen/returnfor that reason.
The refusal block is down from eight reasons to four
The list at the foot of prelude.ml used to be one sentence — every entry needed to produce bytes that did not exist
in its input, and there was no allocator. join, concat, split, to-lower, to-upper, repeat and replace
have moved up into the code; string-from-bytes turned out to be the string builtin all along, and
(string (as-slice v)) is the round trip, free precisely because the layouts are identical.
What remains is refused for four different reasons, and is now written that way: pad/center for nothing at all
except that no caller has asked; format/sprintf for variadics of mixed type; map/filter/reduce/sort-by
for function values; map-keys/map-values for the missing iterator.
Tests: programs/strings.flan, programs/format.flan, programs/algorithms.flan, programs/math2.flan, each at
-O2 and -O0, and strings.flan also in a dev build — the one that checks a container's recorded allocator epoch,
so it is what would catch one of these Vecs being used after the arena under it was released.
Function values, with no capture, and why that was the whole blocker
map, filter, reduce and sort-by could not be written, and the previous lane's sharpening of the reason was
right: function values, not generics. Generics alone would not have fixed it — without something to pass there is
nothing to be generic over — and function values alone did fix it, which is the evidence. The prelude gained all four
the same day, without generics, and is still one copy per element type, which is the half generics would remove.
The shape, and why it was not invented here
The compiler has built and called function values internally since the Map landed. A handler-bind clause is lowered
to a function of its own, its address goes into a flan_handler, and the runtime calls it back through
h->fn(condition, xfer); a Map's hash and equality pair is the same arrangement, reached as Odin reaches
Map_Info's two contextless proc fields. The surface feature is that machinery given a name, not a second one
beside it. check_fn is check_handler_bind's clause lifting with the parameters coming from the type instead of
from the condition, and emit's indirect call is the callee expression handed to the same call_through a direct
call already went through.
A bare name is the function
(map double xs)
and not Common Lisp's #'double. This is a Lisp-1 — one top-level namespace, enforced, so a defn and a defvar
cannot share a name — which is exactly what makes the bare name safe to read: there is no second binding of
double it could have meant instead, so the sharp quote would be punctuation answering a question the language does
not ask.
A Types.Fn is one pointer. There is no environment beside it, so the type resolves to ptr and lays out as eight
bytes, and a call through one is byte-for-byte the call a name would have produced — a Flan function's emitted
signature is its parameters followed by the transfer channel whether it was reached by name or by pointer. That is
why a handler established across a fold still catches a signal raised by the function the fold was handed:
programs/fn-values.flan does exactly that, and it is the case that would fail if an indirect call skipped the
guard.
fn literals take their types from the position
Ast.Fn carries parameter names and no types — that is the surface syntax, not an omission — so an fn is
checkable exactly where something says what is wanted. An argument position does, because named_call already
threads the callee's parameter type into each argument; a bare (let [f (fn [x] x)]) does not, and is refused saying
so (programs/fn-no-type.flan). A name already written as a defn goes anywhere, because it carries its own
signature.
What was built, and what was refused by name
Built: a written (Fn [T ...] R) annotation; a defn's name in value position; an fn literal; a call through
a value, both by the name it is bound to and through a computed head; returning one. Four refusal sites, all four
implemented.
Refused, each with its own reason and its own program:
- Capture does not exist (
fn-capture.flan). Anfnis lifted into a function of its own and handed nothing but its parameters; a reference to a local of the enclosing function is refused by name. This is the same refusal a handler clause has always carried, and the two now share one message with the construct's name in it.spec-memory.md's capture cases, and escaping closures with them, stay deferred — deliberately, and this is what keeps a function value a bare code address that cannot outlive anything. - An
fnwith nothing to say what it takes (fn-no-type.flan), above. - A position that would zero one (
fn-in-struct.flan): a struct field, a global, a fixed array's element,(zeroed). ZII fills an omitted field with all-bytes-zero, and a zeroed function value is a null pointer, which is the one kind of zero that is not a value the type can have — every other type's zero is one:0,false, an empty slice,None, a union's first case. A parameter, a return type and aletbinding are not on the list because none of them is ever conjured, and an(Option (Fn ...))is not either, because aNone's tag is what nobody may look past. Nor are a(Vec (Fn ...))or aMapwith function values: the Vec runtime never zeroes past its length andflan_map_alloczeroes only the hash run, so neither conjures an element nobody pushed or put. A function value as a map key is refused already, byTypes.keyable— hashing an address is a different operation from hashing what it points at. - A foreign function's address (
fn-extern.flan). A Flan function's signature ends with the transfer channel and a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the raw symbol knows nothing about. Wrap it in adefnand pass that.
Fnval, and the one thing a dev build cannot do
Tast.FnAddr had two fnref cases and now has three. Flanfn and Rtfn are the compiler's own uses and want the
symbol, always — a lifted handler clause and a hash pair have no indirection cell to load from. Fnval is a
function value someone wrote, and in a dev build it is the cell's contents rather than the symbol, so a value taken
after a redefinition is the new body. Splitting the case rather than overloading Flanfn is what keeps that true
without breaking the two paths that must not take it.
What that does not give: a value taken before a redefinition and called after it is still the old body. Once the address is in a slot there is nothing left to re-resolve, and the honest fix is a trampoline per function, which is a cost every program would pay for a case no one has hit. Named here rather than papered over.
The two lifted-function name sequences are counted per kind — fn/OWNER/N and handler/OWNER/N/TYPE —
rather than off one list. Sharing a counter would rename every fn in a function the moment a handler-bind was
added above one, which is a rename for a body that did not change, in exactly the names a redefinition module emits.
Tast.CallPtr is its own node for the same kind of reason. Everything that walks this IR treats Call's string as a
link-time edge — Reach roots the callee, Dev finds the cell, Emit may load it — and none of those are
questions an indirect call can answer. Reach gains the Fnval edge, and that edge is load-bearing: a name used as
a value is never a Call, so without it the one function a program passes to map is the one function the link
drops.
A user-written allocator: still refused, and now for two different reasons
NEXT.md said it needed "a defn's name in value position". It has that now, and it is still two things short, neither of them a function-value question:
- The runtime calls
a->proc(a, mode, p, old_size, size, align)— six C arguments and no transfer channel — and every Flan function value's signature ends with one. It is the same mismatch a foreign function's address is refused for, pointing the other way. Allocatoris opaque and pointer-width, so there is nowhere for a program to put theflan_allocatorthat pointer would have to point at.
The refusal message says both, and programs/user-allocator.flan is the row that holds it. (arena-new ...) over a
backing buffer remains the parameterised allocator that does exist.
The prelude's four
map-i32!/map-f32!, filter-i32/filter-f32, reduce-i32/reduce-f32 and sort-i32-by!/sort-f32-by!. Two
rules, both inherited rather than invented: the in-place ones write back into the slice they were handed, because a
slice is non-owning and transforming a thing you already own should not allocate; and filter allocates and the
caller frees, like everything in the building tier.
A map that changes the element type is the one shape that did not come with them — it is one copy per ordered
pair of types rather than per type, which is where a per-type family stops being honest. That entry is what is left
in prelude.ml's refusal block where map, filter, reduce, sort-by used to be, and its reason is generics.
map-next!, the one thing a Map could not do
flan_map_len, _get, _put, _has, _clone, _reserve and _free was the runtime's entire map surface, and
every one of them addresses a single entry by hashing it. Nothing walked the block, so a map's keys and its values
could not be read out at all — the only item on the second tier's list that was blocked on nothing but a missing
function.
flan_map_next is that function and map-next! is the builtin over it.
(let [cur (i64 0) k 0 v 0]
(while (map-next! m (addr cur) (addr k) (addr v))
...))
The cursor is a slot index the caller owns, and there is no iterator struct because there is nothing for one to
hold. A map has no tombstones — removal is deferred (spec-memory.md) — so a slot is either empty or occupied and the
position is the whole of the state. The cursor starts at 0, comes back one past the entry just answered, and is left
at cap by the call that answers false, so a spent cursor keeps answering false rather than wrapping.
Three out-pointers and not a returned pair, because there are no tuples. An (Option K) would answer half an
entry and make the value cost a second hash of the key just handed back. The ! is the cursor: it is the argument
that is written through on the way out.
It is the one map entry point that carries neither a hash nor an equality function. Walking asks nothing about a key. The two sizes are still there, because the runtime is type-erased and the block geometry is computed from them.
The layout, restated, because it is the thing to get wrong here. data is one allocation laid out
keys | values | hashes | scratch, each run cell-packed to a cache line — the arrangement the Valgrind lane described
while explaining why a probe overrun is not observable. A key is reached through flan_cell_at and never as
ks + i * ksize. The hashes are the exception flan_map_clone already relies on: an 8-byte element packs 8 to a
64-byte cell with nothing left over, so g.hs[i] is the right index and a flat one.
Order is block order, which is the hash's order and not the insertion's, and it changes when the map grows.
programs/map-iter.flan is therefore written entirely in sums, counts and lengths — every claim in it is order-free,
which is the contract rather than a weakness of the test. A caller that wants an order sorts what it collected. The
cases that would catch a real mistake: a string key with a struct value, whose two runs have different element sizes
and different packing and so would break if one geometry were used for both; and a 500-entry map, which is several
grows past the minimum and walks a block whose layout has nothing to do with how the entries went in.
map-keys and map-values are still refused, and the reason changed. The refusal block at the foot of
prelude.ml said "a Map iterator"; that is wrong now. What a prelude defn cannot write is
(defn map-keys [m {K V}] (Vec K)) — it has to name its types and there is no K. That is generics. The loop is
three lines at the call site, where K is known, and that is where it stays.
A prelude function may call a prelude macro, and why the fix was not ordering
The handoff said the prelude is never macro-expanded — Macro.program runs over the file being compiled and the
prelude arrives later through Check.program's prepend — and that the fix was to move the prepend before expansion.
Both halves of that are wrong, and the measurement is one command.
Put (clamp prec 0 9) back into format-f64, print names and List.length extra on entry to Macro.compile, and
compile any program that calls a macro. The compiler prints names=[clamp,unless] extra=0 and then the arity error
at <prelude>:1103. So compile was entered: the prelude does reach the expander. The error is raised by the
Check.program inside compile, where building is true and expansion is off.
That is the real shape, and it is a cycle, not an ordering: a macro module is compiled from the prelude, so a prelude function that calls a macro would have to be compiled into the very module that expands it. Moving the prepend earlier changes which pass sees the prelude first and leaves the cycle exactly where it was.
Two things break it, and the second is the one that matters.
The prelude's macros are dropped from mine. Macro.program collected every defmacro in the forms it was given
and handed them back as extra — the forms a macro module is built in addition to the prelude. When the forms it
was given are the prelude, that is the prelude's macros declared twice, refused as a redefinition. They are already
in prelude, which is where the module gets them from.
Macro.reduce: for that one build, the prelude is smaller. Every defn that names a macro is dropped, and then
every defn that names a dropped one, to a fixpoint — a function calling something unbuildable is as unbuildable as
the thing it calls. Prelude.bootstrap is the hook, a ref rather than a parameter because the reader is
Check.program and it cannot be told.
Only defns are dropped: the functions that survive still mention the prelude's types, and a reduced prelude missing
them would not check. A defstruct, defunion, defalias, defenum or defvar therefore stays whatever it names.
There used to be a sharper reason — Parse.prelude_types memoised the prelude's type names for the parser's
return-type guess, and it can be forced for the first time inside a bootstrap build, so a reduced set cached there
would have been wrong for every compile afterwards. That set is gone with the guess; see The return type is the slot
below.
The one restriction that stays, and now names itself. A prelude macro may not call a macro — the module that
expands it is compiled from the prelude, so there is no earlier module for its own call to have been expanded by.
That was already recorded and accepted; what it used to do was fail as an unknown name somewhere inside a clang
driver. reduce checks it directly and refuses with the macro's name and the reason.
format-f64 is written (clamp prec 0 9) now, which is the living proof and also the only place in the prelude that
exercises it. The expansion is (min 9 (max 0 prec)), so nothing about the output moved — the point is that the call
compiles at all.
What it costs. The prelude names a macro now, so Macro.program's short-circuit — the reason a build using no
macro pays nothing — no longer fires for the prelude, and every Check.program dlopens a macro module. The module is
disk-cached under a digest of the prelude source with an empty extra, so it is one .so shared by every build and
every process; dune test is unchanged at 20 seconds. The first build after a prelude edit pays one clang driver.
What was not done. The two expansions are still separate — the prelude is expanded against the prelude's macros,
the file against the prelude's plus its own. That is not a gap, and it is worth stating so the next lane does not
"fix" it: a file macro is never visible to the prelude, and a prelude macro is already visible to the file, so the two
passes cannot disagree. Expanding them together would buy one fewer dlopen and nothing else.
defer may be written in a let
The whole of this project's resource-cleanup answer, and NEXT.md records drop and a with-cleanup form as both
considered and rejected before it.
defer is a compile-time construct: the cleanup is copied into every exit path of the function. That is why a loop
body and a branch stay refused — a loop body's would fire once at function exit rather than once per iteration, and a
branch would have to express "maybe registered", which a form copied into every exit path or into none cannot say.
A let is neither. It is not a frame here: its bindings are function slots like any other and nothing is released at
scope exit, so a let at the top level of a function body has exactly the function's extent and a defer written in
it always registers. It was refused for a reason that does not apply to it.
The rule, stated precisely, because "top level of a function body" is easy to get wrong: a form at the top level of the
body may carry one, and so may a form in the body of a let that itself may — to any depth. A let inside a while
or an if has the loop's extent or the arm's, and inherits the refusal rather than the permission.
The implementation detail worth keeping: the permission is granted again before every form of a body, never once
around the body. check withdraws it as it starts, so granting it once would let the first defer through and refuse
the second — and two resources acquired in one let is the case this exists for. defer-let.flan covers exactly that,
along with nesting, interleaved registration order across the let boundary, and an early return.
This amends spec-memory.md, which states under "When storage is released" that (defer (free v)) for a
let-bound v is "not expressible today" and that no idiom in the spec may depend on it. It is expressible now.
do at the top level of a body has the same extent argument and is deliberately not included: nothing asked for it,
and the rule is easier to state and to trust with one construct in it.
Assets are baked in, and the reason it is a compiler feature
NEXT.md decision 1. (embed "brush.png") is a [u8], (embed "brush.png" string) is a string, and
(embed-dir "assets") is a [n EmbedFile] sorted by name. Odin's #load and #load_directory are the model
(src/parser.cpp, and check_load_directive / check_load_directory_directive in src/check_builtin.cpp); Odin's
# is not imported, because an s-expression language already has a head position for a name and these resolve as
ordinary named calls exactly the way vec-new and heap-allocator do.
The reason for this shape rather than a build flag is the one that decided it. It is a compiler feature, so it
needs no linker arguments and no per-target packaging, and it works identically on desktop and web. That matters more
here than it does for Odin, because Load hands out lflags only to a directory package and main is not exported,
so a program can never be a package: the single file doing (rl/load-texture "brush.png") is structurally the one file
with no link channel at all. The web lane found that hole and did not invent a flag for it. Embedding has no such
hole, because there is nothing to tell the linker.
It costs nothing at run time. The bytes reach the program as a Tast.Str node typed [u8], which emit.ml turns
into the same private unnamed_addr constant every string literal already becomes, and its escape is byte-exact
across the whole 0–255 range, so a PNG survives the round trip through the .ll. Bound with defconst at top level an
embed-dir is an LLVM constant outright, through emit.ml's const.
A Str node typed [u8], not a Bytes prim over a string. This is the one non-obvious choice. Bytes is
identity — emit.ml lowers Types.String and Types.Slice _ to the same %slice — but wrapping the literal in a prim
makes the node non-constant, and const then refuses an embed-dir in a defconst with a global's value must be a
compile-time constant. Both of emit.ml's string emitters take the bytes and ignore the node's type, so it is the same
constant either way and this one is a constant a global can hold.
Two spellings, not one form that changes type with its context. Odin threads a type_hint everywhere and can
afford #load("p") to mean a string here and a []u8 there. With structural equality, no implicit widening and no
coercion anywhere, the same text meaning two types would be a wart, so string is written down when it is wanted. The
site's expectation is a fallback only and nothing depends on it.
The path is a literal and resolves relative to the file the form is written in. Both are Odin's rules and for
Odin's reasons: the bytes must be in hand before any value exists, which is what makes the result free; and a path
relative to the compiler's working directory would make a package's assets depend on where flan was invoked from,
which cannot be right. A missing file is a compile error naming it, never an empty embed — an asset silently absent is
exactly the quiet wrongness this removes. An empty directory is not that case and embeds cleanly as [0 EmbedFile].
The directory lookup is a linear scan, and that is the chosen answer rather than the fallback one. embed-find is
an ordinary prelude function over a [EmbedFile]. A directory embed is tens of entries whose names sit in cache-warm
.rodata; a compile-time perfect hash would be a build-time map with its own failure modes that nothing has asked for,
and sort-and-bisect is the next step if a program ever embeds thousands of files — it would not change the type. It
takes a slice rather than the array, because an array's length is part of its type and there are no generics, so
the call reads (embed-find (slice assets 0 (len assets)) "brush.png"). Entries are sorted by name because readdir
order is filesystem-dependent and an unsorted embed would make two builds of identical sources emit different .ll.
Non-recursive, files only — Odin again.
The sharp edge, inherited and not widened. The slice points into .rodata, so a store through it segfaults at
-O0 and is deleted as undefined behaviour at -O2 — the same trap the prelude's ASCII-case note measures for
(bytes "Hi"), and the same one NEXT.md tracks as "writing through a string literal". Nothing here makes it worse and
nothing here fixes it; provenance is what would. To get a mutable copy, clone the bytes into a Vec. It is worth
saying loudly because an embedded asset is precisely the thing someone will try to decode in place.
What this does not do. sand.flan still calls (rl/load-texture "brush.png"), which hands raylib a path for
raylib to open. Pointing raylib at embedded bytes needs LoadImageFromMemory and LoadTextureFromImage in place of
LoadTexture — a raylib binding question, not an embedding one — so the flagship program is not yet asset-free on the
web. The mechanism it needs is in.
slurp, barf, and the two ways they fail
NEXT.md decisions 2 and 5. (slurp path) and (slurp path allocator) read a whole file into a (Vec u8);
(barf path bytes) writes one.
slurp waited for Vec because its result has no length until the file is read, and it obeys spec-memory.md's
rule without an exception: no allocating operation returns an error. There is no Result here, no out-parameter and
no error code — slurp's type is (Vec u8) and barf's is ().
Two failures, two conditions, and the guards nest rather than merge. Allocation failure is StorageExhausted under
retry, unchanged and reused. File failure is FileError {.path .op .reason} under retry and use-value. They stay
apart because they ask two different answerable questions: the handler that grows an arena is not the handler that
supplies another path, and collapsing them would make one handler guess which it was looking at. file_guard in
check.ml is alloc_guard's shape built from the same nodes — a while, a restart-case and an error — so the
backend learns nothing new.
The restarts are Common Lisp's pair for a file-error. retry for "the file may be there now"; use-value [p string] for "try this other path". use-value is the first restart clause the compiler itself emits with a
parameter — typed restarts landed the same session — and its parameter is the path slot the attempt reads, so the
clause body is empty. emit.ml's bind_params stores the invoker's argument into the slot, the clause falls through,
and the loop re-attempts against the new path. Everything is inside that loop, so a use-value naming a different file
re-measures it and re-allocates for its size; the Vec is freed at the top of each turn, which is why a retry does
not leak, and freeing a Vec that never allocated is a no-op.
The two forms resolve paths by opposite rules, and it is worth saying in one place. An embed path is resolved at
compile time relative to the file the form is written in. A slurp or barf path is resolved at run time by the
host, against the process's working directory — these are ordinary values, and one can arrive from argv or from a
use-value restart. test/programs/slurp.flan reads "programs/assets/a.txt" only because the suite runs from
_build/default/test. Two forms in one section with opposite rules is exactly where someone gets bitten.
The break loop can be offered this restart and cannot fill it in. That is not new behaviour, only a new way to
reach it: a break loop chooses a restart by position and has nothing to supply a parameter with, and emit.ml already
emits a flan_restart_unarmed guard on every clause that takes one, so taking it refuses with the reason rather than
running the clause on a zeroed buffer. slurp's use-value is simply the first such clause the compiler emits —
alloc_guard's retry takes no parameters — and it rides the same path a hand-written one does. The acceptance suite
asserts the guard is on the emitted IR for both.
On the web, barf signals — every time, with the path in the condition. This is the decision worth restating,
because two more obvious answers are both wrong here. A build-time refusal is unusable: Flan has no conditional
compilation, nothing in parse.ml or check.ml reads the target, so "isolate this to desktop" is not expressible in
source and the refusal would have nowhere to be silenced. A silent no-op is worse than either, because that is how
a save file disappears with nothing said. So the program gets a condition and decides — which is the language having
something Odin does not. Odin stubs its whole file API on js/wasm to .Unsupported (core/os/file_js.odin) so that
importing core:os "panics cleanly", and a panic is not a decision. The restriction was taken; the mechanism was
not.
Nothing in the compiler reads the target to do this. The refusal is one #ifdef __EMSCRIPTEN__ in flan_rt.c, which
is where the host ABI is already implemented twice. slurp keeps working on the web — it compiles, runs, and reports
a missing file honestly — and the bytes a web program actually wants come from an embed.
test/programs/web-files.flan is one source built for both targets, and test/test_web.ml runs it under node
rather than asserting the artifact's shape: an artifact-shape assertion would say nothing about the thing the decision
bought. The test checks that the refusal is printed with its path and reason, and that the desktop's success line is
absent — a silent no-op would have taken that branch.
What the host ABI grew by, and why that much
plan.org names the filesystem as the #1 portability risk — "pack assets, one abstraction, never touch paths" — so the widening is written down rather than assumed. It is three calls and one reader:
| Call | What it does |
|---|---|
flan_file_size(path, n, &out) |
how many bytes are there |
flan_file_read(path, n, buf, cap, &got) |
fill a buffer the caller owns |
flan_file_write(path, n, buf, len) |
write a whole file |
flan_file_fail_reason() |
which of the four reasons it was |
They are POSIX-shaped and Vec-ignorant: no handle crosses the boundary, nothing is held between calls, and each
takes a path and answers 1/0 the way every allocator entry point already does. flan_slurp_into — the part that knows
what a Vec is — is runtime glue on this side of the ABI, not a fourth host call, so a second target implements
three functions and inherits the rest. The reason is a global rather than an out-parameter for the same reason
flan_alloc_fail_bytes is: the condition is a value struct on the failing frame's stack with fixed numeric fields and
no rendered message.
These do touch paths, which is the widening plan.org warned about, and decision 2 took it knowingly. embed is the
half that does not: it needs no host ABI at all, so "pack assets" remains the answer for anything known at build time
and slurp is for bytes that genuinely are not.
FileError is one type with a reason field rather than a family, because conditions have no hierarchy today
(spec-conditions.md §1) and a family would need one handler clause per member to say "any file error". NEXT.md decision
4's parent link is the answer to that and is not built; when it is, these reasons can become types without any call
site changing.
One thing slurp and barf reveal that is not theirs to fix. A handler that wants "try to save, and carry on if
you cannot" has nowhere to go: error is diverging (spec-conditions.md §2), a handler returning normally has not
answered it, and neither retry nor use-value means give up. web-files.flan calls exit for that reason. A
continue-style restart — or handler-case, which unwinds — is what the case wants, and neither exists.
Where build time goes
flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object cache is in, and it is now ~110ms:
| Step | Cost |
|---|---|
| frontend: read → parse → load → check → emit | <10ms, below the timer |
clang on the .ll |
60ms — llc does the same codegen in 20ms |
clang on flan_rt.c |
40ms — now cached, paid once |
| link | 20ms |
Every C translation unit a build needs — the host shim and each package's shim — goes through Build.compile_c, which
compiles to a .o under $TMPDIR/flan-objcache and reuses it. The key is a digest of the source text, the compiler
(its path, size and mtime, so an upgrade invalidates without paying a clang --version subprocess per build),
opts.opt and opts.target. The opt level has to be in there: the acceptance table builds the same programs at -O0
and -O2, and an -O2 object must not serve an -O0 build. The object is written to a temporary name and renamed
into place, so two concurrent builds cannot see a half-written one.
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
the cost. The cache is keyed by content, so it never needs invalidating by hand — rm -rf on the directory is only ever
a disk-space decision.
The other cheap win is still open: skip the clang driver for the .ll (llc + link directly), worth another ~40ms. It
is a subset of the dev path's machinery. Check llc's major version against clang's before relying on it — the emitted
IR text is currently absorbed by the driver behind -Wno-override-module, and a version mismatch surfaces as IR parse
errors.
The order it was built in, and why that order
Every item here is done; it is kept because the sequencing is the part worth remembering. Decided in conversation: wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and the dev loop is the thesis of the project, so it came first. Staged so each step was runnable on its own — the failure mode being to build a daemon and a protocol before knowing whether the reload primitive worked.
The reload primitive, measured.Done —Emit.redefinition,Build.shared,test/reload_host.c, ~19ms. See the section above.Indirection cells.Done —Build.opts.dev/flan build --dev,flan_reload_install,runtime/flan_dev.cfor names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above.The agent, in C.Done —vendor/agent/, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above.The daemonand5. the Emacs client— both done, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the sameSessionif something else ever needs to talk to it.
One decision left to settle before step 2, because both change codegen and are painful to retrofit:
Do cells cover globals, or only functions?Settled by step 1: functions only. A redefinition module declares every globalexternal, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding adefvarto a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.- What is a redefinition unit — one function, or a file? A file is much easier to make correct and is what
load-filewants anyway; one function is whatC-c C-cwants and is where the 16ms number comes from.
Deferred until after the dev loop:
wasm32.Done, with one glued joint.flan build --target=wasm32-wasiproduces a module, andtest/programs/sand-headless.flanprints15595743031174623232under it — the same hash as native, byte for byte, at-O2and at-O0. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared across targets, and it compares equal.values.flanandmachine.flanrun there too, which is where a 32-bit pointer would have shown. The acceptance table runs all four, and skips them by probing — it builds the smallest program and runs it — rather than by looking for a binary on PATH.
Three things this cost that were not in the old note:
- The entry point is not
main. wasi-libc's start code calls__main_argc_argv; clang renames C's argc/argvmainto that, and the.llEmitwrites says@mainliterally. The link succeeds and the program traps on a signature-mismatched weak stub.Build.wasm_main_sourceis a two-line C shim that bridges it, and the__asm__("main")label in it is load-bearing: spelling the calleemainmakes clang rename that too and the shim becomes an infinite self-call. - The target has to reach the C compiles, not just the link.
flan_rt.cincludes<stdio.h>; without--sysrootit never gets that far.target_flagsis computed once and passed to both, and the whole flag list — not just the triple — is in the object cache key, so repointing a sysroot cannot serve a stale.o. - Fedora's sysroot is one level deeper than wasi-sdk's:
include/wasm32-wasi/stdio.h, notinclude/stdio.h. Both shapes count.
The glued joint, and the one thing this contradicts in the old note. The old note said the builtins archive has to
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
libcompiler_rt.a, and dropping that in as libclang_rt.builtins.a links and runs. It is a different triple
(wasm32-unknown-emscripten) built by a different clang (22 against Fedora's 20), so it is substituting, and wasi-sdk
is still the proper article. build.ml looks for FLAN_WASM_BUILTINS, then /opt/wasi-sdk/..., then emscripten's
beside emcc on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real include symlinked in.
The runtime is Node. No wasmtime and no wasmer on this machine; test/wasm-run.mjs is twenty lines of
node:wasi and the table prefers wasmtime or wasmer if either appears. --no-warnings, because node:wasi writes
an ExperimentalWarning to stderr on every run and the harness compares combined output.
Refused by name, not half-supported: --dev with a wasm target (the reload path is dlopen), Build.shared with
one (same reason), and flan run --target= (a .wasm is not something this host execs — build it and point a runtime
at it).
Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. Done — and it is a third
target, not a mode of this one. wasi-sdk is right for the headless table and was never going to be right for the game
build. See the next section.
The browser is the third target
flan build --target=web produces a page, its JS and a .wasm, and a raylib example opens in a browser from source
that was not touched. The two wasm targets share the word and almost nothing else, so is_wasi and is_web are
separate predicates and is_wasm is their union — the union is exactly the set of facts about the machine (32-bit
pointers, no dlopen), which is what the refusals are about, and nothing else is shared.
The compiler is emcc, not clang, and that is the whole of the sysroot story. Everything the wasi target has to
find by hand — a sysroot, a builtins archive, a shadow resource directory, the __main_argc_argv shim — is what emcc
is. target_flags for web is the empty list; the only thing checked is that emcc exists, refused by name where the
reason can say so. The one fact that had to be true for any of this: emcc takes a .ll on its command line, which
it does, so Emit's output needs no change and the IR stays target-independent. The object cache keys on the compiler
binary's path, size and mtime as it always did — now of whichever compiler the target uses, so an emcc flan_rt.o
and a clang one cannot collide.
The main loop: -sASYNCIFY, not emscripten_set_main_loop. The older note above had the browser fact right —
it cannot be blocked — and drew the wrong conclusion from it. emscripten_set_main_loop wants the loop body as a
callback, so every one of the eleven examples that writes
(until (rl/window-should-close?) ...)
would have to be split by hand into an init and a tick, and the web program would stop being the native program.
Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is
built for precisely that: WindowShouldClose() on PLATFORM_WEB is an emscripten_sleep(16) that then returns false
(raylib 5.5, platforms/rcore_web.c, read rather than assumed). So the loop yields once a frame at a call it already
makes, and no example changed a character. The price is real and is paid by every web build: asyncify instruments
the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a
question Build can answer, and a flag set that varies per program is a cache key that varies per program.
link lines can be addressed to a target. vendor/raylib/link named libraylib.so.550, which exists on the host
and nowhere else. A line may now carry @native, @wasi or @web, an untagged line applies everywhere — which is
what every existing link file already is — and ${NAME} expands from the environment. The selection happens in
Build and not in Load, which is where the file is read, because Load resolves imports before a target is
chosen: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings.
Load's part in this is to pass the lines through untouched, which it already did.
raylib for the browser is built, not installed. No emscripten port provides it (emcc --show-ports: there is
contrib.glfw3 and no raylib), so vendor/raylib/build-web.sh clones raylib at the 5.5 tag — the one whose
.so.550 the host links, because raylib.flan carries raylib's struct layouts and enum values and two targets built
from different raylibs would disagree about them in silence — and compiles the seven modules with
-DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2 into one archive under vendor/raylib/web/ (gitignored). rglfw.c is not
among them: the web platform uses emscripten's own GLFW port, which is why link carries @web -sUSE_GLFW=3. No
headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.
The HTML shell is a string in Build, not a file in the tree, for the same reason Runtime_src is: it has to be
wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas
and a Module.print that puts stdout on the page; FLAN_WEB_SHELL replaces it. --shell-file is passed only when the
output is a .html, because emcc accepts and ignores it otherwise.
Refused by name, inherited whole: --dev, --debug, Build.shared and flan run --target= are refused for
web exactly as for wasm32, each naming web rather than wasm32 in the message. --sanitize is refused too, but
the web half of that refusal is weaker than the wasi half and says so: emscripten does ship an ASan, and nothing here
has ever run it. A sanitizer that has never been run is one whose silence means nothing.
What the test can honestly check. test/test_web.ml is headless and permanently so. It probes — emscripten may not
be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it
asserts: the three files exist, the module starts with \0asm, the page references its own JS and carries the canvas,
and node runs the emitted JS and gets ok. For raylib it builds core-basic-window.flan unchanged and then reads the
module for the two things that would be false if the mechanism were wrong: an asyncify_start_unwind export, and a
glViewport import that can only have come from raylib's web platform. Import and export names are plain strings in
the binary, so this needs no wasm reader.
sand.flan in a browser
The flagship program builds for the browser and the artifact opens. Three things were between it and the target,
and none of them was the #include the earlier note named.
The brush was a path, and a path is what cannot work. (rl/load-texture "brush.png") hands raylib a filename to
open; a bare relative path has no meaning where there is no filesystem, so raylib would have opened nothing and the
cursor would simply have been missing. It is (embed "brush.png") now, decoded by a new binding —
LoadImageFromMemory, which completes the chain embedded bytes -> Image -> Texture2D that LoadTextureFromImage
already had the other half of. The declaration is (Ptr u8) plus an explicit i32 count, because shim.ml refuses a
slice parameter and its refusal says exactly that; the Flan wrapper beside it takes the slice apart, which is the
idiom collision-point-poly? and load-font-ex already established. One decode serves both textures now: the
unflipped upload first, then ImageFlipHorizontal in place, then the mirrored upload — the order is load-bearing,
and if the two badges look the same it was swapped.
load-texture and load-image lose their only call site in this repository by this change. That is deliberate rather
than an accident of editing: a path-based load is the one shape the browser cannot have, and the bindings stay for the
desktop programs that will want them.
The embedded path resolves relative to the file the form is written in, which is why
test/programs/sand-headless.flan still works: it reaches sand.flan through ../../ out of a sandboxed _build,
and the PNG is found beside sand.flan and not beside the working directory. test/dune therefore lists brush.png
as a dependency of every stanza that builds sand — an embed is read by the checker, so it is a build input and not a
run-time one. The same fact reached test_session's C-c C-k case, which re-evaluates sand.flan's whole text: it now
passes ~origin, which is the buffer path both editor paths already send, because the default <eval> origin would
resolve the embed against the working directory instead.
A package's C may be addressed to one target, the way a link line already could. Load collects a package's
.c files by listing the directory, and there is nowhere in a directory listing to put a tag except the name, so the
tag goes there, before the extension:
vendor/agent/flan_agent.c compiled everywhere, unless displaced
vendor/agent/flan_agent.web.c compiled for the browser, and displaces the above
One rule: a tagged file is compiled only on its own target, and there it replaces the untagged file of the same base
name. Untagged is the default and every existing package is untagged, so nothing that did not opt in changed.
Replacement rather than the pure tagging Go's _windows.go and Odin's file_js.odin use, and the difference is the
point — pure tagging would mean renaming flan_agent.c to flan_agent.native.c to teach the package about a target
it had never heard of, and this way a package gains a target by gaining a file. The selection is in Build and not in
Load, for the reason select_lflags gives: Load resolves imports before a target is chosen.
The dev agent on the web is a no-op, and it is not the barf decision being contradicted. The compile error that
led here — struct timeval incomplete, because emscripten's headers do not pull <sys/time.h> in transitively — is
the surface. Underneath: the agent is a socket server and a browser has no sockets. Adding the include produces an
agent that compiles, links, starts and can never accept a connection.
Refusing vendor:agent on a web target was the other candidate and is ruled out by arithmetic, not taste. Flan has no
conditional compilation, sand.flan calls (agent/start ...) unconditionally, and Reach cannot prune a package
something reachable calls into — so a build-time refusal means the flagship program does not build for the browser at
all without being edited into a second program. A refusal is only honest when the caller has a way to not ask.
The two decisions look contradictory and are not, and the difference is what the caller loses. barf is asked to
make something durable; a no-op returns success to a program that now believes the bytes are on disk, and the loss is
real, is the user's, and is discovered later or never. The agent is asked to accept redefinitions from an editor; on
the web there is no editor, no socket and no session — --dev is refused by name on every wasm target, so a web build
has no cells to install a redefinition into even if one arrived. Nothing is lost because there was never anything
there. sand.flan already says the same about a native release build, at the call site: "Building without --dev
is fine — nothing has cells to install into, so a module is refused on the listener thread and the loop never
notices." A web build reaches that outcome by a shorter route. start returns -1, which is what flan_agent.c
returns for a path it cannot bind; poll and wait return 0, which is what the native build returns on every frame
nothing arrived on. The whole argument is written at the top of vendor/agent/flan_agent.web.c, where the next reader
will meet it.
Building it and opening it
The raylib archive is built once and is not in the tree. From the repository root:
sh vendor/raylib/build-web.sh # clones raylib 5.5 and compiles it with emcc
export FLAN_RAYLIB_WEB=$PWD/vendor/raylib/web/libraylib-5.5.a
build-web.sh prints that export line itself. Then:
flan build sand.flan --target=web -o sand.html
The output must be named .html. --shell-file is passed only when it is, because emcc accepts and ignores it
otherwise — so -o sand produces a module with no shell, no canvas, and a page that looks like it built fine and
paints nothing. Three files land beside it, in whatever directory -o names: sand.html, sand.js, sand.wasm.
A file:// URL will not work. The page fetches sand.wasm, and a browser refuses that from the filesystem. Serve
the directory holding the three files:
python3 -m http.server 8000
and open http://localhost:8000/sand.html. Left mouse paints; the keys are the ones the native build has.
What only a human opening it can settle
Verified headlessly, by test/test_web.ml: the three files exist, the module carries asyncify_start_unwind and
glViewport, and brush.png's own bytes are in the module, whole — the assertion that keeps the embed from
rotting. Not IHDR: stb_image, linked in from raylib, carries that string itself, so an IHDR check would pass on a
build where the embed emitted nothing.
Not verified, and not verifiable here. node sand.js instantiates the module, runs main, and dies inside glfwInit
on window is not defined — which says the module is live and says nothing about the canvas.
- Whether it paints at all. Nothing in CI has ever seen a pixel of this.
- Audio.
start-audiogenerates a tone,ExportWaves it to/tmp/flan-sand-tone.wavand loads it back as a music stream. That is raylib's ownfopen, not Flan'sbarf, so the#ifdefinflan_rt.cdoes not cover it and emscripten's MEMFS may well give it a writable/tmp. Every use is behindmusic-ok/tone-ok, so a failure is silence and not a crash. Separately, browsers suspend the audio context until a user gesture, soaudio-okmay be true while nothing is heard until the first click. - The loop never exits.
WindowShouldClose()onPLATFORM_WEBis anemscripten_sleepthat returns false, sountilnever terminates and none ofmain'sdefers ever run — noCloseWindow, noUnloadTexture. That is correct for a page, which is torn down by the tab closing, and it is worth knowing before reading anything into it. - Canvas size against
screen-width/screen-height. The shell is a string inBuildand its canvas is not sized from the program, so 900x600 may be letterboxed or cropped.
The return type is the slot, and unit is ()
(defn name [param Type ...] ReturnType body ...). The slot after the parameters is unconditionally a type, and a
function that returns nothing writes (). It used to be optional.
What optional cost. (Option f64) and (Some 1) are the same s-expression — a type application and a
constructor call are indistinguishable by shape — so the parser decided which it had by looking the head up in a set
of names that were actually types, collected by a pre-pass over the file's own declarations plus the prelude's. That
is sound, because one top-level namespace means a name cannot be both a type and a value. It is brittle because the
set has to be complete, and it was wrong twice in one day:
- the prelude's type names were not in the set at all, so a macro's
Formin return position read as an unknown value; - the fix for that exposed a second arm reading the same set, which had been parsing
(defn f [] (Rune {.code 65}) (bar))as a function returning aRunewith a one-form body — silently, in every file in the language.
A silent misparse is the worst failure class available. Macros generate definitions now, which widens it, and a table that has to be complete will be incomplete again.
Mandatory removes the guess. Parse.decl stops taking a set of names, is_type_form, qualified_type,
types_in, declared_types and prelude_types are gone, and the pre-pass over a file's declarations that fed them is
gone with them. Whatever is in the slot is a type; whatever follows is the body. Two things fall out:
- A type the parser could not have known is fine. A struct declared further down the file,
rl/Vector2behind an alias that is not resolved until after parsing, a prelude type — none of them needed to be recognised, they just needed to be in the slot. - A typo is a typo.
(defn f [] f65 0.0)reachesCheck.resolve_name, whose near-miss check answers unknown type f65 — did you mean f64?. It used to be parsed as the first form of the body and reported as an unknown name, which points at the wrong mistake.
The cost is () on every void function, against plan.org's deliberate short form. Taken.
Unit is (), ML's spelling. It is honest, and it cannot collide: an empty call is not a valid expression, so
there is no reading of () in value position for a body form to be confused with. The old Unit spelling is
refused, with a message naming the new one — the same rule the colon-to-dot change followed, and for the same
reason: two accepted spellings is how two spellings become permanent.
Internally it is still Types.Unit, and () parses to Ast.Tname "Unit", so the resolver, the shim and the emitter
did not change. Cimport still builds Tname "Unit" for C's void and never goes through the parser, which is why
resolve_name still answers to the word. Diagnostics print (): Types.to_string prints what a person would
write for every other type it knows — [i32], {K V}, (Ptr T) — and Unit was the odd one out the moment the
source spelling changed.
One shape is new. (defn f [] ()) is a function with a return type and no body. The old optional slot could not
produce it: (defn f []) had nowhere to put the type, and a lone form after the parameters was always the body.
The sweep
tools/unit-return.py, kept rather than thrown away, because the lanes that branched before this wrote Flan in the
old spelling and their files want the same pass at merge:
python3 tools/unit-return.py .
python3 tools/unit-return.py --in-strings test/test_flan.ml test/test_acceptance.ml \
test/test_session.ml emacs/test-flan-dev.el emacs/test-flan-mode.el
python3 tools/unit-return.py --raw-ml lib/prelude.ml
python3 tools/unit-return.py --in-html web/index.html
It fills the empty slot with () and rewrites Unit as () wherever a type is spelled. Deciding whether a defn
already had a return type is the whole difficulty, and the script does it by transcribing parse.ml's
is_type_form rather than improving on it — being identical to the parser it replaces is what makes the sweep
meaning-preserving. 440 sites in .flan, 260 more embedded in OCaml, elisp and HTML; -v logs every defn it saw
and what it decided, which is the only practical way to review a sweep that size.
Three hazards it knows about, and one it cannot:
- A snippet split across concatenation.
cursor ^ "(defn f [s [u8]] Cursor ...)"is one OCaml literal that does not contain thedefstructin the other. The embedded modes pool every fragment's type declarations across the whole file and count a pooled name only in bare-symbol position — as a list head it would eat(Some 1)as a return type, which is the misparse this change exists to remove. Sound because no user type takes arguments. - A fragment that cuts off mid-form,
"(defn step [] i64\n", is skipped rather than guessed at.flan-mode.el's"(defn step"search strings are the same case. - A bare
Unitoutside a form is left alone, so the checker's ownTname "Unit"pattern in a test literal is not rewritten into nonsense. - What it cannot know:
test_flan.mldeliberately spells the refused forms, to test that they are refused —(defn f [] (g)),(defn f [] Unit (g)),(defn f [] Nope (bar)),(defn f [] f65 0.0). The script encodes the old rule, so it wants to convert all six of those sites and must not. Re-running it on this tree reports exactly those six; anything else is a real conversion. Read the diff.
The colon belongs to keys, so a field label is a dot
{.x 1.0 .y 2.0} is how a struct is constructed, and {inner .field} is how a pattern names one. The colon is gone
from both, and what is left of it is one job: keys — map keys and enum members.
What was wrong with the colon. Nothing, taken alone. The problem was that it had two jobs and the dot had one.
(.x v) already read a field; {:x 1.0} also named a field, while :space named an enum member. So the dot meant
"field" and the colon meant "field, or member, depending". Moving the label to the dot leaves each mark with one
meaning, and it costs nothing to read because the delimiter already disambiguates: (.x v) is a list and
therefore a call and therefore an access; {.x 1.0} is a brace form and therefore a construction. There is no
position where the two could be confused, which is why the same spelling can serve both.
Why it had to land before Map, and this is the real reason. A map literal wants to be {:key value}. While
struct construction owned that exact spelling, a map literal and a struct literal were the same syntax, and the
only thing that could tell them apart was what the checker expected at that position. That is a context-sensitive
grammar for no gain. Reserving the colon for keys keeps the two visibly distinct at the reader, before any type is
known. Doing it after Map landed would have meant changing both; doing it first meant changing one.
:keys kept its colon, and that is the point rather than an inconsistency. {:keys [x y]} is the one thing in a
brace that is not a field name — it is an instruction to the compiler that happens to sit there, and it takes a
vector rather than a value. Giving it a dot would have made the dot mean "a field, or the word keys". Leaving it a
colon lets the dot mean exactly one thing, this names a field, which is the whole reason the colon was given up.
Everything else Clojure puts in that position — :as, :or, :strs, :syms — is still refused by its own name.
The old spelling is refused, not accepted quietly, and the refusal names the new one: a field label is written .x, not :x — the colon is for keys. Two accepted spellings is how two spellings become permanent, and the standing
rule here is that what is not supported is rejected explicitly with the reason. Both refusals are tested by their
reason, on the construction side and on the destructuring side, which is what stops the colon drifting back.
The sweep is a tool, not a one-off. tools/colon-to-dot.py converted 681 labels across 45 .flan files,
vendor/ included, and 94 more in the Flan embedded in lib/prelude.ml and the tests. It works on forms, not on text: a keyword
becomes a dot only where it sits in a field-label position inside a brace, so an enum member in value position
({.k :hi}), a genuine EDN map inside a string (test/programs/edn.flan), and a type-position {K V} are all left
alone. It was kept in the tree because several lanes branched before it and their Flan needs the same pass at merge.
What it deliberately did not change: the printed form. render.ml still prints (V {:x 1.5 :y 0}). That string
is a wire format — emacs/flan-inspect.el parses it back and hard-codes the colon when it reads a field out — so
moving the printer alone would break struct inspection in the dev loop without breaking any test that says so. The
printer moves when its reader does, in the Emacs lane. It is the one place the old spelling is still correct, and
the reason is worth keeping: a format with two ends only changes at both.
Since written: the printer moved, and both ends moved together. render.ml prints (V {.x 1.5 .y 0}) now and
emacs/flan-inspect.el reads the dot, which is the "moves when its reader does" the paragraph above was waiting on.
The reader tells ... from a field label by one character of lookahead, because both begin with a dot and a field
name never starts with a second one.
Two ways to root a walk, and why neither subsumes the other
i in the break buffer sent a local's name to be evaluated. An expression is evaluated where the evaluator
stands, so on the innermost frame that lands in the right frame by luck; on any other it may resolve to a global, to
another binding of the same name, or to nothing — with the locals listing right above it showing the frame's own
storage and nothing saying the two disagree. The display was right and the inspector was not, which is the worst
arrangement of the two.
The obvious fix was tried and rejected, and the rejection was half wrong. Rooting the walk at the slot's address
does not work on its own: an address is not an expression, so the first RET has nothing to build the next expression
from and navigation dies at step one. What that argument assumed is that the step has to be an expression too, and
the shadow stack is what stopped that being true. The daemon holds the frame's address and every slot's type, so
stepping into a field is an address plus an offset with that field's type — which is exactly the arithmetic
Render.render already does for the locals listing. So Session.render_slot is render_locals with a path applied
to the root before the walk and one line out instead of one per slot. No second walk was written and no backend
change was needed.
The verb is (:op "inspect" :frame N :slot I :path (...)). A path step is a string for a struct field, an integer for
an array or slice element, and the symbol some for an option's payload; a union case's field is spelled
Union.case.field, because the payload's offset depends on which case the value is in and only the renderer knows
which case it currently holds — it wrote the head (Union.case {…}). Guessing the case from a field name two cases
share would read one case's layout over another's payload. Every step that does not fit the type in hand is refused
by name with its reason. A pointer is still never followed; that is the renderer's rule and not this mode's.
The slot travels by index, not by name. check.ml's fresh_slot only ever allocates, so (let [v 22] …) inside
(let [v 11] …) is two slots both called v and both are in the listing; and a refused slot is not in the listing at
all, so its position there is not an identifier either. locals therefore puts the slot index on each entry as a
fourth element, and that is what the break buffer hands back.
The frame checks are the listing's, by construction. Dev.stopped_frame is one function and locals, globals
and inspect all go through it: alive, stopped, the frame exists, it is the program's and not a C-x C-e thunk's,
its body is one this session holds, the slot count matches, and Emit.slot_fingerprint matches. An inspector with its
own copy of those conditions would be free to read a frame whose body was redefined since it was entered, which is
precisely the stale-slot answer the listing refuses. inspect adds one refusal of its own, for the listing's reason:
an unbound slot is a null address and a thunk that read it would fault on the game thread of a program that is already
stopped.
What each root cannot do that the other can
Both are wanted and the buffer says which it is on.
The expression root works on a running program and starts from anything you can write, a call included. It cannot name a frame — that is the bug — and it cannot reach an option's payload, because the compiler gets at that as field 1 and nothing in the surface language does.
The slot root is exact to one frame and one slot, and it reaches an option's payload and a union case's fields,
which have offsets but no accessor form to write. It needs a stopped program, it is refused when the frame's body
was redefined since it was entered — the same fingerprint the listing is refused by — and it cannot root at an
expression at all, so g after the program resumes is refused rather than quietly answered from somewhere else.
A refusal someone can read is the point of the second one existing. The failure being fixed was not "no answer", it was a confident answer from the wrong place.
l does not cross between them, structurally
A stack entry in flan-inspect.el is (ROOT PATH . POINT). RET only ever appends a step to the path under the root
the buffer already has, and every new root — flan-inspect, flan-inspect-slot — starts with an empty stack. A stack
with two kinds of root in it therefore cannot be constructed, so the question of what l should do when it crosses
one does not arise. That stays true if a third rooting mode is added, which is why it is worth having as structure
rather than as a rule in a comment.
The Emacs state is a root plus a path rather than a retained value for the same reason the expression stack was:
nothing on this side can hold a Flan value. A value has no header, the thunk that rendered it is dlclosed the moment
it returns, and there is no heap to retain it in. So every step and every g is a fresh request, which is what keeps
the view from ever being stale — and it is also why g is a key someone presses rather than a timer, since an
expression root with an effect in it would fire once a second for ever.
One wire detail worth recording. An empty :path is sent by omission. Emacs prints an empty list as nil, which
is a symbol on the wire and would be read as a step, so there is no way for a client in that language to spell ().
The daemon reads a missing :path — and nil — as the slot itself.
The watch window, and why it is the only listing that is pushed
The port of the author's Clojure watch.el, with the good idea kept and the transport turned round. The original is
85 lines and three of its decisions survive contact unchanged:
- The program decides what is shown. Emacs paints what one function hands it. There is no watch-expression
machinery, no per-variable registration, no UI for building a query. Everything a watch list would otherwise have to
answer — where it lives, whether it survives a restart, whether it gets committed by accident — stops being a
question once the list is the code, edited with
C-c C-clike anything else. - The request is async. A synchronous call on a 0.2s timer blocks Emacs's UI every tick. The original says so in a comment, having evidently learned it.
replace-buffer-contents, not erase-and-insert. It diffs, so point and scroll survive every repaint. Erasing yanks the cursor to the top five times a second, which makes the buffer useless for the one thing anyone wants to do in it — look at a particular line while the program runs.
The one thing that does not port, and it inverts the design
In Clojure an eval is cheap. Here eval-expr compiles a module and dlopens it — tens of milliseconds and a new
.so each time, in a directory nothing sweeps. Polling (watch/render) at 5Hz would produce hundreds of shared
objects a minute to read a number that was already in a register.
The first answer written down — in NEXT.md, now superseded — was to compile the render thunk once and re-invoke it cheaply per tick. That is the right instinct and it is still a poll, and a poll has a defect no amount of caching fixes: it cannot answer while the program is stopped. A thunk runs at a frame boundary, a stopped program has no more frame boundaries, and a break loop is precisely when you most want to see what the last frame held.
So the direction is reversed. The program pushes. It calls into a table in flan_dev.c from inside its own loop;
Emacs reads the table, which is memory rather than an evaluation. Both halves are cheap for opposite reasons, and two
properties fall out that no poll has:
- the values are as fresh as the last frame, whatever the repaint interval happens to be — the timer decides how often the picture is redrawn, not how current it is;
- and they are still there while the program is stopped, because nothing has to run to produce them.
This makes watch the only listing in the daemon that compiles nothing. Every other one — locals, globals,
inspect, eval-expr — is a thunk built from the types, delivered, and run at a frame boundary. That is affordable
at the rate a person presses a key and ruinous at the rate a HUD refreshes, and the difference in rate is the whole
reason this one is shaped differently.
What the frame thread is allowed to do, and what the table is therefore made of
The writer is the game thread, mid-frame, every frame. That is a stricter constraint than the rest of flan_dev.c is
under, and it decides the storage:
- No allocation. Names are fixed
chararrays inside the table, notstrdup'd the wayinterndoes it alongside.internruns at module load and maymalloc; this runs at 60fps and may not. - No lock, because the reader is the agent's listener thread and neither side may wait for the other.
- No call into OCaml, which is the rule that keeps the collector off the frame thread. Nothing in the table is OCaml.
The result buffer is deliberately not reused, and this was the tempting mistake. The renderers are the same
shape, so sharing looks free — and it is wrong, because result is written once per C-x C-e and this is written
every frame. Sharing would mean watch traffic overwriting the value of every expression anyone evaluated. Separate
storage, separate counters.
One seqlock per slot rather than one for the table. A table-wide counter makes a read all-or-nothing: the reader has to copy every slot inside a single even generation, which means catching the gap between two frames' worth of writes — a window that at 60fps is whatever the program does after its last watch call, and may be nothing. Per-slot, the reader retries one slot at a time and always gets somewhere. The worst it can produce is a snapshot whose entries come from adjacent frames, which for a HUD is not a defect: a frame counter one ahead of a position read a millisecond earlier is what a HUD looks like anyway. It would be a defect for anything where two values have to agree, and that is a different op rather than a bigger counter.
A torn slot is still listed, with an empty value, rather than dropped. Dropping it would make the buffer's rows move under the reader every time the game happened to be mid-write, which is much worse to look at than one value that is blank for a tick.
The bounds, and what happens past each
64 slots. Past that a name is dropped, not fatal. This is the one place the house style in flan_dev.c — die,
loudly — would be wrong: a watch is a diagnostic, and killing the program because somebody watched a 65th value is the
diagnostic shooting the patient. It is not silent either. An overflow flag is read back with the table and the buffer
says so, because a value that simply never appeared would send someone looking for a bug in their program.
A flag and not a count, which was a correction. The first version counted, and a counter on the write path counts writes: the write path runs once per watched value per frame, so one name too many at 60fps reads back as "3847 names found no slot" within a minute — a false sentence about a true problem. What a reader needs is "the table is full and something is not being shown", which is one bit, and one bit cannot drift into a wrong number. Counting distinct names that missed would mean remembering which ones had, which is exactly the bookkeeping the frame thread has no room for.
31 bytes of name, 192 bytes of rendered value. Both truncate; the value's truncation shows as an ellipsis, the
same as flan_dev_result_end does, so a clipped value does not read as a complete one.
What it costs when nobody is watching
Nothing writes the table until a watch buffer is open. M-x flan-watch sends watch-enable :on t and closing it
sends :on nil, so arming is a message rather than something the daemon infers — the program is the writer and it
has to be told.
So the cost of a watch call in a program nobody is debugging is one relaxed load and a not-taken branch, and that
is the same number in a release build as in a dev one: flan_dev.c is linked into every build (Build, which says
why), so the symbols resolve either way and there is no second version of the file.
It is not free, and the distinction is worth keeping honest. Eliding the call entirely needs the compiler to know
the form, which is the check.ml arm below. A load and a branch per watched value per frame is the real number.
Scalars work today; composites need a check.ml arm that was not built
The four entry points a program can reach through declare-c are the whole feature for a scalar:
(declare-c watch-i64 [name string x i64] i32 "flan_dev_watch_i64")
(watch-i64 "ticks" ticks)
No arm in the checker, no new special form, nothing the compiler has to learn. They return i32 rather than nothing
for a blunt reason: declare-c refuses a void return outright — "which is not a value C can carry", shim.ml — so a
function a program can declare has to return something, and since it must, it returns the useful thing: 1 if the value
was written, 0 if nobody is watching or the table is full.
A composite — a struct, a slice, a union — cannot be reached this way, and that is not a shortcoming of the four. A
Flan value carries no header, so nothing at run time can say what it is, and rendering one is a compile-time walk over
its type. That is the same reason C-x C-e renders in the thunk rather than marshalling anything, and it is the
layout decision's bill, paid in the same place.
The missing piece is one arm in check.ml, and it was deliberately not written — that file is held by another
lane. It sits beside print (check.ml:3480) and is the same shape as it:
| "watch" ->
arity loc name 2 args; (* a name and a value *)
(* a read, not a move — as print is, for the same reason: (watch "v" v)
must not consume a Vec and make that its last showing *)
let n = check ctx (List.nth args 0) in (* must be String *)
let a = borrowed ctx target (fun () -> check ctx (List.nth args 1)) in
(* begin, the walk, end — with the emitter aimed at the four
flan_dev_watch_emit_* rather than at WriteStdout *)
Render.render { rc with emit = watch_emitter } 0 a
with flan/watch-begin, flan/watch-end and the four emit functions declared as externs the way Session.externs
already declares flan_dev_emit*. Nothing else has to move: Render.render is unchanged, the runtime side is built
and tested, and the daemon and the editor cannot tell which kind of caller filled the table.
That arm is also what ghost text is gated on. It was not, and ghost text is built without it — see
"Ghost text finds its anchor in the buffer, not in the table" below. The claim was that values shown inline need a
place and nothing in the table has one, so a source location would have to be carried per entry, so the call site
would have to be generated. True of the table and false of the conclusion: the call site is in the buffer. The
composite renderer still wants the form, for its own reason, and it is the only one of the two that does.
An error is a value, and there is more than one of them
lib/loc.ml used to carry a point and a message, and Loc.Error was the frontend's one exception, so the first
error ended the run. The author's workflow is write everything, compile at the end, work through the list — which
cannot happen when there is never a list. The messages themselves were already good; they state the reason and name
what to write instead, and none of them changed. What was missing was structure and volume.
The location is a span
Loc.t grew an exclusive end, defaulting to the start. That is the whole trick: a location nobody widened is a
zero-width span at a point, so every call site that existed before means exactly what it meant, and Loc.to_string
still prints file:line:col. Only the reader knows where a form ends, so only the reader fills them in — one helper
in the one place that holds both ends, which is why nothing above Reader had to learn a span exists. Form, Ast
and Tast were not touched and did not need to be.
A column number cannot draw an underline and a span can. That is what the field is for and it is the only reason it is there.
The error itself
type diag = {
kind : string; (* "reader/unclosed", stable *)
dloc : t; (* the primary span *)
dmsg : string;
notes : note list; (* each with its own span and severity *)
expansion : (string * t) option; (* the macro it came out of *)
}
exception Error of diag
exception Errors of diag list
Three parts, each buying something the old pair could not express.
kind is a stable id. It classifies with no prose parsed, so a message can be reworded without breaking anything
that depends on which error this is. The reader's fourteen refusals all carry one; in the checker they go on the
errors a test names and the handful common enough to be worth classifying. Not a hundred of them. jank has about
a hundred because it is mature, and the count is not the feature — with 163 refusal sites in check.ml alone,
minting an id for each would be a sweep that never ends and that nothing reads.
notes are the part that was actually missing, and they are the secret of an Elm-quality message. Each carries
its own span and its own severity, so an error says "this is wrong here" and "because of that, over there",
and points at both. One location and one string can only ever state one of the two. What has them today:
- a name defined twice points at the second, because that is the one to delete, and notes the first;
- a duplicate parameter and a duplicate field do the same;
- an unknown field, an unknown struct and a non-exhaustive
matchnote the declaration and list what is actually there, so the reader's next move arrives with the question instead of after it; - the reader's unclosed bracket is the clearest case — the error sits on the bracket, because that is where the fix goes, and the note sits where the file ran out, because that is the surprise. A mismatched closer is the mirror of it: the wrong closer is where the mistake reads, and the opener is what makes it wrong, and neither alone says which bracket to change.
expansion names the macro an error is really about. It rides on the location, not on the form, because the
location is the thing that already travels: Expand.unmarshal stamps the call site onto every node a macro answers
with, and that stamp goes on through the AST and the typed IR untouched. Tagging it there means an error raised
anywhere downstream can name the macro with no field added to Form, to Ast or to Tast. Outermost wins — the
macro the author wrote is the one worth naming, not whatever it expanded into on the way down.
The squiggle
The first line of an entry is exactly file:line:col: message, which is the GNU format compilation-mode parses
with no configuration. That is the whole of the editor story: once more than one comes out, M-x compile gives a
clickable list and next-error walks it. Everything under the first line is indented, and compilation-mode ignores
indented continuation lines, so the underline is free:
prog.flan:6:3: Cursor has no field pos
6 | (.pos c))
| ^^^^^^^^
prog.flan:1:1: info: Cursor is declared here, with row, col
1 | (defstruct Cursor
| -----------------
A note gets an entry of its own rather than being folded into the error's block. That is gcc's shape and it is the point of notes having locations at all: the second place becomes a place the compilation buffer knows about.
Stated at its true strength, because it was checked rather than assumed (compile.el, Emacs 30.2). The gnu
entry in compilation-error-regexp-alist-alist puts Note/note in the same capture group as Info/info —
group 7, level 0 — while warning is group 6, level 1. compilation-skip-threshold defaults to 1, "skip
anything less than warning". So: errors are navigable with next-error out of the box, which is the claim that
matters and the one M-x compile rests on. Notes are parsed, coloured and clickable, and next-error steps over
them until compilation-skip-threshold is 0. Renaming the label from info: to note: does not change that — same
group. Labelling notes warning: would make them navigable at the default, and is refused: a note is not a
warning, and a compile whose only complaint is an error would start reporting warnings that are not warnings.
Every part of it degrades to the bare first line. A location the checker invented has line 0 and a file called
<unknown>, the prelude and the REPL have names that are not paths, and a file can change under us between being
read and being blamed. An error printer that can raise is worse than one that prints less. Placeless diagnostics sort
last: a wrong main signature is raised against unknown, and sorting on the line number alone put it above every
error that could actually be clicked.
The source cache in loc.ml is process-lifetime, which is right for flan build — a fresh process per run. The
daemon is long-lived and never calls report; the interactive path draws no squiggle, it takes a location and a
message. Loc.forget_sources exists for the day that changes.
Collecting, and where it stops
A sink holds what a pass found so the pass can go on to the next thing. It is switched on by the caller, not by the code that raises. Two resync points, and both are places the work already had a boundary:
- In the parser, a top-level form. The reader already found where each declaration ends, so skipping a bad one
costs nothing and cannot lose its place. Inside a declaration there is no such landmark, so one bad
defnstays one error. - In the checker, the two passes. Pass one — which builds every name, type and signature — still stops at the first refusal, and that is deliberate rather than unfinished. A signature it could not make sense of leaves a hole that pass two would report once per mention; thirty "unknown name" lines under one wrong signature are not thirty errors, they are the same one. Pass two is where the volume is and where collecting pays, and by then every signature is sound, so a body that fails cannot make the next body fail. That is what makes a declaration a resync point needing no resynchronising.
The reader does not collect at all. There is no resynchronising a paren stream: after an unclosed bracket the
reader has no way to know whether the next ) closes the form it is in or the one above it, and guessing produces a
file-shaped pile of nonsense. First error, stop. That is a decision, not an omission.
What the daemon sees, which was the open question
Changing the error type without touching dev.ml and session.ml needed a compatible way to get one location and
one message out. The answer is that the single-diagnostic exception is still the single-diagnostic exception.
Session.eval and the daemon evaluate one form and have one failure to report; they keep catching Loc.Error and
take the pair out of it with Loc.summary. Only a driver that compiles a whole file raises Loc.Errors.
That guarantee is structural and not conventional. Parse.program / Check.program stop at the first refusal;
Parse.program_all / Check.program_all collect. Two names rather than one function with a ~keep_going label,
because Loc.Errors is a second exception that the session's handlers do not name — a list reaching them would be an
unhandled exception and a dead session, which is the one thing the dev loop exists to prevent. With a label that was
one keystroke away at a call site the session already uses. With two names, somebody has to edit the session.
What it looks like
$ flan check bad.flan
bad.flan:2:8: unknown name bogus
2 | (+ a bogus))
| ^^^^^
bad.flan:5:8: unknown name nope
5 | (- a nope))
| ^^^^
bad.flan:8:3: unknown function mystery
8 | (mystery 1 2))
| ^^^^^^^^^^^^^
3 errors
No editor work was needed and none was done. Flycheck and a structured JSON report were both considered and are not wanted: the workflow is compile-at-the-end, not live linting, and the GNU first line already buys the clickable list.
The four raylib lines siam-farmer needed
PORTING.md measured the author's game against the binding and found the renderer unwritable in a default build.
Four lines closed it, and the interesting part is not the lines.
draw-texture-pro is the one that mattered. It is the only call in the package that takes both a source rectangle and
a destination rectangle, which is what a tilemap is: source picks a cell out of an atlas, dest says where it lands
and how big, and a 16px tile drawn at 4x is a dest four times the source. draw-texture-rec has the source and no
scale; draw-texture-ex has the scale and no source. Neither half draws a tile.
It was reachable — with FLAN_RAYLIB_H exported the importer brings it in with 250-odd others — and that is the
finding worth keeping. vendor/raylib/headers keeps the import opt-in on purpose, so a build needs libraylib linkable
and not raylib-devel installed. That property is worth keeping and it means the default build had no draw call for a
grid-based game. The rule that follows: a raylib function on a game's per-frame path is hand-written in
raylib.flan and checked against the header; it is not left to the import. The import widens the surface; it must
not be load-bearing.
The other three: image-from-image (the non-mutating image-crop — carving a sheet into twenty tiles with
image-crop destroys the sheet on the first one, and 5.5 has no ImageCopy), window-ready? (an engine's "am I
already running" guard), and left-shift 340 in the Key defenum. Nothing else went into the enum: it carries the
keys that have a customer, and PORTING.md §5 checked every other value the game touches and found them all present.
What could be tested, and what could not. PORTING.md asked for an acceptance case making raylib compute with the
source rect so a permuted Rectangle goes red. That cannot exist for DrawTexturePro — it needs a GL context, and
raylib.flan's Shapes comment already says none of the drawing calls can be in the table. So raylib-ffi.flan links
it instead: the call sits behind (when (rl/window-ready?) …), false headless, so the shim is generated and the
symbol resolves at link time and the body never runs. That catches a name or an arity libraylib does not have. It does
not catch the argument order, and three structs in a row is where an argument order goes wrong. Only looking at
the screen catches that, and saying so is better than a test that implies otherwise.
The computed case moved to image-from-image, which is CPU-side and is exactly what the Images section says is
assertable. raylib-image.flan carves one 6×3 sheet twice at two different ys and then re-reads the sheet: x,
y, width and height are each pinned by an answer that moves if they do, and the sheet surviving both carves is
what distinguishes this from image-crop. Bind it to ImageCrop by mistake and the second carve reads out of a 2×1
image and the case goes red.
An index out of range is a condition
flan_bounds_fail printed the source location, the index and the length, and called exit(134). That was defensible
while flan dev was two processes. It is not now: the compiler runs inside the program, so the trap took the
whole session with it — and the session not having to restart is the project's thesis. PORTING.md found the customer
and found it on the most ordinary path there is: a grid indexed straight from a mouse position is out of bounds the
first time the pointer leaves the window, and the author's Common Lisp port had to add an in-bounds-p to survive it.
A failed bounds check now signals BoundsError with error, the same way a failed allocation signals
StorageExhausted. runtime/flan_rt.c has flan_bounds_error and flan_slice_error; each walks the handlers, then
offers the break loop, and only if neither transferred does it tail into the flan_bounds_fail/flan_slice_fail that
were there before — same message, same status 134. Nothing was removed; a die was demoted to a last resort.
(defstruct BoundsError [low i64 high i64 length i64]) is in the prelude. Fixed numeric fields and no rendered
message, for StorageExhausted's reason: formatting allocates, and a condition raised on a path that may be out of
storage must not. low and high are the same index for an (at xs i) and the two ends of the range for a (slice xs lo hi), so one condition type covers both and a handler that wants to survive a bad index writes one clause
rather than two. The three int64_ts in flan_rt.c have to agree with that defstruct field for field — the same
hand-kept agreement flan_name_id already keeps with Check.type_id, and for the same reason: a struct is a layout,
a type is a number, and neither side can see the other.
Why no restart is established at the failing index
This is the decision, and it is a decision rather than an omission.
alloc_guard offers retry and file_guard offers retry/use-value because their attempt is repeatable. A
handler frees something and the allocation succeeds; a handler supplies another path and the open succeeds. That is
what makes them plan.org's named exceptions to "restarts go at the resync point, once" — a restart at an outer loop
cannot re-attempt an allocation, and only the allocation site can.
Nothing a handler can do makes index 51 valid for a length-50 array. There is no attempt to re-run, so there is nothing for a site restart to resume into, and bounds falls on the default side of that rule.
continueis wrong.(at xs i)has to produce a value of the element type and there is none to produce. It would mean something in the(set (at xs i) v)position and nothing in the other, andatis one form.use-valuefor the index is the near miss and is still wrong. It costs the hot path, not just the cold block:idxis an SSA value feeding the gep, and retrying needs it in an alloca reloaded per attempt, plus a restart frame pushed and popped on every indexing operation. What it buys is a different element, silently — the class of answer this codebase refuses everywhere else.- What actually answers a bad index is already on the stack. A frame loop's
continue—sand.flan's shape — is an ordinaryrestart-case,flan_find_restartwalks to it, and the break loop lists it. Signalling is the whole fix. A site restart would add nothing the frame loop does not already offer, at a cost on every index in the program.
Release, and what a build without the agent does
flan_break_hook is NULL unless the agent package was imported, so a release build — or any program that did not
import it — signals, finds no handler, finds no hook, and dies with the message and the status it always had. That is
still right: there is nowhere to stand. The change is not "bounds failures stopped being fatal"; it is "a bounds
failure is now answerable, and is fatal when unanswered".
Defer, which had to be answered rather than inherited
The note at the top of this file said a trap runs no defers, which follows from the bounds-check shape (noreturn
then unreachable) rather than being a separate decision. The shape changed, so the consequence could not be
inherited. It now splits:
- An answered bounds failure runs the defers.
guardroutes throughcurrent_pad, which with no enclosingrestart-casesetsf.unwoundand branches to the function's unwind block — the same pathreturnuses, which is where §5's defers already live. So a transfer out of a bad index is an ordinary transfer and runs cleanup innermost-first, like every other one. - An unanswered one still runs none, because it is still a
rt_die()inside C with no Flan frame involved.
bounds-condition.flan asserts the first of those directly: five defers across five abandoned and finished frames,
counted.
Both halves are tested, and they are different tests
bounds-condition.flan (acceptance table, at -O2, -O0 and as a dev build) is the answered half: a
handler-bind over five routes to a bad index, taking the frame loop's continue each time. It does not import the
agent, so flan_break_hook is NULL in all three rows and nothing there says anything about the break loop.
dev-break-bounds.flan (test_dev.ml) is the half the change is actually for: nothing handles it, so the signal
walks the handlers, finds none, and reaches the hook. The daemon sees a program that stopped without being told to;
BoundsError is what the break reports; its own name resolves to a layout whose fields are low, high, length,
so the conditions buffer shows the numbers with nothing special-cased for it; the restart list is exactly
continue — the program's own, which is the visible consequence of establishing none at the site — and taking it
resumes, with an ordinary evaluation working on the far side. That last step is the whole claim: the session outlived
the index.
Vec and Map
(at v i) and (at arr i) are the same form in the source, so shipping one signalling and the other exiting would
have read as a bug. A Vec's bounds check lives inside flan_vec_at and flan_vec_as_slice rather than in emitted
IR (which is also why --no-bounds-checks never reached it), so both grew a trailing transfer-channel parameter and
Emit's Rt arm guards those two symbols and no others — they are the only ones in that family that can transfer;
everything else there is arithmetic over a container header.
A Map's bounds and a Vec's stale-allocator check still die. flan_vec_stale_fail is a different kind of
failure — the region a container lived in was released, and there is no frame to go back to that would not read freed
memory — and the map path was left alone rather than converted half-way. Written down here so it is a known edge
rather than a discovery.
An address answers with a type, and nothing grew a tag word
A Flan struct is exactly its C layout. No header, no tag word — deliberately, and it is what makes a struct free and
what makes the FFI work. The consequence is stated elsewhere in this file more than once: a Flan value carries no
header, so nothing at run time can say what it is, which is why a rendering is a compile-time walk over a type and
why (watch "v" v) cannot reach a composite without an arm in check.ml.
The allocation registry does not answer that question. It sidesteps it. The allocator's caller knows the type at the moment it asks for memory, and the compiler is standing right there, so a dev build writes it down: base address, extent, element size, and the type's printed spelling. Nothing about any value's layout changes, and raylib never finds out.
The split, which is the whole design
Two halves, and they are in different files because they need different things.
- Recording the type is emitted.
check.mlbuilds oneflan_dev_reg_note_*call after every operation that may have allocated —vec-new,push,reserve,clone,pool-new,insert,map-new,put,slurp. Here is the only place the concrete element type exists, so here is the only place that can name it. It is built in every build, because a tree that differed by build flag would make every pass between the checker and the backend ask which one it was looking at;Emit'sRtarm drops the family whendevis off. - Recording that a block died is not emitted. An address needs no type, so
flan_rt.ccalls into the table directly fromflan_heap_proc's free, from a heap resize for the block it moved away from, from the arena'sfree-all, and fromarena-destroy. No signature in the allocator grew a type name and no ABI moved.
The drop in Emit happens before the arguments are walked, not after. A note takes the address of the container
it describes; emitting that address and then discarding the call would leave an escaped alloca behind, and an
escaped alloca is one mem2reg will not promote. So a release build's IR is the same IR it always was.
It is armed by a constructor, and that is not fastidiousness
@llvm.global_ctors in the dev module, not a line at the top of main. A defvar initialiser can allocate, and it
runs before main does; a note that arrived before the flag was set would be a block the table never heard of, which
is a live pointer the inspector would call dead. That is the one failure mode worse than no registry at all.
Lookup is containment, and that is not an optimisation
An entry is a block, not a value. Every pointer a program can hold into heap storage is interior: (at v i) is
v->ptr + i*size, and (resolve p h) is an item in the middle of a pool's items array. Neither is ever a base
address. A table that answered only exact hits would answer nothing anyone can ask it.
The two sides of the table therefore look different, and the difference is which thread is asking:
flan_dev_reg_noteandflan_dev_reg_deadare probes — a hash slot and a linear walk from it. Both are on the writer's side, and a free hands back the same base address the allocator gave out, so equality is the whole question there.flan_dev_reg_liveand the epitaph are a scan of all 4096 slots. The reader is a person pressing a key, so the cost belongs there.flan_dev_reg_dead_rangeis a scan too, and it is the one that runs per frame: an arena'sfree-allhas no list of what it handed out, so the region is matched against the table rather than the other way round. That is a real per-frame cost in a dev build and it is named here rather than discovered later.
Dead entries are kept. That is the second thing the registry buys — an address that was freed still names what died — and an entry is dropped only when the allocator hands the same address out again, which is exactly when the old answer stopped being true. When the table fills it is compacted, dropping the dead and re-inserting the live; in a long-running program the dead are the bulk of it.
What it is for: permission, not identification
This is the part worth stating plainly, because the obvious reading is wrong.
(Ptr Enemy) already says Enemy, at compile time, in Render's walk. The type at the far end was never the
difficulty. What was missing is permission: whether it is still true to read the storage there. render.ml's old
comment said a pointer is never followed because "dereferencing one a REPL was handed is not a safe thing to do on
someone's behalf", and that sentence is still correct — the registry just makes the safety checkable. So the arm
became a branch:
("live" "(Ptr Enemy)" "<ptr (Enemy {.hp 41 .x 2})>")
("dead" "(Ptr Enemy)" "<ptr dead: was Enemy, freed at step 15>")
The recorded type name is therefore not what selects the renderer. It is what the epitaph says, and a cross-check available to anything that wants one.
An address the registry never saw renders <ptr>, unchanged. That is a stack local, a global, or a pointer from C,
and the shadow stack and the static type table already answer for the first two by name.
println does not follow a pointer, and will not. spec-memory.md fixes what a printed Ptr prints, a printed
line belongs to the program and has to read the same in a release build, and a release build has no registry to ask.
The two callers of Render already differ in an emitter record; they now differ in a pointers record too, and
check.ml passes None. This is also why the epitaph carries no address: an address is not stable across two
runs, so printing one would make a rendering — and any test that reads one — depend on where the heap landed. It is
the rule Render already follows for an allocator.
The arena hole: answerable, not reported
test_valgrind.ml measures a hole and this does not close it. free-all is retain-capacity: the offset goes to zero,
the pages stay mapped, and from malloc's point of view nothing died — so memcheck is never told, and a later read of
stale bytes is a read of memory that is, as far as it knows, perfectly alive.
The registry is told. What that buys is that the same read is answerable: a pointer into a released region comes
back dead and names what used to be there. Memcheck still says nothing, and closing that half is
VALGRIND_MAKE_MEM_UNDEFINED in flan_arena_proc, which test_valgrind.ml names as the fix and which is still not
written. The two must not be blurred into one claim.
What a release build actually carries, said honestly
"Release builds carry none of it" is the goal and it is not quite true, in exactly two places:
- A load and a not-taken branch per free, and per
free-all.flan_dev.cis compiled into every build (seeBuild, which says why), soflan_rt.ccalls the dead-marking hooks unconditionally and each begins by testing a flag only a dev build sets. The alternative is a second version of the allocator selected by a build flag, which is worse than a branch for the reason theVecheader already carries its two dev words in every build: a layout or a code path that changes with a flag is one that can disagree across the redefinition boundary silently. - Nothing else. The table is
calloc'd when it is armed, not declared as an array — a fixed 4096-entry table would have been a quarter of a megabyte of BSS in a shipped game for something that build never writes. A release binary carries a null pointer, a zero flag, and the declarations, which cost nothing.
The @llvm.global_ctors entry, the notes, and everything that reads them are dev-only, which is the same arrangement
the indirection cells and the shadow stack have.
Tested twice, and one thing tested by hand
programs/registry.flan is one program read twice in the acceptance table. A dev build answers for an address at
the heap tier, the arena tier and the pool tier; a release build answers 0 to every question. The difference between
the two expectations is the assertion, and writing it as one program means nobody can change what a dev build does
without the release row noticing.
The inspector's pointer arm is not covered. test/programs/dev-ptr.flan is the program, its header carries the
two lines above, and they were read off a running session by hand — the test_dev.ml case that would drive it belongs
to another lane's file. NEXT.md says so rather than letting the verification read as automated.
A restart is not a transaction
Written down in three places rather than fixed, because it is a property and not a defect. If a frame mutates a global
and then signals, taking a retry re-runs the mutation. Control resumes at the restart-case and runs forward
from there; nothing is undone. Common Lisp has exactly this property and offers no help either — rollback would mean
journalling every store, which is a different language.
The discipline is that the author chooses where the retry boundary is. A restart-case at the top of a frame
re-runs everything including mutations already applied; one placed after the mutations re-runs only what follows them.
So: put the restart before anything mutates, make the retried section idempotent, or snapshot what will be re-applied.
§3 of spec-conditions.md places a restart syntactically — every clause body and the body share a type — and
nothing places it semantically.
It matters more here than in most Lisps because the intended use is a game loop, where the plan is to skip a frame
and carry on rather than die. That stopped being hypothetical when a failed bounds check began signalling
BoundsError instead of ending the process (see "An index out of range is a condition"): a frame can now be abandoned
and retried, which is exactly the case a non-idempotent mutation spoils.
Both conditions.org and web/index.html already carried the mechanical half — "a restart re-runs whatever sits
between it and the target" — as a one-line gotcha. Those were rewritten in place into the full statement rather than
having a second bullet added beside them, and the same reasoning went into spec-conditions.md §5, which is the
section that already enumerates what a transfer does and does not do: it runs defers, it skips errdefers, and it
does not undo. No numbered case changed meaning, so the freeze holds.
Ghost text finds its anchor in the buffer, not in the table
M-x flan-watch-ghost-mode paints each watched value inline, after the line holding the call that wrote it, as an
overlay after-string. It is an addition to the watch buffer and not a replacement, and both can be on at once:
the buffer is the picture you want when you want everything, inline is the picture you want when you are reading one
function.
Why it was thought to be blocked, and why it was not. The earlier note in flan-watch.el — and two paragraphs
above, now struck — said ghost text needed a source location per table entry, which needs a generated call site,
which means the (watch ...) form in check.ml. Every step of that is true of the table: it holds a name and a
rendered string, and (watch-i64 "ticks" ticks) says what the value is called, not where it was written. The
conclusion did not follow. The call site is in the buffer. The name in the table is the string literal in that
call, so the editor searches its own text for the anchor instead of being told it. Nothing new is asked of the daemon,
and the composite-struct renderer is now the only one of the two things that really does want the checker arm.
The head of the call cannot be hardcoded. watch-i64 is a name the program's author chose in their own
declare-c; only flan_dev_watch_i64 is fixed, and the editor never sees it. Hence
flan-watch-ghost-call-regexp. Matching on the string literal rather than on the head is what makes that safe — a
name in the table came from one of these call sites by construction — and the syntax check (nth 8 of a
syntax-ppss) is what stops a call written inside a string being taken for one.
One reply, two pictures. Ghost text does not poll. It is painted from flan-watch--absorb, the same function
that paints the buffer, from the same reply, so the two cannot disagree and there is no second :op "watch" in
flight — the one-request invariant flan-dev-settle-hook exists to keep. What had to change is that the watch
buffer used to be the subscription: killing it cancelled the timer and disarmed the table. That was right while it
was the only consumer and wrong the moment it was not, so arming and the timer now hang off
flan-watch--consumers, and only the last consumer out turns the lights off.
Overlays are replaced wholesale on every repaint, never followed through edits. That is the entire answer to the invalidation problem the earlier note called the work: a line that moved cannot strand an overlay, because no overlay outlives a tick. The cost is bounded by scanning only buffers shown in a window — an overlay nobody can see is worth nothing, and the project's other files are not touched. It also means a file you scroll to is annotated within one tick with nothing to hook.
The questions the design had to settle, and the answers:
- Two sites, one name. Both get the overlay, both show the same value, and the text says
one slot, 2 sites. The table has one slot per name and the last writer in the frame wins, so that is the value at both. Showing it at one site would imply the other was not running; showing it at both in silence would read as a coincidence. - A watch inside a loop shows the last value written, exactly as the buffer does. "The last of 4000 iterations" is often not the interesting one, and every better answer is a UI for building a query — the one thing this design exists not to have. Settled, not open.
- Stopped. The values survive a break, which is half the reason for pushing rather than polling. But they are the
last frame's, and inline they sit in code that looks perfectly live with no modeline beside them, so each one says
last frameand changes face. The buffer needs no such marker: you opened it deliberately andflan:stopped(...)is already in view. - A site with no row is annotated only when the table reports overflow, where it says so. Then the value exists and was dropped, which is the case silence would send you hunting for a bug over; the buffer's overflow line says the same thing without being able to say which name. With room to spare, a site with no row has simply not run yet, and annotating every one of those at startup is noise.
- A name the runtime clipped. It holds 31 bytes, so a longer name in the source never matches a row exactly. The lookup falls back to a prefix match, guarded on the row being a full 31 bytes so a short name cannot claim a site that merely starts with it.
What gets no ghost text, and it is the honest limit of anchoring on text: a watch call produced by a macro, or one whose name is not a literal. There is nothing in the source to find — the editor is reading the file, not debug info — and both still appear in the watch buffer.
One bug worth recording, because it cost a debugging session and reads as impossible: syntax-ppss moves point and
clobbers the match data. Calling it inside a re-search-forward loop and then reading match-string or
line-end-position puts the scan back where it started, and the loop never ends. Everything wanted from a match is
now read out before the check, and the loop advances to a saved match-end.
emacs/test-flan-watch.el covers it, loaded from test-flan-cider.el for the reason test-flan-mode.el gives:
emacs/*.el is already a dependency of that dune stanza, so a new .el needs no build change. Nothing in it needs a
daemon — ghost text is a function from a table of rows and the text in a buffer to overlays, and both halves are
fixtures.
A hot loop keeps five numbers, and the window is the editor's
The scalar watch above keeps one value per name. From a hot inner loop that is nearly useless: you see whichever of
the 91,200 cells happened to run last, and watch.clj's own docstring says so — "from a hot loop you only ever see
whichever cell happened to run last — use spy-long there instead." This is that other half, and it is the piece
PORTING.md item 5 calls least obvious and most valuable.
(watch-num-i64 "cell" (at grid i)) reaches flan_dev_watch_num_i64 through the same plain declare-c the four
scalars use. No arm in the checker, no special form, nothing the compiler learns — the same deliberate non-ownership
of check.ml that the scalars were built under.
What a slot keeps: count, min, max, last, mean. The argument for those five is that each answers a question you
can ask without building a query. n is how many times the expression ran, which is the first thing that is wrong
when a loop is wrong — a count that tracks the frame counter rather than the cells is a loop that is not running.
min and max are the range, which is the thing a single sample can never show you and the thing you are looking
for when you suspect an index or a velocity is leaving the region it should stay in. last is the one sample, kept
because it is what the scalar watch would have given you and losing it would be a regression. mean is carried as a
running sum and divided at read time, because a mean accumulated as a mean drifts and a sum does not.
What it deliberately does not keep: a ring, or a history. A small ring of the last N samples was the other candidate and it loses on the only ground that matters: N samples out of 91,200 is a sample of the tail of the loop, not of the loop, so it answers "what did the last few cells do" when the question is "what did the cells do". Beyond five numbers every richer answer is a UI for building a query, and a query builder is the one thing this design exists not to be — the same reason the ghost text section gives for showing the last value in a loop rather than offering to pick one.
The write path does no formatting, and that is the whole feature. An snprintf per sample at thousands of
samples a frame is a HUD that costs more than the game. A sample is a relaxed load, five compares and stores, and the
slot's seqlock; the reader — the agent's listener thread, once per editor tick — turns the five numbers into text.
watch.clj reaches the same place with a double-array per label and a render that Emacs calls, and the reasoning
is the author's rather than ours. It is also why the slot grew a num flag instead of a second table: the read path
already copies under a seqlock, so it copies five doubles instead of 192 bytes and renders them out of locals after
the counter check.
The window is since the last reset, and this is a deliberate divergence from watch.clj. There the stats are
cumulative until reset-spies! is called by hand. Cumulative is the wrong default for a frame loop: a min and a
max over a whole session reach the session's extremes within a few seconds of play and then never move again, so
the two most useful of the five go dead exactly when you start interacting with the thing you are debugging. This
tool exists to show you a number while you drag the mouse. So flan-watch--tick sends :reset t beside its read and
the displayed range is "since you last looked" — a fifth of a second, a dozen frames. A caller that wants the
cumulative numbers gets them by not resetting; the setting lives in the editor, not in the runtime.
Reset is its own message, and it is not a side effect of reading. A destructive read was the tempting shape and
is wrong: it makes looking change what is there, so anything that polls — a test's await, a second editor, a
person reading twice — silently shortens the window and gets a count that is noise. This was caught before the test
was written rather than after, which is the only reason the test has assertions about n at all. watch reset is a
line in flan_agent.c beside watch on/watch off, :reset t is a field on the read op, and dev.ml sends it
after the read so the tick reports the window it just closed.
The reader never writes the table. Reset bumps one global epoch counter and touches no slot; a slot clears itself
on its next sample, when it notices the epoch has moved, and does that inside its own odd-generation window so a
reader can never catch a half-cleared slot. The game thread stays the only writer of the table, which is the
invariant the whole of flan_dev.c's watch section rests on. The cost is that a new window begins when the program
next runs rather than at the instant of the reset — and for a frame loop that is the only moment it could sensibly
begin. The test waits for it rather than reading once, and says why.
An i64 accumulates as a double, so a magnitude past 2^53 loses precision in the sum and in the ends of the
range. Recorded rather than designed around: a count, a coordinate and a tile index are what this is pointed at, and
a second integer accumulator for a case nobody has would be two code paths for one tool. watch-num-i64 and
watch-num-f64 are two entry points only so a program need not cast at the call site, which is noise in the one
place this is meant to be droppable into.
A whole number prints as one. watch.clj's fmt-num does this and the reason survives the port: a watch on an
array index that reads 66.0000 sends you looking for a rounding bug that is not there.
Ghost text needed one character. flan-watch-ghost-call-regexp was watch\(?:-[[:alnum:]]+\)? — one optional
hyphenated segment, which matches watch-i64 and backtracks to failure on watch-num-i64, so a numeric watch got
no inline value at all while appearing normally in the buffer. A * for the ? is the whole fix. It is worth
recording as the predictable cost of the decision that section defends: anchoring on the name rather than on the
head of the call is what makes the head a defcustom, and a defcustom with an enumerated default is a default that
needs widening when the set of heads grows. That is a cheaper failure than the alternative and it is not a free one.
A breakpoint is a function call, and the editor only says where
C-u before an evaluation marks a form so the program stops when that form runs — Clojure's convention, and
DISCUSS.md §9's request. Nothing in the compiler knows what a breakpoint is.
The mark is an ordinary (pause) call spliced into the tree. pause is already in the prelude and is already
error under a restart-case with a continue clause, so an instrumented body is a body that calls one more
function, and the break loop it lands in is the one an unhandled condition already builds. A paused : bool on
Ast.expr would have had to be threaded through Check, Tast and Emit for a feature the prelude implements as a
function.
It travels as a position beside the code, not as text spliced into it. §9 proposed sending the top-level form with
the target span replaced by (do (pause) <span>). That shifts every line and column after the insertion, and the
error overlays, layout, the break loop's frame locations and DWARF all read those. So the request carries
:pause (LINE COL) and the daemon applies it to the declarations, once parsing has attached the locations and
Load has qualified the names — Ast.mark_pause, pre-order, first hit wins. Applied after qualification, so a
package that defines a pause of its own cannot capture the synthesized call.
Pre-order and first-hit because desugaring makes locations non-unique. parse.ml gives several nested nodes the
same location — (when c a) becomes an If whose branch is a Do at the when's own position. The outermost node
at that position is the one the editor pointed at, and the walk stops there.
A whole defn cannot be wrapped, so marking one means stopping on entry. (do (pause) (defn ...)) is not an
expression. §9's first target therefore puts the call at the front of fbody instead, which is the same thing anyone
asking to stop at a function meant. Ast.map_children is exhaustive on purpose: a constructor left out would be a
form the mark silently cannot be set inside.
A position matching nothing is refused. Installing an unmarked body and answering ok would report a breakpoint
that is not there — the silent-success failure the session refuses everywhere else. The reply echoes :pause "LINE:COL" on success, and the editor draws its overlay off that echo rather than off what it asked for, so it can
never show a mark the session declined.
It sticks with no extra machinery. The marked declaration is what goes into Session.t.decls, so it stays marked
until an evaluation replaces it — an ordinary C-c C-c over the same form with no :pause, or C-c C-k over the
buffer. That is §9's settled behaviour and it costs one statement that was already there. A parallel paused list on
the session would have been a second source of truth that drifts the first time some path replaces decls without
touching it, and clearing would have had to be written rather than falling out.
C-u C-x C-e is a flag, not a position. flan-eval-last-sexp sends a raw buffer-substring with no line
padding — unlike flan-dev--text, which pads a snippet back onto its own line — so buffer coordinates do not survive
that path. They are also not needed: the expression sent is the target, so :pause t says everything there is to
say, and Session.eval_expr wraps the parsed expression before Check.expression. It does not stick and cannot: a
thunk is built and thrown away, so there is no declaration for the mark to live in.
The trap that path sets, and the reason wait is three-way. Dev.eval_expr waits five seconds for the thunk to
produce a value and otherwise answers "the program did not reach a frame boundary; is it calling (agent/poll)?" —
which is exactly what a thunk parked at a breakpoint looks like from the daemon's side. Left alone it would report the
working feature as a failure, after a five-second stall. So wait answers `Value | `Stopped | `Timeout, and the
Stopped question is asked only when a pause was requested: without one, a thunk that stops did so by erroring,
and the timeout message is the answer that path has always given — test_dev.ml pins it. The `Stopped reply
carries no :value, because there is not one yet; :stopped t :condition "Pause" rides on it the way it rides on
every reply, from with_break.
And it asks for Stopped "Pause" by name, not for "stopped at all". The break loop allows evaluating, so this
path is reachable from a program already parked on something else — and a match on Stopped _ would then fire on the
first iteration, answering for a thunk that has not run yet, on a reply whose own :condition names the other
condition. Asked by name it waits through the outer break until the thunk reaches its own (pause), which the agent
reports because a nested break overwrites condition_name and restores it on the way out (flan_agent.c, around the
status verb). A program already parked on a Pause is the one case this cannot tell apart, and nothing could: both
answers are "stopped at a pause".
The editor's column is a byte offset. flan-dev--wire-position is the inverse of flan-dev--position and has to
count bytes for the same reason: the reader walks the source a byte at a time, so current-column would be short by
one per extra byte in every non-ASCII character earlier on the line and the daemon would find nothing where it was
pointed. The line is the buffer's own, which works because flan-dev--text pads the snippet with leading newlines.
One key, three targets. flan-eval-defun takes C-u for the innermost form point is inside — backward-up-list,
falling back to the defun when point is not nested — and C-u C-u for the top-level form itself. With C-u C-x C-e
that is all three of §9's targets and no new binding; flan-mode.el did not change.
The overlay is an annotation, not feedback. flan-dev-pause-face is drawn over the marked form and deliberately
does not copy the error overlays' lifetime. An error overlay is about the command that just failed and the next
keystroke takes it down; a pause mark is about the running program, and it has to survive pre-command-hook or the
buffer stops showing a breakpoint that is still there. What clears it is what clears the mark itself: an accepted
evaluation with no :pause on it, over a region that intersects it — and C-c C-k clears the whole buffer, because
every declaration in it was just replaced. The face inherits warning rather than error, agreeing with
flan-cnr.el, which already renders Pause that way: a breakpoint is a stop, not a failure.
The test that matters is the second one. test_dev.ml's pause block marks step's (+ ticks 1), waits for
:stopped t :condition "Pause" and checks continue is on offer — and then re-evaluates the form plainly, takes
continue, and polls for half a second confirming it never stops again. A single sample after the resume proves
nothing: the frame that resumed is still inside the old marked body, past the (pause) call, so the first look reads
as running whether or not the mark was cleared. Half a second is about a hundred calls through the body just
installed. test/programs/dev-pause.flan exists because dev-loop.flan calls step four times, which is too tight
for that, and because a program that stops on its own — dev-break.flan — would prove nothing about what stopped it.