2772 lines
203 KiB
Markdown
2772 lines
203 KiB
Markdown
# How the parts that exist work
|
||
|
||
The reasons behind the code, kept separate from `NEXT.md` so that what to do next is not buried under what was already
|
||
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever `dlclose`d, why a call
|
||
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
|
||
than a function in the runtime. Deleting it would mean deriving it again.
|
||
|
||
`NEXT.md` is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
|
||
first. Come here when you need to know why something is the shape it is.
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a
|
||
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top
|
||
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer
|
||
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
|
||
bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at
|
||
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot,
|
||
so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed
|
||
type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is
|
||
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets
|
||
the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type
|
||
in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
|
||
per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
|
||
in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning
|
||
garbage rather than as a link error.
|
||
|
||
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
|
||
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
|
||
copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare`
|
||
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
### The wrappers are generated now — `declare-c`, `lib/shim.ml`
|
||
|
||
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
|
||
generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory
|
||
holds `raylib.flan` and `link` and no C at all.
|
||
|
||
One binding is now one line:
|
||
|
||
```
|
||
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
|
||
```
|
||
|
||
`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
|
||
any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the
|
||
function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an
|
||
ordinary Flan `defn` above it when the signature has a struct in it. `flan shim <file>` prints the whole file.
|
||
|
||
**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path
|
||
string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h
|
||
i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural
|
||
rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited.
|
||
|
||
**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same
|
||
`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what
|
||
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
|
||
that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real
|
||
signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was
|
||
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
|
||
for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct
|
||
— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves.
|
||
Everything where they could diverge is refused at the field.
|
||
|
||
**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a
|
||
scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the
|
||
hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against
|
||
the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree.
|
||
|
||
**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path,
|
||
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
|
||
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
|
||
malloc failure, where the alternative is handing C a null pointer.
|
||
|
||
**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice
|
||
and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is
|
||
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
|
||
recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes
|
||
`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned
|
||
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
|
||
the reason, and the acceptance table asserts on the reasons.
|
||
|
||
**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc +
|
||
`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
|
||
that calls an existing binding is unaffected.
|
||
|
||
`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
|
||
`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17
|
||
34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity
|
||
would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed.
|
||
|
||
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write.
|
||
|
||
Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows.
|
||
|
||
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
|
||
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
|
||
GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins
|
||
`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all —
|
||
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`,
|
||
`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four
|
||
ways for that reason.
|
||
|
||
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
|
||
library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora
|
||
ships the runtime library without the `.so` symlink.
|
||
|
||
### The header is read now — `headers`, `lib/cimport.ml`
|
||
|
||
The section above ends by naming what the generator *trusts*: that the
|
||
`defstruct` matches the library's real struct, and that the `declare-c`
|
||
signature is the function's real signature. "No header is read, deliberately,
|
||
so nothing can check either." A header is read now, and both are checked.
|
||
|
||
**The dependency, which is the crux, and which this project already answered
|
||
once.** Zig's old `@cImport` ran clang as a *library*. That is exactly the
|
||
dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++
|
||
library breaks routinely on upgrade, a binary on PATH does not. So this shells
|
||
out for `clang -Xclang -ast-dump=json -fsyntax-only`, which is the same binary
|
||
every build already runs and adds nothing that is not already being paid for.
|
||
`lib/cjson.ml` is enough JSON to read that dump and no more, so it adds no opam
|
||
package either.
|
||
|
||
A note worth recording, because it strengthens the argument rather than
|
||
weakening it: **Zig has since abandoned clang here too.** `translate_c.zig` is
|
||
gone; `lib/compiler/translate-c/` is built on Aro, a C frontend written in Zig.
|
||
Their reason was to ship a compiler containing no clang at all — the opposite
|
||
premise to this one, where `clang` on PATH *is* the toolchain assumption. Both
|
||
projects walked away from linking libclang; only the destination differs.
|
||
|
||
**What is imported: functions, and only functions.** Not structs, not enums,
|
||
not macros. The bound on how much is not a curated list but the package's own
|
||
`defstruct`s — a function whose signature mentions a struct the package has not
|
||
described is refused with that reason, so `vendor/raylib` describing thirteen
|
||
structs is what makes the import thirteen structs wide, and describing a
|
||
fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are
|
||
refused, and 172 are left alone because the package already binds them by hand.
|
||
|
||
Not generating `defstruct`s is what makes the check possible at all. Generate
|
||
them and the header becomes the authority on layout, and comparing the
|
||
package's layouts against the header's would be comparing the header with
|
||
itself. A `_Static_assert` on `sizeof`/`offsetof` was rejected in the section
|
||
above as circular for exactly that reason; **this is not circular, because the
|
||
two sides have different authors.** It is the cheapest real closure of the
|
||
trusted-not-guaranteed gap.
|
||
|
||
**Refusing by demotion, which is the one thing taken wholesale from Zig.** Zig's
|
||
translator never drops a declaration it cannot handle: `failDecl` binds the name
|
||
to a `@compileError` carrying the reason, so the name still exists, the program
|
||
still compiles, and asking for *that one name* fails at the use site with the
|
||
reason. A wholesale import has a hundred and fifty refusals and a caller cares
|
||
about the one they typed. Flan already had that mechanism — `Load.refuse_hidden`,
|
||
built for `main` — so `rl/get-gamepad-name` is not a name, and a program that
|
||
writes it is told *the return type is a string, and a string only crosses as a
|
||
parameter* rather than "unknown name".
|
||
|
||
That is also the split `shim.ml` needed and did not have. It refuses through
|
||
`Loc.fail`, which is right when a human named one function and wrong for a
|
||
wholesale import, where one returned `const char *` would kill the header. Same
|
||
judgement, different disposition; a hand-written `declare-c` still hard-fails
|
||
and `shim.ml` is untouched.
|
||
|
||
**Two C spellings mean something in a parameter that they mean nowhere else.**
|
||
`const char *` is a string going in, and the generator already knows how to hand
|
||
one over. `char *` without the const is very often a buffer the callee *writes*,
|
||
and handing it a NUL-terminated temporary would lose the writes with no
|
||
diagnostic anywhere — const is the only thing in the header that separates the
|
||
two, so it is what decides, and an out-buffer keeps a hand-written binding
|
||
saying `(Ptr u8)`. `long`, `size_t` and the rest are refused rather than
|
||
guessed, and for a reason specific to this project: it builds for x86-64, for
|
||
wasm32-wasi and for the browser, and `long` is 64 bits on the first and 32 on
|
||
the others, so a guess would be right for the target that gets tested and
|
||
silently wrong for two that do not.
|
||
|
||
**Naming.** `rl/InitWindow` is `rl/init-window`. Reversibility is not a property
|
||
of the rule — the C symbol is stored verbatim in the declaration, so the wrapper
|
||
reads the library's spelling rather than reconstructing it. What the rule must
|
||
be is *injective over one header*, since two C functions arriving under one Flan
|
||
name would surface as a duplicate declaration about a name nobody wrote. A
|
||
boundary goes before an uppercase letter after a lowercase one, before an
|
||
uppercase letter between an uppercase and a lowercase, and before a digit after
|
||
a lowercase; nowhere else. So `SetTargetFPS` is `set-target-fps` and not
|
||
`set-target-f-p-s`, `BeginMode2D` is `begin-mode-2d`, `UnloadUTF8` is
|
||
`unload-utf8`. raylib's 581 names are injective under it. **When two do collide,
|
||
neither takes the name** — resolving by order would mean that moving two lines
|
||
in somebody else's header silently rebinds a name a program is already calling.
|
||
Both are refused, both say why, and the author binds the one they want with a
|
||
`declare-c`.
|
||
|
||
**Where it runs.** `headers` beside `link` in the package directory, read the
|
||
same way: a path, any clang flags it needs, `${NAME}` expanded from the
|
||
environment. What comes back is ordinary `declare-c` declarations, generated
|
||
before the package's names are qualified, so they arrive as `rl/…` exactly like
|
||
the hand-written ones and nothing downstream can tell which is which. No new
|
||
form, no new `decl_kind`, no reader or parser change. A C symbol the package
|
||
already binds by hand is left alone, so `declare-c` stays the escape hatch and
|
||
stays the thing that wins.
|
||
|
||
A leading `?` makes a line optional, and `vendor/raylib` uses it. "No raylib
|
||
headers are needed" is a real property — a build needs libraylib linkable, not
|
||
raylib-devel installed — and requiring a header would take it from everyone in
|
||
order to give the check to whoever has one. Unset `FLAN_RAYLIB_H` and the build
|
||
is exactly what it was; set it and every signature is checked. A path that is
|
||
*set and wrong* is an error naming it, because silently behaving as though
|
||
nobody had opted in is the difference between an opt-in and a trap.
|
||
|
||
#### What the diff found
|
||
|
||
The evidence the whole lane exists for. Against raylib **5.5** — the version
|
||
whose `.so` `link` names — **all 16 `defstruct`s and all 172 hand-written
|
||
`declare-c` agree exactly.** The half BUILT.md called trusted is now checked,
|
||
and it was right.
|
||
|
||
That is only worth stating because the check has teeth. Against the **5.1-dev**
|
||
header installed in `/usr/local` it reports ten differences: nine functions that
|
||
version does not have (`CheckCollisionCircleLine`, the six `Is*Valid` renames,
|
||
`DrawRectangleRoundedLinesEx`) and `DrawRectangleRoundedLines`, which gained a
|
||
parameter. Picking the wrong header is therefore loud, which matters, because
|
||
the two headers are on the same machine and only one matches the linked library.
|
||
|
||
Both comparisons run **at build time** and stop the build, not just in the tool.
|
||
Verified by breaking them: a permuted `Texture2D` fails naming the field that
|
||
moved, and `f64` where raylib says `float` fails naming the parameter — which is
|
||
the hazard the section above calls out by name and says only a test can catch.
|
||
The message points at the line in `raylib.flan`, not at the header.
|
||
|
||
`flan import-c <header> [package.flan…]` prints what it would produce, what it
|
||
refused and why, and both comparisons, without building anything. That also
|
||
makes "generate once and commit the result" available for the cost of a
|
||
printer — explicit in the source, checked against reality, no header read at
|
||
build time.
|
||
|
||
#### What it costs, measured
|
||
|
||
The number that decides how much to import, because `reach.ml` was the reason to
|
||
think a wholesale import could be free.
|
||
|
||
| | today (172 by hand) | + 256 imported |
|
||
|---|---|---|
|
||
| release build, cold | 0.298s | 0.312s |
|
||
| release build, warm | 0.078s | 0.082s |
|
||
| redefinition (`flan reload`) | 31.0ms | 46.5ms |
|
||
| dev build, cold | 0.649s | 0.982s |
|
||
|
||
**`Reach.link` already drops a generated wrapper whose declaration nothing
|
||
reachable calls, and that is what makes the release column nearly flat.**
|
||
Confirmed on the case it exists for: a wasm32-wasi build of a program that
|
||
imports raylib and calls none of it still links without libraylib, with 256
|
||
extra declarations in play. Dev builds are not pruned, on purpose, so one
|
||
compiles all 428 wrappers — once, at session start, since `Build.shared` is
|
||
llc + `ld -shared` and compiles no C.
|
||
|
||
Reading the header is cached, and the cache earned itself against a measurement
|
||
rather than a guess: 64ms of a 72ms check, against 8ms for the whole program
|
||
without it. What is cached is the *extracted* signatures and not clang's JSON,
|
||
because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest
|
||
is reading it. Keyed the way the object cache is keyed, on everything that could
|
||
change the answer: the header's path, size and mtime, the full flag list, and a
|
||
format version, since the value is marshalled. That takes the delta to 17ms.
|
||
|
||
**The 15.5ms on redefinition is the real cost and it is the argument against
|
||
importing at build time**, on the branch where the dev loop is the priority. It
|
||
is the strongest case for the third option — generate from the header, commit
|
||
the result, regenerate when the library moves — and that decision is open.
|
||
|
||
#### Where that 15.5ms actually is — and it is not the header
|
||
|
||
Written above as though a redefinition were re-reading the header on every
|
||
reload. It is not, and the number was never taken apart. Taken apart now, with a
|
||
timer around each stage of the same `flan reload` (raylib 5.5, sand.flan, one
|
||
`defn` in the forms file, warm caches):
|
||
|
||
| | no header | header imported | delta |
|
||
|---|---|---|---|
|
||
| `Session.create` | 7.0ms | 21.5ms | +14.5 |
|
||
| `Session.eval` | 4.8ms | 8.4ms | +3.6 |
|
||
| `llc` + `ld` | 21ms | 21ms | 0 |
|
||
|
||
And inside that `create`, the header's own share: reading the cached dump
|
||
**0.33ms**, turning it into declarations **3.3ms**, checking the package's
|
||
layouts and its hand-written signatures against it **0.55ms**. Roughly 4ms of
|
||
the 14.5, once. The other ~10ms is `Load` and `Check` doing their ordinary work
|
||
over 256 more declarations, and so is the +3.6ms on `eval`.
|
||
|
||
**So a `C-c C-c` in a daemon pays no header cost at all, and never did.**
|
||
`Session.eval` puts the evaluated forms through `Load.program`, and a form list
|
||
with no `(import …)` in it reads no package and therefore no header. The import
|
||
runs when the session is created, and again on a `C-c C-k` whose buffer carries
|
||
the package's own `import` line. `flan reload` is a fresh process per
|
||
redefinition, which is why its number carries a session's startup cost — it is
|
||
the CLI's shape, not the dev loop's.
|
||
|
||
What a redefinition does pay for an imported package is the +3.6ms of checking
|
||
and emitting against a program with 256 more declarations in it. That is in
|
||
`Check` and in `Emit.redefinition` declaring every sibling; it is real, it is
|
||
the number to attack next, and no cache touches it.
|
||
|
||
#### Two caches, and what a header change mid-session means
|
||
|
||
Both levels exist now. **On disk**, keyed on the header's realpath, size, mtime,
|
||
the full flag list and a format version — the object cache's own convention, and
|
||
it now lives in the object cache directory rather than a second one beside it.
|
||
Cold that read is 50ms of clang and parse; served from the file it is 0.33ms.
|
||
|
||
**In the session**, two tables in `Cimport`: the extracted dump by header, and
|
||
the generated declarations by header *and* by what the package already declares
|
||
(names taken, structs and enums known, C symbols bound by hand), so a package an
|
||
evaluation has added a decl to is worked out again rather than served a stale
|
||
answer. A repeat import in one process goes from 3.65ms to nothing — measured on
|
||
a whole-buffer evaluation, which imports twice.
|
||
|
||
**A header edited mid-session does not take effect until the session is
|
||
restarted**, and that is deliberate rather than incidental: the in-memory key is
|
||
the header's path and flags, with no mtime and no `stat`. It is the rule a
|
||
changed `.c` file already follows — the daemon compiled the package's C at
|
||
startup and a redefinition does not recompile it — and the rule the running
|
||
program follows, since its struct layouts are the ones it was built with. Taking
|
||
a new header mid-session would mean type-checking new signatures against a
|
||
process still running the old layouts, which is the silent disagreement the
|
||
whole header check exists to catch. The disk cache does key on mtime, so the
|
||
next session reads the new header rather than the old dump.
|
||
|
||
One attribution to not repeat: the disk cache is **not** what the dev build's
|
||
+333ms cold is about. Measured on this machine, a cold dev build of sand.flan is
|
||
1.37s without the header and 1.83s with it and both caches cold; clear only the
|
||
header cache and the same build is 0.92s against 0.83s. So the header read is
|
||
~60–90ms of a fresh session and the rest of the cold delta is the object cache
|
||
compiling a shim with 428 wrappers in it, which is the object cache's business
|
||
and already warm after one build.
|
||
|
||
### What a headless FFI test can and cannot pin
|
||
|
||
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
|
||
|
||
- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C
|
||
writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and
|
||
getting identical output.
|
||
- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the
|
||
way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance,
|
||
passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from.
|
||
Distances are worse: the reflection does not even reach them.
|
||
- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does
|
||
not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision
|
||
cases look like they cover it.
|
||
- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different
|
||
field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.
|
||
|
||
- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available.
|
||
`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values
|
||
in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely,
|
||
including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU
|
||
reads its width, height or mipmaps at all.
|
||
- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image
|
||
`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and
|
||
answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more.
|
||
- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the
|
||
package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the
|
||
`x`/`y` permutation above.
|
||
|
||
The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail
|
||
by permuting the fields and watching it go red. A case not verified that way is decoration.
|
||
|
||
One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It
|
||
measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no
|
||
window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime`
|
||
and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are
|
||
exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is
|
||
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares
|
||
becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
|
||
declarations whose names contain a slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file
|
||
named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below.
|
||
|
||
**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also
|
||
a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import
|
||
all four. A file carries no `.c` and no `link` file; those belong to a directory.
|
||
|
||
**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that
|
||
is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under
|
||
one set of names or the checker sees every declaration twice. It is the one decision here that would be hardest to
|
||
change later, because a qualified name is an identity the `layout` op and the break loop both resolve by.
|
||
|
||
A directory is keyed by its real path and read once, so a **diamond** shares one copy of its bottom package.
|
||
`programs/pkg-diamond.flan` is the case that proves it at the level that matters: `area/` and `draw/` both import
|
||
`shape/`, a `shape/Box` is built inside `area/` and passed to a function declared inside `draw/`. Read `shape/` twice
|
||
and there are two structs both named `shape/Box` which do not unify, so what proves the dedupe is that the program
|
||
prints `3 6 20` and not that it compiles.
|
||
|
||
The same directory under two *different* aliases is refused, including when one of the two is a package's own import
|
||
and the other is pages away in the entry file — `programs/pkg-alias-clash.flan`.
|
||
|
||
**A ring is refused and named.** Earlier it was not: the read-once table swallowed the second arrival, so mutually
|
||
dependent packages appeared to work, and BUILT.md said so. What that cost is a definite package order, which is what
|
||
the macro expander needs — every `defmacro` has to be compiled before anything that calls it — so it is now a refusal
|
||
that names the ring, `a -> b -> c -> a`, rather than the single import that happened to close it. The two questions are
|
||
kept apart by two pieces of state: the chain currently being read, and the set already finished. Found in the first is
|
||
a cycle; found only in the second is the diamond's second route and still a no-op.
|
||
|
||
`Load.t.pkgs` comes back **dependencies first** — the topological order the acyclic rule buys. The declaration list is
|
||
deliberately unsorted: `check.ml` collects every top-level name in one pass before it checks any body, so top-level
|
||
names are order-independent by construction. The order exists for the expander, which cannot work that way.
|
||
|
||
**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment
|
||
anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would
|
||
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
|
||
checker it would be "unknown name", which is true and useless.
|
||
|
||
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
|
||
The gap is surface syntax and not `load.ml` — `exported` is one predicate and the refusal machinery that points at the
|
||
line which tried already exists, so a second rule is a line. What does not exist is any way for a package to *mark* a
|
||
name private, and inventing one is a reader and parser change.
|
||
|
||
## The link follows the program
|
||
|
||
`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the
|
||
importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that
|
||
link cannot succeed.
|
||
|
||
`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it,
|
||
follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing
|
||
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
|
||
|
||
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
|
||
`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs**
|
||
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
|
||
dropped one is a silently different program.
|
||
|
||
**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far.
|
||
|
||
The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and
|
||
`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker
|
||
arguments together, so a caller cannot take one without the other.
|
||
|
||
## sand.flan is one program
|
||
|
||
It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a
|
||
program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both
|
||
drivers imported it.
|
||
|
||
Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports
|
||
`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
|
||
raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both
|
||
targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation.
|
||
|
||
What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no
|
||
input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and
|
||
`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
Two claims that got run together in an earlier note, for the record:
|
||
|
||
- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the
|
||
**wasi** path, which is what the headless table targets, and which has no GL and no browser.
|
||
- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with
|
||
`emscripten_set_main_loop` instead of a `while`. ~~That is a different `main`, not a different program.~~ **The premise
|
||
held and the conclusion did not.** It is the same `main` and the same program: `-sASYNCIFY` answers the same browser
|
||
fact without cutting anything in half. See "The browser is the third target" below.
|
||
|
||
**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`,
|
||
and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`.
|
||
- `(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what
|
||
cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three
|
||
check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo
|
||
<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the
|
||
same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor
|
||
that split them would break the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s
|
||
unchecked. Indistinguishable.
|
||
|
||
## Sanitizers, and why hand-written IR does not get them for free
|
||
|
||
`--sanitize` builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last
|
||
of those is not what passing `-fsanitize=address` to the clang run over the `.ll` gets you, and the gap is silent.
|
||
|
||
**AddressSanitizer is an LLVM pass, but it instruments only functions carrying the `sanitize_address` attribute.** That
|
||
attribute is put there by clang's C frontend. `Emit` writes `.ll` by hand, so it wrote none, so the pass walked past
|
||
every Flan function and instrumented `flan_rt.c`. The measurement that settled it: an out-of-bounds read of a `defvar`
|
||
array in a `--no-bounds-checks` build printed its garbage and exited 0; with an `attributes #0 = { sanitize_address }`
|
||
group named on every `define`, the same program reports `global-buffer-overflow in flan.main`. Globals are the exception
|
||
— the module pass redzones them whether or not any function is attributed — which is why the *shape* of a sanitized
|
||
build looked right long before it worked.
|
||
|
||
**UndefinedBehaviorSanitizer has no equivalent lever.** Its checks are not a pass: the C frontend emits branches to
|
||
`__ubsan_handle_*` inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and
|
||
`(<< 1 32)` is still unremarked under `-fsanitize=undefined`. Shift UB, alignment and the f32→i32 cast on NaN are
|
||
therefore a compiler feature if they are wanted — checks emitted from `Emit` behind the flag, the same shape the bounds
|
||
checks already have — and not a flag away. `test_sanitize` pins both halves with controls: one program that must report
|
||
and one that must not, so either fact changing is a test failure rather than a discovery.
|
||
|
||
`-fno-sanitize=signed-integer-overflow` is the only exclusion, because wrapping is what this language's arithmetic
|
||
means and without it every program trips on its first `+`.
|
||
|
||
**The flag deliberately does not force `-O0`,** unlike `--debug`, whose reason (mem2reg deletes the alloca a
|
||
`llvm.dbg.declare` describes) does not apply. The optimiser is half of what is being measured, and `bounds.flan` proves
|
||
it: with checks off, its read past the end of a string constant is reported at `-O0` and silent at `-O2`, because an
|
||
out-of-bounds `inbounds` getelementptr into a constant is poison and LLVM folds the load away. The program then prints a
|
||
wrong answer instead of touching memory. Same family as `(<< 1 32)` compiling to a bare `retq`.
|
||
|
||
**What ASan covers of the bounds checks' job, since `--sanitize --no-bounds-checks` is the run that asks.** Three of
|
||
`bounds.flan`'s six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is
|
||
layout rather than anything about the access: ASan lays a global out as `{data, redzone}`, so reading before one lands
|
||
in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is.
|
||
Measured both ways — silent in `bounds.flan`, reported as soon as another `defvar` is declared in front of `arr`. A
|
||
reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees
|
||
out-of-*object* access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical
|
||
bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a
|
||
second net, not a replacement.
|
||
|
||
The sweep lives on its own dune alias rather than on `dune test`: a sanitized program links to a statically linked 1.8MB
|
||
binary, and twenty-eight of them twice over is minutes against the suite's seconds.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had
|
||
expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is
|
||
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
|
||
criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the
|
||
semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is*
|
||
the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody
|
||
addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a
|
||
`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding,
|
||
so wasm32 needs no exception proposal.
|
||
|
||
## The reload primitive — dev loop steps 1 and 2, measured
|
||
|
||
`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled
|
||
into its own object and called inside a process that is already running, twice, with a changed body the second time.
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| `Emit.redefinition` | below the timer (<0.1ms) |
|
||
| `llc -O2 -filetype=obj` | 15–17ms |
|
||
| `ld -shared` | 3ms |
|
||
| `dlopen` + `dlsym` | **0.04ms** |
|
||
|
||
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
|
||
number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost,
|
||
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since
|
||
the driver absorbs IR the bare tools reject.
|
||
|
||
`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that
|
||
*is* the mechanism. What the new module does **not** define is the whole design:
|
||
|
||
- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a
|
||
redefinition can change a function's body and can never re-initialise the program's data. Define the global and the
|
||
loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the
|
||
thesis.
|
||
- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a
|
||
private copy of it.
|
||
- **no `main`.** This module is loaded, not started.
|
||
|
||
Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss
|
||
because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is
|
||
spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a
|
||
slice parameter.
|
||
|
||
**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the
|
||
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
|
||
agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release
|
||
builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_read`, which reads as a compiler
|
||
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and
|
||
the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason.
|
||
|
||
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a
|
||
loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property
|
||
of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol.
|
||
|
||
Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**,
|
||
since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since
|
||
`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
|
||
`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares
|
||
`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
|
||
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
|
||
|
||
String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's —
|
||
worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would
|
||
fail at link time. The fixtures each print a literal so that path is actually exercised.
|
||
|
||
### Cells — how a call site follows a redefinition
|
||
|
||
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes
|
||
every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current.
|
||
|
||
```
|
||
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
|
||
%p = load ptr, ptr @"flan.cell.bump" ; every call site
|
||
%r = call i64 %p()
|
||
```
|
||
|
||
Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and
|
||
exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what
|
||
makes a frame-boundary swap a non-event.
|
||
|
||
The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of
|
||
one.
|
||
|
||
Four things about this that are not free choices:
|
||
|
||
- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on
|
||
whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are
|
||
separate on purpose.
|
||
- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to
|
||
*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would
|
||
publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage,
|
||
because the failure is silent.
|
||
- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined
|
||
function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture
|
||
recurses on purpose, and would print the old body's text if it did not.
|
||
- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is
|
||
the first time `opts` means something semantic rather than an optimisation level.
|
||
|
||
LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46
|
||
indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well;
|
||
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
|
||
|
||
### Names that did not exist when the process started
|
||
|
||
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF
|
||
cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else:
|
||
|
||
```
|
||
void **flan_dev_cell(const char *name); /* a new function's cell */
|
||
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
|
||
```
|
||
|
||
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
|
||
new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores
|
||
on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
|
||
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
|
||
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
|
||
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
|
||
common case pays nothing for the general one.
|
||
|
||
**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c
|
||
C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a
|
||
redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet.
|
||
|
||
Four rules, each of which is a silent failure if broken:
|
||
|
||
- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are
|
||
still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`.
|
||
- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the
|
||
old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so.
|
||
This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
|
||
- **Nothing that published anything is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it
|
||
leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being
|
||
*pointed into*, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient
|
||
thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer,
|
||
or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the
|
||
handle *value* rather than the mapping: `dlopen` refcounts by path, so re-sending the same bad file raised a count
|
||
nothing could lower, and the one reference that could was dropped on the floor.
|
||
- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed
|
||
capacity with a loud failure rather than growable.
|
||
|
||
The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at
|
||
run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell*
|
||
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
|
||
pass and the transcript would read 246 instead of 432.
|
||
|
||
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout
|
||
calculator in OCaml that would have to agree with LLVM's on every target.
|
||
|
||
### The agent — dev loop step 3
|
||
|
||
`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link`
|
||
asks for `-lpthread`.
|
||
|
||
```
|
||
(agent/start path) listen on a unix socket; once, at startup
|
||
(agent/poll) install whatever has arrived; returns how many
|
||
(agent/wait ms) the same, but waits for something first
|
||
```
|
||
|
||
The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded —
|
||
so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a
|
||
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
|
||
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
|
||
loader.
|
||
|
||
**A full ring is refused, at the sender, before the `dlopen`.** Of the three honest answers this is the only one that
|
||
reaches the person who asked: dropping loses a reload the sender was told was `ok`, which is the same lie more quietly,
|
||
and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also
|
||
stop answering `status` and `abort`, leaving the dev loop with no way to reach a program that had stopped listening to
|
||
it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away,
|
||
since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says
|
||
what to do about it — call `agent/poll`.
|
||
|
||
**The way out of the break loop is `_exit`, not `exit`.** `exit` runs the atexit chain and the ELF destructors, which
|
||
want the loader lock the listener thread may be holding inside `dlopen`; a program asked to abort would hang instead of
|
||
dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134
|
||
stays because that is what a trap exits with.
|
||
|
||
`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits
|
||
for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first
|
||
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
|
||
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
|
||
|
||
Two details found by running it:
|
||
|
||
- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a
|
||
program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at
|
||
all, which is how this was found.
|
||
- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program
|
||
can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`.
|
||
- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows
|
||
when it is between frames.
|
||
|
||
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under
|
||
Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body
|
||
that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is
|
||
refused on the listener thread and the loop never notices.
|
||
|
||
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the
|
||
names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what
|
||
the session will track automatically.
|
||
|
||
### The session
|
||
|
||
`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change
|
||
accepted since.
|
||
|
||
**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a
|
||
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
|
||
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that
|
||
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
|
||
|
||
Two things the session knows that no single evaluation could:
|
||
|
||
- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST,
|
||
because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `println` reads
|
||
as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
|
||
- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded:
|
||
|
||
| Change | What it would have broken |
|
||
|---|---|
|
||
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below |
|
||
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
|
||
| a struct's fields | the values the process is holding have the old layout |
|
||
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it |
|
||
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
|
||
|
||
**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should
|
||
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
|
||
existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked
|
||
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
|
||
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
|
||
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
|
||
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
|
||
should not be read as the final answer.
|
||
|
||
A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past
|
||
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global`
|
||
record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than
|
||
the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is
|
||
the same fact that makes them unreloadable.
|
||
|
||
Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first,
|
||
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
|
||
uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for
|
||
that reason.
|
||
|
||
A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A
|
||
dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a
|
||
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
|
||
is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the
|
||
folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could.
|
||
|
||
**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in
|
||
`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s
|
||
own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a
|
||
brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it
|
||
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
|
||
did nothing.
|
||
|
||
It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a
|
||
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
|
||
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
|
||
either.
|
||
|
||
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise
|
||
re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name
|
||
pass rejects it. `C-c C-k` on sand.flan's own text is the test.
|
||
|
||
`flan reload <program.flan> <forms.flan>` is that path from the command line: a session over the program the process was
|
||
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw`
|
||
and 910 consecutive frames drew it.
|
||
|
||
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
|
||
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
|
||
has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to
|
||
`<eval>`, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
|
||
the real buffer path, which is the same key CIDER's `eval` carries.
|
||
|
||
### The daemon — `flan dev`
|
||
|
||
`flan dev <program.flan>` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside
|
||
the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part
|
||
of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe
|
||
the process that is actually running rather than a guess about it.
|
||
|
||
**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client
|
||
became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form,
|
||
from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the
|
||
editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of
|
||
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
|
||
newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor.
|
||
|
||
```
|
||
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
|
||
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
|
||
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
|
||
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
|
||
(:op "close")
|
||
```
|
||
|
||
`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the
|
||
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
|
||
It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing
|
||
else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none
|
||
to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself,
|
||
which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step
|
||
[i64 f32] i64`: types only.
|
||
|
||
The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan
|
||
dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by
|
||
guessing which directory it was relative to.
|
||
|
||
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
|
||
— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one
|
||
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
|
||
|
||
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so without it every error an editor shows
|
||
points into a file that does not exist.
|
||
|
||
Two things the daemon must not paper over, both of which would look like a successful evaluation:
|
||
|
||
- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan
|
||
says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead
|
||
fails silently: the module compiles, is built, and nobody receives it.
|
||
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed
|
||
connect or a reply that is not `ok` becomes an error the editor sees.
|
||
|
||
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads
|
||
like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather
|
||
than leaving an editor waiting on a socket nobody is serving.
|
||
|
||
### The Emacs client
|
||
|
||
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is
|
||
s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is
|
||
Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is
|
||
written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords.
|
||
|
||
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a
|
||
request and `read` reads a reply.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `C-c C-c` | the top-level form at point, recompiled and installed |
|
||
| `C-c C-k` | the whole buffer, as **one** module |
|
||
| `C-x C-e` | the expression before point, evaluated *in the running program* |
|
||
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
|
||
| `C-c C-o` | the running program's own output, in `*flan-output*` |
|
||
| `C-c C-r` | a prompt on the running program (`*flan-repl*`) |
|
||
| `C-c C-b` | what a **stopped** program is offering, and which to take |
|
||
| `C-c C-d` | what the running program currently defines |
|
||
| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR |
|
||
| `M-.` / `M-,` | where a name is written, through an `xref` backend |
|
||
|
||
eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on
|
||
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
|
||
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just
|
||
installed completes at once.
|
||
|
||
The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away,
|
||
which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the
|
||
socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the
|
||
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
|
||
|
||
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to
|
||
arrive in the same load, or the first refers to storage that does not exist yet.
|
||
|
||
**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary,
|
||
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
|
||
has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives
|
||
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
|
||
mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml`
|
||
and fails here.
|
||
|
||
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
|
||
next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte
|
||
offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
|
||
`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the
|
||
daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1
|
||
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
|
||
reader skips, so the reply's line numbers are the buffer's own.
|
||
|
||
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
|
||
success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the
|
||
one point looked like it was in.
|
||
|
||
**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to
|
||
into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form
|
||
"asm"|"ir")` is the op.
|
||
|
||
What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d
|
||
--disassemble=flan.<name>` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared`
|
||
deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was,
|
||
so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else
|
||
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
|
||
bookkeeping.
|
||
|
||
**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket
|
||
takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address
|
||
space, and `C-x C-e` renders a pointer as `<ptr>` on purpose — so nothing the daemon can ask would tell it what a cell
|
||
holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first
|
||
instruction:
|
||
|
||
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*;
|
||
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
|
||
- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed,
|
||
since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further
|
||
installs until it resumes.
|
||
|
||
From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch
|
||
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
|
||
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
|
||
the debug build lands, that is the line to delete.
|
||
|
||
**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an
|
||
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
|
||
`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the
|
||
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
|
||
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
|
||
in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the
|
||
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
|
||
is Emacs' contract and a test claiming to check it would be checking nothing.
|
||
|
||
**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the
|
||
next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the
|
||
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
|
||
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
|
||
accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
|
||
grow the daemon without limit.
|
||
|
||
### `C-x C-e` — evaluating an expression
|
||
|
||
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
|
||
into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by
|
||
exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an
|
||
expression reading the program's state sees a point the program agrees is consistent.
|
||
|
||
**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say
|
||
what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout
|
||
decision's bill, and it is why the printer set is small rather than universal.
|
||
|
||
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
|
||
dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's
|
||
socket. The read is safe without a handshake because the counter is a **seqlock**, and it had to be made into a real
|
||
one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived
|
||
and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to
|
||
be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the
|
||
bare pointer was the bug rather than the ordering. `flan_dev_result_read` copies into the caller's buffer and checks
|
||
the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader
|
||
that loses the race reports the last *complete* generation and no bytes, so a daemon polling for a new value keeps
|
||
polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's
|
||
"has it moved" still means what it meant. Marking the counter odd needs a release *fence* and not a release store — a
|
||
release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which
|
||
is the original bug with more ceremony. And the odd mark is *set* rather than incremented, because a `begin` with no
|
||
`end` is reachable: the thunk calls `begin` before it evaluates anything, so an expression that signals is stopped
|
||
inside that window, and a restart taken from the break transfers past the thunk and `end` never runs. Incrementing
|
||
would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit
|
||
means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has
|
||
reached a frame boundary.
|
||
|
||
**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a
|
||
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
|
||
structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth
|
||
and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three
|
||
bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a
|
||
builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time
|
||
type to walk. Not the walk itself.
|
||
|
||
The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise
|
||
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
|
||
allocator the language does not have.
|
||
|
||
```
|
||
big 18446744073709551615
|
||
col :blue
|
||
(.pos b) (V {:x 1.5 :y 0})
|
||
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
|
||
(slice (.tags b) 0 3) [ 0 42 0]
|
||
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
|
||
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
|
||
```
|
||
|
||
Details that are decisions rather than formatting:
|
||
|
||
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come
|
||
back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point
|
||
instead.
|
||
- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather
|
||
than as the value it is.
|
||
- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are
|
||
erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is
|
||
exactly what you would want to see.
|
||
- **A pointer is never followed** — `<ptr>`. It is the only thing that could make the walk cycle, and dereferencing one
|
||
a REPL was handed is not a safe thing to do on someone's behalf.
|
||
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does
|
||
not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K
|
||
and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it
|
||
means no renderer carries a budget.
|
||
- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is
|
||
not evaluated once per element.
|
||
|
||
What still refuses by name: `Map`, `Fn`, a type variable.
|
||
|
||
A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer.
|
||
`flan run calc-me.flan "1.5 * 2.0"` has always said `3`.
|
||
|
||
An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe`
|
||
does not fill up with `eval/N` for every expression ever typed.
|
||
|
||
**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by
|
||
`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
|
||
into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the
|
||
agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition*
|
||
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
|
||
and can never claim this.
|
||
|
||
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop
|
||
would exhaust it.
|
||
|
||
The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must
|
||
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
|
||
|
||
### The REPL buffer
|
||
|
||
`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No
|
||
new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as
|
||
deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess
|
||
behind it — the "process" is a stub comint needs in order to have a prompt at all.
|
||
|
||
Three things about it that are decisions:
|
||
|
||
- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in
|
||
sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it
|
||
because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
|
||
- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a
|
||
paren inside a string does not count.
|
||
- **A value and the program's output are different things and arrive by different routes.** The value is the result of
|
||
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
|
||
and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the
|
||
separation.
|
||
|
||
That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so
|
||
`(println "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A Unit
|
||
expression is almost always a call made for its effect, and is now evaluated and *then* reported.
|
||
|
||
### Conditions — step 1: `handler-bind` and `signal`
|
||
|
||
`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters
|
||
control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling
|
||
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
|
||
yet, and no signature changed.
|
||
|
||
```
|
||
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
|
||
(load-all))
|
||
```
|
||
|
||
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
|
||
function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame
|
||
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
|
||
step.
|
||
|
||
Three decisions worth keeping:
|
||
|
||
- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and
|
||
every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
|
||
- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is
|
||
nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the
|
||
name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address
|
||
instead.
|
||
- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a
|
||
branch in the function that wrote it.
|
||
- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from
|
||
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
|
||
particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the
|
||
duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still
|
||
valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the
|
||
stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When
|
||
`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
|
||
|
||
Which gives the two refusals, both by the house rule rather than by accident:
|
||
|
||
- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a
|
||
reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in
|
||
scope, which is what the accumulation case needs.
|
||
|
||
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
|
||
outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by
|
||
value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete
|
||
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
|
||
biggest usability limit in conditions is not behind the thing that was just deferred.
|
||
- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would
|
||
leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block.
|
||
|
||
### The break loop — conditions step 3
|
||
|
||
Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the
|
||
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
|
||
|
||
```
|
||
flan: unhandled Missing — stopped, not dead.
|
||
0. restart: retry
|
||
1. restart: use-placeholder
|
||
```
|
||
|
||
Four decisions, each of which is the reason something is where it is:
|
||
|
||
- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the
|
||
release runtime and must not depend on something a program may never import. A program with no agent leaves the hook
|
||
null and dies exactly as it did before.
|
||
- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes,
|
||
reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same
|
||
act, lowered once. Nothing about §6 needed changing to support it.
|
||
- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a
|
||
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
|
||
that did not drain that queue would hang `C-x C-e` exactly when it is most wanted.
|
||
- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must
|
||
not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new —
|
||
and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the
|
||
cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of
|
||
stopping.
|
||
|
||
**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but
|
||
a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also
|
||
`compute-restarts`' data, whenever that arrives.
|
||
|
||
**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused
|
||
there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that
|
||
cannot happen.
|
||
|
||
The socket verbs are `restarts`, `restart-at <n> [name]`, `restart <name>` and `abort`, and all four are refused with
|
||
the reason when the program is not stopped — there is no restart stack to walk from a running one. `restarts` answers
|
||
`<index> <+|-> <name>` per line: the index is the identity, and the flag says whether a transfer to that frame has
|
||
anywhere to land. `test/programs/break.flan` errors three times — two restarts taken by name, then a shadowed pair
|
||
where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved
|
||
by name could pass.
|
||
|
||
### The break loop in the editor — conditions step 3, the other half
|
||
|
||
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it
|
||
follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a
|
||
question an editor chose to ask.
|
||
|
||
So the state is learned **twice, deliberately**:
|
||
|
||
- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`,
|
||
or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now
|
||
errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding
|
||
out *after* the echo area had said the evaluation was fine.
|
||
- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is
|
||
drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies
|
||
that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left
|
||
behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the
|
||
`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs
|
||
timers, so a poll firing inside a read would eat the reply that read was waiting for.
|
||
|
||
Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is
|
||
stopped, so the poll and the prompt cannot disagree.
|
||
|
||
```
|
||
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
|
||
:stopped t :condition "Missing")
|
||
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
|
||
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
|
||
(:op "abort") → (:status "ok" :note "the program is exiting; …")
|
||
```
|
||
|
||
`:restarts` is *positional* — innermost first, duplicates kept — because the position is what `restart-at` takes.
|
||
`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is
|
||
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
|
||
prompt cannot take a different restart than the one it showed.
|
||
|
||
`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by
|
||
someone about to choose from it.
|
||
|
||
**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against
|
||
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
|
||
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
|
||
lets the next poll settle it.
|
||
|
||
The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running
|
||
one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly
|
||
right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key —
|
||
it is the thing you pick when none of the restarts is the answer.
|
||
|
||
**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A
|
||
`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
|
||
cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the
|
||
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
|
||
breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices
|
||
each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an
|
||
expression that errors and resumes it, which fails against the old shape.
|
||
|
||
The agent grew one verb, `status`, answered in **both** states — `running` or `stopped <condition>`. Everything else the
|
||
break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had
|
||
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
|
||
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
|
||
|
||
`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the
|
||
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
|
||
round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e`
|
||
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
|
||
|
||
### `layout` — a type's fields, with no program involved
|
||
|
||
```
|
||
(:op "layout" :type "sim/Cell") → (:status "ok" :type "sim/Cell"
|
||
:fields (("heat" "f32") ("next" "(Option sim/Cell)")))
|
||
→ (:status "error" :message "Missing is not a qualified name; …"
|
||
:candidates ("a/Missing" "b/Missing"))
|
||
```
|
||
|
||
The daemon can answer this with nothing running. A layout is a fact about the *build*, and the daemon owns the build —
|
||
`Tast.structs` is sitting in the session it compiled the process from. That is why the conditions buffer can name and
|
||
type a condition's fields while every one of their *values* stays refused: the shape is knowable and the contents are
|
||
not, and drawing them apart says more than drawing neither.
|
||
|
||
**The type is a name, and the name is the qualified one.** This was the open question — a class name is not an identity,
|
||
and two packages each declaring `Missing` would leave the daemon unable to pick. It turned out to need no new
|
||
machinery: `Load.qualify_decl` rewrites `Defstruct (n, …)` to `Defstruct (alias/n, …)` at import, so by the time
|
||
anything reaches `Tast.structs` the names are one flat namespace in which a collision cannot exist. The name *is* the
|
||
type id, with no table to keep in step across a reload, and the existing spelling of a type — `Types.to_string` —
|
||
already prints it.
|
||
|
||
**And the break loop was already speaking it.** `Emit.struct_name_of` takes `Types.Named n` — the qualified name — and
|
||
passes it to `flan_error`; the agent holds it in `condition_name`; `break` answers it as `:condition`. So the string
|
||
the conditions buffer already had in hand resolves as `:type` by construction, and `test_dev.ml` round-trips exactly
|
||
that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat
|
||
worth writing down — `condition_name` is a `char[128]`, so the round trip holds for names up to 127 bytes and a longer
|
||
one is truncated and will not resolve.
|
||
|
||
**A bare name is refused, not resolved**, even when only one struct's last segment matches it. Resolving a unique
|
||
suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot
|
||
rely on. The refusal carries `:candidates`, so a person is one copy-paste from the answer and a client has its
|
||
completion list — the same shape as `package_of` refusing a directory imported under two aliases rather than picking
|
||
one. An enum is refused by *kind* (`X is an enum, not a struct`): its members are erased to `i32` before `Tast.program`
|
||
exists, which is the same fact that makes a `defenum` unreloadable. A union is refused the same way and for its own
|
||
reason — it is declared, and union *values* are milestone 6. Both are `Types.Named` at a use site, so falling through
|
||
to "no struct is named X" would say a type does not exist about one that plainly does.
|
||
|
||
**`render.ml` is not reused, and that is not a second walk.** It walks a *value* and emits the code that prints it;
|
||
this describes a *type* and emits text. What is shared is the spelling: field types go through `Types.to_string`, which
|
||
is what `defs` spells a signature with, so `(Option T)`, `[T]`, `[n T]` and `(Ptr T)` read the same in a layout, in a
|
||
signature and in the source. A field that is itself a struct shows its qualified name — which is a `:type` this op
|
||
accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude
|
||
structs are answered like any other, because `Render` resolves against the same list and an editor that could see a
|
||
`Split` printed but not ask about it would be the two disagreeing.
|
||
|
||
On the Emacs side `flan-cnr-layout` makes the request and `flan-cnr-show` passes the result into
|
||
`flan-cnr-state-from-reply`, which stays a function from data to data so the fixture-driven tests keep working without
|
||
a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and
|
||
turning `C-c C-b` into an error would take away the restarts — the decision the buffer exists for — over a missing
|
||
annotation.
|
||
|
||
### The shadow stack, and `backtrace`
|
||
|
||
plan.org's "Dev vs release builds" table has had **Frames: shadow stack** in the dev column since the project began and
|
||
nothing had ever built it. This is it. It is what `(:op "backtrace")` is made of, and it is dev-only, so a shipped game
|
||
pays nothing — the same bargain the indirection cells already make.
|
||
|
||
Chosen over the DWARF route deliberately, and the author's reason is the one that decides it: **the more a break loop
|
||
can show, the less often a real debugger is needed.** DWARF buys frames in lldb; this buys them in the break loop,
|
||
where someone already is when they want them.
|
||
|
||
A frame is four words on the calling function's own stack — the frame it displaced, a pointer to a static description
|
||
of the function, and two words reserved for locals. The description is per function and not per call, because nothing
|
||
about a function changes between two calls to it: the qualified name, `file:line:col` as `Loc` spells it, and how many
|
||
slots the frame has. So the name in a backtrace comes off the frame itself and needs no debug information, no symbol
|
||
table and no agreement with the optimiser about what a stack frame looks like. It is the same mechanism on wasm32,
|
||
which is the other reason it is not `.eh_frame`: plan.org lowers every non-local exit explicitly rather than through
|
||
platform unwinding, so there is no unwinder here to borrow.
|
||
|
||
The compiler emits the push and the pop **inline** rather than calling into the runtime. This is on every call in a dev
|
||
build, and a call made to record a call would be most of what it costs.
|
||
|
||
**The pop is at every `ret`, and the transfer path is the one that matters.** There are five: an explicit `return` with
|
||
a value and without, the `none` arm of `(some x)`, the tail of the body, and the landing block a transfer leaves
|
||
through. `emit.ml` funnels all five through one `ret`, because the way to get this wrong is to write the pop on the
|
||
normal path and not on the other one — and then every *handled* error leaves a dead frame behind, and the backtrace
|
||
after the fifth one is five frames of fiction. `test_dev.ml` takes five breaks and resumes all of them by condition
|
||
transfer before asking for a backtrace, and the answer has to be two frames. Same lesson `with-allocator` learned about
|
||
restoring at the pad.
|
||
|
||
**A plain global, not a thread-local.** It matches what the handler stack and the restart stack in `flan_rt.c` already
|
||
assume: one thread runs Flan, and the listener thread runs C and the loader and never enters a Flan body. If the
|
||
language grows threads this becomes thread-local and the compiler's two stores become TLS-relative, which is the whole
|
||
of the change.
|
||
|
||
**The chain is snapshotted on the stopped thread**, into the same `snapshot` the restarts are copied into, at the same
|
||
moment and for exactly the same reason: the break loop *polls*, a poll runs Flan, and a chain read by the listener
|
||
thread is a chain that can be popped underneath the reader. Names are copied as bytes rather than kept as pointers,
|
||
because a transient `C-x C-e` module does get `dlclose`d and its rodata with it. Frame *addresses* are kept beside the
|
||
text, because reading a frame's locals means going back to that frame and not to whatever is at index 2 by then.
|
||
|
||
```
|
||
(:op "backtrace") → (:status "ok" :frames (("fetch" "/game.flan:15:7" "program" 0)
|
||
("main" "/game.flan:27:7" "program" 0))
|
||
:more 0 :stopped t :condition "Missing")
|
||
```
|
||
|
||
Innermost first. `:more` is how many frames deep recursion left off the end — the innermost ones are what the question
|
||
is about. **`origin` is `"program"` or `"eval"`**: a break inside a `C-x C-e` thunk has that thunk's frames on top of
|
||
the program's, and "where is my program" answered with `eval/7` is true and not the question. The boundary is recorded
|
||
at the call, in `flan_agent_poll`, exactly where `restart_floor` is and for the same reason — except that its
|
||
out-of-a-thunk value is `-1` rather than `0`, because zero restarts on the stack is a real answer and zero frames
|
||
belonging to the program is not.
|
||
|
||
**Refused while the program is running**, like every other break verb. The chain is the game thread's and it is pushed
|
||
and popped on every call; a walk of it from the daemon would have the shape of a backtrace and the contents of a race.
|
||
|
||
**What it costs, measured rather than assumed.** Two benchmarks, one compiler built per variant, every binary kept and
|
||
then run alternately. The figure is the **minimum of nine runs**, because this machine is shared and a mean measures
|
||
whatever else was running; the whole table reproduces to three digits on a second pass.
|
||
|
||
| Dev build, -O2 | no frames | frames | frames + slots |
|
||
|---|---|---|---|
|
||
| 2000 sweeps of a 100×100 grid — sand's inner loop, in miniature | 30.0 ms | 40.0 ms (+33%) | 48.3 ms (+61%) |
|
||
| fib(30) plus 20M calls in a loop — nothing but calls | 28.7 ms | 31.1 ms (+8%) | 35.1 ms (+22%) |
|
||
|
||
Per call that is about **0.1 ns** for the frame and **0.3 ns** for the frame and the slot table together; per sweep of
|
||
the grid, 5 µs and 10 µs, which is 0.03% and 0.06% of a 16.6 ms frame at 60fps. The dev loop's premise is that
|
||
redefinition does not stutter a running game, and a sixteenth of a percent of a frame does not. A dev build is already
|
||
deliberately slower than a release one — every call goes through a cell, every index is checked, no `defconst` folds —
|
||
and this joins that list rather than starting a new one.
|
||
|
||
The chain's *shape* was chosen by measurement and not before it. An array with a stack pointer — no alloca, no address
|
||
escaping — was built and timed as the obvious alternative and is worse on both benchmarks: the frame record on the
|
||
calling function's own stack is already hot, and an indexed store into a megabyte of BSS is not. It also has a fixed
|
||
depth, which a chain of stack records does not.
|
||
|
||
### Locals of a stopped frame
|
||
|
||
The half the shadow stack was built for. `(:op "locals" :frame N)` answers what a stopped frame's named locals hold.
|
||
|
||
```
|
||
(:op "locals" :frame 0) → (:status "ok" :frame "look"
|
||
:locals (("n" "i64" "3") ("label" "string" "\"hello\"")
|
||
("p" "Point" "(Point {:x 1.5 :y 2.5})")
|
||
("xs" "[3 i32]" "[ 10 20 30]") ("flag" "bool" "true"))
|
||
:refused (("after" "not bound yet at the point the program stopped")))
|
||
```
|
||
|
||
**Nothing is copied out of the program, and nothing could be.** A Flan value carries no header, so bytes read from
|
||
another process would be bytes with no meaning. What the daemon has instead is the *type* — `Tast.fn.slots`, from the
|
||
build it owns — and the name beside it in `snames`, which was already there for the DWARF work. So it compiles a thunk
|
||
that renders those types **at those addresses, in the program**, on the stopped thread, and reads the text back through
|
||
the same seqlocked result buffer `C-x C-e` uses. The only fact that comes from the running program is where the frame
|
||
is. That is `render.ml`'s existing walk with its root changed: `Render.render` over `(Deref (Ptr T) (flan/dev-slot f
|
||
i))` instead of over an expression — the **pointer-rooted render thunk** NEXT.md said locals needed, and it turned out
|
||
to need one new arm in the whole backend (a pointer-to-pointer cast, which under opaque pointers emits nothing).
|
||
|
||
**A slot's entry is its address, and null until it is bound.** That is the whole of the liveness answer: there is no
|
||
analysis, no PC-to-scope map and no bitmap. The store that binds a slot stores the address, so a slot the program has
|
||
not reached yet reads as null and is refused by name. Without it, `(let [after 99] …)` sitting past an `error` would
|
||
render whatever the stack held, and a slice or a struct of garbage does not misprint — it faults, on the game thread of
|
||
a program that is already stopped, which is the worst moment this project has to offer. The daemon asks the program
|
||
which slots are bound *before* it builds the thunk, so the set it emits code for is the set the thunk will resolve.
|
||
|
||
**Only named slots are recorded**, and this is where most of the cost went. A slot whose address is stored anywhere
|
||
escapes, and an escaped alloca is one `mem2reg` cannot promote — so recording a slot is paying for it in every call to
|
||
that function, for ever. The slots that would hurt most are exactly the ones with nothing to show: `dotimes`'s hidden
|
||
bound, the temporaries `(min)` and `(max)` evaluate their operands into, the render walk's own scratch. They keep their
|
||
promotion and are refused by name (`s4`, "a slot the compiler made up") rather than shown under an invented one.
|
||
Recording every slot instead was built and timed and came out inside the noise on both benchmarks, so the rule stands
|
||
on what it shows rather than on what it saves.
|
||
|
||
**Shadowing is right here, and that is not an accident of this design — it is the thing the DWARF route still owes.**
|
||
`check.ml`'s `fresh_slot` only ever allocates, so `(let [v 22] …)` inside `(let [v 11] …)` is two slots, both named
|
||
`v`, and both appear with their own values. lldb answers `p v` with 11 in that program and will until a
|
||
`!DILexicalBlock` per `Let` exists.
|
||
|
||
**Four refusals, each by name and with its reason.** Three per slot — invented, not yet bound, and no printer for the
|
||
type (a map, a function value, a type variable; the arm exists, no program the checker accepts has reached it yet, so
|
||
it is written and untested) — and two whole frames: one belonging to a `C-x C-e` thunk, whose `Tast` the session does
|
||
not keep, and one whose *body* is not the body this session holds.
|
||
|
||
**The superseded frame, and why a count could not find it.** Installing while stopped is deliberately allowed — it is
|
||
the fix-it-and-retry loop — so the frame on the stack and the body the session holds can be two bodies of one function.
|
||
A slot count catches a body that gained or lost a binding and nothing else; the case that matters is a **rename**, which
|
||
changes neither the count nor the types, and which would otherwise show every *new* name against the *old* body's
|
||
storage with nothing saying so. That is the confident wrong answer, in the one place someone is working out what went
|
||
wrong.
|
||
|
||
So each `%fninfo` carries a **slot fingerprint**: `Emit.slot_fingerprint` over every slot's name together with the
|
||
spelling of its type, since either can change on its own. The frame carries the value for the body it was compiled from,
|
||
`flan_dev_frame_slotsig` reads it, the agent snapshots it with the rest of the frame and puts it on the backtrace line,
|
||
and `Dev.locals` recomputes it from the body it holds and compares. Different means refused by name — *this frame's body
|
||
was redefined since it was entered, so its names no longer describe its values* — rather than answered. The count check
|
||
stays in front of it because its message is the more specific one.
|
||
|
||
Computed in `emit.ml` and read from there by `dev.ml`, so there is one definition of it and the two ends cannot drift.
|
||
It stays **off the wire**: `(:op "backtrace")` still answers four fields per frame, because a hash is not something an
|
||
editor can act on and the refusal says the fact in words instead. The same is true of the globals fingerprint that
|
||
now travels beside it. The bound is worth stating: it is a 30-bit hash, so a
|
||
collision is possible in principle, and it would reproduce exactly the silent wrong answer this catches — but only
|
||
between two *differing* bodies of a function whose qualified name has already matched, since `find_fn` is what gates the
|
||
comparison at all.
|
||
|
||
The first build of this shipped the fingerprint into `%fninfo` and stopped there — no accessor, no wire field, no
|
||
comparison — and was left in the tree with its own test red. Four of five hand-offs missing looks exactly like one
|
||
hand-off dropping a number, which is what the note left behind said it was.
|
||
|
||
**A `(Vec T)` shows as `<vec>` and a `(Ptr T)` as `<ptr>`**, because that is what `render.ml` already does for them
|
||
everywhere else: following a pointer a REPL was handed is not a safe thing to do on someone's behalf, and walking a
|
||
`Vec` structurally is a walk over storage the frame does not own — `(print (as-slice v))` is how that is asked for,
|
||
and it says at the call site that it borrowed.
|
||
|
||
Each slot is rendered **from its address** rather than copied into the thunk first. A copy would be one `alloca` the
|
||
size of the slot — 40KB for sand's grid — and the walk only ever shows eight elements of it. The cost is one call to
|
||
`flan/dev-slot` per leaf the walk reaches rather than one per slot, which the depth and span caps already bound.
|
||
|
||
### Globals of a stopped stack
|
||
|
||
The other half, and in this language arguably the more useful one: a game keeps most of its state in top-level
|
||
`defvar`s and `sand.flan` holds its entire grid that way, so "what is this program's state right now" was a question
|
||
with nowhere to ask it.
|
||
|
||
```
|
||
(:op "globals") → (:status "ok"
|
||
:globals (("grid" "[4 i32]" "[ 7 5 0 0]" (0 1))
|
||
("pressure" "i64" "12" (0))
|
||
("label" "string" "\"running\"" (1)))
|
||
:refused () :skipped ())
|
||
```
|
||
|
||
**Not per frame, and that is the design rather than a layout preference.** A global is not part of a frame — it is
|
||
program state the frame happened to touch — so nesting it under one implies an ownership that is not there and repeats
|
||
the name once per frame that reads it. So: one section, whose contents are the **union of the globals every frame on
|
||
the current stack references**.
|
||
|
||
**The compiler does the choosing.** `Reach.expr_refs` is the walk that already computes what a function refers to — it
|
||
is how the link drops a package nothing calls — and pointed at one body it answers that body's reference set. Listing
|
||
*all* of a program's globals instead would bury the one that matters under the prelude's PRNG state; taking only what
|
||
the stack reaches is a filter the compiler can apply and a person cannot. Direct references only, with no transitive
|
||
closure through calls: a callee that reads a global is either on this stack, contributing its own references already,
|
||
or it is not on it and is not part of where the program stopped.
|
||
|
||
**Each entry carries the frames that touch it, by index**, which is what the conditions buffer already numbers frames
|
||
by. That recovers what per-frame nesting would have told you — "the whole chain is reading this" reads differently from
|
||
"only the innermost does" — at no cost in duplication. **Ordered by the innermost frame that touches it**, because a
|
||
deep stack makes the union large and proximity to the error is what puts the likely culprit on top; ties keep
|
||
declaration order.
|
||
|
||
**The mechanism is one step simpler than `locals`.** A local is reached by *address*, and only the stopped program
|
||
knows where its frame is — hence `flan/dev-slot`, hence the `bound_slots` round trip, hence the null-until-bound
|
||
refusal. A global is reached by *name*: `Emit.redefinition` writes a global the host already has as `external`, so the
|
||
dlopened thunk binds to the program's own storage and the dynamic linker does the work. Nothing is asked of the stopped
|
||
thread at all, and there is no not-yet-bound case, because a global's storage exists from the moment the process
|
||
started. It is `Render.render` over `(Global n)` — the same root `C-x C-e` uses when you type a global's name at it.
|
||
|
||
**A frame that cannot be attributed contributes nothing and says so.** An `eval` frame has no declaration in this
|
||
session; a lifted handler clause has none of its own; a frame whose body was redefined since it was entered holds a
|
||
body whose reference set is a claim about different code. All three go into `:skipped` with the reason, because "the
|
||
union is incomplete and here is why" and "these are all of them" are different answers and the second one is the lie.
|
||
|
||
**The hole this had, and the second fingerprint that closes it.** `Emit.slot_fingerprint` hashes a body's *slots*. For
|
||
`locals` that is exactly the right cut: identical slots means the names still describe the storage, so the answer is
|
||
still true. Here it was not, because a body can change which globals it names without touching a single slot — and then
|
||
this section showed the *new* body's reference set attributed to the *old* frame. The values were never at risk; they
|
||
are read from the program's storage by name. What was, is one frame's membership in the union and the frame numbers
|
||
beside an entry.
|
||
|
||
So `%fninfo` carries a second number beside the slot fingerprint: `Reach.ref_fingerprint`, over the **set** of globals
|
||
the body names — sorted and deduplicated, because a reference set is not ordered and a body that mentions the same two
|
||
globals the other way round is the same body, where slot indices make the slot fingerprint order-sensitive on purpose.
|
||
It travels the same path the first one does: `flan_dev_frame_refsig`, the agent's snapshot, the backtrace line, and
|
||
`Dev.globals_op` recomputing it from the body it holds. Different means that frame is in `:skipped` with its own
|
||
reason — *this frame's body names different globals than the one this session holds* — and the rest of the stack still
|
||
contributes.
|
||
|
||
**Two numbers and not one combined**, which is the whole reason this is a second fingerprint rather than a wider first
|
||
one. They are different facts: a frame whose slots match and whose globals do not has locals that are perfectly
|
||
readable and attribution that is not, and one hash over both would make `locals` refuse a frame with nothing wrong
|
||
with it. So `locals` still checks the slot fingerprint alone.
|
||
|
||
It lives in `reach.ml` rather than `emit.ml` because `Reach.expr_refs` is already the walk that answers "what does this
|
||
body refer to" and is the same walk the union above is built from — the two ends cannot disagree about what counts as a
|
||
reference. Which names are globals is the caller's to say: the emitter knows the program's globals, and so does the
|
||
session. `test_dev.ml` now drives the exact case — a redefinition that binds identical locals and names `untouched`
|
||
where the stopped frame's body names `pressure` — and with the check disabled it fails twice: once on the missing
|
||
refusal, once on `untouched` appearing in the union under frame 0.
|
||
|
||
### Conditions — step 2: `restart-case` and `invoke-restart`
|
||
|
||
`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
|
||
`restart-case` further out.
|
||
|
||
```
|
||
(defn fetch [n i32] i32
|
||
(restart-case (middle n) ; its value if nothing transfers
|
||
(use-placeholder [] -1)
|
||
(retry [] 7)))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
|
||
(fetch 2)) ; -1
|
||
```
|
||
|
||
**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an
|
||
`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the
|
||
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
|
||
slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers.
|
||
`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather
|
||
than a hunt.
|
||
|
||
**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare
|
||
pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and
|
||
plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight
|
||
through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform
|
||
also means redefinition acquires no new refusal class.
|
||
|
||
**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous
|
||
note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only
|
||
probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The
|
||
frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is
|
||
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
|
||
Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's
|
||
"innermost offering the name" is just the order of the walk.
|
||
|
||
**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever
|
||
frames it established and either catches the transfer or forwards it outward:
|
||
|
||
- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts
|
||
the target back and goes on out;
|
||
- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes,
|
||
and without it the handler stack would be left pointing into a frame that has gone;
|
||
- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is
|
||
still refused by name.
|
||
|
||
A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would
|
||
jump straight past the very form that was supposed to catch it.
|
||
|
||
**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is
|
||
guarded; with the channel still set the first of them would branch straight back into the landing block it came from.
|
||
Same reason the clause body starts with it null.
|
||
|
||
`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
|
||
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
|
||
cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and
|
||
there are no function values yet, so nothing can call *back* into Flan across one.
|
||
|
||
Scope, each piece refused by name with its reason and a test on the reason:
|
||
|
||
- **restarts took no parameters.** That covered §1's own `load-texture` example and skipped argument marshalling and
|
||
§3's runtime check. Both are in now; see "Conditions — step 3" below.
|
||
- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops.
|
||
- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a
|
||
choice nothing in the source shows.
|
||
- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting
|
||
there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a
|
||
defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see
|
||
it.
|
||
- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past
|
||
everything. There is nowhere to resume, so there is nothing else to do.
|
||
|
||
**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal`
|
||
returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past
|
||
it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked
|
||
`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case`
|
||
body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled
|
||
case: it cannot be an `outputs` row, because it does not exit 0.
|
||
|
||
`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart`
|
||
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
|
||
|
||
**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a
|
||
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
|
||
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
|
||
nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value.
|
||
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
|
||
|
||
`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca`
|
||
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
|
||
crosses `probe`, which the host was compiled with. Those three do not meet anywhere else.
|
||
|
||
Two things found by writing it:
|
||
|
||
- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates
|
||
the body's slots into a record the function never sees again and the indices collide. It was harmless only because no
|
||
`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now.
|
||
- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not
|
||
fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere
|
||
else entirely.
|
||
|
||
`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant
|
||
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
|
||
of those calls goes through a cell.
|
||
|
||
### Conditions — step 3: restarts take parameters
|
||
|
||
§3's other half, and the half the comparative studies all ask for: `use-value` and `store-value` are the two restarts
|
||
whose answer is not in the program.
|
||
|
||
```
|
||
(restart-case (middle n)
|
||
(use-value [v i32] (* v 2))
|
||
(retry [] 7))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7)) ; 42
|
||
```
|
||
|
||
**The parameters live in a buffer the restart-case owns.** The obvious place is the invoker's frame — it is where the
|
||
values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so
|
||
the invoking side is gone by then. The invoker stores into the *target's* buffer while both are still alive, which is
|
||
the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that
|
||
buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any
|
||
other.
|
||
|
||
**What a restart takes is checked at run time, and it has to be.** §4 finds a restart by name on a dynamic stack: the
|
||
invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker
|
||
to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and
|
||
`invoke-restart` compares both against its own before it stores anything. Every frame carries them, parameterless ones
|
||
included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not
|
||
redundant with the hash: it is what keeps a hash collision between two *different* signatures harmless, since a
|
||
collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well,
|
||
because the message has to say what was wanted and what was given and neither end knows both:
|
||
|
||
```
|
||
restarts.flan:79:36: restart use-value takes (i32), given ()
|
||
restarts.flan:87:36: restart retry takes (), given (i32)
|
||
```
|
||
|
||
**The arguments are evaluated into slots before the invoke node, not hung off it.** Two reasons and both are real. An
|
||
argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside
|
||
an argument has to be on the walk `Reach` and `Load` already do — `InvokeRestart` was a leaf to both, and a leaf that
|
||
grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker.
|
||
`restarts.flan` has exactly that function, `half`, to keep the claim tested.
|
||
|
||
**The break loop can take a restart it cannot fill in, so it is refused.** A transfer has two sources: an
|
||
`invoke-restart`, which writes the arguments first, and the break loop, which aims the channel at a frame by position
|
||
and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them
|
||
apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked
|
||
unfilled, `invoke-restart` marks it filled, and a clause with parameters checks the mark before reading. Choosing
|
||
`use-value` from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the
|
||
top of `NEXT.md`: the answer is a Flan expression, and there is already something that compiles one against the live
|
||
program.
|
||
|
||
**Lookup stayed by name, and the signature is checked against what it found.** `flan_find_restart` matches the name
|
||
hash and nothing else, so an inner `(use-value [s string] ...)` shadows an outer `(use-value [v i32] ...)` and an i32
|
||
is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first
|
||
frame offering the name") and it is the thing a reader will assume works the other way, so `restarts.flan` has a case
|
||
for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments,
|
||
which is overload resolution on a dynamic stack.
|
||
|
||
Both of these guards go through `fail_block` unconditionally, unlike the bounds checks: `--no-bounds-checks` does not
|
||
remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or
|
||
written to a different layout, is memory corruption.
|
||
|
||
`runtime/flan_rt.c` gained two message functions and nothing else. The restart frame's first four fields are the ones C
|
||
declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.
|
||
|
||
§3's other open point, a **report string per clause**, is still open and was not settled first as §3 asks. The field
|
||
and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the
|
||
agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing
|
||
that listing anyway.
|
||
|
||
### What is left
|
||
- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays.
|
||
|
||
**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the
|
||
memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary.
|
||
Attaching to a process someone else built is not a thing to support by default.
|
||
|
||
## Allocators, `(Vec T)` and `StorageExhausted`
|
||
|
||
`spec-memory.md`'s Allocators section is frozen and settles what to build. This is why the built thing is the shape it
|
||
is, and — separately and loudly — the three places it **amends** that section plus the one thing it adds to it.
|
||
|
||
### `Allocator` is a builtin opaque type, so none of milestone 5 was needed
|
||
|
||
The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and
|
||
`check.ml` refuses function values four ways — a written `(Fn ...)` annotation, a written `fn` literal, a `defn`'s name
|
||
in value position, and calling anything other than a named function. Read straight off those lines, containers need
|
||
function values and the work doubles.
|
||
|
||
None of the four is anywhere near this. All four are about *surface syntax*, and a value the compiler builds that no
|
||
surface form names trips none of them:
|
||
|
||
- `Allocator` is a `Types.t` case with no user-writable constructor, the way `string` is a builtin ptr+len. No Flan
|
||
type names its procedure.
|
||
- The procedure is a C symbol the emitter names.
|
||
- `vec-new`, `push`, `free`, `free-all` and the rest are ordinary named calls, which `check_call` already routes
|
||
through `named_call`.
|
||
|
||
The precedent was already in the repo twice: a `handler-bind` clause is lowered to a function called back through
|
||
`h->fn(condition, xfer)` and built as its own `Tast.fn`, never an `Ast.Fn`; and a dev build's `call` loads a pointer
|
||
out of a cell and calls through it, which is the indirect call the surface language refuses.
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: "here is my proc, make an `Allocator` from it" wants a
|
||
`defn`'s name in value position. `make-allocator`, `allocator-from` and `allocator` are refused by name with that
|
||
reason, rather than coming back as unknown functions.
|
||
|
||
**An `Allocator` value is a pointer to the runtime's struct, never a copy of one.** That is forced, not chosen. The
|
||
capability set has to be readable from wherever a container landed, and `free-all` bumps an epoch every container made
|
||
from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never
|
||
fires.
|
||
|
||
### The surface
|
||
|
||
| Name | What |
|
||
|---|---|
|
||
| `(heap-allocator)` | the general-purpose tier: malloc, aligned, with free |
|
||
| `(arena-new bytes)` | a bump arena over one fixed backing buffer |
|
||
| `(arena-destroy a)` | hands the pages back — see the amendment below |
|
||
| `(free-all a)` | releases everything the allocator holds, retain-capacity |
|
||
| `(can-free? a)` / `(can-free-all? a)` | the capability set, read at run time |
|
||
| `(alloc-epoch a)` / `(alloc-id a)` | the counter `free-all` bumps; the allocator's identity |
|
||
| `(alloc-budget a)` / `(set-alloc-budget a n)` | a ceiling on live bytes — an addition, see below |
|
||
| `(alloc-live-blocks a)` | "did you forget to free", answered at the tier that can answer it |
|
||
| `context/allocator` / `context/temp` | the current implicit allocator, and the per-frame arena |
|
||
| `(with-allocator a body...)` | rebinds for a dynamic extent and releases nothing |
|
||
| `(vec-new T)` / `(vec-new T a)` / `(vec-new)` | a Vec, against the context or a named allocator |
|
||
| `(push v x)` / `(reserve v n)` | `Unit`, both |
|
||
| `(at v i)` / `(len v)` | the array names, extended — not a parallel pair |
|
||
| `(as-slice v)` / `(as-slice v lo hi)` | a non-owning `[T]` view |
|
||
| `(clone v)` / `(clone v a)` | the only copy; assignment moves |
|
||
| `(free v)` | consumes its argument |
|
||
|
||
### Three amendments to a frozen spec, and one addition
|
||
|
||
**1. `free-all` is retain-capacity, and `arena-destroy` is the operation that hands pages back.** The spec's table has
|
||
`free-all` and nothing else. Zig's `ArenaAllocator.reset` takes a `ResetMode` of `free_all` / `retain_capacity` /
|
||
`retain_with_limit`; Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing
|
||
buffer and the call only sets `offset = 0`. For a frame arena reset every frame, retain-capacity is the normal case and
|
||
handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown
|
||
the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being
|
||
the same does not make a container built before the reset valid, which is the whole point of the trap.
|
||
|
||
**2. `context/allocator` and `context/temp` are dynamic variables with save and restore, not extra parameters.** The
|
||
spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the
|
||
FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other
|
||
lane working in `emit.ml`. The dynamic variable is the implementation; the literal reading is deferred and is a
|
||
performance question (a parameter avoids a global load), not a semantic one.
|
||
|
||
**3. The `Vec` header is six words in every build, not four in release.** The spec fixes the release layout at
|
||
`ptr + len + cap + allocator` with the generation word and the epoch dev-only. A layout that changes with a build flag
|
||
is a layout that can disagree *silently* across the reload boundary: a redefinition module is built by `llc` and `ld`
|
||
against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and
|
||
so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag
|
||
before it can land.
|
||
|
||
**The addition: a budget.** `flan_allocator` grew a ceiling on live bytes, 0 for none. The spec's `retry` restart is
|
||
answerable only by a handler that can make the *same* request succeed, and for a fixed backing store the handler that
|
||
works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which
|
||
is exactly what the epoch check catches. The spec names "grows the arena and then invokes `retry`" as the handler that
|
||
works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on
|
||
purpose.
|
||
|
||
### `with-allocator` is its own IR node because of the transfer path
|
||
|
||
Save, run, restore — and *restore again at the pad*. That second restore is the whole reason it is a node rather than a
|
||
`let` and two calls. A body that errors, or one a handler transfers out of, leaves through `current_pad`, and a context
|
||
allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is
|
||
precisely where something is about to allocate in order to render a condition. `test/programs/allocators.flan` asserts
|
||
that path by taking a restart out of a `with-allocator` body.
|
||
|
||
It releases nothing, per the spec: not at the end of a `let`, not at the end of a function, not at the end of the body.
|
||
The program proves it by reading the epoch either side.
|
||
|
||
### `(Vec T)`: one type-erased runtime, and the backend learned almost nothing
|
||
|
||
The element type appears nowhere below the call site. `size_of` and `align_of` are produced where the concrete type is
|
||
known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement
|
||
(`base/runtime/dynamic_array_internal.odin`). The backend grew four prims in total:
|
||
|
||
- `Rt of string` — a call into the runtime's C named by symbol, with argument and result LLVM types read off the
|
||
expression nodes. The container runtime is type-erased and therefore *is* a list of C entry points, so one arm covers
|
||
all of them. A `Vec` argument crosses as its address, which is also what lets an operation mutate the caller's Vec.
|
||
- `SizeOf` / `AlignOf` of a type, filled in from the same layout calculator DWARF uses — the one the acceptance test
|
||
already checks against LLVM's own `getelementptr` answers.
|
||
- `AddrOf` of any expression, place or not, because the element a `push` copies may be computed. The backend already
|
||
spilled a non-place to a temporary for exactly this.
|
||
|
||
`at` and `len` were already the names for a fixed array and a slice, so a Vec extends them rather than adding a
|
||
parallel pair — the asymmetry `nth` was removed for. The value form `(at v i)` and the place form `(set (at v i) x)` go
|
||
through one helper, so they cannot drift apart the way `nth` did.
|
||
|
||
A Vec's length and index are `i32`, like every other length here. Widening indices is one change across every
|
||
container and not a Vec question.
|
||
|
||
`let` has no type annotation — `parse.ml` settles that a triple binding is ambiguous and that types are inferred — so a
|
||
local Vec has nowhere to say what it holds and the element type is written at the call: `(vec-new i32)`. This is *not*
|
||
the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an
|
||
ordinary type rather than binding a type variable. Where the context does say — a `defvar`'s type, a return type, an
|
||
argument — it may be left out.
|
||
|
||
A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is
|
||
Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a
|
||
different reason (below), but the adopt rule is what makes `Zero` of a Vec a usable value rather than a null deref.
|
||
|
||
### Move-only is a dead set, and it is flow-sensitive at a join
|
||
|
||
Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing
|
||
one to a function, binding it, returning it and `free`ing it are all moves and all reach one place, and the handful of
|
||
operations that only look at a container (`at`, `len`, `as-slice`, `push`, `reserve`, `clone`) say so. Only a
|
||
*syntactically simple* target counts as a borrow — in `(len (f v))` the call still moves `v`.
|
||
|
||
At an `if` and at a `match`, every arm is checked from the state before the form and the **union** of what they moved
|
||
survives the join. A flat set is wrong in both directions: it refuses `(if c (free v) (free v))`, which is legal, and it
|
||
accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union
|
||
says.
|
||
|
||
The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move,
|
||
not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a
|
||
move of a binding declared outside the loop is refused, with that as the reason.
|
||
|
||
### What ownership is not transitive through yet, and why each is refused
|
||
|
||
The spec says ownership is structural — a struct containing a Vec is itself move-only, `free` recurses into owning
|
||
fields, and a field cannot be freed on its own. That machinery is the recursive teardown `drop` brings. Until it lands,
|
||
three shapes are refused where they are declared, each naming `drop`:
|
||
|
||
- **a struct field of `Vec` type**, because the struct copies its header on assignment and nothing records a move;
|
||
- **a global of `Vec` type**, because the dead set is per function — two functions each freeing it is a double free
|
||
nothing could see, and a global read does not go through the move path at all, so even the one-function case would be
|
||
accepted. Half a rule is worse than none. A global **`Allocator`** is a different thing and stays legal: an allocator
|
||
is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot
|
||
see the locals of the function that established it;
|
||
- **a `Vec` of a `Vec`**, because the type-erased runtime copies and releases elements bytewise: `clone` would
|
||
duplicate inner headers instead of copying what they own and `free` would drop their buffers.
|
||
|
||
And a `Vec` does not cross to C: handing a header that owns storage to C hands out an owner. `(as-slice v)` as
|
||
`(Ptr T)` plus `(len v)` is the shape that does cross, and the refusal says so.
|
||
|
||
### `StorageExhausted` went in *with* `Vec`, not after it
|
||
|
||
No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the
|
||
operation signals `(StorageExhausted {.bytes n .align a .allocator id})` with `error` — whose type is `Never` — inside a
|
||
`restart-case` offering `retry`. One rule over every allocating operation, which is what keeps `push` and `reserve` at
|
||
`Unit`, `clone` at the container, and no signature anywhere growing a `Result`.
|
||
|
||
It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every
|
||
allocating operation, which is the point of having decided it first. Odin's `append` returns an ignorable
|
||
`Allocator_Error`, and its type-erased path returns the old length on a failed reserve; an append that appends nothing
|
||
and says nothing is the outcome this rule exists to make impossible.
|
||
|
||
The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:
|
||
|
||
```
|
||
(let [ok false]
|
||
(while (not ok)
|
||
(restart-case
|
||
(do (set ok ATTEMPT)
|
||
(if (not ok) (error (StorageExhausted {...}))))
|
||
(retry []))))
|
||
```
|
||
|
||
A handler that frees something, releases a scratch region or raises the ceiling and then invokes `retry` lands in the
|
||
clause, the clause falls through, and the `while` re-attempts the **same** request. Every argument to the attempt is
|
||
bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a
|
||
`push` was given. With nothing handling it, `error` stops the program on the frame that erred.
|
||
|
||
This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the
|
||
failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site
|
||
can. The condition is a value struct with fixed numeric fields and **no rendered message**, because formatting would
|
||
allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.
|
||
|
||
### The epoch trap, which is the shipping answer to an open question
|
||
|
||
`spec-memory.md` leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which
|
||
allocator a construction used, and `with-allocator` plus `context/allocator` are exactly the mechanisms that deny that
|
||
knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, `free-all` bumps
|
||
that counter, and any operation on a container whose recorded epoch has moved traps naming the site.
|
||
`test/programs/stale-region.flan` is the case, and the point of it is that `v` is still in scope, still looks fine, and
|
||
nothing marked it — which is precisely what a static rule cannot see.
|
||
|
||
**The generation word has no reader.** It is bumped on every reallocation, as specified, and the stale-slice trap it
|
||
exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said
|
||
plainly here rather than implied by the word's presence in the header.
|
||
|
||
### What this leaves for steps 5 to 7
|
||
|
||
`(Map K V)` is built — see below. What is left: `drop` and with it the transitive move-only rule, recursive teardown,
|
||
and the refusal to construct a drop-carrying container against an allocator without `can-free`; `(Result T E)` and
|
||
`try`; generics; the macro expander.
|
||
And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy:
|
||
capture does not exist at all, and the spec's captured-`Vec`-by-pointer rule has never had to exist because every
|
||
capturable type today is a value type. It is its own item and should be planned as one.
|
||
|
||
## `(Map K V)`, which is Odin's map
|
||
|
||
Step 4 of the container build order, over the runtime `Vec` already established. `spec-memory.md` is followed as
|
||
written; the two places this departs from *Odin* are named below, and there is one amendment to the spec and one
|
||
restriction under it.
|
||
|
||
### The three properties that were the point
|
||
|
||
Odin's header states them and they are why it was the thing to follow (`base/runtime/dynamic_map_internal.odin`).
|
||
|
||
- **Open-addressed Robin Hood hashing at a 75% load factor.** No buckets and no per-entry allocation: one block holds
|
||
every key, every value and every hash. Robin Hood is the part that earns its keep — on insert, an element that is
|
||
further from the slot its hash wanted than the occupant it is looking at takes the slot and sends the occupant on.
|
||
Probe distances even out, and a lookup may stop the moment it is further from home than the occupant it is looking
|
||
at, because no element is ever further from home than one it passed. That early exit is why a miss costs about what
|
||
a hit does.
|
||
- **Cache-line cell packing.** A flat `[capacity]K` array lets one key straddle two cache lines, so a probe walking
|
||
four slots can touch five lines. A cell packs as many `K` as fit in 64 bytes and pads the rest, so no key ever
|
||
straddles a line. Keys, values and hashes are three separate runs, so a probe — which reads hashes, and only then
|
||
one key — touches hash lines and nothing else until it has a candidate.
|
||
- **Pointer-width integers throughout**, so no sign extension or masking gets into the probe loop.
|
||
|
||
### Two departures from Odin, both deliberate
|
||
|
||
**There are no tombstones**, because `spec-memory.md` defers removal ("Move-aware lookup, removal, and owned entries
|
||
are deferred"). A slot is empty or occupied and nothing else, which deletes Odin's backward-shift loop entirely — it
|
||
is the single largest reason this file is shorter than the original. When removal arrives, that loop is what it costs.
|
||
|
||
**The header does not tag the capacity into the data pointer.** Odin stuffs `log2cap` into the low six bits because
|
||
its `Raw_Map` must be three words. This header already carries an allocator, a generation and an epoch, so the tagging
|
||
would buy nothing, cost a mask on every access, and — the part that actually matters — make correctness depend on the
|
||
block being 64-byte aligned. Alignment is *requested*; an arena whose base is not cache-aligned now gives a slower map
|
||
rather than a wrong one.
|
||
|
||
The header is six words, 48 bytes, the same as a `Vec`'s and for the same reason: a layout that changes with a build
|
||
flag can disagree across the reload boundary.
|
||
|
||
data len log2cap allocator gen epoch
|
||
|
||
### The hash and equality pair, and why most key types do not get one
|
||
|
||
`spec-memory.md` restricts keys to built-in structural types and makes hashing and equality compiler-provided
|
||
structural operations. So there is no dispatch to design: a key type resolves to a pair of symbols, passed to the
|
||
type-erased runtime the way Odin hangs two contextless `proc`s off a `Map_Info`.
|
||
|
||
Most key types need nothing emitted. A key whose equality is bytewise and whose bytes are all present — every integer,
|
||
enum, bool, and fixed array of those — is served by one runtime pair over `(pointer, size)`. Two kinds are not, and
|
||
the reasons are worth keeping because they are what a bytewise shortcut would have got wrong:
|
||
|
||
- a **string** is ptr+len and its bytes are elsewhere, so two equal strings at different addresses must hash alike;
|
||
- a **struct** may have padding, whose bytes are indeterminate — two structs equal field by field can differ bytewise
|
||
— and it may hold a string, which brings the first problem inside it.
|
||
|
||
A struct therefore gets a pair emitted for it, walking its fields in declaration order and addressing nothing but
|
||
fields. Two maps with the same key type share one pair, and a struct reached twice through two fields emits one.
|
||
|
||
`Tast.FnAddr` is what carries the symbol, and it is **not** a function value: nothing in the surface language can
|
||
produce one, name its type, or call through it. It is the same escape the allocator used — the one NEXT.md predicted
|
||
would work — and `reach.ml` learned the edge, because a function reached only by address is invisible to the
|
||
reachability walk otherwise. That was already true of handler clauses.
|
||
|
||
A pair emitted for a struct is an ordinary Flan function, so its signature ends with the transfer channel like every
|
||
other. The runtime's typedef spells that pointer out rather than hoping nothing writes through it, and each built-in
|
||
hasher exists in two spellings — the pointer form that matches the typedef, and the direct form an emitted hasher
|
||
calls per field, which has no channel to hand on.
|
||
|
||
### The surface
|
||
|
||
| Name | What |
|
||
|---|---|
|
||
| `(map-new)` `(map-new K V)` `(map-new a)` `(map-new K V a)` | a new map; the pair may be omitted where the context says |
|
||
| `(put m k v)` | upsert, `Unit` |
|
||
| `(get m k)` | `(Option V)` — absence is `None` |
|
||
| `(has-key? m k)` | `bool`, copying no value — **an addition; the spec does not name it** |
|
||
| `(len m)` `(reserve m n)` `(clone m)` `(clone m a)` `(free m)` | extended, not duplicated |
|
||
|
||
`has-key?` is **not in `spec-memory.md`** and is an addition, flagged because everything else here is the spec's.
|
||
`(get m k)` answers the same question, but through an `Option` the caller then has to match, and the common use is a
|
||
condition. It copies no value, which is also why it is not just `get` with the result thrown away. The `?` suffix
|
||
follows `can-free?`.
|
||
|
||
`len`, `reserve`, `clone` and `free` were **extended rather than given map-shaped names of their own**, which is what
|
||
`at` and `len` already did for `Vec`: one question, one word. `reserve`'s `n` is entries, not slots — the runtime sizes
|
||
the block so `n` still sits under the load factor, which is the only reading of "room for n" that does not reallocate
|
||
on the nth put.
|
||
|
||
`get` builds the `Option` in the checker, not the runtime, which has no idea what an `Option`'s layout is; keeping it
|
||
that way is what lets one entry point serve every value type. `put` and `get` bind their arguments to slots before the
|
||
allocation guard, so a `retry` re-attempts the allocation and not the expressions that produced the key and value.
|
||
|
||
**`{K V}` is the type spelling and there is no map literal.** A bare map form in expression position is a struct
|
||
literal's field list, and giving the same braces two meanings is what the colon-to-dot change was for. A map is built
|
||
with `map-new` and filled with `put`.
|
||
|
||
### The refusals, each by name
|
||
|
||
- A **float key** — not a milestone question, which is why it is said separately. NaN is not equal to itself, and
|
||
`0.0` and `-0.0` are equal while differing bytewise. There is no equality there for a map to hash.
|
||
- A **`Ptr`, slice, `Vec` or `Map` key** — would hash an address rather than what it points at, which is a different
|
||
operation.
|
||
- A **fixed array whose elements are not compared bytewise** — an array of structs or of strings needs the
|
||
per-element walk a struct key gets, driven by a loop rather than a field list. Nothing has wanted one, so it is
|
||
refused rather than written untested, and the shape that does work (a struct holding the array) is named beside it.
|
||
**This is narrower than the spec**, which lists fixed arrays without qualification.
|
||
- A **move-only value** — the refusal `(Vec (Vec T))` already carries, for the identical reason: the runtime copies
|
||
entries bytewise, so `clone` would duplicate headers and `free` would leak what they own. Owned entries arrive with
|
||
`drop`.
|
||
- **`Unit` as a value** — there is nothing to store, and the cell geometry divides a cache line by the element size.
|
||
Named rather than left to divide by zero, because it is the natural spelling of a set.
|
||
|
||
`StorageExhausted` under `retry` holds over `map-new`, `put`, `reserve` and `clone`, reusing the machinery that landed
|
||
with `Vec`; no operation returns an error. A map is the harder of the two containers for that rule, and
|
||
`map-exhausted.flan` is why it gets its own program: a `Vec`'s failing allocation leaves the `Vec` untouched, whereas a
|
||
map's growth allocates a new block, rehashes into it, and only then releases the old one — so a failure partway must
|
||
leave the map exactly as it was, or the retry re-attempts against a half-moved map.
|
||
|
||
The epoch trap covers the map too, and `map-stale-region.flan` is separate from `stale-region.flan` because the two
|
||
reach the check by different routes: a `Vec`'s operations check on the way in and stop there, while a map's `get` goes
|
||
on to call a hash and an equality function through pointers into the block. A missing check there is not a wrong
|
||
number, it is a probe loop walking released memory.
|
||
|
||
### Is this another Python dict? — measured
|
||
|
||
The question was asked directly and the answer is that **Python's algorithm is fine**. What makes CPython's dict slow
|
||
is that every key and every value is a separately allocated, reference-counted object and hashing goes through
|
||
`__hash__` and `__eq__` calls that cannot be inlined. This map stores raw bytes in the block and compiles hashing and
|
||
comparison concretely per key type. That is most of the gap before any cleverness.
|
||
|
||
Measured on this machine, `i64` to `i64`, against CPython 3.13's dict on the same workload:
|
||
|
||
| Working set | Flan | CPython dict |
|
||
|---|---|---|
|
||
| 10k entries, 10M lookups (cache-resident) | 21 ns/lookup | 132 ns/lookup |
|
||
| 1M entries, 10M puts + 10M lookups | 1.41 s | 1.16 s |
|
||
|
||
**Six times quicker cache-resident, and slower at a million entries** — and the second row is written down rather than
|
||
left out. Both are waiting on memory there, and this layout waits longer: keys, values and hashes are three separate
|
||
runs, so a lookup that misses everything takes three cache misses where a compact dict takes two, and the hash run is a
|
||
full eight bytes a slot. Cell packing buys probe locality, which is a win while the hash run is resident and a loss
|
||
once nothing is. One byte of metadata a slot — the Swiss-table arrangement — is the known answer and is not built.
|
||
|
||
The path from 35 ns to 18 ns (the cache-resident floor, at 500 entries) was **profiled, and the first two guesses were
|
||
both wrong**: the per-slot cell division and the block-size divisions were each replaced first and neither moved the
|
||
number. What did: FNV one byte at a time became eight, and a key that is one machine word became one load and one mix
|
||
with no loop; equality on eight bytes stopped being a call into libc's vectorised `memcmp`, and copying a value out
|
||
stopped being a call into `memmove`; the block geometry stopped being recomputed five times over in a function that ran
|
||
twice per lookup; and the seed stopped being a five-multiply avalanche on the critical path for mixing the hasher does
|
||
again immediately after. `64/size` is a table, which is Odin's `Map_Cell_Info` by another route — Odin precomputes it
|
||
per type because the probe loop must not divide, and here the sizes arrive as ordinary arguments.
|
||
|
||
What remains at 18 ns is the type erasure itself: a non-inlinable call into the runtime and two non-inlinable indirect
|
||
calls to the pair. That is the trade `spec-memory.md` chose deliberately — "It is type-erased on purpose… No generics
|
||
are involved, and none are needed" — and monomorphisation is what would buy it back, at the cost the spec declined.
|
||
|
||
## Unions, and the tag they carry
|
||
|
||
`defunion` parsed and its shape was checked long before this; naming the type (`check.ml:312`) and constructing a
|
||
value (`:1075`) were both refused as milestone 6. They are not any more.
|
||
|
||
### It is closer than the milestone number suggested, and the reason is `Option`
|
||
|
||
`Tast.arm` already carried `acase` (a case name) and `binds` (the slots a payload binds to). That is union shape,
|
||
built and exercised, because **`Option` is a two-case union wearing a special coat** and `match` over it worked
|
||
already. So `check_match` grew a second *subject* rather than a second path: one function decides which case each arm
|
||
names and what type each name it binds has, and everything after it — the dead-set join across alternatives, the
|
||
result type, the exhaustiveness check — is the code that was there.
|
||
|
||
`Option` was **not** desugared into a declared union, and that is deliberate: `Option` is generic and no declared
|
||
union is. The coat is the part that cannot be taken off until generics exist.
|
||
|
||
A union is `Types.Named` exactly as a struct is. One case in `Types.t` covers both, and *which table the name is in*
|
||
is the only thing that tells them apart. That is what let a union be a field, a parameter, a return type, a slot and a
|
||
copy without a single one of those paths learning that unions exist.
|
||
|
||
### The layout, which is the part that has to be exactly right
|
||
|
||
```
|
||
%"U" = type { i32, [k x iA] }
|
||
%"U.Case" = type { the case's fields, in declaration order }
|
||
```
|
||
|
||
A tag, then room for the largest case, with `A` the alignment the *widest member of any case* needs and `k` the size
|
||
rounded up to it. Writing the payload as an array of `iA` rather than of `i8` is what makes LLVM align it without an
|
||
explicit `align` on a type — and it is what makes the whole thing
|
||
|
||
```c
|
||
struct { int tag; union { ... } u; }
|
||
```
|
||
|
||
byte for byte. **That agreement is the point.** A macro is `[Form] -> Form`, so `Form` has to be the same bytes in
|
||
the compiler and in the `dlopen`ed macro, and nothing at run time would notice a disagreement.
|
||
|
||
The tag is an `i32` and not an `i8`. With an 8-byte payload alignment the two cost the same, and `i32` is what a C
|
||
`enum` field is — the spelling the macro lane will have to write by hand on the other side of the boundary.
|
||
|
||
Both the union and one struct per case are emitted as *named* LLVM types, so every reader geps rather than computing
|
||
byte offsets of its own. Construction is a `store` of the tag and a `store` of the case struct through a gep into the
|
||
payload; `match` is a `load` of the tag and a gep the other way. There is one function that knows how a payload is
|
||
read — `case_field_addr` — because there are two readers: a `match` arm's binds and the structural printer.
|
||
|
||
The layout goes through the same oracle the DWARF section uses: LLVM's own answer for the emitted type, read back as a
|
||
constant-folded `ptrtoint`. Two unions are checked, one whose widest case is a pair of `f64` and one whose cases are
|
||
all `i32`, so the payload size and alignment cannot be constants the test agreed with by accident. The `Shape` in
|
||
`test/programs/unions.flan` was checked against clang's answer for the same declaration in C: 32 bytes aligned 8 with
|
||
the payload at offset 8, and 40/8 for a struct holding one.
|
||
|
||
### The surface
|
||
|
||
```clojure
|
||
(defunion Shape
|
||
[Empty
|
||
(Dot [x f64 y f64])
|
||
(Rect [w i32 h i32])
|
||
(Tag [name string n u8])])
|
||
|
||
(Shape.Rect {.w 3 .h 3}) ; a value: the type and the case, then the fields
|
||
(Shape.Rect {.w 3}) ; ZII, exactly as in a struct literal — .h is 0
|
||
Shape.Empty ; a case with no fields is a whole value, not a call
|
||
|
||
(match s
|
||
Empty "empty"
|
||
(Dot x y) "a dot" ; positional, in declaration order
|
||
(Rect w h) "a rect"
|
||
(Tag name n) name)
|
||
```
|
||
|
||
**Construction is qualified and a pattern is bare.** The two are not inconsistent: at a constructor nothing says which
|
||
union is meant, and at a pattern the scrutinee's type already does. The qualified spelling is accepted in a pattern
|
||
too, since that is how the value was written and writing it again should not be an error. Two unions may therefore
|
||
share a case name, and that is not refused — refusing it would be a restriction with no mechanism behind it.
|
||
|
||
Construction needed **no change to `parse.ml`**. `.` is a symbol constituent, so `Shape.Rect` reads as one name; the
|
||
struct-literal arm fires on a symbol followed by a map; and the field-access arm needs a *leading* dot, so the two
|
||
cannot collide. `(Name {.field value})` is one syntax for both, and which it is, is decided against the tables.
|
||
|
||
### Tags are declaration order, so case order is part of the contract
|
||
|
||
Tag *n* is the *n*th declared case, from zero. The consequence is that an all-bytes-zero union is **the first declared
|
||
case with a zeroed payload** — which is exactly the rule that makes an `Option`'s zero a `None`, and it is what makes
|
||
`(defvar u U)` and an omitted union-typed struct field mean something rather than nothing. Reordering a union's cases
|
||
is a layout change, the same way reordering a struct's fields is.
|
||
|
||
### What is refused, and why each
|
||
|
||
- **A non-exhaustive `match`.** Refused, not defaulted, and the message names the cases with no arm. A match that fell
|
||
through would have to produce a value of the match's type out of nothing, and the case a union grows tomorrow is
|
||
exactly the one a reader wants to be told about today. `_` is how to say "the rest", written where it can be seen.
|
||
- **A case pattern binding some of a case's fields.** All of them or none, positionally — binding a prefix reads the
|
||
wrong field the moment one is inserted above it.
|
||
- **Two arms for one case**, which is a mistake and never an intent.
|
||
- **A union with no cases.** No value of it can exist, so a parameter of that type is a function nothing can call.
|
||
- **A case field that is move-only**, in the same words a struct field already gets and for the same reason:
|
||
recursive teardown arrives with `drop`.
|
||
- **A union as a map key.** The payload past the case in hand is indeterminate, so hashing the blob would make two
|
||
equal values hash differently. Hashing one properly is a per-case walk driven by a switch — a different shape from
|
||
the field list `struct_key_pair` emits, and nothing has wanted it.
|
||
- **`uninit` on a union.** This is the one refusal that is *not* the struct rule. Everywhere else `uninit` is an
|
||
opt-out from ZII and the bytes are whatever they were: a garbage `f64` is a garbage number. A union's tag steers
|
||
control flow — a tag no case names falls past every comparison in a `match` into a block LLVM is entitled to assume
|
||
cannot be reached. That is the one place where garbage becomes "the optimiser may do anything".
|
||
- **`(.x u)`.** A union's fields belong to a case, and which case is being held is what the tag says. `match` is how
|
||
one is opened, and its arms bind the fields of the case they matched.
|
||
- **A global initialised with a case.** `(defvar g U (U.B {.x 1}))` would mean serialising the fields into the payload
|
||
blob at link time, which is a byte-level encoder this compiler does not have and which could not express a `string`
|
||
field at all — that is a pointer the linker has to relocate and a byte array has nowhere to put a relocation. A
|
||
*zeroed* global is fine and needs none of it: it is the first declared case.
|
||
|
||
### Recursion, and the shape `Form` will have
|
||
|
||
A union that contains itself by value has no finite size, and `payload_lay`
|
||
would recurse forever laying one out rather than failing. It does not get the
|
||
chance: `check_finite` already walked a union's cases, so `(defunion T [Leaf
|
||
(Node [l T r T])])` is refused with *"T contains itself by value, so it has no
|
||
size — go through (Ptr T)"*, and so is a pair of unions that contain each
|
||
other. Through a pointer it works, and that is the shape a `Form` has:
|
||
|
||
```clojure
|
||
(defunion Tree [Leaf (Node [l (Ptr Tree) n i32])])
|
||
|
||
(defn depth [t (Ptr Tree)] i32
|
||
(match (deref t)
|
||
Leaf 0
|
||
(Node l n) (+ n (depth l))))
|
||
```
|
||
|
||
### Why `match`'s fall-through is still `unreachable`
|
||
|
||
The block after the last case comparison is `unreachable`, kept from the
|
||
`Option` path. That is only sound if no reachable program can hold a tag no
|
||
case names — and none can: `Zero` is tag 0, which is a real case; every
|
||
construction writes a tag the checker resolved; and `uninit`, the one way to
|
||
get bytes nobody wrote, is refused on a union for exactly this reason. The
|
||
refusal is what pays for the `unreachable`.
|
||
|
||
### The diagnostics bug, fixed
|
||
|
||
`(A {.x 1})` on a case of a union said **"unknown struct A"**, because `env` had no table of case names and could not
|
||
tell a case from a misspelling. It has one now, keyed both by the full spelling `U.C` and by the bare `C`. The full
|
||
spelling is a *key* rather than something split out of a dotted name at the use site, because a union's own name can
|
||
contain a slash — an imported `rl/U` — and string surgery would own an edge this does not have to.
|
||
|
||
### Printing
|
||
|
||
`render.ml` walks a concrete type for `print`, the REPL inspector and the break buffer's locals, and a union fell
|
||
through its `Named` arm to `<Shape>`. It recovers the case from the tag by a chain of comparisons — the same shape the
|
||
enum arm already had, and for the same reason: the name is erased before any backend sees it — and reads the fields of
|
||
**that case only**. It prints `(Shape.Dot {.x 1.5 .y -2.5})`, which is what the source would write.
|
||
|
||
### What is left
|
||
|
||
An **imported** union is still refused by name at `load.ml:312`, so a union is file-local. That is not a blocker for
|
||
the macro expander: the prelude is parsed and prepended into the same flat namespace before `collect` runs, so a
|
||
`defunion Form` in `prelude.ml` is an ordinary same-file declaration needing no import — verified by declaring one
|
||
there and matching it from a program. `dev.ml`'s inspector still says "union values are milestone 6" for a stopped
|
||
frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards
|
||
is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates.
|
||
|
||
## `defer` may be written in a `let`
|
||
|
||
The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both
|
||
considered and rejected before it.
|
||
|
||
`defer` is a **compile-time** construct: the cleanup is copied into every exit path of the function. That is why a loop
|
||
body and a branch stay refused — a loop body's would fire once at function exit rather than once per iteration, and a
|
||
branch would have to express "maybe registered", which a form copied into every exit path or into none cannot say.
|
||
|
||
A `let` is neither. It is not a frame here: its bindings are function slots like any other and **nothing is released at
|
||
scope exit**, so a `let` at the top level of a function body has exactly the function's extent and a `defer` written in
|
||
it always registers. It was refused for a reason that does not apply to it.
|
||
|
||
The rule, stated precisely, because "top level of a function body" is easy to get wrong: a form at the top level of the
|
||
body may carry one, and so may a form in the body of a `let` that itself may — to any depth. A `let` inside a `while`
|
||
or an `if` has the loop's extent or the arm's, and inherits the refusal rather than the permission.
|
||
|
||
The implementation detail worth keeping: **the permission is granted again before every form of a body, never once
|
||
around the body.** `check` withdraws it as it starts, so granting it once would let the first `defer` through and refuse
|
||
the second — and two resources acquired in one `let` is the case this exists for. `defer-let.flan` covers exactly that,
|
||
along with nesting, interleaved registration order across the `let` boundary, and an early return.
|
||
|
||
This **amends `spec-memory.md`**, which states under "When storage is released" that `(defer (free v))` for a
|
||
`let`-bound `v` is "not expressible today" and that no idiom in the spec may depend on it. It is expressible now.
|
||
|
||
`do` at the top level of a body has the same extent argument and is deliberately **not** included: nothing asked for it,
|
||
and the rule is easier to state and to trust with one construct in it.
|
||
|
||
## Assets are baked in, and the reason it is a compiler feature
|
||
|
||
NEXT.md decision 1. `(embed "brush.png")` is a `[u8]`, `(embed "brush.png" string)` is a `string`, and
|
||
`(embed-dir "assets")` is a `[n EmbedFile]` sorted by name. Odin's `#load` and `#load_directory` are the model
|
||
(`src/parser.cpp`, and `check_load_directive` / `check_load_directory_directive` in `src/check_builtin.cpp`); Odin's
|
||
`#` is not imported, because an s-expression language already has a head position for a name and these resolve as
|
||
ordinary named calls exactly the way `vec-new` and `heap-allocator` do.
|
||
|
||
**The reason for this shape rather than a build flag is the one that decided it.** It is a *compiler* feature, so it
|
||
needs no linker arguments and no per-target packaging, and it works identically on desktop and web. That matters more
|
||
here than it does for Odin, because `Load` hands out `lflags` only to a directory package and `main` is not exported,
|
||
so a program can never be a package: the single file doing `(rl/load-texture "brush.png")` is structurally the one file
|
||
with **no link channel at all**. The web lane found that hole and did not invent a flag for it. Embedding has no such
|
||
hole, because there is nothing to tell the linker.
|
||
|
||
**It costs nothing at run time.** The bytes reach the program as a `Tast.Str` node typed `[u8]`, which emit.ml turns
|
||
into the same `private unnamed_addr constant` every string literal already becomes, and its `escape` is byte-exact
|
||
across the whole 0–255 range, so a PNG survives the round trip through the `.ll`. Bound with `defconst` at top level an
|
||
`embed-dir` is an LLVM constant outright, through emit.ml's `const`.
|
||
|
||
**A `Str` node typed `[u8]`, not a `Bytes` prim over a `string`.** This is the one non-obvious choice. `Bytes` is
|
||
identity — emit.ml lowers `Types.String` and `Types.Slice _` to the same `%slice` — but wrapping the literal in a prim
|
||
makes the node non-constant, and `const` then refuses an `embed-dir` in a `defconst` with *a global's value must be a
|
||
compile-time constant*. Both of emit.ml's string emitters take the bytes and ignore the node's type, so it is the same
|
||
constant either way and this one is a constant a global can hold.
|
||
|
||
**Two spellings, not one form that changes type with its context.** Odin threads a `type_hint` everywhere and can
|
||
afford `#load("p")` to mean a `string` here and a `[]u8` there. With structural equality, no implicit widening and no
|
||
coercion anywhere, the same text meaning two types would be a wart, so `string` is written down when it is wanted. The
|
||
site's expectation is a fallback only and nothing depends on it.
|
||
|
||
**The path is a literal and resolves relative to the file the form is written in.** Both are Odin's rules and for
|
||
Odin's reasons: the bytes must be in hand before any value exists, which is what makes the result free; and a path
|
||
relative to the compiler's working directory would make a package's assets depend on where `flan` was invoked from,
|
||
which cannot be right. A missing file is a compile error naming it, never an empty embed — an asset silently absent is
|
||
exactly the quiet wrongness this removes. An empty *directory* is not that case and embeds cleanly as `[0 EmbedFile]`.
|
||
|
||
**The directory lookup is a linear scan, and that is the chosen answer rather than the fallback one.** `embed-find` is
|
||
an ordinary prelude function over a `[EmbedFile]`. A directory embed is tens of entries whose names sit in cache-warm
|
||
`.rodata`; a compile-time perfect hash would be a build-time map with its own failure modes that nothing has asked for,
|
||
and sort-and-bisect is the next step if a program ever embeds thousands of files — it would not change the type. It
|
||
takes a **slice** rather than the array, because an array's length is part of its type and there are no generics, so
|
||
the call reads `(embed-find (slice assets 0 (len assets)) "brush.png")`. Entries are sorted by name because `readdir`
|
||
order is filesystem-dependent and an unsorted embed would make two builds of identical sources emit different `.ll`.
|
||
Non-recursive, files only — Odin again.
|
||
|
||
**The sharp edge, inherited and not widened.** The slice points into `.rodata`, so a store through it segfaults at
|
||
`-O0` and is deleted as undefined behaviour at `-O2` — the same trap the prelude's ASCII-case note measures for
|
||
`(bytes "Hi")`, and the same one NEXT.md tracks as "writing through a string literal". Nothing here makes it worse and
|
||
nothing here fixes it; provenance is what would. **To get a mutable copy, clone the bytes into a `Vec`.** It is worth
|
||
saying loudly because an embedded asset is precisely the thing someone will try to decode in place.
|
||
|
||
**What this does not do.** `sand.flan` still calls `(rl/load-texture "brush.png")`, which hands raylib a path for
|
||
raylib to open. Pointing raylib at embedded bytes needs `LoadImageFromMemory` and `LoadTextureFromImage` in place of
|
||
`LoadTexture` — a raylib binding question, not an embedding one — so the flagship program is not yet asset-free on the
|
||
web. The mechanism it needs is in.
|
||
|
||
## `slurp`, `barf`, and the two ways they fail
|
||
|
||
NEXT.md decisions 2 and 5. `(slurp path)` and `(slurp path allocator)` read a whole file into a `(Vec u8)`;
|
||
`(barf path bytes)` writes one.
|
||
|
||
**`slurp` waited for `Vec` because its result has no length until the file is read**, and it obeys spec-memory.md's
|
||
rule without an exception: *no allocating operation returns an error*. There is no `Result` here, no out-parameter and
|
||
no error code — `slurp`'s type is `(Vec u8)` and `barf`'s is `Unit`.
|
||
|
||
**Two failures, two conditions, and the guards nest rather than merge.** Allocation failure is `StorageExhausted` under
|
||
`retry`, unchanged and reused. File failure is `FileError {.path .op .reason}` under `retry` and `use-value`. They stay
|
||
apart because they ask two different answerable questions: the handler that grows an arena is not the handler that
|
||
supplies another path, and collapsing them would make one handler guess which it was looking at. `file_guard` in
|
||
check.ml is `alloc_guard`'s shape built from the same nodes — a `while`, a `restart-case` and an `error` — so the
|
||
backend learns nothing new.
|
||
|
||
**The restarts are Common Lisp's pair for a `file-error`.** `retry` for "the file may be there now"; `use-value [p
|
||
string]` for "try this other path". `use-value` is the first restart clause the **compiler itself** emits with a
|
||
parameter — typed restarts landed the same session — and its parameter *is* the path slot the attempt reads, so the
|
||
clause body is empty. emit.ml's `bind_params` stores the invoker's argument into the slot, the clause falls through,
|
||
and the loop re-attempts against the new path. Everything is inside that loop, so a `use-value` naming a different file
|
||
re-measures it and re-allocates for *its* size; the `Vec` is freed at the top of each turn, which is why a retry does
|
||
not leak, and freeing a `Vec` that never allocated is a no-op.
|
||
|
||
**The two forms resolve paths by opposite rules, and it is worth saying in one place.** An `embed` path is resolved at
|
||
*compile* time relative to the file the form is written in. A `slurp` or `barf` path is resolved at *run* time by the
|
||
host, against the process's working directory — these are ordinary values, and one can arrive from `argv` or from a
|
||
`use-value` restart. `test/programs/slurp.flan` reads `"programs/assets/a.txt"` only because the suite runs from
|
||
`_build/default/test`. Two forms in one section with opposite rules is exactly where someone gets bitten.
|
||
|
||
**The break loop can be offered this restart and cannot fill it in.** That is not new behaviour, only a new way to
|
||
reach it: a break loop chooses a restart by position and has nothing to supply a parameter with, and emit.ml already
|
||
emits a `flan_restart_unarmed` guard on every clause that takes one, so taking it refuses with the reason rather than
|
||
running the clause on a zeroed buffer. `slurp`'s `use-value` is simply the first such clause the *compiler* emits —
|
||
`alloc_guard`'s `retry` takes no parameters — and it rides the same path a hand-written one does. The acceptance suite
|
||
asserts the guard is on the emitted IR for both.
|
||
|
||
**On the web, `barf` signals — every time, with the path in the condition.** This is the decision worth restating,
|
||
because two more obvious answers are both wrong here. A **build-time refusal** is unusable: Flan has *no conditional
|
||
compilation*, nothing in `parse.ml` or `check.ml` reads the target, so "isolate this to desktop" is not expressible in
|
||
source and the refusal would have nowhere to be silenced. A **silent no-op** is worse than either, because that is how
|
||
a save file disappears with nothing said. So the program gets a condition and decides — which is the language having
|
||
something Odin does not. Odin stubs its whole file API on js/wasm to `.Unsupported` (`core/os/file_js.odin`) so that
|
||
importing `core:os` "panics cleanly", and a panic is not a decision. **The restriction was taken; the mechanism was
|
||
not.**
|
||
|
||
Nothing in the compiler reads the target to do this. The refusal is one `#ifdef __EMSCRIPTEN__` in `flan_rt.c`, which
|
||
is where the host ABI is *already* implemented twice. `slurp` keeps working on the web — it compiles, runs, and reports
|
||
a missing file honestly — and the bytes a web program actually wants come from an `embed`.
|
||
|
||
`test/programs/web-files.flan` is one source built for both targets, and `test/test_web.ml` **runs** it under node
|
||
rather than asserting the artifact's shape: an artifact-shape assertion would say nothing about the thing the decision
|
||
bought. The test checks that the refusal is printed with its path and reason, and that the desktop's success line is
|
||
*absent* — a silent no-op would have taken that branch.
|
||
|
||
### What the host ABI grew by, and why that much
|
||
|
||
plan.org names the filesystem as the #1 portability risk — "pack assets, one abstraction, never touch paths" — so the
|
||
widening is written down rather than assumed. It is **three calls and one reader**:
|
||
|
||
| Call | What it does |
|
||
|------|--------------|
|
||
| `flan_file_size(path, n, &out)` | how many bytes are there |
|
||
| `flan_file_read(path, n, buf, cap, &got)` | fill a buffer the caller owns |
|
||
| `flan_file_write(path, n, buf, len)` | write a whole file |
|
||
| `flan_file_fail_reason()` | which of the four reasons it was |
|
||
|
||
They are POSIX-shaped and **Vec-ignorant**: no handle crosses the boundary, nothing is held between calls, and each
|
||
takes a path and answers 1/0 the way every allocator entry point already does. `flan_slurp_into` — the part that knows
|
||
what a `Vec` is — is runtime *glue* on this side of the ABI, not a fourth host call, so a second target implements
|
||
three functions and inherits the rest. The reason is a global rather than an out-parameter for the same reason
|
||
`flan_alloc_fail_bytes` is: the condition is a value struct on the failing frame's stack with fixed numeric fields and
|
||
no rendered message.
|
||
|
||
**These do touch paths, which is the widening plan.org warned about**, and decision 2 took it knowingly. `embed` is the
|
||
half that does not: it needs no host ABI at all, so "pack assets" remains the answer for anything known at build time
|
||
and `slurp` is for bytes that genuinely are not.
|
||
|
||
`FileError` is one type with a `reason` field rather than a family, because conditions have no hierarchy today
|
||
(spec-conditions.md §1) and a family would need one handler clause per member to say "any file error". NEXT.md decision
|
||
4's parent link is the answer to that and is not built; when it is, these reasons can become types without any call
|
||
site changing.
|
||
|
||
**One thing `slurp` and `barf` reveal that is not theirs to fix.** A handler that wants "try to save, and carry on if
|
||
you cannot" has nowhere to go: `error` is diverging (spec-conditions.md §2), a handler returning normally has not
|
||
answered it, and neither `retry` nor `use-value` means *give up*. `web-files.flan` calls `exit` for that reason. A
|
||
`continue`-style restart — or `handler-case`, which unwinds — is what the case wants, and neither exists.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
|
||
| link | 20ms |
|
||
|
||
Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which
|
||
compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler
|
||
(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build),
|
||
`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0`
|
||
and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d
|
||
into place, so two concurrent builds cannot see a half-written one.
|
||
|
||
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
|
||
the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever
|
||
a disk-space decision.
|
||
|
||
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It
|
||
is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted
|
||
IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse
|
||
errors.
|
||
|
||
|
||
## The order it was built in, and why that order
|
||
|
||
Every item here is done; it is kept because the *sequencing* is the part worth remembering. Decided in conversation:
|
||
wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and **the dev loop is the
|
||
thesis of the project**, so it came first. Staged so each step was runnable on its own — the failure mode being to
|
||
build a daemon and a protocol before knowing whether the reload primitive worked.
|
||
|
||
1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms.
|
||
See the section above.
|
||
2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`,
|
||
`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a
|
||
run-time-added function is itself redefined. See the section above.
|
||
3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs,
|
||
and sand.flan polling at the top of its frame. See the section above.
|
||
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than
|
||
nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if
|
||
something else ever needs to talk to it.
|
||
|
||
**One decision left to settle before step 2**, because both change codegen and are painful to retrofit:
|
||
|
||
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module
|
||
declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep
|
||
the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload,
|
||
and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
|
||
- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what
|
||
`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from.
|
||
|
||
Deferred until after the dev loop:
|
||
|
||
6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and
|
||
`test/programs/sand-headless.flan` prints `15595743031174623232` under it — the same hash as native, byte for byte, at
|
||
`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared
|
||
across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer
|
||
would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and
|
||
runs it — rather than by looking for a binary on PATH.
|
||
|
||
Three things this cost that were not in the old note:
|
||
|
||
- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main`
|
||
to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a
|
||
signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")`
|
||
label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite
|
||
self-call.
|
||
- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes `<stdio.h>`; without `--sysroot`
|
||
it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the
|
||
triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`.
|
||
- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both
|
||
shapes count.
|
||
|
||
**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to
|
||
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
|
||
`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple
|
||
(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk
|
||
is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's
|
||
beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
|
||
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
|
||
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in.
|
||
|
||
**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of
|
||
`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes
|
||
an `ExperimentalWarning` to stderr on every run and the harness compares combined output.
|
||
|
||
**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with
|
||
one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime
|
||
at it).
|
||
|
||
~~Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot.~~ **Done — and it is a third
|
||
target, not a mode of this one.** wasi-sdk is right for the headless table and was never going to be right for the game
|
||
build. See the next section.
|
||
|
||
## The browser is the third target
|
||
|
||
`flan build --target=web` produces a page, its JS and a `.wasm`, and a raylib example opens in a browser from source
|
||
that was not touched. The two wasm targets share the word and almost nothing else, so `is_wasi` and `is_web` are
|
||
separate predicates and `is_wasm` is their union — the union is exactly the set of facts about the *machine* (32-bit
|
||
pointers, no `dlopen`), which is what the refusals are about, and nothing else is shared.
|
||
|
||
**The compiler is `emcc`, not `clang`, and that is the whole of the sysroot story.** Everything the wasi target has to
|
||
find by hand — a sysroot, a builtins archive, a shadow resource directory, the `__main_argc_argv` shim — is what emcc
|
||
*is*. `target_flags` for `web` is the empty list; the only thing checked is that emcc exists, refused by name where the
|
||
reason can say so. The one fact that had to be true for any of this: **emcc takes a `.ll` on its command line**, which
|
||
it does, so `Emit`'s output needs no change and the IR stays target-independent. The object cache keys on the compiler
|
||
binary's path, size and mtime as it always did — now of *whichever* compiler the target uses, so an emcc `flan_rt.o`
|
||
and a clang one cannot collide.
|
||
|
||
**The main loop: `-sASYNCIFY`, not `emscripten_set_main_loop`.** The older note above had the browser fact right —
|
||
it cannot be blocked — and drew the wrong conclusion from it. `emscripten_set_main_loop` wants the loop body as a
|
||
callback, so every one of the eleven examples that writes
|
||
|
||
```lisp
|
||
(until (rl/window-should-close?) ...)
|
||
```
|
||
|
||
would have to be split by hand into an init and a tick, and the web program would stop being the native program.
|
||
Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is
|
||
built for precisely that: `WindowShouldClose()` on `PLATFORM_WEB` is an `emscripten_sleep(16)` that then returns false
|
||
(raylib 5.5, `platforms/rcore_web.c`, read rather than assumed). So the loop yields once a frame at a call it already
|
||
makes, and **no example changed a character**. The price is real and is paid by every web build: asyncify instruments
|
||
the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a
|
||
question `Build` can answer, and a flag set that varies per program is a cache key that varies per program.
|
||
|
||
**`link` lines can be addressed to a target.** `vendor/raylib/link` named `libraylib.so.550`, which exists on the host
|
||
and nowhere else. A line may now carry `@native`, `@wasi` or `@web`, an untagged line applies everywhere — which is
|
||
what every existing `link` file already is — and `${NAME}` expands from the environment. The selection happens in
|
||
`Build` and not in `Load`, which is where the file is read, because **`Load` resolves imports before a target is
|
||
chosen**: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings.
|
||
`Load`'s part in this is to pass the lines through untouched, which it already did.
|
||
|
||
**raylib for the browser is built, not installed.** No emscripten port provides it (`emcc --show-ports`: there is
|
||
`contrib.glfw3` and no raylib), so `vendor/raylib/build-web.sh` clones raylib at the **5.5** tag — the one whose
|
||
`.so.550` the host links, because `raylib.flan` carries raylib's struct layouts and enum values and two targets built
|
||
from different raylibs would disagree about them in silence — and compiles the seven modules with
|
||
`-DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2` into one archive under `vendor/raylib/web/` (gitignored). `rglfw.c` is not
|
||
among them: the web platform uses emscripten's own GLFW port, which is why `link` carries `@web -sUSE_GLFW=3`. No
|
||
headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.
|
||
|
||
**The HTML shell is a string in `Build`, not a file in the tree**, for the same reason `Runtime_src` is: it has to be
|
||
wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas
|
||
and a `Module.print` that puts stdout on the page; `FLAN_WEB_SHELL` replaces it. `--shell-file` is passed only when the
|
||
output is a `.html`, because emcc accepts and ignores it otherwise.
|
||
|
||
**Refused by name, inherited whole:** `--dev`, `--debug`, `Build.shared` and `flan run --target=` are refused for
|
||
`web` exactly as for `wasm32`, each naming `web` rather than `wasm32` in the message. `--sanitize` is refused too, but
|
||
the web half of that refusal is weaker than the wasi half and says so: emscripten *does* ship an ASan, and nothing here
|
||
has ever run it. A sanitizer that has never been run is one whose silence means nothing.
|
||
|
||
**What the test can honestly check.** `test/test_web.ml` is headless and permanently so. It probes — emscripten may not
|
||
be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it
|
||
asserts: the three files exist, the module starts with `\0asm`, the page references its own JS and carries the canvas,
|
||
and node runs the emitted JS and gets `ok`. For raylib it builds `core-basic-window.flan` unchanged and then reads the
|
||
module for the two things that would be false if the mechanism were wrong: an `asyncify_start_unwind` export, and a
|
||
`glViewport` import that can only have come from raylib's web platform. Import and export names are plain strings in
|
||
the binary, so this needs no wasm reader.
|
||
|
||
## sand.flan in a browser
|
||
|
||
**The flagship program builds for the browser and the artifact opens.** Three things were between it and the target,
|
||
and none of them was the `#include` the earlier note named.
|
||
|
||
**The brush was a path, and a path is what cannot work.** `(rl/load-texture "brush.png")` hands raylib a filename to
|
||
open; a bare relative path has no meaning where there is no filesystem, so raylib would have opened nothing and the
|
||
cursor would simply have been missing. It is `(embed "brush.png")` now, decoded by a new binding —
|
||
`LoadImageFromMemory`, which completes the chain embedded bytes -> `Image` -> `Texture2D` that `LoadTextureFromImage`
|
||
already had the other half of. The declaration is `(Ptr u8)` plus an explicit `i32` count, because `shim.ml` refuses a
|
||
slice parameter and its refusal says exactly that; the Flan wrapper beside it takes the slice apart, which is the
|
||
idiom `collision-point-poly?` and `load-font-ex` already established. One decode serves both textures now: the
|
||
unflipped upload first, then `ImageFlipHorizontal` in place, then the mirrored upload — **the order is load-bearing**,
|
||
and if the two badges look the same it was swapped.
|
||
|
||
`load-texture` and `load-image` lose their only call site in this repository by this change. That is deliberate rather
|
||
than an accident of editing: a path-based load is the one shape the browser cannot have, and the bindings stay for the
|
||
desktop programs that will want them.
|
||
|
||
The embedded path resolves **relative to the file the form is written in**, which is why
|
||
`test/programs/sand-headless.flan` still works: it reaches `sand.flan` through `../../` out of a sandboxed `_build`,
|
||
and the PNG is found beside `sand.flan` and not beside the working directory. `test/dune` therefore lists `brush.png`
|
||
as a dependency of every stanza that builds sand — an embed is read by the *checker*, so it is a build input and not a
|
||
run-time one. The same fact reached `test_session`'s `C-c C-k` case, which re-evaluates sand.flan's whole text: it now
|
||
passes `~origin`, which is the buffer path both editor paths already send, because the default `<eval>` origin would
|
||
resolve the embed against the working directory instead.
|
||
|
||
**A package's C may be addressed to one target, the way a `link` line already could.** `Load` collects a package's
|
||
`.c` files by listing the directory, and there is nowhere in a directory listing to put a tag except the name, so the
|
||
tag goes there, before the extension:
|
||
|
||
```
|
||
vendor/agent/flan_agent.c compiled everywhere, unless displaced
|
||
vendor/agent/flan_agent.web.c compiled for the browser, and displaces the above
|
||
```
|
||
|
||
One rule: **a tagged file is compiled only on its own target, and there it replaces the untagged file of the same base
|
||
name.** Untagged is the default and every existing package is untagged, so nothing that did not opt in changed.
|
||
Replacement rather than the pure tagging Go's `_windows.go` and Odin's `file_js.odin` use, and the difference is the
|
||
point — pure tagging would mean renaming `flan_agent.c` to `flan_agent.native.c` to teach the package about a target
|
||
it had never heard of, and this way a package gains a target by gaining a file. The selection is in `Build` and not in
|
||
`Load`, for the reason `select_lflags` gives: `Load` resolves imports before a target is chosen.
|
||
|
||
**The dev agent on the web is a no-op, and it is not the `barf` decision being contradicted.** The compile error that
|
||
led here — `struct timeval` incomplete, because emscripten's headers do not pull `<sys/time.h>` in transitively — is
|
||
the surface. Underneath: *the agent is a socket server and a browser has no sockets*. Adding the include produces an
|
||
agent that compiles, links, starts and can never accept a connection.
|
||
|
||
Refusing `vendor:agent` on a web target was the other candidate and is ruled out by arithmetic, not taste. Flan has no
|
||
conditional compilation, `sand.flan` calls `(agent/start ...)` unconditionally, and `Reach` cannot prune a package
|
||
something reachable calls into — so a build-time refusal means the flagship program does not build for the browser at
|
||
all without being edited into a second program. **A refusal is only honest when the caller has a way to not ask.**
|
||
|
||
The two decisions look contradictory and are not, and the difference is *what the caller loses*. `barf` is asked to
|
||
make something durable; a no-op returns success to a program that now believes the bytes are on disk, and the loss is
|
||
real, is the user's, and is discovered later or never. The agent is asked to accept redefinitions from an editor; on
|
||
the web there is no editor, no socket and no session — `--dev` is refused by name on every wasm target, so a web build
|
||
has no cells to install a redefinition into even if one arrived. **Nothing is lost because there was never anything
|
||
there.** `sand.flan` already says the same about a *native* release build, at the call site: "Building without `--dev`
|
||
is fine — nothing has cells to install into, so a module is refused on the listener thread and the loop never
|
||
notices." A web build reaches that outcome by a shorter route. `start` returns `-1`, which is what `flan_agent.c`
|
||
returns for a path it cannot bind; `poll` and `wait` return 0, which is what the native build returns on every frame
|
||
nothing arrived on. The whole argument is written at the top of `vendor/agent/flan_agent.web.c`, where the next reader
|
||
will meet it.
|
||
|
||
### Building it and opening it
|
||
|
||
The raylib archive is built once and is not in the tree. From the repository root:
|
||
|
||
```sh
|
||
sh vendor/raylib/build-web.sh # clones raylib 5.5 and compiles it with emcc
|
||
export FLAN_RAYLIB_WEB=$PWD/vendor/raylib/web/libraylib-5.5.a
|
||
```
|
||
|
||
`build-web.sh` prints that `export` line itself. Then:
|
||
|
||
```sh
|
||
flan build sand.flan --target=web -o sand.html
|
||
```
|
||
|
||
**The output must be named `.html`.** `--shell-file` is passed only when it is, because emcc accepts and ignores it
|
||
otherwise — so `-o sand` produces a module with no shell, no canvas, and a page that looks like it built fine and
|
||
paints nothing. Three files land beside it, in whatever directory `-o` names: `sand.html`, `sand.js`, `sand.wasm`.
|
||
|
||
**A `file://` URL will not work.** The page fetches `sand.wasm`, and a browser refuses that from the filesystem. Serve
|
||
the directory holding the three files:
|
||
|
||
```sh
|
||
python3 -m http.server 8000
|
||
```
|
||
|
||
and open **`http://localhost:8000/sand.html`**. Left mouse paints; the keys are the ones the native build has.
|
||
|
||
### What only a human opening it can settle
|
||
|
||
Verified headlessly, by `test/test_web.ml`: the three files exist, the module carries `asyncify_start_unwind` and
|
||
`glViewport`, and **brush.png's own bytes are in the module, whole** — the assertion that keeps the embed from
|
||
rotting. Not `IHDR`: stb_image, linked in from raylib, carries that string itself, so an `IHDR` check would pass on a
|
||
build where the embed emitted nothing.
|
||
|
||
Not verified, and not verifiable here. `node sand.js` instantiates the module, runs `main`, and dies inside `glfwInit`
|
||
on `window is not defined` — which says the module is live and says nothing about the canvas.
|
||
|
||
- **Whether it paints at all.** Nothing in CI has ever seen a pixel of this.
|
||
- **Audio.** `start-audio` generates a tone, `ExportWave`s it to `/tmp/flan-sand-tone.wav` and loads it back as a
|
||
music stream. That is raylib's own `fopen`, not Flan's `barf`, so the `#ifdef` in `flan_rt.c` does not cover it and
|
||
emscripten's MEMFS may well give it a writable `/tmp`. Every use is behind `music-ok`/`tone-ok`, so a failure is
|
||
silence and not a crash. Separately, browsers suspend the audio context until a user gesture, so `audio-ok` may be
|
||
true while nothing is heard until the first click.
|
||
- **The loop never exits.** `WindowShouldClose()` on `PLATFORM_WEB` is an `emscripten_sleep` that returns false, so
|
||
`until` never terminates and **none of `main`'s `defer`s ever run** — no `CloseWindow`, no `UnloadTexture`. That is
|
||
correct for a page, which is torn down by the tab closing, and it is worth knowing before reading anything into it.
|
||
- **Canvas size against `screen-width`/`screen-height`.** The shell is a string in `Build` and its canvas is not sized
|
||
from the program, so 900x600 may be letterboxed or cropped.
|
||
|
||
## The colon belongs to keys, so a field label is a dot
|
||
|
||
`{.x 1.0 .y 2.0}` is how a struct is constructed, and `{inner .field}` is how a pattern names one. The colon is gone
|
||
from both, and what is left of it is one job: keys — map keys and enum members.
|
||
|
||
**What was wrong with the colon.** Nothing, taken alone. The problem was that it had two jobs and the dot had one.
|
||
`(.x v)` already read a field; `{:x 1.0}` also named a field, while `:space` named an enum member. So the dot meant
|
||
"field" and the colon meant "field, or member, depending". Moving the label to the dot leaves each mark with one
|
||
meaning, and it costs nothing to read because **the delimiter already disambiguates**: `(.x v)` is a list and
|
||
therefore a call and therefore an access; `{.x 1.0}` is a brace form and therefore a construction. There is no
|
||
position where the two could be confused, which is why the same spelling can serve both.
|
||
|
||
**Why it had to land before `Map`, and this is the real reason.** A map literal wants to be `{:key value}`. While
|
||
struct construction owned that exact spelling, a map literal and a struct literal were *the same syntax*, and the
|
||
only thing that could tell them apart was what the checker expected at that position. That is a context-sensitive
|
||
grammar for no gain. Reserving the colon for keys keeps the two visibly distinct at the reader, before any type is
|
||
known. Doing it after `Map` landed would have meant changing both; doing it first meant changing one.
|
||
|
||
**`:keys` kept its colon, and that is the point rather than an inconsistency.** `{:keys [x y]}` is the one thing in a
|
||
brace that is not a field name — it is an instruction to the compiler that happens to sit there, and it takes a
|
||
vector rather than a value. Giving it a dot would have made the dot mean "a field, or the word keys". Leaving it a
|
||
colon lets the dot mean exactly one thing, *this names a field*, which is the whole reason the colon was given up.
|
||
Everything else Clojure puts in that position — `:as`, `:or`, `:strs`, `:syms` — is still refused by its own name.
|
||
|
||
**The old spelling is refused, not accepted quietly**, and the refusal names the new one: `a field label is written
|
||
.x, not :x — the colon is for keys`. Two accepted spellings is how two spellings become permanent, and the standing
|
||
rule here is that what is not supported is rejected explicitly with the reason. Both refusals are tested by their
|
||
reason, on the construction side and on the destructuring side, which is what stops the colon drifting back.
|
||
|
||
**The sweep is a tool, not a one-off.** `tools/colon-to-dot.py` converted 681 labels across 45 `.flan` files,
|
||
`vendor/` included, and 94 more in the Flan embedded in `lib/prelude.ml` and the tests. It works on *forms*, not on text: a keyword
|
||
becomes a dot only where it sits in a field-label position inside a brace, so an enum member in value position
|
||
(`{.k :hi}`), a genuine EDN map inside a string (`test/programs/edn.flan`), and a type-position `{K V}` are all left
|
||
alone. It was kept in the tree because several lanes branched before it and their Flan needs the same pass at merge.
|
||
|
||
**What it deliberately did not change: the printed form.** `render.ml` still prints `(V {:x 1.5 :y 0})`. That string
|
||
is a wire format — `emacs/flan-inspect.el` parses it back and hard-codes the colon when it reads a field out — so
|
||
moving the printer alone would break struct inspection in the dev loop without breaking any test that says so. The
|
||
printer moves when its reader does, in the Emacs lane. It is the one place the old spelling is still correct, and
|
||
the reason is worth keeping: **a format with two ends only changes at both.**
|