The arm was written with the others and through the same deferral, and neither the generics row in test_flan.ml nor the paragraph in BUILT.md that enumerates what defers had it. Its placeholder is get's, for get's reason: it answers an (Option V), so the match around it still has to check while the key is a variable.
5698 lines
420 KiB
Markdown
5698 lines
420 KiB
Markdown
# How the parts that exist work
|
||
|
||
The reasons behind the code, kept separate from `NEXT.md` so that what to do next is not buried under what was already
|
||
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever `dlclose`d, why a call
|
||
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
|
||
than a function in the runtime. Deleting it would mean deriving it again.
|
||
|
||
`NEXT.md` is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
|
||
first. Come here when you need to know why something is the shape it is.
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a
|
||
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top
|
||
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer
|
||
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
|
||
bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision. **Amended** once a bounds
|
||
failure became a signal: an *answered* one leaves through the unwind path and runs them like any other transfer, an
|
||
unanswered one still runs none. See "An index out of range is a condition" at the foot of this file.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at
|
||
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot,
|
||
so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed
|
||
type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is
|
||
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets
|
||
the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value? ...])` gives a type that is an `i32` at run time and its own type
|
||
in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
|
||
A member's value may be left out, and then it is the one above it plus one, starting at 0 — C's rule, because these
|
||
enums are as often a transcription of a header as they are original. A value written twice is an alias and is allowed;
|
||
a value autoincrement *walks into* is refused, naming both members, because nothing in the source chose it.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
|
||
per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
|
||
in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning
|
||
garbage rather than as a link error.
|
||
|
||
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
|
||
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
|
||
copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare`
|
||
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
### The wrappers are generated now — `declare-c`, `lib/shim.ml`
|
||
|
||
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
|
||
generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory
|
||
holds `raylib.flan` and `link` and no C at all.
|
||
|
||
One binding is now one line:
|
||
|
||
```
|
||
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
|
||
```
|
||
|
||
`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
|
||
any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the
|
||
function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an
|
||
ordinary Flan `defn` above it when the signature has a struct in it. `flan shim <file>` prints the whole file.
|
||
|
||
**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path
|
||
string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h
|
||
i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural
|
||
rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited.
|
||
|
||
**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same
|
||
`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what
|
||
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
|
||
that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real
|
||
signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was
|
||
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
|
||
for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct
|
||
— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves.
|
||
Everything where they could diverge is refused at the field.
|
||
|
||
**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a
|
||
scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the
|
||
hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against
|
||
the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree.
|
||
|
||
**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path,
|
||
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
|
||
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
|
||
malloc failure, where the alternative is handing C a null pointer.
|
||
|
||
**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice
|
||
and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is
|
||
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
|
||
recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes
|
||
`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned
|
||
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
|
||
the reason, and the acceptance table asserts on the reasons.
|
||
|
||
**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc +
|
||
`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
|
||
that calls an existing binding is unaffected.
|
||
|
||
`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
|
||
`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17
|
||
34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity
|
||
would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed.
|
||
|
||
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write.
|
||
|
||
Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows.
|
||
|
||
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
|
||
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
|
||
GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins
|
||
`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all —
|
||
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`,
|
||
`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four
|
||
ways for that reason.
|
||
|
||
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
|
||
library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora
|
||
ships the runtime library without the `.so` symlink.
|
||
|
||
### The header is read now — `headers`, `lib/cimport.ml`
|
||
|
||
The section above ends by naming what the generator *trusts*: that the
|
||
`defstruct` matches the library's real struct, and that the `declare-c`
|
||
signature is the function's real signature. "No header is read, deliberately,
|
||
so nothing can check either." A header is read now, and both are checked.
|
||
|
||
**The dependency, which is the crux, and which this project already answered
|
||
once.** Zig's old `@cImport` ran clang as a *library*. That is exactly the
|
||
dependency plan.org rejected in "Why LLVM IR as text": a version-pinned C++
|
||
library breaks routinely on upgrade, a binary on PATH does not. So this shells
|
||
out for `clang -Xclang -ast-dump=json -fsyntax-only`, which is the same binary
|
||
every build already runs and adds nothing that is not already being paid for.
|
||
`lib/cjson.ml` is enough JSON to read that dump and no more, so it adds no opam
|
||
package either.
|
||
|
||
A note worth recording, because it strengthens the argument rather than
|
||
weakening it: **Zig has since abandoned clang here too.** `translate_c.zig` is
|
||
gone; `lib/compiler/translate-c/` is built on Aro, a C frontend written in Zig.
|
||
Their reason was to ship a compiler containing no clang at all — the opposite
|
||
premise to this one, where `clang` on PATH *is* the toolchain assumption. Both
|
||
projects walked away from linking libclang; only the destination differs.
|
||
|
||
**What is imported: functions, and only functions.** Not structs, not enums,
|
||
not macros. The bound on how much is not a curated list but the package's own
|
||
`defstruct`s — a function whose signature mentions a struct the package has not
|
||
described is refused with that reason, so `vendor/raylib` describing thirteen
|
||
structs is what makes the import thirteen structs wide, and describing a
|
||
fourteenth widens it. Of raylib 5.5's 581 functions, 256 import, 153 are
|
||
refused, and 172 are left alone because the package already binds them by hand.
|
||
(253 and 156 once `bindings` excludes raylib's three allocator entry points —
|
||
see the next section.)
|
||
|
||
Not generating `defstruct`s is what makes the check possible at all. Generate
|
||
them and the header becomes the authority on layout, and comparing the
|
||
package's layouts against the header's would be comparing the header with
|
||
itself. A `_Static_assert` on `sizeof`/`offsetof` was rejected in the section
|
||
above as circular for exactly that reason; **this is not circular, because the
|
||
two sides have different authors.** It is the cheapest real closure of the
|
||
trusted-not-guaranteed gap.
|
||
|
||
**Refusing by demotion, which is the one thing taken wholesale from Zig.** Zig's
|
||
translator never drops a declaration it cannot handle: `failDecl` binds the name
|
||
to a `@compileError` carrying the reason, so the name still exists, the program
|
||
still compiles, and asking for *that one name* fails at the use site with the
|
||
reason. A wholesale import has a hundred and fifty refusals and a caller cares
|
||
about the one they typed. Flan already had that mechanism — `Load.refuse_hidden`,
|
||
built for `main` — so `rl/get-gamepad-name` is not a name, and a program that
|
||
writes it is told *the return type is a string, and a string only crosses as a
|
||
parameter* rather than "unknown name".
|
||
|
||
That is also the split `shim.ml` needed and did not have. It refuses through
|
||
`Loc.fail`, which is right when a human named one function and wrong for a
|
||
wholesale import, where one returned `const char *` would kill the header. Same
|
||
judgement, different disposition; a hand-written `declare-c` still hard-fails
|
||
and `shim.ml` is untouched.
|
||
|
||
**Two C spellings mean something in a parameter that they mean nowhere else.**
|
||
`const char *` is a string going in, and the generator already knows how to hand
|
||
one over. `char *` without the const is very often a buffer the callee *writes*,
|
||
and handing it a NUL-terminated temporary would lose the writes with no
|
||
diagnostic anywhere — const is the only thing in the header that separates the
|
||
two, so it is what decides, and an out-buffer keeps a hand-written binding
|
||
saying `(Ptr u8)`. `long`, `size_t` and the rest are refused rather than
|
||
guessed, and for a reason specific to this project: it builds for x86-64, for
|
||
wasm32-wasi and for the browser, and `long` is 64 bits on the first and 32 on
|
||
the others, so a guess would be right for the target that gets tested and
|
||
silently wrong for two that do not.
|
||
|
||
**Naming.** `rl/InitWindow` is `rl/init-window`. Reversibility is not a property
|
||
of the rule — the C symbol is stored verbatim in the declaration, so the wrapper
|
||
reads the library's spelling rather than reconstructing it. What the rule must
|
||
be is *injective over one header*, since two C functions arriving under one Flan
|
||
name would surface as a duplicate declaration about a name nobody wrote. A
|
||
boundary goes before an uppercase letter after a lowercase one, before an
|
||
uppercase letter between an uppercase and a lowercase, and before a digit after
|
||
a lowercase; nowhere else. So `SetTargetFPS` is `set-target-fps` and not
|
||
`set-target-f-p-s`, `BeginMode2D` is `begin-mode-2d`, `UnloadUTF8` is
|
||
`unload-utf8`. raylib's 581 names are injective under it. **When two do collide,
|
||
neither takes the name** — resolving by order would mean that moving two lines
|
||
in somebody else's header silently rebinds a name a program is already calling.
|
||
Both are refused, both say why, and the author binds the one they want with a
|
||
`declare-c`.
|
||
|
||
**Where it runs.** `headers` beside `link` in the package directory, read the
|
||
same way: a path, any clang flags it needs, `${NAME}` expanded from the
|
||
environment. What comes back is ordinary `declare-c` declarations, generated
|
||
before the package's names are qualified, so they arrive as `rl/…` exactly like
|
||
the hand-written ones and nothing downstream can tell which is which. No new
|
||
form, no new `decl_kind`, no reader or parser change. A C symbol the package
|
||
already binds by hand is left alone, so `declare-c` stays the escape hatch and
|
||
stays the thing that wins.
|
||
|
||
A leading `?` makes a line optional, and `vendor/raylib` used it. "No raylib
|
||
headers are needed" is a real property — a build needs libraylib linkable, not
|
||
raylib-devel installed — and requiring a header would take it from everyone in
|
||
order to give the check to whoever has one. Unset `FLAN_RAYLIB_H` and the build
|
||
is exactly what it was; set it and every signature is checked. A path that is
|
||
*set and wrong* is an error naming it, because silently behaving as though
|
||
nobody had opted in is the difference between an opt-in and a trap.
|
||
|
||
*(Superseded. The `?` marker is still what it says here, but `vendor/raylib` no
|
||
longer uses it: the header is committed at `vendor/raylib/raylib-5.5.h`, the
|
||
line names it directly, and the check runs on every build. See "The header is
|
||
committed too" below, which is where the argument in this paragraph is
|
||
answered.)*
|
||
|
||
#### What the diff found
|
||
|
||
The evidence the whole lane exists for. Against raylib **5.5** — the version
|
||
whose `.so` `link` names — **all 16 `defstruct`s and all 172 hand-written
|
||
`declare-c` agree exactly.** The half BUILT.md called trusted is now checked,
|
||
and it was right.
|
||
|
||
That is only worth stating because the check has teeth. Against the **5.1-dev**
|
||
header installed in `/usr/local` it reports ten differences: nine functions that
|
||
version does not have (`CheckCollisionCircleLine`, the six `Is*Valid` renames,
|
||
`DrawRectangleRoundedLinesEx`) and `DrawRectangleRoundedLines`, which gained a
|
||
parameter. Picking the wrong header is therefore loud, which matters, because
|
||
the two headers are on the same machine and only one matches the linked library.
|
||
|
||
Both comparisons run **at build time** and stop the build, not just in the tool.
|
||
Verified by breaking them: a permuted `Texture2D` fails naming the field that
|
||
moved, and `f64` where raylib says `float` fails naming the parameter — which is
|
||
the hazard the section above calls out by name and says only a test can catch.
|
||
The message points at the line in `raylib.flan`, not at the header.
|
||
|
||
`flan import-c <header> [package.flan…]` prints what it would produce, what it
|
||
refused and why, and both comparisons, without building anything. That also
|
||
makes "generate once and commit the result" available for the cost of a
|
||
printer — explicit in the source, checked against reality, no header read at
|
||
build time.
|
||
|
||
#### What it costs, measured
|
||
|
||
The number that decides how much to import, because `reach.ml` was the reason to
|
||
think a wholesale import could be free.
|
||
|
||
| | today (172 by hand) | + 256 imported |
|
||
|---|---|---|
|
||
| release build, cold | 0.298s | 0.312s |
|
||
| release build, warm | 0.078s | 0.082s |
|
||
| redefinition (`flan reload`) | 31.0ms | 46.5ms |
|
||
| dev build, cold | 0.649s | 0.982s |
|
||
|
||
**`Reach.link` already drops a generated wrapper whose declaration nothing
|
||
reachable calls, and that is what makes the release column nearly flat.**
|
||
Confirmed on the case it exists for: a wasm32-wasi build of a program that
|
||
imports raylib and calls none of it still links without libraylib, with 256
|
||
extra declarations in play. Dev builds are not pruned, on purpose, so one
|
||
compiles all 428 wrappers — once, at session start, since `Build.shared` is
|
||
llc + `ld -shared` and compiles no C.
|
||
|
||
Reading the header is cached, and the cache earned itself against a measurement
|
||
rather than a guess: 64ms of a 72ms check, against 8ms for the whole program
|
||
without it. What is cached is the *extracted* signatures and not clang's JSON,
|
||
because the parse is half the cost — 30ms is clang writing 1.8 MB and the rest
|
||
is reading it. Keyed the way the object cache is keyed, on everything that could
|
||
change the answer: the header's path, size and mtime, the full flag list, and a
|
||
format version, since the value is marshalled. That takes the delta to 17ms.
|
||
|
||
**The 15.5ms on redefinition is the real cost and it is the argument against
|
||
importing at build time**, on the branch where the dev loop is the priority. It
|
||
is the strongest case for the third option — generate from the header, commit
|
||
the result, regenerate when the library moves — and that decision is open.
|
||
|
||
#### Where that 15.5ms actually is — and it is not the header
|
||
|
||
Written above as though a redefinition were re-reading the header on every
|
||
reload. It is not, and the number was never taken apart. Taken apart now, with a
|
||
timer around each stage of the same `flan reload` (raylib 5.5, sand.flan, one
|
||
`defn` in the forms file, warm caches):
|
||
|
||
| | no header | header imported | delta |
|
||
|---|---|---|---|
|
||
| `Session.create` | 7.0ms | 21.5ms | +14.5 |
|
||
| `Session.eval` | 4.8ms | 8.4ms | +3.6 |
|
||
| `llc` + `ld` | 21ms | 21ms | 0 |
|
||
|
||
And inside that `create`, the header's own share: reading the cached dump
|
||
**0.33ms**, turning it into declarations **3.3ms**, checking the package's
|
||
layouts and its hand-written signatures against it **0.55ms**. Roughly 4ms of
|
||
the 14.5, once. The other ~10ms is `Load` and `Check` doing their ordinary work
|
||
over 256 more declarations, and so is the +3.6ms on `eval`.
|
||
|
||
**So a `C-c C-c` in a daemon pays no header cost at all, and never did.**
|
||
`Session.eval` puts the evaluated forms through `Load.program`, and a form list
|
||
with no `(import …)` in it reads no package and therefore no header. The import
|
||
runs when the session is created, and again on a `C-c C-k` whose buffer carries
|
||
the package's own `import` line. `flan reload` is a fresh process per
|
||
redefinition, which is why its number carries a session's startup cost — it is
|
||
the CLI's shape, not the dev loop's.
|
||
|
||
What a redefinition does pay for an imported package is the +3.6ms of checking
|
||
and emitting against a program with 256 more declarations in it. That is in
|
||
`Check` and in `Emit.redefinition` declaring every sibling; it is real, it is
|
||
the number to attack next, and no cache touches it.
|
||
|
||
#### Two caches, and what a header change mid-session means
|
||
|
||
Both levels exist now. **On disk**, keyed on the header's realpath, size, mtime,
|
||
the full flag list and a format version — the object cache's own convention, and
|
||
it now lives in the object cache directory rather than a second one beside it.
|
||
Cold that read is 50ms of clang and parse; served from the file it is 0.33ms.
|
||
|
||
**In the session**, two tables in `Cimport`: the extracted dump by header, and
|
||
the generated declarations by header *and* by what the package already declares
|
||
(names taken, structs and enums known, C symbols bound by hand), so a package an
|
||
evaluation has added a decl to is worked out again rather than served a stale
|
||
answer. A repeat import in one process goes from 3.65ms to nothing — measured on
|
||
a whole-buffer evaluation, which imports twice.
|
||
|
||
**A header edited mid-session does not take effect until the session is
|
||
restarted**, and that is deliberate rather than incidental: the in-memory key is
|
||
the header's path and flags, with no mtime and no `stat`. It is the rule a
|
||
changed `.c` file already follows — the daemon compiled the package's C at
|
||
startup and a redefinition does not recompile it — and the rule the running
|
||
program follows, since its struct layouts are the ones it was built with. Taking
|
||
a new header mid-session would mean type-checking new signatures against a
|
||
process still running the old layouts, which is the silent disagreement the
|
||
whole header check exists to catch. The disk cache does key on mtime, so the
|
||
next session reads the new header rather than the old dump.
|
||
|
||
One attribution to not repeat: the disk cache is **not** what the dev build's
|
||
+333ms cold is about. Measured on this machine, a cold dev build of sand.flan is
|
||
1.37s without the header and 1.83s with it and both caches cold; clear only the
|
||
header cache and the same build is 0.92s against 0.83s. So the header read is
|
||
~60–90ms of a fresh session and the rest of the cold delta is the object cache
|
||
compiling a shim with 428 wrappers in it, which is the object cache's business
|
||
and already warm after one build.
|
||
|
||
### The check reaches the constants now — `bindings`, `check_constants`
|
||
|
||
The claim `generate-c` made was that every `defstruct` and every hand-written
|
||
`declare-c` agrees with the header, and that claim held. It said nothing about
|
||
a `defconst` or a `defenum` member — and **a wrong flag bit or a wrong enum
|
||
member is completely silent.** No link error, no type error; a window that does
|
||
not open, or a key that never fires. That is the class the header read exists
|
||
to catch and the class hardest to see by reading, and raylib's package carries
|
||
sixteen `ConfigFlags` bits and eight enums that were all transcribed by hand.
|
||
|
||
**A Flan constant has no C spelling stored anywhere, so one has to be built.**
|
||
A function never needs this: `declare-c` keeps the C symbol verbatim, so the
|
||
wrapper reads the library's spelling rather than reconstructing it. A constant
|
||
has no declaration to keep it in. So the rule is uppercase-and-underscore —
|
||
`left-shift` is `LEFT_SHIFT`, `msaa-4x-hint` is `MSAA_4X_HINT` — and the
|
||
*prefix*, which is nowhere in the Flan name, is declared in `bindings`:
|
||
|
||
```
|
||
enum Key KEY_
|
||
const flag- FLAG_
|
||
constant Gesture/double-tap GESTURE_DOUBLETAP
|
||
```
|
||
|
||
**Nothing goes quiet, in either direction, and that is most of the design.** A
|
||
name the rule builds and the header does not have is *reported*, because a
|
||
mapping that silently matched nothing would read as coverage and provide none
|
||
— worse than no check. A rule that reaches no Flan name is reported too, which
|
||
is what catches a typo in the prefix. And a `defenum` with no `enum` line is
|
||
itself a finding, because otherwise the silence simply moves up one level: the
|
||
next lane adds an enum, adds no line, and nothing notices. `enum Foo -` is how
|
||
a package says out loud that the header has nothing to check `Foo` against —
|
||
a sentence somebody wrote rather than a line nobody did.
|
||
|
||
**The two kinds of finding have different dispositions, which is the one thing
|
||
worth getting right here.** A value that does not match, or a C name the header
|
||
does not have, is the *library* contradicting the package — the same kind of
|
||
thing a permuted `defstruct` is, and an ordinary build stops on it. An enum
|
||
nobody mapped and a rule that reaches nothing are about the package's own
|
||
`bindings` file: real, worth fixing, and not a reason to fail somebody's build
|
||
with a message shaped like "your layout is wrong". Those gate `flan generate-c`
|
||
instead, which is where that file is edited and where the author is standing.
|
||
|
||
`defconst` is deliberately not held to that. A package's constants are mostly
|
||
its own — raylib's 26 colours, an example's screen size — and demanding a line
|
||
for each would be noise with no second author behind it. `gesture-all` is the
|
||
honest case: 1023 is the OR of ten members and no enumerator has that value,
|
||
so nothing claims to check it.
|
||
|
||
**Two things about clang's dump this rests on, both found by looking rather
|
||
than assumed.** raylib's enums are *anonymous* — `typedef enum { FLAG_VSYNC_HINT
|
||
= 0x40, ... } ConfigFlags;` is an `EnumDecl` with no name and a separate
|
||
typedef beside it — so the constants are collected into one flat table, which
|
||
is the table C itself keeps at file scope anyway. And an enumerator written
|
||
without `= n` carries no value in the dump at all, so values are counted the
|
||
way C counts them; `TraceLogLevel` is eight members with one initialiser
|
||
between them, and reading only the explicit ones would have checked one of
|
||
eight and passed the rest.
|
||
|
||
### An enum-typed `defstruct` field is a layout, not a disagreement
|
||
|
||
`Shim.cty` lowers a Flan `defenum` to `int32_t` in a struct field exactly as it
|
||
does in a parameter, and a C enum is an `int`, so the two are the same four
|
||
bytes. The signature check knew that; the layout check did not, and reported
|
||
`field projection is CameraProjection in the defstruct and i32 (int)` — which
|
||
cost `Camera3D` an enum-typed field and bought a conversion function beside it.
|
||
One predicate now serves both, symmetric so the enum may be on either side.
|
||
|
||
The tolerance is for a 32-bit integer and **nothing else**, which is the whole
|
||
point: `f64` where the library says `float` lays out eight bytes where there
|
||
are four, every field after it moves, and it reads as plausible numbers rather
|
||
than as a link error. An enum against an `i16` or an `i64` is a real
|
||
disagreement and stays one.
|
||
|
||
What it buys is at the construction site. `.projection :perspective` resolves
|
||
against the enum's members and a typo is a compile error there — a keyword
|
||
resolves only where an enum type is expected, so an `i32` field would have
|
||
taken any number at all. Fixing the layout check is therefore the whole of that
|
||
second problem for this case: make the field legal and the keyword follows.
|
||
|
||
### The bindings are committed now — `generated.flan`, `bindings`, `flan generate-c`
|
||
|
||
The section above reads the header at build time, behind an opt-in
|
||
`?${FLAN_RAYLIB_H}` in `headers`, because a build should need libraylib
|
||
linkable and not raylib-devel installed. That opt-in was doing two jobs and
|
||
only one of them was defensible: it decided whether a *check* ran, which is
|
||
fine to make optional, and it decided whether a package had 172 bindings or
|
||
428, which is not. `DrawTexturePro` being reachable only by exporting an
|
||
environment variable is the shape of that second job, and it blocked a real
|
||
game — see PORTING.md.
|
||
|
||
**Generate once, commit the result, regenerate when raylib moves.**
|
||
`flan generate-c vendor/raylib` reads the header named by `headers`, writes
|
||
`vendor/raylib/generated.flan`, and that file is checked in. No header is
|
||
needed by anybody: every build gets all 425 declarations, they are greppable,
|
||
and they show up in a diff when the library moves. **Caching is not the
|
||
argument** — the dump is already cached on disk and in memory, so a build that
|
||
reads a header pays for it once either way. The argument is the dependency and
|
||
the diff.
|
||
|
||
**What it costs is the check, so regeneration runs it and the check gates the
|
||
write.** A file on disk has no second opinion, so nothing compares the bindings
|
||
against the library on an ordinary build any more. The one function that writes
|
||
`generated.flan` therefore compares first — every `defstruct` against the
|
||
header's record, every hand-written `declare-c` against the header's signature
|
||
— and writes nothing when they disagree. It is not possible to regenerate
|
||
without comparing, because there is no other way to write the file. Pointed at
|
||
the 5.1-dev header on this machine while the package is written for 5.5, it
|
||
reports the same ten real differences the diff above found and writes nothing.
|
||
|
||
**The 172 hand-written lines stay, and the reason is not caution.** It was
|
||
tempting to delete them: 136 of the 172 are exactly what the kebab rule would
|
||
have produced, and the other 36 could be spelled as name overrides, so the
|
||
generated set really is a superset. The argument against is the one this
|
||
section is about. Everything the generator emits agrees with the header *by
|
||
construction* — the declaration and the prototype come from one dump — so
|
||
diffing generated output against the header it came from is a tautology, and
|
||
replacing the hand-written set would quietly reduce the signature half of the
|
||
check to nothing. The hand-written lines were transcribed from raylib's
|
||
documentation by a person; they are the only declarations in the package a
|
||
header can actually contradict. All ten of the 5.1-dev differences came from
|
||
them. They are not a parallel set to maintain — they are the second opinion,
|
||
and the check is what maintains them.
|
||
|
||
The opt-in did not go away, it stopped deciding anything important. With
|
||
`FLAN_RAYLIB_H` set, an ordinary build still reads the header, and since every
|
||
C symbol is now bound — by hand or by generation — the importer generates
|
||
nothing and the read is purely the check. It is a *better* check than before:
|
||
425 declarations rather than 172, because `generated.flan` is a package file
|
||
like any other and is checked like one.
|
||
|
||
#### The header is committed too, and the check is no longer opt-in
|
||
|
||
The paragraph above is superseded: there is no `FLAN_RAYLIB_H`, and the header
|
||
read is not conditional on anything. The header is committed at
|
||
`vendor/raylib/raylib-5.5.h` and `headers` names that path directly, so the
|
||
check runs on **every** build.
|
||
|
||
**What dissolved the opt-in argument is the commit, not a change of mind about
|
||
the property.** "A build needs libraylib linkable and not raylib-devel
|
||
installed" is still true and still the reason the opt-in existed — but nobody
|
||
needs raylib-devel to have a file that ships with the repository, so requiring
|
||
the header costs nobody anything. The trade the `?` was paying for stopped
|
||
existing.
|
||
|
||
**What being optional actually cost was found the hard way**, and the story is in
|
||
`vendor/raylib/headers` rather than repeated here. The short of it: the header a
|
||
tree happened to have lived under a gitignored directory, so several parallel
|
||
lanes were checking against nothing and were not told. **A check that silently
|
||
does not run is worse than no check** — the failure mode of an opt-in is not
|
||
"the check does not run", it is "the check does not run and the output looks the
|
||
same".
|
||
|
||
There is no environment variable for the path any more. To check against a
|
||
different header, edit the line or replace the file. The check still runs over
|
||
every declaration in the package, hand-written and generated alike, and only the
|
||
hand-written half can actually disagree — for the reason the section above gives,
|
||
which is unchanged.
|
||
|
||
**`bindings`, beside `headers`, is what survives regeneration.** A committed
|
||
generated file cannot be hand-corrected — the next run overwrites it and the
|
||
edit is destroyed without anybody being told, which is the worst shape an edit
|
||
can have — so the corrections have to live somewhere regeneration *reads*. Two
|
||
directives, which are the two things the header cannot decide: `exclude <symbol
|
||
or pattern>` and `name <symbol> <flan-name>`. A postprocessing transform pass
|
||
was considered and rejected: a second program to understand, run over text the
|
||
generator had already committed to.
|
||
|
||
Both are applied *while* the declarations are made, which is not a detail. The
|
||
kebab rule is consulted in exactly one place, so collision groups are computed
|
||
on the name a function will really take — which means renaming one of two
|
||
colliding symbols dissolves the collision instead of leaving both refused, and
|
||
`Spin2D`/`spin2d` gains a way out that is not a hand-written line. An excluded
|
||
symbol still reports that it was excluded rather than going quiet: "there is no
|
||
such binding" and "the package decided against this binding" are different
|
||
answers.
|
||
|
||
What is actually in raylib's: `exclude Mem*`, because raylib exports
|
||
malloc/realloc/free under its own names and binding them would put a second
|
||
untracked heap behind three innocuous-looking Flan names, against plan.org's
|
||
rule that an operation never falls back to a hidden allocator. And 19 `name`
|
||
lines giving the generated predicates the `?` spelling the hand-written ones
|
||
already use — `window-ready?` rather than `is-window-ready`, because
|
||
`key-pressed?` and `is-window-ready` living in one package is precisely the
|
||
split this change exists to remove. The C symbol is kept verbatim in
|
||
`Ast.DeclareC` either way, so a rename costs nothing: it is still what is
|
||
called and still what the check compares against.
|
||
|
||
#### What committing 253 more declarations costs, measured
|
||
|
||
The section above says the cost is the check. That is the cost that mattered,
|
||
but it is not the only one, and this file does not omit measured numbers.
|
||
|
||
Every build now carries 425 `declare-c` where a default build carried 172, and
|
||
the obvious worry is the shim: BUILT.md's own cold-build attribution above
|
||
blames "the object cache compiling a shim with 428 wrappers in it", and that
|
||
was the *opt-in* path. It is not what happens, because `Reach.link` drops the
|
||
bindings nothing reachable calls and the shim comes back in parts for exactly
|
||
that purpose. `sand.flan` links **110** wrappers, not 425 — `nm sand | grep -c
|
||
flan_shim_`. A program that never draws still compiles no drawing wrapper.
|
||
(`flan shim <file>` prints the unpruned view, so it says 427 and is not the
|
||
number a build pays.)
|
||
|
||
What is left is frontend work on 253 more declarations, and it is small:
|
||
|
||
| cold build of `sand.flan`, object cache cleared | best of 3 |
|
||
|---|---|
|
||
| 425 declarations (committed bindings) | 2.16s |
|
||
| 172 declarations (`generated.flan` moved aside) | 2.10s |
|
||
|
||
**+65ms, about 3%, and cold only** — the object cache serves the shim after one
|
||
build, and the redefinition path never recompiles C at all. Against it: the
|
||
header read this removes was 60–90ms of a fresh session by the measurement
|
||
above, and DISCUSS.md 6a measured it at 15.5ms on *every* redefinition, which is
|
||
a 50% increase on the number the dev-loop lane exists to keep small. So for
|
||
anyone who had the opt-in switched on this is a straight win, and for everyone
|
||
else it is 65ms once per cold build in exchange for 253 bindings that were
|
||
previously unreachable.
|
||
|
||
Two bindings the config cannot express stay hand-written, and they are the
|
||
reason `declare-c` remains the escape hatch: `LoadFontEx` and
|
||
`LoadImageFromMemory` are bound `-raw` and wrapped by a Flan function of the
|
||
same name without the suffix, one taking a slice and one answering with an
|
||
`Option`. The importer refuses them by name collision with those wrappers,
|
||
which is the correct answer.
|
||
|
||
### What a headless FFI test can and cannot pin
|
||
|
||
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
|
||
|
||
- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C
|
||
writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and
|
||
getting identical output.
|
||
- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the
|
||
way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance,
|
||
passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from.
|
||
Distances are worse: the reflection does not even reach them.
|
||
- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does
|
||
not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision
|
||
cases look like they cover it.
|
||
- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different
|
||
field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.
|
||
|
||
- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available.
|
||
`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values
|
||
in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely,
|
||
including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU
|
||
reads its width, height or mipmaps at all.
|
||
- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image
|
||
`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and
|
||
answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more.
|
||
- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the
|
||
package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the
|
||
`x`/`y` permutation above.
|
||
|
||
The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail
|
||
by permuting the fields and watching it go red. A case not verified that way is decoration.
|
||
|
||
One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It
|
||
measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no
|
||
window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime`
|
||
and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are
|
||
exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is
|
||
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares
|
||
becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
|
||
declarations whose names contain a slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file
|
||
named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below.
|
||
|
||
**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also
|
||
a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import
|
||
all four. A file carries no `.c` and no `link` file; those belong to a directory.
|
||
|
||
**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that
|
||
is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under
|
||
one set of names or the checker sees every declaration twice. It is the one decision here that would be hardest to
|
||
change later, because a qualified name is an identity the `layout` op and the break loop both resolve by.
|
||
|
||
A directory is keyed by its real path and read once, so a **diamond** shares one copy of its bottom package.
|
||
`programs/pkg-diamond.flan` is the case that proves it at the level that matters: `area/` and `draw/` both import
|
||
`shape/`, a `shape/Box` is built inside `area/` and passed to a function declared inside `draw/`. Read `shape/` twice
|
||
and there are two structs both named `shape/Box` which do not unify, so what proves the dedupe is that the program
|
||
prints `3 6 20` and not that it compiles.
|
||
|
||
The same directory under two *different* aliases is refused, including when one of the two is a package's own import
|
||
and the other is pages away in the entry file — `programs/pkg-alias-clash.flan`.
|
||
|
||
**A ring is refused and named.** Earlier it was not: the read-once table swallowed the second arrival, so mutually
|
||
dependent packages appeared to work, and BUILT.md said so. What that cost is a definite package order, which is what
|
||
the macro expander needs — every `defmacro` has to be compiled before anything that calls it — so it is now a refusal
|
||
that names the ring, `a -> b -> c -> a`, rather than the single import that happened to close it. The two questions are
|
||
kept apart by two pieces of state: the chain currently being read, and the set already finished. Found in the first is
|
||
a cycle; found only in the second is the diamond's second route and still a no-op.
|
||
|
||
`Load.t.pkgs` comes back **dependencies first** — the topological order the acyclic rule buys. The declaration list is
|
||
deliberately unsorted: `check.ml` collects every top-level name in one pass before it checks any body, so top-level
|
||
names are order-independent by construction. The order exists for the expander, which cannot work that way.
|
||
|
||
**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment
|
||
anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would
|
||
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
|
||
checker it would be "unknown name", which is true and useless.
|
||
|
||
**A package may declare a macro, and its name is the package's.** `(import mac "pkgs/mac")` makes `(mac/twice 4)` a
|
||
call and leaves `(twice 4)` an unknown name, exactly as for a `defn`: importing a package makes nothing globally
|
||
visible, macros included. Inside the package the names are unqualified, the rule every other declaration there follows.
|
||
|
||
This used to be a refusal, and the refusal's reasoning was wrong in a way worth recording. It said collecting a
|
||
package's macros would need that package's imports resolved *at the `Form` level, before `Load` runs* — a second import
|
||
resolver — and refused rather than build one. What it did not notice is that **the file being compiled is parsed before
|
||
`Load` runs too**, so no shape of this feature could have left import resolution where it was. The phases moved instead
|
||
of being duplicated: `Load.program` takes **forms**, reads the import forms out of them with `Load.imports_of`, resolves
|
||
them with the one resolver it always had, and only then parses the file — with the packages' macros already in front of
|
||
it. `imports_of` reads one shape and recurses into nothing; `import` is still the only thing that walks a package graph.
|
||
The acyclic-import rule is what makes the order definite, and it was already there for this reason.
|
||
|
||
The qualification cannot be the `Ast` rename every other name goes through. By the time `Parse` is finished with a
|
||
`defmacro` its quasiquote has been desugared, and a quasiquoted `(begin)` is a `Form.Sym` whose name is a *string in an
|
||
argument* — which is precisely the property that makes a quasiquoted call output rather than a dependency, and
|
||
precisely what puts it out of a rename's reach. So a package's macros are renamed over the **forms** their author wrote,
|
||
before desugaring, where `` `(begin) `` and `(begin)` are still the same shape. Locals shadow there as they do in the
|
||
`Ast` rename.
|
||
|
||
`Parse.imported_macros` carries the set, a ref for the same reason `Parse.expander` is one. It matters to the dev loop:
|
||
`C-c C-c` sends one form with no import in sight, so `Session` holds the set and **unions** into it rather than
|
||
replacing it — a session that replaced would expand `mac/twice` on the build and answer "unknown function" on the
|
||
reload.
|
||
|
||
Both unions put **what was just read off disk first**, because `macro_union` keeps the left on a name collision. Edit a
|
||
macro in a package and reload the file that imports it: the session has been holding that macro since it was created,
|
||
`Load` has just re-read it, and the other order goes on expanding the old body and says nothing about it. That is the
|
||
quietest failure in this area and `test_session` pins it in both places — the reload itself, and the `C-c C-c` after
|
||
the reload, which reads the set the session kept rather than the one `Load` handed it.
|
||
|
||
### `C-x C-e` expands too, and a declaration is not an expression
|
||
|
||
`Parse.expr` never ran the expander, so a macro call typed as a bare expression was an unknown name — a package's and
|
||
**the prelude's alike**, which is what said the gap was older than importable macros and not theirs. `(unless c a b)`
|
||
at `C-x C-e` failed exactly as `(mac/twice 4)` did. It is the wrap `Parse.decl` already had, applied to the other entry
|
||
point: quasiquote-desugar, expand, then parse the one form that comes back. `Session.eval_expr` puts
|
||
`Parse.with_imported` in front of it as `Session.eval` does, because the one expression an editor sends carries no
|
||
import and the session is the only thing holding what the imports brought in.
|
||
|
||
**What it expands is the prelude's macros and the imported packages', and not the file's own.** That limit is the
|
||
session's rather than this path's: `Macro.program` collects a file's macros by scanning the forms it is handed, and the
|
||
forms handed to an evaluation are the one thing that was sent. `C-c C-c` has always had the same limit for the same
|
||
reason, and a `defmacro` typed at the REPL becomes an ordinary `Ast.Defn` that nothing records as a macro. Left where
|
||
it was rather than half-fixed here, and pinned in `test_session` so that changing it is a decision.
|
||
|
||
**An expression that expands to a declaration is refused, by name.** `defn`, `defvar`, `defconst`, `defstruct`,
|
||
`defdata`, `defenum`, `defalias`, `defmacro` and `import` are heads `Parse.expr` now rejects — the arm that used to
|
||
say it for `defmacro` alone, generalised. It sits in the head dispatch and not in a walk over what the expander
|
||
answered, so it catches a declaration nested anywhere in the expansion for free, catches one **typed** by hand with the
|
||
same sentence instead of "unknown name defvar", and cannot drift out of sync with `decl`'s list the way a second copy
|
||
would. A *quasiquoted* declaration is deliberately not caught: after desugaring, the name in `` `(defn ...) `` is a
|
||
string inside a `Form.Sym` argument rather than a head — the same property that makes a quasiquoted call output rather
|
||
than a dependency. Building a declaration as a value is what a macro is for; evaluating one is not a thing an
|
||
expression can do.
|
||
|
||
**The non-termination refusals, and where each of them is.** This matters more here than in a build: `eval_expr` runs
|
||
inside the daemon, and a hang there wedges the editor with the program still on screen. The **spin** — a macro that
|
||
expands into a call to itself and does not get smaller — fires on this path, bounded at 200 rounds, and comes back as
|
||
a `Loc.Error` that `Dev.eval_expr` already answers as an error reply. The **ring** never reaches this path, and
|
||
finding out why was worth the trip: a ring is refused at the parse of whichever file first has both of
|
||
its members in scope — the package itself when the ring is internal to one, the importer when it is not — and that is
|
||
always a file `Load` reads before any session exists. So no program holding a ring can be loaded and no session over
|
||
one can be created. The refusal is in front of the path rather than on it, which is the stronger place for it; `test_session` asserts it at
|
||
`Session.create` so that moving the check later shows up as a failing test and not as a wedged daemon.
|
||
`test/programs/pkg-macro-idle.flan` is the fixture the spin needs — it imports and calls nothing, so the session is
|
||
created without expanding anything and the expression is the first thing that ever expands the macro.
|
||
|
||
Expansion happens **before** the thunk is built, so the three-way `` `Value | `Stopped | `Timeout `` wait in
|
||
`Dev.eval_expr` is untouched: a cold macro module costs its ~300ms before that 5-second clock starts, and `before` is
|
||
sampled after `Session.eval_expr` has returned. `C-u C-x C-e` wraps the **expanded** expression, so `Ast.pause_call`
|
||
takes a macro-stamped location — and `Loc.from_macro` sets a name and leaves file, line and column the call site's, so
|
||
the frame the break loop reports is still the line the reader is looking at. `temps` is not reset on this path, unlike `decl`'s — a declaration is
|
||
a fresh top level, an expression is evaluated into a session that has been handing out temporaries all along.
|
||
|
||
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
|
||
The gap is surface syntax and not `load.ml` — `exported` is one predicate and the refusal machinery that points at the
|
||
line which tried already exists, so a second rule is a line. What does not exist is any way for a package to *mark* a
|
||
name private, and inventing one is a reader and parser change.
|
||
|
||
## The link follows the program
|
||
|
||
`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the
|
||
importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that
|
||
link cannot succeed.
|
||
|
||
`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it,
|
||
follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing
|
||
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
|
||
|
||
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
|
||
`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs**
|
||
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
|
||
dropped one is a silently different program.
|
||
|
||
**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far.
|
||
|
||
The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and
|
||
`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker
|
||
arguments together, so a caller cannot take one without the other.
|
||
|
||
## sand.flan is one program
|
||
|
||
It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a
|
||
program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both
|
||
drivers imported it.
|
||
|
||
Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports
|
||
`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
|
||
raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both
|
||
targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation.
|
||
|
||
What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no
|
||
input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and
|
||
`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
Two claims that got run together in an earlier note, for the record:
|
||
|
||
- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the
|
||
**wasi** path, which is what the headless table targets, and which has no GL and no browser.
|
||
- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with
|
||
`emscripten_set_main_loop` instead of a `while`. ~~That is a different `main`, not a different program.~~ **The premise
|
||
held and the conclusion did not.** It is the same `main` and the same program: `-sASYNCIFY` answers the same browser
|
||
fact without cutting anything in half. See "The browser is the third target" below.
|
||
|
||
**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`,
|
||
and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`.
|
||
- `(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what
|
||
cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three
|
||
check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo
|
||
<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the
|
||
same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor
|
||
that split them would break the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s
|
||
unchecked. Indistinguishable.
|
||
|
||
## Sanitizers, and why hand-written IR does not get them for free
|
||
|
||
`--sanitize` builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last
|
||
of those is not what passing `-fsanitize=address` to the clang run over the `.ll` gets you, and the gap is silent.
|
||
|
||
**AddressSanitizer is an LLVM pass, but it instruments only functions carrying the `sanitize_address` attribute.** That
|
||
attribute is put there by clang's C frontend. `Emit` writes `.ll` by hand, so it wrote none, so the pass walked past
|
||
every Flan function and instrumented `flan_rt.c`. The measurement that settled it: an out-of-bounds read of a `defvar`
|
||
array in a `--no-bounds-checks` build printed its garbage and exited 0; with an `attributes #0 = { sanitize_address }`
|
||
group named on every `define`, the same program reports `global-buffer-overflow in flan.main`. Globals are the exception
|
||
— the module pass redzones them whether or not any function is attributed — which is why the *shape* of a sanitized
|
||
build looked right long before it worked.
|
||
|
||
**UndefinedBehaviorSanitizer has no equivalent lever.** Its checks are not a pass: the C frontend emits branches to
|
||
`__ubsan_handle_*` inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and
|
||
`(<< 1 32)` is still unremarked under `-fsanitize=undefined`. Shift UB, alignment and the f32→i32 cast on NaN are
|
||
therefore a compiler feature if they are wanted — checks emitted from `Emit` behind the flag, the same shape the bounds
|
||
checks already have — and not a flag away. `test_sanitize` pins both halves with controls: one program that must report
|
||
and one that must not, so either fact changing is a test failure rather than a discovery.
|
||
|
||
`-fno-sanitize=signed-integer-overflow` is the only exclusion, because wrapping is what this language's arithmetic
|
||
means and without it every program trips on its first `+`.
|
||
|
||
**The flag deliberately does not force `-O0`,** unlike `--debug`, whose reason (mem2reg deletes the alloca a
|
||
`llvm.dbg.declare` describes) does not apply. The optimiser is half of what is being measured, and `bounds.flan` proves
|
||
it: with checks off, its read past the end of a string constant is reported at `-O0` and silent at `-O2`, because an
|
||
out-of-bounds `inbounds` getelementptr into a constant is poison and LLVM folds the load away. The program then prints a
|
||
wrong answer instead of touching memory. Same family as `(<< 1 32)` compiling to a bare `retq`.
|
||
|
||
**What ASan covers of the bounds checks' job, since `--sanitize --no-bounds-checks` is the run that asks.** Three of
|
||
`bounds.flan`'s six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is
|
||
layout rather than anything about the access: ASan lays a global out as `{data, redzone}`, so reading before one lands
|
||
in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is.
|
||
Measured both ways — silent in `bounds.flan`, reported as soon as another `defvar` is declared in front of `arr`. A
|
||
reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees
|
||
out-of-*object* access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical
|
||
bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a
|
||
second net, not a replacement.
|
||
|
||
The sweep lives on its own dune alias rather than on `dune test`: a sanitized program links to a statically linked 1.8MB
|
||
binary, and twenty-eight of them twice over is minutes against the suite's seconds.
|
||
|
||
## Valgrind, and the two questions a sanitizer cannot be asked
|
||
|
||
`dune build --root . @valgrind` runs the headless corpus under memcheck: forty-nine programs checked, twelve of them a
|
||
second time with `--no-bounds-checks`, in **91 seconds** against `@sanitize`'s nine minutes — warm; a cold object cache
|
||
puts it at 162s, which is the compiles. That figure includes
|
||
the compiles, because the programs are built once here rather than twice. Memcheck is 20–50x on execution and the
|
||
corpus is small; the sanitized build's static link was always the expensive half.
|
||
|
||
**It needs nothing from `Emit`, and that is the entire reason it was reachable.** The `sanitize_address` attribute
|
||
story above is a story about an LLVM pass that only instruments what the C frontend marked. Memcheck instruments the
|
||
*binary*: it never sees IR, never sees an attribute, and cannot tell `Emit`'s output from clang's. Hand-written IR,
|
||
the runtime's C and libc arrive on the same footing. This is also why MSan was ruled out and memcheck was not —
|
||
MSan needs every dependency instrumented and raylib settles it, and memcheck needs none.
|
||
|
||
**The two tools are complementary and neither is a superset.** Measured on `bounds.flan`'s six deliberate
|
||
out-of-bounds cases: ASan catches three, memcheck catches **zero**. Every one of them is a global or a stack array,
|
||
and memcheck's *addressability* checking covers heap blocks only — it has no redzone concept for anything else. What
|
||
memcheck has instead is *definedness*, per byte, which ASan does not have at all. Believe neither sweep alone.
|
||
|
||
**The control that justifies the sweep.** Index 3 of a `Vec` with len 2 and cap 4 is inside the allocation — every
|
||
addressability check in existence says it is fine, ASan among them — and was never written. Memcheck reports it, and
|
||
`--track-origins=yes` names `flan_vec_push`'s `aligned_alloc` as where the undefined bytes came from. That is
|
||
NEXT.md's "ASan does not see uninitialised reads" turned into a test. `test_valgrind.ml` also asserts a heap overrun
|
||
that must report, and a `Map` key with two seven-byte holes that must *not* — the last being direct evidence for the
|
||
emitted per-key hash and equality pair walking fields rather than bytes, where `maps.flan` could only show the
|
||
consequence.
|
||
|
||
**What `--no-bounds-checks` actually removes, which is less than its name suggests.** `check_at` and `check_slice` in
|
||
`emit.ml` are behind the flag. A `Vec`'s and a `Map`'s bounds checks are *not*: they live inside `flan_vec_at` and the
|
||
map probe in `flan_rt.c`, are ordinary C, and run in every build. So the flag lowers the guard on fixed arrays and
|
||
slices only, and the single way to reach unguarded heap storage from Flan is a slice taken over a `Vec` — which is
|
||
what both positive controls do. This retro-explains why `unchecked_controls` in `test_sanitize.ml` only ever found
|
||
anything through fixed arrays.
|
||
|
||
### What a clean run does not prove
|
||
|
||
The sweep is clean. Enumerated, in the spirit of "three of six is a ceiling, not a measurement of the risk":
|
||
|
||
- **Globals and stack are outside it.** Measured: 0 of 6 against ASan's 3 of 6.
|
||
- ~~**Arena storage reused after `free-all` is not re-poisoned.**~~ **Closed.** It was the sharpest hole and it is
|
||
now the fourth control. Round one writes four elements; `free-all` resets the offset and keeps the pages; round two
|
||
allocates the same bytes back and reads one it never wrote — and still prints round one's `44`, because nothing
|
||
about the program changed, but memcheck now reports the read and `--track-origins` names `flan_arena_proc` under
|
||
`flan_alloc_free_all` as where the undefined bytes came from. Measured on the same machine: ERROR SUMMARY **0**
|
||
before, **6 errors from 4 contexts** after. `free-all` now issues memcheck's `MAKE_MEM_UNDEFINED` over
|
||
`[base, cap)` beside the registry's `flan_dev_reg_dead_range`, so the uninitialised-read coverage the headline
|
||
control demonstrates holds for the per-frame pattern the arena exists for as well as for the heap.
|
||
|
||
The client request is **vendored into `flan_rt.c`, not included**, and the argument is measurement rather than
|
||
taste: the machine that runs the sweep has `/usr/bin/valgrind` and no `/usr/include/valgrind` — `valgrind-devel` is
|
||
a separate package — so a guarded `#include` would compile to nothing on the one box where it matters and the
|
||
control proving it works would go quiet with no diagnostic. There is also nowhere to put an `-I`: `flan_rt.c` is
|
||
`cat`'d into an OCaml string literal (`lib/dune`, `runtime_src.ml`) and handed to clang in a scratch directory.
|
||
The vendored macro is `#if defined(__x86_64__)`-guarded, load-bearing rather than defensive, because the same
|
||
runtime is compiled for wasm32-wasi and emscripten where the inline asm would not assemble.
|
||
|
||
**Cost, measured.** Outside valgrind the request is four `rolq $n,%rdi` — which leave `%rdi` as they found it — and
|
||
`xchgq %rbx,%rbx`; 23 extra instructions on the `FLAN_ALLOC_FREE_ALL` path only, `flan_arena_proc` going from 151
|
||
to 174 instructions at `-O2`, and `flan_rt.o` from 53,008 to 53,112 bytes. Nothing on `alloc`, `resize` or `free`.
|
||
Fifty million `free-all`s in a loop: best-of-seven **1.19s** before and **1.24s** after, i.e. about **1ns** per
|
||
`free-all`, against a run-to-run spread of 0.18s within each series. At one arena reset per frame that is
|
||
unmeasurable; it is stated as a per-call number rather than "inside noise" because the noise floor here is wider
|
||
than the effect.
|
||
- **Interior overruns inside a single allocation are invisible by construction.** The arena's alignment padding and
|
||
the gap between its offset and its capacity are one block to memcheck, so a read across a sub-object boundary
|
||
crosses nothing. The same holds for the `Map`, whose `data` is *one* allocation laid out `keys | values | hashes |
|
||
scratch`: a probe walking off the end of the hashes array into the values region is not an error memcheck can see.
|
||
"Probe overrun at high load" is therefore not clean — it is **not observable by this tool**.
|
||
- **The union aliasing case is not exercised.** `unions.flan` reassigns a payload across cases and copies a union
|
||
through a struct field, but `match` is tag-dispatched and the checker enforces it, so reading case A's bytes after
|
||
writing case B cannot be written in the language. The sweep says nothing about that `getelementptr` because no
|
||
program can reach it.
|
||
- **raylib and the windowed examples are excluded**, so nothing is claimed about them. Under memcheck this matters
|
||
more than under ASan, not less: memcheck reports on uninstrumented code too, so including them would bury the
|
||
signal rather than lose it.
|
||
- **Both positive controls had to be synthesized.** No program in the corpus reaches a state where an uninitialised
|
||
read is observable. That is what a corpus passing its own acceptance table should look like, but it means the
|
||
sweep's value is as a regression net from here on, not as evidence that the current runtime was audited and found
|
||
sound.
|
||
|
||
`test/valgrind.supp` exists and contains **no suppressions**, which is a finding rather than an oversight: the sweep
|
||
was run with `--gen-suppressions=all` before the file existed and memcheck produced nothing to suppress — no false
|
||
positives from the hand-written IR, the arena or `zeroed`, and no true ones either. The file is the four expected
|
||
complaints with the reason each failed to appear, and the rule for adding to it: paste valgrind's own generated text,
|
||
and write above it why the report is not a bug.
|
||
|
||
One thing the sweep found that is not a memory defect: **`slurp.flan` is not idempotent.** Its last section expects a
|
||
missing file, and its handler `barf`s that file into existence and invokes `retry`; run twice, the second run finds
|
||
the file already there and prints a handler count of 0 where the first printed 1. Running each program plain and then
|
||
under memcheck is exactly two runs, so this presented as "diverges under memcheck" and was nothing of the kind —
|
||
reproduced with no valgrind anywhere near it. `test_acceptance.ml` already cleared the same two filenames for the same
|
||
reason; `test_valgrind.ml` now does too.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had
|
||
expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is
|
||
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
|
||
criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
It stopped being an absence when macros landed. A macro has to run at compile time and there is nothing to interpret
|
||
it with, so the compiler compiles it into a shared object and `dlopen`s it into its own process — see "Macros: the
|
||
compiler dlopens the program". The decision's cost and its mechanism are the same thing.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
() and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the
|
||
semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is*
|
||
the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody
|
||
addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a
|
||
`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding,
|
||
so wasm32 needs no exception proposal.
|
||
|
||
## The reload primitive — dev loop steps 1 and 2, measured
|
||
|
||
`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled
|
||
into its own object and called inside a process that is already running, twice, with a changed body the second time.
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| `Emit.redefinition` | below the timer (<0.1ms) |
|
||
| `llc -O2 -filetype=obj` | 15–17ms |
|
||
| `ld -shared` | 3ms |
|
||
| `dlopen` + `dlsym` | **0.04ms** |
|
||
|
||
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
|
||
number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost,
|
||
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since
|
||
the driver absorbs IR the bare tools reject.
|
||
|
||
`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that
|
||
*is* the mechanism. What the new module does **not** define is the whole design:
|
||
|
||
- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a
|
||
redefinition can change a function's body and can never re-initialise the program's data. Define the global and the
|
||
loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the
|
||
thesis.
|
||
- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a
|
||
private copy of it.
|
||
- **no `main`.** This module is loaded, not started.
|
||
|
||
Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss
|
||
because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is
|
||
spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `()` or for a
|
||
slice parameter.
|
||
|
||
**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the
|
||
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
|
||
agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release
|
||
builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_read`, which reads as a compiler
|
||
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and
|
||
the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason.
|
||
|
||
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a
|
||
loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property
|
||
of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol.
|
||
|
||
Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**,
|
||
since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since
|
||
`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
|
||
`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares
|
||
`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
|
||
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
|
||
|
||
String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's —
|
||
worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would
|
||
fail at link time. The fixtures each print a literal so that path is actually exercised.
|
||
|
||
### Cells — how a call site follows a redefinition
|
||
|
||
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes
|
||
every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current.
|
||
|
||
```
|
||
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
|
||
%p = load ptr, ptr @"flan.cell.bump" ; every call site
|
||
%r = call i64 %p()
|
||
```
|
||
|
||
Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and
|
||
exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what
|
||
makes a frame-boundary swap a non-event.
|
||
|
||
The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of
|
||
one.
|
||
|
||
Four things about this that are not free choices:
|
||
|
||
- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on
|
||
whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are
|
||
separate on purpose.
|
||
- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to
|
||
*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would
|
||
publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage,
|
||
because the failure is silent.
|
||
- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined
|
||
function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture
|
||
recurses on purpose, and would print the old body's text if it did not.
|
||
- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is
|
||
the first time `opts` means something semantic rather than an optimisation level.
|
||
|
||
LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46
|
||
indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well;
|
||
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
|
||
|
||
### Names that did not exist when the process started
|
||
|
||
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF
|
||
cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else:
|
||
|
||
```
|
||
void **flan_dev_cell(const char *name); /* a new function's cell */
|
||
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
|
||
```
|
||
|
||
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
|
||
new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores
|
||
on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
|
||
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
|
||
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
|
||
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
|
||
common case pays nothing for the general one.
|
||
|
||
**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c
|
||
C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a
|
||
redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet.
|
||
|
||
Four rules, each of which is a silent failure if broken:
|
||
|
||
- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are
|
||
still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`.
|
||
- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the
|
||
old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so.
|
||
This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
|
||
- **Nothing that published anything is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it
|
||
leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being
|
||
*pointed into*, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient
|
||
thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer,
|
||
or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the
|
||
handle *value* rather than the mapping: `dlopen` refcounts by path, so re-sending the same bad file raised a count
|
||
nothing could lower, and the one reference that could was dropped on the floor.
|
||
- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed
|
||
capacity with a loud failure rather than growable.
|
||
|
||
The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at
|
||
run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell*
|
||
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
|
||
pass and the transcript would read 246 instead of 432.
|
||
|
||
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout
|
||
calculator in OCaml that would have to agree with LLVM's on every target.
|
||
|
||
### The agent — dev loop step 3
|
||
|
||
`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link`
|
||
asks for `-lpthread`.
|
||
|
||
```
|
||
(agent/start path) listen on a unix socket; once, at startup
|
||
(agent/poll) install whatever has arrived; returns how many
|
||
(agent/wait ms) the same, but waits for something first
|
||
```
|
||
|
||
The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded —
|
||
so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a
|
||
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
|
||
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
|
||
loader.
|
||
|
||
**A full ring is refused, at the sender, before the `dlopen`.** Of the three honest answers this is the only one that
|
||
reaches the person who asked: dropping loses a reload the sender was told was `ok`, which is the same lie more quietly,
|
||
and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also
|
||
stop answering `status` and `abort`, leaving the dev loop with no way to reach a program that had stopped listening to
|
||
it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away,
|
||
since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says
|
||
what to do about it — call `agent/poll`.
|
||
|
||
**The way out of the break loop is `_exit`, not `exit`.** `exit` runs the atexit chain and the ELF destructors, which
|
||
want the loader lock the listener thread may be holding inside `dlopen`; a program asked to abort would hang instead of
|
||
dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134
|
||
stays because that is what a trap exits with.
|
||
|
||
`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits
|
||
for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first
|
||
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
|
||
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
|
||
|
||
Two details found by running it:
|
||
|
||
- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a
|
||
program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at
|
||
all, which is how this was found.
|
||
- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program
|
||
can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`.
|
||
- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows
|
||
when it is between frames.
|
||
|
||
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under
|
||
Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body
|
||
that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is
|
||
refused on the listener thread and the loop never notices.
|
||
|
||
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the
|
||
names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what
|
||
the session will track automatically.
|
||
|
||
### The session
|
||
|
||
`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change
|
||
accepted since.
|
||
|
||
**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a
|
||
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
|
||
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that
|
||
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
|
||
|
||
Two things the session knows that no single evaluation could:
|
||
|
||
- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST,
|
||
because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `println` reads
|
||
as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
|
||
- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded:
|
||
|
||
| Change | What it would have broken |
|
||
|---|---|
|
||
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below |
|
||
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
|
||
| a struct's fields | the values the process is holding have the old layout |
|
||
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it |
|
||
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
|
||
|
||
**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should
|
||
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
|
||
existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked
|
||
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
|
||
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
|
||
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
|
||
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
|
||
should not be read as the final answer.
|
||
|
||
A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past
|
||
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global`
|
||
record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than
|
||
the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is
|
||
the same fact that makes them unreloadable.
|
||
|
||
Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first,
|
||
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
|
||
uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for
|
||
that reason.
|
||
|
||
A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A
|
||
dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a
|
||
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
|
||
is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the
|
||
folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could.
|
||
|
||
**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in
|
||
`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s
|
||
own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a
|
||
brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it
|
||
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
|
||
did nothing.
|
||
|
||
It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a
|
||
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
|
||
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
|
||
either.
|
||
|
||
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise
|
||
re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name
|
||
pass rejects it. `C-c C-k` on sand.flan's own text is the test.
|
||
|
||
`flan reload <program.flan> <forms.flan>` is that path from the command line: a session over the program the process was
|
||
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw`
|
||
and 910 consecutive frames drew it.
|
||
|
||
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
|
||
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
|
||
has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to
|
||
`<eval>`, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
|
||
the real buffer path, which is the same key CIDER's `eval` carries.
|
||
|
||
### The daemon — `flan dev`
|
||
|
||
`flan dev <program.flan>` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside
|
||
the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part
|
||
of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe
|
||
the process that is actually running rather than a guess about it.
|
||
|
||
**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client
|
||
became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form,
|
||
from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the
|
||
editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of
|
||
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
|
||
newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor.
|
||
|
||
```
|
||
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
|
||
(:op "rerun") → (:status "ok" :note "running main again; …")
|
||
→ (:status "error" :message "the program is already running; …")
|
||
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
|
||
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
|
||
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
|
||
(:op "close")
|
||
```
|
||
|
||
`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the
|
||
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
|
||
It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing
|
||
else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none
|
||
to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself,
|
||
which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step
|
||
[i64 f32] i64`: types only.
|
||
|
||
The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan
|
||
dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by
|
||
guessing which directory it was relative to.
|
||
|
||
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
|
||
— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one
|
||
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
|
||
|
||
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so without it every error an editor shows
|
||
points into a file that does not exist.
|
||
|
||
Two things the daemon must not paper over, both of which would look like a successful evaluation:
|
||
|
||
- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan
|
||
says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead
|
||
fails silently: the module compiles, is built, and nobody receives it.
|
||
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed
|
||
connect or a reply that is not `ok` becomes an error the editor sees.
|
||
|
||
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads
|
||
like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather
|
||
than leaving an editor waiting on a socket nobody is serving.
|
||
|
||
### One process — `flan dev` is the program, and the compiler is a thread in it
|
||
|
||
`flan dev` builds **one binary** that is the compiled Flan program *and* the whole OCaml compiler, and `exec`s it.
|
||
There is no child: `/proc/<pid>/exe` on the pid an editor launched points at the program the daemon built, and
|
||
`test/test_dev.ml` checks exactly that rather than taking the claim on trust.
|
||
|
||
Who owns what:
|
||
|
||
| thread | what runs there |
|
||
|---|---|
|
||
| main | the program. C's `main` holds it — macOS needs a window on the main thread |
|
||
| a pthread | `caml_startup`, then `Flan.Dev.merged_setup` and `merged_serve`: the compiler and the editor socket |
|
||
| a pthread | `flan_agent.c`'s accept loop, as it always was |
|
||
|
||
The program keeps `main()` and the compiler comes up beside it, rather than the other way round, because that is the
|
||
shape DISCUSS.md §11 landed on: **the agent is already a server inside the program**, so the compiler moves *into* the
|
||
program's process. It is SLIME's model — you start the image, it serves, the editor connects.
|
||
|
||
`--two-process` is the escape hatch, for a machine where the compiler object cannot be built (no `ocamlfind`, no
|
||
`flan.cmxa` beside the binary). It has its own test and it stays.
|
||
|
||
**The editor socket and its wire protocol did not move.** Emacs cannot tell the difference, which is what made the merge
|
||
testable: the whole existing suite is the check.
|
||
|
||
Two rules hold in one address space and did not before:
|
||
|
||
- **The game thread must never call into OCaml.** A native thread has no safe points, so the collector can never stop
|
||
it — which is precisely why a frame is never paused, and precisely what one convenient direct call would undo. Requests
|
||
reach the compiler by being left somewhere and picked up, never by a call out of the frame loop.
|
||
- **No OCaml `value` goes into Flan storage** without `caml_register_global_root`. That is how "the GC does not touch
|
||
the arenas" stops being true.
|
||
|
||
A Flan `main` does not return — `Emit` ends it with `flan_exit` and an `unreachable` — so in one process that call would
|
||
take the compiler down with a program that merely *finished*. `flan_rt.c` has a hook, null in every other build, that
|
||
the merged entry point uses to flush and park.
|
||
|
||
#### The program is re-runnable — `rerun`
|
||
|
||
The park used to be `for (;;) pause()`, and that was a dead end with the process still standing: you ran a program, it
|
||
opened a window, you closed the window, `main` returned, and the only way to get another window was
|
||
`flan-dev-restart-program` — a new build, a new session, every global gone. Common Lisp and Clojure do not have that
|
||
problem because the image outlives `main` and you call it again. The process here already outlived `main`; nothing
|
||
could wake it.
|
||
|
||
So `main()` is a loop. The hook records the status and `longjmp`s back into a `setjmp` in `main()` — there is no return
|
||
available, since `flan_exit` is reached from wherever the program happened to be — and the thread then waits on a
|
||
condition variable. `(:op "rerun")` reaches `flan_merged_rerun` through a weak symbol in `lib/dynload_stubs.c`, the same
|
||
way `flan_agent_request` is reached, and signals it. **The main thread is the one that runs `main` again**: a window
|
||
belongs to the thread that opened it, and on macOS to the first thread of the process, so running the second `main`
|
||
anywhere else would draw nothing.
|
||
|
||
A `longjmp` pops no frame, so the park first empties the handler stack, the restart stack and the shadow frame chain
|
||
(`flan_condition_stacks_reset`, `flan_dev_frames_reset`). Each was a chain of allocas in stack the next run is about to
|
||
write over; a backtrace taken across that would name whatever the new run had put there.
|
||
|
||
**Nothing else is reset.** The second run sees the globals exactly as the first left them — that is the CL and Clojure
|
||
semantics and it is what was asked for. A clean slate is one evaluation away; a zeroed one cannot be had back.
|
||
|
||
**A re-run while the program is running is refused, not queued.** The test and the signal are under one mutex, so the
|
||
window between them does not exist, and two `main`s writing the same globals at once never starts.
|
||
|
||
`close`ing stdout went with this change, and it had to. That was how the compiler learned the program was done — the
|
||
pipe read EOF, exactly as the two-process daemon learns it from a dead child — but a pipe delivers EOF *once*, so the
|
||
signal and the program's output were the same resource: spending it left the second run with nowhere to print. The
|
||
descriptor hazard that came with it (POSIX hands out the lowest free fd, so the compiler thread's next socket became
|
||
this process's stdout, and the next `llc` inherited it) goes away with the close that caused it. Liveness is asked for
|
||
instead, through `Program.state`, which is a question with an answer rather than an event with one delivery.
|
||
|
||
That makes liveness **three states**, not two. `Dev.liveness` is `Live`, `Parked` or `Gone`, and every guard in
|
||
`lib/dev.ml` branches on it *before* consulting `Dev.state` — the agent's listener is alive while the program is parked
|
||
and answers `status` with "running", so the old order would have told someone whose program had finished that it was
|
||
running. Only `eval` accepts `Parked`: it queues a module and waits for nothing, and the queued module installs at the
|
||
first frame boundary of the next run, so a body can be fixed while the program is parked and the re-run executes it.
|
||
Everything else needs a frame boundary or a stopped stack, has neither, and says so by name — including `globals`,
|
||
whose storage is perfectly readable but whose *renderer* is a thunk the program has to run.
|
||
|
||
`:parked` rides on every reply beside `:stopped`, for the same reason `:stopped` does: a program finishes without
|
||
announcing it, and the commonest way to finish is somebody closing a window with the mouse. `:alive` keeps its old
|
||
meaning — is there still a session — so a parked program is `:alive t :parked t`. In Emacs that is `flan:parked` in the
|
||
modeline and `C-c C-M-x` (`flan-rerun`) to get the program back.
|
||
|
||
#### The internal socket is gone — the transport, measured
|
||
|
||
The merge deliberately deleted nothing, so that it could be tested against the shape it replaced. This is the first
|
||
thing to go: **the unix socket between the compiler and the program**.
|
||
|
||
It was a connect, a write and a read that looped back into the *same address space*. `vendor/agent/flan_agent.c`'s
|
||
verb table is now `handle_line(line, sink *)`, where a `sink` is either an fd or a growing buffer, and there are two
|
||
callers: `serve()`, which reads a line off a connection as before, and `flan_agent_request()`, which the compiler
|
||
thread calls directly. **One verb table, not two** — every answer is assembled from several pieces (a header, a name, a
|
||
newline), so the seam had to be the writing rather than the handlers, or they would have been duplicated and would
|
||
drift. `Dev.deliver`, `Dev.result` and `Dev.ask` were three copies of the same socket dance; they are now three lines
|
||
over one `Dev.request`.
|
||
|
||
**Which path is taken is decided by the linker, not by a flag.** `flan_agent_request` is declared *weak* in
|
||
`lib/dynload_stubs.c`, so it is null in every binary that links no `flan_agent.o` — the `flan` launcher, `flan reload`,
|
||
the two-process daemon, every test — and `Agent.request` answers `None` there, which is what makes `Dev` fall back to
|
||
the socket. A flag could disagree with reality; this cannot.
|
||
|
||
Three things had to be right:
|
||
|
||
- **The ring still has one producer at a time.** `queue_room` checks for space separately from `publish`'s store, which
|
||
was safe only because the accept loop was the sole producer and was single-threaded. The compiler thread can now ask
|
||
while the accept loop is serving, so `handle_line` runs under a mutex. It is held across the `dlopen`, which is what
|
||
the accept loop already did to itself by serving connections inline.
|
||
- **A delivery is still only *queued*.** The direct call publishes to the ring exactly as the listener did; the install
|
||
is one store per function on the game thread at a frame boundary. Installing on the spot would be a frame running half
|
||
in the old code and half in the new, and it would be an easy thing to do by accident here.
|
||
- **The OCaml runtime system is released across the call.** A delivery is a `dlopen` — milliseconds of relocation and
|
||
the loader lock — and holding OCaml's lock through it stalls every other OCaml thread. DISCUSS.md §14's third cost, in
|
||
the one place this change creates it.
|
||
|
||
**What it is worth, measured.** This machine, warm caches, a one-`defn` redefinition driven over the editor socket,
|
||
median of 12; the four columns taken back to back in one sitting, because the run-to-run drift on this machine is
|
||
larger than what is being measured:
|
||
|
||
| | merged, before | merged, after | `--two-process`, before | `--two-process`, after |
|
||
|---|---|---|---|---|
|
||
| redefinition, end to end | 22.1ms | 21.2ms | 20.8ms | 22.1ms |
|
||
| ...of which the build (`:ms`) | 19.5ms | 18.8ms | 18.4ms | 19.5ms |
|
||
| a `break` round trip | 0.061ms | **0.020ms** | 0.060ms | 0.064ms |
|
||
|
||
The `break` row is the editor socket *plus* one question to the agent, so the ~41µs it lost is the whole of the
|
||
internal socket. Everything else is noise: the redefinition column moves by less than its own spread and moves in both
|
||
directions.
|
||
|
||
**The transport was about 40µs of a 21ms redefinition, and removing it does not move that number.** That is the
|
||
finding, and it is worth more than a speedup would have been. The sharpest evidence for it is that the merged and
|
||
two-process columns *change places between runs* — two-process ahead by 1.3ms in the before pair, merged ahead by
|
||
0.9ms in the after pair — which is what a difference made of noise looks like. **The merge's prize was never
|
||
latency.** It is that
|
||
the compiler and the program share an address space, which is what makes the items below deletable at all and what
|
||
unblocks reading the stopped frame's memory directly. Anyone reaching for an in-process JIT on the strength of
|
||
"transport is slow" should read the build row first: code generation is 19 of the 21 milliseconds, and the socket was
|
||
0.2% of it.
|
||
|
||
The test that pins it is a deletion, because a reply cannot say which way it came: `test_dev.ml` **unlinks the agent's
|
||
socket file** once the merged program has bound it, and then runs every evaluation in the file. Unlinking a bound unix
|
||
socket does not disturb the listener, it makes new connects fail — so if the deliveries still install, nothing
|
||
connected. The agent still binds it, for `--two-process` and for a person at a raw socket.
|
||
|
||
#### The 4K result cap is not a transport buffer, and it stays
|
||
|
||
Next on the list, and the answer is no — with half of it deleted anyway, which is the useful part.
|
||
|
||
`RESULT_MAX` reads like a wire size: 4096 bytes, a cap on a value the compiler reads back after `C-x C-e`. It was
|
||
written down **twice**, once in `runtime/flan_dev.c` and once in `vendor/agent/flan_agent.c`, with a run-time check
|
||
that the two had not drifted. That second copy *was* transport — a buffer sized to be sent through a socket — and it
|
||
is gone: the agent asks `flan_dev_result_cap()` and allocates, so the bound is one file's decision now and the
|
||
drift check has nothing left to check.
|
||
|
||
**The bound itself cannot go, and the reason is the rule the merge was built around.** `result` is the buffer the
|
||
**game thread** writes into, from a render thunk at a frame boundary. A growable one means the frame thread calling
|
||
`realloc` — an allocation in the one place this design exists to keep allocation out of. And it would break the
|
||
seqlock, which the premise for this work correctly says must stay: a seqlock is a protocol about torn *contents*, and
|
||
it assumes the address it `memcpy`s from neither moves nor goes away underneath the reader. Growing on the writer's
|
||
side is a use-after-free the counter cannot see.
|
||
|
||
So it is a render budget, not a wire size, and it was only ever mistaken for one because the agent had a copy of it.
|
||
Removing the bound is a redesign of the *read* — probe the length, allocate, re-read, validate the generation, retry —
|
||
and it belongs with moving the read to a frame boundary, which is the seqlock's own decision and its own lane.
|
||
|
||
#### The break snapshot is not marshalling either, and all of it stays
|
||
|
||
Third on the list, and the answer is no, with nothing left over. "The compiler can read the stopped frame's memory
|
||
directly, so copying it is now ceremony" is the right instinct and the wrong diagnosis: **the snapshot was never about
|
||
two address spaces. It is about two threads, and there are still two.**
|
||
|
||
A stopped program is not holding still. The break loop polls, `flan_agent_poll` runs whatever the compiler delivered,
|
||
and a `C-x C-e` thunk is arbitrary Flan — it pushes and pops the one global restart list and the shadow stack while it
|
||
runs. The compiler is a thread beside it either way. So every copy in `snap_push` has the same justification it had
|
||
before:
|
||
|
||
- **Restart names** are copied because serving them off the live list hands the reader a pointer into a frame the
|
||
break loop's own poll may already have popped. A pointer is meaningful to the compiler now; the frame it points into
|
||
is no more alive for that.
|
||
- **Frame names and locations** are copied from the held-still stack for the same reason, and the *fingerprints* have a
|
||
sharper one already written down: the module a frame's description lives in can be unloaded once a replacement is
|
||
installed, and the comparison happens after that.
|
||
- **The generation stamp** (`snap_gen`, `chosen_gen`) is about nested breaks, not about processes. A thunk this loop
|
||
runs can error, push a break of its own, and reach `chosen_ready` first — claiming an index someone chose from the
|
||
outer list. Depth cannot tell those apart, because an outer break resuming and a new one starting reuse the number. A
|
||
generation can. One process changes nothing about that.
|
||
|
||
The fixed caps go with it: the snapshot is taken **on the game thread**, so it cannot allocate, which is why
|
||
`SNAP_MAX`, `FRAME_MAX` and `FRAME_TEXT` are literals and why truncation is reported rather than avoided.
|
||
|
||
What the merge *does* unlock here is one thing and it is the next item: `flan_agent_frame_slot` already hands back the
|
||
address of a slot, and in one process the compiler could read the value at that address instead of compiling a render
|
||
thunk to print it. That is the render-thunk-per-inspection redesign — a different mechanism rather than a deletion, and
|
||
what makes "the inspector can retain a value" reachable. It is deliberately not done here.
|
||
|
||
**So of the three things the merge was expected to make deletable, one was.** The socket was transport and is gone; the
|
||
result cap and the snapshot are both concurrency, and they were only ever mistaken for transport because the socket was
|
||
the thing in front of them.
|
||
|
||
### The Emacs client
|
||
|
||
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is
|
||
s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is
|
||
Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is
|
||
written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords.
|
||
|
||
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a
|
||
request and `read` reads a reply.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `C-c C-c` | the top-level form at point, recompiled and installed |
|
||
| `C-c C-k` | the whole buffer, as **one** module |
|
||
| `C-x C-e` | the expression before point, evaluated *in the running program* |
|
||
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
|
||
| `C-c C-o` | the running program's own output, in `*flan-output*` |
|
||
| `C-c C-r` | a prompt on the running program (`*flan-repl*`) |
|
||
| `C-c C-b` | what a **stopped** program is offering, and which to take |
|
||
| `C-c C-d` | what the running program currently defines |
|
||
| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR |
|
||
| `M-.` / `M-,` | where a name is written, through an `xref` backend |
|
||
|
||
eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on
|
||
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
|
||
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just
|
||
installed completes at once.
|
||
|
||
The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away,
|
||
which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the
|
||
socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the
|
||
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
|
||
|
||
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to
|
||
arrive in the same load, or the first refers to storage that does not exist yet.
|
||
|
||
**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary,
|
||
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
|
||
has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives
|
||
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
|
||
mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml`
|
||
and fails here.
|
||
|
||
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
|
||
next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte
|
||
offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
|
||
`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the
|
||
daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1
|
||
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
|
||
reader skips, so the reply's line numbers are the buffer's own.
|
||
|
||
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
|
||
success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the
|
||
one point looked like it was in.
|
||
|
||
**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to
|
||
into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form
|
||
"asm"|"ir")` is the op.
|
||
|
||
What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d
|
||
--disassemble=flan.<name>` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared`
|
||
deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was,
|
||
so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else
|
||
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
|
||
bookkeeping.
|
||
|
||
**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket
|
||
takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address
|
||
space, and `C-x C-e` renders a pointer as `<ptr>` on purpose — so nothing the daemon can ask would tell it what a cell
|
||
holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first
|
||
instruction:
|
||
|
||
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*;
|
||
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
|
||
- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed,
|
||
since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further
|
||
installs until it resumes.
|
||
|
||
From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch
|
||
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
|
||
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
|
||
the debug build lands, that is the line to delete.
|
||
|
||
**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an
|
||
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
|
||
`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the
|
||
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
|
||
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
|
||
in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the
|
||
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
|
||
is Emacs' contract and a test claiming to check it would be checking nothing.
|
||
|
||
**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the
|
||
next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the
|
||
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
|
||
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
|
||
accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
|
||
grow the daemon without limit.
|
||
|
||
### `C-x C-e` — evaluating an expression
|
||
|
||
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
|
||
into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by
|
||
exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an
|
||
expression reading the program's state sees a point the program agrees is consistent.
|
||
|
||
**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say
|
||
what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout
|
||
decision's bill, and it is why the printer set is small rather than universal.
|
||
|
||
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
|
||
dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's
|
||
socket. The read is safe without a handshake because the counter is a **seqlock**, and it had to be made into a real
|
||
one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived
|
||
and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to
|
||
be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the
|
||
bare pointer was the bug rather than the ordering. `flan_dev_result_read` copies into the caller's buffer and checks
|
||
the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader
|
||
that loses the race reports the last *complete* generation and no bytes, so a daemon polling for a new value keeps
|
||
polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's
|
||
"has it moved" still means what it meant. Marking the counter odd needs a release *fence* and not a release store — a
|
||
release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which
|
||
is the original bug with more ceremony. And the odd mark is *set* rather than incremented, because a `begin` with no
|
||
`end` is reachable: the thunk calls `begin` before it evaluates anything, so an expression that signals is stopped
|
||
inside that window, and a restart taken from the break transfers past the thunk and `end` never runs. Incrementing
|
||
would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit
|
||
means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has
|
||
reached a frame boundary.
|
||
|
||
**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a
|
||
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
|
||
structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth
|
||
and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three
|
||
bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a
|
||
builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time
|
||
type to walk. Not the walk itself.
|
||
|
||
The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise
|
||
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
|
||
allocator the language does not have.
|
||
|
||
```
|
||
big 18446744073709551615
|
||
col :blue
|
||
(.pos b) (V {.x 1.5 .y 0})
|
||
b (Blob {.id 7 .name "sandy \"quoted\"" .pos (V {.x 1.5 .y 0}) .tags [ 0 42 0]})
|
||
(slice (.tags b) 0 3) [ 0 42 0]
|
||
(rl/get-color 0x11223344) (rl/Color {.r 17 .g 34 .b 51 .a 68})
|
||
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
|
||
```
|
||
|
||
Details that are decisions rather than formatting:
|
||
|
||
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come
|
||
back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point
|
||
instead.
|
||
- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather
|
||
than as the value it is.
|
||
- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are
|
||
erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is
|
||
exactly what you would want to see.
|
||
- **A pointer is never followed** — `<ptr>`. It is the only thing that could make the walk cycle, and dereferencing one
|
||
a REPL was handed is not a safe thing to do on someone's behalf.
|
||
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does
|
||
not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K
|
||
and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it
|
||
means no renderer carries a budget.
|
||
- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is
|
||
not evaluated once per element.
|
||
|
||
What still refuses by name: `Map`, `Fn`, a type variable.
|
||
|
||
A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer.
|
||
`flan run calc-me.flan "1.5 * 2.0"` has always said `3`.
|
||
|
||
An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe`
|
||
does not fill up with `eval/N` for every expression ever typed.
|
||
|
||
**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by
|
||
`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
|
||
into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the
|
||
agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition*
|
||
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
|
||
and can never claim this.
|
||
|
||
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop
|
||
would exhaust it.
|
||
|
||
The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must
|
||
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
|
||
|
||
### The REPL buffer
|
||
|
||
`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No
|
||
new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as
|
||
deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess
|
||
behind it — the "process" is a stub comint needs in order to have a prompt at all.
|
||
|
||
Three things about it that are decisions:
|
||
|
||
- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in
|
||
sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it
|
||
because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
|
||
- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a
|
||
paren inside a string does not count.
|
||
- **A value and the program's output are different things and arrive by different routes.** The value is the result of
|
||
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
|
||
and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the
|
||
separation.
|
||
|
||
That test is what caught a real bug: the renderer's unit case emitted `()` without evaluating the expression, so
|
||
`(println "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A `()`
|
||
expression is almost always a call made for its effect, and is now evaluated and *then* reported.
|
||
|
||
### Conditions — step 1: `handler-bind` and `signal`
|
||
|
||
`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters
|
||
control flow**: `signal` returns `()` whatever it finds, a handler that returns normally leaves the signalling
|
||
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
|
||
yet, and no signature changed.
|
||
|
||
```
|
||
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
|
||
(load-all))
|
||
```
|
||
|
||
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
|
||
function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame
|
||
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
|
||
step.
|
||
|
||
Three decisions worth keeping:
|
||
|
||
- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and
|
||
every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
|
||
- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is
|
||
nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the
|
||
name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address
|
||
instead.
|
||
- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a
|
||
branch in the function that wrote it.
|
||
- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from
|
||
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
|
||
particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the
|
||
duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still
|
||
valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the
|
||
stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When
|
||
`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
|
||
|
||
Which gives the two refusals, both by the house rule rather than by accident:
|
||
|
||
- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a
|
||
reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in
|
||
scope, which is what the accumulation case needs.
|
||
|
||
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
|
||
outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by
|
||
value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete
|
||
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
|
||
biggest usability limit in conditions is not behind the thing that was just deferred.
|
||
- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would
|
||
leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block.
|
||
|
||
### The break loop — conditions step 3
|
||
|
||
Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the
|
||
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
|
||
|
||
```
|
||
flan: unhandled Missing — stopped, not dead.
|
||
0. restart: retry
|
||
1. restart: use-placeholder
|
||
```
|
||
|
||
Four decisions, each of which is the reason something is where it is:
|
||
|
||
- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the
|
||
release runtime and must not depend on something a program may never import. A program with no agent leaves the hook
|
||
null and dies exactly as it did before.
|
||
- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes,
|
||
reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same
|
||
act, lowered once. Nothing about §6 needed changing to support it.
|
||
- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a
|
||
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
|
||
that did not drain that queue would hang `C-x C-e` exactly when it is most wanted.
|
||
- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must
|
||
not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new —
|
||
and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the
|
||
cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of
|
||
stopping.
|
||
|
||
**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but
|
||
a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also
|
||
`compute-restarts`' data, whenever that arrives.
|
||
|
||
**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused
|
||
there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that
|
||
cannot happen.
|
||
|
||
The socket verbs are `restarts`, `restart-at <n> [name]`, `restart <name>` and `abort`, and all four are refused with
|
||
the reason when the program is not stopped — there is no restart stack to walk from a running one. `restarts` answers
|
||
`<index> <+|-> <name>` per line: the index is the identity, and the flag says whether a transfer to that frame has
|
||
anywhere to land. `test/programs/break.flan` errors three times — two restarts taken by name, then a shadowed pair
|
||
where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved
|
||
by name could pass.
|
||
|
||
### The break loop in the editor — conditions step 3, the other half
|
||
|
||
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it
|
||
follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a
|
||
question an editor chose to ask.
|
||
|
||
So the state is learned **twice, deliberately**:
|
||
|
||
- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`,
|
||
or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now
|
||
errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding
|
||
out *after* the echo area had said the evaluation was fine.
|
||
- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is
|
||
drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies
|
||
that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left
|
||
behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the
|
||
`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs
|
||
timers, so a poll firing inside a read would eat the reply that read was waiting for.
|
||
|
||
Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is
|
||
stopped, so the poll and the prompt cannot disagree.
|
||
|
||
```
|
||
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
|
||
:stopped t :condition "Missing")
|
||
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
|
||
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
|
||
(:op "abort") → (:status "ok" :note "the program is exiting; …")
|
||
```
|
||
|
||
`:restarts` is *positional* — innermost first, duplicates kept — because the position is what `restart-at` takes.
|
||
`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is
|
||
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
|
||
prompt cannot take a different restart than the one it showed.
|
||
|
||
`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by
|
||
someone about to choose from it.
|
||
|
||
**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against
|
||
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
|
||
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
|
||
lets the next poll settle it.
|
||
|
||
The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running
|
||
one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly
|
||
right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key —
|
||
it is the thing you pick when none of the restarts is the answer.
|
||
|
||
**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A
|
||
`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
|
||
cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the
|
||
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
|
||
breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices
|
||
each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an
|
||
expression that errors and resumes it, which fails against the old shape.
|
||
|
||
The agent grew one verb, `status`, answered in **both** states — `running` or `stopped <condition>`. Everything else the
|
||
break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had
|
||
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
|
||
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
|
||
|
||
`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the
|
||
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
|
||
round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e`
|
||
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
|
||
|
||
### `layout` — a type's fields, with no program involved
|
||
|
||
```
|
||
(:op "layout" :type "sim/Cell") → (:status "ok" :type "sim/Cell"
|
||
:fields (("heat" "f32") ("next" "(Option sim/Cell)")))
|
||
→ (:status "error" :message "Missing is not a qualified name; …"
|
||
:candidates ("a/Missing" "b/Missing"))
|
||
```
|
||
|
||
The daemon can answer this with nothing running. A layout is a fact about the *build*, and the daemon owns the build —
|
||
`Tast.structs` is sitting in the session it compiled the process from. That is why the conditions buffer can name and
|
||
type a condition's fields while every one of their *values* stays refused: the shape is knowable and the contents are
|
||
not, and drawing them apart says more than drawing neither.
|
||
|
||
**The type is a name, and the name is the qualified one.** This was the open question — a class name is not an identity,
|
||
and two packages each declaring `Missing` would leave the daemon unable to pick. It turned out to need no new
|
||
machinery: `Load.qualify_decl` rewrites `Defstruct (n, …)` to `Defstruct (alias/n, …)` at import, so by the time
|
||
anything reaches `Tast.structs` the names are one flat namespace in which a collision cannot exist. The name *is* the
|
||
type id, with no table to keep in step across a reload, and the existing spelling of a type — `Types.to_string` —
|
||
already prints it.
|
||
|
||
**And the break loop was already speaking it.** `Emit.struct_name_of` takes `Types.Named n` — the qualified name — and
|
||
passes it to `flan_error`; the agent holds it in `condition_name`; `break` answers it as `:condition`. So the string
|
||
the conditions buffer already had in hand resolves as `:type` by construction, and `test_dev.ml` round-trips exactly
|
||
that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat
|
||
worth writing down — `condition_name` is a `char[128]`, so the round trip holds for names up to 127 bytes and a longer
|
||
one is truncated and will not resolve.
|
||
|
||
**A bare name is refused, not resolved**, even when only one struct's last segment matches it. Resolving a unique
|
||
suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot
|
||
rely on. The refusal carries `:candidates`, so a person is one copy-paste from the answer and a client has its
|
||
completion list — the same shape as `package_of` refusing a directory imported under two aliases rather than picking
|
||
one. An enum is refused by *kind* (`X is an enum, not a struct`): its members are erased to `i32` before `Tast.program`
|
||
exists, which is the same fact that makes a `defenum` unreloadable. A union is refused the same way and for its own
|
||
reason — it is declared, and union *values* are milestone 6. Both are `Types.Named` at a use site, so falling through
|
||
to "no struct is named X" would say a type does not exist about one that plainly does.
|
||
|
||
**`render.ml` is not reused, and that is not a second walk.** It walks a *value* and emits the code that prints it;
|
||
this describes a *type* and emits text. What is shared is the spelling: field types go through `Types.to_string`, which
|
||
is what `defs` spells a signature with, so `(Option T)`, `[T]`, `[n T]` and `(Ptr T)` read the same in a layout, in a
|
||
signature and in the source. A field that is itself a struct shows its qualified name — which is a `:type` this op
|
||
accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude
|
||
structs are answered like any other, because `Render` resolves against the same list and an editor that could see a
|
||
`Split` printed but not ask about it would be the two disagreeing.
|
||
|
||
On the Emacs side `flan-cnr-layout` makes the request and `flan-cnr-show` passes the result into
|
||
`flan-cnr-state-from-reply`, which stays a function from data to data so the fixture-driven tests keep working without
|
||
a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and
|
||
turning `C-c C-b` into an error would take away the restarts — the decision the buffer exists for — over a missing
|
||
annotation.
|
||
|
||
### The shadow stack, and `backtrace`
|
||
|
||
plan.org's "Dev vs release builds" table has had **Frames: shadow stack** in the dev column since the project began and
|
||
nothing had ever built it. This is it. It is what `(:op "backtrace")` is made of, and it is dev-only, so a shipped game
|
||
pays nothing — the same bargain the indirection cells already make.
|
||
|
||
Chosen over the DWARF route deliberately, and the author's reason is the one that decides it: **the more a break loop
|
||
can show, the less often a real debugger is needed.** DWARF buys frames in lldb; this buys them in the break loop,
|
||
where someone already is when they want them.
|
||
|
||
A frame is four words on the calling function's own stack — the frame it displaced, a pointer to a static description
|
||
of the function, and two words reserved for locals. The description is per function and not per call, because nothing
|
||
about a function changes between two calls to it: the qualified name, `file:line:col` as `Loc` spells it, and how many
|
||
slots the frame has. So the name in a backtrace comes off the frame itself and needs no debug information, no symbol
|
||
table and no agreement with the optimiser about what a stack frame looks like. It is the same mechanism on wasm32,
|
||
which is the other reason it is not `.eh_frame`: plan.org lowers every non-local exit explicitly rather than through
|
||
platform unwinding, so there is no unwinder here to borrow.
|
||
|
||
The compiler emits the push and the pop **inline** rather than calling into the runtime. This is on every call in a dev
|
||
build, and a call made to record a call would be most of what it costs.
|
||
|
||
**The pop is at every `ret`, and the transfer path is the one that matters.** There are five: an explicit `return` with
|
||
a value and without, the `none` arm of `(some x)`, the tail of the body, and the landing block a transfer leaves
|
||
through. `emit.ml` funnels all five through one `ret`, because the way to get this wrong is to write the pop on the
|
||
normal path and not on the other one — and then every *handled* error leaves a dead frame behind, and the backtrace
|
||
after the fifth one is five frames of fiction. `test_dev.ml` takes five breaks and resumes all of them by condition
|
||
transfer before asking for a backtrace, and the answer has to be two frames. Same lesson `with-allocator` learned about
|
||
restoring at the pad.
|
||
|
||
**A plain global, not a thread-local.** It matches what the handler stack and the restart stack in `flan_rt.c` already
|
||
assume: one thread runs Flan, and the listener thread runs C and the loader and never enters a Flan body. If the
|
||
language grows threads this becomes thread-local and the compiler's two stores become TLS-relative, which is the whole
|
||
of the change.
|
||
|
||
**The chain is snapshotted on the stopped thread**, into the same `snapshot` the restarts are copied into, at the same
|
||
moment and for exactly the same reason: the break loop *polls*, a poll runs Flan, and a chain read by the listener
|
||
thread is a chain that can be popped underneath the reader. Names are copied as bytes rather than kept as pointers,
|
||
because a transient `C-x C-e` module does get `dlclose`d and its rodata with it. Frame *addresses* are kept beside the
|
||
text, because reading a frame's locals means going back to that frame and not to whatever is at index 2 by then.
|
||
|
||
```
|
||
(:op "backtrace") → (:status "ok" :frames (("fetch" "/game.flan:15:7" "program" 0)
|
||
("main" "/game.flan:27:7" "program" 0))
|
||
:more 0 :stopped t :condition "Missing")
|
||
```
|
||
|
||
Innermost first. `:more` is how many frames deep recursion left off the end — the innermost ones are what the question
|
||
is about. **`origin` is `"program"` or `"eval"`**: a break inside a `C-x C-e` thunk has that thunk's frames on top of
|
||
the program's, and "where is my program" answered with `eval/7` is true and not the question. The boundary is recorded
|
||
at the call, in `flan_agent_poll`, exactly where `restart_floor` is and for the same reason — except that its
|
||
out-of-a-thunk value is `-1` rather than `0`, because zero restarts on the stack is a real answer and zero frames
|
||
belonging to the program is not.
|
||
|
||
**Refused while the program is running**, like every other break verb. The chain is the game thread's and it is pushed
|
||
and popped on every call; a walk of it from the daemon would have the shape of a backtrace and the contents of a race.
|
||
|
||
**What it costs, measured rather than assumed.** Two benchmarks, one compiler built per variant, every binary kept and
|
||
then run alternately. The figure is the **minimum of nine runs**, because this machine is shared and a mean measures
|
||
whatever else was running; the whole table reproduces to three digits on a second pass.
|
||
|
||
| Dev build, -O2 | no frames | frames | frames + slots |
|
||
|---|---|---|---|
|
||
| 2000 sweeps of a 100×100 grid — sand's inner loop, in miniature | 30.0 ms | 40.0 ms (+33%) | 48.3 ms (+61%) |
|
||
| fib(30) plus 20M calls in a loop — nothing but calls | 28.7 ms | 31.1 ms (+8%) | 35.1 ms (+22%) |
|
||
|
||
Per call that is about **0.1 ns** for the frame and **0.3 ns** for the frame and the slot table together; per sweep of
|
||
the grid, 5 µs and 10 µs, which is 0.03% and 0.06% of a 16.6 ms frame at 60fps. The dev loop's premise is that
|
||
redefinition does not stutter a running game, and a sixteenth of a percent of a frame does not. A dev build is already
|
||
deliberately slower than a release one — every call goes through a cell, every index is checked, no `defconst` folds —
|
||
and this joins that list rather than starting a new one.
|
||
|
||
The chain's *shape* was chosen by measurement and not before it. An array with a stack pointer — no alloca, no address
|
||
escaping — was built and timed as the obvious alternative and is worse on both benchmarks: the frame record on the
|
||
calling function's own stack is already hot, and an indexed store into a megabyte of BSS is not. It also has a fixed
|
||
depth, which a chain of stack records does not.
|
||
|
||
### Locals of a stopped frame
|
||
|
||
The half the shadow stack was built for. `(:op "locals" :frame N)` answers what a stopped frame's named locals hold.
|
||
|
||
```
|
||
(:op "locals" :frame 0) → (:status "ok" :frame "look"
|
||
:locals (("n" "i64" "3") ("label" "string" "\"hello\"")
|
||
("p" "Point" "(Point {:x 1.5 :y 2.5})")
|
||
("xs" "[3 i32]" "[ 10 20 30]") ("flag" "bool" "true"))
|
||
:refused (("after" "not bound yet at the point the program stopped")))
|
||
```
|
||
|
||
**Nothing is copied out of the program, and nothing could be.** A Flan value carries no header, so bytes read from
|
||
another process would be bytes with no meaning. What the daemon has instead is the *type* — `Tast.fn.slots`, from the
|
||
build it owns — and the name beside it in `snames`, which was already there for the DWARF work. So it compiles a thunk
|
||
that renders those types **at those addresses, in the program**, on the stopped thread, and reads the text back through
|
||
the same seqlocked result buffer `C-x C-e` uses. The only fact that comes from the running program is where the frame
|
||
is. That is `render.ml`'s existing walk with its root changed: `Render.render` over `(Deref (Ptr T) (flan/dev-slot f
|
||
i))` instead of over an expression — the **pointer-rooted render thunk** NEXT.md said locals needed, and it turned out
|
||
to need one new arm in the whole backend (a pointer-to-pointer cast, which under opaque pointers emits nothing).
|
||
|
||
**A slot's entry is its address, and null until it is bound.** That is the whole of the liveness answer: there is no
|
||
analysis, no PC-to-scope map and no bitmap. The store that binds a slot stores the address, so a slot the program has
|
||
not reached yet reads as null and is refused by name. Without it, `(let [after 99] …)` sitting past an `error` would
|
||
render whatever the stack held, and a slice or a struct of garbage does not misprint — it faults, on the game thread of
|
||
a program that is already stopped, which is the worst moment this project has to offer. The daemon asks the program
|
||
which slots are bound *before* it builds the thunk, so the set it emits code for is the set the thunk will resolve.
|
||
|
||
**Only named slots are recorded**, and this is where most of the cost went. A slot whose address is stored anywhere
|
||
escapes, and an escaped alloca is one `mem2reg` cannot promote — so recording a slot is paying for it in every call to
|
||
that function, for ever. The slots that would hurt most are exactly the ones with nothing to show: `dotimes`'s hidden
|
||
bound, the temporaries `(min)` and `(max)` evaluate their operands into, the render walk's own scratch. They keep their
|
||
promotion and are refused by name (`s4`, "a slot the compiler made up") rather than shown under an invented one.
|
||
Recording every slot instead was built and timed and came out inside the noise on both benchmarks, so the rule stands
|
||
on what it shows rather than on what it saves.
|
||
|
||
**Shadowing is right here, and that is not an accident of this design — it is the thing the DWARF route still owes.**
|
||
`check.ml`'s `fresh_slot` only ever allocates, so `(let [v 22] …)` inside `(let [v 11] …)` is two slots, both named
|
||
`v`, and both appear with their own values. lldb answers `p v` with 11 in that program and will until a
|
||
`!DILexicalBlock` per `Let` exists.
|
||
|
||
**Four refusals, each by name and with its reason.** Three per slot — invented, not yet bound, and no printer for the
|
||
type (a map, a function value, a type variable; the arm exists, no program the checker accepts has reached it yet, so
|
||
it is written and untested) — and two whole frames: one belonging to a `C-x C-e` thunk, whose `Tast` the session does
|
||
not keep, and one whose *body* is not the body this session holds.
|
||
|
||
**The superseded frame, and why a count could not find it.** Installing while stopped is deliberately allowed — it is
|
||
the fix-it-and-retry loop — so the frame on the stack and the body the session holds can be two bodies of one function.
|
||
A slot count catches a body that gained or lost a binding and nothing else; the case that matters is a **rename**, which
|
||
changes neither the count nor the types, and which would otherwise show every *new* name against the *old* body's
|
||
storage with nothing saying so. That is the confident wrong answer, in the one place someone is working out what went
|
||
wrong.
|
||
|
||
So each `%fninfo` carries a **slot fingerprint**: `Emit.slot_fingerprint` over every slot's name together with the
|
||
spelling of its type, since either can change on its own. The frame carries the value for the body it was compiled from,
|
||
`flan_dev_frame_slotsig` reads it, the agent snapshots it with the rest of the frame and puts it on the backtrace line,
|
||
and `Dev.locals` recomputes it from the body it holds and compares. Different means refused by name — *this frame's body
|
||
was redefined since it was entered, so its names no longer describe its values* — rather than answered. The count check
|
||
stays in front of it because its message is the more specific one.
|
||
|
||
Computed in `emit.ml` and read from there by `dev.ml`, so there is one definition of it and the two ends cannot drift.
|
||
It stays **off the wire**: `(:op "backtrace")` still answers four fields per frame, because a hash is not something an
|
||
editor can act on and the refusal says the fact in words instead. The same is true of the globals fingerprint that
|
||
now travels beside it. The bound is worth stating: it is a 30-bit hash, so a
|
||
collision is possible in principle, and it would reproduce exactly the silent wrong answer this catches — but only
|
||
between two *differing* bodies of a function whose qualified name has already matched, since `find_fn` is what gates the
|
||
comparison at all.
|
||
|
||
The first build of this shipped the fingerprint into `%fninfo` and stopped there — no accessor, no wire field, no
|
||
comparison — and was left in the tree with its own test red. Four of five hand-offs missing looks exactly like one
|
||
hand-off dropping a number, which is what the note left behind said it was.
|
||
|
||
**A `(Vec T)` shows as `<vec>` and a `(Ptr T)` as `<ptr>`**, because that is what `render.ml` already does for them
|
||
everywhere else: following a pointer a REPL was handed is not a safe thing to do on someone's behalf, and walking a
|
||
`Vec` structurally is a walk over storage the frame does not own — `(print (as-slice v))` is how that is asked for,
|
||
and it says at the call site that it borrowed.
|
||
|
||
Each slot is rendered **from its address** rather than copied into the thunk first. A copy would be one `alloca` the
|
||
size of the slot — 40KB for sand's grid — and the walk only ever shows eight elements of it. The cost is one call to
|
||
`flan/dev-slot` per leaf the walk reaches rather than one per slot, which the depth and span caps already bound.
|
||
|
||
### Globals of a stopped stack
|
||
|
||
The other half, and in this language arguably the more useful one: a game keeps most of its state in top-level
|
||
`defvar`s and `sand.flan` holds its entire grid that way, so "what is this program's state right now" was a question
|
||
with nowhere to ask it.
|
||
|
||
```
|
||
(:op "globals") → (:status "ok"
|
||
:globals (("grid" "[4 i32]" "[ 7 5 0 0]" (0 1))
|
||
("pressure" "i64" "12" (0))
|
||
("label" "string" "\"running\"" (1)))
|
||
:refused () :skipped ())
|
||
```
|
||
|
||
**Not per frame, and that is the design rather than a layout preference.** A global is not part of a frame — it is
|
||
program state the frame happened to touch — so nesting it under one implies an ownership that is not there and repeats
|
||
the name once per frame that reads it. So: one section, whose contents are the **union of the globals every frame on
|
||
the current stack references**.
|
||
|
||
**The compiler does the choosing.** `Reach.expr_refs` is the walk that already computes what a function refers to — it
|
||
is how the link drops a package nothing calls — and pointed at one body it answers that body's reference set. Listing
|
||
*all* of a program's globals instead would bury the one that matters under the prelude's PRNG state; taking only what
|
||
the stack reaches is a filter the compiler can apply and a person cannot. Direct references only, with no transitive
|
||
closure through calls: a callee that reads a global is either on this stack, contributing its own references already,
|
||
or it is not on it and is not part of where the program stopped.
|
||
|
||
**Each entry carries the frames that touch it, by index**, which is what the conditions buffer already numbers frames
|
||
by. That recovers what per-frame nesting would have told you — "the whole chain is reading this" reads differently from
|
||
"only the innermost does" — at no cost in duplication. **Ordered by the innermost frame that touches it**, because a
|
||
deep stack makes the union large and proximity to the error is what puts the likely culprit on top; ties keep
|
||
declaration order.
|
||
|
||
**The mechanism is one step simpler than `locals`.** A local is reached by *address*, and only the stopped program
|
||
knows where its frame is — hence `flan/dev-slot`, hence the `bound_slots` round trip, hence the null-until-bound
|
||
refusal. A global is reached by *name*: `Emit.redefinition` writes a global the host already has as `external`, so the
|
||
dlopened thunk binds to the program's own storage and the dynamic linker does the work. Nothing is asked of the stopped
|
||
thread at all, and there is no not-yet-bound case, because a global's storage exists from the moment the process
|
||
started. It is `Render.render` over `(Global n)` — the same root `C-x C-e` uses when you type a global's name at it.
|
||
|
||
**A frame that cannot be attributed contributes nothing and says so.** An `eval` frame has no declaration in this
|
||
session; a lifted handler clause has none of its own; a frame whose body was redefined since it was entered holds a
|
||
body whose reference set is a claim about different code. All three go into `:skipped` with the reason, because "the
|
||
union is incomplete and here is why" and "these are all of them" are different answers and the second one is the lie.
|
||
|
||
**The hole this had, and the second fingerprint that closes it.** `Emit.slot_fingerprint` hashes a body's *slots*. For
|
||
`locals` that is exactly the right cut: identical slots means the names still describe the storage, so the answer is
|
||
still true. Here it was not, because a body can change which globals it names without touching a single slot — and then
|
||
this section showed the *new* body's reference set attributed to the *old* frame. The values were never at risk; they
|
||
are read from the program's storage by name. What was, is one frame's membership in the union and the frame numbers
|
||
beside an entry.
|
||
|
||
So `%fninfo` carries a second number beside the slot fingerprint: `Reach.ref_fingerprint`, over the **set** of globals
|
||
the body names — sorted and deduplicated, because a reference set is not ordered and a body that mentions the same two
|
||
globals the other way round is the same body, where slot indices make the slot fingerprint order-sensitive on purpose.
|
||
It travels the same path the first one does: `flan_dev_frame_refsig`, the agent's snapshot, the backtrace line, and
|
||
`Dev.globals_op` recomputing it from the body it holds. Different means that frame is in `:skipped` with its own
|
||
reason — *this frame's body names different globals than the one this session holds* — and the rest of the stack still
|
||
contributes.
|
||
|
||
**Two numbers and not one combined**, which is the whole reason this is a second fingerprint rather than a wider first
|
||
one. They are different facts: a frame whose slots match and whose globals do not has locals that are perfectly
|
||
readable and attribution that is not, and one hash over both would make `locals` refuse a frame with nothing wrong
|
||
with it. So `locals` still checks the slot fingerprint alone.
|
||
|
||
It lives in `reach.ml` rather than `emit.ml` because `Reach.expr_refs` is already the walk that answers "what does this
|
||
body refer to" and is the same walk the union above is built from — the two ends cannot disagree about what counts as a
|
||
reference. Which names are globals is the caller's to say: the emitter knows the program's globals, and so does the
|
||
session. `test_dev.ml` now drives the exact case — a redefinition that binds identical locals and names `untouched`
|
||
where the stopped frame's body names `pressure` — and with the check disabled it fails twice: once on the missing
|
||
refusal, once on `untouched` appearing in the union under frame 0.
|
||
|
||
### Conditions — step 2: `restart-case` and `invoke-restart`
|
||
|
||
`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
|
||
`restart-case` further out.
|
||
|
||
```
|
||
(defn fetch [n i32] i32
|
||
(restart-case (middle n) ; its value if nothing transfers
|
||
(use-placeholder [] -1)
|
||
(retry [] 7)))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
|
||
(fetch 2)) ; -1
|
||
```
|
||
|
||
**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an
|
||
`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the
|
||
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
|
||
slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers.
|
||
`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather
|
||
than a hunt.
|
||
|
||
**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare
|
||
pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and
|
||
plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight
|
||
through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform
|
||
also means redefinition acquires no new refusal class.
|
||
|
||
**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous
|
||
note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only
|
||
probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The
|
||
frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is
|
||
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
|
||
Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's
|
||
"innermost offering the name" is just the order of the walk.
|
||
|
||
**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever
|
||
frames it established and either catches the transfer or forwards it outward:
|
||
|
||
- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts
|
||
the target back and goes on out;
|
||
- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes,
|
||
and without it the handler stack would be left pointing into a frame that has gone;
|
||
- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is
|
||
still refused by name.
|
||
|
||
A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would
|
||
jump straight past the very form that was supposed to catch it.
|
||
|
||
**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is
|
||
guarded; with the channel still set the first of them would branch straight back into the landing block it came from.
|
||
Same reason the clause body starts with it null.
|
||
|
||
`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
|
||
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
|
||
cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and
|
||
there are no function values yet, so nothing can call *back* into Flan across one.
|
||
|
||
Scope, each piece refused by name with its reason and a test on the reason:
|
||
|
||
- **restarts took no parameters.** That covered §1's own `load-texture` example and skipped argument marshalling and
|
||
§3's runtime check. Both are in now; see "Conditions — step 3" below.
|
||
- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops.
|
||
- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a
|
||
choice nothing in the source shows.
|
||
- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting
|
||
there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a
|
||
defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see
|
||
it.
|
||
- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past
|
||
everything. There is nowhere to resume, so there is nothing else to do.
|
||
|
||
**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal`
|
||
returns `()` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past
|
||
it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked
|
||
`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case`
|
||
body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled
|
||
case: it cannot be an `outputs` row, because it does not exit 0.
|
||
|
||
`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart`
|
||
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
|
||
|
||
**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a
|
||
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
|
||
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
|
||
nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value.
|
||
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
|
||
|
||
`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca`
|
||
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
|
||
crosses `probe`, which the host was compiled with. Those three do not meet anywhere else.
|
||
|
||
Two things found by writing it:
|
||
|
||
- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates
|
||
the body's slots into a record the function never sees again and the indices collide. It was harmless only because no
|
||
`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now.
|
||
- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not
|
||
fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere
|
||
else entirely.
|
||
|
||
`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant
|
||
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
|
||
of those calls goes through a cell.
|
||
|
||
### Conditions — step 3: restarts take parameters
|
||
|
||
§3's other half, and the half the comparative studies all ask for: `use-value` and `store-value` are the two restarts
|
||
whose answer is not in the program.
|
||
|
||
```
|
||
(restart-case (middle n)
|
||
(use-value [v i32] (* v 2))
|
||
(retry [] 7))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7)) ; 42
|
||
```
|
||
|
||
**The parameters live in a buffer the restart-case owns.** The obvious place is the invoker's frame — it is where the
|
||
values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so
|
||
the invoking side is gone by then. The invoker stores into the *target's* buffer while both are still alive, which is
|
||
the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that
|
||
buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any
|
||
other.
|
||
|
||
**What a restart takes is checked at run time, and it has to be.** §4 finds a restart by name on a dynamic stack: the
|
||
invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker
|
||
to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and
|
||
`invoke-restart` compares both against its own before it stores anything. Every frame carries them, parameterless ones
|
||
included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not
|
||
redundant with the hash: it is what keeps a hash collision between two *different* signatures harmless, since a
|
||
collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well,
|
||
because the message has to say what was wanted and what was given and neither end knows both:
|
||
|
||
```
|
||
restarts.flan:79:36: restart use-value takes (i32), given ()
|
||
restarts.flan:87:36: restart retry takes (), given (i32)
|
||
```
|
||
|
||
**The arguments are evaluated into slots before the invoke node, not hung off it.** Two reasons and both are real. An
|
||
argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside
|
||
an argument has to be on the walk `Reach` and `Load` already do — `InvokeRestart` was a leaf to both, and a leaf that
|
||
grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker.
|
||
`restarts.flan` has exactly that function, `half`, to keep the claim tested.
|
||
|
||
**The break loop can take a restart it cannot fill in, so it is refused.** A transfer has two sources: an
|
||
`invoke-restart`, which writes the arguments first, and the break loop, which aims the channel at a frame by position
|
||
and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them
|
||
apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked
|
||
unfilled, `invoke-restart` marks it filled, and a clause with parameters checks the mark before reading. Choosing
|
||
`use-value` from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the
|
||
top of `NEXT.md`: the answer is a Flan expression, and there is already something that compiles one against the live
|
||
program.
|
||
|
||
**Lookup stayed by name, and the signature is checked against what it found.** `flan_find_restart` matches the name
|
||
hash and nothing else, so an inner `(use-value [s string] ...)` shadows an outer `(use-value [v i32] ...)` and an i32
|
||
is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first
|
||
frame offering the name") and it is the thing a reader will assume works the other way, so `restarts.flan` has a case
|
||
for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments,
|
||
which is overload resolution on a dynamic stack.
|
||
|
||
Both of these guards go through `fail_block` unconditionally, unlike the bounds checks: `--no-bounds-checks` does not
|
||
remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or
|
||
written to a different layout, is memory corruption.
|
||
|
||
`runtime/flan_rt.c` gained two message functions and nothing else. The restart frame's first four fields are the ones C
|
||
declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.
|
||
|
||
§3's other open point, a **report string per clause**, is still open and was not settled first as §3 asks. The field
|
||
and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the
|
||
agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing
|
||
that listing anyway.
|
||
|
||
### What is left
|
||
- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays.
|
||
|
||
**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the
|
||
memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary.
|
||
Attaching to a process someone else built is not a thing to support by default.
|
||
|
||
## Allocators, `(Vec T)` and `StorageExhausted`
|
||
|
||
`spec-memory.md`'s Allocators section is frozen and settles what to build. This is why the built thing is the shape it
|
||
is, and — separately and loudly — the three places it **amends** that section plus the one thing it adds to it.
|
||
|
||
### `Allocator` is a builtin opaque type, so none of milestone 5 was needed
|
||
|
||
The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and
|
||
`check.ml` refuses function values four ways — a written `(Fn ...)` annotation, a written `fn` literal, a `defn`'s name
|
||
in value position, and calling anything other than a named function. Read straight off those lines, containers need
|
||
function values and the work doubles.
|
||
|
||
None of the four is anywhere near this. All four are about *surface syntax*, and a value the compiler builds that no
|
||
surface form names trips none of them:
|
||
|
||
- `Allocator` is a `Types.t` case with no user-writable constructor, the way `string` is a builtin ptr+len. No Flan
|
||
type names its procedure.
|
||
- The procedure is a C symbol the emitter names.
|
||
- `vec-new`, `push`, `free`, `free-all` and the rest are ordinary named calls, which `check_call` already routes
|
||
through `named_call`.
|
||
|
||
The precedent was already in the repo twice: a `handler-bind` clause is lowered to a function called back through
|
||
`h->fn(condition, xfer)` and built as its own `Tast.fn`, never an `Ast.Fn`; and a dev build's `call` loads a pointer
|
||
out of a cell and calls through it, which is the indirect call the surface language refuses.
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: "here is my proc, make an `Allocator` from it" wants a
|
||
`defn`'s name in value position. `make-allocator`, `allocator-from` and `allocator` are refused by name with that
|
||
reason, rather than coming back as unknown functions.
|
||
|
||
**An `Allocator` value is a pointer to the runtime's struct, never a copy of one.** That is forced, not chosen. The
|
||
capability set has to be readable from wherever a container landed, and `free-all` bumps an epoch every container made
|
||
from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never
|
||
fires.
|
||
|
||
### The surface
|
||
|
||
| Name | What |
|
||
|---|---|
|
||
| `(heap-allocator)` | the general-purpose tier: malloc, aligned, with free |
|
||
| `(arena-new bytes)` | a bump arena over one fixed backing buffer |
|
||
| `(arena-destroy a)` | hands the pages back — see the amendment below |
|
||
| `(free-all a)` | releases everything the allocator holds, retain-capacity |
|
||
| `(can-free? a)` / `(can-free-all? a)` | the capability set, read at run time |
|
||
| `(alloc-epoch a)` / `(alloc-id a)` | the counter `free-all` bumps; the allocator's identity |
|
||
| `(alloc-budget a)` / `(set-alloc-budget a n)` | a ceiling on live bytes — an addition, see below |
|
||
| `(alloc-live-blocks a)` | "did you forget to free", answered at the tier that can answer it |
|
||
| `context/allocator` / `context/temp` | the current implicit allocator, and the per-frame arena |
|
||
| `(with-allocator a body...)` | rebinds for a dynamic extent and releases nothing |
|
||
| `(vec-new T)` / `(vec-new T a)` / `(vec-new)` | a Vec, against the context or a named allocator |
|
||
| `(push v x)` / `(reserve v n)` | `()`, both |
|
||
| `(at v i)` / `(len v)` | the array names, extended — not a parallel pair |
|
||
| `(as-slice v)` / `(as-slice v lo hi)` | a non-owning `[T]` view |
|
||
| `(clone v)` / `(clone v a)` | the only copy; assignment moves |
|
||
| `(free v)` | consumes its argument |
|
||
|
||
### Three amendments to a frozen spec, and one addition
|
||
|
||
**1. `free-all` is retain-capacity, and `arena-destroy` is the operation that hands pages back.** The spec's table has
|
||
`free-all` and nothing else. Zig's `ArenaAllocator.reset` takes a `ResetMode` of `free_all` / `retain_capacity` /
|
||
`retain_with_limit`; Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing
|
||
buffer and the call only sets `offset = 0`. For a frame arena reset every frame, retain-capacity is the normal case and
|
||
handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown
|
||
the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being
|
||
the same does not make a container built before the reset valid, which is the whole point of the trap.
|
||
|
||
**2. `context/allocator` and `context/temp` are dynamic variables with save and restore, not extra parameters.** The
|
||
spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the
|
||
FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other
|
||
lane working in `emit.ml`. The dynamic variable is the implementation; the literal reading is deferred and is a
|
||
performance question (a parameter avoids a global load), not a semantic one.
|
||
|
||
**3. The `Vec` header is six words in every build, not four in release.** The spec fixes the release layout at
|
||
`ptr + len + cap + allocator` with the generation word and the epoch dev-only. A layout that changes with a build flag
|
||
is a layout that can disagree *silently* across the reload boundary: a redefinition module is built by `llc` and `ld`
|
||
against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and
|
||
so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag
|
||
before it can land.
|
||
|
||
**The addition: a budget.** `flan_allocator` grew a ceiling on live bytes, 0 for none. The spec's `retry` restart is
|
||
answerable only by a handler that can make the *same* request succeed, and for a fixed backing store the handler that
|
||
works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which
|
||
is exactly what the epoch check catches. The spec names "grows the arena and then invokes `retry`" as the handler that
|
||
works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on
|
||
purpose.
|
||
|
||
### `with-allocator` is its own IR node because of the transfer path
|
||
|
||
Save, run, restore — and *restore again at the pad*. That second restore is the whole reason it is a node rather than a
|
||
`let` and two calls. A body that errors, or one a handler transfers out of, leaves through `current_pad`, and a context
|
||
allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is
|
||
precisely where something is about to allocate in order to render a condition. `test/programs/allocators.flan` asserts
|
||
that path by taking a restart out of a `with-allocator` body.
|
||
|
||
It releases nothing, per the spec: not at the end of a `let`, not at the end of a function, not at the end of the body.
|
||
The program proves it by reading the epoch either side.
|
||
|
||
### `(Vec T)`: one type-erased runtime, and the backend learned almost nothing
|
||
|
||
The element type appears nowhere below the call site. `size_of` and `align_of` are produced where the concrete type is
|
||
known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement
|
||
(`base/runtime/dynamic_array_internal.odin`). The backend grew four prims in total:
|
||
|
||
- `Rt of string` — a call into the runtime's C named by symbol, with argument and result LLVM types read off the
|
||
expression nodes. The container runtime is type-erased and therefore *is* a list of C entry points, so one arm covers
|
||
all of them. A `Vec` argument crosses as its address, which is also what lets an operation mutate the caller's Vec.
|
||
- `SizeOf` / `AlignOf` of a type, filled in from the same layout calculator DWARF uses — the one the acceptance test
|
||
already checks against LLVM's own `getelementptr` answers.
|
||
- `AddrOf` of any expression, place or not, because the element a `push` copies may be computed. The backend already
|
||
spilled a non-place to a temporary for exactly this.
|
||
|
||
`at` and `len` were already the names for a fixed array and a slice, so a Vec extends them rather than adding a
|
||
parallel pair — the asymmetry `nth` was removed for. The value form `(at v i)` and the place form `(set (at v i) x)` go
|
||
through one helper, so they cannot drift apart the way `nth` did.
|
||
|
||
A Vec's length and index are `i32`, like every other length here. Widening indices is one change across every
|
||
container and not a Vec question.
|
||
|
||
`let` has no type annotation — `parse.ml` settles that a triple binding is ambiguous and that types are inferred — so a
|
||
local Vec has nowhere to say what it holds and the element type is written at the call: `(vec-new i32)`. This is *not*
|
||
the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an
|
||
ordinary type rather than binding a type variable. Where the context does say — a `defvar`'s type, a return type, an
|
||
argument — it may be left out.
|
||
|
||
A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is
|
||
Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a
|
||
different reason (below), but the adopt rule is what makes `Zero` of a Vec a usable value rather than a null deref.
|
||
|
||
### Move-only is a dead set, and it is flow-sensitive at a join
|
||
|
||
Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing
|
||
one to a function, binding it, returning it and `free`ing it are all moves and all reach one place, and the handful of
|
||
operations that only look at a container (`at`, `len`, `as-slice`, `push`, `reserve`, `clone`) say so. Only a
|
||
*syntactically simple* target counts as a borrow — in `(len (f v))` the call still moves `v`.
|
||
|
||
At an `if` and at a `match`, every arm is checked from the state before the form and the **union** of what they moved
|
||
survives the join. A flat set is wrong in both directions: it refuses `(if c (free v) (free v))`, which is legal, and it
|
||
accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union
|
||
says.
|
||
|
||
The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move,
|
||
not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a
|
||
move of a binding declared outside the loop is refused, with that as the reason.
|
||
|
||
### What ownership is not transitive through yet, and why each is refused
|
||
|
||
The spec says ownership is structural — a struct containing a Vec is itself move-only, `free` recurses into owning
|
||
fields, and a field cannot be freed on its own. That machinery is the recursive teardown `drop` brings. Until it lands,
|
||
three shapes are refused where they are declared, each naming `drop`:
|
||
|
||
- **a struct field of `Vec` type**, because the struct copies its header on assignment and nothing records a move;
|
||
- **a global of `Vec` type**, because the dead set is per function — two functions each freeing it is a double free
|
||
nothing could see, and a global read does not go through the move path at all, so even the one-function case would be
|
||
accepted. Half a rule is worse than none. A global **`Allocator`** is a different thing and stays legal: an allocator
|
||
is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot
|
||
see the locals of the function that established it;
|
||
- **a `Vec` of a `Vec`**, because the type-erased runtime copies and releases elements bytewise: `clone` would
|
||
duplicate inner headers instead of copying what they own and `free` would drop their buffers.
|
||
|
||
And a `Vec` does not cross to C: handing a header that owns storage to C hands out an owner. `(as-slice v)` as
|
||
`(Ptr T)` plus `(len v)` is the shape that does cross, and the refusal says so.
|
||
|
||
### `StorageExhausted` went in *with* `Vec`, not after it
|
||
|
||
No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the
|
||
operation signals `(StorageExhausted {.bytes n .align a .allocator id})` with `error` — whose type is `Never` — inside a
|
||
`restart-case` offering `retry`. One rule over every allocating operation, which is what keeps `push` and `reserve` at
|
||
`()`, `clone` at the container, and no signature anywhere growing a `Result`.
|
||
|
||
It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every
|
||
allocating operation, which is the point of having decided it first. Odin's `append` returns an ignorable
|
||
`Allocator_Error`, and its type-erased path returns the old length on a failed reserve; an append that appends nothing
|
||
and says nothing is the outcome this rule exists to make impossible.
|
||
|
||
The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:
|
||
|
||
```
|
||
(let [ok false]
|
||
(while (not ok)
|
||
(restart-case
|
||
(do (set ok ATTEMPT)
|
||
(if (not ok) (error (StorageExhausted {...}))))
|
||
(retry []))))
|
||
```
|
||
|
||
A handler that frees something, releases a scratch region or raises the ceiling and then invokes `retry` lands in the
|
||
clause, the clause falls through, and the `while` re-attempts the **same** request. Every argument to the attempt is
|
||
bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a
|
||
`push` was given. With nothing handling it, `error` stops the program on the frame that erred.
|
||
|
||
This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the
|
||
failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site
|
||
can. The condition is a value struct with fixed numeric fields and **no rendered message**, because formatting would
|
||
allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.
|
||
|
||
### The epoch trap, which is the shipping answer to an open question
|
||
|
||
`spec-memory.md` leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which
|
||
allocator a construction used, and `with-allocator` plus `context/allocator` are exactly the mechanisms that deny that
|
||
knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, `free-all` bumps
|
||
that counter, and any operation on a container whose recorded epoch has moved traps naming the site.
|
||
`test/programs/stale-region.flan` is the case, and the point of it is that `v` is still in scope, still looks fine, and
|
||
nothing marked it — which is precisely what a static rule cannot see.
|
||
|
||
**The generation word has no reader.** It is bumped on every reallocation, as specified, and the stale-slice trap it
|
||
exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said
|
||
plainly here rather than implied by the word's presence in the header.
|
||
|
||
It is not "not yet", either, and the runtime's own comment now says so. A reader for that word is a third word on every
|
||
slice in the language — a layout `spec-memory.md` fixes — so implementing the trap is a spec amendment and an ABI
|
||
change, not a runtime patch. The two live options are that amendment, or dropping the word from the header and from the
|
||
spec together; neither is a cleanup, and until one is taken the word is carried and trusted by nothing.
|
||
|
||
### What this leaves for steps 5 to 7
|
||
|
||
`(Map K V)` is built — see below. What is left: `drop` and with it the transitive move-only rule, recursive teardown,
|
||
and the refusal to construct a drop-carrying container against an allocator without `can-free`; `(Result T E)` and
|
||
`try`; generics; the macro expander.
|
||
And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy:
|
||
capture does not exist at all, and the spec's captured-`Vec`-by-pointer rule has never had to exist because every
|
||
capturable type today is a value type. It is its own item and should be planned as one.
|
||
|
||
## `(Map K V)`, which is Odin's map
|
||
|
||
Step 4 of the container build order, over the runtime `Vec` already established. `spec-memory.md` is followed as
|
||
written; the two places this departs from *Odin* are named below, and there is one amendment to the spec and one
|
||
restriction under it.
|
||
|
||
### The three properties that were the point
|
||
|
||
Odin's header states them and they are why it was the thing to follow (`base/runtime/dynamic_map_internal.odin`).
|
||
|
||
- **Open-addressed Robin Hood hashing at a 75% load factor.** No buckets and no per-entry allocation: one block holds
|
||
every key, every value and every hash. Robin Hood is the part that earns its keep — on insert, an element that is
|
||
further from the slot its hash wanted than the occupant it is looking at takes the slot and sends the occupant on.
|
||
Probe distances even out, and a lookup may stop the moment it is further from home than the occupant it is looking
|
||
at, because no element is ever further from home than one it passed. That early exit is why a miss costs about what
|
||
a hit does.
|
||
- **Cache-line cell packing.** A flat `[capacity]K` array lets one key straddle two cache lines, so a probe walking
|
||
four slots can touch five lines. A cell packs as many `K` as fit in 64 bytes and pads the rest, so no key ever
|
||
straddles a line. Keys, values and hashes are three separate runs, so a probe — which reads hashes, and only then
|
||
one key — touches hash lines and nothing else until it has a candidate.
|
||
- **Pointer-width integers throughout**, so no sign extension or masking gets into the probe loop.
|
||
|
||
### Two departures from Odin, both deliberate
|
||
|
||
**There are still no tombstones, now that removal exists.** A slot is empty or occupied and nothing else, and
|
||
`flan_map_remove` keeps it that way by shifting the run back over the hole rather than marking it. Odin does the
|
||
opposite, and this paragraph used to say otherwise: read out of
|
||
`base/runtime/dynamic_map_internal.odin`, `map_erase_dynamic` sets a tombstone bit and leaves the repair to the next
|
||
insert, which is why Odin's *insert* carries a backward-shift loop and its every lookup tests for a tombstone. The
|
||
trade is the usual one — erase is O(1) there and the shift is here, and the lookups, which outnumber the removals,
|
||
pay nothing. What `spec-memory.md` still defers is the rest of its sentence: move-aware lookup and owned entries. A
|
||
removed value is copied out, and nothing is dropped.
|
||
|
||
**The header does not tag the capacity into the data pointer.** Odin stuffs `log2cap` into the low six bits because
|
||
its `Raw_Map` must be three words. This header already carries an allocator, a generation and an epoch, so the tagging
|
||
would buy nothing, cost a mask on every access, and — the part that actually matters — make correctness depend on the
|
||
block being 64-byte aligned. Alignment is *requested*; an arena whose base is not cache-aligned now gives a slower map
|
||
rather than a wrong one.
|
||
|
||
The header is six words, 48 bytes, the same as a `Vec`'s and for the same reason: a layout that changes with a build
|
||
flag can disagree across the reload boundary.
|
||
|
||
data len log2cap allocator gen epoch
|
||
|
||
### The hash and equality pair, and why most key types do not get one
|
||
|
||
`spec-memory.md` restricts keys to built-in structural types and makes hashing and equality compiler-provided
|
||
structural operations. So there is no dispatch to design: a key type resolves to a pair of symbols, passed to the
|
||
type-erased runtime the way Odin hangs two contextless `proc`s off a `Map_Info`.
|
||
|
||
Most key types need nothing emitted. A key whose equality is bytewise and whose bytes are all present — every integer,
|
||
enum, bool, and fixed array of those — is served by one runtime pair over `(pointer, size)`. Two kinds are not, and
|
||
the reasons are worth keeping because they are what a bytewise shortcut would have got wrong:
|
||
|
||
- a **string** is ptr+len and its bytes are elsewhere, so two equal strings at different addresses must hash alike;
|
||
- a **struct** may have padding, whose bytes are indeterminate — two structs equal field by field can differ bytewise
|
||
— and it may hold a string, which brings the first problem inside it.
|
||
|
||
A struct therefore gets a pair emitted for it, walking its fields in declaration order and addressing nothing but
|
||
fields. Two maps with the same key type share one pair, and a struct reached twice through two fields emits one.
|
||
|
||
`Tast.FnAddr` is what carries the symbol, and it is **not** a function value: nothing in the surface language can
|
||
produce one, name its type, or call through it. It is the same escape the allocator used — the one NEXT.md predicted
|
||
would work — and `reach.ml` learned the edge, because a function reached only by address is invisible to the
|
||
reachability walk otherwise. That was already true of handler clauses.
|
||
|
||
A pair emitted for a struct is an ordinary Flan function, so its signature ends with the transfer channel like every
|
||
other. The runtime's typedef spells that pointer out rather than hoping nothing writes through it, and each built-in
|
||
hasher exists in two spellings — the pointer form that matches the typedef, and the direct form an emitted hasher
|
||
calls per field, which has no channel to hand on.
|
||
|
||
### The surface
|
||
|
||
| Name | What |
|
||
|---|---|
|
||
| `(map-new)` `(map-new K V)` `(map-new a)` `(map-new K V a)` | a new map; the pair may be omitted where the context says |
|
||
| `(put m k v)` | upsert, `()` |
|
||
| `(get m k)` | `(Option V)` — absence is `None` |
|
||
| `(has-key? m k)` | `bool`, copying no value — **an addition; the spec does not name it** |
|
||
| `(map-remove! m k)` | `(Option V)` — the value that was there, or `None` |
|
||
| `(len m)` `(reserve m n)` `(clone m)` `(clone m a)` `(free m)` | extended, not duplicated |
|
||
|
||
`has-key?` is **not in `spec-memory.md`** and is an addition, flagged because everything else here is the spec's.
|
||
`(get m k)` answers the same question, but through an `Option` the caller then has to match, and the common use is a
|
||
condition. It copies no value, which is also why it is not just `get` with the result thrown away. The `?` suffix
|
||
follows `can-free?`.
|
||
|
||
`len`, `reserve`, `clone` and `free` were **extended rather than given map-shaped names of their own**, which is what
|
||
`at` and `len` already did for `Vec`: one question, one word. `reserve`'s `n` is entries, not slots — the runtime sizes
|
||
the block so `n` still sits under the load factor, which is the only reading of "room for n" that does not reallocate
|
||
on the nth put.
|
||
|
||
`get` builds the `Option` in the checker, not the runtime, which has no idea what an `Option`'s layout is; keeping it
|
||
that way is what lets one entry point serve every value type. `put` and `get` bind their arguments to slots before the
|
||
allocation guard, so a `retry` re-attempts the allocation and not the expressions that produced the key and value.
|
||
|
||
**`{K V}` is the type spelling and there is no map literal.** A bare map form in expression position is a struct
|
||
literal's field list, and giving the same braces two meanings is what the colon-to-dot change was for. A map is built
|
||
with `map-new` and filled with `put`.
|
||
|
||
*(Superseded on the first half only. `(Map K V)` is the type spelling and `{K V}` was withdrawn — braces in type
|
||
position are refused by name, because the brace's value and type meanings never corresponded the way the bracket's do
|
||
and `{}` in type position is wanted for anonymous struct types. The paragraph's actual subject is unchanged: there is
|
||
still no map literal, and a bare map form in expression position is still a struct literal's field list. See
|
||
`lib/parse.ml:68`.)*
|
||
|
||
### The refusals, each by name
|
||
|
||
- A **float key** — not a milestone question, which is why it is said separately. NaN is not equal to itself, and
|
||
`0.0` and `-0.0` are equal while differing bytewise. There is no equality there for a map to hash.
|
||
- A **`Ptr`, slice, `Vec` or `Map` key** — would hash an address rather than what it points at, which is a different
|
||
operation.
|
||
- A **fixed array whose elements are not compared bytewise** — an array of structs or of strings needs the
|
||
per-element walk a struct key gets, driven by a loop rather than a field list. Nothing has wanted one, so it is
|
||
refused rather than written untested, and the shape that does work (a struct holding the array) is named beside it.
|
||
**This is narrower than the spec**, which lists fixed arrays without qualification.
|
||
- A **move-only value** — the refusal `(Vec (Vec T))` already carries, for the identical reason: the runtime copies
|
||
entries bytewise, so `clone` would duplicate headers and `free` would leak what they own. Owned entries arrive with
|
||
`drop`.
|
||
- **`()` as a value** — there is nothing to store, and the cell geometry divides a cache line by the element size.
|
||
Named rather than left to divide by zero, because it is the natural spelling of a set.
|
||
|
||
`StorageExhausted` under `retry` holds over `map-new`, `put`, `reserve` and `clone`, reusing the machinery that landed
|
||
with `Vec`; no operation returns an error. A map is the harder of the two containers for that rule, and
|
||
`map-exhausted.flan` is why it gets its own program: a `Vec`'s failing allocation leaves the `Vec` untouched, whereas a
|
||
map's growth allocates a new block, rehashes into it, and only then releases the old one — so a failure partway must
|
||
leave the map exactly as it was, or the retry re-attempts against a half-moved map.
|
||
|
||
The epoch trap covers the map too, and `map-stale-region.flan` is separate from `stale-region.flan` because the two
|
||
reach the check by different routes: a `Vec`'s operations check on the way in and stop there, while a map's `get` goes
|
||
on to call a hash and an equality function through pointers into the block. A missing check there is not a wrong
|
||
number, it is a probe loop walking released memory.
|
||
|
||
### Is this another Python dict? — measured
|
||
|
||
The question was asked directly and the answer is that **Python's algorithm is fine**. What makes CPython's dict slow
|
||
is that every key and every value is a separately allocated, reference-counted object and hashing goes through
|
||
`__hash__` and `__eq__` calls that cannot be inlined. This map stores raw bytes in the block and compiles hashing and
|
||
comparison concretely per key type. That is most of the gap before any cleverness.
|
||
|
||
Measured on this machine, `i64` to `i64`, against CPython 3.13's dict on the same workload:
|
||
|
||
| Working set | Flan | CPython dict |
|
||
|---|---|---|
|
||
| 10k entries, 10M lookups (cache-resident) | 21 ns/lookup | 132 ns/lookup |
|
||
| 1M entries, 10M puts + 10M lookups | 1.41 s | 1.16 s |
|
||
|
||
**Six times quicker cache-resident, and slower at a million entries** — and the second row is written down rather than
|
||
left out. Both are waiting on memory there, and this layout waits longer: keys, values and hashes are three separate
|
||
runs, so a lookup that misses everything takes three cache misses where a compact dict takes two, and the hash run is a
|
||
full eight bytes a slot. Cell packing buys probe locality, which is a win while the hash run is resident and a loss
|
||
once nothing is. One byte of metadata a slot — the Swiss-table arrangement — is the known answer and is not built.
|
||
|
||
The path from 35 ns to 18 ns (the cache-resident floor, at 500 entries) was **profiled, and the first two guesses were
|
||
both wrong**: the per-slot cell division and the block-size divisions were each replaced first and neither moved the
|
||
number. What did: FNV one byte at a time became eight, and a key that is one machine word became one load and one mix
|
||
with no loop; equality on eight bytes stopped being a call into libc's vectorised `memcmp`, and copying a value out
|
||
stopped being a call into `memmove`; the block geometry stopped being recomputed five times over in a function that ran
|
||
twice per lookup; and the seed stopped being a five-multiply avalanche on the critical path for mixing the hasher does
|
||
again immediately after. `64/size` is a table, which is Odin's `Map_Cell_Info` by another route — Odin precomputes it
|
||
per type because the probe loop must not divide, and here the sizes arrive as ordinary arguments.
|
||
|
||
What remains at 18 ns is the type erasure itself: a non-inlinable call into the runtime and two non-inlinable indirect
|
||
calls to the pair. That is the trade `spec-memory.md` chose deliberately — "It is type-erased on purpose… No generics
|
||
are involved, and none are needed" — and monomorphisation is what would buy it back, at the cost the spec declined.
|
||
|
||
## Unions, and the tag they carry
|
||
|
||
`defdata` parsed and its shape was checked long before this; naming the type (`check.ml:312`) and constructing a
|
||
value (`:1075`) were both refused as milestone 6. They are not any more.
|
||
|
||
### It is closer than the milestone number suggested, and the reason is `Option`
|
||
|
||
`Tast.arm` already carried `acase` (a case name) and `binds` (the slots a payload binds to). That is union shape,
|
||
built and exercised, because **`Option` is a two-case union wearing a special coat** and `match` over it worked
|
||
already. So `check_match` grew a second *subject* rather than a second path: one function decides which case each arm
|
||
names and what type each name it binds has, and everything after it — the dead-set join across alternatives, the
|
||
result type, the exhaustiveness check — is the code that was there.
|
||
|
||
`Option` was **not** desugared into a declared union, and that is deliberate: `Option` is generic and no declared
|
||
union is. The coat is the part that cannot be taken off until generics exist.
|
||
|
||
A union is `Types.Named` exactly as a struct is. One case in `Types.t` covers both, and *which table the name is in*
|
||
is the only thing that tells them apart. That is what let a union be a field, a parameter, a return type, a slot and a
|
||
copy without a single one of those paths learning that unions exist.
|
||
|
||
### The layout, which is the part that has to be exactly right
|
||
|
||
```
|
||
%"U" = type { i32, [k x iA] }
|
||
%"U.Case" = type { the case's fields, in declaration order }
|
||
```
|
||
|
||
A tag, then room for the largest case, with `A` the alignment the *widest member of any case* needs and `k` the size
|
||
rounded up to it. Writing the payload as an array of `iA` rather than of `i8` is what makes LLVM align it without an
|
||
explicit `align` on a type — and it is what makes the whole thing
|
||
|
||
```c
|
||
struct { int tag; union { ... } u; }
|
||
```
|
||
|
||
byte for byte. **That agreement is the point.** A macro is `[Form] -> Form`, so `Form` has to be the same bytes in
|
||
the compiler and in the `dlopen`ed macro, and nothing at run time would notice a disagreement.
|
||
|
||
The tag is an `i32` and not an `i8`. With an 8-byte payload alignment the two cost the same, and `i32` is what a C
|
||
`enum` field is — the spelling the macro lane will have to write by hand on the other side of the boundary.
|
||
|
||
Both the union and one struct per case are emitted as *named* LLVM types, so every reader geps rather than computing
|
||
byte offsets of its own. Construction is a `store` of the tag and a `store` of the case struct through a gep into the
|
||
payload; `match` is a `load` of the tag and a gep the other way. There is one function that knows how a payload is
|
||
read — `case_field_addr` — because there are two readers: a `match` arm's binds and the structural printer.
|
||
|
||
The layout goes through the same oracle the DWARF section uses: LLVM's own answer for the emitted type, read back as a
|
||
constant-folded `ptrtoint`. Two unions are checked, one whose widest case is a pair of `f64` and one whose cases are
|
||
all `i32`, so the payload size and alignment cannot be constants the test agreed with by accident. The `Shape` in
|
||
`test/programs/unions.flan` was checked against clang's answer for the same declaration in C: 32 bytes aligned 8 with
|
||
the payload at offset 8, and 40/8 for a struct holding one.
|
||
|
||
### The surface
|
||
|
||
```clojure
|
||
(defdata Shape
|
||
[Empty
|
||
(Dot [x f64 y f64])
|
||
(Rect [w i32 h i32])
|
||
(Tag [name string n u8])])
|
||
|
||
(Shape.Rect {.w 3 .h 3}) ; a value: the type and the case, then the fields
|
||
(Shape.Rect {.w 3}) ; ZII, exactly as in a struct literal — .h is 0
|
||
Shape.Empty ; a case with no fields is a whole value, not a call
|
||
|
||
(match s
|
||
Empty "empty"
|
||
(Dot x y) "a dot" ; positional, in declaration order
|
||
(Rect w h) "a rect"
|
||
(Tag name n) name)
|
||
```
|
||
|
||
**Construction is qualified and a pattern is bare.** The two are not inconsistent: at a constructor nothing says which
|
||
union is meant, and at a pattern the scrutinee's type already does. The qualified spelling is accepted in a pattern
|
||
too, since that is how the value was written and writing it again should not be an error. Two unions may therefore
|
||
share a case name, and that is not refused — refusing it would be a restriction with no mechanism behind it.
|
||
|
||
Construction needed **no change to `parse.ml`**. `.` is a symbol constituent, so `Shape.Rect` reads as one name; the
|
||
struct-literal arm fires on a symbol followed by a map; and the field-access arm needs a *leading* dot, so the two
|
||
cannot collide. `(Name {.field value})` is one syntax for both, and which it is, is decided against the tables.
|
||
|
||
### Tags are declaration order, so case order is part of the contract
|
||
|
||
Tag *n* is the *n*th declared case, from zero. The consequence is that an all-bytes-zero union is **the first declared
|
||
case with a zeroed payload** — which is exactly the rule that makes an `Option`'s zero a `None`, and it is what makes
|
||
`(defvar u U)` and an omitted union-typed struct field mean something rather than nothing. Reordering a union's cases
|
||
is a layout change, the same way reordering a struct's fields is.
|
||
|
||
### What is refused, and why each
|
||
|
||
- **A non-exhaustive `match`.** Refused, not defaulted, and the message names the cases with no arm. A match that fell
|
||
through would have to produce a value of the match's type out of nothing, and the case a union grows tomorrow is
|
||
exactly the one a reader wants to be told about today. `_` is how to say "the rest", written where it can be seen.
|
||
- **A case pattern binding some of a case's fields.** All of them or none, positionally — binding a prefix reads the
|
||
wrong field the moment one is inserted above it.
|
||
- **Two arms for one case**, which is a mistake and never an intent.
|
||
- **A union with no cases.** No value of it can exist, so a parameter of that type is a function nothing can call.
|
||
- **A case field that is move-only**, in the same words a struct field already gets and for the same reason:
|
||
recursive teardown arrives with `drop`.
|
||
- **A union as a map key.** The payload past the case in hand is indeterminate, so hashing the blob would make two
|
||
equal values hash differently. Hashing one properly is a per-case walk driven by a switch — a different shape from
|
||
the field list `struct_key_pair` emits, and nothing has wanted it.
|
||
- **`uninit` on a union.** This is the one refusal that is *not* the struct rule. Everywhere else `uninit` is an
|
||
opt-out from ZII and the bytes are whatever they were: a garbage `f64` is a garbage number. A union's tag steers
|
||
control flow — a tag no case names falls past every comparison in a `match` into a block LLVM is entitled to assume
|
||
cannot be reached. That is the one place where garbage becomes "the optimiser may do anything".
|
||
- **`(.x u)`.** A union's fields belong to a case, and which case is being held is what the tag says. `match` is how
|
||
one is opened, and its arms bind the fields of the case they matched.
|
||
- **A global initialised with a case.** `(defvar g U (U.B {.x 1}))` would mean serialising the fields into the payload
|
||
blob at link time, which is a byte-level encoder this compiler does not have and which could not express a `string`
|
||
field at all — that is a pointer the linker has to relocate and a byte array has nowhere to put a relocation. A
|
||
*zeroed* global is fine and needs none of it: it is the first declared case.
|
||
|
||
### Recursion, and the shape `Form` will have
|
||
|
||
A union that contains itself by value has no finite size, and `payload_lay`
|
||
would recurse forever laying one out rather than failing. It does not get the
|
||
chance: `check_finite` already walked a union's cases, so `(defdata T [Leaf
|
||
(Node [l T r T])])` is refused with *"T contains itself by value, so it has no
|
||
size — go through (Ptr T)"*, and so is a pair of unions that contain each
|
||
other. Through a pointer it works, and that is the shape a `Form` has:
|
||
|
||
```clojure
|
||
(defdata Tree [Leaf (Node [l (Ptr Tree) n i32])])
|
||
|
||
(defn depth [t (Ptr Tree)] i32
|
||
(match (deref t)
|
||
Leaf 0
|
||
(Node l n) (+ n (depth l))))
|
||
```
|
||
|
||
### Why `match`'s fall-through is still `unreachable`
|
||
|
||
The block after the last case comparison is `unreachable`, kept from the
|
||
`Option` path. That is only sound if no reachable program can hold a tag no
|
||
case names — and none can: `Zero` is tag 0, which is a real case; every
|
||
construction writes a tag the checker resolved; and `uninit`, the one way to
|
||
get bytes nobody wrote, is refused on a union for exactly this reason. The
|
||
refusal is what pays for the `unreachable`.
|
||
|
||
### The diagnostics bug, fixed
|
||
|
||
`(A {.x 1})` on a case of a union said **"unknown struct A"**, because `env` had no table of case names and could not
|
||
tell a case from a misspelling. It has one now, keyed both by the full spelling `U.C` and by the bare `C`. The full
|
||
spelling is a *key* rather than something split out of a dotted name at the use site, because a union's own name can
|
||
contain a slash — an imported `rl/U` — and string surgery would own an edge this does not have to.
|
||
|
||
### Printing
|
||
|
||
`render.ml` walks a concrete type for `print`, the REPL inspector and the break buffer's locals, and a union fell
|
||
through its `Named` arm to `<Shape>`. It recovers the case from the tag by a chain of comparisons — the same shape the
|
||
enum arm already had, and for the same reason: the name is erased before any backend sees it — and reads the fields of
|
||
**that case only**. It prints `(Shape.Dot {.x 1.5 .y -2.5})`, which is what the source would write.
|
||
|
||
### What is left
|
||
|
||
An **imported** union is still refused by name at `load.ml:312`, so a union is file-local. That is not a blocker for
|
||
the macro expander: the prelude is parsed and prepended into the same flat namespace before `collect` runs, so a
|
||
`defdata Form` in `prelude.ml` is an ordinary same-file declaration needing no import — verified by declaring one
|
||
there and matching it from a program. `dev.ml`'s inspector still says "union values are milestone 6" for a stopped
|
||
frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards
|
||
is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates.
|
||
|
||
## `(Handle T)` and the pool, which is what a stale reference answers with
|
||
|
||
A handle is a reference to something that can die, which reports that it died rather than silently resolving to
|
||
whatever reused its slot. `check.ml` refused `(Handle T)` by name as milestone 6; this is what it stood for, and the
|
||
pool came with it because the pool is what makes the report possible.
|
||
|
||
The problem is concrete and is not about memory safety. Entities live in a pool; something holds a reference to one —
|
||
a projectile chasing it, the UI showing its health. The entity dies, the slot is reused, and a raw index now names a
|
||
different entity. Nothing crashes. The projectile chases the wrong thing, at full speed, for the rest of the game.
|
||
|
||
It was built for two reasons, both already recorded. On its own terms, for entities referred to across frames. And
|
||
because **it is the real gate on managed classes**: plan.org's rule is that nothing starts on `defclass` until
|
||
ordinary `struct`, `Handle` and reload semantics work, and `Handle` was the only one of the three missing. That is not
|
||
an incidental precondition — `migrate-instances` has to *enumerate* live instances, and a pool behind generational
|
||
handles gives that by construction where a world arena and an owned region do not. plan.org presents the three storage
|
||
strategies as a free choice and they are not.
|
||
|
||
`PORTING.md` found no customer for handles in the author's real game today, so this is deliberately the smallest
|
||
correct thing rather than a rich API: eight names, no iteration protocol, no cursor type, no `clone`.
|
||
|
||
### A handle is one i64, and the halves are 32 and 32
|
||
|
||
Slot index in the low 32 bits, that slot's generation counter in the high 32. One machine word, so it copies, zeroes
|
||
and compares like the integer it is, and it **owns nothing** — the pool is the single owner. That is what lets a
|
||
handle sit in a struct field and in a global where a `Vec` may not, and it is the reason the ownership rules needed no
|
||
new case: `Types.is_move_only` says yes to `Pool` and no to `Handle`, and the three existing refusals (a struct field,
|
||
a union case, a global) picked the pool up unchanged with the messages they already had.
|
||
|
||
The index is 32 bits because a `Vec`'s index is an `i32` here and widening indices is one change across every
|
||
container, not a pool question.
|
||
|
||
### Live is odd, and two things fall out of it
|
||
|
||
A slot's generation starts at 0 and is bumped on every allocation *and* on every release, so an odd generation means
|
||
live and an even one means dead. Both consequences are load-bearing:
|
||
|
||
- **A zeroed handle resolves to nothing.** Generation 0 is even, so ZII gives a `(Handle T)` field the right meaning
|
||
for free instead of pointing it at slot 0. `handles.flan` prints one: `<handle 0:0>`, and resolving it answers `None`.
|
||
- **Iteration needs no second array and no spare bit.** Asking whether a slot is live is asking whether its generation
|
||
is odd.
|
||
|
||
### The generation wraps by retiring the slot
|
||
|
||
32 bits is 2^31 allocate/release pairs on one slot — every frame at 60fps for a year and a bit. "Rare" is not an
|
||
answer when the failure it produces is the silent wrong one this type exists to prevent, so a release from generation
|
||
`0xFFFFFFFF` bumps to 0 and does **not** put the slot back on the free list. The slot is retired: dead forever, its
|
||
payload leaked, and no future handle can collide with an old one. Leaking is defined behaviour here, and one slot is a
|
||
bounded price for making the collision unrepresentable rather than unlikely.
|
||
|
||
### `resolve` answers `(Option (Ptr T))`, and the spec settled that, not this lane
|
||
|
||
The task asked whether a lookup should answer `(Option T)`, matching `(get m k)`. It answers `(Option (Ptr T))`, and
|
||
`spec-memory.md` already writes it out — its worked example under "Mutating something you matched" is annotated
|
||
`(Option (Ptr Enemy))` — for the reason given one line above it: *pattern bindings bind values, so a matched struct is
|
||
a copy*. A copy cannot be written back, and writing to the pooled entity in place is what a pool is for. `(Option T)`
|
||
would answer a question nobody asked.
|
||
|
||
The `Option` half is `get`'s shape and for `get`'s reason: absence is an answer, not a failure. A trap would be wrong
|
||
here — the entity dying is the *expected* case, not a bug.
|
||
|
||
**The hole, said plainly.** A `(Ptr T)` from `resolve` is invalidated by any `insert` that grows the pool, exactly as
|
||
a slice is invalidated by a `push`. The handle survives that and the pointer does not. This is `spec-memory.md`'s
|
||
explicit Zig/Odin borrowing contract one level down, and it is worth naming rather than implying, because it
|
||
reintroduces the silent-wrong-answer mode the handle just removed for anyone who keeps a resolved pointer across an
|
||
insert. Chunked never-moving storage is the fix and it costs code; taking the contract is the smaller correct thing,
|
||
given `as-slice` already established it.
|
||
|
||
### `len` is the slot high-water and `live` is the count, in that direction
|
||
|
||
`(len p)` is how many slots have ever been handed out. `(live p)` is how many of them are live now. It had to be that
|
||
way round: `0..(len p)` are the indices `(pool-handle p i)` accepts, so a loop bounded by `len` visits every live
|
||
entry. Bounded by the live count instead, it would silently skip entries the moment anything had been released —
|
||
which is exactly the quiet wrong answer the whole type exists to remove.
|
||
|
||
`(pool-handle p i)` answers `(Option (Handle T))`: the handle of slot `i`, or `None` if that slot is dead. That plus
|
||
`len` is the whole of iteration. An index outside `0..(len p)` **traps**, exactly as `(at v i)` traps: an index is an
|
||
index here, and answering `None` for one would hide a bug rather than a death.
|
||
|
||
### A slot is released through the pool, and that is not a third release point
|
||
|
||
`free` consumes its argument as a move, and a handle is a copyable number that owns nothing — consuming one copy would
|
||
say nothing about the others. So `(free h)` is refused by name and the release operation is on the owner:
|
||
`(release p h)`. `spec-memory.md`'s two release points are untouched: `(free p)` is release point 1 applied to the
|
||
pool, and a `free-all` of the region takes the pool with everything else. `release` recycles a slot inside storage the
|
||
pool still owns, which is not a release of storage at all.
|
||
|
||
It answers `bool` rather than `()`: true if this call released it, false if the handle was already gone. The
|
||
generational scheme makes a double release **detectable**, and that is worth handing to the caller — this is the one
|
||
place in the language where freeing something twice is an answer instead of a refusal.
|
||
|
||
### Growth is transactional, because `retry` re-attempts the same call
|
||
|
||
`StorageExhausted`'s restart re-attempts the *same* request, so a failed grow has to leave the pool byte for byte as
|
||
it was — including a `cap` that still agrees with the real block sizes, since the next attempt passes `cap` as the
|
||
allocator's `old_size`. A pool grows two blocks together (payloads and slot headers), so resizing the first in place
|
||
and then failing on the second would leave `cap` describing neither. So the runtime allocates both, copies, and only
|
||
then releases the old pair: nothing is mutated after the last thing that can fail. An allocator without `can-free`
|
||
leaks the first block when the second fails, which is the defined outcome and not a new one — the request failed
|
||
because the region is exhausted, and the region is about to be released whole or its ceiling raised.
|
||
|
||
### Two failures, kept apart
|
||
|
||
A stale handle is an **answer**: `resolve` says `None` and the program carries on. A pool whose allocator was released
|
||
**traps**, through the same epoch check a `Vec` gets — the slot array went with the storage and there is nothing left
|
||
to ask. `test/programs/pool-stale-region.flan` is that case, and keeping the two apart is the same rule that keeps a
|
||
`Vec`'s generation word and its epoch word apart: they answer different questions and must not be conflated.
|
||
|
||
### Two amendments to a frozen spec
|
||
|
||
Both are places where `spec-memory.md` describes a handle doing something that cannot answer "gone", which is the one
|
||
thing the type exists to do. **This amends it: both are deferred, not built.**
|
||
|
||
**1. `.field` and `at` do not auto-deref a handle.** The Places table says `x` may be a struct, a `(Ptr S)` or a
|
||
`(Handle S)`, and that the two forms auto-deref exactly one pointer *or handle* level. They auto-deref one pointer
|
||
level and nothing else. A `(set (.hp h) ...)` through a handle has two possible meanings when the entity is dead — trap,
|
||
or do nothing — and both are worse than the third option, which is the spec's own worked example: resolve first, match,
|
||
and the compiler makes you handle the `None`. The spec contradicts itself here and the example is the half that is
|
||
right.
|
||
|
||
**2. `deref` is not overloaded on `(Handle a)`.** The Generics section says "`deref` yields a value; `resolve` yields a
|
||
pointer. Both are overloaded on `(Ptr a)` and `(Handle a)`." `deref` is `(Ptr a)` only. Same reason: `deref` returns a
|
||
value and has nowhere to put "gone".
|
||
|
||
### What this does not have, and one of the gaps is not a pool question
|
||
|
||
- **No `clone`.** Refused by name. A copied pool would carry the same slot generations, so one handle would resolve in
|
||
both copies and name two different things — the exact confusion the type removes. A program that wants a second world
|
||
builds one and inserts into it, and the new handles say they are new.
|
||
- **No pool of an owning element.** `(Pool (Vec i32))` is refused where `(Vec (Vec i32))` is refused and for the same
|
||
reason: the type-erased runtime copies and releases slots bytewise. Recursive teardown arrives with `drop`.
|
||
- **A handle is not a map key.** For the reason a `Ptr` is not: hashing an identity is a different operation from
|
||
hashing what it names, and a stale handle hashes the same as it always did while naming nothing.
|
||
- **Handles compare with `=` and not with `<`.** `Types` grew a second predicate, `is_equatable`, beside
|
||
`is_comparable`. Two handles are equal exactly when they name the same slot at the same generation, so a stale handle
|
||
is never equal to the live one that replaced it — that is worth one integer compare. Ordering them would order a slot
|
||
index, which is a free-list artefact and means nothing.
|
||
- **A pool passed to a helper is consumed**, because a pool is move-only exactly as a `Vec` is and there is no
|
||
borrowing parameter in the language. `test/programs/handles.flan` is one long function for that reason, and it does
|
||
not work around it. This is a pre-existing gap and not a pool question: the same sentence is true of every `Vec` in
|
||
the tree.
|
||
|
||
### What classes still need
|
||
|
||
The enumeration primitive is the piece migration was blocked on, and it exists now. What is left is `defclass` itself
|
||
and its runtime shape metadata; `migrate-instances`, which is a walk over `(len p)` and `(pool-handle p i)`; generic
|
||
functions and method dispatch, whose expensive half is already built and tested (a generic function is an indirection
|
||
cell whose body is a dispatch table, and a reload extends the table); and the rule that `Enemy@1` stays resolvable for
|
||
as long as any instance holds it — the same rule as "nothing is ever `dlclose`d".
|
||
|
||
## Macros: the compiler dlopens the program
|
||
|
||
"Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing
|
||
that turns that decision into a mechanism rather than an absence: **running a macro at compile time means compiling
|
||
it and loading it into the compiler's own process.** There is nothing to interpret it with and there is not going to
|
||
be, so `Emit.redefinition` → `Build.shared` → `dlopen`, the reload primitive the dev loop already runs, is pointed at
|
||
the compiler instead of at a running program.
|
||
|
||
`(defmacro name [args] body ...)` is one function, `[Form] -> Form`. One parameter, the slice of forms written at the
|
||
call site, which is where variadics come from in a language with no `&rest`: `(len args)` is how many were written.
|
||
|
||
### A defmacro is a defn, and there is no Ast.Defmacro
|
||
|
||
`Parse` turns `(defmacro m [args] body)` into `(defn m [args [Form]] Form body)` and nothing below the parser knows
|
||
the word exists. The checker checks it like any function, the backend emits it like any function, `Reach.link` drops
|
||
it from a program that does not call it like any function. The only thing that makes it a macro is that `Macro` calls
|
||
it at compile time instead of the program calling it at run time.
|
||
|
||
This is also why there is no macro table. Storage was the question the front half deliberately left open, and the
|
||
answer is that there is none: the macro set is recomputed by scanning the top level for the word `defmacro`, which is
|
||
the only place it survives, and the compiled artefact is a `.so` keyed by a digest. The top level scanned is the
|
||
prelude's, the file's own, **and the imported packages'** — the last arriving through `Parse.imported_macros`, already
|
||
qualified under the alias the package was imported as. See "A package may declare a macro" under *Packages*.
|
||
|
||
### `Form`, and the three numbers
|
||
|
||
A macro's parameter and its result are `Form`, so `Form` has to exist on the Flan side: a `defdata` in `prelude.ml`
|
||
mirroring `lib/form.ml`. It mirrors `Form.value` and **not** `Form.t` — there is no `loc` field, deliberately. A macro
|
||
cannot invent a source location, so the unmarshaller stamps the **call site's** `Loc.t` onto every node of what a
|
||
macro returns. That is the structural answer to "keep the source location of the call site attached to what a macro
|
||
produces", and it is what the queued structured-error work reads.
|
||
|
||
Case order is tag order, so the list in the prelude is a layout contract and says so. The widest cases are
|
||
`(Str [s string])` and `(List [xs [Form]])`; a string and a slice are both `%slice` = `{ptr, i64}`, 16 bytes at
|
||
align 8. So the image is `{ i32 tag, [2 x i64] payload }`: **24 bytes, align 8, payload at offset 8**, and every case
|
||
holds its one member at the payload's start, so there is no third offset anywhere in the marshaller.
|
||
|
||
Those three numbers are asserted, not assumed. `test_acceptance.ml`'s "Form's image format" asks LLVM for each of them
|
||
through the same `ptrtoint`-of-`getelementptr`-through-null oracle the DWARF offsets go through. Alignment needed a
|
||
probe the oracle did not have: the offset of field 1 in `{ i8, %"Form" }` *is* `alignof(Form)`, because a struct member
|
||
sits at the first offset its own alignment allows. Reading `[2 x i64]` out of the emitted type and concluding 8 would
|
||
be asserting the layout against itself, which is the circularity that got a `_Static_assert` rejected for the FFI.
|
||
|
||
### Nothing aggregate crosses to C
|
||
|
||
The boundary is `void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer)` — one thunk per macro, written by
|
||
`Emit.macro_thunk`. The thunk builds the `%slice` from `(args, n)` on the LLVM side, calls the macro, and stores the
|
||
result through `%out`.
|
||
|
||
The correction that matters here is not obvious from the diff. The unions work verified a union's **memory** layout
|
||
against clang; that is a different claim from LLVM's calling convention for an aggregate passed or returned **by
|
||
value** in hand-written IR, which is not promised to be clang's C ABI for the equivalent struct. Memory is the
|
||
agreement that actually exists, so pointers and scalars are all that cross. `%xfer` is the transfer channel every Flan
|
||
signature carries; `flan_macro_call` supplies a zeroed one, because a macro that signals with nothing above it to
|
||
handle it aborts inside the compiler, and the channel still has to be a real slot.
|
||
|
||
`Build.macro_module` produces a self-contained `.so`: the runtime linked in, no undefined Flan symbols, `-fPIC` on
|
||
every object including the `.ll`. Self-contained is what keeps `-rdynamic` off the compiler's own link. It goes
|
||
through clang rather than `llc` + `ld -shared`, unlike `Build.shared`, because there are C objects and a libc to find
|
||
— exactly the part of the driver the dev path skips.
|
||
|
||
OCaml has no `dlopen` for ELF (`Dynlink` loads OCaml), so `lib/dynload_stubs.c` is the whole boundary: `dlopen`,
|
||
`dlsym`, `dlclose`, the four-argument call, `calloc`/`free`, and a peek/poke family, because OCaml cannot address raw
|
||
memory and a `Form` image is written into it one field at a time.
|
||
|
||
### Quasiquote runs before the walk, and that is not a preference
|
||
|
||
Quasiquote is a desugaring over `Form` and nothing more: it becomes `form-nil`, `form-cons` per item and
|
||
`form-append` per splice — the prelude's three form-building functions and no fourth. It is pure, it needs nothing
|
||
loaded, and `Parse.program` runs it on the way in, which is what lets the prelude's own macros parse in a process
|
||
that has not built a macro module yet.
|
||
|
||
Running it **before** the expander's walk is load-bearing. A recursive conditional macro's body contains a
|
||
quasiquoted call to itself; with the quasiquote still standing, the walk would see that head and expand it then and
|
||
there, against the wrong arguments. Desugared first, that subform is a `(Form.Sym {.s "cond"})` and there is no head
|
||
left to mistake — so the walk needs no idea that quoting exists.
|
||
|
||
Nesting levels are counted nowhere: not by the reader, which was written that way deliberately, and not by the
|
||
desugaring. A quasiquote inside a quasiquote is refused by name. Only a macro that writes a macro wants one.
|
||
|
||
### A call inside a quasiquote is output, not a dependency
|
||
|
||
This is the distinction that is easy to get wrong, and the first cycle test written for this work got it wrong: it
|
||
quasiquoted, and it was not a cycle at all.
|
||
|
||
A macro body that **calls** another macro outside a quasiquote needs that macro compiled and loaded first, because
|
||
until then the call is a name nothing defines and the body will not compile. That is a compile-order dependency and it
|
||
is what makes the pre-pass a fixpoint. A macro body that **quasiquotes** a call to another macro needs nothing: the
|
||
call is part of what the macro answers, and the answer is expanded again after it returns.
|
||
|
||
So there are two different ways expansion fails to terminate, and they are different failures:
|
||
|
||
- **A ring** — two macros whose bodies call each other for real. There is no order to compile them in, so it is
|
||
refused, naming both. `test/programs/macro-cycle.flan`.
|
||
- **A macro that expands into a call to a macro and does not get smaller.** That is an ordinary loop, not an ordering
|
||
problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site.
|
||
`test/programs/macro-spin.flan`.
|
||
|
||
Both fire when the macros arrive from an **imported package**, and each has its own case for it —
|
||
`test/programs/pkg-macro-ring.flan` and `pkg-macro-spin.flan`. They are worth keeping apart there: a ring has no
|
||
compile order and one that never settles has one, and an import must not quietly turn either into the other. The ring
|
||
is refused inside the package, at the `defmacro` that closed it, because a package's own macros go through the same
|
||
rounds before any importer sees them; the one that never settles is refused at the importer's call site, under the
|
||
qualified name — `expanding s/spin did not settle`.
|
||
|
||
The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest
|
||
against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a
|
||
macro never sees a call to another macro in what it is handed.
|
||
|
||
### Hygiene is an escape hatch, not a system
|
||
|
||
Deliberately non-hygienic, Common Lisp's rule and Clojure's, settled in plan.org's open decision 2. A macro that needs
|
||
a name of its own calls `gensym`, which is a prelude function the loaded module runs while it runs. The name is
|
||
`~g<n>`, and `~` is a delimiter now — it opens an unquote — so no symbol the reader can produce contains one and a
|
||
gensym cannot collide with a name someone wrote. The counter lives in the loaded module rather than in the compiler,
|
||
which is the one place this departs from the original sketch; a module is dlopened once per compiler process, so it is
|
||
process-wide in practice.
|
||
|
||
### -linkall, and why the hook could not be installed by hand
|
||
|
||
Expanding a macro means compiling it, so `Macro` needs `Check`, `Build` and `Emit` and therefore sits **above** the
|
||
parser it feeds. The join is `Parse.expander`, a ref that `Macro` fills in at module initialisation.
|
||
|
||
Nothing references `Macro`, so without `-linkall` the linker drops it from every executable that does not name the
|
||
module — `bin/main.exe` among them — and a program calling a macro fails with an unknown name. Installing by hand at
|
||
every entry point was the alternative and it is not viable: `lib/session.ml` calls `Parse.program` for `C-c C-c` and
|
||
`C-c C-k`, and `test_session.ml` drives the session library in-process rather than through the CLI, so the set of
|
||
places that would need an install call is open-ended and a missed one is silent. `(library_flags (-linkall))` in
|
||
`lib/dune` is the guarantee instead.
|
||
|
||
`Macro.building` is the re-entrancy guard. `Build.macro_module` goes through `Check.program`, which parses the
|
||
prelude, which calls back into `Parse.program` — and that would re-enter the expander forever. Nothing is lost by
|
||
refusing to expand there: a macro compiled in round *n* calls only macros compiled in earlier rounds, and those calls
|
||
were already expanded before the build was entered.
|
||
|
||
### What it costs
|
||
|
||
- A build of a program that **names no macro**: 50ms, unchanged. The pass scans the top level, finds nothing, and no
|
||
compiler runs. This is nearly every program, and it is the reason the prelude can grow a `defmacro` without every
|
||
build paying a clang driver.
|
||
- A program that **calls one**: 310ms the first time, 70ms after. The 240ms is the clang driver; the module is cached
|
||
under the object cache and keyed by a digest of the prelude's source plus the file's `defmacro` forms, so it is paid
|
||
once per change rather than once per build. Every `flan build` is a fresh process, which is what makes the on-disk
|
||
cache rather than a memo the right shape.
|
||
- **Importing a package that declares macros moved the cold number and not the warm one.** What the cache shows is
|
||
that more than one module is built: `test/programs/pkg-macro.flan` leaves four `flan-macros-*.so` behind where
|
||
`macros.flan` leaves two, and cold it is about twice the wait. Why four rather than three or two has not been
|
||
pinned down — the package's macros are expanded once for the package's own parse and again under the importer's
|
||
qualification, and the rounds within each of those are also separate modules; nothing here discriminates the two.
|
||
Warm it is the same 70ms, because every one of those modules is keyed and cached like the first. A program that
|
||
imports a package declaring **no** macro pays nothing new — the extra work is reading the import forms, which is a
|
||
scan of the top level.
|
||
|
||
Measured on this machine, with the cache warm for the runtime and cold for every macro module, so the numbers are
|
||
comparable to each other rather than to the two above: no macro 78ms, imports but no macro 110ms, the file's own
|
||
macros 747ms cold and 72ms warm, a package's macros 1507ms cold and 68ms warm.
|
||
- A hello-world's binary carries exactly one symbol out of all of this: `flan.gensym-n`, eight bytes. `Reach.link`
|
||
drops `unless`, `form-cons`, `form-nil`, `form-append`, `form-rest` and `gensym`, because nothing reachable calls
|
||
them.
|
||
|
||
### `unless` is the proof
|
||
|
||
plan.org milestone 5 says `when`, `unless`, `until`, `cond` and `dotimes` are special forms only until macros land.
|
||
`unless` is the first to stop being one — it is now a `defmacro` in `prelude.ml` and `parse.ml` has nothing to say
|
||
about it — and it was chosen because it is the one the prelude itself does not use. That matters: the prelude is
|
||
compiled into the macro module, so a prelude macro that the prelude's own functions call would need the expander to
|
||
compile the thing the expander needs in order to run.
|
||
|
||
Its coverage is `sand.flan`, seven calls, compiled through `Session` in `test_session` — the in-process path, and the
|
||
reason `-linkall` is not optional. Say plainly what that coverage is not: nothing in `test/programs` used `unless`
|
||
before this landed, so `macro-unless.flan` is a test written after the feature. The corpus written before it is
|
||
`sand.flan` and `web/examples/control.flan`, and both compile unchanged.
|
||
|
||
## `break` and `continue`, and the rule that replaced a blanket refusal
|
||
|
||
Declined once deliberately; NEXT.md's *`break`, and why it was not built* records what was settled then and still
|
||
holds — `dotimes` gets it free because it desugars to a `While`, `defer` is a non-question because it is
|
||
function-scoped and a break does not leave the function, and both are typed `Never` as `exit` and `return` already
|
||
are. What stopped it was two things, and both are answered here.
|
||
|
||
**Labels are Odin's, in the head position.** `(while :outer (< i n) ...)`, and `(break :outer)`. A keyword there is
|
||
unambiguous because a loop condition is never one, so one small `label` function in `parse.ml` serves `while`,
|
||
`until`, `dotimes`, `break` and `continue` and no form has to count its arguments. `until` and `dotimes` were not
|
||
asked for and cost a line each: `until` is a `While` by the time the parser is done with it, and a nested `dotimes`
|
||
scanning a grid is the case the label exists for.
|
||
|
||
**It is not a goto.** A label names one of the loops the form is *lexically inside* — the checker resolves it against
|
||
exactly those and refuses anything else by name — so control can only leave a loop it is already in. That is Odin's
|
||
restriction and it is what keeps the feature small: there is no arbitrary target, no forward jump, and nothing to say
|
||
about scopes being entered.
|
||
|
||
### The `in_frames` question, ruled on
|
||
|
||
`return` is refused inside a `handler-bind` or a `restart-case` because those forms push frames and pop them on the
|
||
way out, and a return that leaves would strand them on the stack pointing into a frame that is gone. That refusal is
|
||
**blanket**, and correctly so: a return *always* crosses.
|
||
|
||
A break crosses only sometimes. A loop written wholly inside a `restart-case` body has a perfectly good local break,
|
||
and refusing it would be refusing the common case for the sake of the uncommon one. So the rule here is **relative**,
|
||
and it is one list rather than a flag: `ctx.loops` holds the loops this form is inside, innermost first, with a
|
||
**barrier** entry pushed by every construct a jump may not cross. A break resolves by walking outwards; a barrier
|
||
reached before the target loop is a refusal that **names the construct** — *break :outer would leave a restart-case,
|
||
which it may not*. A loop nested inside the construct sits below the barrier and is never affected.
|
||
|
||
The barriers, and why each is one:
|
||
|
||
- **`handler-bind` and `restart-case` bodies** — the frames they pushed are popped on the way out, and a `br` past
|
||
the pop leaves a dangling frame. The same reason `return` is refused, scoped to crossings instead of to everything.
|
||
- **A `restart-case` clause body** — it runs after a transfer landed, with the form's frames still to be popped.
|
||
- **A `defer`'s forms** — they are *copied* into the function's exit paths, where the loop they were written beside
|
||
is not running. (Unreachable today, because `defer` is already refused in a loop body for its own reason. Written
|
||
anyway: the rule is about what the forms mean, not about which other refusal happens to fire first.)
|
||
|
||
A **handler clause** is not on the list at all. It is lifted into a function of its own with a fresh `ctx`, so its
|
||
loop stack starts empty and nothing in it can name a loop outside it — the refusal falls out of the lifting.
|
||
|
||
The two rules agree where they overlap, which is the check that the relative one is not weaker: a `return` is a jump
|
||
whose target is always outside every barrier, so the blanket refusal is the special case of this one. `in_frames` is
|
||
left exactly as it was.
|
||
|
||
### `continue` and the latch
|
||
|
||
`Tast.While` is now `expr * expr list * expr list` — condition, body, and a **latch** that runs after the body and
|
||
before the test. `check_dotimes` folded its increment onto the end of the body, and a `continue` branching at the
|
||
header would have jumped straight past it: the counter would never advance and the loop would hang. So the step is
|
||
the latch, `continue` branches to the latch block rather than to the header, and a `while` has an empty latch that
|
||
every optimiser folds away. `emit_while` emits four blocks instead of three.
|
||
|
||
A **labelled** `continue` is the half worth stating outright: `(continue :rows)` branches to the *named* loop's latch,
|
||
so that loop's counter advances and the rest of its body is skipped along with the rest of every loop inside it. It is
|
||
"start the next iteration of `:rows`", not "skip the rest of this innermost body". `loops.flan` asserts exactly that —
|
||
an outer `dotimes` whose tail never prints while its counter still runs out.
|
||
|
||
`Tast.Break` and `Tast.Continue` carry a **relative depth** — how many loops out the target is, innermost first —
|
||
rather than a name or an id, because that is exactly what a backend already has. `emit` keeps one entry per `While`
|
||
it is inside, the same shape and for the same reason as `pads`, and indexes it. The invariant this rests on: the
|
||
checker mints a depth only from its own loop stack, and the two stacks are pushed once per `While` each. A `While`
|
||
the checker *invents* — `alloc_guard`'s retry and the file-failure retry — is built directly and contains no jump, so
|
||
its emit entry matches nothing.
|
||
|
||
**Nothing in `lib/prelude.ml` wants either.** Every early exit there is a `return` from the function — `bytes=?`,
|
||
`index-of-byte`, `index-of-bytes`, `valid-utf8?`, `bytes->i64` — and a break cannot replace one: it leaves the loop
|
||
and the function still has to answer. The loop-with-a-sentinel-flag shape that break exists to remove does not appear
|
||
in the prelude. The two compiler-emitted retry loops in `check.ml` are that shape, and they are the one place it
|
||
cannot help: their sentinel is set inside a `restart-case` body, which is a barrier.
|
||
|
||
## `into`, which fuses at compile time because it is a macro
|
||
|
||
```
|
||
(into xs (vec-new i32) (map double) (filter even?))
|
||
```
|
||
|
||
Source, destination, then any number of transforms — the shape of the `into->` macro the author already uses in
|
||
Clojure. It reads as a sentence (take this, put it there, doing these) and the variadic tail has to trail anyway,
|
||
which is the mechanical reason the transforms cannot sit in the middle.
|
||
|
||
**A macro, and therefore not transducers and not iterators.** Transducers compose at *run time*: function values,
|
||
closures, an allocation, and a chain of indirect calls per element. Rust has no transducers either — it has iterators,
|
||
which fuse into one loop at compile time through monomorphisation, and that needs generics. A macro reaches the same
|
||
place with neither. `(map double)` expands to `(double x)` written straight into the loop body, so the function name
|
||
is **syntax and never a value**: no intermediate collection at any step, no closure, no generics, and nothing to
|
||
inline. `into.flan` counts every call to the transform functions, which is the assertion a unit test cannot make — a
|
||
chain that built a `Vec` per stage would pull a different number.
|
||
|
||
What it gives up is building a transformation at run time and passing it around. That is transducers' actual selling
|
||
point, it is the one part that would need run-time machinery, and it is close to useless in a game. Clojure's
|
||
`:eduction` branch is dropped for exactly that reason.
|
||
|
||
**Why the destination is in the form.** Every collecting operation here allocates from an *explicit* allocator, which
|
||
is a frozen rule in `spec-memory.md`. A `->>` chain hides where the result goes; naming the destination means the
|
||
macro knows its type, emits the right loop, and the rule is honoured by construction. `(vec-new i32 a)` names an
|
||
allocator here as it does anywhere else, because the destination form is written out untouched. This is what `->>`
|
||
threading over slices in `plan.org` is replaced by for the collecting cases.
|
||
|
||
### Reductions do not share the form, and that was the open question
|
||
|
||
`(into xs 0 (map cost) (sum))` reads oddly because zero is not a collection, and the oddness is the tell rather than a
|
||
matter of taste. The whole reason the destination sits in the form is that **the destination is the allocation** — it
|
||
is what makes the explicit-allocator rule checkable by construction. A seed is not an allocation, so a form that
|
||
accepted one would be two forms sharing a spelling, and the destination would have stopped being honest about what it
|
||
is. So `into` collects, and a reducing macro of the same shape is a separate form the day something wants one.
|
||
|
||
### The parts that took a decision
|
||
|
||
- **The destination is a `Vec`**, because `push` is what fills it. A `Map` destination is refused by `push` itself,
|
||
which says *push takes a (Vec T)* and names the real problem. There is no second lowering and no reason to invent
|
||
one before something asks.
|
||
- **A source that is already a name is used as it is; anything else is bound to a gensym.** Both halves are needed and
|
||
neither is cosmetic. Binding is what a source that is a *call* needs — `(len s)` and `(at s i)` have to be the same
|
||
`s`, or the call is made once per element. Not binding a name is what everything else needs: a `(Vec T)` is
|
||
move-only, so `(let [s v] ...)` would hand the caller's `v` to a binding it cannot see and `v` would be dead after
|
||
an `into` that only read it; and a fixed array would be *copied* into the binding, once per `into`. `len` and `at`
|
||
borrow, so used directly the source is only read.
|
||
- **An owning temporary as the source leaks**, and this is the wart. `(into (make-a-vec) ...)` binds the result to a
|
||
name the caller cannot reach and therefore cannot free. The macro cannot know whether the type owns anything. A call
|
||
in that position should borrow — `into.flan`'s does — and the day `drop` exists this stops being a question.
|
||
- **One element name throughout**, shadowed by each `(map f)` stage: `(let [x (f x)] ...)`. A `let` binding's value is
|
||
checked before its name is bound, so the initialiser reads the outer `x` — that is the language's rule, not an
|
||
accident, and `bind`'s `x~2` debug suffix exists precisely so a debugger does not lie about which is which. A
|
||
**type-changing** `map` is the case this most plausibly breaks and it does not: each stage is a fresh slot at the
|
||
stage's own type, and `into.flan` runs an `i32` source into a `(Vec f32)` to say so.
|
||
|
||
### What the prelude's macro limits cost, exactly
|
||
|
||
All four bit, and none blocked anything.
|
||
|
||
- **A macro has no error facility**, so the three refusals are calls to names nothing defines:
|
||
`into-takes-a-source-a-destination-and-transforms`, `into-transform-is-map-or-filter` and
|
||
`…-of-one-function`. The report is *unknown function* at the call site with a *expanded from the macro into* note
|
||
under it, which is the right place and the wrong sentence. The bad transform is passed along as an argument so that
|
||
at least it is named.
|
||
- **A prelude macro may not call another macro**, so `into-wrap` is a plain `defn` and uses only special forms —
|
||
`loop`, `cond`, `when`, `let`, `if`. A `clamp` or an `unless` in there would have put it in the set `Macro.reduce`
|
||
drops.
|
||
- **Nested quasiquote is refused**, and it was not needed: each wrapper is a single-level quasiquote over a `body`
|
||
already built.
|
||
- **Macros are not importable**, which is why `into` is in the prelude rather than a library.
|
||
|
||
`into-wrap` walks the transforms **in reverse**, because the chain is built from the inside out: the innermost form is
|
||
the `push`, and each transform wraps what the ones after it produced. That reverse walk with two accumulators is the
|
||
first thing in the prelude written as a `loop`/`recur`, which landed in the commit before this one.
|
||
|
||
## `loop` and `recur`, and why `recur` is better than tail calls and not only cheaper
|
||
|
||
There is no TCO anywhere in this compiler — nothing emits a tail call, and `plan.org` mentions them only as something
|
||
the backend choice *could* control. `recur` is the answer, and the reason is not that it is cheap. **It is checked.**
|
||
The compiler verifies the call is in the loop body's tail position and turns it into a jump, so writing it in the
|
||
wrong place is a compile error at the place it was written. Under silent TCO the same mistake compiles and is a stack
|
||
overflow at run time, with a backtrace pointing at whatever ran out of stack rather than at what was wrong. Clojure
|
||
adopted `recur` because the JVM lacks TCO; it turned out to be the better design, and it is the better design here for
|
||
the same reason.
|
||
|
||
What it does **not** give is mutual recursion between two functions. That needs real tail calls and is out of scope,
|
||
and the refusal for a `recur` outside any loop says so in as many words.
|
||
|
||
### Nothing new reaches the backend
|
||
|
||
`(loop [x 0 acc 1] body ...)` is a `let` over the names, a `While` whose condition is `true`, and two jumps. `emit.ml`
|
||
is untouched. That is the whole argument for building this on the labelled `break`/`continue` that landed just before
|
||
it: the machinery was already there, and the question `recur` asks — *may this jump cross that* — is the question
|
||
`break` already answers.
|
||
|
||
- **A loop answers with the value of its body.** The result goes into a slot of its own on the way out and is read
|
||
after the loop, so an accumulator comes back without a mutable local and without a sentinel flag.
|
||
- **A `Unit` body needs no slot**, and a **`Never` body needs neither slot nor break** — a body every path of which
|
||
recurs or returns never falls off the end, so there is nothing to break to and nothing to store.
|
||
- **`Set` of a `Never` body is safe**, which is the one thing that had to be checked rather than assumed. `emit`
|
||
closes a block at its terminator and drops what follows (`ins` tests `f.live`), and `Tast.Set` resolves the place
|
||
before the value, where a local's place is an address with no instruction behind it. So when the body ends in a
|
||
jump, the store is simply never written.
|
||
- **`recur` rebinds every name at once.** The new values go into temporaries and are written afterwards, so
|
||
`(recur y x)` swaps. Interleaved writes would give `y y`, and `recur.flan` asserts the swap for exactly that
|
||
reason.
|
||
- **A move-only accumulator goes round.** `(loop [acc (vec-new i32) i 0] ... (recur acc (+ i 1)))` is the shape the
|
||
form exists for, and it is the one the move tracker had to be taught about (below). `recur.flan` carries a `Vec`
|
||
round three iterations and answers with it.
|
||
|
||
`tast.ml` said a `While` the checker *invents* contains no jumps, because the depths would be minted against a stack
|
||
it is not on. `check_loop`'s `While` is the exception, and it is the exception because it is pushed on `ctx.loops`
|
||
like any other: being invented was never the property that mattered, being on the stack is. The comment now says so.
|
||
|
||
### Tail position, as a permission that is withdrawn
|
||
|
||
The alternative was a pre-pass over the `Ast` marking tail positions, which would have to enumerate every constructor
|
||
and stay in step with the type forever. Instead `ctx.tail` is read and withdrawn at the top of `check`, the same
|
||
read-and-withdraw `defer_ok` already does and for the same reason: nothing reached from here inherits it. Three forms
|
||
hand it back on, and they are the only three that pass a tail through — the last form of a `block`, both arms of an
|
||
`if` (including the one-armed `when` shape, which is how nearly every loop is written), and a `match` arm. Everything
|
||
else is non-tail **by construction**, and no walk has to list the cases that are not.
|
||
|
||
The bodies of `restart-case` and `handler-bind` are tails semantically and are deliberately not given the permission
|
||
here — the barrier below refuses them anyway, and with a better sentence.
|
||
|
||
### `loop` is a barrier for `break` and `continue`
|
||
|
||
`lentry` gains `Lrecur`, carrying the slot and type of each of the loop's names. It is the target a `recur` resolves
|
||
to, by the same walk over the same stack `break` makes, refusing on the same barriers — `handler-bind`, `restart-case`
|
||
and a `defer`'s forms — rather than by a second mechanism.
|
||
|
||
It is **also a barrier itself**, and that is a restriction added here rather than one inherited. A loop answers with
|
||
the value of its body; a `break` out of one would have to produce that value from somewhere and there is nowhere, and
|
||
a `continue` would re-run the body without rebinding anything. So both are refused, and the message names the fix
|
||
(*answer with the value, or use a while*). A `while` written **inside** a loop sits below the entry and keeps its own
|
||
perfectly good break, which is the relative rule doing the job it was built for.
|
||
|
||
Two consequences fall out of this and are worth stating:
|
||
|
||
- **`loop` takes no label**, because there is nothing for a label to name. A leading keyword is caught in `parse.ml`
|
||
rather than left to `bindings`, which would have complained that `:outer` has no value.
|
||
- **A `recur` can only ever be at depth 0 in practice.** A loop body is not a tail position, so no `recur` is ever
|
||
written inside a nested loop. `recur_target` counts the depth anyway rather than assuming it, because the count is
|
||
what `emit` indexes.
|
||
|
||
### Two small things the shape forced
|
||
|
||
**A loop binding is a plain name.** `let`'s `bindings` expands a destructuring pattern into several bindings from one
|
||
form, and then `recur`'s argument count would no longer be readable off the binding vector. `loop_bindings` is the
|
||
pairs without the patterns, and it refuses a duplicate name.
|
||
|
||
**The move tracker had to be told.** `in_loop` refuses a body that moves a binding declared outside the loop, because
|
||
the second iteration would use what the first gave away. A loop's own names are bound before the entry is pushed —
|
||
their initial values are evaluated once, outside — so they would have landed in that set, and
|
||
`(loop [v (vec-new i32)] ...)` would have been refused for doing the ordinary thing. `recur` writes every one of them
|
||
on the way round, so the rule is not about them; `in_loop` takes the loop's own slots and excludes them.
|
||
|
||
## `(array 4 rl/Vector2)`, and the one position with no type slot
|
||
|
||
`[4 T]` is the ordinary type spelling and is unchanged. It already works everywhere a type is expected — `(defvar
|
||
points [4 rl/Vector2])`, `(defn draw [pts [4 rl/Vector2]] ...)`, a `defstruct` field, a return. A **`let` binding is
|
||
the single position with no type slot**, and there the brackets are read as what they are in expression position: an
|
||
array *literal* of two elements, whose second element is a type name nothing declares. So `(let [pts [4 rl/Vector2]]
|
||
...)` failed with *unknown name rl/Vector2*, which describes the symptom and not the mistake. It cost 32 hand-written
|
||
`Vector2`s in one raylib example.
|
||
|
||
`(array COUNT TYPE)` is the answer: a zeroed fixed array, told its count and its element type as plain arguments.
|
||
`(array 4 rl/Vector2)` and the type `[4 rl/Vector2]` denote the same type, so the constructor is assignable to a
|
||
declaration written the other way and either spelling can be the parameter — `array-ctor.flan` asserts exactly that.
|
||
|
||
It is a parser form and not a builtin call, because the second argument is a *type* and there are no types in the
|
||
parser's callers. `Parse` assembles the whole `Tarray (len COUNT, TYPE)` itself, which is why the count takes a
|
||
constant's name for free — `len` is the same function `[n T]` goes through — and why a non-type second argument is
|
||
refused by the type reader's own message rather than as an unknown name. The checker resolves it and hands back
|
||
`Tast.Zero`, the same node a declaration with no initialiser gets. There is no new backend node and no new type.
|
||
|
||
**`(zeroed)` was the first proposal and was rejected on how it reads.** `(zeroed [4 rl/Vector2])` is unambiguous to the
|
||
*parser* — a bracket in argument position could be a type there — but to a person it still looks like a two-element
|
||
vector, which is the exact confusion being fixed. `zeroed` keeps its existing job: the empty value of whatever type the
|
||
destination wants, inferred and never written. `array` is the one that is told.
|
||
## The prelude's second tier: the functions that return new storage
|
||
|
||
Everything in the prelude before this was slice-based and allocation-free, and NEXT.md's diagnosis of why was exact:
|
||
there was nothing to allocate from when it was written. `Vec`, `Map`, an arena and `StorageExhausted` changed that,
|
||
and this is the tier that follows — 24 additions, of which the twelve that matter most **return new things**
|
||
instead of writing into a buffer the caller supplies. The rest fill in the slice family at the element types that
|
||
were missing, and one of them is a macro.
|
||
|
||
Three rules hold across all of it, and they are stated once at the head of the section rather than repeated:
|
||
|
||
1. **The result is owned and the caller frees it.** Nothing is released at scope exit — not at the end of a `let`,
|
||
not at the end of a function (`spec-memory.md`). A caller writes `(free v)`, or lets a `(free-all a)` take the
|
||
whole region.
|
||
2. **The allocator is the context's, and `with-allocator` is the override.** This is the one design decision the
|
||
spec did not settle by itself. `(vec-new)` and `(map-new)` take an optional trailing allocator because the
|
||
*checker* builds them and can vary their arity; a Flan `defn` cannot, so the choice was an allocator parameter on
|
||
every signature or none. None — `(with-allocator a (join parts sep))` is the override, the `Vec` records the
|
||
arena, and `free` and `clone` never need it named again.
|
||
3. **No `Result` anywhere.** Allocation failure signals `StorageExhausted` under `retry`, and no allocating
|
||
operation returns an error, so every signature says what it produces and nothing about how it might fail.
|
||
|
||
### The builder is not a type
|
||
|
||
Odin's `strings.Builder` wraps a `[dynamic]u8`. Here the `(Vec u8)` already **is** that and already has `push`, so
|
||
the struct would be a move-only wrapper whose only method is the one it wraps. What was actually missing is appending
|
||
a *run* of bytes, and `append!` is that.
|
||
|
||
It takes a `(Ptr (Vec u8))` and not a `(Vec u8)`, and that is not style: a `Vec` parameter **moves**, so a by-value
|
||
builder would be consumed by its first append and refused on the second.
|
||
|
||
`append-i64!` and `append-f64!` are the argument for the whole shape. NEXT.md's "Sharp edges" records that
|
||
`flan_i64_to_bytes` and its neighbours render into one `static char scratch[64]`, so two formatted numbers cannot be
|
||
held at once; these copy out of that buffer before returning, so the hazard ends at the call and a builder holds as
|
||
many numbers as it likes. `strings.flan` puts two integers and a float on one line, which is the case that could not
|
||
be written before.
|
||
|
||
### `split` answers a `(Vec [u8])`, and the owning shape is unrepresentable
|
||
|
||
The fields are slices *of the input*. That was not a performance choice when this was written: `(Vec (Vec u8))` was
|
||
**refused outright**, so there was no owning shape to have chosen instead. That refusal has since been narrowed — see
|
||
spec-memory.md, "A container of owning elements lives in a region" — and a `(Vec (Vec u8))` is now buildable, against
|
||
a region allocator and nowhere else. `split` is unchanged anyway, and now by choice rather than by refusal: an owning
|
||
`split` would have to allocate one block per field and would only be usable in the tier that can never hand one back,
|
||
while the slices cost nothing and work everywhere. It follows, as before, that the result dies with whatever the input
|
||
pointed at, which is the same contract `trim` and `split-next!` already have.
|
||
|
||
The rule is `split-on-byte`'s, unchanged: n separators always yield n+1 fields, so an empty input yields one empty
|
||
field and a trailing separator yields a trailing empty one. That is Odin's allocating `strings.split` and not Odin's
|
||
`split_by_byte_iterator`, which disagree with each other on exactly that input.
|
||
|
||
Constructing it needed a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`, because `check.ml`'s `vec_new_elem`
|
||
takes the element type as a single bare symbol and `[u8]` is not one — so a `(Vec [u8])` can only be made where the
|
||
*context* names the type, and a return type is a context while a `let` is not. Written down in NEXT.md as a compiler
|
||
gap rather than worked around silently.
|
||
|
||
### `format-f64`, and the rounding rule it does not share with printf
|
||
|
||
`f64->bytes` is `snprintf "%g"`: six significant digits, exponent notation of its own accord, no precision to pass
|
||
it. A frame time of 1/60 comes back `0.0166667` and a score past a million `1.23457e+06`.
|
||
|
||
`format-f64` returns a `Vec`, so it inherits neither that nor the shared scratch buffer, and it renders the integer
|
||
part and the fraction through that buffer in strict sequence — the discipline `append-i64!` exists to make automatic.
|
||
|
||
It rounds **half away from zero at the last digit kept**, which is `round-f32`'s rule and the rest of the prelude's.
|
||
printf rounds the *binary* value to nearest-even at the decimal digit, so `0.125` at two places is `0.13` here and
|
||
`0.12` there. Matching printf would mean pinning a particular libc's answer, and that answer is not the same on every
|
||
target anyway.
|
||
|
||
Three lines in it are the ones a plausible version ships without, and each is a separate test case:
|
||
|
||
- **The carry.** `0.999995` at five places scales to exactly `100000`, which is not a fraction — it is the next
|
||
integer. Without the carry it prints `0.100000`.
|
||
- **The zero padding.** The fraction of `1.005` at three places is `5`, and `5` is not `005`; without the pad it
|
||
prints `1.5`.
|
||
- **The sign.** It belongs to the number, not to its integer part: `-0.5` has an integer part of `0`, and
|
||
`i64->bytes` of `0` carries no sign.
|
||
|
||
`-0.0` prints as `0.00`, because the sign test is `(< x 0.0)`, which `-0.0` fails. Past `9e18` the integer part does
|
||
not fit in an `i64` and there are no fractional bits left anyway, so it falls back to `%g` rather than approximating.
|
||
|
||
### `clamp` is a macro, and `atan2`/`pow` are declares
|
||
|
||
`clamp` is the second prelude `defmacro` after `unless`, and the reason is the prelude's own objection to wrapping
|
||
`(min hi (max lo x))` turned around rather than dropped. `min` and `max` are builtins at *every* numeric type and
|
||
there are no generics, so a clamp **function** is one copy per type — `clamp-i32`, `clamp-f32`, `clamp-i64`. A macro
|
||
is type-agnostic for free and emits nothing at all. `math2.flan` makes the same three-word call at `i32`, `i64`, `u8`
|
||
and `f32` to show it, and counts evaluations to show each argument appears once.
|
||
|
||
`atan2-f32` and `pow-f32` inherit `sin-f32`/`cos-f32`'s caveat in full and not `sqrt-f32`'s: IEEE-754 requires
|
||
nothing of `atan2f` or `powf` either, so they are the third and fourth places in the prelude where native and wasm32
|
||
may differ in the last bit. Every case in `math2.flan` is therefore a value exact in binary — a quadrant boundary, a
|
||
power of two, a perfect square — and the `-O0` run is the one that proves the symbols resolve, since at `-O2` LLVM
|
||
constant-folds a `powf` of two literals and leaves nothing to link.
|
||
|
||
### What could not be built, and why it is not "no generics"
|
||
|
||
Four things on NEXT.md's list did not land, and the interesting part is that the reason differs in each case.
|
||
**Three of the four have since landed** — see "`map-next!`, the one thing a Map could not do", "Function values, with
|
||
no capture" and "A prelude function may call a prelude macro" below — and each was fixed by the thing named here
|
||
rather than by generics, which is the argument this list was making. The fourth, the path-insensitive dead set, is
|
||
still open. Kept as written because the diagnoses are what the later lanes worked from, and one of them turned out to
|
||
be wrong in a way worth being able to see: the prelude *was* reaching the expander.
|
||
|
||
- **`Map` keys and values** need a **map iterator**, and there is none. `flan_map_len`, `_get`, `_put`, `_has`,
|
||
`_clone`, `_reserve`, `_free` is the runtime's entire map surface; nothing walks the open-addressed block. One
|
||
runtime function taking a cursor and one builtin in `check.ml` to emit the key and value sizes is the whole job,
|
||
and none of it is a generics question.
|
||
- **`map`, `filter`, `reduce` and a comparator sort** are blocked on **function values**, which is sharper than "no
|
||
generics" and matters because generics alone would not fix it. `Types.Fn` exists; `check.ml` refuses it with "a
|
||
function type is not implemented yet — milestone 5"; there is nothing in the language to pass. The concrete answer
|
||
is the one that shipped: `sort-f32!` and `sort-bytes!` are the second and third sorts in the language, and
|
||
`sum-i32`/`sum-f32` already are `reduce` with the `+` written in.
|
||
- **The prelude is never macro-expanded**, so a prelude function may not call a prelude macro. `Macro.program` runs
|
||
over the file being compiled; the prelude reaches the checker through `Check.program`'s prepend. The call resolves
|
||
to the macro's underlying `defn` and reports an arity error, which is why `format-f64` writes
|
||
`(min 9 (max 0 prec))`.
|
||
- **A returned `Vec` is a move and the dead set spans the function**, so an early `(return v)` on one branch kills
|
||
the binding at the foot of another. `replace-bytes` guards its empty needle with an `if` rather than a
|
||
`when`/`return` for that reason.
|
||
|
||
### The refusal block is down from eight reasons to four
|
||
|
||
The list at the foot of `prelude.ml` used to be one sentence — every entry needed to produce bytes that did not exist
|
||
in its input, and there was no allocator. `join`, `concat`, `split`, `to-lower`, `to-upper`, `repeat` and `replace`
|
||
have moved up into the code; `string-from-bytes` turned out to be the `string` builtin all along, and
|
||
`(string (as-slice v))` is the round trip, free precisely because the layouts are identical.
|
||
|
||
What remains is refused for four different reasons, and is now written that way: `pad`/`center` for *nothing at all*
|
||
except that no caller has asked; `format`/`sprintf` for variadics of mixed type; `map`/`filter`/`reduce`/`sort-by`
|
||
for function values; `map-keys`/`map-values` for the missing iterator.
|
||
|
||
Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`, each at
|
||
`-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch,
|
||
so it is what would catch one of these `Vec`s being used after the arena under it was released.
|
||
|
||
## Function values, with no capture, and why that was the whole blocker
|
||
|
||
`map`, `filter`, `reduce` and `sort-by` could not be written, and the previous lane's sharpening of the reason was
|
||
right: **function values, not generics**. Generics alone would not have fixed it — without something to pass there is
|
||
nothing to be generic over — and function values alone did fix it, which is the evidence. The prelude gained all four
|
||
the same day, without generics, and is still one copy per element type, which is the half generics would remove.
|
||
|
||
### The shape, and why it was not invented here
|
||
|
||
The compiler has built and called function values internally since the Map landed. A `handler-bind` clause is lowered
|
||
to a function of its own, its address goes into a `flan_handler`, and the runtime calls it back through
|
||
`h->fn(condition, xfer)`; a Map's hash and equality pair is the same arrangement, reached as Odin reaches
|
||
`Map_Info`'s two contextless `proc` fields. **The surface feature is that machinery given a name**, not a second one
|
||
beside it. `check_fn` is `check_handler_bind`'s clause lifting with the parameters coming from the type instead of
|
||
from the condition, and `emit`'s indirect call is the callee expression handed to the same `call_through` a direct
|
||
call already went through.
|
||
|
||
### A bare name is the function
|
||
|
||
```
|
||
(map double xs)
|
||
```
|
||
|
||
and not Common Lisp's `#'double`. **This is a Lisp-1 — one top-level namespace, enforced, so a `defn` and a `defvar`
|
||
cannot share a name** — which is exactly what makes the bare name safe to read: there is no second binding of
|
||
`double` it could have meant instead, so the sharp quote would be punctuation answering a question the language does
|
||
not ask.
|
||
|
||
A `Types.Fn` is one pointer. There is no environment beside it, so the type resolves to `ptr` and lays out as eight
|
||
bytes, and a call through one is byte-for-byte the call a name would have produced — a Flan function's emitted
|
||
signature is its parameters followed by the transfer channel whether it was reached by name or by pointer. That is
|
||
why a handler established across a `fold` still catches a signal raised by the function the fold was handed:
|
||
`programs/fn-values.flan` does exactly that, and it is the case that would fail if an indirect call skipped the
|
||
guard.
|
||
|
||
### `fn` literals take their types from the position
|
||
|
||
`Ast.Fn` carries parameter *names* and no types — that is the surface syntax, not an omission — so an `fn` is
|
||
checkable exactly where something says what is wanted. An argument position does, because `named_call` already
|
||
threads the callee's parameter type into each argument; a bare `(let [f (fn [x] x)])` does not, and is refused saying
|
||
so (`programs/fn-no-type.flan`). A name already written as a `defn` goes anywhere, because it carries its own
|
||
signature.
|
||
|
||
### What was built, and what was refused by name
|
||
|
||
**Built:** a written `(Fn [T ...] R)` annotation; a `defn`'s name in value position; an `fn` literal; a call through
|
||
a value, both by the name it is bound to and through a computed head; returning one. Four refusal sites, all four
|
||
implemented.
|
||
|
||
**Refused, each with its own reason and its own program:**
|
||
|
||
- **Capture does not exist** (`fn-capture.flan`). An `fn` is lifted into a function of its own and handed nothing but
|
||
its parameters; a reference to a local of the enclosing function is refused by name. This is the same refusal a
|
||
handler clause has always carried, and the two now share one message with the construct's name in it.
|
||
`spec-memory.md`'s capture cases, and **escaping closures with them, stay deferred** — deliberately, and this is
|
||
what keeps a function value a bare code address that cannot outlive anything.
|
||
- **An `fn` with nothing to say what it takes** (`fn-no-type.flan`), above.
|
||
- **A position that would zero one** (`fn-in-struct.flan`): a struct field, a global, a fixed array's element,
|
||
`(zeroed)`. ZII fills an omitted field with all-bytes-zero, and **a zeroed function value is a null pointer, which
|
||
is the one kind of zero that is not a value the type can have** — every other type's zero is one: `0`, `false`, an
|
||
empty slice, `None`, a union's first case. A parameter, a return type and a `let` binding are not on the list
|
||
because none of them is ever conjured, and an `(Option (Fn ...))` is not either, because a `None`'s tag is what
|
||
nobody may look past. Nor are a `(Vec (Fn ...))` or a `Map` with function values: the Vec runtime never zeroes
|
||
past its length and `flan_map_alloc` zeroes only the hash run, so neither conjures an element nobody pushed or
|
||
put. A function value as a map *key* is refused already, by `Types.keyable` — hashing an address is a different
|
||
operation from hashing what it points at.
|
||
- **A foreign function's address** (`fn-extern.flan`). A Flan function's signature ends with the transfer channel and
|
||
a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the raw symbol knows
|
||
nothing about. Wrap it in a `defn` and pass that.
|
||
|
||
### `Fnval`, and the one thing a dev build cannot do
|
||
|
||
`Tast.FnAddr` had two `fnref` cases and now has three. `Flanfn` and `Rtfn` are the compiler's own uses and want the
|
||
*symbol*, always — a lifted handler clause and a hash pair have no indirection cell to load from. **`Fnval` is a
|
||
function value someone wrote, and in a dev build it is the cell's contents rather than the symbol**, so a value taken
|
||
after a redefinition is the new body. Splitting the case rather than overloading `Flanfn` is what keeps that true
|
||
without breaking the two paths that must not take it.
|
||
|
||
What that does *not* give: a value taken *before* a redefinition and called after it is still the old body. Once the
|
||
address is in a slot there is nothing left to re-resolve, and the honest fix is a trampoline per function, which is a
|
||
cost every program would pay for a case no one has hit. Named here rather than papered over.
|
||
|
||
The two lifted-function name sequences are counted **per kind** — `fn/OWNER/N` and `handler/OWNER/N/TYPE` —
|
||
rather than off one list. Sharing a counter would rename every `fn` in a function the moment a `handler-bind` was
|
||
added above one, which is a rename for a body that did not change, in exactly the names a redefinition module emits.
|
||
|
||
`Tast.CallPtr` is its own node for the same kind of reason. Everything that walks this IR treats `Call`'s string as a
|
||
*link-time* edge — `Reach` roots the callee, `Dev` finds the cell, `Emit` may load it — and none of those are
|
||
questions an indirect call can answer. `Reach` gains the `Fnval` edge, and that edge is load-bearing: a name used as
|
||
a value is never a `Call`, so without it the one function a program passes to `map` is the one function the link
|
||
drops.
|
||
|
||
### A user-written allocator: still refused, and now for two different reasons
|
||
|
||
NEXT.md said it needed "a defn's name in value position". **It has that now, and it is still two things short**,
|
||
neither of them a function-value question:
|
||
|
||
1. The runtime calls `a->proc(a, mode, p, old_size, size, align)` — six C arguments and no transfer channel — and
|
||
every Flan function value's signature ends with one. It is the same mismatch a foreign function's address is
|
||
refused for, pointing the other way.
|
||
2. `Allocator` is opaque and pointer-width, so there is nowhere for a program to put the `flan_allocator` that
|
||
pointer would have to point at.
|
||
|
||
The refusal message says both, and `programs/user-allocator.flan` is the row that holds it. `(arena-new ...)` over a
|
||
backing buffer remains the parameterised allocator that does exist.
|
||
|
||
### The prelude's four
|
||
|
||
`map-i32!`/`map-f32!`, `filter-i32`/`filter-f32`, `reduce-i32`/`reduce-f32` and `sort-i32-by!`/`sort-f32-by!`. Two
|
||
rules, both inherited rather than invented: the in-place ones write back into the slice they were handed, because a
|
||
slice is non-owning and transforming a thing you already own should not allocate; and `filter` allocates and the
|
||
caller frees, like everything in the building tier.
|
||
|
||
**A `map` that changes the element type is the one shape that did not come with them** — it is one copy per *ordered
|
||
pair* of types rather than per type, which is where a per-type family stops being honest. That entry is what is left
|
||
in `prelude.ml`'s refusal block where `map, filter, reduce, sort-by` used to be, and its reason is generics.
|
||
|
||
## `map-next!`, the one thing a Map could not do
|
||
|
||
`flan_map_len`, `_get`, `_put`, `_has`, `_clone`, `_reserve` and `_free` was the runtime's entire map surface, and
|
||
every one of them addresses a *single* entry by hashing it. Nothing walked the block, so a map's keys and its values
|
||
could not be read out at all — the only item on the second tier's list that was blocked on nothing but a missing
|
||
function.
|
||
|
||
`flan_map_next` is that function and `map-next!` is the builtin over it.
|
||
|
||
```
|
||
(let [cur (i64 0) k 0 v 0]
|
||
(while (map-next! m (addr cur) (addr k) (addr v))
|
||
...))
|
||
```
|
||
|
||
**The cursor is a slot index the caller owns, and there is no iterator struct** because there is nothing for one to
|
||
hold. A map has no tombstones — removal shifts the run back instead of marking a hole — so a slot is either empty or
|
||
occupied and the position is the whole of the state. What a cursor does *not* survive is a removal taken while it is
|
||
in flight: the shift moves entries to lower slots, and a cursor already past them steps over entries it has not
|
||
answered, the same bargain a put that grows already makes. The cursor starts at 0, comes back one past the entry just answered, and is left
|
||
at `cap` by the call that answers false, so a spent cursor keeps answering false rather than wrapping.
|
||
|
||
**Three out-pointers and not a returned pair**, because there are no tuples. An `(Option K)` would answer half an
|
||
entry and make the value cost a second hash of the key just handed back. The `!` is the cursor: it is the argument
|
||
that is written through on the way out.
|
||
|
||
**It is the one map entry point that carries neither a hash nor an equality function.** Walking asks nothing about a
|
||
key. The two sizes are still there, because the runtime is type-erased and the block geometry is computed from them.
|
||
|
||
**The layout, restated, because it is the thing to get wrong here.** `data` is *one* allocation laid out
|
||
keys | values | hashes | scratch, each run cell-packed to a cache line — the arrangement the Valgrind lane described
|
||
while explaining why a probe overrun is not observable. A key is reached through `flan_cell_at` and never as
|
||
`ks + i * ksize`. The hashes are the exception `flan_map_clone` already relies on: an 8-byte element packs 8 to a
|
||
64-byte cell with nothing left over, so `g.hs[i]` is the right index and a flat one.
|
||
|
||
**Order is block order**, which is the hash's order and not the insertion's, and it changes when the map grows.
|
||
`programs/map-iter.flan` is therefore written entirely in sums, counts and lengths — every claim in it is order-free,
|
||
which is the contract rather than a weakness of the test. A caller that wants an order sorts what it collected. The
|
||
cases that would catch a real mistake: a string key with a struct value, whose two runs have different element sizes
|
||
and different packing and so would break if one geometry were used for both; and a 500-entry map, which is several
|
||
grows past the minimum and walks a block whose layout has nothing to do with how the entries went in.
|
||
|
||
**`map-keys` and `map-values` are still refused, and the reason changed.** The refusal block at the foot of
|
||
`prelude.ml` said "a Map iterator"; that is wrong now. What a prelude `defn` cannot write is
|
||
`(defn map-keys [m {K V}] (Vec K))` — it has to name its types and there is no `K`. That is generics. The loop is
|
||
three lines at the call site, where `K` is known, and that is where it stays.
|
||
|
||
## A prelude function may call a prelude macro, and why the fix was not ordering
|
||
|
||
The handoff said the prelude is never macro-expanded — `Macro.program` runs over the file being compiled and the
|
||
prelude arrives later through `Check.program`'s prepend — and that the fix was to move the prepend before expansion.
|
||
**Both halves of that are wrong, and the measurement is one command.**
|
||
|
||
Put `(clamp prec 0 9)` back into `format-f64`, print `names` and `List.length extra` on entry to `Macro.compile`, and
|
||
compile any program that calls a macro. The compiler prints `names=[clamp,unless] extra=0` and *then* the arity error
|
||
at `<prelude>:1103`. So `compile` was entered: the prelude does reach the expander. The error is raised by the
|
||
`Check.program` **inside** `compile`, where `building` is true and expansion is off.
|
||
|
||
That is the real shape, and it is a **cycle, not an ordering**: a macro module is compiled *from* the prelude, so a
|
||
prelude function that calls a macro would have to be compiled into the very module that expands it. Moving the
|
||
prepend earlier changes which pass sees the prelude first and leaves the cycle exactly where it was.
|
||
|
||
Two things break it, and the second is the one that matters.
|
||
|
||
**The prelude's macros are dropped from `mine`.** `Macro.program` collected every `defmacro` in the forms it was given
|
||
and handed them back as `extra` — the forms a macro module is built *in addition to* the prelude. When the forms it
|
||
was given *are* the prelude, that is the prelude's macros declared twice, refused as a redefinition. They are already
|
||
in `prelude`, which is where the module gets them from.
|
||
|
||
**`Macro.reduce`: for that one build, the prelude is smaller.** Every `defn` that names a macro is dropped, and then
|
||
every `defn` that names a dropped one, to a fixpoint — a function calling something unbuildable is as unbuildable as
|
||
the thing it calls. `Prelude.bootstrap` is the hook, a ref rather than a parameter because the reader is
|
||
`Check.program` and it cannot be told.
|
||
|
||
Only `defn`s are dropped: the functions that survive still mention the prelude's types, and a reduced prelude missing
|
||
them would not check. A `defstruct`, `defdata`, `defalias`, `defenum` or `defvar` therefore stays whatever it names.
|
||
There used to be a sharper reason — `Parse.prelude_types` memoised the prelude's type names for the parser's
|
||
return-type guess, and it can be forced for the first time inside a bootstrap build, so a reduced set cached there
|
||
would have been wrong for every compile afterwards. That set is gone with the guess; see *The return type is the slot*
|
||
below.
|
||
|
||
**The one restriction that stays, and now names itself.** A prelude macro may not call a macro — the module that
|
||
expands it is compiled from the prelude, so there is no earlier module for its own call to have been expanded by.
|
||
That was already recorded and accepted; what it used to do was fail as an unknown name somewhere inside a clang
|
||
driver. `reduce` checks it directly and refuses with the macro's name and the reason.
|
||
|
||
`format-f64` is written `(clamp prec 0 9)` now, which is the living proof and also the only place in the prelude that
|
||
exercises it. The expansion is `(min 9 (max 0 prec))`, so nothing about the output moved — the point is that the call
|
||
compiles at all.
|
||
|
||
**What it costs.** The prelude names a macro now, so `Macro.program`'s short-circuit — the reason a build using no
|
||
macro pays nothing — no longer fires for the prelude, and every `Check.program` dlopens a macro module. The module is
|
||
disk-cached under a digest of the prelude source with an empty `extra`, so it is one `.so` shared by every build and
|
||
every process; `dune test` is unchanged at 20 seconds. The first build after a prelude edit pays one clang driver.
|
||
|
||
**What was not done.** The two expansions are still separate — the prelude is expanded against the prelude's macros,
|
||
the file against the prelude's plus its own. That is not a gap, and it is worth stating so the next lane does not
|
||
"fix" it: a file macro is never visible to the prelude, and a prelude macro is already visible to the file, so the two
|
||
passes cannot disagree. Expanding them together would buy one fewer `dlopen` and nothing else.
|
||
|
||
## `defer` may be written in a `let`
|
||
|
||
The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both
|
||
considered and rejected before it.
|
||
|
||
`defer` is a **compile-time** construct: the cleanup is copied into every exit path of the function. That is why a loop
|
||
body and a branch stay refused — a loop body's would fire once at function exit rather than once per iteration, and a
|
||
branch would have to express "maybe registered", which a form copied into every exit path or into none cannot say.
|
||
|
||
A `let` is neither. It is not a frame here: its bindings are function slots like any other and **nothing is released at
|
||
scope exit**, so a `let` at the top level of a function body has exactly the function's extent and a `defer` written in
|
||
it always registers. It was refused for a reason that does not apply to it.
|
||
|
||
The rule, stated precisely, because "top level of a function body" is easy to get wrong: a form at the top level of the
|
||
body may carry one, and so may a form in the body of a `let` that itself may — to any depth. A `let` inside a `while`
|
||
or an `if` has the loop's extent or the arm's, and inherits the refusal rather than the permission.
|
||
|
||
The implementation detail worth keeping: **the permission is granted again before every form of a body, never once
|
||
around the body.** `check` withdraws it as it starts, so granting it once would let the first `defer` through and refuse
|
||
the second — and two resources acquired in one `let` is the case this exists for. `defer-let.flan` covers exactly that,
|
||
along with nesting, interleaved registration order across the `let` boundary, and an early return.
|
||
|
||
This **amends `spec-memory.md`**, which states under "When storage is released" that `(defer (free v))` for a
|
||
`let`-bound `v` is "not expressible today" and that no idiom in the spec may depend on it. It is expressible now.
|
||
|
||
`do` at the top level of a body has the same extent argument and is deliberately **not** included: nothing asked for it,
|
||
and the rule is easier to state and to trust with one construct in it.
|
||
|
||
## Assets are baked in, and the reason it is a compiler feature
|
||
|
||
NEXT.md decision 1. `(embed "brush.png")` is a `[u8]`, `(embed "brush.png" string)` is a `string`, and
|
||
`(embed-dir "assets")` is a `[n EmbedFile]` sorted by name. Odin's `#load` and `#load_directory` are the model
|
||
(`src/parser.cpp`, and `check_load_directive` / `check_load_directory_directive` in `src/check_builtin.cpp`); Odin's
|
||
`#` is not imported, because an s-expression language already has a head position for a name and these resolve as
|
||
ordinary named calls exactly the way `vec-new` and `heap-allocator` do.
|
||
|
||
**The reason for this shape rather than a build flag is the one that decided it.** It is a *compiler* feature, so it
|
||
needs no linker arguments and no per-target packaging, and it works identically on desktop and web. That matters more
|
||
here than it does for Odin, because `Load` hands out `lflags` only to a directory package and `main` is not exported,
|
||
so a program can never be a package: the single file doing `(rl/load-texture "brush.png")` is structurally the one file
|
||
with **no link channel at all**. The web lane found that hole and did not invent a flag for it. Embedding has no such
|
||
hole, because there is nothing to tell the linker.
|
||
|
||
**It costs nothing at run time.** The bytes reach the program as a `Tast.Str` node typed `[u8]`, which emit.ml turns
|
||
into the same `private unnamed_addr constant` every string literal already becomes, and its `escape` is byte-exact
|
||
across the whole 0–255 range, so a PNG survives the round trip through the `.ll`. Bound with `defconst` at top level an
|
||
`embed-dir` is an LLVM constant outright, through emit.ml's `const`.
|
||
|
||
**A `Str` node typed `[u8]`, not a `Bytes` prim over a `string`.** This is the one non-obvious choice. `Bytes` is
|
||
identity — emit.ml lowers `Types.String` and `Types.Slice _` to the same `%slice` — but wrapping the literal in a prim
|
||
makes the node non-constant, and `const` then refuses an `embed-dir` in a `defconst` with *a global's value must be a
|
||
compile-time constant*. Both of emit.ml's string emitters take the bytes and ignore the node's type, so it is the same
|
||
constant either way and this one is a constant a global can hold.
|
||
|
||
**Two spellings, not one form that changes type with its context.** Odin threads a `type_hint` everywhere and can
|
||
afford `#load("p")` to mean a `string` here and a `[]u8` there. With structural equality, no implicit widening and no
|
||
coercion anywhere, the same text meaning two types would be a wart, so `string` is written down when it is wanted. The
|
||
site's expectation is a fallback only and nothing depends on it.
|
||
|
||
**The path is a literal and resolves relative to the file the form is written in.** Both are Odin's rules and for
|
||
Odin's reasons: the bytes must be in hand before any value exists, which is what makes the result free; and a path
|
||
relative to the compiler's working directory would make a package's assets depend on where `flan` was invoked from,
|
||
which cannot be right. A missing file is a compile error naming it, never an empty embed — an asset silently absent is
|
||
exactly the quiet wrongness this removes. An empty *directory* is not that case and embeds cleanly as `[0 EmbedFile]`.
|
||
|
||
**The directory lookup is a linear scan, and that is the chosen answer rather than the fallback one.** `embed-find` is
|
||
an ordinary prelude function over a `[EmbedFile]`. A directory embed is tens of entries whose names sit in cache-warm
|
||
`.rodata`; a compile-time perfect hash would be a build-time map with its own failure modes that nothing has asked for,
|
||
and sort-and-bisect is the next step if a program ever embeds thousands of files — it would not change the type. It
|
||
takes a **slice** rather than the array, because an array's length is part of its type and there are no generics, so
|
||
the call reads `(embed-find (slice assets 0 (len assets)) "brush.png")`. Entries are sorted by name because `readdir`
|
||
order is filesystem-dependent and an unsorted embed would make two builds of identical sources emit different `.ll`.
|
||
Non-recursive, files only — Odin again.
|
||
|
||
**The sharp edge, inherited and not widened.** The slice points into `.rodata`, so a store through it segfaults at
|
||
`-O0` and is deleted as undefined behaviour at `-O2` — the same trap the prelude's ASCII-case note measures for
|
||
`(bytes "Hi")`, and the same one NEXT.md tracks as "writing through a string literal". Nothing here makes it worse and
|
||
nothing here fixes it; provenance is what would. **To get a mutable copy, clone the bytes into a `Vec`.** It is worth
|
||
saying loudly because an embedded asset is precisely the thing someone will try to decode in place.
|
||
|
||
**What this does not do.** `sand.flan` still calls `(rl/load-texture "brush.png")`, which hands raylib a path for
|
||
raylib to open. Pointing raylib at embedded bytes needs `LoadImageFromMemory` and `LoadTextureFromImage` in place of
|
||
`LoadTexture` — a raylib binding question, not an embedding one — so the flagship program is not yet asset-free on the
|
||
web. The mechanism it needs is in.
|
||
|
||
*(Superseded by the cut. `sand.flan` was taken back from 765 lines to 206, to parity with the Clojure, Common Lisp and
|
||
jank ports, and the texture went with it — there is no `load-texture` call and no asset left to embed, only a
|
||
`brush-size` integer that kept the name. The binding question is still open for whatever wants it next; this program is
|
||
no longer the one asking it, and `brush.png` is now unreferenced.)*
|
||
|
||
## `slurp`, `barf`, and the two ways they fail
|
||
|
||
NEXT.md decisions 2 and 5. `(slurp path)` and `(slurp path allocator)` read a whole file into a `(Vec u8)`;
|
||
`(barf path bytes)` writes one.
|
||
|
||
**`slurp` waited for `Vec` because its result has no length until the file is read**, and it obeys spec-memory.md's
|
||
rule without an exception: *no allocating operation returns an error*. There is no `Result` here, no out-parameter and
|
||
no error code — `slurp`'s type is `(Vec u8)` and `barf`'s is `()`.
|
||
|
||
**Two failures, two conditions, and the guards nest rather than merge.** Allocation failure is `StorageExhausted` under
|
||
`retry`, unchanged and reused. File failure is `FileError {.path .op .reason}` under `retry` and `use-value`. They stay
|
||
apart because they ask two different answerable questions: the handler that grows an arena is not the handler that
|
||
supplies another path, and collapsing them would make one handler guess which it was looking at. `file_guard` in
|
||
check.ml is `alloc_guard`'s shape built from the same nodes — a `while`, a `restart-case` and an `error` — so the
|
||
backend learns nothing new.
|
||
|
||
**The restarts are Common Lisp's pair for a `file-error`.** `retry` for "the file may be there now"; `use-value [p
|
||
string]` for "try this other path". `use-value` is the first restart clause the **compiler itself** emits with a
|
||
parameter — typed restarts landed the same session — and its parameter *is* the path slot the attempt reads, so the
|
||
clause body is empty. emit.ml's `bind_params` stores the invoker's argument into the slot, the clause falls through,
|
||
and the loop re-attempts against the new path. Everything is inside that loop, so a `use-value` naming a different file
|
||
re-measures it and re-allocates for *its* size; the `Vec` is freed at the top of each turn, which is why a retry does
|
||
not leak, and freeing a `Vec` that never allocated is a no-op.
|
||
|
||
**The two forms resolve paths by opposite rules, and it is worth saying in one place.** An `embed` path is resolved at
|
||
*compile* time relative to the file the form is written in. A `slurp` or `barf` path is resolved at *run* time by the
|
||
host, against the process's working directory — these are ordinary values, and one can arrive from `argv` or from a
|
||
`use-value` restart. `test/programs/slurp.flan` reads `"programs/assets/a.txt"` only because the suite runs from
|
||
`_build/default/test`. Two forms in one section with opposite rules is exactly where someone gets bitten.
|
||
|
||
**The break loop can be offered this restart and cannot fill it in.** That is not new behaviour, only a new way to
|
||
reach it: a break loop chooses a restart by position and has nothing to supply a parameter with, and emit.ml already
|
||
emits a `flan_restart_unarmed` guard on every clause that takes one, so taking it refuses with the reason rather than
|
||
running the clause on a zeroed buffer. `slurp`'s `use-value` is simply the first such clause the *compiler* emits —
|
||
`alloc_guard`'s `retry` takes no parameters — and it rides the same path a hand-written one does. The acceptance suite
|
||
asserts the guard is on the emitted IR for both.
|
||
|
||
**On the web, `barf` signals — every time, with the path in the condition.** This is the decision worth restating,
|
||
because two more obvious answers are both wrong here. A **build-time refusal** is unusable: Flan has *no conditional
|
||
compilation*, nothing in `parse.ml` or `check.ml` reads the target, so "isolate this to desktop" is not expressible in
|
||
source and the refusal would have nowhere to be silenced. A **silent no-op** is worse than either, because that is how
|
||
a save file disappears with nothing said. So the program gets a condition and decides — which is the language having
|
||
something Odin does not. Odin stubs its whole file API on js/wasm to `.Unsupported` (`core/os/file_js.odin`) so that
|
||
importing `core:os` "panics cleanly", and a panic is not a decision. **The restriction was taken; the mechanism was
|
||
not.**
|
||
|
||
Nothing in the compiler reads the target to do this. The refusal is one `#ifdef __EMSCRIPTEN__` in `flan_rt.c`, which
|
||
is where the host ABI is *already* implemented twice. `slurp` keeps working on the web — it compiles, runs, and reports
|
||
a missing file honestly — and the bytes a web program actually wants come from an `embed`.
|
||
|
||
`test/programs/web-files.flan` is one source built for both targets, and `test/test_web.ml` **runs** it under node
|
||
rather than asserting the artifact's shape: an artifact-shape assertion would say nothing about the thing the decision
|
||
bought. The test checks that the refusal is printed with its path and reason, and that the desktop's success line is
|
||
*absent* — a silent no-op would have taken that branch.
|
||
|
||
### What the host ABI grew by, and why that much
|
||
|
||
plan.org names the filesystem as the #1 portability risk — "pack assets, one abstraction, never touch paths" — so the
|
||
widening is written down rather than assumed. It is **three calls and one reader**:
|
||
|
||
| Call | What it does |
|
||
|------|--------------|
|
||
| `flan_file_size(path, n, &out)` | how many bytes are there |
|
||
| `flan_file_read(path, n, buf, cap, &got)` | fill a buffer the caller owns |
|
||
| `flan_file_write(path, n, buf, len)` | write a whole file |
|
||
| `flan_file_fail_reason()` | which of the four reasons it was |
|
||
|
||
They are POSIX-shaped and **Vec-ignorant**: no handle crosses the boundary, nothing is held between calls, and each
|
||
takes a path and answers 1/0 the way every allocator entry point already does. `flan_slurp_into` — the part that knows
|
||
what a `Vec` is — is runtime *glue* on this side of the ABI, not a fourth host call, so a second target implements
|
||
three functions and inherits the rest. The reason is a global rather than an out-parameter for the same reason
|
||
`flan_alloc_fail_bytes` is: the condition is a value struct on the failing frame's stack with fixed numeric fields and
|
||
no rendered message.
|
||
|
||
**These do touch paths, which is the widening plan.org warned about**, and decision 2 took it knowingly. `embed` is the
|
||
half that does not: it needs no host ABI at all, so "pack assets" remains the answer for anything known at build time
|
||
and `slurp` is for bytes that genuinely are not.
|
||
|
||
`FileError` is one type with a `reason` field rather than a family, because conditions have no hierarchy today
|
||
(spec-conditions.md §1) and a family would need one handler clause per member to say "any file error". NEXT.md decision
|
||
4's parent link is the answer to that and is not built; when it is, these reasons can become types without any call
|
||
site changing.
|
||
|
||
**One thing `slurp` and `barf` reveal that is not theirs to fix.** A handler that wants "try to save, and carry on if
|
||
you cannot" has nowhere to go: `error` is diverging (spec-conditions.md §2), a handler returning normally has not
|
||
answered it, and neither `retry` nor `use-value` means *give up*. `web-files.flan` calls `exit` for that reason. A
|
||
`continue`-style restart — or `handler-case`, which unwinds — is what the case wants, and neither exists.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
|
||
| link | 20ms |
|
||
|
||
Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which
|
||
compiles to a `.o` under `$XDG_CACHE_HOME/flan/objcache` (`~/.cache/flan/objcache`, or `$FLAN_CACHE_DIR`) and
|
||
reuses it. It used to sit under `$TMPDIR`, which meant dune — which gives every run a private `TMPDIR` — never reused
|
||
an object and every build in the test suite was cold. The key is a digest of the source text, the compiler
|
||
(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build),
|
||
`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0`
|
||
and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d
|
||
into place, so two concurrent builds cannot see a half-written one.
|
||
|
||
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
|
||
the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever
|
||
a disk-space decision.
|
||
|
||
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It
|
||
is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted
|
||
IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse
|
||
errors.
|
||
|
||
|
||
## The order it was built in, and why that order
|
||
|
||
Every item here is done; it is kept because the *sequencing* is the part worth remembering. Decided in conversation:
|
||
wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and **the dev loop is the
|
||
thesis of the project**, so it came first. Staged so each step was runnable on its own — the failure mode being to
|
||
build a daemon and a protocol before knowing whether the reload primitive worked.
|
||
|
||
1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms.
|
||
See the section above.
|
||
2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`,
|
||
`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a
|
||
run-time-added function is itself redefined. See the section above.
|
||
3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs,
|
||
and sand.flan polling at the top of its frame. See the section above.
|
||
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than
|
||
nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if
|
||
something else ever needs to talk to it.
|
||
|
||
**One decision left to settle before step 2**, because both change codegen and are painful to retrofit:
|
||
|
||
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module
|
||
declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep
|
||
the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload,
|
||
and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
|
||
- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what
|
||
`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from.
|
||
|
||
Deferred until after the dev loop:
|
||
|
||
6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and
|
||
`test/programs/sand-headless.flan` prints `15595743031174623232` under it — the same hash as native, byte for byte, at
|
||
`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared
|
||
across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer
|
||
would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and
|
||
runs it — rather than by looking for a binary on PATH.
|
||
|
||
Three things this cost that were not in the old note:
|
||
|
||
- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main`
|
||
to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a
|
||
signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")`
|
||
label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite
|
||
self-call.
|
||
- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes `<stdio.h>`; without `--sysroot`
|
||
it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the
|
||
triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`.
|
||
- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both
|
||
shapes count.
|
||
|
||
**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to
|
||
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
|
||
`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple
|
||
(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk
|
||
is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's
|
||
beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
|
||
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
|
||
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in.
|
||
|
||
**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of
|
||
`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes
|
||
an `ExperimentalWarning` to stderr on every run and the harness compares combined output.
|
||
|
||
**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with
|
||
one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime
|
||
at it).
|
||
|
||
~~Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot.~~ **Done — and it is a third
|
||
target, not a mode of this one.** wasi-sdk is right for the headless table and was never going to be right for the game
|
||
build. See the next section.
|
||
|
||
## The browser is the third target
|
||
|
||
`flan build --target=web` produces a page, its JS and a `.wasm`, and a raylib example opens in a browser from source
|
||
that was not touched. The two wasm targets share the word and almost nothing else, so `is_wasi` and `is_web` are
|
||
separate predicates and `is_wasm` is their union — the union is exactly the set of facts about the *machine* (32-bit
|
||
pointers, no `dlopen`), which is what the refusals are about, and nothing else is shared.
|
||
|
||
**The compiler is `emcc`, not `clang`, and that is the whole of the sysroot story.** Everything the wasi target has to
|
||
find by hand — a sysroot, a builtins archive, a shadow resource directory, the `__main_argc_argv` shim — is what emcc
|
||
*is*. `target_flags` for `web` is the empty list; the only thing checked is that emcc exists, refused by name where the
|
||
reason can say so. The one fact that had to be true for any of this: **emcc takes a `.ll` on its command line**, which
|
||
it does, so `Emit`'s output needs no change and the IR stays target-independent. The object cache keys on the compiler
|
||
binary's path, size and mtime as it always did — now of *whichever* compiler the target uses, so an emcc `flan_rt.o`
|
||
and a clang one cannot collide.
|
||
|
||
**The main loop: `-sASYNCIFY`, not `emscripten_set_main_loop`.** The older note above had the browser fact right —
|
||
it cannot be blocked — and drew the wrong conclusion from it. `emscripten_set_main_loop` wants the loop body as a
|
||
callback, so every one of the eleven examples that writes
|
||
|
||
```lisp
|
||
(until (rl/window-should-close?) ...)
|
||
```
|
||
|
||
would have to be split by hand into an init and a tick, and the web program would stop being the native program.
|
||
Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is
|
||
built for precisely that: `WindowShouldClose()` on `PLATFORM_WEB` is an `emscripten_sleep(16)` that then returns false
|
||
(raylib 5.5, `platforms/rcore_web.c`, read rather than assumed). So the loop yields once a frame at a call it already
|
||
makes, and **no example changed a character**. The price is real and is paid by every web build: asyncify instruments
|
||
the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a
|
||
question `Build` can answer, and a flag set that varies per program is a cache key that varies per program.
|
||
|
||
**`link` lines can be addressed to a target.** `vendor/raylib/link` named `libraylib.so.550`, which exists on the host
|
||
and nowhere else. A line may now carry `@native`, `@wasi` or `@web`, an untagged line applies everywhere — which is
|
||
what every existing `link` file already is — and `${NAME}` expands from the environment. The selection happens in
|
||
`Build` and not in `Load`, which is where the file is read, because **`Load` resolves imports before a target is
|
||
chosen**: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings.
|
||
`Load`'s part in this is to pass the lines through untouched, which it already did.
|
||
|
||
**raylib for the browser is built, not installed.** No emscripten port provides it (`emcc --show-ports`: there is
|
||
`contrib.glfw3` and no raylib), so `vendor/raylib/build-web.sh` clones raylib at the **5.5** tag — the one whose
|
||
`.so.550` the host links, because `raylib.flan` carries raylib's struct layouts and enum values and two targets built
|
||
from different raylibs would disagree about them in silence — and compiles the seven modules with
|
||
`-DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2` into one archive under `vendor/raylib/web/` (gitignored). `rglfw.c` is not
|
||
among them: the web platform uses emscripten's own GLFW port, which is why `link` carries `@web -sUSE_GLFW=3`. No
|
||
headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.
|
||
|
||
**The HTML shell is a string in `Build`, not a file in the tree**, for the same reason `Runtime_src` is: it has to be
|
||
wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas
|
||
and a `Module.print` that puts stdout on the page; `FLAN_WEB_SHELL` replaces it. `--shell-file` is passed only when the
|
||
output is a `.html`, because emcc accepts and ignores it otherwise.
|
||
|
||
**Refused by name, inherited whole:** `--dev`, `--debug`, `Build.shared` and `flan run --target=` are refused for
|
||
`web` exactly as for `wasm32`, each naming `web` rather than `wasm32` in the message. `--sanitize` is refused too, but
|
||
the web half of that refusal is weaker than the wasi half and says so: emscripten *does* ship an ASan, and nothing here
|
||
has ever run it. A sanitizer that has never been run is one whose silence means nothing.
|
||
|
||
**What the test can honestly check.** `test/test_web.ml` is headless and permanently so. It probes — emscripten may not
|
||
be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it
|
||
asserts: the three files exist, the module starts with `\0asm`, the page references its own JS and carries the canvas,
|
||
and node runs the emitted JS and gets `ok`. For raylib it builds `core-basic-window.flan` unchanged and then reads the
|
||
module for the two things that would be false if the mechanism were wrong: an `asyncify_start_unwind` export, and a
|
||
`glViewport` import that can only have come from raylib's web platform. Import and export names are plain strings in
|
||
the binary, so this needs no wasm reader.
|
||
|
||
## sand.flan in a browser
|
||
|
||
**The flagship program builds for the browser and the artifact opens.** Three things were between it and the target,
|
||
and none of them was the `#include` the earlier note named.
|
||
|
||
**The brush was a path, and a path is what cannot work.** `(rl/load-texture "brush.png")` hands raylib a filename to
|
||
open; a bare relative path has no meaning where there is no filesystem, so raylib would have opened nothing and the
|
||
cursor would simply have been missing. It is `(embed "brush.png")` now, decoded by a new binding —
|
||
`LoadImageFromMemory`, which completes the chain embedded bytes -> `Image` -> `Texture2D` that `LoadTextureFromImage`
|
||
already had the other half of. The declaration is `(Ptr u8)` plus an explicit `i32` count, because `shim.ml` refuses a
|
||
slice parameter and its refusal says exactly that; the Flan wrapper beside it takes the slice apart, which is the
|
||
idiom `collision-point-poly?` and `load-font-ex` already established. One decode serves both textures now: the
|
||
unflipped upload first, then `ImageFlipHorizontal` in place, then the mirrored upload — **the order is load-bearing**,
|
||
and if the two badges look the same it was swapped.
|
||
|
||
`load-texture` and `load-image` lose their only call site in this repository by this change. That is deliberate rather
|
||
than an accident of editing: a path-based load is the one shape the browser cannot have, and the bindings stay for the
|
||
desktop programs that will want them.
|
||
|
||
The embedded path resolves **relative to the file the form is written in**, which is why
|
||
`test/programs/sand-headless.flan` still works: it reaches `sand.flan` through `../../` out of a sandboxed `_build`,
|
||
and the PNG is found beside `sand.flan` and not beside the working directory. `test/dune` therefore lists `brush.png`
|
||
as a dependency of every stanza that builds sand — an embed is read by the *checker*, so it is a build input and not a
|
||
run-time one. The same fact reached `test_session`'s `C-c C-k` case, which re-evaluates sand.flan's whole text: it now
|
||
passes `~origin`, which is the buffer path both editor paths already send, because the default `<eval>` origin would
|
||
resolve the embed against the working directory instead.
|
||
|
||
**A package's C may be addressed to one target, the way a `link` line already could.** `Load` collects a package's
|
||
`.c` files by listing the directory, and there is nowhere in a directory listing to put a tag except the name, so the
|
||
tag goes there, before the extension:
|
||
|
||
```
|
||
vendor/agent/flan_agent.c compiled everywhere, unless displaced
|
||
vendor/agent/flan_agent.web.c compiled for the browser, and displaces the above
|
||
```
|
||
|
||
One rule: **a tagged file is compiled only on its own target, and there it replaces the untagged file of the same base
|
||
name.** Untagged is the default and every existing package is untagged, so nothing that did not opt in changed.
|
||
Replacement rather than the pure tagging Go's `_windows.go` and Odin's `file_js.odin` use, and the difference is the
|
||
point — pure tagging would mean renaming `flan_agent.c` to `flan_agent.native.c` to teach the package about a target
|
||
it had never heard of, and this way a package gains a target by gaining a file. The selection is in `Build` and not in
|
||
`Load`, for the reason `select_lflags` gives: `Load` resolves imports before a target is chosen.
|
||
|
||
**The dev agent on the web is a no-op, and it is not the `barf` decision being contradicted.** The compile error that
|
||
led here — `struct timeval` incomplete, because emscripten's headers do not pull `<sys/time.h>` in transitively — is
|
||
the surface. Underneath: *the agent is a socket server and a browser has no sockets*. Adding the include produces an
|
||
agent that compiles, links, starts and can never accept a connection.
|
||
|
||
Refusing `vendor:agent` on a web target was the other candidate and is ruled out by arithmetic, not taste. Flan has no
|
||
conditional compilation, `sand.flan` calls `(agent/start ...)` unconditionally, and `Reach` cannot prune a package
|
||
something reachable calls into — so a build-time refusal means the flagship program does not build for the browser at
|
||
all without being edited into a second program. **A refusal is only honest when the caller has a way to not ask.**
|
||
|
||
The two decisions look contradictory and are not, and the difference is *what the caller loses*. `barf` is asked to
|
||
make something durable; a no-op returns success to a program that now believes the bytes are on disk, and the loss is
|
||
real, is the user's, and is discovered later or never. The agent is asked to accept redefinitions from an editor; on
|
||
the web there is no editor, no socket and no session — `--dev` is refused by name on every wasm target, so a web build
|
||
has no cells to install a redefinition into even if one arrived. **Nothing is lost because there was never anything
|
||
there.** `sand.flan` already says the same about a *native* release build, at the call site: "Building without `--dev`
|
||
is fine — nothing has cells to install into, so a module is refused on the listener thread and the loop never
|
||
notices." A web build reaches that outcome by a shorter route. `start` returns `-1`, which is what `flan_agent.c`
|
||
returns for a path it cannot bind; `poll` and `wait` return 0, which is what the native build returns on every frame
|
||
nothing arrived on. The whole argument is written at the top of `vendor/agent/flan_agent.web.c`, where the next reader
|
||
will meet it.
|
||
|
||
### Building it and opening it
|
||
|
||
The raylib archive is built once and is not in the tree. From the repository root:
|
||
|
||
```sh
|
||
sh vendor/raylib/build-web.sh # clones raylib 5.5 and compiles it with emcc
|
||
export FLAN_RAYLIB_WEB=$PWD/vendor/raylib/web/libraylib-5.5.a
|
||
```
|
||
|
||
`build-web.sh` prints that `export` line itself. Then:
|
||
|
||
```sh
|
||
flan build sand.flan --target=web -o sand.html
|
||
```
|
||
|
||
**The output must be named `.html`.** `--shell-file` is passed only when it is, because emcc accepts and ignores it
|
||
otherwise — so `-o sand` produces a module with no shell, no canvas, and a page that looks like it built fine and
|
||
paints nothing. Three files land beside it, in whatever directory `-o` names: `sand.html`, `sand.js`, `sand.wasm`.
|
||
|
||
**A `file://` URL will not work.** The page fetches `sand.wasm`, and a browser refuses that from the filesystem. Serve
|
||
the directory holding the three files:
|
||
|
||
```sh
|
||
python3 -m http.server 8000
|
||
```
|
||
|
||
and open **`http://localhost:8000/sand.html`**. Left mouse paints; the keys are the ones the native build has.
|
||
|
||
### What only a human opening it can settle
|
||
|
||
Verified headlessly, by `test/test_web.ml`: the three files exist, the module carries `asyncify_start_unwind` and
|
||
`glViewport`, and **brush.png's own bytes are in the module, whole** — the assertion that keeps the embed from
|
||
rotting. Not `IHDR`: stb_image, linked in from raylib, carries that string itself, so an `IHDR` check would pass on a
|
||
build where the embed emitted nothing.
|
||
|
||
Not verified, and not verifiable here. `node sand.js` instantiates the module, runs `main`, and dies inside `glfwInit`
|
||
on `window is not defined` — which says the module is live and says nothing about the canvas.
|
||
|
||
- **Whether it paints at all.** Nothing in CI has ever seen a pixel of this.
|
||
- **Audio.** `start-audio` generates a tone, `ExportWave`s it to `/tmp/flan-sand-tone.wav` and loads it back as a
|
||
music stream. That is raylib's own `fopen`, not Flan's `barf`, so the `#ifdef` in `flan_rt.c` does not cover it and
|
||
emscripten's MEMFS may well give it a writable `/tmp`. Every use is behind `music-ok`/`tone-ok`, so a failure is
|
||
silence and not a crash. Separately, browsers suspend the audio context until a user gesture, so `audio-ok` may be
|
||
true while nothing is heard until the first click.
|
||
- **The loop never exits.** `WindowShouldClose()` on `PLATFORM_WEB` is an `emscripten_sleep` that returns false, so
|
||
`until` never terminates and **none of `main`'s `defer`s ever run** — no `CloseWindow`, no `UnloadTexture`. That is
|
||
correct for a page, which is torn down by the tab closing, and it is worth knowing before reading anything into it.
|
||
- **Canvas size against `screen-width`/`screen-height`.** The shell is a string in `Build` and its canvas is not sized
|
||
from the program, so 900x600 may be letterboxed or cropped.
|
||
|
||
## The return type is the slot, and unit is `()`
|
||
|
||
`(defn name [param Type ...] ReturnType body ...)`. The slot after the parameters is unconditionally a type, and a
|
||
function that returns nothing writes `()`. It used to be optional.
|
||
|
||
**What optional cost.** `(Option f64)` and `(Some 1)` are the same s-expression — a type application and a
|
||
constructor call are indistinguishable by shape — so the parser decided which it had by looking the head up in a set
|
||
of names that were actually types, collected by a pre-pass over the file's own declarations plus the prelude's. That
|
||
is *sound*, because one top-level namespace means a name cannot be both a type and a value. It is brittle because the
|
||
set has to be complete, and **it was wrong twice in one day**:
|
||
|
||
- the prelude's type names were not in the set at all, so a macro's `Form` in return position read as an unknown
|
||
value;
|
||
- the fix for that exposed a second arm reading the same set, which had been parsing
|
||
`(defn f [] (Rune {.code 65}) (bar))` as a function *returning* a `Rune` with a one-form body — silently, in every
|
||
file in the language.
|
||
|
||
A silent misparse is the worst failure class available. Macros generate definitions now, which widens it, and a table
|
||
that has to be complete will be incomplete again.
|
||
|
||
**Mandatory removes the guess.** `Parse.decl` stops taking a set of names, `is_type_form`, `qualified_type`,
|
||
`types_in`, `declared_types` and `prelude_types` are gone, and the pre-pass over a file's declarations that fed them is
|
||
gone with them. Whatever is in the slot is a type; whatever follows is the body. Two things fall out:
|
||
|
||
- **A type the parser could not have known is fine.** A struct declared further down the file, `rl/Vector2` behind an
|
||
alias that is not resolved until after parsing, a prelude type — none of them needed to be *recognised*, they just
|
||
needed to be in the slot.
|
||
- **A typo is a typo.** `(defn f [] f65 0.0)` reaches `Check.resolve_name`, whose near-miss check answers *unknown
|
||
type f65 — did you mean f64?*. It used to be parsed as the first form of the body and reported as an unknown
|
||
**name**, which points at the wrong mistake.
|
||
|
||
The cost is `()` on every void function, against `plan.org`'s deliberate short form. Taken.
|
||
|
||
**Unit is `()`**, ML's spelling. It is honest, and it cannot collide: an empty call is not a valid expression, so
|
||
there is no reading of `()` in value position for a body form to be confused with. The old `Unit` spelling is
|
||
**refused**, with a message naming the new one — the same rule the colon-to-dot change followed, and for the same
|
||
reason: two accepted spellings is how two spellings become permanent.
|
||
|
||
Internally it is still `Types.Unit`, and `()` parses to `Ast.Tname "Unit"`, so the resolver, the shim and the emitter
|
||
did not change. `Cimport` still builds `Tname "Unit"` for C's `void` and never goes through the parser, which is why
|
||
`resolve_name` still answers to the word. **Diagnostics print `()`**: `Types.to_string` prints what a person would
|
||
write for every other type it knows — `[i32]`, `{K V}`, `(Ptr T)` — and `Unit` was the odd one out the moment the
|
||
source spelling changed.
|
||
|
||
**One shape is new.** `(defn f [] ())` is a function with a return type and no body. The old optional slot could not
|
||
produce it: `(defn f [])` had nowhere to put the type, and a lone form after the parameters was always the body.
|
||
|
||
### The sweep
|
||
|
||
`tools/unit-return.py`, kept rather than thrown away, because the lanes that branched before this wrote Flan in the
|
||
old spelling and their files want the same pass at merge:
|
||
|
||
```
|
||
python3 tools/unit-return.py .
|
||
python3 tools/unit-return.py --in-strings test/test_flan.ml test/test_acceptance.ml \
|
||
test/test_session.ml emacs/test-flan-dev.el emacs/test-flan-mode.el
|
||
python3 tools/unit-return.py --raw-ml lib/prelude.ml
|
||
python3 tools/unit-return.py --in-html web/index.html
|
||
```
|
||
|
||
It fills the empty slot with `()` and rewrites `Unit` as `()` wherever a type is spelled. Deciding whether a `defn`
|
||
*already* had a return type is the whole difficulty, and the script does it by transcribing `parse.ml`'s
|
||
`is_type_form` rather than improving on it — being identical to the parser it replaces is what makes the sweep
|
||
meaning-preserving. 440 sites in `.flan`, 260 more embedded in OCaml, elisp and HTML; `-v` logs every `defn` it saw
|
||
and what it decided, which is the only practical way to review a sweep that size.
|
||
|
||
Three hazards it knows about, and one it cannot:
|
||
|
||
- **A snippet split across concatenation.** `cursor ^ "(defn f [s [u8]] Cursor ...)"` is one OCaml literal that does
|
||
not contain the `defstruct` in the other. The embedded modes pool every fragment's type declarations across the
|
||
whole file and count a pooled name only in bare-symbol position — as a list head it would eat `(Some 1)` as a
|
||
return type, which is the misparse this change exists to remove. Sound because no *user* type takes arguments.
|
||
- **A fragment that cuts off mid-form**, `"(defn step [] i64\n"`, is skipped rather than guessed at. `flan-mode.el`'s
|
||
`"(defn step"` search strings are the same case.
|
||
- **A bare `Unit` outside a form is left alone**, so the checker's own `Tname "Unit"` pattern in a test literal is not
|
||
rewritten into nonsense.
|
||
- **What it cannot know**: `test_flan.ml` deliberately spells the *refused* forms, to test that they are refused —
|
||
`(defn f [] (g))`, `(defn f [] Unit (g))`, `(defn f [] Nope (bar))`, `(defn f [] f65 0.0)`. The script encodes the
|
||
old rule, so it wants to convert all six of those sites and must not. Re-running it on this tree reports exactly
|
||
those six; anything else is a real conversion. Read the diff.
|
||
|
||
## The colon belongs to keys, so a field label is a dot
|
||
|
||
`{.x 1.0 .y 2.0}` is how a struct is constructed, and `{inner .field}` is how a pattern names one. The colon is gone
|
||
from both, and what is left of it is one job: keys — map keys and enum members.
|
||
|
||
**What was wrong with the colon.** Nothing, taken alone. The problem was that it had two jobs and the dot had one.
|
||
`(.x v)` already read a field; `{:x 1.0}` also named a field, while `:space` named an enum member. So the dot meant
|
||
"field" and the colon meant "field, or member, depending". Moving the label to the dot leaves each mark with one
|
||
meaning, and it costs nothing to read because **the delimiter already disambiguates**: `(.x v)` is a list and
|
||
therefore a call and therefore an access; `{.x 1.0}` is a brace form and therefore a construction. There is no
|
||
position where the two could be confused, which is why the same spelling can serve both.
|
||
|
||
**Why it had to land before `Map`, and this is the real reason.** A map literal wants to be `{:key value}`. While
|
||
struct construction owned that exact spelling, a map literal and a struct literal were *the same syntax*, and the
|
||
only thing that could tell them apart was what the checker expected at that position. That is a context-sensitive
|
||
grammar for no gain. Reserving the colon for keys keeps the two visibly distinct at the reader, before any type is
|
||
known. Doing it after `Map` landed would have meant changing both; doing it first meant changing one.
|
||
|
||
**`:keys` kept its colon, and that is the point rather than an inconsistency.** `{:keys [x y]}` is the one thing in a
|
||
brace that is not a field name — it is an instruction to the compiler that happens to sit there, and it takes a
|
||
vector rather than a value. Giving it a dot would have made the dot mean "a field, or the word keys". Leaving it a
|
||
colon lets the dot mean exactly one thing, *this names a field*, which is the whole reason the colon was given up.
|
||
Everything else Clojure puts in that position — `:as`, `:or`, `:strs`, `:syms` — is still refused by its own name.
|
||
|
||
**The old spelling is refused, not accepted quietly**, and the refusal names the new one: `a field label is written
|
||
.x, not :x — the colon is for keys`. Two accepted spellings is how two spellings become permanent, and the standing
|
||
rule here is that what is not supported is rejected explicitly with the reason. Both refusals are tested by their
|
||
reason, on the construction side and on the destructuring side, which is what stops the colon drifting back.
|
||
|
||
**The sweep is a tool, not a one-off.** `tools/colon-to-dot.py` converted 681 labels across 45 `.flan` files,
|
||
`vendor/` included, and 94 more in the Flan embedded in `lib/prelude.ml` and the tests. It works on *forms*, not on text: a keyword
|
||
becomes a dot only where it sits in a field-label position inside a brace, so an enum member in value position
|
||
(`{.k :hi}`), a genuine EDN map inside a string (`test/programs/edn.flan`), and a type-position `{K V}` are all left
|
||
alone. It was kept in the tree because several lanes branched before it and their Flan needs the same pass at merge.
|
||
|
||
**What it deliberately did not change: the printed form.** `render.ml` still prints `(V {:x 1.5 :y 0})`. That string
|
||
is a wire format — `emacs/flan-inspect.el` parses it back and hard-codes the colon when it reads a field out — so
|
||
moving the printer alone would break struct inspection in the dev loop without breaking any test that says so. The
|
||
printer moves when its reader does, in the Emacs lane. It is the one place the old spelling is still correct, and
|
||
the reason is worth keeping: **a format with two ends only changes at both.**
|
||
|
||
**Since written: the printer moved, and both ends moved together.** `render.ml` prints `(V {.x 1.5 .y 0})` now and
|
||
`emacs/flan-inspect.el` reads the dot, which is the "moves when its reader does" the paragraph above was waiting on.
|
||
The reader tells `...` from a field label by one character of lookahead, because both begin with a dot and a field
|
||
name never starts with a second one.
|
||
|
||
## Two ways to root a walk, and why neither subsumes the other
|
||
|
||
`i` in the break buffer sent a local's **name** to be evaluated. An expression is evaluated where the evaluator
|
||
stands, so on the innermost frame that lands in the right frame by luck; on any other it may resolve to a global, to
|
||
another binding of the same name, or to nothing — with the locals listing right above it showing the frame's own
|
||
storage and nothing saying the two disagree. The display was right and the inspector was not, which is the worst
|
||
arrangement of the two.
|
||
|
||
**The obvious fix was tried and rejected, and the rejection was half wrong.** Rooting the walk at the slot's address
|
||
does not work on its own: an address is not an expression, so the first `RET` has nothing to build the next expression
|
||
from and navigation dies at step one. What that argument assumed is that the *step* has to be an expression too, and
|
||
the shadow stack is what stopped that being true. The daemon holds the frame's address and every slot's type, so
|
||
stepping into a field is an address plus an offset with that field's type — which is exactly the arithmetic
|
||
`Render.render` already does for the locals listing. So `Session.render_slot` is `render_locals` with a path applied
|
||
to the root before the walk and one line out instead of one per slot. No second walk was written and no backend
|
||
change was needed.
|
||
|
||
The verb is `(:op "inspect" :frame N :slot I :path (...))`. A path step is a string for a struct field, an integer for
|
||
an array or slice element, and the symbol `some` for an option's payload; a union case's field is spelled
|
||
`Union.case.field`, because the payload's offset depends on which case the value is in and only the renderer knows
|
||
which case it currently holds — it wrote the head `(Union.case {…})`. Guessing the case from a field name two cases
|
||
share would read one case's layout over another's payload. Every step that does not fit the type in hand is refused
|
||
by name with its reason. A pointer is still never followed; that is the renderer's rule and not this mode's.
|
||
|
||
**The slot travels by index, not by name.** `check.ml`'s `fresh_slot` only ever allocates, so `(let [v 22] …)` inside
|
||
`(let [v 11] …)` is two slots both called `v` and both are in the listing; and a refused slot is not in the listing at
|
||
all, so its position there is not an identifier either. `locals` therefore puts the slot index on each entry as a
|
||
fourth element, and that is what the break buffer hands back.
|
||
|
||
**The frame checks are the listing's, by construction.** `Dev.stopped_frame` is one function and `locals`, `globals`
|
||
and `inspect` all go through it: alive, stopped, the frame exists, it is the program's and not a `C-x C-e` thunk's,
|
||
its body is one this session holds, the slot count matches, and `Emit.slot_fingerprint` matches. An inspector with its
|
||
own copy of those conditions would be free to read a frame whose body was redefined since it was entered, which is
|
||
precisely the stale-slot answer the listing refuses. `inspect` adds one refusal of its own, for the listing's reason:
|
||
an unbound slot is a null address and a thunk that read it would fault on the game thread of a program that is already
|
||
stopped.
|
||
|
||
### What each root cannot do that the other can
|
||
|
||
Both are wanted and the buffer says which it is on.
|
||
|
||
**The expression root** works on a **running** program and starts from anything you can write, a call included. It
|
||
cannot name a frame — that is the bug — and it cannot reach an option's payload, because the compiler gets at that as
|
||
field 1 and nothing in the surface language does.
|
||
|
||
**The slot root** is exact to one frame and one slot, and it reaches an option's payload and a union case's fields,
|
||
which have offsets but no accessor form to write. It needs a **stopped** program, it is refused when the frame's body
|
||
was redefined since it was entered — the same fingerprint the listing is refused by — and it cannot root at an
|
||
expression at all, so `g` after the program resumes is refused rather than quietly answered from somewhere else.
|
||
|
||
A refusal someone can read is the point of the second one existing. The failure being fixed was not "no answer", it
|
||
was a confident answer from the wrong place.
|
||
|
||
### `l` does not cross between them, structurally
|
||
|
||
A stack entry in `flan-inspect.el` is `(ROOT PATH . POINT)`. `RET` only ever appends a step to the path under the root
|
||
the buffer already has, and every new root — `flan-inspect`, `flan-inspect-slot` — starts with an empty stack. A stack
|
||
with two kinds of root in it therefore cannot be constructed, so the question of what `l` should do when it crosses
|
||
one does not arise. That stays true if a third rooting mode is added, which is why it is worth having as structure
|
||
rather than as a rule in a comment.
|
||
|
||
The Emacs state is a root plus a path rather than a retained value for the same reason the expression stack was:
|
||
nothing on this side can hold a Flan value. A value has no header, the thunk that rendered it is `dlclose`d the moment
|
||
it returns, and there is no heap to retain it in. So every step and every `g` is a fresh request, which is what keeps
|
||
the view from ever being stale — and it is also why `g` is a key someone presses rather than a timer, since an
|
||
expression root with an effect in it would fire once a second for ever.
|
||
|
||
**One wire detail worth recording.** An empty `:path` is sent by omission. Emacs prints an empty list as `nil`, which
|
||
is a symbol on the wire and would be read as a step, so there is no way for a client in that language to spell `()`.
|
||
The daemon reads a missing `:path` — and `nil` — as the slot itself.
|
||
|
||
## The watch window, and why it is the only listing that is pushed
|
||
|
||
The port of the author's Clojure `watch.el`, with the good idea kept and the transport turned round. The original is
|
||
85 lines and three of its decisions survive contact unchanged:
|
||
|
||
- **The program decides what is shown.** Emacs paints what one function hands it. There is no watch-expression
|
||
machinery, no per-variable registration, no UI for building a query. Everything a watch list would otherwise have to
|
||
answer — where it lives, whether it survives a restart, whether it gets committed by accident — stops being a
|
||
question once the list *is* the code, edited with `C-c C-c` like anything else.
|
||
- **The request is async.** A synchronous call on a 0.2s timer blocks Emacs's UI every tick. The original says so in a
|
||
comment, having evidently learned it.
|
||
- **`replace-buffer-contents`, not erase-and-insert.** It diffs, so point and scroll survive every repaint. Erasing
|
||
yanks the cursor to the top five times a second, which makes the buffer useless for the one thing anyone wants to do
|
||
in it — look at a particular line while the program runs.
|
||
|
||
### The one thing that does not port, and it inverts the design
|
||
|
||
In Clojure an eval is cheap. Here `eval-expr` **compiles a module and `dlopen`s it** — tens of milliseconds and a new
|
||
`.so` each time, in a directory nothing sweeps. Polling `(watch/render)` at 5Hz would produce hundreds of shared
|
||
objects a minute to read a number that was already in a register.
|
||
|
||
The first answer written down — in NEXT.md, now superseded — was to compile the render thunk *once* and re-invoke it
|
||
cheaply per tick. That is the right instinct and it is still a poll, and a poll has a defect no amount of caching
|
||
fixes: **it cannot answer while the program is stopped.** A thunk runs at a frame boundary, a stopped program has no
|
||
more frame boundaries, and a break loop is precisely when you most want to see what the last frame held.
|
||
|
||
So the direction is reversed. **The program pushes.** It calls into a table in `flan_dev.c` from inside its own loop;
|
||
Emacs reads the table, which is memory rather than an evaluation. Both halves are cheap for opposite reasons, and two
|
||
properties fall out that no poll has:
|
||
|
||
- the values are as fresh as the **last frame**, whatever the repaint interval happens to be — the timer decides how
|
||
often the picture is redrawn, not how current it is;
|
||
- and they are **still there while the program is stopped**, because nothing has to run to produce them.
|
||
|
||
This makes `watch` the only listing in the daemon that compiles nothing. Every other one — `locals`, `globals`,
|
||
`inspect`, `eval-expr` — is a thunk built from the types, delivered, and run at a frame boundary. That is affordable
|
||
at the rate a person presses a key and ruinous at the rate a HUD refreshes, and the difference in rate is the whole
|
||
reason this one is shaped differently.
|
||
|
||
### What the frame thread is allowed to do, and what the table is therefore made of
|
||
|
||
The writer is the game thread, mid-frame, every frame. That is a stricter constraint than the rest of `flan_dev.c` is
|
||
under, and it decides the storage:
|
||
|
||
- **No allocation.** Names are fixed `char` arrays inside the table, not `strdup`'d the way `intern` does it
|
||
alongside. `intern` runs at module load and may `malloc`; this runs at 60fps and may not.
|
||
- **No lock**, because the reader is the agent's listener thread and neither side may wait for the other.
|
||
- **No call into OCaml**, which is the rule that keeps the collector off the frame thread. Nothing in the table is
|
||
OCaml.
|
||
|
||
**The `result` buffer is deliberately not reused**, and this was the tempting mistake. The renderers are the same
|
||
shape, so sharing looks free — and it is wrong, because `result` is written once per `C-x C-e` and this is written
|
||
every frame. Sharing would mean watch traffic overwriting the value of every expression anyone evaluated. Separate
|
||
storage, separate counters.
|
||
|
||
**One seqlock per slot rather than one for the table.** A table-wide counter makes a read all-or-nothing: the reader
|
||
has to copy every slot inside a single even generation, which means catching the gap *between* two frames' worth of
|
||
writes — a window that at 60fps is whatever the program does after its last watch call, and may be nothing. Per-slot,
|
||
the reader retries one slot at a time and always gets somewhere. The worst it can produce is a snapshot whose entries
|
||
come from adjacent frames, which for a HUD is not a defect: a frame counter one ahead of a position read a
|
||
millisecond earlier is what a HUD looks like anyway. It would be a defect for anything where two values have to agree,
|
||
and that is a different op rather than a bigger counter.
|
||
|
||
A torn slot is still **listed**, with an empty value, rather than dropped. Dropping it would make the buffer's rows
|
||
move under the reader every time the game happened to be mid-write, which is much worse to look at than one value
|
||
that is blank for a tick.
|
||
|
||
### The bounds, and what happens past each
|
||
|
||
**64 slots. Past that a name is dropped, not fatal.** This is the one place the house style in `flan_dev.c` — `die`,
|
||
loudly — would be wrong: a watch is a diagnostic, and killing the program because somebody watched a 65th value is the
|
||
diagnostic shooting the patient. It is not silent either. An overflow flag is read back with the table and the buffer
|
||
says so, because a value that simply never appeared would send someone looking for a bug in their program.
|
||
|
||
**A flag and not a count, which was a correction.** The first version counted, and a counter on the write path counts
|
||
*writes*: the write path runs once per watched value per frame, so one name too many at 60fps reads back as "3847
|
||
names found no slot" within a minute — a false sentence about a true problem. What a reader needs is "the table is
|
||
full and something is not being shown", which is one bit, and one bit cannot drift into a wrong number. Counting
|
||
*distinct* names that missed would mean remembering which ones had, which is exactly the bookkeeping the frame thread
|
||
has no room for.
|
||
|
||
**31 bytes of name, 192 bytes of rendered value.** Both truncate; the value's truncation shows as an ellipsis, the
|
||
same as `flan_dev_result_end` does, so a clipped value does not read as a complete one.
|
||
|
||
### What it costs when nobody is watching
|
||
|
||
Nothing writes the table until a watch buffer is open. `M-x flan-watch` sends `watch-enable :on t` and closing it
|
||
sends `:on nil`, so arming is a *message* rather than something the daemon infers — the program is the writer and it
|
||
has to be told.
|
||
|
||
So the cost of a watch call in a program nobody is debugging is **one relaxed load and a not-taken branch**, and that
|
||
is the same number in a release build as in a dev one: `flan_dev.c` is linked into every build (`Build`, which says
|
||
why), so the symbols resolve either way and there is no second version of the file.
|
||
|
||
It is not *free*, and the distinction is worth keeping honest. Eliding the call entirely needs the compiler to know
|
||
the form, which is the `check.ml` arm below. A load and a branch per watched value per frame is the real number.
|
||
|
||
### Scalars work today; composites need a `check.ml` arm that was not built
|
||
|
||
The four entry points a program can reach through `declare-c` are the whole feature for a scalar:
|
||
|
||
```flan
|
||
(declare-c watch-i64 [name string x i64] i32 "flan_dev_watch_i64")
|
||
(watch-i64 "ticks" ticks)
|
||
```
|
||
|
||
No arm in the checker, no new special form, nothing the compiler has to learn. They return `i32` rather than nothing
|
||
for a blunt reason: `declare-c` refuses a void return outright — "which is not a value C can carry", `shim.ml` — so a
|
||
function a program can declare has to return something, and since it must, it returns the useful thing: 1 if the value
|
||
was written, 0 if nobody is watching or the table is full.
|
||
|
||
A composite — a struct, a slice, a union — cannot be reached this way, and that is not a shortcoming of the four. A
|
||
Flan value carries no header, so nothing at run time can say what it is, and rendering one is a compile-time walk over
|
||
its *type*. **That is the same reason `C-x C-e` renders in the thunk rather than marshalling anything**, and it is the
|
||
layout decision's bill, paid in the same place.
|
||
|
||
**The missing piece is one arm in `check.ml`, and it was deliberately not written** — that file is held by another
|
||
lane. It sits beside `print` (`check.ml:3480`) and is the same shape as it:
|
||
|
||
```
|
||
| "watch" ->
|
||
arity loc name 2 args; (* a name and a value *)
|
||
(* a read, not a move — as print is, for the same reason: (watch "v" v)
|
||
must not consume a Vec and make that its last showing *)
|
||
let n = check ctx (List.nth args 0) in (* must be String *)
|
||
let a = borrowed ctx target (fun () -> check ctx (List.nth args 1)) in
|
||
(* begin, the walk, end — with the emitter aimed at the four
|
||
flan_dev_watch_emit_* rather than at WriteStdout *)
|
||
Render.render { rc with emit = watch_emitter } 0 a
|
||
```
|
||
|
||
with `flan/watch-begin`, `flan/watch-end` and the four emit functions declared as externs the way `Session.externs`
|
||
already declares `flan_dev_emit*`. Nothing else has to move: `Render.render` is unchanged, the runtime side is built
|
||
and tested, and the daemon and the editor cannot tell which kind of caller filled the table.
|
||
|
||
~~That arm is also what **ghost text** is gated on.~~ **It was not, and ghost text is built without it** — see
|
||
"Ghost text finds its anchor in the buffer, not in the table" below. The claim was that values shown inline need a
|
||
*place* and nothing in the table has one, so a source location would have to be carried per entry, so the call site
|
||
would have to be generated. True of the table and false of the conclusion: the call site is in the buffer. The
|
||
composite renderer still wants the form, for its own reason, and it is the only one of the two that does.
|
||
|
||
## An error is a value, and there is more than one of them
|
||
|
||
`lib/loc.ml` used to carry a point and a message, and `Loc.Error` was the frontend's one exception, so the first
|
||
error ended the run. The author's workflow is write everything, compile at the end, work through the list — which
|
||
cannot happen when there is never a list. The messages themselves were already good; they state the reason and name
|
||
what to write instead, and **none of them changed**. What was missing was structure and volume.
|
||
|
||
### The location is a span
|
||
|
||
`Loc.t` grew an exclusive end, defaulting to the start. That is the whole trick: a location nobody widened is a
|
||
zero-width span at a point, so every call site that existed before means exactly what it meant, and `Loc.to_string`
|
||
still prints `file:line:col`. Only the reader knows where a form ends, so only the reader fills them in — one helper
|
||
in the one place that holds both ends, which is why nothing above `Reader` had to learn a span exists. `Form`, `Ast`
|
||
and `Tast` were not touched and did not need to be.
|
||
|
||
A column number cannot draw an underline and a span can. That is what the field is for and it is the only reason it
|
||
is there.
|
||
|
||
### The error itself
|
||
|
||
```ocaml
|
||
type diag = {
|
||
kind : string; (* "reader/unclosed", stable *)
|
||
dloc : t; (* the primary span *)
|
||
dmsg : string;
|
||
notes : note list; (* each with its own span and severity *)
|
||
expansion : (string * t) option; (* the macro it came out of *)
|
||
}
|
||
exception Error of diag
|
||
exception Errors of diag list
|
||
```
|
||
|
||
Three parts, each buying something the old pair could not express.
|
||
|
||
**`kind`** is a stable id. It classifies with no prose parsed, so a message can be reworded without breaking anything
|
||
that depends on *which* error this is. The reader's fourteen refusals all carry one; in the checker they go on the
|
||
errors a test names and the handful common enough to be worth classifying. **Not a hundred of them.** jank has about
|
||
a hundred because it is mature, and the count is not the feature — with 163 refusal sites in `check.ml` alone,
|
||
minting an id for each would be a sweep that never ends and that nothing reads.
|
||
|
||
**`notes`** are the part that was actually missing, and they are the secret of an Elm-quality message. Each carries
|
||
its own span and its own severity, so an error says "this is wrong *here*" **and** "because of *that*, over there",
|
||
and points at both. One location and one string can only ever state one of the two. What has them today:
|
||
|
||
- a name defined twice points at the second, because that is the one to delete, and notes the first;
|
||
- a duplicate parameter and a duplicate field do the same;
|
||
- an unknown field, an unknown struct and a non-exhaustive `match` note the *declaration* and list what is actually
|
||
there, so the reader's next move arrives with the question instead of after it;
|
||
- the reader's unclosed bracket is the clearest case — the error sits on the bracket, because that is where the fix
|
||
goes, and the note sits where the file ran out, because that is the surprise. A mismatched closer is the mirror of
|
||
it: the wrong closer is where the mistake reads, and the opener is what makes it wrong, and neither alone says
|
||
which bracket to change.
|
||
|
||
**`expansion`** names the macro an error is really about. It rides on the *location*, not on the form, because the
|
||
location is the thing that already travels: `Expand.unmarshal` stamps the call site onto every node a macro answers
|
||
with, and that stamp goes on through the AST and the typed IR untouched. Tagging it there means an error raised
|
||
anywhere downstream can name the macro with **no field added to `Form`, to `Ast` or to `Tast`**. Outermost wins — the
|
||
macro the author wrote is the one worth naming, not whatever it expanded into on the way down.
|
||
|
||
### The squiggle
|
||
|
||
The first line of an entry is exactly `file:line:col: message`, which is the GNU format `compilation-mode` parses
|
||
with no configuration. That is the whole of the editor story: once more than one comes out, `M-x compile` gives a
|
||
clickable list and `next-error` walks it. Everything under the first line is indented, and `compilation-mode` ignores
|
||
indented continuation lines, so the underline is free:
|
||
|
||
```
|
||
prog.flan:6:3: Cursor has no field pos
|
||
6 | (.pos c))
|
||
| ^^^^^^^^
|
||
prog.flan:1:1: info: Cursor is declared here, with row, col
|
||
1 | (defstruct Cursor
|
||
| -----------------
|
||
```
|
||
|
||
A note gets an **entry of its own** rather than being folded into the error's block. That is gcc's shape and it is the
|
||
point of notes having locations at all: the second place becomes a place the compilation buffer knows about.
|
||
|
||
**Stated at its true strength, because it was checked rather than assumed** (`compile.el`, Emacs 30.2). The `gnu`
|
||
entry in `compilation-error-regexp-alist-alist` puts `Note`/`note` in the *same capture group* as `Info`/`info` —
|
||
group 7, level 0 — while `warning` is group 6, level 1. `compilation-skip-threshold` defaults to **1**, "skip
|
||
anything less than warning". So: **errors** are navigable with `next-error` out of the box, which is the claim that
|
||
matters and the one `M-x compile` rests on. **Notes** are parsed, coloured and clickable, and `next-error` steps over
|
||
them until `compilation-skip-threshold` is 0. Renaming the label from `info:` to `note:` does not change that — same
|
||
group. Labelling notes `warning:` *would* make them navigable at the default, and is refused: a note is not a
|
||
warning, and a compile whose only complaint is an error would start reporting warnings that are not warnings.
|
||
|
||
Every part of it degrades to the bare first line. A location the checker invented has line 0 and a file called
|
||
`<unknown>`, the prelude and the REPL have names that are not paths, and a file can change under us between being
|
||
read and being blamed. An error printer that can raise is worse than one that prints less. Placeless diagnostics sort
|
||
*last*: a wrong `main` signature is raised against `unknown`, and sorting on the line number alone put it above every
|
||
error that could actually be clicked.
|
||
|
||
The source cache in `loc.ml` is process-lifetime, which is right for `flan build` — a fresh process per run. The
|
||
daemon is long-lived and never calls `report`; the interactive path draws no squiggle, it takes a location and a
|
||
message. `Loc.forget_sources` exists for the day that changes.
|
||
|
||
### Collecting, and where it stops
|
||
|
||
A sink holds what a pass found so the pass can go on to the next thing. It is switched on by the caller, not by the
|
||
code that raises. Two resync points, and both are places the work already had a boundary:
|
||
|
||
- **In the parser, a top-level form.** The reader already found where each declaration ends, so skipping a bad one
|
||
costs nothing and cannot lose its place. Inside a declaration there is no such landmark, so one bad `defn` stays
|
||
one error.
|
||
- **In the checker, the two passes.** Pass one — which builds every name, type and signature — **still stops at the
|
||
first refusal**, and that is deliberate rather than unfinished. A signature it could not make sense of leaves a hole
|
||
that pass two would report once per mention; thirty "unknown name" lines under one wrong signature are not thirty
|
||
errors, they are the same one. Pass two is where the volume is and where collecting pays, and by then every
|
||
signature is sound, so a body that fails cannot make the next body fail. That is what makes a declaration a resync
|
||
point needing no resynchronising.
|
||
|
||
**The reader does not collect at all.** There is no resynchronising a paren stream: after an unclosed bracket the
|
||
reader has no way to know whether the next `)` closes the form it is in or the one above it, and guessing produces a
|
||
file-shaped pile of nonsense. First error, stop. That is a decision, not an omission.
|
||
|
||
### What the daemon sees, which was the open question
|
||
|
||
Changing the error type without touching `dev.ml` and `session.ml` needed a compatible way to get one location and
|
||
one message out. The answer is that **the single-diagnostic exception is still the single-diagnostic exception**.
|
||
`Session.eval` and the daemon evaluate one form and have one failure to report; they keep catching `Loc.Error` and
|
||
take the pair out of it with `Loc.summary`. Only a driver that compiles a whole file raises `Loc.Errors`.
|
||
|
||
That guarantee is **structural and not conventional**. `Parse.program` / `Check.program` stop at the first refusal;
|
||
`Parse.program_all` / `Check.program_all` collect. Two names rather than one function with a `~keep_going` label,
|
||
because `Loc.Errors` is a second exception that the session's handlers do not name — a list reaching them would be an
|
||
unhandled exception and a dead session, which is the one thing the dev loop exists to prevent. With a label that was
|
||
one keystroke away at a call site the session already uses. With two names, somebody has to edit the session.
|
||
|
||
### What it looks like
|
||
|
||
```
|
||
$ flan check bad.flan
|
||
bad.flan:2:8: unknown name bogus
|
||
2 | (+ a bogus))
|
||
| ^^^^^
|
||
bad.flan:5:8: unknown name nope
|
||
5 | (- a nope))
|
||
| ^^^^
|
||
bad.flan:8:3: unknown function mystery
|
||
8 | (mystery 1 2))
|
||
| ^^^^^^^^^^^^^
|
||
3 errors
|
||
```
|
||
|
||
**No editor work was needed and none was done.** Flycheck and a structured JSON report were both considered and are
|
||
not wanted: the workflow is compile-at-the-end, not live linting, and the GNU first line already buys the clickable
|
||
list.
|
||
## The four raylib lines siam-farmer needed
|
||
|
||
`PORTING.md` measured the author's game against the binding and found the renderer unwritable in a default build.
|
||
Four lines closed it, and the interesting part is not the lines.
|
||
|
||
`draw-texture-pro` is the one that mattered. It is the only call in the package that takes both a source rectangle and
|
||
a destination rectangle, which is what a tilemap is: `source` picks a cell out of an atlas, `dest` says where it lands
|
||
and how big, and a 16px tile drawn at 4x is a dest four times the source. `draw-texture-rec` has the source and no
|
||
scale; `draw-texture-ex` has the scale and no source. Neither half draws a tile.
|
||
|
||
It *was* reachable — with `FLAN_RAYLIB_H` exported the importer brings it in with 250-odd others — and that is the
|
||
finding worth keeping. `vendor/raylib/headers` kept the import opt-in on purpose, so a build needs libraylib linkable
|
||
and not raylib-devel installed. That property is worth keeping and it means **the default build had no draw call for a
|
||
grid-based game**. (The opt-in is gone — the header and the generated bindings are both committed now, so the default
|
||
build has every declaration. The rule below is unaffected: it is about where a per-frame call is *written*, not about
|
||
what the import happens to reach.) The rule that follows: *a raylib function on a game's per-frame path is hand-written in
|
||
`raylib.flan` and checked against the header; it is not left to the import.* The import widens the surface; it must
|
||
not be load-bearing.
|
||
|
||
The other three: `image-from-image` (the non-mutating `image-crop` — carving a sheet into twenty tiles with
|
||
`image-crop` destroys the sheet on the first one, and 5.5 has no `ImageCopy`), `window-ready?` (an engine's "am I
|
||
already running" guard), and `left-shift 340` in the `Key` `defenum`. Nothing else went into the enum: it carries the
|
||
keys that have a customer, and `PORTING.md` §5 checked every other value the game touches and found them all present.
|
||
|
||
**What could be tested, and what could not.** `PORTING.md` asked for an acceptance case making raylib compute with the
|
||
source rect so a permuted `Rectangle` goes red. That cannot exist for `DrawTexturePro` — it needs a GL context, and
|
||
`raylib.flan`'s Shapes comment already says none of the drawing calls can be in the table. So `raylib-ffi.flan` links
|
||
it instead: the call sits behind `(when (rl/window-ready?) …)`, false headless, so the shim is generated and the
|
||
symbol resolves at link time and the body never runs. That catches a name or an arity libraylib does not have. It does
|
||
**not** catch the argument order, and three structs in a row is where an argument order goes wrong. Only looking at
|
||
the screen catches that, and saying so is better than a test that implies otherwise.
|
||
|
||
The computed case moved to `image-from-image`, which is CPU-side and is exactly what the Images section says is
|
||
assertable. `raylib-image.flan` carves one 6×3 sheet twice at two different `y`s and then re-reads the sheet: `x`,
|
||
`y`, `width` and `height` are each pinned by an answer that moves if they do, and the sheet surviving both carves is
|
||
what distinguishes this from `image-crop`. Bind it to `ImageCrop` by mistake and the second carve reads out of a 2×1
|
||
image and the case goes red.
|
||
|
||
## An index out of range is a condition
|
||
|
||
`flan_bounds_fail` printed the source location, the index and the length, and called `exit(134)`. That was defensible
|
||
while `flan dev` was two processes. It is not now: the compiler runs **inside the program**, so the trap took the
|
||
whole session with it — and the session not having to restart is the project's thesis. `PORTING.md` found the customer
|
||
and found it on the most ordinary path there is: a grid indexed straight from a mouse position is out of bounds the
|
||
first time the pointer leaves the window, and the author's Common Lisp port had to add an `in-bounds-p` to survive it.
|
||
|
||
A failed bounds check now signals **`BoundsError`** with `error`, the same way a failed allocation signals
|
||
`StorageExhausted`. `runtime/flan_rt.c` has `flan_bounds_error` and `flan_slice_error`; each walks the handlers, then
|
||
offers the break loop, and only if neither transferred does it tail into the `flan_bounds_fail`/`flan_slice_fail` that
|
||
were there before — same message, same status 134. Nothing was removed; a die was demoted to a last resort.
|
||
|
||
`(defstruct BoundsError [low i64 high i64 length i64])` is in the prelude. Fixed numeric fields and no rendered
|
||
message, for `StorageExhausted`'s reason: formatting allocates, and a condition raised on a path that may be out of
|
||
storage must not. `low` and `high` are the same index for an `(at xs i)` and the two ends of the range for a `(slice
|
||
xs lo hi)`, so **one** condition type covers both and a handler that wants to survive a bad index writes one clause
|
||
rather than two. The three `int64_t`s in `flan_rt.c` have to agree with that `defstruct` field for field — the same
|
||
hand-kept agreement `flan_name_id` already keeps with `Check.type_id`, and for the same reason: a struct is a layout,
|
||
a type is a number, and neither side can see the other.
|
||
|
||
### Why no restart is established at the failing index
|
||
|
||
This is the decision, and it is a decision rather than an omission.
|
||
|
||
`alloc_guard` offers `retry` and `file_guard` offers `retry`/`use-value` **because their attempt is repeatable**. A
|
||
handler frees something and the allocation succeeds; a handler supplies another path and the open succeeds. That is
|
||
what makes them `plan.org`'s named exceptions to "restarts go at the resync point, once" — a restart at an outer loop
|
||
cannot re-attempt an allocation, and only the allocation site can.
|
||
|
||
Nothing a handler can do makes index 51 valid for a length-50 array. There is no attempt to re-run, so there is
|
||
nothing for a site restart to resume into, and bounds falls on the **default** side of that rule.
|
||
|
||
- **`continue` is wrong.** `(at xs i)` has to produce a value of the element type and there is none to produce. It
|
||
would mean something in the `(set (at xs i) v)` position and nothing in the other, and `at` is one form.
|
||
- **`use-value` for the index is the near miss and is still wrong.** It costs the hot path, not just the cold block:
|
||
`idx` is an SSA value feeding the gep, and retrying needs it in an alloca reloaded per attempt, plus a restart frame
|
||
pushed and popped on **every** indexing operation. What it buys is a *different element*, silently — the class of
|
||
answer this codebase refuses everywhere else.
|
||
- **What actually answers a bad index is already on the stack.** A frame loop's `continue` — `sand.flan`'s shape — is
|
||
an ordinary `restart-case`, `flan_find_restart` walks to it, and the break loop lists it. Signalling is the whole
|
||
fix. A site restart would add nothing the frame loop does not already offer, at a cost on every index in the
|
||
program.
|
||
|
||
### Release, and what a build without the agent does
|
||
|
||
`flan_break_hook` is NULL unless the agent package was imported, so a release build — or any program that did not
|
||
import it — signals, finds no handler, finds no hook, and dies with the message and the status it always had. That is
|
||
still right: there is nowhere to stand. The change is not "bounds failures stopped being fatal"; it is "a bounds
|
||
failure is now answerable, and is fatal when unanswered".
|
||
|
||
### Defer, which had to be answered rather than inherited
|
||
|
||
The note at the top of this file said *a trap runs no defers, which follows from the bounds-check shape (`noreturn`
|
||
then `unreachable`) rather than being a separate decision*. The shape changed, so the consequence could not be
|
||
inherited. It now splits:
|
||
|
||
- **An answered bounds failure runs the defers.** `guard` routes through `current_pad`, which with no enclosing
|
||
`restart-case` sets `f.unwound` and branches to the function's unwind block — the same path `return` uses, which is
|
||
where §5's defers already live. So a transfer out of a bad index is an ordinary transfer and runs cleanup
|
||
innermost-first, like every other one.
|
||
- **An unanswered one still runs none**, because it is still a `rt_die()` inside C with no Flan frame involved.
|
||
|
||
`bounds-condition.flan` asserts the first of those directly: five `defer`s across five abandoned and finished frames,
|
||
counted.
|
||
|
||
### Both halves are tested, and they are different tests
|
||
|
||
`bounds-condition.flan` (acceptance table, at `-O2`, `-O0` and as a dev build) is the **answered** half: a
|
||
`handler-bind` over five routes to a bad index, taking the frame loop's `continue` each time. It does not import the
|
||
agent, so `flan_break_hook` is NULL in all three rows and nothing there says anything about the break loop.
|
||
|
||
`dev-break-bounds.flan` (`test_dev.ml`) is the half the change is actually for: **nothing handles it**, so the signal
|
||
walks the handlers, finds none, and reaches the hook. The daemon sees a program that stopped without being told to;
|
||
`BoundsError` is what the break reports; its own name resolves to a layout whose fields are `low`, `high`, `length`,
|
||
so the conditions buffer shows the numbers with nothing special-cased for it; the restart list is exactly
|
||
`continue` — the *program's* own, which is the visible consequence of establishing none at the site — and taking it
|
||
resumes, with an ordinary evaluation working on the far side. That last step is the whole claim: the session outlived
|
||
the index.
|
||
|
||
### Vec and Map
|
||
|
||
`(at v i)` and `(at arr i)` are the same form in the source, so shipping one signalling and the other exiting would
|
||
have read as a bug. A `Vec`'s bounds check lives *inside* `flan_vec_at` and `flan_vec_as_slice` rather than in emitted
|
||
IR (which is also why `--no-bounds-checks` never reached it), so both grew a trailing transfer-channel parameter and
|
||
`Emit`'s `Rt` arm guards those two symbols and no others — they are the only ones in that family that can transfer;
|
||
everything else there is arithmetic over a container header.
|
||
|
||
**A `Map`'s bounds and a `Vec`'s stale-allocator check still die.** `flan_vec_stale_fail` is a different kind of
|
||
failure — the region a container lived in was released, and there is no frame to go back to that would not read freed
|
||
memory — and the map path was left alone rather than converted half-way. Written down here so it is a known edge
|
||
rather than a discovery.
|
||
|
||
|
||
## An address answers with a type, and nothing grew a tag word
|
||
|
||
A Flan struct is exactly its C layout. No header, no tag word — deliberately, and it is what makes a struct free and
|
||
what makes the FFI work. The consequence is stated elsewhere in this file more than once: *a Flan value carries no
|
||
header, so nothing at run time can say what it is*, which is why a rendering is a compile-time walk over a type and
|
||
why `(watch "v" v)` cannot reach a composite without an arm in `check.ml`.
|
||
|
||
The allocation registry does not answer that question. It sidesteps it. **The allocator's caller knows the type at the
|
||
moment it asks for memory**, and the compiler is standing right there, so a dev build writes it down: base address,
|
||
extent, element size, and the type's printed spelling. Nothing about any value's layout changes, and raylib never
|
||
finds out.
|
||
|
||
### The split, which is the whole design
|
||
|
||
Two halves, and they are in different files because they need different things.
|
||
|
||
- **Recording the type is emitted.** `check.ml` builds one `flan_dev_reg_note_*` call after every operation that may
|
||
have allocated — `vec-new`, `push`, `reserve`, `clone`, `pool-new`, `insert`, `map-new`, `put`, `slurp`. Here is the
|
||
only place the concrete element type exists, so here is the only place that can name it. It is built in **every**
|
||
build, because a tree that differed by build flag would make every pass between the checker and the backend ask
|
||
which one it was looking at; `Emit`'s `Rt` arm drops the family when `dev` is off.
|
||
- **Recording that a block died is not emitted.** An address needs no type, so `flan_rt.c` calls into the table
|
||
directly from `flan_heap_proc`'s free, from a heap resize for the block it moved away from, from the arena's
|
||
`free-all`, and from `arena-destroy`. No signature in the allocator grew a type name and no ABI moved.
|
||
|
||
The drop in `Emit` happens **before** the arguments are walked, not after. A note takes the address of the container
|
||
it describes; emitting that address and then discarding the call would leave an escaped `alloca` behind, and an
|
||
escaped `alloca` is one mem2reg will not promote. So a release build's IR is the same IR it always was.
|
||
|
||
### It is armed by a constructor, and that is not fastidiousness
|
||
|
||
`@llvm.global_ctors` in the dev module, not a line at the top of `main`. A `defvar` initialiser can allocate, and it
|
||
runs before `main` does; a note that arrived before the flag was set would be a block the table never heard of, which
|
||
is a live pointer the inspector would call dead. That is the one failure mode worse than no registry at all.
|
||
|
||
### Lookup is containment, and that is not an optimisation
|
||
|
||
An entry is a **block**, not a value. Every pointer a program can hold into heap storage is interior: `(at v i)` is
|
||
`v->ptr + i*size`, and `(resolve p h)` is an item in the middle of a pool's items array. Neither is ever a base
|
||
address. A table that answered only exact hits would answer nothing anyone can ask it.
|
||
|
||
The two sides of the table therefore look different, and the difference is which thread is asking:
|
||
|
||
- `flan_dev_reg_note` and `flan_dev_reg_dead` are **probes** — a hash slot and a linear walk from it. Both are on the
|
||
writer's side, and a free hands back the same base address the allocator gave out, so equality is the whole
|
||
question there.
|
||
- `flan_dev_reg_live` and the epitaph are a **scan** of all 4096 slots. The reader is a person pressing a key, so the
|
||
cost belongs there.
|
||
- `flan_dev_reg_dead_range` is a scan too, and it is the one that runs per frame: an arena's `free-all` has no list of
|
||
what it handed out, so the region is matched against the table rather than the other way round. That is a real
|
||
per-frame cost in a dev build and it is named here rather than discovered later.
|
||
|
||
**Two threads, so the table has the watch table's seqlock.** The writer is the game thread, inside every allocation and
|
||
every free; the reader is the agent's listener, and the two listing verbs are asked of a *running* program — "what is
|
||
still held" is the question asked in the last moment before a game is killed, which is not a moment anything is stopped
|
||
in. So the frame chain's answer, snapshot it while the thread is parked, is not available here. Each entry carries its
|
||
own counter, odd while it is written; a reader copies the entry and re-reads the counter, and a compaction bumps a
|
||
table-wide counter around itself because it moves entries between slots and a per-slot counter cannot describe that.
|
||
The pair this protects is `type` and `typelen`: they mean nothing apart, and a reader holding the new pointer with the
|
||
old length reads off the end of a string literal. `reg at`, the one verb that makes a claim about a single address
|
||
rather than describing the program, is refused while running instead — the daemon already refused it, and the agent now
|
||
says so too.
|
||
|
||
Dead entries are kept. That is the second thing the registry buys — an address that was freed still names what died —
|
||
and an entry is dropped only when the allocator hands the same address out again, which is exactly when the old answer
|
||
stopped being true. When the table fills it is compacted, dropping the dead and re-inserting the live; in a
|
||
long-running program the dead are the bulk of it.
|
||
|
||
### What it is for: permission, not identification
|
||
|
||
This is the part worth stating plainly, because the obvious reading is wrong.
|
||
|
||
`(Ptr Enemy)` **already says Enemy**, at compile time, in `Render`'s walk. The type at the far end was never the
|
||
difficulty. What was missing is *permission*: whether it is still true to read the storage there. `render.ml`'s old
|
||
comment said a pointer is never followed because "dereferencing one a REPL was handed is not a safe thing to do on
|
||
someone's behalf", and that sentence is still correct — the registry just makes the safety checkable. So the arm
|
||
became a branch:
|
||
|
||
```
|
||
("live" "(Ptr Enemy)" "<ptr (Enemy {.hp 41 .x 2})>")
|
||
("dead" "(Ptr Enemy)" "<ptr dead: was Enemy, freed at step 15>")
|
||
```
|
||
|
||
The recorded type name is therefore not what selects the renderer. It is what the *epitaph* says, and a cross-check
|
||
available to anything that wants one.
|
||
|
||
An address the registry never saw renders `<ptr>`, unchanged. That is a stack local, a global, or a pointer from C,
|
||
and the shadow stack and the static type table already answer for the first two by name.
|
||
|
||
**`println` does not follow a pointer, and will not.** `spec-memory.md` fixes what a printed `Ptr` prints, a printed
|
||
line belongs to the program and has to read the same in a release build, and a release build has no registry to ask.
|
||
The two callers of `Render` already differ in an emitter record; they now differ in a `pointers` record too, and
|
||
`check.ml` passes `None`. This is also why the epitaph carries **no address**: an address is not stable across two
|
||
runs, so printing one would make a rendering — and any test that reads one — depend on where the heap landed. It is
|
||
the rule `Render` already follows for an allocator.
|
||
|
||
### The arena hole: answerable *and* reported — two tools, still two claims
|
||
|
||
`free-all` is retain-capacity: the offset goes to zero, the pages stay mapped, and from `malloc`'s point of view
|
||
nothing died. Both halves of the answer are about telling something that fact, and they tell different somethings.
|
||
|
||
The registry is told by `flan_dev_reg_dead_range`. What that buys is that a later read is *answerable*: a pointer into
|
||
a released region comes back dead and names what used to be there. That is for a person at an editor, in a dev build.
|
||
|
||
Memcheck is told by `FLAN_VG_MAKE_MEM_UNDEFINED` on the line beneath it, added later. What that buys is that the read
|
||
is *reported*, in every build, to whoever runs the sweep — the definedness bits are reset, so round two reading a byte
|
||
it never wrote is an uninitialised read rather than a silent reprint of round one's value.
|
||
|
||
**The two still must not be blurred into one claim.** They reach different people through different tools, and
|
||
neither is evidence for the other: the registry says nothing under memcheck, and the client request renders nothing at
|
||
an editor.
|
||
|
||
### What a release build actually carries, said honestly
|
||
|
||
"Release builds carry none of it" is the goal and it is not quite true, in exactly two places:
|
||
|
||
- **A load and a not-taken branch per free, and per `free-all`.** `flan_dev.c` is compiled into every build (see
|
||
`Build`, which says why), so `flan_rt.c` calls the dead-marking hooks unconditionally and each begins by testing a
|
||
flag only a dev build sets. The alternative is a second version of the allocator selected by a build flag, which is
|
||
worse than a branch for the reason the `Vec` header already carries its two dev words in every build: a layout or a
|
||
code path that changes with a flag is one that can disagree across the redefinition boundary silently.
|
||
- **Nothing else.** The table is `calloc`'d when it is armed, not declared as an array — a fixed 4096-entry table
|
||
would have been a quarter of a megabyte of BSS in a shipped game for something that build never writes. A release
|
||
binary carries a null pointer, a zero flag, and the declarations, which cost nothing.
|
||
|
||
The `@llvm.global_ctors` entry, the notes, and everything that reads them are dev-only, which is the same arrangement
|
||
the indirection cells and the shadow stack have.
|
||
|
||
### Tested twice, and one thing tested by hand
|
||
|
||
`programs/registry.flan` is **one program read twice** in the acceptance table. A dev build answers for an address at
|
||
the heap tier, the arena tier and the pool tier; a release build answers 0 to every question. The difference between
|
||
the two expectations *is* the assertion, and writing it as one program means nobody can change what a dev build does
|
||
without the release row noticing.
|
||
|
||
`test/programs/dev-ptr.flan` covers the inspector's pointer arm, and it is driven now rather than read by hand. Its
|
||
header still carries the two lines a session answers with; `test_dev.ml`'s *"a pointer the registry knows about"* case
|
||
asserts the live one whole and the dead one **around** its step number, which is the registry's event counter and
|
||
moves if anything allocates ahead of that program. It also asserts that no address appears in the epitaph — an
|
||
assertion that would otherwise depend on where the heap landed, which is the point of leaving one out.
|
||
|
||
## An address you have in your hand
|
||
|
||
The table above is the *recording* side. This is what reads it, and it is three things that turned out to be one
|
||
thing: pointing at a bare address, a breakdown by type, and what is still held.
|
||
|
||
### The recorded name, back to a type
|
||
|
||
The table records a **string**, and it has to. The note is built in `check.ml` at the allocation site, where the
|
||
concrete element type exists, and what crosses into the runtime is bytes — an ABI carrying a type would be an ABI that
|
||
had to agree with the checker's representation of one, which is the coupling the whole no-header-no-tag-word design
|
||
refuses.
|
||
|
||
What closes it is that the string is not a *description*. It is `Types.to_string` of the type, which is the **source
|
||
spelling** — `check.ml`'s `reg_note` says so, as the reason the name is worth printing at all — so the round trip is
|
||
the language's own reader, its own `Parse.texpr`, and the session's own `Check.resolve`. `Enemy` resolves against the
|
||
structs this session holds; `(Vec i32)` rebuilds through `Tapp`; `[3 i32]` through `Tarray`. **No table of spellings
|
||
is written down anywhere**, so nothing can fall behind `Types.to_string`.
|
||
|
||
And it is allowed to **fail**, which matters more than it looks. Not every recorded name is a type: `flan_rt.c` notes
|
||
a pool's slot headers as `"pool slots"`, because after a `free-all` an address landing in them must not come back as
|
||
an element. That string is not Flan source and must not become one, so a name that does not resolve is refused with
|
||
the name quoted and never defaulted to bytes.
|
||
|
||
### The address root renders a pointer, not a pointee
|
||
|
||
`(:op "at" :addr N)` — `M-x flan-inspect-address` — builds a `(Ptr T)` at the address and renders **that**. Rendering
|
||
the `T` directly would read the storage whatever the registry said, which is the hex dump this project is trying not
|
||
to be. Rendering the pointer puts the walk through `render.ml`'s pointer arm, which is the arm that asks first, so an
|
||
address root and a slot root reach the same two answers by the same code and permission is asked in exactly one place
|
||
in the compiler.
|
||
|
||
The one piece the thunk cannot do for itself is the address: **Flan has no integer-to-pointer cast**, deliberately, so
|
||
`flan_dev_reg_addr` is an extern beside `flan_agent_frame_slot` and for the same reason — the compiler knows the type
|
||
and something outside the language supplies the address. `flan_dev_reg_number` is the other direction, for a *program*
|
||
that has to say an address out loud; the language still has no operator for either.
|
||
|
||
Three refusals, each by name: an address the registry never saw **with no `:type` given** (a stack local, a global, a
|
||
pointer from C — the first two are answered by name already); an address the block's element size does not divide,
|
||
because rendering the element type there shows one element's tail as another's head, which is a plausible-looking
|
||
answer and therefore the worst kind; and a running program, because live-or-dead is exactly what a running program is
|
||
changing. A named `:type` overrides all of the first two — overriding is the point of being able to say it — and the
|
||
reply carries `:recorded` whenever the table had a name, so the disagreement is never silent.
|
||
|
||
**No `:path`.** A path steps from the pointee, and the pointee is what the registry has only just been asked to bless.
|
||
The whole answer here is the branch.
|
||
|
||
### One walk, two questions, and what "at exit" means
|
||
|
||
`flan_dev_reg_by_type` is the group-by, and there is one of it: a leak report **is** a breakdown with the dead left
|
||
out, and two walks would drift. Formatting is in the agent and ordering is in the daemon — biggest first, by bytes,
|
||
because a breakdown in table order is a list of everything and answers nothing.
|
||
|
||
**"At exit" is not a hook, and the honest reason is that a game is killed.** A program stopped by a signal runs no
|
||
`atexit` handler, no destructor, nothing — so no code written inside the program could report anything about the run
|
||
that matters most. The authoritative reader is therefore `(:op "leaks")`, which reads the same table over the agent
|
||
socket and can be asked at any moment, including the one before the kill. `flan-dev.el` does not ask on teardown
|
||
either: that would put a request on a path that runs every time the editor closes, for an answer nobody asked for.
|
||
|
||
The hook exists for the other program — the one that returns from `main` — and it is two decisions:
|
||
|
||
- **Registered by `atexit` from inside `flan_dev_reg_enable`**, not a file-scope `__attribute__((destructor))`. This
|
||
file is compiled into *every* build, so a destructor would run in a release build too, and that is exactly the third
|
||
place a release build is not free. Registered where it is, a release binary still carries a null pointer, a zero
|
||
flag and the declarations.
|
||
- **Off unless `FLAN_DEV_LEAKS` is set.** The acceptance table reads `programs/registry.flan`'s output with stderr
|
||
folded in, so a report nobody asked for is a report that changes what a dev build prints.
|
||
## A restart is not a transaction
|
||
|
||
Written down in three places rather than fixed, because it is a property and not a defect. If a frame mutates a global
|
||
and then signals, taking a `retry` **re-runs the mutation**. Control resumes at the `restart-case` and runs forward
|
||
from there; nothing is undone. Common Lisp has exactly this property and offers no help either — rollback would mean
|
||
journalling every store, which is a different language.
|
||
|
||
The discipline is that **the author chooses where the retry boundary is**. A `restart-case` at the top of a frame
|
||
re-runs everything including mutations already applied; one placed after the mutations re-runs only what follows them.
|
||
So: put the restart before anything mutates, make the retried section idempotent, or snapshot what will be re-applied.
|
||
§3 of `spec-conditions.md` places a restart *syntactically* — every clause body and the body share a type — and
|
||
nothing places it semantically.
|
||
|
||
It matters more here than in most Lisps because the intended use is a **game loop**, where the plan is to skip a frame
|
||
and carry on rather than die. That stopped being hypothetical when a failed bounds check began signalling
|
||
`BoundsError` instead of ending the process (see "An index out of range is a condition"): a frame can now be abandoned
|
||
and retried, which is exactly the case a non-idempotent mutation spoils.
|
||
|
||
Both `conditions.org` and `web/index.html` already carried the mechanical half — "a restart re-runs whatever sits
|
||
between it and the target" — as a one-line gotcha. Those were rewritten in place into the full statement rather than
|
||
having a second bullet added beside them, and the same reasoning went into `spec-conditions.md` §5, which is the
|
||
section that already enumerates what a transfer does and does not do: it runs `defer`s, it skips `errdefer`s, and it
|
||
does not undo. No numbered case changed meaning, so the freeze holds.
|
||
|
||
## Ghost text finds its anchor in the buffer, not in the table
|
||
|
||
`M-x flan-watch-ghost-mode` paints each watched value inline, after the line holding the call that wrote it, as an
|
||
overlay `after-string`. It is an **addition** to the watch buffer and not a replacement, and both can be on at once:
|
||
the buffer is the picture you want when you want everything, inline is the picture you want when you are reading one
|
||
function.
|
||
|
||
**Why it was thought to be blocked, and why it was not.** The earlier note in `flan-watch.el` — and two paragraphs
|
||
above, now struck — said ghost text needed a source location per table entry, which needs a generated call site,
|
||
which means the `(watch ...)` form in `check.ml`. Every step of that is true *of the table*: it holds a name and a
|
||
rendered string, and `(watch-i64 "ticks" ticks)` says what the value is called, not where it was written. The
|
||
conclusion did not follow. **The call site is in the buffer.** The name in the table is the string literal in that
|
||
call, so the editor searches its own text for the anchor instead of being told it. Nothing new is asked of the daemon,
|
||
and the composite-struct renderer is now the only one of the two things that really does want the checker arm.
|
||
|
||
**The head of the call cannot be hardcoded.** `watch-i64` is a name the *program's* author chose in their own
|
||
`declare-c`; only `flan_dev_watch_i64` is fixed, and the editor never sees it. Hence
|
||
`flan-watch-ghost-call-regexp`. Matching on the string literal rather than on the head is what makes that safe — a
|
||
name in the table came from one of these call sites by construction — and the syntax check (`nth 8` of a
|
||
`syntax-ppss`) is what stops a call *written inside a string* being taken for one.
|
||
|
||
**One reply, two pictures.** Ghost text does not poll. It is painted from `flan-watch--absorb`, the same function
|
||
that paints the buffer, from the same reply, so the two cannot disagree and there is no second `:op "watch"` in
|
||
flight — the one-request invariant `flan-dev-settle-hook` exists to keep. What had to change is that the **watch
|
||
buffer used to be the subscription**: killing it cancelled the timer and disarmed the table. That was right while it
|
||
was the only consumer and wrong the moment it was not, so arming and the timer now hang off
|
||
`flan-watch--consumers`, and only the last consumer out turns the lights off.
|
||
|
||
**Overlays are replaced wholesale on every repaint**, never followed through edits. That is the entire answer to the
|
||
invalidation problem the earlier note called the work: a line that moved cannot strand an overlay, because no overlay
|
||
outlives a tick. The cost is bounded by scanning only buffers **shown in a window** — an overlay nobody can see is
|
||
worth nothing, and the project's other files are not touched. It also means a file you scroll to is annotated within
|
||
one tick with nothing to hook.
|
||
|
||
**The questions the design had to settle, and the answers:**
|
||
|
||
- **Two sites, one name.** Both get the overlay, both show the same value, and the text says `one slot, 2 sites`. The
|
||
table has one slot per name and the last writer in the frame wins, so that *is* the value at both. Showing it at one
|
||
site would imply the other was not running; showing it at both in silence would read as a coincidence.
|
||
- **A watch inside a loop** shows the last value written, exactly as the buffer does. "The last of 4000 iterations" is
|
||
often not the interesting one, and every better answer is a UI for building a query — the one thing this design
|
||
exists not to have. Settled, not open.
|
||
- **Stopped.** The values survive a break, which is half the reason for pushing rather than polling. But they are the
|
||
*last frame's*, and inline they sit in code that looks perfectly live with no modeline beside them, so each one says
|
||
`last frame` and changes face. The buffer needs no such marker: you opened it deliberately and `flan:stopped(...)`
|
||
is already in view.
|
||
- **A site with no row** is annotated only when the table reports **overflow**, where it says so. Then the value
|
||
exists and was dropped, which is the case silence would send you hunting for a bug over; the buffer's overflow line
|
||
says the same thing without being able to say *which* name. With room to spare, a site with no row has simply not
|
||
run yet, and annotating every one of those at startup is noise.
|
||
- **A name the runtime clipped.** It holds 31 bytes, so a longer name in the source never matches a row exactly. The
|
||
lookup falls back to a prefix match, guarded on the row being a full 31 bytes so a short name cannot claim a site
|
||
that merely starts with it.
|
||
|
||
**What gets no ghost text, and it is the honest limit of anchoring on text:** a watch call produced by a macro, or one
|
||
whose name is not a literal. There is nothing in the source to find — the editor is reading the file, not debug info —
|
||
and both still appear in the watch buffer.
|
||
|
||
**One bug worth recording**, because it cost a debugging session and reads as impossible: `syntax-ppss` moves point and
|
||
clobbers the match data. Calling it inside a `re-search-forward` loop and then reading `match-string` or
|
||
`line-end-position` puts the scan back where it started, and the loop never ends. Everything wanted from a match is
|
||
now read out before the check, and the loop advances to a saved `match-end`.
|
||
|
||
`emacs/test-flan-watch.el` covers it, loaded from `test-flan-cider.el` for the reason `test-flan-mode.el` gives:
|
||
`emacs/*.el` is already a dependency of that dune stanza, so a new `.el` needs no build change. Nothing in it needs a
|
||
daemon — ghost text is a function from a table of rows and the text in a buffer to overlays, and both halves are
|
||
fixtures.
|
||
|
||
## A hot loop keeps five numbers, and the window is the editor's
|
||
|
||
The scalar watch above keeps one value per name. From a hot inner loop that is nearly useless: you see whichever of
|
||
the 91,200 cells happened to run last, and `watch.clj`'s own docstring says so — "from a hot loop you only ever see
|
||
whichever cell happened to run last — use `spy-long` there instead." This is that other half, and it is the piece
|
||
`PORTING.md` item 5 calls least obvious and most valuable.
|
||
|
||
`(watch-num-i64 "cell" (at grid i))` reaches `flan_dev_watch_num_i64` through the same plain `declare-c` the four
|
||
scalars use. No arm in the checker, no special form, nothing the compiler learns — the same deliberate non-ownership
|
||
of `check.ml` that the scalars were built under.
|
||
|
||
**What a slot keeps: count, min, max, last, mean.** The argument for those five is that each answers a question you
|
||
can ask without building a query. `n` is how many times the expression ran, which is the first thing that is wrong
|
||
when a loop is wrong — a count that tracks the frame counter rather than the cells is a loop that is not running.
|
||
`min` and `max` are the range, which is the thing a single sample can never show you and the thing you are looking
|
||
for when you suspect an index or a velocity is leaving the region it should stay in. `last` is the one sample, kept
|
||
because it is what the scalar watch would have given you and losing it would be a regression. `mean` is carried as a
|
||
running `sum` and divided at read time, because a mean accumulated as a mean drifts and a sum does not.
|
||
|
||
**What it deliberately does not keep: a ring, or a history.** A small ring of the last N samples was the other
|
||
candidate and it loses on the only ground that matters: N samples out of 91,200 is a sample of the *tail* of the
|
||
loop, not of the loop, so it answers "what did the last few cells do" when the question is "what did the cells do".
|
||
Beyond five numbers every richer answer is a UI for building a query, and a query builder is the one thing this
|
||
design exists not to be — the same reason the ghost text section gives for showing the last value in a loop rather
|
||
than offering to pick one.
|
||
|
||
**The write path does no formatting, and that is the whole feature.** An `snprintf` per sample at thousands of
|
||
samples a frame is a HUD that costs more than the game. A sample is a relaxed load, five compares and stores, and the
|
||
slot's seqlock; the *reader* — the agent's listener thread, once per editor tick — turns the five numbers into text.
|
||
`watch.clj` reaches the same place with a `double-array` per label and a `render` that Emacs calls, and the reasoning
|
||
is the author's rather than ours. It is also why the slot grew a `num` flag instead of a second table: the read path
|
||
already copies under a seqlock, so it copies five doubles instead of 192 bytes and renders them out of locals after
|
||
the counter check.
|
||
|
||
**The window is since the last reset, and this is a deliberate divergence from `watch.clj`.** There the stats are
|
||
cumulative until `reset-spies!` is called by hand. Cumulative is the wrong default for a frame loop: a `min` and a
|
||
`max` over a whole session reach the session's extremes within a few seconds of play and then never move again, so
|
||
the two most useful of the five go dead exactly when you start interacting with the thing you are debugging. This
|
||
tool exists to show you a number while you drag the mouse. So `flan-watch--tick` sends `:reset t` beside its read and
|
||
the displayed range is "since you last looked" — a fifth of a second, a dozen frames. A caller that wants the
|
||
cumulative numbers gets them by not resetting; the setting lives in the editor, not in the runtime.
|
||
|
||
**Reset is its own message, and it is not a side effect of reading.** A destructive read was the tempting shape and
|
||
is wrong: it makes *looking* change what is there, so anything that polls — a test's `await`, a second editor, a
|
||
person reading twice — silently shortens the window and gets a count that is noise. This was caught before the test
|
||
was written rather than after, which is the only reason the test has assertions about `n` at all. `watch reset` is a
|
||
line in `flan_agent.c` beside `watch on`/`watch off`, `:reset t` is a field on the read op, and `dev.ml` sends it
|
||
*after* the read so the tick reports the window it just closed.
|
||
|
||
**The reader never writes the table.** Reset bumps one global epoch counter and touches no slot; a slot clears itself
|
||
on its next sample, when it notices the epoch has moved, and does that *inside* its own odd-generation window so a
|
||
reader can never catch a half-cleared slot. The game thread stays the only writer of the table, which is the
|
||
invariant the whole of `flan_dev.c`'s watch section rests on. The cost is that a new window begins when the program
|
||
next runs rather than at the instant of the reset — and for a frame loop that is the only moment it could sensibly
|
||
begin. The test waits for it rather than reading once, and says why.
|
||
|
||
**The lazy clear stays lazy, and a stopped program is the case that decides it.** The obvious tidy-up is to make the
|
||
reader epoch-aware — compare the slot's epoch with the global one under the seqlock and report an untouched slot as
|
||
empty — which would make a reset visible in the very next tick instead of in the tick after the program's next
|
||
sample. It is the wrong trade, because **a stopped program does not sample**. An epoch-aware reader would blank the
|
||
watch for as long as the program sat in a break loop, and reading the numbers from the moment you stopped is the
|
||
entire point of stopping. The lazy clear gives exactly the right answer there. What was actually wrong was narrower
|
||
and lives in the editor: `flan-watch--tick` was sending `:reset t` five times a second at a program that could not
|
||
answer it. So the reset is now guarded on `flan-dev--stopped` — the read still goes out every tick, only the reset
|
||
field drops — and the runtime is untouched. That keeps the policy where the rest of this section already put it:
|
||
"since you last looked" is the editor's idea, not the table's. `watch_render_num`'s unreachable `n=0` arm is deleted
|
||
rather than commented, since the only way to reach it is the epoch check that was just rejected, and dead code is an
|
||
invitation to add one.
|
||
|
||
**An `i64` accumulates as a double**, so a magnitude past 2^53 loses precision in the sum and in the ends of the
|
||
range. Recorded rather than designed around: a count, a coordinate and a tile index are what this is pointed at, and
|
||
a second integer accumulator for a case nobody has would be two code paths for one tool. `watch-num-i64` and
|
||
`watch-num-f64` are two entry points only so a program need not cast at the call site, which is noise in the one
|
||
place this is meant to be droppable into.
|
||
|
||
**A whole number prints as one.** `watch.clj`'s `fmt-num` does this and the reason survives the port: a watch on an
|
||
array index that reads `66.0000` sends you looking for a rounding bug that is not there.
|
||
|
||
**Ghost text needed one character.** `flan-watch-ghost-call-regexp` was `watch\(?:-[[:alnum:]]+\)?` — one optional
|
||
hyphenated segment, which matches `watch-i64` and backtracks to failure on `watch-num-i64`, so a numeric watch got
|
||
no inline value at all while appearing normally in the buffer. A `*` for the `?` is the whole fix. It is worth
|
||
recording as the predictable cost of the decision that section defends: anchoring on the *name* rather than on the
|
||
head of the call is what makes the head a `defcustom`, and a `defcustom` with an enumerated default is a default that
|
||
needs widening when the set of heads grows. That is a cheaper failure than the alternative and it is not a free one.
|
||
|
||
## A breakpoint is a function call, and the editor only says where
|
||
|
||
`C-u` before an evaluation marks a form so the program stops when that form runs — Clojure's convention, and
|
||
DISCUSS.md §9's request. Nothing in the compiler knows what a breakpoint is.
|
||
|
||
**The mark is an ordinary `(pause)` call spliced into the tree.** `pause` is already in the prelude and is already
|
||
`error` under a `restart-case` with a `continue` clause, so an instrumented body is a body that calls one more
|
||
function, and the break loop it lands in is the one an unhandled condition already builds. A `paused : bool` on
|
||
`Ast.expr` would have had to be threaded through `Check`, `Tast` and `Emit` for a feature the prelude implements as a
|
||
function.
|
||
|
||
**It travels as a position beside the code, not as text spliced into it.** §9 proposed sending the top-level form with
|
||
the target span replaced by `(do (pause) <span>)`. That shifts every line and column after the insertion, and the
|
||
error overlays, `layout`, the break loop's frame locations and DWARF all read those. So the request carries
|
||
`:pause (LINE COL)` and the daemon applies it to the *declarations*, once parsing has attached the locations and
|
||
`Load` has qualified the names — `Ast.mark_pause`, pre-order, first hit wins. Applied after qualification, so a
|
||
package that defines a `pause` of its own cannot capture the synthesized call.
|
||
|
||
**Pre-order and first-hit because desugaring makes locations non-unique.** `parse.ml` gives several nested nodes the
|
||
same location — `(when c a)` becomes an `If` whose branch is a `Do` at the `when`'s own position. The outermost node
|
||
at that position is the one the editor pointed at, and the walk stops there.
|
||
|
||
**A whole `defn` cannot be wrapped, so marking one means stopping on entry.** `(do (pause) (defn ...))` is not an
|
||
expression. §9's first target therefore puts the call at the front of `fbody` instead, which is the same thing anyone
|
||
asking to stop at a function meant. `Ast.map_children` is exhaustive on purpose: a constructor left out would be a
|
||
form the mark silently cannot be set inside.
|
||
|
||
**A position matching nothing is refused.** Installing an unmarked body and answering `ok` would report a breakpoint
|
||
that is not there — the silent-success failure the session refuses everywhere else. The reply echoes `:pause
|
||
"LINE:COL"` on success, and the editor draws its overlay off that echo rather than off what it asked for, so it can
|
||
never show a mark the session declined.
|
||
|
||
**It sticks with no extra machinery.** The marked declaration is what goes into `Session.t.decls`, so it stays marked
|
||
until an evaluation replaces it — an ordinary `C-c C-c` over the same form with no `:pause`, or `C-c C-k` over the
|
||
buffer. That is §9's settled behaviour and it costs one statement that was already there. A parallel `paused` list on
|
||
the session would have been a second source of truth that drifts the first time some path replaces `decls` without
|
||
touching it, and clearing would have had to be written rather than falling out.
|
||
|
||
**`C-u C-x C-e` is a flag, not a position.** `flan-eval-last-sexp` sends a raw `buffer-substring` with no line
|
||
padding — unlike `flan-dev--text`, which pads a snippet back onto its own line — so buffer coordinates do not survive
|
||
that path. They are also not needed: the expression sent *is* the target, so `:pause t` says everything there is to
|
||
say, and `Session.eval_expr` wraps the parsed expression before `Check.expression`. It does not stick and cannot: a
|
||
thunk is built and thrown away, so there is no declaration for the mark to live in.
|
||
|
||
**The trap that path sets, and the reason `wait` is three-way.** `Dev.eval_expr` waits five seconds for the thunk to
|
||
produce a value and otherwise answers *"the program did not reach a frame boundary; is it calling (agent/poll)?"* —
|
||
which is exactly what a thunk parked at a breakpoint looks like from the daemon's side. Left alone it would report the
|
||
working feature as a failure, after a five-second stall. So `wait` answers `` `Value | `Stopped | `Timeout ``, and the
|
||
`Stopped` question is asked **only when a pause was requested**: without one, a thunk that stops did so by erroring,
|
||
and the timeout message is the answer that path has always given — `test_dev.ml` pins it. The `` `Stopped `` reply
|
||
carries no `:value`, because there is not one yet; `:stopped t :condition "Pause"` rides on it the way it rides on
|
||
every reply, from `with_break`.
|
||
|
||
**And it asks for `Stopped "Pause"` by name, not for "stopped at all".** The break loop allows evaluating, so this
|
||
path is reachable from a program already parked on something else — and a match on `Stopped _` would then fire on the
|
||
first iteration, answering for a thunk that has not run yet, on a reply whose own `:condition` names the *other*
|
||
condition. Asked by name it waits through the outer break until the thunk reaches its own `(pause)`, which the agent
|
||
reports because a nested break overwrites `condition_name` and restores it on the way out (`flan_agent.c`, around the
|
||
`status` verb). A program already parked on a `Pause` is the one case this cannot tell apart, and nothing could: both
|
||
answers are "stopped at a pause".
|
||
|
||
**The editor's column is a byte offset.** `flan-dev--wire-position` is the inverse of `flan-dev--position` and has to
|
||
count bytes for the same reason: the reader walks the source a byte at a time, so `current-column` would be short by
|
||
one per extra byte in every non-ASCII character earlier on the line and the daemon would find nothing where it was
|
||
pointed. The *line* is the buffer's own, which works because `flan-dev--text` pads the snippet with leading newlines.
|
||
|
||
**One key, three targets.** `flan-eval-defun` takes `C-u` for the innermost form point is inside — `backward-up-list`,
|
||
falling back to the defun when point is not nested — and `C-u C-u` for the top-level form itself. With `C-u C-x C-e`
|
||
that is all three of §9's targets and no new binding; `flan-mode.el` did not change.
|
||
|
||
**The overlay is an annotation, not feedback.** `flan-dev-pause-face` is drawn over the marked form and deliberately
|
||
does *not* copy the error overlays' lifetime. An error overlay is about the command that just failed and the next
|
||
keystroke takes it down; a pause mark is about the running program, and it has to survive `pre-command-hook` or the
|
||
buffer stops showing a breakpoint that is still there. What clears it is what clears the mark itself: an accepted
|
||
evaluation with no `:pause` on it, over a region that intersects it — and `C-c C-k` clears the whole buffer, because
|
||
every declaration in it was just replaced. The face inherits `warning` rather than `error`, agreeing with
|
||
`flan-cnr.el`, which already renders `Pause` that way: a breakpoint is a stop, not a failure.
|
||
|
||
**The test that matters is the second one.** `test_dev.ml`'s pause block marks `step`'s `(+ ticks 1)`, waits for
|
||
`:stopped t :condition "Pause"` and checks `continue` is on offer — and then re-evaluates the form *plainly*, takes
|
||
`continue`, and polls for half a second confirming it never stops again. A single sample after the resume proves
|
||
nothing: the frame that resumed is still inside the old marked body, past the `(pause)` call, so the first look reads
|
||
as running whether or not the mark was cleared. Half a second is about a hundred calls through the body just
|
||
installed. `test/programs/dev-pause.flan` exists because `dev-loop.flan` calls `step` four times, which is too tight
|
||
for that, and because a program that stops on its own — `dev-break.flan` — would prove nothing about what stopped it.
|
||
|
||
## A generic may key a map, and the predicate is what pays for it
|
||
|
||
The map operations are the second entry on the one list `check.ml` keeps of forms the abstract pass does **not**
|
||
answer where they are written. `print` and `println` were the first, and for an afternoon they were the only ones:
|
||
`hashable?` gated the *type* and not the operations, so a generic could take and return a `(Map $t V)` and could not
|
||
`get` or `put` into one.
|
||
|
||
**The reason was implementation, not design.** A map carries a hash and an equality, and `key_pair` emits them as
|
||
*concrete symbols* chosen from the key type — `flan_hash_str` for a string, `flan_hash_flat` for anything compared
|
||
bytewise, a generated `map/hash/Point` walking a struct's fields. While `$t` is still a variable there is no symbol
|
||
to name and nothing to choose between, so the abstract pass could not build the node. Falling through to the flat
|
||
pair would have been worse than refusing: it would hash a string's pointer and a struct's padding.
|
||
|
||
**What closes it is deferral, and what makes deferral safe is the `where` clause.** `put`, `get`, `has-key?`,
|
||
`map-remove!`, `reserve` and `clone` — the arms that reach `key_fns` — now check their arguments and then, when the
|
||
key is a type variable, return a placeholder of the operation's own type: `Unit` for `put` and `reserve`, `None` for
|
||
`get` and for `map-remove!` so the `(Option V)` around either still checks, `false` for `has-key?`, a zeroed map for
|
||
`clone`. The whole node is
|
||
thrown away with the rest of the abstract pass, exactly as `println`'s is, and the real one is built when the copy
|
||
is checked with `$t` concrete.
|
||
|
||
Every member of that list moves a refusal from the definition to a call site, which is the thing the abstract pass
|
||
exists to prevent, so **the membership rule matters more than the membership**. `print` and `println` pay nothing:
|
||
every type prints, there is no printability predicate because one would always hold, and the deferred check always
|
||
succeeds. The map operations *can* fail at a concrete type — a float key has no equality a map can use — and they
|
||
are on the list anyway because `{:where (hashable? $t)}` is in the signature. The refusal then has something to
|
||
point at: it lands at the call that asked for the type, naming the type, the predicate and the clause, and the
|
||
author of the generic wrote that requirement down. That is categorically different from an unconstrained
|
||
`(+ a b)` failing deep in a body with no signature to blame, which stays refused at its definition.
|
||
|
||
**So a generic that does not declare the predicate gets no deferral.** `deferred_key` checks `declares` before it
|
||
answers yes, and in practice `map_type` has already refused the signature where the type was written — `(Map $t
|
||
i32)` under `{:where (copyable? $t)}` is not a type. `key_pair`'s `Types.Var` arm survives as a backstop for a
|
||
route neither covers, and says so rather than claiming to be a design.
|
||
|
||
`test/programs/generics.flan` runs one written body at two key types; `test/programs/generic-map-reject.flan` is
|
||
the other half, a call site at `f64` refused against the clause.
|
||
|
||
## The begin/end pairs are macros now, and the pair is all they can promise
|
||
|
||
`vendor/raylib/modes.flan` — a second file in the raylib package with no `declare-c` in it, split out on
|
||
`vector.flan`'s reasoning exactly: `raylib.flan` is the package's statement about C and the file the header check
|
||
reads signatures out of, and nothing here can be made wrong by raylib changing because nothing here names C.
|
||
|
||
Five macros, one per pair the package binds: `with-drawing`, `with-mode-2d`, `with-mode-3d`, `with-texture-mode`,
|
||
`with-scissor-mode`. Each expands to `(do (begin-… args) body… (end-…))` — the calls the author used to type, in the
|
||
order they used to type them, with the argument evaluated once where it always was. No `let` and no `gensym`
|
||
anywhere: nothing binds a name, so there is no name a caller's could collide with, and introducing one in a
|
||
deliberately non-hygienic macro that does not need it is a step backwards. NEXT.md called these "three-line macros
|
||
once the expander lands" when it rejected `unwind-protect`, and they are.
|
||
|
||
They arrive qualified, the rule every declaration in a package follows: the importer writes `(rl/with-drawing …)`
|
||
and a bare `(with-drawing …)` is an unknown name. The expansions name `begin-drawing` and friends unqualified and
|
||
the expander qualifies them on the way out — `test/programs/pkgs/mac/mac.flan` is that rule's worked example, and
|
||
these are its first non-test use.
|
||
|
||
**What the macro removes is not what a reader assumes it removes.** It removes *the `End*` is missing, or is the
|
||
wrong one, or drifted away from its `Begin*` during an edit* — which is the whole of the class that actually bites,
|
||
and `EndMode2D` closing a `BeginMode3D` type-checks fine and corrupts the matrix stack. It does **not** remove a
|
||
body that leaves through the unwind path: a `return`, or an `invoke-restart` reaching an enclosing `restart-case`,
|
||
skips the rest of the `do` and the `End*` with it, exactly as it skipped a hand-written one.
|
||
|
||
`defer` is the obvious fix and is not available. It is a compile-time construct — the forms are copied into the
|
||
function's exit paths — so it is refused inside a loop body, and a begin/end pair lives inside the game loop
|
||
essentially always. The refusal was checked rather than assumed:
|
||
|
||
```
|
||
defer is not allowed inside a loop body — a defer is copied into every exit path of the function, so it
|
||
always registers and always runs at function exit.
|
||
```
|
||
|
||
Were it permitted there it would be wrong in the worse direction: one `EndDrawing` at function exit for N
|
||
`BeginDrawing`s, which is the silent-wrongness class the refusal exists for. A macro expanding to a `defer` would
|
||
therefore compile at a function body's top level and be refused in the one place anybody writes it. So the
|
||
discipline `sand.flan` already wrote down stays the discipline — **keep the restart boundary outside the pair**, so
|
||
choosing `continue` for a frame abandons the update and still reaches the drawing. spec-conditions §5 is the rule
|
||
behind that: a transfer runs the intervening frames' `defer`s and moves control, and there are none here to run.
|
||
`sand.flan`'s comment says so at the loop, and the macro's header says so at the definition.
|
||
|
||
**35 call sites converted** across `examples/` and `sand.flan`: 26 `with-drawing`, 5 `with-texture-mode`, 3
|
||
`with-mode-3d`, 1 `with-mode-2d`. Every begin/end pair in the corpus except one — `examples/core-scissor-test.flan`
|
||
turns its scissor on and off from a flag, so its `Begin` and its `End` sit in two separate `when`s with the drawing
|
||
between them and there is no body to wrap. That site is left hand-written, and it is the honest illustration of the
|
||
macro's limit rather than an oversight: a conditional pair is a shape a bracketing macro cannot express without
|
||
duplicating the body.
|
||
|
||
`sand.flan` converted its draw pair and stays at parity — `lisp/sand.lisp` writes `rl:with-drawing` there, which is
|
||
what makes it a port of the reference version rather than an addition to it. Its header note "no user-written
|
||
macros" now reads "no macros of its own", because `rl/with-drawing` is the binding package's, as in the Lisp.
|
||
|
||
`with-scissor-mode` is the one no example exercises, so `test/programs/rl-with.flan` does: all five expanded in a
|
||
`frame` function deliberately **unreachable from `main`**, which is what lets the case run with no libraylib at all
|
||
— `reach.ml` answers the link from the checked program, and `check` still walks every line. Nothing there draws,
|
||
because nothing headless can. `rl-with-reject.flan` is the arity half: a macro cannot signal while it expands, so
|
||
the prelude's idiom is to answer a symbol that is not a name and reads as the sentence the caller needs
|
||
(`with-mode-2d-takes-a-camera-and-a-body`), with the expander's note beneath it naming the macro at the call site.
|
||
|
||
## `slice-from-ptr` refuses in its own words, because its promise is the only context there is
|
||
|
||
`(slice-from-ptr p n)` shipped with a run-time check that was right and a message that was not. The check is
|
||
`icmp sge i64 n, 0`, signed, and it has to be signed: `check_slice`'s comparisons are unsigned, and a negative
|
||
`i32` sign-extended to `i64` is a huge `u64` that passes both of its clauses. What came out of it was
|
||
|
||
```
|
||
slice [0 -2) is out of bounds for length 0
|
||
```
|
||
|
||
— a range and a length the caller never wrote. The arithmetic behind the reuse was defensible: the condition
|
||
violated is `0 <= n`, which is a reversed range spelled the other way, which is why `@flan_slice_error` accepted
|
||
it. The sentence was not. It is about a container, and there is no container here.
|
||
|
||
**This is the one form in the language where the compiler cannot check the thing that matters.** Everywhere else
|
||
the length belongs to the compiler — an array has one, a slice carries one, a `Vec` stores one. Here the caller is
|
||
the only thing that knows how many elements live behind that pointer, and writing `n` *is* the promise. So its
|
||
refusal is the place that promise has to be spelled out, and it was the one place it was not.
|
||
|
||
`flan_slice_promise_error` in `runtime/flan_rt.c` now says it:
|
||
|
||
```
|
||
bounds.flan:34:28: slice-from-ptr was promised -2 elements behind the pointer, and a count of elements is never negative
|
||
the caller promises the pointer addresses n elements and nothing else can know it, so the sign of n is the whole of
|
||
what this check can see
|
||
```
|
||
|
||
Two lines, and the second one is deliberate: it says what is *not* checked, so that a caller does not read a trap
|
||
here as proof that the pointer itself was looked at.
|
||
|
||
It is shaped like the two beside it and not like `exit`. It signals `BoundsError` through `flan_bounds_signal`,
|
||
which walks the handlers and offers the break loop, and only falls through to the message and the status when
|
||
nothing answered — the same "an index out of range is a condition" rule, so a `flan dev` session survives one.
|
||
|
||
**The three condition fields are `(0, n, 0)`**, the violated condition written as a range, which is what those
|
||
fields can carry. Deliberately not `(0, n, n)`: that reads as a range in bounds, and a handler testing
|
||
`high <= length` would wave the failure through. One condition type still covers every bad index in the language,
|
||
so a handler writes one clause and not three.
|
||
|
||
On the emit side it is one `signal_block` call and one `declare`; nothing about the lowering of the form changed,
|
||
and it stays behind `f.md.checks` for the reason the other two do — dropping bounds checks is a release decision,
|
||
not an optimisation one, so it is on at `-O0` and `-O2` alike. `test/programs/bounds.flan`'s `-2` arm asserts the
|
||
new sentence at both, and the `Emit.program ~checks:false` assertion names the new symbol alongside the old two.
|
||
`lib/x86.ml` needs nothing: it lowers `flan_slice_error` for `slice`, and it has no `SliceFromPtr` arm at all.
|
||
|
||
## A session expands the buffer's own macros, and a `defmacro` joins it like a `defn`
|
||
|
||
`(tenfold 7)` at `C-x C-e` was an unknown name, and `(defn x [] i32 (tenfold 7))` at `C-c C-c` was the same
|
||
unknown name, in the very file that declares `tenfold`. The cause is one line of `Macro.program`: it collects
|
||
macros by scanning the forms it is handed, and an evaluation hands it the one form that was sent. The prelude's
|
||
macros worked because they are ambient, and an imported package's worked because `Session` holds them — the
|
||
buffer's own were the set nobody held.
|
||
|
||
**The design question was what "the buffer's own macros" means to a session, and the answer is the one already
|
||
written at the top of `lib/session.ml`: the declarations the program was built from, plus every change accepted
|
||
since.** Applied to macros, that is both halves of the fix and neither is optional:
|
||
|
||
- **Seeded in `Session.create`** from the forms `Load.program` was handed. That is the same read that produced
|
||
`t.decls`, not a second one — so `tenfold` is there from the first evaluation, and no macro can arrive from a
|
||
version of the file the session was never told about.
|
||
- **Added in `Session.eval`**, so a `defmacro` typed at the editor joins the set and the *next* evaluation can
|
||
call it. That is exactly the shape `defn` already has, and a `defmacro` was already an ordinary declaration on
|
||
this path — it parses to a `Defn` and installs a body like any other. The only thing missing was the session
|
||
remembering that the name is a macro.
|
||
|
||
**The boundary of that shape, written down rather than discovered:** the set only ever grows. Delete
|
||
`(defmacro tenfold ...)` from the buffer, reload, and `(tenfold 7)` still expands, because a union never removes.
|
||
That is not a leak to fix later — it is exactly what `defn` does. `Session.eval` keeps `kept @ added`, so a
|
||
function the file no longer declares is still in the session and still callable, and the running process still has
|
||
its body loaded. A session is what the program was built from plus every change *accepted*, and a deletion is not
|
||
a change anything sent. The one operation that forgets is restarting the session, which is what it is for.
|
||
|
||
**The session does not re-read the file, and that was the live alternative.** It has `~origin`, the buffer's own
|
||
path, and re-reading would pick up macros the session has never been told about. It would also read whatever is
|
||
*saved*, while the buffer on screen is whatever is *typed* — so an expansion would silently use a body the reader
|
||
is not looking at. Unsaved-versus-saved skew inside macro expansion is the quiet-wrongness class this codebase
|
||
refuses everywhere else, and it is worse here than elsewhere because a macro decides what the code *is*.
|
||
|
||
The commit point is untouched. The new set is computed into a `ref` at the top and assigned at `t.macros <- ...`
|
||
with everything else, below the checker — so a form that does not check leaves the session exactly as it was,
|
||
macros included. `test_session` pins that with a `defmacro` whose body calls an unknown function: the evaluation
|
||
is refused, and a later call to the name is still an unknown name.
|
||
|
||
Ordering is left-wins throughout, which is what `Load.macro_union` already encodes: the forms just sent, then what
|
||
`Load` just read off disk, then what the session was holding. That is what makes an *edited* macro expand with its
|
||
new body rather than with the stale copy.
|
||
|
||
**One thing had to change below `Session`.** `Macro.program` merged the ambient set with the file's own and said
|
||
in a comment that an import's names carry a slash, so nothing ambient could collide with a name written here. That
|
||
stopped being true the moment a session held unqualified names — and it stopped being true on the most ordinary
|
||
action there is: `C-c C-c` over a `defmacro` the session already knows sends a form declaring a name the ambient
|
||
set also has. Two forms declaring one name reach `Check.program` as a duplicate declaration, refused with a
|
||
sentence nobody would connect to editing a macro. The merge dedupes now, on the same left-wins rule and at the one
|
||
point that joins the two sets.
|
||
|
||
`Expand.quasiquote` is applied on the way in, and it is load-bearing rather than tidiness. `Parse.parse_forms`
|
||
desugars every form before the expander sees it and `Load.qualify_macro` desugars a package's macro on its way
|
||
out, but `Parse.imported_macros` is read by `Macro.program` directly, past that map. An undesugared body still has
|
||
its quasiquote in it, which makes a quasiquoted call look like a real one — the false ring the first cycle test
|
||
walked into.
|
||
|
||
The names stay unqualified. A buffer writes its own macro's bare name, so that is the name the session has to
|
||
answer to; a file inside a package the program also imports ends up holding both, the bare one from here and
|
||
`alias/name` from `Load`, which is what the two call sites each need.
|
||
|
||
**One shape stays refused, and its refusal was already right.** `(defmacro ...)` at `C-x C-e` is a declaration,
|
||
and `Parse.expr`'s head dispatch already names `defmacro` first among the heads it refuses — "defmacro is a
|
||
top-level declaration, not an expression". It matters more now than when it was written: before this, nobody would
|
||
type a `defmacro` at `C-x C-e`, because the session forgot it either way. Now that one typed at the editor means
|
||
something, reaching for `C-x C-e` on it is a reflex, and what comes back says what it is rather than complaining
|
||
about an unknown function. `test_repl` pins it beside the `defvar` case.
|
||
|
||
Both editor paths work and they are different wraps — `Parse.expr`'s for `C-x C-e` and `Parse.decl`'s for
|
||
`C-c C-c`. `test/test_session.ml` covers both, plus a `defmacro` the file never had followed by a call to it,
|
||
plus editing that macro and calling it again; the assertions are on the IR and not on the absence of an
|
||
exception, because an expression that did not expand raises while one that expanded to the *wrong* thing does
|
||
not.
|
||
|
||
## `C-c C-m`: what a macro call expands to, and a printer for `Form`
|
||
|
||
Macros grew far enough today that the editor had to be able to ask. The argument is one line: **a macro is
|
||
importable from a package, arriving qualified**, so `(rl/with-drawing …)` is a call whose `defmacro` lives in
|
||
another directory. There is nothing beside the call site to read — and there would be nothing to read even if the
|
||
`defmacro` were in the buffer, because a macro answers a `Form` and nobody ever wrote that down.
|
||
|
||
### One step on the key, all the way under `C-u`
|
||
|
||
Both, because the difference is real here and the second is nearly free once the first exists. A macro may
|
||
quasiquote a call to another macro — `mac/quad` answers `(mac/twice (mac/twice n))` — so one step and the fixpoint
|
||
are different text.
|
||
|
||
One step is the *bare* key, and the reason is `Loc.from_macro`. It is outermost-wins by design: the macro the
|
||
author actually wrote is the one worth naming. So by the time a full expansion settles, every node of it is stamped
|
||
`mac/quad` and `mac/twice` is unnameable. All the way is the answer the compiler acts on; one step is the only way
|
||
to find out which macro produced what. `C-u` for the other half rather than a second key is the rule `C-c C-a`
|
||
already follows: it is the same question asked of the same form.
|
||
|
||
One step is **outermost-only**, and that is a deliberate divergence from `Macro.expand_form`, which expands a
|
||
call's arguments before calling it. `(mac/twice (mac/twice 1))` one-stepped here is `(+ (mac/twice 1) (mac/twice
|
||
1))`; the compiler's own first move is `(mac/twice (+ 1 1))`. Different intermediates, the same fixpoint. Written
|
||
down because someone will otherwise discover it and file it.
|
||
|
||
### The session's macros, not a fresh read
|
||
|
||
`Session.macroexpand` reads `t.macros` and writes nothing at all. Re-reading the file would answer with macros the
|
||
session was never told about, and with whatever is *saved* rather than what is typed — the unsaved-versus-saved
|
||
skew "A session expands the buffer's own macros" already refused, and it is worse here than anywhere, because an
|
||
expansion that disagrees with what an evaluation does is a lie about what the code *is*.
|
||
|
||
Writing nothing is a property and it is pinned: a `defmacro` handed to `C-c C-m` must not join the session by
|
||
having been looked at, so `test_session` and `test_repl` both expand one and then require the name to still be
|
||
unknown. `C-c C-c` is where a declaration goes.
|
||
|
||
### The refusals, and the one that must not be reachable
|
||
|
||
The verb sits inside the guard the robustness lane put round the whole of `handle` in `Dev.serve` — confirmed
|
||
rather than assumed, and it is why `Dev.macroexpand` has no `Loc.Error` arm of its own. Both non-termination
|
||
refusals raise `Loc.Error` and come back as replies with the call site on them.
|
||
|
||
**The bound has to be reached, and only where it is needed.** A hang in the daemon wedges the editor with the
|
||
program still on screen. So: all the way runs `Macro.expand_form`, hits `Macro.fuel` and names the macro; one step
|
||
makes exactly one call and does not look at what comes back, so `(s/spin)` one-stepped *answers*, with `(s/spin)`
|
||
and the name `s/spin`. A command that refused to show anybody the thing they were trying to see would be the wrong
|
||
kind of safe. The ring is refused a level up, while the package holding it is parsed, so no session over one can
|
||
exist and nothing on this path can reach it.
|
||
|
||
`Macro`'s module handling is now a `Fun.protect`. It used to be two statements after the call, which was enough
|
||
while a build was the only caller — a build that raised was a process about to exit. The daemon is not that
|
||
process: a macro that does not settle raises, the editor is told, and the next `C-c C-m` does it again, so
|
||
`Dynload.owned` must not be a list that only grows over a session's lifetime.
|
||
|
||
### `Form.to_source` and `Form.pretty`, because no printer existed
|
||
|
||
`Form.to_string` is an error-message renderer — one line, no width, `%S` and `%g`. It is also what `Macro.key`
|
||
digests, so every cached macro module on disk is keyed by what it prints; it is untouched. What an expansion needs
|
||
is text a *reader reads back*, and three of its spellings are not round trips, all reachable because a macro may
|
||
build any literal: `%g` prints 1.0 as `1` and truncates at six digits, `%S` is OCaml's escaping where the reader
|
||
takes exactly six escapes, and `Byte` falls through to `\<char>` where the reader names `\nul` and `\return`.
|
||
|
||
`Form.pretty` decides **where the line breaks go and not where the columns do**. Structure is a printer's job;
|
||
indentation is `flan-mode`'s, which is where this project's rules already live — the buffer runs `indent-region`
|
||
over what it is shown, so nothing in OCaml has to know that a `let` aligns its bindings under the bracket. A client
|
||
with no Emacs still gets something readable rather than one very long line.
|
||
|
||
### Where the answer goes
|
||
|
||
`*flan-macroexpansion*`, a read-only `flan-mode` buffer — what is in it is Flan source and reading it is the whole
|
||
point, so the font-lock and the indentation are the ones already there. It follows `flan-disassemble`'s shape,
|
||
header comment lines and all, rather than `flan-cnr`'s: cnr's folding is frame-and-locals machinery with its own
|
||
text properties and there is nothing here to fold. What *was* taken from cnr is the idea its header names — a key
|
||
that expands in place. `m` (or `C-c C-m`) takes the form at point one more step and puts the result where it was,
|
||
which is what makes one-step-by-default usable rather than a thing you press once and lose. `a` goes to the
|
||
fixpoint, `g` asks again — useful after re-evaluating the `defmacro` — and `q` closes.
|
||
|
||
Three keys inherited from `flan-mode-map` are shadowed by name, not unbound, so pressing one says why: `C-c C-c`,
|
||
`C-c C-k` and `C-x C-e` over an expansion would install a body nobody wrote under a name somebody did.
|
||
|
||
**What `Loc.from_macro` means for what is printed** is a header line, because there is nowhere else it could be
|
||
said. Every node below it carries the *call site's* file, line and column with the macro's name stamped on it —
|
||
which is how an error inside expanded code points at the call you wrote — and therefore none of it has a location
|
||
of its own, nothing in it is the text of any file, and there is nothing for `M-.` to jump to.
|
||
|
||
### What gets sent, and the padding that is not `C-x C-e`'s
|
||
|
||
The region is `C-x C-e`'s, plus one case it has no need of: the form *at* point when point is on its opening
|
||
delimiter. A macro call is a form you put point on, and `backward-sexp` from an open paren takes the previous
|
||
sibling, which is never what was meant.
|
||
|
||
The text is padded onto its own line **and its own column**, which `flan-dev--text` does not do and says why it
|
||
does not: a top-level form starts at column 1, so the columns already agreed. A macro call does not — it is
|
||
written well inside a `defn` — and the refusal this path can get carries a column measured from the start of the
|
||
snippet, so an unpadded send would draw "did not settle" at the start of the line. Leading newlines and leading
|
||
spaces are both whitespace the reader skips. `test-flan-dev.el` checks the arithmetic the only way that proves it:
|
||
it evaluates a spinning `defmacro` into a live session, asks for the fixpoint of a call to it from a known buffer
|
||
position, and requires the error overlay to start at exactly that position.
|
||
|
||
The in-buffer `m` sends its text *un*padded, which is the honest shape there: the expansion is in no file, so
|
||
there is no line or column for a refusal to be drawn at. The `:file` still goes on the wire, because that is what
|
||
says which session's macros to expand against.
|