1653 lines
124 KiB
Markdown
1653 lines
124 KiB
Markdown
# How the parts that exist work
|
||
|
||
The reasons behind the code, kept separate from `NEXT.md` so that what to do next is not buried under what was already
|
||
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever `dlclose`d, why a call
|
||
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
|
||
than a function in the runtime. Deleting it would mean deriving it again.
|
||
|
||
`NEXT.md` is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
|
||
first. Come here when you need to know why something is the shape it is.
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a
|
||
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top
|
||
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer
|
||
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
|
||
bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at
|
||
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot,
|
||
so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed
|
||
type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is
|
||
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets
|
||
the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type
|
||
in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
|
||
per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
|
||
in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning
|
||
garbage rather than as a link error.
|
||
|
||
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
|
||
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
|
||
copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare`
|
||
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
### The wrappers are generated now — `declare-c`, `lib/shim.ml`
|
||
|
||
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
|
||
generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory
|
||
holds `raylib.flan` and `link` and no C at all.
|
||
|
||
One binding is now one line:
|
||
|
||
```
|
||
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
|
||
```
|
||
|
||
`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
|
||
any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the
|
||
function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an
|
||
ordinary Flan `defn` above it when the signature has a struct in it. `flan shim <file>` prints the whole file.
|
||
|
||
**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path
|
||
string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h
|
||
i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural
|
||
rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited.
|
||
|
||
**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same
|
||
`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what
|
||
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
|
||
that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real
|
||
signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was
|
||
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
|
||
for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct
|
||
— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves.
|
||
Everything where they could diverge is refused at the field.
|
||
|
||
**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a
|
||
scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the
|
||
hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against
|
||
the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree.
|
||
|
||
**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path,
|
||
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
|
||
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
|
||
malloc failure, where the alternative is handing C a null pointer.
|
||
|
||
**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice
|
||
and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is
|
||
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
|
||
recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes
|
||
`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned
|
||
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
|
||
the reason, and the acceptance table asserts on the reasons.
|
||
|
||
**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc +
|
||
`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
|
||
that calls an existing binding is unaffected.
|
||
|
||
`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
|
||
`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17
|
||
34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity
|
||
would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed.
|
||
|
||
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write.
|
||
|
||
Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows.
|
||
|
||
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
|
||
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
|
||
GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins
|
||
`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all —
|
||
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`,
|
||
`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four
|
||
ways for that reason.
|
||
|
||
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
|
||
library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora
|
||
ships the runtime library without the `.so` symlink.
|
||
|
||
### What a headless FFI test can and cannot pin
|
||
|
||
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
|
||
|
||
- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C
|
||
writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and
|
||
getting identical output.
|
||
- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the
|
||
way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance,
|
||
passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from.
|
||
Distances are worse: the reflection does not even reach them.
|
||
- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does
|
||
not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision
|
||
cases look like they cover it.
|
||
- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different
|
||
field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.
|
||
|
||
- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available.
|
||
`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values
|
||
in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely,
|
||
including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU
|
||
reads its width, height or mipmaps at all.
|
||
- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image
|
||
`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and
|
||
answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more.
|
||
- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the
|
||
package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the
|
||
`x`/`y` permutation above.
|
||
|
||
The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail
|
||
by permuting the fields and watching it go red. A case not verified that way is decoration.
|
||
|
||
One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It
|
||
measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no
|
||
window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime`
|
||
and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are
|
||
exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is
|
||
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares
|
||
becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
|
||
declarations whose names contain a slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file
|
||
named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below.
|
||
|
||
**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also
|
||
a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import
|
||
all four. A file carries no `.c` and no `link` file; those belong to a directory.
|
||
|
||
**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that
|
||
is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under
|
||
one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which
|
||
is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time,
|
||
and the namespace being flat, mutually dependent packages simply work. The same directory under two *different* aliases
|
||
is refused.
|
||
|
||
**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment
|
||
anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would
|
||
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
|
||
checker it would be "unknown name", which is true and useless.
|
||
|
||
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
|
||
|
||
## The link follows the program
|
||
|
||
`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the
|
||
importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that
|
||
link cannot succeed.
|
||
|
||
`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it,
|
||
follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing
|
||
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
|
||
|
||
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
|
||
`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs**
|
||
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
|
||
dropped one is a silently different program.
|
||
|
||
**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far.
|
||
|
||
The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and
|
||
`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker
|
||
arguments together, so a caller cannot take one without the other.
|
||
|
||
## sand.flan is one program
|
||
|
||
It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a
|
||
program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both
|
||
drivers imported it.
|
||
|
||
Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports
|
||
`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
|
||
raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both
|
||
targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation.
|
||
|
||
What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no
|
||
input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and
|
||
`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
Two claims that got run together in an earlier note, for the record:
|
||
|
||
- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the
|
||
**wasi** path, which is what the headless table targets, and which has no GL and no browser.
|
||
- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with
|
||
`emscripten_set_main_loop` instead of a `while`. ~~That is a different `main`, not a different program.~~ **The premise
|
||
held and the conclusion did not.** It is the same `main` and the same program: `-sASYNCIFY` answers the same browser
|
||
fact without cutting anything in half. See "The browser is the third target" below.
|
||
|
||
**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`,
|
||
and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`.
|
||
- `(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what
|
||
cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three
|
||
check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo
|
||
<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the
|
||
same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor
|
||
that split them would break the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s
|
||
unchecked. Indistinguishable.
|
||
|
||
## Sanitizers, and why hand-written IR does not get them for free
|
||
|
||
`--sanitize` builds the whole program under ASan and UBSan — the runtime's C, the generated shim, and the Flan. The last
|
||
of those is not what passing `-fsanitize=address` to the clang run over the `.ll` gets you, and the gap is silent.
|
||
|
||
**AddressSanitizer is an LLVM pass, but it instruments only functions carrying the `sanitize_address` attribute.** That
|
||
attribute is put there by clang's C frontend. `Emit` writes `.ll` by hand, so it wrote none, so the pass walked past
|
||
every Flan function and instrumented `flan_rt.c`. The measurement that settled it: an out-of-bounds read of a `defvar`
|
||
array in a `--no-bounds-checks` build printed its garbage and exited 0; with an `attributes #0 = { sanitize_address }`
|
||
group named on every `define`, the same program reports `global-buffer-overflow in flan.main`. Globals are the exception
|
||
— the module pass redzones them whether or not any function is attributed — which is why the *shape* of a sanitized
|
||
build looked right long before it worked.
|
||
|
||
**UndefinedBehaviorSanitizer has no equivalent lever.** Its checks are not a pass: the C frontend emits branches to
|
||
`__ubsan_handle_*` inline, and no attribute asks anything to produce them. So UBSan covers the C and nothing else, and
|
||
`(<< 1 32)` is still unremarked under `-fsanitize=undefined`. Shift UB, alignment and the f32→i32 cast on NaN are
|
||
therefore a compiler feature if they are wanted — checks emitted from `Emit` behind the flag, the same shape the bounds
|
||
checks already have — and not a flag away. `test_sanitize` pins both halves with controls: one program that must report
|
||
and one that must not, so either fact changing is a test failure rather than a discovery.
|
||
|
||
`-fno-sanitize=signed-integer-overflow` is the only exclusion, because wrapping is what this language's arithmetic
|
||
means and without it every program trips on its first `+`.
|
||
|
||
**The flag deliberately does not force `-O0`,** unlike `--debug`, whose reason (mem2reg deletes the alloca a
|
||
`llvm.dbg.declare` describes) does not apply. The optimiser is half of what is being measured, and `bounds.flan` proves
|
||
it: with checks off, its read past the end of a string constant is reported at `-O0` and silent at `-O2`, because an
|
||
out-of-bounds `inbounds` getelementptr into a constant is poison and LLVM folds the load away. The program then prints a
|
||
wrong answer instead of touching memory. Same family as `(<< 1 32)` compiling to a bare `retq`.
|
||
|
||
**What ASan covers of the bounds checks' job, since `--sanitize --no-bounds-checks` is the run that asks.** Three of
|
||
`bounds.flan`'s six deliberate out-of-bounds cases are caught. A negative index into a global is not, and the reason is
|
||
layout rather than anything about the access: ASan lays a global out as `{data, redzone}`, so reading before one lands
|
||
in whatever precedes it, which is a redzone if something instrumented is there and ordinary memory if nothing is.
|
||
Measured both ways — silent in `bounds.flan`, reported as soon as another `defvar` is declared in front of `arr`. A
|
||
reversed slice is not caught either, having computed a negative length and read nothing at all. And ASan sees
|
||
out-of-*object* access, not out-of-subobject, so a slice into the middle of a larger array can overrun its logical
|
||
bounds without crossing a redzone. Three of six is a ceiling on what it covers, not a measurement of the risk. It is a
|
||
second net, not a replacement.
|
||
|
||
The sweep lives on its own dune alias rather than on `dune test`: a sanitized program links to a statically linked 1.8MB
|
||
binary, and twenty-eight of them twice over is minutes against the suite's seconds.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had
|
||
expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is
|
||
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
|
||
criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the
|
||
semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is*
|
||
the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody
|
||
addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a
|
||
`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding,
|
||
so wasm32 needs no exception proposal.
|
||
|
||
## The reload primitive — dev loop steps 1 and 2, measured
|
||
|
||
`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled
|
||
into its own object and called inside a process that is already running, twice, with a changed body the second time.
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| `Emit.redefinition` | below the timer (<0.1ms) |
|
||
| `llc -O2 -filetype=obj` | 15–17ms |
|
||
| `ld -shared` | 3ms |
|
||
| `dlopen` + `dlsym` | **0.04ms** |
|
||
|
||
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
|
||
number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost,
|
||
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since
|
||
the driver absorbs IR the bare tools reject.
|
||
|
||
`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that
|
||
*is* the mechanism. What the new module does **not** define is the whole design:
|
||
|
||
- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a
|
||
redefinition can change a function's body and can never re-initialise the program's data. Define the global and the
|
||
loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the
|
||
thesis.
|
||
- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a
|
||
private copy of it.
|
||
- **no `main`.** This module is loaded, not started.
|
||
|
||
Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss
|
||
because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is
|
||
spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a
|
||
slice parameter.
|
||
|
||
**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the
|
||
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
|
||
agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release
|
||
builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_read`, which reads as a compiler
|
||
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and
|
||
the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason.
|
||
|
||
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a
|
||
loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property
|
||
of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol.
|
||
|
||
Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**,
|
||
since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since
|
||
`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
|
||
`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares
|
||
`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
|
||
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
|
||
|
||
String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's —
|
||
worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would
|
||
fail at link time. The fixtures each print a literal so that path is actually exercised.
|
||
|
||
### Cells — how a call site follows a redefinition
|
||
|
||
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes
|
||
every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current.
|
||
|
||
```
|
||
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
|
||
%p = load ptr, ptr @"flan.cell.bump" ; every call site
|
||
%r = call i64 %p()
|
||
```
|
||
|
||
Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and
|
||
exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what
|
||
makes a frame-boundary swap a non-event.
|
||
|
||
The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of
|
||
one.
|
||
|
||
Four things about this that are not free choices:
|
||
|
||
- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on
|
||
whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are
|
||
separate on purpose.
|
||
- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to
|
||
*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would
|
||
publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage,
|
||
because the failure is silent.
|
||
- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined
|
||
function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture
|
||
recurses on purpose, and would print the old body's text if it did not.
|
||
- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is
|
||
the first time `opts` means something semantic rather than an optimisation level.
|
||
|
||
LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46
|
||
indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well;
|
||
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
|
||
|
||
### Names that did not exist when the process started
|
||
|
||
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF
|
||
cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else:
|
||
|
||
```
|
||
void **flan_dev_cell(const char *name); /* a new function's cell */
|
||
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
|
||
```
|
||
|
||
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
|
||
new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores
|
||
on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
|
||
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
|
||
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
|
||
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
|
||
common case pays nothing for the general one.
|
||
|
||
**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c
|
||
C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a
|
||
redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet.
|
||
|
||
Four rules, each of which is a silent failure if broken:
|
||
|
||
- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are
|
||
still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`.
|
||
- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the
|
||
old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so.
|
||
This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
|
||
- **Nothing that published anything is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it
|
||
leaves every call site pointing at unmapped memory. That is a constraint on the agent too. The rule is about being
|
||
*pointed into*, which is why it has exactly two exceptions and they are not exceptions to the reasoning: a transient
|
||
thunk, which takes no registry slot and has returned; and a module the agent refuses before queueing it — no installer,
|
||
or no room in the ring — which published nothing and which nothing can name. What was leaked in the second case was the
|
||
handle *value* rather than the mapping: `dlopen` refcounts by path, so re-sending the same bad file raised a count
|
||
nothing could lower, and the one reference that could was dropped on the floor.
|
||
- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed
|
||
capacity with a loud failure rather than growable.
|
||
|
||
The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at
|
||
run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell*
|
||
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
|
||
pass and the transcript would read 246 instead of 432.
|
||
|
||
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout
|
||
calculator in OCaml that would have to agree with LLVM's on every target.
|
||
|
||
### The agent — dev loop step 3
|
||
|
||
`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link`
|
||
asks for `-lpthread`.
|
||
|
||
```
|
||
(agent/start path) listen on a unix socket; once, at startup
|
||
(agent/poll) install whatever has arrived; returns how many
|
||
(agent/wait ms) the same, but waits for something first
|
||
```
|
||
|
||
The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded —
|
||
so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a
|
||
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
|
||
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
|
||
loader.
|
||
|
||
**A full ring is refused, at the sender, before the `dlopen`.** Of the three honest answers this is the only one that
|
||
reaches the person who asked: dropping loses a reload the sender was told was `ok`, which is the same lie more quietly,
|
||
and blocking stalls the accept loop — it serves connections inline, so a program that had stopped polling would also
|
||
stop answering `status` and `abort`, leaving the dev loop with no way to reach a program that had stopped listening to
|
||
it. The check is separate from the store because there is exactly one producer: room, once seen, cannot be taken away,
|
||
since the consumer only ever makes more of it. Sixty-four is a lot of reloads between two frames and the refusal says
|
||
what to do about it — call `agent/poll`.
|
||
|
||
**The way out of the break loop is `_exit`, not `exit`.** `exit` runs the atexit chain and the ELF destructors, which
|
||
want the loader lock the listener thread may be holding inside `dlopen`; a program asked to abort would hang instead of
|
||
dying, which is the failure the break loop exists to replace. The streams are flushed by hand at each call site, and 134
|
||
stays because that is what a trap exits with.
|
||
|
||
`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits
|
||
for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first
|
||
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
|
||
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
|
||
|
||
Two details found by running it:
|
||
|
||
- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a
|
||
program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at
|
||
all, which is how this was found.
|
||
- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program
|
||
can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`.
|
||
- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows
|
||
when it is between frames.
|
||
|
||
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under
|
||
Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body
|
||
that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is
|
||
refused on the listener thread and the loop never notices.
|
||
|
||
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the
|
||
names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what
|
||
the session will track automatically.
|
||
|
||
### The session
|
||
|
||
`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change
|
||
accepted since.
|
||
|
||
**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a
|
||
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
|
||
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that
|
||
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
|
||
|
||
Two things the session knows that no single evaluation could:
|
||
|
||
- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST,
|
||
because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `println` reads
|
||
as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
|
||
- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded:
|
||
|
||
| Change | What it would have broken |
|
||
|---|---|
|
||
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below |
|
||
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
|
||
| a struct's fields | the values the process is holding have the old layout |
|
||
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it |
|
||
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
|
||
|
||
**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should
|
||
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
|
||
existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked
|
||
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
|
||
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
|
||
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
|
||
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
|
||
should not be read as the final answer.
|
||
|
||
A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past
|
||
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global`
|
||
record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than
|
||
the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is
|
||
the same fact that makes them unreloadable.
|
||
|
||
Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first,
|
||
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
|
||
uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for
|
||
that reason.
|
||
|
||
A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A
|
||
dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a
|
||
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
|
||
is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the
|
||
folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could.
|
||
|
||
**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in
|
||
`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s
|
||
own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a
|
||
brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it
|
||
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
|
||
did nothing.
|
||
|
||
It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a
|
||
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
|
||
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
|
||
either.
|
||
|
||
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise
|
||
re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name
|
||
pass rejects it. `C-c C-k` on sand.flan's own text is the test.
|
||
|
||
`flan reload <program.flan> <forms.flan>` is that path from the command line: a session over the program the process was
|
||
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw`
|
||
and 910 consecutive frames drew it.
|
||
|
||
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
|
||
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
|
||
has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to
|
||
`<eval>`, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
|
||
the real buffer path, which is the same key CIDER's `eval` carries.
|
||
|
||
### The daemon — `flan dev`
|
||
|
||
`flan dev <program.flan>` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside
|
||
the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part
|
||
of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe
|
||
the process that is actually running rather than a guess about it.
|
||
|
||
**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client
|
||
became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form,
|
||
from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the
|
||
editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of
|
||
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
|
||
newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor.
|
||
|
||
```
|
||
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
|
||
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
|
||
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
|
||
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
|
||
(:op "close")
|
||
```
|
||
|
||
`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the
|
||
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
|
||
It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing
|
||
else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none
|
||
to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself,
|
||
which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step
|
||
[i64 f32] i64`: types only.
|
||
|
||
The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan
|
||
dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by
|
||
guessing which directory it was relative to.
|
||
|
||
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
|
||
— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one
|
||
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
|
||
|
||
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so without it every error an editor shows
|
||
points into a file that does not exist.
|
||
|
||
Two things the daemon must not paper over, both of which would look like a successful evaluation:
|
||
|
||
- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan
|
||
says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead
|
||
fails silently: the module compiles, is built, and nobody receives it.
|
||
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed
|
||
connect or a reply that is not `ok` becomes an error the editor sees.
|
||
|
||
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads
|
||
like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather
|
||
than leaving an editor waiting on a socket nobody is serving.
|
||
|
||
### The Emacs client
|
||
|
||
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is
|
||
s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is
|
||
Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is
|
||
written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords.
|
||
|
||
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a
|
||
request and `read` reads a reply.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `C-c C-c` | the top-level form at point, recompiled and installed |
|
||
| `C-c C-k` | the whole buffer, as **one** module |
|
||
| `C-x C-e` | the expression before point, evaluated *in the running program* |
|
||
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
|
||
| `C-c C-o` | the running program's own output, in `*flan-output*` |
|
||
| `C-c C-r` | a prompt on the running program (`*flan-repl*`) |
|
||
| `C-c C-b` | what a **stopped** program is offering, and which to take |
|
||
| `C-c C-d` | what the running program currently defines |
|
||
| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR |
|
||
| `M-.` / `M-,` | where a name is written, through an `xref` backend |
|
||
|
||
eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on
|
||
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
|
||
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just
|
||
installed completes at once.
|
||
|
||
The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away,
|
||
which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the
|
||
socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the
|
||
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
|
||
|
||
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to
|
||
arrive in the same load, or the first refers to storage that does not exist yet.
|
||
|
||
**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary,
|
||
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
|
||
has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives
|
||
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
|
||
mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml`
|
||
and fails here.
|
||
|
||
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
|
||
next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte
|
||
offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
|
||
`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the
|
||
daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1
|
||
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
|
||
reader skips, so the reply's line numbers are the buffer's own.
|
||
|
||
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
|
||
success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the
|
||
one point looked like it was in.
|
||
|
||
**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to
|
||
into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form
|
||
"asm"|"ir")` is the op.
|
||
|
||
What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d
|
||
--disassemble=flan.<name>` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared`
|
||
deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was,
|
||
so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else
|
||
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
|
||
bookkeeping.
|
||
|
||
**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket
|
||
takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address
|
||
space, and `C-x C-e` renders a pointer as `<ptr>` on purpose — so nothing the daemon can ask would tell it what a cell
|
||
holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first
|
||
instruction:
|
||
|
||
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*;
|
||
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
|
||
- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed,
|
||
since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further
|
||
installs until it resumes.
|
||
|
||
From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch
|
||
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
|
||
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
|
||
the debug build lands, that is the line to delete.
|
||
|
||
**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an
|
||
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
|
||
`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the
|
||
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
|
||
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
|
||
in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the
|
||
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
|
||
is Emacs' contract and a test claiming to check it would be checking nothing.
|
||
|
||
**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the
|
||
next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the
|
||
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
|
||
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
|
||
accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
|
||
grow the daemon without limit.
|
||
|
||
### `C-x C-e` — evaluating an expression
|
||
|
||
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
|
||
into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by
|
||
exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an
|
||
expression reading the program's state sees a point the program agrees is consistent.
|
||
|
||
**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say
|
||
what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout
|
||
decision's bill, and it is why the printer set is small rather than universal.
|
||
|
||
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
|
||
dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's
|
||
socket. The read is safe without a handshake because the counter is a **seqlock**, and it had to be made into a real
|
||
one: the first version bumped the generation last and handed back the buffer itself, which says a new value has arrived
|
||
and says nothing about whether the bytes the agent then wrote to a socket were that value — the game thread is free to
|
||
be a hundred bytes into the next one by then. A seqlock cannot validate a read that finishes after it returns, so the
|
||
bare pointer was the bug rather than the ordering. `flan_dev_result_read` copies into the caller's buffer and checks
|
||
the counter either side of the copy; the counter is odd for exactly as long as a value is being written, and a reader
|
||
that loses the race reports the last *complete* generation and no bytes, so a daemon polling for a new value keeps
|
||
polling rather than being shown half of one. The count handed out is the number of complete values, so the daemon's
|
||
"has it moved" still means what it meant. Marking the counter odd needs a release *fence* and not a release store — a
|
||
release store orders what precedes it, so the writes to the buffer would be free to become visible ahead of it, which
|
||
is the original bug with more ceremony. And the odd mark is *set* rather than incremented, because a `begin` with no
|
||
`end` is reachable: the thunk calls `begin` before it evaluates anything, so an expression that signals is stopped
|
||
inside that window, and a restart taken from the break transfers past the thunk and `end` never runs. Incrementing
|
||
would leave the counter odd for the life of the process and every later read reporting "in progress"; setting the bit
|
||
means the next evaluation repairs it. The daemon waits for the count to move rather than assuming the program has
|
||
reached a frame boundary.
|
||
|
||
**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a
|
||
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
|
||
structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth
|
||
and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three
|
||
bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a
|
||
builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time
|
||
type to walk. Not the walk itself.
|
||
|
||
The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise
|
||
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
|
||
allocator the language does not have.
|
||
|
||
```
|
||
big 18446744073709551615
|
||
col :blue
|
||
(.pos b) (V {:x 1.5 :y 0})
|
||
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
|
||
(slice (.tags b) 0 3) [ 0 42 0]
|
||
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
|
||
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
|
||
```
|
||
|
||
Details that are decisions rather than formatting:
|
||
|
||
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come
|
||
back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point
|
||
instead.
|
||
- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather
|
||
than as the value it is.
|
||
- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are
|
||
erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is
|
||
exactly what you would want to see.
|
||
- **A pointer is never followed** — `<ptr>`. It is the only thing that could make the walk cycle, and dereferencing one
|
||
a REPL was handed is not a safe thing to do on someone's behalf.
|
||
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does
|
||
not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K
|
||
and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it
|
||
means no renderer carries a budget.
|
||
- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is
|
||
not evaluated once per element.
|
||
|
||
What still refuses by name: `Map`, `Fn`, a type variable.
|
||
|
||
A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer.
|
||
`flan run calc-me.flan "1.5 * 2.0"` has always said `3`.
|
||
|
||
An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe`
|
||
does not fill up with `eval/N` for every expression ever typed.
|
||
|
||
**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by
|
||
`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
|
||
into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the
|
||
agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition*
|
||
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
|
||
and can never claim this.
|
||
|
||
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop
|
||
would exhaust it.
|
||
|
||
The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must
|
||
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
|
||
|
||
### The REPL buffer
|
||
|
||
`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No
|
||
new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as
|
||
deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess
|
||
behind it — the "process" is a stub comint needs in order to have a prompt at all.
|
||
|
||
Three things about it that are decisions:
|
||
|
||
- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in
|
||
sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it
|
||
because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
|
||
- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a
|
||
paren inside a string does not count.
|
||
- **A value and the program's output are different things and arrive by different routes.** The value is the result of
|
||
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
|
||
and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the
|
||
separation.
|
||
|
||
That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so
|
||
`(println "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A Unit
|
||
expression is almost always a call made for its effect, and is now evaluated and *then* reported.
|
||
|
||
### Conditions — step 1: `handler-bind` and `signal`
|
||
|
||
`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters
|
||
control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling
|
||
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
|
||
yet, and no signature changed.
|
||
|
||
```
|
||
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
|
||
(load-all))
|
||
```
|
||
|
||
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
|
||
function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame
|
||
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
|
||
step.
|
||
|
||
Three decisions worth keeping:
|
||
|
||
- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and
|
||
every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
|
||
- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is
|
||
nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the
|
||
name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address
|
||
instead.
|
||
- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a
|
||
branch in the function that wrote it.
|
||
- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from
|
||
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
|
||
particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the
|
||
duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still
|
||
valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the
|
||
stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When
|
||
`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
|
||
|
||
Which gives the two refusals, both by the house rule rather than by accident:
|
||
|
||
- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a
|
||
reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in
|
||
scope, which is what the accumulation case needs.
|
||
|
||
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
|
||
outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by
|
||
value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete
|
||
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
|
||
biggest usability limit in conditions is not behind the thing that was just deferred.
|
||
- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would
|
||
leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block.
|
||
|
||
### The break loop — conditions step 3
|
||
|
||
Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the
|
||
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
|
||
|
||
```
|
||
flan: unhandled Missing — stopped, not dead.
|
||
0. restart: retry
|
||
1. restart: use-placeholder
|
||
```
|
||
|
||
Four decisions, each of which is the reason something is where it is:
|
||
|
||
- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the
|
||
release runtime and must not depend on something a program may never import. A program with no agent leaves the hook
|
||
null and dies exactly as it did before.
|
||
- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes,
|
||
reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same
|
||
act, lowered once. Nothing about §6 needed changing to support it.
|
||
- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a
|
||
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
|
||
that did not drain that queue would hang `C-x C-e` exactly when it is most wanted.
|
||
- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must
|
||
not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new —
|
||
and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the
|
||
cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of
|
||
stopping.
|
||
|
||
**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but
|
||
a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also
|
||
`compute-restarts`' data, whenever that arrives.
|
||
|
||
**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused
|
||
there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that
|
||
cannot happen.
|
||
|
||
The socket verbs are `restarts`, `restart-at <n> [name]`, `restart <name>` and `abort`, and all four are refused with
|
||
the reason when the program is not stopped — there is no restart stack to walk from a running one. `restarts` answers
|
||
`<index> <+|-> <name>` per line: the index is the identity, and the flag says whether a transfer to that frame has
|
||
anywhere to land. `test/programs/break.flan` errors three times — two restarts taken by name, then a shadowed pair
|
||
where only an index can reach the outer one — so neither a loop that always resumed the same way nor one that resolved
|
||
by name could pass.
|
||
|
||
### The break loop in the editor — conditions step 3, the other half
|
||
|
||
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it
|
||
follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a
|
||
question an editor chose to ask.
|
||
|
||
So the state is learned **twice, deliberately**:
|
||
|
||
- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`,
|
||
or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now
|
||
errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding
|
||
out *after* the echo area had said the evaluation was fine.
|
||
- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is
|
||
drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies
|
||
that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left
|
||
behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the
|
||
`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs
|
||
timers, so a poll firing inside a read would eat the reply that read was waiting for.
|
||
|
||
Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is
|
||
stopped, so the poll and the prompt cannot disagree.
|
||
|
||
```
|
||
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
|
||
:stopped t :condition "Missing")
|
||
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
|
||
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
|
||
(:op "abort") → (:status "ok" :note "the program is exiting; …")
|
||
```
|
||
|
||
`:restarts` is *positional* — innermost first, duplicates kept — because the position is what `restart-at` takes.
|
||
`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is
|
||
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
|
||
prompt cannot take a different restart than the one it showed.
|
||
|
||
`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by
|
||
someone about to choose from it.
|
||
|
||
**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against
|
||
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
|
||
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
|
||
lets the next poll settle it.
|
||
|
||
The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running
|
||
one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly
|
||
right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key —
|
||
it is the thing you pick when none of the restarts is the answer.
|
||
|
||
**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A
|
||
`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
|
||
cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the
|
||
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
|
||
breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices
|
||
each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an
|
||
expression that errors and resumes it, which fails against the old shape.
|
||
|
||
The agent grew one verb, `status`, answered in **both** states — `running` or `stopped <condition>`. Everything else the
|
||
break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had
|
||
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
|
||
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
|
||
|
||
`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the
|
||
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
|
||
round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e`
|
||
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
|
||
|
||
### `layout` — a type's fields, with no program involved
|
||
|
||
```
|
||
(:op "layout" :type "sim/Cell") → (:status "ok" :type "sim/Cell"
|
||
:fields (("heat" "f32") ("next" "(Option sim/Cell)")))
|
||
→ (:status "error" :message "Missing is not a qualified name; …"
|
||
:candidates ("a/Missing" "b/Missing"))
|
||
```
|
||
|
||
The daemon can answer this with nothing running. A layout is a fact about the *build*, and the daemon owns the build —
|
||
`Tast.structs` is sitting in the session it compiled the process from. That is why the conditions buffer can name and
|
||
type a condition's fields while every one of their *values* stays refused: the shape is knowable and the contents are
|
||
not, and drawing them apart says more than drawing neither.
|
||
|
||
**The type is a name, and the name is the qualified one.** This was the open question — a class name is not an identity,
|
||
and two packages each declaring `Missing` would leave the daemon unable to pick. It turned out to need no new
|
||
machinery: `Load.qualify_decl` rewrites `Defstruct (n, …)` to `Defstruct (alias/n, …)` at import, so by the time
|
||
anything reaches `Tast.structs` the names are one flat namespace in which a collision cannot exist. The name *is* the
|
||
type id, with no table to keep in step across a reload, and the existing spelling of a type — `Types.to_string` —
|
||
already prints it.
|
||
|
||
**And the break loop was already speaking it.** `Emit.struct_name_of` takes `Types.Named n` — the qualified name — and
|
||
passes it to `flan_error`; the agent holds it in `condition_name`; `break` answers it as `:condition`. So the string
|
||
the conditions buffer already had in hand resolves as `:type` by construction, and `test_dev.ml` round-trips exactly
|
||
that: the condition a stopped program reports, handed straight back, answers with that condition's fields. One caveat
|
||
worth writing down — `condition_name` is a `char[128]`, so the round trip holds for names up to 127 bytes and a longer
|
||
one is truncated and will not resolve.
|
||
|
||
**A bare name is refused, not resolved**, even when only one struct's last segment matches it. Resolving a unique
|
||
suffix would reintroduce the ambiguity the rule exists to remove, and a rule with an exception is one a client cannot
|
||
rely on. The refusal carries `:candidates`, so a person is one copy-paste from the answer and a client has its
|
||
completion list — the same shape as `package_of` refusing a directory imported under two aliases rather than picking
|
||
one. An enum is refused by *kind* (`X is an enum, not a struct`): its members are erased to `i32` before `Tast.program`
|
||
exists, which is the same fact that makes a `defenum` unreloadable. A union is refused the same way and for its own
|
||
reason — it is declared, and union *values* are milestone 6. Both are `Types.Named` at a use site, so falling through
|
||
to "no struct is named X" would say a type does not exist about one that plainly does.
|
||
|
||
**`render.ml` is not reused, and that is not a second walk.** It walks a *value* and emits the code that prints it;
|
||
this describes a *type* and emits text. What is shared is the spelling: field types go through `Types.to_string`, which
|
||
is what `defs` spells a signature with, so `(Option T)`, `[T]`, `[n T]` and `(Ptr T)` read the same in a layout, in a
|
||
signature and in the source. A field that is itself a struct shows its qualified name — which is a `:type` this op
|
||
accepts, so nesting is another request rather than a recursion, and nothing here can be made to walk forever. Prelude
|
||
structs are answered like any other, because `Render` resolves against the same list and an editor that could see a
|
||
`Split` printed but not ask about it would be the two disagreeing.
|
||
|
||
On the Emacs side `flan-cnr-layout` makes the request and `flan-cnr-show` passes the result into
|
||
`flan-cnr-state-from-reply`, which stays a function from data to data so the fixture-driven tests keep working without
|
||
a socket. A refusal is nil, not an error: the buffer already draws a section explaining why a section is empty, and
|
||
turning `C-c C-b` into an error would take away the restarts — the decision the buffer exists for — over a missing
|
||
annotation.
|
||
|
||
### Conditions — step 2: `restart-case` and `invoke-restart`
|
||
|
||
`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
|
||
`restart-case` further out.
|
||
|
||
```
|
||
(defn fetch [n i32] i32
|
||
(restart-case (middle n) ; its value if nothing transfers
|
||
(use-placeholder [] -1)
|
||
(retry [] 7)))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
|
||
(fetch 2)) ; -1
|
||
```
|
||
|
||
**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an
|
||
`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the
|
||
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
|
||
slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers.
|
||
`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather
|
||
than a hunt.
|
||
|
||
**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare
|
||
pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and
|
||
plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight
|
||
through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform
|
||
also means redefinition acquires no new refusal class.
|
||
|
||
**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous
|
||
note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only
|
||
probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The
|
||
frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is
|
||
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
|
||
Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's
|
||
"innermost offering the name" is just the order of the walk.
|
||
|
||
**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever
|
||
frames it established and either catches the transfer or forwards it outward:
|
||
|
||
- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts
|
||
the target back and goes on out;
|
||
- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes,
|
||
and without it the handler stack would be left pointing into a frame that has gone;
|
||
- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is
|
||
still refused by name.
|
||
|
||
A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would
|
||
jump straight past the very form that was supposed to catch it.
|
||
|
||
**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is
|
||
guarded; with the channel still set the first of them would branch straight back into the landing block it came from.
|
||
Same reason the clause body starts with it null.
|
||
|
||
`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
|
||
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
|
||
cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and
|
||
there are no function values yet, so nothing can call *back* into Flan across one.
|
||
|
||
Scope, each piece refused by name with its reason and a test on the reason:
|
||
|
||
- **restarts took no parameters.** That covered §1's own `load-texture` example and skipped argument marshalling and
|
||
§3's runtime check. Both are in now; see "Conditions — step 3" below.
|
||
- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops.
|
||
- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a
|
||
choice nothing in the source shows.
|
||
- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting
|
||
there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a
|
||
defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see
|
||
it.
|
||
- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past
|
||
everything. There is nowhere to resume, so there is nothing else to do.
|
||
|
||
**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal`
|
||
returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past
|
||
it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked
|
||
`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case`
|
||
body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled
|
||
case: it cannot be an `outputs` row, because it does not exit 0.
|
||
|
||
`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart`
|
||
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
|
||
|
||
**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a
|
||
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
|
||
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
|
||
nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value.
|
||
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
|
||
|
||
`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca`
|
||
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
|
||
crosses `probe`, which the host was compiled with. Those three do not meet anywhere else.
|
||
|
||
Two things found by writing it:
|
||
|
||
- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates
|
||
the body's slots into a record the function never sees again and the indices collide. It was harmless only because no
|
||
`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now.
|
||
- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not
|
||
fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere
|
||
else entirely.
|
||
|
||
`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant
|
||
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
|
||
of those calls goes through a cell.
|
||
|
||
### Conditions — step 3: restarts take parameters
|
||
|
||
§3's other half, and the half the comparative studies all ask for: `use-value` and `store-value` are the two restarts
|
||
whose answer is not in the program.
|
||
|
||
```
|
||
(restart-case (middle n)
|
||
(use-value [v i32] (* v 2))
|
||
(retry [] 7))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-value 21))] (supplied 7)) ; 42
|
||
```
|
||
|
||
**The parameters live in a buffer the restart-case owns.** The obvious place is the invoker's frame — it is where the
|
||
values are — and it is wrong: a clause runs after every frame between the invoke and the target has returned (§5), so
|
||
the invoking side is gone by then. The invoker stores into the *target's* buffer while both are still alive, which is
|
||
the one moment they are. A clause's parameters are then ordinary slots of the establishing function, loaded out of that
|
||
buffer in the clause's landing block, and the clause body is in-frame code that sees this function's scope like any
|
||
other.
|
||
|
||
**What a restart takes is checked at run time, and it has to be.** §4 finds a restart by name on a dynamic stack: the
|
||
invoke site cannot see what it will find and the frame cannot see who will find it, so there is nothing for the checker
|
||
to compare. The frame therefore carries its parameter count and a 32-bit hash of how the types are spelled, and
|
||
`invoke-restart` compares both against its own before it stores anything. Every frame carries them, parameterless ones
|
||
included — a clause taking none has to refuse arguments as loudly as one taking two of the wrong type. The count is not
|
||
redundant with the hash: it is what keeps a hash collision between two *different* signatures harmless, since a
|
||
collision would then have to be between two lists of the same length. The spelling itself rides in the frame as well,
|
||
because the message has to say what was wanted and what was given and neither end knows both:
|
||
|
||
```
|
||
restarts.flan:79:36: restart use-value takes (i32), given ()
|
||
restarts.flan:87:36: restart retry takes (), given (i32)
|
||
```
|
||
|
||
**The arguments are evaluated into slots before the invoke node, not hung off it.** Two reasons and both are real. An
|
||
argument can transfer on its own, and that guard must fire before anything aims the channel. And a call written inside
|
||
an argument has to be on the walk `Reach` and `Load` already do — `InvokeRestart` was a leaf to both, and a leaf that
|
||
grew a subexpression would have dropped a function that is called from nowhere else and failed in the linker.
|
||
`restarts.flan` has exactly that function, `half`, to keep the claim tested.
|
||
|
||
**The break loop can take a restart it cannot fill in, so it is refused.** A transfer has two sources: an
|
||
`invoke-restart`, which writes the arguments first, and the break loop, which aims the channel at a frame by position
|
||
and has no value to supply. They reach a clause through the same channel by design, so nothing downstream can tell them
|
||
apart — which is what makes this the kind of hole that ships silently. The frame is pushed with its buffer marked
|
||
unfilled, `invoke-restart` marks it filled, and a clause with parameters checks the mark before reading. Choosing
|
||
`use-value` from a break loop today stops the program and says why. Filling it in is the editor half, and it is now the
|
||
top of `NEXT.md`: the answer is a Flan expression, and there is already something that compiles one against the live
|
||
program.
|
||
|
||
**Lookup stayed by name, and the signature is checked against what it found.** `flan_find_restart` matches the name
|
||
hash and nothing else, so an inner `(use-value [s string] ...)` shadows an outer `(use-value [v i32] ...)` and an i32
|
||
is refused there — the outer frame that would have taken it is never consulted. That is §4 read straight ("the first
|
||
frame offering the name") and it is the thing a reader will assume works the other way, so `restarts.flan` has a case
|
||
for it. Searching outward for a frame whose signature fits would make which restart runs depend on the arguments,
|
||
which is overload resolution on a dynamic stack.
|
||
|
||
Both of these guards go through `fail_block` unconditionally, unlike the bounds checks: `--no-bounds-checks` does not
|
||
remove them. A wrong index is a wrong answer, and a transfer into a clause whose parameters were never written, or
|
||
written to a different layout, is memory corruption.
|
||
|
||
`runtime/flan_rt.c` gained two message functions and nothing else. The restart frame's first four fields are the ones C
|
||
declares and their offsets do not move; everything §3 needed is appended after them, and C never allocates one.
|
||
|
||
§3's other open point, a **report string per clause**, is still open and was not settled first as §3 asks. The field
|
||
and the accessor are both cheap; the only thing that would read them is the break loop's listing, which lives in the
|
||
agent and the daemon, so it would have landed as a field nothing read. It goes with the editor half, which is changing
|
||
that listing anyway.
|
||
|
||
### What is left
|
||
- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays.
|
||
|
||
**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the
|
||
memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary.
|
||
Attaching to a process someone else built is not a thing to support by default.
|
||
|
||
## Allocators, `(Vec T)` and `StorageExhausted`
|
||
|
||
`spec-memory.md`'s Allocators section is frozen and settles what to build. This is why the built thing is the shape it
|
||
is, and — separately and loudly — the three places it **amends** that section plus the one thing it adds to it.
|
||
|
||
### `Allocator` is a builtin opaque type, so none of milestone 5 was needed
|
||
|
||
The spec defines an allocator as "a procedure plus an opaque data pointer", which reads as a function value, and
|
||
`check.ml` refuses function values four ways — a written `(Fn ...)` annotation, a written `fn` literal, a `defn`'s name
|
||
in value position, and calling anything other than a named function. Read straight off those lines, containers need
|
||
function values and the work doubles.
|
||
|
||
None of the four is anywhere near this. All four are about *surface syntax*, and a value the compiler builds that no
|
||
surface form names trips none of them:
|
||
|
||
- `Allocator` is a `Types.t` case with no user-writable constructor, the way `string` is a builtin ptr+len. No Flan
|
||
type names its procedure.
|
||
- The procedure is a C symbol the emitter names.
|
||
- `vec-new`, `push`, `free`, `free-all` and the rest are ordinary named calls, which `check_call` already routes
|
||
through `named_call`.
|
||
|
||
The precedent was already in the repo twice: a `handler-bind` clause is lowered to a function called back through
|
||
`h->fn(condition, xfer)` and built as its own `Tast.fn`, never an `Ast.Fn`; and a dev build's `call` loads a pointer
|
||
out of a cell and calls through it, which is the indirect call the surface language refuses.
|
||
|
||
What *does* need milestone 5 is a **user-written** allocator: "here is my proc, make an `Allocator` from it" wants a
|
||
`defn`'s name in value position. `make-allocator`, `allocator-from` and `allocator` are refused by name with that
|
||
reason, rather than coming back as unknown functions.
|
||
|
||
**An `Allocator` value is a pointer to the runtime's struct, never a copy of one.** That is forced, not chosen. The
|
||
capability set has to be readable from wherever a container landed, and `free-all` bumps an epoch every container made
|
||
from the allocator has to observe. A copied-by-value allocator gives each copy its own epoch and the dev trap never
|
||
fires.
|
||
|
||
### The surface
|
||
|
||
| Name | What |
|
||
|---|---|
|
||
| `(heap-allocator)` | the general-purpose tier: malloc, aligned, with free |
|
||
| `(arena-new bytes)` | a bump arena over one fixed backing buffer |
|
||
| `(arena-destroy a)` | hands the pages back — see the amendment below |
|
||
| `(free-all a)` | releases everything the allocator holds, retain-capacity |
|
||
| `(can-free? a)` / `(can-free-all? a)` | the capability set, read at run time |
|
||
| `(alloc-epoch a)` / `(alloc-id a)` | the counter `free-all` bumps; the allocator's identity |
|
||
| `(alloc-budget a)` / `(set-alloc-budget a n)` | a ceiling on live bytes — an addition, see below |
|
||
| `(alloc-live-blocks a)` | "did you forget to free", answered at the tier that can answer it |
|
||
| `context/allocator` / `context/temp` | the current implicit allocator, and the per-frame arena |
|
||
| `(with-allocator a body...)` | rebinds for a dynamic extent and releases nothing |
|
||
| `(vec-new T)` / `(vec-new T a)` / `(vec-new)` | a Vec, against the context or a named allocator |
|
||
| `(push v x)` / `(reserve v n)` | `Unit`, both |
|
||
| `(at v i)` / `(len v)` | the array names, extended — not a parallel pair |
|
||
| `(as-slice v)` / `(as-slice v lo hi)` | a non-owning `[T]` view |
|
||
| `(clone v)` / `(clone v a)` | the only copy; assignment moves |
|
||
| `(free v)` | consumes its argument |
|
||
|
||
### Three amendments to a frozen spec, and one addition
|
||
|
||
**1. `free-all` is retain-capacity, and `arena-destroy` is the operation that hands pages back.** The spec's table has
|
||
`free-all` and nothing else. Zig's `ArenaAllocator.reset` takes a `ResetMode` of `free_all` / `retain_capacity` /
|
||
`retain_with_limit`; Odin's `arena_free_all` is retain-capacity in effect, because its arena is one fixed backing
|
||
buffer and the call only sets `offset = 0`. For a frame arena reset every frame, retain-capacity is the normal case and
|
||
handing the pages back only to ask for them again is the unusual one. Taking the mode as a parameter would have grown
|
||
the table the spec froze at four names; a second operation does not. The epoch is bumped either way — the pages being
|
||
the same does not make a container built before the reset valid, which is the whole point of the trap.
|
||
|
||
**2. `context/allocator` and `context/temp` are dynamic variables with save and restore, not extra parameters.** The
|
||
spec says the allocator is "part of the calling convention". The literal reading touches every function signature, the
|
||
FFI shim, the dev trampolines and the reload ABI, for the same observable behaviour, and it collides with every other
|
||
lane working in `emit.ml`. The dynamic variable is the implementation; the literal reading is deferred and is a
|
||
performance question (a parameter avoids a global load), not a semantic one.
|
||
|
||
**3. The `Vec` header is six words in every build, not four in release.** The spec fixes the release layout at
|
||
`ptr + len + cap + allocator` with the generation word and the epoch dev-only. A layout that changes with a build flag
|
||
is a layout that can disagree *silently* across the reload boundary: a redefinition module is built by `llc` and `ld`
|
||
against a host built separately, and nothing makes the two agree on a struct size. So the words are unconditional and
|
||
so is the epoch check. The 32-byte release layout is deferred on that, and it needs the reload path to carry the flag
|
||
before it can land.
|
||
|
||
**The addition: a budget.** `flan_allocator` grew a ceiling on live bytes, 0 for none. The spec's `retry` restart is
|
||
answerable only by a handler that can make the *same* request succeed, and for a fixed backing store the handler that
|
||
works is the one that raises the ceiling — releasing the region the container lives in invalidates the container, which
|
||
is exactly what the epoch check catches. The spec names "grows the arena and then invokes `retry`" as the handler that
|
||
works; something has to be growable for that sentence to be true. It doubles as how a test exhausts an allocator on
|
||
purpose.
|
||
|
||
### `with-allocator` is its own IR node because of the transfer path
|
||
|
||
Save, run, restore — and *restore again at the pad*. That second restore is the whole reason it is a node rather than a
|
||
`let` and two calls. A body that errors, or one a handler transfers out of, leaves through `current_pad`, and a context
|
||
allocator left pointing into a region nobody outside the body has heard of would be wrong in the break loop, which is
|
||
precisely where something is about to allocate in order to render a condition. `test/programs/allocators.flan` asserts
|
||
that path by taking a restart out of a `with-allocator` body.
|
||
|
||
It releases nothing, per the spec: not at the end of a `let`, not at the end of a function, not at the end of the body.
|
||
The program proves it by reading the epoch either side.
|
||
|
||
### `(Vec T)`: one type-erased runtime, and the backend learned almost nothing
|
||
|
||
The element type appears nowhere below the call site. `size_of` and `align_of` are produced where the concrete type is
|
||
known — which without generics is simply the concrete call site — and passed in, which is Odin's arrangement
|
||
(`base/runtime/dynamic_array_internal.odin`). The backend grew four prims in total:
|
||
|
||
- `Rt of string` — a call into the runtime's C named by symbol, with argument and result LLVM types read off the
|
||
expression nodes. The container runtime is type-erased and therefore *is* a list of C entry points, so one arm covers
|
||
all of them. A `Vec` argument crosses as its address, which is also what lets an operation mutate the caller's Vec.
|
||
- `SizeOf` / `AlignOf` of a type, filled in from the same layout calculator DWARF uses — the one the acceptance test
|
||
already checks against LLVM's own `getelementptr` answers.
|
||
- `AddrOf` of any expression, place or not, because the element a `push` copies may be computed. The backend already
|
||
spilled a non-place to a temporary for exactly this.
|
||
|
||
`at` and `len` were already the names for a fixed array and a slice, so a Vec extends them rather than adding a
|
||
parallel pair — the asymmetry `nth` was removed for. The value form `(at v i)` and the place form `(set (at v i) x)` go
|
||
through one helper, so they cannot drift apart the way `nth` did.
|
||
|
||
A Vec's length and index are `i32`, like every other length here. Widening indices is one change across every
|
||
container and not a Vec question.
|
||
|
||
`let` has no type annotation — `parse.ml` settles that a triple binding is ambiguous and that types are inferred — so a
|
||
local Vec has nowhere to say what it holds and the element type is written at the call: `(vec-new i32)`. This is *not*
|
||
the explicit instantiation syntax the generics section rules out: nothing here is generic, and the name resolves as an
|
||
ordinary type rather than binding a type variable. Where the context does say — a `defvar`'s type, a return type, an
|
||
argument — it may be left out.
|
||
|
||
A zeroed Vec has a null allocator, and the first operation that needs storage adopts the context allocator, which is
|
||
Odin's behaviour. The alternative was refusing a Vec-typed struct field outright; that is refused anyway, for a
|
||
different reason (below), but the adopt rule is what makes `Zero` of a Vec a usable value rather than a null deref.
|
||
|
||
### Move-only is a dead set, and it is flow-sensitive at a join
|
||
|
||
Reading a move-only local is a move unless the site said it was a borrow. That is the conservative direction: passing
|
||
one to a function, binding it, returning it and `free`ing it are all moves and all reach one place, and the handful of
|
||
operations that only look at a container (`at`, `len`, `as-slice`, `push`, `reserve`, `clone`) say so. Only a
|
||
*syntactically simple* target counts as a borrow — in `(len (f v))` the call still moves `v`.
|
||
|
||
At an `if` and at a `match`, every arm is checked from the state before the form and the **union** of what they moved
|
||
survives the join. A flat set is wrong in both directions: it refuses `(if c (free v) (free v))`, which is legal, and it
|
||
accepts a use after a one-armed move, which is a use-after-free. The arms are alternatives, and that is what a union
|
||
says.
|
||
|
||
The one case a dead set cannot answer is a move inside a loop: merged once at the end of the body it counts one move,
|
||
not two, while the second iteration would use what the first gave away. So it is a rule rather than an inference — a
|
||
move of a binding declared outside the loop is refused, with that as the reason.
|
||
|
||
### What ownership is not transitive through yet, and why each is refused
|
||
|
||
The spec says ownership is structural — a struct containing a Vec is itself move-only, `free` recurses into owning
|
||
fields, and a field cannot be freed on its own. That machinery is the recursive teardown `drop` brings. Until it lands,
|
||
three shapes are refused where they are declared, each naming `drop`:
|
||
|
||
- **a struct field of `Vec` type**, because the struct copies its header on assignment and nothing records a move;
|
||
- **a global of `Vec` type**, because the dead set is per function — two functions each freeing it is a double free
|
||
nothing could see, and a global read does not go through the move path at all, so even the one-function case would be
|
||
accepted. Half a rule is worse than none. A global **`Allocator`** is a different thing and stays legal: an allocator
|
||
is a copyable opaque handle, and it is what makes a handler that owns the arena expressible, since a handler cannot
|
||
see the locals of the function that established it;
|
||
- **a `Vec` of a `Vec`**, because the type-erased runtime copies and releases elements bytewise: `clone` would
|
||
duplicate inner headers instead of copying what they own and `free` would drop their buffers.
|
||
|
||
And a `Vec` does not cross to C: handing a header that owns storage to C hands out an owner. `(as-slice v)` as
|
||
`(Ptr T)` plus `(len v)` is the shape that does cross, and the refusal says so.
|
||
|
||
### `StorageExhausted` went in *with* `Vec`, not after it
|
||
|
||
No allocating operation returns an error and none can fail silently. When the allocator cannot satisfy a request the
|
||
operation signals `(StorageExhausted {:bytes n :align a :allocator id})` with `error` — whose type is `Never` — inside a
|
||
`restart-case` offering `retry`. One rule over every allocating operation, which is what keeps `push` and `reserve` at
|
||
`Unit`, `clone` at the container, and no signature anywhere growing a `Result`.
|
||
|
||
It had to land with step 2 rather than after it: retrofitting adds a transfer check to every call site of every
|
||
allocating operation, which is the point of having decided it first. Odin's `append` returns an ignorable
|
||
`Allocator_Error`, and its type-erased path returns the old length on a failed reserve; an append that appends nothing
|
||
and says nothing is the outcome this rule exists to make impossible.
|
||
|
||
The lowering is built out of nodes that already existed, so the backend learned nothing about allocation:
|
||
|
||
```
|
||
(let [ok false]
|
||
(while (not ok)
|
||
(restart-case
|
||
(do (set ok ATTEMPT)
|
||
(if (not ok) (error (StorageExhausted {...}))))
|
||
(retry []))))
|
||
```
|
||
|
||
A handler that frees something, releases a scratch region or raises the ceiling and then invokes `retry` lands in the
|
||
clause, the clause falls through, and the `while` re-attempts the **same** request. Every argument to the attempt is
|
||
bound to a slot before the loop, so a retry re-attempts the allocation and not the expression that produced the value a
|
||
`push` was given. With nothing handling it, `error` stops the program on the frame that erred.
|
||
|
||
This is the named exception to plan.org's "restarts go at the resync point, once" — the restart is established at the
|
||
failing allocation, because a restart at some outer loop cannot re-attempt an allocation and only the allocation site
|
||
can. The condition is a value struct with fixed numeric fields and **no rendered message**, because formatting would
|
||
allocate and this is the one path that must not; the numbers of the failed request are read back out of the runtime.
|
||
|
||
### The epoch trap, which is the shipping answer to an open question
|
||
|
||
`spec-memory.md` leaves "catching a use-after-release statically" open on purpose: the static rule needs to know which
|
||
allocator a construction used, and `with-allocator` plus `context/allocator` are exactly the mechanisms that deny that
|
||
knowledge. The shipping answer is detection. A Vec records the epoch of the allocator it was made with, `free-all` bumps
|
||
that counter, and any operation on a container whose recorded epoch has moved traps naming the site.
|
||
`test/programs/stale-region.flan` is the case, and the point of it is that `v` is still in scope, still looks fine, and
|
||
nothing marked it — which is precisely what a static rule cannot see.
|
||
|
||
**The generation word has no reader.** It is bumped on every reallocation, as specified, and the stale-slice trap it
|
||
exists for is not implemented: a slice is ptr+len and has nowhere to carry the Vec's identity or its generation. Said
|
||
plainly here rather than implied by the word's presence in the header.
|
||
|
||
### What this leaves for steps 4 to 7
|
||
|
||
`(Map K V)`; `drop` and with it the transitive move-only rule, recursive teardown, and the refusal to construct a
|
||
drop-carrying container against an allocator without `can-free`; `(Result T E)` and `try`; generics; the macro expander.
|
||
And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy:
|
||
capture does not exist at all, and the spec's captured-`Vec`-by-pointer rule has never had to exist because every
|
||
capturable type today is a value type. It is its own item and should be planned as one.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
|
||
| link | 20ms |
|
||
|
||
Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which
|
||
compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler
|
||
(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build),
|
||
`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0`
|
||
and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d
|
||
into place, so two concurrent builds cannot see a half-written one.
|
||
|
||
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
|
||
the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever
|
||
a disk-space decision.
|
||
|
||
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It
|
||
is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted
|
||
IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse
|
||
errors.
|
||
|
||
|
||
## The order it was built in, and why that order
|
||
|
||
Every item here is done; it is kept because the *sequencing* is the part worth remembering. Decided in conversation:
|
||
wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and **the dev loop is the
|
||
thesis of the project**, so it came first. Staged so each step was runnable on its own — the failure mode being to
|
||
build a daemon and a protocol before knowing whether the reload primitive worked.
|
||
|
||
1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms.
|
||
See the section above.
|
||
2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`,
|
||
`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a
|
||
run-time-added function is itself redefined. See the section above.
|
||
3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs,
|
||
and sand.flan polling at the top of its frame. See the section above.
|
||
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than
|
||
nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if
|
||
something else ever needs to talk to it.
|
||
|
||
**One decision left to settle before step 2**, because both change codegen and are painful to retrofit:
|
||
|
||
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module
|
||
declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep
|
||
the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload,
|
||
and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
|
||
- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what
|
||
`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from.
|
||
|
||
Deferred until after the dev loop:
|
||
|
||
6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and
|
||
`test/programs/sand-headless.flan` prints `15595743031174623232` under it — the same hash as native, byte for byte, at
|
||
`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared
|
||
across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer
|
||
would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and
|
||
runs it — rather than by looking for a binary on PATH.
|
||
|
||
Three things this cost that were not in the old note:
|
||
|
||
- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main`
|
||
to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a
|
||
signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")`
|
||
label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite
|
||
self-call.
|
||
- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes `<stdio.h>`; without `--sysroot`
|
||
it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the
|
||
triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`.
|
||
- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both
|
||
shapes count.
|
||
|
||
**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to
|
||
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
|
||
`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple
|
||
(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk
|
||
is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's
|
||
beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
|
||
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
|
||
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in.
|
||
|
||
**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of
|
||
`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes
|
||
an `ExperimentalWarning` to stderr on every run and the harness compares combined output.
|
||
|
||
**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with
|
||
one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime
|
||
at it).
|
||
|
||
~~Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot.~~ **Done — and it is a third
|
||
target, not a mode of this one.** wasi-sdk is right for the headless table and was never going to be right for the game
|
||
build. See the next section.
|
||
|
||
## The browser is the third target
|
||
|
||
`flan build --target=web` produces a page, its JS and a `.wasm`, and a raylib example opens in a browser from source
|
||
that was not touched. The two wasm targets share the word and almost nothing else, so `is_wasi` and `is_web` are
|
||
separate predicates and `is_wasm` is their union — the union is exactly the set of facts about the *machine* (32-bit
|
||
pointers, no `dlopen`), which is what the refusals are about, and nothing else is shared.
|
||
|
||
**The compiler is `emcc`, not `clang`, and that is the whole of the sysroot story.** Everything the wasi target has to
|
||
find by hand — a sysroot, a builtins archive, a shadow resource directory, the `__main_argc_argv` shim — is what emcc
|
||
*is*. `target_flags` for `web` is the empty list; the only thing checked is that emcc exists, refused by name where the
|
||
reason can say so. The one fact that had to be true for any of this: **emcc takes a `.ll` on its command line**, which
|
||
it does, so `Emit`'s output needs no change and the IR stays target-independent. The object cache keys on the compiler
|
||
binary's path, size and mtime as it always did — now of *whichever* compiler the target uses, so an emcc `flan_rt.o`
|
||
and a clang one cannot collide.
|
||
|
||
**The main loop: `-sASYNCIFY`, not `emscripten_set_main_loop`.** The older note above had the browser fact right —
|
||
it cannot be blocked — and drew the wrong conclusion from it. `emscripten_set_main_loop` wants the loop body as a
|
||
callback, so every one of the eleven examples that writes
|
||
|
||
```lisp
|
||
(until (rl/window-should-close?) ...)
|
||
```
|
||
|
||
would have to be split by hand into an init and a tick, and the web program would stop being the native program.
|
||
Asyncify rewrites the module so a call can suspend across a return to the event loop, and raylib's web platform is
|
||
built for precisely that: `WindowShouldClose()` on `PLATFORM_WEB` is an `emscripten_sleep(16)` that then returns false
|
||
(raylib 5.5, `platforms/rcore_web.c`, read rather than assumed). So the loop yields once a frame at a call it already
|
||
makes, and **no example changed a character**. The price is real and is paid by every web build: asyncify instruments
|
||
the whole module, roughly doubling code size. It is not applied per-program because "does this program block" is not a
|
||
question `Build` can answer, and a flag set that varies per program is a cache key that varies per program.
|
||
|
||
**`link` lines can be addressed to a target.** `vendor/raylib/link` named `libraylib.so.550`, which exists on the host
|
||
and nowhere else. A line may now carry `@native`, `@wasi` or `@web`, an untagged line applies everywhere — which is
|
||
what every existing `link` file already is — and `${NAME}` expands from the environment. The selection happens in
|
||
`Build` and not in `Load`, which is where the file is read, because **`Load` resolves imports before a target is
|
||
chosen**: the same program is built for both, and a package's linker arguments arrive here as a flat list of strings.
|
||
`Load`'s part in this is to pass the lines through untouched, which it already did.
|
||
|
||
**raylib for the browser is built, not installed.** No emscripten port provides it (`emcc --show-ports`: there is
|
||
`contrib.glfw3` and no raylib), so `vendor/raylib/build-web.sh` clones raylib at the **5.5** tag — the one whose
|
||
`.so.550` the host links, because `raylib.flan` carries raylib's struct layouts and enum values and two targets built
|
||
from different raylibs would disagree about them in silence — and compiles the seven modules with
|
||
`-DPLATFORM_WEB -DGRAPHICS_API_OPENGL_ES2` into one archive under `vendor/raylib/web/` (gitignored). `rglfw.c` is not
|
||
among them: the web platform uses emscripten's own GLFW port, which is why `link` carries `@web -sUSE_GLFW=3`. No
|
||
headers are installed, for the reason the host build needs none — the generated shim declares its own prototypes.
|
||
|
||
**The HTML shell is a string in `Build`, not a file in the tree**, for the same reason `Runtime_src` is: it has to be
|
||
wherever the compiler is, and a build that cannot find its own shell fails for a reason nobody spelled. It is a canvas
|
||
and a `Module.print` that puts stdout on the page; `FLAN_WEB_SHELL` replaces it. `--shell-file` is passed only when the
|
||
output is a `.html`, because emcc accepts and ignores it otherwise.
|
||
|
||
**Refused by name, inherited whole:** `--dev`, `--debug`, `Build.shared` and `flan run --target=` are refused for
|
||
`web` exactly as for `wasm32`, each naming `web` rather than `wasm32` in the message. `--sanitize` is refused too, but
|
||
the web half of that refusal is weaker than the wasi half and says so: emscripten *does* ship an ASan, and nothing here
|
||
has ever run it. A sanitizer that has never been run is one whose silence means nothing.
|
||
|
||
**What the test can honestly check.** `test/test_web.ml` is headless and permanently so. It probes — emscripten may not
|
||
be installed, and the raylib archive is not in the tree — and skips with the reason rather than going red. What it
|
||
asserts: the three files exist, the module starts with `\0asm`, the page references its own JS and carries the canvas,
|
||
and node runs the emitted JS and gets `ok`. For raylib it builds `core-basic-window.flan` unchanged and then reads the
|
||
module for the two things that would be false if the mechanism were wrong: an `asyncify_start_unwind` export, and a
|
||
`glViewport` import that can only have come from raylib's web platform. Import and export names are plain strings in
|
||
the binary, so this needs no wasm reader.
|