flan/BUILT.md
Joseph Ferano 3e3d3b28f0 Merge branch 'print-sweep' into dev-loop
print and println are the whole printing surface now. About 500 call
sites across 47 files, and the site documents either of them for the
first time.

Two pinned outputs moved and both are corrections. sand-headless hashes
to 15595743031174623232 rather than -2851001042534928384 -- the same 64
bits, printed unsigned now that hash-grid's u64 no longer goes through an
(i64 ...) cast, which is the bug the family's explicit widening invited.
And a trap column shifted because the call it names got shorter.
2026-09-12 05:42:57 +07:00

1184 lines
87 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# How the parts that exist work
The reasons behind the code, kept separate from `NEXT.md` so that what to do next is not buried under what was already
done. Nothing here is a plan. Everything here is load-bearing at least once: why nothing is ever `dlclose`d, why a call
bound at link time cannot be made to notice a redefinition, why the printer is a compile-time walk over a type rather
than a function in the runtime. Deleting it would mean deriving it again.
`NEXT.md` is the live document — what is in flight, what is queued, what is blocked, and the sharp edges. Read that
first. Come here when you need to know why something is the shape it is.
## What milestone 4 added
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
assignable, which makes the generated step its only writer.
**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision.
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
Block-scoped defer is real work and is not done.
**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot,
so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed
type and logical on an unsigned one), and `rand-f32`.
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets
the state. This is what the bitwise operators were added for.
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type
in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
## Why the FFI goes through a C shim
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
```
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
```
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning
garbage rather than as a link error.
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare`
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
ABI, implemented twice", and `flan_rt.c` is the same pattern.
### The wrappers are generated now — `declare-c`, `lib/shim.ml`
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory
holds `raylib.flan` and `link` and no C at all.
One binding is now one line:
```
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
```
`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the
function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an
ordinary Flan `defn` above it when the signature has a struct in it. `flan shim <file>` prints the whole file.
**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path
string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h
i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural
rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited.
**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same
`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real
signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct
— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves.
Everything where they could diverge is refused at the field.
**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a
scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the
hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against
the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree.
**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path,
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
malloc failure, where the alternative is handing C a null pointer.
**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice
and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes
`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
the reason, and the acceptance table asserts on the reasons.
**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc +
`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
that calls an existing binding is unaffected.
`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17
34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity
would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed.
The bindings are 171 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums, raylib's own named colour palette, and the `FLAG_` window hints. Adding one is a single `declare-c` line; there is no C to write.
Two things the ported examples in `examples/` wanted and could not have, both refused for reasons that are right. `GetGamepadName` returns a `char *` into raylib's static storage: *the return type of get-gamepad-name is a string, and a string only crosses as a parameter — a C function that* returns *one returns something Flan has no owner for*. And an enum parameter cannot be indexed — `GetGamepadAxisMovement` takes a `GamepadAxis`, a loop variable is an `i32`, *expected rl/GamepadAxis, found i32*, and a second `declare-c` of the same symbol with an `i32` face is refused too: *one declare-c per C function, and another Flan name for it is a defn* — which cannot help, because a wrapper renames and does not retype. The caller spells the loop as a `cond` over the members it knows.
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins
`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all —
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`,
`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four
ways for that reason.
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora
ships the runtime library without the `.so` symlink.
### What a headless FFI test can and cannot pin
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C
writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and
getting identical output.
- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the
way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance,
passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from.
Distances are worse: the reflection does not even reach them.
- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does
not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision
cases look like they cover it.
- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different
field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.
- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available.
`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values
in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely,
including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU
reads its width, height or mipmaps at all.
- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image
`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and
answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more.
- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the
package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the
`x`/`y` permutation above.
The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail
by permuting the fields and watching it go red. A case not verified that way is decoration.
One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It
measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no
window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime`
and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are
exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them.
## Packages
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares
becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
declarations whose names contain a slash.
A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file
named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below.
**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also
a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import
all four. A file carries no `.c` and no `link` file; those belong to a directory.
**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that
is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under
one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which
is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time,
and the namespace being flat, mutually dependent packages simply work. The same directory under two *different* aliases
is refused.
**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment
anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
checker it would be "unknown name", which is true and useless.
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
## The link follows the program
`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the
importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that
link cannot succeed.
`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it,
follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs**
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
dropped one is a silently different program.
**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far.
The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and
`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker
arguments together, so a caller cannot take one without the other.
## sand.flan is one program
It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a
program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both
drivers imported it.
Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports
`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both
targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation.
What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no
input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and
`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 48, still
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
Two claims that got run together in an earlier note, for the record:
- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the
**wasi** path, which is what the headless table targets, and which has no GL and no browser.
- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with
`emscripten_set_main_loop` instead of a `while`. That is a different `main`, not a different program.
**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes:
- `(defconst gravity 0.05)``(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`,
and there is no implicit widening.
- `(defvar current-color u32)``i32`. It is an index into `colors`, and `(len colors)` is an `i32`.
- `(defn main [])` is unchanged — the short form, as plan.org says.
Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what
cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
## Bounds checks — done at milestone 3
`at` and `slice` emit `icmp``br` → cold block → `call``unreachable`; a failure names the source location. Three
check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo
<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the
same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`.
The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor
that split them would break the write check silently. The test covers both.
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.110.12s checked against 0.120.13s
unchecked. Indistinguishable.
## Why there is no interpreter
Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had
expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
criterion is dropped, and the host ABI moved onto the critical path.
## The layout, which is the whole backend design
```
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
f32 f64 float double
bool i1
an enum i32
[T] and string { ptr, i64 } ptr+len, non-owning
[n T] [n x T] inline, a value
(Ptr T) ptr opaque pointers
(Option T) { i8, T } tag 0 None, 1 Some
a struct a literal struct, declaration order
Unit and Never {}
```
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the
semantics:
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is*
the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody
addressed. `test/programs/values.flan` pins this down.
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a
`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding,
so wasm32 needs no exception proposal.
## The reload primitive — dev loop steps 1 and 2, measured
`llc``ld -shared``dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled
into its own object and called inside a process that is already running, twice, with a changed body the second time.
| Step | Cost |
|---|---|
| `Emit.redefinition` | below the timer (<0.1ms) |
| `llc -O2 -filetype=obj` | 1517ms |
| `ld -shared` | 3ms |
| `dlopen` + `dlsym` | **0.04ms** |
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
number from this codebase. For contrast, `clang -shared` on the same IR is 50ms the driver is again most of the cost,
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since
the driver absorbs IR the bare tools reject.
`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that
*is* the mechanism. What the new module does **not** define is the whole design:
- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a
redefinition can change a function's body and can never re-initialise the program's data. Define the global and the
loaded object gets a second copy sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the
thesis.
- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a
private copy of it.
- **no `main`.** This module is loaded, not started.
Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss
because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is
spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a
slice parameter.
**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it the
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells but the
agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release
builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_get`, which reads as a compiler
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and
the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason.
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a
loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property
of the dev build rather than of every build. `dlsym` on `"flan.bump"` works a dot is legal in an ELF symbol.
Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**,
since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since
`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares
`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's
worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would
fail at link time. The fixtures each print a literal so that path is actually exercised.
### Cells — how a call site follows a redefinition
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes
every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current.
```
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
%p = load ptr, ptr @"flan.cell.bump" ; every call site
%r = call i64 %p()
```
Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and
exposes `flan_reload_install()` that stores its own body into its own cell cost **below a microsecond**, which is what
makes a frame-boundary swap a non-event.
The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of
one.
Four things about this that are not free choices:
- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on
whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are
separate on purpose.
- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to
*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would
publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage,
because the failure is silent.
- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined
function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture
recurses on purpose, and would print the old body's text if it did not.
- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is
the first time `opts` means something semantic rather than an optimisation level.
LLVM cannot fold the indirection away the cell is an external mutable global and a `--dev` build of calc-me keeps 46
indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well;
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
### Names that did not exist when the process started
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF
cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else:
```
void **flan_dev_cell(const char *name); /* a new function's cell */
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
```
Both are idempotent, so the second module to mention a name gets what the first one got which is the entire point. A
new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores
on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
common case pays nothing for the general one.
**The unit is a list of top-level forms**, not one function `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c
C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a
redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet.
Four rules, each of which is a silent failure if broken:
- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are
still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`.
- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the
old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so.
This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
- **Nothing is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it leaves every call site
pointing at unmapped memory. That is a constraint on the agent too.
- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed
capacity with a loud failure rather than growable.
The test that separates this from a plausible wrong version is **v4**, which redefines `added` a name v3 introduced at
run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell*
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
pass and the transcript would read 246 instead of 432.
Sizes are spelled LLVM's way `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` rather than by a layout
calculator in OCaml that would have to agree with LLVM's on every target.
### The agent — dev loop step 3
`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link`
asks for `-lpthread`.
```
(agent/start path) listen on a unix socket; once, at startup
(agent/poll) install whatever has arrived; returns how many
(agent/wait ms) the same, but waits for something first
```
The split between them is the design. `dlopen` relocates a module and takes the loader lock milliseconds, unbounded
so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
loader.
`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits
for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
Two details found by running it:
- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a
program running with a REPL attached shows nothing until it exits and a test driving one cannot see its progress at
all, which is how this was found.
- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`.
- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows
when it is between frames.
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under
Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body
that did not exist when the process started. Building without `--dev` is fine there are no cells, so a module is
refused on the listener thread and the loop never notices.
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the
names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what
the session will track automatically.
### The session
`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change
accepted since.
**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing the accumulated list is simply not
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
Two things the session knows that no single evaluation could:
- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST,
because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `println` reads
as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded:
| Change | What it would have broken |
|---|---|
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it **and this is now a stopgap**, see below |
| a global's type | the storage exists and has a shape reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
| a struct's fields | the values the process is holding have the old layout |
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves so no store can reach it |
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked
caller site still targeting the old signature recompiling one either retargets it or gives an ordinary type error.
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
should not be read as the final answer.
A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global`
record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than
the checked program, because `Tast.program` carries no enums at all they are erased to `i32` in the checker, which is
the same fact that makes them unreloadable.
Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first,
loudly. The session's rules only get a turn on a change the checker accepts one to a name nothing else in the program
uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for
that reason.
A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A
dev build emits every `defconst` as a mutable `global` rather than a `constant` so LLVM cannot fold a read of it and a
module can store into it and a changed one is published at the frame boundary exactly as a new function body is. That
is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the
folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could.
**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in
`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` through `Load`'s
own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a
brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
did nothing.
It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
either.
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise
re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name
pass rejects it. `C-c C-k` on sand.flan's own text is the test.
`flan reload <program.flan> <forms.flan>` is that path from the command line: a session over the program the process was
built from, and a file of the forms that changed. Verified against a running sand under Xvfb a one-form `game-draw`
and 910 consecutive frames drew it.
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to
`<eval>`, so an error in forms sent without one reports positions in a file that does not exist the daemon has to pass
the real buffer path, which is the same key CIDER's `eval` carries.
### The daemon — `flan dev`
`flan dev <program.flan>` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside
the source. What it adds over `flan reload` is that the session *persists* a `defvar` added by one evaluation is part
of what the next one is checked against and that it **owns the build**, which is what makes its layout rules describe
the process that is actually running rather than a guess about it.
**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client
became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form,
from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message no parsing code on the
editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor.
```
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
(:op "close")
```
`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* it is how the
program's output is drained and signatures on that would be paid for every time anyone glanced at the output buffer.
It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing
else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none
to give, because only `Tast.fn` carries one an editor must refuse rather than go looking for the definition itself,
which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step
[i64 f32] i64`: types only.
The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan
dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by
guessing which directory it was relative to.
An evaluation that declares nothing to install a declaration the program already has, with no body and no new storage
is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so without it every error an editor shows
points into a file that does not exist.
Two things the daemon must not paper over, both of which would look like a successful evaluation:
- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path sand.flan
says `/tmp/flan-sand.sock` and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead
fails silently: the module compiles, is built, and nobody receives it.
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed
connect or a reply that is not `ok` becomes an error the editor sees.
It waits for the program to bind before accepting an evaluation one arriving first would fail for a reason that reads
like a compiler bug and it accepts with a timeout so that a program which has exited takes the daemon with it rather
than leaving an editor waiting on a socket nobody is serving.
### The Emacs client
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is
s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is
Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters every binding list and every type is
written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords.
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a
request and `read` reads a reply.
| | |
|---|---|
| `C-c C-c` | the top-level form at point, recompiled and installed |
| `C-c C-k` | the whole buffer, as **one** module |
| `C-x C-e` | the expression before point, evaluated *in the running program* |
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
| `C-c C-o` | the running program's own output, in `*flan-output*` |
| `C-c C-r` | a prompt on the running program (`*flan-repl*`) |
| `C-c C-b` | what a **stopped** program is offering, and which to take |
| `C-c C-d` | what the running program currently defines |
| `C-c C-a` | the code a name compiled to amd64, or `C-u` for the LLVM IR |
| `M-.` / `M-,` | where a name is written, through an `xref` backend |
eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
the two moments the answer can have changed on connect, and after an evaluation the daemon accepted so a `defn` just
installed completes at once.
The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away,
which is ordinary rather than an error `flan dev` ends when its program does so the next request reconnects on the
socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to
arrive in the same load, or the first refers to storage that does not exist yet.
**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary,
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
has of them a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml`
and fails here.
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte
offset**, because the reader walks the source a byte at a time the same rule as the framing, in a different place, and
`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the
daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
reader skips, so the reply's line numbers are the buffer's own.
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the
one point looked like it was in.
**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to
into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form
"asm"|"ir")` is the op.
What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d
--disassemble=flan.<name>` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared`
deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was,
so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` ten reloads in, nothing else
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
bookkeeping.
**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket
takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address
space, and `C-x C-e` renders a pointer as `<ptr>` on purpose so nothing the daemon can ask would tell it what a cell
holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first
instruction:
- nothing has been delivered for this name, so the cell still holds the host's body the one case that is *certain*;
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary unconfirmed;
- a module was delivered and the program is **stopped** which says nothing either way about whether it installed,
since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further
installs until it resumes.
From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch
targets with the file address that duplicates them dropped. Not source interleaving SBCL has the mapping and this
build emits no line tables so the reply says that in words rather than printing a listing with no source in it. When
the debug build lands, that is the line to delete.
**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an
annotation on the source, so the next command in that buffer takes it down edit, motion, evaluation, anything.
`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` the
same call the command loop makes and checks the hook is installed in that buffer and in no other; that Emacs runs it
is Emacs' contract and a test claiming to check it would be checking nothing.
**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the
next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
not a nicety a pipe nobody reads fills at 64K and the next write blocks the program forever so it is read from the
accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
grow the daemon without limit.
### `C-x C-e` — evaluating an expression
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by
exporting `flan_reload_call`, and the agent calls it after the install on the game thread, at a frame boundary, so an
expression reading the program's state sees a point the program agrees is consistent.
**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say
what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout
decision's bill, and it is why the printer set is small rather than universal.
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
dev-only feature must not put a branch in it so `flan_rt.c` is untouched and the value is read back over the agent's
socket. The read is safe without a handshake because `flan_dev_result` bumps a generation counter last; the daemon waits
for it to move rather than assuming the program has reached a frame boundary.
**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth
and length. That is a description of what `Session` already does for `C-x C-e` same walk, same refusals, same three
bounds aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a
builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time
type to walk. Not the walk itself.
The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
allocator the language does not have.
```
big 18446744073709551615
col :blue
(.pos b) (V {:x 1.5 :y 0})
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3) [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
```
Details that are decisions rather than formatting:
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come
back as `-1` but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point
instead.
- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather
than as the value it is.
- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are
erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is
exactly what you would want to see.
- **A pointer is never followed** `<ptr>`. It is the only thing that could make the walk cycle, and dereferencing one
a REPL was handed is not a safe thing to do on someone's behalf.
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does
not become ten thousand render sites in one module. The *output* is bounded once in the runtime `emit` truncates at 4K
and `end` appends `...` because a slice renders through a loop the compiler cannot bound, and one place enforcing it
means no renderer carries a budget.
- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is
not evaluated once per element.
What still refuses by name: `Map`, `Fn`, a type variable.
A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer.
`flan run calc-me.flan "1.5 * 2.0"` has always said `3`.
An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe`
does not fill up with `eval/N` for every expression ever typed.
**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by
`flan_reload_call` rather than through a cell, and it takes no registry slot so once it has returned, nothing points
into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the
agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition*
retains three, permanently and correctly a module that publishes a body exists precisely to leave a pointer behind,
and can never claim this.
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop
would exhaust it.
The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
### The REPL buffer
`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No
new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as
deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess
behind it the "process" is a stub comint needs in order to have a prompt at all.
Three things about it that are decisions:
- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in
sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it
because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a
paren inside a string does not count.
- **A value and the program's output are different things and arrive by different routes.** The value is the result of
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the
separation.
That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so
`(println "x")` the most ordinary thing anyone types at a prompt answered `()` while nothing happened. A Unit
expression is almost always a call made for its effect, and is now evaluated and *then* reported.
### Conditions — step 1: `handler-bind` and `signal`
`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters
control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
yet, and no signature changed.
```
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
(load-all))
```
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
function's own stack, and `signal` with an empty stack is a null check which is what §2 asks for. Popping is by frame
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
step.
Three decisions worth keeping:
- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and
every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is
nothing to copy. What the clause *binds* is the condition itself, though the pointer is a hidden parameter and the
name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address
instead.
- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a
branch in the function that wrote it.
- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
particular body. A handler frame is not a `Fn` value nothing in the language can name it and it is live only for the
duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still
valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the
stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When
`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
Which gives the two refusals, both by the house rule rather than by accident:
- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a
reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in
scope, which is what the accumulation case needs.
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
outlive the function that established it, so this is spec-memory.md's **case 2** a non-escaping `fn` capturing by
value into a stack environment and *not* the escaping closure that plan.org's open decision #5 defers until a concrete
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
biggest usability limit in conditions is not behind the thing that was just deferred.
- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would
leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block.
### The break loop — conditions step 3
Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
```
flan: unhandled Missing — stopped, not dead.
restart: retry
restart: use-placeholder
```
Four decisions, each of which is the reason something is where it is:
- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the
release runtime and must not depend on something a program may never import. A program with no agent leaves the hook
null and dies exactly as it did before.
- **The hook resumes by writing a restart into the transfer channel** the same channel an `invoke-restart` writes,
reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same
act, lowered once. Nothing about §6 needed changing to support it.
- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
that did not drain that queue would hang `C-x C-e` exactly when it is most wanted.
- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must
not be swapped while it is on the stack" is about *mid-frame consistency* half a frame of old code and half of new
and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the
cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of
stopping.
**A restart frame carries its name now**, beside the hash. Matching never needs it that is what the hash is for but
a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also
`compute-restarts`' data, whenever that arrives.
**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused
there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that
cannot happen.
The socket verbs are `restarts`, `restart-at <n> [name]`, `restart <name>` and `abort`, and all four are refused with
the reason when the program is not stopped there is no restart stack to walk from a running one. `restarts` answers
`<index> <+|-> <name>` per line: the index is the identity, and the flag says whether a transfer to that frame has
anywhere to land. `test/programs/break.flan` errors three times two restarts taken by name, then a shadowed pair
where only an index can reach the outer one so neither a loop that always resumed the same way nor one that resolved
by name could pass.
### The break loop in the editor — conditions step 3, the other half
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it
follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a
question an editor chose to ask.
So the state is learned **twice, deliberately**:
- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`,
or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation a body that now
errors and that is a reply the client is already reading. Finding out a second later from a poll would mean finding
out *after* the echo area had said the evaluation was fine.
- **And a timer asks anyway**, once a second, with `describe` the cheap op, which is also how the output pipe is
drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies
that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left
behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the
`lost` state that exists to be seen. It also skips while another request is in flight `accept-process-output` runs
timers, so a poll firing inside a read would eat the reply that read was waiting for.
Three ops, and the *annotation* owns `:stopped`, not the ops one place in the daemon decides whether the program is
stopped, so the poll and the prompt cannot disagree.
```
(:op "break") → (:status "ok" :restarts ("retry" …) :unreachable (2 3)
:stopped t :condition "Missing")
(:op "restart-at" :index 2 :name "retry") → (:status "ok" :index 2 :note "accepted; …")
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
(:op "abort") → (:status "ok" :note "the program is exiting; …")
```
`:restarts` is *positional* innermost first, duplicates kept because the position is what `restart-at` takes.
`:unreachable` names the positions that are on the list and cannot be chosen. `restart-at`'s `:name` is optional and is
not the lookup: the program checks it against the name it holds at that index and refuses if they have drifted, so a
prompt cannot take a different restart than the one it showed.
`break` carries only the restart list, because it costs a second round trip to the program and is wanted only by
someone about to choose from it.
**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
poll once, find it still stopped, and re-open the prompt it had just answered so the client clears its own flag and
lets the next poll settle it.
The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running
one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly
right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key
it is the thing you pick when none of the restarts is the answer.
**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A
`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the
nested one had consumed and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices
each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an
expression that errors and resumes it, which fails against the old shape.
The agent grew one verb, `status`, answered in **both** states `running` or `stopped <condition>`. Everything else the
break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped the
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e`
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
### Conditions — step 2: `restart-case` and `invoke-restart`
`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
`restart-case` further out.
```
(defn fetch [n i32] i32
(restart-case (middle n) ; its value if nothing transfers
(use-placeholder [] -1)
(retry [] 7)))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
(fetch 2)) ; -1
```
**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an
`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the
release one plus a guard, and one pointer threads down the whole chain a callee writes the target into its caller's
slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers.
`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather
than a hunt.
**Every function is transfer-transparent**, release included and that is the ABI, not a stopgap. A cell holds a bare
pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and
plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight
through, but may not drop the parameter a signature that depended on an analysis could not be reloaded into. Uniform
also means redefinition acquires no new refusal class.
**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous
note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only
probably unique two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The
frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's
"innermost offering the name" is just the order of the walk.
**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever
frames it established and either catches the transfer or forwards it outward:
- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts
the target back and goes on out;
- a `handler-bind`'s pops its handler frames and goes on out which is the path a transfer out of a handled body takes,
and without it the handler stack would be left pointing into a frame that has gone;
- the function's own runs its defers 5) and returns early. `errdefer` does not run and never could: `try`/`Result` is
still refused by name.
A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would
jump straight past the very form that was supposed to catch it.
**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is
guarded; with the channel still set the first of them would branch straight back into the landing block it came from.
Same reason the clause body starts with it null.
`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
the one C frame every handler is reached through transparent to a transfer it has to be, or §6's "a transfer cannot
cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and
there are no function values yet, so nothing can call *back* into Flan across one.
Scope, each piece refused by name with its reason and a test on the reason:
- **restarts take no parameters.** That covers §1's own `load-texture` example and skips argument marshalling and §3's
runtime arity check.
- **TODO CL-style interactive recovery.** A stopped program should be able to offer a typed restart such as
`(use-value [value T] value)` or `(use-function [replacement (Fn ...)] ...)`, and the editor should show its signature
and ask for the replacement before invoking it. That is the missing "this variable is None; what should I use instead?"
path: named, hard-coded branches are useful for `retry` and `skip`, but not a substitute for an interactive value or
alternate implementation. It needs restart argument marshalling and validation in the runtime, plus an editor protocol
for entering and checking the supplied value/expression.
- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops.
- one `restart-case` offering a name twice §4 finds the first frame offering it, and two in one frame makes that a
choice nothing in the source shows.
- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting
there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a
defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see
it.
- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past
everything. There is nowhere to resume, so there is nothing else to do.
**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal`
returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past
it, which is why `emit` puts a guard after the call and then `unreachable` and why `flan_error` cannot be marked
`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case`
body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled
case: it cannot be an `outputs` row, because it does not exit 0.
`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart`
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
**A lifted handler clause is named after the function it came out of** `handler/step/0/Missing` and is emitted by a
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value.
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca`
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
crosses `probe`, which the host was compiled with. Those three do not meet anywhere else.
Two things found by writing it:
- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates
the body's slots into a record the function never sees again and the indices collide. It was harmless only because no
`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now.
- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not
fail at link time when the prototype is a parameter short it reads a garbage pointer as the channel and dies somewhere
else entirely.
`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
of those calls goes through a cell.
### What is left
- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays.
**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the
memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary.
Attaching to a process someone else built is not a thing to support by default.
## Where build time goes
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms:
| Step | Cost |
|---|---|
| frontend: read parse load check emit | <10ms, below the timer |
| `clang` on the `.ll` | 60ms `llc` does the same codegen in **20ms** |
| `clang` on `flan_rt.c` | 40ms **now cached, paid once** |
| link | 20ms |
Every C translation unit a build needs the host shim and each package's shim goes through `Build.compile_c`, which
compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler
(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build),
`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0`
and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d
into place, so two concurrent builds cannot see a half-written one.
Measured: calc-me 160ms 110ms; sand ~720ms ~700ms, since sand's time is mostly linking libraylib and its C was never
the cost. The cache is keyed by content, so it never needs invalidating by hand `rm -rf` on the directory is only ever
a disk-space decision.
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It
is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it the emitted
IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse
errors.
## The order it was built in, and why that order
Every item here is done; it is kept because the *sequencing* is the part worth remembering. Decided in conversation:
wasm32 could wait (believed to be a solved problem once the builtins archive was in place), and **the dev loop is the
thesis of the project**, so it came first. Staged so each step was runnable on its own the failure mode being to
build a daemon and a protocol before knowing whether the reload primitive worked.
1. ~~**The reload primitive, measured.**~~ **Done** `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms.
See the section above.
2. ~~**Indirection cells.**~~ **Done** `Build.opts.dev` / `flan build --dev`, `flan_reload_install`,
`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a
run-time-added function is itself redefined. See the section above.
3. ~~**The agent, in C.**~~ **Done** `vendor/agent/`, a listener thread that loads and a game thread that installs,
and sand.flan polling at the top of its frame. See the section above.
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ **both done**, and the protocol is s-expressions rather than
nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if
something else ever needs to talk to it.
**One decision left to settle before step 2**, because both change codegen and are painful to retrofit:
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module
declares every global `external`, so globals live in the host and survive a reload which is what "edit the code, keep
the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload,
and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
- **What is a redefinition unit one function, or a file?** A file is much easier to make correct and is what
`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from.
Deferred until after the dev loop:
6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and
`test/programs/sand-headless.flan` prints `15595743031174623232` under it the same hash as native, byte for byte, at
`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared
across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer
would have shown. The acceptance table runs all four, and skips them by *probing* it builds the smallest program and
runs it rather than by looking for a binary on PATH.
Three things this cost that were not in the old note:
- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main`
to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a
signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")`
label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite
self-call.
- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes `<stdio.h>`; without `--sysroot`
it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list not just the
triple is in the object cache key, so repointing a sysroot cannot serve a stale `.o`.
- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both
shapes count.
**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple
(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk
is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's
beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
root-owned, so the archive is not dropped into it a shadow resource directory is built under the object cache, named
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in.
**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of
`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes
an `ExperimentalWarning` to stderr on every run and the harness compares combined output.
**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with
one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs build it and point a runtime
at it).
Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. wasi-sdk is right for the
headless table; it is not necessarily right for the eventual game build.