nth removed, and the REPL's structural walk lifted into a println that shares it. The example of the narrowing-index rule used nth, which no longer exists.
1595 lines
117 KiB
Markdown
1595 lines
117 KiB
Markdown
# Where this is
|
||
|
||
## Start here — next session
|
||
|
||
**Branch `dev-loop`, 154 commits, working tree clean, `dune test` green.**
|
||
|
||
The dev loop works end to end: `flan dev program.flan`, then `C-c C-c`, `C-x C-e` and `C-c C-r` in Emacs against the
|
||
running process. Conditions are two steps in of four.
|
||
|
||
`(error c)` is in — §2's diverging variant, same lookup, type `Never`. A handler that returns normally has not answered
|
||
it, so only a transfer gets past; with nothing transferring the program stops and names the condition. `flan_error` is
|
||
where the break loop goes.
|
||
|
||
**The break loop is in.** An unhandled `error` now stops the program on the frame that erred, lists the restarts on
|
||
offer, and waits; a choice sent to the agent's socket resumes it. See "The break loop" below. What is left of §2 is the
|
||
*editor* half — a `break` notification on the daemon protocol and a minibuffer prompt — which is what makes it something
|
||
you use rather than something you drive with a socket.
|
||
|
||
**The old next task, now done, was the dev-build break loop, `spec-conditions.md` §2** — where an unhandled `error`
|
||
stops and talks to the daemon instead of `rt_die()`, and where **"a crash kills the program"** finally gets fixed. The
|
||
transfer it needs exists now: §6's channel is in every signature and `restart-case` catches on it, so a break loop is a
|
||
place to *stand* while the program is stopped, not a new way to move.
|
||
|
||
Then restarts offered in the Emacs minibuffer, which wants `compute-restarts` plus two protocol ops. SBCL's restart
|
||
struct carries `report-function` and `interactive-function` for exactly that prompt and `spec-conditions.md` mentions
|
||
neither — worth adding before that step. `find-restart` and `compute-restarts` are both named in §4 and neither exists
|
||
yet; the runtime already has the stack they would walk.
|
||
|
||
Still open from §3, and each refused by name today: **restarts with parameters** (argument marshalling plus the runtime
|
||
arity check), and `handler-case`, which §"What this does not settle" leaves open as possibly a macro over `handler-bind`
|
||
plus a transfer. `error`, `find-restart` and `compute-restarts` now refuse by name too — they used to fall through to a
|
||
call and come back as *unknown name*, which is the house rule's own class of bug.
|
||
|
||
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
|
||
which §6 rules out.
|
||
|
||
### In flight — four tracks, one session, 2026-09-12
|
||
|
||
Four agents are working these in parallel worktrees. Listed so a session reading this cold does not start one of them again.
|
||
|
||
1. **`nth` removed, then a `println` builtin.** `nth` was an *asymmetric* alias of `at`: `check.ml` aliased them, but
|
||
`parse.ml` and `place_of_expr` matched only `at`, so `(set (nth a i) x)` and `(addr (nth a i))` were refused while
|
||
the `at` forms worked. Two names documented as identical that differ on write. Then `println`: `Session.render` is
|
||
already the compile-time structural walk plan.org asks for, and lifting it out of the dev path — parameterised on
|
||
its emitter — gives a real `println` over concrete types with none of milestone 5 involved. The per-type
|
||
`print-f64`/`print-str` family goes.
|
||
2. **The break loop's editor half.** `break` on the daemon protocol, `compute-restarts`, `restart-at`, and a
|
||
minibuffer prompt. `spec-conditions.md` §2's remaining piece.
|
||
3. **The first ten raylib core examples**, plus the window and input bindings they need. The deliverable is the
|
||
*language gap list* as much as the ported files — sand.flan is the only real raylib program today, so this is the
|
||
first time the language is pushed by code it was not designed around.
|
||
4. **Names in DWARF.** The two items under "One line away" below, promoted here because the inspector and the
|
||
conditions buffer are only as good as what lldb can say: let-bound locals print as `s0`, `s2` because `Tast` refers
|
||
to them by slot index and `Check` drops the names, and `Session.eval` never passes the `~debug` that
|
||
`Emit.redefinition` already takes and is already tested for. The second also unblocks source interleaving in the
|
||
disassembly buffer.
|
||
|
||
**DWARF is not missing, and `flan-cnr.el`'s stack pane reads as though it were.** `flan build --debug` emits it and
|
||
`flan-dape.el` drives lldb with it. What the break loop lacks is DWARF *in a dev build* — a `flan dev` process is not a
|
||
`--debug` one. Worth rewording that refusal so it names the real gap.
|
||
|
||
### Managed classes are planned. Do not start them.
|
||
|
||
plan.org grew a `class` facility beside `struct`: identity, runtime shape metadata, an implementation-defined
|
||
representation, generic-function dispatch, and live schema change with an explicit migration at a frame boundary. Its
|
||
own last line is the rule — nothing until ordinary `struct`, `Handle` and reload semantics are working. It is here so
|
||
that a session reading plan.org cold does not take it as the next task. Three things found while reviewing it, none of
|
||
them in plan.org yet:
|
||
|
||
- **A generic function is a cell.** "A later module can add `(defmethod draw ((e Enemy)) ...)` without editing the
|
||
original" means every compiled call site of `draw` has to find the new method — which is the problem the indirection
|
||
cells already solve. A generic function is a cell whose body is a dispatch table and a reload extends the table. The
|
||
expensive half of classes is therefore already built and tested.
|
||
- **The pool is not one storage option among three.** `migrate-instances` has to *enumerate* live instances. A pool
|
||
behind generational `(Handle T)` gives that by construction; a world arena and an owned region do not obviously.
|
||
plan.org presents the three as a free choice and they are not.
|
||
- **`Enemy@1` has to stay resolvable** for `migrate` to dispatch on it, so the session retains every layout version's
|
||
metadata for as long as any instance holds it. Same rule as "nothing is ever `dlclose`d", and worth stating as one.
|
||
|
||
### Open: can a condition be a class?
|
||
|
||
Unanswered, and it wants answering before `handler-case`, because it decides whether handler matching has one path or
|
||
two.
|
||
|
||
It would buy the thing conditions most lack: a **hierarchy**. §1 says flatly there is none, which is why nothing can say
|
||
"any condition" — no catch-all handler and nothing for a break loop to match on. Class inheritance gives it.
|
||
|
||
Three costs, one serious:
|
||
|
||
- **Signalling would allocate.** A struct condition is a stack value and `signal` takes its address; a class instance
|
||
needs a pool slot at the signal site. That is the failure path, sometimes the hot path, and sometimes the thing that
|
||
failed is allocation itself. plan.org also says no implicit allocation anywhere in the core.
|
||
- **§5's lifetime inverts.** Today the condition dies with the signalling frame and a handler that keeps it copies,
|
||
which is free for a value struct. A class instance survives the transfer — nicer, but now something owns and frees it.
|
||
- **Layout versions meet handler frames.** A struct condition cannot change layout; it is refused. A class can, and then
|
||
a frame pushed against `MyError@1` is on the stack while the signaller builds `MyError@2`.
|
||
|
||
The shape that probably wins is both: a struct condition stays exactly what it is — no allocation, matched by name hash,
|
||
dies with the frame — and a class condition is allocated, survives, and matches by walking its class chain. That is two
|
||
matching paths, which is the same bill the struct/class split already signs, so it is consistent rather than a new cost.
|
||
Either way it is an amendment to a **frozen** `spec-conditions.md`, not a gap in it.
|
||
|
||
|
||
**The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running
|
||
program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240
|
||
consecutive frames drew it.
|
||
|
||
Steps 1, 2 and 3 are done — see *The reload primitive* below. A list of top-level forms can be recompiled and installed
|
||
into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was
|
||
never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes
|
||
a redefinition over a socket and installs it between frames.
|
||
|
||
What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the
|
||
running process was built with, and speaks a protocol an editor can talk to.
|
||
|
||
Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that
|
||
runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the
|
||
interpreter was dropped — open decision #7, settled, see below).
|
||
|
||
```
|
||
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
|
||
```
|
||
|
||
| File | What it does |
|
||
|---|---|
|
||
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
|
||
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
|
||
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
|
||
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
|
||
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
|
||
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
|
||
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
|
||
| `lib/tast.ml` | the typed IR the backend consumes |
|
||
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
|
||
| `lib/session.ml` | **a live program: what the process was built from, plus every change since** |
|
||
| `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** |
|
||
| `lib/dev.ml` | **`flan dev`: a session, the program running beside it, and a socket** |
|
||
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
|
||
| `lib/emit.ml` | typed IR → LLVM IR text |
|
||
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
|
||
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
|
||
| `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** |
|
||
| `lib/shim.ml` | **`declare-c` -> the generated C that flattens a struct crossing** |
|
||
| `vendor/raylib/` | **the raylib package: `raylib.flan` and `link`, and no C at all** |
|
||
| `vendor/agent/` | **the dev agent: a socket, a loader thread, install at a frame boundary** |
|
||
| `emacs/` | **`flan-mode.el`, `flan-dev.el`, `flan-repl.el`: the editor half of the dev loop** |
|
||
| `bin/main.ml` | `flan read \| parse \| check \| emit \| shim \| build \| run \| reload \| dev` |
|
||
| `test/test_flan.ml` | reader, parser and checker |
|
||
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |
|
||
| `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** |
|
||
| `test/test_agent.ml` | **a running program taking a redefinition over a socket** |
|
||
| `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** |
|
||
| `test/test_dev.ml` | **the daemon, driven the way an editor drives it** |
|
||
| `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** |
|
||
| `test/programs/conditions.flan` | **`handler-bind` and `signal`, the accumulation case** |
|
||
| `conditions.org` | **a cheatsheet for driving conditions: what works, the exact refusals, the gotchas** |
|
||
| `conditions-play.flan` | **a program to poke at them with, built to be attached to by `flan dev`** |
|
||
| `test/programs/restarts.flan` | **`restart-case` and `invoke-restart`: the transfer, across two frames** |
|
||
| `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** |
|
||
| `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process |
|
||
| `test/wasm-run.mjs` | **a WASI host in twenty lines of `node:wasi`, so the table can run a wasm32 build** |
|
||
|
||
```
|
||
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
|
||
3.5
|
||
$ flan run test/programs/sand-headless.flan
|
||
-2851001042534928384
|
||
$ flan run sand.flan # a window, 120 fps, hold space
|
||
```
|
||
|
||
## What milestone 4 added
|
||
|
||
**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a
|
||
hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not
|
||
assignable, which makes the generated step its only writer.
|
||
|
||
**`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top
|
||
level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it
|
||
stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer
|
||
written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the
|
||
bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision.
|
||
|
||
`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at
|
||
function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about.
|
||
Block-scoped defer is real work and is not done.
|
||
|
||
**New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot,
|
||
so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed
|
||
type and logical on an unsigned one), and `rand-f32`.
|
||
|
||
**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is
|
||
only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets
|
||
the state. This is what the bitwise operators were added for.
|
||
|
||
**Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type
|
||
in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather
|
||
than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.
|
||
|
||
## Why the FFI goes through a C shim
|
||
|
||
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
|
||
|
||
```
|
||
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
|
||
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
|
||
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
|
||
```
|
||
|
||
None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a
|
||
per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that
|
||
in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning
|
||
garbage rather than as a link error.
|
||
|
||
So the boundary has one wrapper per binding, each one flattening the aggregates: a struct returns through an
|
||
out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a
|
||
copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare`
|
||
signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
|
||
ABI, implemented twice", and `flan_rt.c` is the same pattern.
|
||
|
||
### The wrappers are generated now — `declare-c`, `lib/shim.ml`
|
||
|
||
The price above was a hand-written wrapper per raylib call, and the prediction that they were mechanical enough to
|
||
generate "if that ever becomes the bottleneck" came true at 84 of them. `vendor/raylib/shim.c` is gone; the directory
|
||
holds `raylib.flan` and `link` and no C at all.
|
||
|
||
One binding is now one line:
|
||
|
||
```
|
||
(declare-c draw-texture [t Texture2D x i32 y i32 tint Color] "DrawTexture")
|
||
```
|
||
|
||
`declare-c` names raylib's own function in raylib's own signature, and the compiler emits, into a C file compiled like
|
||
any other: the typedefs for the structs involved, made from the Flan `defstruct`s; the `extern` prototype in the
|
||
function's true signature; the wrapper that flattens it; and the flattened `declare` the Flan side calls, with an
|
||
ordinary Flan `defn` above it when the signature has a struct in it. `flan shim <file>` prints the whole file.
|
||
|
||
**It is a second form and not a change to `declare`, for one reason worth remembering:** `(declare start-raw [path
|
||
string] i32 "flan_agent_start")` in `vendor/agent` means the symbol takes ptr+len, and `(declare-c init-window [w i32 h
|
||
i32 title string] "InitWindow")` means it takes a NUL-terminated `char *`. Same shape, opposite claims, so no structural
|
||
rule can separate them. `declare` is untouched, and `sqrtf` and the agent still work unedited.
|
||
|
||
**What the generator guarantees, and what it trusts.** Guaranteed: the C typedef and the Flan struct come from the same
|
||
`defstruct`, so they cannot disagree — permute the `defstruct` and the typedef permutes with it, which is exactly what
|
||
makes the permutation runs below meaningful. And clang type-checks the wrapper against the generated prototype. Trusted:
|
||
that the `defstruct` matches the library's real struct, and that the `declare-c` signature is the function's real
|
||
signature — no header is read, deliberately, so nothing can check either. A `_Static_assert` on `sizeof`/`offsetof` was
|
||
considered and rejected as circular: both sides would come from the same field list. Padding is not a separate hazard:
|
||
for every field type the generator admits — the machine integers, the two floats, `bool`, a pointer and a nested struct
|
||
— LLVM's layout is C's, and `emit.ml` writes no datalayout, so clang applies the target's rules to both halves.
|
||
Everything where they could diverge is refused at the field.
|
||
|
||
**One thing got sharper and should be said plainly:** the prototype is now generated *from the declaration*, so a
|
||
scalar's width carries ABI weight it did not before. `f64` where raylib says `float` used to be narrowed by clang at the
|
||
hand-written call site; now it emits `double` and raylib reads garbage. All 84 migrated prototypes were diffed against
|
||
the deleted `shim.c`'s — which was the ground truth for the true signatures — and agree.
|
||
|
||
**Strings.** The hand-written wrappers sized the NUL-copy per call site: 256 for a window title, `PATH_MAX` for a path,
|
||
512 for drawn text, truncating past it. A generator has no call site to look at, so it must not be the thing deciding a
|
||
string is too long: 256 bytes on the stack, the heap past that, freed after the call. The only truncation left is on
|
||
malloc failure, where the alternative is handing C a null pointer.
|
||
|
||
**Two bindings keep a hand-written wrapper, and both wrappers are Flan, not C.** `collision-point-poly?` takes a slice
|
||
and `collision-lines` answers with an `Option`; neither is raylib's own signature. A slice parameter in a `declare-c` is
|
||
refused by name, because a slice's length crosses as i64 and the type of the C count parameter beside the pointer is not
|
||
recoverable from `[T]` — so that one declares `(Ptr Vector2)` with an explicit `count i32` and the Flan wrapper passes
|
||
`(addr (at points 0))` and `(len points)`. Every other refusal — an Option, a union, a fixed array, a map, a returned
|
||
string, a callback, an unknown type, an unrepresentable struct field, two Flan names for one C symbol — is by name with
|
||
the reason, and the acceptance table asserts on the reasons.
|
||
|
||
**Known edge, not fixed:** a REPL redefinition that introduces a *new* `declare-c` cannot work. `Build.shared` is llc +
|
||
`ld -shared` and compiles no C, so the wrapper would not exist in the running process. Editing the body of a function
|
||
that calls an existing binding is unaffected.
|
||
|
||
`raylib.flan` carries the nice signature and the compiler writes the rest, so the surface sand.flan sees is
|
||
`(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17
|
||
34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity
|
||
would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed.
|
||
|
||
The bindings are 164 calls across thirteen structs: window, keyboard and mouse; drawing (rectangles, circles, lines, triangles, rings, ellipses, text); the eleven `collision-*` predicates; textures; the Image family; `Camera2D`; `RenderTexture2D`; the whole audio surface (device, `Wave`, `Sound`, `Music`); fonts and glyphs; and gamepads, touch and gestures — plus the `Key`, `MouseButton`, `TraceLogLevel`, `GamepadButton`, `GamepadAxis` and `Gesture` enums. Adding one is a single `declare-c` line; there is no C to write.
|
||
|
||
The texture calls are the first ones with no headless test, because loading one needs a GL context. What the acceptance
|
||
case does instead is pin the two new struct layouts using the only things raylib computes from those fields without a
|
||
GPU: `GetCollisionRec`, which pins `Rectangle` completely, and `SetShapesTexture`'s default substitution, which pins
|
||
`Texture2D`'s `id` and `format` and nothing else. Handing a struct over and reading it back proves nothing at all —
|
||
storing and returning is symmetric, so a permuted layout comes back permuted the same way and the case passes. `width`,
|
||
`height` and `mipmaps` are therefore checked only by looking at `sand.flan` running, which draws the brush sprite four
|
||
ways for that reason.
|
||
|
||
No raylib headers are needed: the generated C declares the prototypes it uses, so the build depends on the shared
|
||
library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora
|
||
ships the runtime library without the `.so` symlink.
|
||
|
||
### What a headless FFI test can and cannot pin
|
||
|
||
Worth knowing before writing another one, because two plausible tests in a row turned out to check nothing.
|
||
|
||
- **A struct round trip is worthless.** Hand raylib a struct, read it back, compare: store-and-return is symmetric, so C
|
||
writes and reads the same wrong slots and the test passes for *any* field order. Found by permuting two fields and
|
||
getting identical output.
|
||
- **Axis-aligned geometry cannot pin `Vector2`.** Exchanging `x` and `y` is a reflection, applied to the inputs on the
|
||
way in and undone on the way out, so the printed answer is unchanged. Every collision predicate, and every distance,
|
||
passes with the fields swapped — verified by swapping the `defstruct`, which is what the typedef is now made from.
|
||
Distances are worse: the reflection does not even reach them.
|
||
- **What does pin `Vector2` is the rotated camera**, because a 90-degree rotation is not axis-aligned and therefore does
|
||
not commute with the reflection. That case is load-bearing and must not be deleted on the grounds that the collision
|
||
cases look like they cover it.
|
||
- **What pins `Rectangle` is arithmetic on its fields** — `GetCollisionRec` computes four numbers from four different
|
||
field pairs, and the point/rect predicates turn the wrong way when width and height are exchanged.
|
||
|
||
- **Scalars in, fields out is the strongest shape there is**, and the Image family is where it was finally available.
|
||
`GenImageColor(4, 2, colour)` is handed two integers and answers with a struct reading 4, 2, 1, 7 — four distinct values
|
||
in four adjacent `i32` slots, with no input struct for a permutation to cancel against. That pins `Image` completely,
|
||
including that `data` is present and first; `Texture2D` could never be pinned that way because nothing without a GPU
|
||
reads its width, height or mipmaps at all.
|
||
- **A non-square image is an axis discriminator.** `GetImageColor` indexes `y*width + x`, so on a 4-wide, 2-tall image
|
||
`(3,0)` exists and its transpose does not: exchange the wrapper's `x` and `y` and the read goes out of bounds and
|
||
answers transparent black. `ImageFlipHorizontal` against `ImageFlipVertical` says the same thing twice more.
|
||
- **A file is external ground truth**, so `ExportImage` then `LoadImage` is not the symmetric round trip the rest of the
|
||
package has to avoid — stb's encoder and decoder agree with each other, not with Flan's field order. Verified red by the
|
||
`x`/`y` permutation above.
|
||
|
||
The rule that falls out: make raylib **compute** something whose answer differs per axis, then verify the test can fail
|
||
by permuting the fields and watching it go red. A case not verified that way is decoration.
|
||
|
||
One correction to an assumption that has now cost two lanes a guess: **`MeasureText` is not headless material.** It
|
||
measures with the default font, `LoadFontDefault` is not exported, and nothing but `InitWindow` loads it — so with no
|
||
window it answers 0 for every string. Measured against `libraylib.so.550`, not reasoned about. `GetFrameTime`, `GetTime`
|
||
and `GetScreenWidth`/`Height` are all 0 headless for the same kind of reason. All five are bound, and all five are
|
||
exercised by running `sand.flan` and looking, which is the whole of what can be claimed for them.
|
||
|
||
## Packages
|
||
|
||
`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is
|
||
a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no
|
||
collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares
|
||
becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is
|
||
rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of
|
||
declarations whose names contain a slash.
|
||
|
||
A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file
|
||
named `link` lists extra linker arguments. Whether those reach the build at all is decided *after* checking — see below.
|
||
|
||
**A package may be a single `.flan` file** named outright, rather than a directory. That is for the program that is also
|
||
a library: `sand.flan` shares the repository root with three other loose programs, so naming its directory would import
|
||
all four. A file carries no `.c` and no `link` file; those belong to a directory.
|
||
|
||
**A package may import a package.** The qualification flattens to the *inner* alias — raylib imported by a package that
|
||
is itself imported is still `rl/…`, never `sand/rl/…` — because a directory reached along two routes has to arrive under
|
||
one set of names or the checker sees every declaration twice. A directory is keyed by its real path and read once, which
|
||
is also what ends a cycle: a package that imports itself meets its own entry and contributes nothing the second time,
|
||
and the namespace being flat, mutually dependent packages simply work. The same directory under two *different* aliases
|
||
is refused.
|
||
|
||
**Visibility is one rule: `main` is not exported.** A package carrying one would collide with the importer's the moment
|
||
anything imported it, so a program could never be a package; and `main` is a reachability root, so an imported one would
|
||
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
|
||
checker it would be "unknown name", which is true and useless.
|
||
|
||
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
|
||
|
||
## The link follows the program
|
||
|
||
`Load` used to hand a package's `.c` files and `link` arguments to the build the moment it was imported, whatever the
|
||
importing program did with them. So anything naming `vendor:raylib` linked libraylib on every target, and on wasm32 that
|
||
link cannot succeed.
|
||
|
||
`lib/reach.ml` answers it from the checked program instead. Start at `main` and at the globals that run before it,
|
||
follow every call — including the `Handled` frames, where a lifted handler clause is reached by address and by nothing
|
||
else — and keep what is reached. A package none of whose externs survive contributes no C and no linker argument.
|
||
|
||
Dropping the flags alone would only move the failure: the bodies that called into raylib would still be emitted and
|
||
`wasm-ld` would fail on the symbols rather than on the argument list. So the same walk prunes **functions and externs**
|
||
from the program. Only those. Globals, structs and unions stay, because an unreferenced global is bytes in BSS and a
|
||
dropped one is a silently different program.
|
||
|
||
**Dev builds are not pruned.** What a REPL may redefine next is not a function of what has been called so far.
|
||
|
||
The filtering happens at the call sites — `bin/main.ml`, the tests — because `Build.executable` receives `csrcs` and
|
||
`lflags` from its caller and never sees the import list. `Reach.link` returns the pruned program and its C and linker
|
||
arguments together, so a caller cannot take one without the other.
|
||
|
||
## sand.flan is one program
|
||
|
||
It was two files, and only ever for the reason above: the headless run is the one CI does on native *and* wasm32, and a
|
||
program that imported raylib linked libraylib whatever its `main` did. So the simulation lived in `sand-sim/` and both
|
||
drivers imported it.
|
||
|
||
Now `sand.flan` holds the simulation *and* the raylib front-end, and `test/programs/sand-headless.flan` imports
|
||
`sand.flan` itself — window, raylib bindings, dev agent and all — and still builds for wasm32. Nothing it calls reaches
|
||
raylib; `sand.flan`'s `main` is not exported, so the only `main` is the headless one; and the hash is unchanged on both
|
||
targets at `-O2` and `-O0`, which is the point. A refactor that moved that number would have moved the simulation.
|
||
|
||
What still justifies *two entry points* is smaller and stands on its own: **the headless test needs no window and no
|
||
input on any target.** `sand.flan` could not be that test even natively — with no mouse the grid stays empty and
|
||
`settle` and `move-grain` never run on real data. Measured through the probe: 168 grains painted around row 4–8, still
|
||
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
|
||
|
||
Two claims that got run together in an earlier note, for the record:
|
||
|
||
- *raylib does not work on wasm* — false. It works through emscripten. What is true is that it does not work on the
|
||
**wasi** path, which is what the headless table targets, and which has no GL and no browser.
|
||
- *a game loop cannot be expressed on wasm* — false. The browser cannot be blocked, so a web build drives the loop with
|
||
`emscripten_set_main_loop` instead of a `while`. That is a different `main`, not a different program.
|
||
|
||
**Three edits were made to sand.flan's own text** when it was ported, and they are language decisions rather than fixes:
|
||
|
||
- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`,
|
||
and there is no implicit widening.
|
||
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`.
|
||
- `(defn main [])` is unchanged — the short form, as plan.org says.
|
||
|
||
Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what
|
||
cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
|
||
|
||
## Bounds checks — done at milestone 3
|
||
|
||
`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three
|
||
check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds
|
||
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo
|
||
<= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned.
|
||
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the
|
||
same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`.
|
||
|
||
The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor
|
||
that split them would break the write check silently. The test covers both.
|
||
|
||
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s
|
||
unchecked. Indistinguishable.
|
||
|
||
## Why there is no interpreter
|
||
|
||
Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had
|
||
expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
|
||
perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is
|
||
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
|
||
criterion is dropped, and the host ABI moved onto the critical path.
|
||
|
||
## The layout, which is the whole backend design
|
||
|
||
```
|
||
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
|
||
f32 f64 float double
|
||
bool i1
|
||
an enum i32
|
||
[T] and string { ptr, i64 } ptr+len, non-owning
|
||
[n T] [n x T] inline, a value
|
||
(Ptr T) ptr opaque pointers
|
||
(Option T) { i8, T } tag 0 None, 1 Some
|
||
a struct a literal struct, declaration order
|
||
Unit and Never {}
|
||
```
|
||
|
||
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the
|
||
semantics:
|
||
|
||
- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is*
|
||
the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody
|
||
addressed. `test/programs/values.flan` pins this down.
|
||
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a
|
||
`getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
|
||
|
||
Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding,
|
||
so wasm32 needs no exception proposal.
|
||
|
||
## Blocked and unfinished
|
||
|
||
Everything below was found, decided or half-built and then stopped. Each says what blocks it. Nothing here is a
|
||
vague intention — if it is listed, someone has already established it is real.
|
||
|
||
### Unblocked now, and ranked
|
||
|
||
1. **Allocators, then `Vec` and `Map`.** The critical path, and the only thing standing between this and writing a
|
||
game. **`Vec` does not need generics** — that was wrong and is worth un-learning: Odin's containers are compiler
|
||
builtins over a *type-erased* runtime (`base/runtime/dynamic_array_internal.odin`), where `$T` appears only in thin
|
||
wrappers producing `size_of`/`align_of` at the call site, and per-key hash and equality are compiler-emitted
|
||
procedures passed as a runtime argument (`src/llvm_backend.cpp`, `Map_Info`). That is exactly what
|
||
`spec-memory.md` already specifies. Before writing any of it, settle the four things the Odin and Carp studies
|
||
converged on, because all four are cheap now and expensive after:
|
||
- **When is storage released?** `spec-memory.md` never says. Odin's answer is `defer delete`, which Flan cannot
|
||
express — `defer` is function-scoped and refused in a `let`, a loop or a branch. Carp's answer is scope-end
|
||
frees, which it then could not reconcile with arenas and so has no allocator at all.
|
||
- **A `drop` hook** for a struct owning something that is not memory — a `Texture2D`, a socket, a file handle. Carp
|
||
shipped `delete` and then had to add a separate `drop` interface (`docs/Drop.md`). The hard part is ours alone:
|
||
what runs `drop` when the arena resets underneath the value.
|
||
- **`alignment`** appears nowhere in the design. Every Odin allocation carries it, and `#soa` and component-wise
|
||
fixed arrays want 16-byte alignment.
|
||
- **Allocation failure.** Unspecified. Odin returns an ignorable error, so a failed `append` silently appends
|
||
nothing. Flan has a better answer available for free: a `StorageExhausted` condition with a `retry` restart.
|
||
Decide which, because `push`/`put`/`clone`'s signatures depend on it.
|
||
|
||
2. **Typed restarts — `(use-value [v T] v)`.** The author's third TODO, and the most-wanted thing across every
|
||
comparative study. SBCL's report: restarts without parameters lose "the entire supply-a-value half of the standard
|
||
vocabulary", because `use-value` and `store-value` are the only two whose answer comes from outside the program.
|
||
Needs argument marshalling in `emit.ml` and §3's arity check in `check.ml`; both files are free now. The leverage
|
||
SBCL lacks: `eval` already compiles and runs an expression inside the live program, and the daemon already holds
|
||
the struct layouts, so "ask the human, type-check the answer, hand it over" is a short hop.
|
||
|
||
3. **`handler-case`.** Not a convenience — it is the fix for the loudest gotcha in `conditions.org`. A handler closes
|
||
over nothing *only because* a `handler-bind` clause runs at the signal point; a `handler-case` clause runs in the
|
||
establishing frame, which is ordinary in-frame code exactly like a `restart-case` clause. SBCL's is `handler-bind`
|
||
plus a transfer and nothing more (`src/code/error.lisp:196-268`). Every piece exists.
|
||
|
||
### Bugs found and not yet fixed
|
||
|
||
- **A shadowed restart is offered and cannot be taken.** §4 says lookup takes the *first* frame offering a name, and
|
||
`flan_find_restart` does exactly that — so when two frames offer `retry`, the second one is real, is on the list the
|
||
break loop prints, and is unreachable. The old `C-c C-b` prompt showed `retry` twice and sent the string either way;
|
||
the inner frame took it, silently. `test/programs/restarts.flan`'s own `nested` function is the counterexample, and
|
||
it has been there since the transfer landed. The C&R buffer now draws the shadowed row unbracketed and refuses
|
||
`RET` on it by name, which stops the lie but does not restore the choice. The fix is to take a restart **by index**
|
||
rather than by name — SBCL identifies them positionally for precisely this reason:
|
||
- `(:op "restart-at" :index N)` on the daemon, and a matching agent verb. `flan_restart_name(i, &len)` already
|
||
walks the stack by index, so resuming by index is that loop plus `flan_break_resume`'s tail. C only, no compiler
|
||
work.
|
||
|
||
- **A restart chosen at a break inside a thunk is accepted, announced, and silently not taken.** Demonstrated.
|
||
`flan_reload_call` allocates its own `xfer` and discards it on return, so a transfer aimed at a frame below the
|
||
`flan_agent_poll` C frame unwinds only as far as the thunk. Three statements that the program will resume, none
|
||
true. Fix: record the restart-stack depth on entering `break_loop` and refuse any frame below it.
|
||
- **Restart names are served from a stack that is being mutated.** The stopped thread is not holding still — the break
|
||
loop runs `flan_agent_poll`, which runs arbitrary Flan, and every `restart-case` it enters pushes and pops the same
|
||
global list. `flan_restart_name` can return a pointer into a popped frame, which `send` then reads out of bounds.
|
||
Fix: publish an immutable snapshot when the break loop is entered.
|
||
- **The job ring has no fullness check**, and the comment describing its overflow is wrong. `publish` never consults
|
||
`tail`; past `QUEUE` entries it overwrites the slot the consumer is reading, and `job` is 24 non-atomic bytes.
|
||
Reachable from a program that goes a long time between `agent/poll` calls.
|
||
- **`flan_dev_result_get` is not the seqlock its comment claims** — it reads the generation first, then a non-atomic
|
||
length, then returns a bare pointer the caller sends later.
|
||
- Smaller: `exit(134)` from the break loop with the listener inside `dlopen`; a `dlopen` handle leaked when a module
|
||
has no installer.
|
||
- **`(A {:x 1})` on a union variant says "unknown struct A"** rather than the union refusal `check_struct` plainly
|
||
intends — `env` has no table of variant names. A diagnostics bug, not a backend death.
|
||
|
||
### Test blind spots, from a mutation pass
|
||
|
||
Sixty mutations, nineteen left the whole suite green. The severe cluster is closed (`cleanup.flan`,
|
||
`signedness.flan`); these are not:
|
||
|
||
- `Reach`'s walk of index expressions, `addr` places and `restart-case` clause bodies — each confirmed to prune a
|
||
function a valid program calls, so the build fails to link.
|
||
- `flan_dev_global`'s size-change guard — the layout-drift check, with no test that retypes a global across a reload.
|
||
- A local shadowing an imported name is qualified anyway.
|
||
- The 4K result cap and the registry overflow guard have **no coverage at all**, rather than a missing assertion.
|
||
- The reader accepts an unknown string escape; `+5` stops being a number.
|
||
- And a warning: a reader mutation makes the suite **hang** rather than fail. A green run is not the only outcome to
|
||
plan for in CI.
|
||
|
||
### Asked for by the editor lanes
|
||
|
||
- **`(:op "layout" :type T)` → the struct's fields and their types.** `Tast.structs` is held by the daemon at all
|
||
times because it owns the build, and **no running program is involved** — this is the cheapest real win on the
|
||
list, and the C&R buffer already draws its result.
|
||
- **`(:op "condition")` → the stopped program's condition, rendered.** Two steps: `break_loop` currently does
|
||
`(void)condition;` and *discards the pointer*, so stash it beside `condition_name`; then the daemon builds a render
|
||
thunk aimed at that address, which is `Session.render` rooted at a `Ptr` instead of an expression.
|
||
- **One thing to get right for both:** the type must be an identity the daemon can resolve to a `Tast` type, not a
|
||
bare class name. The hook is handed a string, and two packages both declaring `Missing` leave the daemon unable to
|
||
pick a layout. A qualified name or a type id. The same wrinkle bites locals later, because DWARF also gives a name.
|
||
- **`(:op "backtrace")` is blocked** on frame metadata — unlocked by the DWARF work, then a new agent verb. Locals are
|
||
blocked twice: DWARF for the frame layout, *and* the pointer-rooted render thunk. Restart source locations and
|
||
arity are blocked too — `flan_restart` carries `prev`, `name_id`, `name` and `namelen`, so both need a new field in
|
||
the frame, which means the compiler emitting it.
|
||
|
||
### One line away
|
||
|
||
- **`match` over enums.** Fully desugarable, wanted, and blocked only by `Ast.pattern` needing a keyword case, which
|
||
`load.ml` matches exhaustively.
|
||
- **[in flight]** **DWARF for a redefinition module.** `Emit.redefinition` takes `~debug` and is tested; `Session.eval` does not pass
|
||
it. That also unblocks source interleaving in the disassembly buffer.
|
||
- **[in flight]** **Let-bound locals print as `s0`, `s2`** under lldb. Parameters get their real names; `Tast` refers to the rest by
|
||
slot index and `Check` drops the names.
|
||
- **`Build.executable` returns only `out`**, so the daemon recovers the host `.ll` by recomputing `Build.workdir ()`.
|
||
|
||
### Deferred with a reason
|
||
|
||
- **Writing through a string literal** — see Sharp edges. Needs provenance, which is open decision #3.
|
||
- **`cstring` as a type.** Odin has no `string → cstring` conversion at all; it pays the same copy our shim already
|
||
makes. The one thing it buys is the *return* direction, and nothing in `vendor/raylib` returns a string.
|
||
- **`rune`.** Odin's is a 4-byte integer distinguished by a flag, so `i32` is the same thing. Non-ASCII text is
|
||
blocked on font loading, not on the string layer — and fonts are now bound.
|
||
- **Macro expansion.** The reader and the declaration are in. Running a macro means compiling it and `dlopen`ing it
|
||
into the compiler, which is the reload primitive pointed at ourselves — but a macro is `[Form] -> Form`, so `Form`
|
||
has to be a Flan union whose layout the compiler and the loaded macro agree on, and union *values* are milestone 6.
|
||
|
||
### Documents that contradict the code
|
||
|
||
- **`plan.org`'s jank #947 citation is wrong in its mechanism.** jank does not relink (it calls through vars, which
|
||
are already indirection cells) and never unloads (`remove_symbol` has no callers). The real cause was a
|
||
process-teardown race. We are safe from the repro — because we compile out of process, not because of cells. A
|
||
normative document citing the wrong mechanism protects the wrong invariant.
|
||
- **`plan.org` still lists open decision #7 as open** and the interpreter as a backend. It was settled the other way;
|
||
`NEXT.md` records the consequences as "already applied" to `plan.org`, and they never were.
|
||
- **nREPL's `eval` does carry `file`, `line` and `column`** — jank reads all three. The choice of s-expressions still
|
||
stands on its other grounds; the stated reason does not.
|
||
|
||
## Sharp edges
|
||
|
||
- **Writing through a string literal is undefined, and the two build modes
|
||
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
|
||
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
|
||
memory and the program takes SIGSEGV; at `-O2` LLVM deletes it as undefined
|
||
and the program prints `Hi` and exits 0. Same source, and which way it fails
|
||
depends on a flag — the worst shape available, and worse than either outcome
|
||
alone.
|
||
|
||
Nothing refuses it. `bytes` turns a `string` into a `[u8]`, the language lets
|
||
you write through a slice, and by then nothing records that the bytes came
|
||
from a constant. The honest fix is provenance — knowing a slice's origin —
|
||
which is plan.org open decision #3 and deliberately deferred. A cheaper one
|
||
that is *not* a fix: emitting literals as mutable globals only moves which
|
||
flag misbehaves, and costs their read-only placement.
|
||
|
||
Found by the string lane while deciding whether `lower-ascii` should mutate
|
||
in place. It ships the copying version for exactly this reason, and that is
|
||
the rule to follow until provenance exists: **a function over a `string` must
|
||
not write through it.**
|
||
|
||
|
||
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count,
|
||
both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They
|
||
stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
|
||
|
||
- **An index converts from a narrower integer and never from a wider one.** `(at colors current-color)` with a `u32`
|
||
index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index
|
||
is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
|
||
- **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs,
|
||
unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a
|
||
duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function
|
||
'@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations
|
||
were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the
|
||
second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly
|
||
the same set — the names an import renames — and two copies of that list would drift.
|
||
- **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not
|
||
a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal
|
||
count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to
|
||
`width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The
|
||
prelude's rotate masks its own count; that is now redundant but harmless.
|
||
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that
|
||
a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was
|
||
written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up.
|
||
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a
|
||
global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is
|
||
therefore its own call to `in_range`; there is a regression test.
|
||
- A `let` binding takes no type annotation, which is why `sand.flan` names its FNV constants instead of writing them
|
||
inline.
|
||
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser
|
||
cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous.
|
||
|
||
## The reload primitive — dev loop steps 1 and 2, measured
|
||
|
||
`llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled
|
||
into its own object and called inside a process that is already running, twice, with a changed body the second time.
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| `Emit.redefinition` | below the timer (<0.1ms) |
|
||
| `llc -O2 -filetype=obj` | 15–17ms |
|
||
| `ld -shared` | 3ms |
|
||
| `dlopen` + `dlsym` | **0.04ms** |
|
||
|
||
**~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the
|
||
number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost,
|
||
which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since
|
||
the driver absorbs IR the bare tools reject.
|
||
|
||
`ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that
|
||
*is* the mechanism. What the new module does **not** define is the whole design:
|
||
|
||
- **a global is `external`.** This settles the open question below in the only direction that supports the demo: a
|
||
redefinition can change a function's body and can never re-initialise the program's data. Define the global and the
|
||
loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the
|
||
thesis.
|
||
- **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a
|
||
private copy of it.
|
||
- **no `main`.** This module is loaded, not started.
|
||
|
||
Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss
|
||
because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is
|
||
spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a
|
||
slice parameter.
|
||
|
||
**`flan_dev.c` is compiled into every build, not only a dev one.** Nothing in a release build calls into it — the
|
||
compiler only emits a registry lookup for a name the host was not built with, which cannot arise without cells — but the
|
||
agent package's C refers to it, and a package's C sources are collected whatever `main` does. Leaving it out of release
|
||
builds made `flan build sand.flan` fail at the link with an undefined `flan_dev_result_get`, which reads as a compiler
|
||
bug rather than as a missing flag. The table is BSS, so the cost is address space and not binary size; `-rdynamic` and
|
||
the cells are still what `--dev` means. `test_agent.ml` links the agent program both ways for this reason.
|
||
|
||
**`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a
|
||
loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property
|
||
of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol.
|
||
|
||
Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**,
|
||
since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since
|
||
`dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And
|
||
`helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares
|
||
`helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the
|
||
host's copy. With the two bodies identical nothing at run time would notice a module that grew its own.
|
||
|
||
String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's —
|
||
worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would
|
||
fail at link time. The fixtures each print a literal so that path is actually exercised.
|
||
|
||
### Cells — how a call site follows a redefinition
|
||
|
||
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes
|
||
every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current.
|
||
|
||
```
|
||
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
|
||
%p = load ptr, ptr @"flan.cell.bump" ; every call site
|
||
%r = call i64 %p()
|
||
```
|
||
|
||
Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and
|
||
exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what
|
||
makes a frame-boundary swap a non-event.
|
||
|
||
The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of
|
||
one.
|
||
|
||
Four things about this that are not free choices:
|
||
|
||
- **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on
|
||
whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are
|
||
separate on purpose.
|
||
- **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to
|
||
*taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would
|
||
publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage,
|
||
because the failure is silent.
|
||
- **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined
|
||
function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture
|
||
recurses on purpose, and would print the old body's text if it did not.
|
||
- **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is
|
||
the first time `opts` means something semantic rather than an optimisation level.
|
||
|
||
LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46
|
||
indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well;
|
||
the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function.
|
||
|
||
### Names that did not exist when the process started
|
||
|
||
Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF
|
||
cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else:
|
||
|
||
```
|
||
void **flan_dev_cell(const char *name); /* a new function's cell */
|
||
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
|
||
```
|
||
|
||
Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A
|
||
new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores
|
||
on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not
|
||
reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the
|
||
rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a
|
||
call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the
|
||
common case pays nothing for the general one.
|
||
|
||
**The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c
|
||
C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a
|
||
redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet.
|
||
|
||
Four rules, each of which is a silent failure if broken:
|
||
|
||
- **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are
|
||
still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`.
|
||
- **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the
|
||
old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so.
|
||
This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.
|
||
- **Nothing is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it leaves every call site
|
||
pointing at unmapped memory. That is a constraint on the agent too.
|
||
- **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed
|
||
capacity with a loud failure rather than growable.
|
||
|
||
The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at
|
||
run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell*
|
||
both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still
|
||
pass and the transcript would read 246 instead of 432.
|
||
|
||
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout
|
||
calculator in OCaml that would have to agree with LLVM's on every target.
|
||
|
||
### The agent — dev loop step 3
|
||
|
||
`vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link`
|
||
asks for `-lpthread`.
|
||
|
||
```
|
||
(agent/start path) listen on a unix socket; once, at startup
|
||
(agent/poll) install whatever has arrived; returns how many
|
||
(agent/wait ms) the same, but waits for something first
|
||
```
|
||
|
||
The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded —
|
||
so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a
|
||
redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks.
|
||
The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the
|
||
loader.
|
||
|
||
`wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits
|
||
for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first
|
||
introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if
|
||
it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that.
|
||
|
||
Two details found by running it:
|
||
|
||
- **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a
|
||
program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at
|
||
all, which is how this was found.
|
||
- **The reply goes out before the module is queued.** The other way round, the game thread can install and the program
|
||
can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`.
|
||
- **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows
|
||
when it is between frames.
|
||
|
||
**sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under
|
||
Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body
|
||
that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is
|
||
refused on the listener thread and the loop never notices.
|
||
|
||
`flan reload <file.flan> <fn>... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the
|
||
names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what
|
||
the session will track automatically.
|
||
|
||
### The session
|
||
|
||
`lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change
|
||
accepted since.
|
||
|
||
**Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a
|
||
declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not
|
||
replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that
|
||
follows. There is a test for the case that actually matters: a typo, then a good form, in the same session.
|
||
|
||
Two things the session knows that no single evaluation could:
|
||
|
||
- **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST,
|
||
because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `print-line` reads
|
||
as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic.
|
||
- **What that process's memory looks like.** Three changes are refused with a reason rather than loaded:
|
||
|
||
| Change | What it would have broken |
|
||
|---|---|
|
||
| a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it — **and this is now a stopgap**, see below |
|
||
| a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve |
|
||
| a struct's fields | the values the process is holding have the old layout |
|
||
| a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it |
|
||
| a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too |
|
||
|
||
**The signature row is the one the plan has moved past.** plan.org now says a signature-changing redefinition should
|
||
make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while
|
||
existing callers and stored `Fn` values keep the old version and stay safe, and the session warns at every tracked
|
||
caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error.
|
||
Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines
|
||
(a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because
|
||
the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message
|
||
should not be read as the final answer.
|
||
|
||
A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past
|
||
long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global`
|
||
record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than
|
||
the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is
|
||
the same fact that makes them unreloadable.
|
||
|
||
Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first,
|
||
loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program
|
||
uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for
|
||
that reason.
|
||
|
||
A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A
|
||
dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a
|
||
module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That
|
||
is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the
|
||
folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could.
|
||
|
||
**A form typed into a file that is imported as a package is qualified the way the import qualified it.** `poll` in
|
||
`vendor/agent/agent.flan` becomes `agent/poll`, and its call to `poll-raw` becomes `agent/poll-raw` — through `Load`'s
|
||
own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a
|
||
brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it
|
||
already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently
|
||
did nothing.
|
||
|
||
It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a
|
||
Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written
|
||
nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to
|
||
either.
|
||
|
||
The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise
|
||
re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name
|
||
pass rejects it. `C-c C-k` on sand.flan's own text is the test.
|
||
|
||
`flan reload <program.flan> <forms.flan>` is that path from the command line: a session over the program the process was
|
||
built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw`
|
||
and 910 consecutive frames drew it.
|
||
|
||
Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on
|
||
every invocation, so if the program file has been edited since the process launched, its idea of which names the host
|
||
has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to
|
||
`<eval>`, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass
|
||
the real buffer path, which is the same key CIDER's `eval` carries.
|
||
|
||
### The daemon — `flan dev`
|
||
|
||
`flan dev <program.flan>` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside
|
||
the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part
|
||
of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe
|
||
the process that is actually running rather than a guess about it.
|
||
|
||
**The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client
|
||
became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form,
|
||
from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the
|
||
editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of
|
||
Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains
|
||
newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor.
|
||
|
||
```
|
||
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
|
||
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
|
||
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
|
||
(:op "defs") → (:status "ok" :defs ((name kind signature loc) …))
|
||
(:op "close")
|
||
```
|
||
|
||
`defs` is its own op rather than more fields on `describe`, because `describe` is what an editor *polls* — it is how the
|
||
program's output is drained — and signatures on that would be paid for every time anyone glanced at the output buffer.
|
||
It is asked once on connect and again after each accepted install. Four strings an editor reads with `read` and nothing
|
||
else: eldoc, completion and find-definition want the same three facts about a name. `loc` is empty where there is none
|
||
to give, because only `Tast.fn` carries one — an editor must refuse rather than go looking for the definition itself,
|
||
which in a program of several files finds the wrong one. Parameter *names* are not in the Tast, so a signature is `step
|
||
[i64 f32] i64`: types only.
|
||
|
||
The daemon makes its own source path absolute before building, because every location it reports derives from it. `flan
|
||
dev src/game.flan` run from a project root otherwise answered `src/game.flan:12:7`, which an editor can only resolve by
|
||
guessing which directory it was relative to.
|
||
|
||
An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage
|
||
— is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one
|
||
anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need.
|
||
|
||
`:file` is not decoration: `Session.eval`'s origin defaults to `<eval>`, so without it every error an editor shows
|
||
points into a file that does not exist.
|
||
|
||
Two things the daemon must not paper over, both of which would look like a successful evaluation:
|
||
|
||
- **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan
|
||
says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead
|
||
fails silently: the module compiles, is built, and nobody receives it.
|
||
- **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed
|
||
connect or a reply that is not `ok` becomes an error the editor sees.
|
||
|
||
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads
|
||
like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather
|
||
than leaving an editor waiting on a socket nobody is serving.
|
||
|
||
### The Emacs client
|
||
|
||
`emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is
|
||
s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is
|
||
Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is
|
||
written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords.
|
||
|
||
`emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a
|
||
request and `read` reads a reply.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `C-c C-c` | the top-level form at point, recompiled and installed |
|
||
| `C-c C-k` | the whole buffer, as **one** module |
|
||
| `C-x C-e` | the expression before point, evaluated *in the running program* |
|
||
| `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect |
|
||
| `C-c C-o` | the running program's own output, in `*flan-output*` |
|
||
| `C-c C-r` | a prompt on the running program (`*flan-repl*`) |
|
||
| `C-c C-b` | what a **stopped** program is offering, and which to take |
|
||
| `C-c C-d` | what the running program currently defines |
|
||
| `C-c C-a` | the code a name compiled to — amd64, or `C-u` for the LLVM IR |
|
||
| `M-.` / `M-,` | where a name is written, through an `xref` backend |
|
||
|
||
eldoc, `completion-at-point` and `M-.` all read one cached `defs` reply rather than asking per keystroke: eldoc fires on
|
||
an idle timer and completion inside redisplay, and neither may block on a socket or signal. The cache is refreshed at
|
||
the two moments the answer can have changed — on connect, and after an evaluation the daemon accepted — so a `defn` just
|
||
installed completes at once.
|
||
|
||
The modeline says whether there is a program on the other end, in three states. `lost` is a daemon that has gone away,
|
||
which is ordinary rather than an error — `flan dev` ends when its program does — so the next request reconnects on the
|
||
socket it was on. Strictly *before* a send, never after one: a connection that died mid-request may have died after the
|
||
daemon ran what it was given, and resending would install it twice or evaluate a side-effecting expression twice.
|
||
|
||
`C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to
|
||
arrive in the same load, or the first refers to storage that does not exist yet.
|
||
|
||
**Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary,
|
||
or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload
|
||
has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives
|
||
the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a
|
||
mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml`
|
||
and fails here.
|
||
|
||
An error comes back with a location and the client draws an overlay there, with the message beside the code, cleared the
|
||
next time that buffer's evaluation is accepted. Two things had to be right first. **The column in a `:loc` is a byte
|
||
offset**, because the reader walks the source a byte at a time — the same rule as the framing, in a different place, and
|
||
`forward-char` with it put the marker as many columns right as the line had non-ASCII characters before it. And the
|
||
daemon numbers lines from the start of what it was *sent*, so `C-c C-c` on a defn halfway down a buffer answered line 1
|
||
and every overlay would have sat on the file's first line; the client pads the form with leading newlines, which the
|
||
reader skips, so the reply's line numbers are the buffer's own.
|
||
|
||
An accepted evaluation says which names landed and what the build cost, and flashes the region that was sent. Silent
|
||
success is indistinguishable from silent failure, and `beginning-of-defun` may well have found a different form from the
|
||
one point looked like it was in.
|
||
|
||
**Live disassembly — done.** `C-c C-a` on a name writes the amd64 the running program's copy of it was assembled to
|
||
into `*flan-disassembly*`; `C-u C-c C-a` writes the LLVM IR that body was built from. `(:op "disassemble" :name … :form
|
||
"asm"|"ir")` is the op.
|
||
|
||
What makes it possible is that the daemon owns the build: it compiled every module it sent, so `objdump -d
|
||
--disassemble=flan.<name>` on the right object *is* the disassembly and the retained `.ll` is the IR. `Build.shared`
|
||
deletes its own `.ll` and `Build.executable` leaves the host's under a name that says nothing about which module it was,
|
||
so the daemon now writes its own copy beside each `.so` and keeps the host's as `host.ll` — ten reloads in, nothing else
|
||
on the machine still has that text. A table from function name to the last module accepted for it is the whole of the
|
||
bookkeeping.
|
||
|
||
**What it will not claim is that the code shown is installed**, and this is the interesting half. The agent's socket
|
||
takes a module path and five verbs; none of them reports an address, `flan_dev_cell` lives in the program's address
|
||
space, and `C-x C-e` renders a pointer as `<ptr>` on purpose — so nothing the daemon can ask would tell it what a cell
|
||
holds. The reply carries `:basis` saying which of three things is true, and the buffer prints it above the first
|
||
instruction:
|
||
|
||
- nothing has been delivered for this name, so the cell still holds the host's body — the one case that is *certain*;
|
||
- a module was delivered and the agent queued it, and the program installs it at its next frame boundary — unconfirmed;
|
||
- a module was delivered and the program is **stopped** — which says nothing either way about whether it installed,
|
||
since the commonest way to stop is to install a body and have it error; what is certain is only that nothing further
|
||
installs until it resumes.
|
||
|
||
From SBCL: offsets from the function's own start rather than addresses into an object, and `L0:` labels on branch
|
||
targets with the file address that duplicates them dropped. Not source interleaving — SBCL has the mapping and this
|
||
build emits no line tables — so the reply says that in words rather than printing a listing with no source in it. When
|
||
the debug build lands, that is the line to delete.
|
||
|
||
**Transient error overlays — done.** The diagnostic ghost text is feedback about the evaluation that just failed, not an
|
||
annotation on the source, so the next command in that buffer takes it down — edit, motion, evaluation, anything.
|
||
`pre-command-hook` and not `post-command-hook`, which fires at the end of the *failing* command and would clear the
|
||
overlay before redisplay had drawn it. The hook is buffer-local and lives exactly as long as an overlay does: added
|
||
where one is drawn, removed where they are cleared, so a session of twenty buffers is not running it on every keystroke
|
||
in all of them. `execute-kbd-macro` runs no `pre-command-hook` under `--batch`, so the test drives `run-hooks` — the
|
||
same call the command loop makes — and checks the hook is installed in that buffer and in no other; that Emacs runs it
|
||
is Emacs' contract and a test claiming to check it would be checking nothing.
|
||
|
||
**The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the
|
||
next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the
|
||
output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and
|
||
not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the
|
||
accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot
|
||
grow the daemon without limit.
|
||
|
||
### `C-x C-e` — evaluating an expression
|
||
|
||
A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body
|
||
into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by
|
||
exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an
|
||
expression reading the program's state sees a point the program agrees is consistent.
|
||
|
||
**Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say
|
||
what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout
|
||
decision's bill, and it is why the printer set is small rather than universal.
|
||
|
||
It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a
|
||
dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's
|
||
socket. The read is safe without a handshake because `flan_dev_result` bumps a generation counter last; the daemon waits
|
||
for it to move rather than assuming the program has reached a frame boundary.
|
||
|
||
**This renderer is most of `println`**, which is worth knowing before anyone schedules it. plan.org describes a
|
||
compiler-provided, type-directed intrinsic that selects or emits a structural printer per concrete instantiation, prints
|
||
structs, fixed arrays and options structurally, prints a `Ptr` as its address rather than following it, and bounds depth
|
||
and length. That is a description of what `Session` already does for `C-x C-e` — same walk, same refusals, same three
|
||
bounds — aimed at `flan_dev_emit` and the wire instead of at stdout. What `println` needs on top is a stdout sink, a
|
||
builtin that takes its printer from the argument's type, and the `any`/`Error` dynamic cases, which have no compile-time
|
||
type to walk. Not the walk itself.
|
||
|
||
The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise
|
||
because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an
|
||
allocator the language does not have.
|
||
|
||
```
|
||
big 18446744073709551615
|
||
col :blue
|
||
(.pos b) (V {:x 1.5 :y 0})
|
||
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
|
||
(slice (.tags b) 0 3) [ 0 42 0]
|
||
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
|
||
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
|
||
```
|
||
|
||
Details that are decisions rather than formatting:
|
||
|
||
- **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come
|
||
back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point
|
||
instead.
|
||
- **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather
|
||
than as the value it is.
|
||
- **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are
|
||
erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is
|
||
exactly what you would want to see.
|
||
- **A pointer is never followed** — `<ptr>`. It is the only thing that could make the walk cycle, and dereferencing one
|
||
a REPL was handed is not a safe thing to do on someone's behalf.
|
||
- **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does
|
||
not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K
|
||
and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it
|
||
means no renderer carries a budget.
|
||
- A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is
|
||
not evaluated once per element.
|
||
|
||
What still refuses by name: `Map`, `Fn`, a type variable.
|
||
|
||
A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer.
|
||
`flan run calc-me.flan "1.5 * 2.0"` has always said `3`.
|
||
|
||
An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe`
|
||
does not fill up with `eval/N` for every expression ever typed.
|
||
|
||
**The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by
|
||
`flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points
|
||
into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the
|
||
agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition*
|
||
retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind,
|
||
and can never claim this.
|
||
|
||
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop
|
||
would exhaust it.
|
||
|
||
The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must
|
||
disagree. A value computed in the compiler, or read out of a copy of the program's state, would not.
|
||
|
||
### The REPL buffer
|
||
|
||
`flan-repl.el` is a `comint-mode` buffer whose every line goes through the same `eval-expr` request `C-x C-e` uses. No
|
||
new protocol and no compiler support. Deriving from `comint` rather than hand-rolling a prompt is the same call as
|
||
deriving `flan-mode` from `lisp-mode`: history, the input ring and kill/yank already exist. There is no subprocess
|
||
behind it — the "process" is a stub comint needs in order to have a prompt at all.
|
||
|
||
Three things about it that are decisions:
|
||
|
||
- **It is program-scoped.** A name typed at the prompt resolves against the running program's top-level namespace, so in
|
||
sand you write `sim/settle` and not `settle`. A buffer visiting a package's own file gets the alias applied for it
|
||
because the file says which package it belongs to; a prompt has no file and nothing to derive one from.
|
||
- **RET on a half-typed form opens a line instead of sending it.** Balance is checked with the Flan syntax table, so a
|
||
paren inside a string does not count.
|
||
- **A value and the program's output are different things and arrive by different routes.** The value is the result of
|
||
the request and appears at the prompt; anything the program printed while evaluating it rides along on the same reply
|
||
and goes to `*flan-output*`. Showing them in one place would be convenient and wrong, so there is a test for the
|
||
separation.
|
||
|
||
That test is what caught a real bug: the renderer's `Unit` case emitted `()` without evaluating the expression, so
|
||
`(print-line "x")` — the most ordinary thing anyone types at a prompt — answered `()` while nothing happened. A Unit
|
||
expression is almost always a call made for its effect, and is now evaluated and *then* reported.
|
||
|
||
### Conditions — step 1: `handler-bind` and `signal`
|
||
|
||
`spec-conditions.md` §1 and §2, and nothing else yet. They are worth having on their own because **neither alters
|
||
control flow**: `signal` returns `Unit` whatever it finds, a handler that returns normally leaves the signalling
|
||
function to carry on, and with nothing matching it is a no-op. So none of the transfer machinery §6 describes exists
|
||
yet, and no signature changed.
|
||
|
||
```
|
||
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
|
||
(load-all))
|
||
```
|
||
|
||
The runtime is a linked list: establishing a handler is two stores and a push onto a frame allocated on the establishing
|
||
function's own stack, and `signal` with an empty stack is a null check — which is what §2 asks for. Popping is by frame
|
||
rather than by count, so restoring what this one displaced is right even if something below it left the stack out of
|
||
step.
|
||
|
||
Three decisions worth keeping:
|
||
|
||
- **A condition's type is a hash of its name**, not an index. An index would shift the moment a struct were added, and
|
||
every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
|
||
- **The condition crosses as a pointer**, because a handler runs while the signalling frame is still alive and there is
|
||
nothing to copy. What the clause *binds* is the condition itself, though — the pointer is a hidden parameter and the
|
||
name is a slot loaded from it, so a handler passing `c` to something expecting the struct is not handed an address
|
||
instead.
|
||
- **A clause is lifted into a function of its own.** A handler runs from wherever the signal was, so it cannot be a
|
||
branch in the function that wrote it.
|
||
- **A pushed handler frame holds the clause's body address, not a cell.** This is a deliberate divergence from
|
||
plan.org's rule that a top-level function value is a stable trampoline over the cell and never the address of a
|
||
particular body. A handler frame is not a `Fn` value — nothing in the language can name it — and it is live only for the
|
||
duration of the `handler-bind` body, so a reload landing while it is on the stack finds the clause it pushed still
|
||
valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the
|
||
stack does *not* observe a redefinition of its own clause; the next entry to the `handler-bind` pushes the new one. When
|
||
`Fn` values arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
|
||
|
||
Which gives the two refusals, both by the house rule rather than by accident:
|
||
|
||
- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a
|
||
reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in
|
||
scope, which is what the accumulation case needs.
|
||
|
||
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not
|
||
outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by
|
||
value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete
|
||
use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the
|
||
biggest usability limit in conditions is not behind the thing that was just deferred.
|
||
- **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would
|
||
leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block.
|
||
|
||
### The break loop — conditions step 3
|
||
|
||
Where **"a crash kills the program"** stops being true. An unhandled `error` runs a hook instead of `rt_die()`, on the
|
||
frame that erred with nothing unwound, so the condition and every restart between there and the top are still live.
|
||
|
||
```
|
||
flan: unhandled Missing — stopped, not dead.
|
||
restart: retry
|
||
restart: use-placeholder
|
||
```
|
||
|
||
Four decisions, each of which is the reason something is where it is:
|
||
|
||
- **It is a hook, not a call.** The loop lives in `vendor/agent/`, which is an optional package; `flan_rt.c` is the
|
||
release runtime and must not depend on something a program may never import. A program with no agent leaves the hook
|
||
null and dies exactly as it did before.
|
||
- **The hook resumes by writing a restart into the transfer channel** — the same channel an `invoke-restart` writes,
|
||
reaching the same guard. Choosing a restart from the break loop and choosing one from a handler are therefore the same
|
||
act, lowered once. Nothing about §6 needed changing to support it.
|
||
- **The break loop *is* the poll loop**, run from the error instead of from the frame boundary. That is not a
|
||
convenience: an expression evaluated while stopped is a module the listener queues and the game thread runs, so a loop
|
||
that did not drain that queue would hang `C-x C-e` exactly when it is most wanted.
|
||
- **Installing while stopped is allowed**, which contradicts a rule stated above and should. "A redefined function must
|
||
not be swapped while it is on the stack" is about *mid-frame consistency* — half a frame of old code and half of new —
|
||
and there is no frame in progress here. The old body on the stack keeps running; a `retry` restart calls through the
|
||
cell and reaches the new one. That is the fix-it-and-retry loop, and refusing the install would remove the point of
|
||
stopping.
|
||
|
||
**A restart frame carries its name now**, beside the hash. Matching never needs it — that is what the hash is for — but
|
||
a break loop has to *show* someone their choices and nothing at run time can turn a hash back into a name. It is also
|
||
`compute-restarts`' data, whenever that arrives.
|
||
|
||
**A choice is validated on the listener thread**, against a stack the stopped game thread is holding still, and refused
|
||
there. Accepting it and discovering on the game thread that no frame offers it would answer `ok` for something that
|
||
cannot happen.
|
||
|
||
The socket verbs are `restarts`, `restart <name>` and `abort`, and all three are refused with the reason when the
|
||
program is not stopped — there is no restart stack to walk from a running one. `test/programs/break.flan` errors twice
|
||
and the test takes a *different* restart each time, so a loop that always resumed the same way could not pass.
|
||
|
||
### The break loop in the editor — conditions step 3, the other half
|
||
|
||
The loop above is reachable from a raw socket. This is the half that makes it reachable from Emacs, and the whole of it
|
||
follows from one fact: **a program stops at a moment nobody asked about.** Every other op in the protocol answers a
|
||
question an editor chose to ask.
|
||
|
||
So the state is learned **twice, deliberately**:
|
||
|
||
- **It rides on every reply**, beside the program's output and for the same reason. `:stopped t :condition "Missing"`,
|
||
or `:stopped nil`. The likeliest instant for a program to stop is the one just after an evaluation — a body that now
|
||
errors — and that is a reply the client is already reading. Finding out a second later from a poll would mean finding
|
||
out *after* the echo area had said the evaluation was fine.
|
||
- **And a timer asks anyway**, once a second, with `describe` — the cheap op, which is also how the output pipe is
|
||
drained. A program that stops in a frame of its own game loop produces no reply at all, and folding state into replies
|
||
that never come says nothing. The timer never *reconnects*: `flan-dev--request` reopens a socket a restarted daemon left
|
||
behind, which is right for something a person did and wrong for a background poll, because it would quietly erase the
|
||
`lost` state that exists to be seen. It also skips while another request is in flight — `accept-process-output` runs
|
||
timers, so a poll firing inside a read would eat the reply that read was waiting for.
|
||
|
||
Three ops, and the *annotation* owns `:stopped`, not the ops — one place in the daemon decides whether the program is
|
||
stopped, so the poll and the prompt cannot disagree.
|
||
|
||
```
|
||
(:op "break") → (:status "ok" :restarts ("retry" …) :stopped t :condition "Missing")
|
||
(:op "restart" :name "retry") → (:status "ok" :restart "retry" :note "accepted; …")
|
||
(:op "abort") → (:status "ok" :note "the program is exiting; …")
|
||
```
|
||
|
||
`break` carries only the restart names, because those cost a second round trip to the program and are wanted only by
|
||
someone about to choose one.
|
||
|
||
**`ok` from `restart` means accepted, not resumed.** The choice is validated on the program's listener thread against
|
||
the stopped stack, then taken when that thread next comes round its loop. A client that read it as "running again" would
|
||
poll once, find it still stopped, and re-open the prompt it had just answered — so the client clears its own flag and
|
||
lets the next poll settle it.
|
||
|
||
The modeline is a fourth state, `flan:stopped(Missing)`, before `live`: a stopped program looks exactly like a running
|
||
one from anywhere else in Emacs. The prompt is a `completing-read` over the names with `require-match`, which is exactly
|
||
right for a closed set the program computed, and `abort` is the last entry on that same list rather than a second key —
|
||
it is the thing you pick when none of the restarts is the answer.
|
||
|
||
**`flan_agent_poll` had to become re-entrant**, and that is the one thing here that was a bug rather than an addition. A
|
||
`C-x C-e` thunk may itself error; the break loop that catches it polls again from inside that very call. The old loop
|
||
cached `head` and `tail` and wrote `tail` back at the end, so the outer call rewound the index over everything the
|
||
nested one had consumed — and re-ran the thunk that had just stopped the program, which is an unbounded recursion of
|
||
breaks rather than a stumble. It now claims each job by advancing `tail` before running it, and re-reads both indices
|
||
each time round. Still single-consumer: only the game thread writes `tail`, nesting included. `test_dev.ml` evaluates an
|
||
expression that errors and resumes it, which fails against the old shape.
|
||
|
||
The agent grew one verb, `status`, answered in **both** states — `running` or `stopped <condition>`. Everything else the
|
||
break loop offers is refused while running, and rightly; but the question an editor asks *without already knowing* had
|
||
to have an answer either way, or there would be nothing to poll. The condition is its class name and nothing more: the
|
||
hook is handed a name and an opaque pointer, and nothing at run time can render a value whose type it does not know.
|
||
|
||
`test/programs/dev-break.flan` stops on its first frame, so the daemon meets a program that is *already* stopped — the
|
||
state an editor has to cope with and the hardest one to arrange later. The Emacs test breaks a program the other way
|
||
round, by installing a `step` that errors into a loop that calls it, fixes it while stopped, and then resumes: `C-x C-e`
|
||
answering while the program sits in the break loop is checked there against the real client, not only in OCaml.
|
||
|
||
### Conditions — step 2: `restart-case` and `invoke-restart`
|
||
|
||
`spec-conditions.md` §3 to §6: the transfer. A handler runs where the signal was, decides, and control resumes at a
|
||
`restart-case` further out.
|
||
|
||
```
|
||
(defn fetch [n i32] i32
|
||
(restart-case (middle n) ; its value if nothing transfers
|
||
(use-placeholder [] -1)
|
||
(retry [] 7)))
|
||
|
||
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
|
||
(fetch 2)) ; -1
|
||
```
|
||
|
||
**The channel is an out-parameter**, as §6 now says: one `ptr` appended to every Flan signature, written by an
|
||
`invoke-restart` and checked after every call. The return type stays what the source says, so the disassembly is the
|
||
release one plus a guard, and one pointer threads down the whole chain — a callee writes the target into its caller's
|
||
slot and each frame only has to check and return early, which reuses the existing `return` path and with it §5's defers.
|
||
`Emit.signature` was already the one place a signature is spelled, which is what made this a three-line change rather
|
||
than a hunt.
|
||
|
||
**Every function is transfer-transparent**, release included — and that is the ABI, not a stopgap. A cell holds a bare
|
||
pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and
|
||
plan.org both now say the later optimisation may stop a function *checking* the channel, or pass the pointer straight
|
||
through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform
|
||
also means redefinition acquires no new refusal class.
|
||
|
||
**The transfer target is the restart frame's own address, not a clause id.** This is a correction to what the previous
|
||
note settled. A static id has to be unique against every module a running program may *later* load, and a hash is only
|
||
probably unique — two `restart-case`s colliding means the inner one silently catches a transfer aimed at the outer. The
|
||
frame is an `alloca` in the function that offers it, so its address is exact, and it also says *which* clause, which is
|
||
how clause ids disappeared entirely. §6 says "transferring to frame N" and this is closer to it than the number was.
|
||
Re-entering a `restart-case` then works with nothing extra: each activation allocates its own frames, and §4's
|
||
"innermost offering the name" is just the order of the walk.
|
||
|
||
**Cleanup happens in landing blocks, one per region.** A guard branches to the innermost open one, which pops whatever
|
||
frames it established and either catches the transfer or forwards it outward:
|
||
|
||
- a `restart-case`'s pops its restart frames, compares the target against its own, and either runs that clause or puts
|
||
the target back and goes on out;
|
||
- a `handler-bind`'s pops its handler frames and goes on out — which is the path a transfer out of a handled body takes,
|
||
and without it the handler stack would be left pointing into a frame that has gone;
|
||
- the function's own runs its defers (§5) and returns early. `errdefer` does not run and never could: `try`/`Result` is
|
||
still refused by name.
|
||
|
||
A single function-wide unwind block would have been wrong for the first two: a call inside a `restart-case` body would
|
||
jump straight past the very form that was supposed to catch it.
|
||
|
||
**The channel is cleared before any cleanup runs and put back after.** A defer makes ordinary calls and each one is
|
||
guarded; with the channel still set the first of them would branch straight back into the landing block it came from.
|
||
Same reason the clause body starts with it null.
|
||
|
||
`flan_signal` takes the channel and passes it to each handler, and stops walking once one has written to it. That makes
|
||
the one C frame every handler is reached through transparent to a transfer — it has to be, or §6's "a transfer cannot
|
||
cross a foreign frame" would make `restart-case` useless. It is also the only such frame: `extern` is Flan-to-C only and
|
||
there are no function values yet, so nothing can call *back* into Flan across one.
|
||
|
||
Scope, each piece refused by name with its reason and a test on the reason:
|
||
|
||
- **restarts take no parameters.** That covers §1's own `load-texture` example and skips argument marshalling and §3's
|
||
runtime arity check.
|
||
- **TODO — CL-style interactive recovery.** A stopped program should be able to offer a typed restart such as
|
||
`(use-value [value T] value)` or `(use-function [replacement (Fn ...)] ...)`, and the editor should show its signature
|
||
and ask for the replacement before invoking it. That is the missing "this variable is None; what should I use instead?"
|
||
path: named, hard-coded branches are useful for `retry` and `skip`, but not a substitute for an interactive value or
|
||
alternate implementation. It needs restart argument marshalling and validation in the runtime, plus an editor protocol
|
||
for entering and checking the supplied value/expression.
|
||
- `return` inside a `restart-case` body, exactly as inside `handler-bind`: a bare `ret` skips the pops.
|
||
- one `restart-case` offering a name twice — §4 finds the first frame offering it, and two in one frame makes that a
|
||
choice nothing in the source shows.
|
||
- `invoke-restart` inside a `defer`. A defer *is* the cleanup a transfer runs on its way out, so a transfer starting
|
||
there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a
|
||
defer that reaches one through a call is trapped at run time by `flan_transfer_fail`, because nothing static could see
|
||
it.
|
||
- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past
|
||
everything. There is nowhere to resume, so there is nothing else to do.
|
||
|
||
**`(error c)`, §2.** The same walk as `signal`, and the difference is entirely what happens when the walk ends: `signal`
|
||
returns `Unit` and the signalling function carries on, `error` has type `Never` and stops. So only a transfer gets past
|
||
it, which is why `emit` puts a guard after the call and then `unreachable` — and why `flan_error` cannot be marked
|
||
`noreturn`, since it does return, on exactly one path. Being `Never` is also what lets it stand as a `restart-case`
|
||
body's fall-through, which is the shape §1's `load-texture` example needs. `test/programs/error.flan` is the unhandled
|
||
case: it cannot be an `outputs` row, because it does not exit 0.
|
||
|
||
`flan_transfer_fail` covers the ordinary return path as well as the unwind one: a defer that reaches an `invoke-restart`
|
||
through a call traps either way, and the message names the rule rather than the path, since the rule is the same.
|
||
|
||
**A lifted handler clause is named after the function it came out of** — `handler/step/0/Missing` — and is emitted by a
|
||
redefinition module alongside the body it belongs to, hidden, for the same interposition reason the body is. This was a
|
||
hole step 1 left: the name used to be numbered by position in the whole program's lifted list, so it was neither stable
|
||
nor attributable, and a redefinition of a function containing a `handler-bind` failed in `llc` with an undefined value.
|
||
A clause is reached by address from its parent's body and from nowhere else, so it takes no cell and no registry slot.
|
||
|
||
`test/test_dev.ml` drives that path: a third evaluation redefines `step` to a `restart-case` whose frame is an `alloca`
|
||
in the newly loaded module, whose guarded call goes through the host's cell, and whose transfer starts in a handler and
|
||
crosses `probe`, which the host was compiled with. Those three do not meet anywhere else.
|
||
|
||
Two things found by writing it:
|
||
|
||
- **`{ ctx with in_handler = true }` was a latent bug.** `ctx.slots` and `ctx.slot_tys` are mutable, so a copy allocates
|
||
the body's slots into a record the function never sees again and the indices collide. It was harmless only because no
|
||
`handler-bind` body in the tests had a `let` in it. The flags are set on `ctx` and restored now.
|
||
- **`test/reload_host.c` had to learn the parameter.** It calls `flan.outer` through an `__asm__` label, which does not
|
||
fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere
|
||
else entirely.
|
||
|
||
`test/programs/restarts.flan` runs in the acceptance table at `-O2`, at `-O0` and as a dev build. `-O0` is not redundant
|
||
here: the guard after every call is control flow the optimiser would otherwise launder, and the dev build is where each
|
||
of those calls goes through a cell.
|
||
|
||
### What is left
|
||
- **Editor comforts**: completion, eldoc, jump-to-definition, error overlays.
|
||
|
||
**Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the
|
||
memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary.
|
||
Attaching to a process someone else built is not a thing to support by default.
|
||
|
||
## Where build time goes
|
||
|
||
`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms:
|
||
|
||
| Step | Cost |
|
||
|---|---|
|
||
| frontend: read → parse → load → check → emit | <10ms, below the timer |
|
||
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
|
||
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
|
||
| link | 20ms |
|
||
|
||
Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which
|
||
compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler
|
||
(its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build),
|
||
`opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0`
|
||
and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d
|
||
into place, so two concurrent builds cannot see a half-written one.
|
||
|
||
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its C was never
|
||
the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever
|
||
a disk-space decision.
|
||
|
||
The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It
|
||
is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted
|
||
IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse
|
||
errors.
|
||
|
||
|
||
## Next — the REPL is the priority
|
||
|
||
Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place),
|
||
and **the dev loop is the thesis of the project**, so it comes first. Staged so each step is runnable on its own — the
|
||
failure mode is building a daemon and a protocol before knowing the reload primitive works.
|
||
|
||
1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms.
|
||
See the section above.
|
||
2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`,
|
||
`runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a
|
||
run-time-added function is itself redefined. See the section above.
|
||
3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs,
|
||
and sand.flan polling at the top of its frame. See the section above.
|
||
4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than
|
||
nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if
|
||
something else ever needs to talk to it.
|
||
|
||
**One decision left to settle before step 2**, because both change codegen and are painful to retrofit:
|
||
|
||
- ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module
|
||
declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep
|
||
the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload,
|
||
and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.
|
||
- **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what
|
||
`load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from.
|
||
|
||
Deferred until after the dev loop:
|
||
|
||
6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and
|
||
`test/programs/sand-headless.flan` prints `-2851001042534928384` under it — the same hash as native, byte for byte, at
|
||
`-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared
|
||
across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer
|
||
would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and
|
||
runs it — rather than by looking for a binary on PATH.
|
||
|
||
Three things this cost that were not in the old note:
|
||
|
||
- **The entry point is not `main`.** wasi-libc's start code calls `__main_argc_argv`; clang renames C's argc/argv `main`
|
||
to that, and the `.ll` `Emit` writes says `@main` literally. The link succeeds and the program traps on a
|
||
signature-mismatched weak stub. `Build.wasm_main_source` is a two-line C shim that bridges it, and the `__asm__("main")`
|
||
label in it is load-bearing: spelling the callee `main` makes clang rename *that* too and the shim becomes an infinite
|
||
self-call.
|
||
- **The target has to reach the C compiles, not just the link.** `flan_rt.c` includes `<stdio.h>`; without `--sysroot`
|
||
it never gets that far. `target_flags` is computed once and passed to both, and the whole flag list — not just the
|
||
triple — is in the object cache key, so repointing a sysroot cannot serve a stale `.o`.
|
||
- **Fedora's sysroot is one level deeper** than wasi-sdk's: `include/wasm32-wasi/stdio.h`, not `include/stdio.h`. Both
|
||
shapes count.
|
||
|
||
**The glued joint, and the one thing this contradicts in the old note.** The old note said the builtins archive has to
|
||
come from a wasi-sdk release. It does not have to: emscripten builds the same compiler-rt for wasm32 and calls it
|
||
`libcompiler_rt.a`, and dropping that in as `libclang_rt.builtins.a` links and runs. It is a different triple
|
||
(`wasm32-unknown-emscripten`) built by a different clang (22 against Fedora's 20), so it is *substituting*, and wasi-sdk
|
||
is still the proper article. `build.ml` looks for `FLAN_WASM_BUILTINS`, then `/opt/wasi-sdk/...`, then emscripten's
|
||
beside `emcc` on PATH, and refuses by name listing every path it tried when none is there. clang's resource directory is
|
||
root-owned, so the archive is not dropped into it — a shadow resource directory is built under the object cache, named
|
||
by a digest of clang's own resource dir plus the archive's path, size and mtime, with the real `include` symlinked in.
|
||
|
||
**The runtime is Node.** No `wasmtime` and no `wasmer` on this machine; `test/wasm-run.mjs` is twenty lines of
|
||
`node:wasi` and the table prefers `wasmtime` or `wasmer` if either appears. `--no-warnings`, because `node:wasi` writes
|
||
an `ExperimentalWarning` to stderr on every run and the harness compares combined output.
|
||
|
||
**Refused by name, not half-supported:** `--dev` with a wasm target (the reload path is `dlopen`), `Build.shared` with
|
||
one (same reason), and `flan run --target=` (a `.wasm` is not something this host execs — build it and point a runtime
|
||
at it).
|
||
|
||
Still open: raylib on wasm, which plan.org wants through emscripten and its own sysroot. wasi-sdk is right for the
|
||
headless table; it is not necessarily right for the eventual game build.
|
||
|
||
7. **Loose ends from milestone 4**, none of them blocking: block-scoped `defer`; package visibility, so
|
||
`rl/get-color-raw` is not callable; a package importing a package; imported unions.
|
||
|
||
## Macros — the reader and the declaration are in, the expander is not
|
||
|
||
The front half landed. What exists:
|
||
|
||
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, `(unquote x)` and `(unquote-splicing x)`, exactly
|
||
as `'x` reads as `(quote x)`. It stays dumb: it does not count nesting levels, does not know whether an unquote is
|
||
inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma
|
||
is whitespace in `is_delimiter` and every binding vector in the corpus relies on that. Backtick and tilde are delimiters
|
||
now, so `a~b` is two things.
|
||
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say expansion is not wired up; `unquote` and
|
||
`unquote-splicing` say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature.
|
||
`(defmacro name [params] body ...)` at the top level is checked for shape and *then* refused — a malformed defmacro and
|
||
an unimplemented one get different reasons, so the shape rule is enforced before the feature exists.
|
||
|
||
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, because a table nothing reads is a place
|
||
for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit.
|
||
|
||
### How the expander should work
|
||
|
||
**There is no interpreter** (see "Why there is no interpreter") and there is not going to be one, so running a macro at
|
||
compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured:
|
||
`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload
|
||
primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.
|
||
|
||
The shape it wants:
|
||
|
||
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its result is a form, which means `Form.t`
|
||
has to exist on the Flan side — a `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and accessors.
|
||
That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on
|
||
the *layout* of a `Form`, not merely its shape, so whatever the checker does for unions has to be exact here. Until
|
||
unions are values this cannot start — `check.ml` puts union values and `match` on a union at **milestone 6**, so that is
|
||
milestone 6 work landing before milestone 5's.
|
||
2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: there is no `Ast.Defmacro` and `Parse` refuses
|
||
`defmacro` outright, so an `Ast`-level pass would have nothing to work with. That refusal is not a dead end, it is the
|
||
ordering — the expander runs first and `Parse` never sees a macro call at all. It is also the Clojure ordering, and the
|
||
reason a macro expanding to a special form is ordinary rather than a special case.
|
||
3. **Order matters and files do not have one.** Top-level names in a package are order-independent everywhere else
|
||
(`declared_types`, the constant fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and loaded before a
|
||
call to it is expanded. Either collect every `defmacro` in a pre-pass and compile them as one module, or require
|
||
definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest
|
||
of the frontend already behaves.
|
||
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be
|
||
detected and named rather than looping.
|
||
5. **`gensym` is a runtime function of the compiler**, called by the loaded macro while it runs. It needs a counter that
|
||
lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a
|
||
character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org,
|
||
open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no `macrolet` until a
|
||
concrete use case appears.
|
||
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: `` `(a ~b) `` becomes list-construction
|
||
over quoted pieces, with `~@` splicing. Written once, in the expander, over `Form`.
|
||
|
||
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — were owned by other lanes when the front
|
||
half landed, which is the only reason the expander is not here too.
|
||
|
||
### What would tell you it works
|
||
|
||
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` today, and plan.org milestone 5 says
|
||
they are special forms *only until macros land*. Moving one of them out of the compiler and into the prelude as a
|
||
`defmacro`, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote,
|
||
`gensym` and the ordering pre-pass at once, against a test suite written before any of them existed.
|
||
|
||
## Watch for
|
||
|
||
The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is
|
||
not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
|
||
`Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone
|
||
it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The
|
||
tests assert on the reason, not just on the failure.
|
||
|
||
## Untracked on purpose
|
||
|
||
`calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored
|
||
(`/calc-me`, `/sand`) so the patterns cannot match anything nested.
|
||
|
||
`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root
|
||
`dune` file. Its contents are also in git history at `2c232dd`.
|