# Where this is **The dev loop is closed.** `C-c C-c` in Emacs recompiles the top-level form at point and installs it in a running program, at that program's next frame boundary. Verified against sand: an unsaved buffer edit to `game-draw`, and 240 consecutive frames drew it. Steps 1, 2 and 3 are done — see *The reload primitive* below. A list of top-level forms can be recompiled and installed into a running process; call sites compiled before they existed follow them, and a `defn` or `defvar` the process was never built with can be added and then redefined again. That is the whole of `C-c C-c`, minus an editor: sand.flan takes a redefinition over a socket and installs it between frames. What is left is the *session* — something that holds the checker environment between evaluations, tracks which names the running process was built with, and speaks a protocol an editor can talk to. Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its simulation has a headless acceptance case that runs on the `dune test` path at `-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and runs; the interpreter was dropped — open decision #7, settled, see below). ``` reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅ ``` | File | What it does | |---|---| | `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception | | `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` | | `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex | | `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` | | `lib/parse.ml` | forms → AST; special forms, desugaring, declarations | | `lib/load.ml` | **imports: a package directory → qualified declarations** | | `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere | | `lib/tast.ml` | the typed IR the backend consumes | | `lib/check.ml` | AST → typed IR; two passes, bidirectional | | `lib/session.ml` | **a live program: what the process was built from, plus every change since** | | `lib/wire.ml` | **the editor protocol: one s-expression per message, length framed** | | `lib/dev.ml` | **`flan dev`: a session, the program running beside it, and a socket** | | `lib/prelude.ml` | printers + `rand-f32`, written in Flan | | `lib/emit.ml` | typed IR → LLVM IR text | | `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable | | `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions | | `runtime/flan_dev.c` | **dev only: the by-name registry a run-time-new name needs** | | `vendor/raylib/` | **the raylib package: `raylib.flan`, `shim.c`, `link`** | | `vendor/agent/` | **the dev agent: a socket, a loader thread, install at a frame boundary** | | `emacs/` | **`flan-mode.el` and `flan-dev.el`: the editor half of the dev loop** | | `sand-sim/` | **the falling-sand simulation, with no raylib in it** | | `bin/main.ml` | `flan read \| parse \| check \| emit \| build \| run \| reload \| dev` | | `test/test_flan.ml` | reader, parser and checker | | `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps | | `test/test_reload.ml` | **the reload primitive: recompile one function, load it, call it** | | `test/test_agent.ml` | **a running program taking a redefinition over a socket** | | `test/test_session.ml` | **what a running process cannot be told, and recovering from a typo** | | `test/test_dev.ml` | **the daemon, driven the way an editor drives it** | | `test/test_repl.ml` | **`C-x C-e`: an expression evaluated inside a running program** | | `test/test_emacs.ml` | **the client, driven against a real daemon and a real program** | | `test/reload_host.c` | the C host that loads and installs two rebuilds, in one process | ``` $ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2" 3.5 $ flan run test/programs/sand-headless.flan 2256461126764447066 $ flan run sand.flan # a window, 120 fps, hold space ``` ## What milestone 4 added **`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node. The bound is evaluated once into a hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not assignable, which makes the generated step its only writer. **`defer`** is recognised in `check_fn` and nowhere else, because that is the only place that knows a form is at the top level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it stands. Function exit runs them innermost-first, and an explicit `return` runs the ones registered *above* it — a defer written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the bounds-check shape (`noreturn` then `unreachable`) rather than being a separate decision. `defer` inside a `let`, a loop or a branch is **rejected**, not accepted with function scope. It would run once at function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. Block-scoped defer is real work and is not done. **New builtins:** `zeroed` (takes its type from the place it is stored into), `min`/`max` (each operand through a slot, so neither is evaluated twice), `bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a signed type and logical on an unsigned one), and `rand-f32`. **`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state. It is not libc's, because a grid hash is only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed` sets the state. This is what the bitwise operators were added for. **Enums and keywords.** `(defenum Name [member value ...])` gives a type that is an `i32` at run time and its own type in the checker, so `:space` at a call site resolves against the parameter's enum and a typo is an error there rather than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on. ## Why the FFI goes through a C shim The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64: ``` Vector2 {float,float} → declare <2 x float> @GetMousePosition() Color {u8,u8,u8,u8} → declare void @ClearBackground(i32) Rectangle {4 × int} → declare { i64, i64 } @mkrect() ``` None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a per-target classification the *caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that in `emit.ml` is three classifiers to write and then keep correct forever, and a mistake shows up as `(.y m)` returning garbage rather than as a link error. So `vendor/raylib/shim.c` has one wrapper per binding, each one flattening the aggregates: a struct returns through an out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a copy. clang classifies all of it, per target, for free. `check.ml` enforces the rule — an aggregate in a `declare` signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host ABI, implemented twice", and `flan_rt.c` is the same pattern. The price is a hand-written wrapper per raylib call. They are one-liners and mechanical enough to generate if that ever becomes the bottleneck. `raylib.flan` declares each `-raw` entry point and wraps it in an ordinary Flan function just below, so the surface sand.flan sees is `(rl/get-mouse-position)` returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)` comes back as `17 34 51 68`, four separate bytes — a `Color` is *not* the little-endian reading of the packed integer, so an identity would have passed a weaker test. That case is in the acceptance table, skipped if `libraylib` is not installed. The bindings are 18 calls: window (`init-window`, `close-window`, `window-should-close?`, `set-target-fps`, `set-trace-log-level`), keyboard (`key-pressed?`/`down?`/`released?`), mouse (`mouse-button-pressed?`/`down?`/ `released?`, `get-mouse-position`), `get-color`, and drawing (`begin-drawing`, `end-drawing`, `draw-fps`, `clear-background`, `draw-rectangle`), plus the `Key`, `MouseButton` and `TraceLogLevel` enums. Adding one is three lines: a `declare`, an `extern` prototype, and a one-line wrapper. No raylib headers are needed: `shim.c` declares the prototypes it uses, so the build depends on the shared library being linkable and not on `raylib-devel`. `vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the runtime library without the `.so` symlink. ## Packages `lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs. The directory is the package; `vendor:` is a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no collection is relative to the importing file. Importing is a **rename**: every top-level name the package declares becomes `alias/name`, and every use of one — in a type, in a body, in a struct literal, in an *array length* — is rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of declarations whose names contain a slash. A package may also carry the C it binds to: every `.c` file in the directory is compiled into the build, and a file named `link` lists extra linker arguments. This is not a module system yet. No visibility (hence `rl/get-color-raw` being callable), no cycle detection, and a package cannot import another one. ## sand.flan is two programs plan.org wants sand tested twice — interactive at 120 fps, and headless over N frames with the grid hashed, the version CI runs on native *and* wasm32. Those cannot be one binary: `Load` collects a package's C sources and linker arguments unconditionally, so anything importing the raylib package links libraylib on every target regardless of what its `main` does, and on wasm32 that link cannot succeed. So the simulation moved to `sand-sim/`, which imports nothing. `sand.flan` imports it as `sim/` and adds the window, the mouse and the drawing; `test/programs/sand-headless.flan` imports it and adds a seed, four deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics. The headless case is what actually *verifies* milestone 4 — running the interactive build only proves it enters its loop, because with no mouse input the grid stays empty and `paint-at`, `settle` and `move-grain` never execute on real data. Measured through the probe: 168 grains painted around row 4–8, still 168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost. **Three edits were made to sand.flan's own text**, and they are language decisions rather than fixes: - `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float constant is `f64`, `velocity` is `[f32]`, and there is no implicit widening. - `(defvar current-color u32)` → `i32`. It is an index into `colors`, and `(len colors)` is an `i32`. - The file was split as above, so its body now says `sim/rows` and so on. `(defn main [])` is unchanged — the short form, as plan.org says. Painting is on **hold left mouse button** rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press. ## Bounds checks — done at milestone 3 `at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a failure names the source location. Three check sites: `at` on `[n T]` (static bound, folded by LLVM for a literal index — and a literal that is out of bounds never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime len), and `slice` (two comparisons — `lo <= hi` is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. `Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is what lets the acceptance table run the same programs at `-O0` and `-O2` with identical checks. The flag is `--no-bounds-checks`. The write path is its own case: `(set (at arr n) …)` lowers through `place`/`Pindex`, not through `At`, so a refactor that split them would break the write check silently. The test covers both. Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable. ## Why there is no interpreter Open decision #7 is settled: **the compiled path is the only backend.** Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path. ## The layout, which is the whole backend design ``` i8..i64 / u8..u64 i8..i64 signedness lives in the ops f32 f64 float double bool i1 an enum i32 [T] and string { ptr, i64 } ptr+len, non-owning [n T] [n x T] inline, a value (Ptr T) ptr opaque pointers (Option T) { i8, T } tag 0 None, 1 Some a struct a literal struct, declaration order Unit and Never {} ``` No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics: - **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a `store`, and a `store` of an aggregate *is* the copy `spec-memory.md` requires. `addr` of a local is then just the alloca, and `mem2reg` removes the ones nobody addressed. `test/programs/values.flan` pins this down. - **A place is a pointer, a value is a load from it.** `(set (.pos c) …)` through a `(Ptr Cursor)` becomes a `getelementptr` on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong. Non-local exit is lowered explicitly: `return`, `some` and a failed bounds check are branches, never platform unwinding, so wasm32 needs no exception proposal. ## Sharp edges Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count, both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each says so. They stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it. - **An index converts from a narrower integer and never from a wider one.** `(nth colors current-color)` with a `u32` index works — anything above 2³¹ truncates to a negative `i32` and the unsigned bounds check rejects it. An `i64` index is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all. - **There is one top-level namespace, and `check.ml` now enforces it.** The environment's tables are per-kind — structs, unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a duplicate. `(defn item …)` beside `(defvar item …)` type checked and then died in LLVM as `redefinition of function '@flan.item'`, a message about an emitted symbol with no source location left, and two colliding *type* declarations were not caught anywhere. One pass over `Ast.declared_name` now runs before every other collection pass and rejects the second declaration of a name whatever kind either one is. `declared_name` lives in `ast.ml` because `Load` needs exactly the same set — the names an import renames — and two copies of that list would drift. - **A shift count is bounded, two different ways.** A shift by the operand's own width or more is *poison* in LLVM, not a wrong number: `(defn main [] i32 (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A literal count out of range is now rejected in `check.ml` — that is the typo case — and `emit.ml` masks a computed count to `width - 1`, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The prelude's rotate masks its own count; that is now redundant but harmless. - **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real `u64` and not an error. The cost is that a negative *decimal* literal is accepted as a `u64` too, because the reader records the value and not how it was written. Narrower unsigned types keep the strict check, which is where a typo like `300` for a `u8` actually shows up. - **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted from the folding pass's value, because a global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is therefore its own call to `in_range`; there is a regression test. - A `let` binding takes no type annotation, which is why `sand-sim` names its FNV constants instead of writing them inline. - `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean f64*: with a single body form the parser cannot tell a return type from the first expression. Only the parameter position and `(Option …)` are unambiguous. ## The reload primitive — dev loop steps 1 and 2, measured `llc` → `ld -shared` → `dlopen` → call, with no protocol and no daemon. `dune test` runs it: one function is recompiled into its own object and called inside a process that is already running, twice, with a changed body the second time. | Step | Cost | |---|---| | `Emit.redefinition` | below the timer (<0.1ms) | | `llc -O2 -filetype=obj` | 15–17ms | | `ld -shared` | 3ms | | `dlopen` + `dlsym` | **0.04ms** | **~19ms end to end**, and the load itself is free. plan.org's 16ms was measured with clang somewhere else; this is the number from this codebase. For contrast, `clang -shared` on the same IR is 50ms — the driver is again most of the cost, which is why the dev path skips it. `llc` and `clang` are both 20.1.8 here; check that before trusting the `.ll`, since the driver absorbs IR the bare tools reject. `ld -shared` rather than `clang -shared` for a second reason: a shared object is allowed undefined symbols, and that *is* the mechanism. What the new module does **not** define is the whole design: - **a global is `external`.** This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand's `grid` would reset on every reload, and "edit the code, keep the sand" is the thesis. - **every other function is a `declare`**, so a redefined `settle` calls the host's `move-grain` rather than freezing a private copy of it. - **no `main`.** This module is loaded, not started. Its string constants still come along; omitting them is an undefined `@.str.N` at link time, and it is easy to miss because a one-function module usually has none. `Emit.signature` is now the single place a function's LLVM signature is spelled, because a `define` here and a `declare` there drift the moment one of them grows a case for `Unit` or for a slice parameter. **`-rdynamic` is load-bearing.** A normal executable exports nothing: `nm -D calc-me | grep 'flan\.'` is empty, so a loaded module's `declare`s would have nothing to bind to. The test passes it through `lflags`, which keeps it a property of the dev build rather than of every build. `dlsym` on `"flan.bump"` works — a dot is legal in an ELF symbol. Two things about the test are deliberate and are what make it prove anything: both loads happen in **one process**, since two runs would pass while saying nothing about an in-process swap; and the versions are **two paths**, since `dlopen` caches by path and re-opening one would hand back the handle it already had, so the check would lie. And `helper` is `(* x 2)` in one fixture and `(* x 3)` in the other: the second body is dead text, since the module declares `helper` rather than defining it, so the expected 1024 coming back instead of 1036 is what proves the call landed on the host's copy. With the two bodies identical nothing at run time would notice a module that grew its own. String constants are emitted `private unnamed_addr`, so the module's own `@.str.N` cannot be interposed by the host's — worth knowing, because with external linkage a redefined function would silently print the *old* text and nothing would fail at link time. The fixtures each print a literal so that path is actually exercised. ### Cells — how a call site follows a redefinition Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so **a dev build routes every Flan-to-Flan call through a cell**: a mutable global holding the address of the function that is current. ``` @"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it %p = load ptr, ptr @"flan.cell.bump" ; every call site %r = call i64 %p() ``` Redefinition is then one store. A redefinition module declares the cells `external`, exactly like the globals, and exposes `flan_reload_install()` that stores its own body into its own cell — cost **below a microsecond**, which is what makes a frame-boundary swap a non-event. The cell load is emitted *after* the arguments, so a redefinition landing between two calls cannot land in the middle of one. Four things about this that are not free choices: - **`flan_reload_install` is a named function and not an ELF constructor.** A constructor runs during `dlopen`, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose. - **A redefinition's own body is `hidden`.** Default visibility in a shared object is interposable, and that applies to *taking the address* too: plain `@"flan.bump"` inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent. - **This also fixes the self-call edge**, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not. - **`-rdynamic` is what exports the cells**, so it and cells are one flag: `Build.opts.dev`, `flan build --dev`. This is the first time `opts` means something semantic rather than an optimisation level. LLVM cannot fold the indirection away — the cell is an external mutable global — and a `--dev` build of calc-me keeps 46 indirect calls at `-O2`. The acceptance table now runs `values`, `machine` and `sand-headless` as dev builds as well; the sand hash is the case that matters, since it is the one result that would notice a call reaching the wrong function. ### Names that did not exist when the process started Editing a `defvar` or a `defn` is a symbol the host exports. *Adding* one is not: there is no symbol to bind to and ELF cannot grow one. Those go through `runtime/flan_dev.c`, which is two lookups and nothing else: ``` void **flan_dev_cell(const char *name); /* a new function's cell */ void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */ ``` Both are idempotent, so the second module to mention a name gets what the first one got — which is the entire point. A new global's **declared initial value travels with it**, as a constant the runtime copies on the allocation and ignores on every call after: `calloc` alone is only right for ZII, and the "ignores afterwards" half is where "a reload must not reset the program's state" lives. Putting it in the allocation path rather than in a branch at the call site means the rule cannot be got wrong at one of them. The compiler picks per name: a name the host has is a symbol (one load at a call site), a name it lacks is a registry lookup cached at install time in a module-local slot (two loads). So the common case pays nothing for the general one. **The unit is a list of top-level forms**, not one function — `Emit.redefinition ~fns`. `C-c C-c` passes one name, `C-c C-k` passes a file's worth, one code path either way. It has to be: v3 of the fixture adds `extra` and uses it from a redefined `bump`, and splitting that into two loads would leave a module referring to storage that does not exist yet. Four rules, each of which is a silent failure if broken: - **Every lookup resolves before any body is published.** Publish first and a caller reaches a function whose slots are still null. Not race-testable, so it is asserted on the emitted `flan_reload_install`. - **`flan_dev_global` refuses a size change.** The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart. - **Nothing is ever `dlclose`d.** A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too. - **The registry never moves.** A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable. The test that separates this from a plausible wrong version is **v4**, which redefines `added` — a name v3 introduced at run time. v3's `bump` is already installed and is not rebuilt, so it picks v4 up only if its call goes through a *cell* both modules found by the same name. Had v3 cached the function's address instead, every other assertion would still pass and the transcript would read 246 instead of 432. Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32 1) to i64)` — rather than by a layout calculator in OCaml that would have to agree with LLVM's on every target. ### The agent — dev loop step 3 `vendor/agent/` is a package like any other: `agent.flan` declares three calls, `flan_agent.c` implements them, `link` asks for `-lpthread`. ``` (agent/start path) listen on a unix socket; once, at startup (agent/poll) install whatever has arrived; returns how many (agent/wait ms) the same, but waits for something first ``` The split between them is the design. `dlopen` relocates a module and takes the loader lock — milliseconds, unbounded — so it happens on the listener thread. `flan_reload_install` is one store per function and must not land while a redefined function is on the stack, so it happens on the game thread, at the top of the frame, when the program asks. The two are connected by a single-producer/single-consumer ring and two atomics; the game thread never blocks on the loader. `wait` exists for tests. A test that races the frame rate fails on a loaded machine, so `test/programs/agent.flan` waits for the reload instead of sleeping past it. It takes **two** reloads, which is the daemon's actual loop: the first introduces a global the process was never built with, the second only reads it, and the second can only answer 1007 if it found the storage the first one allocated rather than a fresh zeroed copy. One reload would not have shown that. Two details found by running it: - **stdout is line buffered**, set in `flan_rt_init`. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found. - **The reply goes out before the module is queued.** The other way round, the game thread can install and the program can exit between the two, and the answer reaches the sender as a connection reset rather than as `ok`. - **`ok` means queued, not installed.** The sender does not get to know when the swap happened; only the program knows when it is between frames. **sand.flan calls `agent/poll` at the top of its loop**, which is what step 3 was for. Verified: with sand running under Xvfb, `flan reload sand-probe.flan game-draw` and one line on the socket, and 455 consecutive frames drew from a body that did not exist when the process started. Building without `--dev` is fine — there are no cells, so a module is refused on the listener thread and the loop never notices. `flan reload ... [-o out.so] [--new name,...]` builds one module the way the daemon will. `--new` is the names the host was *not* built with; it is the one thing the command cannot work out for itself, and it is exactly what the session will track automatically. ### The session `lib/session.ml` is the program as a live thing: the declarations the running process was built from, plus every change accepted since. **Transactionality came for free and needed no machinery.** `Check.program` builds a fresh environment from a declaration list on every call, so a form that fails to check mutates nothing — the accumulated list is simply not replaced. Re-checking the whole program each evaluation costs the entire frontend, under 10ms, less than the `llc` that follows. There is a test for the case that actually matters: a typo, then a good form, in the same session. Two things the session knows that no single evaluation could: - **Which names the running process was built with.** It comes from the *checked* program, not from any accumulated AST, because `Check.program` prepends the prelude and no AST contains it. Derive it from declarations and `print-line` reads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. - **What that process's memory looks like.** Three changes are refused with a reason rather than loaded: | Change | What it would have broken | |---|---| | a function's signature | a cell is a bare `ptr`; every call site compiled before the change still passes the old arguments through it | | a global's type | the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve | | a struct's fields | the values the process is holding have the old layout | | a `defconst`'s value, **when the checker consumed it** | it is in the *shape* of the program — `(defconst rows (/ h c))` decides `grid`'s type before anything else resolves — so no store can reach it | | a `defenum` member | `:space` is erased to an `i32` literal in the caller, so it is folded there too | A `defvar`'s *initial value* is deliberately **not** in that table. Its storage holds live state the program moved past long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". Same `Tast.global` record as a `defconst`, opposite answers, told apart by `gconst`. The enum comparison runs over declarations rather than the checked program, because `Tast.program` carries no enums at all — they are erased to `i32` in the checker, which is the same fact that makes them unreloadable. Note what the checker catches on its own: change `helper`'s parameter type and the *caller* fails to type check first, loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program uses, which is exactly where the silent version lives. The fixtures carry an unused `defvar` and a C-called `defn` for that reason. A `defconst` the checker never consumed is a different matter and **can** be changed: it is only ever bytes in memory. A dev build emits every `defconst` as a mutable `global` rather than a `constant` — so LLVM cannot fold a read of it and a module can store into it — and a changed one is published at the frame boundary exactly as a new function body is. That is how sand's `colors` gets tuned live while `rows` stays refused. Release builds emit `constant` and get all the folding back; `Tast.global.gfolded` is what tells the two apart, because nothing downstream of the checker could. **A form typed into a file that is imported as a package is qualified the way the import qualified it.** `settle` in `sand-sim/sim.flan` becomes `sim/settle`, and its call to `move-grain` becomes `sim/move-grain` — through `Load`'s own `qualify_decl`, so the rule cannot drift from the one used at import time. Without this the form spliced as a brand-new unrelated name: the evaluation answered `ok`, and the running program went on calling the `sim/settle` it already had. Since sand's simulation lives in a package, the one thing worth tuning live was the one thing that silently did nothing. It is derived from the file's path and **not sent by the editor**, which is where this departs from CIDER's `ns` key: a Clojure namespace is declared in the file, but a Flan alias is chosen by whatever imported the directory and is written nowhere the editor can see. One directory imported under two aliases is refused with the reason rather than resolved to either. The accumulated list is the **post-`Load`** one, so an evaluated `(import …)` is spliced as its expansion. Otherwise re-evaluating a file that imports something appends a second import, `Load` expands it again, and the duplicate-name pass rejects it. `C-c C-k` on sand.flan's own text is the test. `flan reload ` is that path from the command line: a session over the program the process was built from, and a file of the forms that changed. Verified against a running sand under Xvfb — a one-form `game-draw` and 910 consecutive frames drew it. Two limits of that command specifically, neither of them true of sessions: it builds a fresh session from source on every invocation, so if the program file has been edited since the process launched, its idea of which names the host has and what its memory looks like describes a binary that is not running. And `Session.eval`'s `origin` defaults to ``, so an error in forms sent without one reports positions in a file that does not exist — the daemon has to pass the real buffer path, which is the same key CIDER's `eval` carries. ### The daemon — `flan dev` `flan dev ` holds one `Session`, builds the program, launches it, and listens on `.flan-dev.sock` beside the source. What it adds over `flan reload` is that the session *persists* — a `defvar` added by one evaluation is part of what the next one is checked against — and that it **owns the build**, which is what makes its layout rules describe the process that is actually running rather than a guess about it. **The protocol is s-expressions, not bencode.** nREPL was the plan and the argument for it evaporated once the client became ours too: there is no CIDER to be compatible with, `eval` is string-in/string-out with no slot for *which form, from which file*, and Emacs already has `read` and `prin1`. So it is one sexp per message — no parsing code on the editor side, and on this side the parser is the language's own reader, where `:op` is already a keyword and a payload of Flan source is already a string literal. Framing is a decimal byte count and a newline, because the payload contains newlines. An nREPL front end can sit on the same `Session` later; it should not have gated the editor. ``` (:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t) (:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0) → (:status "error" :message "…" :loc "/buf.flan:1:19") (:op "close") ``` An evaluation that declares nothing to install — a declaration the program already has, with no body and no new storage — is accepted and answered with `:note "nothing to install"` rather than by shipping an empty module. Building one anyway reports success for a change that cannot have taken effect, and costs the program a reload it did not need. `:file` is not decoration: `Session.eval`'s origin defaults to ``, so without it every error an editor shows points into a file that does not exist. Two things the daemon must not paper over, both of which would look like a successful evaluation: - **The agent socket is chosen by the daemon**, not by the program. A program's source has to name some path — sand.flan says `/tmp/flan-sand.sock` — and the daemon overrides it through `FLAN_AGENT_SOCKET` before spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it. - **Delivery is checked.** `agent/start` returning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is not `ok` becomes an error the editor sees. It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving. ### The Emacs client `emacs/flan-mode.el` derives from `prog-mode` with `lisp-mode`'s syntax table, which is most of the work: Flan is s-expressions, so sexp motion, paren matching, `beginning-of-defun` and indentation are already right. What it adds is Flan's own bracket syntax (`[` and `{` are brackets, not symbol characters — every binding list and every type is written with them), the characters a Flan name may contain (`-`, `?`, `/`, `.`), and its keywords. `emacs/flan-dev.el` is the client. There is no parser in it, which is the point of the protocol choice: `prin1` writes a request and `read` reads a reply. | | | |---|---| | `C-c C-c` | the top-level form at point, recompiled and installed | | `C-c C-k` | the whole buffer, as **one** module | | `C-x C-e` | the expression before point, evaluated *in the running program* | | `C-c C-z` / `C-c C-q` | connect (finds `.flan-dev.sock` upward) / disconnect | | `C-c C-o` | the running program's own output, in `*flan-output*` | | `C-c C-d` | what the running program currently defines | `C-c C-k` sends one module rather than a form at a time on purpose: a `defvar` and the function that uses it have to arrive in the same load, or the first refers to storage that does not exist yet. **Framing is in bytes and Emacs counts characters.** Every length goes through `string-bytes` and the process is binary, or a single non-ASCII character in a buffer puts the reply stream out of step by exactly as many bytes as the payload has of them — a bug that would look like a corrupt protocol and appear only for some users. `test/test_emacs.ml` drives the real client against a real daemon for this reason: it is not the same claim as the daemon answering correctly, and a mistake in the framing, in `beginning-of-defun` over Flan's syntax table, or in the reply reader passes `test_dev.ml` and fails here. An error comes back with a location and the client moves point to it when it is this buffer. **The program's stdout is a pipe into the daemon**, and whatever it printed since the last reply rides along with the next one into `*flan-output*`. Having it arrive *with* a reply rather than by a separate request is the point: the output an evaluation itself caused is the output anyone wants to see. Draining that pipe is a liveness requirement and not a nicety — a pipe nobody reads fills at 64K and the next write blocks the program forever — so it is read from the accept loop's `select`, not only when an editor asks, and the buffer is capped so a program printing every frame cannot grow the daemon without limit. ### `C-x C-e` — evaluating an expression A different primitive from redefining a name, and the difference is the whole design. There is no name to install a body into, so the expression is wrapped in a function with nowhere to be called from; the module says *run this once* by exporting `flan_reload_call`, and the agent calls it after the install — on the game thread, at a frame boundary, so an expression reading the program's state sees a point the program agrees is consistent. **Nothing is marshalled back, because nothing could be.** A Flan value carries no header, so no code at run time can say what it is. The compiler knows the type and renders it *there*, in the thunk, into `flan_dev_result`. That is the layout decision's bill, and it is why the printer set is small rather than universal. It does not go through stdout. Stdout belongs to the program, it is in the hot path for anything that prints, and a dev-only feature must not put a branch in it — so `flan_rt.c` is untouched and the value is read back over the agent's socket. The read is safe without a handshake because `flan_dev_result` bumps a generation counter last; the daemon waits for it to move rather than assuming the program has reached a frame boundary. The renderer is a **compile-time walk over the type**, emitting a piece at a time through `flan_dev_emit`. Piecewise because a struct is its fields with punctuation between them, and concatenating that in generated IR would need an allocator the language does not have. ``` big 18446744073709551615 col :blue (.pos b) (V {:x 1.5 :y 0}) b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]}) (slice (.tags b) 0 3) [ 0 42 0] (rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68}) sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...] ``` Details that are decisions rather than formatting: - **`u64` renders in C**, with `%llu`. The language's own `i64->bytes` is signed, so it used to refuse rather than come back as `-1` — but refusing a whole struct because one field is a `u64` is much worse, so the runtime got an entry point instead. - **Strings are quoted and escaped**, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is. - **An enum renders as `:name`**, recovered from the checker's table as a chain of comparisons, because members are erased to `i32` before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see. - **A pointer is never followed** — ``. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf. - **Three separate bounds**, easy to conflate. `depth` (4) and `span` (8) bound the *walk*, so `[100 [100 u32]]` does not become ten thousand render sites in one module. The *output* is bounded once in the runtime — `emit` truncates at 4K and `end` appends `...` — because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget. - A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element. What still refuses by name: `Map`, `Fn`, a type variable. A caveat inherited from the language, not introduced here: `3.0` renders as `3`, indistinguishable from the integer. `flan run calc-me.flan "1.5 * 2.0"` has always said `3`. An evaluation is **not** a declaration: the thunk is built against the program and never spliced into it, so `describe` does not fill up with `eval/N` for every expression ever typed. **The module is unloaded afterwards**, which is the one case where that is safe. The thunk is called directly by `flan_reload_call` rather than through a cell, and it takes no registry slot — so once it has returned, nothing points into its text and the value it produced has been copied out. It declares that with `@flan_reload_transient` and the agent `dlclose`s it. Measured: sixteen expression evaluations retain **zero** mappings, where each *redefinition* retains three, permanently and correctly — a module that publishes a body exists precisely to leave a pointer behind, and can never claim this. Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it. The test that matters is the same expression twice: the fixture increments `ticks` every frame, so two evaluations must disagree. A value computed in the compiler, or read out of a copy of the program's state, would not. ### What is left - **A REPL buffer.** `C-x C-e` echoes into the minibuffer; there is no prompt to type at and no history. - **Printers beyond the scalars.** See below — a struct, an `(Option T)` or a slice of structs still refuses by name. **Session identity is the daemon that owns the build.** A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default. ## Where build time goes `flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object cache is in**, and it is now ~110ms: | Step | Cost | |---|---| | frontend: read → parse → load → check → emit | <10ms, below the timer | | `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** | | `clang` on `flan_rt.c` | 40ms — **now cached, paid once** | | link | 20ms | Every C translation unit a build needs — the host shim and each package's shim — goes through `Build.compile_c`, which compiles to a `.o` under `$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text, the compiler (its path, size and mtime, so an upgrade invalidates without paying a `clang --version` subprocess per build), `opts.opt` and `opts.target`. The opt level has to be in there: the acceptance table builds the same programs at `-O0` and `-O2`, and an `-O2` object must not serve an `-O0` build. The object is written to a temporary name and `rename`d into place, so two concurrent builds cannot see a half-written one. Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its `shim.c` was never the cost. The cache is keyed by content, so it never needs invalidating by hand — `rm -rf` on the directory is only ever a disk-space decision. The other cheap win is still open: skip the clang driver for the `.ll` (`llc` + link directly), worth another ~40ms. It is a subset of the dev path's machinery. Check `llc`'s major version against clang's before relying on it — the emitted IR text is currently absorbed by the driver behind `-Wno-override-module`, and a version mismatch surfaces as IR parse errors. **There is still no REPL.** Nothing does redefinition, `dlopen`, or nREPL. `build` is the only way to run code. ## Next — the REPL is the priority Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place), and **the dev loop is the thesis of the project**, so it comes first. Staged so each step is runnable on its own — the failure mode is building a daemon and a protocol before knowing the reload primitive works. 1. ~~**The reload primitive, measured.**~~ **Done** — `Emit.redefinition`, `Build.shared`, `test/reload_host.c`, ~19ms. See the section above. 2. ~~**Indirection cells.**~~ **Done** — `Build.opts.dev` / `flan build --dev`, `flan_reload_install`, `runtime/flan_dev.c` for names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above. 3. ~~**The agent, in C.**~~ **Done** — `vendor/agent/`, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above. 4. ~~**The daemon**~~ and ~~**5. the Emacs client**~~ — **both done**, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the same `Session` if something else ever needs to talk to it. **One decision left to settle before step 2**, because both change codegen and are painful to retrofit: - ~~**Do cells cover globals, or only functions?**~~ **Settled by step 1: functions only.** A redefinition module declares every global `external`, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding a `defvar` to a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet. - **What is a redefinition unit — one function, or a file?** A file is much easier to make correct and is what `load-file` wants anyway; one function is what `C-c C-c` wants and is where the 16ms number comes from. Deferred until after the dev loop: 6. **wasm32.** The user installed `wasi-libc-devel` and `wasi-libc-static`; the sysroot is `/usr/wasm32-wasi` and `wasm-ld` is present. `clang --target=wasm32-wasi --sysroot=/usr/wasm32-wasi` gets past the headers and then **fails to link**: it wants `lib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a`, which no Fedora package provides (`dnf provides '*libclang_rt.builtins*wasm*'` finds nothing). It has to come from a wasi-sdk release, dropped into clang's resource directory. After that: teach `build.ml` `--sysroot`, and run the acceptance table — `sand-headless.flan` included, which is exactly why it does not import raylib — on both targets in CI. Note plan.org has the *web* build linking raylib via emscripten, which brings its own sysroot: wasi-sdk is right for the headless table, not necessarily for the eventual game build. 7. **Loose ends from milestone 4**, none of them blocking: block-scoped `defer`; package visibility, so `rl/get-color-raw` is not callable; a package importing a package; imported unions. ## Watch for The rule that caught the two misparse bugs applies unchanged: **anything that binds a name, alters control flow, or is not yet implemented must be recognised explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`, `Result`/`try`, union values, closures, quoted symbols, generics and function values *by name*, each with the milestone it belongs to; `load.ml` rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The tests assert on the reason, not just on the failure. ## Untracked on purpose `calc-me` and `sand`, the executables `flan build` drops beside their sources, are now in `.gitignore` — anchored (`/calc-me`, `/sand`) so the patterns cannot also match `sand-sim/` or anything nested. `old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root `dune` file. Its contents are also in git history at `2c232dd`.