plan.org's hot reload section now says a signature-changing redefinition should make a new internal function version with its own trampoline: new code resolves the name to it, existing callers and stored Fn values keep the old one safely, and the session warns at every tracked caller site still on the old signature. session.ml refuses the change outright, with a reason that reads like the final answer. None of the three parts exists - no function versions, no trampolines, and no record of which source location called what - so the refusal stays, because the alternative to refusing is not the new design, it is a silent argument mismatch. What changes here is only that the code and NEXT.md now say which one it is, so the next person reads it as the stopgap it is.
68 KiB
Where this is
Start here — next session
Branch dev-loop, 20 commits, working tree clean, dune test green.
The dev loop works end to end: flan dev program.flan, then C-c C-c,
C-x C-e and C-c C-r in Emacs against the running process. Conditions are
two steps in of four.
(error c) is in — §2's diverging variant, same lookup, type Never. A
handler that returns normally has not answered it, so only a transfer gets
past; with nothing transferring the program stops and names the condition.
flan_error is where the break loop goes.
The next task is the dev-build break loop, spec-conditions.md §2 — where
an unhandled error stops and talks to the daemon instead of rt_die(), and
where "a crash kills the program" finally gets fixed. The transfer it needs
exists now: §6's channel is in every signature and restart-case catches on
it, so a break loop is a place to stand while the program is stopped, not a
new way to move.
Then restarts offered in the Emacs minibuffer, which wants compute-restarts
plus two protocol ops. SBCL's restart struct carries report-function and
interactive-function for exactly that prompt and spec-conditions.md
mentions neither — worth adding before that step. find-restart and
compute-restarts are both named in §4 and neither exists yet; the runtime
already has the stack they would walk.
Still open from §3, and each refused by name today: restarts with
parameters (argument marshalling plus the runtime arity check), and
handler-case, which §"What this does not settle" leaves open as possibly a
macro over handler-bind plus a transfer. error, find-restart and
compute-restarts now refuse by name too — they used to fall through to a call
and come back as unknown name, which is the house rule's own class of bug.
Read SBCL for what restarts should mean and ignore how it moves control: it
transfers with block/return-from, which §6 rules out.
The dev loop is closed. C-c C-c in Emacs recompiles the top-level form
at point and installs it in a running program, at that program's next frame
boundary. Verified against sand: an unsaved buffer edit to game-draw, and 240
consecutive frames drew it.
Steps 1, 2 and 3 are done — see The reload primitive below. A list
of top-level forms can be recompiled and installed into a running process; call
sites compiled before they existed follow them, and a defn or defvar the
process was never built with can be added and then redefined again. That is the
whole of C-c C-c, minus an editor: sand.flan takes a redefinition over a
socket and installs it between frames.
What is left is the session — something that holds the checker environment between evaluations, tracks which names the running process was built with, and speaks a protocol an editor can talk to.
Milestone 4 is done: sand.flan builds, links raylib and runs, and its
simulation has a headless acceptance case that runs on the dune test path at
-O0 and -O2. Milestones 2 and 3 are behind it (calc-me.flan compiles and
runs; the interpreter was dropped — open decision #7, settled, see below).
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
| File | What it does |
|---|---|
lib/loc.ml |
source locations + Loc.Error, the frontend's one exception |
lib/form.ml |
reader output: Sym Kw Int Float Str Byte List Vec Map |
lib/reader.ml |
hand-written S-expression reader, no menhir/ocamllex |
lib/ast.ml |
AST: texpr, expr, place, pattern, decl |
lib/parse.ml |
forms → AST; special forms, desugaring, declarations |
lib/load.ml |
imports: a package directory → qualified declarations |
lib/types.ml |
resolved types; structural equality, Never fits anywhere |
lib/tast.ml |
the typed IR the backend consumes |
lib/check.ml |
AST → typed IR; two passes, bidirectional |
lib/session.ml |
a live program: what the process was built from, plus every change since |
lib/wire.ml |
the editor protocol: one s-expression per message, length framed |
lib/dev.ml |
flan dev: a session, the program running beside it, and a socket |
lib/prelude.ml |
printers + rand-f32, written in Flan |
lib/emit.ml |
typed IR → LLVM IR text |
lib/build.ml |
.ll + the shim + the packages' C → clang → executable |
runtime/flan_rt.c |
the host ABI: argv, stdout, exit, 4 conversions |
runtime/flan_dev.c |
dev only: the by-name registry a run-time-new name needs |
vendor/raylib/ |
the raylib package: raylib.flan, shim.c, link |
vendor/agent/ |
the dev agent: a socket, a loader thread, install at a frame boundary |
emacs/ |
flan-mode.el, flan-dev.el, flan-repl.el: the editor half of the dev loop |
sand-sim/ |
the falling-sand simulation, with no raylib in it |
bin/main.ml |
flan read | parse | check | emit | build | run | reload | dev |
test/test_flan.ml |
reader, parser and checker |
test/test_acceptance.ml |
expression/result pairs + whole programs + the traps |
test/test_reload.ml |
the reload primitive: recompile one function, load it, call it |
test/test_agent.ml |
a running program taking a redefinition over a socket |
test/test_session.ml |
what a running process cannot be told, and recovering from a typo |
test/test_dev.ml |
the daemon, driven the way an editor drives it |
test/test_repl.ml |
C-x C-e: an expression evaluated inside a running program |
test/programs/conditions.flan |
handler-bind and signal, the accumulation case |
conditions.org |
a cheatsheet for driving conditions: what works, the exact refusals, the gotchas |
conditions-play.flan |
a program to poke at them with, built to be attached to by flan dev |
test/programs/restarts.flan |
restart-case and invoke-restart: the transfer, across two frames |
test/test_emacs.ml |
the client, driven against a real daemon and a real program |
test/reload_host.c |
the C host that loads and installs two rebuilds, in one process |
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
3.5
$ flan run test/programs/sand-headless.flan
2256461126764447066
$ flan run sand.flan # a window, 120 fps, hold space
What milestone 4 added
dotimes desugars in check.ml to a Let plus a While — no new IR node.
The bound is evaluated once into a hidden slot before the loop, so a body that
changes it cannot change the trip count, and the loop variable is not
assignable, which makes the generated step its only writer.
defer is recognised in check_fn and nowhere else, because that is the
only place that knows a form is at the top level of a function body. Each one
is checked in place, then registered on the context; it emits nothing where it
stands. Function exit runs them innermost-first, and an explicit return runs
the ones registered above it — a defer written below a return has not
executed yet and must not fire. A trap runs none of them, which follows from
the bounds-check shape (noreturn then unreachable) rather than being a
separate decision.
defer inside a let, a loop or a branch is rejected, not accepted with
function scope. It would run once at function exit rather than once per
iteration, and that is the silent-wrongness class the rule below is about.
Block-scoped defer is real work and is not done.
New builtins: zeroed (takes its type from the place it is stored into),
min/max (each operand through a slot, so neither is evaluated twice),
bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a
signed type and logical on an unsigned one), and rand-f32.
rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state.
It is not libc's, because a grid hash is only a regression test if the sequence
is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed
sets the state. This is what the bitwise operators were added for.
Enums and keywords. (defenum Name [member value ...]) gives a type that
is an i32 at run time and its own type in the checker, so :space at a call
site resolves against the parameter's enum and a typo is an error there rather
than a wrong number later. A keyword means nothing where no enum is expected —
there is no keyword type to fall back on.
Why the FFI goes through a C shim
The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:
Vector2 {float,float} → declare <2 x float> @GetMousePosition()
Color {u8,u8,u8,u8} → declare void @ClearBackground(i32)
Rectangle {4 × int} → declare { i64, i64 } @mkrect()
None of those is the struct's own LLVM type. A small aggregate's calling
convention is not part of its layout — it is a per-target classification the
caller has to reproduce, and x86-64, arm64 and wasm32 classify differently.
Putting that in emit.ml is three classifiers to write and then keep correct
forever, and a mistake shows up as (.y m) returning garbage rather than as a
link error.
So vendor/raylib/shim.c has one wrapper per binding, each one flattening the
aggregates: a struct returns through an out-pointer, a struct argument is
passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates
a copy. clang classifies all of it, per target, for free. check.ml enforces
the rule — an aggregate in a declare signature is rejected with the reason —
so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
ABI, implemented twice", and flan_rt.c is the same pattern.
The price is a hand-written wrapper per raylib call. They are one-liners and mechanical enough to generate if that ever becomes the bottleneck.
raylib.flan declares each -raw entry point and wraps it in an ordinary Flan
function just below, so the surface sand.flan sees is (rl/get-mouse-position)
returning a Vector2. Verified end to end, headless: GetColor(0x11223344)
comes back as 17 34 51 68, four separate bytes — a Color is not the
little-endian reading of the packed integer, so an identity would have passed a
weaker test. That case is in the acceptance table, skipped if libraylib is
not installed.
The bindings are 18 calls: window (init-window, close-window,
window-should-close?, set-target-fps, set-trace-log-level), keyboard
(key-pressed?/down?/released?), mouse (mouse-button-pressed?/down?/
released?, get-mouse-position), get-color, and drawing (begin-drawing,
end-drawing, draw-fps, clear-background, draw-rectangle), plus the
Key, MouseButton and TraceLogLevel enums. Adding one is three lines: a
declare, an extern prototype, and a one-line wrapper.
No raylib headers are needed: shim.c declares the prototypes it uses, so the
build depends on the shared library being linkable and not on raylib-devel.
vendor/raylib/link carries -l:libraylib.so.550 because Fedora ships the
runtime library without the .so symlink.
Packages
lib/load.ml resolves (import rl "vendor:raylib") before the checker runs.
The directory is the package; vendor: is a collection, resolved by walking up
from the importing file until a directory of that name is found; a path with no
collection is relative to the importing file. Importing is a rename: every
top-level name the package declares becomes alias/name, and every use of one
— in a type, in a body, in a struct literal, in an array length — is
rewritten to match. Local bindings shadow. Nothing downstream knows a package
existed; the checker sees one flat list of declarations whose names contain a
slash.
A package may also carry the C it binds to: every .c file in the directory is
compiled into the build, and a file named link lists extra linker arguments.
This is not a module system yet. No visibility (hence rl/get-color-raw being
callable), no cycle detection, and a package cannot import another one.
sand.flan is two programs
plan.org wants sand tested twice — interactive at 120 fps, and headless over N
frames with the grid hashed, the version CI runs on native and wasm32. Those
cannot be one binary: Load collects a package's C sources and linker
arguments unconditionally, so anything importing the raylib package links
libraylib on every target regardless of what its main does, and on wasm32
that link cannot succeed.
So the simulation moved to sand-sim/, which imports nothing. sand.flan
imports it as sim/ and adds the window, the mouse and the drawing;
test/programs/sand-headless.flan imports it and adds a seed, four
deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics.
The headless case is what actually verifies milestone 4 — running the
interactive build only proves it enters its loop, because with no mouse input
the grid stays empty and paint-at, settle and move-grain never execute on
real data. Measured through the probe: 168 grains painted around row 4–8, still
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.
Three edits were made to sand.flan's own text, and they are language decisions rather than fixes:
(defconst gravity 0.05)→(defconst gravity f32 0.05). An untyped float constant isf64,velocityis[f32], and there is no implicit widening.(defvar current-color u32)→i32. It is an index intocolors, and(len colors)is ani32.- The file was split as above, so its body now says
sim/rowsand so on.
(defn main []) is unchanged — the short form, as plan.org says.
Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.
Bounds checks — done at milestone 3
at and slice emit icmp → br → cold block → call → unreachable; a
failure names the source location. Three check sites: at on [n T] (static
bound, folded by LLVM for a literal index — and a literal that is out of bounds
never reaches emit, check.ml rejects it), at on a slice or string (runtime
len), and slice (two comparisons — lo <= hi is not redundant, without it a
reversed range yields a huge unsigned length). All comparisons unsigned.
Build.opts.checks is on by default and not tied to opts.opt, which is
what lets the acceptance table run the same programs at -O0 and -O2 with
identical checks. The flag is --no-bounds-checks.
The write path is its own case: (set (at arr n) …) lowers through
place/Pindex, not through At, so a refactor that split them would break
the write check silently. The test covers both.
Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.
Why there is no interpreter
Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.
The layout, which is the whole backend design
i8..i64 / u8..u64 i8..i64 signedness lives in the ops
f32 f64 float double
bool i1
an enum i32
[T] and string { ptr, i64 } ptr+len, non-owning
[n T] [n x T] inline, a value
(Ptr T) ptr opaque pointers
(Option T) { i8, T } tag 0 None, 1 Some
a struct a literal struct, declaration order
Unit and Never {}
No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:
- Every slot is an
alloca. Reading a local is aload, assigning is astore, and astoreof an aggregate is the copyspec-memory.mdrequires.addrof a local is then just the alloca, andmem2regremoves the ones nobody addressed.test/programs/values.flanpins this down. - A place is a pointer, a value is a load from it.
(set (.pos c) …)through a(Ptr Cursor)becomes agetelementptron the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.
Non-local exit is lowered explicitly: return, some and a failed bounds
check are branches, never platform unwinding, so wasm32 needs no exception
proposal.
Sharp edges
Most of these are edges the language keeps and you should know about. Two — the top-level namespace and the shift count, both found by review after milestone 4 — were bugs that reached LLVM or ran wrong, and are fixed; each says so. They stay written down because each one is now a rule the checker enforces, and a later change could quietly drop it.
- An index converts from a narrower integer and never from a wider one.
(nth colors current-color)with au32index works — anything above 2³¹ truncates to a negativei32and the unsigned bounds check rejects it. Ani64index is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all. - There is one top-level namespace, and
check.mlnow enforces it. The environment's tables are per-kind — structs, unions, aliases, enums, functions, externs and globals each have their own — so only a function was ever checked for a duplicate.(defn item …)beside(defvar item …)type checked and then died in LLVM asredefinition of function '@flan.item', a message about an emitted symbol with no source location left, and two colliding type declarations were not caught anywhere. One pass overAst.declared_namenow runs before every other collection pass and rejects the second declaration of a name whatever kind either one is.declared_namelives inast.mlbecauseLoadneeds exactly the same set — the names an import renames — and two copies of that list would drift. - A shift count is bounded, two different ways. A shift by the operand's
own width or more is poison in LLVM, not a wrong number:
(defn main [] i32 (<< 1 32))compiled at -O2 to a bareretq, returning an undefined value. A literal count out of range is now rejected incheck.ml— that is the typo case — andemit.mlmasks a computed count towidth - 1, which is what the hardware does anyway and which LLVM folds away whenever the count is constant. The prelude's rotate masks its own count; that is now redundant but harmless. - A
u64literal is its 64-bit pattern, so0xcbf29ce484222325is a realu64and not an error. The cost is that a negative decimal literal is accepted as au64too, because the reader records the value and not how it was written. Narrower unsigned types keep the strict check, which is where a typo like300for au8actually shows up. - A folded constant skips
check.(defconst rows (/ h c))is emitted from the folding pass's value, because a global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is therefore its own call toin_range; there is a regression test. - A
letbinding takes no type annotation, which is whysand-simnames its FNV constants instead of writing them inline. (defn f [] f65 0.0)still says unknown name rather than did you mean f64: with a single body form the parser cannot tell a return type from the first expression. Only the parameter position and(Option …)are unambiguous.
The reload primitive — dev loop steps 1 and 2, measured
llc → ld -shared → dlopen → call, with no protocol and no daemon.
dune test runs it: one function is recompiled into its own object and called
inside a process that is already running, twice, with a changed body the second
time.
| Step | Cost |
|---|---|
Emit.redefinition |
below the timer (<0.1ms) |
llc -O2 -filetype=obj |
15–17ms |
ld -shared |
3ms |
dlopen + dlsym |
0.04ms |
~19ms end to end, and the load itself is free. plan.org's 16ms was measured
with clang somewhere else; this is the number from this codebase. For contrast,
clang -shared on the same IR is 50ms — the driver is again most of the cost,
which is why the dev path skips it. llc and clang are both 20.1.8 here;
check that before trusting the .ll, since the driver absorbs IR the bare
tools reject.
ld -shared rather than clang -shared for a second reason: a shared object
is allowed undefined symbols, and that is the mechanism. What the new module
does not define is the whole design:
- a global is
external. This settles the open question below in the only direction that supports the demo: a redefinition can change a function's body and can never re-initialise the program's data. Define the global and the loaded object gets a second copy — sand'sgridwould reset on every reload, and "edit the code, keep the sand" is the thesis. - every other function is a
declare, so a redefinedsettlecalls the host'smove-grainrather than freezing a private copy of it. - no
main. This module is loaded, not started.
Its string constants still come along; omitting them is an undefined @.str.N
at link time, and it is easy to miss because a one-function module usually has
none. Emit.signature is now the single place a function's LLVM signature is
spelled, because a define here and a declare there drift the moment one of
them grows a case for Unit or for a slice parameter.
flan_dev.c is compiled into every build, not only a dev one. Nothing in a
release build calls into it — the compiler only emits a registry lookup for a
name the host was not built with, which cannot arise without cells — but the
agent package's C refers to it, and a package's C sources are collected
whatever main does. Leaving it out of release builds made flan build sand.flan fail at the link with an undefined flan_dev_result_get, which
reads as a compiler bug rather than as a missing flag. The table is BSS, so the
cost is address space and not binary size; -rdynamic and the cells are still
what --dev means. test_agent.ml links the agent program both ways for this
reason.
-rdynamic is load-bearing. A normal executable exports nothing: nm -D calc-me | grep 'flan\.' is empty, so a loaded module's declares would have
nothing to bind to. The test passes it through lflags, which keeps it a
property of the dev build rather than of every build. dlsym on "flan.bump"
works — a dot is legal in an ELF symbol.
Two things about the test are deliberate and are what make it prove anything:
both loads happen in one process, since two runs would pass while saying
nothing about an in-process swap; and the versions are two paths, since
dlopen caches by path and re-opening one would hand back the handle it
already had, so the check would lie. And helper is (* x 2) in one fixture
and (* x 3) in the other: the second body is dead text, since the module
declares helper rather than defining it, so the expected 1024 coming back
instead of 1036 is what proves the call landed on the host's copy. With the two
bodies identical nothing at run time would notice a module that grew its own.
String constants are emitted private unnamed_addr, so the module's own
@.str.N cannot be interposed by the host's — worth knowing, because with
external linkage a redefined function would silently print the old text and
nothing would fail at link time. The fixtures each print a literal so that path
is actually exercised.
Cells — how a call site follows a redefinition
Loading a new body is not installing it. A call bound at link time cannot be made to notice one, so a dev build routes every Flan-to-Flan call through a cell: a mutable global holding the address of the function that is current.
@"flan.cell.bump" = global ptr @"flan.bump" ; the host defines it
%p = load ptr, ptr @"flan.cell.bump" ; every call site
%r = call i64 %p()
Redefinition is then one store. A redefinition module declares the cells
external, exactly like the globals, and exposes flan_reload_install() that
stores its own body into its own cell — cost below a microsecond, which is
what makes a frame-boundary swap a non-event.
The cell load is emitted after the arguments, so a redefinition landing between two calls cannot land in the middle of one.
Four things about this that are not free choices:
flan_reload_installis a named function and not an ELF constructor. A constructor runs duringdlopen, on whatever thread called it, mid-frame. The agent has to choose when the store happens. Loading and installing are separate on purpose.- A redefinition's own body is
hidden. Default visibility in a shared object is interposable, and that applies to taking the address too: plain@"flan.bump"inside the module resolves to the host's copy, so the installer would publish the very function it was replacing and the reload would appear to do nothing. There is a test on the linkage, because the failure is silent. - This also fixes the self-call edge, which the previous version of this section listed as a sharp edge: a redefined function calling itself goes through the cell like any other call, so it reaches the new body. v2 of the fixture recurses on purpose, and would print the old body's text if it did not.
-rdynamicis what exports the cells, so it and cells are one flag:Build.opts.dev,flan build --dev. This is the first timeoptsmeans something semantic rather than an optimisation level.
LLVM cannot fold the indirection away — the cell is an external mutable global
— and a --dev build of calc-me keeps 46 indirect calls at -O2. The
acceptance table now runs values, machine and sand-headless as dev builds
as well; the sand hash is the case that matters, since it is the one result
that would notice a call reaching the wrong function.
Names that did not exist when the process started
Editing a defvar or a defn is a symbol the host exports. Adding one is
not: there is no symbol to bind to and ELF cannot grow one. Those go through
runtime/flan_dev.c, which is two lookups and nothing else:
void **flan_dev_cell(const char *name); /* a new function's cell */
void *flan_dev_global(const char *name, uint64_t); /* a new global's storage */
Both are idempotent, so the second module to mention a name gets what the first
one got — which is the entire point. A new global's declared initial value
travels with it, as a constant the runtime copies on the allocation and
ignores on every call after: calloc alone is only right for ZII, and the
"ignores afterwards" half is where "a reload must not reset the program's
state" lives. Putting it in the allocation path rather than in a branch at the
call site means the rule cannot be got wrong at one of them. The compiler picks per name: a name the
host has is a symbol (one load at a call site), a name it lacks is a registry
lookup cached at install time in a module-local slot (two loads). So the common
case pays nothing for the general one.
The unit is a list of top-level forms, not one function — Emit.redefinition ~fns. C-c C-c passes one name, C-c C-k passes a file's worth, one code
path either way. It has to be: v3 of the fixture adds extra and uses it from
a redefined bump, and splitting that into two loads would leave a module
referring to storage that does not exist yet.
Four rules, each of which is a silent failure if broken:
- Every lookup resolves before any body is published. Publish first and a
caller reaches a function whose slots are still null. Not race-testable, so
it is asserted on the emitted
flan_reload_install. flan_dev_globalrefuses a size change. The running process has already laid that memory out; handing back the old allocation for a differently shaped type means the new body reads fields at the wrong offsets and nothing says so. This is the layout-drift rule's first enforcement point. Retyping a var needs a restart.- Nothing is ever
dlclosed. A cell holds an address inside a module's text; unloading it leaves every call site pointing at unmapped memory. That is a constraint on the agent too. - The registry never moves. A module holds a cell's address for as long as it is loaded, so the table is fixed capacity with a loud failure rather than growable.
The test that separates this from a plausible wrong version is v4, which
redefines added — a name v3 introduced at run time. v3's bump is already
installed and is not rebuilt, so it picks v4 up only if its call goes through a
cell both modules found by the same name. Had v3 cached the function's
address instead, every other assertion would still pass and the transcript
would read 246 instead of 432.
Sizes are spelled LLVM's way — `ptrtoint (ptr getelementptr (T, ptr null, i32
- to i64)` — rather than by a layout calculator in OCaml that would have to agree with LLVM's on every target.
The agent — dev loop step 3
vendor/agent/ is a package like any other: agent.flan declares three calls,
flan_agent.c implements them, link asks for -lpthread.
(agent/start path) listen on a unix socket; once, at startup
(agent/poll) install whatever has arrived; returns how many
(agent/wait ms) the same, but waits for something first
The split between them is the design. dlopen relocates a module and takes the
loader lock — milliseconds, unbounded — so it happens on the listener thread.
flan_reload_install is one store per function and must not land while a
redefined function is on the stack, so it happens on the game thread, at the
top of the frame, when the program asks. The two are connected by a
single-producer/single-consumer ring and two atomics; the game thread never
blocks on the loader.
wait exists for tests. A test that races the frame rate fails on a loaded
machine, so test/programs/agent.flan waits for the reload instead of sleeping
past it. It takes two reloads, which is the daemon's actual loop: the first
introduces a global the process was never built with, the second only reads it,
and the second can only answer 1007 if it found the storage the first one
allocated rather than a fresh zeroed copy. One reload would not have shown
that.
Two details found by running it:
- stdout is line buffered, set in
flan_rt_init. The C default when stdout is a file or a pipe is a 4K block, so a program running with a REPL attached shows nothing until it exits — and a test driving one cannot see its progress at all, which is how this was found. - The reply goes out before the module is queued. The other way round, the
game thread can install and the program can exit between the two, and the
answer reaches the sender as a connection reset rather than as
ok. okmeans queued, not installed. The sender does not get to know when the swap happened; only the program knows when it is between frames.
sand.flan calls agent/poll at the top of its loop, which is what step 3
was for. Verified: with sand running under Xvfb, flan reload sand-probe.flan game-draw and one line on the socket, and 455 consecutive frames drew from a
body that did not exist when the process started. Building without --dev is
fine — there are no cells, so a module is refused on the listener thread and
the loop never notices.
flan reload <file.flan> <fn>... [-o out.so] [--new name,...] builds one
module the way the daemon will. --new is the names the host was not built
with; it is the one thing the command cannot work out for itself, and it is
exactly what the session will track automatically.
The session
lib/session.ml is the program as a live thing: the declarations the running
process was built from, plus every change accepted since.
Transactionality came for free and needed no machinery. Check.program
builds a fresh environment from a declaration list on every call, so a form
that fails to check mutates nothing — the accumulated list is simply not
replaced. Re-checking the whole program each evaluation costs the entire
frontend, under 10ms, less than the llc that follows. There is a test for the
case that actually matters: a typo, then a good form, in the same session.
Two things the session knows that no single evaluation could:
-
Which names the running process was built with. It comes from the checked program, not from any accumulated AST, because
Check.programprepends the prelude and no AST contains it. Derive it from declarations andprint-linereads as new, gets a registry cell nobody publishes, and the first call jumps to null with no diagnostic. -
What that process's memory looks like. Three changes are refused with a reason rather than loaded:
Change What it would have broken a function's signature a cell is a bare ptr; every call site compiled before the change still passes the old arguments through it — and this is now a stopgap, see belowa global's type the storage exists and has a shape — reuse reads at the wrong offsets, replacement discards the state the reload exists to preserve a struct's fields the values the process is holding have the old layout a defconst's value, when the checker consumed itit is in the shape of the program — (defconst rows (/ h c))decidesgrid's type before anything else resolves — so no store can reach ita defenummember:spaceis erased to ani32literal in the caller, so it is folded there tooThe signature row is the one the plan has moved past. plan.org now says a signature-changing redefinition should make a new internal function version with its own trampoline: newly compiled code resolves the name to it, while existing callers and stored
Fnvalues keep the old version and stay safe, and the session warns at every tracked caller site still targeting the old signature — recompiling one either retargets it or gives an ordinary type error. Open decision #6 records it the same way. None of the three parts exists: there are no function versions, no trampolines (a cell holds a body address today), and no record of which source locations called what. So the refusal stays, because the alternative to refusing is not the new design, it is a silent argument mismatch. It is a stopgap and the message should not be read as the final answer.A
defvar's initial value is deliberately not in that table. Its storage holds live state the program moved past long ago, and refusing to change the initialiser would be refusing "edit the code, keep the sand". SameTast.globalrecord as adefconst, opposite answers, told apart bygconst. The enum comparison runs over declarations rather than the checked program, becauseTast.programcarries no enums at all — they are erased toi32in the checker, which is the same fact that makes them unreloadable.Note what the checker catches on its own: change
helper's parameter type and the caller fails to type check first, loudly. The session's rules only get a turn on a change the checker accepts — one to a name nothing else in the program uses, which is exactly where the silent version lives. The fixtures carry an unuseddefvarand a C-calleddefnfor that reason.
A defconst the checker never consumed is a different matter and can be
changed: it is only ever bytes in memory. A dev build emits every defconst as
a mutable global rather than a constant — so LLVM cannot fold a read of it
and a module can store into it — and a changed one is published at the frame
boundary exactly as a new function body is. That is how sand's colors gets
tuned live while rows stays refused. Release builds emit constant and get
all the folding back; Tast.global.gfolded is what tells the two apart, because
nothing downstream of the checker could.
A form typed into a file that is imported as a package is qualified the way
the import qualified it. settle in sand-sim/sim.flan becomes sim/settle,
and its call to move-grain becomes sim/move-grain — through Load's own
qualify_decl, so the rule cannot drift from the one used at import time.
Without this the form spliced as a brand-new unrelated name: the evaluation
answered ok, and the running program went on calling the sim/settle it
already had. Since sand's simulation lives in a package, the one thing worth
tuning live was the one thing that silently did nothing.
It is derived from the file's path and not sent by the editor, which is
where this departs from CIDER's ns key: a Clojure namespace is declared in
the file, but a Flan alias is chosen by whatever imported the directory and is
written nowhere the editor can see. One directory imported under two aliases is
refused with the reason rather than resolved to either.
The accumulated list is the post-Load one, so an evaluated (import …)
is spliced as its expansion. Otherwise re-evaluating a file that imports
something appends a second import, Load expands it again, and the
duplicate-name pass rejects it. C-c C-k on sand.flan's own text is the test.
flan reload <program.flan> <forms.flan> is that path from the command line: a
session over the program the process was built from, and a file of the forms
that changed. Verified against a running sand under Xvfb — a one-form
game-draw and 910 consecutive frames drew it.
Two limits of that command specifically, neither of them true of sessions:
it builds a fresh session from source on every invocation, so if the program
file has been edited since the process launched, its idea of which names the
host has and what its memory looks like describes a binary that is not running.
And Session.eval's origin defaults to <eval>, so an error in forms sent
without one reports positions in a file that does not exist — the daemon has to
pass the real buffer path, which is the same key CIDER's eval carries.
The daemon — flan dev
flan dev <program.flan> holds one Session, builds the program, launches it,
and listens on .flan-dev.sock beside the source. What it adds over flan reload is that the session persists — a defvar added by one evaluation is
part of what the next one is checked against — and that it owns the build,
which is what makes its layout rules describe the process that is actually
running rather than a guess about it.
The protocol is s-expressions, not bencode. nREPL was the plan and the
argument for it evaporated once the client became ours too: there is no CIDER
to be compatible with, eval is string-in/string-out with no slot for which
form, from which file, and Emacs already has read and prin1. So it is one
sexp per message — no parsing code on the editor side, and on this side the
parser is the language's own reader, where :op is already a keyword and a
payload of Flan source is already a string literal. Framing is a decimal byte
count and a newline, because the payload contains newlines. An nREPL front end
can sit on the same Session later; it should not have gated the editor.
(:op "describe") → (:status "ok" :fns (…) :globals (…) :alive t)
(:op "eval" :code "…" :file "/buf.flan") → (:status "ok" :names (…) :fns (…) :ms 19.0)
→ (:status "error" :message "…" :loc "/buf.flan:1:19")
(:op "close")
An evaluation that declares nothing to install — a declaration the program
already has, with no body and no new storage — is accepted and answered with
:note "nothing to install" rather than by shipping an empty module. Building
one anyway reports success for a change that cannot have taken effect, and
costs the program a reload it did not need.
:file is not decoration: Session.eval's origin defaults to <eval>, so
without it every error an editor shows points into a file that does not exist.
Two things the daemon must not paper over, both of which would look like a successful evaluation:
- The agent socket is chosen by the daemon, not by the program. A program's
source has to name some path — sand.flan says
/tmp/flan-sand.sock— and the daemon overrides it throughFLAN_AGENT_SOCKETbefore spawning. Guessing instead fails silently: the module compiles, is built, and nobody receives it. - Delivery is checked.
agent/startreturning 0 means a socket was bound, not that anyone connected. A failed connect or a reply that is notokbecomes an error the editor sees.
It waits for the program to bind before accepting an evaluation — one arriving first would fail for a reason that reads like a compiler bug — and it accepts with a timeout so that a program which has exited takes the daemon with it rather than leaving an editor waiting on a socket nobody is serving.
The Emacs client
emacs/flan-mode.el derives from prog-mode with lisp-mode's syntax table,
which is most of the work: Flan is s-expressions, so sexp motion, paren
matching, beginning-of-defun and indentation are already right. What it adds
is Flan's own bracket syntax ([ and { are brackets, not symbol characters —
every binding list and every type is written with them), the characters a Flan
name may contain (-, ?, /, .), and its keywords.
emacs/flan-dev.el is the client. There is no parser in it, which is the point
of the protocol choice: prin1 writes a request and read reads a reply.
C-c C-c |
the top-level form at point, recompiled and installed |
C-c C-k |
the whole buffer, as one module |
C-x C-e |
the expression before point, evaluated in the running program |
C-c C-z / C-c C-q |
connect (finds .flan-dev.sock upward) / disconnect |
C-c C-o |
the running program's own output, in *flan-output* |
C-c C-r |
a prompt on the running program (*flan-repl*) |
C-c C-d |
what the running program currently defines |
C-c C-k sends one module rather than a form at a time on purpose: a defvar
and the function that uses it have to arrive in the same load, or the first
refers to storage that does not exist yet.
Framing is in bytes and Emacs counts characters. Every length goes through
string-bytes and the process is binary, or a single non-ASCII character in a
buffer puts the reply stream out of step by exactly as many bytes as the
payload has of them — a bug that would look like a corrupt protocol and appear
only for some users. test/test_emacs.ml drives the real client against a real
daemon for this reason: it is not the same claim as the daemon answering
correctly, and a mistake in the framing, in beginning-of-defun over Flan's
syntax table, or in the reply reader passes test_dev.ml and fails here.
An error comes back with a location and the client moves point to it when it is this buffer.
The program's stdout is a pipe into the daemon, and whatever it printed
since the last reply rides along with the next one into *flan-output*. Having
it arrive with a reply rather than by a separate request is the point: the
output an evaluation itself caused is the output anyone wants to see. Draining
that pipe is a liveness requirement and not a nicety — a pipe nobody reads
fills at 64K and the next write blocks the program forever — so it is read from
the accept loop's select, not only when an editor asks, and the buffer is
capped so a program printing every frame cannot grow the daemon without
limit.
C-x C-e — evaluating an expression
A different primitive from redefining a name, and the difference is the whole
design. There is no name to install a body into, so the expression is wrapped
in a function with nowhere to be called from; the module says run this once
by exporting flan_reload_call, and the agent calls it after the install — on
the game thread, at a frame boundary, so an expression reading the program's
state sees a point the program agrees is consistent.
Nothing is marshalled back, because nothing could be. A Flan value carries
no header, so no code at run time can say what it is. The compiler knows the
type and renders it there, in the thunk, into flan_dev_result. That is the
layout decision's bill, and it is why the printer set is small rather than
universal.
It does not go through stdout. Stdout belongs to the program, it is in the hot
path for anything that prints, and a dev-only feature must not put a branch in
it — so flan_rt.c is untouched and the value is read back over the agent's
socket. The read is safe without a handshake because flan_dev_result bumps a
generation counter last; the daemon waits for it to move rather than assuming
the program has reached a frame boundary.
This renderer is most of println, which is worth knowing before anyone
schedules it. plan.org describes a compiler-provided, type-directed intrinsic
that selects or emits a structural printer per concrete instantiation, prints
structs, fixed arrays and options structurally, prints a Ptr as its address
rather than following it, and bounds depth and length. That is a description of
what Session already does for C-x C-e — same walk, same refusals, same
three bounds — aimed at flan_dev_emit and the wire instead of at stdout. What
println needs on top is a stdout sink, a builtin that takes its printer from
the argument's type, and the any/Error dynamic cases, which have no
compile-time type to walk. Not the walk itself.
The renderer is a compile-time walk over the type, emitting a piece at a
time through flan_dev_emit. Piecewise because a struct is its fields with
punctuation between them, and concatenating that in generated IR would need an
allocator the language does not have.
big 18446744073709551615
col :blue
(.pos b) (V {:x 1.5 :y 0})
b (Blob {:id 7 :name "sandy \"quoted\"" :pos (V {:x 1.5 :y 0}) :tags [ 0 42 0]})
(slice (.tags b) 0 3) [ 0 42 0]
(rl/get-color 0x11223344) (rl/Color {:r 17 :g 34 :b 51 :a 68})
sim/grid [ [ 0 0 0 0 0 0 0 0 ...] [ 0 ... ] ...]
Details that are decisions rather than formatting:
u64renders in C, with%llu. The language's owni64->bytesis signed, so it used to refuse rather than come back as-1— but refusing a whole struct because one field is au64is much worse, so the runtime got an entry point instead.- Strings are quoted and escaped, also in C. Unescaped content does not round-trip and reads as a framing bug rather than as the value it is.
- An enum renders as
:name, recovered from the checker's table as a chain of comparisons, because members are erased toi32before the backend sees them. A value outside the declared members falls through to its number, which is exactly what you would want to see. - A pointer is never followed —
<ptr>. It is the only thing that could make the walk cycle, and dereferencing one a REPL was handed is not a safe thing to do on someone's behalf. - Three separate bounds, easy to conflate.
depth(4) andspan(8) bound the walk, so[100 [100 u32]]does not become ten thousand render sites in one module. The output is bounded once in the runtime —emittruncates at 4K andendappends...— because a slice renders through a loop the compiler cannot bound, and one place enforcing it means no renderer carries a budget. - A slice is the one case needing a runtime loop, and the slice goes into a slot first so the expression it came from is not evaluated once per element.
What still refuses by name: Map, Fn, a type variable.
A caveat inherited from the language, not introduced here: 3.0 renders as
3, indistinguishable from the integer. flan run calc-me.flan "1.5 * 2.0"
has always said 3.
An evaluation is not a declaration: the thunk is built against the program
and never spliced into it, so describe does not fill up with eval/N for
every expression ever typed.
The module is unloaded afterwards, which is the one case where that is
safe. The thunk is called directly by flan_reload_call rather than through a
cell, and it takes no registry slot — so once it has returned, nothing points
into its text and the value it produced has been copied out. It declares that
with @flan_reload_transient and the agent dlcloses it. Measured: sixteen
expression evaluations retain zero mappings, where each redefinition
retains three, permanently and correctly — a module that publishes a body
exists precisely to leave a pointer behind, and can never claim this.
Skipping the registry matters for more than tidiness: the table holds 4096 names and an expression evaluated in a loop would exhaust it.
The test that matters is the same expression twice: the fixture increments
ticks every frame, so two evaluations must disagree. A value computed in the
compiler, or read out of a copy of the program's state, would not.
The REPL buffer
flan-repl.el is a comint-mode buffer whose every line goes through the same
eval-expr request C-x C-e uses. No new protocol and no compiler support.
Deriving from comint rather than hand-rolling a prompt is the same call as
deriving flan-mode from lisp-mode: history, the input ring and kill/yank
already exist. There is no subprocess behind it — the "process" is a stub
comint needs in order to have a prompt at all.
Three things about it that are decisions:
- It is program-scoped. A name typed at the prompt resolves against the
running program's top-level namespace, so in sand you write
sim/settleand notsettle. A buffer visiting a package's own file gets the alias applied for it because the file says which package it belongs to; a prompt has no file and nothing to derive one from. - RET on a half-typed form opens a line instead of sending it. Balance is checked with the Flan syntax table, so a paren inside a string does not count.
- A value and the program's output are different things and arrive by
different routes. The value is the result of the request and appears at the
prompt; anything the program printed while evaluating it rides along on the
same reply and goes to
*flan-output*. Showing them in one place would be convenient and wrong, so there is a test for the separation.
That test is what caught a real bug: the renderer's Unit case emitted ()
without evaluating the expression, so (print-line "x") — the most ordinary
thing anyone types at a prompt — answered () while nothing happened. A Unit
expression is almost always a call made for its effect, and is now evaluated
and then reported.
Conditions — step 1: handler-bind and signal
spec-conditions.md §1 and §2, and nothing else yet. They are worth having on
their own because neither alters control flow: signal returns Unit
whatever it finds, a handler that returns normally leaves the signalling
function to carry on, and with nothing matching it is a no-op. So none of the
transfer machinery §6 describes exists yet, and no signature changed.
(handler-bind [(AssetMissing [c] (set seen (+ seen (i64 (.id c)))))]
(load-all))
The runtime is a linked list: establishing a handler is two stores and a push
onto a frame allocated on the establishing function's own stack, and signal
with an empty stack is a null check — which is what §2 asks for. Popping is by
frame rather than by count, so restoring what this one displaced is right even
if something below it left the stack out of step.
Three decisions worth keeping:
- A condition's type is a hash of its name, not an index. An index would shift the moment a struct were added, and every handler a running program had already pushed would then match the wrong type. FNV-1a over the name.
- The condition crosses as a pointer, because a handler runs while the
signalling frame is still alive and there is nothing to copy. What the clause
binds is the condition itself, though — the pointer is a hidden parameter
and the name is a slot loaded from it, so a handler passing
cto something expecting the struct is not handed an address instead. - A clause is lifted into a function of its own. A handler runs from wherever the signal was, so it cannot be a branch in the function that wrote it.
- A pushed handler frame holds the clause's body address, not a cell. This
is a deliberate divergence from plan.org's rule that a top-level function
value is a stable trampoline over the cell and never the address of a
particular body. A handler frame is not a
Fnvalue — nothing in the language can name it — and it is live only for the duration of thehandler-bindbody, so a reload landing while it is on the stack finds the clause it pushed still valid, which is exactly the "old code is never unloaded" guarantee. The consequence to know: a handler already on the stack does not observe a redefinition of its own clause; the next entry to thehandler-bindpushes the new one. WhenFnvalues arrive, this is the one place that stores a body address on purpose and must not be swept up with them.
Which gives the two refusals, both by the house rule rather than by accident:
-
A handler cannot see the establishing function's locals. That is a closure with an explicit environment, so a reference to one is refused for that reason rather than reported as an unknown name. Globals and the condition are in scope, which is what the accumulation case needs.
What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not outlive the function that established it, so this is spec-memory.md's case 2 — a non-escaping
fncapturing by value into a stack environment — and not the escaping closure that plan.org's open decision #5 defers until a concrete use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the biggest usability limit in conditions is not behind the thing that was just deferred. -
returninside ahandler-bindbody is refused. The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape asdeferinside a block.
Conditions — step 2: restart-case and invoke-restart
spec-conditions.md §3 to §6: the transfer. A handler runs where the signal
was, decides, and control resumes at a restart-case further out.
(defn fetch [n i32] i32
(restart-case (middle n) ; its value if nothing transfers
(use-placeholder [] -1)
(retry [] 7)))
(handler-bind [(AssetMissing [c] (invoke-restart 'use-placeholder))]
(fetch 2)) ; -1
The channel is an out-parameter, as §6 now says: one ptr appended to
every Flan signature, written by an invoke-restart and checked after every
call. The return type stays what the source says, so the disassembly is the
release one plus a guard, and one pointer threads down the whole chain — a
callee writes the target into its caller's slot and each frame only has to
check and return early, which reuses the existing return path and with it
§5's defers. Emit.signature was already the one place a signature is spelled,
which is what made this a three-line change rather than a hunt.
Every function is transfer-transparent, release included — and that is the ABI, not a stopgap. A cell holds a bare pointer, so the honest answer to "what can this call?" is "anything"; the same bargain as the indirect call. §6 and plan.org both now say the later optimisation may stop a function checking the channel, or pass the pointer straight through, but may not drop the parameter — a signature that depended on an analysis could not be reloaded into. Uniform also means redefinition acquires no new refusal class.
The transfer target is the restart frame's own address, not a clause id.
This is a correction to what the previous note settled. A static id has to be
unique against every module a running program may later load, and a hash is
only probably unique — two restart-cases colliding means the inner one
silently catches a transfer aimed at the outer. The frame is an alloca in the
function that offers it, so its address is exact, and it also says which
clause, which is how clause ids disappeared entirely. §6 says "transferring to
frame N" and this is closer to it than the number was. Re-entering a
restart-case then works with nothing extra: each activation allocates its
own frames, and §4's "innermost offering the name" is just the order of the
walk.
Cleanup happens in landing blocks, one per region. A guard branches to the innermost open one, which pops whatever frames it established and either catches the transfer or forwards it outward:
- a
restart-case's pops its restart frames, compares the target against its own, and either runs that clause or puts the target back and goes on out; - a
handler-bind's pops its handler frames and goes on out — which is the path a transfer out of a handled body takes, and without it the handler stack would be left pointing into a frame that has gone; - the function's own runs its defers (§5) and returns early.
errdeferdoes not run and never could:try/Resultis still refused by name.
A single function-wide unwind block would have been wrong for the first two:
a call inside a restart-case body would jump straight past the very form that
was supposed to catch it.
The channel is cleared before any cleanup runs and put back after. A defer makes ordinary calls and each one is guarded; with the channel still set the first of them would branch straight back into the landing block it came from. Same reason the clause body starts with it null.
flan_signal takes the channel and passes it to each handler, and stops
walking once one has written to it. That makes the one C frame every handler is
reached through transparent to a transfer — it has to be, or §6's "a transfer
cannot cross a foreign frame" would make restart-case useless. It is also the
only such frame: extern is Flan-to-C only and there are no function values
yet, so nothing can call back into Flan across one.
Scope, each piece refused by name with its reason and a test on the reason:
- restarts take no parameters. That covers §1's own
load-textureexample and skips argument marshalling and §3's runtime arity check. returninside arestart-casebody, exactly as insidehandler-bind: a bareretskips the pops.- one
restart-caseoffering a name twice — §4 finds the first frame offering it, and two in one frame makes that a choice nothing in the source shows. invoke-restartinside adefer. A defer is the cleanup a transfer runs on its way out, so a transfer starting there leaves the function's defers half run with two targets and no way to choose. The lexical case is the checker's; a defer that reaches one through a call is trapped at run time byflan_transfer_fail, because nothing static could see it.- no restart of that name is active: a runtime error at the invoke site, named and located, rather than an unwind past everything. There is nowhere to resume, so there is nothing else to do.
(error c), §2. The same walk as signal, and the difference is entirely
what happens when the walk ends: signal returns Unit and the signalling
function carries on, error has type Never and stops. So only a transfer
gets past it, which is why emit puts a guard after the call and then
unreachable — and why flan_error cannot be marked noreturn, since it does
return, on exactly one path. Being Never is also what lets it stand as a
restart-case body's fall-through, which is the shape §1's load-texture
example needs. test/programs/error.flan is the unhandled case: it cannot be
an outputs row, because it does not exit 0.
flan_transfer_fail covers the ordinary return path as well as the unwind one:
a defer that reaches an invoke-restart through a call traps either way, and
the message names the rule rather than the path, since the rule is the same.
A lifted handler clause is named after the function it came out of —
handler/step/0/Missing — and is emitted by a redefinition module alongside
the body it belongs to, hidden, for the same interposition reason the body is.
This was a hole step 1 left: the name used to be numbered by position in the
whole program's lifted list, so it was neither stable nor attributable, and a
redefinition of a function containing a handler-bind failed in llc with an
undefined value. A clause is reached by address from its parent's body and from
nowhere else, so it takes no cell and no registry slot.
test/test_dev.ml drives that path: a third evaluation redefines step to a
restart-case whose frame is an alloca in the newly loaded module, whose
guarded call goes through the host's cell, and whose transfer starts in a
handler and crosses probe, which the host was compiled with. Those three do
not meet anywhere else.
Two things found by writing it:
{ ctx with in_handler = true }was a latent bug.ctx.slotsandctx.slot_tysare mutable, so a copy allocates the body's slots into a record the function never sees again and the indices collide. It was harmless only because nohandler-bindbody in the tests had aletin it. The flags are set onctxand restored now.test/reload_host.chad to learn the parameter. It callsflan.outerthrough an__asm__label, which does not fail at link time when the prototype is a parameter short — it reads a garbage pointer as the channel and dies somewhere else entirely.
test/programs/restarts.flan runs in the acceptance table at -O2, at -O0
and as a dev build. -O0 is not redundant here: the guard after every call is
control flow the optimiser would otherwise launder, and the dev build is where
each of those calls goes through a cell.
What is left
- Editor comforts: completion, eldoc, jump-to-definition, error overlays.
Session identity is the daemon that owns the build. A session's struct layouts and global types have to describe the memory of the process it is talking to, which is only guaranteed if it is the session that compiled the running binary. Attaching to a process someone else built is not a thing to support by default.
Where build time goes
flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object
cache is in, and it is now ~110ms:
| Step | Cost |
|---|---|
| frontend: read → parse → load → check → emit | <10ms, below the timer |
clang on the .ll |
60ms — llc does the same codegen in 20ms |
clang on flan_rt.c |
40ms — now cached, paid once |
| link | 20ms |
Every C translation unit a build needs — the host shim and each package's shim
— goes through Build.compile_c, which compiles to a .o under
$TMPDIR/flan-objcache and reuses it. The key is a digest of the source text,
the compiler (its path, size and mtime, so an upgrade invalidates without
paying a clang --version subprocess per build), opts.opt and opts.target.
The opt level has to be in there: the acceptance table builds the same programs
at -O0 and -O2, and an -O2 object must not serve an -O0 build. The
object is written to a temporary name and renamed into place, so two
concurrent builds cannot see a half-written one.
Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is
mostly linking libraylib and its shim.c was never the cost. The cache is
keyed by content, so it never needs invalidating by hand — rm -rf on the
directory is only ever a disk-space decision.
The other cheap win is still open: skip the clang driver for the .ll (llc +
link directly), worth another ~40ms. It is a subset of the dev path's
machinery. Check llc's major version against clang's before relying on it —
the emitted IR text is currently absorbed by the driver behind
-Wno-override-module, and a version mismatch surfaces as IR parse errors.
There is still no REPL. Nothing does redefinition, dlopen, or nREPL.
build is the only way to run code.
Next — the REPL is the priority
Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place), and the dev loop is the thesis of the project, so it comes first. Staged so each step is runnable on its own — the failure mode is building a daemon and a protocol before knowing the reload primitive works.
The reload primitive, measured.Done —Emit.redefinition,Build.shared,test/reload_host.c, ~19ms. See the section above.Indirection cells.Done —Build.opts.dev/flan build --dev,flan_reload_install,runtime/flan_dev.cfor names introduced at run time, and a fixture where an untouched call site follows the swap and a run-time-added function is itself redefined. See the section above.The agent, in C.Done —vendor/agent/, a listener thread that loads and a game thread that installs, and sand.flan polling at the top of its frame. See the section above.The daemonand5. the Emacs client— both done, and the protocol is s-expressions rather than nREPL's bencode; see the two sections above for why that changed. An nREPL front end can sit on the sameSessionif something else ever needs to talk to it.
One decision left to settle before step 2, because both change codegen and are painful to retrofit:
Do cells cover globals, or only functions?Settled by step 1: functions only. A redefinition module declares every globalexternal, so globals live in the host and survive a reload — which is what "edit the code, keep the sand" needs. The consequence to watch is the other half: adding adefvarto a file cannot take effect on reload, and changing one's type is a silent mismatch against storage the host already laid out. Nothing detects that yet.- What is a redefinition unit — one function, or a file? A file is much
easier to make correct and is what
load-filewants anyway; one function is whatC-c C-cwants and is where the 16ms number comes from.
Deferred until after the dev loop:
- wasm32. The user installed
wasi-libc-develandwasi-libc-static; the sysroot is/usr/wasm32-wasiandwasm-ldis present.clang --target=wasm32-wasi --sysroot=/usr/wasm32-wasigets past the headers and then fails to link: it wantslib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a, which no Fedora package provides (dnf provides '*libclang_rt.builtins*wasm*'finds nothing). It has to come from a wasi-sdk release, dropped into clang's resource directory. After that: teachbuild.ml--sysroot, and run the acceptance table —sand-headless.flanincluded, which is exactly why it does not import raylib — on both targets in CI. Note plan.org has the web build linking raylib via emscripten, which brings its own sysroot: wasi-sdk is right for the headless table, not necessarily for the eventual game build. - Loose ends from milestone 4, none of them blocking: block-scoped
defer; package visibility, sorl/get-color-rawis not callable; a package importing a package; imported unions.
Watch for
The rule that caught the two misparse bugs applies unchanged: anything that
binds a name, alters control flow, or is not yet implemented must be recognised
explicitly and rejected if unsupported. check.ml rejects Vec, Map,
Result/try, union values, closures, quoted symbols, generics and function
values by name, each with the milestone it belongs to; load.ml rejects the
package shapes it does not handle; and the FFI boundary rejects an aggregate.
The tests assert on the reason, not just on the failure.
Untracked on purpose
calc-me and sand, the executables flan build drops beside their sources,
are now in .gitignore — anchored (/calc-me, /sand) so the patterns cannot
also match sand-sim/ or anything nested.
old-ocaml/ — the pre-rewrite menhir/ocamllex frontend, kept as reference and
excluded from the build by the root dune file. Its contents are also in git
history at 2c232dd.