flan/NEXT.md

Where this is

Milestone 4 is done: sand.flan builds, links raylib and runs, and its simulation has a headless acceptance case that runs on the dune test path at -O0 and -O2. Milestones 2 and 3 are behind it (calc-me.flan compiles and runs; the interpreter was dropped — open decision #7, settled, see below).

reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅

File	What it does
lib/loc.ml	source locations + Loc.Error, the frontend's one exception
lib/form.ml	reader output: Sym Kw Int Float Str Byte List Vec Map
lib/reader.ml	hand-written S-expression reader, no menhir/ocamllex
lib/ast.ml	AST: texpr, expr, place, pattern, decl
lib/parse.ml	forms → AST; special forms, desugaring, declarations
lib/load.ml	imports: a package directory → qualified declarations
lib/types.ml	resolved types; structural equality, Never fits anywhere
lib/tast.ml	the typed IR the backend consumes
lib/check.ml	AST → typed IR; two passes, bidirectional
lib/prelude.ml	printers + rand-f32, written in Flan
lib/emit.ml	typed IR → LLVM IR text
lib/build.ml	.ll + the shim + the packages' C → clang → executable
runtime/flan_rt.c	the host ABI: argv, stdout, exit, 4 conversions
vendor/raylib/	the raylib package: raylib.flan, shim.c, link
sand-sim/	the falling-sand simulation, with no raylib in it
bin/main.ml	flan read | parse | check | emit | build | run
test/test_flan.ml	reader, parser and checker
test/test_acceptance.ml	expression/result pairs + whole programs + the traps

$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
3.5
$ flan run test/programs/sand-headless.flan
2256461126764447066
$ flan run sand.flan                      # a window, 120 fps, hold space

What milestone 4 added

dotimes desugars in check.ml to a Let plus a While — no new IR node. The bound is evaluated once into a hidden slot before the loop, so a body that changes it cannot change the trip count, and the loop variable is not assignable, which makes the generated step its only writer.

defer is recognised in check_fn and nowhere else, because that is the only place that knows a form is at the top level of a function body. Each one is checked in place, then registered on the context; it emits nothing where it stands. Function exit runs them innermost-first, and an explicit return runs the ones registered above it — a defer written below a return has not executed yet and must not fire. A trap runs none of them, which follows from the bounds-check shape (noreturn then unreachable) rather than being a separate decision.

defer inside a let, a loop or a branch is rejected, not accepted with function scope. It would run once at function exit rather than once per iteration, and that is the silent-wrongness class the rule below is about. Block-scoped defer is real work and is not done.

New builtins: zeroed (takes its type from the place it is stored into), min/max (each operand through a slot, so neither is evaluated twice), bit-and/bit-or/bit-xor/<</>> (integers only; >> is arithmetic on a signed type and logical on an unsigned one), and rand-f32.

rand-f32 is in the prelude, in Flan — PCG-XSH-RR 32 over a u64 state. It is not libc's, because a grid hash is only a regression test if the sequence is byte-identical on native and wasm32 (plan.org, RNG is ours). rand-seed sets the state. This is what the bitwise operators were added for.

Enums and keywords. (defenum Name [member value ...]) gives a type that is an i32 at run time and its own type in the checker, so :space at a call site resolves against the parameter's enum and a typo is an error there rather than a wrong number later. A keyword means nothing where no enum is expected — there is no keyword type to fall back on.

Why the FFI goes through a C shim

The decision that shapes the whole raylib package. What clang generates for raylib's own prototypes on x86-64:

Vector2 {float,float}   →  declare <2 x float> @GetMousePosition()
Color   {u8,u8,u8,u8}   →  declare void @ClearBackground(i32)
Rectangle {4 × int}     →  declare { i64, i64 } @mkrect()

None of those is the struct's own LLVM type. A small aggregate's calling convention is not part of its layout — it is a per-target classification the caller has to reproduce, and x86-64, arm64 and wasm32 classify differently. Putting that in emit.ml is three classifiers to write and then keep correct forever, and a mistake shows up as (.y m) returning garbage rather than as a link error.

So vendor/raylib/shim.c has one wrapper per binding, each one flattening the aggregates: a struct returns through an out-pointer, a struct argument is passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates a copy. clang classifies all of it, per target, for free. check.ml enforces the rule — an aggregate in a declare signature is rejected with the reason — so the boundary cannot quietly acquire one. This is plan.org's "one narrow host ABI, implemented twice", and flan_rt.c is the same pattern.

The price is a hand-written wrapper per raylib call. They are one-liners and mechanical enough to generate if that ever becomes the bottleneck.

raylib.flan declares each -raw entry point and wraps it in an ordinary Flan function just below, so the surface sand.flan sees is (rl/get-mouse-position) returning a Vector2. Verified end to end, headless: GetColor(0x11223344) comes back as 17 34 51 68, four separate bytes — a Color is not the little-endian reading of the packed integer, so an identity would have passed a weaker test. That case is in the acceptance table, skipped if libraylib is not installed.

The bindings are 18 calls: window (init-window, close-window, window-should-close?, set-target-fps, set-trace-log-level), keyboard (key-pressed?/down?/released?), mouse (mouse-button-pressed?/down?/ released?, get-mouse-position), get-color, and drawing (begin-drawing, end-drawing, draw-fps, clear-background, draw-rectangle), plus the Key, MouseButton and TraceLogLevel enums. Adding one is three lines: a declare, an extern prototype, and a one-line wrapper.

No raylib headers are needed: shim.c declares the prototypes it uses, so the build depends on the shared library being linkable and not on raylib-devel. vendor/raylib/link carries -l:libraylib.so.550 because Fedora ships the runtime library without the .so symlink.

Packages

lib/load.ml resolves (import rl "vendor:raylib") before the checker runs. The directory is the package; vendor: is a collection, resolved by walking up from the importing file until a directory of that name is found; a path with no collection is relative to the importing file. Importing is a rename: every top-level name the package declares becomes alias/name, and every use of one — in a type, in a body, in a struct literal, in an array length — is rewritten to match. Local bindings shadow. Nothing downstream knows a package existed; the checker sees one flat list of declarations whose names contain a slash.

A package may also carry the C it binds to: every .c file in the directory is compiled into the build, and a file named link lists extra linker arguments.

This is not a module system yet. No visibility (hence rl/get-color-raw being callable), no cycle detection, and a package cannot import another one.

sand.flan is two programs

plan.org wants sand tested twice — interactive at 120 fps, and headless over N frames with the grid hashed, the version CI runs on native and wasm32. Those cannot be one binary: Load collects a package's C sources and linker arguments unconditionally, so anything importing the raylib package links libraylib on every target regardless of what its main does, and on wasm32 that link cannot succeed.

So the simulation moved to sand-sim/, which imports nothing. sand.flan imports it as sim/ and adds the window, the mouse and the drawing; test/programs/sand-headless.flan imports it and adds a seed, four deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics.

The headless case is what actually verifies milestone 4 — running the interactive build only proves it enters its loop, because with no mouse input the grid stays empty and paint-at, settle and move-grain never execute on real data. Measured through the probe: 168 grains painted around row 4–8, still 168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.

Three edits were made to sand.flan's own text, and they are language decisions rather than fixes:

• (defconst gravity 0.05) → (defconst gravity f32 0.05). An untyped float constant is f64, velocity is [f32], and there is no implicit widening.
• (defvar current-color u32) → i32. It is an index into colors, and (len colors) is an i32.
• The file was split as above, so its body now says sim/rows and so on.

(defn main []) is unchanged — the short form, as plan.org says.

Painting is on hold left mouse button rather than on space, since the mouse bindings exist now. Space is still what cycles the colour, on release, which is a leftover and probably wants to move to the right button or to a key press.

Bounds checks — done at milestone 3

at and slice emit icmp → br → cold block → call → unreachable; a failure names the source location. Three check sites: at on [n T] (static bound, folded by LLVM for a literal index — and a literal that is out of bounds never reaches emit, check.ml rejects it), at on a slice or string (runtime len), and slice (two comparisons — lo <= hi is not redundant, without it a reversed range yields a huge unsigned length). All comparisons unsigned. Build.opts.checks is on by default and not tied to opts.opt, which is what lets the acceptance table run the same programs at -O0 and -O2 with identical checks. The flag is --no-bounds-checks.

The write path is its own case: (set (at arr n) …) lowers through place/Pindex, not through At, so a refactor that split them would break the write check silently. The test covers both.

Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.

Why there is no interpreter

Open decision #7 is settled: the compiled path is the only backend. Both arguments for a permanent interpreter had expired — the instrumentation step debugger that wanted it is cut, and compiled redefinition measured at ~16ms, perceptually instant for expression eval too. Milestone 3 did not need an oracle either: the acceptance table is hand-written, so the table is the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path.

The layout, which is the whole backend design

i8..i64 / u8..u64   i8..i64          signedness lives in the ops
f32 f64             float double
bool                i1
an enum             i32
[T] and string      { ptr, i64 }     ptr+len, non-owning
[n T]               [n x T]          inline, a value
(Ptr T)             ptr              opaque pointers
(Option T)          { i8, T }        tag 0 None, 1 Some
a struct            a literal struct, declaration order
Unit and Never      {}

No object headers anywhere, so a Flan struct is exactly its C struct and nothing marshals. Two consequences carry the semantics:

• Every slot is an alloca. Reading a local is a load, assigning is a store, and a store of an aggregate is the copy spec-memory.md requires. addr of a local is then just the alloca, and mem2reg removes the ones nobody addressed. test/programs/values.flan pins this down.
• A place is a pointer, a value is a load from it. (set (.pos c) …) through a (Ptr Cursor) becomes a getelementptr on the pointer, not on a copy. This is the split that would have made a tree-walker silently wrong.

Non-local exit is lowered explicitly: return, some and a failed bounds check are branches, never platform unwinding, so wasm32 needs no exception proposal.

Sharp edges found and left visible

• An index converts from a narrower integer and never from a wider one. (nth colors current-color) with a u32 index works — anything above 2³¹ truncates to a negative i32 and the unsigned bounds check rejects it. An i64 index is refused with the reason: 2³²+5 truncates to 5 and would read the wrong element with no trap at all.
• A u64 literal is its 64-bit pattern, so 0xcbf29ce484222325 is a real u64 and not an error. The cost is that a negative decimal literal is accepted as a u64 too, because the reader records the value and not how it was written. Narrower unsigned types keep the strict check, which is where a typo like 300 for a u8 actually shows up.
• A folded constant skips check. (defconst rows (/ h c)) is emitted from the folding pass's value, because a global's initialiser has to be a compile-time constant and only that pass knows this one is. Its range check is therefore its own call to in_range; there is a regression test.
• A let binding takes no type annotation, which is why sand-sim names its FNV constants instead of writing them inline.
• (defn f [] f65 0.0) still says unknown name rather than did you mean f64: with a single body form the parser cannot tell a return type from the first expression. Only the parameter position and (Option …) are unambiguous.

Where build time goes

flan build calc-me.flan was ~160ms, and ~95% of it was clang. The object cache is in, and it is now ~110ms:

Step	Cost
frontend: read → parse → load → check → emit	<10ms, below the timer
clang on the .ll	60ms — llc does the same codegen in 20ms
clang on flan_rt.c	40ms — now cached, paid once
link	20ms

Every C translation unit a build needs — the host shim and each package's shim — goes through Build.compile_c, which compiles to a .o under $TMPDIR/flan-objcache and reuses it. The key is a digest of the source text, the compiler (its path, size and mtime, so an upgrade invalidates without paying a clang --version subprocess per build), opts.opt and opts.target. The opt level has to be in there: the acceptance table builds the same programs at -O0 and -O2, and an -O2 object must not serve an -O0 build. The object is written to a temporary name and renamed into place, so two concurrent builds cannot see a half-written one.

Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is mostly linking libraylib and its shim.c was never the cost. The cache is keyed by content, so it never needs invalidating by hand — rm -rf on the directory is only ever a disk-space decision.

The other cheap win is still open: skip the clang driver for the .ll (llc + link directly), worth another ~40ms. It is a subset of the dev path's machinery. Check llc's major version against clang's before relying on it — the emitted IR text is currently absorbed by the driver behind -Wno-override-module, and a version mismatch surfaces as IR parse errors.

There is still no REPL. Nothing does redefinition, dlopen, or nREPL. build is the only way to run code.

Next — the REPL is the priority

Decided in conversation: wasm32 can wait (it is believed to be a solved problem once the builtins archive is in place), and the dev loop is the thesis of the project, so it comes first. Staged so each step is runnable on its own — the failure mode is building a daemon and a protocol before knowing the reload primitive works.

1. The reload primitive, measured. llc + ld -shared → .so → dlopen → call. No sockets, no protocol. A test that compiles one function, loads it, calls it, recompiles it changed, and calls it again. plan.org's 16ms was measured with clang in isolation and never in this codebase. It forces the first real change: emit.ml needs a mode that compiles one redefinable function into its own module against the existing globals, rather than as a whole program.
2. Indirection cells. Every cross-function call in a dev build goes through a pointer; redefinition is one atomic store. A fork in emit.ml between dev and release codegen, and the first time Build.opts means something semantic rather than an optimisation level. test/programs/ gets a case where a running loop's callee is swapped mid-run.
3. The agent, in C. A socket listener in the game process, dlopen off the game thread with RTLD_NOW, and the staged cell publish at a frame boundary. It lives next to flan_rt.c — no OCaml runtime in the game binary. sand.flan is the test: redefine settle while grains are falling and see the behaviour change with no stutter and no dropped frame.
4. The daemon and nREPL (bencode over a socket; eval, load-file, describe, interrupt), then 5. the Emacs client — a focused ~3–5k line client, not a CIDER fork. Deliberately last and deliberately separate: the protocol is mechanical once 1–3 exist, and the editor client is where the taste is.

Two decisions to settle before step 2, because both change codegen and are painful to retrofit:

• Do cells cover globals, or only functions? plan.org says redefining a defvar is not covered (open decision #6, milestone 7). But sand's grid is a global, and "edit the code, keep the sand" is exactly the demo — which works only if globals survive a reload, meaning the new .so must not re-emit them.
• What is a redefinition unit — one function, or a file? A file is much easier to make correct and is what load-file wants anyway; one function is what C-c C-c wants and is where the 16ms number comes from.

Deferred until after the dev loop:

6. wasm32. The user installed wasi-libc-devel and wasi-libc-static; the sysroot is /usr/wasm32-wasi and wasm-ld is present. clang --target=wasm32-wasi --sysroot=/usr/wasm32-wasi gets past the headers and then fails to link: it wants lib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a, which no Fedora package provides (dnf provides '*libclang_rt.builtins*wasm*' finds nothing). It has to come from a wasi-sdk release, dropped into clang's resource directory. After that: teach build.ml --sysroot, and run the acceptance table — sand-headless.flan included, which is exactly why it does not import raylib — on both targets in CI. Note plan.org has the web build linking raylib via emscripten, which brings its own sysroot: wasi-sdk is right for the headless table, not necessarily for the eventual game build.
7. Loose ends from milestone 4, none of them blocking: block-scoped defer; package visibility, so rl/get-color-raw is not callable; a package importing a package; imported unions.

Watch for

The rule that caught the two misparse bugs applies unchanged: anything that binds a name, alters control flow, or is not yet implemented must be recognised explicitly and rejected if unsupported. check.ml rejects Vec, Map, Result/try, union values, closures, quoted symbols, generics and function values by name, each with the milestone it belongs to; load.ml rejects the package shapes it does not handle; and the FFI boundary rejects an aggregate. The tests assert on the reason, not just on the failure.

Untracked on purpose

calc-me and sand, the executables flan build drops beside their sources, are now in .gitignore — anchored (/calc-me, /sand) so the patterns cannot also match sand-sim/ or anything nested.

old-ocaml/ — the pre-rewrite menhir/ocamllex frontend, kept as reference and excluded from the build by the root dune file. Its contents are also in git history at 2c232dd.