flan/NEXT.md

# Where this is

Milestone 4 is done: **sand.flan builds, links raylib and runs**, and its
simulation has a headless acceptance case that runs on the `dune test` path at
`-O0` and `-O2`. Milestones 2 and 3 are behind it (`calc-me.flan` compiles and
runs; the interpreter was dropped — open decision #7, settled, see below).

```
reader ✅ → parse ✅ → load ✅ → check ✅ → emit ✅ → clang ✅
```

| File | What it does |
|---|---|
| `lib/loc.ml` | source locations + `Loc.Error`, the frontend's one exception |
| `lib/form.ml` | reader output: `Sym Kw Int Float Str Byte List Vec Map` |
| `lib/reader.ml` | hand-written S-expression reader, no menhir/ocamllex |
| `lib/ast.ml` | AST: `texpr`, `expr`, `place`, `pattern`, `decl` |
| `lib/parse.ml` | forms → AST; special forms, desugaring, declarations |
| `lib/load.ml` | **imports: a package directory → qualified declarations** |
| `lib/types.ml` | resolved types; structural equality, `Never` fits anywhere |
| `lib/tast.ml` | the typed IR the backend consumes |
| `lib/check.ml` | AST → typed IR; two passes, bidirectional |
| `lib/prelude.ml` | printers + `rand-f32`, written in Flan |
| `lib/emit.ml` | typed IR → LLVM IR text |
| `lib/build.ml` | `.ll` + the shim + the packages' C → clang → executable |
| `runtime/flan_rt.c` | the host ABI: argv, stdout, exit, 4 conversions |
| `vendor/raylib/` | **the raylib package: `raylib.flan`, `shim.c`, `link`** |
| `sand-sim/` | **the falling-sand simulation, with no raylib in it** |
| `bin/main.ml` | `flan read \| parse \| check \| emit \| build \| run` |
| `test/test_flan.ml` | reader, parser and checker |
| `test/test_acceptance.ml` | expression/result pairs + whole programs + the traps |

```
$ flan run calc-me.flan "1 + 2 * (3 - 0.5) / 2"
3.5
$ flan run test/programs/sand-headless.flan
2256461126764447066
$ flan run sand.flan                      # a window, 120 fps, hold space
```

## What milestone 4 added

**`dotimes`** desugars in `check.ml` to a `Let` plus a `While` — no new IR node.
The bound is evaluated once into a hidden slot before the loop, so a body that
changes it cannot change the trip count, and the loop variable is not
assignable, which makes the generated step its only writer.

**`defer`** is recognised in `check_fn` and nowhere else, because that is the
only place that knows a form is at the top level of a function body. Each one
is checked in place, then registered on the context; it emits nothing where it
stands. Function exit runs them innermost-first, and an explicit `return` runs
the ones registered *above* it — a defer written below a return has not
executed yet and must not fire. A trap runs none of them, which follows from
the bounds-check shape (`noreturn` then `unreachable`) rather than being a
separate decision.

`defer` inside a `let`, a loop or a branch is **rejected**, not accepted with
function scope. It would run once at function exit rather than once per
iteration, and that is the silent-wrongness class the rule below is about.
Block-scoped defer is real work and is not done.

**New builtins:** `zeroed` (takes its type from the place it is stored into),
`min`/`max` (each operand through a slot, so neither is evaluated twice),
`bit-and`/`bit-or`/`bit-xor`/`<<`/`>>` (integers only; `>>` is arithmetic on a
signed type and logical on an unsigned one), and `rand-f32`.

**`rand-f32` is in the prelude, in Flan** — PCG-XSH-RR 32 over a `u64` state.
It is not libc's, because a grid hash is only a regression test if the sequence
is byte-identical on native and wasm32 (plan.org, RNG is ours). `rand-seed`
sets the state. This is what the bitwise operators were added for.

**Enums and keywords.** `(defenum Name [member value ...])` gives a type that
is an `i32` at run time and its own type in the checker, so `:space` at a call
site resolves against the parameter's enum and a typo is an error there rather
than a wrong number later. A keyword means nothing where no enum is expected —
there is no keyword type to fall back on.

## Why the FFI goes through a C shim

The decision that shapes the whole raylib package. What clang generates for
raylib's own prototypes on x86-64:

```
Vector2 {float,float}   →  declare <2 x float> @GetMousePosition()
Color   {u8,u8,u8,u8}   →  declare void @ClearBackground(i32)
Rectangle {4 × int}     →  declare { i64, i64 } @mkrect()
```

None of those is the struct's own LLVM type. A small aggregate's calling
convention is not part of its layout — it is a per-target classification the
*caller* has to reproduce, and x86-64, arm64 and wasm32 classify differently.
Putting that in `emit.ml` is three classifiers to write and then keep correct
forever, and a mistake shows up as `(.y m)` returning garbage rather than as a
link error.

So `vendor/raylib/shim.c` has one wrapper per binding, each one flattening the
aggregates: a struct returns through an out-pointer, a struct argument is
passed by pointer, a Flan string crosses as ptr+len and the shim NUL-terminates
a copy. clang classifies all of it, per target, for free. `check.ml` enforces
the rule — an aggregate in a `declare` signature is rejected with the reason —
so the boundary cannot quietly acquire one. This is plan.org's "one narrow host
ABI, implemented twice", and `flan_rt.c` is the same pattern.

The price is a hand-written wrapper per raylib call. They are one-liners and
mechanical enough to generate if that ever becomes the bottleneck.

`raylib.flan` declares each `-raw` entry point and wraps it in an ordinary Flan
function just below, so the surface sand.flan sees is `(rl/get-mouse-position)`
returning a `Vector2`. Verified end to end, headless: `GetColor(0x11223344)`
comes back as `17 34 51 68`, four separate bytes — a `Color` is *not* the
little-endian reading of the packed integer, so an identity would have passed a
weaker test. That case is in the acceptance table, skipped if `libraylib` is
not installed.

The bindings are 18 calls: window (`init-window`, `close-window`,
`window-should-close?`, `set-target-fps`, `set-trace-log-level`), keyboard
(`key-pressed?`/`down?`/`released?`), mouse (`mouse-button-pressed?`/`down?`/
`released?`, `get-mouse-position`), `get-color`, and drawing (`begin-drawing`,
`end-drawing`, `draw-fps`, `clear-background`, `draw-rectangle`), plus the
`Key`, `MouseButton` and `TraceLogLevel` enums. Adding one is three lines: a
`declare`, an `extern` prototype, and a one-line wrapper.

No raylib headers are needed: `shim.c` declares the prototypes it uses, so the
build depends on the shared library being linkable and not on `raylib-devel`.
`vendor/raylib/link` carries `-l:libraylib.so.550` because Fedora ships the
runtime library without the `.so` symlink.

## Packages

`lib/load.ml` resolves `(import rl "vendor:raylib")` before the checker runs.
The directory is the package; `vendor:` is a collection, resolved by walking up
from the importing file until a directory of that name is found; a path with no
collection is relative to the importing file. Importing is a **rename**: every
top-level name the package declares becomes `alias/name`, and every use of one
— in a type, in a body, in a struct literal, in an *array length* — is
rewritten to match. Local bindings shadow. Nothing downstream knows a package
existed; the checker sees one flat list of declarations whose names contain a
slash.

A package may also carry the C it binds to: every `.c` file in the directory is
compiled into the build, and a file named `link` lists extra linker arguments.

This is not a module system yet. No visibility (hence `rl/get-color-raw` being
callable), no cycle detection, and a package cannot import another one.

## sand.flan is two programs

plan.org wants sand tested twice — interactive at 120 fps, and headless over N
frames with the grid hashed, the version CI runs on native *and* wasm32. Those
cannot be one binary: `Load` collects a package's C sources and linker
arguments unconditionally, so anything importing the raylib package links
libraylib on every target regardless of what its `main` does, and on wasm32
that link cannot succeed.

So the simulation moved to `sand-sim/`, which imports nothing. `sand.flan`
imports it as `sim/` and adds the window, the mouse and the drawing;
`test/programs/sand-headless.flan` imports it and adds a seed, four
deterministic clouds, 40 frames and an FNV-1a hash. One copy of the physics.

The headless case is what actually *verifies* milestone 4 — running the
interactive build only proves it enters its loop, because with no mouse input
the grid stays empty and `paint-at`, `settle` and `move-grain` never execute on
real data. Measured through the probe: 168 grains painted around row 4–8, still
168 after 40 frames, lowest occupied row 68. Grains fall, and none are lost.

**Three edits were made to sand.flan's own text**, and they are language
decisions rather than fixes:

- `(defconst gravity 0.05)` → `(defconst gravity f32 0.05)`. An untyped float
  constant is `f64`, `velocity` is `[f32]`, and there is no implicit widening.
- `(defvar current-color u32)` → `i32`. It is an index into `colors`, and
  `(len colors)` is an `i32`.
- The file was split as above, so its body now says `sim/rows` and so on.

`(defn main [])` is unchanged — the short form, as plan.org says.

Painting is on **hold left mouse button** rather than on space, since the mouse
bindings exist now. Space is still what cycles the colour, on release, which is
a leftover and probably wants to move to the right button or to a key press.

## Bounds checks — done at milestone 3

`at` and `slice` emit `icmp` → `br` → cold block → `call` → `unreachable`; a
failure names the source location. Three check sites: `at` on `[n T]` (static
bound, folded by LLVM for a literal index — and a literal that is out of bounds
never reaches emit, `check.ml` rejects it), `at` on a slice or string (runtime
len), and `slice` (two comparisons — `lo <= hi` is not redundant, without it a
reversed range yields a huge unsigned length). All comparisons unsigned.
`Build.opts.checks` is on by default and **not** tied to `opts.opt`, which is
what lets the acceptance table run the same programs at `-O0` and `-O2` with
identical checks. The flag is `--no-bounds-checks`.

The write path is its own case: `(set (at arr n) …)` lowers through
`place`/`Pindex`, not through `At`, so a refactor that split them would break
the write check silently. The test covers both.

Cost, measured: a 50M-iteration dependency chain over a 1024-element array runs
at 0.11–0.12s checked against 0.12–0.13s unchecked. Indistinguishable.

## Why there is no interpreter

Open decision #7 is settled: **the compiled path is the only backend.** Both
arguments for a permanent interpreter had expired — the instrumentation step
debugger that wanted it is cut, and compiled redefinition measured at ~16ms,
perceptually instant for expression eval too. Milestone 3 did not need an
oracle either: the acceptance table is hand-written, so the table *is* the
oracle. Consequences already applied: milestone 2's "interpreted calls per
second" criterion is dropped, and the host ABI moved onto the critical path.

## The layout, which is the whole backend design

```
i8..i64 / u8..u64   i8..i64          signedness lives in the ops
f32 f64             float double
bool                i1
an enum             i32
[T] and string      { ptr, i64 }     ptr+len, non-owning
[n T]               [n x T]          inline, a value
(Ptr T)             ptr              opaque pointers
(Option T)          { i8, T }        tag 0 None, 1 Some
a struct            a literal struct, declaration order
Unit and Never      {}
```

No object headers anywhere, so a Flan struct is exactly its C struct and
nothing marshals. Two consequences carry the semantics:

- **Every slot is an `alloca`.** Reading a local is a `load`, assigning is a
  `store`, and a `store` of an aggregate *is* the copy `spec-memory.md`
  requires. `addr` of a local is then just the alloca, and `mem2reg` removes
  the ones nobody addressed. `test/programs/values.flan` pins this down.
- **A place is a pointer, a value is a load from it.** `(set (.pos c) …)`
  through a `(Ptr Cursor)` becomes a `getelementptr` on the pointer, not on a
  copy. This is the split that would have made a tree-walker silently wrong.

Non-local exit is lowered explicitly: `return`, `some` and a failed bounds
check are branches, never platform unwinding, so wasm32 needs no exception
proposal.

## Sharp edges

Most of these are edges the language keeps and you should know about. Two —
the top-level namespace and the shift count, both found by review after
milestone 4 — were bugs that reached LLVM or ran wrong, and are **fixed**; each
says so. They stay written down because each one is now a rule the checker
enforces, and a later change could quietly drop it.

- **An index converts from a narrower integer and never from a wider one.**
  `(nth colors current-color)` with a `u32` index works — anything above 2³¹
  truncates to a negative `i32` and the unsigned bounds check rejects it. An
  `i64` index is refused with the reason: 2³²+5 truncates to 5 and would read
  the wrong element with no trap at all.
- **There is one top-level namespace, and `check.ml` now enforces it.** The
  environment's tables are per-kind — structs, unions, aliases, enums,
  functions, externs and globals each have their own — so only a function was
  ever checked for a duplicate. `(defn item …)` beside `(defvar item …)` type
  checked and then died in LLVM as `redefinition of function '@flan.item'`, a
  message about an emitted symbol with no source location left, and two
  colliding *type* declarations were not caught anywhere. One pass over
  `Ast.declared_name` now runs before every other collection pass and rejects
  the second declaration of a name whatever kind either one is. `declared_name`
  lives in `ast.ml` because `Load` needs exactly the same set — the names an
  import renames — and two copies of that list would drift.
- **A shift count is bounded, two different ways.** A shift by the operand's
  own width or more is *poison* in LLVM, not a wrong number: `(defn main [] i32
  (<< 1 32))` compiled at -O2 to a bare `retq`, returning an undefined value. A
  literal count out of range is now rejected in `check.ml` — that is the typo
  case — and `emit.ml` masks a computed count to `width - 1`, which is what the
  hardware does anyway and which LLVM folds away whenever the count is
  constant. The prelude's rotate masks its own count; that is now redundant but
  harmless.
- **A `u64` literal is its 64-bit pattern**, so `0xcbf29ce484222325` is a real
  `u64` and not an error. The cost is that a negative *decimal* literal is
  accepted as a `u64` too, because the reader records the value and not how it
  was written. Narrower unsigned types keep the strict check, which is where a
  typo like `300` for a `u8` actually shows up.
- **A folded constant skips `check`.** `(defconst rows (/ h c))` is emitted
  from the folding pass's value, because a global's initialiser has to be a
  compile-time constant and only that pass knows this one is. Its range check
  is therefore its own call to `in_range`; there is a regression test.
- A `let` binding takes no type annotation, which is why `sand-sim` names its
  FNV constants instead of writing them inline.
- `(defn f [] f65 0.0)` still says *unknown name* rather than *did you mean
  f64*: with a single body form the parser cannot tell a return type from the
  first expression. Only the parameter position and `(Option …)` are
  unambiguous.

## Where build time goes

`flan build calc-me.flan` was ~160ms, and ~95% of it was clang. **The object
cache is in**, and it is now ~110ms:

| Step | Cost |
|---|---|
| frontend: read → parse → load → check → emit | <10ms, below the timer |
| `clang` on the `.ll` | 60ms — `llc` does the same codegen in **20ms** |
| `clang` on `flan_rt.c` | 40ms — **now cached, paid once** |
| link | 20ms |

Every C translation unit a build needs — the host shim and each package's shim
— goes through `Build.compile_c`, which compiles to a `.o` under
`$TMPDIR/flan-objcache` and reuses it. The key is a digest of the source text,
the compiler (its path, size and mtime, so an upgrade invalidates without
paying a `clang --version` subprocess per build), `opts.opt` and `opts.target`.
The opt level has to be in there: the acceptance table builds the same programs
at `-O0` and `-O2`, and an `-O2` object must not serve an `-O0` build. The
object is written to a temporary name and `rename`d into place, so two
concurrent builds cannot see a half-written one.

Measured: calc-me 160ms → 110ms; sand ~720ms → ~700ms, since sand's time is
mostly linking libraylib and its `shim.c` was never the cost. The cache is
keyed by content, so it never needs invalidating by hand — `rm -rf` on the
directory is only ever a disk-space decision.

The other cheap win is still open: skip the clang driver for the `.ll` (`llc` +
link directly), worth another ~40ms. It is a subset of the dev path's
machinery. Check `llc`'s major version against clang's before relying on it —
the emitted IR text is currently absorbed by the driver behind
`-Wno-override-module`, and a version mismatch surfaces as IR parse errors.

**There is still no REPL.** Nothing does redefinition, `dlopen`, or nREPL.
`build` is the only way to run code.

## Next — the REPL is the priority

Decided in conversation: wasm32 can wait (it is believed to be a solved problem
once the builtins archive is in place), and **the dev loop is the thesis of the
project**, so it comes first. Staged so each step is runnable on its own —
the failure mode is building a daemon and a protocol before knowing the reload
primitive works.

1. **The reload primitive, measured.** `llc` + `ld -shared` → `.so` →
   `dlopen` → call. No sockets, no protocol. A test that compiles one function,
   loads it, calls it, recompiles it changed, and calls it again. plan.org's
   16ms was measured with `clang` in isolation and never in this codebase.
   It forces the first real change: `emit.ml` needs a mode that compiles one
   redefinable function into its own module *against the existing globals*,
   rather than as a whole program.
2. **Indirection cells.** Every cross-function call in a dev build goes through
   a pointer; redefinition is one atomic store. A fork in `emit.ml` between dev
   and release codegen, and the first time `Build.opts` means something
   semantic rather than an optimisation level. `test/programs/` gets a case
   where a running loop's callee is swapped mid-run.
3. **The agent, in C.** A socket listener in the game process, `dlopen` off the
   game thread with `RTLD_NOW`, and the staged cell publish at a frame
   boundary. It lives next to `flan_rt.c` — no OCaml runtime in the game
   binary. sand.flan is the test: redefine `settle` while grains are falling
   and see the behaviour change with no stutter and no dropped frame.
4. **The daemon and nREPL** (bencode over a socket; `eval`, `load-file`,
   `describe`, `interrupt`), then **5. the Emacs client** — a focused ~3–5k
   line client, not a CIDER fork. Deliberately last and deliberately separate:
   the protocol is mechanical once 1–3 exist, and the editor client is where
   the taste is.

**Two decisions to settle before step 2**, because both change codegen and are
painful to retrofit:

- **Do cells cover globals, or only functions?** plan.org says redefining a
  `defvar` is not covered (open decision #6, milestone 7). But sand's `grid` is
  a global, and "edit the code, keep the sand" is exactly the demo — which
  works only if globals *survive* a reload, meaning the new `.so` must not
  re-emit them.
- **What is a redefinition unit — one function, or a file?** A file is much
  easier to make correct and is what `load-file` wants anyway; one function is
  what `C-c C-c` wants and is where the 16ms number comes from.

Deferred until after the dev loop:

6. **wasm32.** The user installed `wasi-libc-devel` and `wasi-libc-static`; the
   sysroot is `/usr/wasm32-wasi` and `wasm-ld` is present. `clang
   --target=wasm32-wasi --sysroot=/usr/wasm32-wasi` gets past the headers and
   then **fails to link**: it wants
   `lib/clang/20/lib/wasm32-unknown-wasi/libclang_rt.builtins.a`, which no
   Fedora package provides (`dnf provides '*libclang_rt.builtins*wasm*'` finds
   nothing). It has to come from a wasi-sdk release, dropped into clang's
   resource directory. After that: teach `build.ml` `--sysroot`, and run the
   acceptance table — `sand-headless.flan` included, which is exactly why it
   does not import raylib — on both targets in CI.
   Note plan.org has the *web* build linking raylib via emscripten, which
   brings its own sysroot: wasi-sdk is right for the headless table, not
   necessarily for the eventual game build.
7. **Loose ends from milestone 4**, none of them blocking: block-scoped
   `defer`; package visibility, so `rl/get-color-raw` is not callable; a
   package importing a package; imported unions.

## Watch for

The rule that caught the two misparse bugs applies unchanged: **anything that
binds a name, alters control flow, or is not yet implemented must be recognised
explicitly and rejected if unsupported.** `check.ml` rejects `Vec`, `Map`,
`Result`/`try`, union values, closures, quoted symbols, generics and function
values *by name*, each with the milestone it belongs to; `load.ml` rejects the
package shapes it does not handle; and the FFI boundary rejects an aggregate.
The tests assert on the reason, not just on the failure.

## Untracked on purpose

`calc-me` and `sand`, the executables `flan build` drops beside their sources,
are now in `.gitignore` — anchored (`/calc-me`, `/sand`) so the patterns cannot
also match `sand-sim/` or anything nested.

`old-ocaml/` — the pre-rewrite menhir/ocamllex frontend, kept as reference and
excluded from the build by the root `dune` file. Its contents are also in git
history at `2c232dd`.