There is no interpreter, and the plan stops promising one

The Compilation section was written around a permanent tree-walking backend for
expression eval. Open decision #7 closed the other way and BUILT.md records it:
compiling is the only way a form is ever run. The diagram, the milestone-2 exit
criterion, milestone 7's "free in the interpreter", the dev/release table and
the decision itself all said otherwise, and lib/expand.ml states the settled
answer at the top of the file.

Also here, because the same section was the place they were missing: the
hand-written x86-64 code generator, which is a second route from the typed IR to
the same observable behaviour rather than a second semantics; DWARF from both
code generators rather than from LLVM alone; OCaml as a settled host language;
and the map-new example, which has named its key and value types since braces
stopped being a type spelling.
This commit is contained in:
Joseph Ferano 2026-09-14 07:27:36 +07:00
parent be39f32cb6
commit 00163bcf34

158
plan.org
View File

@ -101,7 +101,7 @@ world.
- Map keys initially use compiler-provided structural equality and hashing for - Map keys initially use compiler-provided structural equality and hashing for
integers, enums, strings, fixed arrays and value structs; pointers, slices and integers, enums, strings, fixed arrays and value structs; pointers, slices and
owning containers are excluded. A map is homogeneous, and empty construction owning containers are excluded. A map is homogeneous, and empty construction
is type-directed: ~(defvar enemies (Map string Enemy) (map-new))~. ~get~ names its types: ~(let [enemies (map-new string Enemy)] ...)~. ~get~
returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See
spec-memory.md for the deferred move-aware operations. spec-memory.md for the deferred move-aware operations.
- Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~. - Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~.
@ -401,8 +401,8 @@ primitives, and are never bootstrapped away.
The LLVM question does not bear on this: the release backend emits LLVM IR *as The LLVM question does not bear on this: the release backend emits LLVM IR *as
text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or
Rust buy nothing here. What the choice actually turns on is that milestones 25 Rust buy nothing here. What the choice actually turns on is that milestones 25
are a reader, a typed IR, a checker and a tree-walking interpreter — variants and are a reader, a typed IR, a checker and the code generators behind it — variants
exhaustive pattern matching, which is the one domain where OCaml is not a and exhaustive pattern matching, which is the one domain where OCaml is not a
preference but a clear win. There is also a menhir lexer/parser already started preference but a clear win. There is also a menhir lexer/parser already started
in ~old-ocaml/~. in ~old-ocaml/~.
@ -415,7 +415,7 @@ build sequence. For a game language it buys dogfooding at the price of a second
compiler to maintain forever. Choose as if the host language is permanent. compiler to maintain forever. Choose as if the host language is permanent.
* Milestone-2 primitives * Milestone-2 primitives
The interpreter provides these; everything else is written in Flan. Keeping the The runtime provides these; everything else is written in Flan. Keeping the
list short is the whole strategy — it is what makes the LLVM backend and the list short is the whole strategy — it is what makes the LLVM backend and the
wasm32 target cheap, because a primitive is the only thing implemented twice. wasm32 target cheap, because a primitive is the only thing implemented twice.
@ -477,33 +477,54 @@ is Clojure's ~ns~ form: no path that must mirror the directory, no
root-directory aliases. root-directory aliases.
* Compilation * Compilation
*Two backends and three paths.* The split is not dev-vs-release; it is *One evaluator and three paths.* The split is not dev-vs-release; it is
/does this code have a frame budget/. /does this code have a frame budget/. There is no interpreter: open decision #7
is settled the other way from how this section was first written, and BUILT.md's
"Why there is no interpreter" carries the reasoning. Compiling is the only way a
form is ever run, so there is no second evaluator that could disagree with the
first about what a program means.
#+begin_src #+begin_src
expression eval: flan → typed IR → interpreter ~1ms expression eval: flan → typed IR → .ll → llc → ld -shared → dlopen → call
dev redefinition: flan → typed IR → .ll → llc → ld -shared → dlopen → cell store ~19ms (MEASURED)
~16ms (MEASURED) dev redefinition: the same path, ending in a cell store rather than a call
release build: flan → typed IR → .ll → clang --target={native,wasm32} release build: flan → typed IR → .ll → clang --target={native,wasm32}
#+end_src #+end_src
*Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate, *Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate,
because the whole point of the live loop is that you see the result. 16ms is one because the whole point of the live loop is that you see the result. 19ms is
frame at 60fps and under the ~50ms threshold where a response stops feeling around one frame at 60fps and under the ~50ms threshold where a response stops
instantaneous. The rule that buys it: *never invoke the ~clang~ driver on the dev feeling instantaneous. The rule that buys it: *never invoke the ~clang~ driver on
path.* the dev path.*
*Expression eval*~C-c C-e~, calling a function, inspecting a var, running a *Expression eval*~C-x C-e~, calling a function, inspecting a var, running a
test — goes to the tree-walking interpreter. Sub-millisecond, no subprocess. This test — is compiled like everything else, into its own shared object, which is
is the permanent REPL backend, not a milestone-2 scaffold. then loaded and called. What made an interpreter look necessary was the
assumption that this had to be sub-millisecond; the measurement below is that
the compiled route is already inside the threshold, and the one thing an
interpreter would have bought is an oracle the hand-written acceptance table
supplies instead.
*Dev redefinition*~C-c C-c~ on a function inside a running game — cannot use *Dev redefinition*~C-c C-c~ on a function inside a running game — has an 8ms
the interpreter, because that code has an 8ms frame budget. It recompiles the one frame budget to respect. It recompiles the one function, links it, and does the
function, links it, and does the atomic indirection-cell store. This is what the atomic indirection-cell store. This is what the Hot reload section has always
Hot reload section has always described; the interpreter does not replace it. described, and it is the same machinery expression eval uses, one step further
on.
*Release* is whole-program AOT with direct calls and no cells. *Release* is whole-program AOT with direct calls and no cells.
*A second code generator, not a second evaluator.* ~lib/x86.ml~ emits x86-64
machine code directly and is selected with ~--x86~; it exists because ~llc~ is
most of the 19ms above. It is a different route from the same typed IR to the
same observable behaviour, not a different semantics, and what holds it to that
is ~spike/x86/survey.sh~: every program in the corpus is built both ways and
byte-compared on stdout, stderr and exit status. At the time of writing that is
103 MATCH, 0 DIFFER, 0 refused by name. It handles conditions, bounds checks,
indirection cells, redefinition modules and DWARF line tables; what it does not
have, and must not grow, is an aggregate classifier — an ~--x86~ host therefore
takes ~--x86~ modules and an LLVM host takes LLVM ones, and ~lib/build.ml~
refuses the crossed pair by name.
** Measured redefinition latency ** Measured redefinition latency
Single function, x86-64, clang 20.1.8, 20 iterations each: Single function, x86-64, clang 20.1.8, 20 iterations each:
@ -563,9 +584,10 @@ This is why *the dev runtime is multithreaded* — it needs the reload thread. T
is settled, and is independent of whether the /language/ exposes threads, which is settled, and is independent of whether the /language/ exposes threads, which
is still open decision #4. is still open decision #4.
It also bears on whether the interpreter survives: if the agent can ~dlopen~ and This is also what settled the interpreter question: if the agent can ~dlopen~ and
call anything in 16ms, then even "eval this expression against live game state" call anything in under 20ms, then even "eval this expression against live game
can be a compiled ~.so~, and no interpreter is needed inside the game process. state" is a compiled ~.so~, and no interpreter is needed inside the game process.
That is the route ~C-x C-e~ actually takes.
** Why LLVM IR as text ** Why LLVM IR as text
| | text ~.ll~ → ~clang~ | libLLVM bindings | emit C | | | text ~.ll~ → ~clang~ | libLLVM bindings | emit C |
@ -580,32 +602,36 @@ The only column text loses is the JIT one, and the measurement above shows the
loss is ~13ms — below perception. ORC remains addable later behind the same typed loss is ~13ms — below perception. ORC remains addable later behind the same typed
IR without touching the language, but nothing currently argues for it. IR without touching the language, but nothing currently argues for it.
** The interpreter cannot run sand ** The interpreter, and why there is not one
Do not plan around it. 200 × 280 = 56,000 cells, scanned by ~game-update~ and An interpreter could never have run sand, and that was the first half of the
again by ~game-draw~ — ~112,000 interpreted cell-visits per frame against an argument. 200 × 280 = 56,000 cells, scanned by ~game-update~ and again by
8.3ms budget at 120fps. At an optimistic 100ns per visit (environment ~game-draw~ — ~112,000 interpreted cell-visits per frame against an 8.3ms budget
allocation, argument binding, two index computations, a compare) that is 11ms at 120fps. At an optimistic 100ns per visit (environment allocation, argument
before ~settle~, ~paint~, or a single raylib call. Expect 2030fps. binding, two index computations, a compare) that is 11ms before ~settle~,
~paint~, or a single raylib call. Expect 2030fps. Milestone 4's interactive
acceptance test was always going to run on the compiled dev build.
This is an estimate, not a measurement, which is why *milestone 2 exits with a *** Settled: the compiled path is the only backend
measured interpreter throughput number* — before milestone 4 depends on it. This was open decision #7 and it is closed. Compiled redefinition measured at
Milestone 4's interactive acceptance test runs on the compiled dev build; the ~19ms is perceptually instant for expression eval too, so the one thing an
interpreter is not in that loop. interpreter was still wanted for went away; the instrumentation-based step
debugger that wanted it is cut (see Tooling); and milestone 3 did not need it as
an oracle either, because the acceptance table is hand-written and the table /is/
the oracle. What is bought by dropping it is the standing obligation: two
evaluators must agree on observable behaviour forever, and every divergence is a
bug that reproduces in only one of them. BUILT.md's "Why there is no interpreter"
records the decision; ~lib/expand.ml~ states it at the top of the file, because
macros are where the absence stopped being free — a macro has to run at compile
time and there is nothing to interpret it with, so the compiler compiles it into
a shared object and loads it with ~dlopen~ into its own process.
*** Open: does the interpreter survive milestone 3? Consequences applied elsewhere in this document: milestone 2's "interpreted calls
Now that compiled redefinition is measured at 16ms, the case for a /permanent/ per second" exit criterion is dropped, and the host ABI moved onto the critical
interpreter is weaker than it looked. 16ms is perceptually instant for expression path in its place.
eval too, and one backend removes a standing obligation — two backends must agree
on observable behaviour forever, and every divergence is a bug that reproduces in
only one of them.
Against dropping it: the interpreter is clearly right for milestone 2 (far less The paths that remain share the frontend and the typed IR and must agree on
work than an LLVM backend, better error messages, no linking), and the observable behaviour. That agreement is what the acceptance programs test, and
instrumentation-based step debugger wants it. Decide at milestone 3 exit on for the two code generators it is tested byte for byte.
measured numbers, not now.
All three paths share the frontend and the typed IR and must agree on observable
behaviour. That agreement is what the acceptance programs test.
- Non-local exit lowered *explicitly* (result propagation + branch targets), not - Non-local exit lowered *explicitly* (result propagation + branch targets), not
via platform unwinding. Same on both targets, no dependency on the WASM via platform unwinding. Same on both targets, no dependency on the WASM
@ -635,7 +661,7 @@ Deliberately different.
| | Dev | Release | | | Dev | Release |
|---------+---------------------------+------------| |---------+---------------------------+------------|
| Backend | interpreter /and/ LLVM | LLVM/clang | | Backend | LLVM, or ~--x86~ | LLVM/clang |
| Calls | indirection cells | direct | | Calls | indirection cells | direct |
| Code | never freed | static | | Code | never freed | static |
| Frames | shadow stack | none | | Frames | shadow stack | none |
@ -698,8 +724,11 @@ trap handling, frame unwinding), duplicating an enormous existing project.
Neither covers the other's column, so this is not a choice between them. Neither covers the other's column, so this is not a choice between them.
*DAP is nearly free.* No debug adapter is written: emit DWARF from the LLVM *DAP is nearly free.* No debug adapter is written: emit DWARF from the backend
backend and point ~lldb-dap~ at the binary; dape speaks to that. lldb and gdb and point ~lldb-dap~ at the binary; dape speaks to that. Both code generators do
— the hand-written one writes its compile unit, subprograms and line table out as
bytes, since ~.loc~ cannot work against a file whose instructions are ~.byte~
blobs, and what it does not describe is locals and types. lldb and gdb
both ship DAP interfaces already. both ship DAP interfaces already.
This is where /no object headers/ pays off a second time. Flan structs *are* C This is where /no object headers/ pays off a second time. Flan structs *are* C
@ -756,17 +785,21 @@ building the whole live environment at once.
1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any 1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any
code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators, code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators,
~Ptr~, explicit ~clone~; the six restart cases. /Done./ ~Ptr~, explicit ~clone~; the six restart cases. /Done./
2. *Run calc-me.flan on the interpreter.* Reader, typed IR, checker, 2. *Run calc-me.flan.* Reader, typed IR, checker, and a backend that can carry
tree-walking backend. /Exit criterion includes a measured throughput number/ the program end to end. The exit criterion was once a measured interpreter
— interpreted calls per second on a tight loop — because milestone 4's frame throughput number; with no interpreter that criterion is gone and the narrow
budget depends on it (see Compilation). Packages, structs, ~(Ptr T)~ + ~addr~, byte slices, host ABI took its place on the critical path (see Compilation). Packages,
structs, ~(Ptr T)~ + ~addr~, byte slices,
~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~ ~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~
+ ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~, + ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~,
no generics, no user macros, no FFI, no window. Headless, so the acceptance no generics, no user macros, no FFI, no window. Headless, so the acceptance
test is a table of expression/result pairs. test is a table of expression/result pairs.
3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI. 3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI.
Both backends, one test table, one narrow host ABI (argv, stdout, exit). This Both targets, one test table, one narrow host ABI (argv, stdout, exit). This
is where the second target gets proven — while there is almost nothing to port. is where the second target gets proven — while there is almost nothing to port.
The hand-written x86-64 code generator is not on this path: it arrived later,
as a second route to the same behaviour rather than a milestone of its own,
and is held to the LLVM backend's output byte for byte (see Compilation).
4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to 4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to
raylib including keyword→enum coercion. Acceptance test twice: headless (N raylib including keyword→enum coercion. Acceptance test twice: headless (N
frames, hash the grid — runnable in CI on both targets) and interactive at frames, hash the grid — runnable in CI on both targets) and interactive at
@ -776,8 +809,8 @@ building the whole live environment at once.
special forms in the compiler. special forms in the compiler.
6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and 6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and
restarts* against spec-conditions.md, with dedicated tests per numbered case. restarts* against spec-conditions.md, with dedicated tests per numbered case.
7. *Hot reload*free in the interpreter, indirection cells for compiled dev 7. *Hot reload*indirection cells in dev builds, with signature generations
builds, with signature generations and stale-caller warnings, plus the and stale-caller warnings, plus the
remaining compatibility limits written down and enforced: struct layout remaining compatibility limits written down and enforced: struct layout
changes, live callbacks held by C, captured environments. changes, live callbacks held by C, captured environments.
8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor 8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor
@ -823,8 +856,8 @@ monomorphisation, no restarts and no reload.
None of these block milestone 2. The milestone each one must be answered by is None of these block milestone 2. The milestone each one must be answered by is
marked. marked.
1. *Host language: OCaml or Rust*the only thing blocking the scaffold. See 1. *Host language: OCaml or Rust*/settled: OCaml,/ and the compiler has been
Host language. /Milestone 2./ written in it since. See Host language for what the choice turned on.
2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately 2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately
non-hygienic expansion, no local macros until a concrete use case appears. non-hygienic expansion, no local macros until a concrete use case appears.
3. Borrow checking and escaping frame-arena values. /Deferred; revisit after 3. Borrow checking and escaping frame-arena values. /Deferred; revisit after
@ -855,7 +888,8 @@ marked.
a ~defvar~. Each needs an answer of the form "rejected", "accepted with a a ~defvar~. Each needs an answer of the form "rejected", "accepted with a
migration", or "accepted and the old code keeps running". migration", or "accepted and the old code keeps running".
7. Does the interpreter survive milestone 3, or is the compiled path the only 7. Does the interpreter survive milestone 3, or is the compiled path the only
backend? /Milestone 3, on measured numbers./ See Compilation. backend? /Settled: the compiled path is the only one, and there is no
interpreter./ See Compilation, and BUILT.md's "Why there is no interpreter".
8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the 8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the
compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The
CL-vs-Clojure truthiness question is moot under static typing. CL-vs-Clojure truthiness question is moot under static typing.
@ -875,8 +909,10 @@ marked.
- Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~, - Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~,
never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a
batch at a frame boundary. See Compilation. batch at a frame boundary. See Compilation.
- Dev backend → interpreter for milestone 2 certainly. Whether it /survives/ - Dev backend → compiled, and only compiled. The interpreter that milestone 2
milestone 3 is open, not settled — see Compilation. was going to be written against was never needed and does not exist: expression
eval is a compiled shared object like everything else, and a macro is the case
that made the absence load-bearing rather than merely tidy. See Compilation.
- ~set~ on places → a fixed list of assignable forms, not ~setf~. - ~set~ on places → a fixed list of assignable forms, not ~setf~.
- Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and - Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and
~return~; ~loop~/~recur~ only if it later earns its place. It did: both are ~return~; ~loop~/~recur~ only if it later earns its place. It did: both are