JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

There is no interpreter, and the plan stops promising one

Browse Source 
The Compilation section was written around a permanent tree-walking backend for
expression eval. Open decision #7 closed the other way and BUILT.md records it:
compiling is the only way a form is ever run. The diagram, the milestone-2 exit
criterion, milestone 7's "free in the interpreter", the dev/release table and
the decision itself all said otherwise, and lib/expand.ml states the settled
answer at the top of the file.

Also here, because the same section was the place they were missing: the
hand-written x86-64 code generator, which is a second route from the typed IR to
the same observable behaviour rather than a second semantics; DWARF from both
code generators rather than from LLVM alone; OCaml as a settled host language;
and the map-new example, which has named its key and value types since braces
stopped being a type spelling.
This commit is contained in:
Joseph Ferano 2026-09-14 07:27:36 +07:00
parent be39f32cb6
commit 00163bcf34
1 changed files with 97 additions and 61 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

158
plan.org
View File

@ -101,7 +101,7 @@ world.
- Map keys initially use compiler-provided structural equality and hashing for
integers, enums, strings, fixed arrays and value structs; pointers, slices and
owning containers are excluded. A map is homogeneous, and empty construction
is type-directed: ~(defvar enemies (Map string Enemy) (map-new))~. ~get~
names its types: ~(let [enemies (map-new string Enemy)] ...)~. ~get~
returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See
spec-memory.md for the deferred move-aware operations.
- Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~.
@ -401,8 +401,8 @@ primitives, and are never bootstrapped away.
The LLVM question does not bear on this: the release backend emits LLVM IR *as
text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or
Rust buy nothing here. What the choice actually turns on is that milestones 2–5
are a reader, a typed IR, a checker and a tree-walking interpreter — variants and
exhaustive pattern matching, which is the one domain where OCaml is not a
are a reader, a typed IR, a checker and the code generators behind it — variants
and exhaustive pattern matching, which is the one domain where OCaml is not a
preference but a clear win. There is also a menhir lexer/parser already started
in ~old-ocaml/~.
@ -415,7 +415,7 @@ build sequence. For a game language it buys dogfooding at the price of a second
compiler to maintain forever. Choose as if the host language is permanent.
* Milestone-2 primitives
The interpreter provides these; everything else is written in Flan. Keeping the
The runtime provides these; everything else is written in Flan. Keeping the
list short is the whole strategy — it is what makes the LLVM backend and the
wasm32 target cheap, because a primitive is the only thing implemented twice.
@ -477,33 +477,54 @@ is Clojure's ~ns~ form: no path that must mirror the directory, no
root-directory aliases.
* Compilation
*Two backends and three paths.* The split is not dev-vs-release; it is
/does this code have a frame budget/.
*One evaluator and three paths.* The split is not dev-vs-release; it is
/does this code have a frame budget/. There is no interpreter: open decision #7
is settled the other way from how this section was first written, and BUILT.md's
"Why there is no interpreter" carries the reasoning. Compiling is the only way a
form is ever run, so there is no second evaluator that could disagree with the
first about what a program means.
#+begin_src
expression eval: flan → typed IR → interpreter ~1ms
dev redefinition: flan → typed IR → .ll → llc → ld -shared → dlopen → cell store
~16ms (MEASURED)
expression eval: flan → typed IR → .ll → llc → ld -shared → dlopen → call
~19ms (MEASURED)
dev redefinition: the same path, ending in a cell store rather than a call
release build: flan → typed IR → .ll → clang --target={native,wasm32}
#+end_src
*Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate,
because the whole point of the live loop is that you see the result. 16ms is one
frame at 60fps and under the ~50ms threshold where a response stops feeling
instantaneous. The rule that buys it: *never invoke the ~clang~ driver on the dev
path.*
because the whole point of the live loop is that you see the result. 19ms is
around one frame at 60fps and under the ~50ms threshold where a response stops
feeling instantaneous. The rule that buys it: *never invoke the ~clang~ driver on
the dev path.*
*Expression eval* — ~C-c C-e~, calling a function, inspecting a var, running a
test — goes to the tree-walking interpreter. Sub-millisecond, no subprocess. This
is the permanent REPL backend, not a milestone-2 scaffold.
*Expression eval* — ~C-x C-e~, calling a function, inspecting a var, running a
test — is compiled like everything else, into its own shared object, which is
then loaded and called. What made an interpreter look necessary was the
assumption that this had to be sub-millisecond; the measurement below is that
the compiled route is already inside the threshold, and the one thing an
interpreter would have bought is an oracle the hand-written acceptance table
supplies instead.
*Dev redefinition* — ~C-c C-c~ on a function inside a running game — cannot use
the interpreter, because that code has an 8ms frame budget. It recompiles the one
function, links it, and does the atomic indirection-cell store. This is what the
Hot reload section has always described; the interpreter does not replace it.
*Dev redefinition* — ~C-c C-c~ on a function inside a running game — has an 8ms
frame budget to respect. It recompiles the one function, links it, and does the
atomic indirection-cell store. This is what the Hot reload section has always
described, and it is the same machinery expression eval uses, one step further
on.
*Release* is whole-program AOT with direct calls and no cells.
*A second code generator, not a second evaluator.* ~lib/x86.ml~ emits x86-64
machine code directly and is selected with ~--x86~; it exists because ~llc~ is
most of the 19ms above. It is a different route from the same typed IR to the
same observable behaviour, not a different semantics, and what holds it to that
is ~spike/x86/survey.sh~: every program in the corpus is built both ways and
byte-compared on stdout, stderr and exit status. At the time of writing that is
103 MATCH, 0 DIFFER, 0 refused by name. It handles conditions, bounds checks,
indirection cells, redefinition modules and DWARF line tables; what it does not
have, and must not grow, is an aggregate classifier — an ~--x86~ host therefore
takes ~--x86~ modules and an LLVM host takes LLVM ones, and ~lib/build.ml~
refuses the crossed pair by name.
** Measured redefinition latency
Single function, x86-64, clang 20.1.8, 20 iterations each:
@ -563,9 +584,10 @@ This is why *the dev runtime is multithreaded* — it needs the reload thread. T
is settled, and is independent of whether the /language/ exposes threads, which
is still open decision #4.
It also bears on whether the interpreter survives: if the agent can ~dlopen~ and
call anything in 16ms, then even "eval this expression against live game state"
can be a compiled ~.so~, and no interpreter is needed inside the game process.
This is also what settled the interpreter question: if the agent can ~dlopen~ and
call anything in under 20ms, then even "eval this expression against live game
state" is a compiled ~.so~, and no interpreter is needed inside the game process.
That is the route ~C-x C-e~ actually takes.
** Why LLVM IR as text
| | text ~.ll~ → ~clang~ | libLLVM bindings | emit C |
@ -580,32 +602,36 @@ The only column text loses is the JIT one, and the measurement above shows the
loss is ~13ms — below perception. ORC remains addable later behind the same typed
IR without touching the language, but nothing currently argues for it.
** The interpreter cannot run sand
Do not plan around it. 200 × 280 = 56,000 cells, scanned by ~game-update~ and
again by ~game-draw~ — ~112,000 interpreted cell-visits per frame against an
8.3ms budget at 120fps. At an optimistic 100ns per visit (environment
allocation, argument binding, two index computations, a compare) that is 11ms
before ~settle~, ~paint~, or a single raylib call. Expect 20–30fps.
** The interpreter, and why there is not one
An interpreter could never have run sand, and that was the first half of the
argument. 200 × 280 = 56,000 cells, scanned by ~game-update~ and again by
~game-draw~ — ~112,000 interpreted cell-visits per frame against an 8.3ms budget
at 120fps. At an optimistic 100ns per visit (environment allocation, argument
binding, two index computations, a compare) that is 11ms before ~settle~,
~paint~, or a single raylib call. Expect 20–30fps. Milestone 4's interactive
acceptance test was always going to run on the compiled dev build.
This is an estimate, not a measurement, which is why *milestone 2 exits with a
measured interpreter throughput number* — before milestone 4 depends on it.
Milestone 4's interactive acceptance test runs on the compiled dev build; the
interpreter is not in that loop.
*** Settled: the compiled path is the only backend
This was open decision #7 and it is closed. Compiled redefinition measured at
~19ms is perceptually instant for expression eval too, so the one thing an
interpreter was still wanted for went away; the instrumentation-based step
debugger that wanted it is cut (see Tooling); and milestone 3 did not need it as
an oracle either, because the acceptance table is hand-written and the table /is/
the oracle. What is bought by dropping it is the standing obligation: two
evaluators must agree on observable behaviour forever, and every divergence is a
bug that reproduces in only one of them. BUILT.md's "Why there is no interpreter"
records the decision; ~lib/expand.ml~ states it at the top of the file, because
macros are where the absence stopped being free — a macro has to run at compile
time and there is nothing to interpret it with, so the compiler compiles it into
a shared object and loads it with ~dlopen~ into its own process.
*** Open: does the interpreter survive milestone 3?
Now that compiled redefinition is measured at 16ms, the case for a /permanent/
interpreter is weaker than it looked. 16ms is perceptually instant for expression
eval too, and one backend removes a standing obligation — two backends must agree
on observable behaviour forever, and every divergence is a bug that reproduces in
only one of them.
Consequences applied elsewhere in this document: milestone 2's "interpreted calls
per second" exit criterion is dropped, and the host ABI moved onto the critical
path in its place.
Against dropping it: the interpreter is clearly right for milestone 2 (far less
work than an LLVM backend, better error messages, no linking), and the
instrumentation-based step debugger wants it. Decide at milestone 3 exit on
measured numbers, not now.
All three paths share the frontend and the typed IR and must agree on observable
behaviour. That agreement is what the acceptance programs test.
The paths that remain share the frontend and the typed IR and must agree on
observable behaviour. That agreement is what the acceptance programs test, and
for the two code generators it is tested byte for byte.
- Non-local exit lowered *explicitly* (result propagation + branch targets), not
via platform unwinding. Same on both targets, no dependency on the WASM
@ -635,7 +661,7 @@ Deliberately different.
| | Dev | Release |
|---------+---------------------------+------------|
| Backend | interpreter /and/ LLVM | LLVM/clang |
| Backend | LLVM, or ~--x86~ | LLVM/clang |
| Calls | indirection cells | direct |
| Code | never freed | static |
| Frames | shadow stack | none |
@ -698,8 +724,11 @@ trap handling, frame unwinding), duplicating an enormous existing project.
Neither covers the other's column, so this is not a choice between them.
*DAP is nearly free.* No debug adapter is written: emit DWARF from the LLVM
backend and point ~lldb-dap~ at the binary; dape speaks to that. lldb and gdb
*DAP is nearly free.* No debug adapter is written: emit DWARF from the backend
and point ~lldb-dap~ at the binary; dape speaks to that. Both code generators do
— the hand-written one writes its compile unit, subprograms and line table out as
bytes, since ~.loc~ cannot work against a file whose instructions are ~.byte~
blobs, and what it does not describe is locals and types. lldb and gdb
both ship DAP interfaces already.
This is where /no object headers/ pays off a second time. Flan structs *are* C
@ -756,17 +785,21 @@ building the whole live environment at once.
1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any
code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators,
~Ptr~, explicit ~clone~; the six restart cases. /Done./
2. *Run calc-me.flan on the interpreter.* Reader, typed IR, checker,
tree-walking backend. /Exit criterion includes a measured throughput number/
— interpreted calls per second on a tight loop — because milestone 4's frame
budget depends on it (see Compilation). Packages, structs, ~(Ptr T)~ + ~addr~, byte slices,
2. *Run calc-me.flan.* Reader, typed IR, checker, and a backend that can carry
the program end to end. The exit criterion was once a measured interpreter
throughput number; with no interpreter that criterion is gone and the narrow
host ABI took its place on the critical path (see Compilation). Packages,
structs, ~(Ptr T)~ + ~addr~, byte slices,
~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~
+ ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~,
no generics, no user macros, no FFI, no window. Headless, so the acceptance
test is a table of expression/result pairs.
3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI.
Both backends, one test table, one narrow host ABI (argv, stdout, exit). This
Both targets, one test table, one narrow host ABI (argv, stdout, exit). This
is where the second target gets proven — while there is almost nothing to port.
The hand-written x86-64 code generator is not on this path: it arrived later,
as a second route to the same behaviour rather than a milestone of its own,
and is held to the LLVM backend's output byte for byte (see Compilation).
4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to
raylib including keyword→enum coercion. Acceptance test twice: headless (N
frames, hash the grid — runnable in CI on both targets) and interactive at
@ -776,8 +809,8 @@ building the whole live environment at once.
special forms in the compiler.
6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and
restarts* against spec-conditions.md, with dedicated tests per numbered case.
7. *Hot reload* — free in the interpreter, indirection cells for compiled dev
builds, with signature generations and stale-caller warnings, plus the
7. *Hot reload* — indirection cells in dev builds, with signature generations
and stale-caller warnings, plus the
remaining compatibility limits written down and enforced: struct layout
changes, live callbacks held by C, captured environments.
8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor
@ -823,8 +856,8 @@ monomorphisation, no restarts and no reload.
None of these block milestone 2. The milestone each one must be answered by is
marked.
1. *Host language: OCaml or Rust* — the only thing blocking the scaffold. See
Host language. /Milestone 2./
1. *Host language: OCaml or Rust* — /settled: OCaml,/ and the compiler has been
written in it since. See Host language for what the choice turned on.
2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately
non-hygienic expansion, no local macros until a concrete use case appears.
3. Borrow checking and escaping frame-arena values. /Deferred; revisit after
@ -855,7 +888,8 @@ marked.
a ~defvar~. Each needs an answer of the form "rejected", "accepted with a
migration", or "accepted and the old code keeps running".
7. Does the interpreter survive milestone 3, or is the compiled path the only
backend? /Milestone 3, on measured numbers./ See Compilation.
backend? /Settled: the compiled path is the only one, and there is no
interpreter./ See Compilation, and BUILT.md's "Why there is no interpreter".
8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the
compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The
CL-vs-Clojure truthiness question is moot under static typing.
@ -875,8 +909,10 @@ marked.
- Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~,
never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a
batch at a frame boundary. See Compilation.
- Dev backend → interpreter for milestone 2 certainly. Whether it /survives/
milestone 3 is open, not settled — see Compilation.
- Dev backend → compiled, and only compiled. The interpreter that milestone 2
was going to be written against was never needed and does not exist: expression
eval is a compiled shared object like everything else, and a macro is the case
that made the absence load-bearing rather than merely tidy. See Compilation.
- ~set~ on places → a fixed list of assignable forms, not ~setf~.
- Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and
~return~; ~loop~/~recur~ only if it later earns its place. It did: both are