From 00163bcf34bbf039823421c3b81c9411ecfe0b05 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 14 Sep 2026 07:27:36 +0700 Subject: [PATCH] There is no interpreter, and the plan stops promising one The Compilation section was written around a permanent tree-walking backend for expression eval. Open decision #7 closed the other way and BUILT.md records it: compiling is the only way a form is ever run. The diagram, the milestone-2 exit criterion, milestone 7's "free in the interpreter", the dev/release table and the decision itself all said otherwise, and lib/expand.ml states the settled answer at the top of the file. Also here, because the same section was the place they were missing: the hand-written x86-64 code generator, which is a second route from the typed IR to the same observable behaviour rather than a second semantics; DWARF from both code generators rather than from LLVM alone; OCaml as a settled host language; and the map-new example, which has named its key and value types since braces stopped being a type spelling. --- plan.org | 158 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 97 insertions(+), 61 deletions(-) diff --git a/plan.org b/plan.org index 895457f..d0aa2af 100644 --- a/plan.org +++ b/plan.org @@ -101,7 +101,7 @@ world. - Map keys initially use compiler-provided structural equality and hashing for integers, enums, strings, fixed arrays and value structs; pointers, slices and owning containers are excluded. A map is homogeneous, and empty construction - is type-directed: ~(defvar enemies (Map string Enemy) (map-new))~. ~get~ + names its types: ~(let [enemies (map-new string Enemy)] ...)~. ~get~ returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See spec-memory.md for the deferred move-aware operations. - Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~. @@ -401,8 +401,8 @@ primitives, and are never bootstrapped away. The LLVM question does not bear on this: the release backend emits LLVM IR *as text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or Rust buy nothing here. What the choice actually turns on is that milestones 2–5 -are a reader, a typed IR, a checker and a tree-walking interpreter — variants and -exhaustive pattern matching, which is the one domain where OCaml is not a +are a reader, a typed IR, a checker and the code generators behind it — variants +and exhaustive pattern matching, which is the one domain where OCaml is not a preference but a clear win. There is also a menhir lexer/parser already started in ~old-ocaml/~. @@ -415,7 +415,7 @@ build sequence. For a game language it buys dogfooding at the price of a second compiler to maintain forever. Choose as if the host language is permanent. * Milestone-2 primitives -The interpreter provides these; everything else is written in Flan. Keeping the +The runtime provides these; everything else is written in Flan. Keeping the list short is the whole strategy — it is what makes the LLVM backend and the wasm32 target cheap, because a primitive is the only thing implemented twice. @@ -477,33 +477,54 @@ is Clojure's ~ns~ form: no path that must mirror the directory, no root-directory aliases. * Compilation -*Two backends and three paths.* The split is not dev-vs-release; it is -/does this code have a frame budget/. +*One evaluator and three paths.* The split is not dev-vs-release; it is +/does this code have a frame budget/. There is no interpreter: open decision #7 +is settled the other way from how this section was first written, and BUILT.md's +"Why there is no interpreter" carries the reasoning. Compiling is the only way a +form is ever run, so there is no second evaluator that could disagree with the +first about what a program means. #+begin_src -expression eval: flan → typed IR → interpreter ~1ms -dev redefinition: flan → typed IR → .ll → llc → ld -shared → dlopen → cell store - ~16ms (MEASURED) +expression eval: flan → typed IR → .ll → llc → ld -shared → dlopen → call + ~19ms (MEASURED) +dev redefinition: the same path, ending in a cell store rather than a call release build: flan → typed IR → .ll → clang --target={native,wasm32} #+end_src *Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate, -because the whole point of the live loop is that you see the result. 16ms is one -frame at 60fps and under the ~50ms threshold where a response stops feeling -instantaneous. The rule that buys it: *never invoke the ~clang~ driver on the dev -path.* +because the whole point of the live loop is that you see the result. 19ms is +around one frame at 60fps and under the ~50ms threshold where a response stops +feeling instantaneous. The rule that buys it: *never invoke the ~clang~ driver on +the dev path.* -*Expression eval* — ~C-c C-e~, calling a function, inspecting a var, running a -test — goes to the tree-walking interpreter. Sub-millisecond, no subprocess. This -is the permanent REPL backend, not a milestone-2 scaffold. +*Expression eval* — ~C-x C-e~, calling a function, inspecting a var, running a +test — is compiled like everything else, into its own shared object, which is +then loaded and called. What made an interpreter look necessary was the +assumption that this had to be sub-millisecond; the measurement below is that +the compiled route is already inside the threshold, and the one thing an +interpreter would have bought is an oracle the hand-written acceptance table +supplies instead. -*Dev redefinition* — ~C-c C-c~ on a function inside a running game — cannot use -the interpreter, because that code has an 8ms frame budget. It recompiles the one -function, links it, and does the atomic indirection-cell store. This is what the -Hot reload section has always described; the interpreter does not replace it. +*Dev redefinition* — ~C-c C-c~ on a function inside a running game — has an 8ms +frame budget to respect. It recompiles the one function, links it, and does the +atomic indirection-cell store. This is what the Hot reload section has always +described, and it is the same machinery expression eval uses, one step further +on. *Release* is whole-program AOT with direct calls and no cells. +*A second code generator, not a second evaluator.* ~lib/x86.ml~ emits x86-64 +machine code directly and is selected with ~--x86~; it exists because ~llc~ is +most of the 19ms above. It is a different route from the same typed IR to the +same observable behaviour, not a different semantics, and what holds it to that +is ~spike/x86/survey.sh~: every program in the corpus is built both ways and +byte-compared on stdout, stderr and exit status. At the time of writing that is +103 MATCH, 0 DIFFER, 0 refused by name. It handles conditions, bounds checks, +indirection cells, redefinition modules and DWARF line tables; what it does not +have, and must not grow, is an aggregate classifier — an ~--x86~ host therefore +takes ~--x86~ modules and an LLVM host takes LLVM ones, and ~lib/build.ml~ +refuses the crossed pair by name. + ** Measured redefinition latency Single function, x86-64, clang 20.1.8, 20 iterations each: @@ -563,9 +584,10 @@ This is why *the dev runtime is multithreaded* — it needs the reload thread. T is settled, and is independent of whether the /language/ exposes threads, which is still open decision #4. -It also bears on whether the interpreter survives: if the agent can ~dlopen~ and -call anything in 16ms, then even "eval this expression against live game state" -can be a compiled ~.so~, and no interpreter is needed inside the game process. +This is also what settled the interpreter question: if the agent can ~dlopen~ and +call anything in under 20ms, then even "eval this expression against live game +state" is a compiled ~.so~, and no interpreter is needed inside the game process. +That is the route ~C-x C-e~ actually takes. ** Why LLVM IR as text | | text ~.ll~ → ~clang~ | libLLVM bindings | emit C | @@ -580,32 +602,36 @@ The only column text loses is the JIT one, and the measurement above shows the loss is ~13ms — below perception. ORC remains addable later behind the same typed IR without touching the language, but nothing currently argues for it. -** The interpreter cannot run sand -Do not plan around it. 200 × 280 = 56,000 cells, scanned by ~game-update~ and -again by ~game-draw~ — ~112,000 interpreted cell-visits per frame against an -8.3ms budget at 120fps. At an optimistic 100ns per visit (environment -allocation, argument binding, two index computations, a compare) that is 11ms -before ~settle~, ~paint~, or a single raylib call. Expect 20–30fps. +** The interpreter, and why there is not one +An interpreter could never have run sand, and that was the first half of the +argument. 200 × 280 = 56,000 cells, scanned by ~game-update~ and again by +~game-draw~ — ~112,000 interpreted cell-visits per frame against an 8.3ms budget +at 120fps. At an optimistic 100ns per visit (environment allocation, argument +binding, two index computations, a compare) that is 11ms before ~settle~, +~paint~, or a single raylib call. Expect 20–30fps. Milestone 4's interactive +acceptance test was always going to run on the compiled dev build. -This is an estimate, not a measurement, which is why *milestone 2 exits with a -measured interpreter throughput number* — before milestone 4 depends on it. -Milestone 4's interactive acceptance test runs on the compiled dev build; the -interpreter is not in that loop. +*** Settled: the compiled path is the only backend +This was open decision #7 and it is closed. Compiled redefinition measured at +~19ms is perceptually instant for expression eval too, so the one thing an +interpreter was still wanted for went away; the instrumentation-based step +debugger that wanted it is cut (see Tooling); and milestone 3 did not need it as +an oracle either, because the acceptance table is hand-written and the table /is/ +the oracle. What is bought by dropping it is the standing obligation: two +evaluators must agree on observable behaviour forever, and every divergence is a +bug that reproduces in only one of them. BUILT.md's "Why there is no interpreter" +records the decision; ~lib/expand.ml~ states it at the top of the file, because +macros are where the absence stopped being free — a macro has to run at compile +time and there is nothing to interpret it with, so the compiler compiles it into +a shared object and loads it with ~dlopen~ into its own process. -*** Open: does the interpreter survive milestone 3? -Now that compiled redefinition is measured at 16ms, the case for a /permanent/ -interpreter is weaker than it looked. 16ms is perceptually instant for expression -eval too, and one backend removes a standing obligation — two backends must agree -on observable behaviour forever, and every divergence is a bug that reproduces in -only one of them. +Consequences applied elsewhere in this document: milestone 2's "interpreted calls +per second" exit criterion is dropped, and the host ABI moved onto the critical +path in its place. -Against dropping it: the interpreter is clearly right for milestone 2 (far less -work than an LLVM backend, better error messages, no linking), and the -instrumentation-based step debugger wants it. Decide at milestone 3 exit on -measured numbers, not now. - -All three paths share the frontend and the typed IR and must agree on observable -behaviour. That agreement is what the acceptance programs test. +The paths that remain share the frontend and the typed IR and must agree on +observable behaviour. That agreement is what the acceptance programs test, and +for the two code generators it is tested byte for byte. - Non-local exit lowered *explicitly* (result propagation + branch targets), not via platform unwinding. Same on both targets, no dependency on the WASM @@ -635,7 +661,7 @@ Deliberately different. | | Dev | Release | |---------+---------------------------+------------| -| Backend | interpreter /and/ LLVM | LLVM/clang | +| Backend | LLVM, or ~--x86~ | LLVM/clang | | Calls | indirection cells | direct | | Code | never freed | static | | Frames | shadow stack | none | @@ -698,8 +724,11 @@ trap handling, frame unwinding), duplicating an enormous existing project. Neither covers the other's column, so this is not a choice between them. -*DAP is nearly free.* No debug adapter is written: emit DWARF from the LLVM -backend and point ~lldb-dap~ at the binary; dape speaks to that. lldb and gdb +*DAP is nearly free.* No debug adapter is written: emit DWARF from the backend +and point ~lldb-dap~ at the binary; dape speaks to that. Both code generators do +— the hand-written one writes its compile unit, subprograms and line table out as +bytes, since ~.loc~ cannot work against a file whose instructions are ~.byte~ +blobs, and what it does not describe is locals and types. lldb and gdb both ship DAP interfaces already. This is where /no object headers/ pays off a second time. Flan structs *are* C @@ -756,17 +785,21 @@ building the whole live environment at once. 1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators, ~Ptr~, explicit ~clone~; the six restart cases. /Done./ -2. *Run calc-me.flan on the interpreter.* Reader, typed IR, checker, - tree-walking backend. /Exit criterion includes a measured throughput number/ - — interpreted calls per second on a tight loop — because milestone 4's frame - budget depends on it (see Compilation). Packages, structs, ~(Ptr T)~ + ~addr~, byte slices, +2. *Run calc-me.flan.* Reader, typed IR, checker, and a backend that can carry + the program end to end. The exit criterion was once a measured interpreter + throughput number; with no interpreter that criterion is gone and the narrow + host ABI took its place on the critical path (see Compilation). Packages, + structs, ~(Ptr T)~ + ~addr~, byte slices, ~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~ + ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~, no generics, no user macros, no FFI, no window. Headless, so the acceptance test is a table of expression/result pairs. 3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI. - Both backends, one test table, one narrow host ABI (argv, stdout, exit). This + Both targets, one test table, one narrow host ABI (argv, stdout, exit). This is where the second target gets proven — while there is almost nothing to port. + The hand-written x86-64 code generator is not on this path: it arrived later, + as a second route to the same behaviour rather than a milestone of its own, + and is held to the LLVM backend's output byte for byte (see Compilation). 4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to raylib including keyword→enum coercion. Acceptance test twice: headless (N frames, hash the grid — runnable in CI on both targets) and interactive at @@ -776,8 +809,8 @@ building the whole live environment at once. special forms in the compiler. 6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and restarts* against spec-conditions.md, with dedicated tests per numbered case. -7. *Hot reload* — free in the interpreter, indirection cells for compiled dev - builds, with signature generations and stale-caller warnings, plus the +7. *Hot reload* — indirection cells in dev builds, with signature generations + and stale-caller warnings, plus the remaining compatibility limits written down and enforced: struct layout changes, live callbacks held by C, captured environments. 8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor @@ -823,8 +856,8 @@ monomorphisation, no restarts and no reload. None of these block milestone 2. The milestone each one must be answered by is marked. -1. *Host language: OCaml or Rust* — the only thing blocking the scaffold. See - Host language. /Milestone 2./ +1. *Host language: OCaml or Rust* — /settled: OCaml,/ and the compiler has been + written in it since. See Host language for what the choice turned on. 2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately non-hygienic expansion, no local macros until a concrete use case appears. 3. Borrow checking and escaping frame-arena values. /Deferred; revisit after @@ -855,7 +888,8 @@ marked. a ~defvar~. Each needs an answer of the form "rejected", "accepted with a migration", or "accepted and the old code keeps running". 7. Does the interpreter survive milestone 3, or is the compiled path the only - backend? /Milestone 3, on measured numbers./ See Compilation. + backend? /Settled: the compiled path is the only one, and there is no + interpreter./ See Compilation, and BUILT.md's "Why there is no interpreter". 8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The CL-vs-Clojure truthiness question is moot under static typing. @@ -875,8 +909,10 @@ marked. - Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~, never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a batch at a frame boundary. See Compilation. -- Dev backend → interpreter for milestone 2 certainly. Whether it /survives/ - milestone 3 is open, not settled — see Compilation. +- Dev backend → compiled, and only compiled. The interpreter that milestone 2 + was going to be written against was never needed and does not exist: expression + eval is a compiled shared object like everything else, and a macro is the case + that made the absence load-bearing rather than merely tidy. See Compilation. - ~set~ on places → a fixed list of assignable forms, not ~setf~. - Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and ~return~; ~loop~/~recur~ only if it later earns its place. It did: both are