There is no interpreter, and the plan stops promising one
The Compilation section was written around a permanent tree-walking backend for expression eval. Open decision #7 closed the other way and BUILT.md records it: compiling is the only way a form is ever run. The diagram, the milestone-2 exit criterion, milestone 7's "free in the interpreter", the dev/release table and the decision itself all said otherwise, and lib/expand.ml states the settled answer at the top of the file. Also here, because the same section was the place they were missing: the hand-written x86-64 code generator, which is a second route from the typed IR to the same observable behaviour rather than a second semantics; DWARF from both code generators rather than from LLVM alone; OCaml as a settled host language; and the map-new example, which has named its key and value types since braces stopped being a type spelling.
This commit is contained in:
parent
be39f32cb6
commit
00163bcf34
158
plan.org
158
plan.org
@ -101,7 +101,7 @@ world.
|
|||||||
- Map keys initially use compiler-provided structural equality and hashing for
|
- Map keys initially use compiler-provided structural equality and hashing for
|
||||||
integers, enums, strings, fixed arrays and value structs; pointers, slices and
|
integers, enums, strings, fixed arrays and value structs; pointers, slices and
|
||||||
owning containers are excluded. A map is homogeneous, and empty construction
|
owning containers are excluded. A map is homogeneous, and empty construction
|
||||||
is type-directed: ~(defvar enemies (Map string Enemy) (map-new))~. ~get~
|
names its types: ~(let [enemies (map-new string Enemy)] ...)~. ~get~
|
||||||
returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See
|
returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See
|
||||||
spec-memory.md for the deferred move-aware operations.
|
spec-memory.md for the deferred move-aware operations.
|
||||||
- Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~.
|
- Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~len~, ~update~.
|
||||||
@ -401,8 +401,8 @@ primitives, and are never bootstrapped away.
|
|||||||
The LLVM question does not bear on this: the release backend emits LLVM IR *as
|
The LLVM question does not bear on this: the release backend emits LLVM IR *as
|
||||||
text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or
|
text* and shells out to ~clang~, so no language needs LLVM bindings, and C++ or
|
||||||
Rust buy nothing here. What the choice actually turns on is that milestones 2–5
|
Rust buy nothing here. What the choice actually turns on is that milestones 2–5
|
||||||
are a reader, a typed IR, a checker and a tree-walking interpreter — variants and
|
are a reader, a typed IR, a checker and the code generators behind it — variants
|
||||||
exhaustive pattern matching, which is the one domain where OCaml is not a
|
and exhaustive pattern matching, which is the one domain where OCaml is not a
|
||||||
preference but a clear win. There is also a menhir lexer/parser already started
|
preference but a clear win. There is also a menhir lexer/parser already started
|
||||||
in ~old-ocaml/~.
|
in ~old-ocaml/~.
|
||||||
|
|
||||||
@ -415,7 +415,7 @@ build sequence. For a game language it buys dogfooding at the price of a second
|
|||||||
compiler to maintain forever. Choose as if the host language is permanent.
|
compiler to maintain forever. Choose as if the host language is permanent.
|
||||||
|
|
||||||
* Milestone-2 primitives
|
* Milestone-2 primitives
|
||||||
The interpreter provides these; everything else is written in Flan. Keeping the
|
The runtime provides these; everything else is written in Flan. Keeping the
|
||||||
list short is the whole strategy — it is what makes the LLVM backend and the
|
list short is the whole strategy — it is what makes the LLVM backend and the
|
||||||
wasm32 target cheap, because a primitive is the only thing implemented twice.
|
wasm32 target cheap, because a primitive is the only thing implemented twice.
|
||||||
|
|
||||||
@ -477,33 +477,54 @@ is Clojure's ~ns~ form: no path that must mirror the directory, no
|
|||||||
root-directory aliases.
|
root-directory aliases.
|
||||||
|
|
||||||
* Compilation
|
* Compilation
|
||||||
*Two backends and three paths.* The split is not dev-vs-release; it is
|
*One evaluator and three paths.* The split is not dev-vs-release; it is
|
||||||
/does this code have a frame budget/.
|
/does this code have a frame budget/. There is no interpreter: open decision #7
|
||||||
|
is settled the other way from how this section was first written, and BUILT.md's
|
||||||
|
"Why there is no interpreter" carries the reasoning. Compiling is the only way a
|
||||||
|
form is ever run, so there is no second evaluator that could disagree with the
|
||||||
|
first about what a program means.
|
||||||
|
|
||||||
#+begin_src
|
#+begin_src
|
||||||
expression eval: flan → typed IR → interpreter ~1ms
|
expression eval: flan → typed IR → .ll → llc → ld -shared → dlopen → call
|
||||||
dev redefinition: flan → typed IR → .ll → llc → ld -shared → dlopen → cell store
|
~19ms (MEASURED)
|
||||||
~16ms (MEASURED)
|
dev redefinition: the same path, ending in a cell store rather than a call
|
||||||
release build: flan → typed IR → .ll → clang --target={native,wasm32}
|
release build: flan → typed IR → .ll → clang --target={native,wasm32}
|
||||||
#+end_src
|
#+end_src
|
||||||
|
|
||||||
*Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate,
|
*Hard requirement: eval is immediate.* Not "fast enough for a build" — immediate,
|
||||||
because the whole point of the live loop is that you see the result. 16ms is one
|
because the whole point of the live loop is that you see the result. 19ms is
|
||||||
frame at 60fps and under the ~50ms threshold where a response stops feeling
|
around one frame at 60fps and under the ~50ms threshold where a response stops
|
||||||
instantaneous. The rule that buys it: *never invoke the ~clang~ driver on the dev
|
feeling instantaneous. The rule that buys it: *never invoke the ~clang~ driver on
|
||||||
path.*
|
the dev path.*
|
||||||
|
|
||||||
*Expression eval* — ~C-c C-e~, calling a function, inspecting a var, running a
|
*Expression eval* — ~C-x C-e~, calling a function, inspecting a var, running a
|
||||||
test — goes to the tree-walking interpreter. Sub-millisecond, no subprocess. This
|
test — is compiled like everything else, into its own shared object, which is
|
||||||
is the permanent REPL backend, not a milestone-2 scaffold.
|
then loaded and called. What made an interpreter look necessary was the
|
||||||
|
assumption that this had to be sub-millisecond; the measurement below is that
|
||||||
|
the compiled route is already inside the threshold, and the one thing an
|
||||||
|
interpreter would have bought is an oracle the hand-written acceptance table
|
||||||
|
supplies instead.
|
||||||
|
|
||||||
*Dev redefinition* — ~C-c C-c~ on a function inside a running game — cannot use
|
*Dev redefinition* — ~C-c C-c~ on a function inside a running game — has an 8ms
|
||||||
the interpreter, because that code has an 8ms frame budget. It recompiles the one
|
frame budget to respect. It recompiles the one function, links it, and does the
|
||||||
function, links it, and does the atomic indirection-cell store. This is what the
|
atomic indirection-cell store. This is what the Hot reload section has always
|
||||||
Hot reload section has always described; the interpreter does not replace it.
|
described, and it is the same machinery expression eval uses, one step further
|
||||||
|
on.
|
||||||
|
|
||||||
*Release* is whole-program AOT with direct calls and no cells.
|
*Release* is whole-program AOT with direct calls and no cells.
|
||||||
|
|
||||||
|
*A second code generator, not a second evaluator.* ~lib/x86.ml~ emits x86-64
|
||||||
|
machine code directly and is selected with ~--x86~; it exists because ~llc~ is
|
||||||
|
most of the 19ms above. It is a different route from the same typed IR to the
|
||||||
|
same observable behaviour, not a different semantics, and what holds it to that
|
||||||
|
is ~spike/x86/survey.sh~: every program in the corpus is built both ways and
|
||||||
|
byte-compared on stdout, stderr and exit status. At the time of writing that is
|
||||||
|
103 MATCH, 0 DIFFER, 0 refused by name. It handles conditions, bounds checks,
|
||||||
|
indirection cells, redefinition modules and DWARF line tables; what it does not
|
||||||
|
have, and must not grow, is an aggregate classifier — an ~--x86~ host therefore
|
||||||
|
takes ~--x86~ modules and an LLVM host takes LLVM ones, and ~lib/build.ml~
|
||||||
|
refuses the crossed pair by name.
|
||||||
|
|
||||||
** Measured redefinition latency
|
** Measured redefinition latency
|
||||||
Single function, x86-64, clang 20.1.8, 20 iterations each:
|
Single function, x86-64, clang 20.1.8, 20 iterations each:
|
||||||
|
|
||||||
@ -563,9 +584,10 @@ This is why *the dev runtime is multithreaded* — it needs the reload thread. T
|
|||||||
is settled, and is independent of whether the /language/ exposes threads, which
|
is settled, and is independent of whether the /language/ exposes threads, which
|
||||||
is still open decision #4.
|
is still open decision #4.
|
||||||
|
|
||||||
It also bears on whether the interpreter survives: if the agent can ~dlopen~ and
|
This is also what settled the interpreter question: if the agent can ~dlopen~ and
|
||||||
call anything in 16ms, then even "eval this expression against live game state"
|
call anything in under 20ms, then even "eval this expression against live game
|
||||||
can be a compiled ~.so~, and no interpreter is needed inside the game process.
|
state" is a compiled ~.so~, and no interpreter is needed inside the game process.
|
||||||
|
That is the route ~C-x C-e~ actually takes.
|
||||||
|
|
||||||
** Why LLVM IR as text
|
** Why LLVM IR as text
|
||||||
| | text ~.ll~ → ~clang~ | libLLVM bindings | emit C |
|
| | text ~.ll~ → ~clang~ | libLLVM bindings | emit C |
|
||||||
@ -580,32 +602,36 @@ The only column text loses is the JIT one, and the measurement above shows the
|
|||||||
loss is ~13ms — below perception. ORC remains addable later behind the same typed
|
loss is ~13ms — below perception. ORC remains addable later behind the same typed
|
||||||
IR without touching the language, but nothing currently argues for it.
|
IR without touching the language, but nothing currently argues for it.
|
||||||
|
|
||||||
** The interpreter cannot run sand
|
** The interpreter, and why there is not one
|
||||||
Do not plan around it. 200 × 280 = 56,000 cells, scanned by ~game-update~ and
|
An interpreter could never have run sand, and that was the first half of the
|
||||||
again by ~game-draw~ — ~112,000 interpreted cell-visits per frame against an
|
argument. 200 × 280 = 56,000 cells, scanned by ~game-update~ and again by
|
||||||
8.3ms budget at 120fps. At an optimistic 100ns per visit (environment
|
~game-draw~ — ~112,000 interpreted cell-visits per frame against an 8.3ms budget
|
||||||
allocation, argument binding, two index computations, a compare) that is 11ms
|
at 120fps. At an optimistic 100ns per visit (environment allocation, argument
|
||||||
before ~settle~, ~paint~, or a single raylib call. Expect 20–30fps.
|
binding, two index computations, a compare) that is 11ms before ~settle~,
|
||||||
|
~paint~, or a single raylib call. Expect 20–30fps. Milestone 4's interactive
|
||||||
|
acceptance test was always going to run on the compiled dev build.
|
||||||
|
|
||||||
This is an estimate, not a measurement, which is why *milestone 2 exits with a
|
*** Settled: the compiled path is the only backend
|
||||||
measured interpreter throughput number* — before milestone 4 depends on it.
|
This was open decision #7 and it is closed. Compiled redefinition measured at
|
||||||
Milestone 4's interactive acceptance test runs on the compiled dev build; the
|
~19ms is perceptually instant for expression eval too, so the one thing an
|
||||||
interpreter is not in that loop.
|
interpreter was still wanted for went away; the instrumentation-based step
|
||||||
|
debugger that wanted it is cut (see Tooling); and milestone 3 did not need it as
|
||||||
|
an oracle either, because the acceptance table is hand-written and the table /is/
|
||||||
|
the oracle. What is bought by dropping it is the standing obligation: two
|
||||||
|
evaluators must agree on observable behaviour forever, and every divergence is a
|
||||||
|
bug that reproduces in only one of them. BUILT.md's "Why there is no interpreter"
|
||||||
|
records the decision; ~lib/expand.ml~ states it at the top of the file, because
|
||||||
|
macros are where the absence stopped being free — a macro has to run at compile
|
||||||
|
time and there is nothing to interpret it with, so the compiler compiles it into
|
||||||
|
a shared object and loads it with ~dlopen~ into its own process.
|
||||||
|
|
||||||
*** Open: does the interpreter survive milestone 3?
|
Consequences applied elsewhere in this document: milestone 2's "interpreted calls
|
||||||
Now that compiled redefinition is measured at 16ms, the case for a /permanent/
|
per second" exit criterion is dropped, and the host ABI moved onto the critical
|
||||||
interpreter is weaker than it looked. 16ms is perceptually instant for expression
|
path in its place.
|
||||||
eval too, and one backend removes a standing obligation — two backends must agree
|
|
||||||
on observable behaviour forever, and every divergence is a bug that reproduces in
|
|
||||||
only one of them.
|
|
||||||
|
|
||||||
Against dropping it: the interpreter is clearly right for milestone 2 (far less
|
The paths that remain share the frontend and the typed IR and must agree on
|
||||||
work than an LLVM backend, better error messages, no linking), and the
|
observable behaviour. That agreement is what the acceptance programs test, and
|
||||||
instrumentation-based step debugger wants it. Decide at milestone 3 exit on
|
for the two code generators it is tested byte for byte.
|
||||||
measured numbers, not now.
|
|
||||||
|
|
||||||
All three paths share the frontend and the typed IR and must agree on observable
|
|
||||||
behaviour. That agreement is what the acceptance programs test.
|
|
||||||
|
|
||||||
- Non-local exit lowered *explicitly* (result propagation + branch targets), not
|
- Non-local exit lowered *explicitly* (result propagation + branch targets), not
|
||||||
via platform unwinding. Same on both targets, no dependency on the WASM
|
via platform unwinding. Same on both targets, no dependency on the WASM
|
||||||
@ -635,7 +661,7 @@ Deliberately different.
|
|||||||
|
|
||||||
| | Dev | Release |
|
| | Dev | Release |
|
||||||
|---------+---------------------------+------------|
|
|---------+---------------------------+------------|
|
||||||
| Backend | interpreter /and/ LLVM | LLVM/clang |
|
| Backend | LLVM, or ~--x86~ | LLVM/clang |
|
||||||
| Calls | indirection cells | direct |
|
| Calls | indirection cells | direct |
|
||||||
| Code | never freed | static |
|
| Code | never freed | static |
|
||||||
| Frames | shadow stack | none |
|
| Frames | shadow stack | none |
|
||||||
@ -698,8 +724,11 @@ trap handling, frame unwinding), duplicating an enormous existing project.
|
|||||||
|
|
||||||
Neither covers the other's column, so this is not a choice between them.
|
Neither covers the other's column, so this is not a choice between them.
|
||||||
|
|
||||||
*DAP is nearly free.* No debug adapter is written: emit DWARF from the LLVM
|
*DAP is nearly free.* No debug adapter is written: emit DWARF from the backend
|
||||||
backend and point ~lldb-dap~ at the binary; dape speaks to that. lldb and gdb
|
and point ~lldb-dap~ at the binary; dape speaks to that. Both code generators do
|
||||||
|
— the hand-written one writes its compile unit, subprograms and line table out as
|
||||||
|
bytes, since ~.loc~ cannot work against a file whose instructions are ~.byte~
|
||||||
|
blobs, and what it does not describe is locals and types. lldb and gdb
|
||||||
both ship DAP interfaces already.
|
both ship DAP interfaces already.
|
||||||
|
|
||||||
This is where /no object headers/ pays off a second time. Flan structs *are* C
|
This is where /no object headers/ pays off a second time. Flan structs *are* C
|
||||||
@ -756,17 +785,21 @@ building the whole live environment at once.
|
|||||||
1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any
|
1. *Freeze the model.* spec-memory.md and spec-conditions.md — done before any
|
||||||
code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators,
|
code. Fixed arrays, non-owning slices, move-only ~Vec~/~Map~, allocators,
|
||||||
~Ptr~, explicit ~clone~; the six restart cases. /Done./
|
~Ptr~, explicit ~clone~; the six restart cases. /Done./
|
||||||
2. *Run calc-me.flan on the interpreter.* Reader, typed IR, checker,
|
2. *Run calc-me.flan.* Reader, typed IR, checker, and a backend that can carry
|
||||||
tree-walking backend. /Exit criterion includes a measured throughput number/
|
the program end to end. The exit criterion was once a measured interpreter
|
||||||
— interpreted calls per second on a tight loop — because milestone 4's frame
|
throughput number; with no interpreter that criterion is gone and the narrow
|
||||||
budget depends on it (see Compilation). Packages, structs, ~(Ptr T)~ + ~addr~, byte slices,
|
host ABI took its place on the critical path (see Compilation). Packages,
|
||||||
|
structs, ~(Ptr T)~ + ~addr~, byte slices,
|
||||||
~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~
|
~at~/~len~, ~while~, ~set~ on the fixed place list, ~cond~, ~match~, ~Option~
|
||||||
+ ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~,
|
+ ~some~, ~i32~/~u8~/~f64~, recursion, argv, stdout. No allocator, no ~Vec~,
|
||||||
no generics, no user macros, no FFI, no window. Headless, so the acceptance
|
no generics, no user macros, no FFI, no window. Headless, so the acceptance
|
||||||
test is a table of expression/result pairs.
|
test is a table of expression/result pairs.
|
||||||
3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI.
|
3. *Emit LLVM IR and pass the same calc-me test AOT*, on native and wasm32 in CI.
|
||||||
Both backends, one test table, one narrow host ABI (argv, stdout, exit). This
|
Both targets, one test table, one narrow host ABI (argv, stdout, exit). This
|
||||||
is where the second target gets proven — while there is almost nothing to port.
|
is where the second target gets proven — while there is almost nothing to port.
|
||||||
|
The hand-written x86-64 code generator is not on this path: it arrived later,
|
||||||
|
as a second route to the same behaviour rather than a milestone of its own,
|
||||||
|
and is held to the LLVM backend's output byte for byte (see Compilation).
|
||||||
4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to
|
4. *Run sand.flan.* Fixed 2-D arrays, ~dotimes~, ~defer~, and typed FFI to
|
||||||
raylib including keyword→enum coercion. Acceptance test twice: headless (N
|
raylib including keyword→enum coercion. Acceptance test twice: headless (N
|
||||||
frames, hash the grid — runnable in CI on both targets) and interactive at
|
frames, hash the grid — runnable in CI on both targets) and interactive at
|
||||||
@ -776,8 +809,8 @@ building the whole live environment at once.
|
|||||||
special forms in the compiler.
|
special forms in the compiler.
|
||||||
6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and
|
6. *Allocators, ~Vec~/~Map~, ~Result~/~try~/~errdefer~, then conditions and
|
||||||
restarts* against spec-conditions.md, with dedicated tests per numbered case.
|
restarts* against spec-conditions.md, with dedicated tests per numbered case.
|
||||||
7. *Hot reload* — free in the interpreter, indirection cells for compiled dev
|
7. *Hot reload* — indirection cells in dev builds, with signature generations
|
||||||
builds, with signature generations and stale-caller warnings, plus the
|
and stale-caller warnings, plus the
|
||||||
remaining compatibility limits written down and enforced: struct layout
|
remaining compatibility limits written down and enforced: struct layout
|
||||||
changes, live callbacks held by C, captured environments.
|
changes, live callbacks held by C, captured environments.
|
||||||
8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor
|
8. *Debugger, nREPL, async* — last, and 8 splits into transport (8a) and editor
|
||||||
@ -823,8 +856,8 @@ monomorphisation, no restarts and no reload.
|
|||||||
None of these block milestone 2. The milestone each one must be answered by is
|
None of these block milestone 2. The milestone each one must be answered by is
|
||||||
marked.
|
marked.
|
||||||
|
|
||||||
1. *Host language: OCaml or Rust* — the only thing blocking the scaffold. See
|
1. *Host language: OCaml or Rust* — /settled: OCaml,/ and the compiler has been
|
||||||
Host language. /Milestone 2./
|
written in it since. See Host language for what the choice turned on.
|
||||||
2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately
|
2. Macro hygiene is settled for milestone 5: explicit ~gensym~, deliberately
|
||||||
non-hygienic expansion, no local macros until a concrete use case appears.
|
non-hygienic expansion, no local macros until a concrete use case appears.
|
||||||
3. Borrow checking and escaping frame-arena values. /Deferred; revisit after
|
3. Borrow checking and escaping frame-arena values. /Deferred; revisit after
|
||||||
@ -855,7 +888,8 @@ marked.
|
|||||||
a ~defvar~. Each needs an answer of the form "rejected", "accepted with a
|
a ~defvar~. Each needs an answer of the form "rejected", "accepted with a
|
||||||
migration", or "accepted and the old code keeps running".
|
migration", or "accepted and the old code keeps running".
|
||||||
7. Does the interpreter survive milestone 3, or is the compiled path the only
|
7. Does the interpreter survive milestone 3, or is the compiled path the only
|
||||||
backend? /Milestone 3, on measured numbers./ See Compilation.
|
backend? /Settled: the compiled path is the only one, and there is no
|
||||||
|
interpreter./ See Compilation, and BUILT.md's "Why there is no interpreter".
|
||||||
8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the
|
8. ~(Option a)~ /settled:/ an ordinary stdlib union with ~Some~/~None~; the
|
||||||
compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The
|
compiler niche-optimises ~(Option (Ptr T))~ to a nullable pointer. The
|
||||||
CL-vs-Clojure truthiness question is moot under static typing.
|
CL-vs-Clojure truthiness question is moot under static typing.
|
||||||
@ -875,8 +909,10 @@ marked.
|
|||||||
- Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~,
|
- Dev redefinition latency → ~16ms, measured: ~llc~ + ~ld -shared~ + ~dlopen~,
|
||||||
never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a
|
never the ~clang~ driver, ~dlopen~ off the game thread, cells published in a
|
||||||
batch at a frame boundary. See Compilation.
|
batch at a frame boundary. See Compilation.
|
||||||
- Dev backend → interpreter for milestone 2 certainly. Whether it /survives/
|
- Dev backend → compiled, and only compiled. The interpreter that milestone 2
|
||||||
milestone 3 is open, not settled — see Compilation.
|
was going to be written against was never needed and does not exist: expression
|
||||||
|
eval is a compiled shared object like everything else, and a macro is the case
|
||||||
|
that made the absence load-bearing rather than merely tidy. See Compilation.
|
||||||
- ~set~ on places → a fixed list of assignable forms, not ~setf~.
|
- ~set~ on places → a fixed list of assignable forms, not ~setf~.
|
||||||
- Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and
|
- Loop story → imperative ~while~/~until~/~dotimes~ with ~break~/~continue~ and
|
||||||
~return~; ~loop~/~recur~ only if it later earns its place. It did: both are
|
~return~; ~loop~/~recur~ only if it later earns its place. It did: both are
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user