flan/docs/DISCUSS.md

1433 lines
101 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Let's discuss
Open questions, raised and deliberately not answered yet. Nothing here is a decision or a task. Each entry is the
question as asked, plus what is already known in this repo that bears on it — so the investigation starts from what
exists rather than from scratch.
Settled decisions live in `NEXT.md`. Reasons for what already exists live in `BUILT.md`.
---
## 1. `i`, the inspector, and the frame it cannot see — answered and built
Answered, and built as option 2. `BUILT.md`'s "Two ways to root a walk, and why neither subsumes the other" is where
it lives now, including the correction to what this entry said about rooting at an address. The number is kept because
other files cite these by number; nothing here is open.
## 2. Annotating the IR and the disassembly with the source
**The IR half is nearly free and should just be done.** `emit.ml` writes `.ll` as text, so a comment costs nothing and
cannot break anything, and every typed IR node already carries a `Loc.t`.
**The disassembly half, with the optimisation question settled.** `objdump` already interleaves source into a listing
when DWARF is present (`-S`), and the daemon already shells out to objdump — so the `-O0` case is close to a flag.
Settled in conversation: **`-O0` is expected to follow the source and gets the full annotation; `-O2` is not expected to
and gets either nothing or whatever best-effort mapping falls out.** That removes what looked like the blocking
question. `--debug` forcing `-O0` is therefore fine and does not need decoupling for this.
**How SBCL does it, since it came up.** SBCL does *not* shell out. `sb-disassem` is its own disassembler, written in
Lisp, that knows the instruction encodings directly. What that buys is annotation from the inside: it labels constants
the function references, names the functions being called, marks entry points, and shows its own calling conventions —
because it compiled the code object and still holds the metadata.
The relevant observation is that **this project is closer to SBCL's position than the objdump route suggests.** The
daemon owns the build and holds the metadata too; it simply is not feeding much of it into the listing yet. Naming the
function behind an indirection cell, or a constant by its source name, needs no instruction decoder — only the
information the daemon already has. Writing a disassembler is not the interesting part and should stay off the table;
richer annotation of objdump's output is cheap and is where SBCL's advantage actually comes from.
## 3. `def`, `defvar`, `defconst` — three roles, currently two words
There is no `def`. `defvar` is a mutable global that is always re-initialised on redefinition; `defconst` is a
compile-time constant, folded into its use sites.
The problem in practice: `C-c C-k` re-evaluates every top-level form against a *running* program, so today it wipes
state you may have spent a session accumulating.
**Settled in conversation — three roles, three words:**
- **`def`** — always re-initialised. The tweakable: some number you are adjusting and do not care about preserving.
- **`defvar`** — re-initialised *only if it would come out different*. For heavier initialisers: an N×M grid that should
recompute only when the rows or columns actually changed.
- **`defconst`** — folded, and reserved for what genuinely cannot change for the life of the program. `rows`/`cols` in
`sand.flan` are honest `defconst`s: the program does not support resizing on the fly and changing them means resetting
everything anyway.
Note this **`defvar` is not Common Lisp's**, which is "assign only if unbound". This one is value-dependent, which suits
a live-edited game better and is the harder of the two to implement.
**The open question is what "different" means**, and the grid example is what makes it sharp. If the grid is sized by
`rows` and `cols`, the initialising expression's *text* is unchanged when `rows` changes — only its value is. So:
1. **Compare the expression** (textual or AST). Cheap, and does not catch the case above.
2. **Compare the expression plus everything it depends on.** Catches it. Needs dependency tracking.
3. **Evaluate and compare the result.** Correct, and defeats the entire purpose whenever the initialiser is expensive —
which is the only case this feature exists for.
Option 2 is what is wanted.
**The reason to do it: it is the same machinery `defconst` needs.** A folded constant cannot be tuned live, because its
value is baked into every function that used it — so the values you most want to tweak while the game runs are exactly
the ones you cannot. Fixing that means knowing which functions depend on which constants, and rebuilding those. That is
the same dependency tracking `defvar`'s "if different" requires. Built once, it buys live-tunable constants *and* the
conditional re-initialisation. Neither is worth building alone; together they are clearly worth it.
## 4. Structural typing, row polymorphism, anonymous structs
The goal as stated: make this feel Clojure-like while still being typed.
**The representation is already structural.** `types.ml` does structural equality on resolved types, and a Flan struct is
exactly its C layout with no header and no tag word. What is nominal is the *checking*, not the data — so this is a
front-end question, not a data-model change.
**What it buys, in rough order of value here:**
1. **Your destructuring already looks like this, and that is the strongest argument.** Clojure's `{:keys [x y]}` works in
binding position today. Structural typing makes the same notation work in *parameter* position — the pattern you write
to pull fields out becomes the signature saying what you accept. One notation, two places, no separate declaration.
That is precisely the "Clojure-like but typed" feel being asked for, and it needs no new syntax, only a new meaning
for syntax that exists.
2. **Functions over "anything with these fields".** Magnitude over anything with an `x` and a `y` — a position, a
velocity, an enemy, a bullet. In a game this recurs constantly because everything has a position.
3. **Ad-hoc returns.** Returning a hit flag and a point without declaring a type for the pair. The Clojure habit of
returning a map, typed and free.
4. **The FFI stops needing conversions.** raylib's `Vector2` and a locally-defined vector are byte-identical and are
today two nominal types. Structurally they are one.
**Two costs, and the second is the real work:**
- **Not an "explicit over convenient" question — that framing was wrong and is recorded so it is not repeated.** The
standing rules about implicit numeric conversion and about a bare integer not becoming an enum are about *lossy*
things happening silently. A structural match is not lossy; it is a different notion of type *identity*. The two were
lumped together in conversation and the author rejected the pairing, correctly. Strong typing and nominal typing are
separate choices and this only touches the second.
- **Field order is layout, and that is the hard part.** `{x f32, y f32}` and `{y f32, x f32}` have the same fields and
different memory. So either field order becomes significant — surprising, since nothing else about a structural type
should be — or matching a differently-ordered type requires a copy, which breaks the zero-cost property the whole data
model rests on. Accepted in conversation as a limitation to respect rather than fight.
**Writability is what decides whether the copy escape exists.** Fields would be writable on the same terms as any struct:
by value you get a copy and writes are local; through a `(Ptr T)` they reach the original. But that settles the layout
question rather than sitting beside it. *Read-only* structural access has an out — the compiler could gather the
requested fields into a temporary, and then field order would not matter at all. *Writable* structural access cannot:
it has to alias the real storage, so the layout must genuinely line up. Decide writability first; the layout answer
follows from it.
**`defclass` may solve this, for the cases that can afford it.** A class has an implementation-defined representation, so
the compiler owns the layout and access goes through metadata rather than a fixed offset — field order stops being a
user-visible fact. The limit is that a class carries identity and metadata, and a `Vector2` should not pay for either.
So plain structs still need their own answer and classes only cover the cases where the overhead was already accepted.
See item 7.
Also unresolved: the shim generates a C typedef per *named* struct, and an anonymous struct has no name to generate one
from. Either anonymous structs cannot cross the FFI, or the generator learns to name them.
## 5. A JS backend — for web apps, not for games
**The goal, stated in conversation: calling JS libraries, and eventually a hiccup-style DSL for writing web apps in
Flan.** That is a different product from the wasm target and the two do not compete — games go to wasm, web apps go to
JS.
This matters because the obvious objection does not apply. "A JS backend cannot call raylib, so the whole graphics layer
would need reimplementing on canvas" is fatal for a *game* backend and irrelevant here: web apps do not use raylib.
**The design fork, and it should be decided first:**
- **Linear memory** — an `ArrayBuffer` standing in for pointers and structs, with typed-array reads and writes. Fast
arithmetic, and every value is opaque bytes, so every call into a JS library marshals both ways. This is asm.js, which
wasm exists to replace, and it is precisely wrong for an interop-motivated backend.
- **Object mapping** — a Flan struct becomes a plain JS object, a `Vec` becomes a JS array. Slower for tight numeric
loops, natural for interop, readable output. **For this goal this is clearly the right one**, and performance is not
the constraint people assume: DOM work is dominated by the browser, not by arithmetic.
**The consequence to face early: the memory model does not come along.** Object mapping means garbage collection, which
means no pointers, no manual `free`, no arena, no allocator. So this is a **dialect**, not merely a second backend —
some Flan programs will not compile to JS, and that should be named up front rather than discovered. Worth deciding
which subset is the JS-targetable one, and whether the checker enforces it per target.
**The unexpected upside: the dev loop could work on the web.** Conditions, restarts and a break loop all need the
ability to run new code in a stopped program. wasm cannot have this — `--dev` is refused there because reload is
`dlopen` and the browser has no sockets. JS has no such problem: it evaluates new code trivially. A JS target might be
the *only* place the live dev loop and the browser coexist.
**Reader conditionals, decided in conversation.** A cljc-style subset is accepted, and the author chose Clojure's
inline reader conditionals over file-level target naming. Note what that changes: **this language has no conditional
compilation today**, and that absence drove a decision earlier the same day — `barf` signals on web rather than being
compiled out, precisely because "this bit is desktop-only" is not expressible. Note also that the sand-on-web work
introduced file-level target replacement for *C* sources (`foo.web.c` replaces `foo.c`), so the repo now has a
file-level precedent that this decision deliberately does not follow. Both were put; inline was chosen.
**Also wanted, and independent of all of the above:** `#_` to discard the next form, and repeated (`#_#_`) to discard
that many following forms, as in Clojure. Reader-only, small, and useful immediately rather than whenever a JS backend
happens.
**One dependency:** hiccup is a macro, and the macro expander is blocked on `Form` being a Flan union, which is union
values. The backend can start before that; the DSL cannot.
## 6. C interop as seamless as Zig's — built, with two decisions left
**The mechanism is in** (`lib/cimport.ml`, `lib/cjson.ml`, `vendor/raylib/headers`; BUILT.md, "The header
is read now"). Settled and not worth reopening: clang's JSON AST dump over a shelled-out `clang`, never
libclang — and Zig has since abandoned linking clang too, for Aro, which strengthens the argument rather
than weakening it. The import is bounded by the package's own `defstruct`s rather than by a curated list.
Refusals are demotions in Zig's sense: the name exists, cannot be had, and says why at the use site, which
`Load.refuse_hidden` already did for `main`. Names kebab by a rule that is injective over raylib's 581, and
reversibility is by storage — the C symbol is kept verbatim — so the rule never needs an inverse. Where two
names do collide, neither takes it.
**The evidence:** against raylib 5.5, all 16 `defstruct`s and all 172 hand-written `declare-c` agree
exactly. Against the 5.1-dev header also on this machine, ten real differences. Both comparisons stop the
build, and a permuted `Texture2D` or an `f64` for a `float` is caught by name.
What is left is two decisions, and both are the author's.
**6a and 6b are answered. See BUILT.md, "The bindings are committed now".** It became a code generator whose
output is committed, and the 172 hand-written lines were *not* migrated — for a reason 6b did not reach. The two
sections below are kept as the reasoning that got there.
### 6a. Does reading the header stay a build-time step, or become a code generator?
The cost, measured, with the wrappers pruned by `Reach` as they already were:
| | today | + 256 imported |
|---|---|---|
| release build, warm | 0.078s | 0.082s |
| **redefinition** | **31.0ms** | **46.5ms** |
| dev build, cold | 0.649s | 0.982s |
Release is nearly free and that question is answered. **The 15.5ms on redefinition is not nothing on the
branch where the dev loop is the priority** — it is a 50% increase on the number that lane exists to keep
small, and it buys a check of signatures that have not changed since the last build.
So the third option from the original discussion is now the live one: `flan import-c` already prints
`declare-c` lines, so **generate from the header, commit the result, regenerate when raylib moves** costs
nothing more to build. Explicit in the source, checked against reality, no header read at build time, and
the check becomes a thing you run rather than a thing you pay for. Against it: a committed file goes stale
silently, which is the failure the whole lane exists to prevent, and "regenerate when the library moves"
is a discipline rather than a mechanism.
A middle reading worth considering: keep the build-time check but run it only when *not* `--dev`, on the
grounds that a release build is where a wrong signature must not get through and a dev build is where
15.5ms is felt. That is the same shape as `Reach` not pruning dev builds, for a symmetric reason.
### 6b. Do the 172 hand-written lines get migrated?
The diff is clean, so nothing blocks it on correctness. What blocks it is that migration needs the header
present at *every* build, which means vendoring raylib.h into the repo or requiring `raylib-devel` — and
BUILT.md records "a build needs libraylib linkable and not raylib-devel installed" as a property that was
chosen on purpose. That is why `vendor/raylib/headers` is opt-in (`?${FLAN_RAYLIB_H}`) today and the
hand-written lines are untouched.
Worth noting what migration would actually lose, since it is small but real: the hand-written names are
better than the rule's. `IsKeyPressed` is `key-pressed?` by hand and `is-key-pressed` by rule;
`CheckCollisionRecs` is `collision-recs?`. And an enum parameter imports as `i32`, because the header says
`KeyboardKey` and nothing tells the importer the package calls that `Key` — so `(rl/key-down? :space)`
would become an integer at the call site. A migration is therefore not a deletion; it is a deletion plus a
kept list of the lines whose face is deliberately nicer than the header's.
**Answered: no, and the blocker is not the one above.** Vendoring stopped being the question once the *output* was
committed rather than the header — no header is needed at any build. (Vendoring then happened anyway, and for the
other reason: `vendor/raylib/raylib-5.5.h` is committed, `headers` names it with no `${...}` in front of it, and the
signature check runs on every build. What that settled was not "can we build without a header" — the committed
bindings had already settled that — but "does the check actually run", and behind an opt-in the answer in several
worktrees was no.) The count was also smaller than feared: 136 of
the 172 are exactly what the rule produces, and the other 36 are expressible as `name` overrides in `bindings`.
What actually decides it is that migration would gut the check. Everything the generator emits agrees with the header
by construction, so diffing generated output against its own source proves nothing; the hand-written lines have a
different author, so they are the only declarations a header can contradict — and all ten of the 5.1-dev differences
came from them. Delete them and the signature half of the check silently becomes a tautology. The enum point above
survives intact as a second reason: `(rl/key-down? :space)` keeps its `Key` parameter only because that line is
hand-written.
## 7. Watching variables
Raised while designing item 1, and deliberately separated from it.
Item 1 shows globals *per frame*, chosen by what the code in that frame references. That is the right default and it
cannot cover the case where the thing you care about is not named by the frame you are standing in — stopped deep in a
helper, still wanting to see the grid.
So: a way to say "always show me this", surviving a resume and the next break.
Questions it opens:
- **What can be watched.** A global is easy — it has a name and an address that does not move. A *local* is harder: it
belongs to a frame that is gone as soon as you resume, so "watch `y`" either means a different `y` every time or means
nothing. A watched expression — `(at velocity 40 12)` — is the most useful and the most expensive, since it has to be
compiled and run in the program each time, which is what `eval-expr` already does.
- **Where the list lives.** Per project, per session, or in the file. A watch list that vanishes when Emacs restarts is
one you stop using; one in the repo is one you accidentally commit.
- **When it updates.** Only on entering a break is cheap and probably enough. Live-updating while the program runs is a
different feature — closer to a HUD than a debugger — and worth not conflating.
- **Whether it belongs in the break buffer at all**, or in its own window that is useful while the program is *running*,
which is arguably where a game developer wants it.
## 8. Indentation is wrong inside a binding vector
Reported against the `let` in `sand.flan`'s `settle`: the second and later bindings indent one column too far.
**Cause, found by reading `flan-mode.el`.** `flan-indent-function` looks at the head of the *enclosing* form and, when
it is one of Flan's body forms (`defn`, `let`, `if`, `while`, `until`, `dotimes`, `match`, `do`, `loop`, `defer`),
indents one past the open paren. Inside a `let`'s binding *vector* the enclosing open is the `[`, and the symbol after
it is the first binding's **name** — never a body form — so the check fails and it falls through to Emacs's generic
`lisp-indent-function`, which treats the vector as a function call and aligns continuation lines under the first
*argument* rather than under the first *binding*.
**What it is built on:** Emacs's built-in `lisp-indent-function`, with that one override. Not `clojure-mode`, which is
where the missing piece lives — `clojure-mode` special-cases binding vectors and aligns them as pairs.
**The fix, agreed in conversation:** add a binding-vector case to `flan-indent-function`. When the enclosing open is `[`
and the form containing it is a binding form, align continuation lines to the column of the **first binding**, not to
the first argument. Bindings are pairs, so the alignment that reads correctly is name-under-name. Check the other
bracket users in the same pass — `defn` parameter lists and `restart-case` clause parameters have the same shape and
almost certainly the same bug.
**`clojure-mode` should be the reference, not the ancestor and not a dependency.** The distinction matters and was
muddled once in conversation: the proposal is *not* `define-derived-mode` on `clojure-mode`, and not adding an external
package. It is to rewrite `flan-mode`'s indentation and font-lock as our own code, **ported from `clojure-mode`'s source
rather than from `lisp-mode`'s**.
The current mode has the wrong ancestor. It is built on Emacs's `lisp-indent-function`, and Emacs Lisp has none of the
shapes Flan actually uses: no vectors as binding forms, no keywords as map keys, no destructuring, no bracket variety.
So every rule has to be added by hand and the binding-vector bug is simply the first one hit. Clojure's rules already
cover brackets-mean-binding, pairs-align, maps, keywords, `#_`, and reader conditionals — the last two now wanted in
their own right.
Where Flan diverges it diverges deliberately rather than by discovery: `defn` carries a return type between the
parameters and the body; field access is `(.x v)`; field labels are moving from `:x` to `.x`, so `{.x 1.0}` is a struct
literal and not a map with symbol keys. Those get written on purpose.
Deriving at runtime stays rejected — it would add an external dependency to a mode that ships inside this repo and
currently needs nothing beyond stock Emacs, and that community is mid-transition to a tree-sitter mode, so the base is
moving.
## 9. Instrumenting with `(pause)` from Emacs, without editing the buffer
Requested, not scheduled. The Clojure model: `C-u` before an eval marks the form so it stops when it runs. Wanted here
for **three targets** — the top-level form, the last expression, and **the form point is inside**, so that with the
cursor at `(+ 1| 1)` the program stops at that `(+ ...)`.
**What already exists, so this is mostly assembly:**
- **`(pause)` itself**, landed today in `lib/prelude.ml`. It is `error` under a `restart-case` with a `continue`
clause — a breakpoint is not a compiler feature here, it is what a condition system already gives you. Taking
`continue` resumes at the call.
- **`flan-eval-defun`** takes the top-level form's bounds and sends the text; **`flan-eval-last-sexp`** sends the sexp
before point. Both hand a *string* to the daemon, which is the hook — the buffer never has to be modified.
- **`flan-dev--enclosing-head`** (`flan-dev.el:1009`) already walks out to the enclosing form to find its head, for
eldoc. The third target's hard part is already written.
- `current-prefix-arg` is already read in one place (`flan-dev.el:1429`), so the `C-u` convention has precedent.
**So the work is a source-to-source rewrite in Emacs before sending:** find the target form's bounds, send the
top-level form with that span replaced by `(do (pause) <span>)`. Nothing on the compiler side changes.
**The one real difficulty: source locations.** Splicing text shifts every line and column after the insertion, so
error overlays, `layout`, the break loop's frame locations and DWARF would all point slightly wrong for an
instrumented form. Options, none free: pad the inserted text to preserve offsets (ugly, fragile across newlines); send
the instrumentation as a separate field the daemon splices *after* parsing, where locations are already attached; or
accept the drift and say so. **The second is almost certainly right** and makes this a small daemon change rather than
a pure-Emacs one — which also means it should not be built as a pure-Emacs hack first.
**Settled: it sticks, like Clojure's.** The mark stays until the form is evaluated again plainly. That is the
behaviour the workflow expects — you mark it, run the game, hit it repeatedly, and clear it with an ordinary `C-c C-c`
rather than having to re-mark before every run.
## 10. A second backend for dev builds — study SBCL and drop LLVM there?
Raised at the end of the session, not answered. The proposal: write our own code generator for **debug builds and the
interactive workflow**, keep LLVM for release. The motivation is that evaluating arbitrary code — full functions, real
test code, not just constant arithmetic — costs a compile, a file and a `dlopen` every time.
**What SBCL actually does**, since it is the model: its own assembler and code generator, emitting machine code
straight into the heap. No files, no external tools, no dynamic loading. Code objects are ordinary heap objects and are
collected when unreachable. That is also why its `disassemble` can annotate richly and why it can show machine state —
it owns everything.
**The argument against, and it is the same one that killed the interpreter.** Two backends must agree about arithmetic,
struct layout, overflow, evaluation order and calling convention. When they disagree you get *works interactively,
breaks when shipped* — the worst bug class in a live-programming system, and arguably worse than the interpreter
version because the divergence is in code generation rather than in semantics you can read.
Note what plan.org's dev-vs-release table already accepts: indirection cells, a shadow stack, version words, code
never freed. Those are **additive instrumentation on one code generator**, not a second one. This proposal is a
different kind of divergence.
And the cost is enormous: an instruction selector and assembler per architecture — x86-64, arm64, wasm32 — which is
where SBCL's decades went.
**The cheaper route to the same goal, and the thing to weigh it against: ORC, LLVM's in-process JIT.** Measured today,
a 35ms redefinition is **21ms of `llc` plus `ld`** — external tools generating and linking machine code. An in-process
JIT replaces exactly that, with **the same IR and therefore the same semantics**, no second implementation, and code
that can be unloaded. It gets what the proposal wants without the divergence hazard.
plan.org already says ORC "remains addable later behind the same typed IR without touching the language", and the only
column text-IR-plus-clang loses in its comparison table is the JIT one.
**So the honest position:** the thing that would justify revisiting the LLVM decision is not slower eval in the
abstract — it is wanting *arbitrary interactive evaluation of real code* as a first-class workflow. That is now stated
as a want. If it stays a want, ORC is the answer and a hand-written backend is not.
**Where the shared objects live**, since it was asked: `Build.workdir ()` — a per-build temporary directory — with the
object cache separately at `Build.cachedir ()`, stable across builds. Nothing sweeps the workdir, and nothing is ever
`dlclose`d, so a long session accumulates both files and mappings. Sweeping at session end is cheap and unrelated to
any of the above.
## NEXT SESSION STARTS HERE
**The agreed plan: investigate putting the compiler inside the running program.**
Today there are two programs. The *daemon* is the compiler. The *game* is a separate program it launches. They talk
over a socket, so everything the editor shows — locals, globals, the stack, evaluation results — has to be packed up
in one program, sent across, and unpacked in the other. A large share of what `BUILT.md` documents exists only to move
data between those two address spaces.
Putting the compiler inside the game removes all of that. There is nothing to send, because the compiler can read the
program's memory directly. It is what SBCL does, and it is why interactive work in SBCL is instant.
**Read, in this order:** item 13 (what is actually being decided, and why the order matters), then item 11 (the shape,
the two objections and how the author answered them), then item 12 (the specific questions a spike must answer).
**First task, and it is research not building:** the spike in item 12 — what it costs to link the OCaml compiler into
a dev build. OCaml 5.2's multicore runtime has already removed the objection that would have been fatal. If it embeds
acceptably, everything else is reachable.
**Do not start with the backend question.** Merge first, measure, then choose. Item 13 says why.
---
## 11. One process, not two — the biggest open question
Raised at the very end of the session and **not resolved**. It subsumes items 9 and 10, and possibly several decisions
recorded elsewhere as settled. Do not treat any of those as closed until this is answered.
**The claim:** the daemon and the running program should not be two processes. SBCL is one image; Clojure is one
image; in both, the interactive workflow is cheap *because nothing is transported*. The author's reading is that the
split here was forced by not having threads — and that is worth checking, because **threads are already in use**: the
agent runs a listener thread and a loader thread inside the program today.
**The concrete shape**, if it were done: load the *program* into the daemon's process as a shared library rather than
launching it as a child. The reload primitive already `dlopen`s modules; the program itself would be one.
**What would evaporate, and it is a lot.** No socket and no wire protocol. Cell updates become plain pointer stores.
Locals and globals become memory the compiler can read directly — no render thunks compiled per inspection, no 4K
result cap, no seqlock, no snapshot copying, no generation stamping. The macro expander stops needing the compiler's
own `dlopen`. `flan reload` stops being a fresh process paying startup every time. A large amount of machinery
recorded in `BUILT.md` exists **only** to move data between two address spaces.
**What is given up, in order of sharpness:**
1. **Crash isolation.** A segfault in the game currently cannot touch the compiler. `plan.org`'s jank note — corrected
earlier for citing the wrong mechanism — concludes "we are safe from the repro *because we compile out of process*".
In one image a bad pointer takes the session with it. Lisp users live with this and mostly accept it; it is a real
loss and should be chosen knowingly, not discovered.
2. **macOS requires a window on the main thread.** "Run raylib on a separate thread" is fine on Linux and not portable.
The inversion — compiler on the side thread, window on main — probably works, but it needs deciding rather than
assuming, and it changes who owns the main loop.
3. **The OCaml runtime and the game in one address space**: signals, `SIGSEGV` handling for the break loop, the OCaml
GC running while game code holds raw pointers into its own arenas. Each is probably fine; none is free.
**What it does NOT fix, and this is where an earlier answer in this session was wrong.** A hand-written debug backend
was discussed as the way to make evaluation instant. The correction: a new backend removes `llc` and `ld` — about
21ms of a 35ms redefinition — but **not the transport**, because the transport exists due to the process split, not
the backend. Conversely, **merging the processes removes the transport but not the code generation**. They are
independent, and merging is the larger prize of the two.
**Decisions to revisit if this is taken:** the watch design (push was chosen partly because polling costs a compile);
the 4K result cap and its seqlock; the snapshot machinery and its generation stamping; the render-thunk-per-inspection
design for locals and globals; the inspector's inability to retain a value; and whether ORC or a hand-written lowering
is needed at all.
**Both objections were answered by the author, and the second reframed the design — record the corrected shape.**
**Crash isolation: accepted knowingly.** Conditions already catch what the *language* signals — a failed bounds check,
an exhausted allocator, a missing file — and those keep stopping in the break loop in one process exactly as they do
now. What they cannot catch is a real memory fault: a bad pointer through the FFI, a use-after-free in an arena. That
is an OS signal, not a condition, and it takes the image with it. Same deal every Lisp makes.
**macOS: invert the relationship, and this is the right shape rather than a workaround.** Not "load the program into
the daemon" as sketched above — the opposite. **The agent is already a server running inside the program.** So the
compiler moves *into the program's process*, beside the listener that is already there. The game starts, takes the
main thread for its window, and the compiler and listener run on a side thread. That is SLIME's model: you start the
image, it serves, the editor connects.
This also fits a distinction `plan.org` already draws — dev and release builds are deliberately different. A dev build
links the compiler in; a release build ships neither compiler nor server. The same split SBCL has between a
development image and a delivered executable.
One thing that gets *better* rather than merely cheaper: a break loop in the same process as the compiler can offer
restarts that recompile and retry, with no transport in between.
**Suggested next step:** the macOS question is answered, so start instead with what it costs to link the OCaml compiler
into a dev build — the OCaml runtime and the game sharing an address space, signal handling for the break loop, and the
GC running while game code holds raw pointers into its own arenas. It is the only one of the three costs that could
make the whole idea non-portable, and it is answerable by reading raylib and GLFW rather than by building anything.
## 12. What it costs to put the OCaml compiler in the game's process — and whether to port the compiler
Follows from item 11. **Investigate before deciding anything**; this entry is the brief, not the answer.
**One fact established already: this project is on OCaml 5.2.** That is multicore OCaml — real threads, domains, no
global runtime lock. It removes the objection that would have been fatal: the runtime can sit on its own domain beside
the game's threads rather than serialising everything.
**What to find out:**
- **Linking OCaml into a C/native binary.** `ocamlopt -output-obj` and friends are supported; how well, with dune, with
the C stubs this project already has (`dynload_stubs.c`, the runtime, the shims)?
- **Signals.** The OCaml runtime installs handlers. The break loop wants `SIGSEGV` for the crash case, and the agent
already handles socket work. Who wins, and can they coexist?
- **The GC and raw pointers.** OCaml's collector moves its own heap; Flan's arenas and `Vec`s are raw memory OCaml
never sees. That should be fine — they do not point at each other — but "should be" is not an answer.
- **Startup cost and binary size** for a dev build that links the whole compiler.
- **Whether the main thread can be the game's** while the compiler runs on a domain (see item 11's macOS note).
**And the larger question the author raised: port the compiler to another language?**
The honest end state for a Lisp that wants one image is **self-hosting** — the compiler written in Flan. That is what
SBCL is, and it dissolves this entire question: no foreign runtime to embed, no GC to reconcile, no signal conflict,
and the compiler becomes the largest test of the language. It is also an enormous undertaking and needs macros,
unions, `Map` and a string library first — three of which landed today.
Intermediate options if OCaml proves genuinely hostile: C or C++ (no runtime to embed, painful to write a compiler
in), Rust (no GC, embeds cleanly, large rewrite), or Zig (same, and its own C-interop story is already being read for
item 6). **None of these should be entertained on aesthetics.** The bar is a specific, demonstrated obstacle to
embedding OCaml 5.2 — and the multicore runtime means the most likely obstacle has already gone away.
**Order of work:** answer the embedding questions above with a spike, not a rewrite. If OCaml embeds acceptably, item
11 is buildable and no port is needed. Self-hosting stays a long-term ambition rather than a prerequisite.
## 13. The two questions are independent, and the order matters
The end of the session, and the clearest statement of what is actually being decided. Read this before items 10, 11
and 12, which were each written mid-argument.
**Two separate decisions, repeatedly conflated in conversation:**
1. **One process or two** — whether the compiler runs inside the program's process. This is about *transport*: how
compiled code and inspection data move between the compiler and the running program.
2. **How machine code is produced and installed** — the current `llc` + `ld` + `dlopen`, libLLVM's in-process JIT
(ORC), or a hand-written lowering. This is about *code generation*.
Neither implies the other. A hand-written backend removes `llc` and `ld` (about 21ms of a 35ms redefinition) and
leaves the transport. Merging the processes removes the transport and leaves code generation exactly as it is.
**Merging is the larger prize.**
**And there is a third option for question 2 that keeps getting lost: change nothing.** In one process you can still
write a `.so` and `dlopen` it. You lose nothing that works today, and you still delete the socket, the wire protocol,
the 4K result cap, the seqlock, the snapshot copying, the render-thunk-per-inspection, and `flan reload` paying process
startup every time.
**So the order is:**
1. **Spike the embedding** (item 12). OCaml 5.2's multicore runtime removes the objection that would have been fatal.
If it embeds acceptably, everything below is reachable.
2. **Merge the processes**, keeping the existing code path. Biggest win, no new dependency, deletes the most
machinery.
3. **Measure what is left.** With transport gone, a redefinition is whatever code generation costs. That may simply be
fine.
4. **Then decide the backend, on evidence.** libLLVM's JIT means linking the library `plan.org` rejected — for
reasons that were right at the time, when the JIT was worth ~13ms and nothing else. In one process it is worth the
file, the load, the unbounded accumulation *and* cheap evaluation of arbitrary real code, which the author has since
named as a first-class want. The cost is also more bounded than "breaks routinely" suggests: the bindings track
LLVM major versions, so it is a pin you own and upgrade deliberately.
A hand-written lowering buys the same speed plus no dependency and a disassembler you own, at the cost of an
instruction selector per architecture, maintained forever. The drift risk is real but testable — the `@sanitize`
alias already builds 28 programs twice and compares output and exit status, which is exactly the harness two
backends would need.
**The principle to keep:** the original decision was made on a measurement and it held up for months. Make this one the
same way — merge first, measure, then choose. Do not pre-empt step 4 at step 1.
**Also settled in passing:** a release build can include the compiler and server and simply not be connected to, the
way an SBCL executable can. The dev/release split is a build flag, not an architectural constraint.
**And one correction to keep:** OCaml's own wasm support is irrelevant to any of this. OCaml is the compiler's
implementation language; Flan programs reach wasm through LLVM. The two only meet if the *compiler* should run in a
browser, which is not a goal.
## 14. The embedding spike, answered: OCaml 5.2 goes into a Flan dev build, and nothing objects
Item 12's brief, run. **Feasible.** No obstacle was found that argues for porting the compiler, and the one expected to
be sharpest — signals — turned out not to exist on the platform measured.
The apparatus is `spike/embed/`: four shell scripts and sixteen small sources, deliberately not a dune target, driving
`ocamlfind` and `clang` by hand against the `flan.cmxa` dune already builds. `bash spike/embed/run.sh` reproduces
everything below; `sig.sh`, `symbols.sh` and `merged.sh` each answer a question on their own. Nothing under `spike/` is
wired into the build, and `dune test --root .` is green either side of it.
### The headline: one binary, and it compiles itself
`spike/embed/merged.sh` builds a single executable out of, in one `clang` link:
- the emitted Flan program (`test/programs/edn.flan`, `--dev`, `@main` renamed to `flan_program_main`),
- `runtime/flan_rt.c`, `runtime/flan_dev.c`, `vendor/agent/flan_agent.c`,
- and the entire OCaml compiler as one `-output-complete-obj` object.
It runs. A C `main()` holds the main thread and runs the Flan program there; `caml_startup` happens on a `pthread`
beside it, next to where `flan_agent_start` already puts its listener. The compiler inside the binary then compiles
`edn.flan` — the source the program itself was built from — and emits 336,579 bytes of LLVM IR. That single result
answers questions 1 and 2 together; the ladder of smaller probes underneath it is support, not evidence in its own
right.
Nothing is wired up. The two halves share an address space and do not speak to each other. That is the point: the
question was whether they *can*, not what they would say.
### Question 1 — does OCaml link into a native binary here?
Yes, and with less friction than expected.
- `ocamlopt -output-complete-obj` is the one to use, not `-output-obj`: it bundles the runtime, so there is no hunt for
`libasmrun`. The final link needs `-lm -lpthread -ldl` and, on 5.2, **`-lzstd`** — 5.x's marshaller is compressed,
and the missing `ZSTD_*` symbols are the first thing a naive link fails on. That is the whole of the surprise.
- **dune is not in the way, because it does not have to be involved.** dune builds `lib/flan.cmxa` as it does today;
the `-output-complete-obj` step consumes that artifact afterwards. No dune rule had to change, and `lib/dune` and
`bin/dune` are untouched. A real merge would want a dune rule to drive that step, but the spike shows the artifact
boundary is clean, which is the part that could have failed.
- **The existing C stubs come through.** `lib/dynload_stubs.c` was taken verbatim from the unmerged `9e0ae3a` dlopen
branch and compiled into the same object; from inside the embedded runtime, `flan_mem_alloc`/`poke`/`peek`
round-trip correctly and `dlopen`+`dlsym` work. `-output-complete-obj` carries a `foreign_stubs`-shaped C file
through without special handling.
- **No symbol collides** (`symbols.sh`). Flan's own C — `flan_rt.c`, `flan_dev.c`, `flan_agent.c`, `dynload_stubs.c`
defines 211 symbols; `libasmrun.a` defines 1,379; the intersection is empty, and the four Flan files do not collide
with each other either. Worth checking rather than assuming: those four are compiled into *two different processes*
today, and merging puts them in one link for the first time.
### Question 2 — threads, and who owns the main loop
**OCaml 5.2 is confirmed multicore**, and `Domain.recommended_domain_count ()` reports 16 here. A spawned domain does
real work in parallel with the main thread. The objection that would have been fatal is gone.
The macOS shape works, and it was tested as the thing that matters rather than as "can OCaml use threads":
- `caml_startup` can be called from a **C-created, non-main pthread**, while `main()` goes on to a loop it does not
leave. `harness4.c` runs a 298-frame mock game loop on the main thread that never once enters OCaml.
- **A second C thread — one the runtime never created, which is exactly the agent's listener — can call into OCaml**
after `caml_c_thread_register()`, bracketed by `caml_acquire_runtime_system`/`caml_release_runtime_system`. It
compiled the same program successfully from that thread. This is the specific capability the merged design needs
from `flan_agent.c`, and it exists.
The spike could not test macOS. Nothing here is macOS-specific — the inversion is a portable pthread arrangement — but
it is a Linux measurement.
### Question 3 — signals
**On Linux/amd64, OCaml 5.2 installs no signal handlers at all.** Not SIGSEGV, not SIGINT, not SIGFPE, not SIGPIPE,
nothing. The expected conflict does not exist.
`sig.sh` sweeps fifteen signals from inside the runtime at four moments — before `caml_startup`, at module init, from
inside a spawned domain, and after `Domain.join` — and every one is `SIG_DFL`. The reading has to be taken from inside
OCaml rather than from C after `caml_startup` returns, because OCaml 5 starts domains later; the first pass got this
wrong and read `SIG_DFL` for the wrong reason. A plain `ocamlopt` executable was built as a control and behaves
identically, so embedding changes nothing about signals.
The reason is structural, not incidental: OCaml 5 detects stack overflow with an explicit stack-limit check and calls
`caml_raise_stack_overflow` directly, rather than with a guard page and a SIGSEGV handler. `nm` on `libasmrun.a` shows
`sigaltstack` referenced but no SIGSEGV handler defined, which matches.
So **the break loop can take `SIGSEGV` outright, and it does not have to install first or last** — order does not
matter when there is nothing to displace. Measured directly: with the break loop's handler installed, deep OCaml
recursion still raises `Stack_overflow` normally, and a genuine fault at `0x10` reaches the break loop's handler with
`si_addr` correct. Chaining was implemented and compared; it is unnecessary here, but `harness5b.c` keeps it, because
it is what the merged build should do on any platform where the sweep comes back non-empty.
**The caveat, and it is the one thing in this report to re-check rather than trust:** this is `x86_64-pc-linux-gnu`
only. macOS/arm64 OCaml 5 was not testable here, and item 11 raises signals in the same breath as macOS. Re-run
`sig.sh` there before relying on it. `flan_agent.c` needs nothing from this either way — it sends with `MSG_NOSIGNAL`
throughout rather than depending on a SIGPIPE disposition.
### Question 4 — the GC and raw memory
Confirmed, and the confirmation is narrow on purpose. An 8 MiB arena was filled with a checkable pattern and 64 raw
interior pointers taken into it; OCaml then allocated 26.4 million words, took 7 major collections and a full
`Gc.compact ()`. Afterwards: the arena base is unmoved, **0 of 8,388,608 bytes altered**, 0 of 64 interior pointers
invalidated.
What that proves is that the collector traces its own roots and foreign memory is invisible to it. Arenas, `Vec`s and
`Map`s are safe because OCaml never learns they exist.
**What would break the assumption**, stated so it is not rediscovered the hard way:
1. Storing an OCaml `value` in Flan memory — an arena, a `Vec`, a global — across any allocation. The collector will
move the block and will not update that word, because it is not a root. `caml_register_global_root` (or the
generational one) is the only way that is legal, and it is a rule the merged design has to hold: **the boundary
passes pointers and scalars, never `value`s into Flan storage** — the same rule `dynload_stubs.c` already states
for its own reason.
2. A `value` held in a C local across a call that allocates, without `CAMLparam`/`CAMLlocal`. Ordinary OCaml-FFI
discipline; it applies to every stub the merge adds.
3. Long C work on the compiler thread without releasing the runtime system — which corrupts nothing but stalls
whichever domains want a stop-the-world. `llc` and `ld` are `exec`s and would want `caml_release_runtime_system`
around them.
### Question 5 — what it costs
| | bytes |
|---|---|
| `edn.flan`, release build | 71,824 |
| `edn.flan`, dev build, as built today | 114,296 |
| the same dev build with the whole compiler linked in | 4,257,624 |
| **what the compiler adds** | **~4.14 MB** |
**Startup: `caml_startup` takes 0.580.72 ms** across six runs — the runtime coming up and every module initialiser in
the compiler running. Measured with `clock_gettime` around the call itself, not `time(1)` on the process, because exec
and dynamic linking are paid today anyway.
The size reads as 37x, and that framing is misleading. **A dev session today runs two binaries, and the daemon alone is
4,815,368 bytes.** The merged dev build is *smaller than today's compiler process by itself*, and there is one of it
instead of two. Sub-millisecond startup and ~4 MB is not a cost worth designing around.
One more number, recorded because item 13's step 3 will want it and for no other reason: **a full in-process compile of
`edn.flan` — read, parse, load, typecheck, emit — is 12.112.5 ms**, warm and cold alike, with `llc` and `ld` excluded
because they are separate processes. That is where the remaining cost sits once transport is gone. It argues for
nothing; item 13 says the backend is decided at step 4 on a measurement taken at step 3, and this is not that
measurement.
### What this does not answer
- **The agent's handlers are a port, not a recompile.** `harness4.c` proves the *pattern* — a C-created listener
thread can register with the runtime and call OCaml. It does not port `flan_agent.c`'s handlers, which today answer
requests out of the program's own memory and would instead be calling into the compiler.
- **Crash isolation is gone by construction.** Not a finding; item 11 already accepts it knowingly. A bad pointer
through the FFI takes the session, and conditions still catch everything the *language* signals. Noted only so this
entry stands alone.
- **macOS, for signals and for the main-thread inversion.** See above.
- **The backend.** Untouched deliberately.
### The order the real work goes in
1. **Make the `-output-complete-obj` step a dune rule**, producing the compiler-as-object that a dev build links. This
is the only build-system work, and `lib/dune`/`bin/dune` did not need changing to prove it.
2. **Land the `dynload_stubs.c` branch** (`9e0ae3a`, currently reverted). The merged build needs the same
pointer-and-scalar boundary, and it is already written.
3. **Invert the startup**: the Flan program keeps `main()`, and `flan_agent_start` also brings up the OCaml runtime on
its side thread. `merged_main.c` is the sketch.
4. **Port the agent's handlers** from answering out of the program's memory to calling the compiler directly — this is
where the socket, the wire protocol, the 4K result cap, the seqlock, the snapshot copying, the generation stamping
and the render-thunk-per-inspection all get deleted. It is the bulk of the work and the whole of the prize.
5. **Keep `llc` + `ld` + `dlopen` exactly as they are.** Item 13's third option. Nothing here argues against it.
6. **Measure what is left.** Then, and only then, item 13 step 4.
## 15. The backend spike, answered: feasible, and the obstacle is not the one anyone expected
Item 13's step 4, run early and deliberately out of order, as a spike rather than as a decision. **Feasible.** A
function written in Flan goes through the ordinary frontend, is lowered to x86-64 by hand, is written into an `mmap`
and is called, and it answers correctly. That took an afternoon and no reference material beyond `objdump`.
The apparatus is `spike/backend/`: `x86.ml` (an instruction selector), `jit_stubs.c` (three calls OCaml cannot make
for itself, plus the C side of the ABI probes), `probe.flan`, `driver.ml` and `run.sh`. `bash spike/backend/run.sh`
reproduces everything below. There is no dune file under `spike/`, nothing is wired into the build, and
`dune test --root . -j 1` is green either side of it.
**The headline is not the arithmetic.** Ten checks pass, and one of them failed first and mattered: a C callee that
does a 16-byte aligned spill and reports whether it was entered aligned. Called plainly it passed; called from inside
a binary operator it returned `-1`. The evaluator spills its left operand across the right operand's evaluation, so a
call written in the right operand runs with `rsp` 8 bytes out. **That is the raylib crash, reproduced on day one of a
backend that does almost nothing.** It is fixed with a depth counter and an assertion, and it is the clearest single
argument that this work is *tractable but unforgiving*: nothing about the wrong version looked wrong, and only a probe
built to catch it caught it.
### Question 1 — does one function work end to end
Yes. The frontend is the real one — `Reader`, `Parse`, `Load`, `Check` — so what is lowered is the same `Tast.fn`
`emit.ml` gets, not a literal typed to make the exercise come out. Eleven of the 83 functions in a trivial program
(the prelude is most of them) lower with no special handling, including `space?`, `digit?` and `upper-ascii`, which
nobody wrote for this.
The emitter is the trivial one the brief allows: every slot is a stack slot at `rbp - 8(i+1)`, every value is computed
into `rax`, a binary operator spills its left operand. Two registers, no allocator, no liveness. `spike-add` is 58
bytes where clang `-O0` would spend about 20, and that is the correct trade for a debug build.
Proved by comparing numbers, not by reading bytes. `SPIKE_DISASM=1` disassembles the buffers that ran, and that is a
debugging aid kept out of the pass/fail path on purpose: a disassembly that reads correctly beside a function
answering 656 when it should answer 650 is the normal outcome of hand-encoding.
### Question 2 — the real shape of the work
**Layout is already owned, and this is the best news in the report.** `emit.ml`'s `lay` / `lay_fields` /
`payload_lay` compute C struct layout — offsets, padding, tail padding, the union payload blob — because DWARF needs
member offsets as integer literals and `getelementptr` cannot supply one. They are acceptance-tested against LLVM's
own answer for the same struct type. So the drift risk item 10 fears most, *two backends disagreeing silently about
where a field is*, does not arise: there is one layout calculator and a new backend calls it.
Against `tast.ml`'s `expr_kind`, in four buckets:
| | nodes |
|---|---|
| **Done in the spike** | `Int` `Bool` `Local` `Do` `Let` `If` `Return` `Set`/`Plocal`, arithmetic, comparison, bitwise, `Call`, `Rt` |
| **Mechanical** | `While` `Break` `Continue` (the jump patching exists), `Global` `Str` `Zero` `Uninit`, the rest of `place`, `Field` `Deref` `Addr`, `Arr`, `Some_` `None_` `UnwrapSome`, `Match` on a tag |
| **Bulky, not hard** | floats — a second register file, SSE encodings, `Cast`'s eight conversions, and the SSE half of the calling convention. Perhaps a third of the total instruction work for a small fraction of the programs |
| **Fiddly** | aggregate copy on assignment (a struct `store` *is* the copy `spec-memory.md` requires), `Make` `MakeCase` `CaseField` over the payload blob, `CallPtr`, `FnAddr`'s three cases and the cell load behind `Fnval` |
| **No plan** | `Handled` `Signal` `RestartCase` `InvokeRestart` `WithAlloc`, the transfer-channel guard after every call, the landing pad, and `fdefers` on the transfer exit path |
The last row is the one to take seriously. The spike never emitted a guard or a pad, and the guard is on *every call
site* in the real thing — `emit.ml`'s `guard`, `current_pad`, `emit_restart_case` and `emit_with_alloc` are several
hundred lines of control flow that a second backend reimplements from the spec rather than copies. Conditions are not
an advanced feature to defer: `spec-conditions.md` is load-bearing in the prelude already.
### Question 3 — the SysV boundary, and the obstacle nobody named
**The C boundary is the easy half, and `BUILT.md` is why.** `check.ml` rejects an aggregate in a `declare` signature
and the generated shim flattens every struct, so no Flan-emitted call ever passes one to C. A string or slice crosses
as `ptr`+`len` — two arguments, which is the only counting subtlety. The spike calls an eight-argument C function
correctly, including the two that go on the stack, and sets `al` for the variadic case. **No aggregate classifier is
needed for raylib. That is a large piece of `plan.org`'s "three classifiers to write and keep correct forever" that
simply does not apply.**
**The hard half is Flan calling Flan, and it was found by reading `signature`.** That function spells each parameter
with `ll ty` and flattens nothing. Emitting a trivial program and looking at the `define` lines:
```
define i64 @"flan.take-slice"(%slice %p0, i64 %p1, ptr %xfer)
define { i8, i32 } @"flan.index-of-i32"(%slice %p0, i32 %p1, ptr %xfer)
define { i8, float } @"flan.min-f32"(%slice %p0, ptr %xfer)
define { i8, %slice } @"flan.split-next!"(ptr %p0, ptr %xfer)
define %vec @"flan.filter-i32"(%slice %p0, ptr %p1, ptr %xfer)
```
The prelude is wall to wall aggregates by value. And what LLVM does with them, measured by `objdump` on clang's own
output rather than read off a table:
- `%slice` argument → `rdi`:`rsi`, two registers, and the next argument shifts along.
- `{ i8, i64 }` return → tag in `al`, value in `rdx`.
- `{ i8, float }` return → tag in `al`, value in **`xmm0`**.
- `%vec` return (six words) → a hidden `sret` pointer in `rdi`, the real arguments shifted along behind it, and the
pointer returned in `rax`. **That pointer does not appear in the `define` line at all.**
The last two are not the C ABI, and that was checked against a control rather than asserted — the same three shapes
written in C, compiled by the same clang at `-O2`, beside the same shapes written as first-class IR aggregates:
| shape | from C | from IR |
|---|---|---|
| `{ i8, i64 }` | `al` + `rdx` | `al` + `rdx` — agree |
| `{ i8, float }` | **packed into `rax`** (`movd`/`shl`/`or`) | `al` + `xmm0` |
| `{ i64, i64, i64 }` | **`sret` pointer in `rdi`** | **`rax` + `rdx` + `rcx`** |
The 24-byte case is the striking one: C spills to memory through a hidden pointer, and the IR form returns it in three
registers, one of which — `rcx` — the SysV ABI never uses for a return value at all. Somewhere past that, LLVM does
switch to `sret`, which is what `%vec` gets.
So **the internal calling convention is not the C ABI and is not specified anywhere. It is whatever LLVM's backend
does with a first-class struct, discoverable only by disassembling.** That is the sharpest obstacle in this report,
and it is sharper than raylib for three reasons: the reference is an implementation rather than a document; the
failure mode is a garbage field rather than a link error; and it is not stable by contract across LLVM versions, which
is exactly the coupling `plan.org` chose text IR to avoid.
It also forces the decision that determines everything else:
1. **Redefinitions only** — the custom backend emits a new body into an LLVM-built host. Incremental, testable one
function at a time, and the path that fits the dev loop. It requires matching LLVM's aggregate convention
bit-exactly, including the mixed integer/SSE case above.
2. **The whole dev build** — the custom backend owns both sides and *picks* the convention: every aggregate by
pointer, nothing classified, done. No matching problem at all. But it needs complete node coverage on day one,
conditions included, and there is no partial version that runs.
The spike leaned on option 1 without noticing, because the probe called C and C is the flattened half. A real attempt
has to choose deliberately.
### Question 4 — where the language leans on LLVM instead of defining itself
The audit, which stands on its own whatever happens to the backend. Each row is drift you do not get if the language
answers it.
| | today | defined? |
|---|---|---|
| Integer overflow | no `nsw`/`nuw`, "arithmetic wraps (plan.org, Types)" | **yes** |
| Shift count | masked to the operand width; a literal out of range is rejected by `check` | **yes** |
| Evaluation order | `map_lr`, and the comment says left-to-right is *required, not a preference* | **yes** |
| Division by zero | nothing. `prelude.ml` calls a remainder by zero "immediate undefined behaviour" and routes around it | **no** |
| `INT64_MIN / -1` | nothing, and it is a separate case. LLVM says undefined; x86 `idiv` raises `SIGFPE` | **no** |
| `f64``i64` out of range | `fptosi`, undefined in LLVM; x86 `cvttsd2si` answers the "integer indefinite" value | **no** |
| `Uninit` | emitted as `poison` | **no**, and see below |
| `unreachable` | after a `noreturn` call, and after an exhaustive `match` | **no** |
| Alignment | no explicit `align` on loads and stores; LLVM uses the type's ABI alignment | implicitly, via `lay` |
Two are worth more than a table row.
**`Uninit``poison` is the one that actually bites, and it bites in the direction item 10 fears.** A hand backend
gives a stable garbage value: whatever the stack slot held. LLVM's optimiser may reason from poison and delete the
code that reads it. So `(uninit)` is the one construct where the two backends are *supposed* to differ and where
"works in dev, breaks when shipped" is the expected outcome rather than a bug. The language should say what reading an
uninitialised value means before a second backend exists, not after.
**`unreachable` is the cheap one.** The spike emits `ud2`: a defined `SIGILL` at the instruction that fell through.
LLVM's `unreachable` is undefined behaviour and licenses the optimiser to delete the path. Defining it as a trap costs
two bytes and turns a class of miscompile into a crash with an address.
None of this needs a backend. It is a session with `plan.org` and six `check.ml` cases.
### Question 5 — unloading code, which is the prize
**The shadow stack is better than expected and still not sufficient, and the two halves of that are separate
questions.**
*The running half — and the shadow stack does answer it.* The frame push in `emit_fn` is inside a plain `if m.dev`
and is **not** gated: only the *slot table* is gated on `named && n > 0`, and a function with no named slot still
pushes a frame with a null `slots`. So every active Flan function in a dev build is on the chain, lifted handler
clauses included, and each frame points at the `flan_fninfo` belonging to the module it was compiled into — so the
pointer identifies not just the function but *which body*. "No frame on the chain names this body" is answerable
today, with no DWARF and no unwinder.
Two caveats on that half. The pop happens before `leave; ret`, so a body is briefly executing with no record —
irrelevant if reclamation happens at a safe point on the same thread, fatal if another thread reclaims while the game
thread is returning. And the chain is a plain global, not thread-local, which `flan_dev.c` states and justifies.
*The pointed-into half, which the shadow stack cannot see and which is the actual reason nothing is `dlclose`d today.*
`BUILT.md` is explicit, and it is not the reason the brief assumed: "a cell holds an address inside a module's text;
unloading it leaves every call site pointing at unmapped memory. The rule is about being *pointed into*." Owning the
code object answers most of this — you own the cells, so redefinition drops the old body's last cell reference — but
not all of it:
- `FnAddr (Fnval n)` **loads the cell and yields a raw body address**, which can then be stored in a struct, a `Vec`
or a global. Nothing records that it happened.
- `FnAddr (Flanfn _)` and `(Rtfn _)` bypass the cell *by design*`tast.ml` says they "must never take that path" —
so a `Map`'s hash and equality pair and a handler-bind clause's address are raw pointers into a specific body, held
in data.
So: **frames are tracked, escaped code pointers are not.** Unloading needs a rule the language does not have yet. The
cheapest honest one is deferred reclamation — retire a body when no frame names it *and* an epoch has passed with no
new capture — and the cleanest is to make a function value a cell pointer rather than a body pointer, which costs one
indirection on `CallPtr` in dev builds and makes the whole question go away. That second option is worth writing down
now whatever happens to the backend, because it is a change to what a `Fn` value *is*.
### The verdict
**Feasible, unforgiving, and not the next thing to do.**
Feasible: the instruction selection is easy, layout is already owned and tested, the C boundary is already flattened,
and one function ran on day one. Nothing here argues the way item 10 feared — the divergence hazard is real but it is
concentrated in three named places (`Uninit`, division, the float cast), not spread through the whole of arithmetic.
Unforgiving: the internal aggregate convention is defined by LLVM's implementation and not by any document, the
alignment rule is invisible until raylib crashes somewhere else, and conditions are a second full implementation of
`spec-conditions.md` rather than a port.
Not next: item 13's order still holds, and the spike does not disturb it. Transport is 41µs of a 21ms redefinition and
code generation is 19 of the 21, so the *speed* case remains what item 13 said it was. What this spike adds is that
the **introspection** case is also not free — unloading needs a rule about escaped function values that nothing in the
language has, and that rule is worth having whether or not a backend is ever written.
**What to do first if it went ahead**, in order, and the first two are worth doing regardless:
1. **Define the six undefined cases** (question 4). No backend required, and every one is drift avoided rather than
drift managed.
2. **Decide what a `Fn` value is** — body pointer or cell pointer — and write it down. This is the unloading question
and it is a language question, not a backend one.
3. **Choose option 1 or option 2 from question 3**, deliberately. Everything else follows from it.
4. **Only then**, and only if 3 says so, grow `spike/backend/x86.ml` from the node table in question 2 — floats
first, because they gate most of the prelude, and conditions last, because they are the only row with no plan.
## 16. The dev backend, wired: a whole program compiles through `x86.ml` and runs, and conditions were never in the way
Item 15's step 4, taken. `lib/x86.ml` was 502 lines of encoder and frame model with nothing calling it. It now lowers a
whole `Tast.program` to an assembly file, and `flan build --x86` hands that file to the same clang invocation the LLVM
path uses, against the same runtime objects, the same generated shim and the same linker arguments. The flag is off by
default and refused in combination with `--dev`, `--debug`, `--sanitize` and every wasm target. **LLVM stays the release
backend and the default one; nothing on the existing path changed.** `dune test --root .` is green either side.
**41 programs out of `test/programs` build through it, and 40 of them print exactly what the LLVM build prints.** The
41st is `bounds.flan`, and it diverges on purpose — see question 4.
### Question 1 — which program, and what did it actually cost
Measured with `spike/backend/hist.ml` before anything was written, over candidates, not guessed. The one picked is
`spike/x86/p3-fizz.flan` — a `dotimes`, a call, an `if`, a remainder, two string literals and `print`/`println`:
```
p3-fizz.flan: 2 reachable fns
Call 1 Do 4 If 1 Int 15 Let 2 Local 7 Prim 14
Set 1 Str 3 While 1 place/Plocal 1
prim/Add 2 prim/Bytes 3 prim/Cast 1 prim/Eq 1
prim/I64ToBytes 1 prim/Lt 1 prim/Rem 1 prim/WriteStdout 4
```
Nineteen rows, two reachable functions after `Reach` prunes, and **no `Signal`, no `Handled`, no `RestartCase`, no
`Make`, no `Field`, no allocator**. The same is true of a program that only loops and prints: zero of each.
That is worth stating flatly because the expectation going in was the opposite — that the smallest useful program
carries one of each condition node, and that printing a single value drags in `Str`, `Make`, `Field` and `Call` because
the prelude builds a slice to do it. **It does not.** Printing an integer is `Cast``I64ToBytes``WriteStdout`,
three prims and no call at all; `flan_i64_to_bytes` renders into a static buffer in `flan_rt.c` and hands back a slice.
Printing a string literal is `Str``Bytes``WriteStdout`, and `Bytes` is a non-instruction because a string and a
`[u8]` are the same two words.
**What does drag conditions in is the bounds check and the allocator, and neither is a `Tast` node.** `check_at` and
`check_slice` call `flan_bounds_error` with the transfer channel and then `guard`; `Rt flan_vec_at` takes the channel
too. So it is `m.checks` and the container runtime that make a program need the condition machinery, not looping and
not printing — and because both are *emit-time* constructs rather than IR nodes, `hist.ml` cannot see either. That is
the correction to item 15's node table: the "no plan" row is not reached by writing a loop, it is reached by writing
`(at a i)`.
### Question 2 — what runs
| program | what it is for |
|---|---|
| `spike/x86/p1-exit.flan` | `main` returns 0. The assembly path, the runtime link, a real executable. |
| `spike/x86/p2-loop-print.flan` | a `dotimes` that prints — the smallest program that does something |
| `spike/x86/p3-fizz.flan` | the measured target: a call, an `if`, `%`, two string literals |
| `spike/x86/p4-convention.flan` | the *internal calling convention*, which p3 does not touch at all |
| `spike/x86/p5-core.flan` | a global with an initialiser, recursion, `break`, `continue`, the bitwise family, unsigned shifts, both directions of every conversion |
p4 is the one that matters most, because item 15 named the internal aggregate convention as the sharpest obstacle in
the whole report. It passes a struct by value, returns a struct by value, puts an `f32` through the SSE half, calls an
eight-argument function so that two arguments go on the stack, and passes a slice — and it agrees with LLVM. **The
obstacle really did dissolve the way the header claims**: a dev build is compiled entirely here and a release build
entirely by LLVM, the two never meet in one process, so the convention is ours to pick. Every aggregate goes by
pointer, an aggregate return is a hidden pointer in the first integer register returned in `rax`, and there is no
classifier in the file. Nothing had to be discovered by disassembling clang.
Over `test/programs` (111 files):
| | n |
|---|---|
| built through `--x86` and **matched** the LLVM build's output and exit status | **40** |
| built and diverged — `bounds.flan`, by design | 1 |
| refused by name: a node this backend does not lower | 40 |
| no `main` (package and library fixtures) | 6 |
| do not compile at all (the checker-error fixtures) | 22 |
| never terminate on their own (`dev-loop`, `dev-watch`) | 2 |
So of the 81 programs that compile, have a `main` and finish, **41 went through the hand-written backend and 40 were
byte-identical in output.** That includes `edn.flan` — sixty lines of output from a hand-written EDN reader with
unions, options, nested collections and a fixed-depth balance stack.
Proved by comparing output, never by reading bytes. The script builds both ways and diffs stdout and the exit status;
`objdump` was used only after a program already had the wrong answer. Item 15 is right that this is the only honest
order.
### Question 3 — the two bugs, and both are the shape item 15 predicted
**A `(set (.x (at pts 0)) 1.5)` wrote into a copy.** `lvalue` had no case for `At`, so it fell through to "evaluate it",
and the store landed in a temporary while the array kept its zeros. `emit.ml` has this as `addr`'s own `At` case. One
line. `array-ctor.flan` found it, and it found it as a segfault several statements later.
**A discarded value was stored over the return address.** This is the better one. A form whose value is thrown away was
handed a sink, and the sink was spelled as an address — `rbp+0`. That is the saved `rbp`, and `rbp+8` is the return
address, so a non-void form written in statement position stored straight over both; a 16-byte slice did it in one
`rep movsb`. `edn.flan` crashed by jumping into `.rodata`, **several statements after the mistake and in a different
function**, and the assembly at the jump read perfectly. The sink is now compared by identity and never used as an
address: anything with a value that is handed it gets a frame temporary instead.
The second bug cannot exist on the LLVM path, and that is the general point. LLVM has no notion of "store this value
nowhere" — an unused SSA value is simply unused. Every construct this backend has that LLVM does not is a place where
a bug can live that the LLVM backend's own testing can never have covered.
Against that, the thing item 15 was most worried about did *not* happen: **nothing went wrong with the frame or the
stack alignment.** `rsp` is written exactly twice — one rounded `sub` in the prologue that covers the temporaries and
the outgoing-argument area together, and `leave` — so `rsp % 16 == 0` at every call site is a property of one
subtraction rather than an invariant every case maintains. The spike's worst bug has no door to come in by, and p4
calls an eight-argument function to prove it.
### Question 4 — where the two backends now differ, and it is three named places
Item 15's question 4 listed nine undefined cases. Three of them are now *real* divergences with a build on each side,
and they should be written down before anyone uses this for anything.
| | LLVM | here |
|---|---|---|
| **A bounds violation** | `flan_bounds_error` signals; a `restart-case` can catch it; `bounds.flan` exits 134 | **no check at all**; `bounds.flan` exits 139 |
| `(uninit)` | `poison`, and the optimiser may reason from it | whatever the stack slot held — stable garbage |
| an exhausted `match`, a `noreturn` call | `unreachable`, undefined | `ud2` — a defined SIGILL at the instruction that fell through |
**The first is the one that matters, and it is not a footnote: `--x86` is silently a `--no-bounds-checks` build.** It is
silent because it is not a decision the backend made — `check_at` signals, signalling needs the channel and the guard,
and there is no guard here, so there is no check. `bounds.flan` is the direct evidence and `edn.flan`'s 33-brackets-
against-a-32-deep-stack case is the second. Anyone reaching for this flag on a program that indexes anything should
know that the trap is gone.
The other two are improvements and cost nothing. `ud2` in particular is two bytes and turns a class of miscompile into
a crash with an address.
### Question 5 — conditions, which is still the row with no plan
**They were not reached, and converting that into a checked precondition is the most useful thing in this report.**
There is no transfer guard after a call here, no landing pad and no transfer exit. `emit.ml` emits a guard after *every*
call; this emits none. What makes that sound is a whole-program argument rather than a hope: **if nothing in the
reachable set can ever write the channel, no call can ever return with it set.** So `check_no_transfer` walks the
linked program once per build and stops it — with the node's name and the function it is in — the moment it finds a
`signal`, an `invoke-restart`, a `restart-case`, a `handler-bind`, a `with-allocator`, or the two `Rt` symbols whose
bounds check signals.
That is what the 40 refusals are:
```
27 restart-case 4 handler-bind 1 with-allocator
7 signal 1 defers on the transfer path
```
Forty programs refused by name rather than miscompiled, and one line of build output says which node and where. A
backend that quietly omitted the guard would have compiled all forty and been wrong in a way no test distinguishes
from a race.
What this does *not* do is measure what conditions cost. That is still unknown, and it is still the only row of item
15's table with nothing behind it. What is now known is the shape of the bill: the guard is per call site, the pad is
per `restart-case` activation, `fdefers` needs a second exit path that no form in `body` can reach, and **any function
with `fdefers` at all is refused today** — which is most of the prelude's file and container code, and is why the
programs that use a `Vec` are not in the 40.
### The honest no-plan bucket
Everything below is refused by name at build time, not silently wrong.
- **Conditions, entire** — the guard, the landing pad, `emit_restart_case`, `emit_with_alloc`, the transfer exit, and
`fdefers` on it. Several hundred lines of `emit.ml` reimplemented from `spec-conditions.md` rather than ported.
- **Bounds checks**, which are the same work: `check_at` and `check_slice` cannot exist without the guard.
- **`Rt` with an aggregate return**, and with it most of the container runtime; `Vec`, `Map` and `Pool` have not been
exercised at all.
- **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` emits the symbol, which is correct for a whole-program build and
wrong the instant anything is redefined into it. This backend has no cells and no `--dev`; that is a deliberate
restriction and not an oversight, but it is exactly item 15's question 5 waiting where it was left.
- **`f64``i64` out of range**, and **`INT64_MIN / -1`**. `idiv` raises `SIGFPE` where LLVM says undefined, and
`cvttsd2si` answers the integer-indefinite value. Unchanged from item 15: these want a language decision, not a
backend.
- **Debug information.** None. `--x86` and `--debug` together are refused.
- **Code size and speed.** Not measured. Every value is in memory, every intermediate is a frame temporary, and a
block copy is `rep movsb`; that is the trade the brief asks for and nobody has put a number on it.
### The verdict
**The wiring is done and it was the easy half. What is left is conditions, and the measurement moved them from "first
obstacle" to "the only obstacle".**
The order item 15 recommended was floats first and conditions last. Floats turned out to be one afternoon's encodings
and they are done. Conditions are still last and are now the *whole* remainder: they are what stands between 41
programs and the corpus, they are what a bounds check is made of, and `check_no_transfer` is the line that says so out
loud on every build until someone writes them.
Two things are worth doing before that, and both are cheap. Decide what a bounds violation means in a build with no
handler — because "no check" is what it means today and nothing says so. And take item 15's question 4 seriously now
that there are two backends to disagree: `(uninit)` and `unreachable` already differ, deliberately, and the difference
is currently documented only in a comment in `x86.ml`.
## 17. Conditions on the x86 backend, and with them bounds checks: the corpus, not 41 programs
Item 16's verdict was that conditions were the only obstacle left and that a bounds check was made of the same parts.
Both halves held. The transfer channel's guard, the landing pads, the per-function transfer exit, `fdefers` on it,
`emit_restart_case`, `emit_with_alloc`, `signal`, `error`, `handler-bind`, `invoke-restart`, `check_at` and
`check_slice` are all in `lib/x86.ml` now, written from `spec-conditions.md` and `emit.ml`'s semantics rather than
ported. **LLVM is untouched and still the default and the release backend, and `--x86` is still off by default and
still refused with `--dev`, `--debug`, `--sanitize` and every wasm target.**
`dune test --root .` was green twice on this work, and is green now apart from four raylib fixtures — `images` and
`audio`, each at `-O2` and `-O0` — which export to a *hardcoded* `/tmp` path and fail with
`Failed to export wave data` because `/tmp` is full. It is full because of a runaway `llc` in another lane writing a
5.8 GB `m4.o` out of a 13 KB `m4.ll`; the same four fail on two consecutive runs and nothing else does. Nothing in
this lane writes to `/tmp` by a fixed name, and `TMPDIR` does not reach those fixtures.
### Question 1 — the counts, and what was compared
`spike/x86/survey.sh`. Item 16 describes a script that builds every program both ways and diffs the output; it was
never committed, so this one is. It builds each program in `test/programs` and each probe in `spike/x86` twice — once
default, once `--x86`, **with the same bounds-check setting on both sides**, because a checked build compared against
an unchecked one says nothing about `bounds.flan` — runs both, and compares **stdout, stderr and the exit status**.
stderr is not a detail. Every message the new machinery produces goes there — the bounds and slice errors, the three
restart refusals, the transfer failure — and each carries a `Loc.to_string` string this backend emits by hand as a
`.rodata` label and a length in a register. An exit status of 134 with the wrong text beside it is exactly the failure
that reads as a match.
| | before | after |
|---|---|---|
| **MATCH** — same stdout, same stderr, same exit status | **41** | **89** |
| **DIFFER** | 1 (`bounds.flan`) | **0** |
| **refused by name** — a node this backend does not lower | 41 | **0** |
| skipped: does not compile (checker-error fixtures) | 25 | 25 |
| skipped: no `main` (package and library fixtures) | 6 | 6 |
| skipped: never terminates (`dev-loop`, `dev-watch`) | 2 | 2 |
The "before" row is measured, not quoted from item 16 — it is the same corpus seven files larger, and the 41 refusals
split as 26 `restart-case`, 7 `signal`, 4 `handler-bind`, 1 `with-allocator`, 1 `fdefers`, and one each for
`flan_vec_at` and `flan_vec_as_slice`.
**Every program in `test/programs` that compiles, has a `main` and terminates now goes through the hand-written backend
and agrees with the LLVM build.** That is the whole corpus: `restarts.flan`, `conditions.flan`, `bounds.flan`,
`bounds-condition.flan`, `allocators.flan`, `defers.flan`, the `Vec` and `Map` programs, `edn.flan`, `format.flan`.
The programs that only trap when given an argument — `restarts.flan`'s four signature-mismatch cases, `bounds.flan`'s
three — were run by hand with their arguments and agree on stderr and on exit 134 as well.
### Question 2 — what a bounds violation means in a build with no handler
**The same thing it means on the LLVM path, and that is the answer rather than a decision.** `check_at` and
`check_slice` here are `emit.ml`'s: a compare, a branch, a call to `flan_bounds_error` or `flan_slice_error` with the
transfer channel, and then **the guard** — which is why they could not exist before. The call is an ordinary one that
returns only when a handler or the break loop transferred, so the guard is the way out and the fall-through past it is
`ud2` where `emit.ml` writes `unreachable`.
So: a bounds violation signals `BoundsError`; a `handler-bind` can answer it; a `restart-case` catches the transfer
and its clause's value stands; an unanswered one dies inside the runtime and the program exits 134 with the location
and the index. `--no-bounds-checks` omits the check on both backends and both then exit 139. **`bounds.flan` has
stopped being a DIFFER**, and the row of item 16's question-4 table that said "no check at all" is gone rather than
documented.
The transitional refusal item 16 asked for — a build that says out loud that its behaviour differs — was not written,
because `check_at` landed in the same pass and the refusal would have been created and retired inside one commit.
There is nothing left to be loud about.
Item 16's other two divergences are unchanged and still deliberate: `(uninit)` reads whatever the slot held rather
than LLVM's `poison`, and an exhausted `match` is `ud2` rather than `unreachable`. Both are still documented only in
`x86.ml`.
### Question 3 — `check_no_transfer` narrowed, not removed
It was a whole-program argument: this backend emitted no guard, which is sound exactly when nothing reachable can
write the channel, so the build refused by name the moment it found something that could. Every call site is guarded
now and the argument has retired — **in every function.** It still stands in one place, so the walk is still there and
now walks only global initialisers:
A global's initialiser runs from `flan..init-globals`, before `main` and before anything has established a handler or
a restart. It owns its own channel cell because no caller hands it one, so a transfer out of it has nowhere to go —
its exit would return into the loader. `signal`, `error`, `restart-case` and `handler-bind` in a `defvar` initialiser
are refused by name. A bounds check there is *not* refused: it signals into a cell nothing is listening on, finds no
handler, and dies, which is the right answer.
### Question 4 — five bugs, and four are item 16's shape exactly
Found by what the programs printed, never by reading bytes. Two of them existed before this work and only became
reachable once the guard let the programs that expose them compile.
**The body fell through into the transfer exit.** Every `fdefer` ran twice on a normal return, so `restarts.flan`'s
`log` was one too high at every checkpoint and nothing else was wrong. `emit.ml` cannot have this bug: its `ret`
terminates the block, and there is no fall-through to forget. This is the same class as item 16's "a discarded value
was stored over the return address" — a construct this backend has that LLVM does not.
**A `Vec` crossed to the runtime as the address of a copy.** `eval` copies an aggregate into a temporary, so
`flan_vec_push` grew the temporary and the caller's header stayed at length zero — and an *in-bounds* `(at v 1)` then
signalled against a length of 0. `emit.ml` says so in a comment beside its own `addr`; this had no such case. Pre-
existing, and invisible until a program using a `Vec` could build.
**`ucomis` sets CF, ZF and PF together for a NaN**, so `sete` answered *true* for `(= x x)`. Flan's comparisons are
LLVM's *ordered* ones (`oeq`, `olt`, ...), which are false for a NaN; the unsigned table is not those. `<` and `<=`
now swap their operands and ask for a/ae, and `=` and `!=` take a `setnp` beside them. The symptom was
`(/ 0.0 0.0)` formatting as `-9223372036854775808`, because the prelude's NaN test is `(not (= x x))` and nothing
else. Pre-existing; `format.flan` could not build before.
**A union read field 0 through the struct table** and was refused by name. A union is a tag and a payload blob, which
is a two-field struct at that level, and the structural printer reads the tag without unwrapping the value.
**And one that could not have been found later:** `emit_globals_init` stored a null *into* the channel slot rather
than a cell's address into it, so every callee of a global initialiser was handed a null pointer to write a transfer
through. Harmless while nothing could transfer; a fault the first time a guard loaded through it. `emit_main` had it
right and was the model.
Against those: nothing went wrong with the frame, the stack alignment, or the pads' nesting. The one place worth
naming is the one item 16 could not have: **the channel is one indirection deeper here than in `emit.ml`.** There
`%xfer` is an alloca and the target is one `load` away; here `xfer_off` is a frame slot *holding the caller's
pointer*, so reading the target is two loads and clearing the channel is a store *through* the pointer and never a
store to the slot. `chan_into`, `xfer_load`, `xfer_store` and `xfer_clear` exist so that no call site has to remember
which.
### Question 5 — what the corpus does not walk, and the probe that does
Every pad has two halves: the one a body reaches by finishing, and the one a transfer reaches by passing through. The
corpus walks the first everywhere and the second in one place only — `allocators.flan`'s last case aims an
`invoke-restart` out of a `with-allocator` body at a `restart-case` outside it, which is the `wxfer` re-propagation.
The other two it never reaches. In `restarts.flan` every transfer stops at a `restart-case` *inside* the
`handler-bind`'s extent, so the handler frames never come off on the transfer path; and in `nested` and `shadowed` the
*inner* restart frame offers the name, so a restart-case that the transfer is not aimed at never has to put the target
back. `spike/x86/p6-transfer.flan` is those two, beside a defer and a clause parameter, and it agrees with the LLVM
build. It is in the survey, and it is why the count is 89 rather than 83.
The branch nothing exercises is `flan_transfer_fail` — a defer that starts a *second* transfer while the first is
unwinding. It is emitted and refused loudly, and it is untested.
### The honest no-plan bucket
- **`Rt` with an aggregate return.** Still refused by name. `flan_vec_as_slice` returns a slice by value, and
`bounds-condition.flan` exercises it both in and out of bounds through a `restart-case` and matches — so it does not
reach the refusal, and **nobody traced why.** That is a loose end, not a result.
- **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` still emits the symbol. Correct for a whole-program build, wrong
the instant anything is redefined into it; there are no cells here and no `--dev`, deliberately.
- **`f64``i64` out of range**, and **`INT64_MIN / -1`**. Unchanged from items 15 and 16: `idiv` raises `SIGFPE`
where LLVM says undefined. A language decision, not a backend one.
- **`(uninit)` and `unreachable`** still differ from LLVM on purpose and are still written down only in a comment.
- **The `flan_transfer_fail` branch**, above.
- **`"defers on a transfer path nothing reaches"`** — the refusal that replaced the old `fdefers` one. No program in
the corpus hits it; it exists so that if the reasoning behind it is ever wrong, it says so.
- **Debug information.** None. `--x86` and `--debug` together are still refused.
- **Code size and speed.** Still not measured, and now there is more to measure: a guard after every call, two loads
and a branch each, and a bounds check that spends three frame temporaries. Nobody has put a number on any of it.
### The verdict
**The row with no plan is gone.** What item 15 called the last obstacle and item 16 called the only one is written,
and the measurement that made it the only one is the measurement that says it is finished: every program in the corpus
that can run, runs, and prints what LLVM's build prints — down to stderr.
What is left in `x86.ml` is not conditions. It is a container return convention, a redefinition cell, two arithmetic
edge cases the language has not decided, and no debug info. None of those is the shape conditions were: each is a
known thing in a known place, and the guard is not underneath any of them.
## 18. The container return convention was never there, and the redefinition cell now is
Item 17's no-plan bucket opened with two correctness items and a loose end it was honest about. Both items are closed
and the loose end is the reason the first one closed the way it did rather than the way it was written down. **LLVM is
untouched and still the default and the release backend.** `--x86` is still off by default and still refused with
`--debug`, `--sanitize` and every wasm target; it is **no longer refused with `--dev`**, and question 3 is the argument
for that.
### Question 1 — the counts, and a corpus that moved
`spike/x86/survey.sh`, unchanged in what it compares: every program in `test/programs` and every probe in `spike/x86`,
built both ways with the same bounds-check setting, run, and diffed on **stdout, stderr and exit status**.
| | item 17 | measured here, before | after |
|---|---|---|---|
| **MATCH** | 89 | **93** | **97** |
| **DIFFER** | 0 | 0 | **0** |
| **refused by name** | 0 | **2** | **0** |
| skipped: does not compile | 25 | 28 | 28 |
| skipped: no `main` | 6 | 6 | 6 |
| skipped: never terminates | 2 | 2 | 2 |
**The baseline is not the one the brief quoted, and that is the first finding.** Item 17 measured 89/0/0; this lane
measured 93 MATCH and **2 refusals** before writing anything. The corpus moved underneath: another lane landed
`(slice-from-ptr p n)`, and it arrived as two refusals rather than one — `slice-from-ptr.flan` and `bounds.flan` both
stopped building through `--x86`. A backend that refuses by name does not rot quietly, but it does rot, and nothing
was watching. That is an argument for running the survey in CI rather than in a lane.
The form itself is nothing: a `Slice _` is `{ptr, i64}` here exactly as it is in `emit.ml`, so it is one store of the
pointer and one of the length and no new representation at all. The half worth writing down is that **the check has to
be signed**. There is nothing to compare the length against — only the caller knows what is behind that pointer — so
what is checked is that the promise is not absurd, and `check_slice`'s own compares are *unsigned*. A negative `i32`
sign-extended to 64 bits is a huge unsigned value that `jbe` waves straight through, and the result is a slice about
2^64 long that reads as a pass and faults somewhere else entirely.
Nothing in the corpus walks that path, because every length in `slice-from-ptr.flan` is a literal and a negative
literal is refused by `check.ml` before any code is emitted. `spike/x86/p7-slice-from-ptr.flan` takes the length as a
parameter and runs it through a `restart-case`, which puts the condition's `low`/`high`/`length` on stdout beside the
LLVM build's.
The other two of the four new MATCHes are this lane's own probes, below.
### Question 2 — why `flan_vec_as_slice` avoided the aggregate-return refusal
**It is the first of the two possibilities item 17 named: the refusal is narrower than it reads, and nothing is going
right by accident.**
`flan_vec_as_slice`'s Flan-level return type is `Unit`. `check.ml:3721` builds it as `rt loc Types.Unit`,
`flan_rt.c:1165` is `void flan_vec_as_slice(flan_vec *v, void *out, ...)`, and `emit.ml:2464` declares it `void`. So
`is_void rty` answers first and the `is_agg rty` test below it is never reached. The slice comes back through an
out-pointer the checker allocated, which is not one symbol's accident but the convention:
- **Every aggregate-valued runtime result crosses through an out-pointer.** Every other `rt` builder in `check.ml`
answers `Unit`, an `Int`, a `Ptr`, an `Alloc` or a `Handle`. `flan_pool_resolve` answers `Ptr elem` and the `Option`
is built in Flan; `Argv` is its own `Tast` node with its own out-pointer and never comes through here at all.
- **`crossable`**, the other user of the same code path, admits `String` and `Slice _` only as *a parameter* and
refuses an aggregate return from a `declare` outright.
**So there is no sret convention to build for `Rt`, and building one would have been worse than the refusal.** This is
the C boundary, where the header says the backend must match SysV rather than pick: a 16-byte slice comes back in
`rax:rdx`, not through the internal hidden-pointer convention. There is no classifier in the file, there is nothing to
test one against, and "untestable and wrong" is a bad trade against a line that costs nothing. The line stays as a
guard against those two rules changing, and now names which rules and what the work would actually be.
**Item 16's claim that the container runtime is unexercised went with it, and it was already stale when item 17
repeated it.** `Vec` and `Map` run through `vec.flan`, `vec-of-vec.flan`, `vec-in-struct.flan`, `maps.flan` and
`map-iter.flan`; `Pool` — which neither report checked — runs through `registry.flan`, `handles.flan`,
`generics.flan` and `pool-stale-region.flan`. All match, and they have since item 17's guard landed.
### Question 3 — the cell, and why `--x86 --dev` is no longer refused
`FnAddr (Fnval n)` emitted the symbol. It now reads the cell — and **so does every direct call**, which is the half
that matters and is what `emit.ml`'s `body_of` does: a redefinition is one store, and its whole purpose is to reach
call sites that already exist. What is emitted, all of it behind `dev`:
- **One cell per function** in `.data`, `.globl`, initialised to the body this build compiled. Spelled exactly as
`Emit.cellname` spells it, because that is the point of having one here — an LLVM-built redefinition module binds
`@"flan.cell.<n>" = external global ptr` against whatever built the host. `nm -D` over the two builds of the same
program gives identical sets of 68 cell symbols.
- **The cell load placed after the arguments.** `emit.ml` has that as a load-bearing comment: a redefinition landing
between two calls must not land in the middle of one. `CallPtr` stays the other way round, for `emit.ml`'s reason.
- **The `flan_dev_reg_enable` constructor**, which arms the allocation registry.
Not emitted: `Emit.cellptr`, the deeper spelling for a name the host was never built with. It cannot arise in a
whole-program build, where `known` is true of everything, and it belongs with the redefinition module that would
introduce such a name.
**The refusal is relaxed, and the argument is that `flan dev` never reaches that fork.** `--x86` is read in exactly one
place, `flan build`'s argument list; the daemon builds its host through `Build.executable` and its modules through
`Build.shared` without it, and there is no spelling that hands it one. So the flag now means what it says — a host
whose call sites are redefinable, built by this backend — and nothing claims the module that would redefine through
them exists.
### Question 4 — what made that believable, because the corpus cannot
Two measurements, and the second is the one that matters.
**The corpus with `--dev` on both sides: 97 MATCH, 0 DIFFER.** `SURVEY_FLAGS=--dev` is opt-in so the counts in question
1 stay the same measurement. Before the constructor was added it read **96/1**: `registry.flan` asks `(live? ...)` and
got four zeroes, because the allocation registry was never armed. That is the whole of what a dev host does
differently besides the cells, and it is worth saying that the corpus found it — one program out of 97 observes it.
**And the thing the corpus structurally cannot test.** A dev build starts with every cell pointing at the body this
build compiled, so it prints exactly what a release build prints *whether or not anything reads the cell*. The
property that makes the whole corpus a safe test of the cells is the property that makes it a useless one.
So `spike/x86/cells.sh` changes what a cell holds. It preloads a shared object whose constructor looks up
`flan.cell.twice` with `dlsym` — the cells are in `.dynsym` because a dev build is `-rdynamic` — and stores a different
body there. That is the one store a redefinition ends in, done from outside, with no compiler involved. Four builds,
and the two controls are half the test:
| | |
|---|---|
| `llvm --dev` | 22 22 — the cell is read |
| `x86 --dev` | 22 22 — **this lane's claim** |
| `llvm` | 42 42 — no cell; `dlsym` answers NULL |
| `x86` | 42 42 |
`spike/x86/p8-cell.flan` has both call shapes, because they are two different cases in both backends: a direct call by
name, and a function *value*, which is the one `FnAddr` that is not the symbol. The release rows are what say the
change came from the indirection and not from ordinary symbol interposition.
### Question 5 — the licence in the header now has an edge, and it is the cell
This is the thing the next lane inherits, and it is worth more than either item above.
`x86.ml`'s header licenses its own calling convention on the grounds that **a dev build is compiled entirely by this
backend and a release build entirely by LLVM, so the two never meet in one process.** That is what dissolved item 15's
sharpest obstacle and it is why there is no classifier in the file.
**Publishing a cell an LLVM-built module can store into is the first thing that could make it false.** The two
conventions agree on scalars and disagree on every aggregate — this backend passes each by pointer and returns one
through a hidden `sret`, LLVM classifies — so an `Emit.redefinition` module dlopened into an `--x86` host would be
correct exactly until the first redefined function took or returned a struct. `cells.sh` does not reach it, because
the body it installs is `(i64, void *) -> i64` and the conventions agree there.
Nothing in the toolchain does that today: `flan reload` and `flan dev` build host and module through LLVM together,
and neither accepts `--x86`. But the lane that wires this backend into the dev loop will be the one that does it, and
**the answer then is a redefinition emitter here, not a classifier.** Written into both `x86.ml`'s header and
`build.ml`'s refusal so that it is found before it is discovered.
### The honest no-plan bucket
- **A redefinition emitter**, which is question 5 and is new to this list: `Emit.redefinition` has no counterpart here,
so a `--x86 --dev` host has cells nothing in the toolchain can yet write.
- **`f64``i64` out of range**, and **`INT64_MIN / -1`**. Unchanged from items 15, 16 and 17: `idiv` raises `SIGFPE`
where LLVM says undefined. A language decision, not a backend one, and nobody has taken it in three reports.
- **`(uninit)` and `unreachable`** still differ from LLVM on purpose and are still written down only in `x86.ml`.
- **The `flan_transfer_fail` branch** — a defer starting a second transfer while the first unwinds. Emitted, refused
loudly, still untested.
- **`"defers on a transfer path nothing reaches"`** — still reached by no program in the corpus.
- **`flan_dev_reg_note` is not dropped in a release build here.** `emit.ml` drops the whole family when `dev` is off;
this emits real calls to a registry that is disabled, so they are no-ops that cost a call each. Correct, not free,
and `emit.ml`'s stated reason for the drop — an escaped alloca `mem2reg` would refuse — does not apply to a backend
with no `mem2reg`.
- **Debug information.** None. `--x86` and `--debug` together are still refused.
- **Code size and speed.** Still not measured, and the list of what to measure has not got shorter: a guard after every
call, a bounds check spending three frame temporaries, every intermediate in memory, `rep movsb` for a block copy —
and now an extra load at every call site in a dev build, which is the one item on this list that `emit.ml` pays too.
### The verdict
**Neither of the two correctness items was the shape it was written down as, and finding that out was most of the
work.** The container return convention did not exist to be built; the loose end item 17 flagged was the answer and
not a symptom, and one afternoon spent tracing it saved a SysV classifier nobody could have tested. The cell was real,
took thirty lines, and could not be tested by anything in the corpus — which is why the useful artefact from it is a
preloaded `dlsym` and not a program.
What is left is a redefinition emitter, two arithmetic edge cases the language still has not decided, and no debug
info. The backend is no longer the thing standing between here and the dev loop.