1433 lines
101 KiB
Markdown
1433 lines
101 KiB
Markdown
# Let's discuss
|
||
|
||
Open questions, raised and deliberately not answered yet. Nothing here is a decision or a task. Each entry is the
|
||
question as asked, plus what is already known in this repo that bears on it — so the investigation starts from what
|
||
exists rather than from scratch.
|
||
|
||
Settled decisions live in `NEXT.md`. Reasons for what already exists live in `BUILT.md`.
|
||
|
||
---
|
||
|
||
## 1. `i`, the inspector, and the frame it cannot see — answered and built
|
||
|
||
Answered, and built as option 2. `BUILT.md`'s "Two ways to root a walk, and why neither subsumes the other" is where
|
||
it lives now, including the correction to what this entry said about rooting at an address. The number is kept because
|
||
other files cite these by number; nothing here is open.
|
||
|
||
## 2. Annotating the IR and the disassembly with the source
|
||
|
||
**The IR half is nearly free and should just be done.** `emit.ml` writes `.ll` as text, so a comment costs nothing and
|
||
cannot break anything, and every typed IR node already carries a `Loc.t`.
|
||
|
||
**The disassembly half, with the optimisation question settled.** `objdump` already interleaves source into a listing
|
||
when DWARF is present (`-S`), and the daemon already shells out to objdump — so the `-O0` case is close to a flag.
|
||
|
||
Settled in conversation: **`-O0` is expected to follow the source and gets the full annotation; `-O2` is not expected to
|
||
and gets either nothing or whatever best-effort mapping falls out.** That removes what looked like the blocking
|
||
question. `--debug` forcing `-O0` is therefore fine and does not need decoupling for this.
|
||
|
||
**How SBCL does it, since it came up.** SBCL does *not* shell out. `sb-disassem` is its own disassembler, written in
|
||
Lisp, that knows the instruction encodings directly. What that buys is annotation from the inside: it labels constants
|
||
the function references, names the functions being called, marks entry points, and shows its own calling conventions —
|
||
because it compiled the code object and still holds the metadata.
|
||
|
||
The relevant observation is that **this project is closer to SBCL's position than the objdump route suggests.** The
|
||
daemon owns the build and holds the metadata too; it simply is not feeding much of it into the listing yet. Naming the
|
||
function behind an indirection cell, or a constant by its source name, needs no instruction decoder — only the
|
||
information the daemon already has. Writing a disassembler is not the interesting part and should stay off the table;
|
||
richer annotation of objdump's output is cheap and is where SBCL's advantage actually comes from.
|
||
|
||
## 3. `def`, `defvar`, `defconst` — three roles, currently two words
|
||
|
||
There is no `def`. `defvar` is a mutable global that is always re-initialised on redefinition; `defconst` is a
|
||
compile-time constant, folded into its use sites.
|
||
|
||
The problem in practice: `C-c C-k` re-evaluates every top-level form against a *running* program, so today it wipes
|
||
state you may have spent a session accumulating.
|
||
|
||
**Settled in conversation — three roles, three words:**
|
||
|
||
- **`def`** — always re-initialised. The tweakable: some number you are adjusting and do not care about preserving.
|
||
- **`defvar`** — re-initialised *only if it would come out different*. For heavier initialisers: an N×M grid that should
|
||
recompute only when the rows or columns actually changed.
|
||
- **`defconst`** — folded, and reserved for what genuinely cannot change for the life of the program. `rows`/`cols` in
|
||
`sand.flan` are honest `defconst`s: the program does not support resizing on the fly and changing them means resetting
|
||
everything anyway.
|
||
|
||
Note this **`defvar` is not Common Lisp's**, which is "assign only if unbound". This one is value-dependent, which suits
|
||
a live-edited game better and is the harder of the two to implement.
|
||
|
||
**The open question is what "different" means**, and the grid example is what makes it sharp. If the grid is sized by
|
||
`rows` and `cols`, the initialising expression's *text* is unchanged when `rows` changes — only its value is. So:
|
||
|
||
1. **Compare the expression** (textual or AST). Cheap, and does not catch the case above.
|
||
2. **Compare the expression plus everything it depends on.** Catches it. Needs dependency tracking.
|
||
3. **Evaluate and compare the result.** Correct, and defeats the entire purpose whenever the initialiser is expensive —
|
||
which is the only case this feature exists for.
|
||
|
||
Option 2 is what is wanted.
|
||
|
||
**The reason to do it: it is the same machinery `defconst` needs.** A folded constant cannot be tuned live, because its
|
||
value is baked into every function that used it — so the values you most want to tweak while the game runs are exactly
|
||
the ones you cannot. Fixing that means knowing which functions depend on which constants, and rebuilding those. That is
|
||
the same dependency tracking `defvar`'s "if different" requires. Built once, it buys live-tunable constants *and* the
|
||
conditional re-initialisation. Neither is worth building alone; together they are clearly worth it.
|
||
|
||
## 4. Structural typing, row polymorphism, anonymous structs
|
||
|
||
The goal as stated: make this feel Clojure-like while still being typed.
|
||
|
||
**The representation is already structural.** `types.ml` does structural equality on resolved types, and a Flan struct is
|
||
exactly its C layout with no header and no tag word. What is nominal is the *checking*, not the data — so this is a
|
||
front-end question, not a data-model change.
|
||
|
||
**What it buys, in rough order of value here:**
|
||
|
||
1. **Your destructuring already looks like this, and that is the strongest argument.** Clojure's `{:keys [x y]}` works in
|
||
binding position today. Structural typing makes the same notation work in *parameter* position — the pattern you write
|
||
to pull fields out becomes the signature saying what you accept. One notation, two places, no separate declaration.
|
||
That is precisely the "Clojure-like but typed" feel being asked for, and it needs no new syntax, only a new meaning
|
||
for syntax that exists.
|
||
2. **Functions over "anything with these fields".** Magnitude over anything with an `x` and a `y` — a position, a
|
||
velocity, an enemy, a bullet. In a game this recurs constantly because everything has a position.
|
||
3. **Ad-hoc returns.** Returning a hit flag and a point without declaring a type for the pair. The Clojure habit of
|
||
returning a map, typed and free.
|
||
4. **The FFI stops needing conversions.** raylib's `Vector2` and a locally-defined vector are byte-identical and are
|
||
today two nominal types. Structurally they are one.
|
||
|
||
**Two costs, and the second is the real work:**
|
||
|
||
- **Not an "explicit over convenient" question — that framing was wrong and is recorded so it is not repeated.** The
|
||
standing rules about implicit numeric conversion and about a bare integer not becoming an enum are about *lossy*
|
||
things happening silently. A structural match is not lossy; it is a different notion of type *identity*. The two were
|
||
lumped together in conversation and the author rejected the pairing, correctly. Strong typing and nominal typing are
|
||
separate choices and this only touches the second.
|
||
- **Field order is layout, and that is the hard part.** `{x f32, y f32}` and `{y f32, x f32}` have the same fields and
|
||
different memory. So either field order becomes significant — surprising, since nothing else about a structural type
|
||
should be — or matching a differently-ordered type requires a copy, which breaks the zero-cost property the whole data
|
||
model rests on. Accepted in conversation as a limitation to respect rather than fight.
|
||
|
||
**Writability is what decides whether the copy escape exists.** Fields would be writable on the same terms as any struct:
|
||
by value you get a copy and writes are local; through a `(Ptr T)` they reach the original. But that settles the layout
|
||
question rather than sitting beside it. *Read-only* structural access has an out — the compiler could gather the
|
||
requested fields into a temporary, and then field order would not matter at all. *Writable* structural access cannot:
|
||
it has to alias the real storage, so the layout must genuinely line up. Decide writability first; the layout answer
|
||
follows from it.
|
||
|
||
**`defclass` may solve this, for the cases that can afford it.** A class has an implementation-defined representation, so
|
||
the compiler owns the layout and access goes through metadata rather than a fixed offset — field order stops being a
|
||
user-visible fact. The limit is that a class carries identity and metadata, and a `Vector2` should not pay for either.
|
||
So plain structs still need their own answer and classes only cover the cases where the overhead was already accepted.
|
||
See item 7.
|
||
|
||
Also unresolved: the shim generates a C typedef per *named* struct, and an anonymous struct has no name to generate one
|
||
from. Either anonymous structs cannot cross the FFI, or the generator learns to name them.
|
||
|
||
## 5. A JS backend — for web apps, not for games
|
||
|
||
**The goal, stated in conversation: calling JS libraries, and eventually a hiccup-style DSL for writing web apps in
|
||
Flan.** That is a different product from the wasm target and the two do not compete — games go to wasm, web apps go to
|
||
JS.
|
||
|
||
This matters because the obvious objection does not apply. "A JS backend cannot call raylib, so the whole graphics layer
|
||
would need reimplementing on canvas" is fatal for a *game* backend and irrelevant here: web apps do not use raylib.
|
||
|
||
**The design fork, and it should be decided first:**
|
||
|
||
- **Linear memory** — an `ArrayBuffer` standing in for pointers and structs, with typed-array reads and writes. Fast
|
||
arithmetic, and every value is opaque bytes, so every call into a JS library marshals both ways. This is asm.js, which
|
||
wasm exists to replace, and it is precisely wrong for an interop-motivated backend.
|
||
- **Object mapping** — a Flan struct becomes a plain JS object, a `Vec` becomes a JS array. Slower for tight numeric
|
||
loops, natural for interop, readable output. **For this goal this is clearly the right one**, and performance is not
|
||
the constraint people assume: DOM work is dominated by the browser, not by arithmetic.
|
||
|
||
**The consequence to face early: the memory model does not come along.** Object mapping means garbage collection, which
|
||
means no pointers, no manual `free`, no arena, no allocator. So this is a **dialect**, not merely a second backend —
|
||
some Flan programs will not compile to JS, and that should be named up front rather than discovered. Worth deciding
|
||
which subset is the JS-targetable one, and whether the checker enforces it per target.
|
||
|
||
**The unexpected upside: the dev loop could work on the web.** Conditions, restarts and a break loop all need the
|
||
ability to run new code in a stopped program. wasm cannot have this — `--dev` is refused there because reload is
|
||
`dlopen` and the browser has no sockets. JS has no such problem: it evaluates new code trivially. A JS target might be
|
||
the *only* place the live dev loop and the browser coexist.
|
||
|
||
**Reader conditionals, decided in conversation.** A cljc-style subset is accepted, and the author chose Clojure's
|
||
inline reader conditionals over file-level target naming. Note what that changes: **this language has no conditional
|
||
compilation today**, and that absence drove a decision earlier the same day — `barf` signals on web rather than being
|
||
compiled out, precisely because "this bit is desktop-only" is not expressible. Note also that the sand-on-web work
|
||
introduced file-level target replacement for *C* sources (`foo.web.c` replaces `foo.c`), so the repo now has a
|
||
file-level precedent that this decision deliberately does not follow. Both were put; inline was chosen.
|
||
|
||
**Also wanted, and independent of all of the above:** `#_` to discard the next form, and repeated (`#_#_`) to discard
|
||
that many following forms, as in Clojure. Reader-only, small, and useful immediately rather than whenever a JS backend
|
||
happens.
|
||
|
||
**One dependency:** hiccup is a macro, and the macro expander is blocked on `Form` being a Flan union, which is union
|
||
values. The backend can start before that; the DSL cannot.
|
||
|
||
## 6. C interop as seamless as Zig's — built, with two decisions left
|
||
|
||
**The mechanism is in** (`lib/cimport.ml`, `lib/cjson.ml`, `vendor/raylib/headers`; BUILT.md, "The header
|
||
is read now"). Settled and not worth reopening: clang's JSON AST dump over a shelled-out `clang`, never
|
||
libclang — and Zig has since abandoned linking clang too, for Aro, which strengthens the argument rather
|
||
than weakening it. The import is bounded by the package's own `defstruct`s rather than by a curated list.
|
||
Refusals are demotions in Zig's sense: the name exists, cannot be had, and says why at the use site, which
|
||
`Load.refuse_hidden` already did for `main`. Names kebab by a rule that is injective over raylib's 581, and
|
||
reversibility is by storage — the C symbol is kept verbatim — so the rule never needs an inverse. Where two
|
||
names do collide, neither takes it.
|
||
|
||
**The evidence:** against raylib 5.5, all 16 `defstruct`s and all 172 hand-written `declare-c` agree
|
||
exactly. Against the 5.1-dev header also on this machine, ten real differences. Both comparisons stop the
|
||
build, and a permuted `Texture2D` or an `f64` for a `float` is caught by name.
|
||
|
||
What is left is two decisions, and both are the author's.
|
||
|
||
**6a and 6b are answered. See BUILT.md, "The bindings are committed now".** It became a code generator whose
|
||
output is committed, and the 172 hand-written lines were *not* migrated — for a reason 6b did not reach. The two
|
||
sections below are kept as the reasoning that got there.
|
||
|
||
### 6a. Does reading the header stay a build-time step, or become a code generator?
|
||
|
||
The cost, measured, with the wrappers pruned by `Reach` as they already were:
|
||
|
||
| | today | + 256 imported |
|
||
|---|---|---|
|
||
| release build, warm | 0.078s | 0.082s |
|
||
| **redefinition** | **31.0ms** | **46.5ms** |
|
||
| dev build, cold | 0.649s | 0.982s |
|
||
|
||
Release is nearly free and that question is answered. **The 15.5ms on redefinition is not nothing on the
|
||
branch where the dev loop is the priority** — it is a 50% increase on the number that lane exists to keep
|
||
small, and it buys a check of signatures that have not changed since the last build.
|
||
|
||
So the third option from the original discussion is now the live one: `flan import-c` already prints
|
||
`declare-c` lines, so **generate from the header, commit the result, regenerate when raylib moves** costs
|
||
nothing more to build. Explicit in the source, checked against reality, no header read at build time, and
|
||
the check becomes a thing you run rather than a thing you pay for. Against it: a committed file goes stale
|
||
silently, which is the failure the whole lane exists to prevent, and "regenerate when the library moves"
|
||
is a discipline rather than a mechanism.
|
||
|
||
A middle reading worth considering: keep the build-time check but run it only when *not* `--dev`, on the
|
||
grounds that a release build is where a wrong signature must not get through and a dev build is where
|
||
15.5ms is felt. That is the same shape as `Reach` not pruning dev builds, for a symmetric reason.
|
||
|
||
### 6b. Do the 172 hand-written lines get migrated?
|
||
|
||
The diff is clean, so nothing blocks it on correctness. What blocks it is that migration needs the header
|
||
present at *every* build, which means vendoring raylib.h into the repo or requiring `raylib-devel` — and
|
||
BUILT.md records "a build needs libraylib linkable and not raylib-devel installed" as a property that was
|
||
chosen on purpose. That is why `vendor/raylib/headers` is opt-in (`?${FLAN_RAYLIB_H}`) today and the
|
||
hand-written lines are untouched.
|
||
|
||
Worth noting what migration would actually lose, since it is small but real: the hand-written names are
|
||
better than the rule's. `IsKeyPressed` is `key-pressed?` by hand and `is-key-pressed` by rule;
|
||
`CheckCollisionRecs` is `collision-recs?`. And an enum parameter imports as `i32`, because the header says
|
||
`KeyboardKey` and nothing tells the importer the package calls that `Key` — so `(rl/key-down? :space)`
|
||
would become an integer at the call site. A migration is therefore not a deletion; it is a deletion plus a
|
||
kept list of the lines whose face is deliberately nicer than the header's.
|
||
|
||
**Answered: no, and the blocker is not the one above.** Vendoring stopped being the question once the *output* was
|
||
committed rather than the header — no header is needed at any build. (Vendoring then happened anyway, and for the
|
||
other reason: `vendor/raylib/raylib-5.5.h` is committed, `headers` names it with no `${...}` in front of it, and the
|
||
signature check runs on every build. What that settled was not "can we build without a header" — the committed
|
||
bindings had already settled that — but "does the check actually run", and behind an opt-in the answer in several
|
||
worktrees was no.) The count was also smaller than feared: 136 of
|
||
the 172 are exactly what the rule produces, and the other 36 are expressible as `name` overrides in `bindings`.
|
||
|
||
What actually decides it is that migration would gut the check. Everything the generator emits agrees with the header
|
||
by construction, so diffing generated output against its own source proves nothing; the hand-written lines have a
|
||
different author, so they are the only declarations a header can contradict — and all ten of the 5.1-dev differences
|
||
came from them. Delete them and the signature half of the check silently becomes a tautology. The enum point above
|
||
survives intact as a second reason: `(rl/key-down? :space)` keeps its `Key` parameter only because that line is
|
||
hand-written.
|
||
|
||
## 7. Watching variables
|
||
|
||
Raised while designing item 1, and deliberately separated from it.
|
||
|
||
Item 1 shows globals *per frame*, chosen by what the code in that frame references. That is the right default and it
|
||
cannot cover the case where the thing you care about is not named by the frame you are standing in — stopped deep in a
|
||
helper, still wanting to see the grid.
|
||
|
||
So: a way to say "always show me this", surviving a resume and the next break.
|
||
|
||
Questions it opens:
|
||
|
||
- **What can be watched.** A global is easy — it has a name and an address that does not move. A *local* is harder: it
|
||
belongs to a frame that is gone as soon as you resume, so "watch `y`" either means a different `y` every time or means
|
||
nothing. A watched expression — `(at velocity 40 12)` — is the most useful and the most expensive, since it has to be
|
||
compiled and run in the program each time, which is what `eval-expr` already does.
|
||
- **Where the list lives.** Per project, per session, or in the file. A watch list that vanishes when Emacs restarts is
|
||
one you stop using; one in the repo is one you accidentally commit.
|
||
- **When it updates.** Only on entering a break is cheap and probably enough. Live-updating while the program runs is a
|
||
different feature — closer to a HUD than a debugger — and worth not conflating.
|
||
- **Whether it belongs in the break buffer at all**, or in its own window that is useful while the program is *running*,
|
||
which is arguably where a game developer wants it.
|
||
|
||
## 8. Indentation is wrong inside a binding vector
|
||
|
||
Reported against the `let` in `sand.flan`'s `settle`: the second and later bindings indent one column too far.
|
||
|
||
**Cause, found by reading `flan-mode.el`.** `flan-indent-function` looks at the head of the *enclosing* form and, when
|
||
it is one of Flan's body forms (`defn`, `let`, `if`, `while`, `until`, `dotimes`, `match`, `do`, `loop`, `defer`),
|
||
indents one past the open paren. Inside a `let`'s binding *vector* the enclosing open is the `[`, and the symbol after
|
||
it is the first binding's **name** — never a body form — so the check fails and it falls through to Emacs's generic
|
||
`lisp-indent-function`, which treats the vector as a function call and aligns continuation lines under the first
|
||
*argument* rather than under the first *binding*.
|
||
|
||
**What it is built on:** Emacs's built-in `lisp-indent-function`, with that one override. Not `clojure-mode`, which is
|
||
where the missing piece lives — `clojure-mode` special-cases binding vectors and aligns them as pairs.
|
||
|
||
**The fix, agreed in conversation:** add a binding-vector case to `flan-indent-function`. When the enclosing open is `[`
|
||
and the form containing it is a binding form, align continuation lines to the column of the **first binding**, not to
|
||
the first argument. Bindings are pairs, so the alignment that reads correctly is name-under-name. Check the other
|
||
bracket users in the same pass — `defn` parameter lists and `restart-case` clause parameters have the same shape and
|
||
almost certainly the same bug.
|
||
|
||
**`clojure-mode` should be the reference, not the ancestor and not a dependency.** The distinction matters and was
|
||
muddled once in conversation: the proposal is *not* `define-derived-mode` on `clojure-mode`, and not adding an external
|
||
package. It is to rewrite `flan-mode`'s indentation and font-lock as our own code, **ported from `clojure-mode`'s source
|
||
rather than from `lisp-mode`'s**.
|
||
|
||
The current mode has the wrong ancestor. It is built on Emacs's `lisp-indent-function`, and Emacs Lisp has none of the
|
||
shapes Flan actually uses: no vectors as binding forms, no keywords as map keys, no destructuring, no bracket variety.
|
||
So every rule has to be added by hand and the binding-vector bug is simply the first one hit. Clojure's rules already
|
||
cover brackets-mean-binding, pairs-align, maps, keywords, `#_`, and reader conditionals — the last two now wanted in
|
||
their own right.
|
||
|
||
Where Flan diverges it diverges deliberately rather than by discovery: `defn` carries a return type between the
|
||
parameters and the body; field access is `(.x v)`; field labels are moving from `:x` to `.x`, so `{.x 1.0}` is a struct
|
||
literal and not a map with symbol keys. Those get written on purpose.
|
||
|
||
Deriving at runtime stays rejected — it would add an external dependency to a mode that ships inside this repo and
|
||
currently needs nothing beyond stock Emacs, and that community is mid-transition to a tree-sitter mode, so the base is
|
||
moving.
|
||
|
||
## 9. Instrumenting with `(pause)` from Emacs, without editing the buffer
|
||
|
||
Requested, not scheduled. The Clojure model: `C-u` before an eval marks the form so it stops when it runs. Wanted here
|
||
for **three targets** — the top-level form, the last expression, and **the form point is inside**, so that with the
|
||
cursor at `(+ 1| 1)` the program stops at that `(+ ...)`.
|
||
|
||
**What already exists, so this is mostly assembly:**
|
||
|
||
- **`(pause)` itself**, landed today in `lib/prelude.ml`. It is `error` under a `restart-case` with a `continue`
|
||
clause — a breakpoint is not a compiler feature here, it is what a condition system already gives you. Taking
|
||
`continue` resumes at the call.
|
||
- **`flan-eval-defun`** takes the top-level form's bounds and sends the text; **`flan-eval-last-sexp`** sends the sexp
|
||
before point. Both hand a *string* to the daemon, which is the hook — the buffer never has to be modified.
|
||
- **`flan-dev--enclosing-head`** (`flan-dev.el:1009`) already walks out to the enclosing form to find its head, for
|
||
eldoc. The third target's hard part is already written.
|
||
- `current-prefix-arg` is already read in one place (`flan-dev.el:1429`), so the `C-u` convention has precedent.
|
||
|
||
**So the work is a source-to-source rewrite in Emacs before sending:** find the target form's bounds, send the
|
||
top-level form with that span replaced by `(do (pause) <span>)`. Nothing on the compiler side changes.
|
||
|
||
**The one real difficulty: source locations.** Splicing text shifts every line and column after the insertion, so
|
||
error overlays, `layout`, the break loop's frame locations and DWARF would all point slightly wrong for an
|
||
instrumented form. Options, none free: pad the inserted text to preserve offsets (ugly, fragile across newlines); send
|
||
the instrumentation as a separate field the daemon splices *after* parsing, where locations are already attached; or
|
||
accept the drift and say so. **The second is almost certainly right** and makes this a small daemon change rather than
|
||
a pure-Emacs one — which also means it should not be built as a pure-Emacs hack first.
|
||
|
||
**Settled: it sticks, like Clojure's.** The mark stays until the form is evaluated again plainly. That is the
|
||
behaviour the workflow expects — you mark it, run the game, hit it repeatedly, and clear it with an ordinary `C-c C-c`
|
||
rather than having to re-mark before every run.
|
||
|
||
## 10. A second backend for dev builds — study SBCL and drop LLVM there?
|
||
|
||
Raised at the end of the session, not answered. The proposal: write our own code generator for **debug builds and the
|
||
interactive workflow**, keep LLVM for release. The motivation is that evaluating arbitrary code — full functions, real
|
||
test code, not just constant arithmetic — costs a compile, a file and a `dlopen` every time.
|
||
|
||
**What SBCL actually does**, since it is the model: its own assembler and code generator, emitting machine code
|
||
straight into the heap. No files, no external tools, no dynamic loading. Code objects are ordinary heap objects and are
|
||
collected when unreachable. That is also why its `disassemble` can annotate richly and why it can show machine state —
|
||
it owns everything.
|
||
|
||
**The argument against, and it is the same one that killed the interpreter.** Two backends must agree about arithmetic,
|
||
struct layout, overflow, evaluation order and calling convention. When they disagree you get *works interactively,
|
||
breaks when shipped* — the worst bug class in a live-programming system, and arguably worse than the interpreter
|
||
version because the divergence is in code generation rather than in semantics you can read.
|
||
|
||
Note what plan.org's dev-vs-release table already accepts: indirection cells, a shadow stack, version words, code
|
||
never freed. Those are **additive instrumentation on one code generator**, not a second one. This proposal is a
|
||
different kind of divergence.
|
||
|
||
And the cost is enormous: an instruction selector and assembler per architecture — x86-64, arm64, wasm32 — which is
|
||
where SBCL's decades went.
|
||
|
||
**The cheaper route to the same goal, and the thing to weigh it against: ORC, LLVM's in-process JIT.** Measured today,
|
||
a 35ms redefinition is **21ms of `llc` plus `ld`** — external tools generating and linking machine code. An in-process
|
||
JIT replaces exactly that, with **the same IR and therefore the same semantics**, no second implementation, and code
|
||
that can be unloaded. It gets what the proposal wants without the divergence hazard.
|
||
|
||
plan.org already says ORC "remains addable later behind the same typed IR without touching the language", and the only
|
||
column text-IR-plus-clang loses in its comparison table is the JIT one.
|
||
|
||
**So the honest position:** the thing that would justify revisiting the LLVM decision is not slower eval in the
|
||
abstract — it is wanting *arbitrary interactive evaluation of real code* as a first-class workflow. That is now stated
|
||
as a want. If it stays a want, ORC is the answer and a hand-written backend is not.
|
||
|
||
**Where the shared objects live**, since it was asked: `Build.workdir ()` — a per-build temporary directory — with the
|
||
object cache separately at `Build.cachedir ()`, stable across builds. Nothing sweeps the workdir, and nothing is ever
|
||
`dlclose`d, so a long session accumulates both files and mappings. Sweeping at session end is cheap and unrelated to
|
||
any of the above.
|
||
|
||
## NEXT SESSION STARTS HERE
|
||
|
||
**The agreed plan: investigate putting the compiler inside the running program.**
|
||
|
||
Today there are two programs. The *daemon* is the compiler. The *game* is a separate program it launches. They talk
|
||
over a socket, so everything the editor shows — locals, globals, the stack, evaluation results — has to be packed up
|
||
in one program, sent across, and unpacked in the other. A large share of what `BUILT.md` documents exists only to move
|
||
data between those two address spaces.
|
||
|
||
Putting the compiler inside the game removes all of that. There is nothing to send, because the compiler can read the
|
||
program's memory directly. It is what SBCL does, and it is why interactive work in SBCL is instant.
|
||
|
||
**Read, in this order:** item 13 (what is actually being decided, and why the order matters), then item 11 (the shape,
|
||
the two objections and how the author answered them), then item 12 (the specific questions a spike must answer).
|
||
|
||
**First task, and it is research not building:** the spike in item 12 — what it costs to link the OCaml compiler into
|
||
a dev build. OCaml 5.2's multicore runtime has already removed the objection that would have been fatal. If it embeds
|
||
acceptably, everything else is reachable.
|
||
|
||
**Do not start with the backend question.** Merge first, measure, then choose. Item 13 says why.
|
||
|
||
---
|
||
|
||
## 11. One process, not two — the biggest open question
|
||
|
||
Raised at the very end of the session and **not resolved**. It subsumes items 9 and 10, and possibly several decisions
|
||
recorded elsewhere as settled. Do not treat any of those as closed until this is answered.
|
||
|
||
**The claim:** the daemon and the running program should not be two processes. SBCL is one image; Clojure is one
|
||
image; in both, the interactive workflow is cheap *because nothing is transported*. The author's reading is that the
|
||
split here was forced by not having threads — and that is worth checking, because **threads are already in use**: the
|
||
agent runs a listener thread and a loader thread inside the program today.
|
||
|
||
**The concrete shape**, if it were done: load the *program* into the daemon's process as a shared library rather than
|
||
launching it as a child. The reload primitive already `dlopen`s modules; the program itself would be one.
|
||
|
||
**What would evaporate, and it is a lot.** No socket and no wire protocol. Cell updates become plain pointer stores.
|
||
Locals and globals become memory the compiler can read directly — no render thunks compiled per inspection, no 4K
|
||
result cap, no seqlock, no snapshot copying, no generation stamping. The macro expander stops needing the compiler's
|
||
own `dlopen`. `flan reload` stops being a fresh process paying startup every time. A large amount of machinery
|
||
recorded in `BUILT.md` exists **only** to move data between two address spaces.
|
||
|
||
**What is given up, in order of sharpness:**
|
||
|
||
1. **Crash isolation.** A segfault in the game currently cannot touch the compiler. `plan.org`'s jank note — corrected
|
||
earlier for citing the wrong mechanism — concludes "we are safe from the repro *because we compile out of process*".
|
||
In one image a bad pointer takes the session with it. Lisp users live with this and mostly accept it; it is a real
|
||
loss and should be chosen knowingly, not discovered.
|
||
2. **macOS requires a window on the main thread.** "Run raylib on a separate thread" is fine on Linux and not portable.
|
||
The inversion — compiler on the side thread, window on main — probably works, but it needs deciding rather than
|
||
assuming, and it changes who owns the main loop.
|
||
3. **The OCaml runtime and the game in one address space**: signals, `SIGSEGV` handling for the break loop, the OCaml
|
||
GC running while game code holds raw pointers into its own arenas. Each is probably fine; none is free.
|
||
|
||
**What it does NOT fix, and this is where an earlier answer in this session was wrong.** A hand-written debug backend
|
||
was discussed as the way to make evaluation instant. The correction: a new backend removes `llc` and `ld` — about
|
||
21ms of a 35ms redefinition — but **not the transport**, because the transport exists due to the process split, not
|
||
the backend. Conversely, **merging the processes removes the transport but not the code generation**. They are
|
||
independent, and merging is the larger prize of the two.
|
||
|
||
**Decisions to revisit if this is taken:** the watch design (push was chosen partly because polling costs a compile);
|
||
the 4K result cap and its seqlock; the snapshot machinery and its generation stamping; the render-thunk-per-inspection
|
||
design for locals and globals; the inspector's inability to retain a value; and whether ORC or a hand-written lowering
|
||
is needed at all.
|
||
|
||
**Both objections were answered by the author, and the second reframed the design — record the corrected shape.**
|
||
|
||
**Crash isolation: accepted knowingly.** Conditions already catch what the *language* signals — a failed bounds check,
|
||
an exhausted allocator, a missing file — and those keep stopping in the break loop in one process exactly as they do
|
||
now. What they cannot catch is a real memory fault: a bad pointer through the FFI, a use-after-free in an arena. That
|
||
is an OS signal, not a condition, and it takes the image with it. Same deal every Lisp makes.
|
||
|
||
**macOS: invert the relationship, and this is the right shape rather than a workaround.** Not "load the program into
|
||
the daemon" as sketched above — the opposite. **The agent is already a server running inside the program.** So the
|
||
compiler moves *into the program's process*, beside the listener that is already there. The game starts, takes the
|
||
main thread for its window, and the compiler and listener run on a side thread. That is SLIME's model: you start the
|
||
image, it serves, the editor connects.
|
||
|
||
This also fits a distinction `plan.org` already draws — dev and release builds are deliberately different. A dev build
|
||
links the compiler in; a release build ships neither compiler nor server. The same split SBCL has between a
|
||
development image and a delivered executable.
|
||
|
||
One thing that gets *better* rather than merely cheaper: a break loop in the same process as the compiler can offer
|
||
restarts that recompile and retry, with no transport in between.
|
||
|
||
**Suggested next step:** the macOS question is answered, so start instead with what it costs to link the OCaml compiler
|
||
into a dev build — the OCaml runtime and the game sharing an address space, signal handling for the break loop, and the
|
||
GC running while game code holds raw pointers into its own arenas. It is the only one of the three costs that could
|
||
make the whole idea non-portable, and it is answerable by reading raylib and GLFW rather than by building anything.
|
||
|
||
## 12. What it costs to put the OCaml compiler in the game's process — and whether to port the compiler
|
||
|
||
Follows from item 11. **Investigate before deciding anything**; this entry is the brief, not the answer.
|
||
|
||
**One fact established already: this project is on OCaml 5.2.** That is multicore OCaml — real threads, domains, no
|
||
global runtime lock. It removes the objection that would have been fatal: the runtime can sit on its own domain beside
|
||
the game's threads rather than serialising everything.
|
||
|
||
**What to find out:**
|
||
|
||
- **Linking OCaml into a C/native binary.** `ocamlopt -output-obj` and friends are supported; how well, with dune, with
|
||
the C stubs this project already has (`dynload_stubs.c`, the runtime, the shims)?
|
||
- **Signals.** The OCaml runtime installs handlers. The break loop wants `SIGSEGV` for the crash case, and the agent
|
||
already handles socket work. Who wins, and can they coexist?
|
||
- **The GC and raw pointers.** OCaml's collector moves its own heap; Flan's arenas and `Vec`s are raw memory OCaml
|
||
never sees. That should be fine — they do not point at each other — but "should be" is not an answer.
|
||
- **Startup cost and binary size** for a dev build that links the whole compiler.
|
||
- **Whether the main thread can be the game's** while the compiler runs on a domain (see item 11's macOS note).
|
||
|
||
**And the larger question the author raised: port the compiler to another language?**
|
||
|
||
The honest end state for a Lisp that wants one image is **self-hosting** — the compiler written in Flan. That is what
|
||
SBCL is, and it dissolves this entire question: no foreign runtime to embed, no GC to reconcile, no signal conflict,
|
||
and the compiler becomes the largest test of the language. It is also an enormous undertaking and needs macros,
|
||
unions, `Map` and a string library first — three of which landed today.
|
||
|
||
Intermediate options if OCaml proves genuinely hostile: C or C++ (no runtime to embed, painful to write a compiler
|
||
in), Rust (no GC, embeds cleanly, large rewrite), or Zig (same, and its own C-interop story is already being read for
|
||
item 6). **None of these should be entertained on aesthetics.** The bar is a specific, demonstrated obstacle to
|
||
embedding OCaml 5.2 — and the multicore runtime means the most likely obstacle has already gone away.
|
||
|
||
**Order of work:** answer the embedding questions above with a spike, not a rewrite. If OCaml embeds acceptably, item
|
||
11 is buildable and no port is needed. Self-hosting stays a long-term ambition rather than a prerequisite.
|
||
|
||
## 13. The two questions are independent, and the order matters
|
||
|
||
The end of the session, and the clearest statement of what is actually being decided. Read this before items 10, 11
|
||
and 12, which were each written mid-argument.
|
||
|
||
**Two separate decisions, repeatedly conflated in conversation:**
|
||
|
||
1. **One process or two** — whether the compiler runs inside the program's process. This is about *transport*: how
|
||
compiled code and inspection data move between the compiler and the running program.
|
||
2. **How machine code is produced and installed** — the current `llc` + `ld` + `dlopen`, libLLVM's in-process JIT
|
||
(ORC), or a hand-written lowering. This is about *code generation*.
|
||
|
||
Neither implies the other. A hand-written backend removes `llc` and `ld` (about 21ms of a 35ms redefinition) and
|
||
leaves the transport. Merging the processes removes the transport and leaves code generation exactly as it is.
|
||
**Merging is the larger prize.**
|
||
|
||
**And there is a third option for question 2 that keeps getting lost: change nothing.** In one process you can still
|
||
write a `.so` and `dlopen` it. You lose nothing that works today, and you still delete the socket, the wire protocol,
|
||
the 4K result cap, the seqlock, the snapshot copying, the render-thunk-per-inspection, and `flan reload` paying process
|
||
startup every time.
|
||
|
||
**So the order is:**
|
||
|
||
1. **Spike the embedding** (item 12). OCaml 5.2's multicore runtime removes the objection that would have been fatal.
|
||
If it embeds acceptably, everything below is reachable.
|
||
2. **Merge the processes**, keeping the existing code path. Biggest win, no new dependency, deletes the most
|
||
machinery.
|
||
3. **Measure what is left.** With transport gone, a redefinition is whatever code generation costs. That may simply be
|
||
fine.
|
||
4. **Then decide the backend, on evidence.** libLLVM's JIT means linking the library `plan.org` rejected — for
|
||
reasons that were right at the time, when the JIT was worth ~13ms and nothing else. In one process it is worth the
|
||
file, the load, the unbounded accumulation *and* cheap evaluation of arbitrary real code, which the author has since
|
||
named as a first-class want. The cost is also more bounded than "breaks routinely" suggests: the bindings track
|
||
LLVM major versions, so it is a pin you own and upgrade deliberately.
|
||
|
||
A hand-written lowering buys the same speed plus no dependency and a disassembler you own, at the cost of an
|
||
instruction selector per architecture, maintained forever. The drift risk is real but testable — the `@sanitize`
|
||
alias already builds 28 programs twice and compares output and exit status, which is exactly the harness two
|
||
backends would need.
|
||
|
||
**The principle to keep:** the original decision was made on a measurement and it held up for months. Make this one the
|
||
same way — merge first, measure, then choose. Do not pre-empt step 4 at step 1.
|
||
|
||
**Also settled in passing:** a release build can include the compiler and server and simply not be connected to, the
|
||
way an SBCL executable can. The dev/release split is a build flag, not an architectural constraint.
|
||
|
||
**And one correction to keep:** OCaml's own wasm support is irrelevant to any of this. OCaml is the compiler's
|
||
implementation language; Flan programs reach wasm through LLVM. The two only meet if the *compiler* should run in a
|
||
browser, which is not a goal.
|
||
|
||
## 14. The embedding spike, answered: OCaml 5.2 goes into a Flan dev build, and nothing objects
|
||
|
||
Item 12's brief, run. **Feasible.** No obstacle was found that argues for porting the compiler, and the one expected to
|
||
be sharpest — signals — turned out not to exist on the platform measured.
|
||
|
||
The apparatus is `spike/embed/`: four shell scripts and sixteen small sources, deliberately not a dune target, driving
|
||
`ocamlfind` and `clang` by hand against the `flan.cmxa` dune already builds. `bash spike/embed/run.sh` reproduces
|
||
everything below; `sig.sh`, `symbols.sh` and `merged.sh` each answer a question on their own. Nothing under `spike/` is
|
||
wired into the build, and `dune test --root .` is green either side of it.
|
||
|
||
### The headline: one binary, and it compiles itself
|
||
|
||
`spike/embed/merged.sh` builds a single executable out of, in one `clang` link:
|
||
|
||
- the emitted Flan program (`test/programs/edn.flan`, `--dev`, `@main` renamed to `flan_program_main`),
|
||
- `runtime/flan_rt.c`, `runtime/flan_dev.c`, `vendor/agent/flan_agent.c`,
|
||
- and the entire OCaml compiler as one `-output-complete-obj` object.
|
||
|
||
It runs. A C `main()` holds the main thread and runs the Flan program there; `caml_startup` happens on a `pthread`
|
||
beside it, next to where `flan_agent_start` already puts its listener. The compiler inside the binary then compiles
|
||
`edn.flan` — the source the program itself was built from — and emits 336,579 bytes of LLVM IR. That single result
|
||
answers questions 1 and 2 together; the ladder of smaller probes underneath it is support, not evidence in its own
|
||
right.
|
||
|
||
Nothing is wired up. The two halves share an address space and do not speak to each other. That is the point: the
|
||
question was whether they *can*, not what they would say.
|
||
|
||
### Question 1 — does OCaml link into a native binary here?
|
||
|
||
Yes, and with less friction than expected.
|
||
|
||
- `ocamlopt -output-complete-obj` is the one to use, not `-output-obj`: it bundles the runtime, so there is no hunt for
|
||
`libasmrun`. The final link needs `-lm -lpthread -ldl` and, on 5.2, **`-lzstd`** — 5.x's marshaller is compressed,
|
||
and the missing `ZSTD_*` symbols are the first thing a naive link fails on. That is the whole of the surprise.
|
||
- **dune is not in the way, because it does not have to be involved.** dune builds `lib/flan.cmxa` as it does today;
|
||
the `-output-complete-obj` step consumes that artifact afterwards. No dune rule had to change, and `lib/dune` and
|
||
`bin/dune` are untouched. A real merge would want a dune rule to drive that step, but the spike shows the artifact
|
||
boundary is clean, which is the part that could have failed.
|
||
- **The existing C stubs come through.** `lib/dynload_stubs.c` was taken verbatim from the unmerged `9e0ae3a` dlopen
|
||
branch and compiled into the same object; from inside the embedded runtime, `flan_mem_alloc`/`poke`/`peek`
|
||
round-trip correctly and `dlopen`+`dlsym` work. `-output-complete-obj` carries a `foreign_stubs`-shaped C file
|
||
through without special handling.
|
||
- **No symbol collides** (`symbols.sh`). Flan's own C — `flan_rt.c`, `flan_dev.c`, `flan_agent.c`, `dynload_stubs.c` —
|
||
defines 211 symbols; `libasmrun.a` defines 1,379; the intersection is empty, and the four Flan files do not collide
|
||
with each other either. Worth checking rather than assuming: those four are compiled into *two different processes*
|
||
today, and merging puts them in one link for the first time.
|
||
|
||
### Question 2 — threads, and who owns the main loop
|
||
|
||
**OCaml 5.2 is confirmed multicore**, and `Domain.recommended_domain_count ()` reports 16 here. A spawned domain does
|
||
real work in parallel with the main thread. The objection that would have been fatal is gone.
|
||
|
||
The macOS shape works, and it was tested as the thing that matters rather than as "can OCaml use threads":
|
||
|
||
- `caml_startup` can be called from a **C-created, non-main pthread**, while `main()` goes on to a loop it does not
|
||
leave. `harness4.c` runs a 298-frame mock game loop on the main thread that never once enters OCaml.
|
||
- **A second C thread — one the runtime never created, which is exactly the agent's listener — can call into OCaml**
|
||
after `caml_c_thread_register()`, bracketed by `caml_acquire_runtime_system`/`caml_release_runtime_system`. It
|
||
compiled the same program successfully from that thread. This is the specific capability the merged design needs
|
||
from `flan_agent.c`, and it exists.
|
||
|
||
The spike could not test macOS. Nothing here is macOS-specific — the inversion is a portable pthread arrangement — but
|
||
it is a Linux measurement.
|
||
|
||
### Question 3 — signals
|
||
|
||
**On Linux/amd64, OCaml 5.2 installs no signal handlers at all.** Not SIGSEGV, not SIGINT, not SIGFPE, not SIGPIPE,
|
||
nothing. The expected conflict does not exist.
|
||
|
||
`sig.sh` sweeps fifteen signals from inside the runtime at four moments — before `caml_startup`, at module init, from
|
||
inside a spawned domain, and after `Domain.join` — and every one is `SIG_DFL`. The reading has to be taken from inside
|
||
OCaml rather than from C after `caml_startup` returns, because OCaml 5 starts domains later; the first pass got this
|
||
wrong and read `SIG_DFL` for the wrong reason. A plain `ocamlopt` executable was built as a control and behaves
|
||
identically, so embedding changes nothing about signals.
|
||
|
||
The reason is structural, not incidental: OCaml 5 detects stack overflow with an explicit stack-limit check and calls
|
||
`caml_raise_stack_overflow` directly, rather than with a guard page and a SIGSEGV handler. `nm` on `libasmrun.a` shows
|
||
`sigaltstack` referenced but no SIGSEGV handler defined, which matches.
|
||
|
||
So **the break loop can take `SIGSEGV` outright, and it does not have to install first or last** — order does not
|
||
matter when there is nothing to displace. Measured directly: with the break loop's handler installed, deep OCaml
|
||
recursion still raises `Stack_overflow` normally, and a genuine fault at `0x10` reaches the break loop's handler with
|
||
`si_addr` correct. Chaining was implemented and compared; it is unnecessary here, but `harness5b.c` keeps it, because
|
||
it is what the merged build should do on any platform where the sweep comes back non-empty.
|
||
|
||
**The caveat, and it is the one thing in this report to re-check rather than trust:** this is `x86_64-pc-linux-gnu`
|
||
only. macOS/arm64 OCaml 5 was not testable here, and item 11 raises signals in the same breath as macOS. Re-run
|
||
`sig.sh` there before relying on it. `flan_agent.c` needs nothing from this either way — it sends with `MSG_NOSIGNAL`
|
||
throughout rather than depending on a SIGPIPE disposition.
|
||
|
||
### Question 4 — the GC and raw memory
|
||
|
||
Confirmed, and the confirmation is narrow on purpose. An 8 MiB arena was filled with a checkable pattern and 64 raw
|
||
interior pointers taken into it; OCaml then allocated 26.4 million words, took 7 major collections and a full
|
||
`Gc.compact ()`. Afterwards: the arena base is unmoved, **0 of 8,388,608 bytes altered**, 0 of 64 interior pointers
|
||
invalidated.
|
||
|
||
What that proves is that the collector traces its own roots and foreign memory is invisible to it. Arenas, `Vec`s and
|
||
`Map`s are safe because OCaml never learns they exist.
|
||
|
||
**What would break the assumption**, stated so it is not rediscovered the hard way:
|
||
|
||
1. Storing an OCaml `value` in Flan memory — an arena, a `Vec`, a global — across any allocation. The collector will
|
||
move the block and will not update that word, because it is not a root. `caml_register_global_root` (or the
|
||
generational one) is the only way that is legal, and it is a rule the merged design has to hold: **the boundary
|
||
passes pointers and scalars, never `value`s into Flan storage** — the same rule `dynload_stubs.c` already states
|
||
for its own reason.
|
||
2. A `value` held in a C local across a call that allocates, without `CAMLparam`/`CAMLlocal`. Ordinary OCaml-FFI
|
||
discipline; it applies to every stub the merge adds.
|
||
3. Long C work on the compiler thread without releasing the runtime system — which corrupts nothing but stalls
|
||
whichever domains want a stop-the-world. `llc` and `ld` are `exec`s and would want `caml_release_runtime_system`
|
||
around them.
|
||
|
||
### Question 5 — what it costs
|
||
|
||
| | bytes |
|
||
|---|---|
|
||
| `edn.flan`, release build | 71,824 |
|
||
| `edn.flan`, dev build, as built today | 114,296 |
|
||
| the same dev build with the whole compiler linked in | 4,257,624 |
|
||
| **what the compiler adds** | **~4.14 MB** |
|
||
|
||
**Startup: `caml_startup` takes 0.58–0.72 ms** across six runs — the runtime coming up and every module initialiser in
|
||
the compiler running. Measured with `clock_gettime` around the call itself, not `time(1)` on the process, because exec
|
||
and dynamic linking are paid today anyway.
|
||
|
||
The size reads as 37x, and that framing is misleading. **A dev session today runs two binaries, and the daemon alone is
|
||
4,815,368 bytes.** The merged dev build is *smaller than today's compiler process by itself*, and there is one of it
|
||
instead of two. Sub-millisecond startup and ~4 MB is not a cost worth designing around.
|
||
|
||
One more number, recorded because item 13's step 3 will want it and for no other reason: **a full in-process compile of
|
||
`edn.flan` — read, parse, load, typecheck, emit — is 12.1–12.5 ms**, warm and cold alike, with `llc` and `ld` excluded
|
||
because they are separate processes. That is where the remaining cost sits once transport is gone. It argues for
|
||
nothing; item 13 says the backend is decided at step 4 on a measurement taken at step 3, and this is not that
|
||
measurement.
|
||
|
||
### What this does not answer
|
||
|
||
- **The agent's handlers are a port, not a recompile.** `harness4.c` proves the *pattern* — a C-created listener
|
||
thread can register with the runtime and call OCaml. It does not port `flan_agent.c`'s handlers, which today answer
|
||
requests out of the program's own memory and would instead be calling into the compiler.
|
||
- **Crash isolation is gone by construction.** Not a finding; item 11 already accepts it knowingly. A bad pointer
|
||
through the FFI takes the session, and conditions still catch everything the *language* signals. Noted only so this
|
||
entry stands alone.
|
||
- **macOS, for signals and for the main-thread inversion.** See above.
|
||
- **The backend.** Untouched deliberately.
|
||
|
||
### The order the real work goes in
|
||
|
||
1. **Make the `-output-complete-obj` step a dune rule**, producing the compiler-as-object that a dev build links. This
|
||
is the only build-system work, and `lib/dune`/`bin/dune` did not need changing to prove it.
|
||
2. **Land the `dynload_stubs.c` branch** (`9e0ae3a`, currently reverted). The merged build needs the same
|
||
pointer-and-scalar boundary, and it is already written.
|
||
3. **Invert the startup**: the Flan program keeps `main()`, and `flan_agent_start` also brings up the OCaml runtime on
|
||
its side thread. `merged_main.c` is the sketch.
|
||
4. **Port the agent's handlers** from answering out of the program's memory to calling the compiler directly — this is
|
||
where the socket, the wire protocol, the 4K result cap, the seqlock, the snapshot copying, the generation stamping
|
||
and the render-thunk-per-inspection all get deleted. It is the bulk of the work and the whole of the prize.
|
||
5. **Keep `llc` + `ld` + `dlopen` exactly as they are.** Item 13's third option. Nothing here argues against it.
|
||
6. **Measure what is left.** Then, and only then, item 13 step 4.
|
||
|
||
## 15. The backend spike, answered: feasible, and the obstacle is not the one anyone expected
|
||
|
||
Item 13's step 4, run early and deliberately out of order, as a spike rather than as a decision. **Feasible.** A
|
||
function written in Flan goes through the ordinary frontend, is lowered to x86-64 by hand, is written into an `mmap`
|
||
and is called, and it answers correctly. That took an afternoon and no reference material beyond `objdump`.
|
||
|
||
The apparatus is `spike/backend/`: `x86.ml` (an instruction selector), `jit_stubs.c` (three calls OCaml cannot make
|
||
for itself, plus the C side of the ABI probes), `probe.flan`, `driver.ml` and `run.sh`. `bash spike/backend/run.sh`
|
||
reproduces everything below. There is no dune file under `spike/`, nothing is wired into the build, and
|
||
`dune test --root . -j 1` is green either side of it.
|
||
|
||
**The headline is not the arithmetic.** Ten checks pass, and one of them failed first and mattered: a C callee that
|
||
does a 16-byte aligned spill and reports whether it was entered aligned. Called plainly it passed; called from inside
|
||
a binary operator it returned `-1`. The evaluator spills its left operand across the right operand's evaluation, so a
|
||
call written in the right operand runs with `rsp` 8 bytes out. **That is the raylib crash, reproduced on day one of a
|
||
backend that does almost nothing.** It is fixed with a depth counter and an assertion, and it is the clearest single
|
||
argument that this work is *tractable but unforgiving*: nothing about the wrong version looked wrong, and only a probe
|
||
built to catch it caught it.
|
||
|
||
### Question 1 — does one function work end to end
|
||
|
||
Yes. The frontend is the real one — `Reader`, `Parse`, `Load`, `Check` — so what is lowered is the same `Tast.fn`
|
||
`emit.ml` gets, not a literal typed to make the exercise come out. Eleven of the 83 functions in a trivial program
|
||
(the prelude is most of them) lower with no special handling, including `space?`, `digit?` and `upper-ascii`, which
|
||
nobody wrote for this.
|
||
|
||
The emitter is the trivial one the brief allows: every slot is a stack slot at `rbp - 8(i+1)`, every value is computed
|
||
into `rax`, a binary operator spills its left operand. Two registers, no allocator, no liveness. `spike-add` is 58
|
||
bytes where clang `-O0` would spend about 20, and that is the correct trade for a debug build.
|
||
|
||
Proved by comparing numbers, not by reading bytes. `SPIKE_DISASM=1` disassembles the buffers that ran, and that is a
|
||
debugging aid kept out of the pass/fail path on purpose: a disassembly that reads correctly beside a function
|
||
answering 656 when it should answer 650 is the normal outcome of hand-encoding.
|
||
|
||
### Question 2 — the real shape of the work
|
||
|
||
**Layout is already owned, and this is the best news in the report.** `emit.ml`'s `lay` / `lay_fields` /
|
||
`payload_lay` compute C struct layout — offsets, padding, tail padding, the union payload blob — because DWARF needs
|
||
member offsets as integer literals and `getelementptr` cannot supply one. They are acceptance-tested against LLVM's
|
||
own answer for the same struct type. So the drift risk item 10 fears most, *two backends disagreeing silently about
|
||
where a field is*, does not arise: there is one layout calculator and a new backend calls it.
|
||
|
||
Against `tast.ml`'s `expr_kind`, in four buckets:
|
||
|
||
| | nodes |
|
||
|---|---|
|
||
| **Done in the spike** | `Int` `Bool` `Local` `Do` `Let` `If` `Return` `Set`/`Plocal`, arithmetic, comparison, bitwise, `Call`, `Rt` |
|
||
| **Mechanical** | `While` `Break` `Continue` (the jump patching exists), `Global` `Str` `Zero` `Uninit`, the rest of `place`, `Field` `Deref` `Addr`, `Arr`, `Some_` `None_` `UnwrapSome`, `Match` on a tag |
|
||
| **Bulky, not hard** | floats — a second register file, SSE encodings, `Cast`'s eight conversions, and the SSE half of the calling convention. Perhaps a third of the total instruction work for a small fraction of the programs |
|
||
| **Fiddly** | aggregate copy on assignment (a struct `store` *is* the copy `spec-memory.md` requires), `Make` `MakeCase` `CaseField` over the payload blob, `CallPtr`, `FnAddr`'s three cases and the cell load behind `Fnval` |
|
||
| **No plan** | `Handled` `Signal` `RestartCase` `InvokeRestart` `WithAlloc`, the transfer-channel guard after every call, the landing pad, and `fdefers` on the transfer exit path |
|
||
|
||
The last row is the one to take seriously. The spike never emitted a guard or a pad, and the guard is on *every call
|
||
site* in the real thing — `emit.ml`'s `guard`, `current_pad`, `emit_restart_case` and `emit_with_alloc` are several
|
||
hundred lines of control flow that a second backend reimplements from the spec rather than copies. Conditions are not
|
||
an advanced feature to defer: `spec-conditions.md` is load-bearing in the prelude already.
|
||
|
||
### Question 3 — the SysV boundary, and the obstacle nobody named
|
||
|
||
**The C boundary is the easy half, and `BUILT.md` is why.** `check.ml` rejects an aggregate in a `declare` signature
|
||
and the generated shim flattens every struct, so no Flan-emitted call ever passes one to C. A string or slice crosses
|
||
as `ptr`+`len` — two arguments, which is the only counting subtlety. The spike calls an eight-argument C function
|
||
correctly, including the two that go on the stack, and sets `al` for the variadic case. **No aggregate classifier is
|
||
needed for raylib. That is a large piece of `plan.org`'s "three classifiers to write and keep correct forever" that
|
||
simply does not apply.**
|
||
|
||
**The hard half is Flan calling Flan, and it was found by reading `signature`.** That function spells each parameter
|
||
with `ll ty` and flattens nothing. Emitting a trivial program and looking at the `define` lines:
|
||
|
||
```
|
||
define i64 @"flan.take-slice"(%slice %p0, i64 %p1, ptr %xfer)
|
||
define { i8, i32 } @"flan.index-of-i32"(%slice %p0, i32 %p1, ptr %xfer)
|
||
define { i8, float } @"flan.min-f32"(%slice %p0, ptr %xfer)
|
||
define { i8, %slice } @"flan.split-next!"(ptr %p0, ptr %xfer)
|
||
define %vec @"flan.filter-i32"(%slice %p0, ptr %p1, ptr %xfer)
|
||
```
|
||
|
||
The prelude is wall to wall aggregates by value. And what LLVM does with them, measured by `objdump` on clang's own
|
||
output rather than read off a table:
|
||
|
||
- `%slice` argument → `rdi`:`rsi`, two registers, and the next argument shifts along.
|
||
- `{ i8, i64 }` return → tag in `al`, value in `rdx`.
|
||
- `{ i8, float }` return → tag in `al`, value in **`xmm0`**.
|
||
- `%vec` return (six words) → a hidden `sret` pointer in `rdi`, the real arguments shifted along behind it, and the
|
||
pointer returned in `rax`. **That pointer does not appear in the `define` line at all.**
|
||
|
||
The last two are not the C ABI, and that was checked against a control rather than asserted — the same three shapes
|
||
written in C, compiled by the same clang at `-O2`, beside the same shapes written as first-class IR aggregates:
|
||
|
||
| shape | from C | from IR |
|
||
|---|---|---|
|
||
| `{ i8, i64 }` | `al` + `rdx` | `al` + `rdx` — agree |
|
||
| `{ i8, float }` | **packed into `rax`** (`movd`/`shl`/`or`) | `al` + `xmm0` |
|
||
| `{ i64, i64, i64 }` | **`sret` pointer in `rdi`** | **`rax` + `rdx` + `rcx`** |
|
||
|
||
The 24-byte case is the striking one: C spills to memory through a hidden pointer, and the IR form returns it in three
|
||
registers, one of which — `rcx` — the SysV ABI never uses for a return value at all. Somewhere past that, LLVM does
|
||
switch to `sret`, which is what `%vec` gets.
|
||
|
||
So **the internal calling convention is not the C ABI and is not specified anywhere. It is whatever LLVM's backend
|
||
does with a first-class struct, discoverable only by disassembling.** That is the sharpest obstacle in this report,
|
||
and it is sharper than raylib for three reasons: the reference is an implementation rather than a document; the
|
||
failure mode is a garbage field rather than a link error; and it is not stable by contract across LLVM versions, which
|
||
is exactly the coupling `plan.org` chose text IR to avoid.
|
||
|
||
It also forces the decision that determines everything else:
|
||
|
||
1. **Redefinitions only** — the custom backend emits a new body into an LLVM-built host. Incremental, testable one
|
||
function at a time, and the path that fits the dev loop. It requires matching LLVM's aggregate convention
|
||
bit-exactly, including the mixed integer/SSE case above.
|
||
2. **The whole dev build** — the custom backend owns both sides and *picks* the convention: every aggregate by
|
||
pointer, nothing classified, done. No matching problem at all. But it needs complete node coverage on day one,
|
||
conditions included, and there is no partial version that runs.
|
||
|
||
The spike leaned on option 1 without noticing, because the probe called C and C is the flattened half. A real attempt
|
||
has to choose deliberately.
|
||
|
||
### Question 4 — where the language leans on LLVM instead of defining itself
|
||
|
||
The audit, which stands on its own whatever happens to the backend. Each row is drift you do not get if the language
|
||
answers it.
|
||
|
||
| | today | defined? |
|
||
|---|---|---|
|
||
| Integer overflow | no `nsw`/`nuw`, "arithmetic wraps (plan.org, Types)" | **yes** |
|
||
| Shift count | masked to the operand width; a literal out of range is rejected by `check` | **yes** |
|
||
| Evaluation order | `map_lr`, and the comment says left-to-right is *required, not a preference* | **yes** |
|
||
| Division by zero | nothing. `prelude.ml` calls a remainder by zero "immediate undefined behaviour" and routes around it | **no** |
|
||
| `INT64_MIN / -1` | nothing, and it is a separate case. LLVM says undefined; x86 `idiv` raises `SIGFPE` | **no** |
|
||
| `f64` → `i64` out of range | `fptosi`, undefined in LLVM; x86 `cvttsd2si` answers the "integer indefinite" value | **no** |
|
||
| `Uninit` | emitted as `poison` | **no**, and see below |
|
||
| `unreachable` | after a `noreturn` call, and after an exhaustive `match` | **no** |
|
||
| Alignment | no explicit `align` on loads and stores; LLVM uses the type's ABI alignment | implicitly, via `lay` |
|
||
|
||
Two are worth more than a table row.
|
||
|
||
**`Uninit` → `poison` is the one that actually bites, and it bites in the direction item 10 fears.** A hand backend
|
||
gives a stable garbage value: whatever the stack slot held. LLVM's optimiser may reason from poison and delete the
|
||
code that reads it. So `(uninit)` is the one construct where the two backends are *supposed* to differ and where
|
||
"works in dev, breaks when shipped" is the expected outcome rather than a bug. The language should say what reading an
|
||
uninitialised value means before a second backend exists, not after.
|
||
|
||
**`unreachable` is the cheap one.** The spike emits `ud2`: a defined `SIGILL` at the instruction that fell through.
|
||
LLVM's `unreachable` is undefined behaviour and licenses the optimiser to delete the path. Defining it as a trap costs
|
||
two bytes and turns a class of miscompile into a crash with an address.
|
||
|
||
None of this needs a backend. It is a session with `plan.org` and six `check.ml` cases.
|
||
|
||
### Question 5 — unloading code, which is the prize
|
||
|
||
**The shadow stack is better than expected and still not sufficient, and the two halves of that are separate
|
||
questions.**
|
||
|
||
*The running half — and the shadow stack does answer it.* The frame push in `emit_fn` is inside a plain `if m.dev`
|
||
and is **not** gated: only the *slot table* is gated on `named && n > 0`, and a function with no named slot still
|
||
pushes a frame with a null `slots`. So every active Flan function in a dev build is on the chain, lifted handler
|
||
clauses included, and each frame points at the `flan_fninfo` belonging to the module it was compiled into — so the
|
||
pointer identifies not just the function but *which body*. "No frame on the chain names this body" is answerable
|
||
today, with no DWARF and no unwinder.
|
||
|
||
Two caveats on that half. The pop happens before `leave; ret`, so a body is briefly executing with no record —
|
||
irrelevant if reclamation happens at a safe point on the same thread, fatal if another thread reclaims while the game
|
||
thread is returning. And the chain is a plain global, not thread-local, which `flan_dev.c` states and justifies.
|
||
|
||
*The pointed-into half, which the shadow stack cannot see and which is the actual reason nothing is `dlclose`d today.*
|
||
`BUILT.md` is explicit, and it is not the reason the brief assumed: "a cell holds an address inside a module's text;
|
||
unloading it leaves every call site pointing at unmapped memory. The rule is about being *pointed into*." Owning the
|
||
code object answers most of this — you own the cells, so redefinition drops the old body's last cell reference — but
|
||
not all of it:
|
||
|
||
- `FnAddr (Fnval n)` **loads the cell and yields a raw body address**, which can then be stored in a struct, a `Vec`
|
||
or a global. Nothing records that it happened.
|
||
- `FnAddr (Flanfn _)` and `(Rtfn _)` bypass the cell *by design* — `tast.ml` says they "must never take that path" —
|
||
so a `Map`'s hash and equality pair and a handler-bind clause's address are raw pointers into a specific body, held
|
||
in data.
|
||
|
||
So: **frames are tracked, escaped code pointers are not.** Unloading needs a rule the language does not have yet. The
|
||
cheapest honest one is deferred reclamation — retire a body when no frame names it *and* an epoch has passed with no
|
||
new capture — and the cleanest is to make a function value a cell pointer rather than a body pointer, which costs one
|
||
indirection on `CallPtr` in dev builds and makes the whole question go away. That second option is worth writing down
|
||
now whatever happens to the backend, because it is a change to what a `Fn` value *is*.
|
||
|
||
### The verdict
|
||
|
||
**Feasible, unforgiving, and not the next thing to do.**
|
||
|
||
Feasible: the instruction selection is easy, layout is already owned and tested, the C boundary is already flattened,
|
||
and one function ran on day one. Nothing here argues the way item 10 feared — the divergence hazard is real but it is
|
||
concentrated in three named places (`Uninit`, division, the float cast), not spread through the whole of arithmetic.
|
||
|
||
Unforgiving: the internal aggregate convention is defined by LLVM's implementation and not by any document, the
|
||
alignment rule is invisible until raylib crashes somewhere else, and conditions are a second full implementation of
|
||
`spec-conditions.md` rather than a port.
|
||
|
||
Not next: item 13's order still holds, and the spike does not disturb it. Transport is 41µs of a 21ms redefinition and
|
||
code generation is 19 of the 21, so the *speed* case remains what item 13 said it was. What this spike adds is that
|
||
the **introspection** case is also not free — unloading needs a rule about escaped function values that nothing in the
|
||
language has, and that rule is worth having whether or not a backend is ever written.
|
||
|
||
**What to do first if it went ahead**, in order, and the first two are worth doing regardless:
|
||
|
||
1. **Define the six undefined cases** (question 4). No backend required, and every one is drift avoided rather than
|
||
drift managed.
|
||
2. **Decide what a `Fn` value is** — body pointer or cell pointer — and write it down. This is the unloading question
|
||
and it is a language question, not a backend one.
|
||
3. **Choose option 1 or option 2 from question 3**, deliberately. Everything else follows from it.
|
||
4. **Only then**, and only if 3 says so, grow `spike/backend/x86.ml` from the node table in question 2 — floats
|
||
first, because they gate most of the prelude, and conditions last, because they are the only row with no plan.
|
||
|
||
## 16. The dev backend, wired: a whole program compiles through `x86.ml` and runs, and conditions were never in the way
|
||
|
||
Item 15's step 4, taken. `lib/x86.ml` was 502 lines of encoder and frame model with nothing calling it. It now lowers a
|
||
whole `Tast.program` to an assembly file, and `flan build --x86` hands that file to the same clang invocation the LLVM
|
||
path uses, against the same runtime objects, the same generated shim and the same linker arguments. The flag is off by
|
||
default and refused in combination with `--dev`, `--debug`, `--sanitize` and every wasm target. **LLVM stays the release
|
||
backend and the default one; nothing on the existing path changed.** `dune test --root .` is green either side.
|
||
|
||
**41 programs out of `test/programs` build through it, and 40 of them print exactly what the LLVM build prints.** The
|
||
41st is `bounds.flan`, and it diverges on purpose — see question 4.
|
||
|
||
### Question 1 — which program, and what did it actually cost
|
||
|
||
Measured with `spike/backend/hist.ml` before anything was written, over candidates, not guessed. The one picked is
|
||
`spike/x86/p3-fizz.flan` — a `dotimes`, a call, an `if`, a remainder, two string literals and `print`/`println`:
|
||
|
||
```
|
||
p3-fizz.flan: 2 reachable fns
|
||
Call 1 Do 4 If 1 Int 15 Let 2 Local 7 Prim 14
|
||
Set 1 Str 3 While 1 place/Plocal 1
|
||
prim/Add 2 prim/Bytes 3 prim/Cast 1 prim/Eq 1
|
||
prim/I64ToBytes 1 prim/Lt 1 prim/Rem 1 prim/WriteStdout 4
|
||
```
|
||
|
||
Nineteen rows, two reachable functions after `Reach` prunes, and **no `Signal`, no `Handled`, no `RestartCase`, no
|
||
`Make`, no `Field`, no allocator**. The same is true of a program that only loops and prints: zero of each.
|
||
|
||
That is worth stating flatly because the expectation going in was the opposite — that the smallest useful program
|
||
carries one of each condition node, and that printing a single value drags in `Str`, `Make`, `Field` and `Call` because
|
||
the prelude builds a slice to do it. **It does not.** Printing an integer is `Cast` → `I64ToBytes` → `WriteStdout`,
|
||
three prims and no call at all; `flan_i64_to_bytes` renders into a static buffer in `flan_rt.c` and hands back a slice.
|
||
Printing a string literal is `Str` → `Bytes` → `WriteStdout`, and `Bytes` is a non-instruction because a string and a
|
||
`[u8]` are the same two words.
|
||
|
||
**What does drag conditions in is the bounds check and the allocator, and neither is a `Tast` node.** `check_at` and
|
||
`check_slice` call `flan_bounds_error` with the transfer channel and then `guard`; `Rt flan_vec_at` takes the channel
|
||
too. So it is `m.checks` and the container runtime that make a program need the condition machinery, not looping and
|
||
not printing — and because both are *emit-time* constructs rather than IR nodes, `hist.ml` cannot see either. That is
|
||
the correction to item 15's node table: the "no plan" row is not reached by writing a loop, it is reached by writing
|
||
`(at a i)`.
|
||
|
||
### Question 2 — what runs
|
||
|
||
| program | what it is for |
|
||
|---|---|
|
||
| `spike/x86/p1-exit.flan` | `main` returns 0. The assembly path, the runtime link, a real executable. |
|
||
| `spike/x86/p2-loop-print.flan` | a `dotimes` that prints — the smallest program that does something |
|
||
| `spike/x86/p3-fizz.flan` | the measured target: a call, an `if`, `%`, two string literals |
|
||
| `spike/x86/p4-convention.flan` | the *internal calling convention*, which p3 does not touch at all |
|
||
| `spike/x86/p5-core.flan` | a global with an initialiser, recursion, `break`, `continue`, the bitwise family, unsigned shifts, both directions of every conversion |
|
||
|
||
p4 is the one that matters most, because item 15 named the internal aggregate convention as the sharpest obstacle in
|
||
the whole report. It passes a struct by value, returns a struct by value, puts an `f32` through the SSE half, calls an
|
||
eight-argument function so that two arguments go on the stack, and passes a slice — and it agrees with LLVM. **The
|
||
obstacle really did dissolve the way the header claims**: a dev build is compiled entirely here and a release build
|
||
entirely by LLVM, the two never meet in one process, so the convention is ours to pick. Every aggregate goes by
|
||
pointer, an aggregate return is a hidden pointer in the first integer register returned in `rax`, and there is no
|
||
classifier in the file. Nothing had to be discovered by disassembling clang.
|
||
|
||
Over `test/programs` (111 files):
|
||
|
||
| | n |
|
||
|---|---|
|
||
| built through `--x86` and **matched** the LLVM build's output and exit status | **40** |
|
||
| built and diverged — `bounds.flan`, by design | 1 |
|
||
| refused by name: a node this backend does not lower | 40 |
|
||
| no `main` (package and library fixtures) | 6 |
|
||
| do not compile at all (the checker-error fixtures) | 22 |
|
||
| never terminate on their own (`dev-loop`, `dev-watch`) | 2 |
|
||
|
||
So of the 81 programs that compile, have a `main` and finish, **41 went through the hand-written backend and 40 were
|
||
byte-identical in output.** That includes `edn.flan` — sixty lines of output from a hand-written EDN reader with
|
||
unions, options, nested collections and a fixed-depth balance stack.
|
||
|
||
Proved by comparing output, never by reading bytes. The script builds both ways and diffs stdout and the exit status;
|
||
`objdump` was used only after a program already had the wrong answer. Item 15 is right that this is the only honest
|
||
order.
|
||
|
||
### Question 3 — the two bugs, and both are the shape item 15 predicted
|
||
|
||
**A `(set (.x (at pts 0)) 1.5)` wrote into a copy.** `lvalue` had no case for `At`, so it fell through to "evaluate it",
|
||
and the store landed in a temporary while the array kept its zeros. `emit.ml` has this as `addr`'s own `At` case. One
|
||
line. `array-ctor.flan` found it, and it found it as a segfault several statements later.
|
||
|
||
**A discarded value was stored over the return address.** This is the better one. A form whose value is thrown away was
|
||
handed a sink, and the sink was spelled as an address — `rbp+0`. That is the saved `rbp`, and `rbp+8` is the return
|
||
address, so a non-void form written in statement position stored straight over both; a 16-byte slice did it in one
|
||
`rep movsb`. `edn.flan` crashed by jumping into `.rodata`, **several statements after the mistake and in a different
|
||
function**, and the assembly at the jump read perfectly. The sink is now compared by identity and never used as an
|
||
address: anything with a value that is handed it gets a frame temporary instead.
|
||
|
||
The second bug cannot exist on the LLVM path, and that is the general point. LLVM has no notion of "store this value
|
||
nowhere" — an unused SSA value is simply unused. Every construct this backend has that LLVM does not is a place where
|
||
a bug can live that the LLVM backend's own testing can never have covered.
|
||
|
||
Against that, the thing item 15 was most worried about did *not* happen: **nothing went wrong with the frame or the
|
||
stack alignment.** `rsp` is written exactly twice — one rounded `sub` in the prologue that covers the temporaries and
|
||
the outgoing-argument area together, and `leave` — so `rsp % 16 == 0` at every call site is a property of one
|
||
subtraction rather than an invariant every case maintains. The spike's worst bug has no door to come in by, and p4
|
||
calls an eight-argument function to prove it.
|
||
|
||
### Question 4 — where the two backends now differ, and it is three named places
|
||
|
||
Item 15's question 4 listed nine undefined cases. Three of them are now *real* divergences with a build on each side,
|
||
and they should be written down before anyone uses this for anything.
|
||
|
||
| | LLVM | here |
|
||
|---|---|---|
|
||
| **A bounds violation** | `flan_bounds_error` signals; a `restart-case` can catch it; `bounds.flan` exits 134 | **no check at all**; `bounds.flan` exits 139 |
|
||
| `(uninit)` | `poison`, and the optimiser may reason from it | whatever the stack slot held — stable garbage |
|
||
| an exhausted `match`, a `noreturn` call | `unreachable`, undefined | `ud2` — a defined SIGILL at the instruction that fell through |
|
||
|
||
**The first is the one that matters, and it is not a footnote: `--x86` is silently a `--no-bounds-checks` build.** It is
|
||
silent because it is not a decision the backend made — `check_at` signals, signalling needs the channel and the guard,
|
||
and there is no guard here, so there is no check. `bounds.flan` is the direct evidence and `edn.flan`'s 33-brackets-
|
||
against-a-32-deep-stack case is the second. Anyone reaching for this flag on a program that indexes anything should
|
||
know that the trap is gone.
|
||
|
||
The other two are improvements and cost nothing. `ud2` in particular is two bytes and turns a class of miscompile into
|
||
a crash with an address.
|
||
|
||
### Question 5 — conditions, which is still the row with no plan
|
||
|
||
**They were not reached, and converting that into a checked precondition is the most useful thing in this report.**
|
||
|
||
There is no transfer guard after a call here, no landing pad and no transfer exit. `emit.ml` emits a guard after *every*
|
||
call; this emits none. What makes that sound is a whole-program argument rather than a hope: **if nothing in the
|
||
reachable set can ever write the channel, no call can ever return with it set.** So `check_no_transfer` walks the
|
||
linked program once per build and stops it — with the node's name and the function it is in — the moment it finds a
|
||
`signal`, an `invoke-restart`, a `restart-case`, a `handler-bind`, a `with-allocator`, or the two `Rt` symbols whose
|
||
bounds check signals.
|
||
|
||
That is what the 40 refusals are:
|
||
|
||
```
|
||
27 restart-case 4 handler-bind 1 with-allocator
|
||
7 signal 1 defers on the transfer path
|
||
```
|
||
|
||
Forty programs refused by name rather than miscompiled, and one line of build output says which node and where. A
|
||
backend that quietly omitted the guard would have compiled all forty and been wrong in a way no test distinguishes
|
||
from a race.
|
||
|
||
What this does *not* do is measure what conditions cost. That is still unknown, and it is still the only row of item
|
||
15's table with nothing behind it. What is now known is the shape of the bill: the guard is per call site, the pad is
|
||
per `restart-case` activation, `fdefers` needs a second exit path that no form in `body` can reach, and **any function
|
||
with `fdefers` at all is refused today** — which is most of the prelude's file and container code, and is why the
|
||
programs that use a `Vec` are not in the 40.
|
||
|
||
### The honest no-plan bucket
|
||
|
||
Everything below is refused by name at build time, not silently wrong.
|
||
|
||
- **Conditions, entire** — the guard, the landing pad, `emit_restart_case`, `emit_with_alloc`, the transfer exit, and
|
||
`fdefers` on it. Several hundred lines of `emit.ml` reimplemented from `spec-conditions.md` rather than ported.
|
||
- **Bounds checks**, which are the same work: `check_at` and `check_slice` cannot exist without the guard.
|
||
- **`Rt` with an aggregate return**, and with it most of the container runtime; `Vec`, `Map` and `Pool` have not been
|
||
exercised at all.
|
||
- **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` emits the symbol, which is correct for a whole-program build and
|
||
wrong the instant anything is redefined into it. This backend has no cells and no `--dev`; that is a deliberate
|
||
restriction and not an oversight, but it is exactly item 15's question 5 waiting where it was left.
|
||
- **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**. `idiv` raises `SIGFPE` where LLVM says undefined, and
|
||
`cvttsd2si` answers the integer-indefinite value. Unchanged from item 15: these want a language decision, not a
|
||
backend.
|
||
- **Debug information.** None. `--x86` and `--debug` together are refused.
|
||
- **Code size and speed.** Not measured. Every value is in memory, every intermediate is a frame temporary, and a
|
||
block copy is `rep movsb`; that is the trade the brief asks for and nobody has put a number on it.
|
||
|
||
### The verdict
|
||
|
||
**The wiring is done and it was the easy half. What is left is conditions, and the measurement moved them from "first
|
||
obstacle" to "the only obstacle".**
|
||
|
||
The order item 15 recommended was floats first and conditions last. Floats turned out to be one afternoon's encodings
|
||
and they are done. Conditions are still last and are now the *whole* remainder: they are what stands between 41
|
||
programs and the corpus, they are what a bounds check is made of, and `check_no_transfer` is the line that says so out
|
||
loud on every build until someone writes them.
|
||
|
||
Two things are worth doing before that, and both are cheap. Decide what a bounds violation means in a build with no
|
||
handler — because "no check" is what it means today and nothing says so. And take item 15's question 4 seriously now
|
||
that there are two backends to disagree: `(uninit)` and `unreachable` already differ, deliberately, and the difference
|
||
is currently documented only in a comment in `x86.ml`.
|
||
|
||
## 17. Conditions on the x86 backend, and with them bounds checks: the corpus, not 41 programs
|
||
|
||
Item 16's verdict was that conditions were the only obstacle left and that a bounds check was made of the same parts.
|
||
Both halves held. The transfer channel's guard, the landing pads, the per-function transfer exit, `fdefers` on it,
|
||
`emit_restart_case`, `emit_with_alloc`, `signal`, `error`, `handler-bind`, `invoke-restart`, `check_at` and
|
||
`check_slice` are all in `lib/x86.ml` now, written from `spec-conditions.md` and `emit.ml`'s semantics rather than
|
||
ported. **LLVM is untouched and still the default and the release backend, and `--x86` is still off by default and
|
||
still refused with `--dev`, `--debug`, `--sanitize` and every wasm target.**
|
||
|
||
`dune test --root .` was green twice on this work, and is green now apart from four raylib fixtures — `images` and
|
||
`audio`, each at `-O2` and `-O0` — which export to a *hardcoded* `/tmp` path and fail with
|
||
`Failed to export wave data` because `/tmp` is full. It is full because of a runaway `llc` in another lane writing a
|
||
5.8 GB `m4.o` out of a 13 KB `m4.ll`; the same four fail on two consecutive runs and nothing else does. Nothing in
|
||
this lane writes to `/tmp` by a fixed name, and `TMPDIR` does not reach those fixtures.
|
||
|
||
### Question 1 — the counts, and what was compared
|
||
|
||
`spike/x86/survey.sh`. Item 16 describes a script that builds every program both ways and diffs the output; it was
|
||
never committed, so this one is. It builds each program in `test/programs` and each probe in `spike/x86` twice — once
|
||
default, once `--x86`, **with the same bounds-check setting on both sides**, because a checked build compared against
|
||
an unchecked one says nothing about `bounds.flan` — runs both, and compares **stdout, stderr and the exit status**.
|
||
|
||
stderr is not a detail. Every message the new machinery produces goes there — the bounds and slice errors, the three
|
||
restart refusals, the transfer failure — and each carries a `Loc.to_string` string this backend emits by hand as a
|
||
`.rodata` label and a length in a register. An exit status of 134 with the wrong text beside it is exactly the failure
|
||
that reads as a match.
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| **MATCH** — same stdout, same stderr, same exit status | **41** | **89** |
|
||
| **DIFFER** | 1 (`bounds.flan`) | **0** |
|
||
| **refused by name** — a node this backend does not lower | 41 | **0** |
|
||
| skipped: does not compile (checker-error fixtures) | 25 | 25 |
|
||
| skipped: no `main` (package and library fixtures) | 6 | 6 |
|
||
| skipped: never terminates (`dev-loop`, `dev-watch`) | 2 | 2 |
|
||
|
||
The "before" row is measured, not quoted from item 16 — it is the same corpus seven files larger, and the 41 refusals
|
||
split as 26 `restart-case`, 7 `signal`, 4 `handler-bind`, 1 `with-allocator`, 1 `fdefers`, and one each for
|
||
`flan_vec_at` and `flan_vec_as_slice`.
|
||
|
||
**Every program in `test/programs` that compiles, has a `main` and terminates now goes through the hand-written backend
|
||
and agrees with the LLVM build.** That is the whole corpus: `restarts.flan`, `conditions.flan`, `bounds.flan`,
|
||
`bounds-condition.flan`, `allocators.flan`, `defers.flan`, the `Vec` and `Map` programs, `edn.flan`, `format.flan`.
|
||
The programs that only trap when given an argument — `restarts.flan`'s four signature-mismatch cases, `bounds.flan`'s
|
||
three — were run by hand with their arguments and agree on stderr and on exit 134 as well.
|
||
|
||
### Question 2 — what a bounds violation means in a build with no handler
|
||
|
||
**The same thing it means on the LLVM path, and that is the answer rather than a decision.** `check_at` and
|
||
`check_slice` here are `emit.ml`'s: a compare, a branch, a call to `flan_bounds_error` or `flan_slice_error` with the
|
||
transfer channel, and then **the guard** — which is why they could not exist before. The call is an ordinary one that
|
||
returns only when a handler or the break loop transferred, so the guard is the way out and the fall-through past it is
|
||
`ud2` where `emit.ml` writes `unreachable`.
|
||
|
||
So: a bounds violation signals `BoundsError`; a `handler-bind` can answer it; a `restart-case` catches the transfer
|
||
and its clause's value stands; an unanswered one dies inside the runtime and the program exits 134 with the location
|
||
and the index. `--no-bounds-checks` omits the check on both backends and both then exit 139. **`bounds.flan` has
|
||
stopped being a DIFFER**, and the row of item 16's question-4 table that said "no check at all" is gone rather than
|
||
documented.
|
||
|
||
The transitional refusal item 16 asked for — a build that says out loud that its behaviour differs — was not written,
|
||
because `check_at` landed in the same pass and the refusal would have been created and retired inside one commit.
|
||
There is nothing left to be loud about.
|
||
|
||
Item 16's other two divergences are unchanged and still deliberate: `(uninit)` reads whatever the slot held rather
|
||
than LLVM's `poison`, and an exhausted `match` is `ud2` rather than `unreachable`. Both are still documented only in
|
||
`x86.ml`.
|
||
|
||
### Question 3 — `check_no_transfer` narrowed, not removed
|
||
|
||
It was a whole-program argument: this backend emitted no guard, which is sound exactly when nothing reachable can
|
||
write the channel, so the build refused by name the moment it found something that could. Every call site is guarded
|
||
now and the argument has retired — **in every function.** It still stands in one place, so the walk is still there and
|
||
now walks only global initialisers:
|
||
|
||
A global's initialiser runs from `flan..init-globals`, before `main` and before anything has established a handler or
|
||
a restart. It owns its own channel cell because no caller hands it one, so a transfer out of it has nowhere to go —
|
||
its exit would return into the loader. `signal`, `error`, `restart-case` and `handler-bind` in a `defvar` initialiser
|
||
are refused by name. A bounds check there is *not* refused: it signals into a cell nothing is listening on, finds no
|
||
handler, and dies, which is the right answer.
|
||
|
||
### Question 4 — five bugs, and four are item 16's shape exactly
|
||
|
||
Found by what the programs printed, never by reading bytes. Two of them existed before this work and only became
|
||
reachable once the guard let the programs that expose them compile.
|
||
|
||
**The body fell through into the transfer exit.** Every `fdefer` ran twice on a normal return, so `restarts.flan`'s
|
||
`log` was one too high at every checkpoint and nothing else was wrong. `emit.ml` cannot have this bug: its `ret`
|
||
terminates the block, and there is no fall-through to forget. This is the same class as item 16's "a discarded value
|
||
was stored over the return address" — a construct this backend has that LLVM does not.
|
||
|
||
**A `Vec` crossed to the runtime as the address of a copy.** `eval` copies an aggregate into a temporary, so
|
||
`flan_vec_push` grew the temporary and the caller's header stayed at length zero — and an *in-bounds* `(at v 1)` then
|
||
signalled against a length of 0. `emit.ml` says so in a comment beside its own `addr`; this had no such case. Pre-
|
||
existing, and invisible until a program using a `Vec` could build.
|
||
|
||
**`ucomis` sets CF, ZF and PF together for a NaN**, so `sete` answered *true* for `(= x x)`. Flan's comparisons are
|
||
LLVM's *ordered* ones (`oeq`, `olt`, ...), which are false for a NaN; the unsigned table is not those. `<` and `<=`
|
||
now swap their operands and ask for a/ae, and `=` and `!=` take a `setnp` beside them. The symptom was
|
||
`(/ 0.0 0.0)` formatting as `-9223372036854775808`, because the prelude's NaN test is `(not (= x x))` and nothing
|
||
else. Pre-existing; `format.flan` could not build before.
|
||
|
||
**A union read field 0 through the struct table** and was refused by name. A union is a tag and a payload blob, which
|
||
is a two-field struct at that level, and the structural printer reads the tag without unwrapping the value.
|
||
|
||
**And one that could not have been found later:** `emit_globals_init` stored a null *into* the channel slot rather
|
||
than a cell's address into it, so every callee of a global initialiser was handed a null pointer to write a transfer
|
||
through. Harmless while nothing could transfer; a fault the first time a guard loaded through it. `emit_main` had it
|
||
right and was the model.
|
||
|
||
Against those: nothing went wrong with the frame, the stack alignment, or the pads' nesting. The one place worth
|
||
naming is the one item 16 could not have: **the channel is one indirection deeper here than in `emit.ml`.** There
|
||
`%xfer` is an alloca and the target is one `load` away; here `xfer_off` is a frame slot *holding the caller's
|
||
pointer*, so reading the target is two loads and clearing the channel is a store *through* the pointer and never a
|
||
store to the slot. `chan_into`, `xfer_load`, `xfer_store` and `xfer_clear` exist so that no call site has to remember
|
||
which.
|
||
|
||
### Question 5 — what the corpus does not walk, and the probe that does
|
||
|
||
Every pad has two halves: the one a body reaches by finishing, and the one a transfer reaches by passing through. The
|
||
corpus walks the first everywhere and the second in one place only — `allocators.flan`'s last case aims an
|
||
`invoke-restart` out of a `with-allocator` body at a `restart-case` outside it, which is the `wxfer` re-propagation.
|
||
|
||
The other two it never reaches. In `restarts.flan` every transfer stops at a `restart-case` *inside* the
|
||
`handler-bind`'s extent, so the handler frames never come off on the transfer path; and in `nested` and `shadowed` the
|
||
*inner* restart frame offers the name, so a restart-case that the transfer is not aimed at never has to put the target
|
||
back. `spike/x86/p6-transfer.flan` is those two, beside a defer and a clause parameter, and it agrees with the LLVM
|
||
build. It is in the survey, and it is why the count is 89 rather than 83.
|
||
|
||
The branch nothing exercises is `flan_transfer_fail` — a defer that starts a *second* transfer while the first is
|
||
unwinding. It is emitted and refused loudly, and it is untested.
|
||
|
||
### The honest no-plan bucket
|
||
|
||
- **`Rt` with an aggregate return.** Still refused by name. `flan_vec_as_slice` returns a slice by value, and
|
||
`bounds-condition.flan` exercises it both in and out of bounds through a `restart-case` and matches — so it does not
|
||
reach the refusal, and **nobody traced why.** That is a loose end, not a result.
|
||
- **`Fnval`'s indirection cell.** `FnAddr (Fnval n)` still emits the symbol. Correct for a whole-program build, wrong
|
||
the instant anything is redefined into it; there are no cells here and no `--dev`, deliberately.
|
||
- **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**. Unchanged from items 15 and 16: `idiv` raises `SIGFPE`
|
||
where LLVM says undefined. A language decision, not a backend one.
|
||
- **`(uninit)` and `unreachable`** still differ from LLVM on purpose and are still written down only in a comment.
|
||
- **The `flan_transfer_fail` branch**, above.
|
||
- **`"defers on a transfer path nothing reaches"`** — the refusal that replaced the old `fdefers` one. No program in
|
||
the corpus hits it; it exists so that if the reasoning behind it is ever wrong, it says so.
|
||
- **Debug information.** None. `--x86` and `--debug` together are still refused.
|
||
- **Code size and speed.** Still not measured, and now there is more to measure: a guard after every call, two loads
|
||
and a branch each, and a bounds check that spends three frame temporaries. Nobody has put a number on any of it.
|
||
|
||
### The verdict
|
||
|
||
**The row with no plan is gone.** What item 15 called the last obstacle and item 16 called the only one is written,
|
||
and the measurement that made it the only one is the measurement that says it is finished: every program in the corpus
|
||
that can run, runs, and prints what LLVM's build prints — down to stderr.
|
||
|
||
What is left in `x86.ml` is not conditions. It is a container return convention, a redefinition cell, two arithmetic
|
||
edge cases the language has not decided, and no debug info. None of those is the shape conditions were: each is a
|
||
known thing in a known place, and the guard is not underneath any of them.
|
||
|
||
## 18. The container return convention was never there, and the redefinition cell now is
|
||
|
||
Item 17's no-plan bucket opened with two correctness items and a loose end it was honest about. Both items are closed
|
||
and the loose end is the reason the first one closed the way it did rather than the way it was written down. **LLVM is
|
||
untouched and still the default and the release backend.** `--x86` is still off by default and still refused with
|
||
`--debug`, `--sanitize` and every wasm target; it is **no longer refused with `--dev`**, and question 3 is the argument
|
||
for that.
|
||
|
||
### Question 1 — the counts, and a corpus that moved
|
||
|
||
`spike/x86/survey.sh`, unchanged in what it compares: every program in `test/programs` and every probe in `spike/x86`,
|
||
built both ways with the same bounds-check setting, run, and diffed on **stdout, stderr and exit status**.
|
||
|
||
| | item 17 | measured here, before | after |
|
||
|---|---|---|---|
|
||
| **MATCH** | 89 | **93** | **97** |
|
||
| **DIFFER** | 0 | 0 | **0** |
|
||
| **refused by name** | 0 | **2** | **0** |
|
||
| skipped: does not compile | 25 | 28 | 28 |
|
||
| skipped: no `main` | 6 | 6 | 6 |
|
||
| skipped: never terminates | 2 | 2 | 2 |
|
||
|
||
**The baseline is not the one the brief quoted, and that is the first finding.** Item 17 measured 89/0/0; this lane
|
||
measured 93 MATCH and **2 refusals** before writing anything. The corpus moved underneath: another lane landed
|
||
`(slice-from-ptr p n)`, and it arrived as two refusals rather than one — `slice-from-ptr.flan` and `bounds.flan` both
|
||
stopped building through `--x86`. A backend that refuses by name does not rot quietly, but it does rot, and nothing
|
||
was watching. That is an argument for running the survey in CI rather than in a lane.
|
||
|
||
The form itself is nothing: a `Slice _` is `{ptr, i64}` here exactly as it is in `emit.ml`, so it is one store of the
|
||
pointer and one of the length and no new representation at all. The half worth writing down is that **the check has to
|
||
be signed**. There is nothing to compare the length against — only the caller knows what is behind that pointer — so
|
||
what is checked is that the promise is not absurd, and `check_slice`'s own compares are *unsigned*. A negative `i32`
|
||
sign-extended to 64 bits is a huge unsigned value that `jbe` waves straight through, and the result is a slice about
|
||
2^64 long that reads as a pass and faults somewhere else entirely.
|
||
|
||
Nothing in the corpus walks that path, because every length in `slice-from-ptr.flan` is a literal and a negative
|
||
literal is refused by `check.ml` before any code is emitted. `spike/x86/p7-slice-from-ptr.flan` takes the length as a
|
||
parameter and runs it through a `restart-case`, which puts the condition's `low`/`high`/`length` on stdout beside the
|
||
LLVM build's.
|
||
|
||
The other two of the four new MATCHes are this lane's own probes, below.
|
||
|
||
### Question 2 — why `flan_vec_as_slice` avoided the aggregate-return refusal
|
||
|
||
**It is the first of the two possibilities item 17 named: the refusal is narrower than it reads, and nothing is going
|
||
right by accident.**
|
||
|
||
`flan_vec_as_slice`'s Flan-level return type is `Unit`. `check.ml:3721` builds it as `rt loc Types.Unit`,
|
||
`flan_rt.c:1165` is `void flan_vec_as_slice(flan_vec *v, void *out, ...)`, and `emit.ml:2464` declares it `void`. So
|
||
`is_void rty` answers first and the `is_agg rty` test below it is never reached. The slice comes back through an
|
||
out-pointer the checker allocated, which is not one symbol's accident but the convention:
|
||
|
||
- **Every aggregate-valued runtime result crosses through an out-pointer.** Every other `rt` builder in `check.ml`
|
||
answers `Unit`, an `Int`, a `Ptr`, an `Alloc` or a `Handle`. `flan_pool_resolve` answers `Ptr elem` and the `Option`
|
||
is built in Flan; `Argv` is its own `Tast` node with its own out-pointer and never comes through here at all.
|
||
- **`crossable`**, the other user of the same code path, admits `String` and `Slice _` only as *a parameter* and
|
||
refuses an aggregate return from a `declare` outright.
|
||
|
||
**So there is no sret convention to build for `Rt`, and building one would have been worse than the refusal.** This is
|
||
the C boundary, where the header says the backend must match SysV rather than pick: a 16-byte slice comes back in
|
||
`rax:rdx`, not through the internal hidden-pointer convention. There is no classifier in the file, there is nothing to
|
||
test one against, and "untestable and wrong" is a bad trade against a line that costs nothing. The line stays as a
|
||
guard against those two rules changing, and now names which rules and what the work would actually be.
|
||
|
||
**Item 16's claim that the container runtime is unexercised went with it, and it was already stale when item 17
|
||
repeated it.** `Vec` and `Map` run through `vec.flan`, `vec-of-vec.flan`, `vec-in-struct.flan`, `maps.flan` and
|
||
`map-iter.flan`; `Pool` — which neither report checked — runs through `registry.flan`, `handles.flan`,
|
||
`generics.flan` and `pool-stale-region.flan`. All match, and they have since item 17's guard landed.
|
||
|
||
### Question 3 — the cell, and why `--x86 --dev` is no longer refused
|
||
|
||
`FnAddr (Fnval n)` emitted the symbol. It now reads the cell — and **so does every direct call**, which is the half
|
||
that matters and is what `emit.ml`'s `body_of` does: a redefinition is one store, and its whole purpose is to reach
|
||
call sites that already exist. What is emitted, all of it behind `dev`:
|
||
|
||
- **One cell per function** in `.data`, `.globl`, initialised to the body this build compiled. Spelled exactly as
|
||
`Emit.cellname` spells it, because that is the point of having one here — an LLVM-built redefinition module binds
|
||
`@"flan.cell.<n>" = external global ptr` against whatever built the host. `nm -D` over the two builds of the same
|
||
program gives identical sets of 68 cell symbols.
|
||
- **The cell load placed after the arguments.** `emit.ml` has that as a load-bearing comment: a redefinition landing
|
||
between two calls must not land in the middle of one. `CallPtr` stays the other way round, for `emit.ml`'s reason.
|
||
- **The `flan_dev_reg_enable` constructor**, which arms the allocation registry.
|
||
|
||
Not emitted: `Emit.cellptr`, the deeper spelling for a name the host was never built with. It cannot arise in a
|
||
whole-program build, where `known` is true of everything, and it belongs with the redefinition module that would
|
||
introduce such a name.
|
||
|
||
**The refusal is relaxed, and the argument is that `flan dev` never reaches that fork.** `--x86` is read in exactly one
|
||
place, `flan build`'s argument list; the daemon builds its host through `Build.executable` and its modules through
|
||
`Build.shared` without it, and there is no spelling that hands it one. So the flag now means what it says — a host
|
||
whose call sites are redefinable, built by this backend — and nothing claims the module that would redefine through
|
||
them exists.
|
||
|
||
### Question 4 — what made that believable, because the corpus cannot
|
||
|
||
Two measurements, and the second is the one that matters.
|
||
|
||
**The corpus with `--dev` on both sides: 97 MATCH, 0 DIFFER.** `SURVEY_FLAGS=--dev` is opt-in so the counts in question
|
||
1 stay the same measurement. Before the constructor was added it read **96/1**: `registry.flan` asks `(live? ...)` and
|
||
got four zeroes, because the allocation registry was never armed. That is the whole of what a dev host does
|
||
differently besides the cells, and it is worth saying that the corpus found it — one program out of 97 observes it.
|
||
|
||
**And the thing the corpus structurally cannot test.** A dev build starts with every cell pointing at the body this
|
||
build compiled, so it prints exactly what a release build prints *whether or not anything reads the cell*. The
|
||
property that makes the whole corpus a safe test of the cells is the property that makes it a useless one.
|
||
|
||
So `spike/x86/cells.sh` changes what a cell holds. It preloads a shared object whose constructor looks up
|
||
`flan.cell.twice` with `dlsym` — the cells are in `.dynsym` because a dev build is `-rdynamic` — and stores a different
|
||
body there. That is the one store a redefinition ends in, done from outside, with no compiler involved. Four builds,
|
||
and the two controls are half the test:
|
||
|
||
| | |
|
||
|---|---|
|
||
| `llvm --dev` | 22 22 — the cell is read |
|
||
| `x86 --dev` | 22 22 — **this lane's claim** |
|
||
| `llvm` | 42 42 — no cell; `dlsym` answers NULL |
|
||
| `x86` | 42 42 |
|
||
|
||
`spike/x86/p8-cell.flan` has both call shapes, because they are two different cases in both backends: a direct call by
|
||
name, and a function *value*, which is the one `FnAddr` that is not the symbol. The release rows are what say the
|
||
change came from the indirection and not from ordinary symbol interposition.
|
||
|
||
### Question 5 — the licence in the header now has an edge, and it is the cell
|
||
|
||
This is the thing the next lane inherits, and it is worth more than either item above.
|
||
|
||
`x86.ml`'s header licenses its own calling convention on the grounds that **a dev build is compiled entirely by this
|
||
backend and a release build entirely by LLVM, so the two never meet in one process.** That is what dissolved item 15's
|
||
sharpest obstacle and it is why there is no classifier in the file.
|
||
|
||
**Publishing a cell an LLVM-built module can store into is the first thing that could make it false.** The two
|
||
conventions agree on scalars and disagree on every aggregate — this backend passes each by pointer and returns one
|
||
through a hidden `sret`, LLVM classifies — so an `Emit.redefinition` module dlopened into an `--x86` host would be
|
||
correct exactly until the first redefined function took or returned a struct. `cells.sh` does not reach it, because
|
||
the body it installs is `(i64, void *) -> i64` and the conventions agree there.
|
||
|
||
Nothing in the toolchain does that today: `flan reload` and `flan dev` build host and module through LLVM together,
|
||
and neither accepts `--x86`. But the lane that wires this backend into the dev loop will be the one that does it, and
|
||
**the answer then is a redefinition emitter here, not a classifier.** Written into both `x86.ml`'s header and
|
||
`build.ml`'s refusal so that it is found before it is discovered.
|
||
|
||
### The honest no-plan bucket
|
||
|
||
- **A redefinition emitter**, which is question 5 and is new to this list: `Emit.redefinition` has no counterpart here,
|
||
so a `--x86 --dev` host has cells nothing in the toolchain can yet write.
|
||
- **`f64` → `i64` out of range**, and **`INT64_MIN / -1`**. Unchanged from items 15, 16 and 17: `idiv` raises `SIGFPE`
|
||
where LLVM says undefined. A language decision, not a backend one, and nobody has taken it in three reports.
|
||
- **`(uninit)` and `unreachable`** still differ from LLVM on purpose and are still written down only in `x86.ml`.
|
||
- **The `flan_transfer_fail` branch** — a defer starting a second transfer while the first unwinds. Emitted, refused
|
||
loudly, still untested.
|
||
- **`"defers on a transfer path nothing reaches"`** — still reached by no program in the corpus.
|
||
- **`flan_dev_reg_note` is not dropped in a release build here.** `emit.ml` drops the whole family when `dev` is off;
|
||
this emits real calls to a registry that is disabled, so they are no-ops that cost a call each. Correct, not free,
|
||
and `emit.ml`'s stated reason for the drop — an escaped alloca `mem2reg` would refuse — does not apply to a backend
|
||
with no `mem2reg`.
|
||
- **Debug information.** None. `--x86` and `--debug` together are still refused.
|
||
- **Code size and speed.** Still not measured, and the list of what to measure has not got shorter: a guard after every
|
||
call, a bounds check spending three frame temporaries, every intermediate in memory, `rep movsb` for a block copy —
|
||
and now an extra load at every call site in a dev build, which is the one item on this list that `emit.ml` pays too.
|
||
|
||
### The verdict
|
||
|
||
**Neither of the two correctness items was the shape it was written down as, and finding that out was most of the
|
||
work.** The container return convention did not exist to be built; the loose end item 17 flagged was the answer and
|
||
not a symptom, and one afternoon spent tracing it saved a SysV classifier nobody could have tested. The cell was real,
|
||
took thirty lines, and could not be tested by anything in the corpus — which is why the useful artefact from it is a
|
||
preloaded `dlsym` and not a program.
|
||
|
||
What is left is a redefinition emitter, two arithmetic edge cases the language still has not decided, and no debug
|
||
info. The backend is no longer the thing standing between here and the dev loop.
|