Joseph Ferano
57fe91f303
FIX.org, NEXT.md, DISCUSS.org, docs/DISCUSS.md and the session handoff at the root are one TODO.org now: 293 entries under seven subsystem headings, each carrying an org keyword that says where it stands. A DONE entry is a few lines saying what was decided and what that rules out; the reasoning that would not compress — the embedding spike and the four reports the hand-written x86 backend was built from — moved into docs/BUILT.md instead, and its entries point there in one line. Every entry was checked against the tree before it got a keyword, and the prose was wrong in both directions. Things the deleted files called open were built: the first-evaluation stall, main being redefinable, macro parameter lists, the type-limit constants, the array constructors, the byte fills, inc/dec, the discard's fontification, the Emacs buffers, rt_die's _exit, the backtrace surface, and the acceptance failure that could print and still exit zero. Things they called done were not: the backend reports' no-plan buckets had gone stale in the other direction, the value-dependent defvar was superseded rather than built, and macro-expansion source locations are on an unmerged lane, so that entry is NEXT and names the branch. Every comment that cited one of the five by name now cites a heading that exists, in TODO.org or in docs/BUILT.md. The session reports under docs/handoffs/ keep naming the files they worked on, because rewriting them would falsify what those sessions did; each carries a note saying where the content went.
118 lines
8.2 KiB
Markdown
118 lines
8.2 KiB
Markdown
# Handoff — a redefinition emitter in `lib/x86.ml`
|
||
|
||
Branch `dev-loop`, worktree `agent-a884e1e2fcffe5052`, from `b1cc67b`. Item 1 of `HANDOFF-x86-rt.md` §6.
|
||
|
||
**It works and it is checked by running it.** An `--x86` host dlopens an `--x86` redefinition module, the
|
||
module publishes a new body into the host's cell, and the host's un-rebuilt call site follows it.
|
||
`test/test_reload.ml` is green on both backends.
|
||
|
||
## Baseline, measured before and after
|
||
|
||
| | before (`b1cc67b`) | after |
|
||
|---|---|---|
|
||
| `spike/x86/survey.sh` | 97 MATCH / 0 DIFFER / 0 refused | (see §"After", below) |
|
||
| `spike/x86/cells.sh` | 4/4 ok | 4/4 ok |
|
||
| `dune test --root .` | exit 0 | exit 0 |
|
||
|
||
`dune test` prints two `ld` / `clang` failures from inside the `dev-robust` fixture and still exits 0. Check the
|
||
**exit code**, not the output; grepping will mislead you.
|
||
|
||
## Why this had to exist
|
||
|
||
`x86.ml` licenses its own calling convention on the grounds that a dev build is compiled entirely by it and a
|
||
release build entirely by LLVM, *and the two never meet in one process*. The conventions agree on every scalar
|
||
and disagree on **every aggregate** (here: every aggregate by pointer with a hidden `sret` in the first integer
|
||
register; LLVM classifies by eightbyte). An `Emit.redefinition` module dlopened into an `--x86` host is correct
|
||
until the first redefined function takes or returns a struct. So an `--x86` host gets `--x86` modules.
|
||
**There is still no aggregate classifier and there must not be one.**
|
||
|
||
## The crux, and the answer
|
||
|
||
`program` emits an executable; every symbol it names it defines, or the runtime does, and a pc-relative `PC32`
|
||
reference is right for all of them. A redefinition module is a `.so` and almost everything it names belongs to
|
||
the **host executable** — the cells `flan.cell.<n>`, the globals `flan.<g>`, the runtime's entry points. A
|
||
`PC32` relocation against an undefined symbol cannot be used in a `-shared` link; `ld` refuses it.
|
||
|
||
So a reference to a symbol the object does not define goes through the GOT. Measured, not recalled:
|
||
|
||
- The spelling is `.long sym@GOTPCREL - 4`. Quoted symbols (`"flan.cell.x"@GOTPCREL`) are accepted.
|
||
- **`- .` is wrong.** `@GOTPCREL` is already pc-relative; written `- . - 4` as the plain form needs, `as`
|
||
produces an addend of **-8** and the load reads the wrong slot. `readelf -r` on the object is how you see it:
|
||
want `flan.cell.x - 4`.
|
||
- `llc -relocation-model=pic` emits `movq x@GOTPCREL(%rip), %rbx` for `@x = external global i64`, which is the
|
||
same instruction bytes this backend already emits for a `Sym` operand. Only the relocation changes.
|
||
|
||
## What was built
|
||
|
||
| file | what |
|
||
|---|---|
|
||
| `lib/x86.ml` — `modrm_got`, `mem`'s new `Got` case | the `@GOTPCREL` field. `mem_op` and `store_int`'s byte case were the only two matches on `mem` |
|
||
| `lib/x86.ml` — `loc`'s new `Lgot` case | a symbol the object does not define. `lmem` loads the GOT slot into `scratch` and answers `Reg (scratch, d)` — the `Lp` shape, so a field offset is arithmetic on a register and never on the relocation |
|
||
| `lib/x86.ml` — `fnctx.ext` | `string -> bool`, true of a symbol this object does not define. **Default `false`**, so the whole-program path emits byte-identical output and the survey is protected structurally |
|
||
| `lib/x86.ml` — `sym_loc`, `addr_sym`, `load_sym` | the three spellings of naming a symbol; each picks pc-relative or GOT off `ext` |
|
||
| `lib/x86.ml` — `emit_fn ?ext ?hidden` | `.hidden` on a module's own bodies |
|
||
| `lib/x86.ml` — `program`, the no-`main` case | was `unsupported "no main"`, now emits none, as `emit.ml` does. A program linked against a C host that brings its own entry point is legitimate — `reload_host.c` is exactly that, and the refusal made an `--x86` host for the reload tests impossible to build |
|
||
| `lib/x86.ml` — `redefinition` | the emitter. No `main`, no `.init_array`, no `flan..init-globals`, no `flan_dev_reg_enable` ctor, no `.bss` for globals, no `.data` for cells. Bodies `.hidden`; one `flan_reload_install` that reads each cell's address out of the GOT and stores the new body's `lea` into it |
|
||
| `lib/build.ml` — `shared_x86` | `as` + `ld -shared`, the same link `shared` does. No `-relocation-model=pic` to ask for: the emitter already writes every foreign reference through the GOT |
|
||
| `test/test_reload.ml` | the x86 section: an `--x86` host, two `--x86` modules, the same transcript, plus the refusal check |
|
||
|
||
## The one real bug, because it is the shape of the next one
|
||
|
||
`call_flan`'s `` `Cell `` target loaded **one** level: `mov r11, cell@GOTPCREL(%rip)` then `call *%r11`. That
|
||
reads as "call through the cell" in a disassembly and in fact *calls the cell*. The GOT slot holds the cell's
|
||
**address**; reading what the cell holds is two loads, not one. Found by the segfault, not by reading — which is
|
||
../DISCUSS.md item 15's claim (now docs/BUILT.md, "The bugs, and the shape they share"), now with a sixth
|
||
instance. `load_sym` is the fix and is the only helper that
|
||
double-loads; `Lgot` gets it right for free because `lmem` already had the shape.
|
||
|
||
## Scope, deliberately narrower than `Emit.redefinition`
|
||
|
||
Only names the **host already has**. Refused by name, each with the reason in the message:
|
||
|
||
- a function new to the session — needs `flan_dev_cell` and `Emit.cellptr`'s second spelling (a module-local
|
||
slot resolved by string at install time)
|
||
- a global new to the session — needs `flan_dev_global`, plus the init constant that travels with it
|
||
- `consts` — republishing a `defconst`
|
||
- `call` — the transient `flan_reload_call` thunk and the `@flan_reload_transient` marker the agent unloads on
|
||
|
||
That is `C-c C-c` on an existing `defn`, which is the demo, and it is what `test/programs/reload.flan` and
|
||
`reload-v2.flan` exercise (v3 and v4 are the new-name cases and are asserted to refuse).
|
||
|
||
## What remains
|
||
|
||
1. **The new-name path.** `flan_dev_cell` / `flan_dev_global` and `Emit.cellptr`'s spelling, so that a `C-c C-k`
|
||
introducing a function or a `defvar` works on `--x86`. `test/test_reload.ml`'s v3/v4/v5 are the fixtures
|
||
already written; the refusal check there is what to replace.
|
||
2. **The transient thunk** — `flan_reload_call` plus `@flan_reload_transient`, which is what an *expression*
|
||
evaluation (the break loop, `C-x C-e`) compiles to. Note `emit.ml`'s three conditions for emitting the
|
||
marker, including `m.nstr = 0`: a module holding a string literal can never say it is transient.
|
||
3. **Wiring it to `flan dev` / `flan reload`.** Today nothing reaches this: `--x86` is read only by
|
||
`flan build`'s argument list in `bin/main.ml`, and the daemon builds host and module through
|
||
`Build.executable` / `Build.shared` with no `x86` field set. When it is wired, the host's backend and the
|
||
module's backend must be chosen together — that is the whole licence.
|
||
4. **A redefined function that takes and returns a struct.** This is the case that motivated the whole lane —
|
||
the two backends agree on every scalar and disagree on every aggregate — and it is exercised by *nothing*,
|
||
on either side of the reload boundary. `reload.flan`'s `bump` is `-> i64` and `helper` is `i64 -> i64`. The
|
||
argument that an `--x86` host plus `--x86` modules is same-convention-by-construction is sound, but it is an
|
||
argument and not a measurement. Do not edit `reload.flan` for it — its transcript is shared with the LLVM
|
||
path; add a fixture beside it.
|
||
5. Items 2–7 of `HANDOFF-x86-rt.md` §6, unchanged.
|
||
|
||
Also still true and still worth doing: **run `spike/x86/survey.sh` in CI**.
|
||
|
||
## After, measured
|
||
|
||
| | before (`b1cc67b`) | after |
|
||
|---|---|---|
|
||
| `spike/x86/survey.sh` | 97 MATCH / 0 DIFFER / 0 refused | **97 / 0 / 0**, skips `28 does-not-compile / 6 no-main / 2 runs-forever` — identical |
|
||
| `spike/x86/cells.sh` | 4/4 ok | **4/4 ok** |
|
||
| `dune test --root .` | exit 0 | **exit 0**, run without a pipe |
|
||
|
||
The skip breakdown is the line that matters beyond the MATCH count, because the no-`main` relaxation is the one
|
||
change here not gated behind `ext` being false. It cannot in fact move it: `survey.sh:70` classifies `no-main`
|
||
off the **LLVM** build failing at the link, before it ever runs the x86 one.
|
||
|
||
And run `dune test --root .` **without a pipe**: piping to `tail` gives you `tail`'s exit status, which is the
|
||
trap this file warns about at the top, and the `dev-robust` fixture puts `ld` and `clang` failure text in the
|
||
output either way.
|