flan/docs/handoffs/HANDOFF-x86-annotate.md

# Handoff — `flan emit --x86` annotated, so its output can be read

Branch `dev-loop`, from `f772162`.

The x86 backend exists so that a dev build is ours end to end and so that a reader can be told *why* each
instruction is there. Until now `flan emit --x86` printed a three-line file header and then nothing but
`.byte` blobs: no source form, no frame key, no statement of the calling convention, nothing naming the
bookkeeping. Reading it meant `as` plus `objdump`, and even that answers only *what*.

**It is annotated now.** Every run of bytes is headed by the Flan form that produced it; every function
carries a frame map and a statement of how its arguments arrived; and every piece of bookkeeping the
compiler adds has a name where it appears and an explanation once, in a legend at the top of the file.

## What was built

| file | what |
|---|---|
| `lib/loc.ml` | `Loc.snippet` — the text of a span, on one line, whitespace collapsed, or `None` where there is no readable source. The squiggle's other half: the squiggle points at a form in its own file, this quotes it somewhere the file is not |
| `lib/x86.ml` | the annotation machinery: a queue of pending comments on `buf`, the per-form hook in `lower`, `frame_map`, the bookkeeping `note`s, and the legend |
| `bin/main.ml` | `emit --x86` annotates; `--no-annotate` is the bare spelling |
| `spike/x86/annot.sh` | **new** — emits every program in the corpus both ways, assembles both, and compares every section of the two objects byte for byte, in all three of default, `--dev` and `--debug` |
| `spike/x86/dump.sh` | its x86 section now shows the annotated listing *and* the disassembly of the object that listing assembles to — why beside what |

`lib/build.ml` was not touched. A `--x86` build emits exactly the assembly it emitted before.

## The two decisions

**Annotation is always on for `flan emit --x86` and always off for a `--x86` build.** `flan emit` exists to
be read by a person, and there is no reason to make a reader ask for the thing the command is for. A build's
`.s` is a temporary file handed straight to clang and read by nobody, so leaving it untouched costs nothing
and buys something: `survey.sh`'s 103 MATCH stays a statement about the same text it has always been about,
rather than about text this lane rewrote. `--no-annotate` exists for exactly one consumer, `annot.sh`.

**None of this belongs to the LLVM path.** `flan emit` already prints IR that names its values, carries
`!dbg` on every instruction and a `!DILocalVariable` per slot. The problem being fixed here is one this
backend has and LLVM does not.

## Mnemonics: not done, deliberately

The brief allowed a trailing mnemonic per line as a bonus and said not to write a disassembler for it. No
disassembler was written — but the honest reason is not that one would be needed. The encoder knows the
mnemonic at the moment it emits the bytes; threading a text trace through every one of its entry points,
with memory-operand formatting to match, is a larger change than the request and would touch every encoding
function in the file, which is the part of it the survey is a structural check on.

What replaces it is cheaper and arguably better: `dump.sh` now prints the annotated `.s` and `objdump`'s
disassembly of the same object side by side. The disassembly says what the instructions are; the listing
says why they exist. Neither answers the other's question, and having both is what the four-way comparison
was for.

## How the annotation is attached, and why that shape

The bytes accumulate in `buf.pend` and flush as one `.byte` directive. A comment therefore cannot simply be
written when a form starts lowering — the form may emit nothing, and a heading left standing would be read
as belonging to whatever came next.

So headings are **queued, not written**. `annote` puts one on `buf.ann` with a serial; the next byte written
flushes the pending directive and then the queue, so the comment lands immediately above the bytes it is
about; and `unannote` withdraws whatever a form queued and never spent. The serials are monotonic, so
"was mine written?" is one integer comparison against the highest serial ever written.

The hook is in `lower`, beside `dwline`, and for the same reason `dwline` is there: the recursion that
lowers a nested call also lowers its arguments, so a heading queued in `lower` spans exactly the bytes that
form and everything inside it emit. The margin moves with the nesting, so an argument's code steps in and
the call's steps back out and the shape of the expression is visible without reading a word.

**Atoms are not annotated**, and this is the single detail the acceptance test turns on. A literal, a local
or a global would steal its parent's heading: `(* n 2)` lowers as a load, a load and an `imul`, and if the
two operands each queued a heading of their own then the line standing above the `imul` would name the
literal `2`. Skipping them leaves `(* n 2)` queued until the first byte and spanning the whole run. Read the
sample below: the line above the `imul` is the one that had to be right.

A form the checker invented carries `Loc.unknown` and is not annotated either — there is no source text to
quote, and it inherits the heading of the form that contains it, which is where it really came from. A form
at the same position as the last heading written is skipped, which is what stops a macro from printing its
call site once per form of its expansion. A function whose source is not readable from here — the prelude —
gets its frame map, no form headings, and one line saying so, rather than a column of bare positions.

## The frame map

This is the half that matters most and the one no amount of disassembly recovers. LLVM's output names its
values; this backend's cannot, because every value it has is a bump-allocated frame temporary and a
temporary has no name. So `-0x20(%rbp)` is the whole vocabulary of the listing and the map is its key.

Everything in it is read out of state `emit_fn` already keeps — `f.slots` *is* what the prologue stores
through, `fn.snames` is what the source called each slot — so it cannot drift from the code it describes.
The boundary between named slots and temporaries is captured as `fixed` right after the fixed allocations
and before the body is lowered; `maxframe` would be the wrong number, because that is the high-water mark of
the temporaries rather than where they start.

What is deliberately *not* described is any individual temporary. `scoped` reclaims them and a later form
reuses the bytes, so naming an offset that holds something else half the time is worse than saying where the
region begins — the same call the DWARF above it makes about locals, and for the same reason.

## The bookkeeping, named

Five things the compiler adds that no form asked for, each named where it appears and explained once in the
file legend rather than at every site: the transfer guard after every call to Flan code, the bounds check and
its signalling slow path, the arithmetic guard and the float-to-integer range check, `rep movsb` for every
aggregate copy, and the indirection cell a `--dev` build calls through. The prologue, the epilogue, the
transfer exit and C's `main` carry a prose block each.

## A worked sample

`spike/x86/dump.sh small.flan twice`, on

```
(defn twice [n i64] i64
  (* n 2))
```

LLVM at `-O2`:

```
flan.twice:
	movq	%rdi, -8(%rsp)
	leaq	(%rdi,%rdi), %rax
	retq
```

and the same function out of this backend:

```
# ────────────────────────────────────────────────────────────────────
# "flan.twice"   (defn twice [n i64] i64   small.flan:1:7
#
# Arguments: n from rdi.
# Returns i64 in rax.
# The transfer channel arrives last of all, from rsi. It is a pointer to the
# cell a callee writes its target into, and reading it is what every guard
# below does.
#
# The frame is 0x30 bytes below rbp. No call in it passes an argument on the
# stack.
#
#   -0x8     n              i64        parameter 1, from rdi
#   -0x10    <chan>         ptr        the transfer channel this frame passes on
#   -0x18    <ret>          i64        the return value the epilogue loads
# Everything below -0x18 is a temporary. They are bump-allocated and
# reclaimed at the end of the form that made them, so a later form reuses
# the bytes and no one offset down there means one thing for long.
# ────────────────────────────────────────────────────────────────────
	.globl	"flan.twice"
	.type	"flan.twice", @function
"flan.twice":
	# The prologue: save rbp, take the frame in one sub, and spill every incoming
	# register into its slot. rsp is written here and by leave and nowhere else,
	# so rsp % 16 == 0 at every call site below is a property of that one rounded
	# sub rather than an invariant each case has to keep.
	.byte 0x55,0x48,0x89,0xe5,0x48,0x81,0xec,0x30,...
	# (* n 2)                                                    small.flan:2:3
	.byte 0x48,0x8b,0x85,0xf8,0xff,0xff,0xff,0x48,0x89,0x85,0xe0,...
	# The epilogue, and every return and every transfer out of this frame arrives
	# here, so the frame is torn down once.
.Lret1048:
	.byte 0x48,0x8b,0x85,0xe8,0xff,0xff,0xff,0xc9,0xc3
	.size	"flan.twice", . - "flan.twice"
```

Three LLVM instructions against thirteen, and the listing now says where the difference went: a real frame
rather than a red zone, because rsp is written twice per function and never at a call site; a transfer
channel parameter that LLVM's `-O2` dropped as dead and this backend spills because a dev build does not
optimise; and both operands of the multiply through frame temporaries at `-0x20` and `-0x28`, because every
intermediate in this backend is a frame temporary. None of that is visible in the disassembly and all of it
is visible here.

A denser one, from `bounds.flan` — the four instructions after every call, and the bounds triple:

```
	        # (at args 1)                                        bounds.flan:11:35
	        .byte 0x48,0xb8,0x01,...
	        # The bounds check. One unsigned compare catches a negative index as
	        # well as an oversized one, and the not-taken branch is the whole of
	        # the fast path.
	        .byte 0x48,0x63,0x85,0xa4,...,0x48,0x39,0xc8,0x0f,0x82
	        .long .Linb1078 - . - 4
	        # Out of bounds: the location string, the operands, and this frame's
	        # channel, then flan_bounds_error, which signals
	        .byte 0x48,0x8d,0x3d
	        .long .Lk1079 - . - 4
	        ...
	        call flan_bounds_error
	        # The transfer guard, after every call to Flan code: load this
	        # frame's channel, load through it, test, and branch if it is set —
	        # a callee that transferred left a target there and the value in rax
	        # means nothing.
	        .byte 0x4c,0x8b,0x9d,0xd0,...,0x0f,0x85
	        .long .Lxfer1077 - . - 4
	        # ud2, where emit.ml writes unreachable. Nothing answered the signal,
	        # so the runtime already died inside that call and nothing falls
	        # through to here.
	        .byte 0x0f,0x0b
```

## Verification

**Byte identity — `spike/x86/annot.sh`, 342 SAME / 0 DIFFER, measured on this code.** Every program in `test/programs` and in
`spike/x86`, emitted both ways, assembled both ways, and every section of the two objects compared byte for
byte — in the default build, in `--dev`, and in `--debug`. The `--debug` case is the sharp one: a
`.debug_line` row is an address expressed as a label, and annotation issues no labels and consumes no
`uniq`, precisely so that those cannot move. The 84 SKIPs are programs that do not compile at all — the
reject corpus and the generic-milestone ones — and are the same set the survey skips.

**`dune test --root .` — 232 checks, 0 failures**, run against this code. `dune build @page` and
`dune build @cells` both exit 0; `@cells` reports `x86 --dev: 22 22` and `x86 : 42 42`, which is the check
that the indirection cells and the ABI marker still come out where they were.

**`spike/x86/survey.sh` has NOT been run on this work.** Two runs were started and both were invalidated by
racing with a `dune build` that replaced `bin/main.exe` underneath them; a third was killed on instruction
before it finished. **Whoever picks this up must not assume the 103 MATCH / 0 DIFFER / 0 REFUSED baseline
still holds — it is the one check that matters and it is outstanding.** Run it detached and with the
compiler pinned, so it cannot race a rebuild:

```
FLAN=_build/default/bin/main.exe setsid timeout 2400 spike/x86/survey.sh > log 2>&1 </dev/null
```

What makes an unwelcome result unlikely rather than impossible: the survey builds through `flan build --x86`,
which passes `annotate = false`, so the assembly it compiles is produced by the same code path as before with
one field added to a record and two branches that are never taken. The edits that *could* reach it are the
ones to `buf` — `flush` now writes `b.ind`, which is `""` in a build, and `u8`/`dir`/`text`/`lbl` now test
`b.ann`, which is empty in a build. `annot.sh` proves those are inert for the object at 342 SAME across the
corpus in all three of default, `--dev` and `--debug`, but it proves it about the *assembly*, and only the
survey proves it about the programs.

## What is not here

- No mnemonics, for the reason above.
- No annotation of an individual temporary, for the reason above.
- The redefinition emitter (`X86.redefinition`) builds its install stub with annotation off. It is generated
  code with no Flan form behind it and a frame map of two slots; the legend would be longer than the
  function. If the reload path ever wants a readable listing, the `ann` field on its context is the one
  line to change.