flan/HANDOFF-x86-debug.md

Handoff — dropping the registry notes, and debug information for --x86

Branch dev-loop, worktree agent-a123a91f32f4c9779. Started from 957ba07 and rebased onto eacf7c4 when the arithmetic-condition lane landed; the rebase was clean, and everything below was re-measured after it. Items 2 and 6 of HANDOFF-x86-rt.md §6. Both landed.

The finding that changes the brief, read this first

.loc does not work in this backend's output, and no amount of care makes it work. The brief called line tables "by far the least work, since as will build .debug_line for you from .loc directives". That is true of a backend that emits mnemonics. This one emits .byte blobs — the header says so and says why — and GAS builds its line table from dwarf2_emit_insn, which runs only out of md_assemble. No instruction is ever assembled, so nothing flushes the pending .loc.

What GAS does instead is flush the pending .loc when it sees the next .loc, at whatever the location counter has reached by then. Measured on as 2.44:

	.file	1 "foo.flan"
main:	.loc 1 10 1
	.byte	0x55
	.byte	0x48,0x89,0xe5
	.loc	1 11 1
	.byte	0xb8,0x07,0x00,0x00,0x00
	.loc	1 12 1
	.byte	0xc9
	.byte	0xc3

foo.flan     10      0x4     x      <- should be 0x0
foo.flan     11      0x9     x      <- should be 0x4
foo.flan      -      0xb            <- line 12 has no row at all

Every row is one statement late and the last statement of every function gets none. Interposing labels between the .loc and the .byte does not help; it was tested. Do not try to correct this by shifting the .loc directives back by one — the last-statement hole is a correctness gap, not just fragility, and the behaviour being exploited is undocumented.

So .debug_line is written out here by hand, as .byte / .uleb128 / .quad in a .debug_line section, which is the same thing this backend already does for instructions.

.cfi is the opposite finding and it is good news. GAS's CFI machinery computes its advances from frag positions and not from md_assemble, so .cfi_def_cfa_offset and friends interleaved with .byte come out exactly right — measured, readelf --debug-dump=frames on a .byte-only function gives the correct advances. So this lane emits them, in a --debug build — see below.

What landed

Item 2 — a release build stops calling a registry that is switched off

lib/x86.ml, prim's Tast.Rt dispatch, one arm above the general one, mirroring lib/emit.ml:1918 test for test including the strict > 17: bare flan_dev_reg_note is the runtime's own C entry point and is never a Tast.Rt; what check.ml builds is the _vec, _map and _pool wrappers. The node's type is Unit, so the body is () and dst is untouched.

emit.ml is careful to drop the note before the arguments are walked, because emitting the address of the container only to discard the call would leave an escaped alloca that mem2reg refuses. Here the arguments are not touched until call_rt, so the guard is already early enough.

Measured by disassembly, objdump -d | grep -c 'call.*flan_dev_reg_note':

program	--x86 release	--x86 --dev	LLVM release
vec.flan	46 → 1	46	1
maps.flan	55 → 1	55	—
registry.flan	36 → 1	36	—

The one that remains in every release build is not emitted code: it is inside the runtime's own flan_dev_reg_note_vec in flan_rt.c. LLVM's release build has exactly the same one, so the two backends agree.

Item 6 — --x86 --debug works, at parity with LLVM bar locals

lib/build.ml's refusal list no longer names --debug; only --sanitize and wasm are left there. X86.program takes ~debug the way it already took ~dev.

In lib/x86.ml:

piece	what
the Debug information section, above fnctx	dwrow / dwsub / dwarf, and dwfile. Carries the header comment explaining the .loc finding above
fnctx.dw : dwarf option	None in every build but a --debug one, which is what makes the release path structurally unchanged
dwline, beside new_label	the one hook. A label and a row, deduplicated on (file, line, column) and on the byte counter — see below
lower	calls dwline f e.Tast.loc and nothing else
emit_fn ?dw	opens the subprogram, seeds it with a row at the function symbol on the defn's line, closes it with a label one past the last byte
emit_dwarf	.debug_abbrev (two abbreviations), .debug_info (one CU, one DW_TAG_subprogram per function), .debug_line (header, directory table, file table, program)
program ~debug	a numbered .file at the top, .Ldwtext / .Ldwtext_end around the text, and the sections at the end
cfi_after_push / cfi_after_mov / cfi_after_leave, above emit_fn	the whole frame model in five directives, at the three sites that share a prologue: emit_fn, emit_main, emit_globals_init

Two decisions worth knowing about because they are not obvious:

The line program is written in the dumb form. Every row is a full DW_LNE_set_address on a label, then set_file / set_column / advance_line as needed, then DW_LNS_copy — eleven bytes and up per row, where a special opcode would be one. The alternative is DW_LNS_advance_pc with a .uleb128 of a difference of two labels, which asks the assembler to resolve a value whose size changes the offsets after it. That does work, and its failure would look exactly like a DWARF bug. A debug build can afford the bytes.

The .file 1 "..." directive at the top is load-bearing and is not about our own table. Without it, clang's integrated assembler generates a compile unit of its own over the .s, whose rows land at the call mnemonics — the only real instructions this backend emits — and those addresses are inside functions our unit already describes. Two units then claim the same addresses. With the directive the assembler emits an empty line table and nothing else. -gdwarf-4 is added alongside it in build.ml so that empty stub is a version 4 header; at the default version it is a DWARF 5 header whose file table readelf calls corrupt, and a warning on a cross-check is a warning you learn to ignore.

Rows are deduplicated on the byte counter as well as the position. lower recurses, so an outer form and the inner one that emits its first byte both ask for a row at the same address. Keeping only the first — the outer, enclosing form — is both smaller and the better answer, since a debugger resolving a run of rows at one address takes the last. buf.n already existed and was written but never read; this is its first reader.

Verification — a real debugger, on the corpus's own debug fixture

test/programs/debug.flan built --x86 --debug, under gdb 16.x, breaking by file and line:

$ flan build test/programs/debug.flan --x86 --debug -o dbg
$ gdb -batch -x gdb5.cmd ./dbg
Breakpoint 1 at 0x400689: file debug.flan, line 19.

Breakpoint 1, flan.tick () at debug.flan:19
19	    (set (.heat c) (+ (.heat c) 1.5))
#0  flan.tick () at debug.flan:19
#1  0x00000000004007b5 in flan.main () at debug.flan:25
#2  0x0000000000400bea in main ()
Line 19 of "debug.flan" starts at address 0x400689 <flan.tick+97> and ends at 0x4006af <flan.tick+135>.
Current source file is debug.flan
Compilation directory is .../test/programs
Located in .../test/programs/debug.flan
Contains 30 lines.
Source language is c.
Producer is flan (x86-64).
Compiled with DWARF 4 debugging format.

The same script against an LLVM --debug build of the same program, which is the parity this is measured against:

Breakpoint 1 at 0x400644: file debug.flan, line 19.

Breakpoint 1, flan.tick (c=0x7fffffffd8f0, n=41) at debug.flan:19
19	    (set (.heat c) (+ (.heat c) 1.5))
#0  flan.tick (c=0x7fffffffd8f0, n=41) at debug.flan:19
#1  0x00000000004006cf in flan.main () at debug.flan:25
#2  0x0000000000400806 in main ()
Producer is flan.
Compiled with DWARF 5 debugging format.

Identical bar c=0x7fffffffd8f0, n=41 — which is the locals work, and is the honest summary of what is missing.

break tick fails on both backends, with Function "tick" not defined. That is not a gap this lane opened: emit.ml writes name: "tick", linkageName: "flan.tick", gdb takes the linkage name as the search name because flan.tick is not a mangled C++ name, and an LLVM --debug build behaves the same way. The project's own source-level debugging case runs under lldb, which does not have this problem. break debug.flan:19 and break flan.tick both work everywhere. Matching emit.ml was chosen over diverging from it; if this is worth fixing it should be fixed in both places at once.

lldb, which is the debugger this project's own source-level case runs under, does better still — it resolves flan.tick by name, shows the source inline and gives a column:

$ lldb -b -o "breakpoint set --name flan.tick" -o run -o bt ./dbg
* thread #1, stop reason = breakpoint 1.1
    frame #0: 0x0000000000400647 dbg`flan.tick at debug.flan:18:3
   17  	(defn tick [c (Ptr Cell)  n i32] i32
-> 18  	  (let [bump (+ n 1)]
    	  ^
   19  	    (set (.heat c) (+ (.heat c) 1.5))
(lldb) bt
  * frame #0: 0x0000000000400647 dbg`flan.tick at debug.flan:18:3
    frame #1: 0x00000000004007b5 dbg`flan.main at debug.flan:25:28
    frame #2: 0x0000000000400bea dbg`main + 41
    frame #3: 0x00007ffff7cb9575 libc.so.6`__libc_start_call_main + 117

That transcript is now a test rather than a transcript: test/test_acceptance.ml's lldb block has an --x86 arm beside its --dev one, asserting the breakpoint resolves and both Flan frames name debug.flan, and that the program still prints what it printed. It claims no frame variable, which is the gap.

Unwinding out of the C runtime works, which is the case that matters for this project's error paths and which the brief did not ask for but should have. bounds.flan built --x86 --debug, breaking on the C symbol:

Breakpoint 1, flan_bounds_error (loc=0x40a880 "test/programs/bounds.flan:11:44", ...) at flan_rt.c:534
#0  flan_bounds_error (...) at flan_rt.c:534
#1  0x00000000004006be in flan.main () at bounds.flan:11
#2  0x000000000040162e in main ()

The compile unit's range is exact. DW_AT_low_pc 0x400628 is flan.tick, the first function; DW_AT_high_pc 0x5cf puts the end at 0x400bf7, and nm -S says main — the last thing this object puts in .text — ends at 0x400bc1 + 0x36 = 0x400bf7. So .Ldwtext / .Ldwtext_end bracket the four functions this unit emitted and nothing else.

--x86 --dev --debug together, which is a reachable combination and is exercised by nothing else, builds, runs, and gives a line table and an .eh_frame that readelf reads with no warnings — 280 FDEs, ours among the runtime's. That is the one combination where the cells' .data, the .eh_frame and the debug sections all have to be ordered against each other.

Cross-checked on a spread of the corpus — debug, generics, vec, maps, strings, conditions, bounds-condition, algorithms, handles, registry, destructure, edn, twelve in all — every one builds with --x86 --debug, runs, and produces .debug_line that readelf --debug-dump=decodedline reads with zero warnings. edn and generics are the large ones, at 40 and 36 subprograms. generics is the multi-file case: its directory table has one entry and its file table two, generics.flan and <prelude>. <prelude> has no path on disk and a debugger simply finds no source for it, which is the truth.

Baseline, measured on this tree

	before (957ba07)	after, on 957ba07	after, rebased onto eacf7c4
spike/x86/survey.sh	99 MATCH / 0 DIFFER / 0 REFUSED / 0 NOX86, skips 28 + 6 + 2	99 / 0 / 0 / 0, skips 28 + 6 + 2	101 / 0 / 0 / 0, skips 28 + 6 + 2
spike/x86/cells.sh	4/4 ok	4/4 ok	4/4 ok
dune test --root .	exit 0	exit 0	exit 0, run without a pipe; acceptance reports 232 checks, 0 failures

The MATCH count went from 99 to 101 across the rebase and neither is this lane's doing: the arithmetic-condition lane added two probes to spike/x86. The number that matters is that DIFFER, REFUSED and NOX86 are all zero on both sides and the skip breakdown did not move.

The survey is protected structurally as well as measured: every piece of this lane's work on the release path is behind debug, so --x86 without --debug emits the assembly it emitted before — fnctx.dw is None, dwline does nothing, the .Ldwtext label is not written and neither are the .cfi directives.

Run dune test --root . without a pipe — HANDOFF-x86-redef.md is emphatic and right: piping gives you tail's exit status, and the dev-robust fixture puts ld and clang failure text in the output either way while the run still exits 0.

What was not reached, and what the next lane should do with it

1. Locals and types. Not attempted, and the reason is structural rather than a lack of time. emit.ml writes a !DILocalVariable per slot because every slot there is an alloca that llvm.dbg.declare points at and LLVM computes the frame offset. Here a slot is a bump-allocated frame temporary: alloc knows its rbp-relative offset, but scoped reclaims temporaries and a later expression reuses the bytes, so the lifetime is not modelled. A DW_TAG_variable with a DW_OP_fbreg would be right at some addresses and confidently wrong at others. Note that the named slots — fn.Tast.snames — are the slots and not the temporaries, and those do live for the whole function, so a narrower version of this is reachable: DW_TAG_formal_parameter and DW_TAG_variable for snames-named slots only, with DW_AT_frame_base as DW_OP_call_frame_cfa or DW_OP_reg6. That needs a type-DIE emitter, which is the part emit.ml spends most of its debug lines on.

2. .cfi is emitted, but only in a --debug build. Five directives at three sites — emit_fn, emit_main, emit_globals_init, which share a prologue. readelf --debug-dump=frames gives the exact advances. emit_main gets no closing rule because it has no epilogue: it leaves through flan_exit and the ud2 after that is unreachable, so the rbp rule holds to the last byte.

   gdb did not need it — its prologue analyser already unwound out of flan_bounds_error, because push rbp; mov rbp,rsp; sub rsp,N is the pattern it recognises — so this is a description stated rather than guessed, plus a correct unwind from the very first byte of a function, before the push.

   The open decision is whether to make it unconditional. The description is correct in every build and a release build is where a crash would most want it. What stopped it here is only that nothing measures the .eh_frame it would add, and another lane is measuring backend cost right now. One if in three places.

3. X86.redefinition emits no debug information. It is passed no dwarf, so a redefinition module's bodies have no lines. Nothing reaches that yet — flan dev does not build --x86 at all (HANDOFF-x86-redef.md §"What remains" item 3) — but when it does, a break loop stopping in a reloaded body will show raw addresses.

4. emit_main and flan..init-globals have no rows, and this is a gap rather than a wrong answer. Both sit inside the compile unit's low_pc / high_pc range and neither has a line-table sequence, so the worry is that a debugger picks a nearby row out of a neighbouring function's sequence and reports a confident wrong Flan line. It does not. Checked, breaking inside the initialiser of dev-globals.flan built --x86 --debug:

   Breakpoint 1, 0x0000000000400cec in flan..init-globals ()
   #0  0x0000000000400cec in flan..init-globals ()
   #1  0x00007ffff7cb96a4 in __libc_start_main_impl () from /lib64/libc.so.6
   #2  0x00000000004006c5 in _start ()
   No line number information available.
   

   The frame is named from the ELF symbol, the line is honestly absent, and the unwind out to libc is the .cfi working. So this is worth doing and is not urgent: emit_globals_init does lower expressions that carry locations, and giving it a dwsub the way emit_fn has one is the whole of it — it already takes ~cfi, so ?dw goes in beside it.

5. debug-permuted.flan has no --x86 arm. The new acceptance case covers debug.flan only. The permuted fixture exists to catch a member offset that does not follow the declaration, and that is a types claim — there are no type DIEs here to get wrong, so it would test nothing today. It becomes the right test the moment item 1 above is attempted, and it is already written.

Open questions for the author

• break tick naming. Is the name / linkageName split worth keeping, given it costs break <fn> in gdb on both backends? lldb is fine with it, and the project's debugging case is an lldb one, so this may be deliberate. It was left alone rather than diverged from.
• Should --x86 --debug imply --dev? It does not today, and they are independent axes, which seems right — but a debug build you cannot redefine into is half of what the dev loop wants.