diff --git a/HANDOFF-x86-debug.md b/HANDOFF-x86-debug.md index c88e42e..f511d04 100644 --- a/HANDOFF-x86-debug.md +++ b/HANDOFF-x86-debug.md @@ -1,14 +1,14 @@ # Handoff — dropping the registry notes, and debug information for `--x86` Branch `dev-loop`, worktree `agent-a123a91f32f4c9779`, from `957ba07`. Items **2** and **6** of -`HANDOFF-x86-rt.md` §6. +`HANDOFF-x86-rt.md` §6. Both landed. ## The finding that changes the brief, read this first **`.loc` does not work in this backend's output, and no amount of care makes it work.** The brief called line tables "by far the least work, since `as` will build `.debug_line` for you from `.loc` directives". That is true of a backend that emits mnemonics. This one emits `.byte` blobs — the header says so and says why — and -GAS builds its line table from `dwarf2_emit_insn`, which only runs out of `md_assemble`. No instruction is +GAS builds its line table from `dwarf2_emit_insn`, which runs only out of `md_assemble`. No instruction is ever assembled, so nothing flushes the pending `.loc`. What GAS does instead is flush the pending `.loc` when it sees the *next* `.loc`, at whatever the location @@ -32,32 +32,27 @@ foo.flan 11 0x9 x <- should be 0x4 foo.flan - 0xb <- line 12 has no row at all ``` -Every row is one statement late and the last statement in the function gets none. Interposing labels between -the `.loc` and the `.byte` does not help; it was tested. Do **not** try to correct this by shifting the `.loc` -directives back by one statement — the last-statement hole is a correctness gap, not just fragility, and the +Every row is one statement late and the last statement of every function gets none. Interposing labels +between the `.loc` and the `.byte` does not help; it was tested. Do **not** try to correct this by shifting +the `.loc` directives back by one — the last-statement hole is a correctness gap, not just fragility, and the behaviour being exploited is undocumented. -So the line table is written out by hand here, as raw `.byte` / `.uleb128` / `.quad` in `.debug_line`, which is -the same thing this backend already does for instructions. Scope grew accordingly; see the plan below. +So `.debug_line` is written out here by hand, as `.byte` / `.uleb128` / `.quad` in a `.debug_line` section, +which is the same thing this backend already does for instructions. -## Plan, in the order it is being done +**`.cfi` is the opposite finding and it is good news.** GAS's CFI machinery computes its advances from frag +positions and not from `md_assemble`, so `.cfi_def_cfa_offset` and friends interleaved with `.byte` come out +*exactly* right — measured, `readelf --debug-dump=frames` on a `.byte`-only function gives the correct +advances. Nothing here emits them yet (see "What was not reached"), but the next lane should know it can. -1. **Item 2 — `flan_dev_reg_note` dropped in a release build.** Done; see below. -2. A hand-written `.debug_line`, plus the minimal `.debug_info` / `.debug_abbrev` CU that makes a debugger - *find* it. These are one step and not two: `readelf` will read a bare `.debug_line`, but gdb reaches a line - table only by walking `.debug_info` and following `DW_AT_stmt_list`. -3. `DW_TAG_subprogram` DIEs, so `break` and `list` scope by function. Note that gdb already names frames from - the ELF symbol table, which this backend emits via `.globl`, so this buys less than it looks like. -4. `.cfi` — only if it can be placed correctly against `.byte` output. If it cannot, that is said plainly and - gdb's prologue analyser is left to recognise `push rbp; mov rbp,rsp; sub rsp,N`, which is canonical. -5. Locals and types, only if everything above is solid and committed. +## What landed -## Item 2 — done +### Item 2 — a release build stops calling a registry that is switched off -`lib/x86.ml`, `prim`'s `Tast.Rt` dispatch, one arm above the general one, mirroring `lib/emit.ml:1918` test for -test including the strict `> 17`: bare `flan_dev_reg_note` is the runtime's own C entry point and is never a -`Tast.Rt`; what `check.ml` builds is the `_vec`, `_map` and `_pool` wrappers. The node's type is `Unit`, so the -body is `()` and `dst` is untouched. +`lib/x86.ml`, `prim`'s `Tast.Rt` dispatch, one arm above the general one, mirroring `lib/emit.ml:1918` test +for test including the strict `> 17`: bare `flan_dev_reg_note` is the runtime's own C entry point and is never +a `Tast.Rt`; what `check.ml` builds is the `_vec`, `_map` and `_pool` wrappers. The node's type is `Unit`, so +the body is `()` and `dst` is untouched. `emit.ml` is careful to drop the note *before* the arguments are walked, because emitting the address of the container only to discard the call would leave an escaped alloca that mem2reg refuses. Here the arguments are @@ -72,13 +67,150 @@ Measured by disassembly, `objdump -d | grep -c 'call.*flan_dev_reg_note'`: | `registry.flan` | 36 → **1** | 36 | — | The one that remains in every release build is not emitted code: it is inside the runtime's own -`flan_dev_reg_note_vec` in `flan_rt.c`, which calls `flan_dev_reg_note`. LLVM's release build has exactly the -same one, so the two backends now agree. +`flan_dev_reg_note_vec` in `flan_rt.c`. LLVM's release build has exactly the same one, so the two backends +agree. -## Baseline +### Item 6 — `--x86 --debug` works, at parity with LLVM bar locals -Measured on this tree, not recalled. (Filled in below as it is measured.) +`lib/build.ml`'s refusal list no longer names `--debug`; only `--sanitize` and wasm are left there. +`X86.program` takes `~debug` the way it already took `~dev`. -## Open questions +In `lib/x86.ml`: -(none yet) +| piece | what | +|---|---| +| the `Debug information` section, above `fnctx` | `dwrow` / `dwsub` / `dwarf`, and `dwfile`. Carries the header comment explaining the `.loc` finding above | +| `fnctx.dw : dwarf option` | `None` in every build but a `--debug` one, which is what makes the release path structurally unchanged | +| `dwline`, beside `new_label` | the one hook. A label and a row, deduplicated on (file, line, column) *and* on the byte counter — see below | +| `lower` | calls `dwline f e.Tast.loc` and nothing else | +| `emit_fn ?dw` | opens the subprogram, seeds it with a row at the function symbol on the `defn`'s line, closes it with a label one past the last byte | +| `emit_dwarf` | `.debug_abbrev` (two abbreviations), `.debug_info` (one CU, one `DW_TAG_subprogram` per function), `.debug_line` (header, directory table, file table, program) | +| `program ~debug` | a numbered `.file` at the top, `.Ldwtext` / `.Ldwtext_end` around the text, and the sections at the end | + +Two decisions worth knowing about because they are not obvious: + +**The line program is written in the dumb form.** Every row is a full `DW_LNE_set_address` on a label, then +`set_file` / `set_column` / `advance_line` as needed, then `DW_LNS_copy` — eleven bytes and up per row, where +a special opcode would be one. The alternative is `DW_LNS_advance_pc` with a `.uleb128` of a difference of two +labels, which asks the assembler to resolve a value whose *size* changes the offsets after it. That does work, +and its failure would look exactly like a DWARF bug. A debug build can afford the bytes. + +**The `.file 1 "..."` directive at the top is load-bearing and is not about our own table.** Without it, +clang's integrated assembler generates a compile unit of *its own* over the `.s`, whose rows land at the +`call` mnemonics — the only real instructions this backend emits — and those addresses are inside functions +our unit already describes. Two units then claim the same addresses. With the directive the assembler emits an +empty line table and nothing else. `-gdwarf-4` is added alongside it in `build.ml` so that empty stub is a +version 4 header; at the default version it is a DWARF 5 header whose file table `readelf` calls corrupt, and +a warning on a cross-check is a warning you learn to ignore. + +**Rows are deduplicated on the byte counter as well as the position.** `lower` recurses, so an outer form and +the inner one that emits its first byte both ask for a row at the same address. Keeping only the first — the +outer, enclosing form — is both smaller and the better answer, since a debugger resolving a run of rows at one +address takes the last. `buf.n` already existed and was written but never read; this is its first reader. + +## Verification — a real debugger, on the corpus's own debug fixture + +`test/programs/debug.flan` built `--x86 --debug`, under `gdb` 16.x, breaking by file and line: + +``` +$ flan build test/programs/debug.flan --x86 --debug -o dbg +$ gdb -batch -x gdb5.cmd ./dbg +Breakpoint 1 at 0x400689: file debug.flan, line 19. + +Breakpoint 1, flan.tick () at debug.flan:19 +19 (set (.heat c) (+ (.heat c) 1.5)) +#0 flan.tick () at debug.flan:19 +#1 0x00000000004007b5 in flan.main () at debug.flan:25 +#2 0x0000000000400bea in main () +Line 19 of "debug.flan" starts at address 0x400689 and ends at 0x4006af . +Current source file is debug.flan +Compilation directory is .../test/programs +Located in .../test/programs/debug.flan +Contains 30 lines. +Source language is c. +Producer is flan (x86-64). +Compiled with DWARF 4 debugging format. +``` + +The same script against an **LLVM** `--debug` build of the same program, which is the parity this is measured +against: + +``` +Breakpoint 1 at 0x400644: file debug.flan, line 19. + +Breakpoint 1, flan.tick (c=0x7fffffffd8f0, n=41) at debug.flan:19 +19 (set (.heat c) (+ (.heat c) 1.5)) +#0 flan.tick (c=0x7fffffffd8f0, n=41) at debug.flan:19 +#1 0x00000000004006cf in flan.main () at debug.flan:25 +#2 0x0000000000400806 in main () +Producer is flan. +Compiled with DWARF 5 debugging format. +``` + +Identical bar `c=0x7fffffffd8f0, n=41` — which is the locals work, and is the honest summary of what is +missing. + +**`break tick` fails on both backends**, with `Function "tick" not defined`. That is not a gap this lane +opened: `emit.ml` writes `name: "tick", linkageName: "flan.tick"`, gdb takes the linkage name as the search +name because `flan.tick` is not a mangled C++ name, and an LLVM `--debug` build behaves the same way. The +project's own source-level debugging case runs under **lldb**, which does not have this problem. `break +debug.flan:19` and `break flan.tick` both work everywhere. Matching `emit.ml` was chosen over diverging from +it; if this is worth fixing it should be fixed in both places at once. + +Cross-checked on a spread of the corpus — `debug`, `generics`, `vec`, `maps`, `strings`, `conditions`, +`bounds-condition`, `algorithms`, `handles`, `registry` — all build with `--x86 --debug`, run, and produce +`.debug_line` that `readelf --debug-dump=decodedline` reads with **zero warnings**. `generics` is the +multi-file case: its directory table has one entry and its file table two, `generics.flan` and ``. +`` has no path on disk and a debugger simply finds no source for it, which is the truth. + +## Baseline, measured on this tree + +| | before | after | +|---|---|---| +| `spike/x86/survey.sh` | 99 MATCH / 0 DIFFER / 0 REFUSED / 0 NOX86, skips 28 + 6 + 2 | (see below) | +| `spike/x86/cells.sh` | 4/4 ok | (see below) | +| `dune test --root .` | exit 0 | (see below) | + +Run `dune test --root .` **without a pipe** — `HANDOFF-x86-redef.md` is emphatic and right: piping gives you +`tail`'s exit status, and the `dev-robust` fixture puts `ld` and `clang` failure text in the output either way +while the run still exits 0. + +## What was not reached, and what the next lane should do with it + +1. **Locals and types.** Not attempted, and the reason is structural rather than a lack of time. `emit.ml` + writes a `!DILocalVariable` per slot because every slot there is an `alloca` that `llvm.dbg.declare` + points at and LLVM computes the frame offset. Here a slot is a bump-allocated frame temporary: `alloc` + knows its `rbp`-relative offset, but `scoped` reclaims temporaries and a later expression reuses the bytes, + so the *lifetime* is not modelled. A `DW_TAG_variable` with a `DW_OP_fbreg` would be right at some + addresses and confidently wrong at others. Note that the named slots — `fn.Tast.snames` — are the *slots* + and not the temporaries, and those do live for the whole function, so a narrower version of this is + reachable: `DW_TAG_formal_parameter` and `DW_TAG_variable` for `snames`-named slots only, with + `DW_AT_frame_base` as `DW_OP_call_frame_cfa` or `DW_OP_reg6`. That needs a type-DIE emitter, which is the + part `emit.ml` spends most of its debug lines on. +2. **`.cfi`, and it is now known to be cheap.** Nothing here emits it, so gdb unwinds by its prologue + analyser — which works, because `push rbp; mov rbp,rsp; sub rsp,N` is the canonical pattern and this + backend's header guarantees `rsp` is written exactly twice. The backtraces above are that analyser + working. But the experiment at the top of this file says `.cfi` against `.byte` output is exact, and the + frame model makes the content textbook: CFA is `rsp+16` after the `push`, `rbp`-based after the `mov`, and + unchanged for the whole body. It would want `.cfi_startproc` / `.cfi_endproc` in `emit_fn` and three + directives in the prologue. Worth doing; it is what makes a backtrace survive an unwind from inside the + runtime's C. +3. **`X86.redefinition` emits no debug information.** It is passed no `dwarf`, so a redefinition module's + bodies have no lines. Nothing reaches that yet — `flan dev` does not build `--x86` at all + (`HANDOFF-x86-redef.md` §"What remains" item 3) — but when it does, a break loop stopping in a reloaded + body will show raw addresses. +4. **`emit_main` and `flan..init-globals` have no rows.** They are inside the compile unit's `low_pc` / + `high_pc` range, which is honest — a debugger finds no line for an address in them and says so — but a + backtrace through the globals' initialiser names nothing. `emit_globals_init` does lower expressions that + carry locations, so this is a small piece of work: give it a `dwsub` the way `emit_fn` has one. +5. **Nothing tests this in `dune test`.** The corpus's source-level debugging case (`test/programs/debug.flan` + and `debug-permuted.flan`) runs under lldb against the LLVM backend. An `--x86` arm of it would be the + right regression test and does not exist; the verification above is a transcript in a handoff, which rots. + +## Open questions for the author + +- **`break tick` naming.** Is the `name` / `linkageName` split worth keeping, given it costs `break ` in + gdb on *both* backends? lldb is fine with it, and the project's debugging case is an lldb one, so this may + be deliberate. It was left alone rather than diverged from. +- **Should `--x86 --debug` imply `--dev`?** It does not today, and they are independent axes, which seems + right — but a debug build you cannot redefine into is half of what the dev loop wants. diff --git a/lib/build.ml b/lib/build.ml index e4f62ac..e863805 100644 --- a/lib/build.ml +++ b/lib/build.ml @@ -756,11 +756,20 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = []) [flan reload] and [flan dev] both build host and module through LLVM — and the fix when something does is to emit the module through this backend too, not to grow a classifier. *) - if opts.x86 && (wasm_target opts || opts.debug || opts.sanitize) + (* [--debug] used to be in this list too. It is not any more: [x86.ml] emits + a compile unit, a subprogram per function and a line table, all written + out as bytes because [.loc] cannot work against a file whose instructions + are [.byte] blobs -- see that file's own debug-information section. What + it does *not* emit is locals and types, for the reason given there: a + slot is a bump-allocated frame temporary whose lifetime this backend does + not model, so there is nothing honest for a [DW_TAG_variable] to point + at. So `--x86 --debug` gives a backtrace that names Flan files, functions + and lines, and `print x` says the name is not in the current context. *) + if opts.x86 && (wasm_target opts || opts.sanitize) then failwith - "--x86 is the native dev backend on its own: it emits no DWARF and \ - there is no sanitizer pass over hand-written assembly"; + "--x86 is the native dev backend on its own: there is no sanitizer pass \ + over hand-written assembly"; let dir = workdir () in (* The one fork in this function. The x86 backend hands clang an assembly file where LLVM hands it IR text; clang takes either on its command line, @@ -771,7 +780,8 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = []) (Filename.basename out ^ if opts.x86 then ".s" else ".ll") in write ll - (if opts.x86 then X86.program ~checks:opts.checks ~dev:opts.dev p + (if opts.x86 then + X86.program ~checks:opts.checks ~dev:opts.dev ~debug:opts.debug p else Emit.program ~checks:opts.checks ~dev:opts.dev ~debug:opts.debug ~pnames ~sanitize:opts.sanitize p); @@ -820,6 +830,16 @@ let executable ?(opts = default) ?(csrcs = []) ?(lflags = []) ?(pnames = []) the .ll, which is the only place the Flan half of the program gets instrumented at all. *) @ cflags opts + (* The .s carries a hand-written DWARF 4 compile unit. The numbered + [.file] directive in it stops clang's integrated assembler from + generating one of its own, but the directive still leaves an empty + line table behind, and at the default version that stub is a DWARF 5 + header whose file table readelf reports as corrupt. Asking for 4 + makes the stub parse, so a readelf cross-check of the real table is + not read past a warning. It reaches only this command, which + assembles the .s and links; the C objects were compiled by + [compile_c] and are already done. *) + @ (if opts.x86 && opts.debug then [ "-gdwarf-4" ] else []) @ (if opts.dev then [ "-rdynamic" ] else []) @ tflags @ (if web_target opts then web_link_flags ~out else []) diff --git a/lib/x86.ml b/lib/x86.ml index 43a8eb2..61fc65c 100644 --- a/lib/x86.ml +++ b/lib/x86.ml @@ -408,6 +408,90 @@ let gsym n = asm_sym ("flan." ^ n) Byte-for-byte or the link fails and the piece served nothing. *) let csym n = asm_sym ("flan.cell." ^ n) +(* ── Debug information ───────────────────────────────────────────────── *) + +(* DWARF, written out as bytes, for the same reason the instructions are — and + the reason is worth stating first because it is the one thing about this + backend that makes debug information cost more here than it does anywhere + else. + + {b [.loc] does not work against an assembly file that has no instructions + in it.} GAS builds its line table from [dwarf2_emit_insn], which runs only + when an instruction is assembled, and this file assembles none: everything + is a [.byte] blob. A pending [.loc] therefore sits until the *next* [.loc] + and is flushed at whatever the location counter has reached by then, so + every row comes out one statement late and the last statement of every + function gets no row at all. Measured on GAS 2.44 and reproduced with + labels interposed, which do not help. So [.debug_line] is emitted here as + data, which is in any case the only spelling consistent with the rest of + the file. + + DWARF 4 rather than 5. Version 4's file and directory tables are + NUL-terminated strings and a terminator byte where 5 form-codes them, and + nothing above needs a version 5 feature. A compile unit declares its own + version, so a v4 unit sitting beside the v5 ones clang gives the runtime's + C is not a conflict — each is read on its own terms. + + What is described is deliberately shallow: the compile unit, one subprogram + per function, and the line table. {b No locals and no types.} [emit.ml] + writes a [!DILocalVariable] per slot because every slot there is an + [alloca] that [llvm.dbg.declare] can point at and LLVM computes the frame + offset; here a slot is a bump-allocated frame temporary whose offset this + file knows but whose *lifetime* it does not model — [scoped] reclaims + temporaries and a later expression reuses the bytes. Naming an offset that + holds something else half the time is worse than naming nothing, so this + emits nothing rather than a confident wrong answer. That is the same call + [build.ml] makes about wasm32's member offsets. *) + +(* One row of the line table: the label whose address it is, and the position + it names. Addresses are labels rather than numbers because the assembler + places the function and this file does not. *) +type dwrow = { rlbl : string; rfile : int; rline : int; rcol : int } + +type dwsub = { + sname : string; (* what a debugger calls the frame *) + ssym : string; (* the symbol the linker sees *) + sfile : int; + sline : int; + send : string; (* a label one past the function's last byte *) + mutable srows : dwrow list; (* newest first *) +} + +type dwarf = { + dfiles : (string, int) Hashtbl.t; + mutable dpaths : string list; (* newest first *) + mutable dsubs : dwsub list; (* newest first *) + mutable dcur : dwsub option; (* the function being lowered *) + mutable dlast : (int * int * int) option; (* the last row's position *) + (* [buf.n] as it stood when the last row was made. [lower] recurses, so an + outer form and the inner one that emits the first byte of it both ask for + a row at the same address; without this the table carries a run of rows + that a debugger resolves by taking the last, which is the innermost form + rather than the statement. Keeping the first is both smaller and the + better answer. Reset to -1 per function, because the body's byte counter + starts again at 0 and the entry row is at the prologue's 0. *) + mutable dlastn : int; +} + +let new_dwarf () = + { dfiles = Hashtbl.create 8; dpaths = []; dsubs = []; dcur = None; + dlast = None; dlastn = -1 } + +(* File indices are 1-based and handed out in first-seen order, which is the + order the file table is written in below. A program spans more than one + file whenever the prelude or a macro contributed a form, and the checker's + own invented nodes carry [Loc.unknown], whose file is [] — that + one never reaches here, because a row with line 0 is attributed to the + enclosing function's file instead. *) +let dwfile dw path = + match Hashtbl.find_opt dw.dfiles path with + | Some n -> n + | None -> + let n = List.length dw.dpaths + 1 in + Hashtbl.replace dw.dfiles path n; + dw.dpaths <- path :: dw.dpaths; + n + (* ── Function context ────────────────────────────────────────────────── *) type fnctx = { @@ -451,6 +535,11 @@ type fnctx = { module answers true for the host's cells, globals and bodies, and those go through the GOT -- see [modrm_got]. *) ext : string -> bool; + (* The line table under construction, in a [--debug] build. [None] is every + other build, and then [dwline] below is the only thing that looks at it + and does nothing — so a release build's output is byte-identical to what + it was before debug information existed. *) + dw : dwarf option; } (* Module-wide rather than per-function. Two functions each holding an [if] @@ -462,6 +551,36 @@ let uniq = ref 0 let new_label _f tag = incr uniq; Printf.sprintf ".L%s%d" tag !uniq +(* A line-table row at the point the output has reached, unless the last row + already named this position. [lower] calls this for every expression, so + the dedup is what keeps a statement made of a dozen nodes on one line from + costing a dozen rows; the column is part of the key, so two forms on one + line are still told apart. + + Line 0 is [Loc.unknown] — a node the checker invented rather than one + anyone wrote. It is attributed to the enclosing function's own line, for + [emit.ml]'s reason at its [at_loc]: a zero line in DWARF means "no line", + and a debugger given one steps over the whole construct. *) +let dwline f (loc : Loc.t) = + match f.dw with + | None -> () + | Some dw -> + (match dw.dcur with + | None -> () + | Some s -> + let file, line, col = + if loc.Loc.line = 0 then (s.sfile, s.sline, 1) + else (dwfile dw loc.Loc.file, loc.Loc.line, loc.Loc.col) + in + if dw.dlast <> Some (file, line, col) && f.b.n > dw.dlastn then begin + let l = new_label f "dl" in + lbl f.b l; + dw.dlast <- Some (file, line, col); + dw.dlastn <- f.b.n; + s.srows <- + { rlbl = l; rfile = file; rline = line; rcol = col } :: s.srows + end) + (* Bump-allocate a frame temporary and answer its rbp-relative offset. The offset is negative, so the running total is rounded *up* to the alignment; rbp is 16-aligned, so that is the alignment the value actually gets. *) @@ -962,6 +1081,12 @@ let emit_args f (args : arg list) = let sink = Lf 0 let rec lower f (e : Tast.expr) (dst : loc) : unit = + (* The one hook the line table needs, and it is here rather than at + statement granularity on purpose: the same recursion that lowers a nested + call lowers its arguments, and a row per expression is what makes a + backtrace through an argument name the argument rather than the call. It + is inert in every build but a [--debug] one. *) + dwline f e.Tast.loc; if dst == sink && not (is_void e.Tast.ty) then scoped f (fun () -> let o = tmp f e.Tast.ty in @@ -2365,7 +2490,7 @@ let incoming_of ~sret (params : Types.t list) = sret_at, ps, next_int () let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) - ?(hidden = false) (fn : Tast.fn) : string * string = + ?(hidden = false) ?dw (fn : Tast.fn) : string * string = let b = create () in let nslots = Array.length fn.Tast.slots in let f = @@ -2374,7 +2499,35 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) xfer_off = 0; sret_off = 0; retval = 0; frame = 0; maxframe = 0; outgoing = 0; loops = []; pads = []; xfer_lbl = ""; unwound = false; - rodata = Buffer.create 64; externs; fns; ext } + rodata = Buffer.create 64; externs; fns; ext; dw } + in + (* The subprogram this function's rows hang off. Its first row is the + function symbol itself, at the line the [defn] was written on, so the + entry has a position before the prologue has run; every row after it + comes out of [dwline] as the body is lowered. [dlast] is not primed with + it, which is deliberate — the first form of the body sits at a different + column even when it is on the same line, so it gets a row of its own, and + two rows are what let a debugger put a breakpoint after the prologue + rather than on it. *) + let sub = + match dw with + | None -> None + | Some d -> + let line = + if fn.Tast.floc.Loc.line = 0 then 1 else fn.Tast.floc.Loc.line + in + let file = dwfile d fn.Tast.floc.Loc.file in + let s = + { sname = fn.Tast.name; ssym = fsym fn.Tast.name; sfile = file; + sline = line; send = new_label f "fe"; + srows = [ { rlbl = fsym fn.Tast.name; rfile = file; rline = line; + rcol = 1 } ] } + in + d.dcur <- Some s; + d.dlast <- None; + d.dlastn <- -1; + d.dsubs <- s :: d.dsubs; + Some s in (* The header's own frame model: every slot and every temporary is bump-allocated below rbp, and the high-water mark is what the prologue @@ -2567,6 +2720,14 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) Buffer.add_string out (sym ^ ":\n"); Buffer.add_string out (Buffer.contents pb.out); Buffer.add_string out (Buffer.contents f.b.out); + (* One past the last byte, which is what [DW_AT_high_pc] and the line + table's closing [DW_LNE_end_sequence] both want. [.size]'s [. - sym] says + the same thing but is an expression rather than a symbol, and + [DW_FORM_addr] takes a symbol. *) + (match sub with + | Some s -> Buffer.add_string out (s.send ^ ":\n") + | None -> ()); + (match dw with Some d -> d.dcur <- None | None -> ()); Buffer.add_string out (Printf.sprintf "\t.size\t%s, . - %s\n\n" sym sym); Buffer.contents out, Buffer.contents f.rodata @@ -2645,7 +2806,8 @@ let emit_globals_init (md : Emit.m) ~externs ~fns (globals : Tast.global list) = fret = Types.Unit; slots = [||]; xfer_off = 0; sret_off = 0; retval = 0; frame = 0; maxframe = 0; outgoing = 0; loops = []; pads = []; xfer_lbl = ""; unwound = false; - rodata = Buffer.create 64; externs; fns; ext = (fun _ -> false) } + rodata = Buffer.create 64; externs; fns; ext = (fun _ -> false); + dw = None } in (* Two slots, not one: [xfer_off] holds the *pointer* every call passes on, and [cell] is what it points at. Storing a null into [xfer_off] itself — @@ -2773,8 +2935,210 @@ let emit_cells (p : Tast.program) = p.Tast.fns; Buffer.contents out +(* ── The DWARF sections ──────────────────────────────────────────────── *) + +(* A path inside an assembler string literal. Flan source paths contain + neither of these two characters in practice, but a path is user input and + a stray backslash would end the directive rather than the string. *) +let asm_str s = + let b = Buffer.create (String.length s + 8) in + String.iter + (fun c -> + if c = '"' || c = '\\' then Buffer.add_char b '\\'; + Buffer.add_char b c) + s; + Buffer.contents b + +(* Absolute, because the file table below gives every entry directory index 0 + and a debugger then reads the name as written. [emit.ml]'s [dfile] splits + the same absolute path into a [!DIFile]'s filename and directory; DWARF is + happy with either and one string is fewer moving parts. *) +let abspath p = + if Filename.is_relative p then Filename.concat (Sys.getcwd ()) p else p + +(* [DW_LNE_set_address] on a label: the escape opcode, a length of 9, the + sub-opcode, and eight bytes of address the assembler relocates. + + Every row gets one of these rather than a [DW_LNS_advance_pc] with a + computed delta, and the reason is that a delta would be a difference of two + labels inside a [.uleb128], which asks the assembler to resolve a value + whose size affects the values after it. That does work, but it is exactly + the kind of thing whose failure looks like a DWARF bug. Rows are eleven + bytes each here and that is a debug build's business. *) +let dw_set_address b lbl = + Buffer.add_string b (Printf.sprintf "\t.byte\t0, 9, 2\n\t.quad\t%s\n" lbl) + +(* The line-number program: one sequence per function, closed with an + end_sequence at the label one past its last byte. The registers are reset + at the head of every sequence, which is why [line] starts at 1 and [file] + at 1 in each. Rows within a sequence must be address-ordered, and they are + for free: [dwline] appends them in the order the bytes are emitted. *) +let dw_line_program (dw : dwarf) = + let b = Buffer.create 4096 in + List.iter + (fun s -> + let line = ref 1 and file = ref 1 and col = ref 0 in + List.iter + (fun r -> + dw_set_address b r.rlbl; + if r.rfile <> !file then begin + Buffer.add_string b + (Printf.sprintf "\t.byte\t4\n\t.uleb128 %d\n" r.rfile); + file := r.rfile + end; + if r.rcol <> !col then begin + Buffer.add_string b + (Printf.sprintf "\t.byte\t5\n\t.uleb128 %d\n" r.rcol); + col := r.rcol + end; + if r.rline <> !line then begin + Buffer.add_string b + (Printf.sprintf "\t.byte\t3\n\t.sleb128 %d\n" (r.rline - !line)); + line := r.rline + end; + Buffer.add_string b "\t.byte\t1\n") + (List.rev s.srows); + dw_set_address b s.send; + Buffer.add_string b "\t.byte\t0, 1, 1\n") + (List.rev dw.dsubs); + Buffer.contents b + +(* The three sections. [tbeg] and [tend] bracket everything this object puts + in [.text] — the compile unit claims that whole range, including the C + [main] shim and the globals' initialiser, neither of which has any rows. + That is honest: they are code this unit produced, and a debugger that finds + no line for an address inside them says so. *) +let emit_dwarf (dw : dwarf) ~cufile ~tbeg ~tend = + let out = Buffer.create 8192 in + (* The compile unit's directory, and the directory table the file entries + index into. [emit.ml]'s [dfile] splits every path into a basename and a + directory the same way, and the reason to match it is what a debugger + prints: a frame reads [debug.flan:19] rather than the eighty characters + of an absolute path. Directory index 0 is the compile unit's own + directory, so a file sitting beside the one that named the unit -- which + is most of them -- costs no entry at all. *) + let cudir = Filename.dirname (abspath cufile) in + let dirs = ref [] in + let dirix d = + if String.equal d cudir then 0 + else + match List.assoc_opt d !dirs with + | Some n -> n + | None -> + let n = List.length !dirs + 1 in + dirs := !dirs @ [ (d, n) ]; + n + in + let files = + List.map + (fun p -> + let a = abspath p in + (Filename.basename a, dirix (Filename.dirname a))) + (List.rev dw.dpaths) + in + Buffer.add_string out + "\n# Debug information. Written out as data rather than left to the\n\ + # assembler's .loc, which cannot work here: GAS builds its line table\n\ + # when it assembles an instruction, and this file assembles none.\n"; + (* .debug_abbrev. Two abbreviations, because two shapes of DIE are all that + is described. *) + Buffer.add_string out + "\t.section\t.debug_abbrev,\"\",@progbits\n\ + .Ldwabbrev:\n\ + \t.uleb128 1\n\ + \t.uleb128 0x11\t\t# DW_TAG_compile_unit\n\ + \t.byte\t1\t\t# has children\n\ + \t.uleb128 0x25\n\t.uleb128 0x08\t# DW_AT_producer DW_FORM_string\n\ + \t.uleb128 0x13\n\t.uleb128 0x05\t# DW_AT_language DW_FORM_data2\n\ + \t.uleb128 0x03\n\t.uleb128 0x08\t# DW_AT_name DW_FORM_string\n\ + \t.uleb128 0x1b\n\t.uleb128 0x08\t# DW_AT_comp_dir DW_FORM_string\n\ + \t.uleb128 0x11\n\t.uleb128 0x01\t# DW_AT_low_pc DW_FORM_addr\n\ + \t.uleb128 0x12\n\t.uleb128 0x07\t# DW_AT_high_pc DW_FORM_data8\n\ + \t.uleb128 0x10\n\t.uleb128 0x17\t# DW_AT_stmt_list DW_FORM_sec_offset\n\ + \t.byte\t0, 0\n\ + \t.uleb128 2\n\ + \t.uleb128 0x2e\t\t# DW_TAG_subprogram\n\ + \t.byte\t0\t\t# no children\n\ + \t.uleb128 0x3f\n\t.uleb128 0x19\t# DW_AT_external DW_FORM_flag_present\n\ + \t.uleb128 0x03\n\t.uleb128 0x08\t# DW_AT_name DW_FORM_string\n\ + \t.uleb128 0x6e\n\t.uleb128 0x08\t# DW_AT_linkage_name DW_FORM_string\n\ + \t.uleb128 0x3a\n\t.uleb128 0x0f\t# DW_AT_decl_file DW_FORM_udata\n\ + \t.uleb128 0x3b\n\t.uleb128 0x0f\t# DW_AT_decl_line DW_FORM_udata\n\ + \t.uleb128 0x11\n\t.uleb128 0x01\t# DW_AT_low_pc DW_FORM_addr\n\ + \t.uleb128 0x12\n\t.uleb128 0x07\t# DW_AT_high_pc DW_FORM_data8\n\ + \t.byte\t0, 0\n\ + \t.byte\t0\n"; + (* .debug_info. DW_LANG_C99 for [emit.ml]'s reason: it is less a claim about + the source language than the truth about the data model, and it is what + makes a debugger's own struct printing correct against this layout. *) + Buffer.add_string out + (Printf.sprintf + "\t.section\t.debug_info,\"\",@progbits\n\ + .Ldwinfo:\n\ + \t.long\t.Ldwinfo_end - .Ldwinfo_ver\n\ + .Ldwinfo_ver:\n\ + \t.short\t4\n\ + \t.long\t.Ldwabbrev\n\ + \t.byte\t8\n\ + \t.uleb128 1\n\ + \t.asciz\t\"flan (x86-64)\"\n\ + \t.short\t0x000c\t\t# DW_LANG_C99\n\ + \t.asciz\t\"%s\"\n\ + \t.asciz\t\"%s\"\n\ + \t.quad\t%s\n\ + \t.quad\t%s - %s\n\ + \t.long\t.Ldwline\n" + (asm_str (Filename.basename (abspath cufile))) + (asm_str cudir) + tbeg tend tbeg); + List.iter + (fun s -> + Buffer.add_string out + (Printf.sprintf + "\t.uleb128 2\n\t.asciz\t\"%s\"\n\t.asciz\t\"%s\"\n\ + \t.uleb128 %d\n\t.uleb128 %d\n\t.quad\t%s\n\t.quad\t%s - %s\n" + (asm_str s.sname) + (asm_str ("flan." ^ s.sname)) + s.sfile s.sline s.ssym s.send s.ssym)) + (List.rev dw.dsubs); + Buffer.add_string out "\t.byte\t0\n.Ldwinfo_end:\n"; + (* .debug_line. The standard opcode lengths are the standard ones; changing + them would change nothing, since every row below is spelled out of the + three opcodes this emitter uses and never out of a special opcode. *) + Buffer.add_string out + "\t.section\t.debug_line,\"\",@progbits\n\ + .Ldwline:\n\ + \t.long\t.Ldwline_end - .Ldwline_ver\n\ + .Ldwline_ver:\n\ + \t.short\t4\n\ + \t.long\t.Ldwline_prog - .Ldwline_hdr\n\ + .Ldwline_hdr:\n\ + \t.byte\t1\t\t# minimum_instruction_length\n\ + \t.byte\t1\t\t# maximum_operations_per_instruction\n\ + \t.byte\t1\t\t# default_is_stmt\n\ + \t.byte\t0xfb\t\t# line_base = -5\n\ + \t.byte\t14\t\t# line_range\n\ + \t.byte\t13\t\t# opcode_base\n\ + \t.byte\t0,1,1,1,1,0,0,0,1,0,0,1\n"; + List.iter + (fun (d, _) -> + Buffer.add_string out (Printf.sprintf "\t.asciz\t\"%s\"\n" (asm_str d))) + !dirs; + Buffer.add_string out "\t.byte\t0\t\t# end of the directory table\n"; + List.iter + (fun (name, dir) -> + Buffer.add_string out + (Printf.sprintf "\t.asciz\t\"%s\"\n\t.uleb128 %d\n\t.uleb128 0\n\ + \t.uleb128 0\n" + (asm_str name) dir)) + files; + Buffer.add_string out "\t.byte\t0\n.Ldwline_prog:\n"; + Buffer.add_string out (dw_line_program dw); + Buffer.add_string out ".Ldwline_end:\n"; + Buffer.contents out + (* A whole program as one assembly file. *) -let program ~checks ?(dev = false) (p : Tast.program) : string = +let program ~checks ?(dev = false) ?(debug = false) (p : Tast.program) : string = check_no_transfer p; let md = layout_ctx ~checks ~dev p in let externs = Hashtbl.create 16 in @@ -2783,15 +3147,41 @@ let program ~checks ?(dev = false) (p : Tast.program) : string = p.Tast.externs; let fns = Hashtbl.create 64 in List.iter (fun (fn : Tast.fn) -> Hashtbl.replace fns fn.Tast.name ()) p.Tast.fns; + let dw = if debug then Some (new_dwarf ()) else None in + (* The file every diagnostic in this unit is really about: the first + function anyone actually wrote. [emit.ml]'s [new_dbg] picks it the same + way and for the same reason -- the prelude contributes functions too, and + naming the prelude as the compile unit would be true and useless. *) + let cufile = + match + List.find_opt (fun (f : Tast.fn) -> f.Tast.floc.Loc.line > 0) p.Tast.fns + with + | Some f -> f.Tast.floc.Loc.file + | None -> "" + in let text = Buffer.create 65536 and rodata = Buffer.create 4096 in Buffer.add_string text "# Generated by flan's x86-64 backend (the dev one). The instructions are\n\ # .byte blobs so that every byte offset stays exactly known; the few\n\ - # fields that need a relocation are assembler expressions.\n\ - \t.text\n\n"; + # fields that need a relocation are assembler expressions.\n"; + (* A numbered [.file] is what stops clang's integrated assembler from + generating a compile unit of its *own* over this file -- one that names + the .s, and whose rows land at the [call] mnemonics, which are the only + real instructions here. Those addresses are inside the functions this + unit already describes, so two units would claim them. Measured: with the + directive, the assembler emits an empty line table and nothing else. + Harmless in a release build, where it is simply never emitted. *) + if debug then + Buffer.add_string text + (Printf.sprintf "\t.file\t1 \"%s\"\n" (asm_str (abspath cufile))); + (* The label the compile unit's range starts at, and it is emitted only in a + debug build so that a release build's assembly is byte-for-byte what it + was before any of this existed. *) + Buffer.add_string text + (if debug then "\t.text\n.Ldwtext:\n\n" else "\t.text\n\n"); List.iter (fun (fn : Tast.fn) -> - let t, r = emit_fn md ~externs ~fns fn in + let t, r = emit_fn md ~externs ~fns ?dw fn in Buffer.add_string text t; Buffer.add_string rodata r) p.Tast.fns; @@ -2805,6 +3195,11 @@ let program ~checks ?(dev = false) (p : Tast.program) : string = which is what [reload_host.c] is. Refusing here made a --x86 host for the reload tests impossible to build. *) | None -> ()); + (* The end of everything this object puts in .text, and therefore the end of + the compile unit's range. It has to be written while .text is still the + current section, which is why it is here and not beside the sections + below. *) + if debug then Buffer.add_string text ".Ldwtext_end:\n"; let out = Buffer.create 65536 in Buffer.add_buffer out text; (* The globals' initialiser runs before main, through the same constructor @@ -2825,6 +3220,11 @@ let program ~checks ?(dev = false) (p : Tast.program) : string = Buffer.add_string out (emit_globals_data md p.Tast.globals); Buffer.add_string out "\n\t.section\t.rodata\n"; Buffer.add_buffer out rodata; + (match dw with + | Some d -> + Buffer.add_string out + (emit_dwarf d ~cufile ~tbeg:".Ldwtext" ~tend:".Ldwtext_end") + | None -> ()); Buffer.add_string out "\n\t.section\t.note.GNU-stack,\"\",@progbits\n"; Buffer.contents out