diff --git a/HANDOFF-x86-cost.md b/HANDOFF-x86-cost.md new file mode 100644 index 0000000..995b625 --- /dev/null +++ b/HANDOFF-x86-cost.md @@ -0,0 +1,102 @@ +# Handoff — what the x86 backend costs, and keeping it from rotting + +Branch: `dev-loop`. This lane is item 7 of `HANDOFF-x86-rt.md`'s "What remains" — code size and speed, measured +rather than guessed — plus the closing note under it, which asks for the survey to run on its own so a refusal +cannot sit unnoticed for a month again. + +Nothing in `lib/` is touched. Another lane is rewriting `lib/x86.ml` at the same time, so everything here lives in +`spike/x86` and in `test/dune`, and anything this lane finds that would want a compiler change is written down at +the bottom rather than made. + +## 1. The `@x86` alias — done, and it fails when it should + +`test/dune` has a fourth alias beside `@sanitize` and `@valgrind`, shaped the same way and opt-in for the same +reason: it builds every corpus program twice and runs both, which is minutes where `dune test` is forty seconds. + + dune build --root . @x86 + +It is a `(rule ...)` with no `(executable ...)` beside it, which is where it differs from its two neighbours. The +check already exists — `spike/x86/survey.sh` is what every handoff quotes its counts from — and a second +implementation of it in OCaml would be a second thing to drift, which is exactly the failure this alias is meant +to prevent. So the rule runs the script, and the script grew three environment variables to make that possible: + +- `SURVEY_STRICT=1` turns the report into an exit status. A DIFFER is a wrong answer and a refusal by name is a + node the backend has stopped lowering; either fails the build. NOX86 and SKIP do not — the first is usually a + toolchain that is not installed on the machine, the second is the frontend refusing the program on both sides. +- `FLAN=` hands it a compiler somebody else built. Without it the script runs `dune build` on `bin/main.exe`, + and a dune inside a dune action waits on a lock it will never get. The rule has `bin/main.exe` in its deps + instead, exactly as the `(tests ...)` stanza already does. +- `SURVEY_CORPUS` for where the programs are read from, which under dune is the build tree. + +Standalone, with nothing set, the script is byte-for-byte the measurement it always was. + +**Verified both directions.** `dune build --root . @x86` passes on a clean tree and prints the same +**97 MATCH / 0 DIFFER / 0 REFUSED / 0 NOX86 / 36 SKIP** the script prints standalone — the build tree and the +source tree agree about the corpus, which was not obvious in advance. Broken deliberately (a fake DIFFER injected +into the script, the rule narrowed to one program so the round trip was thirty seconds) it fails the dune build +with `x86 survey FAILED: 1 differ, 0 refused` and a nonzero status. Both edits reverted. + +`dune test --root .` is still green and still about forty seconds. The new stanza is a rule on its own alias and +touches no `(modules ...)` list, so it cannot be pulled into `@runtest` by accident. The two +`clang: error: linker command failed` lines it prints are the `dev-robust` fixture doing its job and are called +out in `HANDOFF-x86-rt.md` §4 already. + +## 2. The measurement — `spike/x86/COST.md` + +`COST.md` is the deliverable and it carries the numbers and the reading of them; the raw rows are committed beside +it as `cost-corpus.tsv` and `cost-bench.tsv`. The headline, in one paragraph: + +**This backend emits 3.84× the code LLVM does at `-O2`, and 1.92× what LLVM emits at `-O0`** — half the factor is +the optimiser Flan ships with and not the backend. Per program the second ratio is tight: median 2.21, quartiles +1.94 and 2.98. Speed is unmeasurable over the corpus (every program is 2.5ms of `execve`) except `recur`, which +loops ten million times and is **6× LLVM `-O0`**; on purpose-written loops the backend is 2.8× to 4.7×. + +Of the five suspected costs the old handoff named, the measurement says: the frame-slot round trip on every +intermediate is most of everything and is the item worth fixing; `rep movsb` costs twenty cycles a copy and is +worth fixing cheaply; the bounds check's three temporaries cost 241 bytes and *one cycle*, so they are a code-size +item and not a speed item; the guard after every call is four instructions and invisible; and the dev build's +extra load per call site is not measurable at all. What a dev build does cost — 400× the code, because nothing may +be dropped when anything might be redefined — is `Reach`'s doing and both backends pay it. + +The two scripts are how that was produced: + +- `cost.sh` sweeps the corpus and prints a TSV: file size, `.text`, and **the sum of the `flan.*` defined symbols**, + which is the program's own code with the runtime excluded. `COST_FLAGS=--dev` is the `SURVEY_FLAGS` precedent. +- `bench.sh` runs four programs in `spike/x86/bench/`, each written so that one suspected cost is most of what the + program does, because every program in `test/programs` runs in about two and a half milliseconds of which nearly + all is `execve`. + +They live in `spike/x86/bench/`, a subdirectory, *deliberately*: `survey.sh` globs `spike/x86/*.flan`, and a +benchmark landing in the survey would move the 97 that three handoffs quote. + +## 3. One trap, paid for once + +Two copies of `cost.sh` sharing a scratch directory silently corrupt each other's numbers. The first release sweep +was run, killed, and restarted; `pkill` missed a child, and the survivor kept writing the same `l` and `x` files +the new run was building and timing. The result was a row for `math` carrying `dev-locals`' binaries — which is +visible only because those two happened to collide *exactly*, and a collision between two programs of merely +similar size would have produced a plausible row nobody would question. + +`cost.sh` now makes a `run.$$` subdirectory with a plain `mkdir`, so a second copy cannot land in the first one's. +The sweep behind `COST.md` was re-run clean afterwards. If you extend these scripts, keep that property. + +## 4. Nothing was changed in `lib/`, and nothing needed to be + +No finding here wants a compiler change that this lane was not allowed to make. The costs are all in +`lib/x86.ml`'s lowering and the next lane rewriting that file will meet them; `COST.md`'s last section says which +ones are worth its attention and which are not, which was the point of measuring rather than guessing. + +## 5. What was not done + +- **The `--dev` sweep over the whole corpus was not finished.** It was started, ran into the `dev-*` programs + (each of which waits out a twenty-second timeout twice), and was killed to free the machine for the clean + release re-run. The `--dev` numbers in `COST.md` come from `bench.sh`, which measures the same thing on four + programs and measures it better, because a corpus program's dev cost is invisible under `execve`. Re-running it + is `COST_FLAGS=--dev spike/x86/cost.sh` and about half an hour. +- **No corpus program was disassembled.** The attribution in `COST.md` is all from the four benchmarks, which are + small enough to read whole and were written so that each one is mostly a single suspected cost. The corpus + outliers are explained from the `-O0` column instead — which is what that column is for, and in `bounds`' case + it turns an alarming 11× into an ordinary 2.07×. +- **`LLVM -O0` is only reachable through `--debug`**, which also asks for DWARF. Checked and harmless: DWARF lands + in `.debug_*` and the metric sums `.text` symbols. A plain `-O0` flag on the CLI would be a `bin/main.ml` change + and this lane was not touching compiler sources. diff --git a/spike/x86/COST.md b/spike/x86/COST.md new file mode 100644 index 0000000..0e5835a --- /dev/null +++ b/spike/x86/COST.md @@ -0,0 +1,210 @@ +# What the hand-written x86-64 backend costs + +`survey.sh` has said for three handoffs that this backend agrees with LLVM on all 97 corpus programs it can build. +Item 7 of `HANDOFF-x86-rt.md` is the other half of that sentence — nobody had a number for what the agreement +costs — and it came with a list of suspects: a guard after every call, three frame temporaries per bounds check, +every intermediate in memory, `rep movsb` block copies, and an extra load per call site in a dev build. This is +the measurement. It does not change anything; two of the five suspects turn out not to matter, and the one that +matters most is not on the list. + +Produced by `spike/x86/cost.sh` (the corpus, for size) and `spike/x86/bench.sh` (four purpose-written programs, +for speed). Both are documented in their own headers. Machine: 16-core x86-64, Fedora, clang as the assembler and +linker on both sides, and other work running on it throughout — which is why every time below is the *minimum* of +five or seven runs and why nothing here rests on a difference of a few percent. + +The rows behind the tables are committed beside this file as `cost-corpus.tsv` and `cost-bench.tsv`, so a later +lane can recompute a ratio rather than believe one. + +## What is being compared, and against what + +The interesting column is not the size of the executable. A Flan binary is mostly `flan_rt.o` and libc glue, the +same object on both sides, and it drowns the signal: over the corpus the whole file is only **1.09×** bigger +through this backend and `.text` only **1.18×**, which would be a reassuring number and a meaningless one. + +So the measurement is the sum of the sizes of the defined symbols the compiler *named itself* — everything called +`flan.`. The runtime's C is `flan_` with an underscore, so the two never collide, and a +runtime symbol has the same size on both sides (`flan_map_clone`, `0x4b3` either way), which is the check that +says the difference really is codegen and not a differently-linked runtime. + +There is a third column, and it is what makes the second readable: **LLVM with `--debug`, which forces `-O0`**. +This backend has no optimiser at all, so measuring it against LLVM at `-O2` charges it for the whole of mem2reg, +inlining and constant folding. `-O0` is LLVM's instruction selection with none of that, which is the comparison +that says something about *this* backend rather than about the absence of a middle end. (`--x86 --debug` is +refused — the backend emits no DWARF — so the column exists on one side only. DWARF lands in `.debug_*` sections +and not in `.text`, checked, so it does not contaminate the symbol sums.) + +## The corpus: size + +Ninety-seven programs — the same set `survey.sh` matches on, minus the two that run forever. Summed over all of +them: + +| | LLVM -O2 | LLVM -O0 | x86 | +|---|---|---|---| +| own code, all 97 programs | 221,608 | 444,504 | 851,638 | +| against LLVM -O2 | 1.00× | 2.01× | **3.84×** | +| against LLVM -O0 | | 1.00× | **1.92×** | + +So the headline is two numbers, not one. **This backend emits 3.8× the code LLVM does at `-O2`, and half of that +factor is the optimiser Flan ships with rather than anything about the backend; against LLVM with the optimiser +off it is 1.9×.** Per program the second ratio is tight — median 2.21, quartiles 1.94 and 2.98, the whole range +1.32 to 5.44 — which is itself a finding: the cost is not a few bad nodes, it is a constant tax on everything. + +The ten largest programs, which are where the bytes actually are: + +| program | LLVM -O2 | LLVM -O0 | x86 | x86 / -O2 | x86 / -O0 | +|---|---|---|---|---|---| +| `strings` | 13,395 | 22,681 | 33,265 | 2.48× | 1.47× | +| `maps` | 12,833 | 19,850 | 40,445 | 3.15× | 2.04× | +| `edn` | 12,057 | 34,556 | 48,532 | 4.03× | 1.40× | +| `generics` | 10,137 | 20,427 | 39,241 | 3.87× | 1.92× | +| `slurp` | 9,744 | 14,813 | 24,656 | 2.53× | 1.66× | +| `into` | 8,277 | 12,345 | 21,320 | 2.58× | 1.73× | +| `vec` | 8,039 | 12,089 | 22,770 | 2.83× | 1.88× | +| `map-iter` | 7,211 | 10,374 | 21,715 | 3.01× | 2.09× | +| `algorithms` | 6,776 | 16,616 | 30,875 | 4.56× | 1.86× | +| `slices` | 2,393 | 9,057 | 17,753 | 7.42× | 1.96× | + +And the two ends of the distribution, both of which are more interesting than the middle: + +| program | LLVM -O2 | LLVM -O0 | x86 | x86 / -O2 | x86 / -O0 | why | +|---|---|---|---|---|---|---| +| `bounds` | 363 | 1,951 | 4,039 | **11.13×** | 2.07× | LLVM at `-O2` proves the indices and deletes the checks | +| `array-ctor` | 357 | 1,840 | 3,880 | 10.87× | 2.11× | the same, over a constructor's worth of stores | +| `p2-loop-print` | 283 | 106 | 577 | 2.04× | **5.44×** | a `-O0` build *smaller* than `-O2`: LLVM unrolls the five-iteration loop and `-O0` does not | +| `pkg-return` | 7,024 | 23,635 | 31,716 | 4.52× | 1.34× | mostly prelude, where `-O0` is already fat | + +`bounds` is the clearest case in the table of why the `-O0` column had to exist. Eleven times is a shocking +number and it is not about this backend at all: the program's whole point is indexing, LLVM at `-O2` can see the +indices are in range and removes the check, and neither LLVM at `-O0` nor this backend can. Against the compiler +that also keeps every check, `bounds` is 2.07× — a completely ordinary row. + +## Where the size goes + +Every one of these is from the disassembly of the benchmark programs, which are small enough to read whole. + +**Every intermediate goes through the frame, and so does every constant.** This is the big one and it is not one +feature, it is the shape of the whole backend. `(step acc 1)` in `b1-calls` compiles to: + + movabs $0x1,%rax ; a 10-byte immediate ... + mov %rax,-0x30(%rbp) ; ... stored to a frame slot ... + mov -0x30(%rbp),%rsi ; ... and loaded back into the argument register + +Three instructions and 24 bytes where LLVM writes `mov $1,%esi`, five. The loop bound gets the same treatment +*every iteration* — `movabs $0x1312d00` into a slot, sign-extended out of it, compared — because nothing is +hoisted. So does the loop condition: `cmp`/`setl`/`movzbq`/store a byte to the frame/reload it/`test`/`jne`, +seven instructions for what is `cmp`/`jge` anywhere else. This is most of the 2× against `-O0` and essentially +all of the difference on the programs at the bottom of the table, which have no calls, no bounds checks and no +aggregates in them at all. + +**The guard after every call is four instructions and one dependent load.** + + call 4009f8 + mov -0x18(%rbp),%r11 ; the condition frame, from its own slot + mov 0x0(%r11),%r11 ; ... dereferenced + test %r11,%r11 + jne + +Roughly 25 bytes per call site. Real, cheap, and third in size behind the two above it — on a program that is +nothing *but* calls (`b1`) the whole backend is 2.8× LLVM `-O0`, and the guard is a minority of that. + +**The bounds check is three frame temporaries, as suspected, and it costs code and not time.** The index is +widened, stored, reloaded, stored again, the limit goes to a third slot, and then `cmp`/`jb` — with the failure +path, its `.rodata` location string and its length, inline at the branch target. On `b2-bounds`, `flan.main` is +`0x4b1` with checks and `0x3c0` without: **241 bytes, a quarter of the function.** In time it is 112.6ms against +105.0ms over 20.5 million checked loads — **about 0.4ns, a cycle or two a check** — because the branch predicts +perfectly and the loads were going to memory anyway. That contradicts the way the handoff's list reads. The three +temporaries are a code-size item. They are not a speed item. + +**`rep movsb` is real and it is the most expensive single instruction here.** A 64-byte struct copy lowers to +`lea`/`lea`/`movabs $0x40,%rcx`/`rep movsb`, and `b4-copy` runs 2 million of them in 20.8ms against LLVM `-O0`'s +7.2ms: **about 6.8ns of the difference per copy, some twenty cycles**, which is `rep movsb`'s startup cost and +almost none of it the 64 bytes. It is also the one place where the backend loses to `-O0` by a factor (2.9×) it +does not lose by on straight-line code, and the one item on the suspect list where a targeted fix — inline +16-byte moves under some size threshold — would pay for itself. + +## The corpus: speed, and why there is barely any + +Almost nothing. **Every program in `test/programs` runs in about 2.5 milliseconds, nearly all of it `execve` and +the dynamic loader**, and both backends produce the same 2.5 milliseconds. Best-of-five does not rescue a signal +that is not there. There is exactly one corpus program whose own code is a measurable part of its runtime, and it +is the right one: + +| program | LLVM -O2 | LLVM -O0 | x86 | what it is | +|---|---|---|---|---| +| `recur` | ~1ms | 10ms | 60ms | a ten-million-iteration counting loop, written to prove `recur` is a jump | + +Read that carefully, because the 25× against `-O2` is not a fact about this backend: LLVM folds the loop to its +answer and runs nothing. Against `-O0`, which also runs ten million iterations, it is **6×** — about 6ns an +iteration against 1ns, or roughly eighteen cycles for `i+1` and a compare. That is the frame-slot round trip +above, four or five times over, and it is the honest number. + +## The benchmarks + +Four programs in `spike/x86/bench/`, each written so that one suspected cost is most of what the program does. +They are in a subdirectory on purpose: `survey.sh` globs `spike/x86/*.flan` and a benchmark is not a case. +Times are best-of-seven, in milliseconds. + +| bench | what it is | LLVM -O2 | LLVM -O0 | x86 | x86 / -O0 | own code, -O0 → x86 | +|---|---|---|---|---|---|---| +| `b1-calls` | 20M calls of a one-instruction function | 2.1 | 36.8 | 104.3 | 2.8× | 200 → 807 | +| `b2-bounds` | 20.5M bounds-checked array loads | 4.7 | 27.0 | 112.6 | 4.2× | 407 → 1390 | +| `b3-spill` | 5M iterations of a six-deep arithmetic tree | 22.2 | 29.0 | 137.3 | 4.7× | 248 → 1117 | +| `b4-copy` | 2M copies of a 64-byte struct | 2.0 | 7.2 | 20.8 | 2.9× | 348 → 877 | + +`b2` with `--no-bounds-checks` on both sides: LLVM 4.5ms, x86 105.0ms — the 7.6ms the check costs over 20.5 +million of them, and the 241 bytes it costs in `flan.main`, are the whole of it. + +`b3`'s ratio is the one to distrust slightly: its expression ends in a `%`, which is an `idiv`, and an `idiv` is +twenty-odd cycles on every side. That is most of LLVM's own 22.2ms and a good part of its 29.0ms, so the +denominator is largely a hardware latency this backend cannot do anything about. The absolute gap — 108ms over +5 million iterations, about 21ns of extra work each — is the honest reading of that row. + + + +`b1-calls` and `b3-spill` are the pair to read together, with the `idiv` caveat above in mind. `b1` is 20 million +calls of a one-instruction function and lands at 2.8× `-O0`; `b3` has no calls at all and pays 21ns an iteration +for six dependent arithmetic temporaries. **The backend is worse at arithmetic than it is at calling**, which is +the opposite of what the suspect list implies, and it is because a call already costs enough that four extra +instructions beside it disappear, while an add that should be one instruction costs five. `recur`, which is a +counting loop and nothing else, says the same thing on a corpus program: 6× LLVM `-O0`. + +The `-O2` column in `b1` and `b4` is 2ms — the loop is gone. That is a true fact about the toolchain Flan ships +and a useless one about code generation, which is the whole reason the `-O0` column exists. + +## `--dev`, which is the one axis both backends pay + +`SURVEY_FLAGS=--dev` reported 97 MATCH for the lane before this one, so the comparison is available. The suspected cost was the extra +load per call site — every cross-function call going through its indirection cell. **It is not measurable.** On +`b1-calls`, 20 million calls, x86 release is 104.3ms and x86 `--dev` is 100.9ms: the same number, and the dev +build is nominally the *faster* of the two, which is what a difference below the noise floor looks like. The load +is from a `.data` cell that is in L1 after the first call and the machine was already waiting on the frame. + +What a dev build actually costs is something else entirely, and both backends pay it. `b1-calls` is a program +with two functions in it: + +| build | own code | +|---|---| +| LLVM, release | 82 bytes | +| x86, release | 807 bytes | +| LLVM, `--dev` | 32,714 bytes | +| x86, `--dev` | 83,018 bytes | + +**A dev build emits the entire prelude**, because anything might be redefined and so nothing may be dropped. That +is four hundred times the code for this program, and it dwarfs every item on the suspect list put together. It is +also not a backend cost — LLVM pays a 400× of its own — so it is `Reach`'s business and not `x86.ml`'s. The +backend's share of it is the same ~2.5× it charges everywhere else. + +## What this says to the lane rewriting `lib/x86.ml` + +Ranked by what the numbers actually support, and not by the order of the list in the handoff: + +1. **Keep values in registers across a single expression.** Not a register allocator — just not routing every + constant and every subexpression through a frame slot, and not re-materialising a loop bound every iteration. + This is most of the 2× against `-O0` and most of `recur`'s 6×, and it is what `b3`'s 21ns an iteration buys. +2. **Inline small aggregate copies** instead of `rep movsb`. One instruction, twenty cycles, on a copy that is + four `movdqu` pairs. +3. **`flan_dev_reg_note` in a release build** (item 2 of the old handoff's list) is worth doing and is small. +4. **The call guard is fine.** Four instructions and 25 bytes, invisible in time. Leave it. +5. **The bounds check is fine on time and fat on code.** If it is ever worth touching, it is worth touching for + the 241 bytes — hoisting the failure path out of line would get most of that back without changing a cycle. +6. **The dev call cell is free.** Whatever the redefinition emitter costs, it does not cost this. diff --git a/spike/x86/bench.sh b/spike/x86/bench.sh new file mode 100755 index 0000000..0899f15 --- /dev/null +++ b/spike/x86/bench.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# The speed half of cost.sh, on programs that are long enough to time. +# +# Why this exists beside cost.sh rather than inside it: every program in +# test/programs runs in about two and a half milliseconds, of which nearly +# all is fork, exec and the dynamic loader. Best-of-five does not rescue a +# signal that is not there, and a table of 97 rows that all say "2.5ms vs +# 2.6ms" would be a measurement of execve. So the corpus answers the size +# question and these four answer the speed one, each written so that one +# suspected cost is most of what the program does. +# +# Four builds of each, and the third column is the one to read: +# +# llvm as shipped, -O2. Frequently the loop is simply gone; that is a +# true number about the toolchain and a useless one about codegen. +# llvm -O0 via --debug, which forces it. LLVM's instruction selection with +# its optimiser off -- the fair comparison for a backend that has +# no optimiser. +# x86 this backend. +# x86 nbc --no-bounds-checks, for b2, where the difference is the check. +# +# And --dev on both sides, which is the one suspected cost the two backends +# share: every cross-function call goes through an indirection cell, so it is +# a load and an indirect call where a release build has a direct one. b1 is +# where that has to show. +# +# Usage: spike/x86/bench.sh [name-substring ...] +set -u +orig=$(pwd) +here=$(cd "$(dirname "$0")" && pwd) +root=$(cd "$here/../.." && pwd) +cd "$root" || exit 1 + +if [ -n "${FLAN:-}" ]; then + case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac +else + dune build --root . bin/main.exe 2>&1 | head -30 + flan=$root/_build/default/bin/main.exe +fi +test -x "$flan" || { echo "build failed" >&2; exit 1; } + +out=${COST_OUT:-${TMPDIR:-/tmp}/flan-bench.$$} +mkdir -p "$out" || exit 1 +trap 'rm -rf "$out"' EXIT + +REPS=${COST_REPS:-5} + +best () { + min= + for i in $(seq "$REPS"); do + t0=$(date +%s%N) + timeout 120 "$1" >/dev/null 2>&1 /dev/null \ + | awk 'NF==4 && ($3=="T"||$3=="t") && $4 ~ /^flan\./ {n+=strtonum("0x"$2)} END{print n+0}' +} + +printf 'name\tllvm_us\to0_us\tx86_us\tllvm_nbc_us\tx86_nbc_us\tllvm_dev_us\tx86_dev_us\tllvm_own\to0_own\tx86_own\tx86_dev_own\n' + +for src in "$root"/spike/x86/bench/*.flan; do + name=$(basename "$src" .flan) + if [ $# -gt 0 ]; then + want=0 + for pat in "$@"; do case "$name" in *"$pat"*) want=1;; esac; done + [ $want = 1 ] || continue + fi + "$flan" build "$src" -o "$out/l" >/dev/null 2>&1 || { echo "$name: llvm build failed" >&2; continue; } + "$flan" build "$src" --debug -o "$out/d" >/dev/null 2>&1 || { echo "$name: -O0 build failed" >&2; continue; } + "$flan" build "$src" --x86 -o "$out/x" >/dev/null 2>&1 || { echo "$name: x86 build failed" >&2; continue; } + "$flan" build "$src" --no-bounds-checks -o "$out/ln" >/dev/null 2>&1 + "$flan" build "$src" --x86 --no-bounds-checks -o "$out/xn" >/dev/null 2>&1 + "$flan" build "$src" --dev -o "$out/lv" >/dev/null 2>&1 + "$flan" build "$src" --x86 --dev -o "$out/xv" >/dev/null 2>&1 + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$name" \ + "$(best "$out/l")" "$(best "$out/d")" "$(best "$out/x")" \ + "$(best "$out/ln")" "$(best "$out/xn")" \ + "$(best "$out/lv")" "$(best "$out/xv")" \ + "$(own "$out/l")" "$(own "$out/d")" "$(own "$out/x")" "$(own "$out/xv")" +done diff --git a/spike/x86/bench/b1-calls.flan b/spike/x86/bench/b1-calls.flan new file mode 100644 index 0000000..2cf2f50 --- /dev/null +++ b/spike/x86/bench/b1-calls.flan @@ -0,0 +1,20 @@ +;;;; A hot loop that does nothing but call. +;;;; +;;;; The suspected cost is the guard this backend emits after every call -- +;;;; and, in a dev build, the load of the indirection cell before it. Neither +;;;; is visible in the corpus, where a program's whole run is process startup. +;;;; Here the call is the program: the body is one add, so whatever separates +;;;; this from LLVM at -O0 is the call sequence and not the arithmetic. +;;;; +;;;; Not in spike/x86 proper, where survey.sh would pick it up: the survey's +;;;; counts are quoted in three handoffs and a benchmark is not a case. + +(defn step [a i64 b i64] i64 + (+ a b)) + +(defn main [] i32 + (let [acc (i64 0)] + (dotimes [i 20000000] + (set acc (step acc 1))) + (print acc) (println "")) + 0) diff --git a/spike/x86/bench/b2-bounds.flan b/spike/x86/bench/b2-bounds.flan new file mode 100644 index 0000000..62330cd --- /dev/null +++ b/spike/x86/bench/b2-bounds.flan @@ -0,0 +1,20 @@ +;;;; A hot loop that does nothing but index a bounds-checked array. +;;;; +;;;; The suspected cost is three frame temporaries per check. This one has an +;;;; A/B that the others do not: --no-bounds-checks builds the same program +;;;; with the check gone, on both sides, so the difference between the two +;;;; x86 numbers is the check and nothing else, and the same difference on +;;;; the LLVM side says what the check costs when a compiler is allowed to +;;;; hoist it out of the loop. + +(defvar xs [1024 i32]) + +(defn main [] i32 + (dotimes [i 1024] + (set (at xs i) i)) + (let [acc (i64 0)] + (dotimes [r 20000] + (dotimes [i 1024] + (set acc (+ acc (i64 (at xs i)))))) + (print acc) (println "")) + 0) diff --git a/spike/x86/bench/b3-spill.flan b/spike/x86/bench/b3-spill.flan new file mode 100644 index 0000000..d0bdd9f --- /dev/null +++ b/spike/x86/bench/b3-spill.flan @@ -0,0 +1,21 @@ +;;;; A hot loop of arithmetic and nothing else: no calls, no arrays, no +;;;; copies. +;;;; +;;;; The suspected cost is that every intermediate lives in a frame slot -- +;;;; this backend allocates no registers, so an expression tree becomes a +;;;; chain of stores and reloads. The tree here is deliberately deep and +;;;; entirely dependent, so a register allocator would keep all of it in +;;;; registers and this backend cannot keep any of it. + +(defn main [] i32 + (let [acc (i64 1)] + (dotimes [i 5000000] + (let [a (+ acc 3) + b (* a 2) + c (- b 1) + d (bit-xor c 7) + e (+ d (* a 5)) + f (- e (bit-and d 15))] + (set acc (+ (% f 1000003) 1)))) + (print acc) (println "")) + 0) diff --git a/spike/x86/bench/b4-copy.flan b/spike/x86/bench/b4-copy.flan new file mode 100644 index 0000000..200c245 --- /dev/null +++ b/spike/x86/bench/b4-copy.flan @@ -0,0 +1,20 @@ +;;;; A hot loop of struct copies. +;;;; +;;;; The suspected cost is `rep movsb`: this backend copies an aggregate by +;;;; block-moving bytes, where LLVM either keeps the thing in registers or +;;;; emits a handful of wide moves. Eight i64 fields is 64 bytes -- big +;;;; enough that a copy is a real copy, small enough that `rep movsb` is +;;;; paying its setup cost on every one of them, which is the shape the +;;;; instruction is worst at. + +(defstruct Big [a i64 b i64 c i64 d i64 e i64 f i64 g i64 h i64]) + +(defn main [] i32 + (let [acc (i64 0) + v (Big {.a 1 .b 2 .c 3 .d 4 .e 5 .f 6 .g 7 .h 8})] + (dotimes [i 2000000] + (let [w v] + (set (.a v) (+ (.h w) 1)) + (set acc (+ acc (.a w))))) + (print acc) (println "")) + 0) diff --git a/spike/x86/cost-bench.tsv b/spike/x86/cost-bench.tsv new file mode 100644 index 0000000..a231e38 --- /dev/null +++ b/spike/x86/cost-bench.tsv @@ -0,0 +1,5 @@ +name llvm_us o0_us x86_us llvm_nbc_us x86_nbc_us llvm_dev_us x86_dev_us llvm_own o0_own x86_own x86_dev_own +b1-calls 2108 36764 104266 1959 103336 26827 100874 82 200 807 83018 +b2-bounds 4711 27047 112567 4548 104951 4663 111295 355 407 1390 83596 +b3-spill 22184 28982 137347 22772 132053 22659 132913 168 248 1117 83323 +b4-copy 2043 7170 20792 1980 19559 2058 18497 140 348 877 83083 diff --git a/spike/x86/cost-corpus.tsv b/spike/x86/cost-corpus.tsv new file mode 100644 index 0000000..d8b36d4 --- /dev/null +++ b/spike/x86/cost-corpus.tsv @@ -0,0 +1,98 @@ +name llvm_file llvm_text llvm_own o0_own x86_file x86_text x86_own llvm_us x86_us +agent-longname 82224 42002 79 134 82264 42450 519 -1 -1 +agent-queue 82352 42306 340 567 86488 43698 1747 2491 2546 +agent 82336 42434 454 798 86472 44162 2218 2609 2496 +algorithms 76680 39282 6776 16616 101296 63234 30875 2499 2605 +allocators 67760 33650 1297 1719 71896 37490 5130 3170 3034 +array-ctor 67784 32722 357 1840 71920 36242 3880 2442 2664 +bounds-condition 76416 37074 4641 6963 84648 45858 13493 2673 2469 +bounds 67720 32738 363 1951 71856 36418 4039 3194 2654 +break 82296 42754 819 1182 86432 44482 2549 -1 -1 +bytes2 72120 37010 4576 10893 88544 52018 19654 2573 2510 +cleanup 68176 33986 1583 2077 72312 36946 4583 2351 2402 +conditions 68064 33586 1212 1209 68104 35346 2984 2436 2513 +debug-permuted 67720 32610 261 410 67760 33794 1433 2479 2442 +debug 67720 32610 261 429 67760 33794 1433 2494 2494 +defer-let 67928 34018 1625 1784 72064 36754 4396 2548 2543 +destructure 67792 33650 1290 4577 80120 40946 8583 2534 2523 +dev-break-bounds 82440 42546 588 648 86576 43762 1830 -1 -1 +dev-break 82368 42418 459 715 86504 43682 1755 -1 -1 +dev-globals 82456 42162 206 534 86584 43538 1614 -1 -1 +dev-inspect 82408 42610 652 704 86544 43698 1765 -1 -1 +dev-locals 82336 42418 465 623 86472 43650 1721 -1 -1 +dev-noagent 67720 32434 83 109 67760 32866 507 3116 2939 +dev-pause 82336 42066 112 244 82376 42802 877 -1 -1 +dev-ptr 86632 43954 1970 2859 90768 47490 5543 -1 -1 +dev-repl 82336 42066 112 244 82376 42802 877 -1 -1 +dev-robust 82336 42066 112 244 82376 42802 877 -1 -1 +edn 91208 44706 12057 34556 124016 80898 48532 3218 3165 +embed 67800 33330 977 2461 71936 37906 5550 2440 2650 +enum-compare 67752 32562 201 581 67792 33970 1611 2607 2805 +enum-convert 67752 33202 830 1522 71888 37042 4675 2737 2512 +error 67744 32498 131 163 67784 32946 591 2614 2587 +exhausted-unhandled 67688 33090 749 1156 67728 34850 2491 2442 2591 +exhausted 76192 35778 3415 5139 80336 42674 10321 2409 2469 +fn-values 68296 34226 1745 5124 80632 43026 10672 2405 2406 +format 76032 36802 4413 7076 84264 45394 13038 2449 2388 +frame-rollback 68440 34466 2020 3424 80768 39762 7402 2383 2394 +free-all-refused 67688 32370 28 28 67728 32658 299 2418 2497 +generics 81368 42786 10137 20427 114184 71602 39241 2474 2516 +handles 75880 35826 3482 6289 84112 46338 13981 2520 2470 +higher-order 76648 37154 4652 10329 88976 51314 18948 2413 2439 +into 80256 40658 8277 12345 92584 53682 21320 2428 2412 +loops 67736 33378 1016 1720 71872 37634 5269 2370 2315 +machine 67960 33186 821 1925 72096 37394 5042 2515 2456 +macro-unless 67720 32642 288 615 67760 34610 2243 2466 2457 +macros 67688 32802 461 793 67728 35250 2896 2532 2409 +map-exhausted 76152 36162 3805 5994 84392 44530 12171 2393 2506 +map-iter 76096 39586 7211 10374 92520 54082 21715 2541 2531 +map-stale-region 67688 33650 1307 2106 71824 36898 4537 2354 2407 +maps 88504 45266 12833 19850 117216 72802 40445 3162 3082 +math 67832 33986 1611 3288 72024 39362 7000 2384 2405 +math2 67784 33506 1148 2862 72024 39010 6644 2300 2482 +pkg-diamond 67880 32626 252 616 67928 34162 1801 2353 2227 +pkg-macro-idle 67688 32338 3 3 67728 32562 210 2440 2410 +pkg-macro 67800 32722 354 450 67840 33986 1626 2455 2429 +pkg-return 78416 39570 7024 23635 103032 64082 31716 2432 2375 +pkg-shadow 67848 32642 277 371 67888 33618 1258 2408 2236 +pkg-shared 74560 34066 1676 5593 86888 45650 13283 2422 2648 +pkg-unused 69704 32386 35 35 69744 34658 2298 2351 2271 +pool-stale-region 67688 33282 932 1675 71824 35746 3379 2389 2466 +printers 82464 42098 168 452 82504 43282 1348 -1 -1 +println 76256 36098 3737 7587 88584 50114 17753 2312 2460 +raylib-audio 79536 35522 2478 5610 87768 44178 11225 2574 2769 +raylib-ffi 85848 39538 6245 11506 102272 55650 22537 2607 2852 +raylib-font 79088 35538 2622 6260 83232 43186 10350 2661 2658 +raylib-image 80152 37698 4504 8285 88384 47058 13966 2684 2946 +raylib-imported 70344 33202 612 1003 74480 36658 4066 2688 2602 +reach-walk 67992 33170 774 1082 68032 34802 2443 2552 2308 +recur 67792 34594 2233 4070 76024 40674 8314 2468 62674 +registry 72008 35330 2940 4604 80240 41010 8608 2515 2329 +reload-generic 67944 32930 520 1390 67992 35442 3076 2341 2241 +restarts 77200 39010 6430 9268 89528 49442 17065 2356 2304 +rl-with 69704 32386 35 35 69744 34658 2298 2550 2461 +sand-headless 74728 34546 2139 6326 87056 47634 15269 2783 9172 +signedness 67688 32594 246 386 67728 33938 1576 2418 2411 +slice-from-ptr 67752 33106 755 2549 75984 37458 5101 2421 2367 +slices 68128 34802 2393 9057 88648 50114 17753 2504 2271 +slurp-unhandled 67688 33522 1173 1601 67728 35298 2945 2340 2354 +slurp 80288 42114 9744 14813 100808 57010 24656 2358 2408 +stale-region 67688 33490 1148 1681 71824 35842 3483 2332 2286 +string-of-bytes 67856 33490 981 1925 71992 36338 3817 2319 2353 +strings 88912 45890 13395 22681 109432 65618 33265 2455 2391 +text 72296 37122 4666 8007 84624 49378 17018 2310 2397 +unions 76016 36498 4131 17714 92440 55810 23456 2323 2413 +unit-main 67688 32386 33 33 67728 32674 307 2459 2377 +utf8 81216 39714 7170 21515 114024 73106 40745 2563 2380 +values 67720 32530 180 667 67760 34082 1719 2531 2378 +vec 80080 40402 8039 12089 92408 55122 22770 2370 2389 +virtual-controls-headless 70896 33666 1271 1295 75032 38962 6601 2319 3201 +web-files 67736 33058 706 1000 67776 34610 2246 2321 2382 +p1-exit 67688 32338 3 3 67728 32562 210 2436 2377 +p2-loop-print 67688 32626 283 106 67728 32930 577 2269 2364 +p3-fizz 67720 32530 169 225 67760 33282 928 2512 2359 +p4-convention 67824 32786 405 1005 67864 35138 2778 2436 2332 +p5-core 67792 33346 993 1507 71928 37218 4865 2346 2444 +p6-transfer 68096 34354 1944 2801 72232 37618 5252 2386 2310 +p7-slice-from-ptr 67840 33698 1336 1498 71984 35682 3318 2493 2392 +p8-cell 67752 32514 146 270 67800 33202 847 2402 2381 diff --git a/spike/x86/cost.sh b/spike/x86/cost.sh new file mode 100755 index 0000000..8bc0f35 --- /dev/null +++ b/spike/x86/cost.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# What does the hand-written backend cost, against LLVM, on the same programs? +# +# survey.sh answers "does it agree". This answers "what does agreeing cost", +# which is item 7 of HANDOFF-x86-rt.md and the one thing about this backend +# nobody had a number for. It builds each program the same two ways the +# survey does, and for each records three sizes and a time: +# +# file the whole executable on disk. Mostly runtime and libc glue, and +# the least interesting of the three -- it is here because it is +# the number anybody looks at first, and it should be visible how +# much of it is noise. +# text the .text section, from `size -A`. Still contains flan_rt.o, +# which is the same object on both sides. +# own the sum of the sizes of the defined symbols named `flan.` +# -- the program's *own* code and nothing else. The runtime's C is +# `flan_` with an underscore, so the two do not collide, and +# spot-checking a runtime symbol on both sides (flan_map_clone, +# 0x4b3 either way) says the runtime really is byte-identical and +# the difference in `own` is all codegen. +# +# The `own` column is the measurement; the other two are context. +# +# A fourth build, LLVM with --debug, is the reference that makes the number +# readable. --debug forces -O0, so it is LLVM's codegen with its optimiser +# switched off -- the closest thing available to what this backend is doing, +# which has no optimiser at all. Without it every ratio silently blames the +# backend for the whole of mem2reg and inlining. (--x86 --debug is refused, +# so the column exists on one side only, and that is the point of it.) +# +# Time is best-of-N, not a mean: a mean measures the other tenants of the +# machine. Even so, a corpus program is mostly process startup -- these are +# milliseconds -- so read the time column only where it is tens of +# milliseconds or more, and read the rest as size. +# +# Usage: spike/x86/cost.sh [name-substring ...] -> a TSV on stdout +# COST_FLAGS=--dev extra flags, given to both sides, as SURVEY_FLAGS is +# COST_REPS=5 timing repetitions +# COST_O0=0 skip the LLVM -O0 reference column +set -u +orig=$(pwd) +here=$(cd "$(dirname "$0")" && pwd) +root=$(cd "$here/../.." && pwd) +cd "$root" || exit 1 + +if [ -n "${FLAN:-}" ]; then + case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac +else + dune build --root . bin/main.exe 2>&1 | head -30 + flan=$root/_build/default/bin/main.exe +fi +test -x "$flan" || { echo "build failed" >&2; exit 1; } + +# Not mktemp under /tmp by default: this writes a few hundred executables of +# a megabyte or two, and a full /tmp on this machine has already frozen one +# session. The guard is cheap and a wedged run is not. +# A directory of this run's own, made with a plain mkdir so that a second +# copy of this script cannot land in the first one's: two runs sharing a +# scratch directory overwrite each other's `l` and `x` between the build and +# the timing, and the result is a row of numbers that belong to two different +# programs. That happened once here and the numbers looked entirely ordinary. +work=${COST_OUT:-${TMPDIR:-/tmp}/flan-cost} +mkdir -p "$work" || exit 1 +out=$work/run.$$ +mkdir "$out" || exit 1 +trap 'rm -rf "$out"' EXIT +free=$(df -Pk "$out" | awk 'NR==2 {print $4}') +[ "$free" -gt 2000000 ] || { echo "less than 2GB free at $out" >&2; exit 1; } + +forever="dev-loop dev-watch" +REPS=${COST_REPS:-5} +read -r -a extra <<<"${COST_FLAGS:-}" +o0=${COST_O0:-1} +[ -z "${COST_FLAGS:-}" ] || o0=0 + +# The sum of the defined text symbols the compiler itself named. `nm -S` +# prints value, size, type, name; a symbol with no size is not printed with +# four fields at all, which is why the guard is on NF. +own () { + nm --defined-only -S "$1" 2>/dev/null \ + | awk 'NF==4 && ($3=="T"||$3=="t") && $4 ~ /^flan\./ {n+=strtonum("0x"$2)} END{print n+0}' +} +text () { size -A "$1" 2>/dev/null | awk '$1==".text" {print $2}'; } + +# Best of REPS, in whole microseconds. A program that fails on one run and +# not another would make this meaningless, so the exit status of the first +# run is remembered and a run that disagrees with it poisons the row as -1. +# A program that hits the timeout is not timed at all: several of the corpus +# programs are agents or daemons that sit waiting for something that is not +# there, and five repetitions of a twenty-second wait, twice, is most of an +# afternoon spent measuring `timeout`. +best () { + exe=$1; min=; rc0= + for i in $(seq "$REPS"); do + t0=$(date +%s%N) + ( cd "$out" && timeout 20 "$exe" >/dev/null 2>&1 /dev/null 2>&1 || continue + # No main is a link failure, and it leaves nothing behind to measure. + test -x "$out/l" || continue + "$flan" build "$src" --x86 "${extra[@]}" -o "$out/x" >/dev/null 2>&1 || continue + test -x "$out/x" || continue + + d0=0 + if [ "$o0" = 1 ] && "$flan" build "$src" --debug -o "$out/d" >/dev/null 2>&1; then + d0=$(own "$out/d") + fi + + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$name" \ + "$(stat -c %s "$out/l")" "$(text "$out/l")" "$(own "$out/l")" "$d0" \ + "$(stat -c %s "$out/x")" "$(text "$out/x")" "$(own "$out/x")" \ + "$(best "$out/l")" "$(best "$out/x")" +done diff --git a/spike/x86/survey.sh b/spike/x86/survey.sh index 7f550db..0532ae6 100755 --- a/spike/x86/survey.sh +++ b/spike/x86/survey.sh @@ -32,14 +32,33 @@ # # Usage: spike/x86/survey.sh [name-substring ...] set -u +orig=$(pwd) here=$(cd "$(dirname "$0")" && pwd) root=$(cd "$here/../.." && pwd) cd "$root" || exit 1 -dune build --root . bin/main.exe 2>&1 | head -30 -flan=$root/_build/default/bin/main.exe +# FLAN is how the dune @x86 alias hands this script a compiler that dune has +# already built. Building it here instead would be a second dune inside the +# first one's lock, which does not run at all; and the alias has bin/main.exe +# in its deps precisely so it does not have to. Standalone -- the way the +# baseline in every handoff was measured -- nothing sets it and the build +# below is what it always was. +if [ -n "${FLAN:-}" ]; then + # Resolved against the directory this was invoked from, not against $root: + # dune spells its deps relative to the dune file, and the cd above has + # already happened by the time this is read. + case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac +else + dune build --root . bin/main.exe 2>&1 | head -30 + flan=$root/_build/default/bin/main.exe +fi test -x "$flan" || { echo "build failed"; exit 1; } +# Where the corpus is read from. Under dune the script runs from the build +# tree, where test/programs is present but spike/x86 is not, so the alias +# points this at the source tree and gets both. +corpus=${SURVEY_CORPUS:-$root} + out=$(mktemp -d); trap 'rm -rf "$out"' EXIT # The two that run until something stops them. Not a failure and not a match; @@ -58,7 +77,7 @@ read -r -a extra <<<"${SURVEY_FLAGS:-}" declare -a match=() differ=() refused=() nox86=() skip=() -for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do +for src in "$corpus"/test/programs/*.flan "$corpus"/spike/x86/*.flan; do name=$(basename "$src" .flan) if [ $# -gt 0 ]; then want=0 @@ -127,3 +146,19 @@ if [ "${#skip[@]}" != 0 ] && [ "${SURVEY_QUIET:-}" != 1 ]; then printf '%s\n' "${skip[@]}" | sed 's/^[^:]*://' | sort | uniq -c \ | sed 's/^/ /' fi + +# Strict mode, for the @x86 alias: the counts above are a report, and a report +# nobody reads is how two refusals from another lane's new primitive sat in +# the tree for a month. A DIFFER is a wrong answer and a refusal by name is a +# node this backend has stopped lowering; either is a failure. NOX86 and SKIP +# are not: the first is usually a toolchain that is not installed here, and +# the second is the frontend refusing the program on both sides. +if [ "${SURVEY_STRICT:-}" = 1 ]; then + if [ "${#differ[@]}" != 0 ] || [ "${#refused[@]}" != 0 ]; then + echo + echo "x86 survey FAILED: ${#differ[@]} differ, ${#refused[@]} refused" + exit 1 + fi + echo + echo "x86 survey ok: ${#match[@]} match" +fi diff --git a/test/dune b/test/dune index a511d57..6c7bf6c 100644 --- a/test/dune +++ b/test/dune @@ -169,3 +169,55 @@ (glob_files programs/pkgs/macring/*) (glob_files programs/pkgs/macspin/*)) (action (run ./test_valgrind.exe))) + +; The corpus a fourth time, through the hand-written x86-64 backend, compared +; against LLVM on what each program prints and what it exits with. Its own +; alias for the same reason the two above have one -- it builds every program +; twice and runs both, which is a couple of minutes against `dune test`'s +; seconds -- but the reason it exists at all is different. @sanitize and +; @valgrind ask whether the runtime is sound. This one asks whether the +; second backend still lowers the language: it refuses by name rather than +; miscompiling, so when another lane adds a primitive the backend says so +; loudly, and nothing was listening. Two such refusals sat in the tree for a +; month. Now they fail a build somebody can run. +; +; dune build --root . @x86 +; +; A rule with no executable beside it, unlike @sanitize and @valgrind: the +; check already exists as spike/x86/survey.sh, which is what every handoff +; quotes its counts from, and a second implementation in OCaml would be a +; second thing to drift. SURVEY_STRICT=1 turns its report into an exit +; status. FLAN is passed because the script otherwise runs `dune build` on +; the compiler, and a dune inside a dune action waits on a lock it cannot +; get; main.exe is in the deps instead. SURVEY_QUIET keeps the skip +; breakdown out of a passing build's log. +(rule + (alias x86) + (deps + (file %{workspace_root}/spike/x86/survey.sh) + (glob_files %{workspace_root}/spike/x86/*.flan) + (file %{workspace_root}/bin/main.exe) + (file %{workspace_root}/calc-me.flan) + (file %{workspace_root}/sand.flan) + (file %{workspace_root}/brush.png) + (glob_files %{workspace_root}/vendor/raylib/*) + (glob_files %{workspace_root}/vendor/agent/*) + (glob_files %{workspace_root}/vendor/edn/*) + (glob_files %{workspace_root}/examples/*) + (glob_files programs/*.flan) + (glob_files programs/assets/*) + ; A glob per package directory, because dune's glob does not descend. + (glob_files programs/pkgs/shape/*) + (glob_files programs/pkgs/area/*) + (glob_files programs/pkgs/draw/*) + (glob_files programs/pkgs/ring-a/*) + (glob_files programs/pkgs/ring-b/*) + (glob_files programs/pkgs/ring-c/*) + (glob_files programs/pkgs/mac/*) + (glob_files programs/pkgs/macring/*) + (glob_files programs/pkgs/macspin/*)) + (action + (setenv SURVEY_STRICT 1 + (setenv SURVEY_QUIET 1 + (setenv FLAN %{workspace_root}/bin/main.exe + (run bash %{workspace_root}/spike/x86/survey.sh))))))