Merge branch 'worktree-agent-ab7f6288f142b05df' into dev-loop
This commit is contained in:
commit
2b68262249
102
HANDOFF-x86-cost.md
Normal file
102
HANDOFF-x86-cost.md
Normal file
@ -0,0 +1,102 @@
|
||||
# Handoff — what the x86 backend costs, and keeping it from rotting
|
||||
|
||||
Branch: `dev-loop`. This lane is item 7 of `HANDOFF-x86-rt.md`'s "What remains" — code size and speed, measured
|
||||
rather than guessed — plus the closing note under it, which asks for the survey to run on its own so a refusal
|
||||
cannot sit unnoticed for a month again.
|
||||
|
||||
Nothing in `lib/` is touched. Another lane is rewriting `lib/x86.ml` at the same time, so everything here lives in
|
||||
`spike/x86` and in `test/dune`, and anything this lane finds that would want a compiler change is written down at
|
||||
the bottom rather than made.
|
||||
|
||||
## 1. The `@x86` alias — done, and it fails when it should
|
||||
|
||||
`test/dune` has a fourth alias beside `@sanitize` and `@valgrind`, shaped the same way and opt-in for the same
|
||||
reason: it builds every corpus program twice and runs both, which is minutes where `dune test` is forty seconds.
|
||||
|
||||
dune build --root . @x86
|
||||
|
||||
It is a `(rule ...)` with no `(executable ...)` beside it, which is where it differs from its two neighbours. The
|
||||
check already exists — `spike/x86/survey.sh` is what every handoff quotes its counts from — and a second
|
||||
implementation of it in OCaml would be a second thing to drift, which is exactly the failure this alias is meant
|
||||
to prevent. So the rule runs the script, and the script grew three environment variables to make that possible:
|
||||
|
||||
- `SURVEY_STRICT=1` turns the report into an exit status. A DIFFER is a wrong answer and a refusal by name is a
|
||||
node the backend has stopped lowering; either fails the build. NOX86 and SKIP do not — the first is usually a
|
||||
toolchain that is not installed on the machine, the second is the frontend refusing the program on both sides.
|
||||
- `FLAN=<path>` hands it a compiler somebody else built. Without it the script runs `dune build` on `bin/main.exe`,
|
||||
and a dune inside a dune action waits on a lock it will never get. The rule has `bin/main.exe` in its deps
|
||||
instead, exactly as the `(tests ...)` stanza already does.
|
||||
- `SURVEY_CORPUS` for where the programs are read from, which under dune is the build tree.
|
||||
|
||||
Standalone, with nothing set, the script is byte-for-byte the measurement it always was.
|
||||
|
||||
**Verified both directions.** `dune build --root . @x86` passes on a clean tree and prints the same
|
||||
**97 MATCH / 0 DIFFER / 0 REFUSED / 0 NOX86 / 36 SKIP** the script prints standalone — the build tree and the
|
||||
source tree agree about the corpus, which was not obvious in advance. Broken deliberately (a fake DIFFER injected
|
||||
into the script, the rule narrowed to one program so the round trip was thirty seconds) it fails the dune build
|
||||
with `x86 survey FAILED: 1 differ, 0 refused` and a nonzero status. Both edits reverted.
|
||||
|
||||
`dune test --root .` is still green and still about forty seconds. The new stanza is a rule on its own alias and
|
||||
touches no `(modules ...)` list, so it cannot be pulled into `@runtest` by accident. The two
|
||||
`clang: error: linker command failed` lines it prints are the `dev-robust` fixture doing its job and are called
|
||||
out in `HANDOFF-x86-rt.md` §4 already.
|
||||
|
||||
## 2. The measurement — `spike/x86/COST.md`
|
||||
|
||||
`COST.md` is the deliverable and it carries the numbers and the reading of them; the raw rows are committed beside
|
||||
it as `cost-corpus.tsv` and `cost-bench.tsv`. The headline, in one paragraph:
|
||||
|
||||
**This backend emits 3.84× the code LLVM does at `-O2`, and 1.92× what LLVM emits at `-O0`** — half the factor is
|
||||
the optimiser Flan ships with and not the backend. Per program the second ratio is tight: median 2.21, quartiles
|
||||
1.94 and 2.98. Speed is unmeasurable over the corpus (every program is 2.5ms of `execve`) except `recur`, which
|
||||
loops ten million times and is **6× LLVM `-O0`**; on purpose-written loops the backend is 2.8× to 4.7×.
|
||||
|
||||
Of the five suspected costs the old handoff named, the measurement says: the frame-slot round trip on every
|
||||
intermediate is most of everything and is the item worth fixing; `rep movsb` costs twenty cycles a copy and is
|
||||
worth fixing cheaply; the bounds check's three temporaries cost 241 bytes and *one cycle*, so they are a code-size
|
||||
item and not a speed item; the guard after every call is four instructions and invisible; and the dev build's
|
||||
extra load per call site is not measurable at all. What a dev build does cost — 400× the code, because nothing may
|
||||
be dropped when anything might be redefined — is `Reach`'s doing and both backends pay it.
|
||||
|
||||
The two scripts are how that was produced:
|
||||
|
||||
- `cost.sh` sweeps the corpus and prints a TSV: file size, `.text`, and **the sum of the `flan.*` defined symbols**,
|
||||
which is the program's own code with the runtime excluded. `COST_FLAGS=--dev` is the `SURVEY_FLAGS` precedent.
|
||||
- `bench.sh` runs four programs in `spike/x86/bench/`, each written so that one suspected cost is most of what the
|
||||
program does, because every program in `test/programs` runs in about two and a half milliseconds of which nearly
|
||||
all is `execve`.
|
||||
|
||||
They live in `spike/x86/bench/`, a subdirectory, *deliberately*: `survey.sh` globs `spike/x86/*.flan`, and a
|
||||
benchmark landing in the survey would move the 97 that three handoffs quote.
|
||||
|
||||
## 3. One trap, paid for once
|
||||
|
||||
Two copies of `cost.sh` sharing a scratch directory silently corrupt each other's numbers. The first release sweep
|
||||
was run, killed, and restarted; `pkill` missed a child, and the survivor kept writing the same `l` and `x` files
|
||||
the new run was building and timing. The result was a row for `math` carrying `dev-locals`' binaries — which is
|
||||
visible only because those two happened to collide *exactly*, and a collision between two programs of merely
|
||||
similar size would have produced a plausible row nobody would question.
|
||||
|
||||
`cost.sh` now makes a `run.$$` subdirectory with a plain `mkdir`, so a second copy cannot land in the first one's.
|
||||
The sweep behind `COST.md` was re-run clean afterwards. If you extend these scripts, keep that property.
|
||||
|
||||
## 4. Nothing was changed in `lib/`, and nothing needed to be
|
||||
|
||||
No finding here wants a compiler change that this lane was not allowed to make. The costs are all in
|
||||
`lib/x86.ml`'s lowering and the next lane rewriting that file will meet them; `COST.md`'s last section says which
|
||||
ones are worth its attention and which are not, which was the point of measuring rather than guessing.
|
||||
|
||||
## 5. What was not done
|
||||
|
||||
- **The `--dev` sweep over the whole corpus was not finished.** It was started, ran into the `dev-*` programs
|
||||
(each of which waits out a twenty-second timeout twice), and was killed to free the machine for the clean
|
||||
release re-run. The `--dev` numbers in `COST.md` come from `bench.sh`, which measures the same thing on four
|
||||
programs and measures it better, because a corpus program's dev cost is invisible under `execve`. Re-running it
|
||||
is `COST_FLAGS=--dev spike/x86/cost.sh` and about half an hour.
|
||||
- **No corpus program was disassembled.** The attribution in `COST.md` is all from the four benchmarks, which are
|
||||
small enough to read whole and were written so that each one is mostly a single suspected cost. The corpus
|
||||
outliers are explained from the `-O0` column instead — which is what that column is for, and in `bounds`' case
|
||||
it turns an alarming 11× into an ordinary 2.07×.
|
||||
- **`LLVM -O0` is only reachable through `--debug`**, which also asks for DWARF. Checked and harmless: DWARF lands
|
||||
in `.debug_*` and the metric sums `.text` symbols. A plain `-O0` flag on the CLI would be a `bin/main.ml` change
|
||||
and this lane was not touching compiler sources.
|
||||
210
spike/x86/COST.md
Normal file
210
spike/x86/COST.md
Normal file
@ -0,0 +1,210 @@
|
||||
# What the hand-written x86-64 backend costs
|
||||
|
||||
`survey.sh` has said for three handoffs that this backend agrees with LLVM on all 97 corpus programs it can build.
|
||||
Item 7 of `HANDOFF-x86-rt.md` is the other half of that sentence — nobody had a number for what the agreement
|
||||
costs — and it came with a list of suspects: a guard after every call, three frame temporaries per bounds check,
|
||||
every intermediate in memory, `rep movsb` block copies, and an extra load per call site in a dev build. This is
|
||||
the measurement. It does not change anything; two of the five suspects turn out not to matter, and the one that
|
||||
matters most is not on the list.
|
||||
|
||||
Produced by `spike/x86/cost.sh` (the corpus, for size) and `spike/x86/bench.sh` (four purpose-written programs,
|
||||
for speed). Both are documented in their own headers. Machine: 16-core x86-64, Fedora, clang as the assembler and
|
||||
linker on both sides, and other work running on it throughout — which is why every time below is the *minimum* of
|
||||
five or seven runs and why nothing here rests on a difference of a few percent.
|
||||
|
||||
The rows behind the tables are committed beside this file as `cost-corpus.tsv` and `cost-bench.tsv`, so a later
|
||||
lane can recompute a ratio rather than believe one.
|
||||
|
||||
## What is being compared, and against what
|
||||
|
||||
The interesting column is not the size of the executable. A Flan binary is mostly `flan_rt.o` and libc glue, the
|
||||
same object on both sides, and it drowns the signal: over the corpus the whole file is only **1.09×** bigger
|
||||
through this backend and `.text` only **1.18×**, which would be a reassuring number and a meaningless one.
|
||||
|
||||
So the measurement is the sum of the sizes of the defined symbols the compiler *named itself* — everything called
|
||||
`flan.<something>`. The runtime's C is `flan_<something>` with an underscore, so the two never collide, and a
|
||||
runtime symbol has the same size on both sides (`flan_map_clone`, `0x4b3` either way), which is the check that
|
||||
says the difference really is codegen and not a differently-linked runtime.
|
||||
|
||||
There is a third column, and it is what makes the second readable: **LLVM with `--debug`, which forces `-O0`**.
|
||||
This backend has no optimiser at all, so measuring it against LLVM at `-O2` charges it for the whole of mem2reg,
|
||||
inlining and constant folding. `-O0` is LLVM's instruction selection with none of that, which is the comparison
|
||||
that says something about *this* backend rather than about the absence of a middle end. (`--x86 --debug` is
|
||||
refused — the backend emits no DWARF — so the column exists on one side only. DWARF lands in `.debug_*` sections
|
||||
and not in `.text`, checked, so it does not contaminate the symbol sums.)
|
||||
|
||||
## The corpus: size
|
||||
|
||||
Ninety-seven programs — the same set `survey.sh` matches on, minus the two that run forever. Summed over all of
|
||||
them:
|
||||
|
||||
| | LLVM -O2 | LLVM -O0 | x86 |
|
||||
|---|---|---|---|
|
||||
| own code, all 97 programs | 221,608 | 444,504 | 851,638 |
|
||||
| against LLVM -O2 | 1.00× | 2.01× | **3.84×** |
|
||||
| against LLVM -O0 | | 1.00× | **1.92×** |
|
||||
|
||||
So the headline is two numbers, not one. **This backend emits 3.8× the code LLVM does at `-O2`, and half of that
|
||||
factor is the optimiser Flan ships with rather than anything about the backend; against LLVM with the optimiser
|
||||
off it is 1.9×.** Per program the second ratio is tight — median 2.21, quartiles 1.94 and 2.98, the whole range
|
||||
1.32 to 5.44 — which is itself a finding: the cost is not a few bad nodes, it is a constant tax on everything.
|
||||
|
||||
The ten largest programs, which are where the bytes actually are:
|
||||
|
||||
| program | LLVM -O2 | LLVM -O0 | x86 | x86 / -O2 | x86 / -O0 |
|
||||
|---|---|---|---|---|---|
|
||||
| `strings` | 13,395 | 22,681 | 33,265 | 2.48× | 1.47× |
|
||||
| `maps` | 12,833 | 19,850 | 40,445 | 3.15× | 2.04× |
|
||||
| `edn` | 12,057 | 34,556 | 48,532 | 4.03× | 1.40× |
|
||||
| `generics` | 10,137 | 20,427 | 39,241 | 3.87× | 1.92× |
|
||||
| `slurp` | 9,744 | 14,813 | 24,656 | 2.53× | 1.66× |
|
||||
| `into` | 8,277 | 12,345 | 21,320 | 2.58× | 1.73× |
|
||||
| `vec` | 8,039 | 12,089 | 22,770 | 2.83× | 1.88× |
|
||||
| `map-iter` | 7,211 | 10,374 | 21,715 | 3.01× | 2.09× |
|
||||
| `algorithms` | 6,776 | 16,616 | 30,875 | 4.56× | 1.86× |
|
||||
| `slices` | 2,393 | 9,057 | 17,753 | 7.42× | 1.96× |
|
||||
|
||||
And the two ends of the distribution, both of which are more interesting than the middle:
|
||||
|
||||
| program | LLVM -O2 | LLVM -O0 | x86 | x86 / -O2 | x86 / -O0 | why |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `bounds` | 363 | 1,951 | 4,039 | **11.13×** | 2.07× | LLVM at `-O2` proves the indices and deletes the checks |
|
||||
| `array-ctor` | 357 | 1,840 | 3,880 | 10.87× | 2.11× | the same, over a constructor's worth of stores |
|
||||
| `p2-loop-print` | 283 | 106 | 577 | 2.04× | **5.44×** | a `-O0` build *smaller* than `-O2`: LLVM unrolls the five-iteration loop and `-O0` does not |
|
||||
| `pkg-return` | 7,024 | 23,635 | 31,716 | 4.52× | 1.34× | mostly prelude, where `-O0` is already fat |
|
||||
|
||||
`bounds` is the clearest case in the table of why the `-O0` column had to exist. Eleven times is a shocking
|
||||
number and it is not about this backend at all: the program's whole point is indexing, LLVM at `-O2` can see the
|
||||
indices are in range and removes the check, and neither LLVM at `-O0` nor this backend can. Against the compiler
|
||||
that also keeps every check, `bounds` is 2.07× — a completely ordinary row.
|
||||
|
||||
## Where the size goes
|
||||
|
||||
Every one of these is from the disassembly of the benchmark programs, which are small enough to read whole.
|
||||
|
||||
**Every intermediate goes through the frame, and so does every constant.** This is the big one and it is not one
|
||||
feature, it is the shape of the whole backend. `(step acc 1)` in `b1-calls` compiles to:
|
||||
|
||||
movabs $0x1,%rax ; a 10-byte immediate ...
|
||||
mov %rax,-0x30(%rbp) ; ... stored to a frame slot ...
|
||||
mov -0x30(%rbp),%rsi ; ... and loaded back into the argument register
|
||||
|
||||
Three instructions and 24 bytes where LLVM writes `mov $1,%esi`, five. The loop bound gets the same treatment
|
||||
*every iteration* — `movabs $0x1312d00` into a slot, sign-extended out of it, compared — because nothing is
|
||||
hoisted. So does the loop condition: `cmp`/`setl`/`movzbq`/store a byte to the frame/reload it/`test`/`jne`,
|
||||
seven instructions for what is `cmp`/`jge` anywhere else. This is most of the 2× against `-O0` and essentially
|
||||
all of the difference on the programs at the bottom of the table, which have no calls, no bounds checks and no
|
||||
aggregates in them at all.
|
||||
|
||||
**The guard after every call is four instructions and one dependent load.**
|
||||
|
||||
call 4009f8 <flan.step>
|
||||
mov -0x18(%rbp),%r11 ; the condition frame, from its own slot
|
||||
mov 0x0(%r11),%r11 ; ... dereferenced
|
||||
test %r11,%r11
|
||||
jne <unwind>
|
||||
|
||||
Roughly 25 bytes per call site. Real, cheap, and third in size behind the two above it — on a program that is
|
||||
nothing *but* calls (`b1`) the whole backend is 2.8× LLVM `-O0`, and the guard is a minority of that.
|
||||
|
||||
**The bounds check is three frame temporaries, as suspected, and it costs code and not time.** The index is
|
||||
widened, stored, reloaded, stored again, the limit goes to a third slot, and then `cmp`/`jb` — with the failure
|
||||
path, its `.rodata` location string and its length, inline at the branch target. On `b2-bounds`, `flan.main` is
|
||||
`0x4b1` with checks and `0x3c0` without: **241 bytes, a quarter of the function.** In time it is 112.6ms against
|
||||
105.0ms over 20.5 million checked loads — **about 0.4ns, a cycle or two a check** — because the branch predicts
|
||||
perfectly and the loads were going to memory anyway. That contradicts the way the handoff's list reads. The three
|
||||
temporaries are a code-size item. They are not a speed item.
|
||||
|
||||
**`rep movsb` is real and it is the most expensive single instruction here.** A 64-byte struct copy lowers to
|
||||
`lea`/`lea`/`movabs $0x40,%rcx`/`rep movsb`, and `b4-copy` runs 2 million of them in 20.8ms against LLVM `-O0`'s
|
||||
7.2ms: **about 6.8ns of the difference per copy, some twenty cycles**, which is `rep movsb`'s startup cost and
|
||||
almost none of it the 64 bytes. It is also the one place where the backend loses to `-O0` by a factor (2.9×) it
|
||||
does not lose by on straight-line code, and the one item on the suspect list where a targeted fix — inline
|
||||
16-byte moves under some size threshold — would pay for itself.
|
||||
|
||||
## The corpus: speed, and why there is barely any
|
||||
|
||||
Almost nothing. **Every program in `test/programs` runs in about 2.5 milliseconds, nearly all of it `execve` and
|
||||
the dynamic loader**, and both backends produce the same 2.5 milliseconds. Best-of-five does not rescue a signal
|
||||
that is not there. There is exactly one corpus program whose own code is a measurable part of its runtime, and it
|
||||
is the right one:
|
||||
|
||||
| program | LLVM -O2 | LLVM -O0 | x86 | what it is |
|
||||
|---|---|---|---|---|
|
||||
| `recur` | ~1ms | 10ms | 60ms | a ten-million-iteration counting loop, written to prove `recur` is a jump |
|
||||
|
||||
Read that carefully, because the 25× against `-O2` is not a fact about this backend: LLVM folds the loop to its
|
||||
answer and runs nothing. Against `-O0`, which also runs ten million iterations, it is **6×** — about 6ns an
|
||||
iteration against 1ns, or roughly eighteen cycles for `i+1` and a compare. That is the frame-slot round trip
|
||||
above, four or five times over, and it is the honest number.
|
||||
|
||||
## The benchmarks
|
||||
|
||||
Four programs in `spike/x86/bench/`, each written so that one suspected cost is most of what the program does.
|
||||
They are in a subdirectory on purpose: `survey.sh` globs `spike/x86/*.flan` and a benchmark is not a case.
|
||||
Times are best-of-seven, in milliseconds.
|
||||
|
||||
| bench | what it is | LLVM -O2 | LLVM -O0 | x86 | x86 / -O0 | own code, -O0 → x86 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `b1-calls` | 20M calls of a one-instruction function | 2.1 | 36.8 | 104.3 | 2.8× | 200 → 807 |
|
||||
| `b2-bounds` | 20.5M bounds-checked array loads | 4.7 | 27.0 | 112.6 | 4.2× | 407 → 1390 |
|
||||
| `b3-spill` | 5M iterations of a six-deep arithmetic tree | 22.2 | 29.0 | 137.3 | 4.7× | 248 → 1117 |
|
||||
| `b4-copy` | 2M copies of a 64-byte struct | 2.0 | 7.2 | 20.8 | 2.9× | 348 → 877 |
|
||||
|
||||
`b2` with `--no-bounds-checks` on both sides: LLVM 4.5ms, x86 105.0ms — the 7.6ms the check costs over 20.5
|
||||
million of them, and the 241 bytes it costs in `flan.main`, are the whole of it.
|
||||
|
||||
`b3`'s ratio is the one to distrust slightly: its expression ends in a `%`, which is an `idiv`, and an `idiv` is
|
||||
twenty-odd cycles on every side. That is most of LLVM's own 22.2ms and a good part of its 29.0ms, so the
|
||||
denominator is largely a hardware latency this backend cannot do anything about. The absolute gap — 108ms over
|
||||
5 million iterations, about 21ns of extra work each — is the honest reading of that row.
|
||||
|
||||
|
||||
|
||||
`b1-calls` and `b3-spill` are the pair to read together, with the `idiv` caveat above in mind. `b1` is 20 million
|
||||
calls of a one-instruction function and lands at 2.8× `-O0`; `b3` has no calls at all and pays 21ns an iteration
|
||||
for six dependent arithmetic temporaries. **The backend is worse at arithmetic than it is at calling**, which is
|
||||
the opposite of what the suspect list implies, and it is because a call already costs enough that four extra
|
||||
instructions beside it disappear, while an add that should be one instruction costs five. `recur`, which is a
|
||||
counting loop and nothing else, says the same thing on a corpus program: 6× LLVM `-O0`.
|
||||
|
||||
The `-O2` column in `b1` and `b4` is 2ms — the loop is gone. That is a true fact about the toolchain Flan ships
|
||||
and a useless one about code generation, which is the whole reason the `-O0` column exists.
|
||||
|
||||
## `--dev`, which is the one axis both backends pay
|
||||
|
||||
`SURVEY_FLAGS=--dev` reported 97 MATCH for the lane before this one, so the comparison is available. The suspected cost was the extra
|
||||
load per call site — every cross-function call going through its indirection cell. **It is not measurable.** On
|
||||
`b1-calls`, 20 million calls, x86 release is 104.3ms and x86 `--dev` is 100.9ms: the same number, and the dev
|
||||
build is nominally the *faster* of the two, which is what a difference below the noise floor looks like. The load
|
||||
is from a `.data` cell that is in L1 after the first call and the machine was already waiting on the frame.
|
||||
|
||||
What a dev build actually costs is something else entirely, and both backends pay it. `b1-calls` is a program
|
||||
with two functions in it:
|
||||
|
||||
| build | own code |
|
||||
|---|---|
|
||||
| LLVM, release | 82 bytes |
|
||||
| x86, release | 807 bytes |
|
||||
| LLVM, `--dev` | 32,714 bytes |
|
||||
| x86, `--dev` | 83,018 bytes |
|
||||
|
||||
**A dev build emits the entire prelude**, because anything might be redefined and so nothing may be dropped. That
|
||||
is four hundred times the code for this program, and it dwarfs every item on the suspect list put together. It is
|
||||
also not a backend cost — LLVM pays a 400× of its own — so it is `Reach`'s business and not `x86.ml`'s. The
|
||||
backend's share of it is the same ~2.5× it charges everywhere else.
|
||||
|
||||
## What this says to the lane rewriting `lib/x86.ml`
|
||||
|
||||
Ranked by what the numbers actually support, and not by the order of the list in the handoff:
|
||||
|
||||
1. **Keep values in registers across a single expression.** Not a register allocator — just not routing every
|
||||
constant and every subexpression through a frame slot, and not re-materialising a loop bound every iteration.
|
||||
This is most of the 2× against `-O0` and most of `recur`'s 6×, and it is what `b3`'s 21ns an iteration buys.
|
||||
2. **Inline small aggregate copies** instead of `rep movsb`. One instruction, twenty cycles, on a copy that is
|
||||
four `movdqu` pairs.
|
||||
3. **`flan_dev_reg_note` in a release build** (item 2 of the old handoff's list) is worth doing and is small.
|
||||
4. **The call guard is fine.** Four instructions and 25 bytes, invisible in time. Leave it.
|
||||
5. **The bounds check is fine on time and fat on code.** If it is ever worth touching, it is worth touching for
|
||||
the 241 bytes — hoisting the failure path out of line would get most of that back without changing a cycle.
|
||||
6. **The dev call cell is free.** Whatever the redefinition emitter costs, it does not cost this.
|
||||
86
spike/x86/bench.sh
Executable file
86
spike/x86/bench.sh
Executable file
@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env bash
|
||||
# The speed half of cost.sh, on programs that are long enough to time.
|
||||
#
|
||||
# Why this exists beside cost.sh rather than inside it: every program in
|
||||
# test/programs runs in about two and a half milliseconds, of which nearly
|
||||
# all is fork, exec and the dynamic loader. Best-of-five does not rescue a
|
||||
# signal that is not there, and a table of 97 rows that all say "2.5ms vs
|
||||
# 2.6ms" would be a measurement of execve. So the corpus answers the size
|
||||
# question and these four answer the speed one, each written so that one
|
||||
# suspected cost is most of what the program does.
|
||||
#
|
||||
# Four builds of each, and the third column is the one to read:
|
||||
#
|
||||
# llvm as shipped, -O2. Frequently the loop is simply gone; that is a
|
||||
# true number about the toolchain and a useless one about codegen.
|
||||
# llvm -O0 via --debug, which forces it. LLVM's instruction selection with
|
||||
# its optimiser off -- the fair comparison for a backend that has
|
||||
# no optimiser.
|
||||
# x86 this backend.
|
||||
# x86 nbc --no-bounds-checks, for b2, where the difference is the check.
|
||||
#
|
||||
# And --dev on both sides, which is the one suspected cost the two backends
|
||||
# share: every cross-function call goes through an indirection cell, so it is
|
||||
# a load and an indirect call where a release build has a direct one. b1 is
|
||||
# where that has to show.
|
||||
#
|
||||
# Usage: spike/x86/bench.sh [name-substring ...]
|
||||
set -u
|
||||
orig=$(pwd)
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(cd "$here/../.." && pwd)
|
||||
cd "$root" || exit 1
|
||||
|
||||
if [ -n "${FLAN:-}" ]; then
|
||||
case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac
|
||||
else
|
||||
dune build --root . bin/main.exe 2>&1 | head -30
|
||||
flan=$root/_build/default/bin/main.exe
|
||||
fi
|
||||
test -x "$flan" || { echo "build failed" >&2; exit 1; }
|
||||
|
||||
out=${COST_OUT:-${TMPDIR:-/tmp}/flan-bench.$$}
|
||||
mkdir -p "$out" || exit 1
|
||||
trap 'rm -rf "$out"' EXIT
|
||||
|
||||
REPS=${COST_REPS:-5}
|
||||
|
||||
best () {
|
||||
min=
|
||||
for i in $(seq "$REPS"); do
|
||||
t0=$(date +%s%N)
|
||||
timeout 120 "$1" >/dev/null 2>&1 </dev/null
|
||||
t1=$(date +%s%N)
|
||||
d=$(( (t1 - t0) / 1000 ))
|
||||
if [ -z "$min" ] || [ "$d" -lt "$min" ]; then min=$d; fi
|
||||
done
|
||||
echo "$min"
|
||||
}
|
||||
|
||||
own () {
|
||||
nm --defined-only -S "$1" 2>/dev/null \
|
||||
| awk 'NF==4 && ($3=="T"||$3=="t") && $4 ~ /^flan\./ {n+=strtonum("0x"$2)} END{print n+0}'
|
||||
}
|
||||
|
||||
printf 'name\tllvm_us\to0_us\tx86_us\tllvm_nbc_us\tx86_nbc_us\tllvm_dev_us\tx86_dev_us\tllvm_own\to0_own\tx86_own\tx86_dev_own\n'
|
||||
|
||||
for src in "$root"/spike/x86/bench/*.flan; do
|
||||
name=$(basename "$src" .flan)
|
||||
if [ $# -gt 0 ]; then
|
||||
want=0
|
||||
for pat in "$@"; do case "$name" in *"$pat"*) want=1;; esac; done
|
||||
[ $want = 1 ] || continue
|
||||
fi
|
||||
"$flan" build "$src" -o "$out/l" >/dev/null 2>&1 || { echo "$name: llvm build failed" >&2; continue; }
|
||||
"$flan" build "$src" --debug -o "$out/d" >/dev/null 2>&1 || { echo "$name: -O0 build failed" >&2; continue; }
|
||||
"$flan" build "$src" --x86 -o "$out/x" >/dev/null 2>&1 || { echo "$name: x86 build failed" >&2; continue; }
|
||||
"$flan" build "$src" --no-bounds-checks -o "$out/ln" >/dev/null 2>&1
|
||||
"$flan" build "$src" --x86 --no-bounds-checks -o "$out/xn" >/dev/null 2>&1
|
||||
"$flan" build "$src" --dev -o "$out/lv" >/dev/null 2>&1
|
||||
"$flan" build "$src" --x86 --dev -o "$out/xv" >/dev/null 2>&1
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$name" \
|
||||
"$(best "$out/l")" "$(best "$out/d")" "$(best "$out/x")" \
|
||||
"$(best "$out/ln")" "$(best "$out/xn")" \
|
||||
"$(best "$out/lv")" "$(best "$out/xv")" \
|
||||
"$(own "$out/l")" "$(own "$out/d")" "$(own "$out/x")" "$(own "$out/xv")"
|
||||
done
|
||||
20
spike/x86/bench/b1-calls.flan
Normal file
20
spike/x86/bench/b1-calls.flan
Normal file
@ -0,0 +1,20 @@
|
||||
;;;; A hot loop that does nothing but call.
|
||||
;;;;
|
||||
;;;; The suspected cost is the guard this backend emits after every call --
|
||||
;;;; and, in a dev build, the load of the indirection cell before it. Neither
|
||||
;;;; is visible in the corpus, where a program's whole run is process startup.
|
||||
;;;; Here the call is the program: the body is one add, so whatever separates
|
||||
;;;; this from LLVM at -O0 is the call sequence and not the arithmetic.
|
||||
;;;;
|
||||
;;;; Not in spike/x86 proper, where survey.sh would pick it up: the survey's
|
||||
;;;; counts are quoted in three handoffs and a benchmark is not a case.
|
||||
|
||||
(defn step [a i64 b i64] i64
|
||||
(+ a b))
|
||||
|
||||
(defn main [] i32
|
||||
(let [acc (i64 0)]
|
||||
(dotimes [i 20000000]
|
||||
(set acc (step acc 1)))
|
||||
(print acc) (println ""))
|
||||
0)
|
||||
20
spike/x86/bench/b2-bounds.flan
Normal file
20
spike/x86/bench/b2-bounds.flan
Normal file
@ -0,0 +1,20 @@
|
||||
;;;; A hot loop that does nothing but index a bounds-checked array.
|
||||
;;;;
|
||||
;;;; The suspected cost is three frame temporaries per check. This one has an
|
||||
;;;; A/B that the others do not: --no-bounds-checks builds the same program
|
||||
;;;; with the check gone, on both sides, so the difference between the two
|
||||
;;;; x86 numbers is the check and nothing else, and the same difference on
|
||||
;;;; the LLVM side says what the check costs when a compiler is allowed to
|
||||
;;;; hoist it out of the loop.
|
||||
|
||||
(defvar xs [1024 i32])
|
||||
|
||||
(defn main [] i32
|
||||
(dotimes [i 1024]
|
||||
(set (at xs i) i))
|
||||
(let [acc (i64 0)]
|
||||
(dotimes [r 20000]
|
||||
(dotimes [i 1024]
|
||||
(set acc (+ acc (i64 (at xs i))))))
|
||||
(print acc) (println ""))
|
||||
0)
|
||||
21
spike/x86/bench/b3-spill.flan
Normal file
21
spike/x86/bench/b3-spill.flan
Normal file
@ -0,0 +1,21 @@
|
||||
;;;; A hot loop of arithmetic and nothing else: no calls, no arrays, no
|
||||
;;;; copies.
|
||||
;;;;
|
||||
;;;; The suspected cost is that every intermediate lives in a frame slot --
|
||||
;;;; this backend allocates no registers, so an expression tree becomes a
|
||||
;;;; chain of stores and reloads. The tree here is deliberately deep and
|
||||
;;;; entirely dependent, so a register allocator would keep all of it in
|
||||
;;;; registers and this backend cannot keep any of it.
|
||||
|
||||
(defn main [] i32
|
||||
(let [acc (i64 1)]
|
||||
(dotimes [i 5000000]
|
||||
(let [a (+ acc 3)
|
||||
b (* a 2)
|
||||
c (- b 1)
|
||||
d (bit-xor c 7)
|
||||
e (+ d (* a 5))
|
||||
f (- e (bit-and d 15))]
|
||||
(set acc (+ (% f 1000003) 1))))
|
||||
(print acc) (println ""))
|
||||
0)
|
||||
20
spike/x86/bench/b4-copy.flan
Normal file
20
spike/x86/bench/b4-copy.flan
Normal file
@ -0,0 +1,20 @@
|
||||
;;;; A hot loop of struct copies.
|
||||
;;;;
|
||||
;;;; The suspected cost is `rep movsb`: this backend copies an aggregate by
|
||||
;;;; block-moving bytes, where LLVM either keeps the thing in registers or
|
||||
;;;; emits a handful of wide moves. Eight i64 fields is 64 bytes -- big
|
||||
;;;; enough that a copy is a real copy, small enough that `rep movsb` is
|
||||
;;;; paying its setup cost on every one of them, which is the shape the
|
||||
;;;; instruction is worst at.
|
||||
|
||||
(defstruct Big [a i64 b i64 c i64 d i64 e i64 f i64 g i64 h i64])
|
||||
|
||||
(defn main [] i32
|
||||
(let [acc (i64 0)
|
||||
v (Big {.a 1 .b 2 .c 3 .d 4 .e 5 .f 6 .g 7 .h 8})]
|
||||
(dotimes [i 2000000]
|
||||
(let [w v]
|
||||
(set (.a v) (+ (.h w) 1))
|
||||
(set acc (+ acc (.a w)))))
|
||||
(print acc) (println ""))
|
||||
0)
|
||||
5
spike/x86/cost-bench.tsv
Normal file
5
spike/x86/cost-bench.tsv
Normal file
@ -0,0 +1,5 @@
|
||||
name llvm_us o0_us x86_us llvm_nbc_us x86_nbc_us llvm_dev_us x86_dev_us llvm_own o0_own x86_own x86_dev_own
|
||||
b1-calls 2108 36764 104266 1959 103336 26827 100874 82 200 807 83018
|
||||
b2-bounds 4711 27047 112567 4548 104951 4663 111295 355 407 1390 83596
|
||||
b3-spill 22184 28982 137347 22772 132053 22659 132913 168 248 1117 83323
|
||||
b4-copy 2043 7170 20792 1980 19559 2058 18497 140 348 877 83083
|
||||
|
98
spike/x86/cost-corpus.tsv
Normal file
98
spike/x86/cost-corpus.tsv
Normal file
@ -0,0 +1,98 @@
|
||||
name llvm_file llvm_text llvm_own o0_own x86_file x86_text x86_own llvm_us x86_us
|
||||
agent-longname 82224 42002 79 134 82264 42450 519 -1 -1
|
||||
agent-queue 82352 42306 340 567 86488 43698 1747 2491 2546
|
||||
agent 82336 42434 454 798 86472 44162 2218 2609 2496
|
||||
algorithms 76680 39282 6776 16616 101296 63234 30875 2499 2605
|
||||
allocators 67760 33650 1297 1719 71896 37490 5130 3170 3034
|
||||
array-ctor 67784 32722 357 1840 71920 36242 3880 2442 2664
|
||||
bounds-condition 76416 37074 4641 6963 84648 45858 13493 2673 2469
|
||||
bounds 67720 32738 363 1951 71856 36418 4039 3194 2654
|
||||
break 82296 42754 819 1182 86432 44482 2549 -1 -1
|
||||
bytes2 72120 37010 4576 10893 88544 52018 19654 2573 2510
|
||||
cleanup 68176 33986 1583 2077 72312 36946 4583 2351 2402
|
||||
conditions 68064 33586 1212 1209 68104 35346 2984 2436 2513
|
||||
debug-permuted 67720 32610 261 410 67760 33794 1433 2479 2442
|
||||
debug 67720 32610 261 429 67760 33794 1433 2494 2494
|
||||
defer-let 67928 34018 1625 1784 72064 36754 4396 2548 2543
|
||||
destructure 67792 33650 1290 4577 80120 40946 8583 2534 2523
|
||||
dev-break-bounds 82440 42546 588 648 86576 43762 1830 -1 -1
|
||||
dev-break 82368 42418 459 715 86504 43682 1755 -1 -1
|
||||
dev-globals 82456 42162 206 534 86584 43538 1614 -1 -1
|
||||
dev-inspect 82408 42610 652 704 86544 43698 1765 -1 -1
|
||||
dev-locals 82336 42418 465 623 86472 43650 1721 -1 -1
|
||||
dev-noagent 67720 32434 83 109 67760 32866 507 3116 2939
|
||||
dev-pause 82336 42066 112 244 82376 42802 877 -1 -1
|
||||
dev-ptr 86632 43954 1970 2859 90768 47490 5543 -1 -1
|
||||
dev-repl 82336 42066 112 244 82376 42802 877 -1 -1
|
||||
dev-robust 82336 42066 112 244 82376 42802 877 -1 -1
|
||||
edn 91208 44706 12057 34556 124016 80898 48532 3218 3165
|
||||
embed 67800 33330 977 2461 71936 37906 5550 2440 2650
|
||||
enum-compare 67752 32562 201 581 67792 33970 1611 2607 2805
|
||||
enum-convert 67752 33202 830 1522 71888 37042 4675 2737 2512
|
||||
error 67744 32498 131 163 67784 32946 591 2614 2587
|
||||
exhausted-unhandled 67688 33090 749 1156 67728 34850 2491 2442 2591
|
||||
exhausted 76192 35778 3415 5139 80336 42674 10321 2409 2469
|
||||
fn-values 68296 34226 1745 5124 80632 43026 10672 2405 2406
|
||||
format 76032 36802 4413 7076 84264 45394 13038 2449 2388
|
||||
frame-rollback 68440 34466 2020 3424 80768 39762 7402 2383 2394
|
||||
free-all-refused 67688 32370 28 28 67728 32658 299 2418 2497
|
||||
generics 81368 42786 10137 20427 114184 71602 39241 2474 2516
|
||||
handles 75880 35826 3482 6289 84112 46338 13981 2520 2470
|
||||
higher-order 76648 37154 4652 10329 88976 51314 18948 2413 2439
|
||||
into 80256 40658 8277 12345 92584 53682 21320 2428 2412
|
||||
loops 67736 33378 1016 1720 71872 37634 5269 2370 2315
|
||||
machine 67960 33186 821 1925 72096 37394 5042 2515 2456
|
||||
macro-unless 67720 32642 288 615 67760 34610 2243 2466 2457
|
||||
macros 67688 32802 461 793 67728 35250 2896 2532 2409
|
||||
map-exhausted 76152 36162 3805 5994 84392 44530 12171 2393 2506
|
||||
map-iter 76096 39586 7211 10374 92520 54082 21715 2541 2531
|
||||
map-stale-region 67688 33650 1307 2106 71824 36898 4537 2354 2407
|
||||
maps 88504 45266 12833 19850 117216 72802 40445 3162 3082
|
||||
math 67832 33986 1611 3288 72024 39362 7000 2384 2405
|
||||
math2 67784 33506 1148 2862 72024 39010 6644 2300 2482
|
||||
pkg-diamond 67880 32626 252 616 67928 34162 1801 2353 2227
|
||||
pkg-macro-idle 67688 32338 3 3 67728 32562 210 2440 2410
|
||||
pkg-macro 67800 32722 354 450 67840 33986 1626 2455 2429
|
||||
pkg-return 78416 39570 7024 23635 103032 64082 31716 2432 2375
|
||||
pkg-shadow 67848 32642 277 371 67888 33618 1258 2408 2236
|
||||
pkg-shared 74560 34066 1676 5593 86888 45650 13283 2422 2648
|
||||
pkg-unused 69704 32386 35 35 69744 34658 2298 2351 2271
|
||||
pool-stale-region 67688 33282 932 1675 71824 35746 3379 2389 2466
|
||||
printers 82464 42098 168 452 82504 43282 1348 -1 -1
|
||||
println 76256 36098 3737 7587 88584 50114 17753 2312 2460
|
||||
raylib-audio 79536 35522 2478 5610 87768 44178 11225 2574 2769
|
||||
raylib-ffi 85848 39538 6245 11506 102272 55650 22537 2607 2852
|
||||
raylib-font 79088 35538 2622 6260 83232 43186 10350 2661 2658
|
||||
raylib-image 80152 37698 4504 8285 88384 47058 13966 2684 2946
|
||||
raylib-imported 70344 33202 612 1003 74480 36658 4066 2688 2602
|
||||
reach-walk 67992 33170 774 1082 68032 34802 2443 2552 2308
|
||||
recur 67792 34594 2233 4070 76024 40674 8314 2468 62674
|
||||
registry 72008 35330 2940 4604 80240 41010 8608 2515 2329
|
||||
reload-generic 67944 32930 520 1390 67992 35442 3076 2341 2241
|
||||
restarts 77200 39010 6430 9268 89528 49442 17065 2356 2304
|
||||
rl-with 69704 32386 35 35 69744 34658 2298 2550 2461
|
||||
sand-headless 74728 34546 2139 6326 87056 47634 15269 2783 9172
|
||||
signedness 67688 32594 246 386 67728 33938 1576 2418 2411
|
||||
slice-from-ptr 67752 33106 755 2549 75984 37458 5101 2421 2367
|
||||
slices 68128 34802 2393 9057 88648 50114 17753 2504 2271
|
||||
slurp-unhandled 67688 33522 1173 1601 67728 35298 2945 2340 2354
|
||||
slurp 80288 42114 9744 14813 100808 57010 24656 2358 2408
|
||||
stale-region 67688 33490 1148 1681 71824 35842 3483 2332 2286
|
||||
string-of-bytes 67856 33490 981 1925 71992 36338 3817 2319 2353
|
||||
strings 88912 45890 13395 22681 109432 65618 33265 2455 2391
|
||||
text 72296 37122 4666 8007 84624 49378 17018 2310 2397
|
||||
unions 76016 36498 4131 17714 92440 55810 23456 2323 2413
|
||||
unit-main 67688 32386 33 33 67728 32674 307 2459 2377
|
||||
utf8 81216 39714 7170 21515 114024 73106 40745 2563 2380
|
||||
values 67720 32530 180 667 67760 34082 1719 2531 2378
|
||||
vec 80080 40402 8039 12089 92408 55122 22770 2370 2389
|
||||
virtual-controls-headless 70896 33666 1271 1295 75032 38962 6601 2319 3201
|
||||
web-files 67736 33058 706 1000 67776 34610 2246 2321 2382
|
||||
p1-exit 67688 32338 3 3 67728 32562 210 2436 2377
|
||||
p2-loop-print 67688 32626 283 106 67728 32930 577 2269 2364
|
||||
p3-fizz 67720 32530 169 225 67760 33282 928 2512 2359
|
||||
p4-convention 67824 32786 405 1005 67864 35138 2778 2436 2332
|
||||
p5-core 67792 33346 993 1507 71928 37218 4865 2346 2444
|
||||
p6-transfer 68096 34354 1944 2801 72232 37618 5252 2386 2310
|
||||
p7-slice-from-ptr 67840 33698 1336 1498 71984 35682 3318 2493 2392
|
||||
p8-cell 67752 32514 146 270 67800 33202 847 2402 2381
|
||||
|
135
spike/x86/cost.sh
Executable file
135
spike/x86/cost.sh
Executable file
@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bash
|
||||
# What does the hand-written backend cost, against LLVM, on the same programs?
|
||||
#
|
||||
# survey.sh answers "does it agree". This answers "what does agreeing cost",
|
||||
# which is item 7 of HANDOFF-x86-rt.md and the one thing about this backend
|
||||
# nobody had a number for. It builds each program the same two ways the
|
||||
# survey does, and for each records three sizes and a time:
|
||||
#
|
||||
# file the whole executable on disk. Mostly runtime and libc glue, and
|
||||
# the least interesting of the three -- it is here because it is
|
||||
# the number anybody looks at first, and it should be visible how
|
||||
# much of it is noise.
|
||||
# text the .text section, from `size -A`. Still contains flan_rt.o,
|
||||
# which is the same object on both sides.
|
||||
# own the sum of the sizes of the defined symbols named `flan.<name>`
|
||||
# -- the program's *own* code and nothing else. The runtime's C is
|
||||
# `flan_<name>` with an underscore, so the two do not collide, and
|
||||
# spot-checking a runtime symbol on both sides (flan_map_clone,
|
||||
# 0x4b3 either way) says the runtime really is byte-identical and
|
||||
# the difference in `own` is all codegen.
|
||||
#
|
||||
# The `own` column is the measurement; the other two are context.
|
||||
#
|
||||
# A fourth build, LLVM with --debug, is the reference that makes the number
|
||||
# readable. --debug forces -O0, so it is LLVM's codegen with its optimiser
|
||||
# switched off -- the closest thing available to what this backend is doing,
|
||||
# which has no optimiser at all. Without it every ratio silently blames the
|
||||
# backend for the whole of mem2reg and inlining. (--x86 --debug is refused,
|
||||
# so the column exists on one side only, and that is the point of it.)
|
||||
#
|
||||
# Time is best-of-N, not a mean: a mean measures the other tenants of the
|
||||
# machine. Even so, a corpus program is mostly process startup -- these are
|
||||
# milliseconds -- so read the time column only where it is tens of
|
||||
# milliseconds or more, and read the rest as size.
|
||||
#
|
||||
# Usage: spike/x86/cost.sh [name-substring ...] -> a TSV on stdout
|
||||
# COST_FLAGS=--dev extra flags, given to both sides, as SURVEY_FLAGS is
|
||||
# COST_REPS=5 timing repetitions
|
||||
# COST_O0=0 skip the LLVM -O0 reference column
|
||||
set -u
|
||||
orig=$(pwd)
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(cd "$here/../.." && pwd)
|
||||
cd "$root" || exit 1
|
||||
|
||||
if [ -n "${FLAN:-}" ]; then
|
||||
case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac
|
||||
else
|
||||
dune build --root . bin/main.exe 2>&1 | head -30
|
||||
flan=$root/_build/default/bin/main.exe
|
||||
fi
|
||||
test -x "$flan" || { echo "build failed" >&2; exit 1; }
|
||||
|
||||
# Not mktemp under /tmp by default: this writes a few hundred executables of
|
||||
# a megabyte or two, and a full /tmp on this machine has already frozen one
|
||||
# session. The guard is cheap and a wedged run is not.
|
||||
# A directory of this run's own, made with a plain mkdir so that a second
|
||||
# copy of this script cannot land in the first one's: two runs sharing a
|
||||
# scratch directory overwrite each other's `l` and `x` between the build and
|
||||
# the timing, and the result is a row of numbers that belong to two different
|
||||
# programs. That happened once here and the numbers looked entirely ordinary.
|
||||
work=${COST_OUT:-${TMPDIR:-/tmp}/flan-cost}
|
||||
mkdir -p "$work" || exit 1
|
||||
out=$work/run.$$
|
||||
mkdir "$out" || exit 1
|
||||
trap 'rm -rf "$out"' EXIT
|
||||
free=$(df -Pk "$out" | awk 'NR==2 {print $4}')
|
||||
[ "$free" -gt 2000000 ] || { echo "less than 2GB free at $out" >&2; exit 1; }
|
||||
|
||||
forever="dev-loop dev-watch"
|
||||
REPS=${COST_REPS:-5}
|
||||
read -r -a extra <<<"${COST_FLAGS:-}"
|
||||
o0=${COST_O0:-1}
|
||||
[ -z "${COST_FLAGS:-}" ] || o0=0
|
||||
|
||||
# The sum of the defined text symbols the compiler itself named. `nm -S`
|
||||
# prints value, size, type, name; a symbol with no size is not printed with
|
||||
# four fields at all, which is why the guard is on NF.
|
||||
own () {
|
||||
nm --defined-only -S "$1" 2>/dev/null \
|
||||
| awk 'NF==4 && ($3=="T"||$3=="t") && $4 ~ /^flan\./ {n+=strtonum("0x"$2)} END{print n+0}'
|
||||
}
|
||||
text () { size -A "$1" 2>/dev/null | awk '$1==".text" {print $2}'; }
|
||||
|
||||
# Best of REPS, in whole microseconds. A program that fails on one run and
|
||||
# not another would make this meaningless, so the exit status of the first
|
||||
# run is remembered and a run that disagrees with it poisons the row as -1.
|
||||
# A program that hits the timeout is not timed at all: several of the corpus
|
||||
# programs are agents or daemons that sit waiting for something that is not
|
||||
# there, and five repetitions of a twenty-second wait, twice, is most of an
|
||||
# afternoon spent measuring `timeout`.
|
||||
best () {
|
||||
exe=$1; min=; rc0=
|
||||
for i in $(seq "$REPS"); do
|
||||
t0=$(date +%s%N)
|
||||
( cd "$out" && timeout 20 "$exe" >/dev/null 2>&1 </dev/null )
|
||||
rc=$?
|
||||
t1=$(date +%s%N)
|
||||
[ -n "$rc0" ] || rc0=$rc
|
||||
[ "$rc" = "$rc0" ] || { echo "-1"; return; }
|
||||
d=$(( (t1 - t0) / 1000 ))
|
||||
[ "$d" -lt 15000000 ] || { echo "-1"; return; }
|
||||
if [ -z "$min" ] || [ "$d" -lt "$min" ]; then min=$d; fi
|
||||
done
|
||||
echo "$min"
|
||||
}
|
||||
|
||||
printf 'name\tllvm_file\tllvm_text\tllvm_own\to0_own\tx86_file\tx86_text\tx86_own\tllvm_us\tx86_us\n'
|
||||
|
||||
for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do
|
||||
name=$(basename "$src" .flan)
|
||||
if [ $# -gt 0 ]; then
|
||||
want=0
|
||||
for pat in "$@"; do case "$name" in *"$pat"*) want=1;; esac; done
|
||||
[ $want = 1 ] || continue
|
||||
fi
|
||||
case " $forever " in *" $name "*) continue;; esac
|
||||
|
||||
rm -f "$out/l" "$out/x" "$out/d"
|
||||
"$flan" build "$src" "${extra[@]}" -o "$out/l" >/dev/null 2>&1 || continue
|
||||
# No main is a link failure, and it leaves nothing behind to measure.
|
||||
test -x "$out/l" || continue
|
||||
"$flan" build "$src" --x86 "${extra[@]}" -o "$out/x" >/dev/null 2>&1 || continue
|
||||
test -x "$out/x" || continue
|
||||
|
||||
d0=0
|
||||
if [ "$o0" = 1 ] && "$flan" build "$src" --debug -o "$out/d" >/dev/null 2>&1; then
|
||||
d0=$(own "$out/d")
|
||||
fi
|
||||
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$name" \
|
||||
"$(stat -c %s "$out/l")" "$(text "$out/l")" "$(own "$out/l")" "$d0" \
|
||||
"$(stat -c %s "$out/x")" "$(text "$out/x")" "$(own "$out/x")" \
|
||||
"$(best "$out/l")" "$(best "$out/x")"
|
||||
done
|
||||
@ -32,14 +32,33 @@
|
||||
#
|
||||
# Usage: spike/x86/survey.sh [name-substring ...]
|
||||
set -u
|
||||
orig=$(pwd)
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(cd "$here/../.." && pwd)
|
||||
cd "$root" || exit 1
|
||||
|
||||
dune build --root . bin/main.exe 2>&1 | head -30
|
||||
flan=$root/_build/default/bin/main.exe
|
||||
# FLAN is how the dune @x86 alias hands this script a compiler that dune has
|
||||
# already built. Building it here instead would be a second dune inside the
|
||||
# first one's lock, which does not run at all; and the alias has bin/main.exe
|
||||
# in its deps precisely so it does not have to. Standalone -- the way the
|
||||
# baseline in every handoff was measured -- nothing sets it and the build
|
||||
# below is what it always was.
|
||||
if [ -n "${FLAN:-}" ]; then
|
||||
# Resolved against the directory this was invoked from, not against $root:
|
||||
# dune spells its deps relative to the dune file, and the cd above has
|
||||
# already happened by the time this is read.
|
||||
case $FLAN in /*) flan=$FLAN;; *) flan=$orig/$FLAN;; esac
|
||||
else
|
||||
dune build --root . bin/main.exe 2>&1 | head -30
|
||||
flan=$root/_build/default/bin/main.exe
|
||||
fi
|
||||
test -x "$flan" || { echo "build failed"; exit 1; }
|
||||
|
||||
# Where the corpus is read from. Under dune the script runs from the build
|
||||
# tree, where test/programs is present but spike/x86 is not, so the alias
|
||||
# points this at the source tree and gets both.
|
||||
corpus=${SURVEY_CORPUS:-$root}
|
||||
|
||||
out=$(mktemp -d); trap 'rm -rf "$out"' EXIT
|
||||
|
||||
# The two that run until something stops them. Not a failure and not a match;
|
||||
@ -58,7 +77,7 @@ read -r -a extra <<<"${SURVEY_FLAGS:-}"
|
||||
|
||||
declare -a match=() differ=() refused=() nox86=() skip=()
|
||||
|
||||
for src in "$root"/test/programs/*.flan "$root"/spike/x86/*.flan; do
|
||||
for src in "$corpus"/test/programs/*.flan "$corpus"/spike/x86/*.flan; do
|
||||
name=$(basename "$src" .flan)
|
||||
if [ $# -gt 0 ]; then
|
||||
want=0
|
||||
@ -127,3 +146,19 @@ if [ "${#skip[@]}" != 0 ] && [ "${SURVEY_QUIET:-}" != 1 ]; then
|
||||
printf '%s\n' "${skip[@]}" | sed 's/^[^:]*://' | sort | uniq -c \
|
||||
| sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# Strict mode, for the @x86 alias: the counts above are a report, and a report
|
||||
# nobody reads is how two refusals from another lane's new primitive sat in
|
||||
# the tree for a month. A DIFFER is a wrong answer and a refusal by name is a
|
||||
# node this backend has stopped lowering; either is a failure. NOX86 and SKIP
|
||||
# are not: the first is usually a toolchain that is not installed here, and
|
||||
# the second is the frontend refusing the program on both sides.
|
||||
if [ "${SURVEY_STRICT:-}" = 1 ]; then
|
||||
if [ "${#differ[@]}" != 0 ] || [ "${#refused[@]}" != 0 ]; then
|
||||
echo
|
||||
echo "x86 survey FAILED: ${#differ[@]} differ, ${#refused[@]} refused"
|
||||
exit 1
|
||||
fi
|
||||
echo
|
||||
echo "x86 survey ok: ${#match[@]} match"
|
||||
fi
|
||||
|
||||
52
test/dune
52
test/dune
@ -169,3 +169,55 @@
|
||||
(glob_files programs/pkgs/macring/*)
|
||||
(glob_files programs/pkgs/macspin/*))
|
||||
(action (run ./test_valgrind.exe)))
|
||||
|
||||
; The corpus a fourth time, through the hand-written x86-64 backend, compared
|
||||
; against LLVM on what each program prints and what it exits with. Its own
|
||||
; alias for the same reason the two above have one -- it builds every program
|
||||
; twice and runs both, which is a couple of minutes against `dune test`'s
|
||||
; seconds -- but the reason it exists at all is different. @sanitize and
|
||||
; @valgrind ask whether the runtime is sound. This one asks whether the
|
||||
; second backend still lowers the language: it refuses by name rather than
|
||||
; miscompiling, so when another lane adds a primitive the backend says so
|
||||
; loudly, and nothing was listening. Two such refusals sat in the tree for a
|
||||
; month. Now they fail a build somebody can run.
|
||||
;
|
||||
; dune build --root . @x86
|
||||
;
|
||||
; A rule with no executable beside it, unlike @sanitize and @valgrind: the
|
||||
; check already exists as spike/x86/survey.sh, which is what every handoff
|
||||
; quotes its counts from, and a second implementation in OCaml would be a
|
||||
; second thing to drift. SURVEY_STRICT=1 turns its report into an exit
|
||||
; status. FLAN is passed because the script otherwise runs `dune build` on
|
||||
; the compiler, and a dune inside a dune action waits on a lock it cannot
|
||||
; get; main.exe is in the deps instead. SURVEY_QUIET keeps the skip
|
||||
; breakdown out of a passing build's log.
|
||||
(rule
|
||||
(alias x86)
|
||||
(deps
|
||||
(file %{workspace_root}/spike/x86/survey.sh)
|
||||
(glob_files %{workspace_root}/spike/x86/*.flan)
|
||||
(file %{workspace_root}/bin/main.exe)
|
||||
(file %{workspace_root}/calc-me.flan)
|
||||
(file %{workspace_root}/sand.flan)
|
||||
(file %{workspace_root}/brush.png)
|
||||
(glob_files %{workspace_root}/vendor/raylib/*)
|
||||
(glob_files %{workspace_root}/vendor/agent/*)
|
||||
(glob_files %{workspace_root}/vendor/edn/*)
|
||||
(glob_files %{workspace_root}/examples/*)
|
||||
(glob_files programs/*.flan)
|
||||
(glob_files programs/assets/*)
|
||||
; A glob per package directory, because dune's glob does not descend.
|
||||
(glob_files programs/pkgs/shape/*)
|
||||
(glob_files programs/pkgs/area/*)
|
||||
(glob_files programs/pkgs/draw/*)
|
||||
(glob_files programs/pkgs/ring-a/*)
|
||||
(glob_files programs/pkgs/ring-b/*)
|
||||
(glob_files programs/pkgs/ring-c/*)
|
||||
(glob_files programs/pkgs/mac/*)
|
||||
(glob_files programs/pkgs/macring/*)
|
||||
(glob_files programs/pkgs/macspin/*))
|
||||
(action
|
||||
(setenv SURVEY_STRICT 1
|
||||
(setenv SURVEY_QUIET 1
|
||||
(setenv FLAN %{workspace_root}/bin/main.exe
|
||||
(run bash %{workspace_root}/spike/x86/survey.sh))))))
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user