diff --git a/README.md b/README.md index 1507413..cf9d01c 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,30 @@ Useful build options include `--debug`, `--sanitize`, `--no-bounds-checks`, output should be run with an appropriate WASI runtime or browser. A `.wasm` file is not a tiny native executable in a trench coat. +## Checking it + +```text +dune test the suite. Seconds. Run it constantly. +dune build @checks everything else that can fail. A couple of minutes. +dune build @sanitize the corpus under ASan and UBSan. +dune build @valgrind the corpus under memcheck. Tens of minutes. +``` + +`dune test` means "the language still works". `@checks` — which is `@page`, +`@x86` and `@cells` — means "and everything written down about it is still +true": the reference page's examples still print what the page says, the +hand-written x86 backend still agrees with LLVM, and a `--dev` build still calls +through its indirection cells. + +The two are separate on purpose. A suite that goes red because prose drifted +teaches you to skim past red. But `@checks` only helps if it is run, and nothing +runs it for you — there is no CI here. The convention that has to carry it is +that a lane's handoff quotes `@checks`, the way the x86 handoffs already quote +the survey's counts. Both of this repository's silent failures — two backend +refusals that sat for a month, a page of examples that stopped compiling for two +days — were found by accident, and neither would have survived one person +typing one command. + ## Project map - [web/index.html](web/index.html) — language reference and fuller examples. diff --git a/docs/handoffs/HANDOFF-rot.md b/docs/handoffs/HANDOFF-rot.md index 3dfe2e0..ed754f5 100644 --- a/docs/handoffs/HANDOFF-rot.md +++ b/docs/handoffs/HANDOFF-rot.md @@ -1,29 +1,207 @@ # The checks nobody runs -Twice now a script in this repository has been quietly wrong for weeks. Both -times it was found by accident rather than by anything failing. This is the -inventory of everything in the tree that asserts something real and is not -reached by `dune test`, what each of them says today, and what was done about -the general problem. +Twice now a script in this repository has been quietly wrong for weeks, and both +times it was found by accident rather than by anything failing. `spike/x86/survey.sh` +had two programs refused by name for about a month. `web/examples/check.sh` had been +red since September 12th. This is the inventory of everything in the tree that +asserts something and is not reached by `dune test`, what each one says today, and +what was done about the general problem. -## The plan +A third failure was indeed waiting. It is the interesting one, and it is not the +failure it looked like. -1. Sweep `spike/`, `web/`, `tools/`, `emacs/` and `test/` for anything that - checks something and is not on a dune alias. Run every one of them; record - pass or fail honestly rather than assuming. -2. Give `web/examples/check.sh` and `web/examples/quotes.sh` an opt-in alias in - the shape `@x86` already established, with the same kind of comment saying - why they are opt-in. -3. Decide what would actually have caught the two known failures within a day, - given that an opt-in alias is only marginally better than a script — both - require somebody to type them. +## The inventory -## Red on arrival +Everything executable or fixture-shaped outside `dune test`, run rather than assumed. -`web/examples/quotes.sh` fails, eleven checks of forty-odd. It is not -re-recorded to make it green: every one of its failures is a real statement on -`web/index.html` that has stopped being true, and three of them are the page -claiming a language feature is unimplemented when it now works. That is a -content decision for the author, written up below rather than papered over. +| What | Verifies | Run how | State | +|---|---|---|---| +| `web/examples/check.sh` | each `web/examples/*.flan` against the `.out` beside it, plus `flan shim` on shimdemo and a killed `--dev` build of breakdemo | now `@page` | **passes**, 21 checks | +| `web/examples/quotes.sh` | every block on `index.html` that is not a program — usage text, refusal messages, the LLVM excerpt, the renderer's output, the Emacs keys — re-derived and looked for on the page | now `@page` | **was red, 11 of 42**; fixed, now passes | +| `spike/x86/survey.sh` | stdout, stderr and exit status of every corpus program, x86 backend against LLVM | `@x86`, and by hand | **passes**, 103 match / 0 differ / 0 refused / 38 skip | +| `spike/x86/cells.sh` | a `--dev` build really calls through its indirection cell: an `LD_PRELOAD`ed constructor stores a different body, four builds, `22 22` against `42 42` | nothing — now `@cells` | **passes**, 4 checks | +| `tools/colon-to-dot.py --check` | every `.flan` uses the dot spelling for field labels | nothing | **reports 20 hits, all false.** See below — this one is a hazard, not a check | +| `tools/unit-return.py --check` | every `defn` states a return type, `Unit` spelled `()` | nothing | **reports 30 hits, all false.** Same hazard | +| `emacs/test-flan-dape.el` | dape driving lldb-dap sets a breakpoint from a `.flan` buffer and reports Flan frames | nothing loads it | **cannot run here** — `require 'dape` fails, dape is not installed. Its own header says it is out of `dune test` on purpose | +| `emacs/test-flan-mode.el`, `test-flan-watch.el` | indentation shapes; the watch accumulator | loaded by `test-flan-cider.el` | covered by `dune test` | +| `spike/x86/bench.sh`, `cost.sh` | nothing — they measure and print tables | by hand | measurement, not checks | +| `spike/embed/run.sh`, `merged.sh`, `sig.sh`, `symbols.sh` | nothing — sizes, startup numbers, a symbol-collision report with no exit status | by hand | measurement, not checks | +| `spike/backend/run.sh`, `spike/generics/run.sh` | nothing — they drive a spike by hand | by hand | measurement, not checks | -(Details filled in as the work proceeds.) +Recorded-output fixtures: `find . -name '*.out'` outside `_build` returns twenty +files and all twenty are in `web/examples`. There is no second family of golden +files hiding anywhere. + +## The red one, and why nobody saw it + +`quotes.sh` failed eleven of its forty-odd checks, and the honest account is that +**nine of the eleven were the script's fault, not the page's**. This matters, because +the obvious move — "the page is stale, update the page" — would have been wrong nine +times out of eleven and would have rewritten correct prose. + +**Three were an over-wide needle.** `result`, `quoted` and `i64index` compare a +compiler message against the page. The page has them, exactly. But `flan check` grew +a source excerpt with a caret under the offending form, and the script stuffed the +whole of the output into the needle, so it was comparing three lines against a page +that quotes one. The script now takes the message and nothing else. + +**Six were probes for refusals that no longer happen.** `(Vec T)`, `(Map K V)`, +`(Handle T)`, an `(Fn …)` parameter, a `defer` inside a `let`, and `break` were each +refused as "not implemented yet" when this loop was written. All six work today. The +page's "Not implemented yet" table lost their rows as each one landed — correctly, +and `acda9ed` says so in its own message — but nobody went back to the probe list. +Those six were the last thing in the tree asserting a claim the page had already +stopped making, and they are deleted, with a comment saying what the list used to +cover so the next reader is not left guessing. + +**And what kept this invisible is the part worth keeping.** `flan check` answers a +program that compiles with its entire symbol table. So each of those six probes +failed by printing eighty lines of prelude signatures into a report that was already +forty lines long. The author had built a guard against false greens — the empty-needle +case, with a comment calling it the one failure the script exists to catch — and the +rot arrived instead as a **two-hundred-line unreadable red**. A check whose output +nobody can read is a check nobody reads. A clean exit is now its own one-line failure: +`this program compiles now — the page still says it is refused`. + +Two failures were genuine, and both were a document that had gone stale: + +**`cell load`.** The page quotes an LLVM excerpt from `flan emit --dev`. A `--dev` +`main` now pushes a condition frame before anything else, which shifted the SSA +numbering by one, so the page's `%t1 = load ptr, ptr @"flan.cell.work"` is `%t2` +today. The three quoted lines are now what the compiler prints, with the frame push +marked as elided rather than silently dropped. + +**`renderer`, which is the September 12th failure again and was still there.** The +check greps `NEXT.md` for a line of inspector output. `NEXT.md` is a rolling scratch +document; the line was rewritten out of it, the grep went empty, the empty-needle +guard fired and said "whatever this quotes has moved" — into a report nobody was +reading. Behind that noise the page was genuinely wrong in exactly the way `check.sh` +was wrong two days ago: the colon-to-dot sweep rewrote every field label in the +corpus, `lib/render.ml` writes `.field` today and `test_repl.ml`'s fixtures are dots +throughout, and the page had been showing `(V {:x 1.5 :y 0})` for a renderer that +prints `(V {.x 1.5 .y 0})`. The page is corrected; so is the identical block in +`docs/BUILT.md`, which nothing checks and which had drifted the same way. The check's +anchor moves from `NEXT.md` to `test/programs/raylib-imported.flan`, which `dune test` +builds and runs, so it cannot quietly stop saying it. + +The design lesson is one line: **an anchor in a rolling document is not an anchor.** + +## The hazard in `tools/` + +This is not a rotted check, it is worse, and it deserves its own paragraph. + +`tools/colon-to-dot.py` advertises itself as re-runnable — "converting an already- +converted file is a no-op, so this can be run again over files a parallel branch +wrote in the old spelling". Run `--check` today and it reports twenty field labels to +convert across six files. Every one of them is `{:where (copyable? $t)}`, the generic +constraint clause, which is not a struct literal and names no field. The generics lane +introduced a brace form the tool has never heard of. **Running that tool for real +would corrupt `test/programs/generics.flan`, `reload-generic.flan`, +`generic-map-reject.flan`, `generic-runaway.flan`, `spike/generics/prelude-shapes.flan` +and `syntax-sketch.flan`.** + +`tools/unit-return.py` has the same shape of fault. It decides whether a `defn` +already states a return type by transcribing the old `parse.ml`, and that +transcription knows the prelude's types and the builtins but not a type that arrived +through `declare-c`. So it reads `(defn v2-add [a Vector2 b Vector2] Vector2 …)` as a +function with no return type and would insert `()` in front of sixteen correct +signatures in `vendor/raylib/vector.flan`. + +Neither is wired to an alias and neither should be. They are finished one-shot +migrations whose `--check` mode now reports noise, and the safe move is to delete them +or to put a line at the top of each saying they are historical and must not be re-run +without being taught `{:where}` and imported type names first. **That is the author's +call and nothing here touched them beyond reading.** It is flagged because a +destructive script sitting in `tools/` looking maintained is a worse trap than a red +test. + +## What got wired + +Three stanzas in `test/dune`, in the shape `@x86` established — a `rule` with an +alias rather than a `test`, because a test stanza attaches to `@runtest` and offers no +way to be attached to anything else, and with `FLAN` passed in because a script that +shells out to `dune` from inside a dune action waits on a lock it cannot get. + +**`@page`** runs `check.sh` then `quotes.sh`. 42 checks, eleven seconds, five of which +is breakdemo's deliberate `timeout 5`. Its comment gives a *different* reason for being +opt-in than its three neighbours do, and the difference is the point: `@sanitize`, +`@valgrind` and `@x86` are opt-in because they are slow, and this one is not slow. It +is separate because of what a failure means. These two fail when a **document** has +gone stale, not when the compiler has regressed, and a suite that goes red because +prose drifted teaches whoever runs it to skim past red. `dune test` should mean "the +language broke". `@page` should mean "the page is lying". + +**`@cells`** runs `spike/x86/cells.sh`, which was a real four-check pass/fail script +that nothing in the tree ran. It is not folded into `@x86` because it asks a different +question: the survey asks whether the backend agrees with LLVM about what a program +prints, and no program can answer this one, because a dev build starts with every cell +already pointing at the body this build compiled. + +**`@checks`** is `@page`, `@x86` and `@cells` in one word. + +All three scripts learned to resolve `FLAN` to an absolute path, which turned out to +be the only thing actually standing between them and a dune rule: `%{workspace_root}` +expands relative to the directory the rule is written in, and every one of these +scripts `cd`s somewhere before using it. + +## Item 3: what would genuinely have caught these + +The honest answer first. **There is no local mechanism with teeth, and `@checks` is +not one.** It is discoverability, not enforcement. An alias nobody types is no better +than a script nobody types; it only looks more responsible. What the umbrella buys is +that *deciding to check* and *checking everything* become the same act, so the gap +between "somebody wondered" and "everything was verified" is one command instead of +five — and five names to remember is four too many, paid every time, by whoever is +least likely to know they exist. It shortens the gap. It does not close it. + +**The recommendation is `@checks` plus a written convention, and nothing else.** +`README.md` now has a "Checking it" section giving the four commands, saying which +means what, and saying plainly that nothing runs `@checks` for you. The convention it +proposes is that a lane's handoff quotes `@checks`, the way the x86 handoffs already +quote the survey's counts — which is a habit this repository demonstrably already has, +and habits that already exist are the only ones worth adding a rule to. + +**Against a git hook, which is the tempting answer.** `@checks` is a couple of minutes, +nearly all of it `@x86` building every program twice. A pre-commit hook costing two +minutes gets switched off within a week, and a disabled hook is strictly worse than no +hook because it reads as coverage. A pre-push hook is cheaper in principle, because +pushing is rare — but the remote is not reachable from here and this repository's work +is merged between local worktrees, so there is no push to hang it on. There is no +`.git/hooks` content and no `core.hooksPath` today; that is a reasonable state and it +was left alone. + +**What would actually have caught both failures within a day** is CI on every push, +and this repository has none. That is the real answer and it should not be dressed up. +The failure mode being defended against is not "somebody ran the check and ignored the +result" — it is "a month passed and nobody ran anything" — and only a machine that +runs without being asked fixes that. + +**One thing deliberately not done, because it is the author's call.** `@page` is eleven +seconds against `dune test`'s forty-eight, and five of those eleven are a fixed sleep. +That is a quarter again on top of the suite — not free, but not the reason it is out +either — and putting it in the suite is the one move that would give it actual teeth. The argument against is the one in its comment — +that mixing "the page is lying" into "the language broke" devalues both — and that is +a judgement about what a red suite should mean, not a measurement. The number is here +so the decision can be made on it rather than on an assumption. + +## Verification + +- `dune test --root .` — 232 checks, 0 failures, 47s. Unchanged: nothing was added to + the default suite. +- `dune build --root . @x86` — passes. 103 match, 0 differ, 0 refused, 38 skip. +- `dune build --root . @page` — passes, 42 checks. Verified both ways: + `web/examples/hello.out` was temporarily rewritten to say `goodbye world`, the build + went red with a readable diff and exit 1, and the break was reverted. The tree is + clean. +- `dune build --root . @cells` — passes, 4 checks. + +## Loose ends for the author + +- The two scripts in `tools/` described above. They are live hazards and the fix is a + decision, not a patch. +- `emacs/test-flan-dape.el` cannot run on this machine and nothing loads it. It is the + one check in the inventory whose state is genuinely unknown. +- Three commits — `8bd5547`, `64e4f6c`, `0251cf4` — were on this worktree's branch and + were dropped by the reset to `dev-loop` that this lane was told to perform. They are + your own unmerged edits to `web/index.html`, `plan.org`, `NEXT.md` and `lib/build.ml`, + they are unrelated to anything here, and they are still reachable by hash.