flan/test/dune
Joseph Ferano 76270eac0d The test directory had been copying its own plumbing, file by file
Ten test binaries share a directory and had shared nothing in it but
watchdog.ml. Everything else each one needed it wrote out again: the
failure counter and its FAIL line, the three-line report tail, a poll,
a socket connect, the wait for a [flan dev] daemon to bind, a substring
search, and the Load -> Check -> Reach.link front half of a compile.

[listening] was the clearest case. Three copies, byte for byte apart
from one comment, and two of them said in that comment that they were
kept separate because "these three files have no module between them".
That was not true when it was written: watchdog.ml was already named in
the same (modules ...) stanzas. test_support.ml is the second such
module, wired the same way, and those two sentences go with the copies
they were explaining.

test_repl.ml's [quote] was Wire.quote character for character, in a file
that already links Wire and already names Wire.quote in a comment about
what the case below it is checking. It is Wire.quote now.

One real behaviour change, and it is a fix. [connect] existed twice over
with different retries: the agent's narrowed to ECONNREFUSED with a
comment saying why -- the socket file appears at bind, a moment before
listen -- while dev's and repl's retried any Unix_error, which meant an
ENOENT or an EACCES was retried to the full timeout before raising
something the reader still had to interpret. The shared one takes the
narrow version. Every caller connects to a socket [listening] has
already seen on disk, so the race it does catch is the only one left.

The rest is left where it is, on purpose. The three output-capturing
[run]s differ in what they wrap -- a pid suffix, a sanitizer environment,
a valgrind invocation -- and are not the same function. The report tails
in test_repl, test_web and the two sweep binaries print different things
for different reasons. The per-file scratch prefixes are the feature that
keeps two suites running at once from unlinking each other's sockets, so
the shared helper takes the prefix rather than choosing one. And the
[match Sys.command "command -v clang ..."] probes stay as they are:
their skip lines are output this suite pins.

bin/main.ml has the compile pipeline written out twice more. Left alone
-- this was a test/-scoped change and bin/ should not be reaching into a
test module -- and noted in FIX.org as what it actually needs, which is
the pipeline moving into lib/.

dune test: exit 0, and its output is the same line for line once the
temp-directory hash and the millisecond counts are normalised.
2026-09-20 11:56:30 +07:00

444 lines
21 KiB
Plaintext

(tests
(names test_flan test_acceptance test_reload test_agent test_session test_dev test_emacs test_repl test_cider test_dyn)
; Explicit because test_sanitize lives in this directory and is not one of
; these: two stanzas in one directory have to say which modules are whose.
; watchdog is every binary's clock: a hanging test reports nothing, so each
; of these arms an alarm that turns "for ever" into a failing run. test_support
; is the other shared module: the failure counter, the report tail, the poll,
; the daemon wait and the front half of a compile, which the binaries that
; wanted them had each been carrying their own copy of.
(modules test_flan test_acceptance test_reload test_agent test_session
test_dev test_emacs test_repl test_cider test_dyn watchdog
test_support)
(libraries flan unix)
; The acceptance programs are part of the test corpus: if the reader, the
; parser or the checker regresses on them we want to know here, not at the CLI.
(deps
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/sand.flan)
; The brush sheet. sand.flan no longer embeds it — the front-end was cut back
; to what lisp/sand.lisp and sand.jank have — so nothing in the corpus reads
; it today. Kept as a dependency because it is still in the workspace and an
; embed is read by the *checker*, relative to the file the form is written
; in, which is the rule any program picking it up again would meet.
(file %{workspace_root}/brush.png)
; The raylib bindings, because sand.flan and the FFI case import them and an
; import reads the directory at build time. sand.flan itself is above: the
; headless case imports it as a single-file package.
(glob_files %{workspace_root}/vendor/raylib/*)
; The dev agent package: its Flan declarations and the C that implements them.
(glob_files %{workspace_root}/vendor/agent/*)
; The EDN package — the tokenizer and the dynamic reader over it — which
; programs/edn.flan and edn-read.flan import; and the JSON
; tokenizer, which programs/json.flan does.
(glob_files %{workspace_root}/vendor/edn/*)
(glob_files %{workspace_root}/vendor/json/*)
; The ported raylib examples. Only one of them has a headless acceptance
; case, but it imports its example as a package and that example imports
; examples/digits.flan, so the directory has to be here whole.
(glob_files %{workspace_root}/examples/*)
(glob_files programs/*.flan)
; The package tree the multi-level cases import: pkg-diamond reaches shape
; through area and draw, and pkg-cycle reaches a ring. Each directory is a
; package, so each comes whole — a glob per directory rather than one over
; programs/pkgs/*, because dune's glob does not descend.
(glob_files programs/pkgs/shape/*)
(glob_files programs/pkgs/area/*)
(glob_files programs/pkgs/draw/*)
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*)
; The package that exports a data type, which pkg-data.flan imports.
(glob_files programs/pkgs/tree/*)
; The packages that declare macros: one whose macros a program calls
; qualified, and the two whose macros do not terminate — a ring, and one
; that never settles. Each is its own directory, so each needs its own glob.
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*)
; The synthetic C header the importer's table reads. Committed rather than
; reached for on the machine: the raylib case needs raylib installed, at the
; right version, with a variable set, so it skips everywhere and covers
; nothing. This one does not move.
(glob_files headers/*.h)
; The files programs/embed.flan bakes in. An embed reads them at *compile*
; time, so they are a dependency of the checker run and not of the program.
(glob_files programs/assets/*)
; And the tileset programs/edn-read.flan bakes in, which is under assets/ and
; not in it: embed.flan holds (embed-dir "assets") in a [3 EmbedFile], so a
; fourth file beside those three is a type error in an unrelated program.
; embed-dir does not descend and neither does a glob, so this is its own line.
(glob_files programs/assets/edn/*)
; And the config programs/json-provide.flan derives a struct from, under
; assets/ for the same reason and needing its own line for the same one.
(glob_files programs/assets/json/*)
; The reload primitive's host: a C main that dlopens what Build.shared made.
(file reload_host.c)
; A shared object that is not a redefinition module, for the agent's refusal
; path. Its destructor is what proves the handle was closed rather than lost.
(file noinstall.c)
; The other C main: flan_dev.c's two fixed limits, which no Flan program
; reaches, driven directly.
(file dev_limits.c)
; And the third: the dynamic-value runtime, which has no Flan spelling yet
; at all. Its host program is under programs/ and is picked up by the glob
; above; the header dyn_ops.c includes is the compiler's, dropped into the
; build directory beside each translation unit, so it is not a dependency
; here.
(file dyn_ops.c)
; test_dev runs the compiler itself: flan dev launches and owns a program.
(file %{workspace_root}/bin/main.exe)
; The Emacs client, which test_emacs drives against a real daemon.
(glob_files %{workspace_root}/emacs/*.el)
; The checker's own source, which test_flan reads: there is no reflecting
; over a match, so the only way to assert that Check.builtins still lists
; every builtin arm is to read the arms.
(file %{workspace_root}/lib/check.ml)
; The WASI host the wasm32 case runs its module under, when no wasmtime or
; wasmer is installed.
(file wasm-run.mjs)))
; The web target, in its own stanza rather than in the table above because it
; is the one case whose toolchain is a separate install: emscripten, and a
; raylib archive built by vendor/raylib/build-web.sh. It probes for both and
; skips with the reason, so it is green on a machine that has neither.
(test
(name test_web)
(modules test_web test_support)
(libraries flan unix)
(deps
(glob_files programs/*.flan)
(glob_files programs/assets/*)
(glob_files programs/assets/edn/*)
(glob_files programs/assets/json/*)
; The raylib bindings and the ported example the raylib case builds. The
; example imports examples/digits.flan, so the directory comes whole.
(glob_files %{workspace_root}/vendor/raylib/*)
(glob_files %{workspace_root}/examples/*)
; sand.flan for the browser, with the sheet it embeds and the dev agent it
; imports — the agent's directory has to be whole, because the file that
; makes a web build possible is the one Build selects out of it.
(file %{workspace_root}/sand.flan)
(file %{workspace_root}/brush.png)
(glob_files %{workspace_root}/vendor/agent/*)
; flan run --target=web is refused by the CLI, so the CLI has to be here.
(file %{workspace_root}/bin/main.exe)))
; The corpus a second time under ASan and UBSan. Its own alias and not part of
; `dune test`: a sanitized build is a statically linked 1.8MB binary that takes
; tens of seconds to produce, so the sweep is minutes against the existing
; suite's seconds, and a test nobody will wait for is a test nobody runs.
;
; dune build --root . @sanitize
; An executable plus a rule rather than a (test ...): a test stanza attaches
; to the @runtest alias and offers no way to be attached to another one, which
; is the whole point here.
(executable
(name test_sanitize)
(modules test_sanitize watchdog test_support)
(libraries flan unix))
(rule
(alias sanitize)
(deps
test_sanitize.exe
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/sand.flan)
(file %{workspace_root}/brush.png)
(glob_files %{workspace_root}/vendor/raylib/*)
(glob_files %{workspace_root}/vendor/agent/*)
(glob_files %{workspace_root}/vendor/edn/*)
(glob_files %{workspace_root}/vendor/json/*)
(glob_files %{workspace_root}/examples/*)
(glob_files programs/*.flan)
(glob_files programs/assets/*)
(glob_files programs/assets/edn/*)
; p13-dyn-collect.flan, which lives with the x86 probes because that is the
; lane that wrote it, and is in this sweep because of what it does rather
; than where it is: it is the only program anywhere that allocates past
; flan_dyn.c's one-megabyte floor, so it is the only one under which a mark
; and a sweep actually run. Every other Flan program here agrees with ASan
; by never collecting at all.
(glob_files %{workspace_root}/spike/x86/*.flan)
; The dyn runtime's C main, which is the one thing in this sweep that is not
; a Flan program: flan_dyn.c has no Flan spelling yet. It is also the one
; translation unit here that frees the most, which is what makes it worth a
; sanitized run at all. See [dyn_sweep].
(file dyn_ops.c)
(glob_files programs/assets/json/*))
(action (run ./test_sanitize.exe)))
; The corpus a third time, under Valgrind's memcheck. Its own alias for the
; same reason @sanitize has one, only more so: memcheck runs the program on a
; synthetic CPU, so the corpus is tens of minutes rather than seconds.
;
; dune build --root . @valgrind
;
; Why a third sweep when @sanitize exists: ASan answers "is this address
; mine", and cannot answer "were these bytes ever written". That second
; question is MSan's, MSan needs every dependency instrumented and raylib
; settles it, and memcheck answers both while needing no instrumentation at
; all. NEXT.md asked for exactly this.
(executable
(name test_valgrind)
(modules test_valgrind watchdog test_support)
(libraries flan unix str))
(rule
(alias valgrind)
(deps
test_valgrind.exe
; The suppression file, which is all reasons and no suppressions; its own
; header says why that is the finding rather than an oversight.
(file valgrind.supp)
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/sand.flan)
(file %{workspace_root}/brush.png)
(glob_files %{workspace_root}/vendor/raylib/*)
(glob_files %{workspace_root}/vendor/agent/*)
(glob_files %{workspace_root}/vendor/edn/*)
(glob_files %{workspace_root}/vendor/json/*)
(glob_files %{workspace_root}/examples/*)
(glob_files programs/*.flan)
(glob_files programs/assets/*)
(glob_files programs/assets/edn/*)
(glob_files programs/assets/json/*)
; The package tree the multi-level cases import, as in the test stanza
; above: a glob per directory, because dune's glob does not descend.
(glob_files programs/pkgs/shape/*)
(glob_files programs/pkgs/area/*)
(glob_files programs/pkgs/draw/*)
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*)
(glob_files programs/pkgs/tree/*)
; And the macro-declaring packages, for the same reason.
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*))
(action (run ./test_valgrind.exe)))
; The corpus a fourth time, through the hand-written x86-64 backend, compared
; against LLVM on what each program prints and what it exits with. Its own
; alias for the same reason the two above have one -- it builds every program
; twice and runs both, which is a couple of minutes against `dune test`'s
; seconds -- but the reason it exists at all is different. @sanitize and
; @valgrind ask whether the runtime is sound. This one asks whether the
; second backend still lowers the language: it refuses by name rather than
; miscompiling, so when another lane adds a primitive the backend says so
; loudly, and nothing was listening. Two such refusals sat in the tree for a
; month. Now they fail a build somebody can run.
;
; dune build --root . @x86
;
; A rule with no executable beside it, unlike @sanitize and @valgrind: the
; check already exists as spike/x86/survey.sh, which is what every handoff
; quotes its counts from, and a second implementation in OCaml would be a
; second thing to drift. SURVEY_STRICT=1 turns its report into an exit
; status. FLAN is passed because the script otherwise runs `dune build` on
; the compiler, and a dune inside a dune action waits on a lock it cannot
; get; main.exe is in the deps instead. SURVEY_QUIET keeps the skip
; breakdown out of a passing build's log.
(rule
(alias x86)
(deps
(file %{workspace_root}/spike/x86/survey.sh)
(glob_files %{workspace_root}/spike/x86/*.flan)
; The js spike's programs are plain flan programs and the sweep reads them
; now, so they have to be in the build tree the sweep runs from -- otherwise
; the glob matches nothing, and nothing is exactly what it was reporting
; while p1-int-semantics.flan sat there with a wrong shift in it.
(glob_files %{workspace_root}/spike/js/*.flan)
(file %{workspace_root}/bin/main.exe)
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/sand.flan)
(file %{workspace_root}/brush.png)
(glob_files %{workspace_root}/vendor/raylib/*)
(glob_files %{workspace_root}/vendor/agent/*)
(glob_files %{workspace_root}/vendor/edn/*)
(glob_files %{workspace_root}/vendor/json/*)
(glob_files %{workspace_root}/examples/*)
(glob_files programs/*.flan)
(glob_files programs/assets/*)
(glob_files programs/assets/edn/*)
(glob_files programs/assets/json/*)
; A glob per package directory, because dune's glob does not descend.
(glob_files programs/pkgs/shape/*)
(glob_files programs/pkgs/area/*)
(glob_files programs/pkgs/draw/*)
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*)
(glob_files programs/pkgs/tree/*)
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*))
(action
(setenv SURVEY_STRICT 1
(setenv SURVEY_QUIET 1
(setenv FLAN %{workspace_root}/bin/main.exe
(run bash %{workspace_root}/spike/x86/survey.sh))))))
; The reference page, checked against the compiler that is supposed to have
; produced everything on it. Two scripts, one alias, because they are halves of
; the same claim: web/examples/check.sh runs each program the page shows and
; diffs it against the .out recorded beside it, and web/examples/quotes.sh
; re-derives every *other* block -- the usage text, the refusal messages, the
; LLVM excerpt, the keybindings -- and looks for it in web/index.html. A
; paraphrase reads exactly like a quotation, which is the whole reason the
; second one exists.
;
; dune build --root . @page
;
; Opt-in rather than part of `dune test`, and for a reason that is NOT the one
; @sanitize, @valgrind and @x86 give. Those three are minutes; this is eleven
; seconds, five of which is breakdemo's deliberate sleep. Slowness is measured
; and is not the argument here. The argument is what a failure
; means: these two fail when a *document* has gone stale, not when the compiler
; has regressed, and a suite that goes red because prose drifted teaches the
; person running it to skim past red. `dune test` should mean "the language
; broke". This should mean "the page is lying". Keeping them apart is what lets
; both stay worth reading. See @checks below, which is how they get run.
;
; A rule rather than a (test ...) for the reason the @x86 comment gives: a test
; stanza attaches to @runtest and offers no way to be attached to anything else.
; FLAN is passed for the same reason too -- the scripts otherwise run `dune
; build` themselves, and a dune inside a dune action waits on a lock it cannot
; get. Everything either script reads has to be in the deps, because the action
; runs in _build and a glob that came up empty would let check.sh loop over no
; programs and exit 0 -- an alias that passes because it checked nothing.
(rule
(alias page)
(deps
(glob_files %{workspace_root}/web/examples/*)
; pkg.flan imports "geom" as a package, and dune's glob does not descend.
(glob_files %{workspace_root}/web/examples/geom/*)
(file %{workspace_root}/web/index.html)
(file %{workspace_root}/bin/main.exe)
; quotes.sh re-derives its needles from these: calc-me for the arithmetic
; answer, the corpus for the sand hash and the renderer's field spelling,
; conditions.org and flan-mode.el for the lines quoted verbatim.
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/conditions.org)
(glob_files %{workspace_root}/emacs/*.el)
(glob_files programs/*.flan)
; breakdemo.flan is built --dev, so it imports the agent; raylib is here for
; the binding line quotes.sh greps out of it.
(glob_files %{workspace_root}/vendor/agent/*)
(glob_files %{workspace_root}/vendor/raylib/*))
(action
(progn
(setenv FLAN %{workspace_root}/bin/main.exe
(run sh %{workspace_root}/web/examples/check.sh))
(setenv FLAN %{workspace_root}/bin/main.exe
(run sh %{workspace_root}/web/examples/quotes.sh)))))
; The indirection cell, driven from outside the language. spike/x86/cells.sh
; preloads a shared object whose constructor stores a different body into
; flan.cell.twice with dlsym, and checks that a --dev build notices and a
; release build does not -- 22 22 against 42 42, both backends, four builds.
; Its own alias and not part of @x86 because it is a different question:
; survey.sh asks whether the backend agrees with LLVM about what a program
; prints, and no program can answer this one, because a dev build starts with
; every cell already pointing at the body this build compiled.
;
; dune build --root . @cells
;
; Wired here for the reason everything on this page is wired: it was a real
; pass/fail check that nothing ran, which is how a check becomes decoration.
(rule
(alias cells)
(deps
(file %{workspace_root}/spike/x86/cells.sh)
(file %{workspace_root}/spike/x86/cell-override.c)
(glob_files %{workspace_root}/spike/x86/*.flan)
(file %{workspace_root}/bin/main.exe))
(action
(setenv FLAN %{workspace_root}/bin/main.exe
(run bash %{workspace_root}/spike/x86/cells.sh))))
; Everything that checks something and is not `dune test`, in one word.
;
; dune build --root . @checks
;
; This exists because the opt-in aliases had the same disease as the scripts
; they replaced. @x86 was added after two backend refusals sat unnoticed for a
; month, and an alias nobody types is no better than a script nobody types --
; it only looks more responsible. Five names to remember is four too many, and
; the cost of remembering them is paid every time, by whoever is least likely
; to know they exist.
;
; What is in it is the whole design. @page, @x86 and @cells: a couple of
; minutes between them, they need nothing installed that a build does not
; already need, and they are exactly the two failures this repository has
; actually had plus the one that was found beside them. @sanitize and @valgrind
; are deliberately out. They are tens of minutes and they want ASan and
; Valgrind present, and folding them in would make @checks the thing you do not
; have time for -- which is the disease, not the cure.
;
; What it buys is that deciding to check and checking everything are now the
; same act, so the gap between "somebody wondered" and "everything was
; verified" is one command instead of five.
;
; It used to say here that nothing runs this, because there was no CI. There is
; now: .github/workflows/checks.yml runs `dune build`, `dune test --force` and
; this alias on every push. That does not make the local run redundant, and the
; workflow says why in as many words — an Ubuntu runner has no raylib, no
; emscripten, no wasi sysroot and no lldb, so every one of those cases takes
; the skip path it already had and the tick is green over less than this alias
; covers here. The habit written down in README.md still carries the rest: a
; lane's handoff quotes this alias, the way the x86 handoffs already quote
; survey.sh.
(alias
(name checks)
(deps
(alias page)
(alias x86)
(alias js)
(alias cells)))
; The JS dialect's sweep, opt-in in the same way and for the same reason:
;
; dune build --root . @js
;
; Shaped after @x86 above, deps and all, because it walks the same corpus.
; Two differences. It needs node, which the script probes for and reports on
; rather than failing over -- this is opt-in, and a machine without node
; should say so and stop, not report zero of everything. And SURVEY_STRICT
; means something narrower here: a refusal is a *decision* in a dialect, not
; a gap, so only a DIFFER and a CRASH fail the alias. See the header of
; spike/js/survey.sh.
(rule
(alias js)
(deps
(file %{workspace_root}/spike/js/survey.sh)
(glob_files %{workspace_root}/spike/js/*.flan)
(file %{workspace_root}/bin/main.exe)
(file %{workspace_root}/calc-me.flan)
(file %{workspace_root}/sand.flan)
(file %{workspace_root}/brush.png)
(glob_files %{workspace_root}/vendor/raylib/*)
(glob_files %{workspace_root}/vendor/agent/*)
(glob_files %{workspace_root}/vendor/edn/*)
(glob_files %{workspace_root}/vendor/json/*)
(glob_files %{workspace_root}/examples/*)
(glob_files programs/*.flan)
(glob_files programs/assets/*)
(glob_files programs/assets/edn/*)
(glob_files programs/assets/json/*)
(glob_files programs/pkgs/shape/*)
(glob_files programs/pkgs/area/*)
(glob_files programs/pkgs/draw/*)
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*)
(glob_files programs/pkgs/tree/*)
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*))
(action
(setenv SURVEY_STRICT 1
(setenv SURVEY_QUIET 1
(setenv FLAN %{workspace_root}/bin/main.exe
(run bash %{workspace_root}/spike/js/survey.sh))))))