The noise is gone and the reporting half is verified by breaking the test on purpose
This commit is contained in:
parent
3c4ca20fea
commit
1195e90476
@ -39,15 +39,51 @@ stderr was simply never given the same treatment.
|
||||
The result is four lines of what reads like a broken toolchain on a passing run. A real linker failure in that spot
|
||||
today would be invisible.
|
||||
|
||||
## The plan
|
||||
## What was done
|
||||
|
||||
1. Give the robustness daemon its own stderr log rather than the suite's, the same way its stdout already has one.
|
||||
2. Report that log — and say so — only when something in the block actually failed, so a genuine linker error is
|
||||
still visible, and a deliberate one is not.
|
||||
3. Have the fixture announce the deliberate failure in its own voice before it causes it, so a reader of the output
|
||||
knows a build is about to be made to fail and why.
|
||||
Three changes, all in `test/test_dev.ml` and all inside the `dev-robust` block. The lever that causes the failure
|
||||
is untouched: the existing comment's argument that a read-only cache is "the one lever that reaches the macro module
|
||||
and nothing else" is right, and swapping it for a different cause would mean re-proving that the new one does not
|
||||
leak into other builds. The channel was the defect, not the cause.
|
||||
|
||||
Baseline to not regress: `dune test --root .` exit 0, `spike/x86/survey.sh` 101 MATCH / 0 DIFFER / 0 refused,
|
||||
1. **The daemon gets a stderr of its own** — `rerr = tmp "robust.err"`, passed to `create_process_env` where
|
||||
`Unix.stderr` used to be, the same treatment its stdout already had.
|
||||
2. **That log is reprinted, contents and not path, only when a step in this block failed.** The count is taken
|
||||
against `rfailures`, a snapshot of `!failures` at block entry, because `failures` is the whole file's and an
|
||||
earlier block's failure is not a reason to reprint this daemon's expected complaints. The path itself would be
|
||||
useless: `rerr` lives under dune's per-run `TMPDIR`, which is gone before anyone reads the run.
|
||||
3. **The fixture says so in its own voice**, one line, before the daemon starts. The wording deliberately contains
|
||||
no `error`, `Error` or `FAIL`, because the reader it is written for is scanning for exactly those words.
|
||||
|
||||
## Measured, before and after
|
||||
|
||||
Both from a full `dune test --root .`, exit 0 either way.
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| `cannot open output file` lines | 2 | **0** |
|
||||
| `clang: error: linker command failed` lines | 2 | **0** |
|
||||
| a line saying the failure is deliberate | none | **1** |
|
||||
|
||||
The two occurrences in the baseline were at `baseline.log:563` and `:565`, immediately after
|
||||
`flan dev: ... dev-robust.flan ready on ...` — which places them in this block and nowhere else. **The count is
|
||||
exactly two**, so there is no second site elsewhere in the suite quietly doing the same thing; fixing `test_dev.ml`
|
||||
fixes all of it.
|
||||
|
||||
Two informational lines from this daemon — `flan dev: built dev-robust.flan in 1937ms` and its `ready on` — now go
|
||||
to the log with the rest of its stderr rather than to the terminal. Its siblings still print theirs. That is a small
|
||||
inconsistency and the right side of the trade: the whole point is that this daemon's stderr is a channel nobody
|
||||
should be reading on a green run.
|
||||
|
||||
## Verified
|
||||
|
||||
- `dune test --root .` exit 0, and the four lines are gone. `test_dev` alone: `dev: all tests passed`, exit 0.
|
||||
- **The reporting half was verified by breaking the test on purpose**, which is the part a green run cannot show:
|
||||
a fix that silences unconditionally and a fix that reports-on-failure produce identical output when everything
|
||||
passes. The `Some "16"` expectation was temporarily changed to a value it cannot match, the block failed, and the
|
||||
captured stderr — the linker's own words — was reprinted under its header. Then reverted.
|
||||
|
||||
Baseline not regressed: `dune test --root .` exit 0, `spike/x86/survey.sh` 101 MATCH / 0 DIFFER / 0 refused,
|
||||
`spike/x86/cells.sh` 4/4.
|
||||
|
||||
## Open questions
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user