They were the whole of this binary's wall clock, and none of it was CPU the
machine did not have: one core busy, fifteen idle, for 30-odd seconds of
independent programs run strictly one after another. A small pool now forks
up to eight of them at once and reads results back through a pipe in the
order the rows are written in, so the log is byte-for-byte what it always
was — just produced faster.
Eight, not sixteen: dune already runs this suite's eleven binaries at once,
and test_dev's daemon is the actual critical path at roughly 46s against
this one's 30. Handing the pool the whole box would rob the binary that
matters more to speed up the one that matters less.
Every temporary path a worker touches now carries its own pid, so a
program's -O2 row and its -O0 row, run in different forked processes, can
never collide on the same exe or the same captured stdout underfoot. Three
rows could not be pooled that way regardless: slurp.flan and files.flan
write to fixed relative paths their own variants share, with a clean()
between them that only makes sense run in order, so those three stay on the
direct, unpooled path the whole file used before.
A worker that dies without ever writing its result — signalled, OOM-killed,
segfaulted mid-compile — is counted and named rather than silently read as
a pass; the exit status is trusted over a payload that never arrived. And a
worker clears its own SIGALRM handler on the way in, so the parent's
watchdog can never be triggered from inside a child holding a stale copy of
its sibling list. If the watchdog does fire from the parent, it now kills
every worker still in flight before it exits, so a hang does not also leave
orphans behind it.
Standalone, test_acceptance went from 43.4s to 13.8-20.8s across three runs
with identical output every time. The full dune test did not move — 49.4-
49.9s after against 49.8s before — which is the eleven-way sibling
parallelism already claiming what this pool would have used.
Joseph Ferano
7b3773c344