The reader spun where it should have waited, and called a still table unreadable

test_reload's registry-under-a-writer case failed about one run in five on a
loaded machine and never on an idle one. The refusals were always unread=1:
one slot, on the best of eight walks, that would not copy. In that mode the
table cannot even move — re-noting live blocks kills nothing, the dead count
stays at zero, and the compaction trigger never fires — so the reader was
refusing a table that sat perfectly still for it.

flan_reg_snap re-read a slot's counter sixty-four times with nothing between
the tries. That is right for the writer it was written for, which holds the
counter odd for seven stores. It is hopeless against a writer the scheduler
took the core from mid-write, which holds it odd for a quantum: the reader
burns all sixty-four looks inside a fraction of that, and the looking is what
keeps the core the writer needs. Hence load-only.

The walk-level retry already had the answer written up at length — a spin takes
a core from the thread being waited on. The per-slot retry never got it. One
pause helper now, shared by both: eight bare looks for the running writer, then
the same 250us step. A refusal means the table would not hold still.

480 contended runs of regfull after, no refusals; 46 of 240 before.

FIX.org

@@ -5542,3 +5542,71 @@ condition's payload" reached the struct-field refusal that fired first, never
 the condition arm it was named for. The struct refusal is gone since the
 descriptors landed, so the row is an [accepts] now and a new [rejects_check]
 signals a dyn directly to reach the arm that is still there.
+
+* 2026-09-21 — the registry's flaky read, and why a spin was the bug
+test_reload's "the registry read under a writer" case failed about one run in
+five on a loaded machine and never on an idle one. The mode behind it is
+dev_limits.c's [regfull]: 3100 live blocks in a 4096-slot table, one thread
+re-noting them as fast as it can, and the main thread asking the leak question
+two hundred times. It wants 200 right answers, no zero-row answers, no wrong
+counts and no refusals.
+
+** How it was reproduced
+Not by re-running the test, which takes minutes and fails a fifth of the time.
+dev_limits.c was compiled by hand against runtime/flan_rt.c, flan_dev.c and
+flan_dyn.c — it is a plain C main, so it needs nothing from the compiler — and
+then run twenty-four at a time on a sixteen-core box. Thirty runs on an idle
+machine: all green. Two hundred and forty runs under that contention: 46 of
+them had at least one refusal, highest count 6 of 200. Zero and wrong never
+appeared once, in any run.
+
+That is also the one inference in this entry worth flagging: the original
+failing output from test_reload was never captured, so which of the four
+counters had moved is deduced rather than observed. The deduction is safe —
+zero and wrong did not occur in ~480 contended runs, and refusals reproduced at
+the reported rate — but the next person should know which is which.
+
+** What the race actually was
+A patched copy printed [unread] on every refusal. It was 1 every single time:
+exactly one slot, on the best of the eight walks, that could not be copied.
+Never the epoch. And in this mode the epoch *cannot* move — compaction needs
+[flan_reg_dead >= FLAN_REG_RECLAIM] and re-noting a live block never kills
+one, so the dead count sits at zero for the whole run and no rearrangement is
+possible. The table under the reader was stationary. The reader refused it
+anyway.
+
+runtime/flan_dev.c:1140, [flan_reg_snap], read the slot's counter sixty-four
+times with nothing between the tries. Two different things leave that counter
+odd and they are minutes apart in scale. A writer that is running holds it odd
+for seven stores; a bare re-read wins almost at once, and that is the case the
+sixty-four were written for. A writer that was *descheduled* in the middle of
+those stores holds it odd until the scheduler runs it again — a millisecond or
+more when there are more threads than cores — and no amount of re-reading can
+end that, because the re-reading is what keeps the core busy. So the reader
+spent its entire budget inside a fraction of one quantum, reported the slot as
+unreadable, and [flan_dev_reg_by_type] refused the listing.
+
+The irony is that the file already knew. [flan_reg_scan_wait], the walk-level
+pause, is written up at length with exactly this reasoning — "a spin would take
+a core from the thread being waited on". The per-slot retry never got it.
+
+** The fix
+One pause helper, [flan_reg_wait], shared by both levels, moved above
+[flan_reg_snap]. The slot retry is now eight bare looks — the running-writer
+case, which pays nothing — and then the same 250us step, forty tries in all.
+A refusal now means the table genuinely would not hold still.
+
+Not a widened timeout and not a loosened assertion: the design's own contract
+is that waiting is legal on the reading side and refusal is for a table that is
+being rearranged. A stationary table earning a refusal was the reader lying
+about the writer.
+
+** Proof
+The same 24-at-a-time harness, twice: 480 runs of [regfull], zero refusals,
+zero wrong, zero zero-row. [regchurn] unchanged at 120 runs. [regrace], whose
+answered/refused split is the machine's business and is printed rather than
+pinned, still never produced a zero-row or a wrong count in 120 contended runs.
+
+** Not the dev-pause flake
+test_dev's "rerun: the program is already running" at high load is a different
+mechanism — nothing in that path reads the registry. Left where it was.

docs/BUILT.md

@@ -5324,13 +5324,24 @@ asking is asking because they suspect the opposite. A slot the writer kept winni
 would be a leak report quietly missing the leak — and the agent turns a refusal into an `err` line naming how many
 slots would not hold still, which the daemon already renders as the editor's message.
 
-**And the eight attempts wait between themselves**, which is most of what makes them eight attempts rather than one.
+**And every attempt waits, at both levels**, which is most of what makes them attempts rather than one. A retry on the
-A walk that bails at the epoch check costs almost nothing, so eight of them back to back fit inside the single
+reading side costs a handful of loads, so a whole run of them back to back fits inside the single write they are all
-compaction they are all losing to. A quarter of a millisecond between them -- `flan_agent.c`'s break-loop idiom, and
+losing to and the reader gives up having waited for nothing. A quarter of a millisecond between them -- `flan_agent.c`'s
-legal here because the waiter is the listener thread and never the game loop -- bounds the whole refusal at two
+break-loop idiom, and legal here because the waiter is the listener thread and never the game loop -- was the
-milliseconds, and under a writer churning on top of 3000 live blocks it was the difference between 8 right answers in
+difference, under a writer churning on top of 3000 live blocks, between 8 right answers in 200 and 200 in 200. It is
-200 and 200 in 200. It is not magic: a writer that spends most of its time rearranging the table still gets refused,
+not magic: a writer that spends most of its time rearranging the table still gets refused, which is the honest answer
-which is the honest answer to a question asked of a table that is never still.
+to a question asked of a table that is never still.
+
+**The same pause belongs on the per-slot retry, and its absence was a bug.** `flan_reg_snap` re-read a slot's counter
+sixty-four times with nothing between the tries, and the two things that leave a counter odd are minutes apart in
+scale. A writer that is *running* holds it odd for seven stores, which a bare re-read wins almost at once. A writer
+that was *descheduled* in the middle of those stores holds it odd for however long the scheduler takes to run it again
+-- a millisecond or more on a machine with more threads than cores -- and looking again cannot end that, because the
+looking is what keeps the core busy. So the reader burned its whole budget inside a fraction of one quantum, called a
+perfectly readable slot unreadable, and the listing above it refused a table that never moved at all. It showed up
+only under load, which is exactly when a writer gets descheduled: no refusals at all on an idle machine, and one run in
+five when the cores were oversubscribed. The retry is now eight bare looks and then the same quarter-millisecond step,
+so a refusal means the table really would not hold still. See FIX.org, 2026-09-21.
 
 **"At exit" is not a hook, and the honest reason is that a game is killed.** A program stopped by a signal runs no
 `atexit` handler, no destructor, nothing — so no code written inside the program could report anything about the run

runtime/flan_dev.c

@@ -33,8 +33,9 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
-/* For the one wait in this file: the pause between a listing's attempts at
+/* For the one wait in this file: the pause a listing takes between its
- * reading the table. See flan_reg_scan_wait. Nothing on the writer's side
+ * attempts at reading the table, whether it is waiting out a rearrangement or
+ * a single slot mid-write. See flan_reg_wait. Nothing on the writer's side
  * waits for anything. */
 #include <time.h>
 
@@ -1204,6 +1205,52 @@ static void flan_reg_end(flan_reg_entry *e) {
   __atomic_store_n(&e->gen, (e->gen | 1) + 1, __ATOMIC_RELEASE);
 }
 
+/* A reader's pause between attempts, and the one wait anywhere in this file.
+ *
+ * Retrying immediately looks like several chances and is not. Every retry on
+ * the reading side costs a handful of loads, so a whole run of them fits
+ * inside the one write they are all losing to, and the reader gives up having
+ * waited for nothing. Measured, with a writer allocating and freeing on top of
+ * three thousand live blocks: without this pause 8 listings in 200 were
+ * answered and 192 were refused; with it, 200 of 200.
+ *
+ * A quarter of a millisecond is well over one compaction, and — the reason it
+ * is a sleep and not a spin — well over the handover a descheduled writer
+ * needs. The writer is a game loop and never reaches this; the reader is the
+ * agent's listener thread, or the exiting program's own, and both of them are
+ * answering a person. It is [flan_agent.c]'s break-loop idiom, nanosleep a
+ * step and look again, for its reason too: a spin would take a core from the
+ * thread being waited on, which on a machine with more threads than cores is
+ * exactly the thread that has to run before the wait can end. */
+static void flan_reg_wait(void) {
+  struct timespec step;
+  step.tv_sec = 0;
+  step.tv_nsec = 250000;   /* 250us */
+  nanosleep(&step, NULL);
+}
+
+/* How many times a reader re-reads a slot's counter before it starts waiting
+ * between the tries instead of going straight round again.
+ *
+ * Two different things leave a counter odd, and they are minutes apart in
+ * scale. A writer that is running holds it odd for seven stores, so a reader
+ * that simply looks again wins almost at once and a sleep would be pure
+ * latency. A writer that was descheduled in the middle of those stores holds
+ * it odd for however long the scheduler takes to run it again — a millisecond
+ * or more on a loaded machine — and no amount of looking again will end that,
+ * because the looking is what is keeping the core busy.
+ *
+ * That second case is what this constant is for. It was the whole of a
+ * reproducible wrong answer: with sixty-four bare retries and nothing else,
+ * the reader burned its entire budget inside a fraction of one scheduler
+ * quantum, called the slot unreadable, and the listing above it refused a
+ * table that was perfectly readable and never moved. It showed up only under
+ * load, which is exactly when a writer gets descheduled — no refusals at all
+ * on an idle machine, and one run in five when the cores were oversubscribed.
+ * See FIX.org, 2026-09-21. */
+#define FLAN_REG_SPINS 8
+#define FLAN_REG_TRIES 40   /* 8 spins, then 32 waits: ~8ms of patience */
+
 /* One slot, copied whole or not at all. 0 means the writer kept winning, which
  * a caller reports as a slot it could not read rather than as an empty one.
  *
@@ -1219,8 +1266,14 @@ static void flan_reg_end(flan_reg_entry *e) {
  * program. */
 static int flan_reg_snap(flan_reg_entry *e, flan_reg_entry *out) {
   int attempt;
-  for (attempt = 0; attempt < 64; attempt++) {
+  for (attempt = 0; attempt < FLAN_REG_TRIES; attempt++) {
-    uint64_t g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE);
+    uint64_t g1;
+    /* The wait is only on the path that found a write in flight, so a table
+       nobody is writing pays nothing for it. One writer means at most one slot
+       of a walk is odd at a time, and that slot is almost always readable on
+       the first look after the first sleep. */
+    if (attempt >= FLAN_REG_SPINS) flan_reg_wait();
+    g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE);
     if (g1 & 1) continue;                    /* a write is in progress */
     *out = *e;
     /* Ordered before the second read of the counter, or the check is of a copy
@@ -1246,32 +1299,12 @@ static int flan_reg_scan_ok(uint64_t at) {
   return __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE) == at;
 }
 
-/* And a pause between a reader's attempts, which is most of what makes eight
+/* The pause a walk takes between its own attempts is [flan_reg_wait] above,
- * attempts worth more than one.
+ * the same one a slot takes: a compaction is what a walk loses to, and it is
- *
+ * over in well under the step. The rate matters and the pause is not magic — a
- * Retrying immediately looks like eight chances and is not. A walk that bails
+ * writer rearranging the table more than half the time still gets refused,
- * at the epoch check costs almost nothing, so eight of them back to back fit
+ * which is the honest answer to a question asked of a table that is never
- * inside the single compaction they are all losing to, and the reader refuses
+ * still, and is what the refusal sentence is for. */
- * having waited for nothing. Measured, with a writer allocating and freeing on
- * top of three thousand live blocks: without this pause 8 answers in 200 were
- * right and 192 were refusals; with it, 200 of 200. The rate matters and the
- * pause is not magic — a writer rearranging the table more than half the time
- * still gets refused, which is the honest answer to a question asked of a
- * table that is never still, and is what the refusal sentence is for.
- *
- * Waiting is legal here and only here. The reader is the agent's listener
- * thread, or the exiting program's own; the writer is a game loop and never
- * reaches this. A quarter of a millisecond is well over one compaction and far
- * under what a person waiting for a keypress to answer would notice, and eight
- * of them bound the whole refusal at two milliseconds. It is [flan_agent.c]'s
- * break-loop idiom — nanosleep a step, look again — for its reason too: a spin
- * would take a core from the thread being waited on. */
-static void flan_reg_scan_wait(void) {
-  struct timespec step;
-  step.tv_sec = 0;
-  step.tv_nsec = 250000;   /* 250us */
-  nanosleep(&step, NULL);
-}
 
 /* Allocated by flan_dev_reg_enable and null until then, which is the whole of
  * what a release build carries: a null pointer, a zero flag, and the load and
@@ -1733,7 +1766,7 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
     uint64_t at;
     n = 0;
     missed = 0;
-    if (!flan_reg_scan_open(&at)) { flan_reg_scan_wait(); continue; }
+    if (!flan_reg_scan_open(&at)) { flan_reg_wait(); continue; }
     for (i = 0; i < FLAN_REG_CAP; i++) {
       flan_reg_entry e;
       int64_t j;
@@ -1761,14 +1794,14 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
       }
       n++;
     }
-    if (!flan_reg_scan_ok(at)) { flan_reg_scan_wait(); continue; }
+    if (!flan_reg_scan_ok(at)) { flan_reg_wait(); continue; }
     /* A stable epoch and every slot copied: this is a table that existed. */
     if (missed == 0) return n;
     /* A stable epoch but slots that would not hold still. Worth another walk —
        the writer moves on — and worth remembering the closest one, because it
        is the number the refusal quotes. */
     if (fewest < 0 || missed < fewest) fewest = missed;
-    flan_reg_scan_wait();
+    flan_reg_wait();
   }
   /* Eight walks, and not one of them saw the whole table. Answering with the
      last walk's rows would be answering with a table that never existed, and