diff --git a/FIX.org b/FIX.org index 4649ca5..11f426e 100644 --- a/FIX.org +++ b/FIX.org @@ -5542,3 +5542,71 @@ condition's payload" reached the struct-field refusal that fired first, never the condition arm it was named for. The struct refusal is gone since the descriptors landed, so the row is an [accepts] now and a new [rejects_check] signals a dyn directly to reach the arm that is still there. + +* 2026-09-21 — the registry's flaky read, and why a spin was the bug +test_reload's "the registry read under a writer" case failed about one run in +five on a loaded machine and never on an idle one. The mode behind it is +dev_limits.c's [regfull]: 3100 live blocks in a 4096-slot table, one thread +re-noting them as fast as it can, and the main thread asking the leak question +two hundred times. It wants 200 right answers, no zero-row answers, no wrong +counts and no refusals. + +** How it was reproduced +Not by re-running the test, which takes minutes and fails a fifth of the time. +dev_limits.c was compiled by hand against runtime/flan_rt.c, flan_dev.c and +flan_dyn.c — it is a plain C main, so it needs nothing from the compiler — and +then run twenty-four at a time on a sixteen-core box. Thirty runs on an idle +machine: all green. Two hundred and forty runs under that contention: 46 of +them had at least one refusal, highest count 6 of 200. Zero and wrong never +appeared once, in any run. + +That is also the one inference in this entry worth flagging: the original +failing output from test_reload was never captured, so which of the four +counters had moved is deduced rather than observed. The deduction is safe — +zero and wrong did not occur in ~480 contended runs, and refusals reproduced at +the reported rate — but the next person should know which is which. + +** What the race actually was +A patched copy printed [unread] on every refusal. It was 1 every single time: +exactly one slot, on the best of the eight walks, that could not be copied. +Never the epoch. And in this mode the epoch *cannot* move — compaction needs +[flan_reg_dead >= FLAN_REG_RECLAIM] and re-noting a live block never kills +one, so the dead count sits at zero for the whole run and no rearrangement is +possible. The table under the reader was stationary. The reader refused it +anyway. + +runtime/flan_dev.c:1140, [flan_reg_snap], read the slot's counter sixty-four +times with nothing between the tries. Two different things leave that counter +odd and they are minutes apart in scale. A writer that is running holds it odd +for seven stores; a bare re-read wins almost at once, and that is the case the +sixty-four were written for. A writer that was *descheduled* in the middle of +those stores holds it odd until the scheduler runs it again — a millisecond or +more when there are more threads than cores — and no amount of re-reading can +end that, because the re-reading is what keeps the core busy. So the reader +spent its entire budget inside a fraction of one quantum, reported the slot as +unreadable, and [flan_dev_reg_by_type] refused the listing. + +The irony is that the file already knew. [flan_reg_scan_wait], the walk-level +pause, is written up at length with exactly this reasoning — "a spin would take +a core from the thread being waited on". The per-slot retry never got it. + +** The fix +One pause helper, [flan_reg_wait], shared by both levels, moved above +[flan_reg_snap]. The slot retry is now eight bare looks — the running-writer +case, which pays nothing — and then the same 250us step, forty tries in all. +A refusal now means the table genuinely would not hold still. + +Not a widened timeout and not a loosened assertion: the design's own contract +is that waiting is legal on the reading side and refusal is for a table that is +being rearranged. A stationary table earning a refusal was the reader lying +about the writer. + +** Proof +The same 24-at-a-time harness, twice: 480 runs of [regfull], zero refusals, +zero wrong, zero zero-row. [regchurn] unchanged at 120 runs. [regrace], whose +answered/refused split is the machine's business and is printed rather than +pinned, still never produced a zero-row or a wrong count in 120 contended runs. + +** Not the dev-pause flake +test_dev's "rerun: the program is already running" at high load is a different +mechanism — nothing in that path reads the registry. Left where it was. diff --git a/docs/BUILT.md b/docs/BUILT.md index b4112fa..50f0b97 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -5324,13 +5324,24 @@ asking is asking because they suspect the opposite. A slot the writer kept winni would be a leak report quietly missing the leak — and the agent turns a refusal into an `err` line naming how many slots would not hold still, which the daemon already renders as the editor's message. -**And the eight attempts wait between themselves**, which is most of what makes them eight attempts rather than one. -A walk that bails at the epoch check costs almost nothing, so eight of them back to back fit inside the single -compaction they are all losing to. A quarter of a millisecond between them -- `flan_agent.c`'s break-loop idiom, and -legal here because the waiter is the listener thread and never the game loop -- bounds the whole refusal at two -milliseconds, and under a writer churning on top of 3000 live blocks it was the difference between 8 right answers in -200 and 200 in 200. It is not magic: a writer that spends most of its time rearranging the table still gets refused, -which is the honest answer to a question asked of a table that is never still. +**And every attempt waits, at both levels**, which is most of what makes them attempts rather than one. A retry on the +reading side costs a handful of loads, so a whole run of them back to back fits inside the single write they are all +losing to and the reader gives up having waited for nothing. A quarter of a millisecond between them -- `flan_agent.c`'s +break-loop idiom, and legal here because the waiter is the listener thread and never the game loop -- was the +difference, under a writer churning on top of 3000 live blocks, between 8 right answers in 200 and 200 in 200. It is +not magic: a writer that spends most of its time rearranging the table still gets refused, which is the honest answer +to a question asked of a table that is never still. + +**The same pause belongs on the per-slot retry, and its absence was a bug.** `flan_reg_snap` re-read a slot's counter +sixty-four times with nothing between the tries, and the two things that leave a counter odd are minutes apart in +scale. A writer that is *running* holds it odd for seven stores, which a bare re-read wins almost at once. A writer +that was *descheduled* in the middle of those stores holds it odd for however long the scheduler takes to run it again +-- a millisecond or more on a machine with more threads than cores -- and looking again cannot end that, because the +looking is what keeps the core busy. So the reader burned its whole budget inside a fraction of one quantum, called a +perfectly readable slot unreadable, and the listing above it refused a table that never moved at all. It showed up +only under load, which is exactly when a writer gets descheduled: no refusals at all on an idle machine, and one run in +five when the cores were oversubscribed. The retry is now eight bare looks and then the same quarter-millisecond step, +so a refusal means the table really would not hold still. See FIX.org, 2026-09-21. **"At exit" is not a hook, and the honest reason is that a game is killed.** A program stopped by a signal runs no `atexit` handler, no destructor, nothing — so no code written inside the program could report anything about the run diff --git a/runtime/flan_dev.c b/runtime/flan_dev.c index a1cef0f..ea03b78 100644 --- a/runtime/flan_dev.c +++ b/runtime/flan_dev.c @@ -33,8 +33,9 @@ #include #include #include -/* For the one wait in this file: the pause between a listing's attempts at - * reading the table. See flan_reg_scan_wait. Nothing on the writer's side +/* For the one wait in this file: the pause a listing takes between its + * attempts at reading the table, whether it is waiting out a rearrangement or + * a single slot mid-write. See flan_reg_wait. Nothing on the writer's side * waits for anything. */ #include @@ -1204,6 +1205,52 @@ static void flan_reg_end(flan_reg_entry *e) { __atomic_store_n(&e->gen, (e->gen | 1) + 1, __ATOMIC_RELEASE); } +/* A reader's pause between attempts, and the one wait anywhere in this file. + * + * Retrying immediately looks like several chances and is not. Every retry on + * the reading side costs a handful of loads, so a whole run of them fits + * inside the one write they are all losing to, and the reader gives up having + * waited for nothing. Measured, with a writer allocating and freeing on top of + * three thousand live blocks: without this pause 8 listings in 200 were + * answered and 192 were refused; with it, 200 of 200. + * + * A quarter of a millisecond is well over one compaction, and — the reason it + * is a sleep and not a spin — well over the handover a descheduled writer + * needs. The writer is a game loop and never reaches this; the reader is the + * agent's listener thread, or the exiting program's own, and both of them are + * answering a person. It is [flan_agent.c]'s break-loop idiom, nanosleep a + * step and look again, for its reason too: a spin would take a core from the + * thread being waited on, which on a machine with more threads than cores is + * exactly the thread that has to run before the wait can end. */ +static void flan_reg_wait(void) { + struct timespec step; + step.tv_sec = 0; + step.tv_nsec = 250000; /* 250us */ + nanosleep(&step, NULL); +} + +/* How many times a reader re-reads a slot's counter before it starts waiting + * between the tries instead of going straight round again. + * + * Two different things leave a counter odd, and they are minutes apart in + * scale. A writer that is running holds it odd for seven stores, so a reader + * that simply looks again wins almost at once and a sleep would be pure + * latency. A writer that was descheduled in the middle of those stores holds + * it odd for however long the scheduler takes to run it again — a millisecond + * or more on a loaded machine — and no amount of looking again will end that, + * because the looking is what is keeping the core busy. + * + * That second case is what this constant is for. It was the whole of a + * reproducible wrong answer: with sixty-four bare retries and nothing else, + * the reader burned its entire budget inside a fraction of one scheduler + * quantum, called the slot unreadable, and the listing above it refused a + * table that was perfectly readable and never moved. It showed up only under + * load, which is exactly when a writer gets descheduled — no refusals at all + * on an idle machine, and one run in five when the cores were oversubscribed. + * See FIX.org, 2026-09-21. */ +#define FLAN_REG_SPINS 8 +#define FLAN_REG_TRIES 40 /* 8 spins, then 32 waits: ~8ms of patience */ + /* One slot, copied whole or not at all. 0 means the writer kept winning, which * a caller reports as a slot it could not read rather than as an empty one. * @@ -1219,8 +1266,14 @@ static void flan_reg_end(flan_reg_entry *e) { * program. */ static int flan_reg_snap(flan_reg_entry *e, flan_reg_entry *out) { int attempt; - for (attempt = 0; attempt < 64; attempt++) { - uint64_t g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE); + for (attempt = 0; attempt < FLAN_REG_TRIES; attempt++) { + uint64_t g1; + /* The wait is only on the path that found a write in flight, so a table + nobody is writing pays nothing for it. One writer means at most one slot + of a walk is odd at a time, and that slot is almost always readable on + the first look after the first sleep. */ + if (attempt >= FLAN_REG_SPINS) flan_reg_wait(); + g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE); if (g1 & 1) continue; /* a write is in progress */ *out = *e; /* Ordered before the second read of the counter, or the check is of a copy @@ -1246,32 +1299,12 @@ static int flan_reg_scan_ok(uint64_t at) { return __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE) == at; } -/* And a pause between a reader's attempts, which is most of what makes eight - * attempts worth more than one. - * - * Retrying immediately looks like eight chances and is not. A walk that bails - * at the epoch check costs almost nothing, so eight of them back to back fit - * inside the single compaction they are all losing to, and the reader refuses - * having waited for nothing. Measured, with a writer allocating and freeing on - * top of three thousand live blocks: without this pause 8 answers in 200 were - * right and 192 were refusals; with it, 200 of 200. The rate matters and the - * pause is not magic — a writer rearranging the table more than half the time - * still gets refused, which is the honest answer to a question asked of a - * table that is never still, and is what the refusal sentence is for. - * - * Waiting is legal here and only here. The reader is the agent's listener - * thread, or the exiting program's own; the writer is a game loop and never - * reaches this. A quarter of a millisecond is well over one compaction and far - * under what a person waiting for a keypress to answer would notice, and eight - * of them bound the whole refusal at two milliseconds. It is [flan_agent.c]'s - * break-loop idiom — nanosleep a step, look again — for its reason too: a spin - * would take a core from the thread being waited on. */ -static void flan_reg_scan_wait(void) { - struct timespec step; - step.tv_sec = 0; - step.tv_nsec = 250000; /* 250us */ - nanosleep(&step, NULL); -} +/* The pause a walk takes between its own attempts is [flan_reg_wait] above, + * the same one a slot takes: a compaction is what a walk loses to, and it is + * over in well under the step. The rate matters and the pause is not magic — a + * writer rearranging the table more than half the time still gets refused, + * which is the honest answer to a question asked of a table that is never + * still, and is what the refusal sentence is for. */ /* Allocated by flan_dev_reg_enable and null until then, which is the whole of * what a release build carries: a null pointer, a zero flag, and the load and @@ -1733,7 +1766,7 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts, uint64_t at; n = 0; missed = 0; - if (!flan_reg_scan_open(&at)) { flan_reg_scan_wait(); continue; } + if (!flan_reg_scan_open(&at)) { flan_reg_wait(); continue; } for (i = 0; i < FLAN_REG_CAP; i++) { flan_reg_entry e; int64_t j; @@ -1761,14 +1794,14 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts, } n++; } - if (!flan_reg_scan_ok(at)) { flan_reg_scan_wait(); continue; } + if (!flan_reg_scan_ok(at)) { flan_reg_wait(); continue; } /* A stable epoch and every slot copied: this is a table that existed. */ if (missed == 0) return n; /* A stable epoch but slots that would not hold still. Worth another walk — the writer moves on — and worth remembering the closest one, because it is the number the refusal quotes. */ if (fewest < 0 || missed < fewest) fewest = missed; - flan_reg_scan_wait(); + flan_reg_wait(); } /* Eight walks, and not one of them saw the whole table. Answering with the last walk's rows would be answering with a table that never existed, and