JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

The reader spun where it should have waited, and called a still table unreadable

Browse Source 
test_reload's registry-under-a-writer case failed about one run in five on a
loaded machine and never on an idle one. The refusals were always unread=1:
one slot, on the best of eight walks, that would not copy. In that mode the
table cannot even move — re-noting live blocks kills nothing, the dead count
stays at zero, and the compaction trigger never fires — so the reader was
refusing a table that sat perfectly still for it.

flan_reg_snap re-read a slot's counter sixty-four times with nothing between
the tries. That is right for the writer it was written for, which holds the
counter odd for seven stores. It is hopeless against a writer the scheduler
took the core from mid-write, which holds it odd for a quantum: the reader
burns all sixty-four looks inside a fraction of that, and the looking is what
keeps the core the writer needs. Hence load-only.

The walk-level retry already had the answer written up at length — a spin takes
a core from the thread being waited on. The per-slot retry never got it. One
pause helper now, shared by both: eight bare looks for the running writer, then
the same 250us step. A refusal means the table would not hold still.

480 contended runs of regfull after, no refusals; 46 of 240 before.
This commit is contained in:
Joseph Ferano 2026-09-21 10:26:45 +07:00
parent b8856bef3a
commit 8f05ece441
3 changed files with 152 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
FIX.org
View File

@ -5542,3 +5542,71 @@ condition's payload" reached the struct-field refusal that fired first, never
the condition arm it was named for. The struct refusal is gone since the
descriptors landed, so the row is an [accepts] now and a new [rejects_check]
signals a dyn directly to reach the arm that is still there.
* 2026-09-21 — the registry's flaky read, and why a spin was the bug
test_reload's "the registry read under a writer" case failed about one run in
five on a loaded machine and never on an idle one. The mode behind it is
dev_limits.c's [regfull]: 3100 live blocks in a 4096-slot table, one thread
re-noting them as fast as it can, and the main thread asking the leak question
two hundred times. It wants 200 right answers, no zero-row answers, no wrong
counts and no refusals.
** How it was reproduced
Not by re-running the test, which takes minutes and fails a fifth of the time.
dev_limits.c was compiled by hand against runtime/flan_rt.c, flan_dev.c and
flan_dyn.c — it is a plain C main, so it needs nothing from the compiler — and
then run twenty-four at a time on a sixteen-core box. Thirty runs on an idle
machine: all green. Two hundred and forty runs under that contention: 46 of
them had at least one refusal, highest count 6 of 200. Zero and wrong never
appeared once, in any run.
That is also the one inference in this entry worth flagging: the original
failing output from test_reload was never captured, so which of the four
counters had moved is deduced rather than observed. The deduction is safe —
zero and wrong did not occur in ~480 contended runs, and refusals reproduced at
the reported rate — but the next person should know which is which.
** What the race actually was
A patched copy printed [unread] on every refusal. It was 1 every single time:
exactly one slot, on the best of the eight walks, that could not be copied.
Never the epoch. And in this mode the epoch *cannot* move — compaction needs
[flan_reg_dead >= FLAN_REG_RECLAIM] and re-noting a live block never kills
one, so the dead count sits at zero for the whole run and no rearrangement is
possible. The table under the reader was stationary. The reader refused it
anyway.
runtime/flan_dev.c:1140, [flan_reg_snap], read the slot's counter sixty-four
times with nothing between the tries. Two different things leave that counter
odd and they are minutes apart in scale. A writer that is running holds it odd
for seven stores; a bare re-read wins almost at once, and that is the case the
sixty-four were written for. A writer that was *descheduled* in the middle of
those stores holds it odd until the scheduler runs it again — a millisecond or
more when there are more threads than cores — and no amount of re-reading can
end that, because the re-reading is what keeps the core busy. So the reader
spent its entire budget inside a fraction of one quantum, reported the slot as
unreadable, and [flan_dev_reg_by_type] refused the listing.
The irony is that the file already knew. [flan_reg_scan_wait], the walk-level
pause, is written up at length with exactly this reasoning — "a spin would take
a core from the thread being waited on". The per-slot retry never got it.
** The fix
One pause helper, [flan_reg_wait], shared by both levels, moved above
[flan_reg_snap]. The slot retry is now eight bare looks — the running-writer
case, which pays nothing — and then the same 250us step, forty tries in all.
A refusal now means the table genuinely would not hold still.
Not a widened timeout and not a loosened assertion: the design's own contract
is that waiting is legal on the reading side and refusal is for a table that is
being rearranged. A stationary table earning a refusal was the reader lying
about the writer.
** Proof
The same 24-at-a-time harness, twice: 480 runs of [regfull], zero refusals,
zero wrong, zero zero-row. [regchurn] unchanged at 120 runs. [regrace], whose
answered/refused split is the machine's business and is printed rather than
pinned, still never produced a zero-row or a wrong count in 120 contended runs.
** Not the dev-pause flake
test_dev's "rerun: the program is already running" at high load is a different
mechanism — nothing in that path reads the registry. Left where it was.

25
docs/BUILT.md
View File

@ -5324,13 +5324,24 @@ asking is asking because they suspect the opposite. A slot the writer kept winni
would be a leak report quietly missing the leak — and the agent turns a refusal into an `err` line naming how many
slots would not hold still, which the daemon already renders as the editor's message.
**And the eight attempts wait between themselves**, which is most of what makes them eight attempts rather than one.
A walk that bails at the epoch check costs almost nothing, so eight of them back to back fit inside the single
compaction they are all losing to. A quarter of a millisecond between them -- `flan_agent.c`'s break-loop idiom, and
legal here because the waiter is the listener thread and never the game loop -- bounds the whole refusal at two
milliseconds, and under a writer churning on top of 3000 live blocks it was the difference between 8 right answers in
200 and 200 in 200. It is not magic: a writer that spends most of its time rearranging the table still gets refused,
which is the honest answer to a question asked of a table that is never still.
**And every attempt waits, at both levels**, which is most of what makes them attempts rather than one. A retry on the
reading side costs a handful of loads, so a whole run of them back to back fits inside the single write they are all
losing to and the reader gives up having waited for nothing. A quarter of a millisecond between them -- `flan_agent.c`'s
break-loop idiom, and legal here because the waiter is the listener thread and never the game loop -- was the
difference, under a writer churning on top of 3000 live blocks, between 8 right answers in 200 and 200 in 200. It is
not magic: a writer that spends most of its time rearranging the table still gets refused, which is the honest answer
to a question asked of a table that is never still.
**The same pause belongs on the per-slot retry, and its absence was a bug.** `flan_reg_snap` re-read a slot's counter
sixty-four times with nothing between the tries, and the two things that leave a counter odd are minutes apart in
scale. A writer that is *running* holds it odd for seven stores, which a bare re-read wins almost at once. A writer
that was *descheduled* in the middle of those stores holds it odd for however long the scheduler takes to run it again
-- a millisecond or more on a machine with more threads than cores -- and looking again cannot end that, because the
looking is what keeps the core busy. So the reader burned its whole budget inside a fraction of one quantum, called a
perfectly readable slot unreadable, and the listing above it refused a table that never moved at all. It showed up
only under load, which is exactly when a writer gets descheduled: no refusals at all on an idle machine, and one run in
five when the cores were oversubscribed. The retry is now eight bare looks and then the same quarter-millisecond step,
so a refusal means the table really would not hold still. See FIX.org, 2026-09-21.
**"At exit" is not a hook, and the honest reason is that a game is killed.** A program stopped by a signal runs no
`atexit` handler, no destructor, nothing — so no code written inside the program could report anything about the run

99
runtime/flan_dev.c
View File

@ -33,8 +33,9 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* For the one wait in this file: the pause between a listing's attempts at
* reading the table. See flan_reg_scan_wait. Nothing on the writer's side
/* For the one wait in this file: the pause a listing takes between its
* attempts at reading the table, whether it is waiting out a rearrangement or
* a single slot mid-write. See flan_reg_wait. Nothing on the writer's side
* waits for anything. */
#include <time.h>
@ -1204,6 +1205,52 @@ static void flan_reg_end(flan_reg_entry *e) {
__atomic_store_n(&e->gen, (e->gen | 1) + 1, __ATOMIC_RELEASE);
}
/* A reader's pause between attempts, and the one wait anywhere in this file.
*
* Retrying immediately looks like several chances and is not. Every retry on
* the reading side costs a handful of loads, so a whole run of them fits
* inside the one write they are all losing to, and the reader gives up having
* waited for nothing. Measured, with a writer allocating and freeing on top of
* three thousand live blocks: without this pause 8 listings in 200 were
* answered and 192 were refused; with it, 200 of 200.
*
* A quarter of a millisecond is well over one compaction, and — the reason it
* is a sleep and not a spin — well over the handover a descheduled writer
* needs. The writer is a game loop and never reaches this; the reader is the
* agent's listener thread, or the exiting program's own, and both of them are
* answering a person. It is [flan_agent.c]'s break-loop idiom, nanosleep a
* step and look again, for its reason too: a spin would take a core from the
* thread being waited on, which on a machine with more threads than cores is
* exactly the thread that has to run before the wait can end. */
static void flan_reg_wait(void) {
struct timespec step;
step.tv_sec = 0;
step.tv_nsec = 250000; /* 250us */
nanosleep(&step, NULL);
}
/* How many times a reader re-reads a slot's counter before it starts waiting
* between the tries instead of going straight round again.
*
* Two different things leave a counter odd, and they are minutes apart in
* scale. A writer that is running holds it odd for seven stores, so a reader
* that simply looks again wins almost at once and a sleep would be pure
* latency. A writer that was descheduled in the middle of those stores holds
* it odd for however long the scheduler takes to run it again — a millisecond
* or more on a loaded machine — and no amount of looking again will end that,
* because the looking is what is keeping the core busy.
*
* That second case is what this constant is for. It was the whole of a
* reproducible wrong answer: with sixty-four bare retries and nothing else,
* the reader burned its entire budget inside a fraction of one scheduler
* quantum, called the slot unreadable, and the listing above it refused a
* table that was perfectly readable and never moved. It showed up only under
* load, which is exactly when a writer gets descheduled — no refusals at all
* on an idle machine, and one run in five when the cores were oversubscribed.
* See FIX.org, 2026-09-21. */
#define FLAN_REG_SPINS 8
#define FLAN_REG_TRIES 40 /* 8 spins, then 32 waits: ~8ms of patience */
/* One slot, copied whole or not at all. 0 means the writer kept winning, which
* a caller reports as a slot it could not read rather than as an empty one.
*
@ -1219,8 +1266,14 @@ static void flan_reg_end(flan_reg_entry *e) {
* program. */
static int flan_reg_snap(flan_reg_entry *e, flan_reg_entry *out) {
int attempt;
for (attempt = 0; attempt < 64; attempt++) {
uint64_t g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE);
for (attempt = 0; attempt < FLAN_REG_TRIES; attempt++) {
uint64_t g1;
/* The wait is only on the path that found a write in flight, so a table
nobody is writing pays nothing for it. One writer means at most one slot
of a walk is odd at a time, and that slot is almost always readable on
the first look after the first sleep. */
if (attempt >= FLAN_REG_SPINS) flan_reg_wait();
g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE);
if (g1 & 1) continue; /* a write is in progress */
*out = *e;
/* Ordered before the second read of the counter, or the check is of a copy
@ -1246,32 +1299,12 @@ static int flan_reg_scan_ok(uint64_t at) {
return __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE) == at;
}
/* And a pause between a reader's attempts, which is most of what makes eight
* attempts worth more than one.
*
* Retrying immediately looks like eight chances and is not. A walk that bails
* at the epoch check costs almost nothing, so eight of them back to back fit
* inside the single compaction they are all losing to, and the reader refuses
* having waited for nothing. Measured, with a writer allocating and freeing on
* top of three thousand live blocks: without this pause 8 answers in 200 were
* right and 192 were refusals; with it, 200 of 200. The rate matters and the
* pause is not magic — a writer rearranging the table more than half the time
* still gets refused, which is the honest answer to a question asked of a
* table that is never still, and is what the refusal sentence is for.
*
* Waiting is legal here and only here. The reader is the agent's listener
* thread, or the exiting program's own; the writer is a game loop and never
* reaches this. A quarter of a millisecond is well over one compaction and far
* under what a person waiting for a keypress to answer would notice, and eight
* of them bound the whole refusal at two milliseconds. It is [flan_agent.c]'s
* break-loop idiom — nanosleep a step, look again — for its reason too: a spin
* would take a core from the thread being waited on. */
static void flan_reg_scan_wait(void) {
struct timespec step;
step.tv_sec = 0;
step.tv_nsec = 250000; /* 250us */
nanosleep(&step, NULL);
}
/* The pause a walk takes between its own attempts is [flan_reg_wait] above,
* the same one a slot takes: a compaction is what a walk loses to, and it is
* over in well under the step. The rate matters and the pause is not magic — a
* writer rearranging the table more than half the time still gets refused,
* which is the honest answer to a question asked of a table that is never
* still, and is what the refusal sentence is for. */
/* Allocated by flan_dev_reg_enable and null until then, which is the whole of
* what a release build carries: a null pointer, a zero flag, and the load and
@ -1733,7 +1766,7 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
uint64_t at;
n = 0;
missed = 0;
if (!flan_reg_scan_open(&at)) { flan_reg_scan_wait(); continue; }
if (!flan_reg_scan_open(&at)) { flan_reg_wait(); continue; }
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry e;
int64_t j;
@ -1761,14 +1794,14 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
}
n++;
}
if (!flan_reg_scan_ok(at)) { flan_reg_scan_wait(); continue; }
if (!flan_reg_scan_ok(at)) { flan_reg_wait(); continue; }
/* A stable epoch and every slot copied: this is a table that existed. */
if (missed == 0) return n;
/* A stable epoch but slots that would not hold still. Worth another walk —
the writer moves on — and worth remembering the closest one, because it
is the number the refusal quotes. */
if (fewest < 0 || missed < fewest) fewest = missed;
flan_reg_scan_wait();
flan_reg_wait();
}
/* Eight walks, and not one of them saw the whole table. Answering with the
last walk's rows would be answering with a table that never existed, and