JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge: the registry answers honestly under churn

Browse Source
This commit is contained in:
Joseph Ferano 2026-09-18 07:50:48 +07:00
commit 534907d204
6 changed files with 597 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/BUGS-2026-09-18.md
View File

@ -76,10 +76,12 @@ territory; fix or record, the lane's call.
invariant, not a bounds check, but sits behind `if f.md.checks` in both backends
(`emit.ml:1037`, `x86.ml:2164`; same for SliceFromPtr's `n >= 0`). A negative-length
slice reaches user code. `flan_vec_as_slice` checks unconditionally — the model.
- **`reg leaks` lies at >3072 live blocks** (`runtime/flan_dev.c:1521`): compaction then
fires on every allocation, the epoch stays odd, all 8 scan retries fail, and
`flan_dev_reg_by_type` returns 0 — measured 198/200 wrong answers on a full table.
Plus `flan_reg_snap` failure is `continue`d past, contradicting its own contract.
- ~~**`reg leaks` lies at >3072 live blocks**~~ — **fixed**. Compaction now also asks
whether there is an eighth of a table's worth of dead to reclaim, so a table full of
live blocks stops thrashing the epoch; `flan_dev_reg_by_type` returns −1 with an unread
count rather than zero rows, and the agent refuses in a sentence; a note dropped by a
genuinely full table says so on stderr once. `test/dev_limits.c regfull` pins it under
a writer thread: 3100 live blocks, 200 asks, 199 wrong before and 200 right after.
- **Emacs framing**: a truncated frame raises wrong-type instead of the timeout message
and leaves the partial frame in the buffer, desyncing every later request by one frame
(`flan-dev.el:220`); `extract-reply` reads before it deletes, so an unreadable payload

25
docs/BUILT.md
View File

@ -4977,6 +4977,16 @@ and an entry is dropped only when the allocator hands the same address out again
stopped being true. When the table fills it is compacted, dropping the dead and re-inserting the live; in a
long-running program the dead are the bulk of it.
**And only when the dead are worth reclaiming**, which is a correctness rule and not a tuning one. Compaction reclaims
dead entries and nothing else, so "three quarters full" alone made a program holding more than three quarters of the
table *live* compact on every allocation for the rest of its life, reclaiming nothing each time and holding the
table-wide counter odd while it did. A listing racing that loop lost all eight of its attempts and answered "nothing is
held" about a program holding three thousand blocks — 199 wrong answers in 200, measured. The trigger now also asks
whether there are at least an eighth of a table's worth of dead to reclaim, which bounds the cost from the other side
too: a compaction that runs reclaims at least 512 slots, so it cannot run more than once per 512 allocations. A table
that is genuinely full of live blocks stops rearranging itself, drops the note, and says so on stderr once — the flag
every listing already carried was only visible to somebody who asked.
### What it is for: permission, not identification
This is the part worth stating plainly, because the obvious reading is wrong.
@ -5102,6 +5112,21 @@ The whole answer here is the branch.
out, and two walks would drift. Formatting is in the agent and ordering is in the daemon — biggest first, by bytes,
because a breakdown in table order is a list of everything and answers nothing.
**A walk that could not be taken returns −1, never zero rows.** The two are not the same answer and this is the verb
where confusing them costs the most: no rows is what a program that has freed everything looks like, and the person
asking is asking because they suspect the opposite. A slot the writer kept winning is counted rather than stepped over
— `flan_reg_snap`'s zero means "a slot I could not read", and a walk that skipped one and still called itself whole
would be a leak report quietly missing the leak — and the agent turns a refusal into an `err` line naming how many
slots would not hold still, which the daemon already renders as the editor's message.
**And the eight attempts wait between themselves**, which is most of what makes them eight attempts rather than one.
A walk that bails at the epoch check costs almost nothing, so eight of them back to back fit inside the single
compaction they are all losing to. A quarter of a millisecond between them -- `flan_agent.c`'s break-loop idiom, and
legal here because the waiter is the listener thread and never the game loop -- bounds the whole refusal at two
milliseconds, and under a writer churning on top of 3000 live blocks it was the difference between 8 right answers in
200 and 200 in 200. It is not magic: a writer that spends most of its time rearranging the table still gets refused,
which is the honest answer to a question asked of a table that is never still.
**"At exit" is not a hook, and the honest reason is that a game is killed.** A program stopped by a signal runs no
`atexit` handler, no destructor, nothing — so no code written inside the program could report anything about the run
that matters most. The authoritative reader is therefore `(:op "leaks")`, which reads the same table over the agent

212
runtime/flan_dev.c
View File

@ -33,6 +33,10 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* For the one wait in this file: the pause between a listing's attempts at
* reading the table. See flan_reg_scan_wait. Nothing on the writer's side
* waits for anything. */
#include <time.h>
#define FLAN_DEV_MAX 4096
@ -1039,6 +1043,28 @@ void *flan_dev_frame_slot(const void *frame, int32_t i) {
* see flan_dev_reg_note. */
#define FLAN_REG_CAP 4096
/* How many dead entries make a compaction worth running. Not a tuning knob: it
* is the difference between a diagnostic that works on a big program and one
* that lies to it.
*
* The high-water mark alone said "three quarters full, rearrange". But a
* compaction reclaims *dead* entries and nothing else, so a program holding
* more than three quarters of the table in live blocks met that mark on every
* allocation for the rest of its life, and every one of those allocations
* dragged the table-wide epoch odd and back. A listing racing that loop lost
* all eight of its attempts and answered "nothing is held" about a program
* holding three thousand blocks — measured at 198 wrong answers in 200. The
* verb that exists to find a leak reported the absence of one.
*
* So the trigger asks the question the work can actually answer: is there
* enough dead here to be worth the sweep. An eighth of the table is the floor,
* which bounds the cost from the other side too — a compaction that runs
* reclaims at least 512 slots, so it cannot run more than once per 512
* allocations, and the epoch is quiet in between. A table that is full of
* genuinely live blocks now stops rearranging itself and says it is full,
* which is the truth and was always the truth. */
#define FLAN_REG_RECLAIM (FLAN_REG_CAP / 8)
typedef struct {
const char *type; /* the Flan spelling, e.g. "Enemy" or "(Vec i32)" */
int64_t typelen;
@ -1089,7 +1115,18 @@ static void flan_reg_end(flan_reg_entry *e) {
}
/* One slot, copied whole or not at all. 0 means the writer kept winning, which
* a caller reports as a slot it could not read rather than as an empty one. */
* a caller reports as a slot it could not read rather than as an empty one.
*
* [flan_dev_reg_by_type] keeps that: it counts the slots it could not copy and
* refuses the whole listing rather than describing a table it only partly saw,
* because "no rows" is exactly what a program that had freed everything would
* look like. [flan_dev_reg_at] steps past them instead, and is allowed to
* because it is asking a different question — it wants the one block that
* contains an address, and a slot it could not read either did not hold that
* block, in which case skipping it costs nothing, or did, in which case the
* whole call answers "never heard of this address", which is the answer it
* already gives for a stack local and the daemon already gates it to a stopped
* program. */
static int flan_reg_snap(flan_reg_entry *e, flan_reg_entry *out) {
int attempt;
for (attempt = 0; attempt < 64; attempt++) {
@ -1119,6 +1156,33 @@ static int flan_reg_scan_ok(uint64_t at) {
return __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE) == at;
}
/* And a pause between a reader's attempts, which is most of what makes eight
* attempts worth more than one.
*
* Retrying immediately looks like eight chances and is not. A walk that bails
* at the epoch check costs almost nothing, so eight of them back to back fit
* inside the single compaction they are all losing to, and the reader refuses
* having waited for nothing. Measured, with a writer allocating and freeing on
* top of three thousand live blocks: without this pause 8 answers in 200 were
* right and 192 were refusals; with it, 200 of 200. The rate matters and the
* pause is not magic — a writer rearranging the table more than half the time
* still gets refused, which is the honest answer to a question asked of a
* table that is never still, and is what the refusal sentence is for.
*
* Waiting is legal here and only here. The reader is the agent's listener
* thread, or the exiting program's own; the writer is a game loop and never
* reaches this. A quarter of a millisecond is well over one compaction and far
* under what a person waiting for a keypress to answer would notice, and eight
* of them bound the whole refusal at two milliseconds. It is [flan_agent.c]'s
* break-loop idiom — nanosleep a step, look again — for its reason too: a spin
* would take a core from the thread being waited on. */
static void flan_reg_scan_wait(void) {
struct timespec step;
step.tv_sec = 0;
step.tv_nsec = 250000; /* 250us */
nanosleep(&step, NULL);
}
/* Allocated by flan_dev_reg_enable and null until then, which is the whole of
* what a release build carries: a null pointer, a zero flag, and the load and
* not-taken branch each of the hooks below begins with. A fixed array here
@ -1126,10 +1190,42 @@ static int flan_reg_scan_ok(uint64_t at) {
* table nothing in that build ever writes. */
static flan_reg_entry *flan_reg;
static int64_t flan_reg_used; /* live + dead slots in use */
static int64_t flan_reg_dead; /* how many of those have died */
static int64_t flan_reg_seq; /* a monotonic clock, in events */
static int flan_reg_on; /* only a dev build turns this on */
static int flan_reg_full; /* something found no slot */
/* The table filled, and whoever is reading a listing is told twice: by this
* flag, which every listing carries so that its numbers are read as a floor,
* and once on stderr at the moment it happens.
*
* The line is here because the flag alone is only seen by someone who asks.
* A note that finds no slot is dropped — killing a game because its
* diagnostic ran out of room would be the diagnostic shooting the patient —
* and a drop that nobody is told about is the table quietly becoming a
* different table than the one its reader thinks they are reading.
*
* Once, not per note, and the difference is not tidiness. This runs on the
* game thread inside the allocation hook, and stderr in a dev build is a pipe
* the daemon reads; one bounded write is what a frame can afford, and a line
* per note would be sixty a second into a pipe nobody is draining while a
* request is being served. The second line would carry nothing the first did
* not anyway.
*
* [why] is the caller's sentence because the two ways to get here are not the
* same fact and neither is "the table is full of live blocks": a probe that
* ran out of slots found every slot *in use*, which is not the same as every
* slot live, and the caller is the one holding the count that says which. */
static void flan_reg_say_full(const char *why) {
if (flan_reg_full) return;
flan_reg_full = 1;
fprintf(stderr,
"flan: the allocation registry is full (%d blocks) — %s. Blocks "
"noted from here on are dropped, so every listing is a floor and "
"not a count.\n",
FLAN_REG_CAP, why);
}
/* Armed by the program's entry in a dev build. The free-side hooks in
* flan_rt.c are called unconditionally and begin with this load and a
* not-taken branch, because flan_dev.c is linked into every build and a
@ -1163,9 +1259,12 @@ static size_t flan_reg_slot(uintptr_t a) {
& (FLAN_REG_CAP - 1);
}
/* Drop every dead entry and re-insert the live ones. Called when the table
* fills: in a long-running program the dead are the bulk of it, and losing
* them is much cheaper than losing the live half. */
/* Drop every dead entry and re-insert the live ones. Called when the table is
* filling *and* there is a worthwhile number of dead in it: in a long-running
* program the dead are the bulk of it, and losing them is much cheaper than
* losing the live half. When they are not the bulk of it this does nothing but
* move live entries around and hold the epoch odd while it does — see
* FLAN_REG_RECLAIM for the wrong answers that bought. */
static void flan_reg_compact(void) {
size_t bytes = FLAN_REG_CAP * sizeof(flan_reg_entry);
flan_reg_entry *old = (flan_reg_entry *)malloc(bytes);
@ -1173,7 +1272,11 @@ static void flan_reg_compact(void) {
/* Borrowed rather than kept: a second permanent copy would double what a
dev build holds for a rearrangement that happens rarely. If it cannot be
had, the table simply stays as it is and says it is full. */
if (old == NULL) { flan_reg_full = 1; return; }
if (old == NULL) {
flan_reg_say_full("there was no room to make the scratch copy a "
"rearrangement needs");
return;
}
/* Odd for the duration, so a scan that overlapped this throws its counts
away rather than reporting a table half in one arrangement and half in
the other. */
@ -1192,6 +1295,7 @@ static void flan_reg_compact(void) {
flan_reg_end(e);
}
flan_reg_used = 0;
flan_reg_dead = 0; /* the dead are what this drops; none survive the sweep */
for (i = 0; i < FLAN_REG_CAP; i++) {
size_t s;
int64_t probe;
@ -1224,12 +1328,20 @@ void flan_dev_reg_note(void *base, int64_t bytes, int64_t elem,
size_t s;
int64_t probe;
if (!flan_reg_on || a == 0 || bytes <= 0) return;
if (flan_reg_used * 4 > (int64_t)FLAN_REG_CAP * 3) flan_reg_compact();
/* Both halves, and the second is the one that matters: filling is what makes
a rearrangement urgent, but only the dead make it possible. */
if (flan_reg_used * 4 > (int64_t)FLAN_REG_CAP * 3
&& flan_reg_dead >= FLAN_REG_RECLAIM)
flan_reg_compact();
s = flan_reg_slot(a);
for (probe = 0; probe < FLAN_REG_CAP; probe++) {
size_t j = (s + (size_t)probe) & (FLAN_REG_CAP - 1);
if (flan_reg[j].base != 0 && flan_reg[j].base != a) continue;
if (flan_reg[j].base == 0) flan_reg_used++;
/* Reusing the slot of a block that died un-counts that death: the entry
about to be written is live, and the count the compaction trigger reads
is a count of what a sweep would actually reclaim. */
else if (flan_reg[j].died != 0) flan_reg_dead--;
/* The pair a torn read would get wrong is [type] and [typelen], which is
why the whole entry goes under the counter rather than the two of them
being ordered somehow. */
@ -1244,11 +1356,25 @@ void flan_dev_reg_note(void *base, int64_t bytes, int64_t elem,
flan_reg_end(&flan_reg[j]);
return;
}
/* Full of live blocks. Killing the program because it ran out of diagnostic
/* Every slot in use, and the compaction above declined to run because what
* a sweep would reclaim is not worth the sweep. The honest version of that
* quotes the number rather than claiming the table is all live: it usually
* is, and between one dead entry and FLAN_REG_RECLAIM of them it is not, and
* a reader deciding whether this is a leak wants to know which.
*
* The note is dropped. Killing the program because it ran out of diagnostic
* room would be the diagnostic shooting the patient — the watch table's rule
* and the same answer: a flag, readable by whoever asks, so that a missing
* entry is never mistaken for a freed one. */
flan_reg_full = 1;
* entry is never mistaken for a freed one. What is new is that the drop is
* also said out loud, once, rather than only being discoverable by asking. */
{
char why[160];
snprintf(why, sizeof why,
"every slot is in use and only %lld of them have died, too few "
"to be worth rearranging the table for",
(long long)flan_reg_dead);
flan_reg_say_full(why);
}
}
int flan_dev_reg_overflowed(void) { return flan_reg_full; }
@ -1294,6 +1420,7 @@ void flan_dev_reg_dead(void *base) {
flan_reg_begin(&flan_reg[j]);
flan_reg[j].died = ++flan_reg_seq;
flan_reg_end(&flan_reg[j]);
flan_reg_dead++;
}
return;
}
@ -1317,6 +1444,7 @@ void flan_dev_reg_dead_range(void *base, int64_t bytes) {
flan_reg_begin(e);
e->died = now;
flan_reg_end(e);
flan_reg_dead++;
}
}
}
@ -1476,6 +1604,23 @@ int64_t flan_dev_reg_number(const void *p) { return (int64_t)(uintptr_t)p; }
* told so by the number coming back larger than [cap], which is the same
* contract the watch table's count has.
*
* And a walk that could not be taken returns -1 rather than a row count,
* because the two are not the same answer and this verb is the one place in
* the dev loop where confusing them costs the most. "No rows" is what a
* program that has freed everything looks like. "I could not read it" is what
* a program whose game thread is allocating through the whole read looks like,
* and the person asking is asking precisely because they suspect the second
* program of holding something. A zero handed back for a failed read is this
* file answering "nothing is held" about a table it never managed to see —
* the silent wrong answer everything else in here is written to refuse.
*
* [unread], on a -1, says which failure it was: a positive count is slots that
* were being written every time this looked at them, which is [flan_reg_snap]'s
* own contract being kept — its 0 means "a slot I could not read", and it is
* reported as that and never folded into the rows. A zero means no walk ever
* got a stable epoch at all: the table was being rearranged from end to end of
* every attempt. The caller turns either into a sentence.
*
* The grouping is O(rows x types) on string compare. The table is 4096 slots
* and the reader is a person, so this is the side the cost belongs on — the
* same judgement [flan_reg_find] is written down for. Compared by *content*
@ -1483,9 +1628,10 @@ int64_t flan_dev_reg_number(const void *p) { return (int64_t)(uintptr_t)p; }
* call site, and two modules that both allocate an Enemy emit two of them. */
int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
int64_t *bytes, const char **types,
int64_t *typelens, int64_t cap) {
int64_t i, n = 0;
int64_t *typelens, int64_t cap, int64_t *unread) {
int64_t i, n = 0, missed = 0, fewest = -1;
int attempt;
if (unread) *unread = 0;
if (!flan_reg_on) return 0;
/* Read off a running program, which is what makes the counters below
necessary: a row is a (pointer, length) pair that is only meaningful
@ -1496,12 +1642,16 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
for (attempt = 0; attempt < 8; attempt++) {
uint64_t at;
n = 0;
if (!flan_reg_scan_open(&at)) continue;
missed = 0;
if (!flan_reg_scan_open(&at)) { flan_reg_scan_wait(); continue; }
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry e;
int64_t j;
int found = 0;
if (!flan_reg_snap(&flan_reg[i], &e)) continue;
/* Counted, not stepped over. A slot the writer kept winning may hold a
block, and a walk that skipped one and still called itself whole would
be a leak report quietly missing the leak. */
if (!flan_reg_snap(&flan_reg[i], &e)) { missed++; continue; }
if (e.base == 0) continue;
if (live_only && e.died != 0) continue;
for (j = 0; j < n && j < cap; j++) {
@ -1521,13 +1671,21 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
}
n++;
}
if (flan_reg_scan_ok(at)) return n;
if (!flan_reg_scan_ok(at)) { flan_reg_scan_wait(); continue; }
/* A stable epoch and every slot copied: this is a table that existed. */
if (missed == 0) return n;
/* A stable epoch but slots that would not hold still. Worth another walk —
the writer moves on — and worth remembering the closest one, because it
is the number the refusal quotes. */
if (fewest < 0 || missed < fewest) fewest = missed;
flan_reg_scan_wait();
}
/* Eight walks and a compaction through every one of them. Answering with the
last walk's rows would be answering with a table that never existed, so
this answers with none — the caller prints a header saying how many rows
follow, and zero is a number it can print. */
return 0;
/* Eight walks, and not one of them saw the whole table. Answering with the
last walk's rows would be answering with a table that never existed, and
answering with zero rows would be answering "nothing is held" — so this
answers with neither, and the caller says so in a sentence. */
if (unread) *unread = fewest > 0 ? fewest : 0;
return -1;
}
/* ── What is still held when the program returns ──────────────────────
@ -1552,8 +1710,20 @@ static void flan_reg_report(void) {
enum { ROWS = 128 };
int64_t counts[ROWS], bytes[ROWS], typelens[ROWS];
const char *types[ROWS];
int64_t n, i, blocks = 0, held = 0;
n = flan_dev_reg_by_type(1, counts, bytes, types, typelens, ROWS);
int64_t n, i, unread = 0, blocks = 0, held = 0;
n = flan_dev_reg_by_type(1, counts, bytes, types, typelens, ROWS, &unread);
/* At exit, with the program's own thread standing right here, a walk that
cannot settle means another thread is still allocating — and the report is
the one about what is still held, so saying nothing would be the loudest
possible version of the wrong answer. */
if (n < 0) {
fprintf(stderr,
"flan: the allocation registry could not be read at exit (%s), so "
"there is no report of what was still held\n",
unread > 0 ? "slots were being written throughout"
: "the table was being rearranged throughout");
return;
}
if (n == 0) return;
for (i = 0; i < n && i < ROWS; i++) { blocks += counts[i]; held += bytes[i]; }
fprintf(stderr, "flan: %lld block%s still held at exit, %lld bytes\n",

253
test/dev_limits.c
View File

@ -18,6 +18,7 @@
* name table never shrinks and the overflow case aborts the process.
*/
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
@ -54,6 +55,16 @@ static const char *result_read(uint64_t *gen, uint64_t *len) {
}
void flan_rt_init(int32_t argc, char **argv);
void flan_dev_reg_enable(void);
void flan_dev_reg_note(void *base, int64_t bytes, int64_t elem,
const char *type, int64_t typelen);
void flan_dev_reg_dead(void *base);
int flan_dev_reg_overflowed(void);
int64_t flan_dev_reg_count(int32_t live_only);
int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
int64_t *bytes, const char **types,
int64_t *typelens, int64_t cap, int64_t *unread);
#define CAP 4096
/* Emit more than fits, twice the buffer's worth, and report what came back.
@ -118,14 +129,254 @@ static int names(void) {
return 0;
}
/* ── A table full of live blocks, read while it is being written ─────
*
* The shape of the bug this pins, which is worth writing down because the
* mode looks like a stress test and is not one. A compaction reclaims dead
* entries. The trigger used to be "three quarters full" and nothing else, so a
* program holding more than three quarters of the table live compacted on
* every single allocation, for ever, reclaiming nothing each time. Each
* compaction holds the table-wide epoch odd from end to end, a listing retries
* eight times and each retry meets another one, and [flan_dev_reg_by_type]
* used to answer that with zero rows — "nothing is held", about a program
* holding three thousand blocks. It was measured at 198 wrong answers in 200.
*
* So: 3100 live blocks, which is over the old three-quarter mark and under the
* 4096 the table holds, one thread noting allocations as fast as it can, and
* this thread asking the leak question two hundred times. The count printed is
* how many of those answers were right. Nothing here is timing-dependent in
* the direction that matters — a wrong answer is a wrong answer on any
* machine, and the fixed table means "right" is a number this file knows.
*
* The blocks are fake addresses rather than real allocations. The registry is
* told about a block, it never reads one, and 3100 real mallocs would only be
* measuring the allocator. */
enum { REG_LIVE = 3100, REG_ASKS = 200 };
static volatile int reg_stop;
/* The writer: re-note blocks that are already in the table, which is what a
* game loop overwriting its own allocations looks like from in here, and is
* the path that takes the compaction branch. The addresses are the ones
* already recorded, so the live count does not move and the table never
* fills. */
static void *reg_writer(void *arg) {
int64_t n = 0;
(void)arg;
while (!reg_stop) {
for (int i = 0; i < REG_LIVE && !reg_stop; i++) {
void *p = (void *)(uintptr_t)(0x100000 + (uintptr_t)i * 64);
flan_dev_reg_note(p, 64, 8, "Enemy", 5);
n++;
}
}
return (void *)(uintptr_t)n;
}
static int regfull(void) {
enum { ROWS = 16 };
int64_t counts[ROWS], bytes[ROWS], typelens[ROWS], unread;
const char *types[ROWS];
pthread_t w;
int right = 0, refused = 0, wrong = 0, zero = 0;
flan_dev_reg_enable();
for (int i = 0; i < REG_LIVE; i++) {
void *p = (void *)(uintptr_t)(0x100000 + (uintptr_t)i * 64);
flan_dev_reg_note(p, 64, 8, "Enemy", 5);
}
printf("live %lld\n", (long long)flan_dev_reg_count(1));
if (pthread_create(&w, NULL, reg_writer, NULL) != 0) {
printf("no writer thread\n");
return 2;
}
for (int a = 0; a < REG_ASKS; a++) {
int64_t n = flan_dev_reg_by_type(1, counts, bytes, types, typelens, ROWS,
&unread);
/* One type, 3100 blocks of it. Any other answer is this table described as
something it is not — and zero rows is the specific lie, because it is
the answer a program that had freed everything would get. */
if (n < 0) refused++;
else if (n == 0) zero++;
else if (n == 1 && counts[0] == REG_LIVE) right++;
else wrong++;
}
reg_stop = 1;
pthread_join(w, NULL);
printf("right %d\n", right);
printf("zero %d\n", zero);
printf("wrong %d\n", wrong);
printf("refused %d\n", refused);
return 0;
}
/* ── The dead count the compaction trigger reads ─────────────────────
*
* [regfull] above proves a table of live blocks stops compacting. This proves
* the other half still does, which is the failure that would be silent: the
* trigger now asks how many entries are dead, four places maintain that number
* — a death, an arena's free-all, a dead slot being written over, and the sweep
* itself, which drops all of them — and a count that drifted low would simply
* stop reclaiming. Nothing would crash. The table would pin at its cap, the
* overflow flag would come on, and every listing from then on would be a floor
* quietly described as a count.
*
* So: three thousand blocks that stay live, and a pool of six hundred noted and
* freed over and over on top of them. Used passes the three-quarter mark, the
* dead pass the reclaim floor several times over the run, and the sweep runs.
* The assertion that catches a drifted count is the last one — a table that
* never reclaimed would have overflowed long before this returns. */
enum { CHURN_LIVE = 3000, CHURN_POOL = 600, CHURN_PASSES = 8 };
static void *churn_addr(int i) {
return (void *)(uintptr_t)(0x800000 + (uintptr_t)i * 64);
}
static int regchurn(void) {
enum { ROWS = 16 };
int64_t counts[ROWS], bytes[ROWS], typelens[ROWS], unread, n;
const char *types[ROWS];
flan_dev_reg_enable();
for (int i = 0; i < CHURN_LIVE; i++)
flan_dev_reg_note(churn_addr(i), 64, 8, "Enemy", 5);
/* Noted and freed on top of the live set. A note landing on the slot of a
* block that died is the decrement; the free is the increment; the sweep in
* between is the reset. */
for (int pass = 0; pass < CHURN_PASSES; pass++)
for (int i = 0; i < CHURN_POOL; i++) {
void *p = churn_addr(CHURN_LIVE + i);
flan_dev_reg_note(p, 32, 4, "Bullet", 6);
flan_dev_reg_dead(p);
}
printf("live %lld\n", (long long)flan_dev_reg_count(1));
n = flan_dev_reg_by_type(1, counts, bytes, types, typelens, ROWS, &unread);
printf("rows %lld\n", (long long)n);
printf("blocks %lld\n", (long long)(n == 1 ? counts[0] : -1));
/* The one that catches a dead count that drifted: a table that stopped
reclaiming fills, and a full table is a floor for the rest of the run. */
printf("overflowed %d\n", flan_dev_reg_overflowed());
return 0;
}
/* And a table that really is full, which is the case the trigger above must
* not paper over: 4096 live blocks and nothing dead anywhere, then one more.
* The note is dropped — that is the standing decision, and dying because a
* diagnostic ran out of room would be worse — and what is checked here is that
* the drop is *said*, once, rather than only being discoverable by asking the
* flag. A second and third dropped note must add nothing. */
static int regoverflow(void) {
flan_dev_reg_enable();
for (int i = 0; i < CAP; i++)
flan_dev_reg_note((void *)(uintptr_t)(0x2000000 + (uintptr_t)i * 64), 64, 8,
"Enemy", 5);
printf("live %lld\n", (long long)flan_dev_reg_count(1));
printf("overflowed %d\n", flan_dev_reg_overflowed());
fflush(stdout);
for (int i = 0; i < 3; i++)
flan_dev_reg_note((void *)(uintptr_t)(0x9000000 + (uintptr_t)i * 64), 64, 8,
"Latecomer", 9);
printf("overflowed %d\n", flan_dev_reg_overflowed());
printf("live %lld\n", (long long)flan_dev_reg_count(1));
return 0;
}
/* ── A listing taken while the table is actually being rearranged ────
*
* [regfull] is a table of live blocks, where the fix is that no compaction
* runs at all. This is the other race: a writer churning hard enough that
* compactions do run, under a reader asking the leak question through them.
*
* What it pins is not a rate. Rates here are the machine's, and the counts
* printed are deliberately the two that cannot come out any other way on any
* machine: [zero], an answer of no rows, which is the original lie and the one
* shape a program holding three thousand blocks must never produce; and
* [low], a count that is neither right nor a refusal, which is what a torn
* read or a walk that straddled a compaction would produce. Right-versus-
* refused is the interesting number and it is printed rather than asserted,
* because a refusal is an honest answer and how often it comes up is the
* writer's business. It moves a lot: on the machine this was written on, the
* paced retry turned 8 right in 200 into 200 in 200 at one churn rate and
* changed nothing at all at another.
*
* The throttle is a spin and not a sleep because the whole interesting range
* is under nanosleep's floor of tens of microseconds. */
enum { RACE_LIVE = 3000, RACE_ASKS = 200, RACE_SPIN = 1000 };
static volatile int race_stop;
static void *race_writer(void *arg) {
long long i = 0;
(void)arg;
while (!race_stop) {
/* Noted and freed, which is what makes the dead count climb and the
* compaction actually run — the difference from [regfull]'s writer, which
* re-notes live blocks and therefore never earns a sweep. */
void *p = (void *)(uintptr_t)(0x40000000 + (uintptr_t)(i % 100000) * 64);
flan_dev_reg_note(p, 32, 4, "Bullet", 6);
flan_dev_reg_dead(p);
for (volatile long k = 0; k < RACE_SPIN; k++) { }
i++;
}
return NULL;
}
static int regrace(void) {
enum { ROWS = 16 };
int64_t counts[ROWS], bytes[ROWS], typelens[ROWS], unread;
const char *types[ROWS];
pthread_t w;
int right = 0, refused = 0, zero = 0, low = 0;
flan_dev_reg_enable();
for (int i = 0; i < RACE_LIVE; i++)
flan_dev_reg_note((void *)(uintptr_t)(0x100000 + (uintptr_t)i * 64), 64, 8,
"Enemy", 5);
if (pthread_create(&w, NULL, race_writer, NULL) != 0) {
printf("no writer thread\n");
return 2;
}
for (int a = 0; a < RACE_ASKS; a++) {
int64_t n =
flan_dev_reg_by_type(1, counts, bytes, types, typelens, ROWS, &unread);
if (n < 0) refused++;
else if (n == 0) zero++;
else {
int64_t enemies = 0;
for (int64_t j = 0; j < n && j < ROWS; j++)
if (typelens[j] == 5) enemies = counts[j];
if (enemies == RACE_LIVE) right++; else low++;
}
}
race_stop = 1;
pthread_join(w, NULL);
printf("zero %d\n", zero);
printf("low %d\n", low);
printf("answered %d refused %d\n", right, refused);
return 0;
}
int main(int argc, char **argv) {
flan_rt_init(argc, argv);
if (argc < 2) {
fprintf(stderr, "usage: %s cap|names\n", argv[0]);
fprintf(stderr,
"usage: %s cap|names|regfull|regchurn|regrace|regoverflow\n",
argv[0]);
return 2;
}
if (strcmp(argv[1], "cap") == 0) return cap();
if (strcmp(argv[1], "names") == 0) return names();
if (strcmp(argv[1], "regfull") == 0) return regfull();
if (strcmp(argv[1], "regchurn") == 0) return regchurn();
if (strcmp(argv[1], "regrace") == 0) return regrace();
if (strcmp(argv[1], "regoverflow") == 0) return regoverflow();
fprintf(stderr, "unknown mode %s\n", argv[1]);
return 2;
}

87
test/test_reload.ml
View File

@ -537,6 +537,93 @@ let () =
if not (has err "out of dev name slots") then
fail "the registry overflowed without saying so: %S" err;
(* And the allocation registry read while it is being written, which is
the one of the three that was answering wrongly rather than not being
reached. 3100 live blocks is past the three-quarter mark that used to
trigger a compaction and short of the 4096 the table holds, so every
note taken by the writer thread used to rearrange a table with nothing
dead in it to reclaim — holding the epoch odd, losing every one of a
reader's eight attempts, and answering "no rows" about a program
holding three thousand blocks. Measured at 199 of 200 asks before the
trigger learned to ask whether a sweep would reclaim anything.
The check is on the answers and not on timing: zero rows here is the
exact shape of the wrong answer, because it is what a program that had
freed everything would say. A refusal would be honest and is allowed —
it is what a genuinely unreadable table earns — but it should not be
what this one earns, so it is counted separately and expected to be
none. *)
let code, out, err = mode "regfull" in
let want_reg = "live 3100\nright 200\nzero 0\nwrong 0\nrefused 0\n" in
if code <> 0 || out <> want_reg then
fail
"the registry read under a writer\n got: %S (exit %d, err %S)\n wanted: %S"
out code err want_reg;
(* The other half of the same trigger: it still compacts when compacting
would reclaim something. This one would fail silently if the dead count
the trigger reads ever drifted low — nothing crashes, the table simply
stops reclaiming, pins at its cap, and every listing after that is a
floor being read as a count. [overflowed 0] is the line that catches it:
3000 blocks held live with 600 more noted and freed on top of them, over
and over, fills a 4096-slot table several times over unless the sweep is
running. *)
let code, out, err = mode "regchurn" in
let want_churn = "live 3000\nrows 1\nblocks 3000\noverflowed 0\n" in
if code <> 0 || out <> want_churn then
fail
"the registry under churn\n got: %S (exit %d, err %S)\n wanted: %S"
out code err want_churn;
(* A listing taken while the table really is being rearranged, which
neither case above covers: [regfull]'s table never compacts, and the
churn above is single-threaded, so the epoch retry — the thing that
decides between an answer and a refusal — was exercised by nothing.
What is asserted is only what cannot come out differently on a slower
machine: never an answer of no rows, which is the original lie, and
never a count that is neither right nor refused, which is what a walk
that straddled a compaction would produce. How the rest divides between
answered and refused is the writer's business and is printed rather
than pinned — a refusal is an honest answer, and a case that went red
because a shared runner was busy would teach whoever runs this to skim
past red. *)
let code, out, err = mode "regrace" in
let first_two =
match String.split_on_char '\n' out with
| a :: b :: _ -> a ^ "\n" ^ b ^ "\n"
| _ -> out
in
if code <> 0 || first_two <> "zero 0\nlow 0\n" then
fail
"a listing taken during a compaction\n got: %S (exit %d, err %S)\n wanted no zero-row and no wrong-count answers"
out code err;
(* And a table that is genuinely full, which is the state the trigger above
must not paper over. The note is dropped — a diagnostic that killed the
program because it ran out of room would be the diagnostic shooting the
patient — and the decision this pins is that the drop is *said*, once.
Once matters: this is the game thread inside the allocation hook, and a
line per dropped note would be sixty a second down a pipe nobody drains
while a request is being served. *)
let code, out, err = mode "regoverflow" in
let want_over = "live 4096\noverflowed 0\noverflowed 1\nlive 4096\n" in
if code <> 0 || out <> want_over then
fail "a full registry\n got: %S (exit %d)\n wanted: %S" out
code want_over;
let said_full =
let needle = "the allocation registry is full" in
let rec go i n =
if i + String.length needle > String.length err then n
else if String.sub err i (String.length needle) = needle then
go (i + 1) (n + 1)
else go (i + 1) n
in
go 0 0
in
if said_full <> 1 then
fail "a full registry said so %d times, not once: %S" said_full err;
Printf.printf
"reload: emit %.1fms llc %.1fms ld %.1fms (v2: emit %.1fms llc %.1fms ld %.1fms) host run %.1fms\n"
emit_ms t1.Build.llc_ms t1.Build.link_ms emit2_ms t2.Build.llc_ms

39
vendor/agent/flan_agent.c vendored
View File

@ -108,7 +108,7 @@ int32_t flan_dev_reg_at(const void *p, const char **type, int64_t *typelen,
int64_t *seq, int64_t *died);
int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
int64_t *bytes, const char **types,
int64_t *typelens, int64_t cap);
int64_t *typelens, int64_t cap, int64_t *unread);
int flan_dev_reg_enabled(void);
int flan_dev_reg_overflowed(void);
@ -1147,7 +1147,7 @@ static void handle_line(char *line, sink *o) {
int64_t *bytes = malloc(REG_ROWS * sizeof *bytes);
int64_t *typelens = malloc(REG_ROWS * sizeof *typelens);
const char **types = malloc(REG_ROWS * sizeof *types);
int64_t n, i;
int64_t n, i, unread = 0;
int live_only = line[4] == 'l';
if (counts == NULL || bytes == NULL || typelens == NULL || types == NULL) {
free(counts); free(bytes); free(typelens); free(types);
@ -1160,7 +1160,40 @@ static void handle_line(char *line, sink *o) {
return;
}
n = flan_dev_reg_by_type(live_only ? 1 : 0, counts, bytes, types, typelens,
REG_ROWS);
REG_ROWS, &unread);
/* A read that did not settle is refused in a sentence rather than sent as
* a header saying zero rows. The header cannot carry the difference: a row
* count is a number, and the number that means "I could not look" is the
* same number that means "nothing is held" — which is the answer this verb
* is most often asked to disprove. An [err] line is already a shape the
* daemon turns into the editor's message, so the distinction survives all
* the way to the person who asked.
*
* Both sentences name the *program's* behaviour and not the mechanism,
* because the mechanism is not actionable and the behaviour is: a table
* being rewritten this hard is a game thread allocating flat out, and the
* useful next move is to ask again, or to ask it while stopped. */
if (n < 0) {
free(counts); free(bytes); free(typelens); free(types);
if (unread > 0) {
char msg[192];
int k = snprintf(msg, sizeof msg,
"err the allocation registry could not be read whole: "
"%lld slot%s were being written through every one of "
"eight attempts, so any rows here would be missing "
"blocks; ask again\n",
(long long)unread, unread == 1 ? "" : "s");
/* Clamped, because [snprintf] returns the length it wanted rather
* than the length it wrote — [flan_dev_reg_emit]'s rule. */
if (k > (int)sizeof msg - 1) k = (int)sizeof msg - 1;
if (k > 0) emit(o, msg, (size_t)k);
} else {
reply(o, "err the allocation registry was being rearranged through "
"every one of eight attempts; no listing taken, rather than "
"one of a table that never existed — ask again\n");
}
return;
}
{
char hdr[64];
int k = snprintf(hdr, sizeof hdr, "%lld %d\n",