JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

The allocation registry stops being read out from under its writer

Browse Source 
The writer is the game thread, in every allocation and every free; the reader
is the agent's listener, on a program that is running. Nothing stood between
them. The consequence is not a slightly wrong count: a row's type pointer and
its length mean nothing apart, and a reader that takes the new pointer with the
old length reads off the end of a string literal.

Each entry now carries the watch table's seqlock, odd while it is written, and
a reader copies the entry and re-reads the counter before believing it. The
compaction bumps a table-wide counter around itself, because it moves entries
between slots and no per-slot counter can describe that; a scan that sees that
counter move walks again. It clears the table slot by slot rather than with one
memset, since the memset would zero the counters a reader was holding.

The breakdown and the leak report stay answerable while the program runs, which
is the moment they are for. reg at does not: whether one address is still live
is exactly what a running program is changing, so it is refused the way every
break verb is refused, which is what the daemon already did on its own side.
This commit is contained in:
Joseph Ferano 2026-09-17 22:37:19 +07:00
parent 594a42b54e
commit 9d10e7edb0
3 changed files with 212 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/BUILT.md
View File

@ -4961,6 +4961,17 @@ The two sides of the table therefore look different, and the difference is which
what it handed out, so the region is matched against the table rather than the other way round. That is a real
per-frame cost in a dev build and it is named here rather than discovered later.
**Two threads, so the table has the watch table's seqlock.** The writer is the game thread, inside every allocation and
every free; the reader is the agent's listener, and the two listing verbs are asked of a *running* program — "what is
still held" is the question asked in the last moment before a game is killed, which is not a moment anything is stopped
in. So the frame chain's answer, snapshot it while the thread is parked, is not available here. Each entry carries its
own counter, odd while it is written; a reader copies the entry and re-reads the counter, and a compaction bumps a
table-wide counter around itself because it moves entries between slots and a per-slot counter cannot describe that.
The pair this protects is `type` and `typelen`: they mean nothing apart, and a reader holding the new pointer with the
old length reads off the end of a string literal. `reg at`, the one verb that makes a claim about a single address
rather than describing the program, is refused while running instead — the daemon already refused it, and the agent now
says so too.
Dead entries are kept. That is the second thing the registry buys — an address that was freed still names what died —
and an entry is dropped only when the allocator hands the same address out again, which is exactly when the old answer
stopped being true. When the table fills it is compacted, dropping the dead and re-inserting the live; in a

216
runtime/flan_dev.c
View File

@ -1047,8 +1047,78 @@ typedef struct {
int64_t elem; /* one element's size, or 0 if it is not an array */
int64_t seq; /* when it was made */
int64_t died; /* when it was released, or 0 while it is live */
uint64_t gen; /* this slot's own seqlock; odd while it is written */
} flan_reg_entry;
/* ── Why this table has a seqlock and the watch table's is the model ───
*
* The writer is the game thread, inside every allocation and every free. The
* reader is the agent's listener thread, on a program that is *running* — the
* two listing verbs answer "where did the memory go" and "what is still held",
* and the second of those is asked in the last moment before a game is killed,
* which is a moment the program is not stopped in. So the frame chain's answer
* — snapshot it while the thread is parked — is not available to this table,
* and a plain read of it is a read of eight words another thread is in the
* middle of writing.
*
* The consequence is not a slightly wrong number. [type] and [typelen] are a
* pointer and a length that are only meaningful together, and a reader that
* takes the new pointer with the old length reads off the end of a string
* literal. That is the failure this closes.
*
* Per slot, exactly as [watch_slot] does it: odd while a write is in flight,
* even when it is whole, and a reader copies the slot and re-reads the counter
* to find out whether what it copied ever existed. The compaction is the one
* thing a per-slot counter cannot describe, because it moves entries between
* slots — so it bumps a table-wide counter around itself and a scan that sees
* that counter move starts again. Nothing here blocks the writer: a reader
* that cannot get a clean read gives up after a bounded number of attempts,
* which is the rule everywhere else in this file. */
static uint64_t flan_reg_epoch; /* odd while the table is being compacted */
static void flan_reg_begin(flan_reg_entry *e) {
__atomic_store_n(&e->gen, e->gen | 1, __ATOMIC_RELAXED);
__atomic_thread_fence(__ATOMIC_RELEASE);
}
/* Back to even, so a reader that sees the new count sees the whole entry. The
* [| 1] is [watch]'s: a write abandoned by a break taken inside it must still
* land on an even count. */
static void flan_reg_end(flan_reg_entry *e) {
__atomic_store_n(&e->gen, (e->gen | 1) + 1, __ATOMIC_RELEASE);
}
/* One slot, copied whole or not at all. 0 means the writer kept winning, which
* a caller reports as a slot it could not read rather than as an empty one. */
static int flan_reg_snap(flan_reg_entry *e, flan_reg_entry *out) {
int attempt;
for (attempt = 0; attempt < 64; attempt++) {
uint64_t g1 = __atomic_load_n(&e->gen, __ATOMIC_ACQUIRE);
if (g1 & 1) continue; /* a write is in progress */
*out = *e;
/* Ordered before the second read of the counter, or the check is of a copy
* the compiler was free to make afterwards. */
__atomic_thread_fence(__ATOMIC_ACQUIRE);
if (__atomic_load_n(&e->gen, __ATOMIC_ACQUIRE) == g1) return 1;
}
return 0;
}
/* The table-wide counter, read on the way into a scan and again on the way
* out: a compaction between the two moved entries, so the scan saw some of
* them twice and some not at all. */
static int flan_reg_scan_open(uint64_t *at) {
uint64_t g = __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE);
if (g & 1) return 0;
*at = g;
return 1;
}
static int flan_reg_scan_ok(uint64_t at) {
__atomic_thread_fence(__ATOMIC_ACQUIRE);
return __atomic_load_n(&flan_reg_epoch, __ATOMIC_ACQUIRE) == at;
}
/* Allocated by flan_dev_reg_enable and null until then, which is the whole of
* what a release build carries: a null pointer, a zero flag, and the load and
* not-taken branch each of the hooks below begins with. A fixed array here
@ -1104,8 +1174,23 @@ static void flan_reg_compact(void) {
dev build holds for a rearrangement that happens rarely. If it cannot be
had, the table simply stays as it is and says it is full. */
if (old == NULL) { flan_reg_full = 1; return; }
/* Odd for the duration, so a scan that overlapped this throws its counts
away rather than reporting a table half in one arrangement and half in
the other. */
__atomic_store_n(&flan_reg_epoch, flan_reg_epoch | 1, __ATOMIC_RELAXED);
__atomic_thread_fence(__ATOMIC_RELEASE);
memcpy(old, flan_reg, bytes);
memset(flan_reg, 0, bytes);
/* Cleared slot by slot under each slot's own counter rather than by one
memset over the array: the memset would zero the counters themselves, and
a reader holding one would then validate a read of an entry that was
rewritten underneath it. */
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry *e = &flan_reg[i];
flan_reg_begin(e);
e->type = NULL; e->typelen = 0; e->base = 0;
e->bytes = 0; e->elem = 0; e->seq = 0; e->died = 0;
flan_reg_end(e);
}
flan_reg_used = 0;
for (i = 0; i < FLAN_REG_CAP; i++) {
size_t s;
@ -1115,13 +1200,19 @@ static void flan_reg_compact(void) {
for (probe = 0; probe < FLAN_REG_CAP; probe++) {
size_t j = (s + (size_t)probe) & (FLAN_REG_CAP - 1);
if (flan_reg[j].base == 0) {
flan_reg[j] = old[i];
flan_reg_entry *e = &flan_reg[j];
flan_reg_begin(e);
e->type = old[i].type; e->typelen = old[i].typelen;
e->base = old[i].base; e->bytes = old[i].bytes;
e->elem = old[i].elem; e->seq = old[i].seq; e->died = old[i].died;
flan_reg_end(e);
flan_reg_used++;
break;
}
}
}
free(old);
__atomic_store_n(&flan_reg_epoch, (flan_reg_epoch | 1) + 1, __ATOMIC_RELEASE);
}
/* One note per allocation. [base] replaces whatever was recorded there, live
@ -1139,6 +1230,10 @@ void flan_dev_reg_note(void *base, int64_t bytes, int64_t elem,
size_t j = (s + (size_t)probe) & (FLAN_REG_CAP - 1);
if (flan_reg[j].base != 0 && flan_reg[j].base != a) continue;
if (flan_reg[j].base == 0) flan_reg_used++;
/* The pair a torn read would get wrong is [type] and [typelen], which is
why the whole entry goes under the counter rather than the two of them
being ordered somehow. */
flan_reg_begin(&flan_reg[j]);
flan_reg[j].type = type;
flan_reg[j].typelen = typelen;
flan_reg[j].base = a;
@ -1146,6 +1241,7 @@ void flan_dev_reg_note(void *base, int64_t bytes, int64_t elem,
flan_reg[j].elem = elem;
flan_reg[j].seq = ++flan_reg_seq;
flan_reg[j].died = 0;
flan_reg_end(&flan_reg[j]);
return;
}
/* Full of live blocks. Killing the program because it ran out of diagnostic
@ -1194,7 +1290,11 @@ void flan_dev_reg_dead(void *base) {
size_t j = (s + (size_t)probe) & (FLAN_REG_CAP - 1);
if (flan_reg[j].base == 0) return; /* never noted; nothing to mark */
if (flan_reg[j].base != a) continue;
if (flan_reg[j].died == 0) flan_reg[j].died = ++flan_reg_seq;
if (flan_reg[j].died == 0) {
flan_reg_begin(&flan_reg[j]);
flan_reg[j].died = ++flan_reg_seq;
flan_reg_end(&flan_reg[j]);
}
return;
}
}
@ -1213,7 +1313,11 @@ void flan_dev_reg_dead_range(void *base, int64_t bytes) {
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry *e = &flan_reg[i];
if (e->base == 0 || e->died != 0) continue;
if (e->base >= lo && e->base < hi) e->died = now;
if (e->base >= lo && e->base < hi) {
flan_reg_begin(e);
e->died = now;
flan_reg_end(e);
}
}
}
@ -1305,15 +1409,39 @@ int64_t flan_dev_reg_count(int32_t live_only) {
int32_t flan_dev_reg_at(const void *p, const char **type, int64_t *typelen,
int64_t *off, int64_t *bytes, int64_t *elem,
int64_t *seq, int64_t *died) {
flan_reg_entry *e = flan_reg_on ? flan_reg_find((uintptr_t)p) : NULL;
if (e == NULL) return 0;
if (type) *type = e->type;
if (typelen) *typelen = e->typelen;
if (off) *off = (int64_t)((uintptr_t)p - e->base);
if (bytes) *bytes = e->bytes;
if (elem) *elem = e->elem;
if (seq) *seq = e->seq;
if (died) *died = e->died;
/* The one reader of the containment lookup that is not on the game thread —
* [flan_dev_reg_live] and [flan_dev_reg_emit] above run inside a render
* thunk, which is the writer's own thread — so this one copies each slot
* under its counter instead of pointing into the table. See the seqlock
* note above the entry type. */
uintptr_t a = (uintptr_t)p;
flan_reg_entry best, cur;
int have = 0, attempt;
if (!flan_reg_on || a == 0) return 0;
for (attempt = 0; attempt < 8; attempt++) {
uint64_t at;
int64_t i;
have = 0;
if (!flan_reg_scan_open(&at)) continue;
for (i = 0; i < FLAN_REG_CAP; i++) {
if (!flan_reg_snap(&flan_reg[i], &cur)) continue;
if (cur.base == 0) continue;
if (a < cur.base || a >= cur.base + (uintptr_t)cur.bytes) continue;
/* A live block wins over a dead one covering the same address: the dead
entry is a stale answer the allocator has already contradicted. */
if (!have || (best.died != 0 && cur.died == 0)) { best = cur; have = 1; }
}
if (flan_reg_scan_ok(at)) break;
have = 0;
}
if (!have) return 0;
if (type) *type = best.type;
if (typelen) *typelen = best.typelen;
if (off) *off = (int64_t)(a - best.base);
if (bytes) *bytes = best.bytes;
if (elem) *elem = best.elem;
if (seq) *seq = best.seq;
if (died) *died = best.died;
return 1;
}
@ -1357,31 +1485,49 @@ int64_t flan_dev_reg_by_type(int32_t live_only, int64_t *counts,
int64_t *bytes, const char **types,
int64_t *typelens, int64_t cap) {
int64_t i, n = 0;
int attempt;
if (!flan_reg_on) return 0;
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry *e = &flan_reg[i];
int64_t j;
int found = 0;
if (e->base == 0) continue;
if (live_only && e->died != 0) continue;
for (j = 0; j < n && j < cap; j++) {
if (typelens[j] != e->typelen) continue;
if (memcmp(types[j], e->type, (size_t)e->typelen) != 0) continue;
counts[j]++;
bytes[j] += e->bytes;
found = 1;
break;
/* Read off a running program, which is what makes the counters below
necessary: a row is a (pointer, length) pair that is only meaningful
together, and the memcmp two lines down is where a torn one would read off
the end of a string literal. The whole walk is retried when a compaction
ran through the middle of it, since entries moved and the counts would
hold some blocks twice and some not at all. */
for (attempt = 0; attempt < 8; attempt++) {
uint64_t at;
n = 0;
if (!flan_reg_scan_open(&at)) continue;
for (i = 0; i < FLAN_REG_CAP; i++) {
flan_reg_entry e;
int64_t j;
int found = 0;
if (!flan_reg_snap(&flan_reg[i], &e)) continue;
if (e.base == 0) continue;
if (live_only && e.died != 0) continue;
for (j = 0; j < n && j < cap; j++) {
if (typelens[j] != e.typelen) continue;
if (memcmp(types[j], e.type, (size_t)e.typelen) != 0) continue;
counts[j]++;
bytes[j] += e.bytes;
found = 1;
break;
}
if (found) continue;
if (n < cap) {
types[n] = e.type;
typelens[n] = e.typelen;
counts[n] = 1;
bytes[n] = e.bytes;
}
n++;
}
if (found) continue;
if (n < cap) {
types[n] = e->type;
typelens[n] = e->typelen;
counts[n] = 1;
bytes[n] = e->bytes;
}
n++;
if (flan_reg_scan_ok(at)) return n;
}
return n;
/* Eight walks and a compaction through every one of them. Answering with the
last walk's rows would be answering with a table that never existed, so
this answers with none — the caller prints a header saying how many rows
follow, and zero is a number it can print. */
return 0;
}
/* ── What is still held when the program returns ──────────────────────

24
vendor/agent/flan_agent.c vendored
View File

@ -1057,10 +1057,11 @@ static void handle_line(char *line, sink *o) {
* (Ptr T) to read a type off, and the recorded name is the whole of what
* there is to go on.
*
* Answered while the program is running as well as while it is stopped:
* this reads a table, not a stack, and nothing here walks a chain another
* thread is pushing. Whether the *answer* holds still long enough to be
* worth acting on is the caller's judgement, and the daemon makes it.
* It used to say here that this is answered while the program runs as well
* as while it is stopped, on the grounds that a table is not a stack. The
* table is not a stack and it is still written by the other thread, and the
* answer to one address is a claim that stops being true as it is made — so
* the gate below now says what the daemon already said.
*
* ADDR is read with base 0, so both 0x-hex and decimal arrive; an editor
* that has an address as text has it in one of those two spellings. */
@ -1069,6 +1070,21 @@ static void handle_line(char *line, sink *o) {
int64_t typelen = 0, off = 0, bytes = 0, elem = 0, seq = 0, died = 0;
char *end = NULL;
unsigned long long a;
/* Stopped only, like every break verb. Whether an address is still live is
* exactly what a running program is changing, so the answer would describe
* a table the game thread has already moved on from — the daemon refuses
* the question for that reason before it ever reaches here
* (lib/dev.ml, [inspect_addr]), and this says the same thing to anything
* else that asks. The two listing verbs below are the opposite case and
* stay answerable while running: a breakdown is a description of the
* program and not a claim about one address, and the table carries its own
* seqlock so that reading it while it is written is safe. */
if (!(atomic_load(&depth) > 0)) {
reply(o, "err not stopped: whether one address is still live is what a "
"running program is changing, so this is read from a stopped "
"one\n");
return;
}
if (!flan_dev_reg_enabled()) {
reply(o, "err the allocation registry is off; this is not a dev build\n");
return;