CLHS 4.3.6's update protocol, minus the user hook, on the dyn side's
defclass. Redefining a class used to be silent: a class is sugar for a
constructor defn, so the edit replaced a body and the instances already in
the program kept their old keys for ever.
Three pieces. A registry in flan_dyn.c holding each class's current slot
list and a generation, made only of interned kw_entry pointers so the
collector has nothing to trace in it and no root to push for it. A uint32
generation on the instance, fitted into the padding kind and mark leave in
front of len's alignment — sizeof(flan_obj) is 48 with it and was 48
without, and flan_dyn_obj_size is there so a later field that moves it
fails a test. And a registration thunk per reload, run by the agent
through flan_reload_call after the module's bodies are published: it has
to be a thunk, because the case this exists for is a class redefined and
not constructed.
Migration is lazy, at want_map, len's map arm and dyn_equal's. Slots kept
by name, gained slots nil, dropped slots gone, identity preserved, entries
rebuilt in the class's order so a migrated instance is indistinguishable
from a fresh one. Equality migrates both operands first, so it is over the
class as it is now.
The session had to stop refusing the constructor's signature change, and
does so only for a defclass and only when no compiled caller is left
behind. The checker gets there first in practice; the walk in eval holds
the reason locally rather than inheriting it.
The registry is advisory: a class instance is an open map, so a key a raw
put wrote that the class never declared is dropped by the next migration.
FIX.org says that plainly rather than pretending enforcement.
Ten test binaries share a directory and had shared nothing in it but
watchdog.ml. Everything else each one needed it wrote out again: the
failure counter and its FAIL line, the three-line report tail, a poll,
a socket connect, the wait for a [flan dev] daemon to bind, a substring
search, and the Load -> Check -> Reach.link front half of a compile.
[listening] was the clearest case. Three copies, byte for byte apart
from one comment, and two of them said in that comment that they were
kept separate because "these three files have no module between them".
That was not true when it was written: watchdog.ml was already named in
the same (modules ...) stanzas. test_support.ml is the second such
module, wired the same way, and those two sentences go with the copies
they were explaining.
test_repl.ml's [quote] was Wire.quote character for character, in a file
that already links Wire and already names Wire.quote in a comment about
what the case below it is checking. It is Wire.quote now.
One real behaviour change, and it is a fix. [connect] existed twice over
with different retries: the agent's narrowed to ECONNREFUSED with a
comment saying why -- the socket file appears at bind, a moment before
listen -- while dev's and repl's retried any Unix_error, which meant an
ENOENT or an EACCES was retried to the full timeout before raising
something the reader still had to interpret. The shared one takes the
narrow version. Every caller connects to a socket [listening] has
already seen on disk, so the race it does catch is the only one left.
The rest is left where it is, on purpose. The three output-capturing
[run]s differ in what they wrap -- a pid suffix, a sanitizer environment,
a valgrind invocation -- and are not the same function. The report tails
in test_repl, test_web and the two sweep binaries print different things
for different reasons. The per-file scratch prefixes are the feature that
keeps two suites running at once from unlinking each other's sockets, so
the shared helper takes the prefix rather than choosing one. And the
[match Sys.command "command -v clang ..."] probes stay as they are:
their skip lines are output this suite pins.
bin/main.ml has the compile pipeline written out twice more. Left alone
-- this was a test/-scoped change and bin/ should not be reaching into a
test module -- and noted in FIX.org as what it actually needs, which is
the pipeline moving into lib/.
dune test: exit 0, and its output is the same line for line once the
temp-directory hash and the millisecond counts are normalised.
flan_merged_park called flan_dyn_root_reset, which emptied the collector's
root stack. The frames' roots had to go — main is left by longjmp, so they
name stack the next run overwrites — but the dyn globals' roots are on that
same stack, pushed once by the emitted main and never popped, and the park
took them with the frames.
The park is not a quiet state. It services evaluated thunks, a thunk
allocates, and an allocation collects. So a program with (defvar config dyn)
answered (get config :s) with its string before any thunk ran and with nil
after one that allocated past the heap's floor — a read of memory the sweep
had freed, answering nil by luck of what the freed words decoded as.
The emitted main now brackets its global pushes: flan_dyn_root_globals_begin
empties the stack, the pushes go on, flan_dyn_root_globals_end records how
many of them there are, and the park resets to that line instead of to zero.
Nothing between the two allocates, which is what keeps the globals from being
swept in the window where they are unrooted — and [begin] emptying the stack
rather than adding to it is what makes a re-entered main re-root the same
globals rather than push a second copy of each, which also closes the other
half: a re-run used to re-push roots over slots left dangling by the park.
Both emitters, because the dev loop's default backend is x86 and a fix in one
lowering is not a fix. A program with no dyn globals emits neither call and
its root stack still resets to empty, which is what an empty push list should
leave behind.
flan_dyn_root_pop now clamps at the globals rather than at zero. An
over-popping frame eating the globals is the one way that clamp could turn a
miscount into this same use-after-free.
Covered twice. test/dyn_ops.c's park mode is the runtime's half — a run, a
park with a collecting thunk in it, and another run, three times over,
asserting both that the global survives and that the frame's five hundred
objects do not. Under ASan the old reset reports heap-use-after-free in
flan_dyn_tag with the free in gc_sweep; under memcheck it reports 24 errors
and still prints the right answer, which is the shape of the bug. test_dev.ml
drives the whole daemon over its socket on both backends against
programs/dev-dyn-global.flan.
Not touched, and it wants a decision rather than a patch: a re-run re-enters
flan_program_main, which re-runs the lifted startup function, so every global
with a computed initialiser is reset by a re-run. That contradicts dev.ml's
own note and FIX.org item 1. It is independent of this — the roots are right
whether or not the values are re-initialised.
Nor is this the reload path. A defvar added by an evaluation gets its storage
from flan_dev_global (emit.ml's new_globals, x86.ml's counterpart) and there
is no flan_dyn_root_push anywhere on that path in either backend, so a dyn
global added to a live session is unrooted. That is a separate defect with a
separate fix, and nothing here makes it better or worse.
Relocation was proved sound and stayed sound — a Vec view holding the
header's own address survives a push that grows and moves it, because
there is no snapshot to invalidate. That was never the whole of the hazard.
Refusing every container into dyn outright, before this lane, meant a
dangling view was unreachable; the moment box stopped refusing, three
routes opened at once — a view returned from the function whose frame the
Vec lived in, one stashed in a dyn global and read after that frame is
gone, and one left behind when a condition transfer unwinds it. All three
are stack-use-after-return, reachable for the first time.
The rule: a typed container crosses into dyn as a view only when its own
storage is permanent — a global's. On the dynamic side Flan follows Clojure
and Common Lisp, where holding a value can never hand you garbage; treating
a view as a bare pointer and calling the lifetime the programmer's problem
is the Odin answer, and it is the wrong trade on this side of the language.
check.ml's permanent_root walks the checked expression back to its root: a
global is permanent, a field or an array element of one is permanent at the
same fixed offset, and a slice cut directly from one at the call site
inherits it — the trace is what a slice carries, and it is lost the moment
the slice is bound to a name first, so that case is refused too rather than
guessed at. Everything else answers false: a local, a parameter, a
temporary, and anything reached through a (Ptr T), because a heap-durable
pointer and a frame's own are the same type and the checker cannot tell
them apart — admitting one admits the other, which is the whole hazard this
closes. An arena-held header turns out not to be a separate case at all: an
arena changes where a Vec's elements live, never where its own header — the
binding — lives, so it is already covered by the storage-class check above.
Both directions of the F1 escape were reproduced before the fix (a genuine
ASan stack-use-after-return, reproduced by building the pre-fix tree) and
confirmed refused at check time after it, for all three routes.
Three more findings, all in the runtime rather than the boundary:
view_vec_check, on finding a stale container, rendered the very view it had
just declared unsafe to read — which called back into the same check,
unconditionally, an infinite recursion rather than the intended trap. Fixed
by never rendering the container in the stale message at all; the sentence
names the two epochs and nothing else, which is everything a reader needs
and the one thing that was safe to read.
dyn_equal's VEC arm read x->len and x->u.v.items regardless of kind, which
for a view answers 0 and the union's other member reinterpreted as dyn
words: two views with different contents compared equal, a view and an
equal heap vec compared unequal, and a map keyed by any view collided with
every other view, silently. vecish_len and vecish_at read either shape
correctly and the arm now goes through them. obj_words gets the same
explicit OBJ_VIEW case on the same reasoning, unreachable today only
because mark_push's own gate already excludes the kind — this is the belt
next to that brace.
The three restatements of flan_vec's layout — flan_rt.c's real struct,
flan_dyn.c's mirror, and dyn_ops.c's hand-built one — had a comment
claiming a reorder would not compile or link, which was never true of a
void*-typed forward declaration. flan_vec_layout and
flan_dyn_vec_hdr_layout each report their struct's size and field offsets;
dyn_ops.c's new "layout" mode compares both against offsetof on its own
hand_vec, so a disagreement is a FAIL line in dune test instead of a
silent corruption at whichever view reads through the wrong offset next.
Also: the survey program's comment excusing a by-value parameter's view as
"value semantics, not a hole" was wrong on its own terms — a write through
such a view does reach the caller's storage, only growth diverges — but the
question is moot now: every container the program views is a global, and
the file was rewritten around that rather than patched. And an i32 element
does not cross into a view either, but the refusal used to say why in words
that were true only of a string element; it now says what i32 actually is
and what the restriction is actually for.
Rebased onto dev-loop's item-4 landing (221df5a).
A (Vec T), a slice or a fixed array crossing into dyn no longer refuses; it
is a view, one word in the box, over the container's own storage. Reads box
the element on the way out; writes tag-check the dyn value's tag against the
element type on the way in and trap, by name, on a mismatch, never coercing
or silently storing.
The open question the decision left — whether the descriptor points at the
container or snapshots pointer and length beside it — is settled by kind. A
Vec view holds the address of the Vec's own header (flan_rt.c's flan_vec,
restated in flan_dyn.c under the file's standing "if either table changes,
change both" rule) and reads ptr and len live on every operation, so a push
that reallocates cannot leave it stale: flan_vec_grow overwrites that same
header in place, and there is nothing captured at the crossing for the
growth to invalidate. A slice and a fixed array cannot grow, so a flat view
snapshots data and length once; pointing it at the value's own slot instead
would be worse, since a slot's lifetime is not the slice's.
The element set is i64, f64 and bool, not everything box already handles
typed-to-dyn. A string element's dyn form is a pointer into the collector's
heap, and a typed container's storage is arena or stack memory the collector
never scans — a wider set would let a write plant a live reference nothing
ever traces, which no care at the write site closes. (Vec string) and a
typed (Map K V) keep the "does not cross into dyn yet" refusal, now for that
reason.
flan_dyn.c gains a fourth object kind, OBJ_VIEW, and flan_dyn_len/at/set_at/
push and the printer each grow one branch for it beside the existing vec
one. A view's own stale-container check is the runtime's own spelling
(flan_trap, park-and-inspect) rather than flan_rt.c's rt_die, per the
duplicity doctrine; growing a Vec through a view calls flan_rt.c's own
flan_vec_push rather than re-implementing doubling and allocator adoption a
second time. (set (at target i) x) against a dyn target — a plain dyn vec or
a view alike — was a hole in the base dyn milestone rather than something
item 3 introduced; it is wired to flan_dyn_set_at here because a view's
writes needed it to exist at all.
Both backends: emit.ml and x86.ml both already passed a Vec or a Map to a
runtime call by address rather than by value; a fixed array crossing into a
view needed the same arm added in both, for the same reason — a copy would
view the copy and never see a write to the caller's own array.
test/dyn_ops.c drives the runtime directly with a hand-built Vec header and
a plain C array, ahead of any compiler involvement: reads, writes on both
element kinds, the tag-check refusal on every element kind, the range
refusal, and the push that grows and moves a hand-built header out from
under the view watching it. test_flan.ml turns the old "does not cross into
dyn yet" refusal into acceptances for Vec/slice/array, keeps it for a string
element and for Map, and adds the element-restriction refusal by name.
test/programs/dyn-view.flan is the compiler-level survey: a Vec view mutated
through both sides including the grow-and-move case, a fixed array's and a
slice's views, a bool Vec's view, and its own two trapping modes for the
acceptance rows to run against. test_sanitize.ml carries the survey's happy
path; test_dyn.ml's new refusals are the runtime's own.
A descriptor's symbol was the type's printed form with every character an
assembler would refuse replaced by a dot, and the table was keyed by that.
The mangle is many-to-one — a Flan name may hold -, +, *, ? and / — so row-a
and row+a were one entry, the second of them was pushed with the first's
descriptor, and the collector read at another type's offsets: past the end of
the object when the first was the larger, and never where the second's dyn
actually sat. ASan named it, a stack-buffer-overflow inside gc_mark_all. It
is the same corruption root_plan pools its temporaries to avoid, arriving
through the name rather than through the supply, which is a lesson about where
identity lives: the table is keyed by Types.to_string now, which is an
identity, and the symbol carries a counter so two types cannot collide however
they mangle. dyn-struct.flan grows the pair, held live across the churn, and
an acceptance assertion asks the emitter directly how many descriptors it
wrote under that label — two, or the two are sharing one. That assertion is
the half with teeth: whether an overread off the end of a frame slot lands on
anything is luck, and the run's own output was not red under the defect.
The cap on a flattened array's offsets was bypassable by the thing it was
meant to stop. [4611686018427387904 S] wrapped the multiplication negative,
so the test read as under the cap, the declaration was accepted, and the
emitter then sat building the offset list until something killed it. A
refusal that overflows into an acceptance is worse than no refusal. The count
saturates at one past the cap now and the message says more-than rather than a
figure that came out of a wrap.
dyn_ops.c's second assertion had no teeth: a marker never writes through a
root, so "the word at a non-dyn offset is untouched" passed under any marker
at all. What discriminates offset-driven from word-driven is a dyn word the
descriptor leaves out, holding five hundred objects, that must NOT survive —
and it is checked by adding its offset to the table and watching the line go
red.
dyn_anywhere descended through Ptr and Slice, so (Vec (Ptr Cond)) was refused
with a sentence about a dyn inside a type whose storage contains none. A
vector of pointers to condition structs is an ordinary thing to write. It
stops at a pointer now, which is the line hidden_dyn already took for a bare
(Ptr S) and the line the whole argument rests on: a pointer is a view of
storage something else roots.
Which leaves the one honest hole, and it is named at the boundary where it
opens rather than left in a comment. Storage C hands back was never rooted
and never will be, so a (Ptr S) crossing a declare with a dyn anywhere under S
is refused by name — the same sentence a bare dyn already gets there, one
level down.
dune test --force: green, 0 failures. dyn-struct.flan clean under ASan and
UBSan and identical at -O2, -O0 and --x86.
The crux was never where to put a descriptor; it was how an instance finds
one. A bare struct on the stack has no header to hang a pointer off, and
giving it one would change the layout C interop agrees on, change the stride
of an array and change what embedding a struct in another costs. So it has
none. The instance never carries a pointer to its type and the collector
never derives one from the bytes: the pairing of an address with a descriptor
is made at the *push*, by the code that put the value there and therefore
knows its static type. That is the same trick the shadow stack has always
used, and it makes the stack case the easy one rather than the impossible one.
A descriptor is the size of an instance, a count, and a table of byte offsets,
emitted once per type as private static data. Flattened, not a graph — a
struct held by value contributes its offsets shifted by where it sits, and a
fixed array contributes its element's once per element — so nesting costs
nothing at run time and there is no recursion in the marker. The offsets of a
big array would be a big table, and that is capped with a sentence rather than
half of the repeat form item 3 will bring.
Four places a value of such a type can live, and all four are rooted: a frame
slot, a global, the temporary a call's by-value return is spilled into, and
the slot a condition that is not a place is evaluated into. The last two are
new and are the ones that were not obvious. A callee roots its dyn words and
pops them in its epilogue, so between the return and the caller's store the
only copy is a register, which a collector that finds its roots by address
cannot see; the same hole was open for a Flan call answering a bare dyn and is
closed here too. And a condition crosses as a pointer into the signalling
frame while a handler allocates, which is exactly what the original refusal
said could not be made safe.
dyn_roots grows into root_plan and both backends read it, which is what the
older note about one counter deciding both ends was always for. The aggregate
temporaries are pooled by type rather than handed out in mint order: a
positional supply that drifted would pair an address with another type's
descriptor, and marking arbitrary offsets off a base is corruption where a
missed root is only a bug. Pooled, the worst a drift can do is run out.
What is still refused is a dyn no static offset can reach — inside a typed
container, in a data type's payload or a union's members where the cases
overlay, or under an Option where the payload exists only beneath the tag.
A (Ptr S) and a [S] are deliberately not on that list: neither owns storage,
and the only storage this compiler hands out for such a type is a frame slot,
a global or a fixed array in one, all of them already rooted. That is what
lets a handler clause take its (Ptr Cond) and read a dyn payload.
test/programs/dyn-struct.flan is the evidence. It runs forty thousand rows
past flan_dyn.c's one-megabyte floor, so marks and sweeps really happen, and
it holds live values through them in all four places at once. It has teeth:
with the descriptor walk stubbed out of the marker, the kept vector's length
comes back 24 instead of 628 and its first element is a stale word. Clean
under ASan and UBSan, same output at -O2, -O0 and --x86. dyn_ops.c grows an
aggregate-root mode so the runtime half can be wrong on its own, with a
header word holding a bit pattern that looks boxed and is not a dyn slot.
--no-gc still refuses, and had to be told how: a struct with a dyn field is a
collected value even when no expression in the program ever has the type dyn,
because a zeroed one still has a word the collector is asked to mark.
dune test --force: green, 0 failures across every suite.
Milestone 1 of dynamic-by-default, the runtime half: NaN-boxed values in one
machine word, a mark-sweep heap, and the operations over them.
A double is itself, which is what a language with a physics loop and a float
calculator in its corpus wants; everything else hides in the quiet-NaN space,
three tag bits and a 48-bit payload that is exactly an x86-64 user pointer.
The negative-NaN collision is answered by canonicalising every NaN on the way
in, which flan_rt.c had already decided was the right thing to print. An i64
past the payload goes on the heap rather than becoming a 48-bit integer with a
64-bit name.
The collector is mark-sweep and nothing else -- no generation, no barrier, no
free list -- because the answer to wanting it faster is to type the program.
Roots are pushed, not scanned: NaN-boxing makes a conservative guess wrong in
both directions, and flan_dev.c's frame chain is the precedent. A fixed ring
of the last sixty-four allocations is marked unconditionally, which closes the
window where an expression with two constructors in it can collect its own
first result before the compiler has rooted either.
A type mismatch traps rather than aborting, through a flan_trap exported from
flan_rt.c so it takes the same path the six existing traps take: parked for
inspection in a dev session, dead where it stands otherwise. The sentence
names the operation, both tags as words, and both values.
flan_dyn.c is its own translation unit and nothing in the release runtime
names a symbol in it, so a program with no dyn operation links no collector
and --no-gc can be file-level selection rather than an argument with the
linker.
docs/SPIKE-DYNAMIC.md carries the argument. test/dyn_ops.c drives every
operation and all twenty-four refusals from C, the way dev_limits.c does,
including a million allocations against a hundred live and the control that
says an unrooted object really is reclaimed.
Joseph Ferano
