Joseph Ferano
7d4bec521e
Milestone 1 of dynamic-by-default, the runtime half: NaN-boxed values in one machine word, a mark-sweep heap, and the operations over them. A double is itself, which is what a language with a physics loop and a float calculator in its corpus wants; everything else hides in the quiet-NaN space, three tag bits and a 48-bit payload that is exactly an x86-64 user pointer. The negative-NaN collision is answered by canonicalising every NaN on the way in, which flan_rt.c had already decided was the right thing to print. An i64 past the payload goes on the heap rather than becoming a 48-bit integer with a 64-bit name. The collector is mark-sweep and nothing else -- no generation, no barrier, no free list -- because the answer to wanting it faster is to type the program. Roots are pushed, not scanned: NaN-boxing makes a conservative guess wrong in both directions, and flan_dev.c's frame chain is the precedent. A fixed ring of the last sixty-four allocations is marked unconditionally, which closes the window where an expression with two constructors in it can collect its own first result before the compiler has rooted either. A type mismatch traps rather than aborting, through a flan_trap exported from flan_rt.c so it takes the same path the six existing traps take: parked for inspection in a dev session, dead where it stands otherwise. The sentence names the operation, both tags as words, and both values. flan_dyn.c is its own translation unit and nothing in the release runtime names a symbol in it, so a program with no dyn operation links no collector and --no-gc can be file-level selection rather than an argument with the linker. docs/SPIKE-DYNAMIC.md carries the argument. test/dyn_ops.c drives every operation and all twenty-four refusals from C, the way dev_limits.c does, including a million allocations against a hundred live and the control that says an unrooted object really is reclaimed.
540 lines
32 KiB
Markdown
540 lines
32 KiB
Markdown
# The dynamic-value runtime: NaN-boxed values over a mark-sweep heap
|
||
|
||
Milestone 1 of dynamic-by-default. Unannotated code computes with values that carry their type at run
|
||
time; fully annotated code compiles to exactly what it compiled to before; and a build that wants
|
||
neither a tag nor a collector can be given one that has neither. This document is the runtime half —
|
||
`runtime/flan_dyn.h`, `runtime/flan_dyn.c`, and the C tests in `test/dyn_ops.c` driven by
|
||
`test/test_dyn.ml`. The compiler half — deciding which locals are dyn, emitting the root pushes,
|
||
enforcing `--no-gc` — is a parallel lane and is built against the ABI in the header, which is fixed.
|
||
|
||
Everything here is implemented and green. `dune test --force` includes `test_dyn`, which is 29 processes
|
||
and 0.42 seconds; `dune build @sanitize` covers the same C under ASan and UBSan.
|
||
|
||
---
|
||
|
||
## 1. Value representation: NaN-boxing
|
||
|
||
`flan_dyn` is `uint64_t`, one machine word, so a dyn local is a register or a stack slot and never a
|
||
struct whose return convention two backends would have to agree about. That much was fixed by the ABI.
|
||
What was open was the tagging scheme, and the two candidates are the usual pair.
|
||
|
||
**Low-bit tagging** puts the tag in the bottom three bits of a word and requires every payload to be
|
||
either a pointer to something aligned or an integer with three bits to spare. Its cost lands on floats:
|
||
an `f64` uses all 64 of its bits, so a low-bit scheme either boxes every float on the heap or steals the
|
||
mantissa's low bits and computes with a shortened double.
|
||
|
||
**NaN-boxing** does the opposite. A double is *itself* — the bit pattern is returned unchanged — and
|
||
everything else hides inside the quiet-NaN space, which IEEE 754 leaves as 2^52 unused encodings.
|
||
|
||
This codebase decides it. `f64` is a first-class type here, not a library afterthought: `sand.flan` runs
|
||
a physics loop, `calc-me.flan` is a float calculator, the prelude carries a hand-written `format-f64`
|
||
because `%g` was not enough, and `flan_rt.c` has three paragraphs on how to print a NaN. A representation
|
||
whose float path is the expensive one is the wrong representation for this language. Under NaN-boxing
|
||
`flan_dyn_from_f64` is a `memcpy` of 8 bytes and `flan_dyn_need_f64` is the same in reverse; neither
|
||
allocates, neither branches except on the NaN canonicalisation below.
|
||
|
||
The layout:
|
||
|
||
```
|
||
63 62..52 51 50..48 47..0
|
||
1 1...1 1 tag payload
|
||
^ sign ^ quiet
|
||
```
|
||
|
||
A word is *boxed* when `(v & 0xFFF8000000000000) == 0xFFF8000000000000` — sign bit set, exponent all
|
||
ones, quiet bit set. Bits 50..48 are three tag bits; bits 47..0 are the payload, which is exactly the
|
||
width of an x86-64 user-space pointer (48-bit canonical form, top 16 bits zero for user addresses).
|
||
Anything else is a double, read straight back.
|
||
|
||
Four box tags are used and four are free:
|
||
|
||
| tag | meaning | payload |
|
||
|-----|---------|---------|
|
||
| 0 | nil | 0 |
|
||
| 1 | bool | 0 or 1 |
|
||
| 2 | int | 48-bit two's complement, sign-extended on read |
|
||
| 3 | object | pointer to a `flan_obj` (text, vec, or a wide int) |
|
||
| 4–7 | unassigned | reserved for interop; see §7 |
|
||
|
||
### The negative-NaN collision, and why canonicalising is free here
|
||
|
||
Every NaN-boxing scheme has to answer for the doubles that are already negative quiet NaNs: their bits
|
||
are indistinguishable from a box. `flan_dyn_from_f64` answers it by mapping *every* NaN to the one
|
||
positive quiet NaN on the way in.
|
||
|
||
That is not a new rule invented for this file. `flan_rt.c`'s `flan_f64_to_bytes` already renders every
|
||
NaN as `nan` with no sign, and carries the argument at length: IEEE 754 does not specify the sign of a
|
||
NaN any operation produces, LLVM's constant folder and `divsd` disagree about `(/ 0.0 0.0)`, and the same
|
||
two-line program printed `nan` through one backend and `-nan` through the other. The runtime already
|
||
decided that a NaN's sign is not a fact about the arithmetic. This takes the same line one step further
|
||
and declines to *store* it. Nothing observable is lost: NaN is not equal to itself, so no comparison can
|
||
see which NaN it is, and the printer was already refusing to say. `-0.0` is untouched and keeps its sign,
|
||
which is tested.
|
||
|
||
### Integers: 48 bits inline, the rest on the heap
|
||
|
||
`i64` is first class here and 48 bits is not 64. The two honest options were a 48-bit integer with a
|
||
64-bit name, or a box for the overflow. This takes the box.
|
||
|
||
An `i64` in ±2^47 — that is ±140,737,488,355,327 — is inline: every array index, every byte count, every
|
||
timestamp in milliseconds until the year 6429, every value any program in the corpus computes. Outside
|
||
that range `flan_dyn_from_i64` allocates a `flan_obj` of kind `OBJ_INT` holding the full 64 bits. Both
|
||
shapes report `int` from `flan_dyn_tag`, compare and print identically, and round-trip through
|
||
`flan_dyn_need_i64`; nothing above the ABI can tell them apart. The cost is one range test on the
|
||
constructor and one branch in `dyn_int_value`, and the benefit is that a dyn `i64` is an `i64`.
|
||
|
||
`u64` has no constructor in this ABI, deliberately. Typed Flan distinguishes `u64` from `i64` — that is
|
||
why `flan_u64_to_bytes` exists beside `flan_i64_to_bytes` — and a `u64` above 2^63 has no dyn spelling in
|
||
milestone 1. It is an interop question (§7), not a representation one: the box already holds 64 bits and
|
||
a seventh tag would distinguish the signedness.
|
||
|
||
### Why not a two-word value
|
||
|
||
It would have made all of the above go away: a tag word and a payload word, no NaN games, no integer
|
||
boxing. It was not available — `typedef uint64_t flan_dyn` is in the fixed ABI — and it would have been
|
||
the wrong call anyway, for the reason `flan_rt.c`'s header gives about returning structs by value: the
|
||
emitted `.ll` would then have to agree with the platform's struct-return convention, which works on
|
||
x86-64 and silently does not on wasm32, and this project has two backends and a third target.
|
||
|
||
---
|
||
|
||
## 2. Heap objects
|
||
|
||
One header, three kinds, one singly-linked list of everything allocated.
|
||
|
||
```c
|
||
typedef struct flan_obj {
|
||
struct flan_obj *next; /* every object ever allocated, newest first */
|
||
uint8_t kind; /* OBJ_TEXT | OBJ_VEC | OBJ_INT */
|
||
uint8_t mark;
|
||
int64_t len; /* bytes of a text, elements of a vec */
|
||
union {
|
||
int64_t i; /* OBJ_INT */
|
||
struct { flan_dyn *items; int64_t cap; } v; /* OBJ_VEC */
|
||
} u; /* OBJ_TEXT's bytes trail */
|
||
} flan_obj;
|
||
```
|
||
|
||
- **Text** is immutable and length-prefixed, with the bytes inline after the header — one allocation per
|
||
string, and a length that is a count and not a NUL scan. `flan_dyn_from_bytes` copies, so a literal in
|
||
`.rodata`, a frame slot, or a slice about to be dropped are all legitimate sources. Nothing in the ABI
|
||
writes into a text; `flan_dyn_set_at` on one refuses by name rather than copying on write.
|
||
- **Vec** is mutable, with its elements in a plain `malloc` block hanging off the header rather than in a
|
||
GC object of their own. Growth is then a `realloc` rather than a copy this file writes, and the
|
||
elements are never reachable except through their vec, so giving them an identity would buy nothing
|
||
and cost a header. Their bytes are counted in the live figure and freed when the vec is swept.
|
||
- **Int** is the overflow case above.
|
||
|
||
`mark` is a byte in the header and not a bit in a side table. A side table is the right answer when the
|
||
sweep is the cost, and the sweep is never going to be the cost here — see §3.
|
||
|
||
There is no free list, no size class, and no interning. Two texts built from the same bytes are two
|
||
objects, which `test_dyn` asserts directly: equality is bytewise (§5) and a test that only compared
|
||
structurally would pass against an implementation that had quietly shared.
|
||
|
||
---
|
||
|
||
## 3. The collector
|
||
|
||
**Mark-sweep, precise, non-moving, stop-the-world, in about 120 lines.**
|
||
|
||
The author's constraint was explicit — *"the answer to 'I need more performance' will never be a faster
|
||
GC, it will be to type the whole program"* — so there is no generation, no card table, no write barrier,
|
||
no incremental phase and no reference counting. The design goal is that somebody can read the whole
|
||
collector in one sitting and believe it.
|
||
|
||
### Trigger
|
||
|
||
Collection happens inside `gc_alloc` and nowhere else. The policy is the plainest one that bounds the
|
||
heap:
|
||
|
||
```
|
||
if (gc_bytes + need > gc_next) collect();
|
||
...
|
||
gc_next = gc_bytes * 2; if (gc_next < gc_floor) gc_next = gc_floor;
|
||
```
|
||
|
||
Collect when this allocation would carry the heap past a limit; afterwards set the limit to twice what
|
||
survived, with a floor (1 MiB by default) so a program with a tiny live set does not collect every other
|
||
allocation. That gives amortised O(1) collection work per byte allocated, and a heap bounded at roughly
|
||
twice the live set plus the floor — which is the property `test_dyn`'s million-allocation case asserts,
|
||
as a bound rather than as an exact figure, because pinning the high-water mark would make a tuning change
|
||
a test failure.
|
||
|
||
`flan_gc_set_floor` is an ABI extension and exists for the tests: a megabyte of floor would make the
|
||
million-allocation case a megabyte of arithmetic before it proved anything. It recomputes the trigger
|
||
from the new floor rather than only raising it to meet it — raising alone left a heap that had been given
|
||
a *lower* floor still running to the old one, which is a bug this document's first draft had.
|
||
|
||
A vec's element array growing is a `realloc` and not a collection point, deliberately: the value being
|
||
pushed may not be rooted yet. The bytes are charged to the heap immediately so the trigger sees them at
|
||
the next real allocation, but nothing is swept in the middle of a push.
|
||
|
||
### Mark
|
||
|
||
An explicit worklist, not recursion. A vec of a vec of a vec is an ordinary dyn value and its depth is
|
||
the program's, not the runtime's; a recursive marker would put the heap's depth on the C stack and a long
|
||
enough chain would overflow it *during a collection*, which is the worst available moment. The mark stack
|
||
is grown on demand and kept between collections, so a steady program stops paying for it after the first.
|
||
`test_dyn`'s `nested` mode is a chain 64 deep and would notice a marker that stopped being iterative.
|
||
|
||
Only a vec has anything to trace; a text and a boxed int are leaves and marking them is the whole visit.
|
||
|
||
### Sweep
|
||
|
||
Walk the all-objects list, unlink and free anything unmarked, clear the mark on anything else, and
|
||
subtract the freed bytes. A vec's element array is freed with its header.
|
||
|
||
### What "precise" buys
|
||
|
||
The collector never guesses whether a word is a pointer, which matters more here than usual. NaN-boxing
|
||
makes conservative scanning wrong *in both directions*: a live double is bit-identical to a boxed pointer
|
||
often enough to retain garbage indefinitely, and a payload with the box stripped is not the pointer a
|
||
scanner would recognise. Precision is cheaper than the arguments about it.
|
||
|
||
---
|
||
|
||
## 4. Roots
|
||
|
||
No conservative stack scanning. The compiler emits `flan_dyn_root_push` per dyn local on entry and one
|
||
`flan_dyn_root_pop(n)` for the lot on the way out — shadow-stack style, exactly as `flan_dev.c`'s frame
|
||
chain works and for the same reasons its header gives: a pushed record needs no agreement with the
|
||
optimiser about what a frame looks like, is the same on wasm32 as on x86-64, and needs no `.eh_frame`
|
||
walk to borrow (this language lowers every non-local exit explicitly and has none).
|
||
|
||
The root stack holds *addresses*, not values, so a local that is reassigned needs no second push. It
|
||
grows on demand, because a deep recursion over dyn locals is an ordinary program and a fixed table would
|
||
be a limit nobody could predict. Dyn globals go through the same pair, pushed once at startup and never
|
||
popped.
|
||
|
||
**The one contract the compiler lane must honour: a slot must hold a valid `flan_dyn` before it is
|
||
pushed.** The collector reads every registered address on every mark, and an uninitialised slot is a word
|
||
of stack garbage that will be decoded. Storing `flan_dyn_nil()` at the declaration is the whole of it.
|
||
|
||
`flan_dyn_root_pop` clamps at empty rather than refusing. A pop that outruns its pushes means the frame
|
||
machinery is already out of step, and the useful thing at that point is a heap that still collects.
|
||
|
||
`flan_dyn_root_reset` is an ABI extension with one caller: the merged dev build's `main`, which is
|
||
re-entered by `longjmp` and therefore pops no frame, so every root the finished run pushed still points
|
||
into stack the next run is about to write over. It is the exact counterpart of `flan_dev_frames_reset`
|
||
and `flan_condition_stacks_reset`, which exist for the same `longjmp`.
|
||
|
||
### The temporaries ring, and the hazard it closes
|
||
|
||
"Collection triggers only inside `flan_gc_alloc`, so between pushes nothing moves" protects against
|
||
*relocation*. It does not protect against *freeing*, and mark-sweep frees whatever is unreachable — which
|
||
includes an object allocated a moment ago and not yet stored anywhere the collector looks. Concretely:
|
||
|
||
```c
|
||
flan_dyn_push(v, flan_dyn_add(flan_dyn_from_bytes(p, n), other));
|
||
```
|
||
|
||
Two allocating calls in one expression. C leaves argument evaluation order unspecified, so this is not
|
||
even a window a careful emitter could close by ordering its calls.
|
||
|
||
The answer is the smallest one that does not depend on another agent having read this document: **every
|
||
object `flan_dyn.c` allocates is written into a fixed 64-slot ring, and the marker roots the whole ring
|
||
unconditionally.** Any expression making at most 64 allocations before rooting its result is safe. The
|
||
cost is one store and a masked increment per allocation, plus up to 64 objects of float in the heap,
|
||
which the trigger absorbs because the trigger is a fraction of live bytes and not a count. There is no
|
||
ABI change and no contract for the compiler lane to get wrong.
|
||
|
||
The visible consequence, and the only one: after a full collection up to 64 unreachable objects are still
|
||
alive. `test_dyn`'s `unrooted` mode asserts exactly that — "reclaimed all but the ring" — rather than
|
||
pretending the figure is zero.
|
||
|
||
An expression with 65 allocating calls and no intervening root would be a single Flan form with 65
|
||
constructors in it. If that ever exists, the compiler will have rooted its intermediates and the ring is
|
||
belt on top of braces.
|
||
|
||
---
|
||
|
||
## 5. The operations, as implemented
|
||
|
||
### Arithmetic — `+ - * / %`
|
||
|
||
Two ints answer an int; a float anywhere answers a float; anything non-numeric traps.
|
||
|
||
**The promotion is the one place dyn is more permissive than the typed language, and it is deliberate.**
|
||
Typed Flan has no implicit widening anywhere — `(print-i64 x)` used to force an explicit `(i64 x)` at
|
||
every site, and the prelude's history is full of that. But `(+ 1 2.5)` has exactly one sensible answer,
|
||
and a dynamic language that refuses it is not dynamic in any useful sense. A program that wants the
|
||
refusal annotates, which is the whole bargain of dynamic-by-default.
|
||
|
||
Integer division and remainder by zero trap, matching typed Flan, which signals `ArithError` and dies
|
||
with "divide by zero" if nothing handles it. `INT64_MIN / -1` gets its own sentence for the reason
|
||
`flan_rt.c` gives it one. The float cases are left to IEEE: `1.0/0.0` is `inf` and that is an answer.
|
||
Float `%` is computed by truncated division rather than by calling `fmod`, which would pull `math.h` into
|
||
this translation unit for one operator; the quotient is range-checked first, because casting a double
|
||
past 2^63 to an integer is undefined rather than merely wrong.
|
||
|
||
### Ordering — `< <= > >=`
|
||
|
||
Numbers against numbers (promoting across the two tags), text against text bytewise, and nothing else.
|
||
Bytewise means `memcmp` with the shorter operand first on a tie — the order a sort of `(Vec string)`
|
||
wants, and the only one needing no locale and no collation table.
|
||
|
||
A number against a text traps. The temptation is to order by tag so that everything is comparable and
|
||
sorting never fails; the reason not to is that the resulting order is an artefact of this file's tag
|
||
numbering, and a program sorting a mixed vec would get a stable answer that means nothing.
|
||
|
||
NaN is unordered in all four directions, which is what IEEE says and what `test_dyn` asserts.
|
||
|
||
### Equality — `=`
|
||
|
||
Structural, and **the only operation here that never traps**: two values of unrelated tags are unequal,
|
||
which is an answer. Making it an error would mean a dyn program could not ask "is this the string I
|
||
expected" without first checking that it is a string at all.
|
||
|
||
- Numbers compare by value across the two tags: `(= 1 1.0)` is true.
|
||
- Text is **bytewise, not by identity**. `flan_dyn_from_bytes` copies and does not intern, so two texts
|
||
built from the same bytes are two objects; the only defensible equality for an immutable byte string is
|
||
its bytes. Embedded NULs are compared like any other byte — the length is a count, not a `strlen`.
|
||
- A vec is equal element by element, with an identity shortcut first.
|
||
- NaN is not equal to itself.
|
||
|
||
Cycles: `flan_dyn_set_at` lets a vec contain itself, so the recursion is capped at 64. Past the cap two
|
||
vecs are equal only if they are the same vec, which terminates and answers correctly for the case that
|
||
actually arises — a cycle compared against itself. Two *distinct* cyclic vecs of the same shape answer
|
||
false, which is a wrong answer to a question nobody has asked yet; the honest fix is a visited set and it
|
||
can be added the day somebody needs it.
|
||
|
||
### `len`, `at`, `set-at`, `push`
|
||
|
||
`len` is bytes of a text or elements of a vec. `at` is an element of a vec or a byte of a text as an int
|
||
— which is what `(at s i)` on a `(Slice u8)` does in the typed language; codepoints are `utf8`'s job and
|
||
stay there. `set-at` and `push` are vec-only, and a `set-at` on a text says *"a text is immutable — build
|
||
another one"* rather than quietly copying.
|
||
|
||
An index that is not an int is a different sentence from a container that is not indexable, because
|
||
`(at v "1")` and `(at 3 1)` are different mistakes and one message naming both would name neither.
|
||
|
||
### `print`
|
||
|
||
Structural, and per tag it renders what typed `print` renders. The reference forms were **captured from a
|
||
running program**, not read off `lib/render.ml` — that file is the REPL's inspector and not necessarily
|
||
`println`'s expansion:
|
||
|
||
| tag | rendering | example |
|
||
|-----|-----------|---------|
|
||
| int | `%lld` | `42` |
|
||
| float | `%g`, NaN unsigned | `3.5`, `1`, `nan` |
|
||
| bool | the word | `true` |
|
||
| text | bare at the top level | `hi` |
|
||
| text | quoted and escaped inside a structure | `"a b"`, `"q\"\n"` |
|
||
| vec | a slice's spelling | `[ 1 2 3]` |
|
||
| nil | — | `nil` |
|
||
|
||
The leading space before every element is what `lib/render.ml`'s slice loop emits and what a Flan program
|
||
prints today; a tidier answer would diverge from an acceptance test. The escape table is the third copy
|
||
of the one in `flan_rt.c`'s `flan_escape_bytes` and `flan_dev.c`'s `flan_dev_emit_str` — those two are
|
||
already kept identical because the REPL parses the printed form back, and this one joins them. If that
|
||
table changes, change all three.
|
||
|
||
**A typed `Vec` prints as `<vec>` and a dyn vec does not.** That looks like a mismatch and is not. The
|
||
typed printer's refusal is about walking storage it does not own — `render.ml` says so, and directs you to
|
||
`(print (as-slice v))`, which borrows explicitly at the call site. A dyn vec's storage belongs to the
|
||
collector, the printer is inside the runtime that owns it, and there is nobody to ask permission from. So
|
||
a dyn vec prints the way a *slice* does, which is also the only rendering that carries any information.
|
||
|
||
`print` has a depth cap of 16. A typed value cannot contain itself, so the typed printer needs no
|
||
run-time cap; `set-at` makes a dyn vec that can, so this one has one, and past it prints `...`, which is
|
||
the mark `render.ml` uses for the same idea.
|
||
|
||
---
|
||
|
||
## 6. The typed boundary
|
||
|
||
`flan_dyn_need_i64`, `_need_f64`, `_need_bool` are what an annotated parameter does with a dyn argument
|
||
and what a dyn expression does where the checker wants a machine value. **There is no widening and no
|
||
coercion at the boundary**: `flan_dyn_need_f64` of an int traps.
|
||
|
||
That is the asymmetry worth defending, because the operators *do* promote (§5). The two are different
|
||
questions. An operator has no stated expectation to violate — `(+ 1 2.5)` was written by somebody who
|
||
wanted a number and there is one obvious number. An annotation *is* a stated expectation, written down by
|
||
a person, and quietly turning their `i64` into an `f64` would make the boundary the one place in the
|
||
language where a type changed without anybody writing it. Typed Flan has no implicit widening; the
|
||
boundary into typed Flan should not invent some.
|
||
|
||
The consequence for the compiler lane: a call from dyn code into `(defn f [x f64] ...)` with an integer
|
||
argument traps at run time rather than converting. If that turns out to be too sharp in practice, the
|
||
place to soften it is the *compiler*, by emitting a conversion where the checker can see both sides — not
|
||
here, where the only thing visible is a tag.
|
||
|
||
---
|
||
|
||
## 7. Interop, which is out of scope for M1 and not precluded
|
||
|
||
A typed `(Vec i64)` crossing into dyn code, without copying, is milestone 2. The layout leaves room for
|
||
it in three specific ways.
|
||
|
||
**Four spare tags.** Bits 50..48 give eight box tags and four are used. Tag 4 is the natural home for a
|
||
*typed handle*: payload = pointer to a typed object, with the element type recovered from a side table or
|
||
from a word in the object's own header. Tag 5 can carry `u64` above 2^63, which is the other value typed
|
||
Flan holds and this ABI currently cannot name.
|
||
|
||
**The object header is not the only thing the collector can trace.** `mark_push` dispatches on `kind`,
|
||
and a fourth kind whose tracer is "call a function pointer in a descriptor" is a local change — a handle
|
||
to a typed `(Vec Enemy)` would need the collector to know which fields of `Enemy` are dyn, and a
|
||
per-type descriptor emitted by the compiler is how that arrives. Nothing in the current layout has to
|
||
move for that: the `union` grows an arm.
|
||
|
||
**A typed vec is not a `flan_obj` and does not need to become one.** The interop that matters is the
|
||
cheap direction — a typed vec is `{ptr, len, cap}` in memory the *arena* owns, and a dyn handle to it
|
||
should be a borrow, not a copy. That means the collector must be able to trace *into* memory it did not
|
||
allocate and must never free it. The `kind` byte is what distinguishes "I own these bytes" from "I am a
|
||
window onto someone else's", and the sweep already branches on it.
|
||
|
||
What is genuinely open, and is stated here so the M2 lane does not discover it late: **a typed vec's
|
||
lifetime is its arena's, and the collector has no say in it.** A dyn value outliving the arena its handle
|
||
points into is a dangling read that the current design cannot detect. The allocation registry in
|
||
`flan_dev.c` already answers exactly that question for `Ptr` — `render.ml`'s pointer arm follows a live
|
||
one and mourns a dead one — so the shape of the answer exists. It is not built here.
|
||
|
||
---
|
||
|
||
## 8. Dropping the collector
|
||
|
||
`--no-gc` enforcement is the compiler lane's. This lane's contribution is that **it is possible**, which
|
||
is a property of the build and not a wish:
|
||
|
||
- `flan_dyn.c` is its own translation unit and always has been. It is compiled from
|
||
`Runtime_src.dyn_source` in `lib/build.ml` (both `executable` and `macro_module`) and in `lib/dev.ml`,
|
||
beside `flan_rt.c` and `flan_dev.c` and never merged with them.
|
||
- **The dependency runs one way only.** `flan_dyn.c` calls `flan_write_stdout` and `flan_trap` in
|
||
`flan_rt.c`. Nothing in `flan_rt.c` or `flan_dev.c` names a symbol in `flan_dyn.c`. One
|
||
back-reference — `flan_rt_init` calling `flan_gc_init`, say, which was the obvious thing to write —
|
||
would make the collector unconditional and the refusal a lie, so the heap initialises itself lazily on
|
||
first allocation instead.
|
||
- Consequently a program that calls no dyn operation pulls nothing out of that object, and the linker
|
||
leaves it behind.
|
||
|
||
The link line does **not** currently pass `-ffunction-sections`/`-Wl,--gc-sections`, and this document
|
||
will not claim a drop the build cannot perform. What the above buys is the cheaper mechanism: the object
|
||
is selected at the *file* level, so `--no-gc` is a one-line change at each of the three sites — the same
|
||
per-target selection `select_csrcs` already performs for a package's C — rather than an argument with the
|
||
linker about which sections are live. `flan_dev.c` is compiled into every build today for a reason its
|
||
comment gives at length (a package's C refers to it and package sources are collected whatever `main`
|
||
does); `flan_dyn.c` has no such entanglement and can genuinely be left out.
|
||
|
||
The residual-dynamism refusal itself — deciding that a program is *not* fully annotated and saying which
|
||
form is not — is the compiler's, and is not attempted here.
|
||
|
||
---
|
||
|
||
## 9. Cost sketch, per operation
|
||
|
||
No microbenchmarks; these are counts of what the code does.
|
||
|
||
| operation | cost |
|
||
|-----------|------|
|
||
| `from_f64`, `need_f64` | one 8-byte copy; one compare for the NaN canonicalisation |
|
||
| `from_i64` | one range compare, then a mask and an or — or an allocation past ±2^47 |
|
||
| `need_i64` | tag test, then a shift pair to sign-extend |
|
||
| `from_bool`, `nil`, `need_bool` | mask and or; tag test |
|
||
| `tag` | one mask-compare, one shift, and a load for the object case |
|
||
| `add`/`sub`/`mul` | two tag tests, the arithmetic, one `from_i64` (which may allocate at the extremes) |
|
||
| `div`/`rem` | the above plus a zero test and an overflow test |
|
||
| `lt`/`le`/`gt`/`ge` | two tag tests and a compare; text is `memcmp` |
|
||
| `eq` | word compare first; then tags, then bytes or elements — O(size) at worst |
|
||
| `len` | tag test and a load |
|
||
| `at` | two tag tests, a bounds compare, a load |
|
||
| `set_at` | the same, plus a store; **no write barrier**, because there is no generation to have one for |
|
||
| `push` | tag test, a capacity test, amortised O(1); the doubling is a `realloc` |
|
||
| `print` | O(size), streamed, no allocation |
|
||
| `root_push` / `root_pop` | a store and an increment; a `realloc` when the stack doubles |
|
||
| allocation | a trigger compare, a `malloc`, a header init, a ring store |
|
||
| collection | O(live) to mark, O(all) to sweep; amortised O(1) per byte allocated |
|
||
|
||
The shape to take away: **every operation is a handful of instructions plus whatever the arithmetic
|
||
costs, and the only unbounded ones are `eq`, `print` and collection itself.** If that is not fast enough,
|
||
type the program.
|
||
|
||
---
|
||
|
||
## 10. ABI extensions
|
||
|
||
The header in `runtime/flan_dyn.h` is the agreed ABI with nothing removed and no signature changed.
|
||
Seven things are added; none is something the compiler lane must emit.
|
||
|
||
| addition | why |
|
||
|----------|-----|
|
||
| `flan_dyn_root_reset()` | the merged dev build's `longjmp` re-entry, exactly as `flan_dev_frames_reset` |
|
||
| `flan_dyn_tag()` | the break loop and the inspector; the first question anyone asks a stopped dyn value |
|
||
| `flan_dyn_tag_name()` | the same table the trap messages are written from, so a message and an inspector cannot disagree |
|
||
| `FLAN_DYN_TAG_*` | the numbers `flan_dyn_tag` answers with |
|
||
| `flan_gc_count()` | the tests: a live-bytes figure alone cannot tell a collecting heap from one with small objects |
|
||
| `flan_gc_set_floor()` | the tests: a 1 MiB floor makes the million-allocation case slow and uninformative |
|
||
| `flan_trap()` in `flan_rt.c` | a thin exported wrapper over the existing static `rt_trap`, so a dyn type error takes the *same* path as `flan_bounds_fail` and the other six rather than re-implementing the hook, the flush, the socket and `_exit(134)` |
|
||
|
||
One more thing the compiler lane should know about, which is a build fact rather than an ABI addition:
|
||
**`Build.compile_c` now drops `runtime/flan_dyn.h` into the directory it compiles each translation unit
|
||
in**, so a package's C — or an emitted shim — can `#include "flan_dyn.h"`. `flan_dyn.c` itself does not
|
||
include it and declares its own prototypes, because the object cache is keyed on source text and a header
|
||
edit would serve an object compiled against the previous one. The two copies are kept honest
|
||
mechanically: `test/dyn_ops.c` includes the header and names every function in it, so a divergence is a
|
||
compile or link error in `dune test`.
|
||
|
||
---
|
||
|
||
## 11. Traps
|
||
|
||
A dyn type error is a trap and not an abort: it takes `flan_trap`, which calls `flan_trap_hook` if one is
|
||
installed and then `rt_die`. In a dev session the hook parks the program for inspection; in a standalone
|
||
build nothing installs it and the program dies where it stands. The sentence is the same either way,
|
||
which is the point of routing through the hook rather than calling `abort` in the runtime.
|
||
|
||
The sentence names the operation, both tags as **words**, and both values:
|
||
|
||
```
|
||
dyn +: int and text, and it takes two numbers — (+ 3 "hi")
|
||
dyn at: index 9 is out of bounds for text of length 2 — "hi"
|
||
dyn f64: int, and a float was wanted — (f64 1)
|
||
```
|
||
|
||
Tag names are always words — `int`, `float`, `text`, `vec`, `nil`, `bool` — and never numbers. They come
|
||
from one table that `flan_dyn_tag_name` also reads. Values are rendered into a bounded stack buffer with
|
||
a depth cap of 2, truncated rather than allocated: a trap is the one moment when allocating would be a
|
||
second thing to go wrong.
|
||
|
||
Four trap names, because they are four different mistakes and somebody standing in one wants to know
|
||
which without reading the sentence twice:
|
||
|
||
| name | when |
|
||
|------|------|
|
||
| `DynType` | a tag that is not what the operation wanted |
|
||
| `DynRange` | an index outside a vec or a text |
|
||
| `DynArith` | division by zero, or the one quotient that overflows |
|
||
| `DynHeap` | an allocation the host refused |
|
||
|
||
The bounds case does **not** signal `BoundsError` the way typed indexing does, and that is a limitation
|
||
rather than a preference: `flan_bounds_error` takes a `loc` and a transfer channel, and a dyn operation
|
||
has neither in its hands. It is the same reason the six traps in `flan_rt.c` park rather than signal. If
|
||
the compiler lane threads a `loc` through — it knows one at every call site — the dyn bounds case can
|
||
join the condition system unchanged, and that is the obvious next increment.
|
||
|
||
---
|
||
|
||
## 12. What is tested
|
||
|
||
`test/dyn_ops.c`, driven by `test/test_dyn.ml` against `test/programs/dyn-host.flan` — a C main linked
|
||
against a Flan program with no `main`, the arrangement `dev_limits.c` and `reload_host.c` already use,
|
||
because these are C entry points with no Flan spelling. One binary, 29 runs, 0.42 s.
|
||
|
||
| mode | what it establishes |
|
||
|------|--------------------|
|
||
| `ops` | every operation's happy path; every tag and its word; f64 round trips at `0`, `-0.0`, `1e308` and NaN; i64 at both inline edges, both boxed edges, `INT64_MIN` and `INT64_MAX`; arithmetic crossing into the box; NaN unordered in all four directions; text identity vs equality; embedded NULs; the empty text; vec growth past its initial capacity; a vec that contains itself, compared and printed; every printed form as an exact string |
|
||
| `gc` | a million allocations against a hundred rooted values; heap high-water under 512 KiB where a non-collecting heap would hold ~40 MiB; the live set intact afterwards |
|
||
| `unrooted` | **the positive control** — an object nothing points at is reclaimed. Without it every other case would pass against a collector that never freed |
|
||
| `nested` | a chain of vecs 64 deep traced through one root, across real collections |
|
||
| `sharing` | one vec in three slots: identity (not just equality), a write through one path read through another, survival when the direct root goes, and a single sweep when the last reference does |
|
||
| `refuse:*` | 24 refusals, one process each, asserted on the sentence as well as the exit status — a process that died some other way is not the guard firing, and the code cannot tell them apart |
|
||
|
||
`dune build @sanitize` runs the same C under ASan and UBSan. That sweep matters more here than anywhere
|
||
else in the corpus: every other program in it allocates and never frees, which is a policy ASan can only
|
||
agree with, and this one frees — a mark-sweep collector is precisely a machine for freeing something that
|
||
is still reachable. A use-after-free is what a wrong marker looks like from outside, and it is invisible
|
||
to the assertions above, because the freed bytes are usually still the bytes that were there. Leaks stay
|
||
off, as they are for the whole file: the temporaries ring holds 64 objects alive at exit on purpose, and
|
||
every one of them would be reported.
|