vendor/json is vendor/edn's shape with one decision reversed. edn never
allocates, so its tokens are views into the source buffer and escaped
strings are refused for want of anywhere to put the unescaped copy. This
one has an allocator, so it unescapes, and to unescape it copies —
string-of is the only function in the package that allocates, and it
copies even when there was no escape to resolve, because a Value whose
lifetime depended on which bytes happened to be in it is not a contract
anyone can hold. Odin answered the same question the same way:
tokenizer.odin allocates nothing, parser.odin's unquote_string does the
copy, and it clones in the no-escape branch too.
What that buys is at the bottom of test/programs/json.flan, which is
programs/edn.flan and programs/arena-edn.flan in one file because for
JSON they are one claim. The source buffer is overwritten with `?` bytes
while the document is live and the strings read back afterwards are
still the strings. arena-edn's header has a section admitting it cannot
do that.
Strict JSON and not Odin's JSON5 default, and the difference is where
most of the refusals come from: comments, single quotes, +1, .5, 1.,
0x1f, 01, NaN, Infinity and unquoted keys each get a sentence naming the
dialect they belong to, rather than one shared unexpected-byte. A lone
surrogate is refused too, and that one is forced rather than chosen —
rune-size answers None for the whole D800-DFFF block, so encode-rune!
would write nothing and the character would vanish.
The tokenizer refused #{} because "it needs a hash set to even
represent" — which is a claim about a reader, and a tokenizer represents
nothing. #{ now pushes } on the same balance stack { does, there is one
new token kind and no new closer, and err-set is gone rather than kept
with a message it no longer earns. skip-value needed nothing: it is
written against the depth and not against the kinds.
The dynamic reader moves out of test/programs/arena-edn.flan and into
vendor/edn/read.flan as (edn/read bytes), answering an (Option Value)
against whichever allocator the caller bound. Two decisions are written
down where they are made:
* a set is a Value.Set holding a deduplicated (Vec Value), because
(Map Value bool) does not typecheck — keyable refuses a key holding
a Vec or a Map — and restricting elements to keyable Values would
refuse #{[0 0] [1 0]}, which is the file this was built for. Insert
is O(n) against a structural value=?, so building the tileset's 54
pairs is 1458 comparisons, once.
* a Value copies every string into the allocator where a Token stays
a view. A view handed back out of the function that owns the buffer
is a dangling pointer, and free-all would not even take it. Odin's
json parser clones for the same reason.
An imported defdata was a refusal in load.ml — "not implemented yet
(milestone 4)" — and it had to go first. It is the type's name plus the
Type. half of a constructor symbol, which arrives as a Var node when the
case has no fields and a Struct node when it has; a match pattern needed
nothing, because a case resolves against the scrutinee's type and was
never a top-level name. programs/pkg-data.flan is that on its own.
programs/edn-read.flan reads assets/edn/tileset.edn, which is the
editor's real output: :texture-path and a :selected-cells of 54 integer
pairs, with no type declared for any of it. It also overwrites the
source buffer in place after reading and prints the document back, which
is the copy contract asserted rather than described.
The lo <= hi test in check_slice and slice-from-ptr's n >= 0 sat behind
--no-bounds-checks in both backends, while the comment beside each said
they could not be dropped. They are not bounds checks: hi <= len asks
whether a range fits inside a length, and lo <= hi asks whether the word
about to be written into a %slice's length field is a count at all. The
first stays behind the flag, the second is now emitted everywhere, the
way flan_vec_as_slice has always validated its own l > h in plain C.
emit.ml emits two signal blocks rather than one and i1, so an unchecked
build carries one compare. x86.ml keeps all three frame temporaries
stored outside the flag and gates only the second compare, because the
third is the length the message prints.
The IR assertion in test_acceptance now says the two slice calls are
present under --no-bounds-checks rather than absent, and the same build
is run: case 2 and case -2 of bounds.flan must still die.
Every number-to-text conversion wrote into one file-static in the runtime and
answered a slice over it, and nothing copied. Two of them in one expression
printed the second number twice — no crash, no diagnostic, and nothing a
sanitizer could find, because every byte read was inside an object that was
alive. The wrong object.
The buffer is now the caller's, one frame slot per call site. The slot is
allocated in the checker rather than in either backend: a slot is a
function-lifetime location in both of them, where an x86 backend temporary is
bump-allocated and reclaimed at the end of the expression that made it — which
is the one lifetime a returned slice must outlive. Each backend gains one
pointer argument and no reasoning of its own, which is what keeps them
symmetric.
The static is gone rather than left unused, since a buffer with nothing but a
comment beside it is a loaded gun. What remains is the ordinary lifetime a
pointer into a frame has: storing one of these slices in a container that
outlives the frame, or returning it, is still a copy the caller has to make.
NEXT.md's sharp edge now says that instead of what it used to say.
Both corpora are explicit lists and not globs, so a program added to
test/programs is covered by dune test and by @x86 and by nothing else until
somebody types its name here. files.flan, math3.flan and time.flan are typed.
time.flan is the one with something to say. getenv hands back a slice viewing
the process environment and never a copy, which is the exact shape a
use-after-free or an off-by-one length would be, and neither ASan nor memcheck
had ever seen it. files.flan brings three more path buffers through
flan_path_cstr. math3.flan is the cheap one and is here for completeness.
files.flan makes and removes its own tree, so the sweeps' two runs of it see
the same directory both times.
@sanitize is clean with all three in. @valgrind is not run here -- it is tens
of minutes and opt-in -- so those three entries are checked by the next person
who runs the alias.
A (Vec Value) where a Value may itself hold a (Vec Value) — the recursive
dynamic value an EDN reader has to answer with when nobody hands it a target
struct type — was refused five different ways, and every one of the five gave
the same reason: the container runtime is type-erased, so it copies and
releases slots bytewise and cannot reach inside a slot. A free would release
the slots and leave every block they point at stranded.
That reason is about teardown, and it does not hold for a region. free-all
never releases an individual slot; it takes the whole arena, and every block
the elements own is in it, because they came out of it. The refusals were
over-broad, and what they were guarding was never ownership — ownership
tracking is untouched here, moves are still moves, and Types.is_move_only is
the same function it was.
So the question moved rather than disappeared. It could not stay at the type,
because can-free is a capability on an allocator value and with-allocator
rebinds a dynamic variable: which tier a (vec-new) will meet is not a property
of the place its type is written. What is decided at compile time is only
whether to ask, which is a property of the element type; the answer is a
run-time branch on the allocator, one per container and never per element,
because the alternative is a walk at release and a walk at release is the
registry of destructors the frame tier's reset exists to not have. It is
emitted at every growth and not only at the construction, because ZII means a
container can exist without ever passing through (vec-new) — a case field left
out of a literal, a global that starts zeroed — and those adopt the context on
their first push.
free on such a container is refused rather than made quietly shallow. It cannot
recurse, which is the whole premise, and releasing the outer block alone would
be "I freed it" written over a program that stranded everything inside; this
runtime refuses that collapse everywhere else. The message names free-all,
which is reachable by construction. clone stays refused for a reason the region
does not dissolve, and the old message had bundled the two failures under one
sentence: what disqualifies clone is not that it copies a header — so do at and
get, and they are fine, because they promise nothing — it is that clone
allocates a new block and promises independence, and a bytewise copy hands back
elements still pointing into the original's region.
A struct or union field is admitted only where the field's container holds
owning elements, because that container can only have been built against a
region. A field holding a plain (Vec u8) stays refused: nothing would force
that one into a region, and two copies of the aggregate would be two headers
over one heap block. vec-in-struct.flan still pins that.
The epoch already covered use after free-all, including the case this makes
reachable — an inner header copied out of an arena-held element into a local
still traps, because an Allocator is a pointer and a copied-by-value one would
carry its own epoch.
arena-value.flan builds the value by hand; arena-edn.flan reads a real document
through the tokenizer, and its reader takes no allocator and names none,
because spec-memory.md already puts the allocator in the calling convention.
arena-region.flan is the branch itself: run 0 is the (Vec (Vec i32)) control
that must not trap, and runs 1 and 2 are the two ways this dies.
Load.program takes forms: it reads the import forms, resolves them with the
one resolver it always had, and parses the file with the packages' macros in
front of it. The refusal said this needed a second import resolver at the Form
level. It did not notice that the file being compiled is parsed before Load
runs too, so no shape of the feature could have left import resolution where
it was.
Names arrive qualified, as a defn's do. (mac/twice 4) is a call and (twice 4)
is an unknown name.
Stopped mid-task: dune test was never run and the acceptance wiring is
unfinished. HANDOFF-macros.md has what is left.
handles.flan joins the ASan and memcheck corpora; pool-stale-region.flan
joins memcheck as a seventh program that aborts by design, for the reason
the other six are kept — a trap that stopped firing would be silent. Clean
both ways.
Loading a package kept one table, keyed by real path, and used it for two
different questions. Already loaded meant "skip", which is right for the second
route of a diamond and wrong for a ring: a package that imported itself round a
chain met its own entry, contributed nothing, and appeared to work. The comment
said so and called it a feature.
It is not one. A ring has no package order, and a definite package order is what
the macro expander needs — every defmacro has to be compiled before anything
that calls it. So the chain currently being read is now carried separately from
the set already finished. A directory found in the first is a cycle and is
refused; a directory found only in the second is still the diamond's second
route and still a no-op.
The refusal names the ring — a -> b -> c -> a — and only the ring, not the route
that led to it. "There is a cycle" leaves the reader to find which three imports
it was.
pkgs now comes back dependencies-first, which is the topological order the
acyclic rule buys. The declaration list is left alone: check.ml collects every
top-level name before it checks any body, so declarations are order-independent
by construction and sorting them would be churn in the field every test reads.
The tests are a real tree rather than a second copy of pkg-shared. pkg-diamond
builds a shape/Box inside area/ and hands it to a function declared inside
draw/, which only type-checks if the bottom package was read once — two copies
of one struct are two types. What proves it is the numbers, not the compile.
The mutation pass turned up one defect that did not make the suite go
red: a reader branch that forgets to advance reads the same character
for ever, and dune test waits as long as it is left to. In CI that is a
job killed by the runner with nothing named and no output to read.
watchdog.ml puts an alarm on every test binary — generous, because an
alarm that fires on a slow machine is a flake — and a five-second one
around each read in test_flan, where the budget really is small. The
first read that does not return wedges the rest, so a looping reader
costs five seconds and names the row instead of costing eight minutes
or never finishing. Both were watched: the string-escape loop now fails
in five seconds with the case named, and the per-binary backstop was
armed short and observed to fire.
A negative index into a global is silent in bounds.flan, which is
measured. "Because a global has no left redzone" was the explanation
put on it, and it does not survive the obvious test: declare another
defvar in front of arr and arr[-1] is caught, landing in that global's
right redzone. Underflow detection is a question about what the linker
put in front of the object, not about the access. Corrected in
test_sanitize, BUILT.md and NEXT.md.
NEXT.md's entry also goes back to its stated size. It had grown to 78
lines saying what BUILT.md says in the same commit range -- the
attribute, the -O0 decision, the bounds.flan table -- which is the
half-build-log the file's own header warns about. What stays here is
what is next: the UBSan gap as an undecided compiler question, the four
daemon-path buffers the corpus never reaches, and Valgrind.
NEXT.md's queued section becomes a landed one. The headline is not the
flag: ASan reaches Flan code only because Emit now attributes every
define, and UBSan reaches none of it and has no lever that would, so the
shift-UB and float-cast items that section listed are still open and are
a compiler feature rather than a flag.
The clean result is written with its reach. println.flan pushes a
1100-character string through escaped[1024] on purpose, so that buffer
is genuinely covered; scratch[64] never sees more than 20 characters;
and the 4K result cap, the dev registry guard, SNAP_MAX/SNAP_NAMES and
condition_name[128] are on the daemon path and not in the corpus at all
-- read, not tested. Two defects fixed, both found by reading. Three of
bounds.flan's six out-of-bounds cases caught with the checks off, with
the other three tabulated and explained, and the caveat that ASan sees
out-of-object and not out-of-subobject access, so three of six is a
ceiling and not a measurement.
BUILT.md gets the durable half: the attribute, the absent UBSan lever,
why --sanitize does not force -O0 when --debug does, and the -O0/-O2
divergence that earned it.
(slice s 2 1) has length 2 - 1 - 2 = -1. flan_bytes_to_i64 and
flan_bytes_to_f64 both wrote their clamp as (size_t)n < sizeof buf - 1,
and (size_t)(-1) is 18446744073709551615, which is not less than 511 --
so k took the cap and the memcpy copied 63 or 511 bytes out of a
five-byte string constant. ASan calls it a global-buffer-overflow in
flan_bytes_to_i64; the regression case is in test_sanitize.
Every other (ptr, len) entry point in the runtime already guarded the
negative case -- flan_write_stdout tests n > 0, flan_escape_bytes and
flan_dev_emit both fold a negative length to zero -- so this was two
exceptions rather than a missing convention. A checked build traps on
the reversed slice before reaching either, which is why it took an
--no-bounds-checks run to show.
Also clamps the three snprintf shims that publish scratch as a slice.
snprintf returns what it would have written, not what it did, so a
format that overran the 64-byte buffer would hand out a length past its
end. No format here can: %g is 13 characters and %lld is 20. Found by
reading, and the sweep could not have found it -- nothing in forty
programs prints a number that long.
Twenty-eight programs built twice -- once plain, once sanitized -- and
compared on output and exit status, plus two positive controls that are
the only reason a clean result means anything: an out-of-bounds read
that must report, and a shift by the width of the type that must not,
because UBSan cannot see hand-written IR and this file would otherwise
be claiming coverage it does not have.
Its own alias rather than dune test. A sanitized program is a statically
linked 1.8MB binary and takes tens of seconds to link; the sweep is nine
minutes against the existing suite's seconds, and a test nobody will
wait for is a test nobody runs. dune build --root . @sanitize.
The checked sweep is clean. The unchecked variant -- ASan alone, with
Flan's own bounds checks off -- catches three of bounds.flan's six
deliberate out-of-bounds cases and is listed with why for the other
three: a global has a right redzone and nothing to its left, so arr[-1]
is invisible; a read past a string constant folds away entirely at -O2
and is caught only at -O0; and a reversed slice reads nothing at all.
ASan is not a substitute for the bounds checks, and now there is a table
saying which half it covers.
