The daemon side and the Emacs side both landed with nothing written down. Four
files owed something.
`BUILT.md` gets the whole of it: why rooting at an address alone was rejected
and why that rejection was half wrong, what a path step is and how a union's
case travels with it, why the slot goes by index and not by name, and the two
capability lists side by side — the expression root works on a running program
and cannot name a frame; the slot root names one frame and one slot and reaches
an option's payload and a union case's fields, and needs the program stopped.
Neither contains the other, which is the reason there are two.
`emacs/MANUAL.md` says the same thing in the register that file uses, under the
inspector, because the person pressing `i` is the one who needs to know which
root they got and what it cannot do. The globals section's claim that `i` works
on a global "exactly as it does on a local" was true and is now the interesting
difference, so it says what the difference is.
`NEXT.md`'s decided item is struck with what actually shipped: a frame and a
slot index rather than an address and a type, and `l` crossing between the modes
was predicted as a cost and turned out not to be one.
`DISCUSS.md` item 1 is no longer an open question. The number stays — cimport.ml
and NEXT.md cite these by number — and what stays with it is the one correction
worth keeping: an address is not an expression, but a step does not have to be
one either.
And BUILT.md's last paragraph still said `render.ml` prints `(V {:x 1.5})` and
that the printer would move when its reader did. They moved together some time
ago.
Running a macro means compiling it and loading it into the compiler, and the
step that reads as small in NEXT.md is not: OCaml has no dlopen for ELF, and
lib/dune had no foreign_stubs. So the boundary is built first and the expander
not at all. lib/dynload_stubs.c is the whole of it — dlopen, dlsym, a
four-argument call into a macro thunk, and a peek/poke family, because OCaml
cannot address the raw memory a Form image has to be laid out in.
Nothing aggregate crosses to C. The unions lane verified a union's memory
layout against clang, which is a different claim from LLVM's convention for an
aggregate passed or returned by value in hand-written IR, so Emit.macro_thunk
wraps every macro in void(ptr,i64,ptr,ptr): the slice is built and the result
stored on the LLVM side, and the compiler's side is four pointers.
Build.macro_module links the runtime in rather than declaring it external, so
the module has no undefined symbols and the compiler's own link needs no
-rdynamic. That is the difference from Build.shared, whose host is a running
Flan program.
defunion Form and the list-building surface quasiquote will desugar into are in
the prelude. Form mirrors Form.value and not Form.t: no loc field, so the
compiler stamps the call site's location onto everything a macro returns.
The compiler builds. dune test was not run, and Form's layout is asserted
nowhere — NEXT.md's new handoff section says what the three numbers are, what
the next two commits should be, and the four decisions this made that the
design did not settle.
BUILT.md gets the section and NEXT.md's item 5 and its diagnostics bug are
struck through.
The decisions worth recording are the ones nothing upstream had made: an i32
tag, a payload aligned to the widest member of any case, qualified
construction and bare patterns, declaration-order tags -- so case order is
part of a union's contract the way field order is a struct's -- and a
non-exhaustive match refused rather than defaulted.
And the one finding the macro lane needs: an imported union is still refused
at load.ml:312, but the prelude is prepended into the same flat namespace
before collect runs, so a defunion Form in prelude.ml needs no import and no
load.ml change. Verified by declaring one there and matching it.
The globals section attributed a frame by its slot fingerprint, which is the
wrong cut for it: a redefined body can name entirely different globals while
binding identical locals, so the check saw no change and the new body's
reference set went into the union under the old body's frame, with the frame
numbers beside an entry saying so.
So a second fingerprint. Reach.ref_fingerprint hashes the set of globals a body
names — sorted and deduplicated, because a reference set is not ordered, where
slot indices make the slot fingerprint order-sensitive on purpose — and it
travels the path the first one already cut: %fninfo, flan_dev_frame_refsig, the
agent's snapshot, the backtrace line, Dev.globals_op. Different means the frame
is skipped by name with its reason, and the rest of the stack still contributes.
Two numbers rather than one, because they are two facts. A frame whose slots
match and whose globals do not has locals that are perfectly readable and
attribution that is not, and a combined hash would make locals refuse a frame
with nothing wrong with it. locals still checks the slot fingerprint alone.
It lives in reach.ml because expr_refs is already the walk that answers what a
body refers to, and is the walk the union itself is built from. One consequence:
emit now reaches reach, which closes a cycle through Load if cimport calls
Build.cachedir, so the header cache spells the object cache directory itself.
test_dev.ml drives the exact case — a body that binds identical locals and names
untouched where the stopped frame names pressure. With the check disabled it
fails twice: the missing refusal, and untouched appearing under frame 0.
The 15.5ms attributed to re-reading the header on every reload is not that.
A timer around each stage says the cached dump reads in 0.33ms, the extraction
takes 3.3ms and the checks 0.55ms — about 4ms, once, in Session.create. The
rest of flan reload's delta is Load and Check over 256 more declarations, and
the +3.6ms a redefinition really pays is Check and Emit.redefinition against a
bigger program. A C-c C-c reads no header at all: eval's forms carry no import,
so no package is read.
Both cache levels anyway, because a long-lived process should pay nothing
twice. In the session, two tables: the dump by header, the declarations by
header and by what the package already declares. On disk, the existing cache
moved into the object cache directory beside the .o files. The in-memory key
is the path and the flags with no mtime, so a header edited mid-session is not
picked up until the session restarts — the rule a changed .c file follows, and
the rule that keeps new signatures from being checked against a process still
running the old layouts.
Measured: repeat import 3.65ms to nothing; flan reload unchanged, as it must
be, since it imports once per process.
Loading a package kept one table, keyed by real path, and used it for two
different questions. Already loaded meant "skip", which is right for the second
route of a diamond and wrong for a ring: a package that imported itself round a
chain met its own entry, contributed nothing, and appeared to work. The comment
said so and called it a feature.
It is not one. A ring has no package order, and a definite package order is what
the macro expander needs — every defmacro has to be compiled before anything
that calls it. So the chain currently being read is now carried separately from
the set already finished. A directory found in the first is a cycle and is
refused; a directory found only in the second is still the diamond's second
route and still a no-op.
The refusal names the ring — a -> b -> c -> a — and only the ring, not the route
that led to it. "There is a cycle" leaves the reader to find which three imports
it was.
pkgs now comes back dependencies-first, which is the topological order the
acyclic rule buys. The declaration list is left alone: check.ml collects every
top-level name before it checks any body, so declarations are order-independent
by construction and sorting them would be churn in the field every test reads.
The tests are a real tree rather than a second copy of pkg-shared. pkg-diamond
builds a shape/Box inside area/ and hands it to a function declared inside
draw/, which only type-checks if the bottom package was read once — two copies
of one struct are two types. What proves it is the numbers, not the compile.
BUILT.md gets the Map and the defer relaxation; NEXT.md strikes step 4
and item 3, and records four things that are genuinely open rather than
finished.
The one worth reading is that the Map is slower than CPython's dict at a
million entries while being six times quicker cache-resident. Both are
memory-bound at that size and this layout waits longer: keys, values and
hashes are three separate runs, so a lookup that misses everything costs
three cache misses where a compact dict costs two, and the hash run is a
full eight bytes a slot. Cell packing buys probe locality, which is a
win while the hash run is resident and a loss once nothing is. One byte
of metadata a slot is the known answer and is not built, and the
crossover between the two results is somewhere nobody has looked.
Also recorded: the defer change amends a frozen spec-memory.md, which
said a defer for a let-bound value was not expressible; and the Map is
narrower than the spec on one point, a fixed array being a key only when
its elements compare bytewise.
A global is program state a frame happened to touch, not part of it, so
nesting it under one implies an ownership that is not there and repeats the
name once per frame that reads it. One section instead, holding the union of
the globals every frame on the stack references — the compiler does the
choosing, since Reach.expr_refs already answers a body's reference set, and
listing every global a program has would bury the one that matters under the
prelude's PRNG state.
Each entry says which frames touch it, by the index the stack section already
numbers them with, which recovers what per-frame nesting would have told you
at no cost in duplication. Ordered by the innermost frame that touches it:
a deep stack makes the union large and proximity to the error is what puts
the likely culprit on top.
Simpler than locals, because a global is reached by name rather than by
address. Emit.redefinition writes a global the host has as external, so the
thunk binds to the program's own storage and nothing is asked of the stopped
thread — no dev-slot round trip and no not-yet-bound case to refuse.
A frame that cannot be attributed contributes nothing and is named in
:skipped; the union being incomplete and the union being complete are
different answers. The hole in that is stated rather than papered over:
slot_fingerprint hashes a body's slots, which is the right cut for locals and
not for this, so a body that names different globals while binding the same
locals is not caught. The test drives the case that is.
MANUAL.md also loses a stale paragraph claiming the fingerprint check never
fires with a failing test pinned to it. It fires, and test_dev covers it.
BUILT.md gains "The header is read now", directly under the section whose last
paragraph promised that reading a header was what would convert the trusted
half into a checked one and that it was not built. That sentence is replaced by
a pointer to the one below it, in BUILT.md and in shim.ml's docstring both.
It records the things worth not re-deriving: why the dump and not libclang (and
that Zig left libclang too, which strengthens the argument rather than weakening
it), why the import is bounded by the package's own defstructs, why generating
defstructs would make the check circular in exactly the way a _Static_assert
was rejected for, refusal-by-demotion from Zig's failDecl, the naming rule and
what it must actually guarantee, and both const-vs-non-const char * and the
target-varying widths.
The diff and the costs are stated as measurements, with the table: 16 of 16
defstructs and 172 of 172 declare-c agree against 5.5, ten real differences
against 5.1-dev, release +4ms warm, redefinition 31.0 -> 46.5ms.
DISCUSS.md item 6 is rewritten rather than removed. The mechanism question is
settled and is now in BUILT.md; what is left is narrower and is two decisions
that are the author's — whether the header stays a build-time read or becomes a
committed generator, and whether the 172 hand-written lines migrate. Both have
the argument on each side written out, including what migration would lose:
key-pressed? is a better name than is-key-pressed, and an enum parameter
imports as i32 because nothing tells the importer the package calls KeyboardKey
"Key".
The indentation rules were written and tested but never described anywhere a
user would look. MANUAL.md had no section on editing at all — it starts at
`C-c C-c' and assumes the file is already written — so the rule that cost the
friction, a binding vector lining up name under name, was only visible by
trying it.
What is written down is what was checked, not what the port was aimed at: the
call fallback, the `handler-bind' clause vector, `defn' parameter alignment
with a return type after it, and `restart-case' clause bodies were each
reindented from scratch and the manual quotes the result.
NEXT.md keeps the half of the field-label handover that is still open. The
printer in render.ml has to move in the same commit as the inspector that
parses it, and that is the inspector lane's; the font-lock half is done here,
so only that half is struck.
check.ml's prose carried struct literals in the old spelling in two
comments the form-level scan does not see, OCaml comments not being forms.
The Emacs handoff said MANUAL.md and flan-mode.el's font-lock still show
the colon. MANUAL.md does not mention a struct literal at all. font-lock
does have something, but it is the opposite of what was written: it colours
:name as a constant and has no rule for .name, so a field label is now
unfontified rather than wrongly coloured. Said accurately, with the line.
runtime/flan_rt.c:256 also shows {:name ...} and is left alone on purpose --
it describes the *printed* form, which still uses colons and is correct.
web/index.html's Flan blocks convert and its output blocks do not, which
is the same split render.ml makes: the printed form keeps the colon until
the Emacs inspector that reads it moves too. Same in BUILT.md.
plan.org, spec-conditions.md and spec-memory.md carried struct literals in
the old spelling and now do not.
NEXT.md decision 6 is struck, and batch item 2 with it, naming what to run
at merge. BUILT.md says why the colon belongs to keys -- mostly that a map
literal wants {:key value}, and two literals sharing one syntax would have
left the reader asking the checker which it was looking at.
The sweep was not idempotent and is now: {.k :hi} -- a field already
converted, holding an enum member -- read as a destructuring pair on a
second run and ate the member. A re-run over a lane's files would have
corrupted them silently, which is exactly what the tool exists to do
safely.
BUILT.md's locals section said the second whole-frame refusal was a slot count
mismatch. It is a fingerprint, and the paragraph now says why a count could not
have done the job: the case it exists for is a rename, which changes neither
the count nor the types. It also states the bound honestly — a 30-bit hash can
collide, and a collision would reproduce exactly the wrong answer this catches,
but only between two differing bodies of a function whose name already matched.
NEXT.md's item 1 is struck, and the handoff paragraph that diagnosed this is
marked wrong rather than deleted. It claimed every piece was written and one of
five hand-offs was dropping the number; four were never written. The step it
recommended first could not have found that, and a lane stopping mid-repair
should say which pieces it ran rather than which it believes it wrote.
BUILT.md gains the section: why the path had to become an embed, the rule for
a target-tagged .c file, why the agent is a no-op there and why that is not the
barf decision reversed, the exact commands including the .html output name and
the server a wasm module needs, and the four things only a human opening it can
settle.
NEXT.md strikes web blocker 1 and rewrites blocker 3 — nothing has been opened
in a browser is still true, and is now the only thing left.
locals compares the frame on the stack against the body the session holds:
installing while stopped is allowed, so the two can be different bodies of
one function, and a rename that keeps the slot count pairs every name with
the wrong value. Emit.slot_fingerprint hashes each slot's name and type,
emit_fn puts it in the frame's static description, the agent reports it on
the backtrace line and Dev.locals compares it.
It does not fire. The test that drives it -- a redefinition that renames
every local of a function that is on the stack -- fails, and is committed
failing rather than deleted, because it is the only record of what is
wrong. Everything else in the suite is green; this one check is red.
It builds. See NEXT.md's handoff for where to look first.
The half the shadow stack was built for. A slot's entry in the frame is its
address, null until the binding that fills it has run, so "not bound yet at
this point" is a null and needs no liveness analysis. The daemon compiles a
thunk that renders the types it already knows -- Tast.fn.slots, with snames
beside them -- at the addresses the stopped program supplies, and reads the
text back the way C-x C-e does. Nothing is copied out, because a value with
no header is bytes with no meaning anywhere but in the program that holds
it.
That is render.ml's walk with its root changed, which is the pointer-rooted
thunk NEXT.md said this needed, and one new arm in the backend: a cast from
one pointer type to another, which emits nothing.
Only named slots are recorded. A recorded slot escapes and stops being
promotable, and the slots that would cost most are the ones with nothing to
show -- dotimes' bound, the temporaries min and max use, the walk's own
scratch. They are refused by name rather than shown under an invented one.
Recording every slot was built and timed and is inside the noise, so the
rule stands on what it shows.
Four refusals, each by name and with its reason: a slot nobody named, a
slot the program has not reached, a type the printer has no arm for, and
two whole frames -- an evaluation's thunk, and a frame running a body that
has been redefined since, where every slot index would be a guess.
Measured, minimum of nine runs: +61% on call-heavy code over globals
against +33% for the frames alone, 0.06% of a frame at 60fps.
plan.org has specified a shadow stack in the dev column since the beginning
and nothing had ever built it. A frame is four words on the calling
function's own stack: the one it displaced, a pointer to a static
description of the function, and two words reserved for its locals. The
name and the location travel on the frame, so a backtrace needs no debug
information, no symbol table, and nothing from the platform unwinder that
plan.org deliberately does not use.
The pop is at every ret, the landing block a transfer leaves through
included. That is the half that is easy to get wrong: a pop written only on
the normal path leaves a dead frame behind every handled error, and the
test takes five breaks and resumes all of them by transfer before asking
for two frames.
(:op "backtrace") answers from a snapshot the stopped thread takes, beside
the restarts and for the same reason, and marks which frames belong to the
program and which to the evaluation the break is inside. It is refused
while the program runs.
Measured, interleaved, three pairs of binaries: 29% on 600 frames of sand,
7.6% on a benchmark that is nothing but calls -- 32us per frame of sand, a
fifth of a percent of a frame at 60fps. An array with a stack pointer was
built and timed as the alternative and is worse on both.
NEXT.md strikes decisions 1, 2 and 5, and the web target's "assets are
two questions" item, which the embed answered with a third option
neither half of it considered: make it a compiler feature and neither
question arises. That item's diagnosis was right — the file that needs
the asset is structurally the one file that cannot declare it — and its
conclusion, that the fix must be a link channel or a new declaration,
was wrong.
BUILT.md gets the two sections. The embed one records the choice a
reader would otherwise have to reverse-engineer: the bytes are a Str
node typed [u8] rather than a Bytes prim over a string, because the
prim is identity but makes the node non-constant, and an embed-dir in a
defconst then cannot be an LLVM constant. It also states the .rodata
write hole loudly, because an embedded asset is precisely what someone
will try to decode in place.
The slurp/barf one writes down what the host ABI grew by and why that
much: three POSIX-shaped calls and one reason reader, Vec-ignorant, with
the Vec-aware half as runtime glue rather than a fourth call. And it
records the gap the feature revealed without fixing — a handler that
wants "try to save, carry on if you cannot" has nowhere to go, because
error is diverging and neither restart means give up.
NEXT.md is what is left, so the three steps go, the four questions that were
unsettled under them are answered or replaced, and the reasoning that outlives
the change moves to BUILT.md: why Allocator being a builtin opaque type made
milestone 5 unnecessary, why with-allocator is its own IR node, why the move
rule unions at a join, what ownership is not transitive through yet and what
each refusal waits on, and why StorageExhausted had to land with Vec.
Four things replace the old open list. The use-after-release question is still
open but now has evidence available for the first time, because there is a Vec
to write arena programs with. The release layout is blocked on the reload path
carrying build flags. The generation word has no reader and will not have one
until a slice can carry a Vec's identity. And the budget is an addition to the
spec that wants folding in or replacing with a growable arena.
Says what closed each one and how, and that two of the four named
buffers now have evidence rather than a reading. The count of nineteen
stands as the old one: the mutation pass has not been re-run.
The thunk calls flan_dev_result_begin before it evaluates anything, so an
expression that signals is stopped inside the seqlock's window — and a restart
taken from that break transfers past the thunk, so the matching end never runs.
An unpaired begin cost nothing while the counter only moved at the end. It
costs everything now: incrementing would leave the count odd for the life of
the process, every later read reporting a write in progress, and C-x C-e dead
until the program restarts.
So begin sets the low bit rather than incrementing, and end clears it by
setting rather than adding. The ordinary sequence is unchanged — 2k, 2k+1,
2k+2 — and an abandoned write is over as soon as the next evaluation starts.
What that does not fix, because one buffer cannot: an evaluation running while
another is stopped mid-render shares the buffer, so the inner value is the one
that survives. That was true before the counter was a seqlock and is not a
regression.
Also noted in NEXT.md: rt_die in flan_rt.c has the same exit-with-the-loader-
lock-held shape the break loop just lost. Not fixed with it, because rt_die is
the non-dev path too, where there is no listener to deadlock against — whether
it should be _exit always or only under --dev is a decision.
And the 4K-cap assertions clamp their own String.sub, so a short body prints a
failure instead of raising out of the test.
