The refusal for a frame whose body has been redefined underneath it did not
fire because four of its five hand-offs were never written. `Emit.fninfo` has
been storing `slot_fingerprint` in the last `i32` of every `%fninfo` all along;
`flan_dev.c` called that field `spare`, there was no accessor for it, the agent
never snapshotted it, the backtrace line never carried it, and `Dev.locals`
compared slot counts and nothing else. The handoff note's "every piece is
written and the refusal does not happen" was a guess, and the first step it
suggested — printing both sides of the comparison — could not have found it,
because there was no comparison.
So: `spare` becomes `slotsig` and gets `flan_dev_frame_slotsig`; the agent
snapshots it beside the slot count and puts it on the backtrace line *before*
the location, since the name is the one field that can contain a space and has
to stay last; `Dev.backtrace` parses it; `Dev.locals` compares it against
`Emit.slot_fingerprint` of the body this session holds and refuses by name when
they differ. No change to `emit.ml` — the value was already there.
The mechanism itself is right and stays. `slot_fingerprint` hashes every slot's
name together with the spelling of its type, so a rename that keeps the count
and the types — exactly the case this exists for — changes it. The count check
stays in front of it because its message is the more specific one.
The fingerprint stays off the wire. A hash is not something an editor can act
on, and the refusal says the fact in words: this frame's body was redefined
since it was entered, so its names no longer describe its values.
`test_dev.ml` gains the inverse and the control. A body that drops a `let` is
refused on the count, and `main` — untouched by the redefinition of `look` —
must still answer, which is the assertion that would catch a fingerprint that
never matched anything and made the verb useless while turning the suite green.
Three things stood between the flagship program and the web target, and each
is answered here rather than worked around.
The brush was a path. (rl/load-texture "brush.png") hands raylib a filename to
open, and a bare relative path means nothing on a target with no filesystem.
It is (embed "brush.png") now, decoded through a new binding —
LoadImageFromMemory, declared (Ptr u8) plus an explicit count because the shim
generator refuses a slice parameter and says so, with a Flan wrapper taking
the slice apart exactly as collision-point-poly? and load-font-ex already do.
One decode now serves both textures: the unflipped upload first, then
ImageFlipHorizontal in place, then the mirrored one. load-texture and
load-image lose their only call site in this repository; that is deliberate,
because a path-based load is the thing that cannot work here.
A package's C may now be addressed to one target, the way a link line already
could. A .c file may carry a tag before its extension — flan_agent.web.c — and
on that target it is compiled and *replaces* the untagged file of the same
base name. Replacement rather than plain tagging, so that teaching a package
about a new target is additive: the file that was right on three targets is
not renamed to say so. Selection is in Build and not in Load, for the reason
select_lflags gives.
The dev agent on the web is a no-op, and the reasoning is written at length in
vendor/agent/flan_agent.web.c. Short version: the agent is a socket server and
a browser has no sockets, so the missing <sys/time.h> was the surface and not
the cause. Refusing vendor:agent on a web target was the other candidate and
is ruled out by arithmetic — Flan has no conditional compilation, sand.flan
calls agent/start unconditionally, Reach cannot prune a package something
reachable calls into, so a refusal means the program does not build for the
browser at all. This does not contradict the `barf` decision made earlier
today. `barf` is asked to make something durable, and a no-op returns success
to a program that now believes bytes are on disk. The agent is asked to accept
redefinitions, and on the web there is no editor, no socket and no session —
--dev is refused by name on every wasm target — so there is nothing to lose.
sand.flan already says the same of a native release build at the call site.
test/test_web.ml builds sand.flan for the browser and reads the module for
brush.png's own bytes, whole. Not "IHDR": stb_image carries that string itself,
linked in from raylib, so it would pass on a build where the embed emitted
nothing. It is not run — node has no DOM, so main reaches InitWindow and dies
inside glfwInit on `window is not defined`, which says the module is live and
nothing about whether the canvas paints.
test/dune gains brush.png, because an embed is read by the checker and the
headless case reaches sand.flan through ../../ from a sandboxed _build.
test_session's C-c C-k case now passes ~origin, which is what both editor
paths already send; omitting it was testing a request nobody makes.
dune test is green. Docs follow in the next commit.
The half the shadow stack was built for. A slot's entry in the frame is its
address, null until the binding that fills it has run, so "not bound yet at
this point" is a null and needs no liveness analysis. The daemon compiles a
thunk that renders the types it already knows -- Tast.fn.slots, with snames
beside them -- at the addresses the stopped program supplies, and reads the
text back the way C-x C-e does. Nothing is copied out, because a value with
no header is bytes with no meaning anywhere but in the program that holds
it.
That is render.ml's walk with its root changed, which is the pointer-rooted
thunk NEXT.md said this needed, and one new arm in the backend: a cast from
one pointer type to another, which emits nothing.
Only named slots are recorded. A recorded slot escapes and stops being
promotable, and the slots that would cost most are the ones with nothing to
show -- dotimes' bound, the temporaries min and max use, the walk's own
scratch. They are refused by name rather than shown under an invented one.
Recording every slot was built and timed and is inside the noise, so the
rule stands on what it shows.
Four refusals, each by name and with its reason: a slot nobody named, a
slot the program has not reached, a type the printer has no arm for, and
two whole frames -- an evaluation's thunk, and a frame running a body that
has been redefined since, where every slot index would be a guess.
Measured, minimum of nine runs: +61% on call-heavy code over globals
against +33% for the frames alone, 0.06% of a frame at 60fps.
plan.org has specified a shadow stack in the dev column since the beginning
and nothing had ever built it. A frame is four words on the calling
function's own stack: the one it displaced, a pointer to a static
description of the function, and two words reserved for its locals. The
name and the location travel on the frame, so a backtrace needs no debug
information, no symbol table, and nothing from the platform unwinder that
plan.org deliberately does not use.
The pop is at every ret, the landing block a transfer leaves through
included. That is the half that is easy to get wrong: a pop written only on
the normal path leaves a dead frame behind every handled error, and the
test takes five breaks and resumes all of them by transfer before asking
for two frames.
(:op "backtrace") answers from a snapshot the stopped thread takes, beside
the restarts and for the same reason, and marks which frames belong to the
program and which to the evaluation the break is inside. It is refused
while the program runs.
Measured, interleaved, three pairs of binaries: 29% on 600 frames of sand,
7.6% on a benchmark that is nothing but calls -- 32us per frame of sand, a
fifth of a percent of a frame at 60fps. An array with a stack pointer was
built and timed as the alternative and is worse on both.
The 4K result cap and condition_name[128] are on the agent's socket path, which
is why the sanitizer corpus cannot reach them: a program in the sweep has no
socket and nobody on the other end of it. test_agent has both.
A 5000-byte string literal evaluated into the running program comes back as
exactly 4096 bytes ending in the ellipsis result_end puts there to say it
clamped — and it comes back through the seqlock's copy, so the cap and the new
reader are pinned by the same case. The header also shows the generation as 1,
which is the count of complete values rather than the raw counter.
A condition class of 198 characters comes back from `status` as 127 and a
terminator. Aborting out of that break is what pins the exit status at 134 now
that the loop leaves with _exit rather than exit.
SNAP_MAX, SNAP_NAMES and the dev registry's overflow guard are still read
rather than tested. Sixty-five nested restart-cases and four thousand interned
names are a lot of program to write for a clamp each, and neither is on a path
this session changed.
flan_dev_result_cap() exists so the size is asked for rather than written down
in two files: "the copy is never truncated" is only true while the agent's
buffer and the runtime's bound agree, and the agent checks that where the copy
happens.
The pipe the queue program blocks on is close-on-exec, or the child inherits
the write end and its own stdin never reaches end of file — it sat in its last
read waiting for a byte only it could send.
publish() wrote queue[head % QUEUE] without consulting tail, so the 65th module
queued between two agent/poll calls landed on the slot the game thread was
reading — twenty-four bytes of function pointers copied field by field with no
atomic near them, so the consumer could take half of one job and half of
another and call it. The comment claimed the overflow dropped the oldest
request; nothing did that.
A full ring is refused now, at the sender, before the dlopen. Dropping loses a
reload the sender was told was ok, which is the same lie more quietly; blocking
stalls the accept loop, which serves connections inline, so a program that had
stopped polling would also stop answering status and abort — the dev loop would
have no way to reach a program that had stopped listening to it. The check is
separate from the store because there is one producer: room, once seen, cannot
be taken away.
Two smaller defects in the same file:
A module with no flan_reload_install was refused and its handle dropped on the
floor. Not an exception to "nothing is ever dlclosed" — that rule is about a
module something points into, and this one installed nothing, so no cell names
it. What leaked was the handle value rather than the mapping: dlopen refcounts
by path, so re-sending the same bad file raised a count nothing could lower.
exit(134) from the break loop runs the atexit chain and the ELF destructors,
which want the loader lock the listener thread may be holding inside dlopen. A
program asked to abort would hang instead of dying. _exit, with the streams
flushed by hand at each call site. The deadlock itself is read rather than
tested; what the tests pin is that the exit status is still 134.
programs/agent-queue.flan blocks on stdin so the window is held open by the
test rather than by a timer: it takes 64 modules, refuses the 65th with a
reason, and installs 64 when it finally polls. noinstall.c's destructor prints
while the program is still running, which is the only way to see the close — at
exit the loader runs every destructor whether anything was closed or not. Both
halves fail on the old code.
It read the generation, then a non-atomic length, then returned the buffer
itself — and the agent sent those bytes down a socket some time later, while
the game thread was free to be a hundred bytes into the next value. A seqlock
cannot validate a read that finishes after it returns, so the pointer was the
bug and not the ordering.
Made into a real one rather than documented down to what it guaranteed,
because what it guaranteed was nothing: the generation told the daemon a new
value had arrived and said nothing about whether the bytes it then read were
that value. Writing the honest comment would have left the daemon's only way of
reading a result unsound with a note beside it.
The counter is odd for exactly as long as a value is being written.
flan_dev_result_read copies into the caller's buffer and checks the counter
either side of the copy, retrying if it moved; a reader that loses the race
reports the last complete generation and no bytes, so a daemon polling for a
new value keeps polling rather than being shown half of one. The count handed
out is the number of complete values, so "has it moved" still means what
lib/dev.ml takes it to mean. The agent's buffer is RESULT_MAX, so the copy is
never truncated.
The race itself has no regression test. Arranging it means landing a socket
read inside a render thunk from outside the process, which is the same hook the
snapshot generation wants. What is tested is that eval still reads back the
value it rendered, through test_dev's existing cases.
The snapshot made the listing stand still; it did not make the handoff safe.
A choice is validated against the snapshot on top when the request lands and
resolved against the snapshot on top when the game thread next looks, and
between those the loop runs evaluations — one that errors pushes a break of
its own, whose loop reaches the flag first and takes its own index 2 for the
one somebody picked off the outer list. That is the failure this change exists
to end, arriving by a different door.
So each snapshot carries a generation, a choice carries the generation it was
validated against, and a loop claims only what is addressed to it. A mismatch
is left set rather than dropped: the listener already answered ok, so the break
it was meant for must still be able to take it. Depth could not do this — an
outer break resuming and a new one starting reuses the number. The snapshot is
also popped before the depth comes down now, so the two never describe
different breaks.
The client's own new path gets tested too: the candidate table is pure, so the
shapes a real daemon will not easily produce are checked directly, and the
break-and-resume test now goes through restart-at rather than by name.
flan-dev-unreachable-restarts was dead on arrival — flan-break reads
:unreachable off its own reply — and is gone.
Two frames offering `retry` put both on the break loop's list and only the
inner one within reach: §4's walk takes the first frame offering a name, by
definition, so the outer clause was drawn, offered, and unreachable. The old
prompt showed `retry` twice and sent the string either way. An index is the
only thing that can say which one, which is why SBCL identifies them
positionally too.
An index is worthless against a stack that moves, though, and this one moves:
the break loop is the poll loop, so every restart-case an evaluation enters
pushes and pops the same global list between the listing and the choice. So
the list is read once on entry and copied — names into the agent's own buffer,
frames as the addresses a transfer carries — and every answer comes from that.
The name still travels with the index as a receipt, checked against the
snapshot and refused if the two have drifted, so a bare integer can be wrong
out loud.
And the third state. A restart below the thunk a break is inside was accepted,
announced, and silently not taken: `flan_reload_call` holds its own transfer
channel and drops it on return, so the unwind stops at the thunk. The boundary
is now recorded where it is made, at the call — frames a restart-case inside
the thunk pushes are above it and still work — and such a restart is listed,
marked, and refused with the reason.
`break.flan` grew the shadowed pair, and 900 is a value no by-name lookup in
that file can produce.
Found by a concurrency audit that demonstrated three of them against a running
program rather than reasoning about them.
The break state was a flag, not a depth. A C-x C-e thunk may itself error, and
the break loop that catches it nests inside the first - so the inner loop's
resume stored broken = 0 while the outer one was still stopped. Every verb that
could rescue the program then answered "not stopped", status answered "running",
and the outer loop spun forever with no protocol path out. Only kill recovered
it, and Emacs' modeline read live throughout. The audit showed it with ticks
frozen at 0 beside :stopped nil. It is a depth now, capped, and past the cap the
program says so and exits rather than grinding. The condition name is saved and
restored per frame for the same reason.
An idle connection wedged the whole listener. The accept loop is single-threaded
and serves each connection inline on a blocking read, so a client that connected
and sent nothing - an editor killed mid-request - blocked every later request
including the abort that ends a stopped program. Worse, requests the client had
already given up on were served when its socket finally closed, so an abandoned
abort could kill the program minutes later against a state that had moved on.
Two seconds is generous for one line.
chosen_ready was cleared after the resume attempt, so a restart arriving in that
window was answered ok and then erased. It is claimed into a local and cleared
first now, which also keeps strlen off a buffer the listener may be writing.
And aborting was sticky: an abort that passed its check just as the program
resumed stayed armed and would have killed it at the next unhandled error,
minutes later, in unrelated code, giving nobody the chance to choose.
The break loop was reachable from a raw socket. This is the half that makes
it reachable from an editor, and it all follows from one fact: a program
stops at a moment nobody asked about.
So the state is learned twice, on purpose. It rides on every reply, beside
the program's output and for the same reason -- the likeliest instant for a
program to stop is the one just after an evaluation, which is a reply the
client is already reading, and learning it a second later from a poll would
mean learning it after the echo area had said the evaluation was fine. And a
timer asks anyway, once a second with `describe', because a program that
stops in a frame of its own game loop produces no reply at all and folding
state into replies that never come says nothing. The timer never reconnects
-- that would quietly erase the `lost' state that exists to be seen -- and
skips while a request is in flight, since accept-process-output runs timers
and a poll firing inside a read would eat that read's reply.
Three ops: `break' for the restart names, `restart' and `abort'. The
annotation owns :stopped and :condition rather than the ops, so one place in
the daemon decides whether the program is stopped and the poll and the prompt
cannot disagree. "ok" from `restart' means accepted, not resumed: the choice
is validated against the stopped stack and taken when that thread next comes
round, so it says so and the client clears its own flag rather than polling
once, finding it stopped, and re-opening the prompt it just answered.
The agent grew one verb, `status', answered in both states. Everything else
the break loop offers is refused while running, rightly; but the question an
editor asks without already knowing had to have an answer either way or there
would be nothing to poll.
And flan_agent_poll had to become re-entrant, which was a bug rather than an
addition. A C-x C-e thunk may itself error, and the break loop that catches
it polls again from inside that call. The old loop cached both indices and
stored tail at the end, rewinding over everything the nested poll consumed --
re-running the thunk that had just stopped the program, which is an unbounded
recursion of breaks. Each job is now claimed before it is run. test_dev.ml
evaluates an expression that errors and resumes it, which fails against the
old shape.
spec-conditions.md §2, and the reason the transfer was worth building. An
unhandled error runs a hook instead of rt_die(), on the frame that erred with
nothing unwound, lists the restarts between there and the top, and waits.
A hook rather than a direct call because the loop lives in vendor/agent, which
is an optional package, and flan_rt.c is the release runtime - a program with no
agent leaves it null and dies the way it always did. The hook resumes by writing
a restart into the transfer channel, which is the channel an invoke-restart
writes and reaches the same guard, so choosing from the break loop and choosing
from a handler are one act lowered once. §6 needed no change.
The break loop is the poll loop, run from the error rather than from the frame
boundary. That is load-bearing: an expression evaluated while stopped is a
module the listener queues and the game thread runs, so a loop that did not
drain that queue would hang C-x C-e exactly when it is wanted most. Installing
while stopped is allowed, which contradicts the rule that a redefined function
must not be swapped while it is on the stack - that rule is about mid-frame
consistency and there is no frame in progress here. The old body keeps running
and a retry reaches the new one through the cell, which is the whole point.
A restart frame carries its name now, beside the hash. Matching never needs it;
showing someone their choices does, and nothing at run time can turn a hash back
into a name.
A choice is checked on the listener thread against a stack the stopped game
thread is holding still. Answering ok and finding out on the game thread that
nothing offers that name would report success for something that cannot happen.
The test errors twice and takes a different restart each time, so a loop that
always resumed the same way fails it.
C-x C-e is the case that repeats - you evaluate expressions constantly and
redefine functions occasionally - and it is also the one case where unloading
is safe. The thunk is called directly by flan_reload_call rather than through a
cell, and it takes no registry slot, so once it has returned nothing points
into its text and the value it produced has been copied out. The module says so
with flan_reload_transient and the agent dlcloses it.
Skipping the registry matters for more than tidiness: the table holds 4096
names and an expression evaluated in a loop would have exhausted it.
A module that publishes a body can never make this claim, since leaving a
pointer behind is its whole purpose. Measured on a running program: sixteen
expression evaluations retain zero mappings, each redefinition retains three,
permanently and correctly.
A different primitive from redefining a name. There is no name to install a
body into, so the expression is wrapped in a function with nowhere to be called
from; the module exports flan_reload_call to say "run this once", and the agent
calls it after the install - on the game thread, at a frame boundary, so an
expression that reads the program's state sees a point the program agrees is
consistent.
Nothing is marshalled back because nothing could be. A Flan value carries no
header, so no code at run time can say what it is; the compiler knows the type
and renders it there, in the thunk. That is the layout decision's bill, and it
is why the printer set is the scalars rather than everything.
The rendering does not go through stdout. Stdout belongs to the program, it is
in the hot path for anything that prints, and a dev-only feature must not put a
branch in it - so flan_rt.c is untouched and the value goes to flan_dev_result,
read back over the agent's socket. Safe without a handshake because the
generation counter is bumped last: the daemon waits for it to move rather than
assuming the program has reached a frame boundary.
u64 refuses by name, because i64->bytes is signed and anything past 2^63 would
come back negative. Everything without a derived printer refuses the same way.
A number that is quietly wrong is the failure this whole thing exists to
prevent.
An evaluation is not a declaration: the thunk is built against the program and
never spliced into it, so describe does not fill up with an eval/N for every
expression ever typed.
The test that matters is the same expression twice. The fixture increments
ticks every frame, so two evaluations must disagree - a value computed in the
compiler, or read from a copy of the program's state, would not.
The piece between an editor and everything else. One long-lived Session, the
program it belongs to launched and owned by the same process, and a socket that
takes forms and installs them. What it adds over flan reload is that the
session persists - a defvar added by one evaluation is part of what the next is
checked against - and that it owns the build, which is what makes its layout
rules describe the process actually running rather than a guess about it.
The protocol is s-expressions rather than bencode, and I changed my mind about
that. The case for nREPL was reusing a designed op set and not re-litigating
session identity, but with the client ours too there is no CIDER to be
compatible with, its eval is string-in/string-out with no slot for which form
from which file, and Emacs already has read and prin1. So: one sexp per
message, length framed because the payload contains newlines. No parsing code
on the editor side, and on this side the parser is the language's own reader,
where :op is already a keyword and Flan source is already a string literal. An
nREPL front end can sit on the same Session later; it should not gate the
editor.
Two silent failures the daemon refuses to have. The agent socket is chosen by
the daemon and forced through FLAN_AGENT_SOCKET before spawning, because a
program's source has to name some path and a daemon that guessed would compile,
build and deliver a module to nobody. And delivery is checked: agent/start
returning 0 means a socket was bound, not that anyone connected, so a failed
connect or a reply that is not ok becomes an error the editor sees.
It waits for the program to bind before accepting an evaluation, since one
arriving first fails for a reason that reads like a compiler bug, and it
accepts with a timeout so a program that has exited takes the daemon with it
instead of leaving an editor waiting on a socket nobody serves.
lib/session.ml holds the declarations a running process was built from plus
every change accepted since, which is what an editor needs and what a one-shot
compiler cannot have.
Transactionality came for free. Check.program builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing
and the accumulated list is simply not replaced - no scratch-environment
machinery, which is what I was about to build. Re-checking the whole program
each evaluation costs the frontend, under 10ms, less than the llc after it.
There is a test for the case that matters: a typo, then a good form, in the
same session.
Which names the process was built with comes from the checked program, not from
any accumulated AST, because Check.program prepends the prelude and no AST
contains it. Derive it from declarations and print-line reads as new, gets a
registry cell nobody publishes, and the first call jumps to null.
Three changes are refused with a reason rather than loaded. A function's
signature, because a cell is a bare ptr and every call site compiled before the
change still passes the old arguments through it. A global's type, because the
storage exists and has a shape - reusing it reads at the wrong offsets, and
replacing it discards the state the reload exists to preserve. A struct's
fields, because the values the process is holding have the old layout. Note
what the checker already catches on its own: change a parameter type and the
caller fails to type check first, loudly. These rules only get a turn on a
change the checker accepts, which is a name nothing else in the program uses -
exactly where the silent version lives. Hence an unused defvar and a C-called
defn in the fixtures.
The accumulated list is the post-Load one, so an evaluated import is spliced as
its expansion. Otherwise re-evaluating a file that imports something appends a
second import, Load expands it again, and the duplicate-name pass rejects it.
C-c C-k on sand.flan's own text is the test.
flan reload now takes a program and a file of changed forms rather than a list
of function names and a --new list: the session works out which names are new,
which is the thing a bare CLI could not.
Also fixed, found by running the agent test under load: the agent took SIGPIPE
when a sender read part of a reply and closed. Replies go out with
MSG_NOSIGNAL, per call rather than by installing a handler, because the signal
disposition belongs to the program the agent is embedded in.
vendor/agent/ is a package like any other - agent.flan declares three calls,
flan_agent.c implements them, link asks for -lpthread. start listens on a unix
socket, poll installs whatever arrived and says how many, wait does the same
after waiting for something.
The split between poll and the listener is the whole design. dlopen relocates a
module and takes the loader lock, which is milliseconds and unbounded, so it
happens on the listener thread. flan_reload_install is one store per function
and must not land while a redefined function is on the stack, so it happens on
the game thread at the top of the frame, when the program asks. A ring and two
atomics connect them; the game thread never blocks on the loader.
wait exists for tests. A test that races the frame rate fails on a loaded
machine, so test/programs/agent.flan waits for the reload rather than sleeping
past it. It also sends a junk path first: the daemon is a separate process and
can send anything, and a bad path must be refused rather than take down the
program it was sent to.
Two things came out of running it. The reply goes out before the module is
queued, because the other way round the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection
reset instead of as ok. And ok means queued, not installed - the sender does
not get to know when the swap happened, since only the program knows when it is
between frames.
sand.flan now polls at the top of its loop, which is what this step was for.
Under Xvfb, one line on the socket and 455 consecutive frames drew from a
game-draw that did not exist when the process started. Building without --dev
still works: there are no cells, so a module is refused on the listener thread
and the loop never notices.
flan reload builds one module the way the daemon will. --new names what the
host was not built with, which is the one thing the command cannot work out for
itself and exactly what the session will track.
