Three gaps in the first pass. The park flushes after the poll, which is the
only thing that gets a printing expression's line out of a fully buffered pipe
before the next run — now pinned by its position in the transcript. The globals
section under a parked break answers with everything skipped, which is a
different reply from the timeout a job nobody polled would give, so it is asked
for rather than assumed. And the render thunk's five-second sentence no longer
asks a parked program whether it is calling agent/poll.
The park waited on one flag and could do one thing, so C-x C-e on (+ 1 1) was
refused for want of a frame boundary — an expression that needs nothing from
the program, in a process holding every global the run left.
It waits on two now. A re-run leaves the park; a wake drains the agent's ring
and waits again, with the state still PARKED, which is what makes running the
thunk there exactly as safe as running it at a frame boundary: while parked
there is no concurrency to be unsafe against. The break loop is the precedent
and CL's spawned worker is deliberately not copied.
A thunk that stops now stops on a parked thread, so the restart ops refuse on
whether a break is engaged rather than on the state, and a paused expression
against the park is resumable.
Under --two-process the compiled program is a child of the daemon, and the
daemon is the only thing that ever ends it: the program is a loop with no
reason of its own to stop, and lib/dev.ml kills it from a Fun.protect finally
block on the way out. That block does not run when the daemon is SIGKILLed —
by a harness tearing down a daemon it gave up on, by a watchdog, by anybody —
and what was left behind was a program with ppid 1, sleeping, holding a socket
nothing will ever connect to. Eight of those were on the machine this was
written on, the oldest six days old, and one had been minted by this suite.
The child now arms PR_SET_PDEATHSIG when it starts its agent, gated on a
FLAN_DEV_PARENT the daemon sets and nothing else does, and closes the race
where the daemon died before the arming by comparing getppid against that pid
rather than against 1 — under a subreaper an orphan is not reparented to init.
The signal is SIGPWR and not SIGTERM, because the daemon's ordinary teardown
already sends the child SIGTERM and the child's stderr is the daemon's own; a
handler there would print "the daemon is gone" into the daemon's output on
every clean close. The handler writes one sentence, unlinks the agent socket
it bound, and exits 0, and it does nothing else because it can run on the game
thread inside stdio's lock.
A merged build arms none of this and must not. There the daemon is the
process, so a dead daemon is a dead program; its parent is whoever typed
flan dev, which is a shell or an emacs and not the session's owner. Merged
daemons do leak on this machine — four of the eight — but for a different
reason, which is that accept_loop has no idea whether its editor still exists.
That wants a mechanism that tracks the client rather than the parent, and it
is not this change.
The daemon's read end of the program's stdout pipe is also cloexec now. The
child used to inherit it, which is both a leak and the reason waiting for EOF
on that pipe could never have been the mechanism: the pipe always had a
reader. A printing program now takes SIGPIPE once its daemon is gone, which is
a second net under chatty programs and not a substitute — the test uses
dev-watch, which is silent, so that only the thing under test can end it.
The test spawns the two-process shape, sends one request so that what it kills
is demonstrably serving, reads the program's pid out of the daemon's
/proc children before killing it — it is a grandchild, so waitpid is ECHILD —
and SIGKILLs the daemon. SIGKILL and not SIGTERM: SIGTERM lets the finally
block run and the assertion would be green with this change reverted. It fails
that way, and passes with it, in well under a second.
dev-chatty.flan outlives the surveys' twenty seconds by design, and unlike
dev-repl it prints while it does -- so the two backends stop at different
lines and the diff reports on scheduling rather than on lowering. It joins
dev-loop and dev-watch in the excluded-by-name list in both sweeps, with the
distinction written down.
rt_flush_out is guarded on __wasm__: the pipe it is careful about belongs to a
merged flan dev, which is only ever a native host, and wasm32 need not answer
for a descriptor mode its runtime may model differently.
And three comments that went false with the _exit: the atexit registration in
the merged entry point is no longer there for rt_die, which unlinks the socket
for itself now, so both places that said so say what it is actually left
covering.
It ended "send it again once the program is running", which reads as a
withdrawal of what eval promises a parked program on every other evaluation:
that a body redefined while parked installs when the program is run again.
What is true is narrower — this module was not taken, and the run that drains
the sixty-four already queued is what makes room for it.
The three named arms are not every way a step after the check can fail.
write_file makes the daemon's copy of the module's text before llc is called at
all, and a Sys_error from it went out through serve's guard with the session
already holding the declaration — the same stranded declaration under a
different exception. Dev.eval now restores on the way past anything that
escapes, and carries an accepted flag so that a module the agent has already
taken is never rolled back out from under the process.
That is also the seam the crash can be reproduced through. test_dev takes the
daemon's working directory away from it — its own, not the macro cache the two
failures above use, which cannot fail a build with no macro in it — evaluates a
defn, and then calls it. Without the restore the daemon lists probe-two in
describe and the call closes the socket: the null cell, the game thread, and
address 0.
rt_die began with fflush(stdout) and ended with exit(134), and under a merged
flan dev both could hang. The flush waits on a 64K pipe whose reader is busy;
exit runs the atexit chain and the ELF destructors, which want the loader lock
a dlopening listener thread may be holding, and in that build the chain also
holds OCaml's shutdown. A bounds failure that hangs is worse than one that
kills the process, because nothing is printed and nothing exits.
So the flush every trap does first goes through rt_flush_out, which puts fd 1
into non-blocking mode and accepts a truncated tail, and rt_die takes _exit
and unlinks FLAN_DEV_SOCK by hand -- the same shape as die_now in
flan_agent.c, which argued all of this first and is quoted rather than
diverged from. On a terminal, a file, or a pipe with room, O_NONBLOCK changes
nothing, and the acceptance corpus diffs that output.
One consequence worth knowing: _exit skips ASan's exit-time leak report, so a
trapping program under @sanitize no longer reports leaks.
Closing the window of a printing program left the pipe full, and the flushes
on the way out blocked on a compiler thread that was busy answering the very
request asking to run it again. program_state still said RUNNING, so rerun
refused a finished program as "the program is already running".
Two flushes moved. flan_exit no longer flushes before the hook -- exit(3)
flushes for itself, so that call only ever ran on the merged path, and there
it was the first thing to block. flan_merged_park flips the state under the
lock first and flushes after, because a state flip is two stores and cannot
wait on anything.
What widens is the window in which the program is parked and not yet on the
condvar. Nothing is lost there: program_asked is the fact and a signal
delivered to nobody is discarded. Two re-runs in that window are both
answered ok for one run, which is the trade against a refusal that was false.
The merged build's stdout is a 64K pipe back into the daemon's own process,
and the accept loop is the only thing reading it -- which it is not doing
while serve is answering a request. The two five-second waits for a frame
boundary now drain the pipe on every tick, so a program stopped inside fwrite
is one the daemon lets go rather than one it waits out and then accuses of
not calling agent/poll.
drain and not take: the text stays in the buffer until with_output puts it on
the reply, which is where the output an evaluation caused belongs. And the
drain sits beside the sleep rather than inside the select, because a readable
pipe would make the tick free and count the timeout out in a fraction of it.
dev-chatty.flan prints 4K a frame, which is the only fixture here that fills
the pipe at all; without the drain it fails in 5.1s with the old sentence.
The agent refuses a module it has no room to queue with "the program is not
calling agent/poll". For a running program that is the cause. For a parked one
it is confidently wrong: there is no game thread left to poll with, nothing
drains the ring until somebody runs the program again, and the ring is full
precisely because Dev.eval accepts evaluations while parked and promises they
install at the next run. At install sixty-five that promise breaks, and the
reply sent the reader to inspect a loop that is not running.
The agent cannot know this — parking is the merged shim's state — so the
daemon, which has just asked liveness, rewrites that one reply the way
abi_mismatch rewrites dlerror's. Every other refusal is still quoted in the
agent's own words, and a running program's is unchanged.
Pinned on Dev.refusal rather than end to end: filling the ring is sixty-four
real clang runs, and what the daemon does with the agent's words is the whole
of the change.
Session.eval committed the checked program the moment a form checked, and the
two steps that can still refuse it — the build, and the agent taking the
module — come after that, in the daemon. Either one left the editor holding an
error and the session holding a declaration the process has no body for. The
next module built for that session lists the name in its install prologue,
which interns a cell for it and never stores anything into it, and a dev
build's call through a cell has no null test in front of it: the game thread
jumps to address 0 at the next C-x C-e, locals render or globals refresh.
Session.held takes the four fields eval commits as one unit and Session.restore
puts them back. Dev.eval and Dev.eval_expr take one before checking and restore
it on every arm where nothing was accepted — a failed build, a refused
delivery, an unreachable agent. eval_expr needed it for the generic instances
it keeps, whose own comment already claimed the invariant: "the session must
not come to believe it holds a body that no module was ever written for."
A delivered module that then times out is not a refusal and does not roll back:
the agent has it and will install it at a frame boundary.
The name freed up by the rename now means what C means by it: the members
overlay one storage, the size is the largest of them, the alignment the
strictest, and nothing anywhere records which one was written. It serves
two things that wanted it. Binding a C header means holding the union the
library holds and reading whichever member the library's own tag says is
live -- a tag Flan cannot see, because the rule relating them is prose in
a manual. Overlaying an f32 on a u32 to look at its bits is the other,
and it is the same read.
So that read is defined rather than refused. This is the one place in the
checker where bytes win over safety on purpose, and the alternative was
not a safer language, it was no feature: type punning *is* reading the
member that was not written. The promise is the one C's implementations
make and C's standard does not -- the layout is the target's, the bytes
are the bytes, a read is a reinterpretation of them -- and what is not
promised is anything about bytes nobody wrote, where a member wider than
the one last stored reads a tail that is indeterminate exactly as a
struct's padding is. ZII narrows that to almost nothing: a union starts
all-bytes-zero unless uninit says otherwise.
uninit on one is allowed, unlike on a defdata. The refusal there was
never about garbage; it is that a tag steers, and a tag no case names
falls past every comparison in a match into a block LLVM may treat as
unreachable. An untagged union steers nothing.
Which is also why three things are refused, each for a reason that does
not expire with a milestone. No move-only member: nothing knows which
member is live, so nothing can tear one down, and unlike the struct and
defdata refusals this is not waiting on recursive teardown -- there is no
fact for teardown to read. No bool at any depth: an i1 loaded from a byte
that is neither 0 nor 1 is a value the optimiser may assume cannot exist,
and a union is the only type that can produce one. No defdata at any
depth, for the reason uninit gives, arriving the other way round. An
Option member is fine and the walk says why: its match is a tag test and
a branch, not a chain with an unreachable tail.
Two members in one literal, a match on a union, a union map key and a
member written into a global initialiser are each refused by name.
A union is a field list whose every offset is zero, so it travels as a
Tast.structure and the checker, the emitter and the x86 backend each grow
one table rather than one shape. A value is a zeroed temporary and a
store -- Set over Pfield, which every backend already has -- so there is
no new IR node and no layout rule spelled out a second time per backend.
The LLVM type is the blob clang gives a union, the DWARF is
DW_TAG_union_type with every member at zero, and the printer names the
type and does not walk it: it cannot know which member is live, and one
of them may be a pointer.
cimport can now check what it could not. A C record holding a union
member was not recorded at all, so the defstruct beside it went unchecked
rather than checked wrongly; a named union member resolves to a defunion
now and the whole record is compared field by field. The defunion itself
is compared against the header's union as a set and not in order --
every member is at offset zero, so a permuted one is the same type and
reporting it would be a finding that is not one -- while a member the
header has and Flan lacks is reported, because that is what changes the
size. A defunion against a C struct, or a defstruct against a C union,
is reported in both directions. An anonymous union member is still
skipped, and the comment now says that the gap is on the Flan side:
there is nothing to declare.
flan dev's merged build is the program and the compiler in one -rdynamic
executable, so it exports every flan.* body it has, and ELF gives it precedence
over anything dlopened afterwards. The compiler expands a macro by dlopening a
module into that same process, and the module is built by Emit.program whatever
backend the session uses -- so under --x86 the caller was LLVM's and the body it
landed in was the dev backend's, which is a crossed pair. It died with SIGSEGV
inside flan.[clamp] during the first expansion, before the program had run a
line, and Dev.start refused the combination rather than do that.
Build.macro_module now asks Emit.program for hidden visibility on the module's
own Flan definitions. There is nothing left for the host to interpose, and the
flan.macro.* thunks stay exported because dlsym is how the compiler reaches
them -- nm -D on the built module lists those three and nothing else of Flan's.
The -Wl,-Bsymbolic that had been binding everything locally since 65d14f4 goes
with it: the module links its own flan_rt.c, and binding that locally aimed its
calls at a runtime flan_rt_init never ran on, with a null flan_exit_hook, so a
trap raised inside an expansion would have exited the process instead of parking
it.
Nothing about the host moved, which is what keeps redefinition modules reaching
its cells, its globals and flan_dev_cell. hidden defaults to false, and the 540
IR files this compiler emits for the test corpus are byte-identical to the ones
before it.
test_dev.ml's assertion that the merged daemon refuses --x86 becomes the session
it was standing in for: dev-macro.flan calls a prelude macro at the top level,
so the daemon coming up at all is the old crash not happening, and one build
then carries C-x C-e, a C-c C-c whose body calls a macro again, the park and the
rerun.
Flan's tagged sum has been spelled defunion since it landed, which was
accurate right up until the language wanted C's untagged union as well.
Both cannot be called the same thing, and the tagged one is the one with
an alternative name that says what it is: a case, its fields, and a tag
that steers which case is live is a data type, not a union.
So the form is defdata everywhere -- the parser, the AST, the checker,
both backends, the prelude's Form, the editor's font-locking and imenu,
the docs and every .flan file in the tree. The internal vocabulary moves
with it: Tast.union is Tast.data, uname is dname, the tables the checker
and the emitter keep are datas. Leaving them would have inverted the
words permanently, with surface defunion meaning one thing and
env.unions meaning the other, which is exactly the kind of drift the
comments in those files exist to prevent. What did not move is case,
variant and vfields: a tagged sum still has cases, and it still has one
live at a time.
defunion is not kept as an alias. An alias would compile the day the
untagged form lands and mean the opposite of what it used to -- the same
silent misparse that made defn's return type mandatory, and worse,
because the reader would have no reason to look. The old spelling is a
named refusal instead, parse/defunion-renamed, which says what it is now
called and that the name is reserved for something else. It fires on the
head alone, so (defunion U [A B]) -- which would otherwise have parsed
cleanly as one field A of type B -- is refused with the rest.
basis had the same false promise the eval note did. It branches on the break
state, which answers "running" while the program is parked — the agent's
listener is alive and nothing has stopped — so disassemble reported that a body
delivered while parked installs at the next frame boundary, which is not a
boundary the program will reach until somebody runs it again.
And flan-rerun had no autoload. The keymap binds it, and flan-mode.el says in
as many words why every command there needs a real autoload rather than a
declare-function: a user who loaded only flan-mode would otherwise find the key
bound to nothing at the exact moment they most want it.
You run a program under flan dev, it opens a raylib window, you close the
window, main returns — and there is no way to get another window short of
flan-dev-restart-program, which throws away the build, the session and every
global with it. In Common Lisp or Clojure the image outlives main, so you call
it again. The process here already outlived main: the exit hook flushed, closed
stdout and sat in for (;;) pause(). Nothing could wake it.
So main() is a loop. The hook records the status and longjmps back into a
setjmp in main() — there is no return available, since flan_exit is reached
from wherever the program happened to be — and the thread waits on a condition
variable until the new rerun op signals it. The main thread is the one that
runs main again: a window belongs to the thread that opened it, and on macOS to
the first thread of the process. A longjmp pops no frame, so the park first
empties the handler stack, the restart stack and the shadow frame chain, each
of which was a chain of allocas in stack the next run is about to write over.
Nothing else is reset; the second run reads whatever the first left in the
globals, which is the semantics that was asked for.
Closing stdout had to go with it. That was how the compiler learned the program
was done, but a pipe delivers EOF once, so the signal and the program's output
were the same resource and spending it left the second run with nowhere to
print. The descriptor hazard the old code reopened /dev/null for goes away with
the close that caused it. Liveness is asked for instead, through a weak symbol
in the same style as the agent's, and is now three states rather than two: Live,
Parked and Gone. Every guard branches on that before consulting the break
state, because the agent's listener answers "running" while the program is
parked and telling somebody whose program has finished that it is running is
worse than saying nothing. Only eval accepts a parked program — it queues and
waits for nothing, and the queued module installs at the first frame boundary
of the next run, so a body can be fixed while parked and the re-run executes
it. Everything else needs a frame boundary or a stopped stack, has neither, and
says which, naming the command that gets the program back.
A re-run while the program is running is refused rather than queued: the test
and the signal happen under one mutex, so two mains writing the same globals at
once never starts.
:parked rides on every reply beside :stopped, for the reason :stopped does —
finishing is as unannounced as stopping, more so when the way it happens is a
mouse click on a title bar. Emacs shows flan:parked in the modeline and binds
flan-rerun to C-c C-M-x.
flan dev's one-process build links the program and the compiler into one
-rdynamic executable, so it exports flan.rl/with-drawing -- the package's
defmacro compiled as an ordinary function, whose body was qualified at the
Ast level, after the quasiquote had already become a string literal. ELF
gives the executable precedence over a dlopened object, so the macro module
called the host's copy and (rl/with-drawing ...) expanded to an unqualified
begin-drawing the checker then refused. The module is self-contained, so
-Wl,-Bsymbolic is the whole fix.
merged_setup reports a Loc.Error as a diagnostic now, which is how the
failure had a location at all instead of an exception constructor after an
apparently successful build.
Item 3, and the reason the backend was written. Until now --x86 was read only
by flan build's argument list; the daemon built both halves through LLVM, so
none of this reached the dev loop at all.
The choice is a session setting, not a per-command flag, and it is spelled
exactly as [debug] already is -- one field on Session.t, set once in Dev.start,
carried on every change the session emits. session.ml's comment on [debug]
already gives the reason and it is the same one: the modules have to match the
process they are loaded into. Session.redefinition is the single place that
picks a backend, so the six call sites cannot disagree and the refusal has one
home. Session.change carries the answer beside the text, so the builder and the
text can never come from two different decisions.
There is no fallback and there must not be one. X86.redefinition refusing a form
is reported to the editor; quietly building an LLVM module instead is precisely
the crossed pair flan.abi.x86 exists to refuse at dlopen. A refusal reaches the
editor as a diagnostic like any other -- X86.Unsupported is re-raised as a
Loc.Error at the form it is about, because every caller already handles that and
none handled the other, and a session that died on the first unsupported form
would be worse than one that says so and stays up.
flan reload got the same flag at the same time. A command that could build a
module for a host the other backend compiled is how the crossed pair was
reachable from the CLI at all; the aggregate handoff's two-line reproduction no
longer has a second half.
And the finding: flan dev --x86 refuses the merged daemon. A merged build is the
program and the compiler in one process, and the compiler expands macros by
dlopening a module Build.macro_module made through Emit.program, cached on disk
by the macro source rather than by the backend. The merged host is linked
-rdynamic so a redefinition module can reach its cells, which also exports every
flan.* body it has -- so the macro module's own copy of a prelude function is
interposed by the host's. With an LLVM host nobody notices. With an --x86 host
the caller is LLVM and the body it lands in is this backend's, and the process
dies inside flan.[clamp] during the first macro expansion, before the program
has started. flan.abi.x86 does not catch it and was never meant to: a macro
module deliberately neither defines nor requires a marker. The honest fix is
hidden visibility on a macro module's Flan bodies, which changes the cached
object for both backends and wants a lane of its own. Until then the refusal
names the mechanism and the remedy, and --two-process has no such meeting.
start_merged keeps its --x86 plumbing, unreachable for now, because it is the
half that is right and will be wanted the day the macro module is fixed.
test_dev.ml drives an --x86 daemon through C-c C-c, C-x C-e, a literal, a new
defvar with a value of its own and a new defn, and asserts (twice fresh) is 82 --
which only holds if both registry lookups resolved. The merged refusal is
asserted there too. bin/main.ml learned to print a bare Failure as a sentence
rather than an uncaught exception and its backtrace.
The merged daemon now unlinks its socket on the ways out it does not
control as well as the one it does: an atexit for exit(3), which is what a
runtime trap takes, and by hand in die_now and in main's fallback, which
are _exit and skip the chain on purpose.
The client says so too. A refusal on a path that exists is a leftover, not
a daemon declining, and the raw 'Connection refused' has now misdirected
two investigations.
Separately, and it is separate: dev-repl.flan gets dev-robust.flan's
24000-tick budget. Twenty seconds of program under a two-minute test is a
second flake waiting its turn, and it is not the one fixed above -- that
one fails honestly, saying the program exited.
SIGPIPE's default action was killing the daemon outright. In the merged
build that is the program, the compiler and the listener at once, and the
socket file is left behind - so the next client got ECONNREFUSED on a path
that exists, which reads like anything but what it is.
Two halves: ignore the signal where the listening socket is bound, and
guard the reply write, because an [exception] case on a [match] covers the
scrutinee and not the branch body, so the EPIPE went past both handlers
below it and out of the accept loop.
test_emacs.ml keeps the daemon's wait status and the program's output on a
failure now, instead of removing the evidence before reading the exit code.
C-c C-m. One step on the bare key, the fixpoint under C-u: a macro may
quasiquote a call to another macro, and Loc.from_macro is outermost-wins, so
by the time a full expansion settles the intermediate name is gone. One step
is the only thing that can say which macro produced what.
The expansion runs against the macros the *session* holds -- the prelude's,
its imports', and every defmacro evaluated since it started -- and writes
nothing back: a defmacro handed to C-c C-m does not join the session by having
been looked at.
Both non-termination refusals stay refusals, and only where they are needed.
One step makes one call and does not look at the answer, so (s/spin) one-
stepped answers with itself; all the way hits the fuel and names the macro,
inside Dev.serve's guard, so the daemon replies rather than hanging. Macro's
module handling is a Fun.protect now -- a build that raised was a process
about to exit, and the daemon is not that process.
No printer for a Form existed. Form.to_string is an error-message renderer and
is what Macro.key digests, so it is untouched; Form.to_source round-trips
floats, strings and bytes through the reader, and Form.pretty decides where
the line breaks go and leaves the columns to flan-mode.
The answer is a read-only flan-mode buffer shaped like the disassembly one,
with cnr's idea in it: m expands the form at point one more step in place.
Three inherited keys refuse by name -- an expansion is in no file. The text is
sent padded onto its own line and its own column, unlike C-x C-e, so the
refusal lands on the call and not at the start of its line.
The daemon caught Loc.Error at each op and nothing else. That was survivable
while the frontend was the only thing that could refuse a form; it is not now
that expansion is part of evaluating. Both C-c C-c and C-x C-e run a clang
driver through Build.macro_module, which answers with an exit status and a
Failure, and a dlopen that finds no symbol answers with another one. Neither
is a Loc.Error, so neither was answered, and an exception past serve is not a
refused evaluation — it is a dead daemon with the program still on screen and
a closed socket waiting for the editor's next request.
The boundary is now one place, around the whole of a request, rather than a
new arm at each of the dozens of calls. Out_of_memory, Stack_overflow and
Sys.Break go through it: those say the process cannot continue, and answering
"error" to them would claim a session survived something it did not.
Everything else is about the form that was sent, and the message it carries
is the one the user can act on, so a clang exit status reaches :message
instead of being flattened to "internal error".
The session's own state goes with it. Session.eval wrote the imported macro
set above the checker, so a form that did not check left the session holding
a package's macros and none of its declarations; it is held and committed at
the bottom with decls, program and env. Session.eval_expr committed the
generic copies it had instantiated before emitting the module that carries
them, which is the session believing it holds a body nothing was written for;
that assignment moved below Emit.
Both are pinned. test_session drives the two rollbacks in process, and
test_dev drives a real daemon whose macro module cannot be built — the
expression path and the redefinition path, each followed by the same
evaluation succeeding and by the session still knowing the program.
The allocation registry had a recording side and half a reader. This is the
rest of the reader: point at any heap address, a breakdown by type, what is
still held, and the test that stops dev-ptr.flan's header from being read by
hand.
The recorded name, back to a type. The table records a string and has to —
the note is built where the concrete type exists and what crosses into the
runtime is bytes. What closes it is that the string is Types.to_string, which
is the source spelling, so the round trip is the language's own reader,
Parse.texpr and Check.resolve. No table of spellings is written down, so
nothing can fall behind Types.to_string, and a name that is not a type —
"pool slots" — is refused with the name quoted rather than defaulted.
The address root renders a (Ptr T) and not the pointee, which puts it through
render.ml's pointer arm: permission is asked in one place in the compiler, and
an address root and a slot root reach the same two answers by the same code.
Flan has no integer-to-pointer cast, so flan_dev_reg_addr is an extern beside
flan_agent_frame_slot, for the same reason.
One walk and two questions: a leak report is a breakdown with the dead left
out, so flan_dev_reg_by_type is one function and the agent formats it.
"At exit" is not a hook. A program killed by a signal runs no handler, which
is how a game under the editor ends, so (:op "leaks") is the authoritative
reader and can be asked at any moment including the one before the kill. The
atexit hook is for the program that returns from main, is registered from
inside flan_dev_reg_enable rather than by a file-scope destructor so that a
release build does not grow a third not-free place, and is off unless
FLAN_DEV_LEAKS is set because the acceptance table reads stderr.
The memcheck half of item 6 is deliberately not here.
Build.cachedir sat under TMPDIR, which dune makes private per run, so no
test run ever reused an object and every build in the suite was cold. It
moves to $XDG_CACHE_HOME/flan/objcache (FLAN_CACHE_DIR overrides), which
is safe because the keys are total: compile_c digests the source text,
the compiler's stamp and every flag; wasm_resource_dir digests the
builtins archive; compiler_object digests flan.cmxa and flan.a. Writes
were already .tmp-then-rename, so concurrent dune jobs are fine.
Macro.key was the one key that was not total -- prelude text plus the
call's forms, and nothing about the compiler whose codegen produced the
.so it names, which is dlopened straight back into this binary. Under a
per-run TMPDIR that never showed; under a durable cache it is a stale
expander that crashes rather than a compile error. It carries the
compiler's stamp now, handed across start_merged's exec in
FLAN_COMPILER_STAMP because a merged dev binary lives at a per-session
path and keying on that rebuilt a macro module every dev start.
Measured on dev-repl.flan, launch to bound socket: 2.0s cold against
0.48s warm. Whole-program flan build: 1.44s against 0.06s. Full dune
test 25.7s/30.1s before, 24.0s after, user CPU ~50s down to ~34s.
And the await: one timer covered two waits, a build then a bind, so
'the daemon never listened' was a wrong diagnosis of a build that had
not finished. listening now polls the process alongside the socket and
says which -- exited with a status, or still running and therefore still
building. A daemon that dies fails in milliseconds instead of costing
the whole timeout. Thirty seconds, down from a minute, because the build
it waits on is warm now.
Two follow-ups to the marking commit.
`Dev.eval_expr`'s new wait matched `Stopped _`, which fires on the first
iteration when the program is already parked on something else — the
break loop allows evaluating, so that is reachable — and answers for a
thunk that has not run yet, on a reply whose own `:condition` names the
other condition. It now waits for `Stopped "Pause"`, which the agent
reports under a nested break because `condition_name` is overwritten on
the way in and restored on the way out. `dev-pause.flan` grows a
`Missing` and a `boom` so the test can park the program on something
else first and tell the two apart.
And the flake NEXT.md had as "seen once and unexplained": `the daemon
never listened` is not a race, it is an llc-and-link of the whole
program before `flan dev` binds — ~600ms idle, measured at 6.6s and 6.8s
with the rest of the suite beside it, against a 5s and 8s await. All
three test binaries now wait a minute; the watchdog is what bounds the
run. Two consecutive full runs green.
Finishes DISCUSS.md §9's `pause` marking: the daemon half was already
built, this is the editor half plus the one daemon path it was missing.
`C-u C-c C-c` marks the form point is inside, `C-u C-u C-c C-c` the
top-level form (stop on entry), `C-u C-x C-e` the expression before
point. The buffer is never edited — the position rides beside the code
and the `(pause)` call goes into the tree after parsing, so no source
location moves.
`C-x C-e`'s path needed the daemon: its 5s `wait` answered "the program
did not reach a frame boundary", which is exactly what a thunk parked at
a breakpoint looks like from out here. `wait` is now three-way and asks
`state t = Stopped` only when a pause was requested, so the no-pause
shape `test_dev.ml` pins is unchanged.
The overlay is an annotation and not feedback, so unlike an error marker
it survives `pre-command-hook`; what takes it down is an accepted
evaluation with no `:pause` on it, which is the same thing that takes
the mark itself down.
Tests: a `test_dev.ml` block over the new `dev-pause.flan` that marks,
stops, re-evaluates plainly and then polls half a second confirming it
does not stop again — one sample after `continue` proves nothing, the
resumed frame is still in the old body — and an `emacs/test-flan-dev.el`
block for which form a prefix picks, the byte column, the overlay's
lifetime, and one live round trip.
PORTING.md Tier 1 item 5. The spy half of the watch was already built — the
pushed table, the buffer, the inline ghost text. What was missing is spy-num,
which is the part that item calls least obvious and most valuable, and it is
what this is.
A slot keeps count, min, max, last and mean. Each answers a question you can
ask without building a query: n is the first thing wrong when a loop is wrong,
the range is what one sample can never show you, last is what the scalar watch
would have given you, and the mean is a running sum divided at read time
because a mean accumulated as a mean drifts. A small ring of the last N
samples was the other candidate and loses — N out of 91,200 is a sample of the
tail of the loop rather than of the loop, and past five numbers every richer
answer is a UI for building a query.
The write path does no formatting, which is the feature rather than an
optimisation: a snprintf per sample at thousands a frame is a HUD that costs
more than the game. A sample is a load, five compares and the slot's seqlock;
the listener thread renders once per editor tick.
The window is since the editor's last tick, and that is a deliberate
divergence from watch.clj, where the stats are cumulative until reset-spies!.
Cumulative min and max reach the session's extremes within seconds of play and
then never move again, so the two most useful of the five go dead exactly when
you start interacting with the thing you are debugging — and this tool exists
to show you a number while you drag the mouse. Reset is its own message and
never a side effect of reading, because a destructive read makes looking
change what is there and anything that polls would shorten the window under
the editor that owns it. It bumps one epoch counter and clears no slot; a slot
clears itself on its next sample, so the reader never writes the table.
Ghost text needed one character. The call regexp allowed one hyphenated
segment, so watch-num-i64 backtracked to failure and a numeric watch got no
inline value while appearing normally in the buffer.
dune test is green, run twice. HANDOFF-f3.md carries the reasoning, the two
small gaps left behind it, and what did not work on the way.
C-u before an eval marks a form so the program stops when it runs
(DISCUSS.md 9). The mark arrives as a position in a separate :pause field
and is applied to the Ast after parsing: splicing text into the source
would move every line and column after it, and the error overlays, the
layout, the break loop's frame locations and DWARF all read those.
Ast.mark_pause puts a (pause) call at whatever starts at that position --
wrapping a sub-expression in a do, or going to the front of a defn's body,
since a declaration cannot be wrapped. A position that matches nothing is
refused rather than installed unmarked, which would report a breakpoint
that is not there.
It sticks with no extra state: the marked declaration is what goes into
the session, so an ordinary C-c C-c over the same form clears it.
The daemon half only; the Emacs command and its overlay are not built.
HANDOFF-f2.md has the rest, in order.
Loc.Error now carries a diagnostic: a stable kind, a span, notes that each
have their own span and severity, and the macro expansion it came from. The
notes are the part that was actually missing — "this is wrong here" plus
"because of that, over there" is two places and two explanations, and a
single string can state only one of them.
The compatibility story for the daemon, which was the open question: the
single-diagnostic exception stays the single-diagnostic exception. Session
and dev evaluate one form and have one failure to report, so they take a
location and a message out of it with Loc.summary and are otherwise
unchanged. A second exception carries a list, and only a driver that
compiles a whole file raises it, so nothing interactive has to know it is
there.
No message text changed.
Ported from the author's Clojure watch.el. Three of its decisions survive
unchanged — the program decides what is shown, the request is async, and the
paint is replace-buffer-contents so point survives every tick. The transport
is the part that had to turn round.
An eval here compiles a module and dlopens it, so the planned fix of
compiling the render thunk once and re-invoking it per tick was still a poll,
and a poll has a defect caching does not touch: a thunk runs at a frame
boundary and a stopped program has no more of those. So the program writes
into a table in flan_dev.c from its own loop and Emacs reads the table, which
is memory. The values are then as fresh as the last frame whatever the
repaint interval is, and they are still there while the program is stopped.
The frame thread's constraints decide the storage: no allocation, so names
are fixed arrays rather than strdup'd; no lock, because the reader is the
listener thread; and not the result buffer, which is written once per C-x C-e
and would be overwritten sixty times a second by watch traffic. One seqlock
per slot rather than one for the table, so a reader retries one slot instead
of having to catch the gap between two frames' writes; a snapshot from
adjacent frames is what a HUD looks like anyway.
Sixty-four slots, and past that a name is dropped rather than fatal — killing
the program because somebody watched a 65th value is the diagnostic shooting
the patient. Reported as a flag and not a count: the only number the write
path could keep is of write attempts, which at frame rate says "3847 names"
about one name.
Nothing writes the table until a watch buffer is open, so a watch call in a
program nobody is debugging is a load and a branch that is not taken — the
same number in a release build, since flan_dev.c is linked into both.
Scalars work today through declare-c against four runtime entry points, which
is why this needed no compiler change. A struct or a slice needs a walk over
its type, which is one arm in check.ml beside print; BUILT.md writes it out
rather than reaching into a file another lane holds. Ghost text turns out to
be gated on the same arm, for a different reason: nothing in the table
carries a source location, and a hand-written declare-c call cannot supply
one that does not drift when the line moves.
flan dev is one process, so the unix socket between the compiler and the
program was a connect, a write and a read that never left the machine's own
memory. A delivery is now a call into the agent's verb table.
flan_agent.c's handlers write into a sink that is either an fd or a growing
buffer, so serve() and the new flan_agent_request() share one verb table rather
than two that would drift -- every answer in there is assembled from several
pieces, so the seam had to be the writing. Dev.deliver, Dev.result and Dev.ask
were three copies of the same socket dance and are now three lines over one
Dev.request.
Which path is taken is the linker's answer, not a flag: flan_agent_request is
weak in dynload_stubs.c, null in every binary that links no flan_agent.o, and
Agent.request is None there. So --two-process, flan reload and every test go on
using the socket with nothing to configure.
Three things the direct path must not quietly change. The ring's room check is
separate from its store, which was safe only while the accept loop was the sole
producer, so handle_line runs under a mutex. A delivery is still only published
-- the install is one store per function on the game thread at a frame
boundary, and doing it on the spot would be a frame running half in the old
code and half in the new. And the OCaml runtime system is released across the
call, because a delivery is a dlopen.
Measured, and the number is the point: the transport was ~50us of a 23ms
redefinition, so end to end did not move. Code generation is 19 of the 22
milliseconds. The merge's prize was never latency.
The test is a deletion, because a reply cannot say which way it came:
test_dev.ml unlinks the agent's socket file once the merged program has bound
it, and every evaluation after that still installs.
The merged exit closes stdout so the compiler reads EOF and learns the program
has finished. POSIX then hands descriptor 1 to the next thing that asks — a
socket, a module's object file — and the llc after that inherits it as its
stdout. /dev/null takes the slot back, and the EOF is unaffected because the
pipe's write end is genuinely gone.
Verified beyond the headless suite: sand.flan builds and runs merged under
Xvfb, one process with no children, and a game-draw typed in at the socket is
drawing frames a second later.
`flan dev` now builds one binary that is the compiled Flan program and holds
the whole OCaml compiler, and execs it. The program keeps main() — macOS needs
the window there — and caml_startup happens on a pthread beside it, next to the
listener flan_agent.c already starts. The editor's socket and the wire protocol
are untouched: Emacs cannot tell the difference.
Two rules are written into lib/dev.ml rather than discovered later. The game
thread must never call into OCaml, because a native thread has no safe points
and so can never be stopped by the collector — which is exactly why a frame is
never paused, and exactly what one convenient direct call would undo. And no
OCaml value may be stored in Flan memory without caml_register_global_root,
which is the way the spike's "the GC does not touch the arenas" measurement
stops being true.
The link is spelled in dev.ml out of Build's existing public pieces rather than
as a mode of Build.executable: lib/build.ml belongs to another lane this week.
It should collapse into Build once that lands.
A Flan main does not return — Emit ends it with flan_exit and an unreachable —
so in one process that call would take the compiler down with a program that
merely finished. flan_rt.c grows a hook, null in every other build, that the
merged entry point uses to flush, close stdout and park. The compiler then
learns the program is done the same way the daemon did: the pipe reads EOF.
--two-process keeps the old shape for a machine that cannot build the compiler
object, and nothing has been deleted.
The capability lists were written before the code held the line they claim.
Under an expression root, RET on a field of a union built `(.at s)' and sent
it, and the checker refused it — "a union's fields belong to a case ... they
are reached by (match ...)". A refusal from the far end of a socket is exactly
what this buffer's own comment says not to do: every refusal is by name, here,
with the reason, because RET working on some lines and erroring on others
teaches nothing about the language.
It is a refusal of the *parent* and not of the value at point, which is why it
is not in `flan-inspect-refusal': a struct field that merely holds a union is
an ordinary accessor and has to stay enterable. It is a field of the union
itself that cannot be written. The two cases are one test each.
The slot root steps into it by offset and is unaffected, which is the
difference the manual now claims and the tests now show.
`lib/dev.ml' cited DISCUSS.md item 1 as a hole; item 1 is the answer now, so
it cites BUILT.md instead. And the item 1 stub is two sentences and a pointer
— everything else in it is in BUILT.md verbatim, and DISCUSS.md's own header
says nothing in it is a decision.
`i' on a local sent the local's *name* to be evaluated, and an expression is
evaluated where the evaluator stands. That is the right frame only when the
frame is the innermost one; on any other it may resolve to a global, to
another binding of the same name, or to nothing, with the locals listing right
above it showing the frame's own storage and nothing saying the two disagree.
The daemon verb for the fix landed already. What was missing was the state
layer under it: `flan-inspect--expr' held a bare expression, so there was
nowhere to put a frame. It is `flan-inspect--root' and `flan-inspect--path'
now — `(:expr E)' or `(:slot FRAME SLOT NAME)', plus the steps walked from it
— and a stack entry is `(ROOT PATH . POINT)'. RET appends a step, `l' restores
a pair it pushed. Every step is still a fresh request, so the view is never
stale.
`l' cannot cross between the two roots, and that is structural rather than a
rule someone has to keep: RET only ever extends the path under the root the
buffer already has, and `flan-inspect' and `flan-inspect-slot' both start with
an empty stack, so a mixed stack cannot be built at all. It stays true if a
third rooting mode is added.
The break buffer hands over the frame and the slot *index*, which is the
fourth element `locals' now puts on each line. A name does not identify a
slot: two slots of one frame can share one, and a refused slot is not in the
listing, so its position is not an identifier either. A global still goes in
by name, because a global's name really is an expression that means the same
thing wherever it is evaluated — the loaded thunk binds to the program's own
storage through the dynamic linker.
Two smaller things the wire needed. A field step carries the type it was read
out of, because a union's payload is at an offset that depends on the case and
only the renderer knows which case the value is in — so `Union.case.field',
which is the head the renderer wrote with the field appended. And an empty
path is sent by omission: Emacs prints an empty list as `nil', which is a
symbol on the wire, so the daemon now reads that as no path rather than
refusing it as a step.
`i' in the break buffer sent a local's *name* to be evaluated, and an
expression is evaluated where the evaluator stands. On the innermost frame
that is the right frame; on any other it may resolve to a global, to another
binding of the same name, or to nothing, with the listing above it showing the
frame's own storage and nothing saying the two disagree.
The shadow stack is what makes the second rooting mode cheap: a frame's
address and every slot's type are both here, so a step into a field is an
address plus an offset with that field's type — the arithmetic
`Render.render' already does for the listing. `Session.render_slot' is
`render_locals' with a path applied to the root and one line out.
The slot travels by *index*, because a name is not unique: two `v's is two
slots and both are in the listing, and a refused slot is not, so the position
in the list is not an identifier either. So `locals' now puts the index on
each line.
The frame checks are `locals'' by construction — `stopped_frame' is one
function now, and an inspector with its own copy would be free to read a frame
whose body was redefined since it was entered. The build-and-read tail is one
function too, for the reason this file already records about the fingerprint.
And `layout' said union values were milestone 6, which they have not been
since today.
The globals section attributed a frame by its slot fingerprint, which is the
wrong cut for it: a redefined body can name entirely different globals while
binding identical locals, so the check saw no change and the new body's
reference set went into the union under the old body's frame, with the frame
numbers beside an entry saying so.
So a second fingerprint. Reach.ref_fingerprint hashes the set of globals a body
names — sorted and deduplicated, because a reference set is not ordered, where
slot indices make the slot fingerprint order-sensitive on purpose — and it
travels the path the first one already cut: %fninfo, flan_dev_frame_refsig, the
agent's snapshot, the backtrace line, Dev.globals_op. Different means the frame
is skipped by name with its reason, and the rest of the stack still contributes.
Two numbers rather than one, because they are two facts. A frame whose slots
match and whose globals do not has locals that are perfectly readable and
attribution that is not, and a combined hash would make locals refuse a frame
with nothing wrong with it. locals still checks the slot fingerprint alone.
It lives in reach.ml because expr_refs is already the walk that answers what a
body refers to, and is the walk the union itself is built from. One consequence:
emit now reaches reach, which closes a cycle through Load if cimport calls
Build.cachedir, so the header cache spells the object cache directory itself.
test_dev.ml drives the exact case — a body that binds identical locals and names
untouched where the stopped frame names pressure. With the check disabled it
fails twice: the missing refusal, and untouched appearing under frame 0.
A global is program state a frame happened to touch, not part of it, so
nesting it under one implies an ownership that is not there and repeats the
name once per frame that reads it. One section instead, holding the union of
the globals every frame on the stack references — the compiler does the
choosing, since Reach.expr_refs already answers a body's reference set, and
listing every global a program has would bury the one that matters under the
prelude's PRNG state.
Each entry says which frames touch it, by the index the stack section already
numbers them with, which recovers what per-frame nesting would have told you
at no cost in duplication. Ordered by the innermost frame that touches it:
a deep stack makes the union large and proximity to the error is what puts
the likely culprit on top.
Simpler than locals, because a global is reached by name rather than by
address. Emit.redefinition writes a global the host has as external, so the
thunk binds to the program's own storage and nothing is asked of the stopped
thread — no dev-slot round trip and no not-yet-bound case to refuse.
A frame that cannot be attributed contributes nothing and is named in
:skipped; the union being incomplete and the union being complete are
different answers. The hole in that is stated rather than papered over:
slot_fingerprint hashes a body's slots, which is the right cut for locals and
not for this, so a body that names different globals while binding the same
locals is not caught. The test drives the case that is.
MANUAL.md also loses a stale paragraph claiming the fingerprint check never
fires with a failing test pinned to it. It fires, and test_dev covers it.
"every slot in it is one the compiler made up" is a claim about the body this
session holds, not about the frame, and it was answered before either body
check ran — so a zero-slot frame whose body had since been replaced by one with
slots got that note instead of the refusal. No values were misattributed, which
is why it is not the defect just fixed, but the reason given was untrue. The
count and fingerprint checks now run first and the note is the last arm.
The refusal for a frame whose body has been redefined underneath it did not
fire because four of its five hand-offs were never written. `Emit.fninfo` has
been storing `slot_fingerprint` in the last `i32` of every `%fninfo` all along;
`flan_dev.c` called that field `spare`, there was no accessor for it, the agent
never snapshotted it, the backtrace line never carried it, and `Dev.locals`
compared slot counts and nothing else. The handoff note's "every piece is
written and the refusal does not happen" was a guess, and the first step it
suggested — printing both sides of the comparison — could not have found it,
because there was no comparison.
So: `spare` becomes `slotsig` and gets `flan_dev_frame_slotsig`; the agent
snapshots it beside the slot count and puts it on the backtrace line *before*
the location, since the name is the one field that can contain a space and has
to stay last; `Dev.backtrace` parses it; `Dev.locals` compares it against
`Emit.slot_fingerprint` of the body this session holds and refuses by name when
they differ. No change to `emit.ml` — the value was already there.
The mechanism itself is right and stays. `slot_fingerprint` hashes every slot's
name together with the spelling of its type, so a rename that keeps the count
and the types — exactly the case this exists for — changes it. The count check
stays in front of it because its message is the more specific one.
The fingerprint stays off the wire. A hash is not something an editor can act
on, and the refusal says the fact in words: this frame's body was redefined
since it was entered, so its names no longer describe its values.
`test_dev.ml` gains the inverse and the control. A body that drops a `let` is
refused on the count, and `main` — untouched by the redefinition of `look` —
must still answer, which is the assertion that would catch a fingerprint that
never matched anything and made the verb useless while turning the suite green.
locals compares the frame on the stack against the body the session holds:
installing while stopped is allowed, so the two can be different bodies of
one function, and a rename that keeps the slot count pairs every name with
the wrong value. Emit.slot_fingerprint hashes each slot's name and type,
emit_fn puts it in the frame's static description, the agent reports it on
the backtrace line and Dev.locals compares it.
It does not fire. The test that drives it -- a redefinition that renames
every local of a function that is on the stack -- fails, and is committed
failing rather than deleted, because it is the only record of what is
wrong. Everything else in the suite is green; this one check is red.
It builds. See NEXT.md's handoff for where to look first.
The half the shadow stack was built for. A slot's entry in the frame is its
address, null until the binding that fills it has run, so "not bound yet at
this point" is a null and needs no liveness analysis. The daemon compiles a
thunk that renders the types it already knows -- Tast.fn.slots, with snames
beside them -- at the addresses the stopped program supplies, and reads the
text back the way C-x C-e does. Nothing is copied out, because a value with
no header is bytes with no meaning anywhere but in the program that holds
it.
That is render.ml's walk with its root changed, which is the pointer-rooted
thunk NEXT.md said this needed, and one new arm in the backend: a cast from
one pointer type to another, which emits nothing.
Only named slots are recorded. A recorded slot escapes and stops being
promotable, and the slots that would cost most are the ones with nothing to
show -- dotimes' bound, the temporaries min and max use, the walk's own
scratch. They are refused by name rather than shown under an invented one.
Recording every slot was built and timed and is inside the noise, so the
rule stands on what it shows.
Four refusals, each by name and with its reason: a slot nobody named, a
slot the program has not reached, a type the printer has no arm for, and
two whole frames -- an evaluation's thunk, and a frame running a body that
has been redefined since, where every slot index would be a guess.
Measured, minimum of nine runs: +61% on call-heavy code over globals
against +33% for the frames alone, 0.06% of a frame at 60fps.
