vendor/edn rather than the prelude: the prelude is prepended to every program
and everything in it is emitted, so a reader nobody imports would be a cost
every build pays.
The tokenizer only. A type-directed reader - the compiler emitting a parser
from a walk over a struct's fields, the dual of the printer C-x C-e already has
- lands in check.ml and emit.ml and is not this. What a caller writes today is
a struct reader by hand against the cursor, and the acceptance program carries
one, because that is what proves the API is usable rather than present.
Every token is a slice into the source, so nothing allocates and the buffer has
to outlive the tokens. That contract is stated at the top of the package,
because it is the kind of thing found the hard way.
Escaped strings are refused rather than half-supported: unescaping needs a copy
and there is nowhere to put one, and handing back the raw bytes would return a
three-byte string as four with a backslash in it. Each other refusal carries its
own sentence - #inst and #uuid separately from tagged literals, because a file
is most likely to contain those two and being told tagged literals are refused
would not say that the timestamp is the thing to delete.
Errors live on the cursor, a code and a byte offset, not in the return type: an
Option loses the position, which is the whole point for an editor. A failed
cursor is poisoned so a caller's loop terminates on a malformed file rather than
spinning.
# Conflicts:
# test/test_acceptance.ml
An import reads the directory at build time, so a package that dune has not
copied under the test's build dir does not resolve — and the failure is a
missing collection, not a missing file, which reads like a bug in Load.
The wasi-sdk candidate had an LLVM version in it, which moves release to
release — so the path advertised as the proper article would have matched only
by coincidence, while the emscripten one beside it was derived. Both are
derived now.
calc-me on wasm32 covers what the other three cases cannot: flan_argv hands
Flan an array of flan_slice built in C, so what it pins is the element stride
of a ptr+len pair — 16 bytes native, 12 on wasm32 — rather than a field
offset. It is also the claim in this file's own header, that the table runs on
the second target, honoured for the first time.
flan emit refuses --target rather than stripping it. The IR really is
target-free, so ignoring it is correct and silence about it is not.
sand-headless imports no raylib so that it can run here, and the point of the
case is not that a module exists — it is that the number matches native byte
for byte, which is only possible because rand-f32 is Flan's rather than libc's.
It does, at -O2 and at -O0; -O0 is the cheap way to say the agreement is not a
coincidence of how LLVM folded the float arithmetic. values and machine run
there too, which is where a 32-bit pointer would have shown.
Four separate things can be missing — clang's wasm target, the sysroot, the
builtins, a WASI runtime — so the skip is a probe rather than a lookup: build
the smallest program and run it, and print what went wrong. A which(1) would go
red on the machine where Node is too old, with a reason nobody could read.
No wasmtime and no wasmer here, so the runner is node:wasi, with wasmtime and
wasmer preferred if either appears. --no-warnings because node:wasi writes to
stderr on every run and this harness compares combined output.
A different primitive from redefining a name. There is no name to install a
body into, so the expression is wrapped in a function with nowhere to be called
from; the module exports flan_reload_call to say "run this once", and the agent
calls it after the install - on the game thread, at a frame boundary, so an
expression that reads the program's state sees a point the program agrees is
consistent.
Nothing is marshalled back because nothing could be. A Flan value carries no
header, so no code at run time can say what it is; the compiler knows the type
and renders it there, in the thunk. That is the layout decision's bill, and it
is why the printer set is the scalars rather than everything.
The rendering does not go through stdout. Stdout belongs to the program, it is
in the hot path for anything that prints, and a dev-only feature must not put a
branch in it - so flan_rt.c is untouched and the value goes to flan_dev_result,
read back over the agent's socket. Safe without a handshake because the
generation counter is bumped last: the daemon waits for it to move rather than
assuming the program has reached a frame boundary.
u64 refuses by name, because i64->bytes is signed and anything past 2^63 would
come back negative. Everything without a derived printer refuses the same way.
A number that is quietly wrong is the failure this whole thing exists to
prevent.
An evaluation is not a declaration: the thunk is built against the program and
never spliced into it, so describe does not fill up with an eval/N for every
expression ever typed.
The test that matters is the same expression twice. The fixture increments
ticks every frame, so two evaluations must disagree - a value computed in the
compiler, or read from a copy of the program's state, would not.
C-c C-c recompiles the top-level form at point and installs it in a running
program at that program's next frame boundary. Verified against sand: an
unsaved buffer edit to game-draw, and 240 consecutive frames drew it.
flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is
most of the work - Flan is s-expressions, so sexp motion, paren matching,
beginning-of-defun and indentation are already right. What it adds is Flan's
own brackets ([ and { are brackets and not symbol characters, since every
binding list and every type is written with them), the characters a name may
contain, and its keywords.
flan-dev.el has no parser in it, which is what the protocol choice bought:
prin1 writes a request, read reads a reply. C-c C-k sends a buffer as one
module rather than a form at a time, because a defvar and the function using it
have to arrive in the same load or the first refers to storage that does not
exist yet. An error comes back with a location and point moves there.
Framing is in bytes and Emacs counts characters, so every length goes through
string-bytes and the process is binary. Otherwise one non-ASCII character in a
buffer puts the reply stream out of step by exactly as many bytes as the
payload has of them - a bug that reads as a corrupt protocol and only appears
for some people. test_emacs.ml drives the real client against a real daemon for
that reason: it is not the same claim as the daemon answering correctly, and a
mistake in the framing, in beginning-of-defun over Flan's syntax table, or in
the reply reader passes test_dev.ml and fails here.
The piece between an editor and everything else. One long-lived Session, the
program it belongs to launched and owned by the same process, and a socket that
takes forms and installs them. What it adds over flan reload is that the
session persists - a defvar added by one evaluation is part of what the next is
checked against - and that it owns the build, which is what makes its layout
rules describe the process actually running rather than a guess about it.
The protocol is s-expressions rather than bencode, and I changed my mind about
that. The case for nREPL was reusing a designed op set and not re-litigating
session identity, but with the client ours too there is no CIDER to be
compatible with, its eval is string-in/string-out with no slot for which form
from which file, and Emacs already has read and prin1. So: one sexp per
message, length framed because the payload contains newlines. No parsing code
on the editor side, and on this side the parser is the language's own reader,
where :op is already a keyword and Flan source is already a string literal. An
nREPL front end can sit on the same Session later; it should not gate the
editor.
Two silent failures the daemon refuses to have. The agent socket is chosen by
the daemon and forced through FLAN_AGENT_SOCKET before spawning, because a
program's source has to name some path and a daemon that guessed would compile,
build and deliver a module to nobody. And delivery is checked: agent/start
returning 0 means a socket was bound, not that anyone connected, so a failed
connect or a reply that is not ok becomes an error the editor sees.
It waits for the program to bind before accepting an evaluation, since one
arriving first fails for a reason that reads like a compiler bug, and it
accepts with a timeout so a program that has exited takes the daemon with it
instead of leaving an editor waiting on a socket nobody serves.
lib/session.ml holds the declarations a running process was built from plus
every change accepted since, which is what an editor needs and what a one-shot
compiler cannot have.
Transactionality came for free. Check.program builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing
and the accumulated list is simply not replaced - no scratch-environment
machinery, which is what I was about to build. Re-checking the whole program
each evaluation costs the frontend, under 10ms, less than the llc after it.
There is a test for the case that matters: a typo, then a good form, in the
same session.
Which names the process was built with comes from the checked program, not from
any accumulated AST, because Check.program prepends the prelude and no AST
contains it. Derive it from declarations and print-line reads as new, gets a
registry cell nobody publishes, and the first call jumps to null.
Three changes are refused with a reason rather than loaded. A function's
signature, because a cell is a bare ptr and every call site compiled before the
change still passes the old arguments through it. A global's type, because the
storage exists and has a shape - reusing it reads at the wrong offsets, and
replacing it discards the state the reload exists to preserve. A struct's
fields, because the values the process is holding have the old layout. Note
what the checker already catches on its own: change a parameter type and the
caller fails to type check first, loudly. These rules only get a turn on a
change the checker accepts, which is a name nothing else in the program uses -
exactly where the silent version lives. Hence an unused defvar and a C-called
defn in the fixtures.
The accumulated list is the post-Load one, so an evaluated import is spliced as
its expansion. Otherwise re-evaluating a file that imports something appends a
second import, Load expands it again, and the duplicate-name pass rejects it.
C-c C-k on sand.flan's own text is the test.
flan reload now takes a program and a file of changed forms rather than a list
of function names and a --new list: the session works out which names are new,
which is the thing a bare CLI could not.
Also fixed, found by running the agent test under load: the agent took SIGPIPE
when a sender read part of a reply and closed. Replies go out with
MSG_NOSIGNAL, per call rather than by installing a handler, because the signal
disposition belongs to the program the agent is embedded in.
vendor/agent/ is a package like any other - agent.flan declares three calls,
flan_agent.c implements them, link asks for -lpthread. start listens on a unix
socket, poll installs whatever arrived and says how many, wait does the same
after waiting for something.
The split between poll and the listener is the whole design. dlopen relocates a
module and takes the loader lock, which is milliseconds and unbounded, so it
happens on the listener thread. flan_reload_install is one store per function
and must not land while a redefined function is on the stack, so it happens on
the game thread at the top of the frame, when the program asks. A ring and two
atomics connect them; the game thread never blocks on the loader.
wait exists for tests. A test that races the frame rate fails on a loaded
machine, so test/programs/agent.flan waits for the reload rather than sleeping
past it. It also sends a junk path first: the daemon is a separate process and
can send anything, and a bad path must be refused rather than take down the
program it was sent to.
Two things came out of running it. The reply goes out before the module is
queued, because the other way round the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection
reset instead of as ok. And ok means queued, not installed - the sender does
not get to know when the swap happened, since only the program knows when it is
between frames.
sand.flan now polls at the top of its loop, which is what this step was for.
Under Xvfb, one line on the socket and 455 consecutive frames drew from a
game-draw that did not exist when the process started. Building without --dev
still works: there are no cells, so a module is refused on the listener thread
and the loop never notices.
flan reload builds one module the way the daemon will. --new names what the
host was not built with, which is the one thing the command cannot work out for
itself and exactly what the session will track.
Two things, and either alone is useless, so they are one commit.
Emit.redefinition compiles one function into its own module against a host
that is already running. What it does *not* define is the design: a global is
external, so state survives a reload and sand's grid is not reset by editing
the code; every other function is a declare, so a redefined settle calls the
host's move-grain rather than a frozen copy; there is no main. Build.shared
puts that text through llc + ld -shared. ld, not clang, because a shared object
is allowed undefined symbols and that is the whole mechanism - and because the
driver is 50ms of a 20ms job. Measured here: llc 16ms, ld 3ms, dlopen 0.04ms.
Loading a body is not installing it, though. A call bound at link time cannot
notice a new one, so a dev build routes every Flan-to-Flan call through a cell
- a mutable global holding the address of the function that is current - and a
module publishes itself with one store. The cell load is emitted after the
arguments, so a redefinition between two calls cannot land inside one.
Three details that are not free choices. flan_reload_install is a named
function rather than an ELF constructor, because the agent has to choose when
the store happens and a constructor would do it during dlopen, mid-frame, on
whatever thread called it. A redefinition's own body is hidden, because default
visibility in a shared object is interposable and that applies to taking the
address too: plain @"flan.bump" inside the module resolves to the host's copy,
so the installer would publish the function it was replacing and the reload
would silently do nothing. And -rdynamic is what exports the cells at all, so
it and cells are one flag: Build.opts.dev, flan build --dev, the first time
opts means something semantic rather than an optimisation level.
The test is one process, because two runs would prove nothing about a swap,
and two .so paths, because dlopen caches by path and would hand back the first
handle. Every call in it goes through outer, compiled once into the host and
never rebuilt, so a changed answer can only mean its call site followed. v2
recurses through its own cell, which is the interposition case; it would print
the old body's text if it did not. helper differs between the fixtures purely
as a tripwire for a module that grew its own copy.
LLVM cannot fold the indirection - the cell is an external mutable global - and
a --dev calc-me keeps 46 indirect calls at -O2. values, machine and
sand-headless now run as dev builds in the acceptance table too; the sand hash
is the one result that would notice a call reaching the wrong function.
Second stage of the milestone-2 frontend. calc-me.flan (12 decls) and
sand.flan (20 decls) both parse end to end, and both are test deps so a
regression fails `dune test` rather than surfacing at the CLI.
Three silent-misparse bugs fixed along the way -- all cases that read
cleanly and meant something else:
- dotimes/defer/some/try/fn fell through to Call, discarding their
binding and control-flow meaning. Now special forms. Forms from later
milestones (handler-bind, restart-case, loop/recur, defmacro, signal,
with-allocator, errdefer, await) are rejected outright rather than
parsed as calls.
- (Some 1) in first body position was read as a return type, because
(Option f64) and (Some 1) are identical s-expressions and the
heuristic was capitalisation. Now decided by the set of names actually
declared as types, collected in a pre-pass -- exact, and
order-independent so a type declared below its user still resolves.
- Array literals in value position were rejected outright.
Also adds NEXT.md with the handoff for the checker.
