Milestone 1 of dynamic-by-default, the runtime half: NaN-boxed values in one
machine word, a mark-sweep heap, and the operations over them.
A double is itself, which is what a language with a physics loop and a float
calculator in its corpus wants; everything else hides in the quiet-NaN space,
three tag bits and a 48-bit payload that is exactly an x86-64 user pointer.
The negative-NaN collision is answered by canonicalising every NaN on the way
in, which flan_rt.c had already decided was the right thing to print. An i64
past the payload goes on the heap rather than becoming a 48-bit integer with a
64-bit name.
The collector is mark-sweep and nothing else -- no generation, no barrier, no
free list -- because the answer to wanting it faster is to type the program.
Roots are pushed, not scanned: NaN-boxing makes a conservative guess wrong in
both directions, and flan_dev.c's frame chain is the precedent. A fixed ring
of the last sixty-four allocations is marked unconditionally, which closes the
window where an expression with two constructors in it can collect its own
first result before the compiler has rooted either.
A type mismatch traps rather than aborting, through a flan_trap exported from
flan_rt.c so it takes the same path the six existing traps take: parked for
inspection in a dev session, dead where it stands otherwise. The sentence
names the operation, both tags as words, and both values.
flan_dyn.c is its own translation unit and nothing in the release runtime
names a symbol in it, so a program with no dyn operation links no collector
and --no-gc can be file-level selection rather than an argument with the
linker.
docs/SPIKE-DYNAMIC.md carries the argument. test/dyn_ops.c drives every
operation and all twenty-four refusals from C, the way dev_limits.c does,
including a million allocations against a hundred live and the control that
says an unrooted object really is reclaimed.
`arena-new` is a builtin, so it is in no program's symbol table and `defs`
never mentioned it — which made the editor answer "the running program
defines no arena-new" about a name that works. The fix is not a better
refusal: it is that the seventy-eight names the checker answers without
being told are now in the reply, under a kind of their own.
check.ml carries the table, beside the arms it describes, because a table
in another file drifts from them with nothing said. test_flan reads both
the arms and the table and fails on either having a name the other does
not, in both directions — there is no reflecting over a match, so it reads
the source.
Each entry is a signature in `signature_of_fn`'s shape and one line. The
arms that do not have one shape say what is true instead of pretending:
`?` for an argument that may be left out, `|` for the types an arm really
takes, and the checker's own predicate names for the type-directed ones.
The three user-allocator names carry a bare name and no bracket list,
because they are refused wherever they are written.
`defs` grows a fifth string for the prose, and builtins are appended last
so a completion table does not bury the names being worked on. A defn has
no docstring to put there and does not get one here: the Tast keeps none,
and that is a different piece of work.
The editor reads the kind, not a special case. `flan-doc--where` and the
xref backend both answer before their empty-location branch, because a
global's missing location and a builtin's absent one are different facts
and only one of them is about the daemon.
vendor/json is vendor/edn's shape with one decision reversed. edn never
allocates, so its tokens are views into the source buffer and escaped
strings are refused for want of anywhere to put the unescaped copy. This
one has an allocator, so it unescapes, and to unescape it copies —
string-of is the only function in the package that allocates, and it
copies even when there was no escape to resolve, because a Value whose
lifetime depended on which bytes happened to be in it is not a contract
anyone can hold. Odin answered the same question the same way:
tokenizer.odin allocates nothing, parser.odin's unquote_string does the
copy, and it clones in the no-escape branch too.
What that buys is at the bottom of test/programs/json.flan, which is
programs/edn.flan and programs/arena-edn.flan in one file because for
JSON they are one claim. The source buffer is overwritten with `?` bytes
while the document is live and the strings read back afterwards are
still the strings. arena-edn's header has a section admitting it cannot
do that.
Strict JSON and not Odin's JSON5 default, and the difference is where
most of the refusals come from: comments, single quotes, +1, .5, 1.,
0x1f, 01, NaN, Infinity and unquoted keys each get a sentence naming the
dialect they belong to, rather than one shared unexpected-byte. A lone
surrogate is refused too, and that one is forced rather than chosen —
rune-size answers None for the whole D800-DFFF block, so encode-rune!
would write nothing and the character would vanish.
The tokenizer refused #{} because "it needs a hash set to even
represent" — which is a claim about a reader, and a tokenizer represents
nothing. #{ now pushes } on the same balance stack { does, there is one
new token kind and no new closer, and err-set is gone rather than kept
with a message it no longer earns. skip-value needed nothing: it is
written against the depth and not against the kinds.
The dynamic reader moves out of test/programs/arena-edn.flan and into
vendor/edn/read.flan as (edn/read bytes), answering an (Option Value)
against whichever allocator the caller bound. Two decisions are written
down where they are made:
* a set is a Value.Set holding a deduplicated (Vec Value), because
(Map Value bool) does not typecheck — keyable refuses a key holding
a Vec or a Map — and restricting elements to keyable Values would
refuse #{[0 0] [1 0]}, which is the file this was built for. Insert
is O(n) against a structural value=?, so building the tileset's 54
pairs is 1458 comparisons, once.
* a Value copies every string into the allocator where a Token stays
a view. A view handed back out of the function that owns the buffer
is a dangling pointer, and free-all would not even take it. Odin's
json parser clones for the same reason.
An imported defdata was a refusal in load.ml — "not implemented yet
(milestone 4)" — and it had to go first. It is the type's name plus the
Type. half of a constructor symbol, which arrives as a Var node when the
case has no fields and a Struct node when it has; a match pattern needed
nothing, because a case resolves against the scrutinee's type and was
never a top-level name. programs/pkg-data.flan is that on its own.
programs/edn-read.flan reads assets/edn/tileset.edn, which is the
editor's real output: :texture-path and a :selected-cells of 54 integer
pairs, with no type declared for any of it. It also overwrites the
source buffer in place after reading and prints the document back, which
is the copy contract asserted rather than described.
A GitHub Actions workflow on push: dune build, dune test --force, dune build
@checks. @x86 parity is not under dune test, so the routine suite never
protected it; both of this repository's silent failures would have been caught
by one person typing one command, and the problem was never the command.
The suite step keeps its log and greps it for Fatal error, because a suite
that passes while leaving an unhandled exception on stderr is one that is
telling you something and being ignored.
FLAN_LLC is pinned to the llc matching clang's version rather than left to
PATH order: the live loop goes llc + ld -shared + dlopen and never calls the
clang driver, so a mismatch breaks every reload test while flan build keeps
working, which is a bad failure to debug from a log.
What an Ubuntu runner cannot cover -- raylib by exact Fedora soname,
emscripten, a wasi sysroot, lldb -- is written in the workflow with the skip
path each one already takes, so the tick does not read as more than it is.
README.md and test/dune both said there was no CI; both now say what there is
and what it misses.
The alias is @x86's shape and deps: one rule over spike/js/survey.sh, FLAN
passed so the script does not start a dune inside dune's own lock. It is not
in the default run and node is probed rather than assumed.
@page runs web/examples/check.sh and web/examples/quotes.sh against the compiler
dune just built. @cells runs spike/x86/cells.sh, which was a real pass/fail check
-- four builds, two backends, 22 22 against 42 42 -- that nothing in the tree ran.
@checks is @page, @x86 and @cells together, and its comment argues for where the
boundary sits: everything you can run while making coffee is in, @sanitize and
@valgrind are out because folding tens of minutes in would make the umbrella the
thing nobody has time for, which is the disease rather than the cure.
All three scripts learned to resolve FLAN to an absolute path, which is what
actually stood between them and a dune rule: %{workspace_root} expands relative to
the directory the rule is written in, and every one of these scripts cd's somewhere
before using it. The first run of @page failed with twenty diffs all saying
'../bin/main.exe: No such file or directory', which is at least a failure that says
what is wrong.
docs/BUILT.md carried the same colon-spelled renderer block index.html did, from the
same sweep. Nothing checks BUILT.md, so it is corrected here by hand.
The feature was built and never tested. Three things were missing.
The two packages holding a ring of macros and a macro that never settles
were not dependencies of the test stanza, so both non-termination
refusals failed on "no package at ..." rather than on their own reason.
They fire, and now the suite sees them fire.
The positive half of the rule had no acceptance case at all -- only the
refusal that pins the bare name. pkg-macro.flan is asserted at three opt
levels and on the dev path, which is where six package macros and the
program's own coexist in one file.
And the dev loop was broken in exactly the way that matters most here.
Session held the imported macro set but *replaced* it on every
evaluation, and the one form C-c C-c sends carries no import -- so
(mac/twice 4) compiled on the build and came back "unknown function" on
the first reload. It unions now. test_session drives two evaluations,
because one proves nothing: the first could have re-supplied the set.
BUILT.md said the expander collects from the prelude and the file being
compiled. It collects from imported packages too, and the refusal's old
reasoning -- that this needed a second import resolver -- was wrong for a
reason worth keeping written down.
Cold build cost roughly doubles for a program importing a package that
declares macros: a macro module is built per round and the package's
rounds are its own. Warm is unchanged at ~70ms.
765 lines to 201, at parity with lisp/sand.lisp, clojure/src/fnm/sand.clj
and src/fnm/sand.jank. Gone: audio and tone synthesis, the brush textures
and the embedded PNG, the render-texture scene, the HUD font, the camera,
the world cursor, the HUD and the input-state read-out. None of it is in
any reference version, and none of it was the language being exercised.
Inlined the single-use helpers -- empty-at?, move-grain, draw-grid, paint.
settle also carried an unused (rl/Vector2 ...) binding, which put raylib
inside the section whose banner says there is none, and a stray (pause).
The physics is untouched on purpose: the references disagree there, so
parity does not name a target, and settle is what the pinned hash covers.
It still prints 15595743031174623232 at -O2, -O0 and as a dev build.
test_web.ml asserted brush.png's bytes reached the wasm module, proving an
embed survives the web build. web-files.flan is web-built and *run* under
node and asserts the embedded bytes print, which is the same property
checked harder, so the sand assertion goes rather than the embed staying.
The expander collects defmacros from the prelude and from the file being
compiled. Not from an imported package, and the reason is an ordering one:
Load learns a package's imports by parsing it, so reaching a package's macros
would mean resolving that package's own imports over Forms, before Load runs.
That is a second import resolver, and it is a bigger thing than this lane.
Refused by name, which is the rule that caught the two misparse bugs. Left
alone the call arrives at the checker as an unknown name -- true, and no help.
Refused where the defmacro is written rather than where it is called, because
that is where the fix goes.
The check has to sit in Load's read, because that is the only place that can
see one: by the time Parse is finished a defmacro is an ordinary Ast.Defn and
the word is gone.
Measured while here, since a prelude that grows a defmacro is a cost every
program pays or does not:
- A build of a program that names no macro: 50ms, the same as before. The
pass scans the top level, finds nothing, and no compiler runs.
- A program that calls one: 310ms the first time, 70ms after. The 240ms is
the clang driver building the macro module; it is cached under the object
cache, keyed by the prelude's source and the file's defmacros, so it is
paid once per change rather than once per build.
- A hello-world's binary carries exactly one symbol out of all of this:
flan.gensym-n, eight bytes. Reach.link drops unless, form-cons, form-nil,
form-append, form-rest and gensym, because nothing reachable calls them.
Loading a package kept one table, keyed by real path, and used it for two
different questions. Already loaded meant "skip", which is right for the second
route of a diamond and wrong for a ring: a package that imported itself round a
chain met its own entry, contributed nothing, and appeared to work. The comment
said so and called it a feature.
It is not one. A ring has no package order, and a definite package order is what
the macro expander needs — every defmacro has to be compiled before anything
that calls it. So the chain currently being read is now carried separately from
the set already finished. A directory found in the first is a cycle and is
refused; a directory found only in the second is still the diamond's second
route and still a no-op.
The refusal names the ring — a -> b -> c -> a — and only the ring, not the route
that led to it. "There is a cycle" leaves the reader to find which three imports
it was.
pkgs now comes back dependencies-first, which is the topological order the
acyclic rule buys. The declaration list is left alone: check.ml collects every
top-level name before it checks any body, so declarations are order-independent
by construction and sorting them would be churn in the field every test reads.
The tests are a real tree rather than a second copy of pkg-shared. pkg-diamond
builds a shape/Box inside area/ and hands it to a function declared inside
draw/, which only type-checks if the bottom package was read once — two copies
of one struct are two types. What proves it is the numbers, not the compile.
Nothing in dune test exercised cimport.ml or cjson.ml. The raylib case is the
better evidence and the worse coverage: it needs raylib installed, at the
version whose .so is linked, with FLAN_RAYLIB_H set, so as the only test of
this it would skip everywhere and cover nothing.
test/headers/sample.h is one function per decision the importer makes, and the
table asserts on the reasons rather than the counts — a refusal that fires for
the wrong cause still refuses, and a count still matches. Accepted: an
aggregate in and out, const char * as a string, a pointer parameter, a second
typedef name for a record described once, a C enum against a defenum. Refused,
each by reason: a returned char *, a non-const char * C may write through, a
variadic, a callback, a long, a struct with no defstruct, and a kebab
collision. Plus that nothing is in both lists, which is the bug the collision
case found.
check_structs and diff_bound get a row each for agreeing, for a permuted field
order, for a widened field, and for a symbol the header does not have — the
last being how a package pinned to the wrong release announces itself. The
name rule and the JSON reader get their own rows.
Checked by breaking two of them on purpose and watching both fail.
test/programs/raylib-imported.flan is the end-to-end evidence, back and in the
new struct-literal spelling: four bindings the package does not bind by hand.
ColorToInt of {17,34,51,68} is 0x11223344 and ColorTint by white hands the four
bytes back separately, so field order is pinned by arithmetic and not by a
round trip, which is the trap BUILT.md records.
Three things stood between the flagship program and the web target, and each
is answered here rather than worked around.
The brush was a path. (rl/load-texture "brush.png") hands raylib a filename to
open, and a bare relative path means nothing on a target with no filesystem.
It is (embed "brush.png") now, decoded through a new binding —
LoadImageFromMemory, declared (Ptr u8) plus an explicit count because the shim
generator refuses a slice parameter and says so, with a Flan wrapper taking
the slice apart exactly as collision-point-poly? and load-font-ex already do.
One decode now serves both textures: the unflipped upload first, then
ImageFlipHorizontal in place, then the mirrored one. load-texture and
load-image lose their only call site in this repository; that is deliberate,
because a path-based load is the thing that cannot work here.
A package's C may now be addressed to one target, the way a link line already
could. A .c file may carry a tag before its extension — flan_agent.web.c — and
on that target it is compiled and *replaces* the untagged file of the same
base name. Replacement rather than plain tagging, so that teaching a package
about a new target is additive: the file that was right on three targets is
not renamed to say so. Selection is in Build and not in Load, for the reason
select_lflags gives.
The dev agent on the web is a no-op, and the reasoning is written at length in
vendor/agent/flan_agent.web.c. Short version: the agent is a socket server and
a browser has no sockets, so the missing <sys/time.h> was the surface and not
the cause. Refusing vendor:agent on a web target was the other candidate and
is ruled out by arithmetic — Flan has no conditional compilation, sand.flan
calls agent/start unconditionally, Reach cannot prune a package something
reachable calls into, so a refusal means the program does not build for the
browser at all. This does not contradict the `barf` decision made earlier
today. `barf` is asked to make something durable, and a no-op returns success
to a program that now believes bytes are on disk. The agent is asked to accept
redefinitions, and on the web there is no editor, no socket and no session —
--dev is refused by name on every wasm target — so there is nothing to lose.
sand.flan already says the same of a native release build at the call site.
test/test_web.ml builds sand.flan for the browser and reads the module for
brush.png's own bytes, whole. Not "IHDR": stb_image carries that string itself,
linked in from raylib, so it would pass on a build where the embed emitted
nothing. It is not run — node has no DOM, so main reaches InitWindow and dies
inside glfwInit on `window is not defined`, which says the module is live and
nothing about whether the canvas paints.
test/dune gains brush.png, because an embed is read by the checker and the
headless case reaches sand.flan through ../../ from a sandboxed _build.
test_session's C-c C-k case now passes ~origin, which is what both editor
paths already send; omitting it was testing a request nobody makes.
dune test is green. Docs follow in the next commit.
Decision 1. Odin's #load and #load_directory are the model, spelled as
ordinary named calls — an s-expression language already has a head
position and does not need Odin's `#`. (embed "p") is a [u8], (embed "p"
string) is a string, and (embed-dir "d") is a [n EmbedFile] sorted by
name.
Two spellings rather than one that changes type with its context. Odin
threads a type_hint everywhere and can afford it; with structural
equality and no implicit widening, the same text meaning two types here
would be a wart. The path is a literal and resolves relative to the file
the form is written in, both of which are Odin's rules and for Odin's
reasons: the bytes must be in hand before any value exists, and a
package's assets must not depend on where flan was invoked from.
The bytes reach the program as a [Str] node typed [u8], not as a [Bytes]
prim over a string. [Bytes] is identity — emit.ml lowers String and
Slice _ to the same %slice — and wrapping the literal in a prim would
make the node non-constant, so an (embed-dir) bound with defconst could
not be an LLVM constant. Both string emitters take the bytes and ignore
the node's type, so it is the same constant either way and one a global
can hold. emit.ml's escape is byte-exact, so a PNG survives the .ll.
The directory lookup is a linear scan in the prelude over a slice of
EmbedFile. A directory embed is tens of entries out of cache-warm
.rodata, and a compile-time perfect hash would be a build-time map with
its own failure modes that nothing has asked for. Sorted because readdir
order is filesystem-dependent and an unsorted embed would make two
builds of identical sources emit different .ll.
The slice points into .rodata, so a store through it segfaults at -O0
and is deleted at -O2 — the same measured trap the prelude's ASCII-case
note describes for (bytes "Hi"). Inherited, not widened; clone into a
Vec for a mutable copy.
Neither limit had any coverage: a renderer that emits more than 4K and
a program that introduces more than 4096 run-time names are both past
anything the corpus does, so the truncation and the abort were code
that had never executed. dev_limits.c drives them directly — they are C
entry points with no Flan spelling, and flan_dev.c is compiled into
every build — one process per mode, because the name table never
shrinks and the overflow case aborts.
The cap case pins the length, the ellipsis, a byte from before the cut,
the generation moving exactly once, and the flag being cleared so a
short value after a truncated one does not inherit its ellipsis. The
registry case pins that 4096 fit and the next one stops the process
with its reason. Dropping result_full and moving the slot check by one
were both planted and watched fail.
The mutation pass turned up one defect that did not make the suite go
red: a reader branch that forgets to advance reads the same character
for ever, and dune test waits as long as it is left to. In CI that is a
job killed by the runner with nothing named and no output to read.
watchdog.ml puts an alarm on every test binary — generous, because an
alarm that fires on a slow machine is a flake — and a five-second one
around each read in test_flan, where the budget really is small. The
first read that does not return wedges the rest, so a looping reader
costs five seconds and names the row instead of costing eight minutes
or never finishing. Both were watched: the string-escape loop now fails
in five seconds with the case named, and the per-binary backstop was
armed short and observed to fire.
test_web.ml never opens a browser and never will. What it asserts is the shape
a browser needs — three files, a module that starts with the wasm magic, a
page that references its own JS and carries the canvas — plus the one
execution available without a DOM: node runs the emitted JS and gets "ok".
For raylib it builds core-basic-window.flan unchanged, which is the claim, and
then reads the module for the two things that would be false if the mechanism
were wrong: an asyncify_start_unwind export, and a glViewport import that can
only have come from raylib's web platform. Import and export names are plain
strings in the binary, so this needs no wasm reader.
Both halves probe rather than assume, the way the wasm32 case does: emscripten
may not be installed and the raylib archive is not in the tree, and a missing
piece is a skip with the reason.
The four refusals are asserted by name — --dev, --debug, --sanitize,
Build.shared, and flan run --target=web from the CLI — because "it falls out
of the existing predicate" is the kind of thing that stops being true quietly.
publish() wrote queue[head % QUEUE] without consulting tail, so the 65th module
queued between two agent/poll calls landed on the slot the game thread was
reading — twenty-four bytes of function pointers copied field by field with no
atomic near them, so the consumer could take half of one job and half of
another and call it. The comment claimed the overflow dropped the oldest
request; nothing did that.
A full ring is refused now, at the sender, before the dlopen. Dropping loses a
reload the sender was told was ok, which is the same lie more quietly; blocking
stalls the accept loop, which serves connections inline, so a program that had
stopped polling would also stop answering status and abort — the dev loop would
have no way to reach a program that had stopped listening to it. The check is
separate from the store because there is one producer: room, once seen, cannot
be taken away.
Two smaller defects in the same file:
A module with no flan_reload_install was refused and its handle dropped on the
floor. Not an exception to "nothing is ever dlclosed" — that rule is about a
module something points into, and this one installed nothing, so no cell names
it. What leaked was the handle value rather than the mapping: dlopen refcounts
by path, so re-sending the same bad file raised a count nothing could lower.
exit(134) from the break loop runs the atexit chain and the ELF destructors,
which want the loader lock the listener thread may be holding inside dlopen. A
program asked to abort would hang instead of dying. _exit, with the streams
flushed by hand at each call site. The deadlock itself is read rather than
tested; what the tests pin is that the exit status is still 134.
programs/agent-queue.flan blocks on stdin so the window is held open by the
test rather than by a timer: it takes 64 modules, refuses the 65th with a
reason, and installs 64 when it finally polls. noinstall.c's destructor prints
while the program is still running, which is the only way to see the close — at
exit the loader runs every destructor whether anything was closed or not. Both
halves fail on the old code.
Twenty-eight programs built twice -- once plain, once sanitized -- and
compared on output and exit status, plus two positive controls that are
the only reason a clean result means anything: an out-of-bounds read
that must report, and a shift by the width of the type that must not,
because UBSan cannot see hand-written IR and this file would otherwise
be claiming coverage it does not have.
Its own alias rather than dune test. A sanitized program is a statically
linked 1.8MB binary and takes tens of seconds to link; the sweep is nine
minutes against the existing suite's seconds, and a test nobody will
wait for is a test nobody runs. dune build --root . @sanitize.
The checked sweep is clean. The unchecked variant -- ASan alone, with
Flan's own bounds checks off -- catches three of bounds.flan's six
deliberate out-of-bounds cases and is listed with why for the other
three: a global has a right redzone and nothing to its left, so arr[-1]
is invisible; a read past a string constant folds away entirely at -O2
and is caught only at -O0; and a reversed slice reads nothing at all.
ASan is not a substitute for the bounds checks, and now there is a table
saying which half it covers.
The first ten of raylib's core list, ported. Seven new bindings and the
named colour palette; nothing else was added, because a binding called
by nothing is the same as not having bound it.
The gaps they found are the point. No number reaches draw-text: i64->bytes
answers [u8], draw-text wants a string, and nothing bridges — five of the
ten wanted TextFormat and got a glyph table instead. And an enum parameter
cannot be driven by a loop variable: the index is an i32, the parameter is
an enum, neither converts, and a second declare-c with an i32 face is
refused because one C function gets one binding. Two correct rules that
compose into a wall.
None of the gaps expected blocked anything: no generics, no allocator, no
Vec, no escaping closure, no block-scoped defer. These are input-and-draw
programs over fixed-size state, which is the shape the language has.
C-x C-e renders once and stops at depth 4 and span 8. A field past either
comes back as "..." and nothing recovers it from the echo area. Re-rooting
the walk at that field renders it from depth 0, so the bound moves with you
— that, and not tidiness, is why an inspector is worth having beside the
expression evaluator.
CIDER keeps its inspector stack on the server because a JVM value can be
retained. Nothing here can: a Flan value has no header and the thunk that
rendered it is dlclosed the moment it returns. So the stack is a stack of
expressions on this side, and going into a field means sending a different
one — (.pos b) where the last one was b. It costs a re-evaluation per step,
which buys a view that is never stale and is why refresh is a key someone
presses rather than a timer.
Driven from fixtures, which is also the only way the cases a live program
will not hold still for get tested at all.
The simulation was in a package of its own for one reason: importing raylib
linked libraylib on every target, so the headless run could not name the
package the interactive one needs. That reason is gone, and the split was
never anything else — the physics is the same code either way.
So sim.flan is back inside sand.flan, and test/programs/sand-headless.flan
imports sand.flan itself: window, raylib bindings, dev agent and all. It builds
for wasm32 anyway. Nothing it calls reaches raylib, so no shim is compiled, no
-lraylib is passed, and the front-end's functions are never emitted; sand.flan's
main is not exported, so the only main is the headless one. The hash is
unchanged on both targets at both optimisation levels, which is the point —
a refactor that moved the number would have moved the simulation.
The new cases cover what made it possible rather than only the result: a
package nothing calls into, native and wasm32; raylib reached both directly and
through sand.flan and read once; and the three refusals — sand/main, one
directory under two aliases, and two mains.
test_session's package-qualification case moves to vendor/agent, which is now
the package in the tree with a defn in it.
vendor/edn rather than the prelude: the prelude is prepended to every program
and everything in it is emitted, so a reader nobody imports would be a cost
every build pays.
The tokenizer only. A type-directed reader - the compiler emitting a parser
from a walk over a struct's fields, the dual of the printer C-x C-e already has
- lands in check.ml and emit.ml and is not this. What a caller writes today is
a struct reader by hand against the cursor, and the acceptance program carries
one, because that is what proves the API is usable rather than present.
Every token is a slice into the source, so nothing allocates and the buffer has
to outlive the tokens. That contract is stated at the top of the package,
because it is the kind of thing found the hard way.
Escaped strings are refused rather than half-supported: unescaping needs a copy
and there is nowhere to put one, and handing back the raw bytes would return a
three-byte string as four with a backslash in it. Each other refusal carries its
own sentence - #inst and #uuid separately from tagged literals, because a file
is most likely to contain those two and being told tagged literals are refused
would not say that the timestamp is the thing to delete.
Errors live on the cursor, a code and a byte offset, not in the return type: an
Option loses the position, which is the whole point for an editor. A failed
cursor is poisoned so a caller's loop terminates on a malformed file rather than
spinning.
# Conflicts:
# test/test_acceptance.ml
An import reads the directory at build time, so a package that dune has not
copied under the test's build dir does not resolve — and the failure is a
missing collection, not a missing file, which reads like a bug in Load.
The wasi-sdk candidate had an LLVM version in it, which moves release to
release — so the path advertised as the proper article would have matched only
by coincidence, while the emscripten one beside it was derived. Both are
derived now.
calc-me on wasm32 covers what the other three cases cannot: flan_argv hands
Flan an array of flan_slice built in C, so what it pins is the element stride
of a ptr+len pair — 16 bytes native, 12 on wasm32 — rather than a field
offset. It is also the claim in this file's own header, that the table runs on
the second target, honoured for the first time.
flan emit refuses --target rather than stripping it. The IR really is
target-free, so ignoring it is correct and silence about it is not.
sand-headless imports no raylib so that it can run here, and the point of the
case is not that a module exists — it is that the number matches native byte
for byte, which is only possible because rand-f32 is Flan's rather than libc's.
It does, at -O2 and at -O0; -O0 is the cheap way to say the agreement is not a
coincidence of how LLVM folded the float arithmetic. values and machine run
there too, which is where a 32-bit pointer would have shown.
Four separate things can be missing — clang's wasm target, the sysroot, the
builtins, a WASI runtime — so the skip is a probe rather than a lookup: build
the smallest program and run it, and print what went wrong. A which(1) would go
red on the machine where Node is too old, with a reason nobody could read.
No wasmtime and no wasmer here, so the runner is node:wasi, with wasmtime and
wasmer preferred if either appears. --no-warnings because node:wasi writes to
stderr on every run and this harness compares combined output.
A different primitive from redefining a name. There is no name to install a
body into, so the expression is wrapped in a function with nowhere to be called
from; the module exports flan_reload_call to say "run this once", and the agent
calls it after the install - on the game thread, at a frame boundary, so an
expression that reads the program's state sees a point the program agrees is
consistent.
Nothing is marshalled back because nothing could be. A Flan value carries no
header, so no code at run time can say what it is; the compiler knows the type
and renders it there, in the thunk. That is the layout decision's bill, and it
is why the printer set is the scalars rather than everything.
The rendering does not go through stdout. Stdout belongs to the program, it is
in the hot path for anything that prints, and a dev-only feature must not put a
branch in it - so flan_rt.c is untouched and the value goes to flan_dev_result,
read back over the agent's socket. Safe without a handshake because the
generation counter is bumped last: the daemon waits for it to move rather than
assuming the program has reached a frame boundary.
u64 refuses by name, because i64->bytes is signed and anything past 2^63 would
come back negative. Everything without a derived printer refuses the same way.
A number that is quietly wrong is the failure this whole thing exists to
prevent.
An evaluation is not a declaration: the thunk is built against the program and
never spliced into it, so describe does not fill up with an eval/N for every
expression ever typed.
The test that matters is the same expression twice. The fixture increments
ticks every frame, so two evaluations must disagree - a value computed in the
compiler, or read from a copy of the program's state, would not.
C-c C-c recompiles the top-level form at point and installs it in a running
program at that program's next frame boundary. Verified against sand: an
unsaved buffer edit to game-draw, and 240 consecutive frames drew it.
flan-mode.el derives from prog-mode with lisp-mode's syntax table, which is
most of the work - Flan is s-expressions, so sexp motion, paren matching,
beginning-of-defun and indentation are already right. What it adds is Flan's
own brackets ([ and { are brackets and not symbol characters, since every
binding list and every type is written with them), the characters a name may
contain, and its keywords.
flan-dev.el has no parser in it, which is what the protocol choice bought:
prin1 writes a request, read reads a reply. C-c C-k sends a buffer as one
module rather than a form at a time, because a defvar and the function using it
have to arrive in the same load or the first refers to storage that does not
exist yet. An error comes back with a location and point moves there.
Framing is in bytes and Emacs counts characters, so every length goes through
string-bytes and the process is binary. Otherwise one non-ASCII character in a
buffer puts the reply stream out of step by exactly as many bytes as the
payload has of them - a bug that reads as a corrupt protocol and only appears
for some people. test_emacs.ml drives the real client against a real daemon for
that reason: it is not the same claim as the daemon answering correctly, and a
mistake in the framing, in beginning-of-defun over Flan's syntax table, or in
the reply reader passes test_dev.ml and fails here.
The piece between an editor and everything else. One long-lived Session, the
program it belongs to launched and owned by the same process, and a socket that
takes forms and installs them. What it adds over flan reload is that the
session persists - a defvar added by one evaluation is part of what the next is
checked against - and that it owns the build, which is what makes its layout
rules describe the process actually running rather than a guess about it.
The protocol is s-expressions rather than bencode, and I changed my mind about
that. The case for nREPL was reusing a designed op set and not re-litigating
session identity, but with the client ours too there is no CIDER to be
compatible with, its eval is string-in/string-out with no slot for which form
from which file, and Emacs already has read and prin1. So: one sexp per
message, length framed because the payload contains newlines. No parsing code
on the editor side, and on this side the parser is the language's own reader,
where :op is already a keyword and Flan source is already a string literal. An
nREPL front end can sit on the same Session later; it should not gate the
editor.
Two silent failures the daemon refuses to have. The agent socket is chosen by
the daemon and forced through FLAN_AGENT_SOCKET before spawning, because a
program's source has to name some path and a daemon that guessed would compile,
build and deliver a module to nobody. And delivery is checked: agent/start
returning 0 means a socket was bound, not that anyone connected, so a failed
connect or a reply that is not ok becomes an error the editor sees.
It waits for the program to bind before accepting an evaluation, since one
arriving first fails for a reason that reads like a compiler bug, and it
accepts with a timeout so a program that has exited takes the daemon with it
instead of leaving an editor waiting on a socket nobody serves.
lib/session.ml holds the declarations a running process was built from plus
every change accepted since, which is what an editor needs and what a one-shot
compiler cannot have.
Transactionality came for free. Check.program builds a fresh environment from a
declaration list on every call, so a form that fails to check mutates nothing
and the accumulated list is simply not replaced - no scratch-environment
machinery, which is what I was about to build. Re-checking the whole program
each evaluation costs the frontend, under 10ms, less than the llc after it.
There is a test for the case that matters: a typo, then a good form, in the
same session.
Which names the process was built with comes from the checked program, not from
any accumulated AST, because Check.program prepends the prelude and no AST
contains it. Derive it from declarations and print-line reads as new, gets a
registry cell nobody publishes, and the first call jumps to null.
Three changes are refused with a reason rather than loaded. A function's
signature, because a cell is a bare ptr and every call site compiled before the
change still passes the old arguments through it. A global's type, because the
storage exists and has a shape - reusing it reads at the wrong offsets, and
replacing it discards the state the reload exists to preserve. A struct's
fields, because the values the process is holding have the old layout. Note
what the checker already catches on its own: change a parameter type and the
caller fails to type check first, loudly. These rules only get a turn on a
change the checker accepts, which is a name nothing else in the program uses -
exactly where the silent version lives. Hence an unused defvar and a C-called
defn in the fixtures.
The accumulated list is the post-Load one, so an evaluated import is spliced as
its expansion. Otherwise re-evaluating a file that imports something appends a
second import, Load expands it again, and the duplicate-name pass rejects it.
C-c C-k on sand.flan's own text is the test.
flan reload now takes a program and a file of changed forms rather than a list
of function names and a --new list: the session works out which names are new,
which is the thing a bare CLI could not.
Also fixed, found by running the agent test under load: the agent took SIGPIPE
when a sender read part of a reply and closed. Replies go out with
MSG_NOSIGNAL, per call rather than by installing a handler, because the signal
disposition belongs to the program the agent is embedded in.
vendor/agent/ is a package like any other - agent.flan declares three calls,
flan_agent.c implements them, link asks for -lpthread. start listens on a unix
socket, poll installs whatever arrived and says how many, wait does the same
after waiting for something.
The split between poll and the listener is the whole design. dlopen relocates a
module and takes the loader lock, which is milliseconds and unbounded, so it
happens on the listener thread. flan_reload_install is one store per function
and must not land while a redefined function is on the stack, so it happens on
the game thread at the top of the frame, when the program asks. A ring and two
atomics connect them; the game thread never blocks on the loader.
wait exists for tests. A test that races the frame rate fails on a loaded
machine, so test/programs/agent.flan waits for the reload rather than sleeping
past it. It also sends a junk path first: the daemon is a separate process and
can send anything, and a bad path must be refused rather than take down the
program it was sent to.
Two things came out of running it. The reply goes out before the module is
queued, because the other way round the game thread can install and the program
can exit between the two, and the answer reaches the sender as a connection
reset instead of as ok. And ok means queued, not installed - the sender does
not get to know when the swap happened, since only the program knows when it is
between frames.
sand.flan now polls at the top of its loop, which is what this step was for.
Under Xvfb, one line on the socket and 455 consecutive frames drew from a
game-draw that did not exist when the process started. Building without --dev
still works: there are no cells, so a module is refused on the listener thread
and the loop never notices.
flan reload builds one module the way the daemon will. --new names what the
host was not built with, which is the one thing the command cannot work out for
itself and exactly what the session will track.
Two things, and either alone is useless, so they are one commit.
Emit.redefinition compiles one function into its own module against a host
that is already running. What it does *not* define is the design: a global is
external, so state survives a reload and sand's grid is not reset by editing
the code; every other function is a declare, so a redefined settle calls the
host's move-grain rather than a frozen copy; there is no main. Build.shared
puts that text through llc + ld -shared. ld, not clang, because a shared object
is allowed undefined symbols and that is the whole mechanism - and because the
driver is 50ms of a 20ms job. Measured here: llc 16ms, ld 3ms, dlopen 0.04ms.
Loading a body is not installing it, though. A call bound at link time cannot
notice a new one, so a dev build routes every Flan-to-Flan call through a cell
- a mutable global holding the address of the function that is current - and a
module publishes itself with one store. The cell load is emitted after the
arguments, so a redefinition between two calls cannot land inside one.
Three details that are not free choices. flan_reload_install is a named
function rather than an ELF constructor, because the agent has to choose when
the store happens and a constructor would do it during dlopen, mid-frame, on
whatever thread called it. A redefinition's own body is hidden, because default
visibility in a shared object is interposable and that applies to taking the
address too: plain @"flan.bump" inside the module resolves to the host's copy,
so the installer would publish the function it was replacing and the reload
would silently do nothing. And -rdynamic is what exports the cells at all, so
it and cells are one flag: Build.opts.dev, flan build --dev, the first time
opts means something semantic rather than an optimisation level.
The test is one process, because two runs would prove nothing about a swap,
and two .so paths, because dlopen caches by path and would hand back the first
handle. Every call in it goes through outer, compiled once into the host and
never rebuilt, so a changed answer can only mean its call site followed. v2
recurses through its own cell, which is the interposition case; it would print
the old body's text if it did not. helper differs between the fixtures purely
as a tripwire for a module that grew its own copy.
LLVM cannot fold the indirection - the cell is an external mutable global - and
a --dev calc-me keeps 46 indirect calls at -O2. values, machine and
sand-headless now run as dev builds in the acceptance table too; the sand hash
is the one result that would notice a call reaching the wrong function.
Second stage of the milestone-2 frontend. calc-me.flan (12 decls) and
sand.flan (20 decls) both parse end to end, and both are test deps so a
regression fails `dune test` rather than surfacing at the CLI.
Three silent-misparse bugs fixed along the way -- all cases that read
cleanly and meant something else:
- dotimes/defer/some/try/fn fell through to Call, discarding their
binding and control-flow meaning. Now special forms. Forms from later
milestones (handler-bind, restart-case, loop/recur, defmacro, signal,
with-allocator, errdefer, await) are rejected outright rather than
parsed as calls.
- (Some 1) in first body position was read as a return type, because
(Option f64) and (Some 1) are identical s-expressions and the
heuristic was capitalisation. Now decided by the set of names actually
declared as types, collected in a pre-pass -- exact, and
order-independent so a type declared below its user still resolves.
- Array literals in value position were rejected outright.
Also adds NEXT.md with the handoff for the checker.
