flan dev is one process, so the unix socket between the compiler and the
program was a connect, a write and a read that never left the machine's own
memory. A delivery is now a call into the agent's verb table.
flan_agent.c's handlers write into a sink that is either an fd or a growing
buffer, so serve() and the new flan_agent_request() share one verb table rather
than two that would drift -- every answer in there is assembled from several
pieces, so the seam had to be the writing. Dev.deliver, Dev.result and Dev.ask
were three copies of the same socket dance and are now three lines over one
Dev.request.
Which path is taken is the linker's answer, not a flag: flan_agent_request is
weak in dynload_stubs.c, null in every binary that links no flan_agent.o, and
Agent.request is None there. So --two-process, flan reload and every test go on
using the socket with nothing to configure.
Three things the direct path must not quietly change. The ring's room check is
separate from its store, which was safe only while the accept loop was the sole
producer, so handle_line runs under a mutex. A delivery is still only published
-- the install is one store per function on the game thread at a frame
boundary, and doing it on the spot would be a frame running half in the old
code and half in the new. And the OCaml runtime system is released across the
call, because a delivery is a dlopen.
Measured, and the number is the point: the transport was ~50us of a 23ms
redefinition, so end to end did not move. Code generation is 19 of the 22
milliseconds. The merge's prize was never latency.
The test is a deletion, because a reply cannot say which way it came:
test_dev.ml unlinks the agent's socket file once the merged program has bound
it, and every evaluation after that still installs.
The refusal list was rewritten and the prologues that pointed at it were not,
so prelude.ml claimed in three places that what it now contains is impossible:
the splitting header said `split` is refused at the foot of the file, forty
lines above `split`; the ASCII-case header said Odin's allocating to_lower is
not available here, next to the one that was written; and the UTF-8 header
said the rest of core/strings is refused rather than ported.
Each keeps its point rather than losing it. The iterator is still the shape
that owns nothing and still the right call when there is no result to own;
lower-ascii and bytes-ci=? are still the right calls when a copy is not
wanted, since folding a comparison over two inputs beats lowering both. What
changed is the reason, which used to be the absence of an allocator and is now
a choice between two shapes that both exist.
And strings.flan told the reader the opposite of what it did -- "not freed",
on the line above the free. vec.flan already had the right framing: the free
is written, it keeps the block because an arena cannot release one, and that
is the difference the capability set exists to state.
The list at the foot of prelude.ml was one sentence -- every entry needed to
produce bytes that did not exist in its input, and there was no allocator --
and that sentence has been false since Vec landed. Seven entries move up into
the code, and string-from-bytes turns out to have been the `string` builtin
all along: (string (as-slice v)) is the round trip, free precisely because
the layouts are identical.
What is left is refused for four different reasons and is written that way
now: pad and center for nothing at all except that no caller has asked;
format and sprintf for variadics of mixed type; map, filter, reduce and
sort-by for function values; map-keys and map-values for a map iterator that
does not exist in the runtime.
NEXT.md's queued section is struck and carries the four findings, each with
the change it wants named -- flan_map_next plus one builtin for the iterator;
milestone 5's function values for the higher-order three; vec_new_elem taking
a type expression rather than a bare name, which is what forces slices-new to
exist; and an array literal with no way to say it is [f32], which is what
forces every float in algorithms.flan to be cast. BUILT.md gets the section.
return is refused inside handler-bind and restart-case blanketly, and rightly:
a return always crosses the frames they pushed. A break does not. A loop
written wholly inside a restart-case body has a perfectly good local break, so
the rule is a barrier on the loop stack rather than a flag — a jump is refused
exactly when a barrier stands between it and the loop it names, and the message
says which construct. handler-bind and restart-case bodies are barriers, so is
a restart clause, so are a defer's forms; a handler clause is lifted into its
own function and needs no rule at all. in_frames is untouched: a return is the
special case where the target is always outside every barrier.
continue wanted the other blocker. check_dotimes folded its step onto the end
of the body, which a continue would jump past, so the counter would never
advance and the loop would hang. Tast.While carries a latch now — condition,
body, latch — the step goes there, and emit_while emits four blocks. A while's
latch is empty and folds away.
Labels are Odin's, in the head position: (while :outer c ...) and (break
:outer). A keyword there is unambiguous because a loop condition is never one,
so one label function serves while, until, dotimes, break and continue. It is
not a goto — the checker resolves a label against the loops the form is
lexically inside, so control can only leave a loop it is already in.
Break and Continue carry a relative depth rather than a name, because that is
what a backend already has: emit keeps one entry per While the way it keeps
one pad per frame, and indexes it.
Nothing in the prelude wants either. Every early exit there is a return from
the function, which break cannot replace; the sentinel-flag loop break exists
to remove does not appear in it. The two the compiler emits are that shape and
are the one place it cannot help — their sentinel is set inside a restart-case.
reach.ml and render.ml take the While arity change and nothing else.
sort-i32! was the only sort in the language. sort-f32! and sort-bytes! are
the other two, and they are copies rather than an abstraction for a reason
worth naming precisely: map, filter, reduce and a sort taking a comparator
are not blocked on generics, they are blocked on *function values*. Types.Fn
exists and check.ml refuses it with "a function type is not implemented yet --
milestone 5", and there is nothing else in the language to pass. Generics on
top of that is what would make them one copy instead of one per element type.
The f32 family carries one caveat the i32 family cannot have: a NaN makes the
order undefined, because every comparison against one is false, so the
insertion loop never moves it and never moves anything past it. sum-f32
accumulates in f64 for a stronger version of sum-i32's argument -- an f32
total does not wrap, it absorbs, and the answer comes out silently short.
The test prints the difference rather than the total, because %g hides it.
sort-bytes! is the one a caller of split actually wants, and its ordering is
memcmp's: bytewise, unsigned, prefix first. Not alphabetical -- "Zebra" sorts
before "apple" -- and the note says so, for the same reason the ASCII-case
note refuses a locale. The slices move and the bytes never do, so it sorts
fields borrowed out of a string literal, which an in-place byte sort could
not.
f64->bytes is snprintf "%g": six significant digits, exponent notation of its
own accord, and no precision to pass it. A frame time of 1/60 comes back as
0.0166667 and a score past a million as 1.23457e+06. format-f64 returns a Vec
instead, so it inherits neither that nor the shared static scratch buffer --
and it is the reason append-i64! exists, because it renders the integer part
and the fraction through that one buffer in strict sequence.
Half away from zero at the last digit kept, which is round-f32's rule and not
printf's. 0.125 at two places is 0.13 here and 0.12 there; matching printf
would mean pinning a particular libc's nearest-even on the binary value, and
that answer is not the same on every target anyway.
The three cases that ship broken are each one line and each tested: the
carry, where the rounded fraction equals the scale and is the next integer
(0.999995 at five places prints "0.100000" without it); the zero padding,
without which 1.005 at three places prints "1.5"; and the sign, which belongs
to the number rather than to its integer part, since -0.5 has an integer part
of 0 and 0 carries no sign.
The clamp on the precision is spelled (min 9 (max 0 prec)) and not with the
clamp macro, and the reason is a finding: the prelude is never
macro-expanded. macro.ml's pass runs over the file being compiled, and the
prelude arrives at the checker through Check.program's own prepend, so a
prelude function calling a prelude macro resolves the macro's underlying
defn -- the one that takes a [Form] -- and reports an arity error.
Eight functions that the file used to refuse by name, and the refusal was
always one sentence -- there is no allocator -- which stopped being true when
Vec landed. Three rules hold across all of them and are written at the head
of the section: the result is owned and the caller frees it, the allocator is
the context's, and no signature carries a Result because no allocating
operation returns an error.
The builder is not a type. Odin's strings.Builder wraps a [dynamic]u8; here
the (Vec u8) already is that and already has push, so a wrapper would be a
move-only struct whose only method is the one it wraps. What was missing is
appending a run of bytes, and append! is that -- taking a (Ptr (Vec u8)),
because a Vec parameter moves and a by-value builder would be consumed by its
first append.
append-i64! and append-f64! are the argument for the whole shape. The
runtime renders numbers into one shared static buffer, so two of its results
cannot be held at once; these copy out before returning, so a builder holds as
many numbers as it likes. strings.flan puts two integers and a float on one
line to show it.
split returns a (Vec [u8]) and not a (Vec (Vec u8)): the fields borrow the
input, and the owning shape is refused outright because a Vec copies and
releases its elements bytewise. Constructing it needed a one-line slices-new,
because (vec-new) takes its element type as a bare symbol and [u8] is not
one -- a compiler gap, noted rather than worked around in silence.
replace-bytes guards its empty needle with an if and not an early return: a
returned Vec is a move, the dead set spans the function, and a return on one
branch would kill the binding on the other.
The merged exit closes stdout so the compiler reads EOF and learns the program
has finished. POSIX then hands descriptor 1 to the next thing that asks — a
socket, a module's object file — and the llc after that inherits it as its
stdout. /dev/null takes the slot back, and the EOF is unaffected because the
pipe's write end is genuinely gone.
Verified beyond the headless suite: sand.flan builds and runs merged under
Xvfb, one process with no children, and a game-draw typed in at the socket is
drawing frames a second later.
The two declares inherit the sin/cos caveat in full and not the sqrt one:
IEEE-754 requires nothing of atan2f or powf either, so they are the third and
fourth places in the prelude where native and wasm32 may differ in the last
bit. Every case in math2.flan is therefore a value that is exact in binary --
a quadrant boundary, a power of two, a perfect square -- rather than one that
would pin a particular libm and then fail on wasi.
clamp is the interesting one. The prelude already argued against wrapping
(min hi (max lo x)) in a function, and that argument gets stronger rather
than weaker: min and max are builtins at every numeric type and there are no
generics, so a clamp *function* is one copy per type. A macro is
type-agnostic for free and emits nothing at all. The test calls the same
three words at i32, i64, u8 and f32 to show it, and counts evaluations to
show that each argument appears once -- the shape that names x twice reads
identically and calls it twice.
lo above hi answers hi and is not checked. A macro has no error facility, so
the only diagnostic available would be a run-time one, in the construct whose
whole point is that it costs nothing at run time.
[4 T] is the type syntax and is unchanged; it already works in a defvar, a
parameter, a field and a return. A let binding is the one position with no
type slot, and there the brackets are an array literal of two elements whose
second is a type name — which came back as "unknown name rl/Vector2" and cost
32 hand-written Vector2s in one raylib example.
(array COUNT TYPE) is a parser form rather than a builtin call, because the
second argument is a type and the parser's callers have none. Parse assembles
the Tarray itself, so the count takes a constant's name for free and a value
in the type position is refused by the type reader's own message. The checker
resolves it to Tast.Zero — no new backend node and no new type.
(zeroed [4 T]) was proposed first and rejected: the parser can tell, a person
cannot. zeroed keeps its job of being inferred; array is the one that is told.
`flan dev` now builds one binary that is the compiled Flan program and holds
the whole OCaml compiler, and execs it. The program keeps main() — macOS needs
the window there — and caml_startup happens on a pthread beside it, next to the
listener flan_agent.c already starts. The editor's socket and the wire protocol
are untouched: Emacs cannot tell the difference.
Two rules are written into lib/dev.ml rather than discovered later. The game
thread must never call into OCaml, because a native thread has no safe points
and so can never be stopped by the collector — which is exactly why a frame is
never paused, and exactly what one convenient direct call would undo. And no
OCaml value may be stored in Flan memory without caml_register_global_root,
which is the way the spike's "the GC does not touch the arenas" measurement
stops being true.
The link is spelled in dev.ml out of Build's existing public pieces rather than
as a mode of Build.executable: lib/build.ml belongs to another lane this week.
It should collapse into Build once that lands.
A Flan main does not return — Emit ends it with flan_exit and an unreachable —
so in one process that call would take the compiler down with a program that
merely finished. flan_rt.c grows a hook, null in every other build, that the
merged entry point uses to flush, close stdout and park. The compiler then
learns the program is done the same way the daemon did: the pipe reads EOF.
--two-process keeps the old shape for a machine that cannot build the compiler
object, and nothing has been deleted.
The expander collects defmacros from the prelude and from the file being
compiled. Not from an imported package, and the reason is an ordering one:
Load learns a package's imports by parsing it, so reaching a package's macros
would mean resolving that package's own imports over Forms, before Load runs.
That is a second import resolver, and it is a bigger thing than this lane.
Refused by name, which is the rule that caught the two misparse bugs. Left
alone the call arrives at the checker as an unknown name -- true, and no help.
Refused where the defmacro is written rather than where it is called, because
that is where the fix goes.
The check has to sit in Load's read, because that is the only place that can
see one: by the time Parse is finished a defmacro is an ordinary Ast.Defn and
the word is gone.
Measured while here, since a prelude that grows a defmacro is a cost every
program pays or does not:
- A build of a program that names no macro: 50ms, the same as before. The
pass scans the top level, finds nothing, and no compiler runs.
- A program that calls one: 310ms the first time, 70ms after. The 240ms is
the clang driver building the macro module; it is cached under the object
cache, keyed by the prelude's source and the file's defmacros, so it is
paid once per change rather than once per build.
- A hello-world's binary carries exactly one symbol out of all of this:
flan.gensym-n, eight bytes. Reach.link drops unless, form-cons, form-nil,
form-append, form-rest and gensym, because nothing reachable calls them.
plan.org milestone 5 says when, unless, until, cond and dotimes are special
forms only until macros land. This is the first one to stop being one, and
running test/programs/macro-unless.flan means the compiler built a shared
object, dlopened it into itself and called a Flan function to find out what
(unless c a b) means.
unless is the one that moved because it is the one nothing else needs: zero
uses in the prelude, so moving it cannot make the prelude depend on the
expander that compiles it. Its coverage is sand.flan, seven calls, compiled
through Session in test_session -- which is the in-process path and the reason
lib/dune now passes -linkall. Say plainly what that coverage is not: nothing
in test/programs used unless before today, so macro-unless.flan is a test
written after the feature. The corpus that was written before it is sand.flan
and web/examples/control.flan, and both compile unchanged.
lib/macro.ml is the half of expansion that has to compile something. Expand is
the image format and the quasiquote desugaring and depends on nothing above
Form; this needs Check, Build and Emit, so it sits above the parser it feeds
and arrives through Parse.expander.
What it does, in order:
- Collects every defmacro from the prelude and from the file. Not from an
imported package: Load learns a package's imports by parsing it, so
collecting from one means a second import resolver over Forms, and that is a
bigger thing than this.
- Builds them in rounds, because a macro's body may call a macro and a body
with an unexpanded call in it will not compile at all -- the call is a name
nothing defines. Round 0 takes every macro that names no macro still
waiting; round 1 expands the rest against round 0's module. A round that
takes nothing while macros remain is a ring and is named. macros.flan has
the round-1 case and macro-cycle.flan has the ring, and the distinction
between them is the one thing here that is easy to get wrong: a call inside
a quasiquote is *not* a compile-order dependency. It is part of what the
macro answers, and the answer is expanded again after it returns. The first
macro-cycle.flan written for this commit quasiquoted, and it was not a cycle
at all -- it hit the fuel instead, correctly.
- Walks bottom up, so a macro never sees a call to another macro in what it is
handed, and re-expands what comes back, so a macro that expands into a call
to itself keeps going. That loop is bounded at 200 and says which macro ran
out: macro-spin.flan.
- Skips all of it when the file names no macro, which is nearly every file.
Otherwise every build in the suite would pay a clang driver to answer a
question nobody asked. When it does build, the module is cached under the
object cache and keyed by the prelude's source plus the file's defmacros, so
a second process pays a dlopen.
lib/dune passes -linkall, which is the one line in another lane's file. The
module installs itself into Parse.expander at initialisation and nothing
references it, so without -linkall the linker drops it from every executable
that does not name the module -- bin/main.exe among them -- and a program
calling a macro fails with an unknown name. The alternative was an install
call at every entry point, including ones in files this lane must not touch.
The one thing a macro cannot do that parse.ml could is give a reason. A macro
runs inside the compiler and anything it signals aborts the compile with no
location, so a malformed (unless) answers a name nothing defines and the
report is "unknown name unless-takes-a-test-and-a-body" at the call site --
right place, wrong sentence. NEXT.md says so.
test_flan.ml's "unless -> if(not)" assertion is gone, because it asserted a
desugaring in a file that no longer does one. Nothing else in the suite
changed.
Two things that look like plumbing and are the frontend half of expansion.
A quasiquote becomes calls to the prelude's three form-building functions and
nothing else: form-nil, form-cons for an item, form-append for a splice. It is
pure, it needs nothing loaded, and it runs over every form on the way into
Parse.program and Parse.decl, which is what lets the prelude's own macros parse
in a process that has not built a macro module yet.
Running it *before* the expander's walk is not an ordering preference. A cond
macro's body contains a quasiquoted (cond ...) for its own tail; with the
quasiquote still standing, the walk would see that head and expand it then and
there, against the wrong arguments. Desugared first, that subform is a
(Form.Sym {.s "cond"}) and there is no head left to mistake. So the walk needs
no idea that quoting exists, which is the whole reason this runs first.
Nesting levels are not counted -- not by the reader, which was written that way
deliberately, and not here. A quasiquote inside a quasiquote is refused by
name. Only a macro that writes a macro wants one, nothing in the corpus does,
and CL's level arithmetic costs more than the use case is worth so far.
A defmacro is now an Ast.Defn: (defmacro m [args] body) is
(defn m [args [Form]] Form body). There is no Ast.Defmacro and there is not
going to be one -- a macro is [Form] -> Form, compiled by the same backend as
everything else, and the only thing that makes it a macro is that the expander
calls it at compile time. One parameter, the slice of forms at the call site,
so variadics come free in a language with no &rest; two parameters is a
misunderstanding rather than an arity error and says so.
Parse.expander is the hook the walk arrives through, because expanding a macro
means compiling and dlopening it, so the expander sits above Check and Build
and Parse sits below them. Nothing fills it in yet.
The quasiquote refusal stays as a backstop: it now means a form reached the
parser without coming through program or decl. gensym's refusal is gone -- it
is an ordinary prelude function returning a Form, and a macro body calls it
like any other.
The last commit put the prelude's types into the set the parser uses to tell a
return type from the first form of a body, and put them in plainly. That set is
read by two arms: a bare symbol, and a list head. The list-head arm is why
(defn f [] (Some 1) (bar)) does not lose its body, and the comment above it has
warned about this since it was written -- so adding Rune plainly made
(defn f [] (Rune {.code 65}) (bar)) a function returning a Rune with a
one-form body, silently, in every file in the language. Confirmed before
fixing: it failed with "a map type is {K V}", which is the misparse arriving a
step later wearing someone else's error.
The enums already solve this one comment up, under their own key, for the same
reason. The prelude's types go in the same way. No prelude type takes
arguments, so a bare symbol is the only type position any of them can occupy.
Both halves are pinned in test_flan.ml's return-type section: Form is a return
type, and a prelude struct literal opening a body is not.
The capability lists were written before the code held the line they claim.
Under an expression root, RET on a field of a union built `(.at s)' and sent
it, and the checker refused it — "a union's fields belong to a case ... they
are reached by (match ...)". A refusal from the far end of a socket is exactly
what this buffer's own comment says not to do: every refusal is by name, here,
with the reason, because RET working on some lines and erroring on others
teaches nothing about the language.
It is a refusal of the *parent* and not of the value at point, which is why it
is not in `flan-inspect-refusal': a struct field that merely holds a union is
an ordinary accessor and has to stay enterable. It is a field of the union
itself that cannot be written. The two cases are one test each.
The slot root steps into it by offset and is unaffected, which is the
difference the manual now claims and the tests now show.
`lib/dev.ml' cited DISCUSS.md item 1 as a hole; item 1 is the answer now, so
it cites BUILT.md instead. And the item 1 stub is two sentences and a pointer
— everything else in it is in BUILT.md verbatim, and DISCUSS.md's own header
says nothing in it is a decision.
The boundary was verified by compilation and had never executed. Now it does:
three Flan functions compiled into a .so, dlopened into the test process, and
called with Forms this side laid out in raw memory.
lib/expand.ml is the image format and nothing else yet. A Form is 24 bytes,
align 8, payload at offset 8, and every case holds one member at the payload's
start -- a string and a slice are both { ptr, i64 }, so there is no third
offset anywhere in it. The tag is the case's position in the prelude's
defunion, which is why that list says it is a layout contract; a tag this file
and the prelude disagree about is named rather than read as some other case.
The case sends one Form of every one of the nine shapes through an identity
macro, so a tag nobody thought about is a failure and not a gap. Then two
arguments through a macro that reads the second, because a slice whose length
did not cross reads past its arguments and an identity macro would not notice.
Then a Form the *macro* allocated, through the prelude's form-cons, on the
loaded module's own heap: that is the direction nothing had ever tested, and
it is the one the expander spends all its time in.
Checked by breaking the last expectation before restoring it.
A parser bug this turned up, and it is the reason the previous lane's Form
work could not have been finished as written: is_type_form decides whether a
leading form is a return type or the first form of the body by asking whether
its name is a declared type, and the set it asks was collected from the file's
own declarations only. Check.program prepends the prelude to every program, so
the prelude's types are every file's types -- but nothing told the parser that.
It never mattered while the prelude's structs were only taken as parameters.
A macro is (defn m [args [Form]] Form ...), and bare Form in return position
was parsed as a body expression and reported as an unknown name, while [[Form]]
worked, because a Vec in that position is a type whatever is inside it. The
prelude's types are now part of the base set, read once.
`i' on a local sent the local's *name* to be evaluated, and an expression is
evaluated where the evaluator stands. That is the right frame only when the
frame is the innermost one; on any other it may resolve to a global, to
another binding of the same name, or to nothing, with the locals listing right
above it showing the frame's own storage and nothing saying the two disagree.
The daemon verb for the fix landed already. What was missing was the state
layer under it: `flan-inspect--expr' held a bare expression, so there was
nowhere to put a frame. It is `flan-inspect--root' and `flan-inspect--path'
now — `(:expr E)' or `(:slot FRAME SLOT NAME)', plus the steps walked from it
— and a stack entry is `(ROOT PATH . POINT)'. RET appends a step, `l' restores
a pair it pushed. Every step is still a fresh request, so the view is never
stale.
`l' cannot cross between the two roots, and that is structural rather than a
rule someone has to keep: RET only ever extends the path under the root the
buffer already has, and `flan-inspect' and `flan-inspect-slot' both start with
an empty stack, so a mixed stack cannot be built at all. It stays true if a
third rooting mode is added.
The break buffer hands over the frame and the slot *index*, which is the
fourth element `locals' now puts on each line. A name does not identify a
slot: two slots of one frame can share one, and a refused slot is not in the
listing, so its position is not an identifier either. A global still goes in
by name, because a global's name really is an expression that means the same
thing wherever it is evaluated — the loaded thunk binds to the program's own
storage through the dynamic linker.
Two smaller things the wire needed. A field step carries the type it was read
out of, because a union's payload is at an offset that depends on the case and
only the renderer knows which case the value is in — so `Union.case.field',
which is the head the renderer wrote with the field appended. And an empty
path is sent by omission: Emacs prints an empty list as `nil', which is a
symbol on the wire, so the daemon now reads that as no path rather than
refusing it as a step.
The prelude's own (vec-new Form) was refused with "nothing here says what
(vec-new) is a Vec of" -- a message about a missing annotation, to a program
that had written one. The build went red the moment the Form declaration was
checked against anything, which is why the front half landed unmeasured.
The test a leading bare symbol has to pass was spelled out twice, once in
vec_new_elem and once in map_new_types, and both lists were written before
unions existed: primitives, structs, enums, aliases. resolve_name has known
about unions since they landed, so the two halves disagreed about what a type
name is. Now there is one list, read by both, so the next kind of type cannot
be added to one of them.
The case is in unions.flan rather than in a file of its own, because what it
asserts is that a union is an element type like any other -- (vec-new Shape),
(map-new string Shape) -- and that is a sentence about unions.
Also drops forms.so, a build artefact the last lane committed.
`i' in the break buffer sent a local's *name* to be evaluated, and an
expression is evaluated where the evaluator stands. On the innermost frame
that is the right frame; on any other it may resolve to a global, to another
binding of the same name, or to nothing, with the listing above it showing the
frame's own storage and nothing saying the two disagree.
The shadow stack is what makes the second rooting mode cheap: a frame's
address and every slot's type are both here, so a step into a field is an
address plus an offset with that field's type — the arithmetic
`Render.render' already does for the listing. `Session.render_slot' is
`render_locals' with a path applied to the root and one line out.
The slot travels by *index*, because a name is not unique: two `v's is two
slots and both are in the listing, and a refused slot is not, so the position
in the list is not an identifier either. So `locals' now puts the index on
each line.
The frame checks are `locals'' by construction — `stopped_frame' is one
function now, and an inspector with its own copy would be free to read a frame
whose body was redefined since it was entered. The build-and-read tail is one
function too, for the reason this file already records about the fingerprint.
And `layout' said union values were milestone 6, which they have not been
since today.
Running a macro means compiling it and loading it into the compiler, and the
step that reads as small in NEXT.md is not: OCaml has no dlopen for ELF, and
lib/dune had no foreign_stubs. So the boundary is built first and the expander
not at all. lib/dynload_stubs.c is the whole of it — dlopen, dlsym, a
four-argument call into a macro thunk, and a peek/poke family, because OCaml
cannot address the raw memory a Form image has to be laid out in.
Nothing aggregate crosses to C. The unions lane verified a union's memory
layout against clang, which is a different claim from LLVM's convention for an
aggregate passed or returned by value in hand-written IR, so Emit.macro_thunk
wraps every macro in void(ptr,i64,ptr,ptr): the slice is built and the result
stored on the LLVM side, and the compiler's side is four pointers.
Build.macro_module links the runtime in rather than declaring it external, so
the module has no undefined symbols and the compiler's own link needs no
-rdynamic. That is the difference from Build.shared, whose host is a running
Flan program.
defunion Form and the list-building surface quasiquote will desugar into are in
the prelude. Form mirrors Form.value and not Form.t: no loc field, so the
compiler stamps the call site's location onto everything a macro returns.
The compiler builds. dune test was not run, and Form's layout is asserted
nowhere — NEXT.md's new handoff section says what the three numbers are, what
the next two commits should be, and the four decisions this made that the
design did not settle.
The globals section attributed a frame by its slot fingerprint, which is the
wrong cut for it: a redefined body can name entirely different globals while
binding identical locals, so the check saw no change and the new body's
reference set went into the union under the old body's frame, with the frame
numbers beside an entry saying so.
So a second fingerprint. Reach.ref_fingerprint hashes the set of globals a body
names — sorted and deduplicated, because a reference set is not ordered, where
slot indices make the slot fingerprint order-sensitive on purpose — and it
travels the path the first one already cut: %fninfo, flan_dev_frame_refsig, the
agent's snapshot, the backtrace line, Dev.globals_op. Different means the frame
is skipped by name with its reason, and the rest of the stack still contributes.
Two numbers rather than one, because they are two facts. A frame whose slots
match and whose globals do not has locals that are perfectly readable and
attribution that is not, and a combined hash would make locals refuse a frame
with nothing wrong with it. locals still checks the slot fingerprint alone.
It lives in reach.ml because expr_refs is already the walk that answers what a
body refers to, and is the walk the union itself is built from. One consequence:
emit now reaches reach, which closes a cycle through Load if cimport calls
Build.cachedir, so the header cache spells the object cache directory itself.
test_dev.ml drives the exact case — a body that binds identical locals and names
untouched where the stopped frame names pressure. With the check disabled it
fails twice: the missing refusal, and untouched appearing under frame 0.
Everywhere else uninit is an opt-out from ZII and the bytes are whatever they
were: a garbage f64 is a garbage number. A union is the one type where that
is qualitatively worse. Its tag steers control flow, a tag no case names falls
past every comparison in a match, and the block after those comparisons is
unreachable -- which LLVM is entitled to assume cannot happen. So the one
place where garbage becomes "the optimiser may do anything" is refused by
name, with the zeroed form, which is a real case, named beside it.
(.x u) on a union said "Shape is not a struct, so it has no fields", which
is true and unhelpful. A union's fields belong to a case and which case is
being held is what the tag says, so they are reached by match, whose arms bind
the fields of the case they matched. The message says that.
print, the REPL inspector and the break buffer's locals all walk a concrete
type through render.ml, and a union fell through its Named arm to <Shape>.
It now recovers the case from the tag by a chain of comparisons -- the same
shape the enum arm already had, and for the same reason: the name is erased
before any backend sees it -- and reads the fields of that case only. Reading
the others would be reading a payload that is not there.
It prints (Shape.Dot {.x 1.5 .y -2.5}), which is what the source would write.
The union table has to reach the walk, so Render.ctx grew a field and its
three construction sites in session.ml and one in check.ml pass it. That is
the whole of the session.ml change.
test/programs/unions.flan is the program: a case with no fields, a case wider
than another, a case holding a string, a union in a struct, a union through a
call in both directions, ZII, reassignment, and printing. Its layout was
checked against clang's for the same declaration -- 32 bytes aligned 8 with
the payload at offset 8, and 40/8 for the struct holding it.
The 15.5ms attributed to re-reading the header on every reload is not that.
A timer around each stage says the cached dump reads in 0.33ms, the extraction
takes 3.3ms and the checks 0.55ms — about 4ms, once, in Session.create. The
rest of flan reload's delta is Load and Check over 256 more declarations, and
the +3.6ms a redefinition really pays is Check and Emit.redefinition against a
bigger program. A C-c C-c reads no header at all: eval's forms carry no import,
so no package is read.
Both cache levels anyway, because a long-lived process should pay nothing
twice. In the session, two tables: the dump by header, the declarations by
header and by what the package already declares. On disk, the existing cache
moved into the object cache directory beside the .o files. The in-memory key
is the path and the flags with no mtime, so a header edited mid-session is not
picked up until the session restarts — the rule a changed .c file follows, and
the rule that keeps new signatures from being checked against a process still
running the old layouts.
Measured: repeat import 3.65ms to nothing; flan reload unchanged, as it must
be, since it imports once per process.
defunion parsed and its shape checked; naming the type and constructing a
value were both refused as milestone 6. They are not any more.
A union is Types.Named, exactly as a struct is, so every path that carries a
type -- a field, a parameter, a slot, a copy -- learns nothing about unions.
Which table the name is in is the only thing that tells the two apart.
The layout is a tag then room for the largest case, with the alignment the
widest member of any case needs: %"U" = type { i32, [k x iA] }, and one
named %"U.C" per case laid over the blob. That is C's
struct { int tag; union { ... } u; } byte for byte, which is the requirement
the macro expander's Form will arrive with.
A value is (U.C {.field value ...}), or U.C on its own when the case has no
fields. Construction goes through the struct-literal syntax already there, so
parse.ml is untouched: the dot is a symbol constituent and U.C reads as one
name.
Tags are declaration order from zero, so an all-bytes-zero union is the first
declared case with a zeroed payload -- the same rule that makes an Option's
zero a None, and it makes case order part of a union's contract.
A move-only field in a case is refused in the same words a struct's is, and a
union is refused as a map key: the payload past the case in hand is
indeterminate, so hashing the blob would make two equal values hash
differently.
Loading a package kept one table, keyed by real path, and used it for two
different questions. Already loaded meant "skip", which is right for the second
route of a diamond and wrong for a ring: a package that imported itself round a
chain met its own entry, contributed nothing, and appeared to work. The comment
said so and called it a feature.
It is not one. A ring has no package order, and a definite package order is what
the macro expander needs — every defmacro has to be compiled before anything
that calls it. So the chain currently being read is now carried separately from
the set already finished. A directory found in the first is a cycle and is
refused; a directory found only in the second is still the diamond's second
route and still a no-op.
The refusal names the ring — a -> b -> c -> a — and only the ring, not the route
that led to it. "There is a cycle" leaves the reader to find which three imports
it was.
pkgs now comes back dependencies-first, which is the topological order the
acyclic rule buys. The declaration list is left alone: check.ml collects every
top-level name before it checks any body, so declarations are order-independent
by construction and sorting them would be churn in the field every test reads.
The tests are a real tree rather than a second copy of pkg-shared. pkg-diamond
builds a shape/Box inside area/ and hands it to a function declared inside
draw/, which only type-checks if the bottom package was read once — two copies
of one struct are two types. What proves it is the numbers, not the compile.
A global is program state a frame happened to touch, not part of it, so
nesting it under one implies an ownership that is not there and repeats the
name once per frame that reads it. One section instead, holding the union of
the globals every frame on the stack references — the compiler does the
choosing, since Reach.expr_refs already answers a body's reference set, and
listing every global a program has would bury the one that matters under the
prelude's PRNG state.
Each entry says which frames touch it, by the index the stack section already
numbers them with, which recovers what per-frame nesting would have told you
at no cost in duplication. Ordered by the innermost frame that touches it:
a deep stack makes the union large and proximity to the error is what puts
the likely culprit on top.
Simpler than locals, because a global is reached by name rather than by
address. Emit.redefinition writes a global the host has as external, so the
thunk binds to the program's own storage and nothing is asked of the stopped
thread — no dev-slot round trip and no not-yet-bound case to refuse.
A frame that cannot be attributed contributes nothing and is named in
:skipped; the union being incomplete and the union being complete are
different answers. The hole in that is stated rather than papered over:
slot_fingerprint hashes a body's slots, which is the right cut for locals and
not for this, so a body that names different globals while binding the same
locals is not caught. The test drives the case that is.
MANUAL.md also loses a stale paragraph claiming the fingerprint check never
fires with a failing test pinned to it. It fires, and test_dev covers it.
render.ml's output and emacs/flan-inspect.el's parser are the two ends of one
wire format, which is why the printer was left on the colon when the rest of
the corpus moved: shifting it alone would have broken inspection in the dev
loop without breaking a test that said so. They move together here.
The field list in the inspector is labelled with the dot too, which is the
spelling flan-inspect-step-expr already used to build `(.x b)' — the label and
the expression it stands for now read the same.
One case needed a guard the colon never did: `...' also begins with a dot and
is the renderer saying it stopped, not a field called `..'. A field name never
starts with a second dot, so one character of lookahead separates them.
The colon is not gone from the rendered grammar. An enum member is `:green' and
is a *value*, so the two are now told apart by the character alone, which is
the only thing that distinguishes them.
Also font lock, handed over with the same change: `:name' was the rule that
drew field labels, and with the colon belonging to keywords every label in the
corpus was left unfontified. `.name' is drawn as a constant, in both the places
it appears — the label in `{.x 1.0}' and the accessor in `(.x v)', which are
the same name.
BUILT.md gains "The header is read now", directly under the section whose last
paragraph promised that reading a header was what would convert the trusted
half into a checked one and that it was not built. That sentence is replaced by
a pointer to the one below it, in BUILT.md and in shim.ml's docstring both.
It records the things worth not re-deriving: why the dump and not libclang (and
that Zig left libclang too, which strengthens the argument rather than weakening
it), why the import is bounded by the package's own defstructs, why generating
defstructs would make the check circular in exactly the way a _Static_assert
was rejected for, refusal-by-demotion from Zig's failDecl, the naming rule and
what it must actually guarantee, and both const-vs-non-const char * and the
target-varying widths.
The diff and the costs are stated as measurements, with the table: 16 of 16
defstructs and 172 of 172 declare-c agree against 5.5, ten real differences
against 5.1-dev, release +4ms warm, redefinition 31.0 -> 46.5ms.
DISCUSS.md item 6 is rewritten rather than removed. The mechanism question is
settled and is now in BUILT.md; what is left is narrower and is two decisions
that are the author's — whether the header stays a build-time read or becomes a
committed generator, and whether the 172 hand-written lines migrate. Both have
the argument on each side written out, including what migration would lose:
key-pressed? is a better name than is-key-pressed, and an enum parameter
imports as i32 because nothing tells the importer the package calls KeyboardKey
"Key".
Clojure's spelling and Clojure's semantics. Repeated — #_#_ a b c — discards
that many following forms, and that falls out of the recursion rather than
being counted: the discard reads *a form*, and the form it reads may itself
begin with a discard, so the outer one throws away what the inner one already
stepped past.
It belongs to read_form rather than to the sequence readers, which is what makes
it work in every position a form can appear — top level, inside a list or a
vector or a map, before or after a quote. The two loops that look for a closer
or for end of input skip it as well, because a discard is not an element and a
file ending in one has read everything there is to read.
A trailing #_ with nothing after it is an error, and it is the same error an
unterminated form already gives.
Two C functions whose names kebab to one Flan name used to resolve by order:
the first won the name, the second was refused. Which one that is depends on
the order the header happens to declare them in, so moving two lines in
somebody else's header would silently rebind a name a Flan program is already
calling — and the winner was left in the hidden list too, so using the name it
did get reported that it could not be had.
Neither takes it now. There is no reading of spin-2d that is obviously right
when the header offers both Spin2D and spin2d, so both are refused and both say
why; the author binds the one they want with a hand-written declare-c, which is
what that form is for. Found by test/headers/sample.h, which is why it is a
fixture rather than a raylib case.
raylib is unaffected: its 581 names are injective under the rule.
Reading the header produced declarations and nothing else, so the gap the whole
thing exists to close — that nothing verifies a declaration against the library
— was closed by a command somebody could run rather than by a property the
build had. Now `import` runs both comparisons whenever a header resolves.
Build-stopping, not a note. The package named the header, so the header is the
package's own claim about what it binds; a defstruct that disagrees lays fields
out in the wrong order and reads as five plausible numbers rather than as a
link error. Continuing past a known-wrong layout to produce a program that will
read garbage is the shape the house rule against swallowing things exists to
prevent. Both messages point at the line in raylib.flan, not at the header.
Verified by breaking it on purpose: a permuted Texture2D stops the build naming
the field that moved, and `f64` where raylib says `float` stops it naming the
parameter — which is the hazard BUILT.md calls out by name and says only a test
can catch.
A set-but-wrong FLAN_RAYLIB_H used to be indistinguishable from not opting in:
the line was skipped and nothing was said. Unset still means off and silent; a
path that is not there is now an error naming it. That is the difference
between an opt-in and a trap.
test/headers/sample.h is one function per decision the importer makes. The
raylib case needs raylib installed, at the right version, with a variable set,
so it would skip everywhere and cover nothing; this one does not move. It also
found a bug, fixed next.
Reach still prunes with 256 extra declarations in play: a wasm32-wasi build of
a program that imports raylib and calls none of it links without libraylib,
which is the case Reach.link exists for.
`headers` beside `link`, read the same way: a path, any clang flags that header
needs, ${NAME} expanded from the environment. What comes back is ordinary
declare-c declarations, generated before the package's names are qualified, so
they arrive as rl/… exactly like the hand-written ones and nothing downstream
can tell which is which. No new form, no new decl_kind, no reader or parser
change.
A leading `?` makes a line optional. vendor/raylib uses it, because "a build
needs libraylib linkable and not raylib-devel installed" is a property worth
keeping — requiring a header would take it from everyone to give the check to
whoever has one. Unset FLAN_RAYLIB_H and the build is exactly what it was; set
it and every signature is checked against raylib's own header.
A C symbol the package already binds by hand is left alone, so declare-c
remains the escape hatch and stays the thing that wins. A refused function
becomes a hidden name through Load.refuse_hidden, so writing rl/get-gamepad-name
says "GetGamepadName returns char *, and a string only crosses as a parameter"
rather than "unknown name".
Measured, because the cost is the whole argument for how much to import:
release build +14ms cold, +4ms warm — Reach prunes the wrappers
redefinition 31ms -> 46.5ms
dev build +333ms cold — dev does not prune, 428 wrappers
Reach.link already drops a generated wrapper whose declaration nothing
reachable calls, and that is what makes a wholesale import cost nothing in a
release build. It does not prune dev builds, on purpose, so a dev build
compiles every wrapper once at session start; Build.shared compiles no C, so
redefinition does not pay that again.
Reading the header is cached — 64ms of a 72ms check, against 8ms for the whole
program without it. Keyed like the object cache, on everything that could
change the answer: the header's path, size and mtime, the full flag list, and a
format version, since the cached value is a marshalled dump. The extracted
signatures are cached rather than clang's JSON, because the parse is half the
cost. That takes the delta to 17ms.
Verified end to end and headless, using only imported declarations:
ColorToInt of {17,34,51,68} is 0x11223344 and ColorTint hands the four bytes
back separately, so field order is pinned by arithmetic rather than by a
round trip. TextLength of "hello" is 5, so the string crossing works.
declare-c generates the wrapper, the typedefs and the prototype from one
declaration, so they cannot disagree with each other. What nothing checked was
whether the declaration matched the library — BUILT.md records that as trusted
rather than guaranteed, because no header was ever read.
This reads one. clang is asked for a JSON AST dump of the header and shelled
out to, not linked: -Xclang -ast-dump=json is the same binary on PATH that
every build already runs, which is plan.org's "Why LLVM IR as text" applied a
second time. Zig's old @cImport linked clang as a library and that is precisely
the dependency plan.org rejected.
cjson.ml is enough JSON to read the dump and no more, so this adds no opam
package to parse it.
What comes out of the header is signatures and nothing else — not structs, not
enums, not macros. The bound on how much is imported is the package's own
defstructs: a function whose signature mentions a struct the package has not
described is refused with that reason, so vendor/raylib describing thirteen
structs is what makes the import thirteen structs wide. Keeping the layouts
hand-written is also what makes checking them against the header's records
worth doing — a _Static_assert was rejected in BUILT.md as circular, and this
is not, because the two sides have different authors.
Refusals are demotions, taken from Zig's translator: it never drops a
declaration it cannot handle, it binds the name to a @compileError carrying the
reason so the failure lands at the use site. Load.refuse_hidden is already that
mechanism. So a returned char * does not kill the header — it makes one name
unavailable, with the reason attached.
flan import-c prints what it would produce, what it refused, how the package's
defstructs compare with the header's records, and how the hand-written
declare-c lines compare with the header's signatures.
Against raylib 5.5, the version whose .so vendor/raylib/link names: all 16
defstructs and all 172 hand-written declare-c agree exactly. Against the 5.1-dev
header installed in /usr/local it reports ten differences, nine functions that
version does not have and one that gained a parameter — so the check has teeth
and the clean run is not a vacuous one.
test/programs/maps.flan is seven claims over the Map, each one a
plausible wrong version gets wrong, with the numbers differing per
failure so a single wrong answer names its own cause: an integer key
past eight grows, a struct key whose padding must never be hashed, a
struct key holding a string, an enum key, clone's independence, upsert
not growing the length, and a map living in an arena.
The move refusal said "a Vec is move-only" whatever had been moved, so
moving a Map was reported as a fact about Vecs. It names the type now.
The checker half. {K V} and (Map K V) resolve, and map-new, put, get,
has-key?, len, reserve, clone and free are named calls over the
type-erased runtime, with the two sizes and the key's hash and equality
pair produced at the site because the site is where the concrete types
are known. len, reserve, clone and free were extended rather than given
map-shaped names of their own, which is what at and len already did for
Vec: one question, one word.
The key's pair is resolved per key type and mostly is not emitted at
all. Every integer, enum, bool and fixed array of those is compared
bytewise and served by one runtime pair over (pointer, size). A string
is not, because its bytes are elsewhere and two equal strings at
different addresses must hash alike. A struct is not, because its
padding bytes are indeterminate — two structs equal field by field can
differ bytewise — and because it may hold a string. So a struct gets a
pair emitted for it, walking its fields in declaration order and
addressing nothing but fields, and that is the only case that does. Two
maps with the same key type share one pair, and a struct reached twice
through two fields emits one.
get returns (Option V) and builds it here rather than in the runtime,
which has no idea what an Option's layout is — keeping it that way is
what lets one entry point serve every value type. put is upsert
returning Unit. Both bind their arguments to slots before the guard, so
a retry re-attempts the allocation and not the expressions that produced
the key and the value.
Refusals, each by name: a float key has no usable equality at all, which
is not a milestone question; a Ptr, slice, Vec or Map key would hash an
address rather than what it points at; a move-only value would have its
header duplicated by clone, which is the refusal (Vec (Vec T)) already
carries; Unit as a value has no bytes to store, and it is the natural
spelling of a set, so it is refused by name rather than by dividing a
cache line by zero.
Work in progress: it builds and the runtime is exercised and green, but
no Flan program can reach it yet — the checker half is not written, so
(Map K V) is still refused where it is resolved.
runtime/flan_rt.c is Odin's map, followed deliberately: open-addressed
Robin Hood hashing at a 75% load factor, cache-line cell packing so no
key or value straddles a line, and the probe loop kept to pointer-width
integers. One type-erased runtime over (key size, value size) plus a
hash and equality pair, the same arrangement the Vec runtime has over
(size, align).
Two departures from Odin, both deliberate and both commented where they
are made. There are no tombstones, because removal is deferred by
spec-memory.md, and that deletes the backward-shift loop entirely — it is
the single largest reason this is shorter than the original. And the
header does not stuff log2cap into the low bits of the data pointer:
Odin does that because Raw_Map must be three words, whereas this header
already carries an allocator, a generation and an epoch, so the tagging
would buy nothing, cost a mask on every access, and make correctness
depend on the block being 64-byte aligned rather than merely faster
when it is.
The scaffolding around it: a Map is 48 bytes and six words like a Vec,
it crosses to the runtime by address because it is move-only and must be
mutated in place, and it has a DWARF type showing all six fields.
Tast.FnAddr is new — the address of a function, either one this compiler
emitted or a runtime C symbol. It is not a function value: nothing in
the surface language can produce one, name its type or call through it.
Odin's Map_Info reaches its hash and equality pair exactly this way.
reach.ml learns that edge, because a function reached only by address is
invisible to the reachability walk otherwise, which is the same hazard
handler-bind clauses already had.
The hash and equality pair carries the transfer channel as its last
parameter, because a pair emitted for a struct key is an ordinary Flan
function and every Flan function's signature ends with one.
