JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

The expander is written down where the next person will look

Browse Source 
BUILT.md gains "Macros: the compiler dlopens the program": the image format and
why nothing aggregate crosses to C, quasiquote before the walk and why that is
load-bearing, the distinction between a quasiquoted call and a real one, the
two different non-termination failures, -linkall and the session-path argument
that forced it, the three cost numbers, and what unless proves and what it does
not.

"Why there is no interpreter" gains its consequence: compile-and-dlopen is a
mechanism now rather than an absence.

NEXT.md loses the whole expander design and its handoff, which are done, and
keeps six things that are not: four special forms left, with why when and
dotimes are the hard two -- the prelude uses them 29 and 12 times, so moving
either makes the prelude depend on the macro the macro module has to compile
the prelude to get; that a macro has no way to say why something is wrong,
which is the biggest gap and the reason unless went before cond; macros not
imported; a prelude macro not being able to call a macro; nested quasiquote;
and gensym's counter per module.

One claim corrected rather than left standing: the packages section said the
topological package order exists for the expander. The expander does not read
it, because macros are not imported. The order is right and correct and
nothing uses it yet.
This commit is contained in:
Joseph Ferano 2026-09-12 21:07:06 +07:00
parent 545ef6e0ea
commit 64d4993fee
2 changed files with 225 additions and 134 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

156
BUILT.md
View File

@ -565,6 +565,10 @@ perceptually instant for expression eval too. Milestone 3 did not need an oracle
hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second"
criterion is dropped, and the host ABI moved onto the critical path.
It stopped being an absence when macros landed. A macro has to run at compile time and there is nothing to interpret
it with, so the compiler compiles it into a shared object and `dlopen`s it into its own process — see "Macros: the
compiler dlopens the program". The decision's cost and its mechanism are the same thing.
## The layout, which is the whole backend design
```
@ -2279,6 +2283,158 @@ there and matching it from a program. `dev.ml`'s inspector still says "union val
frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards
is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates.
## Macros: the compiler dlopens the program
"Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing
that turns that decision into a mechanism rather than an absence: **running a macro at compile time means compiling
it and loading it into the compiler's own process.** There is nothing to interpret it with and there is not going to
be, so `Emit.redefinition` → `Build.shared` → `dlopen`, the reload primitive the dev loop already runs, is pointed at
the compiler instead of at a running program.
`(defmacro name [args] body ...)` is one function, `[Form] -> Form`. One parameter, the slice of forms written at the
call site, which is where variadics come from in a language with no `&rest`: `(len args)` is how many were written.
### A defmacro is a defn, and there is no Ast.Defmacro
`Parse` turns `(defmacro m [args] body)` into `(defn m [args [Form]] Form body)` and nothing below the parser knows
the word exists. The checker checks it like any function, the backend emits it like any function, `Reach.link` drops
it from a program that does not call it like any function. The only thing that makes it a macro is that `Macro` calls
it at compile time instead of the program calling it at run time.
This is also why there is no macro table. Storage was the question the front half deliberately left open, and the
answer is that there is none: the macro set is recomputed by scanning the top level for the word `defmacro`, which is
the only place it survives, and the compiled artefact is a `.so` keyed by a digest.
### `Form`, and the three numbers
A macro's parameter and its result are `Form`, so `Form` has to exist on the Flan side: a `defunion` in `prelude.ml`
mirroring `lib/form.ml`. It mirrors `Form.value` and **not** `Form.t` — there is no `loc` field, deliberately. A macro
cannot invent a source location, so the unmarshaller stamps the **call site's** `Loc.t` onto every node of what a
macro returns. That is the structural answer to "keep the source location of the call site attached to what a macro
produces", and it is what the queued structured-error work reads.
Case order is tag order, so the list in the prelude is a layout contract and says so. The widest cases are
`(Str [s string])` and `(List [xs [Form]])`; a string and a slice are both `%slice` = `{ptr, i64}`, 16 bytes at
align 8. So the image is `{ i32 tag, [2 x i64] payload }`: **24 bytes, align 8, payload at offset 8**, and every case
holds its one member at the payload's start, so there is no third offset anywhere in the marshaller.
Those three numbers are asserted, not assumed. `test_acceptance.ml`'s "Form's image format" asks LLVM for each of them
through the same `ptrtoint`-of-`getelementptr`-through-null oracle the DWARF offsets go through. Alignment needed a
probe the oracle did not have: the offset of field 1 in `{ i8, %"Form" }` *is* `alignof(Form)`, because a struct member
sits at the first offset its own alignment allows. Reading `[2 x i64]` out of the emitted type and concluding 8 would
be asserting the layout against itself, which is the circularity that got a `_Static_assert` rejected for the FFI.
### Nothing aggregate crosses to C
The boundary is `void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer)` — one thunk per macro, written by
`Emit.macro_thunk`. The thunk builds the `%slice` from `(args, n)` on the LLVM side, calls the macro, and stores the
result through `%out`.
The correction that matters here is not obvious from the diff. The unions work verified a union's **memory** layout
against clang; that is a different claim from LLVM's calling convention for an aggregate passed or returned **by
value** in hand-written IR, which is not promised to be clang's C ABI for the equivalent struct. Memory is the
agreement that actually exists, so pointers and scalars are all that cross. `%xfer` is the transfer channel every Flan
signature carries; `flan_macro_call` supplies a zeroed one, because a macro that signals with nothing above it to
handle it aborts inside the compiler, and the channel still has to be a real slot.
`Build.macro_module` produces a self-contained `.so`: the runtime linked in, no undefined Flan symbols, `-fPIC` on
every object including the `.ll`. Self-contained is what keeps `-rdynamic` off the compiler's own link. It goes
through clang rather than `llc` + `ld -shared`, unlike `Build.shared`, because there are C objects and a libc to find
— exactly the part of the driver the dev path skips.
OCaml has no `dlopen` for ELF (`Dynlink` loads OCaml), so `lib/dynload_stubs.c` is the whole boundary: `dlopen`,
`dlsym`, `dlclose`, the four-argument call, `calloc`/`free`, and a peek/poke family, because OCaml cannot address raw
memory and a `Form` image is written into it one field at a time.
### Quasiquote runs before the walk, and that is not a preference
Quasiquote is a desugaring over `Form` and nothing more: it becomes `form-nil`, `form-cons` per item and
`form-append` per splice — the prelude's three form-building functions and no fourth. It is pure, it needs nothing
loaded, and `Parse.program` runs it on the way in, which is what lets the prelude's own macros parse in a process
that has not built a macro module yet.
Running it **before** the expander's walk is load-bearing. A recursive conditional macro's body contains a
quasiquoted call to itself; with the quasiquote still standing, the walk would see that head and expand it then and
there, against the wrong arguments. Desugared first, that subform is a `(Form.Sym {.s "cond"})` and there is no head
left to mistake — so the walk needs no idea that quoting exists.
Nesting levels are counted nowhere: not by the reader, which was written that way deliberately, and not by the
desugaring. A quasiquote inside a quasiquote is refused by name. Only a macro that writes a macro wants one.
### A call inside a quasiquote is output, not a dependency
This is the distinction that is easy to get wrong, and the first cycle test written for this work got it wrong: it
quasiquoted, and it was not a cycle at all.
A macro body that **calls** another macro outside a quasiquote needs that macro compiled and loaded first, because
until then the call is a name nothing defines and the body will not compile. That is a compile-order dependency and it
is what makes the pre-pass a fixpoint. A macro body that **quasiquotes** a call to another macro needs nothing: the
call is part of what the macro answers, and the answer is expanded again after it returns.
So there are two different ways expansion fails to terminate, and they are different failures:
- **A ring** — two macros whose bodies call each other for real. There is no order to compile them in, so it is
refused, naming both. `test/programs/macro-cycle.flan`.
- **A macro that expands into a call to a macro and does not get smaller.** That is an ordinary loop, not an ordering
problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site.
`test/programs/macro-spin.flan`.
The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest
against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a
macro never sees a call to another macro in what it is handed.
### Hygiene is an escape hatch, not a system
Deliberately non-hygienic, Common Lisp's rule and Clojure's, settled in plan.org's open decision 2. A macro that needs
a name of its own calls `gensym`, which is a prelude function the loaded module runs while it runs. The name is
`~g<n>`, and `~` is a delimiter now — it opens an unquote — so no symbol the reader can produce contains one and a
gensym cannot collide with a name someone wrote. The counter lives in the loaded module rather than in the compiler,
which is the one place this departs from the original sketch; a module is dlopened once per compiler process, so it is
process-wide in practice.
### -linkall, and why the hook could not be installed by hand
Expanding a macro means compiling it, so `Macro` needs `Check`, `Build` and `Emit` and therefore sits **above** the
parser it feeds. The join is `Parse.expander`, a ref that `Macro` fills in at module initialisation.
Nothing references `Macro`, so without `-linkall` the linker drops it from every executable that does not name the
module — `bin/main.exe` among them — and a program calling a macro fails with an unknown name. Installing by hand at
every entry point was the alternative and it is not viable: `lib/session.ml` calls `Parse.program` for `C-c C-c` and
`C-c C-k`, and `test_session.ml` drives the session library in-process rather than through the CLI, so the set of
places that would need an install call is open-ended and a missed one is silent. `(library_flags (-linkall))` in
`lib/dune` is the guarantee instead.
`Macro.building` is the re-entrancy guard. `Build.macro_module` goes through `Check.program`, which parses the
prelude, which calls back into `Parse.program` — and that would re-enter the expander forever. Nothing is lost by
refusing to expand there: a macro compiled in round *n* calls only macros compiled in earlier rounds, and those calls
were already expanded before the build was entered.
### What it costs
- A build of a program that **names no macro**: 50ms, unchanged. The pass scans the top level, finds nothing, and no
compiler runs. This is nearly every program, and it is the reason the prelude can grow a `defmacro` without every
build paying a clang driver.
- A program that **calls one**: 310ms the first time, 70ms after. The 240ms is the clang driver; the module is cached
under the object cache and keyed by a digest of the prelude's source plus the file's `defmacro` forms, so it is paid
once per change rather than once per build. Every `flan build` is a fresh process, which is what makes the on-disk
cache rather than a memo the right shape.
- A hello-world's binary carries exactly one symbol out of all of this: `flan.gensym-n`, eight bytes. `Reach.link`
drops `unless`, `form-cons`, `form-nil`, `form-append`, `form-rest` and `gensym`, because nothing reachable calls
them.
### `unless` is the proof
plan.org milestone 5 says `when`, `unless`, `until`, `cond` and `dotimes` are special forms only until macros land.
`unless` is the first to stop being one — it is now a `defmacro` in `prelude.ml` and `parse.ml` has nothing to say
about it — and it was chosen because it is the one the prelude itself does not use. That matters: the prelude is
compiled into the macro module, so a prelude macro that the prelude's own functions call would need the expander to
compile the thing the expander needs in order to run.
Its coverage is `sand.flan`, seven calls, compiled through `Session` in `test_session` — the in-process path, and the
reason `-linkall` is not optional. Say plainly what that coverage is not: nothing in `test/programs` used `unless`
before this landed, so `macro-unless.flan` is a test written after the feature. The corpus written before it is
`sand.flan` and `web/examples/control.flan`, and both compile unchanged.
## `defer` may be written in a `let`
The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both

203
NEXT.md
View File

@ -40,6 +40,34 @@ read. It belongs with item 2, where the listing is being changed anyway.
Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`,
which §6 rules out.
### Landed — macros run, and `unless` is not a special form any more
The expander is written and the exit criterion plan.org set for milestone 5 is met: a conditional sugar moved out of
`parse.ml` and into `prelude.ml` as a `defmacro`, with the corpus that was written against the special form
unchanged. Running `test/programs/macro-unless.flan` means the compiler built a shared object, `dlopen`ed it into
itself and called a Flan function to find out what `(unless c a b)` means.
The full explanation is in [`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program". Four things worth
knowing before touching any of it, because each cost something to find:
- **A call inside a quasiquote is output, not a compile-order dependency.** A macro body that *calls* another macro
needs it compiled first; a macro body that *quasiquotes* a call to one needs nothing, because the call is part of
what it answers and the answer is expanded again. The first cycle test written for this got that wrong and was not
a cycle at all. The two non-termination failures are therefore different and are refused differently: a ring is
named, a macro that does not settle is bounded.
- **Quasiquote is desugared before the walk**, and that is load-bearing rather than tidy — with the quasiquote still
standing, the walk expands the call inside it against the wrong arguments.
- **`lib/dune` passes `-linkall`.** `lib/macro.ml` installs itself into `Parse.expander` and nothing references it, so
the linker would otherwise drop it from `bin/main.exe`. Installing by hand is not viable: `session.ml` parses for
`C-c C-c`, and `test_session.ml` drives the session library in-process.
- **Two parser bugs fell out of it**, both in the rule that tells a return type from the first form of a body. The
prelude's types were not in the set that rule consults, so `Form` in return position was read as a body form; and
adding them plainly made `(defn f [] (Rune {.code 65}) (bar))` a function returning a `Rune` with a one-form body,
silently, in every file in the language. Both are pinned in test_flan.ml's return-type section.
Costs: a build that names no macro is unchanged at 50ms; one that calls a macro is 310ms cold and 70ms warm, the
difference being a cached `.so`; and a hello-world carries eight bytes of it, because `Reach.link` drops the rest.
### Landed — a C header is read, so a binding is checked instead of trusted
`lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `BUILT.md`, "The header is read
@ -533,11 +561,9 @@ run one lane at a time; item 4 is disjoint and runs alongside any of them.
namespace before `collect` runs, so a `defunion Form` in `prelude.ml` is an ordinary same-file declaration and
needs no import and no `load.ml` change. Verified by declaring one there and matching it from a program.
**Then macros**, which are blocked on exactly this and nothing else: a macro is `[Form] -> Form`, so `Form` has to be
a Flan union whose *layout* the compiler and the `dlopen`ed macro agree on byte for byte. `NEXT.md`'s macro section
has the expander design — a pre-pass fixpoint before `Parse`, `gensym` as a compiler-side counter, quasiquote as a
desugaring over `Form`. The exit criterion is already written: move `when` or `cond` out of `parse.ml` into the
prelude as a `defmacro` with the existing tests unchanged and still green.
**Macros landed on top of this** and needed no `load.ml` change for `Form`, exactly as this said. See
[`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program", and the short list of what is left of them
below.
Macros are what buy `with-drawing` and `with-mode-2d` over raylib's begin/end pairs, the hiccup DSL if a JS backend
ever happens, and the removal of special forms from the compiler.
@ -670,8 +696,13 @@ libraries. Four things were settled doing it:
needs, since every `defmacro` must be compiled before anything that calls it.
`Load.t.pkgs` now comes back in topological order, dependencies first. The *declaration* list is deliberately not
sorted and does not need to be — `check.ml` collects every top-level name before it checks any body — so the order
exists for the expander, which cannot work that way.
sorted and does not need to be — `check.ml` collects every top-level name before it checks any body.
**The expander did not end up reading that order**, and it is worth saying so rather than leaving the paragraphs above
to imply otherwise. Macros are collected from the prelude and from the file being compiled; a `defmacro` in a package
is refused by name, because reaching one means resolving that package's own imports over `Form`s before `Load` runs.
The order is there and correct and is what package-level macros will read on the day they exist; nothing reads it
today.
**Still missing: package visibility.** `rl/get-color-raw` is callable. The blocker is surface syntax, not `load.ml`:
`exported` and the refusal machinery already exist and take a second rule in one line, but there is no way for a
@ -1328,142 +1359,46 @@ A package importing a package was on this list and is off it. It loads, a diamon
package, the alias clash is refused through a chain as well as inside one file, and a ring is refused by name. What is
left of the item is visibility, which is listed above and needs a marker the parser does not have.
## Macros — the reader and the declaration are in, the expander is not
## Macros — landed; what is left of them
The front half landed. What exists:
**The expander works and `unless` is a prelude `defmacro`.** How all of it fits together is in
[`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program" — the image format, the thunk ABI, why quasiquote
runs before the walk, the two different ways expansion fails to terminate, `-linkall`, and the three cost numbers.
What follows is only the part that is still missing.
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, `(unquote x)` and `(unquote-splicing x)`, exactly
as `'x` reads as `(quote x)`. It stays dumb: it does not count nesting levels, does not know whether an unquote is
inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma
is whitespace in `is_delimiter` and every binding vector in the corpus relies on that. Backtick and tilde are delimiters
now, so `a~b` is two things.
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say expansion is not wired up; `unquote` and
`unquote-splicing` say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature.
`(defmacro name [params] body ...)` at the top level is checked for shape and *then* refused — a malformed defmacro and
an unimplemented one get different reasons, so the shape rule is enforced before the feature exists.
- **Four special forms left**, and two of them are the hard ones. `until` and `cond` are free to move whenever
somebody wants them. `when` and `dotimes` are not: the prelude itself uses them 29 and 12 times, so moving either
makes the prelude depend on the macro that the macro module has to compile the prelude to get. Breaking that needs
either a prelude that stops using them, or a two-stage prelude where the macro module is built from a subset. The
first is a mechanical edit of `prelude.ml` and is probably the answer.
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, because a table nothing reads is a place
for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit.
`cond` has its own snag, and it is the reason `unless` went first: `parse.ml` refuses `(cond a)` with "cond clause
has no body", and a macro cannot produce that (see the next item), so moving `cond` changes an existing test.
### How the expander should work
- **A macro has no error facility**, and this is the biggest gap. A macro runs inside the compiler; anything it
signals aborts the compile with no location. So the prelude's `unless` answers `(unless-takes-a-test-and-a-body)`
when it is handed fewer than two forms, and the report is "unknown name unless-takes-a-test-and-a-body" at the call
site — right place, wrong sentence. What a macro wants is a way to say *this is wrong and here is why*, reported at
the call site. The queued structured-error rewrite is where that belongs, and the call site's `Loc.t` is already
stamped onto everything a macro returns, so the location half is done.
**There is no interpreter** (see "Why there is no interpreter" in `BUILT.md`) and there is not going to be one, so running a macro at
compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured:
`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload
primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's.
- **Macros are not imported.** A `defmacro` in a package is refused by name in `load.ml`. Reaching one would mean
resolving that package's own imports over `Form`s, before `Load` runs — a second import resolver. `programs/pkg-macro.flan`.
The shape it wants:
- **A prelude macro may not call a macro.** The prelude is in every macro module by construction, so there is no
round it could be compiled in after something else. It would fail with an unknown name rather than with a reason,
which is worth fixing the day the prelude wants one.
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its result is a form, which means `Form.t`
has to exist on the Flan side — a `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and accessors.
That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on
the *layout* of a `Form`, not merely its shape, so whatever the checker does for unions has to be exact here. Until
unions are values this cannot start — `check.ml` puts union values and `match` on a union at **milestone 6**, so that is
milestone 6 work landing before milestone 5's.
2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: there is no `Ast.Defmacro` and `Parse` refuses
`defmacro` outright, so an `Ast`-level pass would have nothing to work with. That refusal is not a dead end, it is the
ordering — the expander runs first and `Parse` never sees a macro call at all. It is also the Clojure ordering, and the
reason a macro expanding to a special form is ordinary rather than a special case.
3. **Order matters and files do not have one.** Top-level names in a package are order-independent everywhere else
(`declared_types`, the constant fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and loaded before a
call to it is expanded. Either collect every `defmacro` in a pre-pass and compile them as one module, or require
definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest
of the frontend already behaves.
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be
detected and named rather than looping.
5. **`gensym` is a runtime function of the compiler**, called by the loaded macro while it runs. It needs a counter that
lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a
character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org,
open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no `macrolet` until a
concrete use case appears.
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: `` `(a ~b) `` becomes list-construction
over quoted pieces, with `~@` splicing. Written once, in the expander, over `Form`.
- **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and
not the desugaring. Only a macro that writes a macro wants one.
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — were owned by other lanes when the front
half landed, which is the only reason the expander is not here too.
- **`gensym`'s counter restarts in a second module.** It lives in the loaded module, and a module is dlopened once per
compiler process, so it is process-wide in practice. The rounds already build more than one module for a program
whose macros call macros, and the fix that day is to seed the counter from the module's index.
### Handoff: the boundary is built and verified-by-compilation, the expander is not written
A lane stopped here mid-flight. What exists, exactly:
- **`lib/dynload_stubs.c` and `lib/dynload.ml` — the compiler's own dlopen.** This was the one unvalidated
assumption under the whole design and it is now machinery. OCaml has no dlopen for ELF (`Dynlink` loads OCaml,
not shared objects), and `lib/dune` had no `foreign_stubs`, so "point the reload primitive at the compiler's own
process" was not the small step it reads as. It is `dlopen`/`dlsym`/`dlclose`, a four-argument call into a macro
thunk, `calloc`/`free`, and a peek/poke family — OCaml cannot address raw memory, so a `Form` image is written
into it one field at a time from C. `(c_library_flags (-ldl))` is in `lib/dune`.
- **`Emit.macro_thunk`, and `Emit.program ?macros`.** One thunk per macro:
`void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer)`. It builds the `%slice` from `(args, n)`,
calls the macro, stores the result through `%out`. **Nothing aggregate crosses to C.** This is the correction
that matters and it is not obvious from the diff: the unions lane verified a union's *memory* layout against
clang, which is a different claim from LLVM's calling convention for an aggregate passed or returned **by value**
in hand-written IR. Memory is the only agreement that exists, so the boundary is pointers and scalars only.
- **`Build.macro_module`.** A whole program into a self-contained `.so`: the runtime linked in, no undefined Flan
symbols, `-fPIC` on every object including the `.ll`. Self-contained is what keeps `-rdynamic` off the compiler's
own link. It goes through clang rather than `llc` + `ld -shared` — unlike `Build.shared` — because there are C
objects and a libc to find, which is exactly the part of the driver the dev path skips. Cost is the driver's
~50ms, unmeasured here, paid once per process for the whole macro set.
- **`defunion Form` and the list-building surface, in `prelude.ml`.** Written, and the compiler builds; **not yet
checked against a program, so its layout is unverified.** That is the first thing to do.
Two gates were checked before any of this and both pass, which saves re-deriving them:
- **`check_finite` does not recurse through `Types.Slice`**, only through `Named`, `Array` and `Option`. So a
union case holding `[Form]` is accepted and `Form` needs no `(Ptr Form)` indirection.
- **The default allocator needs no init.** `flan_ctx_alloc = &flan_heap` is statically initialised in
`flan_rt.c`, so a module with no `main` can allocate. `flan_rt_init` is only argv.
**The layout the two sides have to agree on.** `Form` mirrors `Form.value`, **not** `Form.t` — there is no `loc`
field, deliberately. A macro cannot invent a source location, so the unmarshaller stamps the *call site's*
`Loc.t` onto every node of what a macro returns; that is the structural answer to "keep the call site's location
attached to what a macro produces", and it is what the queued structured-error work reads. The cases are
`Sym Kw Int Float Str Byte List Vec Map` and **case order is tag order**, so the list is a layout contract with
the marshaller and may not be reordered. The widest cases are `string` and `[Form]`, both `%slice` = 16 bytes
align 8, so the expected shape is `{ i32 tag, [2 x i64] payload }`: **24 bytes, align 8, payload at offset 8**.
Those three numbers are the whole agreement and **they are asserted nowhere yet** — the next commit should put
them through the same `ptrtoint` layout oracle the unions lane used, not hardcode them on faith.
**What is not written at all:** `lib/expand.ml`. No marshaller, no unmarshaller, no macro collection, no
quasiquote, no fixpoint, no cycle detection. `parse.ml` still refuses `defmacro`, `when`/`unless`/`until`/`cond`/
`dotimes` are still special forms, and the exit criterion is untouched.
**What the next person should do first**, in this order, committing each:
1. Write a program that names `Form` and check its layout through the oracle — 24/8/8. `Vec` is also a case name
and `(Vec T)` is also a type application; if the struct-literal arm and the type arm collide, rename the case and
say so, because that is a layout-contract change.
2. Prove the boundary: one Flan file, `(defn id [args [Form]] Form (at args 0))`, through `Build.macro_module`,
`Dynload.dl_open`, `dl_sym "flan.macro.id"`, a hand-laid `Form` in, the same one back. That is the commit that
makes everything above real rather than plausible.
3. Only then the expander.
Four decisions this lane made that the design in this section did not settle, each of which the next person may
overturn cheaply:
- **A macro takes one parameter, the slice of argument forms** — `[Form] -> Form` read as a single function type,
not as "one declared parameter per argument". It needs no reader or parser change (`[args]` already passes the
existing shape check) and it gives variadics for free, which `when` and `unless` both need since there is no
`&rest`.
- **The thunk ABI above**, rather than letting `%"Form"` cross to C.
- **`gensym`'s counter lives in the loaded module**, not in the compiler process as this section sketches. The
name is `~g<n>`; `~` is a delimiter now, so no symbol the reader produces can contain one and a gensym cannot
collide. A module is dlopened once per compiler process, so the counter is process-wide in practice; a second
module would restart it, and the fix that day is to seed it from the module's index.
- **The macro module is the prelude plus the program's `defmacro`s, and not the program's own functions.**
Compiling the user's `defn`s into it would mean compiling a program that has not been expanded yet, which is the
chicken-and-egg the pre-pass exists to avoid. The cost is that a macro body may call prelude functions and other
macros and nothing else. Worth revisiting; not worth revisiting first.
Left deliberately undone and named so nobody hunts for it: `&rest` sugar, an error carrying the expansion it came
from (only the call-site location is preserved, which is the part that does not make the later work harder), and
the other four special forms.
### What would tell you it works
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` today, and plan.org milestone 5 says
they are special forms *only until macros land*. Moving one of them out of the compiler and into the prelude as a
`defmacro`, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote,
`gensym` and the ordering pre-pass at once, against a test suite written before any of them existed.
- **No `&rest` sugar.** A macro takes one parameter, the slice of forms at its call site, and `(len args)` is the
arity. That is deliberate — it is where variadics come from — but a `when` written against it reads worse than
`parse.ml`'s version did.
## Watch for