Write down the expander so the next lane inherits a decision, not a table

The front half is here and the back half is not, and the reason is that
running a macro means compiling it and dlopening it into the compiler —
which is Emit.redefinition plus Build.shared, already measured at ~19ms,
pointed at our own process instead of the program's.

The part worth recording is what blocks it: a macro is [Form] -> Form,
so Form has to be a Flan union with a layout the compiler and the loaded
macro agree on exactly. That is milestone 6 work landing before
milestone 5's, and it is bigger than the expander.

Nothing is stored on purpose. No macro table and no Ast.Defmacro: a
table nothing reads is where a design rots, and the storage shape is the
expander author's first decision rather than one to inherit from a lane
that could not test it.
This commit is contained in:
Joseph Ferano 2026-09-11 20:17:12 +07:00
parent d4cef99718
commit f590436ed3

80
NEXT.md
View File

@ -1582,6 +1582,86 @@ Deferred until after the dev loop:
`defer`; package visibility, so `rl/get-color-raw` is not callable; a
package importing a package; imported unions.
## Macros — the reader and the declaration are in, the expander is not
The front half landed. What exists:
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`,
`(unquote x)` and `(unquote-splicing x)`, exactly as `'x` reads as
`(quote x)`. It stays dumb: it does not count nesting levels, does not know
whether an unquote is inside a quasiquote, and attaches no meaning to the
three names. Clojure's spelling, not Common Lisp's, because a comma is
whitespace in `is_delimiter` and every binding vector in the corpus relies
on that. Backtick and tilde are delimiters now, so `a~b` is two things.
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say
expansion is not wired up; `unquote` and `unquote-splicing` say they mean
nothing outside a quasiquote, which is a mistake rather than a missing
feature. `(defmacro name [params] body ...)` at the top level is checked for
shape and *then* refused — a malformed defmacro and an unimplemented one get
different reasons, so the shape rule is enforced before the feature exists.
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`,
because a table nothing reads is a place for a design to rot, and the storage
shape is the expander author's first decision, not a decision to inherit.
### How the expander should work
**There is no interpreter** (see "Why there is no interpreter") and there is not
going to be one, so running a macro at compile time means *compiling it and
loading it into the compiler*. That machinery already exists and is measured:
`Emit.redefinition``Build.shared``dlopen` is ~19ms end to end, with the
load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline
pointed at the compiler's own process instead of the program's.
The shape it wants:
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its
result is a form, which means `Form.t` has to exist on the Flan side — a
`defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and
accessors. That is the real work, and it is bigger than the expander itself:
the compiler and the compiled macro have to agree on the *layout* of a
`Form`, not merely its shape, so whatever the checker does for unions has to
be exact here. Until unions are values this cannot start; that is milestone 6
work landing before milestone 5's.
2. **Expansion is a pass between `Parse` and `Check`**, over `Ast`, not over
`Form` — or, better, over `Form` before `Parse` ever runs, which is why
`Parse` refusing `defmacro` rather than storing it is not a dead end: the
expander runs first and `Parse` never sees a macro call at all. That is the
Clojure ordering and the one to prefer, because a macro expanding to a
special form is then ordinary rather than a special case.
3. **Order matters and files do not have one.** Top-level names in a package
are order-independent everywhere else (`declared_types`, the constant
fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and
loaded before a call to it is expanded. Either collect every `defmacro` in a
pre-pass and compile them as one module, or require definition-before-use for
macros specifically and say so in the error. The pre-pass is better and
matches how the rest of the frontend already behaves.
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a
single sweep, and a cycle has to be detected and named rather than looping.
5. **`gensym` is a runtime function of the compiler**, called by the loaded
macro while it runs. It needs a counter that lives in the compiler process
and a name that cannot collide with a reader-produced symbol — the usual
trick is a character no symbol may contain, and this reader now has two new
ones it could reserve. Hygiene is settled (plan.org, open decision 2):
deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no
`macrolet` until a concrete use case appears.
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature:
`` `(a ~b) `` becomes list-construction over quoted pieces, with
`~@` splicing. Written once, in the expander, over `Form`.
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml`
were owned by other lanes when the front half landed, which is the only reason
the expander is not here too.
### What would tell you it works
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml`
today, and plan.org milestone 5 says they are special forms *only until macros
land*. Moving one of them out of the compiler and into the prelude as a
`defmacro`, with the existing tests unchanged and still green, is the exit
criterion — it proves expansion, quasiquote, `gensym` and the ordering pre-pass
at once, against a test suite written before any of them existed.
## Watch for
The rule that caught the two misparse bugs applies unchanged: **anything that