From f590436ed3f9b155c8ad4dc2dfd1e1a6706ce3e5 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Fri, 11 Sep 2026 20:17:12 +0700 Subject: [PATCH] Write down the expander so the next lane inherits a decision, not a table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The front half is here and the back half is not, and the reason is that running a macro means compiling it and dlopening it into the compiler — which is Emit.redefinition plus Build.shared, already measured at ~19ms, pointed at our own process instead of the program's. The part worth recording is what blocks it: a macro is [Form] -> Form, so Form has to be a Flan union with a layout the compiler and the loaded macro agree on exactly. That is milestone 6 work landing before milestone 5's, and it is bigger than the expander. Nothing is stored on purpose. No macro table and no Ast.Defmacro: a table nothing reads is where a design rots, and the storage shape is the expander author's first decision rather than one to inherit from a lane that could not test it. --- NEXT.md | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/NEXT.md b/NEXT.md index da43f24..64e53d6 100644 --- a/NEXT.md +++ b/NEXT.md @@ -1582,6 +1582,86 @@ Deferred until after the dev loop: `defer`; package visibility, so `rl/get-color-raw` is not callable; a package importing a package; imported unions. +## Macros — the reader and the declaration are in, the expander is not + +The front half landed. What exists: + +- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, + `(unquote x)` and `(unquote-splicing x)`, exactly as `'x` reads as + `(quote x)`. It stays dumb: it does not count nesting levels, does not know + whether an unquote is inside a quasiquote, and attaches no meaning to the + three names. Clojure's spelling, not Common Lisp's, because a comma is + whitespace in `is_delimiter` and every binding vector in the corpus relies + on that. Backtick and tilde are delimiters now, so `a~b` is two things. +- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say + expansion is not wired up; `unquote` and `unquote-splicing` say they mean + nothing outside a quasiquote, which is a mistake rather than a missing + feature. `(defmacro name [params] body ...)` at the top level is checked for + shape and *then* refused — a malformed defmacro and an unimplemented one get + different reasons, so the shape rule is enforced before the feature exists. + +Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, +because a table nothing reads is a place for a design to rot, and the storage +shape is the expander author's first decision, not a decision to inherit. + +### How the expander should work + +**There is no interpreter** (see "Why there is no interpreter") and there is not +going to be one, so running a macro at compile time means *compiling it and +loading it into the compiler*. That machinery already exists and is measured: +`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the +load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline +pointed at the compiler's own process instead of the program's. + +The shape it wants: + +1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its + result is a form, which means `Form.t` has to exist on the Flan side — a + `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and + accessors. That is the real work, and it is bigger than the expander itself: + the compiler and the compiled macro have to agree on the *layout* of a + `Form`, not merely its shape, so whatever the checker does for unions has to + be exact here. Until unions are values this cannot start; that is milestone 6 + work landing before milestone 5's. +2. **Expansion is a pass between `Parse` and `Check`**, over `Ast`, not over + `Form` — or, better, over `Form` before `Parse` ever runs, which is why + `Parse` refusing `defmacro` rather than storing it is not a dead end: the + expander runs first and `Parse` never sees a macro call at all. That is the + Clojure ordering and the one to prefer, because a macro expanding to a + special form is then ordinary rather than a special case. +3. **Order matters and files do not have one.** Top-level names in a package + are order-independent everywhere else (`declared_types`, the constant + fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and + loaded before a call to it is expanded. Either collect every `defmacro` in a + pre-pass and compile them as one module, or require definition-before-use for + macros specifically and say so in the error. The pre-pass is better and + matches how the rest of the frontend already behaves. +4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a + single sweep, and a cycle has to be detected and named rather than looping. +5. **`gensym` is a runtime function of the compiler**, called by the loaded + macro while it runs. It needs a counter that lives in the compiler process + and a name that cannot collide with a reader-produced symbol — the usual + trick is a character no symbol may contain, and this reader now has two new + ones it could reserve. Hygiene is settled (plan.org, open decision 2): + deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no + `macrolet` until a concrete use case appears. +6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: + `` `(a ~b) `` becomes list-construction over quoted pieces, with + `~@` splicing. Written once, in the expander, over `Form`. + +The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — +were owned by other lanes when the front half landed, which is the only reason +the expander is not here too. + +### What would tell you it works + +`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` +today, and plan.org milestone 5 says they are special forms *only until macros +land*. Moving one of them out of the compiler and into the prelude as a +`defmacro`, with the existing tests unchanged and still green, is the exit +criterion — it proves expansion, quasiquote, `gensym` and the ordering pre-pass +at once, against a test suite written before any of them existed. + ## Watch for The rule that caught the two misparse bugs applies unchanged: **anything that