diff --git a/BUILT.md b/BUILT.md index 35a4d5d..163d92f 100644 --- a/BUILT.md +++ b/BUILT.md @@ -565,6 +565,10 @@ perceptually instant for expression eval too. Milestone 3 did not need an oracle hand-written, so the table *is* the oracle. Consequences already applied: milestone 2's "interpreted calls per second" criterion is dropped, and the host ABI moved onto the critical path. +It stopped being an absence when macros landed. A macro has to run at compile time and there is nothing to interpret +it with, so the compiler compiles it into a shared object and `dlopen`s it into its own process — see "Macros: the +compiler dlopens the program". The decision's cost and its mechanism are the same thing. + ## The layout, which is the whole backend design ``` @@ -2279,6 +2283,158 @@ there and matching it from a program. `dev.ml`'s inspector still says "union val frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates. +## Macros: the compiler dlopens the program + +"Why there is no interpreter" above decided that the compiled path is the only backend. A macro is the first thing +that turns that decision into a mechanism rather than an absence: **running a macro at compile time means compiling +it and loading it into the compiler's own process.** There is nothing to interpret it with and there is not going to +be, so `Emit.redefinition` → `Build.shared` → `dlopen`, the reload primitive the dev loop already runs, is pointed at +the compiler instead of at a running program. + +`(defmacro name [args] body ...)` is one function, `[Form] -> Form`. One parameter, the slice of forms written at the +call site, which is where variadics come from in a language with no `&rest`: `(len args)` is how many were written. + +### A defmacro is a defn, and there is no Ast.Defmacro + +`Parse` turns `(defmacro m [args] body)` into `(defn m [args [Form]] Form body)` and nothing below the parser knows +the word exists. The checker checks it like any function, the backend emits it like any function, `Reach.link` drops +it from a program that does not call it like any function. The only thing that makes it a macro is that `Macro` calls +it at compile time instead of the program calling it at run time. + +This is also why there is no macro table. Storage was the question the front half deliberately left open, and the +answer is that there is none: the macro set is recomputed by scanning the top level for the word `defmacro`, which is +the only place it survives, and the compiled artefact is a `.so` keyed by a digest. + +### `Form`, and the three numbers + +A macro's parameter and its result are `Form`, so `Form` has to exist on the Flan side: a `defunion` in `prelude.ml` +mirroring `lib/form.ml`. It mirrors `Form.value` and **not** `Form.t` — there is no `loc` field, deliberately. A macro +cannot invent a source location, so the unmarshaller stamps the **call site's** `Loc.t` onto every node of what a +macro returns. That is the structural answer to "keep the source location of the call site attached to what a macro +produces", and it is what the queued structured-error work reads. + +Case order is tag order, so the list in the prelude is a layout contract and says so. The widest cases are +`(Str [s string])` and `(List [xs [Form]])`; a string and a slice are both `%slice` = `{ptr, i64}`, 16 bytes at +align 8. So the image is `{ i32 tag, [2 x i64] payload }`: **24 bytes, align 8, payload at offset 8**, and every case +holds its one member at the payload's start, so there is no third offset anywhere in the marshaller. + +Those three numbers are asserted, not assumed. `test_acceptance.ml`'s "Form's image format" asks LLVM for each of them +through the same `ptrtoint`-of-`getelementptr`-through-null oracle the DWARF offsets go through. Alignment needed a +probe the oracle did not have: the offset of field 1 in `{ i8, %"Form" }` *is* `alignof(Form)`, because a struct member +sits at the first offset its own alignment allows. Reading `[2 x i64]` out of the emitted type and concluding 8 would +be asserting the layout against itself, which is the circularity that got a `_Static_assert` rejected for the FFI. + +### Nothing aggregate crosses to C + +The boundary is `void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer)` — one thunk per macro, written by +`Emit.macro_thunk`. The thunk builds the `%slice` from `(args, n)` on the LLVM side, calls the macro, and stores the +result through `%out`. + +The correction that matters here is not obvious from the diff. The unions work verified a union's **memory** layout +against clang; that is a different claim from LLVM's calling convention for an aggregate passed or returned **by +value** in hand-written IR, which is not promised to be clang's C ABI for the equivalent struct. Memory is the +agreement that actually exists, so pointers and scalars are all that cross. `%xfer` is the transfer channel every Flan +signature carries; `flan_macro_call` supplies a zeroed one, because a macro that signals with nothing above it to +handle it aborts inside the compiler, and the channel still has to be a real slot. + +`Build.macro_module` produces a self-contained `.so`: the runtime linked in, no undefined Flan symbols, `-fPIC` on +every object including the `.ll`. Self-contained is what keeps `-rdynamic` off the compiler's own link. It goes +through clang rather than `llc` + `ld -shared`, unlike `Build.shared`, because there are C objects and a libc to find +— exactly the part of the driver the dev path skips. + +OCaml has no `dlopen` for ELF (`Dynlink` loads OCaml), so `lib/dynload_stubs.c` is the whole boundary: `dlopen`, +`dlsym`, `dlclose`, the four-argument call, `calloc`/`free`, and a peek/poke family, because OCaml cannot address raw +memory and a `Form` image is written into it one field at a time. + +### Quasiquote runs before the walk, and that is not a preference + +Quasiquote is a desugaring over `Form` and nothing more: it becomes `form-nil`, `form-cons` per item and +`form-append` per splice — the prelude's three form-building functions and no fourth. It is pure, it needs nothing +loaded, and `Parse.program` runs it on the way in, which is what lets the prelude's own macros parse in a process +that has not built a macro module yet. + +Running it **before** the expander's walk is load-bearing. A recursive conditional macro's body contains a +quasiquoted call to itself; with the quasiquote still standing, the walk would see that head and expand it then and +there, against the wrong arguments. Desugared first, that subform is a `(Form.Sym {.s "cond"})` and there is no head +left to mistake — so the walk needs no idea that quoting exists. + +Nesting levels are counted nowhere: not by the reader, which was written that way deliberately, and not by the +desugaring. A quasiquote inside a quasiquote is refused by name. Only a macro that writes a macro wants one. + +### A call inside a quasiquote is output, not a dependency + +This is the distinction that is easy to get wrong, and the first cycle test written for this work got it wrong: it +quasiquoted, and it was not a cycle at all. + +A macro body that **calls** another macro outside a quasiquote needs that macro compiled and loaded first, because +until then the call is a name nothing defines and the body will not compile. That is a compile-order dependency and it +is what makes the pre-pass a fixpoint. A macro body that **quasiquotes** a call to another macro needs nothing: the +call is part of what the macro answers, and the answer is expanded again after it returns. + +So there are two different ways expansion fails to terminate, and they are different failures: + +- **A ring** — two macros whose bodies call each other for real. There is no order to compile them in, so it is + refused, naming both. `test/programs/macro-cycle.flan`. +- **A macro that expands into a call to a macro and does not get smaller.** That is an ordinary loop, not an ordering + problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site. + `test/programs/macro-spin.flan`. + +The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest +against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a +macro never sees a call to another macro in what it is handed. + +### Hygiene is an escape hatch, not a system + +Deliberately non-hygienic, Common Lisp's rule and Clojure's, settled in plan.org's open decision 2. A macro that needs +a name of its own calls `gensym`, which is a prelude function the loaded module runs while it runs. The name is +`~g`, and `~` is a delimiter now — it opens an unquote — so no symbol the reader can produce contains one and a +gensym cannot collide with a name someone wrote. The counter lives in the loaded module rather than in the compiler, +which is the one place this departs from the original sketch; a module is dlopened once per compiler process, so it is +process-wide in practice. + +### -linkall, and why the hook could not be installed by hand + +Expanding a macro means compiling it, so `Macro` needs `Check`, `Build` and `Emit` and therefore sits **above** the +parser it feeds. The join is `Parse.expander`, a ref that `Macro` fills in at module initialisation. + +Nothing references `Macro`, so without `-linkall` the linker drops it from every executable that does not name the +module — `bin/main.exe` among them — and a program calling a macro fails with an unknown name. Installing by hand at +every entry point was the alternative and it is not viable: `lib/session.ml` calls `Parse.program` for `C-c C-c` and +`C-c C-k`, and `test_session.ml` drives the session library in-process rather than through the CLI, so the set of +places that would need an install call is open-ended and a missed one is silent. `(library_flags (-linkall))` in +`lib/dune` is the guarantee instead. + +`Macro.building` is the re-entrancy guard. `Build.macro_module` goes through `Check.program`, which parses the +prelude, which calls back into `Parse.program` — and that would re-enter the expander forever. Nothing is lost by +refusing to expand there: a macro compiled in round *n* calls only macros compiled in earlier rounds, and those calls +were already expanded before the build was entered. + +### What it costs + +- A build of a program that **names no macro**: 50ms, unchanged. The pass scans the top level, finds nothing, and no + compiler runs. This is nearly every program, and it is the reason the prelude can grow a `defmacro` without every + build paying a clang driver. +- A program that **calls one**: 310ms the first time, 70ms after. The 240ms is the clang driver; the module is cached + under the object cache and keyed by a digest of the prelude's source plus the file's `defmacro` forms, so it is paid + once per change rather than once per build. Every `flan build` is a fresh process, which is what makes the on-disk + cache rather than a memo the right shape. +- A hello-world's binary carries exactly one symbol out of all of this: `flan.gensym-n`, eight bytes. `Reach.link` + drops `unless`, `form-cons`, `form-nil`, `form-append`, `form-rest` and `gensym`, because nothing reachable calls + them. + +### `unless` is the proof + +plan.org milestone 5 says `when`, `unless`, `until`, `cond` and `dotimes` are special forms only until macros land. +`unless` is the first to stop being one — it is now a `defmacro` in `prelude.ml` and `parse.ml` has nothing to say +about it — and it was chosen because it is the one the prelude itself does not use. That matters: the prelude is +compiled into the macro module, so a prelude macro that the prelude's own functions call would need the expander to +compile the thing the expander needs in order to run. + +Its coverage is `sand.flan`, seven calls, compiled through `Session` in `test_session` — the in-process path, and the +reason `-linkall` is not optional. Say plainly what that coverage is not: nothing in `test/programs` used `unless` +before this landed, so `macro-unless.flan` is a test written after the feature. The corpus written before it is +`sand.flan` and `web/examples/control.flan`, and both compile unchanged. + ## `defer` may be written in a `let` The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both diff --git a/NEXT.md b/NEXT.md index 20725ad..8ef6052 100644 --- a/NEXT.md +++ b/NEXT.md @@ -40,6 +40,34 @@ read. It belongs with item 2, where the listing is being changed anyway. Read SBCL for what restarts should *mean* and ignore how it moves control: it transfers with `block`/`return-from`, which §6 rules out. +### Landed — macros run, and `unless` is not a special form any more + +The expander is written and the exit criterion plan.org set for milestone 5 is met: a conditional sugar moved out of +`parse.ml` and into `prelude.ml` as a `defmacro`, with the corpus that was written against the special form +unchanged. Running `test/programs/macro-unless.flan` means the compiler built a shared object, `dlopen`ed it into +itself and called a Flan function to find out what `(unless c a b)` means. + +The full explanation is in [`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program". Four things worth +knowing before touching any of it, because each cost something to find: + +- **A call inside a quasiquote is output, not a compile-order dependency.** A macro body that *calls* another macro + needs it compiled first; a macro body that *quasiquotes* a call to one needs nothing, because the call is part of + what it answers and the answer is expanded again. The first cycle test written for this got that wrong and was not + a cycle at all. The two non-termination failures are therefore different and are refused differently: a ring is + named, a macro that does not settle is bounded. +- **Quasiquote is desugared before the walk**, and that is load-bearing rather than tidy — with the quasiquote still + standing, the walk expands the call inside it against the wrong arguments. +- **`lib/dune` passes `-linkall`.** `lib/macro.ml` installs itself into `Parse.expander` and nothing references it, so + the linker would otherwise drop it from `bin/main.exe`. Installing by hand is not viable: `session.ml` parses for + `C-c C-c`, and `test_session.ml` drives the session library in-process. +- **Two parser bugs fell out of it**, both in the rule that tells a return type from the first form of a body. The + prelude's types were not in the set that rule consults, so `Form` in return position was read as a body form; and + adding them plainly made `(defn f [] (Rune {.code 65}) (bar))` a function returning a `Rune` with a one-form body, + silently, in every file in the language. Both are pinned in test_flan.ml's return-type section. + +Costs: a build that names no macro is unchanged at 50ms; one that calls a macro is 310ms cold and 70ms warm, the +difference being a cached `.so`; and a hello-world carries eight bytes of it, because `Reach.link` drops the rest. + ### Landed — a C header is read, so a binding is checked instead of trusted `lib/cimport.ml`, `lib/cjson.ml`, a `headers` file beside `link`. Full reasoning in `BUILT.md`, "The header is read @@ -533,11 +561,9 @@ run one lane at a time; item 4 is disjoint and runs alongside any of them. namespace before `collect` runs, so a `defunion Form` in `prelude.ml` is an ordinary same-file declaration and needs no import and no `load.ml` change. Verified by declaring one there and matching it from a program. - **Then macros**, which are blocked on exactly this and nothing else: a macro is `[Form] -> Form`, so `Form` has to be - a Flan union whose *layout* the compiler and the `dlopen`ed macro agree on byte for byte. `NEXT.md`'s macro section - has the expander design — a pre-pass fixpoint before `Parse`, `gensym` as a compiler-side counter, quasiquote as a - desugaring over `Form`. The exit criterion is already written: move `when` or `cond` out of `parse.ml` into the - prelude as a `defmacro` with the existing tests unchanged and still green. + **Macros landed on top of this** and needed no `load.ml` change for `Form`, exactly as this said. See + [`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program", and the short list of what is left of them + below. Macros are what buy `with-drawing` and `with-mode-2d` over raylib's begin/end pairs, the hiccup DSL if a JS backend ever happens, and the removal of special forms from the compiler. @@ -670,8 +696,13 @@ libraries. Four things were settled doing it: needs, since every `defmacro` must be compiled before anything that calls it. `Load.t.pkgs` now comes back in topological order, dependencies first. The *declaration* list is deliberately not -sorted and does not need to be — `check.ml` collects every top-level name before it checks any body — so the order -exists for the expander, which cannot work that way. +sorted and does not need to be — `check.ml` collects every top-level name before it checks any body. + +**The expander did not end up reading that order**, and it is worth saying so rather than leaving the paragraphs above +to imply otherwise. Macros are collected from the prelude and from the file being compiled; a `defmacro` in a package +is refused by name, because reaching one means resolving that package's own imports over `Form`s before `Load` runs. +The order is there and correct and is what package-level macros will read on the day they exist; nothing reads it +today. **Still missing: package visibility.** `rl/get-color-raw` is callable. The blocker is surface syntax, not `load.ml`: `exported` and the refusal machinery already exist and take a second rule in one line, but there is no way for a @@ -1328,142 +1359,46 @@ A package importing a package was on this list and is off it. It loads, a diamon package, the alias clash is refused through a chain as well as inside one file, and a ring is refused by name. What is left of the item is visibility, which is listed above and needs a marker the parser does not have. -## Macros — the reader and the declaration are in, the expander is not +## Macros — landed; what is left of them -The front half landed. What exists: +**The expander works and `unless` is a prelude `defmacro`.** How all of it fits together is in +[`BUILT.md`](BUILT.md), "Macros: the compiler dlopens the program" — the image format, the thunk ABI, why quasiquote +runs before the walk, the two different ways expansion fails to terminate, `-linkall`, and the three cost numbers. +What follows is only the part that is still missing. -- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, `(unquote x)` and `(unquote-splicing x)`, exactly -as `'x` reads as `(quote x)`. It stays dumb: it does not count nesting levels, does not know whether an unquote is -inside a quasiquote, and attaches no meaning to the three names. Clojure's spelling, not Common Lisp's, because a comma -is whitespace in `is_delimiter` and every binding vector in the corpus relies on that. Backtick and tilde are delimiters -now, so `a~b` is two things. -- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say expansion is not wired up; `unquote` and -`unquote-splicing` say they mean nothing outside a quasiquote, which is a mistake rather than a missing feature. -`(defmacro name [params] body ...)` at the top level is checked for shape and *then* refused — a malformed defmacro and -an unimplemented one get different reasons, so the shape rule is enforced before the feature exists. +- **Four special forms left**, and two of them are the hard ones. `until` and `cond` are free to move whenever + somebody wants them. `when` and `dotimes` are not: the prelude itself uses them 29 and 12 times, so moving either + makes the prelude depend on the macro that the macro module has to compile the prelude to get. Breaking that needs + either a prelude that stops using them, or a two-stage prelude where the macro module is built from a subset. The + first is a mechanical edit of `prelude.ml` and is probably the answer. -Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, because a table nothing reads is a place -for a design to rot, and the storage shape is the expander author's first decision, not a decision to inherit. + `cond` has its own snag, and it is the reason `unless` went first: `parse.ml` refuses `(cond a)` with "cond clause + has no body", and a macro cannot produce that (see the next item), so moving `cond` changes an existing test. -### How the expander should work +- **A macro has no error facility**, and this is the biggest gap. A macro runs inside the compiler; anything it + signals aborts the compile with no location. So the prelude's `unless` answers `(unless-takes-a-test-and-a-body)` + when it is handed fewer than two forms, and the report is "unknown name unless-takes-a-test-and-a-body" at the call + site — right place, wrong sentence. What a macro wants is a way to say *this is wrong and here is why*, reported at + the call site. The queued structured-error rewrite is where that belongs, and the call site's `Loc.t` is already + stamped onto everything a macro returns, so the location half is done. -**There is no interpreter** (see "Why there is no interpreter" in `BUILT.md`) and there is not going to be one, so running a macro at -compile time means *compiling it and loading it into the compiler*. That machinery already exists and is measured: -`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the load itself at 0.04ms (see "The reload -primitive"). A macro is that pipeline pointed at the compiler's own process instead of the program's. +- **Macros are not imported.** A `defmacro` in a package is refused by name in `load.ml`. Reaching one would mean + resolving that package's own imports over `Form`s, before `Load` runs — a second import resolver. `programs/pkg-macro.flan`. -The shape it wants: +- **A prelude macro may not call a macro.** The prelude is in every macro module by construction, so there is no + round it could be compiled in after something else. It would fail with an unknown name rather than with a reason, + which is worth fixing the day the prelude wants one. -1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its result is a form, which means `Form.t` -has to exist on the Flan side — a `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and accessors. -That is the real work, and it is bigger than the expander itself: the compiler and the compiled macro have to agree on -the *layout* of a `Form`, not merely its shape, so whatever the checker does for unions has to be exact here. Until -unions are values this cannot start — `check.ml` puts union values and `match` on a union at **milestone 6**, so that is -milestone 6 work landing before milestone 5's. -2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: there is no `Ast.Defmacro` and `Parse` refuses -`defmacro` outright, so an `Ast`-level pass would have nothing to work with. That refusal is not a dead end, it is the -ordering — the expander runs first and `Parse` never sees a macro call at all. It is also the Clojure ordering, and the -reason a macro expanding to a special form is ordinary rather than a special case. -3. **Order matters and files do not have one.** Top-level names in a package are order-independent everywhere else -(`declared_types`, the constant fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and loaded before a -call to it is expanded. Either collect every `defmacro` in a pre-pass and compile them as one module, or require -definition-before-use for macros specifically and say so in the error. The pre-pass is better and matches how the rest -of the frontend already behaves. -4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a single sweep, and a cycle has to be -detected and named rather than looping. -5. **`gensym` is a runtime function of the compiler**, called by the loaded macro while it runs. It needs a counter that -lives in the compiler process and a name that cannot collide with a reader-produced symbol — the usual trick is a -character no symbol may contain, and this reader now has two new ones it could reserve. Hygiene is settled (plan.org, -open decision 2): deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no `macrolet` until a -concrete use case appears. -6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: `` `(a ~b) `` becomes list-construction -over quoted pieces, with `~@` splicing. Written once, in the expander, over `Form`. +- **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and + not the desugaring. Only a macro that writes a macro wants one. -The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — were owned by other lanes when the front -half landed, which is the only reason the expander is not here too. +- **`gensym`'s counter restarts in a second module.** It lives in the loaded module, and a module is dlopened once per + compiler process, so it is process-wide in practice. The rounds already build more than one module for a program + whose macros call macros, and the fix that day is to seed the counter from the module's index. -### Handoff: the boundary is built and verified-by-compilation, the expander is not written - -A lane stopped here mid-flight. What exists, exactly: - -- **`lib/dynload_stubs.c` and `lib/dynload.ml` — the compiler's own dlopen.** This was the one unvalidated -assumption under the whole design and it is now machinery. OCaml has no dlopen for ELF (`Dynlink` loads OCaml, -not shared objects), and `lib/dune` had no `foreign_stubs`, so "point the reload primitive at the compiler's own -process" was not the small step it reads as. It is `dlopen`/`dlsym`/`dlclose`, a four-argument call into a macro -thunk, `calloc`/`free`, and a peek/poke family — OCaml cannot address raw memory, so a `Form` image is written -into it one field at a time from C. `(c_library_flags (-ldl))` is in `lib/dune`. -- **`Emit.macro_thunk`, and `Emit.program ?macros`.** One thunk per macro: -`void @"flan.macro.NAME"(ptr %args, i64 %n, ptr %out, ptr %xfer)`. It builds the `%slice` from `(args, n)`, -calls the macro, stores the result through `%out`. **Nothing aggregate crosses to C.** This is the correction -that matters and it is not obvious from the diff: the unions lane verified a union's *memory* layout against -clang, which is a different claim from LLVM's calling convention for an aggregate passed or returned **by value** -in hand-written IR. Memory is the only agreement that exists, so the boundary is pointers and scalars only. -- **`Build.macro_module`.** A whole program into a self-contained `.so`: the runtime linked in, no undefined Flan -symbols, `-fPIC` on every object including the `.ll`. Self-contained is what keeps `-rdynamic` off the compiler's -own link. It goes through clang rather than `llc` + `ld -shared` — unlike `Build.shared` — because there are C -objects and a libc to find, which is exactly the part of the driver the dev path skips. Cost is the driver's -~50ms, unmeasured here, paid once per process for the whole macro set. -- **`defunion Form` and the list-building surface, in `prelude.ml`.** Written, and the compiler builds; **not yet -checked against a program, so its layout is unverified.** That is the first thing to do. - -Two gates were checked before any of this and both pass, which saves re-deriving them: - -- **`check_finite` does not recurse through `Types.Slice`**, only through `Named`, `Array` and `Option`. So a -union case holding `[Form]` is accepted and `Form` needs no `(Ptr Form)` indirection. -- **The default allocator needs no init.** `flan_ctx_alloc = &flan_heap` is statically initialised in -`flan_rt.c`, so a module with no `main` can allocate. `flan_rt_init` is only argv. - -**The layout the two sides have to agree on.** `Form` mirrors `Form.value`, **not** `Form.t` — there is no `loc` -field, deliberately. A macro cannot invent a source location, so the unmarshaller stamps the *call site's* -`Loc.t` onto every node of what a macro returns; that is the structural answer to "keep the call site's location -attached to what a macro produces", and it is what the queued structured-error work reads. The cases are -`Sym Kw Int Float Str Byte List Vec Map` and **case order is tag order**, so the list is a layout contract with -the marshaller and may not be reordered. The widest cases are `string` and `[Form]`, both `%slice` = 16 bytes -align 8, so the expected shape is `{ i32 tag, [2 x i64] payload }`: **24 bytes, align 8, payload at offset 8**. -Those three numbers are the whole agreement and **they are asserted nowhere yet** — the next commit should put -them through the same `ptrtoint` layout oracle the unions lane used, not hardcode them on faith. - -**What is not written at all:** `lib/expand.ml`. No marshaller, no unmarshaller, no macro collection, no -quasiquote, no fixpoint, no cycle detection. `parse.ml` still refuses `defmacro`, `when`/`unless`/`until`/`cond`/ -`dotimes` are still special forms, and the exit criterion is untouched. - -**What the next person should do first**, in this order, committing each: - -1. Write a program that names `Form` and check its layout through the oracle — 24/8/8. `Vec` is also a case name -and `(Vec T)` is also a type application; if the struct-literal arm and the type arm collide, rename the case and -say so, because that is a layout-contract change. -2. Prove the boundary: one Flan file, `(defn id [args [Form]] Form (at args 0))`, through `Build.macro_module`, -`Dynload.dl_open`, `dl_sym "flan.macro.id"`, a hand-laid `Form` in, the same one back. That is the commit that -makes everything above real rather than plausible. -3. Only then the expander. - -Four decisions this lane made that the design in this section did not settle, each of which the next person may -overturn cheaply: - -- **A macro takes one parameter, the slice of argument forms** — `[Form] -> Form` read as a single function type, -not as "one declared parameter per argument". It needs no reader or parser change (`[args]` already passes the -existing shape check) and it gives variadics for free, which `when` and `unless` both need since there is no -`&rest`. -- **The thunk ABI above**, rather than letting `%"Form"` cross to C. -- **`gensym`'s counter lives in the loaded module**, not in the compiler process as this section sketches. The -name is `~g`; `~` is a delimiter now, so no symbol the reader produces can contain one and a gensym cannot -collide. A module is dlopened once per compiler process, so the counter is process-wide in practice; a second -module would restart it, and the fix that day is to seed it from the module's index. -- **The macro module is the prelude plus the program's `defmacro`s, and not the program's own functions.** -Compiling the user's `defn`s into it would mean compiling a program that has not been expanded yet, which is the -chicken-and-egg the pre-pass exists to avoid. The cost is that a macro body may call prelude functions and other -macros and nothing else. Worth revisiting; not worth revisiting first. - -Left deliberately undone and named so nobody hunts for it: `&rest` sugar, an error carrying the expansion it came -from (only the call-site location is preserved, which is the part that does not make the later work harder), and -the other four special forms. - -### What would tell you it works - -`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` today, and plan.org milestone 5 says -they are special forms *only until macros land*. Moving one of them out of the compiler and into the prelude as a -`defmacro`, with the existing tests unchanged and still green, is the exit criterion — it proves expansion, quasiquote, -`gensym` and the ordering pre-pass at once, against a test suite written before any of them existed. +- **No `&rest` sugar.** A macro takes one parameter, the slice of forms at its call site, and `(len args)` is the + arity. That is deliberate — it is where variadics come from — but a `when` written against it reads worse than + `parse.ml`'s version did. ## Watch for