JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

The reader learns quasiquote, and defmacro says why it does nothing

Browse Source 
Clojure's backtick, tilde and tilde-at rather than Common Lisp's comma forms:
is_delimiter already treats a comma as whitespace and every binding vector in
the corpus assumes it, so freeing the comma would rewrite more of the language
than macros are worth. They read as (quasiquote x), (unquote x) and
(unquote-splicing x), the way 'x already reads as (quote x) - the reader stays
dumb and the meaning is resolved later.

The backtick previously read as an ordinary symbol character, which is exactly
the failure the reader's own header warns about for the apostrophe. Both sigils
are delimiters now, so a~b is two things and can never be one name.

defmacro validates its shape before refusing, because a malformed one and a
well-formed one are different mistakes and deserve different sentences. The
three new reader names are refused by name too, or they would fall through to
Call and come back as unknown name quasiquote. unquote outside a quasiquote is
refused as a mistake rather than as a milestone, since the reader cannot know
where it is.

Nothing is stored: no Ast.Defmacro and no macro table. A new decl variant would
have forced edits to four files other agents hold this session, and a
process-global registry spanning the prelude parse, the package parses and
hundreds of test snippets would make results order-dependent. The storage shape
is the expander author's first decision anyway.

The design note records what the expander needs, and the blocker worth knowing:
a macro is [Form] -> Form, so Form has to be a Flan union whose layout the
compiler and the loaded macro agree on exactly, and union values are milestone
6.
This commit is contained in:
Joseph Ferano 2026-09-11 20:22:03 +07:00
commit 99e59dba9f
4 changed files with 250 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

81
NEXT.md
View File

@ -1617,6 +1617,87 @@ Deferred until after the dev loop:
`defer`; package visibility, so `rl/get-color-raw` is not callable; a
package importing a package; imported unions.
## Macros — the reader and the declaration are in, the expander is not
The front half landed. What exists:
- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`,
`(unquote x)` and `(unquote-splicing x)`, exactly as `'x` reads as
`(quote x)`. It stays dumb: it does not count nesting levels, does not know
whether an unquote is inside a quasiquote, and attaches no meaning to the
three names. Clojure's spelling, not Common Lisp's, because a comma is
whitespace in `is_delimiter` and every binding vector in the corpus relies
on that. Backtick and tilde are delimiters now, so `a~b` is two things.
- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say
expansion is not wired up; `unquote` and `unquote-splicing` say they mean
nothing outside a quasiquote, which is a mistake rather than a missing
feature. `(defmacro name [params] body ...)` at the top level is checked for
shape and *then* refused — a malformed defmacro and an unimplemented one get
different reasons, so the shape rule is enforced before the feature exists.
Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`,
because a table nothing reads is a place for a design to rot, and the storage
shape is the expander author's first decision, not a decision to inherit.
### How the expander should work
**There is no interpreter** (see "Why there is no interpreter") and there is not
going to be one, so running a macro at compile time means *compiling it and
loading it into the compiler*. That machinery already exists and is measured:
`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the
load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline
pointed at the compiler's own process instead of the program's.
The shape it wants:
1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its
result is a form, which means `Form.t` has to exist on the Flan side — a
`defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and
accessors. That is the real work, and it is bigger than the expander itself:
the compiler and the compiled macro have to agree on the *layout* of a
`Form`, not merely its shape, so whatever the checker does for unions has to
be exact here. Until unions are values this cannot start — `check.ml` puts
union values and `match` on a union at **milestone 6**, so that is milestone
6 work landing before milestone 5's.
2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`:
there is no `Ast.Defmacro` and `Parse` refuses `defmacro` outright, so an
`Ast`-level pass would have nothing to work with. That refusal is not a dead
end, it is the ordering — the expander runs first and `Parse` never sees a
macro call at all. It is also the Clojure ordering, and the reason a macro
expanding to a special form is ordinary rather than a special case.
3. **Order matters and files do not have one.** Top-level names in a package
are order-independent everywhere else (`declared_types`, the constant
fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and
loaded before a call to it is expanded. Either collect every `defmacro` in a
pre-pass and compile them as one module, or require definition-before-use for
macros specifically and say so in the error. The pre-pass is better and
matches how the rest of the frontend already behaves.
4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a
single sweep, and a cycle has to be detected and named rather than looping.
5. **`gensym` is a runtime function of the compiler**, called by the loaded
macro while it runs. It needs a counter that lives in the compiler process
and a name that cannot collide with a reader-produced symbol — the usual
trick is a character no symbol may contain, and this reader now has two new
ones it could reserve. Hygiene is settled (plan.org, open decision 2):
deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no
`macrolet` until a concrete use case appears.
6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature:
`` `(a ~b) `` becomes list-construction over quoted pieces, with
`~@` splicing. Written once, in the expander, over `Form`.
The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` —
were owned by other lanes when the front half landed, which is the only reason
the expander is not here too.
### What would tell you it works
`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml`
today, and plan.org milestone 5 says they are special forms *only until macros
land*. Moving one of them out of the compiler and into the prelude as a
`defmacro`, with the existing tests unchanged and still green, is the exit
criterion — it proves expansion, quasiquote, `gensym` and the ordering pre-pass
at once, against a test suite written before any of them existed.
## Watch for
The rule that caught the two misparse bugs applies unchanged: **anything that

50
lib/parse.ml
View File

@ -266,6 +266,32 @@ and form f mk (head : Form.t) (args : Form.t list) : Ast.expr =
and they need argument marshalling and a runtime arity check that \
this version does not do")
(* ── macros ────────────────────────────────────────────────────── *)
(* The reader now produces these three, so they arrive here as ordinary heads
and would fall through to Call — coming back from the checker as "unknown
name quasiquote", which says nothing about what is actually missing. *)
| Sym "quasiquote" ->
fail f "`x is read, but not expanded: macro expansion is not wired up yet \
(NEXT.md says what it needs)"
(* Not a milestone, a mistake: these two mean nothing anywhere else, and the
reader cannot tell, because it does not track where it is. *)
| Sym "unquote" ->
fail f "~x means nothing outside a quasiquote"
| Sym "unquote-splicing" ->
fail f "~@x means nothing outside a quasiquote, and splices only into a \
list or a vector"
| Sym "defmacro" ->
fail f "defmacro is a top-level declaration, not an expression"
(* Neither a reader token nor a special form: an ordinary function that a
macro body calls while the macro runs. There is nowhere for it to run
yet, so it says that rather than arriving as an unknown name. *)
| Sym "gensym" ->
fail f "gensym is only meaningful inside a macro body, and macro expansion \
is not wired up yet (NEXT.md says what it needs)"
(* Recognised, deliberately unimplemented. Rejected rather than left to fall
through to Call, where they would parse and mean nothing. *)
| Sym ("handler-case"
@ -275,7 +301,7 @@ and form f mk (head : Form.t) (args : Form.t list) : Ast.expr =
the restart stack without committing to one. *)
| "find-restart" | "compute-restarts"
| "errdefer" | "with-allocator" | "loop" | "recur"
| "defmacro" | "await" as name) ->
| "await" as name) ->
fail f "%s is not implemented yet (see the build sequence in plan.org)" name
(* ── field access: (.pos c) ────────────────────────────────────── *)
@ -480,6 +506,28 @@ let rec decl types (f : Form.t) : Ast.decl =
| [ n; t; v ] -> mk (Ast.Defconst (sym n, Some (texpr t), expr v))
| _ -> fail f "defconst is (defconst name Type? value)")
(* Checked for shape and then refused, which is deliberate. Getting the shape
wrong and getting the whole feature are two different mistakes, and a
"defmacro is (defmacro ...)" that only ever fired after expansion landed
would be a rule nothing enforced in the meantime.
The refusal is not about parsing. Expanding a macro means running it, and
there is no interpreter — the compiled path is the only backend. So it
means compiling the macro and dlopening it into the compiler, which is
what Emit.redefinition and Build.shared already do for the dev loop.
NEXT.md writes down how that goes together. *)
| List ({ v = Sym "defmacro"; _ } :: args) ->
(match args with
| n :: { v = Form.Vec ps; _ } :: body when body <> [] ->
let name = sym n in
List.iter (fun (p : Form.t) -> ignore (sym p)) ps;
fail f
"defmacro %s parses, but is not expanded: running a macro means \
compiling it and loading it into the compiler, which is not wired \
up yet (NEXT.md says what it needs)" name
| _ ->
fail f "defmacro is (defmacro name [param ...] body ...)")
| List ({ v = Sym s; _ } :: _) -> fail f "unknown top-level form (%s ...)" s
| _ -> fail f "expected a top-level declaration, found %s" (Form.to_string f)

37
lib/reader.ml
View File

@ -7,9 +7,15 @@
restart names are quoted symbols ([(invoke-restart 'skip-form)]) and without
it the apostrophe would silently become part of the symbol's name.
Not handled yet: quasiquote/unquote (milestone 5, with macros) and metadata
([^:async]). Metadata is rejected rather than read as a symbol, so it cannot
rot into a silently-wrong name the way quote would have. *)
[`x], [~x] and [~@x] read as [(quasiquote x)], [(unquote x)] and
[(unquote-splicing x)] by the same rule: the reader stays dumb, and what
those names mean is settled later. Clojure's spelling rather than Common
Lisp's, because [,] is already whitespace here (see [is_delimiter]) and
every binding vector in the corpus relies on that.
Not handled yet: metadata ([^:async]). It is rejected rather than read as a
symbol, so it cannot rot into a silently-wrong name the way quote would
have. *)
type state = {
src : string;
@ -35,9 +41,13 @@ let advance st =
end
(* Symbol constituents. Note '-' and '?' and '!' and '/' and '.' are all
ordinary: `empty-at?`, `rl/draw-fps`, `.pos`, `->>` are single symbols. *)
ordinary: [empty-at?], [rl/draw-fps], [.pos], [->>] are single symbols.
'`' and '~' end a symbol, so [~x] is two things and never one name. That is
the same guard the apostrophe wants and does not have; the corpus has no
symbol containing either character, so closing the class costs nothing. *)
let is_delimiter = function
| '(' | ')' | '[' | ']' | '{' | '}' | '"' | ';' | '\000' -> true
| '(' | ')' | '[' | ']' | '{' | '}' | '"' | ';' | '`' | '~' | '\000' -> true
| c -> c = ' ' || c = '\t' || c = '\n' || c = '\r' || c = ','
let is_digit c = c >= '0' && c <= '9'
@ -154,10 +164,12 @@ let rec read_form st =
| ')' | ']' | '}' as c -> Loc.fail loc "unbalanced %C" c
| '"' -> read_string st
| '\\' -> read_byte st
| '\'' ->
| '\'' -> read_sugar st loc "quote"
| '`' -> read_sugar st loc "quasiquote"
| '~' ->
advance st;
let quoted = read_form st in
Form.make (Form.List [ Form.make (Form.Sym "quote") loc; quoted ]) loc
if peek st = '@' then (advance st; read_wrapped st loc "unquote-splicing")
else read_wrapped st loc "unquote"
| '^' ->
Loc.fail loc "metadata (^) is not supported yet"
@ -165,6 +177,15 @@ let rec read_form st =
| ('-' | '+') when is_digit (peek2 st) -> read_number st
| _ -> read_symbol_or_keyword st
(* One sigil character, then the form it applies to, wrapped in a name. The
name's location is the sigil's, so an error inside the wrapper points at the
character the reader saw rather than at the form after it. *)
and read_sugar st loc name = advance st; read_wrapped st loc name
and read_wrapped st loc name =
let inner = read_form st in
Form.make (Form.List [ Form.make (Form.Sym name) loc; inner ]) loc
and read_seq st open_c loc =
advance st;
let want = closer open_c in

98
test/test_flan.ml
View File

@ -11,6 +11,11 @@ let check name cond =
Printf.printf "FAIL %s\n" name
end
let contains hay needle =
let n = String.length needle and h = String.length hay in
let rec go i = i + n <= h && (String.sub hay i n = needle || go (i + 1)) in
n = 0 || go 0
let reads name src expected =
match Reader.read_all ~file:"<test>" src with
| forms ->
@ -25,10 +30,18 @@ let reads name src expected =
Printf.printf "FAIL %s\n src: %s\n error: %s: %s\n"
name src (Loc.to_string loc) msg
let rejects name src =
(* [needle] is the point: a read error that fires for the wrong reason is not
the test passing. Without it "backtick at end of input" would be green even
if the backtick were still an ordinary symbol character. *)
let rejects ?needle name src =
match Reader.read_all ~file:"<test>" src with
| _ -> incr failures; Printf.printf "FAIL %s: expected a read error\n" name
| exception Loc.Error _ -> ()
| exception Loc.Error (_, msg) ->
(match needle with
| Some n when not (contains msg n) ->
incr failures;
Printf.printf "FAIL %s\n error: %s\n wanted: ...%s...\n" name msg n
| _ -> ())
let () =
(* ── Atoms ─────────────────────────────────────────────────────── *)
@ -77,16 +90,46 @@ let () =
"(invoke-restart (quote use-placeholder))";
reads "quote list" "'(a b)" "(quote (a b))";
(* ── Quasiquote ────────────────────────────────────────────────── *)
(* The bug this closes: a backtick was an ordinary symbol character, so
`(a b) came back as the unknown name "`" — the apostrophe's old failure
mode, still open one sigil over. Clojure's ` ~ ~@ rather than Common
Lisp's ` , ,@ because a comma is whitespace here and every binding vector
depends on that. *)
reads "quasiquote list" "`(a b)" "(quasiquote (a b))";
reads "unquote" "`(a ~b)" "(quasiquote (a (unquote b)))";
reads "unquote-splicing" "`(a ~@bs)" "(quasiquote (a (unquote-splicing bs)))";
reads "unquote a call" "`(+ ~(f x) 1)"
"(quasiquote (+ (unquote (f x)) 1))";
(* Nesting: the reader does not count levels, it just wraps again. Which
level an unquote belongs to is the expander's problem, not the reader's. *)
reads "nested quasiquote" "`(a `(b ~c))"
"(quasiquote (a (quasiquote (b (unquote c)))))";
(* An unquote outside any quasiquote still reads. It has to: the reader is
dumb and has no idea where it is. Parse refuses it — see the parse tests. *)
reads "unquote alone" "~x" "(unquote x)";
reads "splice alone" "~@x" "(unquote-splicing x)";
(* A quote inside a quasiquote stays a quote; the two sigils do not merge. *)
reads "quote in quasi" "`(a 'b)" "(quasiquote (a (quote b)))";
(* The delimiter half of the fix: without it ~x is one symbol named "~x". *)
reads "tilde ends a name" "(f a~b)" "(f a (unquote b))";
reads "backtick ends a name" "(f a`b)" "(f a (quasiquote b))";
reads "backtick in vec" "[`a ~b]" "[(quasiquote a) (unquote b)]";
(* The whole class: no reader-significant character may end up inside a name. *)
let rec bad_names f =
let open Form in
match f.v with
| Sym s | Kw s ->
if String.exists (fun c -> c = '\'' || c = '^') s then [ s ] else []
if String.exists (fun c -> c = '\'' || c = '^' || c = '`' || c = '~') s
then [ s ] else []
| List l | Vec l | Map l -> List.concat_map bad_names l
| _ -> []
in
let corpus = "(invoke-restart 'skip-form) (a 'b [c 'd] {:e 'f}) '(g 'h)" in
let corpus =
"(invoke-restart 'skip-form) (a 'b [c 'd] {:e 'f}) '(g 'h) \
`(i ~j ~@k) `(l `(m ~n)) [`o ~p] {:q `r} (f a~b x`y)"
in
check "no sigils leak into names"
(bad_names (Form.make (Form.List (Reader.read_all ~file:"<test>" corpus))
Loc.unknown) = []);
@ -100,6 +143,11 @@ let () =
rejects "unknown char" "\\bogus";
rejects "metadata" "^:async";
rejects "dangling quote" "'";
(* Each of these asserts the reason, not merely that something failed. *)
rejects "backtick at end" "`" ~needle:"unexpected end of input";
rejects "tilde at end" "~" ~needle:"unexpected end of input";
rejects "splice at end" "~@" ~needle:"unexpected end of input";
rejects "quasiquote unclosed" "`(a b" ~needle:"unclosed";
(* ── Locations ─────────────────────────────────────────────────── *)
(match Reader.read_all ~file:"f.flan" "(a)\n (b)" with
@ -132,10 +180,19 @@ let parse_decl src =
| [ f ] -> Parse.decl f
| _ -> failwith "test source must be exactly one form"
let parse_rejects name src =
(* [needle] again: the house rule is that an unimplemented form is refused by
name with the reason, so a test that only proves *something* failed does not
observe the rule it is there for. *)
let parse_rejects ?needle name src =
match Reader.read_all ~file:"<test>" src |> Parse.program with
| _ -> incr failures; Printf.printf "FAIL %s: expected a parse error\n" name
| exception Loc.Error _ -> ()
| exception Loc.Error (_, msg) ->
(match needle with
| Some n when not (contains msg n) ->
incr failures;
Printf.printf "FAIL %s: wrong reason\n wanted: %s\n got: %s\n"
name n msg
| _ -> ())
let () =
let open Ast in
@ -244,7 +301,34 @@ let () =
parse_rejects "handler-bind" "(handler-bind [E h] body)";
parse_rejects "restart-case" "(restart-case body (r [] 1))";
parse_rejects "loop/recur" "(loop [x 1] (recur x))";
parse_rejects "defmacro" "(defmacro m [] 1)";
(* ── Macros: the front half is here, the expander is not ───────── *)
(* Was "unknown top-level form (defmacro ...)" — refused, but not by name and
with no reason, which is the hole the house rule had at the top level. *)
parse_rejects "defmacro declaration" "(defmacro m [x] x)"
~needle:"not expanded";
(* Shape and feature are separate mistakes and get separate reasons. *)
parse_rejects "defmacro with no body" "(defmacro m [x])"
~needle:"defmacro is (defmacro name [param ...] body ...)";
parse_rejects "defmacro with no params" "(defmacro m x)"
~needle:"defmacro is (defmacro name [param ...] body ...)";
parse_rejects "defmacro with a non-name param" "(defmacro m [1] x)"
~needle:"expected a name";
parse_rejects "defmacro in expression position" "(defn f [] (defmacro m [] 1))"
~needle:"top-level declaration";
(* The reader now hands these three to the parser, so each says what is
actually wrong rather than arriving at the checker as an unknown name. *)
parse_rejects "quasiquote in a function" "(defn f [] `(a b))"
~needle:"not expanded";
(* Not a missing feature — an unquote outside a quasiquote is a mistake, and
the reader cannot catch it because it does not track where it is. *)
parse_rejects "unquote outside a quasiquote" "(defn f [] ~x)"
~needle:"means nothing outside a quasiquote";
parse_rejects "splice where a splice makes no sense" "(defn f [] (+ 1 ~@xs))"
~needle:"splices only into a list or a vector";
parse_rejects "gensym outside a macro" "(defn f [] (gensym))"
~needle:"only meaningful inside a macro body";
(* ── Malformed syntax is caught with a location ────────────────── *)
parse_rejects "odd let bindings" "(let [a])";