diff --git a/NEXT.md b/NEXT.md index 7c0c8fc..4d36d9d 100644 --- a/NEXT.md +++ b/NEXT.md @@ -1617,6 +1617,87 @@ Deferred until after the dev loop: `defer`; package visibility, so `rl/get-color-raw` is not callable; a package importing a package; imported unions. +## Macros — the reader and the declaration are in, the expander is not + +The front half landed. What exists: + +- **The reader** reads `` `x ``, `~x` and `~@x` as `(quasiquote x)`, + `(unquote x)` and `(unquote-splicing x)`, exactly as `'x` reads as + `(quote x)`. It stays dumb: it does not count nesting levels, does not know + whether an unquote is inside a quasiquote, and attaches no meaning to the + three names. Clojure's spelling, not Common Lisp's, because a comma is + whitespace in `is_delimiter` and every binding vector in the corpus relies + on that. Backtick and tilde are delimiters now, so `a~b` is two things. +- **`parse.ml` refuses all four by name.** `quasiquote` and `gensym` say + expansion is not wired up; `unquote` and `unquote-splicing` say they mean + nothing outside a quasiquote, which is a mistake rather than a missing + feature. `(defmacro name [params] body ...)` at the top level is checked for + shape and *then* refused — a malformed defmacro and an unimplemented one get + different reasons, so the shape rule is enforced before the feature exists. + +Nothing is stored. There is deliberately no macro table and no `Ast.Defmacro`, +because a table nothing reads is a place for a design to rot, and the storage +shape is the expander author's first decision, not a decision to inherit. + +### How the expander should work + +**There is no interpreter** (see "Why there is no interpreter") and there is not +going to be one, so running a macro at compile time means *compiling it and +loading it into the compiler*. That machinery already exists and is measured: +`Emit.redefinition` → `Build.shared` → `dlopen` is ~19ms end to end, with the +load itself at 0.04ms (see "The reload primitive"). A macro is that pipeline +pointed at the compiler's own process instead of the program's. + +The shape it wants: + +1. **A macro is a function `[Form] -> Form`.** Its parameters are forms and its + result is a form, which means `Form.t` has to exist on the Flan side — a + `defunion` mirroring `lib/form.ml`, in the prelude, plus constructors and + accessors. That is the real work, and it is bigger than the expander itself: + the compiler and the compiled macro have to agree on the *layout* of a + `Form`, not merely its shape, so whatever the checker does for unions has to + be exact here. Until unions are values this cannot start — `check.ml` puts + union values and `match` on a union at **milestone 6**, so that is milestone + 6 work landing before milestone 5's. +2. **Expansion runs over `Form`, before `Parse`.** Not a pass over `Ast`: + there is no `Ast.Defmacro` and `Parse` refuses `defmacro` outright, so an + `Ast`-level pass would have nothing to work with. That refusal is not a dead + end, it is the ordering — the expander runs first and `Parse` never sees a + macro call at all. It is also the Clojure ordering, and the reason a macro + expanding to a special form is ordinary rather than a special case. +3. **Order matters and files do not have one.** Top-level names in a package + are order-independent everywhere else (`declared_types`, the constant + fixpoint in `check.ml`). Macros cannot be: a macro must be compiled and + loaded before a call to it is expanded. Either collect every `defmacro` in a + pre-pass and compile them as one module, or require definition-before-use for + macros specifically and say so in the error. The pre-pass is better and + matches how the rest of the frontend already behaves. +4. **A macro's own body may call macros**, so the pre-pass is a fixpoint, not a + single sweep, and a cycle has to be detected and named rather than looping. +5. **`gensym` is a runtime function of the compiler**, called by the loaded + macro while it runs. It needs a counter that lives in the compiler process + and a name that cannot collide with a reader-produced symbol — the usual + trick is a character no symbol may contain, and this reader now has two new + ones it could reserve. Hygiene is settled (plan.org, open decision 2): + deliberately non-hygienic, Common Lisp/Clojure style, explicit `gensym`, no + `macrolet` until a concrete use case appears. +6. **Quasiquote itself is a macro-shaped desugaring**, not a compiler feature: + `` `(a ~b) `` becomes list-construction over quoted pieces, with + `~@` splicing. Written once, in the expander, over `Form`. + +The four files this touches — `build.ml`, `check.ml`, `emit.ml`, `load.ml` — +were owned by other lanes when the front half landed, which is the only reason +the expander is not here too. + +### What would tell you it works + +`when`, `unless`, `until`, `cond` and `dotimes` are special forms in `parse.ml` +today, and plan.org milestone 5 says they are special forms *only until macros +land*. Moving one of them out of the compiler and into the prelude as a +`defmacro`, with the existing tests unchanged and still green, is the exit +criterion — it proves expansion, quasiquote, `gensym` and the ordering pre-pass +at once, against a test suite written before any of them existed. + ## Watch for The rule that caught the two misparse bugs applies unchanged: **anything that diff --git a/lib/parse.ml b/lib/parse.ml index c5db730..dc166f4 100644 --- a/lib/parse.ml +++ b/lib/parse.ml @@ -266,6 +266,32 @@ and form f mk (head : Form.t) (args : Form.t list) : Ast.expr = and they need argument marshalling and a runtime arity check that \ this version does not do") + (* ── macros ────────────────────────────────────────────────────── *) + (* The reader now produces these three, so they arrive here as ordinary heads + and would fall through to Call — coming back from the checker as "unknown + name quasiquote", which says nothing about what is actually missing. *) + | Sym "quasiquote" -> + fail f "`x is read, but not expanded: macro expansion is not wired up yet \ + (NEXT.md says what it needs)" + + (* Not a milestone, a mistake: these two mean nothing anywhere else, and the + reader cannot tell, because it does not track where it is. *) + | Sym "unquote" -> + fail f "~x means nothing outside a quasiquote" + | Sym "unquote-splicing" -> + fail f "~@x means nothing outside a quasiquote, and splices only into a \ + list or a vector" + + | Sym "defmacro" -> + fail f "defmacro is a top-level declaration, not an expression" + + (* Neither a reader token nor a special form: an ordinary function that a + macro body calls while the macro runs. There is nowhere for it to run + yet, so it says that rather than arriving as an unknown name. *) + | Sym "gensym" -> + fail f "gensym is only meaningful inside a macro body, and macro expansion \ + is not wired up yet (NEXT.md says what it needs)" + (* Recognised, deliberately unimplemented. Rejected rather than left to fall through to Call, where they would parse and mean nothing. *) | Sym ("handler-case" @@ -275,7 +301,7 @@ and form f mk (head : Form.t) (args : Form.t list) : Ast.expr = the restart stack without committing to one. *) | "find-restart" | "compute-restarts" | "errdefer" | "with-allocator" | "loop" | "recur" - | "defmacro" | "await" as name) -> + | "await" as name) -> fail f "%s is not implemented yet (see the build sequence in plan.org)" name (* ── field access: (.pos c) ────────────────────────────────────── *) @@ -480,6 +506,28 @@ let rec decl types (f : Form.t) : Ast.decl = | [ n; t; v ] -> mk (Ast.Defconst (sym n, Some (texpr t), expr v)) | _ -> fail f "defconst is (defconst name Type? value)") + (* Checked for shape and then refused, which is deliberate. Getting the shape + wrong and getting the whole feature are two different mistakes, and a + "defmacro is (defmacro ...)" that only ever fired after expansion landed + would be a rule nothing enforced in the meantime. + + The refusal is not about parsing. Expanding a macro means running it, and + there is no interpreter — the compiled path is the only backend. So it + means compiling the macro and dlopening it into the compiler, which is + what Emit.redefinition and Build.shared already do for the dev loop. + NEXT.md writes down how that goes together. *) + | List ({ v = Sym "defmacro"; _ } :: args) -> + (match args with + | n :: { v = Form.Vec ps; _ } :: body when body <> [] -> + let name = sym n in + List.iter (fun (p : Form.t) -> ignore (sym p)) ps; + fail f + "defmacro %s parses, but is not expanded: running a macro means \ + compiling it and loading it into the compiler, which is not wired \ + up yet (NEXT.md says what it needs)" name + | _ -> + fail f "defmacro is (defmacro name [param ...] body ...)") + | List ({ v = Sym s; _ } :: _) -> fail f "unknown top-level form (%s ...)" s | _ -> fail f "expected a top-level declaration, found %s" (Form.to_string f) diff --git a/lib/reader.ml b/lib/reader.ml index 6ab0f74..803a1b1 100644 --- a/lib/reader.ml +++ b/lib/reader.ml @@ -7,9 +7,15 @@ restart names are quoted symbols ([(invoke-restart 'skip-form)]) and without it the apostrophe would silently become part of the symbol's name. - Not handled yet: quasiquote/unquote (milestone 5, with macros) and metadata - ([^:async]). Metadata is rejected rather than read as a symbol, so it cannot - rot into a silently-wrong name the way quote would have. *) + [`x], [~x] and [~@x] read as [(quasiquote x)], [(unquote x)] and + [(unquote-splicing x)] by the same rule: the reader stays dumb, and what + those names mean is settled later. Clojure's spelling rather than Common + Lisp's, because [,] is already whitespace here (see [is_delimiter]) and + every binding vector in the corpus relies on that. + + Not handled yet: metadata ([^:async]). It is rejected rather than read as a + symbol, so it cannot rot into a silently-wrong name the way quote would + have. *) type state = { src : string; @@ -35,9 +41,13 @@ let advance st = end (* Symbol constituents. Note '-' and '?' and '!' and '/' and '.' are all - ordinary: `empty-at?`, `rl/draw-fps`, `.pos`, `->>` are single symbols. *) + ordinary: [empty-at?], [rl/draw-fps], [.pos], [->>] are single symbols. + + '`' and '~' end a symbol, so [~x] is two things and never one name. That is + the same guard the apostrophe wants and does not have; the corpus has no + symbol containing either character, so closing the class costs nothing. *) let is_delimiter = function - | '(' | ')' | '[' | ']' | '{' | '}' | '"' | ';' | '\000' -> true + | '(' | ')' | '[' | ']' | '{' | '}' | '"' | ';' | '`' | '~' | '\000' -> true | c -> c = ' ' || c = '\t' || c = '\n' || c = '\r' || c = ',' let is_digit c = c >= '0' && c <= '9' @@ -154,10 +164,12 @@ let rec read_form st = | ')' | ']' | '}' as c -> Loc.fail loc "unbalanced %C" c | '"' -> read_string st | '\\' -> read_byte st - | '\'' -> + | '\'' -> read_sugar st loc "quote" + | '`' -> read_sugar st loc "quasiquote" + | '~' -> advance st; - let quoted = read_form st in - Form.make (Form.List [ Form.make (Form.Sym "quote") loc; quoted ]) loc + if peek st = '@' then (advance st; read_wrapped st loc "unquote-splicing") + else read_wrapped st loc "unquote" | '^' -> Loc.fail loc "metadata (^) is not supported yet" @@ -165,6 +177,15 @@ let rec read_form st = | ('-' | '+') when is_digit (peek2 st) -> read_number st | _ -> read_symbol_or_keyword st +(* One sigil character, then the form it applies to, wrapped in a name. The + name's location is the sigil's, so an error inside the wrapper points at the + character the reader saw rather than at the form after it. *) +and read_sugar st loc name = advance st; read_wrapped st loc name + +and read_wrapped st loc name = + let inner = read_form st in + Form.make (Form.List [ Form.make (Form.Sym name) loc; inner ]) loc + and read_seq st open_c loc = advance st; let want = closer open_c in diff --git a/test/test_flan.ml b/test/test_flan.ml index 9bf26b7..7b1c6d3 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -11,6 +11,11 @@ let check name cond = Printf.printf "FAIL %s\n" name end +let contains hay needle = + let n = String.length needle and h = String.length hay in + let rec go i = i + n <= h && (String.sub hay i n = needle || go (i + 1)) in + n = 0 || go 0 + let reads name src expected = match Reader.read_all ~file:"" src with | forms -> @@ -25,10 +30,18 @@ let reads name src expected = Printf.printf "FAIL %s\n src: %s\n error: %s: %s\n" name src (Loc.to_string loc) msg -let rejects name src = +(* [needle] is the point: a read error that fires for the wrong reason is not + the test passing. Without it "backtick at end of input" would be green even + if the backtick were still an ordinary symbol character. *) +let rejects ?needle name src = match Reader.read_all ~file:"" src with | _ -> incr failures; Printf.printf "FAIL %s: expected a read error\n" name - | exception Loc.Error _ -> () + | exception Loc.Error (_, msg) -> + (match needle with + | Some n when not (contains msg n) -> + incr failures; + Printf.printf "FAIL %s\n error: %s\n wanted: ...%s...\n" name msg n + | _ -> ()) let () = (* ── Atoms ─────────────────────────────────────────────────────── *) @@ -77,16 +90,46 @@ let () = "(invoke-restart (quote use-placeholder))"; reads "quote list" "'(a b)" "(quote (a b))"; + (* ── Quasiquote ────────────────────────────────────────────────── *) + (* The bug this closes: a backtick was an ordinary symbol character, so + `(a b) came back as the unknown name "`" — the apostrophe's old failure + mode, still open one sigil over. Clojure's ` ~ ~@ rather than Common + Lisp's ` , ,@ because a comma is whitespace here and every binding vector + depends on that. *) + reads "quasiquote list" "`(a b)" "(quasiquote (a b))"; + reads "unquote" "`(a ~b)" "(quasiquote (a (unquote b)))"; + reads "unquote-splicing" "`(a ~@bs)" "(quasiquote (a (unquote-splicing bs)))"; + reads "unquote a call" "`(+ ~(f x) 1)" + "(quasiquote (+ (unquote (f x)) 1))"; + (* Nesting: the reader does not count levels, it just wraps again. Which + level an unquote belongs to is the expander's problem, not the reader's. *) + reads "nested quasiquote" "`(a `(b ~c))" + "(quasiquote (a (quasiquote (b (unquote c)))))"; + (* An unquote outside any quasiquote still reads. It has to: the reader is + dumb and has no idea where it is. Parse refuses it — see the parse tests. *) + reads "unquote alone" "~x" "(unquote x)"; + reads "splice alone" "~@x" "(unquote-splicing x)"; + (* A quote inside a quasiquote stays a quote; the two sigils do not merge. *) + reads "quote in quasi" "`(a 'b)" "(quasiquote (a (quote b)))"; + (* The delimiter half of the fix: without it ~x is one symbol named "~x". *) + reads "tilde ends a name" "(f a~b)" "(f a (unquote b))"; + reads "backtick ends a name" "(f a`b)" "(f a (quasiquote b))"; + reads "backtick in vec" "[`a ~b]" "[(quasiquote a) (unquote b)]"; + (* The whole class: no reader-significant character may end up inside a name. *) let rec bad_names f = let open Form in match f.v with | Sym s | Kw s -> - if String.exists (fun c -> c = '\'' || c = '^') s then [ s ] else [] + if String.exists (fun c -> c = '\'' || c = '^' || c = '`' || c = '~') s + then [ s ] else [] | List l | Vec l | Map l -> List.concat_map bad_names l | _ -> [] in - let corpus = "(invoke-restart 'skip-form) (a 'b [c 'd] {:e 'f}) '(g 'h)" in + let corpus = + "(invoke-restart 'skip-form) (a 'b [c 'd] {:e 'f}) '(g 'h) \ + `(i ~j ~@k) `(l `(m ~n)) [`o ~p] {:q `r} (f a~b x`y)" + in check "no sigils leak into names" (bad_names (Form.make (Form.List (Reader.read_all ~file:"" corpus)) Loc.unknown) = []); @@ -100,6 +143,11 @@ let () = rejects "unknown char" "\\bogus"; rejects "metadata" "^:async"; rejects "dangling quote" "'"; + (* Each of these asserts the reason, not merely that something failed. *) + rejects "backtick at end" "`" ~needle:"unexpected end of input"; + rejects "tilde at end" "~" ~needle:"unexpected end of input"; + rejects "splice at end" "~@" ~needle:"unexpected end of input"; + rejects "quasiquote unclosed" "`(a b" ~needle:"unclosed"; (* ── Locations ─────────────────────────────────────────────────── *) (match Reader.read_all ~file:"f.flan" "(a)\n (b)" with @@ -132,10 +180,19 @@ let parse_decl src = | [ f ] -> Parse.decl f | _ -> failwith "test source must be exactly one form" -let parse_rejects name src = +(* [needle] again: the house rule is that an unimplemented form is refused by + name with the reason, so a test that only proves *something* failed does not + observe the rule it is there for. *) +let parse_rejects ?needle name src = match Reader.read_all ~file:"" src |> Parse.program with | _ -> incr failures; Printf.printf "FAIL %s: expected a parse error\n" name - | exception Loc.Error _ -> () + | exception Loc.Error (_, msg) -> + (match needle with + | Some n when not (contains msg n) -> + incr failures; + Printf.printf "FAIL %s: wrong reason\n wanted: %s\n got: %s\n" + name n msg + | _ -> ()) let () = let open Ast in @@ -244,7 +301,34 @@ let () = parse_rejects "handler-bind" "(handler-bind [E h] body)"; parse_rejects "restart-case" "(restart-case body (r [] 1))"; parse_rejects "loop/recur" "(loop [x 1] (recur x))"; - parse_rejects "defmacro" "(defmacro m [] 1)"; + + (* ── Macros: the front half is here, the expander is not ───────── *) + (* Was "unknown top-level form (defmacro ...)" — refused, but not by name and + with no reason, which is the hole the house rule had at the top level. *) + parse_rejects "defmacro declaration" "(defmacro m [x] x)" + ~needle:"not expanded"; + (* Shape and feature are separate mistakes and get separate reasons. *) + parse_rejects "defmacro with no body" "(defmacro m [x])" + ~needle:"defmacro is (defmacro name [param ...] body ...)"; + parse_rejects "defmacro with no params" "(defmacro m x)" + ~needle:"defmacro is (defmacro name [param ...] body ...)"; + parse_rejects "defmacro with a non-name param" "(defmacro m [1] x)" + ~needle:"expected a name"; + parse_rejects "defmacro in expression position" "(defn f [] (defmacro m [] 1))" + ~needle:"top-level declaration"; + + (* The reader now hands these three to the parser, so each says what is + actually wrong rather than arriving at the checker as an unknown name. *) + parse_rejects "quasiquote in a function" "(defn f [] `(a b))" + ~needle:"not expanded"; + (* Not a missing feature — an unquote outside a quasiquote is a mistake, and + the reader cannot catch it because it does not track where it is. *) + parse_rejects "unquote outside a quasiquote" "(defn f [] ~x)" + ~needle:"means nothing outside a quasiquote"; + parse_rejects "splice where a splice makes no sense" "(defn f [] (+ 1 ~@xs))" + ~needle:"splices only into a list or a vector"; + parse_rejects "gensym outside a macro" "(defn f [] (gensym))" + ~needle:"only meaningful inside a macro body"; (* ── Malformed syntax is caught with a location ────────────────── *) parse_rejects "odd let bindings" "(let [a])";