diff --git a/BUILT.md b/BUILT.md index 3d41ea2..0e8dea6 100644 --- a/BUILT.md +++ b/BUILT.md @@ -2733,6 +2733,57 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla `-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch, so it is what would catch one of these `Vec`s being used after the arena under it was released. +## A prelude function may call a prelude macro, and why the fix was not ordering + +The handoff said the prelude is never macro-expanded — `Macro.program` runs over the file being compiled and the +prelude arrives later through `Check.program`'s prepend — and that the fix was to move the prepend before expansion. +**Both halves of that are wrong, and the measurement is one command.** + +Put `(clamp prec 0 9)` back into `format-f64`, print `names` and `List.length extra` on entry to `Macro.compile`, and +compile any program that calls a macro. The compiler prints `names=[clamp,unless] extra=0` and *then* the arity error +at `:1103`. So `compile` was entered: the prelude does reach the expander. The error is raised by the +`Check.program` **inside** `compile`, where `building` is true and expansion is off. + +That is the real shape, and it is a **cycle, not an ordering**: a macro module is compiled *from* the prelude, so a +prelude function that calls a macro would have to be compiled into the very module that expands it. Moving the +prepend earlier changes which pass sees the prelude first and leaves the cycle exactly where it was. + +Two things break it, and the second is the one that matters. + +**The prelude's macros are dropped from `mine`.** `Macro.program` collected every `defmacro` in the forms it was given +and handed them back as `extra` — the forms a macro module is built *in addition to* the prelude. When the forms it +was given *are* the prelude, that is the prelude's macros declared twice, refused as a redefinition. They are already +in `prelude`, which is where the module gets them from. + +**`Macro.reduce`: for that one build, the prelude is smaller.** Every `defn` that names a macro is dropped, and then +every `defn` that names a dropped one, to a fixpoint — a function calling something unbuildable is as unbuildable as +the thing it calls. `Prelude.bootstrap` is the hook, a ref rather than a parameter because the readers are +`Check.program` and `Parse.prelude_types` and neither can be told. + +Only `defn`s are dropped, and that restriction is load-bearing rather than tidy. `Parse.prelude_types` **memoises**, +and it can be forced for the first time inside a bootstrap build; a reduced set of types cached there would be wrong +for every compile afterwards. A `defstruct`, `defunion`, `defalias`, `defenum` or `defvar` therefore stays whatever it +names. + +**The one restriction that stays, and now names itself.** A prelude macro may not call a macro — the module that +expands it is compiled from the prelude, so there is no earlier module for its own call to have been expanded by. +That was already recorded and accepted; what it used to do was fail as an unknown name somewhere inside a clang +driver. `reduce` checks it directly and refuses with the macro's name and the reason. + +`format-f64` is written `(clamp prec 0 9)` now, which is the living proof and also the only place in the prelude that +exercises it. The expansion is `(min 9 (max 0 prec))`, so nothing about the output moved — the point is that the call +compiles at all. + +**What it costs.** The prelude names a macro now, so `Macro.program`'s short-circuit — the reason a build using no +macro pays nothing — no longer fires for the prelude, and every `Check.program` dlopens a macro module. The module is +disk-cached under a digest of the prelude source with an empty `extra`, so it is one `.so` shared by every build and +every process; `dune test` is unchanged at 20 seconds. The first build after a prelude edit pays one clang driver. + +**What was not done.** The two expansions are still separate — the prelude is expanded against the prelude's macros, +the file against the prelude's plus its own. That is not a gap, and it is worth stating so the next lane does not +"fix" it: a file macro is never visible to the prelude, and a prelude macro is already visible to the file, so the two +passes cannot disagree. Expanding them together would buy one fewer `dlopen` and nothing else. + ## `defer` may be written in a `let` The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both diff --git a/NEXT.md b/NEXT.md index bdfa208..87d325f 100644 --- a/NEXT.md +++ b/NEXT.md @@ -577,11 +577,13 @@ them wants a language decision. Two smaller findings, both written down beside the code that ran into them: -- **The prelude is never macro-expanded.** `macro.ml`'s pass runs over the file being compiled; the prelude reaches - the checker through `Check.program`'s own prepend and never goes through the expander. So a prelude *function* - calling a prelude *macro* resolves the macro's underlying `defn` — the one taking a `[Form]` — and reports an arity - error. `format-f64` writes `(min 9 (max 0 prec))` where it wanted `clamp`. This is next to, and not the same as, - "a prelude macro may not call a macro" below. +- ~~**The prelude is never macro-expanded.**~~ **Fixed, and the diagnosis above was wrong in both halves** — see + [`BUILT.md`](BUILT.md), "A prelude function may call a prelude macro". The prelude *does* reach the expander; the + arity error came from the `Check.program` *inside* `Macro.compile`, where expansion is off. It is a cycle and not + an ordering — a macro module is compiled from the prelude — so moving the prepend would have changed nothing. What + fixed it is `Macro.reduce`, which makes the prelude smaller for that one build, plus dropping the prelude's own + macros from the forms fed back as `extra`. `format-f64` is `(clamp prec 0 9)` now. "A prelude macro may not call a + macro" stands and names itself when violated. - **A returned `Vec` is a move, and the dead set spans the function**, so an early `(return v)` on one branch kills the binding for the `v` at the foot of another. `replace-bytes` guards its empty-needle case with an `if` rather diff --git a/lib/macro.ml b/lib/macro.ml index 2ff0ea9..916999b 100644 --- a/lib/macro.ml +++ b/lib/macro.ml @@ -65,17 +65,84 @@ let key (extra : Form.t list) = before the build was entered. *) let building = ref false +(* ── The bootstrap, and what a prelude macro may not call ─────────── + [Check.program] prepends the prelude to every program, this one included, so + the module that expands the prelude's macros is compiled *from* the prelude. + A prelude function that calls a macro therefore cannot be compiled into it: + the call is a name nothing defines yet. That is a cycle and not an ordering + mistake — no amount of moving the prepend around removes it. + + It is broken at one level, which is the restriction already recorded and + kept: a macro module is built from the prelude with every [defn] that + depends on a macro *removed*. Directly or transitively, because a function + calling a dropped one is as unbuildable as the dropped one itself. + + Only [defn]s are dropped. A [defstruct], [defunion], [defalias], [defenum] + or [defvar] stays whatever it names, so [Parse.prelude_types] sees the same + set of types during a bootstrap build as outside one — it memoises, and a + reduced answer cached there would be wrong for every later compile. + + A [defmacro] that lands in the dropped set is the violation of the rule, and + it is refused here by name rather than reaching clang as an unknown symbol. *) + +let head_name (f : Form.t) = + match f.Form.v with + | Form.List ({ Form.v = Form.Sym h; _ } :: { Form.v = Form.Sym n; _ } :: _) -> + Some (h, n) + | _ -> None + +let reduce (forms : Form.t list) : Form.t list = + let macros = macros_in forms in + (* Fixpoint: a form is out once it names something already out. Bounded by + the number of forms, since the set only grows. *) + let out = ref macros in + let changed = ref true in + while !changed do + changed := false; + List.iter + (fun f -> + match head_name f with + | Some (("defn" | "defmacro"), n) when not (List.mem n !out) -> + if names_macro !out f then begin out := n :: !out; changed := true end + | _ -> ()) + forms + done; + (* The macros themselves are in [out] by construction; a macro that is there + for any *other* reason called one, which is the thing that cannot work. *) + List.iter + (fun f -> + match head_name f with + | Some ("defmacro", n) when names_macro macros f -> + Loc.fail f.Form.loc + "the prelude macro %s calls a macro, and a prelude macro may not: \ + the module that expands it is compiled from the prelude, so the \ + call would have to be expanded by a module that does not exist \ + yet. Call a function instead" + n + | _ -> ()) + forms; + List.filter + (fun f -> + match head_name f with + | Some ("defn", n) -> not (List.mem n !out) + | _ -> true) + forms + let compile (names : string list) (extra : Form.t list) : loaded = let out = Filename.concat (Build.cachedir ()) ("flan-macros-" ^ key extra ^ ".so") in if not (Sys.file_exists out) then begin building := true; + Prelude.bootstrap := reduce; Fun.protect - ~finally:(fun () -> building := false) + ~finally:(fun () -> + building := false; + Prelude.bootstrap := (fun fs -> fs)) (fun () -> - (* [Check.program] prepends the prelude itself, so only the file's - own defmacros go in here. *) + (* [Check.program] prepends the prelude itself — reduced, for the one + build that cannot have all of it — so only the file's own defmacros + go in here. *) let p = Check.program (Parse.program extra) in (* Written beside the final name and renamed, so a second process reading the cache never sees a half-written object. *) @@ -186,7 +253,19 @@ let program (forms : Form.t list) : Form.t list = if !building then forms else let prelude = Lazy.force prelude_macros in - let mine = List.filter_map (fun f -> Option.map (fun n -> (n, f)) (macro_name f)) forms in + (* The prelude's own macros are dropped from [mine], and the reason is that + these forms may *be* the prelude: [Check.program] prepends it, so a + prelude macro handed back as [extra] would be declared twice and refused + as a redefinition. They are already in [prelude], which is where the + module gets them from. *) + let mine = + List.filter_map + (fun f -> + match macro_name f with + | Some n when not (List.mem n prelude) -> Some (n, f) + | _ -> None) + forms + in let all = prelude @ List.map fst mine in (* The common case by a wide margin, and the reason a build that uses no macro pays nothing: a file that calls none costs one scan and no diff --git a/lib/prelude.ml b/lib/prelude.ml index d2b50bf..9d8b1d8 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -1111,14 +1111,13 @@ let source = {flan| ;; is almost always a literal, so a refusal would be a run-time condition for a ;; mistake visible in the source. ;; -;; The clamp is written out as (min 9 (max 0 prec)) and not as the `clamp` -;; macro two hundred lines up, and that is a limit rather than a preference: -;; **the prelude is not macro-expanded**. macro.ml's pass runs over the file -;; being compiled, and the prelude reaches the checker through Check.program's -;; own prepend, having never been through the expander — so a prelude function -;; calling a prelude macro resolves the macro's underlying defn, which takes -;; one [Form] argument, and the report is an arity error at the call. It is -;; written down in NEXT.md beside the other macro gaps. +;; It is the `clamp` macro two hundred lines up, and this is the call that +;; proves a prelude function may call a prelude macro — which it could not +;; until macro.ml grew its bootstrap reduction. The cycle it breaks: a macro +;; module is compiled *from* the prelude, so a prelude function calling a macro +;; would have to be compiled into the very module that expands it. For that one +;; build the prelude drops every defn that reaches a macro, this one included. +;; A prelude *macro* may still not call a macro, and says so by name. ;; ;; Three inputs do not have decimal expansions and are named before the cast ;; that would be undefined on them: NaN, which fails every comparison and is @@ -1132,7 +1131,7 @@ let source = {flan| ;; caller that needs the sign of a zero should not be reading it out of text. (defn format-f64 [x f64 prec i32] (Vec u8) (let [b (vec-new u8) - p (min 9 (max 0 prec))] + p (clamp prec 0 9)] (cond (not (= x x)) (append! (addr b) (bytes "nan")) @@ -1392,4 +1391,21 @@ let source = {flan| let file = "" -let forms () = Reader.read_all ~file source +(* The bootstrap hook, and the whole of why a prelude function may now call a + prelude macro. + + [Check.program] prepends this file to every program, and a macro module is + built by running [Check.program] over the prelude — so a prelude function + that calls a macro cannot be compiled *into the very module that would + expand it*. That is a cycle, not an ordering mistake, and it is broken by + making the prelude smaller for exactly the one build that cannot afford it: + while a macro module is being built, [Macro] installs a reduction here that + drops every [defn] depending, directly or transitively, on a macro. A + *macro* that lands in that set is refused by name — see [Macro.reduce]. + + A ref rather than a parameter because the readers are [Check.program] and + [Parse.prelude_types], neither of which can be told, and because [Macro] + sits above both and cannot be depended on from here. *) +let bootstrap : (Form.t list -> Form.t list) ref = ref (fun fs -> fs) + +let forms () = !bootstrap (Reader.read_all ~file source) diff --git a/test/test_flan.ml b/test/test_flan.ml index 06025e1..9307230 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -1476,6 +1476,56 @@ let () = | _ -> false | exception Cjson.Bad _ -> true); + (* ── The prelude's own macro calls, and the bootstrap that allows them ── + A macro module is compiled *from* the prelude, so a prelude function that + calls a prelude macro cannot be in the module that would expand it. The + answer is [Macro.reduce]: for that one build the prelude loses every defn + depending on a macro, directly or transitively. These check the reduction + itself, since the thing it prevents is a cycle and a cycle does not show + up as a wrong answer — it shows up as a build that cannot start. *) + let names_of forms = + List.filter_map + (fun (f : Form.t) -> + match f.Form.v with + | Form.List ({ Form.v = Form.Sym ("defn" | "defmacro"); _ } + :: { Form.v = Form.Sym n; _ } :: _) -> Some n + | _ -> None) + forms + in + let reduced = names_of (Macro.reduce (Prelude.forms ())) in + let full = names_of (Prelude.forms ()) in + check "the reduced prelude drops a defn that calls a macro" + (List.mem "format-f64" full && not (List.mem "format-f64" reduced)); + (* The macros survive — they are what the module is being built to export — + and so does everything that does not reach one, which is almost all of it. *) + check "the reduced prelude keeps the macros themselves" + (List.mem "clamp" reduced && List.mem "unless" reduced); + check "the reduced prelude keeps a defn that calls no macro" + (List.mem "join" reduced && List.mem "split" reduced); + + (* Transitively: a caller of a dropped function is as unbuildable as the + function, so it goes too. Written against a synthetic prelude rather than + the real one, which has no such chain today. *) + let synth src = Reader.read_all ~file:"" src in + let chain = + synth + "(defmacro m [args] `(do))\n\ + (defn a [] Unit (m))\n\ + (defn b [] Unit (a))\n\ + (defn c [] Unit (do))\n" + in + check "the reduction is transitive" + (names_of (Macro.reduce chain) = [ "m"; "c" ]); + + (* And the one rule that stays: a prelude macro may not call a macro. It used + to fail as an unknown name inside a clang build; it names itself now. *) + let ring = synth "(defmacro m [args] `(do))\n(defmacro n [args] (m args))\n" in + check "a prelude macro calling a macro is refused by name" + (match Macro.reduce ring with + | _ -> false + | exception Loc.Error (_, m) -> + contains m "the prelude macro n calls a macro"); + (* ── The acceptance program checks end to end ──────────────────── *) accepts "calc-me.flan type checks" (In_channel.with_open_bin "../calc-me.flan" In_channel.input_all);