JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

A prelude function may call a prelude macro, and the fix was not ordering

Browse Source
This commit is contained in:
Joseph Ferano 2026-09-12 22:19:27 +07:00
parent a26469894e
commit e24aee5120
5 changed files with 217 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
BUILT.md
View File

@ -2733,6 +2733,57 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla
`-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch,
so it is what would catch one of these `Vec`s being used after the arena under it was released.
## A prelude function may call a prelude macro, and why the fix was not ordering
The handoff said the prelude is never macro-expanded — `Macro.program` runs over the file being compiled and the
prelude arrives later through `Check.program`'s prepend — and that the fix was to move the prepend before expansion.
**Both halves of that are wrong, and the measurement is one command.**
Put `(clamp prec 0 9)` back into `format-f64`, print `names` and `List.length extra` on entry to `Macro.compile`, and
compile any program that calls a macro. The compiler prints `names=[clamp,unless] extra=0` and *then* the arity error
at `<prelude>:1103`. So `compile` was entered: the prelude does reach the expander. The error is raised by the
`Check.program` **inside** `compile`, where `building` is true and expansion is off.
That is the real shape, and it is a **cycle, not an ordering**: a macro module is compiled *from* the prelude, so a
prelude function that calls a macro would have to be compiled into the very module that expands it. Moving the
prepend earlier changes which pass sees the prelude first and leaves the cycle exactly where it was.
Two things break it, and the second is the one that matters.
**The prelude's macros are dropped from `mine`.** `Macro.program` collected every `defmacro` in the forms it was given
and handed them back as `extra` — the forms a macro module is built *in addition to* the prelude. When the forms it
was given *are* the prelude, that is the prelude's macros declared twice, refused as a redefinition. They are already
in `prelude`, which is where the module gets them from.
**`Macro.reduce`: for that one build, the prelude is smaller.** Every `defn` that names a macro is dropped, and then
every `defn` that names a dropped one, to a fixpoint — a function calling something unbuildable is as unbuildable as
the thing it calls. `Prelude.bootstrap` is the hook, a ref rather than a parameter because the readers are
`Check.program` and `Parse.prelude_types` and neither can be told.
Only `defn`s are dropped, and that restriction is load-bearing rather than tidy. `Parse.prelude_types` **memoises**,
and it can be forced for the first time inside a bootstrap build; a reduced set of types cached there would be wrong
for every compile afterwards. A `defstruct`, `defunion`, `defalias`, `defenum` or `defvar` therefore stays whatever it
names.
**The one restriction that stays, and now names itself.** A prelude macro may not call a macro — the module that
expands it is compiled from the prelude, so there is no earlier module for its own call to have been expanded by.
That was already recorded and accepted; what it used to do was fail as an unknown name somewhere inside a clang
driver. `reduce` checks it directly and refuses with the macro's name and the reason.
`format-f64` is written `(clamp prec 0 9)` now, which is the living proof and also the only place in the prelude that
exercises it. The expansion is `(min 9 (max 0 prec))`, so nothing about the output moved — the point is that the call
compiles at all.
**What it costs.** The prelude names a macro now, so `Macro.program`'s short-circuit — the reason a build using no
macro pays nothing — no longer fires for the prelude, and every `Check.program` dlopens a macro module. The module is
disk-cached under a digest of the prelude source with an empty `extra`, so it is one `.so` shared by every build and
every process; `dune test` is unchanged at 20 seconds. The first build after a prelude edit pays one clang driver.
**What was not done.** The two expansions are still separate — the prelude is expanded against the prelude's macros,
the file against the prelude's plus its own. That is not a gap, and it is worth stating so the next lane does not
"fix" it: a file macro is never visible to the prelude, and a prelude macro is already visible to the file, so the two
passes cannot disagree. Expanding them together would buy one fewer `dlopen` and nothing else.
## `defer` may be written in a `let`
The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both

12
NEXT.md
View File

@ -577,11 +577,13 @@ them wants a language decision.
Two smaller findings, both written down beside the code that ran into them:
- **The prelude is never macro-expanded.** `macro.ml`'s pass runs over the file being compiled; the prelude reaches
the checker through `Check.program`'s own prepend and never goes through the expander. So a prelude *function*
calling a prelude *macro* resolves the macro's underlying `defn` — the one taking a `[Form]` — and reports an arity
error. `format-f64` writes `(min 9 (max 0 prec))` where it wanted `clamp`. This is next to, and not the same as,
"a prelude macro may not call a macro" below.
- ~~**The prelude is never macro-expanded.**~~ **Fixed, and the diagnosis above was wrong in both halves** — see
[`BUILT.md`](BUILT.md), "A prelude function may call a prelude macro". The prelude *does* reach the expander; the
arity error came from the `Check.program` *inside* `Macro.compile`, where expansion is off. It is a cycle and not
an ordering — a macro module is compiled from the prelude — so moving the prepend would have changed nothing. What
fixed it is `Macro.reduce`, which makes the prelude smaller for that one build, plus dropping the prelude's own
macros from the forms fed back as `extra`. `format-f64` is `(clamp prec 0 9)` now. "A prelude macro may not call a
macro" stands and names itself when violated.
- **A returned `Vec` is a move, and the dead set spans the function**, so an early `(return v)` on one branch kills
the binding for the `v` at the foot of another. `replace-bytes` guards its empty-needle case with an `if` rather

87
lib/macro.ml
View File

@ -65,17 +65,84 @@ let key (extra : Form.t list) =
before the build was entered. *)
let building = ref false
(* ── The bootstrap, and what a prelude macro may not call ───────────
[Check.program] prepends the prelude to every program, this one included, so
the module that expands the prelude's macros is compiled *from* the prelude.
A prelude function that calls a macro therefore cannot be compiled into it:
the call is a name nothing defines yet. That is a cycle and not an ordering
mistake — no amount of moving the prepend around removes it.
It is broken at one level, which is the restriction already recorded and
kept: a macro module is built from the prelude with every [defn] that
depends on a macro *removed*. Directly or transitively, because a function
calling a dropped one is as unbuildable as the dropped one itself.
Only [defn]s are dropped. A [defstruct], [defunion], [defalias], [defenum]
or [defvar] stays whatever it names, so [Parse.prelude_types] sees the same
set of types during a bootstrap build as outside one — it memoises, and a
reduced answer cached there would be wrong for every later compile.
A [defmacro] that lands in the dropped set is the violation of the rule, and
it is refused here by name rather than reaching clang as an unknown symbol. *)
let head_name (f : Form.t) =
match f.Form.v with
| Form.List ({ Form.v = Form.Sym h; _ } :: { Form.v = Form.Sym n; _ } :: _) ->
Some (h, n)
| _ -> None
let reduce (forms : Form.t list) : Form.t list =
let macros = macros_in forms in
(* Fixpoint: a form is out once it names something already out. Bounded by
the number of forms, since the set only grows. *)
let out = ref macros in
let changed = ref true in
while !changed do
changed := false;
List.iter
(fun f ->
match head_name f with
| Some (("defn" | "defmacro"), n) when not (List.mem n !out) ->
if names_macro !out f then begin out := n :: !out; changed := true end
| _ -> ())
forms
done;
(* The macros themselves are in [out] by construction; a macro that is there
for any *other* reason called one, which is the thing that cannot work. *)
List.iter
(fun f ->
match head_name f with
| Some ("defmacro", n) when names_macro macros f ->
Loc.fail f.Form.loc
"the prelude macro %s calls a macro, and a prelude macro may not: \
the module that expands it is compiled from the prelude, so the \
call would have to be expanded by a module that does not exist \
yet. Call a function instead"
n
| _ -> ())
forms;
List.filter
(fun f ->
match head_name f with
| Some ("defn", n) -> not (List.mem n !out)
| _ -> true)
forms
let compile (names : string list) (extra : Form.t list) : loaded =
let out =
Filename.concat (Build.cachedir ()) ("flan-macros-" ^ key extra ^ ".so")
in
if not (Sys.file_exists out) then begin
building := true;
Prelude.bootstrap := reduce;
Fun.protect
~finally:(fun () -> building := false)
~finally:(fun () ->
building := false;
Prelude.bootstrap := (fun fs -> fs))
(fun () ->
(* [Check.program] prepends the prelude itself, so only the file's
own defmacros go in here. *)
(* [Check.program] prepends the prelude itself — reduced, for the one
build that cannot have all of it — so only the file's own defmacros
go in here. *)
let p = Check.program (Parse.program extra) in
(* Written beside the final name and renamed, so a second process
reading the cache never sees a half-written object. *)
@ -186,7 +253,19 @@ let program (forms : Form.t list) : Form.t list =
if !building then forms
else
let prelude = Lazy.force prelude_macros in
let mine = List.filter_map (fun f -> Option.map (fun n -> (n, f)) (macro_name f)) forms in
(* The prelude's own macros are dropped from [mine], and the reason is that
these forms may *be* the prelude: [Check.program] prepends it, so a
prelude macro handed back as [extra] would be declared twice and refused
as a redefinition. They are already in [prelude], which is where the
module gets them from. *)
let mine =
List.filter_map
(fun f ->
match macro_name f with
| Some n when not (List.mem n prelude) -> Some (n, f)
| _ -> None)
forms
in
let all = prelude @ List.map fst mine in
(* The common case by a wide margin, and the reason a build that uses no
macro pays nothing: a file that calls none costs one scan and no

36
lib/prelude.ml
View File

@ -1111,14 +1111,13 @@ let source = {flan|
;; is almost always a literal, so a refusal would be a run-time condition for a
;; mistake visible in the source.
;;
;; The clamp is written out as (min 9 (max 0 prec)) and not as the `clamp`
;; macro two hundred lines up, and that is a limit rather than a preference:
;; **the prelude is not macro-expanded**. macro.ml's pass runs over the file
;; being compiled, and the prelude reaches the checker through Check.program's
;; own prepend, having never been through the expander — so a prelude function
;; calling a prelude macro resolves the macro's underlying defn, which takes
;; one [Form] argument, and the report is an arity error at the call. It is
;; written down in NEXT.md beside the other macro gaps.
;; It is the `clamp` macro two hundred lines up, and this is the call that
;; proves a prelude function may call a prelude macro — which it could not
;; until macro.ml grew its bootstrap reduction. The cycle it breaks: a macro
;; module is compiled *from* the prelude, so a prelude function calling a macro
;; would have to be compiled into the very module that expands it. For that one
;; build the prelude drops every defn that reaches a macro, this one included.
;; A prelude *macro* may still not call a macro, and says so by name.
;;
;; Three inputs do not have decimal expansions and are named before the cast
;; that would be undefined on them: NaN, which fails every comparison and is
@ -1132,7 +1131,7 @@ let source = {flan|
;; caller that needs the sign of a zero should not be reading it out of text.
(defn format-f64 [x f64 prec i32] (Vec u8)
(let [b (vec-new u8)
p (min 9 (max 0 prec))]
p (clamp prec 0 9)]
(cond
(not (= x x))
(append! (addr b) (bytes "nan"))
@ -1392,4 +1391,21 @@ let source = {flan|
let file = "<prelude>"
let forms () = Reader.read_all ~file source
(* The bootstrap hook, and the whole of why a prelude function may now call a
prelude macro.
[Check.program] prepends this file to every program, and a macro module is
built by running [Check.program] over the prelude — so a prelude function
that calls a macro cannot be compiled *into the very module that would
expand it*. That is a cycle, not an ordering mistake, and it is broken by
making the prelude smaller for exactly the one build that cannot afford it:
while a macro module is being built, [Macro] installs a reduction here that
drops every [defn] depending, directly or transitively, on a macro. A
*macro* that lands in that set is refused by name — see [Macro.reduce].
A ref rather than a parameter because the readers are [Check.program] and
[Parse.prelude_types], neither of which can be told, and because [Macro]
sits above both and cannot be depended on from here. *)
let bootstrap : (Form.t list -> Form.t list) ref = ref (fun fs -> fs)
let forms () = !bootstrap (Reader.read_all ~file source)

50
test/test_flan.ml
View File

@ -1476,6 +1476,56 @@ let () =
| _ -> false
| exception Cjson.Bad _ -> true);
(* ── The prelude's own macro calls, and the bootstrap that allows them ──
A macro module is compiled *from* the prelude, so a prelude function that
calls a prelude macro cannot be in the module that would expand it. The
answer is [Macro.reduce]: for that one build the prelude loses every defn
depending on a macro, directly or transitively. These check the reduction
itself, since the thing it prevents is a cycle and a cycle does not show
up as a wrong answer — it shows up as a build that cannot start. *)
let names_of forms =
List.filter_map
(fun (f : Form.t) ->
match f.Form.v with
| Form.List ({ Form.v = Form.Sym ("defn" | "defmacro"); _ }
:: { Form.v = Form.Sym n; _ } :: _) -> Some n
| _ -> None)
forms
in
let reduced = names_of (Macro.reduce (Prelude.forms ())) in
let full = names_of (Prelude.forms ()) in
check "the reduced prelude drops a defn that calls a macro"
(List.mem "format-f64" full && not (List.mem "format-f64" reduced));
(* The macros survive — they are what the module is being built to export —
and so does everything that does not reach one, which is almost all of it. *)
check "the reduced prelude keeps the macros themselves"
(List.mem "clamp" reduced && List.mem "unless" reduced);
check "the reduced prelude keeps a defn that calls no macro"
(List.mem "join" reduced && List.mem "split" reduced);
(* Transitively: a caller of a dropped function is as unbuildable as the
function, so it goes too. Written against a synthetic prelude rather than
the real one, which has no such chain today. *)
let synth src = Reader.read_all ~file:"<synth>" src in
let chain =
synth
"(defmacro m [args] `(do))\n\
(defn a [] Unit (m))\n\
(defn b [] Unit (a))\n\
(defn c [] Unit (do))\n"
in
check "the reduction is transitive"
(names_of (Macro.reduce chain) = [ "m"; "c" ]);
(* And the one rule that stays: a prelude macro may not call a macro. It used
to fail as an unknown name inside a clang build; it names itself now. *)
let ring = synth "(defmacro m [args] `(do))\n(defmacro n [args] (m args))\n" in
check "a prelude macro calling a macro is refused by name"
(match Macro.reduce ring with
| _ -> false
| exception Loc.Error (_, m) ->
contains m "the prelude macro n calls a macro");
(* ── The acceptance program checks end to end ──────────────────── *)
accepts "calc-me.flan type checks"
(In_channel.with_open_bin "../calc-me.flan" In_channel.input_all);