JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

A package's macros survive the reload, and the suite runs them

Browse Source 
The feature was built and never tested. Three things were missing.

The two packages holding a ring of macros and a macro that never settles
were not dependencies of the test stanza, so both non-termination
refusals failed on "no package at ..." rather than on their own reason.
They fire, and now the suite sees them fire.

The positive half of the rule had no acceptance case at all -- only the
refusal that pins the bare name. pkg-macro.flan is asserted at three opt
levels and on the dev path, which is where six package macros and the
program's own coexist in one file.

And the dev loop was broken in exactly the way that matters most here.
Session held the imported macro set but *replaced* it on every
evaluation, and the one form C-c C-c sends carries no import -- so
(mac/twice 4) compiled on the build and came back "unknown function" on
the first reload. It unions now. test_session drives two evaluations,
because one proves nothing: the first could have re-supplied the set.

BUILT.md said the expander collects from the prelude and the file being
compiled. It collects from imported packages too, and the refusal's old
reasoning -- that this needed a second import resolver -- was wrong for a
reason worth keeping written down.

Cold build cost roughly doubles for a program importing a package that
declares macros: a macro module is built per round and the package's
rounds are its own. Warm is unchanged at ~70ms.
This commit is contained in:
Joseph Ferano 2026-09-13 17:35:04 +07:00
parent 1657b87e3d
commit 86174531b7
8 changed files with 152 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
BUILT.md
View File

@ -640,6 +640,31 @@ anything imported it, so a program could never be a package; and `main` is a rea
keep everything it calls alive. Writing `sand/main` is refused at the line that wrote it, with the reason — left to the
checker it would be "unknown name", which is true and useless.
**A package may declare a macro, and its name is the package's.** `(import mac "pkgs/mac")` makes `(mac/twice 4)` a
call and leaves `(twice 4)` an unknown name, exactly as for a `defn`: importing a package makes nothing globally
visible, macros included. Inside the package the names are unqualified, the rule every other declaration there follows.
This used to be a refusal, and the refusal's reasoning was wrong in a way worth recording. It said collecting a
package's macros would need that package's imports resolved *at the `Form` level, before `Load` runs* — a second import
resolver — and refused rather than build one. What it did not notice is that **the file being compiled is parsed before
`Load` runs too**, so no shape of this feature could have left import resolution where it was. The phases moved instead
of being duplicated: `Load.program` takes **forms**, reads the import forms out of them with `Load.imports_of`, resolves
them with the one resolver it always had, and only then parses the file — with the packages' macros already in front of
it. `imports_of` reads one shape and recurses into nothing; `import` is still the only thing that walks a package graph.
The acyclic-import rule is what makes the order definite, and it was already there for this reason.
The qualification cannot be the `Ast` rename every other name goes through. By the time `Parse` is finished with a
`defmacro` its quasiquote has been desugared, and a quasiquoted `(begin)` is a `Form.Sym` whose name is a *string in an
argument* — which is precisely the property that makes a quasiquoted call output rather than a dependency, and
precisely what puts it out of a rename's reach. So a package's macros are renamed over the **forms** their author wrote,
before desugaring, where `` `(begin) `` and `(begin)` are still the same shape. Locals shadow there as they do in the
`Ast` rename.
`Parse.imported_macros` carries the set, a ref for the same reason `Parse.expander` is one. It matters to the dev loop:
`C-c C-c` sends one form with no import in sight, so `Session` holds the set and **unions** into it rather than
replacing it — a session that replaced would expand `mac/twice` on the build and answer "unknown function" on the
reload.
Still missing: a package-private marker for anything other than `main`, which is why `rl/get-color-raw` is callable.
The gap is surface syntax and not `load.ml` — `exported` is one predicate and the refusal machinery that points at the
line which tried already exists, so a second rule is a line. What does not exist is any way for a package to *mark* a
@ -2880,7 +2905,9 @@ it at compile time instead of the program calling it at run time.
This is also why there is no macro table. Storage was the question the front half deliberately left open, and the
answer is that there is none: the macro set is recomputed by scanning the top level for the word `defmacro`, which is
the only place it survives, and the compiled artefact is a `.so` keyed by a digest.
the only place it survives, and the compiled artefact is a `.so` keyed by a digest. The top level scanned is the
prelude's, the file's own, **and the imported packages'** — the last arriving through `Parse.imported_macros`, already
qualified under the alias the package was imported as. See "A package may declare a macro" under *Packages*.
### `Form`, and the three numbers
@ -2956,6 +2983,13 @@ So there are two different ways expansion fails to terminate, and they are diffe
problem, so it is bounded at 200 rounds and the failure says which macro ran out, at the call site.
`test/programs/macro-spin.flan`.
Both fire when the macros arrive from an **imported package**, and each has its own case for it —
`test/programs/pkg-macro-ring.flan` and `pkg-macro-spin.flan`. They are worth keeping apart there: a ring has no
compile order and one that never settles has one, and an import must not quietly turn either into the other. The ring
is refused inside the package, at the `defmacro` that closed it, because a package's own macros go through the same
rounds before any importer sees them; the one that never settles is refused at the importer's call site, under the
qualified name — `expanding s/spin did not settle`.
The rounds themselves: round 0 takes every macro whose body names no macro still waiting, round 1 expands the rest
against round 0's module, and a round that takes nothing while macros remain is the ring. The walk is bottom up, so a
macro never sees a call to another macro in what it is handed.
@ -2995,6 +3029,16 @@ were already expanded before the build was entered.
under the object cache and keyed by a digest of the prelude's source plus the file's `defmacro` forms, so it is paid
once per change rather than once per build. Every `flan build` is a fresh process, which is what makes the on-disk
cache rather than a memo the right shape.
- **Importing a package that declares macros moved the cold number and not the warm one.** A macro module is built per
*round*, so a package whose macros have a compile-order dependency among them costs its own rounds on top of the
importer's: `test/programs/pkg-macro.flan` leaves four `flan-macros-*.so` in the cache where `macros.flan` leaves two,
and cold it is about twice the wait. Warm it is the same 70ms, because every one of those modules is keyed and cached
like the first. A program that imports a package declaring **no** macro pays nothing new — the extra work is reading
the import forms, which is a scan of the top level.
Measured on this machine, with the cache warm for the runtime and cold for every macro module, so the numbers are
comparable to each other rather than to the two above: no macro 78ms, imports but no macro 110ms, the file's own
macros 747ms cold and 72ms warm, a package's macros 1507ms cold and 68ms warm.
- A hello-world's binary carries exactly one symbol out of all of this: `flan.gensym-n`, eight bytes. `Reach.link`
drops `unless`, `form-cons`, `form-nil`, `form-append`, `form-rest` and `gensym`, because nothing reachable calls
them.

76
HANDOFF-macros.md
View File

@ -1,76 +0,0 @@
# Handoff: macros in an imported package
**Stopped mid-task when the session ended.** The feature works; the acceptance wiring is unfinished.
Its author's last words were "Works. Now the acceptance wiring." This file was written by the
coordinator from the diff and from running the program, not by the lane, so treat the "what remains"
list as read off the tree rather than as the author's own account.
## State
`dune build` succeeds. `test/programs/pkg-macro.flan` runs and prints `8 12 10 12 10 8 70 60`,
which is a package macro, a package macro that quasiquotes another, and a program macro, all
coexisting. **`dune test` was never run.** That is the first thing to do.
## The shape picked, and why the refusal's reasoning was wrong
The old refusal in `lib/load.ml` said collecting a package's macros would need that package's
imports resolved **at the Form level, before `Load` runs** — a second import resolver — and refused
rather than build one.
The lane found the premise false, and this is the finding worth keeping:
> the *file being compiled* is parsed before `Load` runs too, so no shape of the feature could have
> left import resolution where it was.
So the phases moved instead of being duplicated. **`Load.program` takes forms now**: it reads the
import forms, resolves them with the one resolver it always had, and parses the file with the
packages' macros already in front of it. There is no second resolver.
The acyclic-import guarantee is what makes this sound, and it was already there for this reason —
`load.ml`'s own comment says a definite package order is what the macro expander needs.
## The rule
Package macros arrive **qualified**, exactly as a `defn` does. A program importing the directory as
`mac` writes `(mac/twice 4)`; `(twice 4)` is an unknown name. Nothing becomes globally visible by
importing a package. Inside the package the names are unqualified, which is the rule every other
declaration there follows.
## Files touched
`bin/main.ml`, `lib/load.ml`, `lib/macro.ml`, `lib/parse.ml`, `lib/session.ml`,
`test/programs/pkg-macro.flan`, `test/programs/pkgs/mac/mac.flan`, `test/test_acceptance.ml`,
`test/test_reload.ml`, `test/test_sanitize.ml`, `test/test_session.ml`, `test/test_valgrind.ml`,
`test/test_web.ml`. 447 insertions, 117 deletions.
The test files are mostly signature churn from `Load.program` taking forms — that call is made in
many places and each had to move.
## What remains
1. **Run `dune test --root .` and make it green.** Nothing has run it. `Load.program`'s signature
changed and it is called from most test binaries, so expect breakage that is mechanical rather
than deep.
2. **Finish the acceptance wiring.** `test_acceptance.ml:1423-1427` was the refusal case
(`refuses "a macro in an imported package"`). The diff touches it; confirm it now asserts the
working program and that a *sibling* refusal pins the qualified-name rule — that `(twice 4)`
unqualified is an unknown name. `pkg-macro.flan`'s header says that refusal exists; verify it
does.
3. **Check the non-termination refusals still fire** when the macros come from a package: a ring of
macros is named, a macro that does not settle is bounded. `BUILT.md` records both. Neither was
confirmed under the new path.
4. **Check the dev loop.** A macro imported by the file being edited must still be available on a
`C-c C-c`. `lib/session.ml` is in the diff, so this was at least considered, but nothing proves
it. `test/test_session.ml` and `test/test_dev.ml` show how sessions are driven.
5. **Measure the build cost** against the recorded baseline in `BUILT.md`: 50ms for a build naming
no macro, 310ms cold and 70ms warm for one calling a macro. Whether reading a package's forms
earlier moved those numbers is unknown.
6. **`vendor/raylib` is the customer.** `with-drawing` and `with-mode-2d` over raylib's
`BeginDrawing`/`EndDrawing` and `BeginMode2D`/`EndMode2D`, so an unbalanced pair stops being
possible. A raylib lane tried and could not; that is what prompted this work. Not started here
and deliberately out of scope — that file was held by another lane at the time.
## Not verified
Everything above item 1. The program runs and the build is clean; that is the whole of what is
known to work.

41
NEXT.md
View File

@ -54,9 +54,9 @@ question 4 seriously now that there are two backends that can disagree.
## Tightening the dev workflow — what is already known to want doing
- **Finish the macro branch** (`worktree-agent-a859480edc827ab73`): `dune test` never ran on it.
`HANDOFF-macros.md` on that branch has six items. Its customer is `with-drawing`/`with-mode-2d`,
which removes a class of unbalanced-pair bug from every raylib program.
- ~~**Finish the macro branch**~~ — done. Macros are importable from a package, qualified like a
`defn`, and the suite is green. Its customer, `with-drawing`/`with-mode-2d`, is now unblocked and
still unwritten.
- **`slice-from-ptr`** — queued below. Blocks three raylib examples and leaves the hand-written `Font`
surface with no example caller.
- **`merged_serve`'s 10s warning path** (`lib/dev.ml:2322-2326`), the last item in `HANDOFF-f1.md`.
@ -99,25 +99,32 @@ everything below it is the standing queue and the decision record.
## Unmerged, and deliberately so
**`worktree-agent-a859480edc827ab73` — macros importable from a package.** Builds clean,
`test/programs/pkg-macro.flan` runs and prints the right answers, **but `dune test` was never run** and
the acceptance wiring is unfinished. `HANDOFF-macros.md` on that branch has the six remaining items.
~~**`worktree-agent-a859480edc827ab73` — macros importable from a package.**~~ **Finished.** The suite
is green, the acceptance wiring asserts both halves of the rule, and the dev loop is covered. See
BUILT.md, "A package may declare a macro, and its name is the package's".
The finding in it is worth keeping whatever happens to the code: the old refusal claimed collecting a
package's macros needed a second import resolver at the Form level. It did not. The file being compiled
is parsed before `Load` runs too, so no shape of the feature could have left import resolution where it
was — `Load.program` takes forms now and uses the one resolver that always existed.
The finding is worth keeping: the old refusal claimed collecting a package's macros needed a second
import resolver at the Form level. It did not. The file being compiled is parsed before `Load` runs
too, so no shape of the feature could have left import resolution where it was — `Load.program` takes
forms now and uses the one resolver that always existed.
Two things found while finishing it. `Session` held the imported macro set but **replaced** it on
every evaluation, and the one form `C-c C-c` sends carries no import — so a package macro worked on
the build and was an unknown name on the first reload. It unions now, and `test_session` drives two
evaluations because one proves nothing. And cold build cost roughly doubles for a program importing a
package that declares macros, because a macro module is built per round and the package's rounds are
its own; warm is unchanged at ~70ms.
Its customer is `with-drawing`/`with-mode-2d` over raylib's begin/end pairs, which a lane tried to add
and could not.
and could not. Unblocked, not written.
**`worktree-agent-a4dca263c97eb0b66`** is an older WIP, "the inspector's address root, half wired on
the Emacs side". **Superseded** — the registry lane built that properly. Delete it.
## What I would do next
1. **Finish the macro branch.** Run `dune test`, fix the mechanical breakage from `Load.program`'s
changed signature, then the five other items in its handoff.
1. **`with-drawing` and `with-mode-2d`**, now that a package may declare a macro. They belong in
`vendor/raylib`, and they remove a class of unbalanced-pair bug from every raylib program.
2. **The pointer-length question** — its own queued section below. It blocks three raylib examples and
leaves the hand-written `Font` surface with no example caller.
3. **Generic structs and `$n` array lengths** — queued below, decided, and now unblocked since the
@ -595,10 +602,10 @@ the symbol and gives up the name, so nothing about the C signature is hand-writt
agreement-by-construction with the header. The third is an `exclude` plus a hand-written line, exactly as `SetExitKey`
and `SetMouseCursor` already were.
**Not built: `with-drawing` and `with-mode-2d`.** An unbalanced begin/end is a real bug and a macro removes it, but a
macro cannot live in a package — the expander collects `defmacro`s from the prelude and from the file being compiled,
and one in an imported package is refused by name. `test/programs/pkg-macro.flan` is that refusal and its whole content
is the case. They have to be written in the program that uses them, or wait for macros to be importable.
**Not built: `with-drawing` and `with-mode-2d`.** An unbalanced begin/end is a real bug and a macro removes it. This
used to be blocked — a macro could not live in a package — and it no longer is: the expander collects `defmacro`s from
the prelude, from the file being compiled **and from imported packages**, qualified under the alias. They can be
written in `vendor/raylib` now, and have not been.
## ~~Queued: a restart is not a transaction, and the docs must say so~~ — **landed**

11
lib/session.ml
View File

@ -328,9 +328,14 @@ let eval ?(origin = "<eval>") ?pause t src : change =
the duplicate-name pass would reject it. *)
let incoming =
let l = Load.program ~file:t.file forms in
(* An evaluated import adds to the session's set, so a macro brought in by
C-c C-k is there for the C-c C-c after it. *)
t.macros <- l.Load.macros;
(* An evaluated import *adds* to the session's set, so a macro brought in
by C-c C-k is there for the C-c C-c after it. A union and not an
assignment: [Load.program] answers the macros of the imports it was
handed, and the one form C-c C-c sends has no import in it, so
replacing would empty the set on the first re-evaluation of a defn —
the macro would work on the build and be an unknown name on the
reload. *)
t.macros <- Load.macro_union t.macros l.Load.macros;
let ds = l.Load.decls in
match package_of t origin with
| None -> ds

12
test/dune
View File

@ -41,8 +41,12 @@
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*)
; The package that declares a macro, which a package may not do yet.
; The packages that declare macros: one whose macros a program calls
; qualified, and the two whose macros do not terminate — a ring, and one
; that never settles. Each is its own directory, so each needs its own glob.
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*)
; The synthetic C header the importer's table reads. Committed rather than
; reached for on the machine: the raylib case needs raylib installed, at the
; right version, with a variable set, so it skips everywhere and covers
@ -159,5 +163,9 @@
(glob_files programs/pkgs/draw/*)
(glob_files programs/pkgs/ring-a/*)
(glob_files programs/pkgs/ring-b/*)
(glob_files programs/pkgs/ring-c/*))
(glob_files programs/pkgs/ring-c/*)
; And the macro-declaring packages, for the same reason.
(glob_files programs/pkgs/mac/*)
(glob_files programs/pkgs/macring/*)
(glob_files programs/pkgs/macspin/*))
(action (run ./test_valgrind.exe)))

22
test/test_acceptance.ml
View File

@ -2052,6 +2052,28 @@ ERR@7 unexpected token: not the kind the caller was reading
outputs ~opt:"-O0" "macros, -O0" "programs/macros.flan" macros_out;
outputs ~dev:true "macros, dev" "programs/macros.flan" macros_out;
(* A macro declared in an imported *package*, which is the half the
refusal at [a package's macro is not visible unqualified] above leaves
out. The program calls six of them qualified and one of its own
unqualified, so what is asserted is that the two sets coexist in one
file: a package macro, a package macro that quasiquotes another one and
a function of its package, a macro that really calls another at expand
time, a macro whose output shadows a top-level name, a package function
calling its own macro unqualified, and the program's own macro wrapped
around a package's.
Three opt levels for the reason the case above has them -- the
expansion is finished before the optimiser exists -- and a dev build
because the dev path is this project's priority and reads the same
collected set. *)
let pkg_macro_out = "8\n12\n10\n12\n10\n8\n70\n60\n" in
outputs "a macro in an imported package" "programs/pkg-macro.flan"
pkg_macro_out;
outputs ~opt:"-O0" "a macro in an imported package, -O0"
"programs/pkg-macro.flan" pkg_macro_out;
outputs ~dev:true "a macro in an imported package, dev"
"programs/pkg-macro.flan" pkg_macro_out;
(* Function values, the non-escaping kind. Three opt levels because the
indirect call is the one shape LLVM is most likely to devirtualise: at
-O2 a name passed straight down becomes a direct call and the pointer

42
test/test_session.ml
View File

@ -195,6 +195,48 @@ let () =
| exception Loc.Error { Loc.dmsg = m; _ } ->
fail "reloading a file with imports failed: %s" m);
(* A macro that came in with an import is still there on the *second*
evaluation, which is the C-c C-c case and the one that can quietly break.
A package's macros are collected by [Load.program] from the import forms
it is handed, and the single form an editor sends has no import in it —
so a session that replaced its set instead of adding to it would expand
[mac/twice] on the build and answer "unknown function" on the reload.
Two evaluations, because one proves nothing: the first is the C-c C-k
that could have re-supplied the set, the second is the one that has to
work without it.
An unexpanded macro call is an unknown function and therefore an
exception, so reaching the assertions at all is what says the expansion
happened. *)
let tm, _ = Session.create ~file:"programs/pkg-macro.flan" () in
(match Session.eval ~origin:"programs/pkg-macro.flan" tm
"(defn twiced [] i32 (mac/twice 21))"
with
| c ->
if not (List.mem "twiced" c.Session.fns) then
fail "a package macro on the first evaluation reported %s"
(String.concat " " c.Session.fns)
| exception Loc.Error { Loc.dmsg = m; _ } ->
fail "a package macro on the first evaluation: %s" m);
(match Session.eval ~origin:"programs/pkg-macro.flan" tm
"(defn quaded [] i32 (mac/quad 3))"
with
| c ->
if not (List.mem "quaded" c.Session.fns) then
fail "a package macro on the second evaluation reported %s"
(String.concat " " c.Session.fns)
| exception Loc.Error { Loc.dmsg = m; _ } ->
fail "a package macro was forgotten by the second evaluation: %s" m);
(* And the rule holds in the session as it does in a build: the bare name is
not a name here either. *)
(match Session.eval ~origin:"programs/pkg-macro.flan" tm
"(defn bare [] i32 (twice 21))"
with
| _ -> fail "an unqualified package macro was accepted in a session"
| exception Loc.Error { Loc.dmsg = m; _ } ->
if not (has m "twice") then
fail "an unqualified package macro said %S" m);
(* A form typed into a file that is *imported as a package* has to be
qualified the way the import qualified it, or it splices as a brand-new
unrelated name: the evaluation reports success and the running program

1
test/test_valgrind.ml
View File

@ -222,6 +222,7 @@ let corpus =
"programs/maps.flan", [];
"programs/math.flan", [];
"programs/pool-stale-region.flan", [];
"programs/pkg-macro.flan", [];
"programs/pkg-diamond.flan", [];
"programs/pkg-return.flan", [];
"programs/pkg-shadow.flan", [];