diff --git a/DISCUSS.org b/DISCUSS.org index 402c35a..9082ebf 100644 --- a/DISCUSS.org +++ b/DISCUSS.org @@ -183,8 +183,208 @@ why MSVC/glibc-style debug allocators use a single repeated byte instead LocalAlloc uninit), 0xFEEEFEEE (Windows HeapFree'd), 0xDEADC0DE, 0xC0FFEE, 0x8BADF00D (Apple watchdog-timeout crash code) +Scope: should take a struct or an array (by place/pointer) and fill it with +the sentinel — general over both, not just raw byte buffers. Ties into the +memcpy discussion (2026-09-20): a struct/array is a first-class aggregate +value in flan today (copies happen via load/store, no memcpy builtin +exists), so this would be the first builtin that reaches into one of those +and overwrites it as raw bytes rather than treating it as a typed value — +same category of operation llvm.memset already does for zero, just exposed +as a real user-facing builtin instead of only an internal codegen detail. + Not designed or implemented, just an idea. +** struct field type change: got a plain type error, not the documented refusal +Hit "expected i32, found u8" after changing Cell's row/col from u8 to i32. +Turned out to be a leftover (u8 (/ my cell-size)) cast at the construction +site that I hadn't updated to i32 yet — an ordinary, correct type error, not +a hot-reload problem. Already fixed in the file. + +Worth remembering: a struct field type change is one of exactly three +redefinitions the session actually refuses outright rather than silently +mis-loading (docs/BUILT.md:1213-1219: "a struct's fields | the values the +process is holding have the old layout"). That refusal has its own distinct +message — if I hit it for real it won't look like an ordinary type mismatch. + +** compiler messages need a persistent, copyable buffer — not just *Messages* +No dedicated log/output buffer for compiler errors exists anywhere in +emacs/flan.el. Today it's only: a transient error overlay +(flan--error-overlays, cleared on the next command) and `message` to +*Messages* (append-only, awkward to copy out of, no structure). Want both +the ghost text AND a persistent buffer that accumulates these, so a message +can actually be copy-pasted (e.g. to hand to Claude) without hunting through +*Messages*. Not designed or implemented. + +** defmacro should support real parameter lists, not just one [Form] arg +Every defmacro today takes exactly one parameter, the whole call's arg list +as [Form] (lib/parse.ml comment: (defmacro m [args] body) is (defn m [args +[Form]] Form body)). Porting a Clojure macro like +(defmacro do-grid [[r rows c cols] & body] ...) means manually picking apart +`args` by hand every time — (at args 0), match on Form.Vec to unwrap a +binding vector, (form-rest args 1) for the body, no arity checking, no +destructuring in the signature itself. with-drawing/with-mode-2d +(vendor/raylib/modes.flan) already do this by hand and it works, but it's +real boilerplate for something Clojure gets for free from defmacro's own +parameter list. Real design gap, not a small fix — would mean parsing macro +params with the same destructuring patterns [let] already has (dvec/dmap, +lib/parse.ml:698-799) plus variadic &body support, applied at the macro +call site before expansion rather than left to the macro body. + +** profiling: use Tracy, don't build our own +No profiling infra or prior discussion exists in FIX.org/plan.org/docs. +Recommendation: bind Tracy (TracyC.h, a pure C API) via declare-c — same +mechanism raylib already goes through. Building a bespoke profiler means +redoing a capture protocol, a viewer, flame graphs and a timeline, all of +which Tracy already does well, for a domain (real-time/game loops) it was +built for. + +Two integration points already fit flan's own design: +- Tracy's frame mark maps directly onto flan's existing frame-boundary + notion — where agent/poll is called each loop iteration. +- flan already has precedent for dev-build-only instrumentation wired in as + an LLVM constructor (the allocation registry, lib/emit.ml ~4165). Automatic + per-function Tracy zones in dev builds, using the same dev/cell + indirection every call already goes through, could follow that pattern + instead of requiring hand-instrumentation everywhere. + +Alternatives considered: +- Optick — similar to Tracy but less active, historically Windows-first. No + real advantage over Tracy. +- Superluminal — great UI, but commercial and Windows-only. Ruled out (Linux). +- perf/Hotspot — free, zero-instrumentation sampling, but not frame-aware, + not live-viewable, Linux-only. Fine as a second opinion, not primary. +- Chrome Trace Event Format / Perfetto (or a tiny header-only exporter for + it) — much smaller lift than Tracy: push timestamped begin/end onto a + buffer, dump JSON, view in Perfetto/chrome://tracing after the fact. No + live view, no socket server — but it's the one option with an actual story + for wasm32 (plan.org's committed third target: write trace to a buffer, + pull out via JS). Tracy's capture side assumes a real OS socket the viewer + connects to, which doesn't really work in a wasm sandbox. + +Conclusion: Tracy for native dev-loop profiling now (best tool, trivial +declare-c binding, fits the frame-boundary idea already in the design). If +wasm32 profiling becomes a real need later, a small Chrome-Trace-Format +exporter is the justified "build our own" — not a Tracy replacement, a +different target Tracy doesn't reach. Not designed or implemented. + +** no type-limit constants: u8-max, i32-max, f32-infinity, etc. — nothing exists +Confirmed missing entirely. No MAX/MIN constants in the prelude for any +sized int type; min/max are just binary comparison builtins, not type-bound +limits. No C limits.h/float.h macro import either (cimport only pulls in +declared functions/structs/typedefs, not #define constants) — no path to +INT_MAX/FLT_MAX that way. + +Worse for infinity/NaN: there's no literal syntax at all. float_repr +(lib/form.ml:87-104) prints inf/nan as words for a runtime value that +happens to be one, but the reader doesn't parse those words back ("nan.0 is +no improvement on a literal no reader accepts either way"). Only way to get ++inf today is at runtime: (/ 1.0 0.0) (float division by zero is IEEE-754 +defined, not a trap the way integer division by zero is) — no way to just +write it down. Real gap, not implemented. + +** investigate SBCL's redefinition model — warn + keep old value until callers update +Right now a global/struct type change is a hard refusal (lib/session.ml: +304-311, docs/BUILT.md:1213-1219) — "restart to change it." I remembered a +design where the OLD value/layout keeps being used until the calling code +is recompiled against the new one, since code using the old shape probably +won't even compile anymore anyway, so nothing unsafe reads mismatched +memory. Found it — it's real, but only planned for a narrower case: + +plan.org's Hot Reload section (~685-696): a signature-changing function +redefinition creates a new internal function version + trampoline. Newly +compiled callers use the new one; existing callers and stored function +values keep the old version safely. The session warns at every tracked old +caller site; recompiling one either updates it or gives a normal type error. + +plan.org's dev/release table (~671-682) also lists "Structs: version word" +for dev builds — implying a similar versioned-layout plan existed for +structs too. But docs/BUILT.md:1223-1230 says none of this is built for +functions either ("no function versions, no trampolines... the refusal +stays, because the alternative to refusing is not the new design, it is a +silent argument mismatch. It is a stopgap"). So today BOTH functions and +globals/structs just hard-refuse; the versioned/warn-and-keep-old-value +design exists on paper for functions but nowhere for structs/globals, and +isn't implemented for either. + +Action: look at what SBCL actually does on struct/type redefinition (it +warns rather than refusing, and instances of the old layout get an +"obsolete instance"-style condition on next access rather than corrupting +memory) as a model for what flan's struct/global version-word plan could +be, versus the current hard-refuse stopgap. + +** need a value-producing array constructor, usable as a defvar initializer +Wanted to fill `grid` with 255 as part of its defvar declaration, not as a +separate mutation step after. Doesn't work today: dotimes returns Unit, not +an array value, so it can only mutate an already-existing place — it can't +be the initializer expression itself, which has to produce the whole typed +value in one go. Had to declare grid zeroed then mutate it in main instead. + +(array 4 rl/Vector2) (Ast.ArrayOf, "the one position with no type slot", +docs/BUILT.md ~3475) already exists but only zero-fills — no way to give it +a fill value or a generator. Want something like (array-fill n v) or a +repeat/generate form that IS an expression (produces the array value +directly, works nested for 2D), so it composes as a defvar initializer the +same way a bracket literal does. Not designed or implemented. + +** revamp the flan buffers: *flan-dev* has no compilation-mode / jump-to-error +Related to the earlier "compiler messages need a persistent buffer" item but +distinct — that one was about eval-time messages; this is *flan-dev*, the +daemon's own startup/build output (emacs/flan.el:729-765, +flan--start-daemon), which is where a `main` that fails to compile shows its +errors. It's a plain buffer — get-buffer-create with no major mode, raw +process output appended via make-process. No compilation-mode, no +compilation-shell-minor-mode, so no next-error / M-g M-n / jump-to-source. + +The fix is probably small: flan's own diagnostics already print in the +ordinary path:line:col: message shape (every error pasted in this session +has been that format) — the same shape Emacs' built-in +compilation-error-regexp-alist already parses (it's the GCC/Clang shape +compile-mode was built around). So this likely isn't a custom-regex job, +just turning on compilation-minor-mode in *flan-dev* (or a dedicated +flan-daemon-mode derived from compilation-mode). Umbrella task: revisit all +the flan-* buffers (*flan-dev*, *flan-output*, error overlays, the eval +result overlay/minibuffer item) together rather than patching each one +separately. Not implemented. + +** implicit numeric conversions with a warning flag, instead of hard errors +Not liking the strictness; want implicit typing with an opt-in warning flag +for implicit conversions instead of a hard type error. + +Worth being clear-eyed first: "no implicit widening/narrowing anywhere" is +not an oversight, it's a core invariant repeated all over the codebase — +lib/check.ml:1774 ("this language has no implicit narrowing anywhere"), +:4736, :5640, :6575-6579 ("no implicit widening, so one side has to decide +it"), docs/BUILT.md:809. It's the Odin/Rust-style bet against C's decades of +silent-precision-loss bugs, and it's load-bearing in how binary-op checking +picks which side "decides" the type. A warn-instead-of-refuse mode isn't a +flag on top of that — it's a second type-checking mode that would need +threading through every one of those sites, not a small change. + +That said, real middle-ground precedent exists: C/C++'s -Wconversion is +exactly "implicit, but warn" bolted on after the fact. Something similar +here would mean: allow widening/narrowing between numeric types silently at +the type-check level, but have the checker also emit a separate warning +list (surfaced same as any other diagnostic) for every site where a +conversion happened that wasn't an explicit cast. Not designed, and a real +philosophy question for the language, not just an implementation task. + +** pos?/neg?/zero? don't exist, and need to be generic over numeric types +Nothing named pos?/neg?/zero?/sign anywhere in the prelude. Trivial to write +per-type ((defn pos? [x i64] bool (> x 0))), but that's the problem: we want +these (and inc/dec from earlier) to work across every numeric type without +writing one copy per type. + +This is blocked on real generics, which don't exist yet — "generic code over +the type variable %s is not implemented yet — milestone 5" (lib/check.ml: +776-778, hit earlier when `int` fell through to the type-variable path). +min/max already do this at the builtin level (special-cased in the checker, +not written as generic Flan functions), which is the current workaround for +"needs to work over any numeric type" — but that doesn't scale to arbitrary +user-defined functions like pos?/neg?/inc/dec. Two related asks logged +together: (1) add pos?/neg?/zero?, (2) make milestone-5 generics real so +functions like these don't need special-casing into the compiler to be +generic. + ** println output goes to *flan-output*, not inline in the repl Deliberate per emacs/flan-repl.el:40-44: "a value and the program's output are different things and arrive by different routes... showing them in one @@ -229,3 +429,70 @@ already disambiguates Vec/Map literals from their type spellings (parse.ml comments near line 266-273); worth checking whether () could get the same treatment — type in type position, unit value in expression position — instead of being refused outright everywhere. + +** error inside a macro-expanded body points at the macro call site, not the real line +Hit with do-grid: "unknown function neg?" reported at the do-grid call line, +not the actual (unless (neg? cell-val) ...) line inside the spliced ~@body. + +Already known/tracked, not a fresh gap: the Form wire format a macro +receives and returns (lib/expand.ml:96-101) is Form.value, not Form.t — it +structurally has no loc field, "a macro cannot invent a source location and +the image has no room for one." So EVERY node a macro returns, including +~@body forms spliced through completely unchanged (my own original source, +not compiler-generated), gets the call site's location stamped on it +(lib/prelude.ml:1863-1868 says the same thing). The comment names the real +fix directly: preserving locations through a macro needs "the +structured-error rewrite," which doesn't exist yet — today's stamping is +explicitly the stopgap "that can be had now without" it. + +So: not asking for something new, just noting I hit the documented +limitation and it's genuinely annoying for macros like do-grid that splice +a large body through — every error in the body mislocates to the macro call. + +** built-in comment +No (comment ...) exists. Trivial as a one-line user macro today — +(defmacro comment [args] `(do)) — and it already has the useful property: +macro args are raw unparsed Form, never checked as expressions, so whatever +is inside never needs to type-check. Want it built in (prelude or special +form) rather than something every project defines for itself. + +** #_ (discard) isn't syntax-highlighted, though it works correctly +Compiler side is fine and deliberate — lib/reader.ml:16-24, 202-262 reads +and throws away the next form, Clojure-style, including #_#_ counting-free +chaining. Purely an editor gap: flan-font-lock-keywords +(emacs/flan-mode.el:126-146) has no rule for #_ at all, and +flan-mode-syntax-table (line 180+, derived from lisp-mode-syntax-table) has +no notion of it either since Common Lisp doesn't have this construct. So a +discarded form renders as plain text, not grayed out/comment-styled. + +real clojure-mode.el's approach is precedent: a syntax-propertize-function +that marks a discarded form's characters with the comment syntax class, so +font-lock renders it as a comment for free. Not implemented here. + +** syntax highlighting: defmacro/when/etc missing, and macros aren't dynamic +Two separate gaps. + +Easy: flan--definers (emacs/flan-mode.el:114-118) is missing `defmacro` +entirely, even though it introduces a top-level name like defn does. +flan--special (120-124) is missing when/cond/and/or/break/continue/recur/ +handler-bind/handler-case/restart-case/invoke-restart — all real lib/parse.ml +special forms, just never added to the static lists. Plain oversights, +trivial to fix. + +Harder: highlighting a user-defined macro like do-grid can't be a static +list — and the daemon doesn't even have the info today. A macro compiles +down to an ordinary Tast.fn (defmacro m [args] body is exactly +defn m [args [Form]] Form body) — macro-ness is erased by check time. +Confirmed: defs (lib/dev.ml:1080-1133) reports every function with +kind:"fn", no "macro" kind exists anywhere, because Tast.fn has no field +recording it came from a defmacro. So even flan--defs (the live cache +already driving eldoc/completion) can't tell do-grid is a macro right now. + +Would need: (1) plumb a macro flag through Check/Tast so defs can report a +real "macro" kind, (2) have flan-mode treat flan--defs as a dynamic +font-lock source — filter macro-kind names, font-lock-add-keywords + +font-lock-flush/fontify-buffer whenever the cache refreshes (same points +flan-refresh-defs already runs at: connect, after an accepted eval). CIDER +does exactly this for clojure-mode off a live nREPL connection — real +working precedent. Not designed or implemented here. + diff --git a/FIX.org b/FIX.org index edb3e06..38458b6 100644 --- a/FIX.org +++ b/FIX.org @@ -2127,3 +2127,71 @@ runs the value half on both backends. One existing row changed: the foreign-spelling pin in test_flan.ml used ~int~, which resolves now, and was moved to ~long~. + +* A macro's parameter list, and the one breaking spelling, 2026-09-20 +DISCUSS.org's "defmacro should support real parameter lists" is built. +=(defmacro do-grid [[r rows c cols] & body] ...)= — positional parameters, a +=[ ]= pattern wherever the argument is a vector, nesting, and =&= for the +tail. The list is read in lib/expand.ml (=params_of=, =check_call=), turned +into bindings by lib/parse.ml (=macro_body=) and checked against a call by +lib/macro.ml (=checked_call=) before anything is expanded. + +** THE BREAKING CHANGE: [args] was the whole call, and is now the first argument +This is the one decision in the lane that changes what existing text means, +and it is here rather than in a commit message because it is the thing to +disagree with if it is wrong. + +A macro's single parameter *was* the whole argument list, so =[args]= meant +"everything written at the call". Under a positional parameter list it cannot +keep meaning that: one named parameter has to be the first argument, the way +it is in every other language with parameter lists and the way Clojure has +it. So the whole list is now spelled =[& args]=. + +The alternative was a legacy mode — one parameter with no =&= keeps the old +meaning — and it was refused. It makes =[a]= and =[a b]= mean unrelated +things, which is the kind of rule nobody can hold in their head, and it +would have left the corpus written in a grammar the documentation no longer +describes. + +So every =defmacro= in the tree was migrated in the same commit. Seventeen +files, mechanical, bodies untouched: + +- lib/prelude.ml — =clamp=, =unless=, =into=, and the dogfood batch's five + (=comment=, =inc=, =dec=, =++=, =--=), which landed on dev-loop after this + lane branched and were migrated at the merge — eight in all +- vendor/raylib/modes.flan — =with-drawing=, =with-mode-2d=, =with-mode-3d=, + =with-texture-mode=, =with-scissor-mode= +- vendor/edn/provide.flan — =defedn=; vendor/json/provide.flan — =defjson= +- test/programs/ — macros.flan (7), macro-cycle.flan (2), macro-spin.flan, + pkg-macro.flan, printers.flan, pkgs/mac (6), pkgs/macring (2), + pkgs/macspin (1) +- test/test_dev.ml, test_flan.ml, test_repl.ml, test_session.ml and + emacs/test-flan.el — the =defmacro= fixtures written as strings + +Nothing was rewritten to *use* the new grammar as part of the migration — +=with-mode-2d= is still =[& args]= picking its camera out by hand, and its +hand-written arity guard still says what it said. That was deliberate: the +migration had to be a spelling change or it proves nothing. The new grammar +is shown off in test/programs/macro-params.flan, which is its own program +beside macros.flan. + +The equivalence is asserted rather than assumed. pkg-macro.flan declares +=tenfold= (=[& args]=, =(at args 0)=) and =tenfold-listed= (=[n]=) with the +same body, and test_session expands both and requires the same text. + +** Map destructuring in a macro's parameter list — deferred, refused by name +=dmap= (lib/parse.ml) is ={:keys [x y]}= over a *struct*: it reads field +names off a declared type. A macro's argument is a =Form=, whose =Map= case +is a flat run of alternating forms with no field names anywhere in it. So +the pattern cannot be translated — it would have to be given a new meaning +(match a keyword key in the literal map written at the call? bind by +position?), and none of those is obviously the one somebody wants. + +Vectors and =&= are the 95% case and are built. A map pattern in a macro's +parameter list is refused by name where it is written: + + map destructuring is not implemented in a macro's parameter list — a + macro's argument is a Form, whose Map case is a flat run of alternating + forms with no fields to name. Take the form and pick it apart in the body + +Pinned in test_flan.ml. Whoever wants it should decide what it means first. diff --git a/docs/BUILT.md b/docs/BUILT.md index 3dad02a..91cffa1 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -3075,16 +3075,49 @@ it and loading it into the compiler's own process.** There is nothing to interpr be, so `Emit.redefinition` → `Build.shared` → `dlopen`, the reload primitive the dev loop already runs, is pointed at the compiler instead of at a running program. -`(defmacro name [args] body ...)` is one function, `[Form] -> Form`. One parameter, the slice of forms written at the -call site, which is where variadics come from in a language with no `&rest`: `(len args)` is how many were written. +`(defmacro name [param ...] body ...)` is one function, `[Form] -> Form`. The *declared* parameter is one and always +has been — the slice of forms written at the call site — but the author writes a real parameter list against it: +positional names, a `[ ]` pattern wherever the argument is a vector, and `&` for the tail. `(defmacro do-grid +[[r rows c cols] & body] ...)` is Clojure's shape and it binds six names out of one slice. + +### `[args]` binds the first argument; `[& args]` binds the whole call + +This is the one breaking change the feature carried and it is worth stating twice. Before parameter lists a macro's +single parameter *was* the whole argument list, so `[args]` meant "everything". Under the list it is positional like +every other name, so `[args]` means "the first argument" and the whole list is spelled `[& args]`. Every `defmacro` in +the tree — the prelude's eight, raylib's five, edn's and json's providers, every test fixture — was migrated to +`[& args]` when this landed. There is one grammar and no legacy mode: `[& args]` is the trivial case of the list. + +### Arity and destructuring are refused at the call, before expansion + +`Expand.params_of` reads the list and `Expand.check_call` measures a call against it. `Macro.checked_call` runs that +check before `Expand.call`, on all four ways into an expansion — the walk, `settle`'s re-expansion, and the editor's +`expand_step` and `expand_all` — so C-c C-m refuses what a build refuses, in the same words. + +Before expansion is the whole point. Every node a macro returns is stamped with the call site's `Loc.t` (see `Form` +below), which is a documented limitation waiting on the structured-error rewrite; a refusal raised *before* the macro +runs has the call's own location and its own column, with no stamping involved. The four shapes are: too few +arguments, too many (when there is no `&`), a `[ ]` pattern meeting a form that is not a vector, and a vector of the +wrong length for its pattern. `test/programs/macro-arity.flan` and the three beside it pin each one. + +Map destructuring is **not** in a macro's parameter list, and it is a deferral rather than an oversight: `dmap` is +`{:keys [x y]}` over a *struct*, and a macro's argument is a `Form` whose `Map` case is a flat run of alternating forms +with no field names in it at all. The pattern would have to mean something new. Refused by name, written down in +FIX.org. ### A defmacro is a defn, and there is no Ast.Defmacro -`Parse` turns `(defmacro m [args] body)` into `(defn m [args [Form]] Form body)` and nothing below the parser knows +`Parse` turns `(defmacro m [a & rest] body)` into `(defn m [macro~args [Form]] Form (let [a (at macro~args 0) rest +(form-rest macro~args 1)] body))` and nothing below the parser knows the word exists. The checker checks it like any function, the backend emits it like any function, `Reach.link` drops it from a program that does not call it like any function. The only thing that makes it a macro is that `Macro` calls it at compile time instead of the program calling it at run time. +The parameter list goes the same way: it is bindings over that one slice and nothing downstream learns a pattern +existed, exactly as nothing downstream learns a `let` had one. `macro~args` is the compiler's own name for the slice, +with a `~` in it for `gensym`'s reason — the reader cannot put that character in a symbol, so no name an author writes +collides with it. The generated extraction is unchecked, because `check_call` already counted the call. + This is also why there is no macro table. Storage was the question the front half deliberately left open, and the answer is that there is none: the macro set is recomputed by scanning the top level for the word `defmacro`, which is the only place it survives, and the compiled artefact is a `.so` keyed by a digest. The top level scanned is the diff --git a/docs/SBCL-REDEFINITION-NOTES.md b/docs/SBCL-REDEFINITION-NOTES.md new file mode 100644 index 0000000..b15fcd7 --- /dev/null +++ b/docs/SBCL-REDEFINITION-NOTES.md @@ -0,0 +1,452 @@ +# SBCL's redefinition model, as a reference point for Flan + +Research notes, 2026-09-20. Untracked. Written for DISCUSS.org's item +"investigate SBCL's redefinition model — warn + keep old value until callers +update". Nothing here is a decision; the last section lists candidates and stops. + +The question behind it: Flan's dev session hard-refuses a struct layout change +and a global type change (`lib/session.ml:304-338`, `docs/BUILT.md:1213-1219`), +and plan.org's dev/release table (~671-682) lists "Structs: version word" for dev +builds with nothing behind it. SBCL is the system the author half-remembered as +"warn, keep the old layout, complain at the stale reader". It does do something +close to that — but the shape of it is not quite what the note says, and the +difference matters for what Flan could copy. + +--- + +## 1. `defstruct` redefinition + +### It is an error, not a warning + +This is the one place DISCUSS.org's summary is wrong, and it is worth stating +first because it inverts the precedent. The note says SBCL "warns rather than +refusing". SBCL signals a **continuable error** — the default behaviour of an +incompatible `defstruct` redefinition at a REPL with no handler is to drop you +into the debugger, and in a non-interactive build it aborts. It proceeds only +because a human (or a handler) picks a restart. + +From `src/code/defstruct.lisp`, `%redefine-defstruct`: + +> "attempt to redefine the ~S class ~S incompatibly with the current definition" + +with two restarts: + +- `continue` — "Use the new definition of ~S, invalidating already-loaded code + and instances." +- `recklessly-continue` — "Use the new definition of ~S as if it were + compatible, allowing old accessors to use new instances and allowing new + accessors to use old instances." Offered only when `mutable-layout-p` holds, + i.e. when the new layout can be overwritten in place onto the old one + (compatible sizes/rawness), so it is not offered for arbitrary changes. + +There is a separate genuine warning for the compatible-ish case, +`redefine-structure-warning`: + +> "incompatibly redefining slots of structure class ~S~@Make sure any uses of +> affected accessors are recompiled" + +The comparison that drives all of this is `compare-slots`, which returns three +lists: slots that **moved**, slots that were **retyped**, slots that were +**deleted**. So SBCL's notion of "incompatible" is finer than Flan's +`compatible_structs` all-or-nothing name+type list equality — a pure append of +trailing slots is a different case from a reorder, and SBCL distinguishes them. + +Note also that ANSI leaves structure redefinition **undefined** +(CLHS `defstruct`); everything above is an SBCL extension, not a standard +protocol. SBCL is not implementing a spec here, it invented a policy. + +### What happens to the live instances + +Picking `continue` calls `register-layout` with `:invalidate t`, which reaches +`%invalidate-layout` in `src/code/class.lisp`: + +> "Mark LAYOUT as invalid. This is called only on CONDITION and STRUCTURE +> subtypes when redefining incompatibly." + +It sets the layout's `invalid` slot, zeroes `layout-clos-hash`, and unhooks the +classoid from every superclass's subclass table. **The instances themselves are +not touched, not walked, not found.** There is no heap scan. Each instance keeps +pointing at the same layout object it always pointed at; that object is now +flagged invalid. + +This is the crucial mechanism, and it is the one Flan cannot have: **an SBCL +structure instance carries a pointer to its layout in its header word.** The +invalidation is one store into a shared object, and every instance in the heap +learns about it for free, because they all reach it through that pointer. + +### When the error surfaces + +At the next access, not at redefinition time. Two paths, and they give different +conditions, which is worth being precise about: + +- **The typed-accessor path.** A `defstruct` accessor in safe code checks the + instance's layout against the expected one. With the layout invalidated the + check fails and you get an ordinary `type-error` — the value is no longer + `typep` the struct type it used to be. (I did not read the exact accessor + emission path in `src/compiler/`; treat "it is specifically a `type-error` + rather than some dedicated condition" as **unverified**.) +- **The PCL path.** When a structure-object with an invalid wrapper goes through + PCL (generic function dispatch, `slot-value`), `%obsolete-instance-trap` + fires. For a structure it signals `sb-pcl::obsolete-structure`, defined in + `src/pcl/std-class.lisp`, reported as: + + > "obsolete structure error for a structure of type ~S" + + This is the condition people quote when they say "SBCL tells you your instance + is stale". There is no restart on it that repairs the instance. There is no + update protocol for structures. + +So the honest one-line summary of the struct story: **SBCL refuses by default, +proceeds only on an explicit human restart, and then converts your old instances +into landmines that raise at first touch.** It never silently reads at the wrong +offsets — except under `recklessly-continue`, which is exactly the "silent +argument mismatch" outcome `docs/BUILT.md:1223-1230` says Flan refuses to ship, +and SBCL offers it only behind a name chosen to shame you. + +## 2. `defclass` redefinition — the part with a real protocol + +This is where CL earns the reputation, and it is a standard protocol (CLHS +4.3.6, `update-instance-for-redefined-class`), not an SBCL extension. + +Redefining a class does **not** error. The sequence: + +1. The new class definition is installed. `make-instances-obsolete` runs — + in SBCL, the `std-class` method calls `%update-lisp-class-layout` and + `%invalidate-wrapper` with the `:obsolete` flag. Again: a flag on the shared + wrapper, no heap walk. `layout-invalid` can hold `nil`, `t`, + `(:flush )` or `(:obsolete )`; `invalid-wrapper-p` is the + predicate on the access path. +2. Nothing else happens until someone touches an instance. CLHS 4.3.6: the + update occurs "at an implementation-dependent time, but no later than the + next time a slot of that instance is read or written", and the instance's + `eq` identity is preserved across it. +3. On that first touch, `%obsolete-instance-trap` runs. It computes the + **added** slots and the **discarded** slots by name, builds a property list + of the discarded slots' values, swaps the instance's storage for the new + shape, and calls: + + ```lisp + (sb-sys:nlx-protect (update-instance-for-redefined-class + instance added discarded plist) + (replace-wrapper-and-slots instance owrapper oslots)) + ``` + + The `nlx-protect` is a detail worth noting: if the user's + `update-instance-for-redefined-class` method signals and the stack unwinds, + the instance is rolled back to its old wrapper and slots rather than left + half-migrated. + +What the protocol guarantees: + +- **Slots present in both definitions keep their values, matched by name.** + This is the whole reason it works, and it is only possible because a CLOS + instance's slots are addressed through a name→index map held in the class, not + baked into compiled call sites. +- **Added slots are initialized from their `:initform`s** — the system-supplied + primary method calls `shared-initialize` on exactly the added-slot names, + passing along the initargs it received. +- **Discarded slots' values are not lost, they are handed to you** in the + property list, so a user method can do `(getf plist 'old-name)` and derive the + new slots from the old ones (the canonical CLHS example is a cartesian→polar + coordinate change). +- **Initargs are validated**; the default method signals on an initarg not + declared valid for the class. +- The return value is ignored. + +The pattern to take away: **the automatic part is name-matching, and the +interesting part is a user hook that gets the old values.** CLOS does not try to +guess what a renamed slot meant; it hands you the corpse and lets you decide. + +## 3. Global variables + +Trivial, as expected, and the interesting bit is that CL already draws Flan's +distinction. + +- `defparameter` re-evaluated "unconditionally assigns the initial-value to the + dynamic variable named name". +- `defvar` "assigns initial-value (if supplied) to the dynamic variable named + name **only if name is not already bound**". + +That is precisely `docs/BUILT.md`'s rule that a `defvar`'s *initial value* is +deliberately not in the refusal table — "edit the code, keep the sand". CL's +`defvar` is the same policy, reached by the same reasoning, thirty years earlier. +Flan's `defvar` and CL's behave the same way here: both accept a changed +initialiser and ignore it while the variable is bound. The divergence is only in +the const twin — Flan's `defconst` refuses a changed value (when the checker +consumed it), whereas CL's `defparameter` simply overwrites. + +There is **no** type-change problem to speak of, because a CL special variable +has no compile-time type and no fixed-shape storage — it is one boxed word. The +entire class of failure that `lib/session.ml:304-311` is protecting against +(storage laid out to a type, reused for another) does not exist in CL. So CL +offers no precedent for Flan's typed-global refusal; it sidesteps it by being +dynamically typed. Worth saying plainly rather than pretending there is guidance +here. + +`defconstant` is the exception and is famously annoying: redefining one to a +non-`eql` value is an error in SBCL, which is the same shape as Flan's +compile-time-consumed `defconst` refusal, for a related reason (the old value +may already have been folded into compiled code). + +## 4. Function redefinition + +Trivially supported, and the reason is the reason Flan's cell design already +works: **a global function call goes through the `fdefinition`**, a name→function +indirection, not to a body address. Redefining `foo` stores a new function +object into the symbol's function cell, and every call site compiled against +`foo` picks it up on its next call with no recompilation and no stale caller. +This is exactly plan.org's "every cross-function call goes through an +indirection cell; body redefinition is one atomic pointer store". Flan already +matches SBCL here for the body-only case. + +Two nuances worth carrying over: + +- **Stale callers do exist in SBCL, in one place: block compilation.** Under + `(declaim (start-block ...))` / `:block-compile`, calls within the block become + local calls resolved at compile time. Redefining a function in a block-compiled + unit leaves its in-block callers calling the old body — they were never going + through the fdefinition. SBCL's answer is not a warning or a trampoline; it is + "block compilation prevents redefinition, that is the trade you made", and the + feature is off by default. Self-recursive calls are the same story in + miniature: SBCL deliberately compiles a self-call as a full call precisely so + that a function can be redefined while running. +- **SBCL does not version signatures.** A function whose lambda list changes is + just a redefinition; there is no new version, no trampoline, no per-caller + warning. Callers compiled against the old arglist get a normal + wrong-number-of-arguments error at runtime (or a compile-time style-warning if + SBCL still has the old ftype recorded and sees the call again). plan.org's + versioned-function design with tracked caller sites is **more** than SBCL does, + not a port of it. If the author was remembering SBCL as the source of that + design, the memory is of something else — the nearest real relatives are + Erlang's two-version code loading (old code + current code, a process running + old code is killed on the third load) and Smalltalk's `become:`. + +## 5. What maps to Flan and what cannot + +### The structural obstacle, stated once + +Every SBCL mechanism above — struct layout invalidation, CLOS wrapper +obsolescence, the lazy trap — rests on a single fact: **an instance carries a +pointer to its shape descriptor in its header.** That is what makes "invalidate +the shape" an O(1) operation that reaches every live instance, and what makes +"detect staleness at next access" a load-and-compare on a word the instance +already has. + +Flan's typed side has none of that. A `defstruct` value is flat and unboxed; it +lives inline in a global, inline in an array element, in registers, spilled on a +stack frame. There is no header word, no per-instance shape pointer, and no way +to enumerate the live instances of a type. So: + +- Layout invalidation as SBCL does it: **structurally unavailable.** There is + nothing to invalidate that instances reach. +- Detecting an obsolete instance at access time: **unavailable in release + shape,** because detection requires a per-instance word. It is available only + if you *add* that word, which is exactly what plan.org's "Structs: version + word, dev only" line means. +- `update-instance-for-redefined-class`-style migration: **doubly unavailable**, + because it needs both the per-instance shape tag *and* the ability to change + an instance's size in place. A flat struct embedded in an array cannot grow. + +### The dev-build version word is SBCL's generation counter + +The connection plan.org's table implies but does not spell out: a version word +in the dev build is the degenerate form of SBCL's layout. SBCL's instance points +at a layout object whose `invalid` flag can be flipped; a Flan dev-build struct +would carry an integer stamped at construction, compared against the type's +current generation on access. Flip the generation on redefinition, and the next +field access on an old-generation value traps. That is `layout-invalid` with the +indirection collapsed into an immediate, which is the only form that survives +"instances have no headers, but dev builds may pay for one extra word". + +What it buys, honestly: **not** the ability to keep using old values. It buys a +*diagnosis* instead of a refusal — the reload is accepted, the program keeps +running, and the first read of a stale value says "this `Cursor` was built +before you changed `Cursor`" with a location, instead of the session saying +"restart" before anything runs. That is genuinely SBCL's bargain (`continue` +then trap), and it is the honest framing: the version word converts a +compile-time refusal into a runtime trap. Whether that is an improvement is a +judgement about which failure the author would rather debug. + +What it costs, all of which is real: + +- A word per struct instance in dev builds, so dev and release layouts differ — + which means array strides, FFI structs, and anything crossing to C differ too, + or need to be excluded. The dev/release divergence plan.org already accepts + for frames and cells gets bigger and more observable. +- A check on every field access in dev builds. +- Structs embedded in other structs, in arrays, and in `Vec`s each need a story; + an array of 4096 cells would need every element stamped and every element + checked. +- Globals are not covered by it at all. A global's storage is a fixed slot with + a shape; a version word on the *global* could detect a stale read, but there + is nothing to read — the point of the refusal is that the old bytes mean + something else now. SBCL has no precedent to offer, per section 3. + +### The dyn side is where CLOS-style update is actually available + +Grounding the claim in the code as landed (FIX.org item 6, commits `8d2bf2a`, +`5af990e`, `6c6024e`; `runtime/flan_dyn.c`): + +- A `defclass` instance is an ordinary dyn map with a `klass` field in the + `flan_obj` header (`flan_dyn.c:255`), set by `flan_dyn_map_new_class` + (`:1088`), read by `flan_dyn_class_of` (`:1100`). It is deliberately in the + header and not an entry, so `len`, `render` and `dyn_equal` do not see it. +- **Slots are keys.** `lib/classes.ml:150-175`: `(defclass point [x y])` + desugars to `(defn point [x dyn y dyn] dyn #point{:x x :y y})` and nothing + else. Access is `get`/`put` by keyword. + +Two consequences, and the second is the one that surprised me: + +1. **Name-based slot carrying — the hard half of + `update-instance-for-redefined-class` — is free here.** A dyn instance already + stores its slots by name. Any migration is a map operation. +2. **There is no class object at runtime to invalidate, because there is no class + registry at all.** The `klass` field holds an interned keyword, not a pointer + to a class descriptor. Redefining a `defclass` today is just redefining its + constructor `defn` — which the session already permits as an ordinary body + change. Old instances keep their old keys and keep working; new ones get the + new keys; `class-of` answers the same keyword for both. So the dyn side + currently has neither the refusal *nor* the protocol: it has silence. + +Whether that silence is a bug depends on what the author wants. It is the +Clojure answer (a map is a map; a "class" is a tag) rather than the CLOS answer. +The CLOS answer would need, concretely: + +- A runtime class registry: keyword → (slot-name list, generation counter). + Nothing like this exists today; `lib/classes.ml` is compile-time only and + emits a plain `defn`. +- A generation stamp per instance, or a cheaper trick: compare the instance's + key set against the registry's current slot list lazily on `get`/`put`. With + a generation word in the `flan_obj` header — there is room, and unlike the + typed side these objects *have* headers — the check is a compare. +- A migration step at first access after a bump: add missing slots as nil, + collect removed slots into a property list, and call a user hook. The obvious + Flan spelling of that hook is a generic function, e.g. + `(defmethod update-for-redefined point [p added discarded] ...)`, which fits + the dispatch mechanism that already exists. +- A decision on whether `put` of an unknown slot stays legal. Today it is — a + class instance is an open map, and FIX.org already defers "refusing an unknown + slot at `(get p :z)`". If unknown slots stay legal, the registry's slot list is + advisory and the whole update protocol is advisory with it. + +This is a real, SBCL/CLOS-precedented design that Flan's runtime can actually +support. It is also a feature with no user yet, since redefinition on the dyn +side currently fails silently rather than loudly. + +--- + +## 6. Candidate designs + +Three, none recommended over the others. Cost estimates are rough. + +### A. Keep the refusal for typed structs and globals; fix the message + +Leave `lib/session.ml` refusing, but rewrite both messages to say what SBCL's +restarts say: name which fields moved, were retyped, or were deleted (SBCL's +`compare-slots` split — Flan compares whole field lists and can cheaply do the +same split), and state the two things the author can do instead (restart, or +rename the type and migrate by hand). Optionally accept the case SBCL's +`mutable-layout-p` accepts and Flan currently refuses: **appending** fields to a +struct no array/FFI type depends on is layout-compatible for every existing +instance, so it need not be refused at all. + +- Cost: small, days. Frontend only, no runtime change, no dev/release divergence. +- Behaviour: unchanged except the append case starts working and the diagnostic + names the offending field. +- Precedent: SBCL refuses by default too — this is the `%redefine-defstruct` + error with no restart taken, plus `compare-slots`' finer classification. The + "SBCL warns instead of refusing" premise in DISCUSS.org does not survive + contact with the source, so "keep refusing" is the SBCL-consistent option, not + the timid one. + +### B. Dev-build struct version word — SBCL's layout generation, collapsed + +Stamp every dev-build struct instance with the type's generation at +construction; bump the generation on an incompatible redefinition; check the +stamp on field access in dev builds and trap with "this value was built before +`Cursor` changed shape" instead of refusing the reload. Release builds carry +neither word nor check, which is plan.org's dev/release table as written. + +- Cost: large. Touches layout, both backends, arrays and `Vec`s of structs, FFI + boundary exclusions, and every field-access site; the dev/release layout + divergence becomes observable wherever a struct crosses to C. This is the one + that wants a written design before any code. +- Behaviour: the reload is accepted and the program keeps running; stale values + trap loudly at first touch rather than reading wrong offsets. It does **not** + let old values keep working — no migration is possible for a flat unboxed + struct — so it converts a pre-run refusal into a mid-run trap. Globals are not + covered and still need option A's refusal. +- Precedent: SBCL's `%invalidate-layout` plus the obsolete-instance trap, with + the layout pointer replaced by an immediate because Flan instances have no + headers. Note that SBCL, having the same choice, still puts the error in the + redefinition path *as well* and requires a restart to reach the trap state. + +### C. CLOS-style update for dyn `defclass` instances only + +Give `defclass` a runtime registry (keyword → slots + generation), stamp a +generation into the `flan_obj` header beside `klass`, and on the first `get`/`put` +after a bump, migrate the instance: add new slots as nil, gather removed slots +into a list, and dispatch a user-overridable generic `update-for-redefined` +before returning. Typed structs and globals are untouched and keep option A's +refusal. + +- Cost: medium and well-contained — `runtime/flan_dyn.c` header field plus a + registry, `lib/classes.ml` emitting the registration, one lazy check on the map + access path. It rides on machinery that already exists: headers, interned + keyword tags, generic dispatch, and slots that are already addressed by name. +- Behaviour: redefining a class stops being silent. Instances survive + redefinition with their shared slots intact, new slots appear as nil, and the + author can write the coordinate-change-style migration by hand. The risk is + that it is a ceremony on top of maps that are already open — if `put` of an + arbitrary key stays legal, the registry describes an intention rather than a + constraint. +- Precedent: this *is* CLHS 4.3.6 and `update-instance-for-redefined-class`, the + only part of CL's story that is a standard protocol rather than an + implementation's policy, and the only part whose prerequisites Flan already + satisfies. + +B and C are independent and could both happen, or neither. A is close to free +and is compatible with both. + +--- + +## Sources + +- SBCL source, `src/code/defstruct.lisp` (`%redefine-defstruct`, `compare-slots`, + `redefine-structure-warning`, `mutable-layout-p`, restart texts) — + +- SBCL source, `src/code/class.lisp` (`%invalidate-layout`, `register-layout`, + `layout-invalid`) — +- SBCL source, `src/pcl/std-class.lisp` (`obsolete-structure` condition, + `%obsolete-instance-trap`, `make-instances-obsolete`, `invalid-wrapper-p`, + `replace-wrapper-and-slots`) — + +- CLHS 4.3.6 Redefining Classes — + +- CLHS `update-instance-for-redefined-class` — + +- CLHS `defparameter`/`defvar` — + +- SBCL Internals, Local Calls — +- Block compilation in SBCL 2.0.2 — + +- Lisp journey, "Structures: lightweight records" (the user-visible restart menu) + — + +The SBCL *manual* () does not document structure +redefinition, obsolete instances or layout invalidation at all — checked, and it +is silent. Everything in sections 1 and 4 above comes from source and mailing +list/blog material, not from the manual. + +### Marked unverified + +- The exact condition signalled when a *compiled typed accessor* (as opposed to + the PCL path) is applied to an instance with an invalidated layout. Stated + above as "an ordinary `type-error`"; I did not read the accessor emission in + `src/compiler/`, so treat the specific condition name as unverified. The + `sb-pcl::obsolete-structure` text is verified from `src/pcl/std-class.lisp`. +- Whether SBCL's `compare-slots` classification ever *avoids* the error for a + pure append (i.e. whether appending slots is accepted silently). The + `mutable-layout-p` gate on `recklessly-continue` strongly implies a + compatible-layout notion exists, but I did not confirm the append case is + error-free. diff --git a/emacs/test-flan.el b/emacs/test-flan.el index 7a009a9..0b958c3 100644 --- a/emacs/test-flan.el +++ b/emacs/test-flan.el @@ -1970,7 +1970,7 @@ already rely on it — so nothing here is a stand-in for the real thing." ;; with the program still on screen. (flan--request (list :op "eval" :file file - :code "(defmacro spinner [args] `(spinner ~@args))")) + :code "(defmacro spinner [& args] `(spinner ~@args))")) (goto-char (point-max)) (insert "\n(defn spun [] i32\n (spinner 1))\n") (goto-char (point-max)) @@ -2005,10 +2005,10 @@ already rely on it — so nothing here is a stand-in for the real thing." ;; only shape where expanding in place has anything to do. (flan--request (list :op "eval" :file file - :code "(defmacro m-inner [args] `(+ ~(at args 0) 1))")) + :code "(defmacro m-inner [& args] `(+ ~(at args 0) 1))")) (flan--request (list :op "eval" :file file - :code "(defmacro m-outer [args] `(m-inner ~(at args 0)))")) + :code "(defmacro m-outer [& args] `(m-inner ~(at args 0)))")) (goto-char (point-max)) (insert "\n(defn outered [] i32 (m-outer 5))\n") (goto-char (point-max)) diff --git a/lib/expand.ml b/lib/expand.ml index e771d4b..ddfe51b 100644 --- a/lib/expand.ml +++ b/lib/expand.ml @@ -227,3 +227,173 @@ let rec quasiquote (f : Form.t) : Form.t = | Form.Vec xs -> Form.make (Form.Vec (List.map quasiquote xs)) f.Form.loc | Form.Map xs -> Form.make (Form.Map (List.map quasiquote xs)) f.Form.loc | _ -> f + +(* ── A macro's parameter list ────────────────────────────────────── + [(defmacro do-grid [[r rows c cols] & body] ...)] — positional parameters, + a destructuring vector wherever one is written, and [&] for the tail. The + grammar is [dvec]'s (lib/parse.ml), read over [Form] instead of over the + values a [let] binds, because a macro's arguments *are* Forms. + + It lives here rather than in [Parse] because both sides of the feature need + it and they are on opposite sides of the parser: [Parse] turns the list into + the bindings a macro body opens with, and [Macro] checks a call against it + before the macro is ever run. This file is below both and depends on nothing + above [Form], which is what lets them share one reading of the list. + + Map destructuring is not here. [dmap] is [{:keys [x y]}] over a *struct*, and + a macro's argument is a [Form] whose [Map] case is a flat list of alternating + forms with no field names in it at all — the pattern would have to mean + something new rather than the same thing over a different value. Refused by + name below, and written down in FIX.org. *) + +type pat = + | Pname of string * Loc.t + (* A [ ] in the parameter list: the argument at this position must be a + [Form.Vec], and its elements are matched against these in turn. *) + (* The [Form] is the pattern as written: a refusal shows the shape the call + failed to match, and nothing else can render it back. *) + | Pvec of pat list * (string * Loc.t) option * Form.t + +type msig = { + ps : pat list; (* the positional parameters, in order *) + rest : (string * Loc.t) option; (* [& name], if there is one *) + src : Form.t; (* the list as written, for the messages *) +} + +(* [a b & rest], shared by the top level and by every destructuring vector + inside it. The three refusals are [dvec]'s, word for word where they say the + same thing: one grammar, so one set of sentences about getting it wrong. *) +let split_amp (items : Form.t list) : Form.t list * (string * Loc.t) option = + let rec go acc = function + | [] -> (List.rev acc, None) + | ({ Form.v = Form.Sym "&"; _ } as amp) :: rest -> + (match rest with + | [ { Form.v = Form.Sym r; loc } ] -> (List.rev acc, Some (r, loc)) + | [] -> Loc.fail amp.Form.loc "& needs a name after it, as in [a b & rest]" + | [ bad ] -> + Loc.fail bad.Form.loc + "& binds one name for the rest of the arguments, and %s is not one \ + — the rest is a slice of forms, so it cannot be destructured \ + further" + (Form.to_string bad) + | _ :: extra :: _ -> + Loc.fail extra.Form.loc + "& takes one name and it is the last thing in the parameter list") + | x :: rest -> go (x :: acc) rest + in + go [] items + +(* Never handed a [&]: every list of items reaching here has been through + [split_amp], which stops at the first one and refuses every way of getting + the tail wrong itself. So there is no arm for it and no sentence about it. *) +let rec pat_of (f : Form.t) : pat = + match f.Form.v with + | Form.Sym s -> Pname (s, f.Form.loc) + | Form.Vec items -> + let elems, rest = split_amp items in + (match elems, rest with + | [], None -> + Loc.fail f.Form.loc + "an empty pattern [] in a macro's parameter list binds nothing — \ + write the names it should bind" + | [], Some (r, loc) -> + Loc.fail loc + "[& %s] binds the whole vector — write %s on its own instead of a \ + pattern" r r + | _ -> ()); + Pvec (List.map pat_of elems, rest, f) + | Form.Map _ -> + Loc.fail f.Form.loc + "map destructuring is not implemented in a macro's parameter list — a \ + macro's argument is a Form, whose Map case is a flat run of alternating \ + forms with no fields to name. Take the form and pick it apart in the \ + body" + | _ -> + Loc.fail f.Form.loc + "a macro's parameter is a name or a [ ] pattern over one, and %s is \ + neither" + (Form.to_string f) + +(* Every name the list binds, so that two of them can be refused where they are + written rather than reaching the checker as a local declared twice. *) +let rec pat_names acc = function + | Pname (s, loc) -> (s, loc) :: acc + | Pvec (ps, rest, _) -> + let acc = List.fold_left pat_names acc ps in + (match rest with None -> acc | Some nl -> nl :: acc) + +let params_of (v : Form.t) : msig = + let items = match v.Form.v with + | Form.Vec items -> items + | _ -> + Loc.fail v.Form.loc "a macro's parameter list is written in [ ], and %s is not" + (Form.to_string v) + in + let elems, rest = split_amp items in + let sg = { ps = List.map pat_of elems; rest; src = v } in + let names = List.fold_left pat_names [] sg.ps in + let names = + match sg.rest with None -> names | Some nl -> nl :: names in + let seen = Hashtbl.create 8 in + List.iter + (fun (n, loc) -> + if Hashtbl.mem seen n then + Loc.fail loc "%s is bound twice in this parameter list" n + else Hashtbl.add seen n ()) + (List.rev names); + sg + +(* ── Checking a call against it ──────────────────────────────────── + Before expansion, so the location is the call's own and not the + [Loc.from_macro] stamp every node of an expansion carries. That is the whole + reason this is a separate pass rather than something the macro body could + do: a macro has no error facility, and by the time its body runs the only + location left is the one it was called from anyway — stamped onto forms the + author never wrote. *) + +let written (sg : msig) = Form.to_string sg.src + +let arity (sg : msig) ~name ~loc (args : Form.t list) = + let n = List.length sg.ps in + let got = List.length args in + let plural k = if k = 1 then "argument" else "arguments" in + match sg.rest with + | Some (r, _) when got < n -> + Loc.fail loc + "%s takes at least %d %s and this call gives %d — its parameter list is \ + %s, where &%s is the rest" + name n (plural n) got (written sg) r + | Some _ -> () + | None when got <> n -> + Loc.fail loc + "%s takes %d %s and this call gives %d — its parameter list is %s" + name n (plural n) got (written sg) + | None -> () + +let rec check_pat ~name (p : pat) (a : Form.t) = + match p with + | Pname _ -> () + | Pvec (ps, rest, src) -> + let items = + match a.Form.v with + | Form.Vec items -> items + | _ -> + Loc.fail a.Form.loc + "%s destructures this argument with %s, so a [ ] belongs here and \ + %s was written" + name (Form.to_string src) (Form.to_string a) + in + let n = List.length ps in + let got = List.length items in + if (rest = None && got <> n) || got < n then + Loc.fail a.Form.loc + "%s destructures this argument with %s, which takes %s%d, and %d %s \ + written here" + name (Form.to_string src) + (if rest = None then "" else "at least ") + n got (if got = 1 then "is" else "are"); + List.iteri (fun i q -> check_pat ~name q (List.nth items i)) ps + +let check_call ~name ~loc (sg : msig) (args : Form.t list) = + arity sg ~name ~loc args; + List.iteri (fun i p -> check_pat ~name p (List.nth args i)) sg.ps diff --git a/lib/macro.ml b/lib/macro.ml index d636183..b67826c 100644 --- a/lib/macro.ml +++ b/lib/macro.ml @@ -53,8 +53,27 @@ let rec names_macro (known : string list) (f : Form.t) = type loaded = { handle : Dynload.handle; fns : (string * Dynload.addr) list; + (* Every macro's parameter list as [Expand] read it, so that a call can be + checked against it *before* it is expanded. That ordering is the whole + point: the refusal then carries the call's own location, where an error + raised from inside a macro body would carry [Loc.from_macro]'s stamp on a + form the author never wrote. *) + sigs : (string * Expand.msig) list; } +(* The parameter list of every [defmacro] in a run of forms. The prelude's are + read from the prelude itself, since its macros are compiled into every + module without ever appearing in [extra]. *) +let sigs_in (forms : Form.t list) : (string * Expand.msig) list = + List.filter_map + (fun (f : Form.t) -> + match f.Form.v with + | Form.List ({ Form.v = Form.Sym "defmacro"; _ } + :: { Form.v = Form.Sym n; _ } :: ps :: _ :: _) -> + Some (n, Expand.params_of ps) + | _ -> None) + forms + (* This compiler's own identity, and it belongs in the key for a reason the other caches do not have. A [.o] under the object cache is decided entirely by the C text and the C compiler that made it, so its key is total without @@ -294,7 +313,11 @@ let compile (names : string list) (extra : Form.t list) : loaded = end; let handle = Dynload.dl_open out in { handle; - fns = List.map (fun n -> (n, Dynload.dl_sym handle (Mangle.macro n))) names } + fns = List.map (fun n -> (n, Dynload.dl_sym handle (Mangle.macro n))) names; + (* [extra] is the file's own macros and an import's; the prelude's are only + ever in the prelude. A name in both is the file's, which is the same + shadowing [loaded_for] applies to the forms themselves. *) + sigs = sigs_in extra @ sigs_in (Prelude.forms ()) } (* ── Where the call site is ──────────────────────────────────────── The one thing a macro cannot find out for itself and the one it needs to @@ -344,11 +367,27 @@ let dir_of (l : loaded) (loc : Loc.t) = expanded again, because a macro that expands into a call to itself — which is what a recursive [cond] is — has to keep going. - That re-expansion is what needs a bound. [(defmacro loop [args] `(loop))] + That re-expansion is what needs a bound. [(defmacro loop [& args] `(loop))] settles at nothing, and the honest answer to a macro that will not settle is to say which one it was, at the call site, rather than to run out of memory. *) +(* Every call goes through here, and there are four ways in: the walk below, + [settle]'s re-expansion of what a macro answered, and the editor's + [expand_step] and [expand_all]. One place, so [C-c C-m] refuses exactly what + a build refuses. + + [List.assoc_opt] rather than [List.assoc]: a macro compiled into the module + always has a signature, and the one thing that could put a name in [fns] + without one is the two lists coming apart — in which case expanding + unchecked is the wrong half to lose. *) +let checked_call (l : loaded) n ~loc (args : Form.t list) : Form.t = + (match List.assoc_opt n l.sigs with + | Some sg -> Expand.check_call ~name:n ~loc sg args + | None -> ()); + dir_of l loc; + Expand.call ~loc:(Loc.from_macro n loc) (List.assoc n l.fns) args + let fuel = 200 let rec expand_form (l : loaded) (f : Form.t) : Form.t = @@ -357,12 +396,11 @@ let rec expand_form (l : loaded) (f : Form.t) : Form.t = | Form.List ({ Form.v = Form.Sym n; _ } :: args) when List.mem_assoc n l.fns -> let args = List.map (expand_form l) args in (* The call site, tagged with the macro it is a call to. [Expand.unmarshal] - stamps this onto every node the macro answers with, so from here down + stamps it onto every node the macro answers with, so from here down every form it produced knows where it came from and an error on one of - them can say so. *) - let from = Loc.from_macro n loc in - dir_of l loc; - settle l n loc (Expand.call ~loc:from (List.assoc n l.fns) args) fuel + them can say so. [checked_call] is where that tagging happens, along + with the arity and destructuring check that has to come first. *) + settle l n loc (checked_call l n ~loc args) fuel | Form.List xs -> Form.make (Form.List (List.map (expand_form l) xs)) loc | Form.Vec xs -> Form.make (Form.Vec (List.map (expand_form l) xs)) loc | Form.Map xs -> Form.make (Form.Map (List.map (expand_form l) xs)) loc @@ -379,10 +417,7 @@ and settle l first loc (f : Form.t) left = first fuel else begin let args = List.map (expand_form l) args in - let from = Loc.from_macro m loc in - dir_of l loc; - settle l first loc (Expand.call ~loc:from (List.assoc m l.fns) args) - (left - 1) + settle l first loc (checked_call l m ~loc args) (left - 1) end (* Settled at the head. The rest of it may still hold macro calls — a cond expands to an if whose else-branch is another cond — so the ordinary walk @@ -570,10 +605,7 @@ let expand_step (f : Form.t) : Form.t * string option = (* [C-c C-m] over a type provider reads the data file, which is the whole of what makes the live loop live: edit the .edn, expand again, see the struct that file now implies. *) - dir_of l f.Form.loc; - ( Expand.call ~loc:(Loc.from_macro n f.Form.loc) (List.assoc n l.fns) - args, - Some n ) + (checked_call l n ~loc:f.Form.loc args, Some n) | _ -> (f, None)) (** To the fixpoint, through exactly the walk a build goes through — so the diff --git a/lib/parse.ml b/lib/parse.ml index 0b6b2fd..1f22cff 100644 --- a/lib/parse.ml +++ b/lib/parse.ml @@ -30,6 +30,13 @@ let temps = ref 0 let fresh_temp what = incr temps; Printf.sprintf "%s~%d" what !temps +(* The one parameter every macro is compiled with, whatever its author wrote as + a parameter list: the slice of forms at the call site, which the bindings + [macro_body] generates read out of. A [~] in it for the same reason + [fresh_temp] puts one there — the reader cannot produce the character in a + symbol, so nothing an author writes collides with it. *) +let macro_args = "macro~args" + (* Destructuring binds in [let] and nowhere else. Every other binding position — a [defn] parameter, a [defstruct] field, an [fn] parameter, a [dotimes] counter, a [match] arm's binds — takes a plain name, and a pattern written @@ -1616,42 +1623,46 @@ let rec decl (f : Form.t) : Ast.decl = | _ -> fail f "defconst is (defconst name Type? value)") (* A macro is an ordinary function, and this is where it becomes one: - [(defmacro m [args] body)] is [(defn m [args [Form]] Form body)]. There is - no [Ast.Defmacro] and there is not going to be one -- a macro has the type - [[Form] -> Form], it is compiled by the same backend as everything else, - and the only thing that makes it a macro is that [Expand] calls it at - compile time instead of the program calling it at run time. + [(defmacro m [a b & body] ...)] is [(defn m [macro~args [Form]] Form (let + [a (at macro~args 0) b (at macro~args 1) body (form-rest macro~args 2)] + ...))]. There is no [Ast.Defmacro] and there is not going to be one -- a + macro has the type [[Form] -> Form], it is compiled by the same backend as + everything else, and the only thing that makes it a macro is that [Expand] + calls it at compile time instead of the program calling it at run time. - One parameter, the slice of the argument forms, rather than one declared - parameter per argument. It needs no reader or parser change and it gives - variadics for free, which is what [unless] and [when] need in a language - with no &rest. + So the declared type is what it always was: one parameter, the slice of + the argument forms. What changed is that the parameter is the compiler's + now and the author writes a real list against it — positional names, a + [ ] pattern wherever an argument is a vector, and [&] for the tail — which + opens the body as bindings over that slice. [Expand]'s [msig] is the + reading of the list, and [Macro] checks a *call* against the same reading + before expanding it, which is where arity and shape are refused with the + call's own location. - The shape rules stay exactly as they were, because they were enforced - before the feature existed on purpose: getting the shape wrong and getting - the whole feature are different mistakes. *) + The one breaking change in this: [[args]] used to bind the whole argument + list and now binds the first argument, because one grammar that means one + thing everywhere is worth more than a legacy spelling. The whole list is + [[& args]], and every macro in the tree was migrated to it. *) | List ({ v = Sym "defmacro"; _ } :: args) -> (match args with - | n :: { v = Form.Vec [ p ]; _ } :: body when body <> [] -> + | n :: ({ v = Form.Vec _; _ } as ps) :: body when body <> [] -> + let sg = Expand.params_of ps in let form_t = { Ast.t = Ast.Tname "Form"; tloc = f.loc } in mk (Ast.Defn { Ast.name = sym n; - params = [ { Ast.fname = sym p; - fty = { Ast.t = Ast.Tslice form_t; tloc = p.loc }; - floc = p.loc } ]; + (* A name the reader cannot produce -- [~] opens an unquote, so + no symbol read out of a source file holds one -- which is + what keeps the compiler's own parameter out of the way of + every name the author might bind. Same trick as [gensym]. *) + params = [ { Ast.fname = macro_args; + fty = { Ast.t = Ast.Tslice form_t; tloc = ps.loc }; + floc = ps.loc } ]; (* Written out, not deferred: a macro takes [[Form]] and returns a [Form], and neither half of that is the user's to leave off. *) praw = None; - ret = Some form_t; fwhere = []; fbody = body_of body; + ret = Some form_t; fwhere = []; fbody = macro_body sg body; nloc = n.loc }) - | _ :: { v = Form.Vec ps; _ } :: body when body <> [] -> - List.iter (fun (p : Form.t) -> ignore (sym p)) ps; - fail f - "a macro takes one parameter, the forms at its call site, and this \ - one names %d. There is no &rest and no arity: (defmacro m [args] \ - ...) and (len args) is how many were written" - (List.length ps) | _ -> fail f "defmacro is (defmacro name [param ...] body ...)") @@ -1667,6 +1678,57 @@ let rec decl (f : Form.t) : Ast.decl = | List ({ v = Sym s; _ } :: _) -> fail f "unknown top-level form (%s ...)" s | _ -> fail f "expected a top-level declaration, found %s" (Form.to_string f) +(* The bindings a macro body opens with, one per name its parameter list binds, + in the order they are written. Built as [Form]s and handed to [expr] rather + than assembled as [Ast] directly: the extraction is [(at ...)] and + [(form-rest ...)] over a slice and [(let ...)] around the body, which is + ordinary Flan and already has a parser. Nothing downstream learns that a + macro had a parameter list, exactly as nothing downstream learns that a + [let] had a pattern. + + The extraction is unchecked on purpose. [Macro] has already run + [Expand.check_call] over this call by the time the body runs, so an [(at + macro~args 2)] here is an index that was counted, and a + [(form-vec-items ...)] is a form already known to be a [Form.Vec]. Checking + twice would mean a second set of sentences, said from inside an expansion + where the location is the call site's stamp rather than the call. *) +and macro_body (sg : Expand.msig) (body : Form.t list) : Ast.expr list = + let loc0 = sg.Expand.src.Form.loc in + let s loc n : Form.t = Form.make (Form.Sym n) loc in + let call loc xs : Form.t = Form.make (Form.List xs) loc in + let idx loc i : Form.t = Form.make (Form.Int (Int64.of_int i)) loc in + let nth loc src i = call loc [ s loc "at"; src; idx loc i ] in + let tail loc src i = call loc [ s loc "form-rest"; src; idx loc i ] in + let out = ref [] in + let add n v = out := (n, v) :: !out in + let rec go (p : Expand.pat) (src : Form.t) = + match p with + | Expand.Pname (n, loc) -> add (s loc n) src + | Expand.Pvec (ps, rest, pf) -> + let loc = pf.Form.loc in + (* The elements of the vector, bound once: every name under this pattern + reads that one slice rather than unwrapping the form again. *) + let t = fresh_temp "macro" in + add (s loc t) (call loc [ s loc "form-vec-items"; src ]); + List.iteri (fun i q -> go q (nth loc (s loc t) i)) ps; + (match rest with + | None -> () + | Some (r, rl) -> add (s rl r) (tail rl (s loc t) (List.length ps))) + in + let av = s loc0 macro_args in + List.iteri (fun i p -> go p (nth loc0 av i)) sg.Expand.ps; + (match sg.Expand.rest with + | None -> () + | Some (r, rl) -> add (s rl r) (tail rl av (List.length sg.Expand.ps))); + match List.rev !out with + (* [(defmacro m [] ...)] binds nothing, and [(let [] ...)] is refused a few + hundred lines up. The body is the body. *) + | [] -> body_of body + | bs -> + let items = List.concat_map (fun (n, v) -> [ n; v ]) bs in + body_of + [ call loc0 (s loc0 "let" :: Form.make (Form.Vec items) loc0 :: body) ] + and variant (f : Form.t) : Ast.variant = match f.v with | Sym n -> { Ast.vname = n; vfields = []; vloc = f.loc } diff --git a/lib/prelude.ml b/lib/prelude.ml index 454684c..c96d2cd 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -662,7 +662,7 @@ let source = {flan| ;; put a complaint: a macro has no error facility (see `unless` at the foot of ;; this file), so a diagnostic would have to be a run-time one, in the one ;; construct whose whole point is that it costs nothing at run time. -(defmacro clamp [args] +(defmacro clamp [& args] (if (!= (len args) 3) `(clamp-takes-a-value-a-low-and-a-high) `(min ~(at args 2) (max ~(at args 1) ~(at args 0))))) @@ -1993,6 +1993,16 @@ let source = {flan| (set i (+ i 1))) (as-slice v))) +;; The elements of a vector form, which is what a [ ] pattern in a macro's +;; parameter list unwraps. The other arm is unreachable from a generated +;; binding -- lib/expand.ml's check_call refuses a non-vector argument at the +;; call site, before the macro runs -- and is here because a macro picking a +;; form apart by hand has the same question and no such guarantee. +(defn form-vec-items [f Form] [Form] + (match f + (Form.Vec xs) xs + _ (form-nil))) + ;; A name no reader can produce. `~` is a delimiter now (it opens an unquote), ;; so no symbol coming out of read_all can contain one, and a gensym therefore ;; cannot collide with a name someone wrote. Non-hygienic expansion with an @@ -2042,7 +2052,7 @@ let source = {flan| ;; halves of one form and only a restriction they both carried would be worth ;; keeping. A test is still required, because there is nothing to negate ;; without one. -(defmacro unless [args] +(defmacro unless [& args] (if (< (len args) 1) `(unless-takes-a-test) `(if (not ~(at args 0)) (do ~@(form-rest args 1))))) @@ -2070,7 +2080,7 @@ let source = {flan| ;; which is what a block of parked code wants. Built in rather than left to ;; every project, because a name this standard should mean the same thing in ;; all of them. -(defmacro comment [args] +(defmacro comment [& args] `(do)) ;; ── inc/dec and ++/-- ───────────────────────────────────────────────── @@ -2099,22 +2109,22 @@ let source = {flan| ;; the *place* without a reference type it does not have, and ;; rl/with-drawing and rl/with-mode-2d already take the same trade on their ;; arguments. Write the index out first if it does anything. -(defmacro inc [args] +(defmacro inc [& args] (if (!= (len args) 1) `(inc-takes-one-number) `(+ ~(at args 0) 1))) -(defmacro dec [args] +(defmacro dec [& args] (if (!= (len args) 1) `(dec-takes-one-number) `(- ~(at args 0) 1))) -(defmacro ++ [args] +(defmacro ++ [& args] (if (!= (len args) 1) `(++-takes-one-place) `(set ~(at args 0) (+ ~(at args 0) 1)))) -(defmacro -- [args] +(defmacro -- [& args] (if (!= (len args) 1) `(---takes-one-place) `(set ~(at args 0) (- ~(at args 0) 1)))) @@ -2235,7 +2245,7 @@ let source = {flan| ;; what was written. len and at borrow, so used directly the source is only ;; read. A source that is a call and produces a Vec is still consumed, which is ;; right: nobody else is holding it. -(defmacro into [args] +(defmacro into [& args] (if (< (len args) 2) `(into-takes-a-source-a-destination-and-transforms) (let [from (at args 0) diff --git a/test/programs/macro-arity-extra.flan b/test/programs/macro-arity-extra.flan new file mode 100644 index 0000000..337bca4 --- /dev/null +++ b/test/programs/macro-arity-extra.flan @@ -0,0 +1,9 @@ +;;;; Too many arguments, which is the half a & would have allowed. There is no +;;;; & in this list, so the count is exact and a third argument has nowhere to +;;;; go. +(defmacro pair [a b] + `(+ ~a ~b)) + +(defn main [] i32 + (print (pair 1 2 3)) + 0) diff --git a/test/programs/macro-arity.flan b/test/programs/macro-arity.flan new file mode 100644 index 0000000..5b5d994 --- /dev/null +++ b/test/programs/macro-arity.flan @@ -0,0 +1,12 @@ +;;;; Too few arguments for the macro's parameter list, refused at the call. +;;;; +;;;; Nothing here is a run-time claim and nothing here expands: check_call +;;;; counts the call against the list before do-grid is ever run, so the +;;;; location is the line below rather than a node of an expansion stamped +;;;; with Loc.from_macro. +(defmacro do-grid [[r rows c cols] & body] + `(dotimes [~r ~rows] (dotimes [~c ~cols] ~@body))) + +(defn main [] i32 + (do-grid) + 0) diff --git a/test/programs/macro-cycle.flan b/test/programs/macro-cycle.flan index 73b48b5..fa3cd69 100644 --- a/test/programs/macro-cycle.flan +++ b/test/programs/macro-cycle.flan @@ -10,10 +10,10 @@ ;;;; to exist first -- see macro-spin.flan, which is bounded rather than ;;;; refused. -(defmacro ping [args] +(defmacro ping [& args] (pong args)) -(defmacro pong [args] +(defmacro pong [& args] (ping args)) (defn main [] i32 diff --git a/test/programs/macro-destructure-arity.flan b/test/programs/macro-destructure-arity.flan new file mode 100644 index 0000000..e979e48 --- /dev/null +++ b/test/programs/macro-destructure-arity.flan @@ -0,0 +1,7 @@ +;;;; A vector of the wrong length for the pattern. Four names, three forms. +(defmacro do-grid [[r rows c cols] & body] + `(dotimes [~r ~rows] (dotimes [~c ~cols] ~@body))) + +(defn main [] i32 + (do-grid [i 2 j] (println "never")) + 0) diff --git a/test/programs/macro-destructure.flan b/test/programs/macro-destructure.flan new file mode 100644 index 0000000..3328755 --- /dev/null +++ b/test/programs/macro-destructure.flan @@ -0,0 +1,9 @@ +;;;; A [ ] pattern meeting an argument that is not a vector. The pattern says +;;;; what the call has to look like, so this is refused where the argument is +;;;; written rather than inside an expansion that read (at ... 0) of a Sym. +(defmacro do-grid [[r rows c cols] & body] + `(dotimes [~r ~rows] (dotimes [~c ~cols] ~@body))) + +(defn main [] i32 + (do-grid 7 (println "never")) + 0) diff --git a/test/programs/macro-params.flan b/test/programs/macro-params.flan new file mode 100644 index 0000000..3eb1fdc --- /dev/null +++ b/test/programs/macro-params.flan @@ -0,0 +1,71 @@ +;;;; A macro's parameter list: positional names, [ ] patterns, and &. +;;;; +;;;; macros.flan is the other half of this and is deliberately not merged with +;;;; it: everything there is written [& args] and picks its arguments apart by +;;;; hand, which is what every macro in the tree looked like before this. Here +;;;; the parameter list does the picking, and the two files together are the +;;;; claim that both spellings are the same grammar rather than two. +;;;; +;;;; Nothing below checks its own arity. It cannot be reached with the wrong +;;;; one: lib/expand.ml's check_call runs over the call *before* the macro is +;;;; expanded, so a miscount is refused at the call with the call's own +;;;; location — see macro-arity.flan and the three beside it. + +;; The shape the feature was asked for (DISCUSS.org): a binding vector +;; destructured in the signature, and & for the body. Without a parameter list +;; this is (at args 0), a match on Form.Vec to unwrap it, four more (at ...) +;; inside that, and (form-rest args 1) for the body. +(defmacro do-grid [[r rows c cols] & body] + `(dotimes [~r ~rows] + (dotimes [~c ~cols] + ~@body))) + +;; One positional parameter, which is where the grammar changed: [x] used to +;; bind the whole argument list and now binds the first argument. The gensym is +;; the ordinary reason it is there — expansion is not hygienic — and not +;; anything to do with the parameter list. +(defmacro doubled [x] + (let [v (gensym)] + `(let [~v ~x] (+ ~v ~v)))) + +;; Patterns nest, because a pattern's elements are patterns. And & is not only +;; the top level's: the tail of a pattern is the tail of that vector. +(defmacro nested [[a [b c]] & body] + `(do (print ~a) (print ~b) (print ~c) ~@body)) + +;; & inside a pattern, which is the same & and means the same thing one level +;; down: the tail of the vector written at the call. +(defmacro first-of [[a & more]] + `(do (print ~a) ~@more)) + +;; & with nothing after it at the call: the rest is an empty slice, ~@ splices +;; nothing, and the expansion is the wrapper alone. The arity check says "at +;; least 1" and one is what this is given. +(defmacro shout [label & body] + `(do (print ~label) ~@body (println "!"))) + +;; The whole argument list, which is what [args] used to mean and is now spelled +;; [& args]. Every macro in the tree was migrated to this line, so it is the +;; one that has to keep working unchanged. +(defmacro all-of [& args] + (if (= (len args) 0) + `true + `(if ~(at args 0) (all-of ~@(form-rest args 1)) false))) + +(defn main [] i32 + (do-grid [i 2 j 3] + (print i) (print j)) + (println "") + + (print (doubled 21)) (println "") + + (nested [1 [2 3]] (println " nested")) + (first-of [4 (print " and") (println " more")]) + + (shout "alone") + (shout "with" (print " body")) + + (print (all-of)) (print " ") + (print (all-of true true true)) (print " ") + (print (all-of true false true)) (println "") + 0) diff --git a/test/programs/macro-spin.flan b/test/programs/macro-spin.flan index cf8ed78..bccc2bd 100644 --- a/test/programs/macro-spin.flan +++ b/test/programs/macro-spin.flan @@ -3,7 +3,7 @@ ;;;; that does not terminate, so it is bounded and the bound says which macro ;;;; ran out rather than the compiler running out of memory. -(defmacro spin [args] +(defmacro spin [& args] `(spin ~@args)) (defn main [] i32 diff --git a/test/programs/macros.flan b/test/programs/macros.flan index c84cbf2..76c9328 100644 --- a/test/programs/macros.flan +++ b/test/programs/macros.flan @@ -5,18 +5,21 @@ ;;;; parsed. What arrives here is the expansion; nothing at run time knows a ;;;; macro was involved. ;;;; -;;;; A macro takes one parameter, the slice of forms written at its call site, -;;;; and answers one form. That is where variadics come from in a language with -;;;; no &rest: (len args) is how many were written. +;;;; Every macro below is written [& args] and picks its arguments apart by +;;;; hand, which is what a macro looked like before parameter lists: & binds +;;;; the whole call as a slice of forms, and (len args) is how many were +;;;; written. macro-params.flan is the other spelling of the same grammar — +;;;; named parameters, [ ] patterns, & only for the tail — and the two files +;;;; are kept apart on purpose, so that each is a whole program in one style. ;; The simplest one there is: two forms, in order. It proves the call site's ;; arguments arrive as forms and come back as code. -(defmacro both [args] +(defmacro both [& args] `(do ~(at args 0) ~(at args 1))) ;; Splicing, which is the only reason ~@ exists: the body is however many forms ;; were written, and they go where a list is expected. -(defmacro when2 [args] +(defmacro when2 [& args] `(if ~(at args 0) (do ~@(form-rest args 1)))) ;; Expansion is not hygienic -- Common Lisp's rule and Clojure's, settled in @@ -27,7 +30,7 @@ ;; ;; Without this, `twice` would bind `tmp` and the caller's own `tmp` would be ;; shadowed inside it. The two calls below are the difference. -(defmacro twice [args] +(defmacro twice [& args] (let [v (gensym)] `(let [~v ~(at args 0)] (+ ~v ~v)))) @@ -36,7 +39,7 @@ ;; nothing: `both` is inside the quasiquote, so it is part of what this macro ;; *returns* and is expanded again after it returns, and `announce` can be ;; compiled without `both` existing. -(defmacro announce [args] +(defmacro announce [& args] `(both (print "-> ") ~(at args 0))) ;; This is the one that makes the pre-pass a fixpoint rather than a sweep. The @@ -45,16 +48,16 @@ ;; and until it is, `id` is a name nothing defines and this body will not ;; compile at all. So round 0 takes `id`, round 1 expands this against it, and ;; the module that finally answers a call holds both. -(defmacro id [args] +(defmacro id [& args] (at args 0)) -(defmacro quiet [args] +(defmacro quiet [& args] (id `(println "a macro that called a macro"))) ;; And a macro that expands into a call to itself, which is what every ;; conditional macro in every Lisp is. It gets smaller each time and stops at ;; the empty case, so the expander's fuel never comes into it. -(defmacro all-of [args] +(defmacro all-of [& args] (if (= (len args) 0) `true `(if ~(at args 0) (all-of ~@(form-rest args 1)) false))) diff --git a/test/programs/pkg-macro.flan b/test/programs/pkg-macro.flan index c749a62..956329c 100644 --- a/test/programs/pkg-macro.flan +++ b/test/programs/pkg-macro.flan @@ -16,9 +16,17 @@ (import mac "pkgs/mac") ;; A macro of the program's own, coexisting with the package's. -(defmacro tenfold [args] +(defmacro tenfold [& args] `(* ~(at args 0) 10)) +;; The same macro again, written with a parameter list instead of by hand. +;; Nothing calls it: it is here for the equivalence case in test_session, which +;; expands (tenfold 7) and (tenfold-listed 7) and requires the same text out of +;; both. [& args] and [n] are one grammar, and this is where that is asserted +;; rather than assumed. +(defmacro tenfold-listed [n] + `(* ~n 10)) + (defn show [n i32] () (print n) (println "")) (defn main [] i32 diff --git a/test/programs/pkgs/mac/mac.flan b/test/programs/pkgs/mac/mac.flan index fb25972..0c5c5c4 100644 --- a/test/programs/pkgs/mac/mac.flan +++ b/test/programs/pkgs/mac/mac.flan @@ -11,7 +11,7 @@ (defn double [n i32] i32 (* n 2)) ;; The plain case: one macro, nothing else needed to compile it. -(defmacro twice [args] +(defmacro twice [& args] `(+ ~(at args 0) ~(at args 0))) ;; A macro that quasiquotes a call to another macro of this package. That is @@ -19,30 +19,30 @@ ;; macro answers and is expanded again after it returns -- so it needs nothing ;; compiled first. What it does need is the name coming out qualified, because ;; the answer lands in the importer's file, where [twice] is not a name. -(defmacro quad [args] +(defmacro quad [& args] `(twice (twice ~(at args 0)))) ;; And one whose output names a *function* of this package, which has the same ;; problem and the same answer. -(defmacro doubled [args] +(defmacro doubled [& args] `(double ~(at args 0))) ;; [wrap] takes a form-valued expression and answers one, so it is a macro ;; another macro's *body* can call for real. -(defmacro wrap [args] +(defmacro wrap [& args] `(do ~(at args 0))) ;; A macro that really calls another, outside a quasiquote. This one *is* a ;; compile-order dependency: [wrap] has to be compiled and loaded before this ;; body will compile at all, which is what the rounds in [Macro] are for, and ;; it is the case a quasiquoted call deliberately is not. -(defmacro also-twice [args] +(defmacro also-twice [& args] (wrap `(+ ~(at args 0) ~(at args 0)))) ;; A shadowing local named like a top-level of this package. The rename must ;; leave it alone, or the expansion would name [mac/double] where the author ;; wrote a let binding. -(defmacro shadowed [args] +(defmacro shadowed [& args] `(let [double ~(at args 0)] (+ double 1))) diff --git a/test/programs/pkgs/macring/macring.flan b/test/programs/pkgs/macring/macring.flan index dfdd6c8..a18f6d5 100644 --- a/test/programs/pkgs/macring/macring.flan +++ b/test/programs/pkgs/macring/macring.flan @@ -3,8 +3,8 @@ ;;;; the same rounds as the file's own, so the refusal has to fire here too. ;;;; Nothing in this package calls them, so the ring is found by the importer. -(defmacro ping [args] +(defmacro ping [& args] (pong args)) -(defmacro pong [args] +(defmacro pong [& args] (ping args)) diff --git a/test/programs/pkgs/macspin/macspin.flan b/test/programs/pkgs/macspin/macspin.flan index c84c254..82454b4 100644 --- a/test/programs/pkgs/macspin/macspin.flan +++ b/test/programs/pkgs/macspin/macspin.flan @@ -4,5 +4,5 @@ ;;;; site. The name in that message is the qualified one, because that is what ;;;; the importer wrote. -(defmacro spin [args] +(defmacro spin [& args] `(spin ~@args)) diff --git a/test/programs/printers.flan b/test/programs/printers.flan index 88aff65..1dcda6e 100644 --- a/test/programs/printers.flan +++ b/test/programs/printers.flan @@ -30,7 +30,7 @@ ;;; declares it. Nothing in this program names it, so no macro module is built ;;; for the build itself -- the first one is paid by the evaluation that calls ;;; it. -(defmacro tenfold [args] +(defmacro tenfold [& args] `(* ~(at args 0) 10)) (defn main [] i32 diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index 5d8a0b3..8993c41 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -3371,6 +3371,26 @@ level "1" outputs ~opt:"-O0" "macros, -O0" "programs/macros.flan" macros_out; outputs ~dev:true "macros, dev" "programs/macros.flan" macros_out; + (* The same feature written the other way round: a real parameter list on + the defmacro, so the arguments are picked apart by the signature rather + than by hand. macros.flan above is every macro in the tree as it was + written before this — [& args] and (at args 0) — and both files are here + because both spellings are one grammar: [& args] is the trivial case of + the list, not a legacy mode kept alive beside it. + + Three opt levels for the reason macros.flan has them, and the dev row + because the dev path is this project's priority. *) + let macro_params_out = + "000102101112\n42\n123 nested\n4 and more\nalone!\nwith body!\n\ + true true false\n" + in + outputs "a macro's parameter list" "programs/macro-params.flan" + macro_params_out; + outputs ~opt:"-O0" "a macro's parameter list, -O0" + "programs/macro-params.flan" macro_params_out; + outputs ~dev:true "a macro's parameter list, dev" + "programs/macro-params.flan" macro_params_out; + (* A macro declared in an imported *package*, which is the half the refusal at [a package's macro is not visible unqualified] above leaves out. The program calls six of them qualified and one of its own @@ -3554,6 +3574,27 @@ level "1" is an ordinary loop and it is bounded. *) refuses "a ring of macros" "programs/macro-cycle.flan" "none can be compiled first"; + (* A call that does not fit the macro's parameter list, in all four of the + ways it can fail to. Every one of them is refused *before* the macro is + expanded, which is why each message carries the call's own location + rather than the [Loc.from_macro] stamp every node of an expansion gets — + that stamping is a documented limitation waiting on the structured-error + rewrite, and these four are the part of it that does not have to wait. *) + refuses "a macro call with too few arguments" "programs/macro-arity.flan" + "do-grid takes at least 1 argument and this call gives 0 — its \ + parameter list is [[r rows c cols] & body], where &body is the rest"; + refuses "a macro call with too many arguments" + "programs/macro-arity-extra.flan" + "pair takes 2 arguments and this call gives 3 — its parameter list is \ + [a b]"; + refuses "a destructuring parameter meeting a form that is not a vector" + "programs/macro-destructure.flan" + "do-grid destructures this argument with [r rows c cols], so a [ ] \ + belongs here and 7 was written"; + refuses "a destructuring parameter meeting a vector of the wrong length" + "programs/macro-destructure-arity.flan" + "do-grid destructures this argument with [r rows c cols], which takes 4, \ + and 3 are written here"; refuses "a macro that does not settle" "programs/macro-spin.flan" "did not settle after"; diff --git a/test/test_dev.ml b/test/test_dev.ml index e8cfbb2..c6bef30 100644 --- a/test/test_dev.ml +++ b/test/test_dev.ml @@ -3928,7 +3928,7 @@ let () = is also the case NEXT.md describes literally: a macro whose module does not build. *) let macro_defn = - "(defmacro plusone [args] `(+ ~(at args 0) 1)) \ + "(defmacro plusone [& args] `(+ ~(at args 0) 1)) \ (defn probe-one [] i64 (plusone 41))" in let before = knows () in diff --git a/test/test_flan.ml b/test/test_flan.ml index b3b33e6..c1f08f3 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -507,24 +507,46 @@ let () = be one: a macro is [Form] -> Form, compiled by the same backend as everything else, and what makes it a macro is that the expander calls it at compile time rather than the program calling it at run time. *) - (match (parse_decl "(defmacro m [args] (at args 0))").d with + (match (parse_decl "(defmacro m [& args] (at args 0))").d with | Defn { name = "m"; params = [ p ]; ret = Some r; _ } -> (match p.fty.t, r.t with | Tslice { t = Tname "Form"; _ }, Tname "Form" -> () | _ -> check "defmacro is [Form] -> Form" false) | _ -> check "defmacro parses as a defn" false); - (* One parameter, the forms at the call site. Two is not an arity mistake, it - is a misunderstanding of what a macro takes, and it gets its own reason. *) - parse_rejects "defmacro with two parameters" "(defmacro m [a b] a)" - ~needle:"a macro takes one parameter"; + (* The declared type is the same whatever the author wrote as a parameter + list: the list is bindings over the one slice, opened by [macro_body], and + nothing below the parser learns there was a list at all. *) + (match (parse_decl "(defmacro m [[a b] c & rest] (at rest 0))").d with + | Defn { name = "m"; params = [ p ]; ret = Some r; _ } -> + (match p.fty.t, r.t with + | Tslice { t = Tname "Form"; _ }, Tname "Form" -> () + | _ -> check "a parameter list is still [Form] -> Form" false) + | _ -> check "a macro with a parameter list parses as a defn" false); + + (* Several parameters is the feature now. What is still refused is a list + that cannot be read: [&] with nothing or too much after it, a pattern that + binds nothing, a name bound twice, and a map pattern — which is deferred + rather than unimplemented by accident, see FIX.org. *) + parse_rejects "defmacro with a dangling &" "(defmacro m [a &] a)" + ~needle:"& needs a name after it"; + parse_rejects "defmacro with two names after &" "(defmacro m [& a b] a)" + ~needle:"& takes one name and it is the last thing"; + parse_rejects "defmacro with a pattern after &" "(defmacro m [& [a b]] a)" + ~needle:"& binds one name for the rest of the arguments"; + parse_rejects "defmacro with an empty pattern" "(defmacro m [a []] a)" + ~needle:"binds nothing"; + parse_rejects "defmacro binding a name twice" "(defmacro m [a [b a]] a)" + ~needle:"a is bound twice in this parameter list"; + parse_rejects "defmacro with a map pattern" "(defmacro m [{:keys [a]}] a)" + ~needle:"map destructuring is not implemented in a macro's parameter list"; (* Shape and feature were separate mistakes and stay separate reasons. *) parse_rejects "defmacro with no body" "(defmacro m [x])" ~needle:"defmacro is (defmacro name [param ...] body ...)"; parse_rejects "defmacro with no params" "(defmacro m x)" ~needle:"defmacro is (defmacro name [param ...] body ...)"; parse_rejects "defmacro with a non-name param" "(defmacro m [1] x)" - ~needle:"expected a name"; + ~needle:"a macro's parameter is a name or a [ ] pattern"; parse_rejects "defmacro in expression position" "(defn f [] () (defmacro m [] 1))" ~needle:"top-level declaration"; @@ -3988,7 +4010,7 @@ let () = let synth src = Reader.read_all ~file:"" src in let chain = synth - "(defmacro m [args] `(do))\n\ + "(defmacro m [& args] `(do))\n\ (defn a [] () (m))\n\ (defn b [] () (a))\n\ (defn c [] () (do))\n" @@ -3998,7 +4020,7 @@ let () = (* And the one rule that stays: a prelude macro may not call a macro. It used to fail as an unknown name inside a clang build; it names itself now. *) - let ring = synth "(defmacro m [args] `(do))\n(defmacro n [args] (m args))\n" in + let ring = synth "(defmacro m [& args] `(do))\n(defmacro n [& args] (m args))\n" in check "a prelude macro calling a macro is refused by name" (match Macro.reduce ring with | _ -> false diff --git a/test/test_repl.ml b/test/test_repl.ml index 2687cc2..de6bba7 100644 --- a/test/test_repl.ml +++ b/test/test_repl.ml @@ -155,7 +155,7 @@ let () = the sentence naming it a declaration rather than an arity complaint about an unknown function. C-c C-c is where a declaration goes, which is the case below. *) - refuses "a defmacro at C-x C-e" "(defmacro m [args] args)" + refuses "a defmacro at C-x C-e" "(defmacro m [& args] args)" "top-level declaration"; (* And the session is untouched by all of it: an evaluation is not a @@ -172,7 +172,7 @@ let () = (let r = request c (Printf.sprintf "(:op \"eval\" :code %s :file \"/tmp/buf.flan\")" - (quote "(defmacro thrice [args] `(* ~(at args 0) 3))")) + (quote "(defmacro thrice [& args] `(* ~(at args 0) 3))")) in if status r <> "ok" then fail "evaluating a defmacro over the socket: %s" @@ -278,7 +278,7 @@ let () = (request c (Printf.sprintf "(:op \"macroexpand\" :code %s :file \"/tmp/buf.flan\")" - (quote "(defmacro looked-at [args] `(* ~(at args 0) 5))"))); + (quote "(defmacro looked-at [& args] `(* ~(at args 0) 5))"))); (let r = evals "(looked-at 3)" in if status r = "ok" then fail "a defmacro joined the session by being macroexpanded"); diff --git a/test/test_session.ml b/test/test_session.ml index 6dff651..360b2e8 100644 --- a/test/test_session.ml +++ b/test/test_session.ml @@ -431,7 +431,7 @@ let () = Out_channel.with_open_bin path (fun oc -> Out_channel.output_string oc text) in let macro op = - Printf.sprintf "(defmacro grow [args] + Printf.sprintf "(defmacro grow [& args] `(%s ~(at args 0) ~(at args 0))) " op in @@ -546,7 +546,7 @@ let () = case here that the create-time seed cannot explain. Two evaluations, because that is what the claim is about. *) (match Session.eval ~origin:"programs/pkg-macro.flan" tm - "(defmacro thrice [args] `(* ~(at args 0) 3))" + "(defmacro thrice [& args] `(* ~(at args 0) 3))" with | _ -> () | exception Loc.Error { Loc.dmsg = m; _ } -> @@ -562,7 +562,7 @@ let () = ordinary editing action and the one that would have reached [Check.program] as a duplicate declaration without the second of those. *) (match Session.eval ~origin:"programs/pkg-macro.flan" tm - "(defmacro thrice [args] `(* ~(at args 0) 4))" + "(defmacro thrice [& args] `(* ~(at args 0) 4))" with | _ -> () | exception Loc.Error { Loc.dmsg = m; _ } -> @@ -582,7 +582,7 @@ let () = and fails at the checker — which is the only interesting place to fail, because a parse failure never reaches the union either. *) (match Session.eval ~origin:"programs/pkg-macro.flan" tm - "(defmacro nope [args] (no-such-function args))" + "(defmacro nope [& args] (no-such-function args))" with | _ -> fail "a defmacro whose body does not check was accepted" | exception Loc.Error _ -> ()); @@ -693,6 +693,30 @@ let () = of what one step is for. *) expands "one step does not expand the arguments first" "(mac/twice (mac/twice 3))" "(+ (mac/twice 3) (mac/twice 3))"; + (* One macro written both ways, expanded to the same text. [tenfold] picks + its argument out of the slice by hand and [tenfold-listed] names it in the + parameter list; there is one grammar under both, so the two expansions + have to be the same string and not merely the same shape. + + This is the migration's evidence. Every [defmacro] in the tree was + rewritten from [args] to [& args] when the list stopped meaning "the whole + call" and started meaning "the first argument", and what makes that a + spelling change rather than a behaviour change is exactly this. *) + expands "a macro that picks its argument out by hand" "(tenfold 7)" "(* 7 10)"; + expands "the same macro with a parameter list" "(tenfold-listed 7)" "(* 7 10)"; + + (* And the call-site check on this path, which is the editor's rather than a + build's. [Macro.checked_call] is one function for all four ways in — the + walk, [settle], C-c C-m's one step and its fixpoint — so C-c C-m over a + miscounted call refuses with the sentence a build would give. *) + (match Session.macroexpand ~origin:"programs/pkg-macro.flan" ~all:false tm + "(tenfold-listed 1 2)" + with + | _ -> fail "C-c C-m expanded a macro call with the wrong arity" + | exception Loc.Error { Loc.dmsg = m; _ } -> + if not (has m "tenfold-listed takes 1 argument and this call gives 2") + then fail "C-c C-m over a miscounted call said: %s" m); + (* Not a macro call at all. The form comes back as it was, and the answer that matters is [xmacro]: nothing ran. *) (match Session.macroexpand ~origin:"programs/pkg-macro.flan" ~all:false tm @@ -724,7 +748,7 @@ let () = *aftermath*, and it is checked the only way it can be — by calling the name and requiring it to still be unknown. *) (match Session.macroexpand ~origin:"programs/pkg-macro.flan" ~all:false tm - "(defmacro looked-at [args] `(* ~(at args 0) 5))" + "(defmacro looked-at [& args] `(* ~(at args 0) 5))" with | _ -> () | exception Loc.Error { Loc.dmsg = m; _ } -> diff --git a/vendor/edn/provide.flan b/vendor/edn/provide.flan index c4b9589..4d2e95f 100644 --- a/vendor/edn/provide.flan +++ b/vendor/edn/provide.flan @@ -528,7 +528,7 @@ ;; points over the whole thing. `C-c C-m` over the call shows all of it, which ;; is the point of generating readable code rather than the smallest code — a ;; provider whose output nobody can look at is a plugin. -(defmacro defedn [args] +(defmacro defedn [& args] (if (!= (len args) 2) (refuse "defedn is (defedn Name \"path.edn\") — a name for the struct, and a path to the file its shape is read out of") (match (at args 1) diff --git a/vendor/json/provide.flan b/vendor/json/provide.flan index 3da211f..af0c56c 100644 --- a/vendor/json/provide.flan +++ b/vendor/json/provide.flan @@ -409,7 +409,7 @@ ;; It answers a `do`, which the top level splices: the nested structs innermost ;; first, then the struct named here, a reader per struct, and the two entry ;; points over the whole thing. -(defmacro defjson [args] +(defmacro defjson [& args] (if (!= (len args) 2) (refuse "defjson is (defjson Name \"path.json\") — a name for the struct, and a path to the file its shape is read out of") (match (at args 1) diff --git a/vendor/raylib/modes.flan b/vendor/raylib/modes.flan index 86ebc60..327bc74 100644 --- a/vendor/raylib/modes.flan +++ b/vendor/raylib/modes.flan @@ -93,7 +93,7 @@ ;; The frame. Everything drawn lands on the back buffer; end-drawing swaps it ;; and waits out the frame time set by set-target-fps. -(defmacro with-drawing [args] +(defmacro with-drawing [& args] (if (or (< (len args) 1) (and (= (len args) 1) (form-empty-list? (at args 0)))) `(with-drawing-takes-a-body) @@ -104,7 +104,7 @@ ;; The 2D camera. The argument is a Camera2D value, evaluated once where it ;; always was. Remember that a fresh (Camera2D {}) has zoom 0.0 and is not ;; usable as an identity — raylib.flan says so beside the struct. -(defmacro with-mode-2d [args] +(defmacro with-mode-2d [& args] (if (or (< (len args) 2) (and (= (len args) 2) (form-empty-list? (at args 1)))) `(with-mode-2d-takes-a-camera-and-a-body) @@ -115,7 +115,7 @@ ;; The 3D camera. Same shape, same argument-once rule, and the pair matters ;; more here than anywhere: ending a 3D mode with end-mode-2d type-checks ;; fine and leaves the projection matrix wrong for everything after it. -(defmacro with-mode-3d [args] +(defmacro with-mode-3d [& args] (if (or (< (len args) 2) (and (= (len args) 2) (form-empty-list? (at args 1)))) `(with-mode-3d-takes-a-camera-and-a-body) @@ -127,7 +127,7 @@ ;; of the GPU upside down, so drawing it back wants a negative source height — ;; that correction is the caller's and is deliberately not hidden here, since ;; it belongs with the draw and not with the mode. -(defmacro with-texture-mode [args] +(defmacro with-texture-mode [& args] (if (or (< (len args) 2) (and (= (len args) 2) (form-empty-list? (at args 1)))) `(with-texture-mode-takes-a-target-and-a-body) @@ -138,7 +138,7 @@ ;; Clip to a rectangle, in screen pixels with y down from the top. Four ;; scalars rather than a Rectangle, because that is what BeginScissorMode ;; takes and this file is not the place to invent a second spelling. -(defmacro with-scissor-mode [args] +(defmacro with-scissor-mode [& args] (if (or (< (len args) 5) (and (= (len args) 5) (form-empty-list? (at args 4)))) `(with-scissor-mode-takes-x-y-width-height-and-a-body)