From ca14394e0f6eb6e853c459b00ee43464046d4354 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sun, 13 Sep 2026 09:27:20 +0700 Subject: [PATCH] into fuses at compile time because it is a macro, not a transducer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (into xs (vec-new i32) (map double) (filter even?)). The function name is syntax and never a value, so (map double) is (double x) written into the loop body: no intermediate collection, no closure, no generics, nothing to inline. Transducers would compose at run time and Rust's iterators need monomorphisation; a macro needs neither. into.flan counts the pulls, which is the assertion a unit test cannot make. The destination is in the form because the destination is the allocation, and that is what makes spec-memory.md's explicit-allocator rule true by construction rather than by convention. Which also settles the open question: reductions do not share the form. A seed is not an allocation, so (into xs 0 (map cost) (sum)) would be a second form wearing the same spelling, and the destination would stop being honest about what it is. A source that is already a name is used as it is, not bound. A (Vec T) is move-only, so binding it would take the caller's ownership for something that only reads; a fixed array would be copied once per into. A source that is anything else is still bound once, which is what a call needs. The wart is that an owning temporary there leaks, because the binding has a name the caller cannot reach — a call in that position should borrow, and drop is what would close it. All four of the prelude's macro limits bit and none blocked anything. The three refusals are names nothing defines, which is the only error facility a macro has. into-wrap is a defn using only special forms, so Macro.reduce does not drop it, and it is the first thing in the prelude written as a loop/recur. --- BUILT.md | 74 ++++++++++++++++++++++++ NEXT.md | 21 ++++++- lib/prelude.ml | 122 ++++++++++++++++++++++++++++++++++++++++ test/programs/into.flan | 81 ++++++++++++++++++++++++++ test/test_acceptance.ml | 6 ++ test/test_flan.ml | 12 ++++ 6 files changed, 315 insertions(+), 1 deletion(-) create mode 100644 test/programs/into.flan diff --git a/BUILT.md b/BUILT.md index 0e03483..14eeaa8 100644 --- a/BUILT.md +++ b/BUILT.md @@ -3006,6 +3006,80 @@ and the function still has to answer. The loop-with-a-sentinel-flag shape that b in the prelude. The two compiler-emitted retry loops in `check.ml` are that shape, and they are the one place it cannot help: their sentinel is set inside a `restart-case` body, which is a barrier. +## `into`, which fuses at compile time because it is a macro + +``` +(into xs (vec-new i32) (map double) (filter even?)) +``` + +Source, destination, then any number of transforms — the shape of the `into->` macro the author already uses in +Clojure. It reads as a sentence (take this, put it there, doing these) and the variadic tail has to trail anyway, +which is the mechanical reason the transforms cannot sit in the middle. + +**A macro, and therefore not transducers and not iterators.** Transducers compose at *run time*: function values, +closures, an allocation, and a chain of indirect calls per element. Rust has no transducers either — it has iterators, +which fuse into one loop at compile time through monomorphisation, and that needs generics. A macro reaches the same +place with neither. `(map double)` expands to `(double x)` written straight into the loop body, so the function name +is **syntax and never a value**: no intermediate collection at any step, no closure, no generics, and nothing to +inline. `into.flan` counts every call to the transform functions, which is the assertion a unit test cannot make — a +chain that built a `Vec` per stage would pull a different number. + +What it gives up is building a transformation at run time and passing it around. That is transducers' actual selling +point, it is the one part that would need run-time machinery, and it is close to useless in a game. Clojure's +`:eduction` branch is dropped for exactly that reason. + +**Why the destination is in the form.** Every collecting operation here allocates from an *explicit* allocator, which +is a frozen rule in `spec-memory.md`. A `->>` chain hides where the result goes; naming the destination means the +macro knows its type, emits the right loop, and the rule is honoured by construction. `(vec-new i32 a)` names an +allocator here as it does anywhere else, because the destination form is written out untouched. This is what `->>` +threading over slices in `plan.org` is replaced by for the collecting cases. + +### Reductions do not share the form, and that was the open question + +`(into xs 0 (map cost) (sum))` reads oddly because zero is not a collection, and the oddness is the tell rather than a +matter of taste. The whole reason the destination sits in the form is that **the destination is the allocation** — it +is what makes the explicit-allocator rule checkable by construction. A seed is not an allocation, so a form that +accepted one would be two forms sharing a spelling, and the destination would have stopped being honest about what it +is. So `into` collects, and a reducing macro of the same shape is a separate form the day something wants one. + +### The parts that took a decision + +- **The destination is a `Vec`**, because `push` is what fills it. A `Map` destination is refused by `push` itself, + which says *push takes a (Vec T)* and names the real problem. There is no second lowering and no reason to invent + one before something asks. +- **A source that is already a name is used as it is; anything else is bound to a gensym.** Both halves are needed and + neither is cosmetic. Binding is what a source that is a *call* needs — `(len s)` and `(at s i)` have to be the same + `s`, or the call is made once per element. Not binding a name is what everything else needs: a `(Vec T)` is + move-only, so `(let [s v] ...)` would hand the caller's `v` to a binding it cannot see and `v` would be dead after + an `into` that only read it; and a fixed array would be *copied* into the binding, once per `into`. `len` and `at` + borrow, so used directly the source is only read. +- **An owning temporary as the source leaks**, and this is the wart. `(into (make-a-vec) ...)` binds the result to a + name the caller cannot reach and therefore cannot free. The macro cannot know whether the type owns anything. A call + in that position should borrow — `into.flan`'s does — and the day `drop` exists this stops being a question. +- **One element name throughout**, shadowed by each `(map f)` stage: `(let [x (f x)] ...)`. A `let` binding's value is + checked before its name is bound, so the initialiser reads the outer `x` — that is the language's rule, not an + accident, and `bind`'s `x~2` debug suffix exists precisely so a debugger does not lie about which is which. + +### What the prelude's macro limits cost, exactly + +All four bit, and none blocked anything. + +- **A macro has no error facility**, so the three refusals are calls to names nothing defines: + `into-takes-a-source-a-destination-and-transforms`, `into-transform-is-map-or-filter` and + `…-of-one-function`. The report is *unknown function* at the call site with a *expanded from the macro into* note + under it, which is the right place and the wrong sentence. The bad transform is passed along as an argument so that + at least it is named. +- **A prelude macro may not call another macro**, so `into-wrap` is a plain `defn` and uses only special forms — + `loop`, `cond`, `when`, `let`, `if`. A `clamp` or an `unless` in there would have put it in the set `Macro.reduce` + drops. +- **Nested quasiquote is refused**, and it was not needed: each wrapper is a single-level quasiquote over a `body` + already built. +- **Macros are not importable**, which is why `into` is in the prelude rather than a library. + +`into-wrap` walks the transforms **in reverse**, because the chain is built from the inside out: the innermost form is +the `push`, and each transform wraps what the ones after it produced. That reverse walk with two accumulators is the +first thing in the prelude written as a `loop`/`recur`, which landed in the commit before this one. + ## `loop` and `recur`, and why `recur` is better than tail calls and not only cheaper There is no TCO anywhere in this compiler — nothing emits a tail call, and `plan.org` mentions them only as something diff --git a/NEXT.md b/NEXT.md index 4bb0aee..ede707f 100644 --- a/NEXT.md +++ b/NEXT.md @@ -789,7 +789,7 @@ Sources are community consensus rather than a specification; the `contains?` com Clojure*. Recorded because these are cheap to honour now and expensive to unpick once a standard library depends on them. -## Queued: `into`, fused transformation without transducers +## ~~Queued: `into`, fused transformation without transducers~~ — **landed** Decided in conversation. **Not transducers, and not Rust's iterators — a macro that fuses the chain at compile time.** @@ -826,6 +826,25 @@ destination is always honest about what it is. Drop Clojure's `:eduction` branch — that is the pass-around case, and the one part that would need runtime machinery. +**Done.** See *`into`, which fuses at compile time because it is a macro* in [`BUILT.md`](BUILT.md). It is a prelude +`defmacro` over a plain `defn` that walks the transforms in reverse, and all four of the macro limits bit without +blocking anything: the three refusals are names nothing defines, `into-wrap` uses only special forms so +`Macro.reduce` does not drop it, the quasiquotes are all single-level, and `into` lives in the prelude because a +macro is not importable. + +**The open question is settled: reductions do not share the form.** The reason the destination sits in `into` at all +is that the destination *is* the allocation, which is what makes `spec-memory.md`'s explicit-allocator rule true by +construction. A seed is not an allocation, so `(into xs 0 (map cost) (sum))` would be a second form wearing the same +spelling and the destination would stop being honest about what it is. A reducing macro of the same shape is a +separate form the day something wants one. + +Two things the design did not anticipate, both written up there. **A source that is already a name is used as it is** +rather than bound — a `(Vec T)` is move-only, so binding it would take the caller's ownership for a read, and a fixed +array would be copied once per `into`; a source that is anything else is still bound once, which is what a call +needs. And **an owning temporary as the source leaks**, because the macro binds it to a name the caller cannot reach +and cannot know whether the type owns anything. A call in that position should borrow. `drop` is what would close +this, and it does not exist. + ## ~~Queued: `loop`/`recur` (the return type is done)~~ — **landed** ~~**1. A `defn` must always state its return type, and unit is written `()`.**~~ **Done.** See *The return type is diff --git a/lib/prelude.ml b/lib/prelude.ml index fa56723..4184cb5 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -1519,6 +1519,128 @@ let source = {flan| `(unless-takes-a-test-and-a-body) `(if (not ~(at args 0)) (do ~@(form-rest args 1))))) +;; ── into: a fused transformation, and not a transducer ──────────────── +;; +;; (into xs (vec-new i32) (map double) (filter even?)) +;; +;; Source, destination, then any number of transforms. It reads as a sentence +;; — take this, put it there, doing these — and the variadic tail has to trail +;; anyway, which is the mechanical reason the transforms cannot sit in the +;; middle. +;; +;; **A macro, not transducers.** Transducers compose at run time: function +;; values, closures, an allocation, and a chain of indirect calls per element. +;; Rust has no transducers either — it has iterators, which fuse at compile +;; time through monomorphisation, and that needs generics. A macro reaches the +;; same place with neither. (map double) expands to (double x) written straight +;; into the loop body, so the function name is *syntax* and never a value: +;; there is no intermediate collection at any step, no closure, no generics and +;; nothing to inline. What it gives up is building a transformation at run time +;; and passing it around, which is transducers' actual selling point and is +;; close to useless in a game. +;; +;; **The destination is in the form on purpose.** Every collecting operation +;; here allocates from an explicit allocator, which is a frozen rule in +;; spec-memory.md. A ->> chain would hide where the result goes; naming the +;; destination means this macro knows its type, emits the right loop, and the +;; rule is honoured by construction. `(vec-new i32 a)` names an allocator here +;; as it does anywhere else, because the destination form is written out +;; untouched. +;; +;; **The destination is a Vec**, because push is what fills it. A Map +;; destination is refused by push, which says "push takes a (Vec T)" and names +;; the real problem; there is no second lowering for it and no reason to invent +;; one before something wants it. +;; +;; **Reductions do not share this form**, and that was the open question. (into +;; xs 0 (map cost) (sum)) reads oddly because zero is not a collection, and the +;; oddness is the tell: the whole reason the destination sits in the form is +;; that it is the allocation, and a seed is not one. Keeping `into` to +;; collections means the destination is always honest about what it is. A +;; reducing macro of the same shape is a separate form when something wants it. + +;; The chain, built from the inside out: the innermost form is the push, and +;; each transform wraps whatever the transforms after it produced. Walked in +;; reverse for that reason, which is what the loop's two names are. +;; +;; One element name throughout, shadowed by each (map f) stage. A let binding's +;; value is checked before the name is bound, so (let [x (f x)] ...) reads the +;; outer x and binds the inner one — that is the language's rule and not an +;; accident of the compiler; the debug-info suffix `x~2` exists precisely so a +;; debugger does not lie about which is which. The name is a gensym, so it +;; cannot collide with anything at the call site. +;; +;; A transform that is neither map nor filter expands to a call to a name +;; nothing defines, which is how a macro reports anything at all: it has no +;; error facility, so the report is "unknown name" at the call site — the right +;; place and the wrong sentence. The bad transform is passed along so that at +;; least it is named. +(defn into-wrap [ts [Form] dst Form x Form] Form + (loop [k (len ts) body `(push ~dst ~x)] + (if (= k 0) + body + (let [t (at ts (- k 1)) + items (form-items t)] + (if (!= (len items) 2) + `(into-transform-is-map-or-filter-of-one-function ~t) + (let [head (at items 0) + f (at items 1)] + (cond + (form-sym? head "map") (recur (- k 1) `(let [~x (~f ~x)] ~body)) + (form-sym? head "filter") (recur (- k 1) `(when (~f ~x) ~body)) + :else `(into-transform-is-map-or-filter ~t)))))))) + +;; The items of a list form, and the empty slice for anything else — a +;; non-list transform falls into the arity complaint above rather than needing +;; a case of its own. +(defn form-items [f Form] [Form] + (match f + (Form.List xs) xs + _ (form-nil))) + +(defn form-sym? [f Form name string] bool + (match f + (Form.Sym s) (bytes=? (bytes s) (bytes name)) + _ false)) + +(defn form-is-sym? [f Form] bool + (match f + (Form.Sym s) true + _ false)) + +(defn form-pair [a Form b Form] [Form] + (form-cons a (form-cons b (form-nil)))) + +;; A source that is already a name is used as it is, and a source that is +;; anything else is bound to one. Both halves matter. +;; +;; Binding it is what a source that is a *call* needs: (len s) and (at s i) +;; have to be the same s, and without the binding the call would be made twice +;; per element. +;; +;; Not binding a name is what everything else needs. A (Vec T) is move-only, so +;; (let [s v] ...) would hand v's ownership to the macro's own binding and the +;; caller would find v dead after an (into v ...) that only read it — and a +;; fixed array would be *copied* into the binding, once per into. Neither is +;; what was written. len and at borrow, so used directly the source is only +;; read. A source that is a call and produces a Vec is still consumed, which is +;; right: nobody else is holding it. +(defmacro into [args] + (if (< (len args) 2) + `(into-takes-a-source-a-destination-and-transforms) + (let [from (at args 0) + named? (form-is-sym? from) + src (if named? from (gensym)) + bind (if named? (form-nil) (form-pair src from)) + dst (gensym) + x (gensym) + i (gensym)] + `(let [~dst ~(at args 1) ~@bind] + (dotimes [~i (len ~src)] + (let [~x (at ~src ~i)] + ~(into-wrap (form-rest args 2) dst x))) + ~dst)))) + |flan} let file = "" diff --git a/test/programs/into.flan b/test/programs/into.flan new file mode 100644 index 0000000..3714892 --- /dev/null +++ b/test/programs/into.flan @@ -0,0 +1,81 @@ +;;;; into — a fused transformation, and not a transducer. +;;;; +;;;; What is asserted here is the thing a unit test cannot see: the loop that +;;;; comes out is one loop, and the values it produces are the ones the chain +;;;; describes in the order it was written. +;;;; +;;;; 1. The order of the transforms is the order of the stages. (filter even?) +;;;; before (map double) is not the same program as after it, and both are +;;;; here with different answers. +;;;; 2. There is no intermediate collection. `pulls` counts every call to the +;;;; transform functions: one pass over the source is one call per element +;;;; per stage it reaches, and a chain that built a Vec per stage would pull +;;;; a different number. +;;;; 3. A source that is already a name is only read. `src` is a (Vec i32), +;;;; which is move-only, and it is still alive and freeable afterwards. +;;;; 4. A source that is a call is evaluated once, not once per element. + +(defvar pulls i32 0) + +(defn double [x i32] i32 + (set pulls (+ pulls 1)) + (* x 2)) + +(defn even? [x i32] bool + (set pulls (+ pulls 1)) + (= (% x 2) 0)) + +(defvar builds i32 0) + +;; A source that is a call. It must be made once, however many elements come +;; out of it. It borrows rather than allocating, which is the shape a call in +;; this position wants: the macro binds the value to a name the caller cannot +;; see, so an owning temporary here would be a leak nobody can reach. +(defn source [xs [i32]] [i32] + (set builds (+ builds 1)) + (slice xs 0 (len xs))) + +(defn show [v [i32]] () + (dotimes [i (len v)] (print (at v i)) (print " ")) + (println "")) + +(defn main [] i32 + ;; No transforms: a copy into the destination named in the form. + (let [xs [7 8 9] + v (into xs (vec-new i32))] + (show (as-slice v)) ; 7 8 9 + (free v)) + + ;; map then filter. + (let [xs [1 2 3 4 5 6] + v (into xs (vec-new i32) (map double) (filter even?))] + (show (as-slice v)) ; 2 4 6 8 10 12 + (free v)) + + ;; filter then map, over the same source: a different answer, because the + ;; stages are in the order they were written. + (let [xs [1 2 3 4 5 6] + v (into xs (vec-new i32) (filter even?) (map double))] + (show (as-slice v)) ; 4 8 12 + (free v)) + + ;; One pass and no intermediate collection. The two chains above pulled + ;; 6 doubles + 6 evens, then 6 evens + 3 doubles: 21. + (print pulls) (println "") ; 21 + + ;; A name as the source is read, not moved: src is still alive here. + (let [src (into [3 1 2] (vec-new i32)) + v (into src (vec-new i32) (map double))] + (show (as-slice v)) ; 6 2 4 + (show (as-slice src)) ; 3 1 2 + (free v) + (free src)) + + ;; A source that is a call is bound once, so it is made once however many + ;; elements come out of it. + (let [xs [1 2 3 4] + v (into (source (slice xs 0 4)) (vec-new i32) (filter even?))] + (show (as-slice v)) ; 2 4 + (free v)) + (print builds) (println "") ; 1 + 0) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index 288b0f0..e00fd4f 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -132,6 +132,12 @@ let () = rebinds every name at once, and interleaved writes would print 1. *) outputs "loop and recur" "programs/recur.flan" "10\n2\n21\n8\n10000000\n64\n012\n0\n4\n6\n"; + (* into. The count of pulls is the assertion a unit test cannot make: one + pass, one call per element per stage it reaches, and no intermediate + collection anywhere. The two show lines either side of it are the same + source transformed in two orders, which have to differ. *) + outputs "into" "programs/into.flan" + "7 8 9 \n2 4 6 8 10 12 \n4 8 12 \n21\n6 2 4 \n3 1 2 \n2 4 \n1\n"; (* The prelude's slice algorithms. Every assertion here is over an input a wrong implementation fails: unsorted with duplicates, negatives and an odd length; a reverse-sorted slice; and a sort of a subslice whose diff --git a/test/test_flan.ml b/test/test_flan.ml index fe24390..0bb7a3a 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -989,6 +989,18 @@ let () = "(defn f [] () (loop :o [i 0] (recur i)))" ~needle:"loop takes no label"; rejects_check "a loop binding is a plain name" "(defn f [] () (loop [[a b] 0] (recur 0)))" ~needle:"destructuring pattern"; + (* into. The expansion is asserted in programs/into.flan, where the values + coming out are the test; what belongs here is the three things it refuses, + each through the one facility a macro has — a name nothing defines. *) + rejects_check "into needs a source and a destination" + "(defn f [] () (free (into [1 2 3])))" + ~needle:"into-takes-a-source-a-destination-and-transforms"; + rejects_check "a transform is map or filter" + "(defn f [] () (free (into [1 2 3] (vec-new i32) (take 2))))" + ~needle:"into-transform-is-map-or-filter"; + rejects_check "a transform names one function" + "(defn f [] () (free (into [1 2 3] (vec-new i32) (map))))" + ~needle:"into-transform-is-map-or-filter-of-one-function"; (* An import is resolved by [Load] before the checker runs, so one that reaches [Check] means a driver skipped that step. *) rejects_check "an unresolved import is a driver bug"