diff --git a/BUILT.md b/BUILT.md index 0e03483..14eeaa8 100644 --- a/BUILT.md +++ b/BUILT.md @@ -3006,6 +3006,80 @@ and the function still has to answer. The loop-with-a-sentinel-flag shape that b in the prelude. The two compiler-emitted retry loops in `check.ml` are that shape, and they are the one place it cannot help: their sentinel is set inside a `restart-case` body, which is a barrier. +## `into`, which fuses at compile time because it is a macro + +``` +(into xs (vec-new i32) (map double) (filter even?)) +``` + +Source, destination, then any number of transforms — the shape of the `into->` macro the author already uses in +Clojure. It reads as a sentence (take this, put it there, doing these) and the variadic tail has to trail anyway, +which is the mechanical reason the transforms cannot sit in the middle. + +**A macro, and therefore not transducers and not iterators.** Transducers compose at *run time*: function values, +closures, an allocation, and a chain of indirect calls per element. Rust has no transducers either — it has iterators, +which fuse into one loop at compile time through monomorphisation, and that needs generics. A macro reaches the same +place with neither. `(map double)` expands to `(double x)` written straight into the loop body, so the function name +is **syntax and never a value**: no intermediate collection at any step, no closure, no generics, and nothing to +inline. `into.flan` counts every call to the transform functions, which is the assertion a unit test cannot make — a +chain that built a `Vec` per stage would pull a different number. + +What it gives up is building a transformation at run time and passing it around. That is transducers' actual selling +point, it is the one part that would need run-time machinery, and it is close to useless in a game. Clojure's +`:eduction` branch is dropped for exactly that reason. + +**Why the destination is in the form.** Every collecting operation here allocates from an *explicit* allocator, which +is a frozen rule in `spec-memory.md`. A `->>` chain hides where the result goes; naming the destination means the +macro knows its type, emits the right loop, and the rule is honoured by construction. `(vec-new i32 a)` names an +allocator here as it does anywhere else, because the destination form is written out untouched. This is what `->>` +threading over slices in `plan.org` is replaced by for the collecting cases. + +### Reductions do not share the form, and that was the open question + +`(into xs 0 (map cost) (sum))` reads oddly because zero is not a collection, and the oddness is the tell rather than a +matter of taste. The whole reason the destination sits in the form is that **the destination is the allocation** — it +is what makes the explicit-allocator rule checkable by construction. A seed is not an allocation, so a form that +accepted one would be two forms sharing a spelling, and the destination would have stopped being honest about what it +is. So `into` collects, and a reducing macro of the same shape is a separate form the day something wants one. + +### The parts that took a decision + +- **The destination is a `Vec`**, because `push` is what fills it. A `Map` destination is refused by `push` itself, + which says *push takes a (Vec T)* and names the real problem. There is no second lowering and no reason to invent + one before something asks. +- **A source that is already a name is used as it is; anything else is bound to a gensym.** Both halves are needed and + neither is cosmetic. Binding is what a source that is a *call* needs — `(len s)` and `(at s i)` have to be the same + `s`, or the call is made once per element. Not binding a name is what everything else needs: a `(Vec T)` is + move-only, so `(let [s v] ...)` would hand the caller's `v` to a binding it cannot see and `v` would be dead after + an `into` that only read it; and a fixed array would be *copied* into the binding, once per `into`. `len` and `at` + borrow, so used directly the source is only read. +- **An owning temporary as the source leaks**, and this is the wart. `(into (make-a-vec) ...)` binds the result to a + name the caller cannot reach and therefore cannot free. The macro cannot know whether the type owns anything. A call + in that position should borrow — `into.flan`'s does — and the day `drop` exists this stops being a question. +- **One element name throughout**, shadowed by each `(map f)` stage: `(let [x (f x)] ...)`. A `let` binding's value is + checked before its name is bound, so the initialiser reads the outer `x` — that is the language's rule, not an + accident, and `bind`'s `x~2` debug suffix exists precisely so a debugger does not lie about which is which. + +### What the prelude's macro limits cost, exactly + +All four bit, and none blocked anything. + +- **A macro has no error facility**, so the three refusals are calls to names nothing defines: + `into-takes-a-source-a-destination-and-transforms`, `into-transform-is-map-or-filter` and + `…-of-one-function`. The report is *unknown function* at the call site with a *expanded from the macro into* note + under it, which is the right place and the wrong sentence. The bad transform is passed along as an argument so that + at least it is named. +- **A prelude macro may not call another macro**, so `into-wrap` is a plain `defn` and uses only special forms — + `loop`, `cond`, `when`, `let`, `if`. A `clamp` or an `unless` in there would have put it in the set `Macro.reduce` + drops. +- **Nested quasiquote is refused**, and it was not needed: each wrapper is a single-level quasiquote over a `body` + already built. +- **Macros are not importable**, which is why `into` is in the prelude rather than a library. + +`into-wrap` walks the transforms **in reverse**, because the chain is built from the inside out: the innermost form is +the `push`, and each transform wraps what the ones after it produced. That reverse walk with two accumulators is the +first thing in the prelude written as a `loop`/`recur`, which landed in the commit before this one. + ## `loop` and `recur`, and why `recur` is better than tail calls and not only cheaper There is no TCO anywhere in this compiler — nothing emits a tail call, and `plan.org` mentions them only as something diff --git a/NEXT.md b/NEXT.md index 4bb0aee..ede707f 100644 --- a/NEXT.md +++ b/NEXT.md @@ -789,7 +789,7 @@ Sources are community consensus rather than a specification; the `contains?` com Clojure*. Recorded because these are cheap to honour now and expensive to unpick once a standard library depends on them. -## Queued: `into`, fused transformation without transducers +## ~~Queued: `into`, fused transformation without transducers~~ — **landed** Decided in conversation. **Not transducers, and not Rust's iterators — a macro that fuses the chain at compile time.** @@ -826,6 +826,25 @@ destination is always honest about what it is. Drop Clojure's `:eduction` branch — that is the pass-around case, and the one part that would need runtime machinery. +**Done.** See *`into`, which fuses at compile time because it is a macro* in [`BUILT.md`](BUILT.md). It is a prelude +`defmacro` over a plain `defn` that walks the transforms in reverse, and all four of the macro limits bit without +blocking anything: the three refusals are names nothing defines, `into-wrap` uses only special forms so +`Macro.reduce` does not drop it, the quasiquotes are all single-level, and `into` lives in the prelude because a +macro is not importable. + +**The open question is settled: reductions do not share the form.** The reason the destination sits in `into` at all +is that the destination *is* the allocation, which is what makes `spec-memory.md`'s explicit-allocator rule true by +construction. A seed is not an allocation, so `(into xs 0 (map cost) (sum))` would be a second form wearing the same +spelling and the destination would stop being honest about what it is. A reducing macro of the same shape is a +separate form the day something wants one. + +Two things the design did not anticipate, both written up there. **A source that is already a name is used as it is** +rather than bound — a `(Vec T)` is move-only, so binding it would take the caller's ownership for a read, and a fixed +array would be copied once per `into`; a source that is anything else is still bound once, which is what a call +needs. And **an owning temporary as the source leaks**, because the macro binds it to a name the caller cannot reach +and cannot know whether the type owns anything. A call in that position should borrow. `drop` is what would close +this, and it does not exist. + ## ~~Queued: `loop`/`recur` (the return type is done)~~ — **landed** ~~**1. A `defn` must always state its return type, and unit is written `()`.**~~ **Done.** See *The return type is diff --git a/lib/prelude.ml b/lib/prelude.ml index fa56723..4184cb5 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -1519,6 +1519,128 @@ let source = {flan| `(unless-takes-a-test-and-a-body) `(if (not ~(at args 0)) (do ~@(form-rest args 1))))) +;; ── into: a fused transformation, and not a transducer ──────────────── +;; +;; (into xs (vec-new i32) (map double) (filter even?)) +;; +;; Source, destination, then any number of transforms. It reads as a sentence +;; — take this, put it there, doing these — and the variadic tail has to trail +;; anyway, which is the mechanical reason the transforms cannot sit in the +;; middle. +;; +;; **A macro, not transducers.** Transducers compose at run time: function +;; values, closures, an allocation, and a chain of indirect calls per element. +;; Rust has no transducers either — it has iterators, which fuse at compile +;; time through monomorphisation, and that needs generics. A macro reaches the +;; same place with neither. (map double) expands to (double x) written straight +;; into the loop body, so the function name is *syntax* and never a value: +;; there is no intermediate collection at any step, no closure, no generics and +;; nothing to inline. What it gives up is building a transformation at run time +;; and passing it around, which is transducers' actual selling point and is +;; close to useless in a game. +;; +;; **The destination is in the form on purpose.** Every collecting operation +;; here allocates from an explicit allocator, which is a frozen rule in +;; spec-memory.md. A ->> chain would hide where the result goes; naming the +;; destination means this macro knows its type, emits the right loop, and the +;; rule is honoured by construction. `(vec-new i32 a)` names an allocator here +;; as it does anywhere else, because the destination form is written out +;; untouched. +;; +;; **The destination is a Vec**, because push is what fills it. A Map +;; destination is refused by push, which says "push takes a (Vec T)" and names +;; the real problem; there is no second lowering for it and no reason to invent +;; one before something wants it. +;; +;; **Reductions do not share this form**, and that was the open question. (into +;; xs 0 (map cost) (sum)) reads oddly because zero is not a collection, and the +;; oddness is the tell: the whole reason the destination sits in the form is +;; that it is the allocation, and a seed is not one. Keeping `into` to +;; collections means the destination is always honest about what it is. A +;; reducing macro of the same shape is a separate form when something wants it. + +;; The chain, built from the inside out: the innermost form is the push, and +;; each transform wraps whatever the transforms after it produced. Walked in +;; reverse for that reason, which is what the loop's two names are. +;; +;; One element name throughout, shadowed by each (map f) stage. A let binding's +;; value is checked before the name is bound, so (let [x (f x)] ...) reads the +;; outer x and binds the inner one — that is the language's rule and not an +;; accident of the compiler; the debug-info suffix `x~2` exists precisely so a +;; debugger does not lie about which is which. The name is a gensym, so it +;; cannot collide with anything at the call site. +;; +;; A transform that is neither map nor filter expands to a call to a name +;; nothing defines, which is how a macro reports anything at all: it has no +;; error facility, so the report is "unknown name" at the call site — the right +;; place and the wrong sentence. The bad transform is passed along so that at +;; least it is named. +(defn into-wrap [ts [Form] dst Form x Form] Form + (loop [k (len ts) body `(push ~dst ~x)] + (if (= k 0) + body + (let [t (at ts (- k 1)) + items (form-items t)] + (if (!= (len items) 2) + `(into-transform-is-map-or-filter-of-one-function ~t) + (let [head (at items 0) + f (at items 1)] + (cond + (form-sym? head "map") (recur (- k 1) `(let [~x (~f ~x)] ~body)) + (form-sym? head "filter") (recur (- k 1) `(when (~f ~x) ~body)) + :else `(into-transform-is-map-or-filter ~t)))))))) + +;; The items of a list form, and the empty slice for anything else — a +;; non-list transform falls into the arity complaint above rather than needing +;; a case of its own. +(defn form-items [f Form] [Form] + (match f + (Form.List xs) xs + _ (form-nil))) + +(defn form-sym? [f Form name string] bool + (match f + (Form.Sym s) (bytes=? (bytes s) (bytes name)) + _ false)) + +(defn form-is-sym? [f Form] bool + (match f + (Form.Sym s) true + _ false)) + +(defn form-pair [a Form b Form] [Form] + (form-cons a (form-cons b (form-nil)))) + +;; A source that is already a name is used as it is, and a source that is +;; anything else is bound to one. Both halves matter. +;; +;; Binding it is what a source that is a *call* needs: (len s) and (at s i) +;; have to be the same s, and without the binding the call would be made twice +;; per element. +;; +;; Not binding a name is what everything else needs. A (Vec T) is move-only, so +;; (let [s v] ...) would hand v's ownership to the macro's own binding and the +;; caller would find v dead after an (into v ...) that only read it — and a +;; fixed array would be *copied* into the binding, once per into. Neither is +;; what was written. len and at borrow, so used directly the source is only +;; read. A source that is a call and produces a Vec is still consumed, which is +;; right: nobody else is holding it. +(defmacro into [args] + (if (< (len args) 2) + `(into-takes-a-source-a-destination-and-transforms) + (let [from (at args 0) + named? (form-is-sym? from) + src (if named? from (gensym)) + bind (if named? (form-nil) (form-pair src from)) + dst (gensym) + x (gensym) + i (gensym)] + `(let [~dst ~(at args 1) ~@bind] + (dotimes [~i (len ~src)] + (let [~x (at ~src ~i)] + ~(into-wrap (form-rest args 2) dst x))) + ~dst)))) + |flan} let file = "" diff --git a/test/programs/into.flan b/test/programs/into.flan new file mode 100644 index 0000000..3714892 --- /dev/null +++ b/test/programs/into.flan @@ -0,0 +1,81 @@ +;;;; into — a fused transformation, and not a transducer. +;;;; +;;;; What is asserted here is the thing a unit test cannot see: the loop that +;;;; comes out is one loop, and the values it produces are the ones the chain +;;;; describes in the order it was written. +;;;; +;;;; 1. The order of the transforms is the order of the stages. (filter even?) +;;;; before (map double) is not the same program as after it, and both are +;;;; here with different answers. +;;;; 2. There is no intermediate collection. `pulls` counts every call to the +;;;; transform functions: one pass over the source is one call per element +;;;; per stage it reaches, and a chain that built a Vec per stage would pull +;;;; a different number. +;;;; 3. A source that is already a name is only read. `src` is a (Vec i32), +;;;; which is move-only, and it is still alive and freeable afterwards. +;;;; 4. A source that is a call is evaluated once, not once per element. + +(defvar pulls i32 0) + +(defn double [x i32] i32 + (set pulls (+ pulls 1)) + (* x 2)) + +(defn even? [x i32] bool + (set pulls (+ pulls 1)) + (= (% x 2) 0)) + +(defvar builds i32 0) + +;; A source that is a call. It must be made once, however many elements come +;; out of it. It borrows rather than allocating, which is the shape a call in +;; this position wants: the macro binds the value to a name the caller cannot +;; see, so an owning temporary here would be a leak nobody can reach. +(defn source [xs [i32]] [i32] + (set builds (+ builds 1)) + (slice xs 0 (len xs))) + +(defn show [v [i32]] () + (dotimes [i (len v)] (print (at v i)) (print " ")) + (println "")) + +(defn main [] i32 + ;; No transforms: a copy into the destination named in the form. + (let [xs [7 8 9] + v (into xs (vec-new i32))] + (show (as-slice v)) ; 7 8 9 + (free v)) + + ;; map then filter. + (let [xs [1 2 3 4 5 6] + v (into xs (vec-new i32) (map double) (filter even?))] + (show (as-slice v)) ; 2 4 6 8 10 12 + (free v)) + + ;; filter then map, over the same source: a different answer, because the + ;; stages are in the order they were written. + (let [xs [1 2 3 4 5 6] + v (into xs (vec-new i32) (filter even?) (map double))] + (show (as-slice v)) ; 4 8 12 + (free v)) + + ;; One pass and no intermediate collection. The two chains above pulled + ;; 6 doubles + 6 evens, then 6 evens + 3 doubles: 21. + (print pulls) (println "") ; 21 + + ;; A name as the source is read, not moved: src is still alive here. + (let [src (into [3 1 2] (vec-new i32)) + v (into src (vec-new i32) (map double))] + (show (as-slice v)) ; 6 2 4 + (show (as-slice src)) ; 3 1 2 + (free v) + (free src)) + + ;; A source that is a call is bound once, so it is made once however many + ;; elements come out of it. + (let [xs [1 2 3 4] + v (into (source (slice xs 0 4)) (vec-new i32) (filter even?))] + (show (as-slice v)) ; 2 4 + (free v)) + (print builds) (println "") ; 1 + 0) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index 288b0f0..e00fd4f 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -132,6 +132,12 @@ let () = rebinds every name at once, and interleaved writes would print 1. *) outputs "loop and recur" "programs/recur.flan" "10\n2\n21\n8\n10000000\n64\n012\n0\n4\n6\n"; + (* into. The count of pulls is the assertion a unit test cannot make: one + pass, one call per element per stage it reaches, and no intermediate + collection anywhere. The two show lines either side of it are the same + source transformed in two orders, which have to differ. *) + outputs "into" "programs/into.flan" + "7 8 9 \n2 4 6 8 10 12 \n4 8 12 \n21\n6 2 4 \n3 1 2 \n2 4 \n1\n"; (* The prelude's slice algorithms. Every assertion here is over an input a wrong implementation fails: unsorted with duplicates, negatives and an odd length; a reverse-sorted slice; and a sort of a subslice whose diff --git a/test/test_flan.ml b/test/test_flan.ml index fe24390..0bb7a3a 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -989,6 +989,18 @@ let () = "(defn f [] () (loop :o [i 0] (recur i)))" ~needle:"loop takes no label"; rejects_check "a loop binding is a plain name" "(defn f [] () (loop [[a b] 0] (recur 0)))" ~needle:"destructuring pattern"; + (* into. The expansion is asserted in programs/into.flan, where the values + coming out are the test; what belongs here is the three things it refuses, + each through the one facility a macro has — a name nothing defines. *) + rejects_check "into needs a source and a destination" + "(defn f [] () (free (into [1 2 3])))" + ~needle:"into-takes-a-source-a-destination-and-transforms"; + rejects_check "a transform is map or filter" + "(defn f [] () (free (into [1 2 3] (vec-new i32) (take 2))))" + ~needle:"into-transform-is-map-or-filter"; + rejects_check "a transform names one function" + "(defn f [] () (free (into [1 2 3] (vec-new i32) (map))))" + ~needle:"into-transform-is-map-or-filter-of-one-function"; (* An import is resolved by [Load] before the checker runs, so one that reaches [Check] means a driver skipped that step. *) rejects_check "an unresolved import is a driver bug"