JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

into fuses at compile time because it is a macro, not a transducer

Browse Source 
(into xs (vec-new i32) (map double) (filter even?)). The function name is
syntax and never a value, so (map double) is (double x) written into the
loop body: no intermediate collection, no closure, no generics, nothing
to inline. Transducers would compose at run time and Rust's iterators
need monomorphisation; a macro needs neither. into.flan counts the pulls,
which is the assertion a unit test cannot make.

The destination is in the form because the destination is the allocation,
and that is what makes spec-memory.md's explicit-allocator rule true by
construction rather than by convention. Which also settles the open
question: reductions do not share the form. A seed is not an allocation,
so (into xs 0 (map cost) (sum)) would be a second form wearing the same
spelling, and the destination would stop being honest about what it is.

A source that is already a name is used as it is, not bound. A (Vec T) is
move-only, so binding it would take the caller's ownership for something
that only reads; a fixed array would be copied once per into. A source
that is anything else is still bound once, which is what a call needs.
The wart is that an owning temporary there leaks, because the binding has
a name the caller cannot reach — a call in that position should borrow,
and drop is what would close it.

All four of the prelude's macro limits bit and none blocked anything. The
three refusals are names nothing defines, which is the only error
facility a macro has. into-wrap is a defn using only special forms, so
Macro.reduce does not drop it, and it is the first thing in the prelude
written as a loop/recur.
This commit is contained in:
Joseph Ferano 2026-09-13 09:27:20 +07:00
parent fedaec3e18
commit ca14394e0f
6 changed files with 315 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

74
BUILT.md
View File

@ -3006,6 +3006,80 @@ and the function still has to answer. The loop-with-a-sentinel-flag shape that b
in the prelude. The two compiler-emitted retry loops in `check.ml` are that shape, and they are the one place it
cannot help: their sentinel is set inside a `restart-case` body, which is a barrier.
## `into`, which fuses at compile time because it is a macro
```
(into xs (vec-new i32) (map double) (filter even?))
```
Source, destination, then any number of transforms — the shape of the `into->` macro the author already uses in
Clojure. It reads as a sentence (take this, put it there, doing these) and the variadic tail has to trail anyway,
which is the mechanical reason the transforms cannot sit in the middle.
**A macro, and therefore not transducers and not iterators.** Transducers compose at *run time*: function values,
closures, an allocation, and a chain of indirect calls per element. Rust has no transducers either — it has iterators,
which fuse into one loop at compile time through monomorphisation, and that needs generics. A macro reaches the same
place with neither. `(map double)` expands to `(double x)` written straight into the loop body, so the function name
is **syntax and never a value**: no intermediate collection at any step, no closure, no generics, and nothing to
inline. `into.flan` counts every call to the transform functions, which is the assertion a unit test cannot make — a
chain that built a `Vec` per stage would pull a different number.
What it gives up is building a transformation at run time and passing it around. That is transducers' actual selling
point, it is the one part that would need run-time machinery, and it is close to useless in a game. Clojure's
`:eduction` branch is dropped for exactly that reason.
**Why the destination is in the form.** Every collecting operation here allocates from an *explicit* allocator, which
is a frozen rule in `spec-memory.md`. A `->>` chain hides where the result goes; naming the destination means the
macro knows its type, emits the right loop, and the rule is honoured by construction. `(vec-new i32 a)` names an
allocator here as it does anywhere else, because the destination form is written out untouched. This is what `->>`
threading over slices in `plan.org` is replaced by for the collecting cases.
### Reductions do not share the form, and that was the open question
`(into xs 0 (map cost) (sum))` reads oddly because zero is not a collection, and the oddness is the tell rather than a
matter of taste. The whole reason the destination sits in the form is that **the destination is the allocation** — it
is what makes the explicit-allocator rule checkable by construction. A seed is not an allocation, so a form that
accepted one would be two forms sharing a spelling, and the destination would have stopped being honest about what it
is. So `into` collects, and a reducing macro of the same shape is a separate form the day something wants one.
### The parts that took a decision
- **The destination is a `Vec`**, because `push` is what fills it. A `Map` destination is refused by `push` itself,
which says *push takes a (Vec T)* and names the real problem. There is no second lowering and no reason to invent
one before something asks.
- **A source that is already a name is used as it is; anything else is bound to a gensym.** Both halves are needed and
neither is cosmetic. Binding is what a source that is a *call* needs — `(len s)` and `(at s i)` have to be the same
`s`, or the call is made once per element. Not binding a name is what everything else needs: a `(Vec T)` is
move-only, so `(let [s v] ...)` would hand the caller's `v` to a binding it cannot see and `v` would be dead after
an `into` that only read it; and a fixed array would be *copied* into the binding, once per `into`. `len` and `at`
borrow, so used directly the source is only read.
- **An owning temporary as the source leaks**, and this is the wart. `(into (make-a-vec) ...)` binds the result to a
name the caller cannot reach and therefore cannot free. The macro cannot know whether the type owns anything. A call
in that position should borrow — `into.flan`'s does — and the day `drop` exists this stops being a question.
- **One element name throughout**, shadowed by each `(map f)` stage: `(let [x (f x)] ...)`. A `let` binding's value is
checked before its name is bound, so the initialiser reads the outer `x` — that is the language's rule, not an
accident, and `bind`'s `x~2` debug suffix exists precisely so a debugger does not lie about which is which.
### What the prelude's macro limits cost, exactly
All four bit, and none blocked anything.
- **A macro has no error facility**, so the three refusals are calls to names nothing defines:
`into-takes-a-source-a-destination-and-transforms`, `into-transform-is-map-or-filter` and
`…-of-one-function`. The report is *unknown function* at the call site with a *expanded from the macro into* note
under it, which is the right place and the wrong sentence. The bad transform is passed along as an argument so that
at least it is named.
- **A prelude macro may not call another macro**, so `into-wrap` is a plain `defn` and uses only special forms —
`loop`, `cond`, `when`, `let`, `if`. A `clamp` or an `unless` in there would have put it in the set `Macro.reduce`
drops.
- **Nested quasiquote is refused**, and it was not needed: each wrapper is a single-level quasiquote over a `body`
already built.
- **Macros are not importable**, which is why `into` is in the prelude rather than a library.
`into-wrap` walks the transforms **in reverse**, because the chain is built from the inside out: the innermost form is
the `push`, and each transform wraps what the ones after it produced. That reverse walk with two accumulators is the
first thing in the prelude written as a `loop`/`recur`, which landed in the commit before this one.
## `loop` and `recur`, and why `recur` is better than tail calls and not only cheaper
There is no TCO anywhere in this compiler — nothing emits a tail call, and `plan.org` mentions them only as something

21
NEXT.md
View File

@ -789,7 +789,7 @@ Sources are community consensus rather than a specification; the `contains?` com
Clojure*. Recorded because these are cheap to honour now and expensive to unpick once a standard library depends on
them.
## Queued: `into`, fused transformation without transducers
## ~~Queued: `into`, fused transformation without transducers~~ — **landed**
Decided in conversation. **Not transducers, and not Rust's iterators — a macro that fuses the chain at compile time.**
@ -826,6 +826,25 @@ destination is always honest about what it is.
Drop Clojure's `:eduction` branch — that is the pass-around case, and the one part that would need runtime machinery.
**Done.** See *`into`, which fuses at compile time because it is a macro* in [`BUILT.md`](BUILT.md). It is a prelude
`defmacro` over a plain `defn` that walks the transforms in reverse, and all four of the macro limits bit without
blocking anything: the three refusals are names nothing defines, `into-wrap` uses only special forms so
`Macro.reduce` does not drop it, the quasiquotes are all single-level, and `into` lives in the prelude because a
macro is not importable.
**The open question is settled: reductions do not share the form.** The reason the destination sits in `into` at all
is that the destination *is* the allocation, which is what makes `spec-memory.md`'s explicit-allocator rule true by
construction. A seed is not an allocation, so `(into xs 0 (map cost) (sum))` would be a second form wearing the same
spelling and the destination would stop being honest about what it is. A reducing macro of the same shape is a
separate form the day something wants one.
Two things the design did not anticipate, both written up there. **A source that is already a name is used as it is**
rather than bound — a `(Vec T)` is move-only, so binding it would take the caller's ownership for a read, and a fixed
array would be copied once per `into`; a source that is anything else is still bound once, which is what a call
needs. And **an owning temporary as the source leaks**, because the macro binds it to a name the caller cannot reach
and cannot know whether the type owns anything. A call in that position should borrow. `drop` is what would close
this, and it does not exist.
## ~~Queued: `loop`/`recur` (the return type is done)~~ — **landed**
~~**1. A `defn` must always state its return type, and unit is written `()`.**~~ **Done.** See *The return type is

122
lib/prelude.ml
View File

@ -1519,6 +1519,128 @@ let source = {flan|
`(unless-takes-a-test-and-a-body)
`(if (not ~(at args 0)) (do ~@(form-rest args 1)))))
;; ── into: a fused transformation, and not a transducer ────────────────
;;
;; (into xs (vec-new i32) (map double) (filter even?))
;;
;; Source, destination, then any number of transforms. It reads as a sentence
;; — take this, put it there, doing these — and the variadic tail has to trail
;; anyway, which is the mechanical reason the transforms cannot sit in the
;; middle.
;;
;; **A macro, not transducers.** Transducers compose at run time: function
;; values, closures, an allocation, and a chain of indirect calls per element.
;; Rust has no transducers either — it has iterators, which fuse at compile
;; time through monomorphisation, and that needs generics. A macro reaches the
;; same place with neither. (map double) expands to (double x) written straight
;; into the loop body, so the function name is *syntax* and never a value:
;; there is no intermediate collection at any step, no closure, no generics and
;; nothing to inline. What it gives up is building a transformation at run time
;; and passing it around, which is transducers' actual selling point and is
;; close to useless in a game.
;;
;; **The destination is in the form on purpose.** Every collecting operation
;; here allocates from an explicit allocator, which is a frozen rule in
;; spec-memory.md. A ->> chain would hide where the result goes; naming the
;; destination means this macro knows its type, emits the right loop, and the
;; rule is honoured by construction. `(vec-new i32 a)` names an allocator here
;; as it does anywhere else, because the destination form is written out
;; untouched.
;;
;; **The destination is a Vec**, because push is what fills it. A Map
;; destination is refused by push, which says "push takes a (Vec T)" and names
;; the real problem; there is no second lowering for it and no reason to invent
;; one before something wants it.
;;
;; **Reductions do not share this form**, and that was the open question. (into
;; xs 0 (map cost) (sum)) reads oddly because zero is not a collection, and the
;; oddness is the tell: the whole reason the destination sits in the form is
;; that it is the allocation, and a seed is not one. Keeping `into` to
;; collections means the destination is always honest about what it is. A
;; reducing macro of the same shape is a separate form when something wants it.
;; The chain, built from the inside out: the innermost form is the push, and
;; each transform wraps whatever the transforms after it produced. Walked in
;; reverse for that reason, which is what the loop's two names are.
;;
;; One element name throughout, shadowed by each (map f) stage. A let binding's
;; value is checked before the name is bound, so (let [x (f x)] ...) reads the
;; outer x and binds the inner one — that is the language's rule and not an
;; accident of the compiler; the debug-info suffix `x~2` exists precisely so a
;; debugger does not lie about which is which. The name is a gensym, so it
;; cannot collide with anything at the call site.
;;
;; A transform that is neither map nor filter expands to a call to a name
;; nothing defines, which is how a macro reports anything at all: it has no
;; error facility, so the report is "unknown name" at the call site — the right
;; place and the wrong sentence. The bad transform is passed along so that at
;; least it is named.
(defn into-wrap [ts [Form] dst Form x Form] Form
(loop [k (len ts) body `(push ~dst ~x)]
(if (= k 0)
body
(let [t (at ts (- k 1))
items (form-items t)]
(if (!= (len items) 2)
`(into-transform-is-map-or-filter-of-one-function ~t)
(let [head (at items 0)
f (at items 1)]
(cond
(form-sym? head "map") (recur (- k 1) `(let [~x (~f ~x)] ~body))
(form-sym? head "filter") (recur (- k 1) `(when (~f ~x) ~body))
:else `(into-transform-is-map-or-filter ~t))))))))
;; The items of a list form, and the empty slice for anything else — a
;; non-list transform falls into the arity complaint above rather than needing
;; a case of its own.
(defn form-items [f Form] [Form]
(match f
(Form.List xs) xs
_ (form-nil)))
(defn form-sym? [f Form name string] bool
(match f
(Form.Sym s) (bytes=? (bytes s) (bytes name))
_ false))
(defn form-is-sym? [f Form] bool
(match f
(Form.Sym s) true
_ false))
(defn form-pair [a Form b Form] [Form]
(form-cons a (form-cons b (form-nil))))
;; A source that is already a name is used as it is, and a source that is
;; anything else is bound to one. Both halves matter.
;;
;; Binding it is what a source that is a *call* needs: (len s) and (at s i)
;; have to be the same s, and without the binding the call would be made twice
;; per element.
;;
;; Not binding a name is what everything else needs. A (Vec T) is move-only, so
;; (let [s v] ...) would hand v's ownership to the macro's own binding and the
;; caller would find v dead after an (into v ...) that only read it — and a
;; fixed array would be *copied* into the binding, once per into. Neither is
;; what was written. len and at borrow, so used directly the source is only
;; read. A source that is a call and produces a Vec is still consumed, which is
;; right: nobody else is holding it.
(defmacro into [args]
(if (< (len args) 2)
`(into-takes-a-source-a-destination-and-transforms)
(let [from (at args 0)
named? (form-is-sym? from)
src (if named? from (gensym))
bind (if named? (form-nil) (form-pair src from))
dst (gensym)
x (gensym)
i (gensym)]
`(let [~dst ~(at args 1) ~@bind]
(dotimes [~i (len ~src)]
(let [~x (at ~src ~i)]
~(into-wrap (form-rest args 2) dst x)))
~dst))))
|flan}
let file = "<prelude>"

81
test/programs/into.flan Normal file
View File

@ -0,0 +1,81 @@
;;;; into — a fused transformation, and not a transducer.
;;;;
;;;; What is asserted here is the thing a unit test cannot see: the loop that
;;;; comes out is one loop, and the values it produces are the ones the chain
;;;; describes in the order it was written.
;;;;
;;;; 1. The order of the transforms is the order of the stages. (filter even?)
;;;; before (map double) is not the same program as after it, and both are
;;;; here with different answers.
;;;; 2. There is no intermediate collection. `pulls` counts every call to the
;;;; transform functions: one pass over the source is one call per element
;;;; per stage it reaches, and a chain that built a Vec per stage would pull
;;;; a different number.
;;;; 3. A source that is already a name is only read. `src` is a (Vec i32),
;;;; which is move-only, and it is still alive and freeable afterwards.
;;;; 4. A source that is a call is evaluated once, not once per element.
(defvar pulls i32 0)
(defn double [x i32] i32
(set pulls (+ pulls 1))
(* x 2))
(defn even? [x i32] bool
(set pulls (+ pulls 1))
(= (% x 2) 0))
(defvar builds i32 0)
;; A source that is a call. It must be made once, however many elements come
;; out of it. It borrows rather than allocating, which is the shape a call in
;; this position wants: the macro binds the value to a name the caller cannot
;; see, so an owning temporary here would be a leak nobody can reach.
(defn source [xs [i32]] [i32]
(set builds (+ builds 1))
(slice xs 0 (len xs)))
(defn show [v [i32]] ()
(dotimes [i (len v)] (print (at v i)) (print " "))
(println ""))
(defn main [] i32
;; No transforms: a copy into the destination named in the form.
(let [xs [7 8 9]
v (into xs (vec-new i32))]
(show (as-slice v)) ; 7 8 9
(free v))
;; map then filter.
(let [xs [1 2 3 4 5 6]
v (into xs (vec-new i32) (map double) (filter even?))]
(show (as-slice v)) ; 2 4 6 8 10 12
(free v))
;; filter then map, over the same source: a different answer, because the
;; stages are in the order they were written.
(let [xs [1 2 3 4 5 6]
v (into xs (vec-new i32) (filter even?) (map double))]
(show (as-slice v)) ; 4 8 12
(free v))
;; One pass and no intermediate collection. The two chains above pulled
;; 6 doubles + 6 evens, then 6 evens + 3 doubles: 21.
(print pulls) (println "") ; 21
;; A name as the source is read, not moved: src is still alive here.
(let [src (into [3 1 2] (vec-new i32))
v (into src (vec-new i32) (map double))]
(show (as-slice v)) ; 6 2 4
(show (as-slice src)) ; 3 1 2
(free v)
(free src))
;; A source that is a call is bound once, so it is made once however many
;; elements come out of it.
(let [xs [1 2 3 4]
v (into (source (slice xs 0 4)) (vec-new i32) (filter even?))]
(show (as-slice v)) ; 2 4
(free v))
(print builds) (println "") ; 1
0)

6
test/test_acceptance.ml
View File

@ -132,6 +132,12 @@ let () =
rebinds every name at once, and interleaved writes would print 1. *)
outputs "loop and recur" "programs/recur.flan"
"10\n2\n21\n8\n10000000\n64\n012\n0\n4\n6\n";
(* into. The count of pulls is the assertion a unit test cannot make: one
pass, one call per element per stage it reaches, and no intermediate
collection anywhere. The two show lines either side of it are the same
source transformed in two orders, which have to differ. *)
outputs "into" "programs/into.flan"
"7 8 9 \n2 4 6 8 10 12 \n4 8 12 \n21\n6 2 4 \n3 1 2 \n2 4 \n1\n";
(* The prelude's slice algorithms. Every assertion here is over an input a
wrong implementation fails: unsorted with duplicates, negatives and an
odd length; a reverse-sorted slice; and a sort of a subslice whose

12
test/test_flan.ml
View File

@ -989,6 +989,18 @@ let () =
"(defn f [] () (loop :o [i 0] (recur i)))" ~needle:"loop takes no label";
rejects_check "a loop binding is a plain name"
"(defn f [] () (loop [[a b] 0] (recur 0)))" ~needle:"destructuring pattern";
(* into. The expansion is asserted in programs/into.flan, where the values
coming out are the test; what belongs here is the three things it refuses,
each through the one facility a macro has — a name nothing defines. *)
rejects_check "into needs a source and a destination"
"(defn f [] () (free (into [1 2 3])))"
~needle:"into-takes-a-source-a-destination-and-transforms";
rejects_check "a transform is map or filter"
"(defn f [] () (free (into [1 2 3] (vec-new i32) (take 2))))"
~needle:"into-transform-is-map-or-filter";
rejects_check "a transform names one function"
"(defn f [] () (free (into [1 2 3] (vec-new i32) (map))))"
~needle:"into-transform-is-map-or-filter-of-one-function";
(* An import is resolved by [Load] before the checker runs, so one that
reaches [Check] means a driver skipped that step. *)
rejects_check "an unresolved import is a driver bug"