diff --git a/FIX.org b/FIX.org index 58259fdc..9c87d8e2 100644 --- a/FIX.org +++ b/FIX.org @@ -6212,5 +6212,96 @@ already call. No new path, no polling. - A data constructor is [Type.Case] and is one symbol. The type half is drawn and the case half is not: the daemon answers with the type's name and knows nothing of its cases. -- [CFn] does not exist anywhere in the tree. [Fn] does — [(Fn [T ...] R)], - parse.ml:99 — and is in the type rule. Nothing was added for the other. +- [CFn] did not exist anywhere in the tree when this pass ran. [Fn] did — + [(Fn [T ...] R)], parse.ml:99 — and is in the type rule. Nothing was added + for the other. The closure lane landed [CFn] the same day, so the type rule + wants it adding. +* Closures, 2026-09-21 — two rulings, and what this lane built +Two rulings, both in the author's words. + +The first, on what to build: + + "do both, capture by value and handle escaping closures, allocated on the + GC side" + +This lane is the first half: capture by value into a stack environment, +spec-memory.md's case 2, non-escaping only. The second half — an environment +the collector allocates, and with it the escaping closure — is a separate lane, +and every refusal this one prints names it. + +The second, on the calling convention, after the first design put an +environment parameter on every Flan signature: + + "while it's dyn first, static side should never have to pay the price for + the existence of the dyn side... if you fully opt out, for instance, using + --no-gc flag, then we should be operating under Odin/C semantics and never + paying any runtime costs" + +So there are two function types, Rust's and Swift's shape: + + (Fn [T ...] R) captures; {code, env}; the common case, short name + (CFn [T ...] R) the bare address; one word; cannot capture + +** The name, ruled 2026-09-21 +CFn, because the C carries information rather than being decoration: a value +with no environment is the only kind that could ever cross to C, and under the +--no-conditions direction below it becomes literally a C function pointer. The +name points at what the type is and at where it is going. + +Rejected: Closure, too long. Proc, because "procedure" is a word we disagree +with Odin about. Fun and Func, because beside Fn they differ only in length, +so nothing tells a reader which one captures. Fnptr, as ugly. + +The thing to be careful about, and it is written into the crossable refusal so +a reader meets it where they would otherwise be misled: CFn is not the type +for C interop *today*. A declare cannot take a function type at all, because a +Flan signature ends with the transfer channel. + +** Nobody needs CFn, and one of the four reasons is the common one +An Fn accepts everything a CFn does, so the narrow one is always reached for +on purpose: + +1. handing a function to C — later, per the item below; +2. a table of bare addresses; +3. forbidding capture at a boundary, where the type is the statement; +4. performance, which is likeliest in practice. A *named* function passed to + an Fn parameter goes through the widening thunk and pays an indirect hop + per call; a CFn parameter is a direct call. (map-in-place s double) is the + example. The prelude's four stay Fn on purpose — a capturing predicate is + what people want — so this is the cost someone would opt out of by writing + their own signature, not one the prelude should have avoided. + +** What the escaping lane inherits, and what it changes +Unchanged by it: a (Fn ...) is two words; the environment is the last +parameter; it is declared by exactly the bodies an Fn value can reach — a +lifted literal in an Fn position, every handler clause, and the widening +thunks. An ordinary defn declares none and is byte-for-byte what it was. + +Changed by it: where the environment points. It is a Make into a frame slot +today and becomes a collector allocation; Check.escape_check goes away with it, +and with it the refusals on returning, storing, pointing at and pushing a +capturing value. + +One thing for the collector to know now rather than discover: a widening +thunk's environment holds a *code pointer*, not a frame address and not a GC +object. When env becomes collector-allocated the marker needs a way not to +follow a thunk's. + +Also still refused and belonging to the second half: capturing a dyn. The +struct dyn field is no longer refused — per-type descriptors landed — but a +synthesised environment has no descriptor, so the refusal stands until it does. + +** Recorded, not built: CFn and C's convention +A CFn is one word and is the right shape for a C callback, and it is still +not one: a Flan function's signature ends with the transfer channel and a C +caller knows nothing about one. Under a future --no-conditions flag a CFn +signature could drop the channel and reach C's exact convention, which is the +direction the author is interested in. The refusal in [crossable] names it. + +** Also worth an item: CFn in a struct or a fixed array +no_zeroed_fn refuses a function value in any position ZII would conjure one, +and it refuses a CFn for the same reason it refuses an Fn: a zeroed function +value is a null pointer, which is the one zero that is not a value the type can +have. But "a table of function pointers" is exactly what CFn is for, and that +objection is about ZII rather than about capture — an (Option (CFn ...)) field +is already legal and is the shape that works. Its own item. diff --git a/docs/BUILT.md b/docs/BUILT.md index 56c23650..a4cd75c7 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -1861,15 +1861,17 @@ stack does *not* observe a redefinition of its own clause; the next entry to the Which gives the two refusals, both by the house rule rather than by accident: -- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a -reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in -scope, which is what the accumulation case needs. +- **A handler cannot see the establishing function's locals.** *Superseded: it can, by value — see "Capture by value" +below. What is still refused is a `set` into one, because a captured name is a copy.* The original note read: that is a +closure with an explicit environment, so a reference to one is refused *for that reason* rather than reported as an +unknown name; globals and the condition are in scope, which is what the accumulation case needs. -What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not -outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by -value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete -use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the -biggest usability limit in conditions is not behind the thing that was just deferred. +The reading that turned out to be right: a handler frame does not outlive the function that established it, so this is +spec-memory.md's **case 2** — a non-escaping `fn` capturing by value into a stack environment — and *not* the escaping +closure that plan.org's open decision #5 defers until a concrete use case. Case 2 was settled, was built, and brought +the handler along with it for one struct field and one argument. #5 says in as many words that without it "conditions +are not worth building", and the biggest usability limit in conditions was indeed never behind the thing that was +deferred. - **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block. @@ -2702,9 +2704,11 @@ spec together; neither is a cleanup, and until one is taken the word is carried `(Map K V)` is built — see below. What is left: `drop` and with it the transitive move-only rule, recursive teardown, and the refusal to construct a drop-carrying container against an allocator without `can-free`; `(Result T E)` and `try`; generics; the macro expander. -And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy: -capture does not exist at all, and the spec's captured-`Vec`-by-pointer rule has never had to exist because every -capturable type today is a value type. It is its own item and should be planned as one. +And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy. +Capture exists now (see "Capture by value" below) and this is still not it: capture is by *value*, so the `fn` would +push into its own copy of the header and leave the enclosing one at the length it had. The spec's +captured-`Vec`-by-pointer rule is exactly the thing that has still never had to exist. It is its own item and should +be planned as one. ## `(Map K V)`, which is Odin's map @@ -3817,9 +3821,10 @@ cannot share a name** — which is exactly what makes the bare name safe to read `double` it could have meant instead, so the sharp quote would be punctuation answering a question the language does not ask. -A `Types.Fn` is one pointer. There is no environment beside it, so the type resolves to `ptr` and lays out as eight -bytes, and a call through one is byte-for-byte the call a name would have produced — a Flan function's emitted -signature is its parameters followed by the transfer channel whether it was reached by name or by pointer. That is +A `Types.Fn` was one pointer when this lane landed. It is two words since capture, and the one-word version has a +name of its own now — `(CFn [T ...] R)`; see "Capture by value" below. A call through either is the call a name +would have produced, with the environment appended for a `Fn`: a Flan function's emitted signature is its +parameters, then the transfer channel, and then the environment on the bodies that can be reached that way. That is why a handler established across a `fold` still catches a signal raised by the function the fold was handed: `programs/fn-values.flan` does exactly that, and it is the case that would fail if an indirect call skipped the guard. @@ -3840,11 +3845,8 @@ implemented. **Refused, each with its own reason and its own program:** -- **Capture does not exist** (`fn-capture.flan`). An `fn` is lifted into a function of its own and handed nothing but - its parameters; a reference to a local of the enclosing function is refused by name. This is the same refusal a - handler clause has always carried, and the two now share one message with the construct's name in it. - `spec-memory.md`'s capture cases, and **escaping closures with them, stay deferred** — deliberately, and this is - what keeps a function value a bare code address that cannot outlive anything. +- **Capture does not exist.** *Superseded — see "Capture by value" below. It exists, the program that was this + refusal's witness now runs, and what is refused in its place is the **escape**.* - **An `fn` with nothing to say what it takes** (`fn-no-type.flan`), above. - **A position that would zero one** (`fn-in-struct.flan`): a struct field, a global, a fixed array's element, `(zeroed)`. ZII fills an omitted field with all-bytes-zero, and **a zeroed function value is a null pointer, which @@ -3855,9 +3857,314 @@ implemented. past its length and `flan_map_alloc` zeroes only the hash run, so neither conjures an element nobody pushed or put. A function value as a map *key* is refused already, by `Types.keyable` — hashing an address is a different operation from hashing what it points at. -- **A foreign function's address** (`fn-extern.flan`). A Flan function's signature ends with the transfer channel and - a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the raw symbol knows - nothing about. Wrap it in a `defn` and pass that. +- **A foreign function's address** (`fn-extern.flan`). A Flan function's signature ends with the environment and the + transfer channel and a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the + raw symbol knows nothing about. Wrap it in a `defn` and pass that. Capture widened this gap rather than closing + it: a Flan function value is now two words and a C symbol is one. + +## Capture by value, and what "non-escaping" had to mean + +`spec-memory.md`'s **case 2**, which the section above listed as the headline refusal and which is now the headline +feature. This compiles: + +``` +(let [bonus 10] + (apply2 (fn [x] (+ x bonus)) 5)) +``` + +`bonus` is **copied** into an environment on the enclosing function's frame at the instant the `fn` value is made, +and the lifted body reads the copy. Not a reference: `fn-capture.flan` changes the local through a pointer *after* +the value exists and *before* it is called, and the `fn` still answers with the old one. That test is the whole +claim, and it is the one no evaluation order can fake. + +### Two function types, because the static side does not pay for the dynamic side + +``` +(Fn [i32] i32) ; captures; {code, env}; the common case +(CFn [i32] i32) ; the bare code address; one word; cannot capture +``` + +The forcing constraint first. A callee that takes a `(Fn [i32] i32)` and calls it knows nothing about where the +value came from — `fold` is handed a value and calls it — so **the environment has to travel with the value** or +there is nowhere to put it. A `Fn` is therefore `{code, env}`: sixteen bytes, classified as an aggregate in both +backends exactly as a slice is. + +The first design put an environment parameter on **every** Flan signature, uniform for the reason the transfer +channel is uniform. The author ruled against it, and the ruling is the principle rather than the case: *"while it's +dyn first, static side should never have to pay the price for the existence of the dyn side… if you fully opt out, +for instance, using `--no-gc`, then we should be operating under Odin/C semantics and never paying any runtime +costs."* A uniform environment taxes every function in every program for a feature most of them never use. + +So there are two types, Rust's and Swift's shape. `Fn` keeps the short name because it is what almost every +higher-order signature wants; `CFn` is the narrow one. + +**The `C` is information and not decoration**, which is what settled the name. A value with no environment is the +only kind that could ever cross to C, and under the `--no-conditions` direction FIX.org records — where a signature +that cannot transfer drops the transfer channel too — one becomes literally a C function pointer. The name points at +what the type *is* and at where it is going. `Closure` was rejected as too long; `Proc` because "procedure" is a +word this language disagrees with Odin about; `Fun` and `Func` because beside `Fn` they differ only in length, so +nothing tells a reader which one captures; `Fnptr` as ugly. + +**It is not a capability today, and the diagnostic says so.** A `declare` cannot take a function type at all — a +Flan signature ends with the transfer channel and a C caller knows nothing about one — so anyone reaching for `CFn` +straight after writing a `declare-c` is reaching too early. `crossable`'s refusal names that in as many words: *"the +C in CFn is about having no environment, which is what a C function pointer would need, and not about crossing +today."* + +**And nobody ever needs it.** `Fn` accepts everything a `CFn` does, so the narrow one is reached for on purpose, for +one of four reasons: + +1. handing a function to C — later, as above; +2. a table of bare addresses; +3. forbidding capture at a boundary, where the type is the statement; +4. **performance, which is likeliest in practice.** A *named* function passed to an `Fn` parameter goes through the + widening thunk and pays an indirect hop per call; a `CFn` parameter is a direct call. `(map-in-place s double)` + is the example — and the prelude's four stay `Fn`, because a capturing predicate is exactly what people want. + +**An ordinary `defn` keeps its exact signature.** Verified rather than asserted: the LLVM for `calc-me.flan` and +fourteen corpus programs was diffed against the same compiler without this lane. Exactly three kinds of difference +appear, and no fourth: + +- two type declarations in the preamble — `%fnv`, and `%handler`'s new `env` field; +- the prelude's `sort-by-slice-u8`, whose *parameter* is now `%fnv` because it is declared `(Fn [$t $t] bool)` and + pays two words for the value it asked for; and the lifted literal it is handed, which gains a trailing `ptr %env` + because an `Fn` value can reach it; +- in a program with a `handler-bind`, its clauses gain the same trailing `ptr %env` — every clause declares one, + see below. + +**No ordinary `defn` gained a parameter, in any program.** `flan_rt.c`'s hash and equality typedefs are untouched; +`main`, the macro thunk, the startup call and the reload thunk emit the calls they always emitted. + +**`handler-bind` is not free, and rounding it to zero would be wrong.** A program that establishes a handler and +captures nothing still pays: `%handler` went from 24 bytes to 32, every push writes an unconditional null into the +new field, every clause gains `ptr %env` plus an alloca and a store at entry, and `flan_signal` passes one more +argument per dispatch. Measured on `loops.flan`: +20 changed lines of x86. It is small and it is real, and every +program with conditions in it pays it — the alternative was a second clause convention beside the capturing one, +which `flan_signal` could not choose between because it calls through one C function-pointer type and cannot know +which clause matched. + +### The environment is the last argument, on exactly the bodies an `Fn` can reach + +The environment is the last parameter, after the transfer channel, and it is declared by **exactly the bodies a +`(Fn ...)` value can reach**: a lifted `fn` literal written into an `Fn` position, capturing or not; every handler +clause, because `flan_signal` passes one to whichever clause matched and cannot know which of them captured; and the +widening thunks below, which exist to read it. Nothing else declares it, which is where an ordinary `defn` keeps +costing nothing. + +So **every indirect call is exactly typed** and nowhere does a caller pass an argument the callee did not declare. + +That was not the first attempt. The first put the environment last and let a body that never asked for one simply +ignore the register it arrived in — legal under SysV, where argument N is classified from arguments 1..N alone, and +exactly **Swift's thin-vs-thick convention**, where a thin function converts to a thick one by pairing with a null +context the thin body ignores. It works on x86-64 and it is dead on **wasm32**, where `call_indirect` compares the +signature at the call site: a spare argument is a trap, not a register nobody reads. `test_web`'s two cases and the +headless `sand` build failed with *"null function or function signature mismatch"*, and the honest reading is that +being exactly typed is checkable by a verifier rather than argued from a calling convention, which is the better +property to have wanted. + +**So there is one adapter, and it is per *signature* rather than per function.** `CFn` → `Fn` is `Tast.Thicken`, +and the pair it builds is `{thunk, the address}`: the thunk's code, with the bare address stored where an +environment would be. The thunk — `thick/`, minted and memoised by the checker the way +`struct_key_pair` mints a map's hash and equality pair — declares the environment, reads the address back out of it, +and calls through it. One small function per distinct shape a program widens, not per function it widens, and it +handles the dynamic case (a `CFn`-typed local or parameter widened at a call) with the same mechanism as the +static one. + +**What it costs, plainly.** A *name* handed to an `Fn`-typed parameter now pays one indirect hop per call: +`(map-in-place s double)` goes through the thunk per element where it used to reach `double` directly. A literal +pays nothing — capturing or not, it is compiled to take an environment and needs no thunk. The escape hatch is +writing `CFn` in the signature, which is what the type is for; the prelude's `map`, `filter`, `reduce` and +`sort-by` correctly stay `Fn`, because a capturing comparator is exactly what people want, so the common +named-function case does pay. That is the one real price of two types, and it buys every function in every program +not paying for an environment it never has. + +A redefinition module carries its own copy of every thunk, hidden. A module that widens a name refers to one, and +the host has no cell for it to be reached through — the same shape of bug as the `Fnval` cell below, found the same +way and closed before it shipped. `flan reload` with a body that widens a name builds on both backends. + +**The memo is keyed on the types and the symbol is a counter**, which review found the hard way. Keyed on a *name* +derived from `mangle_ty` it was not a memo but a collision: that function flattens a whole signature into one +hyphen-joined string, so `(CFn [(Ptr i32)] i32)` and `(CFn [ptr i32] i32)` — the second over a struct someone +called `ptr` — flatten alike, and the second widening silently reused the first's thunk at the wrong arity. Both +backends compiled it without a word and neither ran it. `fn-thunk-share.flan` is that exact pair, and it prints 5 +and 17. (`mangle_ty`'s ambiguity is older than this lane and is still there for the generic instantiation names it +was written for; at one type's granularity it is hard to reach, at a whole signature's it is a line of Flan away.) + +**A generic that binds its variable *through* a function type needed both halves.** `(defn apply2 [f (Fn [$t] $t) +x $t] …)` called as `(apply2 bump 1)` is the shape a bare name has to reach now that a `defn`'s address carries +`CFn`, and it broke in two places that fail apart: `bind_ty` had no arm admitting a `CFn` argument at an `Fn` +pattern (so the instantiation was refused outright — a regression against a program that compiled before this +lane), and `generic_call`'s catch-up pass did not widen it (so the call site handed one word to an instance +declaring two). A parameter that still mentions a variable is checked with *no expectation*, by design — there is +nothing to expect until the argument has spoken — so `expect`, where the widening otherwise lives, never sees the +pair. The same gap hid the new half: a `CFn` argument at a `(CFn [$t] $t)` parameter had no arm either and fell +through to plain equality. `fn-generic.flan` covers both. + +The corpus missed all of it, and the reason is worth keeping: the prelude's higher-order functions bind `$t` from +an *earlier* argument, so `subst_ty` has already made the parameter concrete by the time `bind_ty` sees it. +`(map-in-place s double)` and `higher-order.flan` really were still working. + +The reverse coercion does not exist — there is nowhere for an environment to go — and is refused by the ordinary +type message, which names both spellings (`fn-cfn-narrow.flan`). A capturing literal written into a `CFn` position +is refused by name, with what it captured and the fix in the sentence (`fn-cfn-captures.flan`). + +One trap worth naming, because it is where the map would have broken: `FnAddr` is asked for by unrelated readers. A +`Fn`-typed one is two words; a `CFn`-typed or `Alloc`-typed one is a bare address — the second is what the map's +hash and equality pair and a handler frame's clause are, fields of structs the runtime declares, and they must stay +one word. **The node's type is what discriminates**, in both backends. + +A handler clause is the one place the runtime does the passing, so `flan_handler` grew an `env` field and +`flan_signal` calls `h->fn(condition, xfer, h->env)` — the same trailing position, and a clause that captured +nothing declares nothing and is unaffected. + +### The environment is a struct the checker synthesised + +One field per captured name, in first-reference order, registered in the same table a `defstruct` goes in — so both +backends lay it out with the calculator they already have and neither learns a new shape. Its name is the lifted +function's (`env/fn/OWNER/N`), which is unique and stable for the reason that name is. + +Two ends, and the copy is at the near one. In the *enclosing* frame, a slot holding the struct, filled with a `Make` +of the outer locals: that store is the copy, and it happens where the value is made. In the *lifted* frame, a slot +holding the pointer and a `Let` around the whole body reading each field back into the named slot the body was +checked against — once, at entry, so nothing downstream has to know an environment exists. + +Both of the compiler's new slots are **nameless**, which is how the break loop is told to hide them. That is a +deliberate call, not an omission: what a reader wants at a stop is the captured *copies*, and those are named slots +holding the values under the names the source gave them. An `env` pointer and a struct of bytes would be two rows of +noise above them. + +A redefinition that changes which locals an `fn` names changes an environment's layout, and `Session`'s layout guard +**exempts** these. The guard is about values the running program is holding; an environment can be in exactly one +place, a slot of the frame the literal was written in, written by the same module that reads it on every entry. A +restart for editing a capture list would take the dev loop away from the feature it was built for. + +### Escape, which is what makes "case 2" a bounded claim + +A value carrying an environment may be **called, passed down, and held in a `let`**. It may not be **returned, +stored, pointed at, or pushed into a container**. The check runs over the typed IR of every function the program +ends up with — including the lifted ones, so an `fn` inside an `fn` needs no special case — and classifies +function-typed values as *suspect* or clean: + +- suspect: a capturing literal (`Tast.Closure`, the only node that makes one); a **parameter** of type `Fn`, in + every function; an `Fn` read back out of a struct, a case or a pointer; a local bound to any of those, + transitively; a branch or a valued form whose value is one. +- clean: the address of a name, the result of any call, and **everything of type `CFn`** — the last for free, + because a `CFn` has no environment to dangle and the type says so. The second follows from the first refusal, + which is what stops a function from returning a suspect at all. + +The two types made this pass narrower rather than wider, which is the point of having them: a signature that says +`CFn` has already promised what the analysis would otherwise have to prove, and nothing written against one is +ever examined. + +**The clean set is the enumeration, not the suspect set**, and that is a correction. It read the other way round — +`Field`, `CaseField` and `Deref` named as suspect, everything else clean — and had a hole exactly where a list like +this cannot: `(at s 0)` over a slice of `Fn` is a `Prim`, so it came out clean while the `Vec`, struct and pointer +spellings of the same act were refused. Nothing can write an `Fn` into a slice today, so it was unreachable; but +the pass claims its enumeration is closed, and a default of "clean" is how that claim stops being true without +anyone noticing. `fn-escape-at.flan` pins it. The same inversion fixed which of the two refusal messages an index +read gets. + +Two of those arms are there because leaving them out is unsound rather than merely conservative, and each has a +program. **A function value read out of an environment** (`fn-escape-copy.flan`): a lifted body holds *copies* of +what it captured, read back with `Field(Deref env, i)`, so a copy of a captured function value carries whatever +environment the original did. Treat it as clean and the lifted body can return it, the return arrives at the outer +caller as an ordinary call result, and the whole "a call result is clean" rule has been walked around from inside. +**A valued form's tail** (`fn-escape-handled.flan`): `handler-bind`, `with-allocator` and `restart-case` are +expressions whose value is their body's — and a `restart-case`'s is a clause's too — so each is a way for a suspect +to be a function's answer that a check looking only at `return` and at the last form of a block would step over. + +**The parameter rule is the whole answer to the hard case.** A capturing `fn` passed to a function that stores it is +caught *inside that function*: its parameter is suspect there and the store is refused where it is written. So no +call can leak what its caller passed, and no caller has to be analysed. What it costs is real: +`(defn keep [f (Fn [] i32)] (Fn [] i32) f)` is refused although it is harmless, and so is holding a parameter of +function type in a `Vec` that never leaves the frame. `fn-escape-param.flan` is that refusal, written down as a +refusal of something that would sometimes have been fine. + +Refusing `Addr` of a suspect matters more than it looks: without it, `deref` of a `(Ptr (Fn ...))` launders a +suspect into a clean value and the return refusal has been walked around. Treating the `deref` itself as suspect is +the other half of that door, and it is free: nothing a `(Ptr (Fn ...))` can point at is anywhere but a frame, since +a global and a struct field of function type are both refused already. + +Name resolution inside a lifted body now asks the enclosing function's locals **before** the globals, which is a +deliberate tightening: inside the enclosing function a local shadows a global of the same name, so a body lifted out +of it must mean the same thing. The old order was an accident of where the refusal sat. + +Every one of these messages names **case 3** — the escaping closure, with an environment the collector owns — +because "this cannot be done" and "this cannot be done yet" are different sentences and the second is the true one. +Five programs: `fn-escape-return.flan`, `fn-escape-param.flan`, `fn-escape-store.flan`, `fn-escape-vec.flan`, and +`fn-capture-set.flan`. + +### What may be captured + +Anything but a **dyn**. A scalar, a struct and a fixed array copy whole. A string, a slice and a `Vec` or `Map` +header copy as their words, aliasing whatever they pointed at — which is exactly right while the value cannot +outlive the frame that owns the storage, and is exactly what would break under escape. A function value copies as a +function value, environment included; capturing one into another `fn`'s environment is the one place a suspect may +be written into an aggregate, and it is sound because the outer literal is itself suspect, so the pair of +environments lives and dies with one frame. + +A **dyn is refused**, for the reason a struct field of dyn already is (`A struct cannot hold a dyn field the +collector would never find`): the collector's roots are frames, and nothing pushes the fields of a synthesised +environment. A copy in there would be a live value reachable only through memory the marker never walks. Milestone +2's per-type descriptors lift it, alongside the condition payload's and the struct field's — and case 3's +collector-allocated environment is where it belongs anyway. `fn-capture-dyn.flan`. + +### Handlers, which get this for free and have no case 3 to wait for + +`docs/BUILT.md` said a handler clause cannot see the establishing function's locals, refused for the same reason, +and that this is also case 2. **It is, and it now can.** A handler frame is popped by the body that pushed it and +nothing in the language can name one, so the establishing frame is alive whenever the clause runs — there is no +escaping case here to leave over. The only compiler change beyond the shared machinery is one field on +`flan_handler` and one argument in `flan_signal`, which the uniform signature required anyway. + +What is still refused is a **store** into a captured name, in a clause as in an `fn`: the clause holds a copy, and +writing to it would change the copy and leave the local as it was. Scope is asked first, so a body's own `let` +shadowing a name the enclosing function also has is an ordinary local and an ordinary store — the refusal is about a +captured copy and not about a spelling. + +A **field** of a captured struct is a different matter and is deliberately left alone: `(set (.x p) 9)` inside an +`fn` writes the copy and leaves the enclosing `p` as it was — which is exactly what `(set (.x p) 9)` inside a +function whose `p` is a *parameter* already does, and has always done. Capture takes a copy the way a call takes +one, so the two agree; refusing here would make the `fn` stricter than the `defn` it was written in for no reason +anyone could state. So §1's accumulation case still accumulates into +a global — and now with whatever the establishing function knew readable beside it, which is the half that was +missing. `fn-capture.flan`'s `handles` reads a captured budget in the clause. + +Capturing the establishing frame **by reference** would make the accumulation case work directly and would be sound +here, uniquely — but it is a different feature from case 2, it would fork what "capture" means between the two +constructs, and it is not what was asked for. Named, not done. + +### Recursion, nesting and loops + +Capture is **transitive**: an `fn` inside an `fn` naming a local of the function both were written in makes the +middle one capture it and the inner one copy the middle one's copy. That is the same value, because every copy on +the way was taken at the moment its own value was made and those moments are nested. `check`'s context carries a +`parent` for exactly this, and it is safe to reach into precisely because a lifted body is checked at the point it +is written, with its parent paused there. + +An `fn` written **inside a loop** stores into the same environment slot each time round, so what it sees is the +value on its own iteration and not the last one. `fn-capture.flan` sums `0 + 1 + 2 + 3` through a fresh `fn` per +iteration to say so. The trap this could have been — a closure `set` into a local bound outside the loop and called +after it, seeing the last iteration's copies — cannot be written: a captured value cannot be `set` anywhere, and a +`let` binding scopes to the iteration. + +### What each backend cost + +Very little, which was the point of putting the environment in a frame slot and passing it as an ordinary argument +at the one call that needs it. + +`emit.ml`: a `%fnv` type and a 16-byte layout for `Fn`, `ptr` and eight for `CFn`; an `insertvalue` pair where a +symbol used to stand alone; two `extractvalue`s at a call through an `Fn`; one appended operand on that call and on +no other; one store in the prologue of a body that declared an environment. The four hand-written glue sites — the +startup call, `main`, the macro thunk and the reload thunk — are **unchanged**. + +`x86.ml`: `Fn` becomes an aggregate and `CFn` stays a scalar; one appended argument after the channel on an `Fn` +call; an optional `incoming` slot; the 16-byte value split at a `CallPtr`. The three hand-built entries are +unchanged. + +`flan_rt.c`: one field on `flan_handler` and one argument on the clause typedef, both trailing. The hash and +equality typedefs and their five call sites are **unchanged** — a hasher is reached from inside that file and never +through a function value, so it declares no environment and is handed none. ### `Fnval`, and the one thing a dev build cannot do @@ -3871,6 +4178,14 @@ What that does *not* give: a value taken *before* a redefinition and called afte address is in a slot there is nothing left to re-resolve, and the honest fix is a trampoline per function, which is a cost every program would pay for a case no one has hit. Named here rather than papered over. +**An `fn` literal was asking for `Fnval` and should never have been**, which the capture lane found as a bug rather +than as a design question: `flan reload` on any function containing an `fn` literal failed at `llc` with +`use of undefined value '@flan.cell.fn/OWNER/N'`. A lifted body has no name anyone can type and no way to be +redefined on its own — it is reached by address from the body it was written in, and a redefinition of that body +carries its own copy — so the cell could never hold anything but the symbol, and a redefinition module had no reason +to declare one. It takes `Flanfn` now, which is the choice a handler clause has always made and for the same reason. +`Fnval` remains what it was for: a `defn`'s *name* in value position. + The two lifted-function name sequences are counted **per kind** — `fn/OWNER/N` and `handler/OWNER/N/TYPE` — rather than off one list. Sharing a counter would rename every `fn` in a function the moment a `handler-bind` was added above one, which is a rename for a body that did not change, in exactly the names a redefinition module emits. diff --git a/lib/ast.ml b/lib/ast.ml index c9cab783..a24d2b78 100644 --- a/lib/ast.ml +++ b/lib/ast.ml @@ -18,7 +18,12 @@ and texpr_kind = | Tarray of len * texpr (* [4 f32] [rows [cols u32]] *) | Tmap of texpr * texpr (* (Map string i32) *) | Tapp of string * texpr list (* (Ptr Cursor) (Option f64) *) - | Tfn of texpr list * texpr (* (Fn [a a] bool) *) + (* (Fn [a a] bool) and (CFn [a a] bool). The flag is whether the value + carries an environment: true for [Fn], false for [CFn]. One case + rather than two because everything that walks a type expression treats + them identically — the difference is a fact about the value, and it is + [Check.resolve] that turns it into one. *) + | Tfn of bool * texpr list * texpr (* An array length is an integer or a compile-time constant's name. *) and len = diff --git a/lib/check.ml b/lib/check.ml index f523457b..5d05d087 100644 --- a/lib/check.ml +++ b/lib/check.ml @@ -503,6 +503,34 @@ type ctx = { refusal below says why the enclosing function's locals are not there. Both are the same gap: capture does not exist. *) mutable outer_what : string option; + (* What this body has captured out of [outer], in the order it first named + each one: the source name, the *outer* binding the copy is taken from, + and the slot in this body's own frame the copy is read into. Empty + everywhere [outer] is, which is everywhere but a lifted body. + + The order is the environment's field order, so it is the order the copy + is made in and the order the body reads it back in. First-reference order + rather than declaration order because it is the only one this pass has: + the outer scope is a list and the names on it were not all written for + this body's sake. *) + mutable caught : (string * (binding * int)) list; + (* The context this body was lifted out of, so that capture can be + transitive: an [fn] inside an [fn] naming a local of the function both + were written in is captured by the middle one and then by the inner one + out of the middle one's copy. Without it the inner body would see only + what the middle body happened to have named already, which is a rule + about the text and not about the scope. + + It is safe to reach into while it is on the stack, and only then: a + lifted body is checked at the point it is written, so the parent is + paused exactly there and its scope is the snapshot [outer] holds. *) + parent : ctx option; + (* The slot the environment pointer arrives in, minted the first time + something is captured and [None] until then. Nameless, so the break loop + hides it the way it hides every other slot the compiler made: what a + reader wants to see is the copies, and those are under the names the + source gave them. *) + mutable envslot : int option; (* True wherever handler or restart frames established by this function are on the stack. A [return] from there would leave them pointing into a frame that has gone, so it is refused — the same rule as [defer] inside a @@ -582,26 +610,120 @@ let bind ctx ?what name bty ~assignable = let lookup ctx name = List.assoc_opt name ctx.scope -(* A handler clause is lifted into a function of its own, so the establishing - function's locals are simply not there. Capturing them is a closure with an - explicit environment — spec-memory.md's case 2, a non-escaping [fn] capturing - by value into a stack environment, since a handler frame does not outlive the - function that pushed it — and until that exists a reference to one is refused - for the reason it is really refused for, rather than as a name nobody has - heard of. *) -let captured ctx loc name = - match ctx.outer_what with - | Some what when List.mem_assoc name ctx.outer -> - let why = - if String.equal what "a handler" then - "Use a global, or pass it on the condition" - else - "Pass it in, or use a global" +(* Capture, spec-memory.md's case 2: a body lifted into a function of its own — + an [fn] literal or a handler clause — naming a local of the function it was + written in. + + The copy is taken where the value is made and not where it is read, so the + name in the body means what the local held at that instant and nothing + later can change it. What makes that safe is the extent: the copies live in + a slot of the *enclosing* frame, and a value holding their address may not + outlive it. [escapes] is the whole of what enforces that, and it is where + the escaping half — an environment the collector allocates — is named. + + Answers the binding the body should use, minting one on first reference: + a named slot of this body's own frame, which the prologue fills from the + environment. Not assignable, and that is not an omission — see [captured_set]. + + A dyn is refused, and for the reason a struct field of dyn already is (see + the [defstruct] arm): the collector's roots are frames, a captured copy + lives inside a struct the checker synthesised, and nothing pushes the + fields of one. A dyn in there would be a live value reachable only through + memory the marker never walks. Milestone 2's per-type descriptors lift it, + alongside the condition payload's and the struct field's. *) +let rec capture ctx loc name = + match List.assoc_opt name ctx.caught with + (* Already captured, and named again from a scope that no longer lists it — + a [let] inside the body restores what it displaced, and the copy's + binding goes with it. One field, not two: the environment is keyed by the + source name. *) + | Some (outer, slot) -> Some { slot; bty = outer.bty; assignable = false; bwhat = None } + | None -> + let from_parent () = + (* Not a local of the body directly around this one, so ask whether that + body can capture it in turn. The middle one takes a copy and this one + takes a copy of that — which is the same value, because every copy on + the way was taken at the moment its own value was made, and those + moments are nested. *) + match ctx.parent with + | Some p -> capture p loc name + | None -> None in - Loc.failk "check/capture" loc - "%s cannot see %s — it is a local of the enclosing function. %s" - what name why - | _ -> () + let outer = + match List.assoc_opt name ctx.outer with + | Some b -> Some b + | None -> if ctx.outer_what = None then None else from_parent () + in + match ctx.outer_what, outer with + | Some what, Some (outer : binding) -> + if outer.bty = Types.Dyn then + (* [what] is a descriptor — "an fn", "a handler" — so it reads as the + subject of a sentence and nowhere else. It used to be substituted + into a noun slot as well, which produced "the environment an fn is + handed"; the environment belongs to *this* capture and naming it + twice said less, not more. *) + Loc.failk "check/capture-dyn" loc + "%s cannot capture %s: it is a dyn, and the collector finds its \ + roots by frame — a copy inside the environment would be a live \ + value nothing walks. Pass it in as a parameter, or hold it in a \ + global" + what name; + let slot = bind ctx name outer.bty ~assignable:false in + ctx.caught <- ctx.caught @ [ (name, (outer, slot)) ]; + Some { slot; bty = outer.bty; assignable = false; bwhat = None } + | _ -> None + +(* The one thing [capture] does not answer for. A captured name is a copy, so + a store into it would change this body's copy and leave the local it came + from as it was — which is a silent disagreement and not a feature. The + ordinary "not assignable" message would name the wrong reason, so this one + names the right one. + + [and] rather than a second [let] only so the two read together; neither + calls the other. *) +(* What [capture] would find, without capturing it. A guard that has to know + the *type* of an enclosing local before deciding what a form means — a + name in head position is a call through a value only if the value is a + function — must not take a copy on the way to answering. The binding it + answers with is only good for its type unless it came out of [caught]. *) +and peek_outer ctx name = + if ctx.outer_what = None then None + else + match List.assoc_opt name ctx.caught with + | Some ((b : binding), slot) -> Some { slot; bty = b.bty; assignable = false; bwhat = None } + | None -> + match List.assoc_opt name ctx.outer with + | Some b -> Some b + | None -> + match ctx.parent with Some p -> peek_outer p name | None -> None + +and outer_local ctx name = + ctx.outer_what <> None + && (List.mem_assoc name ctx.caught + || List.mem_assoc name ctx.outer + || (match ctx.parent with + | Some p -> outer_local p name + | None -> false)) + +and captured_set ctx loc name = + if outer_local ctx name then + match ctx.outer_what with + | Some what -> + Loc.failk "check/capture-set" loc + "%s cannot assign to %s: it is a copy of the enclosing function's \ + local, taken where the value was made, so a store here would change \ + the copy and leave %s as it was. %s" + what name name + (if String.equal what "a handler" then + "Accumulate into a global, or put the value on the condition — \ + capture is by value, which is what lets the copy be read at all" + else + "Return the new value, or keep it in a local of this fn") + | None -> () + +(* A binding this body captured, as opposed to one it declared. Used where the + difference matters and nowhere else. *) +let is_captured ctx name = List.mem_assoc name ctx.caught let scoped ctx f = let saved = ctx.scope in @@ -873,6 +995,13 @@ let map_type ?(preds = []) loc (k : Types.t) (v : Types.t) = (* The positions a function value may not be written in, and the one reason they are all the same position: something zeroes it. + Since capture arrived there is a second reason standing behind the first, + and it is the sharper one: every position on this list outlives the frame + a captured environment is on, so even a value nobody zeroed could not be + kept there. The message names the zero because that is the one that applies + to *every* function value and not only to a capturing one — and the escape + is what [escape_check] says, at the store rather than at the declaration. + ZII is the language's rule — an omitted struct field, a fixed array's elements, a [defonce] with no initialiser are all all-bytes-zero — and a zeroed function value is a null pointer with a signature on it, which is the @@ -883,9 +1012,21 @@ let map_type ?(preds = []) loc (k : Types.t) (v : Types.t) = A parameter, a return type, a [let] binding and an [(Option (Fn ...))] are not on the list: none of them is ever conjured, and an [Option]'s zero is a [None] whose tag nobody may look past. *) +(* The two function types as one question, for every place that wants the + signature and does not care which of them carries an environment: a call + site, a shadowing guard, a walk over the parameters. Where the difference + matters it is matched on directly, and there are few such places — the + representation is [Emit]'s business and the coercion is [expect]'s. *) +let fn_sig (t : Types.t) = + match t with + | Types.Fn (ps, r) | Types.CFn (ps, r) -> Some (ps, r) + | _ -> None + +let callable_ty t = fn_sig t <> None + let rec no_zeroed_fn loc what (t : Types.t) = match t with - | Types.Fn _ -> + | Types.Fn _ | Types.CFn _ -> fail loc "%s cannot be %s — it would be zeroed, and a zeroed function value is a \ null pointer. Pass it as a parameter, or hold it in a let" @@ -986,17 +1127,27 @@ let rec resolve env ?(seen = []) (t : Ast.texpr) : Types.t = | Ast.Tmap (k, v) -> map_type ~preds:env.tvpreds loc (resolve env ~seen k) (resolve env ~seen v) - (* (Fn [T ...] R): a function value, which is one code address and no - environment beside it. There is no capture — [check_fn] refuses a - reference to an enclosing local by name — so this is a pointer with a - signature and nothing about it can dangle. + (* (Fn [T ...] R) is a code address and the environment it is called with: + two words. A value made out of a name carries a null there; one made out + of an [fn] that captures carries the address of the copies on the frame + it was written in, and [escape_check] is what stops that address + outliving the frame. + + (CFn [T ...] R) is the address alone, one word, and nothing that can + capture — see [Types] for why the C is information rather than + decoration, and for why it is not yet a capability. Nobody needs it: + [Fn] accepts everything, and the commonest reason to reach for the + narrow one is that a *named* function handed to an [Fn] pays a hop + through the widening thunk where a [CFn] is a direct call. Where one may be *written* is narrower than where the type resolves, and the two rules live apart on purpose: this is what the spelling means, and [no_zeroed_fn] is where a position that would zero one is refused. A - parameter, a return type and a let binding are the positions that work. *) - | Ast.Tfn (ps, r) -> - Types.Fn (List.map (resolve env ~seen) ps, resolve env ~seen r) + parameter, a return type and a let binding are the positions that work, + for both. *) + | Ast.Tfn (env', ps, r) -> + let ps = List.map (resolve env ~seen) ps and r = resolve env ~seen r in + if env' then Types.Fn (ps, r) else Types.CFn (ps, r) | Ast.Tapp (name, args) -> (match name, args with | "Ptr", [ a ] -> Types.Ptr (resolve env ~seen a) @@ -1554,7 +1705,7 @@ let signature_tyvars (fn : Ast.fn) = and a variable cannot stand there: this spike is generic over types, not over type constructors. A [$t] inside the arguments is ordinary. *) | Ast.Tapp (_, args) -> List.iter ty args - | Ast.Tfn (ps, r) -> List.iter ty ps; ty r + | Ast.Tfn (_, ps, r) -> List.iter ty ps; ty r in List.iter (fun (p : Ast.field) -> ty p.Ast.fty) fn.Ast.params; (match fn.Ast.ret with Some r -> ty r | None -> ()); @@ -1565,7 +1716,7 @@ let signature_tyvars (fn : Ast.fn) = and with the same rule: a variable already bound must match what it is bound to, so [(pair 1 2.0)] over [a $t b $t] is a refusal and not a second instantiation. *) -let rec bind_ty subst (pat : Types.t) (arg : Types.t) = +let rec bind_ty ?(widen = false) subst (pat : Types.t) (arg : Types.t) = match pat, arg with | Types.Var v, a -> (match List.assoc_opt v !subst with @@ -1578,7 +1729,36 @@ let rec bind_ty subst (pat : Types.t) (arg : Types.t) = | Types.Array (n, p), Types.Array (m, a) -> Int64.equal n m && bind_ty subst p a | Types.Map (k, v), Types.Map (k', v') -> bind_ty subst k k' && bind_ty subst v v' - | Types.Fn (ps, r), Types.Fn (ps', r') -> + (* Each function type against its own. *) + | Types.Fn (ps, r), Types.Fn (ps', r') + | Types.CFn (ps, r), Types.CFn (ps', r') -> + List.length ps = List.length ps' + && List.for_all2 (bind_ty subst) ps ps' && bind_ty subst r r' + (* And the widening between them, which is admitted at the top of an + argument's type and nowhere inside it. + [(Fn [$t] $t)] against a [(CFn [i32] i32)] is the shape every caller of + a generic higher-order function has, because a [defn]'s name carries + [CFn]: [(apply2 bump 1)]. It has to bind here, where the variables are + decided, and not only in [expect] — [generic_call] binds first and would + have reported the mismatch before [expect] was ever reached. + + The prelude hides that: its higher-order functions bind [$t] from an + earlier argument, so [subst_ty] has already made the parameter concrete + by the time this sees it and [(map-in-place s double)] never took this + path. + + [widen] is why it goes no deeper. The widening is a *value* the caller + builds — a thunk, minted at the call — and there is exactly one place to + build it, around the whole argument. A [(Fn [(Fn [$t] $t)] i32)] + parameter handed a [(CFn [(CFn [i32] i32)] i32)] would need one built + inside the argument's own parameter list, where no caller stands, so the + two types do not meet there and the pattern does not match. What reaches + the fallthrough below is an ordinary mismatch and is refused as one, the + same answer a call with no type variables in it gets. + + One way, as everywhere else: a [CFn] pattern does not admit an [Fn] + argument. *) + | Types.Fn (ps, r), Types.CFn (ps', r') when widen -> List.length ps = List.length ps' && List.for_all2 (bind_ty subst) ps ps' && bind_ty subst r r' (* Nothing generic left on the pattern side: this is ordinary type @@ -1595,6 +1775,8 @@ let rec subst_ty subst (t : Types.t) = | Types.Vec e -> Types.Vec (subst_ty subst e) | Types.Option e -> Types.Option (subst_ty subst e) | Types.Fn (ps, r) -> Types.Fn (List.map (subst_ty subst) ps, subst_ty subst r) + | Types.CFn (ps, r) -> + Types.CFn (List.map (subst_ty subst) ps, subst_ty subst r) | t -> t (* Does this resolved type still mention a variable? *) @@ -1604,7 +1786,8 @@ let rec generic_ty (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> generic_ty e | Types.Map (k, v) -> generic_ty k || generic_ty v - | Types.Fn (ps, r) -> List.exists generic_ty ps || generic_ty r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists generic_ty ps || generic_ty r | _ -> false (* Does a type a call site bound a variable to reach a [dyn] anywhere? See the @@ -1617,7 +1800,8 @@ let rec reaches_dyn (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> reaches_dyn e | Types.Map (k, v) -> reaches_dyn k || reaches_dyn v - | Types.Fn (ps, r) -> List.exists reaches_dyn ps || reaches_dyn r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists reaches_dyn ps || reaches_dyn r | _ -> false (* The refusal plan.org's Types section asks for, in one place so that every @@ -1662,6 +1846,9 @@ let rec mangle_ty (t : Types.t) = | Types.Fn (ps, r) -> Printf.sprintf "fn-%s-to-%s" (String.concat "-" (List.map mangle_ty ps)) (mangle_ty r) + | Types.CFn (ps, r) -> + Printf.sprintf "cfn-%s-to-%s" + (String.concat "-" (List.map mangle_ty ps)) (mangle_ty r) | t -> Types.to_string t (* ── The runaway instantiation, refused by name rather than by depth ──── @@ -1697,7 +1884,7 @@ let rec occurs_in ~needle (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> occurs_in ~needle e | Types.Map (k, v) -> occurs_in ~needle k || occurs_in ~needle v - | Types.Fn (ps, r) -> + | Types.Fn (ps, r) | Types.CFn (ps, r) -> List.exists (occurs_in ~needle) ps || occurs_in ~needle r | _ -> false @@ -1766,6 +1953,69 @@ let mk loc ty e : Tast.expr = { Tast.e; ty; loc } let unit_at loc = mk loc Types.Unit Tast.Unit +(* The environment for a lifted body, built once its own body has been checked + and [caught] is therefore final. spec-memory.md's case 2, and the whole of + its machinery. + + It is a struct the checker synthesises — one field per captured name, in + first-reference order — and it is registered in the same table a + [defstruct] goes in, so both backends lay it out with the calculator they + already have and neither learns a new shape. Its name is the lifted + function's, which is unique and stable for the reason that name is: a + redefinition module emits the lifted functions belonging to the bodies it + replaces, and it emits their environments with them. + + Two ends, and the copy is at the near one: + + - in the *enclosing* frame, a slot holding the struct, filled with a [Make] + of the outer locals. That store is the copy, and it happens where the + value is made. A literal written inside a loop stores into the same slot + each time round, so each value is made from the locals as they were on + its own iteration. + - in the *lifted* frame, a slot holding the pointer, and a [Let] around the + whole body reading each field back into the named slot the body has been + checked against. Once, at entry, for the same reason a handler clause + binds its condition once: what the body names is the copy and not an + address, so nothing downstream has to know an environment exists. + + Both new slots are nameless, which is how the break loop is told to hide + them: a reader wants the captured copies, and those are the named slots the + body reads them into. Answers the prefixed body, the slot the pointer + arrives in, the enclosing frame's binding, and the address to put in the + value. *) +let close_over ~fname (octx : ctx) (fctx : ctx) loc = + match fctx.caught with + | [] -> (fun body -> body), None, None, None + | caught -> + let ename = "env/" ^ fname in + let fields = + List.map + (fun (n, ((b : binding), _)) -> { Tast.fname = n; fty = b.bty }) + caught + in + Hashtbl.replace fctx.env.structs ename { Tast.sname = ename; fields }; + let ety = Types.Named ename in + let eslot = fresh_slot fctx (Types.Ptr ety) in + let binds = + List.mapi + (fun i (_, ((b : binding), slot)) -> + let p = mk loc (Types.Ptr ety) (Tast.Local eslot) in + (slot, mk loc b.bty (Tast.Field (mk loc ety (Tast.Deref p), i)))) + caught + in + let prefix body = [ mk loc fctx.ret (Tast.Let (binds, body)) ] in + let mslot = fresh_slot octx ety in + let make = + mk loc ety + (Tast.Make (ename, + List.map + (fun (_, ((b : binding), _)) -> + mk loc b.bty (Tast.Local b.slot)) + caught)) + in + prefix, Some eslot, Some (mslot, make), + Some (mk loc (Types.Ptr ety) (Tast.Addr (Tast.Plocal mslot))) + (* A source location as a value, for a runtime trap that has to name the site rather than the runtime. The bounds and slice traps get theirs from [Emit], which renders the [Loc.t] it is already carrying; a trap reached through a @@ -2269,7 +2519,7 @@ let box loc (e : Tast.expr) : Tast.expr = caller that starts doing that gets a sentence instead of a silent mis-lowering. *) | Types.Named _ | Types.Enum _ | Types.Option _ | Types.Ptr _ - | Types.Alloc | Types.Fn _ | Types.Var _ -> + | Types.Alloc | Types.Fn _ | Types.CFn _ | Types.Var _ -> no_dyn_yet loc ~into:true e.Tast.ty "" let unbox loc (want : Types.t) (e : Tast.expr) : Tast.expr = @@ -2468,6 +2718,120 @@ let unbox_option ctx loc (t : Types.t) (got : Tast.expr) : Tast.expr = Written once and used by both refusals that can report one — [expect]'s, and the binary operators' when their two operands have no join. *) +(* ── The widening thunk ──────────────────────────────────────────────── + One per signature, and the whole of what a [CFn] costs on its way into + an [Fn]. Its parameters are the signature's, it declares the environment, + and its body calls through what it finds there — because that is where the + original bare address was put. + + Which is the move that makes the two conventions meet in exactly one + place. Every body reachable through an [Fn] value declares the trailing + environment, so every indirect call is exactly typed and nothing anywhere + relies on a callee ignoring an argument it never declared. That was the + first design's hinge and it does not survive wasm32: [call_indirect] + compares the signature at the call and a spare argument is a trap. + + Per *signature* and not per name, so a program pays one small function per + distinct shape it widens rather than one per function it widens. Two + widenings of the same shape share a thunk, which is what the memo below is + for — the same arrangement [struct_key_pair] uses for a map's hash and + equality pair, and for the same reason. + + **The memo is keyed on the types and the symbol spells them back**, and + both halves matter. [mangle_ty] cannot serve as the spelling: it flattens + a whole signature into one hyphen-joined string, which loses arity and + every type boundary with it, so [(CFn [(Ptr i32)] i32)] and + [(CFn [ptr i32] i32)] — the second over a struct someone called [ptr] — + both come out [cfn-ptr-i32-to-i32]. Keyed on that string, the second + widening silently reuses the first's thunk and calls it with the wrong + arity, which is a miscompile on both backends and not a refusal anywhere. + So the key is the types, compared with [Types.equal], and the name is + [thick_enc]'s encoding, which no two signatures share. + + ([mangle_ty]'s ambiguity is older than this and is still there for the + generic instantiation names it was written for. At one type's granularity + it is hard to reach; at a whole signature's it is a line of Flan away.) + + **Why the name has to be the signature and not a counter.** A counter over + the thunks minted so far is unique within one compilation and says nothing + across two: reorder the definitions in the file and [thick/0] is a + different signature than it was. [Session.compatible] compares a reload's + functions against the running program's *by name*, so a thunk that changed + shape under a fixed name reads to it as a function whose signature was + edited, and the dev loop answers a form reorder with "Restart to change + it". Spelled from the types, the name moves with the shape and that + comparison is right again for the same reason it is right everywhere else. + + [fparent] is []: not a name anyone wrote, so [defs] hides it, and a + marker the redefinition modules match on to carry a copy of their own. *) + +(* A signature written so that it can be read back: every type is + self-delimiting, so no two distinct signatures encode alike. + + An atom is its length and then its spelling, which is what closes the gap + [mangle_ty] leaves — a name's boundaries are in the string rather than + inferred from the separators. A constructor is one letter, and the two + that hold a count write it before their children, so [(Fn [i32] i32)] and + [(Fn [] (Fn [i32] i32))] cannot read alike. [Named] and [Enum] carry + different letters because a struct and a C enum may share a spelling. *) +let rec thick_enc (t : Types.t) = + let atom s = Printf.sprintf "%d-%s" (String.length s) s in + let arrow tag ps r = + Printf.sprintf "%s%d-%s" tag (List.length ps) + (String.concat "-" (List.map thick_enc (ps @ [ r ]))) + in + match t with + | Types.Named n -> "n" ^ atom n + | Types.Enum n -> "e" ^ atom n + | Types.Var v -> "y" ^ atom v + | Types.Slice e -> "s" ^ thick_enc e + | Types.Ptr e -> "p" ^ thick_enc e + | Types.Vec e -> "v" ^ thick_enc e + | Types.Option e -> "o" ^ thick_enc e + | Types.Array (n, e) -> Printf.sprintf "a%Ld-%s" n (thick_enc e) + | Types.Map (k, v) -> Printf.sprintf "m%s-%s" (thick_enc k) (thick_enc v) + | Types.Fn (ps, r) -> arrow "f" ps r + | Types.CFn (ps, r) -> arrow "c" ps r + | t -> atom (mangle_ty t) + +let thick_thunk env loc ps r = + let same (f : Tast.fn) = + f.Tast.fparent = Some "" + && List.length f.Tast.params = List.length ps + && List.for_all2 Types.equal f.Tast.params ps + && Types.equal f.Tast.ret r + in + match List.find_opt same env.lifted with + | Some f -> f.Tast.name + | None -> + let n = List.length ps in + let fty = Types.CFn (ps, r) in + let args = List.mapi (fun i t -> mk loc t (Tast.Local i)) ps in + let callee = mk loc fty (Tast.Local n) in + let name = "thick/" ^ thick_enc fty in + env.lifted <- + { Tast.name; params = ps; + slots = Array.of_list (ps @ [ fty ]); + snames = Array.make (n + 1) None; + ret = r; body = [ mk loc r (Tast.CallPtr (callee, args)) ]; + fdefers = []; fenv = Some n; fparent = Some ""; floc = loc } + :: env.lifted; + name + +(* The slot a lifted body's environment arrives in, minted when the body did + not capture anything and so has none of its own. + + Every body that can be *reached* through an [Fn] value declares the + parameter, whether or not it reads it: a lifted [fn] literal in an [Fn] + position, and every handler clause, since [flan_signal] passes the frame's + environment to all of them. The alternative is a call whose signature is + one argument longer than the callee's, which SysV tolerates and wasm32 + does not. The slot is nameless, so the break loop hides it, and a body + that never reads it costs one store the optimiser drops. *) +let declare_env ctx = function + | Some _ as s -> s + | None -> Some (fresh_slot ctx (Types.Ptr Types.Unit)) + let numeric_note ~(want : Types.t) ~(got : Types.t) = if not (Types.is_numeric want && Types.is_numeric got) then "" else if Types.widens_to ~from:want ~into:got then @@ -2521,6 +2885,14 @@ let expect ctx loc ~want (got : Tast.expr) = honest — it admits only conversions that cannot change the number, so the cast this inserts is one no program can tell happened. *) | _ when Types.widens_to ~from:got.Tast.ty ~into:w -> widen loc w got + (* The other widening, and the only coercion between the two function + types. A bare address satisfies a signature that asks for an + environment. It goes this way only — an [Fn] has an environment and a + [CFn] has nowhere to put one — so the reverse falls through to the + ordinary refusal, which names both types and is the right sentence. *) + | Types.Fn (ps, r), Types.CFn (ps', r') + when Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) -> + mk loc w (Tast.Thicken (thick_thunk ctx.env loc ps r, got)) | _ -> got in if Types.fits ~expected:w ~actual:got.Tast.ty then got @@ -2598,7 +2970,7 @@ let hash_ty = Types.Int Types.U64 would share a slot counter. *) let invented_ctx env ret = { env; ret; slots = 0; slot_tys = []; slot_names = []; scope = []; - defers = []; defer_slot = None; outer = []; outer_what = None; in_frames = None; loops = []; tail = false; + defers = []; defer_slot = None; outer = []; outer_what = None; caught = []; envslot = None; parent = None; in_frames = None; loops = []; tail = false; in_defer = false; defer_ok = false; defer_block = "a nested form"; owner = "" } @@ -2711,7 +3083,7 @@ and struct_key_pair env loc n = let placeholder name ret params = { Tast.name; params; slots = Array.of_list params; snames = Array.make (List.length params) None; - ret; body = []; fdefers = []; fparent = None; floc = loc } + ret; body = []; fdefers = []; fenv = None; fparent = None; floc = loc } in env.lifted <- placeholder hname hash_ty hparams @@ -2794,7 +3166,7 @@ and struct_key_pair env loc n = { Tast.name; params; slots = Array.of_list (List.rev ctx.slot_tys); snames = Array.of_list (List.rev ctx.slot_names); - ret; body; fdefers = []; fparent = None; floc = loc } + ret; body; fdefers = []; fenv = None; fparent = None; floc = loc } in env.lifted <- finish hname hash_ty hparams hctx hbody @@ -3440,6 +3812,15 @@ and var ctx ?(qualified = false) loc ~want name = match lookup ctx name with | Some b -> expect ctx loc ~want (mk loc b.bty (Tast.Local b.slot)) + (* A local of the enclosing function, in a body that was lifted out of it: + captured by value, here, where it is first named. Asked *before* the + globals, because that is what the name means at the place it is + written — inside the enclosing function a local shadows a global of the + same name, and a body lifted out of it must not silently mean something + else. *) + | None when capture ctx loc name <> None -> + let b = Option.get (capture ctx loc name) in + expect ctx loc ~want (mk loc b.bty (Tast.Local b.slot)) | None -> match Hashtbl.find_opt ctx.env.globals name with | Some (ty, _) -> @@ -3482,8 +3863,9 @@ and var ctx ?(qualified = false) loc ~want name = "%s is a foreign function, and its address is not a Flan \ function value. Wrap it in a defn and pass that" name; expect ctx loc ~want - (mk loc (Types.Fn (params, ret)) (Tast.FnAddr (Tast.Fnval name))) - | None -> captured ctx loc name; unknown_name ctx loc name) + (mk loc (Types.CFn (params, ret)) + (Tast.FnAddr (Tast.Fnval name))) + | None -> unknown_name ctx loc name) (* What remains of spec-memory.md's ownership section after the repeals of 2026-09-18 is the allocator's side alone: the region rule decides where a @@ -3533,14 +3915,15 @@ and block ctx ?want ?(defer_ok = false) loc body = landed, and the surface feature is that machinery given a name rather than a second one invented beside it. - **No capture, and that is the scope of this milestone.** The body sees its - parameters and the program's globals and nothing else; a reference to a - local of the enclosing function is refused by name (see [captured]) rather - than resolved to something it did not mean. That is what makes the value a - bare code address with no environment behind it, which in turn is what makes - it safe to pass down, return, and store: there is nothing that can outlive - anything. spec-memory.md's capture cases, and escaping closures with them, - stay deferred. + **Capture is by value, and the value may not escape.** The body sees its + parameters, the program's globals, and the locals of the function it was + written in — those last copied into an environment on that function's + frame at the instant the value is made (see [capture] and [close_over]). + So the value is two words, the second of them an address into a frame, and + what keeps that address good is [escape_check]: it may be called, passed + down and let-bound, and may not be returned, stored or pushed anywhere. + spec-memory.md's case 3 — an environment the collector owns, and with it + the escaping closure — is a separate lane, and every refusal names it. **The parameter types come from the position.** [Ast.Fn] carries names and no types — that is the surface syntax, not an omission here — so an fn is @@ -3553,26 +3936,36 @@ and check_fn ctx ~want ?gen loc (params : string list) body = [(Fn ...)] want to say so, and the return is the annotated element type, or [None] to take the body's own. Everything else threads [want]. The caller has already checked the arity, in its own words. *) + (* Which of the two function types was asked for. [CFn] is a bare + address, so a literal written into one has nowhere to put an + environment — it is checked exactly as an [Fn] is and then refused *if + it turned out to capture*, which is a decision only the finished body + can make. Nothing else differs. *) + let bare = match want with Some (Types.CFn _) -> true | _ -> false in let pts, ret0 = match gen with | Some (pts, r) -> pts, r | None -> - match want with - | Some (Types.Fn (ps, r)) when List.length ps = List.length params -> + match Option.map fn_sig want with + | Some (Some (ps, r)) when List.length ps = List.length params -> ps, Some r - | Some (Types.Fn (ps, r)) -> + | Some (Some (ps, r)) -> fail loc "this fn has %d parameter%s and %s was wanted here" (List.length params) (if List.length params = 1 then "" else "s") - (Types.to_string (Types.Fn (ps, r))) - | Some other when other <> Types.Never -> - fail loc "expected %s, found an fn" (Types.to_string other) + (Types.to_string + (if bare then Types.CFn (ps, r) else Types.Fn (ps, r))) | _ -> - fail loc - "nothing here says what this fn's parameters are — an fn takes its \ - types from the position it is written in. Write it as an argument \ - whose parameter is a (Fn [T ...] R)" + match want with + | Some other when other <> Types.Never -> + fail loc "expected %s, found an fn" (Types.to_string other) + | _ -> + fail loc + "nothing here says what this fn's parameters are — an fn takes \ + its types from the position it is written in. Write it as an \ + argument whose parameter is a (Fn [T ...] R), or a \ + (CFn [T ...] R) when it captures nothing" in (* Its own frame and its own empty scope, with [outer] kept only so that a reference to the enclosing function's locals is refused for the reason it @@ -3582,7 +3975,8 @@ and check_fn ctx ~want ?gen loc (params : string list) body = is an expression, and no machinery is built for the form nobody writes. *) let fctx = { (invented_ctx ctx.env (Option.value ret0 ~default:Types.Unit)) with - outer = ctx.scope; outer_what = Some "an fn"; owner = ctx.owner } + outer = ctx.scope; outer_what = Some "an fn"; parent = Some ctx; + owner = ctx.owner } in List.iter2 (fun n t -> ignore (bind fctx n t ~assignable:false)) params pts; @@ -3635,25 +4029,73 @@ and check_fn ctx ~want ?gen loc (params : string list) body = in Printf.sprintf "fn/%s/%d" ctx.owner (List.length mine) in + (* And the environment, now that the body has named everything it is going + to. [close_over] allocates in both frames, so it runs after the body's + slots and before the lifted function is recorded. *) + let prefix, fenv, bind, addr = close_over ~fname ctx fctx loc in + (* A [CFn] is a bare address and has nowhere to keep an environment, so a + literal that captured cannot be one. Refused with the name of what it + captured, because that is the fact the writer has to act on — and with + the fix named, which is the wider type. *) + if bare && fctx.caught <> [] then begin + let names = List.map fst fctx.caught in + Loc.failk "check/cfn-captures" loc + "this fn captures %s, so it is a (Fn [%s] %s) and not a (CFn [%s] \ + %s): a CFn is the bare address, one word, with nowhere for the \ + copies to live. Widen the position to Fn, or pass %s in as a parameter" + (String.concat ", " names) + (String.concat " " (List.map Types.to_string pts)) (Types.to_string ret) + (String.concat " " (List.map Types.to_string pts)) (Types.to_string ret) + (match names with [ n ] -> n | _ -> "them") + end; + (* An [Fn]-position literal declares the environment whether or not it + captured: it is reached by a call that passes one. A [CFn]-position + one must not — it is reached by calls that pass none, and a parameter + nobody supplies is read off whatever the register held. *) + let fenv = if bare then fenv else declare_env fctx fenv in ctx.env.lifted <- { Tast.name = fname; params = pts; slots = Array.of_list (List.rev fctx.slot_tys); snames = Array.of_list (List.rev fctx.slot_names); - ret; body = fbody; fdefers = []; - fparent = Some ctx.owner; floc = loc } + ret; body = prefix fbody; fdefers = []; + fenv; fparent = Some ctx.owner; floc = loc } :: ctx.env.lifted; - expect ctx loc ~want - (mk loc (Types.Fn (pts, ret)) (Tast.FnAddr (Tast.Fnval fname))) + let fty = if bare then Types.CFn (pts, ret) else Types.Fn (pts, ret) in + (* [Flanfn] and not [Fnval], which is the handler clause's choice and is the + same choice for the same reason. [Fnval] exists so that a *name* taken as + a value in a dev build answers with the body that is current, which means + a load from that name's indirection cell. A lifted body has no name + anyone can type and no way to be redefined on its own: it is reached by + address from the body it was written in, and a redefinition of that body + carries its own copy. So the cell would never hold anything but this + symbol, and asking for one is how a redefinition module came to reference + a cell nothing declares. *) + let v = + match addr with + | None -> mk loc fty (Tast.FnAddr (Tast.Flanfn fname)) + | Some a -> + (* The value, and the store that fills its environment around it. What + stops the value leaving this frame is [escape_check], which reads the + finished body: a rule about where a value may *go* cannot be settled + at the point it is made. *) + let c = mk loc fty (Tast.Closure (Tast.Flanfn fname, a)) in + mk loc fty (Tast.Let ([ Option.get bind ], [ c ])) + in + expect ctx loc ~want v (* A handler runs where the *signal* was, not where it was established, so it cannot be a branch in the function that wrote it: it is lifted into a function of its own and reached through a pointer. - Which means it cannot see the establishing function's locals. Capturing them - is a closure with an explicit environment — the non-escaping kind, captured - by value onto this frame — and until that exists a reference to one is - rejected by name rather than silently resolving to something else. Globals and the condition itself are - in scope, which is enough for the accumulation case §1 is about. + It *can* see the establishing function's locals, by value: the same capture + an [fn] literal gets, and the one place it has no escaping case left over. + A handler frame is popped by the body that pushed it and nothing in the + language can name one, so the establishing frame is alive whenever the + clause runs and the copies on it are good. What is still refused is a store + into a captured name — the clause holds a copy, and writing to it would + leave the local as it was — so §1's accumulation case still accumulates + into a global, and now with whatever the establishing function knew + readable beside it. The body may not [return] either. The frames are pushed and popped around it, and an early exit would leave them on the stack pointing into a function @@ -3680,7 +4122,8 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = the enclosing one. *) let hctx = { (invented_ctx ctx.env Types.Unit) with - outer = ctx.scope; outer_what = Some "a handler" } + outer = ctx.scope; outer_what = Some "a handler"; + parent = Some ctx } in (* The condition crosses as a pointer, because the handler runs while the signalling frame is still alive and there is nothing to copy. @@ -3716,16 +4159,35 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = in Printf.sprintf "handler/%s/%d/%s" ctx.owner (List.length mine) name in + (* And the environment, the same machinery an [fn] literal's capture + uses and settled by the same argument — only with no escaping + case to leave over. A handler frame is popped by the body that + pushed it and nothing in the language can name one, so the clause + cannot be reached from anywhere the establishing frame is not + alive. There is nothing here that case 3 would change. *) + let prefix, fenv, bind, addr = + close_over ~fname ctx hctx c.Ast.hloc + in + (* Every clause declares the environment, captured or not: + [flan_signal] reads it off the frame and passes it to whichever + clause matched, and it cannot know which of them captured. *) + let fenv = declare_env hctx fenv in ctx.env.lifted <- { Tast.name = fname; params = [ Types.Ptr ty ]; slots = Array.of_list (List.rev hctx.slot_tys); snames = Array.of_list (List.rev hctx.slot_names); - ret = Types.Unit; body = hbody; fdefers = []; - fparent = Some ctx.owner; floc = c.Ast.hloc } + ret = Types.Unit; body = prefix hbody; fdefers = []; + fenv; fparent = Some ctx.owner; floc = c.Ast.hloc } :: ctx.env.lifted; - { Tast.htype = type_id name; hfn = fname }) + { Tast.htype = type_id name; hfn = fname; henv = addr }, bind) clauses in + (* The stores that fill the environments, one per clause that captured, + around the whole form: a handler frame carries the address and the frame + is pushed before the body runs, so the copies have to be made before + either. *) + let envbinds = List.filter_map snd frames in + let frames = List.map fst frames in (* The flag is set on [ctx] itself and restored, not on a copy: [ctx.slots] and [ctx.slot_tys] are mutable, so a copy would allocate the body's slots into a record the function never sees again and the indices would @@ -3765,7 +4227,11 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = go body) in ctx.in_frames <- saved; - expect ctx loc ~want (mk loc ty (Tast.Handled (frames, body))) + let h = mk loc ty (Tast.Handled (frames, body)) in + let h = + if envbinds = [] then h else mk loc ty (Tast.Let (envbinds, [ h ])) + in + expect ctx loc ~want h (* (restart-case BODY (name [] BODY-1) ...) — spec-conditions.md §3 and §6. @@ -5064,8 +5530,10 @@ and check_array_gen ctx ~want loc dims f = | _ -> check ctx f in let elem = - match f.Tast.ty with - | Types.Fn (ps, r) -> + (* Either function type: the generator is called and nothing here cares + whether an environment rides along. *) + match fn_sig f.Tast.ty with + | Some (ps, r) -> let got = List.length ps in if got <> rank then fail f.Tast.loc @@ -5081,12 +5549,12 @@ and check_array_gen ctx ~want loc dims f = (k + 1) (Types.to_string p)) ps; r - | other -> + | None -> fail f.Tast.loc "array-gen's second element is a function value, called once per \ element with one i32 index per dimension, and this is %s — for one \ value repeated, write array-fill" - (Types.to_string other) + (Types.to_string f.Tast.ty) in no_zeroed_fn loc "a fixed array's element" elem; let fs = fresh_slot ctx f.Tast.ty in @@ -5468,14 +5936,29 @@ and refuse_string_place loc (ty : Types.t) = and check_place ctx loc (p : Ast.place) : Tast.place * Types.t = match p with | Ast.Pvar name -> + (* Scope first, and the capture refusal only where scope did not settle + it. A body's *own* [let] may shadow a name the enclosing function also + has, and a store into that one is an ordinary store — asking about the + capture before looking would refuse it with a message about a copy that + is not the thing being written to. *) (match lookup ctx name with | Some b -> - if not b.assignable then + if not b.assignable then begin + (* Not assignable, so it is either a parameter or a captured copy. + Which one decides the message, and the copy's reason is its + own. *) + (match List.assoc_opt name ctx.caught with + | Some (_, slot) when slot = b.slot -> captured_set ctx loc name + | _ -> ()); fail loc "%s is a parameter, and a parameter is not assignable — bind a \ - local with let" name; + local with let" name + end; Tast.Plocal b.slot, b.bty | None -> + (* Not in scope here at all: a name of the enclosing function, which a + lifted body may read as a copy and may not write to. *) + captured_set ctx loc name; match Hashtbl.find_opt ctx.env.globals name with | Some (_, true) -> (* Four words, before: the name and the fact, and nothing about what @@ -5492,7 +5975,7 @@ and check_place ctx loc (p : Ast.place) : Tast.place * Types.t = "%s is a constant, and a constant is not assignable. Declare it \ with defonce if it has to change" name | Some (ty, false) -> Tast.Pglobal name, ty - | None -> captured ctx loc name; unknown_name ~setting:true ctx loc name) + | None -> unknown_name ~setting:true ctx loc name) | Ast.Pfield (target, name) -> let target, sname = struct_target ctx target in let s = Option.get (fields_named ctx.env sname) in @@ -5609,8 +6092,11 @@ and check_call ctx ~want loc (head : Ast.expr) (args : Ast.expr list) = above and by a name that resolved to a local or a parameter of function type, which is the shape every caller of [map] has. *) and call_value ctx ~want loc (callee : Tast.expr) args = - match callee.Tast.ty with - | Types.Fn (params, ret) -> + (* Either function type: calling one is calling the other, and the + difference — whether an environment rides along — is the backend's to + lower. Nothing here has to know which. *) + match fn_sig callee.Tast.ty with + | Some (params, ret) -> if List.length args <> List.length params then fail loc "this function value takes %d argument%s, given %d" (List.length params) @@ -5618,9 +6104,9 @@ and call_value ctx ~want loc (callee : Tast.expr) args = (List.length args); let args = map2_lr (fun p a -> check ctx ~want:p a) params args in expect ctx loc ~want (mk loc ret (Tast.CallPtr (callee, args))) - | other -> + | None -> fail loc "this is a %s and not a function, so it cannot be called" - (Types.to_string other) + (Types.to_string callee.Tast.ty) (* A builtin's arity. The count is the builtin's and can only be the builtin's: a defn of the same name written in the program now takes the @@ -6563,7 +7049,7 @@ and named_call ?(qualified = false) ctx ~want loc name args = | Types.Option _ -> "it carries a tag saying whether the value is there, and a \ filled one says yes over a payload nobody wrote" - | Types.Fn _ -> + | Types.Fn _ | Types.CFn _ -> "it is a code address, and a call through a filled one jumps \ into whatever 0xDE bytes happen to address" | Types.Named n when Hashtbl.mem ctx.env.datas n -> @@ -8311,16 +8797,26 @@ and ordinary_call ctx ~want loc name args = before the global function table: a binding shadows a defn of the same name (one namespace, ordinary lexical scoping). A local of any *other* type falls through to the table, so a program that shadows a function - name with an i32 and then calls the function still means the function. *) + name with an i32 and then calls the function still means the function. + + A local of the *enclosing* function holding one is the same case: a + lifted body captures it by value and then calls the copy. [peek_outer] + rather than [capture] in the guard, because a guard must not take a copy + on its way to deciding what a form means. *) | _ when (match lookup ctx name with - | Some b -> (match b.bty with Types.Fn _ -> true | _ -> false) - | None -> false) -> + | Some b -> callable_ty b.bty + | None -> + match peek_outer ctx name with + | Some b -> callable_ty b.bty + | None -> false) -> (* The binding the guard already found, read directly. Going back through - [check] would repeat the lookup and walk the capture path for a type - that is not capturable. *) + [check] would repeat the lookup. *) (match lookup ctx name with | Some b -> call_value ctx ~want loc (mk loc b.bty (Tast.Local b.slot)) args - | None -> assert false) + | None -> + match capture ctx loc name with + | Some b -> call_value ctx ~want loc (mk loc b.bty (Tast.Local b.slot)) args + | None -> assert false) | _ when Hashtbl.mem ctx.env.gsigs name -> let vars, params, ret = Hashtbl.find ctx.env.gsigs name in generic_call ctx ~want loc name vars params ret args @@ -8548,7 +9044,8 @@ and generic_call ctx ~want loc name vars pats pret args = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> mentions v e | Types.Map (k, w) -> mentions v k || mentions v w - | Types.Fn (ps, r) -> List.exists (mentions v) ps || mentions v r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists (mentions v) ps || mentions v r | _ -> false in let bound_exactly v = @@ -8649,7 +9146,9 @@ and generic_call ctx ~want loc name vars pats pret args = true) | _ -> false in - if (not handled) && not (bind_ty subst p a.Tast.ty) then + (* [~widen]: this is the top of an argument's type, which is the one + place a widening thunk can be built around it. See [bind_ty]. *) + if (not handled) && not (bind_ty ~widen:true subst p a.Tast.ty) then fail a.Tast.loc "%s expects %s here, found %s" name (Types.to_string p) (Types.to_string a.Tast.ty); a) @@ -8698,7 +9197,38 @@ and generic_call ctx ~want loc name vars pats pret args = && Types.widens_to ~from:a.Tast.ty ~into:f -> widen a.Tast.loc f a | _ -> a) - | _ -> a) + (* And the other widening, for the same reason and at the same + moment: a [CFn] argument against an [(Fn [$t] $t)] parameter. + A parameter that still mentioned a variable was checked with no + expectation at all — there was nothing to expect until the + argument had spoken — so [expect] never saw the pair and never + built the value the instance's signature needs. It is built here, + once the binding is final, exactly as the numeric catch-up above + is. + + A *concrete* [Fn] parameter never reaches this: it was checked + with a want in the first pass and [expect] widened it there. + + The arm is total over the pair, and that is the point of writing + it as an [if] rather than as a guard. [bind_ty]'s fallthrough is + [Types.fits], which admits [Never] where the instance's signature + wants a type — so a binding can succeed over a pair these two + words cannot bridge, and a fallthrough of "hand the argument over + unchanged" would pass one word where the instance declares two. + Every [CFn] arriving at an [Fn] parameter either gets its thunk + here or gets the refusal, which is the answer [expect] gives a + call with no type variables in it. *) + | _ -> + (match subst_ty !subst pat, a.Tast.ty with + | Types.Fn (ps, r), Types.CFn (ps', r') -> + if Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) then + mk a.Tast.loc (Types.Fn (ps, r)) + (Tast.Thicken (thick_thunk ctx.env a.Tast.loc ps r, a)) + else + fail a.Tast.loc "%s expects %s here, found %s" name + (Types.to_string (Types.Fn (ps, r))) + (Types.to_string a.Tast.ty) + | _ -> a)) pats targs in (* **A type variable is not instantiated at dyn.** Nothing stopped it before: @@ -8975,7 +9505,8 @@ and trial ctx f = resource failure into a wrong answer. *) let[@warning "+9"] { env = _; ret = _; slots; slot_tys; slot_names; scope; defers; defer_slot; defer_ok; defer_block; outer = _; - outer_what; in_frames; loops; tail; in_defer; + outer_what; caught; envslot; parent = _; + in_frames; loops; tail; in_defer; owner = _ } = ctx in match f () with | r -> Ok r @@ -8985,6 +9516,7 @@ and trial ctx f = ctx.defers <- defers; ctx.defer_slot <- defer_slot; ctx.defer_ok <- defer_ok; ctx.defer_block <- defer_block; ctx.outer_what <- outer_what; ctx.in_frames <- in_frames; + ctx.caught <- caught; ctx.envslot <- envslot; ctx.loops <- loops; ctx.tail <- tail; ctx.in_defer <- in_defer; Error d @@ -9720,6 +10252,25 @@ let collect env (decls : Ast.decl list) = "%s of %s is dyn, which does not cross to C — take the value \ at a written type and pass that" what fn.Ast.name + (* Its own arm too, because "pass (Ptr T)" is nonsense for a + function and the real objection is worth stating. A [CFn] is + one word and is the right *shape* for a C callback — that is + what it is for — but a Flan function's emitted signature still + ends with the transfer channel, and a C caller knows nothing + about one. So the address is not a C function pointer yet, and + what would make it one is dropping the channel from a signature + that cannot transfer (FIX.org, 2026-09-21). A [Fn] is two words + and is not even the right shape. *) + | Types.CFn _ | Types.Fn _ -> + fail loc + "%s of %s is %s, and a Flan function's address is not a C \ + function pointer yet — not even a CFn's. Its signature ends \ + with the transfer channel, and a C caller knows nothing \ + about one; the C in CFn is about having no environment, \ + which is what a C function pointer would need, and not about \ + crossing today. Write the callback in C, or give the binding \ + a (Ptr ()) and let the shim pass C's own" + what fn.Ast.name (Types.to_string t) | _ -> fail loc "%s of %s is %s, which cannot cross to C directly — pass \ @@ -10126,7 +10677,7 @@ let rec check_fn env (fn : Ast.fn) : Tast.fn = (match ctx.defer_slot with | None -> ctx.defers | Some s -> guarded_defers s ctx.defers); - fparent = None; floc = fn.Ast.nloc } + fenv = None; fparent = None; floc = fn.Ast.nloc } (* The generic body, checked once with its variables abstract. Nothing is kept — the [Tast.fn] it produces is thrown away, and so is anything it lifted — @@ -10420,7 +10971,7 @@ let lift_ginit ctx loc n ty (v : Tast.expr) = (match ctx.defer_slot with | None -> ctx.defers | Some s -> guarded_defers s ctx.defers); - fparent = Some n; floc = loc } + fenv = None; fparent = Some n; floc = loc } :: ctx.env.lifted; { Tast.e = Tast.Call (fname, []); ty; loc } @@ -11085,6 +11636,240 @@ let dyn_descriptors (p : Tast.program) = "What C hands back points at storage this compiler never rooted") p.Tast.externs; value_sites p (fun ~slot:_ loc what t -> check loc what t) +(* ── Escape, which is the other half of capture ──────────────────────── + spec-memory.md's case 2 is the *non-escaping* fn, and this is what makes + the word mean something. A captured copy lives in a slot of the frame the + literal was written in, so a value holding that frame's address may be + called, passed down and copied about as much as anyone likes — and must + never outlive the frame. Case 3, the escaping closure with an environment + the collector allocates, is a separate lane; every refusal here names it, + because "this cannot be done" and "this cannot be done yet" are different + sentences and the second one is the true one. + + Run over the typed IR rather than over the surface, and over every + function the program ends up with rather than only over the ones anyone + wrote. Two reasons, and both are about not having to be careful: the IR + has one node per way a value can be stored, so the list below is closed; + and a lifted body is checked by exactly the same pass as the body it came + out of, so an [fn] inside an [fn] needs no special case. + + **What is suspect.** A value of function type that may carry an + environment, decided by a rule that needs no interprocedural anything: + + - a capturing literal, which is a [Closure] node and is the only place one + is made; + - a *parameter* of function type, in every function, because nothing at a + definition can see what its callers will pass; + - a local bound to either of those, transitively; + - a branch or a block whose value is one. + + Everything else of function type is clean: the address of a name, and the + result of any call — the second follows from the first refusal below, which + is what stops a function from returning a suspect in the first place. + + **Why the parameter rule is the whole answer to the hard case.** An [fn] + that captures, passed to a function that stores it, is caught *inside that + function*: its parameter is suspect there, and the store is refused where + it is written. So no call can leak what its caller passed, and no caller + has to be analysed. What it costs is real and worth naming: [(defn id [f + (Fn [] i32)] (Fn [] i32) f)] is refused although it is harmless, and so is + holding a parameter of function type in a Vec that never leaves the frame. + Both become writable when case 3 lands, and neither is worth an analysis + before then. + + **Where a suspect may stand**: an argument of a call, the callee of one, a + [let] binding, and the field of an environment another literal captures it + into — that last is the [env/] exemption below, and it is sound for the + same reason everything here is: the outer literal is itself suspect, so + the pair of environments lives and dies with one frame. *) +let rec escaping suspects (e : Tast.expr) = + (* The value of a body is its last form, which is the only part of one that + can be this expression's own value. *) + let tail body = + match List.rev body with x :: _ -> escaping suspects x | [] -> false + in + match e.Tast.ty with + | Types.Fn _ -> + (match e.Tast.e with + | Tast.Closure _ -> true + | Tast.Local s -> List.mem s !suspects + | Tast.If (_, a, b) -> escaping suspects a || escaping suspects b + | Tast.Do body | Tast.Let (_, body) -> tail body + | Tast.Match (_, arms) -> List.exists (fun (a : Tast.arm) -> tail a.Tast.abody) arms + (* The forms that establish something around a body and yield the body's + value. Easy to forget and not safe to: each of them is an expression, + so each of them is a way for a suspect to be the answer. A + restart-case yields its body's value *or* a clause's, so every clause + is a tail too. *) + | Tast.Handled (_, body) | Tast.WithAlloc (_, body) -> tail body + | Tast.RestartCase (cs, body) -> + escaping suspects body + || List.exists (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs + (* And then the clean list, which is short, closed, and stated as a list + rather than as a default — a *read* of an [Fn] from anywhere is + suspect unless it is one of these. + + It used to be the other way round, with [Field], [CaseField] and + [Deref] named as suspect and everything else clean, and that spelling + had a hole in it exactly where a list like this cannot: an [(at s 0)] + over a slice of [Fn] is a [Prim], so it read as clean while the [Vec] + and pointer spellings of the same thing were refused. Nothing can + write an [Fn] into a slice today, so it was not reachable — but the + header above claims this enumeration is closed, and a default of + [false] is how that claim stops being true without anyone noticing. + + What is clean, and why. The address of a name never carried an + environment. A widening carries a thunk and a code pointer, which is + not a frame address. And the result of a call cannot carry one, + because a function that would return one is refused below — that + refusal is what this arm rests on, which is why the two have to be + read together. *) + | Tast.FnAddr _ | Tast.Thicken _ | Tast.Call _ | Tast.CallPtr _ -> false + (* Everything else that can produce an [Fn]: read out of a struct — which + for a function value means read out of an *environment*, the one + aggregate a capture may be written into — out of a case, through a + pointer, or out of a container. A copy of a captured function value + carries whatever environment the original did, so it is suspect + exactly as the original was: without this, a lifted body could hand + back its copy of a captured value and the result would arrive at the + caller as an ordinary call result, which is to say as clean. *) + | _ -> true) + | _ -> false + +(* The environment struct a capture built, which is the one aggregate a + suspect may be written into. See the header. *) +let is_env_struct n = String.length n >= 4 && String.sub n 0 4 = "env/" + +let escape_check (fn : Tast.fn) = + let suspects = ref [] in + List.iteri + (fun i ty -> match ty with Types.Fn _ -> suspects := i :: !suspects | _ -> ()) + fn.Tast.params; + (* Two refusals, because the two cases know different amounts. A literal + written here captures, full stop, and the message can name the frame its + copies are on. A function value that arrived as a parameter *may* carry + an environment and nothing at a definition can tell — so it is refused + where it is written rather than at the calls that would have been fine, + and the message says that is what happened. *) + let owner = match fn.Tast.fparent with Some p -> p | None -> fn.Tast.name in + (* Which of the two this is, found by following the same tails [escaping] + followed. The refused expression is often a form that *yields* the + suspect — a handler-bind, a branch — and the message has to describe what + is actually escaping and not the shape it arrived in. *) + let rec written_here (e : Tast.expr) = + let tail body = + match List.rev body with x :: _ -> written_here x | [] -> true + in + match e.Tast.e with + | Tast.Closure _ -> true + | Tast.If (_, a, b) -> written_here a && written_here b + | Tast.Do body | Tast.Let (_, body) | Tast.Handled (_, body) + | Tast.WithAlloc (_, body) -> tail body + | Tast.Match (_, arms) -> + List.for_all (fun (a : Tast.arm) -> tail a.Tast.abody) arms + | Tast.RestartCase (cs, body) -> + written_here body + && List.for_all (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs + (* Everything else is a *read* of a value made elsewhere — a local, a + field, an index, a load through a pointer — and the message that fits + one is the other message, about a value this definition did not make + and cannot see into. The literal is the short list here, exactly as + the clean set is the short list in [escaping]; whichever is short is + the one to write out. *) + | _ -> false + in + let refuse (e : Tast.expr) where = + if not (written_here e) then + Loc.failk "check/fn-escapes" e.Tast.loc + "this function value may carry an environment, and %s would outlive \ + the frame that environment is on. A value reaching %s as a parameter \ + was made by a caller this definition cannot see, so it is refused \ + here rather than at the calls that would be safe. Call it, pass it \ + down, or hold it in a let — an fn whose environment the collector \ + owns is spec-memory.md's case 3 and is not built yet" + where owner + else + Loc.failk "check/fn-escapes" e.Tast.loc + (* No "hold it in a let" here, unlike the message above: an fn literal + takes its types from the position it is written in, so there is no + let binding to offer — see fn-no-type.flan. Every suggestion this + compiler prints has to compile. *) + "this fn captures, and %s would outlive the frame its copies are on. \ + The copies are slots of %s, taken where the value was made, so a \ + reader reached after that frame has gone would read whatever \ + replaced them. Call it, or pass it down — an fn whose environment \ + the collector owns is spec-memory.md's case 3 and is not built yet" + where owner + in + let deny where es = List.iter (fun e -> if escaping suspects e then refuse e where) es in + let go (e : Tast.expr) = + (match e.Tast.e with + | Tast.Let (bs, _) -> + (* A binding is where a suspect spreads, and one of the two places it + does. *) + List.iter + (fun (slot, v) -> if escaping suspects v then suspects := slot :: !suspects) + bs + (* And the other: an arm's pattern binds the case's fields to slots, and + the store that fills them is inside the branch rather than in a form + this walk reads as a binding. Reading the same field by hand is a + [CaseField] and suspect — a copy of a captured function value carries + whatever environment the original did — so the slot the pattern binds + it to is suspect too, or [(match o (Some f) f ...)] would hand back + through a name what [(case-field o ...)] cannot hand back at all. + + Every arm's binds, not only an [Option]'s: a data type's field of + function type is written through [MakeCase], which denies suspects, + and read back through this. *) + | Tast.Match (_, arms) -> + List.iter + (fun (a : Tast.arm) -> + List.iter + (fun s -> + match fn.Tast.slots.(s) with + | Types.Fn _ -> suspects := s :: !suspects + | _ -> ()) + a.Tast.binds) + arms + | Tast.Set (_, v) -> deny "a store" [ v ] + | Tast.Return (Some v) -> deny "a return" [ v ] + | Tast.Some_ v -> deny "an Option" [ v ] + | Tast.Arr es -> deny "a fixed array" es + | Tast.MakeCase (_, _, es) -> deny "a data type's field" es + | Tast.Make (n, es) -> if not (is_env_struct n) then deny "a struct field" es + | Tast.Addr (Tast.Plocal s) -> + if List.mem s !suspects then + refuse e "a pointer to it" + (* Everything the runtime takes: a push into a Vec, a put into a Map, a + box into a dyn. All of them put the value somewhere this frame does + not own — and all of them take it *by address*, because the container + runtime is type-erased, so the address is what has to be caught and + not the value beside it. *) + | Tast.Prim (Tast.Rt _, es) -> + deny "a container" es; + List.iter + (fun (a : Tast.expr) -> + match a.Tast.e with + | Tast.Prim (Tast.AddrOf, [ v ]) -> deny "a container" [ v ] + | _ -> ()) + es + | Tast.Prim (Tast.AddrOf, [ v ]) -> deny "a pointer to it" [ v ] + | Tast.InvokeRestart (_, _, es, _, _, _) -> deny "a restart's argument" es + | _ -> ()) + in + (* Outermost first, which is the order [Tast.walk] gives and the order a + [let] has to be seen in: a binding must be recorded before anything that + reads the slot. *) + List.iter (fun e -> Tast.walk go e) fn.Tast.body; + (* And the defers on the transfer path, which are the same forms again but + are not reachable from [body] — they hang off the function, and a store + written in one is a store. *) + List.iter (fun e -> Tast.walk go e) fn.Tast.fdefers; + (* And the tail, which is a return with nothing written. *) + (match List.rev fn.Tast.body with + | last :: _ when (match fn.Tast.ret with Types.Fn _ -> true | _ -> false) -> + deny "a return" [ last ] + | _ -> ()) let build_program ~keep_going (decls : Ast.decl list) : Tast.program * env = let env = new_env () in @@ -11169,6 +11954,10 @@ let build_program ~keep_going (decls : Ast.decl list) : Tast.program * env = they are reached *by name* from arbitrary call sites, so they carry no [fparent] and a dev build gives each its own cell. *) let fns = fns @ List.rev env.instances in + (* Where a captured copy may go, asked of every function the program ended + up with. Here rather than inside [check] because it is a question about a + finished body — see the header on [escaping]. *) + List.iter escape_check fns; (* And the order the computed initialisers run in, which needs the whole function list: what a global reads is transitive through what it calls. *) let globals = init_order globals fns in diff --git a/lib/cimport.ml b/lib/cimport.ml index 6a7f1da6..3c9b8f95 100644 --- a/lib/cimport.ml +++ b/lib/cimport.ml @@ -413,8 +413,8 @@ let rec ty_source (t : Ast.texpr) = | Ast.Tarray (Ast.Lname n, e) -> Printf.sprintf "[%s %s]" n (ty_source e) | Ast.Tmap (k, v) -> Printf.sprintf "(Map %s %s)" (ty_source k) (ty_source v) - | Ast.Tfn (ps, r) -> - Printf.sprintf "(Fn [%s] %s)" + | Ast.Tfn (env, ps, r) -> + Printf.sprintf "(%s [%s] %s)" (if env then "Fn" else "CFn") (String.concat " " (List.map ty_source ps)) (ty_source r) let tname n = ty (Ast.Tname n) diff --git a/lib/dev.ml b/lib/dev.ml index 1a64e9c0..fbafd1f1 100644 --- a/lib/dev.ml +++ b/lib/dev.ml @@ -2605,7 +2605,7 @@ let render_addr (s : Session.t) ~addr ~(ty : Types.t) let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: what is being shown is storage this thunk reaches by address. *) diff --git a/lib/emit.ml b/lib/emit.ml index 215cfb03..8181c726 100644 --- a/lib/emit.ml +++ b/lib/emit.ml @@ -85,6 +85,38 @@ let cellname n = "@" ^ quoted (Mangle.cell n) written only by the guards this file emits. *) let xfer_param = "%xfer" +(* The environment parameter, the other name that is not a Flan name: the + address of the captured copies a function value was made with. + + **It is declared by exactly the bodies that can be reached through a + [(Fn ...)] value**, and it is the *last* parameter, after the transfer + channel. That set is: a lifted [fn] literal written into an [Fn] position, + capturing or not; every handler clause, because [flan_signal] passes one + to whichever clause matched and cannot know which of them captured; and + the widening thunks ([Tast.Thicken]), which exist to read it. + + Nothing else declares it. An ordinary [defn] therefore emits exactly the + signature it always did — its parameters and then the channel, and not a + byte more — and a call to it by name is unchanged. That is the whole of + what keeps capture free for everyone who does not use it, and it is the + author's ruling: the static side does not pay for the dynamic side. + + So **every indirect call is exactly typed**. The two conventions meet in + one place, the thunk, and nowhere does a caller pass an argument the callee + did not declare. An earlier design did rely on that — the environment last, + ignored by a body that never asked for it, which SysV allows and Swift's + thin-vs-thick convention is built on — and wasm32 killed it: [call_indirect] + compares the signature at the call site, so a spare argument is a trap and + not a register nobody reads. Being exactly typed is checkable by a verifier + rather than argued from a calling convention, which is the better property + to have had all along. *) +let env_param = "%env" + +(* What a call through a [(Fn ...)] value passes when it has no environment — + a value made out of a name, or one widened from a [CFn]. Spelled once so + the sites cannot drift. *) +let no_env = "ptr null" + (* The condition's own name, for the message an unhandled [error] prints. The checker has already refused anything that is not a struct. *) let struct_name_of (t : Types.t) = @@ -130,10 +162,13 @@ module Rt = struct let ll_of = function Ptr -> "ptr" | I32 -> "i32" | I64 -> "i64" let size_of = function Ptr | I64 -> 8 | I32 -> 4 - (* A handler frame: the one it displaced, the condition type it matches, and - the lifted function that runs. *) + (* A handler frame: the one it displaced, the condition type it matches, the + lifted function that runs, and the environment that function is handed — + the establishing function's captured copies, or null when the clause + captured nothing. *) let handler = - { sname = "handler"; fields = [ "prev", Ptr; "type", I32; "fn", Ptr ] } + { sname = "handler"; + fields = [ "prev", Ptr; "type", I32; "fn", Ptr; "env", Ptr ] } (* A restart frame. The first four fields are what the runtime's own [flan_restart] declares and their offsets do not move; the rest are §3's @@ -246,11 +281,13 @@ let rec ll (t : Types.t) = (* An [Allocator] is a pointer to the runtime's [flan_allocator] and never a copy of one: see Types. Opaque here in the same sense [ptr] is. *) | Types.Alloc -> "ptr" - (* A function value is a code address and nothing else. There is no - environment beside it — capture does not exist (check.ml refuses it by - name) — so it is one pointer, the same width as any other, and a backend - needs to know no more about it than that. *) - | Types.Fn _ -> "ptr" + (* A code address and the environment it is called with: two words, always, + whether or not this particular value captured anything. See [%fnv]. *) + | Types.Fn _ -> "%fnv" + (* The bare address, and nothing beside it: one pointer, the width of any + other. A [CFn] cannot capture, so there is nothing an environment + would hold. *) + | Types.CFn _ -> "ptr" (* ptr + len + cap + allocator, and two more words the runtime owns: see flan_rt.c's (Vec T) header for why they are in every build. Nothing in this file reads a field of one — every operation is a runtime call taking @@ -454,7 +491,8 @@ let rec lay m (t : Types.t) : int * int = | Types.Enum _ -> 4, 4 | Types.Ptr _ -> 8, 8 | Types.Alloc -> 8, 8 - | Types.Fn _ -> 8, 8 + | Types.Fn _ -> 16, 8 + | Types.CFn _ -> 8, 8 | Types.Vec _ | Types.Map _ -> 40, 8 (* [n x T] adds no padding of its own: T's size already carries its tail. *) | Types.Array (n, e) -> let s, a = lay m e in Int64.to_int n * s, a @@ -813,13 +851,26 @@ let rec dty m d (t : Types.t) : int = ("len", Types.Int Types.I64); ("log2cap", Types.Int Types.I64); ("allocator", Types.Alloc); ("epoch", Types.Int Types.I64) ] |> fun n -> ignore k; ignore v; n - (* A pointer to code, and lldb is told exactly that and no more. DWARF - has DW_TAG_subroutine_type for the signature behind it, and spelling - one out here would buy a reader nothing they cannot get from the - function it points at — [p f] answers with an address either way, and - the address is what resolves to a symbol. The name carries the - signature, which is where it is actually legible. *) + (* Two words, and shown as two, the same rule the Vec and the Map above + follow: a debugger told a function value were one pointer would put + every offset after it out by eight. [code] is the address that + resolves to a symbol, which is what [p f] was ever worth; [env] is + the captured copies, and there is nothing here that could say what is + in them — the environment is a struct the checker synthesised for one + literal, and DWARF for it would describe a type the program cannot + name. A reader who wants the copies asks the break loop for the + locals, where they are under the names the source gave them. *) | Types.Fn _ -> + composite (Types.to_string t) + [ ("code", Types.Ptr Types.Unit); ("env", Types.Ptr Types.Unit) ] + (* And the bare one is what it always was: a pointer to code, and lldb + is told exactly that and no more. DWARF has DW_TAG_subroutine_type + for the signature behind it, and spelling one out would buy a reader + nothing they cannot get from the function it points at — [p f] + answers with an address either way, and the address is what resolves + to a symbol. The name carries the signature, which is where it is + actually legible. *) + | Types.CFn _ -> dnode d (Printf.sprintf "!DIDerivedType(tag: DW_TAG_pointer_type, name: \"%s\", \ @@ -1763,19 +1814,41 @@ and value_at f (e : Tast.expr) : string = | Tast.Local _ | Tast.Global _ | Tast.Field _ | Tast.Deref _ -> (* Everything that denotes a location is a load from its address. *) load f (addr f e) e.Tast.ty - (* The symbol itself, not a load from it: a function's address is a link-time - constant. The same spelling the handler frames use for a lifted clause. *) - | Tast.FnAddr (Tast.Flanfn n) -> fname n - | Tast.FnAddr (Tast.Rtfn n) -> "@" ^ n - (* A function value someone wrote, which is the one [FnAddr] that is not the - symbol. In a dev build it is the cell's contents, so that a value taken - after a redefinition is the new body — the same load a direct call to the - same name would do, at the point the *address* is taken rather than at the - call. What that does not give is a value taken before a redefinition and - called after it: that one is still the old body, because there is nothing - left to re-resolve once the address is in a slot. Named in docs/BUILT.md rather - than papered over with a trampoline. *) - | Tast.FnAddr (Tast.Fnval n) -> body_of f n + (* A [(Fn ...)] value, which is two words: a code address and the + environment it is called with. A value made out of a name captures + nothing, so the second word is null and [zeroinitializer] has already put + it there. See [%fnv]. + + Only a [Fn]-typed one. The same three [fnref] constructors are also asked + for as bare addresses — carrying [CFn], and carrying [Alloc] for the + map's hash and equality pair and a handler frame's clause, which are + fields of structs the runtime declares — and those stay one word. The + node's type is what says which is being asked for. *) + | (Tast.FnAddr _ | Tast.Closure _ | Tast.Thicken _) + when (match e.Tast.ty with Types.Fn _ -> true | _ -> false) -> + let code, env = + match e.Tast.e with + | Tast.FnAddr r -> fnaddr f r, "null" + | Tast.Closure (r, env) -> fnaddr f r, value f env + (* The widening: the thunk's code, with the bare address stored where + an environment would be. The thunk reads it back out and calls it, + which is what keeps every indirect call exactly typed. *) + | Tast.Thicken (n, p) -> fname n, value f p + | _ -> assert false + in + let a = fresh f in + ins f "%s = insertvalue %%fnv zeroinitializer, ptr %s, 0" a code; + if String.equal env "null" then a + else begin + let b = fresh f in + ins f "%s = insertvalue %%fnv %s, ptr %s, 1" b a env; + b + end + | Tast.FnAddr r -> fnaddr f r + | Tast.Closure _ | Tast.Thicken _ -> + (* Unreachable: both are [Fn] values and the arm above has already taken + every [Fn]-typed node. Here because nothing else could be meant. *) + failwith "a closure is a function value" | Tast.Addr p -> fst (place f p) | Tast.Prim (p, args) -> prim f e p args | Tast.Call (name, args) -> @@ -2192,14 +2265,57 @@ and call f ret flan args = the middle of an argument list). *) and call_ptr f ret callee args = let c = value f callee in + (* A [(Fn ...)] is two words and both are taken before the arguments are + evaluated: an argument may itself make a function value, and the two + halves of *this* one have to come out of the same value. A + [(CFn ...)] is the address alone, and the call that follows is the + call a name would have produced. *) + let code, env = + match callee.Tast.ty with + | Types.Fn _ -> + let code = fresh f in + ins f "%s = extractvalue %%fnv %s, 0" code c; + let env = fresh f in + ins f "%s = extractvalue %%fnv %s, 1" env c; + code, Some ("ptr " ^ env) + | _ -> c, None + in let vs = map_lr (fun (a : Tast.expr) -> let v = value f a in Printf.sprintf "%s %s" (ll a.Tast.ty) v) args in - call_through f ret c vs + call_through f ?env ret code vs -and call_through f ret callee vs = +(* The code address behind one of the three [fnref]s, which is the same string + whether it is wanted as a bare [Alloc] pointer or as the first word of a + function value. + + [Flanfn] and [Rtfn] are the symbol itself, not a load from it: a function's + address is a link-time constant. [Fnval] is the one that is not — in a dev + build it is the cell's contents, so that a value taken after a redefinition + is the new body, the same load a direct call to the same name would do, at + the point the *address* is taken rather than at the call. What that does + not give is a value taken before a redefinition and called after it: that + one is still the old body, because there is nothing left to re-resolve once + the address is in a slot. Named in docs/BUILT.md rather than papered over + with a trampoline. *) +and fnaddr f (r : Tast.fnref) = + match r with + | Tast.Flanfn n -> fname n + | Tast.Rtfn n -> "@" ^ n + | Tast.Fnval n -> body_of f n + +(* [env] is present on exactly one kind of call: one through a [(Fn ...)] + value, which cannot know whether the body it reaches declared one. Every + other call — by name, through a [(CFn ...)] — passes what it always + passed. See [env_param] for why appending it is safe when the callee did + not ask for it. *) +and call_through f ?env ret callee vs = let t = fresh f in - ins f "%s = call %s %s(%s)" t (ll ret) callee - (String.concat ", " (vs @ [ "ptr " ^ xfer_param ])); + let tail = + match env with + | None -> [ "ptr " ^ xfer_param ] + | Some e -> [ "ptr " ^ xfer_param; e ] + in + ins f "%s = call %s %s(%s)" t (ll ret) callee (String.concat ", " (vs @ tail)); guard f; (* An aggregate with a dyn in it is spilled into a rooted slot the instant it arrives, the same move a dyn word gets in [prim] and for a sharper reason: @@ -2293,6 +2409,16 @@ and emit_handled f frames body = stack finds what it pushed still valid, which is what "old code is never unloaded" means. See NEXT.md, conditions step 1. *) ins f "store ptr %s, ptr %s" (fname h.Tast.hfn) fp; + (* And the environment the clause is called with, which is a pointer + into this very frame. Written unconditionally — null when the + clause captured nothing — because a frame the runtime reads a + field of must have every field written, not only the ones this + clause happens to use. *) + let ep = fresh f in + ins f "%s = getelementptr inbounds %%handler, ptr %s, i32 0, i32 3" + ep slot; + ins f "store ptr %s, ptr %s" + (match h.Tast.henv with Some e -> value f e | None -> "null") ep; ins f "call void @flan_handler_push(ptr %s)" slot; slot) frames @@ -3136,6 +3262,14 @@ let signature ~named (fn : Tast.fn) = analysis is an optimisation, and in a dev build a cell can hold anything, so the honest answer to "what can this call?" is "anything". *) let params = params @ [ (if named then "ptr " ^ xfer_param else "ptr") ] in + (* And the environment, last, and only on a body that can be reached + through an [Fn] value: see [env_param]. Everything else emits the + signature it always did. *) + let params = + match fn.Tast.fenv with + | None -> params + | Some _ -> params @ [ (if named then "ptr " ^ env_param else "ptr") ] + in Printf.sprintf "%s %s(%s)" (ll fn.Tast.ret) (fname fn.Tast.name) (String.concat ", " params) @@ -3205,6 +3339,14 @@ let emit_fn m ?(hidden = false) ?(pnames = []) (fn : Tast.fn) = Buffer.add_string f.allocas (Printf.sprintf " store %s %%p%d, ptr %s\n" (ll ty) i f.slots.(i))) fn.Tast.params; + (* And the environment, on the one kind of function that has one. Every + other function is handed it too and never reads it; there is no slot for + it there and nothing to store. *) + (match fn.Tast.fenv with + | Some slot -> + Buffer.add_string f.allocas + (Printf.sprintf " store ptr %s, ptr %s\n" env_param f.slots.(slot)) + | None -> ()); (* The dyn roots, and this is not gated on [m.dev]: the shadow stack below is a debugging convenience and a release build does without it, while a collector that cannot find its roots is a collector that frees live @@ -3709,7 +3851,7 @@ let emit_startup m ?(hidden = false) (globals : Tast.global list) = List.iter (emit_global m ~hidden) flags; emit_fn m ~hidden { Tast.name = ".init-globals"; params = []; slots = [||]; snames = [||]; - ret = Types.Unit; body; fdefers = []; fparent = None; + ret = Types.Unit; body; fdefers = []; fenv = None; fparent = None; floc = (List.hd computed).Tast.ginit.Tast.loc }; true @@ -3758,6 +3900,13 @@ let header = {|; Generated by flan. The layout is C's: no object headers anywher ; so a Flan struct is exactly its C struct and nothing marshals. %slice = type { ptr, i64 } +; A function value: the code address, and the environment the captured copies +; live in. Two words rather than one because the environment has to travel +; *with* the value — a callee that takes a (Fn [T] R) and calls it knows +; nothing about where the value came from, so there is nowhere else to put it. +; A value that captures nothing carries a null there and every call passes it +; on regardless; see [env_param]. +%fnv = type { ptr, ptr } ; (Vec T), spec-memory.md. The element type is nowhere in it: the runtime is ; type-erased and every operation is handed size and align at its call site. %vec = type { ptr, i64, i64, ptr, i64 } @@ -3765,8 +3914,9 @@ let header = {|; Generated by flan. The layout is C's: no object headers anywher ; nor value type appears in it, for the same reason: one type-erased runtime, ; handed the two sizes and a hash/equality pair at each call site. %map = type { ptr, i64, i64, ptr, i64 } -; A handler frame: the one it displaced, the condition type it matches, and -; the lifted function that runs. Allocated on the establishing frame's stack. +; A handler frame: the one it displaced, the condition type it matches, the +; lifted function that runs, and the environment that function is handed. +; Allocated on the establishing frame's stack. |} ^ Rt.ll_type Rt.handler ^ {| ; A restart frame: the one it displaced and the name it offers. There is no ; target field, because the frame's own address *is* the target — which makes @@ -4521,11 +4671,18 @@ let redefinition ?(checks = true) ?(dev = false) ?(debug = false) (* A clause lifted out of one of these comes with it: its body may have changed too, and it is reached by address from inside the module rather than through a cell. Every other lifted clause is invisible here — it - needs no declaration, since nothing in this module names it. *) + needs no declaration, since nothing in this module names it. + + And the widening thunks, every one of them, whichever body they belong + to: a module that hands a name to an [Fn]-typed parameter names one, and + the host has no cell for it to be reached through. They are hidden and + tiny, so a copy per module is the whole cost — and the alternative is an + undefined symbol at dlopen, which is the shape of bug [Fnval] was. *) let lifted = List.filter (fun (f : Tast.fn) -> match f.Tast.fparent with + | Some "" -> true | Some p -> List.mem p fns | None -> false) p.Tast.fns diff --git a/lib/js.ml b/lib/js.ml index a5fdd744..489b8bdb 100644 --- a/lib/js.ml +++ b/lib/js.ml @@ -220,7 +220,8 @@ let rec refuse_ty loc (t : Types.t) = | Types.Never | Types.Named _ | Types.Enum _ -> () | Types.Slice t | Types.Array (_, t) | Types.Option t -> refuse_ty loc t | Types.Vec t -> refuse_ty loc t - | Types.Fn (ps, r) -> List.iter (refuse_ty loc) ps; refuse_ty loc r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.iter (refuse_ty loc) ps; refuse_ty loc r | Types.Ptr _ -> at loc "(Ptr T) is not in the JS dialect — JavaScript has no addresses, so a \ @@ -698,6 +699,17 @@ let rec value f (e : Tast.expr) : string = "the runtime entry point %s has no JS counterpart — it is C in \ flan_rt.c, and this dialect has no C" n + (* A capturing fn literal. A JS function closes over its enclosing scope for + free, so this dialect would not need the environment at all — but the + environment is a struct the checker synthesised and the captured copies + are read out of it by index, which is machinery this backend has nothing + to lower. Refused by name rather than emitted as a plain function that + would read the *current* value of a local instead of the copy. *) + | Tast.Closure _ | Tast.Thicken _ -> + at e.Tast.loc + "an fn that captures has no JS lowering yet — the environment is a \ + struct laid out for the two native backends, and this dialect has no \ + layout" | Tast.Prim (p, args) -> prim f e p args | Tast.Call (n, args) -> Printf.sprintf "%s(%s)" (fname n) (String.concat ", " (call_args f args)) diff --git a/lib/load.ml b/lib/load.ml index c1d3af8c..4636fd21 100644 --- a/lib/load.ml +++ b/lib/load.ml @@ -206,8 +206,9 @@ let rec rename_texpr owned alias (t : Ast.texpr) : Ast.texpr = Ast.Tmap (rename_texpr owned alias k, rename_texpr owned alias v) | Ast.Tapp (n, args) -> Ast.Tapp (n, List.map (rename_texpr owned alias) args) - | Ast.Tfn (ps, r) -> - Ast.Tfn (List.map (rename_texpr owned alias) ps, rename_texpr owned alias r) + | Ast.Tfn (env, ps, r) -> + Ast.Tfn (env, List.map (rename_texpr owned alias) ps, + rename_texpr owned alias r) in { t with Ast.t = k } @@ -746,7 +747,7 @@ let rec texpr_uses acc (t : Ast.texpr) = texpr_uses acc e | Ast.Tmap (k, v) -> texpr_uses acc k; texpr_uses acc v | Ast.Tapp (_, args) -> List.iter (texpr_uses acc) args - | Ast.Tfn (ps, r) -> List.iter (texpr_uses acc) ps; texpr_uses acc r + | Ast.Tfn (_, ps, r) -> List.iter (texpr_uses acc) ps; texpr_uses acc r let rec expr_uses acc (e : Ast.expr) = let go = expr_uses acc in diff --git a/lib/parse.ml b/lib/parse.ml index 52fb2470..4181c0a0 100644 --- a/lib/parse.ml +++ b/lib/parse.ml @@ -96,11 +96,16 @@ let rec texpr (f : Form.t) : Ast.texpr = confused with. *) | Map _ -> fail f "a map type is written (Map K V), not in braces" - | List ({ v = Sym "Fn"; _ } :: rest) -> + (* The two function types. [Fn] is the one almost every signature wants — a + value that may carry an environment — and [CFn] is the bare address, + for a C callback or a table of them. Parsed together because they differ + in one word and the refusal should name both. *) + | List ({ v = Sym (("Fn" | "CFn") as which); _ } :: rest) -> + let env = String.equal which "Fn" in (match rest with | [ { v = Vec params; _ }; ret ] -> - mk (Ast.Tfn (List.map texpr params, texpr ret)) - | _ -> fail f "a function type is (Fn [T ...] R)") + mk (Ast.Tfn (env, List.map texpr params, texpr ret)) + | _ -> fail f "a function type is (%s [T ...] R)" which) | List ({ v = Sym name; _ } :: args) when args <> [] -> mk (Ast.Tapp (name, List.map texpr args)) | _ -> fail f "expected a type, found %s" (Form.to_string f) diff --git a/lib/reach.ml b/lib/reach.ml index f7803d46..a3000a3b 100644 --- a/lib/reach.ml +++ b/lib/reach.ml @@ -50,7 +50,12 @@ let expr_refs f (e : Tast.expr) = name used as a value is never a [Call], so without the second one the one function a program passes to [map] is the one function the link drops. [Rtfn] is C in flan_rt.c and is linked whatever happens. *) - | Tast.FnAddr (Tast.Flanfn n) | Tast.FnAddr (Tast.Fnval n) -> f n + | Tast.FnAddr (Tast.Flanfn n) | Tast.FnAddr (Tast.Fnval n) + | Tast.Closure (Tast.Flanfn n, _) | Tast.Closure (Tast.Fnval n, _) + (* And the widening thunk, which is reached by address from the value + it builds and from nowhere else. Without this edge the one function + a program widens is the one function the link drops. *) + | Tast.Thicken (n, _) -> f n | Tast.Set (Tast.Pglobal n, _) | Tast.Addr (Tast.Pglobal n) -> f n | Tast.Handled (frames, _) -> List.iter (fun (h : Tast.hframe) -> f h.Tast.hfn) frames diff --git a/lib/session.ml b/lib/session.ml index 77dcf351..f6ea36f2 100644 --- a/lib/session.ml +++ b/lib/session.ml @@ -282,8 +282,16 @@ let compatible ?(origin = fun _ -> None) ?(relaxed = []) ~loc refusal is about a function the source does not name." gname ) in + (* The parameters and the return as a [defn] writes them, and not + as [(Fn [...] ...)]. That spelling was harmless while [Fn] was + the only function type and is not now: it is a real type, it is + not the same as [(CFn [...] ...)], and a *declaration* is + neither of them — rendering one as a type invites a reader to + go looking for which of the two this function's name carries, + which is a question about taking its address and not about the + edit that was refused. *) fail loc - "%s changes signature, from (Fn [%s] %s) to (Fn [%s] %s).%s \ + "%s changes signature, from [%s] %s to [%s] %s.%s \ Restart to change it." what (String.concat " " (List.map Types.to_string g.Tast.params)) @@ -373,6 +381,16 @@ let compatible ?(origin = fun _ -> None) ?(relaxed = []) ~loc new_.Tast.globals; List.iter (fun (s : Tast.structure) -> + (* An environment the checker synthesised for a capturing fn is not + subject to this rule, and that is not a loophole. The layout rule is + about values the running program is *holding*: every other struct can + be in a global, in a container, in a frame that is on the stack right + now. An environment can be in exactly one place — a slot of the frame + the literal was written in — and it is written there by the same + module that reads it, on every entry. So editing which locals an fn + names is an ordinary body change, and demanding a restart for it + would take the dev loop away from the feature it was built for. *) + if Check.is_env_struct s.Tast.sname then () else match List.find_opt (fun (r : Tast.structure) -> String.equal r.Tast.sname s.Tast.sname) @@ -992,7 +1010,7 @@ let eval ?(origin = "") ?pause t src : change = Some { Tast.name = Printf.sprintf "install/%d" t.thunks; params = []; ret = Types.Unit; body; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = [||]; snames = [||] } in let ir = @@ -1319,7 +1337,7 @@ let render_locals ?(origin = "") t ~frame ~(fn : Tast.fn) ~bound let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: the locals being shown are the *other* frame's, and this thunk reaches them by address. *) @@ -1408,7 +1426,7 @@ let render_condition t ~(st : Tast.structure) : change * (string * string) list let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); snames = Array.make (List.length !extra) None } in @@ -1662,7 +1680,7 @@ let render_slot ?(origin = "") t ~frame ~(fn : Tast.fn) ~slot ~path { Tast.name = tname; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); snames = Array.make (List.length !extra) None } in @@ -1932,7 +1950,7 @@ let write_slot ?(origin = "") t ~frame ~(fn : Tast.fn) ~slot ~path stores @ (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.append base (Array.of_list (List.rev !extra)); (* The stored expressions' own [let]s keep their names; the slots [render] added behind them are the walk's own @@ -2030,7 +2048,7 @@ let render_globals ?(origin = "") t ~(globals : Tast.global list) let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: what is being shown is the program's storage, which this thunk reaches by name. *) @@ -2118,7 +2136,7 @@ let eval_expr ?(origin = "") ?(pause = false) t src : change = t.thunks <- t.thunks + 1; let name = Printf.sprintf "eval/%d" t.thunks in let thunk : Tast.fn = - { Tast.name; params = []; ret = Types.Unit; body; fdefers = []; fparent = None; floc = loc; + { Tast.name; params = []; ret = Types.Unit; body; fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.append base (Array.of_list (List.rev !extra)); (* The expression's own [let]s keep their names; the slots [render] added behind them are the walk's own scratch and have none to keep. *) diff --git a/lib/tast.ml b/lib/tast.ml index c2ff3feb..deec3a91 100644 --- a/lib/tast.ml +++ b/lib/tast.ml @@ -112,6 +112,42 @@ and expr_kind = dev build is not the symbol but whatever the indirection cell holds, and carries the Flan type [Fn]. *) | FnAddr of fnref + (* A function value with an environment: the lifted body, and the address of + the copies the enclosing frame is holding for it. The environment is a + [Make] of a struct the checker synthesised, stored into a slot of the + frame the literal was written in, so this node's second half is an + [Addr (Plocal _)] and the copies were taken where the value was made. + + Its own node rather than a field on [FnAddr] because the two answer + different questions: [FnAddr] is an address, and is asked for by three + unrelated readers that want a bare symbol ([Alloc]-typed, see [fnref]), + while this is a *value* of type [Fn] and can never be anything else. + + What stops it dangling is the checker, not this node: a value carrying an + environment may not leave the frame that owns it, so every position that + would outlive the frame is refused. spec-memory.md's case 2, and the + escaping half — an environment the collector allocates — is the case the + refusals name. *) + | Closure of fnref * expr + (* A (CFn ...) value where a (Fn ...) is wanted. The one coercion between + the two function types, and it goes this way only: there is nowhere for + an environment to go in the other direction. + + The pair it builds is {thunk, the address}: the *thunk's* code, one per + signature, with the original bare address stored where an environment + would be. The thunk reads it back out and calls it. So a value reached + through this is reached by a body that really does take an environment, + which is what keeps every indirect call exactly typed — including on + wasm32, where [call_indirect] checks the signature and an argument the + callee did not declare is a trap rather than a register nobody reads. + + The string is the thunk's name, minted and memoised by the checker: the + backends emit the pair and derive nothing. What it costs is one hop per + call, paid by a *name* handed to an [Fn]-typed parameter and by nothing + else — a literal, capturing or not, is compiled to take an environment + and needs no thunk. A signature that wants the address alone writes + [CFn] and pays nothing at all, which is what the type is for. *) + | Thicken of string * expr (* A call through a function value: the callee is an expression of type [Fn], not a name. Its own node rather than a [Call] with an expression in the name slot, because everything that walks this IR treats [Call]'s @@ -246,9 +282,16 @@ and place = | Pindex of expr * expr list | Pderef of expr -(* A pushed handler: which condition type it matches, and the lifted function - that runs when one is signalled. *) -and hframe = { htype : int; hfn : string } +(* A pushed handler: which condition type it matches, the lifted function that + runs when one is signalled, and the environment that function is handed. + + [henv] is the address of the establishing frame's copies of whatever the + clause captured, or [None] when it captured nothing. It is sound for the + same reason the frame itself is: a handler frame is popped by the body that + pushed it, so it can never be reached from outside the extent of the + function whose stack both it and the environment live on. There is no + escaping case here to defer. *) +and hframe = { htype : int; hfn : string; henv : expr option } (* A restart clause. [rname_id] is what [invoke-restart] matches by name; the body is a branch in the function that wrote it, because unlike a handler a @@ -307,6 +350,19 @@ type fn = { cell and no registry slot, and a redefinition of the parent carries its own copy. *) fparent : string option; + (* The slot the environment parameter is stored into, on a function that + was lifted out of something and captures one of its locals. Every + emitted signature takes the environment (see Emit's [env_param]) and + almost every function ignores it; this is the one that does not, and it + says where the pointer goes rather than fixing an index by convention, + because the slot is minted by [fresh_slot] like any other and a rule of + the form "the slot after the parameters" would be a second thing to keep + in step with the allocation order. + + [None] on everything anyone wrote. A capturing body reads its copies out + of this pointer once, at entry, into named slots of its own — so the + copy the value was made with is the copy the body sees. *) + fenv : int option; floc : Loc.t; } @@ -403,9 +459,10 @@ let rec walk (f : expr -> unit) (e : expr) = | Set (p, v) -> walk_place f p; go v | Addr p -> walk_place f p | Field (t, _) | Deref t | CaseField (t, _, _) | Some_ t | UnwrapSome t - | Signal (_, _, t) -> go t + | Signal (_, _, t) | Closure (_, t) | Thicken (_, t) -> go t | Match (sc, arms) -> go sc; List.iter (fun a -> gos a.abody) arms - | Handled (_, body) -> gos body + | Handled (hs, body) -> + List.iter (fun h -> Option.iter go h.henv) hs; gos body | RestartCase (cs, body) -> List.iter (fun c -> gos c.rbody) cs; go body | WithAlloc (a, body) -> go a; gos body diff --git a/lib/types.ml b/lib/types.ml index 20f026ba..2cc236e3 100644 --- a/lib/types.ml +++ b/lib/types.ml @@ -48,7 +48,50 @@ type t = at the call site, which is exactly where the two numbers are produced. *) | Vec of t | Option of t (* (Option T) *) + (* The two function types, and the difference between them is what a value + of each one *is* rather than what it may do. + + [(Fn [T ...] R)] is a code address and the environment it is called + with: two words. It is the common case and keeps the short name, because + it is what almost every higher-order signature wants — a caller may pass + it a name, a non-capturing literal, or one that captured half the frame, + and the callee neither knows nor cares. + + [(CFn [T ...] R)] is the bare address: one word, no environment, and + therefore nothing that can capture. + + **The [C] is information, not decoration.** A value with no environment + is the only kind that could ever cross to C, and under the + [--no-conditions] direction FIX.org records — where a signature that + cannot transfer drops the channel too — one becomes literally a C + function pointer. The name points at what the type *is* and at where it + is going. + + What it does **not** point at is a capability that exists now: a + [declare] cannot take a function type at all today, because a Flan + signature ends with the transfer channel and a C caller knows nothing + about one. Anyone reaching for [CFn] straight after writing a + [declare-c] is reaching too early, and [crossable] says so where they + will meet it. + + The whole of the reason there are two: a uniform environment would tax + every function in every program for a feature most of them never use, + and the static side is not to pay for the dynamic side's existence. With + two types an ordinary [defn] keeps exactly the signature it always had. + + **Nobody ever needs [CFn].** [Fn] accepts everything a [CFn] does, so + the narrow one is reached for on purpose, for one of four reasons: + handing a function to C (later, as above); a table of bare addresses; + forbidding capture at a boundary; and the one that is likeliest in + practice — a *named* function passed to an [Fn] parameter goes through + the widening thunk and pays an indirect hop per call, where a [CFn] + parameter is a direct call. [(map-in-place s double)] is the example. + + One-way: a [CFn] value satisfies an [Fn] (paired with a null + environment), and an [Fn] does not satisfy a [CFn] — there is nowhere + for the environment to go. *) | Fn of t list * t (* (Fn [T ...] R) *) + | CFn of t list * t (* (CFn [T ...] R) *) | Var of string (* a type variable — milestone 5 *) (* [dyn]: one machine word whose contents the runtime knows and this module does not. It is a written type — [(defonce x dyn 5)] boxes the 5 — and it @@ -142,7 +185,10 @@ let rec equal a b = | Alloc, Alloc -> true | Vec x, Vec y -> equal x y | Option x, Option y -> equal x y - | Fn (ps, r), Fn (ps', r') -> + (* The two are *not* equal to each other, in either direction. One-way + coercion lives in [Check.expect], where it can build the value the + wider type needs; here there is only identity. *) + | Fn (ps, r), Fn (ps', r') | CFn (ps, r), CFn (ps', r') -> List.length ps = List.length ps' && List.for_all2 equal ps ps' && equal r r' @@ -167,6 +213,9 @@ let rec to_string = function | Fn (ps, r) -> Printf.sprintf "(Fn [%s] %s)" (String.concat " " (List.map to_string ps)) (to_string r) + | CFn (ps, r) -> + Printf.sprintf "(CFn [%s] %s)" + (String.concat " " (List.map to_string ps)) (to_string r) | Var n -> n | Dyn -> "dyn" diff --git a/lib/x86.ml b/lib/x86.ml index 2bc86f38..b1c52d1c 100644 --- a/lib/x86.ml +++ b/lib/x86.ml @@ -498,9 +498,15 @@ let alignof md t = snd (Emit.lay md t) than SysV's eight. *) let is_agg (t : Types.t) = match t with + (* A [(CFn ...)] is one word and crosses exactly as a pointer does, which + is the whole of its reason for existing. *) | Types.Int _ | Types.Float _ | Types.Bool | Types.Ptr _ | Types.Enum _ - | Types.Alloc | Types.Fn _ -> false + | Types.Alloc | Types.CFn _ -> false | Types.Unit | Types.Never -> false + (* A [(Fn ...)] is two words — the code address and the environment beside + it — so it crosses the way a slice does. [Emit.lay] is the one place that + says how wide it is and this agrees with it by asking. *) + | Types.Fn _ -> true | Types.String | Types.Slice _ | Types.Array _ | Types.Map _ | Types.Vec _ | Types.Option _ | Types.Named _ -> true (* A scalar, and trivially one: runtime/flan_dyn.h says [typedef uint64_t @@ -1189,6 +1195,27 @@ let load_sym f ~dst s = end else load_int f.b ~dst ~mm:(Sym (s, 0)) ~size:8 ~signed:false +(* The code address behind one of the three [fnref]s, which is the same + sequence whether it is wanted as a bare [Alloc] pointer or as the first + word of a function value. + + [Flanfn] and [Rtfn] are the symbol itself, not a load from it: a function's + address is a link-time constant, and [Flanfn] is the spelling a lifted + handler clause is reached by. [Fnval] is the one that is not — in a release + build there is nothing to redefine and it is the symbol after all; in a dev + build it is the cell's contents, so that a value taken after a redefinition + is the new body. What that does not give — and [emit.ml] names it rather + than papering over it with a trampoline — is a value taken *before* a + redefinition and called after it. Once the address is in a slot there is + nothing left to re-resolve. *) +let fnaddr f ~reg (r : Tast.fnref) = + match r with + | Tast.Flanfn n -> addr_sym f ~dst:reg (fsym n) + | Tast.Rtfn n -> addr_sym f ~dst:reg n + | Tast.Fnval n -> + if f.md.Emit.dev then load_sym f ~dst:reg (csym n) + else addr_sym f ~dst:reg (fsym n) + let scalar_size f (t : Types.t) = match t with Types.Bool -> 1 | _ -> max 1 (sizeof f.md t) @@ -1454,6 +1481,7 @@ let agg_tmp f (ty : Types.t) = let h_size = Emit.Rt.size Emit.Rt.handler let h_type = Emit.Rt.field Emit.Rt.handler "type" let h_fn = Emit.Rt.field Emit.Rt.handler "fn" +let h_env = Emit.Rt.field Emit.Rt.handler "env" let r_size = Emit.Rt.size Emit.Rt.restart let r_field = Emit.Rt.field Emit.Rt.restart @@ -1720,27 +1748,34 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit = let l = place f p in addr_into f ~reg:rax l; store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - (* The symbol itself, not a load from it: a function's address is a - link-time constant, and this is the spelling a lifted handler clause is - reached by. [emit.ml] says the same of [Flanfn]. *) - | Tast.FnAddr (Tast.Flanfn n) -> - addr_sym f ~dst:rax (fsym n); - store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - (* A function value someone wrote, which is the one [FnAddr] that is not the - symbol. In a release build there is nothing to redefine and it is the - symbol after all; in a dev build it is the cell's contents, so that a - value taken after a redefinition is the new body. What that does not give - — and [emit.ml] names it rather than papering over it with a trampoline — - is a value taken *before* a redefinition and called after it. Once the - address is in a slot there is nothing left to re-resolve. *) - | Tast.FnAddr (Tast.Fnval n) -> - if f.md.Emit.dev then - load_sym f ~dst:rax (csym n) - else addr_sym f ~dst:rax (fsym n); - store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - | Tast.FnAddr (Tast.Rtfn n) -> - addr_sym f ~dst:rax n; + (* A [(Fn ...)] value: the code address, then the environment beside it. + Two words — see [Emit]'s %fnv. Only a [Fn]-typed node; the same three + constructors are also asked for as bare addresses, carrying [CFn] or + [Alloc], and those stay one word. The node's type says which. *) + | (Tast.FnAddr _ | Tast.Closure _ | Tast.Thicken _) + when (match t with Types.Fn _ -> true | _ -> false) -> + let env = + match e.Tast.e with + | Tast.FnAddr r -> fnaddr f ~reg:rax r; None + | Tast.Closure (r, env) -> fnaddr f ~reg:rax r; Some env + (* The widening: the thunk's code, with the bare address stored where + an environment would be. The thunk reads it back out and calls it, + which is what keeps every indirect call exactly typed. *) + | Tast.Thicken (n, p) -> addr_sym f ~dst:rax (fsym n); Some p + | _ -> assert false + in + store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8; + (match env with + | None -> xor_rr f.b ~dst:rax ~src:rax + | Some ev -> let l = eval f ev in load_loc f ~reg:rax l (Types.Ptr Types.Unit)); + store_int f.b ~src:rax ~mm:(lmem f (shift dst 8) ~scratch:r11) ~size:8 + | Tast.FnAddr r -> + fnaddr f ~reg:rax r; store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 + | Tast.Closure _ | Tast.Thicken _ -> + (* Unreachable: the arm above has taken every [Fn]-typed node, and both of + these are function values and can be nothing else. *) + unsupported "a closure that is not a function value" | Tast.Prim (p, args) -> prim f e p args dst | Tast.Call (name, args) -> (match Hashtbl.find_opt f.externs name with @@ -1752,8 +1787,17 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit = ~target:(if f.md.Emit.dev then `Cell (csym name) else `Sym (fsym name)) ~args ~rty:t dst) | Tast.CallPtr (callee, args) -> + (* Through a [(Fn ...)]: both words out of one value, the code address as + the call target and the environment beside it as the extra argument. + Through a [(CFn ...)]: the address alone, and the call that follows + is the call a name would have produced. *) let c = eval f callee in - call_flan f ~target:(`Loc c) ~args ~rty:t dst + let env = + match callee.Tast.ty with + | Types.Fn _ -> Some (Aint (shift c 8, Types.Ptr Types.Unit)) + | _ -> None + in + call_flan f ?env ~target:(`Loc c) ~args ~rty:t dst | Tast.Do body -> block f body dst t | Tast.Let (bs, body) -> List.iter @@ -1953,6 +1997,16 @@ and emit_handled f frames body dst t = name it and it lives only for this body. *) addr_sym f ~dst:rax (fsym h.Tast.hfn); store_int f.b ~src:rax ~mm:(Frame (slot + h_fn)) ~size:8; + (* And the environment the clause is called with, a pointer into this + very frame. Written unconditionally — null when the clause + captured nothing — because the runtime reads the field either + way. *) + (match h.Tast.henv with + | Some ev -> + let l = scoped f (fun () -> eval f ev) in + load_loc f ~reg:rax l (Types.Ptr Types.Unit) + | None -> xor_rr f.b ~dst:rax ~src:rax); + store_int f.b ~src:rax ~mm:(Frame (slot + h_env)) ~size:8; lea f.b ~dst:rdi ~mm:(Frame slot); xor_rr f.b ~dst:rax ~src:rax; call_sym f.b "flan_handler_push"; @@ -2812,7 +2866,7 @@ and ret_loc f = if is_agg f.fret then Lp (f.sret_off, 0) else Lf f.retval integer or SSE sequence, every aggregate by pointer, a hidden [sret] in the first integer register when the result is an aggregate, and the transfer channel last of all. *) -and call_flan f ~target ~args ~rty dst = +and call_flan f ?env ~target ~args ~rty dst = let vals = List.map (fun (a : Tast.expr) -> eval f a, a.Tast.ty) args in let callee = match target with @@ -2832,7 +2886,13 @@ and call_flan f ~target ~args ~rty dst = (* The channel is this frame's own: a callee that transfers writes through the pointer we were handed, so one cell serves the whole chain. *) let chan = [ Aint (Lf f.xfer_off, Types.Ptr Types.Unit) ] in - ignore (emit_args f (head @ body @ chan)); + (* And the environment last of all, on exactly one kind of call: one through + a [(Fn ...)] value, which cannot know whether the body it reaches + declared one. Every other call passes what it always passed — this is + where an ordinary [defn] keeps costing nothing. [Emit.env_param] is where + the position is argued. *) + let tail = match env with None -> [] | Some a -> [ a ] in + ignore (emit_args f (head @ body @ chan @ tail)); (* The cell is loaded *after* the arguments, and [emit.ml] has the same as a load-bearing comment: a redefinition that lands between two calls still must not land in the middle of one. [r11] is scratch and no argument @@ -3371,11 +3431,12 @@ let frame_bytes f = ((f.maxframe + f.outgoing + 15) / 16) * 16 (* Where each argument arrives, in the order the header lays down: a hidden [sret] first when the result is an aggregate, then the parameters, then the - transfer channel. Answers one entry per incoming value — a register number, - or a positive [rbp] displacement for the ones that came on the stack. *) + environment, then the transfer channel. Answers one entry per incoming + value — a register number, or a positive [rbp] displacement for the ones + that came on the stack. *) type incoming = Ireg of int | Isse of int | Istk of int -let incoming_of ~sret (params : Types.t list) = +let incoming_of ~sret ~env (params : Types.t list) = let ints = ref 0 and sses = ref 0 and stk = ref 0 in let next_int () = if !ints < n_int_args then (incr ints; Ireg int_args.(!ints - 1)) @@ -3395,7 +3456,17 @@ let incoming_of ~sret (params : Types.t list) = else next_int ()) params in - sret_at, ps, next_int () + (* Left to right, and the two [next_int ()] calls must be sequenced: OCaml's + argument evaluation order is unspecified, so a tuple built in one + expression could hand the channel's register to the environment. + + The channel, then the environment, and the environment only on a body + that declared one — which is exactly the set of bodies an [Fn] value can + reach. Every other function is never the target of an env-passing call, + so the two never meet out of step. See [Emit.env_param]. *) + let xfer_at = next_int () in + let env_at = if env then Some (next_int ()) else None in + sret_at, ps, env_at, xfer_at (* ── The frame map ───────────────────────────────────────────────────── *) @@ -3422,7 +3493,7 @@ let where_from = function let frame_map (md : Emit.m) (fn : Tast.fn) ~slots ~fixed ~total ~outgoing ~xfer_off ~sret_off ~retval ~dframe ~dslotv ~sret ~sret_at ~param_at - ~xfer_at = + ~env_at ~xfer_at = let b = Buffer.create 1024 in let line s = Buffer.add_string b (if s = "" then "#\n" else "# " ^ s ^ "\n") in (* The prose paragraphs wrap; the table below does not, because its columns @@ -3493,10 +3564,22 @@ let frame_map (md : Emit.m) (fn : Tast.fn) ~slots ~fixed ~total ~outgoing (Types.to_string fn.Tast.ret) (if is_float fn.Tast.ret then "xmm0" else "rax"))); para (Printf.sprintf - "The transfer channel arrives last of all, %s. It is a pointer to the cell a \ - callee writes its target into, and reading it is what every guard below \ - does." - (where_from xfer_at)); + "The transfer channel arrives after the parameters, %s. It is a pointer to the \ + cell a callee writes its target into, and reading it is what every guard \ + below does.%s" + (where_from xfer_at) + (match env_at with + | None -> + " Nothing follows it: no (Fn ...) value can reach this function, so it \ + declares no environment — which is what lets an ordinary defn cost \ + exactly what it did before capture existed." + | Some at -> + Printf.sprintf + " And then the environment, %s, because a (Fn ...) value can reach this \ + function and every such call passes one. It holds the captured copies \ + when there are any and is ignored when there are not; either way the \ + signature declares it, so the call is exactly typed." + (where_from at))); line ""; para (Printf.sprintf "The frame is 0x%x bytes below rbp. %s" total @@ -3716,7 +3799,9 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) high-water mark of the temporaries and this is the boundary below which they start. It is the frame map's last line. *) let fixed = f.frame in - let sret_at, param_at, xfer_at = incoming_of ~sret fn.Tast.params in + let sret_at, param_at, env_at, xfer_at = + incoming_of ~sret ~env:(fn.Tast.fenv <> None) fn.Tast.params + in (* An aggregate parameter arrives as a pointer to the caller's copy and has to be copied into its slot before anything else runs — and [rep movsb] eats rdi, rsi and rcx, which is where three of the other parameters still @@ -3975,6 +4060,18 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) | _ -> max 1 (fst (Emit.lay md ty))) end) fn.Tast.params; + (* The environment, on the one kind of function that declared one. Every + other function never asks where it is, which is exactly why a call site + may append it whether or not the callee wanted it. *) + (match fn.Tast.fenv, env_at with + | Some slot, Some at -> + (match at with + | Ireg r -> store_int pb ~src:r ~mm:(Frame f.slots.(slot)) ~size:8 + | Istk d -> + load_int pb ~dst:rax ~mm:(Frame d) ~size:8 ~signed:false; + store_int pb ~src:rax ~mm:(Frame f.slots.(slot)) ~size:8 + | Isse _ -> unsupported "the environment in an SSE register") + | _ -> ()); (match xfer_at with | Ireg r -> store_int pb ~src:r ~mm:(Frame f.xfer_off) ~size:8 | Istk d -> @@ -4052,7 +4149,7 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) (frame_map md fn ~slots:f.slots ~fixed ~total:(frame_bytes f) ~outgoing:f.outgoing ~xfer_off:f.xfer_off ~sret_off:f.sret_off ~retval:f.retval ~dframe:f.dframe ~dslotv:f.dslotv ~sret ~sret_at - ~param_at ~xfer_at); + ~param_at ~env_at ~xfer_at); Buffer.add_string out (Printf.sprintf "\t.globl\t%s\n" sym); (* [emit.ml:2072] says this is load-bearing and it is: default visibility in a shared object is interposable, and that applies to taking the address @@ -4191,6 +4288,8 @@ let emit_main ?(cfi = false) ?(ann = false) ?(startup = false) ?(gc = false) let xfer = -8 and argv = -32 in xor_rr b ~dst:rax ~src:rax; store_int b ~src:rax ~mm:(Frame xfer) ~size:8; + (* [main] declares no environment — it is not reached through a function + value — so this is the call it always was. *) (match fn.Tast.params with | [] -> lea b ~dst:rdi ~mm:(Frame xfer) | [ _ ] -> @@ -4857,10 +4956,14 @@ let redefinition ~checks ?(dev = true) ?(known = fun _ -> true) (* A clause lifted out of a target comes with it: its body may have changed too, and it is reached by address from inside this module rather than through a cell. Every other lifted clause is invisible here. *) + (* The widening thunks come whole, for the reason [Emit.redefinition] + gives: a module that hands a name to an [Fn]-typed parameter names one + and the host has no cell for it. *) let lifted = List.filter (fun (f : Tast.fn) -> match f.Tast.fparent with + | Some "" -> true | Some q -> List.mem q fns | None -> false) p.Tast.fns diff --git a/runtime/flan_rt.c b/runtime/flan_rt.c index 2e79e6c9..50d92863 100644 --- a/runtime/flan_rt.c +++ b/runtime/flan_rt.c @@ -33,10 +33,21 @@ * A type is a number rather than a pointer to anything, so that a module * compiled later against a running program agrees with it: see Check.type_id. */ +/* [env] is the establishing function's copies of whatever the clause + * captured, or NULL. It is passed after the channel, and *every* clause + * declares it whether or not it captured — this walk cannot know which one + * it is about to reach, and a call whose signature is one argument longer + * than the callee's is a trap on wasm32, where call_indirect compares them. + * Emit's env_param is where the rule is written. + * + * It points into the establishing frame, which is alive for exactly as long + * as the handler frame below it is on this stack — a handler frame is popped + * by the body that pushed it, so there is no dangling case here to defer. */ typedef struct flan_handler { struct flan_handler *prev; uint32_t type_id; - void (*fn)(void *condition, void *xfer); + void (*fn)(void *condition, void *xfer, void *env); + void *env; } flan_handler; static flan_handler *handlers; @@ -64,7 +75,7 @@ void flan_handler_pop(flan_handler *h) { void flan_signal(uint32_t type_id, void *condition, void *xfer) { for (flan_handler *h = handlers; h != NULL; h = h->prev) if (h->type_id == type_id) { - h->fn(condition, xfer); + h->fn(condition, xfer, h->env); if (*(void **)xfer != NULL) return; } } @@ -1992,7 +2003,9 @@ static uint64_t flan_hash_mem(const uint8_t *p, int64_t n, uint64_t seed) { * * The pointer form has to match flan_hash_fn, whose last parameter exists * because a hash function emitted for a struct key is an ordinary Flan - * function and every Flan function's signature ends with the transfer channel. + * function and every Flan function's signature ends with the transfer + * channel. No environment: a hasher is reached from this file and never + * through a function value, so it declares none and is handed none. * The direct form has to match what such an emitted function *calls*, and an * emitted function has no channel to hand on — it would be passing its own, * which is not the same thing and not something a leaf hasher should see. So diff --git a/test/programs/fn-capture-dyn.flan b/test/programs/fn-capture-dyn.flan new file mode 100644 index 00000000..03ee5b6d --- /dev/null +++ b/test/programs/fn-capture-dyn.flan @@ -0,0 +1,13 @@ +;; A dyn is the one thing a capture refuses outright, and for the reason a +;; struct field of dyn already refuses: the collector's roots are frames, and +;; nothing pushes the fields of the environment struct a capture synthesises. +;; A copy in there would be a live value reachable only through memory the +;; marker never walks. Milestone 2's per-type descriptors lift it, alongside +;; the condition payload's and the struct field's. +(defn run [f (Fn [] i64)] i64 (f)) + +;; [d] is unannotated, which is what makes it a dyn. +(defn use [d] i64 + (run (fn [] (i64 d)))) + +(defn main [] i32 (println (use 7)) 0) diff --git a/test/programs/fn-capture-set.flan b/test/programs/fn-capture-set.flan new file mode 100644 index 00000000..6e2bc66f --- /dev/null +++ b/test/programs/fn-capture-set.flan @@ -0,0 +1,10 @@ +;; A captured name is a copy, taken where the value was made. A store into it +;; would change the copy and leave the local it came from as it was, which is +;; a silent disagreement — so it is refused, and the message says which of the +;; two would have moved. +(defn run [f (Fn [] i32)] i32 (f)) + +(defn main [] i32 + (let [n 1] + (println (run (fn [] (set n 2) n)))) + 0) diff --git a/test/programs/fn-capture.flan b/test/programs/fn-capture.flan index 7b9975f3..934e8b8e 100644 --- a/test/programs/fn-capture.flan +++ b/test/programs/fn-capture.flan @@ -1,11 +1,116 @@ -;; Capture does not exist. An fn is lifted into a function of its own and is -;; handed nothing but its parameters, so a reference to a local of the -;; enclosing function is refused by name rather than resolved to something it -;; did not mean. spec-memory.md's capture cases, and escaping closures with -;; them, are deferred; this is the refusal that says so where it happens. -(defn use [f (Fn [] i32)] i32 (f)) +;; Capture by value into a stack environment — spec-memory.md's case 2. +;; +;; An fn is still lifted into a function of its own, but it is no longer handed +;; nothing but its parameters: a local of the enclosing function that it names +;; is *copied* into an environment on that function's frame when the value is +;; made, and the lifted body reads the copy. The value is the code address and +;; that environment beside it, which is why a callee that knows only +;; (Fn [i32] i32) can still call it. +;; +;; What is not here is the escaping half — a value carrying an environment may +;; not outlive the frame the copies are on, and fn-escape*.flan is where each +;; of those refusals is written down. + +(defn double [x i32] i32 (* x 2)) + +(defn apply2 [f (Fn [i32] i32) x i32] i32 (f x)) +(defn call0 [f (Fn [] i32)] i32 (f)) +(defn twice [f (Fn [i32] i32) x i32] i32 (f (f x))) + +;; The copy is taken where the value is made and not where it is read, and +;; this is what proves it: the local is changed *after* the fn value exists +;; and before it is called, through a pointer, so nothing about the order can +;; be an accident of evaluation. +(defn bump-then-call [f (Fn [] i32) p (Ptr i32)] i32 + (set (deref p) 99) + (f)) + +(defstruct Pt [x i32 y i32]) + +(defstruct TooBig [n i32]) +(defonce seen i32) + +(defn checked [x i32] i32 + (when (> x 100) (signal (TooBig {.n x}))) + x) + +;; A handler clause is lifted the same way and captures the same way, and is +;; sound with nothing left over: a handler frame is popped by the body that +;; pushed it, so the establishing frame is alive whenever the clause runs. +;; [budget] is read out of the environment; the accumulator is a global, +;; because a captured copy is a copy and a store into one would leave the +;; local it came from as it was. +(defn handles [] i32 + (let [budget 1000 + xs [5 200 7 300] + s (slice xs 0 4) + t 0] + (handler-bind [(TooBig [c] (set seen (+ seen (+ budget (.n c)))))] + (dotimes [i 4] + (set t (+ t (checked (at s i)))))) + (print t) (print " ") (println seen) + seen)) (defn main [] i32 - (let [n 7] - (println (use (fn [] n)))) + ;; The motivating program. + (let [bonus 10] + (println (apply2 (fn [x] (+ x bonus)) 5))) + + ;; Copy at creation: the fn answers 1 and the local is 99. + (let [n 1] + (print (bump-then-call (fn [] n) (addr n))) + (print " ") + (println n)) + + ;; What may be captured. A string and a slice are two words copied as two + ;; words — the bytes stay whoever's they were, which is fine exactly while + ;; the value cannot outlive the frame that owns them. A struct and a fixed + ;; array are copied whole. A function value is copied as a function value. + (let [s "hi" + arr [1 2 3 4] + sl (slice arr 0 4) + p (Pt {.x 3 .y 4}) + g double] + (println (call0 (fn [] (i32 (length s))))) + (println (call0 (fn [] (at sl 2)))) + (println (call0 (fn [] (+ (.x p) (.y p))))) + (println (call0 (fn [] (at arr 3)))) + (println (apply2 (fn [x] (g (+ x 1))) 4))) + + ;; An fn inside an fn, each capturing. The inner one names a local neither + ;; of them declared, so the outer one captures it too and the inner one + ;; copies the outer one's copy. + (let [a 100 + b 20] + (println (apply2 (fn [x] (+ x (call0 (fn [] (+ a b))))) 3))) + + ;; A loop variable: what the fn sees is the value at the iteration it was + ;; made on, not the last one. 0 + 1 + 2 + 3. + (let [total 0] + (dotimes [i 4] + (set total (+ total (call0 (fn [] i))))) + (println total)) + + ;; And the same again where the loop variable is rebound by a recur rather + ;; than stepped by a dotimes, which is a store into the slot the copy is + ;; taken from: 100 + 101 + 102. + (println + (let [base 100] + (loop [i 0 acc 0] + (if (< i 3) + (recur (+ i 1) (+ acc (call0 (fn [] (+ base i))))) + acc)))) + + ;; Called twice, so the environment is read more than once and a body that + ;; consumed it would show. + (let [k 5] + (println (twice (fn [x] (+ x k)) 1))) + + ;; An fn's own let may shadow a name the enclosing function also has, and a + ;; store into *that* one is an ordinary store: the refusal is about a + ;; captured copy and not about the spelling. 5 + 1. + (let [n 5] + (println (+ n (call0 (fn [] (let [n 0] (set n 1) n)))))) + + (println (handles)) 0) diff --git a/test/programs/fn-cfn-captures.flan b/test/programs/fn-cfn-captures.flan new file mode 100644 index 00000000..d021597b --- /dev/null +++ b/test/programs/fn-cfn-captures.flan @@ -0,0 +1,10 @@ +;; A CFn is the bare address, so a literal written into one has nowhere to +;; keep the copies. Refused with the name of what it captured, because that is +;; the fact to act on, and with the fix named: widen the position to Fn, which +;; is what the type is for. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +(defn main [] i32 + (let [bonus 10] + (println (apply-bare (fn [x] (+ x bonus)) 5))) + 0) diff --git a/test/programs/fn-cfn-narrow.flan b/test/programs/fn-cfn-narrow.flan new file mode 100644 index 00000000..cd101f38 --- /dev/null +++ b/test/programs/fn-cfn-narrow.flan @@ -0,0 +1,13 @@ +;; Coercion between the two function types goes one way only. A (CFn ...) +;; widens into a (Fn ...) through a per-signature thunk, and a (Fn ...) does +;; not narrow: there is nowhere for the environment to go, and nothing at this +;; definition can know whether there is one. +;; +;; Refused by the ordinary type message, which names both spellings and is the +;; right sentence for it: the fix is to widen the position, not to convert the +;; value. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +(defn hand-on [f (Fn [i32] i32) x i32] i32 (apply-bare f x)) + +(defn main [] i32 0) diff --git a/test/programs/fn-cfn.flan b/test/programs/fn-cfn.flan new file mode 100644 index 00000000..2ab8ed9a --- /dev/null +++ b/test/programs/fn-cfn.flan @@ -0,0 +1,67 @@ +;; The narrow function type. A (CFn [T ...] R) is the bare code address — +;; one word, no environment, and therefore nothing that can capture. A +;; (Fn [T ...] R) is that address and the environment beside it, two words. +;; +;; The reason there are two rather than one: an environment on every signature +;; would tax every function in every program for a feature most of them never +;; use. With CFn written where it is wanted, an ordinary defn emits exactly +;; the signature it emitted before capture existed, and a call to it by name +;; is byte-for-byte what it was. +;; +;; The C is information and not decoration. A value with no environment is the +;; only kind that could ever cross to C, and under the --no-conditions +;; direction FIX.org records — where a signature that cannot transfer drops +;; the channel too — one becomes literally a C function pointer. It is not +;; that today: a declare cannot take a function type at all, and the refusal +;; it meets says so. The name points at what the type is, and at where it is +;; going. +;; +;; **Nobody needs CFn.** An Fn accepts everything a CFn does, so the narrow +;; one is reached for on purpose, for one of four reasons: handing a function +;; to C, later; a table of bare addresses; forbidding capture at a boundary; +;; and the one that is likeliest in practice — a *named* function handed to an +;; Fn parameter goes through the widening thunk and pays an indirect hop per +;; call, where a CFn parameter is a direct call. (map-in-place s double) is +;; the example, and [apply-bare] below is it in miniature. +;; +;; Coercion is one-way. A defn's address and a non-capturing literal satisfy +;; both. An Fn does not narrow to a CFn — there is nowhere for the +;; environment to go — and fn-cfn-narrow.flan is that refusal. + +(defn double [x i32] i32 (* x 2)) +(defn negate [x i32] i32 (- 0 x)) + +;; Taking the narrow one. Nothing that reaches here can carry an environment, +;; which is what the signature is saying. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +;; And the wide one, which is what almost every higher-order signature wants. +(defn apply-any [f (Fn [i32] i32) x i32] i32 (f x)) + +;; A CFn returned. It is a link-time constant with nothing behind it, so +;; handing one back is no different from handing it down — which is exactly +;; what a capturing value cannot do. +(defn pick [up bool] (CFn [i32] i32) (if up double negate)) + +;; A CFn parameter widened to an Fn at a call: the address goes where an +;; environment would be and the thunk reads it back out. This is the hop the +;; narrow type exists to avoid. +(defn through [f (CFn [i32] i32) x i32] i32 (apply-any f x)) + +(defn main [] i32 + ;; A name into a CFn, and into an Fn. + (println (apply-bare double 4)) + (println (apply-any negate 4)) + ;; A literal that captures nothing into a CFn. + (println (apply-bare (fn [x] (+ x 1)) 4)) + ;; And one that does capture, into an Fn. + (let [k 10] + (println (apply-any (fn [x] (+ x k)) 4))) + ;; A returned CFn, called through a computed head. + (println ((pick true) 21)) + (println ((pick false) 21)) + ;; The widening, twice over: a CFn local through a CFn parameter into + ;; an Fn parameter. + (let [g double] + (println (through g 5))) + 0) diff --git a/test/programs/fn-escape-at.flan b/test/programs/fn-escape-at.flan new file mode 100644 index 00000000..33e7816e --- /dev/null +++ b/test/programs/fn-escape-at.flan @@ -0,0 +1,13 @@ +;; An index read is a read, and the escape check's clean list has to be a +;; list. A function value out of a slice is refused exactly as one out of a +;; Vec, a struct or a pointer is — the four are the same act and there is no +;; reason for a reader to have to remember which spellings were enumerated. +;; +;; Not reachable today: nothing can write an Fn into a slice, because every +;; position that would have to hold one is refused. It is here so that the +;; day one can, this is already true — the alternative was a default of +;; "clean" for anything the enumeration had not thought of, which is how a +;; closed list quietly stops being closed. +(defn leak [s [(Fn [] i32)]] (Fn [] i32) (at s 0)) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-copy.flan b/test/programs/fn-escape-copy.flan new file mode 100644 index 00000000..b9f3a1ea --- /dev/null +++ b/test/programs/fn-escape-copy.flan @@ -0,0 +1,19 @@ +;; The hole a capture could otherwise be laundered through, and the reason +;; "the result of a call is clean" is a rule and not a hope. +;; +;; [sneak]'s literal captures [g], so its body holds a *copy* of a function +;; value that may itself carry an environment — and the copy is read out of an +;; environment, which is the one aggregate a function value is ever stored in. +;; If a copy read back out were treated as clean, the literal could return it, +;; the return would arrive at [sneak]'s caller as an ordinary call result, and +;; a capturing value would be out of the frame that owns it with nothing +;; having refused anything. +;; +;; So a function value read out of a struct, a case or a pointer is suspect, +;; and the refusal lands inside the lifted body where the return is written. +(defn getf [f (Fn [] (Fn [] i32))] (Fn [] i32) (f)) + +(defn sneak [g (Fn [] i32)] (Fn [] i32) + (getf (fn [] g))) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-handled.flan b/test/programs/fn-escape-handled.flan new file mode 100644 index 00000000..8e8fd54e --- /dev/null +++ b/test/programs/fn-escape-handled.flan @@ -0,0 +1,12 @@ +;; A handler-bind is an expression and its value is its body's, so it is a way +;; for a function value to be a function's answer — and it would have walked +;; straight past a check that only looked at [return] and at the last form of +;; a block. with-allocator and restart-case are the same shape and are checked +;; the same way. +(defstruct C [id i32]) +(defonce seen i32) + +(defn keep [f (Fn [] i32)] (Fn [] i32) + (handler-bind [(C [c] (set seen (.id c)))] f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-match.flan b/test/programs/fn-escape-match.flan new file mode 100644 index 00000000..1fe2b10c --- /dev/null +++ b/test/programs/fn-escape-match.flan @@ -0,0 +1,16 @@ +;; A match arm's binding is a binding, and the escape check has to see it. +;; +;; Reading the payload by hand is a case-field read, which is suspect: a copy +;; of a function value carries whatever environment the original did. Binding +;; it to a name in an arm is the same read, and the store that fills the arm's +;; slot is inside the branch rather than in any form the walk reads as a +;; binding — so without the arm's slots being taken as suspect too, the Vec, +;; slice, struct and pointer spellings of this were all refused while the one +;; that goes through Option and a name was not. + +(defn leak [o (Option (Fn [] i32))] (Fn [] i32) + (match o + (Some f) f + None (fn [] 0))) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-param.flan b/test/programs/fn-escape-param.flan new file mode 100644 index 00000000..6cba4ed2 --- /dev/null +++ b/test/programs/fn-escape-param.flan @@ -0,0 +1,11 @@ +;; The hard case, answered without looking at a single call site: a function +;; value that arrives as a parameter may carry an environment on its caller's +;; frame, so a function that *stores* one is refused where it is written. +;; +;; That is what makes passing a capturing fn down safe everywhere — no callee +;; can keep it — and it is also the conservative half: this particular [keep] +;; would be harmless for a caller that passed a name, and there is no way for +;; the definition to know that it did. +(defn keep [f (Fn [] i32)] (Fn [] i32) f) + +(defn main [] i32 (println ((keep (fn [] 1)))) 0) diff --git a/test/programs/fn-escape-return.flan b/test/programs/fn-escape-return.flan new file mode 100644 index 00000000..c56617af --- /dev/null +++ b/test/programs/fn-escape-return.flan @@ -0,0 +1,10 @@ +;; The refusal that defines "non-escaping". The copies live in a slot of +;; [make]'s frame, and the value would still be pointing at them after that +;; frame has gone. +;; +;; A returned function value is still fine when it captures nothing — +;; fn-values.flan returns one — so this is about the environment and not about +;; the shape of the value. +(defn make [n i32] (Fn [] i32) (fn [] n)) + +(defn main [] i32 (println ((make 3))) 0) diff --git a/test/programs/fn-escape-store.flan b/test/programs/fn-escape-store.flan new file mode 100644 index 00000000..248c3619 --- /dev/null +++ b/test/programs/fn-escape-store.flan @@ -0,0 +1,7 @@ +;; A store through a pointer is the same escape wearing a different hat: the +;; pointer names storage this frame does not own, so the value would outlive +;; the environment it carries. +(defn stash [p (Ptr (Fn [] i32)) f (Fn [] i32)] () + (set (deref p) f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-vec.flan b/test/programs/fn-escape-vec.flan new file mode 100644 index 00000000..affa50f5 --- /dev/null +++ b/test/programs/fn-escape-vec.flan @@ -0,0 +1,9 @@ +;; A Vec's elements are in a block the allocator owns and the frame does not, +;; so a function value pushed into one outlives whatever environment it +;; carries. Refused for that, and not for the shape of the element type: a Vec +;; of function values is a perfectly good thing to want, and is what case 3 +;; is for. +(defn stash [v (Vec (Fn [] i32)) f (Fn [] i32)] () + (push v f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-extern.flan b/test/programs/fn-extern.flan index 1cf28821..aa74ed0e 100644 --- a/test/programs/fn-extern.flan +++ b/test/programs/fn-extern.flan @@ -1,6 +1,8 @@ ;; A foreign function's address is not a Flan function value. A Flan -;; function's emitted signature ends with the transfer channel and a C one -;; does not, so nothing could call the resulting pointer correctly — and an +;; function's emitted signature ends with the environment and the transfer +;; channel and a C one does not, so nothing could call the resulting pointer +;; correctly — and the gap is wider since capture arrived, because a Flan +;; function value is two words and a C symbol is one — and an ;; aggregate crossing the boundary is flattened by a generated shim, which the ;; raw symbol knows nothing about. Refused for what it is, with the wrapper ;; named as the way to get one. diff --git a/test/programs/fn-generic-nested-return.flan b/test/programs/fn-generic-nested-return.flan new file mode 100644 index 00000000..37343151 --- /dev/null +++ b/test/programs/fn-generic-nested-return.flan @@ -0,0 +1,15 @@ +;; The same line drawn in return position, which is the half that is easy to +;; miss: a (Fn [] (Fn [] $t)) parameter handed a (CFn [] (CFn [] i32)) needs +;; the inner widening built by whatever calls the *argument*, and that is the +;; generic's own body, which was compiled against the parameter's type and not +;; against this caller's. + +(defn inner [] i32 3) + +(defn outer [] (CFn [] i32) inner) + +(defn call-twice [g (Fn [] (Fn [] $t))] $t ((g))) + +(defn main [] i32 + (println (call-twice outer)) + 0) diff --git a/test/programs/fn-generic-nested.flan b/test/programs/fn-generic-nested.flan new file mode 100644 index 00000000..51b74840 --- /dev/null +++ b/test/programs/fn-generic-nested.flan @@ -0,0 +1,19 @@ +;; The widening between the two function types is a value the caller builds — +;; a thunk, minted at the call — so there is exactly one place to build it: +;; around the whole argument. Nested inside one, there is no caller standing +;; where the thunk would have to go. +;; +;; The parameter here is (Fn [(Fn [$t] $t)] i32) and taker's address carries +;; (CFn [(CFn [i32] i32)] i32). Admitting that structurally binds $t and then +;; hands one word where the instance declares two, in a position no later pass +;; can widen: the catch-up that builds the outer thunk compares the whole +;; substituted parameter list and this mismatch is inside it. A call with no +;; type variables in it is refused, so this one is too. + +(defn taker [h (CFn [i32] i32)] i32 (h 1)) + +(defn hof [g (Fn [(Fn [$t] $t)] i32) k $t] i32 (g (fn [x] x))) + +(defn main [] i32 + (println (hof taker 0)) + 0) diff --git a/test/programs/fn-generic.flan b/test/programs/fn-generic.flan new file mode 100644 index 00000000..6505a72c --- /dev/null +++ b/test/programs/fn-generic.flan @@ -0,0 +1,33 @@ +;; A generic whose function parameter binds the type variable, which is the +;; shape a name now has to reach: a defn's address carries (CFn [i32] i32), +;; and (apply2 bump 1) has to bind $t from it and then widen the argument. +;; +;; Both halves are here because they fail apart. The binding is Check's +;; bind_ty, which runs inside generic_call and decides the instantiation; the +;; widening is the catch-up pass at the end of the same function, because a +;; parameter that still mentioned a variable was checked with no expectation +;; at all and expect never saw the pair. Get the first without the second and +;; the call site hands one word to an instance that declares two. +;; +;; The prelude does not cover this and that is worth saying: its higher-order +;; functions bind $t from an *earlier* argument, so the parameter is already +;; concrete by the time the function value is reached, and (map-in-place s +;; double) never walks this path at all. +;; +;; The CFn half is new rather than restored: a matching bare address against +;; a (CFn [$t] $t) parameter had no arm either, and fell through to plain +;; equality. + +(defn apply2 [f (Fn [$t] $t) x $t] $t (f (f x))) +(defn applyc [f (CFn [$t] $t) x $t] $t (f (f x))) + +(defn bump [n i32] i32 (+ n 1)) +(defn twice [x f64] f64 (* x 2.0)) + +(defn main [] i32 + (println (apply2 bump 1)) + (println (applyc bump 1)) + ;; A second instantiation, so the two copies are really two and the thunk + ;; the first one minted is not reused at the wrong signature. + (println (apply2 twice 1.5)) + 0) diff --git a/test/programs/fn-in-struct.flan b/test/programs/fn-in-struct.flan index 510b9b6c..661e0c32 100644 --- a/test/programs/fn-in-struct.flan +++ b/test/programs/fn-in-struct.flan @@ -4,6 +4,17 @@ ;; union's first case. So it is refused where the field is written rather than ;; left to crash at the call, and the same rule covers a global, a fixed ;; array's element and (zeroed). +;; +;; Capture sharpened the reason behind this one without changing it. A struct +;; outlives the frame it was built on, so a field could not hold a value +;; carrying an environment either — see fn-escape-*.flan. The zero is still +;; what the message names, because it is the objection that applies to every +;; function value and not only to a capturing one. +;; +;; Which means a (CFn ...) field is refused too, and for the zero alone — +;; a table of function pointers is exactly what that type is for, and nothing +;; about capture stands in its way. An (Option (CFn ...)) field is already +;; legal and is the shape that works; FIX.org carries the rest as its own item. (defstruct Ops [run (Fn [i32] i32)]) (defn main [] i32 0) diff --git a/test/programs/fn-no-type.flan b/test/programs/fn-no-type.flan index 2189b49e..944a5af2 100644 --- a/test/programs/fn-no-type.flan +++ b/test/programs/fn-no-type.flan @@ -2,6 +2,10 @@ ;; so it takes them from the position it is written in. An argument position ;; says what is wanted, because the callee's signature is threaded into every ;; argument; a let binding does not, and is refused saying so. +;; +;; The one thing capture did not change. It is about where the *types* come +;; from and not about what the body may see, so an fn is still written where +;; something says what it takes. (defn main [] i32 (let [f (fn [x] (* x 2))] (println (f 3))) diff --git a/test/programs/fn-thunk-reload.flan b/test/programs/fn-thunk-reload.flan new file mode 100644 index 00000000..b8f815bd --- /dev/null +++ b/test/programs/fn-thunk-reload.flan @@ -0,0 +1,24 @@ +;; Two widenings of different signatures in one program, so the dev loop has +;; two thunks to tell apart across a reload. +;; +;; A thunk is not a function anybody wrote, so nothing in the source names it +;; and nothing can be edited to rename it. But Session.compatible compares a +;; reload's functions against the running program's *by name*, and a name that +;; means whichever thunk was minted first means a different signature the +;; moment the forms are reordered — which reads to the session as a function +;; whose signature was edited, and answers an ordinary edit with "Restart to +;; change it". So the name is the signature, spelled so that it can be read +;; back, and reordering these two calls is an ordinary body change. + +(defn a1 [x i32] i32 (+ x 1)) +(defn b1 [x i64] i64 (+ x 1)) + +(defn use32 [f (Fn [i32] i32)] i32 (f 1)) +(defn use64 [f (Fn [i64] i64)] i64 (f 1)) + +(defn both [] i32 + (println (use32 a1)) + (println (use64 b1)) + 0) + +(defn main [] i32 (both)) diff --git a/test/programs/fn-thunk-share.flan b/test/programs/fn-thunk-share.flan new file mode 100644 index 00000000..946ed730 --- /dev/null +++ b/test/programs/fn-thunk-share.flan @@ -0,0 +1,25 @@ +;; The widening thunk is memoised per signature, and the key is the types. +;; +;; It used to be a *name*, derived from mangle_ty, which flattens a whole +;; signature into one hyphen-joined string and loses arity and every type +;; boundary with it: (CFn [(Ptr i32)] i32) and (CFn [ptr i32] i32) — the +;; second over a struct someone called ptr — both flatten to the same thing. +;; Keyed on that, the second widening reuses the first's thunk and calls it +;; with the wrong arity, which both backends compile without a word and +;; neither runs. This program is that pair, and it prints 5 and 17. +;; +;; A struct named ptr is legal and ordinary; nothing about the collision +;; needed a program written to provoke it, only two signatures that happened +;; to flatten alike. +(defstruct ptr [a i32 b i32]) + +(defn f1 [p (Ptr i32)] i32 (deref p)) +(defn f2 [a ptr b i32] i32 (+ (.a a) b)) + +(defn use1 [f (Fn [(Ptr i32)] i32)] i32 (let [x 5] (f (addr x)))) +(defn use2 [f (Fn [ptr i32] i32)] i32 (f (ptr {.a 10 .b 0}) 7)) + +(defn main [] i32 + (println (use1 f1)) + (println (use2 f2)) + 0) diff --git a/test/programs/fn-values.flan b/test/programs/fn-values.flan index e618b5a8..52362f23 100644 --- a/test/programs/fn-values.flan +++ b/test/programs/fn-values.flan @@ -1,5 +1,13 @@ -;; Function values, the non-escaping kind: a code address and no environment -;; beside it. Capture does not exist, so nothing here can outlive anything. +;; Function values, and specifically the ones with no environment. Nothing +;; here captures, which is what makes every one of these safe to return and to +;; hand around — fn-capture.flan is the other half, and fn-escape-*.flan is +;; the line between them. +;; +;; Every signature below says (Fn ...), which is the wide one: it admits a +;; capturing value and so pays for a two-word value and a widening thunk where +;; a name is handed to it. Written as (CFn ...) these would pay neither, and +;; fn-cfn.flan is where that is spelled out — the spellings are kept apart +;; here so that the two programs cover the two conventions between them. ;; ;; This is a Lisp-1 — one top-level namespace, enforced — so a bare function ;; name *is* the function and there is no #' to write. diff --git a/test/reload_host.c b/test/reload_host.c index c22ac294..c0c0448f 100644 --- a/test/reload_host.c +++ b/test/reload_host.c @@ -43,7 +43,10 @@ /* The trailing ptr is the transfer channel spec-conditions.md §6 puts in every * Flan signature. This host never transfers, so it passes a slot of its own * that stays null — but the parameter is not optional: getting it wrong reads - * garbage as the channel and fails nowhere near here. */ + * garbage as the channel and fails nowhere near here. + * + * No environment: [outer] is called by name and not through a function value, + * so it declares none. That is the point of there being two function types. */ extern int64_t flan_outer(void *xfer) __asm__("flan.outer"); extern int64_t flan_counter __asm__("flan.counter"); diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index e6e5b395..6de3de75 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -3928,12 +3928,121 @@ level "1" outputs ~opt:"-O0" "the prelude's map, filter, reduce and sort-by, -O0" "programs/higher-order.flan" higher_order_out; - (* What function values do *not* include, each refused by name. Capture is - the headline: an fn is lifted into a function of its own and handed - nothing but its parameters, so spec-memory.md's capture cases and - escaping closures with them stay deferred. *) - refuses "an fn cannot capture" "programs/fn-capture.flan" - "cannot see n"; + (* Capture by value into a stack environment — spec-memory.md's case 2. + Three opt levels for the reason the case above has them, and for one + more: the environment is a struct in the frame and the value carries + its address, which is exactly the shape -O2 is entitled to make + disappear. -O0 is what proves there is a real store and a real load + behind it. A dev build is here because the value's code half still + comes out of the indirection cell and the environment half must not + have disturbed that. + + The two lines worth naming. "1 99" is copy-at-creation: the local is + changed through a pointer after the value exists and before it is + called, so no evaluation order can account for the fn still answering + 1. And "512 2500" is the handler clause reading a captured budget, + which is the same machinery in the one place where there is no + escaping case left over. *) + let fn_capture_out = + "15\n1 99\n2\n3\n7\n4\n10\n123\n6\n303\n11\n6\n512 2500\n2500\n" + in + outputs "an fn capturing by value" "programs/fn-capture.flan" + fn_capture_out; + outputs ~opt:"-O0" "an fn capturing by value, -O0" "programs/fn-capture.flan" + fn_capture_out; + (* The narrow function type, and the one-way coercion. What this asserts + that no checker test can: a CFn widened into an Fn and called through + the wider signature reaches the same body and answers the same thing, + on both opt levels — so the null environment a widening pairs with the + address really is ignored by a body that declared none. *) + let fn_ptr_out = "8\n-4\n5\n14\n42\n-21\n10\n" in + outputs "the two function types" "programs/fn-cfn.flan" fn_ptr_out; + outputs ~opt:"-O0" "the two function types, -O0" "programs/fn-cfn.flan" + fn_ptr_out; + (* And a dev build, which is the one that exercises the widening thunk + over an indirection cell: a name widened into an Fn is a cell load for + the address and the thunk for the call, and the two have to compose. *) + outputs ~dev:true "the two function types, dev" "programs/fn-cfn.flan" + fn_ptr_out; + + (* Two signatures that flatten to one string under [mangle_ty], which is + how the thunk memo used to be keyed. Keyed on the name, the second + widening reuses the first's thunk at the wrong arity — a miscompile + both backends emit without a word. Keyed on the types, this prints + 5 and 17. *) + outputs "two signatures that mangle alike" "programs/fn-thunk-share.flan" + "5\n17\n"; + outputs ~opt:"-O0" "two signatures that mangle alike, -O0" + "programs/fn-thunk-share.flan" "5\n17\n"; + + (* A generic whose *function* parameter binds the type variable, which is + the shape a bare name has to reach now that a defn's address carries + CFn. It fails in two places and they fail apart: the binding, in + bind_ty, and the widening, in generic_call's catch-up pass — a + parameter that still mentioned a variable was checked with no + expectation, so expect never saw the pair. The prelude misses it + entirely, because its higher-order functions bind $t from an earlier + argument and the parameter is concrete by the time the function value + is reached. *) + let fn_generic_out = "3\n3\n6\n" in + outputs "a generic that binds its variable through a function type" + "programs/fn-generic.flan" fn_generic_out; + outputs ~opt:"-O0" + "a generic that binds its variable through a function type, -O0" + "programs/fn-generic.flan" fn_generic_out; + (* And where the same binding stops. The widening is a value the caller + builds around the whole argument, so a function type nested inside an + argument's own type has no caller standing where its thunk would go — + and the catch-up pass that builds the outer one compares the whole + substituted parameter list, so a mismatch inside it is not something + any later pass can repair. Both positions, because the return one is + the easier of the two to leave open. *) + refuses "a nested function type does not widen" + "programs/fn-generic-nested.flan" + "hof expects (Fn [(Fn [t] t)] i32) here"; + refuses "and neither does one in return position" + "programs/fn-generic-nested-return.flan" + "call-twice expects (Fn [] (Fn [] t)) here"; + outputs ~dev:true "an fn capturing by value, dev" "programs/fn-capture.flan" + fn_capture_out; + + (* What function values do *not* include, each refused by name. Escape is + the headline now that capture is not: the copies live in the frame the + literal was written in, so a value carrying their address may be + called, passed down and copied about, and may not outlive that frame. + Each of these names case 3 — the collector-allocated environment — + because "not yet" is the true sentence. *) + refuses "a captured fn cannot be returned" "programs/fn-escape-return.flan" + "a return would outlive the frame"; + refuses "a function value parameter cannot be kept" + "programs/fn-escape-param.flan" "may carry an environment"; + refuses "a function value cannot be stored through a pointer" + "programs/fn-escape-store.flan" "a store would outlive the frame"; + refuses "a function value cannot be pushed into a Vec" + "programs/fn-escape-vec.flan" "a container would outlive the frame"; + (* The two an escape check written by eye would have missed. A function + value read back out of an environment is a copy of something that may + carry one, and a handler-bind is an expression whose value is its + body's — so both are ways for a suspect to be a function's answer. *) + refuses "an index read is a read like any other" + "programs/fn-escape-at.flan" "may carry an environment"; + refuses "a match arm's binding is a binding" + "programs/fn-escape-match.flan" "may carry an environment"; + refuses "a captured function value cannot be handed back" + "programs/fn-escape-copy.flan" "a return would outlive the frame"; + refuses "a handler-bind's value is a return too" + "programs/fn-escape-handled.flan" "a return would outlive the frame"; + refuses "a captured local is a copy and cannot be assigned" + "programs/fn-capture-set.flan" "cannot assign to n"; + (* The two function types, and the line between them. A CFn is the bare + address, so nothing that captures can be one and nothing that may + capture can narrow into one. *) + refuses "an fn that captures is not a CFn" + "programs/fn-cfn-captures.flan" "and not a (CFn [i32] i32)"; + refuses "an Fn does not narrow to a CFn" + "programs/fn-cfn-narrow.flan" "expected (CFn [i32] i32)"; + refuses "an fn cannot capture a dyn" "programs/fn-capture-dyn.flan" + "the collector finds its roots by frame"; refuses "an fn with no type to take" "programs/fn-no-type.flan" "nothing here says what this fn"; refuses "a function value would be zeroed" "programs/fn-in-struct.flan" diff --git a/test/test_flan.ml b/test/test_flan.ml index d3faadd2..183c9747 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -473,7 +473,7 @@ let () = (match ty "(Map string i32)" with | Tapp ("Map", [ _; _ ]) -> () | _ -> check "(Map K V) is a map type" false); (match ty "(Fn [a a] bool)" with - | Tfn ([ _; _ ], _) -> () | _ -> check "(Fn [T] R)" false); + | Tfn (_, [ _; _ ], _) -> () | _ -> check "(Fn [T] R)" false); (* ── Declarations ──────────────────────────────────────────────── *) (match (parse_decl "(defn f [x i32] bool x)").d with @@ -3668,14 +3668,21 @@ let () = rejects_check "signal in value position" "(defstruct C [id i32])\n\ (defn f [] i32 (signal (C {.id 1})))" ~needle:"expected i32"; - (* A handler is lifted into a function of its own, so the establishing - function's locals are not there. Capturing them is a closure, which is - milestone 5 — until then it is refused for the reason it is refused for - rather than as an unknown name. *) - rejects_check "a handler capturing a local" + (* A handler clause captures the establishing function's locals by value — + spec-memory.md's case 2 — so it can read one. A *store* is the thing that + is not there: the clause holds a copy, and writing to it would leave the + local it came from as it was, which is a silent disagreement and not a + feature. Refused for that reason, with the accumulation case pointed at a + global. *) + accepts "a handler reading a local" + "(defstruct C [id i32])\n\ + (defonce seen i32)\n\ + (defn f [] () (let [n 7] (handler-bind [(C [c] (set seen (+ n (.id c))))] \ + (signal (C {.id 2})))))"; + rejects_check "a handler assigning to a captured local" "(defstruct C [id i32])\n\ (defn f [] () (let [n 0] (handler-bind [(C [c] (set n 1))] (signal (C {.id 2})))))" - ~needle:"a handler cannot see n"; + ~needle:"a handler cannot assign to n"; (* The frames are popped on the way out of the body, so an early exit would leave them on the stack pointing into a function that has gone. *) rejects_check "return inside handler-bind" @@ -3761,14 +3768,18 @@ let () = accepts "handler-case with several clauses" (boom ^ "(defn f [] i32 (handler-case 1 [(Boom [c] (.id c)) \ (Dud [c] (+ 1 (.id c)))]))"); - (* The whole difference from handler-bind: a clause runs at the form, in the - function that wrote it, so it sees that function's locals. The same body - under a handler-bind is refused by name. *) + (* The difference from handler-bind is narrower than it was. Both see the + establishing function's locals now — a handler-case clause *is* that + function, and a handler-bind clause captures them by value. What only a + handler-case clause can do is *assign* to one, because it is not holding + a copy. *) accepts "a handler-case clause sees the establishing function's locals" (boom ^ "(defn f [] i32 (let [n 1] (handler-case 0 [(Boom [c] n)])))"); + accepts "and may assign to one, which a handler-bind clause may not" + (boom ^ "(defn f [] i32 (let [n 1] (handler-case 0 [(Boom [c] (set n 2) n)])))"); rejects_check "a handler-bind clause still cannot" (boom ^ "(defn f [] i32 (let [n 1] (handler-bind [(Boom [c] (set n 2))] 0)))") - ~needle:"a handler cannot see n — it is a local of the enclosing function"; + ~needle:"a handler cannot assign to n"; (* Nothing static refuses a condition no clause lists: it installs no frame that matches, so it goes past untouched and the body carries on. *) accepts "a condition no clause lists" diff --git a/test/test_session.ml b/test/test_session.ml index 8bb2d271..d3fbfd30 100644 --- a/test/test_session.ml +++ b/test/test_session.ml @@ -1177,6 +1177,30 @@ let () = | exception Loc.Error { Loc.dmsg = m; _ } -> fail "an expression that instantiates a generic: %s" m); + (* ── The widening thunks, across a reorder ────────────────────── + A thunk is a function nobody wrote and nothing in the source names, so + the only way one can change is the program growing or losing a widening + — and reordering two calls is neither. But [compatible] compares by + name, so a thunk named for the order it was minted in means one + signature before the edit and another after, and the session answers a + body change with "Restart to change it" about a name the programmer + cannot find. The name spells the signature, so this reload is ordinary. + + Both directions of the pair are here — the same two calls, swapped — + because a name that is a counter is wrong for exactly one of them and + the test has to be the one that is wrong. *) + (let t, _ = Session.create ~file:"programs/fn-thunk-reload.flan" () in + match + Session.eval t + "(defn both [] i32 (println (use64 b1)) (println (use32 a1)) 0)" + with + | c -> + if not (List.mem "both" c.Session.fns) then + fail "reordering two widenings installed %s" + (String.concat " " c.Session.fns) + | exception Loc.Error { Loc.dmsg = m; _ } -> + fail "reordering two widenings was refused: %s" m); + (* ── A class whose slots changed ──────────────────────────────── The dev loop's half of CLHS 4.3.6. Three things have to be true of the session for the runtime's migration to ever be reached: a changed slot @@ -1305,7 +1329,7 @@ let () = if Session.strip_rebind "~2" <> "~2" then fail "a name that is only a suffix was stripped"; (let fn snames : Tast.fn = { Tast.name = "f"; params = []; ret = Types.Unit; body = []; - fdefers = []; fparent = None; floc = Loc.unknown; + fdefers = []; fenv = None; fparent = None; floc = Loc.unknown; slots = Array.make (Array.length snames) (Types.Int Types.I32); snames } in diff --git a/web/index.html b/web/index.html index d48628ac..be27bf4d 100644 --- a/web/index.html +++ b/web/index.html @@ -519,7 +519,8 @@ notation reads as exactly one data item.

(Handle T)a reference into a pool that reports a dead referentindex and generation packed into an i64 (Ptr T)raw pointera pointer (Option T)Some / Nonetag byte + T -(Fn [T ...] R)a function valuea pointer +(Fn [T ...] R)a function value, which may have captureda code address and an environment pointer +(CFn [T ...] R)a function value that cannot capture — the C is what a C function pointer would need, not a way to reach C todaya pointer Allocatoran opaque builtin: a proc, its data and a capability seta pointer to that $ta type variable — see genericswhatever it is instantiated at a structvalue typefields in declaration order