From 2572f0a5371434349cda3434c3cfe0ee63100acb Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 21 Sep 2026 12:54:13 +0700 Subject: [PATCH 1/4] An fn sees the locals it was written among, and Fn says so in its type spec-memory.md's case 2, capture by value into a stack environment, and the calling convention the author's rulings asked for. (Fn [i32] i32) captures; {code, env}; the common case (CFn [i32] i32) the bare address; one word; cannot capture A local of the enclosing function that an fn names is copied into a struct the checker synthesises, held in a slot of that function's frame, and the value carries its address; the lifted body reads the copies back into named slots of its own, once, at entry. So the name in the body means what the local held at the instant the value was made -- fn-capture.flan changes the local through a pointer after the value exists and the fn still answers with the old one. Two types rather than a uniform environment parameter: "while it's dyn first, static side should never have to pay the price for the existence of the dyn side... if you fully opt out, for instance, using --no-gc flag, then we should be operating under Odin/C semantics and never paying any runtime costs." The environment is declared by exactly the bodies an (Fn ...) value can reach -- a lifted literal in an Fn position, every handler clause, and the widening thunks -- and by nothing else. An ordinary defn emits the signature it always did; calc-me and fourteen corpus programs were diffed to say so. CFn, because the C carries information: a value with no environment is the only kind that could ever cross to C, and under the --no-conditions direction FIX.org records it becomes literally a C function pointer. It is not that today -- a declare cannot take a function type at all -- and crossable's refusal says so where a reader would otherwise be misled. Nobody needs CFn: Fn accepts everything, and the commonest reason to reach for the narrow one is that a *named* function handed to an Fn pays a hop through the widening thunk where a CFn is a direct call. That thunk is one small function per distinct signature widened, which reads the bare address back out of the environment and calls it. The cheaper trick -- the environment last, ignored by a body that never declared it -- is legal under SysV and is a trap under wasm32's call_indirect, which compares the signature at the call. Every indirect call is exactly typed now. A handler clause captures the same way and is sound with nothing left over: its frame is popped by the body that pushed it. What is refused there is a *store* into a captured name -- it is a copy, and writing to it would leave the local as it was. And the other half, which is what "non-escaping" means: a value carrying an environment may be called, passed down and let-bound, and may not be returned, stored, pointed at or pushed into a container. A parameter of type Fn is treated as one, which answers "passed to something that stores it" with no interprocedural analysis -- the store is refused inside the callee. Everything of type CFn is clean for free, which is the second thing having two types buys. Every refusal names case 3, the environment the collector owns. Two pre-existing bugs fell out on the way. A lifted fn asked for Fnval, so `flan reload' on any function containing an fn literal died at llc with an undefined cell; it takes Flanfn now, which is the choice a handler clause always made. And a redefinition module now carries its own hidden copy of every thunk it names, which is the same bug shape caught before it shipped. --- lib/ast.ml | 7 +- lib/check.ml | 801 ++++++++++++++++++++++++--- lib/cimport.ml | 4 +- lib/dev.ml | 2 +- lib/emit.ml | 229 ++++++-- lib/js.ml | 14 +- lib/load.ml | 7 +- lib/parse.ml | 11 +- lib/reach.ml | 7 +- lib/session.ml | 24 +- lib/tast.ml | 67 ++- lib/types.ml | 51 +- lib/x86.ml | 179 ++++-- runtime/flan_rt.c | 19 +- test/programs/fn-capture-dyn.flan | 13 + test/programs/fn-capture-set.flan | 10 + test/programs/fn-capture.flan | 121 +++- test/programs/fn-cfn-captures.flan | 10 + test/programs/fn-cfn-narrow.flan | 13 + test/programs/fn-cfn.flan | 67 +++ test/programs/fn-escape-copy.flan | 19 + test/programs/fn-escape-handled.flan | 12 + test/programs/fn-escape-param.flan | 11 + test/programs/fn-escape-return.flan | 10 + test/programs/fn-escape-store.flan | 7 + test/programs/fn-escape-vec.flan | 9 + test/programs/fn-extern.flan | 6 +- test/programs/fn-in-struct.flan | 11 + test/programs/fn-no-type.flan | 4 + test/programs/fn-values.flan | 12 +- test/reload_host.c | 5 +- test/test_acceptance.ml | 78 ++- test/test_flan.ml | 33 +- test/test_session.ml | 2 +- 34 files changed, 1654 insertions(+), 221 deletions(-) create mode 100644 test/programs/fn-capture-dyn.flan create mode 100644 test/programs/fn-capture-set.flan create mode 100644 test/programs/fn-cfn-captures.flan create mode 100644 test/programs/fn-cfn-narrow.flan create mode 100644 test/programs/fn-cfn.flan create mode 100644 test/programs/fn-escape-copy.flan create mode 100644 test/programs/fn-escape-handled.flan create mode 100644 test/programs/fn-escape-param.flan create mode 100644 test/programs/fn-escape-return.flan create mode 100644 test/programs/fn-escape-store.flan create mode 100644 test/programs/fn-escape-vec.flan diff --git a/lib/ast.ml b/lib/ast.ml index c9cab783..a24d2b78 100644 --- a/lib/ast.ml +++ b/lib/ast.ml @@ -18,7 +18,12 @@ and texpr_kind = | Tarray of len * texpr (* [4 f32] [rows [cols u32]] *) | Tmap of texpr * texpr (* (Map string i32) *) | Tapp of string * texpr list (* (Ptr Cursor) (Option f64) *) - | Tfn of texpr list * texpr (* (Fn [a a] bool) *) + (* (Fn [a a] bool) and (CFn [a a] bool). The flag is whether the value + carries an environment: true for [Fn], false for [CFn]. One case + rather than two because everything that walks a type expression treats + them identically — the difference is a fact about the value, and it is + [Check.resolve] that turns it into one. *) + | Tfn of bool * texpr list * texpr (* An array length is an integer or a compile-time constant's name. *) and len = diff --git a/lib/check.ml b/lib/check.ml index f523457b..83feaf4c 100644 --- a/lib/check.ml +++ b/lib/check.ml @@ -503,6 +503,34 @@ type ctx = { refusal below says why the enclosing function's locals are not there. Both are the same gap: capture does not exist. *) mutable outer_what : string option; + (* What this body has captured out of [outer], in the order it first named + each one: the source name, the *outer* binding the copy is taken from, + and the slot in this body's own frame the copy is read into. Empty + everywhere [outer] is, which is everywhere but a lifted body. + + The order is the environment's field order, so it is the order the copy + is made in and the order the body reads it back in. First-reference order + rather than declaration order because it is the only one this pass has: + the outer scope is a list and the names on it were not all written for + this body's sake. *) + mutable caught : (string * (binding * int)) list; + (* The context this body was lifted out of, so that capture can be + transitive: an [fn] inside an [fn] naming a local of the function both + were written in is captured by the middle one and then by the inner one + out of the middle one's copy. Without it the inner body would see only + what the middle body happened to have named already, which is a rule + about the text and not about the scope. + + It is safe to reach into while it is on the stack, and only then: a + lifted body is checked at the point it is written, so the parent is + paused exactly there and its scope is the snapshot [outer] holds. *) + parent : ctx option; + (* The slot the environment pointer arrives in, minted the first time + something is captured and [None] until then. Nameless, so the break loop + hides it the way it hides every other slot the compiler made: what a + reader wants to see is the copies, and those are under the names the + source gave them. *) + mutable envslot : int option; (* True wherever handler or restart frames established by this function are on the stack. A [return] from there would leave them pointing into a frame that has gone, so it is refused — the same rule as [defer] inside a @@ -582,26 +610,115 @@ let bind ctx ?what name bty ~assignable = let lookup ctx name = List.assoc_opt name ctx.scope -(* A handler clause is lifted into a function of its own, so the establishing - function's locals are simply not there. Capturing them is a closure with an - explicit environment — spec-memory.md's case 2, a non-escaping [fn] capturing - by value into a stack environment, since a handler frame does not outlive the - function that pushed it — and until that exists a reference to one is refused - for the reason it is really refused for, rather than as a name nobody has - heard of. *) -let captured ctx loc name = - match ctx.outer_what with - | Some what when List.mem_assoc name ctx.outer -> - let why = - if String.equal what "a handler" then - "Use a global, or pass it on the condition" - else - "Pass it in, or use a global" +(* Capture, spec-memory.md's case 2: a body lifted into a function of its own — + an [fn] literal or a handler clause — naming a local of the function it was + written in. + + The copy is taken where the value is made and not where it is read, so the + name in the body means what the local held at that instant and nothing + later can change it. What makes that safe is the extent: the copies live in + a slot of the *enclosing* frame, and a value holding their address may not + outlive it. [escapes] is the whole of what enforces that, and it is where + the escaping half — an environment the collector allocates — is named. + + Answers the binding the body should use, minting one on first reference: + a named slot of this body's own frame, which the prologue fills from the + environment. Not assignable, and that is not an omission — see [captured_set]. + + A dyn is refused, and for the reason a struct field of dyn already is (see + the [defstruct] arm): the collector's roots are frames, a captured copy + lives inside a struct the checker synthesised, and nothing pushes the + fields of one. A dyn in there would be a live value reachable only through + memory the marker never walks. Milestone 2's per-type descriptors lift it, + alongside the condition payload's and the struct field's. *) +let rec capture ctx loc name = + match List.assoc_opt name ctx.caught with + (* Already captured, and named again from a scope that no longer lists it — + a [let] inside the body restores what it displaced, and the copy's + binding goes with it. One field, not two: the environment is keyed by the + source name. *) + | Some (outer, slot) -> Some { slot; bty = outer.bty; assignable = false; bwhat = None } + | None -> + let from_parent () = + (* Not a local of the body directly around this one, so ask whether that + body can capture it in turn. The middle one takes a copy and this one + takes a copy of that — which is the same value, because every copy on + the way was taken at the moment its own value was made, and those + moments are nested. *) + match ctx.parent with + | Some p -> capture p loc name + | None -> None in - Loc.failk "check/capture" loc - "%s cannot see %s — it is a local of the enclosing function. %s" - what name why - | _ -> () + let outer = + match List.assoc_opt name ctx.outer with + | Some b -> Some b + | None -> if ctx.outer_what = None then None else from_parent () + in + match ctx.outer_what, outer with + | Some what, Some (outer : binding) -> + if outer.bty = Types.Dyn then + Loc.failk "check/capture-dyn" loc + "%s cannot capture %s: it is a dyn, and the collector finds its \ + roots by frame — a copy inside the environment %s is handed would \ + be a live value nothing walks. Pass it in as a parameter, or hold \ + it in a global" + what name what; + let slot = bind ctx name outer.bty ~assignable:false in + ctx.caught <- ctx.caught @ [ (name, (outer, slot)) ]; + Some { slot; bty = outer.bty; assignable = false; bwhat = None } + | _ -> None + +(* The one thing [capture] does not answer for. A captured name is a copy, so + a store into it would change this body's copy and leave the local it came + from as it was — which is a silent disagreement and not a feature. The + ordinary "not assignable" message would name the wrong reason, so this one + names the right one. + + [and] rather than a second [let] only so the two read together; neither + calls the other. *) +(* What [capture] would find, without capturing it. A guard that has to know + the *type* of an enclosing local before deciding what a form means — a + name in head position is a call through a value only if the value is a + function — must not take a copy on the way to answering. The binding it + answers with is only good for its type unless it came out of [caught]. *) +and peek_outer ctx name = + if ctx.outer_what = None then None + else + match List.assoc_opt name ctx.caught with + | Some ((b : binding), slot) -> Some { slot; bty = b.bty; assignable = false; bwhat = None } + | None -> + match List.assoc_opt name ctx.outer with + | Some b -> Some b + | None -> + match ctx.parent with Some p -> peek_outer p name | None -> None + +and outer_local ctx name = + ctx.outer_what <> None + && (List.mem_assoc name ctx.caught + || List.mem_assoc name ctx.outer + || (match ctx.parent with + | Some p -> outer_local p name + | None -> false)) + +and captured_set ctx loc name = + if outer_local ctx name then + match ctx.outer_what with + | Some what -> + Loc.failk "check/capture-set" loc + "%s cannot assign to %s: it is a copy of the enclosing function's \ + local, taken where the value was made, so a store here would change \ + the copy and leave %s as it was. %s" + what name name + (if String.equal what "a handler" then + "Accumulate into a global, or put the value on the condition — \ + capture is by value, which is what lets the copy be read at all" + else + "Return the new value, or keep it in a local of this fn") + | None -> () + +(* A binding this body captured, as opposed to one it declared. Used where the + difference matters and nowhere else. *) +let is_captured ctx name = List.mem_assoc name ctx.caught let scoped ctx f = let saved = ctx.scope in @@ -873,6 +990,13 @@ let map_type ?(preds = []) loc (k : Types.t) (v : Types.t) = (* The positions a function value may not be written in, and the one reason they are all the same position: something zeroes it. + Since capture arrived there is a second reason standing behind the first, + and it is the sharper one: every position on this list outlives the frame + a captured environment is on, so even a value nobody zeroed could not be + kept there. The message names the zero because that is the one that applies + to *every* function value and not only to a capturing one — and the escape + is what [escape_check] says, at the store rather than at the declaration. + ZII is the language's rule — an omitted struct field, a fixed array's elements, a [defonce] with no initialiser are all all-bytes-zero — and a zeroed function value is a null pointer with a signature on it, which is the @@ -883,9 +1007,21 @@ let map_type ?(preds = []) loc (k : Types.t) (v : Types.t) = A parameter, a return type, a [let] binding and an [(Option (Fn ...))] are not on the list: none of them is ever conjured, and an [Option]'s zero is a [None] whose tag nobody may look past. *) +(* The two function types as one question, for every place that wants the + signature and does not care which of them carries an environment: a call + site, a shadowing guard, a walk over the parameters. Where the difference + matters it is matched on directly, and there are few such places — the + representation is [Emit]'s business and the coercion is [expect]'s. *) +let fn_sig (t : Types.t) = + match t with + | Types.Fn (ps, r) | Types.CFn (ps, r) -> Some (ps, r) + | _ -> None + +let callable_ty t = fn_sig t <> None + let rec no_zeroed_fn loc what (t : Types.t) = match t with - | Types.Fn _ -> + | Types.Fn _ | Types.CFn _ -> fail loc "%s cannot be %s — it would be zeroed, and a zeroed function value is a \ null pointer. Pass it as a parameter, or hold it in a let" @@ -986,17 +1122,27 @@ let rec resolve env ?(seen = []) (t : Ast.texpr) : Types.t = | Ast.Tmap (k, v) -> map_type ~preds:env.tvpreds loc (resolve env ~seen k) (resolve env ~seen v) - (* (Fn [T ...] R): a function value, which is one code address and no - environment beside it. There is no capture — [check_fn] refuses a - reference to an enclosing local by name — so this is a pointer with a - signature and nothing about it can dangle. + (* (Fn [T ...] R) is a code address and the environment it is called with: + two words. A value made out of a name carries a null there; one made out + of an [fn] that captures carries the address of the copies on the frame + it was written in, and [escape_check] is what stops that address + outliving the frame. + + (CFn [T ...] R) is the address alone, one word, and nothing that can + capture — see [Types] for why the C is information rather than + decoration, and for why it is not yet a capability. Nobody needs it: + [Fn] accepts everything, and the commonest reason to reach for the + narrow one is that a *named* function handed to an [Fn] pays a hop + through the widening thunk where a [CFn] is a direct call. Where one may be *written* is narrower than where the type resolves, and the two rules live apart on purpose: this is what the spelling means, and [no_zeroed_fn] is where a position that would zero one is refused. A - parameter, a return type and a let binding are the positions that work. *) - | Ast.Tfn (ps, r) -> - Types.Fn (List.map (resolve env ~seen) ps, resolve env ~seen r) + parameter, a return type and a let binding are the positions that work, + for both. *) + | Ast.Tfn (env', ps, r) -> + let ps = List.map (resolve env ~seen) ps and r = resolve env ~seen r in + if env' then Types.Fn (ps, r) else Types.CFn (ps, r) | Ast.Tapp (name, args) -> (match name, args with | "Ptr", [ a ] -> Types.Ptr (resolve env ~seen a) @@ -1554,7 +1700,7 @@ let signature_tyvars (fn : Ast.fn) = and a variable cannot stand there: this spike is generic over types, not over type constructors. A [$t] inside the arguments is ordinary. *) | Ast.Tapp (_, args) -> List.iter ty args - | Ast.Tfn (ps, r) -> List.iter ty ps; ty r + | Ast.Tfn (_, ps, r) -> List.iter ty ps; ty r in List.iter (fun (p : Ast.field) -> ty p.Ast.fty) fn.Ast.params; (match fn.Ast.ret with Some r -> ty r | None -> ()); @@ -1595,6 +1741,8 @@ let rec subst_ty subst (t : Types.t) = | Types.Vec e -> Types.Vec (subst_ty subst e) | Types.Option e -> Types.Option (subst_ty subst e) | Types.Fn (ps, r) -> Types.Fn (List.map (subst_ty subst) ps, subst_ty subst r) + | Types.CFn (ps, r) -> + Types.CFn (List.map (subst_ty subst) ps, subst_ty subst r) | t -> t (* Does this resolved type still mention a variable? *) @@ -1604,7 +1752,8 @@ let rec generic_ty (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> generic_ty e | Types.Map (k, v) -> generic_ty k || generic_ty v - | Types.Fn (ps, r) -> List.exists generic_ty ps || generic_ty r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists generic_ty ps || generic_ty r | _ -> false (* Does a type a call site bound a variable to reach a [dyn] anywhere? See the @@ -1617,7 +1766,8 @@ let rec reaches_dyn (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> reaches_dyn e | Types.Map (k, v) -> reaches_dyn k || reaches_dyn v - | Types.Fn (ps, r) -> List.exists reaches_dyn ps || reaches_dyn r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists reaches_dyn ps || reaches_dyn r | _ -> false (* The refusal plan.org's Types section asks for, in one place so that every @@ -1662,6 +1812,9 @@ let rec mangle_ty (t : Types.t) = | Types.Fn (ps, r) -> Printf.sprintf "fn-%s-to-%s" (String.concat "-" (List.map mangle_ty ps)) (mangle_ty r) + | Types.CFn (ps, r) -> + Printf.sprintf "cfn-%s-to-%s" + (String.concat "-" (List.map mangle_ty ps)) (mangle_ty r) | t -> Types.to_string t (* ── The runaway instantiation, refused by name rather than by depth ──── @@ -1697,7 +1850,7 @@ let rec occurs_in ~needle (t : Types.t) = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> occurs_in ~needle e | Types.Map (k, v) -> occurs_in ~needle k || occurs_in ~needle v - | Types.Fn (ps, r) -> + | Types.Fn (ps, r) | Types.CFn (ps, r) -> List.exists (occurs_in ~needle) ps || occurs_in ~needle r | _ -> false @@ -1766,6 +1919,69 @@ let mk loc ty e : Tast.expr = { Tast.e; ty; loc } let unit_at loc = mk loc Types.Unit Tast.Unit +(* The environment for a lifted body, built once its own body has been checked + and [caught] is therefore final. spec-memory.md's case 2, and the whole of + its machinery. + + It is a struct the checker synthesises — one field per captured name, in + first-reference order — and it is registered in the same table a + [defstruct] goes in, so both backends lay it out with the calculator they + already have and neither learns a new shape. Its name is the lifted + function's, which is unique and stable for the reason that name is: a + redefinition module emits the lifted functions belonging to the bodies it + replaces, and it emits their environments with them. + + Two ends, and the copy is at the near one: + + - in the *enclosing* frame, a slot holding the struct, filled with a [Make] + of the outer locals. That store is the copy, and it happens where the + value is made. A literal written inside a loop stores into the same slot + each time round, so each value is made from the locals as they were on + its own iteration. + - in the *lifted* frame, a slot holding the pointer, and a [Let] around the + whole body reading each field back into the named slot the body has been + checked against. Once, at entry, for the same reason a handler clause + binds its condition once: what the body names is the copy and not an + address, so nothing downstream has to know an environment exists. + + Both new slots are nameless, which is how the break loop is told to hide + them: a reader wants the captured copies, and those are the named slots the + body reads them into. Answers the prefixed body, the slot the pointer + arrives in, the enclosing frame's binding, and the address to put in the + value. *) +let close_over ~fname (octx : ctx) (fctx : ctx) loc = + match fctx.caught with + | [] -> (fun body -> body), None, None, None + | caught -> + let ename = "env/" ^ fname in + let fields = + List.map + (fun (n, ((b : binding), _)) -> { Tast.fname = n; fty = b.bty }) + caught + in + Hashtbl.replace fctx.env.structs ename { Tast.sname = ename; fields }; + let ety = Types.Named ename in + let eslot = fresh_slot fctx (Types.Ptr ety) in + let binds = + List.mapi + (fun i (_, ((b : binding), slot)) -> + let p = mk loc (Types.Ptr ety) (Tast.Local eslot) in + (slot, mk loc b.bty (Tast.Field (mk loc ety (Tast.Deref p), i)))) + caught + in + let prefix body = [ mk loc fctx.ret (Tast.Let (binds, body)) ] in + let mslot = fresh_slot octx ety in + let make = + mk loc ety + (Tast.Make (ename, + List.map + (fun (_, ((b : binding), _)) -> + mk loc b.bty (Tast.Local b.slot)) + caught)) + in + prefix, Some eslot, Some (mslot, make), + Some (mk loc (Types.Ptr ety) (Tast.Addr (Tast.Plocal mslot))) + (* A source location as a value, for a runtime trap that has to name the site rather than the runtime. The bounds and slice traps get theirs from [Emit], which renders the [Loc.t] it is already carrying; a trap reached through a @@ -2269,7 +2485,7 @@ let box loc (e : Tast.expr) : Tast.expr = caller that starts doing that gets a sentence instead of a silent mis-lowering. *) | Types.Named _ | Types.Enum _ | Types.Option _ | Types.Ptr _ - | Types.Alloc | Types.Fn _ | Types.Var _ -> + | Types.Alloc | Types.Fn _ | Types.CFn _ | Types.Var _ -> no_dyn_yet loc ~into:true e.Tast.ty "" let unbox loc (want : Types.t) (e : Tast.expr) : Tast.expr = @@ -2468,6 +2684,59 @@ let unbox_option ctx loc (t : Types.t) (got : Tast.expr) : Tast.expr = Written once and used by both refusals that can report one — [expect]'s, and the binary operators' when their two operands have no join. *) +(* ── The widening thunk ──────────────────────────────────────────────── + One per signature, and the whole of what a [CFn] costs on its way into + an [Fn]. Its parameters are the signature's, it declares the environment, + and its body calls through what it finds there — because that is where the + original bare address was put. + + Which is the move that makes the two conventions meet in exactly one + place. Every body reachable through an [Fn] value declares the trailing + environment, so every indirect call is exactly typed and nothing anywhere + relies on a callee ignoring an argument it never declared. That was the + first design's hinge and it does not survive wasm32: [call_indirect] + compares the signature at the call and a spare argument is a trap. + + Per *signature* and not per name, so a program pays one small function per + distinct shape it widens rather than one per function it widens. The name + is derived from the type, so two widenings of the same shape share it and + the memo below finds it — the same arrangement [struct_key_pair] uses for + a map's hash and equality pair, and for the same reason. + + [fparent] is []: not a name anyone wrote, so [defs] hides it, and a + marker the redefinition modules match on to carry a copy of their own. *) +let thick_thunk env loc ps r = + let name = "thick/" ^ mangle_ty (Types.CFn (ps, r)) in + if not (List.exists (fun (f : Tast.fn) -> f.Tast.name = name) env.lifted) + then begin + let n = List.length ps in + let fty = Types.CFn (ps, r) in + let args = List.mapi (fun i t -> mk loc t (Tast.Local i)) ps in + let callee = mk loc fty (Tast.Local n) in + env.lifted <- + { Tast.name; params = ps; + slots = Array.of_list (ps @ [ fty ]); + snames = Array.make (n + 1) None; + ret = r; body = [ mk loc r (Tast.CallPtr (callee, args)) ]; + fdefers = []; fenv = Some n; fparent = Some ""; floc = loc } + :: env.lifted + end; + name + +(* The slot a lifted body's environment arrives in, minted when the body did + not capture anything and so has none of its own. + + Every body that can be *reached* through an [Fn] value declares the + parameter, whether or not it reads it: a lifted [fn] literal in an [Fn] + position, and every handler clause, since [flan_signal] passes the frame's + environment to all of them. The alternative is a call whose signature is + one argument longer than the callee's, which SysV tolerates and wasm32 + does not. The slot is nameless, so the break loop hides it, and a body + that never reads it costs one store the optimiser drops. *) +let declare_env ctx = function + | Some _ as s -> s + | None -> Some (fresh_slot ctx (Types.Ptr Types.Unit)) + let numeric_note ~(want : Types.t) ~(got : Types.t) = if not (Types.is_numeric want && Types.is_numeric got) then "" else if Types.widens_to ~from:want ~into:got then @@ -2521,6 +2790,14 @@ let expect ctx loc ~want (got : Tast.expr) = honest — it admits only conversions that cannot change the number, so the cast this inserts is one no program can tell happened. *) | _ when Types.widens_to ~from:got.Tast.ty ~into:w -> widen loc w got + (* The other widening, and the only coercion between the two function + types. A bare address satisfies a signature that asks for an + environment. It goes this way only — an [Fn] has an environment and a + [CFn] has nowhere to put one — so the reverse falls through to the + ordinary refusal, which names both types and is the right sentence. *) + | Types.Fn (ps, r), Types.CFn (ps', r') + when Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) -> + mk loc w (Tast.Thicken (thick_thunk ctx.env loc ps r, got)) | _ -> got in if Types.fits ~expected:w ~actual:got.Tast.ty then got @@ -2598,7 +2875,7 @@ let hash_ty = Types.Int Types.U64 would share a slot counter. *) let invented_ctx env ret = { env; ret; slots = 0; slot_tys = []; slot_names = []; scope = []; - defers = []; defer_slot = None; outer = []; outer_what = None; in_frames = None; loops = []; tail = false; + defers = []; defer_slot = None; outer = []; outer_what = None; caught = []; envslot = None; parent = None; in_frames = None; loops = []; tail = false; in_defer = false; defer_ok = false; defer_block = "a nested form"; owner = "" } @@ -2711,7 +2988,7 @@ and struct_key_pair env loc n = let placeholder name ret params = { Tast.name; params; slots = Array.of_list params; snames = Array.make (List.length params) None; - ret; body = []; fdefers = []; fparent = None; floc = loc } + ret; body = []; fdefers = []; fenv = None; fparent = None; floc = loc } in env.lifted <- placeholder hname hash_ty hparams @@ -2794,7 +3071,7 @@ and struct_key_pair env loc n = { Tast.name; params; slots = Array.of_list (List.rev ctx.slot_tys); snames = Array.of_list (List.rev ctx.slot_names); - ret; body; fdefers = []; fparent = None; floc = loc } + ret; body; fdefers = []; fenv = None; fparent = None; floc = loc } in env.lifted <- finish hname hash_ty hparams hctx hbody @@ -3440,6 +3717,15 @@ and var ctx ?(qualified = false) loc ~want name = match lookup ctx name with | Some b -> expect ctx loc ~want (mk loc b.bty (Tast.Local b.slot)) + (* A local of the enclosing function, in a body that was lifted out of it: + captured by value, here, where it is first named. Asked *before* the + globals, because that is what the name means at the place it is + written — inside the enclosing function a local shadows a global of the + same name, and a body lifted out of it must not silently mean something + else. *) + | None when capture ctx loc name <> None -> + let b = Option.get (capture ctx loc name) in + expect ctx loc ~want (mk loc b.bty (Tast.Local b.slot)) | None -> match Hashtbl.find_opt ctx.env.globals name with | Some (ty, _) -> @@ -3482,8 +3768,9 @@ and var ctx ?(qualified = false) loc ~want name = "%s is a foreign function, and its address is not a Flan \ function value. Wrap it in a defn and pass that" name; expect ctx loc ~want - (mk loc (Types.Fn (params, ret)) (Tast.FnAddr (Tast.Fnval name))) - | None -> captured ctx loc name; unknown_name ctx loc name) + (mk loc (Types.CFn (params, ret)) + (Tast.FnAddr (Tast.Fnval name))) + | None -> unknown_name ctx loc name) (* What remains of spec-memory.md's ownership section after the repeals of 2026-09-18 is the allocator's side alone: the region rule decides where a @@ -3533,14 +3820,15 @@ and block ctx ?want ?(defer_ok = false) loc body = landed, and the surface feature is that machinery given a name rather than a second one invented beside it. - **No capture, and that is the scope of this milestone.** The body sees its - parameters and the program's globals and nothing else; a reference to a - local of the enclosing function is refused by name (see [captured]) rather - than resolved to something it did not mean. That is what makes the value a - bare code address with no environment behind it, which in turn is what makes - it safe to pass down, return, and store: there is nothing that can outlive - anything. spec-memory.md's capture cases, and escaping closures with them, - stay deferred. + **Capture is by value, and the value may not escape.** The body sees its + parameters, the program's globals, and the locals of the function it was + written in — those last copied into an environment on that function's + frame at the instant the value is made (see [capture] and [close_over]). + So the value is two words, the second of them an address into a frame, and + what keeps that address good is [escape_check]: it may be called, passed + down and let-bound, and may not be returned, stored or pushed anywhere. + spec-memory.md's case 3 — an environment the collector owns, and with it + the escaping closure — is a separate lane, and every refusal names it. **The parameter types come from the position.** [Ast.Fn] carries names and no types — that is the surface syntax, not an omission here — so an fn is @@ -3553,26 +3841,36 @@ and check_fn ctx ~want ?gen loc (params : string list) body = [(Fn ...)] want to say so, and the return is the annotated element type, or [None] to take the body's own. Everything else threads [want]. The caller has already checked the arity, in its own words. *) + (* Which of the two function types was asked for. [CFn] is a bare + address, so a literal written into one has nowhere to put an + environment — it is checked exactly as an [Fn] is and then refused *if + it turned out to capture*, which is a decision only the finished body + can make. Nothing else differs. *) + let bare = match want with Some (Types.CFn _) -> true | _ -> false in let pts, ret0 = match gen with | Some (pts, r) -> pts, r | None -> - match want with - | Some (Types.Fn (ps, r)) when List.length ps = List.length params -> + match Option.map fn_sig want with + | Some (Some (ps, r)) when List.length ps = List.length params -> ps, Some r - | Some (Types.Fn (ps, r)) -> + | Some (Some (ps, r)) -> fail loc "this fn has %d parameter%s and %s was wanted here" (List.length params) (if List.length params = 1 then "" else "s") - (Types.to_string (Types.Fn (ps, r))) - | Some other when other <> Types.Never -> - fail loc "expected %s, found an fn" (Types.to_string other) + (Types.to_string + (if bare then Types.CFn (ps, r) else Types.Fn (ps, r))) | _ -> - fail loc - "nothing here says what this fn's parameters are — an fn takes its \ - types from the position it is written in. Write it as an argument \ - whose parameter is a (Fn [T ...] R)" + match want with + | Some other when other <> Types.Never -> + fail loc "expected %s, found an fn" (Types.to_string other) + | _ -> + fail loc + "nothing here says what this fn's parameters are — an fn takes \ + its types from the position it is written in. Write it as an \ + argument whose parameter is a (Fn [T ...] R), or a \ + (CFn [T ...] R) when it captures nothing" in (* Its own frame and its own empty scope, with [outer] kept only so that a reference to the enclosing function's locals is refused for the reason it @@ -3582,7 +3880,8 @@ and check_fn ctx ~want ?gen loc (params : string list) body = is an expression, and no machinery is built for the form nobody writes. *) let fctx = { (invented_ctx ctx.env (Option.value ret0 ~default:Types.Unit)) with - outer = ctx.scope; outer_what = Some "an fn"; owner = ctx.owner } + outer = ctx.scope; outer_what = Some "an fn"; parent = Some ctx; + owner = ctx.owner } in List.iter2 (fun n t -> ignore (bind fctx n t ~assignable:false)) params pts; @@ -3635,25 +3934,73 @@ and check_fn ctx ~want ?gen loc (params : string list) body = in Printf.sprintf "fn/%s/%d" ctx.owner (List.length mine) in + (* And the environment, now that the body has named everything it is going + to. [close_over] allocates in both frames, so it runs after the body's + slots and before the lifted function is recorded. *) + let prefix, fenv, bind, addr = close_over ~fname ctx fctx loc in + (* A [CFn] is a bare address and has nowhere to keep an environment, so a + literal that captured cannot be one. Refused with the name of what it + captured, because that is the fact the writer has to act on — and with + the fix named, which is the wider type. *) + if bare && fctx.caught <> [] then begin + let names = List.map fst fctx.caught in + Loc.failk "check/cfn-captures" loc + "this fn captures %s, so it is a (Fn [%s] %s) and not a (CFn [%s] \ + %s): a CFn is the bare address, one word, with nowhere for the \ + copies to live. Widen the position to Fn, or pass %s in as a parameter" + (String.concat ", " names) + (String.concat " " (List.map Types.to_string pts)) (Types.to_string ret) + (String.concat " " (List.map Types.to_string pts)) (Types.to_string ret) + (match names with [ n ] -> n | _ -> "them") + end; + (* An [Fn]-position literal declares the environment whether or not it + captured: it is reached by a call that passes one. A [CFn]-position + one must not — it is reached by calls that pass none, and a parameter + nobody supplies is read off whatever the register held. *) + let fenv = if bare then fenv else declare_env fctx fenv in ctx.env.lifted <- { Tast.name = fname; params = pts; slots = Array.of_list (List.rev fctx.slot_tys); snames = Array.of_list (List.rev fctx.slot_names); - ret; body = fbody; fdefers = []; - fparent = Some ctx.owner; floc = loc } + ret; body = prefix fbody; fdefers = []; + fenv; fparent = Some ctx.owner; floc = loc } :: ctx.env.lifted; - expect ctx loc ~want - (mk loc (Types.Fn (pts, ret)) (Tast.FnAddr (Tast.Fnval fname))) + let fty = if bare then Types.CFn (pts, ret) else Types.Fn (pts, ret) in + (* [Flanfn] and not [Fnval], which is the handler clause's choice and is the + same choice for the same reason. [Fnval] exists so that a *name* taken as + a value in a dev build answers with the body that is current, which means + a load from that name's indirection cell. A lifted body has no name + anyone can type and no way to be redefined on its own: it is reached by + address from the body it was written in, and a redefinition of that body + carries its own copy. So the cell would never hold anything but this + symbol, and asking for one is how a redefinition module came to reference + a cell nothing declares. *) + let v = + match addr with + | None -> mk loc fty (Tast.FnAddr (Tast.Flanfn fname)) + | Some a -> + (* The value, and the store that fills its environment around it. What + stops the value leaving this frame is [escape_check], which reads the + finished body: a rule about where a value may *go* cannot be settled + at the point it is made. *) + let c = mk loc fty (Tast.Closure (Tast.Flanfn fname, a)) in + mk loc fty (Tast.Let ([ Option.get bind ], [ c ])) + in + expect ctx loc ~want v (* A handler runs where the *signal* was, not where it was established, so it cannot be a branch in the function that wrote it: it is lifted into a function of its own and reached through a pointer. - Which means it cannot see the establishing function's locals. Capturing them - is a closure with an explicit environment — the non-escaping kind, captured - by value onto this frame — and until that exists a reference to one is - rejected by name rather than silently resolving to something else. Globals and the condition itself are - in scope, which is enough for the accumulation case §1 is about. + It *can* see the establishing function's locals, by value: the same capture + an [fn] literal gets, and the one place it has no escaping case left over. + A handler frame is popped by the body that pushed it and nothing in the + language can name one, so the establishing frame is alive whenever the + clause runs and the copies on it are good. What is still refused is a store + into a captured name — the clause holds a copy, and writing to it would + leave the local as it was — so §1's accumulation case still accumulates + into a global, and now with whatever the establishing function knew + readable beside it. The body may not [return] either. The frames are pushed and popped around it, and an early exit would leave them on the stack pointing into a function @@ -3680,7 +4027,8 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = the enclosing one. *) let hctx = { (invented_ctx ctx.env Types.Unit) with - outer = ctx.scope; outer_what = Some "a handler" } + outer = ctx.scope; outer_what = Some "a handler"; + parent = Some ctx } in (* The condition crosses as a pointer, because the handler runs while the signalling frame is still alive and there is nothing to copy. @@ -3716,16 +4064,35 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = in Printf.sprintf "handler/%s/%d/%s" ctx.owner (List.length mine) name in + (* And the environment, the same machinery an [fn] literal's capture + uses and settled by the same argument — only with no escaping + case to leave over. A handler frame is popped by the body that + pushed it and nothing in the language can name one, so the clause + cannot be reached from anywhere the establishing frame is not + alive. There is nothing here that case 3 would change. *) + let prefix, fenv, bind, addr = + close_over ~fname ctx hctx c.Ast.hloc + in + (* Every clause declares the environment, captured or not: + [flan_signal] reads it off the frame and passes it to whichever + clause matched, and it cannot know which of them captured. *) + let fenv = declare_env hctx fenv in ctx.env.lifted <- { Tast.name = fname; params = [ Types.Ptr ty ]; slots = Array.of_list (List.rev hctx.slot_tys); snames = Array.of_list (List.rev hctx.slot_names); - ret = Types.Unit; body = hbody; fdefers = []; - fparent = Some ctx.owner; floc = c.Ast.hloc } + ret = Types.Unit; body = prefix hbody; fdefers = []; + fenv; fparent = Some ctx.owner; floc = c.Ast.hloc } :: ctx.env.lifted; - { Tast.htype = type_id name; hfn = fname }) + { Tast.htype = type_id name; hfn = fname; henv = addr }, bind) clauses in + (* The stores that fill the environments, one per clause that captured, + around the whole form: a handler frame carries the address and the frame + is pushed before the body runs, so the copies have to be made before + either. *) + let envbinds = List.filter_map snd frames in + let frames = List.map fst frames in (* The flag is set on [ctx] itself and restored, not on a copy: [ctx.slots] and [ctx.slot_tys] are mutable, so a copy would allocate the body's slots into a record the function never sees again and the indices would @@ -3765,7 +4132,11 @@ and check_handler_bind ctx ?want ?(what = "handler-bind") loc clauses body = go body) in ctx.in_frames <- saved; - expect ctx loc ~want (mk loc ty (Tast.Handled (frames, body))) + let h = mk loc ty (Tast.Handled (frames, body)) in + let h = + if envbinds = [] then h else mk loc ty (Tast.Let (envbinds, [ h ])) + in + expect ctx loc ~want h (* (restart-case BODY (name [] BODY-1) ...) — spec-conditions.md §3 and §6. @@ -5064,8 +5435,10 @@ and check_array_gen ctx ~want loc dims f = | _ -> check ctx f in let elem = - match f.Tast.ty with - | Types.Fn (ps, r) -> + (* Either function type: the generator is called and nothing here cares + whether an environment rides along. *) + match fn_sig f.Tast.ty with + | Some (ps, r) -> let got = List.length ps in if got <> rank then fail f.Tast.loc @@ -5081,12 +5454,12 @@ and check_array_gen ctx ~want loc dims f = (k + 1) (Types.to_string p)) ps; r - | other -> + | None -> fail f.Tast.loc "array-gen's second element is a function value, called once per \ element with one i32 index per dimension, and this is %s — for one \ value repeated, write array-fill" - (Types.to_string other) + (Types.to_string f.Tast.ty) in no_zeroed_fn loc "a fixed array's element" elem; let fs = fresh_slot ctx f.Tast.ty in @@ -5468,14 +5841,29 @@ and refuse_string_place loc (ty : Types.t) = and check_place ctx loc (p : Ast.place) : Tast.place * Types.t = match p with | Ast.Pvar name -> + (* Scope first, and the capture refusal only where scope did not settle + it. A body's *own* [let] may shadow a name the enclosing function also + has, and a store into that one is an ordinary store — asking about the + capture before looking would refuse it with a message about a copy that + is not the thing being written to. *) (match lookup ctx name with | Some b -> - if not b.assignable then + if not b.assignable then begin + (* Not assignable, so it is either a parameter or a captured copy. + Which one decides the message, and the copy's reason is its + own. *) + (match List.assoc_opt name ctx.caught with + | Some (_, slot) when slot = b.slot -> captured_set ctx loc name + | _ -> ()); fail loc "%s is a parameter, and a parameter is not assignable — bind a \ - local with let" name; + local with let" name + end; Tast.Plocal b.slot, b.bty | None -> + (* Not in scope here at all: a name of the enclosing function, which a + lifted body may read as a copy and may not write to. *) + captured_set ctx loc name; match Hashtbl.find_opt ctx.env.globals name with | Some (_, true) -> (* Four words, before: the name and the fact, and nothing about what @@ -5492,7 +5880,7 @@ and check_place ctx loc (p : Ast.place) : Tast.place * Types.t = "%s is a constant, and a constant is not assignable. Declare it \ with defonce if it has to change" name | Some (ty, false) -> Tast.Pglobal name, ty - | None -> captured ctx loc name; unknown_name ~setting:true ctx loc name) + | None -> unknown_name ~setting:true ctx loc name) | Ast.Pfield (target, name) -> let target, sname = struct_target ctx target in let s = Option.get (fields_named ctx.env sname) in @@ -5609,8 +5997,11 @@ and check_call ctx ~want loc (head : Ast.expr) (args : Ast.expr list) = above and by a name that resolved to a local or a parameter of function type, which is the shape every caller of [map] has. *) and call_value ctx ~want loc (callee : Tast.expr) args = - match callee.Tast.ty with - | Types.Fn (params, ret) -> + (* Either function type: calling one is calling the other, and the + difference — whether an environment rides along — is the backend's to + lower. Nothing here has to know which. *) + match fn_sig callee.Tast.ty with + | Some (params, ret) -> if List.length args <> List.length params then fail loc "this function value takes %d argument%s, given %d" (List.length params) @@ -5618,9 +6009,9 @@ and call_value ctx ~want loc (callee : Tast.expr) args = (List.length args); let args = map2_lr (fun p a -> check ctx ~want:p a) params args in expect ctx loc ~want (mk loc ret (Tast.CallPtr (callee, args))) - | other -> + | None -> fail loc "this is a %s and not a function, so it cannot be called" - (Types.to_string other) + (Types.to_string callee.Tast.ty) (* A builtin's arity. The count is the builtin's and can only be the builtin's: a defn of the same name written in the program now takes the @@ -6563,7 +6954,7 @@ and named_call ?(qualified = false) ctx ~want loc name args = | Types.Option _ -> "it carries a tag saying whether the value is there, and a \ filled one says yes over a payload nobody wrote" - | Types.Fn _ -> + | Types.Fn _ | Types.CFn _ -> "it is a code address, and a call through a filled one jumps \ into whatever 0xDE bytes happen to address" | Types.Named n when Hashtbl.mem ctx.env.datas n -> @@ -8311,16 +8702,26 @@ and ordinary_call ctx ~want loc name args = before the global function table: a binding shadows a defn of the same name (one namespace, ordinary lexical scoping). A local of any *other* type falls through to the table, so a program that shadows a function - name with an i32 and then calls the function still means the function. *) + name with an i32 and then calls the function still means the function. + + A local of the *enclosing* function holding one is the same case: a + lifted body captures it by value and then calls the copy. [peek_outer] + rather than [capture] in the guard, because a guard must not take a copy + on its way to deciding what a form means. *) | _ when (match lookup ctx name with - | Some b -> (match b.bty with Types.Fn _ -> true | _ -> false) - | None -> false) -> + | Some b -> callable_ty b.bty + | None -> + match peek_outer ctx name with + | Some b -> callable_ty b.bty + | None -> false) -> (* The binding the guard already found, read directly. Going back through - [check] would repeat the lookup and walk the capture path for a type - that is not capturable. *) + [check] would repeat the lookup. *) (match lookup ctx name with | Some b -> call_value ctx ~want loc (mk loc b.bty (Tast.Local b.slot)) args - | None -> assert false) + | None -> + match capture ctx loc name with + | Some b -> call_value ctx ~want loc (mk loc b.bty (Tast.Local b.slot)) args + | None -> assert false) | _ when Hashtbl.mem ctx.env.gsigs name -> let vars, params, ret = Hashtbl.find ctx.env.gsigs name in generic_call ctx ~want loc name vars params ret args @@ -8975,7 +9376,8 @@ and trial ctx f = resource failure into a wrong answer. *) let[@warning "+9"] { env = _; ret = _; slots; slot_tys; slot_names; scope; defers; defer_slot; defer_ok; defer_block; outer = _; - outer_what; in_frames; loops; tail; in_defer; + outer_what; caught; envslot; parent = _; + in_frames; loops; tail; in_defer; owner = _ } = ctx in match f () with | r -> Ok r @@ -8985,6 +9387,7 @@ and trial ctx f = ctx.defers <- defers; ctx.defer_slot <- defer_slot; ctx.defer_ok <- defer_ok; ctx.defer_block <- defer_block; ctx.outer_what <- outer_what; ctx.in_frames <- in_frames; + ctx.caught <- caught; ctx.envslot <- envslot; ctx.loops <- loops; ctx.tail <- tail; ctx.in_defer <- in_defer; Error d @@ -9720,6 +10123,25 @@ let collect env (decls : Ast.decl list) = "%s of %s is dyn, which does not cross to C — take the value \ at a written type and pass that" what fn.Ast.name + (* Its own arm too, because "pass (Ptr T)" is nonsense for a + function and the real objection is worth stating. A [CFn] is + one word and is the right *shape* for a C callback — that is + what it is for — but a Flan function's emitted signature still + ends with the transfer channel, and a C caller knows nothing + about one. So the address is not a C function pointer yet, and + what would make it one is dropping the channel from a signature + that cannot transfer (FIX.org, 2026-09-21). A [Fn] is two words + and is not even the right shape. *) + | Types.CFn _ | Types.Fn _ -> + fail loc + "%s of %s is %s, and a Flan function's address is not a C \ + function pointer yet — not even a CFn's. Its signature ends \ + with the transfer channel, and a C caller knows nothing \ + about one; the C in CFn is about having no environment, \ + which is what a C function pointer would need, and not about \ + crossing today. Write the callback in C, or give the binding \ + a (Ptr ()) and let the shim pass C's own" + what fn.Ast.name (Types.to_string t) | _ -> fail loc "%s of %s is %s, which cannot cross to C directly — pass \ @@ -10126,7 +10548,7 @@ let rec check_fn env (fn : Ast.fn) : Tast.fn = (match ctx.defer_slot with | None -> ctx.defers | Some s -> guarded_defers s ctx.defers); - fparent = None; floc = fn.Ast.nloc } + fenv = None; fparent = None; floc = fn.Ast.nloc } (* The generic body, checked once with its variables abstract. Nothing is kept — the [Tast.fn] it produces is thrown away, and so is anything it lifted — @@ -10420,7 +10842,7 @@ let lift_ginit ctx loc n ty (v : Tast.expr) = (match ctx.defer_slot with | None -> ctx.defers | Some s -> guarded_defers s ctx.defers); - fparent = Some n; floc = loc } + fenv = None; fparent = Some n; floc = loc } :: ctx.env.lifted; { Tast.e = Tast.Call (fname, []); ty; loc } @@ -11085,6 +11507,199 @@ let dyn_descriptors (p : Tast.program) = "What C hands back points at storage this compiler never rooted") p.Tast.externs; value_sites p (fun ~slot:_ loc what t -> check loc what t) +(* ── Escape, which is the other half of capture ──────────────────────── + spec-memory.md's case 2 is the *non-escaping* fn, and this is what makes + the word mean something. A captured copy lives in a slot of the frame the + literal was written in, so a value holding that frame's address may be + called, passed down and copied about as much as anyone likes — and must + never outlive the frame. Case 3, the escaping closure with an environment + the collector allocates, is a separate lane; every refusal here names it, + because "this cannot be done" and "this cannot be done yet" are different + sentences and the second one is the true one. + + Run over the typed IR rather than over the surface, and over every + function the program ends up with rather than only over the ones anyone + wrote. Two reasons, and both are about not having to be careful: the IR + has one node per way a value can be stored, so the list below is closed; + and a lifted body is checked by exactly the same pass as the body it came + out of, so an [fn] inside an [fn] needs no special case. + + **What is suspect.** A value of function type that may carry an + environment, decided by a rule that needs no interprocedural anything: + + - a capturing literal, which is a [Closure] node and is the only place one + is made; + - a *parameter* of function type, in every function, because nothing at a + definition can see what its callers will pass; + - a local bound to either of those, transitively; + - a branch or a block whose value is one. + + Everything else of function type is clean: the address of a name, and the + result of any call — the second follows from the first refusal below, which + is what stops a function from returning a suspect in the first place. + + **Why the parameter rule is the whole answer to the hard case.** An [fn] + that captures, passed to a function that stores it, is caught *inside that + function*: its parameter is suspect there, and the store is refused where + it is written. So no call can leak what its caller passed, and no caller + has to be analysed. What it costs is real and worth naming: [(defn id [f + (Fn [] i32)] (Fn [] i32) f)] is refused although it is harmless, and so is + holding a parameter of function type in a Vec that never leaves the frame. + Both become writable when case 3 lands, and neither is worth an analysis + before then. + + **Where a suspect may stand**: an argument of a call, the callee of one, a + [let] binding, and the field of an environment another literal captures it + into — that last is the [env/] exemption below, and it is sound for the + same reason everything here is: the outer literal is itself suspect, so + the pair of environments lives and dies with one frame. *) +let rec escaping suspects (e : Tast.expr) = + (* The value of a body is its last form, which is the only part of one that + can be this expression's own value. *) + let tail body = + match List.rev body with x :: _ -> escaping suspects x | [] -> false + in + match e.Tast.ty with + | Types.Fn _ -> + (match e.Tast.e with + | Tast.Closure _ -> true + | Tast.Local s -> List.mem s !suspects + | Tast.If (_, a, b) -> escaping suspects a || escaping suspects b + | Tast.Do body | Tast.Let (_, body) -> tail body + | Tast.Match (_, arms) -> List.exists (fun (a : Tast.arm) -> tail a.Tast.abody) arms + (* The forms that establish something around a body and yield the body's + value. Easy to forget and not safe to: each of them is an expression, + so each of them is a way for a suspect to be the answer. A + restart-case yields its body's value *or* a clause's, so every clause + is a tail too. *) + | Tast.Handled (_, body) | Tast.WithAlloc (_, body) -> tail body + | Tast.RestartCase (cs, body) -> + escaping suspects body + || List.exists (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs + (* Read out of a struct, which for a function value means read out of an + *environment*: that is the one aggregate a capture may be written into + (see [is_env_struct]), and so the one this can be reading. A copy of a + captured function value carries whatever environment the original did, + so it is suspect exactly as the original was — without this, a lifted + body could hand back its copy of a captured value and the result would + arrive at the caller as an ordinary call result, which is to say as + clean. The same goes for a load through a pointer: nothing a + [(Ptr (Fn ...))] can point at is anywhere but a frame, since a global + and a struct field of function type are both refused. *) + | Tast.Field _ | Tast.CaseField _ | Tast.Deref _ -> true + (* The address of a name, and the result of a call. Neither can carry an + environment: the first never did, and the second cannot because a + function that would return one is refused below. *) + | _ -> false) + | _ -> false + +(* The environment struct a capture built, which is the one aggregate a + suspect may be written into. See the header. *) +let is_env_struct n = String.length n >= 4 && String.sub n 0 4 = "env/" + +let escape_check (fn : Tast.fn) = + let suspects = ref [] in + List.iteri + (fun i ty -> match ty with Types.Fn _ -> suspects := i :: !suspects | _ -> ()) + fn.Tast.params; + (* Two refusals, because the two cases know different amounts. A literal + written here captures, full stop, and the message can name the frame its + copies are on. A function value that arrived as a parameter *may* carry + an environment and nothing at a definition can tell — so it is refused + where it is written rather than at the calls that would have been fine, + and the message says that is what happened. *) + let owner = match fn.Tast.fparent with Some p -> p | None -> fn.Tast.name in + (* Which of the two this is, found by following the same tails [escaping] + followed. The refused expression is often a form that *yields* the + suspect — a handler-bind, a branch — and the message has to describe what + is actually escaping and not the shape it arrived in. *) + let rec written_here (e : Tast.expr) = + let tail body = + match List.rev body with x :: _ -> written_here x | [] -> true + in + match e.Tast.e with + | Tast.Closure _ -> true + | Tast.Local _ | Tast.Field _ | Tast.CaseField _ | Tast.Deref _ -> false + | Tast.If (_, a, b) -> written_here a && written_here b + | Tast.Do body | Tast.Let (_, body) | Tast.Handled (_, body) + | Tast.WithAlloc (_, body) -> tail body + | Tast.Match (_, arms) -> + List.for_all (fun (a : Tast.arm) -> tail a.Tast.abody) arms + | Tast.RestartCase (cs, body) -> + written_here body + && List.for_all (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs + | _ -> true + in + let refuse (e : Tast.expr) where = + if not (written_here e) then + Loc.failk "check/fn-escapes" e.Tast.loc + "this function value may carry an environment, and %s would outlive \ + the frame that environment is on. A value reaching %s as a parameter \ + was made by a caller this definition cannot see, so it is refused \ + here rather than at the calls that would be safe. Call it, pass it \ + down, or hold it in a let — an fn whose environment the collector \ + owns is spec-memory.md's case 3 and is not built yet" + where owner + else + Loc.failk "check/fn-escapes" e.Tast.loc + (* No "hold it in a let" here, unlike the message above: an fn literal + takes its types from the position it is written in, so there is no + let binding to offer — see fn-no-type.flan. Every suggestion this + compiler prints has to compile. *) + "this fn captures, and %s would outlive the frame its copies are on. \ + The copies are slots of %s, taken where the value was made, so a \ + reader reached after that frame has gone would read whatever \ + replaced them. Call it, or pass it down — an fn whose environment \ + the collector owns is spec-memory.md's case 3 and is not built yet" + where owner + in + let deny where es = List.iter (fun e -> if escaping suspects e then refuse e where) es in + let go (e : Tast.expr) = + (match e.Tast.e with + | Tast.Let (bs, _) -> + (* A binding is where a suspect spreads, and the only place it does. *) + List.iter + (fun (slot, v) -> if escaping suspects v then suspects := slot :: !suspects) + bs + | Tast.Set (_, v) -> deny "a store" [ v ] + | Tast.Return (Some v) -> deny "a return" [ v ] + | Tast.Some_ v -> deny "an Option" [ v ] + | Tast.Arr es -> deny "a fixed array" es + | Tast.MakeCase (_, _, es) -> deny "a data type's field" es + | Tast.Make (n, es) -> if not (is_env_struct n) then deny "a struct field" es + | Tast.Addr (Tast.Plocal s) -> + if List.mem s !suspects then + refuse e "a pointer to it" + (* Everything the runtime takes: a push into a Vec, a put into a Map, a + box into a dyn. All of them put the value somewhere this frame does + not own — and all of them take it *by address*, because the container + runtime is type-erased, so the address is what has to be caught and + not the value beside it. *) + | Tast.Prim (Tast.Rt _, es) -> + deny "a container" es; + List.iter + (fun (a : Tast.expr) -> + match a.Tast.e with + | Tast.Prim (Tast.AddrOf, [ v ]) -> deny "a container" [ v ] + | _ -> ()) + es + | Tast.Prim (Tast.AddrOf, [ v ]) -> deny "a pointer to it" [ v ] + | Tast.InvokeRestart (_, _, es, _, _, _) -> deny "a restart's argument" es + | _ -> ()) + in + (* Outermost first, which is the order [Tast.walk] gives and the order a + [let] has to be seen in: a binding must be recorded before anything that + reads the slot. *) + List.iter (fun e -> Tast.walk go e) fn.Tast.body; + (* And the defers on the transfer path, which are the same forms again but + are not reachable from [body] — they hang off the function, and a store + written in one is a store. *) + List.iter (fun e -> Tast.walk go e) fn.Tast.fdefers; + (* And the tail, which is a return with nothing written. *) + (match List.rev fn.Tast.body with + | last :: _ when (match fn.Tast.ret with Types.Fn _ -> true | _ -> false) -> + deny "a return" [ last ] + | _ -> ()) let build_program ~keep_going (decls : Ast.decl list) : Tast.program * env = let env = new_env () in @@ -11169,6 +11784,10 @@ let build_program ~keep_going (decls : Ast.decl list) : Tast.program * env = they are reached *by name* from arbitrary call sites, so they carry no [fparent] and a dev build gives each its own cell. *) let fns = fns @ List.rev env.instances in + (* Where a captured copy may go, asked of every function the program ended + up with. Here rather than inside [check] because it is a question about a + finished body — see the header on [escaping]. *) + List.iter escape_check fns; (* And the order the computed initialisers run in, which needs the whole function list: what a global reads is transitive through what it calls. *) let globals = init_order globals fns in diff --git a/lib/cimport.ml b/lib/cimport.ml index 6a7f1da6..3c9b8f95 100644 --- a/lib/cimport.ml +++ b/lib/cimport.ml @@ -413,8 +413,8 @@ let rec ty_source (t : Ast.texpr) = | Ast.Tarray (Ast.Lname n, e) -> Printf.sprintf "[%s %s]" n (ty_source e) | Ast.Tmap (k, v) -> Printf.sprintf "(Map %s %s)" (ty_source k) (ty_source v) - | Ast.Tfn (ps, r) -> - Printf.sprintf "(Fn [%s] %s)" + | Ast.Tfn (env, ps, r) -> + Printf.sprintf "(%s [%s] %s)" (if env then "Fn" else "CFn") (String.concat " " (List.map ty_source ps)) (ty_source r) let tname n = ty (Ast.Tname n) diff --git a/lib/dev.ml b/lib/dev.ml index c2bfdc5d..05b2e603 100644 --- a/lib/dev.ml +++ b/lib/dev.ml @@ -2484,7 +2484,7 @@ let render_addr (s : Session.t) ~addr ~(ty : Types.t) let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: what is being shown is storage this thunk reaches by address. *) diff --git a/lib/emit.ml b/lib/emit.ml index 215cfb03..8181c726 100644 --- a/lib/emit.ml +++ b/lib/emit.ml @@ -85,6 +85,38 @@ let cellname n = "@" ^ quoted (Mangle.cell n) written only by the guards this file emits. *) let xfer_param = "%xfer" +(* The environment parameter, the other name that is not a Flan name: the + address of the captured copies a function value was made with. + + **It is declared by exactly the bodies that can be reached through a + [(Fn ...)] value**, and it is the *last* parameter, after the transfer + channel. That set is: a lifted [fn] literal written into an [Fn] position, + capturing or not; every handler clause, because [flan_signal] passes one + to whichever clause matched and cannot know which of them captured; and + the widening thunks ([Tast.Thicken]), which exist to read it. + + Nothing else declares it. An ordinary [defn] therefore emits exactly the + signature it always did — its parameters and then the channel, and not a + byte more — and a call to it by name is unchanged. That is the whole of + what keeps capture free for everyone who does not use it, and it is the + author's ruling: the static side does not pay for the dynamic side. + + So **every indirect call is exactly typed**. The two conventions meet in + one place, the thunk, and nowhere does a caller pass an argument the callee + did not declare. An earlier design did rely on that — the environment last, + ignored by a body that never asked for it, which SysV allows and Swift's + thin-vs-thick convention is built on — and wasm32 killed it: [call_indirect] + compares the signature at the call site, so a spare argument is a trap and + not a register nobody reads. Being exactly typed is checkable by a verifier + rather than argued from a calling convention, which is the better property + to have had all along. *) +let env_param = "%env" + +(* What a call through a [(Fn ...)] value passes when it has no environment — + a value made out of a name, or one widened from a [CFn]. Spelled once so + the sites cannot drift. *) +let no_env = "ptr null" + (* The condition's own name, for the message an unhandled [error] prints. The checker has already refused anything that is not a struct. *) let struct_name_of (t : Types.t) = @@ -130,10 +162,13 @@ module Rt = struct let ll_of = function Ptr -> "ptr" | I32 -> "i32" | I64 -> "i64" let size_of = function Ptr | I64 -> 8 | I32 -> 4 - (* A handler frame: the one it displaced, the condition type it matches, and - the lifted function that runs. *) + (* A handler frame: the one it displaced, the condition type it matches, the + lifted function that runs, and the environment that function is handed — + the establishing function's captured copies, or null when the clause + captured nothing. *) let handler = - { sname = "handler"; fields = [ "prev", Ptr; "type", I32; "fn", Ptr ] } + { sname = "handler"; + fields = [ "prev", Ptr; "type", I32; "fn", Ptr; "env", Ptr ] } (* A restart frame. The first four fields are what the runtime's own [flan_restart] declares and their offsets do not move; the rest are §3's @@ -246,11 +281,13 @@ let rec ll (t : Types.t) = (* An [Allocator] is a pointer to the runtime's [flan_allocator] and never a copy of one: see Types. Opaque here in the same sense [ptr] is. *) | Types.Alloc -> "ptr" - (* A function value is a code address and nothing else. There is no - environment beside it — capture does not exist (check.ml refuses it by - name) — so it is one pointer, the same width as any other, and a backend - needs to know no more about it than that. *) - | Types.Fn _ -> "ptr" + (* A code address and the environment it is called with: two words, always, + whether or not this particular value captured anything. See [%fnv]. *) + | Types.Fn _ -> "%fnv" + (* The bare address, and nothing beside it: one pointer, the width of any + other. A [CFn] cannot capture, so there is nothing an environment + would hold. *) + | Types.CFn _ -> "ptr" (* ptr + len + cap + allocator, and two more words the runtime owns: see flan_rt.c's (Vec T) header for why they are in every build. Nothing in this file reads a field of one — every operation is a runtime call taking @@ -454,7 +491,8 @@ let rec lay m (t : Types.t) : int * int = | Types.Enum _ -> 4, 4 | Types.Ptr _ -> 8, 8 | Types.Alloc -> 8, 8 - | Types.Fn _ -> 8, 8 + | Types.Fn _ -> 16, 8 + | Types.CFn _ -> 8, 8 | Types.Vec _ | Types.Map _ -> 40, 8 (* [n x T] adds no padding of its own: T's size already carries its tail. *) | Types.Array (n, e) -> let s, a = lay m e in Int64.to_int n * s, a @@ -813,13 +851,26 @@ let rec dty m d (t : Types.t) : int = ("len", Types.Int Types.I64); ("log2cap", Types.Int Types.I64); ("allocator", Types.Alloc); ("epoch", Types.Int Types.I64) ] |> fun n -> ignore k; ignore v; n - (* A pointer to code, and lldb is told exactly that and no more. DWARF - has DW_TAG_subroutine_type for the signature behind it, and spelling - one out here would buy a reader nothing they cannot get from the - function it points at — [p f] answers with an address either way, and - the address is what resolves to a symbol. The name carries the - signature, which is where it is actually legible. *) + (* Two words, and shown as two, the same rule the Vec and the Map above + follow: a debugger told a function value were one pointer would put + every offset after it out by eight. [code] is the address that + resolves to a symbol, which is what [p f] was ever worth; [env] is + the captured copies, and there is nothing here that could say what is + in them — the environment is a struct the checker synthesised for one + literal, and DWARF for it would describe a type the program cannot + name. A reader who wants the copies asks the break loop for the + locals, where they are under the names the source gave them. *) | Types.Fn _ -> + composite (Types.to_string t) + [ ("code", Types.Ptr Types.Unit); ("env", Types.Ptr Types.Unit) ] + (* And the bare one is what it always was: a pointer to code, and lldb + is told exactly that and no more. DWARF has DW_TAG_subroutine_type + for the signature behind it, and spelling one out would buy a reader + nothing they cannot get from the function it points at — [p f] + answers with an address either way, and the address is what resolves + to a symbol. The name carries the signature, which is where it is + actually legible. *) + | Types.CFn _ -> dnode d (Printf.sprintf "!DIDerivedType(tag: DW_TAG_pointer_type, name: \"%s\", \ @@ -1763,19 +1814,41 @@ and value_at f (e : Tast.expr) : string = | Tast.Local _ | Tast.Global _ | Tast.Field _ | Tast.Deref _ -> (* Everything that denotes a location is a load from its address. *) load f (addr f e) e.Tast.ty - (* The symbol itself, not a load from it: a function's address is a link-time - constant. The same spelling the handler frames use for a lifted clause. *) - | Tast.FnAddr (Tast.Flanfn n) -> fname n - | Tast.FnAddr (Tast.Rtfn n) -> "@" ^ n - (* A function value someone wrote, which is the one [FnAddr] that is not the - symbol. In a dev build it is the cell's contents, so that a value taken - after a redefinition is the new body — the same load a direct call to the - same name would do, at the point the *address* is taken rather than at the - call. What that does not give is a value taken before a redefinition and - called after it: that one is still the old body, because there is nothing - left to re-resolve once the address is in a slot. Named in docs/BUILT.md rather - than papered over with a trampoline. *) - | Tast.FnAddr (Tast.Fnval n) -> body_of f n + (* A [(Fn ...)] value, which is two words: a code address and the + environment it is called with. A value made out of a name captures + nothing, so the second word is null and [zeroinitializer] has already put + it there. See [%fnv]. + + Only a [Fn]-typed one. The same three [fnref] constructors are also asked + for as bare addresses — carrying [CFn], and carrying [Alloc] for the + map's hash and equality pair and a handler frame's clause, which are + fields of structs the runtime declares — and those stay one word. The + node's type is what says which is being asked for. *) + | (Tast.FnAddr _ | Tast.Closure _ | Tast.Thicken _) + when (match e.Tast.ty with Types.Fn _ -> true | _ -> false) -> + let code, env = + match e.Tast.e with + | Tast.FnAddr r -> fnaddr f r, "null" + | Tast.Closure (r, env) -> fnaddr f r, value f env + (* The widening: the thunk's code, with the bare address stored where + an environment would be. The thunk reads it back out and calls it, + which is what keeps every indirect call exactly typed. *) + | Tast.Thicken (n, p) -> fname n, value f p + | _ -> assert false + in + let a = fresh f in + ins f "%s = insertvalue %%fnv zeroinitializer, ptr %s, 0" a code; + if String.equal env "null" then a + else begin + let b = fresh f in + ins f "%s = insertvalue %%fnv %s, ptr %s, 1" b a env; + b + end + | Tast.FnAddr r -> fnaddr f r + | Tast.Closure _ | Tast.Thicken _ -> + (* Unreachable: both are [Fn] values and the arm above has already taken + every [Fn]-typed node. Here because nothing else could be meant. *) + failwith "a closure is a function value" | Tast.Addr p -> fst (place f p) | Tast.Prim (p, args) -> prim f e p args | Tast.Call (name, args) -> @@ -2192,14 +2265,57 @@ and call f ret flan args = the middle of an argument list). *) and call_ptr f ret callee args = let c = value f callee in + (* A [(Fn ...)] is two words and both are taken before the arguments are + evaluated: an argument may itself make a function value, and the two + halves of *this* one have to come out of the same value. A + [(CFn ...)] is the address alone, and the call that follows is the + call a name would have produced. *) + let code, env = + match callee.Tast.ty with + | Types.Fn _ -> + let code = fresh f in + ins f "%s = extractvalue %%fnv %s, 0" code c; + let env = fresh f in + ins f "%s = extractvalue %%fnv %s, 1" env c; + code, Some ("ptr " ^ env) + | _ -> c, None + in let vs = map_lr (fun (a : Tast.expr) -> let v = value f a in Printf.sprintf "%s %s" (ll a.Tast.ty) v) args in - call_through f ret c vs + call_through f ?env ret code vs -and call_through f ret callee vs = +(* The code address behind one of the three [fnref]s, which is the same string + whether it is wanted as a bare [Alloc] pointer or as the first word of a + function value. + + [Flanfn] and [Rtfn] are the symbol itself, not a load from it: a function's + address is a link-time constant. [Fnval] is the one that is not — in a dev + build it is the cell's contents, so that a value taken after a redefinition + is the new body, the same load a direct call to the same name would do, at + the point the *address* is taken rather than at the call. What that does + not give is a value taken before a redefinition and called after it: that + one is still the old body, because there is nothing left to re-resolve once + the address is in a slot. Named in docs/BUILT.md rather than papered over + with a trampoline. *) +and fnaddr f (r : Tast.fnref) = + match r with + | Tast.Flanfn n -> fname n + | Tast.Rtfn n -> "@" ^ n + | Tast.Fnval n -> body_of f n + +(* [env] is present on exactly one kind of call: one through a [(Fn ...)] + value, which cannot know whether the body it reaches declared one. Every + other call — by name, through a [(CFn ...)] — passes what it always + passed. See [env_param] for why appending it is safe when the callee did + not ask for it. *) +and call_through f ?env ret callee vs = let t = fresh f in - ins f "%s = call %s %s(%s)" t (ll ret) callee - (String.concat ", " (vs @ [ "ptr " ^ xfer_param ])); + let tail = + match env with + | None -> [ "ptr " ^ xfer_param ] + | Some e -> [ "ptr " ^ xfer_param; e ] + in + ins f "%s = call %s %s(%s)" t (ll ret) callee (String.concat ", " (vs @ tail)); guard f; (* An aggregate with a dyn in it is spilled into a rooted slot the instant it arrives, the same move a dyn word gets in [prim] and for a sharper reason: @@ -2293,6 +2409,16 @@ and emit_handled f frames body = stack finds what it pushed still valid, which is what "old code is never unloaded" means. See NEXT.md, conditions step 1. *) ins f "store ptr %s, ptr %s" (fname h.Tast.hfn) fp; + (* And the environment the clause is called with, which is a pointer + into this very frame. Written unconditionally — null when the + clause captured nothing — because a frame the runtime reads a + field of must have every field written, not only the ones this + clause happens to use. *) + let ep = fresh f in + ins f "%s = getelementptr inbounds %%handler, ptr %s, i32 0, i32 3" + ep slot; + ins f "store ptr %s, ptr %s" + (match h.Tast.henv with Some e -> value f e | None -> "null") ep; ins f "call void @flan_handler_push(ptr %s)" slot; slot) frames @@ -3136,6 +3262,14 @@ let signature ~named (fn : Tast.fn) = analysis is an optimisation, and in a dev build a cell can hold anything, so the honest answer to "what can this call?" is "anything". *) let params = params @ [ (if named then "ptr " ^ xfer_param else "ptr") ] in + (* And the environment, last, and only on a body that can be reached + through an [Fn] value: see [env_param]. Everything else emits the + signature it always did. *) + let params = + match fn.Tast.fenv with + | None -> params + | Some _ -> params @ [ (if named then "ptr " ^ env_param else "ptr") ] + in Printf.sprintf "%s %s(%s)" (ll fn.Tast.ret) (fname fn.Tast.name) (String.concat ", " params) @@ -3205,6 +3339,14 @@ let emit_fn m ?(hidden = false) ?(pnames = []) (fn : Tast.fn) = Buffer.add_string f.allocas (Printf.sprintf " store %s %%p%d, ptr %s\n" (ll ty) i f.slots.(i))) fn.Tast.params; + (* And the environment, on the one kind of function that has one. Every + other function is handed it too and never reads it; there is no slot for + it there and nothing to store. *) + (match fn.Tast.fenv with + | Some slot -> + Buffer.add_string f.allocas + (Printf.sprintf " store ptr %s, ptr %s\n" env_param f.slots.(slot)) + | None -> ()); (* The dyn roots, and this is not gated on [m.dev]: the shadow stack below is a debugging convenience and a release build does without it, while a collector that cannot find its roots is a collector that frees live @@ -3709,7 +3851,7 @@ let emit_startup m ?(hidden = false) (globals : Tast.global list) = List.iter (emit_global m ~hidden) flags; emit_fn m ~hidden { Tast.name = ".init-globals"; params = []; slots = [||]; snames = [||]; - ret = Types.Unit; body; fdefers = []; fparent = None; + ret = Types.Unit; body; fdefers = []; fenv = None; fparent = None; floc = (List.hd computed).Tast.ginit.Tast.loc }; true @@ -3758,6 +3900,13 @@ let header = {|; Generated by flan. The layout is C's: no object headers anywher ; so a Flan struct is exactly its C struct and nothing marshals. %slice = type { ptr, i64 } +; A function value: the code address, and the environment the captured copies +; live in. Two words rather than one because the environment has to travel +; *with* the value — a callee that takes a (Fn [T] R) and calls it knows +; nothing about where the value came from, so there is nowhere else to put it. +; A value that captures nothing carries a null there and every call passes it +; on regardless; see [env_param]. +%fnv = type { ptr, ptr } ; (Vec T), spec-memory.md. The element type is nowhere in it: the runtime is ; type-erased and every operation is handed size and align at its call site. %vec = type { ptr, i64, i64, ptr, i64 } @@ -3765,8 +3914,9 @@ let header = {|; Generated by flan. The layout is C's: no object headers anywher ; nor value type appears in it, for the same reason: one type-erased runtime, ; handed the two sizes and a hash/equality pair at each call site. %map = type { ptr, i64, i64, ptr, i64 } -; A handler frame: the one it displaced, the condition type it matches, and -; the lifted function that runs. Allocated on the establishing frame's stack. +; A handler frame: the one it displaced, the condition type it matches, the +; lifted function that runs, and the environment that function is handed. +; Allocated on the establishing frame's stack. |} ^ Rt.ll_type Rt.handler ^ {| ; A restart frame: the one it displaced and the name it offers. There is no ; target field, because the frame's own address *is* the target — which makes @@ -4521,11 +4671,18 @@ let redefinition ?(checks = true) ?(dev = false) ?(debug = false) (* A clause lifted out of one of these comes with it: its body may have changed too, and it is reached by address from inside the module rather than through a cell. Every other lifted clause is invisible here — it - needs no declaration, since nothing in this module names it. *) + needs no declaration, since nothing in this module names it. + + And the widening thunks, every one of them, whichever body they belong + to: a module that hands a name to an [Fn]-typed parameter names one, and + the host has no cell for it to be reached through. They are hidden and + tiny, so a copy per module is the whole cost — and the alternative is an + undefined symbol at dlopen, which is the shape of bug [Fnval] was. *) let lifted = List.filter (fun (f : Tast.fn) -> match f.Tast.fparent with + | Some "" -> true | Some p -> List.mem p fns | None -> false) p.Tast.fns diff --git a/lib/js.ml b/lib/js.ml index a5fdd744..489b8bdb 100644 --- a/lib/js.ml +++ b/lib/js.ml @@ -220,7 +220,8 @@ let rec refuse_ty loc (t : Types.t) = | Types.Never | Types.Named _ | Types.Enum _ -> () | Types.Slice t | Types.Array (_, t) | Types.Option t -> refuse_ty loc t | Types.Vec t -> refuse_ty loc t - | Types.Fn (ps, r) -> List.iter (refuse_ty loc) ps; refuse_ty loc r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.iter (refuse_ty loc) ps; refuse_ty loc r | Types.Ptr _ -> at loc "(Ptr T) is not in the JS dialect — JavaScript has no addresses, so a \ @@ -698,6 +699,17 @@ let rec value f (e : Tast.expr) : string = "the runtime entry point %s has no JS counterpart — it is C in \ flan_rt.c, and this dialect has no C" n + (* A capturing fn literal. A JS function closes over its enclosing scope for + free, so this dialect would not need the environment at all — but the + environment is a struct the checker synthesised and the captured copies + are read out of it by index, which is machinery this backend has nothing + to lower. Refused by name rather than emitted as a plain function that + would read the *current* value of a local instead of the copy. *) + | Tast.Closure _ | Tast.Thicken _ -> + at e.Tast.loc + "an fn that captures has no JS lowering yet — the environment is a \ + struct laid out for the two native backends, and this dialect has no \ + layout" | Tast.Prim (p, args) -> prim f e p args | Tast.Call (n, args) -> Printf.sprintf "%s(%s)" (fname n) (String.concat ", " (call_args f args)) diff --git a/lib/load.ml b/lib/load.ml index c1d3af8c..4636fd21 100644 --- a/lib/load.ml +++ b/lib/load.ml @@ -206,8 +206,9 @@ let rec rename_texpr owned alias (t : Ast.texpr) : Ast.texpr = Ast.Tmap (rename_texpr owned alias k, rename_texpr owned alias v) | Ast.Tapp (n, args) -> Ast.Tapp (n, List.map (rename_texpr owned alias) args) - | Ast.Tfn (ps, r) -> - Ast.Tfn (List.map (rename_texpr owned alias) ps, rename_texpr owned alias r) + | Ast.Tfn (env, ps, r) -> + Ast.Tfn (env, List.map (rename_texpr owned alias) ps, + rename_texpr owned alias r) in { t with Ast.t = k } @@ -746,7 +747,7 @@ let rec texpr_uses acc (t : Ast.texpr) = texpr_uses acc e | Ast.Tmap (k, v) -> texpr_uses acc k; texpr_uses acc v | Ast.Tapp (_, args) -> List.iter (texpr_uses acc) args - | Ast.Tfn (ps, r) -> List.iter (texpr_uses acc) ps; texpr_uses acc r + | Ast.Tfn (_, ps, r) -> List.iter (texpr_uses acc) ps; texpr_uses acc r let rec expr_uses acc (e : Ast.expr) = let go = expr_uses acc in diff --git a/lib/parse.ml b/lib/parse.ml index 52fb2470..4181c0a0 100644 --- a/lib/parse.ml +++ b/lib/parse.ml @@ -96,11 +96,16 @@ let rec texpr (f : Form.t) : Ast.texpr = confused with. *) | Map _ -> fail f "a map type is written (Map K V), not in braces" - | List ({ v = Sym "Fn"; _ } :: rest) -> + (* The two function types. [Fn] is the one almost every signature wants — a + value that may carry an environment — and [CFn] is the bare address, + for a C callback or a table of them. Parsed together because they differ + in one word and the refusal should name both. *) + | List ({ v = Sym (("Fn" | "CFn") as which); _ } :: rest) -> + let env = String.equal which "Fn" in (match rest with | [ { v = Vec params; _ }; ret ] -> - mk (Ast.Tfn (List.map texpr params, texpr ret)) - | _ -> fail f "a function type is (Fn [T ...] R)") + mk (Ast.Tfn (env, List.map texpr params, texpr ret)) + | _ -> fail f "a function type is (%s [T ...] R)" which) | List ({ v = Sym name; _ } :: args) when args <> [] -> mk (Ast.Tapp (name, List.map texpr args)) | _ -> fail f "expected a type, found %s" (Form.to_string f) diff --git a/lib/reach.ml b/lib/reach.ml index f7803d46..a3000a3b 100644 --- a/lib/reach.ml +++ b/lib/reach.ml @@ -50,7 +50,12 @@ let expr_refs f (e : Tast.expr) = name used as a value is never a [Call], so without the second one the one function a program passes to [map] is the one function the link drops. [Rtfn] is C in flan_rt.c and is linked whatever happens. *) - | Tast.FnAddr (Tast.Flanfn n) | Tast.FnAddr (Tast.Fnval n) -> f n + | Tast.FnAddr (Tast.Flanfn n) | Tast.FnAddr (Tast.Fnval n) + | Tast.Closure (Tast.Flanfn n, _) | Tast.Closure (Tast.Fnval n, _) + (* And the widening thunk, which is reached by address from the value + it builds and from nowhere else. Without this edge the one function + a program widens is the one function the link drops. *) + | Tast.Thicken (n, _) -> f n | Tast.Set (Tast.Pglobal n, _) | Tast.Addr (Tast.Pglobal n) -> f n | Tast.Handled (frames, _) -> List.iter (fun (h : Tast.hframe) -> f h.Tast.hfn) frames diff --git a/lib/session.ml b/lib/session.ml index 77dcf351..73ecd749 100644 --- a/lib/session.ml +++ b/lib/session.ml @@ -373,6 +373,16 @@ let compatible ?(origin = fun _ -> None) ?(relaxed = []) ~loc new_.Tast.globals; List.iter (fun (s : Tast.structure) -> + (* An environment the checker synthesised for a capturing fn is not + subject to this rule, and that is not a loophole. The layout rule is + about values the running program is *holding*: every other struct can + be in a global, in a container, in a frame that is on the stack right + now. An environment can be in exactly one place — a slot of the frame + the literal was written in — and it is written there by the same + module that reads it, on every entry. So editing which locals an fn + names is an ordinary body change, and demanding a restart for it + would take the dev loop away from the feature it was built for. *) + if Check.is_env_struct s.Tast.sname then () else match List.find_opt (fun (r : Tast.structure) -> String.equal r.Tast.sname s.Tast.sname) @@ -992,7 +1002,7 @@ let eval ?(origin = "") ?pause t src : change = Some { Tast.name = Printf.sprintf "install/%d" t.thunks; params = []; ret = Types.Unit; body; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = [||]; snames = [||] } in let ir = @@ -1319,7 +1329,7 @@ let render_locals ?(origin = "") t ~frame ~(fn : Tast.fn) ~bound let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: the locals being shown are the *other* frame's, and this thunk reaches them by address. *) @@ -1408,7 +1418,7 @@ let render_condition t ~(st : Tast.structure) : change * (string * string) list let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); snames = Array.make (List.length !extra) None } in @@ -1662,7 +1672,7 @@ let render_slot ?(origin = "") t ~frame ~(fn : Tast.fn) ~slot ~path { Tast.name = tname; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); snames = Array.make (List.length !extra) None } in @@ -1932,7 +1942,7 @@ let write_slot ?(origin = "") t ~frame ~(fn : Tast.fn) ~slot ~path stores @ (nullary "flan/dev-begin" :: parts) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.append base (Array.of_list (List.rev !extra)); (* The stored expressions' own [let]s keep their names; the slots [render] added behind them are the walk's own @@ -2030,7 +2040,7 @@ let render_globals ?(origin = "") t ~(globals : Tast.global list) let thunk : Tast.fn = { Tast.name; params = []; ret = Types.Unit; body = (nullary "flan/dev-begin" :: body) @ [ nullary "flan/dev-end" ]; - fdefers = []; fparent = None; floc = loc; + fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.of_list (List.rev !extra); (* Every slot in here is the walk's own scratch: what is being shown is the program's storage, which this thunk reaches by name. *) @@ -2118,7 +2128,7 @@ let eval_expr ?(origin = "") ?(pause = false) t src : change = t.thunks <- t.thunks + 1; let name = Printf.sprintf "eval/%d" t.thunks in let thunk : Tast.fn = - { Tast.name; params = []; ret = Types.Unit; body; fdefers = []; fparent = None; floc = loc; + { Tast.name; params = []; ret = Types.Unit; body; fdefers = []; fenv = None; fparent = None; floc = loc; slots = Array.append base (Array.of_list (List.rev !extra)); (* The expression's own [let]s keep their names; the slots [render] added behind them are the walk's own scratch and have none to keep. *) diff --git a/lib/tast.ml b/lib/tast.ml index c2ff3feb..deec3a91 100644 --- a/lib/tast.ml +++ b/lib/tast.ml @@ -112,6 +112,42 @@ and expr_kind = dev build is not the symbol but whatever the indirection cell holds, and carries the Flan type [Fn]. *) | FnAddr of fnref + (* A function value with an environment: the lifted body, and the address of + the copies the enclosing frame is holding for it. The environment is a + [Make] of a struct the checker synthesised, stored into a slot of the + frame the literal was written in, so this node's second half is an + [Addr (Plocal _)] and the copies were taken where the value was made. + + Its own node rather than a field on [FnAddr] because the two answer + different questions: [FnAddr] is an address, and is asked for by three + unrelated readers that want a bare symbol ([Alloc]-typed, see [fnref]), + while this is a *value* of type [Fn] and can never be anything else. + + What stops it dangling is the checker, not this node: a value carrying an + environment may not leave the frame that owns it, so every position that + would outlive the frame is refused. spec-memory.md's case 2, and the + escaping half — an environment the collector allocates — is the case the + refusals name. *) + | Closure of fnref * expr + (* A (CFn ...) value where a (Fn ...) is wanted. The one coercion between + the two function types, and it goes this way only: there is nowhere for + an environment to go in the other direction. + + The pair it builds is {thunk, the address}: the *thunk's* code, one per + signature, with the original bare address stored where an environment + would be. The thunk reads it back out and calls it. So a value reached + through this is reached by a body that really does take an environment, + which is what keeps every indirect call exactly typed — including on + wasm32, where [call_indirect] checks the signature and an argument the + callee did not declare is a trap rather than a register nobody reads. + + The string is the thunk's name, minted and memoised by the checker: the + backends emit the pair and derive nothing. What it costs is one hop per + call, paid by a *name* handed to an [Fn]-typed parameter and by nothing + else — a literal, capturing or not, is compiled to take an environment + and needs no thunk. A signature that wants the address alone writes + [CFn] and pays nothing at all, which is what the type is for. *) + | Thicken of string * expr (* A call through a function value: the callee is an expression of type [Fn], not a name. Its own node rather than a [Call] with an expression in the name slot, because everything that walks this IR treats [Call]'s @@ -246,9 +282,16 @@ and place = | Pindex of expr * expr list | Pderef of expr -(* A pushed handler: which condition type it matches, and the lifted function - that runs when one is signalled. *) -and hframe = { htype : int; hfn : string } +(* A pushed handler: which condition type it matches, the lifted function that + runs when one is signalled, and the environment that function is handed. + + [henv] is the address of the establishing frame's copies of whatever the + clause captured, or [None] when it captured nothing. It is sound for the + same reason the frame itself is: a handler frame is popped by the body that + pushed it, so it can never be reached from outside the extent of the + function whose stack both it and the environment live on. There is no + escaping case here to defer. *) +and hframe = { htype : int; hfn : string; henv : expr option } (* A restart clause. [rname_id] is what [invoke-restart] matches by name; the body is a branch in the function that wrote it, because unlike a handler a @@ -307,6 +350,19 @@ type fn = { cell and no registry slot, and a redefinition of the parent carries its own copy. *) fparent : string option; + (* The slot the environment parameter is stored into, on a function that + was lifted out of something and captures one of its locals. Every + emitted signature takes the environment (see Emit's [env_param]) and + almost every function ignores it; this is the one that does not, and it + says where the pointer goes rather than fixing an index by convention, + because the slot is minted by [fresh_slot] like any other and a rule of + the form "the slot after the parameters" would be a second thing to keep + in step with the allocation order. + + [None] on everything anyone wrote. A capturing body reads its copies out + of this pointer once, at entry, into named slots of its own — so the + copy the value was made with is the copy the body sees. *) + fenv : int option; floc : Loc.t; } @@ -403,9 +459,10 @@ let rec walk (f : expr -> unit) (e : expr) = | Set (p, v) -> walk_place f p; go v | Addr p -> walk_place f p | Field (t, _) | Deref t | CaseField (t, _, _) | Some_ t | UnwrapSome t - | Signal (_, _, t) -> go t + | Signal (_, _, t) | Closure (_, t) | Thicken (_, t) -> go t | Match (sc, arms) -> go sc; List.iter (fun a -> gos a.abody) arms - | Handled (_, body) -> gos body + | Handled (hs, body) -> + List.iter (fun h -> Option.iter go h.henv) hs; gos body | RestartCase (cs, body) -> List.iter (fun c -> gos c.rbody) cs; go body | WithAlloc (a, body) -> go a; gos body diff --git a/lib/types.ml b/lib/types.ml index 20f026ba..2cc236e3 100644 --- a/lib/types.ml +++ b/lib/types.ml @@ -48,7 +48,50 @@ type t = at the call site, which is exactly where the two numbers are produced. *) | Vec of t | Option of t (* (Option T) *) + (* The two function types, and the difference between them is what a value + of each one *is* rather than what it may do. + + [(Fn [T ...] R)] is a code address and the environment it is called + with: two words. It is the common case and keeps the short name, because + it is what almost every higher-order signature wants — a caller may pass + it a name, a non-capturing literal, or one that captured half the frame, + and the callee neither knows nor cares. + + [(CFn [T ...] R)] is the bare address: one word, no environment, and + therefore nothing that can capture. + + **The [C] is information, not decoration.** A value with no environment + is the only kind that could ever cross to C, and under the + [--no-conditions] direction FIX.org records — where a signature that + cannot transfer drops the channel too — one becomes literally a C + function pointer. The name points at what the type *is* and at where it + is going. + + What it does **not** point at is a capability that exists now: a + [declare] cannot take a function type at all today, because a Flan + signature ends with the transfer channel and a C caller knows nothing + about one. Anyone reaching for [CFn] straight after writing a + [declare-c] is reaching too early, and [crossable] says so where they + will meet it. + + The whole of the reason there are two: a uniform environment would tax + every function in every program for a feature most of them never use, + and the static side is not to pay for the dynamic side's existence. With + two types an ordinary [defn] keeps exactly the signature it always had. + + **Nobody ever needs [CFn].** [Fn] accepts everything a [CFn] does, so + the narrow one is reached for on purpose, for one of four reasons: + handing a function to C (later, as above); a table of bare addresses; + forbidding capture at a boundary; and the one that is likeliest in + practice — a *named* function passed to an [Fn] parameter goes through + the widening thunk and pays an indirect hop per call, where a [CFn] + parameter is a direct call. [(map-in-place s double)] is the example. + + One-way: a [CFn] value satisfies an [Fn] (paired with a null + environment), and an [Fn] does not satisfy a [CFn] — there is nowhere + for the environment to go. *) | Fn of t list * t (* (Fn [T ...] R) *) + | CFn of t list * t (* (CFn [T ...] R) *) | Var of string (* a type variable — milestone 5 *) (* [dyn]: one machine word whose contents the runtime knows and this module does not. It is a written type — [(defonce x dyn 5)] boxes the 5 — and it @@ -142,7 +185,10 @@ let rec equal a b = | Alloc, Alloc -> true | Vec x, Vec y -> equal x y | Option x, Option y -> equal x y - | Fn (ps, r), Fn (ps', r') -> + (* The two are *not* equal to each other, in either direction. One-way + coercion lives in [Check.expect], where it can build the value the + wider type needs; here there is only identity. *) + | Fn (ps, r), Fn (ps', r') | CFn (ps, r), CFn (ps', r') -> List.length ps = List.length ps' && List.for_all2 equal ps ps' && equal r r' @@ -167,6 +213,9 @@ let rec to_string = function | Fn (ps, r) -> Printf.sprintf "(Fn [%s] %s)" (String.concat " " (List.map to_string ps)) (to_string r) + | CFn (ps, r) -> + Printf.sprintf "(CFn [%s] %s)" + (String.concat " " (List.map to_string ps)) (to_string r) | Var n -> n | Dyn -> "dyn" diff --git a/lib/x86.ml b/lib/x86.ml index 2bc86f38..2c4dc4c8 100644 --- a/lib/x86.ml +++ b/lib/x86.ml @@ -498,9 +498,15 @@ let alignof md t = snd (Emit.lay md t) than SysV's eight. *) let is_agg (t : Types.t) = match t with + (* A [(CFn ...)] is one word and crosses exactly as a pointer does, which + is the whole of its reason for existing. *) | Types.Int _ | Types.Float _ | Types.Bool | Types.Ptr _ | Types.Enum _ - | Types.Alloc | Types.Fn _ -> false + | Types.Alloc | Types.CFn _ -> false | Types.Unit | Types.Never -> false + (* A [(Fn ...)] is two words — the code address and the environment beside + it — so it crosses the way a slice does. [Emit.lay] is the one place that + says how wide it is and this agrees with it by asking. *) + | Types.Fn _ -> true | Types.String | Types.Slice _ | Types.Array _ | Types.Map _ | Types.Vec _ | Types.Option _ | Types.Named _ -> true (* A scalar, and trivially one: runtime/flan_dyn.h says [typedef uint64_t @@ -1189,6 +1195,27 @@ let load_sym f ~dst s = end else load_int f.b ~dst ~mm:(Sym (s, 0)) ~size:8 ~signed:false +(* The code address behind one of the three [fnref]s, which is the same + sequence whether it is wanted as a bare [Alloc] pointer or as the first + word of a function value. + + [Flanfn] and [Rtfn] are the symbol itself, not a load from it: a function's + address is a link-time constant, and [Flanfn] is the spelling a lifted + handler clause is reached by. [Fnval] is the one that is not — in a release + build there is nothing to redefine and it is the symbol after all; in a dev + build it is the cell's contents, so that a value taken after a redefinition + is the new body. What that does not give — and [emit.ml] names it rather + than papering over it with a trampoline — is a value taken *before* a + redefinition and called after it. Once the address is in a slot there is + nothing left to re-resolve. *) +let fnaddr f ~reg (r : Tast.fnref) = + match r with + | Tast.Flanfn n -> addr_sym f ~dst:reg (fsym n) + | Tast.Rtfn n -> addr_sym f ~dst:reg n + | Tast.Fnval n -> + if f.md.Emit.dev then load_sym f ~dst:reg (csym n) + else addr_sym f ~dst:reg (fsym n) + let scalar_size f (t : Types.t) = match t with Types.Bool -> 1 | _ -> max 1 (sizeof f.md t) @@ -1454,6 +1481,7 @@ let agg_tmp f (ty : Types.t) = let h_size = Emit.Rt.size Emit.Rt.handler let h_type = Emit.Rt.field Emit.Rt.handler "type" let h_fn = Emit.Rt.field Emit.Rt.handler "fn" +let h_env = Emit.Rt.field Emit.Rt.handler "env" let r_size = Emit.Rt.size Emit.Rt.restart let r_field = Emit.Rt.field Emit.Rt.restart @@ -1477,6 +1505,11 @@ type arg = | Aflt of loc * Types.t | Aptr of loc | Alen of loc + (* A null pointer, which is what a call site with no environment to pass + hands over — every Flan signature takes one. It is its own case rather + than a frame temporary holding zero because there is nothing to spill: + the register is zeroed where it is placed. *) + | Anull (* The C boundary, and the one place this backend must match SysV rather than pick. [check.ml] rejects an aggregate in a [declare] signature and the shim @@ -1525,6 +1558,7 @@ let emit_args f (args : arg list) = | Alen l -> load_int f.b ~dst:reg ~mm:(lmem f (shift l 8) ~scratch:r11) ~size:8 ~signed:true + | Anull -> xor_rr f.b ~dst:reg ~src:reg in (* The stack half first, because it uses rax as its courier and a register argument must not already be sitting in rax while that happens. *) @@ -1720,27 +1754,34 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit = let l = place f p in addr_into f ~reg:rax l; store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - (* The symbol itself, not a load from it: a function's address is a - link-time constant, and this is the spelling a lifted handler clause is - reached by. [emit.ml] says the same of [Flanfn]. *) - | Tast.FnAddr (Tast.Flanfn n) -> - addr_sym f ~dst:rax (fsym n); - store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - (* A function value someone wrote, which is the one [FnAddr] that is not the - symbol. In a release build there is nothing to redefine and it is the - symbol after all; in a dev build it is the cell's contents, so that a - value taken after a redefinition is the new body. What that does not give - — and [emit.ml] names it rather than papering over it with a trampoline — - is a value taken *before* a redefinition and called after it. Once the - address is in a slot there is nothing left to re-resolve. *) - | Tast.FnAddr (Tast.Fnval n) -> - if f.md.Emit.dev then - load_sym f ~dst:rax (csym n) - else addr_sym f ~dst:rax (fsym n); - store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 - | Tast.FnAddr (Tast.Rtfn n) -> - addr_sym f ~dst:rax n; + (* A [(Fn ...)] value: the code address, then the environment beside it. + Two words — see [Emit]'s %fnv. Only a [Fn]-typed node; the same three + constructors are also asked for as bare addresses, carrying [CFn] or + [Alloc], and those stay one word. The node's type says which. *) + | (Tast.FnAddr _ | Tast.Closure _ | Tast.Thicken _) + when (match t with Types.Fn _ -> true | _ -> false) -> + let env = + match e.Tast.e with + | Tast.FnAddr r -> fnaddr f ~reg:rax r; None + | Tast.Closure (r, env) -> fnaddr f ~reg:rax r; Some env + (* The widening: the thunk's code, with the bare address stored where + an environment would be. The thunk reads it back out and calls it, + which is what keeps every indirect call exactly typed. *) + | Tast.Thicken (n, p) -> addr_sym f ~dst:rax (fsym n); Some p + | _ -> assert false + in + store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8; + (match env with + | None -> xor_rr f.b ~dst:rax ~src:rax + | Some ev -> let l = eval f ev in load_loc f ~reg:rax l (Types.Ptr Types.Unit)); + store_int f.b ~src:rax ~mm:(lmem f (shift dst 8) ~scratch:r11) ~size:8 + | Tast.FnAddr r -> + fnaddr f ~reg:rax r; store_int f.b ~src:rax ~mm:(lmem f dst ~scratch:r11) ~size:8 + | Tast.Closure _ | Tast.Thicken _ -> + (* Unreachable: the arm above has taken every [Fn]-typed node, and both of + these are function values and can be nothing else. *) + unsupported "a closure that is not a function value" | Tast.Prim (p, args) -> prim f e p args dst | Tast.Call (name, args) -> (match Hashtbl.find_opt f.externs name with @@ -1752,8 +1793,17 @@ and lower_at f (e : Tast.expr) (dst : loc) : unit = ~target:(if f.md.Emit.dev then `Cell (csym name) else `Sym (fsym name)) ~args ~rty:t dst) | Tast.CallPtr (callee, args) -> + (* Through a [(Fn ...)]: both words out of one value, the code address as + the call target and the environment beside it as the extra argument. + Through a [(CFn ...)]: the address alone, and the call that follows + is the call a name would have produced. *) let c = eval f callee in - call_flan f ~target:(`Loc c) ~args ~rty:t dst + let env = + match callee.Tast.ty with + | Types.Fn _ -> Some (Aint (shift c 8, Types.Ptr Types.Unit)) + | _ -> None + in + call_flan f ?env ~target:(`Loc c) ~args ~rty:t dst | Tast.Do body -> block f body dst t | Tast.Let (bs, body) -> List.iter @@ -1953,6 +2003,16 @@ and emit_handled f frames body dst t = name it and it lives only for this body. *) addr_sym f ~dst:rax (fsym h.Tast.hfn); store_int f.b ~src:rax ~mm:(Frame (slot + h_fn)) ~size:8; + (* And the environment the clause is called with, a pointer into this + very frame. Written unconditionally — null when the clause + captured nothing — because the runtime reads the field either + way. *) + (match h.Tast.henv with + | Some ev -> + let l = scoped f (fun () -> eval f ev) in + load_loc f ~reg:rax l (Types.Ptr Types.Unit) + | None -> xor_rr f.b ~dst:rax ~src:rax); + store_int f.b ~src:rax ~mm:(Frame (slot + h_env)) ~size:8; lea f.b ~dst:rdi ~mm:(Frame slot); xor_rr f.b ~dst:rax ~src:rax; call_sym f.b "flan_handler_push"; @@ -2812,7 +2872,7 @@ and ret_loc f = if is_agg f.fret then Lp (f.sret_off, 0) else Lf f.retval integer or SSE sequence, every aggregate by pointer, a hidden [sret] in the first integer register when the result is an aggregate, and the transfer channel last of all. *) -and call_flan f ~target ~args ~rty dst = +and call_flan f ?env ~target ~args ~rty dst = let vals = List.map (fun (a : Tast.expr) -> eval f a, a.Tast.ty) args in let callee = match target with @@ -2832,7 +2892,13 @@ and call_flan f ~target ~args ~rty dst = (* The channel is this frame's own: a callee that transfers writes through the pointer we were handed, so one cell serves the whole chain. *) let chan = [ Aint (Lf f.xfer_off, Types.Ptr Types.Unit) ] in - ignore (emit_args f (head @ body @ chan)); + (* And the environment last of all, on exactly one kind of call: one through + a [(Fn ...)] value, which cannot know whether the body it reaches + declared one. Every other call passes what it always passed — this is + where an ordinary [defn] keeps costing nothing. [Emit.env_param] is where + the position is argued. *) + let tail = match env with None -> [] | Some a -> [ a ] in + ignore (emit_args f (head @ body @ chan @ tail)); (* The cell is loaded *after* the arguments, and [emit.ml] has the same as a load-bearing comment: a redefinition that lands between two calls still must not land in the middle of one. [r11] is scratch and no argument @@ -3371,11 +3437,12 @@ let frame_bytes f = ((f.maxframe + f.outgoing + 15) / 16) * 16 (* Where each argument arrives, in the order the header lays down: a hidden [sret] first when the result is an aggregate, then the parameters, then the - transfer channel. Answers one entry per incoming value — a register number, - or a positive [rbp] displacement for the ones that came on the stack. *) + environment, then the transfer channel. Answers one entry per incoming + value — a register number, or a positive [rbp] displacement for the ones + that came on the stack. *) type incoming = Ireg of int | Isse of int | Istk of int -let incoming_of ~sret (params : Types.t list) = +let incoming_of ~sret ~env (params : Types.t list) = let ints = ref 0 and sses = ref 0 and stk = ref 0 in let next_int () = if !ints < n_int_args then (incr ints; Ireg int_args.(!ints - 1)) @@ -3395,7 +3462,17 @@ let incoming_of ~sret (params : Types.t list) = else next_int ()) params in - sret_at, ps, next_int () + (* Left to right, and the two [next_int ()] calls must be sequenced: OCaml's + argument evaluation order is unspecified, so a tuple built in one + expression could hand the channel's register to the environment. + + The channel, then the environment, and the environment only on a body + that declared one — which is exactly the set of bodies an [Fn] value can + reach. Every other function is never the target of an env-passing call, + so the two never meet out of step. See [Emit.env_param]. *) + let xfer_at = next_int () in + let env_at = if env then Some (next_int ()) else None in + sret_at, ps, env_at, xfer_at (* ── The frame map ───────────────────────────────────────────────────── *) @@ -3422,7 +3499,7 @@ let where_from = function let frame_map (md : Emit.m) (fn : Tast.fn) ~slots ~fixed ~total ~outgoing ~xfer_off ~sret_off ~retval ~dframe ~dslotv ~sret ~sret_at ~param_at - ~xfer_at = + ~env_at ~xfer_at = let b = Buffer.create 1024 in let line s = Buffer.add_string b (if s = "" then "#\n" else "# " ^ s ^ "\n") in (* The prose paragraphs wrap; the table below does not, because its columns @@ -3493,10 +3570,22 @@ let frame_map (md : Emit.m) (fn : Tast.fn) ~slots ~fixed ~total ~outgoing (Types.to_string fn.Tast.ret) (if is_float fn.Tast.ret then "xmm0" else "rax"))); para (Printf.sprintf - "The transfer channel arrives last of all, %s. It is a pointer to the cell a \ - callee writes its target into, and reading it is what every guard below \ - does." - (where_from xfer_at)); + "The transfer channel arrives after the parameters, %s. It is a pointer to the \ + cell a callee writes its target into, and reading it is what every guard \ + below does.%s" + (where_from xfer_at) + (match env_at with + | None -> + " Nothing follows it: no (Fn ...) value can reach this function, so it \ + declares no environment — which is what lets an ordinary defn cost \ + exactly what it did before capture existed." + | Some at -> + Printf.sprintf + " And then the environment, %s, because a (Fn ...) value can reach this \ + function and every such call passes one. It holds the captured copies \ + when there are any and is ignored when there are not; either way the \ + signature declares it, so the call is exactly typed." + (where_from at))); line ""; para (Printf.sprintf "The frame is 0x%x bytes below rbp. %s" total @@ -3716,7 +3805,9 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) high-water mark of the temporaries and this is the boundary below which they start. It is the frame map's last line. *) let fixed = f.frame in - let sret_at, param_at, xfer_at = incoming_of ~sret fn.Tast.params in + let sret_at, param_at, env_at, xfer_at = + incoming_of ~sret ~env:(fn.Tast.fenv <> None) fn.Tast.params + in (* An aggregate parameter arrives as a pointer to the caller's copy and has to be copied into its slot before anything else runs — and [rep movsb] eats rdi, rsi and rcx, which is where three of the other parameters still @@ -3975,6 +4066,18 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) | _ -> max 1 (fst (Emit.lay md ty))) end) fn.Tast.params; + (* The environment, on the one kind of function that declared one. Every + other function never asks where it is, which is exactly why a call site + may append it whether or not the callee wanted it. *) + (match fn.Tast.fenv, env_at with + | Some slot, Some at -> + (match at with + | Ireg r -> store_int pb ~src:r ~mm:(Frame f.slots.(slot)) ~size:8 + | Istk d -> + load_int pb ~dst:rax ~mm:(Frame d) ~size:8 ~signed:false; + store_int pb ~src:rax ~mm:(Frame f.slots.(slot)) ~size:8 + | Isse _ -> unsupported "the environment in an SSE register") + | _ -> ()); (match xfer_at with | Ireg r -> store_int pb ~src:r ~mm:(Frame f.xfer_off) ~size:8 | Istk d -> @@ -4052,7 +4155,7 @@ let emit_fn (md : Emit.m) ~externs ~fns ?(ext = fun _ -> false) (frame_map md fn ~slots:f.slots ~fixed ~total:(frame_bytes f) ~outgoing:f.outgoing ~xfer_off:f.xfer_off ~sret_off:f.sret_off ~retval:f.retval ~dframe:f.dframe ~dslotv:f.dslotv ~sret ~sret_at - ~param_at ~xfer_at); + ~param_at ~env_at ~xfer_at); Buffer.add_string out (Printf.sprintf "\t.globl\t%s\n" sym); (* [emit.ml:2072] says this is load-bearing and it is: default visibility in a shared object is interposable, and that applies to taking the address @@ -4191,6 +4294,8 @@ let emit_main ?(cfi = false) ?(ann = false) ?(startup = false) ?(gc = false) let xfer = -8 and argv = -32 in xor_rr b ~dst:rax ~src:rax; store_int b ~src:rax ~mm:(Frame xfer) ~size:8; + (* [main] declares no environment — it is not reached through a function + value — so this is the call it always was. *) (match fn.Tast.params with | [] -> lea b ~dst:rdi ~mm:(Frame xfer) | [ _ ] -> @@ -4857,10 +4962,14 @@ let redefinition ~checks ?(dev = true) ?(known = fun _ -> true) (* A clause lifted out of a target comes with it: its body may have changed too, and it is reached by address from inside this module rather than through a cell. Every other lifted clause is invisible here. *) + (* The widening thunks come whole, for the reason [Emit.redefinition] + gives: a module that hands a name to an [Fn]-typed parameter names one + and the host has no cell for it. *) let lifted = List.filter (fun (f : Tast.fn) -> match f.Tast.fparent with + | Some "" -> true | Some q -> List.mem q fns | None -> false) p.Tast.fns diff --git a/runtime/flan_rt.c b/runtime/flan_rt.c index 2e79e6c9..50d92863 100644 --- a/runtime/flan_rt.c +++ b/runtime/flan_rt.c @@ -33,10 +33,21 @@ * A type is a number rather than a pointer to anything, so that a module * compiled later against a running program agrees with it: see Check.type_id. */ +/* [env] is the establishing function's copies of whatever the clause + * captured, or NULL. It is passed after the channel, and *every* clause + * declares it whether or not it captured — this walk cannot know which one + * it is about to reach, and a call whose signature is one argument longer + * than the callee's is a trap on wasm32, where call_indirect compares them. + * Emit's env_param is where the rule is written. + * + * It points into the establishing frame, which is alive for exactly as long + * as the handler frame below it is on this stack — a handler frame is popped + * by the body that pushed it, so there is no dangling case here to defer. */ typedef struct flan_handler { struct flan_handler *prev; uint32_t type_id; - void (*fn)(void *condition, void *xfer); + void (*fn)(void *condition, void *xfer, void *env); + void *env; } flan_handler; static flan_handler *handlers; @@ -64,7 +75,7 @@ void flan_handler_pop(flan_handler *h) { void flan_signal(uint32_t type_id, void *condition, void *xfer) { for (flan_handler *h = handlers; h != NULL; h = h->prev) if (h->type_id == type_id) { - h->fn(condition, xfer); + h->fn(condition, xfer, h->env); if (*(void **)xfer != NULL) return; } } @@ -1992,7 +2003,9 @@ static uint64_t flan_hash_mem(const uint8_t *p, int64_t n, uint64_t seed) { * * The pointer form has to match flan_hash_fn, whose last parameter exists * because a hash function emitted for a struct key is an ordinary Flan - * function and every Flan function's signature ends with the transfer channel. + * function and every Flan function's signature ends with the transfer + * channel. No environment: a hasher is reached from this file and never + * through a function value, so it declares none and is handed none. * The direct form has to match what such an emitted function *calls*, and an * emitted function has no channel to hand on — it would be passing its own, * which is not the same thing and not something a leaf hasher should see. So diff --git a/test/programs/fn-capture-dyn.flan b/test/programs/fn-capture-dyn.flan new file mode 100644 index 00000000..03ee5b6d --- /dev/null +++ b/test/programs/fn-capture-dyn.flan @@ -0,0 +1,13 @@ +;; A dyn is the one thing a capture refuses outright, and for the reason a +;; struct field of dyn already refuses: the collector's roots are frames, and +;; nothing pushes the fields of the environment struct a capture synthesises. +;; A copy in there would be a live value reachable only through memory the +;; marker never walks. Milestone 2's per-type descriptors lift it, alongside +;; the condition payload's and the struct field's. +(defn run [f (Fn [] i64)] i64 (f)) + +;; [d] is unannotated, which is what makes it a dyn. +(defn use [d] i64 + (run (fn [] (i64 d)))) + +(defn main [] i32 (println (use 7)) 0) diff --git a/test/programs/fn-capture-set.flan b/test/programs/fn-capture-set.flan new file mode 100644 index 00000000..6e2bc66f --- /dev/null +++ b/test/programs/fn-capture-set.flan @@ -0,0 +1,10 @@ +;; A captured name is a copy, taken where the value was made. A store into it +;; would change the copy and leave the local it came from as it was, which is +;; a silent disagreement — so it is refused, and the message says which of the +;; two would have moved. +(defn run [f (Fn [] i32)] i32 (f)) + +(defn main [] i32 + (let [n 1] + (println (run (fn [] (set n 2) n)))) + 0) diff --git a/test/programs/fn-capture.flan b/test/programs/fn-capture.flan index 7b9975f3..934e8b8e 100644 --- a/test/programs/fn-capture.flan +++ b/test/programs/fn-capture.flan @@ -1,11 +1,116 @@ -;; Capture does not exist. An fn is lifted into a function of its own and is -;; handed nothing but its parameters, so a reference to a local of the -;; enclosing function is refused by name rather than resolved to something it -;; did not mean. spec-memory.md's capture cases, and escaping closures with -;; them, are deferred; this is the refusal that says so where it happens. -(defn use [f (Fn [] i32)] i32 (f)) +;; Capture by value into a stack environment — spec-memory.md's case 2. +;; +;; An fn is still lifted into a function of its own, but it is no longer handed +;; nothing but its parameters: a local of the enclosing function that it names +;; is *copied* into an environment on that function's frame when the value is +;; made, and the lifted body reads the copy. The value is the code address and +;; that environment beside it, which is why a callee that knows only +;; (Fn [i32] i32) can still call it. +;; +;; What is not here is the escaping half — a value carrying an environment may +;; not outlive the frame the copies are on, and fn-escape*.flan is where each +;; of those refusals is written down. + +(defn double [x i32] i32 (* x 2)) + +(defn apply2 [f (Fn [i32] i32) x i32] i32 (f x)) +(defn call0 [f (Fn [] i32)] i32 (f)) +(defn twice [f (Fn [i32] i32) x i32] i32 (f (f x))) + +;; The copy is taken where the value is made and not where it is read, and +;; this is what proves it: the local is changed *after* the fn value exists +;; and before it is called, through a pointer, so nothing about the order can +;; be an accident of evaluation. +(defn bump-then-call [f (Fn [] i32) p (Ptr i32)] i32 + (set (deref p) 99) + (f)) + +(defstruct Pt [x i32 y i32]) + +(defstruct TooBig [n i32]) +(defonce seen i32) + +(defn checked [x i32] i32 + (when (> x 100) (signal (TooBig {.n x}))) + x) + +;; A handler clause is lifted the same way and captures the same way, and is +;; sound with nothing left over: a handler frame is popped by the body that +;; pushed it, so the establishing frame is alive whenever the clause runs. +;; [budget] is read out of the environment; the accumulator is a global, +;; because a captured copy is a copy and a store into one would leave the +;; local it came from as it was. +(defn handles [] i32 + (let [budget 1000 + xs [5 200 7 300] + s (slice xs 0 4) + t 0] + (handler-bind [(TooBig [c] (set seen (+ seen (+ budget (.n c)))))] + (dotimes [i 4] + (set t (+ t (checked (at s i)))))) + (print t) (print " ") (println seen) + seen)) (defn main [] i32 - (let [n 7] - (println (use (fn [] n)))) + ;; The motivating program. + (let [bonus 10] + (println (apply2 (fn [x] (+ x bonus)) 5))) + + ;; Copy at creation: the fn answers 1 and the local is 99. + (let [n 1] + (print (bump-then-call (fn [] n) (addr n))) + (print " ") + (println n)) + + ;; What may be captured. A string and a slice are two words copied as two + ;; words — the bytes stay whoever's they were, which is fine exactly while + ;; the value cannot outlive the frame that owns them. A struct and a fixed + ;; array are copied whole. A function value is copied as a function value. + (let [s "hi" + arr [1 2 3 4] + sl (slice arr 0 4) + p (Pt {.x 3 .y 4}) + g double] + (println (call0 (fn [] (i32 (length s))))) + (println (call0 (fn [] (at sl 2)))) + (println (call0 (fn [] (+ (.x p) (.y p))))) + (println (call0 (fn [] (at arr 3)))) + (println (apply2 (fn [x] (g (+ x 1))) 4))) + + ;; An fn inside an fn, each capturing. The inner one names a local neither + ;; of them declared, so the outer one captures it too and the inner one + ;; copies the outer one's copy. + (let [a 100 + b 20] + (println (apply2 (fn [x] (+ x (call0 (fn [] (+ a b))))) 3))) + + ;; A loop variable: what the fn sees is the value at the iteration it was + ;; made on, not the last one. 0 + 1 + 2 + 3. + (let [total 0] + (dotimes [i 4] + (set total (+ total (call0 (fn [] i))))) + (println total)) + + ;; And the same again where the loop variable is rebound by a recur rather + ;; than stepped by a dotimes, which is a store into the slot the copy is + ;; taken from: 100 + 101 + 102. + (println + (let [base 100] + (loop [i 0 acc 0] + (if (< i 3) + (recur (+ i 1) (+ acc (call0 (fn [] (+ base i))))) + acc)))) + + ;; Called twice, so the environment is read more than once and a body that + ;; consumed it would show. + (let [k 5] + (println (twice (fn [x] (+ x k)) 1))) + + ;; An fn's own let may shadow a name the enclosing function also has, and a + ;; store into *that* one is an ordinary store: the refusal is about a + ;; captured copy and not about the spelling. 5 + 1. + (let [n 5] + (println (+ n (call0 (fn [] (let [n 0] (set n 1) n)))))) + + (println (handles)) 0) diff --git a/test/programs/fn-cfn-captures.flan b/test/programs/fn-cfn-captures.flan new file mode 100644 index 00000000..d021597b --- /dev/null +++ b/test/programs/fn-cfn-captures.flan @@ -0,0 +1,10 @@ +;; A CFn is the bare address, so a literal written into one has nowhere to +;; keep the copies. Refused with the name of what it captured, because that is +;; the fact to act on, and with the fix named: widen the position to Fn, which +;; is what the type is for. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +(defn main [] i32 + (let [bonus 10] + (println (apply-bare (fn [x] (+ x bonus)) 5))) + 0) diff --git a/test/programs/fn-cfn-narrow.flan b/test/programs/fn-cfn-narrow.flan new file mode 100644 index 00000000..cd101f38 --- /dev/null +++ b/test/programs/fn-cfn-narrow.flan @@ -0,0 +1,13 @@ +;; Coercion between the two function types goes one way only. A (CFn ...) +;; widens into a (Fn ...) through a per-signature thunk, and a (Fn ...) does +;; not narrow: there is nowhere for the environment to go, and nothing at this +;; definition can know whether there is one. +;; +;; Refused by the ordinary type message, which names both spellings and is the +;; right sentence for it: the fix is to widen the position, not to convert the +;; value. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +(defn hand-on [f (Fn [i32] i32) x i32] i32 (apply-bare f x)) + +(defn main [] i32 0) diff --git a/test/programs/fn-cfn.flan b/test/programs/fn-cfn.flan new file mode 100644 index 00000000..2ab8ed9a --- /dev/null +++ b/test/programs/fn-cfn.flan @@ -0,0 +1,67 @@ +;; The narrow function type. A (CFn [T ...] R) is the bare code address — +;; one word, no environment, and therefore nothing that can capture. A +;; (Fn [T ...] R) is that address and the environment beside it, two words. +;; +;; The reason there are two rather than one: an environment on every signature +;; would tax every function in every program for a feature most of them never +;; use. With CFn written where it is wanted, an ordinary defn emits exactly +;; the signature it emitted before capture existed, and a call to it by name +;; is byte-for-byte what it was. +;; +;; The C is information and not decoration. A value with no environment is the +;; only kind that could ever cross to C, and under the --no-conditions +;; direction FIX.org records — where a signature that cannot transfer drops +;; the channel too — one becomes literally a C function pointer. It is not +;; that today: a declare cannot take a function type at all, and the refusal +;; it meets says so. The name points at what the type is, and at where it is +;; going. +;; +;; **Nobody needs CFn.** An Fn accepts everything a CFn does, so the narrow +;; one is reached for on purpose, for one of four reasons: handing a function +;; to C, later; a table of bare addresses; forbidding capture at a boundary; +;; and the one that is likeliest in practice — a *named* function handed to an +;; Fn parameter goes through the widening thunk and pays an indirect hop per +;; call, where a CFn parameter is a direct call. (map-in-place s double) is +;; the example, and [apply-bare] below is it in miniature. +;; +;; Coercion is one-way. A defn's address and a non-capturing literal satisfy +;; both. An Fn does not narrow to a CFn — there is nowhere for the +;; environment to go — and fn-cfn-narrow.flan is that refusal. + +(defn double [x i32] i32 (* x 2)) +(defn negate [x i32] i32 (- 0 x)) + +;; Taking the narrow one. Nothing that reaches here can carry an environment, +;; which is what the signature is saying. +(defn apply-bare [f (CFn [i32] i32) x i32] i32 (f x)) + +;; And the wide one, which is what almost every higher-order signature wants. +(defn apply-any [f (Fn [i32] i32) x i32] i32 (f x)) + +;; A CFn returned. It is a link-time constant with nothing behind it, so +;; handing one back is no different from handing it down — which is exactly +;; what a capturing value cannot do. +(defn pick [up bool] (CFn [i32] i32) (if up double negate)) + +;; A CFn parameter widened to an Fn at a call: the address goes where an +;; environment would be and the thunk reads it back out. This is the hop the +;; narrow type exists to avoid. +(defn through [f (CFn [i32] i32) x i32] i32 (apply-any f x)) + +(defn main [] i32 + ;; A name into a CFn, and into an Fn. + (println (apply-bare double 4)) + (println (apply-any negate 4)) + ;; A literal that captures nothing into a CFn. + (println (apply-bare (fn [x] (+ x 1)) 4)) + ;; And one that does capture, into an Fn. + (let [k 10] + (println (apply-any (fn [x] (+ x k)) 4))) + ;; A returned CFn, called through a computed head. + (println ((pick true) 21)) + (println ((pick false) 21)) + ;; The widening, twice over: a CFn local through a CFn parameter into + ;; an Fn parameter. + (let [g double] + (println (through g 5))) + 0) diff --git a/test/programs/fn-escape-copy.flan b/test/programs/fn-escape-copy.flan new file mode 100644 index 00000000..b9f3a1ea --- /dev/null +++ b/test/programs/fn-escape-copy.flan @@ -0,0 +1,19 @@ +;; The hole a capture could otherwise be laundered through, and the reason +;; "the result of a call is clean" is a rule and not a hope. +;; +;; [sneak]'s literal captures [g], so its body holds a *copy* of a function +;; value that may itself carry an environment — and the copy is read out of an +;; environment, which is the one aggregate a function value is ever stored in. +;; If a copy read back out were treated as clean, the literal could return it, +;; the return would arrive at [sneak]'s caller as an ordinary call result, and +;; a capturing value would be out of the frame that owns it with nothing +;; having refused anything. +;; +;; So a function value read out of a struct, a case or a pointer is suspect, +;; and the refusal lands inside the lifted body where the return is written. +(defn getf [f (Fn [] (Fn [] i32))] (Fn [] i32) (f)) + +(defn sneak [g (Fn [] i32)] (Fn [] i32) + (getf (fn [] g))) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-handled.flan b/test/programs/fn-escape-handled.flan new file mode 100644 index 00000000..8e8fd54e --- /dev/null +++ b/test/programs/fn-escape-handled.flan @@ -0,0 +1,12 @@ +;; A handler-bind is an expression and its value is its body's, so it is a way +;; for a function value to be a function's answer — and it would have walked +;; straight past a check that only looked at [return] and at the last form of +;; a block. with-allocator and restart-case are the same shape and are checked +;; the same way. +(defstruct C [id i32]) +(defonce seen i32) + +(defn keep [f (Fn [] i32)] (Fn [] i32) + (handler-bind [(C [c] (set seen (.id c)))] f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-param.flan b/test/programs/fn-escape-param.flan new file mode 100644 index 00000000..6cba4ed2 --- /dev/null +++ b/test/programs/fn-escape-param.flan @@ -0,0 +1,11 @@ +;; The hard case, answered without looking at a single call site: a function +;; value that arrives as a parameter may carry an environment on its caller's +;; frame, so a function that *stores* one is refused where it is written. +;; +;; That is what makes passing a capturing fn down safe everywhere — no callee +;; can keep it — and it is also the conservative half: this particular [keep] +;; would be harmless for a caller that passed a name, and there is no way for +;; the definition to know that it did. +(defn keep [f (Fn [] i32)] (Fn [] i32) f) + +(defn main [] i32 (println ((keep (fn [] 1)))) 0) diff --git a/test/programs/fn-escape-return.flan b/test/programs/fn-escape-return.flan new file mode 100644 index 00000000..c56617af --- /dev/null +++ b/test/programs/fn-escape-return.flan @@ -0,0 +1,10 @@ +;; The refusal that defines "non-escaping". The copies live in a slot of +;; [make]'s frame, and the value would still be pointing at them after that +;; frame has gone. +;; +;; A returned function value is still fine when it captures nothing — +;; fn-values.flan returns one — so this is about the environment and not about +;; the shape of the value. +(defn make [n i32] (Fn [] i32) (fn [] n)) + +(defn main [] i32 (println ((make 3))) 0) diff --git a/test/programs/fn-escape-store.flan b/test/programs/fn-escape-store.flan new file mode 100644 index 00000000..248c3619 --- /dev/null +++ b/test/programs/fn-escape-store.flan @@ -0,0 +1,7 @@ +;; A store through a pointer is the same escape wearing a different hat: the +;; pointer names storage this frame does not own, so the value would outlive +;; the environment it carries. +(defn stash [p (Ptr (Fn [] i32)) f (Fn [] i32)] () + (set (deref p) f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-escape-vec.flan b/test/programs/fn-escape-vec.flan new file mode 100644 index 00000000..affa50f5 --- /dev/null +++ b/test/programs/fn-escape-vec.flan @@ -0,0 +1,9 @@ +;; A Vec's elements are in a block the allocator owns and the frame does not, +;; so a function value pushed into one outlives whatever environment it +;; carries. Refused for that, and not for the shape of the element type: a Vec +;; of function values is a perfectly good thing to want, and is what case 3 +;; is for. +(defn stash [v (Vec (Fn [] i32)) f (Fn [] i32)] () + (push v f)) + +(defn main [] i32 0) diff --git a/test/programs/fn-extern.flan b/test/programs/fn-extern.flan index 1cf28821..aa74ed0e 100644 --- a/test/programs/fn-extern.flan +++ b/test/programs/fn-extern.flan @@ -1,6 +1,8 @@ ;; A foreign function's address is not a Flan function value. A Flan -;; function's emitted signature ends with the transfer channel and a C one -;; does not, so nothing could call the resulting pointer correctly — and an +;; function's emitted signature ends with the environment and the transfer +;; channel and a C one does not, so nothing could call the resulting pointer +;; correctly — and the gap is wider since capture arrived, because a Flan +;; function value is two words and a C symbol is one — and an ;; aggregate crossing the boundary is flattened by a generated shim, which the ;; raw symbol knows nothing about. Refused for what it is, with the wrapper ;; named as the way to get one. diff --git a/test/programs/fn-in-struct.flan b/test/programs/fn-in-struct.flan index 510b9b6c..661e0c32 100644 --- a/test/programs/fn-in-struct.flan +++ b/test/programs/fn-in-struct.flan @@ -4,6 +4,17 @@ ;; union's first case. So it is refused where the field is written rather than ;; left to crash at the call, and the same rule covers a global, a fixed ;; array's element and (zeroed). +;; +;; Capture sharpened the reason behind this one without changing it. A struct +;; outlives the frame it was built on, so a field could not hold a value +;; carrying an environment either — see fn-escape-*.flan. The zero is still +;; what the message names, because it is the objection that applies to every +;; function value and not only to a capturing one. +;; +;; Which means a (CFn ...) field is refused too, and for the zero alone — +;; a table of function pointers is exactly what that type is for, and nothing +;; about capture stands in its way. An (Option (CFn ...)) field is already +;; legal and is the shape that works; FIX.org carries the rest as its own item. (defstruct Ops [run (Fn [i32] i32)]) (defn main [] i32 0) diff --git a/test/programs/fn-no-type.flan b/test/programs/fn-no-type.flan index 2189b49e..944a5af2 100644 --- a/test/programs/fn-no-type.flan +++ b/test/programs/fn-no-type.flan @@ -2,6 +2,10 @@ ;; so it takes them from the position it is written in. An argument position ;; says what is wanted, because the callee's signature is threaded into every ;; argument; a let binding does not, and is refused saying so. +;; +;; The one thing capture did not change. It is about where the *types* come +;; from and not about what the body may see, so an fn is still written where +;; something says what it takes. (defn main [] i32 (let [f (fn [x] (* x 2))] (println (f 3))) diff --git a/test/programs/fn-values.flan b/test/programs/fn-values.flan index e618b5a8..52362f23 100644 --- a/test/programs/fn-values.flan +++ b/test/programs/fn-values.flan @@ -1,5 +1,13 @@ -;; Function values, the non-escaping kind: a code address and no environment -;; beside it. Capture does not exist, so nothing here can outlive anything. +;; Function values, and specifically the ones with no environment. Nothing +;; here captures, which is what makes every one of these safe to return and to +;; hand around — fn-capture.flan is the other half, and fn-escape-*.flan is +;; the line between them. +;; +;; Every signature below says (Fn ...), which is the wide one: it admits a +;; capturing value and so pays for a two-word value and a widening thunk where +;; a name is handed to it. Written as (CFn ...) these would pay neither, and +;; fn-cfn.flan is where that is spelled out — the spellings are kept apart +;; here so that the two programs cover the two conventions between them. ;; ;; This is a Lisp-1 — one top-level namespace, enforced — so a bare function ;; name *is* the function and there is no #' to write. diff --git a/test/reload_host.c b/test/reload_host.c index c22ac294..c0c0448f 100644 --- a/test/reload_host.c +++ b/test/reload_host.c @@ -43,7 +43,10 @@ /* The trailing ptr is the transfer channel spec-conditions.md §6 puts in every * Flan signature. This host never transfers, so it passes a slot of its own * that stays null — but the parameter is not optional: getting it wrong reads - * garbage as the channel and fails nowhere near here. */ + * garbage as the channel and fails nowhere near here. + * + * No environment: [outer] is called by name and not through a function value, + * so it declares none. That is the point of there being two function types. */ extern int64_t flan_outer(void *xfer) __asm__("flan.outer"); extern int64_t flan_counter __asm__("flan.counter"); diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index e6e5b395..51a8864e 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -3928,12 +3928,78 @@ level "1" outputs ~opt:"-O0" "the prelude's map, filter, reduce and sort-by, -O0" "programs/higher-order.flan" higher_order_out; - (* What function values do *not* include, each refused by name. Capture is - the headline: an fn is lifted into a function of its own and handed - nothing but its parameters, so spec-memory.md's capture cases and - escaping closures with them stay deferred. *) - refuses "an fn cannot capture" "programs/fn-capture.flan" - "cannot see n"; + (* Capture by value into a stack environment — spec-memory.md's case 2. + Three opt levels for the reason the case above has them, and for one + more: the environment is a struct in the frame and the value carries + its address, which is exactly the shape -O2 is entitled to make + disappear. -O0 is what proves there is a real store and a real load + behind it. A dev build is here because the value's code half still + comes out of the indirection cell and the environment half must not + have disturbed that. + + The two lines worth naming. "1 99" is copy-at-creation: the local is + changed through a pointer after the value exists and before it is + called, so no evaluation order can account for the fn still answering + 1. And "512 2500" is the handler clause reading a captured budget, + which is the same machinery in the one place where there is no + escaping case left over. *) + let fn_capture_out = + "15\n1 99\n2\n3\n7\n4\n10\n123\n6\n303\n11\n6\n512 2500\n2500\n" + in + outputs "an fn capturing by value" "programs/fn-capture.flan" + fn_capture_out; + outputs ~opt:"-O0" "an fn capturing by value, -O0" "programs/fn-capture.flan" + fn_capture_out; + (* The narrow function type, and the one-way coercion. What this asserts + that no checker test can: a CFn widened into an Fn and called through + the wider signature reaches the same body and answers the same thing, + on both opt levels — so the null environment a widening pairs with the + address really is ignored by a body that declared none. *) + let fn_ptr_out = "8\n-4\n5\n14\n42\n-21\n10\n" in + outputs "the two function types" "programs/fn-cfn.flan" fn_ptr_out; + outputs ~opt:"-O0" "the two function types, -O0" "programs/fn-cfn.flan" + fn_ptr_out; + (* And a dev build, which is the one that exercises the widening thunk + over an indirection cell: a name widened into an Fn is a cell load for + the address and the thunk for the call, and the two have to compose. *) + outputs ~dev:true "the two function types, dev" "programs/fn-cfn.flan" + fn_ptr_out; + outputs ~dev:true "an fn capturing by value, dev" "programs/fn-capture.flan" + fn_capture_out; + + (* What function values do *not* include, each refused by name. Escape is + the headline now that capture is not: the copies live in the frame the + literal was written in, so a value carrying their address may be + called, passed down and copied about, and may not outlive that frame. + Each of these names case 3 — the collector-allocated environment — + because "not yet" is the true sentence. *) + refuses "a captured fn cannot be returned" "programs/fn-escape-return.flan" + "a return would outlive the frame"; + refuses "a function value parameter cannot be kept" + "programs/fn-escape-param.flan" "may carry an environment"; + refuses "a function value cannot be stored through a pointer" + "programs/fn-escape-store.flan" "a store would outlive the frame"; + refuses "a function value cannot be pushed into a Vec" + "programs/fn-escape-vec.flan" "a container would outlive the frame"; + (* The two an escape check written by eye would have missed. A function + value read back out of an environment is a copy of something that may + carry one, and a handler-bind is an expression whose value is its + body's — so both are ways for a suspect to be a function's answer. *) + refuses "a captured function value cannot be handed back" + "programs/fn-escape-copy.flan" "a return would outlive the frame"; + refuses "a handler-bind's value is a return too" + "programs/fn-escape-handled.flan" "a return would outlive the frame"; + refuses "a captured local is a copy and cannot be assigned" + "programs/fn-capture-set.flan" "cannot assign to n"; + (* The two function types, and the line between them. A CFn is the bare + address, so nothing that captures can be one and nothing that may + capture can narrow into one. *) + refuses "an fn that captures is not a CFn" + "programs/fn-cfn-captures.flan" "and not a (CFn [i32] i32)"; + refuses "an Fn does not narrow to a CFn" + "programs/fn-cfn-narrow.flan" "expected (CFn [i32] i32)"; + refuses "an fn cannot capture a dyn" "programs/fn-capture-dyn.flan" + "the collector finds its roots by frame"; refuses "an fn with no type to take" "programs/fn-no-type.flan" "nothing here says what this fn"; refuses "a function value would be zeroed" "programs/fn-in-struct.flan" diff --git a/test/test_flan.ml b/test/test_flan.ml index d3faadd2..183c9747 100644 --- a/test/test_flan.ml +++ b/test/test_flan.ml @@ -473,7 +473,7 @@ let () = (match ty "(Map string i32)" with | Tapp ("Map", [ _; _ ]) -> () | _ -> check "(Map K V) is a map type" false); (match ty "(Fn [a a] bool)" with - | Tfn ([ _; _ ], _) -> () | _ -> check "(Fn [T] R)" false); + | Tfn (_, [ _; _ ], _) -> () | _ -> check "(Fn [T] R)" false); (* ── Declarations ──────────────────────────────────────────────── *) (match (parse_decl "(defn f [x i32] bool x)").d with @@ -3668,14 +3668,21 @@ let () = rejects_check "signal in value position" "(defstruct C [id i32])\n\ (defn f [] i32 (signal (C {.id 1})))" ~needle:"expected i32"; - (* A handler is lifted into a function of its own, so the establishing - function's locals are not there. Capturing them is a closure, which is - milestone 5 — until then it is refused for the reason it is refused for - rather than as an unknown name. *) - rejects_check "a handler capturing a local" + (* A handler clause captures the establishing function's locals by value — + spec-memory.md's case 2 — so it can read one. A *store* is the thing that + is not there: the clause holds a copy, and writing to it would leave the + local it came from as it was, which is a silent disagreement and not a + feature. Refused for that reason, with the accumulation case pointed at a + global. *) + accepts "a handler reading a local" + "(defstruct C [id i32])\n\ + (defonce seen i32)\n\ + (defn f [] () (let [n 7] (handler-bind [(C [c] (set seen (+ n (.id c))))] \ + (signal (C {.id 2})))))"; + rejects_check "a handler assigning to a captured local" "(defstruct C [id i32])\n\ (defn f [] () (let [n 0] (handler-bind [(C [c] (set n 1))] (signal (C {.id 2})))))" - ~needle:"a handler cannot see n"; + ~needle:"a handler cannot assign to n"; (* The frames are popped on the way out of the body, so an early exit would leave them on the stack pointing into a function that has gone. *) rejects_check "return inside handler-bind" @@ -3761,14 +3768,18 @@ let () = accepts "handler-case with several clauses" (boom ^ "(defn f [] i32 (handler-case 1 [(Boom [c] (.id c)) \ (Dud [c] (+ 1 (.id c)))]))"); - (* The whole difference from handler-bind: a clause runs at the form, in the - function that wrote it, so it sees that function's locals. The same body - under a handler-bind is refused by name. *) + (* The difference from handler-bind is narrower than it was. Both see the + establishing function's locals now — a handler-case clause *is* that + function, and a handler-bind clause captures them by value. What only a + handler-case clause can do is *assign* to one, because it is not holding + a copy. *) accepts "a handler-case clause sees the establishing function's locals" (boom ^ "(defn f [] i32 (let [n 1] (handler-case 0 [(Boom [c] n)])))"); + accepts "and may assign to one, which a handler-bind clause may not" + (boom ^ "(defn f [] i32 (let [n 1] (handler-case 0 [(Boom [c] (set n 2) n)])))"); rejects_check "a handler-bind clause still cannot" (boom ^ "(defn f [] i32 (let [n 1] (handler-bind [(Boom [c] (set n 2))] 0)))") - ~needle:"a handler cannot see n — it is a local of the enclosing function"; + ~needle:"a handler cannot assign to n"; (* Nothing static refuses a condition no clause lists: it installs no frame that matches, so it goes past untouched and the body carries on. *) accepts "a condition no clause lists" diff --git a/test/test_session.ml b/test/test_session.ml index 8bb2d271..a5017ac3 100644 --- a/test/test_session.ml +++ b/test/test_session.ml @@ -1305,7 +1305,7 @@ let () = if Session.strip_rebind "~2" <> "~2" then fail "a name that is only a suffix was stripped"; (let fn snames : Tast.fn = { Tast.name = "f"; params = []; ret = Types.Unit; body = []; - fdefers = []; fparent = None; floc = Loc.unknown; + fdefers = []; fenv = None; fparent = None; floc = Loc.unknown; slots = Array.make (Array.length snames) (Types.Int Types.I32); snames } in From 4cb53a0a9f90dd4a49bbf4ee0623c8b895f1188c Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 21 Sep 2026 12:54:13 +0700 Subject: [PATCH 2/4] What the two function types changed in the documents The four refusals the function-value lane wrote down are now three and a half: capture is a feature, the escape is the refusal in its place, and a struct field or a global of function type is refused for the same zero as before with a sharper reason behind it. The handler note is amended rather than deleted -- its reading was right, and saying so is worth more than a clean paragraph. The convention section carries both rulings, the name and every name it beat, the warning that CFn is not C interop today, the four reasons to reach for it, what was measured rather than asserted about an ordinary defn, and the wasm32 finding that killed the first design -- being exactly typed is checkable by a verifier and the alternative was an argument from a calling convention. FIX.org carries the rulings verbatim, what the escaping lane inherits and changes, the thunk's environment holding a code pointer rather than a GC object, and two things recorded and not built: a CFn under a future --no-conditions reaching C's exact convention, and CFn in a struct or a fixed array, which ZII refuses and capture has nothing to do with. --- FIX.org | 89 ++++++++++++++ docs/BUILT.md | 321 +++++++++++++++++++++++++++++++++++++++++++++---- web/index.html | 3 +- 3 files changed, 390 insertions(+), 23 deletions(-) diff --git a/FIX.org b/FIX.org index d0b9333f..ca4acdca 100644 --- a/FIX.org +++ b/FIX.org @@ -6079,3 +6079,92 @@ table names. Harmless today only because check.ml refuses a dyn field in a struct — the stopgap item 2 of the M2 queue lifts. Whoever lifts it has to root this buffer, or a collection that runs inside the construction will not see what has been built so far. +* Closures, 2026-09-21 — two rulings, and what this lane built +Two rulings, both in the author's words. + +The first, on what to build: + + "do both, capture by value and handle escaping closures, allocated on the + GC side" + +This lane is the first half: capture by value into a stack environment, +spec-memory.md's case 2, non-escaping only. The second half — an environment +the collector allocates, and with it the escaping closure — is a separate lane, +and every refusal this one prints names it. + +The second, on the calling convention, after the first design put an +environment parameter on every Flan signature: + + "while it's dyn first, static side should never have to pay the price for + the existence of the dyn side... if you fully opt out, for instance, using + --no-gc flag, then we should be operating under Odin/C semantics and never + paying any runtime costs" + +So there are two function types, Rust's and Swift's shape: + + (Fn [T ...] R) captures; {code, env}; the common case, short name + (CFn [T ...] R) the bare address; one word; cannot capture + +** The name, ruled 2026-09-21 +CFn, because the C carries information rather than being decoration: a value +with no environment is the only kind that could ever cross to C, and under the +--no-conditions direction below it becomes literally a C function pointer. The +name points at what the type is and at where it is going. + +Rejected: Closure, too long. Proc, because "procedure" is a word we disagree +with Odin about. Fun and Func, because beside Fn they differ only in length, +so nothing tells a reader which one captures. Fnptr, as ugly. + +The thing to be careful about, and it is written into the crossable refusal so +a reader meets it where they would otherwise be misled: CFn is not the type +for C interop *today*. A declare cannot take a function type at all, because a +Flan signature ends with the transfer channel. + +** Nobody needs CFn, and one of the four reasons is the common one +An Fn accepts everything a CFn does, so the narrow one is always reached for +on purpose: + +1. handing a function to C — later, per the item below; +2. a table of bare addresses; +3. forbidding capture at a boundary, where the type is the statement; +4. performance, which is likeliest in practice. A *named* function passed to + an Fn parameter goes through the widening thunk and pays an indirect hop + per call; a CFn parameter is a direct call. (map-in-place s double) is the + example. The prelude's four stay Fn on purpose — a capturing predicate is + what people want — so this is the cost someone would opt out of by writing + their own signature, not one the prelude should have avoided. + +** What the escaping lane inherits, and what it changes +Unchanged by it: a (Fn ...) is two words; the environment is the last +parameter; it is declared by exactly the bodies an Fn value can reach — a +lifted literal in an Fn position, every handler clause, and the widening +thunks. An ordinary defn declares none and is byte-for-byte what it was. + +Changed by it: where the environment points. It is a Make into a frame slot +today and becomes a collector allocation; Check.escape_check goes away with it, +and with it the refusals on returning, storing, pointing at and pushing a +capturing value. + +One thing for the collector to know now rather than discover: a widening +thunk's environment holds a *code pointer*, not a frame address and not a GC +object. When env becomes collector-allocated the marker needs a way not to +follow a thunk's. + +Also still refused and belonging to the second half: capturing a dyn. The +struct dyn field is no longer refused — per-type descriptors landed — but a +synthesised environment has no descriptor, so the refusal stands until it does. + +** Recorded, not built: CFn and C's convention +A CFn is one word and is the right shape for a C callback, and it is still +not one: a Flan function's signature ends with the transfer channel and a C +caller knows nothing about one. Under a future --no-conditions flag a CFn +signature could drop the channel and reach C's exact convention, which is the +direction the author is interested in. The refusal in [crossable] names it. + +** Also worth an item: CFn in a struct or a fixed array +no_zeroed_fn refuses a function value in any position ZII would conjure one, +and it refuses a CFn for the same reason it refuses an Fn: a zeroed function +value is a null pointer, which is the one zero that is not a value the type can +have. But "a table of function pointers" is exactly what CFn is for, and that +objection is about ZII rather than about capture — an (Option (CFn ...)) field +is already legal and is the shape that works. Its own item. diff --git a/docs/BUILT.md b/docs/BUILT.md index 56c23650..bb08a35a 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -1861,15 +1861,17 @@ stack does *not* observe a redefinition of its own clause; the next entry to the Which gives the two refusals, both by the house rule rather than by accident: -- **A handler cannot see the establishing function's locals.** That is a closure with an explicit environment, so a -reference to one is refused *for that reason* rather than reported as an unknown name. Globals and the condition are in -scope, which is what the accumulation case needs. +- **A handler cannot see the establishing function's locals.** *Superseded: it can, by value — see "Capture by value" +below. What is still refused is a `set` into one, because a captured name is a copy.* The original note read: that is a +closure with an explicit environment, so a reference to one is refused *for that reason* rather than reported as an +unknown name; globals and the condition are in scope, which is what the accumulation case needs. -What it needs is narrower than it looks, and worth getting right before anyone schedules it: a handler frame does not -outlive the function that established it, so this is spec-memory.md's **case 2** — a non-escaping `fn` capturing by -value into a stack environment — and *not* the escaping closure that plan.org's open decision #5 defers until a concrete -use case. Case 2 is settled, and #5 says in as many words that without it "conditions are not worth building". So the -biggest usability limit in conditions is not behind the thing that was just deferred. +The reading that turned out to be right: a handler frame does not outlive the function that established it, so this is +spec-memory.md's **case 2** — a non-escaping `fn` capturing by value into a stack environment — and *not* the escaping +closure that plan.org's open decision #5 defers until a concrete use case. Case 2 was settled, was built, and brought +the handler along with it for one struct field and one argument. #5 says in as many words that without it "conditions +are not worth building", and the biggest usability limit in conditions was indeed never behind the thing that was +deferred. - **`return` inside a `handler-bind` body is refused.** The frames are popped on the way out and an early exit would leave them on the stack pointing into a function that has gone. Same shape as `defer` inside a block. @@ -2702,9 +2704,11 @@ spec together; neither is a cleanup, and until one is taken the word is carried `(Map K V)` is built — see below. What is left: `drop` and with it the transitive move-only rule, recursive teardown, and the refusal to construct a drop-carrying container against an allocator without `can-free`; `(Result T E)` and `try`; generics; the macro expander. -And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy: -capture does not exist at all, and the spec's captured-`Vec`-by-pointer rule has never had to exist because every -capturable type today is a value type. It is its own item and should be planned as one. +And the **accumulation pattern** — `(fn [c] (push errors c) ...)` over an enclosing Vec — which `Vec` does not buy. +Capture exists now (see "Capture by value" below) and this is still not it: capture is by *value*, so the `fn` would +push into its own copy of the header and leave the enclosing one at the length it had. The spec's +captured-`Vec`-by-pointer rule is exactly the thing that has still never had to exist. It is its own item and should +be planned as one. ## `(Map K V)`, which is Odin's map @@ -3817,9 +3821,10 @@ cannot share a name** — which is exactly what makes the bare name safe to read `double` it could have meant instead, so the sharp quote would be punctuation answering a question the language does not ask. -A `Types.Fn` is one pointer. There is no environment beside it, so the type resolves to `ptr` and lays out as eight -bytes, and a call through one is byte-for-byte the call a name would have produced — a Flan function's emitted -signature is its parameters followed by the transfer channel whether it was reached by name or by pointer. That is +A `Types.Fn` was one pointer when this lane landed. It is two words since capture, and the one-word version has a +name of its own now — `(CFn [T ...] R)`; see "Capture by value" below. A call through either is the call a name +would have produced, with the environment appended for a `Fn`: a Flan function's emitted signature is its +parameters, then the transfer channel, and then the environment on the bodies that can be reached that way. That is why a handler established across a `fold` still catches a signal raised by the function the fold was handed: `programs/fn-values.flan` does exactly that, and it is the case that would fail if an indirect call skipped the guard. @@ -3840,11 +3845,8 @@ implemented. **Refused, each with its own reason and its own program:** -- **Capture does not exist** (`fn-capture.flan`). An `fn` is lifted into a function of its own and handed nothing but - its parameters; a reference to a local of the enclosing function is refused by name. This is the same refusal a - handler clause has always carried, and the two now share one message with the construct's name in it. - `spec-memory.md`'s capture cases, and **escaping closures with them, stay deferred** — deliberately, and this is - what keeps a function value a bare code address that cannot outlive anything. +- **Capture does not exist.** *Superseded — see "Capture by value" below. It exists, the program that was this + refusal's witness now runs, and what is refused in its place is the **escape**.* - **An `fn` with nothing to say what it takes** (`fn-no-type.flan`), above. - **A position that would zero one** (`fn-in-struct.flan`): a struct field, a global, a fixed array's element, `(zeroed)`. ZII fills an omitted field with all-bytes-zero, and **a zeroed function value is a null pointer, which @@ -3855,9 +3857,276 @@ implemented. past its length and `flan_map_alloc` zeroes only the hash run, so neither conjures an element nobody pushed or put. A function value as a map *key* is refused already, by `Types.keyable` — hashing an address is a different operation from hashing what it points at. -- **A foreign function's address** (`fn-extern.flan`). A Flan function's signature ends with the transfer channel and - a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the raw symbol knows - nothing about. Wrap it in a `defn` and pass that. +- **A foreign function's address** (`fn-extern.flan`). A Flan function's signature ends with the environment and the + transfer channel and a C one does not, and an aggregate crossing the boundary is flattened by a generated shim the + raw symbol knows nothing about. Wrap it in a `defn` and pass that. Capture widened this gap rather than closing + it: a Flan function value is now two words and a C symbol is one. + +## Capture by value, and what "non-escaping" had to mean + +`spec-memory.md`'s **case 2**, which the section above listed as the headline refusal and which is now the headline +feature. This compiles: + +``` +(let [bonus 10] + (apply2 (fn [x] (+ x bonus)) 5)) +``` + +`bonus` is **copied** into an environment on the enclosing function's frame at the instant the `fn` value is made, +and the lifted body reads the copy. Not a reference: `fn-capture.flan` changes the local through a pointer *after* +the value exists and *before* it is called, and the `fn` still answers with the old one. That test is the whole +claim, and it is the one no evaluation order can fake. + +### Two function types, because the static side does not pay for the dynamic side + +``` +(Fn [i32] i32) ; captures; {code, env}; the common case +(CFn [i32] i32) ; the bare code address; one word; cannot capture +``` + +The forcing constraint first. A callee that takes a `(Fn [i32] i32)` and calls it knows nothing about where the +value came from — `fold` is handed a value and calls it — so **the environment has to travel with the value** or +there is nowhere to put it. A `Fn` is therefore `{code, env}`: sixteen bytes, classified as an aggregate in both +backends exactly as a slice is. + +The first design put an environment parameter on **every** Flan signature, uniform for the reason the transfer +channel is uniform. The author ruled against it, and the ruling is the principle rather than the case: *"while it's +dyn first, static side should never have to pay the price for the existence of the dyn side… if you fully opt out, +for instance, using `--no-gc`, then we should be operating under Odin/C semantics and never paying any runtime +costs."* A uniform environment taxes every function in every program for a feature most of them never use. + +So there are two types, Rust's and Swift's shape. `Fn` keeps the short name because it is what almost every +higher-order signature wants; `CFn` is the narrow one. + +**The `C` is information and not decoration**, which is what settled the name. A value with no environment is the +only kind that could ever cross to C, and under the `--no-conditions` direction FIX.org records — where a signature +that cannot transfer drops the transfer channel too — one becomes literally a C function pointer. The name points at +what the type *is* and at where it is going. `Closure` was rejected as too long; `Proc` because "procedure" is a +word this language disagrees with Odin about; `Fun` and `Func` because beside `Fn` they differ only in length, so +nothing tells a reader which one captures; `Fnptr` as ugly. + +**It is not a capability today, and the diagnostic says so.** A `declare` cannot take a function type at all — a +Flan signature ends with the transfer channel and a C caller knows nothing about one — so anyone reaching for `CFn` +straight after writing a `declare-c` is reaching too early. `crossable`'s refusal names that in as many words: *"the +C in CFn is about having no environment, which is what a C function pointer would need, and not about crossing +today."* + +**And nobody ever needs it.** `Fn` accepts everything a `CFn` does, so the narrow one is reached for on purpose, for +one of four reasons: + +1. handing a function to C — later, as above; +2. a table of bare addresses; +3. forbidding capture at a boundary, where the type is the statement; +4. **performance, which is likeliest in practice.** A *named* function passed to an `Fn` parameter goes through the + widening thunk and pays an indirect hop per call; a `CFn` parameter is a direct call. `(map-in-place s double)` + is the example — and the prelude's four stay `Fn`, because a capturing predicate is exactly what people want. + +**An ordinary `defn` keeps its exact signature.** Verified rather than asserted: the LLVM for `calc-me.flan` and +fourteen corpus programs was diffed against the same compiler without this lane. Exactly three kinds of difference +appear, and no fourth: + +- two type declarations in the preamble — `%fnv`, and `%handler`'s new `env` field; +- the prelude's `sort-by-slice-u8`, whose *parameter* is now `%fnv` because it is declared `(Fn [$t $t] bool)` and + pays two words for the value it asked for; and the lifted literal it is handed, which gains a trailing `ptr %env` + because an `Fn` value can reach it; +- in a program with a `handler-bind`, its clauses gain the same trailing `ptr %env` — every clause declares one, + see below. + +**No ordinary `defn` gained a parameter, in any program.** `flan_rt.c`'s hash and equality typedefs are untouched; +`main`, the macro thunk, the startup call and the reload thunk emit the calls they always emitted. + +### The environment is the last argument, on exactly the bodies an `Fn` can reach + +The environment is the last parameter, after the transfer channel, and it is declared by **exactly the bodies a +`(Fn ...)` value can reach**: a lifted `fn` literal written into an `Fn` position, capturing or not; every handler +clause, because `flan_signal` passes one to whichever clause matched and cannot know which of them captured; and the +widening thunks below, which exist to read it. Nothing else declares it, which is where an ordinary `defn` keeps +costing nothing. + +So **every indirect call is exactly typed** and nowhere does a caller pass an argument the callee did not declare. + +That was not the first attempt. The first put the environment last and let a body that never asked for one simply +ignore the register it arrived in — legal under SysV, where argument N is classified from arguments 1..N alone, and +exactly **Swift's thin-vs-thick convention**, where a thin function converts to a thick one by pairing with a null +context the thin body ignores. It works on x86-64 and it is dead on **wasm32**, where `call_indirect` compares the +signature at the call site: a spare argument is a trap, not a register nobody reads. `test_web`'s two cases and the +headless `sand` build failed with *"null function or function signature mismatch"*, and the honest reading is that +being exactly typed is checkable by a verifier rather than argued from a calling convention, which is the better +property to have wanted. + +**So there is one adapter, and it is per *signature* rather than per function.** `CFn` → `Fn` is `Tast.Thicken`, +and the pair it builds is `{thunk, the address}`: the thunk's code, with the bare address stored where an +environment would be. The thunk — `thick/`, minted and memoised by the checker the way +`struct_key_pair` mints a map's hash and equality pair — declares the environment, reads the address back out of it, +and calls through it. One small function per distinct shape a program widens, not per function it widens, and it +handles the dynamic case (a `CFn`-typed local or parameter widened at a call) with the same mechanism as the +static one. + +**What it costs, plainly.** A *name* handed to an `Fn`-typed parameter now pays one indirect hop per call: +`(map-in-place s double)` goes through the thunk per element where it used to reach `double` directly. A literal +pays nothing — capturing or not, it is compiled to take an environment and needs no thunk. The escape hatch is +writing `CFn` in the signature, which is what the type is for; the prelude's `map`, `filter`, `reduce` and +`sort-by` correctly stay `Fn`, because a capturing comparator is exactly what people want, so the common +named-function case does pay. That is the one real price of two types, and it buys every function in every program +not paying for an environment it never has. + +A redefinition module carries its own copy of every thunk, hidden. A module that widens a name refers to one, and +the host has no cell for it to be reached through — the same shape of bug as the `Fnval` cell below, found the same +way and closed before it shipped. `flan reload` with a body that widens a name builds on both backends. + +The reverse coercion does not exist — there is nowhere for an environment to go — and is refused by the ordinary +type message, which names both spellings (`fn-cfn-narrow.flan`). A capturing literal written into a `CFn` position +is refused by name, with what it captured and the fix in the sentence (`fn-cfn-captures.flan`). + +One trap worth naming, because it is where the map would have broken: `FnAddr` is asked for by unrelated readers. A +`Fn`-typed one is two words; a `CFn`-typed or `Alloc`-typed one is a bare address — the second is what the map's +hash and equality pair and a handler frame's clause are, fields of structs the runtime declares, and they must stay +one word. **The node's type is what discriminates**, in both backends. + +A handler clause is the one place the runtime does the passing, so `flan_handler` grew an `env` field and +`flan_signal` calls `h->fn(condition, xfer, h->env)` — the same trailing position, and a clause that captured +nothing declares nothing and is unaffected. + +### The environment is a struct the checker synthesised + +One field per captured name, in first-reference order, registered in the same table a `defstruct` goes in — so both +backends lay it out with the calculator they already have and neither learns a new shape. Its name is the lifted +function's (`env/fn/OWNER/N`), which is unique and stable for the reason that name is. + +Two ends, and the copy is at the near one. In the *enclosing* frame, a slot holding the struct, filled with a `Make` +of the outer locals: that store is the copy, and it happens where the value is made. In the *lifted* frame, a slot +holding the pointer and a `Let` around the whole body reading each field back into the named slot the body was +checked against — once, at entry, so nothing downstream has to know an environment exists. + +Both of the compiler's new slots are **nameless**, which is how the break loop is told to hide them. That is a +deliberate call, not an omission: what a reader wants at a stop is the captured *copies*, and those are named slots +holding the values under the names the source gave them. An `env` pointer and a struct of bytes would be two rows of +noise above them. + +A redefinition that changes which locals an `fn` names changes an environment's layout, and `Session`'s layout guard +**exempts** these. The guard is about values the running program is holding; an environment can be in exactly one +place, a slot of the frame the literal was written in, written by the same module that reads it on every entry. A +restart for editing a capture list would take the dev loop away from the feature it was built for. + +### Escape, which is what makes "case 2" a bounded claim + +A value carrying an environment may be **called, passed down, and held in a `let`**. It may not be **returned, +stored, pointed at, or pushed into a container**. The check runs over the typed IR of every function the program +ends up with — including the lifted ones, so an `fn` inside an `fn` needs no special case — and classifies +function-typed values as *suspect* or clean: + +- suspect: a capturing literal (`Tast.Closure`, the only node that makes one); a **parameter** of type `Fn`, in + every function; an `Fn` read back out of a struct, a case or a pointer; a local bound to any of those, + transitively; a branch or a valued form whose value is one. +- clean: the address of a name, the result of any call, and **everything of type `CFn`** — the last for free, + because a `CFn` has no environment to dangle and the type says so. The second follows from the first refusal, + which is what stops a function from returning a suspect at all. + +The two types made this pass narrower rather than wider, which is the point of having them: a signature that says +`CFn` has already promised what the analysis would otherwise have to prove, and nothing written against one is +ever examined. + +Two of those arms are there because leaving them out is unsound rather than merely conservative, and each has a +program. **A function value read out of an environment** (`fn-escape-copy.flan`): a lifted body holds *copies* of +what it captured, read back with `Field(Deref env, i)`, so a copy of a captured function value carries whatever +environment the original did. Treat it as clean and the lifted body can return it, the return arrives at the outer +caller as an ordinary call result, and the whole "a call result is clean" rule has been walked around from inside. +**A valued form's tail** (`fn-escape-handled.flan`): `handler-bind`, `with-allocator` and `restart-case` are +expressions whose value is their body's — and a `restart-case`'s is a clause's too — so each is a way for a suspect +to be a function's answer that a check looking only at `return` and at the last form of a block would step over. + +**The parameter rule is the whole answer to the hard case.** A capturing `fn` passed to a function that stores it is +caught *inside that function*: its parameter is suspect there and the store is refused where it is written. So no +call can leak what its caller passed, and no caller has to be analysed. What it costs is real: +`(defn keep [f (Fn [] i32)] (Fn [] i32) f)` is refused although it is harmless, and so is holding a parameter of +function type in a `Vec` that never leaves the frame. `fn-escape-param.flan` is that refusal, written down as a +refusal of something that would sometimes have been fine. + +Refusing `Addr` of a suspect matters more than it looks: without it, `deref` of a `(Ptr (Fn ...))` launders a +suspect into a clean value and the return refusal has been walked around. Treating the `deref` itself as suspect is +the other half of that door, and it is free: nothing a `(Ptr (Fn ...))` can point at is anywhere but a frame, since +a global and a struct field of function type are both refused already. + +Name resolution inside a lifted body now asks the enclosing function's locals **before** the globals, which is a +deliberate tightening: inside the enclosing function a local shadows a global of the same name, so a body lifted out +of it must mean the same thing. The old order was an accident of where the refusal sat. + +Every one of these messages names **case 3** — the escaping closure, with an environment the collector owns — +because "this cannot be done" and "this cannot be done yet" are different sentences and the second is the true one. +Five programs: `fn-escape-return.flan`, `fn-escape-param.flan`, `fn-escape-store.flan`, `fn-escape-vec.flan`, and +`fn-capture-set.flan`. + +### What may be captured + +Anything but a **dyn**. A scalar, a struct and a fixed array copy whole. A string, a slice and a `Vec` or `Map` +header copy as their words, aliasing whatever they pointed at — which is exactly right while the value cannot +outlive the frame that owns the storage, and is exactly what would break under escape. A function value copies as a +function value, environment included; capturing one into another `fn`'s environment is the one place a suspect may +be written into an aggregate, and it is sound because the outer literal is itself suspect, so the pair of +environments lives and dies with one frame. + +A **dyn is refused**, for the reason a struct field of dyn already is (`A struct cannot hold a dyn field the +collector would never find`): the collector's roots are frames, and nothing pushes the fields of a synthesised +environment. A copy in there would be a live value reachable only through memory the marker never walks. Milestone +2's per-type descriptors lift it, alongside the condition payload's and the struct field's — and case 3's +collector-allocated environment is where it belongs anyway. `fn-capture-dyn.flan`. + +### Handlers, which get this for free and have no case 3 to wait for + +`docs/BUILT.md` said a handler clause cannot see the establishing function's locals, refused for the same reason, +and that this is also case 2. **It is, and it now can.** A handler frame is popped by the body that pushed it and +nothing in the language can name one, so the establishing frame is alive whenever the clause runs — there is no +escaping case here to leave over. The only compiler change beyond the shared machinery is one field on +`flan_handler` and one argument in `flan_signal`, which the uniform signature required anyway. + +What is still refused is a **store** into a captured name, in a clause as in an `fn`: the clause holds a copy, and +writing to it would change the copy and leave the local as it was. Scope is asked first, so a body's own `let` +shadowing a name the enclosing function also has is an ordinary local and an ordinary store — the refusal is about a +captured copy and not about a spelling. + +A **field** of a captured struct is a different matter and is deliberately left alone: `(set (.x p) 9)` inside an +`fn` writes the copy and leaves the enclosing `p` as it was — which is exactly what `(set (.x p) 9)` inside a +function whose `p` is a *parameter* already does, and has always done. Capture takes a copy the way a call takes +one, so the two agree; refusing here would make the `fn` stricter than the `defn` it was written in for no reason +anyone could state. So §1's accumulation case still accumulates into +a global — and now with whatever the establishing function knew readable beside it, which is the half that was +missing. `fn-capture.flan`'s `handles` reads a captured budget in the clause. + +Capturing the establishing frame **by reference** would make the accumulation case work directly and would be sound +here, uniquely — but it is a different feature from case 2, it would fork what "capture" means between the two +constructs, and it is not what was asked for. Named, not done. + +### Recursion, nesting and loops + +Capture is **transitive**: an `fn` inside an `fn` naming a local of the function both were written in makes the +middle one capture it and the inner one copy the middle one's copy. That is the same value, because every copy on +the way was taken at the moment its own value was made and those moments are nested. `check`'s context carries a +`parent` for exactly this, and it is safe to reach into precisely because a lifted body is checked at the point it +is written, with its parent paused there. + +An `fn` written **inside a loop** stores into the same environment slot each time round, so what it sees is the +value on its own iteration and not the last one. `fn-capture.flan` sums `0 + 1 + 2 + 3` through a fresh `fn` per +iteration to say so. The trap this could have been — a closure `set` into a local bound outside the loop and called +after it, seeing the last iteration's copies — cannot be written: a captured value cannot be `set` anywhere, and a +`let` binding scopes to the iteration. + +### What each backend cost + +Very little, which was the point of putting the environment in a frame slot and passing it as an ordinary argument +at the one call that needs it. + +`emit.ml`: a `%fnv` type and a 16-byte layout for `Fn`, `ptr` and eight for `CFn`; an `insertvalue` pair where a +symbol used to stand alone; two `extractvalue`s at a call through an `Fn`; one appended operand on that call and on +no other; one store in the prologue of a body that declared an environment. The four hand-written glue sites — the +startup call, `main`, the macro thunk and the reload thunk — are **unchanged**. + +`x86.ml`: `Fn` becomes an aggregate and `CFn` stays a scalar; one appended argument after the channel on an `Fn` +call; an optional `incoming` slot; the 16-byte value split at a `CallPtr`. The three hand-built entries are +unchanged. + +`flan_rt.c`: one field on `flan_handler` and one argument on the clause typedef, both trailing. The hash and +equality typedefs and their five call sites are **unchanged** — a hasher is reached from inside that file and never +through a function value, so it declares no environment and is handed none. ### `Fnval`, and the one thing a dev build cannot do @@ -3871,6 +4140,14 @@ What that does *not* give: a value taken *before* a redefinition and called afte address is in a slot there is nothing left to re-resolve, and the honest fix is a trampoline per function, which is a cost every program would pay for a case no one has hit. Named here rather than papered over. +**An `fn` literal was asking for `Fnval` and should never have been**, which the capture lane found as a bug rather +than as a design question: `flan reload` on any function containing an `fn` literal failed at `llc` with +`use of undefined value '@flan.cell.fn/OWNER/N'`. A lifted body has no name anyone can type and no way to be +redefined on its own — it is reached by address from the body it was written in, and a redefinition of that body +carries its own copy — so the cell could never hold anything but the symbol, and a redefinition module had no reason +to declare one. It takes `Flanfn` now, which is the choice a handler clause has always made and for the same reason. +`Fnval` remains what it was for: a `defn`'s *name* in value position. + The two lifted-function name sequences are counted **per kind** — `fn/OWNER/N` and `handler/OWNER/N/TYPE` — rather than off one list. Sharing a counter would rename every `fn` in a function the moment a `handler-bind` was added above one, which is a rename for a body that did not change, in exactly the names a redefinition module emits. diff --git a/web/index.html b/web/index.html index d48628ac..be27bf4d 100644 --- a/web/index.html +++ b/web/index.html @@ -519,7 +519,8 @@ notation reads as exactly one data item.

(Handle T)a reference into a pool that reports a dead referentindex and generation packed into an i64 (Ptr T)raw pointera pointer (Option T)Some / Nonetag byte + T -(Fn [T ...] R)a function valuea pointer +(Fn [T ...] R)a function value, which may have captureda code address and an environment pointer +(CFn [T ...] R)a function value that cannot capture — the C is what a C function pointer would need, not a way to reach C todaya pointer Allocatoran opaque builtin: a proc, its data and a capability seta pointer to that $ta type variable — see genericswhatever it is instantiated at a structvalue typefields in declaration order From a28aeda6339a4f6cd1a4057dd676c68256ac52ba Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 21 Sep 2026 13:41:36 +0700 Subject: [PATCH 3/4] The thunk memo was keyed on a name that two signatures can share Review found it, and it is a silent miscompile on both backends rather than a refusal anywhere. mangle_ty flattens a whole signature into one hyphen-joined string, so (CFn [(Ptr i32)] i32) and (CFn [ptr i32] i32) -- the second over a struct someone called ptr -- flatten alike; keyed on that, the second widening reused the first's thunk at the wrong arity. The key is the types now, compared with Types.equal, and the symbol is a counter over the thunks already minted, so nothing is derived from a spelling. fn-thunk-share.flan is the pair, and it prints 5 and 17. And a regression beside it: a generic whose function parameter binds the type variable. (defn apply2 [f (Fn [$t] $t) x $t] ...) called as (apply2 bump 1) compiled before this lane and stopped, because bind_ty had no arm admitting a CFn argument at an Fn pattern -- and once it had one, the call still handed one word to an instance declaring two, because a parameter that still mentions a variable is checked with no expectation and expect never sees the pair. Both halves: the arm, and the widening in generic_call's catch-up pass beside the numeric one. The same gap hid new functionality -- a CFn argument at a (CFn [$t] $t) parameter had no arm either -- and mentions had no CFn case, so bound_exactly answered wrong for a variable living only inside one. The corpus missed all of it because the prelude binds $t from an earlier argument, so the parameter is concrete before bind_ty sees it. While here: escaping's enumeration is the *clean* set now rather than the suspect set. It had a hole where a list like that cannot -- an (at s 0) over a slice of Fn read as clean while the Vec, struct and pointer spellings were refused. Unreachable today, and the header claims the list is closed. The same inversion fixes which refusal message an index read gets. Three minors: Anull was defined and never constructed; the capture-dyn message substituted a descriptor into a noun slot; and session.ml rendered a defn's changed signature as (Fn [...] ...), which is now a real type and not the same as (CFn [...] ...) -- it writes the parameters and the return the way a defn writes them. And one rounding corrected in the docs: handler-bind is not free for a program that captures nothing. %handler grew from 24 bytes to 32, every push writes a null into the new field, every clause gains ptr %env with an alloca and a store, and flan_signal passes one more argument per dispatch -- twenty changed x86 lines on loops.flan. Small, real, and paid by every conditions program. --- docs/BUILT.md | 38 ++++++++ lib/check.ml | 156 +++++++++++++++++++++++------- lib/session.ml | 10 +- lib/x86.ml | 6 -- test/programs/fn-escape-at.flan | 13 +++ test/programs/fn-generic.flan | 33 +++++++ test/programs/fn-thunk-share.flan | 25 +++++ test/test_acceptance.ml | 28 ++++++ 8 files changed, 268 insertions(+), 41 deletions(-) create mode 100644 test/programs/fn-escape-at.flan create mode 100644 test/programs/fn-generic.flan create mode 100644 test/programs/fn-thunk-share.flan diff --git a/docs/BUILT.md b/docs/BUILT.md index bb08a35a..a4cd75c7 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -3935,6 +3935,14 @@ appear, and no fourth: **No ordinary `defn` gained a parameter, in any program.** `flan_rt.c`'s hash and equality typedefs are untouched; `main`, the macro thunk, the startup call and the reload thunk emit the calls they always emitted. +**`handler-bind` is not free, and rounding it to zero would be wrong.** A program that establishes a handler and +captures nothing still pays: `%handler` went from 24 bytes to 32, every push writes an unconditional null into the +new field, every clause gains `ptr %env` plus an alloca and a store at entry, and `flan_signal` passes one more +argument per dispatch. Measured on `loops.flan`: +20 changed lines of x86. It is small and it is real, and every +program with conditions in it pays it — the alternative was a second clause convention beside the capturing one, +which `flan_signal` could not choose between because it calls through one C function-pointer type and cannot know +which clause matched. + ### The environment is the last argument, on exactly the bodies an `Fn` can reach The environment is the last parameter, after the transfer channel, and it is declared by **exactly the bodies a @@ -3974,6 +3982,28 @@ A redefinition module carries its own copy of every thunk, hidden. A module that the host has no cell for it to be reached through — the same shape of bug as the `Fnval` cell below, found the same way and closed before it shipped. `flan reload` with a body that widens a name builds on both backends. +**The memo is keyed on the types and the symbol is a counter**, which review found the hard way. Keyed on a *name* +derived from `mangle_ty` it was not a memo but a collision: that function flattens a whole signature into one +hyphen-joined string, so `(CFn [(Ptr i32)] i32)` and `(CFn [ptr i32] i32)` — the second over a struct someone +called `ptr` — flatten alike, and the second widening silently reused the first's thunk at the wrong arity. Both +backends compiled it without a word and neither ran it. `fn-thunk-share.flan` is that exact pair, and it prints 5 +and 17. (`mangle_ty`'s ambiguity is older than this lane and is still there for the generic instantiation names it +was written for; at one type's granularity it is hard to reach, at a whole signature's it is a line of Flan away.) + +**A generic that binds its variable *through* a function type needed both halves.** `(defn apply2 [f (Fn [$t] $t) +x $t] …)` called as `(apply2 bump 1)` is the shape a bare name has to reach now that a `defn`'s address carries +`CFn`, and it broke in two places that fail apart: `bind_ty` had no arm admitting a `CFn` argument at an `Fn` +pattern (so the instantiation was refused outright — a regression against a program that compiled before this +lane), and `generic_call`'s catch-up pass did not widen it (so the call site handed one word to an instance +declaring two). A parameter that still mentions a variable is checked with *no expectation*, by design — there is +nothing to expect until the argument has spoken — so `expect`, where the widening otherwise lives, never sees the +pair. The same gap hid the new half: a `CFn` argument at a `(CFn [$t] $t)` parameter had no arm either and fell +through to plain equality. `fn-generic.flan` covers both. + +The corpus missed all of it, and the reason is worth keeping: the prelude's higher-order functions bind `$t` from +an *earlier* argument, so `subst_ty` has already made the parameter concrete by the time `bind_ty` sees it. +`(map-in-place s double)` and `higher-order.flan` really were still working. + The reverse coercion does not exist — there is nowhere for an environment to go — and is refused by the ordinary type message, which names both spellings (`fn-cfn-narrow.flan`). A capturing literal written into a `CFn` position is refused by name, with what it captured and the fix in the sentence (`fn-cfn-captures.flan`). @@ -4026,6 +4056,14 @@ The two types made this pass narrower rather than wider, which is the point of h `CFn` has already promised what the analysis would otherwise have to prove, and nothing written against one is ever examined. +**The clean set is the enumeration, not the suspect set**, and that is a correction. It read the other way round — +`Field`, `CaseField` and `Deref` named as suspect, everything else clean — and had a hole exactly where a list like +this cannot: `(at s 0)` over a slice of `Fn` is a `Prim`, so it came out clean while the `Vec`, struct and pointer +spellings of the same act were refused. Nothing can write an `Fn` into a slice today, so it was unreachable; but +the pass claims its enumeration is closed, and a default of "clean" is how that claim stops being true without +anyone noticing. `fn-escape-at.flan` pins it. The same inversion fixed which of the two refusal messages an index +read gets. + Two of those arms are there because leaving them out is unsound rather than merely conservative, and each has a program. **A function value read out of an environment** (`fn-escape-copy.flan`): a lifted body holds *copies* of what it captured, read back with `Field(Deref env, i)`, so a copy of a captured function value carries whatever diff --git a/lib/check.ml b/lib/check.ml index 83feaf4c..43c244f2 100644 --- a/lib/check.ml +++ b/lib/check.ml @@ -657,12 +657,17 @@ let rec capture ctx loc name = match ctx.outer_what, outer with | Some what, Some (outer : binding) -> if outer.bty = Types.Dyn then + (* [what] is a descriptor — "an fn", "a handler" — so it reads as the + subject of a sentence and nowhere else. It used to be substituted + into a noun slot as well, which produced "the environment an fn is + handed"; the environment belongs to *this* capture and naming it + twice said less, not more. *) Loc.failk "check/capture-dyn" loc "%s cannot capture %s: it is a dyn, and the collector finds its \ - roots by frame — a copy inside the environment %s is handed would \ - be a live value nothing walks. Pass it in as a parameter, or hold \ - it in a global" - what name what; + roots by frame — a copy inside the environment would be a live \ + value nothing walks. Pass it in as a parameter, or hold it in a \ + global" + what name; let slot = bind ctx name outer.bty ~assignable:false in ctx.caught <- ctx.caught @ [ (name, (outer, slot)) ]; Some { slot; bty = outer.bty; assignable = false; bwhat = None } @@ -1724,7 +1729,24 @@ let rec bind_ty subst (pat : Types.t) (arg : Types.t) = | Types.Array (n, p), Types.Array (m, a) -> Int64.equal n m && bind_ty subst p a | Types.Map (k, v), Types.Map (k', v') -> bind_ty subst k k' && bind_ty subst v v' - | Types.Fn (ps, r), Types.Fn (ps', r') -> + (* Both function types, and the widening between them. + [(Fn [$t] $t)] against a [(CFn [i32] i32)] is the shape every caller of + a generic higher-order function now has, because a [defn]'s name carries + [CFn]: [(apply2 bump 1)]. It has to bind here, where the variables are + decided, and not only in [expect] — [generic_call] binds first and would + have reported the mismatch before [expect] was ever reached, which is + how this arrived as a regression against a program that used to compile. + + The prelude hides it: its higher-order functions bind [$t] from an + earlier argument, so [subst_ty] has already made the parameter concrete + by the time this sees it and [(map-in-place s double)] never took this + path. That is why the corpus stayed green over a real break. + + One way, as everywhere else: a [CFn] pattern does not admit an [Fn] + argument. *) + | Types.Fn (ps, r), Types.Fn (ps', r') + | Types.CFn (ps, r), Types.CFn (ps', r') + | Types.Fn (ps, r), Types.CFn (ps', r') -> List.length ps = List.length ps' && List.for_all2 (bind_ty subst) ps ps' && bind_ty subst r r' (* Nothing generic left on the pattern side: this is ordinary type @@ -2698,30 +2720,59 @@ let unbox_option ctx loc (t : Types.t) (got : Tast.expr) : Tast.expr = compares the signature at the call and a spare argument is a trap. Per *signature* and not per name, so a program pays one small function per - distinct shape it widens rather than one per function it widens. The name - is derived from the type, so two widenings of the same shape share it and - the memo below finds it — the same arrangement [struct_key_pair] uses for - a map's hash and equality pair, and for the same reason. + distinct shape it widens rather than one per function it widens. Two + widenings of the same shape share a thunk, which is what the memo below is + for — the same arrangement [struct_key_pair] uses for a map's hash and + equality pair, and for the same reason. + + **The memo is keyed on the types and the symbol is a counter**, and this + is not a matter of taste. [mangle_ty] flattens a whole signature into one + hyphen-joined string, which loses arity and every type boundary with it: + [(CFn [(Ptr i32)] i32)] and [(CFn [ptr i32] i32)] — the second over a + struct someone called [ptr] — both flatten to [cfn-ptr-i32-to-i32]. Keyed + on that string, the second widening silently reuses the first's thunk and + calls it with the wrong arity, which is a miscompile on both backends and + not a refusal anywhere. The types are the key, compared with + [Types.equal], and nothing is derived from a name. + + ([mangle_ty]'s ambiguity is older than this and is still there for the + generic instantiation names it was written for. At one type's granularity + it is hard to reach; at a whole signature's it is a line of Flan away.) [fparent] is []: not a name anyone wrote, so [defs] hides it, and a marker the redefinition modules match on to carry a copy of their own. *) let thick_thunk env loc ps r = - let name = "thick/" ^ mangle_ty (Types.CFn (ps, r)) in - if not (List.exists (fun (f : Tast.fn) -> f.Tast.name = name) env.lifted) - then begin + let same (f : Tast.fn) = + f.Tast.fparent = Some "" + && List.length f.Tast.params = List.length ps + && List.for_all2 Types.equal f.Tast.params ps + && Types.equal f.Tast.ret r + in + match List.find_opt same env.lifted with + | Some f -> f.Tast.name + | None -> let n = List.length ps in let fty = Types.CFn (ps, r) in let args = List.mapi (fun i t -> mk loc t (Tast.Local i)) ps in let callee = mk loc fty (Tast.Local n) in + (* Counted over the thunks already minted, which is a fact about this + compilation and not about the signature — so no two of them can share + a name however the types are spelled. *) + let name = + Printf.sprintf "thick/%d" + (List.length + (List.filter + (fun (f : Tast.fn) -> f.Tast.fparent = Some "") + env.lifted)) + in env.lifted <- { Tast.name; params = ps; slots = Array.of_list (ps @ [ fty ]); snames = Array.make (n + 1) None; ret = r; body = [ mk loc r (Tast.CallPtr (callee, args)) ]; fdefers = []; fenv = Some n; fparent = Some ""; floc = loc } - :: env.lifted - end; - name + :: env.lifted; + name (* The slot a lifted body's environment arrives in, minted when the body did not capture anything and so has none of its own. @@ -8949,7 +9000,8 @@ and generic_call ctx ~want loc name vars pats pret args = | Types.Slice e | Types.Array (_, e) | Types.Ptr e | Types.Vec e | Types.Option e -> mentions v e | Types.Map (k, w) -> mentions v k || mentions v w - | Types.Fn (ps, r) -> List.exists (mentions v) ps || mentions v r + | Types.Fn (ps, r) | Types.CFn (ps, r) -> + List.exists (mentions v) ps || mentions v r | _ -> false in let bound_exactly v = @@ -9099,7 +9151,24 @@ and generic_call ctx ~want loc name vars pats pret args = && Types.widens_to ~from:a.Tast.ty ~into:f -> widen a.Tast.loc f a | _ -> a) - | _ -> a) + (* And the other widening, for the same reason and at the same + moment: a [CFn] argument against an [(Fn [$t] $t)] parameter. + A parameter that still mentioned a variable was checked with no + expectation at all — there was nothing to expect until the + argument had spoken — so [expect] never saw the pair and never + built the value the instance's signature needs. It is built here, + once the binding is final, exactly as the numeric catch-up above + is. + + A *concrete* [Fn] parameter never reaches this: it was checked + with a want in the first pass and [expect] widened it there. *) + | _ -> + (match subst_ty !subst pat, a.Tast.ty with + | Types.Fn (ps, r), Types.CFn (ps', r') + when Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) -> + mk a.Tast.loc (Types.Fn (ps, r)) + (Tast.Thicken (thick_thunk ctx.env a.Tast.loc ps r, a)) + | _ -> a)) pats targs in (* **A type variable is not instantiated at dyn.** Nothing stopped it before: @@ -11576,21 +11645,35 @@ let rec escaping suspects (e : Tast.expr) = | Tast.RestartCase (cs, body) -> escaping suspects body || List.exists (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs - (* Read out of a struct, which for a function value means read out of an - *environment*: that is the one aggregate a capture may be written into - (see [is_env_struct]), and so the one this can be reading. A copy of a - captured function value carries whatever environment the original did, - so it is suspect exactly as the original was — without this, a lifted - body could hand back its copy of a captured value and the result would - arrive at the caller as an ordinary call result, which is to say as - clean. The same goes for a load through a pointer: nothing a - [(Ptr (Fn ...))] can point at is anywhere but a frame, since a global - and a struct field of function type are both refused. *) - | Tast.Field _ | Tast.CaseField _ | Tast.Deref _ -> true - (* The address of a name, and the result of a call. Neither can carry an - environment: the first never did, and the second cannot because a - function that would return one is refused below. *) - | _ -> false) + (* And then the clean list, which is short, closed, and stated as a list + rather than as a default — a *read* of an [Fn] from anywhere is + suspect unless it is one of these. + + It used to be the other way round, with [Field], [CaseField] and + [Deref] named as suspect and everything else clean, and that spelling + had a hole in it exactly where a list like this cannot: an [(at s 0)] + over a slice of [Fn] is a [Prim], so it read as clean while the [Vec] + and pointer spellings of the same thing were refused. Nothing can + write an [Fn] into a slice today, so it was not reachable — but the + header above claims this enumeration is closed, and a default of + [false] is how that claim stops being true without anyone noticing. + + What is clean, and why. The address of a name never carried an + environment. A widening carries a thunk and a code pointer, which is + not a frame address. And the result of a call cannot carry one, + because a function that would return one is refused below — that + refusal is what this arm rests on, which is why the two have to be + read together. *) + | Tast.FnAddr _ | Tast.Thicken _ | Tast.Call _ | Tast.CallPtr _ -> false + (* Everything else that can produce an [Fn]: read out of a struct — which + for a function value means read out of an *environment*, the one + aggregate a capture may be written into — out of a case, through a + pointer, or out of a container. A copy of a captured function value + carries whatever environment the original did, so it is suspect + exactly as the original was: without this, a lifted body could hand + back its copy of a captured value and the result would arrive at the + caller as an ordinary call result, which is to say as clean. *) + | _ -> true) | _ -> false (* The environment struct a capture built, which is the one aggregate a @@ -11619,7 +11702,6 @@ let escape_check (fn : Tast.fn) = in match e.Tast.e with | Tast.Closure _ -> true - | Tast.Local _ | Tast.Field _ | Tast.CaseField _ | Tast.Deref _ -> false | Tast.If (_, a, b) -> written_here a && written_here b | Tast.Do body | Tast.Let (_, body) | Tast.Handled (_, body) | Tast.WithAlloc (_, body) -> tail body @@ -11628,7 +11710,13 @@ let escape_check (fn : Tast.fn) = | Tast.RestartCase (cs, body) -> written_here body && List.for_all (fun (c : Tast.rclause) -> tail c.Tast.rbody) cs - | _ -> true + (* Everything else is a *read* of a value made elsewhere — a local, a + field, an index, a load through a pointer — and the message that fits + one is the other message, about a value this definition did not make + and cannot see into. The literal is the short list here, exactly as + the clean set is the short list in [escaping]; whichever is short is + the one to write out. *) + | _ -> false in let refuse (e : Tast.expr) where = if not (written_here e) then diff --git a/lib/session.ml b/lib/session.ml index 73ecd749..f6ea36f2 100644 --- a/lib/session.ml +++ b/lib/session.ml @@ -282,8 +282,16 @@ let compatible ?(origin = fun _ -> None) ?(relaxed = []) ~loc refusal is about a function the source does not name." gname ) in + (* The parameters and the return as a [defn] writes them, and not + as [(Fn [...] ...)]. That spelling was harmless while [Fn] was + the only function type and is not now: it is a real type, it is + not the same as [(CFn [...] ...)], and a *declaration* is + neither of them — rendering one as a type invites a reader to + go looking for which of the two this function's name carries, + which is a question about taking its address and not about the + edit that was refused. *) fail loc - "%s changes signature, from (Fn [%s] %s) to (Fn [%s] %s).%s \ + "%s changes signature, from [%s] %s to [%s] %s.%s \ Restart to change it." what (String.concat " " (List.map Types.to_string g.Tast.params)) diff --git a/lib/x86.ml b/lib/x86.ml index 2c4dc4c8..b1c52d1c 100644 --- a/lib/x86.ml +++ b/lib/x86.ml @@ -1505,11 +1505,6 @@ type arg = | Aflt of loc * Types.t | Aptr of loc | Alen of loc - (* A null pointer, which is what a call site with no environment to pass - hands over — every Flan signature takes one. It is its own case rather - than a frame temporary holding zero because there is nothing to spill: - the register is zeroed where it is placed. *) - | Anull (* The C boundary, and the one place this backend must match SysV rather than pick. [check.ml] rejects an aggregate in a [declare] signature and the shim @@ -1558,7 +1553,6 @@ let emit_args f (args : arg list) = | Alen l -> load_int f.b ~dst:reg ~mm:(lmem f (shift l 8) ~scratch:r11) ~size:8 ~signed:true - | Anull -> xor_rr f.b ~dst:reg ~src:reg in (* The stack half first, because it uses rax as its courier and a register argument must not already be sitting in rax while that happens. *) diff --git a/test/programs/fn-escape-at.flan b/test/programs/fn-escape-at.flan new file mode 100644 index 00000000..33e7816e --- /dev/null +++ b/test/programs/fn-escape-at.flan @@ -0,0 +1,13 @@ +;; An index read is a read, and the escape check's clean list has to be a +;; list. A function value out of a slice is refused exactly as one out of a +;; Vec, a struct or a pointer is — the four are the same act and there is no +;; reason for a reader to have to remember which spellings were enumerated. +;; +;; Not reachable today: nothing can write an Fn into a slice, because every +;; position that would have to hold one is refused. It is here so that the +;; day one can, this is already true — the alternative was a default of +;; "clean" for anything the enumeration had not thought of, which is how a +;; closed list quietly stops being closed. +(defn leak [s [(Fn [] i32)]] (Fn [] i32) (at s 0)) + +(defn main [] i32 0) diff --git a/test/programs/fn-generic.flan b/test/programs/fn-generic.flan new file mode 100644 index 00000000..6505a72c --- /dev/null +++ b/test/programs/fn-generic.flan @@ -0,0 +1,33 @@ +;; A generic whose function parameter binds the type variable, which is the +;; shape a name now has to reach: a defn's address carries (CFn [i32] i32), +;; and (apply2 bump 1) has to bind $t from it and then widen the argument. +;; +;; Both halves are here because they fail apart. The binding is Check's +;; bind_ty, which runs inside generic_call and decides the instantiation; the +;; widening is the catch-up pass at the end of the same function, because a +;; parameter that still mentioned a variable was checked with no expectation +;; at all and expect never saw the pair. Get the first without the second and +;; the call site hands one word to an instance that declares two. +;; +;; The prelude does not cover this and that is worth saying: its higher-order +;; functions bind $t from an *earlier* argument, so the parameter is already +;; concrete by the time the function value is reached, and (map-in-place s +;; double) never walks this path at all. +;; +;; The CFn half is new rather than restored: a matching bare address against +;; a (CFn [$t] $t) parameter had no arm either, and fell through to plain +;; equality. + +(defn apply2 [f (Fn [$t] $t) x $t] $t (f (f x))) +(defn applyc [f (CFn [$t] $t) x $t] $t (f (f x))) + +(defn bump [n i32] i32 (+ n 1)) +(defn twice [x f64] f64 (* x 2.0)) + +(defn main [] i32 + (println (apply2 bump 1)) + (println (applyc bump 1)) + ;; A second instantiation, so the two copies are really two and the thunk + ;; the first one minted is not reused at the wrong signature. + (println (apply2 twice 1.5)) + 0) diff --git a/test/programs/fn-thunk-share.flan b/test/programs/fn-thunk-share.flan new file mode 100644 index 00000000..946ed730 --- /dev/null +++ b/test/programs/fn-thunk-share.flan @@ -0,0 +1,25 @@ +;; The widening thunk is memoised per signature, and the key is the types. +;; +;; It used to be a *name*, derived from mangle_ty, which flattens a whole +;; signature into one hyphen-joined string and loses arity and every type +;; boundary with it: (CFn [(Ptr i32)] i32) and (CFn [ptr i32] i32) — the +;; second over a struct someone called ptr — both flatten to the same thing. +;; Keyed on that, the second widening reuses the first's thunk and calls it +;; with the wrong arity, which both backends compile without a word and +;; neither runs. This program is that pair, and it prints 5 and 17. +;; +;; A struct named ptr is legal and ordinary; nothing about the collision +;; needed a program written to provoke it, only two signatures that happened +;; to flatten alike. +(defstruct ptr [a i32 b i32]) + +(defn f1 [p (Ptr i32)] i32 (deref p)) +(defn f2 [a ptr b i32] i32 (+ (.a a) b)) + +(defn use1 [f (Fn [(Ptr i32)] i32)] i32 (let [x 5] (f (addr x)))) +(defn use2 [f (Fn [ptr i32] i32)] i32 (f (ptr {.a 10 .b 0}) 7)) + +(defn main [] i32 + (println (use1 f1)) + (println (use2 f2)) + 0) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index 51a8864e..fc9083ea 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -3964,6 +3964,32 @@ level "1" the address and the thunk for the call, and the two have to compose. *) outputs ~dev:true "the two function types, dev" "programs/fn-cfn.flan" fn_ptr_out; + + (* Two signatures that flatten to one string under [mangle_ty], which is + how the thunk memo used to be keyed. Keyed on the name, the second + widening reuses the first's thunk at the wrong arity — a miscompile + both backends emit without a word. Keyed on the types, this prints + 5 and 17. *) + outputs "two signatures that mangle alike" "programs/fn-thunk-share.flan" + "5\n17\n"; + outputs ~opt:"-O0" "two signatures that mangle alike, -O0" + "programs/fn-thunk-share.flan" "5\n17\n"; + + (* A generic whose *function* parameter binds the type variable, which is + the shape a bare name has to reach now that a defn's address carries + CFn. It fails in two places and they fail apart: the binding, in + bind_ty, and the widening, in generic_call's catch-up pass — a + parameter that still mentioned a variable was checked with no + expectation, so expect never saw the pair. The prelude misses it + entirely, because its higher-order functions bind $t from an earlier + argument and the parameter is concrete by the time the function value + is reached. *) + let fn_generic_out = "3\n3\n6\n" in + outputs "a generic that binds its variable through a function type" + "programs/fn-generic.flan" fn_generic_out; + outputs ~opt:"-O0" + "a generic that binds its variable through a function type, -O0" + "programs/fn-generic.flan" fn_generic_out; outputs ~dev:true "an fn capturing by value, dev" "programs/fn-capture.flan" fn_capture_out; @@ -3985,6 +4011,8 @@ level "1" value read back out of an environment is a copy of something that may carry one, and a handler-bind is an expression whose value is its body's — so both are ways for a suspect to be a function's answer. *) + refuses "an index read is a read like any other" + "programs/fn-escape-at.flan" "may carry an environment"; refuses "a captured function value cannot be handed back" "programs/fn-escape-copy.flan" "a return would outlive the frame"; refuses "a handler-bind's value is a return too" From 0c711942f747da96b44180e89682154344d0b5fe Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Mon, 21 Sep 2026 15:44:28 +0700 Subject: [PATCH 4/4] A thunk named for the order it was minted in is a different signature after a reorder Three from a review of the commit before this one, and the first is a dev loop defect. Naming the widening thunk thick/ put traversal order into a symbol, and Session.compatible compares a reload's functions against the running program's by name -- so swapping two calls in a body renamed nothing the programmer can see and reported "thick/0 changes signature, from [i32] i32 to [i64] i64. Restart to change it." The name is the signature again, written by thick_enc: every type self-delimiting, an atom as its length and then its spelling, so the boundaries mangle_ty loses are in the string rather than inferred from the separators. mangle_ty is untouched -- it is load-bearing for the instantiation names a backtrace, a Reach edge and a dev cell show -- and the memo stays keyed on the types. fn-thunk-reload.flan is the pair of widenings and test_session reorders them. The second was a silent miscompile. bind_ty's Fn-pattern-against-CFn-argument arm was structural, so it matched nested function positions too, and the catch-up pass that builds the thunk compares the whole substituted parameter list -- false when the mismatch is inside one, and the fallthrough handed the argument over unchanged: one word where the instance declares two, 255 on both backends with nothing said anywhere. The widening is a value the caller builds around the whole argument and there is nowhere inside one to build it, so the arm is admitted at the top of an argument's type and nowhere else, and the catch-up arm is total over the pair rather than guarded -- bind_ty's fallthrough is Types.fits, which admits Never, so a binding can still succeed over a pair the two words cannot bridge. fn-generic-nested.flan is the parameter position and fn-generic-nested-return.flan the return one. And the third closes the hole the escaping inversion was for. A match arm binds the case's fields to slots and the store that fills them is inside the branch, not in any form escape_check reads as a binding, so (match o (Some f) f ...) handed back through a name what a case-field read cannot hand back at all -- sound only because Some_ denies suspects. An arm's slots of function type are suspect now, for the same reason the read itself is. --- lib/check.ml | 154 +++++++++++++++----- test/programs/fn-escape-match.flan | 16 ++ test/programs/fn-generic-nested-return.flan | 15 ++ test/programs/fn-generic-nested.flan | 19 +++ test/programs/fn-thunk-reload.flan | 24 +++ test/test_acceptance.ml | 15 ++ test/test_session.ml | 24 +++ 7 files changed, 231 insertions(+), 36 deletions(-) create mode 100644 test/programs/fn-escape-match.flan create mode 100644 test/programs/fn-generic-nested-return.flan create mode 100644 test/programs/fn-generic-nested.flan create mode 100644 test/programs/fn-thunk-reload.flan diff --git a/lib/check.ml b/lib/check.ml index 43c244f2..5d05d087 100644 --- a/lib/check.ml +++ b/lib/check.ml @@ -1716,7 +1716,7 @@ let signature_tyvars (fn : Ast.fn) = and with the same rule: a variable already bound must match what it is bound to, so [(pair 1 2.0)] over [a $t b $t] is a refusal and not a second instantiation. *) -let rec bind_ty subst (pat : Types.t) (arg : Types.t) = +let rec bind_ty ?(widen = false) subst (pat : Types.t) (arg : Types.t) = match pat, arg with | Types.Var v, a -> (match List.assoc_opt v !subst with @@ -1729,24 +1729,36 @@ let rec bind_ty subst (pat : Types.t) (arg : Types.t) = | Types.Array (n, p), Types.Array (m, a) -> Int64.equal n m && bind_ty subst p a | Types.Map (k, v), Types.Map (k', v') -> bind_ty subst k k' && bind_ty subst v v' - (* Both function types, and the widening between them. + (* Each function type against its own. *) + | Types.Fn (ps, r), Types.Fn (ps', r') + | Types.CFn (ps, r), Types.CFn (ps', r') -> + List.length ps = List.length ps' + && List.for_all2 (bind_ty subst) ps ps' && bind_ty subst r r' + (* And the widening between them, which is admitted at the top of an + argument's type and nowhere inside it. [(Fn [$t] $t)] against a [(CFn [i32] i32)] is the shape every caller of - a generic higher-order function now has, because a [defn]'s name carries + a generic higher-order function has, because a [defn]'s name carries [CFn]: [(apply2 bump 1)]. It has to bind here, where the variables are decided, and not only in [expect] — [generic_call] binds first and would - have reported the mismatch before [expect] was ever reached, which is - how this arrived as a regression against a program that used to compile. + have reported the mismatch before [expect] was ever reached. - The prelude hides it: its higher-order functions bind [$t] from an + The prelude hides that: its higher-order functions bind [$t] from an earlier argument, so [subst_ty] has already made the parameter concrete by the time this sees it and [(map-in-place s double)] never took this - path. That is why the corpus stayed green over a real break. + path. + + [widen] is why it goes no deeper. The widening is a *value* the caller + builds — a thunk, minted at the call — and there is exactly one place to + build it, around the whole argument. A [(Fn [(Fn [$t] $t)] i32)] + parameter handed a [(CFn [(CFn [i32] i32)] i32)] would need one built + inside the argument's own parameter list, where no caller stands, so the + two types do not meet there and the pattern does not match. What reaches + the fallthrough below is an ordinary mismatch and is refused as one, the + same answer a call with no type variables in it gets. One way, as everywhere else: a [CFn] pattern does not admit an [Fn] argument. *) - | Types.Fn (ps, r), Types.Fn (ps', r') - | Types.CFn (ps, r), Types.CFn (ps', r') - | Types.Fn (ps, r), Types.CFn (ps', r') -> + | Types.Fn (ps, r), Types.CFn (ps', r') when widen -> List.length ps = List.length ps' && List.for_all2 (bind_ty subst) ps ps' && bind_ty subst r r' (* Nothing generic left on the pattern side: this is ordinary type @@ -2725,22 +2737,63 @@ let unbox_option ctx loc (t : Types.t) (got : Tast.expr) : Tast.expr = for — the same arrangement [struct_key_pair] uses for a map's hash and equality pair, and for the same reason. - **The memo is keyed on the types and the symbol is a counter**, and this - is not a matter of taste. [mangle_ty] flattens a whole signature into one - hyphen-joined string, which loses arity and every type boundary with it: - [(CFn [(Ptr i32)] i32)] and [(CFn [ptr i32] i32)] — the second over a - struct someone called [ptr] — both flatten to [cfn-ptr-i32-to-i32]. Keyed - on that string, the second widening silently reuses the first's thunk and - calls it with the wrong arity, which is a miscompile on both backends and - not a refusal anywhere. The types are the key, compared with - [Types.equal], and nothing is derived from a name. + **The memo is keyed on the types and the symbol spells them back**, and + both halves matter. [mangle_ty] cannot serve as the spelling: it flattens + a whole signature into one hyphen-joined string, which loses arity and + every type boundary with it, so [(CFn [(Ptr i32)] i32)] and + [(CFn [ptr i32] i32)] — the second over a struct someone called [ptr] — + both come out [cfn-ptr-i32-to-i32]. Keyed on that string, the second + widening silently reuses the first's thunk and calls it with the wrong + arity, which is a miscompile on both backends and not a refusal anywhere. + So the key is the types, compared with [Types.equal], and the name is + [thick_enc]'s encoding, which no two signatures share. ([mangle_ty]'s ambiguity is older than this and is still there for the generic instantiation names it was written for. At one type's granularity it is hard to reach; at a whole signature's it is a line of Flan away.) + **Why the name has to be the signature and not a counter.** A counter over + the thunks minted so far is unique within one compilation and says nothing + across two: reorder the definitions in the file and [thick/0] is a + different signature than it was. [Session.compatible] compares a reload's + functions against the running program's *by name*, so a thunk that changed + shape under a fixed name reads to it as a function whose signature was + edited, and the dev loop answers a form reorder with "Restart to change + it". Spelled from the types, the name moves with the shape and that + comparison is right again for the same reason it is right everywhere else. + [fparent] is []: not a name anyone wrote, so [defs] hides it, and a marker the redefinition modules match on to carry a copy of their own. *) + +(* A signature written so that it can be read back: every type is + self-delimiting, so no two distinct signatures encode alike. + + An atom is its length and then its spelling, which is what closes the gap + [mangle_ty] leaves — a name's boundaries are in the string rather than + inferred from the separators. A constructor is one letter, and the two + that hold a count write it before their children, so [(Fn [i32] i32)] and + [(Fn [] (Fn [i32] i32))] cannot read alike. [Named] and [Enum] carry + different letters because a struct and a C enum may share a spelling. *) +let rec thick_enc (t : Types.t) = + let atom s = Printf.sprintf "%d-%s" (String.length s) s in + let arrow tag ps r = + Printf.sprintf "%s%d-%s" tag (List.length ps) + (String.concat "-" (List.map thick_enc (ps @ [ r ]))) + in + match t with + | Types.Named n -> "n" ^ atom n + | Types.Enum n -> "e" ^ atom n + | Types.Var v -> "y" ^ atom v + | Types.Slice e -> "s" ^ thick_enc e + | Types.Ptr e -> "p" ^ thick_enc e + | Types.Vec e -> "v" ^ thick_enc e + | Types.Option e -> "o" ^ thick_enc e + | Types.Array (n, e) -> Printf.sprintf "a%Ld-%s" n (thick_enc e) + | Types.Map (k, v) -> Printf.sprintf "m%s-%s" (thick_enc k) (thick_enc v) + | Types.Fn (ps, r) -> arrow "f" ps r + | Types.CFn (ps, r) -> arrow "c" ps r + | t -> atom (mangle_ty t) + let thick_thunk env loc ps r = let same (f : Tast.fn) = f.Tast.fparent = Some "" @@ -2755,16 +2808,7 @@ let thick_thunk env loc ps r = let fty = Types.CFn (ps, r) in let args = List.mapi (fun i t -> mk loc t (Tast.Local i)) ps in let callee = mk loc fty (Tast.Local n) in - (* Counted over the thunks already minted, which is a fact about this - compilation and not about the signature — so no two of them can share - a name however the types are spelled. *) - let name = - Printf.sprintf "thick/%d" - (List.length - (List.filter - (fun (f : Tast.fn) -> f.Tast.fparent = Some "") - env.lifted)) - in + let name = "thick/" ^ thick_enc fty in env.lifted <- { Tast.name; params = ps; slots = Array.of_list (ps @ [ fty ]); @@ -9102,7 +9146,9 @@ and generic_call ctx ~want loc name vars pats pret args = true) | _ -> false in - if (not handled) && not (bind_ty subst p a.Tast.ty) then + (* [~widen]: this is the top of an argument's type, which is the one + place a widening thunk can be built around it. See [bind_ty]. *) + if (not handled) && not (bind_ty ~widen:true subst p a.Tast.ty) then fail a.Tast.loc "%s expects %s here, found %s" name (Types.to_string p) (Types.to_string a.Tast.ty); a) @@ -9161,13 +9207,27 @@ and generic_call ctx ~want loc name vars pats pret args = is. A *concrete* [Fn] parameter never reaches this: it was checked - with a want in the first pass and [expect] widened it there. *) + with a want in the first pass and [expect] widened it there. + + The arm is total over the pair, and that is the point of writing + it as an [if] rather than as a guard. [bind_ty]'s fallthrough is + [Types.fits], which admits [Never] where the instance's signature + wants a type — so a binding can succeed over a pair these two + words cannot bridge, and a fallthrough of "hand the argument over + unchanged" would pass one word where the instance declares two. + Every [CFn] arriving at an [Fn] parameter either gets its thunk + here or gets the refusal, which is the answer [expect] gives a + call with no type variables in it. *) | _ -> (match subst_ty !subst pat, a.Tast.ty with - | Types.Fn (ps, r), Types.CFn (ps', r') - when Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) -> - mk a.Tast.loc (Types.Fn (ps, r)) - (Tast.Thicken (thick_thunk ctx.env a.Tast.loc ps r, a)) + | Types.Fn (ps, r), Types.CFn (ps', r') -> + if Types.equal (Types.Fn (ps, r)) (Types.Fn (ps', r')) then + mk a.Tast.loc (Types.Fn (ps, r)) + (Tast.Thicken (thick_thunk ctx.env a.Tast.loc ps r, a)) + else + fail a.Tast.loc "%s expects %s here, found %s" name + (Types.to_string (Types.Fn (ps, r))) + (Types.to_string a.Tast.ty) | _ -> a)) pats targs in @@ -11745,10 +11805,32 @@ let escape_check (fn : Tast.fn) = let go (e : Tast.expr) = (match e.Tast.e with | Tast.Let (bs, _) -> - (* A binding is where a suspect spreads, and the only place it does. *) + (* A binding is where a suspect spreads, and one of the two places it + does. *) List.iter (fun (slot, v) -> if escaping suspects v then suspects := slot :: !suspects) bs + (* And the other: an arm's pattern binds the case's fields to slots, and + the store that fills them is inside the branch rather than in a form + this walk reads as a binding. Reading the same field by hand is a + [CaseField] and suspect — a copy of a captured function value carries + whatever environment the original did — so the slot the pattern binds + it to is suspect too, or [(match o (Some f) f ...)] would hand back + through a name what [(case-field o ...)] cannot hand back at all. + + Every arm's binds, not only an [Option]'s: a data type's field of + function type is written through [MakeCase], which denies suspects, + and read back through this. *) + | Tast.Match (_, arms) -> + List.iter + (fun (a : Tast.arm) -> + List.iter + (fun s -> + match fn.Tast.slots.(s) with + | Types.Fn _ -> suspects := s :: !suspects + | _ -> ()) + a.Tast.binds) + arms | Tast.Set (_, v) -> deny "a store" [ v ] | Tast.Return (Some v) -> deny "a return" [ v ] | Tast.Some_ v -> deny "an Option" [ v ] diff --git a/test/programs/fn-escape-match.flan b/test/programs/fn-escape-match.flan new file mode 100644 index 00000000..1fe2b10c --- /dev/null +++ b/test/programs/fn-escape-match.flan @@ -0,0 +1,16 @@ +;; A match arm's binding is a binding, and the escape check has to see it. +;; +;; Reading the payload by hand is a case-field read, which is suspect: a copy +;; of a function value carries whatever environment the original did. Binding +;; it to a name in an arm is the same read, and the store that fills the arm's +;; slot is inside the branch rather than in any form the walk reads as a +;; binding — so without the arm's slots being taken as suspect too, the Vec, +;; slice, struct and pointer spellings of this were all refused while the one +;; that goes through Option and a name was not. + +(defn leak [o (Option (Fn [] i32))] (Fn [] i32) + (match o + (Some f) f + None (fn [] 0))) + +(defn main [] i32 0) diff --git a/test/programs/fn-generic-nested-return.flan b/test/programs/fn-generic-nested-return.flan new file mode 100644 index 00000000..37343151 --- /dev/null +++ b/test/programs/fn-generic-nested-return.flan @@ -0,0 +1,15 @@ +;; The same line drawn in return position, which is the half that is easy to +;; miss: a (Fn [] (Fn [] $t)) parameter handed a (CFn [] (CFn [] i32)) needs +;; the inner widening built by whatever calls the *argument*, and that is the +;; generic's own body, which was compiled against the parameter's type and not +;; against this caller's. + +(defn inner [] i32 3) + +(defn outer [] (CFn [] i32) inner) + +(defn call-twice [g (Fn [] (Fn [] $t))] $t ((g))) + +(defn main [] i32 + (println (call-twice outer)) + 0) diff --git a/test/programs/fn-generic-nested.flan b/test/programs/fn-generic-nested.flan new file mode 100644 index 00000000..51b74840 --- /dev/null +++ b/test/programs/fn-generic-nested.flan @@ -0,0 +1,19 @@ +;; The widening between the two function types is a value the caller builds — +;; a thunk, minted at the call — so there is exactly one place to build it: +;; around the whole argument. Nested inside one, there is no caller standing +;; where the thunk would have to go. +;; +;; The parameter here is (Fn [(Fn [$t] $t)] i32) and taker's address carries +;; (CFn [(CFn [i32] i32)] i32). Admitting that structurally binds $t and then +;; hands one word where the instance declares two, in a position no later pass +;; can widen: the catch-up that builds the outer thunk compares the whole +;; substituted parameter list and this mismatch is inside it. A call with no +;; type variables in it is refused, so this one is too. + +(defn taker [h (CFn [i32] i32)] i32 (h 1)) + +(defn hof [g (Fn [(Fn [$t] $t)] i32) k $t] i32 (g (fn [x] x))) + +(defn main [] i32 + (println (hof taker 0)) + 0) diff --git a/test/programs/fn-thunk-reload.flan b/test/programs/fn-thunk-reload.flan new file mode 100644 index 00000000..b8f815bd --- /dev/null +++ b/test/programs/fn-thunk-reload.flan @@ -0,0 +1,24 @@ +;; Two widenings of different signatures in one program, so the dev loop has +;; two thunks to tell apart across a reload. +;; +;; A thunk is not a function anybody wrote, so nothing in the source names it +;; and nothing can be edited to rename it. But Session.compatible compares a +;; reload's functions against the running program's *by name*, and a name that +;; means whichever thunk was minted first means a different signature the +;; moment the forms are reordered — which reads to the session as a function +;; whose signature was edited, and answers an ordinary edit with "Restart to +;; change it". So the name is the signature, spelled so that it can be read +;; back, and reordering these two calls is an ordinary body change. + +(defn a1 [x i32] i32 (+ x 1)) +(defn b1 [x i64] i64 (+ x 1)) + +(defn use32 [f (Fn [i32] i32)] i32 (f 1)) +(defn use64 [f (Fn [i64] i64)] i64 (f 1)) + +(defn both [] i32 + (println (use32 a1)) + (println (use64 b1)) + 0) + +(defn main [] i32 (both)) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index fc9083ea..6de3de75 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -3990,6 +3990,19 @@ level "1" outputs ~opt:"-O0" "a generic that binds its variable through a function type, -O0" "programs/fn-generic.flan" fn_generic_out; + (* And where the same binding stops. The widening is a value the caller + builds around the whole argument, so a function type nested inside an + argument's own type has no caller standing where its thunk would go — + and the catch-up pass that builds the outer one compares the whole + substituted parameter list, so a mismatch inside it is not something + any later pass can repair. Both positions, because the return one is + the easier of the two to leave open. *) + refuses "a nested function type does not widen" + "programs/fn-generic-nested.flan" + "hof expects (Fn [(Fn [t] t)] i32) here"; + refuses "and neither does one in return position" + "programs/fn-generic-nested-return.flan" + "call-twice expects (Fn [] (Fn [] t)) here"; outputs ~dev:true "an fn capturing by value, dev" "programs/fn-capture.flan" fn_capture_out; @@ -4013,6 +4026,8 @@ level "1" body's — so both are ways for a suspect to be a function's answer. *) refuses "an index read is a read like any other" "programs/fn-escape-at.flan" "may carry an environment"; + refuses "a match arm's binding is a binding" + "programs/fn-escape-match.flan" "may carry an environment"; refuses "a captured function value cannot be handed back" "programs/fn-escape-copy.flan" "a return would outlive the frame"; refuses "a handler-bind's value is a return too" diff --git a/test/test_session.ml b/test/test_session.ml index a5017ac3..d3fbfd30 100644 --- a/test/test_session.ml +++ b/test/test_session.ml @@ -1177,6 +1177,30 @@ let () = | exception Loc.Error { Loc.dmsg = m; _ } -> fail "an expression that instantiates a generic: %s" m); + (* ── The widening thunks, across a reorder ────────────────────── + A thunk is a function nobody wrote and nothing in the source names, so + the only way one can change is the program growing or losing a widening + — and reordering two calls is neither. But [compatible] compares by + name, so a thunk named for the order it was minted in means one + signature before the edit and another after, and the session answers a + body change with "Restart to change it" about a name the programmer + cannot find. The name spells the signature, so this reload is ordinary. + + Both directions of the pair are here — the same two calls, swapped — + because a name that is a counter is wrong for exactly one of them and + the test has to be the one that is wrong. *) + (let t, _ = Session.create ~file:"programs/fn-thunk-reload.flan" () in + match + Session.eval t + "(defn both [] i32 (println (use64 b1)) (println (use32 a1)) 0)" + with + | c -> + if not (List.mem "both" c.Session.fns) then + fail "reordering two widenings installed %s" + (String.concat " " c.Session.fns) + | exception Loc.Error { Loc.dmsg = m; _ } -> + fail "reordering two widenings was refused: %s" m); + (* ── A class whose slots changed ──────────────────────────────── The dev loop's half of CLHS 4.3.6. Three things have to be true of the session for the runtime's migration to ever be reached: a changed slot