diff --git a/README.md b/README.md index 94b0892a..368b8a7e 100644 --- a/README.md +++ b/README.md @@ -145,8 +145,7 @@ executable in a trench coat. `flan dev` is the one command that takes it unasked: a dev session is what it was written for, it halves the `C-c C-c` round trip, and nothing it builds outlives the session. `--llvm` is how to ask for the other one there — for a -program this backend refuses by name, for `--debug`, and for the inspector, -which walks a shadow stack it does not push. Every other command here is LLVM +program this backend refuses by name, and for `--debug`. Every other command here is LLVM by default and stays that way; `emacs/MANUAL.md` lists what the dev backend does not do. diff --git a/TODO.org b/TODO.org index 76bef6bc..a081e498 100644 --- a/TODO.org +++ b/TODO.org @@ -957,6 +957,11 @@ ignore order, writable access has to alias the real storage. Flexible field orde waits for classes deliberately, because a class owns its layout and a =Vector2= should not pay for identity and metadata. Not implemented. +** TODO An error in a called generic's body is reported twice +=(defn g [x $t] u64 (nosuch x))= called once from =main= prints "unknown +function nosuch" twice at the same place and counts 2 errors — once from the +abstract pass and once from the instantiation. + * Backends ** DONE The x86 backend tracks LLVM at -O0 @@ -1194,9 +1199,12 @@ for needs a slice that can carry the Vec's identity, and a slice is pointer and length — so either slices grow a word in a dev build or the trap does not exist. Today it does not. -** TODO The allocator's budget is not in the spec -=alloc-budget= and =set-alloc-budget= exist and the spec does not mention them. -Worth folding in or replacing with a growable arena. +** DONE The allocator's budget is not in the spec +CLOSED: [2026-09-25] +spec-memory.md has a Budget subsection under Allocators, as built: a ceiling on +live bytes, 0 for none, that a =retry= handler raises. The failure bullet says +"raises the allocator's budget" where it said "grows the arena", since no arena +grows. A growable arena is not ruled out; nothing here asks for one. ** TODO The Vec header is not the size the spec fixes Five words in every build rather than the spec's four, and for a stated reason: a @@ -1516,8 +1524,8 @@ before shadowed locals are even honest, and that buys locals in lldb rather than the break loop. ** DONE The x86 backend pushes shadow-stack frames -It does, in every dev build. The prose in the daemon that rewrites the agent's -reply to say otherwise is now false and should go with whoever next touches it. +It does, in every dev build. The daemon no longer rewrites the agent's reply to +say otherwise. ** DONE A restart is not a transaction If a frame mutates a global and then signals, taking a =retry= re-runs the @@ -1945,41 +1953,61 @@ Printing the stable kind at the end of the first line, the understood-then- conflicted clause order as a writing rule, and non-cascading multiple errors. Each needs a decision rather than work. -** TODO docs/BUILT.md still describes (Handle T) and the pool as built -It reads as shipped fact for a type the checker has no constructor for and the -runtime has no code for, including an enumeration primitive in the present tense. -The pool was removed on 2026-09-18. +** DONE docs/BUILT.md still describes (Handle T) and the pool as built +CLOSED: [2026-09-25] +The section is a short past-tense record: that neither exists, why they went, +the decisions a library version would face, and that classes landed without the +enumeration the pool was built to give. A stale runtime comment naming +=resolve= went with it. -** TODO The daemon still tells an editor the x86 backend pushes no frames -It does push them, in every dev build. The reply is rewritten from a claim that -stopped being true. +** DONE The daemon still tells an editor the x86 backend pushes no frames +CLOSED: [2026-09-25] +The rewrite in =Dev.ask= is gone, so an x86 session passes the agent's reply +through as an LLVM one does. The Emacs manual and the README no longer list the +inspector as something the x86 backend cannot do. -** TODO web/index.html still claims there is no implicit widening -Two places. Left alone deliberately — the website has its own rewrite lane. +** DONE web/index.html still claims there is no implicit widening +CLOSED: [2026-09-25] +Both sentences say what is true: a conversion that cannot change the number is +implicit, any other is written. The rest of the page is left for its rewrite. -** TODO plan.org's Types section lists a predicate that no longer exists -It still describes =copyable?= and "a type variable is move-only by default", -both of which the ownership repeal removed. Left alone deliberately by the -generics lane as the repeal lane's sentence to retire. +** DONE plan.org's Types section lists a predicate that no longer exists +CLOSED: [2026-09-25] +The five predicates are the checker's five, with =integer?= in place of +=copyable?=, and the section no longer names =(Handle $t)= or =pool-new=. -** TODO plan.org cites the wrong mechanism for jank's relinking bug -The real cause was a process-teardown race; jank calls through vars, which are -already indirection cells. We are safe from the repro because we compile out of -process, not because of cells. A normative document citing the wrong mechanism -protects the wrong invariant. +** DONE plan.org's Data model section still describes move-only containers +CLOSED: [2026-09-25] +The Data model section says assignment copies a container's header and the +copies alias one buffer. The memory tiers and the classes section name the pool +and generational handles as a library over a =Vec=, and the classes gate that +named =Handle= is replaced by what classes are as built. The milestone record +of what was frozen is left as history. -** TODO docs/SPIKE-GENERICS.md lists landed work as remaining -Three items are under "Mechanical" as remaining work and have landed. A dated -report going stale at a live claim. +** DONE plan.org cites the wrong mechanism for jank's relinking bug +CLOSED: [2026-09-25] +plan.org now says the crash was a teardown race (the maintainer's diagnosis on +issue #947), that jank already calls through vars, citing the clone, and that +Flan avoids the repro by compiling out of process. -** TODO Two citations in spec-memory.md do not land where they say -One is off by a line in a reference clone; the other names an arm that is not the -one the claim is about. The claim behind the second is true and cited in the wrong -place, and a third companion citation in the same section is stale too. +** DONE docs/SPIKE-GENERICS.md lists landed work as remaining +CLOSED: [2026-09-25] +The row is marked as landed, with the test that covers it, rather than removed; +the report stays a dated record. -** TODO The sand hash is quoted as prose in five places besides its assertion -The assertion carries the current number; the prose copies do not all. Whoever -re-takes the number has that list. +** DONE Two citations in spec-memory.md do not land where they say +CLOSED: [2026-09-25] +=Map_Cell_Info= is cited at core.odin:351. The =defer= bullet cites the +=defer_ok= field and the =Ast.Defer= arm, and says what the checker accepts: a +=defer= in a top-level =let= is legal, so =(defer (free v))= for a =let=-bound +=v= is expressible and the spec no longer says otherwise. + +** DONE The sand hash is quoted as prose in five places besides its assertion +CLOSED: [2026-09-25] +Four prose copies were left, not five. BUILT.md points at =sand_out= in +test_acceptance.ml instead of quoting a number. The page's two are checked by +quotes.sh against a run, so they stay. The handoff report is dated and keeps the +number it recorded. ** DONE sand.flan is two programs CLOSED: [2026-09-13] diff --git a/docs/BUILT.md b/docs/BUILT.md index 230071a0..de16a5f7 100644 --- a/docs/BUILT.md +++ b/docs/BUILT.md @@ -3316,164 +3316,42 @@ there and matching it from a program. `dev.ml`'s inspector still says "union val frame's locals, and `shim.ml`'s "a Flan union has no C layout" is now inaccurate as prose though the refusal it guards is still right: a union has a C layout and still may not cross to C by value, because the shim flattens aggregates. -## `(Handle T)` and the pool, which is what a stale reference answers with +## `(Handle T)` and the pool, which were built and removed -**Removed 2026-09-18**, in the second round of the repeal: two containers are -enough, the corpus's only pools were its own fixtures, and a slab behind -generational handles is a library over a `Vec` when a program wants one — -which is where Odin keeps it. The `gen` word on `Vec` and `Map` went the same -day (maintained, read by nothing; the header is `ptr len cap allocator -epoch` now), and so did the move-only concept itself — everything copies, and -`copyable?` left the predicate list. spec-memory.md, "The repeal", carries -all three amendments. The section below is kept as the record of what was -built. +Neither exists. The checker has no constructor for `(Handle T)` or `(Pool T)`, and the runtime has no code for either. +Both were built and then **removed on 2026-09-18**, in the second round of the repeal: two containers are enough, the +corpus's only pools were its own fixtures, and a slab behind generational handles is a library over a `Vec` when a +program wants one — which is where Odin keeps it. The `gen` word on `Vec` and `Map` went the same day, and so did the +move-only concept itself. `spec-memory.md`, "The repeal", carries the amendments. -A handle is a reference to something that can die, which reports that it died rather than silently resolving to -whatever reused its slot. `check.ml` refused `(Handle T)` by name as milestone 6; this is what it stood for, and the -pool came with it because the pool is what makes the report possible. +A handle was a reference to something that can die, which reported that it died rather than silently resolving to +whatever reused its slot. The decisions the built version made are the ones a library version faces, so they are kept +here: -The problem is concrete and is not about memory safety. Entities live in a pool; something holds a reference to one — -a projectile chasing it, the UI showing its health. The entity dies, the slot is reused, and a raw index now names a -different entity. Nothing crashes. The projectile chases the wrong thing, at full speed, for the rest of the game. +- **A handle was one i64**: slot index in the low 32 bits, the slot's generation in the high 32. It owned nothing; the +pool was the single owner. +- **Live was odd.** A slot's generation was bumped on every allocation and every release, so a zeroed handle +(generation 0) resolved to nothing and a liveness test needed no second array. +- **The generation wrapped by retiring the slot.** A release from `0xFFFFFFFF` left the slot dead forever rather than +letting a future handle collide with an old one. One leaked slot is a bounded price for making the collision +unrepresentable. +- **`resolve` answered `(Option (Ptr T))`**, because a matched struct is a copy and writing to the pooled entity in +place is what a pool is for. A pointer from `resolve` was invalidated by an `insert` that grew the pool, as a slice is +by a `push`. +- **`length` was the slot high-water and `live` the count**, so a loop bounded by `length` over `(pool-handle p i)` +visited every live entry. That pair was the whole of iteration. +- **A slot was released through the pool**, `(release p h)`, answering `bool` so a double release was an answer +rather than a trap. +- **A stale handle was an answer and a released pool was a trap**, through the epoch check a `Vec` gets. +- **Growth was transactional**: both blocks were allocated and copied before the old pair was released, because +`retry` re-attempts the same request. -It was built for two reasons, both already recorded. On its own terms, for entities referred to across frames. And -because **it is the real gate on managed classes**: plan.org's rule is that nothing starts on `defclass` until -ordinary `struct`, `Handle` and reload semantics work, and `Handle` was the only one of the three missing. That is not -an incidental precondition — `migrate-instances` has to *enumerate* live instances, and a pool behind generational -handles gives that by construction where a world arena and an owned region do not. plan.org presents the three storage -strategies as a free choice and they are not. +### Classes did not need it -`PORTING.md` found no customer for handles in the author's real game today, so this is deliberately the smallest -correct thing rather than a rich API: eight names, no iteration protocol, no cursor type, no `clone`. - -### A handle is one i64, and the halves are 32 and 32 - -Slot index in the low 32 bits, that slot's generation counter in the high 32. One machine word, so it copies, zeroes -and compares like the integer it is, and it **owns nothing** — the pool is the single owner. That is what lets a -handle sit in a struct field and in a global where a `Vec` may not, and it is the reason the ownership rules needed no -new case: `Types.is_move_only` says yes to `Pool` and no to `Handle`, and the three existing refusals (a struct field, -a union case, a global) picked the pool up unchanged with the messages they already had. - -The index is 32 bits because a `Vec`'s index is an `i32` here and widening indices is one change across every -container, not a pool question. - -### Live is odd, and two things fall out of it - -A slot's generation starts at 0 and is bumped on every allocation *and* on every release, so an odd generation means -live and an even one means dead. Both consequences are load-bearing: - -- **A zeroed handle resolves to nothing.** Generation 0 is even, so ZII gives a `(Handle T)` field the right meaning -for free instead of pointing it at slot 0. `handles.flan` prints one: ``, and resolving it answers `None`. -- **Iteration needs no second array and no spare bit.** Asking whether a slot is live is asking whether its generation -is odd. - -### The generation wraps by retiring the slot - -32 bits is 2^31 allocate/release pairs on one slot — every frame at 60fps for a year and a bit. "Rare" is not an -answer when the failure it produces is the silent wrong one this type exists to prevent, so a release from generation -`0xFFFFFFFF` bumps to 0 and does **not** put the slot back on the free list. The slot is retired: dead forever, its -payload leaked, and no future handle can collide with an old one. Leaking is defined behaviour here, and one slot is a -bounded price for making the collision unrepresentable rather than unlikely. - -### `resolve` answers `(Option (Ptr T))`, and the spec settled that, not this lane - -The task asked whether a lookup should answer `(Option T)`, matching `(get m k)`. It answers `(Option (Ptr T))`, and -`spec-memory.md` already writes it out — its worked example under "Mutating something you matched" is annotated -`(Option (Ptr Enemy))` — for the reason given one line above it: *pattern bindings bind values, so a matched struct is -a copy*. A copy cannot be written back, and writing to the pooled entity in place is what a pool is for. `(Option T)` -would answer a question nobody asked. - -The `Option` half is `get`'s shape and for `get`'s reason: absence is an answer, not a failure. A trap would be wrong -here — the entity dying is the *expected* case, not a bug. - -**The hole, said plainly.** A `(Ptr T)` from `resolve` is invalidated by any `insert` that grows the pool, exactly as -a slice is invalidated by a `push`. The handle survives that and the pointer does not. This is `spec-memory.md`'s -explicit Zig/Odin borrowing contract one level down, and it is worth naming rather than implying, because it -reintroduces the silent-wrong-answer mode the handle just removed for anyone who keeps a resolved pointer across an -insert. Chunked never-moving storage is the fix and it costs code; taking the contract is the smaller correct thing, -given `slice` already established it. - -### `length` is the slot high-water and `live` is the count, in that direction - -`(length p)` is how many slots have ever been handed out. `(live p)` is how many of them are live now. It had to be that -way round: `0..(length p)` are the indices `(pool-handle p i)` accepts, so a loop bounded by `length` visits every live -entry. Bounded by the live count instead, it would silently skip entries the moment anything had been released — -which is exactly the quiet wrong answer the whole type exists to remove. - -`(pool-handle p i)` answers `(Option (Handle T))`: the handle of slot `i`, or `None` if that slot is dead. That plus -`length` is the whole of iteration. An index outside `0..(length p)` **traps**, exactly as `(at v i)` traps: an index is an -index here, and answering `None` for one would hide a bug rather than a death. - -### A slot is released through the pool, and that is not a third release point - -`free` consumes its argument as a move, and a handle is a copyable number that owns nothing — consuming one copy would -say nothing about the others. So `(free h)` is refused by name and the release operation is on the owner: -`(release p h)`. `spec-memory.md`'s two release points are untouched: `(free p)` is release point 1 applied to the -pool, and a `free-all` of the region takes the pool with everything else. `release` recycles a slot inside storage the -pool still owns, which is not a release of storage at all. - -It answers `bool` rather than `()`: true if this call released it, false if the handle was already gone. The -generational scheme makes a double release **detectable**, and that is worth handing to the caller — this is the one -place in the language where freeing something twice is an answer instead of a refusal. - -### Growth is transactional, because `retry` re-attempts the same call - -`StorageExhausted`'s restart re-attempts the *same* request, so a failed grow has to leave the pool byte for byte as -it was — including a `cap` that still agrees with the real block sizes, since the next attempt passes `cap` as the -allocator's `old_size`. A pool grows two blocks together (payloads and slot headers), so resizing the first in place -and then failing on the second would leave `cap` describing neither. So the runtime allocates both, copies, and only -then releases the old pair: nothing is mutated after the last thing that can fail. An allocator without `can-free` -leaks the first block when the second fails, which is the defined outcome and not a new one — the request failed -because the region is exhausted, and the region is about to be released whole or its ceiling raised. - -### Two failures, kept apart - -A stale handle is an **answer**: `resolve` says `None` and the program carries on. A pool whose allocator was released -**traps**, through the same epoch check a `Vec` gets — the slot array went with the storage and there is nothing left -to ask. `test/programs/pool-stale-region.flan` is that case, and keeping the two apart is the same rule that keeps a -`Vec`'s generation word and its epoch word apart: they answer different questions and must not be conflated. - -### Two amendments to a frozen spec - -Both are places where `spec-memory.md` describes a handle doing something that cannot answer "gone", which is the one -thing the type exists to do. **This amends it: both are deferred, not built.** - -**1. `.field` and `at` do not auto-deref a handle.** The Places table says `x` may be a struct, a `(Ptr S)` or a -`(Handle S)`, and that the two forms auto-deref exactly one pointer *or handle* level. They auto-deref one pointer -level and nothing else. A `(set (.hp h) ...)` through a handle has two possible meanings when the entity is dead — trap, -or do nothing — and both are worse than the third option, which is the spec's own worked example: resolve first, match, -and the compiler makes you handle the `None`. The spec contradicts itself here and the example is the half that is -right. - -**2. `deref` is not overloaded on `(Handle a)`.** The Generics section says "`deref` yields a value; `resolve` yields a -pointer. Both are overloaded on `(Ptr a)` and `(Handle a)`." `deref` is `(Ptr a)` only. Same reason: `deref` returns a -value and has nowhere to put "gone". - -### What this does not have, and one of the gaps is not a pool question - -- **No `clone`.** Refused by name. A copied pool would carry the same slot generations, so one handle would resolve in -both copies and name two different things — the exact confusion the type removes. A program that wants a second world -builds one and inserts into it, and the new handles say they are new. -- **No pool of an owning element.** `(Pool (Vec i32))` is refused where `(Vec (Vec i32))` is refused and for the same -reason: the type-erased runtime copies and releases slots bytewise. Recursive teardown arrives with `drop`. -- **A handle is not a map key.** For the reason a `Ptr` is not: hashing an identity is a different operation from -hashing what it names, and a stale handle hashes the same as it always did while naming nothing. -- **Handles compare with `=` and not with `<`.** `Types` grew a second predicate, `is_equatable`, beside -`is_comparable`. Two handles are equal exactly when they name the same slot at the same generation, so a stale handle -is never equal to the live one that replaced it — that is worth one integer compare. Ordering them would order a slot -index, which is a free-list artefact and means nothing. -- **A pool passed to a helper is consumed**, because a pool is move-only exactly as a `Vec` is and there is no -borrowing parameter in the language. `test/programs/handles.flan` is one long function for that reason, and it does -not work around it. This is a pre-existing gap and not a pool question: the same sentence is true of every `Vec` in -the tree. - -### What classes still need - -The enumeration primitive is the piece migration was blocked on, and it exists now. What is left is `defclass` itself -and its runtime shape metadata; `migrate-instances`, which is a walk over `(length p)` and `(pool-handle p i)`; generic -functions and method dispatch, whose expensive half is already built and tested (a generic function is an indirection -cell whose body is a dispatch table, and a reload extends the table); and the rule that `Enemy@1` stays resolvable for -as long as any instance holds it — the same rule as "nothing is ever `dlclose`d". +The pool was built as the gate on managed classes, on the reading that `migrate-instances` has to enumerate live +instances. Classes landed without it: a `defclass` instance is a dyn map with its class in the object header, and a +redefined class migrates each instance lazily at its next touch, so nothing is enumerated (TODO.org, "A redefined +defclass migrates its instances lazily"). ## Macros: the compiler dlopens the program @@ -4839,7 +4717,7 @@ and changing one's type is a silent mismatch against storage the host already la Deferred until after the dev loop: 6. ~~**wasm32.**~~ **Done, with one glued joint.** `flan build --target=wasm32-wasi` produces a module, and -`test/programs/sand-headless.flan` prints `15595743031174623232` under it — the same hash as native, byte for byte, at +`test/programs/sand-headless.flan` prints the same hash under it as native, byte for byte (the number is `sand_out` in `test/test_acceptance.ml`), at `-O2` and at `-O0`. That is the milestone: the RNG is ours rather than libc's precisely so that number can be compared across targets, and it compares equal. `values.flan` and `machine.flan` run there too, which is where a 32-bit pointer would have shown. The acceptance table runs all four, and skips them by *probing* — it builds the smallest program and diff --git a/docs/SPIKE-GENERICS.md b/docs/SPIKE-GENERICS.md index c3c04da7..1533da17 100644 --- a/docs/SPIKE-GENERICS.md +++ b/docs/SPIKE-GENERICS.md @@ -228,7 +228,7 @@ with where predicates", in buckets: | | | |---|---| | **Done in the spike** | one or more type variables in a signature; binding through `[t]`, `(Ptr t)`, `(Option t)`, `(Fn [t] t)` and nesting; return types mentioning a variable; left-to-right binding with substitution into later parameters so an `fn` literal gets its types; the Odin-keyed instantiation cache; generic calling generic, transitively; `(vec-new t)`; the abstract refusal pass for `+`, `-`, `*`, `/`, `%`, `=`, `!=`, `<`, `<=`, `>`, `>=`, the bitwise operators and the shifts; instantiations as ordinary `Tast.fn`s with dev cells and `Reach` edges for free; a depth cap so runaway instantiation refuses instead of hanging | -| **Mechanical** | the remaining builtins that take a *type name* as an argument — `pool-new`, `map-new`, `zeroed`, `uninit`, the casts — each of which reaches `type_named`/`resolve_name` by its own path, exactly as `vec-new` did (one line there fixed `vec-new`; the others are one line each); `hash` and the map key-pair path, which must refuse a `Var` rather than assume; the `fns` expansion in `session.ml` described above | +| **Mechanical** (all three have since landed, and `test/programs/generics.flan` exercises the first two; `pool-new` was removed with the pool) | the remaining builtins that take a *type name* as an argument — `pool-new`, `map-new`, `zeroed`, `uninit`, the casts — each of which reaches `type_named`/`resolve_name` by its own path, exactly as `vec-new` did (one line there fixed `vec-new`; the others are one line each); `hash` and the map key-pair path, which must refuse a `Var` rather than assume; the `fns` expansion in `session.ml` described above | | **Bulky, not hard** | error messages that say *where* an instantiation came from — Odin's "in instantiation of" note. Today a refusal inside an instantiated body points at the generic's source with no indication which call site asked for that type, and with three or four instantiations that is the difference between a readable refusal and a puzzle. It is a context stack in `ctx` and a `Loc.note` per frame, and it touches every `fail` under an instantiation | | **Fiddly** | move-only and ownership. `Types.is_move_only (Var _)` is false, but the same variable at `(Vec i32)` is move-only — so the abstract pass **cannot decide ownership at all**, and the dead-set analysis is only sound per instantiation. Today that means a generic body that moves its parameter type-checks abstractly and is caught, if at all, at one instantiation and not another. The rule has to be stated: either ownership is checked only per copy (and the abstract pass skips it, so a generic may be accepted and its instantiation refused), or type variables carry a move-only constraint, which is a constraint system and plan.org says not yet. One smaller thing in the same bucket, found and left alone: the abstract pass over a generic body that calls *another* generic at a concrete type generates that copy and keeps it, so plain `flan emit` can carry a body no call site asked for. It is a valid instantiation and `Reach.link` drops it, so `flan build` and `flan run` are unaffected — but the abstract pass is meant to leave nothing behind and this is the one thing it does | | **No plan** | (1) **Unbounded instantiation.** `(defn grow [x $t] () (grow [x x]))` asks for a copy at `[2 t]`, which asks for one at `[2 [2 t]]`, forever. Before the cap it did not fail, it *hung* — and since `Session.eval` runs this same code, the thing that hangs is `C-c C-c`, with the dev daemon wedged behind it and no error to show. That is the project's stated priority hanging on three lines of ordinary-looking Flan, so the spike stops it: a depth counter in `env`, refusing past 32 and naming the type it had reached (`spike/generics/runaway.flan`). **The number is arbitrary and the designed refusal — one that names the chain of instantiations rather than the depth it gave up at — is still open.** **Odin has no cap of its own**, so there is no implementation to copy. (2) **Generic structs and containers.** `Types.Named` is a bare string with no parameters, so `(defstruct Pair [a $t b $t])` cannot be spelled at all — a parameterised named type is a change to `Types.t` and therefore to every backend, `Render`, DWARF and the layout calculator. (3) **`println` over a type variable.** plan.org's one compiler-provided exception; the abstract pass rejects it (`no printer for t`), see question 6. (4) **Generics across packages.** `Load` flattens imports into one namespace before checking, so it happens to work here, but a package boundary that is ever a real compilation-unit boundary would need the generic's *body* to cross it — the thing separate compilation cannot do and the reason C++ puts templates in headers | diff --git a/emacs/MANUAL.md b/emacs/MANUAL.md index 0ab2ecb5..e08994ed 100644 --- a/emacs/MANUAL.md +++ b/emacs/MANUAL.md @@ -97,13 +97,8 @@ the default, and it is the reason `C-c C-c` is fast: about 28ms at the socket against about 63ms through LLVM, because `as` does in 8ms what `llc` does in 45ms. On a project you are iterating in, that is the difference you feel. -It is not the whole compiler. Three things a session built by it cannot do: +It is not the whole compiler. Two things a session built by it cannot do: -- **No backtrace, no locals, no globals.** The inspector — `C-c C-b`, and - everything the break loop shows you about *where* a stopped program is — walks - a shadow stack of frames, and this backend does not push them. A program built - by it still stops on an unhandled error, still takes a restart, still answers - `C-x C-e` at the break. It just cannot tell you where it stopped. - **No `C-u C-c C-a`.** That view shows the LLVM IR a body was built from, and there is none; the listing this backend produced is assembly. Plain `C-c C-a` disassembles the object and works exactly as before. @@ -128,7 +123,7 @@ command line after the file, and it is how anything that is not the file or the socket reaches `flan dev` from Emacs: ```elisp -(setq flan-daemon-args '("--llvm")) ; the inspector, the IR view, every form +(setq flan-daemon-args '("--llvm")) ; the IR view, every form (setq flan-daemon-args '("--debug")) ; breakpoints, through flan-dape ``` @@ -142,8 +137,8 @@ honest shape of the thing — which backend your sessions use is a property of the project you are working on, not of the keystroke that started this one. **If you were relying on the old default:** every `flan dev` before this one -was an LLVM session, so a workflow built around `C-c C-b` or `C-u C-c C-a` will -find them refused now. One line in your init puts it back. +was an LLVM session, so a workflow built around `C-u C-c C-a` will find it +refused now. One line in your init puts it back. --- diff --git a/lib/dev.ml b/lib/dev.ml index 693575a2..0ae23018 100644 --- a/lib/dev.ml +++ b/lib/dev.ml @@ -295,38 +295,7 @@ let result t = thread on the frame that erred and waits. The agent is where that shows, and the session is the only thing holding it — so an editor asks here or not at all. *) -(* One sentence the agent cannot write, rewritten in the one place every verb - passes through. - - [vendor/agent/flan_agent.c] answers a backtrace, a locals or a globals - request from a program with no shadow-stack frames with "this program was - not built with --dev". That was true of exactly one thing when it was - written -- a release build, which has no frames because it has no dev - machinery at all -- and it stopped being true when [lib/x86.ml] became the - default for [flan dev]. An x86 dev host has its cells, its globals and its - registry; what it has not got is the frame push, so it stops on an error and - then cannot say where. The agent has no way to tell the two apart: it sees - an empty chain either way, and it is inside the program, which knows nothing - about the backend that compiled it. - - The session does know, so it is the one that corrects the sentence. Rewriting - the reply rather than teaching the agent a new environment variable keeps the - claim where the fact is -- and a message that names [--llvm] is the whole of - what the reader needs, where "not built with --dev" sends them to look for a - flag they did not leave off. *) -let mentions hay needle = - let n = String.length hay and m = String.length needle in - let rec go i = i + m <= n && (String.sub hay i m = needle || go (i + 1)) in - m = 0 || go 0 - -let ask t verb = - let text = request t verb in - if t.session.Session.x86 && mentions text "was not built with --dev" then - "err the x86 dev backend pushes no shadow-stack frames, so a stopped \ - program built by it cannot say where it is -- no backtrace, no locals and \ - no globals. It is the default for flan dev because it is about twice as \ - fast; restart the daemon with flan dev --llvm to inspect frames.\n" - else text +let ask t verb = request t verb type state = | Running diff --git a/plan.org b/plan.org index 73d2be6d..e8b45efd 100644 --- a/plan.org +++ b/plan.org @@ -5,7 +5,7 @@ Two documents are normative and are settled ahead of implementation. Anything in this plan that contradicts them is out of date. - [[file:spec-memory.md][spec-memory.md]] — ownership, the four container types, - copies and moves, assignable places, generics without type classes, function + copies, assignable places, generics without type classes, function values. - [[file:spec-conditions.md][spec-conditions.md]] — the six hard cases of conditions/restarts: what ~signal~ returns, no-handler behaviour, restart @@ -56,14 +56,15 @@ Allocators all the way down; ~malloc~ hidden behind them. | Tier | Strategy | Cost | |-----------+---------------------------------+----------------| | Frame | arena, bulk reset each frame | free | -| Entities | pool + generational handles | free | +| Entities | a pool over a ~Vec~, in a library | free | | Subsystem | region, freed wholesale | free | | Dev/REPL | leaks by design, reset on reload| dev only | - Allocator is part of the calling convention, so a refcounted allocator can be added later without a language change. -- Generational handles instead of pointers for cross-references: a stale - reference is detectable, not undefined behaviour. +- Generational handles instead of pointers for cross-references, written as a + library over a ~Vec~ rather than provided by the language: a stale reference + is detectable, not undefined behaviour. - Symbols and code live in a permanent arena that only grows. ** Why no persistent collections @@ -93,8 +94,8 @@ world. |-------------+-----------------+------------------+------| | ~[n T]~ | inline, n items | copies | no | | ~[T]~ | ptr+len | copies the view | no | - | ~(Vec T)~ | ptr+len+cap | *moves* | yes | - | ~(Map K V)~ | open addressing | *moves* | yes | + | ~(Vec T)~ | ptr+len+cap | copies the header | yes | + | ~(Map K V)~ | open addressing | copies the header | yes | ~Vec~ and ~Map~ are monomorphic on element type and record their allocator. Not a Lua-style array/hash hybrid — that is what makes Lua's layout and performance unpredictable. @@ -105,17 +106,15 @@ world. returns ~(Option V)~; ~put~ is the ~()~-returning upsert. See spec-memory.md for the deferred move-aware operations. - Operations: ~get~, ~put~, ~remove~, ~push~, ~pop~, ~at~, ~length~, ~update~. - Copying is explicit: ~(clone m)~, and owning containers move rather than copy on - assignment. No ~!~ convention — nothing is immutable, so it + An independent copy is explicit: ~(clone m)~. Assignment copies a container's + header, and the two headers alias one buffer. No ~!~ convention — nothing is immutable, so it would carry no information. No ~assoc~; it only existed as the copy-returning form. - ~const~ qualifier on references and slices: compile-time contract that a callee will not mutate. Zero runtime cost. -- Value structs copy on assignment — but only *value* structs. Ownership is - structural: a struct is a value type iff every field is, so one ~Vec~ field - makes it move-only. This is what keeps "copies on assignment" from meaning a - shallow copy that aliases owned storage. Deep copies are always explicit: - ~(clone x)~. Value structs are the snapshot / undo / replay story; they need no - separate type. +- Structs copy on assignment. A struct with a ~Vec~ field copies the header, so + the two copies alias one buffer, as in Odin. Deep copies are always explicit: + ~(clone x)~. Structs without owning fields are the snapshot / undo / replay + story; they need no separate type. - Literals live in read-only memory. - *A global's initialiser may be computed.* A value the linker can write goes into the image and costs nothing to start; anything else is stored at @@ -156,8 +155,8 @@ identity, extensibility, and live schema changes: Classes require a managed allocation strategy and runtime class/shape metadata, but *not necessarily a tracing GC*. The initial likely choices are a world or -session arena, pool allocation behind generational ~(Handle T)~ values, or an -explicitly owned region. A small tracing GC confined to class instances remains +session arena, a library pool behind generational handles, or an explicitly +owned region. A small tracing GC confined to class instances remains an option if cyclic graphs prove burdensome; it never changes ~struct~ layout or the C ABI. @@ -187,8 +186,11 @@ surprising lazy mutation on field access: #+end_src The precise class syntax, inheritance, storage strategy, and migration API are -not frozen. Do not add classes until ordinary ~struct~, ~Handle~, and reload -semantics are working. +not frozen. What is built is narrower than this section: a ~defclass~ instance +is a dyn map with its class in the object header, and a redefined class +migrates each instance lazily at its next touch, so nothing enumerates live +instances (TODO.org, "defclass is a named dyn map with a shape tag" and "A +redefined defclass migrates its instances lazily"). Immutability also serves the optimiser: a value known never to be mutated can be copied into registers and stack-allocated freely. Mutability is what forces heap @@ -218,7 +220,7 @@ and on a managed ~class~ instance. An ordinary ~struct~ never carries one. names; the guess was silently wrong twice, so the slot is mandatory. - Every type notation reads as exactly one data item: ~[f32]~, ~[4 f32]~, ~(Vec f32)~, ~(Map string i32)~, ~(Ptr World)~, ~(Fn [f32] bool)~, - ~(Option $t)~, ~(Handle $t)~. The map spelling was ~{string i32}~ once and is + ~(Option $t)~. The map spelling was ~{string i32}~ once and is not any more: braces in type position are refused by name. The ~where~ clause below is a brace form at the head of a body, so keeping the brace type would have put ~(defn f [...] {string i32} {:where ...} body)~ in the language — two @@ -229,7 +231,7 @@ and on a managed ~class~ instance. An ordinary ~struct~ never carries one. HKTs). A type variable is written ~$t~ wherever a *type* goes — parameter, return type, or nested as ~[$t]~ or ~(Vec $t)~ — and bare ~t~ wherever a type's *name* is an argument in expression position: ~(vec-new t)~, ~(map-new t i32)~, - ~(pool-new t)~, and the cast ~(t x)~. This is what makes + and the cast ~(t x)~. This is what makes ~map-in-place~/~filter~/~reduce~ and the monomorphic containers work; it collapsed the prelude's per-type families into one function each. A generic body is checked *abstractly*, with nothing substituted, so ~=~, ~<~, @@ -241,23 +243,16 @@ and on a managed ~class~ instance. An ordinary ~struct~ never carries one. predicates, which is Odin's (~core/slice/slice.odin:289~, ~where intrinsics.type_is_ordered(T)~). It is written as a Clojure-style map at the head of the body — ~{:where (ordered? $t)}~, or a vector for more than one, - ~{:where [(copyable? $t) (copyable? $u)]}~ — on the precedent of Clojure's + ~{:where [(ordered? $t) (hashable? $u)]}~ — on the precedent of Clojure's ~{:pre ... :post ...}~, and because a bare ~{}~ in expression position is already refused so nothing else it could be. ~sort~ declares ~ordered?~ of its variable, the abstract pass then allows ~<~ in the body, and each instantiation checks the concrete type satisfies the predicate and refuses the call site if it does not. There are *five* predicates — ~ordered?~, ~equal?~, ~hashable?~, - ~numeric?~, ~copyable?~ — against Odin's forty-one, and they entail one another - in one direction, so one clause usually does: ~numeric?~ gives ~ordered?~, - ~ordered?~ gives ~equal?~, and any of the four gives ~copyable?~. - ~copyable?~ has no Odin counterpart, because Odin has no move semantics and a - ~$T~ there never has to answer the question. *A type variable is move-only by - default* and ~copyable?~ is the opt-out: whether a variable is move-only is not - decidable abstractly — ~i32~ at one instantiation, ~(Vec i32)~ at the next — so - the checker takes the stricter rule, which can only refuse a program that would - have been fine and never admit one that double-frees. The prior art is Rust's - ~T: Copy~, differing in that the compiler answers the question rather than a - user implementing a trait. + ~numeric?~, ~integer?~ — against Odin's forty-one, and they entail one another + in one direction, so one clause usually does: ~integer?~ gives ~numeric?~, + ~numeric?~ gives ~ordered?~, and ~ordered?~ gives ~equal?~. ~integer?~ exists + because ~numeric?~ admits floats. ~hashable?~ is what lets a variable *key a map*: without it the type ~(Map $t i32)~ is refused where it is written, and with it the refusal moves to the call site that names an unhashable key. @@ -695,8 +690,13 @@ values safely retain the old version. The session immediately warns at each tracked old caller site, and recompiling that caller either updates it or gives a normal type error. This preserves live running code without hiding stale calls. -This is the fix for jank issue #947 (segfault redefining a running loop's -function plus its callee — their JIT relinks and unloads under a running thread). +jank issue #947 (a segfault after redefining a running loop's function and its +callee) is not what cells protect against. jank already calls through vars, +which are indirection cells (~compiler+runtime/src/cpp/jank/codegen/cpp_processor.cpp:565~ +derefs the var at the call). The crash was a teardown race: the REPL's piped +input reached EOF and jank shut LLVM down while another thread was still +JIT-compiling. Flan is safe from that repro because it compiles out of process, +in the daemon, not because of cells. ** What redefinition cannot do Patch a mid-execution frame and continue at the same PC — its register diff --git a/runtime/flan_dev.c b/runtime/flan_dev.c index 2fc3bfe3..fbedac8a 100644 --- a/runtime/flan_dev.c +++ b/runtime/flan_dev.c @@ -1116,8 +1116,8 @@ void *flan_dev_frame_slot(const void *frame, int32_t i) { * What a note records is a *block*, not a value: base, extent, and the size of * one element. So a lookup is containment rather than equality, and that is * not an optimisation — every pointer a program can hold into heap storage is - * interior. (at v i) is v->ptr + i*size and (resolve p h) is an item in the - * middle of a pool's array; neither is ever a base address. A table that + * interior. (at v i) is v->ptr + i*size, which is a base address only at + * i = 0. A table that * answered only exact hits would answer nothing anybody can ask it. * * Dead entries are kept, which is the second thing this buys: an address that diff --git a/runtime/flan_rt.c b/runtime/flan_rt.c index 00a9ddef..f295fa44 100644 --- a/runtime/flan_rt.c +++ b/runtime/flan_rt.c @@ -1185,7 +1185,7 @@ static void *flan_heap_proc(flan_allocator *a, int32_t mode, void *p, * caller passes old_size for exactly this reason, and it is the one * number a wrong answer here would read off the end of. */ void *q; - if (flan_over_budget(a, size - old_size)) return NULL; + if (size > old_size && flan_over_budget(a, size - old_size)) return NULL; q = flan_heap_proc(a, FLAN_ALLOC_ALLOC, NULL, 0, size, align); if (!q) return NULL; if (p && old_size > 0) @@ -1331,6 +1331,9 @@ static void *flan_arena_proc(flan_allocator *a, int32_t mode, void *p, * without copying, which is the common shape. */ if (p && (uint8_t *)p + old_size == ar->base + ar->offset) { int64_t end = (int64_t)((uint8_t *)p - ar->base) + size; + /* The budget is checked here as on every other path; a block grown in + * place is still more live bytes. */ + if (size > old_size && flan_over_budget(a, size - old_size)) return NULL; if (end > ar->cap || end < 0) return NULL; ar->offset = end; if (end > ar->peak) ar->peak = end; diff --git a/spec-memory.md b/spec-memory.md index 19e9fcaf..1220f8f3 100644 --- a/spec-memory.md +++ b/spec-memory.md @@ -450,6 +450,27 @@ An allocator declares which operations it implements. Odin's arena answers equivalent is a **capability set** on the allocator value, readable at run time. The one that is load-bearing below is `can-free`. +### The budget + +Every allocator also carries a ceiling on the bytes it has live, read and set at +run time: + +| Operation | Meaning | +|---------------------------|------------------------------------------------| +| `(alloc-budget a)` | the ceiling, an `i64`; 0 means there is none | +| `(set-alloc-budget a n)` | sets it; a negative `n` sets 0 | + +A request that would take the live total over the ceiling fails as an exhausted +allocator does, and signals `StorageExhausted` (see "Allocation failure"). An +arena is bounded by its backing buffer as well, and whichever limit is lower +applies. Setting a ceiling below what is already live releases nothing; it +refuses the next request. + +The budget is what a `retry` handler raises. Releasing the region a container +lives in invalidates the container, so for a fixed store the handler that makes +the same request succeed is the one that raises the ceiling. It is also how a +program exhausts an allocator on purpose, as `test/programs/exhausted.flan` does. + ### When storage is released There are exactly two release points, and neither of them is a scope. @@ -477,13 +498,14 @@ region it names is released, if ever, by an explicit `free-all` somewhere else. This is deliberate, and it is the point on which the two obvious precedents were rejected: -- **Odin's `defer delete`** cannot be written here. `defer` is function-scoped - (`check.ml:505` refuses it in a `let`, a loop or a branch) and, because `let` - is a block, a top-level `defer` is checked in a scope containing only the - parameters and globals (`check.ml:1670`). `(defer (free v))` for a `let`-bound - `v` is **not expressible today**. It becomes expressible with either - block-scoped `defer` or a sequential top-of-body binder; until one of those - exists, no idiom in this spec may depend on it. +- **Odin's `defer delete`** can be written only where its scope is the whole + function. `defer` is function-scoped: it is accepted at the top level of a + function body and in a `let` that is itself at the top level, to any depth, + and refused in a branch or a loop body (`check.ml:495`, the `defer_ok` field, + and the `Ast.Defer` arm at `check.ml:3661`). So `(defer (free v))` for a + `let`-bound `v` works when that `let` is at the top level of the body, and + runs at function exit rather than at the end of the `let`. There is no + block-scoped `defer`, and no idiom in this spec may depend on one. - **Carp's scope-end frees** are a whole-program linear analysis that inserts a teardown call at every binding's last use (`Memory.hs`, and `Info.hs`'s `Deleter`). Carp could not reconcile that with an arena and therefore has no @@ -655,7 +677,7 @@ not alternatives. Odin arranges it exactly this way: `elem_align` is threaded through every type-erased dynamic-array entry point (`base/runtime/dynamic_array_internal.odin` — `__dynamic_array_reserve`, `__dynamic_array_resize`, `__dynamic_array_append`), -and `align_of_type` sits in `Map_Cell_Info` (`base/runtime/core.odin:350`). The +and `align_of_type` sits in `Map_Cell_Info` (`base/runtime/core.odin:351`). The monomorphised wrapper is the only place the concrete type is known, so it is the only place that can produce the number. @@ -748,8 +770,8 @@ and says nothing is the outcome this rule exists to make impossible. break loop, where a working allocator is known. - Unhandled, `error` enters the dev break loop or aborts in release (spec-conditions.md §2). It is never a no-op; `signal` is not used here. -- A handler that frees something, releases a scratch region, or grows the arena - and then invokes `retry` re-attempts the same request. A handler that wants a +- A handler that frees something, releases a scratch region, or raises the + allocator's budget and then invokes `retry` re-attempts the same request. A handler that wants a *different* allocator needs a restart taking an argument, which does not exist yet; until it does, such a handler rebinds the context allocator and retries. - Because an allocating operation can transfer, every caller of one checks the diff --git a/test/programs/exhausted.flan b/test/programs/exhausted.flan index 206c99bc..0a1a1e2d 100644 --- a/test/programs/exhausted.flan +++ b/test/programs/exhausted.flan @@ -115,6 +115,24 @@ (println "INSERTIONSORT"))) ; INSERTIONSORT (println failures) ; 1 — failed once, retried once + ;; And an arena, whose budget is checked when a Vec grows its block in + ;; place as well as when it allocates a new one. A Vec that is the only thing + ;; pushing into an arena always grows in place, so without that check the + ;; ceiling would never be met. + (set tight (arena-new 65536)) + (set-alloc-budget tight 64) + (set failures 0) + (handler-bind + [(StorageExhausted [c] + (set failures (+ failures 1)) + (set-alloc-budget tight (* 2 (alloc-budget tight))) + (invoke-restart 'retry))] + (let [v (vec-new i32 tight)] + (dotimes [i 1000] (push v i)) + (println (length v)) ; 1000 + (println (at v 999)))) ; 999 + (println (> failures 0)) ; true + ;; And the restart is not once-per-program: it is established at each ;; allocation, so a later one offers it again. (set-alloc-budget tight 0) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index 53273af4..33a39f9d 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -1663,7 +1663,7 @@ let () = flow an optimiser would otherwise launder. *) let exhausted_out = "64\n0\n126\ntrue\ntrue\n4\ntrue\n0\n8\n7\ntrue\n\ - 13\n73\n84\n90\nINSERTIONSORT\n1\n" + 13\n73\n84\n90\nINSERTIONSORT\n1\n1000\n999\ntrue\n" in outputs "storage exhausted, retried" "programs/exhausted.flan" exhausted_out; outputs ~opt:"-O0" "storage exhausted, retried, -O0" "programs/exhausted.flan" diff --git a/web/examples/numbers.flan b/web/examples/numbers.flan index 0fa0a5fb..9cbe216f 100644 --- a/web/examples/numbers.flan +++ b/web/examples/numbers.flan @@ -1,6 +1,6 @@ (defn main [] i32 (let [n 40 ; i32, inferred - big (i64 n) ; every widening is written + big (i64 n) ; a cast is a call named for the type x 1.5] ; f64 (print (+ big 2)) (println "") (print (* x 2.5)) (println "") diff --git a/web/index.html b/web/index.html index ea157cfe..aac03f46 100644 --- a/web/index.html +++ b/web/index.html @@ -539,12 +539,15 @@ notation reads as exactly one data item.

-

There is no implicit widening. Every operand of an arithmetic or -comparison form has one type, and every conversion is written as a cast:

+

A conversion that cannot change the number is implicit; any other is +written. An i32 goes where an i64 or an f64 +is wanted, and an f32 where an f64 is. An i64 into an +i32, or an i32 into an f32, is an error until it is +written as a cast:

(defn main [] i32
   (let [n 40                       ; i32, inferred
-        big (i64 n)                ; every widening is written
+        big (i64 n)                ; a cast is a call named for the type
         x 1.5]                     ; f64
     (print (+ big 2)) (println "")
     (print (* x 2.5)) (println "")
@@ -1367,11 +1370,11 @@ quoted or it could not be told from the punctuation around it.

 since the walk is unrolled at compile time, rather than putting ten thousand printing
 sites in the module.

 
-
That the walk takes the argument's own type matters more here than it would in a
-language that widens implicitly. Nothing widens implicitly in this one, so a printer
-that named a type would need a cast written at every call — and a u64
-above 263 put through a signed one comes out negative. print
-takes the value as it is and prints the number it holds.

+
That the walk takes the argument's own type matters for the types that do not
+widen into one another. A printer that named one type, say i64, would take
+an i32 as it is but need a cast for a u64 — and a
+u64 above 263 put through that cast comes out negative.
+print takes the value as it is and prints the number it holds.

 
 
The prelude