From a8a7ebc3f60f60f39a86970f45844ac80f628f06 Mon Sep 17 00:00:00 2001 From: Joseph Ferano Date: Sat, 12 Sep 2026 22:01:30 +0700 Subject: [PATCH] The refusal block loses half its entries, and the four that stay say why The list at the foot of prelude.ml was one sentence -- every entry needed to produce bytes that did not exist in its input, and there was no allocator -- and that sentence has been false since Vec landed. Seven entries move up into the code, and string-from-bytes turns out to have been the `string` builtin all along: (string (as-slice v)) is the round trip, free precisely because the layouts are identical. What is left is refused for four different reasons and is written that way now: pad and center for nothing at all except that no caller has asked; format and sprintf for variadics of mixed type; map, filter, reduce and sort-by for function values; map-keys and map-values for a map iterator that does not exist in the runtime. NEXT.md's queued section is struck and carries the four findings, each with the change it wants named -- flan_map_next plus one builtin for the iterator; milestone 5's function values for the higher-order three; vec_new_elem taking a type expression rather than a bare name, which is what forces slices-new to exist; and an array literal with no way to say it is [f32], which is what forces every float in algorithms.flan to be cast. BUILT.md gets the section. --- BUILT.md | 127 +++++++++++++++++++++++++++++++++++++++++++++++++ NEXT.md | 81 ++++++++++++++++++++++++------- lib/prelude.ml | 59 +++++++++++++++-------- 3 files changed, 230 insertions(+), 37 deletions(-) diff --git a/BUILT.md b/BUILT.md index b29fc07..24acc60 100644 --- a/BUILT.md +++ b/BUILT.md @@ -2435,6 +2435,133 @@ reason `-linkall` is not optional. Say plainly what that coverage is not: nothin before this landed, so `macro-unless.flan` is a test written after the feature. The corpus written before it is `sand.flan` and `web/examples/control.flan`, and both compile unchanged. +## The prelude's second tier: the functions that return new storage + +Everything in the prelude before this was slice-based and allocation-free, and NEXT.md's diagnosis of why was exact: +there was nothing to allocate from when it was written. `Vec`, `Map`, an arena and `StorageExhausted` changed that, +and this is the tier that follows — 24 additions, of which the twelve that matter most **return new things** +instead of writing into a buffer the caller supplies. The rest fill in the slice family at the element types that +were missing, and one of them is a macro. + +Three rules hold across all of it, and they are stated once at the head of the section rather than repeated: + +1. **The result is owned and the caller frees it.** Nothing is released at scope exit — not at the end of a `let`, + not at the end of a function (`spec-memory.md`). A caller writes `(free v)`, or lets a `(free-all a)` take the + whole region. +2. **The allocator is the context's, and `with-allocator` is the override.** This is the one design decision the + spec did not settle by itself. `(vec-new)` and `(map-new)` take an optional trailing allocator because the + *checker* builds them and can vary their arity; a Flan `defn` cannot, so the choice was an allocator parameter on + every signature or none. None — `(with-allocator a (join parts sep))` is the override, the `Vec` records the + arena, and `free` and `clone` never need it named again. +3. **No `Result` anywhere.** Allocation failure signals `StorageExhausted` under `retry`, and no allocating + operation returns an error, so every signature says what it produces and nothing about how it might fail. + +### The builder is not a type + +Odin's `strings.Builder` wraps a `[dynamic]u8`. Here the `(Vec u8)` already **is** that and already has `push`, so +the struct would be a move-only wrapper whose only method is the one it wraps. What was actually missing is appending +a *run* of bytes, and `append!` is that. + +It takes a `(Ptr (Vec u8))` and not a `(Vec u8)`, and that is not style: a `Vec` parameter **moves**, so a by-value +builder would be consumed by its first append and refused on the second. + +`append-i64!` and `append-f64!` are the argument for the whole shape. NEXT.md's "Sharp edges" records that +`flan_i64_to_bytes` and its neighbours render into one `static char scratch[64]`, so two formatted numbers cannot be +held at once; these copy out of that buffer before returning, so the hazard ends at the call and a builder holds as +many numbers as it likes. `strings.flan` puts two integers and a float on one line, which is the case that could not +be written before. + +### `split` answers a `(Vec [u8])`, and the owning shape is unrepresentable + +The fields are slices *of the input*. That is not a performance choice — `(Vec (Vec u8))` is **refused outright** +(`programs/vec-of-vec.flan`, "copies and releases elements bytewise"), so there is no owning shape to have chosen +instead. It follows that the result dies with whatever the input pointed at, which is the same contract `trim` and +`split-next!` already have. + +The rule is `split-on-byte`'s, unchanged: n separators always yield n+1 fields, so an empty input yields one empty +field and a trailing separator yields a trailing empty one. That is Odin's allocating `strings.split` and not Odin's +`split_by_byte_iterator`, which disagree with each other on exactly that input. + +Constructing it needed a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`, because `check.ml`'s `vec_new_elem` +takes the element type as a single bare symbol and `[u8]` is not one — so a `(Vec [u8])` can only be made where the +*context* names the type, and a return type is a context while a `let` is not. Written down in NEXT.md as a compiler +gap rather than worked around silently. + +### `format-f64`, and the rounding rule it does not share with printf + +`f64->bytes` is `snprintf "%g"`: six significant digits, exponent notation of its own accord, no precision to pass +it. A frame time of 1/60 comes back `0.0166667` and a score past a million `1.23457e+06`. + +`format-f64` returns a `Vec`, so it inherits neither that nor the shared scratch buffer, and it renders the integer +part and the fraction through that buffer in strict sequence — the discipline `append-i64!` exists to make automatic. + +It rounds **half away from zero at the last digit kept**, which is `round-f32`'s rule and the rest of the prelude's. +printf rounds the *binary* value to nearest-even at the decimal digit, so `0.125` at two places is `0.13` here and +`0.12` there. Matching printf would mean pinning a particular libc's answer, and that answer is not the same on every +target anyway. + +Three lines in it are the ones a plausible version ships without, and each is a separate test case: + +- **The carry.** `0.999995` at five places scales to exactly `100000`, which is not a fraction — it is the next + integer. Without the carry it prints `0.100000`. +- **The zero padding.** The fraction of `1.005` at three places is `5`, and `5` is not `005`; without the pad it + prints `1.5`. +- **The sign.** It belongs to the number, not to its integer part: `-0.5` has an integer part of `0`, and + `i64->bytes` of `0` carries no sign. + +`-0.0` prints as `0.00`, because the sign test is `(< x 0.0)`, which `-0.0` fails. Past `9e18` the integer part does +not fit in an `i64` and there are no fractional bits left anyway, so it falls back to `%g` rather than approximating. + +### `clamp` is a macro, and `atan2`/`pow` are declares + +`clamp` is the second prelude `defmacro` after `unless`, and the reason is the prelude's own objection to wrapping +`(min hi (max lo x))` turned around rather than dropped. `min` and `max` are builtins at *every* numeric type and +there are no generics, so a clamp **function** is one copy per type — `clamp-i32`, `clamp-f32`, `clamp-i64`. A macro +is type-agnostic for free and emits nothing at all. `math2.flan` makes the same three-word call at `i32`, `i64`, `u8` +and `f32` to show it, and counts evaluations to show each argument appears once. + +`atan2-f32` and `pow-f32` inherit `sin-f32`/`cos-f32`'s caveat in full and not `sqrt-f32`'s: IEEE-754 requires +nothing of `atan2f` or `powf` either, so they are the third and fourth places in the prelude where native and wasm32 +may differ in the last bit. Every case in `math2.flan` is therefore a value exact in binary — a quadrant boundary, a +power of two, a perfect square — and the `-O0` run is the one that proves the symbols resolve, since at `-O2` LLVM +constant-folds a `powf` of two literals and leaves nothing to link. + +### What could not be built, and why it is not "no generics" + +Four things on NEXT.md's list did not land, and the interesting part is that the reason differs in each case. + +- **`Map` keys and values** need a **map iterator**, and there is none. `flan_map_len`, `_get`, `_put`, `_has`, + `_clone`, `_reserve`, `_free` is the runtime's entire map surface; nothing walks the open-addressed block. One + runtime function taking a cursor and one builtin in `check.ml` to emit the key and value sizes is the whole job, + and none of it is a generics question. +- **`map`, `filter`, `reduce` and a comparator sort** are blocked on **function values**, which is sharper than "no + generics" and matters because generics alone would not fix it. `Types.Fn` exists; `check.ml` refuses it with "a + function type is not implemented yet — milestone 5"; there is nothing in the language to pass. The concrete answer + is the one that shipped: `sort-f32!` and `sort-bytes!` are the second and third sorts in the language, and + `sum-i32`/`sum-f32` already are `reduce` with the `+` written in. +- **The prelude is never macro-expanded**, so a prelude function may not call a prelude macro. `Macro.program` runs + over the file being compiled; the prelude reaches the checker through `Check.program`'s prepend. The call resolves + to the macro's underlying `defn` and reports an arity error, which is why `format-f64` writes + `(min 9 (max 0 prec))`. +- **A returned `Vec` is a move and the dead set spans the function**, so an early `(return v)` on one branch kills + the binding at the foot of another. `replace-bytes` guards its empty needle with an `if` rather than a + `when`/`return` for that reason. + +### The refusal block is down from eight reasons to four + +The list at the foot of `prelude.ml` used to be one sentence — every entry needed to produce bytes that did not exist +in its input, and there was no allocator. `join`, `concat`, `split`, `to-lower`, `to-upper`, `repeat` and `replace` +have moved up into the code; `string-from-bytes` turned out to be the `string` builtin all along, and +`(string (as-slice v))` is the round trip, free precisely because the layouts are identical. + +What remains is refused for four different reasons, and is now written that way: `pad`/`center` for *nothing at all* +except that no caller has asked; `format`/`sprintf` for variadics of mixed type; `map`/`filter`/`reduce`/`sort-by` +for function values; `map-keys`/`map-values` for the missing iterator. + +Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`, each at +`-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch, +so it is what would catch one of these `Vec`s being used after the arena under it was released. + ## `defer` may be written in a `let` The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both diff --git a/NEXT.md b/NEXT.md index ca0bea9..b52474b 100644 --- a/NEXT.md +++ b/NEXT.md @@ -518,26 +518,60 @@ hand-written backend is needed at all. It is research first, not building. The batch below stays valid and none of it is blocked by the question. -## Queued: a second tier of the standard library, after macros +## ~~Queued: a second tier of the standard library, after macros~~ — **landed** -Blocked only on `lib/prelude.ml`, which the macro lane holds. Start it when that merges. +See [`BUILT.md`](BUILT.md), "The prelude's second tier". The diagnosis here was right and the prelude had 44 +allocation-free functions because there was nothing to allocate from; there are 24 more now, and the "Refused, by +name" block at the foot of `prelude.ml` is down from eight entries to four, each with a *different* reason rather +than the one shared sentence. -**The gap, stated plainly: the whole prelude predates the allocator.** All 44 functions are slice-based and -allocation-free, because when they were written there was nothing to allocate from. `Vec` and `Map` now exist, so a -second tier is possible — functions that *return new things* rather than writing into a buffer the caller supplies. +What landed: `append!`/`append-i64!`/`append-f64!` (the builder), `concat`, `join`, `split` returning a +`(Vec [u8])`, `repeat-bytes`, `replace-bytes`, `to-lower`, `to-upper`, `slices-new`; `format-f64` with a precision; +`atan2-f32` and `pow-f32`; `clamp` as a `defmacro`; and the slice family at two more element types — +`sort-f32!`, `reverse-f32!`, `swap-f32!`, `min-f32`, `max-f32`, `sum-f32`, `bytesbytes` directly — nothing was taken away — but a caller assembling a line of + text has a way not to meet it. + - **Writing through a string literal is undefined, and the two build modes disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into a `private unnamed_addr constant`. At `-O0` that is a store to read-only @@ -1417,6 +1457,15 @@ What follows is only the part that is still missing. round it could be compiled in after something else. It would fail with an unknown name rather than with a reason, which is worth fixing the day the prelude wants one. +- **A prelude *function* may not call a prelude macro either**, which is the neighbouring gap and was found by + walking into it. `Macro.program` runs over the file being compiled; the prelude arrives at the checker through + `Check.program`'s own prepend and is never handed to the expander at all. A `defmacro` is an ordinary `defn` taking + one `[Form]` by the time the checker sees it, so the call resolves to that and the report is "clamp takes 1 + argument, given 3" — pointing at the prelude, about a call the author wrote as a macro use. `format-f64` writes + `(min 9 (max 0 prec))` in place of `(clamp prec 0 9)` because of it. The fix is not obviously cheap: expanding the + prelude means building a macro module to compile the prelude that the macro module is built from, which is the same + bootstrap the `when`/`dotimes` item above describes. + - **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and not the desugaring. Only a macro that writes a macro wants one. diff --git a/lib/prelude.ml b/lib/prelude.ml index 48e6b51..760586f 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -1172,29 +1172,46 @@ let source = {flan| (append! (addr b) d)))))))) b)) -;; ── Refused, by name ────────────────────────────────────────────────── +;; ── Still refused, and what the reason is now ───────────────────────── ;; -;; Every one of these needs to produce bytes that did not exist in its input, -;; and there is no allocator, so each is absent rather than approximated. -;; None of them is hard to write once `(Vec u8)` and an allocator exist; all -;; of them are impossible to write honestly today. +;; This list used to be one sentence long — every entry needed to produce bytes +;; that did not exist in its input, and there was no allocator. That sentence +;; stopped being true when `Vec` landed, and most of the list has moved up into +;; the building section above: join, concat, split, to-lower, to-upper, repeat +;; and replace are all written now, and `string-from-bytes` turned out to be +;; the `string` builtin all along — (string (as-slice v)) is the round trip, +;; and the layouts being identical is exactly why it is free. ;; -;; join, concat build one buffer out of several inputs. -;; to-lower, to-upper a new string, per Odin's conversion.odin. The -;; byte-wise and folding-comparison forms above are -;; what is available without one. -;; split the *sequence* of fields is itself an allocation. -;; split-on-byte / split-next! above is the same -;; information with no sequence to own. -;; replace, repeat, pad same reason as join. -;; string-from-bytes a [u8] cannot become a `string` here even though -;; the layouts are identical; see the report. -;; format, sprintf Odin's fmt.aprintf family, all allocating. -;; Builder strings.Builder is (defstruct Builder [buf -;; (Vec u8)]), which spec-memory.md already makes -;; move-only by the rule that a struct containing a -;; Vec is move-only. It needs the Vec, not a spec -;; change. +;; What is left is refused for four *different* reasons, which is why they are +;; named separately rather than under one heading. +;; +;; pad, center Nothing. These are three lines each over repeat-bytes +;; and concat, and they are absent only because no +;; caller has asked. Write them when one does. +;; format, sprintf A format *string* — Odin's fmt.aprintf family. It +;; needs variadic arguments of mixed type, which is a +;; function-value and generics question, not an +;; allocation one. format-f64 above is the piece of it +;; that was actually wanted, and `print`/`println` are +;; already the structural walk over any one value. +;; map, filter, reduce Function values. See the head of the slice-algorithm +;; sort-by section: check.ml refuses a function type outright, +;; and there is nothing in the language to pass. +;; map-keys, map-values A Map iterator. `len` reaches a Map and `get`, +;; `put` and `has-key?` address one entry, but there is +;; no entry point in the runtime that walks the block — +;; flan_map_len, _get, _put, _has, _clone, _reserve and +;; _free is the whole surface. This is the one item on +;; NEXT.md's second-tier list that could not be built +;; here at all, and it wants one runtime function and +;; one builtin rather than anything from the language. +;; +;; Builder Not refused — declined. strings.Builder in Odin +;; wraps a [dynamic]u8; here the (Vec u8) *is* that and +;; already has push, so the struct would be a move-only +;; wrapper whose only method is the one it wraps. What +;; was missing was appending a run of bytes, and +;; `append!` above is that. ;; ── Files: embedding, slurp and barf ────────────────────────────────── ;; ;; One entry per file in an (embed-dir "...") — Odin's Load_Directory_File