JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

The refusal block loses half its entries, and the four that stay say why

Browse Source 
The list at the foot of prelude.ml was one sentence -- every entry needed to
produce bytes that did not exist in its input, and there was no allocator --
and that sentence has been false since Vec landed. Seven entries move up into
the code, and string-from-bytes turns out to have been the `string` builtin
all along: (string (as-slice v)) is the round trip, free precisely because
the layouts are identical.

What is left is refused for four different reasons and is written that way
now: pad and center for nothing at all except that no caller has asked;
format and sprintf for variadics of mixed type; map, filter, reduce and
sort-by for function values; map-keys and map-values for a map iterator that
does not exist in the runtime.

NEXT.md's queued section is struck and carries the four findings, each with
the change it wants named -- flan_map_next plus one builtin for the iterator;
milestone 5's function values for the higher-order three; vec_new_elem taking
a type expression rather than a bare name, which is what forces slices-new to
exist; and an array literal with no way to say it is [f32], which is what
forces every float in algorithms.flan to be cast. BUILT.md gets the section.
This commit is contained in:
Joseph Ferano 2026-09-12 22:01:30 +07:00
parent 7ce6043c47
commit a8a7ebc3f6
3 changed files with 230 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

127
BUILT.md
View File

@ -2435,6 +2435,133 @@ reason `-linkall` is not optional. Say plainly what that coverage is not: nothin
before this landed, so `macro-unless.flan` is a test written after the feature. The corpus written before it is
`sand.flan` and `web/examples/control.flan`, and both compile unchanged.
## The prelude's second tier: the functions that return new storage
Everything in the prelude before this was slice-based and allocation-free, and NEXT.md's diagnosis of why was exact:
there was nothing to allocate from when it was written. `Vec`, `Map`, an arena and `StorageExhausted` changed that,
and this is the tier that follows — 24 additions, of which the twelve that matter most **return new things**
instead of writing into a buffer the caller supplies. The rest fill in the slice family at the element types that
were missing, and one of them is a macro.
Three rules hold across all of it, and they are stated once at the head of the section rather than repeated:
1. **The result is owned and the caller frees it.** Nothing is released at scope exit — not at the end of a `let`,
not at the end of a function (`spec-memory.md`). A caller writes `(free v)`, or lets a `(free-all a)` take the
whole region.
2. **The allocator is the context's, and `with-allocator` is the override.** This is the one design decision the
spec did not settle by itself. `(vec-new)` and `(map-new)` take an optional trailing allocator because the
*checker* builds them and can vary their arity; a Flan `defn` cannot, so the choice was an allocator parameter on
every signature or none. None — `(with-allocator a (join parts sep))` is the override, the `Vec` records the
arena, and `free` and `clone` never need it named again.
3. **No `Result` anywhere.** Allocation failure signals `StorageExhausted` under `retry`, and no allocating
operation returns an error, so every signature says what it produces and nothing about how it might fail.
### The builder is not a type
Odin's `strings.Builder` wraps a `[dynamic]u8`. Here the `(Vec u8)` already **is** that and already has `push`, so
the struct would be a move-only wrapper whose only method is the one it wraps. What was actually missing is appending
a *run* of bytes, and `append!` is that.
It takes a `(Ptr (Vec u8))` and not a `(Vec u8)`, and that is not style: a `Vec` parameter **moves**, so a by-value
builder would be consumed by its first append and refused on the second.
`append-i64!` and `append-f64!` are the argument for the whole shape. NEXT.md's "Sharp edges" records that
`flan_i64_to_bytes` and its neighbours render into one `static char scratch[64]`, so two formatted numbers cannot be
held at once; these copy out of that buffer before returning, so the hazard ends at the call and a builder holds as
many numbers as it likes. `strings.flan` puts two integers and a float on one line, which is the case that could not
be written before.
### `split` answers a `(Vec [u8])`, and the owning shape is unrepresentable
The fields are slices *of the input*. That is not a performance choice — `(Vec (Vec u8))` is **refused outright**
(`programs/vec-of-vec.flan`, "copies and releases elements bytewise"), so there is no owning shape to have chosen
instead. It follows that the result dies with whatever the input pointed at, which is the same contract `trim` and
`split-next!` already have.
The rule is `split-on-byte`'s, unchanged: n separators always yield n+1 fields, so an empty input yields one empty
field and a trailing separator yields a trailing empty one. That is Odin's allocating `strings.split` and not Odin's
`split_by_byte_iterator`, which disagree with each other on exactly that input.
Constructing it needed a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`, because `check.ml`'s `vec_new_elem`
takes the element type as a single bare symbol and `[u8]` is not one — so a `(Vec [u8])` can only be made where the
*context* names the type, and a return type is a context while a `let` is not. Written down in NEXT.md as a compiler
gap rather than worked around silently.
### `format-f64`, and the rounding rule it does not share with printf
`f64->bytes` is `snprintf "%g"`: six significant digits, exponent notation of its own accord, no precision to pass
it. A frame time of 1/60 comes back `0.0166667` and a score past a million `1.23457e+06`.
`format-f64` returns a `Vec`, so it inherits neither that nor the shared scratch buffer, and it renders the integer
part and the fraction through that buffer in strict sequence — the discipline `append-i64!` exists to make automatic.
It rounds **half away from zero at the last digit kept**, which is `round-f32`'s rule and the rest of the prelude's.
printf rounds the *binary* value to nearest-even at the decimal digit, so `0.125` at two places is `0.13` here and
`0.12` there. Matching printf would mean pinning a particular libc's answer, and that answer is not the same on every
target anyway.
Three lines in it are the ones a plausible version ships without, and each is a separate test case:
- **The carry.** `0.999995` at five places scales to exactly `100000`, which is not a fraction — it is the next
integer. Without the carry it prints `0.100000`.
- **The zero padding.** The fraction of `1.005` at three places is `5`, and `5` is not `005`; without the pad it
prints `1.5`.
- **The sign.** It belongs to the number, not to its integer part: `-0.5` has an integer part of `0`, and
`i64->bytes` of `0` carries no sign.
`-0.0` prints as `0.00`, because the sign test is `(< x 0.0)`, which `-0.0` fails. Past `9e18` the integer part does
not fit in an `i64` and there are no fractional bits left anyway, so it falls back to `%g` rather than approximating.
### `clamp` is a macro, and `atan2`/`pow` are declares
`clamp` is the second prelude `defmacro` after `unless`, and the reason is the prelude's own objection to wrapping
`(min hi (max lo x))` turned around rather than dropped. `min` and `max` are builtins at *every* numeric type and
there are no generics, so a clamp **function** is one copy per type — `clamp-i32`, `clamp-f32`, `clamp-i64`. A macro
is type-agnostic for free and emits nothing at all. `math2.flan` makes the same three-word call at `i32`, `i64`, `u8`
and `f32` to show it, and counts evaluations to show each argument appears once.
`atan2-f32` and `pow-f32` inherit `sin-f32`/`cos-f32`'s caveat in full and not `sqrt-f32`'s: IEEE-754 requires
nothing of `atan2f` or `powf` either, so they are the third and fourth places in the prelude where native and wasm32
may differ in the last bit. Every case in `math2.flan` is therefore a value exact in binary — a quadrant boundary, a
power of two, a perfect square — and the `-O0` run is the one that proves the symbols resolve, since at `-O2` LLVM
constant-folds a `powf` of two literals and leaves nothing to link.
### What could not be built, and why it is not "no generics"
Four things on NEXT.md's list did not land, and the interesting part is that the reason differs in each case.
- **`Map` keys and values** need a **map iterator**, and there is none. `flan_map_len`, `_get`, `_put`, `_has`,
`_clone`, `_reserve`, `_free` is the runtime's entire map surface; nothing walks the open-addressed block. One
runtime function taking a cursor and one builtin in `check.ml` to emit the key and value sizes is the whole job,
and none of it is a generics question.
- **`map`, `filter`, `reduce` and a comparator sort** are blocked on **function values**, which is sharper than "no
generics" and matters because generics alone would not fix it. `Types.Fn` exists; `check.ml` refuses it with "a
function type is not implemented yet — milestone 5"; there is nothing in the language to pass. The concrete answer
is the one that shipped: `sort-f32!` and `sort-bytes!` are the second and third sorts in the language, and
`sum-i32`/`sum-f32` already are `reduce` with the `+` written in.
- **The prelude is never macro-expanded**, so a prelude function may not call a prelude macro. `Macro.program` runs
over the file being compiled; the prelude reaches the checker through `Check.program`'s prepend. The call resolves
to the macro's underlying `defn` and reports an arity error, which is why `format-f64` writes
`(min 9 (max 0 prec))`.
- **A returned `Vec` is a move and the dead set spans the function**, so an early `(return v)` on one branch kills
the binding at the foot of another. `replace-bytes` guards its empty needle with an `if` rather than a
`when`/`return` for that reason.
### The refusal block is down from eight reasons to four
The list at the foot of `prelude.ml` used to be one sentence — every entry needed to produce bytes that did not exist
in its input, and there was no allocator. `join`, `concat`, `split`, `to-lower`, `to-upper`, `repeat` and `replace`
have moved up into the code; `string-from-bytes` turned out to be the `string` builtin all along, and
`(string (as-slice v))` is the round trip, free precisely because the layouts are identical.
What remains is refused for four different reasons, and is now written that way: `pad`/`center` for *nothing at all*
except that no caller has asked; `format`/`sprintf` for variadics of mixed type; `map`/`filter`/`reduce`/`sort-by`
for function values; `map-keys`/`map-values` for the missing iterator.
Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`, each at
`-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch,
so it is what would catch one of these `Vec`s being used after the arena under it was released.
## `defer` may be written in a `let`
The whole of this project's resource-cleanup answer, and NEXT.md records `drop` and a `with-cleanup` form as both

81
NEXT.md
View File

@ -518,26 +518,60 @@ hand-written backend is needed at all.
It is research first, not building. The batch below stays valid and none of it is blocked by the question.
## Queued: a second tier of the standard library, after macros
## ~~Queued: a second tier of the standard library, after macros~~ — **landed**
Blocked only on `lib/prelude.ml`, which the macro lane holds. Start it when that merges.
See [`BUILT.md`](BUILT.md), "The prelude's second tier". The diagnosis here was right and the prelude had 44
allocation-free functions because there was nothing to allocate from; there are 24 more now, and the "Refused, by
name" block at the foot of `prelude.ml` is down from eight entries to four, each with a *different* reason rather
than the one shared sentence.
**The gap, stated plainly: the whole prelude predates the allocator.** All 44 functions are slice-based and
allocation-free, because when they were written there was nothing to allocate from. `Vec` and `Map` now exist, so a
second tier is possible — functions that *return new things* rather than writing into a buffer the caller supplies.
What landed: `append!`/`append-i64!`/`append-f64!` (the builder), `concat`, `join`, `split` returning a
`(Vec [u8])`, `repeat-bytes`, `replace-bytes`, `to-lower`, `to-upper`, `slices-new`; `format-f64` with a precision;
`atan2-f32` and `pow-f32`; `clamp` as a `defmacro`; and the slice family at two more element types —
`sort-f32!`, `reverse-f32!`, `swap-f32!`, `min-f32`, `max-f32`, `sum-f32`, `bytes<?`, `swap-bytes!`, `sort-bytes!`.
Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.flan`, `programs/math2.flan`.
Wanted, in rough order of how often it will be missed:
`string-from-bytes`, refused in that block, turned out to already exist: `string` is a builtin and
`(string (as-slice v))` is the round trip.
- **String building.** A `Vec u8` builder, `join`, and a `split` that returns a `Vec` instead of the
`split-next!`/`split-on-byte` iterator dance the current one requires.
- **`Vec` algorithms** — `map`, `filter`, `reduce`, and a `sort` that is not integers-only. `sort-i32!` is the only
sort there is.
- **`Map` helpers** — keys, values.
- **Maths gaps**: `atan2`, `pow`, `clamp`. `sin-f32`/`cos-f32` exist with the caveat that IEEE-754 does not make them
correctly rounded, so native and wasm32 may differ bit for bit; anything added here inherits that and should say so.
- **Number formatting with a precision.** Note the sharp edge that constrains this: `flan_i64_to_bytes` and friends
share one `static char scratch[64]`, so two formatted numbers cannot be held at once. A `Vec`-returning formatter
would not have that problem, which is an argument for building it.
**What could not be built, and why each one could not.** All four want a compiler or runtime change, and none of
them wants a language decision.
- **`Map` keys and values.** The only item on the list above that could not be built at all. `len` reaches a Map and
`get`/`put`/`has-key?` address one entry, but nothing walks the block: `flan_map_len`, `_get`, `_put`, `_has`,
`_clone`, `_reserve` and `_free` is the runtime's whole map surface, with no iterator among them. It wants one
runtime function — `flan_map_next` over the open-addressed block, taking a cursor — and one builtin in `check.ml`
to emit the key and value sizes at the call site. It is *not* a generics problem, and it is a small job.
- **`map`, `filter`, `reduce`, and a sort taking a comparator.** Blocked on **function values**, not on generics,
which is the sharper statement than the one this list made. `Types.Fn` exists; `check.ml` refuses it with "a
function type is not implemented yet — milestone 5"; and there is nothing else in the language to pass. Generics on
top of that is what would make them one copy rather than one per element type, but without function values there is
nothing to be generic *over*. `sort-f32!` and `sort-bytes!` are the concrete answer in the meantime, and `sum-i32`
and `sum-f32` already are `reduce` with the `+` written in.
- **`(vec-new [u8])` is refused**, so a `(Vec [u8])` can only be made where the *context* names the type.
`check.ml`'s `vec_new_elem` accepts a single bare symbol naming a type and nothing else, and a `let` has no type
annotation to say it the other way round — so `split` needs a one-line `(defn slices-new [] (Vec [u8]) (vec-new))`
standing in as the place where the type is said. The fix is to let `vec_new_elem` take a type *expression* rather
than a name, which is the same parser that already reads `[u8]` in a parameter list.
- **An array literal cannot say it is `[f32]`.** A float literal defaults to `f64`, an array literal has no context,
and a `let` has no annotation, so `[3.5 -1.0]` is an `[f64]` and every element in `programs/algorithms.flan` is
written `(f32 3.5)`. Same shape of gap as the one above and probably the same fix.
Two smaller findings, both written down beside the code that ran into them:
- **The prelude is never macro-expanded.** `macro.ml`'s pass runs over the file being compiled; the prelude reaches
the checker through `Check.program`'s own prepend and never goes through the expander. So a prelude *function*
calling a prelude *macro* resolves the macro's underlying `defn` — the one taking a `[Form]` — and reports an arity
error. `format-f64` writes `(min 9 (max 0 prec))` where it wanted `clamp`. This is next to, and not the same as,
"a prelude macro may not call a macro" below.
- **A returned `Vec` is a move, and the dead set spans the function**, so an early `(return v)` on one branch kills
the binding for the `v` at the foot of another. `replace-bytes` guards its empty-needle case with an `if` rather
than a `when`/`return` for that reason. Probably correct as it stands — the analysis is not path-sensitive and
making it so is a real piece of work — but it is a shape that reads as though it should compile.
Already present and easy to miss: an **EDN parser**, at `vendor/edn/edn.flan`.
@ -1327,6 +1361,12 @@ sanitized sweep (`@sanitize`) is under the same watchdog but has never been obse
sequences itself strictly for this reason. `rl/draw-text` is safe because the shim's `flan_shim_cstr` copies out of
ptr+len before the call.
**The prelude now has the shape that does not have this problem**, and it is the reason that shape exists.
`append-i64!` and `append-f64!` copy out of the scratch buffer into a `(Vec u8)` before returning, so a builder
holds as many rendered numbers as it likes, and `format-f64` answers a `Vec` rather than a view. The hazard is
unchanged for anyone calling `i64->bytes` directly — nothing was taken away — but a caller assembling a line of
text has a way not to meet it.
- **Writing through a string literal is undefined, and the two build modes
disagree about how.** `(let [s (bytes "Hi")] (set (at s 0) \h))` stores into
a `private unnamed_addr constant`. At `-O0` that is a store to read-only
@ -1417,6 +1457,15 @@ What follows is only the part that is still missing.
round it could be compiled in after something else. It would fail with an unknown name rather than with a reason,
which is worth fixing the day the prelude wants one.
- **A prelude *function* may not call a prelude macro either**, which is the neighbouring gap and was found by
walking into it. `Macro.program` runs over the file being compiled; the prelude arrives at the checker through
`Check.program`'s own prepend and is never handed to the expander at all. A `defmacro` is an ordinary `defn` taking
one `[Form]` by the time the checker sees it, so the call resolves to that and the report is "clamp takes 1
argument, given 3" — pointing at the prelude, about a call the author wrote as a macro use. `format-f64` writes
`(min 9 (max 0 prec))` in place of `(clamp prec 0 9)` because of it. The fix is not obviously cheap: expanding the
prelude means building a macro module to compile the prelude that the macro module is built from, which is the same
bootstrap the `when`/`dotimes` item above describes.
- **A quasiquote inside a quasiquote is refused.** Nothing counts nesting levels — not the reader, deliberately, and
not the desugaring. Only a macro that writes a macro wants one.

59
lib/prelude.ml
View File

@ -1172,29 +1172,46 @@ let source = {flan|
(append! (addr b) d))))))))
b))
;; ── Refused, by name ──────────────────────────────────────────────────
;; ── Still refused, and what the reason is now ─────────────────────────
;;
;; Every one of these needs to produce bytes that did not exist in its input,
;; and there is no allocator, so each is absent rather than approximated.
;; None of them is hard to write once `(Vec u8)` and an allocator exist; all
;; of them are impossible to write honestly today.
;; This list used to be one sentence long — every entry needed to produce bytes
;; that did not exist in its input, and there was no allocator. That sentence
;; stopped being true when `Vec` landed, and most of the list has moved up into
;; the building section above: join, concat, split, to-lower, to-upper, repeat
;; and replace are all written now, and `string-from-bytes` turned out to be
;; the `string` builtin all along — (string (as-slice v)) is the round trip,
;; and the layouts being identical is exactly why it is free.
;;
;; join, concat build one buffer out of several inputs.
;; to-lower, to-upper a new string, per Odin's conversion.odin. The
;; byte-wise and folding-comparison forms above are
;; what is available without one.
;; split the *sequence* of fields is itself an allocation.
;; split-on-byte / split-next! above is the same
;; information with no sequence to own.
;; replace, repeat, pad same reason as join.
;; string-from-bytes a [u8] cannot become a `string` here even though
;; the layouts are identical; see the report.
;; format, sprintf Odin's fmt.aprintf family, all allocating.
;; Builder strings.Builder is (defstruct Builder [buf
;; (Vec u8)]), which spec-memory.md already makes
;; move-only by the rule that a struct containing a
;; Vec is move-only. It needs the Vec, not a spec
;; change.
;; What is left is refused for four *different* reasons, which is why they are
;; named separately rather than under one heading.
;;
;; pad, center Nothing. These are three lines each over repeat-bytes
;; and concat, and they are absent only because no
;; caller has asked. Write them when one does.
;; format, sprintf A format *string* — Odin's fmt.aprintf family. It
;; needs variadic arguments of mixed type, which is a
;; function-value and generics question, not an
;; allocation one. format-f64 above is the piece of it
;; that was actually wanted, and `print`/`println` are
;; already the structural walk over any one value.
;; map, filter, reduce Function values. See the head of the slice-algorithm
;; sort-by section: check.ml refuses a function type outright,
;; and there is nothing in the language to pass.
;; map-keys, map-values A Map iterator. `len` reaches a Map and `get`,
;; `put` and `has-key?` address one entry, but there is
;; no entry point in the runtime that walks the block —
;; flan_map_len, _get, _put, _has, _clone, _reserve and
;; _free is the whole surface. This is the one item on
;; NEXT.md's second-tier list that could not be built
;; here at all, and it wants one runtime function and
;; one builtin rather than anything from the language.
;;
;; Builder Not refused — declined. strings.Builder in Odin
;; wraps a [dynamic]u8; here the (Vec u8) *is* that and
;; already has push, so the struct would be a move-only
;; wrapper whose only method is the one it wraps. What
;; was missing was appending a run of bytes, and
;; `append!` above is that.
;; ── Files: embedding, slurp and barf ──────────────────────────────────
;;
;; One entry per file in an (embed-dir "...") — Odin's Load_Directory_File