JosephFerano/flan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

A cursor over the block, because nothing walked it

Browse Source
This commit is contained in:
Joseph Ferano 2026-09-12 22:24:05 +07:00
parent e24aee5120
commit 772d1d5b18
8 changed files with 281 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
BUILT.md
View File

@ -2733,6 +2733,51 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla
`-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch,
so it is what would catch one of these `Vec`s being used after the arena under it was released.
## `map-next!`, the one thing a Map could not do
`flan_map_len`, `_get`, `_put`, `_has`, `_clone`, `_reserve` and `_free` was the runtime's entire map surface, and
every one of them addresses a *single* entry by hashing it. Nothing walked the block, so a map's keys and its values
could not be read out at all — the only item on the second tier's list that was blocked on nothing but a missing
function.
`flan_map_next` is that function and `map-next!` is the builtin over it.
```
(let [cur (i64 0) k 0 v 0]
(while (map-next! m (addr cur) (addr k) (addr v))
...))
```
**The cursor is a slot index the caller owns, and there is no iterator struct** because there is nothing for one to
hold. A map has no tombstones — removal is deferred (`spec-memory.md`) — so a slot is either empty or occupied and the
position is the whole of the state. The cursor starts at 0, comes back one past the entry just answered, and is left
at `cap` by the call that answers false, so a spent cursor keeps answering false rather than wrapping.
**Three out-pointers and not a returned pair**, because there are no tuples. An `(Option K)` would answer half an
entry and make the value cost a second hash of the key just handed back. The `!` is the cursor: it is the argument
that is written through on the way out.
**It is the one map entry point that carries neither a hash nor an equality function.** Walking asks nothing about a
key. The two sizes are still there, because the runtime is type-erased and the block geometry is computed from them.
**The layout, restated, because it is the thing to get wrong here.** `data` is *one* allocation laid out
keys | values | hashes | scratch, each run cell-packed to a cache line — the arrangement the Valgrind lane described
while explaining why a probe overrun is not observable. A key is reached through `flan_cell_at` and never as
`ks + i * ksize`. The hashes are the exception `flan_map_clone` already relies on: an 8-byte element packs 8 to a
64-byte cell with nothing left over, so `g.hs[i]` is the right index and a flat one.
**Order is block order**, which is the hash's order and not the insertion's, and it changes when the map grows.
`programs/map-iter.flan` is therefore written entirely in sums, counts and lengths — every claim in it is order-free,
which is the contract rather than a weakness of the test. A caller that wants an order sorts what it collected. The
cases that would catch a real mistake: a string key with a struct value, whose two runs have different element sizes
and different packing and so would break if one geometry were used for both; and a 500-entry map, which is several
grows past the minimum and walks a block whose layout has nothing to do with how the entries went in.
**`map-keys` and `map-values` are still refused, and the reason changed.** The refusal block at the foot of
`prelude.ml` said "a Map iterator"; that is wrong now. What a prelude `defn` cannot write is
`(defn map-keys [m {K V}] (Vec K))` — it has to name its types and there is no `K`. That is generics. The loop is
three lines at the call site, where `K` is known, and that is where it stays.
## A prelude function may call a prelude macro, and why the fix was not ordering
The handoff said the prelude is never macro-expanded — `Macro.program` runs over the file being compiled and the

10
NEXT.md
View File

@ -552,11 +552,11 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla
**What could not be built, and why each one could not.** All four want a compiler or runtime change, and none of
them wants a language decision.
- **`Map` keys and values.** The only item on the list above that could not be built at all. `len` reaches a Map and
`get`/`put`/`has-key?` address one entry, but nothing walks the block: `flan_map_len`, `_get`, `_put`, `_has`,
`_clone`, `_reserve` and `_free` is the runtime's whole map surface, with no iterator among them. It wants one
runtime function — `flan_map_next` over the open-addressed block, taking a cursor — and one builtin in `check.ml`
to emit the key and value sizes at the call site. It is *not* a generics problem, and it is a small job.
- ~~**`Map` keys and values.**~~ **Iteration is built** — `flan_map_next` and the `map-next!` builtin, exactly the
shape this described. See [`BUILT.md`](BUILT.md), "`map-next!`, the one thing a Map could not do".
`map-keys`/`map-values` as *prelude functions* stay refused, and the reason is now generics rather than the
iterator: a `defn` has to name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`. The loop is three
lines at the call site, where `K` is known.
- **`map`, `filter`, `reduce`, and a sort taking a comparator.** Blocked on **function values**, not on generics,
which is the sharper statement than the one this list made. `Types.Fn` exists; `check.ml` refuses it with "a

42
lib/check.ml
View File

@ -2871,6 +2871,48 @@ and named_call ctx ~want loc name args =
[ mk loc oty (Tast.If (cond, some, none)) ])))
| _ -> assert false)
(* (map-next! m (addr cur) (addr k) (addr v)) -> bool, and the whole of map
iteration. Before it there was no way to read a map's keys or its values
at all: every other map operation addresses one entry by hashing it, and
nothing walked the block.
Three out-pointers rather than a returned pair, because there are no
tuples and a (Option K) would answer only half of an entry — the value
would then cost a second hash of the key just answered. The cursor is an
i64 the caller owns and the loop reads as one:
(let [cur 0 k 0 v 0]
(while (map-next! m (addr cur) (addr k) (addr v))
...))
It is *not* a generic (map-keys m): a Vec of them needs a signature naming
K, and a prelude defn cannot be written at every K. That one is generics,
not iteration, and it stays refused for that reason.
No hash and no equality pair go with it — walking asks nothing about a
key — so this is the one map entry point whose signature carries neither,
and the sizes are still needed because the runtime is type-erased. *)
| "map-next!" ->
arity loc name 4 args;
(match args with
| [ target; cur; k; v ] ->
let target = borrowed ctx target (fun () -> check ctx target) in
let kt, vt = map_kv loc "map-next!" target.Tast.ty in
let cur = check ctx ~want:(Types.Ptr (Types.Int Types.I64)) cur in
let k = check ctx ~want:(Types.Ptr kt) k in
let v = check ctx ~want:(Types.Ptr vt) v in
let found =
rt loc (Types.Int Types.I8) "flan_map_next"
[ target; cur; k; v; size_of loc kt; size_of loc vt; here loc ]
in
expect loc ~want
(mk loc Types.Bool
(Tast.Prim
(Tast.Ne,
[ found;
mk loc (Types.Int Types.I8) (Tast.Int (0L, Types.I8)) ])))
| _ -> assert false)
(* (has-key? m k). (get m k) answers the same question, but through an
Option the caller then has to match; this is the form a condition wants,
and it copies no value. *)

4
lib/emit.ml
View File

@ -2273,6 +2273,10 @@ declare i8 @flan_map_has(ptr, ptr, i64, i64, ptr, ptr, ptr, i64)
declare i8 @flan_map_reserve(ptr, i64, i64, i64, ptr, ptr, i64)
declare i8 @flan_map_clone(ptr, ptr, ptr, i64, i64, ptr, ptr, i64)
declare i64 @flan_map_len(ptr, ptr, i64)
; The cursor step. No hash and no equality pair: walking the block asks
; nothing about a key, which is why this is the one map entry point whose
; signature does not carry them.
declare i8 @flan_map_next(ptr, ptr, ptr, ptr, i64, i64, ptr, i64)
declare void @flan_map_free(ptr, i64, i64, ptr, i64)
; The pointer forms, whose signatures end with the transfer channel because a
; hash emitted for a struct key is an ordinary Flan function. Only ever taken

17
lib/prelude.ml
View File

@ -1203,14 +1203,15 @@ let source = {flan|
;; map, filter, reduce Function values. See the head of the slice-algorithm
;; sort-by section: check.ml refuses a function type outright,
;; and there is nothing in the language to pass.
;; map-keys, map-values A Map iterator. `len` reaches a Map and `get`,
;; `put` and `has-key?` address one entry, but there is
;; no entry point in the runtime that walks the block —
;; flan_map_len, _get, _put, _has, _clone, _reserve and
;; _free is the whole surface. This is the one item on
;; NEXT.md's second-tier list that could not be built
;; here at all, and it wants one runtime function and
;; one builtin rather than anything from the language.
;; map-keys, map-values Generics — and the reason changed, which is the
;; point of naming them separately. It used to be the
;; missing Map iterator; `map-next!` is that iterator
;; and walking a map is expressible now. What a defn
;; still cannot say is (defn map-keys [m {K V}] (Vec K)):
;; a prelude function has to name its types, and there
;; is no K. The loop is three lines at the call site,
;; where K is known, and that is where it stays until
;; there are generics.
;;
;; Builder Not refused — declined. strings.Builder in Odin
;; wraps a [dynamic]u8; here the (Vec u8) *is* that and

52
runtime/flan_rt.c
View File

@ -1695,6 +1695,58 @@ int64_t flan_map_len(flan_map *m, const uint8_t *loc, int64_t loclen) {
return m->len;
}
/* The cursor step, and the whole of iteration.
*
* Everything else in this file addresses *one* entry: get, put and has each
* hash a key and probe. Nothing walked the block, so a map's keys and its
* values could not be read out at all, and this is the one function that
* changes it.
*
* The cursor is a slot index the caller owns, and the contract is the one a
* slot index gives for free: it starts at 0, it is written back one past the
* entry just answered, and a 0 answer leaves it at [cap] so calling again is
* still 0. There is no iterator struct because there is nothing for one to
* hold — a map has no tombstones (removal is deferred), so no state beyond the
* position is needed to know where to resume.
*
* Invalidated by anything that moves the block, exactly as a Vec's slice is:
* a put that grows rehashes into a new block and every index before it means a
* different entry. The epoch check below catches a released arena and nothing
* catches a resize, which is the same bargain [as-slice] already makes.
*
* The layout is the one the geometry describes and is worth restating because
* it is the thing most likely to be got wrong here: [data] is *one* allocation
* laid out keys | values | hashes | scratch, each run cell-packed, so a key is
* reached through [flan_cell_at] and never by [ks + i * ksize]. The hashes are
* the exception the clone loop already relies on — an 8-byte element packs 8
* to a 64-byte cell with nothing left over, so a flat index is the right
* index. Order is block order, which is the hash's order and not the
* insertion's; two maps holding the same entries may walk them differently. */
int8_t flan_map_next(flan_map *m, int64_t *cursor, void *kout, void *vout,
int64_t ksize, int64_t vsize,
const uint8_t *loc, int64_t loclen) {
flan_map_geom g;
int64_t cap, i;
flan_map_check(m, loc, loclen);
if (!m->data || m->len == 0) return 0;
cap = flan_map_cap(m);
i = *cursor;
if (i < 0) i = 0;
if (i >= cap) { *cursor = cap; return 0; }
flan_map_geometry(m, ksize, vsize, cap, &g);
for (; i < cap; i++) {
if (g.hs[i] == 0) continue;
memcpy(kout, flan_cell_at(g.ks, ksize, g.kepc, g.kcell, g.kshift, i),
(size_t)ksize);
memcpy(vout, flan_cell_at(g.vs, vsize, g.vepc, g.vcell, g.vshift, i),
(size_t)vsize);
*cursor = i + 1;
return 1;
}
*cursor = cap;
return 0;
}
/* Room for [n] entries without reallocating, which means a block whose 75%
* threshold is at least n. */
int8_t flan_map_reserve(flan_map *m, int64_t n, int64_t ksize, int64_t vsize,

109
test/programs/map-iter.flan Normal file
View File

@ -0,0 +1,109 @@
;; Walking a map, which is what flan_map_next exists for. Every other map
;; operation addresses one entry by hashing it; this is the only thing that
;; reads the block in order.
;;
;; Block order is the hash's order and not the insertion's, so nothing here
;; may depend on which entry comes first: the checks are a sum, a count and a
;; membership test, all of them order-free. That is not a weakness of the test,
;; it is the contract — a caller that wants an order sorts what it collected.
(defn sum-and-count [] Unit
(let [m (map-new i32 i32)]
(put m 1 10)
(put m 2 20)
(put m 3 30)
(put m 4 40)
(let [cur (i64 0)
k 0
v 0
keys 0
vals 0
n 0]
(while (map-next! m (addr cur) (addr k) (addr v))
(set keys (+ keys k))
(set vals (+ vals v))
(set n (+ n 1)))
(print n) (print " ") (print keys) (print " ") (print vals) (println ""))
(free m)))
;; A map that never allocated has no block at all, and one that allocated and
;; holds nothing has a block of nothing but zeroed hashes. Both walk zero
;; times, and they are different code paths to get there.
(defn the-empty-cases [] Unit
(let [m (map-new i32 i32)
cur (i64 0)
k 0
v 0
n 0]
(while (map-next! m (addr cur) (addr k) (addr v))
(set n (+ n 1)))
(print "never allocated: ") (print n) (println "")
(reserve m 64)
(set cur (i64 0))
(while (map-next! m (addr cur) (addr k) (addr v))
(set n (+ n 1)))
(print "allocated and empty: ") (print n) (println "")
(free m)))
;; A cursor left past the end keeps answering false rather than wrapping, so a
;; second loop over a spent cursor is empty and not a repeat.
(defn a-spent-cursor [] Unit
(let [m (map-new i32 i32)]
(put m 5 50)
(put m 6 60)
(let [cur (i64 0) k 0 v 0 n 0]
(while (map-next! m (addr cur) (addr k) (addr v))
(set n (+ n 1)))
(while (map-next! m (addr cur) (addr k) (addr v))
(set n (+ n 1)))
(print "spent: ") (print n) (println ""))
(free m)))
;; A string key and a struct value: the key run and the value run have
;; different element sizes and different cell packing, so this is the case that
;; would catch the two runs being indexed with one geometry.
(defstruct Point [x i32 y i32])
(defn wider-entries [] Unit
(let [m (map-new string Point)]
(put m "a" (Point {.x 1 .y 2}))
(put m "bb" (Point {.x 3 .y 4}))
(put m "ccc" (Point {.x 5 .y 6}))
(let [cur (i64 0)
k ""
v (Point {})
chars 0
xs 0
ys 0]
(while (map-next! m (addr cur) (addr k) (addr v))
(set chars (+ chars (len k)))
(set xs (+ xs (.x v)))
(set ys (+ ys (.y v))))
(print chars) (print " ") (print xs) (print " ") (print ys) (println ""))
(free m)))
;; Growth past the 75% threshold rehashes into a new block, so this walks a map
;; whose layout is nothing like its insertion order and at a capacity several
;; doublings past the minimum.
(defn after-growth [] Unit
(let [m (map-new i64 i64)]
(dotimes [i 500]
(put m (i64 i) (* (i64 i) 2)))
(let [cur (i64 0)
k (i64 0)
v (i64 0)
n 0
doubled 0]
(while (map-next! m (addr cur) (addr k) (addr v))
(set n (+ n 1))
(when (= v (* k 2)) (set doubled (+ doubled 1))))
(print n) (print " ") (print doubled) (print " ") (print (len m)) (println ""))
(free m)))
(defn main [] i32
(sum-and-count)
(the-empty-cases)
(a-spent-cursor)
(wider-entries)
(after-growth)
0)

15
test/test_acceptance.ml
View File

@ -1763,6 +1763,21 @@ ERR@7 unexpected token: not the kind the caller was reading
body behind an indirection cell, so it is the build that would notice. *)
outputs ~dev:true "maps, dev" "programs/maps.flan" maps_out;
(* Iteration, which no map could do at all until flan_map_next. Every case
here is order-free on purpose — block order is the hash's order, not the
insertion's — so the numbers are sums, counts and lengths and never a
first entry. The string-keyed, struct-valued map is the one that would
catch the key and value runs being indexed with a single geometry, since
their element sizes and cell packing differ; and the 500-entry map is
several grows past the minimum, so it walks a block whose layout has
nothing to do with the order the entries went in. *)
let map_iter_out =
"4 10 100\nnever allocated: 0\nallocated and empty: 0\nspent: 2\n\
6 9 12\n500 500 500\n"
in
outputs "map iteration" "programs/map-iter.flan" map_iter_out;
outputs ~opt:"-O0" "map iteration, -O0" "programs/map-iter.flan" map_iter_out;
(* The allocation-failure rule is one rule over every allocating operation,
so it has to hold for map-new, put, reserve and clone as it does for the
Vec's four. A map is the harder case: its growth allocates a new block,