diff --git a/BUILT.md b/BUILT.md index 0e8dea6..eae583e 100644 --- a/BUILT.md +++ b/BUILT.md @@ -2733,6 +2733,51 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla `-O2` and `-O0`, and `strings.flan` also in a dev build — the one that checks a container's recorded allocator epoch, so it is what would catch one of these `Vec`s being used after the arena under it was released. +## `map-next!`, the one thing a Map could not do + +`flan_map_len`, `_get`, `_put`, `_has`, `_clone`, `_reserve` and `_free` was the runtime's entire map surface, and +every one of them addresses a *single* entry by hashing it. Nothing walked the block, so a map's keys and its values +could not be read out at all — the only item on the second tier's list that was blocked on nothing but a missing +function. + +`flan_map_next` is that function and `map-next!` is the builtin over it. + +``` +(let [cur (i64 0) k 0 v 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + ...)) +``` + +**The cursor is a slot index the caller owns, and there is no iterator struct** because there is nothing for one to +hold. A map has no tombstones — removal is deferred (`spec-memory.md`) — so a slot is either empty or occupied and the +position is the whole of the state. The cursor starts at 0, comes back one past the entry just answered, and is left +at `cap` by the call that answers false, so a spent cursor keeps answering false rather than wrapping. + +**Three out-pointers and not a returned pair**, because there are no tuples. An `(Option K)` would answer half an +entry and make the value cost a second hash of the key just handed back. The `!` is the cursor: it is the argument +that is written through on the way out. + +**It is the one map entry point that carries neither a hash nor an equality function.** Walking asks nothing about a +key. The two sizes are still there, because the runtime is type-erased and the block geometry is computed from them. + +**The layout, restated, because it is the thing to get wrong here.** `data` is *one* allocation laid out +keys | values | hashes | scratch, each run cell-packed to a cache line — the arrangement the Valgrind lane described +while explaining why a probe overrun is not observable. A key is reached through `flan_cell_at` and never as +`ks + i * ksize`. The hashes are the exception `flan_map_clone` already relies on: an 8-byte element packs 8 to a +64-byte cell with nothing left over, so `g.hs[i]` is the right index and a flat one. + +**Order is block order**, which is the hash's order and not the insertion's, and it changes when the map grows. +`programs/map-iter.flan` is therefore written entirely in sums, counts and lengths — every claim in it is order-free, +which is the contract rather than a weakness of the test. A caller that wants an order sorts what it collected. The +cases that would catch a real mistake: a string key with a struct value, whose two runs have different element sizes +and different packing and so would break if one geometry were used for both; and a 500-entry map, which is several +grows past the minimum and walks a block whose layout has nothing to do with how the entries went in. + +**`map-keys` and `map-values` are still refused, and the reason changed.** The refusal block at the foot of +`prelude.ml` said "a Map iterator"; that is wrong now. What a prelude `defn` cannot write is +`(defn map-keys [m {K V}] (Vec K))` — it has to name its types and there is no `K`. That is generics. The loop is +three lines at the call site, where `K` is known, and that is where it stays. + ## A prelude function may call a prelude macro, and why the fix was not ordering The handoff said the prelude is never macro-expanded — `Macro.program` runs over the file being compiled and the diff --git a/NEXT.md b/NEXT.md index 87d325f..86a6c31 100644 --- a/NEXT.md +++ b/NEXT.md @@ -552,11 +552,11 @@ Tests: `programs/strings.flan`, `programs/format.flan`, `programs/algorithms.fla **What could not be built, and why each one could not.** All four want a compiler or runtime change, and none of them wants a language decision. -- **`Map` keys and values.** The only item on the list above that could not be built at all. `len` reaches a Map and - `get`/`put`/`has-key?` address one entry, but nothing walks the block: `flan_map_len`, `_get`, `_put`, `_has`, - `_clone`, `_reserve` and `_free` is the runtime's whole map surface, with no iterator among them. It wants one - runtime function — `flan_map_next` over the open-addressed block, taking a cursor — and one builtin in `check.ml` - to emit the key and value sizes at the call site. It is *not* a generics problem, and it is a small job. +- ~~**`Map` keys and values.**~~ **Iteration is built** — `flan_map_next` and the `map-next!` builtin, exactly the + shape this described. See [`BUILT.md`](BUILT.md), "`map-next!`, the one thing a Map could not do". + `map-keys`/`map-values` as *prelude functions* stay refused, and the reason is now generics rather than the + iterator: a `defn` has to name its types and `(defn map-keys [m {K V}] (Vec K))` has no `K`. The loop is three + lines at the call site, where `K` is known. - **`map`, `filter`, `reduce`, and a sort taking a comparator.** Blocked on **function values**, not on generics, which is the sharper statement than the one this list made. `Types.Fn` exists; `check.ml` refuses it with "a diff --git a/lib/check.ml b/lib/check.ml index 2bf97d3..772ad6b 100644 --- a/lib/check.ml +++ b/lib/check.ml @@ -2871,6 +2871,48 @@ and named_call ctx ~want loc name args = [ mk loc oty (Tast.If (cond, some, none)) ]))) | _ -> assert false) + (* (map-next! m (addr cur) (addr k) (addr v)) -> bool, and the whole of map + iteration. Before it there was no way to read a map's keys or its values + at all: every other map operation addresses one entry by hashing it, and + nothing walked the block. + + Three out-pointers rather than a returned pair, because there are no + tuples and a (Option K) would answer only half of an entry — the value + would then cost a second hash of the key just answered. The cursor is an + i64 the caller owns and the loop reads as one: + + (let [cur 0 k 0 v 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + ...)) + + It is *not* a generic (map-keys m): a Vec of them needs a signature naming + K, and a prelude defn cannot be written at every K. That one is generics, + not iteration, and it stays refused for that reason. + + No hash and no equality pair go with it — walking asks nothing about a + key — so this is the one map entry point whose signature carries neither, + and the sizes are still needed because the runtime is type-erased. *) + | "map-next!" -> + arity loc name 4 args; + (match args with + | [ target; cur; k; v ] -> + let target = borrowed ctx target (fun () -> check ctx target) in + let kt, vt = map_kv loc "map-next!" target.Tast.ty in + let cur = check ctx ~want:(Types.Ptr (Types.Int Types.I64)) cur in + let k = check ctx ~want:(Types.Ptr kt) k in + let v = check ctx ~want:(Types.Ptr vt) v in + let found = + rt loc (Types.Int Types.I8) "flan_map_next" + [ target; cur; k; v; size_of loc kt; size_of loc vt; here loc ] + in + expect loc ~want + (mk loc Types.Bool + (Tast.Prim + (Tast.Ne, + [ found; + mk loc (Types.Int Types.I8) (Tast.Int (0L, Types.I8)) ]))) + | _ -> assert false) + (* (has-key? m k). (get m k) answers the same question, but through an Option the caller then has to match; this is the form a condition wants, and it copies no value. *) diff --git a/lib/emit.ml b/lib/emit.ml index 27f34cf..a979463 100644 --- a/lib/emit.ml +++ b/lib/emit.ml @@ -2273,6 +2273,10 @@ declare i8 @flan_map_has(ptr, ptr, i64, i64, ptr, ptr, ptr, i64) declare i8 @flan_map_reserve(ptr, i64, i64, i64, ptr, ptr, i64) declare i8 @flan_map_clone(ptr, ptr, ptr, i64, i64, ptr, ptr, i64) declare i64 @flan_map_len(ptr, ptr, i64) +; The cursor step. No hash and no equality pair: walking the block asks +; nothing about a key, which is why this is the one map entry point whose +; signature does not carry them. +declare i8 @flan_map_next(ptr, ptr, ptr, ptr, i64, i64, ptr, i64) declare void @flan_map_free(ptr, i64, i64, ptr, i64) ; The pointer forms, whose signatures end with the transfer channel because a ; hash emitted for a struct key is an ordinary Flan function. Only ever taken diff --git a/lib/prelude.ml b/lib/prelude.ml index 9d8b1d8..022f5eb 100644 --- a/lib/prelude.ml +++ b/lib/prelude.ml @@ -1203,14 +1203,15 @@ let source = {flan| ;; map, filter, reduce Function values. See the head of the slice-algorithm ;; sort-by section: check.ml refuses a function type outright, ;; and there is nothing in the language to pass. -;; map-keys, map-values A Map iterator. `len` reaches a Map and `get`, -;; `put` and `has-key?` address one entry, but there is -;; no entry point in the runtime that walks the block — -;; flan_map_len, _get, _put, _has, _clone, _reserve and -;; _free is the whole surface. This is the one item on -;; NEXT.md's second-tier list that could not be built -;; here at all, and it wants one runtime function and -;; one builtin rather than anything from the language. +;; map-keys, map-values Generics — and the reason changed, which is the +;; point of naming them separately. It used to be the +;; missing Map iterator; `map-next!` is that iterator +;; and walking a map is expressible now. What a defn +;; still cannot say is (defn map-keys [m {K V}] (Vec K)): +;; a prelude function has to name its types, and there +;; is no K. The loop is three lines at the call site, +;; where K is known, and that is where it stays until +;; there are generics. ;; ;; Builder Not refused — declined. strings.Builder in Odin ;; wraps a [dynamic]u8; here the (Vec u8) *is* that and diff --git a/runtime/flan_rt.c b/runtime/flan_rt.c index e38f01d..6f38c55 100644 --- a/runtime/flan_rt.c +++ b/runtime/flan_rt.c @@ -1695,6 +1695,58 @@ int64_t flan_map_len(flan_map *m, const uint8_t *loc, int64_t loclen) { return m->len; } +/* The cursor step, and the whole of iteration. + * + * Everything else in this file addresses *one* entry: get, put and has each + * hash a key and probe. Nothing walked the block, so a map's keys and its + * values could not be read out at all, and this is the one function that + * changes it. + * + * The cursor is a slot index the caller owns, and the contract is the one a + * slot index gives for free: it starts at 0, it is written back one past the + * entry just answered, and a 0 answer leaves it at [cap] so calling again is + * still 0. There is no iterator struct because there is nothing for one to + * hold — a map has no tombstones (removal is deferred), so no state beyond the + * position is needed to know where to resume. + * + * Invalidated by anything that moves the block, exactly as a Vec's slice is: + * a put that grows rehashes into a new block and every index before it means a + * different entry. The epoch check below catches a released arena and nothing + * catches a resize, which is the same bargain [as-slice] already makes. + * + * The layout is the one the geometry describes and is worth restating because + * it is the thing most likely to be got wrong here: [data] is *one* allocation + * laid out keys | values | hashes | scratch, each run cell-packed, so a key is + * reached through [flan_cell_at] and never by [ks + i * ksize]. The hashes are + * the exception the clone loop already relies on — an 8-byte element packs 8 + * to a 64-byte cell with nothing left over, so a flat index is the right + * index. Order is block order, which is the hash's order and not the + * insertion's; two maps holding the same entries may walk them differently. */ +int8_t flan_map_next(flan_map *m, int64_t *cursor, void *kout, void *vout, + int64_t ksize, int64_t vsize, + const uint8_t *loc, int64_t loclen) { + flan_map_geom g; + int64_t cap, i; + flan_map_check(m, loc, loclen); + if (!m->data || m->len == 0) return 0; + cap = flan_map_cap(m); + i = *cursor; + if (i < 0) i = 0; + if (i >= cap) { *cursor = cap; return 0; } + flan_map_geometry(m, ksize, vsize, cap, &g); + for (; i < cap; i++) { + if (g.hs[i] == 0) continue; + memcpy(kout, flan_cell_at(g.ks, ksize, g.kepc, g.kcell, g.kshift, i), + (size_t)ksize); + memcpy(vout, flan_cell_at(g.vs, vsize, g.vepc, g.vcell, g.vshift, i), + (size_t)vsize); + *cursor = i + 1; + return 1; + } + *cursor = cap; + return 0; +} + /* Room for [n] entries without reallocating, which means a block whose 75% * threshold is at least n. */ int8_t flan_map_reserve(flan_map *m, int64_t n, int64_t ksize, int64_t vsize, diff --git a/test/programs/map-iter.flan b/test/programs/map-iter.flan new file mode 100644 index 0000000..c5718c1 --- /dev/null +++ b/test/programs/map-iter.flan @@ -0,0 +1,109 @@ +;; Walking a map, which is what flan_map_next exists for. Every other map +;; operation addresses one entry by hashing it; this is the only thing that +;; reads the block in order. +;; +;; Block order is the hash's order and not the insertion's, so nothing here +;; may depend on which entry comes first: the checks are a sum, a count and a +;; membership test, all of them order-free. That is not a weakness of the test, +;; it is the contract — a caller that wants an order sorts what it collected. + +(defn sum-and-count [] Unit + (let [m (map-new i32 i32)] + (put m 1 10) + (put m 2 20) + (put m 3 30) + (put m 4 40) + (let [cur (i64 0) + k 0 + v 0 + keys 0 + vals 0 + n 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + (set keys (+ keys k)) + (set vals (+ vals v)) + (set n (+ n 1))) + (print n) (print " ") (print keys) (print " ") (print vals) (println "")) + (free m))) + +;; A map that never allocated has no block at all, and one that allocated and +;; holds nothing has a block of nothing but zeroed hashes. Both walk zero +;; times, and they are different code paths to get there. +(defn the-empty-cases [] Unit + (let [m (map-new i32 i32) + cur (i64 0) + k 0 + v 0 + n 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + (set n (+ n 1))) + (print "never allocated: ") (print n) (println "") + (reserve m 64) + (set cur (i64 0)) + (while (map-next! m (addr cur) (addr k) (addr v)) + (set n (+ n 1))) + (print "allocated and empty: ") (print n) (println "") + (free m))) + +;; A cursor left past the end keeps answering false rather than wrapping, so a +;; second loop over a spent cursor is empty and not a repeat. +(defn a-spent-cursor [] Unit + (let [m (map-new i32 i32)] + (put m 5 50) + (put m 6 60) + (let [cur (i64 0) k 0 v 0 n 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + (set n (+ n 1))) + (while (map-next! m (addr cur) (addr k) (addr v)) + (set n (+ n 1))) + (print "spent: ") (print n) (println "")) + (free m))) + +;; A string key and a struct value: the key run and the value run have +;; different element sizes and different cell packing, so this is the case that +;; would catch the two runs being indexed with one geometry. +(defstruct Point [x i32 y i32]) + +(defn wider-entries [] Unit + (let [m (map-new string Point)] + (put m "a" (Point {.x 1 .y 2})) + (put m "bb" (Point {.x 3 .y 4})) + (put m "ccc" (Point {.x 5 .y 6})) + (let [cur (i64 0) + k "" + v (Point {}) + chars 0 + xs 0 + ys 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + (set chars (+ chars (len k))) + (set xs (+ xs (.x v))) + (set ys (+ ys (.y v)))) + (print chars) (print " ") (print xs) (print " ") (print ys) (println "")) + (free m))) + +;; Growth past the 75% threshold rehashes into a new block, so this walks a map +;; whose layout is nothing like its insertion order and at a capacity several +;; doublings past the minimum. +(defn after-growth [] Unit + (let [m (map-new i64 i64)] + (dotimes [i 500] + (put m (i64 i) (* (i64 i) 2))) + (let [cur (i64 0) + k (i64 0) + v (i64 0) + n 0 + doubled 0] + (while (map-next! m (addr cur) (addr k) (addr v)) + (set n (+ n 1)) + (when (= v (* k 2)) (set doubled (+ doubled 1)))) + (print n) (print " ") (print doubled) (print " ") (print (len m)) (println "")) + (free m))) + +(defn main [] i32 + (sum-and-count) + (the-empty-cases) + (a-spent-cursor) + (wider-entries) + (after-growth) + 0) diff --git a/test/test_acceptance.ml b/test/test_acceptance.ml index d2574fc..5691f24 100644 --- a/test/test_acceptance.ml +++ b/test/test_acceptance.ml @@ -1763,6 +1763,21 @@ ERR@7 unexpected token: not the kind the caller was reading body behind an indirection cell, so it is the build that would notice. *) outputs ~dev:true "maps, dev" "programs/maps.flan" maps_out; + (* Iteration, which no map could do at all until flan_map_next. Every case + here is order-free on purpose — block order is the hash's order, not the + insertion's — so the numbers are sums, counts and lengths and never a + first entry. The string-keyed, struct-valued map is the one that would + catch the key and value runs being indexed with a single geometry, since + their element sizes and cell packing differ; and the 500-entry map is + several grows past the minimum, so it walks a block whose layout has + nothing to do with the order the entries went in. *) + let map_iter_out = + "4 10 100\nnever allocated: 0\nallocated and empty: 0\nspent: 2\n\ + 6 9 12\n500 500 500\n" + in + outputs "map iteration" "programs/map-iter.flan" map_iter_out; + outputs ~opt:"-O0" "map iteration, -O0" "programs/map-iter.flan" map_iter_out; + (* The allocation-failure rule is one rule over every allocating operation, so it has to hold for map-new, put, reserve and clone as it does for the Vec's four. A map is the harder case: its growth allocates a new block,