;;;; The dynamic reader: an EDN document, and no type to read it into. ;;;; ;;;; `edn.flan` answers "what is the next token". This answers "what is in the ;;;; file", for a caller that has no struct to hand — a config file whose keys ;;;; are not known until it is read, a tileset, a save. `(read-edn Enemy bytes)` ;;;; is the other direction and is not here: it wants a compile-time walk over ;;;; a struct's fields, there is no run-time type information in this language, ;;;; and it is its own project (NEXT.md item 9). ;;;; ;;;; ── Where the storage comes from ───────────────────────────────────── ;;;; ;;;; `read` takes no allocator and names none. It does not need to: ;;;; spec-memory.md puts the allocator in the calling convention, so every ;;;; (vec-new) and (map-new) below takes the *context*, and the caller chooses ;;;; the tier by writing (with-allocator frame (edn/read bytes)). An explicit ;;;; allocator at a construction site overrides that, which is how this would ;;;; take one as a parameter if the idiom could not say it — and the idiom ;;;; says it, so there is no allocator parameter here and nothing lost. ;;;; ;;;; `read-file` is the one exception and states its own reason at the bottom ;;;; of this file: the buffer it slurps is named against the heap because its ;;;; life is strictly inside the call and the caller cannot observe it, so it ;;;; is not the caller's tier to choose. The document it answers still lands ;;;; wherever the context says. ;;;; ;;;; The tier has to be a region, and that is enforced rather than documented: ;;;; a (Vec Value) whose elements own storage traps at its construction against ;;;; any allocator that can free one block. Calling `read` with the heap in the ;;;; context dies at the first collection in the document, naming the line. ;;;; See test/programs/arena-region.flan. ;;;; ;;;; There is no teardown in this file — no drop, no destructor, no recursive ;;;; free. One (free-all frame) releases the whole document, because every part ;;;; of it came out of the one region. ;;;; ;;;; ── A Value owns its strings, and a Token does not ─────────────────── ;;;; ;;;; This is the one place the two layers of this package disagree, and it is ;;;; deliberate. A Token's text is a slice INTO the source buffer; a Value's ;;;; strings are copies, in the allocator, and the document does not point at ;;;; the source at all once `read` has returned. ;;;; ;;;; A view would be cheaper and would be a trap. `read` hands its answer back ;;;; out of the function that owns the buffer, which is exactly the case ;;;; edn.flan's lifetime contract says a view cannot survive: the caller frees ;;;; the bytes it slurped, or reads the next file into them, and every string ;;;; in the document is garbage with nothing to say so. A free-all on the arena ;;;; would not even take them, because they were never in it. ;;;; ;;;; Odin settles the same question the same way: core/encoding/json's parser ;;;; clones a string even when it holds no escapes (parser.odin:388), clones ;;;; keys (:254), and its destroy_value frees them (types.odin:96). A reader ;;;; whose result is self-contained is the only kind that can be a library. ;;;; ;;;; ── Sets are a Vec, and why they are not a Map ─────────────────────── ;;;; ;;;; `Value.Set` holds a (Vec Value), deduplicated on insert by a structural ;;;; `value=?`. The obvious shape — a (Map Value bool) — does not typecheck and ;;;; cannot be made to: lib/types.ml `keyable` refuses a key type holding a Vec ;;;; or a Map, and Value holds both. Restricting set elements to the Values ;;;; that *are* keyable was the other way out and is worse: `#{[0 0] [1 0]}` is ;;;; legal EDN and is the exact shape this was built for, so the restriction ;;;; would refuse the motivating file to buy a faster insert. ;;;; ;;;; The cost is stated rather than hidden: insert is O(n) and building a set ;;;; of n elements is O(n²). For the file this was written for — 54 integer ;;;; pairs — that is 1458 comparisons, once, at load. A set large enough for ;;;; the quadratic to matter is one this shape is wrong for, and the reader ;;;; will know before this comment does. (defdata Value [(Nil []) (Bool [b bool]) (Int [n i64]) (Float [x f64]) (Text [s string]) (Key [s string]) (List [items (Vec Value)]) (Set [items (Vec Value)]) (Table [entries (Map string Value)])]) ;; ── Copying a token's text ────────────────────────────────────────── ;; The (Vec u8) is the copy; the string is a view of it, and the Vec header is ;; dropped here on purpose. Nothing individually owns a block in a region — ;; free-all owns all of them — so keeping the header around to free through ;; would be keeping a handle for an operation that never happens. (defn copy-text [s [u8]] string (let [b (vec-new u8)] (append (addr b) s) (string (as-slice b)))) ;; ── Structural equality ───────────────────────────────────────────── ;; What the set's dedup is written against. Recursive, because a set element ;; may be a vector or a map or another set, and `=` on a Value would compare a ;; Vec header against a Vec header — two copies of one document would never be ;; equal and two aliases of one block always would. (defn value=? [a Value b Value] bool (match a Nil (match b Nil true _ false) ;; `=` on two bools is refused by the language (plan.org, Types), so the ;; comparison is written as the thing it means. (Bool x) (match b (Bool y) (if x y (not y)) _ false) (Int x) (match b (Int y) (= x y) _ false) (Float x) (match b (Float y) (= x y) _ false) ;; Text and Key are compared by their bytes and never to each other: ;; "a" and :a are two values in EDN and stay two here. (Text x) (match b (Text y) (bytes=? (bytes x) (bytes y)) _ false) (Key x) (match b (Key y) (bytes=? (bytes x) (bytes y)) _ false) (List xs) (match b (List ys) (items=? xs ys) _ false) (Set xs) (match b (Set ys) (sets=? xs ys) _ false) (Table e) (match b (Table f) (tables=? e f) _ false))) ;; A vector is equal element by element, in order. (defn items=? [xs (Vec Value) ys (Vec Value)] bool (when (!= (len xs) (len ys)) (return false)) (dotimes [i (len xs)] (when (not (value=? (at xs i) (at ys i))) (return false))) true) ;; A set is not. #{1 2} and #{2 1} are one value written two ways, and a ;; positional compare would make `#{#{1 2} #{2 1}}` a two-element set — which ;; is the case that decides whether this function is worth having separately ;; from items=?. (defn sets=? [xs (Vec Value) ys (Vec Value)] bool (when (!= (len xs) (len ys)) (return false)) (dotimes [i (len xs)] (when (not (member? ys (at xs i))) (return false))) true) (defn member? [xs (Vec Value) v Value] bool (dotimes [i (len xs)] (when (value=? (at xs i) v) (return true))) false) ;; Maps compare by size and then by lookup, which is what makes the walk ;; order-independent — two maps built by inserting the same pairs in different ;; orders iterate differently and are the same map. (defn tables=? [a (Map string Value) b (Map string Value)] bool (when (!= (len a) (len b)) (return false)) (let [cur (i64 0) k "" v Value.Nil] (while (map-next a (addr cur) (addr k) (addr v)) (match (get b k) (Some w) (when (not (value=? v w)) (return false)) None (return false)))) true) ;; ── Reading ───────────────────────────────────────────────────────── ;; One token in hand, and the cursor for whatever that token opens. Public ;; because it is the entry point for a caller who wants the error *position*: ;; a caller driving its own Cursor can ask (edn/error-pos c) afterwards, and ;; `read` below cannot, because the cursor it made is gone. (defn read-value [c (Ptr Cursor) t Token] Value (cond (= (.kind t) tok-bool) (Value.Bool {.b (match (bool-of t) (Some v) v None false)}) (= (.kind t) tok-int) (Value.Int {.n (match (int-of t) (Some v) v None (i64 0))}) (= (.kind t) tok-float) (Value.Float {.x (match (float-of t) (Some v) v None 0.0)}) (= (.kind t) tok-string) (Value.Text {.s (copy-text (.text t))}) (= (.kind t) tok-keyword) (Value.Key {.s (copy-text (.text t))}) ;; A symbol becomes a Key. There is no Symbol case, because nothing that ;; reads a document this way tells the two apart — and a case nobody can ;; act on differently is a case that only makes matches longer. (= (.kind t) tok-symbol) (Value.Key {.s (copy-text (.text t))}) (= (.kind t) tok-vec-open) (let [items (vec-new Value) u (next c)] (while (and (ok? c) (!= (.kind u) tok-vec-close) (!= (.kind u) tok-eof)) (push items (read-value c u)) (set u (next c))) (Value.List {.items items})) ;; A set ends on tok-map-close, because `}` is the byte that ends it. The ;; dedup is here and not at the end: a set with a duplicate in it never ;; exists, so nothing downstream has to know that one might. (= (.kind t) tok-set-open) (let [items (vec-new Value) u (next c)] (while (and (ok? c) (!= (.kind u) tok-map-close) (!= (.kind u) tok-eof)) (let [v (read-value c u)] (when (not (member? items v)) (push items v))) (set u (next c))) (Value.Set {.items items})) ;; A map's key is whatever token is there, and its text is the key — ;; copied, so :a and "a" collide as keys here where EDN keeps them apart. ;; That is a real narrowing and it is the price of a (Map string Value): ;; the alternative is a (Map Value Value), which `keyable` refuses for the ;; reason the set's comment above gives. (= (.kind t) tok-map-open) (let [entries (map-new string Value) k (next c)] (while (and (ok? c) (!= (.kind k) tok-map-close) (!= (.kind k) tok-eof)) (let [v (next c)] (put entries (copy-text (.text k)) (read-value c v))) (set k (next c))) (Value.Table {.entries entries})) :else Value.Nil)) ;; The whole document, from a byte slice, in the calling convention's ;; allocator. ;; ;; (Option Value) and not Value, which is the one place this departs from ;; edn.flan's "errors live on the cursor, not in the return type". The cursor ;; is made inside this function and dies with it, so there is nothing left for ;; a caller to ask — and a Value.Nil answer would be indistinguishable from the ;; document that is literally `nil`, which is the class of quiet wrongness the ;; package's refusals exist to avoid. A caller who needs the byte offset builds ;; the Cursor itself and calls read-value; that is the three lines below. ;; ;; One collision the Option does NOT resolve, said here rather than discovered: ;; empty input answers (Some Value.Nil), the same as the document `nil`. Empty ;; is not malformed — a tokenizer over no bytes reports no error, correctly — ;; and the alternative is this function deciding that an empty file is a ;; failure, which is the caller's question and not the reader's. (defn read [src [u8]] (Option Value) (let [c (cursor src) t (next (addr c)) v (read-value (addr c) t)] (if (ok? (addr c)) (Some v) None))) ;; The same, from a path, and the reason it is worth having is the line above ;; it: **the source buffer is dead the moment `read` returns.** Every string in ;; the document is a copy in the allocator — that is this file's "A Value owns ;; its strings" section, and test_acceptance.ml's edn-read case proves it by ;; overwriting every byte of the buffer after the read and still printing the ;; string it found. So the buffer has no reader once this returns, which means ;; it does not have to be the caller's to hold, and a caller writing ;; ;; (defconst raw (embed "game-data.edn")) ;; (edn/read raw) ;; ;; or a slurp-and-free pair around `read` is keeping a name alive for a value ;; whose whole life fits inside one call. ;; ;; **The heap is named, and this is the one place in the package that names an ;; allocator.** The file header argues that `read` takes none because the tier ;; is the caller's choice; that argument does not reach this buffer, because ;; the caller can never observe it. Left to the context, (with-allocator frame ;; (edn/read-file p)) would grow the region by the file's size for bytes that ;; die immediately, and the `free` below would buy nothing back — a bump ;; allocator has no FLAN_CAN_FREE, so freeing into one is a no-op. Against the ;; heap the free is real, and the document still lands in whatever tier the ;; caller chose, because that is where `read`'s own (vec-new) and (map-new) go. ;; ;; The `defer` rather than a trailing (free src) is for the transfer path: ;; `read` allocates, so it can signal StorageExhausted, and a handler that ;; answers by transferring out would otherwise leave the buffer behind. ;; ;; **A FileError passes straight through, and that is the decision, not an ;; omission.** The return type here is already saying something: `None` means ;; the document was malformed, which the header above argues at length has to ;; stay distinguishable from the document that is literally `nil`. Folding "the ;; file was not there" into that same `None` would collapse the distinction the ;; Option exists for. And this function has nothing to answer a FileError ;; *with* — `use-value` wants a path only the caller knows, and whether a ;; missing file is fatal or is a cue to write a default is the caller's policy ;; in every program. Nothing here establishes a handler, so slurp's condition ;; reaches the caller's with both restarts still armed; edn-read.flan runs the ;; `use-value` half, where the handler names another path, the read happens ;; against that file, and this function never learns that anything went wrong. (defn read-file [path string] (Option Value) (let [src (slurp path (heap-allocator))] (defer (free src)) (read (as-slice src))))