flan/vendor/edn/read.flan

;;;; The dynamic reader: an EDN document, and no type to read it into.
;;;;
;;;; `edn.flan` answers "what is the next token". This answers "what is in the
;;;; file", for a caller that has no struct to hand — a config file whose keys
;;;; are not known until it is read, a tileset, a save. `(read-edn Enemy bytes)`
;;;; is the other direction and is not here: it wants a compile-time walk over
;;;; a struct's fields, there is no run-time type information in this language,
;;;; and it is its own project (NEXT.md item 9).
;;;;
;;;; ── Where the storage comes from ─────────────────────────────────────
;;;;
;;;; `read` takes no allocator and names none. It does not need to:
;;;; spec-memory.md puts the allocator in the calling convention, so every
;;;; (vec-new) and (map-new) below takes the *context*, and the caller chooses
;;;; the tier by writing (with-allocator frame (edn/read bytes)). An explicit
;;;; allocator at a construction site overrides that, which is how this would
;;;; take one as a parameter if the idiom could not say it — and the idiom
;;;; says it, so there is no allocator parameter here and nothing lost.
;;;;
;;;; `read-file` is the one exception and states its own reason at the bottom
;;;; of this file: the buffer it slurps is named against the heap because its
;;;; life is strictly inside the call and the caller cannot observe it, so it
;;;; is not the caller's tier to choose. The document it answers still lands
;;;; wherever the context says.
;;;;
;;;; The tier has to be a region, and that is enforced rather than documented:
;;;; a (Vec Value) whose elements own storage traps at its construction against
;;;; any allocator that can free one block. Calling `read` with the heap in the
;;;; context dies at the first collection in the document, naming the line.
;;;; See test/programs/arena-region.flan.
;;;;
;;;; There is no teardown in this file — no drop, no destructor, no recursive
;;;; free. One (free-all frame) releases the whole document, because every part
;;;; of it came out of the one region.
;;;;
;;;; ── A Value owns its strings, and a Token does not ───────────────────
;;;;
;;;; This is the one place the two layers of this package disagree, and it is
;;;; deliberate. A Token's text is a slice INTO the source buffer; a Value's
;;;; strings are copies, in the allocator, and the document does not point at
;;;; the source at all once `read` has returned.
;;;;
;;;; A view would be cheaper and would be a trap. `read` hands its answer back
;;;; out of the function that owns the buffer, which is exactly the case
;;;; edn.flan's lifetime contract says a view cannot survive: the caller frees
;;;; the bytes it slurped, or reads the next file into them, and every string
;;;; in the document is garbage with nothing to say so. A free-all on the arena
;;;; would not even take them, because they were never in it.
;;;;
;;;; Odin settles the same question the same way: core/encoding/json's parser
;;;; clones a string even when it holds no escapes (parser.odin:388), clones
;;;; keys (:254), and its destroy_value frees them (types.odin:96). A reader
;;;; whose result is self-contained is the only kind that can be a library.
;;;;
;;;; ── Sets are a Vec, and why they are not a Map ───────────────────────
;;;;
;;;; `Value.Set` holds a (Vec Value), deduplicated on insert by a structural
;;;; `value=?`. The obvious shape — a (Map Value bool) — does not typecheck and
;;;; cannot be made to: lib/types.ml `keyable` refuses a key type holding a Vec
;;;; or a Map, and Value holds both. Restricting set elements to the Values
;;;; that *are* keyable was the other way out and is worse: `#{[0 0] [1 0]}` is
;;;; legal EDN and is the exact shape this was built for, so the restriction
;;;; would refuse the motivating file to buy a faster insert.
;;;;
;;;; The cost is stated rather than hidden: insert is O(n) and building a set
;;;; of n elements is O(n²). For the file this was written for — 54 integer
;;;; pairs — that is 1458 comparisons, once, at load. A set large enough for
;;;; the quadratic to matter is one this shape is wrong for, and the reader
;;;; will know before this comment does.

(defdata Value
  [(Nil   [])
   (Bool  [b bool])
   (Int   [n i64])
   (Float [x f64])
   (Text  [s string])
   (Key   [s string])
   (List  [items (Vec Value)])
   (Set   [items (Vec Value)])
   (Table [entries (Map string Value)])])

;; ── Copying a token's text ──────────────────────────────────────────

;; The (Vec u8) is the copy; the string is a view of it, and the Vec header is
;; dropped here on purpose. Nothing individually owns a block in a region —
;; free-all owns all of them — so keeping the header around to free through
;; would be keeping a handle for an operation that never happens.
(defn copy-text [s [u8]] string
  (let [b (vec-new u8)]
    (append! (addr b) s)
    (string (as-slice b))))

;; ── Structural equality ─────────────────────────────────────────────

;; What the set's dedup is written against. Recursive, because a set element
;; may be a vector or a map or another set, and `=` on a Value would compare a
;; Vec header against a Vec header — two copies of one document would never be
;; equal and two aliases of one block always would.
(defn value=? [a Value  b Value] bool
  (match a
    Nil       (match b Nil true _ false)
    ;; `=` on two bools is refused by the language (plan.org, Types), so the
    ;; comparison is written as the thing it means.
    (Bool x)  (match b (Bool y)  (if x y (not y)) _ false)
    (Int x)   (match b (Int y)   (= x y) _ false)
    (Float x) (match b (Float y) (= x y) _ false)
    ;; Text and Key are compared by their bytes and never to each other:
    ;; "a" and :a are two values in EDN and stay two here.
    (Text x)  (match b (Text y)  (bytes=? (bytes x) (bytes y)) _ false)
    (Key x)   (match b (Key y)   (bytes=? (bytes x) (bytes y)) _ false)
    (List xs) (match b (List ys) (items=? xs ys) _ false)
    (Set xs)  (match b (Set ys)  (sets=? xs ys)  _ false)
    (Table e) (match b (Table f) (tables=? e f)  _ false)))

;; A vector is equal element by element, in order.
(defn items=? [xs (Vec Value)  ys (Vec Value)] bool
  (when (!= (len xs) (len ys))
    (return false))
  (dotimes [i (len xs)]
    (when (not (value=? (at xs i) (at ys i)))
      (return false)))
  true)

;; A set is not. #{1 2} and #{2 1} are one value written two ways, and a
;; positional compare would make `#{#{1 2} #{2 1}}` a two-element set — which
;; is the case that decides whether this function is worth having separately
;; from items=?.
(defn sets=? [xs (Vec Value)  ys (Vec Value)] bool
  (when (!= (len xs) (len ys))
    (return false))
  (dotimes [i (len xs)]
    (when (not (member? ys (at xs i)))
      (return false)))
  true)

(defn member? [xs (Vec Value)  v Value] bool
  (dotimes [i (len xs)]
    (when (value=? (at xs i) v)
      (return true)))
  false)

;; Maps compare by size and then by lookup, which is what makes the walk
;; order-independent — two maps built by inserting the same pairs in different
;; orders iterate differently and are the same map.
(defn tables=? [a (Map string Value)  b (Map string Value)] bool
  (when (!= (len a) (len b))
    (return false))
  (let [cur (i64 0)
        k ""
        v Value.Nil]
    (while (map-next! a (addr cur) (addr k) (addr v))
      (match (get b k)
        (Some w) (when (not (value=? v w)) (return false))
        None     (return false))))
  true)

;; ── Reading ─────────────────────────────────────────────────────────

;; One token in hand, and the cursor for whatever that token opens. Public
;; because it is the entry point for a caller who wants the error *position*:
;; a caller driving its own Cursor can ask (edn/error-pos c) afterwards, and
;; `read` below cannot, because the cursor it made is gone.
(defn read-value [c (Ptr Cursor)  t Token] Value
  (cond
    (= (.kind t) tok-bool)
      (Value.Bool {.b (match (bool-of t) (Some v) v None false)})
    (= (.kind t) tok-int)
      (Value.Int {.n (match (int-of t) (Some v) v None (i64 0))})
    (= (.kind t) tok-float)
      (Value.Float {.x (match (float-of t) (Some v) v None 0.0)})
    (= (.kind t) tok-string)  (Value.Text {.s (copy-text (.text t))})
    (= (.kind t) tok-keyword) (Value.Key  {.s (copy-text (.text t))})
    ;; A symbol becomes a Key. There is no Symbol case, because nothing that
    ;; reads a document this way tells the two apart — and a case nobody can
    ;; act on differently is a case that only makes matches longer.
    (= (.kind t) tok-symbol)  (Value.Key  {.s (copy-text (.text t))})

    (= (.kind t) tok-vec-open)
      (let [items (vec-new Value)
            u (next c)]
        (while (and (ok? c)
                    (!= (.kind u) tok-vec-close)
                    (!= (.kind u) tok-eof))
          (push items (read-value c u))
          (set u (next c)))
        (Value.List {.items items}))

    ;; A set ends on tok-map-close, because `}` is the byte that ends it. The
    ;; dedup is here and not at the end: a set with a duplicate in it never
    ;; exists, so nothing downstream has to know that one might.
    (= (.kind t) tok-set-open)
      (let [items (vec-new Value)
            u (next c)]
        (while (and (ok? c)
                    (!= (.kind u) tok-map-close)
                    (!= (.kind u) tok-eof))
          (let [v (read-value c u)]
            (when (not (member? items v))
              (push items v)))
          (set u (next c)))
        (Value.Set {.items items}))

    ;; A map's key is whatever token is there, and its text is the key —
    ;; copied, so :a and "a" collide as keys here where EDN keeps them apart.
    ;; That is a real narrowing and it is the price of a (Map string Value):
    ;; the alternative is a (Map Value Value), which `keyable` refuses for the
    ;; reason the set's comment above gives.
    (= (.kind t) tok-map-open)
      (let [entries (map-new string Value)
            k (next c)]
        (while (and (ok? c)
                    (!= (.kind k) tok-map-close)
                    (!= (.kind k) tok-eof))
          (let [v (next c)]
            (put entries (copy-text (.text k)) (read-value c v)))
          (set k (next c)))
        (Value.Table {.entries entries}))

    :else Value.Nil))

;; The whole document, from a byte slice, in the calling convention's
;; allocator.
;;
;; (Option Value) and not Value, which is the one place this departs from
;; edn.flan's "errors live on the cursor, not in the return type". The cursor
;; is made inside this function and dies with it, so there is nothing left for
;; a caller to ask — and a Value.Nil answer would be indistinguishable from the
;; document that is literally `nil`, which is the class of quiet wrongness the
;; package's refusals exist to avoid. A caller who needs the byte offset builds
;; the Cursor itself and calls read-value; that is the three lines below.
;;
;; One collision the Option does NOT resolve, said here rather than discovered:
;; empty input answers (Some Value.Nil), the same as the document `nil`. Empty
;; is not malformed — a tokenizer over no bytes reports no error, correctly —
;; and the alternative is this function deciding that an empty file is a
;; failure, which is the caller's question and not the reader's.
(defn read [src [u8]] (Option Value)
  (let [c (cursor src)
        t (next (addr c))
        v (read-value (addr c) t)]
    (if (ok? (addr c)) (Some v) None)))

;; The same, from a path, and the reason it is worth having is the line above
;; it: **the source buffer is dead the moment `read` returns.** Every string in
;; the document is a copy in the allocator — that is this file's "A Value owns
;; its strings" section, and test_acceptance.ml's edn-read case proves it by
;; overwriting every byte of the buffer after the read and still printing the
;; string it found. So the buffer has no reader once this returns, which means
;; it does not have to be the caller's to hold, and a caller writing
;;
;;   (defconst raw (embed "game-data.edn"))
;;   (edn/read raw)
;;
;; or a slurp-and-free pair around `read` is keeping a name alive for a value
;; whose whole life fits inside one call.
;;
;; **The heap is named, and this is the one place in the package that names an
;; allocator.** The file header argues that `read` takes none because the tier
;; is the caller's choice; that argument does not reach this buffer, because
;; the caller can never observe it. Left to the context, (with-allocator frame
;; (edn/read-file p)) would grow the region by the file's size for bytes that
;; die immediately, and the `free` below would buy nothing back — a bump
;; allocator has no FLAN_CAN_FREE, so freeing into one is a no-op. Against the
;; heap the free is real, and the document still lands in whatever tier the
;; caller chose, because that is where `read`'s own (vec-new) and (map-new) go.
;;
;; The `defer` rather than a trailing (free src) is for the transfer path:
;; `read` allocates, so it can signal StorageExhausted, and a handler that
;; answers by transferring out would otherwise leave the buffer behind.
;;
;; **A FileError passes straight through, and that is the decision, not an
;; omission.** The return type here is already saying something: `None` means
;; the document was malformed, which the header above argues at length has to
;; stay distinguishable from the document that is literally `nil`. Folding "the
;; file was not there" into that same `None` would collapse the distinction the
;; Option exists for. And this function has nothing to answer a FileError
;; *with* — `use-value` wants a path only the caller knows, and whether a
;; missing file is fatal or is a cue to write a default is the caller's policy
;; in every program. Nothing here establishes a handler, so slurp's condition
;; reaches the caller's with both restarts still armed; edn-read.flan runs the
;; `use-value` half, where the handler names another path, the read happens
;; against that file, and this function never learns that anything went wrong.
(defn read-file [path string] (Option Value)
  (let [src (slurp path (heap-allocator))]
    (defer (free src))
    (read (as-slice src))))