flan/vendor/edn/read.flan

;;;; The dynamic reader: an EDN document, and no type to read it into.
;;;;
;;;; `edn.flan` answers "what is the next token". This answers "what is in the
;;;; file", for a caller that has no struct to hand — a config file whose keys
;;;; are not known until it is read, a tileset, a save. `(read-edn Enemy bytes)`
;;;; is the other direction and is not here: it wants a compile-time walk over
;;;; a struct's fields, there is no run-time type information in this language,
;;;; and it is its own project (NEXT.md item 9).
;;;;
;;;; ── Where the storage comes from ─────────────────────────────────────
;;;;
;;;; `read` takes no allocator and names none. It does not need to:
;;;; spec-memory.md puts the allocator in the calling convention, so every
;;;; (vec-new) and (map-new) below takes the *context*, and the caller chooses
;;;; the tier by writing (with-allocator frame (edn/read bytes)). An explicit
;;;; allocator at a construction site overrides that, which is how this would
;;;; take one as a parameter if the idiom could not say it — and the idiom
;;;; says it, so there is no allocator parameter here and nothing lost.
;;;;
;;;; The tier has to be a region, and that is enforced rather than documented:
;;;; a (Vec Value) whose elements own storage traps at its construction against
;;;; any allocator that can free one block. Calling `read` with the heap in the
;;;; context dies at the first collection in the document, naming the line.
;;;; See test/programs/arena-region.flan.
;;;;
;;;; There is no teardown in this file — no drop, no destructor, no recursive
;;;; free. One (free-all frame) releases the whole document, because every part
;;;; of it came out of the one region.
;;;;
;;;; ── A Value owns its strings, and a Token does not ───────────────────
;;;;
;;;; This is the one place the two layers of this package disagree, and it is
;;;; deliberate. A Token's text is a slice INTO the source buffer; a Value's
;;;; strings are copies, in the allocator, and the document does not point at
;;;; the source at all once `read` has returned.
;;;;
;;;; A view would be cheaper and would be a trap. `read` hands its answer back
;;;; out of the function that owns the buffer, which is exactly the case
;;;; edn.flan's lifetime contract says a view cannot survive: the caller frees
;;;; the bytes it slurped, or reads the next file into them, and every string
;;;; in the document is garbage with nothing to say so. A free-all on the arena
;;;; would not even take them, because they were never in it.
;;;;
;;;; Odin settles the same question the same way: core/encoding/json's parser
;;;; clones a string even when it holds no escapes (parser.odin:388), clones
;;;; keys (:254), and its destroy_value frees them (types.odin:96). A reader
;;;; whose result is self-contained is the only kind that can be a library.
;;;;
;;;; ── Sets are a Vec, and why they are not a Map ───────────────────────
;;;;
;;;; `Value.Set` holds a (Vec Value), deduplicated on insert by a structural
;;;; `value=?`. The obvious shape — a (Map Value bool) — does not typecheck and
;;;; cannot be made to: lib/types.ml `keyable` refuses a key type holding a Vec
;;;; or a Map, and Value holds both. Restricting set elements to the Values
;;;; that *are* keyable was the other way out and is worse: `#{[0 0] [1 0]}` is
;;;; legal EDN and is the exact shape this was built for, so the restriction
;;;; would refuse the motivating file to buy a faster insert.
;;;;
;;;; The cost is stated rather than hidden: insert is O(n) and building a set
;;;; of n elements is O(n²). For the file this was written for — 54 integer
;;;; pairs — that is 1458 comparisons, once, at load. A set large enough for
;;;; the quadratic to matter is one this shape is wrong for, and the reader
;;;; will know before this comment does.

(defdata Value
  [(Nil   [])
   (Bool  [b bool])
   (Int   [n i64])
   (Float [x f64])
   (Text  [s string])
   (Key   [s string])
   (List  [items (Vec Value)])
   (Set   [items (Vec Value)])
   (Table [entries (Map string Value)])])

;; ── Copying a token's text ──────────────────────────────────────────

;; The (Vec u8) is the copy; the string is a view of it, and the Vec header is
;; dropped here on purpose. Nothing individually owns a block in a region —
;; free-all owns all of them — so keeping the header around to free through
;; would be keeping a handle for an operation that never happens.
(defn copy-text [s [u8]] string
  (let [b (vec-new u8)]
    (append! (addr b) s)
    (string (as-slice b))))

;; ── Structural equality ─────────────────────────────────────────────

;; What the set's dedup is written against. Recursive, because a set element
;; may be a vector or a map or another set, and `=` on a Value would compare a
;; Vec header against a Vec header — two copies of one document would never be
;; equal and two aliases of one block always would.
(defn value=? [a Value  b Value] bool
  (match a
    Nil       (match b Nil true _ false)
    ;; `=` on two bools is refused by the language (plan.org, Types), so the
    ;; comparison is written as the thing it means.
    (Bool x)  (match b (Bool y)  (if x y (not y)) _ false)
    (Int x)   (match b (Int y)   (= x y) _ false)
    (Float x) (match b (Float y) (= x y) _ false)
    ;; Text and Key are compared by their bytes and never to each other:
    ;; "a" and :a are two values in EDN and stay two here.
    (Text x)  (match b (Text y)  (bytes=? (bytes x) (bytes y)) _ false)
    (Key x)   (match b (Key y)   (bytes=? (bytes x) (bytes y)) _ false)
    (List xs) (match b (List ys) (items=? xs ys) _ false)
    (Set xs)  (match b (Set ys)  (sets=? xs ys)  _ false)
    (Table e) (match b (Table f) (tables=? e f)  _ false)))

;; A vector is equal element by element, in order.
(defn items=? [xs (Vec Value)  ys (Vec Value)] bool
  (when (!= (len xs) (len ys))
    (return false))
  (dotimes [i (len xs)]
    (when (not (value=? (at xs i) (at ys i)))
      (return false)))
  true)

;; A set is not. #{1 2} and #{2 1} are one value written two ways, and a
;; positional compare would make `#{#{1 2} #{2 1}}` a two-element set — which
;; is the case that decides whether this function is worth having separately
;; from items=?.
(defn sets=? [xs (Vec Value)  ys (Vec Value)] bool
  (when (!= (len xs) (len ys))
    (return false))
  (dotimes [i (len xs)]
    (when (not (member? ys (at xs i)))
      (return false)))
  true)

(defn member? [xs (Vec Value)  v Value] bool
  (dotimes [i (len xs)]
    (when (value=? (at xs i) v)
      (return true)))
  false)

;; Maps compare by size and then by lookup, which is what makes the walk
;; order-independent — two maps built by inserting the same pairs in different
;; orders iterate differently and are the same map.
(defn tables=? [a (Map string Value)  b (Map string Value)] bool
  (when (!= (len a) (len b))
    (return false))
  (let [cur (i64 0)
        k ""
        v Value.Nil]
    (while (map-next! a (addr cur) (addr k) (addr v))
      (match (get b k)
        (Some w) (when (not (value=? v w)) (return false))
        None     (return false))))
  true)

;; ── Reading ─────────────────────────────────────────────────────────

;; One token in hand, and the cursor for whatever that token opens. Public
;; because it is the entry point for a caller who wants the error *position*:
;; a caller driving its own Cursor can ask (edn/error-pos c) afterwards, and
;; `read` below cannot, because the cursor it made is gone.
(defn read-value [c (Ptr Cursor)  t Token] Value
  (cond
    (= (.kind t) tok-bool)
      (Value.Bool {.b (match (bool-of t) (Some v) v None false)})
    (= (.kind t) tok-int)
      (Value.Int {.n (match (int-of t) (Some v) v None (i64 0))})
    (= (.kind t) tok-float)
      (Value.Float {.x (match (float-of t) (Some v) v None 0.0)})
    (= (.kind t) tok-string)  (Value.Text {.s (copy-text (.text t))})
    (= (.kind t) tok-keyword) (Value.Key  {.s (copy-text (.text t))})
    ;; A symbol becomes a Key. There is no Symbol case, because nothing that
    ;; reads a document this way tells the two apart — and a case nobody can
    ;; act on differently is a case that only makes matches longer.
    (= (.kind t) tok-symbol)  (Value.Key  {.s (copy-text (.text t))})

    (= (.kind t) tok-vec-open)
      (let [items (vec-new Value)
            u (next c)]
        (while (and (ok? c)
                    (!= (.kind u) tok-vec-close)
                    (!= (.kind u) tok-eof))
          (push items (read-value c u))
          (set u (next c)))
        (Value.List {.items items}))

    ;; A set ends on tok-map-close, because `}` is the byte that ends it. The
    ;; dedup is here and not at the end: a set with a duplicate in it never
    ;; exists, so nothing downstream has to know that one might.
    (= (.kind t) tok-set-open)
      (let [items (vec-new Value)
            u (next c)]
        (while (and (ok? c)
                    (!= (.kind u) tok-map-close)
                    (!= (.kind u) tok-eof))
          (let [v (read-value c u)]
            (when (not (member? items v))
              (push items v)))
          (set u (next c)))
        (Value.Set {.items items}))

    ;; A map's key is whatever token is there, and its text is the key —
    ;; copied, so :a and "a" collide as keys here where EDN keeps them apart.
    ;; That is a real narrowing and it is the price of a (Map string Value):
    ;; the alternative is a (Map Value Value), which `keyable` refuses for the
    ;; reason the set's comment above gives.
    (= (.kind t) tok-map-open)
      (let [entries (map-new string Value)
            k (next c)]
        (while (and (ok? c)
                    (!= (.kind k) tok-map-close)
                    (!= (.kind k) tok-eof))
          (let [v (next c)]
            (put entries (copy-text (.text k)) (read-value c v)))
          (set k (next c)))
        (Value.Table {.entries entries}))

    :else Value.Nil))

;; The whole document, from a byte slice, in the calling convention's
;; allocator.
;;
;; (Option Value) and not Value, which is the one place this departs from
;; edn.flan's "errors live on the cursor, not in the return type". The cursor
;; is made inside this function and dies with it, so there is nothing left for
;; a caller to ask — and a Value.Nil answer would be indistinguishable from the
;; document that is literally `nil`, which is the class of quiet wrongness the
;; package's refusals exist to avoid. A caller who needs the byte offset builds
;; the Cursor itself and calls read-value; that is the three lines below.
;;
;; One collision the Option does NOT resolve, said here rather than discovered:
;; empty input answers (Some Value.Nil), the same as the document `nil`. Empty
;; is not malformed — a tokenizer over no bytes reports no error, correctly —
;; and the alternative is this function deciding that an empty file is a
;; failure, which is the caller's question and not the reader's.
(defn read [src [u8]] (Option Value)
  (let [c (cursor src)
        t (next (addr c))
        v (read-value (addr c) t)]
    (if (ok? (addr c)) (Some v) None)))