Joseph Ferano
ab82e46119
The dyn runtime gets a map object and an interned keyword, alongside the
vec it already had. {:a 1 :b s} is a map literal wherever a struct
literal isn't — the parser tells the two apart by whether the first form
in the braces is a .field symbol — and a bracket literal builds the
runtime's own vec rather than a typed array wherever a dyn is wanted, which
is what lets a map literal's values nest arrays and maps freely. get, put,
len and has-key? all learn a dyn-map arm alongside the typed-map one they
already had, and (keyword s) builds the same interned value a :foo literal
does, for a name that only exists at run time. nil is now a literal, the
dyn absence value that get answers for a key a map does not hold.
On the runtime side, flan_dyn.c gets an OBJ_MAP that shares the vec's
storage arm and doubles its accounting, a linear-scan intern table for
keywords that makes equality an identity compare, and structural map
equality by lookup rather than position. The marker traces a map's
interleaved keys and values the same way it already traced a vec.
edn/read and its callers move off the old (Option Value) union entirely:
a document is plain dyn now, sets are dyn maps to true, and arena-edn.flan
is retired along with the union it demonstrated. The acceptance suite's
edn-read and json rows were recaptured against the new shape, and a new
dyn-map.flan program exercises the map and keyword operations end to end,
including a 200k-iteration churn loop against a rooted map that runs
GC for real, across the LLVM, -O0 and x86 rows, and under the sanitizer.
Keywords are dyn everywhere an enum isn't expected, which changed what a
couple of existing checker tests actually see refused; both were updated
to the sentence the checker gives now rather than the one it used to.
162 lines
8.1 KiB
Plaintext
162 lines
8.1 KiB
Plaintext
;;;; The dynamic reader: an EDN document, and no type to read it into.
|
|
;;;;
|
|
;;;; `edn.flan` answers "what is the next token". This answers "what is in the
|
|
;;;; file", for a caller that has no struct to hand — a config file whose keys
|
|
;;;; are not known until it is read, a tileset, a save. `(edn/defedn T path)`
|
|
;;;; in provide.flan is the other direction: the shape known at compile time,
|
|
;;;; read into a struct, at no run-time cost. Together they are the two sides
|
|
;;;; of one capability, and this is the dynamic one.
|
|
;;;;
|
|
;;;; ── What a document reads as ─────────────────────────────────────────
|
|
;;;;
|
|
;;;; Plain dyn values, the runtime's own:
|
|
;;;;
|
|
;;;; nil true 42 1.5 nil, bool, int, float
|
|
;;;; "text" a dyn text — a copy, owned by the collector
|
|
;;;; :key symbol a keyword, interned, so equality is identity
|
|
;;;; [1 2] a dyn vec
|
|
;;;; {:a 1} a dyn map — keys are whole values, so :a and "a"
|
|
;;;; stay two keys the way EDN says they are
|
|
;;;; #{1 2} a dyn map from each element to true. There is no
|
|
;;;; set kind; a map's put already replaces an equal
|
|
;;;; key, so the dedup is the representation's own, and
|
|
;;;; (has-key? s x) is the membership test.
|
|
;;;;
|
|
;;;; This file used to define edn/Value — a tagged union with its own
|
|
;;;; structural equality, its own set dedup and its own table compare, written
|
|
;;;; before the dyn runtime existed. That was one capability implemented twice
|
|
;;;; on the same side, and the duplicity audit (docs/SPIKE-DUPLICITY.md §5)
|
|
;;;; retired it: everything value=? and its four helpers did is what
|
|
;;;; flan_dyn_eq does, and the (Map Value Value) the typed side refused —
|
|
;;;; keyable says no — is exactly what the dyn map serves without being asked.
|
|
;;;;
|
|
;;;; ── Where the storage comes from ─────────────────────────────────────
|
|
;;;;
|
|
;;;; The collector's heap, all of it. A dyn value's storage is the dyn
|
|
;;;; runtime's — that is what lets the collector find the values inside it —
|
|
;;;; so `read` neither takes an allocator nor consults the ambient one, and
|
|
;;;; there is no free-all to call and nothing to tear down. The strings are
|
|
;;;; copies: boxing a string is flan_dyn_from_bytes, which copies into the
|
|
;;;; heap, so the document does not point at the source buffer at all once
|
|
;;;; `read` has returned. Overwrite the buffer, free it, read the next file
|
|
;;;; into it — the document stands.
|
|
;;;;
|
|
;;;; ── Malformed input is nil, and the narrowing is stated ──────────────
|
|
;;;;
|
|
;;;; `read` answers nil for a document that failed to tokenize — and nil is
|
|
;;;; also what the document `nil` reads as. The old (Option Value) return kept
|
|
;;;; those apart; a dyn nil cannot, and wrapping dyn in an Option today would
|
|
;;;; put the document where the collector cannot see it (a dyn inside a typed
|
|
;;;; container is unrooted until the per-type descriptors land — the queue's
|
|
;;;; item 2). A caller who needs the distinction drives its own cursor and
|
|
;;;; asks it afterwards, which is also how the error *position* has always
|
|
;;;; been got:
|
|
;;;;
|
|
;;;; (let [c (edn/cursor src)
|
|
;;;; v (edn/read-value (addr c) (edn/next (addr c)))]
|
|
;;;; (if (edn/ok? (addr c)) ... (edn/error-pos (addr c)) ...))
|
|
|
|
;; ── Copying a token's text ──────────────────────────────────────────
|
|
|
|
;; Not the dyn reader's own — everything below boxes through the runtime,
|
|
;; which copies for itself — but provide.flan's generated readers build typed
|
|
;; strings out of token text and this is where that copy has always lived.
|
|
;; The (Vec u8) is the copy; the string is a view of it, and the Vec header is
|
|
;; dropped here on purpose. Nothing individually owns a block in a region —
|
|
;; free-all owns all of them — so keeping the header around to free through
|
|
;; would be keeping a handle for an operation that never happens.
|
|
(defn copy-text [s [u8]] string
|
|
(let [b (vec-new u8)]
|
|
(append (addr b) s)
|
|
(string (as-slice b))))
|
|
|
|
;; ── Reading ─────────────────────────────────────────────────────────
|
|
|
|
;; One token in hand, and the cursor for whatever that token opens. Public
|
|
;; because it is the entry point for a caller who wants the error position —
|
|
;; see the header.
|
|
(defn read-value [c (Ptr Cursor) t Token] dyn
|
|
(cond
|
|
(= (.kind t) tok-nil) nil
|
|
(= (.kind t) tok-bool) (match (bool-of t) (Some v) v None false)
|
|
(= (.kind t) tok-int) (match (int-of t) (Some v) v None (i64 0))
|
|
(= (.kind t) tok-float) (match (float-of t) (Some v) v None 0.0)
|
|
;; The box copies the bytes into the collector's heap, which is the "a
|
|
;; document owns its strings" rule this file has always had: a view into
|
|
;; the source buffer would be garbage with nothing to say so the moment
|
|
;; the caller reads the next file into it.
|
|
(= (.kind t) tok-string) (string (.text t))
|
|
(= (.kind t) tok-keyword) (keyword (.text t))
|
|
;; A symbol becomes a keyword. Nothing that reads a document this way
|
|
;; tells the two apart, and a case nobody can act on differently is a
|
|
;; case that only makes matches longer.
|
|
(= (.kind t) tok-symbol) (keyword (.text t))
|
|
|
|
(= (.kind t) tok-vec-open)
|
|
(let [items (vec-new dyn)
|
|
u (next c)]
|
|
(while (and (ok? c)
|
|
(!= (.kind u) tok-vec-close)
|
|
(!= (.kind u) tok-eof))
|
|
(push items (read-value c u))
|
|
(set u (next c)))
|
|
items)
|
|
|
|
;; A set ends on tok-map-close, because `}` is the byte that ends it. The
|
|
;; dedup is the map's own: put replaces the value of an equal key, so a
|
|
;; set with a duplicate in it never exists and #{[0 0] [0 0]} is one
|
|
;; element by structure, not by header identity.
|
|
(= (.kind t) tok-set-open)
|
|
(let [s {}
|
|
u (next c)]
|
|
(while (and (ok? c)
|
|
(!= (.kind u) tok-map-close)
|
|
(!= (.kind u) tok-eof))
|
|
(put s (read-value c u) true)
|
|
(set u (next c)))
|
|
s)
|
|
|
|
;; A map's key is a whole value, read by the same recursion as anything
|
|
;; else — :a and "a" are two keys, [0 0] can key a map, and the old
|
|
;; (Map string Value) narrowing that collapsed them is gone with the type
|
|
;; that forced it.
|
|
(= (.kind t) tok-map-open)
|
|
(let [m {}
|
|
k (next c)]
|
|
(while (and (ok? c)
|
|
(!= (.kind k) tok-map-close)
|
|
(!= (.kind k) tok-eof))
|
|
(let [key (read-value c k)
|
|
u (next c)]
|
|
(put m key (read-value c u)))
|
|
(set k (next c)))
|
|
m)
|
|
|
|
:else nil))
|
|
|
|
;; The whole document, from a byte slice. nil when the input was malformed —
|
|
;; the header says what that conflates and what to do when it matters.
|
|
(defn read [src [u8]] dyn
|
|
(let [c (cursor src)
|
|
t (next (addr c))
|
|
v (read-value (addr c) t)]
|
|
(if (ok? (addr c)) v nil)))
|
|
|
|
;; The same, from a path. The buffer is slurped against the heap, read, and
|
|
;; freed on the way out — it can be, because the document copies every byte it
|
|
;; keeps. The heap is named rather than left to the context because the caller
|
|
;; can never observe this buffer, so its tier was never the caller's to
|
|
;; choose; the document itself lands in the collector's heap wherever this is
|
|
;; called from.
|
|
;;
|
|
;; A FileError passes straight through, and that is the decision, not an
|
|
;; omission: this function has nothing to answer one with — `use-value` wants
|
|
;; a path only the caller knows, and whether a missing file is fatal or a cue
|
|
;; to write a default is the caller's policy in every program. Nothing here
|
|
;; establishes a handler, so slurp's condition reaches the caller's with both
|
|
;; restarts still armed.
|
|
(defn read-file [path string] dyn
|
|
(let [src (slurp path (heap-allocator))]
|
|
(defer (free src))
|
|
(read (as-slice src))))
|