585 lines
26 KiB
Plaintext
585 lines
26 KiB
Plaintext
;;;; An EDN tokenizer, in Flan, over a [const u8].
|
|
;;;;
|
|
;;;; This is the bottom layer of a reader. It answers one question — "what is
|
|
;;;; the next token, and where" — and it answers it without allocating
|
|
;;;; anything: every token's text is a `slice` of the input buffer, not a copy
|
|
;;;; of it.
|
|
;;;;
|
|
;;;; Two layers sit above it. `read.flan`, in this package, is the one that
|
|
;;;; exists: `(edn/read bytes)` walks this cursor and answers a dynamic
|
|
;;;; `Value`. The other, `(read-edn Enemy bytes)` emitting a parser from a
|
|
;;;; compile-time walk over a struct's fields, belongs to the compiler and is
|
|
;;;; not here; until it exists a caller writes the struct reader by hand
|
|
;;;; against this cursor, and test/programs/edn.flan is a worked example of
|
|
;;;; doing exactly that.
|
|
;;;;
|
|
;;;; ── The lifetime contract, which the type system does not state ─────
|
|
;;;;
|
|
;;;; A Token's `text` is a slice INTO the buffer the Cursor was built over.
|
|
;;;; It is ptr+len and it owns nothing. Therefore:
|
|
;;;;
|
|
;;;; * the input buffer must outlive every Token taken from it, and every
|
|
;;;; Cursor over it;
|
|
;;;; * mutating the input while tokens are live changes their text under
|
|
;;;; them, because they are views and not copies;
|
|
;;;; * a Token returned out of the function that owns the buffer is a
|
|
;;;; dangling pointer, and nothing in the language will say so.
|
|
;;;;
|
|
;;;; That is the price of not allocating, and it is written here because it is
|
|
;;;; the kind of contract that otherwise gets discovered from a corrupted
|
|
;;;; string three frames later.
|
|
;;;;
|
|
;;;; **`read.flan` does not follow this rule, deliberately.** The two layers of
|
|
;;;; this package diverge on exactly this point: a Token is a view, and a Value
|
|
;;;; owns copies of every string in it. The reason is that a view is a fine
|
|
;;;; thing for a cursor a caller is driving inside the function that holds the
|
|
;;;; buffer, and a trap for a document handed back out of one. Said the other
|
|
;;;; way: the contract above is a property of the *layer*, not of the package,
|
|
;;;; and a caller who mixes them — holding a Token out of an `(edn/read ...)`
|
|
;;;; that has returned — is on the tokenizer's terms and not the reader's.
|
|
;;;;
|
|
;;;; ── What is refused, and why ────────────────────────────────────────
|
|
;;;;
|
|
;;;; Every refusal below is a *named* one with a reason attached, reachable as
|
|
;;;; (edn/error-message code). A tokenizer that quietly skipped what it did not
|
|
;;;; understand would hand a caller a value that is not the one in the file.
|
|
;;;;
|
|
;;;; escaped strings "a\nb", "a\"b" — the important one. Unescaping needs
|
|
;;;; somewhere to put the unescaped copy, and there is no
|
|
;;;; allocator, so there is nowhere. Returning the raw
|
|
;;;; bytes including the backslash would be quietly wrong:
|
|
;;;; a caller comparing against "a\nb" would get a 4-byte
|
|
;;;; answer where it expected 3, and a caller printing it
|
|
;;;; would print a backslash. So a backslash inside a
|
|
;;;; string is an error at the byte it appears on.
|
|
;;;; tagged literals #foo {} — the tag decides the type, and dispatching on
|
|
;;;; a tag at run time is what a type-directed reader
|
|
;;;; exists to avoid.
|
|
;;;; #inst, #uuid named separately from tagged literals because they are
|
|
;;;; the two a real file is most likely to contain, and
|
|
;;;; "tagged literals are refused" would not tell a caller
|
|
;;;; that a timestamp is the thing to remove.
|
|
;;;; ratios 22/7 — there is no rational type.
|
|
;;;; metadata ^{:a 1} — it attaches to the value after it, and a
|
|
;;;; flat token stream has nowhere to attach anything.
|
|
;;;; characters \a — outside the requested subset; a char is not
|
|
;;;; a byte once anything is non-ASCII, and there is no
|
|
;;;; code point type.
|
|
;;;;
|
|
;;;; ── One place this is not EDN, on the record ────────────────────────
|
|
;;;;
|
|
;;;; `.5` is a float here. In EDN a number must begin with a digit and `.` is
|
|
;;;; a legal symbol-start byte, so strictly `.5` is the *symbol* `.5` — which
|
|
;;;; makes this a reinterpretation of a legal token and not an extension, and
|
|
;;;; therefore the kind of thing that gets written down rather than discovered.
|
|
;;;; It is this way because is-number-start runs before the symbol case and
|
|
;;;; parse-f64 accepts a leading dot; a caller who needs the symbol reading
|
|
;;;; should not be writing `.5` at all. `-`, by contrast, is a symbol, because
|
|
;;;; is-number-start requires a digit after the sign.
|
|
;;;;
|
|
;;;; ── Errors ──────────────────────────────────────────────────────────
|
|
;;;;
|
|
;;;; On the cursor, not in the return type. `next` answers a Token whose kind
|
|
;;;; is tok-error, and the cursor carries the code and the byte offset it was
|
|
;;;; found at; (edn/error-message code) turns the code into the sentence. The
|
|
;;;; offset is the point: an editor underlines a byte range, and an Option with
|
|
;;;; no position could not tell it where. An (Option Token) was the alternative
|
|
;;;; and it loses exactly that — None says something went wrong, and a second
|
|
;;;; out-parameter for the position is the same two fields with a worse shape.
|
|
;;;;
|
|
;;;; A failed cursor is poisoned: every later `next` answers the same error
|
|
;;;; token without advancing. That is what stops a caller's `while` loop from
|
|
;;;; spinning on a malformed file forever.
|
|
|
|
;; ── Token kinds ─────────────────────────────────────────────────────
|
|
;;
|
|
;; Plain i32 constants and not a `defenum`, which is the shape that wants
|
|
;; explaining. An enum here is FFI-only: `=` on an Enum value fails in emit,
|
|
;; and a keyword is not a pattern, so `match` cannot see one either. Both fixes
|
|
;; live in check.ml and emit.ml, which this lane does not touch. An i32 loses
|
|
;; the compile-time typo check on a keyword and gains a token kind a caller can
|
|
;; actually branch on, which is the whole job.
|
|
|
|
const tok-eof = 0 ; the input is exhausted; text is empty
|
|
const tok-error = 1 ; see (edn/error c) and (edn/error-message ...)
|
|
const tok-nil = 2 ; nil
|
|
const tok-bool = 3 ; true / false — text is the word
|
|
const tok-int = 4 ; text parses as i64
|
|
const tok-float = 5 ; text parses as f64
|
|
const tok-string = 6 ; text is the CONTENTS, without the quotes
|
|
const tok-keyword = 7 ; text is WITHOUT the leading colon
|
|
const tok-symbol = 8 ; text is the symbol, namespace and all
|
|
const tok-vec-open = 9 ; [
|
|
const tok-vec-close = 10 ; ]
|
|
const tok-map-open = 11 ; {
|
|
const tok-map-close = 12 ; }
|
|
const tok-list-open = 13 ; (
|
|
const tok-list-close = 14 ; )
|
|
|
|
;; #{ — and there is deliberately no tok-set-close. A set closes on `}`, the
|
|
;; same byte a map closes on, and a closer that answered a different kind
|
|
;; depending on what was open would be asking the caller to track a thing the
|
|
;; opener already told it. Appended rather than slotted in beside the other
|
|
;; openers because these are the numbers a caller branches on.
|
|
const tok-set-open = 15 ; #{
|
|
|
|
;; ── Error codes ─────────────────────────────────────────────────────
|
|
|
|
const err-none = 0
|
|
const err-unexpected-byte = 1
|
|
const err-unterminated = 2
|
|
const err-string-escape = 3 ; refusal
|
|
const err-tagged = 4 ; refusal
|
|
const err-inst = 5 ; refusal
|
|
const err-uuid = 6 ; refusal
|
|
const err-metadata = 7 ; refusal
|
|
const err-ratio = 8 ; refusal
|
|
const err-char = 9 ; refusal
|
|
const err-bad-number = 10
|
|
const err-empty-keyword = 11
|
|
const err-unbalanced = 12 ; a closer that does not match what is open
|
|
const err-too-deep = 13
|
|
const err-unexpected-token = 14 ; raised by a caller, not by the tokenizer
|
|
|
|
;; err-set was 4 and is gone rather than kept with a new message. A code that
|
|
;; nothing raises is a code a caller can still test for and never see, and
|
|
;; renumbering the rest is free: these are named constants, and the only place
|
|
;; a number appears is in this list.
|
|
|
|
;; How deep a nesting the balance check can follow. A fixed array in the
|
|
;; Cursor and not a growable stack, because there is no allocator; 32 is far
|
|
;; past anything a hand-written config file contains, and past it the answer is
|
|
;; err-too-deep rather than a silently unchecked closer.
|
|
const max-depth = 32
|
|
|
|
;; ── The types ───────────────────────────────────────────────────────
|
|
|
|
;; `text` is a slice of the Cursor's `src`. Read the lifetime contract at the
|
|
;; top of this file before storing one anywhere.
|
|
;;
|
|
;; `pos` is the offset of the token's first byte in the ORIGINAL buffer — of
|
|
;; the opening quote for a string, of the colon for a keyword — so it stays a
|
|
;; usable underline position even though `text` is narrower than the token.
|
|
struct Token(kind: i32, text: [const u8], pos: i32)
|
|
|
|
;; The cursor owns no storage either: `src` is the caller's buffer.
|
|
;;
|
|
;; `open` is the stack of delimiters still open, holding the tok-*-close kind
|
|
;; each one is waiting for. Balance is checked in `next` itself rather than
|
|
;; left to a parser, because `[1 2}` is malformed in a way only the tokenizer
|
|
;; has the position for.
|
|
struct Cursor
|
|
src: [const u8]
|
|
pos: i32
|
|
err: i32
|
|
err-pos: i32
|
|
open: [max-depth i32]
|
|
depth: i32
|
|
|
|
;; ── Construction ────────────────────────────────────────────────────
|
|
|
|
fn cursor(src: [const u8]) -> Cursor
|
|
Cursor{.src src .pos 0 .err err-none .err-pos 0 .depth 0}
|
|
|
|
fn is-ok(c: Ptr(Cursor)) -> bool = c.err == err-none
|
|
|
|
fn error(c: Ptr(Cursor)) -> i32 = c.err
|
|
|
|
fn error-pos(c: Ptr(Cursor)) -> i32 = c.err-pos
|
|
|
|
;; Each refusal names itself and says why, so a file that uses one fails with
|
|
;; the sentence explaining what to do about it rather than with a code.
|
|
fn error-message(code: i32) -> str
|
|
if code == err-none
|
|
"no error"
|
|
elif code == err-unexpected-byte
|
|
"unexpected byte: not the start of any EDN value"
|
|
elif code == err-unterminated
|
|
"unterminated string: end of input before the closing quote"
|
|
elif code == err-string-escape
|
|
"escaped strings are refused: unescaping needs a copy of the bytes, and there is no allocator to put one in"
|
|
elif code == err-tagged
|
|
"tagged literals #tag are refused: the tag would pick the type at run time, which is what a type-directed reader exists to avoid"
|
|
elif code == err-inst
|
|
"#inst is refused: it is a tagged literal, and there is no timestamp type to read it into"
|
|
elif code == err-uuid
|
|
"#uuid is refused: it is a tagged literal, and there is no uuid type to read it into"
|
|
elif code == err-metadata
|
|
"metadata ^ is refused: it attaches to the value after it, and a flat token stream has nowhere to attach it"
|
|
elif code == err-ratio
|
|
"ratios are refused: there is no rational type, and rounding one to a float would change the value"
|
|
elif code == err-char
|
|
"character literals are refused: a character is not a byte once it is not ASCII, and there is no code point type"
|
|
elif code == err-bad-number
|
|
"not a number: the token starts like one but does not parse as an integer or a float"
|
|
elif code == err-empty-keyword
|
|
"empty keyword: a colon with no name after it"
|
|
elif code == err-unbalanced
|
|
"unbalanced: this closing delimiter does not match the one that is open"
|
|
elif code == err-too-deep
|
|
"nesting is too deep: the balance stack is a fixed array and it is full"
|
|
elif code == err-unexpected-token
|
|
"unexpected token: not the kind the caller was reading"
|
|
else
|
|
"unknown error code"
|
|
|
|
;; Marks the cursor failed. Public, because a caller's own reader needs to
|
|
;; report "expected an integer here" with a position the same way this file
|
|
;; does, and there is nowhere else the position would come from.
|
|
;;
|
|
;; The first failure wins: a later one would overwrite the offset that
|
|
;; explains the file, with an offset that is merely downstream of it.
|
|
fn fail(c: Ptr(Cursor), code: i32, pos: i32) -> ()
|
|
if c.err == err-none
|
|
c.err = code
|
|
c.err-pos = pos
|
|
|
|
;; ── Byte classes ────────────────────────────────────────────────────
|
|
|
|
;; A comma is whitespace in EDN, which is the rule most hand-written readers
|
|
;; get wrong: {:a 1, :b 2} is one map and the comma is not a token.
|
|
fn- is-ws(b: u8) -> bool = is-space(b) or b == \,
|
|
|
|
;; Everything that ends an unquoted token. Note `;` is here: `[1;c` has the
|
|
;; comment start immediately after the 1, with no space, and a scanner that
|
|
;; only stopped on whitespace and brackets would read "1;c" as one number.
|
|
fn- is-delim(b: u8) -> bool
|
|
is-ws(b) or b == \( or b == \) or b == \[ or b == \] or b == \{ or b == \} or b == \" or b == \;
|
|
|
|
fn- is-alpha(b: u8) -> bool = (b >= \a and b <= \z) or (b >= \A and b <= \Z)
|
|
|
|
;; What EDN lets a symbol begin with. It matters that this is a list and not
|
|
;; "anything that is not a delimiter": without it every stray byte becomes a
|
|
;; one-character symbol, and `@` or a backtick — a Clojure reader macro, not
|
|
;; EDN — reads as a name instead of being reported at the byte it is on.
|
|
fn- is-sym-start(b: u8) -> bool
|
|
is-alpha(b) or b == \. or b == \* or b == \+ or b == \! or b == \- or b == \_ or b == \? or b == \$ or b == \% or b == \& or b == \= or b == \< or b == \> or b == \/
|
|
|
|
;; ── Internal helpers ────────────────────────────────────────────────
|
|
;;
|
|
;; "Internal" by intent and not by enforcement: a package has no visibility
|
|
;; yet, so edn/scan-atom and edn/push-open are as callable as edn/next is.
|
|
;; Nothing below is part of the API and none of it will keep its shape.
|
|
|
|
fn- is-at-end(c: Ptr(Cursor)) -> bool = c.pos >= length(c.src)
|
|
|
|
;; An empty slice of src, positioned at p. Used for the tokens that have no
|
|
;; text of their own — eof, error, and every delimiter. It is still a slice of
|
|
;; the input rather than a slice of nothing, so `text` has one meaning for all
|
|
;; token kinds.
|
|
fn- empty-at(c: Ptr(Cursor), p: i32) -> [const u8] = slice(c.src, p, p)
|
|
|
|
fn- token(c: Ptr(Cursor), kind: i32, lo: i32, hi: i32, p: i32) -> Token
|
|
Token{.kind kind .text slice(c.src, lo, hi) .pos p}
|
|
|
|
fn- error-token(c: Ptr(Cursor)) -> Token
|
|
Token{.kind tok-error .text empty-at(c, c.err-pos) .pos c.err-pos}
|
|
|
|
;; Whitespace, commas, and `;` comments, which run to the newline or to the end
|
|
;; of input — a comment on the last line of a file with no trailing newline is
|
|
;; the case that decides whether the loop tests the length before the byte.
|
|
fn- skip-trivia(c: Ptr(Cursor)) -> ()
|
|
while not is-at-end(c)
|
|
let b = c.src[c.pos]
|
|
if is-ws(b)
|
|
c.pos += 1
|
|
elif b == \;
|
|
while not is-at-end(c) and c.src[c.pos] != \newline
|
|
c.pos += 1
|
|
;; The newline itself, if there is one. If there is not, is-at-end is
|
|
;; already true and the outer loop stops.
|
|
if not is-at-end(c)
|
|
c.pos += 1
|
|
else
|
|
return
|
|
|
|
;; The end of the unquoted token starting at lo: the first delimiter, or the
|
|
;; end of input.
|
|
fn scan-atom(c: Ptr(Cursor), lo: i32) -> i32
|
|
let i = lo
|
|
while i < length(c.src) and not is-delim(c.src[i])
|
|
i += 1
|
|
i
|
|
|
|
fn push-open(c: Ptr(Cursor), closer: i32, p: i32) -> bool
|
|
if c.depth >= max-depth
|
|
fail(c, err-too-deep, p)
|
|
return false
|
|
c.open[c.depth] = closer
|
|
c.depth += 1
|
|
true
|
|
|
|
fn- pop-close(c: Ptr(Cursor), closer: i32, p: i32) -> bool
|
|
if c.depth == 0 or c.open[c.depth - 1] != closer
|
|
fail(c, err-unbalanced, p)
|
|
return false
|
|
c.depth -= 1
|
|
true
|
|
|
|
;; ── Numbers ─────────────────────────────────────────────────────────
|
|
|
|
;; A token starting with a digit, or with a sign or a dot followed by one.
|
|
;; `-` alone is a symbol in EDN and stays one here.
|
|
fn- is-number-start(c: Ptr(Cursor), i: i32) -> bool
|
|
let s = c.src
|
|
if i >= length(s)
|
|
return false
|
|
if is-digit(s[i])
|
|
return true
|
|
(s[i] == \- or s[i] == \+ or s[i] == \.) and i + 1 < length(s) and is-digit(s[i + 1])
|
|
|
|
fn- read-number(c: Ptr(Cursor), lo: i32) -> Token
|
|
let hi = scan-atom(c, lo)
|
|
c.pos = hi
|
|
let text = slice(c.src, lo, hi)
|
|
;; A ratio is caught here and not by a "contains a slash" rule over every
|
|
;; token, because a slash is perfectly ordinary in a symbol: foo/bar is a
|
|
;; namespaced name and must stay one.
|
|
if match(index-of(text, \/), Some(_), true, None, false)
|
|
fail(c, err-ratio, lo)
|
|
return error-token(c)
|
|
if match(parse-i64(text), Some(_), true, None, false)
|
|
return token(c, tok-int, lo, hi, lo)
|
|
if match(parse-f64(text), Some(_), true, None, false)
|
|
return token(c, tok-float, lo, hi, lo)
|
|
;; "12x", and also EDN's own 1N and 1M, which have no type here.
|
|
fail(c, err-bad-number, lo)
|
|
error-token(c)
|
|
|
|
;; ── Strings ─────────────────────────────────────────────────────────
|
|
|
|
;; The whole reason this is not three lines. `text` is the interior, between
|
|
;; the quotes — so the bytes are usable directly — but `pos` is the opening
|
|
;; quote, so an editor underlines the literal and not its contents.
|
|
;;
|
|
;; A backslash anywhere inside is the refusal, reported at the backslash
|
|
;; rather than at the start of the string, because the backslash is what has
|
|
;; to be removed.
|
|
fn- read-string(c: Ptr(Cursor), lo: i32) -> Token
|
|
let i = lo + 1
|
|
s = c.src
|
|
while i < length(s)
|
|
let b = s[i]
|
|
if b == \\
|
|
c.pos = i
|
|
fail(c, err-string-escape, i)
|
|
return error-token(c)
|
|
if b == \"
|
|
c.pos = i + 1
|
|
return token(c, tok-string, lo + 1, i, lo)
|
|
i += 1
|
|
;; Ran off the end with the string still open. Reported at the opening
|
|
;; quote: that is the byte a caller has to look at, not the end of the file.
|
|
c.pos = i
|
|
fail(c, err-unterminated, lo)
|
|
error-token(c)
|
|
|
|
;; ── The dispatch ────────────────────────────────────────────────────
|
|
|
|
;; The one call a caller makes. Advances the cursor past the token it returns.
|
|
;;
|
|
;; A cursor that has already failed keeps answering the same error token and
|
|
;; does not advance, so `(while (!= (.kind t) tok-eof) ...)` terminates on a
|
|
;; malformed file instead of spinning.
|
|
fn next(c: Ptr(Cursor)) -> Token
|
|
if not is-ok(c)
|
|
return error-token(c)
|
|
skip-trivia(c)
|
|
if is-at-end(c)
|
|
;; Something still open at the end of input is malformed, and the position
|
|
;; that helps is the end — the file stopped, not the value.
|
|
if c.depth > 0
|
|
fail(c, err-unbalanced, c.pos)
|
|
return error-token(c)
|
|
return Token{.kind tok-eof .text empty-at(c, c.pos) .pos c.pos}
|
|
let s = c.src
|
|
lo = c.pos
|
|
b = s[lo]
|
|
;; ── Delimiters, each of which moves the balance stack ──────────
|
|
if b == \[
|
|
c.pos = lo + 1
|
|
if push-open(c, tok-vec-close, lo)
|
|
token(c, tok-vec-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \]
|
|
c.pos = lo + 1
|
|
if pop-close(c, tok-vec-close, lo)
|
|
token(c, tok-vec-close, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \{
|
|
c.pos = lo + 1
|
|
if push-open(c, tok-map-close, lo)
|
|
token(c, tok-map-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \}
|
|
c.pos = lo + 1
|
|
if pop-close(c, tok-map-close, lo)
|
|
token(c, tok-map-close, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \(
|
|
c.pos = lo + 1
|
|
if push-open(c, tok-list-close, lo)
|
|
token(c, tok-list-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \)
|
|
c.pos = lo + 1
|
|
if pop-close(c, tok-list-close, lo)
|
|
token(c, tok-list-close, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \"
|
|
read-string(c, lo)
|
|
;; ── Keywords ───────────────────────────────────────────────────
|
|
elif b == \:
|
|
let hi = scan-atom(c, lo + 1)
|
|
c.pos = hi
|
|
if hi == lo + 1
|
|
fail(c, err-empty-keyword, lo)
|
|
error-token(c)
|
|
;; text drops the colon: a caller comparing against "name" should not
|
|
;; have to write ":name", and the compiler-side reader will want the
|
|
;; bare name to match a field against.
|
|
else
|
|
token(c, tok-keyword, lo + 1, hi, lo)
|
|
;; ── The refusals that have their own byte ──────────────────────
|
|
elif b == \^
|
|
c.pos = lo + 1
|
|
fail(c, err-metadata, lo)
|
|
error-token(c)
|
|
elif b == \\
|
|
c.pos = lo + 1
|
|
fail(c, err-char, lo)
|
|
error-token(c)
|
|
elif b == \#
|
|
let hi = scan-atom(c, lo + 1)
|
|
c.pos = hi
|
|
;; #{ — the brace is a delimiter, so scan-atom stopped before it and
|
|
;; hi is lo+1; the brace itself is consumed here, which is why the
|
|
;; position moves to lo+2 and not to hi.
|
|
;;
|
|
;; The closer pushed is tok-map-close, because the byte that closes a
|
|
;; set is `}`. That is not a compromise: the balance stack holds the
|
|
;; *closing kind still owed*, and a set and a map owe the same one.
|
|
if lo + 1 < length(s) and s[lo + 1] == \{
|
|
c.pos = lo + 2
|
|
if push-open(c, tok-map-close, lo)
|
|
token(c, tok-set-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif is-bytes-equal(slice(s, lo + 1, hi), bytes-view("inst"))
|
|
fail(c, err-inst, lo)
|
|
error-token(c)
|
|
elif is-bytes-equal(slice(s, lo + 1, hi), bytes-view("uuid"))
|
|
fail(c, err-uuid, lo)
|
|
error-token(c)
|
|
else
|
|
fail(c, err-tagged, lo)
|
|
error-token(c)
|
|
;; ── Numbers, then everything else as a symbol ──────────────────
|
|
elif is-number-start(c, lo)
|
|
read-number(c, lo)
|
|
else
|
|
let hi = scan-atom(c, lo)
|
|
;; Two ways to get here without a symbol. `hi = lo` would be a
|
|
;; zero-length atom and an infinite loop; a byte that is not a symbol
|
|
;; start is `@` or a backtick, which are Clojure and not EDN. Both
|
|
;; advance one byte before failing, so the position is the offending
|
|
;; byte and the loop cannot spin on it.
|
|
if hi == lo or not is-sym-start(b)
|
|
c.pos = lo + 1
|
|
fail(c, err-unexpected-byte, lo)
|
|
return error-token(c)
|
|
c.pos = hi
|
|
let text = slice(s, lo, hi)
|
|
if is-bytes-equal(text, bytes-view("nil"))
|
|
token(c, tok-nil, lo, hi, lo)
|
|
elif is-bytes-equal(text, bytes-view("true"))
|
|
token(c, tok-bool, lo, hi, lo)
|
|
elif is-bytes-equal(text, bytes-view("false"))
|
|
token(c, tok-bool, lo, hi, lo)
|
|
else
|
|
token(c, tok-symbol, lo, hi, lo)
|
|
|
|
;; ── Reading values out of a token ───────────────────────────────────
|
|
;;
|
|
;; Each checks the kind first. None for the wrong kind rather than a parse of
|
|
;; whatever bytes happened to be there, which is the same reason parse-i64 is
|
|
;; Flan and not strtoll.
|
|
|
|
fn int-of(t: Token) -> Option(i64)
|
|
if t.kind == tok-int then parse-i64(t.text) else None
|
|
|
|
;; Accepts an integer token too: 1 and 1.0 are the same number, and a config
|
|
;; file that writes `:speed 2` for an f32 field is not making a mistake.
|
|
fn float-of(t: Token) -> Option(f64)
|
|
if t.kind == tok-float or t.kind == tok-int then parse-f64(t.text) else None
|
|
|
|
fn bool-of(t: Token) -> Option(bool)
|
|
if t.kind == tok-bool
|
|
Some(is-bytes-equal(t.text, bytes-view("true")))
|
|
else
|
|
None
|
|
|
|
fn- is-text-equal(t: Token, s: str) -> bool
|
|
is-bytes-equal(t.text, bytes-view(s))
|
|
|
|
;; A keyword whose name is s. The leading colon is not part of `text`, so this
|
|
;; is written (is-keyword-equal t "hp") and not (is-keyword-equal t ":hp").
|
|
fn is-keyword-equal(t: Token, s: str) -> bool
|
|
t.kind == tok-keyword and is-bytes-equal(t.text, bytes-view(s))
|
|
|
|
;; ── Reading past a value ────────────────────────────────────────────
|
|
|
|
;; Consumes exactly one value — a scalar, or a whole collection with everything
|
|
;; nested inside it. This is what a struct reader calls on a map key it does
|
|
;; not know, so an extra field in a data file is ignored rather than fatal.
|
|
;;
|
|
;; Iterative on the cursor's own balance depth and not recursive: the depth is
|
|
;; already tracked, and a recursive skip would put the nesting on the C stack
|
|
;; where a deep file is a crash rather than err-too-deep.
|
|
;;
|
|
;; Being written against the depth and not against the kinds is also why sets
|
|
;; cost this function nothing: `#{` pushes in `next` like every other opener,
|
|
;; so a set was already a collection here before it was one anywhere else. An
|
|
;; arm per collection kind would have been a second list to forget to add to.
|
|
fn skip-value(c: Ptr(Cursor)) -> bool
|
|
let start = c.depth
|
|
t = next(c)
|
|
if not is-ok(c)
|
|
return false
|
|
if t.kind == tok-eof
|
|
fail(c, err-unexpected-token, t.pos)
|
|
return false
|
|
;; A scalar is one token and we are done. A closer here is a value ending
|
|
;; that never began, which pop-close has already reported.
|
|
if c.depth <= start
|
|
return true
|
|
while c.depth > start
|
|
let u = next(c)
|
|
if not is-ok(c)
|
|
return false
|
|
if u.kind == tok-eof
|
|
;; next already failed on the open depth; this is belt and braces.
|
|
fail(c, err-unbalanced, u.pos)
|
|
return false
|
|
true
|
|
|
|
;; ── Expecting a kind ────────────────────────────────────────────────
|
|
|
|
;; The shape a hand-written reader is built out of: take the next token, and if
|
|
;; it is not the kind wanted, fail the cursor at that token's position with a
|
|
;; reason. The returned token is the error token in that case, so a caller that
|
|
;; forgets to test is-ok still does not read a value out of the wrong kind —
|
|
;; int-of and friends answer None for tok-error.
|
|
fn expect(c: Ptr(Cursor), kind: i32) -> Token
|
|
let t = next(c)
|
|
if is-ok(c) and t.kind != kind
|
|
fail(c, err-unexpected-token, t.pos)
|
|
return error-token(c)
|
|
t
|