805 lines
36 KiB
Plaintext
805 lines
36 KiB
Plaintext
;;;; A JSON tokenizer, in Flan, over a [const u8].
|
|
;;;;
|
|
;;;; The shape is vendor/edn's, deliberately: a Cursor over a caller's buffer,
|
|
;;;; one `next` that answers a Token, errors accumulated on the cursor with the
|
|
;;;; byte offset they were found at, and every refusal named and given a reason
|
|
;;;; reachable as json/error-message(code). Read vendor/edn/edn.fln first if
|
|
;;;; you have not; everything that file argues for is argued for there and only
|
|
;;;; the differences are argued for here.
|
|
;;;;
|
|
;;;; The differences are the whole of what is interesting, so they are first.
|
|
;;;;
|
|
;;;; ── Where this DIVERGES from vendor/edn: strings are copied ──────────
|
|
;;;;
|
|
;;;; edn's tokens are views and the package never allocates. That is a real
|
|
;;;; decision there and it does not carry over, because it forced a second one:
|
|
;;;; edn refuses escaped strings outright, since unescaping "a\nb" needs three
|
|
;;;; bytes that exist nowhere in the input and there was no allocator to put
|
|
;;;; them in. JSON without escapes is not JSON — \n and \uXXXX are how the
|
|
;;;; format writes a newline and a non-ASCII character at all — so this package
|
|
;;;; unescapes, and to unescape it allocates.
|
|
;;;;
|
|
;;;; The split follows Odin's, which faced the same question and answered it in
|
|
;;;; two files: core/encoding/json/tokenizer.odin allocates nothing and hands
|
|
;;;; back the raw literal including its quotes, and parser.odin's
|
|
;;;; unquote_string does the copy against an allocator. So here too:
|
|
;;;;
|
|
;;;; * a Token's `text` is a slice INTO the buffer, exactly as edn's is, and
|
|
;;;; for a string token it is the RAW interior — backslashes and all;
|
|
;;;; * json/string-of(t) is the one call that allocates. It answers a
|
|
;;;; string whose bytes are a fresh copy in the context's allocator, with
|
|
;;;; every escape resolved.
|
|
;;;;
|
|
;;;; string-of copies even when there is no escape to resolve, where handing
|
|
;;;; back the view would have been free. That is Odin's call too — parser.odin
|
|
;;;; clones in the no-escape branch (:388) and types.odin's destroy_value frees
|
|
;;;; every String it finds (:96) — and the reason is that the alternative is a
|
|
;;;; value whose lifetime depends on which bytes happened to be in it. A
|
|
;;;; document read into an arena and then released by one free-all has to take
|
|
;;;; its strings with it; one that sometimes aliased the source buffer would
|
|
;;;; survive the free-all in the cases that had no escapes and not in the
|
|
;;;; others, which is the kind of contract nobody can hold in their head.
|
|
;;;;
|
|
;;;; Note what that buys against vendor/edn/read.fln's dyn reader, which
|
|
;;;; settled the same question a different way: it takes no allocator at all,
|
|
;;;; because every string it produces is boxed onto the collector's heap on
|
|
;;;; the spot — read-value's "a document owns its strings" rule. This
|
|
;;;; package's split is the one to take when there is a context allocator to
|
|
;;;; put the copy in and a caller who may not want one for every string in the
|
|
;;;; document; the dyn reader's is the one to take when there is no allocator
|
|
;;;; in the picture at all. Either way, a Token's raw `text` is never what
|
|
;;;; outlives the read — only what string-of or the box copies out of it does.
|
|
;;;;
|
|
;;;; ── The allocator ───────────────────────────────────────────────────
|
|
;;;;
|
|
;;;; string-of names no allocator and takes no parameter. spec-memory.md puts
|
|
;;;; the allocator in the calling convention, so the vec-new(u8) inside it
|
|
;;;; takes the context's, and with-allocator(frame, json/string-of(t)) is the
|
|
;;;; whole of "reading against an arena". An explicit allocator at the
|
|
;;;; construction site would override that, which is how this would take one if
|
|
;;;; the idiom could not express it. It can.
|
|
;;;;
|
|
;;;; ── Strict JSON, on the record ──────────────────────────────────────
|
|
;;;;
|
|
;;;; Odin's DEFAULT_SPECIFICATION is JSON5, and its tokenizer carries a `spec`
|
|
;;;; field to switch between three dialects. This one has no such field and
|
|
;;;; reads strict JSON only. A dialect parameter is a run-time branch in every
|
|
;;;; byte class, and the caller who wants JSON5 wants a different file.
|
|
;;;;
|
|
;;;; So this is STRICTER than its reference, and the strictness is where the
|
|
;;;; refusals come from. Each one below is a thing a hand-written file really
|
|
;;;; contains, named separately so the message says what to remove:
|
|
;;;;
|
|
;;;; comments // and /* — legal in JSON5, not in JSON. The single
|
|
;;;; most common reason a config file fails to parse.
|
|
;;;; single quotes 'x' — legal in JSON5.
|
|
;;;; +1 — a leading plus is JSON5's.
|
|
;;;; .5 and 1. — JSON5 allows a bare leading or trailing
|
|
;;;; decimal point. Worth contrasting with edn.fln, which
|
|
;;;; goes the other way and reads `.5` as a float on
|
|
;;;; purpose; here it is an error, and named.
|
|
;;;; 0x1f — hexadecimal is JSON5's.
|
|
;;;; 01 — JSON's number grammar allows a leading
|
|
;;;; zero only when the zero is the whole integer part.
|
|
;;;; NaN, Infinity — JSON5 has both and JSON has neither, and
|
|
;;;; there is no spelling of either in the format.
|
|
;;;; bare words {name: 1} — an unquoted object key is JSON5's. Named
|
|
;;;; rather than folded into "unexpected byte" because the
|
|
;;;; byte is a perfectly ordinary letter and the message
|
|
;;;; has to say what is wrong with it.
|
|
;;;; \v \0 \x41 \' — JSON has exactly nine escapes and these
|
|
;;;; are not among them.
|
|
;;;; a raw control byte inside a string — JSON requires anything below 0x20
|
|
;;;; to be escaped. A literal tab inside a literal is the
|
|
;;;; one a file written by hand actually hits.
|
|
;;;;
|
|
;;;; Whitespace is space, tab, newline and carriage return, which is exactly
|
|
;;;; JSON's set and exactly what the prelude's is-space answers, so there is no
|
|
;;;; is-ws of its own here. A vertical tab or a form feed between values is an
|
|
;;;; unexpected byte, which is correct and worth knowing before it surprises
|
|
;;;; someone: JSON5 allows both.
|
|
;;;;
|
|
;;;; ── Separators are tokens ───────────────────────────────────────────
|
|
;;;;
|
|
;;;; `,` and `:` come back as tok-comma and tok-colon rather than being skipped
|
|
;;;; as trivia, which is the other visible difference from edn — where a comma
|
|
;;;; IS whitespace and is not a token at all. JSON's commas are grammar: [1 2]
|
|
;;;; is malformed and a tokenizer that swallowed the separator would hand a
|
|
;;;; reader no way to notice. Placement is the reader's to check, the way
|
|
;;;; balance is not: the cursor tracks what is open because it has the position
|
|
;;;; for the closer, and it does not track what a comma may follow because that
|
|
;;;; is a grammar with a stack of its own.
|
|
;;;;
|
|
;;;; ── There is no json/read-file, and that is not an oversight ────────
|
|
;;;;
|
|
;;;; vendor/edn grew one and this did not, because the thing that made it safe
|
|
;;;; there is the thing this package does not have. `edn/read-file` can slurp a
|
|
;;;; buffer, read it, and free the buffer inside the one call only because
|
|
;;;; `edn/read` answers a Value whose every string is a copy — the source is
|
|
;;;; dead the moment the read returns. This package's whole surface is the
|
|
;;;; cursor, and a Token's `text` is a slice INTO the caller's buffer, which
|
|
;;;; the divergence section at the top of this file argues for at length. A
|
|
;;;; read-file here would hand back a cursor over memory it had just released.
|
|
;;;;
|
|
;;;; So the prerequisite is not two lines, it is a `json/read` answering a
|
|
;;;; self-contained document — and there is no Value type here to answer with.
|
|
;;;; Write that first and read-file follows it for free; until then the caller
|
|
;;;; holds the buffer, which is what the cursor's contract already says. What
|
|
;;;; DOES carry over is the prelude's or-else and is-some: string-of, int-of,
|
|
;;;; float-of and bool-of all answer an Option and all take them.
|
|
;;;;
|
|
;;;; ── Errors ──────────────────────────────────────────────────────────
|
|
;;;;
|
|
;;;; On the cursor, not in the return type — edn's argument, unchanged: an
|
|
;;;; editor underlines a byte range and an Option has nowhere to put one. A
|
|
;;;; failed cursor is poisoned and keeps answering the same error token without
|
|
;;;; advancing, so a `while` over a malformed document terminates.
|
|
;;;;
|
|
;;;; Escapes are validated where they are SCANNED and not where they are
|
|
;;;; unescaped. read-string walks every byte of the literal already, so it is
|
|
;;;; the only place that still knows the offset of the offending backslash;
|
|
;;;; string-of works over a literal the cursor has already accepted and cannot
|
|
;;;; fail on its contents. That is also Odin's split — its scan_escape runs in
|
|
;;;; get_token — and it is why string-of is an (Option str) on the token's
|
|
;;;; kind alone and not on its bytes.
|
|
|
|
;; ── Token kinds ─────────────────────────────────────────────────────
|
|
;;
|
|
;; Plain i32 constants and not an `enum`, for the reason vendor/edn gives:
|
|
;; `=` on an Enum value fails in emit and a keyword is not a pattern, so an
|
|
;; enum here would be a token kind a caller could not branch on.
|
|
|
|
const tok-eof = 0 ; the input is exhausted; text is empty
|
|
const tok-error = 1 ; see json/error(c) and json/error-message(...)
|
|
const tok-null = 2 ; null
|
|
const tok-bool = 3 ; true / false — text is the word
|
|
const tok-int = 4 ; a number with no fraction and no exponent
|
|
const tok-float = 5 ; a number with either
|
|
const tok-string = 6 ; text is the RAW interior, without the quotes
|
|
const tok-array-open = 7 ; [
|
|
const tok-array-close = 8 ; ]
|
|
const tok-object-open = 9 ; {
|
|
const tok-object-close = 10 ; }
|
|
const tok-colon = 11 ; :
|
|
const tok-comma = 12 ; ,
|
|
|
|
;; ── Error codes ─────────────────────────────────────────────────────
|
|
|
|
const err-none = 0
|
|
const err-unexpected-byte = 1
|
|
const err-unterminated = 2
|
|
const err-control-char = 3 ; a raw byte below 0x20 inside a string
|
|
const err-bad-escape = 4 ; refusal: not one of JSON's nine
|
|
const err-bad-unicode = 5 ; \u without four hex digits after it
|
|
const err-lone-surrogate = 6
|
|
const err-comment = 7 ; refusal
|
|
const err-single-quote = 8 ; refusal
|
|
const err-leading-plus = 9 ; refusal
|
|
const err-leading-dot = 10 ; refusal
|
|
const err-trailing-dot = 11 ; refusal
|
|
const err-hex-number = 12 ; refusal
|
|
const err-leading-zero = 13 ; refusal
|
|
const err-nan-inf = 14 ; refusal
|
|
const err-bare-word = 15 ; refusal
|
|
const err-bad-number = 16
|
|
const err-unbalanced = 17 ; a closer that does not match what is open
|
|
const err-too-deep = 18
|
|
const err-unexpected-token = 19 ; raised by a caller, not by the tokenizer
|
|
const err-trailing-comma = 20 ; raised by a caller; refusal
|
|
|
|
;; How deep a nesting the balance check can follow. A fixed array in the Cursor
|
|
;; and not a growable stack: the cursor is the half of this package that does
|
|
;; not allocate, and keeping it that way is what lets a caller tokenize with no
|
|
;; allocator bound at all. 32 is past anything hand-written, and past it the
|
|
;; answer is err-too-deep rather than a silently unchecked closer.
|
|
const max-depth = 32
|
|
|
|
;; ── The types ───────────────────────────────────────────────────────
|
|
|
|
;; `text` is a slice of the Cursor's `src`, and for tok-string it is the raw
|
|
;; interior: the bytes between the quotes, escapes unresolved. (string-of t) is
|
|
;; what turns that into text you can keep.
|
|
;;
|
|
;; `pos` is the offset of the token's first byte in the ORIGINAL buffer — of
|
|
;; the opening quote for a string, not of its contents — so it stays a usable
|
|
;; underline position even where `text` is narrower than the token.
|
|
struct Token(kind: i32, text: [const u8], pos: i32)
|
|
|
|
;; The cursor owns no storage: `src` is the caller's buffer.
|
|
;;
|
|
;; `open` is the stack of delimiters still open, holding the tok-*-close kind
|
|
;; each one waits for, so that {"a": [1} fails at the brace with a position
|
|
;; rather than confusing a reader two levels up.
|
|
struct Cursor
|
|
src: [const u8]
|
|
pos: i32
|
|
err: i32
|
|
err-pos: i32
|
|
open: [max-depth i32]
|
|
depth: i32
|
|
|
|
;; ── Construction ────────────────────────────────────────────────────
|
|
|
|
fn cursor(src: [const u8]) -> Cursor
|
|
Cursor{.src src .pos 0 .err err-none .err-pos 0 .depth 0}
|
|
|
|
fn is-ok(c: Ptr(Cursor)) -> bool = c.err == err-none
|
|
|
|
fn error(c: Ptr(Cursor)) -> i32 = c.err
|
|
|
|
fn error-pos(c: Ptr(Cursor)) -> i32 = c.err-pos
|
|
|
|
;; Each refusal names itself and says why, so a document that uses one fails
|
|
;; with the sentence explaining what to do about it rather than with a code.
|
|
;; Most of these say "JSON5" somewhere, because most of them are a thing that
|
|
;; is legal in the dialect the file was probably written for.
|
|
fn error-message(code: i32) -> str
|
|
if code == err-none
|
|
"no error"
|
|
elif code == err-unexpected-byte
|
|
"unexpected byte: not the start of any JSON value"
|
|
elif code == err-unterminated
|
|
"unterminated string: end of input before the closing quote"
|
|
elif code == err-control-char
|
|
"a raw control byte inside a string: JSON requires anything below 0x20 to be written as an escape"
|
|
elif code == err-bad-escape
|
|
"unknown escape: JSON has \\\" \\\\ \\/ \\b \\f \\n \\r \\t and \\uXXXX, and nothing else"
|
|
elif code == err-bad-unicode
|
|
"bad \\u escape: four hexadecimal digits have to follow it"
|
|
elif code == err-lone-surrogate
|
|
"lone surrogate: \\uD800-\\uDFFF is half of a pair and encodes no character on its own"
|
|
elif code == err-comment
|
|
"comments are refused: // and /* are JSON5, and JSON has no comment syntax"
|
|
elif code == err-single-quote
|
|
"single-quoted strings are refused: they are JSON5, and JSON quotes with \""
|
|
elif code == err-leading-plus
|
|
"a leading + is refused: it is JSON5, and JSON writes a positive number without a sign"
|
|
elif code == err-leading-dot
|
|
"a number cannot begin with a decimal point: write 0.5 rather than .5"
|
|
elif code == err-trailing-dot
|
|
"a number cannot end with a decimal point: digits have to follow it"
|
|
elif code == err-hex-number
|
|
"hexadecimal numbers are refused: 0x is JSON5, and JSON has decimal only"
|
|
elif code == err-leading-zero
|
|
"a leading zero is refused: JSON allows 0 as a whole integer part and nothing in front of another digit"
|
|
elif code == err-nan-inf
|
|
"NaN and Infinity are refused: they are JSON5, and JSON has no spelling for either"
|
|
elif code == err-bare-word
|
|
"an unquoted name is refused: bare keys and identifiers are JSON5, and JSON quotes every string"
|
|
elif code == err-bad-number
|
|
"not a number: the token starts like one but does not match JSON's number grammar"
|
|
elif code == err-unbalanced
|
|
"unbalanced: this closing delimiter does not match the one that is open"
|
|
elif code == err-too-deep
|
|
"nesting is too deep: the balance stack is a fixed array and it is full"
|
|
elif code == err-unexpected-token
|
|
"unexpected token: not the kind the caller was reading"
|
|
elif code == err-trailing-comma
|
|
"trailing comma: it is JSON5, and JSON has no separator before a closing bracket"
|
|
else
|
|
"unknown error code"
|
|
|
|
;; Marks the cursor failed. Public, because a caller's own reader has to report
|
|
;; "expected a colon here" with a position the same way this file does, and
|
|
;; there is nowhere else the position would come from.
|
|
;;
|
|
;; The first failure wins: a later one would overwrite the offset that explains
|
|
;; the document with an offset that is merely downstream of it.
|
|
fn fail(c: Ptr(Cursor), code: i32, pos: i32) -> ()
|
|
if c.err == err-none
|
|
c.err = code
|
|
c.err-pos = pos
|
|
|
|
;; ── Byte classes ────────────────────────────────────────────────────
|
|
|
|
;; Everything that ends an unquoted token. `/` is in here because a comment can
|
|
;; start against the byte before it — `1//x` — and a scanner that stopped only
|
|
;; on whitespace and brackets would read `1//x` as one atom and then report a
|
|
;; bad number instead of a comment.
|
|
fn- is-delim(b: u8) -> bool
|
|
is-space(b) or b == \( or b == \) or b == \[ or b == \] or b == \{ or b == \} or b == \" or b == \' or b == \, or b == \: or b == \/
|
|
|
|
fn- is-alpha(b: u8) -> bool = (b >= \a and b <= \z) or (b >= \A and b <= \Z)
|
|
|
|
fn- is-hex(b: u8) -> bool
|
|
is-digit(b) or (b >= \a and b <= \f) or (b >= \A and b <= \F)
|
|
|
|
fn- hex-val(b: u8) -> i32
|
|
if is-digit(b)
|
|
i32(b) - i32(\0)
|
|
elif b >= \a and b <= \f
|
|
10 + (i32(b) - i32(\a))
|
|
else
|
|
10 + (i32(b) - i32(\A))
|
|
|
|
;; ── Internal helpers ────────────────────────────────────────────────
|
|
;;
|
|
;; "Internal" by intent and not by enforcement: a package has no visibility
|
|
;; yet, so json/scan-atom and json/push-open are as callable as json/next is.
|
|
;; Nothing below is part of the API and none of it will keep its shape.
|
|
|
|
fn- is-at-end(c: Ptr(Cursor)) -> bool = c.pos >= length(c.src)
|
|
|
|
;; An empty slice of src, positioned at p. Used for the tokens that have no
|
|
;; text of their own — eof, error, and every delimiter — so that `text` has one
|
|
;; meaning for every token kind and not two.
|
|
fn- empty-at(c: Ptr(Cursor), p: i32) -> [const u8] = slice(c.src, p, p)
|
|
|
|
fn- token(c: Ptr(Cursor), kind: i32, lo: i32, hi: i32, p: i32) -> Token
|
|
Token{.kind kind .text slice(c.src, lo, hi) .pos p}
|
|
|
|
fn- error-token(c: Ptr(Cursor)) -> Token
|
|
Token{.kind tok-error .text empty-at(c, c.err-pos) .pos c.err-pos}
|
|
|
|
;; Whitespace only. There is no comment case here and there is deliberately no
|
|
;; comment case anywhere: a `/` reaches the dispatch and is refused by name.
|
|
fn- skip-trivia(c: Ptr(Cursor)) -> ()
|
|
while not is-at-end(c) and is-space(c.src[c.pos])
|
|
c.pos += 1
|
|
|
|
;; The end of the unquoted token starting at lo: the first delimiter, or the
|
|
;; end of input.
|
|
fn scan-atom(c: Ptr(Cursor), lo: i32) -> i32
|
|
let i = lo
|
|
while i < length(c.src) and not is-delim(c.src[i])
|
|
i += 1
|
|
i
|
|
|
|
fn push-open(c: Ptr(Cursor), closer: i32, p: i32) -> bool
|
|
if c.depth >= max-depth
|
|
fail(c, err-too-deep, p)
|
|
return false
|
|
c.open[c.depth] = closer
|
|
c.depth += 1
|
|
true
|
|
|
|
fn- pop-close(c: Ptr(Cursor), closer: i32, p: i32) -> bool
|
|
if c.depth == 0 or c.open[c.depth - 1] != closer
|
|
fail(c, err-unbalanced, p)
|
|
return false
|
|
c.depth -= 1
|
|
true
|
|
|
|
;; ── Numbers ─────────────────────────────────────────────────────────
|
|
|
|
;; What enters read-number. A digit or a `-` is JSON's own rule; `+` and `.`
|
|
;; are here so that they reach the scanner and get the refusal that names them,
|
|
;; instead of falling through to "unexpected byte" — which would be true and
|
|
;; would not tell anyone that the fix is to write 0.5.
|
|
fn- is-number-start(b: u8) -> bool
|
|
is-digit(b) or b == \- or b == \+ or b == \.
|
|
|
|
;; JSON's number grammar, written out rather than left to parse-i64: -?(0 |
|
|
;; [1-9][0-9]*)(.[0-9]+)?([eE][+-]?[0-9]+)?. Every one of the named refusals
|
|
;; below is a production this grammar does not have and JSON5's does, and each
|
|
;; is caught at the byte where the two diverge — so `0x1f` is reported as
|
|
;; hexadecimal and not as "12x"-style garbage, which is what a parse-first
|
|
;; scanner would have to call it.
|
|
;;
|
|
;; A number with neither a fraction nor an exponent is tok-int and anything
|
|
;; else is tok-float, so 1e3 is a float even though its value is whole. That is
|
|
;; the grammar's own split and not a guess about the caller's field.
|
|
fn- read-number(c: Ptr(Cursor), lo: i32) -> Token
|
|
let s = c.src
|
|
i = lo
|
|
is-float = false
|
|
if s[lo] == \+
|
|
c.pos = lo + 1
|
|
fail(c, err-leading-plus, lo)
|
|
return error-token(c)
|
|
if s[lo] == \.
|
|
c.pos = lo + 1
|
|
fail(c, err-leading-dot, lo)
|
|
return error-token(c)
|
|
if s[i] == \-
|
|
i += 1
|
|
if i >= length(s) or not is-digit(s[i])
|
|
;; `-` with nothing usable after it. The scan-atom is only so that the
|
|
;; cursor lands past the whole bad token and not in the middle of it.
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-bad-number, lo)
|
|
return error-token(c)
|
|
;; The integer part, and the two shapes that are JSON5's.
|
|
if s[i] == \0
|
|
i += 1
|
|
if i < length(s) and (s[i] == \x or s[i] == \X)
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-hex-number, lo)
|
|
return error-token(c)
|
|
if i < length(s) and is-digit(s[i])
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-leading-zero, lo)
|
|
return error-token(c)
|
|
else
|
|
while i < length(s) and is-digit(s[i])
|
|
i += 1
|
|
;; The fraction. A dot with no digit after it is `1.` or `1.e3`, both of
|
|
;; which JSON5 accepts and JSON does not.
|
|
if i < length(s) and s[i] == \.
|
|
is-float = true
|
|
i += 1
|
|
if i >= length(s) or not is-digit(s[i])
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-trailing-dot, lo)
|
|
return error-token(c)
|
|
while i < length(s) and is-digit(s[i])
|
|
i += 1
|
|
;; The exponent. Its sign is allowed where the number's own leading + was
|
|
;; not, which is JSON's rule and reads like an inconsistency until you have
|
|
;; seen 1e+3 in a file written by a serialiser.
|
|
if i < length(s) and (s[i] == \e or s[i] == \E)
|
|
is-float = true
|
|
i += 1
|
|
if i < length(s) and (s[i] == \+ or s[i] == \-)
|
|
i += 1
|
|
if i >= length(s) or not is-digit(s[i])
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-bad-number, lo)
|
|
return error-token(c)
|
|
while i < length(s) and is-digit(s[i])
|
|
i += 1
|
|
;; Whatever is left has to end the token. Without this `12x` and `1.5.5`
|
|
;; would be accepted as their own prefixes and the junk would be read as
|
|
;; the next value.
|
|
if i < length(s) and not is-delim(s[i])
|
|
c.pos = scan-atom(c, lo)
|
|
fail(c, err-bad-number, lo)
|
|
return error-token(c)
|
|
c.pos = i
|
|
let text = slice(s, lo, i)
|
|
if not is-float
|
|
;; Grammar-legal and still not representable: an i64 has 19 digits and
|
|
;; JSON's integers have no bound. It is a number, so it becomes a float
|
|
;; rather than an error — which loses precision and says so here, since
|
|
;; the alternative is refusing a document over a field nobody reads.
|
|
if match(parse-i64(text), Some(_), true, None, false)
|
|
return token(c, tok-int, lo, i, lo)
|
|
if match(parse-f64(text), Some(_), true, None, false)
|
|
return token(c, tok-float, lo, i, lo)
|
|
fail(c, err-bad-number, lo)
|
|
error-token(c)
|
|
|
|
;; ── Strings ─────────────────────────────────────────────────────────
|
|
|
|
;; Four hex digits starting at i, as a code point, or -1. Used twice — for an
|
|
;; escape and for the low half of a surrogate pair — which is the whole reason
|
|
;; it is a function.
|
|
fn- hex4(c: Ptr(Cursor), i: i32) -> i32
|
|
let s = c.src
|
|
if i + 4 > length(s)
|
|
return -1
|
|
let v = 0
|
|
for k in range(4)
|
|
let b = s[i + k]
|
|
if not is-hex(b)
|
|
return -1
|
|
v = v * 16 + hex-val(b)
|
|
v
|
|
|
|
fn- is-high-surrogate(r: i32) -> bool = r >= 0xd800 and r <= 0xdbff
|
|
|
|
fn- is-low-surrogate(r: i32) -> bool = r >= 0xdc00 and r <= 0xdfff
|
|
|
|
;; The whole reason this is not three lines. `text` is the interior, between
|
|
;; the quotes and RAW; `pos` is the opening quote, so an editor underlines the
|
|
;; literal and not its contents.
|
|
;;
|
|
;; Every escape is validated here, at the byte it starts on, because this is
|
|
;; the last place that knows that offset — string-of runs over a literal this
|
|
;; has already accepted, and an error out of it would have to point at the
|
|
;; token rather than at the backslash. Surrogate PAIRING is checked here too,
|
|
;; and not only escape syntax, so that string-of's encode-rune can never be
|
|
;; handed a code point the prelude refuses.
|
|
fn- read-string(c: Ptr(Cursor), lo: i32) -> Token
|
|
let s = c.src
|
|
i = lo + 1
|
|
while i < length(s)
|
|
let b = s[i]
|
|
if b == \"
|
|
c.pos = i + 1
|
|
return token(c, tok-string, lo + 1, i, lo)
|
|
;; A literal tab or newline inside a literal. JSON says escape it;
|
|
;; this is the refusal a hand-written file actually meets.
|
|
elif b < 0x20
|
|
c.pos = i
|
|
fail(c, err-control-char, i)
|
|
return error-token(c)
|
|
elif b == \\
|
|
if i + 1 >= length(s)
|
|
c.pos = i + 1
|
|
fail(c, err-unterminated, lo)
|
|
return error-token(c)
|
|
let e = s[i + 1]
|
|
if e == \" or e == \\ or e == \/ or e == \b or e == \f or e == \n or e == \r or e == \t
|
|
i += 2
|
|
elif e == \u
|
|
let r = hex4(c, i + 2)
|
|
if r < 0
|
|
c.pos = i
|
|
fail(c, err-bad-unicode, i)
|
|
return error-token(c)
|
|
i += 6
|
|
;; A high surrogate has to be followed by \u and a low one.
|
|
;; A low one on its own, or a high one followed by anything
|
|
;; else, encodes no character: rune-size answers None for the
|
|
;; whole D800-DFFF block and encode-rune writes nothing, so
|
|
;; the alternative to refusing here is a silently dropped
|
|
;; character later. That refusal is forced by the prelude
|
|
;; rather than chosen, and it is the reason the pairing rule
|
|
;; is in the tokenizer at all.
|
|
if is-high-surrogate(r)
|
|
let ok = i + 1 < length(s) and s[i] == \\ and s[i + 1] == \u and is-low-surrogate(hex4(c, i + 2))
|
|
if not ok
|
|
c.pos = i
|
|
fail(c, err-lone-surrogate, i - 6)
|
|
return error-token(c)
|
|
i += 6
|
|
if is-low-surrogate(r)
|
|
c.pos = i
|
|
fail(c, err-lone-surrogate, i - 6)
|
|
return error-token(c)
|
|
else
|
|
c.pos = i
|
|
fail(c, err-bad-escape, i)
|
|
return error-token(c)
|
|
else
|
|
i += 1
|
|
;; Ran off the end with the literal still open. Reported at the opening
|
|
;; quote: that is the byte a caller has to look at, not the end of the file.
|
|
c.pos = i
|
|
fail(c, err-unterminated, lo)
|
|
error-token(c)
|
|
|
|
;; ── The dispatch ────────────────────────────────────────────────────
|
|
|
|
;; The one call a caller makes. Advances the cursor past the token it returns.
|
|
;;
|
|
;; A cursor that has already failed keeps answering the same error token and
|
|
;; does not advance, so `while t.kind != tok-eof ...` terminates on a
|
|
;; malformed document instead of spinning.
|
|
fn next(c: Ptr(Cursor)) -> Token
|
|
if not is-ok(c)
|
|
return error-token(c)
|
|
skip-trivia(c)
|
|
if is-at-end(c)
|
|
;; Something still open at the end of input is malformed, and the position
|
|
;; that helps is the end — the document stopped, not the value.
|
|
if c.depth > 0
|
|
fail(c, err-unbalanced, c.pos)
|
|
return error-token(c)
|
|
return Token{.kind tok-eof .text empty-at(c, c.pos) .pos c.pos}
|
|
let s = c.src
|
|
lo = c.pos
|
|
b = s[lo]
|
|
;; ── Delimiters, each of which moves the balance stack ──────────
|
|
if b == \[
|
|
c.pos = lo + 1
|
|
if push-open(c, tok-array-close, lo)
|
|
token(c, tok-array-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \]
|
|
c.pos = lo + 1
|
|
if pop-close(c, tok-array-close, lo)
|
|
token(c, tok-array-close, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \{
|
|
c.pos = lo + 1
|
|
if push-open(c, tok-object-close, lo)
|
|
token(c, tok-object-open, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
elif b == \}
|
|
c.pos = lo + 1
|
|
if pop-close(c, tok-object-close, lo)
|
|
token(c, tok-object-close, lo, lo, lo)
|
|
else
|
|
error-token(c)
|
|
;; ── The separators, which are grammar and not trivia ───────────
|
|
elif b == \:
|
|
c.pos = lo + 1
|
|
token(c, tok-colon, lo, lo, lo)
|
|
elif b == \,
|
|
c.pos = lo + 1
|
|
token(c, tok-comma, lo, lo, lo)
|
|
elif b == \"
|
|
read-string(c, lo)
|
|
;; ── The refusals that have their own byte ──────────────────────
|
|
elif b == \'
|
|
c.pos = lo + 1
|
|
fail(c, err-single-quote, lo)
|
|
error-token(c)
|
|
;; Both comment forms, and a `/` that begins neither. The last is an
|
|
;; unexpected byte and not a comment, because saying "comments are
|
|
;; refused" about a stray slash would send someone looking for a comment
|
|
;; that is not there.
|
|
elif b == \/
|
|
c.pos = lo + 1
|
|
if lo + 1 < length(s) and (s[lo + 1] == \/ or s[lo + 1] == \*)
|
|
fail(c, err-comment, lo)
|
|
else
|
|
fail(c, err-unexpected-byte, lo)
|
|
error-token(c)
|
|
;; ── Numbers, then the three words, then everything else ────────
|
|
elif is-number-start(b)
|
|
read-number(c, lo)
|
|
elif is-alpha(b)
|
|
let hi = scan-atom(c, lo)
|
|
c.pos = hi
|
|
let text = slice(s, lo, hi)
|
|
if is-bytes-equal(text, bytes-view("null"))
|
|
token(c, tok-null, lo, hi, lo)
|
|
elif is-bytes-equal(text, bytes-view("true"))
|
|
token(c, tok-bool, lo, hi, lo)
|
|
elif is-bytes-equal(text, bytes-view("false"))
|
|
token(c, tok-bool, lo, hi, lo)
|
|
elif is-bytes-equal(text, bytes-view("NaN"))
|
|
fail(c, err-nan-inf, lo)
|
|
error-token(c)
|
|
elif is-bytes-equal(text, bytes-view("Infinity"))
|
|
fail(c, err-nan-inf, lo)
|
|
error-token(c)
|
|
else
|
|
fail(c, err-bare-word, lo)
|
|
error-token(c)
|
|
else
|
|
;; One byte forward before failing, so the offset is the offending byte
|
|
;; and a caller's loop cannot spin on it.
|
|
c.pos = lo + 1
|
|
fail(c, err-unexpected-byte, lo)
|
|
error-token(c)
|
|
|
|
;; ── Reading values out of a token ───────────────────────────────────
|
|
;;
|
|
;; Each checks the kind first. None for the wrong kind rather than a parse of
|
|
;; whatever bytes happened to be there.
|
|
|
|
fn int-of(t: Token) -> Option(i64)
|
|
if t.kind == tok-int then parse-i64(t.text) else None
|
|
|
|
;; Accepts an integer token too: 1 and 1.0 are the same number, and a document
|
|
;; that writes 2 for a float field is not making a mistake.
|
|
fn float-of(t: Token) -> Option(f64)
|
|
if t.kind == tok-float or t.kind == tok-int then parse-f64(t.text) else None
|
|
|
|
fn bool-of(t: Token) -> Option(bool)
|
|
if t.kind == tok-bool
|
|
Some(is-bytes-equal(t.text, bytes-view("true")))
|
|
else
|
|
None
|
|
|
|
;; ── The one call that allocates ─────────────────────────────────────
|
|
|
|
;; A string token's text, unescaped, COPIED into the context's allocator. Read
|
|
;; the header before using it; the two rules that matter are:
|
|
;;
|
|
;; * the result does not point into the source buffer, so it outlives it —
|
|
;; which is the opposite of edn's contract and the whole reason this
|
|
;; function exists;
|
|
;; * it points into the allocator instead, so under an arena it dies with a
|
|
;; free-all and under the heap it is a block nobody has freed. There is no
|
|
;; third owner.
|
|
;;
|
|
;; The builder is a (Vec u8) and the answer is (str (slice b)) taken ONCE
|
|
;; at the end. That ordering is load-bearing rather than stylistic: a push
|
|
;; after the slice has been taken can grow the Vec into a new block, and in an
|
|
;; arena the old block is still mapped, so the already-taken string would go on
|
|
;; reading bytes that are no longer the answer. There is no (free b) either,
|
|
;; for the same reason — the returned string is that block.
|
|
;;
|
|
;; None only for a token that is not a string. Every way the bytes themselves
|
|
;; could be wrong was already refused by read-string, which is why this cannot
|
|
;; fail halfway through and leave a half-built answer behind.
|
|
fn string-of(t: Token) -> Option(str)
|
|
if t.kind != tok-string
|
|
return None
|
|
let s = t.text
|
|
b = vec-new(u8)
|
|
i = 0
|
|
while i < length(s)
|
|
let ch = s[i]
|
|
if ch != \\
|
|
push(b, ch)
|
|
i += 1
|
|
else
|
|
let e = s[i + 1]
|
|
if e == \"
|
|
push(b, \")
|
|
i += 2
|
|
elif e == \\
|
|
push(b, \\)
|
|
i += 2
|
|
elif e == \/
|
|
push(b, \/)
|
|
i += 2
|
|
elif e == \b
|
|
push(b, u8(8))
|
|
i += 2
|
|
elif e == \f
|
|
push(b, u8(12))
|
|
i += 2
|
|
elif e == \n
|
|
push(b, \newline)
|
|
i += 2
|
|
elif e == \r
|
|
push(b, \return)
|
|
i += 2
|
|
elif e == \t
|
|
push(b, \tab)
|
|
i += 2
|
|
else
|
|
;; \uXXXX, and the surrogate pair read-string has already proved
|
|
;; is there. The arithmetic is UTF-16's: the high half carries the
|
|
;; top ten bits of a code point above 0xffff and the low half the
|
|
;; bottom ten.
|
|
let r = 0
|
|
for k in range(4)
|
|
r = r * 16 + hex-val(s[i + 2 + k])
|
|
i += 6
|
|
if is-high-surrogate(r)
|
|
let lo2 = 0
|
|
for k in range(4)
|
|
lo2 = lo2 * 16 + hex-val(s[i + 2 + k])
|
|
i += 6
|
|
r = 0x10000 + (r - 0xd800 << 10 || lo2 - 0xdc00)
|
|
;; A four-byte buffer and not a push per byte, because
|
|
;; encode-rune is the prelude's answer for this and writing
|
|
;; the shifts again here would be a second copy of UTF-8.
|
|
let buf = array(4, u8)
|
|
match encode-rune(slice(buf, 0, 4), r)
|
|
Some(w) -> append(addr(b), slice(buf, 0, w))
|
|
;; Unreachable: read-string refuses every code point
|
|
;; rune-size refuses. Written as a no-op rather than a trap
|
|
;; because a dropped character is not worth a crash and the
|
|
;; tokenizer is the place that already said no.
|
|
None -> ()
|
|
Some(str(slice(b)))
|
|
|
|
;; ── Reading past a value ────────────────────────────────────────────
|
|
|
|
;; Consumes exactly one value — a scalar, or a whole array or object with
|
|
;; everything nested inside it. This is what a reader calls on an object key it
|
|
;; does not know, so an extra field in a document is ignored rather than fatal.
|
|
;;
|
|
;; Iterative on the cursor's own balance depth and not recursive: the depth is
|
|
;; already tracked, and a recursive skip would put the nesting on the C stack
|
|
;; where a deep document is a crash rather than err-too-deep.
|
|
;;
|
|
;; The separators inside the skipped value are swallowed by the depth loop and
|
|
;; are not checked, which is the honest reading of "skip": a caller that wanted
|
|
;; the commas inside an unknown field validated wanted to read it.
|
|
fn skip-value(c: Ptr(Cursor)) -> bool
|
|
let start = c.depth
|
|
t = next(c)
|
|
if not is-ok(c)
|
|
return false
|
|
if t.kind == tok-eof
|
|
fail(c, err-unexpected-token, t.pos)
|
|
return false
|
|
if c.depth <= start
|
|
return true
|
|
while c.depth > start
|
|
let u = next(c)
|
|
if not is-ok(c)
|
|
return false
|
|
if u.kind == tok-eof
|
|
;; next already failed on the open depth; this is belt and braces.
|
|
fail(c, err-unbalanced, u.pos)
|
|
return false
|
|
true
|
|
|
|
;; ── Expecting a kind ────────────────────────────────────────────────
|
|
|
|
;; The shape a reader is built out of: take the next token, and if it is not
|
|
;; the kind wanted, fail the cursor at that token's position with a reason. The
|
|
;; returned token is the error token in that case, so a caller that forgets to
|
|
;; test is-ok still does not read a value out of the wrong kind — int-of and
|
|
;; string-of answer None for tok-error.
|
|
fn expect(c: Ptr(Cursor), kind: i32) -> Token
|
|
let t = next(c)
|
|
if is-ok(c) and t.kind != kind
|
|
fail(c, err-unexpected-token, t.pos)
|
|
return error-token(c)
|
|
t
|