Joseph Ferano 19be614f22 A tokenizer is what fits without an allocator
The type-directed half — (read-edn Enemy bytes), a parser emitted from a
compile-time walk over a struct — is the compiler's work and is not here.
What a running program can have today is the half underneath it, and the
shape of that half is decided entirely by there being no heap: a token is a
slice of the input, so reading a file costs one buffer and nothing else, and
the cost is a lifetime contract the types cannot state. It is stated in the
header instead, because a dangling [u8] is otherwise found from a corrupted
string several frames later.

A package and not the prelude. The prelude is prepended to every program and
everything in it is emitted, so a reader nobody imports would be a tax on
every build.

Token kinds are i32 constants rather than a defenum, which reads like a
downgrade and is not one: an Enum value cannot be compared with `=` (emit
fails) and a keyword is not a pattern (`match` refuses one), so a defenum here
is FFI-only and a caller could not branch on a kind at all. Both fixes live in
check.ml and emit.ml, which this lane does not touch.

Errors land on the cursor — a code and a byte offset — rather than in an
(Option Token). None says something went wrong; an editor needs to know where,
and a second out-parameter for the position is the same two fields with a
worse shape. A failed cursor is poisoned so a caller's while loop stops
instead of spinning. error-message turns a code into the sentence, and every
refusal gets its own: escapes, sets, tagged literals, #inst and #uuid
separately, metadata, ratios and characters each name themselves and say why,
so a file using one fails with what to remove rather than with a number.

Escapes are the refusal that had to be a refusal. Unescaping needs somewhere
to put the copy and there is nowhere; returning the raw bytes would hand back
a three-byte string as four, with a backslash in it, and nothing would say so.

Balance is checked in `next` against a fixed [32 i32] stack in the cursor,
because `[1 2}` is malformed in a way only the tokenizer has the position for,
and a growable stack is another thing there is no allocator for. Past 32 the
answer is err-too-deep rather than a closer that quietly went unchecked.

Symbol starts are a list and not "anything that is not a delimiter". Without
that, `@` and a backtick read as one-character symbols instead of being
reported; the ratio test is likewise digit-started only, so foo/bar stays a
namespaced symbol.
2026-09-11 19:58:04 +07:00
2026-09-10 18:55:55 +07:00
2024-07-09 21:01:55 +10:00
2026-09-11 09:22:44 +07:00
2026-09-10 14:56:35 +07:00
2026-09-10 14:40:34 +07:00
2026-09-11 19:39:29 +07:00
2026-09-10 14:40:34 +07:00
2026-09-10 14:40:34 +07:00
Description
No description provided
MIT 7.5 MiB
Languages
OCaml 67.2%
Emacs Lisp 15.2%
C 10.4%
HTML 2.9%
Standard ML 2.8%
Other 1.5%