A decoder that only masks and shifts gets every well-formed character right,
so a corpus of real text passes it. What separates it from a correct one is
the second group here: an overlong two- and three-byte "/", a surrogate, a
code point past U+10FFFF, a lead byte that leads nothing, a lone continuation
byte, and a character truncated by the end of its slice. Each isolates one row
of the accept_sizes table, and each must answer width 1 so a scan advances.
The invalid sequences are byte arrays because no valid string contains them
and the reader has no \xNN escape to spell them with.
Encoding is checked by round trip. An encoder and a decoder wrong in the same
direction agree with each other, and expected bytes would not catch that.
The emoji line caught a use-after-return while this was being written: a
(defn whole [a [4 u8]] [u8] (slice a 0 4)) helper returns a slice into the
copy a [n T] parameter makes in the callee's frame. The compiler accepts it in
silence. The comment stays where the helper was.
Joseph Ferano
273dd18eb0