text

written against zig 0.16.

zig has no string type. text is []const u8, and the language makes no promise about what encoding those bytes are in. indexing and slicing are byte operations, which is correct for a byte slice and wrong for most of the things people write with it.

the two vocabularies

almost every text operation is described in one of two units, and they are easy to mix up because ASCII makes them identical:

• bytes — buffer sizes, length caps, wire formats. slicing is right here.
• characters — "first character", "last three characters", "one character per class". slicing is wrong here the moment input leaves ASCII.

"’s" is four bytes: E2 80 99 73. taking token[0] yields 0xE2, which is not a character. it is the lead byte of one, and alone it identifies nothing — the same first byte begins thousands of different characters.

what this looks like in practice

spacez extracts four features per token for spaCy's NER model, three of which are defined over characters — the first character, the last three characters, and a per-character shape string. all three were implemented with byte slices:

prefix_buf[0] = token[0];            // first byte
var suffix_buf: [3]u8 = undefined;   // last three bytes
for (token) |c| { ... }              // shape, per byte

for ASCII this is exactly right and every test passed. for a curly apostrophe the prefix hashed a lone 0xE2, the suffix came from the middle of a character, and the shape emitted three symbols where spaCy emitted one. the model got embeddings for tokens that do not exist, and answered confidently.

nothing crashed. bytes are always valid bytes.

decoding

std.unicode has what you need to work in characters:

const n = try std.unicode.utf8ByteSequenceLength(bytes[0]);  // 1..4
const cp = try std.unicode.utf8Decode(bytes[0..n]);
var it = (try std.unicode.Utf8View.init(s)).iterator();       // codepoint walk

going backwards has no helper. continuation bytes are 10xxxxxx, so walking back to a character boundary is a loop over b & 0xC0 == 0x80.

classification stops at ASCII

std.ascii answers isAlphabetic, isUpper, toLower for bytes below 128. above that, std has no case mapping and no character categories — those are large tables and zig does not ship them.

this matters when a spec is written in terms of "letter" or "uppercase". spaCy's shape feature calls Проверка five letters and lowercases Ω to ω; reproducing that needs unicode tables from somewhere. the honest options are to carry the tables, take a dependency, or document the gap. picking one of ASCII's answers and hoping is the option that looks like it works.

truncation

a byte cap that lands mid-character produces invalid UTF-8, which json encoders and terminals both object to. walk characters and stop before the one that would cross the limit:

/// truncate to at most `max` bytes without splitting a codepoint
pub fn utf8TruncateLen(s: []const u8, max: usize) usize {
    const limit = @min(s.len, max);
    var i: usize = 0;
    while (i < limit) {
        const seq_len = std.unicode.utf8ByteSequenceLength(s[i]) catch {
            i += 1; // invalid lead byte
            continue;
        };
        if (i + seq_len > limit) break;
        i += seq_len;
    }
    return i;
}

any fixed-size field holding user text wants this — entity names, log lines, anything copied into a [64]u8.

rules of thumb

• if a spec says "character", decode. if it says "byte", slice.
• a fixed-size buffer is a byte count; the loop that fills it is a character loop. keeping those straight in the same function is most of the work.
• test with a curly apostrophe, an emoji, and a Cyrillic word. ASCII tests pass either way, which is what makes this class of bug survive a test suite.

related

• structs — slices are views; the same discipline applies to what a slice points at
• json — invalid UTF-8 surfaces first at the encoder

sources

• spacez src/embed.zig — prefix, suffix and shape computed over bytes; fixed in be0f81f (2026-08-18)
• coral backend/src/utf8.zig — the truncation helper