Text Is Not Bytes — UTF-8

Everything so far pretended one byte = one character. For 1970s terminals that was true; for yours it can't be — type é, →, or 日 into any modern shell and multiple bytes flow. Modern terminals are UTF-8 native: it's what programs write to them and what keyboards send from them. A terminal that mishandles it doesn't get to call itself one.

The design (worth knowing the story)

In 1992, needing an encoding for Plan 9, Ken Thompson sketched UTF-8 on a New Jersey diner placemat (Rob Pike's telling of it is in the extended reading, and it's a great read). The constraints: encode all of Unicode; keep ASCII files valid unchanged; never produce a 0x00 or other ASCII byte inside a multi-byte character (so C strings, /-separated paths, and every existing Unix tool keep working); and make it possible to find character boundaries from anywhere in a stream. The result:

Code point range	Bytes	Pattern
U+0000 – U+007F	1	0xxxxxxx
U+0080 – U+07FF	2	110xxxxx 10xxxxxx
U+0800 – U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 – U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The payload bits (x) carry the code point, big-end first. Read the patterns as a self-describing header: the number of leading 1-bits in the first byte is the sequence length, and every continuation byte is unmistakable (10xxxxxx, values 0x80–0xBF). Consequences:

• ASCII is already UTF-8. Every file you've ever compiled: valid.
• Self-synchronizing. Land at a random byte and you can find the next character start without backing up: skip continuation bytes.
• No aliasing with ASCII. A multi-byte character contains no bytes < 0x80 — strchr(s, '/') can't match inside one.

Worked example — é is U+00E9, binary 000 1110 1001 (11 bits → needs the 2-byte form's 11 payload slots):

110 00011   10 101001
    ^^^^^      ^^^^^^
    00011      101001   →  0xC3 0xA9

Decoding, and the ways bytes go wrong

A decoder collects a leading byte, then the right number of continuation bytes, and reassembles:

cp = (lead & payload_mask);
for each continuation byte b:  cp = (cp << 6) | (b & 0x3F);

But terminals eat arbitrary byte streams, so the error cases are not optional:

• A stray continuation byte (0x80–0xBF with no leading byte).
• A leading byte followed by a non-continuation — the sequence was truncated by whoever produced it.
• Overlong encodings: 0xC0 0x80 decodes by-the-rules to U+0000 — the 2-byte form of a 1-byte value. Forbidden by the spec, and historically a real security hole (encode / as 0xC0 0xAF and sail past a path check that only looked for byte 0x2F). Reject: 2-byte forms below U+0080, 3-byte below U+0800, 4-byte below U+10000.
• UTF-16 surrogates (U+D800–U+DFFF) and values past U+10FFFF: not characters; reject.

The universal convention for all of these — what every terminal, browser, and editor does — is to emit U+FFFD REPLACEMENT CHARACTER (�), consume a minimal prefix (we'll consume exactly one byte), and carry on. Never stall, never crash, never let one bad byte eat good ones after it.

There's one more case that is not an error: the buffer simply ends mid-character because read() split it. Same answer as the VT parser and the key decoder: return "need more bytes" and let the caller re-present the tail later. (Sensing the pattern? Incremental interfaces over byte buffers, all the way down. They compose: GROUND-state print bytes flow from the VT parser into the UTF-8 decoder, whose code points land in screen cells.)

One honest caveat before the challenge: code point ≠ column. 日 occupies two terminal columns; combining marks occupy zero; that's the wcwidth() problem, and real emulators carry Unicode tables for it. Our editor sticks to one-column characters, but know the dragon is there.

size_t utf8_decode(const unsigned char *buf, size_t len, uint32_t *cp);