Parsing the Protocol — a VT State Machine

So far you've been speaking the protocol. A terminal emulator's defining job is the opposite: listening. Everything the programs on the pty slave print — shells, compilers, vim itself — arrives at your master fd as one undifferentiated byte stream, escape sequences mixed into text, and you must decide, byte by byte: is this a character to draw, or part of a command?

Why this must be a state machine

The tempting approach — "when I see ESC, read ahead until the sequence looks complete" — collapses on contact with reality, for one deep reason: the stream has no message boundaries. read() hands you whatever chunk happened to be in the kernel buffer. A single ESC[38;5;208m may arrive as ESC[3 in one read and 8;5;208m in the next; or a thousand sequences may arrive glued together in one 64 KiB read. Your parser can never assume "the rest is here" and can never afford to block waiting for it — there may be a screenful of printable text after the split point.

The clean solution is a pushdown-free state machine: an object that eats exactly one byte at a time, remembers where it is between bytes, and emits an event whenever a unit completes:

struct vt_parser p;
vt_parser_init(&p);

struct vt_event ev;
for (each byte b of whatever read() returned)
    if (vt_feed(&p, b, &ev))
        handle(&ev);       /* print a char, run a command, ... */

Because all context lives in the struct, chunk boundaries simply don't matter: feed it ESC [ 3 and it sits in the CSI state holding a half-built parameter; feed 8 ; 5 ... tomorrow and it carries on. This is the same architecture as every production emulator; xterm, st, and Alacritty differ in table encoding, not in shape. (The canonical description, reverse-engineered from real DEC hardware, is Paul Flo Williams' state diagram at vt100.net — linked in extended reading. Ours is a faithful subset.)

The states

For the VT100 core plus OSC-skipping, five states suffice:

GROUND ──ESC──► ESC_SEEN ──'['──► CSI ──final byte──► GROUND (emit CSI event)
   │                │
   │                ├─']'──► OSC ──BEL or ESC \──► GROUND (no event)
   │                │
   │                ├─'(' or ')'──► CHARSET ──any byte──► GROUND (ignore)
   │                │
   │                └─other──► GROUND (emit simple-ESC event)
   │
   ├── printable byte ──► GROUND (emit PRINT event)
   └── C0 control    ──► GROUND (emit CTRL event)

State by state:

• GROUND — the resting state. Bytes ≥ 0x20 (and, for now, bytes ≥ 0x80 — that's UTF-8, next lesson's problem) are PRINT events, with one carve-out: 0x7F (DEL) is neither printable nor a C0 control — real terminals just swallow it, so the parser absorbs it silently and emits nothing. Bytes below 0x20 are C0 controls: ESC (0x1B) changes state; the rest (\n, \r, \b, \t, BEL…) are CTRL events for the screen layer to interpret.
• ESC_SEEN — one byte of lookahead decides everything: [ opens a control sequence, ] opens an OSC string, ( / ) are the old charset-designation sequences (consume one more byte, ignore — you still see ESC ( B in the wild), anything else is a complete two-byte command (ESC 7, ESC 8, ESC c) — emit it.
• CSI — the parameter grinder, detailed below.
• OSC — swallow bytes (a window title, say) until the terminator: BEL, or ESC \ (which costs a tiny sub-state: an ESC inside OSC might be the start of the terminator). Emit nothing. Skipping correctly matters: mishandle one OSC and every byte after it is interpreted in the wrong state — the classic "my terminal went insane" failure.
• CHARSET — consume exactly one byte, return to GROUND.

Grinding CSI parameters

Inside CSI, each byte is one of four things:

• 0–9 — a digit of the current parameter: cur = cur * 10 + (b - '0').
• ; — parameter separator: append cur to the list, reset the accumulator. An empty slot (ESC[;5H) appends 0 — by convention 0 and "absent" both mean "use the default", so storing 0 loses nothing.
• ? (and its neighbors < = >, bytes 0x3C–0x3F) — mark the sequence private. Real parsers keep which byte; a flag is enough for us. Intermediates 0x20–0x2F may be silently ignored.
• final byte 0x40–0x7E — flush the pending parameter (if the slot was started — digits seen or a ; consumed earlier), emit one CSI event carrying the final byte, the parameter list, and the private flag, and return to GROUND.

Two disciplines keep this robust against hostile streams (remember: cat /dev/urandom is a legal input!):

• Bound the parameter list. ECMA-48 says implementations may limit parameters; 16 slots is generous. Extra parameters are parsed but dropped — never written past the array.
• Bound the values. ESC[99999999999999H must not overflow int. Clamp each parameter at some sane ceiling (we use 65535) while accumulating.

Note what the parser does not do: it never interprets. It doesn't know that H moves cursors or that 2J clears screens — it only knows the grammar. Meaning is the screen layer's job, two lessons away. This separation is what makes both halves testable in isolation.