Semantic Terminal

The Problem: AI Tools Expose Terminals, Not APIs

Every AI coding tool — Claude Code, Gemini CLI, Codex CLI — exposes the same interface to the outside world: a terminal. Not a structured API. Not a WebSocket with typed messages. A raw pseudo-terminal emitting ANSI escape codes, Unicode characters, and implicit state transitions that were designed for human eyes, not programmatic consumption.

If you want to build infrastructure on top of these tools — orchestrate multiple agents, detect when one is stuck, auto-approve safe operations, or simply know whether the agent is thinking or idle — you face a choice: wait for each vendor to expose a structured API (which may never happen, or may differ incompatibly between vendors), or find a way to understand the terminal output directly.

This paper describes the second approach. We present a multi-layer parsing pipeline that transforms raw PTY byte streams into a finite state machine with 8 states, enabling vendor-agnostic observation and orchestration of any AI coding tool — present and future — without requiring vendor cooperation or API access.

Related Work

Computer Use (Anthropic, 2024) approaches the problem from the visual layer: screenshots + pixel-level understanding. This is powerful but expensive — each observation requires a vision model inference. Our approach operates at the character stream level, orders of magnitude cheaper and faster, with deterministic rather than probabilistic state detection.

Model Context Protocol (MCP) provides structured tool-use interfaces, but requires explicit vendor adoption. It solves the invocation problem (how to call tools) but not the observation problem (what is the agent doing right now?). A running Claude Code session with MCP tools still needs external observation to know when it's idle, stuck, or waiting for permission.

Terminal emulators (xterm.js, VTE, alacritty) parse ANSI sequences for rendering — producing a visual grid of characters and colors. We parse for semantics — producing a state machine that captures what the AI agent is doing, not what it's displaying. The distinction is fundamental: two visually identical screens may represent different states (e.g., “thinking” vs “responding” with identical output), and the same state may render differently across terminal sizes, color schemes, and locales.

Process supervision tools (supervisord, systemd, launchd) manage processes but have zero semantic understanding of what's happening inside a PTY session. They can tell you if a process is alive; they cannot tell you if an AI agent is waiting for permission approval.

The Approach: Multi-Layer Parsing Pipeline

The semantic terminal parser transforms raw PTY output into structured state through a five-stage pipeline:

1. Pattern Config (YAML) — Each supported CLI engine has a YAML file defining regex patterns for state detection. Claude Code, Gemini CLI, and Codex each have their own pattern files. Adding support for a new AI tool means writing a new YAML file — zero code changes to the parser.
2. Fingerprint Registry — Structural hashing of screen regions. Rather than matching exact strings (which break on version updates, locale changes, or terminal resizing), the registry computes fingerprints of screen structure: where prompts appear, where spinners animate, where tool output is rendered. Fingerprints are robust against cosmetic variations.
3. State Parser — Ordered rule evaluation against the current screen state. Rules are evaluated top-to-bottom with early exit. The first matching rule determines the state. Rule priority is calibrated per CLI engine — for example, Claude Code's “Confirming” state takes priority over “Responding” because permission dialogs must be detected immediately.
4. Confirm Parser — Specialized detection for permission dialogs. These are the highest-priority signals because a stuck permission prompt blocks the entire session. The confirm parser detects dialog structure, extracts the question text, and identifies the available actions (approve, deny, always approve).
5. Tool Output Parser — Tracks tool invocations within the terminal output. Detects tool start, captures output, identifies tool completion. This enables the orchestration layer to know not just that the agent is running a tool, but which tool and what it returned.

The State Machine: 8 States

The parser outputs one of 8 states, forming a finite state machine with well-defined transitions:

Starting ──→ Idle ──→ Thinking ──→ Responding ──→ ToolRunning
               ↑          ↑            ↑               ↑
               └──────────┴────────────┴───────────────┘
                          (cycles back to Idle)

               Idle ──→ SlashMenu ──→ Idle
               Any  ──→ Confirming ──→ Idle | ToolRunning
               Any  ──→ Error

• Starting — Process spawned, CLI loading. Detected by boot sequence patterns (version string, initialization output).
• Idle — Agent waiting for input. The critical state for orchestration: this is when you can safely send new instructions. Detected by prompt character patterns that vary per CLI engine.
• Thinking — Agent processing, no output yet. Detected by spinner animations, “thinking” indicators, or absence of output with active process.
• Responding — Agent generating text output. Distinguished from Thinking by active character emission.
• ToolRunning — Agent executing a tool (file read, bash command, etc.). Detected by tool invocation patterns and output framing.
• Confirming — Agent asking for human permission. The highest-priority detection — a stuck confirmation blocks everything.
• SlashMenu — Transient state for slash command menus. Detectable by menu rendering patterns.
• Error — Unrecoverable error state. Process crash, authentication failure, or fatal parsing error.

Technical Challenges

ANSI Escape Sequences

Terminal output is not a clean text stream. It contains escape sequences for cursor movement (ESC[H), color (ESC[38;2;r;g;bm), screen clearing (ESC[2J), and dozens of other control operations. The parser must strip or interpret these correctly before pattern matching. A naive approach (regex on raw bytes) fails immediately because escape sequences interleave with content unpredictably.

Partial Writes and Race Conditions

PTY output arrives in chunks of arbitrary size. A single logical line may arrive as 3 separate write events. A state transition indicator (e.g., the “thinking” spinner) may be split across chunks. The parser maintains a screen buffer that accumulates writes and re-evaluates state on each update, rather than parsing individual write events in isolation.

Screen Reflows

When the terminal is resized, content reflows. A prompt that was on line 10 may move to line 12. Fingerprint-based detection (structural hashing of screen regions rather than absolute positions) handles this naturally — the structure of a permission dialog is the same regardless of where it appears on screen.

Timer Detection

AI coding tools display elapsed-time indicators during processing (e.g., (3s · or (1m 23s ·). These are active signals — a changing timer proves the process is alive and working, not hung. The parser detects timer patterns and uses their presence as a positive health signal, distinct from static output that might indicate a hang.

Multi-CLI Extensibility

The parser currently supports Claude Code and Gemini CLI as first-class engines, with Codex CLI support planned. Each engine shares the same 8-state abstraction but has completely independent detection logic:

patterns/
  claude-code.yaml    # Claude's prompt patterns, spinner, tool framing
  gemini-cli.yaml     # Gemini's prompt patterns, response framing
  codex-cli.yaml      # (planned)

fingerprints/
  claude-code/        # Structural hashes for Claude's UI elements
  gemini-cli/         # Structural hashes for Gemini's UI elements

The orchestration layer above the parser is completely engine-agnostic. It receives state transitions and acts on them without knowing or caring which CLI produced them. This means infrastructure built on the semantic terminal — auto-approval policies, stuck detection, multi-agent coordination — automatically works with any CLI the parser supports.

Practical Applications

• Multi-agent orchestration — Spawn N background AI agents, detect when each becomes Idle, dispatch new tasks to available agents. This is the core use case in MissionD's slot-based compute management.
• Auto-approval policies — When the parser detects Confirming state, extract the permission question and apply configurable policies (always approve file reads, always deny destructive operations, prompt human for everything else).
• Stuck detection — If an agent stays in Thinking for longer than a configurable threshold with no timer advancement, it's likely hung. The orchestrator can kill and restart it.
• Session recording and replay — Capture state transitions as a structured timeline rather than raw terminal output. Enables post-hoc analysis of agent behavior.
• Cost attribution — Track time spent in each state per session. Thinking time correlates with token consumption. This enables cost analysis across agents and tasks.

Design Decisions

Why the terminal, not the process? We considered intercepting API calls (LD_PRELOAD / dtrace) and parsing JSONL log files. API interception is fragile across runtime updates and OS versions. Log file parsing is delayed (written after the fact) and format-dependent. The terminal is the real-time, universal, stable interface that every CLI tool must expose to function.

Why fingerprints, not exact strings? Early prototypes used exact string matching for state detection. These broke on every CLI version update, locale change, and terminal width variation. Structural fingerprinting — hashing the shape of screen regions rather than their exact content — provides resilience against cosmetic changes while maintaining detection accuracy.

Why YAML pattern configs? Pattern definitions must be modifiable without recompilation. When a CLI vendor updates their UI (e.g., changes the spinner character), the fix is a one-line YAML edit deployed without rebuilding the parser binary.

The Thesis

The terminal is the universal API for AI coding tools. Not because it was designed to be — it wasn't — but because every tool must expose one to function, and the information it contains is sufficient for orchestration.

By parsing terminals semantically rather than requiring structured APIs, we achieve vendor-agnostic agent observation that works with any current AI coding tool and automatically extends to future ones. The cost of supporting a new tool is a YAML configuration file, not an API integration. The state machine abstraction is universal enough to capture the behavior of any interactive CLI agent, yet specific enough to enable practical orchestration.

As AI coding tools proliferate, the ability to observe and orchestrate them without vendor cooperation becomes critical infrastructure. The semantic terminal is that infrastructure.