diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 62bb93e79..b5f5c59be 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -391,11 +391,9 @@ These are read as runtime signals; they are usually set by the terminal/OS rathe | `PI_NOTIFICATIONS` | `off` / `0` / `false` suppress desktop notifications | | `PI_TUI_WRITE_LOG` | If set, logs TUI writes to file | | `PI_HARDWARE_CURSOR` | If `1`, enables hardware cursor mode | -| `PI_CLEAR_ON_SHRINK` | If `1`, clears empty rows when content shrinks | | `PI_NO_SYNC_OUTPUT` | If `1`, disables DEC 2026 synchronized-output wrappers while keeping TUI autowrap guards | | `PI_NO_DECCARA` | If set (truthy), disables Kitty DECCARA rectangular-SGR background fills (forces padded-string rendering) | | `PI_DEBUG_REDRAW` | If `1`, enables redraw debug logging | -| `PI_TUI_DEBUG` | If `1`, enables deep TUI debug dump path | | `PI_FORCE_IMAGE_PROTOCOL` | Forces terminal image protocol detection (`kitty`, `iterm2`/`iterm`, `sixel`, `none`) | --- diff --git a/docs/tui-core-renderer.md b/docs/tui-core-renderer.md index ba9705e8c..cb93617cc 100644 --- a/docs/tui-core-renderer.md +++ b/docs/tui-core-renderer.md @@ -1,295 +1,227 @@ -# TUI core renderer — invariants & failure modes +# TUI core renderer — the append-only contract What you are dealing with before you touch the rendering engine. This is the companion to [`tui-runtime-internals.md`](./tui-runtime-internals.md): that doc -maps the *flow* (input → component tree → render); this doc explains what -**does not work, why it keeps breaking, and the invariants you must not +maps the *flow* (input → component tree → render); this doc explains the +**render contract, why it is shaped this way, and the invariants you must not violate**. Scope is the core engine only: -- [`packages/tui/src/tui.ts`](../packages/tui/src/tui.ts) — render planner, intent emitters, native-scrollback bookkeeping, cursor placement. +- [`packages/tui/src/tui.ts`](../packages/tui/src/tui.ts) — frame pipeline, commit ledger, window math, emitters, cursor placement. - [`packages/tui/src/terminal.ts`](../packages/tui/src/terminal.ts) — `ProcessTerminal`, capability probes, private-CSI reassembly. -- [`packages/tui/src/terminal-capabilities.ts`](../packages/tui/src/terminal-capabilities.ts) — `TERMINAL` profile, ED3 risk / sync-output / DECCARA / image detection. +- [`packages/tui/src/terminal-capabilities.ts`](../packages/tui/src/terminal-capabilities.ts) — `TERMINAL` profile, sync-output / DECCARA / image detection. - [`packages/tui/src/stdin-buffer.ts`](../packages/tui/src/stdin-buffer.ts) — escape-sequence reassembly. - [`packages/tui/src/utils.ts`](../packages/tui/src/utils.ts) — width/slice/wrap (the width model). - [`packages/tui/src/kitty-graphics.ts`](../packages/tui/src/kitty-graphics.ts) + [`components/image.ts`](../packages/tui/src/components/image.ts) — inline images. - [`packages/tui/src/deccara.ts`](../packages/tui/src/deccara.ts) — rectangular-fill optimizer. Application-layer renderers (transcript, tool calls, session tree, editor, -widgets) are **out of scope** — they live in `packages/coding-agent`. +widgets) are **out of scope** — they live in `packages/coding-agent`. The one +app-layer file that is load-bearing for this contract is +[`transcript-container.ts`](../packages/coding-agent/src/modes/components/transcript-container.ts), +which implements the commit-boundary seam described below. --- ## 1. The one thing to understand first -> **The renderer cannot observe the terminal's scroll position on most hosts it -> runs on.** Every decision about rewriting native scrollback is therefore a -> *guess*, and the guess has two opposite failure modes that cannot both be -> avoided by a single policy. +> **The renderer cannot observe the terminal's scroll position** (ConPTY's +> probe lies; POSIX has no API at all). The previous engine tried to *guess* +> when it was safe to rewrite native scrollback, and every policy choice over +> that unobservable variable traded one failure family for another (yank ↔ +> flash ↔ corruption ↔ invisible-until-resize — see the git history of this +> file for the full war journal). The current engine removes the guess +> entirely: **native scrollback is append-only.** -We keep our transcript on the **normal screen**. We deliberately have not moved -the engine to the alternate screen: alt-screen would make the terminal handle -viewport isolation, but the transcript/resume affordances would disappear with -the alternate buffer. Keeping the normal screen means -*we* own native scrollback, which means we must decide, per frame, whether it is -safe to rebuild it. To rebuild history we emit xterm **ED3** (`CSI 3 J`, erase -saved lines). Deciding when ED3 is safe requires knowing whether the user has -scrolled up — and we usually can't: +We keep the transcript on the **normal screen** (native scrollback, native +selection, transcript persists after exit). The engine maintains one ledger: -- **ConPTY hosts** (Windows Terminal, Tabby, Hyper, VS Code, conhost): the - pseudo-console buffer is pinned to the visible grid, so any "am I at the - bottom?" console query answers "yes" even when the reader scrolled up. The - probe *lies*. -- **POSIX terminals**: there is no scroll-position API at all. The probe is - *absent*. +- **`committedRows` (C)** — frame rows `[0, C)` have been physically scrolled + into terminal history. They are **immutable**: the engine never rewrites + them, and components must never change them. +- **`windowTopRow` (W)** — the frame row mapped to grid row 0. The visible + window is frame rows `[W, W + height)`, repainted in place with relative + cursor moves. +- **commit boundary (B)** — reported by the component tree per frame + (`NativeScrollbackLiveRegion`): `B = commitSafeEnd ?? liveRegionStart ?? + frame.length`. Rows below B may still re-layout and must not enter history. -So `Terminal.isNativeViewportAtBottom()` returns `true` / `false` / **`undefined`**, -and `undefined` ("unknown") is the common case. The whole renderer is built -around not trusting `undefined`. +Per ordinary frame: `W = max(C, L − height)`, `C' = max(C, min(B, W))`, and the +only bytes that ever touch history are the **chunk** `frame[C, C')` written at +the scrollback seam. Scrollback therefore equals `frame[0..C)` — every row +exactly once, in order, with its content at commit time. There is nothing to +guess, nothing to defer, and nothing to reconcile: the scroll position is +irrelevant because ordinary updates never rewrite anything a scrolled reader +could be looking at. -### The two-way bind +### What this costs (the accepted tradeoffs) -| If you guess… | …and you're wrong | Symptom | +- A block that has scrolled past the window top cannot reflow in place. Blocks + stay in the live region (below B) until they are final; a late mutation of + committed content is ignored (the stale committed copy stays in history). +- A component tree that reports **no seam** gets shell semantics: whatever + scrolls off is final. Shrinking such a frame into its committed prefix + re-anchors the window and leaves the stale copy in history (§3). +- Inside multiplexers, a resize leaves the pane history wrapped at the old + width (same as any shell output). + +--- + +## 2. The frame pipeline (what you are editing) + +`#doRender` per frame: + +1. Compose the frame (`render(width)`), collecting `liveRegionStart` / + `commitSafeEnd` from the root children (absolute row indices). +2. Classify: **fullPaint** (first paint, `clearScrollback` session replace, or + geometry change outside a multiplexer — all user gestures) or **update**. +3. Window math as in §1. Two special rules: + - **Overlays freeze commits** (`C' = C`): composited rows must never enter + history; the hidden gap backfills via the chunk after the overlay closes. + - **Shrink into the committed prefix** (`L ≤ C`, only possible without a + seam): re-anchor `W = max(0, L − height)`, reset `C = min(B, W)`, keep the + stale history above (no gesture, no erase). +4. Extract the cursor marker, prepare lines (width fitting), slice the window, + composite overlays **into the window slice only** (screen coordinates — an + overlay never touches the frame or the ledger). +5. Emit: + +| Emitter | Bytes | When | |---|---|---| -| **Eager** (rebuild now → emit `CSI 3 J`) | reader was scrolled up | **YANK** to top + **FLASH** on terminals that snap scroll on ED3 | -| **Defer** (emit nothing, reconcile later) | viewport really was at the bottom | **CORRUPTION** (stale/duplicated rows) + **invisible-until-resize** | +| `#emitFullPaint` | clears + `frame[0, C')` + window rows | gestures only. `clearScrollback` ⇒ `\x1b[2J\x1b[H\x1b[3J`; otherwise ED22 (when supported) + `\x1b[2J\x1b[H` | +| `#emitUpdate` scroll-append | `\r\n` + new bottom rows + changed-row range | the rows leaving the screen are exactly the chunk, content untouched since painted | +| `#emitUpdate` in-window diff | relative move + changed-row range rewrite | nothing scrolls, nothing commits (cursor-only when nothing changed) | +| `#emitUpdate` seam rewrite | chunk rows + full window rewrite | commit advance, window re-anchor, hidden-gap backfill, mux resize | -Yank, flash, and buffer corruption are **the same bug wearing three masks.** -Historically, every fix that suppressed one mask for one terminal class -re-enabled the opposite mask for a neighbouring class, and the follow-on -complaint landed within a day. If you "fix flashing" by making rebuilds more -eager, you will reintroduce yank. If you "fix yank" by deferring more, you will -reintroduce corruption / invisibility. **Do not move this lever without the -fidelity harness (§9) green.** +**ED3 (`CSI 3 J`) is emitted in exactly one place** — `#emitFullPaint` with +`clearScrollback: true` — and is reached only by user gestures: session +replace/branch/resume (`requestRender(true, { clearScrollback: true })`), +resize outside a multiplexer, `resetDisplay()` (Ctrl+L). A gesture pins the +user to the tail, so the snap is acceptable; multiplexers never get ED3 (it is +a no-op there and a replay would duplicate pane history). + +The ordinary update path never emits ED2/ED3 or an absolute cursor home — +several terminal families snap a scrolled reader to the bottom on those. + +### The commit-boundary seam (the load-bearing app contract) + +`NativeScrollbackLiveRegion` (tui.ts) is how a component keeps mutable rows out +of history: + +- `getNativeScrollbackLiveRegionStart()` — first row that may still mutate + (everything below it, including root chrome rendered after it, stays in the + window). +- `getNativeScrollbackCommitSafeEnd()` — optional deeper boundary: the + append-only prefix of the live region (a streaming assistant message's + settled rows). Without it, a single live block taller than the window would + hold its head out of history until it finalizes. + +`TranscriptContainer` implements this for the coding agent: finalized blocks +freeze (their render is snapshotted, so their content can never drift after +the engine may have committed it), still-mutating blocks +(`isTranscriptBlockFinalized?.() === false`) anchor the live region, and +`deriveLiveCommitState` detects the append-only stable prefix of a streaming +block (a rewrite of an interior row suspends commits for +`VOLATILE_REARM_FRAMES` clean frames). Freezing is unconditional — it is the +engine's required guarantee, not a per-terminal optimization. --- -## 2. The render-intent planner (what you are editing) +## 3. Invariants — MUST / NEVER -`#doRender` is split into a **planner** (`#planRender`) that classifies a frame -into exactly one `RenderIntent`, and one `#emit*` method per intent that owns -the bytes written and the state update. All state flows through a single -`#commit` checkpoint at the end of every emitter. The intent union -(`tui.ts`, search `type RenderIntent`): - -| Intent | Emits | When | -|---|---|---| -| `noop` | cursor only | nothing visible changed | -| `initial` | clear viewport, paint transcript, **keep** prior shell scrollback | first paint after `start()` | -| `sessionReplace` | clear viewport **+ ED3** (outside multiplexers) | caller forced `{ clearScrollback: true }` (switch/branch/reload/resume) | -| `historyRebuild` | clear viewport **+ ED3** (outside multiplexers) | geometry change rewrapped history, or a proven-at-tail rebuild | -| `overlayRebuild` | rebuild viewport with overlay composite | overlay visibility changed | -| `liveRegionPinned` | relative moves + per-row rewrite/suffix-clear + `\r\n` | foreground streaming on an ED3-risk host, commit-as-you-go | -| `viewportRepaint` | rewrite the visible viewport in place (optional `appendFrom` tail first) | safe non-destructive repaint | -| `deferredShrink` | padded viewport repaint, history left dirty | bottom-anchored shrink, viewport unobservable | -| `deferredMutation` | **zero bytes**, history left dirty | row-reindexing edit while possibly scrolled | -| `shrink` / `diff` | trailing-row clear / changed-line diff | ordinary in-place updates | - -**ED3 (`CSI 3 J`) is emitted in exactly one place** — `#emitFullPaint` when -`clearScrollback: true` (`\x1b[2J\x1b[H\x1b[3J`). The ordinary clear is -**non-destructive**: `\x1b[22J` (copy-screen-to-scrollback, only when -`TERMINAL.supportsScreenToScrollback`) then `\x1b[2J\x1b[H`, **no `3J`**. ED3 is -reached only by `sessionReplace`/`historyRebuild`/`overlayRebuild`, and those -suppress the scrollback clear inside multiplexers (`isMultiplexerSession()` = -`TMUX || STY || ZELLIJ`). - -### The predicate gates - -Three private predicates encode the guessing policy. Do not "simplify" them — -each branch is load-bearing: - -- `#canReplayNativeScrollbackAtCheckpoint(atBottom)` → `atBottom === true`. A - rebuild at a **keystroke checkpoint** (prompt submit) is allowed only with a - *positive* at-tail proof. A prompt submit is **no longer** treated as implicit - proof for an unobservable host. -- `#canRebuildNativeScrollbackLive(atBottom, allowUnknown)` → `true` iff - `atBottom === true`, **or** (`atBottom === undefined && allowUnknown && - platform !== "win32"`). i.e. live ED3 during streaming requires either proof - or an explicit direct-user-input opt-in, and **never** on win32. -- `#nativeViewportIsScrolled(atBottom, allowUnknown)` → `true` if - `atBottom === false`, or (`undefined && win32 && !allowUnknown`). Used to - decide deferral. - -`allowUnknownViewportMutation` is the **direct-user-input opt-in** (autocomplete -/ IME / a keystroke the user just typed). A keystroke pins the host viewport to -the bottom, so it is safe to repaint live then. It is **not** set by passive -streaming. `setEagerNativeScrollbackRebuild(true)` is the streaming opt-in; on -ED3-risk hosts it is downgraded so it never promotes to a live ED3 clear. - -### Deferral + checkpoint discipline - -When the viewport is unobservable during **passive streaming**, the planner -defers (`deferredMutation`/`deferredShrink`/`viewportRepaint`) and marks native -scrollback dirty (`#markNativeScrollbackDirty()`). Reconciliation happens later -at a checkpoint via `refreshNativeScrollbackIfDirty()` — and only if -`#canReplayNativeScrollbackAtCheckpoint` proves at-tail. The streaming-defer + -live-region-pin seam (`NativeScrollbackLiveRegion`, -`getNativeScrollbackLiveRegionStart` / `getNativeScrollbackCommitSafeEnd`) is the -**actively-churning** part of the engine; if you change how transient rows are -committed, every structural-mutation branch (shrink **and** grow/offscreen-edit) -must defer **symmetrically**, or you reopen the corruption family. - ---- - -## 3. The five fault families - -### YANK — viewport snapped to top — NOT fully converged -- **Mechanism:** a live `historyRebuild` fires `CSI 3 J` while the reader is - scrolled up; ED3-snap terminals reset the visible viewport to the top of the - (now-erased) scrollback. -- **Trigger to avoid:** treating an unobservable probe as "at bottom" during - *passive* streaming, or OR-ing an eager-streaming flag into the live ED3 path. -- **Current stance:** never emit ED3 on an unobservable host during passive - streaming; defer and reconcile at a keystroke checkpoint. ConPTY/win32 never - trust the probe at all. - -### CORRUPTION — duplicated / stale rows — NOT fully converged -- **Mechanism:** the flip side of the yank fix. A deferred/repainted frame - leaves rows already committed to native scrollback out of sync with the live - viewport; the scrollback↔viewport seam duplicates (e.g. a 2-row dup, a - streaming-tail dup, or an async-expansion dup). -- **Trigger to avoid:** repainting the viewport over scrollback that still holds - the old copy; a frozen/deferred block whose snapshot no longer matches after - the region above it reflowed; one mutation branch deferring while its mirror - branch repaints. -- **Current stance:** commit only the **stable prefix** line-count to native - history; keep unstable rows out; reconcile drift at the checkpoint; park the - hardware cursor at real content bottom, not padded bottom. - -### FLASH (and invisible-until-resize) — NOT fully converged -- **Two distinct causes, one symptom:** - - *Flash* = eager ED3 rebuild wrapped in DEC 2026 BSU/ESU fired per streaming - frame on a terminal that clamps scroll on ED3 (VTE/GNOME family). - - *Invisible-until-resize* = the defer fix over-firing, so a structural frame - emits **zero bytes** (`deferredMutation` returns nothing) until a resize - forces a repaint. -- **Trigger to avoid:** env-detection that misses a flashing terminal (SSH - strips `VTE_VERSION`; some hosts set no distinguishing var); collapsing an - `undefined` probe into a definite scrolled/at-bottom verdict. -- **Current stance:** confine ED3 to the destructive path; auto-disable DEC 2026 - at runtime when the terminal reports it unsupported (DECRQM), with - `PI_NO_SYNC_OUTPUT` as a manual hatch; keep autowrap discipline regardless. - -### WIDTH — measurement crashes / fidelity — crash class dead, accuracy unproven -- **Mechanism:** the measured column width of a line disagreed with the - terminal's painted cells (emoji, wide graphemes, combining marks, Hangul - jamo), and the old render loop **threw** on any mismatch — a 1-cell cosmetic - error became a fatal whole-agent crash. -- **Current stance:** **never throw in the render hot path — clamp.** The loop - truncates over-wide lines with `truncateToWidth`/`sliceByColumn` and logs - (under debug) instead of dying. Width is owned end-to-end by one native UAX#11 - engine shared by measure/slice/wrap (see §6). Accuracy across all scripts - (e.g. RTL/combining marks) is still not proven by a green gate. - -### PROBE — stray bytes injected as keystrokes — RESOLVED -- **Mechanism:** a private-CSI probe reply (DA1 / kitty / mode 2031) split - across a stdin flush; the unmatched prefix was dropped and the continuation - bytes were forwarded as keystrokes. -- **Current stance:** buffer-and-reassemble partial CSI responses; give each - probe a typed sentinel owner. This is the **one cleanly-closed family** — - because its contract is *bounded and observable* (bytes in = bytes out), - unlike the unobservable-viewport families. See §7. - ---- - -## 4. Invariants — MUST / NEVER - -These are the rules the recurrence taught us. Treat them as load-bearing. - -1. **NEVER add a new `CSI 3 J` (ED3) callsite.** ED3 must flow only through - `#emitFullPaint({ clearScrollback: true })`, for the existing destructive - intents (`sessionReplace`, proven/safe `historyRebuild`, `overlayRebuild`). - Ordinary redraws use the non-destructive `\x1b[22J` + `\x1b[2J\x1b[H` clear. -2. **NEVER trust an unobservable viewport probe (`undefined`) for *passive* - streaming.** Only a positive at-tail proof, or a direct-user-input opt-in - (`allowUnknownViewportMutation`), authorizes a live rebuild — and never on - win32/ConPTY. -3. **NEVER throw in the render hot path.** Clamp over-wide lines; a width - mismatch is cosmetic, not fatal. -4. **NEVER let a defer path emit a structurally-changed frame as zero bytes - while at the bottom** — that is invisible-until-resize. `deferredMutation`/ - `deferredShrink` are only safe when the viewport is (or may be) scrolled. -5. **Defer symmetrically.** If one structural-mutation branch (shrink) defers on - an unobservable ED3-risk host, the mirror branch (grow / offscreen-edit) must - too. Asymmetry reopens corruption. -6. **Commit only the stable prefix to native history.** Transient/unsettled rows - stay out of scrollback until a checkpoint; reconcile drift at the checkpoint. -7. **Park the hardware cursor at real content bottom**, not the padded viewport - bottom, or height shrinks scroll live rows into scrollback and duplicate them +1. **NEVER add a new `CSI 3 J` (ED3) callsite.** ED3 flows only through + `#emitFullPaint({ clearScrollback: true })`, only for gestures, never inside + multiplexers. +2. **NEVER rewrite a committed row.** No emitter may touch frame rows `< C`, + and `W ≥ C` always (re-showing a committed row on the grid duplicates it + for a scrolling reader — the historical corruption family). +3. **Commits are exactly the chunk.** Any byte shape that scrolls the screen + must scroll *only* rows accounted for by `C' − C` — that is what makes + scrollback provably `frame[0..C)`. +4. **NEVER probe the viewport position or fork on platform in the update + path.** win32 behaves like POSIX. The probe APIs are gone; do not + reintroduce them. +5. **Mutable content stays below the commit boundary.** App-layer renderers + must finalize-before-commit; the engine trusts B and clamps, it does not + verify content. +6. **Park the hardware cursor at real content bottom**, not the padded window + bottom, or height shrinks scroll live rows into history and duplicate them per resize step. -8. **Cursor writes live *inside* the synchronized-output frame**, before ESU — - never as a second frame after it (that teleports/blinks the caret). -9. **Detect terminal *risk*, not terminal *brand*, and default unknown to - risky.** Env sniffing is necessarily incomplete (see §5); never assume an - un-enumerated host is safe. -10. **Multiplexers (tmux/screen/zellij) get no destructive scrollback clear and - no viewport probe.** ED3 is a no-op there and a full replay duplicates the - transcript; repaint in place and rely on the pinned/commit-as-you-go path. -11. **Any change to the eager/defer lever, the predicates, or the live-region - seam must be validated by the render-stress fidelity harness (§9)** across - `{win32, POSIX} × {unknown, scrolled, at-bottom}`, not by a single-terminal - smoke test. +7. **Cursor writes live inside the synchronized-output frame**, before ESU — + never as a second frame after it. +8. **NEVER throw in the render hot path.** Clamp over-wide lines + (`truncateToWidth`); a width mismatch is cosmetic, not fatal. +9. **Multiplexers get no destructive clear and no history rewrap on resize** — + repaint the window in place; pane history keeps its old wrap. +10. **Any change to the ledger math, the emitters, or the seam must be + validated by the stress harness (§6)** across its full scenario matrix, + not by a single-terminal smoke test. --- -## 5. Terminal capability detection (and why it is fragile) +## 4. Terminal capability detection `TERMINAL` (`terminal-capabilities.ts`) is resolved once at import from -`TERMINAL_ID` plus environment sniffing. The detection helpers are pure and -parameterized over `(env, platform)` so they are unit-testable: +`TERMINAL_ID` plus environment sniffing; detection helpers are pure over +`(env, platform)` and unit-testable. -- `detectTerminalEagerEraseScrollbackRisk(env, platform)` → is a live ED3 - rebuild unsafe here? Current policy: `false` on win32 (dedicated ConPTY - deferral paths handle it) and when `PI_TUI_ED3_SAFE=1`; otherwise **`true`** - for `WT_SESSION` (WT fronting WSL), SSH/tmux/screen/zellij, known - ED3-snap/scrollback-clearing terminals (WezTerm, kitty, ghostty, alacritty, - VTE, iTerm2, Apple Terminal, GNOME Terminal, Ptyxis, xfce4-terminal), Linux - truecolor, **and every other unknown POSIX terminal**. The default is *risky* - on purpose. -- `shouldEnableSynchronizedOutputByDefault(env, id)` → DEC 2026 default. Precedence: - user opt-out (`PI_NO_SYNC_OUTPUT`/`PI_TUI_SYNC_OUTPUT=0`) → user force-on - (`PI_FORCE_SYNC_OUTPUT=1`/`PI_TUI_SYNC_OUTPUT=1`) → `TERM_FEATURES` advertises - `Sy` → `WT_SESSION` (WT/WSL) → known direct terminals - (kitty/ghostty/wezterm/iterm2/alacritty/vscode; SSH passes through) → off for - risky multiplexers and everything else (VTE-family, GNU screen, Apple Terminal, - legacy conhost, unknown). Reconciled at runtime by the DECRQM mode-2026 report: - a positive report **enables** sync (upgrading default-off muxes like - zellij/tmux-master), a negative one disables it; a user override still wins. - `synchronizedOutputUserOverride(env)` is the shared opt-out/force resolver. -- `detectRectangularSgrSupport(id, env)` → DECCARA fills: **kitty only** - (ghostty does not implement the SGR-background extension), off in multiplexers - and under `PI_NO_DECCARA`. +- `shouldEnableSynchronizedOutputByDefault(env, id)` → DEC 2026 default. + Precedence: user opt-out (`PI_NO_SYNC_OUTPUT`/`PI_TUI_SYNC_OUTPUT=0`) → user + force-on (`PI_FORCE_SYNC_OUTPUT=1`/`PI_TUI_SYNC_OUTPUT=1`) → `TERM_FEATURES` + advertises `Sy` → `WT_SESSION` → known direct terminals → off for risky + multiplexers and unknowns. Reconciled at runtime by the DECRQM mode-2026 + report; a user override still wins. +- `detectRectangularSgrSupport(id, env)` → DECCARA fills: kitty only, off in + multiplexers and under `PI_NO_DECCARA`. +- `supportsScreenToScrollback` → kitty's ED22 (used once, on the initial + paint, to preserve the pre-existing shell screen). -**Why this keeps leaking:** terminal class is inferred from env vars that are -**not durable**. `VTE_VERSION` is stripped by `sshd` (default `AcceptEnv`); -`COLORTERM` is also not in default `AcceptEnv`; some hosts (Tabby) set no -distinguishing var; WSL-fronting-WT is neither pure win32 nor pure POSIX. Every -missed env var is a missed terminal class is a new complaint. The mitigations -are: (a) **default unknown to risky** rather than safe, and (b) detect by -*behavior/handshake* (DECRQM) where possible rather than a host allow-list. When -you add a terminal, add it to the pure detector and add the **SSH-stripped env -shape** to the test, not just the env-present shape. +The old ED3-risk classifier (`eagerEraseScrollbackRisk`, `PI_TUI_ED3_SAFE`, +`submitPinsViewportToTail`) is gone: behavior no longer depends on which +terminal is rendering, so there is no risk class to detect. Env sniffing now +only selects *optimizations* (sync output, DECCARA, images), where a miss is +cosmetic, not corrupting. --- -## 6. Width model +## 5. Width model `visibleWidth` / `truncateToWidth` / `sliceByColumn` / `wrapTextWithAnsi` -(`utils.ts`) all route through **one native UAX#11 engine** (`@oh-my-pi/pi-natives`, -Rust `unicode-width`). We deliberately dropped `Bun.stringWidth` because it -disagreed with the engine on combining marks and jamo, and mixing two width -models in measure-vs-slice produced the crashes. +(`utils.ts`) all route through **one native UAX#11 engine** +(`@oh-my-pi/pi-natives`, Rust `unicode-width`). `Bun.stringWidth` was dropped +deliberately — mixing two width models in measure-vs-slice produced crashes. - Fast path: printable ASCII is one cell per code unit. -- ZWJ pictographic emoji take the `visibleWidthByGrapheme` override (ANSI spans - excised first, then `Intl.Segmenter`), because the native scanner double-counts - SGR bytes when a sequence is split by the segmenter. -- OSC 66 sized text (`\x1b]66;…`) takes the native path. +- ZWJ pictographic emoji take the `visibleWidthByGrapheme` override. +- OSC 66 sized text takes the native path. -**Rule:** if you add a code path that measures width, route it through these -helpers. Never reintroduce `Bun.stringWidth` or a parallel width table — the -measure model and the slice/wrap model must agree, or you get over-wide lines -that the hot-path clamp silently truncates (cosmetic loss) or, worse, seam -duplication. +**Rule:** any new measuring code routes through these helpers, and the hot +path clamps instead of throwing. Known residual: combining-heavy scripts +(Arabic harakat) survive painting verbatim, but ghostty-web's cell readback can +migrate non-spacing marks across cells — the stress harness compares those rows +with marks stripped (`sameLinesAllowingMarkDrift`). + +--- + +## 6. The fidelity gate (use it) + +`packages/tui/test/render-stress-harness.ts` drives the renderer's **real +emitted ANSI** into a ghostty-web `VirtualTerminal` across randomized op +sequences and parameterized terminal shapes, and validates the contract with a +**shadow commit ledger**: an independent reimplementation of §1's math, fed +only by observed frames (a `render` wrap) and observed bytes (a `write` wrap). +Per op it asserts: + +- the whole tape (scrollback + grid) equals `shadowTape + window slice`, row + for row, including across resizes; +- scrolled readers stay pinned and visible history rows are never rewritten; +- multiplexer pane history grows by exactly the committed chunk; +- sync-output/autowrap bracket discipline, cursor parking, background columns, + duplicate accounting. + +Run it — plus `render-regressions.test.ts`, +`streaming-scrollback-defer.test.ts`, and the `issue-*-repro.test.ts` files — +before changing ledger math, emitters, or the seam. A change that passes one +terminal and one seed is not verified. --- @@ -300,90 +232,68 @@ a non-answering terminal is detected when DA1 returns first. Replies can arrive **split across a stdin flush**, so: - `#privateCsiResponseBuffer` accumulates `\x1b[?…` partials while a sentinel is - outstanding, rejoins on the terminator byte (0x40–0x7e), then runs the - DA1/kitty/mode-2031 handlers on the **complete** reply. A new `\x1b` - mid-reassembly or >256 bytes abandons the partial so real keys (e.g. arrow - `\x1b[A`) still reach input. -- `#da1SentinelOwners` is a **typed FIFO** discriminated by `kind` (`keyboard`, - `osc11`, `privateMode`, `kittyGraphicsProbe`, `osc99Probe`) so a keyboard DA1 - cannot be mistaken for an OSC 11 / DECRQM / graphics-probe sentinel. -- DECRQM probes (`#queryPrivateMode(2026/2048/2031)`) record support via DECRPM - and drive runtime feature gating (e.g. auto-disabling DEC 2026 sync output). + outstanding, rejoins on the terminator byte, then runs the handlers on the + **complete** reply. A new `\x1b` mid-reassembly or >256 bytes abandons the + partial so real keys still reach input. +- `#da1SentinelOwners` is a **typed FIFO** discriminated by `kind` so a + keyboard DA1 cannot be mistaken for an OSC 11 / DECRQM / graphics-probe + sentinel. +- DECRQM probes (2026/2048/2031) drive runtime feature gating. -**Rule:** any new probe must own a typed sentinel and survive a split reply. The -contract is bytes-in = bytes-out; it is testable, so test it (feed the reply -byte-by-byte and assert nothing leaks to the input handler). +**Rule:** any new probe must own a typed sentinel and survive a split reply +(feed the reply byte-by-byte in a test and assert nothing leaks to input). --- ## 8. Inline images & memory -Kitty images are **transmit-once, place-many** (`kitty-graphics.ts`): -`encodeKittyTransmit` (`a=t`, keyed by a stable `i=`) writes the base64 a single -time; repaints emit only `encodeKittyPlacement` (`a=p`). Text clears -(`CSI 2 J` / `CSI 3 J`) do **not** purge the terminal's image store — only -`encodeKittyDeleteImage` (`a=d,d=I`) does. `ImageBudget` (`components/image.ts`) -keeps only the most-recent N images live; demoted images render their text -fallback and are explicitly purged. +Kitty images are **transmit-once, place-many** (`kitty-graphics.ts`). +`ImageBudget` keeps only the most-recent N images live; when the cap is +exceeded the demoted image's pixels are deleted by id (`a=d,d=I`) and its +visible rows re-render as the text fallback through the ordinary window diff — +**no destructive replay**. A demoted placement already committed to history +simply loses its pixels (committed rows are immutable). -**Rule:** never re-emit full base64 per frame (it pegged RAM and pinned the UI -thread). Kitty Unicode placeholders are default-on only for kitty/ghostty -(`PI_NO_KITTY_PLACEHOLDERS` / `PI_KITTY_PLACEHOLDERS`); other Kitty-protocol -hosts render placeholder cells as literal PUA glyphs, so they fall back to -direct `a=p` placement. +**Rule:** never re-emit full base64 per frame. Kitty Unicode placeholders are +default-on only for kitty/ghostty (`PI_NO_KITTY_PLACEHOLDERS` / +`PI_KITTY_PLACEHOLDERS`). --- -## 9. The fidelity gate (use it) - -`packages/tui/test/render-stress-harness.ts` renders the renderer's **real emitted ANSI** into -a ghostty-web `VirtualTerminal` and asserts viewport fidelity (a scrolled reader -stays put), background-column fidelity, and scrollback-buffer fidelity, across -parameterized terminal shapes and randomized op sequences. - -This harness is the structural fix for the whole recurrence: every guess-flip and -sniffing-gap regression historically **shipped blind and was caught by a user**, -because no automated "a scrolled-up reader stays pinned across kitty/WT/WSL/ -ConPTY" assertion gated CI. **Before you change the eager/defer lever, a -predicate, the live-region seam, or width math, run the stress harness and the -targeted repro tests** (`packages/tui/test/render-regressions.test.ts`, -`packages/tui/test/streaming-scrollback-defer.test.ts`, the `issue-*-repro.test.ts` files). -A change that passes one terminal and one seed is not verified. - ---- - -## 10. Escape hatches (env vars) +## 9. Escape hatches (env vars) | Var | Effect | |---|---| -| `PI_NO_SYNC_OUTPUT=1` | Disable DEC 2026 BSU/ESU wrappers (autowrap discipline stays on). For terminals that advertise but mishandle mode 2026. | +| `PI_NO_SYNC_OUTPUT=1` | Disable DEC 2026 BSU/ESU wrappers (autowrap discipline stays on). | | `PI_TUI_SYNC_OUTPUT=0\|1` / `PI_FORCE_SYNC_OUTPUT=1` | Force sync output off / on. | -| `PI_TUI_ED3_SAFE=1` | Declare the terminal safe for live ED3 (disables `eagerEraseScrollbackRisk`). | -| `PI_NO_DECCARA` | Disable Kitty DECCARA rectangular-fill optimization (force padded-string fills). | +| `PI_NO_DECCARA` | Disable Kitty DECCARA rectangular-fill optimization. | | `PI_FORCE_IMAGE_PROTOCOL=kitty\|iterm2\|sixel\|off` | Override image protocol detection. | | `PI_NO_KITTY_PLACEHOLDERS=1` / `PI_KITTY_PLACEHOLDERS=1` | Force Kitty Unicode placeholders off / on. | -| `PI_CLEAR_ON_SHRINK=1` | Clear empty rows when content shrinks (default off). | | `PI_HARDWARE_CURSOR=1` | Show the real hardware cursor instead of a rendered one. | | `PI_NOTIFICATIONS=off\|0\|false` | Suppress terminal notifications. | -| `PI_DEBUG_REDRAW=1` | Log the chosen render intent per frame to the debug log. | -| `PI_TUI_DEBUG=1` | Dump per-render diff state under `/tmp/tui`. | +| `PI_DEBUG_REDRAW=1` | Log the chosen render intent + ledger state per frame to the debug log. | + +Removed with the old engine: `PI_TUI_ED3_SAFE` (no ED3-risk lever exists), +`PI_CLEAR_ON_SHRINK` (shrinks always clear exactly), `PI_TUI_DEBUG` (per-render +dump superseded by `PI_DEBUG_REDRAW` ledger logging and the stress harness +replay/reduce tooling). --- -## 11. Before you touch the render core — checklist +## 10. Before you touch the render core — checklist -- [ ] Are you about to emit `CSI 3 J` anywhere other than the destructive - `clearScrollback` path? **Stop.** -- [ ] Does your change trust `isNativeViewportAtBottom() === undefined` as - "at bottom" during passive streaming? **Stop.** -- [ ] Did you change one structural-mutation branch without mirroring its - sibling (shrink ↔ grow)? **Defer symmetrically.** -- [ ] Could any frame now emit zero bytes while the viewport is at the bottom? - That's invisible-until-resize. -- [ ] Did you add a terminal by brand instead of by behavior, or skip the - SSH-stripped env shape in the test? -- [ ] Did you run `packages/tui/test/render-stress-harness.ts` + the repro suite across - win32/POSIX × unknown/scrolled/at-bottom — not just one terminal? +- [ ] Are you about to emit `CSI 3 J` anywhere other than the gesture-driven + `clearScrollback` full paint? **Stop.** +- [ ] Could any code path rewrite, or re-show on the grid, a frame row below + `committedRows`? **Stop.** +- [ ] Does your byte shape scroll rows that are not the commit chunk? That + breaks `scrollback == frame[0..C)`. +- [ ] Are you adding a viewport probe, a platform fork, or a terminal-brand + branch to the update path? The contract exists so none are needed. +- [ ] New mutable UI above the editor? It must report (or live inside) the + live-region seam, or it will freeze at first commit. +- [ ] Did you run the stress harness and the repro suite across the full + scenario matrix — not just one terminal and one seed? - [ ] New probe? Typed sentinel owner + split-reply test. - [ ] New width path? Routed through the shared native engine, clamped (never thrown) in the hot path. diff --git a/docs/tui-runtime-internals.md b/docs/tui-runtime-internals.md index 79df9186d..c37639ae9 100644 --- a/docs/tui-runtime-internals.md +++ b/docs/tui-runtime-internals.md @@ -29,7 +29,7 @@ Boundary rule: the TUI engine is message-agnostic. It only knows `Component.rend ## Boot and component tree assembly -`InteractiveMode` constructs `TUI(new ProcessTerminal(), settings.get("showHardwareCursor"))`, applies `clearOnShrink`, `tui.maxInlineImages`, and Kitty text-sizing settings, then creates persistent containers: +`InteractiveMode` constructs `TUI(new ProcessTerminal(), settings.get("showHardwareCursor"))`, applies `tui.maxInlineImages` and Kitty text-sizing settings, then creates persistent containers: - `chatContainer` - `pendingMessagesContainer` @@ -97,29 +97,22 @@ Routing details: This keeps key parsing/editor mechanics in `packages/tui` and mode semantics in coding-agent controllers. -## Render loop and diffing strategy +## Render loop and the append-only contract `TUI.requestRender()` coalesces render requests and rate-limits ordinary frames: -- forced renders (`requestRender(true, ...)`) schedule an immediate frame and set `#forceViewportRepaintOnNextRender`; with `clearScrollback`, they also queue `sessionReplace` +- forced renders (`requestRender(true, ...)`) schedule an immediate frame and force a full window rewrite; with `clearScrollback`, they trigger a destructive full paint (ED3 outside multiplexers) - ordinary renders schedule through `#scheduleRender()` and respect `TUI.#MIN_RENDER_INTERVAL_MS` - repeated requests while a render is pending collapse into the same scheduled frame `#doRender()` pipeline: -1. Render root component tree to `newLines`. -2. Composite visible overlays (if any). -3. Extract and strip `CURSOR_MARKER` from the visible viewport. -4. Normalize non-image lines and append reset/hyperlink terminators. -5. Classify the frame into a render intent: - - initial paint / forced viewport repaint - - explicit session replacement or native scrollback rebuild - - viewport repaint for width/height/offscreen mutations - - deferred mutation/shrink when native scrollback is scrolled - - trailing shrink - - changed-line diff - - noop -6. Emit only the bytes required by the intent and commit cached frame/cursor/viewport state. +1. Render root component tree, collecting the commit-boundary seam (`NativeScrollbackLiveRegion`) from the children. +2. Advance the append-only ledger: `windowTop = max(committedRows, frame.length - height)`, commit chunk = settled rows crossing the window top (never past the seam). +3. Extract and strip `CURSOR_MARKER`, normalize lines, slice the visible window, composite overlays into the window slice (screen coordinates; overlays freeze commits). +4. Emit one of: gesture-driven full paint (initial / session replace / resize), scroll-append (chunk rows only), in-window row diff, or seam rewrite (chunk + full window). + +Native scrollback always equals the committed frame prefix — rows enter history exactly once, in order, when the seam says they are final. There are no viewport probes and no deferred reconciliation; see [`tui-core-renderer.md`](./tui-core-renderer.md). Render writes use synchronized output mode (`CSI ? 2026 h/l`) when enabled; capability detection, DECRQM, or `PI_NO_SYNC_OUTPUT` can disable the wrappers while leaving autowrap discipline on. @@ -145,9 +138,8 @@ Resize events are event-driven from `ProcessTerminal` to `TUI.requestRender()`. Effects: -- Width or height changes repaint or rebuild because terminal reflow invalidates wrapping, viewport, and cursor anchors. -- Inside terminal multiplexers, resize uses viewport repaint instead of destructive native-scrollback replay; pane history cannot be erased safely and a full replay duplicates transcript rows. -- Viewport/top tracking (`#viewportTopRow`, `#maxLinesRendered`, scrollback high-water state) avoids invalid relative cursor math and defers destructive native scrollback rewrites while the user is scrolled into history. +- A resize is an explicit user gesture: outside multiplexers the engine erases and replays (`ED3` + full paint) so history rewraps at the new geometry; the commit ledger restarts from the replayed frame. +- Inside terminal multiplexers, resize repaints the visible window in place after a settle debounce (issue #2088); pane history keeps its old wrap, like any shell output, because pane scrollback cannot be erased safely. - Overlay visibility can depend on terminal dimensions (`OverlayOptions.visible`); focus is corrected when overlays become non-visible after resize. ## Streaming and incremental UI updates diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index f7a3c0119..250dcff1c 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -13,9 +13,14 @@ ### Fixed - Fixed the model selector dropping an immediate Enter when cached models were available but the selector's offline refresh was still pending. -- Removed transcript block freezing: blocks no longer replay a frozen render snapshot after crossing out of the live region, which could pin the visible window on stale or partial content (a tool stuck on its pending preview after a late result, an assistant message missing post-finalize updates like error pinning, collapsed/expanded toggles not reflected). Every block now renders its current content on every frame; the live-region seam still keeps still-mutating rows out of native scrollback, and history already committed to the terminal is never rewritten — the window simply always reflects the present state. +- Transcript block freezing is now unconditional instead of gated on ED3-risk terminal detection: every finalized block replays its frozen snapshot once it crosses out of the live region, on all terminals including Windows, because the rewritten renderer's committed scrollback is immutable everywhere. Still-mutating blocks (pending tools, streaming messages, async thinking renderers) anchor the live region and keep repainting until they finalize, which structurally fixes stale/duplicated output from late async expansions ([#1823](https://github.com/can1357/oh-my-pi/issues/1823)). - Fixed the edit tool's post-edit diff preview occasionally echoing a context line twice with out-of-order numbering. Block-boundary context injection classified space-prefixed diff rows as old-file-only, so an unchanged line sitting in a net-offset region (old N / new N+k) was missing from the new file's visibility window; `findBlockContextLines` then re-surfaced it under its post-edit number and the row was spliced in after the adjacent change run. New-file boundary lines are now translated back to pre-edit numbers (the compact-preview renumbering contract) and merged into a single old-numbered insertion pass — also fixing closers below a net-offset edit being dropped or renumbered incorrectly. +### Removed + +- Removed the `clearOnShrink` setting and its `PI_CLEAR_ON_SHRINK` environment variable: the rewritten renderer always clears shrunken rows exactly, so the flicker/perf tradeoff the setting controlled no longer exists. Existing config entries are ignored. +- Removed the prompt-submit native-scrollback reconciliation checkpoint and the eager streaming render mode from the interactive controllers — the renderer's append-only contract made both obsolete. + ## [15.10.9] - 2026-06-09 ### Fixed diff --git a/packages/coding-agent/test/event-controller-abort-render.test.ts b/packages/coding-agent/test/event-controller-abort-render.test.ts index f032e0033..9b5a44042 100644 --- a/packages/coding-agent/test/event-controller-abort-render.test.ts +++ b/packages/coding-agent/test/event-controller-abort-render.test.ts @@ -59,7 +59,7 @@ function createFixture(opts: { const ctx = { isInitialized: true, init: vi.fn(async () => {}), - ui: { requestRender, setEagerNativeScrollbackRebuild: vi.fn() }, + ui: { requestRender }, statusLine: { invalidate: vi.fn() }, updateEditorTopBorder: vi.fn(), streamingComponent, diff --git a/packages/coding-agent/test/event-controller-error-banner.test.ts b/packages/coding-agent/test/event-controller-error-banner.test.ts index 7400343cb..dcd2102dd 100644 --- a/packages/coding-agent/test/event-controller-error-banner.test.ts +++ b/packages/coding-agent/test/event-controller-error-banner.test.ts @@ -65,7 +65,7 @@ function createFixture(streamingMessage?: AssistantMessage) { const ctx = { isInitialized: true, init: vi.fn(async () => {}), - ui: { requestRender: vi.fn(), setEagerNativeScrollbackRebuild: vi.fn() }, + ui: { requestRender: vi.fn() }, statusLine: { invalidate: vi.fn() }, updateEditorTopBorder: vi.fn(), ensureLoadingAnimation: vi.fn(), diff --git a/packages/coding-agent/test/input-controller-skill-queue.test.ts b/packages/coding-agent/test/input-controller-skill-queue.test.ts index 36154cc31..b4ea43b0c 100644 --- a/packages/coding-agent/test/input-controller-skill-queue.test.ts +++ b/packages/coding-agent/test/input-controller-skill-queue.test.ts @@ -483,7 +483,7 @@ function createEventControllerFixtureForE10() { const ctx = { isInitialized: true, init: vi.fn(async () => {}), - ui: { requestRender, setEagerNativeScrollbackRebuild: vi.fn() }, + ui: { requestRender }, statusLine: { invalidate: vi.fn() }, updateEditorTopBorder: vi.fn(), addMessageToChat, diff --git a/packages/coding-agent/test/modes/controllers/event-controller-idle-compaction.test.ts b/packages/coding-agent/test/modes/controllers/event-controller-idle-compaction.test.ts index 39b37e38c..9fea85655 100644 --- a/packages/coding-agent/test/modes/controllers/event-controller-idle-compaction.test.ts +++ b/packages/coding-agent/test/modes/controllers/event-controller-idle-compaction.test.ts @@ -53,7 +53,7 @@ describe("EventController idle compaction teardown", () => { streamingMessage: undefined, pendingTools: new Map(), flushPendingModelSwitch: async () => {}, - ui: { requestRender: vi.fn(), setEagerNativeScrollbackRebuild: vi.fn() }, + ui: { requestRender: vi.fn() }, chatContainer: { removeChild: vi.fn() }, statusContainer: { clear: vi.fn() }, statusLine: { invalidate: vi.fn() }, diff --git a/packages/coding-agent/test/modes/controllers/event-controller-interrupt.test.ts b/packages/coding-agent/test/modes/controllers/event-controller-interrupt.test.ts index 88adf8539..5a4e6d9c9 100644 --- a/packages/coding-agent/test/modes/controllers/event-controller-interrupt.test.ts +++ b/packages/coding-agent/test/modes/controllers/event-controller-interrupt.test.ts @@ -21,7 +21,7 @@ function createContext() { setWorkingMessage, clearPinnedError: vi.fn(), ensureLoadingAnimation: vi.fn(), - ui: { setEagerNativeScrollbackRebuild: vi.fn(), requestRender: vi.fn() }, + ui: { requestRender: vi.fn() }, session: { getToolByName: () => undefined }, } as unknown as InteractiveModeContext; return { ctx, pendingTools, setWorkingMessage }; diff --git a/packages/coding-agent/test/modes/controllers/event-controller-message-start.test.ts b/packages/coding-agent/test/modes/controllers/event-controller-message-start.test.ts index f5a796393..cb576fd61 100644 --- a/packages/coding-agent/test/modes/controllers/event-controller-message-start.test.ts +++ b/packages/coding-agent/test/modes/controllers/event-controller-message-start.test.ts @@ -39,7 +39,7 @@ function createContext(options: { isInitialized: true, statusLine: { invalidate: vi.fn() }, updateEditorTopBorder: vi.fn(), - ui: { requestRender: vi.fn(), setEagerNativeScrollbackRebuild: vi.fn() }, + ui: { requestRender: vi.fn() }, editor, addMessageToChat, updatePendingMessagesDisplay, diff --git a/packages/coding-agent/test/modes/controllers/event-controller-read-grouping.test.ts b/packages/coding-agent/test/modes/controllers/event-controller-read-grouping.test.ts index 6f61483e2..3ea3122d4 100644 --- a/packages/coding-agent/test/modes/controllers/event-controller-read-grouping.test.ts +++ b/packages/coding-agent/test/modes/controllers/event-controller-read-grouping.test.ts @@ -73,7 +73,7 @@ function createFixture() { init: vi.fn(async () => {}), statusLine: { invalidate: vi.fn() }, updateEditorTopBorder: vi.fn(), - ui: { requestRender: vi.fn(), setEagerNativeScrollbackRebuild: vi.fn(), imageBudget: undefined }, + ui: { requestRender: vi.fn(), imageBudget: undefined }, chatContainer, pendingTools: new Map(), settings: { get: () => false }, diff --git a/packages/coding-agent/test/modes/controllers/event-controller-tool-render-mode.test.ts b/packages/coding-agent/test/modes/controllers/event-controller-tool-render-mode.test.ts deleted file mode 100644 index c63eb3d82..000000000 --- a/packages/coding-agent/test/modes/controllers/event-controller-tool-render-mode.test.ts +++ /dev/null @@ -1,125 +0,0 @@ -import { afterEach, beforeEach, describe, expect, it, vi } from "bun:test"; -import { resetSettingsForTest, Settings } from "@oh-my-pi/pi-coding-agent/config/settings"; -import { EventController } from "@oh-my-pi/pi-coding-agent/modes/controllers/event-controller"; -import type { InteractiveModeContext } from "@oh-my-pi/pi-coding-agent/modes/types"; -import type { AgentSessionEvent } from "@oh-my-pi/pi-coding-agent/session/agent-session"; - -function createContext() { - const setEagerNativeScrollbackRebuild = vi.fn(); - const ensureLoadingAnimation = vi.fn(); - const pendingTools = new Map(); - const chatContainer = { addChild: vi.fn(), removeChild: vi.fn() }; - const ctx = { - isInitialized: true, - settings: { get: () => false }, - statusLine: { invalidate: vi.fn() }, - updateEditorTopBorder: vi.fn(), - pendingTools, - chatContainer, - hideThinkingBlock: false, - editor: { getText: vi.fn(() => "") }, - flushPendingModelSwitch: vi.fn(), - sessionManager: { getSessionName: () => undefined }, - session: { - agent: { state: { messages: [] } }, - isCompacting: false, - isTtsrAbortPending: false, - retryAttempt: 0, - }, - ui: { setEagerNativeScrollbackRebuild, requestRender: vi.fn() }, - clearPinnedError: vi.fn(), - ensureLoadingAnimation, - } as unknown as InteractiveModeContext; - return { ctx, pendingTools, setEagerNativeScrollbackRebuild, ensureLoadingAnimation }; -} - -// A tool_execution_update for an id that is not pending is a no-op in its handler, -// so dispatching it exercises only the gated post-dispatch refresh in handleEvent — -// which is what syncs the TUI eager-rebuild flag to foreground-tool activity. -const REFRESH_TRIGGER = { - type: "tool_execution_update", - toolCallId: "not-pending", - partialResult: { content: [], details: {} }, -} as unknown as AgentSessionEvent; - -const ASSISTANT_MESSAGE = { - role: "assistant", - content: [{ type: "text", text: "" }], - api: "anthropic-messages", - provider: "anthropic", - model: "test-model", - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: 0, -} as const; - -describe("EventController tool render mode", () => { - beforeEach(async () => { - resetSettingsForTest(); - await Settings.init({ inMemory: true }); - }); - - afterEach(() => { - resetSettingsForTest(); - vi.restoreAllMocks(); - }); - - it("enables eager native scrollback rebuild before starting the idle Working loader", async () => { - const { ctx, ensureLoadingAnimation, setEagerNativeScrollbackRebuild } = createContext(); - const controller = new EventController(ctx); - - await controller.handleEvent({ type: "agent_start" } as unknown as AgentSessionEvent); - - expect(setEagerNativeScrollbackRebuild).toHaveBeenCalledWith(true); - expect(setEagerNativeScrollbackRebuild.mock.invocationCallOrder[0]!).toBeLessThan( - ensureLoadingAnimation.mock.invocationCallOrder[0]!, - ); - }); - it("enables eager native scrollback rebuild while a foreground tool is pending", async () => { - const { ctx, pendingTools, setEagerNativeScrollbackRebuild } = createContext(); - const controller = new EventController(ctx); - - pendingTools.set("call-1", {}); - await controller.handleEvent(REFRESH_TRIGGER); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(true); - - pendingTools.clear(); - await controller.handleEvent(REFRESH_TRIGGER); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(false); - }); - - it("enables eager native scrollback rebuild while assistant text is streaming", async () => { - const { ctx, setEagerNativeScrollbackRebuild } = createContext(); - const controller = new EventController(ctx); - - await controller.handleEvent({ - type: "message_start", - message: ASSISTANT_MESSAGE, - } as unknown as AgentSessionEvent); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(true); - - await controller.handleEvent({ type: "message_end", message: ASSISTANT_MESSAGE } as unknown as AgentSessionEvent); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(false); - }); - - it("resets eager native scrollback rebuild when a stream ends without assistant message_end", async () => { - const { ctx, setEagerNativeScrollbackRebuild } = createContext(); - const controller = new EventController(ctx); - - await controller.handleEvent({ - type: "message_start", - message: ASSISTANT_MESSAGE, - } as unknown as AgentSessionEvent); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(true); - - await controller.handleEvent({ type: "agent_end" } as unknown as AgentSessionEvent); - expect(setEagerNativeScrollbackRebuild).toHaveBeenLastCalledWith(false); - }); -}); diff --git a/packages/tui/CHANGELOG.md b/packages/tui/CHANGELOG.md index 36e88c3bb..5715bba51 100644 --- a/packages/tui/CHANGELOG.md +++ b/packages/tui/CHANGELOG.md @@ -1,6 +1,18 @@ # Changelog ## [Unreleased] +### Changed + +- Rewrote the render core around an append-only native-scrollback contract. Committed rows are immutable: rows enter terminal history exactly once, in order, when the component-reported commit boundary (`NativeScrollbackLiveRegion`) marks them final, and the visible window repaints in place with relative moves. The engine no longer probes the terminal's scroll position or guesses whether a destructive rebuild is safe — the entire ED3-risk/defer/checkpoint machinery (viewport probes, eager streaming mode, dirty-scrollback reconciliation, deferred shrink/mutation intents, streaming high-water rebuilds, ConPTY-specific defer paths) is deleted. ED3 (`CSI 3 J`) now fires only on explicit user gestures: session replace, resize outside multiplexers, and `resetDisplay()`. This structurally removes the yank / flash / duplicated-rows / invisible-until-resize failure families tracked across #1610, #1635, #1651, #1682, #1719, #1746, #1799, #1823, #1962, #1974, #2000, #2011, #2154. +- A frame that shrinks into its committed prefix re-anchors the visible window at the new tail and restarts commit bookkeeping; previously committed rows stay in history (history is never rewritten without a gesture). +- Overlays now composite into the visible window slice only and freeze commits while visible, so overlay pixels can never enter native scrollback and closing an overlay no longer triggers a destructive history rebuild. +- Inline-image budget demotion now deletes the demoted image's graphics by id and lets the window diff repaint the text fallback — no more mid-session destructive full replay when the image cap is exceeded. +- The render-stress harness now validates the contract with a shadow commit ledger (an independent reimplementation of the ledger math fed only by observed frames and bytes), asserting scrollback equals the committed prefix row-for-row across randomized op sequences, resizes, overlays, and multiplexer scenarios. + +### Removed + +- Removed the probe/defer API surface: `TUI.setEagerNativeScrollbackRebuild()`, `TUI.refreshNativeScrollbackIfDirty()`, `TUI.setClearOnShrink()`/`getClearOnShrink()`, `RenderRequestOptions.allowUnknownViewportMutation`, `NativeScrollbackRefreshOptions`, `Terminal.isNativeViewportAtBottom()`, `Terminal.hasEagerEraseScrollbackRisk()`, and the `eagerEraseScrollbackRisk`/`submitPinsViewportToTail` capability fields with their detectors. +- Removed the `PI_TUI_ED3_SAFE`, `PI_CLEAR_ON_SHRINK`, and `PI_TUI_DEBUG` environment variables (the levers they tuned no longer exist; `PI_DEBUG_REDRAW` now logs the commit-ledger state per frame). ## [15.10.9] - 2026-06-09 @@ -1228,4 +1240,4 @@ Initial release under @oh-my-pi scope. See previous releases at [badlogic/pi-mon ### Fixed -- **Readline-style Ctrl+W**: Now skips trailing whitespace before deleting the preceding word, matching standard readline behavior. ([#306](https://github.com/badlogic/pi-mono/pull/306) by [@kim0](https://github.com/kim0)) +- **Readline-style Ctrl+W**: Now skips trailing whitespace before deleting the preceding word, matching standard readline behavior. ([#306](https://github.com/badlogic/pi-mono/pull/306) by [@kim0](https://github.com/kim0)) \ No newline at end of file diff --git a/packages/tui/src/tui.ts b/packages/tui/src/tui.ts index f0054d170..f8877aead 100644 --- a/packages/tui/src/tui.ts +++ b/packages/tui/src/tui.ts @@ -1700,20 +1700,33 @@ export class TUI extends Container { if (fullPaint) { windowTop = Math.max(0, frameLength - height); chunkTo = Math.min(commitBoundary, windowTop); - } else if (geometryChanged) { - // Multiplexer resize: keep committed rows as-is (their old wrap - // stays in pane history, same as any shell output) and re-anchor - // the window at the new geometry. - windowTop = Math.max(this.#committedRows, frameLength - height, 0); - chunkTo = this.#committedRows; + } else if (frameLength <= this.#committedRows) { + // The frame shrank into (or below) the committed prefix: the app + // replaced content it had already let scroll into history without + // requesting a session replace (only possible without a live-region + // seam). History is immutable without a gesture, so the stale + // committed copy stays in scrollback; re-anchor the window at the + // tail and restart commit bookkeeping there so the live grid shows + // the real content instead of a blank pinned window. + windowTop = Math.max(0, frameLength - height); + chunkTo = Math.min(commitBoundary, windowTop); + this.#committedRows = chunkTo; } else { - windowTop = Math.max(prevWindowTop, frameLength - height); + // Re-anchor to the frame tail, floored at the committed boundary: a + // shrink (or overlay close) pulls the window back down, but never + // onto rows already in native history — re-showing those on the + // grid would duplicate them for a scrolling reader. On a + // multiplexer resize the pane reflowed its own history; committed + // rows keep their old wrap there, same as any shell output. + windowTop = Math.max(this.#committedRows, frameLength - height, 0); // Overlays freeze commits: composited rows must never enter // history, and the hidden gap backfills via the chunk once the - // overlay closes. - chunkTo = hasVisibleOverlay - ? this.#committedRows - : Math.max(this.#committedRows, Math.min(commitBoundary, windowTop)); + // overlay closes. A multiplexer resize also commits nothing — the + // pane keeps its own (old-wrap) history. + chunkTo = + hasVisibleOverlay || geometryChanged + ? this.#committedRows + : Math.max(this.#committedRows, Math.min(commitBoundary, windowTop)); } // 5. Extract the hardware-cursor marker (frame coordinates) before @@ -2246,6 +2259,7 @@ export class TUI extends Container { // In-window diff: nothing scrolls, nothing commits. if (chunkLength === 0 && scroll === 0) { + if (forceWindowRewrite) this.#fullRedrawCount += 1; let firstChanged = forceWindowRewrite ? 0 : -1; let lastChanged = forceWindowRewrite ? height - 1 : -1; if (!forceWindowRewrite) { @@ -2304,6 +2318,7 @@ export class TUI extends Container { // Cursor moves to the window top with a relative move; the chunk rows // pass through the screen and scroll off as the window rows are written // below them, so the rows entering scrollback are exactly the chunk. + this.#fullRedrawCount += 1; let buffer = this.#paintBeginSequence + purgeSequence; if (currentScreenRow > 0) buffer += `\x1b[${currentScreenRow}A`; buffer += "\r"; diff --git a/packages/tui/test/image-budget.test.ts b/packages/tui/test/image-budget.test.ts index f6b9d4c54..cc21a8627 100644 --- a/packages/tui/test/image-budget.test.ts +++ b/packages/tui/test/image-budget.test.ts @@ -364,7 +364,7 @@ describe("TUI inline-image budget", () => { ); } - it("hides the oldest image via a full redraw + graphics purge once a new image exceeds the cap", async () => { + it("purges demoted image graphics and repaints the fallback without a destructive replay", async () => { const term = new VirtualTerminal(40, 12); const writes: string[] = []; const realWrite = term.write.bind(term); @@ -381,7 +381,6 @@ describe("TUI inline-image budget", () => { try { tui.start(); await settle(term); - const redrawsBefore = tui.fullRedraws; writes.length = 0; // A second image arrives, exceeding the cap of 1. @@ -389,8 +388,10 @@ describe("TUI inline-image budget", () => { tui.requestRender(); await settle(term); - // The demotion forces at least one extra full redraw... - expect(tui.fullRedraws).toBeGreaterThan(redrawsBefore); + // The demotion never forces a destructive replay: committed + // placements are immutable, so no ED2/ED3 is emitted... + expect(writes.join("")).not.toContain("\x1b[2J"); + expect(writes.join("")).not.toContain("\x1b[3J"); // ...purges the now-hidden image's graphics by id... expect(writes.join("")).toContain(encodeKittyDeleteImage(oldId)); // ...and the oldest image is now shown as text, with one image still live. diff --git a/packages/tui/test/render-regressions.test.ts b/packages/tui/test/render-regressions.test.ts index 9ef51d6e2..98172ee88 100644 --- a/packages/tui/test/render-regressions.test.ts +++ b/packages/tui/test/render-regressions.test.ts @@ -84,25 +84,6 @@ class WrappingLinesComponent implements Component { } } -class FocusedInputComponent implements Component, Focusable { - focused = false; - #onInput: () => void; - - constructor(onInput: () => void) { - this.#onInput = onInput; - } - - handleInput(): void { - this.#onInput(); - } - - invalidate(): void {} - - render(): string[] { - return [this.focused ? `prompt>${CURSOR_MARKER}` : "prompt>"]; - } -} - class UnknownViewportTerminal extends VirtualTerminal { isNativeViewportAtBottom(): undefined { return undefined; @@ -470,8 +451,11 @@ describe("TUI terminal-state regressions", () => { } }); - it("does not yank a scrolled viewport for pure tail appends", async () => { - const term = new VirtualTerminal(20, 3, 5); + it("appends at the seam without yanking a scrolled reader", async () => { + // Law 7: the engine writes the same bytes regardless of scroll position — + // a pure tail append only commits rows at the seam and rewrites grid + // rows, so a reader scrolled into native scrollback keeps their view. + const term = new VirtualTerminal(20, 3, 100); const tui = new TUI(term); const component = new MutableLinesComponent(rows("L", 8)); tui.addChild(component); @@ -481,19 +465,26 @@ describe("TUI terminal-state regressions", () => { await settle(term); term.scrollLines(-1); - const beforePosition = term.getBufferPosition(); + const beforeViewportY = term.getBufferPosition().viewportY; const beforeView = visible(term); + const writes = captureWrites(term); component.setLines(rows("L", 9)); tui.requestRender(); await settle(term); - expect(term.getBufferPosition()).toEqual(beforePosition); + // No yank bytes: ordinary updates never home the cursor or clear. + const paint = writes.join(""); + expect(paint).not.toContain("\x1b[H"); + expect(paint).not.toContain("\x1b[2J"); + expect(paint).not.toContain("\x1b[3J"); + // The reader's anchor and view are untouched by the append. + expect(term.getBufferPosition().viewportY).toBe(beforeViewportY); expect(visible(term)).toEqual(beforeView); term.scrollLines(1_000_000); await term.flush(); - expect(term.getScrollBuffer().map(line => line.trimEnd())).toEqual(rows("L", 9).slice(1)); + expect(term.getScrollBuffer().map(line => line.trimEnd())).toEqual(rows("L", 9)); } finally { tui.stop(); } @@ -1450,7 +1441,7 @@ describe("TUI terminal-state regressions", () => { }); }); - it("tmux: offscreen shrink preserving the visible tail emits no repaint bytes", async () => { + it("tmux: shrink deleting a committed row repaints the window floored at the commit boundary", async () => { await withEnvPatch({ TMUX: "1", STY: undefined, ZELLIJ: undefined }, async () => { const term = new UnknownViewportTerminal(40, 4, 10_000); const tui = new TUI(term); @@ -1472,12 +1463,19 @@ describe("TUI terminal-state regressions", () => { expect(visible(term)).toEqual(["tail-0", "tail-1", "tail-2", "tail-3"]); const writes = captureWrites(term); + // Deleting "remove-me" (already committed to pane history) shifts + // every later row index up by one. Committed rows are immutable, so + // the window stays floored at the commit boundary (law 5): it shows + // the shorter tail plus a trailing blank instead of re-showing + // committed rows, and pane history keeps the stale copy. component.setLines(["old-0", "old-2", "old-3", "tail-0", "tail-1", "tail-2", "tail-3"]); tui.requestRender(); await settle(term); - expect(visible(term)).toEqual(["tail-0", "tail-1", "tail-2", "tail-3"]); - expect(writes).toEqual([]); + expect(visible(term)).toEqual(["tail-1", "tail-2", "tail-3", ""]); + expect(writes.join("")).not.toContain("\x1b[3J"); + const history = term.getScrollBuffer().slice(0, term.getBufferPosition().baseY); + expect(history.map(line => line.trimEnd())).toEqual(["old-0", "remove-me", "old-2", "old-3"]); } finally { tui.stop(); } @@ -1696,7 +1694,7 @@ describe("TUI terminal-state regressions", () => { tui.stop(); } }); - it("rebuilds history when offscreen expansion and append land together", async () => { + it("keeps an offscreen expansion out of history while seam commits continue in order", async () => { const term = new VirtualTerminal(32, 6); const tui = new TUI(term); const component = new MutableLinesComponent(["status-0", ...rows("line-", 11)]); @@ -1714,6 +1712,9 @@ describe("TUI terminal-state regressions", () => { "line-10", ]); + // Rows 0..5 (status-0, line-0..line-4) are committed. The frame edits + // row 0 and inserts a row above the commit boundary while a tail + // append lands in the same frame. component.setLines(["status-1", "expanded-details", ...rows("line-", 12)]); tui.requestRender(); await settle(term); @@ -1726,12 +1727,18 @@ describe("TUI terminal-state regressions", () => { "line-10", "line-11", ]); - const scrollback = term.getScrollBuffer(); - expect(scrollback.join("\n")).toContain("expanded-details"); - for (let i = 0; i < 12; i++) { - const pattern = new RegExp(`\\bline-${i}\\b`); - expect(countMatches(scrollback, pattern), `line-${i} should appear exactly once`).toBe(1); - } + const buffer = term.getScrollBuffer().map(line => line.trimEnd()); + const history = buffer.slice(0, term.getBufferPosition().baseY); + // Committed rows are immutable: the offscreen edit and insertion never + // reach native history. + expect(history).not.toContain("status-1"); + expect(history).not.toContain("expanded-details"); + // Rows that crossed the seam this frame committed with their content + // at commit time (law 1): the two-row insertion shifted indices, so + // the newly committed rows repeat line-4/line-5 — accepted artifact. + expect(history).toEqual(["status-0", ...rows("line-", 5), "line-4", "line-5"]); + // The appended tail row reaches the screen exactly once. + expect(buffer.filter(row => row === "line-11").length).toBe(1); } finally { tui.stop(); } @@ -1750,28 +1757,33 @@ describe("TUI terminal-state regressions", () => { expect(term.isNativeViewportAtBottom()).toBe(true); expect(visible(term).map(line => line.trim())).toEqual(["a", "b", "c", "d"]); - // An offscreen edit (E0 -> E0x, above the viewport top) lands together - // with a tail append whose rows make the prior last line "d" recur one - // row early. The append-tail heuristic then mis-locates the tail and, - // before the fix, scrolled an extra row into history — duplicating the - // viewport-top row "b" just above the viewport. + // An offscreen edit (E0 -> E0x, above the commit boundary) lands + // together with a tail append whose rows make the prior last line "d" + // recur one row early. The seam must advance by exactly the growth — + // committing one row — without duplicating the viewport-top row "b". component.setLines(["E0x", "E1", "a", "b", "d", "e", "f"]); tui.requestRender(); await settle(term); expect(visible(term).map(line => line.trim())).toEqual(["b", "d", "e", "f"]); const buffer = term.getScrollBuffer().map(line => line.trimEnd()); - for (const line of ["E0x", "E1", "a", "b", "d", "e", "f"]) { + for (const line of ["E1", "a", "b", "d", "e", "f"]) { expect(buffer.filter(row => row === line).length, `${line} should appear exactly once`).toBe(1); } - // The offscreen edit must be reflected in history, not left stale. - expect(buffer).not.toContain("E0"); + // Committed rows are immutable: the stale "E0" copy survives in + // history and the offscreen edit "E0x" never paints anywhere. + expect(buffer.filter(row => row === "E0").length).toBe(1); + expect(buffer).not.toContain("E0x"); } finally { tui.stop(); } }); - it("removes collapsed ctrl-o markers from scrollback after offscreen expansion", async () => { + it("keeps stale collapsed ctrl-o markers in history while the expansion repaints the window", async () => { + // Committed marker rows are immutable: a Ctrl+O expansion repaints the + // live window, but the collapsed markers that already scrolled into + // native history stay there — one stale copy each, never rewritten and + // never duplicated (accepted artifact). const term = new VirtualTerminal(48, 6); const tui = new TUI(term); const collapsedLines = [ @@ -1803,12 +1815,23 @@ describe("TUI terminal-state regressions", () => { tui.requestRender(); await settle(term); + expect(visible(term).map(line => line.trim())).toEqual([ + "json-6", + "json-7", + "json-8", + "json-9", + "status", + "editor", + ]); const scrollback = term.getScrollBuffer(); const scrollbackText = scrollback.join("\n"); - expect(scrollbackText).not.toContain("ctrl+o"); - expect(scrollbackText).toContain("code line 1"); - expect(scrollbackText).toContain("output line 1"); - for (let i = 0; i < 10; i++) { + expect(countMatches(scrollback, /Ctrl\+O: Expand/)).toBe(1); + expect(countMatches(scrollback, /ctrl\+o/)).toBe(1); + // The expanded rows sit above the commit boundary: they never paint. + expect(scrollbackText).not.toContain("code line"); + expect(scrollbackText).not.toContain("output line"); + // Rows still below the boundary appear exactly once. + for (let i = 6; i < 10; i++) { const pattern = new RegExp(`\\bjson-${i}\\b`); expect(countMatches(scrollback, pattern), `json-${i} should appear exactly once`).toBe(1); } @@ -1861,12 +1884,11 @@ describe("TUI terminal-state regressions", () => { } }); - it("rebuilds scrollback on a user-driven offscreen expansion when the viewport position is unknown", async () => { - // Pressing Ctrl+O is a direct user keystroke. On a terminal that cannot - // report viewport position (POSIX), the offscreen structural mutation must - // still promote to a clean history rebuild instead of a partial viewport - // repaint — otherwise the collapsed preview rows linger above the fold and - // the expansion renders garbled. + it("paints an offscreen expansion identically when the viewport probe is unavailable", async () => { + // Law 3: there is no probe and no platform fork. A terminal that cannot + // report its native viewport position gets exactly the same treatment as + // one that can: committed rows stay immutable, the live window repaints, + // and no clear/home bytes are emitted. const term = new UnknownViewportTerminal(48, 6); const tui = new TUI(term); const component = new MutableLinesComponent([ @@ -1885,6 +1907,7 @@ describe("TUI terminal-state regressions", () => { expect(term.isNativeViewportAtBottom()).toBeUndefined(); expect(term.getScrollBuffer().join("\n")).toContain("ctrl+o"); + const writes = captureWrites(term); component.setLines([ "frame-top", "code line 0", @@ -1898,15 +1921,19 @@ describe("TUI terminal-state regressions", () => { tui.requestRender(); await settle(term); - const scrollback = term.getScrollBuffer(); - const scrollbackText = scrollback.join("\n"); - expect(scrollbackText).not.toContain("ctrl+o"); - expect(scrollbackText).toContain("code line 1"); - expect(scrollbackText).toContain("output line 1"); - for (let i = 0; i < 10; i++) { - const pattern = new RegExp(`\\bjson-${i}\\b`); - expect(countMatches(scrollback, pattern), `json-${i} should appear exactly once`).toBe(1); - } + const paint = writes.join(""); + expect(paint).not.toContain("\x1b[3J"); + expect(paint).not.toContain("\x1b[2J"); + expect(visible(term).map(line => line.trim())).toEqual([ + "json-6", + "json-7", + "json-8", + "json-9", + "status", + "editor", + ]); + // History is never rewritten: the stale markers survive offscreen. + expect(term.getScrollBuffer().join("\n")).toContain("ctrl+o"); } finally { tui.stop(); } @@ -1961,7 +1988,11 @@ describe("TUI terminal-state regressions", () => { } }); - it("rebuilds scrollback when a bottom-anchored high-water preview collapses", async () => { + it("re-anchors a bottom-anchored high-water collapse and keeps stale history above", async () => { + // Law 4: the collapse shrinks the frame below the committed count, so the + // engine re-anchors the window at the new tail and resets its commit + // counter. Native history is append-only: the high-water preview copy + // stays in scrollback above (accepted artifact) — never clawed back. const term = new VirtualTerminal(40, 5); const highWaterFrame = [...rows("base-", 8), ...rows("preview-", 10)]; const finalFrame = [...rows("base-", 8), "result-0", "result-1"]; @@ -1975,12 +2006,21 @@ describe("TUI terminal-state regressions", () => { expect(term.getScrollBuffer().map(line => line.trimEnd())).toEqual(highWaterFrame); expect(term.getBufferPosition().viewportY).toBe(term.getBufferPosition().baseY); + const writes = captureWrites(term); component.setLines(finalFrame); tui.requestRender(); await settle(term); - expect(term.getScrollBuffer().map(line => line.trimEnd())).toEqual(finalFrame); - expect(term.getScrollBuffer().join("\n")).not.toContain("preview-"); + expect(writes.join("")).not.toContain("\x1b[3J"); + expect(visible(term).map(line => line.trim())).toEqual([ + "base-5", + "base-6", + "base-7", + "result-0", + "result-1", + ]); + const history = term.getScrollBuffer().slice(0, term.getBufferPosition().baseY); + expect(history.map(line => line.trimEnd())).toEqual(highWaterFrame.slice(0, 13)); } finally { tui.stop(); } @@ -2012,7 +2052,11 @@ describe("TUI terminal-state regressions", () => { } }); - it("defers offscreen expansion while native scrollback is scrolled", async () => { + it("writes the same offscreen-expansion bytes while the reader is parked in scrollback", async () => { + // Law 7: the engine neither knows nor cares where the reader is — an + // offscreen expansion only appends commits at the seam and rewrites grid + // rows, so a reader scrolled fully into native scrollback keeps a stable + // view and is never yanked. const term = new VirtualTerminal(32, 5); const tui = new TUI(term); const component = new MutableLinesComponent(rows("line-", 12)); @@ -2021,18 +2065,24 @@ describe("TUI terminal-state regressions", () => { try { tui.start(); await settle(term); - term.scrollLines(-2); + term.scrollLines(-5); const before = term.getBufferPosition(); - expect(before.viewportY).toBeGreaterThan(0); - expect(visible(term).map(line => line.trim())).toEqual(["line-5", "line-6", "line-7", "line-8", "line-9"]); + expect(before.viewportY).toBe(2); + expect(visible(term).map(line => line.trim())).toEqual(["line-2", "line-3", "line-4", "line-5", "line-6"]); + const writes = captureWrites(term); component.setLines(["line-0", "line-1", "expanded-0", "expanded-1", ...rows("line-", 12).slice(2)]); tui.requestRender(); await settle(term); - const after = term.getBufferPosition(); - expect(after.viewportY).toBe(before.viewportY); - expect(visible(term).map(line => line.trim())).toEqual(["line-5", "line-6", "line-7", "line-8", "line-9"]); + const paint = writes.join(""); + expect(paint).not.toContain("\x1b[3J"); + expect(paint).not.toContain("\x1b[2J"); + expect(paint).not.toContain("\x1b[H"); + expect(term.getBufferPosition().viewportY).toBe(before.viewportY); + expect(visible(term).map(line => line.trim())).toEqual(["line-2", "line-3", "line-4", "line-5", "line-6"]); + // The expansion rows sit above the commit boundary: they never enter + // history — not now, and not after the reader returns to the bottom. expect(term.getScrollBuffer().join("\n")).not.toContain("expanded-0"); term.scrollLines(999); @@ -2041,13 +2091,24 @@ describe("TUI terminal-state regressions", () => { const finalPosition = term.getBufferPosition(); expect(finalPosition.viewportY).toBe(finalPosition.baseY); - expect(term.getScrollBuffer().join("\n")).toContain("expanded-0"); + expect(term.getScrollBuffer().join("\n")).not.toContain("expanded-0"); + expect(visible(term).map(line => line.trim())).toEqual([ + "line-7", + "line-8", + "line-9", + "line-10", + "line-11", + ]); } finally { tui.stop(); } }); - it("defers height-changing tail preview while native scrollback is scrolled", async () => { + it("paints a height-changing tail preview in the window while the reader is parked in scrollback", async () => { + // Same law-7 shape as above, but the inserted row lands BELOW the commit + // boundary: it paints into the live window immediately. The scrolled + // reader still gets the same bytes — no clear, no yank — and finds the + // preview row waiting in the window when they scroll back down. const term = new VirtualTerminal(32, 5); const tui = new TUI(term); const component = new MutableLinesComponent(rows("line-", 12)); @@ -2056,27 +2117,35 @@ describe("TUI terminal-state regressions", () => { try { tui.start(); await settle(term); - term.scrollLines(-2); + term.scrollLines(-5); const before = term.getBufferPosition(); - expect(before.viewportY).toBeGreaterThan(0); - expect(visible(term).map(line => line.trim())).toEqual(["line-5", "line-6", "line-7", "line-8", "line-9"]); + expect(before.viewportY).toBe(2); + expect(visible(term).map(line => line.trim())).toEqual(["line-2", "line-3", "line-4", "line-5", "line-6"]); + const writes = captureWrites(term); component.setLines([...rows("line-", 9), "preview-appeared", ...rows("line-", 12).slice(9)]); tui.requestRender(); await settle(term); - const after = term.getBufferPosition(); - expect(after.viewportY).toBe(before.viewportY); - expect(visible(term).map(line => line.trim())).toEqual(["line-5", "line-6", "line-7", "line-8", "line-9"]); - expect(term.getScrollBuffer().join("\n")).not.toContain("preview-appeared"); + const paint = writes.join(""); + expect(paint).not.toContain("\x1b[3J"); + expect(paint).not.toContain("\x1b[2J"); + expect(paint).not.toContain("\x1b[H"); + expect(term.getBufferPosition().viewportY).toBe(before.viewportY); + expect(visible(term).map(line => line.trim())).toEqual(["line-2", "line-3", "line-4", "line-5", "line-6"]); + // The preview painted into the live grid, not into history. + const history = term.getScrollBuffer().slice(0, term.getBufferPosition().baseY); + expect(history.join("\n")).not.toContain("preview-appeared"); term.scrollLines(999); - tui.requestRender(); - await settle(term); - - const finalPosition = term.getBufferPosition(); - expect(finalPosition.viewportY).toBe(finalPosition.baseY); - expect(term.getScrollBuffer().join("\n")).toContain("preview-appeared"); + await term.flush(); + expect(visible(term).map(line => line.trim())).toEqual([ + "line-8", + "preview-appeared", + "line-9", + "line-10", + "line-11", + ]); } finally { tui.stop(); } @@ -2110,39 +2179,6 @@ describe("TUI terminal-state regressions", () => { } }); - it("keeps the unknown Windows viewport guard on ordinary focused input", async () => { - const originalPlatform = process.platform; - Object.defineProperty(process, "platform", { configurable: true, value: "win32" }); - const term = new UnknownViewportTerminal(32, 5); - const tui = new TUI(term); - const transcript = new MutableLinesComponent(rows("line-", 12)); - const input = new FocusedInputComponent(() => { - transcript.setLines([...rows("line-", 6), "typed-token", ...rows("line-", 12).slice(6)]); - }); - tui.addChild(transcript); - tui.addChild(input); - tui.setFocus(input); - - try { - tui.start(); - await settle(term); - term.scrollLines(-2); - const before = term.getBufferPosition(); - const beforeViewport = visible(term).map(line => line.trim()); - expect(before.viewportY).toBeGreaterThan(0); - - term.sendInput("x"); - await settle(term); - - const after = term.getBufferPosition(); - expect(after.viewportY).toBe(before.viewportY); - expect(visible(term).map(line => line.trim())).toEqual(beforeViewport); - expect(term.getScrollBuffer().join("\n")).not.toContain("typed-token"); - } finally { - Object.defineProperty(process, "platform", { configurable: true, value: originalPlatform }); - tui.stop(); - } - }); it("defers bottom-anchored shrink when POSIX viewport state is unknown", async () => { // Repro for #1566 follow-up (kitty/Linux): a bottom-anchored shrink across the // viewport boundary used to fall through to `viewportRepaint`, which redrew the @@ -2244,13 +2280,12 @@ describe("TUI terminal-state regressions", () => { scrolledTui.stop(); } }); - it("rebuilds history when a shrink leaves no real rows above the scrollback boundary", async () => { - // Reviewer scenario (#1599): a large completion-style collapse (e.g. a 100-row - // streamed transcript shrinking to a 20-row final cell in a 10-row viewport) - // must NOT use the padded `deferredShrink` — the viewport would fall entirely - // past the end of `newLines` and render as all blanks (no prompt visible) until - // the next checkpoint. Yank the scrollback instead so the new tail stays on - // screen. + it("re-anchors a huge completion-style collapse at the new tail and keeps stale history", async () => { + // Law 4 (#1599 lineage): a 100-row transcript collapsing to 20 rows in a + // 10-row window must keep the new tail (including the prompt) on screen. + // The window re-anchors at `newLength - height` and the commit counter + // resets; the committed line-* rows remain in native history above as an + // accepted stale artifact — history is never rewritten. const term = new UnknownViewportTerminal(40, 10); const tui = new TUI(term); const body = rows("line-", 99); @@ -2279,60 +2314,76 @@ describe("TUI terminal-state regressions", () => { "short-18", "prompt-row", ]); - const scrollback = term.getScrollBuffer(); - for (let i = 0; i < short.length; i++) { - const pattern = new RegExp(`\\bshort-${i}\\b`); - expect(countMatches(scrollback, pattern), `short-${i} appears once`).toBe(1); + const buffer = term.getScrollBuffer(); + // The re-anchored window is the only place short-* rows exist; rows + // 0..9 of the new frame fall under the stale committed prefix and are + // never painted at all. + for (let i = 10; i < short.length; i++) { + expect(countMatches(buffer, new RegExp(`\\bshort-${i}\\b`)), `short-${i} appears once`).toBe(1); } - expect(scrollback.join("\n")).not.toContain("line-"); + for (let i = 0; i < 10; i++) { + expect(countMatches(buffer, new RegExp(`\\bshort-${i}\\b`)), `short-${i} never paints`).toBe(0); + } + const history = buffer.slice(0, term.getBufferPosition().baseY).map(line => line.trimEnd()); + expect(history).toEqual(body.slice(0, 90)); } finally { tui.stop(); } }); - it("defers ED3-risk huge shrink while unknown viewport is scrolled", async () => { - // The huge-shrink fallback normally prefers `historyRebuild` over a blank - // padded viewport. On terminals where ED3 can move an unobservable - // scrollback viewport, that fallback is worse: it yanks the reader to the - // top. Keep the old visible history frozen and rebuild only at checkpoint. - const originalPlatform = process.platform; - Object.defineProperty(process, "platform", { configurable: true, value: "linux" }); + it("re-anchors a huge collapse without clears while the reader is parked in scrollback", async () => { + // Law 4 + law 2: even a 100→20 row collapse never emits ED2/ED3 — the + // re-anchor is a window rewrite, so a reader parked in native scrollback + // keeps a byte-stable view and their anchor. + const term = new UnknownViewportTerminal(40, 10); + const tui = new TUI(term); + const body = rows("line-", 99); + const component = new MutableLinesComponent([...body, "prompt-row"]); + tui.addChild(component); + try { - const term = new UnknownViewportTerminal(40, 10); - const tui = new TUI(term); - const body = rows("line-", 99); - const component = new MutableLinesComponent([...body, "prompt-row"]); - tui.addChild(component); + tui.start(); + await settle(term); + term.scrollLines(-10); + const before = term.getBufferPosition(); + const beforeViewport = visible(term).map(line => line.trim()); + expect(before.viewportY).toBeGreaterThan(0); - try { - tui.start(); - await settle(term); - term.scrollLines(-2); - const before = term.getBufferPosition(); - const beforeViewport = visible(term).map(line => line.trim()); - expect(before.viewportY).toBeGreaterThan(0); + const writes = captureWrites(term); + const short = rows("short-", 19); + component.setLines([...short, "prompt-row"]); + tui.requestRender(); + await settle(term); - const short = rows("short-", 19); - component.setLines([...short, "prompt-row"]); - tui.requestRender(); - await settle(term); + const paint = writes.join(""); + expect(paint).not.toContain("\x1b[3J"); + expect(paint).not.toContain("\x1b[2J"); + expect(term.getBufferPosition().viewportY).toBe(before.viewportY); + expect(visible(term).map(line => line.trim())).toEqual(beforeViewport); - const after = term.getBufferPosition(); - expect(after.viewportY).toBe(before.viewportY); - expect(visible(term).map(line => line.trim())).toEqual(beforeViewport); - expect(term.getScrollBuffer().join("\n")).not.toContain("short-"); - - term.scrollLines(999); - await settle(term); - expect(term.getScrollBuffer().join("\n")).not.toContain("short-"); - } finally { - tui.stop(); - } + term.scrollLines(999); + await settle(term); + expect(visible(term).map(line => line.trim())).toEqual([ + "short-10", + "short-11", + "short-12", + "short-13", + "short-14", + "short-15", + "short-16", + "short-17", + "short-18", + "prompt-row", + ]); + // Stale committed history stays above — never clawed back. + const history = term.getScrollBuffer().slice(0, term.getBufferPosition().baseY); + expect(history.join("\n")).toContain("line-89"); + expect(history.join("\n")).not.toContain("short-"); } finally { - Object.defineProperty(process, "platform", { configurable: true, value: originalPlatform }); + tui.stop(); } }); - it("rebuilds history when prior POSIX repaint left the padded viewport past the new tail", async () => { + it("re-anchors a shrink after an offscreen-edit grow advanced the commit boundary", async () => { const term = new UnknownViewportTerminal(40, 10); const tui = new TUI(term); const initial = rows("line-", 19); @@ -2343,11 +2394,9 @@ describe("TUI terminal-state regressions", () => { tui.start(); await settle(term); - // Unknown-POSIX offscreen mutation: repainting the viewport commits the - // 120-row logical frame, but `#emitViewportRepaint` intentionally does not - // advance `#scrollbackHighWater` (it remains at the original 20-row frame's - // 10-row overflow). The later shrink must compare against the padded viewport - // top (`120 - height`) rather than the stale high-water mark. + // Offscreen edit (row 0) + 100-row growth in one frame: the edit is + // ignored (row 0 is committed) while the growth advances the commit + // boundary — rows commit with their content at commit time (law 1). const expanded = ["edited-line", ...rows("line-", 118), "prompt-row"]; component.setLines(expanded); tui.requestRender(); @@ -2364,7 +2413,10 @@ describe("TUI terminal-state regressions", () => { "line-117", "prompt-row", ]); + expect(term.getScrollBuffer().join("\n")).not.toContain("edited-line"); + // Collapse far below the commit boundary: law 4 re-anchors the window + // at the new tail; the stale committed transcript stays above. const short = [...rows("short-", 14), "prompt-row"]; component.setLines(short); tui.requestRender(); @@ -2382,7 +2434,10 @@ describe("TUI terminal-state regressions", () => { "short-13", "prompt-row", ]); - expect(term.getScrollBuffer().join("\n")).not.toContain("line-"); + const history = term.getScrollBuffer().slice(0, term.getBufferPosition().baseY).join("\n"); + expect(history).toContain("line-108"); + expect(history).not.toContain("short-"); + expect(history).not.toContain("edited-line"); } finally { tui.stop(); } @@ -2787,86 +2842,6 @@ describe("TUI terminal-state regressions", () => { } }); - it("still defers when the native viewport probe confirms a scrolled-up reader", async () => { - // Counterpart to the two paints above: when the probe is *reliable* and reports - // `false`, the reader is parked in scrollback and a live-frame write is wasted. - // `deferredMutation` (a no-op) must stay in place so the next checkpoint can - // reconcile cleanly, and no bytes hit the terminal during the deferred frame. - const originalPlatform = process.platform; - Object.defineProperty(process, "platform", { configurable: true, value: "win32" }); - try { - await withEnvPatch( - { WT_SESSION: undefined, TMUX: undefined, STY: undefined, ZELLIJ: undefined }, - async () => { - const term = new VirtualTerminal(32, 5); - const tui = new TUI(term); - const transcript = new MutableLinesComponent(rows("seed-", 5)); - const status = new MutableLinesComponent(["STATUS-OLD"]); - const prompt = new MutableLinesComponent(["prompt>"]); - tui.addChild(transcript); - tui.addChild(status); - tui.addChild(prompt); - - try { - tui.start(); - await settle(term); - - // Pin the probe to a confirmed-scrolled answer (host reports `false`). - (term as unknown as { isNativeViewportAtBottom: () => boolean }).isNativeViewportAtBottom = () => - false; - - const writes: string[] = []; - const realWrite = term.write.bind(term); - (term as unknown as { write: (s: string) => void }).write = (data: string) => { - writes.push(data); - realWrite(data); - }; - - // Same structural mutation as the slash-command test — but with the probe - // telling us the user can't see the live frame, the planner stays a no-op. - status.setLines(["STATUS-NEW", "EXTRA"]); - tui.requestRender(); - await settle(term); - - // Zero bytes written — the deferral is intentional and protects the reader. - expect(writes.join("")).toBe(""); - } finally { - tui.stop(); - } - }, - ); - } finally { - Object.defineProperty(process, "platform", { configurable: true, value: originalPlatform }); - } - }); - - it("refreshes deferred native scrollback when the native viewport reaches bottom", async () => { - const term = new VirtualTerminal(32, 5); - const tui = new TUI(term); - const component = new MutableLinesComponent(rows("line-", 12)); - tui.addChild(component); - - try { - tui.start(); - await settle(term); - term.scrollLines(-2); - - component.setLines(rows("line-", 8)); - tui.requestRender(); - await settle(term); - - term.scrollLines(999); - tui.requestRender(); - await settle(term); - - const position = term.getBufferPosition(); - expect(position.viewportY).toBe(position.baseY); - expect(visible(term).map(line => line.trim())).toEqual(["line-3", "line-4", "line-5", "line-6", "line-7"]); - } finally { - tui.stop(); - } - }); - it("keeps transient checkpoint rows out of clean rebuilt scrollback", async () => { const term = new VirtualTerminal(32, 5); const tui = new TUI(term); @@ -2900,11 +2875,11 @@ describe("TUI terminal-state regressions", () => { } }); - it("tail-cell mutation is cleaned up before the next native scrollback checkpoint", async () => { - // Once a header has scrolled into terminal history, a bottom-anchored - // tail cell shrink must rebuild immediately. Deferring until the next - // checkpoint leaves stale high-water rows above the viewport and duplicates - // retained header/tail rows when users scroll back. + it("repaints a tail-cell mutation inside the window on the next ordinary frame", async () => { + // Once header rows have scrolled into native history they are immutable; + // a tail cell collapsing and regrowing inside the live window repaints + // immediately on the next frame — there is no deferred reconciliation + // pass — and the seam only ever advances, so nothing duplicates. const term = new VirtualTerminal(40, 10); const tui = new TUI(term); const header = new MutableLinesComponent(["HEADER-0", "HEADER-1", "HEADER-2", "HEADER-3", "HEADER-4"]); @@ -2916,7 +2891,8 @@ describe("TUI terminal-state regressions", () => { tui.start(); await settle(term); - // Stream output until the transcript exceeds the viewport. + // Stream output until the transcript exceeds the viewport and the + // header rows are committed. const out: string[] = []; for (let i = 0; i < 15; i++) { out.push(`cell-${i}`); @@ -2925,34 +2901,53 @@ describe("TUI terminal-state regressions", () => { await settle(term); } - // Repeatedly shrink (collapse preview) and grow (more output) - // across the previous viewport bottom. This is what triggers - // the duplication: each shrink-then-grow cycle would otherwise - // re-emit HEADER rows that are already in scrollback. - for (let cycle = 0; cycle < 6; cycle++) { - tail.setLines([...out.slice(0, 5), "[summary]", "[footer]"]); - tui.requestRender(); - await settle(term); - - out.push(`cell-grew-${cycle}-a`, `cell-grew-${cycle}-b`); - tail.setLines([...out, "[footer]"]); - tui.requestRender(); - await settle(term); - } - - // Final completion-style collapse: the rebuild happens on this render - // while the viewport is bottom-anchored. - tail.setLines(["[completed: many lines]", "[footer]"]); + // Collapse the streamed preview into a summary. The collapse stays + // below the commit boundary, so the window repaints on this very + // frame: shorter tail plus trailing blanks (law 5). + tail.setLines([...out.slice(0, 8), "[summary]", "[footer]"]); tui.requestRender(); await settle(term); - term.scrollLines(999); + expect(visible(term).map(line => line.trim())).toEqual([ + "cell-6", + "cell-7", + "[summary]", + "[footer]", + "", + "", + "", + "", + "", + "", + ]); + + // Regrow: streaming resumes, the summary disappears before its row + // ever reaches the seam, and commits continue in order, exactly once. + out.push("cell-15", "cell-16"); + tail.setLines([...out, "[footer]"]); + tui.requestRender(); await settle(term); + + expect(visible(term).map(line => line.trim())).toEqual([ + "cell-8", + "cell-9", + "cell-10", + "cell-11", + "cell-12", + "cell-13", + "cell-14", + "cell-15", + "cell-16", + "[footer]", + ]); const scrollback = term.getScrollBuffer(); + expect(scrollback.join("\n")).not.toContain("[summary]"); for (let i = 0; i < 5; i++) { const pattern = new RegExp(`\\bHEADER-${i}\\b`); - expect(countMatches(scrollback, pattern), `HEADER-${i} should appear at most once`).toBeLessThanOrEqual( - 1, - ); + expect(countMatches(scrollback, pattern), `HEADER-${i} appears exactly once`).toBe(1); + } + for (let i = 0; i < 17; i++) { + const pattern = new RegExp(`\\bcell-${i}\\b`); + expect(countMatches(scrollback, pattern), `cell-${i} appears exactly once`).toBe(1); } } finally { tui.stop(); @@ -4061,21 +4056,15 @@ describe("foreground-tool streaming on ED3-risk terminals", () => { }); // Repro of the "injected notification chip renders over the active tool - // render" report. A foreground tool (an active `write`) streams on an - // ED3-risk terminal (ghostty/kitty/…) whose viewport position is - // unobservable. Its header carries a live elapsed-time counter that ticks - // every frame; once output scrolls it above the viewport top, each tick is an - // OFFSCREEN edit. An offscreen-edit-with-growth frame repaints the viewport in place - // (`viewportRepaint`) — advancing the rendered line count WITHOUT committing - // the new overflow to native history. `#scrollbackHighWater` then lags the - // logical viewport top. A later shrink whose changes land in the visible - // region finds `naturalViewportTop >= #scrollbackHighWater`, slips past the - // shrink-across-boundary guard, and reaches the diff emitter, which anchors to - // `#maxLinesRendered - height`: it rewrites only the suffix, drops the newly - // exposed top row, and leaves a blank at the bottom — so every row below the - // edit renders one row too high, painting over the rows above. The shrink must - // instead re-anchor the bottom-anchored viewport. - it("re-anchors a visible-region shrink after an offscreen-edit grow lags native history", async () => { + // render" report. A foreground tool streams while its header (carrying a + // ticking elapsed-time counter) has scrolled above the window top. The tick + // is an offscreen edit — committed rows are immutable, so it is ignored — + // while injected chips grow the frame and advance the commit boundary. When + // a visible chip then collapses, the window cannot re-show committed rows + // (law 5: that would visually duplicate them for a scrolling reader): it + // stays floored at the commit boundary and shows the shorter tail with a + // trailing blank row instead of drifting content upward over the rows above. + it("floors a visible-region shrink at the commit boundary after an offscreen-edit grow", async () => { const term = new UnknownViewportTerminal(40, 6); const tui = new TUI(term); // done-* are completed messages that have scrolled into history; the @@ -4134,7 +4123,9 @@ describe("foreground-tool streaming on ED3-risk terminals", () => { // Frame C: a visible chip collapses (a shrink whose first change lands in // the visible region) while the header does NOT tick this frame. The - // viewport must re-anchor one row up, not drift its content upward. + // window stays floored at the commit boundary: chip-0 committed when the + // chips scrolled the seam forward, so the shorter tail renders with a + // trailing blank row rather than re-showing chip-0. const frameC = [ "done-0", "done-1", @@ -4156,7 +4147,7 @@ describe("foreground-tool streaming on ED3-risk terminals", () => { component.setLines(frameC); tui.requestRender(); await term.waitForRender(); - expect(visible(term)).toEqual(["chip-0", "chip-1", "chip-2", "loader", "todos", "editor"]); + expect(visible(term)).toEqual(["chip-1", "chip-2", "loader", "todos", "editor", ""]); } finally { tui.stop(); } diff --git a/packages/tui/test/render-stress-harness.ts b/packages/tui/test/render-stress-harness.ts index 41e6d766d..fea8c53ae 100644 --- a/packages/tui/test/render-stress-harness.ts +++ b/packages/tui/test/render-stress-harness.ts @@ -303,6 +303,7 @@ interface Snapshot { height: number; frame: string[]; atBottom: boolean; + shadowTapeLength: number; } interface AppliedOperation { @@ -1119,7 +1120,21 @@ class StressDriver { // the duplicate oracle must allow them cumulatively, not just against the // current frame. #everDuplicatedFrameLines = new Set(); - #nativeScrollbackAuditBlocked = false; + // Shadow commit ledger mirroring the engine's append-only law, fed only by + // observed frames (render wrap) and observed bytes (write wrap) — never by + // engine internals. `#shadowTape` is what native scrollback must contain; + // `#shadowWindowTop` is the frame row mapped to grid row 0. Double-entry + // bookkeeping for committedRows/windowTop: the engine and this ledger must + // independently arrive at the same terminal state. + #shadowTape: string[] = []; + #shadowCommitted = 0; + #shadowWindowTop = 0; + #shadowFrame: string[] = []; + #shadowFrameHeight = 0; + #shadowFrameWidth = 0; + #shadowFrameOverlay = false; + #shadowFrameGeometryChanged = false; + #shadowAltActive = false; // Every byte the renderer wrote to the terminal, in order. The sync-output // discipline oracle audits bracket balance incrementally from #writeLogScanned // and carries partial private CSI sequences across write chunks. @@ -1156,9 +1171,27 @@ class StressDriver { (this.#term as { write: (data: string) => void }).write = (data: string) => { this.#writeLog.push(data); realWrite(data); + this.#applyShadowWrite(data); }; this.#tui = new TUI(this.#term, true, { renderScheduler: this.#scheduler }); this.#tui.addChild(this.#component); + const realRender = this.#tui.render.bind(this.#tui); + (this.#tui as { render: (width: number) => string[] }).render = (width: number) => { + const lines = realRender(width); + this.#shadowFrameGeometryChanged = + this.#shadowFrameWidth > 0 && + (width !== this.#shadowFrameWidth || this.#term.rows !== this.#shadowFrameHeight); + // Markers are engine-internal sentinels and never reach the terminal; + // strip them before normalization (stripVTControlCharacters otherwise + // swallows everything after an APC introducer). + this.#shadowFrame = lines.map(line => + expectedTerminalLine(line.includes(CURSOR_MARKER) ? line.replaceAll(CURSOR_MARKER, "") : line, width), + ); + this.#shadowFrameWidth = width; + this.#shadowFrameHeight = this.#term.rows; + this.#shadowFrameOverlay = this.#tui.hasOverlay(); + return lines; + }; } async run(): Promise { @@ -1224,6 +1257,7 @@ class StressDriver { height: this.#term.rows, frame: expected.frame, atBottom: position.viewportY >= position.baseY, + shadowTapeLength: this.#shadowTape.length, }; } @@ -2046,8 +2080,6 @@ class StressDriver { this.#assertNoFrameNeutralScrollbackGrowth(op, before, after, index); this.#assertCursor(op, before, after, index); this.#assertScrolledDeferral(op, before, after, index); - this.#assertRowAccounting(op, before, after, index); - this.#assertScrollbackGrowthMatchesFrameGrowth(op, before, after, index); this.#assertMultiplexerPaneHistoryGrowth(op, before, after, index); this.#assertHistoryPrefixStability(op, before, after, index); this.#assertNativeScrollbackReplay(op, before, after, index); @@ -2217,45 +2249,26 @@ class StressDriver { #assertViewportFidelity(op: AppliedOperation, before: Snapshot, after: Snapshot, index: number): void { if (this.#hasVisibleOverlay()) return; if (!after.atBottom) return; - // Multiplexer mode: the buffer snapshot is just the view, so the - // buffer-length alignment precondition below can never hold once the frame - // overflows. Check fidelity on geometry-changed frames instead — tmux - // reflows the pane grid on resize, and the renderer must repaint the whole - // visible window at the new geometry (any output anchored to pre-reflow - // rows splices phantom rows into the pane). Geometry repaints write every - // row, so ghost trailing blanks cannot occur and the comparison is exact. - if (this.#traits.preservesPaneHistory) { - if (!op.geometryChanged) return; - const expectedAfterResize = expectedViewport(after.frame, after.height); - if (!sameLines(after.view, expectedAfterResize)) { - this.#fail("viewport fidelity", op, before, after, index, { expected: expectedAfterResize }); - } - return; + // The grid must show the shadow window slice: the frame tail anchored at + // the ledger's window top (which floors at the committed boundary after + // a shrink, leaving blank rows below the content instead of re-showing + // committed rows). Multiplexer mode only checks geometry frames — tmux + // reflows the pane grid on resize and the renderer must repaint the + // whole visible window at the new geometry. + if (this.#traits.preservesPaneHistory && !op.geometryChanged) return; + const expected: string[] = []; + for (let r = 0; r < after.height; r++) { + expected.push(after.frame[this.#shadowWindowTop + r] ?? ""); } - // Foreground-tool streaming never legitimately defers: the eager opt-in keeps - // the live tail current every frame (a shrink repaints in place rather than - // padding and pinning the pre-shrink viewport), so the visible window must be - // exactly bottom-anchored even when stale rows still sit in native scrollback - // (those reconcile at the next checkpoint). Asserting the visible rows - // directly — without the ghost-row buffer-length bail below — is what catches - // the "injected chip rendered over the tool render" drift head-on instead of - // skipping the frame because the drift left a length mismatch. - if (this.#traits.foregroundStreaming) { - const expected = expectedViewport(after.frame, after.height); - if (!sameLines(after.view, expected)) { - this.#fail("foreground-stream viewport fidelity", op, before, after, index, { expected }); - } - return; - } - // Strict bottom-anchoring only holds when the buffer carries no ghost/stale - // extra rows. A trailing shrink clears the bottom row in place (it cannot pull - // a scrollback line down without a disruptive full repaint), leaving the - // content top-aligned with a ghost blank below — buffer.length then exceeds - // the clean expectation until the next forced repaint/checkpoint re-anchors it. - if (after.buffer.length !== this.#expectedScrollbackBuffer(after).length) return; - const expected = expectedViewport(after.frame, after.height); - if (!sameLines(after.view, expected)) { - this.#fail("viewport fidelity", op, before, after, index, { expected }); + if (!sameLinesAllowingMarkDrift(after.view, expected)) { + this.#fail( + this.#traits.foregroundStreaming ? "foreground-stream viewport fidelity" : "viewport fidelity", + op, + before, + after, + index, + { expected, shadowWindowTop: this.#shadowWindowTop }, + ); } } @@ -2265,10 +2278,14 @@ class StressDriver { if (!this.#bufferReflectsFrame(before.buffer, before.frame, before.height)) return; const expected = this.#expectedScrollbackBuffer(after); if (after.buffer.length !== expected.length) return; - if (!sameLines(after.buffer, expected)) { + if (!sameLinesAllowingMarkDrift(after.buffer, expected)) { + const mismatch = firstMismatchIndex(after.buffer, expected); this.#fail("aligned buffer fidelity", op, before, after, index, { expectedLength: expected.length, actualLength: after.buffer.length, + firstMismatch: mismatch, + expectedWindow: windowAround(expected, mismatch), + actualWindow: windowAround(after.buffer, mismatch), }); } } @@ -2297,7 +2314,7 @@ class StressDriver { // Exact cursor parking is only predictable when the buffer is bottom-anchored // (no ghost/stale rows). After a trailing shrink the cursor sits on the // de-anchored last content row, which is checked once a repaint re-anchors. - if (after.buffer.length !== this.#expectedScrollbackBuffer(after).length) return; + if (!this.#isCleanBuffer(after.buffer, after.frame, after.height)) return; if (after.cursor.row !== expectedCursor.row) { this.#fail("focused cursor row", op, before, after, index, { expectedRow: expectedCursor.row, @@ -2352,77 +2369,19 @@ class StressDriver { } } - #assertRowAccounting(op: AppliedOperation, before: Snapshot, after: Snapshot, index: number): void { - if (!this.#traits.strictNativeScrollback || this.#hasVisibleOverlay()) return; - if (!op.mutatesContent || !op.checksRowAccounting || op.geometryChanged || op.forcedRender) return; - if (!before.atBottom || !after.atBottom) return; - if (this.#scrollbackCapReached(before) || this.#scrollbackCapReached(after)) return; - if (before.redraws !== after.redraws) return; - // Row accounting is only meaningful once content overflows the viewport. While - // content fits within `height`, xterm pins buffer.length at `height`, so a - // content row added inside the viewport grows the buffer by 0 — `ΔB == ΔF` - // does not apply until rows are actually being pushed into scrollback. - if (before.frame.length < before.height) return; - const deltaFrame = after.frame.length - before.frame.length; - if (deltaFrame < 0) return; - const deltaBuffer = after.buffer.length - before.buffer.length; - const incremental = deltaBuffer === deltaFrame; - const clean = this.#isCleanBuffer(after.buffer, after.frame, after.height); - if (!incremental && !clean) { - this.#fail("buffer row accounting", op, before, after, index, { - deltaFrame, - deltaBuffer, - clean, - expected: "deltaBuffer === deltaFrame OR clean full reconstruction", - }); - } - } - - #assertScrollbackGrowthMatchesFrameGrowth( - op: AppliedOperation, - before: Snapshot, - after: Snapshot, - index: number, - ): void { - if (!this.#traits.strictNativeScrollback || this.#hasVisibleOverlay()) return; - if (op.checkpoint || op.geometryChanged) return; - if (!before.atBottom || !after.atBottom) return; - const deltaBuffer = after.buffer.length - before.buffer.length; - if (this.#scrollbackCapReached(before) || this.#scrollbackCapReached(after)) return; - if (deltaBuffer <= 0) return; - const clean = this.#isCleanBuffer(after.buffer, after.frame, after.height); - if (clean) return; - const deltaFrame = Math.max(0, after.frame.length - before.frame.length); - if (deltaBuffer > deltaFrame) { - this.#fail("scrollback grew faster than frame", op, before, after, index, { - deltaFrame, - deltaBuffer, - expected: "dirty live scrollback growth must not exceed logical frame growth", - }); - } - const expectedTail = after.frame.slice(after.frame.length - deltaBuffer); - const actualTail = after.buffer.slice(after.buffer.length - deltaBuffer); - if (!sameLines(actualTail, expectedTail)) { - this.#fail("scrollback growth tail mismatch", op, before, after, index, { - deltaBuffer, - expectedTail, - actualTail, - }); - } - } - // Multiplexer panes never receive a destructive scrollback clear (the // renderer forces clearScrollback off inside tmux/screen/zellij because pane // history is intentionally preserved), so any full-frame replay during live // rendering appends a complete duplicate copy of the transcript to pane // history. Users see every transcript row twice (or more) when scrolling - // back, and the per-frame write cost becomes O(frame). Bound live-frame pane - // history growth by the rows the frame actually appended; only explicit - // checkpoints may replay the transcript wholesale. Geometry-changed frames - // are exempt except for pure height resizes, where xterm/tmux reflow is + // back, and the per-frame write cost becomes O(frame). Pane history may grow + // exactly by the rows the shadow ledger committed during the op (appends, + // plus backfill of a chunk frozen during an overlay or geometry frame); + // anything beyond that is a replay leaking into preserved history. Geometry + // frames are exempt except pure height resizes, where xterm/tmux reflow is // bounded: a height shrink moves at most (oldHeight - newHeight) rows into - // pane history and a height grow moves rows back out — width changes rewrap - // pane history with unbounded row deltas and cannot be bounded from here. + // pane history — width changes rewrap pane history with unbounded row + // deltas and cannot be bounded from here. #assertMultiplexerPaneHistoryGrowth(op: AppliedOperation, before: Snapshot, after: Snapshot, index: number): void { if (!this.#traits.preservesPaneHistory) return; if (op.checkpoint) return; @@ -2431,19 +2390,13 @@ class StressDriver { const reflowAllowance = heightOnlyResize ? Math.max(0, before.height - after.height) : 0; const deltaBaseY = after.position.baseY - before.position.baseY; if (deltaBaseY <= 0) return; - // Rows appended at any point during the op (including transient preview - // expansions that later collapsed) legitimately scroll into pane history - // — terminal scrolling is how appends work, and pane history can never be - // retracted. The invariant targets full-frame replays, which grow history - // by ~frame.length instead of by the number of appended rows. - const allowedGrowth = - Math.max(Math.max(0, after.frame.length - before.frame.length), op.transientFrameGrowth ?? 0) + - reflowAllowance; + const committedDelta = Math.max(0, after.shadowTapeLength - before.shadowTapeLength); + const allowedGrowth = committedDelta + reflowAllowance; if (deltaBaseY > allowedGrowth) { - this.#fail("multiplexer pane history grew faster than frame", op, before, after, index, { + this.#fail("multiplexer pane history grew faster than committed rows", op, before, after, index, { deltaBaseY, allowedGrowth, - transientFrameGrowth: op.transientFrameGrowth ?? null, + committedDelta, expected: "live frames must not replay the transcript into preserved pane history", }); } @@ -2467,16 +2420,11 @@ class StressDriver { #assertNativeScrollbackReplay(op: AppliedOperation, before: Snapshot, after: Snapshot, index: number): void { if (!this.#traits.strictNativeScrollback) return; - if (op.geometryChanged) { - this.#nativeScrollbackAuditBlocked = true; - return; - } if (this.#hasVisibleOverlay()) return; - if (this.#nativeScrollbackAuditBlocked && !op.checkpoint) return; if (!after.atBottom) return; - if (!op.mutatesContent && !op.forcedRender && !op.checkpoint) return; + if (!op.mutatesContent && !op.forcedRender && !op.checkpoint && !op.geometryChanged) return; const expected = this.#expectedScrollbackBuffer(after); - if (!sameLines(after.buffer, expected)) { + if (!sameLinesAllowingMarkDrift(after.buffer, expected)) { const mismatch = firstMismatchIndex(after.buffer, expected); this.#fail("native scrollback buffer fidelity", op, before, after, index, { expectedLength: expected.length, @@ -2486,7 +2434,6 @@ class StressDriver { actualWindow: windowAround(after.buffer, mismatch), }); } - this.#nativeScrollbackAuditBlocked = false; const probes = scrollbackProbePositions(after.position.baseY, expected.length, after.height); try { @@ -2495,7 +2442,7 @@ class StressDriver { this.#term.scrollLines(viewportY - current); const actual = normalizeLines(this.#term.getViewport()); const expectedView = fixedViewportSlice(expected, viewportY, after.height); - if (!sameLines(actual, expectedView)) { + if (!sameLinesAllowingMarkDrift(actual, expectedView)) { this.#fail("native scrollback viewport fidelity", op, before, after, index, { viewportY, expected: expectedView, @@ -2511,7 +2458,7 @@ class StressDriver { #assertCleanBuffer(op: AppliedOperation, before: Snapshot, after: Snapshot, index: number): void { if (this.#hasVisibleOverlay()) return; const expected = this.#expectedScrollbackBuffer(after); - if (!sameLines(after.buffer, expected)) { + if (!sameLinesAllowingMarkDrift(after.buffer, expected)) { this.#fail("clean checkpoint reconstruction", op, before, after, index, { expectedLength: expected.length, actualLength: after.buffer.length, @@ -2520,7 +2467,60 @@ class StressDriver { } #expectedScrollbackBuffer(snapshot: Snapshot): string[] { - return expectedScrollbackBuffer(snapshot.frame, snapshot.height, this.#scenario.scrollback); + const height = snapshot.height; + const expected = [...this.#shadowTape]; + for (let r = 0; r < height; r++) { + expected.push(this.#shadowFrame[this.#shadowWindowTop + r] ?? ""); + } + const cap = height + this.#scenario.scrollback; + return expected.length > cap ? expected.slice(expected.length - cap) : expected; + } + + /** + * Advance the shadow commit ledger for one observed write. Classification + * is byte-driven: ED3 = destructive replay, ED2-without-ED3 = + * non-destructive replay (initial paint / multiplexer replace), anything + * else = ordinary update following the engine's append-only law. + */ + #applyShadowWrite(data: string): void { + if (data.includes("\x1b[?1049h")) this.#shadowAltActive = true; + if (data.includes("\x1b[?1049l")) { + this.#shadowAltActive = false; + return; + } + if (this.#shadowAltActive) return; + const frame = this.#shadowFrame; + const height = Math.max(1, this.#shadowFrameHeight); + const length = frame.length; + if (data.includes("\x1b[3J")) { + this.#shadowCommitted = Math.max(0, length - height); + this.#shadowWindowTop = this.#shadowCommitted; + this.#shadowTape = frame.slice(0, this.#shadowCommitted); + return; + } + if (data.includes("\x1b[2J")) { + // Grid cleared in place, committed prefix scrolls above it; prior + // history rows stay (and are erased only by the ED3 branch above). + const chunkTo = Math.max(0, length - height); + for (let i = 0; i < chunkTo; i++) this.#shadowTape.push(frame[i] ?? ""); + this.#shadowCommitted = chunkTo; + this.#shadowWindowTop = chunkTo; + return; + } + if (length <= this.#shadowCommitted) { + // Shrink into the committed prefix: the engine re-anchors and + // restarts commit bookkeeping; stale history stays on the tape. + this.#shadowCommitted = Math.max(0, length - height); + this.#shadowWindowTop = this.#shadowCommitted; + return; + } + const windowTop = Math.max(this.#shadowCommitted, length - height, 0); + this.#shadowWindowTop = windowTop; + // Overlays and multiplexer geometry frames freeze commits. + if (this.#shadowFrameOverlay || this.#shadowFrameGeometryChanged) return; + const chunkTo = Math.max(this.#shadowCommitted, Math.min(length, windowTop)); + for (let i = this.#shadowCommitted; i < chunkTo; i++) this.#shadowTape.push(frame[i] ?? ""); + this.#shadowCommitted = chunkTo; } #scrollbackCapReached(snapshot: Snapshot): boolean { return Math.max(snapshot.height, snapshot.frame.length) > snapshot.height + this.#scenario.scrollback; @@ -2565,10 +2565,27 @@ class StressDriver { if (!this.#scenario.uniqueContent) return; // Accumulate even when the check below is skipped (scrolled/overlay): the // frame's legitimate duplicates commit to scrollback regardless of where - // the viewport is parked. + // the viewport is parked. The shadow tape contributes too: a no-seam + // offscreen insert re-indexes committed content, so the shifted rows + // legitimately commit a second time (the exact tape-equality oracle has + // already proven the buffer matches the ledger row for row). for (const line of duplicateNonblankLines(after.frame)) { this.#everDuplicatedFrameLines.add(line); } + const tapeSeen = new Set(); + for (const line of this.#shadowTape) { + if (line.length === 0) continue; + if (tapeSeen.has(line)) this.#everDuplicatedFrameLines.add(line); + tapeSeen.add(line); + } + // A committed row that still sits in the visible window (window floored + // at the commit boundary) legitimately appears in both regions of the + // whole-tape buffer snapshot. + for (let r = 0; r < after.height; r++) { + const line = this.#shadowFrame[this.#shadowWindowTop + r] ?? ""; + if (line.length === 0) continue; + if (tapeSeen.has(line)) this.#everDuplicatedFrameLines.add(line); + } if (this.#hasVisibleOverlay() || !after.atBottom) return; const allowed = this.#everDuplicatedFrameLines; const seen = new Set(); @@ -2674,6 +2691,24 @@ function sameLines(left: readonly string[], right: readonly string[]): boolean { return true; } +// ghostty-web's cell-grid text extraction can migrate or merge Unicode +// non-spacing marks across neighboring cells for combining-heavy scripts +// (Arabic harakat), so a byte-exact round trip through the virtual terminal is +// not achievable for those rows (the engine paints them verbatim; see the +// WIDTH notes in docs/tui-core-renderer.md). Fall back to comparing with +// non-spacing marks stripped — row count, order, and all spacing content stay +// exact. +const NONSPACING_MARKS = /\p{Mn}/gu; +function sameLinesAllowingMarkDrift(left: readonly string[], right: readonly string[]): boolean { + if (sameLines(left, right)) return true; + if (left.length !== right.length) return false; + for (let i = 0; i < left.length; i++) { + if (left[i] === right[i]) continue; + if (left[i]!.replace(NONSPACING_MARKS, "") !== right[i]!.replace(NONSPACING_MARKS, "")) return false; + } + return true; +} + function firstMismatchIndex(left: readonly string[], right: readonly string[]): number { const maxLength = Math.max(left.length, right.length); for (let i = 0; i < maxLength; i++) { diff --git a/packages/tui/test/streaming-scrollback-defer.test.ts b/packages/tui/test/streaming-scrollback-defer.test.ts index 548de2bbd..704421d92 100644 --- a/packages/tui/test/streaming-scrollback-defer.test.ts +++ b/packages/tui/test/streaming-scrollback-defer.test.ts @@ -209,7 +209,7 @@ describe("streaming scrollback defer", () => { } }); - it("defers scrollback growth during streaming", async () => { + it("commits scrolled streaming rows to history exactly once without ED3", async () => { if (process.platform === "win32") return; const term = new VirtualTerminal(40, 10); overrideProbe(term, undefined); @@ -222,16 +222,18 @@ describe("streaming scrollback defer", () => { await settle(term); const writes = capture(term); - const scrollbackBefore = term.getScrollBuffer().length; - // Grow content past the viewport — capped, no rows enter native - // scrollback during streaming, and no ED3 erase fires. - component.setLines([...rows("stream-", 10), ...rows("more-", 30), "prompt"]); + // Grow content past the viewport — without a live-region seam the + // scrolled-off rows commit to native history as they pass the seam + // (shell semantics): exactly once, in frame order, with no ED3. + const frame1 = [...rows("init-", 10), ...rows("stream-", 30), "prompt"]; + component.setLines(frame1); tui.requestRender(); await settle(term); expect(eraseScrollbackCount(writes)).toBe(0); - expect(term.getScrollBuffer().length).toBe(scrollbackBefore); + let buffer = term.getScrollBuffer().map(line => line.trimEnd()); + expect(buffer).toEqual(frame1.slice(0, buffer.length)); expect( term .getViewport() @@ -239,13 +241,17 @@ describe("streaming scrollback defer", () => { .at(-1), ).toBe("prompt"); - // Grow even more — still capped, still no ED3. - component.setLines([...rows("stream-", 10), ...rows("more-", 50), "prompt"]); + // Grow further — history extends append-only: still no ED3, no + // duplicates, and previously committed rows are untouched. + const frame2 = [...rows("init-", 10), ...rows("stream-", 50), "prompt"]; + component.setLines(frame2); tui.requestRender(); await settle(term); expect(eraseScrollbackCount(writes)).toBe(0); - expect(term.getScrollBuffer().length).toBe(scrollbackBefore); + buffer = term.getScrollBuffer().map(line => line.trimEnd()); + expect(buffer).toEqual(frame2.slice(0, buffer.length)); + expect(buffer.length).toBeGreaterThan(frame1.length - 10); } finally { tui.stop(); } @@ -377,15 +383,15 @@ describe("streaming scrollback defer", () => { await settle(term); const writes = capture(term); - const scrollbackBefore = term.getScrollBuffer().length; - // Stream past the viewport: the cap keeps transient rows out of native - // history and no ED3 fires. + // Stream past the viewport: scrolled rows commit to history in + // order (shell semantics) and no ED3 fires. component.setLines([...rows("stream-", 30), "prompt"]); tui.requestRender(); await settle(term); expect(eraseScrollbackCount(writes)).toBe(0); - expect(term.getScrollBuffer().length).toBe(scrollbackBefore); + const streamed = term.getScrollBuffer().map(line => line.trimEnd()); + expect(streamed).toEqual([...rows("stream-", 30), "prompt"].slice(0, streamed.length)); // Resize mid-stream. The terminal re-wrapped its saved lines at the old // width, so the rebuild must erase them (ED 3) rather than capping to a