From 323a3bc29cf60906049c3b95c1c335b52d216dbc Mon Sep 17 00:00:00 2001 From: enieuwy <121954036+enieuwy@users.noreply.github.com> Date: Sun, 29 Mar 2026 23:50:36 +0800 Subject: [PATCH] feat(core): add first-class retry fallback chains for model/provider failover (#541) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(rules): implemented alwaysApply auto-injection into system prompt rules with alwaysApply: true were parsed by all providers and used to exclude the rule from rulebookRules, but the inclusion half was never built — rule content was silently dropped. now: - full content is injected directly into the system prompt (before the rulebook rules section) in both default and custom prompt templates - rules remain addressable via rule:// for re-reading - ttsr rules still take priority (condition + alwaysApply goes to ttsr only) updated rulebook-matching-pipeline.md to reflect the three-bucket split (ttsr > always-apply > rulebook) and corrected the rule:// resolution docs. * feat(browser): implement screenshot path saving - Add `browser.screenshotDir` setting (tools tab) for a persistent default screenshot directory, configurable via /settings - Honour the existing `path` parameter in the screenshot action, which was declared in the schema but never consumed by the implementation - Resolution order: params.path (abs) > join(screenshotDir, params.path) > join(screenshotDir, screenshot-.png) > /tmp only - Writes full-resolution buffer to disk (not the API-compressed copy) - Creates destination directory recursively if it doesn't exist - details.screenshotPath reflects the actual saved location Fixes: path param silently ignored since removal in v11.5.0 * feat(browser): expand ~ in screenshotDir and path params Users can now configure browser.screenshotDir as ~/Downloads or ~/Pictures and use params.path as ~/screenshots/foo.png without needing to supply fully-qualified paths. * fix(browser): write screenshot to exactly one location Previously wrote to /tmp unconditionally then copied to user path, resulting in two files. Now resolves a single destination upfront: 1. params.path (absolute, or relative to screenshotDir/cwd) 2. screenshotDir + auto-timestamp filename 3. /tmp fallback (original behaviour, unchanged) Also: user-defined destinations receive the full-res buffer; /tmp fallback retains the API-compressed copy as before. * feat(settings): add text input support for plain string settings Settings with type: "string" and no submenu now render as an editable text field in the /settings TUI panel instead of being silently skipped. - settings-defs.ts: add TextInputSettingDef interface; pathToSettingDef falls through to { type: "text" } for any plain string schema entry - settings-selector.ts: add TextInputSubmenu class (mirrors ConfigInputSubmenu from plugin-settings.ts); add "text" case to #defToItem; add #createTextInput method This makes browser.screenshotDir visible and editable in the Tools tab. Empty field on submit clears the setting (falls back to /tmp behavior). * fix(settings): text input — cursor at end, block tab navigation - Cursor: call handleInput(ctrl+e) after setValue to jump to end of pre-filled string instead of leaving it at position 0 - Tab/arrow guard: add #textInputActive flag; suppress tab-switch and left/right routing to tab bar while a TextInputSubmenu is open, so arrow keys reach the Input component's cursor movement handlers instead of switching settings tabs * fix(browser): expand ~\ (Windows backslash) in screenshot paths expandHome now handles both Unix ~/... and Windows ~\... separators, matching user expectation on all platforms. Addresses Codex review. * fix(coding-agent): use expandPath for screenshots * docs: add changelog for screenshot path option Bug fixes: - Add browser.screenshotDir to settings-schema.ts (lost during rebase conflict) - Fix stale expandHome comment to expandPath in settings-selector.ts - Final screenshotDir description: Directory to save screenshots with ~ support * fix: biome format corrections for screenshot-path-option branch * fix: add missing #textInputActive field and screenshotDir default * fix(browser): align screenshot metadata with saved file contents When screenshotDir or params.path is set, the full-resolution PNG buffer is written to disk. Previously mimeType/bytes in details still reflected the resized payload sent to the model, making metadata inconsistent with the actual saved file. Now savedBuffer/savedMimeType track what is written, and details reflects that. Display output distinguishes 'Saved' vs 'Model' when full-res is used, and collapses to a single Format/Dimensions line for temp-only. * fix(browser): resolve params.path relative to cwd regardless of screenshotDir screenshotDir is a default save location, not an anchor for explicit paths. A relative params.path should always resolve against cwd so its semantics are stable and predictable regardless of user settings. * fix(tui): enforce strict line budget for collapsed tool output The grep, ast_grep, and ast_edit renderers used group-count-based collapse that always included the first group unconditionally, allowing collapsed output to remain visually large when a single group contained many lines. Add maxCollapsedLines to renderTreeList that enforces a strict total-line cap in collapsed mode. Items that exceed the remaining budget are skipped entirely (no broken fragments). The isLast tree branch is computed after the budget check to avoid double-last branches when a summary line follows. Remove the per-tool getCollapsedMatchLimit / getCollapsedChangeLimit helpers that are now redundant. Fixes #455 Made-with: Cursor * WIP * WIP * cleanup * docs(coding-agent): update changelog for custom model tags and cycle order * Fix custom model precedence across load and refresh * feat(ask): add multiline editor support for custom input * feat(extension-ui): add dialog options and abort signal support to editor * docs(ask): add multiline editor and timeout behavior guidance * docs(ask): simplify multiline input documentation * fix(tui): preserve terminal scrollback during full redraws Replace destructive \x1b[3J\x1b[2J\x1b[H full-redraw sequence with scrollback-preserving repaint helpers: - seedTranscript: first paint with no prior frame, writes full transcript without clearing scrollback - repaintViewport: trusted-frame repaints that scroll viewport-shift delta into scrollback before overwriting visible rows in-place - Height-increase handler that pushes revealed scrollback back before reclaiming the display Replace requestRender(true) state destruction with a one-shot without resetting #previousLines or cursor bookkeeping. Track #previousHeight to detect terminal height increases that pull scrollback lines into the visible area. fixes #507 * fix(tui): fix exit gaps, content shrink drift, and overlay cursor recovery - Rewrite stop() to use viewport-relative cursor positioning instead of content-length, preventing blank gaps when content is shorter than viewport - Replace trailing \r\n\x1b[2K clear loops in repaintViewport() and height-increase path with \r\n\x1b[J to avoid cursor drift past content - Add 12 TUI regression tests for exit gaps, content shrink, and overlay dismiss cursor recovery - Add 6 coding-agent controller tests for /new and /tree commands * fix(tui): redraw sparse height increases atomically * fix(coding-agent,tui): address review regressions * fix(tui): reseed after terminal resume * test(ai): avoid leaking kagi module mocks * fix(ask): keep multiline custom input in prompt gutter * fix(ask): preserve multiselect choices on editor dismiss * fix(ask): honor app interrupt in prompt editor * fix(coding-agent): preserve new-session approval state * feat(core): add session-scoped model/provider retry fallback policy * feat(core): validate retry fallback chains on session startup * fix(ask): preserve prior answer when custom editor dismissed in single-select * fix(core): harden retry fallback policy semantics * fix(core): correct retry fallback edge cases * test(coding-agent): removed macOS fallback test case from theme detection - Removed test case for macOS fallback behavior inside Zellij. * refactor: simplified null checks using optional chaining across TypeScript and Rust modules - Simplified null/empty checks across TypeScript codebase using optional chaining operator (?.) for improved readability. - Replaced explicit null checks in validation logic with optional chaining in oauth-discovery, gemini-cli, claude, zai, and lsp modules. - Updated error handling in Rust command invocation to use double question mark operator (??) for cmd_result. - Consolidated null validation patterns across tools (bash-skill-urls, browser, gemini-image, resolve) and keybindings using optional chaining. * chore: bump version to 13.15.1 * fix(core): address PR review on fallback retry behavior * test(ai): refactored auth storage test to use spyOn for cleaner mocks - Refactored auth storage test to use vi.spyOn() instead of vi.mock() for cleaner mock management. - Simplified mock type definitions by leveraging bun:test's Mock type import. * chore: bump version to 13.15.2 * Fix stale OpenAI Responses replay across session boundaries (#534) * Fix stale OpenAI Responses replay across session boundaries Fixes #505 * Fix CI tests for session replay change * Harden session reload and switch rollback * Guard session switch snapshots * fix(coding-agent): preserve responses replay snapshots * Normalize pasted image formats before attach (#543) Co-authored-by: iter * Allow overriding the Codex web search model (#516) * Allow codex web search model override * Handle blank codex web search model * feat(ai): add gemini-3.1-pro-preview models to google-vertex provider (#521) Add gemini-3.1-pro-preview and gemini-3.1-pro-preview-customtools to the google-vertex provider in models.json, matching the existing google-generative-ai entries. Fixes #520 Co-authored-by: Muness Castle * Make temporary model selector keybinding configurable (#539) Fixes #533 * chore: bump models.json * refactor(coding-agent): migrated test mocks to vitest spyOn with Symbol.dispose cleanup - Migrated test mocking from bun:test mock API to vitest spyOn pattern across 9 test files. - Extracted mock setup logic into reusable helper functions with Symbol.dispose cleanup pattern. - Replaced manual beforeEach/afterEach and try-finally blocks with TypeScript 5.2 using declarations. - Removed 178 lines of boilerplate mock initialization and restoration code from test suite. * chore: bump version to 13.15.3 * fix(ask): restore prompt-style enter handling * fix(tui): avoid blank scrollback regressions * fix(models): keep same-id replacements authoritative * fix(tui): enforce collapsed line budgets * fix(models): unify selector role sources * fix(rules): dedupe always-apply prompt injection * fix(browser): show saved screenshot path * style: format merged PR fixes * refactor(coding-agent): restructured screenshot and prompt utilities into focused helpers - Extracted screenshot formatting logic into dedicated `formatScreenshot()` function with options support. - Consolidated prompt source deduplication into `dedupePromptSource()` helper to prevent rule duplication. - Refactored editor text sanitization to use `replaceTabs()` utility for consistent tab width handling. - Added test coverage verifying editor respects configured tab width when loading text programmatically. * refactor(coding-agent): restructured validation and rendering for consistency - Refactored theme color validation to use single source of truth with THEME_COLOR_RECORD object. - Simplified model registry to defer per-model overrides to dedicated method and use constant for role IDs. - Refactored tree list rendering to pre-render items once for consistent line counts across phases. - Refactored question result formatting to use early returns and consistently include question ID in output. - Updated hook editor hint text to include ctrl+g external editor option when prompt style is enabled. - Removed unused isLogicalLineStart property from LayoutLine interface in editor component. * revert: 535 due to TUI regressions * test(coding-agent): corrected hook-editor assertion for keybinding render - Corrected assertion in hook-editor test to verify external editor keybinding is rendered. * feat(tools): added root path alias to resolve bare / to working directory - Added root path alias feature to resolve bare `/` to session working directory in path resolution. - Updated browser tool to use `resolveToCwd()` for consistent workspace-relative path handling. - Added comprehensive test suite validating root path alias resolution across grep, read, find, ast_grep, and ast_edit tools. * feat(prompts/tools): clarified hashline block boundary handling with examples - Improved hashline tool documentation with clearer guidance on block boundary handling and closing delimiter duplication prevention. - Added concrete example demonstrating correct anchor placement when replacing entire blocks including closing braces. - Reorganized boundary duplication warnings into actionable self-check guidance with visual comparison steps. * chore: bump version to 13.16.0 * fix(coding-agent): fixed python kernel startup hangs (#548) * fix(coding-agent): fixed python kernel startup hangs * fix(coding-agent): fixed startup timeout regressions * fix(coding-agent): preserved startup cancellation typing * perf(pi-natives): optimized memory allocation with MiMalloc integration - Integrated MiMalloc as global allocator to improve memory allocation performance. * feat: fff - Added SearchDb class for stateful shared search database instances enabling persistent file indexing and frecency tracking across grep, glob, and fuzzyFind operations. - Added optional db parameter to grep(), glob(), and fuzzyFind() functions for database-backed searching with improved performance via cached file indices. - Replaced grep-searcher with fff-grep and added fff-search dependency for enhanced file discovery and search capabilities with memory-mapped file support. - Migrated fuzzy file discovery from fd module to fff module with SearchDb integration for stateful caching and improved search performance. - Exported SearchDb type from @oh-my-pi/pi-natives public API for type-safe usage in grep, glob, and fuzzyFind workflows. * feat(pi-natives): added unified picker coordination for file search operations - Added `wait_for_picker_scan()` function to search_db module with cancellation token support for polling picker scan completion. - Integrated picker-based file search into glob matching logic with `collect_files_from_picker()` helper to reuse shared SearchDb picker results. - Refactored fff and grep modules to use centralized `wait_for_picker_scan()` wrapper instead of direct FilePicker calls, improving cancellation handling. - Propagated SearchDb instance through agent session initialization and input controller to enable unified file picker coordination across search operations. * chore: bump version to 13.16.1 * fix: install zig in CI * feat(tui): make inline image max-width configurable via tui.maxInlineImageColumns (#551) * feat(tui): make inline image max-width configurable via tui.maxInlineImageColumns * fix(tui): handle 0 as unlimited in maxInlineImageColumns; drop || undefined coercion * feat(browser): auto-detect NixOS and use system Chromium (#550) Puppeteer's bundled Chromium is a dynamically-linked FHS binary that cannot run on NixOS. On startup, resolveSystemChromium() checks for /etc/NIXOS and searches for a usable binary in order: 1. chromium on PATH 2. chromium-browser on PATH 3. ~/.nix-profile/bin/chromium 4. /run/current-system/sw/bin/chromium The resolved path is passed as executablePath to puppeteer.launch(). Result is cached per process. On non-NixOS systems the function returns undefined immediately, leaving Puppeteer's default resolution intact. * feat(pi-natives): added automatic parenthesis escaping in regex patterns - Added automatic escaping of unescaped parentheses in regex patterns when group syntax errors occur, enabling literal function call patterns like `fetchAnthropicProvider(` to work as search queries. - Extracted regex matcher builder into separate function for reusability and error recovery logic. - Added 2 test cases validating parenthesis escaping behavior for both escaped and literal parentheses. - Fixed documentation formatting in sanitize_braces comment. * style: reformat * chore: bump version to 13.16.2 * fix: only show update banner when npm version is strictly newer (#552) * fix(ai): corrected OAuth credential updates to replace in-place instead of accumulating soft-deleted rows - Fixed OAuth credential updates to replace matching credentials in-place rather than creating disabled rows, preventing unbounded accumulation of soft-deleted credentials. - Modified OAuth credential saving to preserve unrelated identities instead of replacing all credentials for a provider. - Updated credential identity resolution to use provider context for more accurate email deduplication. - Implemented upsertAuthCredentialForProvider method to handle credential matching and in-place updates. - Added 5 test cases covering credential preservation across reauth, multi-account scenarios, and stale cache handling. * chore: bump version to 13.16.3 * feat: introduced unified range API for hashline edits and model catalog updates - Simplified hashline edit location API by replacing separate `line` and `block` properties with unified `range` property accepting `{ pos, end }` anchors. - Renamed hashline helper functions from `hlineref`/`hlinefull` to `href`/`hline` for improved brevity and consistency. - Added detection for `kysely-codegen` generated files in auto-generated file guard with corresponding test coverage. - Added 13 new AI model configurations and updated token limits and pricing for existing models across multiple providers. - Enhanced file type validation in grep native to reject symlinks, FIFOs, sockets, and non-regular files with improved error handling. * chore: bump version to 13.16.4 * fix: pin rustc-hash to 2.1.1 to avoid SIGILL on CI rustc-hash 2.1.2 (released today) refactored hash_bytes to use split_first_chunk, which produces illegal instructions when compiled with nightly + -C target-cpu=x86-64-v3 on CI runners. * fix(ci): pin nightly to 2026-03-27 to avoid codegen SIGILL regression Today's nightly produces illegal instructions when compiled with -C target-cpu=x86-64-v3. Reverts the unnecessary rustc-hash pin from the previous commit since the real cause is the nightly compiler. Also lets Cargo.lock return to rustc-hash 2.1.2 (not the culprit). * fix(ci): add rustup target fallback for pinned nightly cross-compile * fix(coding-agent): do not prompt to use grep and find tools if they are disabled (#566) Co-authored-by: le-cameleon <200889489+le-cameleon@users.noreply.github.com> * fix(natives): skipped grep special files (#565) avoided opening fifos and other special filesystem nodes during grep and added fifo regressions in native and coding-agent tests. * fix: skill baseDir regex fails on Windows backslash paths (#554) The regex that strips SKILL.md from the path to compute baseDir only matches forward slashes. On Windows where paths use backslashes, the replace is a no-op and baseDir equals the full SKILL.md file path. This breaks sub-path resolution for skills: the subpath gets appended to the SKILL.md file path instead of the skill directory. Fix: use character class matching both path separators. --------- Co-authored-by: deadcode-walker <268043493+deadcode-walker@users.noreply.github.com> Co-authored-by: Rens Tillmann Co-authored-by: haiyang.zhou Co-authored-by: Leo P Co-authored-by: Zakhar Kogan <36503576+zaharkogan@users.noreply.github.com> Co-authored-by: Vu Anh Nguyen Co-authored-by: can1357 Co-authored-by: daandden <64765666+daandden@users.noreply.github.com> Co-authored-by: iter <72358817+itertea@users.noreply.github.com> Co-authored-by: iter Co-authored-by: Cheol Kang Co-authored-by: Muness Castle <931+muness@users.noreply.github.com> Co-authored-by: Muness Castle Co-authored-by: zamo Co-authored-by: elikoga Co-authored-by: BayLee4 <63376748+BayLee4@users.noreply.github.com> Co-authored-by: le-cameleon <200889489+le-cameleon@users.noreply.github.com> Co-authored-by: Wiedzmin <56316383+art-wiedzmin@users.noreply.github.com> --- README.md | 29 +- .../coding-agent/src/config/model-registry.ts | 37 ++ .../src/config/settings-schema.ts | 12 + .../src/modes/components/settings-defs.ts | 9 + .../src/modes/controllers/event-controller.ts | 10 + .../src/modes/interactive-mode.ts | 5 + .../coding-agent/src/session/agent-session.ts | 298 +++++++++++++- .../test/agent-session-retry-fallback.test.ts | 389 ++++++++++++++++++ 8 files changed, 775 insertions(+), 14 deletions(-) create mode 100644 packages/coding-agent/test/agent-session-retry-fallback.test.ts diff --git a/README.md b/README.md index b9711948e..83b23504f 100644 --- a/README.md +++ b/README.md @@ -885,16 +885,33 @@ theme: dark: titanium light: light +enabledModels: + - "anthropic/*" + - "*gpt*" + - "gemini-2.5-pro:high" + modelRoles: default: anthropic/claude-sonnet-4-20250514 plan: anthropic/claude-opus-4-1:high smol: anthropic/claude-sonnet-4-20250514 defaultThinkingLevel: high -enabledModels: - - anthropic/* - - "*gpt*" - - gemini-2.5-pro:high +retry: + enabled: true + # Number of retries before giving up on rate limits/server errors + maxRetries: 3 + # Wait this long as a base (exponentially backed off) unless the API provides a retry-after-ms + baseDelayMs: 2000 + # Configure role-specific model fallback chains + fallbackChains: + default: + - "openai/gpt-4o-mini" + - "openai/gpt-4o" + plan: + - "anthropic/claude-sonnet-4-6:high" + - "openai/o3:high" + # Whether to revert to the primary model when a fallback's cooldown expires + fallbackRevertPolicy: cooldown-expiry steeringMode: one-at-a-time followUpMode: one-at-a-time interruptMode: immediate @@ -914,10 +931,6 @@ compaction: skills: enabled: true -retry: - enabled: true - maxRetries: 3 - baseDelayMs: 2000 terminal: showImages: true diff --git a/packages/coding-agent/src/config/model-registry.ts b/packages/coding-agent/src/config/model-registry.ts index e17679a7f..ff85de248 100644 --- a/packages/coding-agent/src/config/model-registry.ts +++ b/packages/coding-agent/src/config/model-registry.ts @@ -28,6 +28,7 @@ import { import { isRecord, logger } from "@oh-my-pi/pi-utils"; import { type Static, Type } from "@sinclair/typebox"; import { type ConfigError, ConfigFile } from "../config"; +import { parseModelString } from "../config/model-resolver"; import { isValidThemeColor, type ThemeColor } from "../modes/theme/theme"; import type { AuthStorage, OAuthCredential } from "../session/auth-storage"; import type { Settings } from "./settings"; @@ -721,6 +722,14 @@ function buildCustomModel( return finalizeCustomModel(model, options); } +function normalizeSuppressedSelector(selector: string): string { + const trimmed = selector.trim(); + if (!trimmed) return trimmed; + const parsed = parseModelString(trimmed); + if (!parsed) return trimmed; + return `${parsed.provider}/${parsed.id}`; +} + /** * Model registry - loads and manages models, resolves API keys via AuthStorage. */ @@ -737,6 +746,7 @@ export class ModelRegistry { #registeredProviderSources: Set = new Set(); #providerDiscoveryStates: Map = new Map(); #cacheDbPath?: string; + #suppressedSelectors: Map = new Map(); #backgroundRefresh?: Promise; #lastDiscoveryWarnings: Map = new Map(); @@ -766,6 +776,7 @@ export class ModelRegistry { */ async refresh(strategy: ModelRefreshStrategy = "online-if-uncached"): Promise { this.#reloadStaticModels(); + this.#suppressedSelectors.clear(); await this.#refreshRuntimeDiscoveries(strategy); } @@ -789,6 +800,11 @@ export class ModelRegistry { async refreshProvider(providerId: string, strategy: ModelRefreshStrategy = "online"): Promise { this.#reloadStaticModels(); + for (const selector of this.#suppressedSelectors.keys()) { + if (selector.startsWith(`${providerId}/`)) { + this.#suppressedSelectors.delete(selector); + } + } await this.#refreshRuntimeDiscoveries(strategy, new Set([providerId])); } @@ -1825,6 +1841,27 @@ export class ModelRegistry { }); } } + + /** + * Suppress a specific model selector (e.g., "provider/id") until a specific timestamp. + */ + suppressSelector(selector: string, untilMs: number): void { + this.#suppressedSelectors.set(normalizeSuppressedSelector(selector), untilMs); + } + + /** + * Check if a model selector is currently suppressed due to rate limits. + */ + isSelectorSuppressed(selector: string): boolean { + const normalizedSelector = normalizeSuppressedSelector(selector); + const suppressedUntil = this.#suppressedSelectors.get(normalizedSelector); + if (!suppressedUntil) return false; + if (suppressedUntil <= Date.now()) { + this.#suppressedSelectors.delete(normalizedSelector); + return false; + } + return true; + } } /** diff --git a/packages/coding-agent/src/config/settings-schema.ts b/packages/coding-agent/src/config/settings-schema.ts index 17ecfcf44..865b5783e 100644 --- a/packages/coding-agent/src/config/settings-schema.ts +++ b/packages/coding-agent/src/config/settings-schema.ts @@ -525,6 +525,18 @@ export const SETTINGS_SCHEMA = { }, "retry.baseDelayMs": { type: "number", default: 2000 }, + "retry.fallbackChains": { type: "record", default: {} as Record }, + "retry.fallbackRevertPolicy": { + type: "enum", + values: ["cooldown-expiry", "never"] as const, + default: "cooldown-expiry", + ui: { + tab: "model", + label: "Fallback Revert Policy", + description: "When to return to the primary model after a fallback", + submenu: true, + }, + }, // ──────────────────────────────────────────────────────────────────────── // Interaction diff --git a/packages/coding-agent/src/modes/components/settings-defs.ts b/packages/coding-agent/src/modes/components/settings-defs.ts index 4c474448a..bd5a1de37 100644 --- a/packages/coding-agent/src/modes/components/settings-defs.ts +++ b/packages/coding-agent/src/modes/components/settings-defs.ts @@ -117,6 +117,15 @@ const OPTION_PROVIDERS: Partial> = { { value: "5", label: "5 retries" }, { value: "10", label: "10 retries" }, ], + // Retry fallback revert policy + "retry.fallbackRevertPolicy": [ + { + value: "cooldown-expiry", + label: "Cooldown expiry", + description: "Return to the primary model after its suppression window ends", + }, + { value: "never", label: "Never", description: "Stay on the fallback model until manually changed" }, + ], // Task max concurrency "task.maxConcurrency": [ { value: "0", label: "Unlimited" }, diff --git a/packages/coding-agent/src/modes/controllers/event-controller.ts b/packages/coding-agent/src/modes/controllers/event-controller.ts index d7085930c..7e18e4986 100644 --- a/packages/coding-agent/src/modes/controllers/event-controller.ts +++ b/packages/coding-agent/src/modes/controllers/event-controller.ts @@ -534,6 +534,16 @@ export class EventController { break; } + case "retry_fallback_applied": { + this.ctx.showWarning(`Fallback: ${event.from} -> ${event.to}`); + break; + } + + case "retry_fallback_succeeded": { + this.ctx.showStatus(`Fallback succeeded on ${event.model}`); + break; + } + case "ttsr_triggered": { const component = new TtsrNotificationComponent(event.rules); component.setExpanded(this.ctx.toolOutputExpanded); diff --git a/packages/coding-agent/src/modes/interactive-mode.ts b/packages/coding-agent/src/modes/interactive-mode.ts index caf1d8621..66acf5ebc 100644 --- a/packages/coding-agent/src/modes/interactive-mode.ts +++ b/packages/coding-agent/src/modes/interactive-mode.ts @@ -306,6 +306,11 @@ export class InteractiveMode implements InteractiveModeContext { const startupQuiet = settings.get("startup.quiet"); + for (const warning of this.session.configWarnings) { + this.ui.addChild(new Text(theme.fg("warning", `Warning: ${warning}`), 1, 0)); + this.ui.addChild(new Spacer(1)); + } + if (!startupQuiet) { // Add welcome header const welcome = new WelcomeComponent(this.#version, modelName, providerName, recentSessions, lspServerInfo); diff --git a/packages/coding-agent/src/session/agent-session.ts b/packages/coding-agent/src/session/agent-session.ts index d9e4737d9..c6bf87d62 100644 --- a/packages/coding-agent/src/session/agent-session.ts +++ b/packages/coding-agent/src/session/agent-session.ts @@ -55,7 +55,12 @@ import { abortableSleep, getAgentDbPath, isEnoent, logger } from "@oh-my-pi/pi-u import type { AsyncJob, AsyncJobManager } from "../async"; import type { Rule } from "../capability/rule"; import { MODEL_ROLE_IDS, type ModelRegistry } from "../config/model-registry"; -import { extractExplicitThinkingSelector, parseModelString, resolveModelRoleValue } from "../config/model-resolver"; +import { + extractExplicitThinkingSelector, + formatModelString, + parseModelString, + resolveModelRoleValue, +} from "../config/model-resolver"; import { expandPromptTemplate, type PromptTemplate, renderPromptTemplate } from "../config/prompt-templates"; import type { Settings, SkillsSettings } from "../config/settings"; import { type BashResult, executeBash as executeBashCommand } from "../exec/bash-executor"; @@ -170,6 +175,8 @@ export type AgentSessionEvent = } | { type: "auto_retry_start"; attempt: number; maxAttempts: number; delayMs: number; errorMessage: string } | { type: "auto_retry_end"; success: boolean; attempt: number; finalError?: string } + | { type: "retry_fallback_applied"; from: string; to: string; role: string } + | { type: "retry_fallback_succeeded"; model: string; role: string } | { type: "ttsr_triggered"; rules: Rule[] } | { type: "todo_reminder"; todos: TodoItem[]; attempt: number; maxAttempts: number } | { type: "todo_auto_clear" }; @@ -315,6 +322,46 @@ interface HandoffOptions { const AUTO_HANDOFF_THRESHOLD_FOCUS = renderPromptTemplate(autoHandoffThresholdFocusPrompt); +type RetryFallbackChains = Record; + +type RetryFallbackRevertPolicy = "never" | "cooldown-expiry"; + +interface RetryFallbackSelector { + raw: string; + provider: string; + id: string; + thinkingLevel: ThinkingLevel | undefined; +} + +interface ActiveRetryFallbackState { + role: string; + originalSelector: string; + originalThinkingLevel: ThinkingLevel | undefined; + lastAppliedFallbackThinkingLevel: ThinkingLevel | undefined; +} + +function parseRetryFallbackSelector(selector: string): RetryFallbackSelector | undefined { + const trimmed = selector.trim(); + if (!trimmed) return undefined; + const parsed = parseModelString(trimmed); + if (!parsed) return undefined; + return { + raw: trimmed, + provider: parsed.provider, + id: parsed.id, + thinkingLevel: parsed.thinkingLevel, + }; +} + +function formatRetryFallbackSelector(model: Model, thinkingLevel: ThinkingLevel | undefined): string { + const selector = formatModelString(model); + return thinkingLevel ? `${selector}:${thinkingLevel}` : selector; +} + +function formatRetryFallbackBaseSelector(selector: RetryFallbackSelector): string { + return `${selector.provider}/${selector.id}`; +} + const noOpUIContext: ExtensionUIContext = { select: async (_title, _options, _dialogOptions) => undefined, confirm: async (_title, _message, _dialogOptions) => false, @@ -352,6 +399,7 @@ export class AgentSession { readonly sessionManager: SessionManager; readonly settings: Settings; readonly searchDb: SearchDb | undefined; + readonly configWarnings: string[] = []; #asyncJobManager: AsyncJobManager | undefined = undefined; #scopedModels: Array<{ model: Model; thinkingLevel?: ThinkingLevel }>; @@ -391,7 +439,7 @@ export class AgentSession { #retryAttempt = 0; #retryPromise: Promise | undefined = undefined; #retryResolve: (() => void) | undefined = undefined; - + #activeRetryFallback: ActiveRetryFallbackState | undefined = undefined; // Todo completion reminder state #todoReminderCount = 0; #todoPhases: TodoPhase[] = []; @@ -478,6 +526,7 @@ export class AgentSession { this.#customCommands = config.customCommands ?? []; this.#skillsSettings = config.skillsSettings; this.#modelRegistry = config.modelRegistry; + this.#validateRetryFallbackChains(); this.#toolRegistry = config.toolRegistry ?? new Map(); this.#transformContext = config.transformContext ?? (messages => messages); this.#onPayload = config.onPayload; @@ -787,6 +836,13 @@ export class AgentSession { assistantMsg.stopReason !== "aborted" && this.#retryAttempt > 0 ) { + if (this.#activeRetryFallback && this.model) { + await this.#emitSessionEvent({ + type: "retry_fallback_succeeded", + model: formatRetryFallbackSelector(this.model, this.thinkingLevel), + role: this.#activeRetryFallback.role, + }); + } await this.#emitSessionEvent({ type: "auto_retry_end", success: true, @@ -985,6 +1041,7 @@ export class AgentSession { return; } try { + await this.#maybeRestoreRetryFallbackPrimary(); await this.agent.continue(); } catch { options?.onError?.(); @@ -2288,6 +2345,8 @@ export class AgentSession { // Reset todo reminder count on new user prompt this.#todoReminderCount = 0; + await this.#maybeRestoreRetryFallbackPrimary(); + // Validate model if (!this.model) { throw new Error( @@ -3108,6 +3167,7 @@ export class AgentSession { throw new Error(`No API key for ${model.provider}/${model.id}`); } + this.#clearActiveRetryFallback(); this.#setModelWithProviderSessionReset(model); this.sessionManager.appendModelChange(`${model.provider}/${model.id}`, role); this.settings.setModelRole(role, this.#formatRoleModelValue(role, model)); @@ -3128,6 +3188,7 @@ export class AgentSession { throw new Error(`No API key for ${model.provider}/${model.id}`); } + this.#clearActiveRetryFallback(); this.#setModelWithProviderSessionReset(model); this.sessionManager.appendModelChange(`${model.provider}/${model.id}`, "temporary"); this.settings.getStorage()?.recordModelUsage(`${model.provider}/${model.id}`); @@ -3253,6 +3314,7 @@ export class AgentSession { const next = scopedModels[nextIndex]; // Apply model + this.#clearActiveRetryFallback(); this.#setModelWithProviderSessionReset(next.model); this.sessionManager.appendModelChange(`${next.model.provider}/${next.model.id}`); this.settings.setModelRole("default", this.#formatRoleModelValue("default", next.model)); @@ -3281,11 +3343,11 @@ export class AgentSession { throw new Error(`No API key for ${nextModel.provider}/${nextModel.id}`); } + this.#clearActiveRetryFallback(); this.#setModelWithProviderSessionReset(nextModel); this.sessionManager.appendModelChange(`${nextModel.provider}/${nextModel.id}`); this.settings.setModelRole("default", this.#formatRoleModelValue("default", nextModel)); this.settings.getStorage()?.recordModelUsage(`${nextModel.provider}/${nextModel.id}`); - // Re-apply the current thinking level for the newly selected model this.setThinkingLevel(this.thinkingLevel); @@ -4768,6 +4830,217 @@ export class AgentSession { ); } + #getRetryFallbackChains(): RetryFallbackChains { + const configuredChains = this.settings.get("retry.fallbackChains"); + if (!configuredChains || typeof configuredChains !== "object") return {}; + return configuredChains as RetryFallbackChains; + } + + #validateRetryFallbackChains(): void { + const configuredChains = this.settings.get("retry.fallbackChains"); + if (configuredChains === undefined) return; + if (!configuredChains || typeof configuredChains !== "object" || Array.isArray(configuredChains)) { + const msg = "retry.fallbackChains must be a mapping of role names to selector arrays."; + logger.warn(msg); + this.configWarnings.push(msg); + return; + } + + for (const [role, chain] of Object.entries(configuredChains)) { + if (!Array.isArray(chain)) { + const msg = `Fallback chain for role '${role}' must be an array of selector strings.`; + logger.warn(msg); + this.configWarnings.push(msg); + continue; + } + for (const selectorStr of chain) { + if (typeof selectorStr !== "string") { + const msg = `Fallback chain for role '${role}' contains a non-string selector.`; + logger.warn(msg); + this.configWarnings.push(msg); + continue; + } + const parsed = parseRetryFallbackSelector(selectorStr); + if (!parsed) { + const msg = `Invalid fallback selector format in role '${role}': ${selectorStr}`; + logger.warn(msg); + this.configWarnings.push(msg); + continue; + } + const exists = this.#modelRegistry.find(parsed.provider, parsed.id); + if (!exists) { + const msg = `Fallback chain for role '${role}' references unknown model: ${selectorStr}`; + logger.warn(msg); + this.configWarnings.push(msg); + } + } + } + } + + #getRetryFallbackRevertPolicy(): RetryFallbackRevertPolicy { + return this.settings.get("retry.fallbackRevertPolicy") === "never" ? "never" : "cooldown-expiry"; + } + + #getRetryFallbackPrimarySelector(role: string): RetryFallbackSelector | undefined { + const configuredSelector = this.settings.getModelRole(role); + return configuredSelector ? parseRetryFallbackSelector(configuredSelector) : undefined; + } + + #clearActiveRetryFallback(): void { + this.#activeRetryFallback = undefined; + } + + #isRetryFallbackSelectorSuppressed(selector: RetryFallbackSelector): boolean { + return this.#modelRegistry.isSelectorSuppressed(selector.raw); + } + + #noteRetryFallbackCooldown(currentSelector: string, retryAfterMs: number | undefined, errorMessage: string): void { + let cooldownMs = retryAfterMs; + if (!cooldownMs || cooldownMs <= 0) { + const reason = parseRateLimitReason(errorMessage); + cooldownMs = reason === "UNKNOWN" ? 5 * 60 * 1000 : calculateRateLimitBackoffMs(reason); + } + this.#modelRegistry.suppressSelector(currentSelector, Date.now() + cooldownMs); + } + + #resolveRetryFallbackRole(currentSelector: string): string | undefined { + const parsedCurrent = parseRetryFallbackSelector(currentSelector); + if (!parsedCurrent) return undefined; + const currentBaseSelector = formatRetryFallbackBaseSelector(parsedCurrent); + for (const role of Object.keys(this.#getRetryFallbackChains())) { + const primarySelector = this.#getRetryFallbackPrimarySelector(role); + if (!primarySelector) continue; + if (primarySelector.raw === currentSelector) return role; + if (formatRetryFallbackBaseSelector(primarySelector) === currentBaseSelector) return role; + } + return undefined; + } + + #getRetryFallbackEffectiveChain(role: string): RetryFallbackSelector[] { + const primarySelector = this.#getRetryFallbackPrimarySelector(role); + if (!primarySelector) return []; + const chain = [primarySelector]; + const seen = new Set([primarySelector.raw]); + for (const selector of this.#getRetryFallbackChains()[role] ?? []) { + const parsed = parseRetryFallbackSelector(selector); + if (!parsed || seen.has(parsed.raw)) continue; + seen.add(parsed.raw); + chain.push(parsed); + } + return chain; + } + + #findRetryFallbackCandidates(role: string, currentSelector: string): RetryFallbackSelector[] { + const chain = this.#getRetryFallbackEffectiveChain(role); + if (chain.length <= 1) return []; + const parsedCurrent = parseRetryFallbackSelector(currentSelector); + const currentBaseSelector = parsedCurrent ? formatRetryFallbackBaseSelector(parsedCurrent) : undefined; + const exactIndex = chain.findIndex(selector => selector.raw === currentSelector); + if (exactIndex >= 0) return chain.slice(exactIndex + 1); + const baseIndex = currentBaseSelector + ? chain.findIndex(selector => formatRetryFallbackBaseSelector(selector) === currentBaseSelector) + : -1; + if (baseIndex >= 0) return chain.slice(baseIndex + 1); + return chain.slice(1); + } + + async #applyRetryFallbackCandidate( + role: string, + selector: RetryFallbackSelector, + currentSelector: string, + ): Promise { + const candidate = this.#modelRegistry.find(selector.provider, selector.id); + if (!candidate) { + throw new Error(`Retry fallback model not found: ${selector.raw}`); + } + const apiKey = await this.#modelRegistry.getApiKey(candidate, this.sessionId); + if (!apiKey) { + throw new Error(`No API key for retry fallback ${selector.raw}`); + } + + const currentThinkingLevel = this.thinkingLevel; + const nextThinkingLevel = selector.thinkingLevel ?? currentThinkingLevel; + + this.#setModelWithProviderSessionReset(candidate); + this.sessionManager.appendModelChange(`${candidate.provider}/${candidate.id}`, "temporary"); + this.settings.getStorage()?.recordModelUsage(`${candidate.provider}/${candidate.id}`); + this.setThinkingLevel(nextThinkingLevel); + if (!this.#activeRetryFallback) { + this.#activeRetryFallback = { + role, + originalSelector: currentSelector, + originalThinkingLevel: currentThinkingLevel, + lastAppliedFallbackThinkingLevel: nextThinkingLevel, + }; + } else { + this.#activeRetryFallback.lastAppliedFallbackThinkingLevel = nextThinkingLevel; + } + await this.#emitSessionEvent({ + type: "retry_fallback_applied", + from: currentSelector, + to: selector.raw, + role, + }); + } + + async #tryRetryModelFallback(currentSelector: string): Promise { + const role = this.#activeRetryFallback?.role ?? this.#resolveRetryFallbackRole(currentSelector); + if (!role) return false; + + for (const selector of this.#findRetryFallbackCandidates(role, currentSelector)) { + if (this.#isRetryFallbackSelectorSuppressed(selector)) continue; + const candidate = this.#modelRegistry.find(selector.provider, selector.id); + if (!candidate) continue; + const apiKey = await this.#modelRegistry.getApiKey(candidate, this.sessionId); + if (!apiKey) continue; + await this.#applyRetryFallbackCandidate(role, selector, currentSelector); + return true; + } + + return false; + } + + async #maybeRestoreRetryFallbackPrimary(): Promise { + if (!this.#activeRetryFallback) return; + if (this.#getRetryFallbackRevertPolicy() !== "cooldown-expiry") return; + + const { + originalSelector: originalSelectorRaw, + originalThinkingLevel, + lastAppliedFallbackThinkingLevel, + } = this.#activeRetryFallback; + const originalSelector = parseRetryFallbackSelector(originalSelectorRaw); + if (!originalSelector) { + this.#clearActiveRetryFallback(); + return; + } + + const currentModel = this.model; + if (!currentModel) return; + const currentSelector = formatRetryFallbackSelector(currentModel, this.thinkingLevel); + if (currentSelector === originalSelector.raw) { + if (!this.#isRetryFallbackSelectorSuppressed(originalSelector)) { + this.#clearActiveRetryFallback(); + } + return; + } + if (this.#isRetryFallbackSelectorSuppressed(originalSelector)) return; + + const primaryModel = this.#modelRegistry.find(originalSelector.provider, originalSelector.id); + if (!primaryModel) return; + const apiKey = await this.#modelRegistry.getApiKey(primaryModel, this.sessionId); + if (!apiKey) return; + + const currentThinkingLevel = this.thinkingLevel; + const thinkingToApply = + currentThinkingLevel === lastAppliedFallbackThinkingLevel ? originalThinkingLevel : currentThinkingLevel; + this.#setModelWithProviderSessionReset(primaryModel); + this.sessionManager.appendModelChange(`${primaryModel.provider}/${primaryModel.id}`, "temporary"); + this.settings.getStorage()?.recordModelUsage(`${primaryModel.provider}/${primaryModel.id}`); + this.setThinkingLevel(thinkingToApply); + this.#clearActiveRetryFallback(); + } + #parseRetryAfterMsFromError(errorMessage: string): number | undefined { const now = Date.now(); const retryAfterMsMatch = /retry-after-ms\s*[:=]\s*(\d+)/i.exec(errorMessage); @@ -4847,12 +5120,13 @@ export class AgentSession { } const errorMessage = message.errorMessage || "Unknown error"; + const parsedRetryAfterMs = this.#parseRetryAfterMsFromError(errorMessage); let delayMs = retrySettings.baseDelayMs * 2 ** (this.#retryAttempt - 1); + let switchedCredential = false; + let switchedModel = false; if (this.model && isUsageLimitError(errorMessage)) { - const retryAfterMs = - this.#parseRetryAfterMsFromError(errorMessage) ?? - calculateRateLimitBackoffMs(parseRateLimitReason(errorMessage)); + const retryAfterMs = parsedRetryAfterMs ?? calculateRateLimitBackoffMs(parseRateLimitReason(errorMessage)); const switched = await this.#modelRegistry.authStorage.markUsageLimitReached( this.model.provider, this.sessionId, @@ -4862,6 +5136,7 @@ export class AgentSession { }, ); if (switched) { + switchedCredential = true; delayMs = 0; } else if (retryAfterMs > delayMs) { // No more accounts to switch to — wait out the backoff @@ -4869,6 +5144,17 @@ export class AgentSession { } } + const currentSelector = this.model ? formatRetryFallbackSelector(this.model, this.thinkingLevel) : undefined; + if (!switchedCredential && currentSelector) { + this.#noteRetryFallbackCooldown(currentSelector, parsedRetryAfterMs, errorMessage); + switchedModel = await this.#tryRetryModelFallback(currentSelector); + if (switchedModel) { + delayMs = 0; + } else if (parsedRetryAfterMs && parsedRetryAfterMs > delayMs) { + delayMs = parsedRetryAfterMs; + } + } + await this.#emitSessionEvent({ type: "auto_retry_start", attempt: this.#retryAttempt, diff --git a/packages/coding-agent/test/agent-session-retry-fallback.test.ts b/packages/coding-agent/test/agent-session-retry-fallback.test.ts new file mode 100644 index 000000000..ec2cf75a2 --- /dev/null +++ b/packages/coding-agent/test/agent-session-retry-fallback.test.ts @@ -0,0 +1,389 @@ +import { afterEach, beforeEach, describe, expect, it } from "bun:test"; +import * as path from "node:path"; +import { Agent } from "@oh-my-pi/pi-agent-core"; +import { type AssistantMessage, Effort, getBundledModel, type Model } from "@oh-my-pi/pi-ai"; +import { AssistantMessageEventStream } from "@oh-my-pi/pi-ai/utils/event-stream"; +import { ModelRegistry } from "@oh-my-pi/pi-coding-agent/config/model-registry"; +import { Settings } from "@oh-my-pi/pi-coding-agent/config/settings"; +import { AgentSession, type AgentSessionEvent } from "@oh-my-pi/pi-coding-agent/session/agent-session"; +import { AuthStorage } from "@oh-my-pi/pi-coding-agent/session/auth-storage"; +import { SessionManager } from "@oh-my-pi/pi-coding-agent/session/session-manager"; +import { TempDir } from "@oh-my-pi/pi-utils"; + +class MockAssistantStream extends AssistantMessageEventStream {} + +function createAssistantMessage( + model: Model, + options: { text?: string; stopReason: "stop" | "error"; errorMessage?: string }, +): AssistantMessage { + return { + role: "assistant", + content: options.text ? [{ type: "text", text: options.text }] : [], + api: model.api, + provider: model.provider, + model: model.id, + usage: { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: options.stopReason, + errorMessage: options.errorMessage, + timestamp: Date.now(), + }; +} + +async function _waitFor(predicate: () => boolean, timeoutMs = 1000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (predicate()) return; + await Bun.sleep(10); + } + throw new Error("Timed out waiting for condition"); +} + +describe("AgentSession retry fallback", () => { + let tempDir: TempDir; + let authStorage: AuthStorage; + let modelRegistry: ModelRegistry; + let session: AgentSession | undefined; + + beforeEach(async () => { + tempDir = TempDir.createSync("@pi-retry-fallback-"); + authStorage = await AuthStorage.create(path.join(tempDir.path(), "testauth.db")); + authStorage.setRuntimeApiKey("anthropic", "anthropic-test-key"); + authStorage.setRuntimeApiKey("openai", "openai-test-key"); + modelRegistry = new ModelRegistry(authStorage); + }); + + afterEach(async () => { + if (session) { + await session.dispose(); + session = undefined; + } + authStorage.close(); + tempDir.removeSync(); + }); + + it("advances through a role-keyed fallback chain across retries", async () => { + const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5"); + const firstFallback = getBundledModel("openai", "gpt-4o-mini"); + const secondFallback = getBundledModel("openai", "gpt-4o"); + if (!primaryModel || !firstFallback || !secondFallback) { + throw new Error("Expected bundled test models to exist"); + } + + const requestedModels: string[] = []; + const retryStartEvents: Array> = []; + const retryEndEvents: Array> = []; + const fallbackAppliedEvents: Array> = []; + const fallbackSucceededEvents: Array> = []; + + const agent = new Agent({ + getApiKey: provider => `${provider}-test-key`, + initialState: { + model: primaryModel, + systemPrompt: "Test", + tools: [], + messages: [], + }, + streamFn: model => { + requestedModels.push(`${model.provider}/${model.id}`); + const stream = new MockAssistantStream(); + queueMicrotask(() => { + if (model.provider === primaryModel.provider && model.id === primaryModel.id) { + const message = createAssistantMessage(model, { + stopReason: "error", + errorMessage: "overloaded_error: provider returned error 503", + }); + stream.push({ type: "start", partial: message }); + stream.push({ type: "error", reason: "error", error: message }); + return; + } + if (model.provider === firstFallback.provider && model.id === firstFallback.id) { + const message = createAssistantMessage(model, { + stopReason: "error", + errorMessage: "service unavailable: 503 overloaded", + }); + stream.push({ type: "start", partial: message }); + stream.push({ type: "error", reason: "error", error: message }); + return; + } + if (model.provider === secondFallback.provider && model.id === secondFallback.id) { + const message = createAssistantMessage(model, { + text: "Recovered on second fallback", + stopReason: "stop", + }); + stream.push({ + type: "start", + partial: createAssistantMessage(model, { text: "", stopReason: "stop" }), + }); + stream.push({ type: "done", reason: "stop", message }); + return; + } + throw new Error(`Unexpected model requested during retry fallback test: ${model.provider}/${model.id}`); + }); + return stream; + }, + }); + + const settings = Settings.isolated({ + "compaction.enabled": false, + "retry.baseDelayMs": 5, + "retry.fallbackChains": { + default: [ + `${firstFallback.provider}/${firstFallback.id}`, + `${secondFallback.provider}/${secondFallback.id}`, + ], + }, + }); + settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}`); + + session = new AgentSession({ + agent, + sessionManager: SessionManager.inMemory(), + settings, + modelRegistry, + }); + + session.subscribe(event => { + if (event.type === "auto_retry_start") { + retryStartEvents.push(event); + } + if (event.type === "auto_retry_end") { + retryEndEvents.push(event); + } + if (event.type === "retry_fallback_applied") { + fallbackAppliedEvents.push(event); + } + if (event.type === "retry_fallback_succeeded") { + fallbackSucceededEvents.push(event); + } + }); + + await session.prompt("Recover from rate limits"); + await session.waitForIdle(); + + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${firstFallback.provider}/${firstFallback.id}`, + `${secondFallback.provider}/${secondFallback.id}`, + ]); + expect(session.model?.provider).toBe(secondFallback.provider); + expect(session.model?.id).toBe(secondFallback.id); + expect(retryStartEvents.map(event => event.delayMs)).toEqual([0, 0]); + expect(fallbackAppliedEvents).toEqual([ + { + type: "retry_fallback_applied", + from: `${primaryModel.provider}/${primaryModel.id}`, + to: `${firstFallback.provider}/${firstFallback.id}`, + role: "default", + }, + { + type: "retry_fallback_applied", + from: `${firstFallback.provider}/${firstFallback.id}`, + to: `${secondFallback.provider}/${secondFallback.id}`, + role: "default", + }, + ]); + expect(retryEndEvents).toHaveLength(1); + expect(retryEndEvents[0]).toMatchObject({ success: true, attempt: 2 }); + expect(fallbackSucceededEvents).toEqual([ + { + type: "retry_fallback_succeeded", + model: `${secondFallback.provider}/${secondFallback.id}`, + role: "default", + }, + ]); + }); + + it("suppresses cooled selectors and lazily reverts to the role primary after cooldown expiry", async () => { + const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5"); + const fallbackModel = getBundledModel("openai", "gpt-4o-mini"); + if (!primaryModel || !fallbackModel) { + throw new Error("Expected bundled test models to exist"); + } + + const requestedModels: string[] = []; + let primaryAttempts = 0; + + const agent = new Agent({ + getApiKey: provider => `${provider}-test-key`, + initialState: { + model: primaryModel, + systemPrompt: "Test", + tools: [], + messages: [], + }, + streamFn: model => { + requestedModels.push(`${model.provider}/${model.id}`); + const stream = new MockAssistantStream(); + queueMicrotask(() => { + if (model.provider === primaryModel.provider && model.id === primaryModel.id && primaryAttempts === 0) { + primaryAttempts += 1; + const message = createAssistantMessage(model, { + stopReason: "error", + errorMessage: "rate limit exceeded retry-after-ms=200", + }); + stream.push({ type: "start", partial: message }); + stream.push({ type: "error", reason: "error", error: message }); + return; + } + const message = createAssistantMessage(model, { + text: `ok:${model.provider}/${model.id}`, + stopReason: "stop", + }); + stream.push({ type: "start", partial: createAssistantMessage(model, { text: "", stopReason: "stop" }) }); + stream.push({ type: "done", reason: "stop", message }); + }); + return stream; + }, + }); + + const settings = Settings.isolated({ + "compaction.enabled": false, + "retry.baseDelayMs": 5, + "retry.fallbackChains": { + default: [`${fallbackModel.provider}/${fallbackModel.id}`], + }, + "retry.fallbackRevertPolicy": "cooldown-expiry", + }); + settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}`); + + session = new AgentSession({ + agent, + sessionManager: SessionManager.inMemory(), + settings, + modelRegistry, + }); + + await session.prompt("First prompt triggers fallback"); + await session.waitForIdle(); + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + ]); + expect(session.model?.provider).toBe(fallbackModel.provider); + expect(session.model?.id).toBe(fallbackModel.id); + + await session.prompt("Immediate second prompt should stay on fallback"); + await session.waitForIdle(); + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + ]); + expect(session.model?.provider).toBe(fallbackModel.provider); + expect(session.model?.id).toBe(fallbackModel.id); + + await Bun.sleep(240); + await session.prompt("Third prompt should lazily revert to primary"); + await session.waitForIdle(); + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + `${primaryModel.provider}/${primaryModel.id}`, + ]); + expect(session.model?.provider).toBe(primaryModel.provider); + expect(session.model?.id).toBe(primaryModel.id); + }); + + it("preserves thinking on bare fallback selectors and does not overwrite user thinking on restore", async () => { + const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5"); + const fallbackModel = getBundledModel("openai", "gpt-4o-mini"); + if (!primaryModel || !fallbackModel) { + throw new Error("Expected bundled test models to exist"); + } + + const requestedModels: string[] = []; + let primaryAttempts = 0; + + const agent = new Agent({ + getApiKey: provider => `${provider}-test-key`, + initialState: { + model: primaryModel, + systemPrompt: "Test", + tools: [], + messages: [], + }, + streamFn: model => { + requestedModels.push(`${model.provider}/${model.id}`); + const stream = new MockAssistantStream(); + queueMicrotask(() => { + if (model.provider === primaryModel.provider && model.id === primaryModel.id && primaryAttempts === 0) { + primaryAttempts += 1; + const message = createAssistantMessage(model, { + stopReason: "error", + errorMessage: "rate limit exceeded retry-after-ms=200", + }); + stream.push({ type: "start", partial: message }); + stream.push({ type: "error", reason: "error", error: message }); + return; + } + const message = createAssistantMessage(model, { + text: `ok:${model.provider}/${model.id}`, + stopReason: "stop", + }); + stream.push({ type: "start", partial: createAssistantMessage(model, { text: "", stopReason: "stop" }) }); + stream.push({ type: "done", reason: "stop", message }); + }); + return stream; + }, + }); + + const settings = Settings.isolated({ + "compaction.enabled": false, + "retry.baseDelayMs": 5, + "retry.fallbackChains": { + default: [`${fallbackModel.provider}/${fallbackModel.id}`], + }, + "retry.fallbackRevertPolicy": "cooldown-expiry", + }); + settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}:high`); + + session = new AgentSession({ + agent, + sessionManager: SessionManager.inMemory(), + settings, + modelRegistry, + thinkingLevel: Effort.High, + }); + + await session.prompt("First prompt triggers bare-selector fallback"); + await session.waitForIdle(); + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + ]); + expect(session.model?.provider).toBe(fallbackModel.provider); + expect(session.model?.id).toBe(fallbackModel.id); + expect(session.thinkingLevel).toBeUndefined(); + + session.setThinkingLevel(Effort.Low); + await Bun.sleep(240); + await session.prompt("Second prompt should restore model but preserve user thinking change"); + await session.waitForIdle(); + expect(requestedModels).toEqual([ + `${primaryModel.provider}/${primaryModel.id}`, + `${fallbackModel.provider}/${fallbackModel.id}`, + `${primaryModel.provider}/${primaryModel.id}`, + ]); + expect(session.model?.provider).toBe(primaryModel.provider); + expect(session.model?.id).toBe(primaryModel.id); + expect(session.thinkingLevel).toBeUndefined(); + }); + + it("normalizes suppression by base selector and clears it on model refresh", async () => { + const future = Date.now() + 60_000; + modelRegistry.suppressSelector("openai/gpt-4o:high", future); + expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o")).toBe(true); + expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o:low")).toBe(true); + + await modelRegistry.refresh("offline"); + expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o")).toBe(false); + }); +});