feat(core): add first-class retry fallback chains for model/provider failover (#541)

* feat(rules): implemented alwaysApply auto-injection into system prompt

rules with alwaysApply: true were parsed by all providers and used to
exclude the rule from rulebookRules, but the inclusion half was never
built — rule content was silently dropped. now:

- full content is injected directly into the system prompt (before the
  rulebook rules section) in both default and custom prompt templates
- rules remain addressable via rule:// for re-reading
- ttsr rules still take priority (condition + alwaysApply goes to ttsr only)

updated rulebook-matching-pipeline.md to reflect the three-bucket split
(ttsr > always-apply > rulebook) and corrected the rule:// resolution
docs.

* feat(browser): implement screenshot path saving

- Add `browser.screenshotDir` setting (tools tab) for a persistent
  default screenshot directory, configurable via /settings
- Honour the existing `path` parameter in the screenshot action,
  which was declared in the schema but never consumed by the implementation
- Resolution order: params.path (abs) > join(screenshotDir, params.path)
  > join(screenshotDir, screenshot-<timestamp>.png) > /tmp only
- Writes full-resolution buffer to disk (not the API-compressed copy)
- Creates destination directory recursively if it doesn't exist
- details.screenshotPath reflects the actual saved location

Fixes: path param silently ignored since removal in v11.5.0

* feat(browser): expand ~ in screenshotDir and path params

Users can now configure browser.screenshotDir as ~/Downloads or
~/Pictures and use params.path as ~/screenshots/foo.png without
needing to supply fully-qualified paths.

* fix(browser): write screenshot to exactly one location

Previously wrote to /tmp unconditionally then copied to user path,
resulting in two files. Now resolves a single destination upfront:
  1. params.path (absolute, or relative to screenshotDir/cwd)
  2. screenshotDir + auto-timestamp filename
  3. /tmp fallback (original behaviour, unchanged)

Also: user-defined destinations receive the full-res buffer;
/tmp fallback retains the API-compressed copy as before.

* feat(settings): add text input support for plain string settings

Settings with type: "string" and no submenu now render as an editable
text field in the /settings TUI panel instead of being silently skipped.

- settings-defs.ts: add TextInputSettingDef interface; pathToSettingDef
  falls through to { type: "text" } for any plain string schema entry
- settings-selector.ts: add TextInputSubmenu class (mirrors
  ConfigInputSubmenu from plugin-settings.ts); add "text" case to
  #defToItem; add #createTextInput method

This makes browser.screenshotDir visible and editable in the Tools tab.
Empty field on submit clears the setting (falls back to /tmp behavior).

* fix(settings): text input — cursor at end, block tab navigation

- Cursor: call handleInput(ctrl+e) after setValue to jump to end of
  pre-filled string instead of leaving it at position 0
- Tab/arrow guard: add #textInputActive flag; suppress tab-switch and
  left/right routing to tab bar while a TextInputSubmenu is open, so
  arrow keys reach the Input component's cursor movement handlers
  instead of switching settings tabs

* fix(browser): expand ~\ (Windows backslash) in screenshot paths

expandHome now handles both Unix ~/... and Windows ~\... separators,
matching user expectation on all platforms. Addresses Codex review.

* fix(coding-agent): use expandPath for screenshots

* docs: add changelog for screenshot path option

Bug fixes:
- Add browser.screenshotDir to settings-schema.ts (lost during rebase conflict)
- Fix stale expandHome comment to expandPath in settings-selector.ts
- Final screenshotDir description: Directory to save screenshots with ~ support

* fix: biome format corrections for screenshot-path-option branch

* fix: add missing #textInputActive field and screenshotDir default

* fix(browser): align screenshot metadata with saved file contents

When screenshotDir or params.path is set, the full-resolution PNG buffer
is written to disk. Previously mimeType/bytes in details still reflected
the resized payload sent to the model, making metadata inconsistent with
the actual saved file.

Now savedBuffer/savedMimeType track what is written, and details reflects
that. Display output distinguishes 'Saved' vs 'Model' when full-res is
used, and collapses to a single Format/Dimensions line for temp-only.

* fix(browser): resolve params.path relative to cwd regardless of screenshotDir

screenshotDir is a default save location, not an anchor for explicit
paths. A relative params.path should always resolve against cwd so its
semantics are stable and predictable regardless of user settings.

* fix(tui): enforce strict line budget for collapsed tool output

The grep, ast_grep, and ast_edit renderers used group-count-based
collapse that always included the first group unconditionally,
allowing collapsed output to remain visually large when a single
group contained many lines.

Add maxCollapsedLines to renderTreeList that enforces a strict
total-line cap in collapsed mode. Items that exceed the remaining
budget are skipped entirely (no broken fragments). The isLast tree
branch is computed after the budget check to avoid double-last
branches when a summary line follows.

Remove the per-tool getCollapsedMatchLimit / getCollapsedChangeLimit
helpers that are now redundant.

Fixes #455

Made-with: Cursor

* WIP

* WIP

* cleanup

* docs(coding-agent): update changelog for custom model tags and cycle order

* Fix custom model precedence across load and refresh

* feat(ask): add multiline editor support for custom input

* feat(extension-ui): add dialog options and abort signal support to editor

* docs(ask): add multiline editor and timeout behavior guidance

* docs(ask): simplify multiline input documentation

* fix(tui): preserve terminal scrollback during full redraws

Replace destructive \x1b[3J\x1b[2J\x1b[H full-redraw sequence with
scrollback-preserving repaint helpers:

- seedTranscript: first paint with no prior frame, writes full transcript
  without clearing scrollback
- repaintViewport: trusted-frame repaints that scroll viewport-shift
  delta into scrollback before overwriting visible rows in-place
- Height-increase handler that pushes revealed scrollback back before
  reclaiming the display

Replace requestRender(true) state destruction with a one-shot
without resetting #previousLines or cursor bookkeeping.

Track #previousHeight to detect terminal height increases that pull
scrollback lines into the visible area.

fixes #507

* fix(tui): fix exit gaps, content shrink drift, and overlay cursor recovery

- Rewrite stop() to use viewport-relative cursor positioning instead of
  content-length, preventing blank gaps when content is shorter than viewport
- Replace trailing \r\n\x1b[2K clear loops in repaintViewport() and
  height-increase path with \r\n\x1b[J to avoid cursor drift past content
- Add 12 TUI regression tests for exit gaps, content shrink, and overlay
  dismiss cursor recovery
- Add 6 coding-agent controller tests for /new and /tree commands

* fix(tui): redraw sparse height increases atomically

* fix(coding-agent,tui): address review regressions

* fix(tui): reseed after terminal resume

* test(ai): avoid leaking kagi module mocks

* fix(ask): keep multiline custom input in prompt gutter

* fix(ask): preserve multiselect choices on editor dismiss

* fix(ask): honor app interrupt in prompt editor

* fix(coding-agent): preserve new-session approval state

* feat(core): add session-scoped model/provider retry fallback policy

* feat(core): validate retry fallback chains on session startup

* fix(ask): preserve prior answer when custom editor dismissed in single-select

* fix(core): harden retry fallback policy semantics

* fix(core): correct retry fallback edge cases

* test(coding-agent): removed macOS fallback test case from theme detection

- Removed test case for macOS fallback behavior inside Zellij.

* refactor: simplified null checks using optional chaining across TypeScript and Rust modules

- Simplified null/empty checks across TypeScript codebase using optional chaining operator (?.) for improved readability.
- Replaced explicit null checks in validation logic with optional chaining in oauth-discovery, gemini-cli, claude, zai, and lsp modules.
- Updated error handling in Rust command invocation to use double question mark operator (??) for cmd_result.
- Consolidated null validation patterns across tools (bash-skill-urls, browser, gemini-image, resolve) and keybindings using optional chaining.

* chore: bump version to 13.15.1

* fix(core): address PR review on fallback retry behavior

* test(ai): refactored auth storage test to use spyOn for cleaner mocks

- Refactored auth storage test to use vi.spyOn() instead of vi.mock() for cleaner mock management.
- Simplified mock type definitions by leveraging bun:test's Mock type import.

* chore: bump version to 13.15.2

* Fix stale OpenAI Responses replay across session boundaries (#534)

* Fix stale OpenAI Responses replay across session boundaries

Fixes #505

* Fix CI tests for session replay change

* Harden session reload and switch rollback

* Guard session switch snapshots

* fix(coding-agent): preserve responses replay snapshots

* Normalize pasted image formats before attach (#543)

Co-authored-by: iter <itertoolz@gmail.com>

* Allow overriding the Codex web search model (#516)

* Allow codex web search model override

* Handle blank codex web search model

* feat(ai): add gemini-3.1-pro-preview models to google-vertex provider (#521)

Add gemini-3.1-pro-preview and gemini-3.1-pro-preview-customtools to
the google-vertex provider in models.json, matching the existing
google-generative-ai entries.

Fixes #520

Co-authored-by: Muness Castle <munesscastle@artium.ai>

* Make temporary model selector keybinding configurable (#539)

Fixes #533

* chore: bump models.json

* refactor(coding-agent): migrated test mocks to vitest spyOn with Symbol.dispose cleanup

- Migrated test mocking from bun:test mock API to vitest spyOn pattern across 9 test files.
- Extracted mock setup logic into reusable helper functions with Symbol.dispose cleanup pattern.
- Replaced manual beforeEach/afterEach and try-finally blocks with TypeScript 5.2 using declarations.
- Removed 178 lines of boilerplate mock initialization and restoration code from test suite.

* chore: bump version to 13.15.3

* fix(ask): restore prompt-style enter handling

* fix(tui): avoid blank scrollback regressions

* fix(models): keep same-id replacements authoritative

* fix(tui): enforce collapsed line budgets

* fix(models): unify selector role sources

* fix(rules): dedupe always-apply prompt injection

* fix(browser): show saved screenshot path

* style: format merged PR fixes

* refactor(coding-agent): restructured screenshot and prompt utilities into focused helpers

- Extracted screenshot formatting logic into dedicated `formatScreenshot()` function with options support.
- Consolidated prompt source deduplication into `dedupePromptSource()` helper to prevent rule duplication.
- Refactored editor text sanitization to use `replaceTabs()` utility for consistent tab width handling.
- Added test coverage verifying editor respects configured tab width when loading text programmatically.

* refactor(coding-agent): restructured validation and rendering for consistency

- Refactored theme color validation to use single source of truth with THEME_COLOR_RECORD object.
- Simplified model registry to defer per-model overrides to dedicated method and use constant for role IDs.
- Refactored tree list rendering to pre-render items once for consistent line counts across phases.
- Refactored question result formatting to use early returns and consistently include question ID in output.
- Updated hook editor hint text to include ctrl+g external editor option when prompt style is enabled.
- Removed unused isLogicalLineStart property from LayoutLine interface in editor component.

* revert: 535 due to TUI regressions

* test(coding-agent): corrected hook-editor assertion for keybinding render

- Corrected assertion in hook-editor test to verify external editor keybinding is rendered.

* feat(tools): added root path alias to resolve bare / to working directory

- Added root path alias feature to resolve bare `/` to session working directory in path resolution.
- Updated browser tool to use `resolveToCwd()` for consistent workspace-relative path handling.
- Added comprehensive test suite validating root path alias resolution across grep, read, find, ast_grep, and ast_edit tools.

* feat(prompts/tools): clarified hashline block boundary handling with examples

- Improved hashline tool documentation with clearer guidance on block boundary handling and closing delimiter duplication prevention.
- Added concrete example demonstrating correct anchor placement when replacing entire blocks including closing braces.
- Reorganized boundary duplication warnings into actionable self-check guidance with visual comparison steps.

* chore: bump version to 13.16.0

* fix(coding-agent): fixed python kernel startup hangs (#548)

* fix(coding-agent): fixed python kernel startup hangs

* fix(coding-agent): fixed startup timeout regressions

* fix(coding-agent): preserved startup cancellation typing

* perf(pi-natives): optimized memory allocation with MiMalloc integration

- Integrated MiMalloc as global allocator to improve memory allocation performance.

* feat: fff

- Added SearchDb class for stateful shared search database instances enabling persistent file indexing and frecency tracking across grep, glob, and fuzzyFind operations.
- Added optional db parameter to grep(), glob(), and fuzzyFind() functions for database-backed searching with improved performance via cached file indices.
- Replaced grep-searcher with fff-grep and added fff-search dependency for enhanced file discovery and search capabilities with memory-mapped file support.
- Migrated fuzzy file discovery from fd module to fff module with SearchDb integration for stateful caching and improved search performance.
- Exported SearchDb type from @oh-my-pi/pi-natives public API for type-safe usage in grep, glob, and fuzzyFind workflows.

* feat(pi-natives): added unified picker coordination for file search operations

- Added `wait_for_picker_scan()` function to search_db module with cancellation token support for polling picker scan completion.
- Integrated picker-based file search into glob matching logic with `collect_files_from_picker()` helper to reuse shared SearchDb picker results.
- Refactored fff and grep modules to use centralized `wait_for_picker_scan()` wrapper instead of direct FilePicker calls, improving cancellation handling.
- Propagated SearchDb instance through agent session initialization and input controller to enable unified file picker coordination across search operations.

* chore: bump version to 13.16.1

* fix: install zig in CI

* feat(tui): make inline image max-width configurable via tui.maxInlineImageColumns (#551)

* feat(tui): make inline image max-width configurable via tui.maxInlineImageColumns

* fix(tui): handle 0 as unlimited in maxInlineImageColumns; drop || undefined coercion

* feat(browser): auto-detect NixOS and use system Chromium (#550)

Puppeteer's bundled Chromium is a dynamically-linked FHS binary that
cannot run on NixOS. On startup, resolveSystemChromium() checks for
/etc/NIXOS and searches for a usable binary in order:

  1. chromium on PATH
  2. chromium-browser on PATH
  3. ~/.nix-profile/bin/chromium
  4. /run/current-system/sw/bin/chromium

The resolved path is passed as executablePath to puppeteer.launch().
Result is cached per process. On non-NixOS systems the function returns
undefined immediately, leaving Puppeteer's default resolution intact.

* feat(pi-natives): added automatic parenthesis escaping in regex patterns

- Added automatic escaping of unescaped parentheses in regex patterns when group syntax errors occur, enabling literal function call patterns like `fetchAnthropicProvider(` to work as search queries.
- Extracted regex matcher builder into separate function for reusability and error recovery logic.
- Added 2 test cases validating parenthesis escaping behavior for both escaped and literal parentheses.
- Fixed documentation formatting in sanitize_braces comment.

* style: reformat

* chore: bump version to 13.16.2

* fix: only show update banner when npm version is strictly newer (#552)

* fix(ai): corrected OAuth credential updates to replace in-place instead of accumulating soft-deleted rows

- Fixed OAuth credential updates to replace matching credentials in-place rather than creating disabled rows, preventing unbounded accumulation of soft-deleted credentials.
- Modified OAuth credential saving to preserve unrelated identities instead of replacing all credentials for a provider.
- Updated credential identity resolution to use provider context for more accurate email deduplication.
- Implemented upsertAuthCredentialForProvider method to handle credential matching and in-place updates.
- Added 5 test cases covering credential preservation across reauth, multi-account scenarios, and stale cache handling.

* chore: bump version to 13.16.3

* feat: introduced unified range API for hashline edits and model catalog updates

- Simplified hashline edit location API by replacing separate `line` and `block` properties with unified `range` property accepting `{ pos, end }` anchors.
- Renamed hashline helper functions from `hlineref`/`hlinefull` to `href`/`hline` for improved brevity and consistency.
- Added detection for `kysely-codegen` generated files in auto-generated file guard with corresponding test coverage.
- Added 13 new AI model configurations and updated token limits and pricing for existing models across multiple providers.
- Enhanced file type validation in grep native to reject symlinks, FIFOs, sockets, and non-regular files with improved error handling.

* chore: bump version to 13.16.4

* fix: pin rustc-hash to 2.1.1 to avoid SIGILL on CI

rustc-hash 2.1.2 (released today) refactored hash_bytes to use
split_first_chunk, which produces illegal instructions when compiled
with nightly + -C target-cpu=x86-64-v3 on CI runners.

* fix(ci): pin nightly to 2026-03-27 to avoid codegen SIGILL regression

Today's nightly produces illegal instructions when compiled with
-C target-cpu=x86-64-v3. Reverts the unnecessary rustc-hash pin from
the previous commit since the real cause is the nightly compiler.

Also lets Cargo.lock return to rustc-hash 2.1.2 (not the culprit).

* fix(ci): add rustup target fallback for pinned nightly cross-compile

* fix(coding-agent): do not prompt to use grep and find tools if they are disabled (#566)

Co-authored-by: le-cameleon <200889489+le-cameleon@users.noreply.github.com>

* fix(natives): skipped grep special files (#565)

avoided opening fifos and other special filesystem nodes during grep and added fifo regressions in native and coding-agent tests.

* fix: skill baseDir regex fails on Windows backslash paths (#554)

The regex that strips SKILL.md from the path to compute baseDir only
matches forward slashes. On Windows where paths use backslashes, the
replace is a no-op and baseDir equals the full SKILL.md file path.

This breaks sub-path resolution for skills: the subpath gets appended
to the SKILL.md file path instead of the skill directory.

Fix: use character class matching both path separators.

---------

Co-authored-by: deadcode-walker <268043493+deadcode-walker@users.noreply.github.com>
Co-authored-by: Rens Tillmann <rens@super-forms.com>
Co-authored-by: haiyang.zhou <haiyang.zhou@seamoney.com>
Co-authored-by: Leo P <junk@slact.net>
Co-authored-by: Zakhar Kogan <36503576+zaharkogan@users.noreply.github.com>
Co-authored-by: Vu Anh Nguyen <vuanhng00@gmail.com>
Co-authored-by: can1357 <me@can.ac>
Co-authored-by: daandden <64765666+daandden@users.noreply.github.com>
Co-authored-by: iter <72358817+itertea@users.noreply.github.com>
Co-authored-by: iter <itertoolz@gmail.com>
Co-authored-by: Cheol Kang <dev@cheol.me>
Co-authored-by: Muness Castle <931+muness@users.noreply.github.com>
Co-authored-by: Muness Castle <munesscastle@artium.ai>
Co-authored-by: zamo <falby97@proton.me>
Co-authored-by: elikoga <elikowa@gmail.com>
Co-authored-by: BayLee4 <63376748+BayLee4@users.noreply.github.com>
Co-authored-by: le-cameleon <200889489+le-cameleon@users.noreply.github.com>
Co-authored-by: Wiedzmin <56316383+art-wiedzmin@users.noreply.github.com>
This commit is contained in:
enieuwy
2026-03-29 23:50:36 +08:00
committed by GitHub
parent ccfd53c318
commit 323a3bc29c
8 changed files with 775 additions and 14 deletions
+21 -8
View File
@@ -885,16 +885,33 @@ theme:
dark: titanium
light: light
enabledModels:
- "anthropic/*"
- "*gpt*"
- "gemini-2.5-pro:high"
modelRoles:
default: anthropic/claude-sonnet-4-20250514
plan: anthropic/claude-opus-4-1:high
smol: anthropic/claude-sonnet-4-20250514
defaultThinkingLevel: high
enabledModels:
- anthropic/*
- "*gpt*"
- gemini-2.5-pro:high
retry:
enabled: true
# Number of retries before giving up on rate limits/server errors
maxRetries: 3
# Wait this long as a base (exponentially backed off) unless the API provides a retry-after-ms
baseDelayMs: 2000
# Configure role-specific model fallback chains
fallbackChains:
default:
- "openai/gpt-4o-mini"
- "openai/gpt-4o"
plan:
- "anthropic/claude-sonnet-4-6:high"
- "openai/o3:high"
# Whether to revert to the primary model when a fallback's cooldown expires
fallbackRevertPolicy: cooldown-expiry
steeringMode: one-at-a-time
followUpMode: one-at-a-time
interruptMode: immediate
@@ -914,10 +931,6 @@ compaction:
skills:
enabled: true
retry:
enabled: true
maxRetries: 3
baseDelayMs: 2000
terminal:
showImages: true
@@ -28,6 +28,7 @@ import {
import { isRecord, logger } from "@oh-my-pi/pi-utils";
import { type Static, Type } from "@sinclair/typebox";
import { type ConfigError, ConfigFile } from "../config";
import { parseModelString } from "../config/model-resolver";
import { isValidThemeColor, type ThemeColor } from "../modes/theme/theme";
import type { AuthStorage, OAuthCredential } from "../session/auth-storage";
import type { Settings } from "./settings";
@@ -721,6 +722,14 @@ function buildCustomModel(
return finalizeCustomModel(model, options);
}
function normalizeSuppressedSelector(selector: string): string {
const trimmed = selector.trim();
if (!trimmed) return trimmed;
const parsed = parseModelString(trimmed);
if (!parsed) return trimmed;
return `${parsed.provider}/${parsed.id}`;
}
/**
* Model registry - loads and manages models, resolves API keys via AuthStorage.
*/
@@ -737,6 +746,7 @@ export class ModelRegistry {
#registeredProviderSources: Set<string> = new Set();
#providerDiscoveryStates: Map<string, ProviderDiscoveryState> = new Map();
#cacheDbPath?: string;
#suppressedSelectors: Map<string, number> = new Map();
#backgroundRefresh?: Promise<void>;
#lastDiscoveryWarnings: Map<string, string> = new Map();
@@ -766,6 +776,7 @@ export class ModelRegistry {
*/
async refresh(strategy: ModelRefreshStrategy = "online-if-uncached"): Promise<void> {
this.#reloadStaticModels();
this.#suppressedSelectors.clear();
await this.#refreshRuntimeDiscoveries(strategy);
}
@@ -789,6 +800,11 @@ export class ModelRegistry {
async refreshProvider(providerId: string, strategy: ModelRefreshStrategy = "online"): Promise<void> {
this.#reloadStaticModels();
for (const selector of this.#suppressedSelectors.keys()) {
if (selector.startsWith(`${providerId}/`)) {
this.#suppressedSelectors.delete(selector);
}
}
await this.#refreshRuntimeDiscoveries(strategy, new Set([providerId]));
}
@@ -1825,6 +1841,27 @@ export class ModelRegistry {
});
}
}
/**
* Suppress a specific model selector (e.g., "provider/id") until a specific timestamp.
*/
suppressSelector(selector: string, untilMs: number): void {
this.#suppressedSelectors.set(normalizeSuppressedSelector(selector), untilMs);
}
/**
* Check if a model selector is currently suppressed due to rate limits.
*/
isSelectorSuppressed(selector: string): boolean {
const normalizedSelector = normalizeSuppressedSelector(selector);
const suppressedUntil = this.#suppressedSelectors.get(normalizedSelector);
if (!suppressedUntil) return false;
if (suppressedUntil <= Date.now()) {
this.#suppressedSelectors.delete(normalizedSelector);
return false;
}
return true;
}
}
/**
@@ -525,6 +525,18 @@ export const SETTINGS_SCHEMA = {
},
"retry.baseDelayMs": { type: "number", default: 2000 },
"retry.fallbackChains": { type: "record", default: {} as Record<string, string[]> },
"retry.fallbackRevertPolicy": {
type: "enum",
values: ["cooldown-expiry", "never"] as const,
default: "cooldown-expiry",
ui: {
tab: "model",
label: "Fallback Revert Policy",
description: "When to return to the primary model after a fallback",
submenu: true,
},
},
// ────────────────────────────────────────────────────────────────────────
// Interaction
@@ -117,6 +117,15 @@ const OPTION_PROVIDERS: Partial<Record<SettingPath, OptionProvider>> = {
{ value: "5", label: "5 retries" },
{ value: "10", label: "10 retries" },
],
// Retry fallback revert policy
"retry.fallbackRevertPolicy": [
{
value: "cooldown-expiry",
label: "Cooldown expiry",
description: "Return to the primary model after its suppression window ends",
},
{ value: "never", label: "Never", description: "Stay on the fallback model until manually changed" },
],
// Task max concurrency
"task.maxConcurrency": [
{ value: "0", label: "Unlimited" },
@@ -534,6 +534,16 @@ export class EventController {
break;
}
case "retry_fallback_applied": {
this.ctx.showWarning(`Fallback: ${event.from} -> ${event.to}`);
break;
}
case "retry_fallback_succeeded": {
this.ctx.showStatus(`Fallback succeeded on ${event.model}`);
break;
}
case "ttsr_triggered": {
const component = new TtsrNotificationComponent(event.rules);
component.setExpanded(this.ctx.toolOutputExpanded);
@@ -306,6 +306,11 @@ export class InteractiveMode implements InteractiveModeContext {
const startupQuiet = settings.get("startup.quiet");
for (const warning of this.session.configWarnings) {
this.ui.addChild(new Text(theme.fg("warning", `Warning: ${warning}`), 1, 0));
this.ui.addChild(new Spacer(1));
}
if (!startupQuiet) {
// Add welcome header
const welcome = new WelcomeComponent(this.#version, modelName, providerName, recentSessions, lspServerInfo);
@@ -55,7 +55,12 @@ import { abortableSleep, getAgentDbPath, isEnoent, logger } from "@oh-my-pi/pi-u
import type { AsyncJob, AsyncJobManager } from "../async";
import type { Rule } from "../capability/rule";
import { MODEL_ROLE_IDS, type ModelRegistry } from "../config/model-registry";
import { extractExplicitThinkingSelector, parseModelString, resolveModelRoleValue } from "../config/model-resolver";
import {
extractExplicitThinkingSelector,
formatModelString,
parseModelString,
resolveModelRoleValue,
} from "../config/model-resolver";
import { expandPromptTemplate, type PromptTemplate, renderPromptTemplate } from "../config/prompt-templates";
import type { Settings, SkillsSettings } from "../config/settings";
import { type BashResult, executeBash as executeBashCommand } from "../exec/bash-executor";
@@ -170,6 +175,8 @@ export type AgentSessionEvent =
}
| { type: "auto_retry_start"; attempt: number; maxAttempts: number; delayMs: number; errorMessage: string }
| { type: "auto_retry_end"; success: boolean; attempt: number; finalError?: string }
| { type: "retry_fallback_applied"; from: string; to: string; role: string }
| { type: "retry_fallback_succeeded"; model: string; role: string }
| { type: "ttsr_triggered"; rules: Rule[] }
| { type: "todo_reminder"; todos: TodoItem[]; attempt: number; maxAttempts: number }
| { type: "todo_auto_clear" };
@@ -315,6 +322,46 @@ interface HandoffOptions {
const AUTO_HANDOFF_THRESHOLD_FOCUS = renderPromptTemplate(autoHandoffThresholdFocusPrompt);
type RetryFallbackChains = Record<string, string[]>;
type RetryFallbackRevertPolicy = "never" | "cooldown-expiry";
interface RetryFallbackSelector {
raw: string;
provider: string;
id: string;
thinkingLevel: ThinkingLevel | undefined;
}
interface ActiveRetryFallbackState {
role: string;
originalSelector: string;
originalThinkingLevel: ThinkingLevel | undefined;
lastAppliedFallbackThinkingLevel: ThinkingLevel | undefined;
}
function parseRetryFallbackSelector(selector: string): RetryFallbackSelector | undefined {
const trimmed = selector.trim();
if (!trimmed) return undefined;
const parsed = parseModelString(trimmed);
if (!parsed) return undefined;
return {
raw: trimmed,
provider: parsed.provider,
id: parsed.id,
thinkingLevel: parsed.thinkingLevel,
};
}
function formatRetryFallbackSelector(model: Model, thinkingLevel: ThinkingLevel | undefined): string {
const selector = formatModelString(model);
return thinkingLevel ? `${selector}:${thinkingLevel}` : selector;
}
function formatRetryFallbackBaseSelector(selector: RetryFallbackSelector): string {
return `${selector.provider}/${selector.id}`;
}
const noOpUIContext: ExtensionUIContext = {
select: async (_title, _options, _dialogOptions) => undefined,
confirm: async (_title, _message, _dialogOptions) => false,
@@ -352,6 +399,7 @@ export class AgentSession {
readonly sessionManager: SessionManager;
readonly settings: Settings;
readonly searchDb: SearchDb | undefined;
readonly configWarnings: string[] = [];
#asyncJobManager: AsyncJobManager | undefined = undefined;
#scopedModels: Array<{ model: Model; thinkingLevel?: ThinkingLevel }>;
@@ -391,7 +439,7 @@ export class AgentSession {
#retryAttempt = 0;
#retryPromise: Promise<void> | undefined = undefined;
#retryResolve: (() => void) | undefined = undefined;
#activeRetryFallback: ActiveRetryFallbackState | undefined = undefined;
// Todo completion reminder state
#todoReminderCount = 0;
#todoPhases: TodoPhase[] = [];
@@ -478,6 +526,7 @@ export class AgentSession {
this.#customCommands = config.customCommands ?? [];
this.#skillsSettings = config.skillsSettings;
this.#modelRegistry = config.modelRegistry;
this.#validateRetryFallbackChains();
this.#toolRegistry = config.toolRegistry ?? new Map();
this.#transformContext = config.transformContext ?? (messages => messages);
this.#onPayload = config.onPayload;
@@ -787,6 +836,13 @@ export class AgentSession {
assistantMsg.stopReason !== "aborted" &&
this.#retryAttempt > 0
) {
if (this.#activeRetryFallback && this.model) {
await this.#emitSessionEvent({
type: "retry_fallback_succeeded",
model: formatRetryFallbackSelector(this.model, this.thinkingLevel),
role: this.#activeRetryFallback.role,
});
}
await this.#emitSessionEvent({
type: "auto_retry_end",
success: true,
@@ -985,6 +1041,7 @@ export class AgentSession {
return;
}
try {
await this.#maybeRestoreRetryFallbackPrimary();
await this.agent.continue();
} catch {
options?.onError?.();
@@ -2288,6 +2345,8 @@ export class AgentSession {
// Reset todo reminder count on new user prompt
this.#todoReminderCount = 0;
await this.#maybeRestoreRetryFallbackPrimary();
// Validate model
if (!this.model) {
throw new Error(
@@ -3108,6 +3167,7 @@ export class AgentSession {
throw new Error(`No API key for ${model.provider}/${model.id}`);
}
this.#clearActiveRetryFallback();
this.#setModelWithProviderSessionReset(model);
this.sessionManager.appendModelChange(`${model.provider}/${model.id}`, role);
this.settings.setModelRole(role, this.#formatRoleModelValue(role, model));
@@ -3128,6 +3188,7 @@ export class AgentSession {
throw new Error(`No API key for ${model.provider}/${model.id}`);
}
this.#clearActiveRetryFallback();
this.#setModelWithProviderSessionReset(model);
this.sessionManager.appendModelChange(`${model.provider}/${model.id}`, "temporary");
this.settings.getStorage()?.recordModelUsage(`${model.provider}/${model.id}`);
@@ -3253,6 +3314,7 @@ export class AgentSession {
const next = scopedModels[nextIndex];
// Apply model
this.#clearActiveRetryFallback();
this.#setModelWithProviderSessionReset(next.model);
this.sessionManager.appendModelChange(`${next.model.provider}/${next.model.id}`);
this.settings.setModelRole("default", this.#formatRoleModelValue("default", next.model));
@@ -3281,11 +3343,11 @@ export class AgentSession {
throw new Error(`No API key for ${nextModel.provider}/${nextModel.id}`);
}
this.#clearActiveRetryFallback();
this.#setModelWithProviderSessionReset(nextModel);
this.sessionManager.appendModelChange(`${nextModel.provider}/${nextModel.id}`);
this.settings.setModelRole("default", this.#formatRoleModelValue("default", nextModel));
this.settings.getStorage()?.recordModelUsage(`${nextModel.provider}/${nextModel.id}`);
// Re-apply the current thinking level for the newly selected model
this.setThinkingLevel(this.thinkingLevel);
@@ -4768,6 +4830,217 @@ export class AgentSession {
);
}
#getRetryFallbackChains(): RetryFallbackChains {
const configuredChains = this.settings.get("retry.fallbackChains");
if (!configuredChains || typeof configuredChains !== "object") return {};
return configuredChains as RetryFallbackChains;
}
#validateRetryFallbackChains(): void {
const configuredChains = this.settings.get("retry.fallbackChains");
if (configuredChains === undefined) return;
if (!configuredChains || typeof configuredChains !== "object" || Array.isArray(configuredChains)) {
const msg = "retry.fallbackChains must be a mapping of role names to selector arrays.";
logger.warn(msg);
this.configWarnings.push(msg);
return;
}
for (const [role, chain] of Object.entries(configuredChains)) {
if (!Array.isArray(chain)) {
const msg = `Fallback chain for role '${role}' must be an array of selector strings.`;
logger.warn(msg);
this.configWarnings.push(msg);
continue;
}
for (const selectorStr of chain) {
if (typeof selectorStr !== "string") {
const msg = `Fallback chain for role '${role}' contains a non-string selector.`;
logger.warn(msg);
this.configWarnings.push(msg);
continue;
}
const parsed = parseRetryFallbackSelector(selectorStr);
if (!parsed) {
const msg = `Invalid fallback selector format in role '${role}': ${selectorStr}`;
logger.warn(msg);
this.configWarnings.push(msg);
continue;
}
const exists = this.#modelRegistry.find(parsed.provider, parsed.id);
if (!exists) {
const msg = `Fallback chain for role '${role}' references unknown model: ${selectorStr}`;
logger.warn(msg);
this.configWarnings.push(msg);
}
}
}
}
#getRetryFallbackRevertPolicy(): RetryFallbackRevertPolicy {
return this.settings.get("retry.fallbackRevertPolicy") === "never" ? "never" : "cooldown-expiry";
}
#getRetryFallbackPrimarySelector(role: string): RetryFallbackSelector | undefined {
const configuredSelector = this.settings.getModelRole(role);
return configuredSelector ? parseRetryFallbackSelector(configuredSelector) : undefined;
}
#clearActiveRetryFallback(): void {
this.#activeRetryFallback = undefined;
}
#isRetryFallbackSelectorSuppressed(selector: RetryFallbackSelector): boolean {
return this.#modelRegistry.isSelectorSuppressed(selector.raw);
}
#noteRetryFallbackCooldown(currentSelector: string, retryAfterMs: number | undefined, errorMessage: string): void {
let cooldownMs = retryAfterMs;
if (!cooldownMs || cooldownMs <= 0) {
const reason = parseRateLimitReason(errorMessage);
cooldownMs = reason === "UNKNOWN" ? 5 * 60 * 1000 : calculateRateLimitBackoffMs(reason);
}
this.#modelRegistry.suppressSelector(currentSelector, Date.now() + cooldownMs);
}
#resolveRetryFallbackRole(currentSelector: string): string | undefined {
const parsedCurrent = parseRetryFallbackSelector(currentSelector);
if (!parsedCurrent) return undefined;
const currentBaseSelector = formatRetryFallbackBaseSelector(parsedCurrent);
for (const role of Object.keys(this.#getRetryFallbackChains())) {
const primarySelector = this.#getRetryFallbackPrimarySelector(role);
if (!primarySelector) continue;
if (primarySelector.raw === currentSelector) return role;
if (formatRetryFallbackBaseSelector(primarySelector) === currentBaseSelector) return role;
}
return undefined;
}
#getRetryFallbackEffectiveChain(role: string): RetryFallbackSelector[] {
const primarySelector = this.#getRetryFallbackPrimarySelector(role);
if (!primarySelector) return [];
const chain = [primarySelector];
const seen = new Set<string>([primarySelector.raw]);
for (const selector of this.#getRetryFallbackChains()[role] ?? []) {
const parsed = parseRetryFallbackSelector(selector);
if (!parsed || seen.has(parsed.raw)) continue;
seen.add(parsed.raw);
chain.push(parsed);
}
return chain;
}
#findRetryFallbackCandidates(role: string, currentSelector: string): RetryFallbackSelector[] {
const chain = this.#getRetryFallbackEffectiveChain(role);
if (chain.length <= 1) return [];
const parsedCurrent = parseRetryFallbackSelector(currentSelector);
const currentBaseSelector = parsedCurrent ? formatRetryFallbackBaseSelector(parsedCurrent) : undefined;
const exactIndex = chain.findIndex(selector => selector.raw === currentSelector);
if (exactIndex >= 0) return chain.slice(exactIndex + 1);
const baseIndex = currentBaseSelector
? chain.findIndex(selector => formatRetryFallbackBaseSelector(selector) === currentBaseSelector)
: -1;
if (baseIndex >= 0) return chain.slice(baseIndex + 1);
return chain.slice(1);
}
async #applyRetryFallbackCandidate(
role: string,
selector: RetryFallbackSelector,
currentSelector: string,
): Promise<void> {
const candidate = this.#modelRegistry.find(selector.provider, selector.id);
if (!candidate) {
throw new Error(`Retry fallback model not found: ${selector.raw}`);
}
const apiKey = await this.#modelRegistry.getApiKey(candidate, this.sessionId);
if (!apiKey) {
throw new Error(`No API key for retry fallback ${selector.raw}`);
}
const currentThinkingLevel = this.thinkingLevel;
const nextThinkingLevel = selector.thinkingLevel ?? currentThinkingLevel;
this.#setModelWithProviderSessionReset(candidate);
this.sessionManager.appendModelChange(`${candidate.provider}/${candidate.id}`, "temporary");
this.settings.getStorage()?.recordModelUsage(`${candidate.provider}/${candidate.id}`);
this.setThinkingLevel(nextThinkingLevel);
if (!this.#activeRetryFallback) {
this.#activeRetryFallback = {
role,
originalSelector: currentSelector,
originalThinkingLevel: currentThinkingLevel,
lastAppliedFallbackThinkingLevel: nextThinkingLevel,
};
} else {
this.#activeRetryFallback.lastAppliedFallbackThinkingLevel = nextThinkingLevel;
}
await this.#emitSessionEvent({
type: "retry_fallback_applied",
from: currentSelector,
to: selector.raw,
role,
});
}
async #tryRetryModelFallback(currentSelector: string): Promise<boolean> {
const role = this.#activeRetryFallback?.role ?? this.#resolveRetryFallbackRole(currentSelector);
if (!role) return false;
for (const selector of this.#findRetryFallbackCandidates(role, currentSelector)) {
if (this.#isRetryFallbackSelectorSuppressed(selector)) continue;
const candidate = this.#modelRegistry.find(selector.provider, selector.id);
if (!candidate) continue;
const apiKey = await this.#modelRegistry.getApiKey(candidate, this.sessionId);
if (!apiKey) continue;
await this.#applyRetryFallbackCandidate(role, selector, currentSelector);
return true;
}
return false;
}
async #maybeRestoreRetryFallbackPrimary(): Promise<void> {
if (!this.#activeRetryFallback) return;
if (this.#getRetryFallbackRevertPolicy() !== "cooldown-expiry") return;
const {
originalSelector: originalSelectorRaw,
originalThinkingLevel,
lastAppliedFallbackThinkingLevel,
} = this.#activeRetryFallback;
const originalSelector = parseRetryFallbackSelector(originalSelectorRaw);
if (!originalSelector) {
this.#clearActiveRetryFallback();
return;
}
const currentModel = this.model;
if (!currentModel) return;
const currentSelector = formatRetryFallbackSelector(currentModel, this.thinkingLevel);
if (currentSelector === originalSelector.raw) {
if (!this.#isRetryFallbackSelectorSuppressed(originalSelector)) {
this.#clearActiveRetryFallback();
}
return;
}
if (this.#isRetryFallbackSelectorSuppressed(originalSelector)) return;
const primaryModel = this.#modelRegistry.find(originalSelector.provider, originalSelector.id);
if (!primaryModel) return;
const apiKey = await this.#modelRegistry.getApiKey(primaryModel, this.sessionId);
if (!apiKey) return;
const currentThinkingLevel = this.thinkingLevel;
const thinkingToApply =
currentThinkingLevel === lastAppliedFallbackThinkingLevel ? originalThinkingLevel : currentThinkingLevel;
this.#setModelWithProviderSessionReset(primaryModel);
this.sessionManager.appendModelChange(`${primaryModel.provider}/${primaryModel.id}`, "temporary");
this.settings.getStorage()?.recordModelUsage(`${primaryModel.provider}/${primaryModel.id}`);
this.setThinkingLevel(thinkingToApply);
this.#clearActiveRetryFallback();
}
#parseRetryAfterMsFromError(errorMessage: string): number | undefined {
const now = Date.now();
const retryAfterMsMatch = /retry-after-ms\s*[:=]\s*(\d+)/i.exec(errorMessage);
@@ -4847,12 +5120,13 @@ export class AgentSession {
}
const errorMessage = message.errorMessage || "Unknown error";
const parsedRetryAfterMs = this.#parseRetryAfterMsFromError(errorMessage);
let delayMs = retrySettings.baseDelayMs * 2 ** (this.#retryAttempt - 1);
let switchedCredential = false;
let switchedModel = false;
if (this.model && isUsageLimitError(errorMessage)) {
const retryAfterMs =
this.#parseRetryAfterMsFromError(errorMessage) ??
calculateRateLimitBackoffMs(parseRateLimitReason(errorMessage));
const retryAfterMs = parsedRetryAfterMs ?? calculateRateLimitBackoffMs(parseRateLimitReason(errorMessage));
const switched = await this.#modelRegistry.authStorage.markUsageLimitReached(
this.model.provider,
this.sessionId,
@@ -4862,6 +5136,7 @@ export class AgentSession {
},
);
if (switched) {
switchedCredential = true;
delayMs = 0;
} else if (retryAfterMs > delayMs) {
// No more accounts to switch to — wait out the backoff
@@ -4869,6 +5144,17 @@ export class AgentSession {
}
}
const currentSelector = this.model ? formatRetryFallbackSelector(this.model, this.thinkingLevel) : undefined;
if (!switchedCredential && currentSelector) {
this.#noteRetryFallbackCooldown(currentSelector, parsedRetryAfterMs, errorMessage);
switchedModel = await this.#tryRetryModelFallback(currentSelector);
if (switchedModel) {
delayMs = 0;
} else if (parsedRetryAfterMs && parsedRetryAfterMs > delayMs) {
delayMs = parsedRetryAfterMs;
}
}
await this.#emitSessionEvent({
type: "auto_retry_start",
attempt: this.#retryAttempt,
@@ -0,0 +1,389 @@
import { afterEach, beforeEach, describe, expect, it } from "bun:test";
import * as path from "node:path";
import { Agent } from "@oh-my-pi/pi-agent-core";
import { type AssistantMessage, Effort, getBundledModel, type Model } from "@oh-my-pi/pi-ai";
import { AssistantMessageEventStream } from "@oh-my-pi/pi-ai/utils/event-stream";
import { ModelRegistry } from "@oh-my-pi/pi-coding-agent/config/model-registry";
import { Settings } from "@oh-my-pi/pi-coding-agent/config/settings";
import { AgentSession, type AgentSessionEvent } from "@oh-my-pi/pi-coding-agent/session/agent-session";
import { AuthStorage } from "@oh-my-pi/pi-coding-agent/session/auth-storage";
import { SessionManager } from "@oh-my-pi/pi-coding-agent/session/session-manager";
import { TempDir } from "@oh-my-pi/pi-utils";
class MockAssistantStream extends AssistantMessageEventStream {}
function createAssistantMessage(
model: Model,
options: { text?: string; stopReason: "stop" | "error"; errorMessage?: string },
): AssistantMessage {
return {
role: "assistant",
content: options.text ? [{ type: "text", text: options.text }] : [],
api: model.api,
provider: model.provider,
model: model.id,
usage: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 0,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: options.stopReason,
errorMessage: options.errorMessage,
timestamp: Date.now(),
};
}
async function _waitFor(predicate: () => boolean, timeoutMs = 1000): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (predicate()) return;
await Bun.sleep(10);
}
throw new Error("Timed out waiting for condition");
}
describe("AgentSession retry fallback", () => {
let tempDir: TempDir;
let authStorage: AuthStorage;
let modelRegistry: ModelRegistry;
let session: AgentSession | undefined;
beforeEach(async () => {
tempDir = TempDir.createSync("@pi-retry-fallback-");
authStorage = await AuthStorage.create(path.join(tempDir.path(), "testauth.db"));
authStorage.setRuntimeApiKey("anthropic", "anthropic-test-key");
authStorage.setRuntimeApiKey("openai", "openai-test-key");
modelRegistry = new ModelRegistry(authStorage);
});
afterEach(async () => {
if (session) {
await session.dispose();
session = undefined;
}
authStorage.close();
tempDir.removeSync();
});
it("advances through a role-keyed fallback chain across retries", async () => {
const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5");
const firstFallback = getBundledModel("openai", "gpt-4o-mini");
const secondFallback = getBundledModel("openai", "gpt-4o");
if (!primaryModel || !firstFallback || !secondFallback) {
throw new Error("Expected bundled test models to exist");
}
const requestedModels: string[] = [];
const retryStartEvents: Array<Extract<AgentSessionEvent, { type: "auto_retry_start" }>> = [];
const retryEndEvents: Array<Extract<AgentSessionEvent, { type: "auto_retry_end" }>> = [];
const fallbackAppliedEvents: Array<Extract<AgentSessionEvent, { type: "retry_fallback_applied" }>> = [];
const fallbackSucceededEvents: Array<Extract<AgentSessionEvent, { type: "retry_fallback_succeeded" }>> = [];
const agent = new Agent({
getApiKey: provider => `${provider}-test-key`,
initialState: {
model: primaryModel,
systemPrompt: "Test",
tools: [],
messages: [],
},
streamFn: model => {
requestedModels.push(`${model.provider}/${model.id}`);
const stream = new MockAssistantStream();
queueMicrotask(() => {
if (model.provider === primaryModel.provider && model.id === primaryModel.id) {
const message = createAssistantMessage(model, {
stopReason: "error",
errorMessage: "overloaded_error: provider returned error 503",
});
stream.push({ type: "start", partial: message });
stream.push({ type: "error", reason: "error", error: message });
return;
}
if (model.provider === firstFallback.provider && model.id === firstFallback.id) {
const message = createAssistantMessage(model, {
stopReason: "error",
errorMessage: "service unavailable: 503 overloaded",
});
stream.push({ type: "start", partial: message });
stream.push({ type: "error", reason: "error", error: message });
return;
}
if (model.provider === secondFallback.provider && model.id === secondFallback.id) {
const message = createAssistantMessage(model, {
text: "Recovered on second fallback",
stopReason: "stop",
});
stream.push({
type: "start",
partial: createAssistantMessage(model, { text: "", stopReason: "stop" }),
});
stream.push({ type: "done", reason: "stop", message });
return;
}
throw new Error(`Unexpected model requested during retry fallback test: ${model.provider}/${model.id}`);
});
return stream;
},
});
const settings = Settings.isolated({
"compaction.enabled": false,
"retry.baseDelayMs": 5,
"retry.fallbackChains": {
default: [
`${firstFallback.provider}/${firstFallback.id}`,
`${secondFallback.provider}/${secondFallback.id}`,
],
},
});
settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}`);
session = new AgentSession({
agent,
sessionManager: SessionManager.inMemory(),
settings,
modelRegistry,
});
session.subscribe(event => {
if (event.type === "auto_retry_start") {
retryStartEvents.push(event);
}
if (event.type === "auto_retry_end") {
retryEndEvents.push(event);
}
if (event.type === "retry_fallback_applied") {
fallbackAppliedEvents.push(event);
}
if (event.type === "retry_fallback_succeeded") {
fallbackSucceededEvents.push(event);
}
});
await session.prompt("Recover from rate limits");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${firstFallback.provider}/${firstFallback.id}`,
`${secondFallback.provider}/${secondFallback.id}`,
]);
expect(session.model?.provider).toBe(secondFallback.provider);
expect(session.model?.id).toBe(secondFallback.id);
expect(retryStartEvents.map(event => event.delayMs)).toEqual([0, 0]);
expect(fallbackAppliedEvents).toEqual([
{
type: "retry_fallback_applied",
from: `${primaryModel.provider}/${primaryModel.id}`,
to: `${firstFallback.provider}/${firstFallback.id}`,
role: "default",
},
{
type: "retry_fallback_applied",
from: `${firstFallback.provider}/${firstFallback.id}`,
to: `${secondFallback.provider}/${secondFallback.id}`,
role: "default",
},
]);
expect(retryEndEvents).toHaveLength(1);
expect(retryEndEvents[0]).toMatchObject({ success: true, attempt: 2 });
expect(fallbackSucceededEvents).toEqual([
{
type: "retry_fallback_succeeded",
model: `${secondFallback.provider}/${secondFallback.id}`,
role: "default",
},
]);
});
it("suppresses cooled selectors and lazily reverts to the role primary after cooldown expiry", async () => {
const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5");
const fallbackModel = getBundledModel("openai", "gpt-4o-mini");
if (!primaryModel || !fallbackModel) {
throw new Error("Expected bundled test models to exist");
}
const requestedModels: string[] = [];
let primaryAttempts = 0;
const agent = new Agent({
getApiKey: provider => `${provider}-test-key`,
initialState: {
model: primaryModel,
systemPrompt: "Test",
tools: [],
messages: [],
},
streamFn: model => {
requestedModels.push(`${model.provider}/${model.id}`);
const stream = new MockAssistantStream();
queueMicrotask(() => {
if (model.provider === primaryModel.provider && model.id === primaryModel.id && primaryAttempts === 0) {
primaryAttempts += 1;
const message = createAssistantMessage(model, {
stopReason: "error",
errorMessage: "rate limit exceeded retry-after-ms=200",
});
stream.push({ type: "start", partial: message });
stream.push({ type: "error", reason: "error", error: message });
return;
}
const message = createAssistantMessage(model, {
text: `ok:${model.provider}/${model.id}`,
stopReason: "stop",
});
stream.push({ type: "start", partial: createAssistantMessage(model, { text: "", stopReason: "stop" }) });
stream.push({ type: "done", reason: "stop", message });
});
return stream;
},
});
const settings = Settings.isolated({
"compaction.enabled": false,
"retry.baseDelayMs": 5,
"retry.fallbackChains": {
default: [`${fallbackModel.provider}/${fallbackModel.id}`],
},
"retry.fallbackRevertPolicy": "cooldown-expiry",
});
settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}`);
session = new AgentSession({
agent,
sessionManager: SessionManager.inMemory(),
settings,
modelRegistry,
});
await session.prompt("First prompt triggers fallback");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
]);
expect(session.model?.provider).toBe(fallbackModel.provider);
expect(session.model?.id).toBe(fallbackModel.id);
await session.prompt("Immediate second prompt should stay on fallback");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
]);
expect(session.model?.provider).toBe(fallbackModel.provider);
expect(session.model?.id).toBe(fallbackModel.id);
await Bun.sleep(240);
await session.prompt("Third prompt should lazily revert to primary");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
`${primaryModel.provider}/${primaryModel.id}`,
]);
expect(session.model?.provider).toBe(primaryModel.provider);
expect(session.model?.id).toBe(primaryModel.id);
});
it("preserves thinking on bare fallback selectors and does not overwrite user thinking on restore", async () => {
const primaryModel = getBundledModel("anthropic", "claude-sonnet-4-5");
const fallbackModel = getBundledModel("openai", "gpt-4o-mini");
if (!primaryModel || !fallbackModel) {
throw new Error("Expected bundled test models to exist");
}
const requestedModels: string[] = [];
let primaryAttempts = 0;
const agent = new Agent({
getApiKey: provider => `${provider}-test-key`,
initialState: {
model: primaryModel,
systemPrompt: "Test",
tools: [],
messages: [],
},
streamFn: model => {
requestedModels.push(`${model.provider}/${model.id}`);
const stream = new MockAssistantStream();
queueMicrotask(() => {
if (model.provider === primaryModel.provider && model.id === primaryModel.id && primaryAttempts === 0) {
primaryAttempts += 1;
const message = createAssistantMessage(model, {
stopReason: "error",
errorMessage: "rate limit exceeded retry-after-ms=200",
});
stream.push({ type: "start", partial: message });
stream.push({ type: "error", reason: "error", error: message });
return;
}
const message = createAssistantMessage(model, {
text: `ok:${model.provider}/${model.id}`,
stopReason: "stop",
});
stream.push({ type: "start", partial: createAssistantMessage(model, { text: "", stopReason: "stop" }) });
stream.push({ type: "done", reason: "stop", message });
});
return stream;
},
});
const settings = Settings.isolated({
"compaction.enabled": false,
"retry.baseDelayMs": 5,
"retry.fallbackChains": {
default: [`${fallbackModel.provider}/${fallbackModel.id}`],
},
"retry.fallbackRevertPolicy": "cooldown-expiry",
});
settings.setModelRole("default", `${primaryModel.provider}/${primaryModel.id}:high`);
session = new AgentSession({
agent,
sessionManager: SessionManager.inMemory(),
settings,
modelRegistry,
thinkingLevel: Effort.High,
});
await session.prompt("First prompt triggers bare-selector fallback");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
]);
expect(session.model?.provider).toBe(fallbackModel.provider);
expect(session.model?.id).toBe(fallbackModel.id);
expect(session.thinkingLevel).toBeUndefined();
session.setThinkingLevel(Effort.Low);
await Bun.sleep(240);
await session.prompt("Second prompt should restore model but preserve user thinking change");
await session.waitForIdle();
expect(requestedModels).toEqual([
`${primaryModel.provider}/${primaryModel.id}`,
`${fallbackModel.provider}/${fallbackModel.id}`,
`${primaryModel.provider}/${primaryModel.id}`,
]);
expect(session.model?.provider).toBe(primaryModel.provider);
expect(session.model?.id).toBe(primaryModel.id);
expect(session.thinkingLevel).toBeUndefined();
});
it("normalizes suppression by base selector and clears it on model refresh", async () => {
const future = Date.now() + 60_000;
modelRegistry.suppressSelector("openai/gpt-4o:high", future);
expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o")).toBe(true);
expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o:low")).toBe(true);
await modelRegistry.refresh("offline");
expect(modelRegistry.isSelectorSuppressed("openai/gpt-4o")).toBe(false);
});
});