Keep ordinary 401 retries bounded while replay-safe quota failures
walk every distinct eligible credential. Anchor blocks to the failed
credential and stop on cycles, aborts, or 64 attempts.
Migrate 203 test files (356 call sites) from fs.rm/fs.rmSync to
removeWithRetries/removeSyncWithRetries to reduce EBUSY test failures
on Windows. removeWithRetries is now exported from @oh-my-pi/pi-utils.
The migration uses a regex-based approach that:
- Replaces fs.rm(path, { recursive, force }) → removeWithRetries(path)
- Replaces fs.rmSync(path, { recursive, force }) → removeSyncWithRetries(path)
- Replaces fs.rm(path) → removeWithRetries(path) (no options)
- Skips fs.rm/fs.rmSync inside template literals (bun --eval scripts)
- Adds imports to existing @oh-my-pi/pi-utils import or creates new one
- Removes unused fs imports where fs.rm was the only fs usage (4 files)
- Added auth-storage regression coverage in `packages/ai` for `markUsageLimitReached` returning the earliest sibling unblock time when all credentials are momentarily blocked.
- Added an agent-session retry-cap test ensuring usage-limit 429 retries wait for sibling unblock and succeed within `retry.maxDelayMs` instead of giving up.
- Updated retry-policy docs and both package changelogs to document the sibling-unblock fallback behavior.
- Derived descriptors, default-model map, env keys, login list, and refresh dispatch from one ProviderDefinition per provider.
- Disabled OpenAI Codex stream obfuscation and interrupted whitespace-only tool-call argument deltas.
- Derived auth-broker callback ports and paste-code login set from the registry.
Centralizing OAuth refresh in AuthStorage (e6893515) introduced five
follow-on bugs surfaced by an audit of the commit; this fixes all of
them and updates the tests that relied on the old refresh seam.
1. packages/ai/src/auth-storage.ts (#tryOAuthCredential):
For built-in providers the path went directly to `getOAuthApiKey`
with the (possibly still-expired) selection.credential when the
pre-refresh at line 2587 caught a transient error. `getOAuthApiKey`
then threw the "expired … must be refreshed via AuthStorage"
precondition error, which the disable classifier matched against
`/expired.*refresh/` and soft-disabled the row. A single network
blip during refresh could permanently kill a still-valid Anthropic /
OpenAI / Gemini-CLI / Copilot credential. Built-in providers now
route through the broker-aware single-flighted
`#refreshOAuthCredential` first, so transient failures surface as
network errors (5-min temp block) instead of definitive auth
failures.
2. packages/ai/src/auth-storage.ts (#fetchUsageUncached):
The usage refresh check only fired once `Date.now() >= expiresAt`,
missing the 60-second skew that `getApiKey` honors. A token
expiring inside the skew window was posted to the usage endpoint
and 401'd mid-flight, briefly hiding quota in the UI. Aligned with
`OAUTH_REFRESH_SKEW_MS`.
3. packages/coding-agent/src/web/search/index.ts (webSearchCustomTool):
The CustomTool counterpart of WebSearchTool dropped sessionId so
SDK callers that opted into `web_search` via toolNames lost
per-session credential stickiness — multi-account users saw the
provider round-robin between searches in the same session. Threads
`ctx.sessionManager.getSessionId()` through to `executeSearch`.
4. packages/coding-agent/src/web/search/providers/perplexity.ts
(findOAuthToken):
`authStorage.getApiKey("perplexity")` returns runtime/config
overrides, stored api_key credentials, OAuth bearers, and env keys.
Filtering only env keys meant a config-pinned `pplx-…` API key was
POSTed to `www.perplexity.ai/rest/sse/perplexity_ask` (the OAuth
endpoint) instead of falling through to
`api.perplexity.ai/chat/completions`, producing 401s. Switched to
`getOAuthAccess` so only true OAuth bearers reach the OAuth
branch; api_key credentials/overrides correctly fall through.
5. packages/ai/scripts/generate-models.ts:
`getOAuthApiKey` was being called directly with possibly-expired
credentials. The new contract throws on expired, the broad catch
swallowed it, and the build silently fell back to bundled models
instead of refreshing. Both helpers now route through
AuthStorage's `getApiKey` / `getOAuthAccess`, which trigger the
full broker-aware refresh pipeline.
Test updates:
- auth-storage-credential-disabled-event.test.ts,
sdk-credential-disabled-bridge.test.ts: the `failOAuthRefresh`
helper used to spy on `getOAuthApiKey` to inject invalid_grant.
With refresh now happening before that helper, the spy never fired.
Switched to spying on `refreshOAuthToken` so the simulated failure
reaches the disable classifier.
- auth-storage-rotation.test.ts: stub `refreshOAuthToken` so the test
doesn't hit a real OAuth endpoint when the seeded credential lands
inside the 60s skew window.
- Documented and removed `utils/oauth` from the `ai` package entrypoint, noting it as a breaking change.
- Refactored `cli`, `auth-storage`, and `utils/oauth` to load provider modules via scoped dynamic `import()` calls.
- Removed top-level provider imports and barrel exports from `utils/oauth/index.ts`, streamlining oauth module loading.
- Consolidated OAuth symbol, type, and provider imports in coding-agent and tests to `@oh-my-pi/pi-ai/utils/oauth` modules.
- Defined `DEFAULT_LOCAL_TOKEN` locally in model-registry and removed its cross-package OAuth import usage.
- Added `close()` method to SessionManager and AuthStorage for proper resource cleanup and finalization of prepared statements.
- Added `initiatorOverride` option support in OpenAI and Anthropic providers for message attribution control.
- Fixed resource leaks in RpcClient timeout handling by centralizing timeout creation with unref() and adding explicit clearTimeout() calls.
- Fixed AgentSession disposal to call SessionManager's `close()` method for guaranteed resource cleanup instead of fallback flush.
- Updated all test suites to properly dispose AuthStorage instances in cleanup hooks to prevent resource leaks between tests.
can1357