Treated an explicit allowed=true and limit_reached=false verdict as authoritative when Codex rounds usage to 100 percent. AuthStorage now honors normalized non-exhausted statuses before numeric fallbacks, keeping a usable Team credential ahead of rejected siblings.
Fixes#7617
- The broker heals its store synchronously while serving /v1/usage, but the
remote client snapshot only converges via the background long-poll; the
assertion raced that poll and flaked on slow CI runners.
- Pull one explicit snapshot after fetchUsageReports, matching the sibling
same-deadline re-upsert test.
Codex reports one account-level limit flag alongside independent chat and Spark meters. Treating that flag, ranking windows, or persisted backoff as provider-wide made an exhausted meter block requests that still had headroom.
Scope exhaustion, ranking, and reactive backoff to the meter each request spends. Every selection path honors the request scope plus the legacy shared scope, including sibling availability, session pinning, and final OAuth resolution.
Reconciliation now evaluates and deletes each persisted meter block independently. The local and broker-backed stores support targeted deletion while the existing all-block deletion endpoint remains compatible.
Signed-off-by: Christian Stewart <christian@aperture.us>
The one-hour warm window comes from Anthropic's verified prompt-cache
retention. Applying it generically could re-rank Codex credentials while its
24-hour prompt cache remains warm, and other providers have no verified TTL.
- Applied the idle staleness gate only to Anthropic.
- Preserved indefinite stickiness for providers without a verified boundary.
- Added a Codex regression proving the pin survives more than one hour idle.
Fixes#5966
Post-TTL prompt-cache entries are deleted promptly but not instantaneously, so
"expired" overstated the guarantee. Wording-only change across the comment,
regression test, and changelog; no behavior change.
Fixes#5966
Session credential stickiness pinned a session to its last-used Anthropic
credential and skipped usage-based re-ranking whenever the pin was available,
using purely structural conditions (exists / refreshable / unblocked). The pin
carried no timestamp, so ranking was suppressed for up to 30 days even after the
=<1h prompt cache it protects had certainly expired, silently degrading
multi-account load balancing across long idle gaps.
- Store lastUsedAtMs in the pin (in-memory map + persisted JSON row).
- Gate the shouldRank short-circuit on staleness against a 1h warm window
(SESSION_STICKY_CACHE_WARM_MS); idle-past-TTL sessions rank again.
- When ranking runs, seed the pinned credential first in the evaluation order
as a tie-break instead of an absolute front-of-queue hoist, so a clearly
better sibling wins while ties still keep the same account.
Fixes#5966
- Added a Codex rate-limit parser for `x-codex-*` headers and registered it with the Codex usage provider.
- Updated auth-storage to require parsed usage headers before ingesting usage data, treated exhaustion as non-throttled, and renamed ranked candidate drain fields.
- Reworked key ranking math to use `headroom / remainingHours` with a 1-minute minimum, then applied measured-usage precedence with hot-window demotion behavior.
- Updated session handling and tests to support provider-aware header ingestion with deterministic, exhaustion-aware account selection.
Sol/Luna set a plan requirement, so #resolveOAuthSelection re-ranked OAuth
candidates every request even when the session-preferred credential was still
usable. #orderRankedOAuthCandidates mixes the stable session hash with
usage-derived weights, so when 5h/7d headroom flipped between two eligible Plus
accounts the same session hash landed on the sibling and #recordSessionCredential
overwrote the sticky binding — surfacing as /usage alternating accounts.
Promote the session-preferred candidate back to the front after ranking whenever
it is unblocked and (for plan-gated models) still plan-eligible, so a session
stays sticky unless the sticky account is blocked, exhausted, or ineligible.
Fixes#5203
- Implemented model-specific keys and provider wildcards for `retry.fallbackChains` with updated resolution logic.
- Added interactive fallback chain management in the model roles UI, including support for reordering and editing.
- Improved fallback chain specificity rules and added comprehensive validation with startup warnings.
- Fixed mouse interaction alignment and hover state coordinate mapping in the roles view.
- Implement a multi-pass resolution ladder to handle scenarios where plan-eligible accounts are usage-blocked.
- Ensure that exhausted paid accounts are prioritized over unblocked free accounts for plan-gated models, allowing callers to receive proper usage-limit error semantics from the API provider.
- Add a test case to verify credential selection when all plan-eligible accounts are currently rate-limited.
- Sent credential-block updatedAtMs through broker snapshots so same-deadline block refreshes are observable by remote clients.
- Refreshed RemoteAuthCredentialStore reconciliation guards when updatedAtMs changes even if blockedUntilMs is unchanged.
- Added broker regression coverage for same-deadline Codex block re-upserts after a local guard expires.
Fixes#4980
- Derived SQLite credential-block reconciliation delays from persisted updated_at so fresh blocks survive process restarts and sibling local stores.
- Added reopened-SQLite regression coverage for healthy usage lag after a persisted Codex 429 block.
- Aged explicit stale-block fixtures by moving updated_at outside the guard window.
Fixes#4980
- Protected Codex blocks present in RemoteAuthCredentialStore initial snapshots from immediate healthy-usage reconciliation.
- Added regression coverage for clients that start after a broker peer already persisted a fresh 429 block.
- Kept stale broker reconciliation explicit by expiring the test guard before the stale-block refresh path.
Fixes#4980
- Preserved fresh broker-sourced credential blocks by exposing store-level reconciliation delays to AuthStorage.
- Tracked fresh block observations in SQLite and remote broker stores without protecting initial stale snapshots.
- Added broker sibling regression coverage for healthy usage lag after a shared 429 block.
Fixes#4980
- Delayed healthy-usage reconciliation for newly-set local Codex blocks so lagging /usage responses cannot immediately undo a real 429 backoff.
- Added regression coverage for fresh usage-limit blocks that see healthy usage during selection.
- Kept broker stale-block reconciliation covered by seeding a persisted-only block.
Fixes#4980
- Restored all-limit Codex block clearing so recovered primary windows do not clear blocks while another reported quota remains exhausted.
- Added regression coverage for primary-recovered/secondary-exhausted selection-path reconciliation.
Fixes#4980
- Re-fetched usage for blocked Codex OAuth candidates during ranking so fresh recovered windows can clear stale persisted blocks.
- Relaxed Codex block reconciliation to trust live allowed/limitReached metadata with an available primary window.
- Added regression coverage for selection-path stale block recovery.
Fixes#4980
- Implemented plan-based tier classification for OpenAI Codex usage to ensure correct account routing.
- Updated authentication logic to normalize metadata and prioritize eligible accounts for GPT-5.6 models.
- Added fallback mechanisms to ensure standard usage ranking persists when specific tier requirements are not met.
- Validated routing behavior and model-specific selection through comprehensive unit test coverage.
- markUsageLimitReached persists codex blocks under the strategy blockScope
("shared" since 0e259bb640); the healthy-usage reconcile guard read only
unscoped state, so the self-heal from PR #4844 never fired on current main.
- Guard now resolves the ranking-strategy blockScope like selection does.
- Updated PR tests to write/read shared-scoped blocks.
- Added client-side usage overlays to the auth broker to support granular, credential-specific usage reporting.
- Integrated Fable weekly usage windows and limits into the Claude rate-limiting and ranking strategies.
- Optimized rate-limit handling by caching null results during backoff and disabling retries for 429 status codes.
- Updated credential storage to ingest and persist overlay-based usage data for improved account selection.
Batch-migrated all packages/ai/test/ files that used fs.rm/fs.rmSync
directly to use removeWithRetries/removeSyncWithRetries from
@oh-my-pi/pi-utils. This fixes EBUSY failures on Windows where SQLite
database files are still locked by the process when afterEach cleanup
hooks run.
Files migrated (24 total):
- 5 auth-broker test files (auth-broker-*.test.ts)
- 11 auth-storage test files (auth-storage-*.test.ts)
- 4 non-auth test files (aws-credentials, issue-1417, issue-957, model-cache)
- 3 remaining test files (remote-auth-store, request-debug, stream)
- 2 sync migration files (anthropic-alignment, image-limits)
Total: 28 fs.rm/fs.rmSync calls replaced with retry-enabled versions.
The retry logic: 40 retries, 25ms delay, EBUSY/EPERM/ENOTEMPTY on Windows.
Recognized bare HTTP 429 and insufficient_quota payloads as credential-rotatable quota failures in the auth retry path.
Added regression coverage for Codex OAuth sibling rotation before content emission and temporary credential blocking.
Fixes#3231
- Derived descriptors, default-model map, env keys, login list, and refresh dispatch from one ProviderDefinition per provider.
- Disabled OpenAI Codex stream obfuscation and interrupted whitespace-only tool-call argument deltas.
- Derived auth-broker callback ports and paste-code login set from the registry.
The codex oauth ranking test asserted both `maxConcurrent === 3` and a wall-clock
bound (`elapsed < 2x refreshDelay`). The concurrency counter already proves parallel
execution deterministically — serial refreshes can never raise peak in-flight above
1 — while the wall-clock bound flaked on loaded CI runners (192ms vs 150ms) and
broke the 15.10.3 release. Dropped the redundant timing assertion.
- Distributed sessions across unblocked credentials via priority-weighted hashing.
- Bucketed candidates by ranking priority for proportional fallback weighting.
- Extracted candidate comparison and ordering into dedicated helpers.
- Switched `grapheme_width_str` to compute multi-character widths from `UnicodeWidthStr::width` and then apply the macOS Hangul Compatibility Jamo correction on top of that result.
- Added regression tests in native and TUI text-width tests for VS16 emoji-presentation symbols, keycaps, and related bare-symbol cases.
- Adjusted the AuthStorage codex ranking timeout test fallback to 2s with a 1s elapsed assertion so slow aborted requests cannot mask the fast timeout path.
- Added `AuthBrokerClient`, `RemoteAuthCredentialStore`, `AuthBrokerRefresher`, and `startAuthBroker` server in `packages/ai/src/auth-broker`.
- Renamed `AuthCredentialStore` class to `SqliteAuthCredentialStore`; extracted `AuthCredentialStore` as a persistence interface.
- Added `exportSnapshot`, `forceRefreshCredentialById`, `disableCredentialById`, and `upsertCredential` to `AuthStorage` for broker wire protocol.
- Added `omp auth-broker` CLI subcommand (serve, token, login, logout, import, status) and `discoverAuthStorage` broker-mode path keyed on `OMP_AUTH_BROKER_URL`.
can1357