Replace the sunset V0 Search API with V1 (POST /api/v1/search) under the
existing `kagi` provider id instead of shipping a parallel `kagi-v1`
provider. Credentials still resolve through the shared AuthStorage broker
(Bearer token, KAGI_API_KEY, /login kagi), and recency now maps to a
UTC-deterministic filters.after date.
- Merge V1 client into src/web/kagi.ts (categorized result buckets, direct
answer, related/adjacent questions)
- Keep classifyProviderHttpError mapping for auth/quota signals
- Drop the kagi-v1 entries from the provider registry, order, type union,
and settings schema
- Consolidate tests into web-search-kagi.test.ts
- Added `ToolTier`, `ToolApproval`, and `ToolApprovalDecision` types and exported approval APIs.
- Updated approval-mode options from `auto|prompt|custom` to `always-ask|write|yolo` and defaulted mode to `yolo`.
- Changed approval resolution to apply per-tool decisions first, then mode-tier limits, with legacy-mode migration.
- Assigned read/write/exec `approval` and approval-detail prompts across built-in, custom, extension, and MCP tools.
Centralizing OAuth refresh in AuthStorage (e6893515) introduced five
follow-on bugs surfaced by an audit of the commit; this fixes all of
them and updates the tests that relied on the old refresh seam.
1. packages/ai/src/auth-storage.ts (#tryOAuthCredential):
For built-in providers the path went directly to `getOAuthApiKey`
with the (possibly still-expired) selection.credential when the
pre-refresh at line 2587 caught a transient error. `getOAuthApiKey`
then threw the "expired … must be refreshed via AuthStorage"
precondition error, which the disable classifier matched against
`/expired.*refresh/` and soft-disabled the row. A single network
blip during refresh could permanently kill a still-valid Anthropic /
OpenAI / Gemini-CLI / Copilot credential. Built-in providers now
route through the broker-aware single-flighted
`#refreshOAuthCredential` first, so transient failures surface as
network errors (5-min temp block) instead of definitive auth
failures.
2. packages/ai/src/auth-storage.ts (#fetchUsageUncached):
The usage refresh check only fired once `Date.now() >= expiresAt`,
missing the 60-second skew that `getApiKey` honors. A token
expiring inside the skew window was posted to the usage endpoint
and 401'd mid-flight, briefly hiding quota in the UI. Aligned with
`OAUTH_REFRESH_SKEW_MS`.
3. packages/coding-agent/src/web/search/index.ts (webSearchCustomTool):
The CustomTool counterpart of WebSearchTool dropped sessionId so
SDK callers that opted into `web_search` via toolNames lost
per-session credential stickiness — multi-account users saw the
provider round-robin between searches in the same session. Threads
`ctx.sessionManager.getSessionId()` through to `executeSearch`.
4. packages/coding-agent/src/web/search/providers/perplexity.ts
(findOAuthToken):
`authStorage.getApiKey("perplexity")` returns runtime/config
overrides, stored api_key credentials, OAuth bearers, and env keys.
Filtering only env keys meant a config-pinned `pplx-…` API key was
POSTed to `www.perplexity.ai/rest/sse/perplexity_ask` (the OAuth
endpoint) instead of falling through to
`api.perplexity.ai/chat/completions`, producing 401s. Switched to
`getOAuthAccess` so only true OAuth bearers reach the OAuth
branch; api_key credentials/overrides correctly fall through.
5. packages/ai/scripts/generate-models.ts:
`getOAuthApiKey` was being called directly with possibly-expired
credentials. The new contract throws on expired, the broad catch
swallowed it, and the build silently fell back to bundled models
instead of refreshing. Both helpers now route through
AuthStorage's `getApiKey` / `getOAuthAccess`, which trigger the
full broker-aware refresh pipeline.
Test updates:
- auth-storage-credential-disabled-event.test.ts,
sdk-credential-disabled-bridge.test.ts: the `failOAuthRefresh`
helper used to spy on `getOAuthApiKey` to inject invalid_grant.
With refresh now happening before that helper, the spy never fired.
Switched to spying on `refreshOAuthToken` so the simulated failure
reaches the disable classifier.
- auth-storage-rotation.test.ts: stub `refreshOAuthToken` so the test
doesn't hit a real OAuth endpoint when the seeded credential lands
inside the 60s skew window.
- packages/ai/test/issue-957-repro.test.ts now tests:
- refreshKimiToken applies the 5-minute server-side skew (Kimi-specific)
- AuthStorage refreshes kimi-code credentials inside its 60s skew window
- packages/ai/test/anthropic-stream-timeout.test.ts: raise the
streamFirstEventTimeoutMs from 10ms to 5000ms so slow CI scheduling
cannot fire the first-event watchdog before the mocked events arrive.
The test still exercises the (1ms) idle path it was written for.
fix(web): allow Parallel extract via PARALLEL_API_KEY env var without storage
The fetch tool and YouTube scraper previously gated the Parallel extract
branch behind `storage && findParallelApiKey(storage)`. With no
AgentStorage the env key was never consulted, so callers that ran
without a per-session storage (e.g. ReadTool sessions in unit tests, and
in practice any caller that has only an env API key) silently fell back
to raw-html / no-ytdlp paths.
- findCredential/findParallelApiKey now accept null or undefined storage
and rely solely on the env-first path when no storage is supplied.
- searchWithParallel/extractWithParallel mirror the same nullable shape.
- Drop the redundant `storage && ` guards in fetch.ts and youtube.ts;
the inner findParallelApiKey call already returns null when no
credential is available.
- Centralized OAuth access lifecycle in `AuthStorage`, returning identity metadata and new access-result types.
- Added 60-second skew and strict expiry checks, returning undefined/throws for stale or expired OAuth credentials.
- Removed provider-local token refresh flows from Gemini, Gemini CLI, Antigravity, Kimi, and related OAuth helpers.
- Migrated web-search providers from `AgentStorage` to `AuthStorage` session-aware lookup with `authStorage`/`sessionId`/`signal` flow.
- Replaced `findAnthropicAuth`/DB auth lookup with `buildAnthropicAuthConfig` and explicit base-url override/env fallback ordering.
- Cleaned up tests, made them more useful
- Fix a bug in the OpenAPI spec with Kagi v1 where 'trace' was mapped to
'id' incorrectly from Kagi's documentation
- Added OpenAI Codex and Gemini web search provider options with updated setup/auth descriptions.
- Updated Codex OAuth flow to refresh near-expiry tokens during web_search and persist the refreshed credentials.
- Plumbed AgentStorage through search orchestrator, scrapers, and fetch paths so providers share session credentials.
- Refactored web provider and credential helpers to accept caller-provided AgentStorage and resolve keys synchronously.
Replace relative time strings ("1d ago", "1w ago", "1mo ago", "1y ago") with
computed YYYY-MM-DD dates in the Kagi V1 search request's `filters.after`
field. The new `recencyToDate()` helper handles month/year drift and leap-day
rollover correctly.
This aligns with the Kagi V1 API's documented expectation of ISO date strings
in the filters object. Relative time strings were a carryover from the V0
integration and caused silent filter mismatches when the API stopped resolving
them on its end.
- Change filters.time_after/time_before → filters.after/before (V1 API
field names; old keys were silently ignored, breaking every recency query)
- Update KagiV1SearchData to include all V1 response categories
(podcast, adjacent_question, interesting_news, etc.)
- Update KagiV1SearchResultItem.props to Record<string, unknown>
- Parse related_search entries as result objects, extracting question
text from props.question/props.query/title
- Read direct_answer from standard result object fields (snippet/title)
instead of the non-existent .text field
Adds a shared classifyProviderHttpError helper that maps the well-known
failure shapes (status 401/402/403 and bodies matching credits / quota
/ insufficient) into compact SearchProviderError messages so the
orchestrator advances to the next provider instead of bailing.
Wired into every HTTP-talking provider (codex, exa, gemini, anthropic,
brave, jina, kimi, perplexity, searxng, synthetic, tavily, zai) and the
two wrapped-error providers (kagi via KagiApiError, parallel via
ParallelApiError). The orchestrator now collects per-provider failures
and emits a joined summary ("exa: 403 forbidden; codex: credits
exhausted; ...") when the whole chain fails.
Test updates: web-search-{exa,tavily,kagi} message expectations now
assert the compact "<id>: <status> <reason>" form.
Bun's WinHTTP backend can ignore AbortSignal once a TCP/TLS connection
stalls (oven-sh/bun#15275, oven-sh/bun#18536), so Esc never reached the
in-flight `web_search` fetch on Windows and the session froze until
Ctrl+C. Only kimi shipped any timeout at all (server-side); every other
provider passed `signal` to `fetch` with no client-side bound.
Introduced `withHardTimeout(signal, ms=60_000)` in providers/utils and
wired it into every web-search provider's outbound fetch — anthropic,
brave, codex, exa, gemini, jina, kagi, kimi, parallel, perplexity
(api-key and oauth), searxng, synthetic, tavily, z.ai. 60s tolerates
legitimate slow LLM-mediated responses while still guaranteeing the
request settles within a minute when Bun's abort fails to propagate.
Independently, `executeSearch`'s provider-fallback loop swallowed every
`AbortError` as a regular provider error and returned
"All web search providers failed", masking cancellation on every
platform. The catch block now calls `throwIfAborted(signal)` first so a
caller-initiated cancel propagates as `ToolAbortError`.
Fixes#1221
- Updated Perplexity JWT parsing in the ai OAuth utilities to return a far-future sentinel when `exp` is missing and use that when computing token expiry.
- Updated `getOAuthApiKey` to prefer the JWT expiry and normalized legacy one-hour `expires` values to a non-expiring value.
- Updated coding-agent web search token lookup to verify Perplexity credentials against the JWT `exp` claim and treat missing claims as non-expiring.
When the Codex backend returned the literal "(see attached image)" as
the answer for a text query and streamedAnswer was empty, the wrapper
accepted the placeholder as the response and the chain never advanced.
Throw SearchProviderError so the next provider is tried.
- Replaced all StringEnum(...) usages with z.enum([...]) across tools, examples, and tests.
- Removed StringEnum re-export from @oh-my-pi/pi-coding-agent public API.
- Condensed verbose tool parameter descriptions to minimal lowercase phrases.
- Renamed AuthCredentialStore to SqliteAuthCredentialStore at usage sites.
- Unified line-ending normalization to `replace(/\r\n?/g, "\\n")` in editor, scraper, benchmark, and utils modules.
- Added terminal-aware line sanitization in code-cell rendering to collapse inline carriage returns and avoid overwrite corruption.
- Tightened editor and paste sanitizers to trim control characters consistently after CR normalization.
- Added canonical `pi.zod` schema API exports and removed TypeBox package exports/imports.
- Migrated Tool schema typing from TypeBox to shared `TSchema`/Zod flow with legacy TypeBox compatibility.
- Updated AI provider adapters and MCP/agent builders to convert tool params through `toolWireSchema()`.
- Reworked schema validation from AJV to Zod-safe parsing with `fromTypeBox`, `toolWireSchema`, and meta schema checks.
- anthropic: add signal to AnthropicSearchParams, callSearch(), and
AnthropicProvider.search()
- exa: add signal to ExaSearchParams, callExaSearch(), and ExaProvider.search()
- jina: add signal to JinaSearchParams, callJinaSearch(), and JinaProvider.search()
- zai: add signal to ZaiSearchParams, callZaiTool(), and ZaiProvider.search()
- gemini: add signal to GeminiSearchParams, callGeminiSearch() and
buildInit() so both fetchWithRetry calls (initial + auth-refresh retry)
carry the signal
The five providers listed above never forwarded SearchParams.signal to the
underlying HTTP layer, so pressing Esc during a web_search call had no effect
and the session froze until the request resolved or Ctrl+C was pressed.
brave, kimi, perplexity, searxng, tavily, synthetic, codex, kagi, and parallel
already thread the signal correctly and are unchanged.
Fixes#1044
- Removed local `abortableSleep` in favour of Node's built-in `scheduler.wait` from `node:timers/promises`.
- Consolidated per-provider retry/fetch loops into a shared `fetchWithRetry` utility in `packages/utils`.
- Moved `extractHttpStatusFromError`, `isRetryableError`, and related helpers out of `packages/ai` into `packages/utils`.
- Deleted `extractRetryDelay` in favour of `extractRetryHint` with unified header and body parsing.
- Extended executeSearch to accept an optional AbortSignal parameter.
- Passed the provided signal through to the search provider invocation.
- Updated execute methods to forward incoming signals into executeSearch.
- Added optional `loadMode` and `summary` fields to `AgentTool` and related type declarations.
- Added `loadMode` and `summary` metadata to built-in tool classes for discoverable/essential behavior.
- Replaced `BUILTIN_TOOL_METADATA` with per-tool fields in discovery code paths.
- Updated `search_tool_bm25` and discovery indexing to use each tool's `summary` text.
- Updated discovery tests to validate tool `loadMode` and summary completeness.
- Converted systemPrompt APIs and state types to ordered `string[]` across agent, AI, and coding-agent surfaces.
- Added `normalizeSystemPrompts` and applied it to context normalization before building provider request payloads.
- Updated AI providers to emit separate normalized prompt blocks/messages instead of a single merged system prompt.
- Removed dedicated `projectPrompt` state and remapped that context into system-context buckets in session, dump, and token accounting.
- Aligned tests and changelogs to pass and assert `systemPrompt` as arrays with ordered prompt semantics.
- Consolidated AI provider imports through register-builtins and moved Gemini/Antigravity header helpers to a shared module.
- Added lazy loading for heavy providers and SDK-backed modules with cached initialization to trim startup cost.
- Converted markdown conversion helpers to async and awaited htmlToBasicMarkdown in affected scraper and kernel output paths.
- Parsed bundled agent definitions on-demand and moved BrowserTool prompt rendering behind a memoized getter.
- Added cached validation/error handling paths by replacing AJV runtime checks with Value.Check and trimming validation error output.
- Documented and removed `utils/oauth` from the `ai` package entrypoint, noting it as a breaking change.
- Refactored `cli`, `auth-storage`, and `utils/oauth` to load provider modules via scoped dynamic `import()` calls.
- Removed top-level provider imports and barrel exports from `utils/oauth/index.ts`, streamlining oauth module loading.
- Consolidated OAuth symbol, type, and provider imports in coding-agent and tests to `@oh-my-pi/pi-ai/utils/oauth` modules.
- Defined `DEFAULT_LOCAL_TOKEN` locally in model-registry and removed its cross-package OAuth import usage.
- Renamed the built-in `grep` content-search tool to `search` across settings, schemas, and SDK exports.
- Switched execution wiring so `Task`, `Plan`, cursor, and shell mapping now invoke `search` instead of `grep`.
- Updated prompts, plan-mode docs, and example tool lists to replace `grep`/`ls` references with `search` guidance.
- Aligned `Grep*`/`grep` event, renderer, and hook types to `Search*`/`search` across runtime and tests.
- Documented and fixed `search` result rendering budget behavior and added internal-URL/path-list transcript notes.
Add explicit Basic auth credentials for SearXNG while preserving Bearer token fallback.
Keep optional SearXNG settings out of the TUI and allow empty RFC 7617 username/password fields.
- Updated the ZAI provider constant to use the `web_search_prime` tool name.
- This aligned the search provider configuration with the expected MCP endpoint identifier.
- Updated the root `fix:ts` script to run `bun run --workspaces --if-present fix` after `fix:tools`.
- Updated `fix:ts:all` to run `fix:tools:all` followed by workspace-wide `fix` tasks.
- Applied minor SearXNG request cleanup, including direct `Authorization` header assignment and formatting-only formatting changes.
When the Codex Responses API synthesizes an answer without emitting
url_citation annotations, previously-empty sources made cited results
look ungrounded. Add:
- tool_choice: { type: "web_search" } so Codex must call the tool
- markdown-link + bare-URL extraction from the answer as a fallback
that only runs when no structured citations were returned
The model-resolution path is unchanged; getBundledModels already
returns a usable Codex catalog on main.
Closes#724
- Standardized missing-file read errors and now return `File not found: <path>` for absent edit targets.
- Centralized AI provider, usage, and OAuth helpers into shared modules to remove duplicated logic.
- Migrated OAuth/API-key login flows to shared factory helpers and removed inline prompt/token-exchange code.
- Reused shared tools and formatter utilities for discovery, stream tails, LSP batching, and source formatting.
- Consolidated repeated test helpers and fixtures into shared modules, replacing inline helper duplicates.
Removed unconditional topic:news coupling in buildRequestBody that scoped
Tavily index to news publications whenever recency was set. Technical queries
with --recency now search the general index filtered by time only.
Tightened SearchParams.recency contract in base.ts: providers MUST interpret
recency as a pure time filter and MUST NOT change topic scope as a side effect.
- SearXNG time_range only supports day/month/year; map week→month
- Add SearXNG option to settings-defs.ts submenu so users can
select it from the Settings UI (Providers tab)
- Import settings singleton directly (matches Exa provider pattern)
- Replace getEnvApiKey with process.env for env var fallbacks
(getEnvApiKey only works for registered provider IDs)
- Remove settings parameter from searchSearXNG/findEndpoint/findToken
- Add try/catch around settings.get calls for initialization safety
Fixes Codex review P1 issues on PR #712
Add SearXNG as a search backend for the web_search tool, enabling
privacy-friendly, self-hosted metasearch without API keys or usage limits.
Changes:
- New SearXNGProvider implementing SearchProvider interface
- Supports bearer token auth, categories, language, and recency filters
- Configurable via settings (searxng.endpoint, .token, .categories, .language)
or environment variables (SEARXNG_ENDPOINT, SEARXNG_TOKEN)
- Registered in provider chain and settings schema
- Maps SearXNG JSON API response to unified SearchResponse
Closes#545
- Added `extractReadableFromHtml()` utility function with dual-path content extraction using Readability library and CSS selector fallback.
- Integrated Turndown library with GitHub Flavored Markdown plugin for improved HTML-to-markdown conversion supporting tables, strikethrough, and task lists.
- Refactored `getPageReadable()` action to use new extraction function, consolidating content parsing logic and improving maintainability.
- Added TypeScript type declarations for turndown-plugin-gfm module with custom Turndown rules for enhanced markdown formatting.
- Fixed duplicate synthetic tool results by tracking persisted tool call IDs and skipping synthesis when real results exist.
- Fixed Codex search streamed answer handling to detect and skip image placeholder text, preferring final answer.
- Added test coverage for duplicate tool result prevention in message transformation pipeline.
Can Bölük