- Removed unreliable Bing and Yahoo HTML-scraping search providers.
- Deleted `src/web/search/providers/bing.ts` and `src/web/search/providers/yahoo.ts` implementation files.
- Updated `provider.ts`, `types.ts`, and `public.ts` to prune provider registration and configuration.
- Adjusted `web-search-public.test.ts` to exclude removed engines from test coverage.
Lazy-initialized the header-generator dependency so compiled runtimes without its fs-loaded data_files fall back to the bundled Chrome header profile instead of failing extension imports.
Added a regression test that hides header-generator data_files in a fresh Bun subprocess and verifies fallback headers are returned.
Fixes#5178
- Added six new search providers (Bing, Yahoo, Ecosia, Startpage, Mojeek, and Public) to expand coverage and parallel search capabilities.
- Implemented a unified `browserFetch` utility with headless-browser fallback and randomized Chrome profiles to improve scrape reliability.
- Integrated automated bot-defense mechanisms including CAPTCHA detection, ALTCHA proof-of-work, and homepage-token flows.
- Fixed hanging search CLI commands by ensuring proper closure of AuthStorage connections.
- Added a mandatory check to ensure authentication storage is successfully initialized before executing searches.
- Implemented a finally block to close the discovered authentication storage after the search execution completes.
- Implemented a new Google search provider using headless browser scraping and HTML parsing.
- Integrated automated bot-challenge detection and error reporting for search operations.
- Consolidated navigation headers into a shared utility and updated existing DuckDuckGo provider to use it.
- Added comprehensive test suites for Google search parsing, deduplication, and browser operation diagnostics.
- Extracted gunzipRustdocJson() with overridable maxOutputLength so the cap contract is testable with real gzip payloads instead of the banned mock.module().
Tightened xAI web_search's xai-oauth preference so lower-priority xai-oauth api_key or fallback credentials do not get shadowed by the shared XAI_API_KEY fallback.
Added regression coverage for the stored xai-oauth API-key plus shared XAI_API_KEY case, preserving explicit xai runtime credential routing.
Refs #4536
Restricted the xai-oauth preference in web_search to dedicated credentials (hasNonEnvCredential("xai-oauth") or XAI_OAUTH_TOKEN) so an XAI_API_KEY-only environment no longer routes an explicit xai runtime/config credential through the xai-oauth resolver.
Added regression tests covering the shared-env case and the xai-only availability check.
Refs #4536
- Stopped adding Responses Agent Tools-incompatible search_parameters to xAI web_search requests.
- Kept limit and numSearchResults enforcement as a local cap over parsed sources and citations.
- Added regression coverage for limit, numSearchResults, recency, and local cap request shapes.
Fixes#4537
Added providers.webSearchGeminiModel and GEMINI_SEARCH_MODEL so Gemini web_search requests use a selected grounding model while keeping gemini-2.5-flash as the fallback.
Covered OAuth, Developer API, and missing modelVersion fallback paths in Gemini web search tests.
Fixes#4312
docs-rs.ts:handleDocsRs downloads up to 50 MB of compressed rustdoc JSON (MAX_BYTES) and then decompresses it with gunzipSync without a size bound, so a zip bomb can expand 10:1+ and OOM/crash the process. Pass { maxOutputLength: 256 * 1024 * 1024 } to gunzipSync at line 402 to cap decompressed output; if the limit is exceeded it throws and falls through to the existing catch (signal check / return null), preserving the failure contract. Coding-agent typecheck (bun run check:types) and Biome check on the changed file pass; no dedicated test exists for this path.
Closes#4249
- Updated the default browser User-Agent string to emulate a modern version of Chrome.
- Added typical browser headers to the outgoing fetch request, including Sec-Ch-Ua, Sec-Fetch flags, and Referer.
- Added a blank "b" parameter to the form body to match native DuckDuckGo HTML search behavior.
Formatted fallback-chain provider errors through the shared formatter so Codex auth failures and DuckDuckGo bot-detection failures give actionable guidance.
Documented DuckDuckGo as a best-effort fallback for datacenter/shared-egress IPs and covered the provider guidance in regression tests.
Fixes#3863
Enabled the Gemini web search provider to use standard Google developer API credentials when Cloud Code Assist OAuth is absent.
Added developer API request coverage for native Google Search grounding and preserved existing OAuth request serialization.
Fixes#3810
The DuckDuckGo provider hit api.duckduckgo.com (the Instant Answer API),
which only serves Wikipedia / Wolfram-Alpha-style topics — empty
AbstractText / Results / RelatedTopics for the vast majority of agent
queries. The orchestrator then rejected the empty response and surfaced
'DuckDuckGo returned no renderable search content', leaving users with
no working free fallback.
Switch the provider to POST html.duckduckgo.com/html/ (the no-JS HTML
frontend) with a browser User-Agent, parse the result blocks (unwrapping
//duckduckgo.com/l/?uddg=… redirect URLs), and map recency to the df
form field (d/w/m/y). When DuckDuckGo serves the bot-detection modal
(HTTP 200/202 with anomaly-modal body) we surface a clear
SearchProviderError so the orchestrator can fall through to the next
provider with cause attached.
Fixes#3799
Distinguished an absent provider (use configured preferred provider) from an explicit `--provider auto` (one-shot bypass that still respects exclusions) in executeSearch.
Fixes#3793
Initialized the Z.AI Streamable HTTP MCP session before calling web_search_prime and preserved the returned session id on subsequent requests.
Added regression coverage for the authenticated MCP request sequence.
Fixes#3619
Raced queued Exa throttle waits against the caller abort signal so requests cancelled behind an earlier throttle wait reject immediately without breaking the serialized throttle chain.
Added regression coverage for cancelling a third Exa request queued behind another delayed request.
Fixes#3271
Made Exa request pacing observe cancellation during the configured delay instead of waiting for the full delay before checking the signal.
Added regression coverage for a queued Exa request cancelled while throttled.
Fixes#3271
Added configurable Exa search request pacing via exa.searchDelayMs so repeated web_search calls no longer burst directly into Exa rate limits.
Covered the provider contract with a focused Exa test and recorded the back-to-back request repro.
Fixes#3271
PerplexityProvider.isAvailable() accepted authStorage.hasAuth("openrouter")
as a valid credential, so any user with an OpenRouter key configured (for
LLM access) had every webSearch: auto request silently routed through
OpenRouter's perplexity/sonar-pro endpoint. Since Perplexity sits first in
SEARCH_PROVIDER_ORDER, downstream providers like Gemini were never reached
and users saw unexpected charges on their OpenRouter billing.
Auto-chain admission now requires a direct Perplexity credential
(PERPLEXITY_COOKIES, Perplexity OAuth, or PERPLEXITY_API_KEY).
isExplicitlyAvailable still returns true, so users who want the
OpenRouter-backed perplexity/sonar-pro path can opt in by setting
webSearch: perplexity explicitly — the existing OpenRouter fallback in
getApiConfigs handles that case unchanged.
Fixes#3251
- Centralized Parallel API utilities and parsing logic into a single module.
- Exported constants and helper functions from `parallel.ts` to replace duplicated definitions in the search provider.
- Updated the search provider to leverage the unified `parseParallelSearchPayload` function with metadata parsing toggled off.
- Moved authentication logic to `perplexity-auth.ts` to share logic between search providers and CLI commands.
- Updated authentication priority to prefer browser cookies over OAuth tokens during search operations.
- Modified the `token` CLI command to display active OAuth tokens when both an OAuth token and an API key are configured.
- Added comprehensive unit tests in `perplexity.test.ts` to verify authentication priority and precedence.
- Replaced inconsistent list truncation messages with a standardized `[...N items elided...]` format across all scrapers.
- Updated documentation reference to use consistent variable naming convention.
- Updated various truncation and overflow messages across `pi-shell` and `coding-agent` to use the consistent `[...N units elided...]` format.
- Improved clarity of elided output by explicitly stating the count and type of omitted information.
- Integrated comprehensive loop guard support for DeepSeek and assistant prose patterns, including configurable stream checks.
- Implemented Moonshot Flavored JSON Schema (MFJS) normalization for improved tool compatibility and enum type inference.
- Added support for Ollama reasoning effort backfilling and Grok-specific service tier cost tracking across providers.
- Expanded model catalog with new entries and unified compatibility logic for improved OpenRouter API integration.
- Derived the `SEARCH_PROVIDER_ORDER`, `SEARCH_PROVIDER_PREFERENCES`, and `SEARCH_PROVIDER_LABELS` metadata dynamically from a single `SEARCH_PROVIDER_OPTIONS` source of truth.
- Reprioritized the default search provider sequence, shifting higher-order choices like Perplexity, Gemini, and Anthropic ahead of Tavily and Brave.
- Updated documentation to reflect the new search provider evaluation order.
can1357