- Added `extractReadableFromHtml()` utility function with dual-path content extraction using Readability library and CSS selector fallback.
- Integrated Turndown library with GitHub Flavored Markdown plugin for improved HTML-to-markdown conversion supporting tables, strikethrough, and task lists.
- Refactored `getPageReadable()` action to use new extraction function, consolidating content parsing logic and improving maintainability.
- Added TypeScript type declarations for turndown-plugin-gfm module with custom Turndown rules for enhanced markdown formatting.
- Fixed duplicate synthetic tool results by tracking persisted tool call IDs and skipping synthesis when real results exist.
- Fixed Codex search streamed answer handling to detect and skip image placeholder text, preferring final answer.
- Added test coverage for duplicate tool result prevention in message transformation pipeline.
- Extracted prompt rendering and formatting utilities from coding-agent to centralized pi-utils package with new API surface (prompt.render, prompt.format, prompt.registerHelper).
- Migrated parseFrontmatter utility from coding-agent to pi-utils package; updated 8 files to import from @oh-my-pi/pi-utils.
- Removed 170-line prompt-format.ts module and consolidated 192 lines of Handlebars helper registrations into pi-utils prompt module.
- Updated 60+ files across coding-agent and typescript-edit-benchmark to use new prompt.render() and prompt.format() API from pi-utils.
- Simplified prompt-templates.ts by delegating core functionality to pi-utils while retaining custom helper registrations (jtdToTypeScript, jsonStringify, etc.).
- Replaced Python-based markitdown CLI with native markit-ai library for document and notebook conversion.
- Added support for converting Jupyter notebooks (.ipynb) to markdown via markit-ai integration.
- Created markit utility module with file/buffer conversion wrappers and standardized error handling.
- Removed markitdown from Python tools manager and eliminated external CLI dependency.
- Updated fetch, read, and scraper tools to use new markit conversion API across codebase.
- Added test coverage for ipynb file conversion through markit utility.
- Removed code_search tool and all related Exa API integration from web search module.
- Deleted code-search.ts implementation including CodeSearchResponse types and executeCodeSearch() function.
- Updated getSearchTools() to return only webSearchCustomTool, removing dual-tool support.
- Removed code_search tool documentation from README and prompt files.
- Added pluggable code search provider system supporting Exa and grep.app with provider selection via `providers.codeSearch` setting.
- Removed Exa-specific tools (`exa_linkedin`, `exa_company`, `exa_search_deep`, `exa_crawl`) and simplified web search tools to focus on core functionality.
- Refactored code search from Exa-only to provider-agnostic architecture with new `code_search` tool supporting context-aware grep.app queries and Exa fallback.
- Removed `exa.enableLinkedin` and `exa.enableCompany` configuration settings in favor of provider-based architecture.
- Added comprehensive test coverage for code search functionality including grep.app result normalization and provider fallback behavior.
- Added Parallel AI provider integration for web search with fast and research modes.
- Added Parallel extract API for URL content and YouTube video extraction with fallback support.
- Added /login parallel command and PARALLEL_API_KEY environment variable authentication.
- Added providers.parallelFetch configuration setting to control Parallel extract usage.
- Integrated Parallel provider into web search priority order between Exa and Kagi.
- Updated HTML-to-text and YouTube scrapers to prefer Parallel extract over fallback providers.
- Removed provider parameter from web search tool schema; provider selection now handled internally.
- Removed deprecated no_fallback option from search parameters; fallback behavior is now automatic.
- Renamed SearchParams type to SearchToolParams and introduced SearchQueryParams for CLI queries.
- Updated executeSearch() to accept SearchQueryParams with optional provider selection.
- Added Tavily web search provider with API key authentication and credential discovery from environment or database.
- Integrated Tavily as highest-priority search provider in fallback chain with structured response mapping and error handling.
- Added Tavily OAuth login flow in CLI and auth-storage with manual API key input and validation.
- Added comprehensive test suite for Tavily provider covering registration, response mapping, error handling, and credential validation.
Fixes#313
- Removed Kagi Universal Summarizer integration from fetch tool and YouTube scraper.
- Removed `fetch.useKagiSummarizer` configuration setting from settings schema.
- Simplified renderHtmlToText() and renderUrl() functions by removing Kagi summarization fallback logic.
- Fixed indentation inconsistencies in test files from tabs to spaces.
findAnthropicAuth() only checked OAuth credentials in agent.db, missing
api_key type credentials entirely. Users authenticated via stored API key
(not env var, not OAuth) got null from isAvailable(), causing the provider
chain to skip Anthropic.
Added tier 4 (api_key in agent.db) between OAuth check and env var
fallback. Refactored store lifecycle so tiers 3-4 share one instance.
Also fixed ExaProvider.isAvailable() which unconditionally returned true,
ignoring exa.enabled/exa.enableSearch settings and never checking for
credentials. It now respects both settings and requires an actual API key.
- Fixed WebSocket stream fallback logic to safely replay buffered output over SSE when WebSocket fails after partial content has been streamed.
- Added tracking flag to prevent unsafe replays of tool calls and terminal events during fallback transitions.
- Enhanced error recovery to reset output state when replaying buffered content over SSE connection.
- Added docs.rs scraper for extracting Rust crate documentation from rustdoc JSON, supporting modules, functions, structs, traits, enums, and other Rust items with intelligent caching.
- Implemented rustdoc JSON parsing with type rendering for complex Rust types including generics, lifetimes, trait bounds, and qualified paths.
- Added caching layer for rustdoc JSON with date-based versioning for 'latest' releases to reduce repeated fetches.
- Added automatic Ollama model capability detection via /api/show endpoint to discover reasoning and input modality support.
- Improved Kagi API error handling with structured error parsing for JSON and plain text response formats.
- Fixed Cerebras streaming compatibility by omitting stream_options.include_usage parameter.
- Simplified API key credential storage to always replace credentials instead of merging for non-minimax providers.
- Updated Kagi Search API key format from 'kagi_...' to 'KG_...' and clarified beta access requirement in provider description.
Fixes#326.
Fixes#321.
Fixes#298.
- Added kebabToCamel and normalizeKeys utility functions to convert kebab-case keys to camelCase recursively. Updated parseFrontmatter to normalize all parsed keys, ensuring consistent camelCase property access throughout the codebase. Updated all frontmatter key accesses to use camelCase notation (e.g., thinkingLevel, spdxId).
- Removed await from stream() calls that return async iterables directly.
- Removed await from synchronous utility functions like findApiKey(), resolvePythonRuntime(), and getToolPath().
- Removed await from prerenderMermaid() and findExaKey() calls that don't require awaiting.
- Added Kagi Universal Summarizer integration for URL and YouTube video summarization with fallback support.
- Exported `searchWithKagi` and `summarizeUrlWithKagi` functions from new shared `web/kagi` module for reuse across components.
- Changed HTML-to-text rendering priority to attempt Kagi summarization first before Jina, Trafilatura, and Lynx.
- Refactored Kagi search provider to use shared utilities from `web/kagi` module, reducing code duplication.
- Added `KagiApiError` exception class for Kagi API-specific error handling with optional status code tracking.
- Added `/login <provider>` command for direct OAuth provider login with automatic selector routing.
- Added Kagi API key authentication support as new OAuth provider across ai and coding-agent packages.
- Added optional `providerId` parameter to `showOAuthSelector()` for direct provider selection without manual input.
- Simplified web search result formatting by removing empty sections (Answer, Sources, Meta, Related).
- Fixed MCP tool schema dereferencing and Ajv validation warnings for non-standard format keywords.
- Refactored OAuth login/logout logic into extracted handler methods for improved maintainability.
buildAnthropicSystemBlocks was unconditionally adding cache_control to system
blocks for OAuth cloaking. This triggered the early-return guard in
applyPromptCaching, which bailed before placing cache_control on conversation
messages. Result: only the system prompt (~21K tokens) was cached; conversation
history was sent as fresh uncached input every turn, dropping cache hit rate
from ~98% to ~40%.
Fix: make cache_control in buildAnthropicSystemBlocks opt-in via a cacheControl
option. The main streaming path (buildParams) omits it since applyPromptCaching
handles all cache breakpoint placement. External callers like the Anthropic
search provider that bypass applyPromptCaching pass cacheControl explicitly.
The guard in applyPromptCaching now only checks messages (for external callers
that pre-place breakpoints), not system/tool blocks.
Also removes dead hasCacheControlInBlocks function.
Introduced by: e7daffdda (feat: Claude fingerprint hardening)
Verified: cache hit rate restored from 20.8% to 99.3%.
- Add KagiProvider implementation (Search API v0)
- Register kagi in SearchProviderId type, provider map, and fallback order
- Add kagi to tool schema enum, SearchParams type, and settings UI
- Register KAGI_API_KEY in service provider map
- Update CHANGELOG
- Added exponential backoff retry mechanism with configurable max retries for transient failures. Implemented rate limit budget tracking to prevent excessive delays on 429 responses. Fixed HTTP status propagation to prevent network errors from being retried when explicit HTTP failures occur.
providers/google-gemini-cli:
- parseGeminiCliCredentials() handles legacy, alias (project_id/refresh/expires),
and enriched credential JSON formats
- shouldRefreshGeminiCliCredentials() + refreshGeminiCliCredentialsIfNeeded()
proactively refresh OAuth tokens 60s before expiry for both providers
- normalizeAntigravityTools() converts parametersJsonSchema -> parameters
in function declarations for Antigravity compatibility
- VALIDATED tool calling config applied for Antigravity + Claude model combos
- maxOutputTokens removed from generation config for Antigravity non-Claude models
- Antigravity system instruction injection scoped to Claude + gemini-3-pro-high models
- Antigravity session ID: signed decimal int63 derived from SHA-256 of first user
message (or random bounded int63), replacing truncated hex hash
- Antigravity requestId uses agent-{uuid}; non-Antigravity requests omit
requestId/userAgent/requestType from payload
- ANTIGRAVITY_DAILY_ENDPOINT corrected to daily-cloudcode-pa.googleapis.com;
sandbox kept as fallback
- ANTIGRAVITY_SYSTEM_INSTRUCTION exported
oauth/google-antigravity:
- PKCE removed from OAuth flow (no code_challenge)
- loadCodeAssist metadata ideType changed to ANTIGRAVITY
- discoverProject uses single production endpoint; falls back to onboardUser LRO
(up to 5 retries, 2s interval) instead of hardcoded default project ID
- ANTIGRAVITY_LOAD_CODE_ASSIST_METADATA exported
oauth/google-gemini-cli:
- PKCE removed from OAuth flow
oauth/index:
- getOAuthApiKey includes refreshToken, expiresAt, email, accountId in
Gemini/Antigravity JSON payload for proactive refresh support
discovery/antigravity:
- Tries production daily endpoint first, sandbox as fallback
- Removed recommended/agentModelSorts filter; applies denylist instead
- ANTIGRAVITY_DISCOVERY_DENYLIST filters low-quality/internal models
- Request body no longer includes project field
coding-agent:
- gemini_image: corrected responseModalities to uppercase IMAGE/TEXT
- Gemini web search: endpoint fallback (daily->sandbox) with retry on 429/5xx,
aligned Antigravity request metadata, ANTIGRAVITY_SYSTEM_INSTRUCTION injection
- buildGeminiRequestTools() helper for composable googleSearch/codeExecution/urlContext
- Web search schema: expose max_tokens, temperature, num_search_results params
- Web search: explicit provider falls back to auto chain when unavailable
Tests: google-antigravity-auth, google-gemini-cli-alignment, web-search-gemini
providers/anthropic:
- Bump claudeCodeVersion to 2.1.63; system instruction identifies as Claude Agent SDK
- X-Stainless-Os and X-Stainless-Arch now runtime-computed via mapStainlessOs/mapStainlessArch
- Remove X-Stainless-Helper-Method; update package version to 0.74.0, runtime to v24.3.0
- Remove fine-grained-tool-streaming-2025-05-14 from default beta set; add
context-management-2025-06-27 and prompt-caching-scope-2026-01-05
- Accept-Encoding updated to 'gzip, deflate, br, zstd'
- Inject x-anthropic-billing-header block (SHA-256 payload fingerprint) and
Claude Agent SDK identity block with ephemeral 1h cache-control for OAuth requests
- Auto-generate cloaking user IDs for OAuth metadata.user_id when absent/invalid
- applyClaudeToolPrefix / stripClaudeToolPrefix skip Anthropic built-in tool names
- buildClaudeCodeTlsFetchOptions attaches SNI + default TLS ciphers for api.anthropic.com
- Non-Anthropic base URLs now use Bearer auth regardless of OAuth status
- Prompt-caching no longer strips then re-applies; skips if blocks already have cache_control
oauth/anthropic:
- Token URL changed from platform.claude.com to api.anthropic.com
- OAuth scopes trimmed to org:create_api_key user:profile user:inference
- Code exchange strips URL fragment from callback code (fragment used as state override)
- AnthropicOAuthFlow exported
- OAuth callback server timeout extended from 2 min to 5 min
usage/claude:
- user-agent updated to claude-cli/2.1.63 (external, cli)
- anthropic-beta header extended with full production beta set
coding-agent web search:
- Anthropic provider uses buildAnthropicSearchHeaders instead of buildAnthropicHeaders
Tests: anthropic-alignment, anthropic-oauth, claude-usage-headers, web-search-anthropic
- Replaced node-html-parser with linkedom library across all web scrapers for improved DOM API compatibility.
- Updated DOM property access from .text to .textContent and .parentNode to .parentElement for linkedom API.
- Refactored extractDocumentLinks() to use regex-based parsing with 20-link limit instead of full DOM traversal.
- Added type assertions and Array.from() wrappers for querySelectorAll() results to ensure proper typing.
Support Perplexity web search via session cookies extracted from
the desktop app (Electron/AppImage). Accepts either a raw cookie
string or a bare session token value.
Auth resolution order: PERPLEXITY_COOKIES > agent.db OAuth > PERPLEXITY_API_KEY
Request contents.summary from Exa /search API and combine up to
3 non-empty summaries into SearchResponse.answer, replacing the
previous 'No answer text returned' output.
Changes:
- Add contents.summary to every Exa search request body
- Add summary field to ExaSearchResult interface
- Export synthesizeAnswer() to build combined answer from summaries
- Export buildExaRequestBody() for testability
- Export normalizeSearchType() (was private)
- Prefer summary over text/highlights for snippet field (falsy check)
- Only synthesize answer from results that have a URL (mirrors sources filter)
Tests: 37 new tests covering normalizeSearchType, buildExaRequestBody,
synthesizeAnswer (pure unit tests) and searchExa (mocked fetch)
- Consolidated @oh-my-pi/pi-utils subpath imports into single package root import across 100+ files.
- Moved tryParseJson utility from local web scrapers module to @oh-my-pi/pi-utils package for centralized JSON parsing.
- Renamed loadSkillsFromDir to scanSkillsFromDir and refactored skill discovery to use fs.promises.readdir instead of glob-based approach.
- Replaced custom parseJSON with tryParseJson across discovery modules for consistent error handling.
- Removed emitCustomToolSessionEvent method and cleanupSshResources function, consolidating shutdown logic into dispose method.
- Updated glob pattern construction to use GlobBuilder with literal_separator(true) for improved path handling.
- Added retry logic to PubMed and OpenLibrary fetches.
- Implemented XML parsing fallback for Chocolatey package details.
- Restructured Hackage scraper to use .json and .cabal for metadata.
- Generated informative markdown for OpenCorporates API failures.
- Updated User-Agent headers for Repology.
- Moved artifact management from ToolSession to SessionManager for centralized lifecycle control and caching.
- Replaced getArtifactManager() with allocateOutputArtifact() async method in ToolSession interface for simplified artifact allocation.
- Updated bash, fetch, python, and ssh tools to call session.allocateOutputArtifact() directly with optional chaining fallback.
- Fixed Lobsters scraper to handle user fields as strings instead of nested objects in API responses.
- Extracted credential storage to shared @oh-my-pi/pi-ai package with AuthCredentialStore and AuthStorage classes.
- Consolidated UI formatting logic from ToolUIKit class into standalone utility functions across render-utils and output-meta modules.
- Moved utility functions (parseCommandArgs, substituteArgs, expandPath, normalizeUnicode) to dedicated modules for improved code reuse.
- Extracted JTD type definitions and type guards to jtd-utils module for shared use across schema conversion tools.
- Updated Claude model pricing and added cache read costs in models.json for accurate billing calculations.
- Refactored agent-storage to delegate credential management to AuthCredentialStore instead of direct SQLite operations.
- Consolidated truncation and output utilities from tools/truncate.ts and tools/output-utils.ts into session/streaming-output.ts with improved UTF-8 boundary handling.
- Renamed formatSize() to formatBytes() across codebase for consistency and clarity in byte-level formatting.
- Refactored OutputSink to use windowed byte truncation instead of full-buffer encoding, improving memory efficiency on large outputs.
- Migrated from Buffer to Uint8Array in web scrapers for better cross-platform compatibility and native browser support.
- Added getArtifactManager() lazy-initialization method to ToolSession for deferred artifact manager instantiation.
- Simplified API surface with wildcard exports from tools and session modules, reducing import complexity.
- Consolidated Gemini CLI and Antigravity headers into central definitions.
- Split Antigravity headers into distinct authentication and streaming sets.
- Enabled User-Agent configuration via environment variables.
- Ensured consistent header usage across relevant services.
Export getAntigravityHeaders() from pi-ai and replace duplicated
ANTIGRAVITY_HEADERS constants in gemini-image tool and Gemini search
provider with calls to the centralized function.
Fixes#132
Replace hardcoded "antigravity/1.11.5 darwin/arm64" User-Agent strings with
the centralized getAntigravityUserAgent() function from @oh-my-pi/pi-ai in
gemini-image tool and Gemini search provider.
Fixes#132
can1357