- Removed the `fflate` dependency in favor of using `node:zlib` for ZIP operations.
- Updated documentation and internal code comments to reflect the transition to native `node:zlib` DEFLATE support.
- Replaced `fflate` dependency with `node:zlib` and manual ZIP framing in `src/utils/zip.ts`.
- Implemented lazy loading for site-specific scrapers to reduce cold-start latency.
- Unified ZIP compression and extraction logic to use native `zlib` stream decoders.
- Optimized ZIP member decoding by implementing memory-safe length-bounded inflation.
- Centralized archive operations into a new `utils/zip.ts` module with unified support for ZIP, tar, and tar.gz formats.
- Optimized ZIP reading using lazy, ranged central-directory access and implemented ZIP64 support for large files.
- Hardened archive extraction with directory traversal protection and configured memory limits for loading and extraction.
- Refactored tool-specific logic to utilize the new centralized utility and deleted the redundant `archive-reader.ts`.
- Added lazy async module loaders for @babel/parser, linkedom, puppeteer/browsers, @mozilla/readability, @xterm/headless, and mnemopi to avoid loading them during cold startup.
- Added an interactive startup splash before session construction and skipped it for resume/fork/continue, quiet mode, timing mode, or non-TTY runs.
- Updated JS import-rewrite and memory tests to match async parser loading and preloaded mnemopi modules for sync state helpers.
- Introduced memoized dynamic import loaders for Babel parser, mnemopi modules, puppeteer, and HTML-related packages.
- Refactored eval import-rewrite helpers and runtime call sites to use asynchronous wrapping and parsing flows.
- Shifted fetch and web-scraper linkedom usage to on-demand imports so heavy modules load only when needed.
Completes the injectable-fetch transport wiring (15.10.8) that the feature
left half-done, fixing the deterministic CI test failures:
- compaction.compact() rebuilt summaryOptions field-by-field but dropped
`fetch`, so the injected transport never reached
requestOpenAiRemoteCompaction / generateSummary's remote path. Thread it.
- Read-tool URL pipeline had no fetch seam: renderHtmlToText gained a
fetchOverride param but renderUrl/ToolSession never carried one, so the
jina/parallel reader backends always used global fetch. Add
ToolSession.fetch -> renderUrl -> renderHtmlToText (defaults to global).
- searchWithParallel mirrored extractWithParallel but missed the fetch
option; add it.
- Repair tests whose deleted hookFetch interceptors were never replaced
with a FetchImpl seam (fetch-kagi-toggle, web-search-parallel,
issue-970 discovery).
- Update issue-1746 POSIX case to the #2154 preserved-scrollback contract:
unknown-viewport streaming deferral is now platform-independent.
- Added optional FetchImpl fields to compaction, proxy, AI, coding-agent, and mnemopi options.
- Threaded injected fetch implementations through OAuth, discovery, and search/LLM request flows.
- Removed exported hookFetch utility and its package entrypoint from utils.
- Replaced global-fetch test monkeypatching with per-test FetchImpl mocks across test suites.
- Adjusted `LspTool` to retry only zero/decl-only references with two 250ms waits for project-aware servers.
- Removed raw-mode GitHub repo README API fallback in `read`, so `:raw` now renders the page directly.
- Replaced regex heuristics in `isImagePlaceholderAnswer` with a fixed normalized placeholder set.
- Added `:raw` repo-root resolution via GitHub API `/readme` for decoded markdown.
- Preserved fallback to default raw HTML rendering when the README payload was unusable.
- Added regression coverage for successful README decoding and fallback behavior.
- Added archive format sniffing to identify ZIP, TAR, and TAR.GZ from file bytes.
- Added MIME/extension and header-based routing for notebook, sqlite, and archive payloads.
- Added archive entry rendering with slash-terminated dirs and size suffixes.
- Added tests for archive, sqlite, notebook, and fallback binary dispatch scenarios.
- Replaced `providers.parallelFetch` with a new `providers.fetch` enum in settings and added migration cleanup for the legacy key.
- Updated `renderHtmlToText` to follow configured reader preference with ordered fallback attempts and remote-reader timeout handling before local conversion.
- Updated YouTube and fetch tests to use `providers.fetch` and cover Jina-first stall fallback behavior.
- Added image-reference rendering to make `[Image #N]` placeholders clickable in chat.
- Added MIME-aware image blob materialization with extensioned sidecar paths.
- Added clickable path, line, and URL hyperlinks for read, search, and fetch outputs.
- Hardened OSC8 hyperlink emission with URI validation and control-byte/idempotency checks.
A stalled Jina reader request shared the overall reader-mode AbortSignal
with the downstream trafilatura/lynx/native fallbacks. When Jina hung
until the budget timer fired, the shared signal aborted and the catch
handler's signal?.throwIfAborted() re-threw before any local fallback
ran.
- Bound Jina and Parallel extract to their own per-attempt sub-budget
(REMOTE_READER_MAX_MS, capped at 10s) so a remote stall cannot consume
the whole overall reader-mode budget.
- Catch handlers now rethrow only on real userSignal cancellation, not
on remote sub-budget or overall budget expiry.
- Wrap trafilatura/lynx in their own try/catch so a subprocess failure
or abort does not skip the in-process native renderer.
- Always attempt the native renderer last: it works on already-loaded
HTML with no network or subprocess, so even an exhausted overall
budget still yields a result.
Fixes#1449
- Added support for multi-range line selectors on URLs (e.g., `:5-10,20-30`) and combining `:raw` mode with line range selectors.
- Added support for line range selectors on directory listings with offset and limit parameters.
- Fixed `:raw` selector being ignored for JSON and feed URLs and directory listing line selectors dropping offset parameter.
- Added clear error message for line offset beyond directory listing end.
- Refactored URL parsing and directory reading to support multiple comma-separated ranges and improved line-based slicing logic.
- Added comprehensive test coverage for multi-range selectors, raw mode combinations, and directory range operations.
- Parser now buffers markdown-style `#` lines and skips them when they directly precede a hashline operation.
- It now preserves comment lines that are not immediately before an operation while still handling blank separators as regular raw input.
- Added focused parser tests plus prompt and changelog updates describing the new comment-skipping behavior.
- packages/ai/test/issue-957-repro.test.ts now tests:
- refreshKimiToken applies the 5-minute server-side skew (Kimi-specific)
- AuthStorage refreshes kimi-code credentials inside its 60s skew window
- packages/ai/test/anthropic-stream-timeout.test.ts: raise the
streamFirstEventTimeoutMs from 10ms to 5000ms so slow CI scheduling
cannot fire the first-event watchdog before the mocked events arrive.
The test still exercises the (1ms) idle path it was written for.
fix(web): allow Parallel extract via PARALLEL_API_KEY env var without storage
The fetch tool and YouTube scraper previously gated the Parallel extract
branch behind `storage && findParallelApiKey(storage)`. With no
AgentStorage the env key was never consulted, so callers that ran
without a per-session storage (e.g. ReadTool sessions in unit tests, and
in practice any caller that has only an env API key) silently fell back
to raw-html / no-ytdlp paths.
- findCredential/findParallelApiKey now accept null or undefined storage
and rely solely on the env-first path when no storage is supplied.
- searchWithParallel/extractWithParallel mirror the same nullable shape.
- Drop the redundant `storage && ` guards in fetch.ts and youtube.ts;
the inner findParallelApiKey call already returns null when no
credential is available.
- Added OpenAI Codex and Gemini web search provider options with updated setup/auth descriptions.
- Updated Codex OAuth flow to refresh near-expiry tokens during web_search and persist the refreshed credentials.
- Plumbed AgentStorage through search orchestrator, scrapers, and fetch paths so providers share session credentials.
- Refactored web provider and credential helpers to accept caller-provided AgentStorage and resolve keys synchronously.
- Updated event-controller read-tool streaming handling to wait for a parseable target before routing tool calls, avoiding early component binding for unresolved arguments.
- Allowed internal-URL read calls to bypass the regular read grouping path and fall through to direct tool execution.
- Adjusted read tool rendering to honor the computed `expanded` flag for completed output instead of forcing expanded output.
- Fixed `issue://owner/diff` and `pr://owner/diff` parsing so they resolve to issue and PR list outputs.
- Fixed `pr` short-form parsing by requiring `scheme==='pr'` and a numeric host before `diff` matching.
- Fixed PR unified-diff parsing to decode quoted header paths and count `----`/`++++` hunk lines as one deletion/addition.
- Fixed `read` error rendering to emit status blocks with cleaned, range-aware, tab-normalized lines.
- Stopped `github-cache` from chmod-ing existing parent directories, preserving pre-existing permission modes.
- Updated the read tool docs to define URL selectors as `:50`, `:50-100`, and `:50+150` and to document the `https://host/:port` form for URLs with explicit ports.
- Changed URL selector parsing to use the file-style numeric range format, treated `raw` as a separate token, and updated the invalid zero-line message to direct users to `:1`.
- Refined embedded URL selector extraction to validate the base URL first and then match selectors against the new numeric range regex.
- Removed the read CLI argument and tool schema field so read requests no longer accept custom timeouts.
- Updated URL read handling to stop forwarding timeout values and execute URL reads without a timeout parameter.
- Standardized URL read fetching to a fixed 30-second timeout and dropped timeout metadata from URL call rendering.
- Updated read schema, path utilities, and dispatch to parse selectors from :raw/:L suffixes on path, removing standalone sel usage.
- Changed truncation/error notices to continue with :<nextOffset> and :1 guidance for read and sqlite pagination.
- Added summarizeCode support with tree-sitter summaries, read.summarize settings, and N-API Summary types/exports.
- Added tests and docs updates for path-embedded selectors, summary behavior, and explicit raw/offset SQL/read cases.
- Updated `read` selector parsing for file and URL reads to accept optional leading `L` and `+` count-style ranges.
- Adjusted truncation notices and schema/help text to emit and suggest `sel` offsets without the `L` prefix, including continuation and suggestion messages.
- Updated hashline/output parsing and related tests to recognize the new `sel` formatting in truncation notices.
- Canonicalized file and CLI defaults from `read` to `open` across tool registration and prompts.
- Added `resolveToolAlias()` and applied alias-normalized tool selection so legacy `read` maps to `open`.
- Updated runtime, UI, and export layers to treat `open` as first-class while preserving `read` compatibility.
- Renamed read prompt docs to `open.md`/`open-chunk.md` and refreshed system guidance to recommend `open`.
- Updated tool-related tests and expectations from `read` to `open` (including test fixtures and aliases).
- Standardized missing-file read errors and now return `File not found: <path>` for absent edit targets.
- Centralized AI provider, usage, and OAuth helpers into shared modules to remove duplicated logic.
- Migrated OAuth/API-key login flows to shared factory helpers and removed inline prompt/token-exchange code.
- Reused shared tools and formatter utilities for discovery, stream tails, LSP batching, and source formatting.
- Consolidated repeated test helpers and fixtures into shared modules, replacing inline helper duplicates.
- Reduced default image size limits to 1568px and 500KB to match Anthropic's internal thresholds.
- Optimized screenshot compression with 1024px max dimensions, 150KB budget, and 70% JPEG quality for aggressive payload reduction.
- Added fast-path optimization to skip re-encoding when images fit dimensions and are within 25% of byte budget.
- Refactored image encoding strategy to JPEG-only in quality/dimension reduction loops with improved quality ladder steps.
- Added support for embedded URL selectors with `:raw` and `:L#-L#` line range syntax in read command.
- Implemented `parseReadUrlTarget()` function to parse and validate URL read targets with line range support.
- Updated read CLI to delegate URL inputs through read tool pipeline instead of treating as local file paths.
- Added comprehensive test coverage for URL selector parsing and CLI URL delegation.
- Refactored URL handling in read tool to use structured `ParsedReadUrlTarget` object.
- Replaced all Bun.which() calls with $which() utility from @oh-my-pi/pi-utils across 22 files.
- Removed findBashOnPath() wrapper function from procmgr.ts, consolidating binary path resolution.
- Updated AGENTS.md documentation to reflect new $which() API usage pattern.
- Centralized binary detection logic through shared utility, reducing code duplication.
- Consolidated fetch tool into read tool with URL reading capability and caching support.
- Removed standalone fetch tool from all agent prompts and CLI documentation.
- Extended read tool schema with timeout and raw parameters for URL fetch control.
- Added URL caching mechanism to prevent redundant network requests during read operations.
- Refactored fetch module from class-based tool to standalone executeReadUrl function.
- Updated read tool documentation to describe multi-purpose capabilities including web pages, GitHub, Stack Overflow, Wikipedia, Reddit, NPM, arXiv, blogs, and feeds.
- Replaced Python-based markitdown CLI with native markit-ai library for document and notebook conversion.
- Added support for converting Jupyter notebooks (.ipynb) to markdown via markit-ai integration.
- Created markit utility module with file/buffer conversion wrappers and standardized error handling.
- Removed markitdown from Python tools manager and eliminated external CLI dependency.
- Updated fetch, read, and scraper tools to use new markit conversion API across codebase.
- Added test coverage for ipynb file conversion through markit utility.
- Fixed boolean type coercion in fetch and executor modules by wrapping truncation flags with Boolean() cast.
- Removed maxBytes property from truncation metadata to simplify output metadata structure.
- Normalized optional result properties with explicit fallbacks in output-meta module.
- Updated test expectations to reflect undefined truncation properties instead of false/null values.
- Threaded Settings parameter through renderUrl and renderHtmlToText functions for dependency injection.
- Changed settings import from default export to type-only import in fetch.ts.
- Added vi.clearAllMocks() calls to test setup hooks for proper mock isolation.
- Added Parallel AI provider integration for web search with fast and research modes.
- Added Parallel extract API for URL content and YouTube video extraction with fallback support.
- Added /login parallel command and PARALLEL_API_KEY environment variable authentication.
- Added providers.parallelFetch configuration setting to control Parallel extract usage.
- Integrated Parallel provider into web search priority order between Exa and Kagi.
- Updated HTML-to-text and YouTube scrapers to prefer Parallel extract over fallback providers.
- Corrected buffer-to-base64 conversion by wrapping raw buffers with Buffer.from() to ensure proper encoding across image processing utilities.
- Updated 6 buffer conversion calls in fetch, image-input, and image-resize modules for consistency.
- Removed Kagi Universal Summarizer integration from fetch tool and YouTube scraper.
- Removed `fetch.useKagiSummarizer` configuration setting from settings schema.
- Simplified renderHtmlToText() and renderUrl() functions by removing Kagi summarization fallback logic.
- Fixed indentation inconsistencies in test files from tabs to spaces.
- Moved llms.txt endpoint discovery to fallback strategy when rendered page content is low quality, prioritizing page-specific content over site-wide files.
- Enhanced llms.txt endpoint detection to scope candidates to the requested URL path, searching section-specific files before site-wide ones.
- Replaced getOrigin() with buildLlmEndpointCandidates() to generate path-scoped endpoint candidates with depth-based fallback strategy.
- Updated tryLlmEndpoints() to accept full URL and return endpoint metadata alongside content for better fallback tracking.
- Added 2 integration tests validating section-scoped llms.txt discovery and preference for rendered content over site-wide files.
* idiomatic rust fixes
* idiomatic rust fixes
* display an image if we are fetching an image
* MIME type strictness
* codex nagging me
* codex nagging
* handoff instead of compaction as context filled strategy and surfacing
* handoff instead of compaction as context filled strategy and surfacing p2
* handoff instead of compaction as context filled strategy and surfacing p3
* handoff instead of compaction as context filled strategy and surfacing p4
* handoff instead of compaction as context filled strategy and surfacing p5
* handoff instead of compaction as context filled strategy and surfacing, fixes
* failing fetch test from the fetch tool updates
* handoff focus prompt skeleton
* handoff focus prompt skeleton p2
* fetch bugs
* further codex improvements
* further codex improvements
---------
Co-authored-by: Brit <lol@no.com>
* idiomatic rust fixes
* idiomatic rust fixes
* display an image if we are fetching an image
* MIME type strictness
* codex nagging me
* resize so we do not blow up the terminal we are in
* codex nagging
* codex nagging
* codex nagging
* codex nagging
* codex nagging
---------
Co-authored-by: Brit <lol@no.com>
- Added `fetch.useKagiSummarizer` configuration setting to toggle Kagi Universal Summarizer usage in fetch tool.
- Updated fetch tool to conditionally apply Kagi summarization based on configuration setting.
- Added comprehensive test coverage for Kagi summarizer toggle behavior with mocked dependencies.
- Added Kagi Universal Summarizer integration for URL and YouTube video summarization with fallback support.
- Exported `searchWithKagi` and `summarizeUrlWithKagi` functions from new shared `web/kagi` module for reuse across components.
- Changed HTML-to-text rendering priority to attempt Kagi summarization first before Jina, Trafilatura, and Lynx.
- Refactored Kagi search provider to use shared utilities from `web/kagi` module, reducing code duplication.
- Added `KagiApiError` exception class for Kagi API-specific error handling with optional status code tracking.
- Fixed fetch result rendering to properly wrap long lines instead of truncating them.
- Replaced Bun.wrapAnsi with unified wrapTextWithAnsi utility across debug and output modules.
- Corrected content preview calculation to use actual wrapped line count instead of truncated count.
can1357