The two search()-level cancel tests (`fallback_walk_observes_cancel_flag`
/ `fast_walk_observes_cancel_flag`) pre-set the cancel flag before
invoking search(), but search() and try_search_fast each have a
pre-loop guard that breaks out on cancelled before either walker call
is invoked. That means both tests still pass with the walker call
sites reverted to no-op heartbeats, so they don't defend the fix.
Replace them with symmetric walker-level tests that drive the two
walker APIs fd uses:
- `cancel_heartbeat_aborts_collect_with_heartbeat` — the fallback
path's `collect_with_heartbeat` call; asserts WalkError::Interrupted
(was `cancel_heartbeat_aborts_walker_when_flag_is_set`, renamed for
API-specific clarity and given a filler-file seed so the outcome
shows a real drain on regression).
- `cancel_heartbeat_aborts_for_each_entry_with_heartbeat` — the fast
path's streaming API; asserts WalkError::Interrupted AND that the
visitor never received any entry.
Both fail against the pre-fix no-op heartbeat with WalkStatus::Complete
/ a fully drained WalkOutcome. Keep
`walk_completes_normally_when_cancel_flag_is_unset` as the positive
regression pin — that path does exercise search() end to end because
the outer guard passes with cancelled=false.
The in-process fd builtin passed no-op heartbeats to pi_walker for
both its gitignore-respecting fallback path (`collect_with_heartbeat`
in `search`) and its fast path (`for_each_entry_with_heartbeat` in
`try_search_fast`), so cancellation of a large or slow directory walk
was deferred until traversal completed. The shell wrapper flips the
shared `AtomicBool` cancel flag when the runtime cancellation token
fires and then awaits the blocking task; with no heartbeat hookup the
walker had no way to observe the flag mid-walk and kept collecting the
whole tree before the wrapper could return exit 130.
Introduce `cancel_heartbeat(&AtomicBool)` — the walker-level heartbeat
that returns `io::ErrorKind::Interrupted` when the flag is set — and
plug it into both walker calls. Both call sites recognize the resulting
`WalkError::Interrupted` alongside `cancelled` and break silently
instead of surfacing an `fd:` diagnostic on stderr; the shell wrapper
owns the user-visible exit code.
Regression cover: a walker-level test pre-sets the cancel flag and
asserts `collect_with_heartbeat(cancel_heartbeat(&flag))` surfaces
`WalkError::Interrupted` instead of collecting the tree; two
higher-level `search` tests exercise the silent break for both the
fallback and fast paths; a fourth pins the non-cancelled contract so
the added heartbeat can't stall normal searches. Neuter the helper to
a no-op and the walker-level test fails with the pre-fix `WalkOutcome`
showing every entry scanned — the exact bug the issue reports.
Fixes#3949
Previously each retained match cloned an Arc to the whole source, keeping large generated/vendor files alive until the scan finished. Now the retained heap stores just the per-match matched text, so the retained payload stays bounded to the small span even when a broad page window keeps a few matches from very large files.
Refs #3935
Retained only the requested AST search page window in native ast_grep/ast_match and the coding-agent multi-target wrapper while preserving exact totals.
Fixes#3935
rg --files used collect_filtered_files for directory operands. When a
walker heartbeat observed the uutils cancellation flag, the helper
returned Ok(Vec::new()), making the caller treat the aborted walk like an
empty directory.
Return an error for cancelled collection and let rg --files / sorted rg
callers map it to a silent had-error outcome, stopping later operands
without writing post-cancel output.
Fixes#3933
Recursive grep and rg already stopped the active directory walk when the
uutils scope cancel flag was observed, but the outer operand loop could
still continue into later regular-file operands before the shell wrapper
returned 130.
Stop subsequent grep, rg, and rg --files operands once cancellation is
set. Extend the cancellation regressions with a later regular-file
operand so the tests cover the exact post-directory continuation case.
Fixes#3933
The grep and rg in-process builtins passed no-op heartbeats to pi_walker
when recursing into a directory, so the uutils scope's cancel flag was
ignored mid-walk. The shell wrapper sets that flag on abort/timeout and
then awaits the blocking task — with no heartbeat hookup, a cancelled
recursive grep/rg waited for the whole tree to be scanned before the
shell could return exit 130.
Add pi_uutils_ctx::is_cancelled() and have grep's search_dir, rg's
search_dir, and rg's collect_filtered_files supply heartbeats that
return io::ErrorKind::Interrupted when the flag is set. The walker maps
that to WalkError::Interrupted, which the utilities now silently treat
as a harness cancellation (no spurious 'native directory scan
interrupted' on the command's stderr — the shell wrapper owns the
user-visible exit code).
Regression tests pre-set the cancel flag and assert grep -r /
rg / rg --files exit without scanning the matching file in the tree.
Fixes#3933
- Added `#streamTurnNonce` to prevent aborted streaming turns from corrupting content indexes of subsequent messages.
- Implemented temporary stream-key generation using content position and turn nonces for previewing tool calls without native IDs.
- Added migration logic to key pending tool previews by their real ID and rekey `ToolArgsRevealController` once the real ID is parsed.
- Removed the ignore-based fallback walker implementation and its associated dependencies.
- Simplified the directory traversal to consistently use native scanners across all platforms.
- Eliminated `WalkError::Unsupported` and `WalkStatus::Unsupported` error handling across all dependent crates.
- Added a depth-first sorting fallback for native traversal and implemented recursion detection using a symlink stack.
- Replaced standard and tokio mpsc channels with flume channels across workspace crates to simplify thread synchronization.
- Swapped standard Mutex guards for parking_lot Mutexes to avoid manual lock poisoning handling and improve performance.
- Declared workspace-wide dependencies for flume and parking_lot in root and member Cargo manifests.
- Introduced a two-pass file processing architecture with `ReadPolicy` and `FileOutcome` state tracking.
- Deferred oversized files to a second pass where only their leading segment is read and searched.
- Integrated the two-pass processing logic into both sequential and native parallel grep execution paths.
- Updated agent tool definitions and user-visible messages to reflect partial coverage of large files instead of skipping them.
- Replaced custom fast-walk and fs-cache implementations in pi-natives with a new dedicated pi-walker library.
- Integrated the thread-safe, parallel pi-walker library across pi-natives, pi-shell, pi-uu-grep, and uu-find.
- Rewrote file search, fuzzy finding, and glob-matching logic to leverage pi-walker configurations and visitor traits.
- Optimized shell process tracking in pi-shell by replacing global descendant-diff logic with an isolated, per-run SpawnRegistry.
- Added parallel rayon-based walking and optimized fast paths for directory scanning and entry classification.
- Added a `SpawnObserver` trait to intercept freshly spawned external command PIDs and process group IDs.
- Exposed `set_spawn_observer` and `spawn_observer` on `ExecutionParameters` to allow tracking process lifecycles.
- Triggered the observer on successful external command execution unless the process is detached or reparented.
- Added a platform-native fast filesystem traversal implementation leveraging `getattrlistbulk` on macOS, `getdents64` and `statx` on Linux, and `NtQueryDirectoryFile` on Windows.
- Integrated the fast traversal path as a preferred, high-performance fallback before standard walk operations in both cache collection and streaming search routines.
- Consolidated grep search logic by extracting a unified file-matching helper to support both standard and fast-walk execution paths.
- Introduced platform-specific Windows system dependency configurations and unit tests validating correct hidden-file filtering during scanning.
- Added Silver.ttf TrueType font support to `pi-natives` with automated fallback logic for bitmap font rendering.
- Implemented wide code point detection and cell-width calculation to improve CJK character handling and layout.
- Introduced dynamic font-aware preflight probing via `resolveShapeForText` and `renderabilityProbeText` for better font selection.
- Enabled semantic emoji folding and improved text normalization to handle non-Latin characters and emoji filtering.
The in-process grep builtin in pi-shell (backed by pi-uu-grep) shadowed /usr/bin/grep with a clap subset that rejected three universal GNU-grep flags. The common bashrc alias 'grep --color=auto' guaranteed that bare 'grep' in any pipeline failed with exit 2, and probes like 'grep --version' from shell startup scripts errored out the same way.
Add --color[=WHEN] (alias --colour) as an accepted-and-ignored flag — the builtin writes to in-process file descriptors, never a TTY, so injecting ANSI escapes would corrupt downstream consumers. Add --version routed through clap so it lands on the context stdout via the same path as --help.
Fixes#3755
- Switch build profile to `panic = "unwind"` to allow catching panics in vendored uutils code.
- Add `run_caught` to wrap utility execution in `catch_unwind`, converting panics into non-zero exit codes.
- Update `pi-natives` to distinguish between recoverable utility panics and fatal process crashes, preventing recovered panics from appearing in user-facing crash reports.
- Propagated IO errors from the bounded tail internal logic to avoid panics.
- Updated `print_target_section` to return `io::Result` instead of unwrapping.
- Added a test case to verify that `BrokenPipe` is correctly surfaced when the consumer closes the pipe.
- Implement an argv-independent `format_usage` within `pi-uutils-ctx`.
- Update all vendor crates to import the function from the new location.
Fixes#3727
- Added a fallback mechanism to treat invalid regex patterns as literal search strings instead of aborting.
- Implemented a targeted recovery for unclosed or extra parentheses that attempts to escape only the problematic characters while preserving active regex syntax.
- Updated `pi-uu-grep` to perform pattern-specific fallback when multiple search strings are provided.
- Added regression tests to verify graceful degradation for invalid patterns alongside standard regex operation.
- Added a fallback mechanism that recovers from invalid regex patterns by treating them as literals instead of failing.
- Enabled this behavior by default, allowing valid regex alternatives to retain their functionality when mixed with invalid patterns.
- Updated the extended regex flag to enforce strict syntax parsing and return errors for invalid patterns.
- Added an `fd` command-line utility for searching the filesystem.
- Implemented file filtering capabilities for type, size, modification time, and ownership.
- Integrated directory traversal and ignore-pattern support for search optimization.
- Registered the builtin in the shell and added comprehensive integration tests for filtering and globbing.
- Implemented `rg` shell builtin using ripgrep libraries to enable file and directory searching.
- Added `RgSink` to handle output formatting, context, and vimgrep compatibility.
- Updated shell context and IO flags to support stdin as search input.
- Added integration tests for directory recursion, ignore filtering, and stdin handling.
- Added tree-sitter markdown support to resolve headings into full sections in `pi-ast`.
- Enabled block operations (`SWAP.BLK`, `DEL.BLK`, `INS.BLK.POST`) on markdown headings so they encompass the entire section, including nested deeper headings.
- Updated system prompt to guide agents in using structured markdown heading edits for plans.
- Fixed `plan-mode-guard` to correctly resolve local protocol options for subagents.
- Added support for `--quiet` (`-q`) and `--line-regexp` (`-x`) to the `grep` builtin.
- Enabled short-circuiting behavior for `-q` to suppress output and return early on the first match.
- Configured exit status logic to prioritize successful matches over error states when using `-q`.
- Added integration tests to verify correct exit status codes and line anchoring behavior.
can1357