Addresses a broad audit of built-in shell utilities against their real counterparts: timeout gains signal delivery, -s/-k/--preserve-status/--foreground/-v, and GNU exit codes; diff defaults to normal format and gains -w/-b/-B/-i/-c/-x/-L/-s/--strip-trailing-cr and proper -r gating; find fixes -newerXY timestamp comparison direction, anchors -regex to whole paths, and gains BSD -perm +mode, -type lists, -size T/P suffixes, -E/-x/-s flags; date gains BSD -r epoch, -v adjustments, -j -f strptime, and non-greedy -I; tail/head accept obsolete -N/+N at any position with any file count; rg resolves case flags by last occurrence and gains --path-separator and clean -0 output; stat prints integer epochs for %X/%Y/%Z and gains BSD -s/-x/-t; cksum is registered as a builtin; truncate implements -o/--io-blocks and b/= size suffixes; sleep/timeout accept infinity; yes/errno/kill accept hyphen-prefixed operands; nohup -- cmd no longer runs --; which gains BSD -s.
- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.
- Implemented the `ctok` Rust native tokenization engine with offline support for Claude V3, V47, V5, and V5Sonnet families.
- Replaced global token estimation with model-scoped `Tokenizer` instances and provider-anchored transcript accounting across packages.
- Added vocabulary generation scripts, test fixtures, and comprehensive unit tests for tokenizer routing and matching modes.
bazel test sandboxes stage only the crate's declared inputs, so the repository sweep found zero sources and tripped its own evidence guard ('corpus sample too small to be evidence: 0'). Skip on a missing or empty corpus; any non-empty scan still enforces the >40-file evidence floor.
std::env::vars() panics the moment a host env key or value is not valid
Unicode, before any command can run. A corrupt GHOSTTY_BIN_DIR (bytes 9d d9 50)
staged by cmux/Ghostty tripped both sinks:
- pi-shell's session env copy in create_session_for_run (also merged PATH)
- brush-core's get_host_env_vars, which process builtins (sleep, timeout,
pgrep, ...) use to inherit the host env into the shells they build
Both now read via std::env::vars_os() and skip entries that cannot be decoded
as Unicode: a corrupt entry carries no usable meaning. PATH merge behavior is
unchanged. Regression tests inject a non-UTF-8 key and value and assert the
shell still starts and PATH survives.
Reported in issue #8925.
collect_boundaries walked every node in the file even though the answer is
bounded by the visible window, which cost roughly twice the parse: on an
81KB source the walk was 8.95ms against a 4.32ms parse, and on 1MB it was
138.8ms against 87.6ms.
A node contributes a boundary only when one of its own endpoint lines is
visible, and both of those lines lie inside its raw row span. Every
descendant's span is contained in its ancestor's, so a span holding no
visible line rules out that node and everything beneath it. Skip such
subtrees with a binary search over the merged visible ranges.
The test is the raw span, not endpoint visibility: a node whose span merely
straddles the window has both endpoints outside it yet can contain a child
that opens exactly on a visible line. The raw span is also conservative
relative to node_content_end_line, so the prune needs no reasoning about
that newline adjustment.
Equivalence is proven differentially rather than argued: the pre-prune walk
is retained under cfg(test) and compared for exact Option<Vec<u32>> equality
across 4827 .ts/.py/.rs files and 38,616 comparisons over eight window
shapes, including whole-file-visible, past-EOF, disjoint ranges, empty range
lists and files that fail to parse. Zero mismatches. root.has_error() is
still evaluated on the whole tree before the walk, so pruning cannot change
a None verdict.
Measured on the built addon with a mid-file 40-line window, medians of 20,
against the parse cache alone: 81KB 13.4ms -> 4.45ms cold and 9.04ms ->
0.149ms warm; 1.06MB 188.1ms -> 55.8ms cold and 131.7ms -> 0.440ms warm.
`enclosing_block_boundaries`, `block_range_at` and `summarize_code` each
re-parsed the whole file on every call. The results are not cacheable —
boundaries depend on the caller's visible ranges, which differ per call —
but the `tree_sitter::Tree` is, so cache that instead and hand out
`ts_tree_copy` clones.
Keyed on (xxh64 of the source, source length, language). The hash is a
bucket selector only: a hit re-verifies the stored source against the
request byte-for-byte before returning the tree, so a collision costs a
re-parse and can never yield a tree built from other content. Language is
in the key because the same bytes parsed as TypeScript and as Python are
different trees.
Bounded at 12 slots and 4 MiB of retained source with LRU eviction;
sources above 4 MiB are parsed but never retained. `Tree` is `Send` but
not `Sync`, so entries sit behind a `Mutex` that is held only for a map
probe, a byte compare and a refcount bump, never across a parse or walk.
Error trees are cached like any other: `has_error()` is a property of the
tree, so the callers' own checks reach an identical verdict from a cached
tree, and repeated "does this parse" probes get the speedup too.
Measured (M4 Max, bazel-built .node, median of 20, 1-40 visible):
read.ts 81 KB 13.34 ms -> 8.86 ms on repeat; 1 MB synthetic 225.7 ms ->
138.3 ms. Parser::new + set_language measured at 0.30 us against a
3.91 ms parse, so no parser pooling.
The default snapcompact frame fonts (X.org 8x13 for every provider, plus the selectable 6x12 and legacy 5x8) drew digit zero as a bare oval visually indistinguishable from letter O. Image-based compaction OCRs the frames back, so 0 and O were mixed up and compacted identifiers (e.g. Slack IDs) got corrupted.
Zero now carries a disambiguating interior mark the O lacks: an ascending slash in 8x13 and a center bar in 6x12/5x8. unscii-8 (8x8/6x6u shapes) already shipped a slashed zero and is unchanged.
Fixes#8713
- Added live tracking and stale status warnings for agent activity snapshots.
- Fixed text wrapping with ANSI escape sequences to defer style open sequences after whitespace.
- Added VirtualRenderScheduler for deterministic virtual-clock rendering tests.
- Kept ANSI sequences after visible content with the current wrap token so closing resets cannot migrate into discarded whitespace.
- Added a regression for a codespan ending exactly at the wrap width.
Fixes#8582
- Replaced the custom MuPDF-WASM PDF extraction and rendering pipeline with the new `pdfToMarkdown` native function from `@oh-my-pi/pi-natives`.
- Removed legacy MuPDF extraction modules, WASM embedding scripts, and PDF image extraction tools.
- Added OCR warnings and browser/text redirection for unsupported PDF image reads.
- Updated native package definitions, documentation, and test suites for the new PDF inspection capability.
-[DCDevice isSupported] synchronously opens an XPC connection to the per-user DeviceCheck metadata daemon, which exists only in an interactive GUI login session. From a session without graphic access (SSH, launchd LaunchDaemon, CI runner, service account, sandbox) the connection setup hits _xpc_api_misuse and aborts the process with SIGTRAP before any completion handler runs, so the promise never rejects and every openai-codex/* OAuth model becomes unusable.
Check the caller's security session for the sessionHasGraphicAccess attribute via SessionGetInfo before touching DeviceCheck; resolve { supported: false, error } when it is absent, mirroring the non-macOS stub and letting the caller send an error-coded attestation instead of dying.
Fixes#8353
The shipped win32-x64 pi_natives addon linked the dynamic MSVC CRT (/MD)
and imported VCRUNTIME140.dll from the Visual C++ Redistributable, which
is absent on a clean Windows install. LoadLibrary of the extracted .node
then failed with error 126 ("The specified module could not be found"),
so omp could not start after a fresh `irm install.ps1 | iex`.
Static-link the CRT for the win32 addon: +crt-static for rustc (crate
BUILD select) plus the static_link_msvcrt cc feature enabled for win32 in
the native_addon transition, so its C deps (opus/cmake, tree-sitter,
blake3, ring) compile /MT in lock-step. The rebuilt .node imports only
core Windows system DLLs -- no VCRUNTIME140.dll, no api-ms-win-crt-*.
Fixes#8439
- Add comprehensive Nix flake definitions, derivations, modules, and CI workflows.
- Update tests and executables to resolve binaries from PATH rather than absolute paths.
- Ensure byte reproducibility and zeroed timestamps in embedded dashboard archives.
- Add handling for Nix-managed installations in CLI update checks.
Routed sed -f script files and the in-script r/w/s///w file paths through brush-core's shell path normalizer, so /c and /mnt/c aliases open the live Windows drive instead of a phantom current-drive path. Added a Windows -f regression.
Fixes#8355
Translated the /c and /mnt/c tail per char over the valid UTF-8 suffix instead of copying raw bytes as chars, so non-ASCII path components no longer mojibake. Removed the now-unused byte separator helper and added a non-ASCII regression.
Fixes#8355
Normalized utility operands through brush-core's shared shell path resolver so /c and /mnt/c aliases address the live Windows drive. Added Windows-only regression coverage at the Host boundary.
Fixes#8355
The leak regression waited only 100ms while the leaked job could not write its marker until a 1s sleep elapsed, so the pre-fix path passed vacuously. Wait past the delay; verified the test fails when the drop-time abort is neutralized.
Fixes#8341
Abort shell-internal background tasks when their owning session is dropped, and propagate task abortion into blocking utility cancellation so infinite writers stop.
Fixes#8341
- Synced pi-builtins bazel crate_features with cargo's resolved default
set: bazel features are literal, so the meta-features never expanded
and the procs/rg cluster (nohup, pgrep, pidwait, pkill, proc-match,
ps, rg, sleep, timeout, top) was silently compiled out, leaving the
process builtins unregistered under bazel and failing pi-shell tests.
- Repaired windows compilation of pi-builtins: cfg-gated the
uucore::mode import in mkdir, imported std::env in sort's non-unix
locale probe, mapped ProcInfo::pid through a closure in kill, brought
MetadataExt into scope in wc, and replaced stat's unstable
windows_by_handle metadata with a stable GetFileInformationByHandle
query (volume serial, link count, file index, no-dereference aware).
- Imported HashSet for pi-shell's windows-only PATH merge.
- Added a clippy-ported bazel config + CI bucket so pi-builtins keeps
its manifest-declared clippy allows under the bazel aspect while rustc
warnings stay denied, and zeroed the remaining windows-target rustc
warnings (unused params/imports in find, mv, rm, proc_match, ps).
- Added util.procs to the bazel crate_features: cargo resolves the
builtin.kill -> util.procs implication automatically, but bazel
crate_features are literal, so kill.rs failed with E0432 on
proc_snapshot once HostProcesses became unconditional.
- Imported std::os::fd::AsFd in the linux/android splice path of the wc
builtin (E0405); the import is target-gated like its callers.
- Verified with cargo check -p pi-builtins --no-default-features using
the exact bazel feature list on both the host and (via zig cc)
x86_64-unknown-linux-gnu targets.
- Add minimum character threshold to bypass minimization for short outputs.
- Add preserve-if-empty configuration and pipeline support for filters.
- Update test fixtures and integration tests to meet length thresholds.
- Added `HostProcesses` snapshot and `ChainNode` validation to track ancestry and prevent pid recycling.
- Updated process matching and signal handling to refuse signalling the shell or its ancestor processes.
- Added regression tests verifying that kill builtins safely block ancestor targeting while permitting unrelated processes.
- Added the `smallvec` dependency to support efficient process snapshot tracking.
clippy-strict (nursery redundant_pub_crate, -D warnings) rejects pub(crate)
items inside the crate-private device module; plain pub is equivalent there.
Applied across all platform backends since CI only lints the linux cfg.
can1357