- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.
- Implemented the `ctok` Rust native tokenization engine with offline support for Claude V3, V47, V5, and V5Sonnet families.
- Replaced global token estimation with model-scoped `Tokenizer` instances and provider-anchored transcript accounting across packages.
- Added vocabulary generation scripts, test fixtures, and comprehensive unit tests for tokenizer routing and matching modes.
`enclosing_block_boundaries`, `block_range_at` and `summarize_code` each
re-parsed the whole file on every call. The results are not cacheable —
boundaries depend on the caller's visible ranges, which differ per call —
but the `tree_sitter::Tree` is, so cache that instead and hand out
`ts_tree_copy` clones.
Keyed on (xxh64 of the source, source length, language). The hash is a
bucket selector only: a hit re-verifies the stored source against the
request byte-for-byte before returning the tree, so a collision costs a
re-parse and can never yield a tree built from other content. Language is
in the key because the same bytes parsed as TypeScript and as Python are
different trees.
Bounded at 12 slots and 4 MiB of retained source with LRU eviction;
sources above 4 MiB are parsed but never retained. `Tree` is `Send` but
not `Sync`, so entries sit behind a `Mutex` that is held only for a map
probe, a byte compare and a refcount bump, never across a parse or walk.
Error trees are cached like any other: `has_error()` is a property of the
tree, so the callers' own checks reach an identical verdict from a cached
tree, and repeated "does this parse" probes get the speedup too.
Measured (M4 Max, bazel-built .node, median of 20, 1-40 visible):
read.ts 81 KB 13.34 ms -> 8.86 ms on repeat; 1 MB synthetic 225.7 ms ->
138.3 ms. Parser::new + set_language measured at 0.30 us against a
3.91 ms parse, so no parser pooling.
- Replaced the custom MuPDF-WASM PDF extraction and rendering pipeline with the new `pdfToMarkdown` native function from `@oh-my-pi/pi-natives`.
- Removed legacy MuPDF extraction modules, WASM embedding scripts, and PDF image extraction tools.
- Added OCR warnings and browser/text redirection for unsupported PDF image reads.
- Updated native package definitions, documentation, and test suites for the new PDF inspection capability.
- Added `HostProcesses` snapshot and `ChainNode` validation to track ancestry and prevent pid recycling.
- Updated process matching and signal handling to refuse signalling the shell or its ancestor processes.
- Added regression tests verifying that kill builtins safely block ancestor targeting while permitting unrelated processes.
- Added the `smallvec` dependency to support efficient process snapshot tracking.
- Replaced the miniaudio dependency with custom OS audio device abstractions and backends.
- Implemented platform-specific audio playback and capture for macOS (Audio Queue), Windows (WASAPI), and Linux (PulseAudio/ALSA).
- Added a fallback stub backend returning errors for unsupported platforms.
- Updated audio stream handling with reliable fill guard wakeups and streamlined rate validation.
- zune-jpeg 0.5.15 (image 0.25's JPEG decoder) cannot compile with its
non-default log feature off: zune-core's no-log warn! stub is not
expression-safe. A feature-activation-only workspace dep on
zune-jpeg { features = ["log"] } fixes the cold build; log stays 0.4.33.
- model-registry-default-config's local ModelSnapshot type gains the optional
streamIdleTimeoutMs the Bedrock watchdog compat now emits.
- Tracked pointer and keyboard grants from the RemoteDesktop response.
- Drained asynchronous EIS announcements after the first resumed device.
- Covered GNOME's keyboard-before-pointer ordering with a regression test.
Fixes#7926
- Replaced monolithic desktop native bindings and action batching with a modular cross-platform backend structure supporting Wayland, X11, macOS, and Win32.
- Updated the computer tool schema and supervisor to execute persistent JavaScript script runs with timeout clamping and asynchronous tool calling.
- Integrated accessibility (AX) tree snapshotting, node querying, and bounds-based hit testing across platform desktop layers.
- Added native clipboard bindings and updated coding-agent prompts, renderers, and tests to validate script-based computer workflows.
- Added window discovery, targeting, and frame validation for macOS, Windows, and Linux platforms.
- Updated computer tool and protocol definitions to support listing and targeting specific application windows.
- Refactored native computer tool exposure to function-only representation across models.
- Updated documentation and test suites to reflect window-scoped computer control capabilities.
html-to-markdown-rs 2.30 could recurse through pathological HTML until
a native stack overflow aborted the entire OMP process. Upstream 3.9.2
bounds those traversals and reports omitted subtrees as a
machine-readable DepthLimitExceeded warning.
Upgrade html-to-markdown-rs to 3.9.2. Upstream treats a depth-limited
conversion as a successful partial Markdown result plus a warning;
deliberately promote that warning to a rejected `htmlToMarkdown`
promise, allowing the Read tool to fall back without terminating OMP.
Normal conversions and unrelated warnings keep their existing behavior.
Add a rejection-contract test plus a child-process regression proving
OMP survives deeply nested and malformed input.
- Extracted audio capture and playback implementations, along with the WebRTC peer engine, from `pi-natives` into a new `pi-voice` library crate.
- Updated `pi-natives` bindings to consume the extracted `pi_voice` audio streams and live peer core.
- Added release validation gate jobs, parallelized Linux binary builds, and introduced a concurrent macOS release build job in the CI workflow.
- Updated Bazel workspace configurations, Cargo manifests, and documentation to include the new `pi-voice` crate and its dependencies.
The 17.1.6 Bazel migration dropped maudio's generate-bindings feature, so
the darwin addon shipped maudio-sys's Linux-shaped pregenerated miniaudio
bindings. Those bindings omit the MA_SUPPORT_COREAUDIO backend-state union
members that the macOS build of miniaudio.c actually contains, producing a
Rust/C ABI mismatch: capture callbacks saw zero channels and yielded no PCM.
Re-enable generate-bindings for apple targets only. Patch maudio-sys's build
script to branch on Cargo's TARGET rather than the build-script host, so a
macOS-hosted Linux/Windows cross-build keeps target-family pregenerated
bindings instead of incorrectly running Darwin bindgen. Repin the Bazel graph
to apply the patch and gate bindgen behind the apple target selects.
Fixes#6846
- Replaced the napi-cli/cargo-zigbuild/cargo-xwin/sccache build path with
Bazel: rules_rust + crate_universe over Cargo.lock, hermetic zig cc
toolchains (linux-gnu pinned to glibc 2.17, linux-musl), host Xcode for
darwin, and a repo-local hermetic clang-cl + llvm-ml + xwin toolchain for
windows-msvc (bazel/toolchains/msvc).
- All eight shipped addons build as //:natives-<target> via the release
transition in bazel/defs.bzl (opt, thin LTO, cgu=16, stripped, canonical
.node naming); scripts/bazel-natives.ts is the single driver for local
dev and CI.
- Rust validation moved to bazel test + clippy aspects (strict workspace
policy for opted-in crates, default lints elsewhere, mirroring cargo
semantics) and the rustfmt aspect; cargo stays as the dev-iteration
surface, with brush-core/brush-builtins promoted to workspace members
and excluded from cargo dev tasks to keep their historical scope.
- CI caches through an in-cluster bazel-remote action cache (TLS + basic
auth, cluster-internal only); GitHub-hosted runners never touch the
infrastructure and use an actions/cache-backed disk cache instead.
- Deleted the hand-rolled caching machinery: ci-target-cache,
ci-native-artifact-cache, ci-build-native, native-source-hash,
find-native-artifacts, restore-linux-native, native-prewarm workflow,
ensure-* toolchain actions, and all sccache/Swatinem wiring.
- Warm native rebuilds drop from ~20 minutes to seconds; a cold client
with a warm remote cache rebuilds the linux x64 pair in ~2.5 minutes.
The vendored uu-stat routed every non-Unix invocation to a stub that
printed "stat: unsupported on this platform" and exited 1, so the bash
tool's stat builtin was dead on native Windows even though it was still
advertised and registered.
Add a Windows-native backend that reimplements the GNU stat directives on
top of std::fs::Metadata, the windows_by_handle metadata extensions
(inode, hard-link count, and device via GetFileInformationByHandle), and
the Win32 volume APIs for --file-system mode. Permission bits, file type,
and timestamps are synthesized from Windows attributes; ownership fields
that have no Windows equivalent report 0/UNKNOWN. The format parser,
printer, and BSD-compat layer are now shared across platforms and Unix
behavior is unchanged.
Fixes#6723
- Added six builtin commands: ts, sponge, ifne, isutf8, combine, and errno to pi-shell.
- Created moreutils module with independent implementations for all tools.
- Added jiff dependency for timestamp and timezone functionality.
- Integrated builtins into shell execution pipeline with in-process execution.
- Added integration tests verifying pipe chains like ts | sponge with isutf8.
can1357