- Replaced the `ctok` implementation with the `utok` universal tokenizer supporting multiple model families and UTF text encodings.
- Added tokenizer support and embedding data for Qwen3, DeepSeek V3, Kimi K2, and GLM-5 model variants.
- Added fixture generation scripts, vocabulary packers, and golden test suites for validating tokenization parity.
- Updated dependency requirements and Bazel workspace definitions for new crates and tools.
- Implemented the `ctok` Rust native tokenization engine with offline support for Claude V3, V47, V5, and V5Sonnet families.
- Replaced global token estimation with model-scoped `Tokenizer` instances and provider-anchored transcript accounting across packages.
- Added vocabulary generation scripts, test fixtures, and comprehensive unit tests for tokenizer routing and matching modes.
- Switched Docker images to build native addons via cargo/napi-rs (`OMP_NATIVE_BUILD_BACKEND=cargo`) instead of Bazel.
- Updated Cargo.toml workspace members explicitly to prevent loading errors from stale directories under crates/.
- Added depth-agnostic patterns to .dockerignore files to exclude nested build outputs from Docker build contexts.
- Added OMP_NATIVE_CARGO_PROFILE environment variable to support configuring cargo profiles for addon builds.
- Added Anthropic prompt-cache refresh scheduling and state management to keep prompts warm across idle sessions.
- Updated pricing models and database stats tracking to calculate and store cost-weighted cache savings.
- Integrated cache savings metrics and efficiency displays into the stats CLI, dashboard routes, and UI components.
- Added support for package renaming, manifest pointer tracking, and installation migration during CLI updates.
Runtime: the loader's AVX2 probe used [System.Runtime.Intrinsics.X86.Avx2]
via powershell.exe, which only exists on .NET Core — stock Windows PowerShell
5.1 raised TypeNotFound, so every Windows host silently loaded the baseline
addon and paid ~270ms for the spawn on the startup path. Ask the kernel via
bun:ffi (IsProcessorFeaturePresent(PF_AVX2_INSTRUCTIONS_AVAILABLE), ~0.5ms)
and fall back to pwsh-then-powershell for Node embeds. scripts/host-detect.ts
shared the same broken probe for build-time variant selection.
Build: `bun run build:native` on a win32 host died deep inside the bazel msvc
repo rules (linux/mac exec hosts only, by design). The `host` pseudo-target
now delegates to the local napi build against real VS Build Tools, and other
targets fail fast with guidance. build-bindings.ts resolves the @napi-rs/cli
JS entry from its manifest (the node_modules/.bin shim is a PE launcher Bun
cannot parse) and auto-appends VS Build Tools' bundled CMake/Ninja to PATH
via vswhere so a vcvars prompt is no longer required. Cap maudio at
opt-level=1 to dodge a rustc ICE codegenning MaybeUninit<ma_fence> for
x86_64-pc-windows-msvc under the pinned nightly (Bazel's older pin is
unaffected).
Compile: Bun.Glob.scan yields backslash paths on Windows; the legacy Pi
virtual module used them verbatim for export keys and generated identifiers,
producing invalid JavaScript in compiled-binary builds.
Tests: strip ConPTY negotiation escapes in the pty argv test; ignore the
bazel-oh-my-pi convenience symlink.
build-bindings.ts built the failure error from captured stderr only, but
napi-rs/cargo route much of the failure detail to stdout (e.g. `cargo
metadata exited with code 101 ...`). When stderr was empty the thrown
error collapsed to a bare "napi build failed", hiding the real cause.
Attach the exit code plus tail-capped stdout and stderr sections to the
thrown error so the actionable output survives on the error object.
Fixes#6796
- Replaced the napi-cli/cargo-zigbuild/cargo-xwin/sccache build path with
Bazel: rules_rust + crate_universe over Cargo.lock, hermetic zig cc
toolchains (linux-gnu pinned to glibc 2.17, linux-musl), host Xcode for
darwin, and a repo-local hermetic clang-cl + llvm-ml + xwin toolchain for
windows-msvc (bazel/toolchains/msvc).
- All eight shipped addons build as //:natives-<target> via the release
transition in bazel/defs.bzl (opt, thin LTO, cgu=16, stripped, canonical
.node naming); scripts/bazel-natives.ts is the single driver for local
dev and CI.
- Rust validation moved to bazel test + clippy aspects (strict workspace
policy for opted-in crates, default lints elsewhere, mirroring cargo
semantics) and the rustfmt aspect; cargo stays as the dev-iteration
surface, with brush-core/brush-builtins promoted to workspace members
and excluded from cargo dev tasks to keep their historical scope.
- CI caches through an in-cluster bazel-remote action cache (TLS + basic
auth, cluster-internal only); GitHub-hosted runners never touch the
infrastructure and use an actions/cache-backed disk cache instead.
- Deleted the hand-rolled caching machinery: ci-target-cache,
ci-native-artifact-cache, ci-build-native, native-source-hash,
find-native-artifacts, restore-linux-native, native-prewarm workflow,
ensure-* toolchain actions, and all sccache/Swatinem wiring.
- Warm native rebuilds drop from ~20 minutes to seconds; a cold client
with a warm remote cache rebuilds the linux x64 pair in ~2.5 minutes.
clang-cl enforces per-function target features unlike MSVC, so opus' silk/x86 SSE4.1 units fail under cargo-xwin. The win32 x64 addon floor is x86-64-v2 (SSE4.2 inclusive), so enable it for all C deps via CFLAGS_<target>.
- Added ensure-cmake action installing pinned cmake/ninja on omp-kata pods; audiopus_sys builds bundled libopus via CMake (Ninja for MSVC cross).
- Set CMAKE_POLICY_VERSION_MINIMUM=3.5 globally and in build-native.ts: the bundled opus tree declares cmake_minimum_required below 3.5, which CMake 4.x refuses.
- Dropped the -C target-cpu=native fallback for non-x64 native builds: it baked build-host CPU features into shipped darwin arm64 addons and trips ring 0.17's aarch64-apple const assertion.
- Built the generic Windows release binary with Bun's baseline x64 runtime so older Windows 10 CPUs do not hit the AVX2-only modern executable.
- Forced pi-natives release builds to link PCRE2 statically so macOS installs do not depend on Homebrew's libpcre2 dylib.
- Added release dry-run and native-build regression coverage for the portable artifact contracts.
Fixes#5172
The build script joined PATH entries with ':' (Unix), so Bun.which could not
locate the @napi-rs/cli napi binary on Windows where the separator is ';'.
This blocked native builds and every test importing countTokens (its import
chain eager-loads the .node).
Native linux-x64/arm64 builds moved onto the Ubuntu 24.04 (glibc 2.39)
omp-kata runner. The x64 addon was a plain host build that linked the
runner's glibc and failed to dlopen with `version 'GLIBC_2.39' not found`
on older distros; the arm64 cross-build floated up to GLIBC_2.30. Build
the shipped linux-gnu addons through cargo-zigbuild against a pinned 2.17
floor so they load on any glibc >= 2.17.
- build-native.ts: key the tree-sitter-just `-UNDEBUG` CFLAGS off the
bare triple (cargo-zigbuild strips the `.2.17` glibc suffix before
invoking cargo) and symlink the suffixed target dir napi 3.7.0 expects
to the bare dir cargo-zigbuild writes, so postBuild copyArtifact finds
the cdylib.
- build-native action: add a `glibc` input plus a resolve step deriving
the zigbuild cross_target (suffixed) and the rustup bare_target
(stripped); gate zig/cargo-zigbuild install on cross_target so the
host-arch x64 build still runs native Rust tests.
- ci.yml: GLIBC_FLOOR=2.17 fed to the linux-x64 and linux-arm64 native
jobs.
Re-tags 15.13.1, whose release failed at the linux-x64 binary smoke
before any publish step ran.
- Added repeatable `--tag` argument parsing in `gen-npm-packages.ts` and threaded parsed tags into `generateNpmPackages` for targeted leaf publishing.
- Exported `prepareNativeCorePackage` and expanded package manifest typing to support scripted manifest rewrites used by release workflows.
- Reworked install-test smoke logic to pack the host leaf package, pack the rewritten natives core, and assert the platform leaf package resolves from the core optional dependency.
- CI now enabled OIDC publishing in the build matrix, installed Node 24/npm, and added a per-target native addon publish step.
- The release script now accepts `--native-leaf <tag>` and publishes only the matching generated native leaf package.
- Native package generation gained optional tag filtering with validation of requested leaf tags for targeted release publishing.
- Added `gen-npm-packages.ts` script to generate per-platform leaf packages under `npm/-/`.
- Updated CI release publisher to generate and publish leaf packages before rewriting the core manifest with pinned `optionalDependencies`.
- Core package now ships only JS loader and declarations; installs fetch only the host platform's `.node` binary.
- Added embedded addon tarball output in `embed-native.ts` using `embedded-addons.<platform>.tar.gz` artifacts.
- Added metadata-rich addon types with `size`, `filePath`, and optional `archive` fields.
- Updated extraction to prioritize archive unpacking and skip cached `.node` files when sizes match.
- Added `extractEmbeddedAddonArchive()` with archive parsing and safe validation of archive entry names and kinds.
- Adjusted release profile to disable line-table generation and strip symbols in `Cargo.toml`.
- Added CI-only ELF validation for forbidden sections and regression coverage for issue-823 archive extraction.
- Updated the build-native action to install cargo-zigbuild for non-MSVC targets and cargo-xwin with LLVM tooling for MSVC targets.
- Removed the fixed aarch64 linker variable and narrowed Rust test execution to skip duplicate macOS runs.
- Reworked CI matrices to build win32-x64 artifacts on ubuntu with x86_64-pc-windows-msvc and simplified smoke testing to skip those Windows binaries via matrix gating.
- Added Windows node_modules staging checks so non-compiled addons are copied into versioned native directories.
- Updated loader candidate resolution to prefer embedded/staged/versioned candidates before default runtime paths when staging is enabled.
- Added runtime version-sentinel export __piNativesV15_0_1 and load-time validation to avoid stale binary drift.
- Added typing and release-flow updates to expose staging options and keep per-release sentinel IDs aligned.
Standalone Bun binaries on WSL (and any host where the user moves the
binary away from the build-host's checkout) failed to load
pi_natives.<platform>-<arch>*.node. The loader's isCompiledBinary
detection relied on two signals that are both false in shipped binaries:
process.env.PI_COMPILED (bun --define PI_COMPILED=true substitutes the
bare identifier, not property accesses on process.env) and
__filename.includes("$bunfs") (Bun retains the build-host absolute path
in __filename for required CJS modules — only import.meta.url is
rewritten). Detection therefore returned false, embedded-addon
extraction was skipped, and the only candidates probed were the
build-host nativeDir and execDir.
Make embedded-addon presence the authoritative compiled-mode signal
(it is null in the post-build --reset stub, populated when embed:native
ran for the standalone build), eagerly require the manifest, and
extract candidate-path computation into a pure helper covered by a
host-platform-agnostic unit test. Also fix the build-time --define so
process.env.PI_COMPILED is genuinely set at runtime as a defensive
fallback.
Fixes#823
- Replaced `atom.sed` string syntax with `{pat,rep,g?,F?,i?}` objects in schema, prompts, and tests.
- Changed sed validation defaults so `pat` is regex, `g` defaults to global replacement, and `F:true` forces literal matching.
- Fixed `sed` edge cases by allowing sequential same-anchor edits, handling zero-length matches, and preserving chaining.
- Updated native build flow to local `--profile` defaults, refactored enum generation, and switched totals to `total_ms`.
- Added a `ci` profile to the Cargo workspace that inherits from `release`.
- Updated the native build script to use the `ci` profile for non-local runs and append a ` (ci)` suffix.
- Skipped resolving a managed Cargo target directory when using the local profile.
- Passed the user context when setting the session name during subprocess execution.
- Updated native build scripts to import detectHostAvx2Support from the correct shared module paths.
- Added `/rename <title>` slash command to set explicit session names and update header/tab titles.
- Added `session_name` status segment with hash-derived accent color for session titles.
- Fixed shell execution failure sanitization to preserve all `execResult` fields after redacting stderr.
- Fixed tool execution completion output to pass original `toolResult` text instead of sanitized `content`.
- Stored the process report in a local variable before reading the header for host detection.
- Updated the type assertion so the header path is optional and guarded.
- Kept the Linux zig target selection logic unchanged while fixing the runtime check.
native/index.js does `module.exports = loadNative()` which replaces the
exports object. The generated const enum assignments (`exports.Ellipsis`,
etc.) wrote to the original (now disconnected) exports object, so they
were silently discarded at runtime.
Fix gen-enums.ts to emit `module.exports.X` instead of `exports.X`.
Also remove "type": "module" since the package only has CJS output.
- resolveBuiltAddonPath: napi-rs 3.x emits pi_natives.<platformArchABI>.node
where platformArchABI is e.g. darwin-x64 (no ABI). The previous filter
excluded the plain <platform>-<arch>.node form because it was in the
canonicalFilenames set, breaking darwin-x64 tag builds. Only exclude
sibling variant-tagged files (modern/baseline) that do not match our
target canonical name.
- ci.yml: pass matrix.variants as TARGET_VARIANTS so ci-build-native.ts
actually builds both baseline+modern on x64 platforms in tag releases.
- Added fully-qualified anchor paths in chunk read output showing complete path hierarchy with CRC checksums.
- Added exact chunk selector resolution with CRC validation to enforce canonical chunk paths in edit operations.
- Added `?` selector option to read syntax-aware chunks by canonical path listing when anchor paths are unclear.
- Improved tree-sitter range handling with `end_row_as_line()` function to correctly convert half-open ranges to 1-indexed line numbers.
- Extracted anchor label logic into `chunk_anchor_label()` const function and consolidated style-based selection.
- Removed --dev flag support from native build pipeline and dev:native npm scripts.
- build-native.ts: bunx @napi-rs/cli was picking up Mono's /usr/bin/cli
on Ubuntu runners; resolve the napi bin from node_modules/.bin instead.
- power.rs: non-macOS build strips the cfg body, leaving stop() as a no-op
that clippy wants const. Allow the lint since the macOS branch can't be const.
- Migrated native bindings from TypeScript wrappers to NAPI-RS generated modules with auto-generated type definitions and runtime enums.
- Replaced chunk tree API with stateful ChunkState class supporting render, edit, and resolve operations with improved error handling.
- Converted callback signatures to error-first pattern (error, result) for shell, PTY, glob, and grep operations.
- Introduced type-safe enums for MacOSAppearance, GrepOutputMode, KeyEventType, ImageFormat, and AstMatchStrictness replacing string literals.
- Refactored chunk tree implementation with dedicated modules for edit, indent, resolve, and state management with comprehensive validation.
- Moved clipboard utilities from native bindings to coding-agent package with improved OSC 52 and Termux compatibility.
- Added `new_segment` parameter to `init_experiment` to force new segment creation when contract fields match.
- Added `skip_restore` parameter to `log_experiment` to preserve working tree state and pre-existing uncommitted changes.
- Changed `log_experiment` to revert only run-modified files instead of entire working tree, preserving user changes.
- Changed `init_experiment` to detect matching contract fields and skip re-initialization unless `new_segment=true`.
- Added git status parsing utilities to track pre-run dirty paths and distinguish tracked vs untracked modifications.
- Removed secondary metrics validation requirement, making secondary metrics informational only.
- Migrated timing instrumentation from custom utils/timings module to centralized logger.time() and logger.timeAsync() methods across coding-agent and natives packages.
- Removed PI_DEBUG_STARTUP environment variable and debugStartup() function, consolidating debug output to logger.debug() for conditional logging.
- Replaced custom TextDecoder buffer decoding with native toString('utf-8') method in natives package for simplified string conversion.
- Added RingBuffer<T> circular buffer implementation to pi-utils with comprehensive test coverage for efficient queue operations.
- Enhanced logger interface with time() and timeAsync() methods providing unified operation timing and performance tracking across packages.
- Added support for braced escape sequences (\p{...}, \P{...}, \x{...}, \u{...}) in regex pattern sanitization to properly preserve unicode property escapes and hex escapes.
- Improved LSP client message writing by simplifying header encoding and refactoring abort signal handling with the untilAborted utility.
- Enhanced native addon installation script to dynamically fetch release tags from GitHub API instead of using hardcoded versions.
- Relaxed native addon validation to only error when no addons are found rather than when any are missing, improving flexibility in addon availability.
- Added CPU variant support for x64 native addons with modern (AVX2/x86-64-v3) and baseline (x86-64-v2) variants, enabling automatic fallback when modern variants are unavailable.
- Added automatic AVX2 CPU detection on Linux, macOS, and Windows platforms to select appropriate native addon variant at runtime.
- Updated native addon filename scheme to include CPU variant suffix (e.g., pi_natives.linux-x64-modern.node) for x64 platforms.
- Updated CLI update mechanism to support downloading and installing multiple native addon variants per platform with fallback support.
- Removed fallback untagged pi_natives.node binary creation; platform and variant-tagged binaries are now required.
- Added RUSTFLAGS environment variable configuration to CI workflow for x86-64-v3 CPU optimization across multiple build jobs.
- Added per-matrix rustflags configuration for native builds on x64 platforms (Linux, macOS Intel, Windows).
- Updated build-native script to default to native CPU optimization for local builds when not cross-compiling.
- Replaced ASCII ellipsis characters (three dots '...') with Unicode ellipsis character ('...') throughout the codebase for improved typography.
- Adjusted string truncation logic to account for single-character Unicode ellipsis instead of three-character ASCII ellipsis, reducing reserved space from 3 to 1 character in truncation calculations.
- Updated truncation offsets in multiple files (session-manager, agent, executor, footer) to preserve 2 additional characters before ellipsis due to more compact Unicode representation.
- Migrated all environment variable access from Node.js `process.env` to Bun runtime `Bun.env` API across the entire codebase.
- Updated 71 files including source code, tests, documentation, and configuration to use Bun's native environment variable API.
- Added comprehensive environment variables documentation in `packages/coding-agent/docs/environment-variables.md` covering 100+ environment variables organized by category.
- Updated CHANGELOG entries to reflect migration from `process.env` to `Bun.env` and environment variable prefix changes (PI_* vs OMP_*).
- Maintained identical behavior and logic across all changes - only the environment variable access method was updated.
can1357