smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into feat/security-native

Browse Source 
# Conflicts:
#	packages/coding-agent/src/tools/index.ts
This commit is contained in:
Kyle McCleary
2026-07-29 23:26:22 -07:00
parent 0b968bbb49 c3d1586740
commit 4337797565
129 changed files with 5901 additions and 2408 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.bazelrc
+2 -1
View File
@@ -22,7 +22,8 @@ build --@rules_rust//rust/toolchain/channel=nightly
# --- Rust validation (replaces cargo clippy/nextest in CI) --------------------
# Mirrors cargo semantics: crates with `[lints] workspace = true` (pi-ast,
# pi-iso, pi-natives, pi-shell, pi-walker) get the strict workspace policy
# pi-iso, pi-natives, pi-shell, pi-voice, pi-walker) get the strict workspace
# policy
# (clippy-strict; bazel/clippy.bazelrc is generated from Cargo.toml); everything
# else gets default clippy + -Dwarnings (clippy).
build:clippy --aspects=@rules_rust//rust:defs.bzl%rust_clippy_aspect
.github/actionlint.yaml
+6
View File
@@ -0,0 +1,6 @@
# actionlint only knows GitHub-hosted labels; register the self-hosted ARC
# runner scale set (infra/docs/04-arc-and-caching.md) so `runs-on: omp-kata`
# lints clean.
self-hosted-runner:
labels:
- omp-kata
.github/actions/bazel-cache/action.yml
+34 -5
View File
@@ -12,11 +12,17 @@ description: >
The v3 key is <config hash>-<source hash>: config covers toolchain and
build settings, source covers crates/** + BUILD.bazel. Restores fall
back to the config-scoped prefix, so a source generation the exact key
has never seen still seeds from the previous generation's archive, and
the refreshed archive is saved after the build (an exact hit suppresses
back to the config-scoped prefix, then to a bare scope+os+arch prefix:
the config hash covers Cargo.toml/Cargo.lock, which every release
version bump rewrites, so without the bare fallback each release built
the darwin addons fully cold (~40-50 min on macos-15-intel; CI run
30357804722). A stale-config archive is safe — bazel keys every
action by content digest, so mismatched entries just miss — and the
refreshed archive is saved after the build (an exact hit suppresses
the save — without the source component the first archive for a config
generation would permanently shadow newer source states).
generation would permanently shadow newer source states). Restored
entries untouched for 14 days are pruned before the build so
ever-seeding archives don't grow without bound.
inputs:
scope:
@@ -58,10 +64,12 @@ runs:
fi
key="bazel-disk-v3-${{ inputs.scope }}-${{ runner.os }}-${{ runner.arch }}-$CONFIG_HASH-$SOURCE_HASH"
prefix="bazel-disk-v3-${{ inputs.scope }}-${{ runner.os }}-${{ runner.arch }}-$CONFIG_HASH-"
bare_prefix="bazel-disk-v3-${{ inputs.scope }}-${{ runner.os }}-${{ runner.arch }}-"
{
echo "remote=$remote"
echo "cache-key=$key"
echo "cache-prefix=$prefix"
echo "cache-bare-prefix=$bare_prefix"
} >> "$GITHUB_OUTPUT"
- name: Restore bazel disk cache
@@ -69,10 +77,23 @@ runs:
if: steps.backend.outputs.remote != 'true'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-bazel-disk
# Both caches ride one archive: the action cache AND the repository
# cache (downloads + extracted repo contents at repo/contents, which
# reproducible repo rules reuse across output bases). Without the
# repo half, every hosted job re-downloaded and re-extracted the
# rust/LLVM/zig toolchains. Save sites must list the same paths.
path: |
~/.cache/omp-bazel-disk
~/.cache/omp-bazel-repo
key: ${{ steps.backend.outputs.cache-key }}
# Order matters: exact key, same-config prefix, then any archive for
# this scope+os+arch. The bare fallback is what keeps release version
# bumps (Cargo.toml/Cargo.lock churn -> new CONFIG_HASH) from going
# fully cold; bazel's content-addressed action keys make a
# stale-config archive a partial hit, never a wrong output.
restore-keys: |
${{ steps.backend.outputs.cache-prefix }}
${{ steps.backend.outputs.cache-bare-prefix }}
- name: Compose cache config
id: compose
@@ -112,6 +133,14 @@ runs:
} > "$rc"
else
mkdir -p "$HOME/.cache/omp-bazel-disk" "$HOME/.cache/omp-bazel-repo"
# Seeding from stale archives means entries dead source
# generations wrote would otherwise ride along forever; drop
# anything untouched for 14 days (tar preserves mtimes across
# the actions/cache round trip, so age survives restores).
# Covers the repo download/contents cache too — bazel's own
# contents-cache GC only runs on idle servers, which short-lived
# CI invocations never have.
find "$HOME/.cache/omp-bazel-disk" "$HOME/.cache/omp-bazel-repo" -type f -mtime +14 -delete 2>/dev/null || true
{
echo "common --config=ci"
echo "common --disk_cache=$HOME/.cache/omp-bazel-disk"
.github/actions/bazel-natives/action.yml
+4 -1
View File
@@ -43,5 +43,8 @@ runs:
if: steps.cache.outputs.save-needed == 'true'
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-bazel-disk
# Must mirror the restore path list in the bazel-cache action.
path: |
~/.cache/omp-bazel-disk
~/.cache/omp-bazel-repo
key: ${{ steps.cache.outputs.cache-key }}
.github/actions/bun-install/action.yml
+24 -3
View File
@@ -42,13 +42,25 @@ runs:
curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.14"
echo "${BUN_INSTALL}/bin" >> "$GITHUB_PATH"
# Off-infra (GitHub-hosted): stock actions/cache for the bun store.
- name: Cache bun store (GitHub cache)
# Off-infra (GitHub-hosted): actions/cache for the bun store, split into
# restore + save. actions/cache entries are scoped per ref but count
# against the shared 10 GB repo quota — letting every PR branch save its
# own copy of the ~500 MB store duplicated it 19x, evicting the far more
# valuable bazel disk-cache archives (which is why release darwin builds
# kept going cold). PRs only restore; non-PR runs (main pushes, releases,
# dispatches) produce the shared entry every PR can see.
- name: Restore bun store (GitHub cache)
id: bun-cache
if: steps.env.outputs.cache == 'gha'
uses: actions/cache@v4
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('**/bun.lock') }}
# A lockfile change seeds from the previous store: entries are
# content-hash-named tarballs, so stale extras waste only space and
# bun downloads just the delta.
restore-keys: |
bun-${{ runner.os }}-
# On-infra (omp-kata): the pod mounts a shared PVC at bun's store path.
- name: Prepare mounted bun store
@@ -75,3 +87,12 @@ runs:
echo "::warning title=bun install retry::shared bun store install failed; retrying with a clean job-local cache"
retry_cache="$(mktemp -d "${RUNNER_TEMP:-/tmp}/bun-cache-retry.XXXXXX")"
bun install --frozen-lockfile --cache-dir="$retry_cache"
# Producer half of the split cache: only non-PR runs save, so the store
# exists once per lockfile generation instead of once per PR ref.
- name: Save bun store (GitHub cache)
if: steps.env.outputs.cache == 'gha' && github.event_name != 'pull_request' && steps.bun-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('**/bun.lock') }}
.github/actions/native-inputs/action.yml
-78
View File
@@ -1,78 +0,0 @@
name: "Native inputs: change detection + artifact cache key"
description: >
Single source of truth for what counts as a native-affecting change.
`rust` gates Rust validation (tests, clippy, rustfmt); `cache-key`
addresses the prebuilt Linux x64 addon pair published by trusted main
builds. The detector pathspec and the hashed file set MUST cover the same
inputs — drift means a native change could ship without validation or be
tested against stale addons. The key embeds a schema version, OS, arch,
target pair, and build profile so a future target/profile change can
never resolve valid-but-wrong .node files under the same source hash.
outputs:
rust:
description: Whether the event touches native inputs (always true off pull_request)
value: ${{ steps.changes.outputs.rust }}
source-hash:
description: 16-hex fingerprint over every native build input
value: ${{ steps.hash.outputs.source-hash }}
cache-key:
description: Exact actions/cache key for the prebuilt Linux x64 addon pair
value: ${{ steps.hash.outputs.cache-key }}
runs:
using: composite
steps:
- name: Detect native-affecting changes
id: changes
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" != "pull_request" ]; then
echo "rust=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Write the diff to a file before matching: in a pipeline, an API
# failure is indistinguishable from "no native changes" (skips
# validation — fail-open), and grep -q can close the pipe early so
# gh dies on SIGPIPE and pipefail flips a MATCH to rust=false.
changed_files="$RUNNER_TEMP/native-changed-files"
gh pr diff ${{ github.event.pull_request.number }} --name-only > "$changed_files"
if grep -qE '^(crates/|bazel/|Cargo\.(toml|lock)|MODULE\.bazel(\.lock)?|BUILD\.bazel|\.bazelrc|\.bazelignore|\.bazelversion|rust-toolchain\.toml|rustfmt\.toml|scripts/bazel-natives\.ts|\.github/actions/(bazel-cache|bazel-natives|native-artifacts|native-inputs)/|\.github/workflows/ci\.yml)' "$changed_files"; then
echo "rust=true" >> "$GITHUB_OUTPUT"
else
echo "No native-affecting changes; skipping Rust validation."
echo "rust=false" >> "$GITHUB_OUTPUT"
fi
# Content-addresses the addon bytes from git index entries (mode,
# blob identity, path) — `git ls-files -s` covers the executable bit
# and never follows symlinks into worktree bytes. `--error-unmatch`
# fails the step loudly when a listed path disappears instead of
# silently narrowing the key.
- name: Compute native source hash
id: hash
shell: bash
run: |
set -euo pipefail
source_hash=$(git ls-files -s -z --error-unmatch -- \
crates bazel \
Cargo.toml Cargo.lock \
MODULE.bazel MODULE.bazel.lock BUILD.bazel \
.bazelrc .bazelignore .bazelversion \
rust-toolchain.toml rustfmt.toml \
scripts/bazel-natives.ts \
.github/actions/bazel-cache .github/actions/bazel-natives \
.github/actions/native-artifacts .github/actions/native-inputs \
.github/workflows/ci.yml \
| sort -z \
| sha256sum \
| cut -c1-16)
{
echo "source-hash=$source_hash"
echo "cache-key=native-addons-v1-linux-x64-baseline+modern-opt-$source_hash"
} >> "$GITHUB_OUTPUT"
echo "Native source hash: $source_hash"
.github/workflows/bazel-cache-warm.yml
+65 -47
View File
@@ -1,24 +1,38 @@
name: Warm bazel disk cache
# GitHub-hosted PR runners cannot use the cluster remote cache and only see
# actions/cache entries created on the default branch. Bazel action keys do
# not transfer across runner environments (a kata-produced disk cache misses
# every action on ubuntu-22.04), so seed the disk cache from the same image
# PR jobs run on. Runs the full hosted action set — Rust validation (test,
# clippy, rustfmt) plus the native addons — so one exact-key archive covers
# both rust_validate and native_addons for this source generation. The v3
# key embeds a crates/** source fingerprint, so a native change on main
# means an exact miss: the build seeds from the previous generation via the
# config-scoped prefix and the refreshed archive is saved. An exact hit
# makes every invocation a cache replay and saves nothing.
# GitHub-hosted runners cannot use the cluster remote cache and only see
# actions/cache entries created on the default branch, and bazel action keys
# do not transfer across runner environments (a kata-produced disk cache
# misses every action on a hosted image) — so seed each hosted consumer from
# the same image it runs on:
#
# warm_darwin (macos-*) -> scopes `release-darwin-*`, consumed by the
# release_binary darwin jobs. Releases are the
# only other producers for these scopes and a
# release always changes CONFIG_HASH (version
# bump rewrites Cargo.toml/Cargo.lock), so
# without warming here every release rebuilt the
# darwin addons cold (~40-50 min on
# macos-15-intel — run 30357804722).
# warm_bun (ubuntu-22.04)-> the shared `bun-<os>-<lockhash>` store entry.
# PR jobs restore it but never save (per-PR-ref
# copies of the ~500 MB store once ate ~9.4 GB of
# the 10 GB repo cache quota and evicted every
# bazel archive).
#
# The v3 key embeds a crates/** source fingerprint, so a native change on
# main means an exact miss: the build seeds from a previous generation via
# the prefix/bare restore keys and the refreshed archive is saved. An exact
# hit makes every invocation a cache replay and saves nothing.
#
# Paths mirror the cache-key inputs (bazel-cache action) plus bun.lock — a
# packages/** push cannot change any archive, so it does not trigger a warm.
on:
push:
branches: [main]
paths:
- "packages/**"
- "crates/**"
- "scripts/**"
- "bazel/**"
- "MODULE.bazel"
- "MODULE.bazel.lock"
@@ -30,50 +44,54 @@ on:
- "Cargo.lock"
- "rust-toolchain.toml"
- "rustfmt.toml"
- "bun.lock"
- ".github/**"
workflow_dispatch:
permissions:
contents: read
# No cancel-in-progress: a cancelled warm saves nothing, and the first darwin
# warm on a cold runner takes long enough that back-to-back native pushes
# would cancel it forever. Superseded queued runs still collapse to the
# newest one, and staleness is harmless — consumers seed via prefix keys.
concurrency:
group: bazel-cache-warm
cancel-in-progress: true
cancel-in-progress: false
jobs:
warm:
name: Seed hosted bazel disk cache
# Keeps the release_binary darwin scopes seeded near HEAD so a release's
# bazel build is the version-bump delta, not a cold graph. Scope, runner
# image, and target must stay in lockstep with the release_binary matrix
# in ci.yml — a mismatched scope warms an archive nobody restores.
warm_darwin:
name: "Seed darwin release bazel cache: ${{ matrix.target }}"
strategy:
fail-fast: false
matrix:
include:
- { os: macos-15-intel, target: darwin-x64-baseline, scope: release-darwin-x64 }
- { os: macos-14, target: darwin-arm64, scope: release-darwin-arm64 }
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: "1.3"
# scripts/bazel-natives.ts is dependency-free (node builtins only), so
# no `bun install` is needed; the bazel-natives action restores,
# builds, and saves the scope's disk cache.
- uses: ./.github/actions/bazel-natives
with:
targets: ${{ matrix.target }}
cache-scope: ${{ matrix.scope }}
# Produces the shared bun store entry PR jobs restore. bun-install's save
# half only runs on non-PR events, and main CI jobs run on omp-kata (PVC
# store, no GitHub cache), so this is the sole hosted-runner producer.
warm_bun:
name: Seed bun store cache
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- id: cache
uses: ./.github/actions/bazel-cache
with:
scope: linux
# Invocation set mirrors the hosted CI jobs (rust_validate +
# native_addons) so the saved archive serves both.
- name: Rust tests
run: |
set -euo pipefail
if [ "$(ulimit -Sn)" != unlimited ] && [ "$(ulimit -Sn)" -lt 65536 ]; then ulimit -Sn 65536 || true; fi
bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" test //crates/...
- name: Clippy (workspace lint policy on opted-in crates)
run: |
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
| xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy-strict --
- name: Clippy (default lints elsewhere)
run: |
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (//crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
| xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy --
- name: Rustfmt
run: bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=rustfmt //crates/...
- name: Build native addons
run: |
set -euo pipefail
bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build //:natives-linux-x64-baseline //:natives-linux-x64-modern
- name: Save bazel disk cache
if: steps.cache.outputs.save-needed == 'true'
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-bazel-disk
key: ${{ steps.cache.outputs.cache-key }}
- uses: ./.github/actions/bun-install
.github/workflows/ci.yml
+243 -196
View File
@@ -18,6 +18,13 @@ on:
- "Cargo.lock"
- "rust-toolchain.toml"
- "rustfmt.toml"
# A root dependency change (lockfile/catalog/patches) affects every
# job; without these a lockfile-only push shipped untested — and a
# release retagged onto such a commit never triggered its run.
- "bun.lock"
- "bunfig.toml"
- "package.json"
- "patches/**"
- ".github/**"
pull_request:
branches: [main]
@@ -36,6 +43,10 @@ on:
- "Cargo.lock"
- "rust-toolchain.toml"
- "rustfmt.toml"
- "bun.lock"
- "bunfig.toml"
- "package.json"
- "patches/**"
- ".github/**"
workflow_dispatch:
inputs:
@@ -75,7 +86,10 @@ jobs:
# ref (or from a tagged main HEAD) is also treated as a release.
release_metadata:
name: Resolve release metadata
runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-22.04' || 'omp-kata' }}
# Hosted, not omp-kata: this ~20 s git-tag probe gates the whole release
# chain, and queueing it behind the 4-runner kata pool adds dead minutes
# to the darwin critical path.
runs-on: ubuntu-22.04
outputs:
is-release: ${{ steps.detect.outputs.is-release }}
release-tag: ${{ steps.detect.outputs.release-tag }}
@@ -124,26 +138,23 @@ jobs:
- name: Build collab web
run: bun run collab:web:build
# Rust validation (tests, clippy, rustfmt) and native addon production are
# separate jobs: TS test shards wait only on the addons, and on
# native-changing PRs validation no longer delays artifact production.
# Rust validation (tests, clippy, rustfmt) runs on non-PR events only.
# Native-changing PRs are rare enough that they do not warrant PR-side
# Rust validation or a PR-side addon build (see native_addons): Rust is
# validated post-merge on main (kata remote cache, warm) and again at
# release. A skipped required check still satisfies branch protection.
rust_validate:
name: Validate Rust workspace (bazel)
runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-22.04' || 'omp-kata' }}
if: github.event_name != 'pull_request'
runs-on: omp-kata
steps:
- uses: actions/checkout@v4
- id: inputs
uses: ./.github/actions/native-inputs
# TS-only PRs skip Rust validation entirely.
- if: steps.inputs.outputs.rust == 'true'
uses: ./.github/actions/bun-install
- uses: ./.github/actions/bun-install
- id: cache
if: steps.inputs.outputs.rust == 'true'
uses: ./.github/actions/bazel-cache
with:
scope: linux
- name: Rust tests
if: steps.inputs.outputs.rust == 'true'
# The ulimit guard runs in the step that launches the bazel server
# (limits are per-process and the server persists across steps).
run: |
@@ -154,67 +165,49 @@ jobs:
# `[lints] workspace = true` get the workspace policy, the vendored
# brush fork is exempt (same as run-rs-task.ts's cargo excludes).
- name: Clippy (workspace lint policy on opted-in crates)
if: steps.inputs.outputs.rust == 'true'
run: |
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-voice/... + //crates/pi-walker/...)" \
| xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy-strict --
- name: Clippy (default lints elsewhere)
if: steps.inputs.outputs.rust == 'true'
run: |
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (//crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (//crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-voice/... + //crates/pi-walker/...) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
| xargs bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=clippy --
- name: Rustfmt
if: steps.inputs.outputs.rust == 'true'
run: bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build --config=rustfmt //crates/...
# No disk-cache save here: native_addons saves this same key
# concurrently, and an immutable archive can have only one producer
# per key — a validation-only archive winning the race would
# suppress the addon half on every later exact hit. Validation
# actions are seeded by the main cache warmer's combined archive.
# Builds the native addons every downstream job installs. TS-only PRs
# restore the prebuilt Linux x64 pair published by trusted main builds
# (exact content-addressed key, smoke-loaded before use) and skip Bazel
# entirely; anything else builds with bazel. Main builds all Linux-hosted
# targets, pull requests only the x64 pair tests require.
# Provides the native addons every downstream TS job installs. PRs never
# build: they fetch the latest release's Linux x64 pair from the
# @oh-my-pi/pi-natives-linux-x64 npm leaf instead. The workspace loader
# skips its version sentinel for workspace loads, so release addons load
# fine under a newer checkout; a PR whose TS tests depend on changed
# native behavior fails visibly and the native side lands via main.
# Main (kata, cluster remote cache) builds all Linux-hosted targets with
# bazel. Job name says "bazel" for continuity with required checks.
native_addons:
name: Build native addons (bazel)
runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-22.04' || 'omp-kata' }}
steps:
- uses: actions/checkout@v4
- id: inputs
uses: ./.github/actions/native-inputs
# Trusted fast path. Exact-key restores only — a prefix fallback
# could resolve valid-but-wrong .node files under a changed
# target/profile (the key embeds schema + os/arch + target pair +
# build profile + input hash; see native-inputs action).
- name: Restore prebuilt native addons
id: prebuilt
# ---- PR path: latest release addons from npm ----
- name: Fetch release native addons (npm)
if: github.event_name == 'pull_request'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-native-addons
key: ${{ steps.inputs.outputs.cache-key }}
- name: Stage cached addons
if: steps.prebuilt.outputs.cache-hit == 'true'
shell: bash
# Canonical filenames come from the target map in
# scripts/bazel-natives.ts; staging reproduces the bazel-bin
# layout the artifact upload below globs.
run: |
set -euo pipefail
tarball="$(npm view @oh-my-pi/pi-natives-linux-x64@latest dist.tarball)"
echo "Fetching $tarball"
curl -fsSL --retry 3 "$tarball" | tar -xz -C "$RUNNER_TEMP"
mkdir -p bazel-bin/natives-linux-x64-baseline bazel-bin/natives-linux-x64-modern
cp ~/.cache/omp-native-addons/pi_natives.linux-x64-baseline.node bazel-bin/natives-linux-x64-baseline/
cp ~/.cache/omp-native-addons/pi_natives.linux-x64-modern.node bazel-bin/natives-linux-x64-modern/
# A poisoned cache entry must not ship: load both addons before
# trusting them. Failure falls back to a full build (loudly) rather
# than failing the PR.
- name: Smoke cached addons
id: smoke
if: steps.prebuilt.outputs.cache-hit == 'true'
cp "$RUNNER_TEMP/package/pi_natives.linux-x64-baseline.node" bazel-bin/natives-linux-x64-baseline/
cp "$RUNNER_TEMP/package/pi_natives.linux-x64-modern.node" bazel-bin/natives-linux-x64-modern/
# A corrupt download must fail here, not as a confusing dlopen error
# in every downstream test shard.
- name: Smoke release addons
if: github.event_name == 'pull_request'
shell: bash
run: |
set -uo pipefail
set -euo pipefail
# NOTE: `bun -e 'require("./x.node")'` swallows dlopen failures
# (exit 0 on a bogus addon); a script file enforces them in
# both bun and node. Verified against a corrupt .node fixture.
@@ -229,50 +222,40 @@ jobs:
EOF
loader=node
if command -v bun >/dev/null 2>&1; then loader=bun; fi
if "$loader" "$RUNNER_TEMP/smoke-addons.js" \
"$PWD/bazel-bin/natives-linux-x64-baseline/pi_natives.linux-x64-baseline.node" \
"$PWD/bazel-bin/natives-linux-x64-modern/pi_natives.linux-x64-modern.node"; then
echo "ok=true" >> "$GITHUB_OUTPUT"
else
echo "::warning::cached native addons failed to load; falling back to a full bazel build"
rm -rf bazel-bin
echo "ok=false" >> "$GITHUB_OUTPUT"
fi
- name: Decide build path
id: decide
"$loader" "$RUNNER_TEMP/smoke-addons.js" \
"$PWD/bazel-bin/natives-linux-x64-baseline/pi_natives.linux-x64-baseline.node" \
"$PWD/bazel-bin/natives-linux-x64-modern/pi_natives.linux-x64-modern.node"
# Make the policy visible on the rare native-touching PR instead of
# leaving a reviewer to wonder which addons the tests exercised.
# Best-effort: a diff-API failure must not fail the job over a notice.
- name: Note native changes tested against release addons
if: github.event_name == 'pull_request'
shell: bash
env:
HIT: ${{ steps.prebuilt.outputs.cache-hit }}
SMOKE: ${{ steps.smoke.outputs.ok }}
GH_TOKEN: ${{ github.token }}
run: |
if [ "$HIT" = "true" ] && [ "$SMOKE" = "true" ]; then
echo "Prebuilt addons restored and verified; skipping bazel."
echo "needed=false" >> "$GITHUB_OUTPUT"
else
echo "needed=true" >> "$GITHUB_OUTPUT"
if gh pr diff ${{ github.event.pull_request.number }} --name-only 2>/dev/null \
| grep -qE '^(crates/|bazel/|Cargo\.(toml|lock)|MODULE\.bazel(\.lock)?|BUILD\.bazel|\.bazelrc|\.bazelignore|\.bazelversion|rust-toolchain\.toml|rustfmt\.toml)'; then
echo "::notice title=Native sources changed::PR CI tests against the latest release addons by design; native changes are validated post-merge on main and at release."
fi
# ---- main path: bazel build of all Linux-hosted targets ----
- id: cache
if: steps.decide.outputs.needed == 'true'
if: github.event_name != 'pull_request'
uses: ./.github/actions/bazel-cache
with:
scope: linux
- name: Build native addons once
if: steps.decide.outputs.needed == 'true'
env:
EVENT_NAME: ${{ github.event_name }}
if: github.event_name != 'pull_request'
run: |
set -eo pipefail
if [ "$(ulimit -Sn)" != unlimited ] && [ "$(ulimit -Sn)" -lt 65536 ]; then ulimit -Sn 65536 || true; fi
targets=(//:natives-linux-x64-baseline //:natives-linux-x64-modern)
if [ "$EVENT_NAME" != "pull_request" ]; then
targets=(//:natives-linux-all)
fi
bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build "${targets[@]}" 2>&1 | tee "$RUNNER_TEMP/bazel-build.log"
bazelisk --bazelrc="${{ steps.cache.outputs.rc }}" build //:natives-linux-all 2>&1 | tee "$RUNNER_TEMP/bazel-build.log"
# Cache-hit visibility: a supposedly warm build that executes
# thousands of actions is the failure mode that made CI slow — make
# it visible in the run summary instead of discovering it weeks in.
- name: Report bazel cache stats
if: steps.decide.outputs.needed == 'true'
if: github.event_name != 'pull_request'
shell: bash
run: |
# Bazel runs with --color=yes (config=ci); strip ANSI before
@@ -280,57 +263,7 @@ jobs:
# the raw log.
summary=$(sed -E 's/\x1b\[[0-9;]*m//g' "$RUNNER_TEMP/bazel-build.log" | grep -E '[0-9]+ processes:' | tail -1 || true)
echo "::notice title=Bazel build summary::${summary:-no process summary found}"
- name: Save Bazel disk cache
if: steps.cache.outputs.save-needed == 'true'
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-bazel-disk
key: ${{ steps.cache.outputs.cache-key }}
# Producer side of the fast path: trusted main builds publish the
# x64 pair under the content-addressed key. Smoke-load before save —
# an exact-key archive is never overwritten, so bad bytes would
# poison every TS-only PR until manual eviction.
- name: Stage built addons for cache
if: github.event_name != 'pull_request'
shell: bash
run: |
set -euo pipefail
mkdir -p ~/.cache/omp-native-addons
cp bazel-bin/natives-linux-x64-baseline/pi_natives.linux-x64-baseline.node \
bazel-bin/natives-linux-x64-modern/pi_natives.linux-x64-modern.node \
~/.cache/omp-native-addons/
- name: Smoke addons before caching
if: github.event_name != 'pull_request'
shell: bash
run: |
set -euo pipefail
cd ~/.cache/omp-native-addons
cat > "$RUNNER_TEMP/smoke-addons.js" <<'EOF'
for (const f of process.argv.slice(2)) {
const m = require(f);
if (!m || Object.keys(m).length === 0) {
console.error(`addon failed to load: ${f}`);
process.exit(1);
}
}
EOF
bun "$RUNNER_TEMP/smoke-addons.js" \
"$PWD/pi_natives.linux-x64-baseline.node" \
"$PWD/pi_natives.linux-x64-modern.node"
- name: Look up native addon cache
id: addon-cache
if: github.event_name != 'pull_request'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-native-addons
key: ${{ steps.inputs.outputs.cache-key }}
lookup-only: true
- name: Save native addon cache
if: github.event_name != 'pull_request' && steps.addon-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.cache/omp-native-addons
key: ${{ steps.inputs.outputs.cache-key }}
- name: Upload native addon artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
@@ -482,8 +415,11 @@ jobs:
OMP_INSTALL_TEST_SKIP_NATIVE_BUILD: "1"
run: bun run ci:test:install-methods
release_binary:
name: "Release binary: ${{ matrix.target_id }}"
# Aggregates every validation job so publish-side release jobs gate on one
# result. Binary BUILDS deliberately do not wait for this gate — they run
# in parallel with the test fan-out and only publishing is held back.
release_gate:
name: Release validation gate
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() &&
needs.rust_validate.result == 'success' &&
needs.native_addons.result == 'success' &&
@@ -496,6 +432,20 @@ jobs:
needs.test_smoke.result == 'success' && needs.check.result == 'success' &&
needs.install_methods.result == 'success' }}
needs: [release_metadata, check, rust_validate, native_addons, test_workspace, test_coding_agent_singleton, test_ts_native, test_coding_agent_ui, test_coding_agent_runtime, test_coding_agent_native, test_smoke, install_methods]
runs-on: ubuntu-22.04
steps:
- run: echo "release validation green"
# Builds (does not publish) the Linux-hosted release binaries in parallel
# with the test fan-out; the single need is native_addons, whose artifact
# supplies their addons. Darwin builds live in release_binary_darwin so
# they start at release_metadata time instead. Publishing (npm leaves,
# GitHub release, core npm) is gated on release_gate downstream.
release_binary:
name: "Release binary: ${{ matrix.target_id }}"
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() &&
needs.native_addons.result == 'success' }}
needs: [release_metadata, native_addons]
strategy:
fail-fast: false
matrix:
@@ -534,22 +484,6 @@ jobs:
binary_path: packages/coding-agent/binaries/omp-linux-musl-arm64,
native_targets: linux-musl-arm64,
}
- {
os: macos-15-intel,
platform: darwin,
arch: x64,
target_id: darwin-x64,
binary_path: packages/coding-agent/binaries/omp-darwin-x64,
native_targets: darwin-x64-baseline,
}
- {
os: macos-14,
platform: darwin,
arch: arm64,
target_id: darwin-arm64,
binary_path: packages/coding-agent/binaries/omp-darwin-arm64,
native_targets: darwin-arm64,
}
- {
os: ubuntu-22.04,
platform: win32,
@@ -561,60 +495,25 @@ jobs:
runs-on: ${{ matrix.os }}
permissions:
contents: read
id-token: write
env:
MACOS_SIGNING: ${{ secrets.APPLE_CERTIFICATE_P12 != '' && secrets.APPLE_CERTIFICATE_PASSWORD != '' && secrets.APPLE_API_KEY_ID != '' && secrets.APPLE_API_ISSUER_ID != '' && secrets.APPLE_API_KEY != '' }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: "1.3"
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "24"
registry-url: "https://registry.npmjs.org"
# npm runs under Bun when invoked by the release script; npm 12
# requires a newer emulated Node version than Bun 1.3 provides.
- name: Ensure npm supports trusted publishing
if: ${{ !inputs.skip_npm }}
run: npm install -g npm@11.17.0
- name: Cache bun dependencies
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('**/bun.lock') }}
- run: bun install --frozen-lockfile
# Linux and Windows addons come from the native_addons job. Darwin runners
# build only their own architecture because cross-hosted artifacts do
# not exist for macOS.
- name: Install prebuilt native addon(s)
if: matrix.platform != 'darwin'
uses: ./.github/actions/native-artifacts
with:
targets: ${{ matrix.native_targets }}
- name: Build native addon (bazel)
if: matrix.platform == 'darwin'
uses: ./.github/actions/bazel-natives
with:
targets: ${{ matrix.native_targets }}
cache-scope: release-${{ matrix.target_id }}
- name: Build release binary
env:
RELEASE_TARGETS: ${{ matrix.target_id }}
run: bun run ci:release:build-binaries
- name: Sign and notarize macOS binary (Developer ID)
# Replaces the ad-hoc signature with a Developer ID + hardened-runtime
# one (+JIT/library-validation entitlements; omp dlopens its
# runtime-extracted native addon, which has a different Team ID) and
# notarizes. Auto-skips until the APPLE_* secrets are configured.
if: matrix.platform == 'darwin' && env.MACOS_SIGNING == 'true'
env:
APPLE_CERTIFICATE_P12: ${{ secrets.APPLE_CERTIFICATE_P12 }}
APPLE_CERTIFICATE_PASSWORD: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
APPLE_API_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
APPLE_API_ISSUER_ID: ${{ secrets.APPLE_API_ISSUER_ID }}
APPLE_API_KEY: ${{ secrets.APPLE_API_KEY }}
run: bash scripts/ci-macos-sign.sh "${{ matrix.binary_path }}"
# Windows binary is cross-built on Linux, so we have no Windows runner
# to smoke it on. Cross-build correctness is verified via the bun
# `--compile --target=bun-windows-x64-*` cross-compile. Musl binaries
@@ -638,25 +537,168 @@ jobs:
HOME="$runtime_dir/home" XDG_DATA_HOME="$runtime_dir/xdg" omp --version
HOME="$runtime_dir/home" XDG_DATA_HOME="$runtime_dir/xdg" omp --smoke-test
'
- name: Publish native addon package
if: ${{ !inputs.skip_npm && matrix.libc != 'musl' }}
env:
# Fallback auth: setup-node wrote an .npmrc referencing
# NODE_AUTH_TOKEN; npm uses it only when OIDC has no trusted
# publisher for the package (or on a first publish).
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: bun run ci:release:publish-native-leaf ${{ matrix.target_id }}
- name: Upload release binary artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: omp-binary-${{ matrix.target_id }}
path: ${{ matrix.binary_path }}
# Darwin release builds are the longest leg of a release (bazel-built
# addons on macOS runners), build their own addons, and touch nothing from
# native_addons — so they gate on release_metadata alone and start the
# moment the release run is detected, fully overlapped with the tests.
release_binary_darwin:
name: "Release binary: ${{ matrix.target_id }}"
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() }}
needs: [release_metadata]
strategy:
fail-fast: false
matrix:
include:
- {
os: macos-15-intel,
target_id: darwin-x64,
binary_path: packages/coding-agent/binaries/omp-darwin-x64,
native_targets: darwin-x64-baseline,
}
- {
os: macos-14,
target_id: darwin-arm64,
binary_path: packages/coding-agent/binaries/omp-darwin-arm64,
native_targets: darwin-arm64,
}
runs-on: ${{ matrix.os }}
permissions:
contents: read
env:
MACOS_SIGNING: ${{ secrets.APPLE_CERTIFICATE_P12 != '' && secrets.APPLE_CERTIFICATE_PASSWORD != '' && secrets.APPLE_API_KEY_ID != '' && secrets.APPLE_API_ISSUER_ID != '' && secrets.APPLE_API_KEY != '' }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: "1.3"
- name: Cache bun dependencies
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('**/bun.lock') }}
- run: bun install --frozen-lockfile
# Darwin runners build their own architecture; cross-hosted artifacts
# do not exist for macOS. The warm workflow keeps these scopes seeded
# near HEAD, so this is normally the version-bump delta.
- name: Build native addon (bazel)
uses: ./.github/actions/bazel-natives
with:
targets: ${{ matrix.native_targets }}
cache-scope: release-${{ matrix.target_id }}
- name: Build release binary
env:
RELEASE_TARGETS: ${{ matrix.target_id }}
run: bun run ci:release:build-binaries
- name: Sign and notarize macOS binary (Developer ID)
# Replaces the ad-hoc signature with a Developer ID + hardened-runtime
# one (+JIT/library-validation entitlements; omp dlopens its
# runtime-extracted native addon, which has a different Team ID) and
# notarizes. Auto-skips until the APPLE_* secrets are configured.
if: env.MACOS_SIGNING == 'true'
env:
APPLE_CERTIFICATE_P12: ${{ secrets.APPLE_CERTIFICATE_P12 }}
APPLE_CERTIFICATE_PASSWORD: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
APPLE_API_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
APPLE_API_ISSUER_ID: ${{ secrets.APPLE_API_ISSUER_ID }}
APPLE_API_KEY: ${{ secrets.APPLE_API_KEY }}
run: bash scripts/ci-macos-sign.sh "${{ matrix.binary_path }}"
- name: Smoke release binary
run: |
runtime_dir="$(mktemp -d)"
HOME="$runtime_dir/home" XDG_DATA_HOME="$runtime_dir/xdg" "${{ matrix.binary_path }}" --version
HOME="$runtime_dir/home" XDG_DATA_HOME="$runtime_dir/xdg" "${{ matrix.binary_path }}" --smoke-test
# Darwin addons exist only on these runners; export them so
# release_native_leaves can publish every leaf package after the
# validation gate. Non-darwin leaves reuse the native-addons artifact.
- name: Upload darwin native addon artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: native-addons-${{ matrix.target_id }}
path: packages/natives/native/pi_natives.${{ matrix.target_id }}*.node
if-no-files-found: error
retention-days: 1
- name: Upload release binary artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: omp-binary-${{ matrix.target_id }}
path: ${{ matrix.binary_path }}
# Publishes the five @oh-my-pi/pi-natives-<tag> leaf packages once
# validation passes and every binary built. Runs on one linux runner: leaf
# publishing is platform-agnostic file packaging (gen-npm-packages reads
# packages/natives/native), so no per-platform runners are needed.
release_native_leaves:
name: Publish native leaf packages
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() &&
needs.release_gate.result == 'success' &&
needs.release_binary.result == 'success' &&
needs.release_binary_darwin.result == 'success' &&
!inputs.skip_npm }}
needs: [release_metadata, release_gate, release_binary, release_binary_darwin]
runs-on: ubuntu-22.04
# id-token lets npm mint the OIDC token for trusted publishing; the
# NODE_AUTH_TOKEN below is the fallback for unconfigured packages.
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: "1.3"
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "24"
registry-url: "https://registry.npmjs.org"
# npm runs under Bun when invoked by the release script; npm 12
# requires a newer emulated Node version than Bun 1.3 provides.
- name: Ensure npm supports trusted publishing
run: npm install -g npm@11.17.0
- name: Cache bun dependencies
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('**/bun.lock') }}
- run: bun install --frozen-lockfile
# Linux + win32 addons from the native_addons artifact (gnu targets
# only — musl reuses the linux filenames and has no leaf package).
- name: Install linux and win32 native addons
uses: ./.github/actions/native-artifacts
with:
targets: linux-x64-baseline linux-x64-modern linux-arm64 win32-x64-baseline
# Darwin addons come flat (canonical filenames) from the
# release_binary_darwin legs.
- name: Install darwin native addons
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: native-addons-darwin-*
path: packages/natives/native
merge-multiple: true
- name: Publish native leaf packages
env:
# Fallback auth: setup-node wrote an .npmrc referencing
# NODE_AUTH_TOKEN; npm uses it only when OIDC has no trusted
# publisher for the package (or on a first publish).
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: |
set -euo pipefail
for tag in linux-x64 linux-arm64 darwin-x64 darwin-arm64 win32-x64; do
bun run ci:release:publish-native-leaf "$tag"
done
release_github:
name: Publish GitHub release
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() &&
needs.release_binary.result == 'success' }}
needs: [release_metadata, release_binary]
needs.release_gate.result == 'success' &&
needs.release_binary.result == 'success' &&
needs.release_binary_darwin.result == 'success' }}
needs: [release_metadata, release_gate, release_binary, release_binary_darwin]
runs-on: ubuntu-22.04
permissions:
contents: write
@@ -727,9 +769,14 @@ jobs:
name: Publish to npm
if: ${{ needs.release_metadata.outputs.is-release == 'true' && !cancelled() &&
needs.release_binary.result == 'success' &&
needs.release_binary_darwin.result == 'success' &&
needs.release_github_verify.result == 'success' &&
needs.release_native_leaves.result == 'success' &&
!inputs.skip_npm }}
needs: [release_metadata, release_binary, release_github_verify]
# release_native_leaves must finish first: the core package lists the
# leaf packages as lockstep-versioned optionalDependencies, so installs
# resolve them the moment the core version is live.
needs: [release_metadata, release_binary, release_binary_darwin, release_github_verify, release_native_leaves]
runs-on: ubuntu-22.04
# `id-token: write` lets npm mint the GitHub OIDC token it exchanges for a
# short-lived publish token (trusted publishing + provenance). When a
.gitignore
+2
View File
@@ -26,6 +26,8 @@ target/
# OS
*.tmp
# Bun compile temp artifacts, left in cwd if a `Bun.build({ compile })` run is killed mid-write
*.bun-build
.DS_Store
# Logs & profiling
Cargo.lock
Generated
+43 -30
View File
@@ -1065,9 +1065,9 @@ dependencies = [
[[package]]
name = "clang-sys"
version = "1.8.1"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
checksum = "157a8ba7b480713b56f4c09fd13fc3e0a22a5dfab8097ba61cbc5feef950788a"
dependencies = [
"glob",
"libc",
@@ -1439,9 +1439,9 @@ dependencies = [
[[package]]
name = "ctor"
version = "1.0.11"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9bb72bb94fdc1bd619f4c18cc91ecf6302aeb333d31b3c6ec0bb841cd920209"
checksum = "2d83cb7e7a873830708d6b02a78cd36a592c6fa14bf267b68725103b85c0d77f"
[[package]]
name = "ctr"
@@ -1692,13 +1692,13 @@ dependencies = [
[[package]]
name = "displaydoc"
version = "0.2.6"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
@@ -2202,9 +2202,9 @@ checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
[[package]]
name = "fontdue"
version = "0.9.3"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e57e16b3fe8ff4364c0661fdaac543fb38b29ea9bc9c2f45612d90adf931d2b"
checksum = "7894823fa221401399e2598f8b63f81ac77ff5c63248b7656779bff1632d7d3d"
dependencies = [
"ttf-parser",
]
@@ -4732,7 +4732,7 @@ dependencies = [
[[package]]
name = "pi-ast"
version = "17.1.8"
version = "17.2.0"
dependencies = [
"anyhow",
"ast-grep-core",
@@ -4801,7 +4801,7 @@ dependencies = [
[[package]]
name = "pi-iso"
version = "17.1.8"
version = "17.2.0"
dependencies = [
"async-trait",
"libc",
@@ -4813,14 +4813,12 @@ dependencies = [
[[package]]
name = "pi-natives"
version = "17.1.8"
version = "17.2.0"
dependencies = [
"anyhow",
"arboard",
"ast-grep-core",
"audiopus_sys",
"base64",
"bytes",
"clap",
"clipboard-win",
"core-graphics",
@@ -4838,17 +4836,16 @@ dependencies = [
"image",
"inferno",
"libc",
"maudio",
"napi",
"napi-build",
"napi-derive",
"opus",
"parking_lot",
"phf 0.13.1",
"pi-ast",
"pi-iso",
"pi-shell",
"pi-uutils-ctx",
"pi-voice",
"pi-walker",
"png",
"portable-pty",
@@ -4865,7 +4862,6 @@ dependencies = [
"toml",
"unicode-segmentation",
"unicode-width 0.2.2",
"webrtc",
"windows-sys 0.61.2",
"winreg 0.56.0",
"x11rb",
@@ -4876,7 +4872,7 @@ dependencies = [
[[package]]
name = "pi-shell"
version = "17.1.8"
version = "17.2.0"
dependencies = [
"anyhow",
"brush-builtins",
@@ -4961,9 +4957,23 @@ dependencies = [
"libc",
]
[[package]]
name = "pi-voice"
version = "17.2.0"
dependencies = [
"audiopus_sys",
"bytes",
"flume",
"maudio",
"opus",
"parking_lot",
"tokio",
"webrtc",
]
[[package]]
name = "pi-walker"
version = "17.1.8"
version = "17.2.0"
dependencies = [
"dashmap",
"globset",
@@ -5758,9 +5768,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.42"
version = "0.23.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3c54fcab019b409d04215d3a17cb438fd7fbf192ee61461f20f4fe18704bc138"
checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
dependencies = [
"once_cell",
"ring",
@@ -6513,13 +6523,13 @@ dependencies = [
[[package]]
name = "tokio-macros"
version = "2.7.1"
version = "2.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6328af13490e73a9b4694030fafd93f8c8c6a9dede33e821c3fc63eddf8042ba"
checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
@@ -6542,9 +6552,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.1.3+spec-1.1.0"
version = "1.1.4+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53c96ecdfa941c8fc4fcaed14f99ada8ebed502eef533015095a07e3301d4c3c"
checksum = "3aace63f4bbcdfc2c965b059de67119c89c4017a70d633be6c104910f67056f5"
dependencies = [
"indexmap",
"serde_core",
@@ -6749,9 +6759,9 @@ dependencies = [
[[package]]
name = "tree-sitter-elisp"
version = "1.6.1"
version = "1.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e2e9a6ab3cebf24ca41cdc1f985549b9b33f7ef25c19ac7d18e53a4ee24da09"
checksum = "5d68ec866ff7af14f1514f11b0eba7aa837adf2ad1837fa9b30c360461bff481"
dependencies = [
"cc",
"tree-sitter-language",
@@ -7223,9 +7233,12 @@ checksum = "b40688ea6389c8171614b25491f71d4a27946e0c7ce2da1c6de27e25abf1a0ae"
[[package]]
name = "ttf-parser"
version = "0.21.1"
version = "0.25.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c591d83f69777866b9126b24c6dd9a18351f177e49d625920d19f989fd31cf8"
checksum = "d2df906b07856748fa3f6e0ad0cbaa047052d4a7dd609e231c4f72cee8c36f31"
dependencies = [
"core_maths",
]
[[package]]
name = "turn"
Cargo.toml
+2 -1
View File
@@ -3,7 +3,7 @@ members = ["crates/pi-*", "crates/vendor/*"]
resolver = "3"
[workspace.package]
version = "17.1.8"
version = "17.2.0"
edition = "2024"
license = "MIT"
authors = ["Can Boluk"]
@@ -177,6 +177,7 @@ too_many_lines = "allow" # Arbitrary limits don't account for necessary comp
pi-ast = { path = "crates/pi-ast" }
pi-iso = { path = "crates/pi-iso" }
pi-shell = { path = "crates/pi-shell" }
pi-voice = { path = "crates/pi-voice" }
pi-walker = { path = "crates/pi-walker" }
brush-core = { path = "crates/vendor/brush-core" }
brush-builtins = { path = "crates/vendor/brush-builtins" }
MODULE.bazel
+39
View File
@@ -37,6 +37,45 @@ rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")
rust.toolchain(
edition = "2024",
versions = ["nightly/2026-04-29"],
# Pinned archive checksums for every toolchain component. Not (only) a
# supply-chain nicety: rules_rust marks a toolchain-tools repo as
# reproducible — and therefore eligible for Bazel's repo contents cache —
# only when every sha256 it needs was predeclared here (repositories.bzl:
# `reproducible = sha256s == dict(ctx.attr.sha256s)`). Without these,
# every fresh output base (= every ephemeral CI pod) re-downloaded and
# re-extracted ~5 rustc toolchains (~20 s each). Regenerate after a
# version bump: for each `<tool>-nightly-<triple>` below, take the value
# from https://static.rust-lang.org/dist/<date>/<file>.tar.xz.sha256.
sha256s = {
"2026-04-29/cargo-nightly-aarch64-apple-darwin.tar.xz": "9165010618ad581a90bd3e74a36cab092e6f509c8e7ffdcfe00293257cafabb7",
"2026-04-29/cargo-nightly-aarch64-unknown-linux-gnu.tar.xz": "f9b463c3eadaf040b5028590c3d3072c8b0893f9863ed25138dfc54c254388e1",
"2026-04-29/cargo-nightly-x86_64-apple-darwin.tar.xz": "03cf3edb1d0b9564fd466553092ccec5c37be729c35df1d77f7c5a9206801638",
"2026-04-29/cargo-nightly-x86_64-unknown-linux-gnu.tar.xz": "0ae6ed5a9e262d082003d5804cd7fc6977057cfa3df721451e356347962f391f",
"2026-04-29/clippy-nightly-aarch64-apple-darwin.tar.xz": "84b933f8a2837c81adca3e3a44fed971ec0de27673e2c56f122f6d2b35ef9371",
"2026-04-29/clippy-nightly-aarch64-unknown-linux-gnu.tar.xz": "827c90b264a716475e85f51bd1999344af14648882cdd5a6d517a73c7e4c02c4",
"2026-04-29/clippy-nightly-x86_64-apple-darwin.tar.xz": "2c749f10bb18c752323b489e056c55f2e6e04ff3939d16444f4f0d960463d2d2",
"2026-04-29/clippy-nightly-x86_64-unknown-linux-gnu.tar.xz": "36a9409a58b3ac16ce9823d4e705055cad8144dd7f18464c028e3a06ae5d87c8",
"2026-04-29/llvm-tools-nightly-aarch64-apple-darwin.tar.xz": "c6a5cfd0c6133e1fae913a624bef2387a1b9c1e2e75316f9d9b6872a592a8b2c",
"2026-04-29/llvm-tools-nightly-aarch64-unknown-linux-gnu.tar.xz": "7e51dc58df2d0d9e263d629e6b149785bb61b716e267cc841556a4641ed52a50",
"2026-04-29/llvm-tools-nightly-x86_64-apple-darwin.tar.xz": "ab6cd3bae1c979fd807982b8c39e911227f9344dab5051d45f612b440dad13ff",
"2026-04-29/llvm-tools-nightly-x86_64-unknown-linux-gnu.tar.xz": "0fb4bed6714359d9505b870c8bd67c722266c5a3894f66523bc231b9c479db31",
"2026-04-29/rust-src-nightly.tar.xz": "2a9c5ae7696a808e30fc5169ddfea20d69571a11997f142944c022a0d30af48b",
"2026-04-29/rust-std-nightly-aarch64-apple-darwin.tar.xz": "cd9e231f3e343b4ed5cff5c384c813cecba13fca94ef767fa8dc4aa0ff58a171",
"2026-04-29/rust-std-nightly-aarch64-unknown-linux-gnu.tar.xz": "80dceec313e144e3cfc5d67326826d85c664e762a27b53fea018a96d537a2942",
"2026-04-29/rust-std-nightly-aarch64-unknown-linux-musl.tar.xz": "9d5cb99d5678b2310bd69b2b654e9dcbbc3c5a50f9b66d03a21745cff4444927",
"2026-04-29/rust-std-nightly-x86_64-apple-darwin.tar.xz": "4992887472dfdea66a7f96c85b81104b543974204f91722742bbe5453bce8ec1",
"2026-04-29/rust-std-nightly-x86_64-pc-windows-msvc.tar.xz": "890d91c1b86e482aaee32d793b366ee3889a9326143cd7f82a2b7507a4f4d592",
"2026-04-29/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz": "09774060c01c82005a2ae4e4b46a899c5e6157d6cc7188dfbb750c0a0f7c52cf",
"2026-04-29/rust-std-nightly-x86_64-unknown-linux-musl.tar.xz": "de5936933a2994ebff227b62fe41fca5707496d13da33563f2bfee3049e7b122",
"2026-04-29/rustc-nightly-aarch64-apple-darwin.tar.xz": "61fc2a3fd11637fbe9f961319aa16d84615d2a033de46909742000448290c687",
"2026-04-29/rustc-nightly-aarch64-unknown-linux-gnu.tar.xz": "d8c9d9117a0499b6891e4622c55c7be9568e6b0cbed4f53540e51459ab514633",
"2026-04-29/rustc-nightly-x86_64-apple-darwin.tar.xz": "66efb8a2fb4155cd010bafda3382f643d87f337b646542f7830eda36cf1571a4",
"2026-04-29/rustc-nightly-x86_64-unknown-linux-gnu.tar.xz": "b0db9dc4957c0f12ff8be0a30d22b411d4e2a7f434d747d767156be80d782f1d",
"2026-04-29/rustfmt-nightly-aarch64-apple-darwin.tar.xz": "56dcf5ac72bcf663687273b0b9cdf953537081f38a4592d3577e9a6502707b4b",
"2026-04-29/rustfmt-nightly-aarch64-unknown-linux-gnu.tar.xz": "b495726b0f311dcb53a8cf5325c61ef321f110a3739247536bc87109d428a3cc",
"2026-04-29/rustfmt-nightly-x86_64-apple-darwin.tar.xz": "9c64403758b4b631b60057f6ed4a93b02f19286a59a60f8ea235b4f7d4e1ba4c",
"2026-04-29/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz": "397e4ead440f29a1fc10903388a377f5607a1d404c29dc3e3a979d2bc813a801",
},
extra_target_triples = [
"x86_64-unknown-linux-gnu",
"aarch64-unknown-linux-gnu",
MODULE.bazel.lock
Generated
+35 -34
View File
File diff suppressed because one or more lines are too long
bazel/patches/hermetic_cc_toolchain-isolated-compile-cache.patch
+26 -1
View File
@@ -12,9 +12,18 @@ dir. Link steps keep the shared cache: they are the ones that benefit
battle-tested path for concurrent artifact builds, and after first warm they
are pure readers.
Also opt @zig_config into Bazel's repo contents cache (repo_metadata,
bazel >= 8.3): the fetch is deterministic given the pinned zig archive and
wrapper source, and re-running it cost ~20 s of wrapper compilation on every
fresh output base — every job on ephemeral CI pods.
Note: bazel's own patch parser (unlike GNU patch) requires the `diff --git`
separator line before each file section of a multi-file patch.
diff --git a/toolchain/zig-wrapper.zig b/toolchain/zig-wrapper.zig
--- a/toolchain/zig-wrapper.zig
+++ b/toolchain/zig-wrapper.zig
@@ -252,6 +252,32 @@ fn parseArgs(
@@ -267,6 +267,32 @@
if (run_mode == RunMode.cc)
try resolveColonLibraries(arena, cwd, &args);
@@ -47,3 +56,19 @@ are pure readers.
// Add -target as the last parameter. The wrapper should overwrite
// the target specified by other tools calling the wrapper.
// Some tools might pass LLVM target triple, which are rejected by zig.
diff --git a/toolchain/defs.bzl b/toolchain/defs.bzl
--- a/toolchain/defs.bzl
+++ b/toolchain/defs.bzl
@@ -278,6 +278,12 @@
)
repository_ctx.symlink("tools/zig-wrapper{}".format(exe), tool_path)
+ # omp: opt into the repo contents cache — the fetch is deterministic
+ # given the pinned zig archive and the wrapper source above.
+ if hasattr(repository_ctx, "repo_metadata"):
+ return repository_ctx.repo_metadata(reproducible = True)
+ return None
+
zig_repository = repository_rule(
attrs = {
"version": attr.string(),
bazel/toolchains/msvc/llvm.bzl
+6
View File
@@ -120,6 +120,12 @@ def _llvm_msvc_tools_impl(rctx):
rctx.file("BUILD.bazel", _BUILD.format(version = _LLVM_VERSION, host = key), executable = False)
# Opt into the repo contents cache: the fetch is a pure function of the
# pinned URL+sha256 and the deterministic prune above, and the ~172 s
# extraction of the 2 GiB archive was the single largest cost of every
# fresh output base on ephemeral CI pods (profiled: run 30510579596).
return rctx.repo_metadata(reproducible = True)
llvm_msvc_tools_repository = repository_rule(
implementation = _llvm_msvc_tools_impl,
doc = "Pruned LLVM release binaries (clang-cl/lld-link/llvm-lib/llvm-rc) for the exec host.",
bazel/toolchains/msvc/sysroot.bzl
+8
View File
@@ -161,6 +161,14 @@ def _xwin_sysroot_impl(rctx):
executable = False,
)
# Opt into the repo contents cache. The splat is channel-stable, not
# bit-reproducible forever (see the module docstring) — caching freezes a
# known-good splat, which also keeps remote action keys stable across
# pods until this file changes or the cache entry ages out (14 d GC).
# OMP_XWIN_CACHE_DIR is a predeclared input (environ attr), so kata and
# dev fetches key separate entries.
return rctx.repo_metadata(reproducible = True)
xwin_sysroot_repository = repository_rule(
implementation = _xwin_sysroot_impl,
doc = "MSVC CRT + Windows SDK sysroot splatted by a pinned xwin release.",
bun.lock
+38 -56
View File
@@ -21,7 +21,7 @@
},
"packages/agent": {
"name": "@oh-my-pi/pi-agent-core",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@oh-my-pi/pi-ai": "catalog:",
"@oh-my-pi/pi-catalog": "catalog:",
@@ -39,7 +39,7 @@
},
"packages/ai": {
"name": "@oh-my-pi/pi-ai",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@bufbuild/protobuf": "catalog:",
"@oh-my-pi/pi-catalog": "catalog:",
@@ -55,7 +55,7 @@
},
"packages/catalog": {
"name": "@oh-my-pi/pi-catalog",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@bufbuild/protobuf": "catalog:",
"@oh-my-pi/pi-utils": "catalog:",
@@ -69,7 +69,7 @@
},
"packages/coding-agent": {
"name": "@oh-my-pi/pi-coding-agent",
"version": "17.1.8",
"version": "17.2.0",
"bin": {
"omp": "src/cli.ts",
},
@@ -143,7 +143,7 @@
},
"packages/hashline": {
"name": "@oh-my-pi/hashline",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@oh-my-pi/pi-natives": "catalog:",
"lru-cache": "catalog:",
@@ -187,7 +187,7 @@
},
"packages/mnemopi": {
"name": "@oh-my-pi/pi-mnemopi",
"version": "17.1.8",
"version": "17.2.0",
"bin": {
"mnemopi": "src/cli.ts",
},
@@ -214,7 +214,7 @@
},
"packages/natives": {
"name": "@oh-my-pi/pi-natives",
"version": "17.1.8",
"version": "17.2.0",
"devDependencies": {
"@napi-rs/cli": "catalog:",
"@types/bun": "catalog:",
@@ -222,7 +222,7 @@
},
"packages/snapcompact": {
"name": "@oh-my-pi/snapcompact",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@oh-my-pi/pi-ai": "catalog:",
"@oh-my-pi/pi-natives": "catalog:",
@@ -235,7 +235,7 @@
},
"packages/stats": {
"name": "@oh-my-pi/omp-stats",
"version": "17.1.8",
"version": "17.2.0",
"bin": {
"omp-stats": "./src/index.ts",
},
@@ -262,7 +262,7 @@
},
"packages/swarm-extension": {
"name": "@oh-my-pi/swarm-extension",
"version": "17.1.8",
"version": "17.2.0",
"bin": {
"omp-swarm": "src/cli.ts",
},
@@ -278,7 +278,7 @@
},
"packages/tui": {
"name": "@oh-my-pi/pi-tui",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@oh-my-pi/pi-natives": "catalog:",
"@oh-my-pi/pi-utils": "catalog:",
@@ -317,7 +317,7 @@
},
"packages/utils": {
"name": "@oh-my-pi/pi-utils",
"version": "17.1.8",
"version": "17.2.0",
"dependencies": {
"@oh-my-pi/pi-natives": "catalog:",
"handlebars": "catalog:",
@@ -330,7 +330,7 @@
},
"packages/wire": {
"name": "@oh-my-pi/pi-wire",
"version": "17.1.8",
"version": "17.2.0",
"devDependencies": {
"@types/bun": "catalog:",
},
@@ -371,18 +371,18 @@
"@huggingface/transformers": "^4.2.0",
"@mozilla/readability": "^0.6.0",
"@napi-rs/cli": "3.7.2",
"@oh-my-pi/hashline": "17.1.8",
"@oh-my-pi/omp-stats": "17.1.8",
"@oh-my-pi/pi-agent-core": "17.1.8",
"@oh-my-pi/pi-ai": "17.1.8",
"@oh-my-pi/pi-catalog": "17.1.8",
"@oh-my-pi/pi-coding-agent": "17.1.8",
"@oh-my-pi/pi-mnemopi": "17.1.8",
"@oh-my-pi/pi-natives": "17.1.8",
"@oh-my-pi/pi-tui": "17.1.8",
"@oh-my-pi/pi-utils": "17.1.8",
"@oh-my-pi/pi-wire": "17.1.8",
"@oh-my-pi/snapcompact": "17.1.8",
"@oh-my-pi/hashline": "17.2.0",
"@oh-my-pi/omp-stats": "17.2.0",
"@oh-my-pi/pi-agent-core": "17.2.0",
"@oh-my-pi/pi-ai": "17.2.0",
"@oh-my-pi/pi-catalog": "17.2.0",
"@oh-my-pi/pi-coding-agent": "17.2.0",
"@oh-my-pi/pi-mnemopi": "17.2.0",
"@oh-my-pi/pi-natives": "17.2.0",
"@oh-my-pi/pi-tui": "17.2.0",
"@oh-my-pi/pi-utils": "17.2.0",
"@oh-my-pi/pi-wire": "17.2.0",
"@oh-my-pi/snapcompact": "17.2.0",
"@opentelemetry/api": "^1.9.1",
"@opentelemetry/api-logs": "^0.220.0",
"@opentelemetry/context-async-hooks": "^2.9.0",
@@ -524,7 +524,7 @@
"@emnapi/core": ["@emnapi/core@1.9.2", "", { "dependencies": { "@emnapi/wasi-threads": "1.2.1", "tslib": "^2.4.0" } }, "sha512-UC+ZhH3XtczQYfOlu3lNEkdW/p4dsJ1r/bP7H8+rhao3TTTMO1ATq/4DdIi23XuGoFY+Cz0JmCbdVl0hz9jZcA=="],
"@emnapi/runtime": ["@emnapi/runtime@1.11.1", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw=="],
"@emnapi/runtime": ["@emnapi/runtime@1.11.3", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA=="],
"@emnapi/wasi-threads": ["@emnapi/wasi-threads@1.2.1", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w=="],
@@ -1044,7 +1044,7 @@
"base64-js": ["base64-js@1.5.1", "", {}, "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA=="],
"baseline-browser-mapping": ["baseline-browser-mapping@2.11.1", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-HYXq73DDpCtNzOmrFsm9eSwCvWCql0RzqjpDzXN9EadiLJ4DNat0nsZ/Bzmy+Ud12mb4/zKDY0cQ805ZzN+i0A=="],
"baseline-browser-mapping": ["baseline-browser-mapping@2.11.4", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-s4+sLr9mZ/CyqeRritFeYV/Zx73OAtmaHn6kkBS1XRoJn1hrg3xIDUcpicAEX68tkcIN0iBCgti31C8zxtkhsQ=="],
"before-after-hook": ["before-after-hook@4.0.0", "", {}, "sha512-q6tR3RPqIB1pMiTRMFcZwuG5T8vwp+vUvEG0vuI6B+Rikh5BfPp2fQ82c925FOs+b0lcFQ8CFrL+KbilfZFhOQ=="],
@@ -1322,7 +1322,7 @@
"lru-cache": ["lru-cache@11.5.2", "", {}, "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g=="],
"lucide-react": ["lucide-react@1.26.0", "", { "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, "sha512-raglYVR2+VkMfJL158krjVmE+rV5ST2lzA/KQm1FRSjMHT4MnWaegHxoVEpmc2So3nOEhp9oGejJwAPX8MoAjg=="],
"lucide-react": ["lucide-react@1.27.0", "", { "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, "sha512-rJicGl/3Fly/E0rOH1YmPZ6e49JCnKknh1ox1vpHnkfjujAkKA6sqUZvH3MTAaXXjgexyUwgNwTJzTtYuAFYJw=="],
"magic-string": ["magic-string@0.30.21", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.5" } }, "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ=="],
@@ -1488,7 +1488,7 @@
"string-argv": ["string-argv@0.3.2", "", {}, "sha512-aqD2Q0144Z+/RqG52NeHEkZauTAUWJO8c6yTftGJKO3Tja5tUgIfmIl6kExvhtxSDP7fXB6DvzkfMpCd/F3G+Q=="],
"string-width": ["string-width@7.2.0", "", { "dependencies": { "emoji-regex": "^10.3.0", "get-east-asian-width": "^1.0.0", "strip-ansi": "^7.1.0" } }, "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ=="],
"string-width": ["string-width@8.2.2", "", { "dependencies": { "get-east-asian-width": "^1.5.0", "strip-ansi": "^7.1.2" } }, "sha512-GaPUh5gfdrYzqeVNZvUfT23vYYxXzKYidUcnMtJg/3rxRV63EFZy3k6xfKlmfeJD0176lnUV/Usr3XcwSvFzpg=="],
"string_decoder": ["string_decoder@1.3.0", "", { "dependencies": { "safe-buffer": "~5.2.0" } }, "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA=="],
@@ -1574,7 +1574,7 @@
"yaml": ["yaml@2.9.0", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA=="],
"yargs": ["yargs@18.0.0", "", { "dependencies": { "cliui": "^9.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "string-width": "^7.2.0", "y18n": "^5.0.5", "yargs-parser": "^22.0.0" } }, "sha512-4UEqdc2RYGHZc7Doyqkrqiln3p9X2DZVxaGbwhn2pi7MrRagKaOcIKe8L3OxYcbhXLgLFUS3zAYuQjKBQgmuNg=="],
"yargs": ["yargs@18.1.0", "", { "dependencies": { "cliui": "^9.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "string-width": "^8.2.1", "y18n": "^5.0.5", "yargs-parser": "^22.0.0" } }, "sha512-2rAgRKu54VsHkqI0/tYkmluGXHD4KW7yZoycuqDQ15QOTnc2VVfy0nN/1eMhnQLO00A+dwtK20xuCnc1YGeUyg=="],
"yargs-parser": ["yargs-parser@22.0.0", "", {}, "sha512-rwu/ClNdSMpkSrUb+d6BRsSkLUq1fmfsY6TOpYzTwvwkg1/NRG85KBy3kq++A8LKQwX6lsu+aWad+2khvuXrqw=="],
@@ -1636,11 +1636,13 @@
"@rolldown/binding-wasm32-wasi/@emnapi/core": ["@emnapi/core@1.11.1", "", { "dependencies": { "@emnapi/wasi-threads": "1.2.2", "tslib": "^2.4.0" } }, "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/core": ["@emnapi/core@1.11.1", "", { "dependencies": { "@emnapi/wasi-threads": "1.2.2", "tslib": "^2.4.0" }, "bundled": true }, "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ=="],
"@rolldown/binding-wasm32-wasi/@emnapi/runtime": ["@emnapi/runtime@1.11.1", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/runtime": ["@emnapi/runtime@1.11.1", "", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/core": ["@emnapi/core@1.11.3", "", { "dependencies": { "@emnapi/wasi-threads": "1.2.3", "tslib": "^2.4.0" }, "bundled": true }, "sha512-zLpS5asjEb7lq8jYLq37N6XKaE41DIexlY1rF/z4/tIl3wo13Sqm28fRyfIsKZD+NZ8mM5RoKkpW/rBcuoSZSg=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/wasi-threads": ["@emnapi/wasi-threads@1.2.2", "", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/runtime": ["@emnapi/runtime@1.11.3", "", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/wasi-threads": ["@emnapi/wasi-threads@1.2.3", "", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-ELEBe8PsLvvJ6QMr0zLt8ffvOHW/dc1m3CEzNMg7aJUv3bMaoDtw2TXyDAwkYBuroxxuHEwhRTLJSe5sya547g=="],
"@tailwindcss/oxide-wasm32-wasi/@napi-rs/wasm-runtime": ["@napi-rs/wasm-runtime@1.1.6", "", { "dependencies": { "@tybys/wasm-util": "^0.10.3" }, "peerDependencies": { "@emnapi/core": "^1.7.1", "@emnapi/runtime": "^1.7.1" }, "bundled": true }, "sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg=="],
@@ -1654,6 +1656,8 @@
"cli-progress/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="],
"cliui/string-width": ["string-width@7.2.0", "", { "dependencies": { "emoji-regex": "^10.3.0", "get-east-asian-width": "^1.0.0", "strip-ansi": "^7.1.0" } }, "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ=="],
"dom-serializer/domelementtype": ["domelementtype@3.0.0", "", {}, "sha512-umCQid3jKbDmVjx8jGaW7uUykm4DEUeyV21hPxNMo2nV955DhUThwqyOIDtreepP31hl84X7G5U9ZfsWvIB3Pg=="],
"dom-serializer/entities": ["entities@8.0.0", "", {}, "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA=="],
@@ -1686,7 +1690,7 @@
"string_decoder/safe-buffer": ["safe-buffer@5.2.1", "", {}, "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="],
"vite/lightningcss": ["lightningcss@1.33.0", "", { "dependencies": { "detect-libc": "^2.0.3" }, "optionalDependencies": { "lightningcss-android-arm64": "1.33.0", "lightningcss-darwin-arm64": "1.33.0", "lightningcss-darwin-x64": "1.33.0", "lightningcss-freebsd-x64": "1.33.0", "lightningcss-linux-arm-gnueabihf": "1.33.0", "lightningcss-linux-arm64-gnu": "1.33.0", "lightningcss-linux-arm64-musl": "1.33.0", "lightningcss-linux-x64-gnu": "1.33.0", "lightningcss-linux-x64-musl": "1.33.0", "lightningcss-win32-arm64-msvc": "1.33.0", "lightningcss-win32-x64-msvc": "1.33.0" } }, "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA=="],
"wrap-ansi/string-width": ["string-width@7.2.0", "", { "dependencies": { "emoji-regex": "^10.3.0", "get-east-asian-width": "^1.0.0", "strip-ansi": "^7.1.0" } }, "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ=="],
"@babel/helper-compilation-targets/lru-cache/yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
@@ -1714,28 +1718,6 @@
"jszip/readable-stream/string_decoder": ["string_decoder@1.1.1", "", { "dependencies": { "safe-buffer": "~5.1.0" } }, "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg=="],
"vite/lightningcss/lightningcss-android-arm64": ["lightningcss-android-arm64@1.33.0", "", { "os": "android", "cpu": "arm64" }, "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg=="],
"vite/lightningcss/lightningcss-darwin-arm64": ["lightningcss-darwin-arm64@1.33.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg=="],
"vite/lightningcss/lightningcss-darwin-x64": ["lightningcss-darwin-x64@1.33.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ=="],
"vite/lightningcss/lightningcss-freebsd-x64": ["lightningcss-freebsd-x64@1.33.0", "", { "os": "freebsd", "cpu": "x64" }, "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg=="],
"vite/lightningcss/lightningcss-linux-arm-gnueabihf": ["lightningcss-linux-arm-gnueabihf@1.33.0", "", { "os": "linux", "cpu": "arm" }, "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ=="],
"vite/lightningcss/lightningcss-linux-arm64-gnu": ["lightningcss-linux-arm64-gnu@1.33.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg=="],
"vite/lightningcss/lightningcss-linux-arm64-musl": ["lightningcss-linux-arm64-musl@1.33.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ=="],
"vite/lightningcss/lightningcss-linux-x64-gnu": ["lightningcss-linux-x64-gnu@1.33.0", "", { "os": "linux", "cpu": "x64" }, "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg=="],
"vite/lightningcss/lightningcss-linux-x64-musl": ["lightningcss-linux-x64-musl@1.33.0", "", { "os": "linux", "cpu": "x64" }, "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw=="],
"vite/lightningcss/lightningcss-win32-arm64-msvc": ["lightningcss-win32-arm64-msvc@1.33.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA=="],
"vite/lightningcss/lightningcss-win32-x64-msvc": ["lightningcss-win32-x64-msvc@1.33.0", "", { "os": "win32", "cpu": "x64" }, "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA=="],
"@huggingface/transformers/onnxruntime-node/global-agent/matcher": ["matcher@3.0.0", "", { "dependencies": { "escape-string-regexp": "^4.0.0" } }, "sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng=="],
"@huggingface/transformers/onnxruntime-node/global-agent/serialize-error": ["serialize-error@7.0.1", "", { "dependencies": { "type-fest": "^0.13.1" } }, "sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw=="],
crates/pi-natives/BUILD.bazel
+1
View File
@@ -48,6 +48,7 @@ rust_shared_library(
"//crates/pi-iso",
"//crates/pi-shell",
"//crates/pi-uutils-ctx",
"//crates/pi-voice",
"//crates/pi-walker",
],
)
crates/pi-natives/Cargo.toml
+1 -15
View File
@@ -15,11 +15,9 @@ workspace = true
[dependencies]
anyhow.workspace = true
audiopus_sys.workspace = true
arboard.workspace = true
ast-grep-core.workspace = true
base64.workspace = true
bytes.workspace = true
clap.workspace = true
globset.workspace = true
fontdue.workspace = true
@@ -32,16 +30,15 @@ icy_sixel.workspace = true
ignore.workspace = true
image = { workspace = true, features = ["bmp"] }
inferno.workspace = true
maudio.workspace = true
napi.workspace = true
napi-derive.workspace = true
opus.workspace = true
parking_lot.workspace = true
phf.workspace = true
flume.workspace = true
pi-ast.workspace = true
pi-iso.workspace = true
pi-shell.workspace = true
pi-voice.workspace = true
pi-walker.workspace = true
pi-uutils-ctx = { path = "../pi-uutils-ctx" }
portable-pty.workspace = true
@@ -58,7 +55,6 @@ tokio-util.workspace = true
toml.workspace = true
unicode-segmentation.workspace = true
unicode-width.workspace = true
webrtc.workspace = true
xxhash-rust.workspace = true
[target.'cfg(target_os = "linux")'.dependencies]
@@ -72,16 +68,6 @@ xcap = { version = "=0.9.6", default-features = false }
[target.'cfg(target_os = "macos")'.dependencies]
tempfile = "=3.27.0"
core-graphics = "0.25"
# Regenerate miniaudio FFI bindings from the compiled headers on macOS only.
# The maudio-sys pregenerated `unix.rs` bindings are Linux-shaped: their
# `ma_device`/`ma_context` backend-state unions omit the `#ifdef
# MA_SUPPORT_COREAUDIO` members that the macOS build of miniaudio.c actually
# contains, so shipping them on darwin is an ABI mismatch that silently breaks
# CoreAudio capture (device init/start succeed but no render callbacks fire).
# darwin builds with host Xcode (libclang available), so bindgen is fine here;
# the hermetic zig/musl/windows toolchains keep the bindgen-free pregen path,
# whose bindings are correct for those platforms.
maudio = { workspace = true, features = ["generate-bindings"] }
[target.'cfg(unix)'.dependencies]
libc.workspace = true
crates/pi-natives/src/audio.rs
+12 -380
View File
@@ -1,310 +1,26 @@
//! Cross-platform microphone capture and streaming speaker playback.
//! N-API bindings for microphone capture and speaker playback.
//!
//! miniaudio owns platform device discovery, format conversion, channel mixing,
//! and resampling. The N-API classes expose one stable mono `f32` contract to
//! TypeScript while the internal playback stream is shared with native WebRTC.
//! The engine — device discovery, format conversion, mixing, drain semantics —
//! lives in `pi_voice::audio`; these classes adapt its mono `f32` contract to
//! TypeScript callbacks and `Float32Array` buffers.
use std::sync::{
Arc,
atomic::{AtomicBool, AtomicU32, Ordering},
};
use std::sync::Arc;
use flume::TryRecvError;
use maudio::{
audio::{performance::PerformanceProfile, sample_rate::SampleRate},
backend::Backend,
device::{
Device,
device_builder::{DeviceBuilder, DeviceBuilderOps},
},
};
use napi::{
bindgen_prelude::{Float32Array, Result},
threadsafe_function::{ThreadsafeFunction, ThreadsafeFunctionCallMode, UnknownReturnValue},
};
use napi_derive::napi;
use parking_lot::Mutex;
use tokio::sync::Notify;
const AUDIO_CHANNELS: u32 = 1;
// PulseAudio TCP playback stutters with a 20 ms target buffer; 50 ms absorbs
// transport jitter while preserving interactive latency.
#[cfg(target_os = "linux")]
const PLAYBACK_PERIOD_MS: u32 = 50;
#[cfg(not(target_os = "linux"))]
const PLAYBACK_PERIOD_MS: u32 = 20;
// miniaudio's PulseAudio backend reserves three periods. Android's OpenSL ES
// source emits 125 ms fragments, so Linux capture needs at least 150 ms queued.
#[cfg(target_os = "linux")]
const CAPTURE_PERIOD_MS: u32 = 50;
#[cfg(not(target_os = "linux"))]
const CAPTURE_PERIOD_MS: u32 = 20;
// PulseAudio can retain its default three periods after the producer closes.
// Wait for all of them before stopping the device so the tail reaches the sink.
#[cfg(target_os = "linux")]
const PLAYBACK_DRAIN_CALLBACKS: usize = 3;
#[cfg(not(target_os = "linux"))]
const PLAYBACK_DRAIN_CALLBACKS: usize = 2;
#[cfg(target_os = "macos")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::CoreAudio];
#[cfg(target_os = "windows")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::Wasapi];
#[cfg(target_os = "linux")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::PulseAudio, Backend::Alsa, Backend::Jack];
#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))]
const AUDIO_BACKENDS: &[Backend] = &[Backend::Sndio, Backend::Audio4, Backend::Oss];
use pi_voice::audio::{CaptureStream, PlaybackState, PlaybackStream};
type CaptureCallback = ThreadsafeFunction<Float32Array, UnknownReturnValue>;
type NativeResult<T> = std::result::Result<T, String>;
struct PlaybackState {
gain_bits: AtomicU32,
drained: AtomicBool,
stopped: AtomicBool,
notify: Notify,
}
impl PlaybackState {
fn new() -> Self {
Self {
gain_bits: AtomicU32::new(1.0f32.to_bits()),
drained: AtomicBool::new(false),
stopped: AtomicBool::new(false),
notify: Notify::new(),
}
}
fn gain(&self) -> f32 {
f32::from_bits(self.gain_bits.load(Ordering::Acquire))
}
fn set_gain(&self, gain: f32) {
self.gain_bits.store(gain.to_bits(), Ordering::Release);
}
fn mark_drained(&self) {
if !self.drained.swap(true, Ordering::AcqRel) {
self.notify.notify_waiters();
}
}
fn mark_stopped(&self) {
self.stopped.store(true, Ordering::Release);
self.notify.notify_waiters();
}
async fn wait_for_drain(&self) {
loop {
let notified = self.notify.notified();
if self.drained.load(Ordering::Acquire) || self.stopped.load(Ordering::Acquire) {
return;
}
notified.await;
}
}
}
/// Producer endpoint for one native playback device.
#[derive(Clone)]
pub(crate) struct PlaybackWriter {
tx: flume::Sender<Vec<f32>>,
state: Arc<PlaybackState>,
}
impl PlaybackWriter {
/// Queue mono floating-point samples without blocking the caller.
pub(crate) fn write(&self, samples: &[f32]) -> NativeResult<()> {
if samples.is_empty() {
return Ok(());
}
if self.state.stopped.load(Ordering::Acquire) || self.state.drained.load(Ordering::Acquire) {
return Err("Native audio playback is closed".to_owned());
}
self
.tx
.send(samples.to_vec())
.map_err(|_| "Native audio playback is closed".to_owned())
}
}
/// Running mono playback stream shared by N-API playback and native WebRTC.
pub(crate) struct PlaybackStream {
device: Option<Device<f32>>,
writer: Option<PlaybackWriter>,
state: Arc<PlaybackState>,
}
impl PlaybackStream {
/// Open and start the default speaker at the requested logical sample rate.
pub(crate) fn start(sample_rate: u32) -> NativeResult<Self> {
let sample_rate = audio_sample_rate(sample_rate)?;
let state = Arc::new(PlaybackState::new());
let (tx, rx) = flume::unbounded::<Vec<f32>>();
let callback_state = Arc::clone(&state);
let mut current = Vec::new();
let mut cursor = 0;
let mut empty_callbacks = 0;
let mut builder = DeviceBuilder::playback().f32();
builder
.sample_rate(sample_rate)
.playback_channels(AUDIO_CHANNELS)
.period_size_millis(PLAYBACK_PERIOD_MS)
.performance_profile(PerformanceProfile::LowLatency)
.backends(AUDIO_BACKENDS);
let mut device = builder
.with_callback(move |_device, output| {
fill_playback(
&rx,
&mut current,
&mut cursor,
output,
&callback_state,
&mut empty_callbacks,
);
})
.map_err(|error| format!("Failed to open the default speaker: {error}"))?;
device
.device_start()
.map_err(|error| format!("Failed to start speaker playback: {error}"))?;
Ok(Self {
device: Some(device),
writer: Some(PlaybackWriter { tx, state: Arc::clone(&state) }),
state,
})
}
/// Clone the producer endpoint used by the remote-audio decoder.
pub(crate) fn writer(&self) -> NativeResult<PlaybackWriter> {
self
.writer
.clone()
.ok_or_else(|| "Native audio playback is closed".to_owned())
}
fn state(&self) -> Arc<PlaybackState> {
Arc::clone(&self.state)
}
fn finish_input(&mut self) {
self.writer.take();
}
fn set_gain(&self, gain: f32) -> NativeResult<()> {
if !gain.is_finite() {
return Err("Audio playback gain must be finite".to_owned());
}
self.state.set_gain(gain.max(0.0));
Ok(())
}
/// Stop playback immediately and release the default speaker.
pub(crate) fn stop(&mut self) -> NativeResult<()> {
self.writer.take();
self.state.mark_stopped();
let Some(mut device) = self.device.take() else {
return Ok(());
};
device
.device_stop()
.map_err(|error| format!("Failed to stop speaker playback: {error}"))
}
}
impl Drop for PlaybackStream {
fn drop(&mut self) {
let _ = self.stop();
}
}
fn audio_sample_rate(sample_rate: u32) -> NativeResult<SampleRate> {
SampleRate::try_from(sample_rate)
.map_err(|error| format!("Unsupported audio sample rate {sample_rate}: {error}"))
}
fn fill_playback(
rx: &flume::Receiver<Vec<f32>>,
current: &mut Vec<f32>,
cursor: &mut usize,
output: &mut [f32],
state: &PlaybackState,
empty_callbacks: &mut usize,
) {
output.fill(0.0);
if state.stopped.load(Ordering::Acquire) {
return;
}
let gain = state.gain();
let mut output_offset = 0;
while output_offset < output.len() {
if *cursor == current.len() {
match rx.try_recv() {
Ok(next) => {
*current = next;
*cursor = 0;
*empty_callbacks = 0;
},
Err(TryRecvError::Empty) => {
*empty_callbacks = 0;
break;
},
Err(TryRecvError::Disconnected) => {
*empty_callbacks += 1;
if *empty_callbacks >= PLAYBACK_DRAIN_CALLBACKS {
state.mark_drained();
}
break;
},
}
}
let count = (current.len() - *cursor).min(output.len() - output_offset);
let source = &current[*cursor..*cursor + count];
let destination = &mut output[output_offset..output_offset + count];
if gain == 1.0 {
destination.copy_from_slice(source);
} else {
for (destination, source) in destination.iter_mut().zip(source) {
*destination = *source * gain;
}
}
*cursor += count;
output_offset += count;
}
}
fn start_capture_device<C>(sample_rate: u32, mut on_audio: C) -> NativeResult<Device<f32>>
where
C: FnMut(&[f32]) + Send + 'static,
{
let sample_rate = audio_sample_rate(sample_rate)?;
let mut builder = DeviceBuilder::capture().f32();
builder
.sample_rate(sample_rate)
.capture_channels(AUDIO_CHANNELS)
.period_size_millis(CAPTURE_PERIOD_MS)
.performance_profile(PerformanceProfile::LowLatency)
.backends(AUDIO_BACKENDS);
let mut device = builder
.with_callback(move |_device, samples| {
if !samples.is_empty() {
on_audio(samples);
}
})
.map_err(|error| format!("Failed to open the default microphone: {error}"))?;
device
.device_start()
.map_err(|error| format!("Failed to start microphone capture: {error}"))?;
Ok(device)
}
/// Default-microphone capture converted to mono `f32` at the requested sample
/// rate.
#[napi]
pub struct AudioCapture {
device: Mutex<Option<Device<f32>>>,
stream: Mutex<Option<CaptureStream>>,
}
#[napi]
@@ -316,32 +32,22 @@ impl AudioCapture {
#[napi(ts_arg_type = "(error: Error | null, samples: Float32Array) => void")]
on_audio: CaptureCallback,
) -> Result<Self> {
let device = start_capture_device(sample_rate, move |samples| {
let stream = CaptureStream::start(sample_rate, move |samples| {
on_audio
.call(Ok(Float32Array::new(samples.to_vec())), ThreadsafeFunctionCallMode::NonBlocking);
})
.map_err(napi::Error::from_reason)?;
Ok(Self { device: Mutex::new(Some(device)) })
Ok(Self { stream: Mutex::new(Some(stream)) })
}
/// Stop capture immediately and release the microphone.
#[napi]
pub fn stop(&self) -> Result<()> {
let device = self.device.lock().take();
let Some(mut device) = device else {
let stream = self.stream.lock().take();
let Some(mut stream) = stream else {
return Ok(());
};
device.device_stop().map_err(|error| {
napi::Error::from_reason(format!("Failed to stop microphone capture: {error}"))
})
}
}
impl Drop for AudioCapture {
fn drop(&mut self) {
if let Some(mut device) = self.device.get_mut().take() {
let _ = device.device_stop();
}
stream.stop().map_err(napi::Error::from_reason)
}
}
@@ -416,77 +122,3 @@ impl AudioPlayback {
Ok(())
}
}
impl Drop for AudioPlayback {
fn drop(&mut self) {
if let Some(mut stream) = self.stream.get_mut().take() {
let _ = stream.stop();
}
}
}
#[cfg(test)]
mod tests {
use std::{
env,
mem::forget,
sync::atomic::AtomicUsize,
thread::sleep,
time::{Duration, Instant},
};
use super::*;
#[test]
fn playback_preserves_chunk_order_and_applies_render_gain() {
let state = PlaybackState::new();
state.set_gain(0.5);
let (tx, rx) = flume::unbounded();
tx.send(vec![1.0, -1.0]).expect("receiver is live");
tx.send(vec![0.5, -0.5]).expect("receiver is live");
drop(tx);
let mut current = Vec::new();
let mut cursor = 0;
let mut empty_callbacks = 0;
let mut output = [9.0; 5];
fill_playback(&rx, &mut current, &mut cursor, &mut output, &state, &mut empty_callbacks);
assert_eq!(output, [0.5, -0.5, 0.25, -0.25, 0.0]);
assert!(!state.drained.load(Ordering::Acquire));
let mut silence = [1.0; 2];
while empty_callbacks < PLAYBACK_DRAIN_CALLBACKS {
silence.fill(1.0);
fill_playback(&rx, &mut current, &mut cursor, &mut silence, &state, &mut empty_callbacks);
assert_eq!(silence, [0.0, 0.0]);
assert_eq!(
state.drained.load(Ordering::Acquire),
empty_callbacks >= PLAYBACK_DRAIN_CALLBACKS
);
}
}
#[test]
fn opt_in_default_capture_receives_frames() {
if env::var_os("OMP_NATIVE_AUDIO_CAPTURE_TEST").is_none() {
return;
}
let callbacks = Arc::new(AtomicUsize::new(0));
let callback_count = Arc::clone(&callbacks);
let mut device = start_capture_device(16_000, move |_samples| {
callback_count.fetch_add(1, Ordering::Relaxed);
})
.expect("default capture device starts");
let deadline = Instant::now() + Duration::from_secs(5);
while callbacks.load(Ordering::Relaxed) == 0 && Instant::now() < deadline {
sleep(Duration::from_millis(20));
}
if callbacks.load(Ordering::Relaxed) == 0 {
forget(device);
panic!("capture device started but delivered no frames within five seconds");
}
device.device_stop().expect("capture device stops");
}
}
crates/pi-natives/src/lib.rs
+1 -1
View File
@@ -260,7 +260,7 @@ fn create_windows_napi_tokio_runtime() -> Option<tokio::runtime::Runtime> {
/// MUST stay in sync with `VERSION_SENTINEL_EXPORT` in
/// `packages/natives/native/index.js` (which derives the name from
/// `package.json#version`).
#[napi(js_name = "__piNativesV17_1_8")]
#[napi(js_name = "__piNativesV17_2_0")]
pub const fn pi_natives_version_sentinel() {}
/// Native module entry point: install crash diagnostics before any tool can
crates/pi-natives/src/live.rs
+16 -696
View File
@@ -1,365 +1,21 @@
//! Native WebRTC media transport for Codex live conversations.
//! N-API bindings for the Codex live WebRTC peer.
//!
//! The TypeScript host owns authenticated signaling and the sideband protocol;
//! this module owns the realtime WebRTC peer, Opus media, and speaker playback.
//! the realtime peer, Opus media, and speaker playback live in
//! `pi_voice::live`. This class adapts its callbacks to non-blocking
//! threadsafe functions and its PCM input to `Float32Array`.
use std::{
sync::{
Arc, Weak,
atomic::{AtomicBool, AtomicUsize, Ordering},
},
time::Duration,
};
use std::sync::Arc;
use bytes::Bytes;
use napi::{
bindgen_prelude::{Float32Array, Result},
threadsafe_function::{ThreadsafeFunction, ThreadsafeFunctionCallMode, UnknownReturnValue},
};
use napi_derive::napi;
use opus::{Application, Channels, Decoder, Encoder};
use parking_lot::Mutex;
use tokio::{sync::watch, task::JoinHandle};
use webrtc::{
api::{
APIBuilder,
interceptor_registry::register_default_interceptors,
media_engine::{MIME_TYPE_OPUS, MediaEngine},
},
data_channel::{RTCDataChannel, data_channel_message::DataChannelMessage},
interceptor::registry::Registry,
media::Sample,
peer_connection::{
RTCPeerConnection, configuration::RTCConfiguration,
peer_connection_state::RTCPeerConnectionState,
sdp::session_description::RTCSessionDescription,
},
rtp_transceiver::{
rtp_codec::{RTCRtpCodecCapability, RTCRtpCodecParameters, RTPCodecType},
rtp_sender::RTCRtpSender,
},
track::{
track_local::{TrackLocal, track_local_static_sample::TrackLocalStaticSample},
track_remote::TrackRemote,
},
};
use crate::audio::{PlaybackStream, PlaybackWriter};
const DATA_CHANNEL_LABEL: &str = "oai-events";
const INPUT_SAMPLE_RATE: u32 = 16_000;
const INPUT_FRAME_SAMPLES: usize = 320;
const INPUT_FRAME_DURATION: Duration = Duration::from_millis(20);
const MAX_ENCODED_OPUS_BYTES: usize = 1_275;
const MAX_QUEUED_INPUT_SAMPLES: usize = 32_000;
const OUTPUT_SAMPLE_RATE: u32 = 48_000;
const MAX_DECODED_OPUS_SAMPLES: usize = 5_760;
const OUTPUT_LEVEL_SAMPLES: usize = 2_400;
const OUTPUT_FRAME_SAMPLES: usize = 960;
const DEFAULT_OPEN_TIMEOUT_MS: u32 = 20_000;
const DISCONNECT_GRACE: Duration = Duration::from_secs(2);
const CLOSE_TASK_TIMEOUT: Duration = Duration::from_secs(1);
const OPUS_CAPABILITY: RTCRtpCodecCapability = RTCRtpCodecCapability {
mime_type: String::new(),
clock_rate: OUTPUT_SAMPLE_RATE,
channels: 2,
sdp_fmtp_line: String::new(),
rtcp_feedback: Vec::new(),
};
use pi_voice::live::{DEFAULT_OPEN_TIMEOUT_MS, LiveCallbacks, LivePeerCore};
type StringCallback = ThreadsafeFunction<String, UnknownReturnValue>;
type LevelCallback = ThreadsafeFunction<f64, UnknownReturnValue>;
type NativeResult<T> = std::result::Result<T, String>;
#[derive(Clone, Debug)]
enum PeerSignal {
Connecting,
Open,
Failed(String),
Closed,
}
enum InputCommand {
Audio(Vec<f32>),
Muted(bool),
Close,
}
struct LiveCallbacks {
event: StringCallback,
level: LevelCallback,
failure: StringCallback,
}
struct LiveResources {
peer: Arc<RTCPeerConnection>,
data_channel: Arc<RTCDataChannel>,
input_tx: flume::Sender<InputCommand>,
input_task: JoinHandle<()>,
rtcp_task: JoinHandle<()>,
playback: PlaybackStream,
}
struct LivePeerCore {
callbacks: LiveCallbacks,
resources: Mutex<Option<LiveResources>>,
signal_tx: watch::Sender<PeerSignal>,
started: AtomicBool,
closing: AtomicBool,
muted: AtomicBool,
failure_reported: AtomicBool,
queued_samples: AtomicUsize,
}
impl LivePeerCore {
fn new(callbacks: LiveCallbacks) -> Self {
let (signal_tx, _) = watch::channel(PeerSignal::Connecting);
Self {
callbacks,
resources: Mutex::new(None),
signal_tx,
started: AtomicBool::new(false),
closing: AtomicBool::new(false),
muted: AtomicBool::new(false),
failure_reported: AtomicBool::new(false),
queued_samples: AtomicUsize::new(0),
}
}
async fn create_offer(self: &Arc<Self>) -> NativeResult<String> {
if self.started.swap(true, Ordering::AcqRel) {
return Err("Native live WebRTC peer has already started".to_owned());
}
if self.closing.load(Ordering::Acquire) {
return Err("Native live WebRTC peer is closed".to_owned());
}
let playback = PlaybackStream::start(OUTPUT_SAMPLE_RATE)?;
let playback_tx = playback.writer()?;
let mut media_engine = MediaEngine::default();
let capability = opus_capability();
media_engine
.register_codec(
RTCRtpCodecParameters {
capability: capability.clone(),
payload_type: 111,
..Default::default()
},
RTPCodecType::Audio,
)
.map_err(|error| format!("Failed to register the live Opus codec: {error}"))?;
let registry = register_default_interceptors(Registry::new(), &mut media_engine)
.map_err(|error| format!("Failed to configure live WebRTC interceptors: {error}"))?;
let api = APIBuilder::new()
.with_media_engine(media_engine)
.with_interceptor_registry(registry)
.build();
let peer = Arc::new(
api.new_peer_connection(RTCConfiguration::default())
.await
.map_err(|error| format!("Failed to create the live WebRTC peer: {error}"))?,
);
let track = Arc::new(TrackLocalStaticSample::new(
capability,
"audio".to_owned(),
"omp-live".to_owned(),
));
let sender = match peer
.add_track(Arc::clone(&track) as Arc<dyn TrackLocal + Send + Sync>)
.await
{
Ok(sender) => sender,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to add the live audio track: {error}"));
},
};
install_peer_callbacks(&peer, Arc::downgrade(self), playback_tx);
let data_channel = match peer.create_data_channel(DATA_CHANNEL_LABEL, None).await {
Ok(channel) => channel,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to create the live data channel: {error}"));
},
};
install_data_channel_callbacks(&data_channel, Arc::downgrade(self));
let offer = match peer.create_offer(None).await {
Ok(offer) => offer,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to create the live SDP offer: {error}"));
},
};
if let Err(error) = peer.set_local_description(offer.clone()).await {
let _ = peer.close().await;
return Err(format!("Failed to install the live SDP offer: {error}"));
}
let mut resources_slot = self.resources.lock();
if self.closing.load(Ordering::Acquire) {
drop(resources_slot);
let _ = peer.close().await;
return Err("Native live WebRTC peer was closed while starting".to_owned());
}
let (input_tx, input_rx) = flume::unbounded();
let input_task = tokio::spawn(run_input_audio(track, input_rx, Arc::downgrade(self)));
let rtcp_task = tokio::spawn(drain_rtcp(sender));
let resources =
LiveResources { peer, data_channel, input_tx, input_task, rtcp_task, playback };
*resources_slot = Some(resources);
Ok(offer.sdp)
}
async fn accept_answer(&self, sdp: String) -> NativeResult<()> {
let peer = self
.resources
.lock()
.as_ref()
.map(|resources| Arc::clone(&resources.peer))
.ok_or_else(|| "Native live WebRTC peer has not started".to_owned())?;
let answer = RTCSessionDescription::answer(sdp)
.map_err(|error| format!("Codex returned an invalid live SDP answer: {error}"))?;
peer
.set_remote_description(answer)
.await
.map_err(|error| format!("Failed to install the live SDP answer: {error}"))
}
async fn wait_for_open(&self, timeout_ms: u32) -> NativeResult<()> {
let mut signal_rx = self.signal_tx.subscribe();
let wait = async {
loop {
let signal = signal_rx.borrow().clone();
match signal {
PeerSignal::Open => return Ok(()),
PeerSignal::Failed(message) => return Err(message),
PeerSignal::Closed => {
return Err("Native live WebRTC peer closed before opening".to_owned());
},
PeerSignal::Connecting => {},
}
signal_rx
.changed()
.await
.map_err(|_| "Native live WebRTC peer stopped before opening".to_owned())?;
}
};
tokio::time::timeout(Duration::from_millis(u64::from(timeout_ms)), wait)
.await
.map_err(|_| "Timed out waiting for the live data channel to open".to_owned())?
}
fn push_audio(&self, samples: &[f32]) -> NativeResult<()> {
if samples.is_empty() || self.muted.load(Ordering::Acquire) {
return Ok(());
}
let input_tx = self
.resources
.lock()
.as_ref()
.map(|resources| resources.input_tx.clone())
.ok_or_else(|| "Native live WebRTC peer has not started".to_owned())?;
let sample_count = samples.len().min(MAX_QUEUED_INPUT_SAMPLES);
let retained = &samples[samples.len() - sample_count..];
let queued = self
.queued_samples
.fetch_add(sample_count, Ordering::AcqRel);
if queued.saturating_add(sample_count) > MAX_QUEUED_INPUT_SAMPLES {
self
.queued_samples
.fetch_sub(sample_count, Ordering::AcqRel);
return Ok(());
}
if input_tx
.send(InputCommand::Audio(retained.to_vec()))
.is_err()
{
self
.queued_samples
.fetch_sub(sample_count, Ordering::AcqRel);
return Err("Native live audio input is closed".to_owned());
}
Ok(())
}
fn set_muted(&self, muted: bool) -> NativeResult<()> {
self.muted.store(muted, Ordering::Release);
let input_tx = self
.resources
.lock()
.as_ref()
.map(|resources| resources.input_tx.clone());
if let Some(input_tx) = input_tx {
input_tx
.send(InputCommand::Muted(muted))
.map_err(|_| "Native live audio input is closed".to_owned())?;
}
Ok(())
}
fn report_event(&self, payload: String) {
self
.callbacks
.event
.call(Ok(payload), ThreadsafeFunctionCallMode::NonBlocking);
}
fn report_level(&self, level: f64) {
self
.callbacks
.level
.call(Ok(level.clamp(0.0, 1.0)), ThreadsafeFunctionCallMode::NonBlocking);
}
fn mark_open(&self) {
if !self.closing.load(Ordering::Acquire) {
self.signal_tx.send_replace(PeerSignal::Open);
}
}
fn report_failure(&self, message: String) {
if self.closing.load(Ordering::Acquire) || self.failure_reported.swap(true, Ordering::AcqRel)
{
return;
}
self
.signal_tx
.send_replace(PeerSignal::Failed(message.clone()));
self
.callbacks
.failure
.call(Ok(message), ThreadsafeFunctionCallMode::NonBlocking);
}
async fn close(&self) {
if self.closing.swap(true, Ordering::AcqRel) {
let mut signal_rx = self.signal_tx.subscribe();
while !matches!(*signal_rx.borrow(), PeerSignal::Closed) {
if signal_rx.changed().await.is_err() {
break;
}
}
return;
}
let resources = self.resources.lock().take();
if let Some(mut resources) = resources {
let _ = resources.input_tx.send(InputCommand::Close);
let _ = resources.peer.close().await;
let _ = resources.playback.stop();
let _ = tokio::time::timeout(CLOSE_TASK_TIMEOUT, resources.input_task).await;
resources.rtcp_task.abort();
let _ = resources.rtcp_task.await;
drop(resources.data_channel);
}
self.queued_samples.store(0, Ordering::Release);
self.signal_tx.send_replace(PeerSignal::Closed);
}
}
/// WebRTC peer that accepts 16 kHz mono PCM and renders remote Opus audio.
#[napi]
@@ -381,9 +37,15 @@ impl LiveWebRtcPeer {
) -> Self {
Self {
inner: Arc::new(LivePeerCore::new(LiveCallbacks {
event: on_event,
level: on_level,
failure: on_failure,
event: Box::new(move |payload| {
on_event.call(Ok(payload), ThreadsafeFunctionCallMode::NonBlocking);
}),
level: Box::new(move |level| {
on_level.call(Ok(level), ThreadsafeFunctionCallMode::NonBlocking);
}),
failure: Box::new(move |message| {
on_failure.call(Ok(message), ThreadsafeFunctionCallMode::NonBlocking);
}),
})),
}
}
@@ -446,7 +108,7 @@ impl LiveWebRtcPeer {
impl Drop for LiveWebRtcPeer {
fn drop(&mut self) {
if self.inner.closing.load(Ordering::Acquire) {
if self.inner.is_closing() {
return;
}
let inner = Arc::clone(&self.inner);
@@ -457,345 +119,3 @@ impl Drop for LiveWebRtcPeer {
}
}
}
fn opus_capability() -> RTCRtpCodecCapability {
RTCRtpCodecCapability {
mime_type: MIME_TYPE_OPUS.to_owned(),
clock_rate: OPUS_CAPABILITY.clock_rate,
channels: OPUS_CAPABILITY.channels,
sdp_fmtp_line: "minptime=10;useinbandfec=1".to_owned(),
rtcp_feedback: Vec::new(),
}
}
fn install_peer_callbacks(
peer: &Arc<RTCPeerConnection>,
core: Weak<LivePeerCore>,
playback_tx: PlaybackWriter,
) {
let output_sender = Arc::new(Mutex::new(Some(playback_tx)));
let output_sender_for_track = Arc::clone(&output_sender);
let core_for_track = core.clone();
peer.on_track(Box::new(move |track, _receiver, _transceiver| {
let output_sender = output_sender_for_track.lock().take();
let core = core_for_track.clone();
Box::pin(async move {
if track.kind() != RTPCodecType::Audio {
return;
}
let Some(output_sender) = output_sender else {
if let Some(core) = core.upgrade() {
core.report_failure(
"Codex live returned more than one remote audio track".to_owned(),
);
}
return;
};
tokio::spawn(receive_output_audio(track, output_sender, core));
})
}));
let peer_for_state = Arc::downgrade(peer);
peer.on_peer_connection_state_change(Box::new(move |state| {
let core = core.clone();
let peer = peer_for_state.clone();
Box::pin(async move {
let Some(core) = core.upgrade() else {
return;
};
match state {
RTCPeerConnectionState::Failed => {
core.report_failure("Live WebRTC peer connection failed".to_owned());
},
RTCPeerConnectionState::Closed if !core.closing.load(Ordering::Acquire) => {
core.report_failure("Live WebRTC peer connection closed unexpectedly".to_owned());
},
RTCPeerConnectionState::Disconnected => {
tokio::time::sleep(DISCONNECT_GRACE).await;
if peer.upgrade().is_some_and(|peer| {
peer.connection_state() == RTCPeerConnectionState::Disconnected
}) {
core.report_failure("Live WebRTC peer connection disconnected".to_owned());
}
},
_ => {},
}
})
}));
}
fn install_data_channel_callbacks(data_channel: &Arc<RTCDataChannel>, core: Weak<LivePeerCore>) {
let core_for_open = core.clone();
data_channel.on_open(Box::new(move || {
Box::pin(async move {
if let Some(core) = core_for_open.upgrade() {
core.mark_open();
}
})
}));
let core_for_message = core.clone();
data_channel.on_message(Box::new(move |message: DataChannelMessage| {
let core = core_for_message.clone();
Box::pin(async move {
if !message.is_string {
return;
}
if let (Some(core), Ok(payload)) =
(core.upgrade(), String::from_utf8(message.data.to_vec()))
{
core.report_event(payload);
}
})
}));
let core_for_close = core.clone();
data_channel.on_close(Box::new(move || {
let core = core_for_close.clone();
Box::pin(async move {
if let Some(core) = core.upgrade() {
core.report_failure("Live data channel closed unexpectedly".to_owned());
}
})
}));
data_channel.on_error(Box::new(move |error| {
let core = core.clone();
Box::pin(async move {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Live data channel failed: {error}"));
}
})
}));
}
async fn run_input_audio(
track: Arc<TrackLocalStaticSample>,
input_rx: flume::Receiver<InputCommand>,
core: Weak<LivePeerCore>,
) {
let mut encoder = match Encoder::new(INPUT_SAMPLE_RATE, Channels::Mono, Application::Voip) {
Ok(encoder) => encoder,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to initialize the live Opus encoder: {error}"));
}
return;
},
};
if let Err(error) = encoder.set_inband_fec(true) {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to configure the live Opus encoder: {error}"));
}
return;
}
let mut muted = false;
let mut pending = Vec::with_capacity(INPUT_FRAME_SAMPLES * 2);
let mut encoded = [0u8; MAX_ENCODED_OPUS_BYTES];
let mut ticker = tokio::time::interval(INPUT_FRAME_DURATION);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Burst);
ticker.tick().await;
loop {
tokio::select! {
biased;
command = input_rx.recv_async() => {
let Ok(command) = command else {
break;
};
match command {
InputCommand::Audio(samples) => {
if let Some(core) = core.upgrade() {
core.queued_samples.fetch_sub(samples.len(), Ordering::AcqRel);
}
if muted {
continue;
}
if samples.len() >= MAX_QUEUED_INPUT_SAMPLES {
pending.clear();
pending.extend_from_slice(&samples[samples.len() - MAX_QUEUED_INPUT_SAMPLES..]);
continue;
}
let overflow = pending
.len()
.saturating_add(samples.len())
.saturating_sub(MAX_QUEUED_INPUT_SAMPLES);
if overflow > 0 {
pending.drain(..overflow);
}
pending.extend_from_slice(&samples);
},
InputCommand::Muted(next_muted) => {
muted = next_muted;
pending.clear();
},
InputCommand::Close => break,
}
},
_ = ticker.tick() => {
let mut frame = [0.0f32; INPUT_FRAME_SAMPLES];
if !muted {
let consumed = pending.len().min(INPUT_FRAME_SAMPLES);
frame[..consumed].copy_from_slice(&pending[..consumed]);
if consumed > 0 {
pending.copy_within(consumed.., 0);
pending.truncate(pending.len() - consumed);
}
}
let encoded_len = match encoder.encode_float(&frame, &mut encoded) {
Ok(encoded_len) => encoded_len,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to encode live microphone audio: {error}"));
}
return;
},
};
let sample = Sample {
data: Bytes::copy_from_slice(&encoded[..encoded_len]),
duration: INPUT_FRAME_DURATION,
..Default::default()
};
if let Err(error) = track.write_sample(&sample).await {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to send live microphone audio: {error}"));
}
return;
}
},
}
}
}
async fn drain_rtcp(sender: Arc<RTCRtpSender>) {
while sender.read_rtcp().await.is_ok() {}
}
async fn receive_output_audio(
track: Arc<TrackRemote>,
playback_tx: PlaybackWriter,
core: Weak<LivePeerCore>,
) {
if !track
.codec()
.capability
.mime_type
.eq_ignore_ascii_case(MIME_TYPE_OPUS)
{
if let Some(core) = core.upgrade() {
core.report_failure(format!(
"Codex live negotiated unsupported audio codec {}",
track.codec().capability.mime_type
));
}
return;
}
let mut decoder = match Decoder::new(OUTPUT_SAMPLE_RATE, Channels::Mono) {
Ok(decoder) => decoder,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to initialize the live Opus decoder: {error}"));
}
return;
},
};
let mut decoded = vec![0.0f32; MAX_DECODED_OPUS_SAMPLES].into_boxed_slice();
let mut expected_sequence: Option<u16> = None;
let mut level = OutputLevel::default();
loop {
let packet = match track.read_rtp().await {
Ok((packet, _attributes)) => packet,
Err(error) => {
if let Some(core) = core.upgrade()
&& !core.closing.load(Ordering::Acquire)
{
core.report_failure(format!("Live remote audio track failed: {error}"));
}
return;
},
};
let sequence = packet.header.sequence_number;
if let Some(expected) = expected_sequence {
let gap = sequence.wrapping_sub(expected);
if gap >= u16::MAX / 2 {
continue;
}
if gap > 0 {
for _ in 1..gap.min(5) {
if let Ok(samples) =
decoder.decode_float(&[], &mut decoded[..OUTPUT_FRAME_SAMPLES], false)
{
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
}
}
if let Ok(samples) = decoder.decode_float(&packet.payload, &mut decoded, true) {
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
}
}
}
expected_sequence = Some(sequence.wrapping_add(1));
match decoder.decode_float(&packet.payload, &mut decoded, false) {
Ok(samples) => {
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
},
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to decode live speaker audio: {error}"));
}
return;
},
}
}
}
fn write_output(playback_tx: &PlaybackWriter, samples: &[f32], core: &Weak<LivePeerCore>) -> bool {
match playback_tx.write(samples) {
Ok(()) => true,
Err(error) => {
if let Some(core) = core.upgrade()
&& !core.closing.load(Ordering::Acquire)
{
core.report_failure(format!("Live speaker playback failed: {error}"));
}
false
},
}
}
#[derive(Default)]
struct OutputLevel {
sum_squares: f64,
samples: usize,
}
impl OutputLevel {
fn observe(&mut self, decoded: &[f32], core: &Weak<LivePeerCore>) {
let mut offset = 0;
while offset < decoded.len() {
let take = (OUTPUT_LEVEL_SAMPLES - self.samples).min(decoded.len() - offset);
for &sample in &decoded[offset..offset + take] {
let sample = f64::from(sample);
self.sum_squares = sample.mul_add(sample, self.sum_squares);
}
self.samples += take;
offset += take;
if self.samples == OUTPUT_LEVEL_SAMPLES {
if let Some(core) = core.upgrade() {
core.report_level((self.sum_squares / self.samples as f64).sqrt());
}
self.sum_squares = 0.0;
self.samples = 0;
}
}
}
}
crates/pi-voice/BUILD.bazel
+29
View File
@@ -0,0 +1,29 @@
load("@crates//:defs.bzl", "aliases", "all_crate_deps")
load("@rules_rust//rust:defs.bzl", "rust_library", "rust_test")
package(default_visibility = ["//visibility:public"])
exports_files(["Cargo.toml"])
# napi-free voice engine (miniaudio capture/playback + WebRTC/Opus live peer).
# Keeping the webrtc/opus/maudio graph behind this rlib means the pi_natives
# leaf — which recompiles on every release (version-sentinel edit in lib.rs) —
# no longer pays for the voice stack's monomorphization.
rust_library(
name = "pi-voice",
srcs = glob(["src/**/*.rs"]),
aliases = aliases(),
crate_name = "pi_voice",
edition = "2024",
proc_macro_deps = all_crate_deps(proc_macro = True),
version = "17.1.5",
deps = all_crate_deps(normal = True),
)
rust_test(
name = "pi-voice_test",
crate = ":pi-voice",
edition = "2024",
proc_macro_deps = all_crate_deps(proc_macro_dev = True),
deps = all_crate_deps(normal_dev = True),
)
crates/pi-voice/Cargo.toml
+33
View File
@@ -0,0 +1,33 @@
[package]
name = "pi-voice"
version.workspace = true
edition.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[lints]
workspace = true
[dependencies]
# Forces the static libopus build for the `opus` codec crate.
audiopus_sys.workspace = true
bytes.workspace = true
flume.workspace = true
maudio.workspace = true
opus.workspace = true
parking_lot.workspace = true
tokio.workspace = true
webrtc.workspace = true
[target.'cfg(target_os = "macos")'.dependencies]
# Regenerate miniaudio FFI bindings from the compiled headers on macOS only.
# The maudio-sys pregenerated `unix.rs` bindings are Linux-shaped: their
# `ma_device`/`ma_context` backend-state unions omit the `#ifdef
# MA_SUPPORT_COREAUDIO` members that the macOS build of miniaudio.c actually
# contains, so shipping them on darwin is an ABI mismatch that silently breaks
# CoreAudio capture (device init/start succeed but no render callbacks fire).
# darwin builds with host Xcode (libclang available), so bindgen is fine here;
# the hermetic zig/musl/windows toolchains keep the bindgen-free pregen path,
# whose bindings are correct for those platforms.
maudio = { workspace = true, features = ["generate-bindings"] }
crates/pi-voice/src/audio.rs
+400
View File
@@ -0,0 +1,400 @@
//! Cross-platform microphone capture and streaming speaker playback.
//!
//! miniaudio owns platform device discovery, format conversion, channel mixing,
//! and resampling. The engine exposes one stable mono `f32` contract: the
//! N-API classes in pi-natives adapt it to TypeScript, and [`crate::live`]
//! shares [`PlaybackStream`] for remote-audio rendering.
use std::sync::{
Arc,
atomic::{AtomicBool, AtomicU32, Ordering},
};
use flume::TryRecvError;
use maudio::{
audio::{performance::PerformanceProfile, sample_rate::SampleRate},
backend::Backend,
device::{
Device,
device_builder::{DeviceBuilder, DeviceBuilderOps},
},
};
use tokio::sync::Notify;
use crate::VoiceResult;
const AUDIO_CHANNELS: u32 = 1;
// PulseAudio TCP playback stutters with a 20 ms target buffer; 50 ms absorbs
// transport jitter while preserving interactive latency.
#[cfg(target_os = "linux")]
const PLAYBACK_PERIOD_MS: u32 = 50;
#[cfg(not(target_os = "linux"))]
const PLAYBACK_PERIOD_MS: u32 = 20;
// miniaudio's PulseAudio backend reserves three periods. Android's OpenSL ES
// source emits 125 ms fragments, so Linux capture needs at least 150 ms queued.
#[cfg(target_os = "linux")]
const CAPTURE_PERIOD_MS: u32 = 50;
#[cfg(not(target_os = "linux"))]
const CAPTURE_PERIOD_MS: u32 = 20;
// PulseAudio can retain its default three periods after the producer closes.
// Wait for all of them before stopping the device so the tail reaches the sink.
#[cfg(target_os = "linux")]
const PLAYBACK_DRAIN_CALLBACKS: usize = 3;
#[cfg(not(target_os = "linux"))]
const PLAYBACK_DRAIN_CALLBACKS: usize = 2;
#[cfg(target_os = "macos")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::CoreAudio];
#[cfg(target_os = "windows")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::Wasapi];
#[cfg(target_os = "linux")]
const AUDIO_BACKENDS: &[Backend] = &[Backend::PulseAudio, Backend::Alsa, Backend::Jack];
#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "linux")))]
const AUDIO_BACKENDS: &[Backend] = &[Backend::Sndio, Backend::Audio4, Backend::Oss];
/// Shared render-time state for one playback device: gain, drain, stop.
///
/// Held as an `Arc` by both the stream and its N-API wrapper so
/// [`PlaybackState::wait_for_drain`] can outlive the stream lock.
pub struct PlaybackState {
gain_bits: AtomicU32,
drained: AtomicBool,
stopped: AtomicBool,
notify: Notify,
}
impl PlaybackState {
fn new() -> Self {
Self {
gain_bits: AtomicU32::new(1.0f32.to_bits()),
drained: AtomicBool::new(false),
stopped: AtomicBool::new(false),
notify: Notify::new(),
}
}
fn gain(&self) -> f32 {
f32::from_bits(self.gain_bits.load(Ordering::Acquire))
}
fn set_gain(&self, gain: f32) {
self.gain_bits.store(gain.to_bits(), Ordering::Release);
}
fn mark_drained(&self) {
if !self.drained.swap(true, Ordering::AcqRel) {
self.notify.notify_waiters();
}
}
fn mark_stopped(&self) {
self.stopped.store(true, Ordering::Release);
self.notify.notify_waiters();
}
/// Resolve once every queued sample reached the speaker (or the stream
/// stopped). Used by the N-API `AudioPlayback.end()` graceful-close path.
pub async fn wait_for_drain(&self) {
loop {
let notified = self.notify.notified();
if self.drained.load(Ordering::Acquire) || self.stopped.load(Ordering::Acquire) {
return;
}
notified.await;
}
}
}
/// Producer endpoint for one native playback device. Cloned into the WebRTC
/// remote-audio decoder so it can feed the same speaker stream.
#[derive(Clone)]
pub struct PlaybackWriter {
tx: flume::Sender<Vec<f32>>,
state: Arc<PlaybackState>,
}
impl PlaybackWriter {
/// Queue mono floating-point samples without blocking the caller.
pub fn write(&self, samples: &[f32]) -> VoiceResult<()> {
if samples.is_empty() {
return Ok(());
}
if self.state.stopped.load(Ordering::Acquire) || self.state.drained.load(Ordering::Acquire) {
return Err("Native audio playback is closed".to_owned());
}
self
.tx
.send(samples.to_vec())
.map_err(|_| "Native audio playback is closed".to_owned())
}
}
/// Running mono playback stream shared by N-API playback and native WebRTC.
pub struct PlaybackStream {
device: Option<Device<f32>>,
writer: Option<PlaybackWriter>,
state: Arc<PlaybackState>,
}
impl PlaybackStream {
/// Open and start the default speaker at the requested logical sample rate.
pub fn start(sample_rate: u32) -> VoiceResult<Self> {
let sample_rate = audio_sample_rate(sample_rate)?;
let state = Arc::new(PlaybackState::new());
let (tx, rx) = flume::unbounded::<Vec<f32>>();
let callback_state = Arc::clone(&state);
let mut current = Vec::new();
let mut cursor = 0;
let mut empty_callbacks = 0;
let mut builder = DeviceBuilder::playback().f32();
builder
.sample_rate(sample_rate)
.playback_channels(AUDIO_CHANNELS)
.period_size_millis(PLAYBACK_PERIOD_MS)
.performance_profile(PerformanceProfile::LowLatency)
.backends(AUDIO_BACKENDS);
let mut device = builder
.with_callback(move |_device, output| {
fill_playback(
&rx,
&mut current,
&mut cursor,
output,
&callback_state,
&mut empty_callbacks,
);
})
.map_err(|error| format!("Failed to open the default speaker: {error}"))?;
device
.device_start()
.map_err(|error| format!("Failed to start speaker playback: {error}"))?;
Ok(Self {
device: Some(device),
writer: Some(PlaybackWriter { tx, state: Arc::clone(&state) }),
state,
})
}
/// Clone the producer endpoint used by the remote-audio decoder.
pub fn writer(&self) -> VoiceResult<PlaybackWriter> {
self
.writer
.clone()
.ok_or_else(|| "Native audio playback is closed".to_owned())
}
/// Shared render-time state, cloned out so callers can await drain after
/// releasing the stream lock.
pub fn state(&self) -> Arc<PlaybackState> {
Arc::clone(&self.state)
}
/// Close the producer side so the render callback can detect drain.
pub fn finish_input(&mut self) {
self.writer.take();
}
/// Scale audio at render time so gain changes affect already queued
/// samples. Rejects non-finite gains; negative gains clamp to silence.
pub fn set_gain(&self, gain: f32) -> VoiceResult<()> {
if !gain.is_finite() {
return Err("Audio playback gain must be finite".to_owned());
}
self.state.set_gain(gain.max(0.0));
Ok(())
}
/// Stop playback immediately and release the default speaker.
pub fn stop(&mut self) -> VoiceResult<()> {
self.writer.take();
self.state.mark_stopped();
let Some(mut device) = self.device.take() else {
return Ok(());
};
device
.device_stop()
.map_err(|error| format!("Failed to stop speaker playback: {error}"))
}
}
impl Drop for PlaybackStream {
fn drop(&mut self) {
let _ = self.stop();
}
}
fn audio_sample_rate(sample_rate: u32) -> VoiceResult<SampleRate> {
SampleRate::try_from(sample_rate)
.map_err(|error| format!("Unsupported audio sample rate {sample_rate}: {error}"))
}
fn fill_playback(
rx: &flume::Receiver<Vec<f32>>,
current: &mut Vec<f32>,
cursor: &mut usize,
output: &mut [f32],
state: &PlaybackState,
empty_callbacks: &mut usize,
) {
output.fill(0.0);
if state.stopped.load(Ordering::Acquire) {
return;
}
let gain = state.gain();
let mut output_offset = 0;
while output_offset < output.len() {
if *cursor == current.len() {
match rx.try_recv() {
Ok(next) => {
*current = next;
*cursor = 0;
*empty_callbacks = 0;
},
Err(TryRecvError::Empty) => {
*empty_callbacks = 0;
break;
},
Err(TryRecvError::Disconnected) => {
*empty_callbacks += 1;
if *empty_callbacks >= PLAYBACK_DRAIN_CALLBACKS {
state.mark_drained();
}
break;
},
}
}
let count = (current.len() - *cursor).min(output.len() - output_offset);
let source = &current[*cursor..*cursor + count];
let destination = &mut output[output_offset..output_offset + count];
if gain == 1.0 {
destination.copy_from_slice(source);
} else {
for (destination, source) in destination.iter_mut().zip(source) {
*destination = *source * gain;
}
}
*cursor += count;
output_offset += count;
}
}
/// Running default-microphone capture delivering low-latency mono `f32`
/// chunks to its callback. Wraps the miniaudio device so N-API callers never
/// see maudio types.
pub struct CaptureStream {
device: Option<Device<f32>>,
}
impl CaptureStream {
/// Open the default microphone at the requested sample rate. `on_audio`
/// runs on the realtime audio thread — it must not block.
pub fn start<C>(sample_rate: u32, mut on_audio: C) -> VoiceResult<Self>
where
C: FnMut(&[f32]) + Send + 'static,
{
let sample_rate = audio_sample_rate(sample_rate)?;
let mut builder = DeviceBuilder::capture().f32();
builder
.sample_rate(sample_rate)
.capture_channels(AUDIO_CHANNELS)
.period_size_millis(CAPTURE_PERIOD_MS)
.performance_profile(PerformanceProfile::LowLatency)
.backends(AUDIO_BACKENDS);
let mut device = builder
.with_callback(move |_device, samples| {
if !samples.is_empty() {
on_audio(samples);
}
})
.map_err(|error| format!("Failed to open the default microphone: {error}"))?;
device
.device_start()
.map_err(|error| format!("Failed to start microphone capture: {error}"))?;
Ok(Self { device: Some(device) })
}
/// Stop capture immediately and release the microphone.
pub fn stop(&mut self) -> VoiceResult<()> {
let Some(mut device) = self.device.take() else {
return Ok(());
};
device
.device_stop()
.map_err(|error| format!("Failed to stop microphone capture: {error}"))
}
}
impl Drop for CaptureStream {
fn drop(&mut self) {
let _ = self.stop();
}
}
#[cfg(test)]
mod tests {
use std::{
env,
mem::forget,
sync::atomic::AtomicUsize,
thread::sleep,
time::{Duration, Instant},
};
use super::*;
#[test]
fn playback_preserves_chunk_order_and_applies_render_gain() {
let state = PlaybackState::new();
state.set_gain(0.5);
let (tx, rx) = flume::unbounded();
tx.send(vec![1.0, -1.0]).expect("receiver is live");
tx.send(vec![0.5, -0.5]).expect("receiver is live");
drop(tx);
let mut current = Vec::new();
let mut cursor = 0;
let mut empty_callbacks = 0;
let mut output = [9.0; 5];
fill_playback(&rx, &mut current, &mut cursor, &mut output, &state, &mut empty_callbacks);
assert_eq!(output, [0.5, -0.5, 0.25, -0.25, 0.0]);
assert!(!state.drained.load(Ordering::Acquire));
let mut silence = [1.0; 2];
while empty_callbacks < PLAYBACK_DRAIN_CALLBACKS {
silence.fill(1.0);
fill_playback(&rx, &mut current, &mut cursor, &mut silence, &state, &mut empty_callbacks);
assert_eq!(silence, [0.0, 0.0]);
assert_eq!(
state.drained.load(Ordering::Acquire),
empty_callbacks >= PLAYBACK_DRAIN_CALLBACKS
);
}
}
#[test]
fn opt_in_default_capture_receives_frames() {
if env::var_os("OMP_NATIVE_AUDIO_CAPTURE_TEST").is_none() {
return;
}
let callbacks = Arc::new(AtomicUsize::new(0));
let callback_count = Arc::clone(&callbacks);
let mut stream = CaptureStream::start(16_000, move |_samples| {
callback_count.fetch_add(1, Ordering::Relaxed);
})
.expect("default capture device starts");
let deadline = Instant::now() + Duration::from_secs(5);
while callbacks.load(Ordering::Relaxed) == 0 && Instant::now() < deadline {
sleep(Duration::from_millis(20));
}
if callbacks.load(Ordering::Relaxed) == 0 {
forget(stream);
panic!("capture device started but delivered no frames within five seconds");
}
stream.stop().expect("capture device stops");
}
}
crates/pi-voice/src/lib.rs
+26
View File
@@ -0,0 +1,26 @@
//! Native voice engine: microphone capture, speaker playback, and the WebRTC
//! live-conversation peer.
//!
//! napi-free by design. `pi-natives` wraps these types in thin `#[napi]`
//! adapters, so the webrtc/opus/miniaudio dependency graph compiles once into
//! this rlib and rebuilds only when voice code changes — never when the
//! N-API surface elsewhere in the addon does.
//!
//! # Architecture
//! ```text
//! JS (packages/natives) -> #[napi] adapters (pi-natives audio.rs / live.rs)
//! -> pi_voice::audio (miniaudio capture/playback engine)
//! -> pi_voice::live (WebRTC peer + Opus media, feeds audio playback)
//! ```
//!
//! Async entry points ([`live::LivePeerCore::create_offer`] and friends) run
//! on the caller's ambient tokio runtime — napi-rs's `tokio_rt` inside the
//! addon. Callback fields ([`live::LiveCallbacks`]) are invoked from tokio
//! worker threads and must not block.
pub mod audio;
pub mod live;
/// Engine-level result: a human-readable failure message the N-API layer maps
/// to a JS error verbatim.
pub type VoiceResult<T> = std::result::Result<T, String>;
crates/pi-voice/src/live.rs
+722
View File
@@ -0,0 +1,722 @@
//! Native WebRTC media transport for Codex live conversations.
//!
//! The TypeScript host owns authenticated signaling and the sideband protocol;
//! this module owns the realtime WebRTC peer, Opus media, and speaker
//! playback. The N-API class in pi-natives is a thin adapter over
//! [`LivePeerCore`].
use std::{
sync::{
Arc, Weak,
atomic::{AtomicBool, AtomicUsize, Ordering},
},
time::Duration,
};
use bytes::Bytes;
use opus::{Application, Channels, Decoder, Encoder};
use parking_lot::Mutex;
use tokio::{sync::watch, task::JoinHandle};
use webrtc::{
api::{
APIBuilder,
interceptor_registry::register_default_interceptors,
media_engine::{MIME_TYPE_OPUS, MediaEngine},
},
data_channel::{RTCDataChannel, data_channel_message::DataChannelMessage},
interceptor::registry::Registry,
media::Sample,
peer_connection::{
RTCPeerConnection, configuration::RTCConfiguration,
peer_connection_state::RTCPeerConnectionState,
sdp::session_description::RTCSessionDescription,
},
rtp_transceiver::{
rtp_codec::{RTCRtpCodecCapability, RTCRtpCodecParameters, RTPCodecType},
rtp_sender::RTCRtpSender,
},
track::{
track_local::{TrackLocal, track_local_static_sample::TrackLocalStaticSample},
track_remote::TrackRemote,
},
};
use crate::{
VoiceResult,
audio::{PlaybackStream, PlaybackWriter},
};
const DATA_CHANNEL_LABEL: &str = "oai-events";
const INPUT_SAMPLE_RATE: u32 = 16_000;
const INPUT_FRAME_SAMPLES: usize = 320;
const INPUT_FRAME_DURATION: Duration = Duration::from_millis(20);
const MAX_ENCODED_OPUS_BYTES: usize = 1_275;
const MAX_QUEUED_INPUT_SAMPLES: usize = 32_000;
const OUTPUT_SAMPLE_RATE: u32 = 48_000;
const MAX_DECODED_OPUS_SAMPLES: usize = 5_760;
const OUTPUT_LEVEL_SAMPLES: usize = 2_400;
const OUTPUT_FRAME_SAMPLES: usize = 960;
/// Default `wait_for_open` timeout, exposed so the N-API adapter can apply it
/// when TypeScript passes no override.
pub const DEFAULT_OPEN_TIMEOUT_MS: u32 = 20_000;
const DISCONNECT_GRACE: Duration = Duration::from_secs(2);
const CLOSE_TASK_TIMEOUT: Duration = Duration::from_secs(1);
const OPUS_CAPABILITY: RTCRtpCodecCapability = RTCRtpCodecCapability {
mime_type: String::new(),
clock_rate: OUTPUT_SAMPLE_RATE,
channels: 2,
sdp_fmtp_line: String::new(),
rtcp_feedback: Vec::new(),
};
#[derive(Clone, Debug)]
enum PeerSignal {
Connecting,
Open,
Failed(String),
Closed,
}
enum InputCommand {
Audio(Vec<f32>),
Muted(bool),
Close,
}
/// Host callbacks for peer lifecycle and media events. Every callback is
/// invoked from tokio worker threads and must not block (the N-API adapter
/// forwards through non-blocking threadsafe functions).
pub struct LiveCallbacks {
/// One `oai-events` data-channel text payload.
pub event: Box<dyn Fn(String) + Send + Sync>,
/// RMS output level in `[0, 1]`, one report per level window.
pub level: Box<dyn Fn(f64) + Send + Sync>,
/// Terminal transport failure; reported at most once per peer.
pub failure: Box<dyn Fn(String) + Send + Sync>,
}
struct LiveResources {
peer: Arc<RTCPeerConnection>,
data_channel: Arc<RTCDataChannel>,
input_tx: flume::Sender<InputCommand>,
input_task: JoinHandle<()>,
rtcp_task: JoinHandle<()>,
playback: PlaybackStream,
}
/// WebRTC live-conversation peer: accepts 16 kHz mono PCM input and renders
/// remote Opus audio to the default speaker. Owned as an `Arc` by the N-API
/// `LiveWebRtcPeer` wrapper.
pub struct LivePeerCore {
callbacks: LiveCallbacks,
resources: Mutex<Option<LiveResources>>,
signal_tx: watch::Sender<PeerSignal>,
started: AtomicBool,
closing: AtomicBool,
muted: AtomicBool,
failure_reported: AtomicBool,
queued_samples: AtomicUsize,
}
impl LivePeerCore {
/// Create an idle peer with its host callbacks registered.
pub fn new(callbacks: LiveCallbacks) -> Self {
let (signal_tx, _) = watch::channel(PeerSignal::Connecting);
Self {
callbacks,
resources: Mutex::new(None),
signal_tx,
started: AtomicBool::new(false),
closing: AtomicBool::new(false),
muted: AtomicBool::new(false),
failure_reported: AtomicBool::new(false),
queued_samples: AtomicUsize::new(0),
}
}
/// Start the native media peer and return its SDP offer.
///
/// Fails when called twice, after close, or when the speaker, codec,
/// peer, track, or data channel cannot be set up.
pub async fn create_offer(self: &Arc<Self>) -> VoiceResult<String> {
if self.started.swap(true, Ordering::AcqRel) {
return Err("Native live WebRTC peer has already started".to_owned());
}
if self.closing.load(Ordering::Acquire) {
return Err("Native live WebRTC peer is closed".to_owned());
}
let playback = PlaybackStream::start(OUTPUT_SAMPLE_RATE)?;
let playback_tx = playback.writer()?;
let mut media_engine = MediaEngine::default();
let capability = opus_capability();
media_engine
.register_codec(
RTCRtpCodecParameters {
capability: capability.clone(),
payload_type: 111,
..Default::default()
},
RTPCodecType::Audio,
)
.map_err(|error| format!("Failed to register the live Opus codec: {error}"))?;
let registry = register_default_interceptors(Registry::new(), &mut media_engine)
.map_err(|error| format!("Failed to configure live WebRTC interceptors: {error}"))?;
let api = APIBuilder::new()
.with_media_engine(media_engine)
.with_interceptor_registry(registry)
.build();
let peer = Arc::new(
api.new_peer_connection(RTCConfiguration::default())
.await
.map_err(|error| format!("Failed to create the live WebRTC peer: {error}"))?,
);
let track = Arc::new(TrackLocalStaticSample::new(
capability,
"audio".to_owned(),
"omp-live".to_owned(),
));
let sender = match peer
.add_track(Arc::clone(&track) as Arc<dyn TrackLocal + Send + Sync>)
.await
{
Ok(sender) => sender,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to add the live audio track: {error}"));
},
};
install_peer_callbacks(&peer, Arc::downgrade(self), playback_tx);
let data_channel = match peer.create_data_channel(DATA_CHANNEL_LABEL, None).await {
Ok(channel) => channel,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to create the live data channel: {error}"));
},
};
install_data_channel_callbacks(&data_channel, Arc::downgrade(self));
let offer = match peer.create_offer(None).await {
Ok(offer) => offer,
Err(error) => {
let _ = peer.close().await;
return Err(format!("Failed to create the live SDP offer: {error}"));
},
};
if let Err(error) = peer.set_local_description(offer.clone()).await {
let _ = peer.close().await;
return Err(format!("Failed to install the live SDP offer: {error}"));
}
let mut resources_slot = self.resources.lock();
if self.closing.load(Ordering::Acquire) {
drop(resources_slot);
let _ = peer.close().await;
return Err("Native live WebRTC peer was closed while starting".to_owned());
}
let (input_tx, input_rx) = flume::unbounded();
let input_task = tokio::spawn(run_input_audio(track, input_rx, Arc::downgrade(self)));
let rtcp_task = tokio::spawn(drain_rtcp(sender));
let resources =
LiveResources { peer, data_channel, input_tx, input_task, rtcp_task, playback };
*resources_slot = Some(resources);
Ok(offer.sdp)
}
/// Apply the remote SDP answer returned by Codex signaling.
pub async fn accept_answer(&self, sdp: String) -> VoiceResult<()> {
let peer = self
.resources
.lock()
.as_ref()
.map(|resources| Arc::clone(&resources.peer))
.ok_or_else(|| "Native live WebRTC peer has not started".to_owned())?;
let answer = RTCSessionDescription::answer(sdp)
.map_err(|error| format!("Codex returned an invalid live SDP answer: {error}"))?;
peer
.set_remote_description(answer)
.await
.map_err(|error| format!("Failed to install the live SDP answer: {error}"))
}
/// Wait until the `oai-events` data channel is open, failing on peer
/// failure, close, or timeout.
pub async fn wait_for_open(&self, timeout_ms: u32) -> VoiceResult<()> {
let mut signal_rx = self.signal_tx.subscribe();
let wait = async {
loop {
let signal = signal_rx.borrow().clone();
match signal {
PeerSignal::Open => return Ok(()),
PeerSignal::Failed(message) => return Err(message),
PeerSignal::Closed => {
return Err("Native live WebRTC peer closed before opening".to_owned());
},
PeerSignal::Connecting => {},
}
signal_rx
.changed()
.await
.map_err(|_| "Native live WebRTC peer stopped before opening".to_owned())?;
}
};
tokio::time::timeout(Duration::from_millis(u64::from(timeout_ms)), wait)
.await
.map_err(|_| "Timed out waiting for the live data channel to open".to_owned())?
}
/// Queue 16 kHz mono PCM for Opus transmission. Silently drops audio while
/// muted or when the bounded input queue is full.
pub fn push_audio(&self, samples: &[f32]) -> VoiceResult<()> {
if samples.is_empty() || self.muted.load(Ordering::Acquire) {
return Ok(());
}
let input_tx = self
.resources
.lock()
.as_ref()
.map(|resources| resources.input_tx.clone())
.ok_or_else(|| "Native live WebRTC peer has not started".to_owned())?;
let sample_count = samples.len().min(MAX_QUEUED_INPUT_SAMPLES);
let retained = &samples[samples.len() - sample_count..];
let queued = self
.queued_samples
.fetch_add(sample_count, Ordering::AcqRel);
if queued.saturating_add(sample_count) > MAX_QUEUED_INPUT_SAMPLES {
self
.queued_samples
.fetch_sub(sample_count, Ordering::AcqRel);
return Ok(());
}
if input_tx
.send(InputCommand::Audio(retained.to_vec()))
.is_err()
{
self
.queued_samples
.fetch_sub(sample_count, Ordering::AcqRel);
return Err("Native live audio input is closed".to_owned());
}
Ok(())
}
/// Enable or disable microphone transmission, discarding partial muted
/// frames.
pub fn set_muted(&self, muted: bool) -> VoiceResult<()> {
self.muted.store(muted, Ordering::Release);
let input_tx = self
.resources
.lock()
.as_ref()
.map(|resources| resources.input_tx.clone());
if let Some(input_tx) = input_tx {
input_tx
.send(InputCommand::Muted(muted))
.map_err(|_| "Native live audio input is closed".to_owned())?;
}
Ok(())
}
/// Whether close has begun; lets the adapter's `Drop` skip spawning a
/// redundant close task.
pub fn is_closing(&self) -> bool {
self.closing.load(Ordering::Acquire)
}
fn report_event(&self, payload: String) {
(self.callbacks.event)(payload);
}
fn report_level(&self, level: f64) {
(self.callbacks.level)(level.clamp(0.0, 1.0));
}
fn mark_open(&self) {
if !self.closing.load(Ordering::Acquire) {
self.signal_tx.send_replace(PeerSignal::Open);
}
}
fn report_failure(&self, message: String) {
if self.closing.load(Ordering::Acquire) || self.failure_reported.swap(true, Ordering::AcqRel)
{
return;
}
self
.signal_tx
.send_replace(PeerSignal::Failed(message.clone()));
(self.callbacks.failure)(message);
}
/// Close media, the data channel, the peer connection, and speaker
/// playback. Concurrent calls wait for the first closer to finish.
pub async fn close(&self) {
if self.closing.swap(true, Ordering::AcqRel) {
let mut signal_rx = self.signal_tx.subscribe();
while !matches!(*signal_rx.borrow(), PeerSignal::Closed) {
if signal_rx.changed().await.is_err() {
break;
}
}
return;
}
let resources = self.resources.lock().take();
if let Some(mut resources) = resources {
let _ = resources.input_tx.send(InputCommand::Close);
let _ = resources.peer.close().await;
let _ = resources.playback.stop();
let _ = tokio::time::timeout(CLOSE_TASK_TIMEOUT, resources.input_task).await;
resources.rtcp_task.abort();
let _ = resources.rtcp_task.await;
drop(resources.data_channel);
}
self.queued_samples.store(0, Ordering::Release);
self.signal_tx.send_replace(PeerSignal::Closed);
}
}
fn opus_capability() -> RTCRtpCodecCapability {
RTCRtpCodecCapability {
mime_type: MIME_TYPE_OPUS.to_owned(),
clock_rate: OPUS_CAPABILITY.clock_rate,
channels: OPUS_CAPABILITY.channels,
sdp_fmtp_line: "minptime=10;useinbandfec=1".to_owned(),
rtcp_feedback: Vec::new(),
}
}
fn install_peer_callbacks(
peer: &Arc<RTCPeerConnection>,
core: Weak<LivePeerCore>,
playback_tx: PlaybackWriter,
) {
let output_sender = Arc::new(Mutex::new(Some(playback_tx)));
let output_sender_for_track = Arc::clone(&output_sender);
let core_for_track = core.clone();
peer.on_track(Box::new(move |track, _receiver, _transceiver| {
let output_sender = output_sender_for_track.lock().take();
let core = core_for_track.clone();
Box::pin(async move {
if track.kind() != RTPCodecType::Audio {
return;
}
let Some(output_sender) = output_sender else {
if let Some(core) = core.upgrade() {
core.report_failure(
"Codex live returned more than one remote audio track".to_owned(),
);
}
return;
};
tokio::spawn(receive_output_audio(track, output_sender, core));
})
}));
let peer_for_state = Arc::downgrade(peer);
peer.on_peer_connection_state_change(Box::new(move |state| {
let core = core.clone();
let peer = peer_for_state.clone();
Box::pin(async move {
let Some(core) = core.upgrade() else {
return;
};
match state {
RTCPeerConnectionState::Failed => {
core.report_failure("Live WebRTC peer connection failed".to_owned());
},
RTCPeerConnectionState::Closed if !core.closing.load(Ordering::Acquire) => {
core.report_failure("Live WebRTC peer connection closed unexpectedly".to_owned());
},
RTCPeerConnectionState::Disconnected => {
tokio::time::sleep(DISCONNECT_GRACE).await;
if peer.upgrade().is_some_and(|peer| {
peer.connection_state() == RTCPeerConnectionState::Disconnected
}) {
core.report_failure("Live WebRTC peer connection disconnected".to_owned());
}
},
_ => {},
}
})
}));
}
fn install_data_channel_callbacks(data_channel: &Arc<RTCDataChannel>, core: Weak<LivePeerCore>) {
let core_for_open = core.clone();
data_channel.on_open(Box::new(move || {
Box::pin(async move {
if let Some(core) = core_for_open.upgrade() {
core.mark_open();
}
})
}));
let core_for_message = core.clone();
data_channel.on_message(Box::new(move |message: DataChannelMessage| {
let core = core_for_message.clone();
Box::pin(async move {
if !message.is_string {
return;
}
if let (Some(core), Ok(payload)) =
(core.upgrade(), String::from_utf8(message.data.to_vec()))
{
core.report_event(payload);
}
})
}));
let core_for_close = core.clone();
data_channel.on_close(Box::new(move || {
let core = core_for_close.clone();
Box::pin(async move {
if let Some(core) = core.upgrade() {
core.report_failure("Live data channel closed unexpectedly".to_owned());
}
})
}));
data_channel.on_error(Box::new(move |error| {
let core = core.clone();
Box::pin(async move {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Live data channel failed: {error}"));
}
})
}));
}
async fn run_input_audio(
track: Arc<TrackLocalStaticSample>,
input_rx: flume::Receiver<InputCommand>,
core: Weak<LivePeerCore>,
) {
let mut encoder = match Encoder::new(INPUT_SAMPLE_RATE, Channels::Mono, Application::Voip) {
Ok(encoder) => encoder,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to initialize the live Opus encoder: {error}"));
}
return;
},
};
if let Err(error) = encoder.set_inband_fec(true) {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to configure the live Opus encoder: {error}"));
}
return;
}
let mut muted = false;
let mut pending = Vec::with_capacity(INPUT_FRAME_SAMPLES * 2);
let mut encoded = [0u8; MAX_ENCODED_OPUS_BYTES];
let mut ticker = tokio::time::interval(INPUT_FRAME_DURATION);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Burst);
ticker.tick().await;
loop {
tokio::select! {
biased;
command = input_rx.recv_async() => {
let Ok(command) = command else {
break;
};
match command {
InputCommand::Audio(samples) => {
if let Some(core) = core.upgrade() {
core.queued_samples.fetch_sub(samples.len(), Ordering::AcqRel);
}
if muted {
continue;
}
if samples.len() >= MAX_QUEUED_INPUT_SAMPLES {
pending.clear();
pending.extend_from_slice(&samples[samples.len() - MAX_QUEUED_INPUT_SAMPLES..]);
continue;
}
let overflow = pending
.len()
.saturating_add(samples.len())
.saturating_sub(MAX_QUEUED_INPUT_SAMPLES);
if overflow > 0 {
pending.drain(..overflow);
}
pending.extend_from_slice(&samples);
},
InputCommand::Muted(next_muted) => {
muted = next_muted;
pending.clear();
},
InputCommand::Close => break,
}
},
_ = ticker.tick() => {
let mut frame = [0.0f32; INPUT_FRAME_SAMPLES];
if !muted {
let consumed = pending.len().min(INPUT_FRAME_SAMPLES);
frame[..consumed].copy_from_slice(&pending[..consumed]);
if consumed > 0 {
pending.copy_within(consumed.., 0);
pending.truncate(pending.len() - consumed);
}
}
let encoded_len = match encoder.encode_float(&frame, &mut encoded) {
Ok(encoded_len) => encoded_len,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to encode live microphone audio: {error}"));
}
return;
},
};
let sample = Sample {
data: Bytes::copy_from_slice(&encoded[..encoded_len]),
duration: INPUT_FRAME_DURATION,
..Default::default()
};
if let Err(error) = track.write_sample(&sample).await {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to send live microphone audio: {error}"));
}
return;
}
},
}
}
}
async fn drain_rtcp(sender: Arc<RTCRtpSender>) {
while sender.read_rtcp().await.is_ok() {}
}
async fn receive_output_audio(
track: Arc<TrackRemote>,
playback_tx: PlaybackWriter,
core: Weak<LivePeerCore>,
) {
if !track
.codec()
.capability
.mime_type
.eq_ignore_ascii_case(MIME_TYPE_OPUS)
{
if let Some(core) = core.upgrade() {
core.report_failure(format!(
"Codex live negotiated unsupported audio codec {}",
track.codec().capability.mime_type
));
}
return;
}
let mut decoder = match Decoder::new(OUTPUT_SAMPLE_RATE, Channels::Mono) {
Ok(decoder) => decoder,
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to initialize the live Opus decoder: {error}"));
}
return;
},
};
let mut decoded = vec![0.0f32; MAX_DECODED_OPUS_SAMPLES].into_boxed_slice();
let mut expected_sequence: Option<u16> = None;
let mut level = OutputLevel::default();
loop {
let packet = match track.read_rtp().await {
Ok((packet, _attributes)) => packet,
Err(error) => {
if let Some(core) = core.upgrade()
&& !core.closing.load(Ordering::Acquire)
{
core.report_failure(format!("Live remote audio track failed: {error}"));
}
return;
},
};
let sequence = packet.header.sequence_number;
if let Some(expected) = expected_sequence {
let gap = sequence.wrapping_sub(expected);
if gap >= u16::MAX / 2 {
continue;
}
if gap > 0 {
for _ in 1..gap.min(5) {
if let Ok(samples) =
decoder.decode_float(&[], &mut decoded[..OUTPUT_FRAME_SAMPLES], false)
{
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
}
}
if let Ok(samples) = decoder.decode_float(&packet.payload, &mut decoded, true) {
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
}
}
}
expected_sequence = Some(sequence.wrapping_add(1));
match decoder.decode_float(&packet.payload, &mut decoded, false) {
Ok(samples) => {
if !write_output(&playback_tx, &decoded[..samples], &core) {
return;
}
level.observe(&decoded[..samples], &core);
},
Err(error) => {
if let Some(core) = core.upgrade() {
core.report_failure(format!("Failed to decode live speaker audio: {error}"));
}
return;
},
}
}
}
fn write_output(playback_tx: &PlaybackWriter, samples: &[f32], core: &Weak<LivePeerCore>) -> bool {
match playback_tx.write(samples) {
Ok(()) => true,
Err(error) => {
if let Some(core) = core.upgrade()
&& !core.closing.load(Ordering::Acquire)
{
core.report_failure(format!("Live speaker playback failed: {error}"));
}
false
},
}
}
#[derive(Default)]
struct OutputLevel {
sum_squares: f64,
samples: usize,
}
impl OutputLevel {
fn observe(&mut self, decoded: &[f32], core: &Weak<LivePeerCore>) {
let mut offset = 0;
while offset < decoded.len() {
let take = (OUTPUT_LEVEL_SAMPLES - self.samples).min(decoded.len() - offset);
for &sample in &decoded[offset..offset + take] {
let sample = f64::from(sample);
self.sum_squares = sample.mul_add(sample, self.sum_squares);
}
self.samples += take;
offset += take;
if self.samples == OUTPUT_LEVEL_SAMPLES {
if let Some(core) = core.upgrade() {
core.report_level((self.sum_squares / self.samples as f64).sqrt());
}
self.sum_squares = 0.0;
self.samples = 0;
}
}
}
}
docs/natives-build-release-debugging.md
+11 -14
View File
@@ -136,15 +136,17 @@ build --tls_certificate=infra/bazel-remote/ca.crt
### Split Rust validation and addon production
`.github/workflows/ci.yml` separates `rust_validate` from `native_addons`. Both run on `omp-kata` pods for pushes and `ubuntu-22.04` for pull requests, but TypeScript jobs depend only on `native_addons`.
`.github/workflows/ci.yml` separates `rust_validate` from `native_addons`; TypeScript jobs depend only on `native_addons`.
`rust_validate` uses `.github/actions/native-inputs` to inspect the complete pull-request file list. TypeScript-only pull requests skip every Rust step; native-affecting changes and all non-PR events run:
**Pull requests never build or validate Rust.** Native-affecting PRs are rare enough that they don't warrant a PR-side bazel build: `rust_validate` is skipped entirely (`if: github.event_name != 'pull_request'`), and `native_addons` fetches the latest release's Linux x64 addon pair from the `@oh-my-pi/pi-natives-linux-x64` npm leaf, smoke-loads both, and uploads them as the `native-addons` workflow artifact. The loader skips its version sentinel for workspace loads, so release-versioned addons load fine under a newer checkout. A PR whose TypeScript tests depend on changed native behavior fails visibly (and CI emits a notice on any native-touching PR); the Rust side is validated post-merge on main and again at release.
On non-PR events both jobs run on `omp-kata` pods against the cluster remote cache. `rust_validate` runs:
```bash
bazelisk --bazelrc="$rc" test //crates/... # full Rust suite
# clippy scope mirrors `cargo clippy --workspace` (libraries only), split by
# lint policy via a query kind filter:
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-voice/... + //crates/pi-walker/...)" \
| xargs bazelisk --bazelrc="$rc" build --config=clippy-strict --
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (…strict set…) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
| xargs bazelisk --bazelrc="$rc" build --config=clippy --
@@ -153,19 +155,14 @@ bazelisk --bazelrc="$rc" build --config=rustfmt //crates/...
- `--config=clippy` = rules_rust clippy aspect + `-Dwarnings`; `--config=clippy-strict` layers the generated `bazel/clippy.bazelrc` for crates with `[lints] workspace = true`.
- `--config=rustfmt` = rustfmt aspect against the workspace `rustfmt.toml`.
- `rust_validate` never saves a hosted disk-cache archive: `native_addons` may concurrently own the same immutable key, while the main-branch warmer publishes a combined validation/addon archive.
`native_addons` is the artifact producer for every downstream TypeScript and release job:
- Pull requests first restore the exact `native-addons-v1-linux-x64-baseline+modern-opt-<source-hash>` cache entry published by trusted main builds. Both addons are loaded before use; a miss or failed smoke check falls back to building the Linux x64 pair.
- Main and other non-PR runs build `//:natives-linux-all`, smoke the x64 pair before publishing its exact addon cache, and upload every `.node` output as the `native-addons` workflow artifact.
- Downstream jobs use `.github/actions/native-artifacts` to download that workflow artifact and install the requested target set without invoking Bazel.
`native_addons` on main builds `//:natives-linux-all` and uploads every `.node` output as the `native-addons` workflow artifact. Downstream jobs use `.github/actions/native-artifacts` to download that artifact and install the requested target set without invoking Bazel.
No toolchain setup steps are required for native jobs: bazelisk is on the GitHub images and baked into the kata runner image; Bazel fetches Rust/zig/LLVM/xwin hermetically.
### Hosted cache warmer
`.github/workflows/bazel-cache-warm.yml` runs the full hosted validation and Linux x64 addon invocation set on `ubuntu-22.04`. Main-branch input changes restore the previous config-compatible generation, rebuild incrementally, and publish one combined exact-key disk-cache archive visible to pull requests.
`.github/workflows/bazel-cache-warm.yml` seeds the GitHub-hosted caches that have no other reliable producer: the `release-darwin-*` bazel disk caches (built on the same macOS images as the `release_binary` darwin matrix, so a release's bazel build is the version-bump delta instead of a ~40-min cold graph) and the shared bun store entry PR jobs restore but never save. It triggers only on pushes that can change those archives (crate/bazel/lock inputs, `bun.lock`, `.github/**`).
### `bazel-cache` action (`.github/actions/bazel-cache`)
@@ -176,15 +173,15 @@ Single source of truth for cache wiring, emitted as a bazelrc fragment (its `rc`
| omp-kata pod | `--config=ci --config=cache-rw --remote_cache=grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092 --tls_certificate=infra/bazel-remote/ca.crt --remote_header='authorization=Basic <b64 ci creds>'` |
| GitHub-hosted | `--config=ci --disk_cache=~/.cache/omp-bazel-disk --repository_cache=~/.cache/omp-bazel-repo` |
Hosted disk caches use `bazel-disk-v3-<scope>-<os>-<arch>-<config-hash>-<source-hash>`. The config hash covers Cargo/Bazel/toolchain settings; the source hash covers `crates/**` and root `BUILD.bazel`. An exact miss restores the newest config-compatible generation and permits one refreshed exact-key save. The remote endpoint resolves only inside the cluster.
Hosted disk caches use `bazel-disk-v3-<scope>-<os>-<arch>-<config-hash>-<source-hash>`. The config hash covers Cargo/Bazel/toolchain settings; the source hash covers `crates/**` and root `BUILD.bazel`. Restores fall back from the exact key to the config-scoped prefix, then to a bare `<scope>-<os>-<arch>` prefix — the bare fallback is what keeps release version bumps (which rewrite `Cargo.toml`/`Cargo.lock` and thus the config hash) from rebuilding cold; bazel's content-addressed action keys make a stale archive a partial hit, never a wrong output. An inexact restore permits one refreshed exact-key save, and restored entries untouched for 14 days are pruned so archives don't grow without bound. The remote endpoint resolves only inside the cluster.
### Native artifact actions
`.github/actions/bazel-natives` is the direct builder: `bazel-cache` → `OMP_BAZEL_RC=<rc> bun scripts/bazel-natives.ts <targets> --dest <dest>`, followed by a disk-cache save after a hosted miss. `.github/actions/native-artifacts` is the no-build consumer: download `native-addons` → run the same driver with `--source`.
### `release_binary`
### Release binary builds and publishing
Linux and Windows release matrices install addons from the `native_addons` workflow artifact. Darwin artifacts cannot be cross-built on Linux, so each macOS matrix builds only its own architecture through `bazel-natives` with scope `release-<target_id>`, then `bun run ci:release:build-binaries` embeds and compiles the executable.
Binary builds are build-only and run in parallel with the test fan-out. `release_binary` (Linux + Windows matrices) needs only `native_addons`, whose workflow artifact supplies their addons. `release_binary_darwin` needs only `release_metadata` and starts the moment a release run is detected: darwin artifacts cannot be cross-built on Linux, so each macOS leg builds its own architecture through `bazel-natives` with scope `release-<target_id>` (seeded near HEAD by the warm workflow — normally just the version-bump delta), then `bun run ci:release:build-binaries` embeds and compiles the executable. Publishing is held behind `release_gate` (the aggregate of every validation job): `release_native_leaves` downloads all built addons and publishes the five `@oh-my-pi/pi-natives-<tag>` leaves from one linux runner, and the GitHub release / verify / core npm chain runs beside it.
## Debugging playbook
@@ -228,7 +225,7 @@ bazelisk build --nobuild //:natives-win32-x64-baseline
### Cache behavior
- **omp-kata:** read-write gRPC to the in-cluster bazel-remote (`grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092`, TLS via the committed `infra/bazel-remote/ca.crt`, htpasswd user `ci`). `--remote_local_fallback` plus retries make an outage degrade to local execution rather than fail the build.
- **GitHub-hosted:** no cluster access. The v3 `actions/cache` disk key separates config and source generations; `.github/workflows/bazel-cache-warm.yml` publishes the combined default-branch archive from the same `ubuntu-22.04` image as pull-request consumers. The smaller final-addon cache is independent and is trusted only after both addons load successfully.
- **GitHub-hosted:** no cluster access; only the darwin release/warm jobs build with bazel here. The v3 `actions/cache` disk key separates config and source generations with prefix + bare fallbacks (see the `bazel-cache` action section above); `.github/workflows/bazel-cache-warm.yml` publishes the `release-darwin-*` archives from the same macOS images as the release consumers.
- **msvc repos:** the ~2 GiB LLVM download is sha256-pinned and repository-cache backed; the ~1 GiB xwin CRT/SDK splat is fetched from the Microsoft CDN inside the repo rule and is **not** repo-cache backed — a cold output base re-downloads it. Microsoft advances the VS channel payload over time, so remote-cache hit rates for win32 actions degrade gracefully after an MS bump (same property the previous cross toolchain had). Win32 link actions also don't share cache entries across host OSes (linux vs mac clang binaries).
- Server-side operations (deploy, TLS/auth, egress, poisoning boundary): `infra/docs/04-arc-and-caching.md` §5.
docs/tools/edit.md
+34 -30
View File
@@ -8,7 +8,7 @@
- Key collaborators:
- `packages/coding-agent/src/utils/edit-mode.ts` — selects active edit mode
- `packages/hashline/src/grammar.lark` — canonical constrained-decoding grammar
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `DEL`, `INS`)
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `CUT`, `INS`, `PASTE`)
- `packages/hashline/src/input.ts` — parses `[PATH#TAG]` sections
- `packages/hashline/src/tokenizer.ts` / `packages/hashline/src/parser.ts` — tokenizes and parses ops
- `packages/hashline/src/apply.ts` — applies parsed edits to file text
@@ -30,18 +30,21 @@ Patch language inside `input`:
- **Operations**:
- `SWAP N.=M:` — replace original lines N.=M with the body rows below.
- `SWAP.BLK N:` — replace the whole tree-sitter block beginning on line N (its header line through its closing line) with the body rows. The line span is resolved at apply time from the file's parse tree; point N at the line that opens the construct. The resolved span is exactly the node that begins on line N — a leading decorator, attribute, or doc-comment is a separate node and is not included; point N at the first decorator line (Python wraps `@dec` + `def` as one block) or fall back to `SWAP N.=M:` to take a leading line-comment that parses as its own node (e.g. Rust `///`). On success the result echoes the matched span (`SWAP.BLK N → resolved lines A-B`). Errors (and steers to `SWAP N.=M:`) when the language is unsupported, line N is blank or a closing delimiter, no node begins there, or the resolved block has a syntax error.
- `DEL N.=M` — delete original lines N.=M. No body.
- `DEL.BLK N` — delete the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`DEL.BLK N → resolved lines A-B`). Same resolution failure modes and `DEL N.=M` fallback.
- `CUT N.=M` — delete original lines N.=M and capture them in the clipboard. No body. A standalone cut is valid; the latest cut replaces the clipboard contents.
- `CUT.BLK N` — delete and capture the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`CUT.BLK N → resolved lines A-B`). Same resolution failure modes and `CUT N.=M` fallback.
- `INS.PRE N:` — insert body rows immediately before line N.
- `INS.POST N:` — insert body rows immediately after line N.
- `INS.BLK.POST N:` — insert body rows after the last line of the tree-sitter block beginning on line N. Point N at the line that opens the construct, never its closing delimiter / last visible line; if you can see the last line already, use plain `INS.POST M:`. An anchor that can't resolve to a block is lowered to plain `INS.POST N:` with a warning instead of failing the patch.
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all three block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `DEL.BLK` drops the section, `SWAP.BLK` rewrites it, `INS.BLK.POST` lands after it. A heading with no body resolves to a single line and falls back to the plain op like any other single-line block.
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all four block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `CUT.BLK` drops and captures the section, `SWAP.BLK` rewrites it, and `INS.BLK.POST` / `PASTE.BLK.POST` land after it. A heading with no body resolves to a single line and is rejected with guidance to use the corresponding plain line op.
- `INS.HEAD:` — insert body rows at the start of the file.
- `INS.TAIL:` — insert body rows at the end of the file.
- `PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` / `PASTE.TAIL` — insert the clipboard at that position. No body. An empty clipboard is an error.
- `PASTE.BLK.POST N` — insert the clipboard after the resolved block's last line. An unresolvable anchor lowers to `PASTE.POST N` with a warning, matching `INS.BLK.POST`.
- **Clipboard**: operations execute top-to-bottom across all patch sections. The latest `CUT` wins; `PASTE` does not consume the clipboard and may be repeated. The coding agent persists the register across edit calls in the same session, enabling cross-file moves. Keep each path under one header when clipboard operations would otherwise be interleaved around another file's section.
- **Body rows**:
- Only body-bearing headers end in `:`.
- Every body row is `+TEXT`; `+` alone adds a blank line.
- `DEL` never has body rows.
- `CUT` and `PASTE` never have body rows.
- There is no repeat row kind. To keep a line, leave it out of every range; split edits into multiple hunks when needed.
- `-` rows are invalid. Literal Markdown bullets or text beginning with `-` / `+` must be written as `+- item` / `++ item`.
@@ -52,7 +55,7 @@ Anchors come from `read`/`grep` output. `read` emits a `[PATH#TAG]` header from
The canonical grammar is strict, but the hand parser accepts a few non-dangerous variants:
- `SWAP N:` — accepted as `SWAP N.=N:`.
- `DEL N` — accepted as single-line delete.
- `CUT N` — accepted as a single-line cut/delete.
- Missing trailing colon on `SWAP` or `INS` — accepted.
- `SWAP N-M:`, `SWAP N…M:`, `SWAP N M:`, and legacy `SWAP N..M:` — accepted as `SWAP N.=M:`.
- Bare body rows with no `+` prefix are auto-prepended with `+` and a `BARE_BODY_AUTO_PIPED_WARNING` is appended.
@@ -61,16 +64,17 @@ The canonical grammar is strict, but the hand parser accepts a few non-dangerous
- Some malformed bracketed headers are recovered after stripping apply-patch path noise such as `Update File:` / `Add File:` and extra `***`, but the recovered header still needs a valid four-hex tag for the patcher to apply it.
- `*** Update File:` / `*** Add File:` / `*** Delete File:` / `*** Move to:` apply_patch sentinels inside the diff body throw an `apply_patch sentinel … is not valid in hashline` error.
- `@@`-bracketed hunk headers are rejected with guidance to write a verb header.
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `DEL`.
- `DEL N.=M:` and any body rows under `DEL` / `DEL.BLK` are rejected.
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` (no body rows) is treated as `DEL N.=M`.
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `CUT`.
- A trailing colon on `CUT N.=M:` / `CUT.BLK N:` is tolerated and ignored, but body rows under `CUT`, `CUT.BLK`, or any `PASTE` form are rejected.
- Bare `PASTE` is rejected because the insertion position is required.
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` deletes the range, though `CUT N.=M` is the canonical deletion form.
- `-` body rows are rejected with `MINUS_ROW_REJECTED` unless the hunk is unambiguously a Markdown bullet list (see Tolerated input shapes).
- `SWAP.BLK N:` / `DEL.BLK N` / `INS.BLK.POST N:` require a wired tree-sitter resolver; `SWAP.BLK` and `INS.BLK.POST` additionally need at least one `+TEXT` body row, while `DEL.BLK` takes none. An unresolvable block (unsupported language, blank/closing-delimiter line, no node beginning on N, or a syntax error in the resolved block) rejects a `SWAP.BLK` / `DEL.BLK` on the apply/final-preview path (the streaming preview silently drops it instead). `INS.BLK.POST N:` is never rejected this way — it is lowered to plain `INS.POST N:` with a warning: a closing-delimiter-anchor warning when line N is a pure closer (inserting after that end is exactly what the plain form does), a generic unresolved-anchor warning otherwise.
- `SWAP.BLK N:` / `CUT.BLK N` / `INS.BLK.POST N:` / `PASTE.BLK.POST N` consult the wired tree-sitter resolver. `SWAP.BLK` and `INS.BLK.POST` need at least one `+TEXT` body row; `CUT.BLK` and `PASTE.BLK.POST` take none. A null resolution rejects `SWAP.BLK` / `CUT.BLK` on the apply or final-preview path (the streaming preview silently drops it), while `INS.BLK.POST` / `PASTE.BLK.POST` lower to the corresponding plain `POST` form with a warning. A single-line resolution rejects every block form with guidance to use its plain line equivalent.
## Outputs
- Single-shot tool result; hashline mode does not use the staged preview/apply devices (`/xdev/resolve`, `/xdev/reject`).
- `content` contains one text block per call. For a successful single-file edit it is the post-edit `[path#TAG]` section header (a fresh snapshot tag for the written content), followed by a compact diff preview from `packages/hashline/src/diff-preview.ts` when one is emitted.
- When the patch used `SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST` ops (and the apply matched the tagged content), one `SWAP.BLK N → resolved lines A-B (K lines)` line per block op (single-line spans render `resolved line A (1 line)`; INS.BLK.POST appends `; body lands after line B`) is inserted between the `[PATH#TAG]` header and the diff preview, so the caller can confirm tree-sitter resolved the construct it intended.
- When the patch used `SWAP.BLK` / `CUT.BLK` / `INS.BLK.POST` / `PASTE.BLK.POST` ops (and the apply matched the tagged content), one `<OP> N → resolved lines A-B (K lines)` line per block op is inserted between the `[PATH#TAG]` header and the diff preview. Single-line spans render `resolved line A (1 line)`; `INS.BLK.POST` appends `body lands after line B`, and `PASTE.BLK.POST` appends `clipboard lands after line B`.
- Parse, apply, or recovery warnings are appended as:
```text
@@ -126,11 +130,11 @@ INS.PRE 5:
+console.log(X + Y);
```
Delete lines 4.=5 entirely:
Delete lines 4.=5 entirely and leave them in the clipboard:
```text
[a.ts#0A3B]
DEL 4.=5
CUT 4.=5
```
Insert at start and end of file:
@@ -143,20 +147,19 @@ INS.TAIL:
+// trailer
```
Multi-file:
Move line 4 from `src/a.ts` to after line 20 in `src/b.ts`:
```text
[src/a.ts#0A3B]
SWAP 4.=4:
+const enabled = true;
CUT 4
[src/b.ts#1F7C]
DEL 20
PASTE.POST 20
```
## Limits & Caps
- File snapshot tags are exactly four uppercase-hex chars — content-derived hashes (`computeFileHash()`) recorded in the per-session snapshot store.
- The visible mismatch report shows 2 lines of context on each side (`MISMATCH_CONTEXT`) in `packages/hashline/src/messages.ts`.
- Stale-anchor recovery uses `fuzzFactor: 0` in `packages/hashline/src/recovery.ts`.
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and hunk keyword constants are `SWAP` / `DEL` / `INS` (`packages/hashline/src/format.ts`).
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and line/clipboard hunk keyword constants are `SWAP` / `CUT` / `INS` / `PASTE` (`packages/hashline/src/format.ts`).
## Errors
- Missing section header:
@@ -164,28 +167,29 @@ DEL 20
- Missing tag for any section:
- `Missing hashline snapshot tag for <path>; use \`[<path>#tag]\` from your latest read/search output. To create a new file, use the write tool.`
- Stray payload line:
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
- Minus row (unless auto-piped as an unambiguous Markdown bullet — see Tolerated input shapes):
- ``line N: `-` rows are not valid; the range already names the lines being changed. For Markdown bullets or other literal `-` lines, prefix the literal row with `+`: `+- item`.``
- Empty body-bearing hunk:
- `line N: \`INS\` needs at least one \`+TEXT\` body row.`
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`DEL.BLK N\`.`
- Unresolvable block anchor — `SWAP.BLK` / `DEL.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by a blank line and numbered `*`-marked context rows around line X (same shape as the mismatch preview). `DEL.BLK X` produces the same message with a `DEL X.=M` fallback.
- `INS.BLK.POST X:` never reaches this error — an unresolvable anchor is lowered to plain `INS.POST X:` with a warning (see Tolerated input shapes).
- Delete with body:
- `line N: \`DEL N.=M\` does not take body rows. Remove the body, or use \`SWAP N.=M:\`.`
- `line N: \`DEL.BLK N\` does not take body rows. Remove the body, or use \`SWAP.BLK N:\`.`
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`CUT.BLK N\`.`
- Unresolvable block anchor — `SWAP.BLK` / `CUT.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by numbered context and, when available, a nearby block suggestion. `CUT.BLK X` produces the corresponding message with a `CUT X.=M` fallback.
- `INS.BLK.POST X:` and `PASTE.BLK.POST X` never reach this error when no block resolves — they lower to plain `INS.POST X:` / `PASTE.POST X` with a warning.
- Clipboard operation errors:
- `line N: \`CUT N.=M\` captures + deletes lines and takes no body rows. To replace lines with new content, use \`SWAP N.=M:\`.`
- `line N: \`PASTE\` inserts the clipboard content and takes no \`+\` body rows. To insert literal text, use \`INS\`.`
- `line N: \`PASTE\` found nothing in the clipboard. Ops run top-to-bottom across the whole patch (sections included): put \`CUT N.=M\` or \`CUT.BLK N\` above the \`PASTE\`.`
- Range out of order:
- `line N: range A.=B ends before it starts.`
- Overlapping hunks on the same anchor:
- `line N: anchor line X is already targeted by another hunk on line Y. Issue ONE hunk per range; payload is only the final desired content, never a before/after pair.`
- apply_patch / unified-diff contamination:
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: \`@@\`-bracketed hunk header "@@ …" is not valid in hashline. Drop the \`@@ ... @@\` brackets and write a verb header such as \`SWAP N.=M:\`.`
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`DEL N\` to delete.`
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`DEL ${bareRange[1]}.=${bareRange[2]}\`.`
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`CUT N\` to delete.`
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`CUT ${bareRange[1]}.=${bareRange[2]}\`.`
- Out-of-range anchor:
- `Line N does not exist (file has M lines)`
- Stale snapshot tag: the `Patcher` first attempts snapshot-based recovery. When recovery cannot prove a valid result it throws `MismatchError`, which distinguishes recognized-but-drifted hashes from never-recorded hashes. The error includes the current file hash plus context around each anchor.
infra/docs/04-arc-and-caching.md
+26 -16
View File
@@ -161,7 +161,7 @@ githubConfigUrl: "https://github.com/<OWNER>/<REPO>"
githubConfigSecret: arc-github
runnerScaleSetName: omp-kata
minRunners: 0
maxRunners: 4
maxRunners: 8
# none: each job runs inside the runner container, which itself lives in a Kata microVM
containerMode:
type: ""
@@ -201,12 +201,15 @@ template:
mountPath: /opt/bazel-repo-cache
subPath: bazel-repo-cache
resources:
# Burstable on purpose: requests bin-pack 8 runners onto the
# 32-vCPU / 125 GiB host; limits are each Kata VM's hotplug
# ceiling. Keep sum(memory limits) under host RAM.
requests:
cpu: "8"
memory: "24Gi"
cpu: "3"
memory: "10Gi"
limits:
cpu: "8"
memory: "24Gi"
memory: "14Gi"
volumes:
- name: runner-cache
persistentVolumeClaim:
@@ -220,10 +223,14 @@ Field by field:
- **`githubConfigSecret: arc-github`** - the auth secret from [step 1](#1-github-app-and-the-arc-github-secret).
- **`runnerScaleSetName: omp-kata`** - the runner label. This is the string that
goes in a workflow's `runs-on:`.
- **`minRunners: 0` / `maxRunners: 4`** - **scale-to-zero**. With no queued jobs
there are zero runner microVMs. Each admitted runner gets an honest 8-vCPU,
24-GiB request and limit; excess jobs queue instead of ten 16-vCPU guests
fighting over the reference host's 32 physical CPUs.
- **`minRunners: 0` / `maxRunners: 8`** - **scale-to-zero**. With no queued jobs
there are zero runner microVMs. Runner pods are **burstable**: a small
request (3 vCPU / 10 GiB) bin-packs eight runners onto the reference host,
while the limit (8 vCPU / 14 GiB) is each Kata VM's hotplug ceiling, so a
lone heavy job still gets 8 vCPUs. Keep the sum of memory *limits* under
host RAM — host OOM under Kata kills VMs unpredictably. (The original
guaranteed sizing, 4 x 8 vCPU / 24 GiB requests=limits, reserved the whole
host and queued every >4-job workflow fan-out for minutes.)
- **`containerMode.type: ""`** - **none**. The default chart offers `dind`
(Docker-in-Docker sidecar) or `kubernetes` mode for job-container isolation;
both are unnecessary here because the *whole runner pod* is already isolated in
@@ -255,14 +262,17 @@ Field by field:
mounts to the `arc-runners/runner-cache` PVC. `ReadWriteOnce` is enough on this
single-node k3s host; use a RWX-capable storage class before spreading runners
across nodes.
- **`resources`** - requests `8` CPU / `24Gi`, limits `8` CPU / `24Gi`. Kata reads
these and sizes the guest accordingly: the VM now boots at the same
guaranteed floor (`default_vcpus: 2`, `default_memory: 4096`) and only
hotplugs beyond that toward the limits, with `default_maxvcpus: 0` allowing up
to all host CPUs. Effectively the **requests are the boot-time VM size** and
the **limits are the hotplug ceiling**. See [02-kata-runtime.md](02-kata-runtime.md)
for the runtime knobs and [`infra/tune-kata-runtime.sh`](../tune-kata-runtime.sh)
for the SSH-driven patch helper.
- **`resources`** - requests `3` CPU / `10Gi`, limits `8` CPU / `14Gi`
(burstable; see the `maxRunners` bullet above). Kata sizes the guest from
these: every VM boots at the fixed floor from the runtime config
(`default_vcpus: 2`, `default_memory: 4096` — deliberately at or below the
pod request so boot stays cheap) and hotplugs beyond it toward the pod
**limits**, with `default_maxvcpus: 0` allowing up to all host CPUs.
Effectively the **boot shape is a fixed floor**, the **requests are the
scheduler's bin-packing unit**, and the **limits are the hotplug ceiling**.
See [02-kata-runtime.md](02-kata-runtime.md) for the runtime knobs and
[`infra/tune-kata-runtime.sh`](../tune-kata-runtime.sh) for the SSH-driven
patch helper.
---
infra/docs/README.md
+1 -1
View File
@@ -47,7 +47,7 @@ flowchart LR
Key properties baked into this design:
- **One job = one VM.** Runner pods are ephemeral and JIT-registered; there is no VM templating or pooling, so a job never inherits state from a previous job.
- **Scale-to-zero.** `minRunners: 0` / `maxRunners: 4` — when no jobs are queued, zero runner pods (and zero microVMs) exist.
- **Scale-to-zero.** `minRunners: 0` / `maxRunners: 8` — when no jobs are queued, zero runner pods (and zero microVMs) exist. Runners are burstable (3-vCPU/10-GiB requests, 8-vCPU/14-GiB limits) so a full workflow fan-out runs 8-wide while a lone heavy job still bursts to 8 vCPUs.
- **Host-kernel isolation.** Jobs see the microVM's guest kernel, not the host kernel, so a kernel exploit in a job does not reach the host.
- **No external registry.** The runner image is built on the host and imported straight into k3s' containerd.
- **Shared, in-cluster cache.** bazel-remote stores Bazel action results and CAS blobs (Rust compilation, tests, native `.node` addons) behind TLS + htpasswd auth, with a public read-mostly NodePort for GitHub-hosted runners; the runner-cache PVC stores Bun/Cargo downloads. Cache traffic stays on the host.
infra/reload-runner.sh
+26 -13
View File
@@ -32,9 +32,20 @@
# CONTAINERD_SOCKET_REMOTE remote containerd socket [/run/k3s/containerd/containerd.sock]
# NERDCTL_VERSION nerdctl release to bootstrap on demand [2.1.6]
# BUILDKIT_VERSION BuildKit release to bootstrap on demand [0.25.1]
# RUNNER_MAX_RUNNERS maximum concurrent Kata runner pods [4]
# RUNNER_CPU requested and limited CPU cores per runner [8]
# RUNNER_MEMORY requested and limited memory per runner [24Gi]
# RUNNER_MAX_RUNNERS maximum concurrent Kata runner pods [8]
# RUNNER_CPU_REQUEST requested CPU cores per runner [3]
# RUNNER_CPU_LIMIT CPU-core limit per runner (Kata hotplug ceiling) [8]
# RUNNER_MEMORY_REQUEST requested memory per runner [10Gi]
# RUNNER_MEMORY_LIMIT memory limit per runner (Kata hotplug ceiling) [14Gi]
#
# Runner pods are deliberately BURSTABLE, not guaranteed: requests size the
# scheduler's bin-packing (8 x 3 cpu / 10Gi fits the 32-vCPU / 125 GiB host
# with headroom), while limits set each Kata VM's hotplug ceiling so a lone
# native build still gets 8 vCPUs. requests==limits previously capped the
# host at 4 runners and queued every >4-job workflow fan-out for minutes.
# Worst-case sum of memory limits (8 x 14Gi = 112Gi) stays under the host's
# 125 GiB because host OOM under Kata kills VMs unpredictably — keep it that
# way when retuning.
set -euo pipefail
: "${CI_HOST:?set CI_HOST to the ssh target of your CI host, e.g. CI_HOST=my-ci-host}"
@@ -48,9 +59,11 @@ BUILD_BACKEND="${BUILD_BACKEND:-auto}"
CONTAINERD_SOCKET_REMOTE="${CONTAINERD_SOCKET_REMOTE:-/run/k3s/containerd/containerd.sock}"
NERDCTL_VERSION="${NERDCTL_VERSION:-2.1.6}"
BUILDKIT_VERSION="${BUILDKIT_VERSION:-0.25.1}"
RUNNER_MAX_RUNNERS="${RUNNER_MAX_RUNNERS:-4}"
RUNNER_CPU="${RUNNER_CPU:-8}"
RUNNER_MEMORY="${RUNNER_MEMORY:-24Gi}"
RUNNER_MAX_RUNNERS="${RUNNER_MAX_RUNNERS:-8}"
RUNNER_CPU_REQUEST="${RUNNER_CPU_REQUEST:-3}"
RUNNER_CPU_LIMIT="${RUNNER_CPU_LIMIT:-8}"
RUNNER_MEMORY_REQUEST="${RUNNER_MEMORY_REQUEST:-10Gi}"
RUNNER_MEMORY_LIMIT="${RUNNER_MEMORY_LIMIT:-14Gi}"
arg="${1:-$(date +%Y-%m-%d-%H%M%S)}"
case "$arg" in *:*) IMAGE="$arg";; *) IMAGE="omp-kata-runner:$arg";; esac
@@ -70,12 +83,12 @@ scp -q "$here/runner.Dockerfile" "${CI_HOST}:${REMOTE_CTX}/Dockerfile"
ssh "$CI_HOST" bash -s -- \
"$IMAGE" "$REMOTE_CTX" "$ARC_VALUES" "$ARC_RELEASE" "$ARC_NAMESPACE" "$ARC_CHART_VERSION" \
"$KUBECONFIG_REMOTE" "$BUILD_BACKEND" "$CONTAINERD_SOCKET_REMOTE" "$NERDCTL_VERSION" "$BUILDKIT_VERSION" \
"$RUNNER_MAX_RUNNERS" "$RUNNER_CPU" "$RUNNER_MEMORY" <<'REMOTE'
"$RUNNER_MAX_RUNNERS" "$RUNNER_CPU_REQUEST" "$RUNNER_CPU_LIMIT" "$RUNNER_MEMORY_REQUEST" "$RUNNER_MEMORY_LIMIT" <<'REMOTE'
set -euo pipefail
IMAGE="$1"; REMOTE_CTX="$2"; ARC_VALUES="$3"; ARC_RELEASE="$4"; ARC_NAMESPACE="$5"; ARC_CHART_VERSION="$6"
export KUBECONFIG="$7"
BUILD_BACKEND="$8"; CONTAINERD_SOCKET="$9"; NERDCTL_VERSION="${10}"; BUILDKIT_VERSION="${11}"
RUNNER_MAX_RUNNERS="${12}"; RUNNER_CPU="${13}"; RUNNER_MEMORY="${14}"
RUNNER_MAX_RUNNERS="${12}"; RUNNER_CPU_REQUEST="${13}"; RUNNER_CPU_LIMIT="${14}"; RUNNER_MEMORY_REQUEST="${15}"; RUNNER_MEMORY_LIMIT="${16}"
cd "$REMOTE_CTX"
TOOLS_DIR="$REMOTE_CTX/.containerd-build-tools"
@@ -244,10 +257,10 @@ if ! grep -q 'name: bazel-remote-ci' "$ARC_VALUES"; then
fi
helm upgrade "$ARC_RELEASE" --namespace "$ARC_NAMESPACE" --version "$ARC_CHART_VERSION" \
-f "$ARC_VALUES" \
--set-string "template.spec.containers[0].resources.requests.cpu=$RUNNER_CPU" \
--set-string "template.spec.containers[0].resources.limits.cpu=$RUNNER_CPU" \
--set-string "template.spec.containers[0].resources.requests.memory=$RUNNER_MEMORY" \
--set-string "template.spec.containers[0].resources.limits.memory=$RUNNER_MEMORY" \
--set-string "template.spec.containers[0].resources.requests.cpu=$RUNNER_CPU_REQUEST" \
--set-string "template.spec.containers[0].resources.limits.cpu=$RUNNER_CPU_LIMIT" \
--set-string "template.spec.containers[0].resources.requests.memory=$RUNNER_MEMORY_REQUEST" \
--set-string "template.spec.containers[0].resources.limits.memory=$RUNNER_MEMORY_LIMIT" \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set >/dev/null
echo "==> [5/5] verifying rollout"
@@ -256,7 +269,7 @@ live="$(kubectl get autoscalingrunnerset "$ARC_RELEASE" -n "$ARC_NAMESPACE" \
live_max="$(kubectl get autoscalingrunnerset "$ARC_RELEASE" -n "$ARC_NAMESPACE" -o jsonpath='{.spec.maxRunners}')"
live_resources="$(kubectl get autoscalingrunnerset "$ARC_RELEASE" -n "$ARC_NAMESPACE" \
-o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}/{.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory}/{.spec.template.spec.containers[0].resources.limits.memory}')"
expected_resources="$RUNNER_CPU/$RUNNER_CPU $RUNNER_MEMORY/$RUNNER_MEMORY"
expected_resources="$RUNNER_CPU_REQUEST/$RUNNER_CPU_LIMIT $RUNNER_MEMORY_REQUEST/$RUNNER_MEMORY_LIMIT"
echo "ARC runner image/resources: $live | max=$live_max | $live_resources"
[ "$live" = "$IMAGE" ] || { echo "MISMATCH: expected image $IMAGE"; exit 1; }
[ "$live_max" = "$RUNNER_MAX_RUNNERS" ] || { echo "MISMATCH: expected maxRunners $RUNNER_MAX_RUNNERS"; exit 1; }
infra/tune-kata-runtime.sh
+7 -2
View File
@@ -1,6 +1,11 @@
#!/usr/bin/env bash
# Patch the live Kata QEMU config to match the runner pod's guaranteed boot
# shape, raise guest and host open-file limits, and enlarge the virtiofsd worker pool.
# Patch the live Kata QEMU config: set the guest BOOT floor (default_vcpus /
# default_memory), raise guest and host open-file limits, and enlarge the
# virtiofsd worker pool. Runner pods are burstable (see reload-runner.sh):
# the boot floor stays deliberately at or below the pod's CPU/memory REQUEST
# so VMs boot small and fast, and Kata hotplugs each guest toward the pod
# LIMIT under load. Keep BOOT_VCPUS/BOOT_MEMORY_MIB <= the request when
# retuning either side.
# The script runs over SSH, which keeps the desired values version-controlled.
# It then smoke-tests a new kata-qemu pod.
#
package.json
+12 -12
View File
@@ -26,18 +26,18 @@
"@huggingface/transformers": "^4.2.0",
"@mozilla/readability": "^0.6.0",
"@napi-rs/cli": "3.7.2",
"@oh-my-pi/hashline": "17.1.8",
"@oh-my-pi/omp-stats": "17.1.8",
"@oh-my-pi/pi-agent-core": "17.1.8",
"@oh-my-pi/pi-ai": "17.1.8",
"@oh-my-pi/pi-catalog": "17.1.8",
"@oh-my-pi/pi-coding-agent": "17.1.8",
"@oh-my-pi/pi-mnemopi": "17.1.8",
"@oh-my-pi/pi-natives": "17.1.8",
"@oh-my-pi/pi-tui": "17.1.8",
"@oh-my-pi/pi-utils": "17.1.8",
"@oh-my-pi/pi-wire": "17.1.8",
"@oh-my-pi/snapcompact": "17.1.8",
"@oh-my-pi/hashline": "17.2.0",
"@oh-my-pi/omp-stats": "17.2.0",
"@oh-my-pi/pi-agent-core": "17.2.0",
"@oh-my-pi/pi-ai": "17.2.0",
"@oh-my-pi/pi-catalog": "17.2.0",
"@oh-my-pi/pi-coding-agent": "17.2.0",
"@oh-my-pi/pi-mnemopi": "17.2.0",
"@oh-my-pi/pi-natives": "17.2.0",
"@oh-my-pi/pi-tui": "17.2.0",
"@oh-my-pi/pi-utils": "17.2.0",
"@oh-my-pi/pi-wire": "17.2.0",
"@oh-my-pi/snapcompact": "17.2.0",
"@opentelemetry/api": "^1.9.1",
"@opentelemetry/api-logs": "^0.220.0",
"@opentelemetry/context-async-hooks": "^2.9.0",
packages/agent/CHANGELOG.md
+2
View File
@@ -2,6 +2,8 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Fixed
- Provider-native compaction failures now surface their transport error instead of silently switching to generic summarization; streaming V2 still falls back to native V1 when available.
packages/agent/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"type": "module",
"name": "@oh-my-pi/pi-agent-core",
"version": "17.1.8",
"version": "17.2.0",
"description": "General-purpose agent with transport abstraction, state management, and attachment support",
"homepage": "https://omp.sh",
"author": "Can Boluk",
packages/ai/CHANGELOG.md
+3
View File
@@ -2,6 +2,8 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Added
- Added first-class parentTurnId support for nested Codex requests, allowing stream options and metadata helpers to accept and safely propagate the initiating turn's ID.
@@ -10,6 +12,7 @@
- Added interactive Exa API-key login through `/login exa`, opening the official API-key dashboard and saving pasted keys to the credential store ([#1798](https://github.com/can1357/oh-my-pi/issues/1798)).
- Cursor's modern exec wire protocol is now handled end to end. `agent.proto` models the frames current Cursor CLI builds emit — the seven Pi tools (`ExecServerMessage` 45-51), hooks, subagents, allowlist prechecks, MCP state, smart-mode classification, canvas diagnostics, conversation search, agent-store conflicts and git diff — and every one of them gets a typed answer. The Pi frames run their local equivalents (`read`/`bash`/`edit`/`write`/`grep`/`glob`); the rest answer with the error, not-found or empty-but-valid variant that is actually true of this client. Frames this build cannot name at all now raise `ExecClientControlMessage.throw` with `unknown_exec_variant`, and recognised frames with no truthful answer (`git_diff_request`, whose `GetDiffResponse` has no error variant) raise `exec_variant_unsupported`, instead of a silent ack that leaves the server waiting.
- `lsp` is advertised in the MCP tool catalog again. It was filtered out as a Cursor-native tool, but the native `diagnostics` frame covers one of roughly ten LSP actions, so the other nine were unreachable.
- Added `pinSessionOAuthAccount` support for backdating the sticky's last-use timestamp (`options.lastUsedAtMs`), so pins restored from persisted sessions keep the provider's warm-window semantics: resumes inside the prompt-cache TTL reuse the account, stale resumes still re-rank.
### Changed
packages/ai/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"type": "module",
"name": "@oh-my-pi/pi-ai",
"version": "17.1.8",
"version": "17.2.0",
"description": "Unified LLM API with automatic model discovery and provider configuration",
"homepage": "https://omp.sh",
"author": "Can Boluk",
packages/ai/src/auth-storage.ts
+19 -4
View File
@@ -1811,15 +1811,20 @@ export class AuthStorage {
}
}
/** Records which credential was used for a session (for rate-limit switching). */
/**
* Records which credential was used for a session (for rate-limit switching).
* `lastUsedAtMs` backdates the sticky (session-file pin restores on resume);
* it defaults to now for live selections.
*/
#recordSessionCredential(
provider: string,
sessionId: string | undefined,
type: AuthCredential["type"],
index: number,
lastUsedAtMs?: number,
): void {
if (!sessionId) return;
const nowMs = Date.now();
const nowMs = lastUsedAtMs ?? Date.now();
const sessionMap = this.#sessionLastCredential.get(provider) ?? new Map();
sessionMap.set(sessionId, { type, index, lastUsedAtMs: nowMs });
this.#sessionLastCredential.set(provider, sessionMap);
@@ -5403,8 +5408,18 @@ export class AuthStorage {
* The durable credential id keeps the pin stable across credential refreshes,
* storage reordering, and process restarts. Normal auth retry and usage-limit
* handling may still route around an unavailable account.
*
* `options.lastUsedAtMs` backdates the sticky's last-use timestamp so a pin
* restored from a persisted session keeps the provider's warm-window
* semantics: a resume inside the prompt-cache TTL reuses the account, a
* stale resume still re-ranks.
*/
pinSessionOAuthAccount(provider: string, sessionId: string, credentialId: number): boolean {
pinSessionOAuthAccount(
provider: string,
sessionId: string,
credentialId: number,
options?: { lastUsedAtMs?: number },
): boolean {
if (!sessionId || this.#runtimeOverrides.has(provider) || this.#configOverrides.has(provider)) {
return false;
}
@@ -5412,7 +5427,7 @@ export class AuthStorage {
const index = stored.findIndex(entry => entry.id === credentialId);
const target = stored[index];
if (target?.credential.type !== "oauth") return false;
this.#recordSessionCredential(provider, sessionId, "oauth", index);
this.#recordSessionCredential(provider, sessionId, "oauth", index, options?.lastUsedAtMs);
return true;
}
packages/ai/src/providers/openai-codex-responses.ts
+1 -1
View File
@@ -252,7 +252,7 @@ const CODEX_WEBSOCKET_IDLE_TIMEOUT_MS = Number($env.PI_CODEX_WEBSOCKET_IDLE_TIME
* SSE. Generous default so legitimately slow first-token providers still get
* a chance on the WS transport before falling through.
*/
const CODEX_WEBSOCKET_FIRST_EVENT_TIMEOUT_MS = Number($env.PI_CODEX_WEBSOCKET_FIRST_EVENT_TIMEOUT_MS || 60_000);
const CODEX_WEBSOCKET_FIRST_EVENT_TIMEOUT_MS = Number($env.PI_CODEX_WEBSOCKET_FIRST_EVENT_TIMEOUT_MS || 300_000);
const CODEX_WEBSOCKET_RETRY_BUDGET = Number($env.PI_CODEX_WEBSOCKET_RETRY_BUDGET || CODEX_MAX_RETRIES);
const CODEX_WEBSOCKET_RETRY_DELAY_MS = Number($env.PI_CODEX_WEBSOCKET_RETRY_DELAY_MS || CODEX_RETRY_DELAY_MS);
const CODEX_WEBSOCKET_TRANSPORT_ERROR_PREFIX = "Codex websocket transport error";
packages/ai/src/utils/idle-iterator.ts
+2 -2
View File
@@ -1,8 +1,8 @@
import { $env } from "@oh-my-pi/pi-utils";
import * as AIError from "../error";
const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 120_000;
const DEFAULT_STREAM_FIRST_EVENT_TIMEOUT_MS = 100_000;
const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000;
const DEFAULT_STREAM_FIRST_EVENT_TIMEOUT_MS = 300_000;
/** Re-mint persistent race promises every N iterations (see hoisted-racer comment). */
const RACER_REMINT_INTERVAL = 1024;
packages/ai/test/proxy.test.ts
-1
View File
@@ -61,7 +61,6 @@ async function waitForSocketClose(socket: net.Socket): Promise<void> {
socket.once("close", () => closed.resolve());
await closed.promise;
}
const isProxyEnvKey = (k: string): boolean =>
k.startsWith("PI_PROXY") ||
k === "HTTP_PROXY" ||
packages/ai/test/stream-timeout-defaults.test.ts
+2 -2
View File
@@ -101,8 +101,8 @@ describe("getStreamFirstEventTimeoutMs(idleTimeoutMs, fallbackMs)", () => {
expect(getStreamFirstEventTimeoutMs(undefined, 300_000)).toBeUndefined();
});
it("falls back to the 100s global default when no fallback or env is provided", () => {
expect(getStreamFirstEventTimeoutMs()).toBe(100_000);
it("falls back to the 300s global default when no fallback or env is provided", () => {
expect(getStreamFirstEventTimeoutMs()).toBe(300_000);
});
});
packages/catalog/CHANGELOG.md
+2
View File
@@ -2,6 +2,8 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Added
- Regenerated the Cursor agent protobufs (`discovery/cursor-gen/agent_pb.ts`) against the modern `agent.proto`, adding the message and enum families current Cursor CLI builds emit: Pi tool exec frames, hook queries and responses, subagents, allowlist prechecks, MCP state, smart-mode classification, canvas diagnostics, conversation search, agent-store conflicts and git diff. Purely additive — no existing exported symbol changed shape.
packages/catalog/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"type": "module",
"name": "@oh-my-pi/pi-catalog",
"version": "17.1.8",
"version": "17.2.0",
"description": "Model catalog for omp: bundled model database, provider discovery descriptors, model identity, classification, and equivalence",
"homepage": "https://omp.sh",
"author": "Can Boluk",
packages/coding-agent/CHANGELOG.md
+10
View File
@@ -2,12 +2,19 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Breaking Changes
- Removed the `DEL`, `DEL.BLK`, `COPY`, and `COPY.BLK` hashline edit operations. Use `CUT` / `CUT.BLK` for deletion; removed content remains available to `PASTE`.
### Added
- Added server-name autocomplete for `/mcp` commands (`enable`, `disable`, `test`, `remove`, `reconnect`, `reauth`, `unauth`) using configured and runtime-discovered MCP servers.
- Added `--from-claude` and `--from-codex` session imports, also available from `/resume @claude` and `/resume @codex`.
- Added an opt-in OMP-native software-security workflow (`security.enabled`, default off) with immutable scan plans, exact-account Codex subscription affinity, native task-worker review, canonical findings/coverage/SARIF publication, project-scoped history, explicit dispositions, producer-differential comparison, and the read-only `security://` resource namespace. Generic SARIF and official Codex Security bundles normalize into the same OMP-owned store.
- Added explicit Codex Security cloud operations to the opt-in security workflow: list and start account-pinned cloud scans, inspect their progress, and import current findings into OMP's canonical store and `security://` namespace without changing the native scan engine or spoofing official runtime attribution.
- Added `CUT` and `PASTE` ops to the hashline edit tool for moving code without retyping it: `CUT N.=M` (and `.BLK` block forms) capture lines into a clipboard register, and `PASTE` operations insert them. The register flows across sections within a patch (cross-file moves) and persists across edit calls per session.
- Added `--from-claude` and `--from-codex` session imports (including compaction state for Codex), also available from `/resume @claude` and `/resume @codex`.
- Added interactive Exa API-key onboarding through `/login exa`, opening the official key dashboard and saving pasted keys for authenticated web search while preserving `EXA_API_KEY` and explicit-selection public MCP fallback behavior ([#1798](https://github.com/can1357/oh-my-pi/issues/1798)).
- Added `ExtensionContext.getAsyncJobSnapshot()` so extensions can read the owning session's async-job state without relying on process-global job-manager identity
@@ -26,6 +33,9 @@
### Fixed
- Fixed Anthropic prompt-cache cold misses on session resume with multiple OAuth accounts: the account that served a session is now recorded in the session file (as a `credential_pin` sha-256 of the account + org/project scope, so exports carry no plaintext identity) and re-pinned on resume with the session's effective last-use time, so a fresh process no longer re-ranks accounts by usage headroom — which systematically routed away from the just-used account and cold-missed the entire account-scoped cache prefix. Sticky routing was previously stored only in the auth store's KV cache, which is in-memory when a remote auth broker is configured.
- Fixed Anthropic prompt-cache cold misses on session resume with multiple OAuth accounts: the account that served a session is now recorded in the session file (as a PII-free `credential_pin` hash) and re-pinned on resume, so a fresh process no longer re-ranks accounts by usage headroom — which systematically routed away from the just-used account and cold-missed the entire account-scoped cache prefix. Sticky routing was previously stored only in the auth store's KV cache, which is in-memory when a remote auth broker is configured.
- Fixed concurrent `createAgentSession` calls with the default agent id failing initialization with `Agent "Main" was replaced during session initialization` — each in-process embedder (e.g. the edit benchmark runner) can now pass a private registry via the newly exported `AgentRegistry`, keeping every top-level session's "Main" out of the process-global roster race.
- Fixed task tool blocks duplicating their per-agent progress rows into terminal scrollback on every update: live task frames now pin the transcript live region so mid-run rows are never recorded as frozen snapshots, and a detached background task freezes its progress the moment any of its rows commit to scrollback instead of mutating committed history.
- Fixed Codex reset fireworks comparing different quota tiers or plans, preventing false celebrations when usage reports switch between Spark and base weekly limits.
- Fixed Cursor ranged-read results losing the full file byte size after applying the requested window.
packages/coding-agent/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"type": "module",
"name": "@oh-my-pi/pi-coding-agent",
"version": "17.1.8",
"version": "17.2.0",
"description": "Coding agent CLI with read, bash, edit, write tools and session management",
"homepage": "https://omp.sh",
"author": "Can Boluk",
packages/coding-agent/src/edit/edit-clipboard.ts
+23
View File
@@ -0,0 +1,23 @@
/**
* Session-bound edit clipboard register.
*
* `CUT`/`PASTE` hashline ops share one {@link Clipboard} per session so
* content captured by one edit call can be pasted by a later one (and across
* files within one call). The executor works on a fork per batch and commits
* it back only after every write lands — see `executeHashlineSingle`.
*/
import type { Clipboard } from "@oh-my-pi/hashline";
interface EditClipboardOwner {
editClipboard?: Clipboard;
}
/**
* Look up (or lazily create) the clipboard register attached to a session.
* Storage lives on `session.editClipboard` so it ages out exactly with the
* session itself.
*/
export function getEditClipboard(session: EditClipboardOwner): Clipboard {
session.editClipboard ??= {};
return session.editClipboard;
}
packages/coding-agent/src/edit/hashline/diff.ts
+49 -10
View File
@@ -13,6 +13,7 @@ import * as path from "node:path";
import {
type ApplyResult,
applyEdits,
type Clipboard,
type Cursor,
computeFileHash,
type Edit,
@@ -26,8 +27,10 @@ import {
parsePatchStreaming,
Recovery,
resolveBlockEdits,
resolveClipboardEdits,
type SnapshotStore,
stripBom,
validateClipboardSequence,
} from "@oh-my-pi/hashline";
import { resolveToCwd } from "../../tools/path-utils";
import { generateDiffString } from "../diff";
@@ -48,6 +51,14 @@ export interface HashlineDiffOptions {
* authoring input; the final apply path still validates through Patcher.
*/
skipHashValidation?: boolean;
/**
* Clipboard register shared across the sections of one patch preview.
* `CUT` in an earlier section feeds `PASTE` in a later one, so the preview
* matches apply. Multi-section previews MUST thread one register through
* sections in patch order; omitted, each section gets a
* private register (same-file cut/paste still previews correctly).
*/
clipboard?: Clipboard;
}
async function readSectionText(absolutePath: string, sectionPath: string): Promise<string> {
@@ -144,6 +155,7 @@ function hasAnchorScopedEdit(edits: readonly Edit[]): boolean {
return edits.some(edit => {
if (edit.kind === "delete") return true;
if (edit.kind === "block") return true;
if (edit.kind === "cut") return true;
return edit.cursor.kind === "before_anchor" || edit.cursor.kind === "after_anchor";
});
}
@@ -204,14 +216,24 @@ function applyPreviewEdits(args: {
const liveMatches = expected !== undefined && computeFileHash(normalized) === expected;
const edits = parsePreviewEdits(section, options.streaming);
const resolved = resolvePreviewEdits({ section, absolutePath, normalized, snapshots, expected, liveMatches, edits });
if (options.skipHashValidation || expected === undefined || liveMatches) return applyEdits(normalized, resolved);
if (!hasAnchorScopedEdit(resolved)) return applyEdits(normalized, resolved);
const clipboard = options.clipboard ?? {};
// Mirror the Patcher: surface clipboard sequencing mistakes with their
// targeted message before the recovery path below swallows them. Streaming
// previews stay lenient — a mid-typed op transiently violating sequencing
// must not flash an error frame.
if (!options.streaming) validateClipboardSequence(resolved, clipboard);
const applyOptions = { clipboard, ...(options.streaming ? { onEmptyPaste: "drop" as const } : {}) };
if (options.skipHashValidation || expected === undefined || liveMatches) {
return applyEdits(normalized, resolved, applyOptions);
}
if (!hasAnchorScopedEdit(resolved)) return applyEdits(normalized, resolved, applyOptions);
const recovered = new Recovery(snapshots).tryRecover({
path: absolutePath,
currentText: normalized,
fileHash: expected,
edits: resolved,
clipboard,
});
if (recovered) return recovered;
throw createMismatchError(section, absolutePath, normalized, snapshots, expected);
@@ -250,18 +272,36 @@ function insertCursorLine(cursor: Cursor, fileLineCount: number): number {
function buildStreamingSectionDiff(
section: PatchSection,
normalized: string,
clipboard: Clipboard,
): { diff: string; firstChangedLine: number | undefined } | { error: string } {
const { edits, fileOp } = parsePatchStreaming(section.diff);
const resolved = resolveBlockEdits(edits, normalized, section.path, nativeBlockResolver, { onUnresolved: "drop" });
const blockResolved = resolveBlockEdits(edits, normalized, section.path, nativeBlockResolver, {
onUnresolved: "drop",
});
const fileLines = normalized.split("\n");
// Expand clipboard ops so a `PASTE` previews the moved rows. Resolve into
// a scratch register and commit it back only on success: a mid-typed
// capture (out-of-range while the range digits are still streaming,
// transient sequencing violations) throws AFTER earlier captures may have
// landed, and leaking those into the shared register would let a later
// section preview a paste that never happened. On failure, fall back to
// filtering the clipboard edits out of this section's frame.
const scratch: Clipboard = { ...clipboard };
let resolved: readonly Edit[];
try {
resolved = resolveClipboardEdits(blockResolved, fileLines, scratch, { onEmptyPaste: "drop" });
if (scratch.lines === undefined) delete clipboard.lines;
else clipboard.lines = scratch.lines;
} catch {
resolved = blockResolved.filter(edit => edit.kind !== "cut" && edit.kind !== "paste");
}
if (resolved.length === 0) {
// A whole-file op (REM / MV) carries no line edits: the change is the
// delete/move itself, conveyed by the result header, so emit an empty
// diff rather than a misleading "No changes" error.
// A whole-file op (REM / MV) carries no line edits; the result header
// conveys the change, so emit an empty diff.
if (fileOp) return { diff: "", firstChangedLine: undefined };
return { error: `No changes would be made to ${section.path}.` };
}
const fileLines = normalized.split("\n");
const rows: string[] = [];
let firstChangedLine: number | undefined;
@@ -323,11 +363,10 @@ export async function computeHashlineSectionDiff(
// diff: feed the in-flight ops through the natural-order builder so the
// streamed cursor stays pinned to the bottom. The args-complete pass
// (`streaming` unset) falls through to the real Myers diff below.
if (options.streaming) return buildStreamingSectionDiff(section, normalized);
if (options.streaming) return buildStreamingSectionDiff(section, normalized, options.clipboard ?? {});
const result = applyPreviewEdits({ section, absolutePath, normalized, snapshots, options });
if (normalized === result.text) {
// REM/MV-only sections change no text; the header conveys the
// delete/move, so don't surface a "No changes" error.
// REM/MV-only sections change no text; the header conveys the op.
if (section.fileOp) return { diff: "", firstChangedLine: undefined };
return { error: `No changes would be made to ${section.path}.` };
}
packages/coding-agent/src/edit/hashline/execute.ts
+38 -6
View File
@@ -13,6 +13,9 @@
import {
type BlockResolution,
buildCompactDiffPreview,
type Clipboard,
commitClipboard,
forkClipboard,
MismatchError as HashlineMismatchError,
Patch,
Patcher,
@@ -25,6 +28,7 @@ import type { ToolSession } from "../../tools";
import { outputMeta } from "../../tools/output-meta";
import { ToolError } from "../../tools/tool-errors";
import { generateDiffString } from "../diff";
import { getEditClipboard } from "../edit-clipboard";
import { getFileSnapshotStore } from "../file-snapshot-store";
import type { EditToolDetails, EditToolPerFileResult, LspBatchRequest } from "../renderer";
import { pruneOversizedEditSnapshots } from "../snapshot-details";
@@ -98,12 +102,24 @@ interface RenderedSection {
perFileResult: EditToolPerFileResult;
}
const BLOCK_OP_LABELS: Record<BlockResolution["op"], string> = {
replace: "SWAP.BLK",
insert_after: "INS.BLK.POST",
cut: "CUT.BLK",
paste_after: "PASTE.BLK.POST",
};
function formatBlockResolution(resolution: BlockResolution): string {
const op = resolution.op === "delete" ? "DEL.BLK" : resolution.op === "insert_after" ? "INS.BLK.POST" : "SWAP.BLK";
const op = BLOCK_OP_LABELS[resolution.op];
const lines = resolution.end - resolution.start + 1;
const span =
resolution.start === resolution.end ? `line ${resolution.start}` : `lines ${resolution.start}-${resolution.end}`;
const suffix = resolution.op === "insert_after" ? `; body lands after line ${resolution.end}` : "";
const suffix =
resolution.op === "insert_after"
? `; body lands after line ${resolution.end}`
: resolution.op === "paste_after"
? `; clipboard lands after line ${resolution.end}`
: "";
return `${op} ${resolution.anchorLine} → resolved ${span} (${lines} line${lines === 1 ? "" : "s"})${suffix}`;
}
@@ -213,12 +229,19 @@ export async function executeHashlineSingle(
const enforceSeenLines = options.session.settings.get("edit.enforceSeenLines");
const patcher = new Patcher({ fs, snapshots, blockResolver: nativeBlockResolver, enforceSeenLines });
// The clipboard register is session-persistent: `CUT` in one edit call can
// `PASTE` in a later one. Each batch works on a fork and publishes it back
// only after writes land.
const sessionClipboard = getEditClipboard(options.session);
const clipboard = forkClipboard(sessionClipboard);
// Single-section fast path: prepare, commit, render.
const inputHash = hashPatchInput(options.input);
if (patch.sections.length === 1) {
fs.setBatchRequest(narrowBatchRequest(options.batchRequest, true));
const prepared = await patcher.prepare(patch.sections[0]);
const prepared = await patcher.prepare(patch.sections[0], clipboard);
const sectionResult = await patcher.commit(prepared);
commitClipboard(clipboard, sessionClipboard);
if (sectionResult.op === "noop") {
const { count, escalate } = recordNoopEdit(options.session, sectionResult.canonicalPath, inputHash);
if (escalate) {
@@ -231,9 +254,18 @@ export async function executeHashlineSingle(
}
// Multi-section: prepare every section up front so we fail fast before
// any write hits the filesystem.
// any write hits the filesystem. One clipboard register spans the batch,
// so `CUT` in one section feeds `PASTE` in a later one.
const prepared: PreparedSection[] = [];
for (const section of patch.sections) prepared.push(await patcher.prepare(section));
// Register state after each section's prepare. Commits are non-atomic: a
// mid-batch write failure leaves earlier sections on disk, so the session
// register must reflect exactly the landed prefix — content a landed CUT
// deleted would otherwise be lost.
const sectionStates: Clipboard[] = [];
for (const section of patch.sections) {
prepared.push(await patcher.prepare(section, clipboard));
sectionStates.push(forkClipboard(clipboard));
}
assertUniqueCanonicalPaths(prepared);
for (const entry of prepared) {
if (entry.isNoop) {
@@ -251,6 +283,7 @@ export async function executeHashlineSingle(
const isLast = i === prepared.length - 1;
fs.setBatchRequest(narrowBatchRequest(options.batchRequest, isLast));
const sectionResult = await patcher.commit(prepared[i]);
commitClipboard(sectionStates[i], sessionClipboard);
if (sectionResult.op === "noop") {
const { count, escalate } = recordNoopEdit(options.session, sectionResult.canonicalPath, inputHash);
throw escalate
@@ -260,7 +293,6 @@ export async function executeHashlineSingle(
resetNoopEdit(options.session, sectionResult.canonicalPath);
rendered.push(renderSection(sectionResult, fs.consumeDiagnostics(sectionResult.path), prepared[i].section.path));
}
return {
content: [
{
packages/coding-agent/src/edit/renderer.ts
+4 -4
View File
@@ -140,7 +140,7 @@ interface HashlineInputEntry {
path: string;
op?: Operation;
rename?: string;
/** A SWAP/DEL/INS line-editing op precedes the file op — keeps a move framed. */
/** A SWAP/CUT/INS line-editing op precedes the file op — keeps a move framed. */
hasLineEdits?: boolean;
}
@@ -568,9 +568,9 @@ function parseHashlineInputPreviewHeader(line: string): string | null {
return previewPath.length > 0 ? previewPath : null;
}
// Line-editing op headers (SWAP/DEL/INS family), distinct from the file-level
// REM/MV ops. Body rows are always `+TEXT`, so this only matches real headers.
const HL_LINE_OP_HEADER = /^(?:SWAP|DEL|INS)\b/;
// Line-editing op headers (SWAP/CUT/INS family), distinct from file-level
// REM/MV ops. Body rows are `+TEXT`, so this only matches real headers.
const HL_LINE_OP_HEADER = /^(?:SWAP|CUT|INS)\b/;
/**
* Walk a (possibly mid-stream) hashline payload into per-section descriptors:
packages/coding-agent/src/edit/streaming.ts
+16 -1
View File
@@ -16,8 +16,10 @@
import {
ABORT_MARKER,
BEGIN_PATCH_MARKER,
type Clipboard,
containsRecognizableHashlineOperations,
END_PATCH_MARKER,
forkClipboard,
type PatchSection as HashlineInputSection,
Patch as HashlinePatch,
type SnapshotStore,
@@ -50,6 +52,13 @@ export interface StreamingDiffContext {
* not flicker in the preview.
*/
isStreaming?: boolean;
/**
* Session-persistent clipboard register (`CUT`/`PASTE`). Previews
* fork it per frame — never mutating it — so a `PASTE` of content cut in
* an earlier edit call (or an earlier section of this patch) renders the
* real rows.
*/
clipboard?: Clipboard;
}
/**
@@ -542,7 +551,9 @@ const hashlineStrategy: EditStreamingStrategy<HashlineArgs> = {
// to parse; suppress until the next chunk arrives. Once args are
// complete, surface the error so the model sees what went wrong.
if (ctx.isStreaming) return null;
const result = await computeHashlineDiff({ input }, ctx.cwd, ctx.snapshots);
const result = await computeHashlineDiff({ input }, ctx.cwd, ctx.snapshots, {
clipboard: forkClipboard(ctx.clipboard),
});
ctx.signal.throwIfAborted();
return [toPerFilePreview("", result)];
}
@@ -558,12 +569,16 @@ const hashlineStrategy: EditStreamingStrategy<HashlineArgs> = {
const trailingProcessedIndex = sectionsToProcess.length - 1;
const previews: PerFileDiffPreview[] = [];
// Fork the session register per preview frame: sections feed each other
// in patch order, but a preview must never mutate the live register.
const clipboard = forkClipboard(ctx.clipboard);
for (let i = 0; i < sectionsToProcess.length; i++) {
ctx.signal.throwIfAborted();
const section = sectionsToProcess[i];
const result = await computeHashlineSectionDiff(section, ctx.cwd, ctx.snapshots, {
streaming: ctx.isStreaming,
skipHashValidation: ctx.isStreaming === true,
clipboard,
});
ctx.signal.throwIfAborted();
// Ignore parse/apply errors from the trailing (actively-typed)
packages/coding-agent/src/export/html/template.js
+1 -1
View File
@@ -395,7 +395,7 @@
}
// Apply filter mode
const isSettingsEntry = ['label', 'custom', 'model_change', 'thinking_level_change', 'mode_change', 'ttsr_injection', 'session_init'].includes(entry.type);
const isSettingsEntry = ['label', 'custom', 'model_change', 'thinking_level_change', 'mode_change', 'ttsr_injection', 'session_init', 'credential_pin'].includes(entry.type);
let passesFilter = true;
switch (filterMode) {
packages/coding-agent/src/modes/components/tool-execution.ts
+6 -1
View File
@@ -1,4 +1,4 @@
import type { SnapshotStore } from "@oh-my-pi/hashline";
import type { Clipboard, SnapshotStore } from "@oh-my-pi/hashline";
import type { AgentTool } from "@oh-my-pi/pi-agent-core";
import {
Box,
@@ -222,6 +222,8 @@ export interface ToolExecutionUi {
export interface ToolExecutionOptions {
snapshots?: SnapshotStore;
/** Session-persistent edit clipboard register, forked per preview frame. */
clipboard?: Clipboard;
showImages?: boolean; // default: true (only used if terminal supports images)
editFuzzyThreshold?: number;
editAllowFuzzy?: boolean;
@@ -285,6 +287,7 @@ export class ToolExecutionComponent extends Container implements NativeScrollbac
#editFuzzyThreshold: number | undefined;
#editAllowFuzzy: boolean | undefined;
#snapshots?: SnapshotStore;
#clipboard?: Clipboard;
#isPartial = true;
#resultVersion = 0;
#lastDisplayKey: string | undefined;
@@ -393,6 +396,7 @@ export class ToolExecutionComponent extends Container implements NativeScrollbac
this.#editFuzzyThreshold = options.editFuzzyThreshold;
this.#editAllowFuzzy = options.editAllowFuzzy;
this.#snapshots = options.snapshots;
this.#clipboard = options.clipboard;
this.#liveRegion = options.liveRegion;
this.#tool = tool;
this.#ui = ui;
@@ -550,6 +554,7 @@ export class ToolExecutionComponent extends Container implements NativeScrollbac
cwd: this.#cwd,
signal: controller.signal,
snapshots: this.#snapshots!,
clipboard: this.#clipboard,
fuzzyThreshold: this.#editFuzzyThreshold,
allowFuzzy: this.#editAllowFuzzy,
isStreaming,
packages/coding-agent/src/modes/controllers/event-controller.ts
+3
View File
@@ -6,6 +6,7 @@ import { logger, prompt, sanitizeText } from "@oh-my-pi/pi-utils";
import { INTENT_FIELD } from "@oh-my-pi/pi-wire";
import { extractTextContent } from "../../commit/utils";
import { settings } from "../../config/settings";
import { getEditClipboard } from "../../edit/edit-clipboard";
import { getFileSnapshotStore } from "../../edit/file-snapshot-store";
import { AssistantMessageComponent } from "../../modes/components/assistant-message";
import { detectCacheInvalidation } from "../../modes/components/cache-invalidation-marker";
@@ -810,6 +811,7 @@ export class EventController {
renderArgs,
{
snapshots: getFileSnapshotStore(this.ctx.viewSession),
clipboard: getEditClipboard(this.ctx.viewSession),
showImages: settings.get("terminal.showImages"),
editFuzzyThreshold: settings.get("edit.fuzzyThreshold"),
editAllowFuzzy: settings.get("edit.fuzzyMatch"),
@@ -1033,6 +1035,7 @@ export class EventController {
event.args,
{
snapshots: getFileSnapshotStore(this.ctx.viewSession),
clipboard: getEditClipboard(this.ctx.viewSession),
showImages: settings.get("terminal.showImages"),
editFuzzyThreshold: settings.get("edit.fuzzyThreshold"),
editAllowFuzzy: settings.get("edit.fuzzyMatch"),
packages/coding-agent/src/modes/utils/ui-helpers.ts
+2
View File
@@ -5,6 +5,7 @@ import { type Component, Spacer, Text, TruncatedText } from "@oh-my-pi/pi-tui";
import type { AdvisorMessageDetails } from "../../advisor";
import { COLLAB_PROMPT_MESSAGE_TYPE, type CollabPromptDetails } from "../../collab/protocol";
import { settings } from "../../config/settings";
import { getEditClipboard } from "../../edit/edit-clipboard";
import { getFileSnapshotStore } from "../../edit/file-snapshot-store";
import { createAdvisorMessageCard } from "../../modes/components/advisor-message";
import { AssistantMessageComponent } from "../../modes/components/assistant-message";
@@ -497,6 +498,7 @@ export class UiHelpers {
renderArgs,
{
snapshots: getFileSnapshotStore(this.ctx.viewSession),
clipboard: getEditClipboard(this.ctx.viewSession),
showImages: settings.get("terminal.showImages"),
editFuzzyThreshold: settings.get("edit.fuzzyThreshold"),
editAllowFuzzy: settings.get("edit.fuzzyMatch"),
packages/coding-agent/src/prompts/system/plan-mode-active.md
+2 -2
View File
@@ -30,10 +30,10 @@ Use `{{editToolName}}` for incremental edits and `{{writeToolName}}` only to cre
{{#if isHashlineEditMode}}
Structure the plan as `##`/`###` markdown sections so you can revise it section-by-section: with `{{editToolName}}`, a heading anchors its WHOLE section (through every nested deeper heading, up to the next same-or-higher heading). Rely on the block ops to grow the plan without rewriting the file:
- `SWAP.BLK N:` on a heading line — rewrite that entire section in place.
- `DEL.BLK N` on a heading line — drop the whole section.
- `CUT.BLK N` on a heading line — drop the whole section.
- `INS.BLK.POST N:` on a heading line — add a new section AFTER that one (end the inserted body with a blank line so the next heading stays separated).
Write each section together with its body — block ops need a multi-line section; a bare heading with no body falls back to plain `INS.POST`/`DEL`/`SWAP`.
Write each section together with its body — block ops need a multi-line section; a bare heading with no body falls back to plain `INS.POST`/`CUT`/`SWAP`.
{{/if}}
## Ground every claim
packages/coding-agent/src/sdk.ts
+4
View File
@@ -625,6 +625,10 @@ export type * from "./extensibility/extensions";
export type { Skill } from "./extensibility/skills";
export type { FileSlashCommand } from "./extensibility/slash-commands";
export type { MCPManager, MCPServerConfig, MCPServerConnection, MCPToolsLoadResult } from "./mcp";
// Agent registry: pass a private instance per `createAgentSession` when
// embedding several concurrent top-level sessions in one process (the default
// global registry admits only one "Main" per process generation).
export { type AgentRef, AgentRegistry, MAIN_AGENT_ID } from "./registry/agent-registry";
export type { Tool } from "./tools";
export { buildDirectoryTree, buildWorkspaceTree, type DirectoryTree, type WorkspaceTree } from "./workspace-tree";
packages/coding-agent/src/session/agent-session.ts
+21 -1
View File
@@ -19,7 +19,7 @@ import * as path from "node:path";
import { scheduler } from "node:timers/promises";
import { isPromise } from "node:util/types";
import type { InMemorySnapshotStore } from "@oh-my-pi/hashline";
import type { Clipboard, InMemorySnapshotStore } from "@oh-my-pi/hashline";
import {
type AfterToolCallContext,
type AfterToolCallResult,
@@ -249,6 +249,7 @@ import {
shouldEvaluateCodexAutoRedeem,
shouldPromptCodexAutoRedeem,
} from "./codex-auto-reset";
import { recordCredentialPin, seedCredentialPins } from "./credential-pin";
import { EvalRunner, type EvalRunnerHost } from "./eval-runner";
import {
collectPendingToolCalls,
@@ -409,6 +410,8 @@ export class AgentSession {
getXdevToolEntries: () => Array<{ name: string; summary: string }>;
readonly yieldQueue: YieldQueue;
fileSnapshotStore?: InMemorySnapshotStore;
/** Per-session `CUT`/`PASTE` clipboard register shared across edit calls. */
editClipboard?: Clipboard;
#powerAssertion: MacOSPowerAssertion | undefined;
@@ -2428,6 +2431,15 @@ export class AgentSession {
},
costUsd: assistantMsg.usage.cost.total,
});
// Persist which account served this turn so a resumed process can
// re-pin it and keep the provider's account-scoped prompt cache
// warm (broker-mode sticky routing is process-local).
recordCredentialPin(
this.#modelRegistry.authStorage,
this.sessionManager,
this.sessionId,
assistantMsg.provider,
);
}
if (event.message.role === "toolResult") {
const { toolName, toolCallId, isError, content } = event.message;
@@ -3400,6 +3412,14 @@ export class AgentSession {
this.agent.setMetadataResolver((provider: string) =>
buildSessionMetadata(sid, provider, this.#modelRegistry.authStorage),
);
// Restore the session's recorded provider accounts before the first
// request routes: sticky rows are process-local under a remote auth
// broker, and losing them re-ranks onto a different account, cold-missing
// the account-scoped prompt cache. Skipped for fresh provider sessions —
// those explicitly want new routing identity.
if (!this.#freshProviderSessionId) {
seedCredentialPins(this.#modelRegistry.authStorage, this.sessionManager, sid);
}
// Keep every live advisor's provider identity in lockstep with the primary's
// across every session-boundary transition — including branch paths that
// skip conversation restore — so advisors never emit the previous
packages/coding-agent/src/session/credential-pin.ts
+93
View File
@@ -0,0 +1,93 @@
/**
* Session-file persistence of the OAuth account that served a session.
*
* Provider prompt caches are account-scoped (Anthropic bills a full cache
* re-write after an account flip), and the auth store's session-sticky routing
* is process-local when a remote auth broker is configured — the broker
* store's KV cache is in-memory, so sticky rows die with the CLI process.
* Resuming a session in a fresh process then re-ranks accounts by usage
* headroom, which is biased *away* from the account that just served the
* session (it has the highest recent burn), cold-missing the entire prefix.
*
* These helpers close the loop through the session file itself: after each
* assistant turn the serving account is recorded as a `credential_pin` entry,
* and on session adoption the pin is matched against the stored accounts and
* seeded back into the auth store with the session's effective last-use
* timestamp, so the provider's warm-window semantics still decide whether to
* stick or re-rank.
*/
import type { AuthStorage } from "./auth-storage";
import type { SessionManager } from "./session-manager";
/** Account fields shared by `OAuthAccountIdentity` and `OAuthAccountSummary`. */
interface CredentialPinIdentity {
accountId?: string;
email?: string;
projectId?: string;
orgId?: string;
}
/**
* Stable identifier for a provider account within its billing scope. The
* digest covers the full scope tuple — the same account in two orgs (Anthropic
* multi-subscription) or projects (Gemini) is two distinct cache domains and
* must produce two distinct pins. The digest input is the persisted contract
* for `CredentialPinEntry.hash` — changing it orphans every recorded pin.
*
* Hashing avoids embedding raw emails/uuids in session files, but an unsalted
* digest of a guessable email is still linkable — treat exported sessions
* accordingly.
*
* Returns `undefined` when the identity carries no account key at all.
*/
export function credentialPinHash(provider: string, identity: CredentialPinIdentity): string | undefined {
if (!identity.accountId && !identity.email) return undefined;
const key = [
provider,
identity.accountId ?? "",
identity.email ?? "",
identity.orgId ?? "",
identity.projectId ?? "",
].join("\0");
return new Bun.CryptoHasher("sha256").update(key).digest("hex");
}
/**
* Record the account that served the latest assistant turn for `provider`.
* Appends a `credential_pin` entry only when the account differs from the
* branch's latest pin, so steady-state sessions add a single entry; the
* effective last-use time is derived from later assistant turns on read
* (see `SessionManager.getCredentialPins`).
*/
export function recordCredentialPin(
authStorage: AuthStorage,
sessionManager: SessionManager,
sessionId: string,
provider: string,
): void {
const identity = authStorage.getOAuthAccountIdentity(provider, sessionId);
if (!identity) return;
const hash = credentialPinHash(provider, identity);
if (!hash || sessionManager.getCredentialPins().get(provider)?.hash === hash) return;
sessionManager.appendCredentialPin(provider, hash);
}
/**
* Re-pin the accounts recorded in the session file onto the auth store's
* session stickiness. No-op per provider when the account is gone (logged out)
* or when a live sticky already exists (same-process branch/session switches
* must not clobber fresher routing). Seeds with the session's effective
* last-use time so stale resumes still fall through to usage ranking.
*/
export function seedCredentialPins(authStorage: AuthStorage, sessionManager: SessionManager, sessionId: string): void {
for (const [provider, pin] of sessionManager.getCredentialPins()) {
const accounts = authStorage.listOAuthAccounts(provider, sessionId);
if (accounts.length === 0 || accounts.some(account => account.active)) continue;
const match = accounts.find(account => credentialPinHash(provider, account) === pin.hash);
if (!match) continue;
authStorage.pinSessionOAuthAccount(provider, sessionId, match.credentialId, {
lastUsedAtMs: pin.lastUsedAt,
});
}
}
packages/coding-agent/src/session/session-entries.ts
+21 -1
View File
@@ -154,6 +154,7 @@ export interface TitleChangeEntry extends SessionEntryBase {
declare module "@oh-my-pi/pi-agent-core/compaction/entries" {
interface CustomCompactionSessionEntries {
titleChange: TitleChangeEntry;
credentialPin: CredentialPinEntry;
}
}
@@ -164,6 +165,24 @@ export interface TtsrInjectionEntry extends SessionEntryBase {
injectedRules: string[];
}
/**
* Records which OAuth account served this session's requests for a provider.
*
* Provider prompt caches (Anthropic in particular) are account-scoped, and the
* auth store's session-sticky routing is process-local under a remote auth
* broker, so resume must re-pin the same account to reuse the warm cache
* prefix. Stores a sha-256 of the account + billing-scope tuple instead of
* the raw email/uuid/org; note an unsalted digest of a guessable email is
* still linkable, so exported sessions are pseudonymous, not anonymous.
*/
export interface CredentialPinEntry extends SessionEntryBase {
type: "credential_pin";
/** Provider id the pin applies to (e.g. "anthropic"). */
provider: string;
/** `credentialPinHash()` of the serving account's identity + scope tuple. */
hash: string;
}
/** Session init entry - captures initial context for subagent sessions (debugging/replay). */
export interface SessionInitEntry extends SessionEntryBase {
type: "session_init";
@@ -230,7 +249,8 @@ export type SessionEntry =
| TitleChangeEntry
| TtsrInjectionEntry
| SessionInitEntry
| ModeChangeEntry;
| ModeChangeEntry
| CredentialPinEntry;
/** Raw logical file entry after loaders strip any fixed-width title slot. */
export type FileEntry = SessionHeader | SessionEntry;
packages/coding-agent/src/session/session-manager.ts
+37
View File
@@ -35,6 +35,7 @@ import { type BuildSessionContextOptions, buildSessionContext, type SessionConte
import {
type BranchSummaryEntry,
type CompactionEntry,
type CredentialPinEntry,
CURRENT_SESSION_VERSION,
type CustomEntry,
type CustomMessageEntry,
@@ -179,6 +180,7 @@ function isDraftOnlyMetadataEntry(entry: SessionEntry): boolean {
case "thinking_level_change":
case "service_tier_change":
case "mode_change":
case "credential_pin":
return true;
default:
return false;
@@ -2062,6 +2064,41 @@ export class SessionManager {
return [...names];
}
/** Append a credential pin recording which OAuth account served `provider`. */
appendCredentialPin(provider: string, hash: string): string {
const entry: CredentialPinEntry = {
type: "credential_pin",
...this.#freshEntryFields(),
provider,
hash,
};
this.#recordEntry(entry);
return entry.id;
}
/**
* Latest credential pin per provider on the current branch (root → leaf),
* with the effective last-use time of the pinned account.
*
* Pins are appended only when the serving account *changes*, so a long
* session on one account carries a single old pin entry. Any assistant turn
* for the same provider after that pin was necessarily served by the pinned
* account, so its timestamp advances `lastUsedAt` — a resume seconds after
* the last turn seeds a warm sticky instead of a stale one.
*/
getCredentialPins(): Map<string, { hash: string; lastUsedAt: number }> {
const pins = new Map<string, { hash: string; lastUsedAt: number }>();
for (const entry of this.getBranch()) {
if (entry.type === "credential_pin") {
pins.set(entry.provider, { hash: entry.hash, lastUsedAt: new Date(entry.timestamp).getTime() });
} else if (entry.type === "message" && entry.message.role === "assistant") {
const pin = pins.get(entry.message.provider);
if (pin) pin.lastUsedAt = Math.max(pin.lastUsedAt, entry.message.timestamp);
}
}
return pins;
}
getLeafId(): string | null {
return this.#index.leafId();
}
packages/coding-agent/src/tools/index.ts
+5 -1
View File
@@ -1,4 +1,4 @@
import type { InMemorySnapshotStore } from "@oh-my-pi/hashline";
import type { Clipboard, InMemorySnapshotStore } from "@oh-my-pi/hashline";
import type { AgentOptions, AgentTelemetryConfig, AgentTool } from "@oh-my-pi/pi-agent-core";
import type { FetchImpl, ImageContent, Model, ServiceTierByFamily, ToolChoice } from "@oh-my-pi/pi-ai";
import { logger } from "@oh-my-pi/pi-utils";
@@ -357,6 +357,10 @@ export interface ToolSession {
* file changed out-of-band. Lazily initialized by `getFileSnapshotStore`. */
fileSnapshotStore?: InMemorySnapshotStore;
/** Per-session `CUT`/`PASTE` clipboard register shared across edit
* calls. Lazily initialized by `getEditClipboard`. */
editClipboard?: Clipboard;
/** Per-session log of unresolved git merge conflict regions surfaced by
* `read`. Each entry gets a stable id N referenced by `write conflict://N`
* to splice the recorded region with replacement content. Lazily initialized
packages/coding-agent/src/utils/git.ts
+6
View File
@@ -1727,6 +1727,12 @@ export const revList = {
async range(cwd: string, base: string, head: string, signal?: AbortSignal): Promise<string[]> {
return splitLines(await runText(cwd, ["rev-list", "--reverse", `${base}..${head}`], { readOnly: true, signal }));
},
/** Commits reachable from `ref` that touched `file`, newest first, capped at `limit`. */
async touching(cwd: string, ref: string, file: string, limit: number, signal?: AbortSignal): Promise<string[]> {
return splitLines(
await runText(cwd, ["rev-list", `--max-count=${limit}`, ref, "--", file], { readOnly: true, signal }),
);
},
};
// ════════════════════════════════════════════════════════════════════════════
packages/coding-agent/test/core/block-replace.test.ts
+8 -8
View File
@@ -90,11 +90,11 @@ describe("SWAP.BLK — native tree-sitter resolution end-to-end", () => {
});
});
it("deletes the resolved `if` block (line 2) end-to-end via `DEL.BLK`", async () => {
it("deletes the resolved `if` block (line 2) end-to-end via `CUT.BLK`", async () => {
await withTempDir(async tempDir => {
const session = makeSession(tempDir);
const { filePath, header } = await seedFile(tempDir, session, "x.ts", TS_SOURCE);
const input = `${header}\nDEL.BLK 2`;
const input = `${header}\nCUT.BLK 2`;
await executeHashlineSingle(executeOptions(tempDir, input, session));
@@ -166,17 +166,17 @@ describe("SWAP.BLK — native tree-sitter resolution end-to-end", () => {
});
});
it("echoes the resolved span in the result text for DEL.BLK", async () => {
it("echoes the resolved span in the result text for CUT.BLK", async () => {
await withTempDir(async tempDir => {
const session = makeSession(tempDir);
const { header } = await seedFile(tempDir, session, "x.ts", TS_SOURCE);
const input = `${header}\nDEL.BLK 2`;
const input = `${header}\nCUT.BLK 2`;
const result = await executeHashlineSingle(executeOptions(tempDir, input, session));
const text = result.content.map(part => (part.type === "text" ? part.text : "")).join("\n");
// `if (y) {` opens on line 2; resolves lines 2-3.
expect(text).toContain("DEL.BLK 2 → resolved lines 2-3 (2 lines)");
expect(text).toContain("CUT.BLK 2 → resolved lines 2-3 (2 lines)");
});
});
@@ -256,13 +256,13 @@ const MD_PLAN = [
].join("\n");
describe("block ops on markdown headings — whole-section resolution end-to-end", () => {
it("DEL.BLK at a `## H2` deletes the entire section, including nested subsections", async () => {
it("CUT.BLK at a `## H2` deletes the entire section, including nested subsections", async () => {
await withTempDir(async tempDir => {
const session = makeSession(tempDir);
// Line 4 is `## Context`; its section runs through `### Detail` and
// the trailing blank, up to `## Approach`.
const { filePath, header } = await seedFile(tempDir, session, "plan.md", MD_PLAN);
const input = `${header}\nDEL.BLK 4`;
const input = `${header}\nCUT.BLK 4`;
const result = await executeHashlineSingle(executeOptions(tempDir, input, session));
const text = result.content.map(part => (part.type === "text" ? part.text : "")).join("\n");
@@ -270,7 +270,7 @@ describe("block ops on markdown headings — whole-section resolution end-to-end
expect(await Bun.file(filePath).text()).toBe(
["# Plan", "intro", "", "## Approach", "step one", ""].join("\n"),
);
expect(text).toContain("DEL.BLK 4 → resolved lines 4-10 (7 lines)");
expect(text).toContain("CUT.BLK 4 → resolved lines 4-10 (7 lines)");
});
});
packages/coding-agent/test/credential-pin.test.ts
+185
View File
@@ -0,0 +1,185 @@
import { Database } from "bun:sqlite";
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import { TempDir } from "@oh-my-pi/pi-utils";
import { AuthStorage, SqliteAuthCredentialStore } from "../src/session/auth-storage";
import { credentialPinHash, recordCredentialPin, seedCredentialPins } from "../src/session/credential-pin";
import { SessionManager } from "../src/session/session-manager";
const ANTHROPIC_ENV = ["ANTHROPIC_API_KEY", "ANTHROPIC_OAUTH_TOKEN"] as const;
const savedEnv: Partial<Record<(typeof ANTHROPIC_ENV)[number], string | undefined>> = {};
function mintOAuthCredential(suffix: string, extra?: { orgId?: string }) {
return {
type: "oauth" as const,
access: `access-${suffix}`,
refresh: `refresh-${suffix}`,
expires: Date.now() + 60_000,
accountId: `account-${suffix}`,
email: `${suffix}@example.com`,
...extra,
};
}
function assistantMessage(provider: string, timestamp: number) {
return {
role: "assistant" as const,
content: [{ type: "text" as const, text: "hi" }],
api: "anthropic-messages",
provider,
model: "claude-test",
usage: {
input: 1,
output: 1,
cacheRead: 0,
cacheWrite: 0,
totalTokens: 2,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
},
stopReason: "stop" as const,
timestamp,
};
}
describe("credential pins", () => {
let tempDir: TempDir;
let storage: AuthStorage;
beforeEach(async () => {
for (const key of ANTHROPIC_ENV) {
savedEnv[key] = process.env[key];
delete process.env[key];
}
tempDir = TempDir.createSync("@pi-credential-pin-");
const store = new SqliteAuthCredentialStore(new Database(":memory:"));
store.saveOAuth("anthropic", mintOAuthCredential("a"));
store.saveOAuth("anthropic", mintOAuthCredential("b"));
storage = new AuthStorage(store);
await storage.reload();
});
afterEach(() => {
for (const key of ANTHROPIC_ENV) {
const value = savedEnv[key];
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
tempDir[Symbol.dispose]();
});
test("pin entries survive a session reload and the latest pin per provider wins", async () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
manager.appendMessage({ role: "user", content: "hello", timestamp: Date.now() });
manager.appendMessage(assistantMessage("anthropic", Date.now()));
manager.appendCredentialPin("anthropic", "hash-old");
manager.appendCredentialPin("openai-codex", "hash-codex");
manager.appendCredentialPin("anthropic", "hash-new");
await manager.flush();
const file = manager.getSessionFile();
if (!file) throw new Error("expected a persisted session file");
const reopened = await SessionManager.open(file);
const pins = reopened.getCredentialPins();
expect(pins.get("anthropic")?.hash).toBe("hash-new");
expect(pins.get("openai-codex")?.hash).toBe("hash-codex");
});
test("later assistant turns advance the pin's effective last-use; other providers and new pins do not", () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const pinId = manager.appendCredentialPin("anthropic", "hash-a");
const pinnedAt = new Date(manager.getEntry(pinId)!.timestamp).getTime();
// Long session on one account: no new pin entries, only assistant turns.
const lastTurnAt = pinnedAt + 3 * 60 * 60 * 1000;
manager.appendMessage(assistantMessage("anthropic", pinnedAt + 60_000));
manager.appendMessage(assistantMessage("anthropic", lastTurnAt));
expect(manager.getCredentialPins().get("anthropic")?.lastUsedAt).toBe(lastTurnAt);
// A different provider's turn never advances this provider's pin.
manager.appendMessage(assistantMessage("openai-codex", lastTurnAt + 60_000));
expect(manager.getCredentialPins().get("anthropic")?.lastUsedAt).toBe(lastTurnAt);
// An account change re-bases last-use at the new pin.
const newPinId = manager.appendCredentialPin("anthropic", "hash-b");
const newPinnedAt = new Date(manager.getEntry(newPinId)!.timestamp).getTime();
expect(manager.getCredentialPins().get("anthropic")?.lastUsedAt).toBe(newPinnedAt);
});
test("seeding re-pins the recorded account in a store with no session stickiness", () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const sessionId = manager.getSessionId();
const hash = credentialPinHash("anthropic", { accountId: "account-b", email: "b@example.com" });
if (!hash) throw new Error("expected a pin hash");
manager.appendCredentialPin("anthropic", hash);
// Fresh process: no sticky exists yet (the broker-mode resume scenario).
expect(storage.listOAuthAccounts("anthropic", sessionId).some(account => account.active)).toBe(false);
seedCredentialPins(storage, manager, sessionId);
const active = storage.listOAuthAccounts("anthropic", sessionId).find(account => account.active);
expect(active?.accountId).toBe("account-b");
});
test("pins are org-scoped: the same account in two orgs re-pins the matching org credential", async () => {
const store = new SqliteAuthCredentialStore(new Database(":memory:"));
store.saveOAuth("anthropic", mintOAuthCredential("x", { orgId: "org-1" }));
store.saveOAuth("anthropic", mintOAuthCredential("x", { orgId: "org-2" }));
const orgStorage = new AuthStorage(store);
await orgStorage.reload();
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const sessionId = manager.getSessionId();
const identity = { accountId: "account-x", email: "x@example.com" };
const orgTwoHash = credentialPinHash("anthropic", { ...identity, orgId: "org-2" });
if (!orgTwoHash) throw new Error("expected a pin hash");
expect(orgTwoHash).not.toBe(credentialPinHash("anthropic", { ...identity, orgId: "org-1" }));
manager.appendCredentialPin("anthropic", orgTwoHash);
seedCredentialPins(orgStorage, manager, sessionId);
const active = orgStorage.listOAuthAccounts("anthropic", sessionId).find(account => account.active);
expect(active?.orgId).toBe("org-2");
});
test("seeding never clobbers a live sticky from the same process", () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const sessionId = manager.getSessionId();
const accounts = storage.listOAuthAccounts("anthropic", sessionId);
const accountA = accounts.find(account => account.accountId === "account-a");
expect(storage.pinSessionOAuthAccount("anthropic", sessionId, accountA!.credentialId)).toBe(true);
const hash = credentialPinHash("anthropic", { accountId: "account-b", email: "b@example.com" });
manager.appendCredentialPin("anthropic", hash!);
seedCredentialPins(storage, manager, sessionId);
const active = storage.listOAuthAccounts("anthropic", sessionId).find(account => account.active);
expect(active?.accountId).toBe("account-a");
});
test("seeding is a no-op when the pinned account is no longer stored", () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const sessionId = manager.getSessionId();
const hash = credentialPinHash("anthropic", { accountId: "account-gone" });
manager.appendCredentialPin("anthropic", hash!);
seedCredentialPins(storage, manager, sessionId);
expect(storage.listOAuthAccounts("anthropic", sessionId).some(account => account.active)).toBe(false);
});
test("recording appends the serving account's hash once and dedupes repeats", () => {
const manager = SessionManager.create(tempDir.path(), tempDir.path());
const sessionId = manager.getSessionId();
const accounts = storage.listOAuthAccounts("anthropic", sessionId);
const accountA = accounts.find(account => account.accountId === "account-a");
storage.pinSessionOAuthAccount("anthropic", sessionId, accountA!.credentialId);
recordCredentialPin(storage, manager, sessionId, "anthropic");
recordCredentialPin(storage, manager, sessionId, "anthropic");
const entries = manager.getBranch().filter(entry => entry.type === "credential_pin");
expect(entries).toHaveLength(1);
const identity = storage.getOAuthAccountIdentity("anthropic", sessionId);
expect(manager.getCredentialPins().get("anthropic")?.hash).toBe(credentialPinHash("anthropic", identity!));
});
});
packages/coding-agent/test/edit-diff.test.ts
+68 -1
View File
@@ -2,7 +2,12 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { formatHashlineHeader, InMemorySnapshotStore, missingSnapshotTagMessage } from "@oh-my-pi/hashline";
import {
forkClipboard,
formatHashlineHeader,
InMemorySnapshotStore,
missingSnapshotTagMessage,
} from "@oh-my-pi/hashline";
import {
adjustIndentation,
computeEditDiff,
@@ -268,6 +273,68 @@ describe("computeHashlineDiff", () => {
}
});
test("previews a same-file CUT + PASTE as the moved content", async () => {
const sourcePath = path.join(tempDir, "source.txt");
const text = "l1\nl2\nl3\n";
await Bun.write(sourcePath, text);
const snapshotStore = new InMemorySnapshotStore();
const tag = snapshotStore.record(sourcePath, text);
const result = await computeHashlineDiff(
{ input: `${formatHashlineHeader(sourcePath, tag)}\nCUT 1.=1\nPASTE.TAIL` },
tempDir,
snapshotStore,
);
expect("diff" in result).toBe(true);
if ("diff" in result) {
expect(result.diff).toContain("-1|l1");
expect(result.diff).toContain("+3|l1");
}
});
test("previews a PASTE from a forked session register without touching the source", async () => {
const sourcePath = path.join(tempDir, "source.txt");
const text = "l1\n";
await Bun.write(sourcePath, text);
const snapshotStore = new InMemorySnapshotStore();
const tag = snapshotStore.record(sourcePath, text);
// Mirror the streaming-strategy contract: the session register is forked
// once per preview frame; sections then thread the fork in patch order.
const sessionRegister = { lines: ["carried"] };
const result = await computeHashlineDiff(
{ input: `${formatHashlineHeader(sourcePath, tag)}\nPASTE.TAIL` },
tempDir,
snapshotStore,
{ clipboard: forkClipboard(sessionRegister) },
);
expect("diff" in result).toBe(true);
if ("diff" in result) {
expect(result.diff).toContain("|carried");
}
expect(sessionRegister.lines).toEqual(["carried"]);
});
test("streams paste rows in the natural-order preview", async () => {
const sourcePath = path.join(tempDir, "source.txt");
const text = "l1\nl2\nl3\n";
await Bun.write(sourcePath, text);
const snapshotStore = new InMemorySnapshotStore();
const tag = snapshotStore.record(sourcePath, text);
const result = await computeHashlineDiff(
{ input: `${formatHashlineHeader(sourcePath, tag)}\nCUT 2.=2\nPASTE.HEAD` },
tempDir,
snapshotStore,
{ streaming: true, skipHashValidation: true },
);
expect("diff" in result).toBe(true);
if ("diff" in result) {
expect(result.diff).toContain("-2|l2");
expect(result.diff).toContain("+1|l2");
}
});
test("rejects a tagless head/tail insert in the preview path, matching apply", async () => {
const relativePath = "source.txt";
await Bun.write(path.join(tempDir, relativePath), "first\n");
packages/coding-agent/test/edit-streaming-preview.test.ts
+2 -2
View File
@@ -305,12 +305,12 @@ describe("apply_patch streaming preview (trailing partial line)", () => {
describe("matcherDigest", () => {
test("hashline: digests stripped `+` body rows only, never headers or op lines", () => {
const input = ["[a.ts#AB12]", "SWAP 1.=2:", "+const x = 1;", "+const y = 2;", "DEL 5", ""].join("\n");
const input = ["[a.ts#AB12]", "SWAP 1.=2:", "+const x = 1;", "+const y = 2;", "CUT 5", ""].join("\n");
expect(EDIT_MODE_STRATEGIES.hashline.matcherDigest({ input })).toBe("const x = 1;\nconst y = 2;");
});
test("hashline: grammar-only payload digests to empty, missing input to undefined", () => {
expect(EDIT_MODE_STRATEGIES.hashline.matcherDigest({ input: "[a.ts#AB12]\nDEL 3\n" })).toBe("");
expect(EDIT_MODE_STRATEGIES.hashline.matcherDigest({ input: "[a.ts#AB12]\nCUT 3\n" })).toBe("");
expect(EDIT_MODE_STRATEGIES.hashline.matcherDigest({})).toBeUndefined();
});
packages/coding-agent/test/edit/seen-line-guard.test.ts
+2 -2
View File
@@ -262,7 +262,7 @@ describe("read → edit seen-line guard", () => {
expect(seen?.has(1122)).toBe(true);
await executeHashlineSingle(
execOptions(
`[src/main.c#${tag}]\nINS.POST 1122:\n+\tbeep_3k8hz_on();\n+\tk_sleep(K_MSEC(300));\n+\tbeep_3k8hz_off();\nDEL 1288.=1291`,
`[src/main.c#${tag}]\nINS.POST 1122:\n+\tbeep_3k8hz_on();\n+\tk_sleep(K_MSEC(300));\n+\tbeep_3k8hz_off();\nCUT 1288.=1291`,
session,
),
);
@@ -313,7 +313,7 @@ describe("read → edit seen-line guard", () => {
const tag = tagFromOutput(resultText(read));
// Anchor 60 unseen lines — deliberately over the 40-line cap.
const dels = Array.from({ length: 60 }, (_, i) => `DEL ${100 + i}`).join("\n");
const dels = Array.from({ length: 60 }, (_, i) => `CUT ${100 + i}`).join("\n");
let message: string | undefined;
try {
await executeHashlineSingle(execOptions(`[long.txt#${tag}]\n${dels}`, session));
packages/coding-agent/test/export-html-template.test.ts
+9 -5
View File
@@ -19,9 +19,9 @@ interface TemplateProbeResult {
}
const expectedTemplate: TemplateProbeResult = {
chars: 377_268,
bytes: 377_446,
sha256: "023cf3773498db52ff0ec508b22ef2c002710f9e9fcb90fea361659d5b0205df",
chars: 377_286,
bytes: 377_464,
sha256: "6598b2d74f905e3105474ea8033c938e92df4c33035421603ea363d72d1c81c1",
stableCache: true,
assetsRemoved: 0,
};
@@ -78,6 +78,7 @@ function composeExpectedTemplate(): string {
async function runProbe(command: string[]): Promise<TemplateProbeResult> {
const proc = Bun.spawn(command, {
cwd: unrelatedCwd,
stdin: "ignore",
stderr: "pipe",
stdout: "pipe",
});
@@ -149,8 +150,9 @@ describe("HTML export template", () => {
fs.writeFileSync(staleAssetPath, "stale");
const build = Bun.spawn([process.execPath, "run", "gen:bundle"], {
cwd: packageDir,
stderr: "pipe",
stdin: "ignore",
stdout: "pipe",
stderr: "pipe",
});
const [buildStdout, buildStderr, buildExitCode] = await Promise.all([
new Response(build.stdout).text(),
@@ -162,8 +164,9 @@ describe("HTML export template", () => {
const proc = Bun.spawn([process.execPath, "pm", "pack", "--dry-run", "--ignore-scripts"], {
cwd: packageDir,
stderr: "pipe",
stdin: "ignore",
stdout: "pipe",
stderr: "pipe",
});
const [stdout, stderr, exitCode] = await Promise.all([
new Response(proc.stdout).text(),
@@ -195,6 +198,7 @@ describe("HTML export template", () => {
test("does not retain source asset strings during a static import", async () => {
const proc = Bun.spawn([process.execPath, heapProbePath], {
stdin: "ignore",
stderr: "pipe",
stdout: "pipe",
});
packages/coding-agent/test/modes/controllers/resume-preflight.test.ts
+2 -1
View File
@@ -220,8 +220,9 @@ describe("SelectorController.handleResumeSession preflight flush", () => {
selector!.handleInput("\n");
expect(selectionPromise).toBeDefined();
await expect(selectionPromise!).rejects.toBe(switchError);
await selectionPromise;
expect(ctx.showError).toHaveBeenCalledWith("switch failed");
expect(ctx.settings.flush).toHaveBeenCalledTimes(1);
expect(switchSession).toHaveBeenCalledWith(session.path);
expect(hide).toHaveBeenCalledTimes(1);
packages/collab-web/CHANGELOG.md
+2
View File
@@ -2,6 +2,8 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Fixed
- Fixed an issue where the agent would stop silently without a message by ensuring terminal auto-retry failures are properly surfaced as error notices.
packages/hashline/CHANGELOG.md
+16
View File
@@ -2,6 +2,22 @@
## [Unreleased]
## [17.2.0] - 2026-07-30
### Breaking Changes
- Removed `DEL`, `DEL.BLK`, `COPY`, and `COPY.BLK` from the patch language. Use `CUT` / `CUT.BLK` for deletion; a cut does not require a following `PASTE` and leaves the removed content available to later pastes.
### Added
- Added clipboard ops: `CUT N.=M` captures lines into a register (and deletes them), `CUT.BLK N` captures tree-sitter blocks, and `PASTE.PRE|POST N` / `PASTE.HEAD|TAIL` / `PASTE.BLK.POST N` insert the captured lines without retyping. The register flows top-to-bottom across sections, so content moves between files in one patch; `PASTE` does not consume it and the last capture wins.
- Added `PatcherOptions.clipboard` for a host-owned register that persists across `Patcher.apply` batches. Batches work on a fork (`forkClipboard`) published per landed section (`commitClipboard`), so failed batches never poison the register and a mid-batch write failure still preserves content already cut from disk.
- Added clipboard safety guards: a `PASTE` with an empty register, a capture overwriting un-pasted `CUT` content, and clipboard ops in same-path sections interleaved across another file's section are all rejected with targeted diagnostics. `CUT` ranges participate in overlap validation, the seen-lines guard, and drift recovery (every captured line must remap).
### Changed
- Simplified `grammar.lark` around shared target and position shapes, collapsing the concrete and block `CUT` forms plus the `INS` / `PASTE` position variants into their common grammar rules.
### Fixed
- Prevented CPU and memory exhaustion in streaming previews by rejecting line anchors above Number.MAX_SAFE_INTEGER and ranges spanning more than 100,000 lines.
packages/hashline/README.md
+2 -1
View File
@@ -49,9 +49,10 @@ session-aware recovery on mismatch.
Inside a section:
- `SWAP A.=B:` — replace lines A.=B with following `+TEXT` body rows.
- `SWAP.BLK A:` — replace the syntactic block beginning on line A.
- `DEL A.=B` / `DEL.BLK A` — delete concrete lines or a resolved block.
- `CUT A.=B` / `CUT.BLK A` — delete concrete lines or a resolved block and capture them for `PASTE`.
- `INS.PRE A:` / `INS.POST A:` / `INS.HEAD:` / `INS.TAIL:` — insert following body rows.
- `INS.BLK.POST A:` — insert following body rows after the resolved block's last line.
- `PASTE.PRE A` / `PASTE.POST A` / `PASTE.HEAD` / `PASTE.TAIL` / `PASTE.BLK.POST A` — insert the clipboard; no body rows. The register flows top-to-bottom across sections (cross-file moves) and can be host-persisted across batches via `PatcherOptions.clipboard`.
- `REM` — delete the whole file named by the section header.
- `MV DEST` — move/rename the section file to `DEST` (optionally after line edits).
- `+TEXT` — literal body row (use `+` alone for a blank line).
packages/hashline/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"type": "module",
"name": "@oh-my-pi/hashline",
"version": "17.1.8",
"version": "17.2.0",
"description": "Hashline: a compact, line-anchored patch language and applier. Pluggable FS/IO so it works over disk, in-memory, or any custom backend.",
"homepage": "https://omp.sh",
"author": "Can Boluk",
packages/hashline/src/apply.ts
+26 -5
View File
@@ -7,6 +7,8 @@
* which absorbs common model mistakes where a payload restates unchanged range
* boundaries or duplicates/drops structural closers.
*/
import { resolveClipboardEdits } from "./clipboard";
import {
afterInsertLandingShiftWarning,
ambiguousBoundaryEchoMessage,
@@ -15,7 +17,7 @@ import {
UNRESOLVED_BLOCK_INTERNAL,
} from "./messages";
import { cloneCursor } from "./tokenizer";
import type { Anchor, ApplyResult, Cursor, Edit } from "./types";
import type { Anchor, ApplyResult, Clipboard, Cursor, Edit } from "./types";
type LineOrigin = "original" | "insert" | "replacement";
@@ -1221,24 +1223,43 @@ function repairAfterInsertLandings(
return { edits: out ?? edits, warnings };
}
/** Optional knobs for {@link applyEdits}. */
export interface ApplyEditsOptions {
/**
* Clipboard register filled by `cut` edits and read by `paste` edits.
* Thread one register through every section of a batch to move content
* across files; omitted, the call gets a private register.
*/
clipboard?: Clipboard;
/** `PASTE` with an empty register: `throw` (default) or `drop` (streaming previews). */
onEmptyPaste?: "throw" | "drop";
}
/**
* Apply a parsed list of edits to a text body. Pure function — no I/O.
*
* Returns the post-edit text and the first changed line number (1-indexed).
* Throws if an anchor is out of bounds.
*/
export function applyEdits(text: string, edits: readonly Edit[]): ApplyResult {
export function applyEdits(text: string, edits: readonly Edit[], options: ApplyEditsOptions = {}): ApplyResult {
if (edits.length === 0) return { text, firstChangedLine: undefined };
const fileLines = text.split("\n");
// Clipboard pre-pass: capture `cut` ranges from the original lines and
// expand `paste` edits into plain inserts in authored order.
const concrete = resolveClipboardEdits(edits, fileLines, options.clipboard ?? {}, {
...(options.onEmptyPaste === undefined ? {} : { onEmptyPaste: options.onEmptyPaste }),
});
// Block edits are deferred until `resolveBlockEdits` expands them into
// concrete inserts + deletes. Reaching the applier with one still present
// is an internal wiring bug, not authored-input error.
for (const edit of edits) {
for (const edit of concrete) {
if (edit.kind === "block") throw new Error(UNRESOLVED_BLOCK_INTERNAL);
}
const appliedEdits = edits as readonly AppliedEdit[];
const appliedEdits = concrete as readonly AppliedEdit[];
const fileLines = text.split("\n");
const lineOrigins: LineOrigin[] = fileLines.map(() => "original");
let firstChangedLine: number | undefined;
packages/hashline/src/block.ts
+52 -24
View File
@@ -1,25 +1,22 @@
/**
* Expand deferred block edits (`replace_block N:` / `delete_block N` /
* `insert_after_block N:`) into concrete inserts + deletes.
* Expand deferred block edits into concrete inserts, cuts, pastes, and deletes.
*
* The hashline parser cannot expand a block edit on its own — the line span is
* unknown until file text + path (→ language) are available. This transform
* runs at every apply/preview boundary that has text: it calls the injected
* {@link BlockResolver} to resolve each block's `[start, end]` span, then emits
* the exact same edits the concrete form produces in the parser: `replace
* start.=end:` inserts + deletes for a replace, a pure range delete for a
* delete, and plain `after_anchor` inserts at `end` for an insert-after. After
* it runs, no `block` edits remain, so {@link applyEdits} (and recovery) only
* ever see resolved edits.
* The parser cannot expand a block edit until file text and language are
* available. This transform resolves each anchored span, then emits the same
* low-level edits as the corresponding concrete operation. After it runs, no
* `block` edits remain, so {@link applyEdits} and recovery see concrete edits.
*/
import { STRUCTURAL_CLOSER_RE } from "./apply";
import {
BLOCK_RESOLVER_UNAVAILABLE,
type BlockDiagnosticSuggestions,
type BlockOp,
blockSingleLineMessage,
blockUnresolvedMessage,
insertAfterBlockCloserLoweredWarning,
insertAfterBlockUnresolvedLoweredWarning,
pasteAfterBlockCloserLoweredWarning,
pasteAfterBlockUnresolvedLoweredWarning,
} from "./messages";
import type { BlockResolution, BlockResolver, BlockSpan, Cursor, Edit } from "./types";
@@ -67,15 +64,12 @@ function findEnclosingBlock(
return null;
}
/** Optional knobs for {@link resolveBlockEdits}. */
export interface ResolveBlockEditsOptions {
/**
* How to handle a replace/delete block edit that cannot be resolved
* (missing resolver or a `null` span). `"throw"` (default) raises a
* `blockUnresolvedMessage` error — used by the authoritative apply + final
* preview paths. `"drop"` silently skips the edit — used by the streaming
* preview, where a half-written file or transient parse error must not
* throw. Unresolvable `insert_after_block N:` edits never reach this: they
* are lowered to plain `insert after N:` with a warning.
* How to handle a replace/cut block edit that cannot be resolved. `"throw"`
* (default) raises a block error; `"drop"` skips it for streaming previews.
* Unresolvable after-block edits lower to their plain after-line form.
*/
onUnresolved?: "throw" | "drop";
/**
@@ -124,7 +118,7 @@ export function resolveBlockEdits(
resolved.push(edit);
continue;
}
const op = edit.mode === "insert_after" ? "insert_after" : edit.payloads.length === 0 ? "delete" : "replace";
const op: BlockOp = edit.mode ?? "replace";
const span = resolver ? resolver({ path, text, line: edit.anchor.line }) : null;
if (span === null) {
// `insert_after_block N:` never fails the patch — lower it to plain
@@ -135,9 +129,19 @@ export function resolveBlockEdits(
// - otherwise (unsupported language, blank line, unparsable block,
// or no resolver wired): "after the block at N" degrades to
// "after line N" — warn to verify the landing line.
if (op === "insert_after") {
if (op === "insert_after" || op === "paste_after") {
const anchorText = text.split("\n")[edit.anchor.line - 1];
const isCloser = anchorText !== undefined && STRUCTURAL_CLOSER_RE.test(anchorText);
if (op === "paste_after") {
options.onWarning?.(
isCloser
? pasteAfterBlockCloserLoweredWarning(edit.anchor.line)
: pasteAfterBlockUnresolvedLoweredWarning(edit.anchor.line),
);
const cursor: Cursor = { kind: "after_anchor", anchor: { line: edit.anchor.line } };
resolved.push({ kind: "paste", cursor, lineNum: edit.lineNum, index: synthIndex++ });
continue;
}
options.onWarning?.(
isCloser
? insertAfterBlockCloserLoweredWarning(edit.anchor.line)
@@ -183,6 +187,32 @@ export function resolveBlockEdits(
end: span.end,
op,
});
if (op === "paste_after") {
// Mirror the block-lowered insert: paste after the block's last
// line, tagging `blockStart` so landing correction can slide a body
// claiming a depth inside the block back across its trailing closers.
resolved.push({
kind: "paste",
cursor: { kind: "after_anchor", anchor: { line: span.end } },
lineNum: edit.lineNum,
index: synthIndex++,
blockStart: span.start,
});
continue;
}
if (op === "cut") {
// Capture the resolved span before deleting it line-by-line.
resolved.push({
kind: "cut",
range: { start: { line: span.start }, end: { line: span.end } },
lineNum: edit.lineNum,
index: synthIndex++,
});
for (let line = span.start; line <= span.end; line++) {
resolved.push({ kind: "delete", anchor: { line }, lineNum: edit.lineNum, index: synthIndex++ });
}
continue;
}
if (op === "insert_after") {
// Mirror the parser's `insert after N:` lowering: one `after_anchor`
// insert per payload row, anchored on the block's last line. The
@@ -202,10 +232,8 @@ export function resolveBlockEdits(
}
continue;
}
// Mirror the parser's `replace start.=end:` expansion exactly: one
// `before_anchor` replacement insert per payload row at `span.start`,
// then one delete per line across `[span.start, span.end]`. An empty
// `payloads` (from `delete_block N`) emits no inserts — a pure deletion.
// Mirror `SWAP start.=end:`: replacement inserts at `span.start`, then
// one delete per line across the resolved span.
for (const payload of edit.payloads) {
const cursor: Cursor = { kind: "before_anchor", anchor: { line: span.start } };
resolved.push({
packages/hashline/src/clipboard.ts
+110
View File
@@ -0,0 +1,110 @@
/**
* Clipboard register support for `CUT` / `PASTE` ops.
*
* `CUT` captures its current source lines before ordinary delete edits apply.
* `PASTE` expands the latest capture into inserts. One register flows through
* patch sections in authored order, so content moves across files; the latest
* cut wins and paste does not consume it.
*/
import { HL_CUT_KEYWORD, HL_RANGE_SEP } from "./format";
import { EMPTY_PASTE } from "./messages";
import { cloneCursor } from "./tokenizer";
import type { Clipboard, Edit } from "./types";
type CutEdit = Extract<Edit, { kind: "cut" }>;
function describeCutEdit(edit: CutEdit): string {
const { start, end } = edit.range;
const range = start.line === end.line ? `${start.line}` : `${start.line}${HL_RANGE_SEP}${end.line}`;
return `${HL_CUT_KEYWORD} ${range}`;
}
/** True when at least one edit reads or writes the clipboard register. */
export function hasClipboardEdit(edits: readonly Edit[]): boolean {
return edits.some(
edit =>
edit.kind === "cut" ||
edit.kind === "paste" ||
(edit.kind === "block" && (edit.mode === "cut" || edit.mode === "paste_after")),
);
}
/** Optional knobs for {@link resolveClipboardEdits}. */
export interface ResolveClipboardEditsOptions {
/** `PASTE` with an empty register: `throw` (default) or `drop` (streaming previews). */
onEmptyPaste?: "throw" | "drop";
}
/**
* Resolve clipboard edits against the original file lines in authored order.
* Cuts fill the register and emit nothing; pastes become plain inserts.
*/
export function resolveClipboardEdits(
edits: readonly Edit[],
fileLines: readonly string[],
clipboard: Clipboard,
options: ResolveClipboardEditsOptions = {},
): readonly Edit[] {
if (!hasClipboardEdit(edits)) return edits;
const onEmptyPaste = options.onEmptyPaste ?? "throw";
const resolved: Edit[] = [];
let synthIndex = 0;
for (const edit of edits) {
if (edit.kind === "cut") {
const { start, end } = edit.range;
if (start.line < 1 || end.line > fileLines.length) {
throw new Error(
`line ${edit.lineNum}: \`${describeCutEdit(edit)}\` is out of range (file has ${fileLines.length} lines).`,
);
}
clipboard.lines = fileLines.slice(start.line - 1, end.line);
continue;
}
if (edit.kind === "paste") {
const lines = clipboard.lines;
if (lines === undefined) {
if (onEmptyPaste === "drop") continue;
throw new Error(`line ${edit.lineNum}: ${EMPTY_PASTE}`);
}
for (const text of lines) {
resolved.push({
kind: "insert",
cursor: cloneCursor(edit.cursor),
text,
lineNum: edit.lineNum,
index: synthIndex++,
...(edit.blockStart === undefined ? {} : { blockStart: edit.blockStart }),
});
}
continue;
}
resolved.push(edit);
}
return resolved;
}
/** Create a transactional working copy of a clipboard register. */
export function forkClipboard(source?: Clipboard): Clipboard {
return source === undefined ? {} : { ...source };
}
/** Publish a clipboard fork back to its source register. */
export function commitClipboard(fork: Clipboard, target: Clipboard): void {
if (fork.lines === undefined) delete target.lines;
else target.lines = fork.lines;
}
/**
* Validate that every paste has a preceding or persisted capture without
* mutating the register or reading file content.
*/
export function validateClipboardSequence(edits: readonly Edit[], clipboard: Clipboard): void {
let hasLines = clipboard.lines !== undefined;
for (const edit of edits) {
if (edit.kind === "cut") {
hasLines = true;
} else if (edit.kind === "paste" && !hasLines) {
throw new Error(`line ${edit.lineNum}: ${EMPTY_PASTE}`);
}
}
}
packages/hashline/src/format.ts
+11 -7
View File
@@ -15,8 +15,6 @@ export const HL_PAYLOAD_REPLACE = "+";
/** Hunk-header keyword for concrete line replacement. */
export const HL_REPLACE_KEYWORD = "SWAP";
/** Hunk-header keyword for concrete line deletion. */
export const HL_DELETE_KEYWORD = "DEL";
/** Hunk-header keyword for insertion operations. */
export const HL_INSERT_KEYWORD = "INS";
/** Insert position keyword for inserting before a concrete line. */
@@ -29,10 +27,16 @@ export const HL_INSERT_HEAD = "HEAD";
export const HL_INSERT_TAIL = "TAIL";
/** Hunk-header keyword: `SWAP.BLK N:` resolves N to a tree-sitter block range and replaces its span. */
export const HL_REPLACE_BLOCK_KEYWORD = "SWAP.BLK";
/** Hunk-header keyword: `DEL.BLK N` resolves N to a tree-sitter block range and deletes its span. */
export const HL_DELETE_BLOCK_KEYWORD = "DEL.BLK";
/** Hunk-header keyword: `INS.BLK.POST N:` inserts after the last line of the tree-sitter block at N. */
export const HL_INSERT_AFTER_BLOCK_KEYWORD = "INS.BLK.POST";
/** Hunk-header keyword: `CUT N.=M` captures lines into the clipboard register and deletes them. */
export const HL_CUT_KEYWORD = "CUT";
/** Hunk-header keyword: `CUT.BLK N` captures the tree-sitter block at N and deletes its span. */
export const HL_CUT_BLOCK_KEYWORD = "CUT.BLK";
/** Hunk-header keyword prefix: `PASTE.PRE|POST N` / `PASTE.HEAD|TAIL` inserts the clipboard. */
export const HL_PASTE_KEYWORD = "PASTE";
/** Hunk-header keyword: `PASTE.BLK.POST N` inserts the clipboard after the tree-sitter block at N. */
export const HL_PASTE_AFTER_BLOCK_KEYWORD = "PASTE.BLK.POST";
/** File-level keyword: `REM` deletes the whole file named by the section header. */
export const HL_REM_KEYWORD = "REM";
/** File-level keyword: `MV DEST` renames/moves the section file to `DEST`. */
@@ -63,9 +67,9 @@ export function formatReplaceHeader(start: number, end: number): string {
return `${HL_REPLACE_KEYWORD} ${start}${HL_RANGE_SEP}${end}${HL_HEADER_COLON}`;
}
/** Format a concrete deletion hunk header. */
export function formatDeleteHeader(start: number, end = start): string {
return start === end ? `${HL_DELETE_KEYWORD} ${start}` : `${HL_DELETE_KEYWORD} ${start}${HL_RANGE_SEP}${end}`;
/** Format a concrete cut hunk header. */
export function formatCutHeader(start: number, end = start): string {
return start === end ? `${HL_CUT_KEYWORD} ${start}` : `${HL_CUT_KEYWORD} ${start}${HL_RANGE_SEP}${end}`;
}
/** Format an insertion hunk header for a cursor position. */
packages/hashline/src/grammar.lark
+12 -15
View File
@@ -7,21 +7,18 @@ file_header: "[" filename "#" file_hash "]" LF
file_hash: /[0-9A-F]{4}/
filename: /[^#\r\n]+/
hunk: replace_hunk | replace_block_hunk | insert_hunk | insert_block_hunk | delete_hunk | delete_block_hunk | remove_hunk | move_hunk
replace_hunk: replace_anchor LF emit_op*
replace_block_hunk: replace_block_anchor LF emit_op+
insert_hunk: insert_anchor LF emit_op+
insert_block_hunk: insert_block_anchor LF emit_op+
delete_hunk: "DEL " header_range LF
delete_block_hunk: "DEL.BLK " LID LF
remove_hunk: "REM" LF
move_hunk: "MV " filename LF emit_op*
replace_anchor: "SWAP " header_range ":"
replace_block_anchor: "SWAP.BLK " LID ":"
insert_anchor: "INS." insert_pos ":"
insert_block_anchor: "INS.BLK.POST " LID ":"
insert_pos: "PRE " LID | "POST " LID | "HEAD" | "TAIL"
emit_op: "+" /(.*)/ LF
hunk: swap_hunk | ins_hunk | cut_hunk | paste_hunk | rem_hunk | mv_hunk
swap_hunk: "SWAP " header_range ":" LF body*
| "SWAP.BLK " LID ":" LF body+
ins_hunk: "INS." pos ":" LF body+
cut_hunk: "CUT" target LF
paste_hunk: "PASTE." pos LF
rem_hunk: "REM" LF
mv_hunk: "MV " filename LF body*
target: " " header_range | ".BLK " LID
pos: "PRE " LID | "POST " LID | "BLK.POST " LID | "HEAD" | "TAIL"
body: "+" /(.*)/ LF
header_range: LID ".=" LID
LID: /[1-9]\d*/
packages/hashline/src/index.ts
+1
View File
@@ -1,5 +1,6 @@
export * from "./apply";
export * from "./block";
export * from "./clipboard";
export * from "./diff-preview";
export * from "./format";
export * from "./fs";
packages/hashline/src/input.ts
+44 -7
View File
@@ -10,10 +10,12 @@
import * as path from "node:path";
import { applyEdits } from "./apply";
import { resolveBlockEdits } from "./block";
import { hasClipboardEdit } from "./clipboard";
import { HL_FILE_HASH_EXAMPLES, HL_FILE_HASH_LENGTH, HL_FILE_HASH_SEP, HL_FILE_PREFIX, HL_FILE_SUFFIX } from "./format";
import { CLIPBOARD_INTERLEAVED_SECTIONS } from "./messages";
import { parsePatch, parsePatchStreaming } from "./parser";
import { Tokenizer } from "./tokenizer";
import type { ApplyResult, BlockResolver, Edit, FileOp, SplitOptions } from "./types";
import type { ApplyResult, BlockResolver, Clipboard, Edit, FileOp, SplitOptions } from "./types";
// Pure classification — single shared tokenizer is safe.
const TOKENIZER = new Tokenizer();
@@ -97,6 +99,14 @@ interface RawSection {
path: string;
fileHash?: string;
diff: string;
/**
* True when this section coalesced same-path sections that were NOT
* adjacent in the authored input (another file's section sat between
* them). Merging moves the later ops up to the first occurrence, which
* would silently reorder the clipboard register sequence — so clipboard
* ops are rejected in such sections.
*/
interleaved?: boolean;
}
/**
@@ -239,10 +249,13 @@ export class PatchSection {
readonly diff: string;
#parsed: { edits: Edit[]; fileOp?: FileOp; warnings: string[] } | undefined;
#interleavedMerge: boolean;
constructor(raw: RawSection) {
this.path = raw.path;
this.fileHash = raw.fileHash;
this.diff = raw.diff;
this.#interleavedMerge = raw.interleaved === true;
}
/**
@@ -253,6 +266,12 @@ export class PatchSection {
parse(): { edits: Edit[]; fileOp?: FileOp; warnings: readonly string[] } {
this.#parsed ??= parsePatch(this.diff);
const parsed = this.#parsed;
// Same-path sections merge into their first occurrence; when that merge
// crossed another file's section, the authored top-to-bottom register
// order is gone, so clipboard ops cannot apply deterministically.
if (this.#interleavedMerge && hasClipboardEdit(parsed.edits)) {
throw new Error(CLIPBOARD_INTERLEAVED_SECTIONS);
}
const fileOp =
parsed.fileOp === undefined
? undefined
@@ -289,6 +308,8 @@ export class PatchSection {
if (edit.kind === "delete") return true;
// A `replace_block N:` edit is anchored to concrete content on line N.
if (edit.kind === "block") return true;
// A `CUT` range reads concrete content.
if (edit.kind === "cut") return true;
return edit.cursor.kind === "before_anchor" || edit.cursor.kind === "after_anchor";
});
}
@@ -305,6 +326,10 @@ export class PatchSection {
lines.add(edit.anchor.line);
continue;
}
if (edit.kind === "cut") {
for (let line = edit.range.start.line; line <= edit.range.end.line; line++) lines.add(line);
continue;
}
if (edit.cursor.kind === "before_anchor" || edit.cursor.kind === "after_anchor") {
lines.add(edit.cursor.anchor.line);
}
@@ -321,15 +346,18 @@ export class PatchSection {
*
* `blockResolver` resolves any `replace_block N:` edits against `text`; an
* unresolvable block throws (this is the final, authoritative preview path).
*
* `clipboard` is the register shared by `CUT`/`PASTE` ops. Pass one when
* applying several sections so content can move across files.
*/
applyTo(text: string, blockResolver?: BlockResolver): ApplyResult {
applyTo(text: string, blockResolver?: BlockResolver, clipboard?: Clipboard): ApplyResult {
const { edits, warnings } = this.parse();
const resolveWarnings: string[] = [];
const resolved = resolveBlockEdits(edits, text, this.path, blockResolver, {
onUnresolved: "throw",
onWarning: warning => resolveWarnings.push(warning),
});
const result = applyEdits(text, resolved);
const result = applyEdits(text, resolved, { clipboard: clipboard ?? {} });
// Preserve parse warnings so consumers don't need to call `parse()`
// separately.
const merged = [...warnings, ...resolveWarnings, ...(result.warnings ?? [])];
@@ -347,16 +375,16 @@ export class PatchSection {
*
* `blockResolver` resolves any `replace_block N:` edits against `text`; an
* unresolvable block is silently dropped so a half-written file does not
* throw mid-stream.
* throw mid-stream. A `PASTE` with an empty register is dropped too.
*/
applyPartialTo(text: string, blockResolver?: BlockResolver): ApplyResult {
applyPartialTo(text: string, blockResolver?: BlockResolver, clipboard?: Clipboard): ApplyResult {
const { edits, warnings } = parsePatchStreaming(this.diff);
const resolveWarnings: string[] = [];
const resolved = resolveBlockEdits(edits, text, this.path, blockResolver, {
onUnresolved: "drop",
onWarning: warning => resolveWarnings.push(warning),
});
const result = applyEdits(text, resolved);
const result = applyEdits(text, resolved, { clipboard: clipboard ?? {}, onEmptyPaste: "drop" });
const merged = [...warnings, ...resolveWarnings, ...(result.warnings ?? [])];
return merged.length > 0
? { ...result, warnings: merged }
@@ -374,6 +402,7 @@ export class PatchSection {
path,
...(this.fileHash !== undefined ? { fileHash: this.fileHash } : {}),
diff: this.diff,
...(this.#interleavedMerge ? { interleaved: true } : {}),
});
next.#parsed = this.#parsed;
return next;
@@ -432,7 +461,8 @@ export class Patch {
* fails. Path order is preserved by first occurrence.
*/
function mergeSamePathSections(sections: RawSection[]): RawSection[] {
const byPath = new Map<string, { fileHash?: string; diffs: string[] }>();
const byPath = new Map<string, { fileHash?: string; diffs: string[]; interleaved: boolean }>();
let previousPath: string | undefined;
for (const section of sections) {
const existing = byPath.get(section.path);
if (existing) {
@@ -446,17 +476,24 @@ function mergeSamePathSections(sections: RawSection[]): RawSection[] {
);
}
if (existing.fileHash === undefined && section.fileHash !== undefined) existing.fileHash = section.fileHash;
// Merging across another file's section moves these ops up to the
// first occurrence; flag it so clipboard ops can refuse the reorder.
if (previousPath !== section.path) existing.interleaved = true;
existing.diffs.push(section.diff);
previousPath = section.path;
continue;
}
byPath.set(section.path, {
...(section.fileHash !== undefined ? { fileHash: section.fileHash } : {}),
diffs: [section.diff],
interleaved: false,
});
previousPath = section.path;
}
return Array.from(byPath, ([sectionPath, entry]) => ({
path: sectionPath,
...(entry.fileHash !== undefined ? { fileHash: entry.fileHash } : {}),
diff: entry.diffs.join("\n"),
...(entry.interleaved ? { interleaved: true } : {}),
}));
}
packages/hashline/src/messages.ts
+96 -59
View File
@@ -31,7 +31,19 @@ export function formatAnchoredContext(anchorLines: readonly number[], fileLines:
return rows;
}
/** Concrete range operation rejected because its absolute end precedes its start. */
export type AbsoluteRangeOp = "replace" | "delete";
export type AbsoluteRangeOp = "replace" | "cut";
/** Header forms per concrete-range op, used to compose retry suggestions. */
const RANGE_OP_FORMS: Record<AbsoluteRangeOp, { keyword: string; colon: string; blockKeyword: string }> = {
replace: { keyword: "SWAP", colon: ":", blockKeyword: "SWAP.BLK" },
cut: { keyword: "CUT", colon: "", blockKeyword: "CUT.BLK" },
};
/** `OP.BLK N`-style header for a concrete-range op (`SWAP.BLK 5:` / `CUT.BLK 5`). */
function blockFormAt(op: AbsoluteRangeOp, line: number): string {
const forms = RANGE_OP_FORMS[op];
return `${forms.blockKeyword} ${line}${forms.colon}`;
}
/** Explain absolute range endpoints and provide safe, non-applying retry forms. */
export function invalidAbsoluteRangeMessage(
@@ -41,15 +53,14 @@ export function invalidAbsoluteRangeMessage(
op: AbsoluteRangeOp,
block?: BlockSpan,
): string {
const single = op === "replace" ? `SWAP ${start}${HL_RANGE_SEP}${start}:` : `DEL ${start}`;
const forms = RANGE_OP_FORMS[op];
const single = op === "replace" ? `SWAP ${start}${HL_RANGE_SEP}${start}:` : `${forms.keyword} ${start}`;
const countedEnd = start + end - 1;
const counted =
Number.isSafeInteger(countedEnd) && countedEnd >= start
? op === "replace"
? `SWAP ${start}${HL_RANGE_SEP}${countedEnd}:`
: `DEL ${start}${HL_RANGE_SEP}${countedEnd}`
? `${forms.keyword} ${start}${HL_RANGE_SEP}${countedEnd}${forms.colon}`
: null;
const blockForm = op === "replace" ? `SWAP.BLK ${start}:` : `DEL.BLK ${start}`;
const blockForm = blockFormAt(op, start);
let message =
`line ${patchLine}: Invalid absolute range: start ${start}, end ${end}. ` +
`The value after \`${HL_RANGE_SEP}\` is an absolute source line, not a line count or replacement length. ` +
@@ -99,10 +110,10 @@ export const MINUS_ROW_REJECTED =
"`-` rows are not valid; the range already names the lines being changed. For Markdown bullets or other literal `-` lines, prefix the literal row with `+`: `+- item`.";
/** Replace hunk with no body. */
export const EMPTY_REPLACE = `\`SWAP N${HL_RANGE_SEP}M:\` needs at least one \`+TEXT\` body row. To delete lines, use \`DEL N${HL_RANGE_SEP}M\`.`;
export const EMPTY_REPLACE = `\`SWAP N${HL_RANGE_SEP}M:\` needs at least one \`+TEXT\` body row. To delete lines, use \`CUT N${HL_RANGE_SEP}M\`.`;
/** `replace_block N:` hunk with no body. */
export const EMPTY_BLOCK = "`SWAP.BLK N:` needs at least one `+TEXT` body row. To delete a block, use `DEL.BLK N`.";
/** `SWAP.BLK N:` hunk with no body. */
export const EMPTY_BLOCK = "`SWAP.BLK N:` needs at least one `+TEXT` body row. To delete a block, use `CUT.BLK N`.";
/** Optional source-aware suggestions appended to block-anchor diagnostics. */
export interface BlockDiagnosticSuggestions {
@@ -113,26 +124,24 @@ export interface BlockDiagnosticSuggestions {
}
/**
* Block-anchored replace/delete could not resolve to a syntactic block
* (unsupported language, blank/out-of-range line, no node beginning on N, or
* parse error). Appends a {@link formatAnchoredContext} preview when
* `fileLines` is given. `insert_after_block N:` never reaches this — it is
* lowered to plain `insert after N:` instead (see
* {@link insertAfterBlockUnresolvedLoweredWarning}).
* A block-anchored replace/cut could not resolve to a syntactic block.
* Appends a {@link formatAnchoredContext} preview when `fileLines` is given.
* `INS.BLK.POST N:` never reaches this path; it lowers to `INS.POST N:`.
*/
export function blockUnresolvedMessage(
line: number,
op: "replace" | "delete" = "replace",
op: AbsoluteRangeOp = "replace",
fileLines?: readonly string[],
suggestions: BlockDiagnosticSuggestions = {},
): string {
const phrase = op === "delete" ? `DEL.BLK ${line}` : `SWAP.BLK ${line}:`;
const fallback = op === "delete" ? `DEL ${line}${HL_RANGE_SEP}M` : `SWAP ${line}${HL_RANGE_SEP}M:`;
const forms = RANGE_OP_FORMS[op];
const phrase = blockFormAt(op, line);
const fallback = `${forms.keyword} ${line}${HL_RANGE_SEP}M${forms.colon}`;
const anchorText = fileLines?.[line - 1];
const nextBlock = suggestions.nextBlock;
let message: string;
if (anchorText !== undefined && anchorText.trim().length === 0 && nextBlock) {
const retry = op === "delete" ? `DEL.BLK ${nextBlock.start}` : `SWAP.BLK ${nextBlock.start}:`;
const retry = blockFormAt(op, nextBlock.start);
message =
`Line ${line} is blank; no syntactic block can begin there. ` +
`The next multi-line block begins at line ${nextBlock.start} and ends at line ${nextBlock.end}. ` +
@@ -144,7 +153,7 @@ export function blockUnresolvedMessage(
}
const enclosingBlock = suggestions.enclosingBlock;
if (enclosingBlock) {
const retry = op === "delete" ? `DEL.BLK ${enclosingBlock.start}` : `SWAP.BLK ${enclosingBlock.start}:`;
const retry = blockFormAt(op, enclosingBlock.start);
message +=
` The nearest enclosing multi-line block begins at line ${enclosingBlock.start} ` +
`and ends at line ${enclosingBlock.end}; use \`${retry}\` to target it.`;
@@ -156,26 +165,46 @@ export function blockUnresolvedMessage(
return message;
}
/** Block-anchored edit reached a path with no {@link BlockResolver} wired in — a host-configuration bug. */
/** Block-anchored edit reached a path with no {@link BlockResolver} wired in. */
export const BLOCK_RESOLVER_UNAVAILABLE =
"`SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST` are not available here (no block resolver configured). Use a concrete line range.";
"Block ops (`SWAP.BLK`, `INS.BLK.POST`, `CUT.BLK`, `PASTE.BLK.POST`) are not available here (no block resolver configured). Use a concrete line range.";
/**
* `insert_after_block N:` anchored on a closing-delimiter line, lowered to
* plain `insert after N:` — the closer ends a block, and inserting after it
* is exactly what the plain form does.
* An after-block op anchored on a closing-delimiter line, lowered to its
* plain after-line form — the closer ends a block, and inserting after it is
* exactly what the plain form does.
*/
export function insertAfterBlockCloserLoweredWarning(line: number): string {
return `\`INS.BLK.POST ${line}:\` anchors on a closing delimiter, so it was applied as plain \`INS.POST ${line}:\`. Anchor on the line that OPENS the construct.`;
function closerLoweredWarning(blockForm: string, plainForm: string): string {
return `\`${blockForm}\` anchors on a closing delimiter, so it was applied as plain \`${plainForm}\`. Anchor on the line that OPENS the construct.`;
}
/**
* `insert_after_block N:` anchor unresolvable (unsupported language, blank
* line, parse error, or no resolver), lowered to plain `insert after N:` —
* applying with a warning beats failing the patch.
* An after-block op whose anchor was unresolvable (unsupported language,
* blank line, parse error, or no resolver), lowered to its plain after-line
* form — applying with a warning beats failing the patch.
*/
function unresolvedLoweredWarning(blockForm: string, line: number, plainForm: string): string {
return `\`${blockForm}\` could not resolve a syntactic block on line ${line}, so it was applied as plain \`${plainForm}\`. Verify the landing line; anchor on a line that OPENS a construct.`;
}
/** `INS.BLK.POST N:` anchored on a closing-delimiter line; applied as `INS.POST N:`. */
export function insertAfterBlockCloserLoweredWarning(line: number): string {
return closerLoweredWarning(`INS.BLK.POST ${line}:`, `INS.POST ${line}:`);
}
/** `INS.BLK.POST N:` anchor unresolvable; applied as `INS.POST N:`. */
export function insertAfterBlockUnresolvedLoweredWarning(line: number): string {
return `\`INS.BLK.POST ${line}:\` could not resolve a syntactic block on line ${line}, so it was applied as plain \`INS.POST ${line}:\`. Verify the landing line; anchor on a line that OPENS a construct.`;
return unresolvedLoweredWarning(`INS.BLK.POST ${line}:`, line, `INS.POST ${line}:`);
}
/** `PASTE.BLK.POST N` anchored on a closing-delimiter line; applied as `PASTE.POST N`. */
export function pasteAfterBlockCloserLoweredWarning(line: number): string {
return closerLoweredWarning(`PASTE.BLK.POST ${line}`, `PASTE.POST ${line}`);
}
/** `PASTE.BLK.POST N` anchor unresolvable; applied as `PASTE.POST N`. */
export function pasteAfterBlockUnresolvedLoweredWarning(line: number): string {
return unresolvedLoweredWarning(`PASTE.BLK.POST ${line}`, line, `PASTE.POST ${line}`);
}
/**
* A one-sided boundary echo whose payload is too short to be the widened
@@ -228,14 +257,11 @@ export function ambiguousCloserSpareMessage(
}
/**
* Internal invariant: `applyEdits` received an unresolved `replace_block N:`
* edit; `resolveBlockEdits` must run first. Wiring bug, not authored input.
* Internal invariant: `applyEdits` received an unresolved block edit;
* `resolveBlockEdits` must run first.
*/
export const UNRESOLVED_BLOCK_INTERNAL =
"internal error: unresolved `SWAP.BLK` edit reached the applier (resolveBlockEdits was not run).";
/** Delete hunk received a body row. */
export const DELETE_TAKES_NO_BODY = `\`DEL N${HL_RANGE_SEP}M\` does not take body rows. Remove the body, or use \`SWAP N${HL_RANGE_SEP}M:\`.`;
"internal error: unresolved block edit reached the applier (resolveBlockEdits was not run).";
/** `REM` received a body row or coexists with line edits. */
export const REM_TAKES_NO_BODY =
@@ -245,8 +271,23 @@ export const REM_TAKES_NO_BODY =
export const MOVE_TAKES_NO_BODY =
"`MV DEST` does not take body rows. Put line edits above the `MV` row; the destination path follows `MV` on the same line.";
/** `delete_block N` hunk received a body row. */
export const DELETE_BLOCK_TAKES_NO_BODY = "`DEL.BLK N` does not take body rows. Remove the body, or use `SWAP.BLK N:`.";
/** `CUT N.=M` hunk received a body row. */
export const CUT_TAKES_NO_BODY = `\`CUT N${HL_RANGE_SEP}M\` captures + deletes lines and takes no body rows. To replace lines with new content, use \`SWAP N${HL_RANGE_SEP}M:\`.`;
/** `PASTE` hunk received a body row. */
export const PASTE_TAKES_NO_BODY =
"`PASTE` inserts the clipboard content and takes no `+` body rows. To insert literal text, use `INS`.";
/** `PASTE` ran with an empty clipboard register. */
export const EMPTY_PASTE = `\`PASTE\` found nothing in the clipboard. Ops run top-to-bottom across the whole patch (sections included): put \`CUT N${HL_RANGE_SEP}M\` or \`CUT.BLK N\` above the \`PASTE\`.`;
/**
* Clipboard ops inside a same-path section that was merged across another
* file's section. Same-path sections coalesce into their first occurrence, so
* an interleaved layout would silently reorder the register sequence.
*/
export const CLIPBOARD_INTERLEAVED_SECTIONS =
"`CUT`/`PASTE` cannot be used in a file whose sections are interleaved with another file's: same-path sections merge into the first occurrence, which would reorder the clipboard sequence. Keep each file's ops under ONE `[path#TAG]` header.";
/** Insert hunk with no body. */
export const EMPTY_INSERT = "`INS` needs at least one `+TEXT` body row.";
@@ -413,33 +454,29 @@ export function unseenLinesMessage(
}
/** Op kind of a deferred block edit, for {@link blockSingleLineMessage}. */
export type BlockOp = "replace" | "delete" | "insert_after";
export type BlockOp = "replace" | "insert_after" | "cut" | "paste_after";
/** Display forms per deferred-block op: block keyword, trailing colon, and single-line plain form. */
const BLOCK_OP_FORMS: Record<BlockOp, { keyword: string; colon: string; plain: (line: number) => string }> = {
replace: { keyword: "SWAP.BLK", colon: ":", plain: line => `SWAP ${line}${HL_RANGE_SEP}${line}:` },
insert_after: { keyword: "INS.BLK.POST", colon: ":", plain: line => `INS.POST ${line}:` },
cut: { keyword: "CUT.BLK", colon: "", plain: line => `CUT ${line}` },
paste_after: { keyword: "PASTE.BLK.POST", colon: "", plain: line => `PASTE.POST ${line}` },
};
/**
* A `replace_block`/`delete_block`/`insert_after_block` anchor resolved to a
* single line — almost always a bare statement the model mis-anchored, not a
* multi-line construct. The plain op is unambiguous for one line; the block
* form only earns its keep when it spares counting a closing line you cannot
* see. Reject and point at both fixes.
* A block-op anchor resolved to a single line: line N is a bare statement,
* not the opening line of a multi-line construct. The plain op is exact for
* one line, so reject and point at it.
*/
export function blockSingleLineMessage(line: number, op: BlockOp, enclosingBlock?: BlockSpan): string {
const blockForm = op === "insert_after" ? "INS.BLK.POST" : op === "delete" ? "DEL.BLK" : "SWAP.BLK";
const plainForm =
op === "insert_after"
? `INS.POST ${line}:`
: op === "delete"
? `DEL ${line}`
: `SWAP ${line}${HL_RANGE_SEP}${line}:`;
const forms = BLOCK_OP_FORMS[op];
const plainForm = forms.plain(line);
let message =
`\`${blockForm} ${line}\` resolved a single-line block — line ${line} is a bare statement, not the opening line ` +
`\`${forms.keyword} ${line}\` resolved a single-line block — line ${line} is a bare statement, not the opening line ` +
`of a multi-line construct. For only this statement use \`${plainForm}\`.`;
if (enclosingBlock) {
const enclosingForm =
op === "insert_after"
? `INS.BLK.POST ${enclosingBlock.start}:`
: op === "delete"
? `DEL.BLK ${enclosingBlock.start}`
: `SWAP.BLK ${enclosingBlock.start}:`;
const enclosingForm = `${forms.keyword} ${enclosingBlock.start}${forms.colon}`;
message +=
` The nearest enclosing multi-line block begins at line ${enclosingBlock.start} ` +
`and ends at line ${enclosingBlock.end}; use \`${enclosingForm}\` to target it.`;
packages/hashline/src/parser.ts
+70 -25
View File
@@ -7,14 +7,14 @@ import { HL_PAYLOAD_REPLACE, HL_RANGE_SEP } from "./format";
import {
type AbsoluteRangeOp,
BARE_BODY_AUTO_PIPED_WARNING,
DELETE_BLOCK_TAKES_NO_BODY,
DELETE_TAKES_NO_BODY,
CUT_TAKES_NO_BODY,
EMPTY_BLOCK,
EMPTY_INSERT,
invalidAbsoluteRangeMessage,
MINUS_BULLET_AUTO_PIPED_WARNING,
MINUS_ROW_REJECTED,
MOVE_TAKES_NO_BODY,
PASTE_TAKES_NO_BODY,
REM_TAKES_NO_BODY,
} from "./messages";
import { stripOneLeadingHashlinePrefix } from "./prefixes";
@@ -75,6 +75,23 @@ function isSkippableCommentLine(line: string): boolean {
return line.trimStart().startsWith("#");
}
/**
* Body-row rejection message for targets that take no `+TEXT` rows, or `null`
* for targets whose header is followed by a body.
*/
function bodylessTargetMessage(target: BlockTarget): string | null {
switch (target.kind) {
case "cut":
case "cut_block":
return CUT_TAKES_NO_BODY;
case "paste":
case "paste_after_block":
return PASTE_TAKES_NO_BODY;
default:
return null;
}
}
/**
* Stripped remainder of a bare `N: <value>` row that is a lone quoted or
* numeric literal (optionally comma-terminated) — the shape of a numeric-keyed
@@ -102,13 +119,13 @@ function detectApplyPatchContamination(text: string, _hasPending: boolean): stri
return (
`apply_patch sentinel ${JSON.stringify(preview)} is not valid in hashline. ` +
"File sections start with `[path#HASH]` (no `Update File:` / `Add File:` keyword). " +
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`DEL N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`CUT N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
);
}
if (/^@@\s+[-+]?\d+,\d+\s+[-+]?\d+,\d+\s+@@/.test(trimmed)) {
return (
"unified-diff hunk header (`@@ -N,M +N,M @@`) is not valid in hashline. " +
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`DEL N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`CUT N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
);
}
if (trimmed.startsWith("@@")) {
@@ -118,17 +135,20 @@ function detectApplyPatchContamination(text: string, _hasPending: boolean): stri
`Drop the \`@@ ... @@\` brackets and write a verb header such as \`SWAP N${HL_RANGE_SEP}M:\`.`
);
}
if (/^DEL\s+[1-9]\d*(?:\s*(?:\.\.|\.=|-|…|\s)\s*[1-9]\d*)?\s*:/.test(trimmed)) {
return `\`DEL N${HL_RANGE_SEP}M\` has no colon and no body. Remove the colon and body rows.`;
// Bare `PASTE` (optionally `PASTE 5` / `PASTE:`) — the op requires an
// explicit position suffix; a bare form would otherwise surface as a
// confusing body-row rejection under the preceding hunk.
if (/^PASTE(?:\s+[1-9]\d*)?\s*:?\s*$/.test(trimmed)) {
return "`PASTE` needs a position: use `PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` / `PASTE.TAIL` / `PASTE.BLK.POST N`.";
}
if (/^[1-9]\d*\s*$/.test(trimmed)) {
return `hunk headers need a verb. Use \`SWAP ${trimmed}${HL_RANGE_SEP}${trimmed}:\` to replace, or \`DEL ${trimmed}\` to delete.`;
return `hunk headers need a verb. Use \`SWAP ${trimmed}${HL_RANGE_SEP}${trimmed}:\` to replace, or \`CUT ${trimmed}\` to delete.`;
}
const bareRange = /^([1-9]\d*)\s*[-. …=]+\s*([1-9]\d*)\s*:?$/.exec(trimmed);
if (bareRange !== null) {
return (
`bare range hunk header ${JSON.stringify(trimmed)} is not valid. ` +
`Hunk headers need a verb: write \`SWAP ${bareRange[1]}${HL_RANGE_SEP}${bareRange[2]}:\` or \`DEL ${bareRange[1]}${HL_RANGE_SEP}${bareRange[2]}\`.`
`Hunk headers need a verb: write \`SWAP ${bareRange[1]}${HL_RANGE_SEP}${bareRange[2]}:\` or \`CUT ${bareRange[1]}${HL_RANGE_SEP}${bareRange[2]}\`.`
);
}
return null;
@@ -207,8 +227,11 @@ export class Executor {
return;
case "op-block":
this.#discardPendingSkippableComments();
if (token.target.kind === "replace" || token.target.kind === "delete") {
validateRange(token.target.range, token.lineNum, token.target.kind);
if (token.target.kind === "replace") {
validateRange(token.target.range, token.lineNum, "replace");
}
if (token.target.kind === "cut") {
validateRange(token.target.range, token.lineNum, "cut");
}
if (token.target.kind === "rem") {
this.#flushPending();
@@ -241,8 +264,7 @@ export class Executor {
endStreaming(): { edits: Edit[]; fileOp?: FileOp; warnings: string[] } {
this.#consumePendingSkippableComments();
if (this.#pending && this.#pending.payloads.length > 0) this.#flushPending();
else if (this.#pending?.target.kind === "delete" || this.#pending?.target.kind === "delete_block")
this.#flushPending();
else if (this.#pending && bodylessTargetMessage(this.#pending.target) !== null) this.#flushPending();
else this.#pending = undefined;
this.#validateFileOp();
this.#validateNoOverlappingDeletes();
@@ -312,8 +334,8 @@ export class Executor {
`Got ${JSON.stringify(`${HL_PAYLOAD_REPLACE}${text}`)}.`,
);
}
if (pending.target.kind === "delete") throw new Error(`line ${lineNum}: ${DELETE_TAKES_NO_BODY}`);
if (pending.target.kind === "delete_block") throw new Error(`line ${lineNum}: ${DELETE_BLOCK_TAKES_NO_BODY}`);
const noBodyOnLiteral = bodylessTargetMessage(pending.target);
if (noBodyOnLiteral !== null) throw new Error(`line ${lineNum}: ${noBodyOnLiteral}`);
this.#commitDeferredBlanks(pending);
pending.payloads.push({ kind: "literal", text, lineNum });
}
@@ -327,9 +349,8 @@ export class Executor {
this.#handleBlank(text, lineNum);
return;
}
if (this.#pending.target.kind === "delete") throw new Error(`line ${lineNum}: ${DELETE_TAKES_NO_BODY}`);
if (this.#pending.target.kind === "delete_block")
throw new Error(`line ${lineNum}: ${DELETE_BLOCK_TAKES_NO_BODY}`);
const noBodyOnRaw = bodylessTargetMessage(this.#pending.target);
if (noBodyOnRaw !== null) throw new Error(`line ${lineNum}: ${noBodyOnRaw}`);
const row: PayloadRow = { kind: "literal", text, lineNum, bare: true };
// `-` rows are held and judged at flush time by #resolveMinusRows,
// once the whole body is visible.
@@ -350,7 +371,7 @@ export class Executor {
if (text.trim().length === 0) return;
throw new Error(
`line ${lineNum}: payload line has no preceding hunk header. ` +
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`DEL N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got ${JSON.stringify(text)}.`,
`Use \`SWAP N${HL_RANGE_SEP}M:\`, \`CUT N${HL_RANGE_SEP}M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got ${JSON.stringify(text)}.`,
);
}
@@ -364,7 +385,7 @@ export class Executor {
#handleBlank(text: string, lineNum: number): void {
const pending = this.#pending;
if (!pending) return;
if (pending.target.kind === "delete" || pending.target.kind === "delete_block") return;
if (bodylessTargetMessage(pending.target) !== null) return;
if (pending.payloads.length === 0) return;
pending.deferredBlanks.push({ kind: "literal", text, lineNum, bare: true });
}
@@ -455,7 +476,24 @@ export class Executor {
for (let line = range.start.line; line <= range.end.line; line++) this.#pushDelete({ line }, lineNum);
}
#pushBlock(anchor: Anchor, payloads: readonly PayloadRow[], lineNum: number, mode?: "insert_after"): void {
#pushCut(range: ParsedRange, lineNum: number): void {
this.#edits.push({
kind: "cut",
range: { start: { ...range.start }, end: { ...range.end } },
lineNum,
index: this.#editIndex++,
});
// Capture before ordinary per-line deletes are applied. Keeping deletion
// as low-level edits preserves overlap validation and recovery remapping.
this.#pushDeleteRange(range, lineNum);
}
#pushBlock(
anchor: Anchor,
payloads: readonly PayloadRow[],
lineNum: number,
mode?: "insert_after" | "cut" | "paste_after",
): void {
this.#edits.push({
kind: "block",
anchor: { ...anchor },
@@ -477,13 +515,20 @@ export class Executor {
this.#resolveMinusRows(payloads);
this.#stripBarePrefixesIfUniform(payloads);
this.#pending = undefined;
if (target.kind === "delete") {
this.#pushDeleteRange(target.range, lineNum);
if (target.kind === "cut") {
this.#pushCut(target.range, lineNum);
return;
}
if (target.kind === "delete_block") {
// A block edit with no payloads resolves to a pure block deletion.
this.#pushBlock(target.anchor, [], lineNum);
if (target.kind === "cut_block") {
this.#pushBlock(target.anchor, [], lineNum, "cut");
return;
}
if (target.kind === "paste") {
this.#edits.push({ kind: "paste", cursor: cloneCursor(target.cursor), lineNum, index: this.#editIndex++ });
return;
}
if (target.kind === "paste_after_block") {
this.#pushBlock(target.anchor, [], lineNum, "paste_after");
return;
}
if (target.kind === "block") {
packages/hashline/src/patcher.ts
+56 -12
View File
@@ -25,6 +25,7 @@
import * as path from "node:path";
import { applyEdits } from "./apply";
import { hasBlockEdit, resolveBlockEdits } from "./block";
import { commitClipboard, forkClipboard, validateClipboardSequence } from "./clipboard";
import { computeFileHash, formatHashlineHeader } from "./format";
import type { Filesystem, WriteResult } from "./fs";
import { isNotFound } from "./fs";
@@ -42,7 +43,7 @@ import { detectLineEnding, type LineEnding, normalizeToLF, restoreLineEndings, s
import { InvalidAbsoluteRangeError } from "./parser";
import { Recovery, type RecoveryResult } from "./recovery";
import type { Snapshot, SnapshotStore } from "./snapshots";
import type { ApplyResult, BlockResolution, BlockResolver, BlockSpan, Edit, FileOp } from "./types";
import type { ApplyResult, BlockResolution, BlockResolver, BlockSpan, Clipboard, Edit, FileOp } from "./types";
/**
* Upper bound on the number of unseen anchor lines whose actual file content
@@ -81,6 +82,12 @@ export interface PatcherOptions {
* validate on content hash alone and any anchor into the tagged content applies.
*/
enforceSeenLines?: boolean;
/**
* Host-owned clipboard register shared across batches, so `CUT` content
* can be `PASTE`d by a later {@link Patcher.apply} call. Each batch works
* on a fork and publishes it back only after writes land.
*/
clipboard?: Clipboard;
}
/** Per-section result returned by {@link Patcher.apply} / {@link Patcher.commit}. */
@@ -116,9 +123,8 @@ export interface PatchSectionResult {
/** Destination path when this section includes `MV DEST`. */
moveDest?: string;
/**
* Resolved spans for any `replace_block`/`delete_block` ops, present when the
* apply matched the tagged content. Undefined for patches with no block ops
* (and for resolutions routed through drift recovery, where numbers shift).
* Resolved spans for block ops, present when the apply matched the tagged
* content. Undefined for patches with no block ops and for drift recovery.
*/
blockResolutions?: BlockResolution[];
}
@@ -158,6 +164,8 @@ function hasAnchorScopedEdit(edits: readonly Edit[]): boolean {
if (edit.kind === "delete") return true;
// A `replace_block N:` edit anchors to concrete content on line N.
if (edit.kind === "block") return true;
// A `CUT` range reads concrete content.
if (edit.kind === "cut") return true;
return edit.cursor.kind === "before_anchor" || edit.cursor.kind === "after_anchor";
});
}
@@ -211,6 +219,7 @@ export class Patcher {
readonly snapshots: SnapshotStore;
readonly recovery: Recovery;
readonly blockResolver: BlockResolver | undefined;
readonly clipboard: Clipboard | undefined;
readonly #enforceSeenLines: boolean;
constructor(options: PatcherOptions) {
@@ -221,6 +230,7 @@ export class Patcher {
this.snapshots = options.snapshots;
this.recovery = new Recovery(options.snapshots);
this.blockResolver = options.blockResolver;
this.clipboard = options.clipboard;
this.#enforceSeenLines = options.enforceSeenLines ?? true;
}
@@ -231,16 +241,34 @@ export class Patcher {
* {@link PatchSectionResult} per section in the original patch order.
*/
async apply(patch: Patch): Promise<PatcherApplyResult> {
// One register per batch: `CUT` in one section feeds `PASTE` in a later
// one, so content can move across files. A host-owned register
// (see PatcherOptions.clipboard) additionally persists across batches:
// work on a fork and publish it per landed section, so a failed batch
// never poisons the persistent register and a mid-batch failure still
// preserves content already cut from disk.
const clipboard = forkClipboard(this.clipboard);
// Single-section fast path.
if (patch.sections.length === 1) {
const prepared = await this.prepare(patch.sections[0]);
return { sections: [await this.commit(prepared)] };
const prepared = await this.prepare(patch.sections[0], clipboard);
const result = await this.commit(prepared);
if (this.clipboard !== undefined) commitClipboard(clipboard, this.clipboard);
return { sections: [result] };
}
// Prepare every section first so any failure (stale hash, missing
// file, parse error, in-memory no-op) surfaces before any write.
const prepared: PreparedSection[] = [];
for (const section of patch.sections) prepared.push(await this.prepare(section));
// Register state after each section's prepare. Commits are non-atomic:
// when a later write fails, the sections before it are already on disk,
// so the host register must reflect exactly the landed prefix — content
// a landed CUT deleted would otherwise be lost.
const sectionStates: Clipboard[] = [];
for (const section of patch.sections) {
prepared.push(await this.prepare(section, clipboard));
sectionStates.push(forkClipboard(clipboard));
}
assertUniqueCanonicalPaths(prepared);
for (const entry of prepared) {
if (entry.isNoop) {
@@ -266,6 +294,7 @@ export class Patcher {
{ cause: error },
);
}
if (this.clipboard !== undefined) commitClipboard(sectionStates[index], this.clipboard);
}
return { sections: results };
}
@@ -275,8 +304,10 @@ export class Patcher {
* No writes hit the filesystem. Use for CI checks and dry runs.
*/
async preflight(patch: Patch): Promise<void> {
// Dry run: fork the register and never publish it back.
const clipboard = forkClipboard(this.clipboard);
const prepared: PreparedSection[] = [];
for (const section of patch.sections) prepared.push(await this.prepare(section));
for (const section of patch.sections) prepared.push(await this.prepare(section, clipboard));
assertUniqueCanonicalPaths(prepared);
for (const entry of prepared) {
if (entry.isNoop) {
@@ -310,10 +341,13 @@ export class Patcher {
* {@link PreparedSection} which can be fed to {@link commit} to land
* the result on the filesystem.
*
* `clipboard` is the register shared by `CUT`/`PASTE` ops. Pass the batch
* register when preparing several sections so content can move across files.
*
* Throws on parse error, missing-file-for-anchored-edit, or unrecovered
* tag mismatch ({@link MismatchError}).
*/
async prepare(section: PatchSection): Promise<PreparedSection> {
async prepare(section: PatchSection, clipboard?: Clipboard): Promise<PreparedSection> {
const parsed = await this.#parseWithRangeDiagnostics(section);
const parseWarnings = [...parsed.warnings];
const fileOp = parsed.fileOp;
@@ -360,6 +394,7 @@ export class Patcher {
const lineEnding = detectLineEnding(text);
const normalized = normalizeToLF(text);
const register = clipboard ?? {};
const applyResult =
fileOp?.kind === "rem"
? this.#applyWithRecovery({
@@ -368,6 +403,7 @@ export class Patcher {
exists: read.exists,
normalized,
edits: [],
clipboard: register,
})
: this.#applyWithRecovery({
section: target,
@@ -375,6 +411,7 @@ export class Patcher {
exists: read.exists,
normalized,
edits: parsed.edits,
clipboard: register,
});
return new PreparedSection(
@@ -639,8 +676,9 @@ export class Patcher {
exists: boolean;
normalized: string;
edits: readonly Edit[];
clipboard: Clipboard;
}): ApplyResult {
const { section, canonicalPath, exists, normalized, edits } = args;
const { section, canonicalPath, exists, normalized, edits, clipboard } = args;
const expected = exists ? section.fileHash : undefined;
// The 4-hex tag is content-derived: when the live text hashes to it,
// trust the match and apply directly. `storedSnapshotForTag` feeds the
@@ -673,6 +711,11 @@ export class Patcher {
onWarning: warning => resolveWarnings.push(warning),
});
}
// Surface clipboard sequencing mistakes (`PASTE` before any capture,
// capturing over un-pasted `CUT` content) with their targeted message
// before the recovery path below, which swallows apply failures and
// re-surfaces them as tag-mismatch errors.
validateClipboardSequence(resolved, clipboard);
const withResolveWarnings = (result: ApplyResult): ApplyResult =>
resolveWarnings.length === 0
? result
@@ -689,7 +732,7 @@ export class Patcher {
if (expected !== undefined && this.#enforceSeenLines) {
this.#assertSeenLines(section, expected, matchedSnapshot);
}
const result = applyEdits(normalized, resolved);
const result = applyEdits(normalized, resolved, { clipboard });
return withResolveWarnings(blockResolutions.length > 0 ? { ...result, blockResolutions } : result);
}
// Head/tail-only inserts are position-stable: "start"/"end" cannot move
@@ -697,7 +740,7 @@ export class Patcher {
// content and warn instead of hard-failing — unlike an anchored
// mismatch, which cannot be safely relocated and must reject.
if (!hasAnchorScopedEdit(resolved)) {
const result = applyEdits(normalized, resolved);
const result = applyEdits(normalized, resolved, { clipboard });
return withResolveWarnings({ ...result, warnings: [HEADTAIL_DRIFT_WARNING, ...(result.warnings ?? [])] });
}
// File drifted: map every anchor from the tagged snapshot to unchanged
@@ -707,6 +750,7 @@ export class Patcher {
currentText: normalized,
fileHash: expected,
edits: resolved,
clipboard,
});
if (recovered) return withResolveWarnings(recoveryToApplyResult(recovered));
const hashRecognized = this.snapshots.byHash(canonicalPath, expected) !== null;
packages/hashline/src/prompt.md
+51 -89
View File
@@ -1,49 +1,43 @@
Your patch language names lines to replace, delete, or insert at, then lists the new content. Rule of thumb: a header ending in `:` is followed by `+` body rows; `DEL` has no body.
Line-anchored patch language: name original lines to replace, cut, or insert at, then list new content. A header ending in `:` takes `+` body rows; `CUT`, `PASTE`, `REM`, `MV` take none.
<headers>
Every file section starts with `[PATH#TAG]`. `TAG` = 4-hex snapshot tag from your latest `read`/`search`, REQUIRED on every section — no hashless form. Create new files with `write`; hashline only edits existing files.
Every file section starts `[PATH#TAG]`. `TAG` = 4-hex snapshot tag from your latest `read`/`search` — REQUIRED on every section. Create new files with `write`; hashline only edits existing files.
</headers>
<ops>
`SWAP N.=M:` — replace original lines N.=M with the body rows below. INCLUSIVE — line M is consumed too.
`SWAP.BLK N:` — replace the whole syntactic block that BEGINS on line N; tree-sitter resolves the closing line. Body rows below.
`DEL N.=M` — delete original lines N.=M. No body.
`DEL.BLK N` — delete the whole syntactic block that BEGINS on line N.
`INS.PRE N:` — insert the body rows immediately before line N.
`INS.POST N:` — insert the body rows immediately after line N.
`INS.BLK.POST N:` — insert the body rows after the END of the block that BEGINS on line N — outside it, at sibling depth. To append inside a block, use `INS.POST`.
`INS.HEAD:` / `INS.TAIL:` — insert the body rows at the very start / end of the file.
`REM` — delete the whole file named by the section header. No body, no line ops.
`MV DEST` — move/rename the section file to `DEST` (a path, quoted when it contains spaces). Line edits above `MV` land on the source first, then the final content is written at `DEST`.
Single line: `SWAP N.=N:` / `DEL N`. The range is the ORIGINAL lines you touch; body length is irrelevant (replacing 1 line with 10 is still `SWAP N.=N:`).
`SWAP N.=M:` — replace original lines N.=M (INCLUSIVE).
`SWAP.BLK N:` — replace the whole syntactic block BEGINNING on line N; its closing line is resolved for you.
`CUT N.=M` / `CUT.BLK N` — delete lines N.=M / the block beginning at N, and capture them for `PASTE`.
`INS.PRE N:` / `INS.POST N:` — insert immediately before / after line N.
`INS.BLK.POST N:` — insert after the END of the block beginning at N, outside it at sibling depth. Append inside a block → `INS.POST`.
`INS.HEAD:` / `INS.TAIL:` — insert at the very start / end of the file.
`PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` / `PASTE.TAIL` / `PASTE.BLK.POST N` — insert the clipboard at the position (the clipboard IS the body).
`REM` — delete the whole section file. `MV DEST` — move/rename to `DEST` (quote paths with spaces); edits above `MV` land on the source first, final content written at `DEST`.
Single line: `SWAP N.=N:` / `CUT N`. Range = ORIGINAL lines touched; body length irrelevant (1 line → 10 is still `SWAP N.=N:`).
</ops>
<body-rows>
Body rows appear only under a `:` header. Every body row is `+TEXT` — add a literal line `TEXT`, verbatim (leading whitespace kept); `+` alone adds a blank line. No other row kind. NEVER write `-old` or a bare/context line. To keep a line, leave it out of every range. Literal lines starting with `-`/`+` still need the body prefix: Markdown `- item` → `+- item`, `+ item` → `++ item`.
Only under a `:` header. Every row is `+TEXT`, verbatim (leading whitespace kept); `+` alone = blank line. NEVER `-old` or bare/context rows — the range deletes; the body is only the final content. Keep a line: leave it out of every range. Literal leading `-`/`+` keeps the prefix: `- item` → `+- item`, `+ item` → `++ item`.
</body-rows>
<rules>
- Line numbers + `[PATH#TAG]` header come from your latest `read`/`search` (`LINE:TEXT` rows).
- Numbers refer to the ORIGINAL file; never shift as hunks apply.
- They die with the call: every applied edit mints a fresh `#TAG` and renumbers — anchor the next edit on the edit response or a fresh `read`.
- Touch only lines your latest `read`/`search` literally displayed as `LINE:TEXT`; the tag certifies the snapshot, not your memory. A hunk anchored on a line you never displayed is REJECTED — re-`read` first. Seeing a line ≠ it holds the code you mean; confirm numbers map to the construct you intend, especially far from your read window.
- Elided regions are UNSEEN: `…`/`..` markers and a collapsed `N-M:` summary row (only boundary lines N and M shown) hide their interior. NEVER place or span a hunk inside one — `read` the range first.
- Never start or end a range mid-expression or mid-block.
- Indent body rows exactly for the depth they should live at.
- On a stale-tag rejection or any surprising result: STOP and re-`read` before further edits.
- One hunk per range; body = final content, never an old/new pair.
- Ranges cover ONLY lines whose content changes. Never widen over unchanged lines — a stale wide range shreds everything it spans.
- Whole construct → `SWAP.BLK N` (tree-sitter resolves the end); lines inside it → `SWAP N.=M`.
- `SWAP.BLK N` resolves EXACTLY the node at N. Leading decorators/attributes/doc-comments are separate nodes: point N at the FIRST decorator to sweep both; standalone line-comments are never swept — use `SWAP N.=M`.
- Block ops (`SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST`) anchor the OPENING line of a MULTI-LINE construct — never its closer, last line, or a bare inner statement. Anchoring one statement resolves to ONE line and is REJECTED: use the plain op (`SWAP N.=N` / `DEL N` / `INS.POST N`), or point N at the real opener. Saw the closer? Use plain `INS.POST M:`.
- Markdown: a heading line IS a block opener — `SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST` on a `##`/`###` heading resolves its WHOLE section (heading through every nested deeper heading, up to the next same-or-higher heading). So `DEL.BLK` drops the section, `SWAP.BLK` rewrites it, `INS.BLK.POST` lands after it (end the inserted body with a blank line to keep the next heading separated).
- Non-adjacent changes = separate hunks; untouched lines stay out of every range.
- Pure additions use `INS.PRE` / `INS.POST` / `INS.HEAD` / `INS.TAIL`, never a widened `SWAP` — retyped keepers are exactly what gets dropped. (A multi-line `SWAP` whose body restates the line just past the range is auto-dropped as an off-by-one keeper with a warning — issue the payload for the range only; never lean on the repair.)
- NEVER format/restyle code with this tool; run the project formatter instead.
- Line numbers + `#TAG` come from your latest `read`/`search` (`LINE:TEXT` rows); numbers name ORIGINAL lines, never shifted by applied hunks.
- Applied edits renumber the file and change the `#TAG` — take the next edit's numbers from the edit response or a fresh `read`.
- Touch only displayed lines — hunks on undisplayed lines are REJECTED. Far from your read window? Re-`read`; confirm numbers map to the intended construct.
- Elided regions are UNSEEN (`…`/`..` markers, collapsed `N-M:` summary rows) — NEVER place or span a hunk inside one; `read` the range first.
- NEVER start or end a range mid-expression or mid-block.
- Ranges cover ONLY changed lines — never widen over keepers. Non-adjacent changes = separate hunks.
- Whole construct → `SWAP.BLK N`; lines inside one → `SWAP N.=M`.
- `SWAP.BLK` resolves EXACTLY the node at N: leading decorators/attributes/doc-comments are separate nodes — point N at the FIRST decorator to sweep both; standalone line-comments are never swept (use `SWAP N.=M`).
- Block ops anchor the OPENING line of a MULTI-LINE construct — never the closer, last line, or a bare inner statement; one statement → plain op (`SWAP N.=N:` / `CUT N` / `INS.POST N:`). Saw the closer? `INS.POST M:`.
- Markdown: a heading IS a block opener — block ops on `##`/`###` resolve the WHOLE section (through deeper nested headings, up to the next same-or-higher heading). `INS.BLK.POST` after a section: end the body with a blank line to keep the next heading separated.
- Pure additions → `INS.PRE`/`INS.POST`/`INS.HEAD`/`INS.TAIL`, never a widened `SWAP`.
- Move code with `CUT`+`PASTE`, never retype. Clipboard: top-to-bottom across the whole patch (cross-file moves), persists across edit calls, latest `CUT` wins, and `PASTE` repeats freely. Pasted indentation is verbatim; re-indent via `SWAP`.
- NEVER format/restyle code with this tool; run the project formatter.
</rules>
<example>
Original (the exact shape `read` returns):
`read` output shape:
```
[greet.py#A1B2]
1:def greet(name):
@@ -52,40 +46,7 @@ Original (the exact shape `read` returns):
4:greet("world")
```
Insert a guard after line 1:
```
[greet.py#A1B2]
INS.POST 1:
+ if not name: name = "stranger"
```
Replace line 2 with two lines:
```
[greet.py#A1B2]
SWAP 2.=2:
+ greeting = "Hi"
+ msg = f"{greeting}, {name}"
```
Delete line 3:
```
[greet.py#A1B2]
DEL 3
```
Delete the whole file:
```
[greet.py#A1B2]
REM
```
Rename or move the file:
```
[greet.py#A1B2]
MV greet_v2.py
```
Move after editing:
Edit, then move:
```
[greet.py#A1B2]
SWAP 1.=3:
@@ -94,16 +55,7 @@ SWAP 1.=3:
MV lib/greet.py
```
Add a header and trailer:
```
[greet.py#A1B2]
INS.HEAD:
+# generated header
INS.TAIL:
+greet("everyone")
```
Insert Markdown bullets — the leading `+` is the body-row marker; the file receives `- task`:
Markdown bullets — the file receives `- task`:
```
[PLAN.md#A1B2]
INS.POST 2:
@@ -111,7 +63,15 @@ INS.POST 2:
+ - nested task
```
Replace the whole `greet` function block — `SWAP.BLK 1:` resolves lines 1–3 (the `def` header through `print(msg)`); line 4 is a separate statement and stays:
Move `greet` to a sibling file — clipboard flows across sections:
```
[greet.py#A1B2]
CUT.BLK 1
[other.py#3C4D]
PASTE.HEAD
```
`SWAP.BLK 1:` resolves lines 1–3 (`def` header through `print(msg)`); line 4 is a separate statement and stays:
```
[greet.py#A1B2]
SWAP.BLK 1:
@@ -119,7 +79,7 @@ SWAP.BLK 1:
+ print(f"Hello, {name}")
```
A decorator/doc-comment is a SEPARATE block — `SWAP.BLK` on the `def`/`fn` line keeps it. Point N at the decorator to take both; here line 1 is `@cache`, so anchoring on the `def` (line 2) would orphan `@cache`:
Decorator/doc-comment = SEPARATE block — point N at the decorator to take both; anchoring the `def` (line 2) would orphan `@cache`:
```
[svc.py#C3D4]
SWAP.BLK 1:
@@ -130,14 +90,14 @@ SWAP.BLK 1:
</example>
<anti-patterns>
# WRONG — empty `SWAP` to delete. RIGHT: DEL 4
# WRONG — empty `SWAP` to delete. RIGHT: CUT 4
SWAP 4.=4:
# WRONG — range describes post-edit size. RIGHT: SWAP 1.=1: (body length is irrelevant)
# WRONG — range sized to the post-edit content. RIGHT: SWAP 1.=1: (body length irrelevant)
SWAP 1.=2:
+def greet(name):
# WRONG — `-` rows / bare context lines do not exist. The range deletes; the body is only the new content.
# WRONG — `-` rows / bare context lines do not exist; the range deletes, the body is only new content.
SWAP 3.=3:
msg = "Hello, " + name
- print(msg)
@@ -146,27 +106,29 @@ SWAP 3.=3:
SWAP 3.=3:
+ return msg
# WRONG — a pure insertion done as a widened `SWAP`: you want to add one line after 2,
# but you replace 2.=4, retype the keepers, and drop one (here line 4, `greet("world")`).
# WRONG — pure insertion as a widened `SWAP`: retyped keepers get dropped (here line 4).
SWAP 2.=4:
+ msg = "Hello, " + name
+ extra = compute(name)
+ print(msg)
# RIGHT — touch nothing you keep; the new line is the whole body.
# RIGHT — touch nothing you keep.
INS.POST 2:
+ extra = compute(name)
# WRONG — `INS.BLK.POST N:` anchored on a closing delimiter / last visible line. RIGHT: plain `INS.POST M:`
# WRONG — `INS.BLK.POST` anchored on the closing delimiter / last visible line. RIGHT: plain `INS.POST M:`
INS.BLK.POST 3:
+after()
# RIGHT
INS.POST 3:
+after()
# WRONG — body rows under PASTE; the clipboard is the body. RIGHT: capture first, then a bodyless `PASTE.POST 20`.
PASTE.POST 20:
+function f() {}
</anti-patterns>
<critical>
If you remember nothing else:
1. RE-GROUND AFTER EVERY EDIT. Every apply mints a fresh `#TAG` and renumbers — take the next edit's numbers from the edit response or a fresh `read`. Stale tag or surprise? STOP, re-`read`.
2. RANGES ARE TIGHT. Cover only lines that change; a stale wide range shreds everything it spans. Whole construct → `SWAP.BLK N`.
3. THE BODY IS THE FINAL CONTENT. Every body row starts with `+`; Markdown bullets use `+- item`, not `- item`.
1. RE-GROUND AFTER EVERY EDIT — applied edits renumber the file and change the `#TAG`; take next numbers from the edit response or a fresh `read`. Stale tag or surprise? STOP, re-`read`.
2. RANGES ARE TIGHT — cover only lines that change. Whole construct → `SWAP.BLK N`.
3. BODY = FINAL CONTENT — every body row starts with `+`; Markdown bullets use `+- item`, not `- item`.
</critical>
packages/hashline/src/recovery.ts
+28 -4
View File
@@ -10,13 +10,15 @@ import { diffLineRuns } from "@oh-my-pi/pi-natives";
import { applyEdits } from "./apply";
import { RECOVERY_EXTERNAL_WARNING, RECOVERY_LINE_REMAP_WARNING, RECOVERY_SESSION_CHAIN_WARNING } from "./messages";
import type { SnapshotStore } from "./snapshots";
import type { Anchor, ApplyResult, Edit } from "./types";
import type { Anchor, ApplyResult, Clipboard, Edit } from "./types";
export interface RecoveryArgs {
path: string;
currentText: string;
fileHash: string;
edits: readonly Edit[];
/** Shared clipboard register for `cut`/`paste` edits, threaded into the replay apply. */
clipboard?: Clipboard;
}
export interface RecoveryResult {
@@ -41,6 +43,12 @@ function getEditAnchors(edit: Edit): Anchor[] {
// Recovery only ever receives already-resolved edits (no `block`); this arm
// exists for type-exhaustiveness over the full `Edit` union.
if (edit.kind === "block") return [edit.anchor];
if (edit.kind === "cut") {
// Every captured line is an anchor: changed interior content is unsafe.
const anchors: Anchor[] = [];
for (let line = edit.range.start.line; line <= edit.range.end.line; line++) anchors.push({ line });
return anchors;
}
return edit.cursor.kind === "before_anchor" || edit.cursor.kind === "after_anchor" ? [edit.cursor.anchor] : [];
}
@@ -213,6 +221,21 @@ function remapEditsToCurrent(previousText: string, currentText: string, edits: r
remapped.push({ ...edit, anchor });
continue;
}
if (edit.kind === "cut") {
// Map every captured line; an unmapped interior line means the
// content drifted and cannot be moved safely. Uniform offsets keep
// the mapped range contiguous.
const start = mapLine(edit.range.start.line);
if (start === null) return null;
let end = start;
for (let line = edit.range.start.line + 1; line <= edit.range.end.line; line++) {
const mapped = mapLine(line);
if (mapped === null) return null;
end = mapped;
}
remapped.push({ ...edit, range: { start: { line: start }, end: { line: end } } });
continue;
}
let blockStart = edit.blockStart;
if (blockStart !== undefined) {
@@ -243,12 +266,13 @@ function replayRemappedAnchorsOnCurrent(
currentText: string,
edits: readonly Edit[],
recoveryWarning: string,
clipboard: Clipboard | undefined,
): RecoveryResult | null {
const remapped = remapEditsToCurrent(previousText, currentText, edits);
if (remapped === null) return null;
let applied: ApplyResult;
try {
applied = applyEdits(currentText, remapped.edits);
applied = applyEdits(currentText, remapped.edits, clipboard === undefined ? {} : { clipboard });
} catch {
return null;
}
@@ -276,13 +300,13 @@ export class Recovery {
* caller should then surface a {@link MismatchError}.
*/
tryRecover(args: RecoveryArgs): RecoveryResult | null {
const { path, currentText, fileHash, edits } = args;
const { path, currentText, fileHash, edits, clipboard } = args;
// When retained texts collide on the 16-bit tag, use the latest one.
// Recovery still requires its anchors and context to map unambiguously.
const snapshot = this.store.byHash(path, fileHash);
if (!snapshot) return null;
const recoveryWarning =
this.store.head(path) === snapshot ? RECOVERY_EXTERNAL_WARNING : RECOVERY_SESSION_CHAIN_WARNING;
return replayRemappedAnchorsOnCurrent(snapshot.text, currentText, edits, recoveryWarning);
return replayRemappedAnchorsOnCurrent(snapshot.text, currentText, edits, recoveryWarning, clipboard);
}
}
packages/hashline/src/tokenizer.ts
+84 -38
View File
@@ -10,8 +10,8 @@
*/
import {
describeAnchorExamples,
HL_DELETE_BLOCK_KEYWORD,
HL_DELETE_KEYWORD,
HL_CUT_BLOCK_KEYWORD,
HL_CUT_KEYWORD,
HL_FILE_HASH_LENGTH,
HL_FILE_HASH_SEP,
HL_FILE_PREFIX,
@@ -24,6 +24,8 @@ import {
HL_INSERT_KEYWORD,
HL_INSERT_TAIL,
HL_MOVE_KEYWORD,
HL_PASTE_AFTER_BLOCK_KEYWORD,
HL_PASTE_KEYWORD,
HL_PAYLOAD_REPLACE,
HL_REM_KEYWORD,
HL_REPLACE_BLOCK_KEYWORD,
@@ -215,11 +217,13 @@ function scanHeaderRange(line: string, index = 0, end = trimEndIndex(line), allo
export type BlockTarget =
| { kind: "replace"; range: ParsedRange }
| { kind: "block"; anchor: Anchor }
| { kind: "delete"; range: ParsedRange }
| { kind: "delete_block"; anchor: Anchor }
| { kind: "insert_before"; anchor: Anchor }
| { kind: "insert_after"; anchor: Anchor }
| { kind: "insert_after_block"; anchor: Anchor }
| { kind: "cut"; range: ParsedRange }
| { kind: "cut_block"; anchor: Anchor }
| { kind: "paste"; cursor: Cursor }
| { kind: "paste_after_block"; anchor: Anchor }
| { kind: "rem" }
| { kind: "move"; dest: string }
| { kind: "bof" }
@@ -270,30 +274,52 @@ function consumeReplaceColon(line: string, index: number, end: number): number {
return skipWhitespace(line, afterEquals + 1, end);
}
function scanInsertTarget(line: string, index: number, end: number): TargetScan | null {
interface PositionScan {
cursor: Cursor;
nextIndex: number;
}
/** Scan the `.PRE N` / `.POST N` / `.HEAD` / `.TAIL` positional suffix shared by `INS` and `PASTE`. */
function scanPositionalSuffix(line: string, index: number, end: number): PositionScan | null {
if (index >= end || line.charCodeAt(index) !== CHAR_DOT) return null;
const cursor = skipWhitespace(line, index + 1, end);
const beforeEnd = scanKeyword(line, cursor, end, HL_INSERT_BEFORE);
const probe = skipWhitespace(line, index + 1, end);
const beforeEnd = scanKeyword(line, probe, end, HL_INSERT_BEFORE);
if (beforeEnd !== null) {
const anchor = scanLineNumber(line, skipWhitespace(line, beforeEnd, end), end);
if (anchor === null) return null;
const nextIndex = consumeOptionalColon(line, anchor.nextIndex, end);
return { target: { kind: "insert_before", anchor: { line: anchor.line } }, nextIndex };
return { cursor: { kind: "before_anchor", anchor: { line: anchor.line } }, nextIndex };
}
const afterEnd = scanKeyword(line, cursor, end, HL_INSERT_AFTER);
const afterEnd = scanKeyword(line, probe, end, HL_INSERT_AFTER);
if (afterEnd !== null) {
const anchor = scanLineNumber(line, skipWhitespace(line, afterEnd, end), end);
if (anchor === null) return null;
const nextIndex = consumeOptionalColon(line, anchor.nextIndex, end);
return { target: { kind: "insert_after", anchor: { line: anchor.line } }, nextIndex };
return { cursor: { kind: "after_anchor", anchor: { line: anchor.line } }, nextIndex };
}
const headEnd = scanKeyword(line, cursor, end, HL_INSERT_HEAD);
if (headEnd !== null) return { target: { kind: "bof" }, nextIndex: consumeOptionalColon(line, headEnd, end) };
const tailEnd = scanKeyword(line, cursor, end, HL_INSERT_TAIL);
if (tailEnd !== null) return { target: { kind: "eof" }, nextIndex: consumeOptionalColon(line, tailEnd, end) };
const headEnd = scanKeyword(line, probe, end, HL_INSERT_HEAD);
if (headEnd !== null) return { cursor: { kind: "bof" }, nextIndex: consumeOptionalColon(line, headEnd, end) };
const tailEnd = scanKeyword(line, probe, end, HL_INSERT_TAIL);
if (tailEnd !== null) return { cursor: { kind: "eof" }, nextIndex: consumeOptionalColon(line, tailEnd, end) };
return null;
}
function scanInsertTarget(line: string, index: number, end: number): TargetScan | null {
const scan = scanPositionalSuffix(line, index, end);
if (scan === null) return null;
const { cursor, nextIndex } = scan;
switch (cursor.kind) {
case "before_anchor":
return { target: { kind: "insert_before", anchor: cursor.anchor }, nextIndex };
case "after_anchor":
return { target: { kind: "insert_after", anchor: cursor.anchor }, nextIndex };
case "bof":
return { target: { kind: "bof" }, nextIndex };
case "eof":
return { target: { kind: "eof" }, nextIndex };
}
}
function unquotePath(pathText: string): string {
if (pathText.length < 2) return pathText;
const first = pathText[0];
@@ -361,28 +387,6 @@ function scanHunkAnchor(line: string, start: number, end: number): TargetScan |
nextIndex: consumeReplaceColon(line, range.nextIndex, end),
};
}
// `delete_block N` — resolve N to a tree-sitter block range at apply time
// and delete its whole span. Like `delete N.=M`, it takes no body and no
// trailing colon.
const deleteBlockEnd = scanKeyword(line, cursor, end, HL_DELETE_BLOCK_KEYWORD);
if (deleteBlockEnd !== null) {
const anchor = scanLineNumber(line, skipWhitespace(line, deleteBlockEnd, end), end);
if (anchor === null) return null;
let next = skipWhitespace(line, anchor.nextIndex, end);
next = skipStrayDot(line, next, end);
if (next < end && line.charCodeAt(next) === CHAR_COLON) return null;
return { target: { kind: "delete_block", anchor: { line: anchor.line } }, nextIndex: next };
}
// `delete N.=M` — like `delete_block N`, takes no body and no trailing
// colon; a colon here falls through to contamination detection.
const deleteEnd = scanKeyword(line, cursor, end, HL_DELETE_KEYWORD);
if (deleteEnd !== null) {
const range = scanHeaderRange(line, deleteEnd, end, true);
if (range === null) return null;
const next = skipStrayDot(line, range.nextIndex, end);
if (next < end && line.charCodeAt(next) === CHAR_COLON) return null;
return { target: { kind: "delete", range: range.range }, nextIndex: next };
}
// `insert_after_block N:` — insert after the last line of the tree-sitter
// block at N.
const insertAfterBlockEnd = scanKeyword(line, cursor, end, HL_INSERT_AFTER_BLOCK_KEYWORD);
@@ -394,6 +398,47 @@ function scanHunkAnchor(line: string, start: number, end: number): TargetScan |
nextIndex: consumeOptionalColon(line, anchor.nextIndex, end),
};
}
// `PASTE.BLK.POST N` — insert the clipboard after the tree-sitter block
// at N. Like all clipboard ops, takes no body rows.
const pasteAfterBlockEnd = scanKeyword(line, cursor, end, HL_PASTE_AFTER_BLOCK_KEYWORD);
if (pasteAfterBlockEnd !== null) {
const anchor = scanLineNumber(line, skipWhitespace(line, pasteAfterBlockEnd, end), end);
if (anchor === null) return null;
return {
target: { kind: "paste_after_block", anchor: { line: anchor.line } },
nextIndex: consumeOptionalColon(line, anchor.nextIndex, end),
};
}
// `PASTE.PRE|POST N` / `PASTE.HEAD|TAIL` — insert the clipboard at the position.
const pasteEnd = scanKeyword(line, cursor, end, HL_PASTE_KEYWORD);
if (pasteEnd !== null) {
const scan = scanPositionalSuffix(line, pasteEnd, end);
if (scan === null) return null;
return { target: { kind: "paste", cursor: scan.cursor }, nextIndex: scan.nextIndex };
}
// `CUT.BLK N` captures and deletes the tree-sitter block beginning at N.
// Scan it before the plain form so `.BLK` is not parsed as a range.
const cutBlockEnd = scanKeyword(line, cursor, end, HL_CUT_BLOCK_KEYWORD);
if (cutBlockEnd !== null) {
const anchor = scanLineNumber(line, skipWhitespace(line, cutBlockEnd, end), end);
if (anchor === null) return null;
return {
target: { kind: "cut_block", anchor: { line: anchor.line } },
nextIndex: consumeOptionalColon(line, anchor.nextIndex, end),
};
}
// `CUT N.=M` captures and deletes concrete lines. A trailing colon is
// tolerated and ignored; body rows are rejected by the parser.
const cutEnd = scanKeyword(line, cursor, end, HL_CUT_KEYWORD);
if (cutEnd !== null) {
const range = scanHeaderRange(line, cutEnd, end, true);
if (range === null) return null;
const next = skipStrayDot(line, range.nextIndex, end);
return {
target: { kind: "cut", range: range.range },
nextIndex: consumeOptionalColon(line, next, end),
};
}
const insertEnd = scanKeyword(line, cursor, end, HL_INSERT_KEYWORD);
if (insertEnd !== null) return scanInsertTarget(line, insertEnd, end);
return null;
@@ -488,10 +533,11 @@ function classifyLine(line: string, lineNum: number): Token {
const lead = skipWhitespace(line, 0);
const isHunkLead =
line.startsWith(HL_REPLACE_KEYWORD, lead) ||
line.startsWith(HL_DELETE_KEYWORD, lead) ||
line.startsWith(HL_INSERT_KEYWORD, lead) ||
line.startsWith(HL_REM_KEYWORD, lead) ||
line.startsWith(HL_MOVE_KEYWORD, lead);
line.startsWith(HL_MOVE_KEYWORD, lead) ||
line.startsWith(HL_CUT_KEYWORD, lead) ||
line.startsWith(HL_PASTE_KEYWORD, lead);
if (isHunkLead) {
const hunk = tryParseHunkHeader(line);
if (hunk !== null) return { kind: "op-block", lineNum, target: hunk.target };
packages/hashline/src/types.ts
+60 -30
View File
@@ -42,21 +42,46 @@ export type Edit =
| { kind: "delete"; anchor: Anchor; lineNum: number; index: number; oldAssertion?: string }
| {
/**
* Deferred block edit (`replace_block N:` / `delete_block N` /
* `insert_after_block N:`). The exact line span is unknown at parse
* time — it is computed by {@link resolveBlockEdits} once file text +
* path (→ language) are available, then expanded into concrete edits:
* a non-empty `payloads` without `mode` (from `replace_block`) becomes
* the same `replacement` inserts + deletes that `replace start.=end:`
* produces; an empty `payloads` (from `delete_block`) becomes a pure
* range deletion; `mode: "insert_after"` becomes plain `after_anchor`
* inserts at the block's last line. `applyEdits` never sees this
* variant.
* Clipboard cut (`CUT N.=M`, or the resolved form of `CUT.BLK N`).
* Captures the range's current lines into the {@link Clipboard}
* register during the applier's clipboard pre-pass and lowers to one
* `delete` per range line at parse/resolve time.
*/
kind: "cut";
range: ParsedRange;
lineNum: number;
index: number;
}
| {
/**
* Clipboard insertion (`PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` /
* `PASTE.TAIL`, or the resolved form of `PASTE.BLK.POST N`). Expanded
* by the clipboard pre-pass into one plain insert per captured line.
* `blockStart` mirrors the insert variant's field for block-lowered
* pastes so landing correction can slide the body across trailing
* closer lines.
*/
kind: "paste";
cursor: Cursor;
lineNum: number;
index: number;
blockStart?: number;
}
| {
/**
* Deferred block edit (`SWAP.BLK N:`, `INS.BLK.POST N:`,
* `CUT.BLK N`, or `PASTE.BLK.POST N`). The exact line span is
* unknown at parse time; {@link resolveBlockEdits} computes it once
* file text and language are available, then expands it into
* concrete edits. `mode: "insert_after"` becomes plain
* `after_anchor` inserts, `"cut"` becomes a clipboard cut plus
* per-line deletes, and `"paste_after"` becomes a paste after the
* resolved block. No mode denotes a block replacement.
*/
kind: "block";
anchor: Anchor;
payloads: string[];
mode?: "insert_after";
mode?: "insert_after" | "cut" | "paste_after";
lineNum: number;
index: number;
};
@@ -73,10 +98,9 @@ export interface ApplyResult {
/** Diagnostic warnings collected by the parser, patcher, or recovery. */
warnings?: string[];
/**
* Resolved spans for each `replace_block`/`delete_block` op in this apply,
* in patch order. Present only when the apply matched the tagged content
* (the common no-drift path), so the line numbers line up with what the
* caller read. Absent when there were no block ops.
* Resolved spans for each block op in this apply, in patch order. Present
* only when the apply matched the tagged content, so the line numbers match
* what the caller read. Absent when there were no block ops.
*/
blockResolutions?: BlockResolution[];
}
@@ -124,9 +148,7 @@ export interface CompactDiffOptions {
maxUnchangedRun?: number;
}
/**
* Resolved 1-indexed inclusive line span of a `replace_block N:` target.
*/
/** Resolved 1-indexed inclusive line span of a block target. */
export interface BlockSpan {
/** First line of the block (1-indexed, inclusive). */
start: number;
@@ -135,11 +157,9 @@ export interface BlockSpan {
}
/**
* One `replace_block N:` / `delete_block N` / `insert_after_block N:` anchor
* resolved to its concrete line span. Surfaced on {@link ApplyResult} so the
* host can echo "block N → lines start.=end" and let the model catch a wrong
* opener — e.g. a decorator or doc-comment that sits in a separate node
* outside the resolved block.
* One block-op anchor resolved to its concrete line span. Surfaced on
* {@link ApplyResult} so the host can echo the resolved range and let the
* model catch an anchor on the wrong opener.
*/
export interface BlockResolution {
/** The 1-indexed line the block op was anchored on (the `N`). */
@@ -149,10 +169,10 @@ export interface BlockResolution {
/** Last line of the resolved span (1-indexed, inclusive). */
end: number;
/** Which block op produced this resolution. */
op: "replace" | "delete" | "insert_after";
op: "replace" | "insert_after" | "cut" | "paste_after";
}
/** Request handed to a {@link BlockResolver} to resolve one `replace_block N:` anchor. */
/** Request handed to a {@link BlockResolver} to resolve one block-op anchor. */
export interface BlockResolverRequest {
/** Target file path (used to infer language by extension). */
path: string;
@@ -163,10 +183,20 @@ export interface BlockResolverRequest {
}
/**
* Resolves a `replace_block N:` anchor to the line span of the syntactic block
* that begins on line N. Returns `null` when no block can be resolved
* (unrecognized language, blank/out-of-range line, no node begins there, or the
* resolved subtree has a syntax error). Pure seam: the hashline core declares
* the contract; the host injects a tree-sitter-backed implementation.
* Resolves a block-op anchor to the line span of the syntactic block beginning
* on that line. Returns `null` for an unsupported language, invalid line, absent
* opener, or syntax error. The host injects the tree-sitter-backed implementation.
*/
export type BlockResolver = (request: BlockResolverRequest) => BlockSpan | null;
/**
* Mutable clipboard register threaded through one patch application. Filled
* by `CUT` edits and read by `PASTE` edits in patch source order, across
* sections, so content can move between files. Create one per batch (`{}`)
* and hand it to every {@link Patcher.prepare} / `applyTo` call in that batch;
* callers that omit it get a private per-call register.
*/
export interface Clipboard {
/** Lines captured by the most recent `CUT`, or unset. */
lines?: readonly string[];
}
packages/hashline/test/block.test.ts
+18 -18
View File
@@ -140,19 +140,19 @@ describe("resolveBlockEdits", () => {
expect(error?.message).not.toContain("\n\n");
});
it("fires onResolved with the resolved span for replace and delete blocks", () => {
it("fires onResolved with the resolved span for replace and cut blocks", () => {
const seen: BlockResolution[] = [];
// stubResolver maps line N → span [N, N+1].
resolveBlockEdits(parsePatch("SWAP.BLK 2:\n+A\n+B").edits, "ignored", PATH, stubResolver, {
onResolved: resolution => seen.push(resolution),
});
resolveBlockEdits(parsePatch("DEL.BLK 5").edits, "ignored", PATH, stubResolver, {
resolveBlockEdits(parsePatch("CUT.BLK 5").edits, "ignored", PATH, stubResolver, {
onResolved: resolution => seen.push(resolution),
});
expect(seen).toEqual([
{ anchorLine: 2, start: 2, end: 3, op: "replace" },
{ anchorLine: 5, start: 5, end: 6, op: "delete" },
{ anchorLine: 5, start: 5, end: 6, op: "cut" },
]);
});
@@ -305,11 +305,11 @@ describe("Patcher with a block resolver", () => {
});
});
describe("DEL.BLK", () => {
describe("CUT.BLK", () => {
const text = "function x() {\n if (y) {\n }\n}\n";
it("parses `DEL.BLK N` into a block edit with no payloads", () => {
const { edits } = parsePatch("DEL.BLK 2");
it("parses `CUT.BLK N` into a cut block edit", () => {
const { edits } = parsePatch("CUT.BLK 2");
expect(edits).toHaveLength(1);
const edit = edits[0];
@@ -317,38 +317,38 @@ describe("DEL.BLK", () => {
if (edit?.kind !== "block") throw new Error("expected a block edit");
expect(edit.anchor.line).toBe(2);
expect(edit.payloads).toEqual([]);
expect(edit.mode).toBe("cut");
});
it("rejects body rows under `DEL.BLK N`", () => {
expect(() => parsePatch("DEL.BLK 2\n+X")).toThrow("`DEL.BLK N` does not take body rows");
it("rejects body rows under `CUT.BLK N`", () => {
expect(() => parsePatch("CUT.BLK 2\n+X")).toThrow("`CUT N.=M` captures + deletes lines");
});
it("resolveBlockEdits expands a delete-block edit into pure deletes", () => {
const edits = parsePatch("DEL.BLK 2").edits;
const resolved = resolveBlockEdits(edits, "ignored", PATH, stubResolver);
it("resolveBlockEdits expands a cut block into capture and deletes", () => {
const resolved = resolveBlockEdits(parsePatch("CUT.BLK 2").edits, "ignored", PATH, stubResolver);
expect(resolved.every(edit => edit.kind === "delete")).toBe(true);
expect(resolved.map(edit => (edit.kind === "delete" ? edit.anchor.line : -1))).toEqual([2, 3]);
expect(resolved.map(edit => edit.kind)).toEqual(["cut", "delete", "delete"]);
expect(resolved.flatMap(edit => (edit.kind === "delete" ? [edit.anchor.line] : []))).toEqual([2, 3]);
});
it("applyTo deletes the resolved block span", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nDEL.BLK 2`);
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT.BLK 2`);
// stub span [2,3] → drop " if (y) {" and " }".
expect(section.applyTo(text, stubResolver).text).toBe("function x() {\n}\n");
});
it("applyPartialTo drops an unresolvable delete-block edit instead of throwing", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nDEL.BLK 2`);
it("applyPartialTo drops an unresolvable cut-block edit", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT.BLK 2`);
expect(section.applyPartialTo(text).text).toBe(text);
});
it("Patcher applies a delete-block edit on the hash-match path", async () => {
it("Patcher applies a cut-block edit on the hash-match path", async () => {
const fs = new InMemoryFilesystem([[PATH, text]]);
const snapshots = new InMemorySnapshotStore();
const tag = snapshots.record(PATH, text);
const patcher = new Patcher({ fs, snapshots, blockResolver: stubResolver });
const result = await patcher.apply(Patch.parse(`[${PATH}#${tag}]\nDEL.BLK 2`));
const result = await patcher.apply(Patch.parse(`[${PATH}#${tag}]\nCUT.BLK 2`));
expect(result.sections[0]?.op).toBe("update");
expect(fs.get(PATH)).toBe("function x() {\n}\n");
packages/hashline/test/boundary-repair.test.ts
+4 -4
View File
@@ -400,11 +400,11 @@ describe("boundary-balance repair", () => {
expect(warnings.filter(warning => /structural closing line/.test(warning))).toHaveLength(1);
});
// #3142: the range's deleted `}` is matched by an opener another hunk deletes
// (`DEL 1`). The patch nets to balanced, so the closer must stay deleted —
// (`CUT 1`). The patch nets to balanced, so the closer must stay deleted —
// the per-group repair wrongly kept it, leaving a stray `}`.
it("does not keep a deleted closer when another hunk removes its opener (#3142)", () => {
const file = ["if enabled {", '\tText("Old")', "}", '\tText("Tail")'].join("\n");
const diff = ["DEL 1", "SWAP 2.=3:", '+Text("New")'].join("\n");
const diff = ["CUT 1", "SWAP 2.=3:", '+Text("New")'].join("\n");
const { text, warnings } = apply(file, diff);
expect(text).toBe(['Text("New")', '\tText("Tail")'].join("\n"));
expect(warnings.filter(warning => /structural closing line/.test(warning))).toHaveLength(0);
@@ -414,7 +414,7 @@ describe("boundary-balance repair", () => {
// residual must be spent on the genuine hunk, not the wrapper-removed one.
it("spends the missing-closer residual on the genuine hunk, not an earlier wrapper removal", () => {
const file = ["if enabled {", '\tText("Old")', "}", "const config = {", "\ta: 1,", "};"].join("\n");
const diff = ["DEL 1", "SWAP 2.=3:", '+Text("New")', "SWAP 6.=6:", "+\tb: 2,"].join("\n");
const diff = ["CUT 1", "SWAP 2.=3:", '+Text("New")', "SWAP 6.=6:", "+\tb: 2,"].join("\n");
const { text, warnings } = apply(file, diff);
expect(text).toBe(['Text("New")', "const config = {", "\ta: 1,", "\tb: 2,", "};"].join("\n"));
expect(warnings.filter(warning => /structural closing line/.test(warning))).toHaveLength(1);
@@ -470,7 +470,7 @@ describe("boundary-balance repair", () => {
it("ignores non-contiguously deleted openers when choosing which closer to keep", () => {
const file = ["if (a) {", "\told();", "\tmore();", "}", "const obj = {", "\ta: 1,", "};"].join("\n");
const diff = ["DEL 1", "SWAP 3.=4:", "+\tnew();", "SWAP 7.=7:", "+\tb: 2,"].join("\n");
const diff = ["CUT 1", "SWAP 3.=4:", "+\tnew();", "SWAP 7.=7:", "+\tb: 2,"].join("\n");
const { text, warnings } = apply(file, diff);
expect(text).toBe(["\told();", "\tnew();", "const obj = {", "\ta: 1,", "\tb: 2,", "};"].join("\n"));
expect(warnings.filter(warning => /structural closing line/.test(warning))).toHaveLength(1);
packages/hashline/test/clipboard.test.ts
+288
View File
@@ -0,0 +1,288 @@
import { describe, expect, it } from "bun:test";
import {
applyEdits,
type BlockResolver,
type BlockSpan,
type Clipboard,
computeFileHash,
InMemoryFilesystem,
InMemorySnapshotStore,
Patch,
Patcher,
parsePatch,
parsePatchStreaming,
resolveBlockEdits,
} from "@oh-my-pi/hashline";
const PATH = "x.ts";
// Deterministic stub: the block beginning on line N spans [N, N+1].
const stubResolver: BlockResolver = ({ line }): BlockSpan => ({ start: line, end: line + 1 });
function taggedPatcher(files: Array<[string, string]>): {
fs: InMemoryFilesystem;
snapshots: InMemorySnapshotStore;
patcher: Patcher;
tags: Map<string, string>;
} {
const fs = new InMemoryFilesystem(files);
const snapshots = new InMemorySnapshotStore();
const tags = new Map<string, string>();
for (const [path, text] of files) tags.set(path, snapshots.record(path, text));
return { fs, snapshots, patcher: new Patcher({ fs, snapshots }), tags };
}
describe("clipboard parsing", () => {
it("lowers `CUT N.=M` to a capture plus per-line deletes", () => {
const cut = parsePatch("CUT 2.=3").edits;
expect(cut.map(edit => edit.kind)).toEqual(["cut", "delete", "delete"]);
expect(cut[0]).toMatchObject({ kind: "cut", range: { start: { line: 2 }, end: { line: 3 } } });
});
it("parses every PASTE position, tolerating a trailing colon", () => {
const cursors = parsePatch("CUT 1\nPASTE.PRE 2\nPASTE.POST 3:\nPASTE.HEAD\nPASTE.TAIL:").edits.flatMap(edit =>
edit.kind === "paste" ? [edit.cursor] : [],
);
expect(cursors).toEqual([
{ kind: "before_anchor", anchor: { line: 2 } },
{ kind: "after_anchor", anchor: { line: 3 } },
{ kind: "bof" },
{ kind: "eof" },
]);
});
it("rejects bare `PASTE` without a position", () => {
expect(() => parsePatch("PASTE")).toThrow(/`PASTE` needs a position/);
});
it("rejects body rows under clipboard ops", () => {
expect(() => parsePatch("CUT 1.=2\n+x")).toThrow(/`CUT N.=M` captures \+ deletes/);
expect(() => parsePatch("CUT 1\nPASTE.POST 1\n+x")).toThrow(/`PASTE` inserts the clipboard content/);
});
it("rejects a CUT range overlapping another hunk's range", () => {
expect(() => parsePatch("CUT 2.=4\nSWAP 3.=3:\n+x")).toThrow(/already targeted by another hunk/);
});
it("reports inverted CUT ranges with op-specific retry forms", () => {
expect(() => parsePatch("CUT 5.=2")).toThrow(/`CUT 5`.*`CUT 5.=6`/);
});
it("flushes a trailing bodyless clipboard op in streaming mode", () => {
const { edits } = parsePatchStreaming("CUT 1\nPASTE.TAIL");
expect(edits.map(edit => edit.kind)).toEqual(["cut", "delete", "paste"]);
});
});
describe("clipboard apply semantics", () => {
it("moves a range within a file (CUT + PASTE)", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 2.=3\nPASTE.POST 5`);
expect(section.applyTo("l1\nl2\nl3\nl4\nl5\n").text).toBe("l1\nl4\nl5\nl2\nl3\n");
});
it("repeats CUT content without consuming the clipboard", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 2.=2\nPASTE.HEAD\nPASTE.TAIL`);
expect(section.applyTo("l1\nl2\nl3\n").text).toBe("l2\nl1\nl3\nl2\n");
});
it("swaps two regions with sequential CUT/PASTE pairs on original coordinates", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 1.=2\nPASTE.POST 4\nCUT 3.=4\nPASTE.PRE 1`);
expect(section.applyTo("a1\na2\nb1\nb2").text).toBe("b1\nb2\na1\na2");
});
it("rejects PASTE with an empty register", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nPASTE.POST 1`);
expect(() => section.applyTo("l1\nl2\n")).toThrow(/found nothing in the clipboard/);
});
it("drops an empty-register PASTE on the streaming-tolerant path", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nPASTE.POST 1`);
expect(section.applyPartialTo("l1\nl2\n").text).toBe("l1\nl2\n");
});
it("allows a CUT without a following PASTE", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 2.=2`);
expect(section.applyTo("l1\nl2\nl3\n").text).toBe("l1\nl3\n");
});
it("allows consecutive CUTs and pastes the latest capture", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 1.=1\nCUT 3.=3\nPASTE.TAIL`);
expect(section.applyTo("l1\nl2\nl3\n").text).toBe("l2\nl3\n");
});
it("rejects an out-of-range capture", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT 8.=9\nPASTE.HEAD`);
expect(() => section.applyTo("l1\nl2\n")).toThrow(/out of range \(file has 3 lines\)/);
});
it("threads a caller-owned register across applyEdits calls", () => {
const clipboard: Clipboard = {};
const cut = parsePatch("CUT 1.=1").edits;
const paste = parsePatch("PASTE.TAIL").edits;
expect(applyEdits("a\nb", cut, { clipboard }).text).toBe("b");
expect(applyEdits("x\ny", paste, { clipboard }).text).toBe("x\ny\na");
});
});
describe("clipboard block ops", () => {
it("expands CUT.BLK to a span capture plus per-line deletes", () => {
const edits = parsePatch("CUT.BLK 2\nPASTE.TAIL").edits;
const resolved = resolveBlockEdits(edits, "l1\nl2\nl3\nl4", PATH, stubResolver);
expect(resolved.map(edit => edit.kind)).toEqual(["cut", "delete", "delete", "paste"]);
expect(resolved[0]).toMatchObject({ kind: "cut", range: { start: { line: 2 }, end: { line: 3 } } });
});
it("moves a block after another block via PASTE.BLK.POST", () => {
const section = Patch.parseSingle(`[${PATH}#1A2B]\nCUT.BLK 1\nPASTE.BLK.POST 3`);
// stub blocks: [1,2] and [3,4].
expect(section.applyTo("a1\na2\nb1\nb2\nrest", stubResolver).text).toBe("b1\nb2\na1\na2\nrest");
});
it("echoes clipboard block resolutions with their op", () => {
const seen: string[] = [];
resolveBlockEdits(parsePatch("CUT.BLK 2\nPASTE.TAIL").edits, "l1\nl2\nl3", PATH, stubResolver, {
onResolved: resolution => seen.push(resolution.op),
});
expect(seen).toEqual(["cut"]);
});
it("rejects a single-line CUT.BLK resolution with the plain-op retry", () => {
const single: BlockResolver = ({ line }): BlockSpan => ({ start: line, end: line });
const edits = parsePatch("CUT.BLK 2\nPASTE.TAIL").edits;
expect(() => resolveBlockEdits(edits, "a\nb\nc", PATH, single)).toThrow(/use `CUT 2`/);
});
it("lowers an unresolvable PASTE.BLK.POST to a plain paste with a warning", () => {
const warnings: string[] = [];
const resolved = resolveBlockEdits(parsePatch("CUT 1\nPASTE.BLK.POST 2").edits, "a\nb\nc", PATH, () => null, {
onWarning: warning => warnings.push(warning),
});
expect(resolved.map(edit => edit.kind)).toEqual(["cut", "delete", "paste"]);
expect(warnings.some(warning => warning.includes("`PASTE.BLK.POST 2`"))).toBe(true);
});
});
describe("clipboard across sections and batches", () => {
it("moves lines between files within one patch", async () => {
const a = "keep\nmove1\nmove2\n";
const b = "b1\n";
const { fs, patcher, tags } = taggedPatcher([
["a.ts", a],
["b.ts", b],
]);
const patch = Patch.parse(`[a.ts#${tags.get("a.ts")}]\nCUT 2.=3\n[b.ts#${tags.get("b.ts")}]\nPASTE.TAIL`);
await patcher.apply(patch);
expect(fs.get("a.ts")).toBe("keep\n");
expect(fs.get("b.ts")).toBe("b1\nmove1\nmove2\n");
});
it("applies a batch-local CUT without requiring PASTE", async () => {
const a = "l1\nl2\n";
const { fs, patcher, tags } = taggedPatcher([["a.ts", a]]);
await patcher.apply(Patch.parse(`[a.ts#${tags.get("a.ts")}]\nCUT 1.=1`));
expect(fs.get("a.ts")).toBe("l2\n");
});
it("persists a host-owned register across apply calls", async () => {
const a = "l1\nl2\n";
const b = "b1\n";
const fs = new InMemoryFilesystem([
["a.ts", a],
["b.ts", b],
]);
const snapshots = new InMemorySnapshotStore();
const tagA = snapshots.record("a.ts", a);
const tagB = snapshots.record("b.ts", b);
const clipboard: Clipboard = {};
const patcher = new Patcher({ fs, snapshots, clipboard });
const first = await patcher.apply(Patch.parse(`[a.ts#${tagA}]\nCUT 1.=1`));
expect(fs.get("a.ts")).toBe("l2\n");
expect(first.sections[0]?.warnings).toEqual([]);
expect(clipboard).toEqual({ lines: ["l1"] });
await patcher.apply(Patch.parse(`[b.ts#${tagB}]\nPASTE.TAIL`));
expect(fs.get("b.ts")).toBe("b1\nl1\n");
expect(clipboard).toEqual({ lines: ["l1"] });
});
it("does not publish register changes from a failed batch", async () => {
const a = "l1\nl2\n";
const b = "b1\n";
const fs = new InMemoryFilesystem([
["a.ts", a],
["b.ts", b],
]);
const snapshots = new InMemorySnapshotStore();
const tagA = snapshots.record("a.ts", a);
snapshots.record("b.ts", b);
const clipboard: Clipboard = {};
const patcher = new Patcher({ fs, snapshots, clipboard });
// Second section carries a bogus tag, so the batch fails in prepare.
const patch = Patch.parse(`[a.ts#${tagA}]\nCUT 1.=1\n[b.ts#0000]\nSWAP 1.=1:\n+x`);
await expect(patcher.apply(patch)).rejects.toThrow();
expect(fs.get("a.ts")).toBe(a);
expect(clipboard.lines).toBeUndefined();
});
it("recovers clipboard anchors when the file drifted by a uniform offset", async () => {
const original = "x1\nx2\nx3\n";
const live = "new\nx1\nx2\nx3\n";
const fs = new InMemoryFilesystem([["a.ts", live]]);
const snapshots = new InMemorySnapshotStore();
const tag = snapshots.record("a.ts", original);
const patcher = new Patcher({ fs, snapshots });
const result = await patcher.apply(Patch.parse(`[a.ts#${tag}]\nCUT 2.=2\nPASTE.TAIL`));
expect(fs.get("a.ts")).toBe("new\nx1\nx3\nx2\n");
expect(result.sections[0]?.warnings.length).toBeGreaterThan(0);
});
it("rejects clipboard ops in same-path sections interleaved across another file", () => {
const patch = Patch.parse(`[a.ts#1A2B]\nCUT 1.=1\n[b.ts#3C4D]\nSWAP 1.=1:\n+x\n[a.ts#1A2B]\nPASTE.TAIL`);
expect(() => patch.sections[0]?.edits).toThrow(/interleaved with another file/);
});
it("still merges interleaved same-path sections without clipboard ops", () => {
const patch = Patch.parse(
`[a.ts#1A2B]\nSWAP 1.=1:\n+a1\n[b.ts#3C4D]\nSWAP 1.=1:\n+b1\n[a.ts#1A2B]\nSWAP 3.=3:\n+a3`,
);
expect(patch.sections).toHaveLength(2);
expect(patch.sections[0]?.edits.map(edit => edit.kind)).toEqual(["insert", "delete", "insert", "delete"]);
});
it("surfaces a targeted sequencing error instead of a mismatch on the drift path", async () => {
const original = "x1\nx2\nx3\n";
const live = "new\nx1\nx2\nx3\n";
const fs = new InMemoryFilesystem([["a.ts", live]]);
const snapshots = new InMemorySnapshotStore();
const tag = snapshots.record("a.ts", original);
const patcher = new Patcher({ fs, snapshots });
// PASTE with nothing captured, against a drifted tag: the recovery path
// swallows apply errors, so the sequencing check must fire first.
await expect(patcher.apply(Patch.parse(`[a.ts#${tag}]\nPASTE.POST 2\nCUT 2.=2`))).rejects.toThrow(
/found nothing in the clipboard/,
);
});
});
describe("clipboard header hash interplay", () => {
it("applies a paste-only section on the clean-tag fast path", async () => {
const text = "l1\nl2\n";
const fs = new InMemoryFilesystem([["a.ts", text]]);
const snapshots = new InMemorySnapshotStore();
expect(computeFileHash(text)).toBe(snapshots.record("a.ts", text));
const clipboard: Clipboard = { lines: ["from-before"] };
const patcher = new Patcher({ fs, snapshots, clipboard });
await patcher.apply(Patch.parse(`[a.ts#${computeFileHash(text)}]\nPASTE.POST 1`));
expect(fs.get("a.ts")).toBe("l1\nfrom-before\nl2\n");
});
});
packages/hashline/test/core-contracts.test.ts
+7 -7
View File
@@ -93,7 +93,7 @@ describe("hashline parser — range-anchor contracts", () => {
expect(section.applyTo("aaa\nbbb\nccc").text).toBe("aaa\nbbb\ntail\nccc");
});
it("applies replace/delete/insert operations against concrete anchors", () => {
it("applies replace/cut/insert operations against concrete anchors", () => {
const diff = [
`INS.PRE ${tag(2)}:`,
repl("before b"),
@@ -105,8 +105,8 @@ describe("hashline parser — range-anchor contracts", () => {
repl("tail"),
].join("\n");
expect(applyDiff(content, diff)).toBe("top\naaa\nbefore b\nbbb\nafter b\nccc\ntail");
expect(applyDiff(content, `DEL ${tag(2)}`)).toBe("aaa\nccc");
expect(applyDiff(content, `DEL ${tag(2)}..${tag(3)}`)).toBe("aaa");
expect(applyDiff(content, `CUT ${tag(2)}`)).toBe("aaa\nccc");
expect(applyDiff(content, `CUT ${tag(2)}..${tag(3)}`)).toBe("aaa");
expect(applyDiff(content, `${sameLineRange(tag(2))}\n${repl("BBB")}`)).toBe("aaa\nBBB\nccc");
});
@@ -283,10 +283,10 @@ describe("hashline abort sentinel", () => {
});
});
describe("hashline parser — delete and blank payload semantics", () => {
it("applies inline delete and empty replace operations", () => {
expect(applyDiff("line1\nline2\nline3\n", splitHashlineInput("[a.ts]\nDEL 2\n").diff)).toBe("line1\nline3\n");
expect(applyDiff("line1\nline2\nline3\nline4\n", splitHashlineInput("[a.ts]\nDEL 2.=3\n").diff)).toBe(
describe("hashline parser — cut and blank payload semantics", () => {
it("applies inline cut and empty replace operations", () => {
expect(applyDiff("line1\nline2\nline3\n", splitHashlineInput("[a.ts]\nCUT 2\n").diff)).toBe("line1\nline3\n");
expect(applyDiff("line1\nline2\nline3\nline4\n", splitHashlineInput("[a.ts]\nCUT 2.=3\n").diff)).toBe(
"line1\nline4\n",
);
expect(applyDiff("line1\nline2\nline3\n", splitHashlineInput("[a.ts]\nSWAP 2.=2:\n").diff)).toBe(
packages/hashline/test/format-v2.test.ts
+17 -11
View File
@@ -15,17 +15,17 @@ describe("hashline format v4", () => {
it("deletes a single source line", () => {
const text = "a\nb\nc";
expect(applyPatch(text, "DEL 2")).toBe("a\nc");
expect(applyPatch(text, "CUT 2")).toBe("a\nc");
});
it("deletes a concrete range", () => {
const text = "a\nb\nc\nd";
expect(applyPatch(text, "DEL 2.=3")).toBe("a\nd");
expect(applyPatch(text, "CUT 2.=3")).toBe("a\nd");
});
it("accepts a single dot between integer range endpoints", () => {
const text = "a\nb\nc\nd";
expect(applyPatch(text, "DEL 2.3")).toBe("a\nd");
expect(applyPatch(text, "CUT 2.3")).toBe("a\nd");
expect(applyPatch(text, "SWAP 2.3:\n+middle")).toBe("a\nmiddle\nd");
});
@@ -47,8 +47,14 @@ describe("hashline format v4", () => {
expect(() => parsePatch("INS.HEAD:")).toThrow(/needs at least one/);
});
it("rejects body rows under delete", () => {
expect(() => parsePatch("DEL 2\n+replacement")).toThrow(/does not take body rows/);
it("rejects body rows under cut", () => {
expect(() => parsePatch("CUT 2\n+replacement")).toThrow(/takes no body rows/);
});
it("does not recognize removed DEL or COPY headers", () => {
for (const header of ["DEL 2", "DEL.BLK 2", "COPY 2", "COPY.BLK 2"]) {
expect(() => parsePatch(header)).toThrow(/payload line has no preceding hunk header/);
}
});
it("auto-pipes bare body rows as literal text", () => {
@@ -85,14 +91,14 @@ describe("hashline format v4", () => {
);
});
it("ignores deleting the trailing blank sentinel of a newline-terminated file", () => {
it("ignores cutting the trailing blank sentinel of a newline-terminated file", () => {
// "a\nb\n" splits into ["a", "b", ""]; line 3 is the phantom sentinel.
const edits = parsePatch("DEL 3").edits;
const edits = parsePatch("CUT 3").edits;
expect(applyEdits("a\nb\n", edits).text).toBe("a\nb\n");
});
it("treats a delete range ending at the trailing sentinel as ending at the last real line", () => {
const edits = parsePatch("DEL 2.=3").edits;
it("treats a cut range ending at the trailing sentinel as ending at the last real line", () => {
const edits = parsePatch("CUT 2.=3").edits;
expect(applyEdits("a\nb\n", edits).text).toBe("a\n");
});
@@ -106,9 +112,9 @@ describe("hashline format v4", () => {
expect(applyEdits("a\nb\n", edits).text).toBe("a\nb\n\ntail");
});
it("still deletes a genuine empty last line of a non-newline-terminated file", () => {
it("still cuts a genuine last line of a non-newline-terminated file", () => {
// "a\nb" has no sentinel; line 2 is real content.
const edits = parsePatch("DEL 2").edits;
const edits = parsePatch("CUT 2").edits;
expect(applyEdits("a\nb", edits).text).toBe("a");
});
packages/hashline/test/landing-shift.test.ts
+2 -2
View File
@@ -85,8 +85,8 @@ describe("after-insert landing shift", () => {
});
it("refuses to cross a line targeted by another hunk", () => {
const { text, warnings } = apply(FILE, "INS.POST 3:\n+ c();\nDEL 4");
// The closer on line 4 is owned by the delete; the insert stays put.
const { text, warnings } = apply(FILE, "INS.POST 3:\n+ c();\nCUT 4");
// The closer on line 4 is owned by the cut; the insert stays put.
expect(text).toBe(["function f() {", " if (x) {", " a();", " c();", " b();", "}", ""].join("\n"));
expect(warnings).toHaveLength(0);
});
packages/hashline/test/leniency.test.ts
+9 -13
View File
@@ -48,7 +48,7 @@ describe("hashline section headers", () => {
it("reports bracket syntax with a 4-hex example when the header is missing", () => {
try {
Patch.parse("DEL 38.=40");
Patch.parse("CUT 38.=40");
throw new Error("expected missing-header error");
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
@@ -68,18 +68,18 @@ describe("hashline core — verb header forms", () => {
expect(() => parsePatch("2 3\n+X")).toThrow(/Hunk headers need a verb/);
});
it("accepts canonical replace/delete/insert forms", () => {
it("accepts canonical replace/cut/insert forms", () => {
expect(applyPatch(FILE, "SWAP 2.=3:\n+X")).toBe("a\nX\nd\ne");
expect(applyPatch(FILE, "DEL 2.=3")).toBe("a\nd\ne");
expect(applyPatch(FILE, "CUT 2.=3")).toBe("a\nd\ne");
expect(applyPatch(FILE, "INS.PRE 2:\n+X")).toBe("a\nX\nb\nc\nd\ne");
expect(applyPatch(FILE, "INS.POST 2:\n+X")).toBe("a\nb\nX\nc\nd\ne");
expect(applyPatch(FILE, "INS.HEAD:\n+X")).toBe("X\na\nb\nc\nd\ne");
expect(applyPatch(FILE, "INS.TAIL:\n+X")).toBe("a\nb\nc\nd\ne\nX");
});
it("accepts single-number replace and delete shorthand", () => {
it("accepts single-number replace and cut shorthand", () => {
expect(applyPatch(FILE, "SWAP 2:\n+X")).toBe("a\nX\nc\nd\ne");
expect(applyPatch(FILE, "DEL 2")).toBe("a\nc\nd\ne");
expect(applyPatch(FILE, "CUT 2")).toBe("a\nc\nd\ne");
});
it("accepts alternate replace range separators and missing colon", () => {
@@ -103,8 +103,8 @@ describe("hashline core — verb header forms", () => {
// `INS.POST 2.:` instead of `INS.POST 2:`
expect(applyPatch(FILE, "INS.POST 2.:\n+X")).toBe("a\nb\nX\nc\nd\ne");
expect(applyPatch(FILE, "INS.PRE 2.:\n+X")).toBe("a\nX\nb\nc\nd\ne");
// `DEL 2.=3.` instead of `DEL 2.=3` (stray dot, no colon)
expect(applyPatch(FILE, "DEL 2.=3.")).toBe("a\nd\ne");
// `CUT 2.=3.` instead of `CUT 2.=3`
expect(applyPatch(FILE, "CUT 2.=3.")).toBe("a\nd\ne");
// `INS.HEAD.:` and `INS.TAIL.:` with stray dot
expect(applyPatch(FILE, "INS.HEAD.:\n+X")).toBe("X\na\nb\nc\nd\ne");
expect(applyPatch(FILE, "INS.TAIL.:\n+X")).toBe("a\nb\nc\nd\ne\nX");
@@ -202,12 +202,8 @@ describe("hashline body contracts", () => {
expect(() => parsePatch("INS.TAIL:")).toThrow(/`INS` needs/);
});
it("rejects delete with a body", () => {
expect(() => parsePatch("DEL 2\n+X")).toThrow(/does not take body rows/);
});
it("rejects delete with a colon", () => {
expect(() => parsePatch("DEL 2:\n+X")).toThrow(/has no colon/);
it("rejects cut with a body", () => {
expect(() => parsePatch("CUT 2\n+X")).toThrow(/takes no body rows/);
});
});
packages/hashline/test/patcher.test.ts
+1 -1
View File
@@ -337,7 +337,7 @@ describe("Patcher seen-line provenance", () => {
const patcher = new Patcher({ fs, snapshots });
// Anchor 60 unseen lines — over the 40-line inline reveal cap.
const dels = Array.from({ length: 60 }, (_, i) => `DEL ${100 + i}`).join("\n");
const dels = Array.from({ length: 60 }, (_, i) => `CUT ${100 + i}`).join("\n");
let message: string | undefined;
try {
await patcher.apply(Patch.parse(`[${PATH}#${tag}]\n${dels}`));
packages/metaharness/adapters/edit/cli.ts
Regular → Executable
+309 -43
View File
@@ -1,97 +1,363 @@
#!/usr/bin/env bun
/** Manager-owned executable adapter for the TypeScript edit benchmark. */
/**
* Executable adapter for the TypeScript edit benchmark.
*
* Two audiences share this entry point:
* - The metaharness manager spawns it headless (`--model … --output result.json`)
* and consumes only the continuously-rewritten report file.
* - Humans run it directly and get the interactive experience: a config
* banner, a live progress bar with per-run failure diffs (stderr, TTY-aware),
* a runtime-stats summary, and a markdown or JSON report.
*/
import * as fs from "node:fs/promises";
import * as path from "node:path";
import { parseArgs } from "node:util";
import { TempDir } from "@oh-my-pi/pi-utils";
import { loadTasksFromDir } from "@oh-my-pi/typescript-edit-benchmark/tasks";
import { generateJsonReport } from "./report";
import { type BenchmarkConfig, runBenchmark } from "./runner";
import { type ResolvedThinkingLevel, ThinkingLevel } from "@oh-my-pi/pi-agent-core";
import { THINKING_EFFORTS } from "@oh-my-pi/pi-ai";
import { postmortem, TempDir } from "@oh-my-pi/pi-utils";
import { loadTasksFromDir, validateFixturesFromDir } from "@oh-my-pi/typescript-edit-benchmark/tasks";
import { LiveProgress } from "./live-progress";
import { generateJsonReport, generateReport } from "./report";
import { type BenchmarkConfig, type BenchmarkResult, buildBenchmarkResult, runBenchmark } from "./runner";
const EDIT_PACKAGE = path.resolve(import.meta.dir, "..", "..", "..", "typescript-edit-benchmark");
const RUNS_DIR = path.resolve(import.meta.dir, "..", "..", "..", "..", "runs");
async function extractFixtures(): Promise<{ dir: string; temp: TempDir }> {
type ReportFormat = "markdown" | "json";
function log(line = ""): void {
process.stderr.write(`${line}\n`);
}
function fail(message: string): never {
throw new Error(message);
}
function parseThinkingLevel(value: string): ResolvedThinkingLevel {
if ([ThinkingLevel.Off, ...THINKING_EFFORTS].includes(value as ResolvedThinkingLevel)) {
return value as ResolvedThinkingLevel;
}
return fail(`Invalid thinking level: ${value}. Valid levels: ${[ThinkingLevel.Off, ...THINKING_EFFORTS].join(", ")}`);
}
function parsePositiveInt(raw: string | undefined, flag: string, fallback: number): number {
if (raw === undefined) return fallback;
const parsed = Number.parseInt(raw, 10);
if (Number.isNaN(parsed) || parsed < 1) fail(`Invalid ${flag} value: ${raw}. Must be a positive integer.`);
return parsed;
}
function parseFuzzy(raw: string | undefined): boolean | "auto" | undefined {
if (raw === undefined) return undefined;
if (raw === "auto") return "auto";
if (raw === "true" || raw === "1") return true;
if (raw === "false" || raw === "0") return false;
return fail(`Invalid --edit-fuzzy: ${raw}. Must be true, false, 1, 0, or auto.`);
}
function parseFuzzyThreshold(raw: string | undefined): number | "auto" | undefined {
if (raw === undefined) return undefined;
if (raw === "auto") return "auto";
const parsed = Number.parseFloat(raw);
if (Number.isNaN(parsed) || parsed < 0 || parsed > 1) {
fail(`Invalid --edit-fuzzy-threshold: ${raw}. Must be 0-1 or auto.`);
}
return parsed;
}
function resolveFormat(explicit: string | undefined, outputPath: string): ReportFormat {
if (explicit === "markdown" || explicit === "json") return explicit;
if (explicit !== undefined) fail(`Invalid --format: ${explicit}. Must be markdown or json.`);
return outputPath.endsWith(".md") || outputPath.endsWith(".markdown") ? "markdown" : "json";
}
function generateReportFilename(config: BenchmarkConfig): string {
const modelName = config.model
.split("/")
.pop()!
.replace(/[^a-zA-Z0-9-]/g, "_");
const variant = config.editVariant ?? "auto";
const timestamp = new Date().toISOString().replace(/:/g, "-").replace(/\..+$/, "Z");
return path.join(RUNS_DIR, `${modelName}_${variant}_${timestamp}.md`);
}
/** Sibling `.dump` directory for an output path, timestamped on collision. */
async function resolveConversationDumpDir(outputPath: string): Promise<string> {
const parsed = path.parse(outputPath);
const preferred = path.join(parsed.dir, `${parsed.name}.dump`);
try {
await fs.stat(preferred);
} catch {
return preferred;
}
const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
return path.join(parsed.dir, `${parsed.name}.${timestamp}.dump`);
}
interface ResolvedFixtures {
dir: string;
cleanup?: () => Promise<void>;
}
/** Resolve `--fixtures` (directory or tarball; default: the built-in tarball). */
async function resolveFixtures(fixturesArg: string | undefined): Promise<ResolvedFixtures> {
const source = fixturesArg ?? path.join(EDIT_PACKAGE, "fixtures.tar.gz");
if (!source.endsWith(".tar.gz") && !source.endsWith(".tgz")) {
return { dir: source };
}
const temp = await TempDir.create("@metaharness-edit-fixtures-");
const archive = new Bun.Archive(await Bun.file(path.join(EDIT_PACKAGE, "fixtures.tar.gz")).arrayBuffer());
const archive = new Bun.Archive(await Bun.file(source).arrayBuffer());
for (const [filePath, file] of await archive.files()) {
await Bun.write(path.join(temp.path(), filePath), file);
}
const entries = await fs.readdir(temp.path(), { withFileTypes: true });
const directories = entries.filter(entry => entry.isDirectory());
const files = entries.filter(entry => entry.isFile());
return { dir: directories.length === 1 && files.length === 0 ? path.join(temp.path(), directories[0]!.name) : temp.path(), temp };
const dir = directories.length === 1 && files.length === 0 ? path.join(temp.path(), directories[0]!.name) : temp.path();
return { dir, cleanup: () => temp.remove() };
}
/** Execute an edit benchmark and continuously materialize its normalized source artifact. */
function printUsage(): void {
log(`
Edit Benchmark - Evaluate patch application success rates
Usage:
bun adapters/edit/cli.ts --model <provider/model> [options]
Options:
--model <id> Provider/model ID, e.g. anthropic/claude-sonnet-4 (required)
--provider <id> Override provider (auto-detected from model prefix)
--thinking <level> Thinking level: off, minimal, low, medium, high, xhigh, max
--runs <n> Runs per task (default: 1)
--timeout <ms> Timeout per run in ms (default: 120000)
--connection-timeout <ms> Timeout for first event before fast-retry (default: 30000)
--max-turns <n> Max turns per attempt before failing (default: 30)
--task-concurrency <n> Max tasks to run in parallel (default: 32)
--tasks <ids> Comma-separated task IDs to run (default: sampled)
--max-tasks <n> Max tasks to sample evenly (default: 80, 0 = all)
--fixtures <path> Fixtures directory or .tar.gz archive (default: built-in)
--edit-variant <v> Edit variant, e.g. hashline, replace, apply_patch (default: auto)
--edit-fuzzy <bool> Fuzzy matching: true, false, auto
--edit-fuzzy-threshold <n> Fuzzy threshold 0-1 or auto
--guided Include an authoritative suggested edit payload
--max-attempts <n> Max prompt attempts per run (default: 1)
--no-early-stop-on-match Don't short-circuit when output matches expected
--output <file> Report file (default: runs/<model>_<variant>_<ts>.md)
--format <fmt> Report format: markdown, json (default: by extension)
--check-fixtures Validate fixtures and exit
--quiet Suppress the live progress view
--list Print task ids as JSON and exit
--help Show this help message
Examples:
# Full run against the default sample of 80 tasks
bun adapters/edit/cli.ts --model anthropic/claude-sonnet-4
# Compare edit variants
bun adapters/edit/cli.ts --model openai/gpt-5 --edit-variant hashline --output hashline.md
bun adapters/edit/cli.ts --model openai/gpt-5 --edit-variant apply_patch --output apply_patch.md
# Specific tasks, more runs
bun adapters/edit/cli.ts --model anthropic/claude-sonnet-4 --tasks logic-flip-strict-equality-001 --runs 5
`);
}
/** Execute an edit benchmark and continuously materialize its report artifact. */
export async function main(argv = process.argv.slice(2)): Promise<void> {
const { values } = parseArgs({
args: argv,
options: {
model: { type: "string" },
output: { type: "string" },
"max-tasks": { type: "string", default: "80" },
tasks: { type: "string" },
"task-concurrency": { type: "string", default: "32" },
provider: { type: "string" },
thinking: { type: "string" },
runs: { type: "string", default: "1" },
timeout: { type: "string", default: "120000" },
"connection-timeout": { type: "string", default: "30000" },
"max-turns": { type: "string", default: "30" },
"task-concurrency": { type: "string", default: "32" },
tasks: { type: "string" },
"max-tasks": { type: "string", default: "80" },
fixtures: { type: "string" },
"edit-variant": { type: "string" },
"edit-fuzzy": { type: "string" },
"edit-fuzzy-threshold": { type: "string" },
guided: { type: "boolean", default: false },
"max-attempts": { type: "string", default: "1" },
"no-op-retry-limit": { type: "string", default: "2" },
"max-timeout-retries": { type: "string", default: "3" },
"max-provider-retries": { type: "string", default: "3" },
"mutation-scope-window": { type: "string", default: "20" },
"no-early-stop-on-match": { type: "boolean", default: false },
output: { type: "string" },
format: { type: "string" },
"check-fixtures": { type: "boolean", default: false },
quiet: { type: "boolean", default: false },
list: { type: "boolean", default: false },
help: { type: "boolean", default: false },
},
strict: true,
});
const fixtures = await extractFixtures();
if (values.help) {
printUsage();
return;
}
const fixtures = await resolveFixtures(values.fixtures);
try {
if (values["check-fixtures"]) {
const issues = await validateFixturesFromDir(fixtures.dir);
if (issues.length === 0) {
log("Fixtures OK");
return;
}
log("Fixture validation failed:");
for (const issue of issues) log(` - ${issue.taskId}: ${issue.message}`);
process.exitCode = 1;
return;
}
let tasks = await loadTasksFromDir(fixtures.dir);
if (values.list) {
process.stdout.write(`${JSON.stringify(tasks.map(task => ({ id: task.id, name: task.name })))}\n`);
return;
}
if (!values.model || !values.output) throw new Error("edit adapter requires --model and --output");
if (!values.model) fail("edit adapter requires --model (see --help)");
if (values.tasks) {
const selected = new Set(values.tasks.split(",").map(value => value.trim()));
tasks = tasks.filter(task => selected.has(task.id));
if (tasks.length !== selected.size) throw new Error("one or more edit task ids were not found");
if (tasks.length !== selected.size) fail("one or more edit task ids were not found (see --list)");
} else {
const limit = Number(values["max-tasks"]);
if (limit > 0 && tasks.length > limit) {
// Deterministic even sampling across the id-sorted list keeps
// mutation-category coverage representative.
const sorted = tasks.slice().sort((a, b) => a.id.localeCompare(b.id));
const step = sorted.length / limit;
tasks = Array.from({ length: limit }, (_, index) => sorted[Math.floor(index * step)]!);
}
}
const slash = values.model.indexOf("/");
const model = values.model;
const slash = model.indexOf("/");
const config: BenchmarkConfig = {
provider: slash === -1 ? "anthropic" : values.model.slice(0, slash),
model: values.model,
runsPerTask: Number(values.runs),
timeout: 120_000,
connectionTimeout: 30_000,
maxTurns: 30,
taskConcurrency: Number(values["task-concurrency"]),
guided: false,
maxAttempts: 1,
noOpRetryLimit: 2,
maxTimeoutRetries: 3,
maxProviderFailureRetries: 3,
mutationScopeWindow: 20,
conversationDumpDir: path.join(path.dirname(values.output), "result.dump"),
provider: values.provider ?? (slash === -1 ? "anthropic" : model.slice(0, slash)),
model,
...(values.thinking === undefined ? {} : { thinkingLevel: parseThinkingLevel(values.thinking) }),
runsPerTask: parsePositiveInt(values.runs, "--runs", 1),
timeout: parsePositiveInt(values.timeout, "--timeout", 120_000),
connectionTimeout: parsePositiveInt(values["connection-timeout"], "--connection-timeout", 30_000),
maxTurns: parsePositiveInt(values["max-turns"], "--max-turns", 30),
taskConcurrency: parsePositiveInt(values["task-concurrency"], "--task-concurrency", 32),
guided: values.guided,
maxAttempts: parsePositiveInt(values["max-attempts"], "--max-attempts", 1),
noOpRetryLimit: parsePositiveInt(values["no-op-retry-limit"], "--no-op-retry-limit", 2),
maxTimeoutRetries: parsePositiveInt(values["max-timeout-retries"], "--max-timeout-retries", 3),
maxProviderFailureRetries: parsePositiveInt(values["max-provider-retries"], "--max-provider-retries", 3),
mutationScopeWindow: parsePositiveInt(values["mutation-scope-window"], "--mutation-scope-window", 20),
inProcess: true,
earlyStopOnMatch: true,
earlyStopOnMatch: !values["no-early-stop-on-match"],
};
let writes = Promise.resolve();
const result = await runBenchmark(tasks, config, undefined, snapshot => {
writes = writes.then(async () => {
await Bun.write(values.output!, generateJsonReport(snapshot));
});
const editVariant = values["edit-variant"];
if (editVariant) config.editVariant = editVariant;
const editFuzzy = parseFuzzy(values["edit-fuzzy"]);
if (editFuzzy !== undefined) config.editFuzzy = editFuzzy;
const editFuzzyThreshold = parseFuzzyThreshold(values["edit-fuzzy-threshold"]);
if (editFuzzyThreshold !== undefined) config.editFuzzyThreshold = editFuzzyThreshold;
let outputPath = values.output;
if (outputPath === undefined) {
await fs.mkdir(RUNS_DIR, { recursive: true });
outputPath = generateReportFilename(config);
}
const format = resolveFormat(values.format, outputPath);
config.conversationDumpDir = await resolveConversationDumpDir(outputPath);
log("Edit Benchmark");
log("==============");
log(`Provider: ${config.provider}`);
log(`Model: ${config.model}`);
if (config.thinkingLevel) log(`Thinking: ${config.thinkingLevel}`);
log(`Runs per task: ${config.runsPerTask}`);
log(`Timeout: ${config.timeout}ms`);
log(`Task concurrency: ${config.taskConcurrency}`);
log(`Guided mode: ${config.guided ? "enabled" : "disabled"}`);
if (config.editVariant) log(`Edit variant: ${config.editVariant}`);
if (config.editFuzzy !== undefined) log(`Edit fuzzy: ${config.editFuzzy}`);
if (config.editFuzzyThreshold !== undefined) log(`Edit fuzzy threshold: ${config.editFuzzyThreshold}`);
log(`Tasks: ${tasks.length}`);
log(`Report: ${outputPath}`);
log(`Conversation dumps: ${config.conversationDumpDir}`);
log();
const renderReport = (result: BenchmarkResult): string =>
format === "json" ? generateJsonReport(result) : generateReport(result);
const progress = values.quiet ? undefined : new LiveProgress(tasks.length * config.runsPerTask, config.runsPerTask);
let latestResult = buildBenchmarkResult({
tasks,
config,
resultsByTask: new Map(),
startTime: new Date().toISOString(),
});
let writes = Promise.resolve();
const queueReportWrite = (result: BenchmarkResult) => {
writes = writes.then(async () => {
await Bun.write(outputPath, renderReport(result));
});
};
// A killed/interrupted run still leaves a usable partial report behind.
const unregisterCleanup = postmortem.register("edit-benchmark-report", async reason => {
if (reason === postmortem.Reason.EXIT) return;
progress?.finish();
log("Benchmark interrupted; writing partial report...");
await Bun.write(outputPath, renderReport(latestResult));
await fixtures.cleanup?.();
});
const result = await runBenchmark(
tasks,
config,
event => progress?.handleEvent(event),
snapshot => {
latestResult = snapshot;
queueReportWrite(snapshot);
},
);
latestResult = result;
progress?.finish();
queueReportWrite(result);
await writes;
await Bun.write(values.output, generateJsonReport(result));
unregisterCleanup();
log();
log("Benchmark complete!");
log(
` Task success rate (best of ${config.runsPerTask}): ${(result.summary.taskSuccessRate * 100).toFixed(1)}% (${result.summary.successfulTasks}/${result.summary.totalTasks})`,
);
log(` Total tokens (best): ${result.summary.totalTokens.input} in / ${result.summary.totalTokens.output} out`);
log(
` Tokens/task (best): mean=${result.summary.avgTokensPerTask.total} median=${result.summary.medianTokensPerTask.total} p1=${result.summary.p1TokensPerTask.total} p99=${result.summary.p99TokensPerTask.total} reasoning=${result.summary.avgTokensPerTask.reasoning}`,
);
if (result.summary.ghostRuns > 0) log(` Ghost runs (0/0/0): ${result.summary.ghostRuns}`);
if (result.summary.timeoutRuns > 0) log(` Timeout runs: ${result.summary.timeoutRuns}`);
log(`Report written to: ${outputPath}`);
} finally {
await fixtures.temp.remove();
await fixtures.cleanup?.();
}
}
if (import.meta.main) {
main().catch(error => {
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
process.exitCode = 1;
});
main()
.then(async () => {
// In-process benchmark runs can leave provider keep-alive sockets and
// background AgentSession timers alive after the report is written.
// Treat the final report as the CLI boundary.
await postmortem.quit(Number(process.exitCode ?? 0));
})
.catch(async error => {
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
await postmortem.quit(1);
});
}
packages/metaharness/adapters/edit/live-progress.ts
+241
View File
@@ -0,0 +1,241 @@
/**
* Live terminal renderer for edit benchmark runs.
*
* Thin view over {@link runBenchmark}'s `onProgress` stream: a single
* in-place progress line (bar, pass rates, token/latency averages, in-flight
* count) on a TTY, plain per-run lines otherwise, plus inline colored diffs
* for failed runs and a runtime-stats block at the end. Renders to stderr so
* stdout stays free for machine output (`--list` JSON), and the report file
* remains the only artifact the manager consumes.
*/
import { percentile, type ProgressEvent } from "./runner";
const ANSI = {
reset: "\x1b[0m",
bold: "\x1b[1m",
dim: "\x1b[2m",
red: "\x1b[31m",
green: "\x1b[32m",
yellow: "\x1b[33m",
cyan: "\x1b[36m",
} as const;
function rateColor(percent: number): string {
if (percent >= 80) return ANSI.green;
if (percent >= 50) return ANSI.yellow;
return ANSI.red;
}
export class LiveProgress {
readonly #totalRuns: number;
readonly #runsPerTask: number;
readonly #isTty: boolean;
readonly #colors: boolean;
#started = 0;
#completed = 0;
#success = 0;
#totalInput = 0;
#totalOutput = 0;
#totalDuration = 0;
#totalReads = 0;
#totalEdits = 0;
#totalWrites = 0;
#totalEditSuccesses = 0;
#totalToolInputChars = 0;
#indentScores: number[] = [];
#inputTokens: number[] = [];
#outputTokens: number[] = [];
#totalTokens: number[] = [];
#oneShotSuccessTokens: number[] = [];
#lastLineLength = 0;
constructor(totalRuns: number, runsPerTask: number) {
this.#totalRuns = totalRuns;
this.#runsPerTask = runsPerTask;
this.#isTty = Boolean(process.stderr.isTTY);
this.#colors = this.#isTty && !process.env.NO_COLOR;
}
#paint(code: string, text: string): string {
return this.#colors ? `${code}${text}${ANSI.reset}` : text;
}
#log(line: string): void {
process.stderr.write(`${line}\n`);
}
handleEvent(event: ProgressEvent): void {
if (event.status === "started") {
this.#started += 1;
if (!this.#isTty) {
this.#log(` [${event.taskId}] Run ${event.runIndex + 1}/${this.#runsPerTask} started...`);
}
this.#renderLine();
return;
}
this.#completed += 1;
if (event.result) {
if (event.result.success) {
this.#success += 1;
}
if (event.result.success && event.runIndex === 0) {
this.#oneShotSuccessTokens.push(event.result.tokens.total);
}
this.#totalInput += event.result.tokens.input;
this.#totalOutput += event.result.tokens.output;
this.#inputTokens.push(event.result.tokens.input);
this.#outputTokens.push(event.result.tokens.output);
this.#totalTokens.push(event.result.tokens.total);
this.#totalDuration += event.result.duration;
this.#totalReads += event.result.toolCalls.read;
this.#totalEdits += event.result.toolCalls.edit;
this.#totalWrites += event.result.toolCalls.write;
this.#totalEditSuccesses += event.result.toolCalls.editSuccesses;
this.#totalToolInputChars += event.result.toolCalls.totalInputChars;
if (typeof event.result.indentScore === "number") {
this.#indentScores.push(event.result.indentScore);
}
}
const result = event.result;
if (result && !result.success && result.error) {
this.#flushLine();
const header = this.#paint(
ANSI.red,
`[${event.taskId}] Run ${event.runIndex + 1}/${this.#runsPerTask} failed:`,
);
this.#log(` ${header} ${result.error}`);
if (result.diff) {
const changeLines = result.diff
.split("\n")
.filter(line => /^[-+@]/.test(line) && !/^(---|\+\+\+)/.test(line));
const maxLines = 40;
for (const line of changeLines.slice(0, maxLines)) {
let color: string | undefined;
if (line.startsWith("@@")) color = ANSI.cyan;
else if (line.startsWith("-")) color = ANSI.red;
else if (line.startsWith("+")) color = ANSI.green;
this.#log(` ${color ? this.#paint(color, line) : line}`);
}
if (changeLines.length > maxLines) {
this.#log(this.#paint(ANSI.dim, ` ... (${changeLines.length - maxLines} more change lines)`));
}
}
}
if (result?.editFailures && result.editFailures.length > 0) {
this.#flushLine();
for (const [i, failure] of result.editFailures.entries()) {
const args = (failure.args ?? {}) as Record<string, unknown>;
const target =
typeof args.path === "string" ? args.path : typeof args.file === "string" ? args.file : undefined;
const op = typeof args.operation === "string" ? args.operation : undefined;
const oneLine = failure.error.replace(/\s+/g, " ").trim();
const clipped = oneLine.length > 240 ? `${oneLine.slice(0, 237)}...` : oneLine;
const tag = this.#paint(ANSI.yellow, `[${event.taskId}] schema #${i + 1}`);
const metaParts = [op, target].filter((value): value is string => Boolean(value));
const meta = metaParts.length > 0 ? this.#paint(ANSI.dim, metaParts.join(" ")) : "";
this.#log(` ${tag}${meta ? ` ${meta}` : ""} ${clipped}`);
if (failure.rawBlock) {
const rawLine = failure.rawBlock.replace(/\s+/g, " ").trim();
const clippedRaw = rawLine.length > 240 ? `${rawLine.slice(0, 237)}...` : rawLine;
this.#log(` ${this.#paint(ANSI.dim, "raw")} ${clippedRaw}`);
}
}
}
if (!this.#isTty) {
const status = event.result?.success ? "completed" : "failed";
this.#log(` [${event.taskId}] Run ${event.runIndex + 1}/${this.#runsPerTask} ${status}`);
}
this.#renderLine();
}
finish(): void {
this.#flushLine();
this.#printSummary();
}
#printSummary(): void {
const n = this.#completed;
const denom = n || 1;
const successRate = (this.#success / denom) * 100;
const editSuccessRate = this.#totalEdits > 0 ? (this.#totalEditSuccesses / this.#totalEdits) * 100 : 100;
const avgIndent =
this.#indentScores.length > 0 ? this.#indentScores.reduce((a, b) => a + b, 0) / this.#indentScores.length : 0;
this.#log("");
this.#log(this.#paint(ANSI.bold, "Runtime Stats:"));
this.#log(
` Task success: ${this.#paint(rateColor(successRate), `${successRate.toFixed(1)}% (${this.#success}/${n})`)}`,
);
this.#log(
` Edit success: ${this.#paint(rateColor(editSuccessRate), `${editSuccessRate.toFixed(1)}% (${this.#totalEditSuccesses}/${this.#totalEdits})`)}`,
);
this.#log(` Avg indent score: ${avgIndent.toFixed(2)}`);
this.#log(` Tool calls: read=${this.#totalReads} edit=${this.#totalEdits} write=${this.#totalWrites}`);
this.#log(` Tool input chars: ${this.#totalToolInputChars.toLocaleString()}`);
const fmtTokens = (samples: number[]): string => {
if (samples.length === 0) return "mean=0 median=0 p1=0 p99=0";
const sorted = [...samples].sort((a, b) => a - b);
const mean = Math.round(sorted.reduce((a, b) => a + b, 0) / sorted.length);
return `mean=${mean} median=${Math.round(percentile(sorted, 50))} p1=${Math.round(percentile(sorted, 1))} p99=${Math.round(percentile(sorted, 99))}`;
};
this.#log(` Tokens/task in: ${fmtTokens(this.#inputTokens)}`);
this.#log(` Tokens/task out: ${fmtTokens(this.#outputTokens)}`);
this.#log(` Tokens/task tot: ${fmtTokens(this.#totalTokens)}`);
this.#log(` Tokens/task (one-shot successes): ${fmtTokens(this.#oneShotSuccessTokens)}`);
this.#log(` Avg time/task: ${Math.round(this.#totalDuration / denom)}ms`);
}
#renderLine(): void {
if (!this.#isTty) {
return;
}
const successRate = this.#completed > 0 ? (this.#success / this.#completed) * 100 : 0;
const editRate = this.#totalEdits > 0 ? (this.#totalEditSuccesses / this.#totalEdits) * 100 : 100;
const avgInput = this.#completed > 0 ? Math.round(this.#totalInput / this.#completed) : 0;
const avgOutput = this.#completed > 0 ? Math.round(this.#totalOutput / this.#completed) : 0;
const avgDuration = this.#completed > 0 ? Math.round(this.#totalDuration / this.#completed) : 0;
const inFlight = this.#started - this.#completed;
const bar = this.#renderBar(this.#completed, this.#totalRuns, 20);
const progress = this.#paint(ANSI.bold, `${this.#completed}/${this.#totalRuns}`);
const taskCol = `task=${this.#paint(rateColor(successRate), `${successRate.toFixed(0)}%`)}`;
const editCol = `edit=${this.#paint(rateColor(editRate), `${editRate.toFixed(0)}%`)}`;
const tokCol = this.#paint(ANSI.dim, `tok=${avgInput}/${avgOutput}`);
const durCol = this.#paint(ANSI.dim, `${avgDuration}ms`);
const rewCol = this.#paint(ANSI.dim, `r/e/w=${this.#totalReads}/${this.#totalEdits}/${this.#totalWrites}`);
const flyCol = `fly=${this.#paint(ANSI.cyan, String(inFlight))}`;
const line = ` ${bar} ${progress} ${taskCol} ${editCol} ${tokCol} ${durCol} ${rewCol} ${flyCol}`;
this.#writeLine(line);
}
#renderBar(done: number, total: number, width: number): string {
const ratio = total === 0 ? 0 : done / total;
const filled = Math.round(ratio * width);
const empty = Math.max(0, width - filled);
const filledPart = this.#paint(ANSI.green, "#".repeat(filled));
const emptyPart = this.#paint(ANSI.dim, "-".repeat(empty));
return `[${filledPart}${emptyPart}]`;
}
#writeLine(line: string): void {
const lineWidth = Bun.stringWidth(line);
const pad = this.#lastLineLength > lineWidth ? " ".repeat(this.#lastLineLength - lineWidth) : "";
process.stderr.write(`\r${line}${pad}`);
this.#lastLineLength = lineWidth;
}
#flushLine(): void {
if (!this.#isTty) {
return;
}
if (this.#lastLineLength > 0) {
process.stderr.write(`\r${" ".repeat(this.#lastLineLength)}\r`);
this.#lastLineLength = 0;
}
}
}
packages/metaharness/adapters/edit/prompts/benchmark-system.md
+4 -5
View File
@@ -3,13 +3,12 @@ You are participating in a code-edit benchmark inside a repository with {{#if mu
This benchmark is scored on exactness. Get the edit right.
## Important constraints
- Make the minimum change necessary. Do not refactor, improve, or clean up other code.
- If you see multiple similar patterns, only change the ONE that is buggy (there is only one intended mutation).
- Preserve exact code structure. Do not rearrange statements or change formatting.
- Make exactly the change the task specifies — nothing more. Do not refactor, improve, or clean up other code.
- Tasks range from single-token fixes to multi-hunk block rewrites. When the task shows replacement code, reproduce it byte-for-byte: indentation, tabs vs spaces, and blank lines included.
- If the file contains multiple similar regions, change only the one(s) the task identifies.
- Your output is verified by exact text diff against an expected fixture. Equivalent code, reordered imports, reordered object keys, or formatting changes will fail.
- Prefer copying the original line(s) and changing only the specific token(s) required. Do not rewrite whole statements.
- Never modify comments or license headers unless the task explicitly asks.
- Re-read the changed region after editing to confirm you only touched the intended line(s).
- Re-read the changed region after editing to confirm it matches the task exactly.
{{#if multiFile}}- Only modify the file(s) referenced by the task or follow-up messages. Leave all other files unchanged.
{{/if}}
## Process

Some files were not shown because too many files have changed in this diff Show More