smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b550858265e12fee3d6d4235ba638744786a008a
oh-my-pi/docs/natives-build-release-debugging.md
T
can1357 b550858265 ci: upgraded continuous integration workflows and migrated bazel dependency locking 
- Updated CI workflows and GitHub actions to enhance Bazel cache keying, credential masking, and validation checks.
- Migrated dependency locking from Cargo.Bazel.lock to MODULE.bazel.lock using rules_rust crate_universe.
- Updated build configuration, documentation, and tooling scripts to reflect the lockfile and cache changes.
2026-07-28 12:53:23 +02:00

434 lines
34 KiB
Markdown
Raw Blame History

# Natives Build, Release, and Debugging Runbook
This runbook describes how `@oh-my-pi/pi-natives` produces `.node` addons, generated declarations, and compiled-binary embedded payloads, and how to debug loader/build failures.
Addon **artifacts are built by Bazel** (`rules_rust` + `crate_universe` + hermetic cc toolchains); the cargo workspace stays authoritative for local Rust iteration (rust-analyzer, `cargo nextest`) and for napi typedef regeneration. Runtime loading and embedding are unchanged.
It follows the architecture terms from `docs/natives-architecture.md`:
- **build-time artifact production** (Bazel `//:natives-<target>` via `scripts/bazel-natives.ts`)
- **embedded addon manifest generation** (`scripts/embed-native.ts`)
- **runtime addon loading** (`native/index.js`, `native/loader-state.js`)
## Implementation files
Build side:
- `BUILD.bazel` (root) — the eight `//:natives-<target>` addon targets + aggregate filegroups
- `bazel/defs.bzl` — the `native_addon` rule/transition
- `bazel/platforms/BUILD.bazel` — one `platform()` per shipped addon
- `bazel/variants/BUILD.bazel` — `baseline`/`modern` ISA constraint values
- `bazel/toolchains/` — musl rustc disambiguation + the msvc cross cc toolchain (`msvc/NOTES.md`)
- `bazel/clippy.bazelrc` — generated from `[workspace.lints]` in `Cargo.toml`
- `MODULE.bazel`, `MODULE.bazel.lock`, `.bazelrc`, `.bazelversion` (Bazel 9.2.0)
- `scripts/bazel-natives.ts` — the canonical driver (build + locate + install)
- `crates/pi-natives/BUILD.bazel`, `crates/pi-natives/Cargo.toml`
Package side (unchanged runtime/packaging):
- `packages/natives/scripts/build-bindings.ts` — dev-only typedef regeneration
- `packages/natives/scripts/embed-native.ts`, `gen-enums.ts`, `gen-npm-packages.ts`
- `packages/natives/package.json`
- `packages/natives/native/index.js`, `native/loader-state.js`
## Build architecture
### 1) `//:natives-<target>` addon targets
Root `BUILD.bazel` instantiates one `native_addon` per shipped `(platform, arch, ISA-variant)`:
| Target | Platform | Canonical output |
| --------------------------------- | --------------------------------------- | ------------------------------------ |
| `//:natives-linux-x64-baseline` | `//bazel/platforms:linux-x64-baseline` | `pi_natives.linux-x64-baseline.node` |
| `//:natives-linux-x64-modern` | `//bazel/platforms:linux-x64-modern` | `pi_natives.linux-x64-modern.node` |
| `//:natives-linux-arm64` | `//bazel/platforms:linux-arm64` | `pi_natives.linux-arm64.node` |
| `//:natives-linux-musl-x64-baseline` | `//bazel/platforms:linux-musl-x64-baseline` | `pi_natives.linux-x64-baseline.node` |
| `//:natives-linux-musl-arm64` | `//bazel/platforms:linux-musl-arm64` | `pi_natives.linux-arm64.node` |
| `//:natives-darwin-x64-baseline` | `//bazel/platforms:darwin-x64-baseline` | `pi_natives.darwin-x64-baseline.node` |
| `//:natives-darwin-arm64` | `//bazel/platforms:darwin-arm64` | `pi_natives.darwin-arm64.node` |
| `//:natives-win32-x64-baseline` | `//bazel/platforms:win32-x64-baseline` | `pi_natives.win32-x64-baseline.node` |
Notes:
- musl addons **intentionally reuse** the plain `linux-<arch>` filenames — the loader never sees gnu and musl side by side; release jobs keep them in separate invocations/dest dirs (`scripts/bazel-natives.ts` hard-errors on a basename collision within one run).
- Aggregates: `//:natives-linux-all` (all linux targets + the msvc cross build, i.e. everything buildable from a linux-x64 host) and `//:natives-darwin-all` (mac hosts only).
### 2) `native_addon` rule (`bazel/defs.bzl`)
`native_addon` wraps `//crates/pi-natives:pi_natives` (a `rust_shared_library`) in a configuration transition that pins, per target:
- `--platforms=<the addon's platform>`
- `--compilation_mode=opt`
- `@rules_rust//rust/settings:lto=thin`
- extra rustc flags `-Ccodegen-units=16 -Cstrip=symbols`
This mirrors the old cargo `ci` profile. Because the profile lives **in the transition**, a bare `bazel build //:natives-<t>` is always release-grade regardless of `-c`, and every addon shares one cache entry per (platform, source) pair. The rule then symlinks the produced shared library to the loader's canonical `pi_natives.<platform>-<arch>[-<variant>].node` name, scoped under the rule name (`bazel-bin/natives-<t>/…`) so gnu/musl outputs with identical basenames cannot collide at the package level.
Per-target codegen that is not part of the transition lives in `crates/pi-natives/BUILD.bazel` `rustc_flags` selects: `-Ctarget-cpu=x86-64-v2` (baseline) / `x86-64-v3` (modern) via `//bazel/variants`, the napi link args (`-Wl,-undefined,dynamic_lookup` on macOS, `-Wl,-z,nodelete` on linux — `build.rs`/`napi_build::setup()` is deliberately not wired in), and `-Ctarget-feature=-crt-static` for musl.
### 3) Platforms and toolchains
| Target family | cc toolchain | Notes |
| --- | --- | --- |
| linux gnu (x64/arm64) | `@zig_sdk//libc_aware/toolchain:linux_*_gnu.2.17` (hermetic zig cc) | glibc **2.17** portability floor — same floor the previous cross builds used |
| linux musl (x64/arm64) | `@zig_sdk//libc_aware/toolchain:linux_*_musl` | dynamic CRT (`-Ctarget-feature=-crt-static` in the crate BUILD) |
| darwin (x64/arm64) | host Xcode toolchain | Apple frameworks aren't redistributable; darwin addons build on mac hosts only |
| win32-x64 msvc | `//bazel/toolchains/msvc` (`@msvc_cc`): clang-cl + lld-link + xwin CRT/SDK | hermetic cross-link from linux-x64 CI pods and darwin dev hosts; see `bazel/toolchains/msvc/NOTES.md` |
Rust toolchains are nightly (pinned in `MODULE.bazel`), with repo-local musl re-registrations in `//bazel/toolchains` carrying an explicit `@zig_sdk//libc:musl` constraint (rules_rust's generated gnu and musl toolchains otherwise share (os, cpu) constraints).
### 4) Third-party crates (`crate_universe`)
`@crates//...` is generated from the workspace `Cargo.toml`/`Cargo.lock`, restricted to exactly the seven shipped triples. Crate-specific build fixes live as `crate.annotation`s in `MODULE.bazel` (see the debugging playbook below).
The root module intentionally omits `crate_universe`'s optional rendering lock. The first evaluation after crate inputs change splices the workspace and generates external repository specs from the pinned `Cargo.lock`; Bazel records that extension result in `MODULE.bazel.lock`, so later clean output bases reuse it. Cargo manifest, lock, and annotation edits therefore require no separate repin step.
## Local development
### Building addons
```bash
# Addon for the current host (x64 hosts pick modern vs baseline via AVX2 detection),
# installed into packages/natives/native/:
bun --cwd=packages/natives run build # = bun ../../scripts/bazel-natives.ts host --dest native
# same, from the repo root:
bun run build:native
# The driver directly — targets are //:natives-* names plus pseudo-targets
# host / linux-all / darwin-all:
bun scripts/bazel-natives.ts <target>... [--dest <dir>] [-- <extra bazel args>]
bun scripts/bazel-natives.ts linux-x64-baseline linux-x64-modern --dest packages/natives/native
bun scripts/bazel-natives.ts darwin-all
# Or bazelisk directly (outputs stay in bazel-bin, nothing is installed):
bazelisk build //:natives-darwin-arm64
bazelisk build //:natives-linux-all
```
The driver runs one `bazel build` for all requested targets, locates outputs via `bazel cquery --output=files` (falling back to the `bazel-bin/natives-<t>/<canonical>.node` path convention), and copies them dereferenced into `--dest` (default `packages/natives/native`). Extra args after `--` go to bazel verbatim. It resolves `bazelisk` (or `bazel`) from `PATH` and honors an `OMP_BAZEL_RC` env var as a `--bazelrc=` startup option (that's how CI injects cache wiring).
Building `linux-all` into one dest would clobber gnu addons with musl ones (shared basenames) — the driver refuses; use separate invocations with separate `--dest` dirs.
### Typedef regeneration (napi CLI, dev-only)
`native/index.js`/`index.d.ts` are **committed**, so Bazel artifact builds never need the napi CLI. Only when the Rust API surface changes its exported typedefs:
```bash
bun --cwd=packages/natives run build:bindings # = bun scripts/build-bindings.ts
```
This runs the napi CLI (host-only, local cargo profile) against `crates/pi-natives`, installs the regenerated `index.d.ts`, normalizes the addon filename, and re-renders the explicit ESM exports + runtime enum objects via `gen-enums.ts`. Commit the resulting `index.js`/`index.d.ts` changes.
### Opt-in remote cache (`.bazelrc.user`)
`.bazelrc` ends with `try-import %workspace%/.bazelrc.user` (gitignored). The bazel-remote endpoint is cluster-internal only; if you can reach it (VPN/tailnet), wire it read-only:
```
# .bazelrc.user
build --config=cache-ro
build --remote_cache=grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092
build --tls_certificate=infra/bazel-remote/ca.crt
```
`cache-ro`/`cache-rw` in `.bazelrc` carry only policy (upload on/off, `--remote_local_fallback`, retries/timeout so a cache outage never fails the build); endpoint + credentials are always composed by the consumer. A plain `--disk_cache=<dir>` line also works fine here.
## CI
### Split Rust validation and addon production
`.github/workflows/ci.yml` separates `rust_validate` from `native_addons`. Both run on `omp-kata` pods for pushes and `ubuntu-22.04` for pull requests, but TypeScript jobs depend only on `native_addons`.
`rust_validate` uses `.github/actions/native-inputs` to inspect the complete pull-request file list. TypeScript-only pull requests skip every Rust step; native-affecting changes and all non-PR events run:
```bash
bazelisk --bazelrc="$rc" test //crates/... # full Rust suite
# clippy scope mirrors `cargo clippy --workspace` (libraries only), split by
# lint policy via a query kind filter:
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
| xargs bazelisk --bazelrc="$rc" build --config=clippy-strict --
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (…strict set…) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
| xargs bazelisk --bazelrc="$rc" build --config=clippy --
bazelisk --bazelrc="$rc" build --config=rustfmt //crates/...
```
- `--config=clippy` = rules_rust clippy aspect + `-Dwarnings`; `--config=clippy-strict` layers the generated `bazel/clippy.bazelrc` for crates with `[lints] workspace = true`.
- `--config=rustfmt` = rustfmt aspect against the workspace `rustfmt.toml`.
- `rust_validate` never saves a hosted disk-cache archive: `native_addons` may concurrently own the same immutable key, while the main-branch warmer publishes a combined validation/addon archive.
`native_addons` is the artifact producer for every downstream TypeScript and release job:
- Pull requests first restore the exact `native-addons-v1-linux-x64-baseline+modern-opt-<source-hash>` cache entry published by trusted main builds. Both addons are loaded before use; a miss or failed smoke check falls back to building the Linux x64 pair.
- Main and other non-PR runs build `//:natives-linux-all`, smoke the x64 pair before publishing its exact addon cache, and upload every `.node` output as the `native-addons` workflow artifact.
- Downstream jobs use `.github/actions/native-artifacts` to download that workflow artifact and install the requested target set without invoking Bazel.
No toolchain setup steps are required for native jobs: bazelisk is on the GitHub images and baked into the kata runner image; Bazel fetches Rust/zig/LLVM/xwin hermetically.
### Hosted cache warmer
`.github/workflows/bazel-cache-warm.yml` runs the full hosted validation and Linux x64 addon invocation set on `ubuntu-22.04`. Main-branch input changes restore the previous config-compatible generation, rebuild incrementally, and publish one combined exact-key disk-cache archive visible to pull requests.
### `bazel-cache` action (`.github/actions/bazel-cache`)
Single source of truth for cache wiring, emitted as a bazelrc fragment (its `rc` output) that consumers pass via `bazelisk --bazelrc=...` or `OMP_BAZEL_RC`. Two modes are selected via `BAZEL_REMOTE_USER`/`BAZEL_REMOTE_PASSWORD`:
| Runner | Fragment contents |
| --- | --- |
| omp-kata pod | `--config=ci --config=cache-rw --remote_cache=grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092 --tls_certificate=infra/bazel-remote/ca.crt --remote_header='authorization=Basic <b64 ci creds>'` |
| GitHub-hosted | `--config=ci --disk_cache=~/.cache/omp-bazel-disk --repository_cache=~/.cache/omp-bazel-repo` |
Hosted disk caches use `bazel-disk-v3-<scope>-<os>-<arch>-<config-hash>-<source-hash>`. The config hash covers Cargo/Bazel/toolchain settings; the source hash covers `crates/**` and root `BUILD.bazel`. An exact miss restores the newest config-compatible generation and permits one refreshed exact-key save. The remote endpoint resolves only inside the cluster.
### Native artifact actions
`.github/actions/bazel-natives` is the direct builder: `bazel-cache` → `OMP_BAZEL_RC=<rc> bun scripts/bazel-natives.ts <targets> --dest <dest>`, followed by a disk-cache save after a hosted miss. `.github/actions/native-artifacts` is the no-build consumer: download `native-addons` → run the same driver with `--source`.
### `release_binary`
Linux and Windows release matrices install addons from the `native_addons` workflow artifact. Darwin artifacts cannot be cross-built on Linux, so each macOS matrix builds only its own architecture through `bazel-natives` with scope `release-<target_id>`, then `bun run ci:release:build-binaries` embeds and compiles the executable.
## Debugging playbook
### Where things land / how to inspect
```bash
# Outputs (workspace-relative): bazel-bin/natives-<target>/pi_natives.<...>.node
bazelisk cquery --output=files //:natives-linux-x64-baseline
# What actions/flags a target produces (add the same --config flags as the build):
bazelisk aquery 'outputs(".*\.node", deps(//:natives-linux-arm64))'
bazelisk aquery 'mnemonic("Rustc", deps(//crates/pi-natives:pi_natives))'
# Which toolchain resolved (e.g. confirm @msvc_cc, not host cc, for win32):
bazelisk cquery 'deps(//:natives-win32-x64-baseline)' | grep msvc_cc
# Keep the sandbox dir + print the full command line of a failing action:
bazelisk build --sandbox_debug --verbose_failures //:natives-<t>
# Analyze without building (cheap cross-target sanity check):
bazelisk build --nobuild //:natives-win32-x64-baseline
```
`scripts/bazel-natives.ts` streams bazel stderr live and repeats a 40-line tail on failure; when its cquery step fails it falls back to the `bazel-bin` path convention.
### Common failure classes (seen during bring-up — fixes already in tree, cite when they resurface)
| Symptom | Cause | Fix (in tree) |
| --- | --- | --- |
| musl build "succeeds" but emits no `.node` | musl defaults to `+crt-static`; rustc silently emits no cdylib | `-Ctarget-feature=-crt-static` select in `crates/pi-natives/BUILD.bazel` |
| opus/cmake `try_compile` fails linking UBSan runtime | zig cc enables UBSan by default; cmake's test exe links with the raw wrapper (no toolchain features) | `CFLAGS=-fno-sanitize=undefined` in the `audiopus_sys` annotation (`MODULE.bazel`) |
| `tree-sitter-just` scanner.c `#error` under opt | scanner hard-errors when `NDEBUG` is set (opt-mode cc default) | `CFLAGS=-UNDEBUG` annotation (cc-rs appends env CFLAGS last, so `-U` wins) |
| rstest macro: "Cargo.toml not found" in a vendored test | rstest verifies `Cargo.toml` exists in the manifest dir | `compile_data = ["Cargo.toml"]` on the `rust_test` (see `crates/vendor/uu-tail/BUILD.bazel`) |
| vendored tests fail on bare `test_data/...` paths / symlink into srcs | tests assume cargo's cwd, incompatible with runfiles execution | `tags = ["manual"]` (e.g. `//crates/vendor/uu-find:uu-find_test`); run via `cargo nextest` when touching the fork; hermetic sibling test covers the contract |
| blake3 msvc: `ml64.exe` not found | cc-rs resolves MASM from build-script PATH on non-windows hosts | `bin/ml64.exe → llvm-ml -m64` shim in `@msvc_cc`, prepended via the `blake3` annotation PATH |
| audiopus_sys msvc: cmake demands VS generator / rc+mt tools; `try_compile` wants `msvcrtd.lib` | cross cmake on linux/mac hosts; Debug config → `/MDd` which the lean xwin splat lacks | `CMAKE_GENERATOR_x86_64_pc_windows_msvc=Ninja` + `@msvc_cc`'s `toolchain.cmake` (`CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc`) pinning wrappers + Release try-compile + `/MD` |
| win32 link oddities generally | — | read `bazel/toolchains/msvc/NOTES.md` first: wrapper self-location, `lld-link` flavor/driver-link behavior, `LIB`, `/MD` CRT choice, xwin splat caveats |
| `rust_test(crate = ...)` "can't find crate" at macro expansion | rmeta-only pipelined deps break macro_rules re-export harness compiles | rust pipelined_compilation stays OFF (`.bazelrc` note) |
| build script can't find cmake/ninja | `--incompatible_strict_action_env` — no host env leaks | explicit `PATH` in the crate annotation (`MODULE.bazel`), not host env |
### Cache behavior
- **omp-kata:** read-write gRPC to the in-cluster bazel-remote (`grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092`, TLS via the committed `infra/bazel-remote/ca.crt`, htpasswd user `ci`). `--remote_local_fallback` plus retries make an outage degrade to local execution rather than fail the build.
- **GitHub-hosted:** no cluster access. The v3 `actions/cache` disk key separates config and source generations; `.github/workflows/bazel-cache-warm.yml` publishes the combined default-branch archive from the same `ubuntu-22.04` image as pull-request consumers. The smaller final-addon cache is independent and is trusted only after both addons load successfully.
- **msvc repos:** the ~2 GiB LLVM download is sha256-pinned and repository-cache backed; the ~1 GiB xwin CRT/SDK splat is fetched from the Microsoft CDN inside the repo rule and is **not** repo-cache backed — a cold output base re-downloads it. Microsoft advances the VS channel payload over time, so remote-cache hit rates for win32 actions degrade gracefully after an MS bump (same property the previous cross toolchain had). Win32 link actions also don't share cache entries across host OSes (linux vs mac clang binaries).
- Server-side operations (deploy, TLS/auth, egress, poisoning boundary): `infra/docs/04-arc-and-caching.md` §5.
## Target/variant model and naming conventions
## Platform tag
Both build and runtime use platform tag:
`<platform>-<arch>` (example: `darwin-arm64`, `linux-x64`).
## Variant model (x64 only)
x64 supports CPU variants, encoded as `//bazel/variants` constraint values on the platform (baseline → `-Ctarget-cpu=x86-64-v2`, modern → `x86-64-v3`):
- `modern` (AVX2-capable path)
- `baseline` (fallback)
Non-x64 uses a single default artifact with no variant suffix. There is no build-time variant *switch*: each variant is its own `//:natives-*` target, and the `host` pseudo-target picks modern vs baseline via AVX2 detection.
### Output filenames
- x64: `pi_natives.<platform>-<arch>-modern.node` or `...-baseline.node`
- non-x64: `pi_natives.<platform>-<arch>.node`
Runtime x64 candidate order also includes the unsuffixed default filename after the selected variant candidates.
## Runtime flags
- `PI_NATIVE_VARIANT`: x64 runtime override; valid values are `modern` and `baseline`.
- `PI_COMPILED`: legacy compiled-mode signal. A populated embedded-addon manifest is also a compiled-mode signal; compiled release builds additionally define `process.env.PI_COMPILED="true"` during `bun build --compile`.
## Embed lifecycle (`embed-native.ts`)
1. **Init**: compute the platform tag (host values, overridable by the release packaging script for cross-target archives).
2. **Candidate set**:
- x64 looks for `modern` and `baseline` files;
- non-x64 looks for one default file.
3. **Validate availability**: at least one expected file must exist in `packages/natives/native`.
4. **Generate archive + manifest**: write `native/embedded-addons.<platform>-<arch>.tar.gz` containing all available target addon files and `native/embedded-addon.js` with package version, archive metadata, and file sizes.
5. **Runtime extraction ready** for compiled mode.
`--reset` writes the null manifest stub (`embeddedAddon = null`) without validating addon availability, and deletes any existing `embedded-addons.*.tar.gz` archives from `native/`.
## Dev workflow vs shipped/compiled behavior
## Local development workflow
Typical local loop:
1. Build addon: `bun --cwd=packages/natives run build`.
2. Loader resolves platform npm leaf-package candidates (`@oh-my-pi/pi-natives-<platform>-<arch>`, when resolvable), then package-local `native/` and executable-dir fallback candidates.
3. Generated declarations in `native/index.d.ts` describe the public TS API (regenerate with `build:bindings` only when the Rust API surface changes).
## Shipped/compiled binary workflow
In compiled mode (`PI_COMPILED`, Bun embedded URL markers, or populated embedded manifest):
1. Loader computes versioned cache dir: `<getNativesDir()>/<packageVersion>`.
2. If embedded manifest matches current platform+version, loader extracts the selected file from `embedded-addons.<tag>.tar.gz` into that versioned dir when the cached file is absent or has the wrong size.
3. Runtime candidate order includes:
- extracted versioned cache path, if available,
- versioned cache dir,
- legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere),
- package/executable directories.
4. First successfully loaded addon with the expected version sentinel is returned.
This is why packaging + runtime loader expectations must align: filenames, platform tags, CPU variants, and embedded manifest version must match what `native/loader-state.js` probes.
## JS API ↔ Rust export mapping (build sanity subset)
Generated declarations currently include exports from these Rust modules:
| Area | Representative JS exports | Rust source |
| ---------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Search/workspace | `grep`, `search`, `hasMatch`, `fuzzyFind`, `glob`, `listWorkspace`, `invalidateFsScanCache` | `grep.rs`, `fd.rs`, `glob.rs`, `workspace.rs`, `fs_cache.rs` |
| AST/block/summary | `astGrep`, `astEdit`, `blockRangeAt`, `summarizeCode` | `ast.rs`, `block.rs`, `summary.rs` |
| Text/highlight/tokens | `visibleWidth`, `truncateToWidth`, `highlightCode`, `countTokens` | `text.rs`, `highlight.rs`, `tokens.rs` |
| Shell/PTY/process/keys | `executeShell`, `Shell`, `PtySession`, `Process`, `parseKey`, `applyBashFixups` | `shell.rs`, `pty.rs`, `ps.rs`, `keys.rs` |
| Media/system/iso | `encodeSixel`, clipboard, macOS appearance/power, `getWorkProfile`, `isoBackend`, `isoStart`, `isoDiff` | `sixel.rs`, `clipboard.rs`, `appearance.rs`, `power.rs`, `prof.rs`, `iso.rs` |
## Failure behavior and diagnostics
## Build-time failures
- Bazel analysis/compile failure: `scripts/bazel-natives.ts` surfaces the exit code plus a stderr tail; re-run the printed `bazel build` line directly (add `--verbose_failures`, `--sandbox_debug`) to iterate.
- Unknown target name: the driver errors with the full known-target list (`//:natives-*` names + `host`/`linux-all`/`darwin-all`).
- No `.node` outputs located after a successful build: driver exits 1 (check `bazel cquery --output=files` manually).
- Basename collision (gnu + musl in one invocation): driver refuses to install and names both sources — split into separate `--dest` dirs.
- `build:bindings` (napi) failure: script surfaces non-zero exit and stderr; artifact builds are unaffected (Bazel never runs the napi CLI).
## Runtime loader failures (`native/loader-state.js`)
- Unsupported platform tag: throws with supported platform list after probing fails.
- No candidate could load: throws with full candidate error list and mode-specific remediation hints.
- Embedded extraction and Windows staging problems: archive/mkdir/write/copy errors are recorded and included in final diagnostics if load fails.
- Version mismatch: install/compiled loads that lack the package-version sentinel are rejected during candidate probing.
## Troubleshooting matrix
| Symptom | Likely cause | Verify | Fix |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `Cannot find module` or dynamic library load error for every candidate | Missing release artifact, wrong platform tag, or stale compiled cache | Inspect loader error list and `packages/natives/native` filenames | Build correct target (`bun scripts/bazel-natives.ts <t> --dest packages/natives/native`); delete stale cache for the package version |
| Export is missing at runtime but present in TypeScript | Stale `.node` loaded, generated declarations newer than binary, or Rust export not compiled | Require the actual candidate and inspect `Object.keys(mod)` | Rebuild native package and remove stale candidate/cache paths |
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or modern file unavailable | Check env and filenames in `native/` | Build and ship the modern target (`bun scripts/bazel-natives.ts linux-x64-modern --dest packages/natives/native`) |
| gnu addon overwritten by musl (or vice versa) | Both built into one dest — they share canonical basenames by design | Compare `bazel-bin/natives-<t>/` sources vs installed file | Separate invocations with separate `--dest` dirs (release matrix already does this) |
| Compiled binary fails after upgrade | Stale extracted cache, embedded archive mismatch, or embedded manifest version mismatch | Inspect `<getNativesDir()>/<version>` and loader error list | Delete versioned cache for the package version; regenerate embedded archive/manifest during packaging |
| `gen:native` fails with `No native addons found` | Required platform artifact was not built before embedding | Check expected list in error text | Build at least one expected artifact for the target, then rerun `gen:native` |
## Operational commands
```bash
# Addon for the current host, installed into packages/natives/native/
bun --cwd=packages/natives run build
# Explicit targets (x64 variants are separate targets, not env switches)
bun scripts/bazel-natives.ts linux-x64-modern linux-x64-baseline --dest packages/natives/native
# Raw bazel (output: bazel-bin/natives-<t>/pi_natives.<...>.node)
bazelisk build //:natives-darwin-arm64
# Regenerate TS typedefs + enum exports (napi CLI, only on Rust API changes)
bun --cwd=packages/natives run build:bindings
# Generate embedded addon manifest from built native files
bun run gen:native
# Output archive: packages/natives/native/embedded-addons.<platform>-<arch>.tar.gz
# Reset embedded manifest to null stub
bun run gen:native:reset
```
## Orchestrator-side content-addressed build cache (robomp)
When `pi-natives` is built inside the robomp orchestrator (`python/robomp/`), workspaces share built artifacts through a content-addressed cache instead of rebuilding from scratch in every per-issue worktree. The cache is **orchestrator-side only** — `bun --cwd=packages/natives run build` itself is unchanged; the cache lives outside the build pipeline and is populated/captured around `ensure_workspace` and post-task success in `python/robomp/src/natives_cache.py`.
### What is cached
The complete set of files in `packages/natives/native/` that are pure functions of the cache-key inputs:
- `pi_natives.<platform>-<arch>[-variant].node` (glob `pi_natives.*.node`)
- `index.d.ts`
- `index.js`
- `embedded-addon.js`
- `manifest.json` (cache metadata: key, target triple, capture timestamp, source workspace, commit)
An entry is only considered a hit when the `.node` glob matches AND every companion plus the manifest is present. Partial entries are evicted on GC.
### Cache key
The key is `sha256` over `(path \t git-tree-hash \n)` pairs for the following inputs, in this order (order is significant), followed by the target triple:
1. `crates` (whole subtree — pi-natives transitively depends on other workspace crates)
2. `Cargo.lock`
3. `Cargo.toml`
4. `rust-toolchain.toml`
5. `packages/natives` (whole subtree — build script, `scripts/*`, package.json)
Tree hashes come from one `git cat-file --batch-check` invocation against `HEAD`; paths missing from `HEAD` fold in as a fixed null hash so the key stays deterministic across repos that don't ship every input. The target-triple suffix matches the addon basename convention (`<platform>-<arch>` for non-x64, `<platform>-<arch>-<variant>` for x64).
Anything outside this input set (Bazel definition files such as `MODULE.bazel`/`BUILD.bazel`, host glibc, env vars) is **not** in the key. If you need to invalidate after such a change, delete the cache directory by hand or bump one of the input files.
### Layout and ownership
- Root: `/data/cache/pi-natives` (provisioned by `entrypoint.sh` alongside the cargo caches, owned `root:omp`, mode `02770` setgid so cached files inherit `gid=omp` and stay readable by every slot user).
- Per-repo subdirectory: `<root>/<repo-slug>/` where the slug is `owner__repo` (mirrors `SandboxManager.pool_path`).
- Per-entry directory: `<root>/<repo-slug>/<sha256-key>/` containing the cached files plus `manifest.json`.
- Per-repo lockfile: `<root>/<repo-slug>/.lock` (advisory `fcntl.flock`, exclusive on capture and GC).
- Staging dirs (`.<key>.tmp.<pid>`) during capture; renamed atomically into the final entry path. Stale staging dirs from crashed captures are swept on GC.
### Populate and capture semantics
- **Populate** (workspace ← cache) runs inside `ensure_workspace`. On a key hit the `.node` is **hardlinked** into the workspace (zero-copy, shared inode); the companion `index.d.ts` / `index.js` / `embedded-addon.js` are **copied** (independent inodes) because the bindings regeneration flow (`build-bindings.ts`'s `installGeneratedBindings` and `gen-enums.ts`) rewrites those files via `open(..., 'w')` — an in-place truncate that would otherwise propagate through a hardlink and corrupt the cache. Cross-device hardlink failures (`EXDEV`) fall back to copy.
- **Capture** (cache ← workspace) runs from the post-task success path when the build produced a complete artifact set. Capture uses **copy**, not hardlink: hardlinking a slot-owned workspace file would preserve slot UID ownership on the cached inode and defeat the shared-group model. Copying creates a fresh root-owned, `gid=omp` inode via the setgid cache root. Capture is idempotent under the per-repo flock: a concurrent capture for the same key returns the existing entry.
### Garbage collection
A periodic GC loop runs in `WorkerPool` with two caps per repo. When either cap is exceeded, oldest entries (by `manifest.json.captured_at`) are dropped first:
- entry count cap (`max_entries_per_repo`, default 8)
- byte cap (`max_bytes`, default 4 GiB)
Workspaces that hardlinked a `.node` before GC retain access via the kernel inode refcount — `rmtree` of the cache entry does not delete the file from the workspace.
### Configuration (settings on `robomp.config.Settings`)
| Env var | Default | Effect |
| ------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------- |
| `ROBOMP_NATIVES_CACHE_ENABLED` | `true` | Master switch. When false the populate/capture hooks no-op and every workspace builds from scratch. |
| `ROBOMP_NATIVES_CACHE_ROOT` | `/data/cache/pi-natives` | Cache root directory. Must be `root:omp 02770` for cross-slot reads. |
| `ROBOMP_NATIVES_CACHE_MAX_ENTRIES_PER_REPO` | `8` | LRU entry-count cap, per repo slug. |
| `ROBOMP_NATIVES_CACHE_MAX_BYTES` | `4294967296` (4 GiB) | LRU byte cap, per repo slug. |
| `ROBOMP_NATIVES_CACHE_GC_INTERVAL_SECONDS` | `3600` | Period of the background GC loop in `WorkerPool`. |
### Manual invalidation
- One key: `rm -rf /data/cache/pi-natives/<repo-slug>/<sha256>`.
- One repo: `rm -rf /data/cache/pi-natives/<repo-slug>`.
- Everything: `rm -rf /data/cache/pi-natives/*` (preserve the root so its setgid mode survives).
- Stuck lock: `rm /data/cache/pi-natives/<repo-slug>/.lock` (only when no orchestrator process is touching the repo).
Trigger an automatic miss by editing any path in the key set: a single touched byte under `crates/`, `Cargo.lock`, `Cargo.toml`, `rust-toolchain.toml`, or `packages/natives/` shifts the tree hash and forces a fresh build at the next populate.
Reference in New Issue View Git Blame Copy Permalink