feat(build): migrated native pipeline to bazel with remote caching
- Replaced the napi-cli/cargo-zigbuild/cargo-xwin/sccache build path with Bazel: rules_rust + crate_universe over Cargo.lock, hermetic zig cc toolchains (linux-gnu pinned to glibc 2.17, linux-musl), host Xcode for darwin, and a repo-local hermetic clang-cl + llvm-ml + xwin toolchain for windows-msvc (bazel/toolchains/msvc). - All eight shipped addons build as //:natives-<target> via the release transition in bazel/defs.bzl (opt, thin LTO, cgu=16, stripped, canonical .node naming); scripts/bazel-natives.ts is the single driver for local dev and CI. - Rust validation moved to bazel test + clippy aspects (strict workspace policy for opted-in crates, default lints elsewhere, mirroring cargo semantics) and the rustfmt aspect; cargo stays as the dev-iteration surface, with brush-core/brush-builtins promoted to workspace members and excluded from cargo dev tasks to keep their historical scope. - CI caches through an in-cluster bazel-remote action cache (TLS + basic auth, cluster-internal only); GitHub-hosted runners never touch the infrastructure and use an actions/cache-backed disk cache instead. - Deleted the hand-rolled caching machinery: ci-target-cache, ci-native-artifact-cache, ci-build-native, native-source-hash, find-native-artifacts, restore-linux-native, native-prewarm workflow, ensure-* toolchain actions, and all sccache/Swatinem wiring. - Warm native rebuilds drop from ~20 minutes to seconds; a cold client with a warm remote cache rebuilds the linux x64 pair in ~2.5 minutes.
This commit is contained in:
@@ -2,61 +2,229 @@
|
||||
|
||||
This runbook describes how `@oh-my-pi/pi-natives` produces `.node` addons, generated declarations, and compiled-binary embedded payloads, and how to debug loader/build failures.
|
||||
|
||||
Addon **artifacts are built by Bazel** (`rules_rust` + `crate_universe` + hermetic cc toolchains); the cargo workspace stays authoritative for local Rust iteration (rust-analyzer, `cargo nextest`) and for napi typedef regeneration. Runtime loading and embedding are unchanged.
|
||||
|
||||
It follows the architecture terms from `docs/natives-architecture.md`:
|
||||
|
||||
- **build-time artifact production** (`scripts/build-native.ts`)
|
||||
- **build-time artifact production** (Bazel `//:natives-<target>` via `scripts/bazel-natives.ts`)
|
||||
- **embedded addon manifest generation** (`scripts/embed-native.ts`)
|
||||
- **runtime addon loading** (`native/index.js`, `native/loader-state.js`)
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/scripts/build-native.ts`
|
||||
- `packages/natives/scripts/embed-native.ts`
|
||||
- `packages/natives/scripts/gen-enums.ts`
|
||||
Build side:
|
||||
|
||||
- `BUILD.bazel` (root) — the eight `//:natives-<target>` addon targets + aggregate filegroups
|
||||
- `bazel/defs.bzl` — the `native_addon` rule/transition
|
||||
- `bazel/platforms/BUILD.bazel` — one `platform()` per shipped addon
|
||||
- `bazel/variants/BUILD.bazel` — `baseline`/`modern` ISA constraint values
|
||||
- `bazel/toolchains/` — musl rustc disambiguation + the msvc cross cc toolchain (`msvc/NOTES.md`)
|
||||
- `bazel/clippy.bazelrc` — generated from `[workspace.lints]` in `Cargo.toml`
|
||||
- `MODULE.bazel`, `.bazelrc`, `.bazelversion` (Bazel 9.2.0), `Cargo.Bazel.lock`
|
||||
- `scripts/bazel-natives.ts` — the canonical driver (build + locate + install)
|
||||
- `crates/pi-natives/BUILD.bazel`, `crates/pi-natives/Cargo.toml`
|
||||
|
||||
Package side (unchanged runtime/packaging):
|
||||
|
||||
- `packages/natives/scripts/build-bindings.ts` — dev-only typedef regeneration
|
||||
- `packages/natives/scripts/embed-native.ts`, `gen-enums.ts`, `gen-npm-packages.ts`
|
||||
- `packages/natives/package.json`
|
||||
- `packages/natives/native/index.js`
|
||||
- `packages/natives/native/loader-state.js`
|
||||
- `crates/pi-natives/Cargo.toml`
|
||||
- `packages/natives/native/index.js`, `native/loader-state.js`
|
||||
|
||||
## Build pipeline overview
|
||||
## Build architecture
|
||||
|
||||
### 1) Build entrypoints
|
||||
### 1) `//:natives-<target>` addon targets
|
||||
|
||||
`packages/natives/package.json` scripts:
|
||||
Root `BUILD.bazel` instantiates one `native_addon` per shipped `(platform, arch, ISA-variant)`:
|
||||
|
||||
- `bun scripts/build-native.ts` (`build`) → N-API build, addon install, generated declarations install, explicit ESM export and enum runtime patch.
|
||||
- `bun scripts/embed-native.ts` (`gen:native`) → generate `native/embedded-addon.js` plus `native/embedded-addons.<tag>.tar.gz` from built files.
|
||||
- `bun scripts/gen-npm-packages.ts` (`gen:npm`) → generate per-platform npm leaf packages (`@oh-my-pi/pi-natives-<platform>-<arch>`, installed as optional dependencies of the core package) under `npm/` from built addon files.
|
||||
| Target | Platform | Canonical output |
|
||||
| --------------------------------- | --------------------------------------- | ------------------------------------ |
|
||||
| `//:natives-linux-x64-baseline` | `//bazel/platforms:linux-x64-baseline` | `pi_natives.linux-x64-baseline.node` |
|
||||
| `//:natives-linux-x64-modern` | `//bazel/platforms:linux-x64-modern` | `pi_natives.linux-x64-modern.node` |
|
||||
| `//:natives-linux-arm64` | `//bazel/platforms:linux-arm64` | `pi_natives.linux-arm64.node` |
|
||||
| `//:natives-linux-musl-x64-baseline` | `//bazel/platforms:linux-musl-x64-baseline` | `pi_natives.linux-x64-baseline.node` |
|
||||
| `//:natives-linux-musl-arm64` | `//bazel/platforms:linux-musl-arm64` | `pi_natives.linux-arm64.node` |
|
||||
| `//:natives-darwin-x64-baseline` | `//bazel/platforms:darwin-x64-baseline` | `pi_natives.darwin-x64-baseline.node` |
|
||||
| `//:natives-darwin-arm64` | `//bazel/platforms:darwin-arm64` | `pi_natives.darwin-arm64.node` |
|
||||
| `//:natives-win32-x64-baseline` | `//bazel/platforms:win32-x64-baseline` | `pi_natives.win32-x64-baseline.node` |
|
||||
|
||||
Root scripts include `build:native` as `bun --cwd=packages/natives run build`.
|
||||
Notes:
|
||||
|
||||
### 2) N-API/Rust artifact build
|
||||
- musl addons **intentionally reuse** the plain `linux-<arch>` filenames — the loader never sees gnu and musl side by side; release jobs keep them in separate invocations/dest dirs (`scripts/bazel-natives.ts` hard-errors on a basename collision within one run).
|
||||
- Aggregates: `//:natives-linux-all` (all linux targets + the msvc cross build, i.e. everything buildable from a linux-x64 host) and `//:natives-darwin-all` (mac hosts only).
|
||||
|
||||
`build-native.ts` invokes the `@napi-rs/cli` binary directly from `node_modules/.bin` with:
|
||||
### 2) `native_addon` rule (`bazel/defs.bzl`)
|
||||
|
||||
- `napi build`
|
||||
- `--manifest-path crates/pi-natives/Cargo.toml`
|
||||
- `--package-json-path packages/natives/package.json`
|
||||
- `--platform`
|
||||
- `--no-js`
|
||||
- `--dts index.d.ts`
|
||||
- `--profile local` for non-CI local native builds, otherwise `--profile ci`
|
||||
- `-o <isolated temp output dir>`
|
||||
- optional `--target <CROSS_TARGET>` plus `--cross-compile` (napi picks the `cargo-zigbuild` or `cargo-xwin` backend from the target) for cross builds
|
||||
`native_addon` wraps `//crates/pi-natives:pi_natives` (a `rust_shared_library`) in a configuration transition that pins, per target:
|
||||
|
||||
`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`; napi-rs emits `.node` artifacts plus generated `index.d.ts` in an isolated temporary output directory under `packages/natives/native/.build/`.
|
||||
- `--platforms=<the addon's platform>`
|
||||
- `--compilation_mode=opt`
|
||||
- `@rules_rust//rust/settings:lto=thin`
|
||||
- extra rustc flags `-Ccodegen-units=16 -Cstrip=symbols`
|
||||
|
||||
### 3) Artifact install
|
||||
This mirrors the old cargo `ci` profile. Because the profile lives **in the transition**, a bare `bazel build //:natives-<t>` is always release-grade regardless of `-c`, and every addon shares one cache entry per (platform, source) pair. The rule then symlinks the produced shared library to the loader's canonical `pi_natives.<platform>-<arch>[-<variant>].node` name, scoped under the rule name (`bazel-bin/natives-<t>/…`) so gnu/musl outputs with identical basenames cannot collide at the package level.
|
||||
|
||||
After napi-rs succeeds, `build-native.ts`:
|
||||
Per-target codegen that is not part of the transition lives in `crates/pi-natives/BUILD.bazel` `rustc_flags` selects: `-Ctarget-cpu=x86-64-v2` (baseline) / `x86-64-v3` (modern) via `//bazel/variants`, the napi link args (`-Wl,-undefined,dynamic_lookup` on macOS, `-Wl,-z,nodelete` on linux — `build.rs`/`napi_build::setup()` is deliberately not wired in), and `-Ctarget-feature=-crt-static` for musl.
|
||||
|
||||
1. resolves the built addon in the isolated output directory;
|
||||
2. normalizes its name to `pi_natives.<platform>-<arch>(-variant).node` when needed;
|
||||
3. installs the addon into `packages/natives/native/` with temp-file + rename semantics;
|
||||
4. copies generated `index.d.ts` into `packages/natives/native/`;
|
||||
5. runs `generateEnumExports()` to render explicit named ESM exports for classes/functions and runtime enum objects in the checked-in `native/index.js`.
|
||||
### 3) Platforms and toolchains
|
||||
|
||||
Windows locked-DLL update failures are handled at runtime by staging install candidates into the versioned native cache; install/rename failures during local builds still include explicit file-operation diagnostics.
|
||||
| Target family | cc toolchain | Notes |
|
||||
| --- | --- | --- |
|
||||
| linux gnu (x64/arm64) | `@zig_sdk//libc_aware/toolchain:linux_*_gnu.2.17` (hermetic zig cc) | glibc **2.17** portability floor — same floor the previous cross builds used |
|
||||
| linux musl (x64/arm64) | `@zig_sdk//libc_aware/toolchain:linux_*_musl` | dynamic CRT (`-Ctarget-feature=-crt-static` in the crate BUILD) |
|
||||
| darwin (x64/arm64) | host Xcode toolchain | Apple frameworks aren't redistributable; darwin addons build on mac hosts only |
|
||||
| win32-x64 msvc | `//bazel/toolchains/msvc` (`@msvc_cc`): clang-cl + lld-link + xwin CRT/SDK | hermetic cross-link from linux-x64 CI pods and darwin dev hosts; see `bazel/toolchains/msvc/NOTES.md` |
|
||||
|
||||
Rust toolchains are nightly (pinned in `MODULE.bazel`), with repo-local musl re-registrations in `//bazel/toolchains` carrying an explicit `@zig_sdk//libc:musl` constraint (rules_rust's generated gnu and musl toolchains otherwise share (os, cpu) constraints).
|
||||
|
||||
### 4) Third-party crates (`crate_universe`)
|
||||
|
||||
`@crates//...` is generated from the workspace `Cargo.toml`/`Cargo.lock` (lockfile: `Cargo.Bazel.lock`), restricted to exactly the seven shipped triples. Crate-specific build fixes live as `crate.annotation`s in `MODULE.bazel` (see the debugging playbook below).
|
||||
|
||||
**Repin flow:** after any `Cargo.toml`/`Cargo.lock` change (or annotation edit), run
|
||||
|
||||
```bash
|
||||
bun run bazel:repin # = CARGO_BAZEL_REPIN=1 bazelisk fetch @crates//...
|
||||
```
|
||||
|
||||
and commit the updated `Cargo.Bazel.lock`. A stale lockfile fails analysis with a "lockfile out of date" style error.
|
||||
|
||||
## Local development
|
||||
|
||||
### Building addons
|
||||
|
||||
```bash
|
||||
# Addon for the current host (x64 hosts pick modern vs baseline via AVX2 detection),
|
||||
# installed into packages/natives/native/:
|
||||
bun --cwd=packages/natives run build # = bun ../../scripts/bazel-natives.ts host --dest native
|
||||
# same, from the repo root:
|
||||
bun run build:native
|
||||
|
||||
# The driver directly — targets are //:natives-* names plus pseudo-targets
|
||||
# host / linux-all / darwin-all:
|
||||
bun scripts/bazel-natives.ts <target>... [--dest <dir>] [-- <extra bazel args>]
|
||||
bun scripts/bazel-natives.ts linux-x64-baseline linux-x64-modern --dest packages/natives/native
|
||||
bun scripts/bazel-natives.ts darwin-all
|
||||
|
||||
# Or bazelisk directly (outputs stay in bazel-bin, nothing is installed):
|
||||
bazelisk build //:natives-darwin-arm64
|
||||
bazelisk build //:natives-linux-all
|
||||
```
|
||||
|
||||
The driver runs one `bazel build` for all requested targets, locates outputs via `bazel cquery --output=files` (falling back to the `bazel-bin/natives-<t>/<canonical>.node` path convention), and copies them dereferenced into `--dest` (default `packages/natives/native`). Extra args after `--` go to bazel verbatim. It resolves `bazelisk` (or `bazel`) from `PATH` and honors an `OMP_BAZEL_RC` env var as a `--bazelrc=` startup option (that's how CI injects cache wiring).
|
||||
|
||||
Building `linux-all` into one dest would clobber gnu addons with musl ones (shared basenames) — the driver refuses; use separate invocations with separate `--dest` dirs.
|
||||
|
||||
### Typedef regeneration (napi CLI, dev-only)
|
||||
|
||||
`native/index.js`/`index.d.ts` are **committed**, so Bazel artifact builds never need the napi CLI. Only when the Rust API surface changes its exported typedefs:
|
||||
|
||||
```bash
|
||||
bun --cwd=packages/natives run build:bindings # = bun scripts/build-bindings.ts
|
||||
```
|
||||
|
||||
This runs the napi CLI (host-only, local cargo profile) against `crates/pi-natives`, installs the regenerated `index.d.ts`, normalizes the addon filename, and re-renders the explicit ESM exports + runtime enum objects via `gen-enums.ts`. Commit the resulting `index.js`/`index.d.ts` changes.
|
||||
|
||||
### Opt-in remote cache (`.bazelrc.user`)
|
||||
|
||||
`.bazelrc` ends with `try-import %workspace%/.bazelrc.user` (gitignored). The bazel-remote endpoint is cluster-internal only; if you can reach it (VPN/tailnet), wire it read-only:
|
||||
|
||||
```
|
||||
# .bazelrc.user
|
||||
build --config=cache-ro
|
||||
build --remote_cache=grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092
|
||||
build --tls_certificate=infra/bazel-remote/ca.crt
|
||||
```
|
||||
|
||||
`cache-ro`/`cache-rw` in `.bazelrc` carry only policy (upload on/off, `--remote_local_fallback`, retries/timeout so a cache outage never fails the build); endpoint + credentials are always composed by the consumer. A plain `--disk_cache=<dir>` line also works fine here.
|
||||
|
||||
## CI
|
||||
|
||||
### `rust` job (validate + cache warm)
|
||||
|
||||
`.github/workflows/ci.yml` `rust` runs on `omp-kata` pods for pushes and `ubuntu-22.04` for PRs, composes cache wiring via the `bazel-cache` action, then:
|
||||
|
||||
```bash
|
||||
bazelisk --bazelrc="$rc" test //crates/... # full Rust suite
|
||||
# clippy scope mirrors `cargo clippy --workspace` (libraries only), split by
|
||||
# lint policy via a query kind filter:
|
||||
bazelisk query "kind('rust_library|rust_shared_library', //crates/pi-ast/... + //crates/pi-iso/... + //crates/pi-natives/... + //crates/pi-shell/... + //crates/pi-walker/...)" \
|
||||
| xargs bazelisk --bazelrc="$rc" build --config=clippy-strict --
|
||||
bazelisk query "kind('rust_library|rust_shared_library', //crates/... - (…strict set…) - //crates/vendor/brush-core/... - //crates/vendor/brush-builtins/...)" \
|
||||
| xargs bazelisk --bazelrc="$rc" build --config=clippy --
|
||||
bazelisk --bazelrc="$rc" build --config=rustfmt //crates/...
|
||||
```
|
||||
|
||||
- `--config=clippy` = rules_rust clippy aspect + `-Dwarnings`; `--config=clippy-strict` layers the generated `bazel/clippy.bazelrc` (rendered from `[workspace.lints]` in `Cargo.toml` — regenerate it when workspace lints change) for the crates with `[lints] workspace = true`.
|
||||
- `--config=rustfmt` = rustfmt aspect against the workspace `rustfmt.toml`.
|
||||
- On main pushes (read-write cache) the job additionally runs `bazelisk build //:natives-linux-all` to warm the shared cache for every downstream job.
|
||||
|
||||
No toolchain setup steps: bazelisk is on the GitHub images and baked into the kata runner image; Bazel fetches Rust/zig/LLVM/xwin hermetically.
|
||||
|
||||
### `bazel-cache` action (`.github/actions/bazel-cache`)
|
||||
|
||||
Single source of truth for cache wiring, emitted as a bazelrc fragment (its `rc` output) that consumers pass via `bazelisk --bazelrc=...` (or `OMP_BAZEL_RC` for the driver). Two modes, detected via `BAZEL_REMOTE_USER`/`BAZEL_REMOTE_PASSWORD` (injected from the `bazel-remote-ci` secret on kata pods only):
|
||||
|
||||
| Runner | Fragment contents |
|
||||
| --- | --- |
|
||||
| omp-kata pod | `--config=ci --config=cache-rw --remote_cache=grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092 --tls_certificate=infra/bazel-remote/ca.crt --remote_header='authorization=Basic <b64 ci creds>'` |
|
||||
| GitHub-hosted | `--config=ci --disk_cache=~/.cache/omp-bazel-disk --repository_cache=~/.cache/omp-bazel-repo`, persisted by `actions/cache` keyed on `bazel-disk-<scope>-<os>-<arch>-<hash(Cargo.Bazel.lock, MODULE.bazel, rust-toolchain.toml)>` |
|
||||
|
||||
The remote endpoint resolves **only inside the cluster** (see `infra/bazel-remote/` and `infra/docs/04-arc-and-caching.md` §5); GitHub-hosted runners never talk to it. The `scope` input separates disk-cache keys per target set (`linux-x64-pair`, `release-<target_id>`, …) so jobs don't evict each other's entries.
|
||||
|
||||
### `bazel-natives` action (`.github/actions/bazel-natives`)
|
||||
|
||||
Thin composite: `bazel-cache` (with `cache-scope`) → `OMP_BAZEL_RC=<rc> bun scripts/bazel-natives.ts <targets> --dest <dest>`. Every TS test job uses it with `targets: linux-x64-baseline linux-x64-modern`, `cache-scope: linux-x64-pair`.
|
||||
|
||||
### `release_binary`
|
||||
|
||||
Release runners are GitHub-hosted and build addons **inline** (disk-cache mode — repeat releases with unchanged Rust are mostly local cache hits): the `bazel-natives` action runs with the matrix's `native_targets` (`linux-x64-baseline linux-x64-modern`, `linux-musl-x64-baseline`, `linux-arm64`, `linux-musl-arm64`, `darwin-all` on both mac runners, `win32-x64-baseline` cross-built from `ubuntu-22.04`) and `cache-scope: release-<target_id>`, then `bun run ci:release:build-binaries` embeds and compiles.
|
||||
|
||||
## Debugging playbook
|
||||
|
||||
### Where things land / how to inspect
|
||||
|
||||
```bash
|
||||
# Outputs (workspace-relative): bazel-bin/natives-<target>/pi_natives.<...>.node
|
||||
bazelisk cquery --output=files //:natives-linux-x64-baseline
|
||||
|
||||
# What actions/flags a target produces (add the same --config flags as the build):
|
||||
bazelisk aquery 'outputs(".*\.node", deps(//:natives-linux-arm64))'
|
||||
bazelisk aquery 'mnemonic("Rustc", deps(//crates/pi-natives:pi_natives))'
|
||||
|
||||
# Which toolchain resolved (e.g. confirm @msvc_cc, not host cc, for win32):
|
||||
bazelisk cquery 'deps(//:natives-win32-x64-baseline)' | grep msvc_cc
|
||||
|
||||
# Keep the sandbox dir + print the full command line of a failing action:
|
||||
bazelisk build --sandbox_debug --verbose_failures //:natives-<t>
|
||||
|
||||
# Analyze without building (cheap cross-target sanity check):
|
||||
bazelisk build --nobuild //:natives-win32-x64-baseline
|
||||
```
|
||||
|
||||
`scripts/bazel-natives.ts` streams bazel stderr live and repeats a 40-line tail on failure; when its cquery step fails it falls back to the `bazel-bin` path convention.
|
||||
|
||||
### Common failure classes (seen during bring-up — fixes already in tree, cite when they resurface)
|
||||
|
||||
| Symptom | Cause | Fix (in tree) |
|
||||
| --- | --- | --- |
|
||||
| musl build "succeeds" but emits no `.node` | musl defaults to `+crt-static`; rustc silently emits no cdylib | `-Ctarget-feature=-crt-static` select in `crates/pi-natives/BUILD.bazel` |
|
||||
| opus/cmake `try_compile` fails linking UBSan runtime | zig cc enables UBSan by default; cmake's test exe links with the raw wrapper (no toolchain features) | `CFLAGS=-fno-sanitize=undefined` in the `audiopus_sys` annotation (`MODULE.bazel`) |
|
||||
| `tree-sitter-just` scanner.c `#error` under opt | scanner hard-errors when `NDEBUG` is set (opt-mode cc default) | `CFLAGS=-UNDEBUG` annotation (cc-rs appends env CFLAGS last, so `-U` wins) |
|
||||
| rstest macro: "Cargo.toml not found" in a vendored test | rstest verifies `Cargo.toml` exists in the manifest dir | `compile_data = ["Cargo.toml"]` on the `rust_test` (see `crates/vendor/uu-tail/BUILD.bazel`) |
|
||||
| vendored tests fail on bare `test_data/...` paths / symlink into srcs | tests assume cargo's cwd, incompatible with runfiles execution | `tags = ["manual"]` (e.g. `//crates/vendor/uu-find:uu-find_test`); run via `cargo nextest` when touching the fork; hermetic sibling test covers the contract |
|
||||
| blake3 msvc: `ml64.exe` not found | cc-rs resolves MASM from build-script PATH on non-windows hosts | `bin/ml64.exe → llvm-ml -m64` shim in `@msvc_cc`, prepended via the `blake3` annotation PATH |
|
||||
| audiopus_sys msvc: cmake demands VS generator / rc+mt tools; `try_compile` wants `msvcrtd.lib` | cross cmake on linux/mac hosts; Debug config → `/MDd` which the lean xwin splat lacks | `CMAKE_GENERATOR_x86_64_pc_windows_msvc=Ninja` + `@msvc_cc`'s `toolchain.cmake` (`CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc`) pinning wrappers + Release try-compile + `/MD` |
|
||||
| win32 link oddities generally | — | read `bazel/toolchains/msvc/NOTES.md` first: wrapper self-location, `lld-link` flavor/driver-link behavior, `LIB`, `/MD` CRT choice, xwin splat caveats |
|
||||
| `rust_test(crate = ...)` "can't find crate" at macro expansion | rmeta-only pipelined deps break macro_rules re-export harness compiles | rust pipelined_compilation stays OFF (`.bazelrc` note) |
|
||||
| build script can't find cmake/ninja | `--incompatible_strict_action_env` — no host env leaks | explicit `PATH` in the crate annotation (`MODULE.bazel`), not host env |
|
||||
|
||||
### Cache behavior
|
||||
|
||||
- **omp-kata (push/main, release_binary is not here):** read-write gRPC to in-cluster bazel-remote (`grpcs://bazel-remote.bazel-cache.svc.cluster.local:9092`, TLS via the committed `infra/bazel-remote/ca.crt`, htpasswd user `ci`). Expect `remote cache hit` counts in the build summary; the `rust` job's `//:natives-linux-all` warm build on main pushes is what seeds it. `--remote_local_fallback` + retries mean a cache outage degrades to a local build, never a failure.
|
||||
- **GitHub-hosted (PRs, macOS, releases):** no remote cache at all — `--disk_cache`/`--repository_cache` persisted by `actions/cache`, keyed on `(scope, os, arch, hash(Cargo.Bazel.lock, MODULE.bazel, rust-toolchain.toml))` with a prefix restore key. A lockfile/module change starts from the nearest previous entry.
|
||||
- **msvc repos:** the ~2 GiB LLVM download is sha256-pinned and repository-cache backed; the ~1 GiB xwin CRT/SDK splat is fetched from the Microsoft CDN inside the repo rule and is **not** repo-cache backed — a cold output base re-downloads it. Microsoft advances the VS channel payload over time, so remote-cache hit rates for win32 actions degrade gracefully after an MS bump (same property the previous cross toolchain had). Win32 link actions also don't share cache entries across host OSes (linux vs mac clang binaries).
|
||||
- Server-side operations (deploy, TLS/auth, egress, poisoning boundary): `infra/docs/04-arc-and-caching.md` §5.
|
||||
|
||||
## Target/variant model and naming conventions
|
||||
|
||||
@@ -68,12 +236,12 @@ Both build and runtime use platform tag:
|
||||
|
||||
## Variant model (x64 only)
|
||||
|
||||
x64 supports CPU variants:
|
||||
x64 supports CPU variants, encoded as `//bazel/variants` constraint values on the platform (baseline → `-Ctarget-cpu=x86-64-v2`, modern → `x86-64-v3`):
|
||||
|
||||
- `modern` (AVX2-capable path)
|
||||
- `baseline` (fallback)
|
||||
|
||||
Non-x64 uses a single default artifact with no variant suffix.
|
||||
Non-x64 uses a single default artifact with no variant suffix. There is no build-time variant *switch*: each variant is its own `//:natives-*` target, and the `host` pseudo-target picks modern vs baseline via AVX2 detection.
|
||||
|
||||
### Output filenames
|
||||
|
||||
@@ -82,50 +250,14 @@ Non-x64 uses a single default artifact with no variant suffix.
|
||||
|
||||
Runtime x64 candidate order also includes the unsuffixed default filename after the selected variant candidates.
|
||||
|
||||
## Environment flags and build options
|
||||
|
||||
## Runtime flags
|
||||
|
||||
- `PI_NATIVE_VARIANT`: x64 runtime override; valid values are `modern` and `baseline`.
|
||||
- `PI_COMPILED`: legacy compiled-mode signal. A populated embedded-addon manifest is also a compiled-mode signal; compiled release builds additionally define `process.env.PI_COMPILED="true"` during `bun build --compile`.
|
||||
|
||||
## Build-time flags/options
|
||||
## Embed lifecycle (`embed-native.ts`)
|
||||
|
||||
- `CROSS_TARGET`: passed to napi-rs as `--target <CROSS_TARGET>`.
|
||||
- `TARGET_PLATFORM`: override output platform tag naming.
|
||||
- `TARGET_ARCH`: override output arch naming.
|
||||
- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy.
|
||||
- `CARGO_TARGET_DIR`: respected if set; otherwise the default `target/` dir is used so `Swatinem/rust-cache` can cache cleanly.
|
||||
- `RUSTFLAGS`:
|
||||
- if unset and not cross-compiling, script sets:
|
||||
- modern: `-C target-cpu=x86-64-v3`
|
||||
- baseline: `-C target-cpu=x86-64-v2`
|
||||
- non-x64 / no variant: `-C target-cpu=native`
|
||||
- if already set, script does not override.
|
||||
|
||||
## Build state/lifecycle transitions
|
||||
|
||||
### Build lifecycle (`build-native.ts`)
|
||||
|
||||
1. **Init**: parse env, resolve target tuple, cross/local mode, profile label.
|
||||
2. **Variant resolve**:
|
||||
- non-x64 → no variant;
|
||||
- x64 + `TARGET_VARIANT` → explicit variant;
|
||||
- x64 cross-build without `TARGET_VARIANT` → hard error;
|
||||
- x64 local build without override → detect host AVX2.
|
||||
3. **CPU policy**: set `RUSTFLAGS` for the resolved variant unless the caller already provided one.
|
||||
4. **Compile**: run napi-rs against `crates/pi-natives` into an isolated output directory.
|
||||
5. **Locate artifact**: accept the canonical filename or a single napi-rs-generated `pi_natives.<platform>-<arch>*.node` candidate.
|
||||
6. **Install**: copy/rename addon into `packages/natives/native`.
|
||||
7. **Install generated declarations**: copy `index.d.ts`.
|
||||
8. **Patch exports/enums**: regenerate explicit ESM exports and enum runtime objects.
|
||||
9. **Cleanup**: remove the temporary build output directory.
|
||||
|
||||
Failure exits have explicit error text for invalid variants, failed napi build, missing/multiple output artifacts, generated binding install failure, stripped CI ELF artifacts that still contain forbidden symbol/string-table sections, and install/rename failure.
|
||||
|
||||
### Embed lifecycle (`embed-native.ts`)
|
||||
|
||||
1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values.
|
||||
1. **Init**: compute the platform tag (host values, overridable by the release packaging script for cross-target archives).
|
||||
2. **Candidate set**:
|
||||
- x64 looks for `modern` and `baseline` files;
|
||||
- non-x64 looks for one default file.
|
||||
@@ -143,7 +275,7 @@ Typical local loop:
|
||||
|
||||
1. Build addon: `bun --cwd=packages/natives run build`.
|
||||
2. Loader resolves platform npm leaf-package candidates (`@oh-my-pi/pi-natives-<platform>-<arch>`, when resolvable), then package-local `native/` and executable-dir fallback candidates.
|
||||
3. Generated declarations in `native/index.d.ts` describe the public TS API.
|
||||
3. Generated declarations in `native/index.d.ts` describe the public TS API (regenerate with `build:bindings` only when the Rust API surface changes).
|
||||
|
||||
## Shipped/compiled binary workflow
|
||||
|
||||
@@ -176,14 +308,12 @@ Generated declarations currently include exports from these Rust modules:
|
||||
|
||||
## Build-time failures
|
||||
|
||||
- Invalid variant configuration:
|
||||
- `TARGET_VARIANT` set on non-x64 → immediate error.
|
||||
- unsupported `TARGET_VARIANT` value → immediate error.
|
||||
- x64 cross-build without explicit `TARGET_VARIANT` → immediate error.
|
||||
- napi-rs build failure: script surfaces non-zero exit and stderr.
|
||||
- Artifact not found or ambiguous: script prints expected/candidate filenames and output directory contents.
|
||||
- Install failure: explicit message; Windows includes locked-file hint.
|
||||
- Generated binding install failure: explicit source/destination message.
|
||||
- Bazel analysis/compile failure: `scripts/bazel-natives.ts` surfaces the exit code plus a stderr tail; re-run the printed `bazel build` line directly (add `--verbose_failures`, `--sandbox_debug`) to iterate.
|
||||
- Unknown target name: the driver errors with the full known-target list (`//:natives-*` names + `host`/`linux-all`/`darwin-all`).
|
||||
- No `.node` outputs located after a successful build: driver exits 1 (check `bazel cquery --output=files` manually).
|
||||
- Basename collision (gnu + musl in one invocation): driver refuses to install and names both sources — split into separate `--dest` dirs.
|
||||
- Stale `Cargo.Bazel.lock` after a `Cargo.{toml,lock}` change: run `bun run bazel:repin`.
|
||||
- `build:bindings` (napi) failure: script surfaces non-zero exit and stderr; artifact builds are unaffected (Bazel never runs the napi CLI).
|
||||
|
||||
## Runtime loader failures (`native/loader-state.js`)
|
||||
|
||||
@@ -196,22 +326,30 @@ Generated declarations currently include exports from these Rust modules:
|
||||
|
||||
| Symptom | Likely cause | Verify | Fix |
|
||||
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| `Cannot find module` or dynamic library load error for every candidate | Missing release artifact, wrong platform tag, or stale compiled cache | Inspect loader error list and `packages/natives/native` filenames | Build correct target/variant; delete stale cache for the package version |
|
||||
| `Cannot find module` or dynamic library load error for every candidate | Missing release artifact, wrong platform tag, or stale compiled cache | Inspect loader error list and `packages/natives/native` filenames | Build correct target (`bun scripts/bazel-natives.ts <t> --dest packages/natives/native`); delete stale cache for the package version |
|
||||
| Export is missing at runtime but present in TypeScript | Stale `.node` loaded, generated declarations newer than binary, or Rust export not compiled | Require the actual candidate and inspect `Object.keys(mod)` | Rebuild native package and remove stale candidate/cache paths |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or modern file unavailable | Check env and filenames in `native/` | Build modern variant (`TARGET_VARIANT=modern ... build`) and ship it |
|
||||
| Cross-build produces wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing x64 variant | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or modern file unavailable | Check env and filenames in `native/` | Build and ship the modern target (`bun scripts/bazel-natives.ts linux-x64-modern --dest packages/natives/native`) |
|
||||
| gnu addon overwritten by musl (or vice versa) | Both built into one dest — they share canonical basenames by design | Compare `bazel-bin/natives-<t>/` sources vs installed file | Separate invocations with separate `--dest` dirs (release matrix already does this) |
|
||||
| Compiled binary fails after upgrade | Stale extracted cache, embedded archive mismatch, or embedded manifest version mismatch | Inspect `<getNativesDir()>/<version>` and loader error list | Delete versioned cache for the package version; regenerate embedded archive/manifest during packaging |
|
||||
| `gen:native` fails with `No native addons found` | Required platform artifact was not built before embedding | Check expected list in error text | Build at least one expected artifact for the target, then rerun `gen:native` |
|
||||
|
||||
## Operational commands
|
||||
|
||||
```bash
|
||||
# Release artifact for current host
|
||||
# Addon for the current host, installed into packages/natives/native/
|
||||
bun --cwd=packages/natives run build
|
||||
|
||||
# Build explicit x64 variants
|
||||
TARGET_VARIANT=modern bun --cwd=packages/natives run build
|
||||
TARGET_VARIANT=baseline bun --cwd=packages/natives run build
|
||||
# Explicit targets (x64 variants are separate targets, not env switches)
|
||||
bun scripts/bazel-natives.ts linux-x64-modern linux-x64-baseline --dest packages/natives/native
|
||||
|
||||
# Raw bazel (output: bazel-bin/natives-<t>/pi_natives.<...>.node)
|
||||
bazelisk build //:natives-darwin-arm64
|
||||
|
||||
# Refresh Cargo.Bazel.lock after Cargo.{toml,lock} or annotation changes
|
||||
bun run bazel:repin
|
||||
|
||||
# Regenerate TS typedefs + enum exports (napi CLI, only on Rust API changes)
|
||||
bun --cwd=packages/natives run build:bindings
|
||||
|
||||
# Generate embedded addon manifest from built native files
|
||||
bun run gen:native
|
||||
@@ -245,11 +383,11 @@ The key is `sha256` over `(path \t git-tree-hash \n)` pairs for the following in
|
||||
2. `Cargo.lock`
|
||||
3. `Cargo.toml`
|
||||
4. `rust-toolchain.toml`
|
||||
5. `packages/natives` (whole subtree — build script, `scripts/*`, package.json with napi config)
|
||||
5. `packages/natives` (whole subtree — build script, `scripts/*`, package.json)
|
||||
|
||||
Tree hashes come from one `git cat-file --batch-check` invocation against `HEAD`; paths missing from `HEAD` fold in as a fixed null hash so the key stays deterministic across repos that don't ship every input. The target-triple suffix matches the napi addon basename convention (`<platform>-<arch>` for non-x64, `<platform>-<arch>-<variant>` for x64). When `TARGET_VARIANT` is unset on an x64 host the variant component is `host` rather than autodetected — the key is stable on a given machine but a `modern`/`baseline` build with an explicit `TARGET_VARIANT` gets a different key.
|
||||
Tree hashes come from one `git cat-file --batch-check` invocation against `HEAD`; paths missing from `HEAD` fold in as a fixed null hash so the key stays deterministic across repos that don't ship every input. The target-triple suffix matches the addon basename convention (`<platform>-<arch>` for non-x64, `<platform>-<arch>-<variant>` for x64).
|
||||
|
||||
Anything outside this input set (Rust toolchain auto-installed delta, host glibc, env vars other than `TARGET_VARIANT`) is **not** in the key. If you need to invalidate after such a change, delete the cache directory by hand or bump one of the input files.
|
||||
Anything outside this input set (Bazel definition files like `MODULE.bazel`/`BUILD.bazel`/`Cargo.Bazel.lock`, host glibc, env vars) is **not** in the key. If you need to invalidate after such a change, delete the cache directory by hand or bump one of the input files.
|
||||
|
||||
### Layout and ownership
|
||||
|
||||
@@ -261,7 +399,7 @@ Anything outside this input set (Rust toolchain auto-installed delta, host glibc
|
||||
|
||||
### Populate and capture semantics
|
||||
|
||||
- **Populate** (workspace ← cache) runs inside `ensure_workspace`. On a key hit the `.node` is **hardlinked** into the workspace (zero-copy, shared inode); the companion `index.d.ts` / `index.js` / `embedded-addon.js` are **copied** (independent inodes) because the napi build's `installGeneratedBindings` and `gen-enums.ts` rewrite those files via `open(..., 'w')` — an in-place truncate that would otherwise propagate through a hardlink and corrupt the cache. Cross-device hardlink failures (`EXDEV`) fall back to copy.
|
||||
- **Populate** (workspace ← cache) runs inside `ensure_workspace`. On a key hit the `.node` is **hardlinked** into the workspace (zero-copy, shared inode); the companion `index.d.ts` / `index.js` / `embedded-addon.js` are **copied** (independent inodes) because the bindings regeneration flow (`build-bindings.ts`'s `installGeneratedBindings` and `gen-enums.ts`) rewrites those files via `open(..., 'w')` — an in-place truncate that would otherwise propagate through a hardlink and corrupt the cache. Cross-device hardlink failures (`EXDEV`) fall back to copy.
|
||||
- **Capture** (cache ← workspace) runs from the post-task success path when the build produced a complete artifact set. Capture uses **copy**, not hardlink: hardlinking a slot-owned workspace file would preserve slot UID ownership on the cached inode and defeat the shared-group model. Copying creates a fresh root-owned, `gid=omp` inode via the setgid cache root. Capture is idempotent under the per-repo flock: a concurrent capture for the same key returns the existing entry.
|
||||
|
||||
### Garbage collection
|
||||
|
||||
Reference in New Issue
Block a user