docs: update docs
This commit is contained in:
can1357
@@ -1,18 +1,22 @@
|
||||
# Natives Build, Release, and Debugging Runbook
|
||||
|
||||
This runbook describes how the `@oh-my-pi/pi-natives` build pipeline produces `.node` addons, how compiled distributions load them, and how to debug loader/build failures.
|
||||
This runbook describes how `@oh-my-pi/pi-natives` produces `.node` addons, generated declarations, and compiled-binary embedded payloads, and how to debug loader/build failures.
|
||||
|
||||
It follows the architecture terms from `docs/natives-architecture.md`:
|
||||
|
||||
- **build-time artifact production** (`scripts/build-native.ts`)
|
||||
- **embedded addon manifest generation** (`scripts/embed-native.ts`)
|
||||
- **runtime addon loading + validation gate** (`src/native.ts`)
|
||||
- **runtime addon loading** (`native/index.js`, `native/loader-state.js`)
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/scripts/build-native.ts`
|
||||
- `packages/natives/scripts/embed-native.ts`
|
||||
- `packages/natives/scripts/gen-enums.ts`
|
||||
- `packages/natives/scripts/zig-safe-wrapper.ts`
|
||||
- `packages/natives/package.json`
|
||||
- `packages/natives/src/native.ts`
|
||||
- `packages/natives/native/index.js`
|
||||
- `packages/natives/native/loader-state.js`
|
||||
- `crates/pi-natives/Cargo.toml`
|
||||
|
||||
## Build pipeline overview
|
||||
@@ -21,37 +25,37 @@ It follows the architecture terms from `docs/natives-architecture.md`:
|
||||
|
||||
`packages/natives/package.json` scripts:
|
||||
|
||||
- `bun scripts/build-native.ts` (`build`) → release build
|
||||
- `bun scripts/embed-native.ts` (`embed:native`) → generate `src/embedded-addon.ts` from built files
|
||||
- `bun scripts/build-native.ts` (`build`) → N-API build, addon install, generated declarations install, enum export patch.
|
||||
- `bun scripts/embed-native.ts` (`embed:native`) → generate `native/embedded-addon.js` from built files.
|
||||
|
||||
### 2) Rust artifact build
|
||||
Root scripts include `build:native` as `bun --cwd=packages/natives run build`.
|
||||
|
||||
`build-native.ts` runs Cargo in `crates/pi-natives`:
|
||||
### 2) N-API/Rust artifact build
|
||||
|
||||
- base command: `cargo build --release`
|
||||
- cross target adds `--target <CROSS_TARGET>`
|
||||
`build-native.ts` invokes the `@napi-rs/cli` binary directly from `node_modules/.bin` with:
|
||||
|
||||
`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`, so Cargo emits a shared library (`.so`/`.dylib`/`.dll`) that is then copied/renamed to a `.node` addon filename.
|
||||
- `napi build`
|
||||
- `--manifest-path crates/pi-natives/Cargo.toml`
|
||||
- `--package-json-path packages/natives/package.json`
|
||||
- `--platform`
|
||||
- `--no-js`
|
||||
- `--dts index.d.ts`
|
||||
- `--profile local` for non-CI local native builds, otherwise `--profile ci`
|
||||
- optional `--target <CROSS_TARGET>`
|
||||
|
||||
### 3) Artifact discovery and install
|
||||
`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`; napi-rs emits `.node` artifacts plus generated `index.d.ts` in an isolated temporary output directory under `packages/natives/native/.build/`.
|
||||
|
||||
After Cargo completes, `build-native.ts` scans candidate output directories in order:
|
||||
### 3) Artifact install
|
||||
|
||||
1. `${CARGO_TARGET_DIR}` (if set)
|
||||
2. `<repo>/target`
|
||||
3. `crates/pi-natives/target`
|
||||
After napi-rs succeeds, `build-native.ts`:
|
||||
|
||||
For each root it checks profile directories:
|
||||
- cross build: `<root>/<crossTarget>/<profile>` then `<root>/<profile>`
|
||||
- native build: `<root>/<profile>`
|
||||
1. resolves the built addon in the isolated output directory;
|
||||
2. normalizes its name to `pi_natives.<platform>-<arch>(-variant).node` when needed;
|
||||
3. installs the addon into `packages/natives/native/` with temp-file + rename semantics;
|
||||
4. copies generated `index.js` and `index.d.ts` into `packages/natives/native/` when present;
|
||||
5. runs `generateEnumExports()` to append enum runtime objects to `native/index.js`.
|
||||
|
||||
Then it looks for one of:
|
||||
- `libpi_natives.so`
|
||||
- `libpi_natives.dylib`
|
||||
- `pi_natives.dll`
|
||||
- `libpi_natives.dll`
|
||||
|
||||
When found, it atomically installs into `packages/natives/native/` with temp-file + rename semantics (Windows fallback handles locked DLL replacement failures explicitly).
|
||||
Windows locked-DLL replacement failures are reported with an explicit close-running-processes hint.
|
||||
|
||||
## Target/variant model and naming conventions
|
||||
|
||||
@@ -59,75 +63,78 @@ When found, it atomically installs into `packages/natives/native/` with temp-fil
|
||||
|
||||
Both build and runtime use platform tag:
|
||||
|
||||
`<platform>-<arch>` (example: `darwin-arm64`, `linux-x64`)
|
||||
`<platform>-<arch>` (example: `darwin-arm64`, `linux-x64`).
|
||||
|
||||
## Variant model (x64 only)
|
||||
|
||||
x64 supports CPU variants:
|
||||
|
||||
- `modern` (AVX2-capable path)
|
||||
- `baseline` (fallback)
|
||||
|
||||
Non-x64 uses a single default artifact (no variant suffix).
|
||||
Non-x64 uses a single default artifact with no variant suffix.
|
||||
|
||||
### Output filenames
|
||||
|
||||
Release builds:
|
||||
- x64: `pi_natives.<platform>-<arch>-modern.node` or `...-baseline.node`
|
||||
- non-x64: `pi_natives.<platform>-<arch>.node`
|
||||
|
||||
Runtime loader candidate order in `native.ts`:
|
||||
- release candidates
|
||||
- compiled mode prepends extracted/cache candidates before package-local files
|
||||
Runtime x64 candidate order also includes the unsuffixed default filename after the selected variant candidates.
|
||||
|
||||
## Environment flags and build options
|
||||
|
||||
## Runtime flags
|
||||
|
||||
- `PI_NATIVE_VARIANT` (loader behavior, x64 only): force `modern` or `baseline` selection at runtime
|
||||
- `PI_COMPILED` (loader behavior): enable compiled-binary candidate/extraction behavior
|
||||
- `PI_NATIVE_VARIANT`: x64 runtime override; valid values are `modern` and `baseline`.
|
||||
- `PI_COMPILED`: legacy compiled-mode signal. A populated embedded-addon manifest is also a compiled-mode signal and is the authoritative signal for Bun standalone builds that do not preserve `process.env.PI_COMPILED`.
|
||||
|
||||
## Build-time flags/options
|
||||
|
||||
- `CROSS_TARGET`: passed to Cargo `--target`
|
||||
- `TARGET_PLATFORM`: override output platform tag naming
|
||||
- `TARGET_ARCH`: override output arch naming
|
||||
- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy
|
||||
- `CARGO_TARGET_DIR`: additional root when searching Cargo outputs
|
||||
- `CROSS_TARGET`: passed to napi-rs as `--target <CROSS_TARGET>`.
|
||||
- `TARGET_PLATFORM`: override output platform tag naming.
|
||||
- `TARGET_ARCH`: override output arch naming.
|
||||
- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy.
|
||||
- `CARGO_TARGET_DIR`: if set, respected; otherwise CI/cross builds use an isolated managed target directory under `target/napi-build/...`.
|
||||
- `RUSTFLAGS`:
|
||||
- if unset and not cross-compiling, script sets:
|
||||
- modern: `-C target-cpu=x86-64-v3`
|
||||
- baseline: `-C target-cpu=x86-64-v2`
|
||||
- non-x64 / no variant: `-C target-cpu=native`
|
||||
- if already set, script does not override
|
||||
- if already set, script does not override.
|
||||
- `ZIG`: optional real Zig path used when the host Zig CPU contract wrapper is enabled.
|
||||
- `PI_NATIVE_REAL_ZIG`, `PI_NATIVE_ZIG_TARGET`, `PI_NATIVE_ZIG_CPU`: set internally for `zig-safe-wrapper.ts` when building local x64 Linux/macOS artifacts with Zig available.
|
||||
|
||||
## Build state/lifecycle transitions
|
||||
|
||||
### Build lifecycle (`build-native.ts`)
|
||||
|
||||
1. **Init**: parse args/env (target overrides, cross flags)
|
||||
1. **Init**: parse env, resolve target tuple, cross/local mode, profile label.
|
||||
2. **Variant resolve**:
|
||||
- non-x64 → no variant
|
||||
- x64 + `TARGET_VARIANT` → explicit variant
|
||||
- x64 cross-build without `TARGET_VARIANT` → hard error
|
||||
- x64 local build without override → detect host AVX2
|
||||
3. **Compile**: run Cargo with resolved profile/target
|
||||
4. **Locate artifact**: scan target roots/profile dirs/library names
|
||||
5. **Install**: copy + atomic rename into `packages/natives/native`
|
||||
6. **Complete**: output addon ready for loader candidates
|
||||
- non-x64 → no variant;
|
||||
- x64 + `TARGET_VARIANT` → explicit variant;
|
||||
- x64 cross-build without `TARGET_VARIANT` → hard error;
|
||||
- x64 local build without override → detect host AVX2.
|
||||
3. **CPU policy**: set `RUSTFLAGS` if allowed; optionally route Zig through `zig-safe-wrapper.ts` to avoid leaking newer host CPU instructions into x64 artifacts.
|
||||
4. **Compile**: run napi-rs against `crates/pi-natives` into an isolated output directory.
|
||||
5. **Locate artifact**: accept the canonical filename or a single napi-rs-generated `pi_natives.<platform>-<arch>*.node` candidate.
|
||||
6. **Install**: copy/rename addon into `packages/natives/native`.
|
||||
7. **Install generated bindings**: copy `index.js`/`index.d.ts` if needed.
|
||||
8. **Patch enums**: append generated enum runtime exports.
|
||||
9. **Cleanup**: remove the temporary build output directory.
|
||||
|
||||
Failure exits happen at any stage with explicit error text (invalid variant, failed cargo build, missing output library, install/rename failure).
|
||||
Failure exits have explicit error text for invalid variants, failed napi build, missing/multiple output artifacts, generated binding install failure, and install/rename failure.
|
||||
|
||||
### Embed lifecycle (`embed-native.ts`)
|
||||
|
||||
1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values
|
||||
1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values.
|
||||
2. **Candidate set**:
|
||||
- x64 expects both `modern` and `baseline`
|
||||
- non-x64 expects one default file
|
||||
3. **Validate availability** in `packages/natives/native`
|
||||
4. **Generate manifest** (`src/embedded-addon.ts`) with Bun `file` imports and package version
|
||||
5. **Runtime extraction ready** for compiled mode
|
||||
- x64 looks for `modern` and `baseline` files;
|
||||
- non-x64 looks for one default file.
|
||||
3. **Validate availability**: at least one expected file must exist in `packages/natives/native`.
|
||||
4. **Generate manifest** (`native/embedded-addon.js`) with Bun `file` imports and package version.
|
||||
5. **Runtime extraction ready** for compiled mode.
|
||||
|
||||
`--reset` bypasses validation and writes a null manifest stub (`embeddedAddon = null`).
|
||||
`--reset` writes the null manifest stub (`embeddedAddon = null`) without validating addon availability.
|
||||
|
||||
## Dev workflow vs shipped/compiled behavior
|
||||
|
||||
@@ -135,75 +142,65 @@ Failure exits happen at any stage with explicit error text (invalid variant, fai
|
||||
|
||||
Typical local loop:
|
||||
|
||||
1. Build addon: `bun --cwd=packages/natives run build`
|
||||
2. Loader in `native.ts` resolves package-local `native/` (and executable-dir fallback) candidates
|
||||
3. `validateNative` enforces export compatibility before wrappers use the binding
|
||||
1. Build addon: `bun --cwd=packages/natives run build`.
|
||||
2. Loader resolves package-local `native/` candidates, then executable-dir fallback candidates.
|
||||
3. Generated declarations in `native/index.d.ts` describe the public TS API.
|
||||
|
||||
## Shipped/compiled binary workflow
|
||||
|
||||
In compiled mode (`PI_COMPILED` or Bun embedded markers):
|
||||
In compiled mode (`PI_COMPILED`, Bun embedded URL markers, or populated embedded manifest):
|
||||
|
||||
1. Loader computes versioned cache dir: `<getNativesDir()>/<packageVersion>` (operationally `~/.omp/natives/<version>`)
|
||||
2. If embedded manifest matches current platform+version, loader may extract selected embedded file into that versioned dir
|
||||
1. Loader computes versioned cache dir: `<getNativesDir()>/<packageVersion>`.
|
||||
2. If embedded manifest matches current platform+version, loader may extract the selected embedded file into that versioned dir.
|
||||
3. Runtime candidate order includes:
|
||||
- versioned cache dir
|
||||
- legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere)
|
||||
- package/executable directories
|
||||
4. First successfully loaded addon still must pass `validateNative`
|
||||
- versioned cache dir,
|
||||
- legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere),
|
||||
- package/executable directories.
|
||||
4. First successfully loaded addon is returned.
|
||||
|
||||
This is why packaging + runtime loader expectations must align: filenames, platform tags, and exported symbols must match what `native.ts` probes and validates.
|
||||
This is why packaging + runtime loader expectations must align: filenames, platform tags, CPU variants, and embedded manifest version must match what `native/index.js` probes.
|
||||
|
||||
## JS API ↔ Rust export mapping (validation gate subset)
|
||||
## JS API ↔ Rust export mapping (build sanity subset)
|
||||
|
||||
`native.ts` requires these JS-visible exports to exist on the loaded addon. They map to Rust N-API exports in `crates/pi-natives/src`:
|
||||
Generated declarations currently include exports from these Rust modules:
|
||||
|
||||
| JS name required by `validateNative` | Rust export declaration | Rust source file |
|
||||
| --- | --- | --- |
|
||||
| `glob` | `#[napi] pub fn glob(...)` | `crates/pi-natives/src/glob.rs` |
|
||||
| `grep` | `#[napi] pub fn grep(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `search` | `#[napi] pub fn search(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `highlightCode` | `#[napi] pub fn highlight_code(...)` | `crates/pi-natives/src/highlight.rs` |
|
||||
| `getSystemInfo` | `#[napi] pub fn get_system_info(...)` | `crates/pi-natives/src/system_info.rs` |
|
||||
| `getWorkProfile` | `#[napi] pub fn get_work_profile(...)` (camel-cased export) | `crates/pi-natives/src/prof.rs` |
|
||||
| `invalidateFsScanCache` | `#[napi] pub fn invalidate_fs_scan_cache(...)` | `crates/pi-natives/src/fs_cache.rs` |
|
||||
|
||||
If any required symbol is missing, loader fails fast with a rebuild hint.
|
||||
| Area | Representative JS exports | Rust source |
|
||||
| ---------------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| Search | `grep`, `search`, `hasMatch`, `fuzzyFind`, `glob`, `invalidateFsScanCache` | `grep.rs`, `fd.rs`, `glob.rs`, `fs_cache.rs` |
|
||||
| AST | `astGrep`, `astEdit` | `ast.rs` |
|
||||
| Text/highlight/tokens | `visibleWidth`, `truncateToWidth`, `highlightCode`, `countTokens` | `text.rs`, `highlight.rs`, `tokens.rs` |
|
||||
| Shell/PTY/process/keys | `executeShell`, `Shell`, `PtySession`, `killTree`, `parseKey` | `shell.rs`, `pty.rs`, `ps.rs`, `keys.rs` |
|
||||
| Media/system | `PhotonImage`, `encodeSixel`, clipboard, macOS appearance/power, `getWorkProfile`, ProjFS helpers | `image.rs`, `clipboard.rs`, `appearance.rs`, `power.rs`, `prof.rs`, `projfs_overlay.rs` |
|
||||
|
||||
## Failure behavior and diagnostics
|
||||
|
||||
## Build-time failures
|
||||
|
||||
- Invalid variant configuration:
|
||||
- `TARGET_VARIANT` set on non-x64 → immediate error
|
||||
- x64 cross-build without explicit `TARGET_VARIANT` → immediate error
|
||||
- Cargo build failure:
|
||||
- script surfaces non-zero exit and stderr
|
||||
- Artifact not found:
|
||||
- script prints every checked profile directory
|
||||
- Install failure:
|
||||
- explicit message; Windows includes locked-file hint
|
||||
- `TARGET_VARIANT` set on non-x64 → immediate error.
|
||||
- unsupported `TARGET_VARIANT` value → immediate error.
|
||||
- x64 cross-build without explicit `TARGET_VARIANT` → immediate error.
|
||||
- napi-rs build failure: script surfaces non-zero exit and stderr.
|
||||
- Artifact not found or ambiguous: script prints expected/candidate filenames and output directory contents.
|
||||
- Install failure: explicit message; Windows includes locked-file hint.
|
||||
- Generated binding install failure: explicit source/destination message.
|
||||
|
||||
## Runtime loader failures (`native.ts`)
|
||||
## Runtime loader failures (`native/index.js`)
|
||||
|
||||
- Unsupported platform tag:
|
||||
- throws with supported platform list
|
||||
- No candidate could load:
|
||||
- throws with full candidate error list and mode-specific remediation hints
|
||||
- Missing exports:
|
||||
- throws with exact missing symbol names and rebuild command
|
||||
- Embedded extraction problems:
|
||||
- extraction mkdir/write errors recorded and included in final diagnostics
|
||||
- Unsupported platform tag: throws with supported platform list after probing fails.
|
||||
- No candidate could load: throws with full candidate error list and mode-specific remediation hints.
|
||||
- Embedded extraction problems: extraction mkdir/write errors are recorded and included in final diagnostics if load fails.
|
||||
|
||||
## Troubleshooting matrix
|
||||
|
||||
| Symptom | Likely cause | Verify | Fix |
|
||||
| --- | --- | --- | --- |
|
||||
| `Native addon missing exports ... Missing: <name>` | Stale `.node` binary, Rust export name mismatch, or wrong binary loaded | Inspect export list for the binary | Rebuild `build`; ensure Rust `#[napi]` export name (or explicit alias when needed) matches JS key; remove stale cached/versioned files |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or only baseline file present | Check `PI_NATIVE_VARIANT`; inspect `native/` for `-modern` file | Build modern variant (`TARGET_VARIANT=modern ... build`) and ensure file is shipped |
|
||||
| Cross-build produces unusable/wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing `TARGET_VARIANT` for x64 | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` |
|
||||
| Compiled binary fails after upgrade | Stale extracted cache (`~/.omp/natives/<old-or-mismatched-version>`) or embedded manifest mismatch | Inspect versioned natives dir and loader error list | Delete versioned natives cache for the package version and rerun; regenerate embedded manifest during packaging |
|
||||
| Loader probes many paths and none work | Platform mismatch or missing release artifact in package `native/` | Check `platformTag` vs actual filename(s) | Ensure built filename exactly matches `pi_natives.<platform>-<arch>(-variant).node` convention and package includes `native/` |
|
||||
| `embed:native` fails with "Incomplete native addons" | Required variant files not built before embedding | Check expected vs found list in error text | Build required files first (x64: both modern+baseline; non-x64: default), then rerun `embed:native` |
|
||||
| Symptom | Likely cause | Verify | Fix |
|
||||
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `Cannot find module` or dynamic library load error for every candidate | Missing release artifact, wrong platform tag, or stale compiled cache | Inspect loader error list and `packages/natives/native` filenames | Build correct target/variant; delete stale cache for the package version |
|
||||
| Export is missing at runtime but present in TypeScript | Stale `.node` loaded, generated declarations newer than binary, or Rust export not compiled | Require the actual candidate and inspect `Object.keys(mod)` | Rebuild native package and remove stale candidate/cache paths |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or modern file unavailable | Check env and filenames in `native/` | Build modern variant (`TARGET_VARIANT=modern ... build`) and ship it |
|
||||
| Cross-build produces wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing x64 variant | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` |
|
||||
| Compiled binary fails after upgrade | Stale extracted cache or embedded manifest version mismatch | Inspect `<getNativesDir()>/<version>` and loader error list | Delete versioned cache for the package version; regenerate embedded manifest during packaging |
|
||||
| `embed:native` fails with `No native addons found` | Required platform artifact was not built before embedding | Check expected list in error text | Build at least one expected artifact for the target, then rerun `embed:native` |
|
||||
|
||||
## Operational commands
|
||||
|
||||
@@ -220,4 +217,4 @@ bun --cwd=packages/natives run embed:native
|
||||
|
||||
# Reset embedded manifest to null stub
|
||||
bun --cwd=packages/natives run embed:native -- --reset
|
||||
```
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user