docs: added comprehensive documentation for @oh-my-pi/pi-natives architecture
- Added comprehensive documentation for @oh-my-pi/pi-natives architecture covering three-layer design (TypeScript wrapper, addon loading/validation, Rust N-API module). - Added deep-dive documentation for addon loading runtime including platform support, CPU variant selection, embedded addon extraction lifecycle, and validation contracts. - Added binding contract documentation defining TypeScript-side API surface, wrapper behavior, and N-API addon validation with maintainer checklist. - Added subsystem deep-dives for text/search pipeline (grep, glob, fs_cache, ANSI utilities), shell/PTY/process execution, task cancellation patterns, and media/system utilities.
This commit is contained in:
can1357
@@ -0,0 +1,237 @@
|
||||
# Natives Build, Release, and Debugging Runbook
|
||||
|
||||
This runbook describes how the `@oh-my-pi/pi-natives` build pipeline produces `.node` addons, how compiled distributions load them, and how to debug loader/build failures.
|
||||
|
||||
It follows the architecture terms from `docs/natives-architecture.md`:
|
||||
- **build-time artifact production** (`scripts/build-native.ts`)
|
||||
- **embedded addon manifest generation** (`scripts/embed-native.ts`)
|
||||
- **runtime addon loading + validation gate** (`src/native.ts`)
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/scripts/build-native.ts`
|
||||
- `packages/natives/scripts/embed-native.ts`
|
||||
- `packages/natives/package.json`
|
||||
- `packages/natives/src/native.ts`
|
||||
- `crates/pi-natives/Cargo.toml`
|
||||
|
||||
## Build pipeline overview
|
||||
|
||||
### 1) Build entrypoints
|
||||
|
||||
`packages/natives/package.json` scripts:
|
||||
|
||||
- `bun scripts/build-native.ts` (`build:native`) → release build
|
||||
- `bun scripts/build-native.ts --dev` (`dev:native`) → debug/dev build
|
||||
- `bun scripts/embed-native.ts` (`embed:native`) → generate `src/embedded-addon.ts` from built files
|
||||
|
||||
### 2) Rust artifact build
|
||||
|
||||
`build-native.ts` runs Cargo in `crates/pi-natives`:
|
||||
|
||||
- base command: `cargo build`
|
||||
- release mode adds `--release` unless `--dev` is passed
|
||||
- cross target adds `--target <CROSS_TARGET>`
|
||||
|
||||
`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`, so Cargo emits a shared library (`.so`/`.dylib`/`.dll`) that is then copied/renamed to a `.node` addon filename.
|
||||
|
||||
### 3) Artifact discovery and install
|
||||
|
||||
After Cargo completes, `build-native.ts` scans candidate output directories in order:
|
||||
|
||||
1. `${CARGO_TARGET_DIR}` (if set)
|
||||
2. `<repo>/target`
|
||||
3. `crates/pi-natives/target`
|
||||
|
||||
For each root it checks profile directories:
|
||||
- cross build: `<root>/<crossTarget>/<profile>` then `<root>/<profile>`
|
||||
- native build: `<root>/<profile>`
|
||||
|
||||
Then it looks for one of:
|
||||
- `libpi_natives.so`
|
||||
- `libpi_natives.dylib`
|
||||
- `pi_natives.dll`
|
||||
- `libpi_natives.dll`
|
||||
|
||||
When found, it atomically installs into `packages/natives/native/` with temp-file + rename semantics (Windows fallback handles locked DLL replacement failures explicitly).
|
||||
|
||||
## Target/variant model and naming conventions
|
||||
|
||||
## Platform tag
|
||||
|
||||
Both build and runtime use platform tag:
|
||||
|
||||
`<platform>-<arch>` (example: `darwin-arm64`, `linux-x64`)
|
||||
|
||||
## Variant model (x64 only)
|
||||
|
||||
x64 supports CPU variants:
|
||||
- `modern` (AVX2-capable path)
|
||||
- `baseline` (fallback)
|
||||
|
||||
Non-x64 uses a single default artifact (no variant suffix).
|
||||
|
||||
### Output filenames
|
||||
|
||||
Release builds:
|
||||
- x64: `pi_natives.<platform>-<arch>-modern.node` or `...-baseline.node`
|
||||
- non-x64: `pi_natives.<platform>-<arch>.node`
|
||||
|
||||
Dev build (`--dev`):
|
||||
- `pi_natives.dev.node`
|
||||
|
||||
Runtime loader candidate order in `native.ts`:
|
||||
- if `PI_DEV` is set: try `pi_natives.dev.node` first
|
||||
- then release candidates
|
||||
- compiled mode prepends extracted/cache candidates before package-local files
|
||||
|
||||
## Environment flags and build options
|
||||
|
||||
## Runtime flags
|
||||
|
||||
- `PI_DEV` (loader behavior): prefer dev addon candidates first
|
||||
- `PI_NATIVE_VARIANT` (loader behavior, x64 only): force `modern` or `baseline` selection at runtime
|
||||
- `PI_COMPILED` (loader behavior): enable compiled-binary candidate/extraction behavior
|
||||
|
||||
## Build-time flags/options
|
||||
|
||||
- `--dev` (script arg): build debug profile and emit `pi_natives.dev.node`
|
||||
- `CROSS_TARGET`: passed to Cargo `--target`
|
||||
- `TARGET_PLATFORM`: override output platform tag naming
|
||||
- `TARGET_ARCH`: override output arch naming
|
||||
- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy
|
||||
- `CARGO_TARGET_DIR`: additional root when searching Cargo outputs
|
||||
- `RUSTFLAGS`:
|
||||
- if unset and not cross-compiling, script sets:
|
||||
- modern: `-C target-cpu=x86-64-v3`
|
||||
- baseline: `-C target-cpu=x86-64-v2`
|
||||
- non-x64 / no variant: `-C target-cpu=native`
|
||||
- if already set, script does not override
|
||||
|
||||
## Build state/lifecycle transitions
|
||||
|
||||
### Build lifecycle (`build-native.ts`)
|
||||
|
||||
1. **Init**: parse args/env (`--dev`, target overrides, cross flags)
|
||||
2. **Variant resolve**:
|
||||
- non-x64 → no variant
|
||||
- x64 + `TARGET_VARIANT` → explicit variant
|
||||
- x64 cross-build without `TARGET_VARIANT` → hard error
|
||||
- x64 local build without override → detect host AVX2
|
||||
3. **Compile**: run Cargo with resolved profile/target
|
||||
4. **Locate artifact**: scan target roots/profile dirs/library names
|
||||
5. **Install**: copy + atomic rename into `packages/natives/native`
|
||||
6. **Complete**: output addon ready for loader candidates
|
||||
|
||||
Failure exits happen at any stage with explicit error text (invalid variant, failed cargo build, missing output library, install/rename failure).
|
||||
|
||||
### Embed lifecycle (`embed-native.ts`)
|
||||
|
||||
1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values
|
||||
2. **Candidate set**:
|
||||
- x64 expects both `modern` and `baseline`
|
||||
- non-x64 expects one default file
|
||||
3. **Validate availability** in `packages/natives/native`
|
||||
4. **Generate manifest** (`src/embedded-addon.ts`) with Bun `file` imports and package version
|
||||
5. **Runtime extraction ready** for compiled mode
|
||||
|
||||
`--reset` bypasses validation and writes a null manifest stub (`embeddedAddon = null`).
|
||||
|
||||
## Dev workflow vs shipped/compiled behavior
|
||||
|
||||
## Local development workflow
|
||||
|
||||
Typical local loop:
|
||||
|
||||
1. Build addon:
|
||||
- release: `bun --cwd=packages/natives run build:native`
|
||||
- debug: `bun --cwd=packages/natives run dev:native`
|
||||
2. Set `PI_DEV=1` when testing debug addon loading
|
||||
3. Loader in `native.ts` resolves package-local `native/` (and executable-dir fallback) candidates
|
||||
4. `validateNative` enforces export compatibility before wrappers use the binding
|
||||
|
||||
## Shipped/compiled binary workflow
|
||||
|
||||
In compiled mode (`PI_COMPILED` or Bun embedded markers):
|
||||
|
||||
1. Loader computes versioned cache dir: `<getNativesDir()>/<packageVersion>` (operationally `~/.omp/natives/<version>`)
|
||||
2. If embedded manifest matches current platform+version, loader may extract selected embedded file into that versioned dir
|
||||
3. Runtime candidate order includes:
|
||||
- versioned cache dir
|
||||
- legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere)
|
||||
- package/executable directories
|
||||
4. First successfully loaded addon still must pass `validateNative`
|
||||
|
||||
This is why packaging + runtime loader expectations must align: filenames, platform tags, and exported symbols must match what `native.ts` probes and validates.
|
||||
|
||||
## JS API ↔ Rust export mapping (validation gate subset)
|
||||
|
||||
`native.ts` requires these JS-visible exports to exist on the loaded addon. They map to Rust N-API exports in `crates/pi-natives/src`:
|
||||
|
||||
| JS name required by `validateNative` | Rust export declaration | Rust source file |
|
||||
| --- | --- | --- |
|
||||
| `glob` | `#[napi(js_name = "glob")] pub fn glob(...)` | `crates/pi-natives/src/glob.rs` |
|
||||
| `grep` | `#[napi(js_name = "grep")] pub fn grep(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `search` | `#[napi(js_name = "search")] pub fn search(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `highlightCode` | `#[napi(js_name = "highlightCode")] pub fn highlight_code(...)` | `crates/pi-natives/src/highlight.rs` |
|
||||
| `getSystemInfo` | `#[napi(js_name = "getSystemInfo")] pub fn get_system_info(...)` | `crates/pi-natives/src/system_info.rs` |
|
||||
| `getWorkProfile` | `#[napi] pub fn get_work_profile(...)` (camel-cased export) | `crates/pi-natives/src/prof.rs` |
|
||||
| `invalidateFsScanCache` | `#[napi(js_name = "invalidateFsScanCache")] pub fn invalidate_fs_scan_cache(...)` | `crates/pi-natives/src/fs_cache.rs` |
|
||||
|
||||
If any required symbol is missing, loader fails fast with a rebuild hint.
|
||||
|
||||
## Failure behavior and diagnostics
|
||||
|
||||
## Build-time failures
|
||||
|
||||
- Invalid variant configuration:
|
||||
- `TARGET_VARIANT` set on non-x64 → immediate error
|
||||
- x64 cross-build without explicit `TARGET_VARIANT` → immediate error
|
||||
- Cargo build failure:
|
||||
- script surfaces non-zero exit and stderr
|
||||
- Artifact not found:
|
||||
- script prints every checked profile directory
|
||||
- Install failure:
|
||||
- explicit message; Windows includes locked-file hint
|
||||
|
||||
## Runtime loader failures (`native.ts`)
|
||||
|
||||
- Unsupported platform tag:
|
||||
- throws with supported platform list
|
||||
- No candidate could load:
|
||||
- throws with full candidate error list and mode-specific remediation hints
|
||||
- Missing exports:
|
||||
- throws with exact missing symbol names and rebuild command
|
||||
- Embedded extraction problems:
|
||||
- extraction mkdir/write errors recorded and included in final diagnostics
|
||||
|
||||
## Troubleshooting matrix
|
||||
|
||||
| Symptom | Likely cause | Verify | Fix |
|
||||
| --- | --- | --- | --- |
|
||||
| `Native addon missing exports ... Missing: <name>` | Stale `.node` binary, Rust export name mismatch, or wrong binary loaded | Run with `PI_DEV=1` to see loaded path; inspect export list for that file | Rebuild `build:native`; ensure Rust `#[napi(js_name=...)]` matches JS name; remove stale cached/versioned files |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or only baseline file present | Check `PI_NATIVE_VARIANT`; inspect `native/` for `-modern` file | Build modern variant (`TARGET_VARIANT=modern ... build:native`) and ensure file is shipped |
|
||||
| Cross-build produces unusable/wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing `TARGET_VARIANT` for x64 | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` |
|
||||
| Compiled binary fails after upgrade | Stale extracted cache (`~/.omp/natives/<old-or-mismatched-version>`) or embedded manifest mismatch | Inspect versioned natives dir and loader error list | Delete versioned natives cache for the package version and rerun; regenerate embedded manifest during packaging |
|
||||
| Loader probes many paths and none work | Platform mismatch or missing release artifact in package `native/` | Check `platformTag` vs actual filename(s) | Ensure built filename exactly matches `pi_natives.<platform>-<arch>(-variant).node` convention and package includes `native/` |
|
||||
| `embed:native` fails with "Incomplete native addons" | Required variant files not built before embedding | Check expected vs found list in error text | Build required files first (x64: both modern+baseline; non-x64: default), then rerun `embed:native` |
|
||||
|
||||
## Operational commands
|
||||
|
||||
```bash
|
||||
# Release artifact for current host
|
||||
bun --cwd=packages/natives run build:native
|
||||
|
||||
# Debug artifact (load first when PI_DEV=1)
|
||||
bun --cwd=packages/natives run dev:native
|
||||
|
||||
# Build explicit x64 variants
|
||||
TARGET_VARIANT=modern bun --cwd=packages/natives run build:native
|
||||
TARGET_VARIANT=baseline bun --cwd=packages/natives run build:native
|
||||
|
||||
# Generate embedded addon manifest from built native files
|
||||
bun --cwd=packages/natives run embed:native
|
||||
|
||||
# Reset embedded manifest to null stub
|
||||
bun --cwd=packages/natives run embed:native -- --reset
|
||||
```
|
||||
Reference in New Issue
Block a user