oh-my-pi/docs/natives-architecture.md

# Natives Architecture

`@oh-my-pi/pi-natives` is a two-layer package around an ESM loader:

1. **ESM loader/package entrypoint** resolves and loads the correct `.node` addon with `createRequire`, validates the release sentinel outside workspace-dev loads, and re-exports generated classes/functions plus enum runtime objects as explicit named ESM exports.
2. **Rust N-API module layer** implements the exported functions/classes and emits the generated TypeScript declarations.

This document is the foundation for deeper module-level docs.

## Implementation files

- `packages/natives/native/index.js`
- `packages/natives/native/index.d.ts`
- `packages/natives/native/loader-state.js`
- `packages/natives/native/embedded-addon.js`
- `packages/natives/scripts/build-native.ts`
- `packages/natives/scripts/embed-native.ts`
- `packages/natives/scripts/gen-enums.ts`
- `packages/natives/package.json`
- `crates/pi-natives/src/lib.rs`

## Package entrypoint and public surface

`packages/natives/package.json` points at generated native artifacts:

- `main`: `./native/index.js`
- `types`: `./native/index.d.ts`
- `exports["."].types`: `./native/index.d.ts`
- `exports["."].import`: `./native/index.js`

There is no current `packages/natives/src` TypeScript wrapper layer. Consumers import functions/classes/enums directly from `@oh-my-pi/pi-natives`; the type contract is the generated `native/index.d.ts` plus the explicit named exports generated into `native/index.js` by `scripts/gen-enums.ts`.

Current capability groups in the generated API include:

- **Search/text/code primitives**: `grep`, `search`, `hasMatch`, `fuzzyFind`, `glob`, `astGrep`, `astEdit`, `blockRangeAt`, `summarizeCode`, text width/slicing/wrapping/sanitization, syntax highlighting, token counting.
- **Execution/process/terminal primitives**: `executeShell`, `Shell`, `PtySession`, `Process`, key parsing, bash fixups.
- **System/media/isolation/conversion primitives**: clipboard, SIXEL encoding, HTML-to-Markdown, macOS appearance/power helpers, work profiling, workspace scanning, isolation backend helpers (`iso*`).

## Loader layer

`packages/natives/native/index.js` is the package entrypoint; it calls `loadNative()` from `loader-state.js`, which owns runtime addon selection and optional embedded extraction.

### Candidate resolution model

- Platform tag is `${process.platform}-${process.arch}`.
- Supported tags are currently:
  - `linux-x64`
  - `linux-arm64`
  - `darwin-x64`
  - `darwin-arm64`
  - `win32-x64`
- x64 can use CPU variants:
  - `modern` (AVX2-capable)
  - `baseline` (fallback)
- Non-x64 uses the default filename without a variant suffix.

Filename strategy:

- Default: `pi_natives.<platform>-<arch>.node`
- x64 variant: `pi_natives.<platform>-<arch>-modern.node` or `...-baseline.node`
- x64 runtime fallback includes the unsuffixed default filename after variant candidates.

### Platform-specific variant detection

For x64, variant selection uses:

- Linux: `/proc/cpuinfo`
- macOS: `sysctl -n machdep.cpu.leaf7_features`, then `machdep.cpu.features`
- Windows: PowerShell check for `System.Runtime.Intrinsics.X86.Avx2`

`PI_NATIVE_VARIANT` can force `modern` or `baseline`; invalid values are ignored.

### Binary distribution and extraction model

The published `@oh-my-pi/pi-natives` package ships **only** the loader layer in `native/`: the ESM loader (`index.js`), generated declarations (`index.d.ts`), the `loader-state.js`/`.d.ts` helpers, and the embedded-addon manifest stub (`embedded-addon.js`). It carries no `.node` binaries.

Each platform's prebuilt `.node` is published as a separate optional-dependency leaf package — `@oh-my-pi/pi-natives-<platform>-<arch>`, one per supported tag — which the core lists in `optionalDependencies` at the lockstep version during publish. npm/bun install only the leaf whose `os`/`cpu` match the host. The working-tree package keeps built `.node` files under `native/` for local dev; the release-publish rewrite (`prepareNativeCorePackage` in `scripts/ci-release-publish.ts`) strips them from the core tarball, and the leaves are generated by `packages/natives/scripts/gen-npm-packages.ts` (`LEAF_TARGETS`). Adding a build target therefore requires a matching `LEAF_TARGETS` entry, or the binary never reaches npm users.

For compiled binaries, loader behavior is:

1. Check versioned user cache path: `<getNativesDir()>/<packageVersion>/...`.
2. Check legacy compiled-binary location:
   - Windows: `%LOCALAPPDATA%/omp` (fallback `%USERPROFILE%/AppData/Local/omp`)
   - non-Windows: `~/.local/bin`
3. Fall back to packaged `native/` and executable directory candidates.

`getNativesDir()` uses `$XDG_DATA_HOME/omp/natives` when `$XDG_DATA_HOME/omp` exists; otherwise it uses `~/.omp/natives`.

If a populated embedded addon manifest is present, it is also treated as a compiled-binary signal. Current embedded manifests point at a gzip-compressed tar archive (`embedded-addons.<tag>.tar.gz`) that contains one or more matching `.node` files. The loader extracts the archive into the versioned cache directory, validates the selected file by size, and prepends that cache path before normal candidate probing.

For npm/bun installs (non-compiled), `loader-state.js` resolves the platform leaf directory via `require.resolve("@oh-my-pi/pi-natives-<tag>/package.json")` and probes its `.node` **before** the core package's `native/` directory and the executable directory. The optional-dependency binary is therefore preferred over any `.node` left in the core (e.g. a stale local-dev build). On Windows `node_modules` installs, the loader first stages the selected leaf/core addon into `<getNativesDir()>/<packageVersion>/...` and prepends that staged path so running processes do not lock the `node_modules` copy during global updates.

### Failure modes

Loader failures are explicit:

- **Unsupported platform tag**: after failed probing, throws with supported platform list.
- **No loadable candidate**: throws with all attempted paths and remediation hints.
- **Embedded/staging errors**: directory/write/archive/staging failures are recorded and included in final load diagnostics if no candidate loads.
- **Release mismatch**: outside workspace-dev loads, a candidate that loads but lacks the version sentinel export for `package.json#version` is rejected with a reinstall hint.

## Rust N-API module layer

`crates/pi-natives/src/lib.rs` declares exported module ownership:

- `appearance`
- `ast`
- `block`
- `clipboard`
- `crash_handler`
- `fd`
- `fs_cache`
- `glob`
- `glob_util`
- `grep`
- `highlight`
- `html`
- `iso`
- `keys`
- `language` (re-exported from `pi_ast`)
- `power`
- `prof`
- `ps`
- `pty`
- `shell`
- `sixel`
- `snapcompact`
- `summary`
- `task`
- `text`
- `tokens`
- `utils` (crate-private helpers)
- `workspace`

N-API exports are generated from Rust `#[napi]` functions/classes/objects/enums. Snake_case Rust names are exposed as camelCase JavaScript names unless explicitly configured by napi-rs.

## Ownership boundaries

- **Loader/package ownership (`packages/natives/native`, `packages/natives/scripts`)**
  - runtime binary selection
  - CPU variant selection and override handling
  - compiled-binary embedded archive extraction
  - Windows `node_modules` addon staging
  - generated TypeScript declarations and explicit ESM export/enum patching
- **Rust ownership (`crates/pi-natives/src`)**
  - algorithmic and system-level implementation
  - platform-native behavior and performance-sensitive logic
  - N-API symbol implementation consumed directly by package callers
- **Consumer ownership (`packages/coding-agent`, `packages/tui`)**
  - user-facing policy and fallbacks that are not built into the native API
  - higher-level rendering, artifact, shell-session, and command behavior

## Runtime flow (high level)

1. Consumer imports from `@oh-my-pi/pi-natives`.
2. `native/index.js` computes platform/arch/variant and candidate paths.
3. Optional embedded archive extraction or Windows `node_modules` staging can prepend a versioned-cache candidate.
4. Each candidate is `require(...)`d; install/compiled loads must expose the package-version sentinel.
5. The loaded addon object is bound to explicit named ESM exports, including generated enum objects.
6. Caller invokes generated N-API functions/classes directly.

## Glossary

- **Native addon**: A `.node` binary loaded via Node-API (N-API).
- **Platform tag**: Runtime tuple `platform-arch` (for example `darwin-arm64`).
- **Platform leaf package**: Per-platform npm package `@oh-my-pi/pi-natives-<tag>` that carries one platform's prebuilt `.node`. The core depends on every leaf via `optionalDependencies`; the package manager installs only the host-matching one (`os`/`cpu`).
- **Variant**: x64 CPU-specific build flavor (`modern` AVX2, `baseline` fallback).
- **Generated binding declaration**: `native/index.d.ts` emitted by napi-rs during `build-native.ts`.
- **Version sentinel**: Rust export named from the package version (for example `__piNativesV16_0_3`) that lets the loader reject a `.node` from a different release.
- **Compiled binary mode**: Runtime mode where the CLI is bundled and native addons are resolved from embedded/cache paths before package-local paths.
- **Embedded addon**: Build artifact metadata and archive reference generated into `native/embedded-addon.js` so compiled binaries can extract matching `.node` payloads.