oh-my-pi/docs/natives-architecture.md

Natives Architecture

@oh-my-pi/pi-natives is now a two-layer package around a loader:

1. CommonJS loader/package entrypoint resolves and loads the correct .node addon and patches generated enum objects onto the export object.
2. Rust N-API module layer implements the exported functions/classes and emits the generated TypeScript declarations.

This document is the foundation for deeper module-level docs.

Implementation files

• packages/natives/native/index.js
• packages/natives/native/index.d.ts
• packages/natives/native/loader-state.js
• packages/natives/native/embedded-addon.js
• packages/natives/scripts/build-native.ts
• packages/natives/scripts/embed-native.ts
• packages/natives/scripts/gen-enums.ts
• packages/natives/package.json
• crates/pi-natives/src/lib.rs

Package entrypoint and public surface

packages/natives/package.json points directly at generated native bindings:

• main: ./native/index.js
• types: ./native/index.d.ts
• exports["."].types: ./native/index.d.ts
• exports["."].import: ./native/index.js

There is no current packages/natives/src TypeScript wrapper layer. Consumers import functions/classes/enums directly from @oh-my-pi/pi-natives; the type contract is the generated native/index.d.ts plus enum exports appended by scripts/gen-enums.ts.

Current capability groups in the generated API include:

• Search/text/code primitives: grep, search, hasMatch, fuzzyFind, glob, astGrep, astEdit, text width/slicing/wrapping/sanitization, syntax highlighting, token counting.
• Execution/process/terminal primitives: executeShell, Shell, PtySession, process-tree helpers, key parsing.
• System/media/conversion primitives: clipboard, image resize/encode/SIXEL, HTML-to-Markdown, macOS appearance/power helpers, work profiling, Windows ProjFS overlay helpers.

Loader layer

packages/natives/native/index.js owns runtime addon selection and optional embedded extraction.

Candidate resolution model

• Platform tag is ${process.platform}-${process.arch}.
• Supported tags are currently:
  • linux-x64
  • linux-arm64
  • darwin-x64
  • darwin-arm64
  • win32-x64
• x64 can use CPU variants:
  • modern (AVX2-capable)
  • baseline (fallback)
• Non-x64 uses the default filename without a variant suffix.

Filename strategy:

• Default: pi_natives.<platform>-<arch>.node
• x64 variant: pi_natives.<platform>-<arch>-modern.node or ...-baseline.node
• x64 runtime fallback includes the unsuffixed default filename after variant candidates.

Platform-specific variant detection

For x64, variant selection uses:

• Linux: /proc/cpuinfo
• macOS: sysctl -n machdep.cpu.leaf7_features, then machdep.cpu.features
• Windows: PowerShell check for System.Runtime.Intrinsics.X86.Avx2

PI_NATIVE_VARIANT can force modern or baseline; invalid values are ignored.

Binary distribution and extraction model

The published @oh-my-pi/pi-natives package ships only the loader layer in native/: the CommonJS loader (index.js), generated declarations (index.d.ts), the loader-state.js/.d.ts helpers, and the embedded-addon manifest stub (embedded-addon.js). It carries no .node binaries.

Each platform's prebuilt .node is published as a separate optional-dependency leaf package — @oh-my-pi/pi-natives-<platform>-<arch>, one per supported tag — which the core lists in optionalDependencies at the lockstep version. npm/bun install only the leaf whose os/cpu match the host. The working-tree package keeps built .node files under native/ for local dev; the release-publish rewrite (prepareNativeCorePackage in scripts/ci-release-publish.ts) strips them from the core tarball, and the leaves are generated by packages/natives/scripts/gen-npm-packages.ts (LEAF_TARGETS). Adding a build target therefore requires a matching LEAF_TARGETS entry, or the binary never reaches npm users.

For compiled binaries, loader behavior is:

1. Check versioned user cache path: <getNativesDir()>/<packageVersion>/....
2. Check legacy compiled-binary location:
   • Windows: %LOCALAPPDATA%/omp (fallback %USERPROFILE%/AppData/Local/omp)
   • non-Windows: ~/.local/bin
3. Fall back to packaged native/ and executable directory candidates.

getNativesDir() uses $XDG_DATA_HOME/omp/natives when $XDG_DATA_HOME/omp exists; otherwise it uses ~/.omp/natives.

If a populated embedded addon manifest is present, it is also treated as a compiled-binary signal. The loader can extract the matching embedded .node into the versioned cache directory before candidate probing.

For npm/bun installs (non-compiled), loader-state.js resolves the platform leaf directory via require.resolve("@oh-my-pi/pi-natives-<tag>/package.json") and probes its .node before the core package's native/ directory and the executable directory. The optional-dependency binary is therefore preferred over any .node left in the core (e.g. a stale local-dev build).

Failure modes

Loader failures are explicit:

• Unsupported platform tag: after failed probing, throws with supported platform list.
• No loadable candidate: throws with all attempted paths and remediation hints.
• Embedded extraction errors: directory/write failures are recorded and included in final load diagnostics if no candidate loads.

The current loader does not perform a separate post-require export validation pass.

Rust N-API module layer

crates/pi-natives/src/lib.rs declares exported module ownership:

• appearance
• ast
• clipboard
• fd
• fs_cache
• glob
• glob_util
• grep
• highlight
• html
• image
• keys
• language
• power
• prof
• projfs_overlay
• ps
• pty
• shell
• task
• text
• tokens
• utils (crate-private helpers)

N-API exports are generated from Rust #[napi] functions/classes/objects/enums. Snake_case Rust names are exposed as camelCase JavaScript names unless explicitly configured by napi-rs.

Ownership boundaries

• Loader/package ownership (packages/natives/native, packages/natives/scripts)
  • runtime binary selection
  • CPU variant selection and override handling
  • compiled-binary embedded extraction
  • generated TypeScript declarations and enum export patching
• Rust ownership (crates/pi-natives/src)
  • algorithmic and system-level implementation
  • platform-native behavior and performance-sensitive logic
  • N-API symbol implementation consumed directly by package callers
• Consumer ownership (packages/coding-agent, packages/tui)
  • user-facing policy and fallbacks that are not built into the native API
  • higher-level rendering, artifact, shell-session, and command behavior

Runtime flow (high level)

1. Consumer imports from @oh-my-pi/pi-natives.
2. native/index.js computes platform/arch/variant and candidate paths.
3. Optional embedded binary extraction occurs for compiled distributions.
4. The first require(candidate) that succeeds becomes the exported addon object.
5. Generated enum objects are appended to module.exports.
6. Caller invokes generated N-API functions/classes directly.

Glossary

• Native addon: A .node binary loaded via Node-API (N-API).
• Platform tag: Runtime tuple platform-arch (for example darwin-arm64).
• Platform leaf package: Per-platform npm package @oh-my-pi/pi-natives-<tag> that carries one platform's prebuilt .node. The core depends on every leaf via optionalDependencies; the package manager installs only the host-matching one (os/cpu).
• Variant: x64 CPU-specific build flavor (modern AVX2, baseline fallback).
• Generated binding declaration: native/index.d.ts emitted by napi-rs during build-native.ts.
• Compiled binary mode: Runtime mode where the CLI is bundled and native addons are resolved from embedded/cache paths before package-local paths.
• Embedded addon: Build artifact metadata and file references generated into native/embedded-addon.js so compiled binaries can extract matching .node payloads.