oh-my-pi/docs/natives-addon-loader-runtime.md

# Natives Addon Loader Runtime

This document covers the runtime loader shipped by `@oh-my-pi/pi-natives`: how `native/index.js` decides which `.node` file to require, how compiled-binary embedded payloads are extracted, and what startup failures report.

## Implementation files

- `packages/natives/native/index.js`
- `packages/natives/native/loader-state.js`
- `packages/natives/native/embedded-addon.js`
- `packages/natives/scripts/embed-native.ts`
- `packages/natives/package.json`

## Scope and responsibility

The loader is intentionally narrow:

- Build a platform/CPU-aware candidate list for addon filenames and directories.
- Treat an embedded-addon manifest as the authoritative compiled-binary signal when present.
- Optionally materialize an embedded addon into a versioned per-user cache directory.
- Attempt candidates in deterministic order and return the first addon that `require(...)` loads.

The current loader does **not** run a separate `validateNative(...)` export-presence gate. API shape is provided by the generated N-API binding file (`native/index.d.ts`) and the loaded addon itself. A stale binary therefore normally fails as a missing property or native load error rather than as a custom "missing exports" validation error.

## Runtime inputs and derived state

At module initialization, `native/index.js` computes:

- **Platform tag**: `${process.platform}-${process.arch}` (for example `darwin-arm64`).
- **Package version**: from `packages/natives/package.json`.
- **Core directories**:
  - `leafPackageDir`: directory of the platform leaf package, resolved via `require.resolve("@oh-my-pi/pi-natives-<tag>/package.json")`; `null` when no leaf is installed (e.g. local dev).
  - `nativeDir`: package-local `packages/natives/native`.
  - `execDir`: directory containing `process.execPath`.
  - `versionedDir`: `<getNativesDir()>/<packageVersion>`.
  - `userDataDir` fallback:
    - Windows: `%LOCALAPPDATA%/omp` or `%USERPROFILE%/AppData/Local/omp`.
    - Non-Windows: `~/.local/bin`.
- **Natives cache root** (`getNativesDir()`):
  - if `$XDG_DATA_HOME/omp` exists, `$XDG_DATA_HOME/omp/natives`;
  - otherwise `~/.omp/natives`.
- **Compiled-binary mode** (`detectCompiledBinary`): true if any of:
  - embedded-addon manifest is non-null,
  - `PI_COMPILED` env var is set,
  - `import.meta.url` contains Bun embedded markers (`$bunfs`, `~BUN`, `%7EBUN`).
- **Variant override**: `PI_NATIVE_VARIANT` (`modern`/`baseline` only; invalid values ignored).
- **Selected variant**: explicit override, otherwise runtime AVX2 detection on x64 (`modern` if AVX2, else `baseline`).

## Platform support and tag resolution

`SUPPORTED_PLATFORMS` is fixed to:

- `linux-x64`
- `linux-arm64`
- `darwin-x64`
- `darwin-arm64`
- `win32-x64`

Unsupported platforms are not rejected before probing. The loader first tries the computed candidate paths. If all fail and `platformTag` is unsupported, it throws an unsupported-platform error listing supported tags.

## Variant selection (`modern` / `baseline` / default)

### x64 behavior

1. `PI_NATIVE_VARIANT=modern|baseline` wins when valid.
2. Otherwise AVX2 support is detected:
   - Linux: scan `/proc/cpuinfo` for `avx2`.
   - macOS: `sysctl -n machdep.cpu.leaf7_features`, then `machdep.cpu.features`.
   - Windows: PowerShell `[System.Runtime.Intrinsics.X86.Avx2]::IsSupported`.
3. AVX2 selects `modern`; unavailable or undetectable AVX2 selects `baseline`.

### Non-x64 behavior

No variant suffix is used; the filename is `pi_natives.<platform>-<arch>.node`.

### Filename construction

`loader-state.js#getAddonFilenames` returns:

- Non-x64 or no variant: `pi_natives.<tag>.node`
- x64 + `modern`:
  1. `pi_natives.<tag>-modern.node`
  2. `pi_natives.<tag>-baseline.node`
  3. `pi_natives.<tag>.node`
- x64 + `baseline`:
  1. `pi_natives.<tag>-baseline.node`
  2. `pi_natives.<tag>.node`

The default unsuffixed fallback remains part of the x64 candidate list.

## Candidate path construction and fallback ordering

`resolveLoaderCandidates(...)` expands every filename across directories, then de-duplicates while preserving first occurrence order.

### Non-compiled runtime

For each filename, candidates are, in order:

1. `<leafPackageDir>/<filename>` (omitted when `leafPackageDir` is `null`)
2. `<nativeDir>/<filename>`
3. `<execDir>/<filename>`

The leaf package dir comes first so the optional-dependency binary published with the release is preferred over any `.node` left in the core package's `native/` (e.g. a stale local-dev build).

On Windows installs where `nativeDir` is inside a `node_modules` segment (`shouldStageNodeModulesAddon`), `<versionedDir>/<filename>` staging candidates are prepended ahead of the leaf candidates so a locked `node_modules` binary can be sidestepped during `bun install -g` updates.

### Compiled runtime

For each filename, candidates are:

1. `<versionedDir>/<filename>`
2. `<userDataDir>/<filename>`
3. `<nativeDir>/<filename>`
4. `<execDir>/<filename>`

At load time, an extracted embedded candidate, when produced, is prepended ahead of these de-duplicated candidates.

## Embedded addon extraction lifecycle

`embedded-addon.js` is generated by `scripts/embed-native.ts`. The reset stub exports `embeddedAddon = null`. A populated manifest has:

- `platformTag`
- `version`
- `files[]` entries with `variant`, `filename`, and `filePath`

Extraction (`maybeExtractEmbeddedAddon`) runs only when:

1. compiled-binary mode is true,
2. `embeddedAddon` is non-null,
3. manifest `platformTag` equals the runtime platform tag,
4. manifest `version` equals the package version,
5. a variant-appropriate embedded file exists.

Variant file selection:

- Non-x64: prefer `default`, then first available file.
- x64 + `modern`: prefer `modern`, fallback to `baseline`.
- x64 + `baseline`: require `baseline`.

Materialization:

1. Ensure `<versionedDir>` exists.
2. Reuse `<versionedDir>/<selected filename>` if it already exists.
3. Otherwise read `selectedEmbeddedFile.filePath` and write the target path.
4. Return the target path as the first candidate.

Directory creation or write failures are appended to the loader error list; probing continues through normal candidates.

## Lifecycle and state transitions

```text
Init
  -> Load package metadata and embedded-addon manifest
  -> Compute platform/version/variant/filenames/candidate paths
  -> (compiled + embedded manifest matches?)
       yes -> try extract to versionedDir (record errors, continue)
       no  -> skip extraction
  -> For each runtime candidate in order:
       require(candidate)
       -> success: return addon exports (READY)
       -> failure: record error, continue
  -> none loaded:
       if unsupported platform tag -> throw Unsupported platform
       else -> throw Failed to load (tried-path diagnostics + hints)
```

## Failure behavior and diagnostics

### Unsupported platform

If all candidates fail and `platformTag` is not supported, the loader throws:

- `Unsupported platform: <tag>`
- supported platform list
- issue-reporting guidance

### No loadable candidate

If the platform is supported but no candidate can be loaded, the final error includes:

- `Failed to load pi_natives native addon for <platformTag>` or `<platformTag> (<variant>)`
- every attempted path with the corresponding `require(...)` error
- mode-specific remediation hints

### Compiled-binary startup failures

Compiled mode diagnostics include:

- expected versioned cache target paths (`<versionedDir>/<filename>`),
- remediation to delete the versioned cache and rerun,
- direct release download `curl` commands for each expected filename.

### Non-compiled startup failures

Normal package/runtime diagnostics include:

- reinstall hint (`bun install @oh-my-pi/pi-natives`),
- local rebuild command (`bun --cwd=packages/natives run build`),
- optional x64 variant build hint (`TARGET_VARIANT=baseline|modern bun --cwd=packages/natives run build`).