oh-my-pi/docs/natives-addon-loader-runtime.md

Natives Addon Loader Runtime

This document covers the runtime loader shipped by @oh-my-pi/pi-natives: how native/index.js decides which .node file to require, how compiled-binary embedded payloads are extracted, and what startup failures report.

Implementation files

• packages/natives/native/index.js
• packages/natives/native/loader-state.js
• packages/natives/native/embedded-addon.js
• packages/natives/scripts/embed-native.ts
• packages/natives/package.json

Scope and responsibility

The loader is intentionally narrow:

• Build a platform/CPU-aware candidate list for addon filenames and directories.
• Treat an embedded-addon manifest as the authoritative compiled-binary signal when present.
• Optionally materialize an embedded addon into a versioned per-user cache directory.
• Attempt candidates in deterministic order and return the first addon that require(...) loads.

The current loader does not run a separate validateNative(...) export-presence gate. API shape is provided by the generated N-API binding file (native/index.d.ts) and the loaded addon itself. A stale binary therefore normally fails as a missing property or native load error rather than as a custom "missing exports" validation error.

Runtime inputs and derived state

At module initialization, native/index.js computes:

• Platform tag: ${process.platform}-${process.arch} (for example darwin-arm64).
• Package version: from packages/natives/package.json.
• Core directories:
  • leafPackageDir: directory of the platform leaf package, resolved via require.resolve("@oh-my-pi/pi-natives-<tag>/package.json"); null when no leaf is installed (e.g. local dev).
  • nativeDir: package-local packages/natives/native.
  • execDir: directory containing process.execPath.
  • versionedDir: <getNativesDir()>/<packageVersion>.
  • userDataDir fallback:
    • Windows: %LOCALAPPDATA%/omp or %USERPROFILE%/AppData/Local/omp.
    • Non-Windows: ~/.local/bin.
• Natives cache root (getNativesDir()):
  • if $XDG_DATA_HOME/omp exists, $XDG_DATA_HOME/omp/natives;
  • otherwise ~/.omp/natives.
• Compiled-binary mode (detectCompiledBinary): true if any of:
  • embedded-addon manifest is non-null,
  • PI_COMPILED env var is set,
  • import.meta.url contains Bun embedded markers ($bunfs, ~BUN, %7EBUN).
• Variant override: PI_NATIVE_VARIANT (modern/baseline only; invalid values ignored).
• Selected variant: explicit override, otherwise runtime AVX2 detection on x64 (modern if AVX2, else baseline).

Platform support and tag resolution

SUPPORTED_PLATFORMS is fixed to:

• linux-x64
• linux-arm64
• darwin-x64
• darwin-arm64
• win32-x64

Unsupported platforms are not rejected before probing. The loader first tries the computed candidate paths. If all fail and platformTag is unsupported, it throws an unsupported-platform error listing supported tags.

Variant selection (modern / baseline / default)

x64 behavior

1. PI_NATIVE_VARIANT=modern|baseline wins when valid.
2. Otherwise AVX2 support is detected:
   • Linux: scan /proc/cpuinfo for avx2.
   • macOS: sysctl -n machdep.cpu.leaf7_features, then machdep.cpu.features.
   • Windows: PowerShell [System.Runtime.Intrinsics.X86.Avx2]::IsSupported.
3. AVX2 selects modern; unavailable or undetectable AVX2 selects baseline.

Non-x64 behavior

No variant suffix is used; the filename is pi_natives.<platform>-<arch>.node.

Filename construction

loader-state.js#getAddonFilenames returns:

• Non-x64 or no variant: pi_natives.<tag>.node
• x64 + modern:
  1. pi_natives.<tag>-modern.node
  2. pi_natives.<tag>-baseline.node
  3. pi_natives.<tag>.node
• x64 + baseline:
  1. pi_natives.<tag>-baseline.node
  2. pi_natives.<tag>.node

The default unsuffixed fallback remains part of the x64 candidate list.

Candidate path construction and fallback ordering

resolveLoaderCandidates(...) expands every filename across directories, then de-duplicates while preserving first occurrence order.

Non-compiled runtime

For each filename, candidates are, in order:

1. <leafPackageDir>/<filename> (omitted when leafPackageDir is null)
2. <nativeDir>/<filename>
3. <execDir>/<filename>

The leaf package dir comes first so the optional-dependency binary published with the release is preferred over any .node left in the core package's native/ (e.g. a stale local-dev build).

On Windows installs where nativeDir is inside a node_modules segment (shouldStageNodeModulesAddon), <versionedDir>/<filename> staging candidates are prepended ahead of the leaf candidates so a locked node_modules binary can be sidestepped during bun install -g updates.

Compiled runtime

For each filename, candidates are:

1. <versionedDir>/<filename>
2. <userDataDir>/<filename>
3. <nativeDir>/<filename>
4. <execDir>/<filename>

At load time, an extracted embedded candidate, when produced, is prepended ahead of these de-duplicated candidates.

Embedded addon extraction lifecycle

embedded-addon.js is generated by scripts/embed-native.ts. The reset stub exports embeddedAddon = null. A populated manifest has:

• platformTag
• version
• files[] entries with variant, filename, and filePath

Extraction (maybeExtractEmbeddedAddon) runs only when:

1. compiled-binary mode is true,
2. embeddedAddon is non-null,
3. manifest platformTag equals the runtime platform tag,
4. manifest version equals the package version,
5. a variant-appropriate embedded file exists.

Variant file selection:

• Non-x64: prefer default, then first available file.
• x64 + modern: prefer modern, fallback to baseline.
• x64 + baseline: require baseline.

Materialization:

1. Ensure <versionedDir> exists.
2. Reuse <versionedDir>/<selected filename> if it already exists.
3. Otherwise read selectedEmbeddedFile.filePath and write the target path.
4. Return the target path as the first candidate.

Directory creation or write failures are appended to the loader error list; probing continues through normal candidates.

Lifecycle and state transitions

Init
  -> Load package metadata and embedded-addon manifest
  -> Compute platform/version/variant/filenames/candidate paths
  -> (compiled + embedded manifest matches?)
       yes -> try extract to versionedDir (record errors, continue)
       no  -> skip extraction
  -> For each runtime candidate in order:
       require(candidate)
       -> success: return addon exports (READY)
       -> failure: record error, continue
  -> none loaded:
       if unsupported platform tag -> throw Unsupported platform
       else -> throw Failed to load (tried-path diagnostics + hints)

Failure behavior and diagnostics

Unsupported platform

If all candidates fail and platformTag is not supported, the loader throws:

• Unsupported platform: <tag>
• supported platform list
• issue-reporting guidance

No loadable candidate

If the platform is supported but no candidate can be loaded, the final error includes:

• Failed to load pi_natives native addon for <platformTag> or <platformTag> (<variant>)
• every attempted path with the corresponding require(...) error
• mode-specific remediation hints

Compiled-binary startup failures

Compiled mode diagnostics include:

• expected versioned cache target paths (<versionedDir>/<filename>),
• remediation to delete the versioned cache and rerun,
• direct release download curl commands for each expected filename.

Non-compiled startup failures

Normal package/runtime diagnostics include:

• reinstall hint (bun install @oh-my-pi/pi-natives),
• local rebuild command (bun --cwd=packages/natives run build),
• optional x64 variant build hint (TARGET_VARIANT=baseline|modern bun --cwd=packages/natives run build).