diff --git a/docs/natives-addon-loader-runtime.md b/docs/natives-addon-loader-runtime.md new file mode 100644 index 000000000..86832196e --- /dev/null +++ b/docs/natives-addon-loader-runtime.md @@ -0,0 +1,268 @@ +# Natives Addon Loader Runtime + +This document deep-dives the addon loading/validation layer in `@oh-my-pi/pi-natives`: how `native.ts` decides which `.node` file to load, when embedded payload extraction runs, and how startup failures are reported. + +## Implementation files + +- `packages/natives/src/native.ts` +- `packages/natives/src/embedded-addon.ts` +- `packages/natives/src/bindings.ts` +- `packages/natives/package.json` + +## Scope and responsibility + +Loader/runtime responsibilities are intentionally narrow: + +- Build a platform/CPU-aware candidate list for addon filenames and directories. +- Optionally materialize an embedded addon into a versioned per-user cache directory. +- Attempt candidates in deterministic order. +- Reject stale or incompatible addons via `validateNative` before exposing bindings. + +Out of scope here: module-specific grep/text/highlight behavior. + +## Runtime inputs and derived state + +At module initialization (`export const native = loadNative();`), `native.ts` computes static context: + +- **Platform tag**: ``${process.platform}-${process.arch}`` (for example `darwin-arm64`). +- **Package version**: from `packages/natives/package.json` (`version` field). +- **Core directories**: + - `nativeDir`: package-local `packages/natives/native`. + - `execDir`: directory containing `process.execPath`. + - `versionedDir`: `/`. + - `userDataDir` fallback: + - Windows: `%LOCALAPPDATA%/omp` (or `%USERPROFILE%/AppData/Local/omp`). + - Non-Windows: `~/.local/bin`. +- **Compiled-binary mode** (`isCompiledBinary`): true if any of: + - `PI_COMPILED` env var is set, or + - `import.meta.url` contains Bun-embedded markers (`$bunfs`, `~BUN`, `%7EBUN`). +- **Variant override**: `PI_NATIVE_VARIANT` (`modern`/`baseline` only; invalid values ignored). +- **Selected variant**: explicit override, otherwise runtime AVX2 detection on x64 (`modern` if AVX2, else `baseline`). + +## Platform support and tag resolution + +`SUPPORTED_PLATFORMS` is fixed to: + +- `linux-x64` +- `linux-arm64` +- `darwin-x64` +- `darwin-arm64` +- `win32-x64` + +Behavior detail: + +- Unsupported platforms are not rejected up-front. +- Loader still tries all computed candidates first. +- If nothing loads, it throws an explicit unsupported-platform error listing supported tags. + +This preserves useful diagnostics for near-miss cases while still failing hard for truly unsupported targets. + +## Variant selection (`modern` / `baseline` / default) + +### x64 behavior + +1. If `PI_NATIVE_VARIANT` is `modern` or `baseline`, that value wins. +2. Else detect AVX2 support: + - Linux: scan `/proc/cpuinfo` for `avx2`. + - macOS: query `sysctl` (`machdep.cpu.leaf7_features`, fallback `machdep.cpu.features`). + - Windows: run PowerShell `[System.Runtime.Intrinsics.X86.Avx2]::IsSupported`. +3. Result: + - AVX2 available -> `modern` + - AVX2 unavailable/undetectable -> `baseline` + +### Non-x64 behavior + +- No variant is used; loader stays on the default filename (`pi_natives.-.node`). + +### Filename construction + +Given `tag = -`: + +- Non-x64 or no variant: `pi_natives..node` +- x64 + `modern`: try in order + 1. `pi_natives.-modern.node` + 2. `pi_natives.-baseline.node` (intentional fallback) +- x64 + `baseline`: only `pi_natives.-baseline.node` + +The `addonLabel` used in final error messages is either `` or ` ()`. + +## Candidate path construction and fallback ordering + +`native.ts` builds candidate pools before any `require(...)` call. + +### Debug/dev candidates (only when `PI_DEV` is set) + +Prepended first: + +1. `/pi_natives.dev.node` +2. `/pi_natives.dev.node` + +This path is explicit debug intent and always outranks release candidates. + +### Release candidates + +Built from variant-resolved filename list and searched in this order: + +- **Non-compiled runtime**: + 1. `/` + 2. `/` + +- **Compiled runtime** (`PI_COMPILED` or Bun embedded markers): + 1. `/` + 2. `/` + 3. `/` + 4. `/` + +`dedupedCandidates` removes duplicates while preserving first occurrence order. + +### Final runtime sequence + +At load time: + +1. Optional embedded extraction candidate (if produced) is inserted at the front. +2. Remaining deduplicated candidates are tried in order. +3. First candidate that both `require(...)`s and passes `validateNative(...)` wins. + +## Embedded addon extraction lifecycle + +`embedded-addon.ts` defines a generated manifest shape: + +- `platformTag` +- `version` +- `files[]` where each entry has `variant`, `filename`, `filePath` + +Current checked-in default is `embeddedAddon: null`; compiled artifacts may replace this with real metadata. + +### Extraction state machine + +Extraction (`maybeExtractEmbeddedAddon`) runs only when all gates pass: + +1. `isCompiledBinary === true` +2. `embeddedAddon !== null` +3. `embeddedAddon.platformTag === platformTag` +4. `embeddedAddon.version === packageVersion` +5. A variant-appropriate embedded file is found + +Variant file selection mirrors runtime variant intent: + +- Non-x64: prefer `default`, then first available file. +- x64 + `modern`: prefer `modern`, fallback to `baseline`. +- x64 + `baseline`: require `baseline`. + +Materialization behavior: + +1. Ensure `` exists (`mkdirSync(..., { recursive: true })`). +2. If `/` already exists, reuse it (no rewrite). +3. Else read embedded source `filePath` and write target file. +4. Return target path for highest-priority load attempt. + +On failure, extraction does not crash immediately; it appends an error entry (directory creation or write failure) and loader proceeds to normal candidate probing. + +## Lifecycle and state transitions + +```text +Init + -> Compute platform/version/variant/candidate lists + -> (Compiled + embedded manifest matches?) + yes -> Try extract embedded to versionedDir (record errors, continue) + no -> Skip extraction + -> For each runtime candidate in order: + require(candidate) + -> success: validateNative + -> pass: return bindings (READY) + -> fail: record error, continue + -> failure: record error, continue + -> none loaded: + if unsupported platform tag -> throw Unsupported platform + else -> throw Failed to load (full tried-path diagnostics + hints) +``` + +## `validateNative` contract checks + +`validateNative(bindings, source)` enforces a function-only contract over `NativeBindings` at startup. + +Mechanics: + +- For each required export name, it checks `typeof bindings[name] === "function"`. +- Missing names are aggregated. +- If any are missing, loader throws: + - source addon path, + - missing export list, + - rebuild command hint. + +This is a hard compatibility gate against stale binaries, partial builds, and symbol/name drift. + +### JS API ↔ native export mapping (validation gate) + +| JS binding name checked in `validateNative` | Expected native export name | +| --- | --- | +| `grep` | `grep` | +| `glob` | `glob` | +| `highlightCode` | `highlightCode` | +| `executeShell` | `executeShell` | +| `PtySession` | `PtySession` | +| `Shell` | `Shell` | +| `visibleWidth` | `visibleWidth` | +| `getSystemInfo` | `getSystemInfo` | +| `getWorkProfile` | `getWorkProfile` | +| `invalidateFsScanCache` | `invalidateFsScanCache` | + +Note: `bindings.ts` declares only the base `cancelWork(id)` member; module `types.ts` files declaration-merge additional symbols that `validateNative` enforces. + +## Failure behavior and diagnostics + +## Unsupported platform + +If all candidates fail and `platformTag` is not in `SUPPORTED_PLATFORMS`, loader throws: + +- `Unsupported platform: ` +- Full supported-platform list +- Explicit issue-reporting guidance + +## Stale binary / mismatch symptoms + +Typical stale mismatch signal: + +- `Native addon missing exports (). Missing: ...` + +Common causes: + +- Old `.node` binary from previous package version/API shape. +- Wrong variant artifact selected (for x64). +- New Rust export not present in loaded artifact. + +Loader behavior: + +- Records per-candidate missing-export failures. +- Continues probing remaining candidates. +- If no candidate validates, final error includes every attempted path with each failure message. + +## Compiled-binary startup failures + +In compiled mode final diagnostics include: + +- expected versioned cache target paths (`/`), +- remediation to delete stale `` and rerun, +- direct release download `curl` commands for each expected filename. + +## Non-compiled startup failures + +In normal package/runtime mode final diagnostics include: + +- reinstall hint (`bun install @oh-my-pi/pi-natives`), +- local rebuild command (`bun --cwd=packages/natives run build:native`), +- optional x64 variant build hint (`TARGET_VARIANT=baseline|modern ...`). + +## Dev/debug versus release behavior + +When `PI_DEV` is set: + +- `pi_natives.dev.node` candidates are prepended ahead of all release candidates. +- Loader emits per-candidate console diagnostics (`Loaded native addon...` and load errors). + +Without `PI_DEV`: + +- Only release candidate chain is used. +- No dev console diagnostics are emitted. + +Operationally, this means debug sessions can validate an ad-hoc dev addon first, while production/release runs remain on deterministic release artifact probing. \ No newline at end of file diff --git a/docs/natives-architecture.md b/docs/natives-architecture.md new file mode 100644 index 000000000..6fdda1a63 --- /dev/null +++ b/docs/natives-architecture.md @@ -0,0 +1,168 @@ +# Natives Architecture + +`@oh-my-pi/pi-natives` is a three-layer stack: + +1. **TypeScript wrapper/API layer** exposes stable JS/TS entrypoints. +2. **Addon loading/validation layer** resolves and validates the `.node` binary for the current runtime. +3. **Rust N-API module layer** implements performance-critical primitives exported to JS. + +This document is the foundation for deeper module-level docs. + +## Implementation files + +- `packages/natives/src/index.ts` +- `packages/natives/src/native.ts` +- `packages/natives/src/bindings.ts` +- `packages/natives/src/embedded-addon.ts` +- `packages/natives/scripts/build-native.ts` +- `packages/natives/scripts/embed-native.ts` +- `packages/natives/package.json` +- `crates/pi-natives/src/lib.rs` + +## Layer 1: TypeScript wrapper/API layer + +`packages/natives/src/index.ts` is the public barrel. It groups exports by capability domain and re-exports typed wrappers rather than exposing raw N-API bindings directly. + +Current top-level groups: + +- **Search/text primitives**: `grep`, `glob`, `text`, `highlight` +- **Execution/process/terminal primitives**: `shell`, `pty`, `ps`, `keys` +- **System/media/conversion primitives**: `image`, `html`, `clipboard`, `system-info`, `work` + +`packages/natives/src/bindings.ts` defines the base interface contract: + +- `NativeBindings` starts with shared members (`cancelWork(id: number)`) +- module-specific bindings are added by declaration merging from each module’s `types.ts` +- `Cancellable` standardizes timeout and abort-signal options for wrappers that expose cancellation + +**Guaranteed contract (API-facing):** consumers import from `@oh-my-pi/pi-natives` and use typed wrappers. + +**Implementation detail (may change):** declaration merging and internal wrapper layout (`src//index.ts`, `src//types.ts`). + +## Layer 2: Addon loading and validation + +`packages/natives/src/native.ts` owns runtime addon selection, optional extraction, and export validation. + +### Candidate resolution model + +- Platform tag is `"${process.platform}-${process.arch}"`. +- Supported tags are currently: + - `linux-x64` + - `linux-arm64` + - `darwin-x64` + - `darwin-arm64` + - `win32-x64` +- x64 can use CPU variants: + - `modern` (AVX2-capable) + - `baseline` (fallback) +- Non-x64 uses the default filename (no variant suffix). + +Filename strategy: + +- Release: `pi_natives.-.node` +- x64 variant release: `pi_natives.--modern.node` and/or `...-baseline.node` +- Dev: `pi_natives.dev.node` (preferred when `PI_DEV` is set) + +### Platform-specific variant detection + +For x64, variant selection uses: + +- **Linux**: `/proc/cpuinfo` +- **macOS**: `sysctl machdep.cpu.leaf7_features` / `machdep.cpu.features` +- **Windows**: PowerShell check for `System.Runtime.Intrinsics.X86.Avx2` + +`PI_NATIVE_VARIANT` can explicitly force `modern` or `baseline`. + +### Binary distribution and extraction model + +`packages/natives/package.json` includes both `src` and `native` in published files. The `native/` directory stores prebuilt platform artifacts. + +For compiled binaries (`PI_COMPILED` or Bun embedded runtime markers), loader behavior is: + +1. Check versioned user cache path: `//...` +2. Check legacy compiled-binary location: + - Windows: `%LOCALAPPDATA%/omp` (fallback `%USERPROFILE%/AppData/Local/omp`) + - non-Windows: `~/.local/bin` +3. Fall back to packaged `native/` and executable directory candidates + +If an embedded addon manifest is present (`embedded-addon.ts` generated by `scripts/embed-native.ts`), `native.ts` can materialize the matching embedded binary into the versioned cache directory before loading. + +### Validation and failure modes + +After `require(candidate)`, `validateNative(...)` verifies required exports (for example `grep`, `glob`, `highlightCode`, `PtySession`, `Shell`, `getSystemInfo`, `getWorkProfile`, `invalidateFsScanCache`). + +Failure paths are explicit: + +- **Unsupported platform tag**: throws with supported platform list +- **No loadable candidate**: throws with all attempted paths and remediation hints +- **Missing exports**: throws with exact missing names and rebuild command +- **Embedded extraction errors**: records directory/write failures and includes them in final load diagnostics + +**Guaranteed contract (API-facing):** addon load either succeeds with a validated binding set or fails fast with actionable error text. + +**Implementation detail (may change):** exact candidate search order and compiled-binary fallback path ordering. + +## Layer 3: Rust N-API module layer + +`crates/pi-natives/src/lib.rs` is the Rust entry module that declares exported module ownership: + +- `clipboard` +- `fd` +- `fs_cache` +- `glob` +- `glob_util` +- `grep` +- `highlight` +- `html` +- `image` +- `keys` +- `prof` +- `ps` +- `pty` +- `shell` +- `system_info` +- `task` +- `text` + +These modules implement the N-API symbols consumed and validated by `native.ts`. JS-level names are surfaced through the TS wrappers in `packages/natives/src`. + +**Guaranteed contract (API-facing):** Rust module exports must match the binding names expected by `validateNative` and wrapper modules. + +**Implementation detail (may change):** internal Rust module decomposition and helper module boundaries (`glob_util`, `task`, etc.). + +## Ownership boundaries + +At architecture level, ownership is split as follows: + +- **TS wrapper/API ownership (`packages/natives/src`)** + - public API grouping, option typing, and stable JS ergonomics + - cancellation surface (`timeoutMs`, `AbortSignal`) exposed to callers +- **Loader ownership (`packages/natives/src/native.ts`)** + - runtime binary selection + - CPU variant selection and override handling + - compiled-binary extraction and candidate probing + - hard validation of required native exports +- **Rust ownership (`crates/pi-natives/src`)** + - algorithmic and system-level implementation + - platform-native behavior and performance-sensitive logic + - N-API symbol implementation that TS wrappers consume + +## Runtime flow (high level) + +1. Consumer imports from `@oh-my-pi/pi-natives`. +2. Wrapper module calls into singleton `native` binding. +3. `native.ts` selects candidate binary for platform/arch/variant. +4. Optional embedded binary extraction occurs for compiled distributions. +5. Addon is loaded and export set is validated. +6. Wrapper returns typed results to caller. + +## Glossary + +- **Native addon**: A `.node` binary loaded via Node-API (N-API). +- **Platform tag**: Runtime tuple `platform-arch` (for example `darwin-arm64`). +- **Variant**: x64 CPU-specific build flavor (`modern` AVX2, `baseline` fallback). +- **Wrapper**: TS function/class that provides typed API over raw native exports. +- **Declaration merging**: TS technique used by module `types.ts` files to extend `NativeBindings`. +- **Compiled binary mode**: Runtime mode where the CLI is bundled and native addons are resolved from extracted/cache paths instead of only package-local paths. +- **Embedded addon**: Build artifact metadata and file references generated into `embedded-addon.ts` so compiled binaries can extract matching `.node` payloads. +- **Validation gate**: `validateNative(...)` check that rejects stale/mismatched binaries missing required exports. diff --git a/docs/natives-binding-contract.md b/docs/natives-binding-contract.md new file mode 100644 index 000000000..867487c26 --- /dev/null +++ b/docs/natives-binding-contract.md @@ -0,0 +1,221 @@ +# Natives Binding Contract (TypeScript Side) + +This document defines the TypeScript-side contract that sits between `@oh-my-pi/pi-natives` callers and the loaded N-API addon. + +It focuses on three pieces: + +1. contract shape (`NativeBindings` + module augmentation), +2. wrapper behavior (`src//index.ts`), +3. public export surface (`src/index.ts`). + +## Implementation files + +- `packages/natives/src/bindings.ts` +- `packages/natives/src/native.ts` +- `packages/natives/src/index.ts` +- `packages/natives/src/clipboard/types.ts` +- `packages/natives/src/clipboard/index.ts` +- `packages/natives/src/glob/types.ts` +- `packages/natives/src/glob/index.ts` +- `packages/natives/src/grep/types.ts` +- `packages/natives/src/grep/index.ts` +- `packages/natives/src/highlight/types.ts` +- `packages/natives/src/highlight/index.ts` +- `packages/natives/src/html/types.ts` +- `packages/natives/src/html/index.ts` +- `packages/natives/src/image/types.ts` +- `packages/natives/src/image/index.ts` +- `packages/natives/src/keys/types.ts` +- `packages/natives/src/keys/index.ts` +- `packages/natives/src/ps/types.ts` +- `packages/natives/src/ps/index.ts` +- `packages/natives/src/pty/types.ts` +- `packages/natives/src/pty/index.ts` +- `packages/natives/src/shell/types.ts` +- `packages/natives/src/shell/index.ts` +- `packages/natives/src/system-info/types.ts` +- `packages/natives/src/system-info/index.ts` +- `packages/natives/src/text/types.ts` +- `packages/natives/src/text/index.ts` +- `packages/natives/src/work/types.ts` +- `packages/natives/src/work/index.ts` + +## Contract model + +`packages/natives/src/bindings.ts` defines the base contract: + +- `NativeBindings` (base interface, currently includes `cancelWork(id: number): void`) +- `Cancellable` (`timeoutMs?: number`, `signal?: AbortSignal`) +- `TsFunc` callback shape used by N-API threadsafe callbacks + +Each module adds its own fields by declaration merging: + +```ts +// packages/natives/src//types.ts +declare module "../bindings" { + interface NativeBindings { + grep(options: GrepOptions, onMatch?: TsFunc): Promise; + } +} +``` + +This keeps one aggregate binding interface without a monolithic central type file. + +## Declaration-merging lifecycle and state transitions + +### 1) Compile-time type assembly + +- `bindings.ts` provides the base `NativeBindings` symbol. +- Every `src//types.ts` augments `NativeBindings`. +- `src/native.ts` imports all `.//types` files for side effects so the merged contract is in scope where `NativeBindings` is used. + +State transition: **Base contract** → **Merged contract**. + +### 2) Runtime addon load and validation gate + +- `src/native.ts` loads candidate `.node` binaries. +- Loaded object is treated as `NativeBindings` and immediately passed through `validateNative(...)`. +- `validateNative` verifies required export keys by `typeof bindings[name] === "function"`. + +State transition: **Untrusted addon object** → **Validated native binding object** (or hard failure). + +### 3) Wrapper invocation + +- Module wrappers in `src//index.ts` call `native.`. +- Wrappers adapt defaults and callback shape (`(err, value)` to value-only callback patterns in JS APIs). +- `src/index.ts` re-exports module wrappers/types as the public package API. + +State transition: **Validated raw bindings** → **Ergonomic public API**. + +## Wrapper responsibilities + +Wrappers are intentionally thin; they do not re-implement native logic. + +Primary responsibilities: + +- **Argument normalization/defaulting** + - `glob()` resolves `options.path` to absolute path and defaults `hidden`, `gitignore`, `recursive`. + - `hasMatch()` fills default flags (`ignoreCase`, `multiline`) before native call. +- **Callback adaptation** + - `grep()`, `glob()`, `executeShell()` convert `TsFunc` (`error, value`) into user callback receiving only successful values. +- **Environment or policy behavior around native calls** + - Clipboard wrapper adds OSC52/Termux/headless handling and treats copy as best effort. +- **Public naming and re-export curation** + - `searchContent()` maps to native export `search`. + +## Public export surface organization + +`packages/natives/src/index.ts` is the canonical public barrel. It groups exports by capability domain: + +- Search/text: `grep`, `glob`, `text`, `highlight` +- Execution/process/terminal: `shell`, `pty`, `ps`, `keys` +- System/media/conversion: `image`, `html`, `clipboard`, `system-info`, `work` + +Maintainer rule: if a wrapper is not re-exported from `src/index.ts`, it is not part of the intended public package surface. + +## JS API ↔ native export mapping (representative) + +The Rust side uses N-API export names (typically via `#[napi(js_name = ...)]`) that must match these binding keys. + +| Category | Public JS API (wrapper) | Native binding key | Return type | Async? | +|---|---|---|---|---| +| Grep | `grep(options, onMatch?)` | `grep` | `Promise` | Yes | +| Grep | `searchContent(content, options)` | `search` | `SearchResult` | No | +| Grep | `hasMatch(content, pattern, opts?)` | `hasMatch` | `boolean` | No | +| Grep | `fuzzyFind(options)` | `fuzzyFind` | `Promise` | Yes | +| Glob | `glob(options, onMatch?)` | `glob` | `Promise` | Yes | +| Glob | `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `void` | No | +| Shell | `executeShell(options, onChunk?)` | `executeShell` | `Promise` | Yes | +| Shell | `Shell` | `Shell` | class constructor | N/A | +| PTY | `PtySession` | `PtySession` | class constructor | N/A | +| Text | `truncateToWidth(...)` | `truncateToWidth` | `string` | No | +| Text | `sliceWithWidth(...)` | `sliceWithWidth` | `SliceWithWidthResult` | No | +| Text | `visibleWidth(text)` | `visibleWidth` | `number` | No | +| Highlight | `highlightCode(code, lang, colors)` | `highlightCode` | `string` | No | +| HTML | `htmlToMarkdown(html, options?)` | `htmlToMarkdown` | `Promise` | Yes | +| System | `getSystemInfo()` | `getSystemInfo` | `SystemInfo` | No | +| Work | `getWorkProfile(lastSeconds)` | `getWorkProfile` | `WorkProfile` | No | +| Process | `killTree(pid, signal)` | `killTree` | `number` | No | +| Process | `listDescendants(pid)` | `listDescendants` | `number[]` | No | +| Clipboard | `copyToClipboard(text)` | `copyToClipboard` | `Promise` (best effort wrapper behavior) | Yes | +| Clipboard | `readImageFromClipboard()` | `readImageFromClipboard` | `Promise` | Yes | +| Keys | `parseKey(data, kittyProtocolActive)` | `parseKey` | `string \| null` | No | + +## Sync vs async contract differences + +The contract mixes sync and async APIs; wrappers preserve native call style rather than forcing one model: + +- **Promise-based async exports** for I/O or long-running work (`grep`, `glob`, `htmlToMarkdown`, `executeShell`, clipboard, image operations). +- **Synchronous exports** for deterministic in-memory transforms/parsers (`search`, `hasMatch`, highlighting, text width/slicing, key parsing, process queries). +- **Constructor exports** for stateful runtime objects (`Shell`, `PtySession`, `PhotonImage`). + +Implication for maintainers: changing sync ↔ async for an existing export is a breaking API and contract change across wrappers and callers. + +## Object and enum typing patterns + +### Object patterns (`#[napi(object)]`-style JS objects) + +TS models object-shaped native values as interfaces, for example: + +- `GrepResult`, `SearchResult`, `GlobResult` +- `SystemInfo`, `WorkProfile` +- `ClipboardImage`, `ParsedKittyResult` + +These are structural contracts at compile time; runtime shape correctness is owned by native implementation. + +### Enum patterns + +Numeric native enums are represented as `const enum` values in TS: + +- `FileType` (`1=file`, `2=dir`, `3=symlink`) +- `ImageFormat` (`0=PNG`, `1=JPEG`, `2=WEBP`, `3=GIF`) +- `SamplingFilter`, `Ellipsis`, `KeyEventType` + +Callers see named enum members; the binding boundary passes numbers. + +## How mismatches are caught + +Mismatch detection happens at two layers: + +1. **Compile-time TypeScript contract checks** + - Wrappers call `native.` against merged `NativeBindings`. + - Missing/renamed binding keys break TS type-checking in wrappers. + +2. **Runtime validation in `validateNative`** + - After load, `native.ts` checks required exports and throws if any are missing. + - Error message includes missing keys and rebuild instruction. + +This catches the common stale-binary drift: wrapper/type exists but loaded `.node` lacks the export. + +## Failure behavior and caveats + +### Load/validation failures (hard failures) + +- Addon load failure or unsupported platform throws during module init in `native.ts`. +- Missing required exports throws before wrappers are usable. + +Effect: package fails fast rather than deferring failure to first call. + +### Wrapper-level behavior differences + +- Some wrappers intentionally soften failures (`copyToClipboard` is best effort and swallows native failure). +- Streaming callbacks ignore callback error payloads and only forward successful value events. + +### Type-level caveats (runtime stricter than TS) + +- TS optional fields do not guarantee semantic validity; native layer can still reject malformed values. +- `const enum` typing does not prevent out-of-range numeric values from untyped callers at runtime. +- `validateNative` checks only presence/function-ness of required exports, not deep argument/return-shape compatibility. +- `bindings.ts` includes `cancelWork(id)` in the base interface, but current runtime validation list does not enforce that key. + +## Maintainer checklist for binding changes + +When adding/changing an export, update all of: + +1. `src//types.ts` (augmentation + contract types) +2. `src//index.ts` (wrapper behavior) +3. `src/native.ts` imports for the module types (if new module) +4. `validateNative` required export checks +5. `src/index.ts` public re-exports + +Skipping any step creates either compile-time drift or runtime load-time failure. \ No newline at end of file diff --git a/docs/natives-build-release-debugging.md b/docs/natives-build-release-debugging.md new file mode 100644 index 000000000..94e997b67 --- /dev/null +++ b/docs/natives-build-release-debugging.md @@ -0,0 +1,237 @@ +# Natives Build, Release, and Debugging Runbook + +This runbook describes how the `@oh-my-pi/pi-natives` build pipeline produces `.node` addons, how compiled distributions load them, and how to debug loader/build failures. + +It follows the architecture terms from `docs/natives-architecture.md`: +- **build-time artifact production** (`scripts/build-native.ts`) +- **embedded addon manifest generation** (`scripts/embed-native.ts`) +- **runtime addon loading + validation gate** (`src/native.ts`) + +## Implementation files + +- `packages/natives/scripts/build-native.ts` +- `packages/natives/scripts/embed-native.ts` +- `packages/natives/package.json` +- `packages/natives/src/native.ts` +- `crates/pi-natives/Cargo.toml` + +## Build pipeline overview + +### 1) Build entrypoints + +`packages/natives/package.json` scripts: + +- `bun scripts/build-native.ts` (`build:native`) → release build +- `bun scripts/build-native.ts --dev` (`dev:native`) → debug/dev build +- `bun scripts/embed-native.ts` (`embed:native`) → generate `src/embedded-addon.ts` from built files + +### 2) Rust artifact build + +`build-native.ts` runs Cargo in `crates/pi-natives`: + +- base command: `cargo build` +- release mode adds `--release` unless `--dev` is passed +- cross target adds `--target ` + +`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`, so Cargo emits a shared library (`.so`/`.dylib`/`.dll`) that is then copied/renamed to a `.node` addon filename. + +### 3) Artifact discovery and install + +After Cargo completes, `build-native.ts` scans candidate output directories in order: + +1. `${CARGO_TARGET_DIR}` (if set) +2. `/target` +3. `crates/pi-natives/target` + +For each root it checks profile directories: +- cross build: `//` then `/` +- native build: `/` + +Then it looks for one of: +- `libpi_natives.so` +- `libpi_natives.dylib` +- `pi_natives.dll` +- `libpi_natives.dll` + +When found, it atomically installs into `packages/natives/native/` with temp-file + rename semantics (Windows fallback handles locked DLL replacement failures explicitly). + +## Target/variant model and naming conventions + +## Platform tag + +Both build and runtime use platform tag: + +`-` (example: `darwin-arm64`, `linux-x64`) + +## Variant model (x64 only) + +x64 supports CPU variants: +- `modern` (AVX2-capable path) +- `baseline` (fallback) + +Non-x64 uses a single default artifact (no variant suffix). + +### Output filenames + +Release builds: +- x64: `pi_natives.--modern.node` or `...-baseline.node` +- non-x64: `pi_natives.-.node` + +Dev build (`--dev`): +- `pi_natives.dev.node` + +Runtime loader candidate order in `native.ts`: +- if `PI_DEV` is set: try `pi_natives.dev.node` first +- then release candidates +- compiled mode prepends extracted/cache candidates before package-local files + +## Environment flags and build options + +## Runtime flags + +- `PI_DEV` (loader behavior): prefer dev addon candidates first +- `PI_NATIVE_VARIANT` (loader behavior, x64 only): force `modern` or `baseline` selection at runtime +- `PI_COMPILED` (loader behavior): enable compiled-binary candidate/extraction behavior + +## Build-time flags/options + +- `--dev` (script arg): build debug profile and emit `pi_natives.dev.node` +- `CROSS_TARGET`: passed to Cargo `--target` +- `TARGET_PLATFORM`: override output platform tag naming +- `TARGET_ARCH`: override output arch naming +- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy +- `CARGO_TARGET_DIR`: additional root when searching Cargo outputs +- `RUSTFLAGS`: + - if unset and not cross-compiling, script sets: + - modern: `-C target-cpu=x86-64-v3` + - baseline: `-C target-cpu=x86-64-v2` + - non-x64 / no variant: `-C target-cpu=native` + - if already set, script does not override + +## Build state/lifecycle transitions + +### Build lifecycle (`build-native.ts`) + +1. **Init**: parse args/env (`--dev`, target overrides, cross flags) +2. **Variant resolve**: + - non-x64 → no variant + - x64 + `TARGET_VARIANT` → explicit variant + - x64 cross-build without `TARGET_VARIANT` → hard error + - x64 local build without override → detect host AVX2 +3. **Compile**: run Cargo with resolved profile/target +4. **Locate artifact**: scan target roots/profile dirs/library names +5. **Install**: copy + atomic rename into `packages/natives/native` +6. **Complete**: output addon ready for loader candidates + +Failure exits happen at any stage with explicit error text (invalid variant, failed cargo build, missing output library, install/rename failure). + +### Embed lifecycle (`embed-native.ts`) + +1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values +2. **Candidate set**: + - x64 expects both `modern` and `baseline` + - non-x64 expects one default file +3. **Validate availability** in `packages/natives/native` +4. **Generate manifest** (`src/embedded-addon.ts`) with Bun `file` imports and package version +5. **Runtime extraction ready** for compiled mode + +`--reset` bypasses validation and writes a null manifest stub (`embeddedAddon = null`). + +## Dev workflow vs shipped/compiled behavior + +## Local development workflow + +Typical local loop: + +1. Build addon: + - release: `bun --cwd=packages/natives run build:native` + - debug: `bun --cwd=packages/natives run dev:native` +2. Set `PI_DEV=1` when testing debug addon loading +3. Loader in `native.ts` resolves package-local `native/` (and executable-dir fallback) candidates +4. `validateNative` enforces export compatibility before wrappers use the binding + +## Shipped/compiled binary workflow + +In compiled mode (`PI_COMPILED` or Bun embedded markers): + +1. Loader computes versioned cache dir: `/` (operationally `~/.omp/natives/`) +2. If embedded manifest matches current platform+version, loader may extract selected embedded file into that versioned dir +3. Runtime candidate order includes: + - versioned cache dir + - legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere) + - package/executable directories +4. First successfully loaded addon still must pass `validateNative` + +This is why packaging + runtime loader expectations must align: filenames, platform tags, and exported symbols must match what `native.ts` probes and validates. + +## JS API ↔ Rust export mapping (validation gate subset) + +`native.ts` requires these JS-visible exports to exist on the loaded addon. They map to Rust N-API exports in `crates/pi-natives/src`: + +| JS name required by `validateNative` | Rust export declaration | Rust source file | +| --- | --- | --- | +| `glob` | `#[napi(js_name = "glob")] pub fn glob(...)` | `crates/pi-natives/src/glob.rs` | +| `grep` | `#[napi(js_name = "grep")] pub fn grep(...)` | `crates/pi-natives/src/grep.rs` | +| `search` | `#[napi(js_name = "search")] pub fn search(...)` | `crates/pi-natives/src/grep.rs` | +| `highlightCode` | `#[napi(js_name = "highlightCode")] pub fn highlight_code(...)` | `crates/pi-natives/src/highlight.rs` | +| `getSystemInfo` | `#[napi(js_name = "getSystemInfo")] pub fn get_system_info(...)` | `crates/pi-natives/src/system_info.rs` | +| `getWorkProfile` | `#[napi] pub fn get_work_profile(...)` (camel-cased export) | `crates/pi-natives/src/prof.rs` | +| `invalidateFsScanCache` | `#[napi(js_name = "invalidateFsScanCache")] pub fn invalidate_fs_scan_cache(...)` | `crates/pi-natives/src/fs_cache.rs` | + +If any required symbol is missing, loader fails fast with a rebuild hint. + +## Failure behavior and diagnostics + +## Build-time failures + +- Invalid variant configuration: + - `TARGET_VARIANT` set on non-x64 → immediate error + - x64 cross-build without explicit `TARGET_VARIANT` → immediate error +- Cargo build failure: + - script surfaces non-zero exit and stderr +- Artifact not found: + - script prints every checked profile directory +- Install failure: + - explicit message; Windows includes locked-file hint + +## Runtime loader failures (`native.ts`) + +- Unsupported platform tag: + - throws with supported platform list +- No candidate could load: + - throws with full candidate error list and mode-specific remediation hints +- Missing exports: + - throws with exact missing symbol names and rebuild command +- Embedded extraction problems: + - extraction mkdir/write errors recorded and included in final diagnostics + +## Troubleshooting matrix + +| Symptom | Likely cause | Verify | Fix | +| --- | --- | --- | --- | +| `Native addon missing exports ... Missing: ` | Stale `.node` binary, Rust export name mismatch, or wrong binary loaded | Run with `PI_DEV=1` to see loaded path; inspect export list for that file | Rebuild `build:native`; ensure Rust `#[napi(js_name=...)]` matches JS name; remove stale cached/versioned files | +| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or only baseline file present | Check `PI_NATIVE_VARIANT`; inspect `native/` for `-modern` file | Build modern variant (`TARGET_VARIANT=modern ... build:native`) and ensure file is shipped | +| Cross-build produces unusable/wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing `TARGET_VARIANT` for x64 | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` | +| Compiled binary fails after upgrade | Stale extracted cache (`~/.omp/natives/`) or embedded manifest mismatch | Inspect versioned natives dir and loader error list | Delete versioned natives cache for the package version and rerun; regenerate embedded manifest during packaging | +| Loader probes many paths and none work | Platform mismatch or missing release artifact in package `native/` | Check `platformTag` vs actual filename(s) | Ensure built filename exactly matches `pi_natives.-(-variant).node` convention and package includes `native/` | +| `embed:native` fails with "Incomplete native addons" | Required variant files not built before embedding | Check expected vs found list in error text | Build required files first (x64: both modern+baseline; non-x64: default), then rerun `embed:native` | + +## Operational commands + +```bash +# Release artifact for current host +bun --cwd=packages/natives run build:native + +# Debug artifact (load first when PI_DEV=1) +bun --cwd=packages/natives run dev:native + +# Build explicit x64 variants +TARGET_VARIANT=modern bun --cwd=packages/natives run build:native +TARGET_VARIANT=baseline bun --cwd=packages/natives run build:native + +# Generate embedded addon manifest from built native files +bun --cwd=packages/natives run embed:native + +# Reset embedded manifest to null stub +bun --cwd=packages/natives run embed:native -- --reset +``` \ No newline at end of file diff --git a/docs/natives-media-system-utils.md b/docs/natives-media-system-utils.md new file mode 100644 index 000000000..5538e0554 --- /dev/null +++ b/docs/natives-media-system-utils.md @@ -0,0 +1,202 @@ +# Natives media + system utilities + +This document is a subsystem deep-dive for the **system/media/conversion primitives** layer described in [`docs/natives-architecture.md`](./natives-architecture.md): `image`, `html`, `clipboard`, `system-info`, and `work` profiling. + +## Implementation files + +- `crates/pi-natives/src/image.rs` +- `crates/pi-natives/src/html.rs` +- `crates/pi-natives/src/clipboard.rs` +- `crates/pi-natives/src/system_info.rs` +- `crates/pi-natives/src/prof.rs` +- `crates/pi-natives/src/task.rs` +- `packages/natives/src/image/index.ts` +- `packages/natives/src/image/types.ts` +- `packages/natives/src/html/index.ts` +- `packages/natives/src/html/types.ts` +- `packages/natives/src/clipboard/index.ts` +- `packages/natives/src/clipboard/types.ts` +- `packages/natives/src/system-info/index.ts` +- `packages/natives/src/system-info/types.ts` +- `packages/natives/src/work/index.ts` +- `packages/natives/src/work/types.ts` + +> Note: there is no `crates/pi-natives/src/work.rs`; work profiling is implemented in `prof.rs` and fed by instrumentation in `task.rs`. + +## TS API ↔ Rust export/module mapping + +| TS export (packages/natives) | Rust N-API export | Rust module | +| --- | --- | --- | +| `PhotonImage.parse(bytes)` | `PhotonImage::parse` (`js_name = "parse"`) | `image.rs` | +| `PhotonImage#resize(width, height, filter)` | `PhotonImage::resize` (`js_name = "resize"`) | `image.rs` | +| `PhotonImage#encode(format, quality)` | `PhotonImage::encode` (`js_name = "encode"`) | `image.rs` | +| `htmlToMarkdown(html, options)` | `html_to_markdown` (`js_name = "htmlToMarkdown"`) | `html.rs` | +| `copyToClipboard(text)` | `copy_to_clipboard` (`js_name = "copyToClipboard"`) + TS fallback logic | `clipboard.rs` + `clipboard/index.ts` | +| `readImageFromClipboard()` | `read_image_from_clipboard` (`js_name = "readImageFromClipboard"`) | `clipboard.rs` | +| `getSystemInfo()` | `get_system_info` (`js_name = "getSystemInfo"`) | `system_info.rs` | +| `getWorkProfile(lastSeconds)` | `get_work_profile` | `prof.rs` | + +## Data format boundaries and conversions + +### Image (`image`) + +- **JS input boundary**: `Uint8Array` encoded image bytes. +- **Rust decode boundary**: bytes are copied to `Vec`, format is guessed with `ImageReader::with_guessed_format()`, then decoded to `DynamicImage`. +- **In-memory state**: `PhotonImage` stores `Arc`. +- **Output boundary**: `encode(format, quality)` returns `Promise` (Rust `Vec`). + +Format IDs are numeric: + +- `0`: PNG +- `1`: JPEG +- `2`: WebP (lossless encoder) +- `3`: GIF + +Constraints: + +- `quality` is only used for JPEG. +- PNG/WebP/GIF ignore `quality`. +- Unsupported format IDs fail (`Invalid image format: `). + +### HTML conversion (`html`) + +- **JS input boundary**: HTML `string` + optional object `{ cleanContent?: boolean; skipImages?: boolean }`. +- **Rust conversion boundary**: `String` input is converted by `html_to_markdown_rs::convert`. +- **Output boundary**: Markdown `string`. + +Conversion behavior: + +- `cleanContent` defaults to `false`. +- When `cleanContent=true`, preprocessing is enabled with `PreprocessingPreset::Aggressive` and hard-removal flags for navigation/forms. +- `skipImages` defaults to `false`. + +### Clipboard (`clipboard`) + +- **Text path**: + - TS first emits OSC 52 (`\x1b]52;c;\x07`) when stdout is a TTY. + - Same text is then attempted via native clipboard API (`native.copyToClipboard`) as best-effort. + - On Termux, TS attempts `termux-clipboard-set` first. +- **Image read path**: + - Rust reads raw image from `arboard`. + - Rust re-encodes it to PNG bytes (`image` crate), returns `{ data: Uint8Array, mimeType: "image/png" }`. + - TS returns `null` early on Termux or Linux sessions without display server (`DISPLAY`/`WAYLAND_DISPLAY` missing). + +### System info (`system-info`) + +- **Output boundary**: plain object returned synchronously. +- Rust currently populates: `distro`, `kernel`, `cpu`, `disk`. +- Linux distro comes from `/etc/os-release` parsing; macOS may append a marketing name (`Tahoe`, `Sequoia`, etc.) to OS version text. +- Disk summary is normalized to human-readable strings (`used/total (pct%)`), with platform-dependent selection: + - Windows: aggregates each mount entry. + - non-Windows: prefers `/`, falls back to first disk. + +### Work profiling (`work`) + +- **Collection boundary**: profiling samples are produced by `profile_region(tag)` guards in `task::blocking` and `task::future`. +- **Storage format**: fixed-size circular buffer (`MAX_SAMPLES = 10_000`) storing stack path + duration (`μs`) + timestamp (`μs since process start`). +- **Output boundary**: `getWorkProfile(lastSeconds)` returns object: + - `folded`: folded-stack text (flamegraph input) + - `summary`: markdown table summary + - `svg`: optional flamegraph SVG + - `totalMs`, `sampleCount` + +## Lifecycle and state transitions + +### Image lifecycle + +1. `PhotonImage.parse(bytes)` schedules a blocking decode task (`image.decode`). +2. On success, a native `PhotonImage` handle exists in JS. +3. `resize(...)` creates a new native handle (`image.resize`), old and new handles can coexist. +4. `encode(...)` materializes bytes (`image.encode`) without mutating image dimensions. + +Failure transitions: + +- Format detection/decode failure rejects parse promise. +- Encode failure rejects encode promise. +- Invalid format ID rejects encode promise. + +### HTML lifecycle + +1. `htmlToMarkdown(html, options)` schedules a blocking conversion task. +2. Conversion runs with defaulted options (`cleanContent=false`, `skipImages=false`) unless specified. +3. Returns markdown string or rejects. + +Failure transitions: + +- Converter failure returns rejected promise (`Conversion error: ...`). + +### Clipboard lifecycle + +`copyToClipboard(text)` is intentionally best-effort and multi-path: + +1. If TTY: attempt OSC 52 write (base64 payload). +2. Try Termux command when `TERMUX_VERSION` is set. +3. Try native `arboard` text copy. +4. Swallow errors at TS layer. + +`readImageFromClipboard()` strictness differs by stage: + +1. TS hard-gates unsupported runtime contexts (Termux/headless Linux) to `null`. +2. Rust `arboard` read runs only when TS allows it. +3. `ContentNotAvailable` maps to `null`. +4. Other Rust errors reject. + +### System info lifecycle + +1. `getSystemInfo()` refreshes `sysinfo::System` and disk list synchronously. +2. Per-platform helpers derive distro/kernel/cpu/disk snapshots. +3. Object is returned directly; no async task scheduling. + +Failure transitions: + +- Missing optional data degrades to omitted fields (`Option::None`), not thrown errors. +- `/etc/os-release` parse failures are soft-fail (`None` distro). +- Disk total space `0` is treated as unavailable (`None` disk). + +### Work profiling lifecycle + +1. No explicit start: profiling is always on when task helpers execute. +2. Every instrumented task scope records one sample on guard drop. +3. Samples overwrite oldest entries after buffer capacity is reached. +4. `getWorkProfile(lastSeconds)` reads a time window and derives folded/summary/svg artifacts. + +Failure transitions: + +- SVG generation failure is soft-fail (`svg: null`), while folded and summary still return. +- Empty sample window returns empty folded data and `svg: null`, not an error. + +## Unsupported operations and error propagation + +### Image + +- Unsupported decode input or corrupted bytes: strict failure (promise rejection). +- Unsupported encode format ID: strict failure. +- No best-effort fallback path in TS wrapper. + +### HTML + +- Conversion errors are strict failures (rejection). +- Option omission is best-effort defaulting, not failure. + +### Clipboard + +- Text copy is best-effort at TS layer: operational failures are suppressed. +- Image read distinguishes "no image" (`null`) from operational failure (rejection). +- Termux/headless Linux are treated as unsupported contexts for image read (`null`). + +### System info + +- Designed for partial success: fields are optional and may be absent by platform. +- Current TS type is broader than current Rust-populated fields; maintainers should expect sparse payloads unless Rust expands output. + +### Work profiling + +- Retrieval is strict for function call itself, but artifact generation is partially best-effort (`svg` nullable). +- Buffer truncation is expected behavior (ring buffer), not data loss bug. + +## Platform caveats + +- **Clipboard text**: OSC 52 depends on terminal support; native clipboard access depends on desktop environment/session. +- **Clipboard image read**: blocked in TS for Termux and Linux without display server. +- **System info distro**: Linux distro name quality depends on `/etc/os-release` fields; macOS marketing name mapping is version-table-based and may lag new releases. +- **Disk reporting**: Windows returns a comma-separated multi-volume summary; non-Windows returns one primary mount summary. diff --git a/docs/natives-rust-task-cancellation.md b/docs/natives-rust-task-cancellation.md new file mode 100644 index 000000000..813a26fbe --- /dev/null +++ b/docs/natives-rust-task-cancellation.md @@ -0,0 +1,208 @@ +# Native Rust task execution and cancellation (`pi-natives`) + +This document describes how `crates/pi-natives` schedules native work and how cancellation flows from JS options (`timeoutMs`, `AbortSignal`) to Rust execution. + +## Implementation files + +- `crates/pi-natives/src/task.rs` +- `crates/pi-natives/src/grep.rs` +- `crates/pi-natives/src/glob.rs` +- `crates/pi-natives/src/fd.rs` +- `crates/pi-natives/src/shell.rs` +- `crates/pi-natives/src/pty.rs` +- `crates/pi-natives/src/html.rs` +- `crates/pi-natives/src/image.rs` +- `crates/pi-natives/src/clipboard.rs` +- `crates/pi-natives/src/text.rs` +- `crates/pi-natives/src/ps.rs` + +## Core primitives (`task.rs`) + +`task.rs` defines three core pieces: + +1. `task::blocking(tag, cancel_token, work)` + - Wraps `napi::AsyncTask` / `Task`. + - `compute()` runs on libuv worker threads (for CPU-bound or blocking/sync system calls). + - Returns a JS `Promise`. + +2. `task::future(env, tag, work)` + - Wraps `env.spawn_future(...)`. + - Runs async work on Tokio runtime. + - Returns `PromiseRaw<'env, T>`. + +3. `CancelToken` / `AbortToken` / `AbortReason` + - `CancelToken::new(timeout_ms, signal)` combines deadline + optional `AbortSignal`. + - `CancelToken::heartbeat()` is cooperative cancellation for blocking loops. + - `CancelToken::wait()` is async cancellation wait (`Signal` / `Timeout` / `User` Ctrl-C). + - `AbortToken` lets external code request abort (`abort(reason)`). + +## `blocking` vs `future`: execution model and selection + +### Use `task::blocking` + +Use when work is CPU-heavy or fundamentally synchronous/blocking: + +- regex/file scanning (`grep`, `glob`, `fuzzy_find`) +- synchronous PTY loop internals (`run_pty_sync` via `spawn_blocking`) +- clipboard/image/html conversions + +Behavior: + +- Work closure receives a cloned `CancelToken`. +- Cancellation is only observed where code checks `ct.heartbeat()?`. +- Closure `Err(...)` rejects JS promise. + +### Use `task::future` + +Use when work must `await` async operations: + +- shell session orchestration (`shell.run`, `executeShell`) +- task racing (`tokio::select!`) between completion and cancellation + +Behavior: + +- Future can race normal completion against `ct.wait()`. +- On cancel path, async implementations typically propagate cancellation to inner subsystems (e.g., `tokio_util::CancellationToken`) and optionally force abort on grace timeout. + +## JS API ↔ Rust export mapping (task/cancel relevant) + +| JS-facing API | Rust export (`#[napi]`) | Scheduler | Cancellation hookup | +|---|---|---|---| +| `grep(options, onMatch?)` | `grep` | `task::blocking("grep", ct, ...)` | `CancelToken::new(options.timeoutMs, options.signal)` + `ct.heartbeat()` | +| `glob(options, onMatch?)` | `glob` | `task::blocking("glob", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in filter loop | +| `fuzzyFind(options)` | `fuzzy_find` (`js_name = "fuzzyFind"`) | `task::blocking("fuzzy_find", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in scoring loop | +| `shell.run(options, onChunk?)` | `Shell::run` | `task::future(env, "shell.run", ...)` | `ct.wait()` raced against run task; bridges to Tokio `CancellationToken` | +| `executeShell(options, onChunk?)` | `execute_shell` (`js_name = "executeShell"`) | `task::future(env, "shell.execute", ...)` | same as above | +| `pty.start(options, onChunk?)` | `PtySession::start` | `task::future(env, "pty.start", ...)` + inner `spawn_blocking` | `CancelToken` checked in sync PTY loop via `heartbeat()` | +| `htmlToMarkdown(html, options?)` | `html_to_markdown` (`js_name = "htmlToMarkdown"`) | `task::blocking("html_to_markdown", (), ...)` | none (`()` token) | +| `PhotonImage.parse/encode/resize` | `PhotonImage::{parse,encode,resize}` | `task::blocking(...)` | none (`()` token) | +| `copyToClipboard/readImageFromClipboard` | `copy_to_clipboard` / `read_image_from_clipboard` | `task::blocking(...)` | none (`()` token) | + +`text.rs` and `ps.rs` currently do not use `task::blocking`/`task::future` and therefore do not participate in this cancellation path. + +## Cancellation lifecycle and state transitions + +### `CancelToken` lifecycle + +`CancelToken` is cooperative and stateful: + +```text +Created + ├─ no signal + no timeout -> passive token (never aborts unless externally emplaced) + ├─ signal registered -> waits for AbortSignal callback + └─ deadline set -> timeout check becomes active + +Running + ├─ heartbeat()/wait() sees signal -> AbortReason::Signal + ├─ heartbeat()/wait() sees deadline -> AbortReason::Timeout + ├─ wait() sees Ctrl-C -> AbortReason::User + └─ no abort -> continue + +Aborted (terminal) + └─ first abort reason wins (atomic flag + notifier) +``` + +### Before-start vs mid-execution cancellation + +- **Before start / before first cancellation check**: + - `task::future` users that race on `ct.wait()` can resolve cancel immediately once they enter `select!`. + - `task::blocking` users only observe cancellation when closure code reaches `heartbeat()`. If closure does not heartbeat early, cancellation is delayed. + +- **Mid-execution**: + - `blocking`: next `heartbeat()` returns `Err("Aborted: ...")`. + - `future`: `ct.wait()` branch wins `select!`, then code cancels subordinate async machinery (for shell: cancels Tokio token, waits up to 2s, then aborts task). + +## Heartbeat expectations for long-running loops + +`heartbeat()` must run at predictable cadence in loops with unbounded or large work sets. + +Observed patterns: + +- `glob::filter_entries`: check each entry before filtering/matching. +- `fd::score_entries`: check each scanned candidate. +- `grep_sync`: explicit cancellation check before heavy search phase, plus fs-cache calls that also receive token. +- `run_pty_sync`: check every loop tick (~16ms sleep cadence) and kill child on cancellation. + +Practical rule: no loop over external-size input should exceed a short bounded interval without a heartbeat. + +## Failure behavior and error propagation to JS + +### Blocking tasks + +Error path: + +1. Closure returns `Err(napi::Error)` (including `heartbeat()` abort). +2. `Task::compute()` returns `Err`. +3. `AsyncTask` rejects JS promise. + +Typical error strings: + +- `Aborted: Timeout` +- `Aborted: Signal` +- domain errors (`Failed to decode image: ...`, `Conversion error: ...`, etc.) + +### Future tasks + +Error path: + +1. Async body returns `Err(napi::Error)` or join failure is mapped (`... task failed: {err}`). +2. `task::future`-spawned promise rejects. +3. Some APIs intentionally return structured cancellation results instead of rejection (`ShellRunResult`/`ShellExecuteResult` with `cancelled`/`timed_out` flags and `exit_code: None`). + +### Cancellation reporting split + +- **Abort as error**: most blocking exports using `heartbeat()?`. +- **Abort as typed result**: shell/pty style command APIs that model cancellation in result structs. + +Choose one model per API and document it explicitly. + +## Common pitfalls + +1. **Missing heartbeat in blocking loops** + - Symptom: timeout/signal appears ignored until loop ends. + - Fix: add `ct.heartbeat()?` at loop top and before expensive per-item steps. + +2. **Long uncancelable sections** + - Symptom: cancellation latency spikes during single large call (decode, sort, compression, etc.). + - Fix: split work into chunks with heartbeat boundaries; if impossible, document latency. + +3. **Blocking async executor** + - Symptom: async API stalls when sync-heavy code runs directly in future. + - Fix: move CPU/sync blocks to `task::blocking` or `tokio::task::spawn_blocking`. + +4. **Inconsistent cancel semantics** + - Symptom: one API rejects on cancel, another resolves with flags, confusing callers. + - Fix: standardize per domain and keep wrapper docs aligned. + +5. **Forgetting cancellation bridge in nested async tasks** + - Symptom: outer token is cancelled but inner readers/subprocess tasks keep running. + - Fix: bridge cancellation to inner token/signal and enforce grace timeout + forced abort fallback. + +## Checklist for new cancellable exports + +1. Classify work correctly: + - CPU-bound or sync blocking -> `task::blocking` + - async I/O / `await` orchestration -> `task::future` + +2. Expose cancel inputs when needed: + - include `timeoutMs` and `signal` in `#[napi(object)]` options + - create `let ct = task::CancelToken::new(timeout_ms, signal);` + +3. Wire cancellation through all layers: + - blocking loops: `ct.heartbeat()?` at stable intervals + - async orchestration: race with `ct.wait()` and cancel sub-tasks/tokens + +4. Decide cancellation contract: + - reject promise with abort error, or + - resolve typed `{ cancelled, timedOut, ... }` + - keep this contract consistent for the API family + +5. Propagate failures with context: + - map errors via `Error::from_reason(format!("...: {err}"))` + - include stage-specific prefixes (`spawn`, `decode`, `wait`, etc.) + +6. Handle before-start and mid-flight cancellation: + - cancellation check/await must happen before expensive body and during long execution + +7. Validate no executor misuse: + - no long sync work directly inside async futures without `spawn_blocking`/blocking task wrapper diff --git a/docs/natives-shell-pty-process.md b/docs/natives-shell-pty-process.md new file mode 100644 index 000000000..a474c4228 --- /dev/null +++ b/docs/natives-shell-pty-process.md @@ -0,0 +1,268 @@ +# Natives Shell, PTY, Process, and Key Internals + +This document covers the **execution/process/terminal primitives** in `@oh-my-pi/pi-natives`: `shell`, `pty`, `ps`, and `keys`, using the architecture terms from `docs/natives-architecture.md`. + +## Implementation files + +- `crates/pi-natives/src/shell.rs` +- `crates/pi-natives/src/shell/windows.rs` (Windows only) +- `crates/pi-natives/src/pty.rs` +- `crates/pi-natives/src/ps.rs` +- `crates/pi-natives/src/keys.rs` +- `crates/pi-natives/src/task.rs` (shared cancellation behavior used by shell/pty) +- `packages/natives/src/shell/index.ts` +- `packages/natives/src/shell/types.ts` +- `packages/natives/src/pty/index.ts` +- `packages/natives/src/pty/types.ts` +- `packages/natives/src/ps/index.ts` +- `packages/natives/src/ps/types.ts` +- `packages/natives/src/keys/index.ts` +- `packages/natives/src/keys/types.ts` +- `packages/natives/src/bindings.ts` + +## Layer ownership + +- **TS wrapper/API layer** (`packages/natives/src/*`): typed entrypoints, cancellation surface (`timeoutMs`, `AbortSignal`), and JS ergonomics. +- **Rust N-API module layer** (`crates/pi-natives/src/*`): shell/PTY process execution, process-tree traversal/termination, and key-sequence parsing. +- **Validation gate** (`native.ts`, architecture-level): ensures required exports (`Shell`, `executeShell`, `PtySession`, `killTree`, `listDescendants`, key helpers) exist before wrappers are used. + +## Shell subsystem (`shell`) + +### API model + +Two execution modes are exposed: + +1. **One-shot** via `executeShell(options, onChunk?)`. +2. **Persistent session** via `new Shell(options?)` then `shell.run(...)` repeatedly. + +Both stream output through a threadsafe callback and return `{ exitCode?, cancelled, timedOut }`. + +### Session creation and environment model + +Rust creates `brush_core::Shell` with: + +- non-interactive mode, +- `do_not_inherit_env: true`, +- explicit environment reconstruction from host env, +- skip-list for shell-sensitive vars (`PS1`, `PWD`, `SHLVL`, bash function exports, etc.). + +Session env behavior: + +- `ShellOptions.sessionEnv` is applied once at session creation. +- `ShellRunOptions.env` is command-scoped (`EnvironmentScope::Command`) and popped after each run. +- `PATH` is merged specially on Windows with case-insensitive dedupe. + +Windows-only path enrichment (`shell/windows.rs`): discovered Git-for-Windows paths (`cmd`, `bin`, `usr/bin`) are appended if present and not already included. + +### Runtime lifecycle and state transitions + +Persistent shell (`Shell.run`) uses this state machine: + +- **Idle/Uninitialized**: `session: None`. +- **Running**: first `run()` lazily creates session, stores `current_abort` token, executes command. +- **Completed + keepalive**: if execution control flow is `Normal`, `current_abort` is cleared and session is reused. +- **Completed + teardown**: if control flow is loop/script/shell-exit related (`BreakLoop`, `ContinueLoop`, `ReturnFromFunctionOrScript`, `ExitShell`), session is dropped (`session: None`). +- **Cancelled/Timed out**: run task is cancelled, grace wait (2s), then force-abort; session is dropped. +- **Error**: session is dropped. + +One-shot shell (`executeShell`) always creates and drops a fresh session per call. + +### Streaming/output behavior + +- Stdout/stderr are routed into a shared pipe and read concurrently. +- Reader decodes UTF-8 incrementally; invalid byte sequences emit `U+FFFD` replacement chunks. +- After process completion, output drain has idle/max guards (`250ms` idle, `2s` max) to avoid hanging on background jobs keeping descriptors open. + +### Cancellation, timeout, and background jobs + +- `CancelToken` is constructed from `timeoutMs` and optional `AbortSignal`. +- On cancellation/timeout, shell cancellation token is triggered, then task gets a 2s graceful window before forced abort. +- If cancellation occurs, background jobs are terminated (`TERM`, then delayed `KILL`) using brush job metadata. + +`Shell.abort()` behavior: + +- aborts only current running command for that `Shell` instance, +- no-op success when nothing is running. + +### Failure behavior + +Common surfaced errors include: + +- session init failures (`Failed to initialize shell`), +- cwd errors (`Failed to set cwd`), +- env set/pop failures, +- snapshot source failures, +- pipe creation/clone failures, +- execution failure (`Shell execution failed: ...`), +- task wrapper failures (`Shell execution task failed: ...`). + +Result-level cancellation flags: + +- timeout -> `exitCode: undefined`, `timedOut: true`. +- abort signal -> `exitCode: undefined`, `cancelled: true`. + +## PTY subsystem (`pty`) + +### API model + +`new PtySession()` exposes: + +- `start(options, onChunk?) -> Promise<{ exitCode?, cancelled, timedOut }>` +- `write(data)` +- `resize(cols, rows)` +- `kill()` + +### Runtime lifecycle and state transitions + +`PtySession` state machine: + +- **Idle**: `core: None`. +- **Reserved**: `start()` installs control channel synchronously (`core: Some`) before async work begins, so `write/resize/kill` become immediately valid. +- **Running**: blocking PTY loop handles child state, reader events, cancellation heartbeat, and control messages. +- **Terminal closed**: child exit + reader completion. +- **Finalized**: `core` is always reset to `None` after start task completion (success or error). + +Concurrency guard: + +- starting while already running returns `PTY session already running`. + +### Spawn/attach/write/read/terminate patterns + +- PTY opened via `portable_pty::native_pty_system().openpty(...)`. +- Command currently runs as `sh -lc ` with optional `cwd` and env overrides. +- `write()` sends raw bytes to PTY stdin. +- `resize()` clamps dimensions (`cols 20..400`, `rows 5..200`) and calls master resize. +- `kill()` marks run as cancelled and kills child process. + +Output path: + +- dedicated reader thread reads master stream, +- incremental UTF-8 decode with `U+FFFD` replacement on invalid bytes, +- chunks forwarded through N-API threadsafe callback. + +### Cancellation and timeout semantics + +- `timeoutMs` and `AbortSignal` feed a `CancelToken`. +- loop calls `ct.heartbeat()` periodically; abort triggers child kill. +- timeout classification is string-based (`"Timeout"` substring in heartbeat error). + +### Failure behavior + +Error surfaces include: + +- PTY allocation/open failure, +- PTY spawn failure, +- writer/reader acquisition failure, +- child status/wait failures, +- lock poisoning, +- control-channel disconnection (`PTY session is no longer available`). + +Control call failures when not running: + +- `write/resize/kill` return `PTY session is not running`. + +## Process-tree subsystem (`ps`) + +### API model + +- `killTree(pid, signal) -> number` +- `listDescendants(pid) -> number[]` + +TS wrapper also registers native kill-tree integration into shared utils via `setNativeKillTree(native.killTree)`. + +### Platform-specific implementation + +- **Linux**: recursively reads `/proc//task//children`. +- **macOS**: uses `libproc` `proc_listchildpids`. +- **Windows**: snapshots process table with `CreateToolhelp32Snapshot`, builds parent->children map, terminates with `OpenProcess(PROCESS_TERMINATE)` + `TerminateProcess`. + +### Kill-tree behavior + +- Descendants are collected recursively. +- Kill order is bottom-up (deepest descendants first) to reduce orphan re-parenting. +- Root pid is killed last. +- Return value is count of successful terminations. + +Signal behavior: + +- POSIX: provided `signal` is passed to `kill`. +- Windows: `signal` is ignored; termination is unconditional process terminate. + +### Failure behavior + +This module is intentionally non-throwing at API surface: + +- missing/inaccessible process tree branches are skipped, +- per-pid kill failures are counted as unsuccessful (not errors), +- lookup miss typically yields `[]` from `listDescendants` and `0` from `killTree`. + +## Key parsing subsystem (`keys`) + +### API model + +Exposed helpers: + +- `parseKey(data, kittyProtocolActive)` +- `matchesKey(data, keyId, kittyProtocolActive)` +- `parseKittySequence(data)` +- `matchesKittySequence(data, expectedCodepoint, expectedModifier)` +- `matchesLegacySequence(data, keyName)` + +### Parsing model + +The parser combines: + +- direct single-byte mappings (`enter`, `tab`, `ctrl+`, printable ASCII), +- O(1) legacy escape-sequence lookup (PHF map), +- xterm `modifyOtherKeys` parsing, +- Kitty protocol parsing (`CSI u`, `CSI ~`, `CSI 1;...`), +- normalization to key IDs (`ctrl+c`, `shift+tab`, `pageUp`, `f5`, etc.). + +Modifier handling: + +- only shift/alt/ctrl bits are compared for key matching, +- lock bits are masked out before comparisons. + +Layout behavior: + +- base-layout fallback is intentionally constrained so remapped layouts do not create false matches for ASCII letters/symbols. + +### Failure behavior + +- Unrecognized or invalid sequences produce `null` from parse functions. +- Match functions return `false` on parse failure or mismatch. +- No thrown error surface for malformed key input. + +## JS wrapper API ↔ Rust export mapping + +### Shell + PTY + Process + +| TS wrapper API | Rust N-API export | Notes | +|---|---|---| +| `executeShell(options, onChunk?)` | `executeShell` (`execute_shell`) | One-shot shell execution | +| `new Shell(options?)` | `Shell` class | Persistent shell session | +| `shell.run(options, onChunk?)` | `Shell::run` | Reuses session on keepalive control flow | +| `shell.abort()` | `Shell::abort` | Aborts active run for that shell instance | +| `new PtySession()` | `PtySession` class | Stateful PTY session | +| `pty.start(options, onChunk?)` | `PtySession::start` | Interactive PTY run | +| `pty.write(data)` | `PtySession::write` | Raw stdin passthrough | +| `pty.resize(cols, rows)` | `PtySession::resize` | Clamped terminal dimensions | +| `pty.kill()` | `PtySession::kill` | Force-kills active PTY child | +| `killTree(pid, signal)` | `killTree` (`kill_tree`) | Children-first process tree termination | +| `listDescendants(pid)` | `listDescendants` (`list_descendants`) | Recursive descendants listing | + +### Keys + +| TS wrapper API | Rust N-API export | Notes | +|---|---|---| +| `matchesKittySequence(data, cp, mod)` | `matchesKittySequence` (`matches_kitty_sequence`) | Kitty codepoint+modifier match | +| `parseKey(data, kittyProtocolActive)` | `parseKey` (`parse_key`) | Normalized key-id parser | +| `matchesLegacySequence(data, keyName)` | `matchesLegacySequence` (`matches_legacy_sequence`) | Exact legacy sequence map check | +| `parseKittySequence(data)` | `parseKittySequence` (`parse_kitty_sequence_napi`) | Structured Kitty parse result | +| `matchesKey(data, keyId, kittyProtocolActive)` | `matchesKey` (`matches_key`) | High-level key matcher | + +## Abandoned session cleanup and finalization notes + +- **Shell persistent session**: if a run is cancelled/timed out/errors/non-keepalive control flow, Rust explicitly drops the internal session state. Successful normal runs keep the session for reuse. +- **PTY session**: `core` is always cleared after `start()` finishes, including failure paths. +- **No explicit JS finalizer-driven kill contract** is exposed by wrappers; cleanup is primarily tied to run completion/cancellation paths. Callers should use `timeoutMs`, `AbortSignal`, `shell.abort()`, or `pty.kill()` for deterministic teardown. diff --git a/docs/natives-text-search-pipeline.md b/docs/natives-text-search-pipeline.md new file mode 100644 index 000000000..1c537e96c --- /dev/null +++ b/docs/natives-text-search-pipeline.md @@ -0,0 +1,243 @@ +# Natives Text/Search Pipeline + +This document maps the `@oh-my-pi/pi-natives` text/search surface (`grep`, `glob`, `text`, `highlight`) from TypeScript wrappers to Rust N-API exports and back to JS result objects. + +Terminology follows `docs/natives-architecture.md`: +- **Wrapper**: TS API in `packages/natives/src/*` +- **Rust module layer**: N-API exports in `crates/pi-natives/src/*` +- **Shared scan cache**: `fs_cache`-backed directory-entry cache used by discovery/search flows + +## Implementation files + +- `packages/natives/src/grep/index.ts` +- `packages/natives/src/grep/types.ts` +- `packages/natives/src/glob/index.ts` +- `packages/natives/src/glob/types.ts` +- `packages/natives/src/text/index.ts` +- `packages/natives/src/text/types.ts` +- `packages/natives/src/highlight/index.ts` +- `packages/natives/src/highlight/types.ts` +- `crates/pi-natives/src/grep.rs` +- `crates/pi-natives/src/glob.rs` +- `crates/pi-natives/src/glob_util.rs` +- `crates/pi-natives/src/fs_cache.rs` +- `crates/pi-natives/src/text.rs` +- `crates/pi-natives/src/highlight.rs` +- `crates/pi-natives/src/fd.rs` + +## JS API ↔ Rust export mapping + +| JS wrapper API | Rust export (`#[napi(js_name = ...)]`) | Rust module | +| --- | --- | --- | +| `grep(options, onMatch?)` | `grep` | `grep.rs` | +| `searchContent(content, options)` | `search` | `grep.rs` | +| `hasMatch(content, pattern, options?)` | `hasMatch` | `grep.rs` | +| `fuzzyFind(options)` | `fuzzyFind` | `fd.rs` | +| `glob(options, onMatch?)` | `glob` | `glob.rs` | +| `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `fs_cache.rs` | +| `wrapTextWithAnsi(text, width)` | `wrapTextWithAnsi` | `text.rs` | +| `truncateToWidth(text, maxWidth, ellipsis, pad)` | `truncateToWidth` | `text.rs` | +| `sliceWithWidth(line, startCol, length, strict?)` | `sliceWithWidth` | `text.rs` | +| `extractSegments(line, beforeEnd, afterStart, afterLen, strictAfter)` | `extractSegments` | `text.rs` | +| `sanitizeText(text)` | `sanitizeText` | `text.rs` | +| `visibleWidth(text)` | `visibleWidth` | `text.rs` | +| `highlightCode(code, lang, colors)` | `highlightCode` | `highlight.rs` | +| `supportsLanguage(lang)` | `supportsLanguage` | `highlight.rs` | +| `getSupportedLanguages()` | `getSupportedLanguages` | `highlight.rs` | + +## Pipeline overview by subsystem + +## 1) Regex search (`grep`, `searchContent`, `hasMatch`) + +### Input/options flow + +1. TS wrapper forwards options to native: + - `grep/index.ts` passes `options` mostly unchanged and wraps callback from `(match) => void` to napi threadsafe callback shape `(err, match)`. + - `searchContent` and `hasMatch` pass string/`Uint8Array` directly. +2. Rust option structs in `grep.rs` deserialize camelCase fields (`ignoreCase`, `maxCount`, `contextBefore`, `contextAfter`, `maxColumns`, `timeoutMs`). +3. `grep` creates `CancelToken` from `timeoutMs` + `AbortSignal` and runs inside `task::blocking("grep", ...)`. + +### Execution branches + +- **In-memory branch (pure utility)** + - `search` → `search_sync` → `run_search` on provided content bytes. + - No filesystem scan, no `fs_cache`. +- **Single-file branch (filesystem-dependent)** + - `grep_sync` resolves path, checks metadata is file, streams up to `MAX_FILE_BYTES` per file (`4 MiB`) through ripgrep matcher. +- **Directory branch (filesystem-dependent)** + - Optional cache lookup via `fs_cache::get_or_scan` when `cache: true`. + - Fresh scan via `fs_cache::force_rescan` when `cache: false`. + - Optional empty-result recheck when cache age exceeds `empty_recheck_ms()`. + - Entry filtering: file-only + optional glob filter (`glob_util`) + optional type filter mapping (`js`, `ts`, `rust`, etc.). + +### Search/collection semantics + +- Regex engine: `grep_regex::RegexMatcherBuilder` with `ignoreCase` and `multiline`. +- Context resolution: + - `contextBefore/contextAfter` override legacy `context`. + - Non-content modes zero out context collection. +- Output modes: + - `content` => one `GrepMatch` per hit. + - `count` and `filesWithMatches` both map to count-style entries (`lineNumber=0`, `line=""`, `matchCount` set). +- Limits: + - Global `offset` and `maxCount` applied across files. + - Parallel path is used only when `maxCount` is unset and `offset == 0`; otherwise sequential path preserves deterministic global offset/limit semantics. + +### Result shaping back to JS + +- Rust `SearchResult`/`GrepResult` fields map to TS types via `#[napi(js_name = ...)]`. +- Counters are clamped to `u32` before crossing N-API. +- Optional booleans are omitted unless true in some paths (`limitReached`). +- Streaming callback receives each shaped `GrepMatch` (content or count entry). + +### Failure behavior + +- `searchContent` returns `SearchResult.error` for regex/search failures instead of throwing. +- `grep` rejects on hard errors (invalid path, invalid glob/regex, cancellation timeout/abort). +- `hasMatch` returns `Result` and throws on invalid pattern/UTF-8 decoding errors. +- File open/search errors in multi-file scans are skipped per-file; scan continues. + +### Malformed regex handling + +`grep.rs` sanitizes braces before regex compile: +- Invalid repetition-like braces are escaped (`{`/`}` -> `\{`/`\}`) when they cannot form `{N}`, `{N,}`, `{N,M}`. +- This prevents common literal-template fragments (for example `${platform}`) from failing as malformed repetition. +- Remaining invalid regex syntax still returns a regex error. + +## 2) File discovery (`glob`) and fuzzy path search (`fuzzyFind`) + +`glob` and `fuzzyFind` share `fs_cache` scans; matching logic differs. + +### `glob` flow + +1. TS wrapper (`glob/index.ts`): + - `path.resolve(options.path)`. + - Defaults: `pattern="*"`, `hidden=false`, `gitignore=true`, `recursive=true`. +2. Rust `glob` builds `GlobConfig` and compiles pattern via `glob_util::compile_glob`. +3. Entry source: + - `cache=true` => `get_or_scan` + optional stale-empty `force_rescan`. + - `cache=false` => `force_rescan(..., store=false)` (fresh only). +4. Filtering: + - Skip `.git` always. + - Skip `node_modules` unless requested (`includeNodeModules` or pattern mentioning node_modules). + - Apply glob match. + - Apply file-type filter; symlink `file/dir` filters resolve target metadata. +5. Optional sort by mtime desc (`sortByMtime`) before truncating to `maxResults`. + +### `fuzzyFind` flow (implemented in `fd.rs`) + +1. TS wrapper is exported from `grep` module, but Rust implementation lives in `fd.rs`. +2. Shared scan source from `fs_cache` with same cache/no-cache split and stale-empty recheck policy. +3. Scoring: + - exact / starts-with / contains / subsequence-based fuzzy score + - separator/punctuation-normalized scoring path + - directory bonus and deterministic tie-break (`score desc`, then `path asc`) +4. Symlink entries are excluded from fuzzy results. + +### Failure behavior + +- Invalid glob pattern => error from `glob_util::compile_glob`. +- Search root must be an existing directory (`resolve_search_path`), otherwise error. +- Cancellation/timeouts propagate as abort errors via `CancelToken::heartbeat()` checks in loops. + +### Malformed glob handling + +`glob_util::build_glob_pattern` is tolerant: +- Normalizes `\` to `/`. +- Auto-prefixes simple recursive patterns with `**/` when `recursive=true`. +- Auto-closes unbalanced `{...` alternation groups before compile. + +## 3) Shared scan/cache lifecycle (`fs_cache`) + +`fs_cache` stores scan results as normalized relative entries (`path`, `fileType`, optional `mtime`) keyed by: +- canonical search root +- `include_hidden` +- `use_gitignore` + +### Cache state transitions + +1. **Miss / disabled** + - TTL is `0` or key absent/expired -> fresh `collect_entries`. +2. **Hit** + - Entry age `< cache_ttl_ms()` -> return cached entries + `cache_age_ms`. +3. **Stale-empty recheck** (caller policy in `glob`/`grep`/`fd`) + - If query yields zero matches and `cache_age_ms >= empty_recheck_ms()`, force one rescan. +4. **Invalidation** + - `invalidateFsScanCache(path?)`: + - no arg: clear all keys + - path arg: remove keys whose root prefixes that target path + +### Stale-result tradeoff + +- Cache favors low-latency repeated scans over immediate consistency. +- TTL window can return stale positives/negatives. +- Empty-result recheck reduces stale negatives for older cached scans at the cost of one extra scan. +- Explicit invalidation is the intended correctness hook after file mutations. + +## 4) ANSI text utilities (`text`) + +These are pure, in-memory utilities (no filesystem scanning). + +### Boundaries and responsibilities + +- **`text.rs` owns terminal-cell semantics**: + - ANSI sequence parsing + - grapheme-aware width and slicing + - wrap/truncate/sanitize behavior +- **`grep.rs` line truncation (`maxColumns`) is separate**: + - simple character-boundary truncation of matched lines with `...` + - not ANSI-state-preserving and not terminal-cell width aware + +### Key behaviors + +- `wrapTextWithAnsi`: wraps by visible width, carries active SGR codes across wrapped lines. +- `truncateToWidth`: visible-cell truncation with ellipsis policy (`Unicode`, `Ascii`, `Omit`), optional right padding, and fast-path returning original JS string when unchanged. +- `sliceWithWidth`: column slicing with optional strict width enforcement. +- `extractSegments`: extracts before/after segments around an overlay while restoring ANSI state for the `after` segment. +- `sanitizeText`: strips ANSI escapes + control chars, drops lone surrogates, normalizes CR/LF by removing `\r`. +- `visibleWidth`: counts visible terminal cells (tabs use fixed `TAB_WIDTH` from Rust implementation). + +### Failure behavior + +Text functions generally return deterministic transformed output; errors are limited to JS string conversion boundaries (N-API argument conversion failures). + +## 5) Syntax highlighting (`highlight`) + +`highlight.rs` is pure transformation (no FS, no cache). + +### Flow + +1. Wrapper forwards `code`, optional `lang`, and ANSI color palette. +2. Rust resolves syntax by: + - token/name lookup + - extension lookup + - alias table fallback (`ts/tsx/js -> JavaScript`, etc.) + - fallback to plain text syntax when unresolved +3. Parse each line with syntect `ParseState` and scope stack. +4. Map scopes to 11 semantic color categories and inject/reset ANSI color codes. + +### Failure behavior + +- Per-line parse failure does not fail the call: that line is appended unhighlighted and processing continues. +- Unknown/unsupported language falls back to plain text syntax. + +## Pure utility vs filesystem-dependent flows + +| Flow | Filesystem access | Shared cache | Notes | +| --- | --- | --- | --- | +| `searchContent` / `hasMatch` | No | No | regex on provided bytes/string only | +| `text` module functions | No | No | ANSI/width/sanitization only | +| `highlight` module functions | No | No | syntax + ANSI coloring only | +| `glob` | Yes | Optional | directory scans + glob filtering | +| `fuzzyFind` | Yes | Optional | directory scans + fuzzy scoring | +| `grep` (file/dir path) | Yes | Optional (dir mode) | ripgrep over files, optional filters/callback | + +## End-to-end lifecycle summary + +1. Caller invokes TS wrapper with typed options. +2. Wrapper normalizes defaults (notably `glob`) and forwards to `native.*` export. +3. Rust validates/normalizes options and builds matcher/search config. +4. For filesystem flows, entries are scanned (cache hit/miss/rescan) then filtered/scored. +5. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution. +6. Rust shapes outputs into N-API objects (`lineNumber`, `matchCount`, `limitReached`, etc.). +7. TS wrapper returns typed JS objects (and optional per-match callbacks for `grep`/`glob`).