docs: added comprehensive documentation for @oh-my-pi/pi-natives architecture
- Added comprehensive documentation for @oh-my-pi/pi-natives architecture covering three-layer design (TypeScript wrapper, addon loading/validation, Rust N-API module). - Added deep-dive documentation for addon loading runtime including platform support, CPU variant selection, embedded addon extraction lifecycle, and validation contracts. - Added binding contract documentation defining TypeScript-side API surface, wrapper behavior, and N-API addon validation with maintainer checklist. - Added subsystem deep-dives for text/search pipeline (grep, glob, fs_cache, ANSI utilities), shell/PTY/process execution, task cancellation patterns, and media/system utilities.
This commit is contained in:
can1357
@@ -0,0 +1,268 @@
|
||||
# Natives Addon Loader Runtime
|
||||
|
||||
This document deep-dives the addon loading/validation layer in `@oh-my-pi/pi-natives`: how `native.ts` decides which `.node` file to load, when embedded payload extraction runs, and how startup failures are reported.
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/src/native.ts`
|
||||
- `packages/natives/src/embedded-addon.ts`
|
||||
- `packages/natives/src/bindings.ts`
|
||||
- `packages/natives/package.json`
|
||||
|
||||
## Scope and responsibility
|
||||
|
||||
Loader/runtime responsibilities are intentionally narrow:
|
||||
|
||||
- Build a platform/CPU-aware candidate list for addon filenames and directories.
|
||||
- Optionally materialize an embedded addon into a versioned per-user cache directory.
|
||||
- Attempt candidates in deterministic order.
|
||||
- Reject stale or incompatible addons via `validateNative` before exposing bindings.
|
||||
|
||||
Out of scope here: module-specific grep/text/highlight behavior.
|
||||
|
||||
## Runtime inputs and derived state
|
||||
|
||||
At module initialization (`export const native = loadNative();`), `native.ts` computes static context:
|
||||
|
||||
- **Platform tag**: ``${process.platform}-${process.arch}`` (for example `darwin-arm64`).
|
||||
- **Package version**: from `packages/natives/package.json` (`version` field).
|
||||
- **Core directories**:
|
||||
- `nativeDir`: package-local `packages/natives/native`.
|
||||
- `execDir`: directory containing `process.execPath`.
|
||||
- `versionedDir`: `<getNativesDir()>/<packageVersion>`.
|
||||
- `userDataDir` fallback:
|
||||
- Windows: `%LOCALAPPDATA%/omp` (or `%USERPROFILE%/AppData/Local/omp`).
|
||||
- Non-Windows: `~/.local/bin`.
|
||||
- **Compiled-binary mode** (`isCompiledBinary`): true if any of:
|
||||
- `PI_COMPILED` env var is set, or
|
||||
- `import.meta.url` contains Bun-embedded markers (`$bunfs`, `~BUN`, `%7EBUN`).
|
||||
- **Variant override**: `PI_NATIVE_VARIANT` (`modern`/`baseline` only; invalid values ignored).
|
||||
- **Selected variant**: explicit override, otherwise runtime AVX2 detection on x64 (`modern` if AVX2, else `baseline`).
|
||||
|
||||
## Platform support and tag resolution
|
||||
|
||||
`SUPPORTED_PLATFORMS` is fixed to:
|
||||
|
||||
- `linux-x64`
|
||||
- `linux-arm64`
|
||||
- `darwin-x64`
|
||||
- `darwin-arm64`
|
||||
- `win32-x64`
|
||||
|
||||
Behavior detail:
|
||||
|
||||
- Unsupported platforms are not rejected up-front.
|
||||
- Loader still tries all computed candidates first.
|
||||
- If nothing loads, it throws an explicit unsupported-platform error listing supported tags.
|
||||
|
||||
This preserves useful diagnostics for near-miss cases while still failing hard for truly unsupported targets.
|
||||
|
||||
## Variant selection (`modern` / `baseline` / default)
|
||||
|
||||
### x64 behavior
|
||||
|
||||
1. If `PI_NATIVE_VARIANT` is `modern` or `baseline`, that value wins.
|
||||
2. Else detect AVX2 support:
|
||||
- Linux: scan `/proc/cpuinfo` for `avx2`.
|
||||
- macOS: query `sysctl` (`machdep.cpu.leaf7_features`, fallback `machdep.cpu.features`).
|
||||
- Windows: run PowerShell `[System.Runtime.Intrinsics.X86.Avx2]::IsSupported`.
|
||||
3. Result:
|
||||
- AVX2 available -> `modern`
|
||||
- AVX2 unavailable/undetectable -> `baseline`
|
||||
|
||||
### Non-x64 behavior
|
||||
|
||||
- No variant is used; loader stays on the default filename (`pi_natives.<platform>-<arch>.node`).
|
||||
|
||||
### Filename construction
|
||||
|
||||
Given `tag = <platform>-<arch>`:
|
||||
|
||||
- Non-x64 or no variant: `pi_natives.<tag>.node`
|
||||
- x64 + `modern`: try in order
|
||||
1. `pi_natives.<tag>-modern.node`
|
||||
2. `pi_natives.<tag>-baseline.node` (intentional fallback)
|
||||
- x64 + `baseline`: only `pi_natives.<tag>-baseline.node`
|
||||
|
||||
The `addonLabel` used in final error messages is either `<tag>` or `<tag> (<variant>)`.
|
||||
|
||||
## Candidate path construction and fallback ordering
|
||||
|
||||
`native.ts` builds candidate pools before any `require(...)` call.
|
||||
|
||||
### Debug/dev candidates (only when `PI_DEV` is set)
|
||||
|
||||
Prepended first:
|
||||
|
||||
1. `<nativeDir>/pi_natives.dev.node`
|
||||
2. `<execDir>/pi_natives.dev.node`
|
||||
|
||||
This path is explicit debug intent and always outranks release candidates.
|
||||
|
||||
### Release candidates
|
||||
|
||||
Built from variant-resolved filename list and searched in this order:
|
||||
|
||||
- **Non-compiled runtime**:
|
||||
1. `<nativeDir>/<filename>`
|
||||
2. `<execDir>/<filename>`
|
||||
|
||||
- **Compiled runtime** (`PI_COMPILED` or Bun embedded markers):
|
||||
1. `<versionedDir>/<filename>`
|
||||
2. `<userDataDir>/<filename>`
|
||||
3. `<nativeDir>/<filename>`
|
||||
4. `<execDir>/<filename>`
|
||||
|
||||
`dedupedCandidates` removes duplicates while preserving first occurrence order.
|
||||
|
||||
### Final runtime sequence
|
||||
|
||||
At load time:
|
||||
|
||||
1. Optional embedded extraction candidate (if produced) is inserted at the front.
|
||||
2. Remaining deduplicated candidates are tried in order.
|
||||
3. First candidate that both `require(...)`s and passes `validateNative(...)` wins.
|
||||
|
||||
## Embedded addon extraction lifecycle
|
||||
|
||||
`embedded-addon.ts` defines a generated manifest shape:
|
||||
|
||||
- `platformTag`
|
||||
- `version`
|
||||
- `files[]` where each entry has `variant`, `filename`, `filePath`
|
||||
|
||||
Current checked-in default is `embeddedAddon: null`; compiled artifacts may replace this with real metadata.
|
||||
|
||||
### Extraction state machine
|
||||
|
||||
Extraction (`maybeExtractEmbeddedAddon`) runs only when all gates pass:
|
||||
|
||||
1. `isCompiledBinary === true`
|
||||
2. `embeddedAddon !== null`
|
||||
3. `embeddedAddon.platformTag === platformTag`
|
||||
4. `embeddedAddon.version === packageVersion`
|
||||
5. A variant-appropriate embedded file is found
|
||||
|
||||
Variant file selection mirrors runtime variant intent:
|
||||
|
||||
- Non-x64: prefer `default`, then first available file.
|
||||
- x64 + `modern`: prefer `modern`, fallback to `baseline`.
|
||||
- x64 + `baseline`: require `baseline`.
|
||||
|
||||
Materialization behavior:
|
||||
|
||||
1. Ensure `<versionedDir>` exists (`mkdirSync(..., { recursive: true })`).
|
||||
2. If `<versionedDir>/<selected filename>` already exists, reuse it (no rewrite).
|
||||
3. Else read embedded source `filePath` and write target file.
|
||||
4. Return target path for highest-priority load attempt.
|
||||
|
||||
On failure, extraction does not crash immediately; it appends an error entry (directory creation or write failure) and loader proceeds to normal candidate probing.
|
||||
|
||||
## Lifecycle and state transitions
|
||||
|
||||
```text
|
||||
Init
|
||||
-> Compute platform/version/variant/candidate lists
|
||||
-> (Compiled + embedded manifest matches?)
|
||||
yes -> Try extract embedded to versionedDir (record errors, continue)
|
||||
no -> Skip extraction
|
||||
-> For each runtime candidate in order:
|
||||
require(candidate)
|
||||
-> success: validateNative
|
||||
-> pass: return bindings (READY)
|
||||
-> fail: record error, continue
|
||||
-> failure: record error, continue
|
||||
-> none loaded:
|
||||
if unsupported platform tag -> throw Unsupported platform
|
||||
else -> throw Failed to load (full tried-path diagnostics + hints)
|
||||
```
|
||||
|
||||
## `validateNative` contract checks
|
||||
|
||||
`validateNative(bindings, source)` enforces a function-only contract over `NativeBindings` at startup.
|
||||
|
||||
Mechanics:
|
||||
|
||||
- For each required export name, it checks `typeof bindings[name] === "function"`.
|
||||
- Missing names are aggregated.
|
||||
- If any are missing, loader throws:
|
||||
- source addon path,
|
||||
- missing export list,
|
||||
- rebuild command hint.
|
||||
|
||||
This is a hard compatibility gate against stale binaries, partial builds, and symbol/name drift.
|
||||
|
||||
### JS API ↔ native export mapping (validation gate)
|
||||
|
||||
| JS binding name checked in `validateNative` | Expected native export name |
|
||||
| --- | --- |
|
||||
| `grep` | `grep` |
|
||||
| `glob` | `glob` |
|
||||
| `highlightCode` | `highlightCode` |
|
||||
| `executeShell` | `executeShell` |
|
||||
| `PtySession` | `PtySession` |
|
||||
| `Shell` | `Shell` |
|
||||
| `visibleWidth` | `visibleWidth` |
|
||||
| `getSystemInfo` | `getSystemInfo` |
|
||||
| `getWorkProfile` | `getWorkProfile` |
|
||||
| `invalidateFsScanCache` | `invalidateFsScanCache` |
|
||||
|
||||
Note: `bindings.ts` declares only the base `cancelWork(id)` member; module `types.ts` files declaration-merge additional symbols that `validateNative` enforces.
|
||||
|
||||
## Failure behavior and diagnostics
|
||||
|
||||
## Unsupported platform
|
||||
|
||||
If all candidates fail and `platformTag` is not in `SUPPORTED_PLATFORMS`, loader throws:
|
||||
|
||||
- `Unsupported platform: <tag>`
|
||||
- Full supported-platform list
|
||||
- Explicit issue-reporting guidance
|
||||
|
||||
## Stale binary / mismatch symptoms
|
||||
|
||||
Typical stale mismatch signal:
|
||||
|
||||
- `Native addon missing exports (<candidate>). Missing: ...`
|
||||
|
||||
Common causes:
|
||||
|
||||
- Old `.node` binary from previous package version/API shape.
|
||||
- Wrong variant artifact selected (for x64).
|
||||
- New Rust export not present in loaded artifact.
|
||||
|
||||
Loader behavior:
|
||||
|
||||
- Records per-candidate missing-export failures.
|
||||
- Continues probing remaining candidates.
|
||||
- If no candidate validates, final error includes every attempted path with each failure message.
|
||||
|
||||
## Compiled-binary startup failures
|
||||
|
||||
In compiled mode final diagnostics include:
|
||||
|
||||
- expected versioned cache target paths (`<versionedDir>/<filename>`),
|
||||
- remediation to delete stale `<versionedDir>` and rerun,
|
||||
- direct release download `curl` commands for each expected filename.
|
||||
|
||||
## Non-compiled startup failures
|
||||
|
||||
In normal package/runtime mode final diagnostics include:
|
||||
|
||||
- reinstall hint (`bun install @oh-my-pi/pi-natives`),
|
||||
- local rebuild command (`bun --cwd=packages/natives run build:native`),
|
||||
- optional x64 variant build hint (`TARGET_VARIANT=baseline|modern ...`).
|
||||
|
||||
## Dev/debug versus release behavior
|
||||
|
||||
When `PI_DEV` is set:
|
||||
|
||||
- `pi_natives.dev.node` candidates are prepended ahead of all release candidates.
|
||||
- Loader emits per-candidate console diagnostics (`Loaded native addon...` and load errors).
|
||||
|
||||
Without `PI_DEV`:
|
||||
|
||||
- Only release candidate chain is used.
|
||||
- No dev console diagnostics are emitted.
|
||||
|
||||
Operationally, this means debug sessions can validate an ad-hoc dev addon first, while production/release runs remain on deterministic release artifact probing.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Natives Architecture
|
||||
|
||||
`@oh-my-pi/pi-natives` is a three-layer stack:
|
||||
|
||||
1. **TypeScript wrapper/API layer** exposes stable JS/TS entrypoints.
|
||||
2. **Addon loading/validation layer** resolves and validates the `.node` binary for the current runtime.
|
||||
3. **Rust N-API module layer** implements performance-critical primitives exported to JS.
|
||||
|
||||
This document is the foundation for deeper module-level docs.
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/src/index.ts`
|
||||
- `packages/natives/src/native.ts`
|
||||
- `packages/natives/src/bindings.ts`
|
||||
- `packages/natives/src/embedded-addon.ts`
|
||||
- `packages/natives/scripts/build-native.ts`
|
||||
- `packages/natives/scripts/embed-native.ts`
|
||||
- `packages/natives/package.json`
|
||||
- `crates/pi-natives/src/lib.rs`
|
||||
|
||||
## Layer 1: TypeScript wrapper/API layer
|
||||
|
||||
`packages/natives/src/index.ts` is the public barrel. It groups exports by capability domain and re-exports typed wrappers rather than exposing raw N-API bindings directly.
|
||||
|
||||
Current top-level groups:
|
||||
|
||||
- **Search/text primitives**: `grep`, `glob`, `text`, `highlight`
|
||||
- **Execution/process/terminal primitives**: `shell`, `pty`, `ps`, `keys`
|
||||
- **System/media/conversion primitives**: `image`, `html`, `clipboard`, `system-info`, `work`
|
||||
|
||||
`packages/natives/src/bindings.ts` defines the base interface contract:
|
||||
|
||||
- `NativeBindings` starts with shared members (`cancelWork(id: number)`)
|
||||
- module-specific bindings are added by declaration merging from each module’s `types.ts`
|
||||
- `Cancellable` standardizes timeout and abort-signal options for wrappers that expose cancellation
|
||||
|
||||
**Guaranteed contract (API-facing):** consumers import from `@oh-my-pi/pi-natives` and use typed wrappers.
|
||||
|
||||
**Implementation detail (may change):** declaration merging and internal wrapper layout (`src/<module>/index.ts`, `src/<module>/types.ts`).
|
||||
|
||||
## Layer 2: Addon loading and validation
|
||||
|
||||
`packages/natives/src/native.ts` owns runtime addon selection, optional extraction, and export validation.
|
||||
|
||||
### Candidate resolution model
|
||||
|
||||
- Platform tag is `"${process.platform}-${process.arch}"`.
|
||||
- Supported tags are currently:
|
||||
- `linux-x64`
|
||||
- `linux-arm64`
|
||||
- `darwin-x64`
|
||||
- `darwin-arm64`
|
||||
- `win32-x64`
|
||||
- x64 can use CPU variants:
|
||||
- `modern` (AVX2-capable)
|
||||
- `baseline` (fallback)
|
||||
- Non-x64 uses the default filename (no variant suffix).
|
||||
|
||||
Filename strategy:
|
||||
|
||||
- Release: `pi_natives.<platform>-<arch>.node`
|
||||
- x64 variant release: `pi_natives.<platform>-<arch>-modern.node` and/or `...-baseline.node`
|
||||
- Dev: `pi_natives.dev.node` (preferred when `PI_DEV` is set)
|
||||
|
||||
### Platform-specific variant detection
|
||||
|
||||
For x64, variant selection uses:
|
||||
|
||||
- **Linux**: `/proc/cpuinfo`
|
||||
- **macOS**: `sysctl machdep.cpu.leaf7_features` / `machdep.cpu.features`
|
||||
- **Windows**: PowerShell check for `System.Runtime.Intrinsics.X86.Avx2`
|
||||
|
||||
`PI_NATIVE_VARIANT` can explicitly force `modern` or `baseline`.
|
||||
|
||||
### Binary distribution and extraction model
|
||||
|
||||
`packages/natives/package.json` includes both `src` and `native` in published files. The `native/` directory stores prebuilt platform artifacts.
|
||||
|
||||
For compiled binaries (`PI_COMPILED` or Bun embedded runtime markers), loader behavior is:
|
||||
|
||||
1. Check versioned user cache path: `<getNativesDir()>/<packageVersion>/...`
|
||||
2. Check legacy compiled-binary location:
|
||||
- Windows: `%LOCALAPPDATA%/omp` (fallback `%USERPROFILE%/AppData/Local/omp`)
|
||||
- non-Windows: `~/.local/bin`
|
||||
3. Fall back to packaged `native/` and executable directory candidates
|
||||
|
||||
If an embedded addon manifest is present (`embedded-addon.ts` generated by `scripts/embed-native.ts`), `native.ts` can materialize the matching embedded binary into the versioned cache directory before loading.
|
||||
|
||||
### Validation and failure modes
|
||||
|
||||
After `require(candidate)`, `validateNative(...)` verifies required exports (for example `grep`, `glob`, `highlightCode`, `PtySession`, `Shell`, `getSystemInfo`, `getWorkProfile`, `invalidateFsScanCache`).
|
||||
|
||||
Failure paths are explicit:
|
||||
|
||||
- **Unsupported platform tag**: throws with supported platform list
|
||||
- **No loadable candidate**: throws with all attempted paths and remediation hints
|
||||
- **Missing exports**: throws with exact missing names and rebuild command
|
||||
- **Embedded extraction errors**: records directory/write failures and includes them in final load diagnostics
|
||||
|
||||
**Guaranteed contract (API-facing):** addon load either succeeds with a validated binding set or fails fast with actionable error text.
|
||||
|
||||
**Implementation detail (may change):** exact candidate search order and compiled-binary fallback path ordering.
|
||||
|
||||
## Layer 3: Rust N-API module layer
|
||||
|
||||
`crates/pi-natives/src/lib.rs` is the Rust entry module that declares exported module ownership:
|
||||
|
||||
- `clipboard`
|
||||
- `fd`
|
||||
- `fs_cache`
|
||||
- `glob`
|
||||
- `glob_util`
|
||||
- `grep`
|
||||
- `highlight`
|
||||
- `html`
|
||||
- `image`
|
||||
- `keys`
|
||||
- `prof`
|
||||
- `ps`
|
||||
- `pty`
|
||||
- `shell`
|
||||
- `system_info`
|
||||
- `task`
|
||||
- `text`
|
||||
|
||||
These modules implement the N-API symbols consumed and validated by `native.ts`. JS-level names are surfaced through the TS wrappers in `packages/natives/src`.
|
||||
|
||||
**Guaranteed contract (API-facing):** Rust module exports must match the binding names expected by `validateNative` and wrapper modules.
|
||||
|
||||
**Implementation detail (may change):** internal Rust module decomposition and helper module boundaries (`glob_util`, `task`, etc.).
|
||||
|
||||
## Ownership boundaries
|
||||
|
||||
At architecture level, ownership is split as follows:
|
||||
|
||||
- **TS wrapper/API ownership (`packages/natives/src`)**
|
||||
- public API grouping, option typing, and stable JS ergonomics
|
||||
- cancellation surface (`timeoutMs`, `AbortSignal`) exposed to callers
|
||||
- **Loader ownership (`packages/natives/src/native.ts`)**
|
||||
- runtime binary selection
|
||||
- CPU variant selection and override handling
|
||||
- compiled-binary extraction and candidate probing
|
||||
- hard validation of required native exports
|
||||
- **Rust ownership (`crates/pi-natives/src`)**
|
||||
- algorithmic and system-level implementation
|
||||
- platform-native behavior and performance-sensitive logic
|
||||
- N-API symbol implementation that TS wrappers consume
|
||||
|
||||
## Runtime flow (high level)
|
||||
|
||||
1. Consumer imports from `@oh-my-pi/pi-natives`.
|
||||
2. Wrapper module calls into singleton `native` binding.
|
||||
3. `native.ts` selects candidate binary for platform/arch/variant.
|
||||
4. Optional embedded binary extraction occurs for compiled distributions.
|
||||
5. Addon is loaded and export set is validated.
|
||||
6. Wrapper returns typed results to caller.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Native addon**: A `.node` binary loaded via Node-API (N-API).
|
||||
- **Platform tag**: Runtime tuple `platform-arch` (for example `darwin-arm64`).
|
||||
- **Variant**: x64 CPU-specific build flavor (`modern` AVX2, `baseline` fallback).
|
||||
- **Wrapper**: TS function/class that provides typed API over raw native exports.
|
||||
- **Declaration merging**: TS technique used by module `types.ts` files to extend `NativeBindings`.
|
||||
- **Compiled binary mode**: Runtime mode where the CLI is bundled and native addons are resolved from extracted/cache paths instead of only package-local paths.
|
||||
- **Embedded addon**: Build artifact metadata and file references generated into `embedded-addon.ts` so compiled binaries can extract matching `.node` payloads.
|
||||
- **Validation gate**: `validateNative(...)` check that rejects stale/mismatched binaries missing required exports.
|
||||
@@ -0,0 +1,221 @@
|
||||
# Natives Binding Contract (TypeScript Side)
|
||||
|
||||
This document defines the TypeScript-side contract that sits between `@oh-my-pi/pi-natives` callers and the loaded N-API addon.
|
||||
|
||||
It focuses on three pieces:
|
||||
|
||||
1. contract shape (`NativeBindings` + module augmentation),
|
||||
2. wrapper behavior (`src/<module>/index.ts`),
|
||||
3. public export surface (`src/index.ts`).
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/src/bindings.ts`
|
||||
- `packages/natives/src/native.ts`
|
||||
- `packages/natives/src/index.ts`
|
||||
- `packages/natives/src/clipboard/types.ts`
|
||||
- `packages/natives/src/clipboard/index.ts`
|
||||
- `packages/natives/src/glob/types.ts`
|
||||
- `packages/natives/src/glob/index.ts`
|
||||
- `packages/natives/src/grep/types.ts`
|
||||
- `packages/natives/src/grep/index.ts`
|
||||
- `packages/natives/src/highlight/types.ts`
|
||||
- `packages/natives/src/highlight/index.ts`
|
||||
- `packages/natives/src/html/types.ts`
|
||||
- `packages/natives/src/html/index.ts`
|
||||
- `packages/natives/src/image/types.ts`
|
||||
- `packages/natives/src/image/index.ts`
|
||||
- `packages/natives/src/keys/types.ts`
|
||||
- `packages/natives/src/keys/index.ts`
|
||||
- `packages/natives/src/ps/types.ts`
|
||||
- `packages/natives/src/ps/index.ts`
|
||||
- `packages/natives/src/pty/types.ts`
|
||||
- `packages/natives/src/pty/index.ts`
|
||||
- `packages/natives/src/shell/types.ts`
|
||||
- `packages/natives/src/shell/index.ts`
|
||||
- `packages/natives/src/system-info/types.ts`
|
||||
- `packages/natives/src/system-info/index.ts`
|
||||
- `packages/natives/src/text/types.ts`
|
||||
- `packages/natives/src/text/index.ts`
|
||||
- `packages/natives/src/work/types.ts`
|
||||
- `packages/natives/src/work/index.ts`
|
||||
|
||||
## Contract model
|
||||
|
||||
`packages/natives/src/bindings.ts` defines the base contract:
|
||||
|
||||
- `NativeBindings` (base interface, currently includes `cancelWork(id: number): void`)
|
||||
- `Cancellable` (`timeoutMs?: number`, `signal?: AbortSignal`)
|
||||
- `TsFunc<T>` callback shape used by N-API threadsafe callbacks
|
||||
|
||||
Each module adds its own fields by declaration merging:
|
||||
|
||||
```ts
|
||||
// packages/natives/src/<module>/types.ts
|
||||
declare module "../bindings" {
|
||||
interface NativeBindings {
|
||||
grep(options: GrepOptions, onMatch?: TsFunc<GrepMatch>): Promise<GrepResult>;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This keeps one aggregate binding interface without a monolithic central type file.
|
||||
|
||||
## Declaration-merging lifecycle and state transitions
|
||||
|
||||
### 1) Compile-time type assembly
|
||||
|
||||
- `bindings.ts` provides the base `NativeBindings` symbol.
|
||||
- Every `src/<module>/types.ts` augments `NativeBindings`.
|
||||
- `src/native.ts` imports all `./<module>/types` files for side effects so the merged contract is in scope where `NativeBindings` is used.
|
||||
|
||||
State transition: **Base contract** → **Merged contract**.
|
||||
|
||||
### 2) Runtime addon load and validation gate
|
||||
|
||||
- `src/native.ts` loads candidate `.node` binaries.
|
||||
- Loaded object is treated as `NativeBindings` and immediately passed through `validateNative(...)`.
|
||||
- `validateNative` verifies required export keys by `typeof bindings[name] === "function"`.
|
||||
|
||||
State transition: **Untrusted addon object** → **Validated native binding object** (or hard failure).
|
||||
|
||||
### 3) Wrapper invocation
|
||||
|
||||
- Module wrappers in `src/<module>/index.ts` call `native.<export>`.
|
||||
- Wrappers adapt defaults and callback shape (`(err, value)` to value-only callback patterns in JS APIs).
|
||||
- `src/index.ts` re-exports module wrappers/types as the public package API.
|
||||
|
||||
State transition: **Validated raw bindings** → **Ergonomic public API**.
|
||||
|
||||
## Wrapper responsibilities
|
||||
|
||||
Wrappers are intentionally thin; they do not re-implement native logic.
|
||||
|
||||
Primary responsibilities:
|
||||
|
||||
- **Argument normalization/defaulting**
|
||||
- `glob()` resolves `options.path` to absolute path and defaults `hidden`, `gitignore`, `recursive`.
|
||||
- `hasMatch()` fills default flags (`ignoreCase`, `multiline`) before native call.
|
||||
- **Callback adaptation**
|
||||
- `grep()`, `glob()`, `executeShell()` convert `TsFunc<T>` (`error, value`) into user callback receiving only successful values.
|
||||
- **Environment or policy behavior around native calls**
|
||||
- Clipboard wrapper adds OSC52/Termux/headless handling and treats copy as best effort.
|
||||
- **Public naming and re-export curation**
|
||||
- `searchContent()` maps to native export `search`.
|
||||
|
||||
## Public export surface organization
|
||||
|
||||
`packages/natives/src/index.ts` is the canonical public barrel. It groups exports by capability domain:
|
||||
|
||||
- Search/text: `grep`, `glob`, `text`, `highlight`
|
||||
- Execution/process/terminal: `shell`, `pty`, `ps`, `keys`
|
||||
- System/media/conversion: `image`, `html`, `clipboard`, `system-info`, `work`
|
||||
|
||||
Maintainer rule: if a wrapper is not re-exported from `src/index.ts`, it is not part of the intended public package surface.
|
||||
|
||||
## JS API ↔ native export mapping (representative)
|
||||
|
||||
The Rust side uses N-API export names (typically via `#[napi(js_name = ...)]`) that must match these binding keys.
|
||||
|
||||
| Category | Public JS API (wrapper) | Native binding key | Return type | Async? |
|
||||
|---|---|---|---|---|
|
||||
| Grep | `grep(options, onMatch?)` | `grep` | `Promise<GrepResult>` | Yes |
|
||||
| Grep | `searchContent(content, options)` | `search` | `SearchResult` | No |
|
||||
| Grep | `hasMatch(content, pattern, opts?)` | `hasMatch` | `boolean` | No |
|
||||
| Grep | `fuzzyFind(options)` | `fuzzyFind` | `Promise<FuzzyFindResult>` | Yes |
|
||||
| Glob | `glob(options, onMatch?)` | `glob` | `Promise<GlobResult>` | Yes |
|
||||
| Glob | `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `void` | No |
|
||||
| Shell | `executeShell(options, onChunk?)` | `executeShell` | `Promise<ShellExecuteResult>` | Yes |
|
||||
| Shell | `Shell` | `Shell` | class constructor | N/A |
|
||||
| PTY | `PtySession` | `PtySession` | class constructor | N/A |
|
||||
| Text | `truncateToWidth(...)` | `truncateToWidth` | `string` | No |
|
||||
| Text | `sliceWithWidth(...)` | `sliceWithWidth` | `SliceWithWidthResult` | No |
|
||||
| Text | `visibleWidth(text)` | `visibleWidth` | `number` | No |
|
||||
| Highlight | `highlightCode(code, lang, colors)` | `highlightCode` | `string` | No |
|
||||
| HTML | `htmlToMarkdown(html, options?)` | `htmlToMarkdown` | `Promise<string>` | Yes |
|
||||
| System | `getSystemInfo()` | `getSystemInfo` | `SystemInfo` | No |
|
||||
| Work | `getWorkProfile(lastSeconds)` | `getWorkProfile` | `WorkProfile` | No |
|
||||
| Process | `killTree(pid, signal)` | `killTree` | `number` | No |
|
||||
| Process | `listDescendants(pid)` | `listDescendants` | `number[]` | No |
|
||||
| Clipboard | `copyToClipboard(text)` | `copyToClipboard` | `Promise<void>` (best effort wrapper behavior) | Yes |
|
||||
| Clipboard | `readImageFromClipboard()` | `readImageFromClipboard` | `Promise<ClipboardImage \| null>` | Yes |
|
||||
| Keys | `parseKey(data, kittyProtocolActive)` | `parseKey` | `string \| null` | No |
|
||||
|
||||
## Sync vs async contract differences
|
||||
|
||||
The contract mixes sync and async APIs; wrappers preserve native call style rather than forcing one model:
|
||||
|
||||
- **Promise-based async exports** for I/O or long-running work (`grep`, `glob`, `htmlToMarkdown`, `executeShell`, clipboard, image operations).
|
||||
- **Synchronous exports** for deterministic in-memory transforms/parsers (`search`, `hasMatch`, highlighting, text width/slicing, key parsing, process queries).
|
||||
- **Constructor exports** for stateful runtime objects (`Shell`, `PtySession`, `PhotonImage`).
|
||||
|
||||
Implication for maintainers: changing sync ↔ async for an existing export is a breaking API and contract change across wrappers and callers.
|
||||
|
||||
## Object and enum typing patterns
|
||||
|
||||
### Object patterns (`#[napi(object)]`-style JS objects)
|
||||
|
||||
TS models object-shaped native values as interfaces, for example:
|
||||
|
||||
- `GrepResult`, `SearchResult`, `GlobResult`
|
||||
- `SystemInfo`, `WorkProfile`
|
||||
- `ClipboardImage`, `ParsedKittyResult`
|
||||
|
||||
These are structural contracts at compile time; runtime shape correctness is owned by native implementation.
|
||||
|
||||
### Enum patterns
|
||||
|
||||
Numeric native enums are represented as `const enum` values in TS:
|
||||
|
||||
- `FileType` (`1=file`, `2=dir`, `3=symlink`)
|
||||
- `ImageFormat` (`0=PNG`, `1=JPEG`, `2=WEBP`, `3=GIF`)
|
||||
- `SamplingFilter`, `Ellipsis`, `KeyEventType`
|
||||
|
||||
Callers see named enum members; the binding boundary passes numbers.
|
||||
|
||||
## How mismatches are caught
|
||||
|
||||
Mismatch detection happens at two layers:
|
||||
|
||||
1. **Compile-time TypeScript contract checks**
|
||||
- Wrappers call `native.<name>` against merged `NativeBindings`.
|
||||
- Missing/renamed binding keys break TS type-checking in wrappers.
|
||||
|
||||
2. **Runtime validation in `validateNative`**
|
||||
- After load, `native.ts` checks required exports and throws if any are missing.
|
||||
- Error message includes missing keys and rebuild instruction.
|
||||
|
||||
This catches the common stale-binary drift: wrapper/type exists but loaded `.node` lacks the export.
|
||||
|
||||
## Failure behavior and caveats
|
||||
|
||||
### Load/validation failures (hard failures)
|
||||
|
||||
- Addon load failure or unsupported platform throws during module init in `native.ts`.
|
||||
- Missing required exports throws before wrappers are usable.
|
||||
|
||||
Effect: package fails fast rather than deferring failure to first call.
|
||||
|
||||
### Wrapper-level behavior differences
|
||||
|
||||
- Some wrappers intentionally soften failures (`copyToClipboard` is best effort and swallows native failure).
|
||||
- Streaming callbacks ignore callback error payloads and only forward successful value events.
|
||||
|
||||
### Type-level caveats (runtime stricter than TS)
|
||||
|
||||
- TS optional fields do not guarantee semantic validity; native layer can still reject malformed values.
|
||||
- `const enum` typing does not prevent out-of-range numeric values from untyped callers at runtime.
|
||||
- `validateNative` checks only presence/function-ness of required exports, not deep argument/return-shape compatibility.
|
||||
- `bindings.ts` includes `cancelWork(id)` in the base interface, but current runtime validation list does not enforce that key.
|
||||
|
||||
## Maintainer checklist for binding changes
|
||||
|
||||
When adding/changing an export, update all of:
|
||||
|
||||
1. `src/<module>/types.ts` (augmentation + contract types)
|
||||
2. `src/<module>/index.ts` (wrapper behavior)
|
||||
3. `src/native.ts` imports for the module types (if new module)
|
||||
4. `validateNative` required export checks
|
||||
5. `src/index.ts` public re-exports
|
||||
|
||||
Skipping any step creates either compile-time drift or runtime load-time failure.
|
||||
@@ -0,0 +1,237 @@
|
||||
# Natives Build, Release, and Debugging Runbook
|
||||
|
||||
This runbook describes how the `@oh-my-pi/pi-natives` build pipeline produces `.node` addons, how compiled distributions load them, and how to debug loader/build failures.
|
||||
|
||||
It follows the architecture terms from `docs/natives-architecture.md`:
|
||||
- **build-time artifact production** (`scripts/build-native.ts`)
|
||||
- **embedded addon manifest generation** (`scripts/embed-native.ts`)
|
||||
- **runtime addon loading + validation gate** (`src/native.ts`)
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/scripts/build-native.ts`
|
||||
- `packages/natives/scripts/embed-native.ts`
|
||||
- `packages/natives/package.json`
|
||||
- `packages/natives/src/native.ts`
|
||||
- `crates/pi-natives/Cargo.toml`
|
||||
|
||||
## Build pipeline overview
|
||||
|
||||
### 1) Build entrypoints
|
||||
|
||||
`packages/natives/package.json` scripts:
|
||||
|
||||
- `bun scripts/build-native.ts` (`build:native`) → release build
|
||||
- `bun scripts/build-native.ts --dev` (`dev:native`) → debug/dev build
|
||||
- `bun scripts/embed-native.ts` (`embed:native`) → generate `src/embedded-addon.ts` from built files
|
||||
|
||||
### 2) Rust artifact build
|
||||
|
||||
`build-native.ts` runs Cargo in `crates/pi-natives`:
|
||||
|
||||
- base command: `cargo build`
|
||||
- release mode adds `--release` unless `--dev` is passed
|
||||
- cross target adds `--target <CROSS_TARGET>`
|
||||
|
||||
`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`, so Cargo emits a shared library (`.so`/`.dylib`/`.dll`) that is then copied/renamed to a `.node` addon filename.
|
||||
|
||||
### 3) Artifact discovery and install
|
||||
|
||||
After Cargo completes, `build-native.ts` scans candidate output directories in order:
|
||||
|
||||
1. `${CARGO_TARGET_DIR}` (if set)
|
||||
2. `<repo>/target`
|
||||
3. `crates/pi-natives/target`
|
||||
|
||||
For each root it checks profile directories:
|
||||
- cross build: `<root>/<crossTarget>/<profile>` then `<root>/<profile>`
|
||||
- native build: `<root>/<profile>`
|
||||
|
||||
Then it looks for one of:
|
||||
- `libpi_natives.so`
|
||||
- `libpi_natives.dylib`
|
||||
- `pi_natives.dll`
|
||||
- `libpi_natives.dll`
|
||||
|
||||
When found, it atomically installs into `packages/natives/native/` with temp-file + rename semantics (Windows fallback handles locked DLL replacement failures explicitly).
|
||||
|
||||
## Target/variant model and naming conventions
|
||||
|
||||
## Platform tag
|
||||
|
||||
Both build and runtime use platform tag:
|
||||
|
||||
`<platform>-<arch>` (example: `darwin-arm64`, `linux-x64`)
|
||||
|
||||
## Variant model (x64 only)
|
||||
|
||||
x64 supports CPU variants:
|
||||
- `modern` (AVX2-capable path)
|
||||
- `baseline` (fallback)
|
||||
|
||||
Non-x64 uses a single default artifact (no variant suffix).
|
||||
|
||||
### Output filenames
|
||||
|
||||
Release builds:
|
||||
- x64: `pi_natives.<platform>-<arch>-modern.node` or `...-baseline.node`
|
||||
- non-x64: `pi_natives.<platform>-<arch>.node`
|
||||
|
||||
Dev build (`--dev`):
|
||||
- `pi_natives.dev.node`
|
||||
|
||||
Runtime loader candidate order in `native.ts`:
|
||||
- if `PI_DEV` is set: try `pi_natives.dev.node` first
|
||||
- then release candidates
|
||||
- compiled mode prepends extracted/cache candidates before package-local files
|
||||
|
||||
## Environment flags and build options
|
||||
|
||||
## Runtime flags
|
||||
|
||||
- `PI_DEV` (loader behavior): prefer dev addon candidates first
|
||||
- `PI_NATIVE_VARIANT` (loader behavior, x64 only): force `modern` or `baseline` selection at runtime
|
||||
- `PI_COMPILED` (loader behavior): enable compiled-binary candidate/extraction behavior
|
||||
|
||||
## Build-time flags/options
|
||||
|
||||
- `--dev` (script arg): build debug profile and emit `pi_natives.dev.node`
|
||||
- `CROSS_TARGET`: passed to Cargo `--target`
|
||||
- `TARGET_PLATFORM`: override output platform tag naming
|
||||
- `TARGET_ARCH`: override output arch naming
|
||||
- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy
|
||||
- `CARGO_TARGET_DIR`: additional root when searching Cargo outputs
|
||||
- `RUSTFLAGS`:
|
||||
- if unset and not cross-compiling, script sets:
|
||||
- modern: `-C target-cpu=x86-64-v3`
|
||||
- baseline: `-C target-cpu=x86-64-v2`
|
||||
- non-x64 / no variant: `-C target-cpu=native`
|
||||
- if already set, script does not override
|
||||
|
||||
## Build state/lifecycle transitions
|
||||
|
||||
### Build lifecycle (`build-native.ts`)
|
||||
|
||||
1. **Init**: parse args/env (`--dev`, target overrides, cross flags)
|
||||
2. **Variant resolve**:
|
||||
- non-x64 → no variant
|
||||
- x64 + `TARGET_VARIANT` → explicit variant
|
||||
- x64 cross-build without `TARGET_VARIANT` → hard error
|
||||
- x64 local build without override → detect host AVX2
|
||||
3. **Compile**: run Cargo with resolved profile/target
|
||||
4. **Locate artifact**: scan target roots/profile dirs/library names
|
||||
5. **Install**: copy + atomic rename into `packages/natives/native`
|
||||
6. **Complete**: output addon ready for loader candidates
|
||||
|
||||
Failure exits happen at any stage with explicit error text (invalid variant, failed cargo build, missing output library, install/rename failure).
|
||||
|
||||
### Embed lifecycle (`embed-native.ts`)
|
||||
|
||||
1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values
|
||||
2. **Candidate set**:
|
||||
- x64 expects both `modern` and `baseline`
|
||||
- non-x64 expects one default file
|
||||
3. **Validate availability** in `packages/natives/native`
|
||||
4. **Generate manifest** (`src/embedded-addon.ts`) with Bun `file` imports and package version
|
||||
5. **Runtime extraction ready** for compiled mode
|
||||
|
||||
`--reset` bypasses validation and writes a null manifest stub (`embeddedAddon = null`).
|
||||
|
||||
## Dev workflow vs shipped/compiled behavior
|
||||
|
||||
## Local development workflow
|
||||
|
||||
Typical local loop:
|
||||
|
||||
1. Build addon:
|
||||
- release: `bun --cwd=packages/natives run build:native`
|
||||
- debug: `bun --cwd=packages/natives run dev:native`
|
||||
2. Set `PI_DEV=1` when testing debug addon loading
|
||||
3. Loader in `native.ts` resolves package-local `native/` (and executable-dir fallback) candidates
|
||||
4. `validateNative` enforces export compatibility before wrappers use the binding
|
||||
|
||||
## Shipped/compiled binary workflow
|
||||
|
||||
In compiled mode (`PI_COMPILED` or Bun embedded markers):
|
||||
|
||||
1. Loader computes versioned cache dir: `<getNativesDir()>/<packageVersion>` (operationally `~/.omp/natives/<version>`)
|
||||
2. If embedded manifest matches current platform+version, loader may extract selected embedded file into that versioned dir
|
||||
3. Runtime candidate order includes:
|
||||
- versioned cache dir
|
||||
- legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere)
|
||||
- package/executable directories
|
||||
4. First successfully loaded addon still must pass `validateNative`
|
||||
|
||||
This is why packaging + runtime loader expectations must align: filenames, platform tags, and exported symbols must match what `native.ts` probes and validates.
|
||||
|
||||
## JS API ↔ Rust export mapping (validation gate subset)
|
||||
|
||||
`native.ts` requires these JS-visible exports to exist on the loaded addon. They map to Rust N-API exports in `crates/pi-natives/src`:
|
||||
|
||||
| JS name required by `validateNative` | Rust export declaration | Rust source file |
|
||||
| --- | --- | --- |
|
||||
| `glob` | `#[napi(js_name = "glob")] pub fn glob(...)` | `crates/pi-natives/src/glob.rs` |
|
||||
| `grep` | `#[napi(js_name = "grep")] pub fn grep(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `search` | `#[napi(js_name = "search")] pub fn search(...)` | `crates/pi-natives/src/grep.rs` |
|
||||
| `highlightCode` | `#[napi(js_name = "highlightCode")] pub fn highlight_code(...)` | `crates/pi-natives/src/highlight.rs` |
|
||||
| `getSystemInfo` | `#[napi(js_name = "getSystemInfo")] pub fn get_system_info(...)` | `crates/pi-natives/src/system_info.rs` |
|
||||
| `getWorkProfile` | `#[napi] pub fn get_work_profile(...)` (camel-cased export) | `crates/pi-natives/src/prof.rs` |
|
||||
| `invalidateFsScanCache` | `#[napi(js_name = "invalidateFsScanCache")] pub fn invalidate_fs_scan_cache(...)` | `crates/pi-natives/src/fs_cache.rs` |
|
||||
|
||||
If any required symbol is missing, loader fails fast with a rebuild hint.
|
||||
|
||||
## Failure behavior and diagnostics
|
||||
|
||||
## Build-time failures
|
||||
|
||||
- Invalid variant configuration:
|
||||
- `TARGET_VARIANT` set on non-x64 → immediate error
|
||||
- x64 cross-build without explicit `TARGET_VARIANT` → immediate error
|
||||
- Cargo build failure:
|
||||
- script surfaces non-zero exit and stderr
|
||||
- Artifact not found:
|
||||
- script prints every checked profile directory
|
||||
- Install failure:
|
||||
- explicit message; Windows includes locked-file hint
|
||||
|
||||
## Runtime loader failures (`native.ts`)
|
||||
|
||||
- Unsupported platform tag:
|
||||
- throws with supported platform list
|
||||
- No candidate could load:
|
||||
- throws with full candidate error list and mode-specific remediation hints
|
||||
- Missing exports:
|
||||
- throws with exact missing symbol names and rebuild command
|
||||
- Embedded extraction problems:
|
||||
- extraction mkdir/write errors recorded and included in final diagnostics
|
||||
|
||||
## Troubleshooting matrix
|
||||
|
||||
| Symptom | Likely cause | Verify | Fix |
|
||||
| --- | --- | --- | --- |
|
||||
| `Native addon missing exports ... Missing: <name>` | Stale `.node` binary, Rust export name mismatch, or wrong binary loaded | Run with `PI_DEV=1` to see loaded path; inspect export list for that file | Rebuild `build:native`; ensure Rust `#[napi(js_name=...)]` matches JS name; remove stale cached/versioned files |
|
||||
| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or only baseline file present | Check `PI_NATIVE_VARIANT`; inspect `native/` for `-modern` file | Build modern variant (`TARGET_VARIANT=modern ... build:native`) and ensure file is shipped |
|
||||
| Cross-build produces unusable/wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing `TARGET_VARIANT` for x64 | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` |
|
||||
| Compiled binary fails after upgrade | Stale extracted cache (`~/.omp/natives/<old-or-mismatched-version>`) or embedded manifest mismatch | Inspect versioned natives dir and loader error list | Delete versioned natives cache for the package version and rerun; regenerate embedded manifest during packaging |
|
||||
| Loader probes many paths and none work | Platform mismatch or missing release artifact in package `native/` | Check `platformTag` vs actual filename(s) | Ensure built filename exactly matches `pi_natives.<platform>-<arch>(-variant).node` convention and package includes `native/` |
|
||||
| `embed:native` fails with "Incomplete native addons" | Required variant files not built before embedding | Check expected vs found list in error text | Build required files first (x64: both modern+baseline; non-x64: default), then rerun `embed:native` |
|
||||
|
||||
## Operational commands
|
||||
|
||||
```bash
|
||||
# Release artifact for current host
|
||||
bun --cwd=packages/natives run build:native
|
||||
|
||||
# Debug artifact (load first when PI_DEV=1)
|
||||
bun --cwd=packages/natives run dev:native
|
||||
|
||||
# Build explicit x64 variants
|
||||
TARGET_VARIANT=modern bun --cwd=packages/natives run build:native
|
||||
TARGET_VARIANT=baseline bun --cwd=packages/natives run build:native
|
||||
|
||||
# Generate embedded addon manifest from built native files
|
||||
bun --cwd=packages/natives run embed:native
|
||||
|
||||
# Reset embedded manifest to null stub
|
||||
bun --cwd=packages/natives run embed:native -- --reset
|
||||
```
|
||||
@@ -0,0 +1,202 @@
|
||||
# Natives media + system utilities
|
||||
|
||||
This document is a subsystem deep-dive for the **system/media/conversion primitives** layer described in [`docs/natives-architecture.md`](./natives-architecture.md): `image`, `html`, `clipboard`, `system-info`, and `work` profiling.
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `crates/pi-natives/src/image.rs`
|
||||
- `crates/pi-natives/src/html.rs`
|
||||
- `crates/pi-natives/src/clipboard.rs`
|
||||
- `crates/pi-natives/src/system_info.rs`
|
||||
- `crates/pi-natives/src/prof.rs`
|
||||
- `crates/pi-natives/src/task.rs`
|
||||
- `packages/natives/src/image/index.ts`
|
||||
- `packages/natives/src/image/types.ts`
|
||||
- `packages/natives/src/html/index.ts`
|
||||
- `packages/natives/src/html/types.ts`
|
||||
- `packages/natives/src/clipboard/index.ts`
|
||||
- `packages/natives/src/clipboard/types.ts`
|
||||
- `packages/natives/src/system-info/index.ts`
|
||||
- `packages/natives/src/system-info/types.ts`
|
||||
- `packages/natives/src/work/index.ts`
|
||||
- `packages/natives/src/work/types.ts`
|
||||
|
||||
> Note: there is no `crates/pi-natives/src/work.rs`; work profiling is implemented in `prof.rs` and fed by instrumentation in `task.rs`.
|
||||
|
||||
## TS API ↔ Rust export/module mapping
|
||||
|
||||
| TS export (packages/natives) | Rust N-API export | Rust module |
|
||||
| --- | --- | --- |
|
||||
| `PhotonImage.parse(bytes)` | `PhotonImage::parse` (`js_name = "parse"`) | `image.rs` |
|
||||
| `PhotonImage#resize(width, height, filter)` | `PhotonImage::resize` (`js_name = "resize"`) | `image.rs` |
|
||||
| `PhotonImage#encode(format, quality)` | `PhotonImage::encode` (`js_name = "encode"`) | `image.rs` |
|
||||
| `htmlToMarkdown(html, options)` | `html_to_markdown` (`js_name = "htmlToMarkdown"`) | `html.rs` |
|
||||
| `copyToClipboard(text)` | `copy_to_clipboard` (`js_name = "copyToClipboard"`) + TS fallback logic | `clipboard.rs` + `clipboard/index.ts` |
|
||||
| `readImageFromClipboard()` | `read_image_from_clipboard` (`js_name = "readImageFromClipboard"`) | `clipboard.rs` |
|
||||
| `getSystemInfo()` | `get_system_info` (`js_name = "getSystemInfo"`) | `system_info.rs` |
|
||||
| `getWorkProfile(lastSeconds)` | `get_work_profile` | `prof.rs` |
|
||||
|
||||
## Data format boundaries and conversions
|
||||
|
||||
### Image (`image`)
|
||||
|
||||
- **JS input boundary**: `Uint8Array` encoded image bytes.
|
||||
- **Rust decode boundary**: bytes are copied to `Vec<u8>`, format is guessed with `ImageReader::with_guessed_format()`, then decoded to `DynamicImage`.
|
||||
- **In-memory state**: `PhotonImage` stores `Arc<DynamicImage>`.
|
||||
- **Output boundary**: `encode(format, quality)` returns `Promise<Uint8Array>` (Rust `Vec<u8>`).
|
||||
|
||||
Format IDs are numeric:
|
||||
|
||||
- `0`: PNG
|
||||
- `1`: JPEG
|
||||
- `2`: WebP (lossless encoder)
|
||||
- `3`: GIF
|
||||
|
||||
Constraints:
|
||||
|
||||
- `quality` is only used for JPEG.
|
||||
- PNG/WebP/GIF ignore `quality`.
|
||||
- Unsupported format IDs fail (`Invalid image format: <id>`).
|
||||
|
||||
### HTML conversion (`html`)
|
||||
|
||||
- **JS input boundary**: HTML `string` + optional object `{ cleanContent?: boolean; skipImages?: boolean }`.
|
||||
- **Rust conversion boundary**: `String` input is converted by `html_to_markdown_rs::convert`.
|
||||
- **Output boundary**: Markdown `string`.
|
||||
|
||||
Conversion behavior:
|
||||
|
||||
- `cleanContent` defaults to `false`.
|
||||
- When `cleanContent=true`, preprocessing is enabled with `PreprocessingPreset::Aggressive` and hard-removal flags for navigation/forms.
|
||||
- `skipImages` defaults to `false`.
|
||||
|
||||
### Clipboard (`clipboard`)
|
||||
|
||||
- **Text path**:
|
||||
- TS first emits OSC 52 (`\x1b]52;c;<base64>\x07`) when stdout is a TTY.
|
||||
- Same text is then attempted via native clipboard API (`native.copyToClipboard`) as best-effort.
|
||||
- On Termux, TS attempts `termux-clipboard-set` first.
|
||||
- **Image read path**:
|
||||
- Rust reads raw image from `arboard`.
|
||||
- Rust re-encodes it to PNG bytes (`image` crate), returns `{ data: Uint8Array, mimeType: "image/png" }`.
|
||||
- TS returns `null` early on Termux or Linux sessions without display server (`DISPLAY`/`WAYLAND_DISPLAY` missing).
|
||||
|
||||
### System info (`system-info`)
|
||||
|
||||
- **Output boundary**: plain object returned synchronously.
|
||||
- Rust currently populates: `distro`, `kernel`, `cpu`, `disk`.
|
||||
- Linux distro comes from `/etc/os-release` parsing; macOS may append a marketing name (`Tahoe`, `Sequoia`, etc.) to OS version text.
|
||||
- Disk summary is normalized to human-readable strings (`used/total (pct%)`), with platform-dependent selection:
|
||||
- Windows: aggregates each mount entry.
|
||||
- non-Windows: prefers `/`, falls back to first disk.
|
||||
|
||||
### Work profiling (`work`)
|
||||
|
||||
- **Collection boundary**: profiling samples are produced by `profile_region(tag)` guards in `task::blocking` and `task::future`.
|
||||
- **Storage format**: fixed-size circular buffer (`MAX_SAMPLES = 10_000`) storing stack path + duration (`μs`) + timestamp (`μs since process start`).
|
||||
- **Output boundary**: `getWorkProfile(lastSeconds)` returns object:
|
||||
- `folded`: folded-stack text (flamegraph input)
|
||||
- `summary`: markdown table summary
|
||||
- `svg`: optional flamegraph SVG
|
||||
- `totalMs`, `sampleCount`
|
||||
|
||||
## Lifecycle and state transitions
|
||||
|
||||
### Image lifecycle
|
||||
|
||||
1. `PhotonImage.parse(bytes)` schedules a blocking decode task (`image.decode`).
|
||||
2. On success, a native `PhotonImage` handle exists in JS.
|
||||
3. `resize(...)` creates a new native handle (`image.resize`), old and new handles can coexist.
|
||||
4. `encode(...)` materializes bytes (`image.encode`) without mutating image dimensions.
|
||||
|
||||
Failure transitions:
|
||||
|
||||
- Format detection/decode failure rejects parse promise.
|
||||
- Encode failure rejects encode promise.
|
||||
- Invalid format ID rejects encode promise.
|
||||
|
||||
### HTML lifecycle
|
||||
|
||||
1. `htmlToMarkdown(html, options)` schedules a blocking conversion task.
|
||||
2. Conversion runs with defaulted options (`cleanContent=false`, `skipImages=false`) unless specified.
|
||||
3. Returns markdown string or rejects.
|
||||
|
||||
Failure transitions:
|
||||
|
||||
- Converter failure returns rejected promise (`Conversion error: ...`).
|
||||
|
||||
### Clipboard lifecycle
|
||||
|
||||
`copyToClipboard(text)` is intentionally best-effort and multi-path:
|
||||
|
||||
1. If TTY: attempt OSC 52 write (base64 payload).
|
||||
2. Try Termux command when `TERMUX_VERSION` is set.
|
||||
3. Try native `arboard` text copy.
|
||||
4. Swallow errors at TS layer.
|
||||
|
||||
`readImageFromClipboard()` strictness differs by stage:
|
||||
|
||||
1. TS hard-gates unsupported runtime contexts (Termux/headless Linux) to `null`.
|
||||
2. Rust `arboard` read runs only when TS allows it.
|
||||
3. `ContentNotAvailable` maps to `null`.
|
||||
4. Other Rust errors reject.
|
||||
|
||||
### System info lifecycle
|
||||
|
||||
1. `getSystemInfo()` refreshes `sysinfo::System` and disk list synchronously.
|
||||
2. Per-platform helpers derive distro/kernel/cpu/disk snapshots.
|
||||
3. Object is returned directly; no async task scheduling.
|
||||
|
||||
Failure transitions:
|
||||
|
||||
- Missing optional data degrades to omitted fields (`Option::None`), not thrown errors.
|
||||
- `/etc/os-release` parse failures are soft-fail (`None` distro).
|
||||
- Disk total space `0` is treated as unavailable (`None` disk).
|
||||
|
||||
### Work profiling lifecycle
|
||||
|
||||
1. No explicit start: profiling is always on when task helpers execute.
|
||||
2. Every instrumented task scope records one sample on guard drop.
|
||||
3. Samples overwrite oldest entries after buffer capacity is reached.
|
||||
4. `getWorkProfile(lastSeconds)` reads a time window and derives folded/summary/svg artifacts.
|
||||
|
||||
Failure transitions:
|
||||
|
||||
- SVG generation failure is soft-fail (`svg: null`), while folded and summary still return.
|
||||
- Empty sample window returns empty folded data and `svg: null`, not an error.
|
||||
|
||||
## Unsupported operations and error propagation
|
||||
|
||||
### Image
|
||||
|
||||
- Unsupported decode input or corrupted bytes: strict failure (promise rejection).
|
||||
- Unsupported encode format ID: strict failure.
|
||||
- No best-effort fallback path in TS wrapper.
|
||||
|
||||
### HTML
|
||||
|
||||
- Conversion errors are strict failures (rejection).
|
||||
- Option omission is best-effort defaulting, not failure.
|
||||
|
||||
### Clipboard
|
||||
|
||||
- Text copy is best-effort at TS layer: operational failures are suppressed.
|
||||
- Image read distinguishes "no image" (`null`) from operational failure (rejection).
|
||||
- Termux/headless Linux are treated as unsupported contexts for image read (`null`).
|
||||
|
||||
### System info
|
||||
|
||||
- Designed for partial success: fields are optional and may be absent by platform.
|
||||
- Current TS type is broader than current Rust-populated fields; maintainers should expect sparse payloads unless Rust expands output.
|
||||
|
||||
### Work profiling
|
||||
|
||||
- Retrieval is strict for function call itself, but artifact generation is partially best-effort (`svg` nullable).
|
||||
- Buffer truncation is expected behavior (ring buffer), not data loss bug.
|
||||
|
||||
## Platform caveats
|
||||
|
||||
- **Clipboard text**: OSC 52 depends on terminal support; native clipboard access depends on desktop environment/session.
|
||||
- **Clipboard image read**: blocked in TS for Termux and Linux without display server.
|
||||
- **System info distro**: Linux distro name quality depends on `/etc/os-release` fields; macOS marketing name mapping is version-table-based and may lag new releases.
|
||||
- **Disk reporting**: Windows returns a comma-separated multi-volume summary; non-Windows returns one primary mount summary.
|
||||
@@ -0,0 +1,208 @@
|
||||
# Native Rust task execution and cancellation (`pi-natives`)
|
||||
|
||||
This document describes how `crates/pi-natives` schedules native work and how cancellation flows from JS options (`timeoutMs`, `AbortSignal`) to Rust execution.
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `crates/pi-natives/src/task.rs`
|
||||
- `crates/pi-natives/src/grep.rs`
|
||||
- `crates/pi-natives/src/glob.rs`
|
||||
- `crates/pi-natives/src/fd.rs`
|
||||
- `crates/pi-natives/src/shell.rs`
|
||||
- `crates/pi-natives/src/pty.rs`
|
||||
- `crates/pi-natives/src/html.rs`
|
||||
- `crates/pi-natives/src/image.rs`
|
||||
- `crates/pi-natives/src/clipboard.rs`
|
||||
- `crates/pi-natives/src/text.rs`
|
||||
- `crates/pi-natives/src/ps.rs`
|
||||
|
||||
## Core primitives (`task.rs`)
|
||||
|
||||
`task.rs` defines three core pieces:
|
||||
|
||||
1. `task::blocking(tag, cancel_token, work)`
|
||||
- Wraps `napi::AsyncTask` / `Task`.
|
||||
- `compute()` runs on libuv worker threads (for CPU-bound or blocking/sync system calls).
|
||||
- Returns a JS `Promise<T>`.
|
||||
|
||||
2. `task::future(env, tag, work)`
|
||||
- Wraps `env.spawn_future(...)`.
|
||||
- Runs async work on Tokio runtime.
|
||||
- Returns `PromiseRaw<'env, T>`.
|
||||
|
||||
3. `CancelToken` / `AbortToken` / `AbortReason`
|
||||
- `CancelToken::new(timeout_ms, signal)` combines deadline + optional `AbortSignal`.
|
||||
- `CancelToken::heartbeat()` is cooperative cancellation for blocking loops.
|
||||
- `CancelToken::wait()` is async cancellation wait (`Signal` / `Timeout` / `User` Ctrl-C).
|
||||
- `AbortToken` lets external code request abort (`abort(reason)`).
|
||||
|
||||
## `blocking` vs `future`: execution model and selection
|
||||
|
||||
### Use `task::blocking`
|
||||
|
||||
Use when work is CPU-heavy or fundamentally synchronous/blocking:
|
||||
|
||||
- regex/file scanning (`grep`, `glob`, `fuzzy_find`)
|
||||
- synchronous PTY loop internals (`run_pty_sync` via `spawn_blocking`)
|
||||
- clipboard/image/html conversions
|
||||
|
||||
Behavior:
|
||||
|
||||
- Work closure receives a cloned `CancelToken`.
|
||||
- Cancellation is only observed where code checks `ct.heartbeat()?`.
|
||||
- Closure `Err(...)` rejects JS promise.
|
||||
|
||||
### Use `task::future`
|
||||
|
||||
Use when work must `await` async operations:
|
||||
|
||||
- shell session orchestration (`shell.run`, `executeShell`)
|
||||
- task racing (`tokio::select!`) between completion and cancellation
|
||||
|
||||
Behavior:
|
||||
|
||||
- Future can race normal completion against `ct.wait()`.
|
||||
- On cancel path, async implementations typically propagate cancellation to inner subsystems (e.g., `tokio_util::CancellationToken`) and optionally force abort on grace timeout.
|
||||
|
||||
## JS API ↔ Rust export mapping (task/cancel relevant)
|
||||
|
||||
| JS-facing API | Rust export (`#[napi]`) | Scheduler | Cancellation hookup |
|
||||
|---|---|---|---|
|
||||
| `grep(options, onMatch?)` | `grep` | `task::blocking("grep", ct, ...)` | `CancelToken::new(options.timeoutMs, options.signal)` + `ct.heartbeat()` |
|
||||
| `glob(options, onMatch?)` | `glob` | `task::blocking("glob", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in filter loop |
|
||||
| `fuzzyFind(options)` | `fuzzy_find` (`js_name = "fuzzyFind"`) | `task::blocking("fuzzy_find", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in scoring loop |
|
||||
| `shell.run(options, onChunk?)` | `Shell::run` | `task::future(env, "shell.run", ...)` | `ct.wait()` raced against run task; bridges to Tokio `CancellationToken` |
|
||||
| `executeShell(options, onChunk?)` | `execute_shell` (`js_name = "executeShell"`) | `task::future(env, "shell.execute", ...)` | same as above |
|
||||
| `pty.start(options, onChunk?)` | `PtySession::start` | `task::future(env, "pty.start", ...)` + inner `spawn_blocking` | `CancelToken` checked in sync PTY loop via `heartbeat()` |
|
||||
| `htmlToMarkdown(html, options?)` | `html_to_markdown` (`js_name = "htmlToMarkdown"`) | `task::blocking("html_to_markdown", (), ...)` | none (`()` token) |
|
||||
| `PhotonImage.parse/encode/resize` | `PhotonImage::{parse,encode,resize}` | `task::blocking(...)` | none (`()` token) |
|
||||
| `copyToClipboard/readImageFromClipboard` | `copy_to_clipboard` / `read_image_from_clipboard` | `task::blocking(...)` | none (`()` token) |
|
||||
|
||||
`text.rs` and `ps.rs` currently do not use `task::blocking`/`task::future` and therefore do not participate in this cancellation path.
|
||||
|
||||
## Cancellation lifecycle and state transitions
|
||||
|
||||
### `CancelToken` lifecycle
|
||||
|
||||
`CancelToken` is cooperative and stateful:
|
||||
|
||||
```text
|
||||
Created
|
||||
├─ no signal + no timeout -> passive token (never aborts unless externally emplaced)
|
||||
├─ signal registered -> waits for AbortSignal callback
|
||||
└─ deadline set -> timeout check becomes active
|
||||
|
||||
Running
|
||||
├─ heartbeat()/wait() sees signal -> AbortReason::Signal
|
||||
├─ heartbeat()/wait() sees deadline -> AbortReason::Timeout
|
||||
├─ wait() sees Ctrl-C -> AbortReason::User
|
||||
└─ no abort -> continue
|
||||
|
||||
Aborted (terminal)
|
||||
└─ first abort reason wins (atomic flag + notifier)
|
||||
```
|
||||
|
||||
### Before-start vs mid-execution cancellation
|
||||
|
||||
- **Before start / before first cancellation check**:
|
||||
- `task::future` users that race on `ct.wait()` can resolve cancel immediately once they enter `select!`.
|
||||
- `task::blocking` users only observe cancellation when closure code reaches `heartbeat()`. If closure does not heartbeat early, cancellation is delayed.
|
||||
|
||||
- **Mid-execution**:
|
||||
- `blocking`: next `heartbeat()` returns `Err("Aborted: ...")`.
|
||||
- `future`: `ct.wait()` branch wins `select!`, then code cancels subordinate async machinery (for shell: cancels Tokio token, waits up to 2s, then aborts task).
|
||||
|
||||
## Heartbeat expectations for long-running loops
|
||||
|
||||
`heartbeat()` must run at predictable cadence in loops with unbounded or large work sets.
|
||||
|
||||
Observed patterns:
|
||||
|
||||
- `glob::filter_entries`: check each entry before filtering/matching.
|
||||
- `fd::score_entries`: check each scanned candidate.
|
||||
- `grep_sync`: explicit cancellation check before heavy search phase, plus fs-cache calls that also receive token.
|
||||
- `run_pty_sync`: check every loop tick (~16ms sleep cadence) and kill child on cancellation.
|
||||
|
||||
Practical rule: no loop over external-size input should exceed a short bounded interval without a heartbeat.
|
||||
|
||||
## Failure behavior and error propagation to JS
|
||||
|
||||
### Blocking tasks
|
||||
|
||||
Error path:
|
||||
|
||||
1. Closure returns `Err(napi::Error)` (including `heartbeat()` abort).
|
||||
2. `Task::compute()` returns `Err`.
|
||||
3. `AsyncTask` rejects JS promise.
|
||||
|
||||
Typical error strings:
|
||||
|
||||
- `Aborted: Timeout`
|
||||
- `Aborted: Signal`
|
||||
- domain errors (`Failed to decode image: ...`, `Conversion error: ...`, etc.)
|
||||
|
||||
### Future tasks
|
||||
|
||||
Error path:
|
||||
|
||||
1. Async body returns `Err(napi::Error)` or join failure is mapped (`... task failed: {err}`).
|
||||
2. `task::future`-spawned promise rejects.
|
||||
3. Some APIs intentionally return structured cancellation results instead of rejection (`ShellRunResult`/`ShellExecuteResult` with `cancelled`/`timed_out` flags and `exit_code: None`).
|
||||
|
||||
### Cancellation reporting split
|
||||
|
||||
- **Abort as error**: most blocking exports using `heartbeat()?`.
|
||||
- **Abort as typed result**: shell/pty style command APIs that model cancellation in result structs.
|
||||
|
||||
Choose one model per API and document it explicitly.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
1. **Missing heartbeat in blocking loops**
|
||||
- Symptom: timeout/signal appears ignored until loop ends.
|
||||
- Fix: add `ct.heartbeat()?` at loop top and before expensive per-item steps.
|
||||
|
||||
2. **Long uncancelable sections**
|
||||
- Symptom: cancellation latency spikes during single large call (decode, sort, compression, etc.).
|
||||
- Fix: split work into chunks with heartbeat boundaries; if impossible, document latency.
|
||||
|
||||
3. **Blocking async executor**
|
||||
- Symptom: async API stalls when sync-heavy code runs directly in future.
|
||||
- Fix: move CPU/sync blocks to `task::blocking` or `tokio::task::spawn_blocking`.
|
||||
|
||||
4. **Inconsistent cancel semantics**
|
||||
- Symptom: one API rejects on cancel, another resolves with flags, confusing callers.
|
||||
- Fix: standardize per domain and keep wrapper docs aligned.
|
||||
|
||||
5. **Forgetting cancellation bridge in nested async tasks**
|
||||
- Symptom: outer token is cancelled but inner readers/subprocess tasks keep running.
|
||||
- Fix: bridge cancellation to inner token/signal and enforce grace timeout + forced abort fallback.
|
||||
|
||||
## Checklist for new cancellable exports
|
||||
|
||||
1. Classify work correctly:
|
||||
- CPU-bound or sync blocking -> `task::blocking`
|
||||
- async I/O / `await` orchestration -> `task::future`
|
||||
|
||||
2. Expose cancel inputs when needed:
|
||||
- include `timeoutMs` and `signal` in `#[napi(object)]` options
|
||||
- create `let ct = task::CancelToken::new(timeout_ms, signal);`
|
||||
|
||||
3. Wire cancellation through all layers:
|
||||
- blocking loops: `ct.heartbeat()?` at stable intervals
|
||||
- async orchestration: race with `ct.wait()` and cancel sub-tasks/tokens
|
||||
|
||||
4. Decide cancellation contract:
|
||||
- reject promise with abort error, or
|
||||
- resolve typed `{ cancelled, timedOut, ... }`
|
||||
- keep this contract consistent for the API family
|
||||
|
||||
5. Propagate failures with context:
|
||||
- map errors via `Error::from_reason(format!("...: {err}"))`
|
||||
- include stage-specific prefixes (`spawn`, `decode`, `wait`, etc.)
|
||||
|
||||
6. Handle before-start and mid-flight cancellation:
|
||||
- cancellation check/await must happen before expensive body and during long execution
|
||||
|
||||
7. Validate no executor misuse:
|
||||
- no long sync work directly inside async futures without `spawn_blocking`/blocking task wrapper
|
||||
@@ -0,0 +1,268 @@
|
||||
# Natives Shell, PTY, Process, and Key Internals
|
||||
|
||||
This document covers the **execution/process/terminal primitives** in `@oh-my-pi/pi-natives`: `shell`, `pty`, `ps`, and `keys`, using the architecture terms from `docs/natives-architecture.md`.
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `crates/pi-natives/src/shell.rs`
|
||||
- `crates/pi-natives/src/shell/windows.rs` (Windows only)
|
||||
- `crates/pi-natives/src/pty.rs`
|
||||
- `crates/pi-natives/src/ps.rs`
|
||||
- `crates/pi-natives/src/keys.rs`
|
||||
- `crates/pi-natives/src/task.rs` (shared cancellation behavior used by shell/pty)
|
||||
- `packages/natives/src/shell/index.ts`
|
||||
- `packages/natives/src/shell/types.ts`
|
||||
- `packages/natives/src/pty/index.ts`
|
||||
- `packages/natives/src/pty/types.ts`
|
||||
- `packages/natives/src/ps/index.ts`
|
||||
- `packages/natives/src/ps/types.ts`
|
||||
- `packages/natives/src/keys/index.ts`
|
||||
- `packages/natives/src/keys/types.ts`
|
||||
- `packages/natives/src/bindings.ts`
|
||||
|
||||
## Layer ownership
|
||||
|
||||
- **TS wrapper/API layer** (`packages/natives/src/*`): typed entrypoints, cancellation surface (`timeoutMs`, `AbortSignal`), and JS ergonomics.
|
||||
- **Rust N-API module layer** (`crates/pi-natives/src/*`): shell/PTY process execution, process-tree traversal/termination, and key-sequence parsing.
|
||||
- **Validation gate** (`native.ts`, architecture-level): ensures required exports (`Shell`, `executeShell`, `PtySession`, `killTree`, `listDescendants`, key helpers) exist before wrappers are used.
|
||||
|
||||
## Shell subsystem (`shell`)
|
||||
|
||||
### API model
|
||||
|
||||
Two execution modes are exposed:
|
||||
|
||||
1. **One-shot** via `executeShell(options, onChunk?)`.
|
||||
2. **Persistent session** via `new Shell(options?)` then `shell.run(...)` repeatedly.
|
||||
|
||||
Both stream output through a threadsafe callback and return `{ exitCode?, cancelled, timedOut }`.
|
||||
|
||||
### Session creation and environment model
|
||||
|
||||
Rust creates `brush_core::Shell` with:
|
||||
|
||||
- non-interactive mode,
|
||||
- `do_not_inherit_env: true`,
|
||||
- explicit environment reconstruction from host env,
|
||||
- skip-list for shell-sensitive vars (`PS1`, `PWD`, `SHLVL`, bash function exports, etc.).
|
||||
|
||||
Session env behavior:
|
||||
|
||||
- `ShellOptions.sessionEnv` is applied once at session creation.
|
||||
- `ShellRunOptions.env` is command-scoped (`EnvironmentScope::Command`) and popped after each run.
|
||||
- `PATH` is merged specially on Windows with case-insensitive dedupe.
|
||||
|
||||
Windows-only path enrichment (`shell/windows.rs`): discovered Git-for-Windows paths (`cmd`, `bin`, `usr/bin`) are appended if present and not already included.
|
||||
|
||||
### Runtime lifecycle and state transitions
|
||||
|
||||
Persistent shell (`Shell.run`) uses this state machine:
|
||||
|
||||
- **Idle/Uninitialized**: `session: None`.
|
||||
- **Running**: first `run()` lazily creates session, stores `current_abort` token, executes command.
|
||||
- **Completed + keepalive**: if execution control flow is `Normal`, `current_abort` is cleared and session is reused.
|
||||
- **Completed + teardown**: if control flow is loop/script/shell-exit related (`BreakLoop`, `ContinueLoop`, `ReturnFromFunctionOrScript`, `ExitShell`), session is dropped (`session: None`).
|
||||
- **Cancelled/Timed out**: run task is cancelled, grace wait (2s), then force-abort; session is dropped.
|
||||
- **Error**: session is dropped.
|
||||
|
||||
One-shot shell (`executeShell`) always creates and drops a fresh session per call.
|
||||
|
||||
### Streaming/output behavior
|
||||
|
||||
- Stdout/stderr are routed into a shared pipe and read concurrently.
|
||||
- Reader decodes UTF-8 incrementally; invalid byte sequences emit `U+FFFD` replacement chunks.
|
||||
- After process completion, output drain has idle/max guards (`250ms` idle, `2s` max) to avoid hanging on background jobs keeping descriptors open.
|
||||
|
||||
### Cancellation, timeout, and background jobs
|
||||
|
||||
- `CancelToken` is constructed from `timeoutMs` and optional `AbortSignal`.
|
||||
- On cancellation/timeout, shell cancellation token is triggered, then task gets a 2s graceful window before forced abort.
|
||||
- If cancellation occurs, background jobs are terminated (`TERM`, then delayed `KILL`) using brush job metadata.
|
||||
|
||||
`Shell.abort()` behavior:
|
||||
|
||||
- aborts only current running command for that `Shell` instance,
|
||||
- no-op success when nothing is running.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
Common surfaced errors include:
|
||||
|
||||
- session init failures (`Failed to initialize shell`),
|
||||
- cwd errors (`Failed to set cwd`),
|
||||
- env set/pop failures,
|
||||
- snapshot source failures,
|
||||
- pipe creation/clone failures,
|
||||
- execution failure (`Shell execution failed: ...`),
|
||||
- task wrapper failures (`Shell execution task failed: ...`).
|
||||
|
||||
Result-level cancellation flags:
|
||||
|
||||
- timeout -> `exitCode: undefined`, `timedOut: true`.
|
||||
- abort signal -> `exitCode: undefined`, `cancelled: true`.
|
||||
|
||||
## PTY subsystem (`pty`)
|
||||
|
||||
### API model
|
||||
|
||||
`new PtySession()` exposes:
|
||||
|
||||
- `start(options, onChunk?) -> Promise<{ exitCode?, cancelled, timedOut }>`
|
||||
- `write(data)`
|
||||
- `resize(cols, rows)`
|
||||
- `kill()`
|
||||
|
||||
### Runtime lifecycle and state transitions
|
||||
|
||||
`PtySession` state machine:
|
||||
|
||||
- **Idle**: `core: None`.
|
||||
- **Reserved**: `start()` installs control channel synchronously (`core: Some`) before async work begins, so `write/resize/kill` become immediately valid.
|
||||
- **Running**: blocking PTY loop handles child state, reader events, cancellation heartbeat, and control messages.
|
||||
- **Terminal closed**: child exit + reader completion.
|
||||
- **Finalized**: `core` is always reset to `None` after start task completion (success or error).
|
||||
|
||||
Concurrency guard:
|
||||
|
||||
- starting while already running returns `PTY session already running`.
|
||||
|
||||
### Spawn/attach/write/read/terminate patterns
|
||||
|
||||
- PTY opened via `portable_pty::native_pty_system().openpty(...)`.
|
||||
- Command currently runs as `sh -lc <command>` with optional `cwd` and env overrides.
|
||||
- `write()` sends raw bytes to PTY stdin.
|
||||
- `resize()` clamps dimensions (`cols 20..400`, `rows 5..200`) and calls master resize.
|
||||
- `kill()` marks run as cancelled and kills child process.
|
||||
|
||||
Output path:
|
||||
|
||||
- dedicated reader thread reads master stream,
|
||||
- incremental UTF-8 decode with `U+FFFD` replacement on invalid bytes,
|
||||
- chunks forwarded through N-API threadsafe callback.
|
||||
|
||||
### Cancellation and timeout semantics
|
||||
|
||||
- `timeoutMs` and `AbortSignal` feed a `CancelToken`.
|
||||
- loop calls `ct.heartbeat()` periodically; abort triggers child kill.
|
||||
- timeout classification is string-based (`"Timeout"` substring in heartbeat error).
|
||||
|
||||
### Failure behavior
|
||||
|
||||
Error surfaces include:
|
||||
|
||||
- PTY allocation/open failure,
|
||||
- PTY spawn failure,
|
||||
- writer/reader acquisition failure,
|
||||
- child status/wait failures,
|
||||
- lock poisoning,
|
||||
- control-channel disconnection (`PTY session is no longer available`).
|
||||
|
||||
Control call failures when not running:
|
||||
|
||||
- `write/resize/kill` return `PTY session is not running`.
|
||||
|
||||
## Process-tree subsystem (`ps`)
|
||||
|
||||
### API model
|
||||
|
||||
- `killTree(pid, signal) -> number`
|
||||
- `listDescendants(pid) -> number[]`
|
||||
|
||||
TS wrapper also registers native kill-tree integration into shared utils via `setNativeKillTree(native.killTree)`.
|
||||
|
||||
### Platform-specific implementation
|
||||
|
||||
- **Linux**: recursively reads `/proc/<pid>/task/<pid>/children`.
|
||||
- **macOS**: uses `libproc` `proc_listchildpids`.
|
||||
- **Windows**: snapshots process table with `CreateToolhelp32Snapshot`, builds parent->children map, terminates with `OpenProcess(PROCESS_TERMINATE)` + `TerminateProcess`.
|
||||
|
||||
### Kill-tree behavior
|
||||
|
||||
- Descendants are collected recursively.
|
||||
- Kill order is bottom-up (deepest descendants first) to reduce orphan re-parenting.
|
||||
- Root pid is killed last.
|
||||
- Return value is count of successful terminations.
|
||||
|
||||
Signal behavior:
|
||||
|
||||
- POSIX: provided `signal` is passed to `kill`.
|
||||
- Windows: `signal` is ignored; termination is unconditional process terminate.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
This module is intentionally non-throwing at API surface:
|
||||
|
||||
- missing/inaccessible process tree branches are skipped,
|
||||
- per-pid kill failures are counted as unsuccessful (not errors),
|
||||
- lookup miss typically yields `[]` from `listDescendants` and `0` from `killTree`.
|
||||
|
||||
## Key parsing subsystem (`keys`)
|
||||
|
||||
### API model
|
||||
|
||||
Exposed helpers:
|
||||
|
||||
- `parseKey(data, kittyProtocolActive)`
|
||||
- `matchesKey(data, keyId, kittyProtocolActive)`
|
||||
- `parseKittySequence(data)`
|
||||
- `matchesKittySequence(data, expectedCodepoint, expectedModifier)`
|
||||
- `matchesLegacySequence(data, keyName)`
|
||||
|
||||
### Parsing model
|
||||
|
||||
The parser combines:
|
||||
|
||||
- direct single-byte mappings (`enter`, `tab`, `ctrl+<letter>`, printable ASCII),
|
||||
- O(1) legacy escape-sequence lookup (PHF map),
|
||||
- xterm `modifyOtherKeys` parsing,
|
||||
- Kitty protocol parsing (`CSI u`, `CSI ~`, `CSI 1;...<letter>`),
|
||||
- normalization to key IDs (`ctrl+c`, `shift+tab`, `pageUp`, `f5`, etc.).
|
||||
|
||||
Modifier handling:
|
||||
|
||||
- only shift/alt/ctrl bits are compared for key matching,
|
||||
- lock bits are masked out before comparisons.
|
||||
|
||||
Layout behavior:
|
||||
|
||||
- base-layout fallback is intentionally constrained so remapped layouts do not create false matches for ASCII letters/symbols.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- Unrecognized or invalid sequences produce `null` from parse functions.
|
||||
- Match functions return `false` on parse failure or mismatch.
|
||||
- No thrown error surface for malformed key input.
|
||||
|
||||
## JS wrapper API ↔ Rust export mapping
|
||||
|
||||
### Shell + PTY + Process
|
||||
|
||||
| TS wrapper API | Rust N-API export | Notes |
|
||||
|---|---|---|
|
||||
| `executeShell(options, onChunk?)` | `executeShell` (`execute_shell`) | One-shot shell execution |
|
||||
| `new Shell(options?)` | `Shell` class | Persistent shell session |
|
||||
| `shell.run(options, onChunk?)` | `Shell::run` | Reuses session on keepalive control flow |
|
||||
| `shell.abort()` | `Shell::abort` | Aborts active run for that shell instance |
|
||||
| `new PtySession()` | `PtySession` class | Stateful PTY session |
|
||||
| `pty.start(options, onChunk?)` | `PtySession::start` | Interactive PTY run |
|
||||
| `pty.write(data)` | `PtySession::write` | Raw stdin passthrough |
|
||||
| `pty.resize(cols, rows)` | `PtySession::resize` | Clamped terminal dimensions |
|
||||
| `pty.kill()` | `PtySession::kill` | Force-kills active PTY child |
|
||||
| `killTree(pid, signal)` | `killTree` (`kill_tree`) | Children-first process tree termination |
|
||||
| `listDescendants(pid)` | `listDescendants` (`list_descendants`) | Recursive descendants listing |
|
||||
|
||||
### Keys
|
||||
|
||||
| TS wrapper API | Rust N-API export | Notes |
|
||||
|---|---|---|
|
||||
| `matchesKittySequence(data, cp, mod)` | `matchesKittySequence` (`matches_kitty_sequence`) | Kitty codepoint+modifier match |
|
||||
| `parseKey(data, kittyProtocolActive)` | `parseKey` (`parse_key`) | Normalized key-id parser |
|
||||
| `matchesLegacySequence(data, keyName)` | `matchesLegacySequence` (`matches_legacy_sequence`) | Exact legacy sequence map check |
|
||||
| `parseKittySequence(data)` | `parseKittySequence` (`parse_kitty_sequence_napi`) | Structured Kitty parse result |
|
||||
| `matchesKey(data, keyId, kittyProtocolActive)` | `matchesKey` (`matches_key`) | High-level key matcher |
|
||||
|
||||
## Abandoned session cleanup and finalization notes
|
||||
|
||||
- **Shell persistent session**: if a run is cancelled/timed out/errors/non-keepalive control flow, Rust explicitly drops the internal session state. Successful normal runs keep the session for reuse.
|
||||
- **PTY session**: `core` is always cleared after `start()` finishes, including failure paths.
|
||||
- **No explicit JS finalizer-driven kill contract** is exposed by wrappers; cleanup is primarily tied to run completion/cancellation paths. Callers should use `timeoutMs`, `AbortSignal`, `shell.abort()`, or `pty.kill()` for deterministic teardown.
|
||||
@@ -0,0 +1,243 @@
|
||||
# Natives Text/Search Pipeline
|
||||
|
||||
This document maps the `@oh-my-pi/pi-natives` text/search surface (`grep`, `glob`, `text`, `highlight`) from TypeScript wrappers to Rust N-API exports and back to JS result objects.
|
||||
|
||||
Terminology follows `docs/natives-architecture.md`:
|
||||
- **Wrapper**: TS API in `packages/natives/src/*`
|
||||
- **Rust module layer**: N-API exports in `crates/pi-natives/src/*`
|
||||
- **Shared scan cache**: `fs_cache`-backed directory-entry cache used by discovery/search flows
|
||||
|
||||
## Implementation files
|
||||
|
||||
- `packages/natives/src/grep/index.ts`
|
||||
- `packages/natives/src/grep/types.ts`
|
||||
- `packages/natives/src/glob/index.ts`
|
||||
- `packages/natives/src/glob/types.ts`
|
||||
- `packages/natives/src/text/index.ts`
|
||||
- `packages/natives/src/text/types.ts`
|
||||
- `packages/natives/src/highlight/index.ts`
|
||||
- `packages/natives/src/highlight/types.ts`
|
||||
- `crates/pi-natives/src/grep.rs`
|
||||
- `crates/pi-natives/src/glob.rs`
|
||||
- `crates/pi-natives/src/glob_util.rs`
|
||||
- `crates/pi-natives/src/fs_cache.rs`
|
||||
- `crates/pi-natives/src/text.rs`
|
||||
- `crates/pi-natives/src/highlight.rs`
|
||||
- `crates/pi-natives/src/fd.rs`
|
||||
|
||||
## JS API ↔ Rust export mapping
|
||||
|
||||
| JS wrapper API | Rust export (`#[napi(js_name = ...)]`) | Rust module |
|
||||
| --- | --- | --- |
|
||||
| `grep(options, onMatch?)` | `grep` | `grep.rs` |
|
||||
| `searchContent(content, options)` | `search` | `grep.rs` |
|
||||
| `hasMatch(content, pattern, options?)` | `hasMatch` | `grep.rs` |
|
||||
| `fuzzyFind(options)` | `fuzzyFind` | `fd.rs` |
|
||||
| `glob(options, onMatch?)` | `glob` | `glob.rs` |
|
||||
| `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `fs_cache.rs` |
|
||||
| `wrapTextWithAnsi(text, width)` | `wrapTextWithAnsi` | `text.rs` |
|
||||
| `truncateToWidth(text, maxWidth, ellipsis, pad)` | `truncateToWidth` | `text.rs` |
|
||||
| `sliceWithWidth(line, startCol, length, strict?)` | `sliceWithWidth` | `text.rs` |
|
||||
| `extractSegments(line, beforeEnd, afterStart, afterLen, strictAfter)` | `extractSegments` | `text.rs` |
|
||||
| `sanitizeText(text)` | `sanitizeText` | `text.rs` |
|
||||
| `visibleWidth(text)` | `visibleWidth` | `text.rs` |
|
||||
| `highlightCode(code, lang, colors)` | `highlightCode` | `highlight.rs` |
|
||||
| `supportsLanguage(lang)` | `supportsLanguage` | `highlight.rs` |
|
||||
| `getSupportedLanguages()` | `getSupportedLanguages` | `highlight.rs` |
|
||||
|
||||
## Pipeline overview by subsystem
|
||||
|
||||
## 1) Regex search (`grep`, `searchContent`, `hasMatch`)
|
||||
|
||||
### Input/options flow
|
||||
|
||||
1. TS wrapper forwards options to native:
|
||||
- `grep/index.ts` passes `options` mostly unchanged and wraps callback from `(match) => void` to napi threadsafe callback shape `(err, match)`.
|
||||
- `searchContent` and `hasMatch` pass string/`Uint8Array` directly.
|
||||
2. Rust option structs in `grep.rs` deserialize camelCase fields (`ignoreCase`, `maxCount`, `contextBefore`, `contextAfter`, `maxColumns`, `timeoutMs`).
|
||||
3. `grep` creates `CancelToken` from `timeoutMs` + `AbortSignal` and runs inside `task::blocking("grep", ...)`.
|
||||
|
||||
### Execution branches
|
||||
|
||||
- **In-memory branch (pure utility)**
|
||||
- `search` → `search_sync` → `run_search` on provided content bytes.
|
||||
- No filesystem scan, no `fs_cache`.
|
||||
- **Single-file branch (filesystem-dependent)**
|
||||
- `grep_sync` resolves path, checks metadata is file, streams up to `MAX_FILE_BYTES` per file (`4 MiB`) through ripgrep matcher.
|
||||
- **Directory branch (filesystem-dependent)**
|
||||
- Optional cache lookup via `fs_cache::get_or_scan` when `cache: true`.
|
||||
- Fresh scan via `fs_cache::force_rescan` when `cache: false`.
|
||||
- Optional empty-result recheck when cache age exceeds `empty_recheck_ms()`.
|
||||
- Entry filtering: file-only + optional glob filter (`glob_util`) + optional type filter mapping (`js`, `ts`, `rust`, etc.).
|
||||
|
||||
### Search/collection semantics
|
||||
|
||||
- Regex engine: `grep_regex::RegexMatcherBuilder` with `ignoreCase` and `multiline`.
|
||||
- Context resolution:
|
||||
- `contextBefore/contextAfter` override legacy `context`.
|
||||
- Non-content modes zero out context collection.
|
||||
- Output modes:
|
||||
- `content` => one `GrepMatch` per hit.
|
||||
- `count` and `filesWithMatches` both map to count-style entries (`lineNumber=0`, `line=""`, `matchCount` set).
|
||||
- Limits:
|
||||
- Global `offset` and `maxCount` applied across files.
|
||||
- Parallel path is used only when `maxCount` is unset and `offset == 0`; otherwise sequential path preserves deterministic global offset/limit semantics.
|
||||
|
||||
### Result shaping back to JS
|
||||
|
||||
- Rust `SearchResult`/`GrepResult` fields map to TS types via `#[napi(js_name = ...)]`.
|
||||
- Counters are clamped to `u32` before crossing N-API.
|
||||
- Optional booleans are omitted unless true in some paths (`limitReached`).
|
||||
- Streaming callback receives each shaped `GrepMatch` (content or count entry).
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- `searchContent` returns `SearchResult.error` for regex/search failures instead of throwing.
|
||||
- `grep` rejects on hard errors (invalid path, invalid glob/regex, cancellation timeout/abort).
|
||||
- `hasMatch` returns `Result<bool>` and throws on invalid pattern/UTF-8 decoding errors.
|
||||
- File open/search errors in multi-file scans are skipped per-file; scan continues.
|
||||
|
||||
### Malformed regex handling
|
||||
|
||||
`grep.rs` sanitizes braces before regex compile:
|
||||
- Invalid repetition-like braces are escaped (`{`/`}` -> `\{`/`\}`) when they cannot form `{N}`, `{N,}`, `{N,M}`.
|
||||
- This prevents common literal-template fragments (for example `${platform}`) from failing as malformed repetition.
|
||||
- Remaining invalid regex syntax still returns a regex error.
|
||||
|
||||
## 2) File discovery (`glob`) and fuzzy path search (`fuzzyFind`)
|
||||
|
||||
`glob` and `fuzzyFind` share `fs_cache` scans; matching logic differs.
|
||||
|
||||
### `glob` flow
|
||||
|
||||
1. TS wrapper (`glob/index.ts`):
|
||||
- `path.resolve(options.path)`.
|
||||
- Defaults: `pattern="*"`, `hidden=false`, `gitignore=true`, `recursive=true`.
|
||||
2. Rust `glob` builds `GlobConfig` and compiles pattern via `glob_util::compile_glob`.
|
||||
3. Entry source:
|
||||
- `cache=true` => `get_or_scan` + optional stale-empty `force_rescan`.
|
||||
- `cache=false` => `force_rescan(..., store=false)` (fresh only).
|
||||
4. Filtering:
|
||||
- Skip `.git` always.
|
||||
- Skip `node_modules` unless requested (`includeNodeModules` or pattern mentioning node_modules).
|
||||
- Apply glob match.
|
||||
- Apply file-type filter; symlink `file/dir` filters resolve target metadata.
|
||||
5. Optional sort by mtime desc (`sortByMtime`) before truncating to `maxResults`.
|
||||
|
||||
### `fuzzyFind` flow (implemented in `fd.rs`)
|
||||
|
||||
1. TS wrapper is exported from `grep` module, but Rust implementation lives in `fd.rs`.
|
||||
2. Shared scan source from `fs_cache` with same cache/no-cache split and stale-empty recheck policy.
|
||||
3. Scoring:
|
||||
- exact / starts-with / contains / subsequence-based fuzzy score
|
||||
- separator/punctuation-normalized scoring path
|
||||
- directory bonus and deterministic tie-break (`score desc`, then `path asc`)
|
||||
4. Symlink entries are excluded from fuzzy results.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- Invalid glob pattern => error from `glob_util::compile_glob`.
|
||||
- Search root must be an existing directory (`resolve_search_path`), otherwise error.
|
||||
- Cancellation/timeouts propagate as abort errors via `CancelToken::heartbeat()` checks in loops.
|
||||
|
||||
### Malformed glob handling
|
||||
|
||||
`glob_util::build_glob_pattern` is tolerant:
|
||||
- Normalizes `\` to `/`.
|
||||
- Auto-prefixes simple recursive patterns with `**/` when `recursive=true`.
|
||||
- Auto-closes unbalanced `{...` alternation groups before compile.
|
||||
|
||||
## 3) Shared scan/cache lifecycle (`fs_cache`)
|
||||
|
||||
`fs_cache` stores scan results as normalized relative entries (`path`, `fileType`, optional `mtime`) keyed by:
|
||||
- canonical search root
|
||||
- `include_hidden`
|
||||
- `use_gitignore`
|
||||
|
||||
### Cache state transitions
|
||||
|
||||
1. **Miss / disabled**
|
||||
- TTL is `0` or key absent/expired -> fresh `collect_entries`.
|
||||
2. **Hit**
|
||||
- Entry age `< cache_ttl_ms()` -> return cached entries + `cache_age_ms`.
|
||||
3. **Stale-empty recheck** (caller policy in `glob`/`grep`/`fd`)
|
||||
- If query yields zero matches and `cache_age_ms >= empty_recheck_ms()`, force one rescan.
|
||||
4. **Invalidation**
|
||||
- `invalidateFsScanCache(path?)`:
|
||||
- no arg: clear all keys
|
||||
- path arg: remove keys whose root prefixes that target path
|
||||
|
||||
### Stale-result tradeoff
|
||||
|
||||
- Cache favors low-latency repeated scans over immediate consistency.
|
||||
- TTL window can return stale positives/negatives.
|
||||
- Empty-result recheck reduces stale negatives for older cached scans at the cost of one extra scan.
|
||||
- Explicit invalidation is the intended correctness hook after file mutations.
|
||||
|
||||
## 4) ANSI text utilities (`text`)
|
||||
|
||||
These are pure, in-memory utilities (no filesystem scanning).
|
||||
|
||||
### Boundaries and responsibilities
|
||||
|
||||
- **`text.rs` owns terminal-cell semantics**:
|
||||
- ANSI sequence parsing
|
||||
- grapheme-aware width and slicing
|
||||
- wrap/truncate/sanitize behavior
|
||||
- **`grep.rs` line truncation (`maxColumns`) is separate**:
|
||||
- simple character-boundary truncation of matched lines with `...`
|
||||
- not ANSI-state-preserving and not terminal-cell width aware
|
||||
|
||||
### Key behaviors
|
||||
|
||||
- `wrapTextWithAnsi`: wraps by visible width, carries active SGR codes across wrapped lines.
|
||||
- `truncateToWidth`: visible-cell truncation with ellipsis policy (`Unicode`, `Ascii`, `Omit`), optional right padding, and fast-path returning original JS string when unchanged.
|
||||
- `sliceWithWidth`: column slicing with optional strict width enforcement.
|
||||
- `extractSegments`: extracts before/after segments around an overlay while restoring ANSI state for the `after` segment.
|
||||
- `sanitizeText`: strips ANSI escapes + control chars, drops lone surrogates, normalizes CR/LF by removing `\r`.
|
||||
- `visibleWidth`: counts visible terminal cells (tabs use fixed `TAB_WIDTH` from Rust implementation).
|
||||
|
||||
### Failure behavior
|
||||
|
||||
Text functions generally return deterministic transformed output; errors are limited to JS string conversion boundaries (N-API argument conversion failures).
|
||||
|
||||
## 5) Syntax highlighting (`highlight`)
|
||||
|
||||
`highlight.rs` is pure transformation (no FS, no cache).
|
||||
|
||||
### Flow
|
||||
|
||||
1. Wrapper forwards `code`, optional `lang`, and ANSI color palette.
|
||||
2. Rust resolves syntax by:
|
||||
- token/name lookup
|
||||
- extension lookup
|
||||
- alias table fallback (`ts/tsx/js -> JavaScript`, etc.)
|
||||
- fallback to plain text syntax when unresolved
|
||||
3. Parse each line with syntect `ParseState` and scope stack.
|
||||
4. Map scopes to 11 semantic color categories and inject/reset ANSI color codes.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- Per-line parse failure does not fail the call: that line is appended unhighlighted and processing continues.
|
||||
- Unknown/unsupported language falls back to plain text syntax.
|
||||
|
||||
## Pure utility vs filesystem-dependent flows
|
||||
|
||||
| Flow | Filesystem access | Shared cache | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `searchContent` / `hasMatch` | No | No | regex on provided bytes/string only |
|
||||
| `text` module functions | No | No | ANSI/width/sanitization only |
|
||||
| `highlight` module functions | No | No | syntax + ANSI coloring only |
|
||||
| `glob` | Yes | Optional | directory scans + glob filtering |
|
||||
| `fuzzyFind` | Yes | Optional | directory scans + fuzzy scoring |
|
||||
| `grep` (file/dir path) | Yes | Optional (dir mode) | ripgrep over files, optional filters/callback |
|
||||
|
||||
## End-to-end lifecycle summary
|
||||
|
||||
1. Caller invokes TS wrapper with typed options.
|
||||
2. Wrapper normalizes defaults (notably `glob`) and forwards to `native.*` export.
|
||||
3. Rust validates/normalizes options and builds matcher/search config.
|
||||
4. For filesystem flows, entries are scanned (cache hit/miss/rescan) then filtered/scored.
|
||||
5. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution.
|
||||
6. Rust shapes outputs into N-API objects (`lineNumber`, `matchCount`, `limitReached`, etc.).
|
||||
7. TS wrapper returns typed JS objects (and optional per-match callbacks for `grep`/`glob`).
|
||||
Reference in New Issue
Block a user