oh-my-pi/docs/natives-text-search-pipeline.md

Natives Text/Search Pipeline

This document maps the @oh-my-pi/pi-natives text/search/code surface from generated JS/TS exports to Rust N-API modules and back to JS result objects.

Terminology follows docs/natives-architecture.md:

• Generated binding: public API in packages/natives/native/index.d.ts.
• Rust module layer: N-API exports in crates/pi-natives/src/*.
• Shared scan cache: optional pi-walker directory-entry cache used by discovery flows.

Implementation files

• packages/natives/native/index.d.ts
• crates/pi-natives/src/grep.rs
• crates/pi-natives/src/glob.rs
• crates/pi-natives/src/glob_util.rs
• crates/pi-natives/src/iofs.rs
• crates/pi-walker/src/lib.rs
• crates/pi-walker/src/cache.rs
• crates/pi-natives/src/ast.rs
• crates/pi-natives/src/text.rs
• crates/pi-natives/src/highlight.rs
• crates/pi-natives/src/tokens.rs

JS API ↔ Rust export mapping

JS API	Rust export (#[napi], snake_case -> camelCase)	Rust module
grep(options, onMatch?)	grep	grep.rs
search(content, options)	search	grep.rs
hasMatch(content, pattern, ignoreCase?, multiline?)	hasMatch	grep.rs
fuzzyFind(options)	fuzzyFind	fd.rs
glob(options, onMatch?)	glob	glob.rs
invalidateFsScanCache(path?)	invalidateFsScanCache	iofs.rs
astGrep(options)	astGrep	ast.rs
astMatch(options)	astMatch	ast.rs
astEdit(options)	astEdit	ast.rs
wrapTextWithAnsi(text, width, tabWidth)	wrapTextWithAnsi	text.rs
truncateToWidth(text, maxWidth, ellipsis, pad, tabWidth)	truncateToWidth	text.rs
sliceWithWidth(line, startCol, length, strict, tabWidth)	sliceWithWidth	text.rs
extractSegments(line, beforeEnd, afterStart, afterLen, strictAfter, tabWidth)	extractSegments	text.rs
visibleWidth(text, tabWidth)	visibleWidth	text.rs
setHangulCompatJamoWidthOverride(value)	setHangulCompatJamoWidthOverride	text.rs
highlightCode(code, lang, colors)	highlightCode	highlight.rs
supportsLanguage(lang)	supportsLanguage	highlight.rs
getSupportedLanguages()	getSupportedLanguages	highlight.rs
countTokens(input, encoding?)	countTokens	tokens.rs

Pipeline overview by subsystem

1) Regex search (grep, search, hasMatch)

Input/options flow

1. Callers invoke generated native exports directly; there is no package-local TS wrapper that renames search to searchContent.
2. Rust option structs in grep.rs deserialize camelCase fields including ignoreCase, maxCount, maxCountPerFile, contextBefore, contextAfter, maxColumns, and timeoutMs.
3. grep creates CancelToken from timeoutMs + AbortSignal and runs inside task::blocking("grep", ...). Filesystem grep does not expose or use the shared walker cache.
4. search and hasMatch operate on provided string/Uint8Array content and do not scan the filesystem.

Execution branches

• In-memory branch
  • search -> search_sync / search helpers over provided content bytes.
  • hasMatch compiles/checks pattern against provided content and returns a boolean.
  • No filesystem scan or walker cache.
• Single-file branch
  • grep resolves path, checks metadata is file, and searches that file.
• Directory branch
  • Candidate discovery uses pi-walker without scan caching.
  • Walker policy applies hidden/gitignore settings and skips skippable directory errors.
  • Entry filtering: file-only + optional glob filter (glob_util) + optional type filter mapping (js, ts, rust, etc.).

Search/collection semantics

• Matcher selection: the Rust regex engine is tried first, then PCRE2 for features such as lookaround/backreferences. OMP_PCRE2_JIT=0/false disables PCRE2 JIT and 1 enables it; when unset, JIT is enabled except on macOS.
• Context resolution:
  • contextBefore/contextAfter override legacy context.
  • Non-content modes do not collect context.
• Output modes:
  • content -> one GrepMatch per hit.
  • count and filesWithMatches map to count-style entries (lineNumber=0, line="", matchCount set).
  • offset and maxCount are applied during aggregation across sorted file results; maxCountPerFile can additionally prevent one hot file consuming the content-mode budget.
  • Directory searches use parallel filesystem walking/searching, then aggregate per-file results to preserve global offset/limit semantics. Small ordered callback streams may stop early; larger streams use a bounded ordered window.

Result shaping back to JS

• Rust SearchResult/GrepResult fields map to TS interfaces via N-API object conversion.
• Counters are clamped before crossing N-API where needed.
• GrepResult.limitReached is optional and emitted when true; skippedOversized reports files skipped over the 4 MiB limit.
• Streaming callback receives each shaped GrepMatch for content or count-style entries.

Failure behavior

• search returns SearchResult.error for regex/search failures instead of throwing.
• grep rejects on hard errors such as invalid path or cancellation timeout/abort. Patterns rejected by both regex engines fall back to a literal search rather than producing a regex error.
• hasMatch returns a boolean on success; matcher construction uses the same tolerant fallback.
• Unreadable/non-regular files in multi-file scans are skipped; oversized files are counted in skippedOversized.

Malformed regex handling

grep.rs sanitizes braces before regex compile:

• Invalid repetition-like braces are escaped ({/} -> \{/\}) when they cannot form {N}, {N,}, {N,M}.
• This prevents common literal-template fragments (for example ${platform}) from failing as malformed repetition.
• A compile failure for an unclosed/unopened group triggers one targeted retry with unescaped parentheses escaped while preserving the rest of the regex.
• If both engines still reject the pattern, the entire original pattern is escaped and searched literally.

2) File discovery (glob) and fuzzy path search (fuzzyFind)

glob and fuzzyFind share the optional pi-walker scan cache; matching logic differs. Cache use defaults to false for both APIs.

glob flow

1. Caller passes GlobOptions directly. pattern and path are required in the generated type.
2. Rust resolves the search path and compiles pattern via glob_util::compile_glob.
3. Entry source:
   • cache=true -> shared walker cache + optional stale-empty rescan.
   • cache=false -> a fresh scan that neither reads nor updates the cache.
4. Filtering:
   • skip .git always;
   • skip node_modules unless requested (includeNodeModules) or pattern mentions node_modules;
   • apply glob match;
   • apply file-type filter; symlink file/dir filters resolve target metadata.
5. Optional sort by mtime descending (sortByMtime) before truncating to maxResults.

fuzzyFind flow

1. Rust implementation lives in fd.rs; generated export is fuzzyFind.
2. Shared pi-walker scan source with the same opt-in cache and stale-empty recheck policy.
3. Scoring:
   • exact / starts-with / contains / subsequence-based fuzzy score;
   • separator/punctuation-normalized scoring path;
   • directory bonus and deterministic tie-break (score desc, then path asc).
4. Symlink entries are excluded from fuzzy results.

Failure behavior

• Invalid glob pattern returns an error from glob_util::compile_glob.
• Search root must resolve to an existing directory for directory discovery flows.
• Cancellation/timeouts propagate as abort errors via the caller-supplied walker heartbeat and result-processing checks.

Malformed glob handling

glob_util::build_glob_pattern is tolerant:

• normalizes \ to /,
• auto-prefixes simple recursive patterns with **/ when recursive=true,
• auto-closes unbalanced {... alternation groups before compile.

3) AST search/match/edit (astGrep, astMatch, astEdit)

ast.rs exposes syntax-aware code search and rewrite operations.

• astGrep(options) returns matches with byte/line/column coordinates and optional metavariable bindings.
• astMatch(options) runs the same patterns against an in-memory source string instead of files; lang is required (there is no path to infer it from), and the result keeps matches, totalMatches, limitReached, and parse errors but omits the file-count fields.
• astEdit(options) returns replacement changes, per-file counts, searched/touched file counts, parse errors, and whether edits were applied.
• dryRun defaults to true for edit options in the generated documentation.
• Options include language override, path/glob/selector, strictness, limits, parse-error policy, signal, and timeoutMs.

These exports are direct native APIs used by tooling; they are not mediated by a TS wrapper in packages/natives.

4) Shared scan/cache lifecycle (pi-walker)

pi-walker owns traversal and cache policy. crates/pi-natives/src/iofs.rs contains only JavaScript-facing DTO conversion, error mapping, and the invalidation export.

The cache stores normalized relative entries (path, fileType, optional mtime and regular-file size) keyed by canonical search root plus the complete WalkOptions value, including hidden/gitignore, link-following, filtering, ordering, and detail policy.

Configuration is read from environment once:

• FS_SCAN_CACHE_TTL_MS: cache TTL, default 1000.
• FS_SCAN_EMPTY_RECHECK_MS: cached-empty recheck age, default 200.
• FS_SCAN_CACHE_MAX_ENTRIES: maximum entries in the cache map, default 16.
• PI_WALK_WORKERS: walker Rayon pool size, default 4.

Cache state transitions

1. Disabled / miss / expired
   • disabled requests collect fresh without reading or updating the cache;
   • enabled misses and entries at or beyond TTL collect fresh and populate it.
2. Hit
   • an entry younger than TTL returns cached entries and cache age.
3. Stale-empty recheck
   • when the caller enables configured rechecking, an empty cached query at or beyond the threshold is scanned once again.
4. Invalidation
   • invalidateFsScanCache() clears all keys;
   • invalidateFsScanCache(path) removes cached roots containing that path.

Cache favors low-latency repeated scans over immediate consistency. Explicit invalidation is the correctness hook after writes, edits, renames, or deletes.

5) ANSI text utilities (text)

These are pure, in-memory utilities.

Boundaries and responsibilities

• text.rs owns terminal-cell semantics:
  • ANSI sequence parsing,
  • grapheme-aware width and slicing,
  • wrap/truncate/slice behavior,
  • explicit tab-width parameter on width-sensitive APIs.
• grep.rs line truncation (maxColumns) is separate:
  • simple character-boundary truncation of matched lines with ...,
  • not ANSI-state-preserving and not terminal-cell width aware.

Key behaviors

• wrapTextWithAnsi: wraps by visible width, carries active SGR codes across wrapped lines.
• truncateToWidth: visible-cell truncation with ellipsis policy (Unicode, Ascii, Omit), optional right padding.
• sliceWithWidth: column slicing with optional strict width enforcement.
• extractSegments: extracts before/after segments around an overlay while restoring ANSI state for the after segment.
• setHangulCompatJamoWidthOverride(value) controls U+3131–U+318E width correction for client-terminal compatibility: 0 uses the platform fallback, 1 forces one cell, 2 forces two, and 3 follows Unicode width.
• visibleWidth: counts visible terminal cells using caller-supplied tab width.

Failure behavior

Text functions generally return deterministic transformed output; errors are limited to N-API argument/string conversion boundaries.

6) Syntax highlighting (highlight)

highlight.rs is pure transformation; it does not use the filesystem scan cache.

Flow

1. Caller passes code, optional lang, and ANSI color palette.
2. Rust resolves syntax by token/name lookup, extension lookup, alias table fallback, then plain-text fallback.
3. Each line is parsed with syntect ParseState and scope stack.
4. Scopes map to semantic color categories and ANSI color codes are injected/reset.

Failure behavior

• Per-line parse failure does not fail the call: that line is appended unhighlighted and processing continues.
• Unknown/unsupported language falls back to plain text syntax.

7) Token counting (tokens)

countTokens(input, encoding?) is an in-memory utility.

• input may be a single string or an array of strings.
• Arrays return one aggregate count and are encoded in parallel in Rust.
• Default encoding is O200kBase; Cl100kBase is also available.
• The implementation uses ordinary tokenization, not special-token handling.

Pure utility vs filesystem-dependent flows

Flow	Filesystem access	Shared cache	Notes
search / hasMatch	No	No	regex on provided bytes/string only
text module functions	No	No	ANSI/width utilities only
highlight module functions	No	No	syntax + ANSI coloring only
countTokens	No	No	tokenization only
astMatch	No	No	in-memory syntax-aware match (no disk)
astGrep / astEdit	Yes	No	syntax-aware file search/edit
glob	Yes	Optional	directory scans + glob filtering
fuzzyFind	Yes	Optional	directory scans + fuzzy scoring
grep (file/dir path)	Yes	No	walker discovery + regex search, optional filters/callback

End-to-end lifecycle summary

1. Caller invokes generated native export with typed options.
2. Rust validates/normalizes options and builds matcher/search config.
3. For filesystem flows, entries are scanned (cache hit/miss/rescan where applicable) then filtered/scored/searched.
4. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution.
5. Rust shapes outputs into N-API objects (lineNumber, matchCount, limitReached, etc.).
6. Generated bindings return typed JS objects and optional per-match callbacks for grep/glob.