From 9865a4ce6c094da3cd314eaee517a2f46c1cf691 Mon Sep 17 00:00:00 2001 From: can1357 Date: Thu, 30 Apr 2026 06:44:16 +0200 Subject: [PATCH] docs: update docs --- docs/bash-tool-runtime.md | 46 +- docs/blob-artifact-architecture.md | 47 +- docs/compaction.md | 57 ++- docs/config-usage.md | 63 ++- docs/custom-tools.md | 68 +-- docs/environment-variables.md | 404 ++++++++++-------- docs/extension-loading.md | 13 +- docs/extensions.md | 112 ++--- docs/fs-scan-cache-architecture.md | 38 +- docs/gemini-manifest-extensions.md | 10 +- docs/handoff-generation-pipeline.md | 58 +-- docs/hooks.md | 107 ++--- docs/marketplace.md | 105 ++--- docs/mcp-config.md | 21 +- docs/mcp-protocol-transports.md | 64 +-- docs/mcp-runtime-lifecycle.md | 73 ++-- docs/mcp-server-tool-authoring.md | 27 +- docs/memory.md | 68 +-- docs/models.md | 28 +- docs/natives-addon-loader-runtime.md | 225 ++++------ docs/natives-architecture.md | 141 +++--- docs/natives-binding-contract.md | 273 ++++-------- docs/natives-build-release-debugging.md | 209 +++++---- docs/natives-media-system-utils.md | 169 ++++---- docs/natives-rust-task-cancellation.md | 123 +++--- docs/natives-shell-pty-process.md | 142 +++--- docs/natives-text-search-pipeline.md | 260 +++++------ docs/non-compaction-retry-policy.md | 32 +- docs/notebook-tool-runtime.md | 5 +- docs/plugin-manager-installer-plumbing.md | 31 +- docs/porting-from-pi-mono.md | 86 ++-- docs/porting-to-natives.md | 167 ++++---- docs/provider-streaming-internals.md | 21 +- docs/python-repl.md | 15 +- docs/resolve-tool-runtime.md | 104 +++-- docs/rpc.md | 76 +++- docs/rulebook-matching-pipeline.md | 39 +- docs/sdk.md | 148 ++++--- docs/secrets.md | 46 +- ...ion-operations-export-share-fork-resume.md | 34 +- docs/session-switching-and-recent-listing.md | 18 +- docs/session-tree-plan.md | 3 +- docs/session.md | 55 ++- docs/skills.md | 21 +- docs/skills/authoring-extensions.md | 2 +- docs/skills/authoring-hooks.md | 8 +- docs/skills/authoring-marketplaces.md | 6 +- docs/slash-command-internals.md | 23 +- docs/task-agent-discovery.md | 22 +- docs/theme.md | 22 +- docs/tree.md | 12 +- docs/ttsr-injection-lifecycle.md | 60 +-- docs/tui-runtime-internals.md | 27 +- docs/tui.md | 70 +-- 54 files changed, 2145 insertions(+), 1959 deletions(-) diff --git a/docs/bash-tool-runtime.md b/docs/bash-tool-runtime.md index 89c6b9329..f0b7ba6fa 100644 --- a/docs/bash-tool-runtime.md +++ b/docs/bash-tool-runtime.md @@ -10,29 +10,24 @@ There are two different bash execution surfaces in coding-agent: 1. **Tool-call surface** (`toolName: "bash"`): used when the model calls the bash tool. - Entry point: `BashTool.execute()`. + - Parameters include `command`, optional `env`, `timeout`, `cwd`, `head`, `tail`, `pty`, and, when `async.enabled` is true, `async`. 2. **User bang-command surface** (`!cmd` from interactive input or RPC `bash` command): session-level helper path. - Entry point: `AgentSession.executeBash()`. -Both eventually use `executeBash()` in `src/exec/bash-executor.ts` for non-PTY execution, but only the tool-call path runs normalization/interception and tool renderer logic. +Both eventually use `executeBash()` in `src/exec/bash-executor.ts` for non-PTY execution, but only the tool-call path runs normalization/interception, optional managed background-job handling, and tool renderer logic. ## End-to-end tool-call pipeline -## 1) Input normalization and parameter merge +## 1) Input handling and parameter merge -`BashTool.execute()` first normalizes the raw command via `normalizeBashCommand()`: +`BashTool.execute()` currently handles input before execution as follows: -- extracts trailing `| head -n N`, `| head -N`, `| tail -n N`, `| tail -N` into structured limits, -- trims trailing/leading whitespace, -- keeps internal whitespace intact. +- validates optional `env` names against shell-variable syntax, +- extracts a leading `cd && ...` into `cwd` when `cwd` was not supplied, +- rejects `async: true` when `async.enabled` is false, +- uses only explicit `head`/`tail` tool args for post-run filtering. -Then it merges extracted limits with explicit tool args: - -- explicit `head`/`tail` args override extracted values, -- extracted values are fallback only. - -### Caveat - -`bash-normalize.ts` comments mention stripping `2>&1`, but current implementation does not remove it. Runtime behavior is still correct (stdout/stderr are already merged), but the normalization behavior is narrower than comments suggest. +`normalizeBashCommand()` still exists in `src/tools/bash-normalize.ts`, but `BashTool.execute()` does not call it in the current source. Trailing shell pipes such as `| head -n 50` remain part of the shell command unless the caller uses the structured `head`/`tail` args. ## 2) Optional interception (blocked-command path) @@ -80,7 +75,7 @@ Before execution, the tool allocates an artifact path/id (best-effort) for trunc `BashTool` chooses PTY execution only when all are true: -- `bash.virtualTerminal === "on"` +- tool input `pty === true` - `PI_NO_PTY !== "1"` - tool context has UI (`ctx.hasUI === true` and `ctx.ui` set) @@ -100,9 +95,9 @@ That means print mode and non-UI RPC/tool contexts always use non-PTY. - serialized shell env, - optional agent session key. -For session-level executions, `AgentSession.executeBash()` passes `sessionKey: this.sessionId`, isolating reuse per session. +Session-level bang-command executions pass `sessionKey: this.sessionId`. -Tool-call path does **not** pass `sessionKey`, so reuse scope is based on shell config/snapshot/env. +Tool-call executions pass `sessionKey: this.session.getSessionId?.()`, when available. In both surfaces, a session key isolates shell reuse per session; without one, reuse falls back to shell config/snapshot/env. ## Shell config and snapshot behavior @@ -122,7 +117,7 @@ If `prefix` is configured, command becomes: ## Streaming and cancellation -`Shell.run()` streams chunks to callback. Executor pipes each chunk into `OutputSink` and optional `onChunk` callback. +`Shell.run()` streams chunks to `OutputSink` and optional `onChunk` callback. Cancellation: @@ -178,12 +173,14 @@ Both PTY and non-PTY paths use `OutputSink`. Runtime truncation is byte-threshold based in `OutputSink` (50KB default). It does not enforce a hard 2000-line cap in this code path. -## Live tool updates +## Live tool updates and async jobs -For non-PTY execution, `BashTool` uses a separate `TailBuffer` for partial updates and emits `onUpdate` snapshots while command is running. +For non-PTY foreground execution, `BashTool` uses a separate `TailBuffer` for partial updates and emits `onUpdate` snapshots while command is running. For PTY execution, live rendering is handled by custom UI overlay, not by `onUpdate` text chunks. +When `async.enabled` is true and the call passes `async: true`, `BashTool` starts a managed bash job, returns a running job result with a job id, and stores completion through the session managed-job path. Auto-backgrounding can also start this path after `bash.autoBackground.thresholdMs`. + ## Result shaping, metadata, and error mapping After execution: @@ -241,7 +238,7 @@ This component is wired by `CommandController.handleBashCommand()` and fed from | Surface | Entry path | PTY eligible | Live output UX | Error surfacing | | ------------------------------ | ----------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------ | -| Interactive tool call | `BashTool.execute` | Yes, when `bash.virtualTerminal=on` and UI exists and `PI_NO_PTY!=1` | PTY overlay (interactive) or streamed tail updates | Tool errors become `toolResult.isError` | +| Interactive tool call | `BashTool.execute` | Yes, when `pty=true` and UI exists and `PI_NO_PTY!=1` | PTY overlay (interactive) or streamed tail updates | Tool errors become `toolResult.isError` | | Print mode tool call | `BashTool.execute` | No (no UI context) | No TUI overlay; output appears in event stream/final assistant text flow | Same tool error mapping | | RPC tool call (agent tooling) | `BashTool.execute` | Usually no UI -> non-PTY | Structured tool events/results | Same tool error mapping | | Interactive bang command (`!`) | `AgentSession.executeBash` + `BashExecutionComponent` | No (uses executor directly) | Dedicated bash execution component | Controller catches exceptions and shows UI error | @@ -258,13 +255,12 @@ This component is wired by `CommandController.handleBashCommand()` and fed from ## Implementation files -- [`src/tools/bash.ts`](../packages/coding-agent/src/tools/bash.ts) — tool entrypoint, normalization/interception, PTY/non-PTY selection, result/error mapping, bash tool renderer. -- [`src/tools/bash-normalize.ts`](../packages/coding-agent/src/tools/bash-normalize.ts) — command normalization and post-run head/tail filtering. +- [`src/tools/bash.ts`](../packages/coding-agent/src/tools/bash.ts) — tool entrypoint, input handling/interception, async and PTY/non-PTY selection, result/error mapping, bash tool renderer. +- [`src/tools/bash-normalize.ts`](../packages/coding-agent/src/tools/bash-normalize.ts) — post-run head/tail filtering; also contains an unused command-normalization helper. - [`src/tools/bash-interceptor.ts`](../packages/coding-agent/src/tools/bash-interceptor.ts) — interceptor rule matching and blocked-command messages. - [`src/exec/bash-executor.ts`](../packages/coding-agent/src/exec/bash-executor.ts) — non-PTY executor, shell session reuse, cancellation wiring, output sink integration. - [`src/tools/bash-interactive.ts`](../packages/coding-agent/src/tools/bash-interactive.ts) — PTY runtime, overlay UI, input normalization, non-interactive env defaults. -- [`src/session/streaming-output.ts`](../packages/coding-agent/src/session/streaming-output.ts) — `OutputSink` truncation/artifact spill and summary metadata. -- [`src/tools/output-utils.ts`](../packages/coding-agent/src/tools/output-utils.ts) — artifact allocation helpers and streaming tail buffer. +- [`src/session/streaming-output.ts`](../packages/coding-agent/src/session/streaming-output.ts) — `OutputSink`, `TailBuffer`, truncation/artifact spill, and summary metadata. - [`src/tools/output-meta.ts`](../packages/coding-agent/src/tools/output-meta.ts) — truncation metadata shape + notice injection wrapper. - [`src/session/agent-session.ts`](../packages/coding-agent/src/session/agent-session.ts) — session-level `executeBash`, message recording, abort lifecycle. - [`src/modes/components/bash-execution.ts`](../packages/coding-agent/src/modes/components/bash-execution.ts) — interactive `!` command execution component. diff --git a/docs/blob-artifact-architecture.md b/docs/blob-artifact-architecture.md index 0a809793c..afbd0088c 100644 --- a/docs/blob-artifact-architecture.md +++ b/docs/blob-artifact-architecture.md @@ -6,7 +6,7 @@ This document describes how coding-agent stores large/binary payloads outside se The runtime uses two different persistence mechanisms for different data shapes: -- **Content-addressed blobs** (`blob:sha256:`): global, binary-oriented storage used to externalize large image base64 payloads from persisted session entries. +- **Content-addressed blobs** (`blob:sha256:`): global storage used to externalize large image base64 payloads and provider image data URLs from persisted session entries. - **Session-scoped artifacts** (files under `/`): per-session text files used for full tool outputs and subagent outputs. They are intentionally separate: @@ -48,7 +48,7 @@ Artifact types share this directory: ## Blob IDs: content hash -`BlobStore.put()` computes SHA-256 over raw binary bytes and returns: +`BlobStore.put()` computes SHA-256 over the bytes it is given and returns: - `hash`: hex digest, - `path`: `/`, @@ -80,13 +80,14 @@ Before session entries are written (`#rewriteFile` / incremental persist), `Sess Key behaviors: -1. **Large string truncation**: oversized strings are cut and suffixed with `"[Session persistence truncated large content]"`. +1. **Large string truncation**: oversized strings are cut and suffixed with `"[Session persistence truncated large content]"`; signature fields (`thinkingSignature`, `thoughtSignature`, `textSignature`) are cleared instead of truncated. 2. **Transient field stripping**: `partialJson` and `jsonlEvents` are removed from persisted entries. 3. **Image externalization to blobs**: - - only applies to image blocks in `content` arrays, - - only when `data` is not already a blob ref, - - only when base64 length is at least threshold (`BLOB_EXTERNALIZE_THRESHOLD = 1024`), - - replaces inline base64 with `blob:sha256:`. + - image blocks in `content` arrays are externalized when `data` is not already a blob ref and base64 length is at least threshold (`BLOB_EXTERNALIZE_THRESHOLD = 1024`), + - provider-style `image_url` data URLs are externalized when they start with `data:image/` and contain `;base64,`, + - image block `data` is stored as decoded binary bytes, + - provider data URLs are stored as the original UTF-8 data URL string, + - persisted values are replaced with `blob:sha256:`. This keeps session JSONL compact while preserving recoverability. @@ -94,11 +95,12 @@ This keeps session JSONL compact while preserving recoverability. When opening a session (`setSessionFile`), after migrations, `SessionManager` runs `resolveBlobRefsInEntries()`. -For each message/custom-message image block with `blob:sha256:`: +For message/custom-message image blocks with `blob:sha256:` and for persisted provider `image_url` fields with blob refs: - reads blob bytes from blob store, -- converts bytes back to base64, -- mutates in-memory entry to inline base64 for runtime consumers. +- converts image-block bytes back to base64, +- converts provider `image_url` blobs back to the original string, +- mutates in-memory entry fields for runtime consumers. If blob is missing: @@ -200,19 +202,19 @@ Blob implications after fork: ## Failure handling and fallback paths -| Case | Behavior | -| --- | --- | -| Blob file missing during rehydration | Warn and keep `blob:sha256:` ref string in-memory | -| Blob read ENOENT via `BlobStore.get` | Returns `null` | -| Artifact directory missing (`ArtifactManager.listFiles`) | Returns empty list (allocation can start fresh) | -| Artifact directory missing (`artifact://` / `agent://`) | Throws explicit `No artifacts directory found` | -| Artifact ID not found | Throws with available IDs listing | -| OutputSink artifact writer init fails | Continues with tail-only truncation (no full-output artifact) | -| No session file (some task paths) | Task tool falls back to temp artifacts directory for subagent outputs | +| Case | Behavior | +| -------------------------------------------------------- | --------------------------------------------------------------------- | +| Blob file missing during rehydration | Warn and keep `blob:sha256:` ref string in-memory | +| Blob read ENOENT via `BlobStore.get` | Returns `null` | +| Artifact directory missing (`ArtifactManager.listFiles`) | Returns empty list (allocation can start fresh) | +| Artifact directory missing (`artifact://` / `agent://`) | Throws explicit `No artifacts directory found` | +| Artifact ID not found | Throws with available IDs listing | +| OutputSink artifact writer init fails | Continues with tail-only truncation (no full-output artifact) | +| No session file (some task paths) | Task tool falls back to temp artifacts directory for subagent outputs | ## Binary blob externalization vs text-output artifacts -- **Blob externalization** is for binary image payloads inside persisted session entry content; it replaces inline base64 in JSONL with stable content refs. +- **Blob externalization** is for image payloads inside persisted session entry content and provider image data URLs; it replaces inline payload strings in JSONL with stable content refs. - **Artifacts** are plain text files for execution output and subagent output; they are addressable by session-local IDs through internal URLs. The two systems intersect only indirectly (both reduce session JSONL bloat) but have different identity, lifetime, and retrieval paths. @@ -220,13 +222,12 @@ The two systems intersect only indirectly (both reduce session JSONL bloat) but ## Implementation files - [`src/session/blob-store.ts`](../packages/coding-agent/src/session/blob-store.ts) — blob reference format, hashing, put/get, externalize/resolve helpers. -- [`src/session/artifacts.ts`](../packages/coding-agent/src/session/artifacts.ts) — session artifact directory model and numeric artifact ID allocation. +- [`src/session/artifacts.ts`](../packages/coding-agent/src/session/artifacts.ts) — session artifact directory model and numeric artifact ID/path allocation. - [`src/session/streaming-output.ts`](../packages/coding-agent/src/session/streaming-output.ts) — `OutputSink` truncation/spill-to-file behavior and summary metadata. - [`src/session/session-manager.ts`](../packages/coding-agent/src/session/session-manager.ts) — persistence transforms, blob rehydration on load, session fork/move interactions. - [`src/session/agent-session.ts`](../packages/coding-agent/src/session/agent-session.ts) — artifact directory copy during interactive fork. -- [`src/tools/output-utils.ts`](../packages/coding-agent/src/tools/output-utils.ts) — tool artifact manager bootstrap and per-tool artifact path allocation. - [`src/internal-urls/artifact-protocol.ts`](../packages/coding-agent/src/internal-urls/artifact-protocol.ts) — `artifact://` resolver. - [`src/internal-urls/agent-protocol.ts`](../packages/coding-agent/src/internal-urls/agent-protocol.ts) — `agent://` resolver + JSON extraction. - [`src/sdk.ts`](../packages/coding-agent/src/sdk.ts) — internal URL router wiring and artifacts-dir resolver. - [`src/task/output-manager.ts`](../packages/coding-agent/src/task/output-manager.ts) — session-scoped agent output ID allocation for `agent://`. -- [`src/task/executor.ts`](../packages/coding-agent/src/task/executor.ts) — subagent output artifact writes (`.md`) and temp artifact directory fallback. \ No newline at end of file +- [`src/task/executor.ts`](../packages/coding-agent/src/task/executor.ts) — subagent output artifact writes (`.md`) and temp artifact directory fallback. diff --git a/docs/compaction.md b/docs/compaction.md index 6dfdbabc4..430a585a4 100644 --- a/docs/compaction.md +++ b/docs/compaction.md @@ -51,11 +51,12 @@ Those custom roles are then transformed into LLM-facing user messages in `conver ### Triggers -Compaction can run in three ways: +Compaction/context maintenance can run in four ways: -1. **Manual**: `/compact [instructions]` calls `AgentSession.compact(...)`. -2. **Automatic overflow recovery**: after an assistant error that matches context overflow. -3. **Automatic threshold compaction**: after a successful turn when context exceeds threshold. +1. **Manual context compaction**: `/compact [instructions]` calls `AgentSession.compact(...)`. +2. **Automatic overflow recovery**: after a same-model assistant error that matches context overflow. +3. **Automatic threshold maintenance**: after a successful turn when context exceeds the resolved threshold. +4. **Idle maintenance**: `runIdleCompaction()` can invoke the same auto-maintenance path with reason `"idle"`. ### Compaction shape (visual) @@ -91,22 +92,28 @@ What the LLM sees: prompt from cmp messages from firstKeptEntryId ``` +### Overflow-retry vs threshold/idle maintenance -### Overflow-retry vs threshold compaction +The automatic paths are intentionally different: -The two automatic paths are intentionally different: - -- **Overflow-retry compaction** - - Trigger: current-model assistant error is detected as context overflow. +- **Overflow recovery** + - Trigger: current-model assistant error is detected as context overflow and the error is not older than the latest compaction. - The failing assistant error message is removed from active agent state before retry. - - Auto compaction runs with `reason: "overflow"` and `willRetry: true`. + - Context promotion is tried first; if a configured larger model is available, the agent switches model and retries without compacting. + - If promotion is unavailable and compaction is enabled, context-full compaction runs with `reason: "overflow"` and `willRetry: true`; handoff strategy is not used for overflow. - On success, agent auto-continues (`agent.continue()`) after compaction. -- **Threshold compaction** - - Trigger: `contextTokens > contextWindow - compaction.reserveTokens`. - - Runs with `reason: "threshold"` and `willRetry: false`. - - On success, if `compaction.autoContinue !== false`, injects a synthetic prompt: - - `"Continue if you have next steps."` +- **Threshold maintenance** + - Trigger: successful, non-error assistant message whose adjusted context tokens exceed `resolveThresholdTokens(...)`. + - Tool-output pruning can reduce the measured token count before threshold comparison. + - Context promotion is tried before compaction. + - If promotion is unavailable, auto maintenance runs with `reason: "threshold"` and `willRetry: false`. + - With `compaction.strategy: "handoff"`, threshold maintenance starts a new handoff session instead of writing a compaction entry; if handoff returns no document without aborting, it falls back to context-full compaction. + - On success, if `compaction.autoContinue !== false`, schedules an agent-authored developer auto-continue prompt from `prompts/system/auto-continue.md`. + +- **Idle maintenance** + - Trigger: `runIdleCompaction()` when not streaming or already compacting. + - Uses `reason: "idle"` and does not auto-continue afterward. ### Pre-compaction pruning @@ -189,11 +196,12 @@ Prompt selection: - split-turn second pass: `compaction-turn-prefix.md` - short UI summary: `compaction-short-summary.md` -Remote summarization mode: +Remote summarization modes: -- If `compaction.remoteEndpoint` is set, compaction POSTs: +- If `compaction.remoteEndpoint` is set and remote compaction is enabled, local summary generation POSTs: - `{ systemPrompt, prompt }` - Expects JSON containing at least `{ summary }`. +- For OpenAI/OpenAI Codex models, compaction first tries the provider-native `/responses/compact` endpoint when remote compaction is enabled. It preserves provider replacement history in `preserveData.openaiRemoteCompaction` and falls back to local summarization if that native request fails. ### File-operation context in summaries @@ -224,8 +232,8 @@ Summary text gets file tags appended via prompt template: After summary generation (or hook-provided summary), agent session: -1. Appends `CompactionEntry` with `appendCompaction(...)`. -2. Rebuilds context via `buildSessionContext()`. +1. Appends `CompactionEntry` with `appendCompaction(...)` for context-full maintenance; handoff strategy creates a new session and injects a handoff `custom_message` instead. +2. Rebuilds display context from the active leaf via `buildDisplaySessionContext()`. 3. Replaces live agent messages with rebuilt context. 4. Emits `session_compact` hook event. @@ -262,7 +270,6 @@ After navigation with summary: └─ E ─ F (new leaf) ``` - ### Preparation and token budget `generateBranchSummary(...)` computes budget as: @@ -334,8 +341,8 @@ Post-navigation event exposing new/old leaf and optional summary entry. - Manual compaction aborts current agent operation first. - `abortCompaction()` cancels both manual and auto-compaction controllers. - Auto compaction emits start/end session events for UI/state updates. -- Auto compaction can try multiple model candidates and retry transient failures. -- Overflow errors are excluded from generic retry path because they are handled by compaction. +- Auto compaction can try multiple model candidates and retry transient failures; long retry delays prefer the next candidate when one is available. +- Overflow errors are excluded from generic retry path because they are handled by context promotion/compaction. - If auto-compaction fails: - overflow path emits `Context overflow recovery failed: ...` - threshold path emits `Auto-compaction failed: ...` @@ -346,11 +353,15 @@ Post-navigation event exposing new/old leaf and optional summary entry. From `settings-schema.ts`: - `compaction.enabled` = `true` +- `compaction.strategy` = `"context-full"` (`"handoff"` and `"off"` are also supported) - `compaction.reserveTokens` = `16384` - `compaction.keepRecentTokens` = `20000` - `compaction.autoContinue` = `true` +- `compaction.remoteEnabled` = `true` - `compaction.remoteEndpoint` = `undefined` +- `compaction.thresholdPercent` = `-1` and `compaction.thresholdTokens` = `-1`; when no positive override is set, the threshold is `contextWindow - max(15% of contextWindow, reserveTokens)` +- `compaction.idleEnabled` = `true` - `branchSummary.enabled` = `false` - `branchSummary.reserveTokens` = `16384` -These values are consumed at runtime by `AgentSession` and compaction/branch summarization modules. \ No newline at end of file +These values are consumed at runtime by `AgentSession` and compaction/branch summarization modules. diff --git a/docs/config-usage.md b/docs/config-usage.md index e9d505e33..c6b9bd60d 100644 --- a/docs/config-usage.md +++ b/docs/config-usage.md @@ -6,51 +6,45 @@ This document describes how the coding-agent resolves configuration today: which Primary implementation: -- `src/config.ts` -- `src/config/settings.ts` -- `src/config/settings-schema.ts` -- `src/discovery/builtin.ts` -- `src/discovery/helpers.ts` +- `packages/coding-agent/src/config.ts` +- `packages/coding-agent/src/config/settings.ts` +- `packages/coding-agent/src/config/settings-schema.ts` +- `packages/coding-agent/src/discovery/builtin.ts` +- `packages/coding-agent/src/discovery/helpers.ts` Key integration points: -- `src/capability/index.ts` -- `src/discovery/index.ts` -- `src/extensibility/skills.ts` -- `src/extensibility/hooks/loader.ts` -- `src/extensibility/custom-tools/loader.ts` -- `src/extensibility/extensions/loader.ts` +- `packages/coding-agent/src/capability/index.ts` +- `packages/coding-agent/src/discovery/index.ts` +- `packages/coding-agent/src/extensibility/skills.ts` +- `packages/coding-agent/src/extensibility/hooks/loader.ts` +- `packages/coding-agent/src/extensibility/custom-tools/loader.ts` +- `packages/coding-agent/src/extensibility/extensions/loader.ts` --- ## Resolution flow (visual) ```text - Config roots (ordered) + Generic helper order (`config.ts`) ┌───────────────────────────────────────┐ -│ 1) ~/.omp/agent + /.omp │ -│ 2) ~/.claude + /.claude │ -│ 3) ~/.codex + /.codex │ -│ 4) ~/.gemini + /.gemini │ +│ 1) ~/.omp/agent, ~/.claude, ... │ +│ 2) /.omp, /.claude, ... │ └───────────────────────────────────────┘ │ ▼ - config.ts helper resolution - (getConfigDirs/findConfigFile/findNearest...) + capability providers enumerate items + (native provider scans project .omp before user .omp; + other providers have their own loading rules) │ ▼ - capability providers enumerate items - (native, claude, codex, gemini, agents, etc.) - │ - ▼ - priority sort + per-capability dedup + provider priority sort + capability dedup │ ▼ subsystem-specific consumption (settings, skills, hooks, tools, extensions) ``` - ## 1) Config roots and source order ## Canonical roots @@ -212,35 +206,35 @@ Relevant keys: --- -## 6) Native `.omp` provider behavior (`src/discovery/builtin.ts`) +## 6) Native `.omp` provider behavior (`packages/coding-agent/src/discovery/builtin.ts`) -Native provider (`id: native`) reads from: +Native provider (`id: native`) reads native config from: - project: `/.omp/...` - user: `~/.omp/agent/...` -### Directory admission rule +### Directory admission rules -`builtin.ts` only includes a config root if the directory exists **and is non-empty** (`ifNonEmptyDir`). +- Slash commands, rules, prompts, instructions, hooks, tools, extensions, extension modules, and settings use a project/user root only when the root directory exists and is non-empty. +- Skills scan `/.omp/skills` for each ancestor from the current working directory up to the repo root/home boundary, plus `~/.omp/agent/skills`, without requiring the root `.omp` directory itself to be non-empty. +- `SYSTEM.md` and `AGENTS.md` read user-level files directly and use nearest-ancestor project `.omp` lookup for project files, but the project `.omp` directory must be non-empty. ### Scope-specific loading -- Skills: `skills/*/SKILL.md` +- Skills: `/.omp/skills/*/SKILL.md` and `~/.omp/agent/skills/*/SKILL.md` - Slash commands: `commands/*.md` - Rules: `rules/*.{md,mdc}` - Prompts: `prompts/*.md` - Instructions: `instructions/*.md` - Hooks: `hooks/pre/*`, `hooks/post/*` -- Tools: `tools/*.json|*.md` and `tools//index.ts` +- Tools: `tools/*.{json,md,ts,js,sh,bash,py}` and `tools//index.ts` - Extension modules: discovered under `extensions/` (+ legacy `settings.json.extensions` string array) - Extensions: `extensions//gemini-extension.json` - Settings capability: `settings.json` ### Nearest-project lookup nuance -For `SYSTEM.md` and `AGENTS.md`, native provider uses nearest-ancestor project `.omp` directory search (walk-up) but still requires the `.omp` dir to be non-empty. - ---- +## For `SYSTEM.md` and `AGENTS.md`, native provider uses nearest-ancestor project `.omp` directory search (walk-up) and still requires the project `.omp` dir to be non-empty. ## 7) How major subsystems consume config @@ -291,8 +285,7 @@ Settings capability items are not deduplicated; `Settings.#loadProjectSettings() - `ConfigFile` JSON -> YAML migration for YAML-targeted files. - Settings migration from `settings.json` and `agent.db` to `config.yml`. -- Settings key migrations (`queueMode`, `ask.timeout`, flat `theme`). -- Extension manifest compatibility: loader accepts both `package.json.omp` and `package.json.pi` manifest sections. +- Settings key migrations (`queueMode`, `ask.timeout`, flat `theme`, `task.isolation.enabled`, `statusLine.plan_mode`). - Legacy setting names `skills.enablePiUser` / `skills.enablePiProject` are still active gates for native skill source. If these compatibility paths are removed in code, update this document immediately; several runtime behaviors still depend on them today. diff --git a/docs/custom-tools.md b/docs/custom-tools.md index 067c6620f..2c5d1dde7 100644 --- a/docs/custom-tools.md +++ b/docs/custom-tools.md @@ -67,39 +67,45 @@ A custom tool module must export a function (default export preferred): import type { CustomToolFactory } from "@oh-my-pi/pi-coding-agent"; const factory: CustomToolFactory = (pi) => ({ - name: "repo_stats", - label: "Repo Stats", - description: "Counts tracked TypeScript files", - parameters: pi.typebox.Type.Object({ - glob: pi.typebox.Type.Optional(pi.typebox.Type.String({ default: "**/*.ts" })), - }), + name: "repo_stats", + label: "Repo Stats", + description: "Counts tracked TypeScript files", + parameters: pi.typebox.Type.Object({ + glob: pi.typebox.Type.Optional( + pi.typebox.Type.String({ default: "**/*.ts" }), + ), + }), - async execute(toolCallId, params, onUpdate, ctx, signal) { - onUpdate?.({ - content: [{ type: "text", text: "Scanning files..." }], - details: { phase: "scan" }, - }); + async execute(toolCallId, params, onUpdate, ctx, signal) { + onUpdate?.({ + content: [{ type: "text", text: "Scanning files..." }], + details: { phase: "scan" }, + }); - const result = await pi.exec("git", ["ls-files", params.glob ?? "**/*.ts"], { signal, cwd: pi.cwd }); - if (result.killed) { - throw new Error("Scan was cancelled"); - } - if (result.code !== 0) { - throw new Error(result.stderr || "git ls-files failed"); - } + const result = await pi.exec( + "git", + ["ls-files", params.glob ?? "**/*.ts"], + { signal, cwd: pi.cwd }, + ); + if (result.killed) { + throw new Error("Scan was cancelled"); + } + if (result.code !== 0) { + throw new Error(result.stderr || "git ls-files failed"); + } - const files = result.stdout.split("\n").filter(Boolean); - return { - content: [{ type: "text", text: `Found ${files.length} files` }], - details: { count: files.length, sample: files.slice(0, 10) }, - }; - }, + const files = result.stdout.split("\n").filter(Boolean); + return { + content: [{ type: "text", text: `Found ${files.length} files` }], + details: { count: files.length, sample: files.slice(0, 10) }, + }; + }, - onSession(event) { - if (event.reason === "shutdown") { - // cleanup resources if needed - } - }, + onSession(event) { + if (event.reason === "shutdown") { + // cleanup resources if needed + } + }, }); export default factory; @@ -131,7 +137,7 @@ Loader starts with a no-op UI context and requires host code to call `setUIConte `CustomTool.execute` signature: ```ts -execute(toolCallId, params, onUpdate, ctx, signal) +execute(toolCallId, params, onUpdate, ctx, signal); ``` - `params` is statically typed from your TypeBox schema via `Static`. @@ -153,7 +159,7 @@ execute(toolCallId, params, onUpdate, ctx, signal) Optional rendering hooks: -- `renderCall(args, theme)` +- `renderCall(args, options, theme)` - `renderResult(result, options, theme, args?)` Runtime behavior in TUI: diff --git a/docs/environment-variables.md b/docs/environment-variables.md index aca80437b..d022ccaa8 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -16,9 +16,11 @@ Most runtime lookups use `$env` from `@oh-my-pi/pi-utils` (`packages/utils/src/e 1. Existing process environment (`Bun.env`) 2. Project `.env` (`$PWD/.env`) for keys not already set -3. Home `.env` (`~/.env`) for keys not already set +3. Agent `.env` (`~/.omp/agent/.env`, respecting `PI_CONFIG_DIR` / `PI_CODING_AGENT_DIR`) for keys not already set +4. Config-root `.env` (`~/.omp/.env`, respecting `PI_CONFIG_DIR`) for keys not already set +5. Home `.env` (`~/.env`) for keys not already set -Additional rule in `.env` files: `OMP_*` keys are mirrored to `PI_*` keys during parse. +Additional rule inside each `.env` file: `OMP_*` keys are mirrored to `PI_*` keys in that parsed file. --- @@ -28,53 +30,59 @@ These are consumed via `getEnvApiKey()` (`packages/ai/src/stream.ts`) unless not ### Core provider credentials -| Variable | Used for | Required when | Notes / precedence | -|---------------------------------|---|---------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| -| `ANTHROPIC_OAUTH_TOKEN` | Anthropic API auth | Using Anthropic with OAuth token auth | Takes precedence over `ANTHROPIC_API_KEY` for provider auth resolution | -| `ANTHROPIC_API_KEY` | Anthropic API auth | Using Anthropic without OAuth token | Fallback after `ANTHROPIC_OAUTH_TOKEN` | -| `ANTHROPIC_FOUNDRY_API_KEY` | Anthropic via Azure Foundry / enterprise gateway | `CLAUDE_CODE_USE_FOUNDRY` enabled | Takes precedence over `ANTHROPIC_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` when Foundry mode is enabled | -| `OPENAI_API_KEY` | OpenAI auth | Using OpenAI-family providers without explicit apiKey argument | Used by OpenAI Completions/Responses providers | -| `GEMINI_API_KEY` | Google Gemini auth | Using `google` provider models | Primary key for Gemini provider mapping | -| `GOOGLE_API_KEY` | Gemini image tool auth fallback | Using `gemini_image` tool without `GEMINI_API_KEY` | Used by coding-agent image tool fallback path | -| `GROQ_API_KEY` | Groq auth | Using Groq models | | -| `CEREBRAS_API_KEY` | Cerebras auth | Using Cerebras models | | -| `TOGETHER_API_KEY` | Together auth | Using `together` provider | | -| `HUGGINGFACE_HUB_TOKEN` | Hugging Face auth | Using `huggingface` provider | Primary Hugging Face token env var | -| `HF_TOKEN` | Hugging Face auth | Using `huggingface` provider | Fallback when `HUGGINGFACE_HUB_TOKEN` is unset | -| `SYNTHETIC_API_KEY` | Synthetic auth | Using Synthetic models | | -| `NVIDIA_API_KEY` | NVIDIA auth | Using `nvidia` provider | | -| `NANO_GPT_API_KEY` | NanoGPT auth | Using `nanogpt` provider | | -| `VENICE_API_KEY` | Venice auth | Using `venice` provider | | -| `LITELLM_API_KEY` | LiteLLM auth | Using `litellm` provider | OpenAI-compatible LiteLLM proxy key | -| `LM_STUDIO_API_KEY` | LM Studio auth (optional) | Using `lm-studio` provider with authenticated hosts | Local LM Studio usually runs without auth; any non-empty token works when a key is required | -| `OLLAMA_API_KEY` | Ollama auth (optional) | Using `ollama` provider with authenticated hosts | Local Ollama usually runs without auth; any non-empty token works when a key is required | -| `LLAMA_CPP_API_KEY` | Ollama auth (optional) | Using `llama-server` with `--api-key` parameter | Local llama.cpp usually runs without auth; any non-empty token works when a key is configured | -| `XIAOMI_API_KEY` | Xiaomi MiMo auth | Using `xiaomi` provider | | -| `MOONSHOT_API_KEY` | Moonshot auth | Using `moonshot` provider | | -| `XAI_API_KEY` | xAI auth | Using xAI models | | -| `OPENROUTER_API_KEY` | OpenRouter auth | Using OpenRouter models | Also used by image tool when preferred/auto provider is OpenRouter | -| `MISTRAL_API_KEY` | Mistral auth | Using Mistral models | | -| `ZAI_API_KEY` | z.ai auth | Using z.ai models | Also used by z.ai web search provider | -| `MINIMAX_API_KEY` | MiniMax auth | Using `minimax` provider | | -| `MINIMAX_CODE_API_KEY` | MiniMax Code auth | Using `minimax-code` provider | | -| `MINIMAX_CODE_CN_API_KEY` | MiniMax Code CN auth | Using `minimax-code-cn` provider | | -| `OPENCODE_API_KEY` | OpenCode auth | Using OpenCode models | | -| `QIANFAN_API_KEY` | Qianfan auth | Using `qianfan` provider | | -| `QWEN_OAUTH_TOKEN` | Qwen Portal auth | Using `qwen-portal` with OAuth token | Takes precedence over `QWEN_PORTAL_API_KEY` | -| `QWEN_PORTAL_API_KEY` | Qwen Portal auth | Using `qwen-portal` with API key | Fallback after `QWEN_OAUTH_TOKEN` | -| `ZENMUX_API_KEY` | ZenMux auth | Using `zenmux` provider | Used for ZenMux OpenAI and Anthropic-compatible routes | -| `VLLM_API_KEY` | vLLM auth/discovery opt-in | Using `vllm` provider (local OpenAI-compatible servers) | Any non-empty value works for no-auth local servers | -| `CURSOR_ACCESS_TOKEN` | Cursor provider auth | Using Cursor provider | | -| `AI_GATEWAY_API_KEY` | Vercel AI Gateway auth | Using `vercel-ai-gateway` provider | | -| `CLOUDFLARE_AI_GATEWAY_API_KEY` | Cloudflare AI Gateway auth | Using `cloudflare-ai-gateway` provider | Base URL must be configured as `https://gateway.ai.cloudflare.com/v1///anthropic` | +| Variable | Used for | Required when | Notes / precedence | +| ------------------------------- | ------------------------------------------------ | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| `ANTHROPIC_OAUTH_TOKEN` | Anthropic API auth | Using Anthropic with OAuth token auth | Takes precedence over `ANTHROPIC_API_KEY` for provider auth resolution | +| `ANTHROPIC_API_KEY` | Anthropic API auth | Using Anthropic without OAuth token | Fallback after `ANTHROPIC_OAUTH_TOKEN` | +| `ANTHROPIC_FOUNDRY_API_KEY` | Anthropic via Azure Foundry / enterprise gateway | `CLAUDE_CODE_USE_FOUNDRY` enabled | Takes precedence over `ANTHROPIC_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` when Foundry mode is enabled | +| `OPENAI_API_KEY` | OpenAI auth | Using OpenAI-family providers without explicit apiKey argument | Used by OpenAI Completions/Responses providers | +| `GEMINI_API_KEY` | Google Gemini auth | Using `google` provider models | Primary key for Gemini provider mapping | +| `GOOGLE_API_KEY` | Gemini image tool auth fallback | Using `gemini_image` tool without `GEMINI_API_KEY` | Used by coding-agent image tool fallback path | +| `GROQ_API_KEY` | Groq auth | Using Groq models | | +| `CEREBRAS_API_KEY` | Cerebras auth | Using Cerebras models | | +| `FIREWORKS_API_KEY` | Fireworks auth | Using Fireworks models | | +| `TOGETHER_API_KEY` | Together auth | Using `together` provider | | +| `HUGGINGFACE_HUB_TOKEN` | Hugging Face auth | Using `huggingface` provider | Primary Hugging Face token env var | +| `HF_TOKEN` | Hugging Face auth | Using `huggingface` provider | Fallback when `HUGGINGFACE_HUB_TOKEN` is unset | +| `SYNTHETIC_API_KEY` | Synthetic auth | Using Synthetic models | | +| `NVIDIA_API_KEY` | NVIDIA auth | Using `nvidia` provider | | +| `NANO_GPT_API_KEY` | NanoGPT auth | Using `nanogpt` provider | | +| `VENICE_API_KEY` | Venice auth | Using `venice` provider | | +| `LITELLM_API_KEY` | LiteLLM auth | Using `litellm` provider | OpenAI-compatible LiteLLM proxy key | +| `LM_STUDIO_API_KEY` | LM Studio auth (optional) | Using `lm-studio` provider with authenticated hosts | Local LM Studio usually runs without auth; any non-empty token works when a key is required | +| `OLLAMA_API_KEY` | Ollama auth (optional) | Using `ollama` provider with authenticated hosts | Local Ollama usually runs without auth; any non-empty token works when a key is required | +| `LLAMA_CPP_API_KEY` | llama.cpp auth (optional) | Using `llama.cpp` provider with authenticated hosts | Local llama.cpp usually runs without auth; any non-empty token works when a key is configured | +| `XIAOMI_API_KEY` | Xiaomi MiMo auth | Using `xiaomi` provider | | +| `MOONSHOT_API_KEY` | Moonshot auth | Using `moonshot` provider | | +| `XAI_API_KEY` | xAI auth | Using xAI models | | +| `OPENROUTER_API_KEY` | OpenRouter auth | Using OpenRouter models | Also used by image tool when preferred/auto provider is OpenRouter | +| `MISTRAL_API_KEY` | Mistral auth | Using Mistral models | | +| `ZAI_API_KEY` | z.ai auth | Using z.ai models | Also used by z.ai web search provider | +| `MINIMAX_API_KEY` | MiniMax auth | Using `minimax` provider | | +| `MINIMAX_CODE_API_KEY` | MiniMax Code auth | Using `minimax-code` provider | | +| `MINIMAX_CODE_CN_API_KEY` | MiniMax Code CN auth | Using `minimax-code-cn` provider | | +| `OPENCODE_API_KEY` | OpenCode auth | Using `opencode-go` / `opencode-zen` models | | +| `QIANFAN_API_KEY` | Qianfan auth | Using `qianfan` provider | | +| `QWEN_OAUTH_TOKEN` | Qwen Portal auth | Using `qwen-portal` with OAuth token | Takes precedence over `QWEN_PORTAL_API_KEY` | +| `QWEN_PORTAL_API_KEY` | Qwen Portal auth | Using `qwen-portal` with API key | Fallback after `QWEN_OAUTH_TOKEN` | +| `ZENMUX_API_KEY` | ZenMux auth | Using `zenmux` provider | Used for ZenMux OpenAI and Anthropic-compatible routes | +| `VLLM_API_KEY` | vLLM auth/discovery opt-in | Using `vllm` provider (local OpenAI-compatible servers) | Any non-empty value works for no-auth local servers | +| `CURSOR_ACCESS_TOKEN` | Cursor provider auth | Using Cursor provider | | +| `AI_GATEWAY_API_KEY` | Vercel AI Gateway auth | Using `vercel-ai-gateway` provider | | +| `CLOUDFLARE_AI_GATEWAY_API_KEY` | Cloudflare AI Gateway auth | Using `cloudflare-ai-gateway` provider | Base URL must be configured as `https://gateway.ai.cloudflare.com/v1///anthropic` | +| `ALIBABA_CODING_PLAN_API_KEY` | Alibaba Coding Plan auth | Using `alibaba-coding-plan` provider | | +| `DEEPSEEK_API_KEY` | DeepSeek auth | Using DeepSeek models | | +| `KILO_API_KEY` | Kilo auth | Using Kilo models | | +| `OLLAMA_CLOUD_API_KEY` | Ollama Cloud auth | Using `ollama-cloud` provider | | +| `GITLAB_TOKEN` | GitLab Duo auth | Using `gitlab-duo` provider | | ### GitHub/Copilot token chains -| Variable | Used for | Chain | -|---|---|---| -| `COPILOT_GITHUB_TOKEN` | GitHub Copilot provider auth | `COPILOT_GITHUB_TOKEN` → `GH_TOKEN` → `GITHUB_TOKEN` | -| `GH_TOKEN` | Copilot fallback; GitHub API auth in web scraper | In web scraper: `GITHUB_TOKEN` → `GH_TOKEN` | -| `GITHUB_TOKEN` | Copilot fallback; GitHub API auth in web scraper | In web scraper: checked before `GH_TOKEN` | +| Variable | Used for | Chain | +| ---------------------- | ------------------------------------------------ | ---------------------------------------------------- | +| `COPILOT_GITHUB_TOKEN` | GitHub Copilot provider auth | `COPILOT_GITHUB_TOKEN` → `GH_TOKEN` → `GITHUB_TOKEN` | +| `GH_TOKEN` | Copilot fallback; GitHub API auth in web scraper | In web scraper: `GITHUB_TOKEN` → `GH_TOKEN` | +| `GITHUB_TOKEN` | Copilot fallback; GitHub API auth in web scraper | In web scraper: checked before `GH_TOKEN` | --- @@ -94,93 +102,96 @@ When `CLAUDE_CODE_USE_FOUNDRY` is enabled, Anthropic requests switch to Foundry - a filesystem path to PEM content, or - inline PEM (including escaped `\n` sequences). -| Variable | Value type | Behavior | -|---|---|---| -| `CLAUDE_CODE_USE_FOUNDRY` | Boolean-like string (`1`, `true`, `yes`, `on`) | Enables Foundry mode for Anthropic provider | -| `FOUNDRY_BASE_URL` | URL string | Anthropic endpoint base URL in Foundry mode | -| `ANTHROPIC_FOUNDRY_API_KEY` | Token string | Used for `Authorization: Bearer ` | -| `ANTHROPIC_CUSTOM_HEADERS` | Header list string | Extra headers; format `header-a: value, header-b: value` or newline-separated | -| `NODE_EXTRA_CA_CERTS` | PEM path or inline PEM | Extra CA chain for server certificate validation | -| `CLAUDE_CODE_CLIENT_CERT` | PEM path or inline PEM | mTLS client certificate | -| `CLAUDE_CODE_CLIENT_KEY` | PEM path or inline PEM | mTLS client private key (must be paired with cert) | +| Variable | Value type | Behavior | +| --------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------- | +| `CLAUDE_CODE_USE_FOUNDRY` | Boolean-like string (`1`, `true`, `yes`, `on`) | Enables Foundry mode for Anthropic provider | +| `FOUNDRY_BASE_URL` | URL string | Anthropic endpoint base URL in Foundry mode | +| `ANTHROPIC_FOUNDRY_API_KEY` | Token string | Used for `Authorization: Bearer ` | +| `ANTHROPIC_CUSTOM_HEADERS` | Header list string | Extra headers; format `header-a: value, header-b: value` or newline-separated | +| `NODE_EXTRA_CA_CERTS` | PEM path or inline PEM | Extra CA chain for server certificate validation | +| `CLAUDE_CODE_CLIENT_CERT` | PEM path or inline PEM | mTLS client certificate | +| `CLAUDE_CODE_CLIENT_KEY` | PEM path or inline PEM | mTLS client private key (must be paired with cert) | ### Amazon Bedrock -| Variable | Default / behavior | -|---|---| -| `AWS_REGION` | Primary region source | -| `AWS_DEFAULT_REGION` | Fallback if `AWS_REGION` unset | -| `AWS_PROFILE` | Enables named profile auth path | -| `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` | Enables IAM key auth path | -| `AWS_BEARER_TOKEN_BEDROCK` | Enables bearer token auth path | -| `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` / `AWS_CONTAINER_CREDENTIALS_FULL_URI` | Enables ECS task credential path | -| `AWS_WEB_IDENTITY_TOKEN_FILE` + `AWS_ROLE_ARN` | Enables web identity auth path | -| `AWS_BEDROCK_SKIP_AUTH` | If `1`, injects dummy credentials (proxy/non-auth scenarios) | -| `AWS_BEDROCK_FORCE_HTTP1` | If `1`, forces Node HTTP/1 request handler | -| `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` | Routes Bedrock runtime and AWS SSO credential calls through the configured proxy using HTTP/1 | -| `NO_PROXY` | Excludes matching hosts from proxy routing when a proxy variable is configured | +| Variable | Default / behavior | +| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `AWS_REGION` | Primary region source | +| `AWS_DEFAULT_REGION` | Fallback if `AWS_REGION` unset | +| `AWS_PROFILE` | Enables named profile auth path | +| `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` | Enables IAM key auth path | +| `AWS_BEARER_TOKEN_BEDROCK` | Enables bearer token auth path | +| `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` / `AWS_CONTAINER_CREDENTIALS_FULL_URI` | Enables ECS task credential path | +| `AWS_WEB_IDENTITY_TOKEN_FILE` + `AWS_ROLE_ARN` | Enables web identity auth path | +| `AWS_BEDROCK_SKIP_AUTH` | If `1`, injects dummy credentials (proxy/non-auth scenarios) | +| `AWS_BEDROCK_FORCE_HTTP1` | If `1`, forces Node HTTP/1 request handler | +| `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` | Routes Bedrock runtime and AWS SSO credential calls through the configured proxy using HTTP/1 | +| `NO_PROXY` | Excludes matching hosts from proxy routing when a proxy variable is configured | Region fallback in provider code: `options.region` → `AWS_REGION` → `AWS_DEFAULT_REGION` → `us-east-1`. ### Azure OpenAI Responses -| Variable | Default / behavior | -|---|---| -| `AZURE_OPENAI_API_KEY` | Required unless API key passed as option | -| `AZURE_OPENAI_API_VERSION` | Default `v1` | -| `AZURE_OPENAI_BASE_URL` | Direct base URL override | -| `AZURE_OPENAI_RESOURCE_NAME` | Used to construct base URL: `https://.openai.azure.com/openai/v1` | -| `AZURE_OPENAI_DEPLOYMENT_NAME_MAP` | Optional mapping string: `modelId=deploymentName,model2=deployment2` | +| Variable | Default / behavior | +| ---------------------------------- | --------------------------------------------------------------------------- | +| `AZURE_OPENAI_API_KEY` | Required unless API key passed as option | +| `AZURE_OPENAI_API_VERSION` | Default `v1` | +| `AZURE_OPENAI_BASE_URL` | Direct base URL override | +| `AZURE_OPENAI_RESOURCE_NAME` | Used to construct base URL: `https://.openai.azure.com/openai/v1` | +| `AZURE_OPENAI_DEPLOYMENT_NAME_MAP` | Optional mapping string: `modelId=deploymentName,model2=deployment2` | Base URL resolution: option `azureBaseUrl` → env `AZURE_OPENAI_BASE_URL` → option/env resource name → `model.baseUrl`. ### Google Vertex AI -| Variable | Required? | Notes | -|---|---|---| -| `GOOGLE_CLOUD_PROJECT` | Yes (unless passed in options) | Fallback: `GCLOUD_PROJECT` | -| `GCLOUD_PROJECT` | Fallback | Used as alternate project ID source | -| `GOOGLE_CLOUD_LOCATION` | Yes (unless passed in options) | No default in provider | -| `GOOGLE_APPLICATION_CREDENTIALS` | Conditional | If set, file must exist; otherwise ADC fallback path is checked (`~/.config/gcloud/application_default_credentials.json`) | +| Variable | Required? | Notes | +| -------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | +| `GOOGLE_CLOUD_PROJECT` | Yes (unless passed in options) | Fallback: `GCLOUD_PROJECT` | +| `GCLOUD_PROJECT` | Fallback | Used as alternate project ID source | +| `GOOGLE_CLOUD_PROJECT_ID` | OAuth login helper only | Used by Gemini CLI OAuth project discovery | +| `GOOGLE_CLOUD_LOCATION` | Yes (unless passed in options) | No default in provider | +| `GOOGLE_CLOUD_API_KEY` | Conditional | Direct Vertex API-key auth; otherwise ADC fallback can authenticate when project and location are set | +| `GOOGLE_APPLICATION_CREDENTIALS` | Conditional | If set, file must exist; otherwise ADC fallback path is checked (`~/.config/gcloud/application_default_credentials.json`) | ### Kimi -| Variable | Default / behavior | -|---|---| -| `KIMI_CODE_OAUTH_HOST` | Primary OAuth host override | -| `KIMI_OAUTH_HOST` | Fallback OAuth host override | -| `KIMI_CODE_BASE_URL` | Overrides Kimi usage endpoint base URL (`usage/kimi.ts`) | +| Variable | Default / behavior | +| ---------------------- | -------------------------------------------------------- | +| `KIMI_CODE_OAUTH_HOST` | Primary OAuth host override | +| `KIMI_OAUTH_HOST` | Fallback OAuth host override | +| `KIMI_CODE_BASE_URL` | Overrides Kimi usage endpoint base URL (`usage/kimi.ts`) | OAuth host chain: `KIMI_CODE_OAUTH_HOST` → `KIMI_OAUTH_HOST` → `https://auth.kimi.com`. -### Antigravity/Gemini image compatibility +### Gemini CLI compatibility -| Variable | Default / behavior | -|---|---| -| `PI_AI_ANTIGRAVITY_VERSION` | Overrides Antigravity user-agent version tag in Gemini CLI provider | +| Variable | Default / behavior | +| -------------------------- | --------------------------------------------------------------- | +| `PI_AI_GEMINI_CLI_VERSION` | Overrides Gemini CLI user-agent version tag (`0.35.3` if unset) | ### OpenAI Codex responses (feature/debug controls) -| Variable | Behavior | -|---|---| -| `PI_CODEX_DEBUG` | `1`/`true` enables Codex provider debug logging | -| `PI_CODEX_WEBSOCKET` | `1`/`true` enables websocket transport preference | -| `PI_CODEX_WEBSOCKET_V2` | `1`/`true` enables websocket v2 path | -| `PI_CODEX_WEBSOCKET_IDLE_TIMEOUT_MS` | Positive integer override (default 300000) | -| `PI_CODEX_WEBSOCKET_RETRY_BUDGET` | Non-negative integer override (default 5) | -| `PI_CODEX_WEBSOCKET_RETRY_DELAY_MS` | Positive integer base backoff override (default 500) | +| Variable | Behavior | +| ------------------------------------ | ---------------------------------------------------- | +| `PI_CODEX_DEBUG` | `1`/`true` enables Codex provider debug logging | +| `PI_CODEX_WEBSOCKET` | `1`/`true` enables websocket transport preference | +| `PI_CODEX_WEBSOCKET_V2` | `1`/`true` enables websocket v2 path | +| `PI_CODEX_WEBSOCKET_IDLE_TIMEOUT_MS` | Positive integer override (default 300000) | +| `PI_CODEX_WEBSOCKET_RETRY_BUDGET` | Non-negative integer override (default 5) | +| `PI_CODEX_WEBSOCKET_RETRY_DELAY_MS` | Positive integer base backoff override (default 500) | +| `PI_OPENAI_STREAM_IDLE_TIMEOUT_MS` | Positive integer OpenAI stream idle timeout override | ### Cursor provider debug -| Variable | Behavior | -|---|---| -| `DEBUG_CURSOR` | Enables provider debug logs; `2`/`verbose` for detailed payload snippets | -| `DEBUG_CURSOR_LOG` | Optional file path for JSONL debug log output | +| Variable | Behavior | +| ------------------ | ------------------------------------------------------------------------ | +| `DEBUG_CURSOR` | Enables provider debug logs; `2`/`verbose` for detailed payload snippets | +| `DEBUG_CURSOR_LOG` | Optional file path for JSONL debug log output | ### Prompt cache compatibility switch -| Variable | Behavior | -|---|---| +| Variable | Behavior | +| -------------------- | ----------------------------------------------------------------------------------------------------------------- | | `PI_CACHE_RETENTION` | If `long`, enables long retention where supported (`anthropic`, `openai-responses`, Bedrock retention resolution) | --- @@ -189,51 +200,60 @@ OAuth host chain: `KIMI_CODE_OAUTH_HOST` → `KIMI_OAUTH_HOST` → `https://auth ### Search provider credentials -| Variable | Used by | -|---|---| -| `EXA_API_KEY` | Exa search provider and Exa MCP tools | -| `BRAVE_API_KEY` | Brave search provider | -| `PERPLEXITY_API_KEY` | Perplexity search provider API-key mode | -| `TAVILY_API_KEY` | Tavily search provider | -| `ZAI_API_KEY` | z.ai search provider (also checks stored OAuth in `agent.db`) | -| `OPENAI_API_KEY` / Codex OAuth in DB | Codex search provider availability/auth | +| Variable | Used by | +| --------------------------------------------------- | ------------------------------------------------------------- | +| `EXA_API_KEY` | Exa search provider and Exa MCP tools | +| `BRAVE_API_KEY` | Brave search provider | +| `PERPLEXITY_API_KEY` | Perplexity search provider API-key mode | +| `PERPLEXITY_COOKIES` | Perplexity cookie-auth search mode | +| `TAVILY_API_KEY` | Tavily search provider | +| `ZAI_API_KEY` | z.ai search provider (also checks stored OAuth in `agent.db`) | +| `OPENAI_API_KEY` / Codex OAuth in DB | Codex search provider availability/auth | +| `PI_CODEX_WEB_SEARCH_MODEL` | Codex search provider model override | +| `MOONSHOT_SEARCH_API_KEY` / `KIMI_SEARCH_API_KEY` | Kimi/Moonshot search provider env auth | +| `MOONSHOT_SEARCH_BASE_URL` / `KIMI_SEARCH_BASE_URL` | Kimi/Moonshot search endpoint override | +| `KAGI_API_KEY` | Kagi search provider | +| `JINA_API_KEY` | Jina search provider | +| `PARALLEL_API_KEY` | Parallel search provider | +| `SEARXNG_ENDPOINT`, `SEARXNG_TOKEN` | SearXNG endpoint and bearer token | ### Anthropic web search auth chain -`packages/coding-agent/src/web/search/auth.ts` resolves Anthropic web-search credentials in this order: +Anthropic web search uses `findAnthropicAuth()` from `packages/ai/src/utils/anthropic-auth.ts` in this order: 1. `ANTHROPIC_SEARCH_API_KEY` (+ optional `ANTHROPIC_SEARCH_BASE_URL`) -2. `models.json` provider entry with `api: "anthropic-messages"` +2. `ANTHROPIC_FOUNDRY_API_KEY` when `CLAUDE_CODE_USE_FOUNDRY` is enabled 3. Anthropic OAuth credentials from `agent.db` (must not expire within 5-minute buffer) -4. Generic Anthropic env fallback: provider key (`ANTHROPIC_FOUNDRY_API_KEY`/`ANTHROPIC_OAUTH_TOKEN`/`ANTHROPIC_API_KEY`) + optional `ANTHROPIC_BASE_URL` (`FOUNDRY_BASE_URL` when Foundry mode is enabled) +4. Anthropic API-key credentials from `agent.db` +5. Generic Anthropic env fallback: provider key (`ANTHROPIC_FOUNDRY_API_KEY` in Foundry mode, otherwise `ANTHROPIC_OAUTH_TOKEN`/`ANTHROPIC_API_KEY`) + optional `ANTHROPIC_BASE_URL` (`FOUNDRY_BASE_URL` when Foundry mode is enabled) Related vars: -| Variable | Default / behavior | -|---|---| -| `ANTHROPIC_SEARCH_API_KEY` | Highest-priority explicit search key | +| Variable | Default / behavior | +| --------------------------- | ---------------------------------------------------- | +| `ANTHROPIC_SEARCH_API_KEY` | Highest-priority explicit search key | | `ANTHROPIC_SEARCH_BASE_URL` | Defaults to `https://api.anthropic.com` when omitted | -| `ANTHROPIC_SEARCH_MODEL` | Defaults to `claude-haiku-4-5` | -| `ANTHROPIC_BASE_URL` | Generic fallback base URL for tier-4 auth path | +| `ANTHROPIC_SEARCH_MODEL` | Defaults to `claude-haiku-4-5` | +| `ANTHROPIC_BASE_URL` | Generic fallback base URL for tier-4 auth path | ### Perplexity OAuth flow behavior flag -| Variable | Behavior | -|---|---| +| Variable | Behavior | +| ------------------- | ------------------------------------------------------------------------------- | | `PI_AUTH_NO_BORROW` | If set, disables macOS native-app token borrowing path in Perplexity login flow | --- ## 4) Python tooling and kernel runtime -| Variable | Default / behavior | -|---|---| -| `PI_PY` | Python tool mode override: `0`/`bash`=`bash-only`, `1`/`py`=`ipy-only`, `mix`/`both`=`both`; invalid values ignored | -| `PI_PYTHON_SKIP_CHECK` | If `1`, skips Python kernel availability checks/warm checks | -| `PI_PYTHON_GATEWAY_URL` | If set, uses external kernel gateway instead of local shared gateway | -| `PI_PYTHON_GATEWAY_TOKEN` | Optional auth token for external gateway (`Authorization: token `) | -| `PI_PYTHON_IPC_TRACE` | If `1`, enables low-level IPC trace path in kernel module | -| `VIRTUAL_ENV` | Highest-priority venv path for Python runtime resolution | +| Variable | Default / behavior | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `PI_PY` | Python tool mode override: `0`/`bash`=`bash-only`, `1`/`py`=`ipy-only`, `mix`/`both`=`both`; invalid values ignored | +| `PI_PYTHON_SKIP_CHECK` | If `1`, skips Python kernel availability checks/warm checks | +| `PI_PYTHON_GATEWAY_URL` | If set, uses external kernel gateway instead of local shared gateway | +| `PI_PYTHON_GATEWAY_TOKEN` | Optional auth token for external gateway (`Authorization: token `) | +| `PI_PYTHON_IPC_TRACE` | If `1`, enables low-level IPC trace path in kernel module | +| `VIRTUAL_ENV` | Highest-priority venv path for Python runtime resolution | Extra conditional behavior: @@ -244,26 +264,33 @@ Extra conditional behavior: ## 5) Agent/runtime behavior toggles -| Variable | Default / behavior | -|----------------------------|----------------------------------------------------------------------------------------------| -| `PI_SMOL_MODEL` | Ephemeral model-role override for `smol` (CLI `--smol` takes precedence) | -| `PI_SLOW_MODEL` | Ephemeral model-role override for `slow` (CLI `--slow` takes precedence) | -| `PI_PLAN_MODEL` | Ephemeral model-role override for `plan` (CLI `--plan` takes precedence) | -| `PI_NO_TITLE` | If set (any non-empty value), disables auto session title generation on first user message | -| `NULL_PROMPT` | If `true`, system prompt builder returns empty string | -| `PI_BLOCKED_AGENT` | Blocks a specific subagent type in task tool | -| `PI_SUBPROCESS_CMD` | Overrides subagent spawn command (`omp` / `omp.cmd` resolution bypass) | -| `PI_TASK_MAX_OUTPUT_BYTES` | Max captured output bytes per subagent (default `500000`) | -| `PI_TASK_MAX_OUTPUT_LINES` | Max captured output lines per subagent (default `5000`) | -| `PI_TIMING` | If `1`, enables startup/tool timing instrumentation logs | -| `PI_DEBUG_STARTUP` | Enables startup stage debug prints to stderr in multiple startup paths | -| `PI_PACKAGE_DIR` | Overrides package asset base dir resolution (docs/examples/changelog path lookup) | -| `PI_DISABLE_LSPMUX` | If `1`, disables lspmux detection/integration and forces direct LSP server spawning | -| `LM_STUDIO_BASE_URL` | Default implicit LM Studio discovery base URL override (`http://127.0.0.1:1234/v1` if unset) | -| `OLLAMA_BASE_URL` | Default implicit Ollama discovery base URL override (`http://127.0.0.1:11434` if unset) | -| `LLAMA_CPP_BASE_URL` | Default implicit Llama.cpp discovery base URL override (`http://127.0.0.1:8080` if unset) | -| `PI_EDIT_VARIANT` | If `hashline`, forces hashline read/grep display mode when edit tool available | -| `PI_NO_PTY` | If `1`, disables interactive PTY path for bash tool | +| Variable | Default / behavior | +| ---------------------------- | -------------------------------------------------------------------------------------------------- | +| `PI_SMOL_MODEL` | Ephemeral model-role override for `smol` (CLI `--smol` takes precedence) | +| `PI_SLOW_MODEL` | Ephemeral model-role override for `slow` (CLI `--slow` takes precedence) | +| `PI_PLAN_MODEL` | Ephemeral model-role override for `plan` (CLI `--plan` takes precedence) | +| `PI_NO_TITLE` | If set (any non-empty value), disables auto session title generation on first user message | +| `NULL_PROMPT` | If `true`, system prompt builder returns empty string | +| `PI_BLOCKED_AGENT` | Blocks a specific subagent type in task tool | +| `PI_SUBPROCESS_CMD` | Overrides subagent spawn command (`omp` / `omp.cmd` resolution bypass) | +| `PI_TASK_MAX_OUTPUT_BYTES` | Max captured output bytes per subagent (default `500000`) | +| `PI_TASK_MAX_OUTPUT_LINES` | Max captured output lines per subagent (default `5000`) | +| `PI_TIMING` | If `1`, enables startup/tool timing instrumentation logs | +| `PI_DEBUG_STARTUP` | Enables startup stage debug prints to stderr in multiple startup paths | +| `PI_PACKAGE_DIR` | Overrides package asset base dir resolution (docs/examples/changelog path lookup) | +| `PI_DISABLE_LSPMUX` | If `1`, disables lspmux detection/integration and forces direct LSP server spawning | +| `PI_RPC_EMIT_TITLE` | Boolean-like flag enabling title events in RPC mode | +| `SMITHERY_URL` | Smithery web URL override (default `https://smithery.ai`) | +| `SMITHERY_API_URL` | Smithery API base URL override (default `https://api.smithery.ai`) | +| `PUPPETEER_EXECUTABLE_PATH` | Browser tool Chromium executable override | +| `LM_STUDIO_BASE_URL` | Default implicit LM Studio discovery base URL override (`http://127.0.0.1:1234/v1` if unset) | +| `OLLAMA_BASE_URL` | Default implicit Ollama discovery base URL override (`http://127.0.0.1:11434` if unset) | +| `LLAMA_CPP_BASE_URL` | Default implicit Llama.cpp discovery base URL override (`http://127.0.0.1:8080` if unset) | +| `PI_EDIT_VARIANT` | Forces edit tool variant when valid (`patch`, `replace`, `hashline`, `atom`, `vim`, `apply_patch`) | +| `PI_STRICT_EDIT_MODE` | If truthy, disables automatic edit-mode fallback | +| `PI_FORCE_IMAGE_PROTOCOL` | Forces supported image protocol (`kitty`, `iterm2`/`iterm`, `sixel`, `none`) where used | +| `PI_ALLOW_SIXEL_PASSTHROUGH` | Allows SIXEL passthrough when `PI_FORCE_IMAGE_PROTOCOL=sixel` | +| `PI_NO_PTY` | If `1`, disables interactive PTY path for bash tool | `PI_NO_PTY` is also set internally when CLI `--no-pty` is used. @@ -273,11 +300,11 @@ Extra conditional behavior: These are consumed via `@oh-my-pi/pi-utils/dirs` and affect where coding-agent stores data. -| Variable | Default / behavior | -|---|---| -| `PI_CONFIG_DIR` | Config root dirname under home (default `.omp`) | +| Variable | Default / behavior | +| --------------------- | ----------------------------------------------------------------------------- | +| `PI_CONFIG_DIR` | Config root dirname under home (default `.omp`) | | `PI_CODING_AGENT_DIR` | Full override for agent directory (default `~//agent`) | -| `PWD` | Used when matching canonical current working directory in path helpers | +| `PWD` | Used when matching canonical current working directory in path helpers | --- @@ -285,18 +312,18 @@ These are consumed via `@oh-my-pi/pi-utils/dirs` and affect where coding-agent s (From `packages/utils/src/procmgr.ts` and coding-agent bash tool integration.) -| Variable | Behavior | -|---|---| -| `PI_BASH_NO_CI` | Suppresses automatic `CI=true` injection into spawned shell env | -| `CLAUDE_BASH_NO_CI` | Legacy alias fallback for `PI_BASH_NO_CI` | -| `PI_BASH_NO_LOGIN` | Intended to disable login shell mode | -| `CLAUDE_BASH_NO_LOGIN` | Legacy alias fallback for `PI_BASH_NO_LOGIN` | -| `PI_SHELL_PREFIX` | Optional command prefix wrapper | -| `CLAUDE_CODE_SHELL_PREFIX` | Legacy alias fallback for `PI_SHELL_PREFIX` | -| `VISUAL` | Preferred external editor command | -| `EDITOR` | Fallback external editor command | +| Variable | Behavior | +| -------------------------- | ------------------------------------------------------------------------------ | +| `PI_BASH_NO_CI` | Suppresses automatic `CI=true` injection into spawned shell env | +| `CLAUDE_BASH_NO_CI` | Legacy alias fallback for `PI_BASH_NO_CI` | +| `PI_BASH_NO_LOGIN` | Disables login-shell mode; shell args become `['-c']` instead of `['-l','-c']` | +| `CLAUDE_BASH_NO_LOGIN` | Legacy alias fallback for `PI_BASH_NO_LOGIN` | +| `PI_SHELL_PREFIX` | Optional command prefix wrapper | +| `CLAUDE_CODE_SHELL_PREFIX` | Legacy alias fallback for `PI_SHELL_PREFIX` | +| `VISUAL` | Preferred external editor command | +| `EDITOR` | Fallback external editor command | -Current implementation note: `PI_BASH_NO_LOGIN`/`CLAUDE_BASH_NO_LOGIN` are read, but current `getShellArgs()` returns `['-l','-c']` in both branches (effectively no-op today). +Current implementation: `PI_BASH_NO_LOGIN`/`CLAUDE_BASH_NO_LOGIN` are active; when either is set, `getShellArgs()` returns `['-c']`. --- @@ -304,40 +331,41 @@ Current implementation note: `PI_BASH_NO_LOGIN`/`CLAUDE_BASH_NO_LOGIN` are read, These are read as runtime signals; they are usually set by the terminal/OS rather than manually configured. -| Variable | Used for | -|---|---| -| `COLORTERM`, `TERM`, `WT_SESSION` | Color capability detection (theme color mode) | -| `COLORFGBG` | Terminal background light/dark auto-detection | -| `TERM_PROGRAM`, `TERM_PROGRAM_VERSION`, `TERMINAL_EMULATOR` | Terminal identity in system prompt/context | +| Variable | Used for | +| ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | +| `COLORTERM`, `TERM`, `WT_SESSION` | Color capability detection (theme color mode) | +| `COLORFGBG` | Terminal background light/dark auto-detection | +| `TERM_PROGRAM`, `TERM_PROGRAM_VERSION`, `TERMINAL_EMULATOR` | Terminal identity in system prompt/context | | `KDE_FULL_SESSION`, `XDG_CURRENT_DESKTOP`, `DESKTOP_SESSION`, `XDG_SESSION_DESKTOP`, `GDMSESSION`, `WINDOWMANAGER` | Desktop/window-manager detection in system prompt/context | -| `KITTY_WINDOW_ID`, `TMUX_PANE`, `TERM_SESSION_ID`, `WT_SESSION` | Stable per-terminal session breadcrumb IDs | -| `SHELL`, `ComSpec`, `TERM_PROGRAM`, `TERM` | System info diagnostics | -| `APPDATA`, `XDG_CONFIG_HOME` | lspmux config path resolution | -| `HOME` | Path shortening in MCP command UI | +| `KITTY_WINDOW_ID`, `TMUX_PANE`, `TERM_SESSION_ID`, `WT_SESSION` | Stable per-terminal session breadcrumb IDs | +| `SHELL`, `ComSpec`, `TERM_PROGRAM`, `TERM` | System info diagnostics | +| `APPDATA`, `XDG_CONFIG_HOME` | lspmux config path resolution | +| `HOME` | Path shortening in MCP command UI | --- ## 9) TUI runtime flags (shared package, affects coding-agent UX) -| Variable | Behavior | -|---|---| -| `PI_NOTIFICATIONS` | `off` / `0` / `false` suppress desktop notifications | -| `PI_TUI_WRITE_LOG` | If set, logs TUI writes to file | -| `PI_HARDWARE_CURSOR` | If `1`, enables hardware cursor mode | -| `PI_CLEAR_ON_SHRINK` | If `1`, clears empty rows when content shrinks | -| `PI_DEBUG_REDRAW` | If `1`, enables redraw debug logging | -| `PI_TUI_DEBUG` | If `1`, enables deep TUI debug dump path | +| Variable | Behavior | +| ------------------------- | ------------------------------------------------------------------------------------- | +| `PI_NOTIFICATIONS` | `off` / `0` / `false` suppress desktop notifications | +| `PI_TUI_WRITE_LOG` | If set, logs TUI writes to file | +| `PI_HARDWARE_CURSOR` | If `1`, enables hardware cursor mode | +| `PI_CLEAR_ON_SHRINK` | If `1`, clears empty rows when content shrinks | +| `PI_DEBUG_REDRAW` | If `1`, enables redraw debug logging | +| `PI_TUI_DEBUG` | If `1`, enables deep TUI debug dump path | +| `PI_FORCE_IMAGE_PROTOCOL` | Forces terminal image protocol detection (`kitty`, `iterm2`/`iterm`, `sixel`, `none`) | --- -## 11) Commit generation controls +## 10) Commit generation controls -| Variable | Behavior | -|---|---| +| Variable | Behavior | +| ------------------------- | ------------------------------------------------------------------- | | `PI_COMMIT_TEST_FALLBACK` | If `true` (case-insensitive), force commit fallback generation path | -| `PI_COMMIT_NO_FALLBACK` | If `true`, disables fallback when agent returns no proposal | -| `PI_COMMIT_MAP_REDUCE` | If `false`, disables map-reduce commit analysis path | -| `DEBUG` | If set, commit agent error stack traces are printed | +| `PI_COMMIT_NO_FALLBACK` | If `true`, disables fallback when agent returns no proposal | +| `PI_COMMIT_MAP_REDUCE` | If `false`, disables map-reduce commit analysis path | +| `DEBUG` | If set, commit agent error stack traces are printed | --- diff --git a/docs/extension-loading.md b/docs/extension-loading.md index e5df941aa..ac0a07fd1 100644 --- a/docs/extension-loading.md +++ b/docs/extension-loading.md @@ -40,9 +40,15 @@ Notes: - Native auto-discovery is currently `.omp` based. - Legacy `.pi` is still accepted in `package.json` manifest keys (`pi.extensions`), but not as a native root here. -### 2) Explicitly configured paths +### 2) Installed plugin extension entries -After auto-discovery, configured paths are appended and resolved. +After native auto-discovery, `discoverAndLoadExtensions()` appends extension entry points from enabled installed plugins via `getAllPluginExtensionPaths(cwd)`. + +Plugin extension entries come from package `omp.extensions` / `pi.extensions` manifests, including enabled feature entries. + +### 3) Explicitly configured paths + +After plugin extension entries, configured paths are appended and resolved. Configured path sources in the main session startup path (`sdk.ts`): @@ -154,7 +160,8 @@ Rules and constraints: Order: 1. Native auto-discovered modules -2. Explicit configured paths (in provided order) +2. Installed plugin extension entries +3. Explicit configured paths (in provided order) In `sdk.ts`, configured order is: diff --git a/docs/extensions.md b/docs/extensions.md index 29f09be78..3b3dbf861 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -20,7 +20,7 @@ An extension is a TS/JS module exporting a default factory: import type { ExtensionAPI } from "@oh-my-pi/pi-coding-agent"; export default function myExtension(pi: ExtensionAPI) { - // register handlers/tools/commands/renderers + // register handlers/tools/commands/renderers } ``` @@ -69,37 +69,37 @@ import type { ExtensionAPI } from "@oh-my-pi/pi-coding-agent"; import { Type } from "@sinclair/typebox"; export default function (pi: ExtensionAPI) { - pi.setLabel("Safety + Utilities"); + pi.setLabel("Safety + Utilities"); - pi.on("session_start", async (_event, ctx) => { - ctx.ui.notify(`Extension loaded in ${ctx.cwd}`, "info"); - }); + pi.on("session_start", async (_event, ctx) => { + ctx.ui.notify(`Extension loaded in ${ctx.cwd}`, "info"); + }); - pi.on("tool_call", async (event) => { - if (event.toolName === "bash" && event.input.command?.includes("rm -rf")) { - return { block: true, reason: "Blocked by extension policy" }; - } - }); + pi.on("tool_call", async (event) => { + if (event.toolName === "bash" && event.input.command?.includes("rm -rf")) { + return { block: true, reason: "Blocked by extension policy" }; + } + }); - pi.registerTool({ - name: "hello_extension", - label: "Hello Extension", - description: "Return a greeting", - parameters: Type.Object({ name: Type.String() }), - async execute(_toolCallId, params, _signal, _onUpdate, _ctx) { - return { - content: [{ type: "text", text: `Hello, ${params.name}` }], - details: { greeted: params.name }, - }; - }, - }); + pi.registerTool({ + name: "hello_extension", + label: "Hello Extension", + description: "Return a greeting", + parameters: Type.Object({ name: Type.String() }), + async execute(_toolCallId, params, _signal, _onUpdate, _ctx) { + return { + content: [{ type: "text", text: `Hello, ${params.name}` }], + details: { greeted: params.name }, + }; + }, + }); - pi.registerCommand("hello-ext", { - description: "Show queue state", - handler: async (_args, ctx) => { - ctx.ui.notify(`pending=${ctx.hasPendingMessages()}`, "info"); - }, - }); + pi.registerCommand("hello-ext", { + description: "Show queue state", + handler: async (_args, ctx) => { + ctx.ui.notify(`pending=${ctx.hasPendingMessages()}`, "info"); + }, + }); } ``` @@ -240,26 +240,26 @@ Template: ```ts pi.registerTool({ - name: "my_tool", - label: "My Tool", - description: "...", - parameters: Type.Object({}), - async execute(_id, _params, signal, onUpdate, ctx) { - if (signal?.aborted) { - return { content: [{ type: "text", text: "Cancelled" }] }; - } - onUpdate?.({ content: [{ type: "text", text: "Working..." }] }); - return { content: [{ type: "text", text: "Done" }], details: {} }; - }, - onSession(event, ctx) { - // reason: start|switch|branch|tree|shutdown - }, - renderCall(args, theme) { - // optional TUI render - }, - renderResult(result, options, theme, args) { - // optional TUI render - }, + name: "my_tool", + label: "My Tool", + description: "...", + parameters: Type.Object({}), + async execute(_id, _params, signal, onUpdate, ctx) { + if (signal?.aborted) { + return { content: [{ type: "text", text: "Cancelled" }] }; + } + onUpdate?.({ content: [{ type: "text", text: "Working..." }] }); + return { content: [{ type: "text", text: "Done" }], details: {} }; + }, + onSession(event, ctx) { + // reason: start|switch|branch|tree|shutdown + }, + renderCall(args, options, theme) { + // optional TUI render + }, + renderResult(result, options, theme, args) { + // optional TUI render + }, }); ``` @@ -322,13 +322,13 @@ Example reconstruction pattern: ```ts pi.on("session_start", async (_event, ctx) => { - let latest; - for (const entry of ctx.sessionManager.getBranch()) { - if (entry.type === "custom" && entry.customType === "my-state") { - latest = entry.data; - } - } - // restore from latest + let latest; + for (const entry of ctx.sessionManager.getBranch()) { + if (entry.type === "custom" && entry.customType === "my-state") { + latest = entry.data; + } + } + // restore from latest }); ``` @@ -338,7 +338,7 @@ pi.on("session_start", async (_event, ctx) => { ```ts pi.registerMessageRenderer("my-type", (message, { expanded }, theme) => { - // return pi-tui Component + // return pi-tui Component }); ``` diff --git a/docs/fs-scan-cache-architecture.md b/docs/fs-scan-cache-architecture.md index d1aa7f210..9bc516f3d 100644 --- a/docs/fs-scan-cache-architecture.md +++ b/docs/fs-scan-cache-architecture.md @@ -7,6 +7,7 @@ This document defines the current contract for the shared filesystem scan cache The cache stores full directory-scan entry lists (`GlobMatch[]`) keyed by scan scope and traversal policy, then lets higher-level operations (glob filtering, fuzzy scoring, grep file selection) run against those cached entries. Primary goals: + - avoid repeated filesystem walks for repeated discovery/search calls - keep consistency across `glob`, `fuzzyFind`, and `grep` when they share the same scan policy - allow explicit staleness recovery for empty results and explicit invalidation after file mutations @@ -28,27 +29,31 @@ Primary goals: ## Cache key partitioning (hard contract) Each entry is keyed by: + - canonicalized `root` directory path - `include_hidden` boolean - `use_gitignore` boolean +- `skip_node_modules` boolean Implications: + - Hidden and non-hidden scans do **not** share entries. - Gitignore-respecting and ignore-disabled scans do **not** share entries. -- Consumers must pass stable semantics for hidden/gitignore behavior; changing either flag creates a different cache partition. - -`node_modules` inclusion is **not** in the cache key. The cache stores entries with `node_modules` included; per-consumer filtering is applied after retrieval. +- Scans that prune `node_modules` do **not** share entries with scans that include it. +- Consumers must pass stable semantics for hidden/gitignore/node_modules behavior; changing any flag creates a different cache partition. ## Scan collection behavior -Cache population uses a deterministic walker (`ignore::WalkBuilder`) configured by `include_hidden` and `use_gitignore`: +Cache population uses a deterministic walker (`ignore::WalkBuilder`) configured by `include_hidden`, `use_gitignore`, and `skip_node_modules`: + - `follow_links(false)` - sorted by file path - `.git` is always skipped -- `node_modules` is always collected at cache-scan time (and optionally filtered later) +- `node_modules` is pruned at traversal time when `skip_node_modules=true` - entry file type + `mtime` are captured via `symlink_metadata` Search roots are resolved by `resolve_search_path`: + - relative paths are resolved against current cwd - target must be an existing directory - root is canonicalized when possible @@ -56,11 +61,13 @@ Search roots are resolved by `resolve_search_path`: ## Freshness and eviction policy Global policy (environment-overridable): + - `FS_SCAN_CACHE_TTL_MS` (default `1000`) - `FS_SCAN_EMPTY_RECHECK_MS` (default `200`) - `FS_SCAN_CACHE_MAX_ENTRIES` (default `16`) Behavior: + - `get_or_scan(...)` - if TTL is `0`: bypass cache entirely, always fresh scan (`cache_age_ms = 0`) - on cache hit within TTL: return cached entries + non-zero `cache_age_ms` @@ -70,14 +77,17 @@ Behavior: ## Empty-result fast recheck (separate from normal hits) Normal cache hit: + - a cache hit inside TTL returns cached entries and does nothing else. Empty-result fast recheck: + - this is a **caller-side** policy using `ScanResult.cache_age_ms` - if filtered/query result is empty and cached scan age is at least `empty_recheck_ms()`, caller performs one `force_rescan(...)` and retries - intended to reduce stale-negative results when files were recently added but cache is still within TTL Current consumers: + - `glob`: rechecks when filtered matches are empty and scan age exceeds threshold - `fuzzyFind` (`fd.rs`): rechecks only when query is non-empty and scored matches are empty - `grep`: rechecks when selected candidate file list is empty @@ -87,11 +97,13 @@ Current consumers: Cache is opt-in on all exposed APIs (`cache?: boolean`, default `false`). Current defaults in native APIs: -- `glob`: `hidden=false`, `gitignore=true`, `cache=false` -- `fuzzyFind`: `hidden=false`, `gitignore=true`, `cache=false` -- `grep`: `hidden=true`, `cache=false`, and cache scan always uses `use_gitignore=true` + +- `glob`: `hidden=false`, `gitignore=true`, `cache=false`, and `node_modules` included only when the pattern mentions `node_modules` +- `fuzzyFind`: `hidden=false`, `gitignore=true`, `cache=false`, and `node_modules` is skipped +- `grep`: `hidden=true`, `gitignore=true`, `cache=false`, and `node_modules` included only when the glob mentions `node_modules` Coding-agent callers today: + - High-volume mention candidate discovery enables cache: - `packages/coding-agent/src/utils/file-mentions.ts` - profile: `hidden=true`, `gitignore=true`, `includeNodeModules=true`, `cache=true` @@ -101,11 +113,13 @@ Coding-agent callers today: ## Invalidation contract Native invalidation entrypoint: + - `invalidateFsScanCache(path?: string)` - with `path`: remove cache entries whose root is a prefix of target path - without path: clear all scan cache entries Path handling details: + - relative invalidation paths are resolved against cwd - invalidation attempts canonicalization - if target does not exist (e.g., delete), fallback canonicalizes parent and reattaches filename when possible @@ -116,11 +130,13 @@ Path handling details: Coding-agent code must invalidate after successful filesystem mutations. Central helpers: + - `invalidateFsScanAfterWrite(path)` - `invalidateFsScanAfterDelete(path)` - `invalidateFsScanAfterRename(oldPath, newPath)` (invalidates both sides when paths differ) Current mutation tool callsites: + - `packages/coding-agent/src/tools/write.ts` - `packages/coding-agent/src/patch/index.ts` (hashline/patch/replace flows) @@ -131,11 +147,11 @@ Rule: if a flow mutates filesystem content or location and bypasses these helper When introducing cache use in a new scanner/search path: 1. **Use stable scan policy inputs** - - decide hidden/gitignore semantics first + - decide hidden/gitignore/node_modules semantics first - pass them consistently to `get_or_scan`/`force_rescan` so cache partitions are intentional 2. **Treat cache data as pre-filtered only by traversal policy** - - apply tool-specific filtering (glob patterns, type filters, node_modules rules) after retrieval + - apply tool-specific filtering (glob patterns, type filters, scoring) after retrieval - never assume cached entries already reflect your higher-level filters 3. **Implement empty-result fast recheck only for stale-negative risk** @@ -158,5 +174,5 @@ When introducing cache use in a new scanner/search path: - Cache scope is process-local in-memory (`DashMap`), not persisted across process restarts. - Cache stores scan entries, not final tool results. -- `glob`/`fuzzyFind`/`grep` share scan entries only when key dimensions (`root`, `hidden`, `gitignore`) match. +- `glob`/`fuzzyFind`/`grep` share scan entries only when key dimensions (`root`, `hidden`, `gitignore`, `skip_node_modules`) match. - `.git` is always excluded at scan collection time regardless of caller options. diff --git a/docs/gemini-manifest-extensions.md b/docs/gemini-manifest-extensions.md index b6aeefdbb..3a53f9f0c 100644 --- a/docs/gemini-manifest-extensions.md +++ b/docs/gemini-manifest-extensions.md @@ -55,11 +55,11 @@ The capability type defines this manifest shape: ```ts interface ExtensionManifest { - name?: string; - description?: string; - mcpServers?: Record>; - tools?: unknown[]; - context?: unknown; + name?: string; + description?: string; + mcpServers?: Record>; + tools?: unknown[]; + context?: unknown; } ``` diff --git a/docs/handoff-generation-pipeline.md b/docs/handoff-generation-pipeline.md index de3fd082e..00c0e7f33 100644 --- a/docs/handoff-generation-pipeline.md +++ b/docs/handoff-generation-pipeline.md @@ -45,16 +45,23 @@ The same minimum-content guard exists again inside `AgentSession.handoff()` and - Reads current branch entries (`sessionManager.getBranch()`) - Validates minimum message count (`>= 2`) - Creates `#handoffAbortController` -- Builds a fixed, inline prompt requesting a structured handoff document (`Goal`, `Constraints & Preferences`, `Progress`, `Key Decisions`, `Critical Context`, `Next Steps`) +- Renders the fixed prompt template `prompts/system/handoff-document.md` with optional `additionalFocus` - Appends `Additional focus: ...` if custom instructions are provided -Prompt is sent via: +Prompt is sent as an agent-authored developer message via: ```ts -await this.prompt(handoffPrompt, { expandPromptTemplates: false }); +await this.#promptAgentWithIdleRetry([ + { + role: "developer", + content: [{ type: "text", text: handoffPrompt }], + attribution: "agent", + timestamp: Date.now(), + }, +]); ``` -`expandPromptTemplates: false` prevents slash/prompt-template expansion of this internal instruction payload. +Because handoff bypasses `prompt(...)`, normal slash/prompt-template expansion is not applied to this internal instruction payload. ### 2) Capture completion @@ -71,10 +78,10 @@ Important extraction assumptions: ### 3) Cancellation checks -`handoff()` returns `undefined` when either condition holds: +Cancellation throws `Error("Handoff cancelled")`; a completed generation with no extracted text returns `undefined`. -- no captured handoff text, or -- `#handoffAbortController.signal.aborted` is true +- no captured handoff text → returns `undefined` +- aborted handoff signal → throws `Error("Handoff cancelled")` It always clears `#handoffAbortController` in `finally`. @@ -83,13 +90,13 @@ It always clears `#handoffAbortController` in `finally`. If text was captured and not aborted: 1. Flush current session writer (`sessionManager.flush()`) -2. Start a brand-new session (`sessionManager.newSession()`) -3. Reset in-memory agent state (`agent.reset()`) -4. Rebind `agent.sessionId` to new session id -5. Clear queued context arrays (`#steeringMessages`, `#followUpMessages`, `#pendingNextTurnMessages`) -6. Reset todo reminder counter - -`newSession()` creates a fresh header and empty entry list (leaf reset to `null`). In the handoff path, no `parentSession` is passed. +2. Cancel async jobs (`#asyncJobManager?.cancelAll()`) +3. Start a brand-new session (`sessionManager.newSession()`) +4. Reset in-memory agent state (`agent.reset()`) +5. Rebind `agent.sessionId` to new session id +6. Clear queued context arrays (`#steeringMessages`, `#followUpMessages`, `#pendingNextTurnMessages`) and any scheduled hidden next-turn generation +7. Reset todo reminder counter + `newSession()` creates a fresh header and empty entry list (leaf reset to `null`). In the handoff path, no `parentSession` is passed. ### 5) Handoff-context injection @@ -119,9 +126,10 @@ Semantics: After injection: -1. `sessionManager.buildSessionContext()` resolves message list for current leaf +1. `buildDisplaySessionContext()` resolves message list for current leaf 2. `agent.replaceMessages(sessionContext.messages)` makes the injected handoff message active context -3. Method returns `{ document: handoffText }` +3. Todo phases are synchronized from the new branch +4. Method returns `{ document: handoffText, savedPath? }` At this point, the active LLM context in the new session contains the injected handoff message, not the old transcript. @@ -164,11 +172,11 @@ After session reset, handoff is persisted as `custom_message` with `customType: - `abortHandoff()` → aborts `#handoffAbortController` - `isGeneratingHandoff` → true while controller exists -When this abort path is used, the handoff subscriber rejects with `Error("Handoff cancelled")`, and command controller maps it to cancellation UI. +When this abort path is used, the handoff completion waiter resolves as cancelled, `agent.abort()` is called, and `handoff()` throws `Error("Handoff cancelled")`; command controller maps it to cancellation UI. -### Interactive `/handoff` path limitation +### Interactive `/handoff` path -In current interactive controller wiring, `/handoff` does not install a dedicated Escape handler that calls `abortHandoff()` (unlike compaction/branch-summary paths that temporarily override `editor.onEscape`). +The editor does not install a dedicated Escape handler that calls `abortHandoff()` for `/handoff`, but `InputController` treats `session.isGeneratingHandoff` as a busy state. Practical impact: @@ -206,15 +214,16 @@ High-level state flow: 1. Interactive slash command intercepted 2. Preflight message-count guard 3. `#handoffAbortController` created (`isGeneratingHandoff = true`) -4. Internal handoff prompt submitted (visible in chat as normal assistant generation) +4. Internal developer handoff prompt submitted (visible in chat as normal assistant generation) 5. On `agent_end`, last assistant text extracted -6. If missing/aborted → return `undefined` or cancellation error path +6. If missing text → return `undefined`; if aborted → cancellation error path 7. If present: - flush old session + - cancel async jobs - create new empty session - reset runtime queues/counters - append `custom_message(handoff)` - - rebuild and replace active agent messages + - optionally save an auto-triggered handoff document under the session artifacts directory when `compaction.handoffSaveToDisk` is enabled 8. Controller rebuilds chat UI and announces success 9. `#handoffAbortController` cleared (`isGeneratingHandoff = false`) @@ -223,5 +232,6 @@ High-level state flow: - Handoff extraction is heuristic: "last assistant text blocks"; no structural validation. - No hard check that generated markdown follows requested section format. - Missing extracted text is reported as cancellation in controller UX. -- `/handoff` interactive flow currently lacks a dedicated Escape→`abortHandoff()` binding. -- New session lineage metadata (`parentSession`) is not set by this path. \ No newline at end of file +- `/handoff` interactive flow currently lacks a dedicated Escape→`abortHandoff()` binding, though the session reports handoff as a busy state. +- New session lineage metadata (`parentSession`) is not set by this path. +- Auto-triggered handoffs can write a timestamped `handoff-*.md` artifact when `compaction.handoffSaveToDisk` is enabled; write failure is logged and does not fail the handoff. diff --git a/docs/hooks.md b/docs/hooks.md index 8325f9f39..f3eb2ce30 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -25,14 +25,17 @@ So this file documents the hook subsystem implementation itself (types/loader/ru A hook module must default-export a factory: ```ts -import type { HookAPI } from "@oh-my-pi/pi-coding-agent/hooks"; +import type { HookAPI } from "@oh-my-pi/pi-coding-agent/extensibility/hooks"; export default function hook(pi: HookAPI): void { - pi.on("tool_call", async (event, ctx) => { - if (event.toolName === "bash" && String(event.input.command ?? "").includes("rm -rf")) { - return { block: true, reason: "blocked by policy" }; - } - }); + pi.on("tool_call", async (event, ctx) => { + if ( + event.toolName === "bash" && + String(event.input.command ?? "").includes("rm -rf") + ) { + return { block: true, reason: "blocked by policy" }; + } + }); } ``` @@ -126,7 +129,6 @@ tool_call handlers └─ error ──> emit tool_result(isError=true) then rethrow original error ``` - ## Execution model and mutation semantics ### 1) Pre-execution: `tool_call` @@ -252,85 +254,92 @@ Hook status text set via `ctx.ui.setStatus(key, text)` is: ### Block unsafe bash commands ```ts -import type { HookAPI } from "@oh-my-pi/pi-coding-agent/hooks"; +import type { HookAPI } from "@oh-my-pi/pi-coding-agent/extensibility/hooks"; export default function (pi: HookAPI): void { - pi.on("tool_call", async (event, ctx) => { - if (event.toolName !== "bash") return; - const cmd = String(event.input.command ?? ""); - if (!cmd.includes("rm -rf")) return; + pi.on("tool_call", async (event, ctx) => { + if (event.toolName !== "bash") return; + const cmd = String(event.input.command ?? ""); + if (!cmd.includes("rm -rf")) return; - if (!ctx.hasUI) return { block: true, reason: "rm -rf blocked (no UI)" }; - const ok = await ctx.ui.confirm("Dangerous command", `Allow: ${cmd}`); - if (!ok) return { block: true, reason: "user denied command" }; - }); + if (!ctx.hasUI) return { block: true, reason: "rm -rf blocked (no UI)" }; + const ok = await ctx.ui.confirm("Dangerous command", `Allow: ${cmd}`); + if (!ok) return { block: true, reason: "user denied command" }; + }); } ``` ### Redact tool output on post-execution ```ts -import type { HookAPI } from "@oh-my-pi/pi-coding-agent/hooks"; +import type { HookAPI } from "@oh-my-pi/pi-coding-agent/extensibility/hooks"; export default function (pi: HookAPI): void { - pi.on("tool_result", async event => { - if (event.toolName !== "read" || event.isError) return; + pi.on("tool_result", async (event) => { + if (event.toolName !== "read" || event.isError) return; - const redacted = event.content.map(chunk => { - if (chunk.type !== "text") return chunk; - return { ...chunk, text: chunk.text.replaceAll(/API_KEY=\S+/g, "API_KEY=[REDACTED]") }; - }); + const redacted = event.content.map((chunk) => { + if (chunk.type !== "text") return chunk; + return { + ...chunk, + text: chunk.text.replaceAll(/API_KEY=\S+/g, "API_KEY=[REDACTED]"), + }; + }); - return { content: redacted }; - }); + return { content: redacted }; + }); } ``` ### Modify model context per LLM call ```ts -import type { HookAPI } from "@oh-my-pi/pi-coding-agent/hooks"; +import type { HookAPI } from "@oh-my-pi/pi-coding-agent/extensibility/hooks"; export default function (pi: HookAPI): void { - pi.on("context", async event => { - const filtered = event.messages.filter(msg => !(msg.role === "custom" && msg.customType === "debug-only")); - return { messages: filtered }; - }); + pi.on("context", async (event) => { + const filtered = event.messages.filter( + (msg) => !(msg.role === "custom" && msg.customType === "debug-only"), + ); + return { messages: filtered }; + }); } ``` ### Register slash command with command-safe context methods ```ts -import type { HookAPI } from "@oh-my-pi/pi-coding-agent/hooks"; +import type { HookAPI } from "@oh-my-pi/pi-coding-agent/extensibility/hooks"; export default function (pi: HookAPI): void { - pi.registerCommand("handoff", { - description: "Create a new session with setup message", - handler: async (_args, ctx) => { - await ctx.waitForIdle(); - await ctx.newSession({ - parentSession: ctx.sessionManager.getSessionFile(), - setup: async sm => { - sm.appendMessage({ - role: "user", - content: [{ type: "text", text: "Continue from prior session summary." }], - timestamp: Date.now(), - }); - }, - }); - }, - }); + pi.registerCommand("handoff", { + description: "Create a new session with setup message", + handler: async (_args, ctx) => { + await ctx.waitForIdle(); + await ctx.newSession({ + parentSession: ctx.sessionManager.getSessionFile(), + setup: async (sm) => { + sm.appendMessage({ + role: "user", + content: [ + { type: "text", text: "Continue from prior session summary." }, + ], + timestamp: Date.now(), + }); + }, + }); + }, + }); } ``` ## Export surface -`src/extensibility/hooks/index.ts` exports: +`src/extensibility/hooks/index.ts` and the package subpath `@oh-my-pi/pi-coding-agent/extensibility/hooks` export: - loading APIs (`discoverAndLoadHooks`, `loadHooks`) - runner and wrapper (`HookRunner`, `HookToolWrapper`) - all hook types - `execCommand` re-export -And package root (`src/index.ts`) re-exports hook **types** as a legacy compatibility surface. +The package root (`@oh-my-pi/pi-coding-agent`) does not re-export `HookAPI`; import legacy hook types from the hooks subpath. diff --git a/docs/marketplace.md b/docs/marketplace.md index f03b16d5e..70dc2df17 100644 --- a/docs/marketplace.md +++ b/docs/marketplace.md @@ -20,7 +20,7 @@ A **plugin** is a directory containing skills, commands, hooks, MCP servers, or **Scopes**: plugins can be installed at two scopes: - **user** (default) -- available in all projects, stored in `~/.omp/plugins/installed_plugins.json` -- **project** -- available only in the current project, stored in `.omp/installed_plugins.json` +- **project** -- available only in the current project, stored in `.omp/plugins/installed_plugins.json` Project-scoped installs shadow user-scoped installs of the same plugin. @@ -28,28 +28,28 @@ Project-scoped installs shadow user-scoped installs of the same plugin. ### Interactive mode -| Command | Effect | -|---|---| +| Command | Effect | +| -------------- | ----------------------------------------- | | `/marketplace` | Open interactive plugin browser (install) | ### Marketplace management -| Command | Effect | -|---|---| -| `/marketplace add ` | Add a marketplace source | -| `/marketplace remove ` | Remove a marketplace | +| Command | Effect | +| ---------------------------- | -------------------------------------------- | +| `/marketplace add ` | Add a marketplace source | +| `/marketplace remove ` | Remove a marketplace | | `/marketplace update [name]` | Re-fetch catalog(s); omit name to update all | -| `/marketplace list` | List configured marketplaces | +| `/marketplace list` | List configured marketplaces | ### Plugin operations -| Command | Effect | -|---|---| -| `/marketplace discover [marketplace]` | Browse available plugins | -| `/marketplace install [--force] [--scope user\|project] name@marketplace` | Install a plugin | -| `/marketplace uninstall [--scope user\|project] name@marketplace` | Uninstall a plugin | -| `/marketplace installed` | List installed marketplace plugins | -| `/marketplace upgrade [--scope user\|project] [name@marketplace]` | Upgrade one or all plugins | +| Command | Effect | +| ------------------------------------------------------------------------- | ---------------------------------- | +| `/marketplace discover [marketplace]` | Browse available plugins | +| `/marketplace install [--force] [--scope user\|project] name@marketplace` | Install a plugin | +| `/marketplace uninstall [--scope user\|project] name@marketplace` | Uninstall a plugin | +| `/marketplace installed` | List installed marketplace plugins | +| `/marketplace upgrade [--scope user\|project] [name@marketplace]` | Upgrade one or all plugins | ### CLI equivalents @@ -61,19 +61,21 @@ omp plugin marketplace remove omp plugin marketplace update [name] omp plugin marketplace list omp plugin discover [marketplace] -omp plugin install --scope project name@marketplace +omp plugin install [--force] [--scope user|project] name@marketplace +omp plugin uninstall [--scope user|project] name@marketplace +omp plugin upgrade [--scope user|project] [name@marketplace] ``` ## Marketplace sources When you run `/marketplace add `, the system classifies the source: -| Source format | Type | Example | -|---|---|---| -| `owner/repo` | GitHub shorthand | `anthropics/claude-plugins-official` | -| `https://...*.json` | Direct catalog URL | `https://example.com/marketplace.json` | -| `https://...*.git` or `git@...` | Git repository | `https://github.com/org/repo.git` | -| `./path` or `~/path` or `/path` | Local directory | `./my-marketplace` | +| Source format | Type | Example | +| ------------------------------- | ------------------ | -------------------------------------- | +| `owner/repo` | GitHub shorthand | `anthropics/claude-plugins-official` | +| `https://...*.json` | Direct catalog URL | `https://example.com/marketplace.json` | +| `https://...*.git` or `git@...` | Git repository | `https://github.com/org/repo.git` | +| `./path` or `~/path` or `/path` | Local directory | `./my-marketplace` | The system clones the repository (or reads the local directory), locates `.claude-plugin/marketplace.json`, validates it, and caches the catalog locally. @@ -104,41 +106,43 @@ A marketplace catalog lives at `.claude-plugin/marketplace.json` in the reposito ### Required fields -| Field | Description | -|---|---| -| `name` | Marketplace name. Lowercase alphanumeric, hyphens, and dots. Must start and end with alphanumeric. Max 64 chars. | -| `owner.name` | Marketplace owner name | -| `plugins` | Array of plugin entries | +| Field | Description | +| ------------ | ---------------------------------------------------------------------------------------------------------------- | +| `name` | Marketplace name. Lowercase alphanumeric, hyphens, and dots. Must start and end with alphanumeric. Max 64 chars. | +| `owner.name` | Marketplace owner name | +| `plugins` | Array of plugin entries | ### Plugin entry fields -| Field | Required | Description | -|---|---|---| -| `name` | yes | Plugin name (same rules as marketplace name) | -| `source` | yes | Where to find the plugin (see below) | -| `description` | no | Short description | -| `version` | no | Version string | -| `author` | no | `{ name, email? }` | -| `homepage` | no | URL | -| `category` | no | Category string (e.g. `development`, `productivity`, `security`) | -| `tags` | no | Array of string tags | -| `strict` | no | Boolean | -| `commands` | no | Slash commands provided | -| `agents` | no | Agents provided | -| `hooks` | no | Hook definitions | -| `mcpServers` | no | MCP server definitions | -| `lspServers` | no | LSP server definitions | +| Field | Required | Description | +| ------------- | -------- | ---------------------------------------------------------------- | +| `name` | yes | Plugin name (same rules as marketplace name) | +| `source` | yes | Where to find the plugin (see below) | +| `description` | no | Short description | +| `version` | no | Version string | +| `author` | no | `{ name, email? }` | +| `homepage` | no | URL | +| `category` | no | Category string (e.g. `development`, `productivity`, `security`) | +| `tags` | no | Array of string tags | +| `strict` | no | Boolean | +| `commands` | no | Slash commands provided | +| `agents` | no | Agents provided | +| `hooks` | no | Hook definitions | +| `mcpServers` | no | MCP server definitions | +| `lspServers` | no | LSP server definitions | ### Plugin source formats The `source` field supports several formats: **Relative path** (within the marketplace repo): + ```json "source": "./plugins/my-plugin" ``` **Git repository URL**: + ```json "source": { "source": "url", @@ -148,6 +152,7 @@ The `source` field supports several formats: ``` **GitHub shorthand**: + ```json "source": { "source": "github", @@ -158,6 +163,7 @@ The `source` field supports several formats: ``` **Git subdirectory** (monorepo): + ```json "source": { "source": "git-subdir", @@ -169,6 +175,7 @@ The `source` field supports several formats: ``` **npm package**: + ```json "source": { "source": "npm", @@ -181,16 +188,16 @@ The `source` field supports several formats: ``` ~/.omp/ - config/ - marketplaces.json # Registry of added marketplaces + marketplaces.json # Registry of added marketplaces plugins/ - installed_plugins.json # User-scoped installed plugins + installed_plugins.json # User-scoped installed plugins cache/ - marketplaces/ # Cached marketplace catalogs - plugins/ # Cached plugin directories + marketplaces/ # Cached marketplace catalogs + plugins/ # Cached plugin directories /.omp/ - installed_plugins.json # Project-scoped installed plugins + plugins/ + installed_plugins.json # Project-scoped installed plugins ``` ## Naming rules diff --git a/docs/mcp-config.md b/docs/mcp-config.md index 3f4c631ad..5e531d40a 100644 --- a/docs/mcp-config.md +++ b/docs/mcp-config.md @@ -59,7 +59,7 @@ Top-level keys: - `$schema` — optional JSON Schema URL for tooling - `mcpServers` — map of server name to server config -- `disabledServers` — user-level denylist used to turn off discovered servers by name +- `disabledServers` — user-level denylist used to turn off discovered servers by name; runtime loading reads this list from `~/.omp/agent/mcp.json` Server names must match `^[a-zA-Z0-9_.-]{1,100}$`. @@ -317,11 +317,12 @@ This is the part that usually trips people up. ### In `.omp/mcp.json` and `~/.omp/agent/mcp.json` -Before OMP launches a server or makes an HTTP request, it resolves `env` and `headers` values like this: +Before OMP launches a stdio server or makes an HTTP/SSE request, it resolves stdio `env` values and HTTP/SSE `headers` values like this: -1. If a value starts with `!`, OMP runs it as a shell command and uses trimmed stdout. -2. Otherwise OMP first checks whether the value matches an environment variable name. -3. If that environment variable is not set, OMP uses the string literally. +1. If a value starts with `!`, OMP runs the rest as a shell command with a 10s timeout and uses trimmed stdout. +2. If the command fails, times out, or prints only whitespace, that `env`/`headers` entry is omitted. +3. Otherwise OMP checks whether the value names an environment variable. +4. If that environment variable is set to a non-empty value, OMP uses the environment value; otherwise it uses the string literally. Examples: @@ -344,7 +345,7 @@ That means this is valid and convenient for local secrets: ### In root `mcp.json` and `.mcp.json` -The standalone fallback loader also expands `${VAR}` and `${VAR:-default}` inside strings during discovery. +The standalone fallback loader also expands `${VAR}` and `${VAR:-default}` inside strings during discovery for `command`, `args`, `env`, `cwd`, `url`, `headers`, `auth`, and `oauth`. Example: @@ -366,7 +367,7 @@ If you want the least surprising OMP behavior, prefer `.omp/mcp.json` or `~/.omp ## `disabledServers` -`disabledServers` is mainly useful in the user config file (`~/.omp/agent/mcp.json`) when a server is discovered from some other source and you want OMP to ignore it without editing that other tool's config. +`disabledServers` is read from the user config file (`~/.omp/agent/mcp.json`) when a server is discovered from any source and you want OMP to ignore it without editing that other tool's config. Example: @@ -392,6 +393,8 @@ After editing, use: - `/mcp reload` to rediscover and reconnect servers in the current session - `/mcp list` to see which config file a server came from - `/mcp test ` to test a single server +- `/mcp reconnect ` to reconnect one server without rediscovering all configs +- `/mcp resources`, `/mcp prompts`, and `/mcp notifications` to inspect non-tool MCP capabilities ## Validation rules OMP enforces @@ -410,7 +413,7 @@ Practical implications: ## Discovery and precedence -OMP does not merge duplicate server definitions across files. Discovery providers are prioritized, and the higher-priority definition wins. +OMP does not merge duplicate server definitions across files. Discovery providers are prioritized, and the higher-priority definition wins. Separately, `disabledServers` from `~/.omp/agent/mcp.json` can suppress a discovered server by name. In practice: @@ -439,7 +442,7 @@ The JSON is valid, but the server may still be unreachable. Use `/mcp test Promise` - `close()` - `connected` -- optional callbacks: `onClose`, `onError`, `onNotification` +- optional callbacks: `onClose`, `onError`, `onNotification`, `onRequest` Transport implementations own framing and I/O details: - `StdioTransport`: newline-delimited JSON over subprocess stdio - `HttpTransport`: JSON-RPC over HTTP POST, with optional SSE responses/listening -### Important current caveat +### Manager/client wiring -Transport callbacks (`onClose`, `onError`, `onNotification`) are implemented, but current `MCPClient`/`MCPManager` flows do not wire reconnection logic to these callbacks. Notifications are only consumed if caller registers handlers. +`connectToServer()` always installs an `onRequest` handler for standard server-to-client requests. `MCPManager` installs notification handlers, OAuth refresh hooks for HTTP OAuth servers, and `onClose` reconnect handling for managed connections. ## Transport selection @@ -67,7 +69,7 @@ Transport callbacks (`onClose`, `onError`, `onNotification`) are implemented, bu ## Request IDs -Each transport generates per-request IDs (`Math.random` + timestamp string). IDs are transport-local correlation tokens. +Each transport generates per-request IDs with `Snowflake.next()`. IDs are transport-local correlation tokens. ## Stdio correlation path @@ -76,8 +78,9 @@ Each transport generates per-request IDs (`Math.random` + timestamp string). IDs - Read loop parses JSONL from stdout and calls `#handleMessage`. - If inbound message has matching `id`, request resolves/rejects. - If inbound message has `method` and no `id`, treated as notification and sent to `onNotification`. +- If inbound message has both `method` and `id`, treated as a server-to-client request and answered through `onRequest`; without a handler the transport replies with JSON-RPC `-32601 Method not found`. -Unknown IDs are ignored (no rejection, no error callback). +Unknown response IDs are ignored (no rejection, no error callback). ## HTTP correlation path @@ -85,8 +88,9 @@ Unknown IDs are ignored (no rejection, no error callback). - Non-SSE response path: parse one JSON-RPC response and return `result`/throw on `error`. - SSE response path (`Content-Type: text/event-stream`): stream events, return first message whose `id` matches expected request ID and has `result` or `error`. - SSE messages with `method` and no `id` are treated as notifications. +- SSE messages with both `method` and `id` are treated as server-to-client requests and answered with a POSTed JSON-RPC response. -If SSE stream ends before matching response, request fails with `No response received for request ID ...`. +If SSE stream ends before matching response, request fails with `No response received for request ID ...`. After the matching response is captured, the transport drains remaining SSE messages in the background. ## Notifications @@ -95,7 +99,7 @@ Client emits JSON-RPC notifications via `transport.notify(...)`. - Stdio: writes notification frame to stdin (`jsonrpc`, `method`, optional `params`) plus newline. - HTTP: sends POST body without `id`; success accepts `2xx` or `202 Accepted`. -Server-initiated notifications are only surfaced through transport `onNotification`; there is no default global subscriber in manager/client. +Server-initiated notifications are surfaced through transport `onNotification`; `MCPManager` consumes known MCP list/update notifications and can forward all notifications through its own callback. ## Stdio transport internals @@ -168,18 +172,20 @@ So `connected` means "transport usable", not "persistent stream established". - Subsequent requests/notifications include `Mcp-Session-Id`. - `close()` tries to terminate server session with HTTP DELETE; termination failures are ignored. -## Timeout and cancellation +## Timeout, cancellation, and auth refresh -For both `request()` and `notify()`: +For `request()`: - timeout uses `AbortController` (`config.timeout ?? 30000`) - external signal, if provided, is merged via `AbortSignal.any([...])` - AbortError handling distinguishes caller abort vs timeout -Errors thrown: +For `notify()`: -- timeout: `Request timeout after ...ms` (or `SSE response timeout ...`, `Notify timeout ...`) -- caller abort: original AbortError is rethrown when external signal is already aborted +- timeout uses an internal `AbortController` (`config.timeout ?? 30000`) +- there is no external abort option on the transport interface + +For HTTP OAuth configs managed by `MCPManager`, `request()` retries once on `HTTP 401`/`403` if token refresh returns replacement headers. ## HTTP error propagation @@ -204,9 +210,9 @@ Two SSE paths exist: - can process interleaved notifications during same stream 2. **Background SSE listener** (`startSSEListener()`) - - optional GET listener for server-initiated notifications - - currently not automatically started by MCP manager/client - - if GET returns `405`, listener silently disables itself (server does not support this mode) + - optional GET listener for server-initiated notifications and server-to-client requests + - `connectToServer()` starts it for HTTP/SSE transports after `initialize` and before `notifications/initialized` + - if GET returns `405`, another non-OK status, or no body, listener silently disables itself ## Malformed payload and disconnect handling @@ -214,7 +220,7 @@ SSE JSON parsing errors bubble out of `readSseJson` and reject request/listener. - Request SSE parse errors reject the active request. - Background listener errors trigger `onError` (except AbortError). -- No auto-reconnect for background listener. +- Transport does not restart the listener itself; managed connections may reconnect through manager `onClose` handling. ## `json-rpc.ts` utility vs transport abstraction @@ -234,31 +240,31 @@ This path is lightweight but less robust than full transport implementation. Current transport implementations do **not**: -- retry failed requests +- retry ordinary failed requests, except the HTTP transport's single OAuth-refresh retry when `onAuthError` is wired - reconnect after stdio process exit -- reconnect SSE listeners +- reconnect SSE listeners by themselves - resend in-flight requests after disconnect They fail fast and propagate errors. -## Manager/client-level +## Manager/tool-bridge level -`MCPManager` handles discovery/initial connection orchestration and can reconnect only by running connect flows again (`connectToServer`/`discoverAndConnect` paths). It does not auto-heal an already connected transport on runtime failure callbacks. +`MCPManager` wires `transport.onClose` for managed connections and runs `reconnectServer(name)` when a transport closes unexpectedly. Reconnect tears down the stale connection, re-resolves auth/config values, retries with backoff (`500`, `1000`, `2000`, `4000` ms), reloads tools, and preserves stale tools while reconnecting. -`MCPManager` does have startup fallback behavior for slow servers (deferred tools from cache), but that is tool availability fallback, not transport retry. +`MCPTool` and `DeferredMCPTool` also attempt one reconnect + retry for retriable connection errors during a tool call. This is tool availability recovery, not transport-level retry. ## Failure scenarios summary - **Malformed stdio message line**: dropped; stream continues. -- **Stdio stream/process ends**: transport closes; pending requests rejected as `Transport closed`. -- **HTTP non-2xx**: request/notify throws HTTP error. +- **Stdio stream/process ends**: transport closes; pending requests rejected as `Transport closed`; manager-managed connections trigger reconnect. +- **HTTP non-2xx**: request/notify throws HTTP error; managed OAuth requests can refresh auth and retry once on 401/403. - **Invalid JSON response**: parse exception propagated. - **SSE ends without matching id**: request fails with `No response received for request ID ...`. - **Timeout**: transport-specific timeout error. -- **Caller abort**: AbortError/reason propagated from caller signal. +- **Caller abort**: AbortError/reason propagated from caller signal where the method accepts one. ## Practical boundary rule If the concern is message shape, id correlation, or MCP method ordering, it belongs to protocol/client logic. -If the concern is framing (JSONL vs HTTP/SSE), stream parsing, fetch/spawn lifecycle, timeout clocks, or connection teardown, it belongs to transport implementation. \ No newline at end of file +If the concern is framing (JSONL vs HTTP/SSE), stream parsing, fetch/spawn lifecycle, timeout clocks, or connection teardown, it belongs to transport implementation. diff --git a/docs/mcp-runtime-lifecycle.md b/docs/mcp-runtime-lifecycle.md index 85ab1016a..631115dd6 100644 --- a/docs/mcp-runtime-lifecycle.md +++ b/docs/mcp-runtime-lifecycle.md @@ -12,8 +12,9 @@ This document describes how MCP servers are discovered, connected, exposed as to - failures per server, - or cached `DeferredMCPTool`s for still-pending servers. 5. **SDK wiring** merges MCP tools into runtime tool registry for the session. -6. **Live session** can refresh MCP tools via `/mcp` flows (`disconnectAll` + rediscover + `session.refreshMCPTools`). -7. **Teardown** happens when callers invoke `disconnectServer`/`disconnectAll`; manager also clears MCP tool registrations for disconnected servers. +6. **Post-connect enrichment** best-effort loads resources, resource templates, prompts, and optional resource subscriptions. +7. **Live session** can refresh MCP tools via `/mcp` flows (`disconnectAll` + rediscover + `session.refreshMCPTools`) and can reconnect individual servers on transport close or `/mcp reconnect`. +8. **Teardown** happens when callers invoke `disconnectServer`/`disconnectAll`; manager also clears MCP tool/resource/prompt registrations for disconnected servers. ## Discovery and load phase @@ -59,11 +60,13 @@ So startup does not fail the whole agent session when individual MCP servers fai - `#pendingToolLoads: Map>` — connected but tools still loading. - `#tools: CustomTool[]` — current MCP tool view exposed to callers. - `#sources: Map` — provider/source metadata even before connect completes. +- `#pendingReconnections: Map>` — reconnects in progress after a dropped transport or explicit reconnect. +- `#serverConfigs: Map` — original unresolved configs preserved so reconnect can re-resolve credentials without leaking resolved tokens. `getConnectionStatus(name)` derives status from these maps: - `connected` if in `#connections`, -- `connecting` if pending connect or pending tool load, +- `connecting` if pending connect, pending tool load, or pending reconnect, - `disconnected` otherwise. ## Connection establishment and startup timing @@ -73,17 +76,21 @@ So startup does not fail the whole agent session when individual MCP servers fai For each discovered server in `connectServers()`: 1. store/update source metadata, -2. skip if already connected/pending, +2. skip if already connected/pending/reconnecting, 3. validate transport fields (`validateServerConfig`), 4. resolve auth/shell substitutions (`#resolveAuthConfig`), -5. call `connectToServer(name, resolvedConfig)`, -6. call `listTools(connection)`, -7. cache tool definitions (`MCPToolCache.set`) best-effort. +5. call `connectToServer(name, resolvedConfig)` with manager notification/request handlers, +6. wire HTTP OAuth refresh and transport `onClose` reconnect handling, +7. call `listTools(connection)`, +8. cache tool definitions (`MCPToolCache.set`) best-effort, +9. best-effort load resources, resource templates, prompts, and subscriptions after tools load. `connectToServer()` behavior (`src/mcp/client.ts`): - creates stdio or HTTP/SSE transport, -- performs MCP `initialize` + `notifications/initialized`, +- performs MCP `initialize`, +- for HTTP/SSE, starts the optional background SSE listener before `notifications/initialized`, +- sends `notifications/initialized`, - uses timeout (`config.timeout` or 30s default), - closes transport on init failure. @@ -118,14 +125,15 @@ Each pending `toolsPromise` also has a background continuation that eventually: `discoverAndLoadMCPTools()` converts manager tools into `LoadedCustomTool[]` and decorates paths (`mcp: via ` when known). -`createAgentSession()` then pushes these tools into `customTools`, which are wrapped and added to the runtime tool registry with names like `mcp__`. +`createAgentSession()` then pushes these tools into `customTools`, which are wrapped and added to the runtime tool registry with names like `mcp___`. ### Tool calls - `MCPTool` calls tools through an already connected `MCPServerConnection`. - `DeferredMCPTool` waits for `waitForConnection(server)` before calling; this allows cached tools to exist before connection is ready. +- Both attempt a reconnect + single retry for retriable connection failures. -Both return structured tool output and convert transport/tool errors into `MCP error: ...` tool content (abort remains abort). +Both return structured tool output and convert remaining transport/tool errors into `MCP error: ...` tool content (abort remains abort). ## Refresh/reload paths (startup vs live reload) @@ -142,24 +150,26 @@ Both return structured tool output and convert transport/tool errors into `MCP e 2. `mcpManager.discoverAndConnect()`, 3. `session.refreshMCPTools(mcpManager.getTools())`. -`session.refreshMCPTools()` (`src/session/agent-session.ts`) removes all `mcp_` tools, re-wraps latest MCP tools, and re-activates tool set so MCP changes apply without restarting session. +`session.refreshMCPTools()` (`src/session/agent-session.ts`) removes all `mcp__` tools, re-wraps latest MCP tools, and re-activates tool set so MCP changes apply without restarting session. There is also a follow-up path for late connections: after waiting for a specific server, if status becomes `connected`, it re-runs `session.refreshMCPTools(...)` so newly available tools are rebound in-session. ## Health, reconnect, and partial failure behavior -Current runtime behavior is intentionally minimal: +Current runtime behavior is connection-event driven: -- **No autonomous health monitor** in manager/client. -- **No automatic reconnect loop** when a transport drops. -- Manager does not subscribe to transport `onClose`/`onError`; status is registry-driven. -- Reconnect is explicit: reload flow or direct `connectServers()` invocation. +- **No autonomous polling health monitor** in manager/client. +- **Automatic reconnect is wired to `transport.onClose`** for managed connections. +- Reconnect retries with backoff (`500`, `1000`, `2000`, `4000` ms), reloads tools, and notifies consumers on success. +- Tool calls that see retriable connection errors also attempt one reconnect + retry. +- Reconnect is also explicit via `/mcp reconnect ` or broader `/mcp reload`. Operationally: - one server failing does not remove tools from healthy servers, - connect/list failures are isolated per server, -- tool cache and background updates are best-effort (warnings/errors logged, no hard stop). +- stale tools may remain visible while reconnect is attempted; calls report MCP errors if recovery fails, +- tool cache, resource/prompt loading, subscriptions, and background updates are best-effort (warnings/errors logged, no hard stop). ## Teardown semantics @@ -167,30 +177,31 @@ Operationally: `disconnectServer(name)`: -- removes pending entries/source metadata, +- removes pending entries, source metadata, saved config, resource refresh/subscription state, +- detaches `onClose` so explicit close does not trigger reconnect, - closes transport if connected, -- removes that server’s `mcp_` tools from manager state. +- filters manager tool state for names beginning with `mcp__${name}_`. ### Global teardown `disconnectAll()`: -- closes all active transports with `Promise.allSettled`, -- clears pending maps, sources, connections, and manager tool list. +- detaches `onClose` for all active transports, then closes them with `Promise.allSettled`, +- clears pending maps, sources, saved configs, connections, subscriptions, resource refreshes, and manager tool list. -In current wiring, explicit teardown is used in MCP command flows (for reload/remove/disable). There is no separate automatic manager disposal hook in the startup path itself; callers are responsible for invoking manager disconnect methods when they need deterministic MCP shutdown. +In current wiring, explicit teardown is used in MCP command flows (for reload/remove/disable). Startup stores the manager on the session; callers that need deterministic MCP shutdown should invoke manager disconnect methods. ## Failure modes and guarantees -| Scenario | Behavior | Hard fail vs best-effort | -| --- | --- | --- | -| Discovery throws (capability/config load path) | Loader returns empty tools + synthetic `.mcp.json` error | Best-effort session startup | -| Invalid server config | Server skipped with validation error entry | Best-effort per server | -| Connect timeout/init failure | Server error recorded; others continue | Best-effort per server | -| `tools/list` still pending at startup with cache hit | Deferred tools returned immediately | Best-effort fast startup | -| `tools/list` still pending at startup without cache | Startup waits for pending to settle | Hard wait for correctness | -| Late background tool-load failure | Logged after startup gate | Best-effort logging | -| Runtime dropped transport | No automatic reconnect; future calls fail until reconnect/reload | Best-effort recovery via manual action | +| Scenario | Behavior | Hard fail vs best-effort | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | +| Discovery throws (capability/config load path) | Loader returns empty tools + synthetic `.mcp.json` error | Best-effort session startup | +| Invalid server config | Server skipped with validation error entry | Best-effort per server | +| Connect timeout/init failure | Server error recorded; others continue | Best-effort per server | +| `tools/list` still pending at startup with cache hit | Deferred tools returned immediately | Best-effort fast startup | +| `tools/list` still pending at startup without cache | Startup waits for pending to settle | Hard wait for correctness | +| Late background tool-load failure | Logged after startup gate | Best-effort logging | +| Runtime dropped transport | Manager attempts reconnect; stale tools remain while reconnecting and future calls may retry once or fail with MCP errors | Best-effort automatic recovery | ## Public API surface diff --git a/docs/mcp-server-tool-authoring.md b/docs/mcp-server-tool-authoring.md index befe9d6de..4d5415ea1 100644 --- a/docs/mcp-server-tool-authoring.md +++ b/docs/mcp-server-tool-authoring.md @@ -1,6 +1,6 @@ # MCP server and tool authoring -This document explains how MCP server definitions become callable `mcp_*` tools in coding-agent, and what operators should expect when configs are invalid, duplicated, disabled, or auth-gated. +This document explains how MCP server definitions become callable `mcp__*` tools in coding-agent, and what operators should expect when configs are invalid, duplicated, disabled, or auth-gated. ## Architecture at a glance @@ -10,7 +10,8 @@ Config sources (.omp/.claude/.cursor/.vscode/mcp.json, mcp.json, etc.) -> capability loader dedupes by server name (higher provider priority wins) -> loadAllMCPConfigs converts to MCPServerConfig + skips enabled:false -> MCPManager connects/listTools (with auth/header/env resolution) - -> MCPTool/DeferredMCPTool bridge exposes tools as mcp__ + -> manager best-effort loads resources/prompts and subscribes to resource updates when enabled + -> MCPTool/DeferredMCPTool bridge exposes tools as mcp___ -> AgentSession.refreshMCPTools replaces live MCP tools immediately ``` @@ -21,7 +22,7 @@ Config sources (.omp/.claude/.cursor/.vscode/mcp.json, mcp.json, etc.) - `stdio` (default when `type` missing): requires `command`, optional `args`, `env`, `cwd` - `http`: requires `url`, optional `headers` - `sse`: requires `url`, optional `headers` (kept for compatibility) -- shared fields: `enabled`, `timeout`, `auth` +- shared fields: `enabled`, `timeout`, `auth`, `oauth` `validateServerConfig()` (`src/mcp/config.ts`) enforces transport basics: @@ -106,13 +107,13 @@ If credential lookup fails, manager logs a warning and continues with unresolved ### Header/env value resolution -Before connect, manager resolves each header/env value via `resolveConfigValue()` (`src/config/resolve-config-value.ts`): +Before connect, manager resolves stdio `env` values and HTTP/SSE `headers` values via `resolveConfigValue()` (`src/config/resolve-config-value.ts`): - value starting with `!` => execute shell command, use trimmed stdout (cached) +- failed, timed-out, or whitespace-only commands produce `undefined`, so that entry is omitted - otherwise, treat value as environment variable name first (`process.env[name]`), fallback to literal value -- unresolved command/env values are omitted from final headers/env map -Operational caveat: this means a mistyped secret command/env key can silently remove that header/env entry, producing downstream 401/403 or server startup failures. +Operational caveat: a mistyped `!` secret command can silently remove that header/env entry, producing downstream 401/403 or server startup failures. A mistyped environment variable name is sent literally unless that literal happens to be meaningful to the server. ## 4) Tool bridge: MCP -> agent-callable tools @@ -123,7 +124,7 @@ Operational caveat: this means a mistyped secret command/env key can silently re Tool names are generated as: ```text -mcp__ +mcp___ ``` Rules: @@ -137,7 +138,7 @@ This avoids many collisions, but not all. Different raw names can still sanitize ### Schema mapping -`convertSchema()` keeps MCP JSON Schema mostly as-is but patches object schemas missing `properties` with `{}` for provider compatibility. +`tool-bridge.ts` passes each MCP `inputSchema` through `sanitizeSchemaForMCP()` before registering it as a `CustomTool` schema. ### Execution mapping @@ -147,7 +148,8 @@ This avoids many collisions, but not all. Different raw names can still sanitize - flattens MCP content into displayable text - returns structured details (`serverName`, `mcpToolName`, provider metadata) - maps server-reported `isError` to `Error: ...` text result -- maps thrown transport/runtime failures to `MCP error: ...` +- attempts reconnect + one retry for retriable connection errors +- maps remaining thrown transport/runtime failures to `MCP error: ...` - preserves abort semantics by translating AbortError into `ToolAbortError` ## 5) Operator lifecycle: add/edit/remove and live updates @@ -161,7 +163,10 @@ Supported operations: - `enable` / `disable` - `test` - `reauth` / `unauth` +- `reconnect` - `reload` +- `resources`, `prompts`, `notifications` +- Smithery search/login/logout flows Config writes are atomic (`writeMCPConfigFile`: temp file + rename). @@ -171,7 +176,7 @@ After changes, controller calls `#reloadMCP()`: 2. `mcpManager.discoverAndConnect()` 3. `session.refreshMCPTools(mcpManager.getTools())` -`refreshMCPTools()` replaces all `mcp_` registry entries and immediately re-activates the latest MCP tool set, so changes take effect without restarting the session. +`refreshMCPTools()` replaces all `mcp__` registry entries and immediately re-activates the latest MCP tool set, so changes take effect without restarting the session. ### Mode differences @@ -206,7 +211,7 @@ Bad source JSON in discovery is generally handled as warnings/logs; config-write For robust MCP authoring in this codebase: 1. Keep server names globally unique across all MCP-capable config sources. -2. Prefer alphanumeric/underscore names to avoid sanitized-name collisions in generated `mcp_*` tool names. +2. Prefer names that remain distinct after MCP tool-name sanitization to avoid generated `mcp__` collisions. 3. Use explicit `type` to avoid accidental stdio defaults. 4. Treat `enabled: false` as hard-off: server is omitted from runtime connect set. 5. For OAuth configs, store a valid `credentialId`; otherwise auth injection is skipped. diff --git a/docs/memory.md b/docs/memory.md index fff5ec9e4..4b2bc64fc 100644 --- a/docs/memory.md +++ b/docs/memory.md @@ -23,23 +23,23 @@ At session start, if a memory summary exists for the current project, it is inje The agent can read memory files directly using `memory://` URLs with the `read` tool: -| URL | Content | -|---|---| -| `memory://root` | Compact summary injected at startup | -| `memory://root/MEMORY.md` | Full long-term memory document | -| `memory://root/skills//SKILL.md` | A generated skill playbook | +| URL | Content | +| -------------------------------------- | ----------------------------------- | +| `memory://root` | Compact summary injected at startup | +| `memory://root/MEMORY.md` | Full long-term memory document | +| `memory://root/skills//SKILL.md` | A generated skill playbook | ### `/memory` slash command -| Subcommand | Effect | -|---|---| -| `view` | Show the current memory injection payload | -| `clear` / `reset` | Delete all memory data and generated artifacts | -| `enqueue` / `rebuild` | Force consolidation to run at next startup | +| Subcommand | Effect | +| --------------------- | ---------------------------------------------- | +| `view` | Show the current memory injection payload | +| `clear` / `reset` | Delete all memory data and generated artifacts | +| `enqueue` / `rebuild` | Force consolidation to run at next startup | ## How it works -Memories are built by a background pipeline that at startup or manually triggered via slash command. +Memories are built by a background pipeline that runs at startup or when manually triggered via slash command. **Phase 1 — per-session extraction:** For each past session that has changed since it was last processed, a model reads the session history and extracts durable signal: technical decisions, constraints, resolved failures, recurring workflows. Sessions that are too recent, too old, or currently active are skipped. Each extraction produces a raw memory block and a short synopsis for that session. @@ -55,41 +55,41 @@ All output is scanned for secrets before being written to disk. ### Extraction behavior -Memory extraction and consolidation behavior is driven entirely by static prompt files in `src/prompts/memories/`. +Memory extraction and consolidation behavior is driven by static prompt files in `packages/coding-agent/src/prompts/memories/`. -| File | Purpose | Variables | -|---|---|---| -| `stage_one_system.md` | System prompt for per-session extraction | — | -| `stage_one_input.md` | User-turn template wrapping session content | `{{thread_id}}`, `{{response_items_json}}` | -| `consolidation.md` | Prompt for cross-session consolidation | `{{raw_memories}}`, `{{rollout_summaries}}` | -| `read_path.md` | Memory guidance injected into live sessions | `{{memory_summary}}` | +| File | Purpose | Variables | +| --------------------- | ------------------------------------------- | ------------------------------------------- | +| `stage_one_system.md` | System prompt for per-session extraction | — | +| `stage_one_input.md` | User-turn template wrapping session content | `{{thread_id}}`, `{{response_items_json}}` | +| `consolidation.md` | Prompt for cross-session consolidation | `{{raw_memories}}`, `{{rollout_summaries}}` | +| `read_path.md` | Memory guidance injected into live sessions | `{{memory_summary}}` | ### Model selection Memory piggybacks on the model role system. -| Phase | Role | Purpose | -|---|---|---| -| Phase 1 (extraction) | `default` | Per-session knowledge extraction | -| Phase 2 (consolidation) | `smol` | Cross-session synthesis | +| Phase | Role | Purpose | +| ----------------------- | ------------------------------------------------------------------- | -------------------------------- | +| Phase 1 (extraction) | `default` | Per-session knowledge extraction | +| Phase 2 (consolidation) | `smol` (falls back to `default`, then current/first registry model) | Cross-session synthesis | -If `smol` is not configured, Phase 2 falls back to the `default` role. +If the requested memory role is not configured, memory model resolution falls back to the `default` role, then the active session model, then the first model in the registry. ## Configuration -| Setting | Default | Description | -|---|---|---| -| `memories.enabled` | `false` | Master switch | -| `memories.maxRolloutAgeDays` | `30` | Sessions older than this are not processed | -| `memories.minRolloutIdleHours` | `12` | Sessions active more recently than this are skipped | -| `memories.maxRolloutsPerStartup` | `64` | Cap on sessions processed in a single startup | -| `memories.summaryInjectionTokenLimit` | `5000` | Max tokens of the summary injected into the system prompt | +| Setting | Default | Description | +| ------------------------------------- | ------- | --------------------------------------------------------- | +| `memories.enabled` | `false` | Master switch | +| `memories.maxRolloutAgeDays` | `30` | Sessions older than this are not processed | +| `memories.minRolloutIdleHours` | `12` | Sessions active more recently than this are skipped | +| `memories.maxRolloutsPerStartup` | `64` | Cap on sessions processed in a single startup | +| `memories.summaryInjectionTokenLimit` | `5000` | Max tokens of the summary injected into the system prompt | Additional tuning knobs (concurrency, lease durations, token budgets) are available in config for advanced use. ## Key files -- `src/memories/index.ts` — pipeline orchestration, injection, slash command handling -- `src/memories/storage.ts` — SQLite-backed job queue and thread registry -- `src/prompts/memories/` — memory prompt templates -- `src/internal-urls/memory-protocol.ts` — `memory://` URL handler +- `packages/coding-agent/src/memories/index.ts` — pipeline orchestration, injection, slash command handling +- `packages/coding-agent/src/memories/storage.ts` — SQLite-backed job queue and thread registry +- `packages/coding-agent/src/prompts/memories/` — memory prompt templates +- `packages/coding-agent/src/internal-urls/memory-protocol.ts` — `memory://` URL handler diff --git a/docs/models.md b/docs/models.md index 2393aaa8a..444328d8f 100644 --- a/docs/models.md +++ b/docs/models.md @@ -101,8 +101,8 @@ providers: ### Allowed auth/discovery values -- `auth`: `apiKey` (default) or `none` -- `discovery.type`: `ollama` +- `auth`: `apiKey` (default), `none`, or `oauth`; for `models.yml` custom models, `oauth` is accepted by schema but does not waive the `apiKey` requirement +- `discovery.type`: `ollama`, `llama.cpp`, or `lm-studio` ## Validation rules (current) @@ -119,6 +119,8 @@ Required: Must define at least one of: - `baseUrl` +- `headers` +- `compat` - `modelOverrides` - `discovery` @@ -142,7 +144,7 @@ ModelRegistry pipeline (on refresh): 5. Merge custom `models`: - same `provider + id` replaces existing - otherwise append -6. Apply runtime-discovered models (currently Ollama and LM Studio), then re-apply model overrides. +6. Load cached/runtime-discovered models (Ollama, llama.cpp, LM Studio, plus built-in provider managers), then re-apply model overrides. ## Canonical model equivalence and coalescing @@ -153,6 +155,7 @@ Canonical ids are official upstream ids only, for example: - `claude-opus-4-6` - `claude-haiku-4-5` - `gpt-5.3-codex` + ### `models.yml` equivalence config Example: @@ -223,23 +226,22 @@ Provider defaults vs per-model overrides: If `ollama` is not explicitly configured, registry adds an implicit discoverable provider: - provider: `ollama` -- api: `openai-completions` +- api: `openai-responses` - base URL: `OLLAMA_BASE_URL` or `http://127.0.0.1:11434` - auth mode: keyless (`auth: none` behavior) -Runtime discovery calls `GET /api/tags` on Ollama and synthesizes model entries with local defaults. +Runtime discovery calls Ollama endpoints and normalizes discovered OpenAI-compatible models to `openai-responses`. ### Implicit llama.cpp discovery If `llama.cpp` is not explicitly configured, registry adds an implicit discoverable provider: -Note: it's using the newer antropic messages api instead of the openai-competions. - provider: `llama.cpp` - api: `openai-responses` - base URL: `LLAMA_CPP_BASE_URL` or `http://127.0.0.1:8080` - auth mode: keyless (`auth: none` behavior) -Runtime discovery calls `GET models` on llama.cpp and synthesizes model entries with local defaults. +Runtime discovery calls llama.cpp model endpoints and synthesizes model entries with local defaults. ### Implicit LM Studio discovery @@ -260,11 +262,11 @@ You can configure discovery yourself: providers: ollama: baseUrl: http://127.0.0.1:11434 - api: openai-completions + api: openai-responses auth: none discovery: type: ollama - + llama.cpp: baseUrl: http://127.0.0.1:8080 api: openai-responses @@ -348,7 +350,7 @@ Resolution precedence for exact selectors: Supported model roles: -- `default`, `smol`, `slow`, `plan`, `commit` +- `default`, `smol`, `slow`, `vision`, `plan`, `designer`, `commit`, `task` Role aliases like `pi/smol` expand through `settings.modelRoles`. Each role value can also append a thinking selector such as `:minimal`, `:low`, `:medium`, or `:high`. @@ -499,11 +501,7 @@ providers: ## Legacy consumer caveat -Most model configuration now flows through `models.yml` via `ModelRegistry`. - -One notable legacy path remains: web-search Anthropic auth resolution still reads `~/.omp/agent/models.json` directly in `src/web/search/auth.ts`. - -If you rely on that specific path, keep JSON compatibility in mind until that module is migrated. +Most model configuration now flows through `models.yml` via `ModelRegistry`. Explicit `.json` / `.jsonc` paths remain supported only when passed programmatically to `ModelRegistry`; the default user config is `~/.omp/agent/models.yml`. ## Failure mode diff --git a/docs/natives-addon-loader-runtime.md b/docs/natives-addon-loader-runtime.md index db6cac031..5fc02a185 100644 --- a/docs/natives-addon-loader-runtime.md +++ b/docs/natives-addon-loader-runtime.md @@ -1,41 +1,46 @@ # Natives Addon Loader Runtime -This document deep-dives the addon loading/validation layer in `@oh-my-pi/pi-natives`: how `native.ts` decides which `.node` file to load, when embedded payload extraction runs, and how startup failures are reported. +This document covers the runtime loader shipped by `@oh-my-pi/pi-natives`: how `native/index.js` decides which `.node` file to require, how compiled-binary embedded payloads are extracted, and what startup failures report. ## Implementation files -- `packages/natives/src/native.ts` -- `packages/natives/src/embedded-addon.ts` -- `packages/natives/src/bindings.ts` +- `packages/natives/native/index.js` +- `packages/natives/native/loader-state.js` +- `packages/natives/native/embedded-addon.js` +- `packages/natives/scripts/embed-native.ts` - `packages/natives/package.json` ## Scope and responsibility -Loader/runtime responsibilities are intentionally narrow: +The loader is intentionally narrow: - Build a platform/CPU-aware candidate list for addon filenames and directories. +- Treat an embedded-addon manifest as the authoritative compiled-binary signal when present. - Optionally materialize an embedded addon into a versioned per-user cache directory. -- Attempt candidates in deterministic order. -- Reject stale or incompatible addons via `validateNative` before exposing bindings. +- Attempt candidates in deterministic order and return the first addon that `require(...)` loads. -Out of scope here: module-specific grep/text/highlight behavior. +The current loader does **not** run a separate `validateNative(...)` export-presence gate. API shape is provided by the generated N-API binding file (`native/index.d.ts`) and the loaded addon itself. A stale binary therefore normally fails as a missing property or native load error rather than as a custom "missing exports" validation error. ## Runtime inputs and derived state -At module initialization (`export const native = loadNative();`), `native.ts` computes static context: +At module initialization, `native/index.js` computes: -- **Platform tag**: ``${process.platform}-${process.arch}`` (for example `darwin-arm64`). -- **Package version**: from `packages/natives/package.json` (`version` field). +- **Platform tag**: `${process.platform}-${process.arch}` (for example `darwin-arm64`). +- **Package version**: from `packages/natives/package.json`. - **Core directories**: - `nativeDir`: package-local `packages/natives/native`. - `execDir`: directory containing `process.execPath`. - `versionedDir`: `/`. - `userDataDir` fallback: - - Windows: `%LOCALAPPDATA%/omp` (or `%USERPROFILE%/AppData/Local/omp`). + - Windows: `%LOCALAPPDATA%/omp` or `%USERPROFILE%/AppData/Local/omp`. - Non-Windows: `~/.local/bin`. -- **Compiled-binary mode** (`isCompiledBinary`): true if any of: - - `PI_COMPILED` env var is set, or - - `import.meta.url` contains Bun-embedded markers (`$bunfs`, `~BUN`, `%7EBUN`). +- **Natives cache root** (`getNativesDir()`): + - if `$XDG_DATA_HOME/omp` exists, `$XDG_DATA_HOME/omp/natives`; + - otherwise `~/.omp/natives`. +- **Compiled-binary mode** (`detectCompiledBinary`): true if any of: + - embedded-addon manifest is non-null, + - `PI_COMPILED` env var is set, + - `import.meta.url` contains Bun embedded markers (`$bunfs`, `~BUN`, `%7EBUN`). - **Variant override**: `PI_NATIVE_VARIANT` (`modern`/`baseline` only; invalid values ignored). - **Selected variant**: explicit override, otherwise runtime AVX2 detection on x64 (`modern` if AVX2, else `baseline`). @@ -49,201 +54,139 @@ At module initialization (`export const native = loadNative();`), `native.ts` co - `darwin-arm64` - `win32-x64` -Behavior detail: - -- Unsupported platforms are not rejected up-front. -- Loader still tries all computed candidates first. -- If nothing loads, it throws an explicit unsupported-platform error listing supported tags. - -This preserves useful diagnostics for near-miss cases while still failing hard for truly unsupported targets. +Unsupported platforms are not rejected before probing. The loader first tries the computed candidate paths. If all fail and `platformTag` is unsupported, it throws an unsupported-platform error listing supported tags. ## Variant selection (`modern` / `baseline` / default) ### x64 behavior -1. If `PI_NATIVE_VARIANT` is `modern` or `baseline`, that value wins. -2. Else detect AVX2 support: +1. `PI_NATIVE_VARIANT=modern|baseline` wins when valid. +2. Otherwise AVX2 support is detected: - Linux: scan `/proc/cpuinfo` for `avx2`. - - macOS: query `sysctl` (`machdep.cpu.leaf7_features`, fallback `machdep.cpu.features`). - - Windows: run PowerShell `[System.Runtime.Intrinsics.X86.Avx2]::IsSupported`. -3. Result: - - AVX2 available -> `modern` - - AVX2 unavailable/undetectable -> `baseline` + - macOS: `sysctl -n machdep.cpu.leaf7_features`, then `machdep.cpu.features`. + - Windows: PowerShell `[System.Runtime.Intrinsics.X86.Avx2]::IsSupported`. +3. AVX2 selects `modern`; unavailable or undetectable AVX2 selects `baseline`. ### Non-x64 behavior -- No variant is used; loader stays on the default filename (`pi_natives.-.node`). +No variant suffix is used; the filename is `pi_natives.-.node`. ### Filename construction -Given `tag = -`: +`loader-state.js#getAddonFilenames` returns: - Non-x64 or no variant: `pi_natives..node` -- x64 + `modern`: try in order +- x64 + `modern`: 1. `pi_natives.-modern.node` - 2. `pi_natives.-baseline.node` (intentional fallback) -- x64 + `baseline`: only `pi_natives.-baseline.node` + 2. `pi_natives.-baseline.node` + 3. `pi_natives..node` +- x64 + `baseline`: + 1. `pi_natives.-baseline.node` + 2. `pi_natives..node` -The `addonLabel` used in final error messages is either `` or ` ()`. +The default unsuffixed fallback remains part of the x64 candidate list. ## Candidate path construction and fallback ordering -`native.ts` builds candidate pools before any `require(...)` call. +`resolveLoaderCandidates(...)` expands every filename across directories, then de-duplicates while preserving first occurrence order. -### Release candidates +### Non-compiled runtime -Built from variant-resolved filename list and searched in this order: +For each filename, candidates are: -- **Non-compiled runtime**: - 1. `/` - 2. `/` +1. `/` +2. `/` -- **Compiled runtime** (`PI_COMPILED` or Bun embedded markers): - 1. `/` - 2. `/` - 3. `/` - 4. `/` +### Compiled runtime -`dedupedCandidates` removes duplicates while preserving first occurrence order. +For each filename, candidates are: -### Final runtime sequence +1. `/` +2. `/` +3. `/` +4. `/` -At load time: - -1. Optional embedded extraction candidate (if produced) is inserted at the front. -2. Remaining deduplicated candidates are tried in order. -3. First candidate that both `require(...)`s and passes `validateNative(...)` wins. +At load time, an extracted embedded candidate, when produced, is prepended ahead of these de-duplicated candidates. ## Embedded addon extraction lifecycle -`embedded-addon.ts` defines a generated manifest shape: +`embedded-addon.js` is generated by `scripts/embed-native.ts`. The reset stub exports `embeddedAddon = null`. A populated manifest has: - `platformTag` - `version` -- `files[]` where each entry has `variant`, `filename`, `filePath` +- `files[]` entries with `variant`, `filename`, and `filePath` -Current checked-in default is `embeddedAddon: null`; compiled artifacts may replace this with real metadata. +Extraction (`maybeExtractEmbeddedAddon`) runs only when: -### Extraction state machine +1. compiled-binary mode is true, +2. `embeddedAddon` is non-null, +3. manifest `platformTag` equals the runtime platform tag, +4. manifest `version` equals the package version, +5. a variant-appropriate embedded file exists. -Extraction (`maybeExtractEmbeddedAddon`) runs only when all gates pass: - -1. `isCompiledBinary === true` -2. `embeddedAddon !== null` -3. `embeddedAddon.platformTag === platformTag` -4. `embeddedAddon.version === packageVersion` -5. A variant-appropriate embedded file is found - -Variant file selection mirrors runtime variant intent: +Variant file selection: - Non-x64: prefer `default`, then first available file. - x64 + `modern`: prefer `modern`, fallback to `baseline`. - x64 + `baseline`: require `baseline`. -Materialization behavior: +Materialization: -1. Ensure `` exists (`mkdirSync(..., { recursive: true })`). -2. If `/` already exists, reuse it (no rewrite). -3. Else read embedded source `filePath` and write target file. -4. Return target path for highest-priority load attempt. +1. Ensure `` exists. +2. Reuse `/` if it already exists. +3. Otherwise read `selectedEmbeddedFile.filePath` and write the target path. +4. Return the target path as the first candidate. -On failure, extraction does not crash immediately; it appends an error entry (directory creation or write failure) and loader proceeds to normal candidate probing. +Directory creation or write failures are appended to the loader error list; probing continues through normal candidates. ## Lifecycle and state transitions ```text Init - -> Compute platform/version/variant/candidate lists - -> (Compiled + embedded manifest matches?) - yes -> Try extract embedded to versionedDir (record errors, continue) - no -> Skip extraction + -> Load package metadata and embedded-addon manifest + -> Compute platform/version/variant/filenames/candidate paths + -> (compiled + embedded manifest matches?) + yes -> try extract to versionedDir (record errors, continue) + no -> skip extraction -> For each runtime candidate in order: require(candidate) - -> success: validateNative - -> pass: return bindings (READY) - -> fail: record error, continue + -> success: return addon exports (READY) -> failure: record error, continue -> none loaded: if unsupported platform tag -> throw Unsupported platform - else -> throw Failed to load (full tried-path diagnostics + hints) + else -> throw Failed to load (tried-path diagnostics + hints) ``` -## `validateNative` contract checks - -`validateNative(bindings, source)` enforces a function-only contract over `NativeBindings` at startup. - -Mechanics: - -- For each required export name, it checks `typeof bindings[name] === "function"`. -- Missing names are aggregated. -- If any are missing, loader throws: - - source addon path, - - missing export list, - - rebuild command hint. - -This is a hard compatibility gate against stale binaries, partial builds, and symbol/name drift. - -### JS API ↔ native export mapping (validation gate) - -| JS binding name checked in `validateNative` | Expected native export name | -| --- | --- | -| `grep` | `grep` | -| `glob` | `glob` | -| `highlightCode` | `highlightCode` | -| `executeShell` | `executeShell` | -| `PtySession` | `PtySession` | -| `Shell` | `Shell` | -| `visibleWidth` | `visibleWidth` | -| `getSystemInfo` | `getSystemInfo` | -| `getWorkProfile` | `getWorkProfile` | -| `invalidateFsScanCache` | `invalidateFsScanCache` | - -Note: `bindings.ts` declares only the base `cancelWork(id)` member; module `types.ts` files declaration-merge additional symbols that `validateNative` enforces. - ## Failure behavior and diagnostics -## Unsupported platform +### Unsupported platform -If all candidates fail and `platformTag` is not in `SUPPORTED_PLATFORMS`, loader throws: +If all candidates fail and `platformTag` is not supported, the loader throws: - `Unsupported platform: ` -- Full supported-platform list -- Explicit issue-reporting guidance +- supported platform list +- issue-reporting guidance -## Stale binary / mismatch symptoms +### No loadable candidate -Typical stale mismatch signal: +If the platform is supported but no candidate can be loaded, the final error includes: -- `Native addon missing exports (). Missing: ...` +- `Failed to load pi_natives native addon for ` or ` ()` +- every attempted path with the corresponding `require(...)` error +- mode-specific remediation hints -Common causes: +### Compiled-binary startup failures -- Old `.node` binary from previous package version/API shape. -- Wrong variant artifact selected (for x64). -- New Rust export not present in loaded artifact. - -Loader behavior: - -- Records per-candidate missing-export failures. -- Continues probing remaining candidates. -- If no candidate validates, final error includes every attempted path with each failure message. - -## Compiled-binary startup failures - -In compiled mode final diagnostics include: +Compiled mode diagnostics include: - expected versioned cache target paths (`/`), -- remediation to delete stale `` and rerun, +- remediation to delete the versioned cache and rerun, - direct release download `curl` commands for each expected filename. -## Non-compiled startup failures +### Non-compiled startup failures -In normal package/runtime mode final diagnostics include: +Normal package/runtime diagnostics include: - reinstall hint (`bun install @oh-my-pi/pi-natives`), - local rebuild command (`bun --cwd=packages/natives run build`), -- optional x64 variant build hint (`TARGET_VARIANT=baseline|modern ...`). - -## Runtime behavior - -- The loader always uses the release candidate chain. \ No newline at end of file +- optional x64 variant build hint (`TARGET_VARIANT=baseline|modern bun --cwd=packages/natives run build`). diff --git a/docs/natives-architecture.md b/docs/natives-architecture.md index 229cdd49d..f8271da77 100644 --- a/docs/natives-architecture.md +++ b/docs/natives-architecture.md @@ -1,51 +1,48 @@ # Natives Architecture -`@oh-my-pi/pi-natives` is a three-layer stack: +`@oh-my-pi/pi-natives` is now a two-layer package around a loader: -1. **TypeScript wrapper/API layer** exposes stable JS/TS entrypoints. -2. **Addon loading/validation layer** resolves and validates the `.node` binary for the current runtime. -3. **Rust N-API module layer** implements performance-critical primitives exported to JS. +1. **CommonJS loader/package entrypoint** resolves and loads the correct `.node` addon and patches generated enum objects onto the export object. +2. **Rust N-API module layer** implements the exported functions/classes and emits the generated TypeScript declarations. This document is the foundation for deeper module-level docs. ## Implementation files -- `packages/natives/src/index.ts` -- `packages/natives/src/native.ts` -- `packages/natives/src/bindings.ts` -- `packages/natives/src/embedded-addon.ts` +- `packages/natives/native/index.js` +- `packages/natives/native/index.d.ts` +- `packages/natives/native/loader-state.js` +- `packages/natives/native/embedded-addon.js` - `packages/natives/scripts/build-native.ts` - `packages/natives/scripts/embed-native.ts` +- `packages/natives/scripts/gen-enums.ts` - `packages/natives/package.json` - `crates/pi-natives/src/lib.rs` -## Layer 1: TypeScript wrapper/API layer +## Package entrypoint and public surface -`packages/natives/src/index.ts` is the public barrel. It groups exports by capability domain and re-exports typed wrappers rather than exposing raw N-API bindings directly. +`packages/natives/package.json` points directly at generated native bindings: -Current top-level groups: +- `main`: `./native/index.js` +- `types`: `./native/index.d.ts` +- `exports["."].types`: `./native/index.d.ts` +- `exports["."].import`: `./native/index.js` -- **Search/text primitives**: `grep`, `glob`, `text`, `highlight` -- **Execution/process/terminal primitives**: `shell`, `pty`, `ps`, `keys` -- **System/media/conversion primitives**: `image`, `html`, `clipboard`, `system-info`, `work` +There is no current `packages/natives/src` TypeScript wrapper layer. Consumers import functions/classes/enums directly from `@oh-my-pi/pi-natives`; the type contract is the generated `native/index.d.ts` plus enum exports appended by `scripts/gen-enums.ts`. -`packages/natives/src/bindings.ts` defines the base interface contract: +Current capability groups in the generated API include: -- `NativeBindings` starts with shared members (`cancelWork(id: number)`) -- module-specific bindings are added by declaration merging from each module’s `types.ts` -- `Cancellable` standardizes timeout and abort-signal options for wrappers that expose cancellation +- **Search/text/code primitives**: `grep`, `search`, `hasMatch`, `fuzzyFind`, `glob`, `astGrep`, `astEdit`, text width/slicing/wrapping/sanitization, syntax highlighting, token counting. +- **Execution/process/terminal primitives**: `executeShell`, `Shell`, `PtySession`, process-tree helpers, key parsing. +- **System/media/conversion primitives**: clipboard, image resize/encode/SIXEL, HTML-to-Markdown, macOS appearance/power helpers, work profiling, Windows ProjFS overlay helpers. -**Guaranteed contract (API-facing):** consumers import from `@oh-my-pi/pi-natives` and use typed wrappers. +## Loader layer -**Implementation detail (may change):** declaration merging and internal wrapper layout (`src//index.ts`, `src//types.ts`). - -## Layer 2: Addon loading and validation - -`packages/natives/src/native.ts` owns runtime addon selection, optional extraction, and export validation. +`packages/natives/native/index.js` owns runtime addon selection and optional embedded extraction. ### Candidate resolution model -- Platform tag is `"${process.platform}-${process.arch}"`. +- Platform tag is `${process.platform}-${process.arch}`. - Supported tags are currently: - `linux-x64` - `linux-arm64` @@ -55,56 +52,56 @@ Current top-level groups: - x64 can use CPU variants: - `modern` (AVX2-capable) - `baseline` (fallback) -- Non-x64 uses the default filename (no variant suffix). +- Non-x64 uses the default filename without a variant suffix. Filename strategy: -- Release: `pi_natives.-.node` -- x64 variant release: `pi_natives.--modern.node` and/or `...-baseline.node` +- Default: `pi_natives.-.node` +- x64 variant: `pi_natives.--modern.node` or `...-baseline.node` +- x64 runtime fallback includes the unsuffixed default filename after variant candidates. ### Platform-specific variant detection For x64, variant selection uses: -- **Linux**: `/proc/cpuinfo` -- **macOS**: `sysctl machdep.cpu.leaf7_features` / `machdep.cpu.features` -- **Windows**: PowerShell check for `System.Runtime.Intrinsics.X86.Avx2` +- Linux: `/proc/cpuinfo` +- macOS: `sysctl -n machdep.cpu.leaf7_features`, then `machdep.cpu.features` +- Windows: PowerShell check for `System.Runtime.Intrinsics.X86.Avx2` -`PI_NATIVE_VARIANT` can explicitly force `modern` or `baseline`. +`PI_NATIVE_VARIANT` can force `modern` or `baseline`; invalid values are ignored. ### Binary distribution and extraction model -`packages/natives/package.json` includes both `src` and `native` in published files. The `native/` directory stores prebuilt platform artifacts. +`packages/natives/package.json` publishes `native/`, which contains the loader, generated declarations, generated enum patch, embedded-addon manifest stub, and prebuilt `.node` artifacts. -For compiled binaries (`PI_COMPILED` or Bun embedded runtime markers), loader behavior is: +For compiled binaries, loader behavior is: -1. Check versioned user cache path: `//...` +1. Check versioned user cache path: `//...`. 2. Check legacy compiled-binary location: - Windows: `%LOCALAPPDATA%/omp` (fallback `%USERPROFILE%/AppData/Local/omp`) - non-Windows: `~/.local/bin` -3. Fall back to packaged `native/` and executable directory candidates +3. Fall back to packaged `native/` and executable directory candidates. -If an embedded addon manifest is present (`embedded-addon.ts` generated by `scripts/embed-native.ts`), `native.ts` can materialize the matching embedded binary into the versioned cache directory before loading. +`getNativesDir()` uses `$XDG_DATA_HOME/omp/natives` when `$XDG_DATA_HOME/omp` exists; otherwise it uses `~/.omp/natives`. -### Validation and failure modes +If a populated embedded addon manifest is present, it is also treated as a compiled-binary signal. The loader can extract the matching embedded `.node` into the versioned cache directory before candidate probing. -After `require(candidate)`, `validateNative(...)` verifies required exports (for example `grep`, `glob`, `highlightCode`, `PtySession`, `Shell`, `getSystemInfo`, `getWorkProfile`, `invalidateFsScanCache`). +### Failure modes -Failure paths are explicit: +Loader failures are explicit: -- **Unsupported platform tag**: throws with supported platform list -- **No loadable candidate**: throws with all attempted paths and remediation hints -- **Missing exports**: throws with exact missing names and rebuild command -- **Embedded extraction errors**: records directory/write failures and includes them in final load diagnostics +- **Unsupported platform tag**: after failed probing, throws with supported platform list. +- **No loadable candidate**: throws with all attempted paths and remediation hints. +- **Embedded extraction errors**: directory/write failures are recorded and included in final load diagnostics if no candidate loads. -**Guaranteed contract (API-facing):** addon load either succeeds with a validated binding set or fails fast with actionable error text. +The current loader does not perform a separate post-`require` export validation pass. -**Implementation detail (may change):** exact candidate search order and compiled-binary fallback path ordering. +## Rust N-API module layer -## Layer 3: Rust N-API module layer - -`crates/pi-natives/src/lib.rs` is the Rust entry module that declares exported module ownership: +`crates/pi-natives/src/lib.rs` declares exported module ownership: +- `appearance` +- `ast` - `clipboard` - `fd` - `fs_cache` @@ -115,53 +112,49 @@ Failure paths are explicit: - `html` - `image` - `keys` +- `language` +- `power` - `prof` +- `projfs_overlay` - `ps` - `pty` - `shell` -- `system_info` - `task` - `text` +- `tokens` +- `utils` (crate-private helpers) -These modules implement the N-API symbols consumed and validated by `native.ts`. JS-level names are surfaced through the TS wrappers in `packages/natives/src`. - -**Guaranteed contract (API-facing):** Rust module exports must match the binding names expected by `validateNative` and wrapper modules. - -**Implementation detail (may change):** internal Rust module decomposition and helper module boundaries (`glob_util`, `task`, etc.). +N-API exports are generated from Rust `#[napi]` functions/classes/objects/enums. Snake_case Rust names are exposed as camelCase JavaScript names unless explicitly configured by napi-rs. ## Ownership boundaries -At architecture level, ownership is split as follows: - -- **TS wrapper/API ownership (`packages/natives/src`)** - - public API grouping, option typing, and stable JS ergonomics - - cancellation surface (`timeoutMs`, `AbortSignal`) exposed to callers -- **Loader ownership (`packages/natives/src/native.ts`)** +- **Loader/package ownership (`packages/natives/native`, `packages/natives/scripts`)** - runtime binary selection - CPU variant selection and override handling - - compiled-binary extraction and candidate probing - - hard validation of required native exports + - compiled-binary embedded extraction + - generated TypeScript declarations and enum export patching - **Rust ownership (`crates/pi-natives/src`)** - algorithmic and system-level implementation - platform-native behavior and performance-sensitive logic - - N-API symbol implementation that TS wrappers consume + - N-API symbol implementation consumed directly by package callers +- **Consumer ownership (`packages/coding-agent`, `packages/tui`)** + - user-facing policy and fallbacks that are not built into the native API + - higher-level rendering, artifact, shell-session, and command behavior ## Runtime flow (high level) 1. Consumer imports from `@oh-my-pi/pi-natives`. -2. Wrapper module calls into singleton `native` binding. -3. `native.ts` selects candidate binary for platform/arch/variant. -4. Optional embedded binary extraction occurs for compiled distributions. -5. Addon is loaded and export set is validated. -6. Wrapper returns typed results to caller. +2. `native/index.js` computes platform/arch/variant and candidate paths. +3. Optional embedded binary extraction occurs for compiled distributions. +4. The first `require(candidate)` that succeeds becomes the exported addon object. +5. Generated enum objects are appended to `module.exports`. +6. Caller invokes generated N-API functions/classes directly. ## Glossary - **Native addon**: A `.node` binary loaded via Node-API (N-API). - **Platform tag**: Runtime tuple `platform-arch` (for example `darwin-arm64`). - **Variant**: x64 CPU-specific build flavor (`modern` AVX2, `baseline` fallback). -- **Wrapper**: TS function/class that provides typed API over raw native exports. -- **Declaration merging**: TS technique used by module `types.ts` files to extend `NativeBindings`. -- **Compiled binary mode**: Runtime mode where the CLI is bundled and native addons are resolved from extracted/cache paths instead of only package-local paths. -- **Embedded addon**: Build artifact metadata and file references generated into `embedded-addon.ts` so compiled binaries can extract matching `.node` payloads. -- **Validation gate**: `validateNative(...)` check that rejects stale/mismatched binaries missing required exports. +- **Generated binding declaration**: `native/index.d.ts` emitted by napi-rs during `build-native.ts`. +- **Compiled binary mode**: Runtime mode where the CLI is bundled and native addons are resolved from embedded/cache paths before package-local paths. +- **Embedded addon**: Build artifact metadata and file references generated into `native/embedded-addon.js` so compiled binaries can extract matching `.node` payloads. diff --git a/docs/natives-binding-contract.md b/docs/natives-binding-contract.md index 6bad6e359..1821be5b8 100644 --- a/docs/natives-binding-contract.md +++ b/docs/natives-binding-contract.md @@ -1,221 +1,134 @@ -# Natives Binding Contract (TypeScript Side) +# Natives Binding Contract (JavaScript/TypeScript Side) -This document defines the TypeScript-side contract that sits between `@oh-my-pi/pi-natives` callers and the loaded N-API addon. +This document defines the JS/TS contract between `@oh-my-pi/pi-natives` callers and the loaded N-API addon. -It focuses on three pieces: - -1. contract shape (`NativeBindings` + module augmentation), -2. wrapper behavior (`src//index.ts`), -3. public export surface (`src/index.ts`). +Current package shape is direct-to-native: there is no `packages/natives/src/` TypeScript wrapper layer. The public API is the generated `packages/natives/native/index.d.ts` declaration file, the CommonJS loader in `packages/natives/native/index.js`, and the Rust `#[napi]` exports in `crates/pi-natives/src`. ## Implementation files -- `packages/natives/src/bindings.ts` -- `packages/natives/src/native.ts` -- `packages/natives/src/index.ts` -- `packages/natives/src/clipboard/types.ts` -- `packages/natives/src/clipboard/index.ts` -- `packages/natives/src/glob/types.ts` -- `packages/natives/src/glob/index.ts` -- `packages/natives/src/grep/types.ts` -- `packages/natives/src/grep/index.ts` -- `packages/natives/src/highlight/types.ts` -- `packages/natives/src/highlight/index.ts` -- `packages/natives/src/html/types.ts` -- `packages/natives/src/html/index.ts` -- `packages/natives/src/image/types.ts` -- `packages/natives/src/image/index.ts` -- `packages/natives/src/keys/types.ts` -- `packages/natives/src/keys/index.ts` -- `packages/natives/src/ps/types.ts` -- `packages/natives/src/ps/index.ts` -- `packages/natives/src/pty/types.ts` -- `packages/natives/src/pty/index.ts` -- `packages/natives/src/shell/types.ts` -- `packages/natives/src/shell/index.ts` -- `packages/natives/src/system-info/types.ts` -- `packages/natives/src/system-info/index.ts` -- `packages/natives/src/text/types.ts` -- `packages/natives/src/text/index.ts` -- `packages/natives/src/work/types.ts` -- `packages/natives/src/work/index.ts` +- `packages/natives/native/index.js` +- `packages/natives/native/index.d.ts` +- `packages/natives/native/loader-state.js` +- `packages/natives/scripts/build-native.ts` +- `packages/natives/scripts/gen-enums.ts` +- `packages/natives/package.json` +- `crates/pi-natives/src/lib.rs` +- Rust modules under `crates/pi-natives/src/*.rs` ## Contract model -`packages/natives/src/bindings.ts` defines the base contract: +The contract has three parts: -- `NativeBindings` (base interface, currently includes `cancelWork(id: number): void`) -- `Cancellable` (`timeoutMs?: number`, `signal?: AbortSignal`) -- `TsFunc` callback shape used by N-API threadsafe callbacks +1. **Generated runtime loader** (`native/index.js`) + - computes candidates and `require(...)`s the `.node` addon; + - exports the loaded addon object directly; + - appends enum objects generated by `scripts/gen-enums.ts`. +2. **Generated TypeScript declarations** (`native/index.d.ts`) + - generated by napi-rs during `scripts/build-native.ts`; + - declares exported functions, classes, object interfaces, and native enums; + - is the package `types` entry. +3. **Rust N-API exports** (`crates/pi-natives/src`) + - `#[napi]` functions/classes/objects/enums are the source of generated declarations and runtime symbols; + - snake_case Rust names become camelCase JavaScript names by napi-rs convention. -Each module adds its own fields by declaration merging: - -```ts -// packages/natives/src//types.ts -declare module "../bindings" { - interface NativeBindings { - grep(options: GrepOptions, onMatch?: TsFunc): Promise; - } -} -``` - -This keeps one aggregate binding interface without a monolithic central type file. - -## Declaration-merging lifecycle and state transitions - -### 1) Compile-time type assembly - -- `bindings.ts` provides the base `NativeBindings` symbol. -- Every `src//types.ts` augments `NativeBindings`. -- `src/native.ts` imports all `.//types` files for side effects so the merged contract is in scope where `NativeBindings` is used. - -State transition: **Base contract** → **Merged contract**. - -### 2) Runtime addon load and validation gate - -- `src/native.ts` loads candidate `.node` binaries. -- Loaded object is treated as `NativeBindings` and immediately passed through `validateNative(...)`. -- `validateNative` verifies required export keys by `typeof bindings[name] === "function"`. - -State transition: **Untrusted addon object** → **Validated native binding object** (or hard failure). - -### 3) Wrapper invocation - -- Module wrappers in `src//index.ts` call `native.`. -- Wrappers adapt defaults and callback shape (`(err, value)` to value-only callback patterns in JS APIs). -- `src/index.ts` re-exports module wrappers/types as the public package API. - -State transition: **Validated raw bindings** → **Ergonomic public API**. - -## Wrapper responsibilities - -Wrappers are intentionally thin; they do not re-implement native logic. - -Primary responsibilities: - -- **Argument normalization/defaulting** - - `glob()` resolves `options.path` to absolute path and defaults `hidden`, `gitignore`, `recursive`. - - `hasMatch()` fills default flags (`ignoreCase`, `multiline`) before native call. -- **Callback adaptation** - - `grep()`, `glob()`, `executeShell()` convert `TsFunc` (`error, value`) into user callback receiving only successful values. -- **Environment or policy behavior around native calls** - - Clipboard wrapper adds OSC52/Termux/headless handling and treats copy as best effort. -- **Public naming and re-export curation** - - `searchContent()` maps to native export `search`. +There is no current `NativeBindings` declaration-merging lifecycle and no `validateNative(...)` required-export list in the loader. ## Public export surface organization -`packages/natives/src/index.ts` is the canonical public barrel. It groups exports by capability domain: +`packages/natives/package.json` exposes the package root only: -- Search/text: `grep`, `glob`, `text`, `highlight` -- Execution/process/terminal: `shell`, `pty`, `ps`, `keys` -- System/media/conversion: `image`, `html`, `clipboard`, `system-info`, `work` +```json +{ + "main": "./native/index.js", + "types": "./native/index.d.ts", + "exports": { + ".": { + "types": "./native/index.d.ts", + "import": "./native/index.js" + } + } +} +``` -Maintainer rule: if a wrapper is not re-exported from `src/index.ts`, it is not part of the intended public package surface. +Consumers in `packages/coding-agent` and `packages/tui` import directly from `@oh-my-pi/pi-natives`. ## JS API ↔ native export mapping (representative) -The Rust side uses N-API export names (typically from `#[napi]` snake_case -> camelCase conversion, with occasional explicit aliases) that must match these binding keys. - -| Category | Public JS API (wrapper) | Native binding key | Return type | Async? | -|---|---|---|---|---| -| Grep | `grep(options, onMatch?)` | `grep` | `Promise` | Yes | -| Grep | `searchContent(content, options)` | `search` | `SearchResult` | No | -| Grep | `hasMatch(content, pattern, opts?)` | `hasMatch` | `boolean` | No | -| Grep | `fuzzyFind(options)` | `fuzzyFind` | `Promise` | Yes | -| Glob | `glob(options, onMatch?)` | `glob` | `Promise` | Yes | -| Glob | `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `void` | No | -| Shell | `executeShell(options, onChunk?)` | `executeShell` | `Promise` | Yes | -| Shell | `Shell` | `Shell` | class constructor | N/A | -| PTY | `PtySession` | `PtySession` | class constructor | N/A | -| Text | `truncateToWidth(...)` | `truncateToWidth` | `string` | No | -| Text | `sliceWithWidth(...)` | `sliceWithWidth` | `SliceWithWidthResult` | No | -| Text | `visibleWidth(text)` | `visibleWidth` | `number` | No | -| Highlight | `highlightCode(code, lang, colors)` | `highlightCode` | `string` | No | -| HTML | `htmlToMarkdown(html, options?)` | `htmlToMarkdown` | `Promise` | Yes | -| System | `getSystemInfo()` | `getSystemInfo` | `SystemInfo` | No | -| Work | `getWorkProfile(lastSeconds)` | `getWorkProfile` | `WorkProfile` | No | -| Process | `killTree(pid, signal)` | `killTree` | `number` | No | -| Process | `listDescendants(pid)` | `listDescendants` | `number[]` | No | -| Clipboard | `copyToClipboard(text)` | `copyToClipboard` | `Promise` (best effort wrapper behavior) | Yes | -| Clipboard | `readImageFromClipboard()` | `readImageFromClipboard` | `Promise` | Yes | -| Keys | `parseKey(data, kittyProtocolActive)` | `parseKey` | `string \| null` | No | +| Category | Public JS API | Rust source | Return style | +| ----------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ----------------------------- | +| Grep | `grep(options, onMatch?)` | `grep.rs` | `Promise` | +| Grep | `search(content, options)` | `grep.rs` | `SearchResult` | +| Grep | `hasMatch(content, pattern, ignoreCase?, multiline?)` | `grep.rs` | `boolean` | +| Fuzzy path search | `fuzzyFind(options)` | `fd.rs` | `Promise` | +| Glob | `glob(options, onMatch?)` | `glob.rs` | `Promise` | +| Glob cache | `invalidateFsScanCache(path?)` | `fs_cache.rs` | `void` | +| AST search/edit | `astGrep(options)`, `astEdit(options)` | `ast.rs` | `Promise<...>` | +| Shell | `executeShell(options, onChunk?)` | `shell.rs` | `Promise` | +| Shell | `new Shell(options?)`, `shell.run(...)`, `shell.abort()` | `shell.rs` | class / promises | +| PTY | `new PtySession()`, `start/write/resize/kill` | `pty.rs` | class / promises | +| Process | `killTree(pid, signal)`, `listDescendants(pid)` | `ps.rs` | sync | +| Keys | `parseKey`, `matchesKey`, Kitty/legacy helpers | `keys.rs` | sync | +| Text | `wrapTextWithAnsi`, `truncateToWidth`, `sliceWithWidth`, `extractSegments`, `sanitizeText`, `visibleWidth` | `text.rs` | sync | +| Highlight | `highlightCode`, `supportsLanguage`, `getSupportedLanguages` | `highlight.rs` | sync | +| HTML | `htmlToMarkdown(html, options?)` | `html.rs` | `Promise` | +| Image | `PhotonImage`, `encodeSixel` | `image.rs` | class / sync / promises | +| Clipboard | `copyToClipboard`, `readImageFromClipboard` | `clipboard.rs` | sync / promise | +| Tokens | `countTokens(input, encoding?)` | `tokens.rs` | sync | +| System | `detectMacOSAppearance`, `MacAppearanceObserver`, `MacOSPowerAssertion`, `getWorkProfile`, ProjFS helpers | `appearance.rs`, `power.rs`, `prof.rs`, `projfs_overlay.rs` | mixed | ## Sync vs async contract differences -The contract mixes sync and async APIs; wrappers preserve native call style rather than forcing one model: +The contract preserves Rust/N-API call style: -- **Promise-based async exports** for I/O or long-running work (`grep`, `glob`, `htmlToMarkdown`, `executeShell`, clipboard, image operations). -- **Synchronous exports** for deterministic in-memory transforms/parsers (`search`, `hasMatch`, highlighting, text width/slicing, key parsing, process queries). -- **Constructor exports** for stateful runtime objects (`Shell`, `PtySession`, `PhotonImage`). +- **Promise-returning exports** for worker-thread or async runtime work (`grep`, `glob`, `fuzzyFind`, `astGrep`, `astEdit`, `htmlToMarkdown`, shell/PTY runs, image parse/resize/encode, clipboard image read). +- **Synchronous exports** for deterministic in-memory transforms/parsers or direct system calls (`search`, `hasMatch`, highlighting, text utilities, token counting, process queries, `copyToClipboard`, `encodeSixel`). +- **Constructor exports** for stateful runtime objects (`Shell`, `PtySession`, `PhotonImage`, macOS observer/power handles). -Implication for maintainers: changing sync ↔ async for an existing export is a breaking API and contract change across wrappers and callers. +Changing sync ↔ async for an existing export is a breaking public API change because consumers call these exports directly. ## Object and enum typing patterns -### Object patterns (`#[napi(object)]`-style JS objects) +### Object patterns -TS models object-shaped native values as interfaces, for example: +`#[napi(object)]` Rust structs become TS interfaces, for example: -- `GrepResult`, `SearchResult`, `GlobResult` -- `SystemInfo`, `WorkProfile` -- `ClipboardImage`, `ParsedKittyResult` +- `GrepResult`, `SearchResult`, `GlobResult`, `FuzzyFindResult` +- `ShellRunResult`, `ShellExecuteResult`, `PtyRunResult`, `MinimizerResult` +- `AstFindResult`, `AstReplaceResult` +- `System`/media payloads such as `ClipboardImage`, `WorkProfile`, `ParsedKittyResult` -These are structural contracts at compile time; runtime shape correctness is owned by native implementation. +Runtime shape correctness is owned by napi-rs and the Rust implementation. ### Enum patterns -Numeric native enums are represented as `const enum` values in TS: +Native enums are represented in generated declarations and also appended to `module.exports` by `scripts/gen-enums.ts`, because the loader is hand-maintained CommonJS around the generated addon. Current enum objects include: -- `FileType` (`1=file`, `2=dir`, `3=symlink`) -- `ImageFormat` (`0=PNG`, `1=JPEG`, `2=WEBP`, `3=GIF`) -- `SamplingFilter`, `Ellipsis`, `KeyEventType` +- `AstMatchStrictness` +- `Ellipsis` +- `Encoding` +- `FileType` +- `GrepOutputMode` +- `ImageFormat` +- `KeyEventType` +- `MacOSAppearance` +- `SamplingFilter` -Callers see named enum members; the binding boundary passes numbers. +## Error behavior and caveats -## How mismatches are caught - -Mismatch detection happens at two layers: - -1. **Compile-time TypeScript contract checks** - - Wrappers call `native.` against merged `NativeBindings`. - - Missing/renamed binding keys break TS type-checking in wrappers. - -2. **Runtime validation in `validateNative`** - - After load, `native.ts` checks required exports and throws if any are missing. - - Error message includes missing keys and rebuild instruction. - -This catches the common stale-binary drift: wrapper/type exists but loaded `.node` lacks the export. - -## Failure behavior and caveats - -### Load/validation failures (hard failures) - -- Addon load failure or unsupported platform throws during module init in `native.ts`. -- Missing required exports throws before wrappers are usable. - -Effect: package fails fast rather than deferring failure to first call. - -### Wrapper-level behavior differences - -- Some wrappers intentionally soften failures (`copyToClipboard` is best effort and swallows native failure). -- Streaming callbacks ignore callback error payloads and only forward successful value events. - -### Type-level caveats (runtime stricter than TS) - -- TS optional fields do not guarantee semantic validity; native layer can still reject malformed values. -- `const enum` typing does not prevent out-of-range numeric values from untyped callers at runtime. -- `validateNative` checks only presence/function-ness of required exports, not deep argument/return-shape compatibility. -- `bindings.ts` includes `cancelWork(id)` in the base interface, but current runtime validation list does not enforce that key. +- Addon load failure or unsupported platform throws during package import from `native/index.js`. +- The loader does not verify the full export set after `require(...)`; stale or mismatched binaries surface as native load errors or missing members at use sites. +- N-API conversion validates basic argument conversion, but TS optional fields do not guarantee semantic validity for untyped callers. +- Numeric enum declarations do not prevent out-of-range numeric values from untyped callers unless the Rust function rejects them during conversion. +- Callback exports use napi-rs `ThreadsafeFunction` shape: `(error: Error | null, value) => void`. Native code generally emits successful values; hard failures reject/throw through the owning call. ## Maintainer checklist for binding changes When adding/changing an export, update all of: -1. `src//types.ts` (augmentation + contract types) -2. `src//index.ts` (wrapper behavior) -3. `src/native.ts` imports for the module types (if new module) -4. `validateNative` required export checks -5. `src/index.ts` public re-exports +1. Rust `#[napi]` implementation in the owning `crates/pi-natives/src/.rs`. +2. `crates/pi-natives/src/lib.rs` if a new module is added. +3. Any consumer imports/callsites in `packages/coding-agent` or `packages/tui`. +4. Build output by running the natives build so `native/index.d.ts` and `native/index.js` stay in sync. +5. `scripts/gen-enums.ts` if enum runtime export patching needs to change. -Skipping any step creates either compile-time drift or runtime load-time failure. \ No newline at end of file +Do not add a parallel TS wrapper convention unless the package design intentionally moves back to wrappers; current consumers depend on the direct generated API. diff --git a/docs/natives-build-release-debugging.md b/docs/natives-build-release-debugging.md index 75c0463fb..b7e16b080 100644 --- a/docs/natives-build-release-debugging.md +++ b/docs/natives-build-release-debugging.md @@ -1,18 +1,22 @@ # Natives Build, Release, and Debugging Runbook -This runbook describes how the `@oh-my-pi/pi-natives` build pipeline produces `.node` addons, how compiled distributions load them, and how to debug loader/build failures. +This runbook describes how `@oh-my-pi/pi-natives` produces `.node` addons, generated declarations, and compiled-binary embedded payloads, and how to debug loader/build failures. It follows the architecture terms from `docs/natives-architecture.md`: + - **build-time artifact production** (`scripts/build-native.ts`) - **embedded addon manifest generation** (`scripts/embed-native.ts`) -- **runtime addon loading + validation gate** (`src/native.ts`) +- **runtime addon loading** (`native/index.js`, `native/loader-state.js`) ## Implementation files - `packages/natives/scripts/build-native.ts` - `packages/natives/scripts/embed-native.ts` +- `packages/natives/scripts/gen-enums.ts` +- `packages/natives/scripts/zig-safe-wrapper.ts` - `packages/natives/package.json` -- `packages/natives/src/native.ts` +- `packages/natives/native/index.js` +- `packages/natives/native/loader-state.js` - `crates/pi-natives/Cargo.toml` ## Build pipeline overview @@ -21,37 +25,37 @@ It follows the architecture terms from `docs/natives-architecture.md`: `packages/natives/package.json` scripts: -- `bun scripts/build-native.ts` (`build`) → release build -- `bun scripts/embed-native.ts` (`embed:native`) → generate `src/embedded-addon.ts` from built files +- `bun scripts/build-native.ts` (`build`) → N-API build, addon install, generated declarations install, enum export patch. +- `bun scripts/embed-native.ts` (`embed:native`) → generate `native/embedded-addon.js` from built files. -### 2) Rust artifact build +Root scripts include `build:native` as `bun --cwd=packages/natives run build`. -`build-native.ts` runs Cargo in `crates/pi-natives`: +### 2) N-API/Rust artifact build -- base command: `cargo build --release` -- cross target adds `--target ` +`build-native.ts` invokes the `@napi-rs/cli` binary directly from `node_modules/.bin` with: -`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`, so Cargo emits a shared library (`.so`/`.dylib`/`.dll`) that is then copied/renamed to a `.node` addon filename. +- `napi build` +- `--manifest-path crates/pi-natives/Cargo.toml` +- `--package-json-path packages/natives/package.json` +- `--platform` +- `--no-js` +- `--dts index.d.ts` +- `--profile local` for non-CI local native builds, otherwise `--profile ci` +- optional `--target ` -### 3) Artifact discovery and install +`crates/pi-natives/Cargo.toml` declares `crate-type = ["cdylib"]`; napi-rs emits `.node` artifacts plus generated `index.d.ts` in an isolated temporary output directory under `packages/natives/native/.build/`. -After Cargo completes, `build-native.ts` scans candidate output directories in order: +### 3) Artifact install -1. `${CARGO_TARGET_DIR}` (if set) -2. `/target` -3. `crates/pi-natives/target` +After napi-rs succeeds, `build-native.ts`: -For each root it checks profile directories: -- cross build: `//` then `/` -- native build: `/` +1. resolves the built addon in the isolated output directory; +2. normalizes its name to `pi_natives.-(-variant).node` when needed; +3. installs the addon into `packages/natives/native/` with temp-file + rename semantics; +4. copies generated `index.js` and `index.d.ts` into `packages/natives/native/` when present; +5. runs `generateEnumExports()` to append enum runtime objects to `native/index.js`. -Then it looks for one of: -- `libpi_natives.so` -- `libpi_natives.dylib` -- `pi_natives.dll` -- `libpi_natives.dll` - -When found, it atomically installs into `packages/natives/native/` with temp-file + rename semantics (Windows fallback handles locked DLL replacement failures explicitly). +Windows locked-DLL replacement failures are reported with an explicit close-running-processes hint. ## Target/variant model and naming conventions @@ -59,75 +63,78 @@ When found, it atomically installs into `packages/natives/native/` with temp-fil Both build and runtime use platform tag: -`-` (example: `darwin-arm64`, `linux-x64`) +`-` (example: `darwin-arm64`, `linux-x64`). ## Variant model (x64 only) x64 supports CPU variants: + - `modern` (AVX2-capable path) - `baseline` (fallback) -Non-x64 uses a single default artifact (no variant suffix). +Non-x64 uses a single default artifact with no variant suffix. ### Output filenames -Release builds: - x64: `pi_natives.--modern.node` or `...-baseline.node` - non-x64: `pi_natives.-.node` -Runtime loader candidate order in `native.ts`: -- release candidates -- compiled mode prepends extracted/cache candidates before package-local files +Runtime x64 candidate order also includes the unsuffixed default filename after the selected variant candidates. ## Environment flags and build options ## Runtime flags -- `PI_NATIVE_VARIANT` (loader behavior, x64 only): force `modern` or `baseline` selection at runtime -- `PI_COMPILED` (loader behavior): enable compiled-binary candidate/extraction behavior +- `PI_NATIVE_VARIANT`: x64 runtime override; valid values are `modern` and `baseline`. +- `PI_COMPILED`: legacy compiled-mode signal. A populated embedded-addon manifest is also a compiled-mode signal and is the authoritative signal for Bun standalone builds that do not preserve `process.env.PI_COMPILED`. ## Build-time flags/options -- `CROSS_TARGET`: passed to Cargo `--target` -- `TARGET_PLATFORM`: override output platform tag naming -- `TARGET_ARCH`: override output arch naming -- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy -- `CARGO_TARGET_DIR`: additional root when searching Cargo outputs +- `CROSS_TARGET`: passed to napi-rs as `--target `. +- `TARGET_PLATFORM`: override output platform tag naming. +- `TARGET_ARCH`: override output arch naming. +- `TARGET_VARIANT` (x64 only): force `modern` or `baseline` for output filename and RUSTFLAGS policy. +- `CARGO_TARGET_DIR`: if set, respected; otherwise CI/cross builds use an isolated managed target directory under `target/napi-build/...`. - `RUSTFLAGS`: - if unset and not cross-compiling, script sets: - modern: `-C target-cpu=x86-64-v3` - baseline: `-C target-cpu=x86-64-v2` - non-x64 / no variant: `-C target-cpu=native` - - if already set, script does not override + - if already set, script does not override. +- `ZIG`: optional real Zig path used when the host Zig CPU contract wrapper is enabled. +- `PI_NATIVE_REAL_ZIG`, `PI_NATIVE_ZIG_TARGET`, `PI_NATIVE_ZIG_CPU`: set internally for `zig-safe-wrapper.ts` when building local x64 Linux/macOS artifacts with Zig available. ## Build state/lifecycle transitions ### Build lifecycle (`build-native.ts`) -1. **Init**: parse args/env (target overrides, cross flags) +1. **Init**: parse env, resolve target tuple, cross/local mode, profile label. 2. **Variant resolve**: - - non-x64 → no variant - - x64 + `TARGET_VARIANT` → explicit variant - - x64 cross-build without `TARGET_VARIANT` → hard error - - x64 local build without override → detect host AVX2 -3. **Compile**: run Cargo with resolved profile/target -4. **Locate artifact**: scan target roots/profile dirs/library names -5. **Install**: copy + atomic rename into `packages/natives/native` -6. **Complete**: output addon ready for loader candidates + - non-x64 → no variant; + - x64 + `TARGET_VARIANT` → explicit variant; + - x64 cross-build without `TARGET_VARIANT` → hard error; + - x64 local build without override → detect host AVX2. +3. **CPU policy**: set `RUSTFLAGS` if allowed; optionally route Zig through `zig-safe-wrapper.ts` to avoid leaking newer host CPU instructions into x64 artifacts. +4. **Compile**: run napi-rs against `crates/pi-natives` into an isolated output directory. +5. **Locate artifact**: accept the canonical filename or a single napi-rs-generated `pi_natives.-*.node` candidate. +6. **Install**: copy/rename addon into `packages/natives/native`. +7. **Install generated bindings**: copy `index.js`/`index.d.ts` if needed. +8. **Patch enums**: append generated enum runtime exports. +9. **Cleanup**: remove the temporary build output directory. -Failure exits happen at any stage with explicit error text (invalid variant, failed cargo build, missing output library, install/rename failure). +Failure exits have explicit error text for invalid variants, failed napi build, missing/multiple output artifacts, generated binding install failure, and install/rename failure. ### Embed lifecycle (`embed-native.ts`) -1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values +1. **Init**: compute platform tag from `TARGET_PLATFORM`/`TARGET_ARCH` or host values. 2. **Candidate set**: - - x64 expects both `modern` and `baseline` - - non-x64 expects one default file -3. **Validate availability** in `packages/natives/native` -4. **Generate manifest** (`src/embedded-addon.ts`) with Bun `file` imports and package version -5. **Runtime extraction ready** for compiled mode + - x64 looks for `modern` and `baseline` files; + - non-x64 looks for one default file. +3. **Validate availability**: at least one expected file must exist in `packages/natives/native`. +4. **Generate manifest** (`native/embedded-addon.js`) with Bun `file` imports and package version. +5. **Runtime extraction ready** for compiled mode. -`--reset` bypasses validation and writes a null manifest stub (`embeddedAddon = null`). +`--reset` writes the null manifest stub (`embeddedAddon = null`) without validating addon availability. ## Dev workflow vs shipped/compiled behavior @@ -135,75 +142,65 @@ Failure exits happen at any stage with explicit error text (invalid variant, fai Typical local loop: -1. Build addon: `bun --cwd=packages/natives run build` -2. Loader in `native.ts` resolves package-local `native/` (and executable-dir fallback) candidates -3. `validateNative` enforces export compatibility before wrappers use the binding +1. Build addon: `bun --cwd=packages/natives run build`. +2. Loader resolves package-local `native/` candidates, then executable-dir fallback candidates. +3. Generated declarations in `native/index.d.ts` describe the public TS API. ## Shipped/compiled binary workflow -In compiled mode (`PI_COMPILED` or Bun embedded markers): +In compiled mode (`PI_COMPILED`, Bun embedded URL markers, or populated embedded manifest): -1. Loader computes versioned cache dir: `/` (operationally `~/.omp/natives/`) -2. If embedded manifest matches current platform+version, loader may extract selected embedded file into that versioned dir +1. Loader computes versioned cache dir: `/`. +2. If embedded manifest matches current platform+version, loader may extract the selected embedded file into that versioned dir. 3. Runtime candidate order includes: - - versioned cache dir - - legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere) - - package/executable directories -4. First successfully loaded addon still must pass `validateNative` + - versioned cache dir, + - legacy compiled-binary dir (`%LOCALAPPDATA%/omp` on Windows, `~/.local/bin` elsewhere), + - package/executable directories. +4. First successfully loaded addon is returned. -This is why packaging + runtime loader expectations must align: filenames, platform tags, and exported symbols must match what `native.ts` probes and validates. +This is why packaging + runtime loader expectations must align: filenames, platform tags, CPU variants, and embedded manifest version must match what `native/index.js` probes. -## JS API ↔ Rust export mapping (validation gate subset) +## JS API ↔ Rust export mapping (build sanity subset) -`native.ts` requires these JS-visible exports to exist on the loaded addon. They map to Rust N-API exports in `crates/pi-natives/src`: +Generated declarations currently include exports from these Rust modules: -| JS name required by `validateNative` | Rust export declaration | Rust source file | -| --- | --- | --- | -| `glob` | `#[napi] pub fn glob(...)` | `crates/pi-natives/src/glob.rs` | -| `grep` | `#[napi] pub fn grep(...)` | `crates/pi-natives/src/grep.rs` | -| `search` | `#[napi] pub fn search(...)` | `crates/pi-natives/src/grep.rs` | -| `highlightCode` | `#[napi] pub fn highlight_code(...)` | `crates/pi-natives/src/highlight.rs` | -| `getSystemInfo` | `#[napi] pub fn get_system_info(...)` | `crates/pi-natives/src/system_info.rs` | -| `getWorkProfile` | `#[napi] pub fn get_work_profile(...)` (camel-cased export) | `crates/pi-natives/src/prof.rs` | -| `invalidateFsScanCache` | `#[napi] pub fn invalidate_fs_scan_cache(...)` | `crates/pi-natives/src/fs_cache.rs` | - -If any required symbol is missing, loader fails fast with a rebuild hint. +| Area | Representative JS exports | Rust source | +| ---------------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| Search | `grep`, `search`, `hasMatch`, `fuzzyFind`, `glob`, `invalidateFsScanCache` | `grep.rs`, `fd.rs`, `glob.rs`, `fs_cache.rs` | +| AST | `astGrep`, `astEdit` | `ast.rs` | +| Text/highlight/tokens | `visibleWidth`, `truncateToWidth`, `highlightCode`, `countTokens` | `text.rs`, `highlight.rs`, `tokens.rs` | +| Shell/PTY/process/keys | `executeShell`, `Shell`, `PtySession`, `killTree`, `parseKey` | `shell.rs`, `pty.rs`, `ps.rs`, `keys.rs` | +| Media/system | `PhotonImage`, `encodeSixel`, clipboard, macOS appearance/power, `getWorkProfile`, ProjFS helpers | `image.rs`, `clipboard.rs`, `appearance.rs`, `power.rs`, `prof.rs`, `projfs_overlay.rs` | ## Failure behavior and diagnostics ## Build-time failures - Invalid variant configuration: - - `TARGET_VARIANT` set on non-x64 → immediate error - - x64 cross-build without explicit `TARGET_VARIANT` → immediate error -- Cargo build failure: - - script surfaces non-zero exit and stderr -- Artifact not found: - - script prints every checked profile directory -- Install failure: - - explicit message; Windows includes locked-file hint + - `TARGET_VARIANT` set on non-x64 → immediate error. + - unsupported `TARGET_VARIANT` value → immediate error. + - x64 cross-build without explicit `TARGET_VARIANT` → immediate error. +- napi-rs build failure: script surfaces non-zero exit and stderr. +- Artifact not found or ambiguous: script prints expected/candidate filenames and output directory contents. +- Install failure: explicit message; Windows includes locked-file hint. +- Generated binding install failure: explicit source/destination message. -## Runtime loader failures (`native.ts`) +## Runtime loader failures (`native/index.js`) -- Unsupported platform tag: - - throws with supported platform list -- No candidate could load: - - throws with full candidate error list and mode-specific remediation hints -- Missing exports: - - throws with exact missing symbol names and rebuild command -- Embedded extraction problems: - - extraction mkdir/write errors recorded and included in final diagnostics +- Unsupported platform tag: throws with supported platform list after probing fails. +- No candidate could load: throws with full candidate error list and mode-specific remediation hints. +- Embedded extraction problems: extraction mkdir/write errors are recorded and included in final diagnostics if load fails. ## Troubleshooting matrix -| Symptom | Likely cause | Verify | Fix | -| --- | --- | --- | --- | -| `Native addon missing exports ... Missing: ` | Stale `.node` binary, Rust export name mismatch, or wrong binary loaded | Inspect export list for the binary | Rebuild `build`; ensure Rust `#[napi]` export name (or explicit alias when needed) matches JS key; remove stale cached/versioned files | -| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or only baseline file present | Check `PI_NATIVE_VARIANT`; inspect `native/` for `-modern` file | Build modern variant (`TARGET_VARIANT=modern ... build`) and ensure file is shipped | -| Cross-build produces unusable/wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing `TARGET_VARIANT` for x64 | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` | -| Compiled binary fails after upgrade | Stale extracted cache (`~/.omp/natives/`) or embedded manifest mismatch | Inspect versioned natives dir and loader error list | Delete versioned natives cache for the package version and rerun; regenerate embedded manifest during packaging | -| Loader probes many paths and none work | Platform mismatch or missing release artifact in package `native/` | Check `platformTag` vs actual filename(s) | Ensure built filename exactly matches `pi_natives.-(-variant).node` convention and package includes `native/` | -| `embed:native` fails with "Incomplete native addons" | Required variant files not built before embedding | Check expected vs found list in error text | Build required files first (x64: both modern+baseline; non-x64: default), then rerun `embed:native` | +| Symptom | Likely cause | Verify | Fix | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `Cannot find module` or dynamic library load error for every candidate | Missing release artifact, wrong platform tag, or stale compiled cache | Inspect loader error list and `packages/natives/native` filenames | Build correct target/variant; delete stale cache for the package version | +| Export is missing at runtime but present in TypeScript | Stale `.node` loaded, generated declarations newer than binary, or Rust export not compiled | Require the actual candidate and inspect `Object.keys(mod)` | Rebuild native package and remove stale candidate/cache paths | +| x64 machine loads baseline when modern expected | `PI_NATIVE_VARIANT=baseline`, no AVX2 detected, or modern file unavailable | Check env and filenames in `native/` | Build modern variant (`TARGET_VARIANT=modern ... build`) and ship it | +| Cross-build produces wrong-labeled binary | Mismatch between `CROSS_TARGET` and `TARGET_PLATFORM`/`TARGET_ARCH`, or missing x64 variant | Confirm env tuple and output filename | Re-run with consistent env values and explicit x64 `TARGET_VARIANT` | +| Compiled binary fails after upgrade | Stale extracted cache or embedded manifest version mismatch | Inspect `/` and loader error list | Delete versioned cache for the package version; regenerate embedded manifest during packaging | +| `embed:native` fails with `No native addons found` | Required platform artifact was not built before embedding | Check expected list in error text | Build at least one expected artifact for the target, then rerun `embed:native` | ## Operational commands @@ -220,4 +217,4 @@ bun --cwd=packages/natives run embed:native # Reset embedded manifest to null stub bun --cwd=packages/natives run embed:native -- --reset -``` \ No newline at end of file +``` diff --git a/docs/natives-media-system-utils.md b/docs/natives-media-system-utils.md index 87ed0e79b..8a07b75f9 100644 --- a/docs/natives-media-system-utils.md +++ b/docs/natives-media-system-utils.md @@ -1,87 +1,113 @@ # Natives media + system utilities -This document is a subsystem deep-dive for the **system/media/conversion primitives** layer described in [`docs/natives-architecture.md`](./natives-architecture.md): `image`, `html`, `clipboard`, and `work` profiling. +This document covers the media/system/conversion exports in `@oh-my-pi/pi-natives`: image processing, HTML conversion, clipboard access, token counting, macOS appearance/power helpers, ProjFS helpers, and work profiling. ## Implementation files - `crates/pi-natives/src/image.rs` - `crates/pi-natives/src/html.rs` - `crates/pi-natives/src/clipboard.rs` +- `crates/pi-natives/src/tokens.rs` +- `crates/pi-natives/src/appearance.rs` +- `crates/pi-natives/src/power.rs` +- `crates/pi-natives/src/projfs_overlay.rs` - `crates/pi-natives/src/prof.rs` - `crates/pi-natives/src/task.rs` -- `packages/natives/src/image/index.ts` -- `packages/natives/src/image/types.ts` -- `packages/natives/src/html/index.ts` -- `packages/natives/src/html/types.ts` -- `packages/natives/src/clipboard/index.ts` -- `packages/natives/src/clipboard/types.ts` -- `packages/natives/src/work/index.ts` -- `packages/natives/src/work/types.ts` +- `packages/natives/native/index.d.ts` > Note: there is no `crates/pi-natives/src/work.rs`; work profiling is implemented in `prof.rs` and fed by instrumentation in `task.rs`. -## TS API ↔ Rust export/module mapping +## JS API ↔ Rust export/module mapping -| TS export (packages/natives) | Rust N-API export | Rust module | -| ------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------- | -| `PhotonImage.parse(bytes)` | `PhotonImage::parse` | `image.rs` | -| `PhotonImage#resize(width, height, filter)` | `PhotonImage::resize` | `image.rs` | -| `PhotonImage#encode(format, quality)` | `PhotonImage::encode` | `image.rs` | -| `htmlToMarkdown(html, options)` | `html_to_markdown` | `html.rs` | -| `copyToClipboard(text)` | `copy_to_clipboard` + TS fallback logic | `clipboard.rs` + `clipboard/index.ts` | -| `readImageFromClipboard()` | `read_image_from_clipboard` | `clipboard.rs` | -| `getWorkProfile(lastSeconds)` | `get_work_profile` | `prof.rs` | +| JS export | Rust N-API export | Rust module | +| --------------------------------------------------- | ------------------------------ | ------------------- | +| `PhotonImage.parse(bytes)` | `PhotonImage::parse` | `image.rs` | +| `PhotonImage#resize(width, height, filter)` | `PhotonImage::resize` | `image.rs` | +| `PhotonImage#encode(format, quality)` | `PhotonImage::encode` | `image.rs` | +| `encodeSixel(bytes, targetWidthPx, targetHeightPx)` | `encode_sixel` | `image.rs` | +| `htmlToMarkdown(html, options?)` | `html_to_markdown` | `html.rs` | +| `copyToClipboard(text)` | `copy_to_clipboard` | `clipboard.rs` | +| `readImageFromClipboard()` | `read_image_from_clipboard` | `clipboard.rs` | +| `countTokens(input, encoding?)` | `count_tokens` | `tokens.rs` | +| `detectMacOSAppearance()` | `detect_mac_os_appearance` | `appearance.rs` | +| `MacAppearanceObserver.start(callback)` | `MacAppearanceObserver::start` | `appearance.rs` | +| `MacOSPowerAssertion.start(options?)` | `MacOSPowerAssertion::start` | `power.rs` | +| `projfsOverlayProbe/start/stop` | ProjFS exports | `projfs_overlay.rs` | +| `getWorkProfile(lastSeconds)` | `get_work_profile` | `prof.rs` | ## Data format boundaries and conversions ### Image (`image`) -- **JS input boundary**: `Uint8Array` encoded image bytes. -- **Rust decode boundary**: bytes are copied to `Vec`, format is guessed with `ImageReader::with_guessed_format()`, then decoded to `DynamicImage`. +- **JS input boundary**: `Uint8Array` encoded image bytes for `PhotonImage.parse` and `encodeSixel`. +- **Rust decode boundary**: bytes are copied/read, format is guessed with `ImageReader::with_guessed_format()`, then decoded to `DynamicImage`. - **In-memory state**: `PhotonImage` stores `Arc`. -- **Output boundary**: `encode(format, quality)` returns `Promise` (Rust `Vec`). +- **Output boundary**: + - `PhotonImage#encode(format, quality)` returns a promise for encoded bytes (`Vec` in Rust; generated TS currently declares `Promise>`). + - `encodeSixel(...)` returns a SIXEL escape string synchronously. -Format IDs are numeric: +Format IDs: - `0`: PNG - `1`: JPEG -- `2`: WebP (lossless encoder) +- `2`: WebP - `3`: GIF -Constraints: +Encoding behavior: -- `quality` is only used for JPEG. -- PNG/WebP/GIF ignore `quality`. -- Unsupported format IDs fail (`Invalid image format: `). +- JPEG uses the provided `quality` with `JpegEncoder::new_with_quality`. +- WebP uses the `webp` crate encoder with `quality` as `f32` in the same 0..=100 range. +- PNG/GIF ignore `quality`. +- Invalid dimensions for SIXEL (`0` width or height) fail with `Target SIXEL dimensions must be greater than zero`. ### HTML conversion (`html`) -- **JS input boundary**: HTML `string` + optional object `{ cleanContent?: boolean; skipImages?: boolean }`. -- **Rust conversion boundary**: `String` input is converted by `html_to_markdown_rs::convert`. -- **Output boundary**: Markdown `string`. +- **JS input boundary**: HTML `string` + optional `{ cleanContent?: boolean; skipImages?: boolean }`. +- **Rust conversion boundary**: conversion is scheduled through `task::blocking("html_to_markdown", (), ...)`. +- **Output boundary**: Markdown `string` promise. Conversion behavior: - `cleanContent` defaults to `false`. -- When `cleanContent=true`, preprocessing is enabled with `PreprocessingPreset::Aggressive` and hard-removal flags for navigation/forms. +- When `cleanContent=true`, preprocessing uses `PreprocessingPreset::Aggressive` and hard-removal flags for navigation/forms. - `skipImages` defaults to `false`. ### Clipboard (`clipboard`) -- **Text path**: - - TS first emits OSC 52 (`\x1b]52;c;\x07`) when stdout is a TTY. - - Same text is then attempted via native clipboard API (`native.copyToClipboard`) as best-effort. - - On Termux, TS attempts `termux-clipboard-set` first. -- **Image read path**: - - Rust reads raw image from `arboard`. - - Rust re-encodes it to PNG bytes (`image` crate), returns `{ data: Uint8Array, mimeType: "image/png" }`. - - TS returns `null` early on Termux or Linux sessions without display server (`DISPLAY`/`WAYLAND_DISPLAY` missing). +- `copyToClipboard(text)` is a synchronous native call using `arboard::Clipboard::set_text`. +- `readImageFromClipboard()` runs in `task::blocking("clipboard.read_image", (), ...)`. +- Image read returns `null`/`undefined` when `arboard` reports `ContentNotAvailable`. +- Successful image read re-encodes clipboard RGBA data as PNG and returns `{ data: Uint8Array, mimeType: "image/png" }`. +- Clipboard access or image encoding failures reject/throw as native errors. + +There is no current `packages/natives` TS wrapper that emits OSC52, handles Termux, or suppresses native clipboard failures. Any best-effort clipboard policy must live in consumers. + +### Tokens (`tokens`) + +- `countTokens(input, encoding?)` accepts a single string or an array of strings. +- Arrays return one aggregate token count; encoding work is parallelized in Rust. +- Default encoding is `O200kBase`; `Cl100kBase` is also exported. +- The implementation uses ordinary encoding, not special-token handling. + +### macOS appearance and power helpers + +- `detectMacOSAppearance()` returns `"dark"`, `"light"`, or `null` on non-macOS. +- `MacAppearanceObserver.start(callback)` returns a handle with `stop()`; on macOS it uses distributed notifications plus a 2-second polling fallback, and on non-macOS it is a no-op observer. +- `MacOSPowerAssertion.start(options?)` returns a handle with `stop()`; on macOS it acquires an IOKit assertion, and on other platforms it is a no-op handle. + +### Windows ProjFS helpers + +- `projfsOverlayProbe()` reports whether ProjFS APIs are available. +- `projfsOverlayStart(lowerRoot, projectionRoot)` starts an overlay. +- `projfsOverlayStop(projectionRoot)` stops an overlay session. + +These helpers are platform-specific; availability must be checked before relying on overlay behavior. ### Work profiling (`work`) - **Collection boundary**: profiling samples are produced by `profile_region(tag)` guards in `task::blocking` and `task::future`. -- **Storage format**: fixed-size circular buffer (`MAX_SAMPLES = 10_000`) storing stack path + duration (`μs`) + timestamp (`μs since process start`). -- **Output boundary**: `getWorkProfile(lastSeconds)` returns object: +- **Storage format**: fixed-size circular buffer (`MAX_SAMPLES = 10_000`) storing stack path, duration, and timestamp. +- **Output boundary**: `getWorkProfile(lastSeconds)` returns: - `folded`: folded-stack text (flamegraph input) - `summary`: markdown table summary - `svg`: optional flamegraph SVG @@ -93,14 +119,15 @@ Conversion behavior: 1. `PhotonImage.parse(bytes)` schedules a blocking decode task (`image.decode`). 2. On success, a native `PhotonImage` handle exists in JS. -3. `resize(...)` creates a new native handle (`image.resize`), old and new handles can coexist. -4. `encode(...)` materializes bytes (`image.encode`) without mutating image dimensions. +3. `resize(...)` creates a new native handle (`image.resize`); old and new handles can coexist. +4. `encode(...)` schedules `image.encode` and materializes bytes without mutating image dimensions. +5. `encodeSixel(...)` decodes, optionally resizes to exact target dimensions with Lanczos3, and returns SIXEL text synchronously. Failure transitions: -- Format detection/decode failure rejects parse promise. +- Format detection/decode failure rejects parse promise or throws from SIXEL encoding. - Encode failure rejects encode promise. -- Invalid format ID rejects encode promise. +- Invalid SIXEL dimensions throw. ### HTML lifecycle @@ -108,63 +135,49 @@ Failure transitions: 2. Conversion runs with defaulted options (`cleanContent=false`, `skipImages=false`) unless specified. 3. Returns markdown string or rejects. -Failure transitions: - -- Converter failure returns rejected promise (`Conversion error: ...`). - ### Clipboard lifecycle -`copyToClipboard(text)` is intentionally best-effort and multi-path: - -1. If TTY: attempt OSC 52 write (base64 payload). -2. Try Termux command when `TERMUX_VERSION` is set. -3. Try native `arboard` text copy. -4. Swallow errors at TS layer. - -`readImageFromClipboard()` strictness differs by stage: - -1. TS hard-gates unsupported runtime contexts (Termux/headless Linux) to `null`. -2. Rust `arboard` read runs only when TS allows it. -3. `ContentNotAvailable` maps to `null`. -4. Other Rust errors reject. +- Text copy constructs an `arboard::Clipboard` and calls `set_text` synchronously. +- Image read constructs an `arboard::Clipboard`, calls `get_image`, encodes PNG on success, maps `ContentNotAvailable` to `None`, and rejects other errors. ### Work profiling lifecycle -1. No explicit start: profiling is always on when task helpers execute. +1. No explicit start: profiling is active when task helpers execute. 2. Every instrumented task scope records one sample on guard drop. 3. Samples overwrite oldest entries after buffer capacity is reached. 4. `getWorkProfile(lastSeconds)` reads a time window and derives folded/summary/svg artifacts. Failure transitions: -- SVG generation failure is soft-fail (`svg: null`), while folded and summary still return. -- Empty sample window returns empty folded data and `svg: null`, not an error. +- SVG generation failure is soft (`svg` omitted/undefined), while folded and summary still return. +- Empty sample windows return empty folded data and no SVG, not an error. ## Unsupported operations and error propagation ### Image -- Unsupported decode input or corrupted bytes: strict failure (promise rejection). -- Unsupported encode format ID: strict failure. -- No best-effort fallback path in TS wrapper. +- Unsupported decode input or corrupted bytes: strict failure. +- Invalid SIXEL target dimensions: strict failure. +- No JS fallback path in the natives package. ### HTML -- Conversion errors are strict failures (rejection). -- Option omission is best-effort defaulting, not failure. +- Conversion errors are strict failures. +- Option omission is defaulting, not failure. ### Clipboard -- Text copy is best-effort at TS layer: operational failures are suppressed. -- Image read distinguishes "no image" (`null`) from operational failure (rejection). -- Termux/headless Linux are treated as unsupported contexts for image read (`null`). +- Text copy is strict at the native API surface. +- Image read distinguishes "no image" (`null`/`undefined`) from operational failure (rejection). ### Work profiling -- Retrieval is strict for function call itself, but artifact generation is partially best-effort (`svg` nullable). -- Buffer truncation is expected behavior (ring buffer), not data loss bug. +- Retrieval is strict for the function call itself. +- Flamegraph SVG generation is nullable/optional. +- Buffer truncation is expected ring-buffer behavior. ## Platform caveats -- **Clipboard text**: OSC 52 depends on terminal support; native clipboard access depends on desktop environment/session. -- **Clipboard image read**: blocked in TS for Termux and Linux without display server. +- Clipboard access depends on OS/session support exposed through `arboard`. +- macOS appearance and power helpers intentionally return no-op/null behavior on unsupported platforms. +- ProjFS helpers are Windows-specific and should be gated by `projfsOverlayProbe()`. diff --git a/docs/natives-rust-task-cancellation.md b/docs/natives-rust-task-cancellation.md index aac660700..4570d03c2 100644 --- a/docs/natives-rust-task-cancellation.md +++ b/docs/natives-rust-task-cancellation.md @@ -1,6 +1,6 @@ # Native Rust task execution and cancellation (`pi-natives`) -This document describes how `crates/pi-natives` schedules native work and how cancellation flows from JS options (`timeoutMs`, `AbortSignal`) to Rust execution. +This document describes how `crates/pi-natives` schedules native work and how cancellation flows from JS options (`timeoutMs`, `AbortSignal`) into Rust execution. ## Implementation files @@ -8,6 +8,7 @@ This document describes how `crates/pi-natives` schedules native work and how ca - `crates/pi-natives/src/grep.rs` - `crates/pi-natives/src/glob.rs` - `crates/pi-natives/src/fd.rs` +- `crates/pi-natives/src/ast.rs` - `crates/pi-natives/src/shell.rs` - `crates/pi-natives/src/pty.rs` - `crates/pi-natives/src/html.rs` @@ -18,23 +19,26 @@ This document describes how `crates/pi-natives` schedules native work and how ca ## Core primitives (`task.rs`) -`task.rs` defines three core pieces: +`task.rs` defines: 1. `task::blocking(tag, cancel_token, work)` - Wraps `napi::AsyncTask` / `Task`. - - `compute()` runs on libuv worker threads (for CPU-bound or blocking/sync system calls). - - Returns a JS `Promise`. + - `compute()` runs on libuv worker threads. + - Returns a JS `Promise` for exported functions. + - Records a profiling sample through `profile_region(tag)`. 2. `task::future(env, tag, work)` - Wraps `env.spawn_future(...)`. - - Runs async work on Tokio runtime. + - Runs async work on Tokio's runtime. - Returns `PromiseRaw<'env, T>`. + - Records a profiling sample through `profile_region(tag)`. 3. `CancelToken` / `AbortToken` / `AbortReason` - - `CancelToken::new(timeout_ms, signal)` combines deadline + optional `AbortSignal`. + - `CancelToken::new(timeout_ms, signal)` combines an optional deadline and optional JS `AbortSignal` converted from `Unknown`. - `CancelToken::heartbeat()` is cooperative cancellation for blocking loops. - - `CancelToken::wait()` is async cancellation wait (`Signal` / `Timeout` / `User` Ctrl-C). - - `AbortToken` lets external code request abort (`abort(reason)`). + - `CancelToken::wait()` asynchronously waits for signal, timeout, or Ctrl-C. + - `CancelToken::emplace_abort_token()` creates an abortable flag when a later `Shell.abort()`/internal bridge needs one. + - `AbortToken::abort(reason)` lets external code request abort. ## `blocking` vs `future`: execution model and selection @@ -42,54 +46,57 @@ This document describes how `crates/pi-natives` schedules native work and how ca Use when work is CPU-heavy or fundamentally synchronous/blocking: -- regex/file scanning (`grep`, `glob`, `fuzzy_find`) -- synchronous PTY loop internals (`run_pty_sync` via `spawn_blocking`) -- clipboard/image/html conversions +- regex/file scanning (`grep`, `glob`, `fuzzyFind`) +- ast-grep search/edit worker work +- PTY loop internals through `tokio::task::spawn_blocking` +- image decode/resize/encode +- HTML conversion +- clipboard image read Behavior: - Work closure receives a cloned `CancelToken`. - Cancellation is only observed where code checks `ct.heartbeat()?`. -- Closure `Err(...)` rejects JS promise. +- Closure `Err(...)` rejects the JS promise. ### Use `task::future` Use when work must `await` async operations: -- shell session orchestration (`shell.run`, `executeShell`) +- shell session orchestration (`Shell.run`, `executeShell`) +- PTY outer promise (`PtySession.start`) before it enters `spawn_blocking` - task racing (`tokio::select!`) between completion and cancellation Behavior: -- Future can race normal completion against `ct.wait()`. -- On cancel path, async implementations typically propagate cancellation to inner subsystems (e.g., `tokio_util::CancellationToken`) and optionally force abort on grace timeout. +- Future code can race normal completion against `ct.wait()`. +- On cancel path, async implementations typically cancel subordinate machinery and may force-abort after a grace timeout. ## JS API ↔ Rust export mapping (task/cancel relevant) -| JS-facing API | Rust export (`#[napi]`) | Scheduler | Cancellation hookup | -|---|---|---|---| -| `grep(options, onMatch?)` | `grep` | `task::blocking("grep", ct, ...)` | `CancelToken::new(options.timeoutMs, options.signal)` + `ct.heartbeat()` | -| `glob(options, onMatch?)` | `glob` | `task::blocking("glob", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in filter loop | -| `fuzzyFind(options)` | `fuzzy_find` | `task::blocking("fuzzy_find", ct, ...)` | `CancelToken::new(...)` + `ct.heartbeat()` in scoring loop | -| `shell.run(options, onChunk?)` | `Shell::run` | `task::future(env, "shell.run", ...)` | `ct.wait()` raced against run task; bridges to Tokio `CancellationToken` | -| `executeShell(options, onChunk?)` | `execute_shell` | `task::future(env, "shell.execute", ...)` | same as above | -| `pty.start(options, onChunk?)` | `PtySession::start` | `task::future(env, "pty.start", ...)` + inner `spawn_blocking` | `CancelToken` checked in sync PTY loop via `heartbeat()` | -| `htmlToMarkdown(html, options?)` | `html_to_markdown` | `task::blocking("html_to_markdown", (), ...)` | none (`()` token) | -| `PhotonImage.parse/encode/resize` | `PhotonImage::{parse,encode,resize}` | `task::blocking(...)` | none (`()` token) | -| `copyToClipboard/readImageFromClipboard` | `copy_to_clipboard` / `read_image_from_clipboard` | `task::blocking(...)` | none (`()` token) | +| JS-facing API | Rust export | Scheduler | Cancellation hookup | +| --------------------------------------- | ------------------------------------ | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| `grep(options, onMatch?)` | `grep` | `task::blocking("grep", ct, ...)` | `CancelToken::new(options.timeoutMs, options.signal)` + heartbeat checks | +| `glob(options, onMatch?)` | `glob` | `task::blocking("glob", ct, ...)` | `CancelToken::new(...)` + heartbeat checks | +| `fuzzyFind(options)` | `fuzzy_find` | `task::blocking("fuzzy_find", ct, ...)` | `CancelToken::new(...)` + heartbeat checks | +| `astGrep(options)` / `astEdit(options)` | ast exports | blocking worker path | timeout/signal fields are accepted by options and checked cooperatively in worker loops | +| `Shell#run(options, onChunk?)` | `Shell::run` | `task::future(env, "shell.run", ...)` | `ct.wait()` raced against run task; bridges to Tokio cancellation token and `AbortToken` | +| `executeShell(options, onChunk?)` | `execute_shell` | `task::future(env, "shell.execute", ...)` | same cancel race and 2s graceful window | +| `PtySession#start(options, onChunk?)` | `PtySession::start` | `task::future(env, "pty.start", ...)` + inner `spawn_blocking` | `CancelToken` checked in sync PTY loop via `heartbeat()` | +| `htmlToMarkdown(html, options?)` | `html_to_markdown` | `task::blocking("html_to_markdown", (), ...)` | none (`()` token) | +| `PhotonImage.parse/encode/resize` | `PhotonImage::{parse,encode,resize}` | `task::blocking(...)` | none (`()` token) | +| `readImageFromClipboard()` | `read_image_from_clipboard` | `task::blocking("clipboard.read_image", (), ...)` | none (`()` token) | -`text.rs` and `ps.rs` currently do not use `task::blocking`/`task::future` and therefore do not participate in this cancellation path. +`text.rs`, `tokens.rs`, `keys.rs`, most `ps.rs` functions, and synchronous utility exports do not use `task::blocking`/`task::future` and therefore do not participate in this cancellation path. ## Cancellation lifecycle and state transitions ### `CancelToken` lifecycle -`CancelToken` is cooperative and stateful: - ```text Created - ├─ no signal + no timeout -> passive token (never aborts unless externally emplaced) - ├─ signal registered -> waits for AbortSignal callback + ├─ no signal + no timeout -> passive token + ├─ signal registered -> AbortSignal callback can set AbortReason::Signal └─ deadline set -> timeout check becomes active Running @@ -98,19 +105,21 @@ Running ├─ wait() sees Ctrl-C -> AbortReason::User └─ no abort -> continue -Aborted (terminal) - └─ first abort reason wins (atomic flag + notifier) +Aborted + └─ flag stores first observed cause for waiters; heartbeat formats it as "Aborted: " ``` ### Before-start vs mid-execution cancellation - **Before start / before first cancellation check**: - - `task::future` users that race on `ct.wait()` can resolve cancel immediately once they enter `select!`. - - `task::blocking` users only observe cancellation when closure code reaches `heartbeat()`. If closure does not heartbeat early, cancellation is delayed. + - `task::future` users that race on `ct.wait()` can resolve cancellation once they enter `select!`. + - `task::blocking` users only observe cancellation when closure code reaches `heartbeat()`. - **Mid-execution**: - `blocking`: next `heartbeat()` returns `Err("Aborted: ...")`. - - `future`: `ct.wait()` branch wins `select!`, then code cancels subordinate async machinery (for shell: cancels Tokio token, waits up to 2s, then aborts task). + - `future`: `ct.wait()` branch wins `select!`, then code cancels subordinate async machinery. + - shell: cancellation triggers a Tokio cancellation token, waits up to 2 seconds, then aborts the task if needed. + - PTY: heartbeat failure or `kill()` terminates PTY child/process tree and drains output briefly. ## Heartbeat expectations for long-running loops @@ -118,10 +127,10 @@ Aborted (terminal) Observed patterns: -- `glob::filter_entries`: check each entry before filtering/matching. -- `fd::score_entries`: check each scanned candidate. -- `grep_sync`: explicit cancellation check before heavy search phase, plus fs-cache calls that also receive token. -- `run_pty_sync`: check every loop tick (~16ms sleep cadence) and kill child on cancellation. +- `glob` filtering checks entries during scan/filter work. +- `fd` scoring checks scanned candidates. +- `grep` checks before/during expensive search and passes tokens into shared scan/cache helpers. +- `run_pty_sync` checks every loop tick with a maximum 16ms wait cadence. Practical rule: no loop over external-size input should exceed a short bounded interval without a heartbeat. @@ -147,12 +156,12 @@ Error path: 1. Async body returns `Err(napi::Error)` or join failure is mapped (`... task failed: {err}`). 2. `task::future`-spawned promise rejects. -3. Some APIs intentionally return structured cancellation results instead of rejection (`ShellRunResult`/`ShellExecuteResult` with `cancelled`/`timed_out` flags and `exit_code: None`). +3. Shell and PTY command APIs model cancellation as structured results instead of rejection when the cancellation path wins: `exitCode` omitted, `cancelled` or `timedOut` set. ### Cancellation reporting split -- **Abort as error**: most blocking exports using `heartbeat()?`. -- **Abort as typed result**: shell/pty style command APIs that model cancellation in result structs. +- **Abort as error**: blocking exports using `heartbeat()?`. +- **Abort as typed result**: shell/PTY command APIs that model cancellation in result structs. Choose one model per API and document it explicitly. @@ -163,7 +172,7 @@ Choose one model per API and document it explicitly. - Fix: add `ct.heartbeat()?` at loop top and before expensive per-item steps. 2. **Long uncancelable sections** - - Symptom: cancellation latency spikes during single large call (decode, sort, compression, etc.). + - Symptom: cancellation latency spikes during single large call (decode, sort, compression, parser invocation, etc.). - Fix: split work into chunks with heartbeat boundaries; if impossible, document latency. 3. **Blocking async executor** @@ -172,7 +181,7 @@ Choose one model per API and document it explicitly. 4. **Inconsistent cancel semantics** - Symptom: one API rejects on cancel, another resolves with flags, confusing callers. - - Fix: standardize per domain and keep wrapper docs aligned. + - Fix: standardize per domain and keep docs aligned. 5. **Forgetting cancellation bridge in nested async tasks** - Symptom: outer token is cancelled but inner readers/subprocess tasks keep running. @@ -181,28 +190,28 @@ Choose one model per API and document it explicitly. ## Checklist for new cancellable exports 1. Classify work correctly: - - CPU-bound or sync blocking -> `task::blocking` - - async I/O / `await` orchestration -> `task::future` + - CPU-bound or sync blocking -> `task::blocking`. + - async I/O / `await` orchestration -> `task::future`. 2. Expose cancel inputs when needed: - - include `timeoutMs` and `signal` in `#[napi(object)]` options - - create `let ct = task::CancelToken::new(timeout_ms, signal);` + - include `timeoutMs` and `signal` in `#[napi(object)]` options, + - create `let ct = task::CancelToken::new(timeout_ms, signal);`. 3. Wire cancellation through all layers: - - blocking loops: `ct.heartbeat()?` at stable intervals - - async orchestration: race with `ct.wait()` and cancel sub-tasks/tokens + - blocking loops: `ct.heartbeat()?` at stable intervals, + - async orchestration: race with `ct.wait()` and cancel sub-tasks/tokens. 4. Decide cancellation contract: - reject promise with abort error, or - - resolve typed `{ cancelled, timedOut, ... }` - - keep this contract consistent for the API family + - resolve typed `{ cancelled, timedOut, ... }`, + - keep this contract consistent for the API family. 5. Propagate failures with context: - - map errors via `Error::from_reason(format!("...: {err}"))` - - include stage-specific prefixes (`spawn`, `decode`, `wait`, etc.) + - map errors via `Error::from_reason(format!("...: {err}"))`, + - include stage-specific prefixes (`spawn`, `decode`, `wait`, etc.). 6. Handle before-start and mid-flight cancellation: - - cancellation check/await must happen before expensive body and during long execution + - cancellation check/await must happen before expensive body and during long execution. 7. Validate no executor misuse: - - no long sync work directly inside async futures without `spawn_blocking`/blocking task wrapper + - no long sync work directly inside async futures without `spawn_blocking`/blocking task wrapper. diff --git a/docs/natives-shell-pty-process.md b/docs/natives-shell-pty-process.md index c277e15ea..1d6ee8a8e 100644 --- a/docs/natives-shell-pty-process.md +++ b/docs/natives-shell-pty-process.md @@ -1,30 +1,22 @@ # Natives Shell, PTY, Process, and Key Internals -This document covers the **execution/process/terminal primitives** in `@oh-my-pi/pi-natives`: `shell`, `pty`, `ps`, and `keys`, using the architecture terms from `docs/natives-architecture.md`. +This document covers the execution/process/terminal primitives in `@oh-my-pi/pi-natives`: `shell`, `pty`, `ps`, and `keys`, using the architecture terms from `docs/natives-architecture.md`. ## Implementation files - `crates/pi-natives/src/shell.rs` -- `crates/pi-natives/src/shell/windows.rs` (Windows only) +- `crates/pi-natives/src/shell/windows.rs` (Windows-only PATH enrichment) - `crates/pi-natives/src/pty.rs` - `crates/pi-natives/src/ps.rs` - `crates/pi-natives/src/keys.rs` -- `crates/pi-natives/src/task.rs` (shared cancellation behavior used by shell/pty) -- `packages/natives/src/shell/index.ts` -- `packages/natives/src/shell/types.ts` -- `packages/natives/src/pty/index.ts` -- `packages/natives/src/pty/types.ts` -- `packages/natives/src/ps/index.ts` -- `packages/natives/src/ps/types.ts` -- `packages/natives/src/keys/index.ts` -- `packages/natives/src/keys/types.ts` -- `packages/natives/src/bindings.ts` +- `crates/pi-natives/src/task.rs` +- `packages/natives/native/index.d.ts` ## Layer ownership -- **TS wrapper/API layer** (`packages/natives/src/*`): typed entrypoints, cancellation surface (`timeoutMs`, `AbortSignal`), and JS ergonomics. +- **Package entrypoint** (`packages/natives/native/index.js`): loads the `.node` addon and exports generated N-API bindings. - **Rust N-API module layer** (`crates/pi-natives/src/*`): shell/PTY process execution, process-tree traversal/termination, and key-sequence parsing. -- **Validation gate** (`native.ts`, architecture-level): ensures required exports (`Shell`, `executeShell`, `PtySession`, `killTree`, `listDescendants`, key helpers) exist before wrappers are used. +- **Consumers** (`packages/coding-agent`, `packages/tui`): higher-level session policy, output artifact/minimizer handling, render policy, and UI key handling. ## Shell subsystem (`shell`) @@ -35,54 +27,62 @@ Two execution modes are exposed: 1. **One-shot** via `executeShell(options, onChunk?)`. 2. **Persistent session** via `new Shell(options?)` then `shell.run(...)` repeatedly. -Both stream output through a threadsafe callback and return `{ exitCode?, cancelled, timedOut }`. +Both stream output through a threadsafe callback and return `{ exitCode?, cancelled, timedOut, minimized? }`. + +`ShellOptions` supports `sessionEnv`, `snapshotPath`, and optional output `minimizer`. `ShellExecuteOptions` supports command-scoped `env`, session-level `sessionEnv`, `snapshotPath`, timeout/signal, and optional minimizer. `ShellRunOptions` supports command, cwd, command-scoped env, timeout, and signal. ### Session creation and environment model Rust creates `brush_core::Shell` with: -- non-interactive mode, +- non-interactive, non-login mode, +- `no_profile` and `no_rc`, - `do_not_inherit_env: true`, +- bash-mode builtins, with `exec` and `suspend` disabled, - explicit environment reconstruction from host env, - skip-list for shell-sensitive vars (`PS1`, `PWD`, `SHLVL`, bash function exports, etc.). Session env behavior: -- `ShellOptions.sessionEnv` is applied once at session creation. -- `ShellRunOptions.env` is command-scoped (`EnvironmentScope::Command`) and popped after each run. +- `ShellOptions.sessionEnv` / one-shot `sessionEnv` is applied at session creation. +- `ShellRunOptions.env` / one-shot `env` is command-scoped (`EnvironmentScope::Command`) and popped after the command. - `PATH` is merged specially on Windows with case-insensitive dedupe. - -Windows-only path enrichment (`shell/windows.rs`): discovered Git-for-Windows paths (`cmd`, `bin`, `usr/bin`) are appended if present and not already included. +- Windows-only path enrichment (`shell/windows.rs`) appends discovered Git-for-Windows paths when present and not already included. +- `snapshotPath`, when present, is sourced during session creation with stdout/stderr/stdin wired to null files. ### Runtime lifecycle and state transitions Persistent shell (`Shell.run`) uses this state machine: - **Idle/Uninitialized**: `session: None`. -- **Running**: first `run()` lazily creates session, stores `current_abort` token, executes command. -- **Completed + keepalive**: if execution control flow is `Normal`, `current_abort` is cleared and session is reused. -- **Completed + teardown**: if control flow is loop/script/shell-exit related (`BreakLoop`, `ContinueLoop`, `ReturnFromFunctionOrScript`, `ExitShell`), session is dropped (`session: None`). -- **Cancelled/Timed out**: run task is cancelled, grace wait (2s), then force-abort; session is dropped. +- **Running**: first `run()` lazily creates a session, stores an abort token, executes command. +- **Completed + keepalive**: if execution control flow is normal, abort state is cleared and session is reused. +- **Completed + teardown**: if control flow is loop/script/shell-exit related, session is dropped. +- **Cancelled/Timed out**: run task is cancelled, grace wait is 2 seconds, task may be force-aborted, session is dropped if lock can be acquired. - **Error**: session is dropped. One-shot shell (`executeShell`) always creates and drops a fresh session per call. -### Streaming/output behavior +### Streaming/output and minimizer behavior - Stdout/stderr are routed into a shared pipe and read concurrently. - Reader decodes UTF-8 incrementally; invalid byte sequences emit `U+FFFD` replacement chunks. -- After process completion, output drain has idle/max guards (`250ms` idle, `2s` max) to avoid hanging on background jobs keeping descriptors open. +- The command runs in a new process group policy. +- Optional minimizer configuration can capture and rewrite output. When minimization occurs, the result includes `minimized` with filter name, replacement text, original text, and byte counts. +- Consumers are responsible for persisting or displaying minimizer artifacts; the native result only carries the data. -### Cancellation, timeout, and background jobs +### Cancellation, timeout, and abort - `CancelToken` is constructed from `timeoutMs` and optional `AbortSignal`. -- On cancellation/timeout, shell cancellation token is triggered, then task gets a 2s graceful window before forced abort. -- If cancellation occurs, background jobs are terminated (`TERM`, then delayed `KILL`) using brush job metadata. +- On cancellation/timeout, shell cancellation token is triggered, then task gets a 2-second graceful window before forced abort. +- Structured result flags are used: + - timeout -> `exitCode` omitted, `timedOut: true`. + - abort signal / `Shell.abort()` -> `exitCode` omitted, `cancelled: true`. `Shell.abort()` behavior: -- aborts only current running command for that `Shell` instance, -- no-op success when nothing is running. +- aborts the current running command for that `Shell` instance through the stored `AbortToken`, +- resolves successfully even when nothing is running. ### Failure behavior @@ -91,16 +91,11 @@ Common surfaced errors include: - session init failures (`Failed to initialize shell`), - cwd errors (`Failed to set cwd`), - env set/pop failures, -- snapshot source failures, +- snapshot source failures (`Failed to source snapshot`), - pipe creation/clone failures, - execution failure (`Shell execution failed: ...`), - task wrapper failures (`Shell execution task failed: ...`). -Result-level cancellation flags: - -- timeout -> `exitCode: undefined`, `timedOut: true`. -- abort signal -> `exitCode: undefined`, `cancelled: true`. - ## PTY subsystem (`pty`) ### API model @@ -112,6 +107,8 @@ Result-level cancellation flags: - `resize(cols, rows)` - `kill()` +`PtyStartOptions` supports `command`, optional `cwd`, optional `env`, `timeoutMs`, `signal`, `cols`, and `rows`. + ### Runtime lifecycle and state transitions `PtySession` state machine: @@ -119,7 +116,7 @@ Result-level cancellation flags: - **Idle**: `core: None`. - **Reserved**: `start()` installs control channel synchronously (`core: Some`) before async work begins, so `write/resize/kill` become immediately valid. - **Running**: blocking PTY loop handles child state, reader events, cancellation heartbeat, and control messages. -- **Terminal closed**: child exit + reader completion. +- **Terminal closed / drain**: child exit or cancellation starts a short reader drain window. - **Finalized**: `core` is always reset to `None` after start task completion (success or error). Concurrency guard: @@ -130,21 +127,28 @@ Concurrency guard: - PTY opened via `portable_pty::native_pty_system().openpty(...)`. - Command currently runs as `sh -lc ` with optional `cwd` and env overrides. +- Default size is `120x40`; dimensions are clamped (`cols 20..400`, `rows 5..200`). - `write()` sends raw bytes to PTY stdin. -- `resize()` clamps dimensions (`cols 20..400`, `rows 5..200`) and calls master resize. -- `kill()` marks run as cancelled and kills child process. +- `resize()` sends a control message and clamps dimensions again. +- `kill()` sends a control message that marks the run cancelled and terminates the child/process tree. Output path: - dedicated reader thread reads master stream, -- incremental UTF-8 decode with `U+FFFD` replacement on invalid bytes, +- incremental UTF-8 decode emits `U+FFFD` for invalid bytes, - chunks forwarded through N-API threadsafe callback. +Termination path: + +- Unix: terminate process group when known, terminate child tree, call child kill, then repeat with SIGKILL. +- Non-Unix: terminate child tree, call child kill, then repeat with SIGKILL-equivalent process-tree helper. + ### Cancellation and timeout semantics - `timeoutMs` and `AbortSignal` feed a `CancelToken`. -- loop calls `ct.heartbeat()` periodically; abort triggers child kill. -- timeout classification is string-based (`"Timeout"` substring in heartbeat error). +- Loop calls `ct.heartbeat()` periodically with a 16ms maximum wait cadence. +- Timeout classification is based on the heartbeat error string containing `Timeout`. +- Cancellation/kill starts a 300ms post-cancel drain window; normal child exit starts a 300ms post-exit drain window. ### Failure behavior @@ -168,8 +172,6 @@ Control call failures when not running: - `killTree(pid, signal) -> number` - `listDescendants(pid) -> number[]` -TS wrapper also registers native kill-tree integration into shared utils via `setNativeKillTree(native.killTree)`. - ### Platform-specific implementation - **Linux**: recursively reads `/proc//task//children`. @@ -179,7 +181,7 @@ TS wrapper also registers native kill-tree integration into shared utils via `se ### Kill-tree behavior - Descendants are collected recursively. -- Kill order is bottom-up (deepest descendants first) to reduce orphan re-parenting. +- Kill order is bottom-up (deepest descendants first). - Root pid is killed last. - Return value is count of successful terminations. @@ -190,10 +192,10 @@ Signal behavior: ### Failure behavior -This module is intentionally non-throwing at API surface: +This module is intentionally non-throwing at API surface for ordinary process misses: - missing/inaccessible process tree branches are skipped, -- per-pid kill failures are counted as unsuccessful (not errors), +- per-pid kill failures are counted as unsuccessful, - lookup miss typically yields `[]` from `listDescendants` and `0` from `killTree`. ## Key parsing subsystem (`keys`) @@ -233,36 +235,36 @@ Layout behavior: - Match functions return `false` on parse failure or mismatch. - No thrown error surface for malformed key input. -## JS wrapper API ↔ Rust export mapping +## JS API ↔ Rust export mapping ### Shell + PTY + Process -| TS wrapper API | Rust N-API export | Notes | -|---|---|---| -| `executeShell(options, onChunk?)` | `executeShell` (`execute_shell`) | One-shot shell execution | -| `new Shell(options?)` | `Shell` class | Persistent shell session | -| `shell.run(options, onChunk?)` | `Shell::run` | Reuses session on keepalive control flow | -| `shell.abort()` | `Shell::abort` | Aborts active run for that shell instance | -| `new PtySession()` | `PtySession` class | Stateful PTY session | -| `pty.start(options, onChunk?)` | `PtySession::start` | Interactive PTY run | -| `pty.write(data)` | `PtySession::write` | Raw stdin passthrough | -| `pty.resize(cols, rows)` | `PtySession::resize` | Clamped terminal dimensions | -| `pty.kill()` | `PtySession::kill` | Force-kills active PTY child | -| `killTree(pid, signal)` | `killTree` (`kill_tree`) | Children-first process tree termination | -| `listDescendants(pid)` | `listDescendants` (`list_descendants`) | Recursive descendants listing | +| JS API | Rust N-API export | Notes | +| --------------------------------- | -------------------------------------- | ----------------------------------------- | +| `executeShell(options, onChunk?)` | `executeShell` (`execute_shell`) | One-shot shell execution | +| `new Shell(options?)` | `Shell` class | Persistent shell session | +| `shell.run(options, onChunk?)` | `Shell::run` | Reuses session on keepalive control flow | +| `shell.abort()` | `Shell::abort` | Aborts active run for that shell instance | +| `new PtySession()` | `PtySession` class | Stateful PTY session | +| `pty.start(options, onChunk?)` | `PtySession::start` | Interactive PTY run | +| `pty.write(data)` | `PtySession::write` | Raw stdin passthrough | +| `pty.resize(cols, rows)` | `PtySession::resize` | Clamped terminal dimensions | +| `pty.kill()` | `PtySession::kill` | Force-kills active PTY child | +| `killTree(pid, signal)` | `killTree` (`kill_tree`) | Children-first process tree termination | +| `listDescendants(pid)` | `listDescendants` (`list_descendants`) | Recursive descendants listing | ### Keys -| TS wrapper API | Rust N-API export | Notes | -|---|---|---| -| `matchesKittySequence(data, cp, mod)` | `matchesKittySequence` (`matches_kitty_sequence`) | Kitty codepoint+modifier match | -| `parseKey(data, kittyProtocolActive)` | `parseKey` (`parse_key`) | Normalized key-id parser | -| `matchesLegacySequence(data, keyName)` | `matchesLegacySequence` (`matches_legacy_sequence`) | Exact legacy sequence map check | -| `parseKittySequence(data)` | `parseKittySequence` (`parse_kitty_sequence`) | Structured Kitty parse result | -| `matchesKey(data, keyId, kittyProtocolActive)` | `matchesKey` (`matches_key`) | High-level key matcher | +| JS API | Rust N-API export | Notes | +| ---------------------------------------------- | --------------------------------------------------- | ------------------------------- | +| `matchesKittySequence(data, cp, mod)` | `matchesKittySequence` (`matches_kitty_sequence`) | Kitty codepoint+modifier match | +| `parseKey(data, kittyProtocolActive)` | `parseKey` (`parse_key`) | Normalized key-id parser | +| `matchesLegacySequence(data, keyName)` | `matchesLegacySequence` (`matches_legacy_sequence`) | Exact legacy sequence map check | +| `parseKittySequence(data)` | `parseKittySequence` (`parse_kitty_sequence`) | Structured Kitty parse result | +| `matchesKey(data, keyId, kittyProtocolActive)` | `matchesKey` (`matches_key`) | High-level key matcher | ## Abandoned session cleanup and finalization notes -- **Shell persistent session**: if a run is cancelled/timed out/errors/non-keepalive control flow, Rust explicitly drops the internal session state. Successful normal runs keep the session for reuse. +- **Shell persistent session**: if a run is cancelled/timed out/errors/non-keepalive control flow, Rust drops the internal session state. Successful normal runs keep the session for reuse. - **PTY session**: `core` is always cleared after `start()` finishes, including failure paths. - **No explicit JS finalizer-driven kill contract** is exposed by wrappers; cleanup is primarily tied to run completion/cancellation paths. Callers should use `timeoutMs`, `AbortSignal`, `shell.abort()`, or `pty.kill()` for deterministic teardown. diff --git a/docs/natives-text-search-pipeline.md b/docs/natives-text-search-pipeline.md index 46a1c80e8..bdf460d53 100644 --- a/docs/natives-text-search-pipeline.md +++ b/docs/natives-text-search-pipeline.md @@ -1,73 +1,72 @@ # Natives Text/Search Pipeline -This document maps the `@oh-my-pi/pi-natives` text/search surface (`grep`, `glob`, `text`, `highlight`) from TypeScript wrappers to Rust N-API exports and back to JS result objects. +This document maps the `@oh-my-pi/pi-natives` text/search/code surface from generated JS/TS exports to Rust N-API modules and back to JS result objects. Terminology follows `docs/natives-architecture.md`: -- **Wrapper**: TS API in `packages/natives/src/*` -- **Rust module layer**: N-API exports in `crates/pi-natives/src/*` -- **Shared scan cache**: `fs_cache`-backed directory-entry cache used by discovery/search flows + +- **Generated binding**: public API in `packages/natives/native/index.d.ts`. +- **Rust module layer**: N-API exports in `crates/pi-natives/src/*`. +- **Shared scan cache**: `fs_cache`-backed directory-entry cache used by discovery/search flows. ## Implementation files -- `packages/natives/src/grep/index.ts` -- `packages/natives/src/grep/types.ts` -- `packages/natives/src/glob/index.ts` -- `packages/natives/src/glob/types.ts` -- `packages/natives/src/text/index.ts` -- `packages/natives/src/text/types.ts` -- `packages/natives/src/highlight/index.ts` -- `packages/natives/src/highlight/types.ts` +- `packages/natives/native/index.d.ts` - `crates/pi-natives/src/grep.rs` - `crates/pi-natives/src/glob.rs` - `crates/pi-natives/src/glob_util.rs` - `crates/pi-natives/src/fs_cache.rs` +- `crates/pi-natives/src/fd.rs` +- `crates/pi-natives/src/ast.rs` - `crates/pi-natives/src/text.rs` - `crates/pi-natives/src/highlight.rs` -- `crates/pi-natives/src/fd.rs` +- `crates/pi-natives/src/tokens.rs` ## JS API ↔ Rust export mapping -| JS wrapper API | Rust export (`#[napi]`, snake_case -> camelCase) | Rust module | -| --- | --- | --- | -| `grep(options, onMatch?)` | `grep` | `grep.rs` | -| `searchContent(content, options)` | `search` | `grep.rs` | -| `hasMatch(content, pattern, options?)` | `hasMatch` | `grep.rs` | -| `fuzzyFind(options)` | `fuzzyFind` | `fd.rs` | -| `glob(options, onMatch?)` | `glob` | `glob.rs` | -| `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `fs_cache.rs` | -| `wrapTextWithAnsi(text, width)` | `wrapTextWithAnsi` | `text.rs` | -| `truncateToWidth(text, maxWidth, ellipsis, pad)` | `truncateToWidth` | `text.rs` | -| `sliceWithWidth(line, startCol, length, strict?)` | `sliceWithWidth` | `text.rs` | -| `extractSegments(line, beforeEnd, afterStart, afterLen, strictAfter)` | `extractSegments` | `text.rs` | -| `sanitizeText(text)` | `sanitizeText` | `text.rs` | -| `visibleWidth(text)` | `visibleWidth` | `text.rs` | -| `highlightCode(code, lang, colors)` | `highlightCode` | `highlight.rs` | -| `supportsLanguage(lang)` | `supportsLanguage` | `highlight.rs` | -| `getSupportedLanguages()` | `getSupportedLanguages` | `highlight.rs` | +| JS API | Rust export (`#[napi]`, snake_case -> camelCase) | Rust module | +| ------------------------------------------------------------------------------- | ------------------------------------------------ | -------------- | +| `grep(options, onMatch?)` | `grep` | `grep.rs` | +| `search(content, options)` | `search` | `grep.rs` | +| `hasMatch(content, pattern, ignoreCase?, multiline?)` | `hasMatch` | `grep.rs` | +| `fuzzyFind(options)` | `fuzzyFind` | `fd.rs` | +| `glob(options, onMatch?)` | `glob` | `glob.rs` | +| `invalidateFsScanCache(path?)` | `invalidateFsScanCache` | `fs_cache.rs` | +| `astGrep(options)` | `astGrep` | `ast.rs` | +| `astEdit(options)` | `astEdit` | `ast.rs` | +| `wrapTextWithAnsi(text, width, tabWidth)` | `wrapTextWithAnsi` | `text.rs` | +| `truncateToWidth(text, maxWidth, ellipsis, pad, tabWidth)` | `truncateToWidth` | `text.rs` | +| `sliceWithWidth(line, startCol, length, strict, tabWidth)` | `sliceWithWidth` | `text.rs` | +| `extractSegments(line, beforeEnd, afterStart, afterLen, strictAfter, tabWidth)` | `extractSegments` | `text.rs` | +| `sanitizeText(text)` | `sanitizeText` | `text.rs` | +| `visibleWidth(text, tabWidth)` | `visibleWidth` | `text.rs` | +| `highlightCode(code, lang, colors)` | `highlightCode` | `highlight.rs` | +| `supportsLanguage(lang)` | `supportsLanguage` | `highlight.rs` | +| `getSupportedLanguages()` | `getSupportedLanguages` | `highlight.rs` | +| `countTokens(input, encoding?)` | `countTokens` | `tokens.rs` | ## Pipeline overview by subsystem -## 1) Regex search (`grep`, `searchContent`, `hasMatch`) +## 1) Regex search (`grep`, `search`, `hasMatch`) ### Input/options flow -1. TS wrapper forwards options to native: - - `grep/index.ts` passes `options` mostly unchanged and wraps callback from `(match) => void` to napi threadsafe callback shape `(err, match)`. - - `searchContent` and `hasMatch` pass string/`Uint8Array` directly. +1. Callers invoke generated native exports directly; there is no package-local TS wrapper that renames `search` to `searchContent`. 2. Rust option structs in `grep.rs` deserialize camelCase fields (`ignoreCase`, `maxCount`, `contextBefore`, `contextAfter`, `maxColumns`, `timeoutMs`). 3. `grep` creates `CancelToken` from `timeoutMs` + `AbortSignal` and runs inside `task::blocking("grep", ...)`. +4. `search` and `hasMatch` operate on provided string/`Uint8Array` content and do not scan the filesystem. ### Execution branches -- **In-memory branch (pure utility)** - - `search` → `search_sync` → `run_search` on provided content bytes. +- **In-memory branch** + - `search` -> `search_sync` / search helpers over provided content bytes. + - `hasMatch` compiles/checks pattern against provided content and returns a boolean. - No filesystem scan, no `fs_cache`. -- **Single-file branch (filesystem-dependent)** - - `grep_sync` resolves path, checks metadata is file, streams up to `MAX_FILE_BYTES` per file (`4 MiB`) through ripgrep matcher. -- **Directory branch (filesystem-dependent)** +- **Single-file branch** + - `grep` resolves path, checks metadata is file, and searches that file. +- **Directory branch** - Optional cache lookup via `fs_cache::get_or_scan` when `cache: true`. - Fresh scan via `fs_cache::force_rescan` when `cache: false`. - - Optional empty-result recheck when cache age exceeds `empty_recheck_ms()`. + - Optional empty-result recheck when cached results are older than the empty-result recheck threshold. - Entry filtering: file-only + optional glob filter (`glob_util`) + optional type filter mapping (`js`, `ts`, `rust`, etc.). ### Search/collection semantics @@ -75,31 +74,32 @@ Terminology follows `docs/natives-architecture.md`: - Regex engine: `grep_regex::RegexMatcherBuilder` with `ignoreCase` and `multiline`. - Context resolution: - `contextBefore/contextAfter` override legacy `context`. - - Non-content modes zero out context collection. + - Non-content modes do not collect context. - Output modes: - - `content` => one `GrepMatch` per hit. - - `count` and `filesWithMatches` both map to count-style entries (`lineNumber=0`, `line=""`, `matchCount` set). + - `content` -> one `GrepMatch` per hit. + - `count` and `filesWithMatches` map to count-style entries (`lineNumber=0`, `line=""`, `matchCount` set). - Limits: - - Global `offset` and `maxCount` applied across files. + - Global `offset` and `maxCount` apply across files. - Parallel path is used only when `maxCount` is unset and `offset == 0`; otherwise sequential path preserves deterministic global offset/limit semantics. ### Result shaping back to JS -- Rust `SearchResult`/`GrepResult` fields map to TS types via N-API object field conversion. -- Counters are clamped to `u32` before crossing N-API. -- Optional booleans are omitted unless true in some paths (`limitReached`). -- Streaming callback receives each shaped `GrepMatch` (content or count entry). +- Rust `SearchResult`/`GrepResult` fields map to TS interfaces via N-API object conversion. +- Counters are clamped before crossing N-API where needed. +- `GrepResult.limitReached` is optional and emitted when true. +- Streaming callback receives each shaped `GrepMatch` for content or count-style entries. ### Failure behavior -- `searchContent` returns `SearchResult.error` for regex/search failures instead of throwing. -- `grep` rejects on hard errors (invalid path, invalid glob/regex, cancellation timeout/abort). -- `hasMatch` returns `Result` and throws on invalid pattern/UTF-8 decoding errors. +- `search` returns `SearchResult.error` for regex/search failures instead of throwing. +- `grep` rejects on hard errors such as invalid path, invalid glob/regex, or cancellation timeout/abort. +- `hasMatch` returns a boolean on success and throws on invalid pattern/UTF-8 conversion errors. - File open/search errors in multi-file scans are skipped per-file; scan continues. ### Malformed regex handling `grep.rs` sanitizes braces before regex compile: + - Invalid repetition-like braces are escaped (`{`/`}` -> `\{`/`\}`) when they cannot form `{N}`, `{N,}`, `{N,M}`. - This prevents common literal-template fragments (for example `${platform}`) from failing as malformed repetition. - Remaining invalid regex syntax still returns a regex error. @@ -110,62 +110,73 @@ Terminology follows `docs/natives-architecture.md`: ### `glob` flow -1. TS wrapper (`glob/index.ts`): - - `path.resolve(options.path)`. - - Defaults: `pattern="*"`, `hidden=false`, `gitignore=true`, `recursive=true`. -2. Rust `glob` builds `GlobConfig` and compiles pattern via `glob_util::compile_glob`. +1. Caller passes `GlobOptions` directly. `pattern` and `path` are required in the generated type. +2. Rust resolves the search path and compiles pattern via `glob_util::compile_glob`. 3. Entry source: - - `cache=true` => `get_or_scan` + optional stale-empty `force_rescan`. - - `cache=false` => `force_rescan(..., store=false)` (fresh only). + - `cache=true` -> `get_or_scan` + optional stale-empty `force_rescan`. + - `cache=false` -> `force_rescan(..., store=false)` (fresh only). 4. Filtering: - - Skip `.git` always. - - Skip `node_modules` unless requested (`includeNodeModules` or pattern mentioning node_modules). - - Apply glob match. - - Apply file-type filter; symlink `file/dir` filters resolve target metadata. -5. Optional sort by mtime desc (`sortByMtime`) before truncating to `maxResults`. + - skip `.git` always; + - skip `node_modules` unless requested (`includeNodeModules`) or pattern mentions `node_modules`; + - apply glob match; + - apply file-type filter; symlink `file`/`dir` filters resolve target metadata. +5. Optional sort by mtime descending (`sortByMtime`) before truncating to `maxResults`. -### `fuzzyFind` flow (implemented in `fd.rs`) +### `fuzzyFind` flow -1. TS wrapper is exported from `grep` module, but Rust implementation lives in `fd.rs`. -2. Shared scan source from `fs_cache` with same cache/no-cache split and stale-empty recheck policy. +1. Rust implementation lives in `fd.rs`; generated export is `fuzzyFind`. +2. Shared scan source from `fs_cache` with the same cache/no-cache split and stale-empty recheck policy. 3. Scoring: - - exact / starts-with / contains / subsequence-based fuzzy score - - separator/punctuation-normalized scoring path - - directory bonus and deterministic tie-break (`score desc`, then `path asc`) + - exact / starts-with / contains / subsequence-based fuzzy score; + - separator/punctuation-normalized scoring path; + - directory bonus and deterministic tie-break (`score desc`, then `path asc`). 4. Symlink entries are excluded from fuzzy results. ### Failure behavior -- Invalid glob pattern => error from `glob_util::compile_glob`. -- Search root must be an existing directory (`resolve_search_path`), otherwise error. +- Invalid glob pattern returns an error from `glob_util::compile_glob`. +- Search root must resolve to an existing directory for directory discovery flows. - Cancellation/timeouts propagate as abort errors via `CancelToken::heartbeat()` checks in loops. ### Malformed glob handling `glob_util::build_glob_pattern` is tolerant: -- Normalizes `\` to `/`. -- Auto-prefixes simple recursive patterns with `**/` when `recursive=true`. -- Auto-closes unbalanced `{...` alternation groups before compile. -## 3) Shared scan/cache lifecycle (`fs_cache`) +- normalizes `\` to `/`, +- auto-prefixes simple recursive patterns with `**/` when `recursive=true`, +- auto-closes unbalanced `{...` alternation groups before compile. + +## 3) AST search/edit (`astGrep`, `astEdit`) + +`ast.rs` exposes syntax-aware code search and rewrite operations. + +- `astGrep(options)` returns matches with byte/line/column coordinates and optional metavariable bindings. +- `astEdit(options)` returns replacement changes, per-file counts, searched/touched file counts, parse errors, and whether edits were applied. +- `dryRun` defaults to true for edit options in the generated documentation. +- Options include language override, path/glob/selector, strictness, limits, parse-error policy, `signal`, and `timeoutMs`. + +These exports are direct native APIs used by tooling; they are not mediated by a TS wrapper in `packages/natives`. + +## 4) Shared scan/cache lifecycle (`fs_cache`) `fs_cache` stores scan results as normalized relative entries (`path`, `fileType`, optional `mtime`) keyed by: -- canonical search root -- `include_hidden` -- `use_gitignore` + +- canonical search root, +- `include_hidden`, +- `use_gitignore`. ### Cache state transitions 1. **Miss / disabled** - - TTL is `0` or key absent/expired -> fresh `collect_entries`. + - TTL is `0` or key absent/expired -> fresh collection. 2. **Hit** - - Entry age `< cache_ttl_ms()` -> return cached entries + `cache_age_ms`. -3. **Stale-empty recheck** (caller policy in `glob`/`grep`/`fd`) - - If query yields zero matches and `cache_age_ms >= empty_recheck_ms()`, force one rescan. + - Entry age is within TTL -> return cached entries + `cache_age_ms`. +3. **Stale-empty recheck** + - If query yields zero matches and cache age exceeds the empty-result threshold, force one rescan. 4. **Invalidation** - `invalidateFsScanCache(path?)`: - - no arg: clear all keys - - path arg: remove keys whose root prefixes that target path + - no arg: clear all keys; + - path arg: remove keys for roots affected by that path. ### Stale-result tradeoff @@ -174,70 +185,77 @@ Terminology follows `docs/natives-architecture.md`: - Empty-result recheck reduces stale negatives for older cached scans at the cost of one extra scan. - Explicit invalidation is the intended correctness hook after file mutations. -## 4) ANSI text utilities (`text`) +## 5) ANSI text utilities (`text`) -These are pure, in-memory utilities (no filesystem scanning). +These are pure, in-memory utilities. ### Boundaries and responsibilities -- **`text.rs` owns terminal-cell semantics**: - - ANSI sequence parsing - - grapheme-aware width and slicing - - wrap/truncate/sanitize behavior -- **`grep.rs` line truncation (`maxColumns`) is separate**: - - simple character-boundary truncation of matched lines with `...` - - not ANSI-state-preserving and not terminal-cell width aware +- `text.rs` owns terminal-cell semantics: + - ANSI sequence parsing, + - grapheme-aware width and slicing, + - wrap/truncate/sanitize behavior, + - explicit tab-width parameter on width-sensitive APIs. +- `grep.rs` line truncation (`maxColumns`) is separate: + - simple character-boundary truncation of matched lines with `...`, + - not ANSI-state-preserving and not terminal-cell width aware. ### Key behaviors - `wrapTextWithAnsi`: wraps by visible width, carries active SGR codes across wrapped lines. -- `truncateToWidth`: visible-cell truncation with ellipsis policy (`Unicode`, `Ascii`, `Omit`), optional right padding, and fast-path returning original JS string when unchanged. +- `truncateToWidth`: visible-cell truncation with ellipsis policy (`Unicode`, `Ascii`, `Omit`), optional right padding. - `sliceWithWidth`: column slicing with optional strict width enforcement. - `extractSegments`: extracts before/after segments around an overlay while restoring ANSI state for the `after` segment. -- `sanitizeText`: strips ANSI escapes + control chars, drops lone surrogates, normalizes CR/LF by removing `\r`. -- `visibleWidth`: counts visible terminal cells (tabs use fixed `TAB_WIDTH` from Rust implementation). +- `sanitizeText`: strips ANSI escapes + control chars, drops lone surrogates, normalizes line endings. +- `visibleWidth`: counts visible terminal cells using caller-supplied tab width. ### Failure behavior -Text functions generally return deterministic transformed output; errors are limited to JS string conversion boundaries (N-API argument conversion failures). +Text functions generally return deterministic transformed output; errors are limited to N-API argument/string conversion boundaries. -## 5) Syntax highlighting (`highlight`) +## 6) Syntax highlighting (`highlight`) -`highlight.rs` is pure transformation (no FS, no cache). +`highlight.rs` is pure transformation; it does not use the filesystem scan cache. ### Flow -1. Wrapper forwards `code`, optional `lang`, and ANSI color palette. -2. Rust resolves syntax by: - - token/name lookup - - extension lookup - - alias table fallback (`ts/tsx/js -> JavaScript`, etc.) - - fallback to plain text syntax when unresolved -3. Parse each line with syntect `ParseState` and scope stack. -4. Map scopes to 11 semantic color categories and inject/reset ANSI color codes. +1. Caller passes `code`, optional `lang`, and ANSI color palette. +2. Rust resolves syntax by token/name lookup, extension lookup, alias table fallback, then plain-text fallback. +3. Each line is parsed with syntect `ParseState` and scope stack. +4. Scopes map to semantic color categories and ANSI color codes are injected/reset. ### Failure behavior - Per-line parse failure does not fail the call: that line is appended unhighlighted and processing continues. - Unknown/unsupported language falls back to plain text syntax. +## 7) Token counting (`tokens`) + +`countTokens(input, encoding?)` is an in-memory utility. + +- `input` may be a single string or an array of strings. +- Arrays return one aggregate count and are encoded in parallel in Rust. +- Default encoding is `O200kBase`; `Cl100kBase` is also available. +- The implementation uses ordinary tokenization, not special-token handling. + ## Pure utility vs filesystem-dependent flows -| Flow | Filesystem access | Shared cache | Notes | -| --- | --- | --- | --- | -| `searchContent` / `hasMatch` | No | No | regex on provided bytes/string only | -| `text` module functions | No | No | ANSI/width/sanitization only | -| `highlight` module functions | No | No | syntax + ANSI coloring only | -| `glob` | Yes | Optional | directory scans + glob filtering | -| `fuzzyFind` | Yes | Optional | directory scans + fuzzy scoring | -| `grep` (file/dir path) | Yes | Optional (dir mode) | ripgrep over files, optional filters/callback | +| Flow | Filesystem access | Shared cache | Notes | +| ---------------------------- | ----------------- | -------------------- | --------------------------------------------- | +| `search` / `hasMatch` | No | No | regex on provided bytes/string only | +| `text` module functions | No | No | ANSI/width/sanitization only | +| `highlight` module functions | No | No | syntax + ANSI coloring only | +| `countTokens` | No | No | tokenization only | +| `astGrep` / `astEdit` | Yes | No | syntax-aware file search/edit | +| `glob` | Yes | Optional | directory scans + glob filtering | +| `fuzzyFind` | Yes | Optional | directory scans + fuzzy scoring | +| `grep` (file/dir path) | Yes | Optional in dir mode | ripgrep over files, optional filters/callback | ## End-to-end lifecycle summary -1. Caller invokes TS wrapper with typed options. -2. Wrapper normalizes defaults (notably `glob`) and forwards to `native.*` export. -3. Rust validates/normalizes options and builds matcher/search config. -4. For filesystem flows, entries are scanned (cache hit/miss/rescan) then filtered/scored. -5. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution. -6. Rust shapes outputs into N-API objects (`lineNumber`, `matchCount`, `limitReached`, etc.). -7. TS wrapper returns typed JS objects (and optional per-match callbacks for `grep`/`glob`). +1. Caller invokes generated native export with typed options. +2. Rust validates/normalizes options and builds matcher/search config. +3. For filesystem flows, entries are scanned (cache hit/miss/rescan where applicable) then filtered/scored/searched. +4. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution. +5. Rust shapes outputs into N-API objects (`lineNumber`, `matchCount`, `limitReached`, etc.). +6. Generated bindings return typed JS objects and optional per-match callbacks for `grep`/`glob`. diff --git a/docs/non-compaction-retry-policy.md b/docs/non-compaction-retry-policy.md index 29bad0ca7..6e97d5b5d 100644 --- a/docs/non-compaction-retry-policy.md +++ b/docs/non-compaction-retry-policy.md @@ -32,16 +32,16 @@ So: overload/rate/server/network-style failures use this retry policy; context-w - assistant `stopReason === "error"` - `errorMessage` exists - message is **not** context overflow -- `errorMessage` matches `#isRetryableErrorMessage(...)` +- `errorMessage` matches transient transport/envelope patterns or `isUsageLimitError(...)` -Current retryable pattern set (regex-based): +Current retryable inputs are regex/string-classified: -- overloaded +- transient transport/envelope failures, including Anthropic stream-envelope failures before `message_start` +- overloaded/provider-returned-error wording - rate limit / usage limit / too many requests - HTTP-like server classes: 429, 500, 502, 503, 504 -- service unavailable / server error / internal error -- connection error / fetch failed -- `retry delay` wording +- service unavailable / server/internal error +- network/connection/socket failures, refused/closed connections, upstream connect/reset-before-headers, socket hang up, timeout/timed out, fetch failed, terminated, retry delay wording, and unexpected socket close messages This is string-pattern classification, not typed provider error codes. @@ -61,12 +61,13 @@ Flow (`#handleRetryableError`): 3. Increment `#retryAttempt`. 4. Create `#retryPromise` once (first attempt in a chain). 5. If attempt exceeded `retry.maxRetries`, emit final failure event and stop. -6. Compute delay: `retry.baseDelayMs * 2^(attempt-1)`. -7. For usage-limit errors, parse retry hints and call auth storage (`markUsageLimitReached(...)`); if provider/model switch succeeds, force delay to `0`. -8. Emit `auto_retry_start`. -9. Remove the trailing assistant error message from agent runtime state (kept in persisted session history). -10. Sleep with abort support. -11. On wake, schedule `agent.continue()` via `setTimeout(..., 0)`. +6. Compute base delay: `retry.baseDelayMs * 2^(attempt-1)`. +7. For usage-limit errors, parse retry hints and call auth storage (`markUsageLimitReached(...)`); if credential switching succeeds, force delay to `0`, otherwise use a larger retry-after/backoff hint when present. +8. If no credential switch occurred, suppress the current model selector for cooldown, try configured retry model fallback chains, and force delay to `0` on model switch. +9. Emit `auto_retry_start`. +10. Remove the trailing assistant error message from agent runtime state (kept in persisted session history). +11. Sleep with abort support. +12. Schedule `agent.continue()` through the post-prompt task scheduler (`delayMs: 1`) for the same prompt generation. ### What resets retry counters @@ -98,7 +99,7 @@ Backoff sequence with default settings: - attempt 2: 4000 ms - attempt 3: 8000 ms -Delay override inputs are only used in the usage-limit handling path, and only to influence auth-storage model/account switching decision. In the main non-compaction retry path, backoff remains local exponential delay unless switching succeeds (`delayMs = 0`). +Delay override inputs can come from parsed retry headers (`retry-after-ms`, `retry-after`, `x-ratelimit-reset-ms`, `x-ratelimit-reset`) or usage-limit backoff. Credential/model fallback switches set delay to `0`; otherwise parsed hints can extend the exponential local delay. ## Abort mechanics @@ -147,6 +148,8 @@ Defined in settings schema under retry group: - `retry.enabled` - `retry.maxRetries` - `retry.baseDelayMs` +- `retry.fallbackChains` +- `retry.fallbackRevertPolicy` (`"cooldown-expiry"` by default; `"never"` disables automatic restoration) Programmatic toggles in session: @@ -174,6 +177,8 @@ Session-level retry events: - `auto_retry_start { attempt, maxAttempts, delayMs, errorMessage }` - `auto_retry_end { success, attempt, finalError? }` +- `retry_fallback_applied { from, to, role }` +- `retry_fallback_succeeded { model, role }` Propagation: @@ -207,3 +212,4 @@ A new retry chain can still start later on a future retryable error after counte - Classification is regex text matching; provider-specific structured errors are not used here. - Retry strips the failing assistant error from **runtime context** before re-continue, but session history still keeps that error entry. - `RpcSessionState` currently exposes `autoCompactionEnabled` but not an `autoRetryEnabled` field; RPC callers must track their own toggle state or query settings through other APIs. +- Model fallback changes append temporary `model_change` entries and may later restore the primary model when its cooldown expires, depending on `retry.fallbackRevertPolicy`. diff --git a/docs/notebook-tool-runtime.md b/docs/notebook-tool-runtime.md index 7181a0646..e7b748c0a 100644 --- a/docs/notebook-tool-runtime.md +++ b/docs/notebook-tool-runtime.md @@ -117,10 +117,9 @@ Resource exhaustion recovery: ## 4) Environment/session variable injection -Kernel startup receives optional env map from executor: +Kernel startup receives the optional session file path from executor: - `PI_SESSION_FILE` (session state file path) -- `ARTIFACTS` (artifact directory) `PythonKernel.#initializeKernelEnvironment(...)` then runs init script inside kernel to: @@ -130,7 +129,7 @@ Kernel startup receives optional env map from executor: Implication: -- prelude helpers that read session or artifact context rely on these env vars in Python process state. +- prelude helpers that read session context rely on this env var in Python process state. ## 5) Streaming/chunk and display handling (kernel-backed path) diff --git a/docs/plugin-manager-installer-plumbing.md b/docs/plugin-manager-installer-plumbing.md index 5542d90b8..da14fbce5 100644 --- a/docs/plugin-manager-installer-plumbing.md +++ b/docs/plugin-manager-installer-plumbing.md @@ -1,6 +1,6 @@ # Plugin manager and installer plumbing -This document describes how `omp plugin` operations mutate plugin state on disk and how installed plugins become runtime capabilities (tools today, hooks/commands path resolution available). +This document describes how `omp plugin` operations mutate plugin state on disk and how installed plugins become runtime capabilities (tools and extensions today, hooks/commands path resolution available). ## Scope and architecture @@ -19,11 +19,11 @@ There are two plugin-management implementations in the codebase: omp plugin ... -> src/commands/plugin.ts -> runPluginCommand(...) in src/cli/plugin-cli.ts - -> PluginManager method (install/list/uninstall/link/...) + -> PluginManager method (install/list/uninstall/link/...) -> mutate ~/.omp/plugins/{package.json,node_modules,omp-plugins.lock.json} - -> runtime discovery: discoverAndLoadCustomTools(...) - -> getAllPluginToolPaths(cwd) - -> custom tool loader imports tool modules + -> runtime discovery: discoverAndLoadCustomTools(...) and discoverAndLoadExtensions(...) + -> getAllPluginToolPaths(cwd) / getAllPluginExtensionPaths(cwd) + -> custom tool loader imports tool modules; extension loader imports extension modules ``` ### Command entrypoints @@ -164,6 +164,7 @@ Filtering: For each enabled plugin: +- `resolvePluginExtensionPaths(plugin)` - `resolvePluginToolPaths(plugin)` - `resolvePluginHookPaths(plugin)` - `resolvePluginCommandPaths(plugin)` @@ -178,8 +179,9 @@ Missing files are silently skipped (`existsSync` guard). ## Current runtime wiring differences - **Tools are wired into runtime today** via `discoverAndLoadCustomTools` (`custom-tools/loader.ts`), which calls `getAllPluginToolPaths(cwd)`. -- Paths are de-duplicated by resolved absolute path in custom tool discovery (`seen` set, first path wins). -- **Hooks/commands resolvers exist** and are exported, but this code path does not currently wire them into a runtime registry in the same way tools are wired. +- **Extensions are wired into runtime today** via `discoverAndLoadExtensions` (`extensions/loader.ts`), which calls `getAllPluginExtensionPaths(cwd)`. +- Paths are de-duplicated by resolved absolute path in custom tool and extension discovery (`seen` set, first path wins). +- **Hooks/commands resolvers exist** and are exported, but this code path does not currently wire them into a runtime registry in the same way tools and extensions are wired. ## Lock/state management details @@ -226,13 +228,13 @@ Because CLI uses `PluginManager`, these stricter link guards are not currently o The plugin manager is not transactional. -| Operation stage | Failure behavior | Rollback | -| --- | --- | --- | -| `bun install` fails | install aborts with stderr | N/A (no state writes yet) | -| Install succeeds, then manifest/feature validation fails | command fails | No uninstall rollback; dependency may remain in `node_modules`/`package.json` | -| Install succeeds, then lockfile write fails | command fails | No rollback of installed package | -| `bun uninstall` succeeds, lockfile write fails | command fails | Package removed, stale runtime state may remain | -| `link` removes old target then symlink creation fails | command fails | No restoration of previous link/dir | +| Operation stage | Failure behavior | Rollback | +| -------------------------------------------------------- | -------------------------- | ----------------------------------------------------------------------------- | +| `bun install` fails | install aborts with stderr | N/A (no state writes yet) | +| Install succeeds, then manifest/feature validation fails | command fails | No uninstall rollback; dependency may remain in `node_modules`/`package.json` | +| Install succeeds, then lockfile write fails | command fails | No rollback of installed package | +| `bun uninstall` succeeds, lockfile write fails | command fails | Package removed, stale runtime state may remain | +| `link` removes old target then symlink creation fails | command fails | No restoration of previous link/dir | Operationally, `doctor --fix` can repair some drift (`bun install`, orphaned config cleanup, invalid-feature cleanup), but it is best-effort. @@ -262,3 +264,4 @@ Operationally, `doctor --fix` can repair some drift (`bun install`, orphaned con - [`src/extensibility/plugins/parser.ts`](../packages/coding-agent/src/extensibility/plugins/parser.ts) — install spec and package-name parsing helpers - [`src/extensibility/plugins/types.ts`](../packages/coding-agent/src/extensibility/plugins/types.ts) — manifest/runtime/override type contracts - [`src/extensibility/custom-tools/loader.ts`](../packages/coding-agent/src/extensibility/custom-tools/loader.ts) — runtime wiring for plugin-provided tool modules +- [`src/extensibility/extensions/loader.ts`](../packages/coding-agent/src/extensibility/extensions/loader.ts) — runtime wiring for plugin-provided extension modules diff --git a/docs/porting-from-pi-mono.md b/docs/porting-from-pi-mono.md index 24a9f8704..3bfb97bc1 100644 --- a/docs/porting-from-pi-mono.md +++ b/docs/porting-from-pi-mono.md @@ -3,14 +3,14 @@ This guide is a repeatable checklist for porting changes from pi-mono into this repo. Use it for any merge: single file, feature branch, or full release sync. -## Last Sync Point +## Last Sync Point (historical upstream marker) **Commit:** `b21b42d032919de2f2e6920a76fa9a37c3920c0a` **Date:** 2026-03-22 -Update this section after each sync; do not reuse the previous range. +Update this section after each sync; do not reuse the previous range. This commit is an upstream pi-mono marker and may not exist in this repo's local object database. -When starting a new sync, generate patches from this commit forward: +When starting a new sync, generate patches from this commit forward in a pi-mono checkout or remote that contains the commit: ```bash git format-patch b21b42d032919de2f2e6920a76fa9a37c3920c0a..HEAD --stdout > changes.patch @@ -30,13 +30,12 @@ git format-patch b21b42d032919de2f2e6920a76fa9a37c3920c0a..HEAD --stdout > chang ## 2) Match import extension conventions -Most runtime TypeScript sources omit `.js` in internal imports, but some test/bench entrypoints keep `.js` for ESM -runtime compatibility. Follow the local package’s existing style; do not blanket-strip extensions. +Most runtime TypeScript sources omit `.js` in internal imports, but several current entrypoints and tool modules keep `.js` for ESM/runtime compatibility. Follow the surrounding file and package export style; do not blanket-strip or blanket-add extensions. -- In `packages/coding-agent` runtime sources, keep internal imports extensionless unless importing non-TS assets. +- In `packages/coding-agent` runtime sources, prefer extensionless internal imports when the surrounding module does, but preserve existing `.js` imports in files that already require them. - In `packages/tui/test` and `packages/natives/bench`, keep `.js` where surrounding files already use it. -- Keep real file extensions when required by tooling (e.g., `.json`, `.css`, `.md` text embeds). -- Example: `import { x } from "./foo.js";` → `import { x } from "./foo";` (only when the package convention is extensionless). +- Keep real file extensions when required by tooling or import assertions (e.g., `.json`, `.css`, `.md` text embeds). +- Example: `import { x } from "./foo.js";` → `import { x } from "./foo";` only when that package/file convention is extensionless. ## 3) Replace import scopes @@ -51,30 +50,29 @@ Upstream uses different package scopes. Replace them consistently. ## 4) Use Bun APIs where they improve on Node -We run on Bun. Replace Node APIs only when Bun provides a better alternative. +We run on Bun, but the current source intentionally mixes Bun APIs with small Node standard-library APIs. Replace Node APIs only when Bun provides a clearer, safer, or simpler implementation; do not mechanically rewrite every Node import. -**DO replace:** +**Prefer replacing when porting new code:** -- Process spawning: `child_process.spawn` → Bun Shell `$` for simple commands, `Bun.spawn`/`Bun.spawnSync` for streaming or long-running work -- File I/O: `fs.readFileSync` → `Bun.file().text()` / `Bun.write()` +- Process spawning: prefer Bun Shell `$` for simple commands; use `Bun.spawn`/`Bun.spawnSync` for streaming or process control. Keep existing `child_process` only where its exact semantics are needed. - HTTP clients: `node-fetch`, `axios` → native `fetch` -- Crypto hashing: `node:crypto` → Web Crypto or `Bun.hash` - SQLite: `better-sqlite3` → `bun:sqlite` - Env loading: `dotenv` → Bun loads `.env` automatically +- Runtime text/assets: prefer Bun imports such as `with { type: "text" }` or `Bun.file()` over copy steps or bundled fallback file reads. **DO NOT replace (these work fine in Bun):** -- `os.homedir()` — do NOT replace with `Bun.env.HOME`, `Bun.env.HOME`, or literal `"~"` +- `os.homedir()` — do NOT replace with `Bun.env.HOME` or literal `"~"` - `os.tmpdir()` — do NOT replace with `Bun.env.TMPDIR || "/tmp"` or hardcoded paths - `fs.mkdtempSync()` — do NOT replace with manual path construction - `path.join()`, `path.resolve()`, etc. — these are fine -**Import style:** Use the `node:` prefix with namespace imports only (no named imports from `node:fs` or `node:path`). +**Import style:** Use the `node:` prefix for Node standard-library imports. Namespace imports are common, but named imports are acceptable where the surrounding code already uses them. **Additional Bun conventions:** - Prefer Bun Shell `$` for short, non-streaming commands; use `Bun.spawn` only when you need streaming I/O or process control. -- Use `Bun.file()`/`Bun.write()` for files and `node:fs/promises` for directories. +- Use `Bun.file()`/`Bun.write()` for simple files and `node:fs/promises` for directory-oriented operations. Existing synchronous `node:fs` calls are acceptable when the calling flow is intentionally synchronous. - Avoid `Bun.file().exists()` checks; use `isEnoent` handling in try/catch. - Prefer `Bun.sleep(ms)` over `setTimeout` wrappers. @@ -99,14 +97,14 @@ const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "myapp-")); ## 5) Prefer Bun embeds (no copying) -Do not copy runtime assets or vendor files at build time. +Do not add new runtime asset copy steps. Keep assets in repo and prefer Bun embeds/imports; preserve existing explicit generation workflows such as `packages/coding-agent/src/export/html/template.generated.ts`. - If upstream copies assets into a dist folder, replace with Bun-friendly embeds. - Prompts are static `.md` files; use Bun text imports (`with { type: "text" }`) and Handlebars instead of inline prompt strings. - Use `import.meta.dir` + `Bun.file` to load adjacent non-text resources. - Keep assets in-repo and let the bundler include them. -- Eliminate copy scripts unless the user explicitly requests them. -- If upstream reads a bundled fallback file at runtime, replace filesystem reads with a Bun text embed import. +- Eliminate copy scripts unless the user explicitly requests them or the package already has an intentional generation step. +- If upstream reads a bundled fallback file at runtime, replace filesystem reads with a Bun text embed import unless the current package already uses a generated asset pipeline. - Example (Codex instructions fallback): - `const FALLBACK_PROMPT_PATH = join(import.meta.dir, "codex-instructions.md");` -> removed - `import FALLBACK_INSTRUCTIONS from "./codex-instructions.md" with { type: "text" };` @@ -126,19 +124,19 @@ Treat `package.json` as a contract. Merge intentionally. - Keep existing formatting conventions. - Do not introduce `any` unless required. -- Avoid dynamic imports and inline type imports; use top-level imports only. +- Avoid dynamic imports unless they are required for optional dependencies, startup cost, or runtime-only modules; prefer top-level imports otherwise. - Never build prompts in code; prompts are static `.md` files rendered with Handlebars. -- In coding-agent, never use `console.log`/`console.warn`/`console.error`; use `logger` from `@oh-my-pi/pi-utils`. +- In `packages/coding-agent`, use `logger` from `@oh-my-pi/pi-utils` for internal/runtime logging; CLI command files may use `console.*` for intentional user-facing output. - Use `Promise.withResolvers()` instead of `new Promise((resolve, reject) => ...)`. -- **No `private`/`protected`/`public` keywords on class fields or methods.** Use ES `#` private fields for encapsulation; leave accessible members bare (no keyword). The only exception is constructor parameter properties (`constructor(private readonly x: T)`), where the keyword is required by TypeScript. When porting upstream code that uses `private foo` or `protected bar`, convert to `#foo` (private) or bare `bar` (accessible). +- Prefer ES `#` private fields for new encapsulated state. Constructor parameter properties already exist in current code and are acceptable; do not churn unrelated access modifiers while porting. - Prefer existing helpers and utilities over new ad-hoc code. -- Preserve Bun-first infrastructure changes already made in this repo: - - Runtime is Bun (no Node entry points). + Preserve Bun-first infrastructure changes already made in this repo: + - Runtime is Bun (no Node entry points for the main CLI). - Package manager is Bun (no npm lockfiles). - - Heavy Node APIs (`child_process`, `readline`) are replaced with Bun equivalents. + - Heavy Node APIs should not be introduced casually; current source still uses selected Node APIs (`node:crypto`, `node:readline`, synchronous `node:fs`, and `child_process`) where they fit provider, CLI, or process-control semantics. - Lightweight Node APIs (`os.homedir`, `os.tmpdir`, `fs.mkdtempSync`, `path.*`) are kept. - CLI shebangs use `bun` (not `node`, not `tsx`). - - Packages use source files directly (no TypeScript build step). + - TypeScript packages generally use source files directly; `@oh-my-pi/pi-natives` exports generated native bindings from `packages/natives/native`. - CI workflows run Bun for install/check/test. ## 8) Remove old compatibility layers @@ -241,12 +239,12 @@ rg "case \"" path/to/file.ts Use this as a final pass before you finish: - [ ] Import extensions follow the local package convention (no blanket `.js` stripping) -- [ ] No Node-only APIs in new/ported code +- [ ] No newly introduced Node-only APIs unless they match an existing justified pattern - [ ] All package scopes updated - [ ] `package.json` scripts use Bun - [ ] Prompts are `.md` text imports (no inline prompt strings) -- [ ] No `console.*` in coding-agent (use `logger`) -- [ ] Assets load via Bun embed patterns (no copy scripts) +- [ ] No internal/runtime `console.*` in coding-agent; CLI user-facing output is intentional +- [ ] Assets load via Bun embed/import patterns, or through an existing intentional generation pipeline - [ ] Tests or checks run (or explicitly noted as blocked) - [ ] No functionality regressions (see sections 11-12) @@ -304,8 +302,8 @@ Our fork has architectural decisions that differ from upstream. **Do not port th | Upstream | Our Fork | Reason | | ------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------- | | `FooterDataProvider` class | `StatusLineComponent` | Simpler, integrated status line | -| `ctx.ui.setHeader()` / `ctx.ui.setFooter()` | Stub in non-TUI modes | Implemented in TUI, no-op elsewhere | -| `ctx.ui.setEditorComponent()` | Stub in non-TUI modes | Implemented in TUI, no-op elsewhere | +| `ctx.ui.setHeader()` / `ctx.ui.setFooter()` | No-op stubs in current extension contexts | Not currently wired to replace the TUI status/header UI | +| `ctx.ui.setEditorComponent()` | No-op stubs in current extension contexts | Custom editor replacement is not currently wired | | `InteractiveModeOptions` options object | Positional constructor args (options type still exported) | Keep constructor signature; update the type when upstream adds fields | ### Component Naming @@ -327,9 +325,9 @@ Our fork has architectural decisions that differ from upstream. **Do not port th ### File Consolidation -| Upstream | Our Fork | Reason | -| -------------------------------------------------- | --------------------------------------- | --------------------------------------- | -| `clipboard.ts` + `clipboard-image.ts` (tool files) | `@oh-my-pi/pi-natives` clipboard module | Merged into N-API native implementation | +| Upstream | Our Fork | Reason | +| -------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------- | +| `clipboard.ts` + `clipboard-image.ts` (tool files) | `src/utils/clipboard.ts` backed by `@oh-my-pi/pi-natives` | Native implementation with a small TS wrapper | ### Test Framework @@ -340,11 +338,11 @@ Our fork has architectural decisions that differ from upstream. **Do not port th ### Tool Architecture -| Upstream | Our Fork | Notes | -| ----------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------- | -| `createTool(cwd: string, options?)` | `createTools(session: ToolSession)` via `BUILTIN_TOOLS` registry | Tool factories accept `ToolSession` and can return `null` | -| Per-tool `*Operations` interfaces | Per-tool interfaces remain (`FindOperations`, `GrepOperations`) | Used for SSH/remote overrides | -| Node.js `fs/promises` everywhere | `Bun.file()`/`Bun.write()` for files; `node:fs/promises` for dirs | Prefer Bun APIs when they simplify | +| Upstream | Our Fork | Notes | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| `createTool(cwd: string, options?)` | `createTools(session: ToolSession)` via `BUILTIN_TOOLS` registry | Tool factories accept `ToolSession` and can return `null` | +| Per-tool `*Operations` interfaces | Only current per-tool override interfaces remain (for example `FindOperations`) | Used for SSH/remote overrides where present | +| Node.js `fs/promises` everywhere | Bun file APIs for simple file writes/reads, `node:fs/promises` for dirs, selected sync `node:fs` where needed | Prefer Bun APIs when they simplify | ### Auth Storage @@ -355,17 +353,17 @@ Our fork has architectural decisions that differ from upstream. **Do not port th ### Extensions -| Upstream | Our Fork | -| ----------------------------- | ------------------------------------------ | -| `jiti` for TypeScript loading | Native Bun `import()` | -| `pkg.pi` manifest field | `pkg.omp ?? pkg.pi` (prefer our namespace) | +| Upstream | Our Fork | +| ----------------------------- | ------------------------------------------------- | +| `jiti` for TypeScript loading | Native Bun `import()` | +| `pkg.pi` manifest field | `pkg.omp` preferred; fallback to `pkg.pi` remains | ### Skip These Upstream Features When porting, **skip** these files/features entirely: - `footer-data-provider.ts` — we use StatusLineComponent -- `clipboard-image.ts` — clipboard is in `@oh-my-pi/pi-natives` N-API module +- `clipboard-image.ts` — image clipboard support is exposed through `src/utils/clipboard.ts` backed by `@oh-my-pi/pi-natives` - GitHub workflow files — we have our own CI - `models.generated.ts` — auto-generated, regenerate locally (as models.json instead) diff --git a/docs/porting-to-natives.md b/docs/porting-to-natives.md index 88a6dee61..0fb9b84de 100644 --- a/docs/porting-to-natives.md +++ b/docs/porting-to-natives.md @@ -1,6 +1,6 @@ # Porting to pi-natives (N-API) — Field Notes -This is a practical guide for moving hot paths into `crates/pi-natives` and wiring them through the JS bindings. It exists to avoid the same failures happening twice. +This is a practical guide for moving hot paths into `crates/pi-natives` and wiring them through the generated native package entrypoint. It exists to avoid the same failures happening twice. ## When to port @@ -10,113 +10,129 @@ Port when any of these are true: - JS allocations dominate (string churn, regex backtracking, large arrays). - You already have a JS baseline and can benchmark both versions side by side. - The work is CPU-bound or blocking I/O that can run on the libuv thread pool. -- The work is async I/O that can run on Tokio's runtime (e.g., shell execution). +- The work is async I/O that can run on Tokio's runtime (for example shell execution). -Avoid ports that depend on JS-only state or dynamic imports. N-API exports should be pure, data-in/data-out. Long-running work should go through `task::blocking` (CPU-bound/blocking I/O) or `task::future` (async I/O) with cancellation. +Avoid ports that depend on JS-only state or dynamic imports. N-API exports should be data-in/data-out. Long-running work should go through `task::blocking` (CPU-bound/blocking I/O) or `task::future` (async I/O) with cancellation where the caller needs `timeoutMs` or `AbortSignal`. + +## Current package shape + +`@oh-my-pi/pi-natives` no longer has a `packages/natives/src/` TypeScript wrapper layer. The package root points at generated native artifacts: + +- runtime entry: `packages/natives/native/index.js` +- types entry: `packages/natives/native/index.d.ts` +- loader helpers: `packages/natives/native/loader-state.js` +- embedded manifest: `packages/natives/native/embedded-addon.js` + +Consumers import directly from `@oh-my-pi/pi-natives`. The generated declarations are produced during `bun --cwd=packages/natives run build`. ## Anatomy of a native export **Rust side:** -- Implementation lives in `crates/pi-natives/src/.rs`. If you add a new module, register it in `crates/pi-natives/src/lib.rs`. -- Export with `#[napi]`; snake_case exports are converted to camelCase automatically. Use explicit `js_name` only for true aliases/non-default names. Use `#[napi(object)]` for structs. -- Use `task::blocking(tag, cancel_token, work)` (see `crates/pi-natives/src/task.rs`) for CPU-bound or blocking work. Use `task::future(env, tag, work)` for async work that needs Tokio (e.g., shell sessions). Pass a `CancelToken` when you expose `timeoutMs` or `AbortSignal`. +- Implementation lives in `crates/pi-natives/src/.rs`. +- If you add a new module, register it in `crates/pi-natives/src/lib.rs`. +- Export with `#[napi]`; snake_case exports are converted to camelCase automatically. Use explicit JS names only for true aliases/non-default names. Use `#[napi(object)]` for object-shaped structs. +- For CPU-bound or blocking work, use `task::blocking(tag, cancel_token, work)`. +- For async work that needs Tokio, use `task::future(env, tag, work)`. +- Pass a `CancelToken` when the API exposes `timeoutMs` or `AbortSignal`, and call `heartbeat()` inside long loops. -**JS side:** +**Package/build side:** -- `packages/natives/src/bindings.ts` holds the base `NativeBindings` interface. -- `packages/natives/src//types.ts` defines TS types and augments `NativeBindings` via declaration merging. -- `packages/natives/src/native.ts` imports each `/types.ts` file to activate the declarations. -- `packages/natives/src//index.ts` wraps the `native` binding from `packages/natives/src/native.ts`. -- `packages/natives/src/native.ts` loads the addon and `validateNative` enforces required exports. -- `packages/natives/src/index.ts` re-exports the wrapper for callers in `packages/*`. +- `packages/natives/scripts/build-native.ts` runs napi-rs, installs the `.node` artifact, copies generated `index.js`/`index.d.ts`, and appends enum runtime exports. +- `packages/natives/native/index.js` is the loader that chooses a candidate `.node` file and returns the loaded addon. +- `packages/natives/package.json` exposes only the package root (`@oh-my-pi/pi-natives`). + +**Consumer side:** + +- Update direct imports/callsites in `packages/coding-agent` or `packages/tui` when the new export replaces a JS implementation. +- Keep higher-level policy in consumers unless it belongs in the native primitive itself. ## Porting checklist 1. **Add the Rust implementation** - Put the core logic in a plain Rust function. -- If it’s a new module, add it to `crates/pi-natives/src/lib.rs`. +- If it is a new module, add it to `crates/pi-natives/src/lib.rs`. - Expose it with `#[napi]` so the default snake_case -> camelCase mapping stays consistent. -- Keep signatures owned and simple: `String`, `Vec`, `Uint8Array`, or `Either` for large string/byte inputs. -- For CPU-bound or blocking work, use `task::blocking`; for async work, use `task::future`. Pass a `CancelToken` and call `heartbeat()` inside long loops. +- Keep signatures owned and simple: `String`, `Vec`, `Uint8Array`, `Either`, or `#[napi(object)]` structs. +- For CPU-bound or blocking work, use `task::blocking`; for async work, use `task::future`. +- If exposing cancellation, include `timeout_ms: Option` and `signal: Option>` in options, create `CancelToken::new(...)`, and heartbeat in long loops. -2. **Wire JS bindings** +2. **Build generated bindings** -- Add the types and `NativeBindings` augmentation in `packages/natives/src//types.ts`. -- Import `.//types` in `packages/natives/src/native.ts` to trigger declaration merging. -- Add a wrapper in `packages/natives/src//index.ts` that calls `native`. -- Re-export from `packages/natives/src/index.ts`. +- Run `bun --cwd=packages/natives run build`. +- Confirm the generated `packages/natives/native/index.d.ts` includes the new export with the intended JS name/signature. +- Confirm `packages/natives/native/index.js` still has generated enum exports appended when enum changes are involved. -3. **Update native validation** +3. **Update consumers** -- Add `checkFn("newExport")` in `validateNative` (`packages/natives/src/native.ts`). +- Import the new export directly from `@oh-my-pi/pi-natives`. +- Replace only callsites where the native implementation is faster/equivalent and preserves behavior. +- Remove obsolete JS implementation code in the same change when the native path becomes canonical. 4. **Add benchmarks** - Put benchmarks next to the owning package (`packages/tui/bench`, `packages/natives/bench`, or `packages/coding-agent/bench`). - Include a JS baseline and native version in the same run. - Use `Bun.nanoseconds()` and a fixed iteration count. -- Keep the benchmark inputs small and realistic (actual data seen in the hot path). +- Keep benchmark inputs realistic for the hot path. -5. **Build the native binary** +5. **Run focused verification** -- `bun --cwd=packages/natives run build` - -6. **Run the benchmark** - -- `bun run packages//bench/.ts` (or `bun --cwd=packages/natives run bench`) - -7. **Decide on usage** - -- If native is slower, **keep JS** and leave the native export unused. -- If native is faster, switch call sites to the native wrapper. +- Build the native package. +- Run the benchmark. +- Run the narrow tests or scenario covering the changed export/callsites. ## Pain points and how to avoid them -### 1) Stale `pi_natives.node` prevents new exports +### 1) Stale platform/variant artifacts -The loader prefers the platform-tagged binary in `packages/natives/native` (`pi_natives.-.node`). There is also a fallback `pi_natives.node`. Compiled binaries extract to `~/.omp/natives//pi_natives.-.node`. If any of these are stale, exports won’t update. +The loader probes platform-tagged artifacts in deterministic order. For x64, selected variant candidates are tried before the unsuffixed default fallback: -**Fix:** remove the stale file before rebuilding. +- `modern`: `pi_natives.-modern.node`, then `...-baseline.node`, then `pi_natives..node`. +- `baseline`: `pi_natives.-baseline.node`, then `pi_natives..node`. + +Non-x64 uses `pi_natives..node`. + +Compiled binaries also probe `//...` and a legacy user-data directory before package/executable locations. If any earlier candidate is stale, a new export may appear missing. + +**Fix:** remove stale candidate/cache files and rebuild. ```bash -rm packages/natives/native/pi_natives.linux-x64.node -rm packages/natives/native/pi_natives.node +rm packages/natives/native/pi_natives.-.node +rm packages/natives/native/pi_natives.--modern.node +rm packages/natives/native/pi_natives.--baseline.node bun --cwd=packages/natives run build ``` -If you’re running a compiled binary, delete the cached addon directory: +For compiled binaries, delete the versioned addon cache shown in the loader error (normally under `~/.omp/natives/` unless `$XDG_DATA_HOME/omp` is used). + +### 2) Generated types do not match loaded binary + +This can happen when `native/index.d.ts` was regenerated but the `.node` file being loaded is stale or from a different platform/variant. + +Verify the loaded export set from the actual candidate path: ```bash -rm -rf ~/.omp/natives/ +bun -e 'const tag = `${process.platform}-${process.arch}`; const mod = require(`./packages/natives/native/pi_natives.${tag}.node`); console.log(Object.keys(mod).sort())' ``` -Then verify the export exists in the binary: - -```bash -bun -e 'const tag = `${process.platform}-${process.arch}`; const mod = require(`./packages/natives/native/pi_natives.${tag}.node`); console.log(Object.keys(mod).includes("newExport"));' -``` - -### 2) “Missing exports” errors from `validateNative` - -This is **good** — it prevents silent mismatches. When you see this: - -``` -Native addon missing exports ... Missing: visibleWidth -``` - -it means your binary is stale, the Rust export name (or explicit alias when used) doesn’t match the JS name, or the export never compiled in. Fix the build and the naming mismatch, don’t weaken validation. +Fix the build/candidate mismatch. Do not paper over it with optional consumer checks if the export is required. ### 3) Rust signature mismatch -Keep it simple and owned. `String`, `Vec`, and `Uint8Array` work. Avoid references like `&str` in public exports. If you need structured data, wrap it in `#[napi(object)]` structs. +Keep N-API signatures simple and owned. Avoid borrowed references like `&str` in public exports. If you need structured data, use `#[napi(object)]` structs. If you need callbacks, use napi-rs `ThreadsafeFunction` and keep callback error/value behavior explicit. -### 4) Benchmarking mistakes +### 4) Enum runtime exports -- Don’t compare different inputs or allocations. +napi-rs declarations alone are not enough for JS callers that use enum objects at runtime. `scripts/gen-enums.ts` appends enum objects to `native/index.js`. If you add or change a native enum, verify both `native/index.d.ts` and the generated enum export block in `native/index.js`. + +### 5) Benchmarking mistakes + +- Do not compare different inputs or allocations. - Keep JS and native using identical input arrays. - Run both in the same benchmark file to avoid skew. +- Include enough iterations to smooth startup noise, but keep inputs realistic. ## Benchmark template @@ -124,31 +140,34 @@ Keep it simple and owned. `String`, `Vec`, and `Uint8Array` work. Avoid const ITERATIONS = 2000; function bench(name: string, fn: () => void): number { - const start = Bun.nanoseconds(); - for (let i = 0; i < ITERATIONS; i++) fn(); - const elapsed = (Bun.nanoseconds() - start) / 1e6; - console.log(`${name}: ${elapsed.toFixed(2)}ms total (${(elapsed / ITERATIONS).toFixed(6)}ms/op)`); - return elapsed; + const start = Bun.nanoseconds(); + for (let i = 0; i < ITERATIONS; i++) fn(); + const elapsed = (Bun.nanoseconds() - start) / 1e6; + console.log( + `${name}: ${elapsed.toFixed(2)}ms total (${(elapsed / ITERATIONS).toFixed(6)}ms/op)`, + ); + return elapsed; } bench("feature/js", () => { - jsImpl(sample); + jsImpl(sample); }); bench("feature/native", () => { - nativeImpl(sample); + nativeImpl(sample); }); ``` ## Verification checklist -- `validateNative` passes (no missing exports). -- `NativeBindings` is augmented in `packages/natives/src//types.ts` and the wrapper is re-exported in `packages/natives/src/index.ts`. -- `Object.keys(require(...))` includes your new export. -- Bench numbers recorded in the PR/notes. -- Call site updated **only if** native is faster or equal. +- Generated `native/index.d.ts` includes the new export and intended TS signature. +- The loaded `.node` file's `Object.keys(require(candidate))` includes the new export. +- Runtime enum objects are present when the change adds/changes enums. +- Bench numbers are recorded in the PR/notes. +- Call sites are updated only if native is faster/equal and behavior-compatible. +- Obsolete JS code is removed when the native implementation becomes canonical. ## Rule of thumb -- If native is slower, **do not switch**. Keep the export for future work, but the TUI should stay on the faster path. -- If native is faster, switch the call site and keep the benchmark in place to catch regressions. +- If native is slower, do not switch callsites. Keep or remove the export based on whether it has a near-term owner. +- If native is faster and behavior-compatible, switch callsites and keep a benchmark to catch regressions. diff --git a/docs/provider-streaming-internals.md b/docs/provider-streaming-internals.md index 9974ac1d2..f4df818ee 100644 --- a/docs/provider-streaming-internals.md +++ b/docs/provider-streaming-internals.md @@ -5,7 +5,7 @@ This document explains how token/tool streaming is normalized in `@oh-my-pi/pi-a ## End-to-end flow 1. `streamSimple()` (`packages/ai/src/stream.ts`) maps generic options and dispatches to a provider stream function. -2. Provider stream functions (`anthropic.ts`, `openai-responses.ts`, `google.ts`) translate provider-native stream events into the unified `AssistantMessageEvent` sequence. +2. Provider stream functions translate provider-native stream events into the unified `AssistantMessageEvent` sequence. Current built-ins include Anthropic, OpenAI Responses/Completions/Codex/Azure Responses, Google Gemini/Gemini CLI/Vertex, Bedrock Converse, Ollama, Cursor, plus GitLab Duo/Kimi wrappers and extension-registered custom APIs. 3. Each provider pushes events into `AssistantMessageEventStream` (`packages/ai/src/utils/event-stream.ts`), which throttles delta events and exposes: - async iteration for incremental updates - `result()` for final `AssistantMessage` @@ -18,9 +18,9 @@ All providers emit the same shape (`AssistantMessageEvent` in `packages/ai/src/t - `start` - content block lifecycle triplets: - - text: `text_start` → `text_delta`* → `text_end` - - thinking: `thinking_start` → `thinking_delta`* → `thinking_end` - - tool call: `toolcall_start` → `toolcall_delta`* → `toolcall_end` + - text: `text_start` → `text_delta`\* → `text_end` + - thinking: `thinking_start` → `thinking_delta`\* → `thinking_end` + - tool call: `toolcall_start` → `toolcall_delta`\* → `toolcall_end` - terminal event: - `done` with `reason: "stop" | "length" | "toolUse"` - or `error` with `reason: "aborted" | "error"` @@ -66,9 +66,9 @@ Tool-call argument streaming: - `arguments` are reparsed on each delta via `parseStreamingJson()` - `toolcall_end` reparses once more, then strips `partialJson` -## OpenAI Responses (`openai-responses`) +## OpenAI Responses family (`openai-responses`, `openai-codex-responses`, `azure-openai-responses`) -Source: `packages/ai/src/providers/openai-responses.ts` +Sources: `packages/ai/src/providers/openai-responses.ts`, `openai-codex-responses.ts`, and `azure-openai-responses.ts` Normalization points: @@ -182,7 +182,7 @@ Current design favors responsiveness and simple ordering over bounded-buffer flo `AgentSession` then consumes those events for session-level behaviors: -- TTSR watches `message_update.assistantMessageEvent` for `text_delta` and `toolcall_delta` +- TTSR watches `message_update.assistantMessageEvent` for `text_delta`, `thinking_delta`, and `toolcall_delta` - streaming edit guard inspects `toolcall_delta`/`toolcall_end` on `edit` calls and can abort early - persistence writes finalized messages at `message_end` - auto-retry examines assistant `stopReason === "error"` plus `errorMessage` heuristics @@ -207,12 +207,13 @@ Provider-specific (not fully abstracted): ## Implementation files -- [`../../ai/src/stream.ts`](../packages/ai/src/stream.ts) — provider dispatch, option mapping, API key/session plumbing. +- [`../../ai/src/stream.ts`](../packages/ai/src/stream.ts) — provider dispatch, option mapping, API key/session plumbing, custom API dispatch, and provider-specific credential handling. - [`../../ai/src/utils/event-stream.ts`](../packages/ai/src/utils/event-stream.ts) — generic stream queue + assistant delta throttling. - [`../../ai/src/utils/json-parse.ts`](../packages/ai/src/utils/json-parse.ts) — partial JSON parsing for streamed tool arguments. - [`../../ai/src/providers/anthropic.ts`](../packages/ai/src/providers/anthropic.ts) — Anthropic event translation and tool JSON delta accumulation. -- [`../../ai/src/providers/openai-responses.ts`](../packages/ai/src/providers/openai-responses.ts) — OpenAI Responses event translation and status mapping. -- [`../../ai/src/providers/google.ts`](../packages/ai/src/providers/google.ts) — Gemini stream chunk-to-block translation. +- [`../../ai/src/providers/openai-responses.ts`](../packages/ai/src/providers/openai-responses.ts), [`openai-codex-responses.ts`](../packages/ai/src/providers/openai-codex-responses.ts), [`azure-openai-responses.ts`](../packages/ai/src/providers/azure-openai-responses.ts) — Responses-family event translation and status mapping. +- [`../../ai/src/providers/google.ts`](../packages/ai/src/providers/google.ts), [`google-gemini-cli.ts`](../packages/ai/src/providers/google-gemini-cli.ts), [`google-vertex.ts`](../packages/ai/src/providers/google-vertex.ts) — Gemini stream chunk-to-block translation variants. - [`../../ai/src/providers/google-shared.ts`](../packages/ai/src/providers/google-shared.ts) — Gemini finish-reason mapping and shared conversion rules. +- [`../../ai/src/providers/amazon-bedrock.ts`](../packages/ai/src/providers/amazon-bedrock.ts), [`openai-completions.ts`](../packages/ai/src/providers/openai-completions.ts), [`ollama.ts`](../packages/ai/src/providers/ollama.ts), [`cursor.ts`](../packages/ai/src/providers/cursor.ts) — additional built-in stream adapters using the same event contract. - [`../../agent/src/agent-loop.ts`](../packages/agent/src/agent-loop.ts) — provider stream consumption and `message_update` bridging. - [`../src/session/agent-session.ts`](../packages/coding-agent/src/session/agent-session.ts) — session-level handling of streaming updates, abort, retry, and persistence. diff --git a/docs/python-repl.md b/docs/python-repl.md index d1c6bc53d..0a1dbd8f6 100644 --- a/docs/python-repl.md +++ b/docs/python-repl.md @@ -22,7 +22,6 @@ Tool params: { cells: Array<{ code: string; title?: string }>; timeout?: number; // seconds, clamped to 1..600, default 30 - cwd?: string; reset?: boolean; // reset kernel before first cell only } ``` @@ -71,7 +70,7 @@ There are two gateway paths: ## Kernel lifecycle -Each execution uses a kernel created via `POST /api/kernels` on the selected gateway. +Kernels are created via `POST /api/kernels` on the selected gateway when a retained session needs a kernel or when `per-call` mode starts a request. Kernel startup sequence: @@ -92,10 +91,10 @@ Kernel shutdown: ## Session persistence semantics -`python.kernelMode` controls kernel reuse: +`python.kernelMode` controls retained kernel reuse: - `session` (default) - - Reuses kernel sessions keyed by session identity + cwd. + - Reuses kernel sessions keyed by session file plus cwd when a session file exists; otherwise by cwd. - Execution is serialized per session via a queue. - Idle sessions are evicted after 5 minutes. - At most 4 sessions; oldest is evicted on overflow. @@ -135,10 +134,14 @@ Runtime selection order: When a venv is selected, its bin/Scripts path is prepended to `PATH`. -Kernel env initialization inside Python also: +Kernel startup receives the optional session file path from the executor: + +- `PI_SESSION_FILE` (session state file path) + +`PythonKernel.#initializeKernelEnvironment(...)` then runs init script inside kernel to: - `os.chdir(cwd)` -- injects provided env map into `os.environ` +- injects provided env entries into `os.environ` - ensures cwd is in `sys.path` ## Tool availability and mode selection diff --git a/docs/resolve-tool-runtime.md b/docs/resolve-tool-runtime.md index ed1102844..c8256067d 100644 --- a/docs/resolve-tool-runtime.md +++ b/docs/resolve-tool-runtime.md @@ -1,11 +1,10 @@ # Resolve tool runtime internals -This document explains how preview/apply workflows are modeled in coding-agent and how custom tools can participate via `pushPendingAction`. +This document explains how preview/apply workflows are modeled in coding-agent and how built-in or custom tools can participate via the tool-choice queue and `pushPendingAction`. ## Scope and key files - [`src/tools/resolve.ts`](../packages/coding-agent/src/tools/resolve.ts) -- [`src/tools/pending-action.ts`](../packages/coding-agent/src/tools/pending-action.ts) - [`src/tools/ast-edit.ts`](../packages/coding-agent/src/tools/ast-edit.ts) - [`src/extensibility/custom-tools/types.ts`](../packages/coding-agent/src/extensibility/custom-tools/types.ts) - [`src/extensibility/custom-tools/loader.ts`](../packages/coding-agent/src/extensibility/custom-tools/loader.ts) @@ -15,27 +14,29 @@ This document explains how preview/apply workflows are modeled in coding-agent a `resolve` is a hidden tool that finalizes a pending preview action. -- `action: "apply"` executes `apply(reason)` on the pending action and persists changes. -- `action: "discard"` invokes `reject(reason)` if provided; otherwise drops the action with a default "Discarded" message. +- `action: "apply"` executes the queued action's `apply(reason)` callback and returns that result with resolve metadata. +- `action: "discard"` invokes `reject(reason)` if provided; otherwise returns `Discarded: