can1357
5ff277349c
- Added the `xd://` virtual device protocol (`internal-urls/xd-protocol.ts`, `tools/xdev.ts`): tools declaring `loadMode: "discoverable"` are unmounted from the request tools array and driven via `read xd://` (list/docs+schema) and `write xd://<tool>` (execute), gated by the `tools.xdev` setting (default on) and inlined into the system prompt. - Merged the `irc`, `job`, and `launch` tools into a single `hub` tool (`tools/hub/`, `async/job-manager.ts`): messaging keeps `send`/`inbox`/`list`, job control maps to `wait`/`cancel`/`jobs`, process supervision keeps `start`/`logs`/`stop`/`restart`/`describe` with `ps`, and the unified `wait` races background jobs against peer messages; SDK `IrcTool`/`JobTool`/`LaunchTool` are replaced by `HubTool`. - Removed the hidden `resolve` tool in favor of the `xd://resolve`/`xd://reject`/`xd://propose` resolution devices, auto-including `write` whenever a deferrable tool or plan mode is present. - Removed the BM25 tool-discovery system: the `search_tool_bm25` tool, the `tool-discovery` module, the `tools.discoveryMode`/`mcp.discoveryMode`/`mcp.discoveryDefaultServers`/`tools.essentialOverride` settings, per-tool MCP selection, and the `mcp_tool_selection` message type. - Unified tool presentation on `ToolLoadMode` (`essential`|`discoverable`), replacing the custom-tool `xdev?: boolean` opt-out; custom, extension, MCP, RPC host, image-generation, and TTS tools now default to `discoverable`, and added a `satisfies` predicate to `SoftToolRequirement`. - Removed the standalone `ssh` command tool and `ssh/ssh-executor` (the `ssh://` read/write/search protocol stays), and made `--tools` address hidden built-ins. - Updated collab-web to render `xd://` dispatches and `hub` op families, dropped the `search_tool_bm25`/`ssh`/`report-finding` renderers, refreshed tool docs and prompts, and migrated the affected tests and changelogs.
121 lines
11 KiB
Markdown
121 lines
11 KiB
Markdown
# ast_edit
|
|
|
|
> Preview and apply structural rewrites over source files via native ast-grep.
|
|
|
|
## Source
|
|
- Entry: `packages/coding-agent/src/tools/ast-edit.ts`
|
|
- Model-facing prompt: `packages/coding-agent/src/prompts/tools/ast-edit.md`
|
|
- Key collaborators:
|
|
- `crates/pi-natives/src/ast.rs` — native rewrite planning and file mutation
|
|
- `crates/pi-ast/src/language/mod.rs` — language aliases and extension inference used by the native wrapper.
|
|
- `packages/coding-agent/src/tools/path-utils.ts` — path/glob parsing and multi-path resolution
|
|
- `packages/coding-agent/src/tools/resolve.ts` — preview/apply queueing
|
|
- `packages/coding-agent/src/tools/render-utils.ts` — parse-error dedupe and display caps
|
|
- `packages/coding-agent/src/utils/file-display-mode.ts` — hashline vs line-number diff references
|
|
- `packages/hashline/src/format.ts` — stable hashline header formatting for preview anchors
|
|
- `packages/natives/native/index.d.ts` — JS-visible native binding contract
|
|
|
|
## Inputs
|
|
|
|
| Field | Type | Required | Description |
|
|
| --- | --- | --- | --- |
|
|
| `ops` | `{ pat: string; out: string }[]` | Yes | One or more rewrite rules. `pat` must be non-empty. Duplicate `pat` values fail before native execution. Empty `out` deletes the matched node. |
|
|
| `paths` | `string[]` | Yes | One or more files, directories, globs, or internal URLs with backing files. Empty entries are rejected. Globs are forbidden for internal URLs. |
|
|
|
|
Shared AST pattern grammar and language catalog: see [`ast_grep`](./ast-grep.md#inputs).
|
|
|
|
- `ast_edit` uses the same `$NAME`, `$_`, `$$$NAME`, and `$$$` metavariable semantics.
|
|
- The tool prompt adds rewrite-specific constraints:
|
|
- metavariable names must be uppercase and must stand for whole AST nodes,
|
|
- captures from `pat` are substituted into `out`,
|
|
- each rewrite is a 1:1 structural substitution; one capture cannot expand into multiple sibling nodes unless the grammar itself permits that expansion at that position.
|
|
|
|
## Outputs
|
|
- Single-shot preview result from `ast_edit` itself.
|
|
- Model-facing `content` is one text block showing proposed edits, grouped by file for directory/multi-file runs.
|
|
- Each change renders as two lines. Hashline mode uses `-LINE:before` / `+LINE:after` under a `[PATH#TAG]` header; plain mode uses `-LINE:COLUMN before` / `+LINE:COLUMN after`.
|
|
- Only the first line of each `before`/`after` snippet is shown, truncated to 120 characters in the wrapper.
|
|
- `Limit reached; narrow paths.` and formatted parse issues are appended when applicable.
|
|
- If no rewrites match, text is `No replacements made` plus formatted parse issues when present.
|
|
- `details` includes aggregate preview metadata:
|
|
- `totalReplacements`, `filesTouched`, `filesSearched`, `applied`, `limitReached`
|
|
- optional `parseErrors`, `parseErrorsTotal`, `scopePath`, `files`, `fileReplacements`, `displayContent`, `searchPath`, `cwd`, `meta`
|
|
- The tool always previews first (`applied: false` in the direct result). Actual file writes happen only later through a `write /xdev/resolve` dispatch whose plain-text body is the reason.
|
|
- When preview produced replacements, `ast_edit` also queues a pending resolve action. Successful apply returns a separate resolve dispatch result (on the `write` call), not another `ast_edit` result.
|
|
|
|
## Flow
|
|
1. `AstEditTool.execute()` validates each op in `packages/coding-agent/src/tools/ast-edit.ts`:
|
|
- empty `pat` fails,
|
|
- at least one op is required,
|
|
- duplicate `pat` values fail,
|
|
- ops are converted to a `Record<pattern, replacement>`.
|
|
2. The wrapper reads `PI_MAX_AST_FILES` via `$envpos(..., 1000)` and uses that as the native `maxFiles` cap for both preview and apply.
|
|
3. Path normalization, internal URL handling, missing-path partitioning, and multi-path resolution follow the same `path-utils.ts` flow as `ast_grep`.
|
|
4. The scope's `isDirectory` flag (set by a stat in `resolveToolSearchScope`) decides whether to render grouped directory output.
|
|
5. `runAstEditOnce(...)` always runs native `astEdit(...)` with `dryRun: true` and `failOnParseError: false` on the first pass.
|
|
6. Native `ast_edit` in `crates/pi-natives/src/ast.rs`:
|
|
- normalizes the rewrite map and sorts rules by pattern string,
|
|
- resolves strictness (`smart` by default),
|
|
- collects candidate files from a file or gitignore-aware directory scan,
|
|
- infers a single language for the whole call unless `lang` was supplied,
|
|
- compiles every rewrite pattern for that language,
|
|
- parses each file, skips files with syntax-error trees, collects `replace_by(...)` edits for every match, enforces replacement and file caps, and returns textual before/after slices plus source ranges.
|
|
7. The TS wrapper deduplicates and caps parse errors, groups changes by file, and renders preview diff lines.
|
|
8. If preview found replacements and `applied` is false, `queueResolveHandler(...)` registers a non-forcing pending resolve invoker. While it is pending the session surfaces a `SoftToolRequirement` (`toolName: "write"` with a `/xdev/resolve` or `/xdev/reject` `satisfies` predicate) carrying the resolve reminder; the agent runtime injects the reminder and forces `write` only if the model declines that turn (no per-preview `tool_choice` cache bust).
|
|
9. On a `write /xdev/resolve` dispatch, the queued callback reruns the same rewrite set with `dryRun: false`, recomputes counts, and returns an error result if the live result no longer matches the preview (`stalePreview`). The current implementation compares replacement totals and per-file counts after the rerun; if the new run has already written different counts, the result is marked error.
|
|
10. On a non-stale apply, the callback returns `Applied N replacements in M files.` (in hashline mode followed by fresh `[path#tag]` snapshot headers re-recorded from the post-apply content); on discard (`write /xdev/reject`), the dispatch returns a discard message without mutating files.
|
|
|
|
## Modes / Variants
|
|
- Single file: preview or apply against one file.
|
|
- Directory + optional glob: native scan walks the directory, then filters by compiled glob.
|
|
- Multiple explicit paths/globs: wrapper unions them into one synthetic scope or runs per-target native calls when paths only meet at root.
|
|
- Internal URL inputs: only supported when the router resolves them to a backing file path.
|
|
- Preview mode: always the direct `ast_edit` tool result.
|
|
- Apply mode: only reachable through the queued resolve callback (a `write /xdev/resolve` or `/xdev/reject` dispatch) after a preview.
|
|
- Hashline output mode vs plain line/column mode: controlled by `resolveFileDisplayMode()`.
|
|
|
|
## Side Effects
|
|
- Filesystem
|
|
- Preview reads files and scans directories.
|
|
- Apply rewrites files in place with `std::fs::write(...)`, but only when the computed output differs from the original source.
|
|
- Session state (transcript, memory, jobs, checkpoints, registries)
|
|
- Registers a non-forcing pending resolve invoker through `queueResolveHandler(...)`.
|
|
- Surfaces a `SoftToolRequirement` (with the resolve reminder) while pending; the agent runtime forces `write` only on non-compliance — no steering message and no per-preview forced tool choice.
|
|
- User-visible prompts / interactive UI
|
|
- Direct `ast_edit` results are previews.
|
|
- Follow-up apply/discard is exposed through the `/xdev/resolve` and `/xdev/reject` device writes.
|
|
- Background work / cancellation
|
|
- Native preview/apply work runs on a blocking worker via `task::blocking(...)`.
|
|
- Cancellation and optional native timeout are cooperative through `CancelToken::heartbeat()`.
|
|
|
|
## Limits & Caps
|
|
- File cap exposed by the wrapper: `PI_MAX_AST_FILES`, default `1000`, in `packages/coding-agent/src/tools/ast-edit.ts`.
|
|
- Native `maxFiles` and `maxReplacements` are both clamped to at least `1` when provided in `crates/pi-natives/src/ast.rs`.
|
|
- The wrapper never sets `maxReplacements`; native behavior therefore defaults to effectively unbounded replacements for a run.
|
|
- Parse issues are deduplicated and capped at `PARSE_ERRORS_LIMIT = 20` entries via `capParseErrors(...)` in `packages/coding-agent/src/tools/render-utils.ts`; `details.parseErrors` carries the capped list and `details.parseErrorsTotal` the pre-cap deduplicated count.
|
|
- Directory scans use `include_hidden: true`, `use_gitignore: true`, and skip `node_modules` unless the glob text explicitly mentions `node_modules` in `crates/pi-natives/src/ast.rs`.
|
|
- No separate glob-expansion count cap exists. Candidate count is whatever the resolved path/glob expands to after gitignore filtering, then native `maxFiles` stops mutations after the configured number of touched files.
|
|
- Preview text truncates each rendered `before` and `after` first line to 120 characters in `packages/coding-agent/src/tools/ast-edit.ts`.
|
|
|
|
## Errors
|
|
- TS wrapper throws `ToolError` for empty patterns, duplicate rewrite patterns, empty path entries, unsupported internal-URL globs, internal URLs without `sourcePath`, and missing paths.
|
|
- Native code returns hard errors for:
|
|
- inability to infer one language across all candidates when `lang` is absent,
|
|
- unsupported explicit `lang`,
|
|
- bad glob compilation or unreadable search roots,
|
|
- overlapping computed edits (`Overlapping replacements detected; refine pattern to avoid ambiguous edits`),
|
|
- out-of-bounds edit ranges or non-UTF-8 replacement text,
|
|
- write failures during apply,
|
|
- cancellation or timeout.
|
|
- With `failOnParseError: false` (the wrapper always uses this), pattern compile failures and file parse failures become `parseErrors` instead of aborting the whole run.
|
|
- If every rewrite pattern fails to compile, native `ast_edit` returns a successful zero-replacement result with `parseErrors` populated.
|
|
- Files containing tree-sitter error nodes are skipped for rewriting; they do not get partial edits.
|
|
- Apply can fail after a successful preview if the preview becomes stale. The resolve callback compares replacement totals and per-file counts and returns an error result rather than silently reporting success for a mismatched preview.
|
|
|
|
## Notes
|
|
- `ast_edit` does not expose the native `lang`, `strictness`, `selector`, `maxReplacements`, `failOnParseError`, or `timeoutMs` fields to the model. The runtime fixes the call shape to a preview-first, smart-strictness, best-effort parse mode.
|
|
- Because the wrapper does not expose `lang`, mixed-language rewrites only succeed when every candidate infers to the same canonical language. This is stricter than `ast_grep`.
|
|
- Idempotency is not enforced syntactically. A rewrite like `foo($A) -> foo($A)` previews zero changes because output equals input; a rewrite that keeps matching its own output may still produce replacements on repeated calls.
|
|
- Rewrites are accumulated per file, then applied from the end of the file backward after an overlap check. Independent matches can coexist; overlapping matches abort the run.
|
|
- Native rewrite rule order is by pattern-string sort, not by the original `ops` array order, because `normalize_rewrite_map(...)` sorts the `(pattern, rewrite)` pairs.
|
|
- Preview/apply parity is validated by totals and per-file counts after the apply rerun, not by a byte-for-byte diff of every replacement payload. |