chore: update stale tests
This commit is contained in:
@@ -26,6 +26,8 @@ target/
|
||||
|
||||
# OS
|
||||
*.tmp
|
||||
# Bun compile temp artifacts, left in cwd if a `Bun.build({ compile })` run is killed mid-write
|
||||
*.bun-build
|
||||
.DS_Store
|
||||
|
||||
# Logs & profiling
|
||||
|
||||
+34
-30
@@ -8,7 +8,7 @@
|
||||
- Key collaborators:
|
||||
- `packages/coding-agent/src/utils/edit-mode.ts` — selects active edit mode
|
||||
- `packages/hashline/src/grammar.lark` — canonical constrained-decoding grammar
|
||||
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `DEL`, `INS`)
|
||||
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `CUT`, `INS`, `PASTE`)
|
||||
- `packages/hashline/src/input.ts` — parses `[PATH#TAG]` sections
|
||||
- `packages/hashline/src/tokenizer.ts` / `packages/hashline/src/parser.ts` — tokenizes and parses ops
|
||||
- `packages/hashline/src/apply.ts` — applies parsed edits to file text
|
||||
@@ -30,18 +30,21 @@ Patch language inside `input`:
|
||||
- **Operations**:
|
||||
- `SWAP N.=M:` — replace original lines N.=M with the body rows below.
|
||||
- `SWAP.BLK N:` — replace the whole tree-sitter block beginning on line N (its header line through its closing line) with the body rows. The line span is resolved at apply time from the file's parse tree; point N at the line that opens the construct. The resolved span is exactly the node that begins on line N — a leading decorator, attribute, or doc-comment is a separate node and is not included; point N at the first decorator line (Python wraps `@dec` + `def` as one block) or fall back to `SWAP N.=M:` to take a leading line-comment that parses as its own node (e.g. Rust `///`). On success the result echoes the matched span (`SWAP.BLK N → resolved lines A-B`). Errors (and steers to `SWAP N.=M:`) when the language is unsupported, line N is blank or a closing delimiter, no node begins there, or the resolved block has a syntax error.
|
||||
- `DEL N.=M` — delete original lines N.=M. No body.
|
||||
- `DEL.BLK N` — delete the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`DEL.BLK N → resolved lines A-B`). Same resolution failure modes and `DEL N.=M` fallback.
|
||||
- `CUT N.=M` — delete original lines N.=M and capture them in the clipboard. No body. A standalone cut is valid; the latest cut replaces the clipboard contents.
|
||||
- `CUT.BLK N` — delete and capture the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`CUT.BLK N → resolved lines A-B`). Same resolution failure modes and `CUT N.=M` fallback.
|
||||
- `INS.PRE N:` — insert body rows immediately before line N.
|
||||
- `INS.POST N:` — insert body rows immediately after line N.
|
||||
- `INS.BLK.POST N:` — insert body rows after the last line of the tree-sitter block beginning on line N. Point N at the line that opens the construct, never its closing delimiter / last visible line; if you can see the last line already, use plain `INS.POST M:`. An anchor that can't resolve to a block is lowered to plain `INS.POST N:` with a warning instead of failing the patch.
|
||||
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all three block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `DEL.BLK` drops the section, `SWAP.BLK` rewrites it, `INS.BLK.POST` lands after it. A heading with no body resolves to a single line and falls back to the plain op like any other single-line block.
|
||||
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all four block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `CUT.BLK` drops and captures the section, `SWAP.BLK` rewrites it, and `INS.BLK.POST` / `PASTE.BLK.POST` land after it. A heading with no body resolves to a single line and is rejected with guidance to use the corresponding plain line op.
|
||||
- `INS.HEAD:` — insert body rows at the start of the file.
|
||||
- `INS.TAIL:` — insert body rows at the end of the file.
|
||||
- `PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` / `PASTE.TAIL` — insert the clipboard at that position. No body. An empty clipboard is an error.
|
||||
- `PASTE.BLK.POST N` — insert the clipboard after the resolved block's last line. An unresolvable anchor lowers to `PASTE.POST N` with a warning, matching `INS.BLK.POST`.
|
||||
- **Clipboard**: operations execute top-to-bottom across all patch sections. The latest `CUT` wins; `PASTE` does not consume the clipboard and may be repeated. The coding agent persists the register across edit calls in the same session, enabling cross-file moves. Keep each path under one header when clipboard operations would otherwise be interleaved around another file's section.
|
||||
- **Body rows**:
|
||||
- Only body-bearing headers end in `:`.
|
||||
- Every body row is `+TEXT`; `+` alone adds a blank line.
|
||||
- `DEL` never has body rows.
|
||||
- `CUT` and `PASTE` never have body rows.
|
||||
- There is no repeat row kind. To keep a line, leave it out of every range; split edits into multiple hunks when needed.
|
||||
- `-` rows are invalid. Literal Markdown bullets or text beginning with `-` / `+` must be written as `+- item` / `++ item`.
|
||||
|
||||
@@ -52,7 +55,7 @@ Anchors come from `read`/`grep` output. `read` emits a `[PATH#TAG]` header from
|
||||
The canonical grammar is strict, but the hand parser accepts a few non-dangerous variants:
|
||||
|
||||
- `SWAP N:` — accepted as `SWAP N.=N:`.
|
||||
- `DEL N` — accepted as single-line delete.
|
||||
- `CUT N` — accepted as a single-line cut/delete.
|
||||
- Missing trailing colon on `SWAP` or `INS` — accepted.
|
||||
- `SWAP N-M:`, `SWAP N…M:`, `SWAP N M:`, and legacy `SWAP N..M:` — accepted as `SWAP N.=M:`.
|
||||
- Bare body rows with no `+` prefix are auto-prepended with `+` and a `BARE_BODY_AUTO_PIPED_WARNING` is appended.
|
||||
@@ -61,16 +64,17 @@ The canonical grammar is strict, but the hand parser accepts a few non-dangerous
|
||||
- Some malformed bracketed headers are recovered after stripping apply-patch path noise such as `Update File:` / `Add File:` and extra `***`, but the recovered header still needs a valid four-hex tag for the patcher to apply it.
|
||||
- `*** Update File:` / `*** Add File:` / `*** Delete File:` / `*** Move to:` apply_patch sentinels inside the diff body throw an `apply_patch sentinel … is not valid in hashline` error.
|
||||
- `@@`-bracketed hunk headers are rejected with guidance to write a verb header.
|
||||
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `DEL`.
|
||||
- `DEL N.=M:` and any body rows under `DEL` / `DEL.BLK` are rejected.
|
||||
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` (no body rows) is treated as `DEL N.=M`.
|
||||
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `CUT`.
|
||||
- A trailing colon on `CUT N.=M:` / `CUT.BLK N:` is tolerated and ignored, but body rows under `CUT`, `CUT.BLK`, or any `PASTE` form are rejected.
|
||||
- Bare `PASTE` is rejected because the insertion position is required.
|
||||
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` deletes the range, though `CUT N.=M` is the canonical deletion form.
|
||||
- `-` body rows are rejected with `MINUS_ROW_REJECTED` unless the hunk is unambiguously a Markdown bullet list (see Tolerated input shapes).
|
||||
- `SWAP.BLK N:` / `DEL.BLK N` / `INS.BLK.POST N:` require a wired tree-sitter resolver; `SWAP.BLK` and `INS.BLK.POST` additionally need at least one `+TEXT` body row, while `DEL.BLK` takes none. An unresolvable block (unsupported language, blank/closing-delimiter line, no node beginning on N, or a syntax error in the resolved block) rejects a `SWAP.BLK` / `DEL.BLK` on the apply/final-preview path (the streaming preview silently drops it instead). `INS.BLK.POST N:` is never rejected this way — it is lowered to plain `INS.POST N:` with a warning: a closing-delimiter-anchor warning when line N is a pure closer (inserting after that end is exactly what the plain form does), a generic unresolved-anchor warning otherwise.
|
||||
- `SWAP.BLK N:` / `CUT.BLK N` / `INS.BLK.POST N:` / `PASTE.BLK.POST N` consult the wired tree-sitter resolver. `SWAP.BLK` and `INS.BLK.POST` need at least one `+TEXT` body row; `CUT.BLK` and `PASTE.BLK.POST` take none. A null resolution rejects `SWAP.BLK` / `CUT.BLK` on the apply or final-preview path (the streaming preview silently drops it), while `INS.BLK.POST` / `PASTE.BLK.POST` lower to the corresponding plain `POST` form with a warning. A single-line resolution rejects every block form with guidance to use its plain line equivalent.
|
||||
|
||||
## Outputs
|
||||
- Single-shot tool result; hashline mode does not use the staged preview/apply devices (`/xdev/resolve`, `/xdev/reject`).
|
||||
- `content` contains one text block per call. For a successful single-file edit it is the post-edit `[path#TAG]` section header (a fresh snapshot tag for the written content), followed by a compact diff preview from `packages/hashline/src/diff-preview.ts` when one is emitted.
|
||||
- When the patch used `SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST` ops (and the apply matched the tagged content), one `SWAP.BLK N → resolved lines A-B (K lines)` line per block op (single-line spans render `resolved line A (1 line)`; INS.BLK.POST appends `; body lands after line B`) is inserted between the `[PATH#TAG]` header and the diff preview, so the caller can confirm tree-sitter resolved the construct it intended.
|
||||
- When the patch used `SWAP.BLK` / `CUT.BLK` / `INS.BLK.POST` / `PASTE.BLK.POST` ops (and the apply matched the tagged content), one `<OP> N → resolved lines A-B (K lines)` line per block op is inserted between the `[PATH#TAG]` header and the diff preview. Single-line spans render `resolved line A (1 line)`; `INS.BLK.POST` appends `body lands after line B`, and `PASTE.BLK.POST` appends `clipboard lands after line B`.
|
||||
- Parse, apply, or recovery warnings are appended as:
|
||||
|
||||
```text
|
||||
@@ -126,11 +130,11 @@ INS.PRE 5:
|
||||
+console.log(X + Y);
|
||||
```
|
||||
|
||||
Delete lines 4.=5 entirely:
|
||||
Delete lines 4.=5 entirely and leave them in the clipboard:
|
||||
|
||||
```text
|
||||
[a.ts#0A3B]
|
||||
DEL 4.=5
|
||||
CUT 4.=5
|
||||
```
|
||||
|
||||
Insert at start and end of file:
|
||||
@@ -143,20 +147,19 @@ INS.TAIL:
|
||||
+// trailer
|
||||
```
|
||||
|
||||
Multi-file:
|
||||
Move line 4 from `src/a.ts` to after line 20 in `src/b.ts`:
|
||||
```text
|
||||
[src/a.ts#0A3B]
|
||||
SWAP 4.=4:
|
||||
+const enabled = true;
|
||||
CUT 4
|
||||
[src/b.ts#1F7C]
|
||||
DEL 20
|
||||
PASTE.POST 20
|
||||
```
|
||||
|
||||
## Limits & Caps
|
||||
- File snapshot tags are exactly four uppercase-hex chars — content-derived hashes (`computeFileHash()`) recorded in the per-session snapshot store.
|
||||
- The visible mismatch report shows 2 lines of context on each side (`MISMATCH_CONTEXT`) in `packages/hashline/src/messages.ts`.
|
||||
- Stale-anchor recovery uses `fuzzFactor: 0` in `packages/hashline/src/recovery.ts`.
|
||||
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and hunk keyword constants are `SWAP` / `DEL` / `INS` (`packages/hashline/src/format.ts`).
|
||||
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and line/clipboard hunk keyword constants are `SWAP` / `CUT` / `INS` / `PASTE` (`packages/hashline/src/format.ts`).
|
||||
|
||||
## Errors
|
||||
- Missing section header:
|
||||
@@ -164,28 +167,29 @@ DEL 20
|
||||
- Missing tag for any section:
|
||||
- `Missing hashline snapshot tag for <path>; use \`[<path>#tag]\` from your latest read/search output. To create a new file, use the write tool.`
|
||||
- Stray payload line:
|
||||
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
|
||||
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
|
||||
- Minus row (unless auto-piped as an unambiguous Markdown bullet — see Tolerated input shapes):
|
||||
- ``line N: `-` rows are not valid; the range already names the lines being changed. For Markdown bullets or other literal `-` lines, prefix the literal row with `+`: `+- item`.``
|
||||
- Empty body-bearing hunk:
|
||||
- `line N: \`INS\` needs at least one \`+TEXT\` body row.`
|
||||
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`DEL.BLK N\`.`
|
||||
- Unresolvable block anchor — `SWAP.BLK` / `DEL.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
|
||||
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by a blank line and numbered `*`-marked context rows around line X (same shape as the mismatch preview). `DEL.BLK X` produces the same message with a `DEL X.=M` fallback.
|
||||
- `INS.BLK.POST X:` never reaches this error — an unresolvable anchor is lowered to plain `INS.POST X:` with a warning (see Tolerated input shapes).
|
||||
- Delete with body:
|
||||
- `line N: \`DEL N.=M\` does not take body rows. Remove the body, or use \`SWAP N.=M:\`.`
|
||||
- `line N: \`DEL.BLK N\` does not take body rows. Remove the body, or use \`SWAP.BLK N:\`.`
|
||||
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`CUT.BLK N\`.`
|
||||
- Unresolvable block anchor — `SWAP.BLK` / `CUT.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
|
||||
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by numbered context and, when available, a nearby block suggestion. `CUT.BLK X` produces the corresponding message with a `CUT X.=M` fallback.
|
||||
- `INS.BLK.POST X:` and `PASTE.BLK.POST X` never reach this error when no block resolves — they lower to plain `INS.POST X:` / `PASTE.POST X` with a warning.
|
||||
- Clipboard operation errors:
|
||||
- `line N: \`CUT N.=M\` captures + deletes lines and takes no body rows. To replace lines with new content, use \`SWAP N.=M:\`.`
|
||||
- `line N: \`PASTE\` inserts the clipboard content and takes no \`+\` body rows. To insert literal text, use \`INS\`.`
|
||||
- `line N: \`PASTE\` found nothing in the clipboard. Ops run top-to-bottom across the whole patch (sections included): put \`CUT N.=M\` or \`CUT.BLK N\` above the \`PASTE\`.`
|
||||
- Range out of order:
|
||||
- `line N: range A.=B ends before it starts.`
|
||||
- Overlapping hunks on the same anchor:
|
||||
- `line N: anchor line X is already targeted by another hunk on line Y. Issue ONE hunk per range; payload is only the final desired content, never a before/after pair.`
|
||||
- apply_patch / unified-diff contamination:
|
||||
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
|
||||
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
|
||||
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
|
||||
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
|
||||
- `line N: \`@@\`-bracketed hunk header "@@ …" is not valid in hashline. Drop the \`@@ ... @@\` brackets and write a verb header such as \`SWAP N.=M:\`.`
|
||||
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`DEL N\` to delete.`
|
||||
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`DEL ${bareRange[1]}.=${bareRange[2]}\`.`
|
||||
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`CUT N\` to delete.`
|
||||
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`CUT ${bareRange[1]}.=${bareRange[2]}\`.`
|
||||
- Out-of-range anchor:
|
||||
- `Line N does not exist (file has M lines)`
|
||||
- Stale snapshot tag: the `Patcher` first attempts snapshot-based recovery. When recovery cannot prove a valid result it throws `MismatchError`, which distinguishes recognized-but-drifted hashes from never-recorded hashes. The error includes the current file hash plus context around each anchor.
|
||||
|
||||
@@ -19,9 +19,9 @@ interface TemplateProbeResult {
|
||||
}
|
||||
|
||||
const expectedTemplate: TemplateProbeResult = {
|
||||
chars: 377_268,
|
||||
bytes: 377_446,
|
||||
sha256: "023cf3773498db52ff0ec508b22ef2c002710f9e9fcb90fea361659d5b0205df",
|
||||
chars: 377_286,
|
||||
bytes: 377_464,
|
||||
sha256: "6598b2d74f905e3105474ea8033c938e92df4c33035421603ea363d72d1c81c1",
|
||||
stableCache: true,
|
||||
assetsRemoved: 0,
|
||||
};
|
||||
@@ -78,6 +78,7 @@ function composeExpectedTemplate(): string {
|
||||
async function runProbe(command: string[]): Promise<TemplateProbeResult> {
|
||||
const proc = Bun.spawn(command, {
|
||||
cwd: unrelatedCwd,
|
||||
stdin: "ignore",
|
||||
stderr: "pipe",
|
||||
stdout: "pipe",
|
||||
});
|
||||
@@ -149,8 +150,9 @@ describe("HTML export template", () => {
|
||||
fs.writeFileSync(staleAssetPath, "stale");
|
||||
const build = Bun.spawn([process.execPath, "run", "gen:bundle"], {
|
||||
cwd: packageDir,
|
||||
stderr: "pipe",
|
||||
stdin: "ignore",
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
});
|
||||
const [buildStdout, buildStderr, buildExitCode] = await Promise.all([
|
||||
new Response(build.stdout).text(),
|
||||
@@ -162,8 +164,9 @@ describe("HTML export template", () => {
|
||||
|
||||
const proc = Bun.spawn([process.execPath, "pm", "pack", "--dry-run", "--ignore-scripts"], {
|
||||
cwd: packageDir,
|
||||
stderr: "pipe",
|
||||
stdin: "ignore",
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
});
|
||||
const [stdout, stderr, exitCode] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
@@ -195,6 +198,7 @@ describe("HTML export template", () => {
|
||||
|
||||
test("does not retain source asset strings during a static import", async () => {
|
||||
const proc = Bun.spawn([process.execPath, heapProbePath], {
|
||||
stdin: "ignore",
|
||||
stderr: "pipe",
|
||||
stdout: "pipe",
|
||||
});
|
||||
|
||||
@@ -220,8 +220,9 @@ describe("SelectorController.handleResumeSession preflight flush", () => {
|
||||
|
||||
selector!.handleInput("\n");
|
||||
expect(selectionPromise).toBeDefined();
|
||||
await expect(selectionPromise!).rejects.toBe(switchError);
|
||||
await selectionPromise;
|
||||
|
||||
expect(ctx.showError).toHaveBeenCalledWith("switch failed");
|
||||
expect(ctx.settings.flush).toHaveBeenCalledTimes(1);
|
||||
expect(switchSession).toHaveBeenCalledWith(session.path);
|
||||
expect(hide).toHaveBeenCalledTimes(1);
|
||||
|
||||
@@ -9,27 +9,25 @@
|
||||
* every run counts as its own hunk.
|
||||
*
|
||||
* Shapes are reported at two levels:
|
||||
* - per tool call — one sample per edit call;
|
||||
* - per user request — all edit calls between one user message and the next
|
||||
* grouped into one sample (one benchmark fixture ≙ one prompt, and the
|
||||
* runner allows multiple edit calls per prompt). Retried/overlapping edits
|
||||
* within a request are summed, so request sizes lean high, and a request
|
||||
* spans every file a long agentic turn touched — an upper bound for
|
||||
* single-file fixtures.
|
||||
* - per tool call — one sample per edit call. Fixtures are scored on the
|
||||
* final input→expected diff, and a single call's diff is the closest
|
||||
* measured proxy for a net diff, so **this is the calibration reference**
|
||||
* for `generate.ts`.
|
||||
* - per request × file — all edit calls to one file between one user message
|
||||
* and the next, concatenated. This double-counts retries, overlapping
|
||||
* edits, and reversals (no per-file net reconstruction), so treat it as an
|
||||
* **upper bound** on how much a single-prompt fixture could plausibly ask
|
||||
* for, not as a target.
|
||||
*
|
||||
* Parse coverage is printed so a format skew cannot silently bias the numbers.
|
||||
* Reference measurement (2026-07, 2,000 newest sessions against this repo,
|
||||
* 99.9% coverage):
|
||||
*
|
||||
* per request: changed lines: 1 → 6% 2-5 → 11% 6-20 → 20% 21-60 → 25% 61+ → 39%
|
||||
* hunks: 1 → 11% 2 → 8% 3+ → 81% (median 40 changed lines)
|
||||
* per call: changed lines: 1 → 23% 2-5 → 30% 6-20 → 29% 21-60 → 14% 61+ → 5%
|
||||
* hunks: 1 → 48% 2 → 21% 3+ → 31% (median 5 changed lines)
|
||||
* request×file: changed lines: 1 → 18% 2-5 → 28% 6-20 → 28% 21-60 → 17% 61+ → 9%
|
||||
* hunks: 1 → 37% 2 → 21% 3+ → 43% (median 7; upper bound)
|
||||
* op mix (both levels): replace 55% insert 32% delete 12%
|
||||
*
|
||||
* `generate.ts` fixtures are single-file, single-prompt tasks: the suite is
|
||||
* calibrated to sit between the two levels — per-call shapes for token fixes,
|
||||
* request-leaning shapes (large blocks, multi-hunk composites) for the rest.
|
||||
*/
|
||||
import * as fs from "node:fs/promises";
|
||||
import * as path from "node:path";
|
||||
@@ -157,26 +155,46 @@ function runsFromDiff(diff: string): { runs: ChangeRun[] | null; format: "number
|
||||
return { runs: runsFromNumberedDiff(diff), format: "numbered" };
|
||||
}
|
||||
|
||||
/** Per-session parse results: change runs per edit call and per user request, plus coverage counters. */
|
||||
/** Per-session parse results: change runs per edit call and per request×file, plus coverage counters. */
|
||||
interface SessionScan {
|
||||
edits: ChangeRun[][];
|
||||
/** All edit-call runs between one user message and the next, concatenated. */
|
||||
requests: ChangeRun[][];
|
||||
/** All edit-call runs targeting one file between one user message and the next, concatenated. */
|
||||
requestFiles: ChangeRun[][];
|
||||
formats: Map<string, number>;
|
||||
skipped: number;
|
||||
}
|
||||
|
||||
/** Resolve the edited file path from tool-call args across edit-tool variants. */
|
||||
function extractCallPath(args: Record<string, unknown>): string | null {
|
||||
if (typeof args.path === "string" && args.path) return args.path;
|
||||
if (typeof args.input === "string") {
|
||||
const header =
|
||||
args.input.match(/^\u00b6([^#\n]+)#/) ??
|
||||
args.input.match(/^\[([^#\]\n]+)#/) ??
|
||||
args.input.match(/^@@ ([^\n]+)$/m);
|
||||
if (header) return header[1].trim();
|
||||
}
|
||||
if (Array.isArray(args.edits) && args.edits.length > 0 && args.edits[0] && typeof args.edits[0] === "object") {
|
||||
const nested = (args.edits[0] as Record<string, unknown>).path;
|
||||
if (typeof nested === "string" && nested) return nested;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Extract change runs for every successful edit in one session JSONL file. */
|
||||
async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
|
||||
const scan: SessionScan = { edits: [], requests: [], formats: new Map(), skipped: 0 };
|
||||
const scan: SessionScan = { edits: [], requestFiles: [], formats: new Map(), skipped: 0 };
|
||||
const stream = Bun.file(sessionFile).stream();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = "";
|
||||
let currentRequest: ChangeRun[] = [];
|
||||
const callPaths = new Map<string, string>();
|
||||
let currentRequest = new Map<string, ChangeRun[]>();
|
||||
|
||||
const flushRequest = (): void => {
|
||||
if (currentRequest.length > 0) scan.requests.push(currentRequest);
|
||||
currentRequest = [];
|
||||
for (const runs of currentRequest.values()) {
|
||||
if (runs.length > 0) scan.requestFiles.push(runs);
|
||||
}
|
||||
currentRequest = new Map();
|
||||
};
|
||||
|
||||
const handleRecord = (record: unknown): void => {
|
||||
@@ -185,11 +203,23 @@ async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
|
||||
if (rec.type !== "message" || !rec.message || typeof rec.message !== "object") return;
|
||||
const message = rec.message as Record<string, unknown>;
|
||||
// A user message starts a new request; everything until the next one
|
||||
// belongs to the same prompt (one benchmark fixture ≙ one request).
|
||||
// belongs to the same prompt (one benchmark fixture ≙ one prompt on one file).
|
||||
if (message.role === "user") {
|
||||
flushRequest();
|
||||
return;
|
||||
}
|
||||
if (message.role === "assistant" && Array.isArray(message.content)) {
|
||||
for (const block of message.content) {
|
||||
if (!block || typeof block !== "object") continue;
|
||||
const call = block as Record<string, unknown>;
|
||||
if (call.type !== "toolCall" || !EDIT_TOOL_NAMES[String(call.name)]) continue;
|
||||
const args =
|
||||
call.arguments && typeof call.arguments === "object" ? (call.arguments as Record<string, unknown>) : {};
|
||||
const callPath = extractCallPath(args);
|
||||
if (typeof call.id === "string" && callPath) callPaths.set(call.id, callPath);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (message.role !== "toolResult" || !EDIT_TOOL_NAMES[String(message.toolName)] || message.isError) return;
|
||||
const details =
|
||||
message.details && typeof message.details === "object"
|
||||
@@ -201,7 +231,12 @@ async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
|
||||
const { runs, format } = runsFromDiff(diff);
|
||||
if (runs) {
|
||||
scan.edits.push(runs);
|
||||
currentRequest.push(...runs);
|
||||
// Unknown path → each call is its own group (conservative: no false merging).
|
||||
const callId = typeof message.toolCallId === "string" ? message.toolCallId : `#${scan.edits.length}`;
|
||||
const fileKey = callPaths.get(callId) ?? `call:${callId}`;
|
||||
const group = currentRequest.get(fileKey) ?? [];
|
||||
group.push(...runs);
|
||||
currentRequest.set(fileKey, group);
|
||||
scan.formats.set(format, (scan.formats.get(format) ?? 0) + 1);
|
||||
} else {
|
||||
scan.skipped++;
|
||||
@@ -278,7 +313,7 @@ async function main(): Promise<number> {
|
||||
};
|
||||
|
||||
const perCall = newCounters();
|
||||
const perRequest = newCounters();
|
||||
const perRequestFile = newCounters();
|
||||
const formatCounts = new Map<string, number>();
|
||||
let skipped = 0;
|
||||
|
||||
@@ -287,14 +322,14 @@ async function main(): Promise<number> {
|
||||
while (next < sessionFiles.length) {
|
||||
const file = sessionFiles[next++];
|
||||
const scan = await collectSessionEdits(file).catch(
|
||||
(): SessionScan => ({ edits: [], requests: [], formats: new Map(), skipped: 0 }),
|
||||
(): SessionScan => ({ edits: [], requestFiles: [], formats: new Map(), skipped: 0 }),
|
||||
);
|
||||
skipped += scan.skipped;
|
||||
for (const [format, count] of scan.formats) {
|
||||
formatCounts.set(format, (formatCounts.get(format) ?? 0) + count);
|
||||
}
|
||||
for (const runs of scan.edits) accumulate(perCall, runs);
|
||||
for (const runs of scan.requests) accumulate(perRequest, runs);
|
||||
for (const runs of scan.requestFiles) accumulate(perRequestFile, runs);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
@@ -309,8 +344,8 @@ async function main(): Promise<number> {
|
||||
console.log(
|
||||
`\nParse coverage: ${parsedTotal}/${parsedTotal + skipped} diff results (${(coverage * 100).toFixed(1)}%; ${formatSummary})`,
|
||||
);
|
||||
report("per user request (calibration target)", perRequest);
|
||||
report("per tool call", perCall);
|
||||
report("per tool call (calibration reference)", perCall);
|
||||
report("per request × file (cumulative activity — upper bound)", perRequestFile);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
@@ -127,14 +127,7 @@ function parseArguments(): Args {
|
||||
/** Target changed-line range for a generated case. */
|
||||
type SizeRange = [number, number];
|
||||
|
||||
/**
|
||||
* Per-mutation case counts and changed-line targets, calibrated against the
|
||||
* empirical edit-shape distribution measured from real agent sessions
|
||||
* (`edit-shape-stats.ts`): 1 → 27%, 2-5 → 30%, 6-20 → 26%, 21-60 → 13%,
|
||||
* 61+ → 4%. Token-level mutations supply the 1-line mass; small structural and
|
||||
* multi-site mutations the 2-5 band; block-level mutations cycle through the
|
||||
* larger bands via `sizes`.
|
||||
*/
|
||||
/** How many cases to generate for a mutation, and which changed-line bands to cycle through. */
|
||||
interface MutationPlan {
|
||||
count: number;
|
||||
sizes?: SizeRange[];
|
||||
@@ -151,30 +144,40 @@ const BLOCK_SIZES: SizeRange[] = [
|
||||
[21, 60],
|
||||
];
|
||||
|
||||
/**
|
||||
* Per-mutation case counts and changed-line targets, calibrated against the
|
||||
* per-tool-call edit-shape distribution measured from real agent sessions
|
||||
* (`edit-shape-stats.ts`, calibration reference): 1 → 23%, 2-5 → 30%,
|
||||
* 6-20 → 29%, 21-60 → 14%, 61+ → 5%. The request×file cumulative numbers form
|
||||
* an upper bound; the suite deliberately sits at or slightly above the
|
||||
* per-call sizes. Token-level mutations supply the 1-line mass; small
|
||||
* structural and multi-site mutations the 2-5 band; block-level mutations
|
||||
* cycle through the larger bands via `sizes`.
|
||||
*/
|
||||
const MUTATION_PLANS: Record<string, MutationPlan> = {
|
||||
"swap-comparison": { count: 2 },
|
||||
"swap-equality": { count: 2 },
|
||||
"swap-logical": { count: 2 },
|
||||
"remove-negation": { count: 2 },
|
||||
"swap-increment-decrement": { count: 2 },
|
||||
"swap-arithmetic": { count: 2 },
|
||||
"flip-boolean": { count: 2 },
|
||||
"remove-optional-chain": { count: 2 },
|
||||
"swap-call-args": { count: 2 },
|
||||
"swap-nullish": { count: 2 },
|
||||
"swap-regex-quantifier": { count: 2 },
|
||||
"unicode-hyphen": { count: 2 },
|
||||
"off-by-one": { count: 2 },
|
||||
"swap-comparison": { count: 1 },
|
||||
"swap-equality": { count: 1 },
|
||||
"swap-logical": { count: 1 },
|
||||
"remove-negation": { count: 1 },
|
||||
"swap-increment-decrement": { count: 1 },
|
||||
"swap-arithmetic": { count: 1 },
|
||||
"flip-boolean": { count: 1 },
|
||||
"remove-optional-chain": { count: 1 },
|
||||
"swap-call-args": { count: 1 },
|
||||
"swap-nullish": { count: 1 },
|
||||
"swap-regex-quantifier": { count: 1 },
|
||||
"unicode-hyphen": { count: 1 },
|
||||
"off-by-one": { count: 1 },
|
||||
"swap-adjacent-lines": { count: 6 },
|
||||
"duplicate-line-flip": { count: 6 },
|
||||
"identifier-multi-edit": { count: 8 },
|
||||
"identifier-multi-edit": { count: 10 },
|
||||
"swap-if-else": { count: 6 },
|
||||
"wrap-redundant-if": { count: 12, sizes: BLOCK_SIZES },
|
||||
"swap-sibling-blocks": { count: 12, sizes: BLOCK_SIZES },
|
||||
"duplicate-block": { count: 6, sizes: BLOCK_SIZES },
|
||||
"move-distant-block": { count: 14, sizes: BLOCK_SIZES },
|
||||
"remove-case-label": { count: 8 },
|
||||
"composite-multi-edit": { count: 14 },
|
||||
"composite-multi-edit": { count: 16 },
|
||||
};
|
||||
|
||||
async function ensureSourceRepo(typescriptDir: string): Promise<void> {
|
||||
|
||||
@@ -53,6 +53,9 @@ async function runScenario(scenario: string): Promise<ScenarioResult> {
|
||||
XDG_DATA_HOME: "",
|
||||
XDG_STATE_HOME: "",
|
||||
XDG_CACHE_HOME: "",
|
||||
// Empty XDG_CACHE_HOME makes Bun's transpiler cache path relative,
|
||||
// spewing bun/@t@/*.pile into the repo root (the child's cwd) — disable it.
|
||||
BUN_RUNTIME_TRANSPILER_CACHE_PATH: "0",
|
||||
OMP_LOGGER_TEST_NOW: fixedNow,
|
||||
TZ: "Etc/GMT+5",
|
||||
},
|
||||
|
||||
@@ -21,16 +21,13 @@ if (childFlagIndex >= 0) {
|
||||
} else if (process.argv.includes(raceChildFlag)) {
|
||||
const marker = process.argv[process.argv.indexOf(raceChildFlag) + 1];
|
||||
if (!marker) throw new Error("Missing cleanup marker path");
|
||||
let cleanupComplete = false;
|
||||
process.on("exit", () => {
|
||||
fs.writeFileSync(marker, cleanupComplete ? "after cleanup" : "before cleanup");
|
||||
});
|
||||
fs.writeFileSync(marker, "before cleanup");
|
||||
postmortem.registerStdioDisconnectHandling();
|
||||
postmortem.register("stdio-epipe-race-test", async () => {
|
||||
process.stderr.write("cleanup started\n");
|
||||
void Promise.reject(Object.assign(new Error("broken pipe"), { code: "EPIPE", syscall: "write" }));
|
||||
await new Response(Bun.stdin.stream()).text();
|
||||
cleanupComplete = true;
|
||||
fs.writeFileSync(marker, "after cleanup");
|
||||
});
|
||||
let rejectionCount = 0;
|
||||
process.on("unhandledRejection", () => {
|
||||
@@ -63,7 +60,7 @@ describe("postmortem broken-pipe handling", () => {
|
||||
|
||||
it("awaits cleanup and exits successfully when a registered stdio peer disconnects", async () => {
|
||||
const marker = `/tmp/omp-postmortem-stdio-${process.pid}-${Date.now()}`;
|
||||
const child = Bun.spawn([process.execPath, import.meta.path, childFlag, marker], {
|
||||
const child = Bun.spawn([process.execPath, "run", import.meta.path, childFlag, marker], {
|
||||
stdin: "pipe",
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
@@ -93,7 +90,7 @@ describe("postmortem broken-pipe handling", () => {
|
||||
|
||||
it("keeps waiting for active cleanup when another stdio EPIPE arrives", async () => {
|
||||
const marker = `/tmp/omp-postmortem-stdio-race-${process.pid}-${Date.now()}`;
|
||||
const child = Bun.spawn([process.execPath, import.meta.path, raceChildFlag, marker], {
|
||||
const child = Bun.spawn([process.execPath, "run", import.meta.path, raceChildFlag, marker], {
|
||||
stdin: "pipe",
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
|
||||
Reference in New Issue
Block a user