chore: update stale tests

This commit is contained in:
can1357
2026-07-30 07:48:46 +02:00
parent 9856f904d7
commit 38ebd30337
8 changed files with 144 additions and 95 deletions
+2
View File
@@ -26,6 +26,8 @@ target/
# OS
*.tmp
# Bun compile temp artifacts, left in cwd if a `Bun.build({ compile })` run is killed mid-write
*.bun-build
.DS_Store
# Logs & profiling
+34 -30
View File
@@ -8,7 +8,7 @@
- Key collaborators:
- `packages/coding-agent/src/utils/edit-mode.ts` — selects active edit mode
- `packages/hashline/src/grammar.lark` — canonical constrained-decoding grammar
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `DEL`, `INS`)
- `packages/hashline/src/format.ts` — sigils and header constants (`[`, `]`, `#`, `+`, `SWAP`, `CUT`, `INS`, `PASTE`)
- `packages/hashline/src/input.ts` — parses `[PATH#TAG]` sections
- `packages/hashline/src/tokenizer.ts` / `packages/hashline/src/parser.ts` — tokenizes and parses ops
- `packages/hashline/src/apply.ts` — applies parsed edits to file text
@@ -30,18 +30,21 @@ Patch language inside `input`:
- **Operations**:
- `SWAP N.=M:` — replace original lines N.=M with the body rows below.
- `SWAP.BLK N:` — replace the whole tree-sitter block beginning on line N (its header line through its closing line) with the body rows. The line span is resolved at apply time from the file's parse tree; point N at the line that opens the construct. The resolved span is exactly the node that begins on line N — a leading decorator, attribute, or doc-comment is a separate node and is not included; point N at the first decorator line (Python wraps `@dec` + `def` as one block) or fall back to `SWAP N.=M:` to take a leading line-comment that parses as its own node (e.g. Rust `///`). On success the result echoes the matched span (`SWAP.BLK N → resolved lines A-B`). Errors (and steers to `SWAP N.=M:`) when the language is unsupported, line N is blank or a closing delimiter, no node begins there, or the resolved block has a syntax error.
- `DEL N.=M` — delete original lines N.=M. No body.
- `DEL.BLK N` — delete the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`DEL.BLK N → resolved lines A-B`). Same resolution failure modes and `DEL N.=M` fallback.
- `CUT N.=M` — delete original lines N.=M and capture them in the clipboard. No body. A standalone cut is valid; the latest cut replaces the clipboard contents.
- `CUT.BLK N` — delete and capture the whole tree-sitter block beginning on line N (resolved like `SWAP.BLK N`, with the same decorator/comment caveat). No body. On success the result echoes the matched span (`CUT.BLK N → resolved lines A-B`). Same resolution failure modes and `CUT N.=M` fallback.
- `INS.PRE N:` — insert body rows immediately before line N.
- `INS.POST N:` — insert body rows immediately after line N.
- `INS.BLK.POST N:` — insert body rows after the last line of the tree-sitter block beginning on line N. Point N at the line that opens the construct, never its closing delimiter / last visible line; if you can see the last line already, use plain `INS.POST M:`. An anchor that can't resolve to a block is lowered to plain `INS.POST N:` with a warning instead of failing the patch.
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all three block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `DEL.BLK` drops the section, `SWAP.BLK` rewrites it, `INS.BLK.POST` lands after it. A heading with no body resolves to a single line and falls back to the plain op like any other single-line block.
- **Markdown sections**: tree-sitter-md nests a heading and its body (including deeper subsections) in one `section` node, so all four block ops anchored on a `#`/`##`/`###` heading line resolve the whole section — heading through every nested deeper heading, up to the next same-or-higher heading. `CUT.BLK` drops and captures the section, `SWAP.BLK` rewrites it, and `INS.BLK.POST` / `PASTE.BLK.POST` land after it. A heading with no body resolves to a single line and is rejected with guidance to use the corresponding plain line op.
- `INS.HEAD:` — insert body rows at the start of the file.
- `INS.TAIL:` — insert body rows at the end of the file.
- `PASTE.PRE N` / `PASTE.POST N` / `PASTE.HEAD` / `PASTE.TAIL` — insert the clipboard at that position. No body. An empty clipboard is an error.
- `PASTE.BLK.POST N` — insert the clipboard after the resolved block's last line. An unresolvable anchor lowers to `PASTE.POST N` with a warning, matching `INS.BLK.POST`.
- **Clipboard**: operations execute top-to-bottom across all patch sections. The latest `CUT` wins; `PASTE` does not consume the clipboard and may be repeated. The coding agent persists the register across edit calls in the same session, enabling cross-file moves. Keep each path under one header when clipboard operations would otherwise be interleaved around another file's section.
- **Body rows**:
- Only body-bearing headers end in `:`.
- Every body row is `+TEXT`; `+` alone adds a blank line.
- `DEL` never has body rows.
- `CUT` and `PASTE` never have body rows.
- There is no repeat row kind. To keep a line, leave it out of every range; split edits into multiple hunks when needed.
- `-` rows are invalid. Literal Markdown bullets or text beginning with `-` / `+` must be written as `+- item` / `++ item`.
@@ -52,7 +55,7 @@ Anchors come from `read`/`grep` output. `read` emits a `[PATH#TAG]` header from
The canonical grammar is strict, but the hand parser accepts a few non-dangerous variants:
- `SWAP N:` — accepted as `SWAP N.=N:`.
- `DEL N` — accepted as single-line delete.
- `CUT N` — accepted as a single-line cut/delete.
- Missing trailing colon on `SWAP` or `INS` — accepted.
- `SWAP N-M:`, `SWAP N…M:`, `SWAP N M:`, and legacy `SWAP N..M:` — accepted as `SWAP N.=M:`.
- Bare body rows with no `+` prefix are auto-prepended with `+` and a `BARE_BODY_AUTO_PIPED_WARNING` is appended.
@@ -61,16 +64,17 @@ The canonical grammar is strict, but the hand parser accepts a few non-dangerous
- Some malformed bracketed headers are recovered after stripping apply-patch path noise such as `Update File:` / `Add File:` and extra `***`, but the recovered header still needs a valid four-hex tag for the patcher to apply it.
- `*** Update File:` / `*** Add File:` / `*** Delete File:` / `*** Move to:` apply_patch sentinels inside the diff body throw an `apply_patch sentinel … is not valid in hashline` error.
- `@@`-bracketed hunk headers are rejected with guidance to write a verb header.
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `DEL`.
- `DEL N.=M:` and any body rows under `DEL` / `DEL.BLK` are rejected.
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` (no body rows) is treated as `DEL N.=M`.
- Bare `N` and bare `N M` / `N.=M` headers are rejected with guidance to write `SWAP` or `CUT`.
- A trailing colon on `CUT N.=M:` / `CUT.BLK N:` is tolerated and ignored, but body rows under `CUT`, `CUT.BLK`, or any `PASTE` form are rejected.
- Bare `PASTE` is rejected because the insertion position is required.
- Empty `INS` / `SWAP.BLK` hunks are rejected; an empty `SWAP N.=M:` deletes the range, though `CUT N.=M` is the canonical deletion form.
- `-` body rows are rejected with `MINUS_ROW_REJECTED` unless the hunk is unambiguously a Markdown bullet list (see Tolerated input shapes).
- `SWAP.BLK N:` / `DEL.BLK N` / `INS.BLK.POST N:` require a wired tree-sitter resolver; `SWAP.BLK` and `INS.BLK.POST` additionally need at least one `+TEXT` body row, while `DEL.BLK` takes none. An unresolvable block (unsupported language, blank/closing-delimiter line, no node beginning on N, or a syntax error in the resolved block) rejects a `SWAP.BLK` / `DEL.BLK` on the apply/final-preview path (the streaming preview silently drops it instead). `INS.BLK.POST N:` is never rejected this way — it is lowered to plain `INS.POST N:` with a warning: a closing-delimiter-anchor warning when line N is a pure closer (inserting after that end is exactly what the plain form does), a generic unresolved-anchor warning otherwise.
- `SWAP.BLK N:` / `CUT.BLK N` / `INS.BLK.POST N:` / `PASTE.BLK.POST N` consult the wired tree-sitter resolver. `SWAP.BLK` and `INS.BLK.POST` need at least one `+TEXT` body row; `CUT.BLK` and `PASTE.BLK.POST` take none. A null resolution rejects `SWAP.BLK` / `CUT.BLK` on the apply or final-preview path (the streaming preview silently drops it), while `INS.BLK.POST` / `PASTE.BLK.POST` lower to the corresponding plain `POST` form with a warning. A single-line resolution rejects every block form with guidance to use its plain line equivalent.
## Outputs
- Single-shot tool result; hashline mode does not use the staged preview/apply devices (`/xdev/resolve`, `/xdev/reject`).
- `content` contains one text block per call. For a successful single-file edit it is the post-edit `[path#TAG]` section header (a fresh snapshot tag for the written content), followed by a compact diff preview from `packages/hashline/src/diff-preview.ts` when one is emitted.
- When the patch used `SWAP.BLK`/`DEL.BLK`/`INS.BLK.POST` ops (and the apply matched the tagged content), one `SWAP.BLK N → resolved lines A-B (K lines)` line per block op (single-line spans render `resolved line A (1 line)`; INS.BLK.POST appends `; body lands after line B`) is inserted between the `[PATH#TAG]` header and the diff preview, so the caller can confirm tree-sitter resolved the construct it intended.
- When the patch used `SWAP.BLK` / `CUT.BLK` / `INS.BLK.POST` / `PASTE.BLK.POST` ops (and the apply matched the tagged content), one `<OP> N → resolved lines A-B (K lines)` line per block op is inserted between the `[PATH#TAG]` header and the diff preview. Single-line spans render `resolved line A (1 line)`; `INS.BLK.POST` appends `body lands after line B`, and `PASTE.BLK.POST` appends `clipboard lands after line B`.
- Parse, apply, or recovery warnings are appended as:
```text
@@ -126,11 +130,11 @@ INS.PRE 5:
+console.log(X + Y);
```
Delete lines 4.=5 entirely:
Delete lines 4.=5 entirely and leave them in the clipboard:
```text
[a.ts#0A3B]
DEL 4.=5
CUT 4.=5
```
Insert at start and end of file:
@@ -143,20 +147,19 @@ INS.TAIL:
+// trailer
```
Multi-file:
Move line 4 from `src/a.ts` to after line 20 in `src/b.ts`:
```text
[src/a.ts#0A3B]
SWAP 4.=4:
+const enabled = true;
CUT 4
[src/b.ts#1F7C]
DEL 20
PASTE.POST 20
```
## Limits & Caps
- File snapshot tags are exactly four uppercase-hex chars — content-derived hashes (`computeFileHash()`) recorded in the per-session snapshot store.
- The visible mismatch report shows 2 lines of context on each side (`MISMATCH_CONTEXT`) in `packages/hashline/src/messages.ts`.
- Stale-anchor recovery uses `fuzzFactor: 0` in `packages/hashline/src/recovery.ts`.
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and hunk keyword constants are `SWAP` / `DEL` / `INS` (`packages/hashline/src/format.ts`).
- `HL_FILE_PREFIX` is `[`, `HL_FILE_SUFFIX` is `]`, `HL_PAYLOAD_REPLACE` is `+`, `HL_RANGE_SEP` is `.=`, `HL_FILE_HASH_SEP` is `#`, and line/clipboard hunk keyword constants are `SWAP` / `CUT` / `INS` / `PASTE` (`packages/hashline/src/format.ts`).
## Errors
- Missing section header:
@@ -164,28 +167,29 @@ DEL 20
- Missing tag for any section:
- `Missing hashline snapshot tag for <path>; use \`[<path>#tag]\` from your latest read/search output. To create a new file, use the write tool.`
- Stray payload line:
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
- `line N: payload line has no preceding hunk header. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` above the body. Got "...".`
- Minus row (unless auto-piped as an unambiguous Markdown bullet — see Tolerated input shapes):
- ``line N: `-` rows are not valid; the range already names the lines being changed. For Markdown bullets or other literal `-` lines, prefix the literal row with `+`: `+- item`.``
- Empty body-bearing hunk:
- `line N: \`INS\` needs at least one \`+TEXT\` body row.`
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`DEL.BLK N\`.`
- Unresolvable block anchor — `SWAP.BLK` / `DEL.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by a blank line and numbered `*`-marked context rows around line X (same shape as the mismatch preview). `DEL.BLK X` produces the same message with a `DEL X.=M` fallback.
- `INS.BLK.POST X:` never reaches this error — an unresolvable anchor is lowered to plain `INS.POST X:` with a warning (see Tolerated input shapes).
- Delete with body:
- `line N: \`DEL N.=M\` does not take body rows. Remove the body, or use \`SWAP N.=M:\`.`
- `line N: \`DEL.BLK N\` does not take body rows. Remove the body, or use \`SWAP.BLK N:\`.`
- `line N: \`SWAP.BLK N:\` needs at least one \`+TEXT\` body row. To delete a block, use \`CUT.BLK N\`.`
- Unresolvable block anchor — `SWAP.BLK` / `CUT.BLK` only (apply / final-preview path; the streaming preview silently drops the op instead):
- `line N: \`SWAP.BLK X:\` could not resolve a syntactic block beginning on line X (unsupported language, blank/closer line, or parse error). Use \`SWAP X.=M:\` with explicit lines.` — followed by numbered context and, when available, a nearby block suggestion. `CUT.BLK X` produces the corresponding message with a `CUT X.=M` fallback.
- `INS.BLK.POST X:` and `PASTE.BLK.POST X` never reach this error when no block resolves — they lower to plain `INS.POST X:` / `PASTE.POST X` with a warning.
- Clipboard operation errors:
- `line N: \`CUT N.=M\` captures + deletes lines and takes no body rows. To replace lines with new content, use \`SWAP N.=M:\`.`
- `line N: \`PASTE\` inserts the clipboard content and takes no \`+\` body rows. To insert literal text, use \`INS\`.`
- `line N: \`PASTE\` found nothing in the clipboard. Ops run top-to-bottom across the whole patch (sections included): put \`CUT N.=M\` or \`CUT.BLK N\` above the \`PASTE\`.`
- Range out of order:
- `line N: range A.=B ends before it starts.`
- Overlapping hunks on the same anchor:
- `line N: anchor line X is already targeted by another hunk on line Y. Issue ONE hunk per range; payload is only the final desired content, never a before/after pair.`
- apply_patch / unified-diff contamination:
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`DEL N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: apply_patch sentinel "*** …" is not valid in hashline. File sections start with \`[path#HASH]\` (no \`Update File:\` / \`Add File:\` keyword). Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: unified-diff hunk header (\`@@ -N,M +N,M @@\`) is not valid in hashline. Use \`SWAP N.=M:\`, \`CUT N.=M\`, or \`INS.PRE|POST|HEAD|TAIL:\` ops.`
- `line N: \`@@\`-bracketed hunk header "@@ …" is not valid in hashline. Drop the \`@@ ... @@\` brackets and write a verb header such as \`SWAP N.=M:\`.`
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`DEL N\` to delete.`
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`DEL ${bareRange[1]}.=${bareRange[2]}\`.`
- `line N: hunk headers need a verb. Use \`SWAP N.=N:\` to replace, or \`CUT N\` to delete.`
- `line N: bare range hunk header "N M" is not valid. Hunk headers need a verb: write \`SWAP ${bareRange[1]}.=${bareRange[2]}:\` or \`CUT ${bareRange[1]}.=${bareRange[2]}\`.`
- Out-of-range anchor:
- `Line N does not exist (file has M lines)`
- Stale snapshot tag: the `Patcher` first attempts snapshot-based recovery. When recovery cannot prove a valid result it throws `MismatchError`, which distinguishes recognized-but-drifted hashes from never-recorded hashes. The error includes the current file hash plus context around each anchor.
@@ -19,9 +19,9 @@ interface TemplateProbeResult {
}
const expectedTemplate: TemplateProbeResult = {
chars: 377_268,
bytes: 377_446,
sha256: "023cf3773498db52ff0ec508b22ef2c002710f9e9fcb90fea361659d5b0205df",
chars: 377_286,
bytes: 377_464,
sha256: "6598b2d74f905e3105474ea8033c938e92df4c33035421603ea363d72d1c81c1",
stableCache: true,
assetsRemoved: 0,
};
@@ -78,6 +78,7 @@ function composeExpectedTemplate(): string {
async function runProbe(command: string[]): Promise<TemplateProbeResult> {
const proc = Bun.spawn(command, {
cwd: unrelatedCwd,
stdin: "ignore",
stderr: "pipe",
stdout: "pipe",
});
@@ -149,8 +150,9 @@ describe("HTML export template", () => {
fs.writeFileSync(staleAssetPath, "stale");
const build = Bun.spawn([process.execPath, "run", "gen:bundle"], {
cwd: packageDir,
stderr: "pipe",
stdin: "ignore",
stdout: "pipe",
stderr: "pipe",
});
const [buildStdout, buildStderr, buildExitCode] = await Promise.all([
new Response(build.stdout).text(),
@@ -162,8 +164,9 @@ describe("HTML export template", () => {
const proc = Bun.spawn([process.execPath, "pm", "pack", "--dry-run", "--ignore-scripts"], {
cwd: packageDir,
stderr: "pipe",
stdin: "ignore",
stdout: "pipe",
stderr: "pipe",
});
const [stdout, stderr, exitCode] = await Promise.all([
new Response(proc.stdout).text(),
@@ -195,6 +198,7 @@ describe("HTML export template", () => {
test("does not retain source asset strings during a static import", async () => {
const proc = Bun.spawn([process.execPath, heapProbePath], {
stdin: "ignore",
stderr: "pipe",
stdout: "pipe",
});
@@ -220,8 +220,9 @@ describe("SelectorController.handleResumeSession preflight flush", () => {
selector!.handleInput("\n");
expect(selectionPromise).toBeDefined();
await expect(selectionPromise!).rejects.toBe(switchError);
await selectionPromise;
expect(ctx.showError).toHaveBeenCalledWith("switch failed");
expect(ctx.settings.flush).toHaveBeenCalledTimes(1);
expect(switchSession).toHaveBeenCalledWith(session.path);
expect(hide).toHaveBeenCalledTimes(1);
@@ -9,27 +9,25 @@
* every run counts as its own hunk.
*
* Shapes are reported at two levels:
* - per tool call — one sample per edit call;
* - per user request — all edit calls between one user message and the next
* grouped into one sample (one benchmark fixture ≙ one prompt, and the
* runner allows multiple edit calls per prompt). Retried/overlapping edits
* within a request are summed, so request sizes lean high, and a request
* spans every file a long agentic turn touched — an upper bound for
* single-file fixtures.
* - per tool call — one sample per edit call. Fixtures are scored on the
* final input→expected diff, and a single call's diff is the closest
* measured proxy for a net diff, so **this is the calibration reference**
* for `generate.ts`.
* - per request × file — all edit calls to one file between one user message
* and the next, concatenated. This double-counts retries, overlapping
* edits, and reversals (no per-file net reconstruction), so treat it as an
* **upper bound** on how much a single-prompt fixture could plausibly ask
* for, not as a target.
*
* Parse coverage is printed so a format skew cannot silently bias the numbers.
* Reference measurement (2026-07, 2,000 newest sessions against this repo,
* 99.9% coverage):
*
* per request: changed lines: 1 → 6% 2-5 → 11% 6-20 → 20% 21-60 → 25% 61+ → 39%
* hunks: 1 → 11% 2 → 8% 3+ → 81% (median 40 changed lines)
* per call: changed lines: 1 → 23% 2-5 → 30% 6-20 → 29% 21-60 → 14% 61+ → 5%
* hunks: 1 → 48% 2 → 21% 3+ → 31% (median 5 changed lines)
* request×file: changed lines: 1 → 18% 2-5 → 28% 6-20 → 28% 21-60 → 17% 61+ → 9%
* hunks: 1 → 37% 2 → 21% 3+ → 43% (median 7; upper bound)
* op mix (both levels): replace 55% insert 32% delete 12%
*
* `generate.ts` fixtures are single-file, single-prompt tasks: the suite is
* calibrated to sit between the two levels — per-call shapes for token fixes,
* request-leaning shapes (large blocks, multi-hunk composites) for the rest.
*/
import * as fs from "node:fs/promises";
import * as path from "node:path";
@@ -157,26 +155,46 @@ function runsFromDiff(diff: string): { runs: ChangeRun[] | null; format: "number
return { runs: runsFromNumberedDiff(diff), format: "numbered" };
}
/** Per-session parse results: change runs per edit call and per user request, plus coverage counters. */
/** Per-session parse results: change runs per edit call and per request×file, plus coverage counters. */
interface SessionScan {
edits: ChangeRun[][];
/** All edit-call runs between one user message and the next, concatenated. */
requests: ChangeRun[][];
/** All edit-call runs targeting one file between one user message and the next, concatenated. */
requestFiles: ChangeRun[][];
formats: Map<string, number>;
skipped: number;
}
/** Resolve the edited file path from tool-call args across edit-tool variants. */
function extractCallPath(args: Record<string, unknown>): string | null {
if (typeof args.path === "string" && args.path) return args.path;
if (typeof args.input === "string") {
const header =
args.input.match(/^\u00b6([^#\n]+)#/) ??
args.input.match(/^\[([^#\]\n]+)#/) ??
args.input.match(/^@@ ([^\n]+)$/m);
if (header) return header[1].trim();
}
if (Array.isArray(args.edits) && args.edits.length > 0 && args.edits[0] && typeof args.edits[0] === "object") {
const nested = (args.edits[0] as Record<string, unknown>).path;
if (typeof nested === "string" && nested) return nested;
}
return null;
}
/** Extract change runs for every successful edit in one session JSONL file. */
async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
const scan: SessionScan = { edits: [], requests: [], formats: new Map(), skipped: 0 };
const scan: SessionScan = { edits: [], requestFiles: [], formats: new Map(), skipped: 0 };
const stream = Bun.file(sessionFile).stream();
const decoder = new TextDecoder();
let buffer = "";
let currentRequest: ChangeRun[] = [];
const callPaths = new Map<string, string>();
let currentRequest = new Map<string, ChangeRun[]>();
const flushRequest = (): void => {
if (currentRequest.length > 0) scan.requests.push(currentRequest);
currentRequest = [];
for (const runs of currentRequest.values()) {
if (runs.length > 0) scan.requestFiles.push(runs);
}
currentRequest = new Map();
};
const handleRecord = (record: unknown): void => {
@@ -185,11 +203,23 @@ async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
if (rec.type !== "message" || !rec.message || typeof rec.message !== "object") return;
const message = rec.message as Record<string, unknown>;
// A user message starts a new request; everything until the next one
// belongs to the same prompt (one benchmark fixture ≙ one request).
// belongs to the same prompt (one benchmark fixture ≙ one prompt on one file).
if (message.role === "user") {
flushRequest();
return;
}
if (message.role === "assistant" && Array.isArray(message.content)) {
for (const block of message.content) {
if (!block || typeof block !== "object") continue;
const call = block as Record<string, unknown>;
if (call.type !== "toolCall" || !EDIT_TOOL_NAMES[String(call.name)]) continue;
const args =
call.arguments && typeof call.arguments === "object" ? (call.arguments as Record<string, unknown>) : {};
const callPath = extractCallPath(args);
if (typeof call.id === "string" && callPath) callPaths.set(call.id, callPath);
}
return;
}
if (message.role !== "toolResult" || !EDIT_TOOL_NAMES[String(message.toolName)] || message.isError) return;
const details =
message.details && typeof message.details === "object"
@@ -201,7 +231,12 @@ async function collectSessionEdits(sessionFile: string): Promise<SessionScan> {
const { runs, format } = runsFromDiff(diff);
if (runs) {
scan.edits.push(runs);
currentRequest.push(...runs);
// Unknown path → each call is its own group (conservative: no false merging).
const callId = typeof message.toolCallId === "string" ? message.toolCallId : `#${scan.edits.length}`;
const fileKey = callPaths.get(callId) ?? `call:${callId}`;
const group = currentRequest.get(fileKey) ?? [];
group.push(...runs);
currentRequest.set(fileKey, group);
scan.formats.set(format, (scan.formats.get(format) ?? 0) + 1);
} else {
scan.skipped++;
@@ -278,7 +313,7 @@ async function main(): Promise<number> {
};
const perCall = newCounters();
const perRequest = newCounters();
const perRequestFile = newCounters();
const formatCounts = new Map<string, number>();
let skipped = 0;
@@ -287,14 +322,14 @@ async function main(): Promise<number> {
while (next < sessionFiles.length) {
const file = sessionFiles[next++];
const scan = await collectSessionEdits(file).catch(
(): SessionScan => ({ edits: [], requests: [], formats: new Map(), skipped: 0 }),
(): SessionScan => ({ edits: [], requestFiles: [], formats: new Map(), skipped: 0 }),
);
skipped += scan.skipped;
for (const [format, count] of scan.formats) {
formatCounts.set(format, (formatCounts.get(format) ?? 0) + count);
}
for (const runs of scan.edits) accumulate(perCall, runs);
for (const runs of scan.requests) accumulate(perRequest, runs);
for (const runs of scan.requestFiles) accumulate(perRequestFile, runs);
}
});
await Promise.all(workers);
@@ -309,8 +344,8 @@ async function main(): Promise<number> {
console.log(
`\nParse coverage: ${parsedTotal}/${parsedTotal + skipped} diff results (${(coverage * 100).toFixed(1)}%; ${formatSummary})`,
);
report("per user request (calibration target)", perRequest);
report("per tool call", perCall);
report("per tool call (calibration reference)", perCall);
report("per request × file (cumulative activity — upper bound)", perRequestFile);
return 0;
}
@@ -127,14 +127,7 @@ function parseArguments(): Args {
/** Target changed-line range for a generated case. */
type SizeRange = [number, number];
/**
* Per-mutation case counts and changed-line targets, calibrated against the
* empirical edit-shape distribution measured from real agent sessions
* (`edit-shape-stats.ts`): 1 → 27%, 2-5 → 30%, 6-20 → 26%, 21-60 → 13%,
* 61+ → 4%. Token-level mutations supply the 1-line mass; small structural and
* multi-site mutations the 2-5 band; block-level mutations cycle through the
* larger bands via `sizes`.
*/
/** How many cases to generate for a mutation, and which changed-line bands to cycle through. */
interface MutationPlan {
count: number;
sizes?: SizeRange[];
@@ -151,30 +144,40 @@ const BLOCK_SIZES: SizeRange[] = [
[21, 60],
];
/**
* Per-mutation case counts and changed-line targets, calibrated against the
* per-tool-call edit-shape distribution measured from real agent sessions
* (`edit-shape-stats.ts`, calibration reference): 1 → 23%, 2-5 → 30%,
* 6-20 → 29%, 21-60 → 14%, 61+ → 5%. The request×file cumulative numbers form
* an upper bound; the suite deliberately sits at or slightly above the
* per-call sizes. Token-level mutations supply the 1-line mass; small
* structural and multi-site mutations the 2-5 band; block-level mutations
* cycle through the larger bands via `sizes`.
*/
const MUTATION_PLANS: Record<string, MutationPlan> = {
"swap-comparison": { count: 2 },
"swap-equality": { count: 2 },
"swap-logical": { count: 2 },
"remove-negation": { count: 2 },
"swap-increment-decrement": { count: 2 },
"swap-arithmetic": { count: 2 },
"flip-boolean": { count: 2 },
"remove-optional-chain": { count: 2 },
"swap-call-args": { count: 2 },
"swap-nullish": { count: 2 },
"swap-regex-quantifier": { count: 2 },
"unicode-hyphen": { count: 2 },
"off-by-one": { count: 2 },
"swap-comparison": { count: 1 },
"swap-equality": { count: 1 },
"swap-logical": { count: 1 },
"remove-negation": { count: 1 },
"swap-increment-decrement": { count: 1 },
"swap-arithmetic": { count: 1 },
"flip-boolean": { count: 1 },
"remove-optional-chain": { count: 1 },
"swap-call-args": { count: 1 },
"swap-nullish": { count: 1 },
"swap-regex-quantifier": { count: 1 },
"unicode-hyphen": { count: 1 },
"off-by-one": { count: 1 },
"swap-adjacent-lines": { count: 6 },
"duplicate-line-flip": { count: 6 },
"identifier-multi-edit": { count: 8 },
"identifier-multi-edit": { count: 10 },
"swap-if-else": { count: 6 },
"wrap-redundant-if": { count: 12, sizes: BLOCK_SIZES },
"swap-sibling-blocks": { count: 12, sizes: BLOCK_SIZES },
"duplicate-block": { count: 6, sizes: BLOCK_SIZES },
"move-distant-block": { count: 14, sizes: BLOCK_SIZES },
"remove-case-label": { count: 8 },
"composite-multi-edit": { count: 14 },
"composite-multi-edit": { count: 16 },
};
async function ensureSourceRepo(typescriptDir: string): Promise<void> {
@@ -53,6 +53,9 @@ async function runScenario(scenario: string): Promise<ScenarioResult> {
XDG_DATA_HOME: "",
XDG_STATE_HOME: "",
XDG_CACHE_HOME: "",
// Empty XDG_CACHE_HOME makes Bun's transpiler cache path relative,
// spewing bun/@t@/*.pile into the repo root (the child's cwd) — disable it.
BUN_RUNTIME_TRANSPILER_CACHE_PATH: "0",
OMP_LOGGER_TEST_NOW: fixedNow,
TZ: "Etc/GMT+5",
},
+4 -7
View File
@@ -21,16 +21,13 @@ if (childFlagIndex >= 0) {
} else if (process.argv.includes(raceChildFlag)) {
const marker = process.argv[process.argv.indexOf(raceChildFlag) + 1];
if (!marker) throw new Error("Missing cleanup marker path");
let cleanupComplete = false;
process.on("exit", () => {
fs.writeFileSync(marker, cleanupComplete ? "after cleanup" : "before cleanup");
});
fs.writeFileSync(marker, "before cleanup");
postmortem.registerStdioDisconnectHandling();
postmortem.register("stdio-epipe-race-test", async () => {
process.stderr.write("cleanup started\n");
void Promise.reject(Object.assign(new Error("broken pipe"), { code: "EPIPE", syscall: "write" }));
await new Response(Bun.stdin.stream()).text();
cleanupComplete = true;
fs.writeFileSync(marker, "after cleanup");
});
let rejectionCount = 0;
process.on("unhandledRejection", () => {
@@ -63,7 +60,7 @@ describe("postmortem broken-pipe handling", () => {
it("awaits cleanup and exits successfully when a registered stdio peer disconnects", async () => {
const marker = `/tmp/omp-postmortem-stdio-${process.pid}-${Date.now()}`;
const child = Bun.spawn([process.execPath, import.meta.path, childFlag, marker], {
const child = Bun.spawn([process.execPath, "run", import.meta.path, childFlag, marker], {
stdin: "pipe",
stdout: "pipe",
stderr: "pipe",
@@ -93,7 +90,7 @@ describe("postmortem broken-pipe handling", () => {
it("keeps waiting for active cleanup when another stdio EPIPE arrives", async () => {
const marker = `/tmp/omp-postmortem-stdio-race-${process.pid}-${Date.now()}`;
const child = Bun.spawn([process.execPath, import.meta.path, raceChildFlag, marker], {
const child = Bun.spawn([process.execPath, "run", import.meta.path, raceChildFlag, marker], {
stdin: "pipe",
stdout: "pipe",
stderr: "pipe",