diff --git a/packages/natives/bench/diff-results.md b/packages/natives/bench/diff-results.md index e5d290c89..24fcb5895 100644 --- a/packages/natives/bench/diff-results.md +++ b/packages/natives/bench/diff-results.md @@ -1,25 +1,34 @@ # Native diff benchmark -- Source: `5f41203f0` (clean), release/ci-profile native build, `PI_COMPILED=1` +- Source: `1b3e35716e64c16c019564a52c58019b54d9733d` (clean tree), ci-profile native build, `PI_COMPILED=1` - Runtime: bun 1.3.14, darwin-arm64 (Apple M4 Pro) -- Method: seeded synthetic docs, per-scenario warmup + timed iterations, crossing-inclusive (measures the full N-API boundary) -- Command: `PI_COMPILED=1 bun packages/natives/bench/diff.ts` (`BENCH_WARMUP`/`BENCH_ITERATIONS` env overrides) +- Method: seeded synthetic docs, per-scenario warmup + timed iterations, serial runs (no concurrent load), crossing-inclusive (measures the full N-API boundary) +- Command: `PI_COMPILED=1 bun packages/natives/bench/diff.ts` (`BENCH_WARMUP` / `BENCH_ITERATIONS` / `BENCH_MAX_LINES` env overrides) -## Results (warmup 2, iterations 10/scenario) +## High-precision run (warmup 20, iterations 300/scenario, scenarios ≤ 5000 lines) | scenario | jsdiff diffLines | native diffLines | speedup | jsdiff structuredPatch | native hunks | speedup | |---|---|---|---|---|---|---| -| 100 lines / 1% edits | 91.9µs | 16.3µs | 5.6x | 65.0µs | 21.8µs | 3.0x | -| 100 lines / 20% edits | 120.1µs | 26.9µs | 4.5x | 96.3µs | 30.3µs | 3.2x | -| 5000 lines / 1% edits | 1.16ms | 514.4µs | 2.3x | 1.48ms | 464.4µs | 3.2x | -| 5000 lines / 20% edits | 134.47ms | 21.49ms | 6.3x | 125.73ms | 21.23ms | 5.9x | -| 50000 lines / 1% edits | 38.94ms | 9.84ms | 4.0x | 41.51ms | 9.92ms | 4.2x | -| 50000 lines / 20% edits | 22790.94ms | 2350.67ms | 9.7x | 23480.65ms | 2395.80ms | 9.8x | +| 100 lines / 1% edits | 22.3µs | 11.0µs | 2.0x | 33.0µs | 10.4µs | 3.2x | +| 100 lines / 20% edits | 50.0µs | 26.9µs | 1.9x | 62.7µs | 26.1µs | 2.4x | +| 5000 lines / 1% edits | 906.4µs | 562.6µs | 1.6x | 1.33ms | 629.6µs | 2.1x | +| 5000 lines / 20% edits | 165.79ms | 24.51ms | 6.8x | 154.89ms | 22.95ms | 6.7x | -A higher-iteration run (warmup 5, iterations 50) over the first five scenarios reproduced the same band (1.9x–6.5x); the 50k/20% row uses 10 iterations because jsdiff needs ~23s per iteration there. +## Full run including 50k-line scenarios (warmup 2, iterations 10/scenario) + +Reduced iterations because jsdiff needs ~26s per iteration on the heaviest row. + +| scenario | jsdiff diffLines | native diffLines | speedup | jsdiff structuredPatch | native hunks | speedup | +|---|---|---|---|---|---|---| +| 100 lines / 1% edits | 61.7µs | 12.7µs | 4.9x | 61.6µs | 11.4µs | 5.4x | +| 100 lines / 20% edits | 58.8µs | 29.6µs | 2.0x | 91.7µs | 27.6µs | 3.3x | +| 5000 lines / 1% edits | 1.33ms | 602.5µs | 2.2x | 2.11ms | 561.4µs | 3.8x | +| 5000 lines / 20% edits | 182.69ms | 26.80ms | 6.8x | 156.84ms | 24.02ms | 6.5x | +| 50000 lines / 1% edits | 48.30ms | 13.97ms | 3.5x | 50.45ms | 10.59ms | 4.8x | +| 50000 lines / 20% edits | 25749.69ms | 2402.27ms | 10.7x | 29119.74ms | 2882.79ms | 10.1x | ## Notes -- Native wins at every measured size; no crossover where the N-API crossing cost dominates (smallest input, 100 lines, is still 3–5.6x). -- The worst jsdiff case (50k lines / 20% edit density) is a 23s synchronous stall on the render path vs 2.4s native. -- Parity: `packages/natives/test/diff-parity.test.ts` (jsdiff vs native, structural equality; 207 expects) plus existing `edit-diff.test.ts` (42 expects) pass unchanged. +- Native wins at every measured size; no crossover where the N-API crossing cost dominates (100-line inputs are still ~2-3x). +- The worst jsdiff case (50k lines / 20% edit density) is a ~26s synchronous stall on the render path vs ~2.4s native. +- Behavior parity with jsdiff is defended by `packages/natives/test/diff-parity.test.ts` (fixed edge cases, seeded random documents including CRLF/unicode/no-trailing-newline, seeded random word diffs, and a 10k-line document) plus the existing `packages/coding-agent/test/tools/edit-diff.test.ts`, all run with `PI_COMPILED=1 bun test`.