smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(natives): refresh diff bench results at final revision

Browse Source
This commit is contained in:
Wolfgang Schoenberger
2026-07-22 03:13:29 -07:00
parent 1b3e35716e
commit 85b44bb0e4
1 changed files with 23 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/natives/bench/diff-results.md
+23 -14
View File
@@ -1,25 +1,34 @@
# Native diff benchmark
- Source: `5f41203f0` (clean), release/ci-profile native build, `PI_COMPILED=1`
- Source: `1b3e35716e64c16c019564a52c58019b54d9733d` (clean tree), ci-profile native build, `PI_COMPILED=1`
- Runtime: bun 1.3.14, darwin-arm64 (Apple M4 Pro)
- Method: seeded synthetic docs, per-scenario warmup + timed iterations, crossing-inclusive (measures the full N-API boundary)
- Command: `PI_COMPILED=1 bun packages/natives/bench/diff.ts` (`BENCH_WARMUP`/`BENCH_ITERATIONS` env overrides)
- Method: seeded synthetic docs, per-scenario warmup + timed iterations, serial runs (no concurrent load), crossing-inclusive (measures the full N-API boundary)
- Command: `PI_COMPILED=1 bun packages/natives/bench/diff.ts` (`BENCH_WARMUP` / `BENCH_ITERATIONS` / `BENCH_MAX_LINES` env overrides)
## Results (warmup 2, iterations 10/scenario)
## High-precision run (warmup 20, iterations 300/scenario, scenarios ≤ 5000 lines)
| scenario | jsdiff diffLines | native diffLines | speedup | jsdiff structuredPatch | native hunks | speedup |
|---|---|---|---|---|---|---|
| 100 lines / 1% edits | 91.9µs | 16.3µs | 5.6x | 65.0µs | 21.8µs | 3.0x |
| 100 lines / 20% edits | 120.1µs | 26.9µs | 4.5x | 96.3µs | 30.3µs | 3.2x |
| 5000 lines / 1% edits | 1.16ms | 514.4µs | 2.3x | 1.48ms | 464.4µs | 3.2x |
| 5000 lines / 20% edits | 134.47ms | 21.49ms | 6.3x | 125.73ms | 21.23ms | 5.9x |
| 50000 lines / 1% edits | 38.94ms | 9.84ms | 4.0x | 41.51ms | 9.92ms | 4.2x |
| 50000 lines / 20% edits | 22790.94ms | 2350.67ms | 9.7x | 23480.65ms | 2395.80ms | 9.8x |
| 100 lines / 1% edits | 22.3µs | 11.0µs | 2.0x | 33.0µs | 10.4µs | 3.2x |
| 100 lines / 20% edits | 50.0µs | 26.9µs | 1.9x | 62.7µs | 26.1µs | 2.4x |
| 5000 lines / 1% edits | 906.4µs | 562.6µs | 1.6x | 1.33ms | 629.6µs | 2.1x |
| 5000 lines / 20% edits | 165.79ms | 24.51ms | 6.8x | 154.89ms | 22.95ms | 6.7x |
A higher-iteration run (warmup 5, iterations 50) over the first five scenarios reproduced the same band (1.9x–6.5x); the 50k/20% row uses 10 iterations because jsdiff needs ~23s per iteration there.
## Full run including 50k-line scenarios (warmup 2, iterations 10/scenario)
Reduced iterations because jsdiff needs ~26s per iteration on the heaviest row.
| scenario | jsdiff diffLines | native diffLines | speedup | jsdiff structuredPatch | native hunks | speedup |
|---|---|---|---|---|---|---|
| 100 lines / 1% edits | 61.7µs | 12.7µs | 4.9x | 61.6µs | 11.4µs | 5.4x |
| 100 lines / 20% edits | 58.8µs | 29.6µs | 2.0x | 91.7µs | 27.6µs | 3.3x |
| 5000 lines / 1% edits | 1.33ms | 602.5µs | 2.2x | 2.11ms | 561.4µs | 3.8x |
| 5000 lines / 20% edits | 182.69ms | 26.80ms | 6.8x | 156.84ms | 24.02ms | 6.5x |
| 50000 lines / 1% edits | 48.30ms | 13.97ms | 3.5x | 50.45ms | 10.59ms | 4.8x |
| 50000 lines / 20% edits | 25749.69ms | 2402.27ms | 10.7x | 29119.74ms | 2882.79ms | 10.1x |
## Notes
- Native wins at every measured size; no crossover where the N-API crossing cost dominates (smallest input, 100 lines, is still 3–5.6x).
- The worst jsdiff case (50k lines / 20% edit density) is a 23s synchronous stall on the render path vs 2.4s native.
- Parity: `packages/natives/test/diff-parity.test.ts` (jsdiff vs native, structural equality; 207 expects) plus existing `edit-diff.test.ts` (42 expects) pass unchanged.
- Native wins at every measured size; no crossover where the N-API crossing cost dominates (100-line inputs are still ~2-3x).
- The worst jsdiff case (50k lines / 20% edit density) is a ~26s synchronous stall on the render path vs ~2.4s native.
- Behavior parity with jsdiff is defended by `packages/natives/test/diff-parity.test.ts` (fixed edge cases, seeded random documents including CRLF/unicode/no-trailing-newline, seeded random word diffs, and a 10k-line document) plus the existing `packages/coding-agent/test/tools/edit-diff.test.ts`, all run with `PI_COMPILED=1 bun test`.