Commit Graph
96 Commits
Author SHA1 Message Date
vmcall d53cf023b0 fix(task): reconciled structured subagents with upstream
- Preserved the plan-mode capability clamp after upstream removed report_finding.
- Updated persisted-revival coverage for mounted xdev tool activation.
- Applied current formatter output to conflicted runtime files.
2026-07-17 17:38:12 +02:00
vmcall 414ef80c41 fix(task): restored goal activation outside restricted sessions
- Preserved goal-mode tool injection for ordinary explicit tool lists.
- Kept plan-mode LSP and IRC unavailable under the host capability clamp.
- Added regressions for both capability boundaries.
2026-07-17 17:36:59 +02:00
vmcall d944879f21 feat(task): unified structured subagent execution
- Added per-invocation task schemas with strict and permissive validation.
- Shared task and eval agent policy, artifacts, isolation, and lifecycle handling.
- Enabled host-restricted plan-mode eval agents and persisted their capability clamp.

Fixes #5279
2026-07-17 17:36:59 +02:00
can1357 a9c038818d feat(tools): removed separate selector args from read and grep APIs
- Removed `selector`/`sel` arguments from read and grep tool schemas and related execution arg handling.
- Reworked read and grep path processing to parse line selectors from `path` suffixes instead of separate fields, including inline range propagation.
- Updated delegation and execution call paths (including JS/Python preludes and executor tests) to pass selectors embedded in `path`.
- Updated read/grep prompt docs and changelog for the breaking inline-selector API, and removed obsolete selector-specific tests and expectations.
2026-07-15 00:04:02 +02:00
can1357 37b1263a26 Merge PR #5494: fix(eval): delegate Python URI reads to host resolver (@roboomp) 2026-07-14 23:11:12 +02:00
roboomp eef79705a6 fix(eval): passed js uri selectors separately
- Kept opaque MCP resource paths unchanged during JS pagination.
- Sent JS line ranges through the read tool selector field.
- Covered the shipped JS prelude and updated the changelog.

Fixes #5353
2026-07-14 20:31:58 +00:00
can1357 7a2b027e47 fix(eval): keep completion aborts interruptible 2026-07-14 18:45:03 +02:00
can1357 5d28a319fd Merge PR #5015: fix(eval): shield Python agent bridge aborts (@roboomp) 2026-07-14 18:45:03 +02:00
can1357 bfdca36f44 test(coding-agent): use portable eval isolation probe 2026-07-14 18:41:16 +02:00
can1357 79d7b74e24 Merge PR #5327: fix(coding-agent): isolate eval runtimes from terminal (@masonc15) 2026-07-14 18:41:16 +02:00
can1357 f9f6ed9e8d feat(coding-agent): replaced legacy pi/ role alias prefix with
- Replaced legacy `pi/` role alias prefix with canonical `@` syntax across model resolution, documentation, and tests.
- Added support for bare `*` default alias and multiple alias prefix detection with custom role resolution in `resolveConfiguredRolePattern()`.
- Enhanced thinking suffix parsing to accept unambiguous abbreviations (minimum 2 characters) for effort and level selectors.
- Extended `resolveCliModel()` and `filterAvailableModelsByEnabledPatterns()` to accept settings parameter for role alias resolution from `--model` flag.
2026-07-13 23:26:33 +02:00
Colin Mason 95620ccd6c preserve async js fallback ladder 2026-07-13 09:57:20 -04:00
Colin Mason 55d9fcfd14 fall back to bun worker on eval spawn failure 2026-07-13 09:57:20 -04:00
Colin Mason 1a527d9a2f mirror session cwd in js eval subprocess 2026-07-13 09:57:19 -04:00
Colin Mason 9c9da2387d harden js eval subprocess 2026-07-13 09:57:19 -04:00
Colin Mason c40ccdc684 isolate eval runtimes from terminal 2026-07-13 09:57:19 -04:00
roboomp a86c1ec46d fix(eval): shield agent bridge aborts
- Deferred eval cancellation while Python bridge calls are paused so already-started agent() subagents can finish and persist output.
- Rejected new Python bridge calls after an external abort is pending to prevent post-abort fan-out waves.
- Added regression coverage for the bridge shield and Python parallel agent() interruption path.

Fixes #5005
2026-07-10 01:21:39 +00:00
can1357 af146b1118 Merge PR #4439: fix(task): preserve isolated commits on transfer failure (@roboomp) 2026-07-05 13:10:25 +02:00
roboomp a52ed682c7 fix(ai): separated codex orchestration usage
- Added a Usage.orchestration sidecar for provider-side service tokens so Responses/Codex totals and costs stay accurate without inflating visible prompt input/cache buckets.
- Updated Codex/WebSocket usage, session/status aggregates, and usage reporting to preserve orchestration-aware totals.
- Added regressions for OpenAI Responses accounting, Codex WebSocket terminal usage, cost calculation, and session aggregation.

Fixes #4469
2026-07-03 16:44:12 +00:00
roboomp 437a871304 fix(eval): surfaced preserved patch path in agent() failure
runEvalAgent short-circuits on subagent error before mergeIsolatedChanges runs, so branch-mode transfer failures previously threw with just result.error and the caller never learned where the preserved patch lived. Enrich the pre-merge failure and reuse a shared recovery-hint builder for the apply/nested-patch failure paths so the captured patch path, branch name, and nested-patch artifacts are always surfaced.

Fixes #4437
2026-07-03 14:51:52 +00:00
can1357 1cb8608a58 Merge PR #3896: fix(task): respect task.maxConcurrency + task.maxRecursionDepth across spawn paths (@roboomp)
# Conflicts:
#	packages/coding-agent/src/eval/__tests__/agent-bridge.test.ts
#	packages/coding-agent/src/eval/agent-bridge.ts
2026-07-01 22:07:28 +02:00
can1357 88e3e77f3e Merge PR #3927: fix(agent): reject stale yield labels for override schemas (@roboomp) 2026-07-01 21:50:54 +02:00
roboomp 8614b4c086 fix(task): respected restricted spawn defaults
Resolved eval agent() and task tool defaults from the active spawn policy so restricted agents advertise and execute an allowed default.

Fixes #3973
2026-07-01 02:44:08 +00:00
roboomp 00ef58f843 fix(agent): steered override-schema subagents
- Marked eval agent schema calls as caller overrides so subagent prompts can revoke native output/yield instructions.\n- Added override-schema prompt guidance telling agents to ignore conflicting native output labels and terminal-yield the caller schema object.\n- Added prompt coverage for the override notice.\n\nRefs #3926
2026-06-30 22:30:15 +00:00
roboomp 1b9c6be129 fix(task): respect task.maxConcurrency + task.maxRecursionDepth across spawn paths
Three independent paths bypassed the user's subagent caps:

1. TaskTool.#getSpawnSemaphore sized the spawn semaphore from
   task.maxConcurrency only on first use and never re-read the setting,
   so lowering the cap mid-session left every later spawn running
   against the old ceiling. Resize the live semaphore against the
   current setting on each acquire.

2. The task tool prompt threaded MAX_CONCURRENCY through to the
   template but never rendered it. A model with task.maxConcurrency=1
   could still emit oversized tasks[] batches that registered
   immediately and piled up behind the semaphore. Render a 'Concurrency
   cap' directive in task.md whenever the setting is bounded.

3. The eval agent() bridge's assertDepthAllowed gated only against
   the hardcoded EVAL_AGENT_MAX_DEPTH=3 and ignored
   task.maxRecursionDepth, so a user-tightened recursion limit
   (0='None', 1='Single') still let cell-spawned subagents recurse
   to depth 3. Mirror the task tool's canSpawnAtDepth gate, clamped
   by the hard ceiling.

Fixes #3895
2026-06-30 11:37:58 +00:00
can1357 037c528e54 Merge PR #3350: increase Julia prelude test timeout to fix CI flake (@oldschoola)
# Conflicts:
#	packages/coding-agent/src/eval/__tests__/julia-prelude.test.ts
2026-06-28 18:41:29 +02:00
can1357 577d2a8eb8 style: biome format/organize-imports across integrated PRs 2026-06-27 02:06:38 +02:00
can1357 29269d9547 Fix eval agent isolation artifact recovery 2026-06-27 01:39:35 +02:00
can1357 84ef014faa Merge PR #3407: fix(eval): dispose one-shot eval subagents after run (@korri123) 2026-06-26 23:27:40 +02:00
can1357 305db55cd7 fix(coding-agent/eval): surfaced the exception type and message in the
- Surface the exception type and message in the error display by prepending the formatted error string to the traceback array.
- Prevent the host from hiding the actual error by ensuring the traceback is not empty, consistent with other language runners.
2026-06-26 10:06:48 +02:00
Kormákur 0a7ec389a0 fix(coding-agent): verify eval subagent cleanup 2026-06-25 09:27:38 +00:00
Kormákur 9118e2cb95 fix(coding-agent): dispose eval subagents after run 2026-06-24 22:15:03 +00:00
oldschoola 332a761625 test(eval): increase Julia prelude test and afterEach timeout to fix CI flake 2026-06-23 13:37:43 -07:00
can1357 1dd78b207e feat(coding-agent): removed unused eval helper functions
- Removed deprecated eval prelude helpers `append`, `tree`, `diff`, `sort`, `uniq`, and `counter` from all supported runtimes.
- Cleaned up runtime implementations, protocol definitions, and UI rendering logic associated with the removed helpers.
- Updated project documentation, prompts, and test suites to reflect the reduced helper API surface.
- Recorded functional changes in the package changelog.
2026-06-23 01:39:24 +02:00
can1357 24d58033bb refactor(eval): rename agent() params agent_type→agent, return_handle→handle
The eval agent() helper used `agent_type`/`return_handle` (snake_case) in
Python/Ruby/Julia and `agentType`/`returnHandle` (camelCase) in JS, forcing
the prelude docs to repeat every option twice ("JS same but camelcased").
Both are now single lowercase words identical across all four runtimes, and
`agent` matches the `task` tool's existing agent-selection parameter.

- Renamed across py/js/rb/jl preludes (signatures, forwarding, docstrings).
- Renamed the `__agent__` bridge wire protocol + `EvalAgentArgs` (`agentType`
  → `agent`, `returnHandle` → `handle`) so no prelude-side remap is needed.
- Updated prompt docs (workflow-notice.md, tools/eval.md), repo docs
  (docs/tools/eval.md, docs/python-repl.md), and all bridge/prelude tests.
- CHANGELOG: Breaking Changes entry under [Unreleased].
2026-06-22 22:12:24 +02:00
roboompandcan1357 5b6e9f904d fix(eval): paused timeout over baseline capture and surfaced nested stash-restore failures
Two Codex P2 findings landed against 978d2a76d0 that were not in the previously delivered review event:

1) prepareIsolationContext() (which runs captureBaseline → walks nested repos and untracked diffs) was running OUTSIDE withBridgeTimeoutPause; on dirty/large repos the baseline walk can exceed the eval idle timeout while the runtime is blocked. Moved the prep call into the pause closure so the watchdog is suspended for the whole bridge call from prep through cleanup.

2) applyNestedPatches() swallowed git stash pop failures with only a logger.warn, so a stash-pop conflict after a successful agent commit was invisible to the workflow. Changed the helper to return Promise<string[]> of warnings; applyEligibleNestedPatches now wraps them in a <system-notification> appended to the merge summary so the caller actually sees the partial-success case.

Added regression tests:
- bridge: prepare fires after timeout-pause and before timeout-resume.
- runner: applyEligibleNestedPatches surfaces stash-restore warnings as a system-notification.
- worktree (real git): a pre-existing dirty edit on the same file the agent patches causes stash pop to conflict; the helper returns a warning naming the nested repo and the stash entry is preserved for manual recovery.

Fixes #3196
2026-06-22 21:57:35 +02:00
can1357 2f2acaf928 Merge pull request #3205: feat(eval): isolated/apply/merge options for agent() helper
Resolves conflict in test/task/worktree.test.ts by keeping both the
getRepoRoot (main) and applyNestedPatches (PR) describe blocks.

Extends the PR's Python/JS work to the remaining workflow runtimes:
- eval/rb/prelude.rb, eval/jl/prelude.jl: agent() now accepts and
  forwards isolated/apply/merge (as booleans) plus returnHandle, and the
  return_handle node carries isolated/patch_path/branch_name/
  nested_patches/changes_applied/isolation_summary.

Post-merge fixups:
- task/index.ts: drop dead commitStyle var (the dedup refactor reads
  task.isolation.commits inside makeIsolationCommitMessage).
- CHANGELOG: move the misplaced Added entry under [Unreleased], correct
  the stale "defaults track task.isolation.mode" wording to the final
  strict opt-in behavior, and note all four runtimes.

Fixes #3196
2026-06-22 21:56:45 +02:00
roboomp 4c3d785048 test(eval): raised Julia prelude per-test timeout to 30s
The two julia-prelude tests pay a ~11-12s Julia kernel cold-start
(JIT + package precompile) when run with reset: true, exceeding
Bun's default 5000ms per-test timeout. CI exposed this since 33e2594f0
(Julia eval support) ran on PR runners (ubuntu-22.04, Julia
preinstalled) rather than the omp-kata self-hosted runners that lack
Julia and skip the suite.

Bumped both tests to 30_000ms, matching the precedent in
agent-bridge.test.ts:540 for similar persistent-kernel tests.

Fixes #3274
2026-06-22 18:41:45 +00:00
roboomp 15ae84f0de fix(eval): paused timeout through isolation merge/apply
Previously withBridgeTimeoutPause only wrapped the subagent subprocess; mergeIsolatedChanges, applyNestedPatches, nested commit-message generation, and artifact cleanup ran with the eval watchdog re-armed. A cherry-pick or large patch apply could trip the cell timeout and abort successful post-processing.

Moved the entire bridge work (subprocess + merge + nested apply + cleanup + usage recording) inside one withBridgeTimeoutPause block. The pause helper still resumes via its finally on success and on throw, so existing failure paths are unchanged.

Added a regression that captures the emitted op order and asserts merge fires after timeout-pause and before timeout-resume.

Fixes #3196
2026-06-22 18:41:44 +00:00
roboomp 7438efc5b3 style: bun run fix 2026-06-22 18:29:34 +00:00
roboomp fbb7255e8e fix(eval): flipped isolated default to strict opt-in
Per maintainer ruling on #3196, eval agent() now defaults to non-isolated regardless of task.isolation.mode, mirroring the task tool. isolated=true is the only way to turn it on; isolated=true while task.isolation.mode === "none" still throws the same clear error.

Updated tests, workflow-notice.md, and Python agent() docstring to reflect the strict opt-in contract. Existing isolation tests now pass isolated:true explicitly; the inherit-from-settings assertion is replaced with a default-off + isolated=true opt-in regression.

Fixes #3196
2026-06-22 18:29:28 +00:00
roboomp a48f1ea813 fix(eval): persisted nested patches before throwing apply failures
When an isolated apply fails the bridge throws a ToolError and never returns details, so the nested-patch payload that previously lived in details.nestedPatches was unrecoverable after the isolation worktree was torn down.

The bridge now writes each captured nested patch to a file under the per-call artifacts dir (e.g. <agentId>.nested-<index>-<slug>.patch) before throwing and includes the resolved paths in the error message so the caller can apply them manually.

Added a regression test verifying the persisted file exists with the original patch contents and that the path is surfaced in the thrown error.

Fixes #3196
2026-06-22 04:21:11 +00:00
roboomp c1b01f1cb9 fix(eval): surfaced isolated apply failures as ToolError
A failed isolated apply (changesApplied === false) previously only set details.isolationSummary and returned the subagent text. Schema-backed agent() calls then parsed the JSON and returned the object, so workflows saw a successful structured result while none of the edits had landed.

Throw a ToolError when mergeIsolatedChanges reports a failed apply, with the merge summary plus a recovery hint pointing at the preserved patch/branch/nested artifacts so the caller can apply manually.

Added regression tests for the schema and non-schema apply-failure paths.

Fixes #3196
2026-06-22 04:13:43 +00:00
can1357 33e2594f03 feat(coding-agent): added support for Julia and display language icons in code cells
- Added Julia language support to the theme symbol maps.
- Enabled language icons in code cell headers for the eval tool renderer.
2026-06-22 06:13:01 +02:00
roboomp 3677e71e9d fix(eval): exposed nested apply=false patches
Branch-mode isolation can capture nested repository changes without creating a root branch. Eval agent() with apply=false previously treated that shape as no captured changes and returned no recoverable nested patch payload after the isolation worktree was removed.

Expose captured nested patches in EvalAgentResult details and copy them onto JS/Python returnHandle nodes (nestedPatches / nested_patches). Document the return_handle escape hatch and add regression coverage for branch-mode nested-only apply=false runs.

Fixes #3196
2026-06-22 04:08:56 +00:00
roboomp 636fbb4fcc style: bun run fix 2026-06-22 03:59:27 +00:00
roboomp a6ee95df08 fix(eval): preserved returnHandle artifacts and nested branch patches
Eval preludes now forward returnHandle to the bridge so no-session eval runs can preserve the temp artifacts backing returned agent:// handles. The bridge keeps those temporary artifact directories whenever returnHandle is requested, including non-isolated runs and successful isolated applies.

Branch-mode isolation now treats nested-only changes as merge-eligible even when no root branch was produced, letting callers apply nested patches instead of dropping them when the root repo had no diff.

Added regression coverage for returnHandle artifact preservation and nested-only branch isolation.

Fixes #3196
2026-06-22 03:59:17 +00:00
roboomp 54f4652573 fix(eval): exposed apply=false artifacts on the agent() return_handle node
When agent() ran with schema and apply=false, the bridge correctly returned the captured patch/branch in details, but the preludes only forwarded id/agent/handle/data on the returnHandle node. Structured workflows had no way to recover the artifact for a manual apply.

Both runtimes now copy isolated, patchPath/branchName, changesApplied, and isolationSummary onto the returnHandle node (snake_case in Python, camelCase in JS), keeping null changesApplied so apply=false stays distinguishable from a successful apply. Updated the workflow notice and the Python agent() docstring to point callers at return_handle as the artifact escape hatch for isolated+apply=false runs. Added prelude tests locking the new node shape in both runtimes.

Fixes #3196
2026-06-21 17:35:26 +00:00
roboomp 9b33e0e644 style: bun run fix 2026-06-21 17:28:23 +00:00
roboomp 068a27b07f fix(eval): preserved temp artifacts when isolated agent() runs with apply=false
The cleanup gate was treating changesApplied===null (apply=false) the same as a clean apply, deleting the temp artifacts dir before returning details.patchPath. Sessions without a session file (which fall back to a per-call tmp dir) ended up with a patchPath pointing at a removed file, defeating the documented manual-apply path.

Tightened the cleanup condition to remove the temp dir only on a confirmed clean apply (changesApplied===true); apply=false and failed applies both keep the artifact for the caller.

Added regression tests for the apply=false preserve case and the apply-succeeds cleanup case.

Fixes #3196
2026-06-21 17:28:16 +00:00