diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 567accbe6..74419b3e6 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -8,7 +8,7 @@ ### Fixed -- Fixed the tool-argument repair layer applying lossy repairs on union-branch diagnoses: when a value failed every `anyOf`/`oneOf` variant, the first failing branch's issues were treated as authoritative, so object payloads got JSON-stringified into string-typed fields and unrecognized keys were silently deleted — corrupting subagent `yield` payloads (validation "passed" and parents received `summary: "{\"purge\":13,…}"` instead of a retryable error). Issues surfaced from a failed union branch are now marked at every depth and only receive lossless repairs (JSON-string parsing, boolean spellings, scalar coercion); container stringification, key deletion, and singleton-array wrapping require an authoritative non-union diagnosis. +- Fixed the tool-argument repair layer applying lossy repairs on union-branch diagnoses: when a value failed every `anyOf`/`oneOf` variant, the first failing branch's issues were treated as authoritative, so object payloads got JSON-stringified into string-typed fields and unrecognized keys were silently deleted — corrupting subagent `yield` payloads (validation "passed" and parents received `summary: "{\"purge\":13,…}"` instead of a retryable error). Issues from a failed union are now marked at every depth and only receive lossless repairs (JSON-string parsing, boolean spellings, scalar coercion) — unless a `const`/`enum` discriminator uniquely tag-selects the intended variant, whose diagnosis stays fully repairable (and is now the surfaced one, rather than the first failing branch's). - Fixed local OpenAI-compatible servers with strict `chat_template_kwargs` whitelists (e.g. NInfer) failing every Qwen 3.8+ turn with `400 chat_template_kwargs.reasoning_effort is not supported` after the effort routing fix: the reasoning-effort fallback now recognizes a rejection of the kwargs spelling itself, retries with the kwarg stripped while keeping the effort on the standard top-level `reasoning_effort` field (hoisting it there for the kwargs-only vLLM dialect), and remembers the shape for the rest of the session. Value-level rejections and drops now also update the `chat_template_kwargs.reasoning_effort` twin instead of leaving a stale effort for kwargs-reading renderers, and unknown-parameter 400s naming `reasoning_effort` are recognized as effort rejections. ## [17.3.8] - 2026-08-19 diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 6e5b0d805..34e7d19f0 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -11,6 +11,7 @@ - Added `extendedContext` setting (`/settings` → Context → General, default on). When off, models with a premium long-context price tier (OpenAI GPT-5.6 Sol/Terra/Luna bill 2x input / 1.5x output above 272K input tokens, on both the API and subscription Codex) are capped at the standard-pricing threshold — they appear as 272K again and compaction fires before a request crosses into premium billing. Toggling mid-session re-clamps or restores the active model's window immediately. Anthropic Claude 4.6+ serves its full 1M window at standard pricing, so no Anthropic model is affected. - Added click-to-toggle and drag-to-reorder controls for list-valued `/settings` editors. - Added `compaction.asyncEnabled` (Async Compaction, default on): when context enters the band just below the compaction threshold, maintenance speculatively summarizes in the background off a branch snapshot (first configured LLM-backed method — remote, handoff, or soft — isolated from the live turn by a side session id) and holds the armed result; crossing the threshold then splices it in instantly instead of blocking on a summarization round-trip. Armed results are invalidated by branch changes, reset boundaries, model switches that strand provider-native replay payloads, and context growth past `keepRecentTokens` (which re-speculates). The status line pulses the auto-compact icon while a speculation runs and holds it in accent once a result is armed. +- Added auto-backgrounding for eval cells, mirroring the bash policy: with `eval.autoBackground.enabled` (default off), a cell still running after `eval.autoBackground.thresholdMs` (default 60s) — or when a queued user/peer message steers mid-wait — converts into a background job that keeps executing on the kernel and delivers its result automatically. Both settings inherit the RPC-host default treatment alongside their bash counterparts. ### Changed @@ -20,6 +21,7 @@ - `/settings` rows can now carry a risk note: a warning glyph on the row plus a warning-colored line above the description. `External Thinking` (`externalThinking`, `--external-thinking`) is the first user — providers have flagged the request shape it produces as abuse, up to account-level enforcement, so both the settings entry and `--help` now say so. - The todo HUD header now draws a summed progress bar counting closed/total tasks across every stage. Once all tasks close, the bar smoothly collapses before the row disappears. - The todo HUD now carries overall progress in the tree spine instead of a horizontal header bar: the top-level connector column runs unbroken from the `TODO` header down the left edge and closes with a short `└────` elbow tail under the block. The closed/total fraction (summed across every stage) fills that path in accent — down the spine, around the bend, out along the tail. Once all tasks close, the accent smoothly drains back up before the header row disappears. +- The todo HUD now carries overall progress in the tree spine instead of a horizontal header bar: the top-level connector column runs unbroken from the `TODO` header to the last row, and the closed/total fraction (summed across every stage) colors it top-down in accent. Once all tasks close, the accent smoothly drains out of the spine before the header row disappears. - Token counting is now scoped to the model being billed rather than to a process-global tokenizer: session maintenance, stats, advisors, `/context`, snapcompact inline imaging, and `compress` each count through the owning agent's `Tokenizer` (`agent.tokenizer`). Message counting is `Tokenizer.countMessage`/`countMessages` (replacing the free `estimateTokens(message, tokenizer)` helper; the legacy shim keeps a compat `estimateTokens` export for legacy pi extensions). `estimateToolSchemaTokens`, `estimateSkillsTokens`, `computeNonMessageTokens`, and `computeNonMessageBreakdown` take an explicit tokenizer; standalone prompt inspection intentionally keeps the default estimate because it has no resolved catalog model. - The advisor runtime's `maintainContext` hook now receives the pending update as a message instead of a pre-computed token count — sizing it needs the advisor model's tokenizer, which the host owns. - Handoff no longer starts a new session: `/handoff` and the auto-maintenance `handoff` method now commit the generated document as a regular compaction entry on the current session (document becomes the summary, recent history is kept per `compaction.keepRecentTokens`, session id/transcript/cache key unchanged). The `session_before_switch`/`session_switch` extension events no longer fire with reason `"handoff"`, mid-turn maintenance no longer skips the handoff preference, and overflow recovery can now apply a pre-armed handoff result. diff --git a/packages/coding-agent/src/async/auto-background.ts b/packages/coding-agent/src/async/auto-background.ts new file mode 100644 index 000000000..22f151d1d --- /dev/null +++ b/packages/coding-agent/src/async/auto-background.ts @@ -0,0 +1,78 @@ +/** + * Shared foreground-wait helpers for tools that auto-background long-running + * work as {@link AsyncJobManager} jobs (bash commands, eval cells): the + * LLM-facing background notice, the threshold-vs-timeout wait budget, and the + * settlement race against abort/steering signals. + */ + +/** Default foreground-wait threshold before a tool call auto-backgrounds. */ +export const DEFAULT_AUTO_BACKGROUND_THRESHOLD_MS = 60_000; +/** LLM-facing footer appended when a tool call is converted into a background job. */ +export function formatBackgroundNotice(jobId: string): string { + return `Backgrounded as job ${jobId}; result will be delivered automatically.`; +} + +/** + * How long a tool foreground-waits before backgrounding. Bounded by the call's + * own timeout minus a small buffer so a deadline expiry resolves inline instead + * of backgrounding moments before it fires. `0` means background immediately. + */ +export function resolveAutoBackgroundWaitMs(thresholdMs: number, timeoutMs: number | undefined): number { + if (thresholdMs <= 0) return 0; + if (timeoutMs === undefined) return thresholdMs; + const timeoutBufferMs = 1_000; + return Math.max(0, Math.min(thresholdMs, timeoutMs - timeoutBufferMs)); +} + +/** Non-settled outcomes of {@link raceJobSettlement}. */ +export type JobWaitInterrupt = { kind: "running" } | { kind: "steer" } | { kind: "aborted" }; + +/** + * Race a managed job's settlement against the auto-background threshold, the + * caller's abort signal, and the turn's steering signal. Returns the job's own + * completion when it settles first; otherwise reports why the wait ended: + * "running" = threshold elapsed (background it), "steer" = a queued message + * arrived mid-wait, "aborted" = the caller cancelled. + */ +export async function raceJobSettlement( + completion: Promise, + thresholdMs: number, + signal?: AbortSignal, + steeringSignal?: AbortSignal, +): Promise { + if (signal?.aborted) { + return { kind: "aborted" }; + } + if (steeringSignal?.aborted) { + return { kind: "steer" }; + } + + // Cancellable threshold: a bare Bun.sleep(thresholdMs) leaves a live, ref'd + // timer for the full threshold after the job finishes (or abort/steer) wins + // the race first — delaying SDK/headless shutdown and accumulating timers + // under fast completion rates. Settle a withResolvers promise from + // setTimeout so the finally can clear it regardless of which waiter wins. + const { promise: thresholdPromise, resolve: resolveThreshold } = Promise.withResolvers<{ kind: "running" }>(); + const thresholdTimer = setTimeout(() => resolveThreshold({ kind: "running" }), thresholdMs); + const waiters: Array> = [completion, thresholdPromise]; + + const { promise: abortedPromise, resolve: resolveAborted } = Promise.withResolvers<{ kind: "aborted" }>(); + const onAbort = () => resolveAborted({ kind: "aborted" }); + const { promise: steerPromise, resolve: resolveSteer } = Promise.withResolvers<{ kind: "steer" }>(); + const onSteer = () => resolveSteer({ kind: "steer" }); + if (signal) { + signal.addEventListener("abort", onAbort, { once: true }); + waiters.push(abortedPromise); + } + if (steeringSignal) { + steeringSignal.addEventListener("abort", onSteer, { once: true }); + waiters.push(steerPromise); + } + try { + return await Promise.race(waiters); + } finally { + clearTimeout(thresholdTimer); + signal?.removeEventListener("abort", onAbort); + steeringSignal?.removeEventListener("abort", onSteer); + } +} diff --git a/packages/coding-agent/src/async/index.ts b/packages/coding-agent/src/async/index.ts index 7d036b77e..987811f07 100644 --- a/packages/coding-agent/src/async/index.ts +++ b/packages/coding-agent/src/async/index.ts @@ -1 +1,2 @@ +export * from "./auto-background"; export * from "./job-manager"; diff --git a/packages/coding-agent/src/async/job-manager.ts b/packages/coding-agent/src/async/job-manager.ts index 21723720c..722038037 100644 --- a/packages/coding-agent/src/async/job-manager.ts +++ b/packages/coding-agent/src/async/job-manager.ts @@ -29,9 +29,12 @@ interface PollEscalationState { lastPollEndAt: number; } +/** Kind of work a managed job runs; drives job-row badges and delivery labels. */ +export type AsyncJobType = "bash" | "task" | "eval"; + export interface AsyncJob { id: string; - type: "bash" | "task"; + type: AsyncJobType; status: "running" | "completed" | "failed" | "cancelled"; startTime: number; label: string; @@ -183,7 +186,7 @@ export class AsyncJobManager { } register( - type: "bash" | "task", + type: AsyncJobType, label: string, run: (ctx: { jobId: string; diff --git a/packages/coding-agent/src/config/settings-schema.ts b/packages/coding-agent/src/config/settings-schema.ts index 8f18d972a..69f10c99c 100644 --- a/packages/coding-agent/src/config/settings-schema.ts +++ b/packages/coding-agent/src/config/settings-schema.ts @@ -3652,6 +3652,22 @@ export const SETTINGS_SCHEMA = { }, }, + "eval.autoBackground.enabled": { + type: "boolean", + default: false, + ui: { + tab: "shell", + group: "Eval & Runtimes", + label: "Eval Auto-Background", + description: "Automatically background long-running eval cells and deliver the result later", + }, + }, + + "eval.autoBackground.thresholdMs": { + type: "number", + default: 60_000, + }, + // Runtime knobs (consumed by eval backends and the /python slash command) "python.kernelMode": { type: "enum", diff --git a/packages/coding-agent/src/eval/types.ts b/packages/coding-agent/src/eval/types.ts index 2463ada07..64141c989 100644 --- a/packages/coding-agent/src/eval/types.ts +++ b/packages/coding-agent/src/eval/types.ts @@ -45,4 +45,10 @@ export interface EvalToolDetails { languages?: EvalLanguage[]; /** Optional human-readable notice (e.g. fallback explanation). */ notice?: string; + /** Present when the cell was auto-backgrounded as an async job. */ + async?: { + state: "running" | "completed" | "failed"; + jobId: string; + type: "eval"; + }; } diff --git a/packages/coding-agent/src/main.ts b/packages/coding-agent/src/main.ts index 4a8f65cee..0bf73f280 100644 --- a/packages/coding-agent/src/main.ts +++ b/packages/coding-agent/src/main.ts @@ -155,6 +155,8 @@ const RPC_BACKGROUND_DEFAULTED_SETTING_PATHS: SettingPath[] = [ "async.maxJobs", "bash.autoBackground.enabled", "bash.autoBackground.thresholdMs", + "eval.autoBackground.enabled", + "eval.autoBackground.thresholdMs", ]; // Protocol-mode hosts opt into a small set of paths whose host-default we diff --git a/packages/coding-agent/src/modes/utils/transcript-render-helpers.ts b/packages/coding-agent/src/modes/utils/transcript-render-helpers.ts index 81cc58250..c72d1ab29 100644 --- a/packages/coding-agent/src/modes/utils/transcript-render-helpers.ts +++ b/packages/coding-agent/src/modes/utils/transcript-render-helpers.ts @@ -7,6 +7,7 @@ import type { AgentMessage } from "@oh-my-pi/pi-agent-core"; import { type Component, Text } from "@oh-my-pi/pi-tui"; import { formatBytes, formatDuration } from "@oh-my-pi/pi-utils"; +import type { AsyncJobType } from "../../async"; import { type CustomMessage, type FileMentionMessage, @@ -32,10 +33,10 @@ export function buildAsyncResultBlock(message: CustomOrHookMessage): ToolActivit const details = ( message as CustomMessage<{ jobId?: string; - type?: "bash" | "task"; + type?: AsyncJobType; label?: string; durationMs?: number; - jobs?: Array<{ jobId?: string; type?: "bash" | "task"; label?: string; durationMs?: number }>; + jobs?: Array<{ jobId?: string; type?: AsyncJobType; label?: string; durationMs?: number }>; }> ).details; const jobs = diff --git a/packages/coding-agent/src/prompts/tools/eval.md b/packages/coding-agent/src/prompts/tools/eval.md index 8afbdf5cc..6f67e763c 100644 --- a/packages/coding-agent/src/prompts/tools/eval.md +++ b/packages/coding-agent/src/prompts/tools/eval.md @@ -43,3 +43,5 @@ Acyclic waves via `agent(…, handle=true)` + `pipeline`/`parallel`: Prior top-level names survive into the next cell — reuse; NEVER re-import/re-declare. Re-read only if file changed since last read. + +{{#if autoBackgroundEnabled}}Long-running cells may auto-background and deliver later; the kernel stays busy until the cell finishes. Need inline? Raise `timeout`.{{/if}} diff --git a/packages/coding-agent/src/session/async-job-delivery.ts b/packages/coding-agent/src/session/async-job-delivery.ts index aefa4384a..d517ae35e 100644 --- a/packages/coding-agent/src/session/async-job-delivery.ts +++ b/packages/coding-agent/src/session/async-job-delivery.ts @@ -9,7 +9,7 @@ * every completion — regardless of owner — into the first top-level session. */ import { prompt } from "@oh-my-pi/pi-utils"; -import type { AsyncJob } from "../async"; +import type { AsyncJob, AsyncJobType } from "../async"; import asyncResultTemplate from "../prompts/tools/async-result.md" with { type: "text" }; import type { CustomMessage } from "./messages"; @@ -41,7 +41,7 @@ export interface AsyncResultEntry { type AsyncResultJobDetails = { jobId: string; - type?: "bash" | "task"; + type?: AsyncJobType; label?: string; durationMs?: number; }; diff --git a/packages/coding-agent/src/task/executor.ts b/packages/coding-agent/src/task/executor.ts index 64427cf6b..c1ae870df 100644 --- a/packages/coding-agent/src/task/executor.ts +++ b/packages/coding-agent/src/task/executor.ts @@ -888,7 +888,7 @@ export function createSubagentSettings( return Settings.isolated( { ...snapshot, - // Async jobs and bash auto-backgrounding are inherited from the parent: + // Async jobs and bash/eval auto-backgrounding are inherited from the parent: // background jobs are owner-routed to the subagent's own session, and // the run driver's quiescence barrier + teardown reap guarantee no // owner job outlives the run, so worktree capture/cleanup stays diff --git a/packages/coding-agent/src/tools/bash.ts b/packages/coding-agent/src/tools/bash.ts index a8e2939a9..aab2904f3 100644 --- a/packages/coding-agent/src/tools/bash.ts +++ b/packages/coding-agent/src/tools/bash.ts @@ -10,6 +10,12 @@ import type { import type { Component } from "@oh-my-pi/pi-tui"; import { ImageProtocol, TERMINAL } from "@oh-my-pi/pi-tui"; import { getProjectDir, isEnoent, logger, prompt } from "@oh-my-pi/pi-utils"; +import { + DEFAULT_AUTO_BACKGROUND_THRESHOLD_MS, + formatBackgroundNotice, + raceJobSettlement, + resolveAutoBackgroundWaitMs, +} from "../async"; import type { Settings } from "../config/settings"; import { applyDirenvPreflight, type BashResult, executeBash } from "../exec/bash-executor"; import type { RenderResultOptions } from "../extensibility/custom-tools/types"; @@ -57,7 +63,6 @@ import { clampTimeout, TOOL_TIMEOUTS } from "./tool-timeouts"; export const BASH_DEFAULT_PREVIEW_LINES = DEFAULT_TERMINAL_PREVIEW_LINES; const BASH_ENV_NAME_PATTERN = /^[A-Za-z_][A-Za-z0-9_]*$/; -const DEFAULT_AUTO_BACKGROUND_THRESHOLD_MS = 60_000; const BASH_APPROVAL_SHELL_CONTROL_CHARS: Record = { "\n": true, "\r": true, @@ -505,10 +510,6 @@ function formatExitCodeNotice(exitCode: number): string { return `Command exited with code ${exitCode}`; } -function formatBackgroundNotice(jobId: string): string { - return `Backgrounded as job ${jobId}; result will be delivered automatically.`; -} - /** * Strip the trailing occurrence of `notice` (plus a single surrounding newline * on each side) so the TUI can echo the value via a styled footer label @@ -894,60 +895,6 @@ export class BashTool implements AgentTool { - if (signal?.aborted) { - return { kind: "aborted" }; - } - if (steeringSignal?.aborted) { - return { kind: "steer" }; - } - - // Cancellable threshold: a bare Bun.sleep(thresholdMs) leaves a live, ref'd - // timer for the full threshold after the command finishes (or abort/steer) - // wins the race first — delaying SDK/headless shutdown and accumulating - // timers under fast command rates. Settle a withResolvers promise from - // setTimeout so the finally can clear it regardless of which waiter wins. - const { promise: thresholdPromise, resolve: resolveThreshold } = Promise.withResolvers<{ - kind: "running"; - }>(); - const thresholdTimer = setTimeout(() => resolveThreshold({ kind: "running" }), thresholdMs); - const waiters: Array< - Promise - > = [job.completion, thresholdPromise]; - - const { promise: abortedPromise, resolve: resolveAborted } = Promise.withResolvers<{ kind: "aborted" }>(); - const onAbort = () => resolveAborted({ kind: "aborted" }); - const { promise: steerPromise, resolve: resolveSteer } = Promise.withResolvers<{ kind: "steer" }>(); - const onSteer = () => resolveSteer({ kind: "steer" }); - if (signal) { - signal.addEventListener("abort", onAbort, { once: true }); - waiters.push(abortedPromise); - } - if (steeringSignal) { - steeringSignal.addEventListener("abort", onSteer, { once: true }); - waiters.push(steerPromise); - } - try { - return await Promise.race(waiters); - } finally { - clearTimeout(thresholdTimer); - signal?.removeEventListener("abort", onAbort); - steeringSignal?.removeEventListener("abort", onSteer); - } - } - - #resolveAutoBackgroundWaitMs(timeoutMs: number | undefined): number { - if (this.#autoBackgroundThresholdMs <= 0) return 0; - if (timeoutMs === undefined) return this.#autoBackgroundThresholdMs; - const timeoutBufferMs = 1_000; - return Math.max(0, Math.min(this.#autoBackgroundThresholdMs, timeoutMs - timeoutBufferMs)); - } - async execute( _toolCallId: string, { @@ -1099,7 +1046,7 @@ export class BashTool implements AgentTool 0) { @@ -670,6 +674,9 @@ export const evalToolRenderer = { if (noticeLine) { lines.push(noticeLine); } + if (asyncLine) { + lines.push(asyncLine); + } if (warningLine) { lines.push(warningLine); } @@ -693,14 +700,19 @@ export const evalToolRenderer = { ); if (!combinedOutput && statusLines.length === 0) { - const lines = [timeoutLine, noticeLine, warningLine].filter(Boolean) as string[]; + const lines = [timeoutLine, noticeLine, asyncLine, warningLine].filter(Boolean) as string[]; return new Text(lines.join("\n"), 0, 0); } if (!combinedOutput && statusLines.length > 0) { - const lines = [uiTheme.fg("dim", "Status"), ...statusLines, timeoutLine, noticeLine, warningLine].filter( - Boolean, - ) as string[]; + const lines = [ + uiTheme.fg("dim", "Status"), + ...statusLines, + timeoutLine, + noticeLine, + asyncLine, + warningLine, + ].filter(Boolean) as string[]; return new Text(lines.join("\n"), 0, 0); } @@ -714,6 +726,7 @@ export const evalToolRenderer = { ...(statusLines.length > 0 ? [uiTheme.fg("dim", "Status"), ...statusLines] : []), timeoutLine, noticeLine, + asyncLine, warningLine, ].filter(Boolean) as string[]; return new Text(lines.join("\n"), 0, 0); @@ -765,6 +778,9 @@ export const evalToolRenderer = { if (noticeLine) { outputLines.push(truncateToWidth(noticeLine, width)); } + if (asyncLine) { + outputLines.push(truncateToWidth(asyncLine, width)); + } if (warningLine) { outputLines.push(truncateToWidth(warningLine, width)); } diff --git a/packages/coding-agent/src/tools/eval.ts b/packages/coding-agent/src/tools/eval.ts index aa819d442..aa1278760 100644 --- a/packages/coding-agent/src/tools/eval.ts +++ b/packages/coding-agent/src/tools/eval.ts @@ -2,6 +2,12 @@ import { type } from "@oh-my-pi/omptype"; import type { AgentTool, AgentToolContext, AgentToolResult, AgentToolUpdateCallback } from "@oh-my-pi/pi-agent-core"; import type { ImageContent, ToolExample } from "@oh-my-pi/pi-ai"; import { prompt } from "@oh-my-pi/pi-utils"; +import { + DEFAULT_AUTO_BACKGROUND_THRESHOLD_MS, + formatBackgroundNotice, + raceJobSettlement, + resolveAutoBackgroundWaitMs, +} from "../async"; import { jsBackend, juliaBackend, pythonBackend, rubyBackend } from "../eval"; import type { ExecutorBackend, ExecutorBackendResult } from "../eval/backend"; import { EVAL_TIMEOUT_PAUSE_OP, EVAL_TIMEOUT_RESUME_OP } from "../eval/bridge-timeout"; @@ -169,6 +175,8 @@ export interface EvalToolDescriptionOptions { * `false`/`""` hides `agent()`, and a comma list drives the advertised default. */ spawns?: boolean | string | null; + /** Advertise auto-backgrounding of long-running cells in the tool prompt. */ + autoBackgroundEnabled?: boolean; } export function getEvalToolDescription(options: EvalToolDescriptionOptions = {}): string { @@ -182,6 +190,7 @@ export function getEvalToolDescription(options: EvalToolDescriptionOptions = {}) js, rb, jl, + autoBackgroundEnabled: options.autoBackgroundEnabled ?? false, spawns: spawnPolicy.enabled, spawnDefaultAgent: spawnPolicy.defaultAgent, spawnAllowedAgentsText: spawnPolicy.allowedPromptText, @@ -206,6 +215,11 @@ interface ResolvedEvalCell { resolved: ResolvedBackend; } +/** Settlement handed from a managed eval job to its foreground waiter. */ +type ManagedEvalJobCompletion = + | { kind: "completed"; result: AgentToolResult } + | { kind: "failed"; error: unknown }; + function uniqueEvalLanguages(cells: ResolvedEvalCell[]): EvalLanguage[] { return [...new Set(cells.map(cell => cell.resolved.backend.id))]; } @@ -302,6 +316,7 @@ export class EvalTool implements AgentTool { rb: backends.ruby, jl: backends.julia, spawns: sessionSpawns, + autoBackgroundEnabled: this.session.settings.get("eval.autoBackground.enabled"), }); } /** All reuse-chain examples; the `examples` getter filters by enabled languages. */ @@ -394,7 +409,7 @@ export class EvalTool implements AgentTool { params: typeof evalSchema.infer, signal?: AbortSignal, onUpdate?: AgentToolUpdateCallback, - _ctx?: AgentToolContext, + ctx?: AgentToolContext, ): Promise> { if (this.#proxyExecutor) { return this.#proxyExecutor(params, signal); @@ -428,6 +443,185 @@ export class EvalTool implements AgentTool { const languages = uniqueEvalLanguages(cells); const notice = detailsNotice(cells); const sessionAbortController = new AbortController(); + const emitToolUpdate = onUpdate + ? (text: string, details: EvalToolDetails): void => { + onUpdate({ content: [{ type: "text", text }], details }); + } + : undefined; + const run = ( + runSignal: AbortSignal | undefined, + emitUpdate: ((text: string, details: EvalToolDetails) => void) | undefined, + ): Promise> => { + const execution = this.#runCells({ + session, + cells, + languages, + notice, + excludeWebP, + signal: runSignal, + sessionAbortController, + emitUpdate, + }); + return session.trackEvalExecution?.(execution, sessionAbortController) ?? execution; + }; + + const autoBgManager = session.asyncJobManager; + // At the running-job cap, fall through to direct foreground execution + // instead of failing every eval call until a slot frees up. + if (!session.settings.get("eval.autoBackground.enabled") || !autoBgManager || autoBgManager.atCapacity) { + return await run(signal, emitToolUpdate); + } + + const thresholdMs = Math.max( + 0, + Math.floor(session.settings.get("eval.autoBackground.thresholdMs") ?? DEFAULT_AUTO_BACKGROUND_THRESHOLD_MS), + ); + // The wait budget mirrors #runCells' clamped cell timeout. The cell budget + // is runtime work (it pauses across agent()/tool bridge calls), so a cell + // can legitimately outlive it in wall time — exactly the case + // backgrounding exists for. + const cellTimeoutMs = + cells[0].timeoutMs === 0 + ? undefined + : clampTimeout("eval", cells[0].timeoutMs / 1000, session.settings.get("tools.maxTimeout")) * 1000; + const autoBackgroundWaitMs = resolveAutoBackgroundWaitMs(thresholdMs, cellTimeoutMs); + const startBackgrounded = autoBackgroundWaitMs === 0; + + const rawLabel = params.title?.trim() || params.code.trim().split("\n", 1)[0] || "eval cell"; + const label = rawLabel.length > 120 ? `${rawLabel.slice(0, 117)}...` : rawLabel; + + let latestText = ""; + let latestDetails: EvalToolDetails | undefined; + let forwardUpdates = !startBackgrounded; + const completion = Promise.withResolvers(); + + const jobId = autoBgManager.register( + "eval", + label, + async ({ jobId, signal: runSignal, reportProgress }) => { + try { + const result = await run(runSignal, (text, details) => { + latestText = text; + latestDetails = details; + void reportProgress(text, { async: { state: "running", jobId, type: "eval" } }); + if (forwardUpdates) emitToolUpdate?.(text, details); + }); + const finalText = result.content.find(block => block.type === "text")?.text ?? ""; + latestText = finalText; + // Hand the full result (images included) to the foreground waiter + // before deciding the job's terminal state. + completion.resolve({ kind: "completed", result }); + if (result.isError === true) { + // A failed, cancelled, or timed-out cell is a completed execution + // that errored. Re-enter the failure path so the job manager + // records it as failed and delivers the error text. + throw new ToolError(finalText || "Eval cell failed"); + } + await reportProgress(finalText, { async: { state: "completed", jobId, type: "eval" } }); + return finalText; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + latestText = message; + completion.resolve({ kind: "failed", error }); + await reportProgress(message, { async: { state: "failed", jobId, type: "eval" } }); + throw error; + } + }, + { ownerId: session.getAgentId?.() ?? undefined }, + ); + + if (startBackgrounded) { + return this.#buildBackgroundStartResult(jobId, cells, languages, notice, latestText, latestDetails); + } + // Suppress the completion delivery up front so a job finishing while we + // foreground-wait cannot also be injected by the delivery loop. Lifted + // via resumeDeliveries() if we end up backgrounding after all. + autoBgManager.acknowledgeDeliveries([jobId]); + const waitResult = await raceJobSettlement( + completion.promise, + autoBackgroundWaitMs, + signal, + ctx?.toolCall?.steeringSignal, + ); + if (waitResult.kind === "completed") { + return waitResult.result; + } + if (waitResult.kind === "failed") { + throw waitResult.error; + } + if (waitResult.kind === "aborted") { + autoBgManager.cancel(jobId); + throw new ToolAbortError(latestText || "Eval cell aborted"); + } + forwardUpdates = false; + autoBgManager.resumeDeliveries([jobId]); + // "steer": a queued user/peer message arrived mid-wait — background the + // cell (it keeps running) so the message injects promptly. + const steerNotice = + waitResult.kind === "steer" + ? "Backgrounded early to handle an incoming message; the cell keeps running." + : undefined; + return this.#buildBackgroundStartResult(jobId, cells, languages, notice, latestText, latestDetails, steerNotice); + } + + /** + * Tool result returned when a cell converts into a background job: the live + * output tail plus the background notice, with details carrying the running + * cell snapshot and the async job marker the transcript renderer keys on. + */ + #buildBackgroundStartResult( + jobId: string, + cells: ResolvedEvalCell[], + languages: EvalLanguage[], + notice: string | undefined, + previewText: string, + latestDetails: EvalToolDetails | undefined, + extraNotice?: string, + ): AgentToolResult { + // latestDetails snapshots are per-update copies (buildUpdateDetails), so + // tagging the async marker on cannot leak into later job progress. + const details: EvalToolDetails = latestDetails ?? { + language: languages[0], + languages, + cells: cells.map(cell => ({ + index: cell.index, + title: cell.title, + code: cell.code, + language: cell.resolved.backend.id, + output: previewText, + status: "running" as const, + })), + }; + if (notice) details.notice ??= notice; + details.async = { state: "running", jobId, type: "eval" }; + const lines: string[] = []; + const trimmedPreview = previewText.trimEnd(); + if (trimmedPreview.length > 0) { + lines.push(trimmedPreview, ""); + } + if (extraNotice) { + lines.push(extraNotice, ""); + } + lines.push(formatBackgroundNotice(jobId)); + return { content: [{ type: "text", text: lines.join("\n") }], details }; + } + + /** + * Execute the resolved cells against their backends, streaming tail/detail + * updates through `emitUpdate`. Runs identically in the foreground path and + * inside a managed background job (which passes the job's own signal). + */ + async #runCells(options: { + session: ToolSession; + cells: ResolvedEvalCell[]; + languages: EvalLanguage[]; + notice: string | undefined; + excludeWebP: boolean | undefined; + signal: AbortSignal | undefined; + sessionAbortController: AbortController; + emitUpdate?: (text: string, details: EvalToolDetails) => void; + }): Promise> { + const { session, cells, languages, notice, excludeWebP, signal, sessionAbortController, emitUpdate } = options; let outputSink: OutputSink | undefined; let outputSummary: OutputSummary | undefined; let outputDumped = false; @@ -437,309 +631,299 @@ export class EvalTool implements AgentTool { outputDumped = true; return outputSummary; }; + try { + if (signal?.aborted) { + throw new ToolAbortError(); + } + session.assertEvalExecutionAllowed?.(); - const execution = (async (): Promise> => { - try { - if (signal?.aborted) { - throw new ToolAbortError(); - } - session.assertEvalExecutionAllowed?.(); + const tailBuffer = new TailBuffer(DEFAULT_MAX_BYTES * 2); + const jsonOutputs: unknown[] = []; + const images: ImageContent[] = []; + const statusEvents: EvalStatusEvent[] = []; - const tailBuffer = new TailBuffer(DEFAULT_MAX_BYTES * 2); - const jsonOutputs: unknown[] = []; - const images: ImageContent[] = []; - const statusEvents: EvalStatusEvent[] = []; + const cellResults: EvalCellResult[] = cells.map(cell => ({ + index: cell.index, + title: cell.title, + code: cell.code, + language: cell.resolved.backend.id, + output: "", + status: "pending", + })); + const cellOutputs: string[] = []; + // The cell currently inside backend.execute(). Streamed stdout is + // appended to its rendered `output` live so a long-running cell (e.g. a + // sleep loop) shows progress instead of nothing until it returns. A + // dedicated per-cell tail buffer keeps attribution correct and avoids + // double-counting against the aggregate `tailBuffer`; on completion the + // authoritative `cellResult.output` (below) overwrites this live tail. + let activeLiveCell: { result: EvalCellResult; buf: TailBuffer } | undefined; - const cellResults: EvalCellResult[] = cells.map(cell => ({ - index: cell.index, - title: cell.title, - code: cell.code, - language: cell.resolved.backend.id, - output: "", - status: "pending", - })); - const cellOutputs: string[] = []; - // The cell currently inside backend.execute(). Streamed stdout is - // appended to its rendered `output` live so a long-running cell (e.g. a - // sleep loop) shows progress instead of nothing until it returns. A - // dedicated per-cell tail buffer keeps attribution correct and avoids - // double-counting against the aggregate `tailBuffer`; on completion the - // authoritative `cellResult.output` (below) overwrites this live tail. - let activeLiveCell: { result: EvalCellResult; buf: TailBuffer } | undefined; - - const appendTail = (text: string) => { - tailBuffer.append(text); - }; - - const buildUpdateDetails = (): EvalToolDetails => { - const details: EvalToolDetails = { - language: languages[0], - languages, - cells: cellResults.map(cell => ({ - ...cell, - statusEvents: cell.statusEvents ? [...cell.statusEvents] : undefined, - })), - }; - if (jsonOutputs.length > 0) { - details.jsonOutputs = jsonOutputs; - } - if (images.length > 0) { - details.images = images; - } - if (statusEvents.length > 0) { - details.statusEvents = statusEvents; - } - if (notice) { - details.notice = notice; - } - return details; - }; - - const pushUpdate = () => { - if (!onUpdate) return; - const tailText = tailBuffer.text(); - onUpdate({ - content: [{ type: "text", text: tailText }], - details: buildUpdateDetails(), - }); - }; - - const sessionFile = session.getSessionFile?.() ?? undefined; - const kernelOwnerId = session.getEvalKernelOwnerId?.() ?? undefined; - const { path: artifactPath, id: artifactId } = (await session.allocateOutputArtifact?.("eval")) ?? {}; - session.assertEvalExecutionAllowed?.(); - outputSink = new OutputSink({ - artifactPath, - artifactId, - headBytes: resolveOutputSinkHeadBytes(session.settings), - maxColumns: resolveOutputMaxColumns(session.settings), - onChunk: chunk => { - appendTail(chunk); - if (activeLiveCell) { - activeLiveCell.buf.append(chunk); - activeLiveCell.result.output = activeLiveCell.buf.text(); - } - pushUpdate(); - }, - }); - const sessionId = session.getEvalSessionId?.() ?? defaultEvalSessionId(session); - - for (let i = 0; i < cells.length; i++) { - const cell = cells[i]; - const backend = cell.resolved.backend; - // The per-cell `timeout` is a budget on the cell runtime's *own* - // work. Host-side `agent()`/`parallel()`/`completion()` bridge calls suspend - // that budget entirely and restart a fresh timeout window when control - // returns to the active backend runtime. Compute, stdout, `log()`/`phase()`, and - // ordinary tool calls all count against the budget. The watchdog drives - // `combinedSignal`; we pass no wall-clock deadline downstream so the - // backends never arm a competing fixed timer. - const idleTimeoutMs = - cell.timeoutMs === 0 - ? undefined - : clampTimeout("eval", cell.timeoutMs / 1000, session.settings.get("tools.maxTimeout")) * 1000; - const idle = idleTimeoutMs === undefined ? undefined : new IdleTimeout(idleTimeoutMs); - const combinedSignal = - signal && idle - ? AbortSignal.any([signal, idle.signal, sessionAbortController.signal]) - : signal - ? AbortSignal.any([signal, sessionAbortController.signal]) - : idle - ? AbortSignal.any([idle.signal, sessionAbortController.signal]) - : sessionAbortController.signal; - - const cellResult = cellResults[i]; - cellResult.status = "running"; - cellResult.output = ""; - cellResult.statusEvents = undefined; - cellResult.exitCode = undefined; - cellResult.durationMs = undefined; - activeLiveCell = { result: cellResult, buf: new TailBuffer(DEFAULT_MAX_BYTES * 2) }; - pushUpdate(); - - const startTime = Date.now(); - let result: ExecutorBackendResult; - try { - result = await backend.execute(cell.code, { - cwd: session.cwd, - sessionId, - sessionFile: sessionFile ?? undefined, - kernelOwnerId, - signal: combinedSignal, - session, - idleTimeoutMs, - reset: cell.reset, - onChunk: chunk => { - outputSink!.push(chunk); - }, - onStatus: event => { - if (event.op === EVAL_TIMEOUT_PAUSE_OP) { - idle?.pause(); - return; - } - if (event.op === EVAL_TIMEOUT_RESUME_OP) { - idle?.resume(); - return; - } - cellResult.statusEvents ??= []; - upsertStatusEvent(cellResult.statusEvents, event); - pushUpdate(); - }, - }); - } finally { - idle?.dispose(); - activeLiveCell = undefined; - } - const durationMs = Date.now() - startTime; - - const cellStatusEvents: EvalStatusEvent[] = []; - const cellDisplayOutputs: EvalDisplayOutput[] = []; - const cellImageNotes: string[] = []; - let cellHasMarkdown = false; - for (const output of result.displayOutputs) { - if (output.type === "json") { - jsonOutputs.push(output.data); - cellDisplayOutputs.push(output); - } - if (output.type === "image") { - const resized = await resizeImage( - { - type: "image", - data: output.data, - mimeType: output.mimeType, - }, - { excludeWebP }, - ); - const image: ImageContent = { - type: "image", - data: resized.data, - mimeType: resized.mimeType, - }; - images.push(image); - cellDisplayOutputs.push({ - type: "image", - data: image.data, - mimeType: image.mimeType, - }); - const dimensionNote = formatDimensionNote(resized); - if (dimensionNote) { - cellImageNotes.push(`display image ${cellImageNotes.length + 1}: ${dimensionNote}`); - } - } - if (output.type === "status") { - upsertStatusEvent(statusEvents, output.event); - upsertStatusEvent(cellStatusEvents, output.event); - } - if (output.type === "markdown") { - cellHasMarkdown = true; - } - } - - const stdoutTrimmed = result.output.trim(); - const imageText = cellImageNotes.join("\n"); - const displayText = formatDisplayOutputsForText(cellDisplayOutputs); - const visibleDisplayText = - displayText && imageText ? `${displayText}\n\n${imageText}` : displayText || imageText; - const cellOutput = - stdoutTrimmed && visibleDisplayText - ? `${stdoutTrimmed}\n\n${visibleDisplayText}` - : stdoutTrimmed || visibleDisplayText; - cellResult.output = cellOutput; - cellResult.exitCode = result.exitCode; - cellResult.durationMs = durationMs; - cellResult.statusEvents = cellStatusEvents.length > 0 ? cellStatusEvents : undefined; - cellResult.hasMarkdown = cellHasMarkdown || undefined; - - if (cellOutput) { - cellOutputs.push(cellOutput); - appendTail(cellOutput); - } - - if (result.cancelled) { - cellResult.status = "error"; - pushUpdate(); - const errorMsg = result.output || "Command aborted"; - const combinedOutput = cellOutputs.join("\n\n"); - const outputText = combinedOutput || errorMsg; - - const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); - const details: EvalToolDetails = { - language: languages[0], - languages, - cells: cellResults, - jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, - statusEvents: statusEvents.length > 0 ? statusEvents : undefined, - isError: true, - }; - if (notice) details.notice = notice; - - return toolResult(details) - .content([{ type: "text", text: outputText }, ...images]) - .truncationFromSummary(summaryForMeta, { direction: "tail" }) - .done(); - } - - if (result.exitCode !== 0 && result.exitCode !== undefined) { - cellResult.status = "error"; - pushUpdate(); - const combinedOutput = cellOutputs.join("\n\n"); - const outputText = combinedOutput - ? `${combinedOutput}\n\nCommand exited with code ${result.exitCode}` - : `Command exited with code ${result.exitCode}`; - - const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); - const details: EvalToolDetails = { - language: languages[0], - languages, - cells: cellResults, - jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, - statusEvents: statusEvents.length > 0 ? statusEvents : undefined, - isError: true, - }; - if (notice) details.notice = notice; - - return toolResult(details) - .content([{ type: "text", text: outputText }, ...images]) - .truncationFromSummary(summaryForMeta, { direction: "tail" }) - .done(); - } - - cellResult.status = "complete"; - pushUpdate(); - } - - const combinedOutput = cellOutputs.join("\n\n"); - const hasImages = images.length > 0; - const outputText = - combinedOutput || - (hasImages - ? `(displayed ${images.length} image${images.length === 1 ? "" : "s"}; no text output)` - : "(no output)"); - const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); + const appendTail = (text: string) => { + tailBuffer.append(text); + }; + const buildUpdateDetails = (): EvalToolDetails => { const details: EvalToolDetails = { language: languages[0], languages, - cells: cellResults, - jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, - statusEvents: statusEvents.length > 0 ? statusEvents : undefined, + cells: cellResults.map(cell => ({ + ...cell, + statusEvents: cell.statusEvents ? [...cell.statusEvents] : undefined, + })), }; - if (notice) details.notice = notice; - - return toolResult(details) - .content([{ type: "text", text: outputText }, ...images]) - .truncationFromSummary(summaryForMeta, { direction: "tail" }) - .done(); - } finally { - if (!outputDumped) { - try { - await finalizeOutput(); - } catch {} + if (jsonOutputs.length > 0) { + details.jsonOutputs = jsonOutputs; } - } - })(); + if (images.length > 0) { + details.images = images; + } + if (statusEvents.length > 0) { + details.statusEvents = statusEvents; + } + if (notice) { + details.notice = notice; + } + return details; + }; - return await (session.trackEvalExecution?.(execution, sessionAbortController) ?? execution); + const pushUpdate = () => { + emitUpdate?.(tailBuffer.text(), buildUpdateDetails()); + }; + + const sessionFile = session.getSessionFile?.() ?? undefined; + const kernelOwnerId = session.getEvalKernelOwnerId?.() ?? undefined; + const { path: artifactPath, id: artifactId } = (await session.allocateOutputArtifact?.("eval")) ?? {}; + session.assertEvalExecutionAllowed?.(); + outputSink = new OutputSink({ + artifactPath, + artifactId, + headBytes: resolveOutputSinkHeadBytes(session.settings), + maxColumns: resolveOutputMaxColumns(session.settings), + onChunk: chunk => { + appendTail(chunk); + if (activeLiveCell) { + activeLiveCell.buf.append(chunk); + activeLiveCell.result.output = activeLiveCell.buf.text(); + } + pushUpdate(); + }, + }); + const sessionId = session.getEvalSessionId?.() ?? defaultEvalSessionId(session); + + for (let i = 0; i < cells.length; i++) { + const cell = cells[i]; + const backend = cell.resolved.backend; + // The per-cell `timeout` is a budget on the cell runtime's *own* + // work. Host-side `agent()`/`parallel()`/`completion()` bridge calls suspend + // that budget entirely and restart a fresh timeout window when control + // returns to the active backend runtime. Compute, stdout, `log()`/`phase()`, and + // ordinary tool calls all count against the budget. The watchdog drives + // `combinedSignal`; we pass no wall-clock deadline downstream so the + // backends never arm a competing fixed timer. + const idleTimeoutMs = + cell.timeoutMs === 0 + ? undefined + : clampTimeout("eval", cell.timeoutMs / 1000, session.settings.get("tools.maxTimeout")) * 1000; + const idle = idleTimeoutMs === undefined ? undefined : new IdleTimeout(idleTimeoutMs); + const combinedSignal = + signal && idle + ? AbortSignal.any([signal, idle.signal, sessionAbortController.signal]) + : signal + ? AbortSignal.any([signal, sessionAbortController.signal]) + : idle + ? AbortSignal.any([idle.signal, sessionAbortController.signal]) + : sessionAbortController.signal; + + const cellResult = cellResults[i]; + cellResult.status = "running"; + cellResult.output = ""; + cellResult.statusEvents = undefined; + cellResult.exitCode = undefined; + cellResult.durationMs = undefined; + activeLiveCell = { result: cellResult, buf: new TailBuffer(DEFAULT_MAX_BYTES * 2) }; + pushUpdate(); + + const startTime = Date.now(); + let result: ExecutorBackendResult; + try { + result = await backend.execute(cell.code, { + cwd: session.cwd, + sessionId, + sessionFile: sessionFile ?? undefined, + kernelOwnerId, + signal: combinedSignal, + session, + idleTimeoutMs, + reset: cell.reset, + onChunk: chunk => { + outputSink!.push(chunk); + }, + onStatus: event => { + if (event.op === EVAL_TIMEOUT_PAUSE_OP) { + idle?.pause(); + return; + } + if (event.op === EVAL_TIMEOUT_RESUME_OP) { + idle?.resume(); + return; + } + cellResult.statusEvents ??= []; + upsertStatusEvent(cellResult.statusEvents, event); + pushUpdate(); + }, + }); + } finally { + idle?.dispose(); + activeLiveCell = undefined; + } + const durationMs = Date.now() - startTime; + + const cellStatusEvents: EvalStatusEvent[] = []; + const cellDisplayOutputs: EvalDisplayOutput[] = []; + const cellImageNotes: string[] = []; + let cellHasMarkdown = false; + for (const output of result.displayOutputs) { + if (output.type === "json") { + jsonOutputs.push(output.data); + cellDisplayOutputs.push(output); + } + if (output.type === "image") { + const resized = await resizeImage( + { + type: "image", + data: output.data, + mimeType: output.mimeType, + }, + { excludeWebP }, + ); + const image: ImageContent = { + type: "image", + data: resized.data, + mimeType: resized.mimeType, + }; + images.push(image); + cellDisplayOutputs.push({ + type: "image", + data: image.data, + mimeType: image.mimeType, + }); + const dimensionNote = formatDimensionNote(resized); + if (dimensionNote) { + cellImageNotes.push(`display image ${cellImageNotes.length + 1}: ${dimensionNote}`); + } + } + if (output.type === "status") { + upsertStatusEvent(statusEvents, output.event); + upsertStatusEvent(cellStatusEvents, output.event); + } + if (output.type === "markdown") { + cellHasMarkdown = true; + } + } + + const stdoutTrimmed = result.output.trim(); + const imageText = cellImageNotes.join("\n"); + const displayText = formatDisplayOutputsForText(cellDisplayOutputs); + const visibleDisplayText = + displayText && imageText ? `${displayText}\n\n${imageText}` : displayText || imageText; + const cellOutput = + stdoutTrimmed && visibleDisplayText + ? `${stdoutTrimmed}\n\n${visibleDisplayText}` + : stdoutTrimmed || visibleDisplayText; + cellResult.output = cellOutput; + cellResult.exitCode = result.exitCode; + cellResult.durationMs = durationMs; + cellResult.statusEvents = cellStatusEvents.length > 0 ? cellStatusEvents : undefined; + cellResult.hasMarkdown = cellHasMarkdown || undefined; + + if (cellOutput) { + cellOutputs.push(cellOutput); + appendTail(cellOutput); + } + + if (result.cancelled) { + cellResult.status = "error"; + pushUpdate(); + const errorMsg = result.output || "Command aborted"; + const combinedOutput = cellOutputs.join("\n\n"); + const outputText = combinedOutput || errorMsg; + + const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); + const details: EvalToolDetails = { + language: languages[0], + languages, + cells: cellResults, + jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, + statusEvents: statusEvents.length > 0 ? statusEvents : undefined, + isError: true, + }; + if (notice) details.notice = notice; + + return toolResult(details) + .content([{ type: "text", text: outputText }, ...images]) + .truncationFromSummary(summaryForMeta, { direction: "tail" }) + .done(); + } + + if (result.exitCode !== 0 && result.exitCode !== undefined) { + cellResult.status = "error"; + pushUpdate(); + const combinedOutput = cellOutputs.join("\n\n"); + const outputText = combinedOutput + ? `${combinedOutput}\n\nCommand exited with code ${result.exitCode}` + : `Command exited with code ${result.exitCode}`; + + const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); + const details: EvalToolDetails = { + language: languages[0], + languages, + cells: cellResults, + jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, + statusEvents: statusEvents.length > 0 ? statusEvents : undefined, + isError: true, + }; + if (notice) details.notice = notice; + + return toolResult(details) + .content([{ type: "text", text: outputText }, ...images]) + .truncationFromSummary(summaryForMeta, { direction: "tail" }) + .done(); + } + + cellResult.status = "complete"; + pushUpdate(); + } + + const combinedOutput = cellOutputs.join("\n\n"); + const hasImages = images.length > 0; + const outputText = + combinedOutput || + (hasImages + ? `(displayed ${images.length} image${images.length === 1 ? "" : "s"}; no text output)` + : "(no output)"); + const summaryForMeta = await summarizeFinal(combinedOutput, finalizeOutput); + + const details: EvalToolDetails = { + language: languages[0], + languages, + cells: cellResults, + jsonOutputs: jsonOutputs.length > 0 ? jsonOutputs : undefined, + statusEvents: statusEvents.length > 0 ? statusEvents : undefined, + }; + if (notice) details.notice = notice; + + return toolResult(details) + .content([{ type: "text", text: outputText }, ...images]) + .truncationFromSummary(summaryForMeta, { direction: "tail" }) + .done(); + } finally { + if (!outputDumped) { + try { + await finalizeOutput(); + } catch {} + } + } } } diff --git a/packages/coding-agent/src/tools/hub/jobs.ts b/packages/coding-agent/src/tools/hub/jobs.ts index d7402f6bd..b1c2f03cf 100644 --- a/packages/coding-agent/src/tools/hub/jobs.ts +++ b/packages/coding-agent/src/tools/hub/jobs.ts @@ -7,7 +7,7 @@ import type { AgentToolResult } from "@oh-my-pi/pi-agent-core"; import type { Component } from "@oh-my-pi/pi-tui"; import { Text } from "@oh-my-pi/pi-tui"; -import type { AsyncJob, AsyncJobManager } from "../../async"; +import type { AsyncJob, AsyncJobManager, AsyncJobType } from "../../async"; import { settings } from "../../config/settings"; import type { RenderResultOptions } from "../../extensibility/custom-tools/types"; import { shimmerEnabled, shimmerText } from "../../modes/theme/shimmer"; @@ -145,7 +145,7 @@ function describeAgents(agents: AgentActivitySnapshot[]): string[] { interface TrackedJobLike { id: string; - type: "bash" | "task"; + type: AsyncJobType; status: string; label: string; startTime: number; diff --git a/packages/coding-agent/src/tools/hub/types.ts b/packages/coding-agent/src/tools/hub/types.ts index 5685e3563..74878145f 100644 --- a/packages/coding-agent/src/tools/hub/types.ts +++ b/packages/coding-agent/src/tools/hub/types.ts @@ -5,6 +5,7 @@ */ import type { AgentToolResult } from "@oh-my-pi/pi-agent-core"; +import type { AsyncJobType } from "../../async"; import type { IrcDeliveryReceipt, IrcMessage } from "../../irc/bus"; import type { LaunchParams, LaunchToolDetails } from "./launch"; @@ -42,7 +43,7 @@ export interface HubPeerInfo { /** Background-job row surfaced by `wait`/`cancel`/`jobs` results. */ export interface JobSnapshot { id: string; - type: "bash" | "task"; + type: AsyncJobType; status: "running" | "completed" | "failed" | "cancelled"; label: string; durationMs: number; diff --git a/packages/coding-agent/test/async-yield-queue.test.ts b/packages/coding-agent/test/async-yield-queue.test.ts index 3a404dbe2..5c372ea21 100644 --- a/packages/coding-agent/test/async-yield-queue.test.ts +++ b/packages/coding-agent/test/async-yield-queue.test.ts @@ -5,7 +5,7 @@ import { ASIDE_MESSAGE_DISCARD, type CommittableAsideMessage, } from "@oh-my-pi/pi-agent-core"; -import { type AsyncJob, AsyncJobManager } from "@oh-my-pi/pi-coding-agent/async"; +import { type AsyncJob, AsyncJobManager, type AsyncJobType } from "@oh-my-pi/pi-coding-agent/async"; import type { CustomMessage } from "@oh-my-pi/pi-coding-agent/session/messages"; import { YieldQueue } from "@oh-my-pi/pi-coding-agent/session/yield-queue"; import type { ToolSession } from "@oh-my-pi/pi-coding-agent/tools"; @@ -21,7 +21,7 @@ type AsyncEntry = { type AsyncDetails = { jobs: Array<{ jobId: string; - type?: "bash" | "task"; + type?: AsyncJobType; label?: string; durationMs?: number; }>; diff --git a/packages/coding-agent/test/tools/eval-auto-background.test.ts b/packages/coding-agent/test/tools/eval-auto-background.test.ts new file mode 100644 index 000000000..ed53c95a8 --- /dev/null +++ b/packages/coding-agent/test/tools/eval-auto-background.test.ts @@ -0,0 +1,215 @@ +import { afterEach, describe, expect, it, vi } from "bun:test"; +import type { AgentToolContext } from "@oh-my-pi/pi-agent-core"; +import { AsyncJobManager } from "@oh-my-pi/pi-coding-agent/async"; +import { Settings } from "@oh-my-pi/pi-coding-agent/config/settings"; +import * as evalIndex from "@oh-my-pi/pi-coding-agent/eval"; +import { SessionManager } from "@oh-my-pi/pi-coding-agent/session/session-manager"; +import type { ToolSession } from "@oh-my-pi/pi-coding-agent/tools"; +import { EvalTool } from "@oh-my-pi/pi-coding-agent/tools/eval"; + +function makeSession(settings: Settings, asyncJobManager: AsyncJobManager): ToolSession { + return { + cwd: "/tmp/eval-test", + hasUI: false, + getSessionFile: () => null, + getSessionSpawns: () => null, + settings, + asyncJobManager, + }; +} + +function baseResult(overrides: Record = {}) { + return { + output: "", + exitCode: 0, + cancelled: false, + truncated: false, + artifactId: undefined, + totalLines: 0, + totalBytes: 0, + outputLines: 0, + outputBytes: 0, + displayOutputs: [] as unknown[], + ...overrides, + }; +} + +/** + * Mock the JS backend with a cell that streams one chunk immediately and then + * blocks until the returned `release()` gate opens — so backgrounding is decided + * by the tool's own threshold/steer race, never by a guessed sleep. + */ +function mockGatedCell(finalOutput: string): { release: () => void } { + const gate = Promise.withResolvers(); + vi.spyOn(evalIndex.jsBackend, "execute").mockImplementation((async ( + _code: string, + options: { onChunk?: (chunk: string) => void }, + ) => { + options.onChunk?.("start\n"); + await gate.promise; + return baseResult({ output: finalOutput }); + }) as never); + return { release: gate.resolve }; +} + +function steeringContext(steeringSignal: AbortSignal): AgentToolContext { + return { + sessionManager: SessionManager.inMemory(), + modelRegistry: { + find: () => undefined, + getAll: () => [], + getApiKey: async () => undefined, + } as unknown as AgentToolContext["modelRegistry"], + model: undefined, + isIdle: () => true, + hasQueuedMessages: () => false, + abort: () => {}, + toolNames: [], + toolCall: { + batchId: "batch-1", + index: 0, + total: 1, + toolCalls: [{ id: "call-steer", name: "eval" }], + steeringSignal, + }, + } as AgentToolContext; +} + +/** + * Defends the eval auto-background contract (mirror of bash's): a cell that + * finishes before the threshold resolves inline with no job leftovers, a cell + * that outlives the threshold converts into a running async job whose result is + * delivered later, and a steering interrupt backgrounds the cell immediately so + * the queued message can inject while the kernel keeps working. + */ +describe("EvalTool auto-background", () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("keeps fast cells inline and suppresses their job delivery", async () => { + const deliveries: string[] = []; + const asyncJobManager = new AsyncJobManager({ + onJobComplete: async (_jobId, text) => { + deliveries.push(text); + }, + }); + vi.spyOn(evalIndex.jsBackend, "execute").mockImplementation((async () => + baseResult({ output: "quick\n" })) as never); + + const tool = new EvalTool( + makeSession( + Settings.isolated({ + "eval.autoBackground.enabled": true, + "eval.autoBackground.thresholdMs": 2_000, + }), + asyncJobManager, + ), + ); + const result = await tool.execute("call-inline", { language: "js", code: "print('quick')" }); + + const text = result.content.map(c => (c.type === "text" ? c.text : "")).join("\n"); + expect(text).toContain("quick"); + expect(result.details?.async).toBeUndefined(); + expect(result.details?.cells?.[0]?.status).toBe("complete"); + await asyncJobManager.drainDeliveries({ timeoutMs: 1 }); + expect(deliveries).toEqual([]); + await asyncJobManager.dispose(); + }); + + it("backgrounds a cell that outlives the threshold and delivers its result", async () => { + const deliveries: Array<{ jobId: string; text: string }> = []; + const updates: string[] = []; + const asyncJobManager = new AsyncJobManager({ + onJobComplete: async (jobId, text) => { + deliveries.push({ jobId, text }); + }, + }); + const cell = mockGatedCell("start\ndone\n"); + + const tool = new EvalTool( + makeSession( + Settings.isolated({ + "eval.autoBackground.enabled": true, + "eval.autoBackground.thresholdMs": 10, + }), + asyncJobManager, + ), + ); + // The gated cell cannot finish on its own, so execute() returning proves + // the threshold path backgrounded it. + const result = await tool.execute( + "call-background", + { language: "js", code: "print('start'); await work(); print('done')" }, + undefined, + update => { + updates.push(update.content?.find(block => block.type === "text")?.text ?? ""); + }, + ); + + expect(result.details?.async?.state).toBe("running"); + expect(result.details?.async?.type).toBe("eval"); + const text = result.content.map(c => (c.type === "text" ? c.text : "")).join("\n"); + expect(text).toContain("Backgrounded as job"); + // The snapshot keeps the running cell (with its streamed tail) for the transcript. + expect(result.details?.cells?.[0]?.status).toBe("running"); + + const jobId = result.details?.async?.jobId; + if (!jobId) { + throw new Error("expected an auto-backgrounded job id"); + } + const runningJob = asyncJobManager.getJob(jobId); + expect(runningJob?.status).toBe("running"); + const updatesAtBackground = updates.slice(); + cell.release(); + await runningJob?.promise; + await asyncJobManager.drainDeliveries({ timeoutMs: 1 }); + expect(deliveries).toHaveLength(1); + expect(deliveries[0]?.jobId).toBe(jobId); + expect(deliveries[0]?.text).toContain("done"); + // Tool-call updates stop once the cell is backgrounded. + expect(updates).toEqual(updatesAtBackground); + await asyncJobManager.dispose(); + }); + + it("backgrounds a running cell when the steering signal fires mid-wait", async () => { + const asyncJobManager = new AsyncJobManager({}); + const cell = mockGatedCell("steered\n"); + + const tool = new EvalTool( + makeSession( + Settings.isolated({ + "eval.autoBackground.enabled": true, + // High threshold: only the steering signal can background this. + "eval.autoBackground.thresholdMs": 60_000, + }), + asyncJobManager, + ), + ); + const steering = new AbortController(); + steering.abort(); + const result = await tool.execute( + "call-steer", + { language: "js", code: "await work()" }, + undefined, + undefined, + steeringContext(steering.signal), + ); + + // The steer backgrounds the cell instead of killing it: the call returns a + // running job and the cell finishes on its own. + expect(result.details?.async?.state).toBe("running"); + const text = result.content.map(c => (c.type === "text" ? c.text : "")).join("\n"); + expect(text).toContain("Backgrounded early to handle an incoming message"); + const jobId = result.details?.async?.jobId; + if (!jobId) { + throw new Error("expected a steer-backgrounded job id"); + } + const job = asyncJobManager.getJob(jobId); + expect(job?.status).toBe("running"); + cell.release(); + await job?.promise; + expect(asyncJobManager.getJob(jobId)?.status).toBe("completed"); + await asyncJobManager.dispose(); + }); +});