refactor(eval): changed timeout from inactivity to wall-clock budget
- Only bridge heartbeats (`agent()`/`llm()`) now re-arm the watchdog; compute, stdout, `log()`/`phase()`, and ordinary tool calls count against the budget. - Emitted an immediate heartbeat at bridge call start to avoid early abort near budget edge. - Removed `idle` flag and "of inactivity" suffix from timeout annotation strings. - Updated docs, prompts, and comments to reflect the new wall-clock semantics.
This commit is contained in:
+3
-3
@@ -166,9 +166,9 @@ Python prelude helpers include `agent(prompt, *, agent_type="task", model=None,
|
|||||||
|
|
||||||
### Cell timeout
|
### Cell timeout
|
||||||
|
|
||||||
Each eval cell `timeout` is in seconds, defaults to 30, and is clamped to `1..600`. It is an **inactivity (idle) budget, not a hard wall-clock cap**: the watchdog (`IdleTimeout`, `src/eval/idle-timeout.ts`) only fires once the cell goes the full window with **no progress signal**. Every status event re-arms it — `agent()` progress snapshots, `log()`/`phase()`, and tool-bridge activity all count — so a long-running fanout that keeps reporting progress runs to completion instead of being killed mid-stream.
|
Each eval cell `timeout` is in seconds, defaults to 30, and is clamped to `1..600`. It is a **wall-clock budget on the cell's own work** that the watchdog (`IdleTimeout`, `src/eval/idle-timeout.ts`) enforces, **but it is paused while a host-side `agent()`/`parallel()`/`llm()` bridge call is in flight**: those calls pump a heartbeat (`withBridgeHeartbeat`, `src/eval/heartbeat.ts`) that re-arms the watchdog, so a long fanout or a slow completion runs to completion instead of being killed mid-stream.
|
||||||
|
|
||||||
Raw `stdout`/`stderr` does **not** re-arm the watchdog, so a pure-compute runaway loop with no progress reporting is still bounded by `timeout`. The tool combines the caller abort signal, the session abort signal, and the idle watchdog's signal with `AbortSignal.any(...)`; no wall-clock deadline is passed to the backend, so neither runtime arms a competing fixed timer.
|
The heartbeat is the **sole** signal that extends the budget. Everything else the cell does — compute, `stdout`/`stderr`, `log()`/`phase()`, and ordinary (non-agent) tool calls — counts against `timeout`, so a cell that is not delegating to an agent/llm is bounded by a plain wall-clock timeout. The tool combines the caller abort signal, the session abort signal, and the watchdog's signal with `AbortSignal.any(...)`; no wall-clock deadline is passed to the backend, so neither runtime arms a competing fixed timer.
|
||||||
|
|
||||||
### Kernel execution cancellation
|
### Kernel execution cancellation
|
||||||
|
|
||||||
@@ -176,7 +176,7 @@ On abort/timeout:
|
|||||||
|
|
||||||
- The host sends `kill("SIGINT")` to the runner subprocess.
|
- The host sends `kill("SIGINT")` to the runner subprocess.
|
||||||
- The runner's exec-time signal handler raises `KeyboardInterrupt` inside the user code.
|
- The runner's exec-time signal handler raises `KeyboardInterrupt` inside the user code.
|
||||||
- Result includes `cancelled=true`; the timeout path annotates output as `Command timed out after <n> seconds of inactivity`.
|
- Result includes `cancelled=true`; the timeout path annotates output as `Command timed out after <n> seconds`.
|
||||||
- Between requests the runner installs `SIG_IGN` for SIGINT so a stray cancel does not tear down the kernel.
|
- Between requests the runner installs `SIG_IGN` for SIGINT so a stray cancel does not tear down the kernel.
|
||||||
|
|
||||||
If a second cancel is required (runner stuck in C code), the host escalates to `SIGTERM` and the session restarts on the next call.
|
If a second cancel is required (runner stuck in C code), the host escalates to `SIGTERM` and the session restarts on the next call.
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
- Fixed the `eval` tool aborting in-flight `agent()`/`parallel()` subagents and `llm()` requests by mistaking them for a stalled cell. The per-cell `timeout` is an *inactivity* budget that only re-arms on status events, but a host-side bridge call can legitimately run long stretches with no intermediate status (a subagent's time-to-first-token on a reasoning model, a long quiet nested tool, or an entire oneshot `llm()` request). Those calls now pump a lightweight heartbeat while they await, re-arming the idle watchdog through the existing status channel; the heartbeat is a pure keepalive and is never persisted or rendered, so a genuinely stalled cell is still interrupted once the call settles.
|
- Fixed the `eval` tool's per-cell `timeout` killing cells that were not stalled. The timeout is now a plain wall-clock budget on the cell's **own** work that is **paused only while a host-side `agent()`/`parallel()`/`llm()` bridge call is in flight** — those calls pump a heartbeat that re-arms the watchdog, so a long fanout or a slow (e.g. reasoning-tier) completion runs to completion instead of being aborted mid-flight (a subagent's time-to-first-token, a long quiet nested tool, or an entire oneshot `llm()` request no longer trip it). Nothing else re-arms the budget: ordinary compute, `print`/stdout, `log()`/`phase()`, and non-agent tool calls all count against it, so a cell that is not delegating to an agent/llm is bounded by the regular wall-clock timeout (and the timeout message no longer says "of inactivity"). The heartbeat is a pure keepalive — never persisted or rendered.
|
||||||
|
|
||||||
## [15.7.5] - 2026-06-01
|
## [15.7.5] - 2026-06-01
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import { AgentOutputManager } from "../../task/output-manager";
|
|||||||
import type { AgentDefinition, AgentProgress, SingleResult } from "../../task/types";
|
import type { AgentDefinition, AgentProgress, SingleResult } from "../../task/types";
|
||||||
import type { ToolSession } from "../../tools";
|
import type { ToolSession } from "../../tools";
|
||||||
import { EVAL_AGENT_MAX_DEPTH, runEvalAgent } from "../agent-bridge";
|
import { EVAL_AGENT_MAX_DEPTH, runEvalAgent } from "../agent-bridge";
|
||||||
import { setBridgeHeartbeatIntervalMs } from "../heartbeat";
|
import { EVAL_HEARTBEAT_OP, setBridgeHeartbeatIntervalMs } from "../heartbeat";
|
||||||
import { IdleTimeout } from "../idle-timeout";
|
import { IdleTimeout } from "../idle-timeout";
|
||||||
import { disposeAllVmContexts } from "../js/context-manager";
|
import { disposeAllVmContexts } from "../js/context-manager";
|
||||||
import { executeJs } from "../js/executor";
|
import { executeJs } from "../js/executor";
|
||||||
@@ -451,14 +451,73 @@ describe("agent() through eval runtimes", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// Mirror the eval tool's wiring: an IdleTimeout drives cancellation and
|
// Mirror the eval tool's wiring: an IdleTimeout drives cancellation and
|
||||||
// every status event re-arms it.
|
// ONLY a bridge heartbeat re-arms it.
|
||||||
using idle = new IdleTimeout(60);
|
using idle = new IdleTimeout(60);
|
||||||
const result = await runEvalAgent(
|
const result = await runEvalAgent(
|
||||||
{ prompt: "investigate" },
|
{ prompt: "investigate" },
|
||||||
{ session, signal: idle.signal, emitStatus: () => idle.bump() },
|
{
|
||||||
|
session,
|
||||||
|
signal: idle.signal,
|
||||||
|
emitStatus: event => {
|
||||||
|
if (event.op === EVAL_HEARTBEAT_OP) idle.bump();
|
||||||
|
},
|
||||||
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
expect(idle.signal.aborted).toBe(false);
|
expect(idle.signal.aborted).toBe(false);
|
||||||
expect(result.text).toBe("done");
|
expect(result.text).toBe("done");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("does not let agent() progress snapshots re-arm the watchdog without a heartbeat", async () => {
|
||||||
|
using tempDir = TempDir.createSync("@omp-eval-agent-progress-no-rearm-");
|
||||||
|
const { session } = makeEvalSession(tempDir, "js-agent-progress-no-rearm");
|
||||||
|
mockAgents();
|
||||||
|
// Heartbeat slower than the budget: only the immediate beat at call start
|
||||||
|
// fires, so after the budget elapses nothing re-arms the watchdog.
|
||||||
|
setBridgeHeartbeatIntervalMs(10_000);
|
||||||
|
|
||||||
|
// Stream frequent progress snapshots (op:"agent") for well past the budget.
|
||||||
|
// Progress is rendered but MUST NOT count as activity — only heartbeats do.
|
||||||
|
vi.spyOn(taskExecutor, "runSubprocess").mockImplementation(async options => {
|
||||||
|
for (let i = 0; i < 40; i++) {
|
||||||
|
options.onProgress?.({
|
||||||
|
index: options.index,
|
||||||
|
id: options.id,
|
||||||
|
agent: options.agent.name,
|
||||||
|
agentSource: options.agent.source,
|
||||||
|
status: "running",
|
||||||
|
task: options.task,
|
||||||
|
assignment: options.assignment,
|
||||||
|
description: options.description,
|
||||||
|
recentTools: [],
|
||||||
|
recentOutput: [],
|
||||||
|
toolCount: i,
|
||||||
|
tokens: 0,
|
||||||
|
cost: 0,
|
||||||
|
durationMs: i * 10,
|
||||||
|
});
|
||||||
|
await Bun.sleep(10);
|
||||||
|
}
|
||||||
|
return singleResult(options, { output: "done" });
|
||||||
|
});
|
||||||
|
|
||||||
|
const ops: string[] = [];
|
||||||
|
using idle = new IdleTimeout(80);
|
||||||
|
await runEvalAgent(
|
||||||
|
{ prompt: "investigate" },
|
||||||
|
{
|
||||||
|
session,
|
||||||
|
signal: idle.signal,
|
||||||
|
emitStatus: event => {
|
||||||
|
ops.push(event.op);
|
||||||
|
if (event.op === EVAL_HEARTBEAT_OP) idle.bump();
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
// Progress streamed, but the watchdog still fired: agent snapshots never
|
||||||
|
// re-armed it, and the lone start heartbeat lapsed before the call ended.
|
||||||
|
expect(ops).toContain("agent");
|
||||||
|
expect(idle.signal.aborted).toBe(true);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -31,6 +31,24 @@ describe("withBridgeHeartbeat", () => {
|
|||||||
expect(events.length).toBe(settledCount);
|
expect(events.length).toBe(settledCount);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("emits a heartbeat immediately so a bridge call extends the budget at once", async () => {
|
||||||
|
// Interval far longer than the operation: the only beat that can fire is
|
||||||
|
// the immediate one at call start. It must still reach the sink.
|
||||||
|
setBridgeHeartbeatIntervalMs(10_000);
|
||||||
|
const events: JsStatusEvent[] = [];
|
||||||
|
|
||||||
|
await withBridgeHeartbeat(
|
||||||
|
event => events.push(event),
|
||||||
|
async () => {
|
||||||
|
await Bun.sleep(30);
|
||||||
|
return "done";
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(events.length).toBe(1);
|
||||||
|
expect(events[0]?.op).toBe(EVAL_HEARTBEAT_OP);
|
||||||
|
});
|
||||||
|
|
||||||
it("runs the operation without emitting when no status sink is wired", async () => {
|
it("runs the operation without emitting when no status sink is wired", async () => {
|
||||||
setBridgeHeartbeatIntervalMs(5);
|
setBridgeHeartbeatIntervalMs(5);
|
||||||
let ran = 0;
|
let ran = 0;
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import type { ModelRegistry } from "../../config/model-registry";
|
|||||||
import { Settings } from "../../config/settings";
|
import { Settings } from "../../config/settings";
|
||||||
import type { ToolSession } from "../../tools";
|
import type { ToolSession } from "../../tools";
|
||||||
import { ToolError } from "../../tools/tool-errors";
|
import { ToolError } from "../../tools/tool-errors";
|
||||||
import { setBridgeHeartbeatIntervalMs } from "../heartbeat";
|
import { EVAL_HEARTBEAT_OP, setBridgeHeartbeatIntervalMs } from "../heartbeat";
|
||||||
import { IdleTimeout } from "../idle-timeout";
|
import { IdleTimeout } from "../idle-timeout";
|
||||||
import { disposeAllVmContexts } from "../js/context-manager";
|
import { disposeAllVmContexts } from "../js/context-manager";
|
||||||
import { executeJs } from "../js/executor";
|
import { executeJs } from "../js/executor";
|
||||||
@@ -230,7 +230,14 @@ describe("runEvalLlm", () => {
|
|||||||
using idle = new IdleTimeout(60);
|
using idle = new IdleTimeout(60);
|
||||||
const result = await runEvalLlm(
|
const result = await runEvalLlm(
|
||||||
{ prompt: "q", model: "smol" },
|
{ prompt: "q", model: "smol" },
|
||||||
{ session: makeSession(), signal: idle.signal, emitStatus: () => idle.bump() },
|
{
|
||||||
|
session: makeSession(),
|
||||||
|
signal: idle.signal,
|
||||||
|
// Mirror the eval tool: only a bridge heartbeat re-arms the watchdog.
|
||||||
|
emitStatus: event => {
|
||||||
|
if (event.op === EVAL_HEARTBEAT_OP) idle.bump();
|
||||||
|
},
|
||||||
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
expect(idle.signal.aborted).toBe(false);
|
expect(idle.signal.aborted).toBe(false);
|
||||||
|
|||||||
@@ -1,22 +1,25 @@
|
|||||||
/**
|
/**
|
||||||
* Keepalive for in-flight host-side eval bridge calls.
|
* Keepalive for in-flight host-side eval bridge calls.
|
||||||
*
|
*
|
||||||
* The eval idle watchdog ({@link ../tools/eval IdleTimeout}) treats a cell's
|
* The eval watchdog ({@link ../tools/eval IdleTimeout}) caps a cell's `timeout`
|
||||||
* `timeout` as an *inactivity* budget and only re-arms when a status event
|
* as a wall-clock budget on the cell's *own* work, but pauses that budget while
|
||||||
* reaches it. Host-side bridge helpers — `agent()`/`parallel()` (via
|
* a host-side `agent()`/`parallel()` (via `runSubprocess`) or `llm()` (a single
|
||||||
* `runSubprocess`) and `llm()` (a single completion) — can legitimately run for
|
* completion) call is in flight. Those calls are the only thing that re-arms the
|
||||||
* long stretches with **no** intermediate status: a subagent's time-to-first
|
* watchdog — and they can run for long stretches with **no** status of their own
|
||||||
* token on a reasoning model, a long quiet nested tool, or the entire body of a
|
* (a subagent's time-to-first-token on a reasoning model, a long quiet nested
|
||||||
* oneshot `llm()` call. Without a keepalive the watchdog mistakes that work for
|
* tool, or the entire body of a oneshot `llm()` call). Without a keepalive the
|
||||||
* a stall and aborts the cell mid-flight, killing the subagent.
|
* watchdog would mistake that delegated work for the cell stalling and abort it
|
||||||
|
* mid-flight, killing the subagent.
|
||||||
*
|
*
|
||||||
* {@link withBridgeHeartbeat} fixes that by pumping a synthetic
|
* {@link withBridgeHeartbeat} bridges that gap by emitting a synthetic
|
||||||
* {@link EVAL_HEARTBEAT_OP} status event on a fixed cadence while the wrapped
|
* {@link EVAL_HEARTBEAT_OP} status event immediately when the call begins and
|
||||||
* operation is pending. The event rides the same `emitStatus → onStatus` channel
|
* then on a fixed cadence until it settles. The event rides the same
|
||||||
* both runtimes already forward, so it re-arms the watchdog without any new
|
* `emitStatus → onStatus` channel both runtimes already forward, so it re-arms
|
||||||
* plumbing. Consumers MUST treat the heartbeat as a pure keepalive: bump the
|
* the watchdog without any new plumbing. The heartbeat is the *sole* signal that
|
||||||
|
* extends the budget: consumers MUST treat it as a pure keepalive — bump the
|
||||||
* watchdog and drop it (never persist or render it) — see the executor display
|
* watchdog and drop it (never persist or render it) — see the executor display
|
||||||
* sinks and the eval tool's `onStatus` handler.
|
* sinks and the eval tool's `onStatus` handler. Every other status event
|
||||||
|
* (compute helpers, `log()`/`phase()`, tool results) counts against the budget.
|
||||||
*/
|
*/
|
||||||
import type { JsStatusEvent } from "./js/shared/types";
|
import type { JsStatusEvent } from "./js/shared/types";
|
||||||
|
|
||||||
@@ -47,14 +50,19 @@ export function setBridgeHeartbeatIntervalMs(ms?: number): void {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Run {@link operation}, pumping {@link EVAL_HEARTBEAT_OP} status events through
|
* Run {@link operation}, pumping {@link EVAL_HEARTBEAT_OP} status events through
|
||||||
* {@link emitStatus} on a fixed cadence until it settles. A no-op wrapper when
|
* {@link emitStatus} — one immediately, then on a fixed cadence — until it
|
||||||
* no `emitStatus` sink is wired (the heartbeat would reach nobody).
|
* settles. The immediate beat pauses the watchdog the instant the call begins,
|
||||||
|
* so a bridge call that starts close to the budget edge (after the cell already
|
||||||
|
* spent most of it computing) is not aborted before the first interval tick. A
|
||||||
|
* no-op wrapper when no `emitStatus` sink is wired (the heartbeat would reach
|
||||||
|
* nobody).
|
||||||
*/
|
*/
|
||||||
export async function withBridgeHeartbeat<T>(
|
export async function withBridgeHeartbeat<T>(
|
||||||
emitStatus: ((event: JsStatusEvent) => void) | undefined,
|
emitStatus: ((event: JsStatusEvent) => void) | undefined,
|
||||||
operation: () => Promise<T>,
|
operation: () => Promise<T>,
|
||||||
): Promise<T> {
|
): Promise<T> {
|
||||||
if (!emitStatus) return operation();
|
if (!emitStatus) return operation();
|
||||||
|
emitStatus({ op: EVAL_HEARTBEAT_OP });
|
||||||
const timer = setInterval(() => emitStatus({ op: EVAL_HEARTBEAT_OP }), heartbeatIntervalMs);
|
const timer = setInterval(() => emitStatus({ op: EVAL_HEARTBEAT_OP }), heartbeatIntervalMs);
|
||||||
// Never keep the event loop alive for the heartbeat alone.
|
// Never keep the event loop alive for the heartbeat alone.
|
||||||
timer.unref?.();
|
timer.unref?.();
|
||||||
|
|||||||
@@ -60,11 +60,10 @@ function isTimeoutReason(reason: unknown): boolean {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
function formatJsTimeoutAnnotation(timeoutMs: number | undefined, idle: boolean): string {
|
function formatJsTimeoutAnnotation(timeoutMs: number | undefined): string {
|
||||||
const suffix = idle ? " of inactivity" : "";
|
|
||||||
if (timeoutMs === undefined) return "Command timed out";
|
if (timeoutMs === undefined) return "Command timed out";
|
||||||
const secs = Math.max(1, Math.round(timeoutMs / 1000));
|
const secs = Math.max(1, Math.round(timeoutMs / 1000));
|
||||||
return `Command timed out after ${secs} seconds${suffix}`;
|
return `Command timed out after ${secs} seconds`;
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function executeJs(code: string, options: JsExecutorOptions): Promise<JsResult> {
|
export async function executeJs(code: string, options: JsExecutorOptions): Promise<JsResult> {
|
||||||
@@ -86,10 +85,9 @@ export async function executeJs(code: string, options: JsExecutorOptions): Promi
|
|||||||
options.signal && timeoutSignal
|
options.signal && timeoutSignal
|
||||||
? AbortSignal.any([options.signal, timeoutSignal])
|
? AbortSignal.any([options.signal, timeoutSignal])
|
||||||
: (options.signal ?? timeoutSignal);
|
: (options.signal ?? timeoutSignal);
|
||||||
// Idle mode: the eval tool drives cancellation via an idle-aware `signal` and
|
// The eval tool drives cancellation via an idle-aware `signal` and passes only
|
||||||
// passes only an inactivity budget. Use it for worker cold-start headroom and
|
// an inactivity budget; use it solely as worker cold-start headroom and never
|
||||||
// timeout-annotation text; never derive a competing fixed timer from it.
|
// derive a competing fixed timer from it.
|
||||||
const idleMode = legacyTimeoutMs === undefined && options.idleTimeoutMs !== undefined;
|
|
||||||
const acquireBudgetMs = legacyTimeoutMs ?? options.idleTimeoutMs;
|
const acquireBudgetMs = legacyTimeoutMs ?? options.idleTimeoutMs;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
@@ -133,7 +131,7 @@ export async function executeJs(code: string, options: JsExecutorOptions): Promi
|
|||||||
if (signal?.aborted || isAbortError(error)) {
|
if (signal?.aborted || isAbortError(error)) {
|
||||||
const timedOut = Boolean(timeoutSignal?.aborted) || isTimeoutReason(options.signal?.reason);
|
const timedOut = Boolean(timeoutSignal?.aborted) || isTimeoutReason(options.signal?.reason);
|
||||||
if (timedOut) {
|
if (timedOut) {
|
||||||
outputSink.push(formatJsTimeoutAnnotation(legacyTimeoutMs ?? options.idleTimeoutMs, idleMode));
|
outputSink.push(formatJsTimeoutAnnotation(legacyTimeoutMs ?? options.idleTimeoutMs));
|
||||||
}
|
}
|
||||||
const summary = await outputSink.dump();
|
const summary = await outputSink.dump();
|
||||||
return {
|
return {
|
||||||
|
|||||||
@@ -232,20 +232,18 @@ async function waitForPromiseWithCancellation<T>(
|
|||||||
// Result formatting
|
// Result formatting
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
function formatTimeoutAnnotation(timeoutMs?: number, idle = false): string | undefined {
|
function formatTimeoutAnnotation(timeoutMs?: number): string | undefined {
|
||||||
const suffix = idle ? " of inactivity" : "";
|
|
||||||
if (timeoutMs === undefined) return "Command timed out";
|
if (timeoutMs === undefined) return "Command timed out";
|
||||||
const secs = Math.max(1, Math.round(timeoutMs / 1000));
|
const secs = Math.max(1, Math.round(timeoutMs / 1000));
|
||||||
return `Command timed out after ${secs} seconds${suffix}`;
|
return `Command timed out after ${secs} seconds`;
|
||||||
}
|
}
|
||||||
|
|
||||||
function formatKernelTimeoutAnnotation(timeoutMs: number | undefined, kernelKilled: boolean, idle = false): string {
|
function formatKernelTimeoutAnnotation(timeoutMs: number | undefined, kernelKilled: boolean): string {
|
||||||
const secs = timeoutMs === undefined ? undefined : Math.max(1, Math.round(timeoutMs / 1000));
|
const secs = timeoutMs === undefined ? undefined : Math.max(1, Math.round(timeoutMs / 1000));
|
||||||
const suffix = idle ? " of inactivity" : "";
|
|
||||||
if (kernelKilled) {
|
if (kernelKilled) {
|
||||||
return `eval cell timed out${suffix} and the kernel was unresponsive to interrupt; the kernel has been killed and will be recreated on the next call.`;
|
return "eval cell timed out and the kernel was unresponsive to interrupt; the kernel has been killed and will be recreated on the next call.";
|
||||||
}
|
}
|
||||||
const duration = secs === undefined ? "the configured timeout" : `${secs}s${suffix}`;
|
const duration = secs === undefined ? "the configured timeout" : `${secs}s`;
|
||||||
return `eval cell timed out after ${duration}; kernel interrupted but remains running. Reset the kernel via { reset: true } if state appears corrupted.`;
|
return `eval cell timed out after ${duration}; kernel interrupted but remains running. Reset the kernel via { reset: true } if state appears corrupted.`;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -489,10 +487,6 @@ async function executeWithKernel(
|
|||||||
const displayOutputs: KernelDisplayOutput[] = [];
|
const displayOutputs: KernelDisplayOutput[] = [];
|
||||||
const deadlineMs = getExecutionDeadlineMs(options);
|
const deadlineMs = getExecutionDeadlineMs(options);
|
||||||
let executionTimeoutMs: number | undefined;
|
let executionTimeoutMs: number | undefined;
|
||||||
// Idle mode: the caller (eval tool) drives cancellation via an idle-aware
|
|
||||||
// signal and passes no wall-clock deadline, so annotate timeouts with the
|
|
||||||
// configured inactivity budget rather than a remaining-deadline figure.
|
|
||||||
const idleMode = deadlineMs === undefined && options?.idleTimeoutMs !== undefined;
|
|
||||||
|
|
||||||
// Collect every display output and, for status events, stream them live so
|
// Collect every display output and, for status events, stream them live so
|
||||||
// long-running bridge helpers (e.g. `agent()`) surface progress mid-cell.
|
// long-running bridge helpers (e.g. `agent()`) surface progress mid-cell.
|
||||||
@@ -530,11 +524,7 @@ async function executeWithKernel(
|
|||||||
|
|
||||||
if (result.cancelled) {
|
if (result.cancelled) {
|
||||||
const annotation = result.timedOut
|
const annotation = result.timedOut
|
||||||
? formatKernelTimeoutAnnotation(
|
? formatKernelTimeoutAnnotation(executionTimeoutMs ?? options?.idleTimeoutMs, result.kernelKilled ?? false)
|
||||||
executionTimeoutMs ?? options?.idleTimeoutMs,
|
|
||||||
result.kernelKilled ?? false,
|
|
||||||
idleMode,
|
|
||||||
)
|
|
||||||
: undefined;
|
: undefined;
|
||||||
return {
|
return {
|
||||||
exitCode: undefined,
|
exitCode: undefined,
|
||||||
@@ -572,7 +562,7 @@ async function executeWithKernel(
|
|||||||
displayOutputs,
|
displayOutputs,
|
||||||
stdinRequested: false,
|
stdinRequested: false,
|
||||||
...(await sink.dump(
|
...(await sink.dump(
|
||||||
timedOut ? formatTimeoutAnnotation(executionTimeoutMs ?? options?.idleTimeoutMs, idleMode) : undefined,
|
timedOut ? formatTimeoutAnnotation(executionTimeoutMs ?? options?.idleTimeoutMs) : undefined,
|
||||||
)),
|
)),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ Cell fields:
|
|||||||
- `language` — {{#if py}}`"py"` for the IPython kernel{{/if}}{{#ifAll py js}}, {{/ifAll}}{{#if js}}`"js"` for the persistent JavaScript VM{{/if}}.
|
- `language` — {{#if py}}`"py"` for the IPython kernel{{/if}}{{#ifAll py js}}, {{/ifAll}}{{#if js}}`"js"` for the persistent JavaScript VM{{/if}}.
|
||||||
- `code` — cell body, verbatim. Newlines, quotes, and indentation are JSON-encoded; no fences, no headers.
|
- `code` — cell body, verbatim. Newlines, quotes, and indentation are JSON-encoded; no fences, no headers.
|
||||||
- `title` (optional) — short label shown in the transcript (e.g. `"imports"`, `"load config"`).
|
- `title` (optional) — short label shown in the transcript (e.g. `"imports"`, `"load config"`).
|
||||||
- `timeout` (optional) — per-cell **inactivity** budget in seconds (1-600). Default 30. The cell is interrupted only after this long with no progress, and every status event (`agent()` updates, `log()`/`phase()`, tool activity) resets the clock — so a long `agent()`/`parallel()` fanout that keeps reporting progress is not killed. Raw `print`/stdout does not reset it; raise `timeout` for a cell that runs long without emitting status.
|
- `timeout` (optional) — per-cell wall-clock budget in seconds (1-600). Default 30. It bounds the cell's **own** work, but is paused while an `agent()`/`parallel()`/`llm()` call is in flight — so a long fanout or a slow completion runs to completion, while the cell itself is still bounded. Compute, `print`/stdout, `log()`/`phase()`, and ordinary tool calls all count against the budget; raise `timeout` for a cell that does heavy local work or long non-agent tool calls.
|
||||||
- `reset` (optional) — wipe this cell's language kernel before running.{{#ifAll py js}} Reset is per-language: a `py` cell's reset does not touch the JavaScript VM and vice versa.{{/ifAll}}
|
- `reset` (optional) — wipe this cell's language kernel before running.{{#ifAll py js}} Reset is per-language: a `py` cell's reset does not touch the JavaScript VM and vice versa.{{/ifAll}}
|
||||||
|
|
||||||
**Work incrementally:**
|
**Work incrementally:**
|
||||||
|
|||||||
@@ -348,15 +348,16 @@ export class EvalTool implements AgentTool<typeof evalSchema> {
|
|||||||
for (let i = 0; i < cells.length; i++) {
|
for (let i = 0; i < cells.length; i++) {
|
||||||
const cell = cells[i];
|
const cell = cells[i];
|
||||||
const backend = cell.resolved.backend;
|
const backend = cell.resolved.backend;
|
||||||
// The per-cell `timeout` is an *inactivity* budget, not a hard
|
// The per-cell `timeout` is a wall-clock budget on the cell's *own*
|
||||||
// wall-clock cap: it bounds the gap between progress signals
|
// work, but it is paused while a host-side `agent()`/`llm()` bridge
|
||||||
// (status events — agent() updates, log()/phase(), tool-bridge
|
// call is in flight: those calls pump a heartbeat (see
|
||||||
// activity), so a long fanout that keeps reporting progress runs to
|
// `withBridgeHeartbeat`) that re-arms the watchdog, so a long fanout
|
||||||
// completion while a genuinely stalled cell (no progress for the
|
// or a slow completion runs to completion. Nothing else re-arms it —
|
||||||
// whole window) is still interrupted. Raw stdout deliberately does
|
// compute, stdout, `log()`/`phase()`, and ordinary tool calls all
|
||||||
// NOT re-arm it, so pure-compute runaway loops stay bounded. The
|
// count against the budget — so a cell that is not delegating to an
|
||||||
// watchdog drives `combinedSignal`; we pass no wall-clock deadline
|
// agent/llm is bounded by a plain wall-clock timeout. The watchdog
|
||||||
// downstream so the backends never arm a competing fixed timer.
|
// drives `combinedSignal`; we pass no wall-clock deadline downstream
|
||||||
|
// so the backends never arm a competing fixed timer.
|
||||||
const idleTimeoutMs = timeoutSecondsFromMs(cell.timeoutMs) * 1000;
|
const idleTimeoutMs = timeoutSecondsFromMs(cell.timeoutMs) * 1000;
|
||||||
const idle = new IdleTimeout(idleTimeoutMs);
|
const idle = new IdleTimeout(idleTimeoutMs);
|
||||||
const combinedSignal = signal
|
const combinedSignal = signal
|
||||||
@@ -389,12 +390,16 @@ export class EvalTool implements AgentTool<typeof evalSchema> {
|
|||||||
outputSink!.push(chunk);
|
outputSink!.push(chunk);
|
||||||
},
|
},
|
||||||
onStatus: event => {
|
onStatus: event => {
|
||||||
// Every status event re-arms the inactivity watchdog. A
|
// Only a bridge heartbeat re-arms the watchdog: it is the
|
||||||
// heartbeat is a pure keepalive emitted while a host-side
|
// keepalive `agent()`/`llm()` pump while a host-side call is
|
||||||
// bridge call (agent()/llm()) runs: it bumps the timer but
|
// in flight, so those calls effectively pause the budget. It
|
||||||
// carries no payload, so don't persist or render it.
|
// carries no payload — bump and drop it. Every other event
|
||||||
idle.bump();
|
// (compute helpers, log()/phase(), tool results) renders but
|
||||||
if (event.op === EVAL_HEARTBEAT_OP) return;
|
// counts against the plain wall-clock budget.
|
||||||
|
if (event.op === EVAL_HEARTBEAT_OP) {
|
||||||
|
idle.bump();
|
||||||
|
return;
|
||||||
|
}
|
||||||
cellResult.statusEvents ??= [];
|
cellResult.statusEvents ??= [];
|
||||||
upsertStatusEvent(cellResult.statusEvents, event);
|
upsertStatusEvent(cellResult.statusEvents, event);
|
||||||
pushUpdate();
|
pushUpdate();
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import { afterAll, describe, expect, it } from "bun:test";
|
||||||
|
import { Settings } from "@oh-my-pi/pi-coding-agent/config/settings";
|
||||||
|
import { disposeAllVmContexts } from "@oh-my-pi/pi-coding-agent/eval/js/context-manager";
|
||||||
|
import type { ToolSession } from "@oh-my-pi/pi-coding-agent/tools";
|
||||||
|
import { EvalTool } from "@oh-my-pi/pi-coding-agent/tools/eval";
|
||||||
|
|
||||||
|
function makeSession(): ToolSession {
|
||||||
|
return {
|
||||||
|
cwd: process.cwd(),
|
||||||
|
hasUI: false,
|
||||||
|
getSessionFile: () => null,
|
||||||
|
getSessionSpawns: () => null,
|
||||||
|
settings: Settings.isolated(),
|
||||||
|
} as unknown as ToolSession;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Defends the contract that a cell which does not delegate to an `agent()`/
|
||||||
|
* `llm()` bridge call is bounded by a *plain wall-clock* timeout — not the
|
||||||
|
* activity watchdog, which now only extends the budget while a bridge call is in
|
||||||
|
* flight. Regression guard for the watchdog killing ordinary compute cells and
|
||||||
|
* surfacing a misleading "of inactivity" message.
|
||||||
|
*/
|
||||||
|
describe("EvalTool timeout semantics", () => {
|
||||||
|
afterAll(async () => {
|
||||||
|
await disposeAllVmContexts();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("bounds a compute cell (no agent/llm) by a plain wall-clock timeout", async () => {
|
||||||
|
const tool = new EvalTool(makeSession());
|
||||||
|
// 1s budget; the cell idles for 5s and emits no status, so nothing extends
|
||||||
|
// the budget — it must be cut off at the wall-clock limit.
|
||||||
|
const result = await tool.execute("call-compute-timeout", {
|
||||||
|
cells: [{ language: "js", code: "await Bun.sleep(5000); return 'never';", timeout: 1 }],
|
||||||
|
});
|
||||||
|
|
||||||
|
const text = result.content
|
||||||
|
.filter((block): block is { type: "text"; text: string } => block.type === "text")
|
||||||
|
.map(block => block.text)
|
||||||
|
.join("\n");
|
||||||
|
expect(text).toContain("timed out after 1 seconds");
|
||||||
|
// The new wording is a plain wall-clock timeout, not an inactivity stall.
|
||||||
|
expect(text).not.toContain("inactivity");
|
||||||
|
expect(text).not.toContain("never");
|
||||||
|
|
||||||
|
const cell = result.details?.cells?.[0];
|
||||||
|
expect(cell?.exitCode).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user