feat(eval): added per-cell inactivity timeout budgets in eval executors
- Changed eval timeout behavior from hard wall-clock deadlines to per-cell inactivity budgets in all executors. - Added IdleTimeout watchdog support, including bumps on status/tool activity and timer cleanup after execution. - Updated executor option plumbing to replace deadlineMs with idleTimeoutMs and emit inactivity timeout annotations. - Added IdleTimeout and shared-executor tests and updated prompt/repl docs for the new timeout contract.
This commit is contained in:
+5
-3
@@ -27,7 +27,7 @@ Tool params:
|
||||
language: "py" | "js";
|
||||
code: string;
|
||||
title?: string;
|
||||
timeout?: number; // seconds, clamped to 1..600, default 30
|
||||
timeout?: number; // seconds, clamped to 1..600, default 30. Inactivity budget — see "Cell timeout".
|
||||
reset?: boolean; // reset this cell's selected runtime before execution
|
||||
}>;
|
||||
}
|
||||
@@ -166,7 +166,9 @@ Python prelude helpers include `agent(prompt, *, agent_type="task", model=None,
|
||||
|
||||
### Cell timeout
|
||||
|
||||
Each eval cell timeout is in seconds, defaults to 30, and is clamped to `1..600`. The tool combines caller abort signal, session abort signal, and the current cell timeout with `AbortSignal.any(...)`.
|
||||
Each eval cell `timeout` is in seconds, defaults to 30, and is clamped to `1..600`. It is an **inactivity (idle) budget, not a hard wall-clock cap**: the watchdog (`IdleTimeout`, `src/eval/idle-timeout.ts`) only fires once the cell goes the full window with **no progress signal**. Every status event re-arms it — `agent()` progress snapshots, `log()`/`phase()`, and tool-bridge activity all count — so a long-running fanout that keeps reporting progress runs to completion instead of being killed mid-stream.
|
||||
|
||||
Raw `stdout`/`stderr` does **not** re-arm the watchdog, so a pure-compute runaway loop with no progress reporting is still bounded by `timeout`. The tool combines the caller abort signal, the session abort signal, and the idle watchdog's signal with `AbortSignal.any(...)`; no wall-clock deadline is passed to the backend, so neither runtime arms a competing fixed timer.
|
||||
|
||||
### Kernel execution cancellation
|
||||
|
||||
@@ -174,7 +176,7 @@ On abort/timeout:
|
||||
|
||||
- The host sends `kill("SIGINT")` to the runner subprocess.
|
||||
- The runner's exec-time signal handler raises `KeyboardInterrupt` inside the user code.
|
||||
- Result includes `cancelled=true`; timeout path annotates output as `Command timed out after <n> seconds`.
|
||||
- Result includes `cancelled=true`; the timeout path annotates output as `Command timed out after <n> seconds of inactivity`.
|
||||
- Between requests the runner installs `SIG_IGN` for SIGINT so a stray cancel does not tear down the kernel.
|
||||
|
||||
If a second cancel is required (runner stuck in C code), the host escalates to `SIGTERM` and the session restarts on the next call.
|
||||
|
||||
Reference in New Issue
Block a user