feat(eval): added per-cell inactivity timeout budgets in eval executors

- Changed eval timeout behavior from hard wall-clock deadlines to per-cell inactivity budgets in all executors.
- Added IdleTimeout watchdog support, including bumps on status/tool activity and timer cleanup after execution.
- Updated executor option plumbing to replace deadlineMs with idleTimeoutMs and emit inactivity timeout annotations.
- Added IdleTimeout and shared-executor tests and updated prompt/repl docs for the new timeout contract.
This commit is contained in:
can1357
2026-05-31 10:11:59 +02:00
parent d7a5fe1ce5
commit 2003d7382e
12 changed files with 283 additions and 47 deletions
+5 -3
View File
@@ -27,7 +27,7 @@ Tool params:
language: "py" | "js";
code: string;
title?: string;
timeout?: number; // seconds, clamped to 1..600, default 30
timeout?: number; // seconds, clamped to 1..600, default 30. Inactivity budget — see "Cell timeout".
reset?: boolean; // reset this cell's selected runtime before execution
}>;
}
@@ -166,7 +166,9 @@ Python prelude helpers include `agent(prompt, *, agent_type="task", model=None,
### Cell timeout
Each eval cell timeout is in seconds, defaults to 30, and is clamped to `1..600`. The tool combines caller abort signal, session abort signal, and the current cell timeout with `AbortSignal.any(...)`.
Each eval cell `timeout` is in seconds, defaults to 30, and is clamped to `1..600`. It is an **inactivity (idle) budget, not a hard wall-clock cap**: the watchdog (`IdleTimeout`, `src/eval/idle-timeout.ts`) only fires once the cell goes the full window with **no progress signal**. Every status event re-arms it — `agent()` progress snapshots, `log()`/`phase()`, and tool-bridge activity all count — so a long-running fanout that keeps reporting progress runs to completion instead of being killed mid-stream.
Raw `stdout`/`stderr` does **not** re-arm the watchdog, so a pure-compute runaway loop with no progress reporting is still bounded by `timeout`. The tool combines the caller abort signal, the session abort signal, and the idle watchdog's signal with `AbortSignal.any(...)`; no wall-clock deadline is passed to the backend, so neither runtime arms a competing fixed timer.
### Kernel execution cancellation
@@ -174,7 +176,7 @@ On abort/timeout:
- The host sends `kill("SIGINT")` to the runner subprocess.
- The runner's exec-time signal handler raises `KeyboardInterrupt` inside the user code.
- Result includes `cancelled=true`; timeout path annotates output as `Command timed out after <n> seconds`.
- Result includes `cancelled=true`; the timeout path annotates output as `Command timed out after <n> seconds of inactivity`.
- Between requests the runner installs `SIG_IGN` for SIGINT so a stray cancel does not tear down the kernel.
If a second cancel is required (runner stuck in C code), the host escalates to `SIGTERM` and the session restarts on the next call.