feat(coding-agent): added parser support for t/rst in Cell eval

- Introduced canonical `*** Cell` headers with `t:` and `rst` attributes in eval prompts, schema, and docs.
- Updated parser and grammar to parse `*** Cell` blocks, stop on `*** End`/next header/EOF, and handle invalid `rst` with errors.
- Added quote-aware attribute tokenizers and split HTML eval parsing into `Cell` and legacy `Begin` handlers with `py` defaults.
- Expanded parsing behavior and tests for `rst` booleans, title aliases, abort boundaries, and stray-line skips between cells.
This commit is contained in:
can1357
2026-05-12 10:01:56 +02:00
parent c562f4dc2a
commit 028a442f18
9 changed files with 570 additions and 245 deletions
+21 -13
View File
@@ -28,22 +28,30 @@
`input` syntax accepted at runtime:
- Cell header: `*** Begin <LANG>`; parser accepts `PY`, `PYTHON`, `IPY`, `IPYTHON`, `JS`, `JAVASCRIPT`, `TS`, `TYPESCRIPT` case-insensitively.
- Optional attributes immediately after the header, first occurrence wins:
- `*** Title: ...`
- `*** Timeout: <n>[ms|s|m]` (default 30s)
- `*** Reset`
- Cell body: every following line until `*** End ...`, the next `*** Begin ...`, or `*** Abort`.
- Cell header: `*** Cell <attrs...>`. Attributes are space-separated tokens with quoted titles (`"..."` or `'...'`).
- Canonical tokens (advertised in the prompt):
- `<lang>:"<title>"` — language + title shorthand. `lang` is `py` or `js` (lenient: also `ts`, plus the long-form aliases `python`, `javascript`, `typescript`, `ipy`, `ipython`).
- `t:<n>[ms|s|m]` — per-cell timeout (default 30s).
- `rst` — wipe this cell's language kernel before running.
- Lenient additional tokens (accepted by the parser, not advertised):
- bare language token (`py`, `js`)
- `id:"..."` / `title:"..."` / `name:"..."` / `cell:"..."` / `file:"..."` / `label:"..."` — title aliases
- `timeout:` / `duration:` / `time:` — `t:` aliases
- `reset` — `rst` alias
- `rst:true|false|1|0|yes|no|on|off` — explicit boolean form
- a bare positional duration token (`30s`, `2m`, `500ms`)
- any unclassified bare token folds into a positional title fragment
- Cell body: every following line until the next `*** Cell ...`, the optional `*** End`, or `*** Abort`. `*** End` is a quirk fix for GPT-trained models that emit terminators and is not documented in the prompt.
Leniencies in `packages/coding-agent/src/eval/parse.ts`:
- Markers accept two or more leading `*` and flexible whitespace.
- `*** End` does not need to repeat the language token.
- Missing end markers between adjacent cells are tolerated; the next `*** Begin` closes the prior cell.
- `*** End` is optional everywhere; the parser silently consumes trailing tokens (e.g. `*** End py`).
- Missing terminators between adjacent cells are tolerated; the next `*** Cell` closes the prior cell, and stray non-marker lines between cells fold into the prior cell's body without crashing.
- Bare code or a single markdown fence such as ```` ```py ```` is treated as one implicit cell.
- If `*** Abort` appears, the in-progress cell is dropped and the result carries an abort warning.
- If `*** Abort` appears, the in-progress cell is dropped and the result carries an abort warning. To preserve a completed cell before `*** Abort`, emit `*** End` first.
The tool also exposes a custom Lark grammar from `packages/coding-agent/src/eval/eval.lark` for constrained sampling. That grammar is stricter than the runtime parser: it only advertises `PY` / `JS` / `TS` headers and an `*** End Cell` closer.
The tool also exposes a custom Lark grammar from `packages/coding-agent/src/eval/eval.lark` for constrained sampling. That grammar is stricter than the runtime parser: it requires the canonical `*** Cell <lang>:"title"` header form with a fixed attribute order, advertises only `py` / `js`, and pins the trailing `*** End` so GPT-trained models' natural terminator habit aligns with the constrained output.
## Outputs
@@ -107,7 +115,7 @@ Side-channel artifacts:
### Parsing modes
- Explicit multi-cell format with `*** Begin ...` / `*** End ...`
- Explicit multi-cell format with `*** Cell ...` headers
- Implicit single-cell fallback for bare code or a single fenced block
- Abort-recovery parse path when `*** Abort` is present
@@ -124,7 +132,7 @@ Side-channel artifacts:
Implemented in `packages/coding-agent/src/eval/js/context-manager.ts` and `packages/coding-agent/src/eval/js/prelude.txt`.
- Persistent `vm.Context` instances keyed by `js:${sessionId}` in `vmContexts`
- `*** Reset` calls `resetVmContext(sessionKey)` before the cell executes
- `rst` calls `resetVmContext(sessionKey)` before the cell executes
- Top-level `await` and bare `return` are supported by wrapping code in an async IIFE when `wrapCode()` sees `await` or `return`
- Top-level static `import ... from ...` is rewritten to `await import(...)` by `rewriteStaticImports()`
- The prelude installs globals:
@@ -145,7 +153,7 @@ Implemented in `packages/coding-agent/src/eval/py/executor.ts`, `packages/coding
- Default mode is retained `session` kernels keyed by `python:${sessionId}`
- Optional `python.kernelMode = "per-call"` creates a fresh kernel for each cell and shuts it down afterward
- `*** Reset` disposes the retained kernel for that session before the cell runs; later Python cells in the same tool call reuse the fresh kernel
- `rst` disposes the retained kernel for that session before the cell runs; later Python cells in the same tool call reuse the fresh kernel
- Startup path:
- availability check
- create/connect kernel
+17
View File
@@ -1,6 +1,23 @@
# Changelog
## [Unreleased]
### Breaking Changes
- Changed the `eval` tool input format to a single-line `*** Cell <lang>:"<title>" [t:<duration>] [rst]` header per cell, replacing the `*** Begin <LANG>` / `*** End <LANG>` envelope and the standalone `*** Title:` / `*** Timeout:` / `*** Reset` directives. The lark grammar enforces a fixed attribute order; the runtime parser remains lenient (alias keys, bare positional tokens, single-quoted titles).
### Added
- Added support for explicit boolean `rst` values (`rst:true`, `rst:false`, `rst:1`, `rst:0`, `rst:yes`, `rst:no`, `rst:on`, `rst:off`) in `*** Cell` headers
### Changed
- Changed the HTML transcript renderer to parse the new `*** Cell` headers while keeping the older `*** Begin <LANG>` and `===== ... =====` formats renderable for historical sessions.
- Changed the `eval` tool parser so a stray non-marker line between cells no longer crashes with `null is not an object (evaluating 'BEGIN_RE.exec(lines[i])[1]')`; stray content is consumed without aborting parsing.
- Changed `*** End` to be an optional, undocumented per-cell terminator (kept in the lark to satisfy GPT-trained models' natural terminator habit during constrained sampling).
### Fixed
- Improved `*** Cell` header parsing to reject invalid `rst` values with a clear `invalid rst value` error
## [14.9.7] - 2026-05-12
+30 -10
View File
@@ -1,16 +1,36 @@
start: cell+
// Canonical Eval input. Each cell is introduced by a single header line:
//
// *** Cell <LANG>:"<title>" [t:<duration>] [rst]
//
// Attribute order is fixed: language+title, then optional timeout, then
// optional reset flag. Title may be empty (`py:""`).
//
// Tokens:
//
// py:"..." | js:"..." language plus title (required)
// t:<digits>(ms|s|m)? per-cell timeout (default 30s)
// rst reset this language's kernel before running
//
// Everything between one header line and the next (or the optional trailing
// `*** End`, or end of input) is the cell's code, verbatim. The runtime
// parser additionally accepts content before the first header as an implicit
// default-language cell, but that is lenient fallback and MUST NOT be relied
// on.
cell: begin_cell attr* code_line* end_cell
begin_cell: "*** Begin " LANG LF
end_cell: "*** End Cell" LF?
start: cell+ end_marker
attr: title | timeout | reset
title: "*** Title: " /(.+)/ LF
timeout: "*** Timeout: " /\d+(ms|s|m)?/ LF
reset: "*** Reset" LF
cell: cell_header code_line*
code_line: /[^\r\n]*/ LF
cell_header: "*** Cell" WS_INLINE LANG_TITLE (WS_INLINE T_ATTR)? (WS_INLINE RST_FLAG)? LF
LANG: "JS" | "TS" | "PY"
end_marker: "*** End" LF?
code_line: CODE_TEXT LF | LF
CODE_TEXT: /([^*\r\n]|\*\*?[^*\r\n])+\*{0,2}|\*{1,2}/
LANG_TITLE: ("py" | "js") ":\"" /[^"\r\n]*/ "\""
T_ATTR: "t:" /\d+(ms|s|m)?/
RST_FLAG: "rst"
%import common.LF
%import common.WS_INLINE
+212 -55
View File
@@ -43,16 +43,19 @@ const LANGUAGE_MAP: Record<string, EvalLanguage> = {
TYPESCRIPT: "js",
};
// Markers are case-insensitive, accept ≥2 leading stars (so `**Begin` and
// `*** Begin` both work), and tolerate any whitespace (including tabs)
// Markers are case-insensitive, accept ≥2 leading stars (so `**Cell` and
// `*** Cell` both work), and tolerate any whitespace (including tabs)
// between tokens. Models that can't constrain-sample frequently emit minor
// variations like `**End`, `*** end py`, or `***\tTitle: foo`.
// variations like `**End` or `*** cell py`.
const STARS = String.raw`\*{2,}`;
const BEGIN_RE = new RegExp(`^${STARS}\\s*Begin\\b\\s*(\\S+)?\\s*$`, "i");
// Cell header: `*** Cell <attrs...>`. The remainder of the line is captured
// and tokenized separately so we can handle quoted values.
const CELL_RE = new RegExp(`^${STARS}\\s*Cell\\b\\s*(.*)$`, "i");
// `*** End` is a tolerated cell/file terminator. Documented as required at
// the file level in the lark grammar (the trailing `*** End` quirks GPT-
// trained models naturally produce), but optional at the parser level.
const END_RE = new RegExp(`^${STARS}\\s*End\\b.*$`, "i");
const TITLE_RE = new RegExp(`^${STARS}\\s*Title\\s*:\\s*(.+?)\\s*$`, "i");
const TIMEOUT_RE = new RegExp(`^${STARS}\\s*Timeout\\s*:\\s*(\\S+)\\s*$`, "i");
const RESET_RE = new RegExp(`^${STARS}\\s*Reset\\s*$`, "i");
// `*** Abort` is the harmony-leak recovery sentinel; see ABORT_WARNING.
const ABORT_RE = new RegExp(`^${STARS}\\s*Abort\\s*$`, "i");
/**
@@ -62,6 +65,7 @@ const ABORT_RE = new RegExp(`^${STARS}\\s*Abort\\s*$`, "i");
*/
export const ABORT_WARNING =
"Tool stream truncated mid-call due to detected output corruption. Earlier cells (if any) executed normally; their state persists. Re-issue the aborted cell.";
const DURATION_RE = /^(\d+)(ms|s|m)?$/i;
function resolveLang(token: string | undefined): EvalLanguage | undefined {
@@ -88,7 +92,7 @@ const FENCE_OPEN_RE = /^```\s*([A-Za-z]\w*)?\s*$/;
const FENCE_CLOSE_RE = /^```\s*$/;
/**
* Last-resort fallback when the input has no recognizable `*** Begin` header.
* Last-resort fallback when the input has no recognizable `*** Cell` header.
* Models that can't constrain-sample sometimes pass bare code or wrap it in
* a markdown fence (```py / ```python / bare ```). Treat the whole input as
* a single implicit cell, sniffing the language from the body.
@@ -122,6 +126,187 @@ function parseImplicitCell(lines: string[]): ParsedEvalCell {
};
}
/**
* Tokenize a `*** Cell` header's attribute list while preserving quoted
* segments (`id:"some title"`, `py:"hi"`, single quotes too) as single
* tokens. Outer whitespace separates tokens; the quote characters
* themselves are kept verbatim so attribute parsing can strip them later.
*/
function tokenizeCellAttrs(input: string): string[] {
const tokens: string[] = [];
let i = 0;
while (i < input.length) {
while (i < input.length && /\s/.test(input[i])) i++;
if (i >= input.length) break;
let token = "";
while (i < input.length && !/\s/.test(input[i])) {
const ch = input[i];
if (ch === '"' || ch === "'") {
token += ch;
i++;
while (i < input.length && input[i] !== ch) {
token += input[i];
i++;
}
if (i < input.length) {
token += input[i];
i++;
}
} else {
token += ch;
i++;
}
}
tokens.push(token);
}
return tokens;
}
interface CellHeader {
language: EvalLanguage | undefined;
languageOrigin: EvalLanguageOrigin;
title: string | undefined;
timeoutMs: number | undefined;
reset: boolean;
}
/**
* Map an attribute key (from `key:value` or bare `key`) to one of the three
* canonical roles. Canonical keys: `id`, `t`, `rst`. Fallback aliases —
* accepted but not advertised in the prompt — cover common synonyms LLMs
* reach for instead of the short canonical.
*/
const ID_KEYS = new Set(["id", "title", "name", "cell", "file", "label"]);
const T_KEYS = new Set(["t", "timeout", "duration", "time"]);
const RST_KEYS = new Set(["rst", "reset"]);
function classifyAttrKey(key: string): "id" | "t" | "rst" | null {
if (ID_KEYS.has(key)) return "id";
if (T_KEYS.has(key)) return "t";
if (RST_KEYS.has(key)) return "rst";
return null;
}
// `key:value` form. `value` may be `"..."`, `'...'`, or a bare run.
const ATTR_TOKEN_RE = /^([a-zA-Z][\w-]*)(?::(?:"([^"]*)"|'([^']*)'|(.*)))?$/;
// Bare positional duration (lenient — `t:` is canonical).
const DURATION_TOKEN_RE = /^\d+(?:ms|s|m)?$/;
function parseBooleanFlag(value: string): boolean | undefined {
const v = value.trim().toLowerCase();
if (v === "true" || v === "1" || v === "yes" || v === "on") return true;
if (v === "false" || v === "0" || v === "no" || v === "off") return false;
return undefined;
}
/**
* Decode a `*** Cell` header's attribute list into language, title,
* timeout, and reset flag.
*
* Token forms (all optional, any order):
* - `py` / `js` / `ts` bare language
* - `py:"..."` / `js:"..."` / `ts:"..."` language + title shorthand
* - `id:"..."` cell title (canonical)
* - `t:<duration>` per-cell timeout (canonical)
* - `<duration>` (e.g. `30s`) bare positional duration
* - `rst` reset flag (canonical)
* - `rst:true|false|1|0|yes|no|on|off` reset flag with explicit value
*
* Fallback aliases (accepted but not advertised in the prompt):
* - id: title, name, cell, file, label
* - t: timeout, duration, time
* - rst: reset
*
* Quotes may be `"` or `'`. Truly unknown keys are silently dropped. First
* occurrence wins when a key is repeated (canonical or alias). Anything
* that doesn't classify accumulates as a positional title fragment joined
* by spaces.
*/
function parseCellHeader(rest: string, lineNumber: number): CellHeader {
const tokens = tokenizeCellAttrs(rest);
let language: EvalLanguage | undefined;
let titleAttr: string | undefined;
let positionalDurationMs: number | undefined;
let tAttr: string | undefined;
let rstAttr: string | undefined;
let bareReset = false;
const titleParts: string[] = [];
for (const token of tokens) {
// Bare reset flag (canonical or alias).
if (RST_KEYS.has(token.toLowerCase())) {
bareReset = true;
continue;
}
const attrMatch = ATTR_TOKEN_RE.exec(token);
if (attrMatch && token.includes(":")) {
const key = attrMatch[1].toLowerCase();
const value = attrMatch[2] ?? attrMatch[3] ?? attrMatch[4] ?? "";
// Language-with-title shorthand: `py:"foo"`, `js:'bar'`, etc.
const langCandidate = resolveLang(key);
if (langCandidate) {
if (language === undefined) language = langCandidate;
if (titleAttr === undefined && value !== "") titleAttr = value;
continue;
}
const role = classifyAttrKey(key);
if (role === "id" && titleAttr === undefined) titleAttr = value;
else if (role === "t" && tAttr === undefined) tAttr = value;
else if (role === "rst" && rstAttr === undefined) rstAttr = value;
// unknown / repeated keys silently dropped
continue;
}
// Bare language token (no colon).
const lang = resolveLang(token);
if (lang && language === undefined) {
language = lang;
continue;
}
// Bare positional duration (lenient — `t:` is canonical).
if (positionalDurationMs === undefined && DURATION_TOKEN_RE.test(token)) {
positionalDurationMs = parseDurationMs(token, lineNumber);
continue;
}
titleParts.push(token);
}
const explicitTitle = (titleAttr ?? "").trim();
const positionalTitle = titleParts.join(" ").trim();
const title = explicitTitle.length > 0 ? explicitTitle : positionalTitle.length > 0 ? positionalTitle : undefined;
let timeoutMs: number | undefined;
if (tAttr !== undefined) {
timeoutMs = parseDurationMs(tAttr, lineNumber);
} else if (positionalDurationMs !== undefined) {
timeoutMs = positionalDurationMs;
}
let reset = false;
if (rstAttr !== undefined) {
const parsed = parseBooleanFlag(rstAttr);
if (parsed === undefined) {
throw new Error(`Eval line ${lineNumber}: invalid rst value \`${rstAttr}\`; use true or false.`);
}
reset = parsed;
} else if (bareReset) {
reset = true;
}
return {
language,
languageOrigin: language ? "header" : "default",
title,
timeoutMs,
reset,
};
}
export function parseEvalInput(input: string): ParsedEvalInput {
const normalized = input.replace(/\r\n?/g, "\n");
const lines = normalized.split("\n");
@@ -134,10 +319,10 @@ export function parseEvalInput(input: string): ParsedEvalInput {
// Skip leading blank lines.
while (i < lines.length && lines[i].trim() === "") i++;
// Lenient fallback: if the input has no recognizable begin marker, treat
// Lenient fallback: if the input has no recognizable cell header, treat
// the entire input as one implicit cell — unless that content contains
// `*** Abort`, in which case the body is incomplete/unsafe and we drop it.
if (i < lines.length && !BEGIN_RE.test(lines[i])) {
if (i < lines.length && !CELL_RE.test(lines[i])) {
const tail = lines.slice(i);
if (tail.some(line => ABORT_RE.test(line))) {
return { cells, aborted: true };
@@ -148,54 +333,26 @@ export function parseEvalInput(input: string): ParsedEvalInput {
}
while (i < lines.length) {
const beginMatch = BEGIN_RE.exec(lines[i]);
if (!beginMatch) {
const headerLine = lines[i];
const cellMatch = CELL_RE.exec(headerLine);
if (!cellMatch) {
// Stray content between/after cells (blank lines were already
// consumed). `*** Abort` here terminates parsing; anything else
// — typically a harmony-leak fragment or model garbage — is
// skipped rather than crashing the parser.
if (ABORT_RE.test(lines[i])) {
// consumed). `*** Abort` here terminates parsing; `*** End` is
// the optional file-level terminator (silently consumed). Anything
// else — typically a harmony-leak fragment — is skipped.
if (ABORT_RE.test(headerLine)) {
aborted = true;
break;
}
i++;
continue;
}
const langToken = beginMatch[1];
const explicitLanguage = resolveLang(langToken);
const header = parseCellHeader(cellMatch[1] ?? "", i + 1);
i++;
let title: string | undefined;
let timeoutMs: number | undefined;
let reset = false;
while (i < lines.length) {
const line = lines[i];
const lineNumber = i + 1;
const titleMatch = TITLE_RE.exec(line);
if (titleMatch) {
if (title === undefined) title = titleMatch[1];
i++;
continue;
}
const timeoutMatch = TIMEOUT_RE.exec(line);
if (timeoutMatch) {
if (timeoutMs === undefined) timeoutMs = parseDurationMs(timeoutMatch[1], lineNumber);
i++;
continue;
}
if (RESET_RE.test(line)) {
reset = true;
i++;
continue;
}
break;
}
// Collect cell body. Close on `*** End` OR on the next `*** Begin`
// (implicit end — leniency for models that drop end markers between
// back-to-back cells). `*** Abort` (recovery sentinel) drops the
// in-progress cell entirely: its body is partial and unsafe to run.
// Collect cell body. Close on `*** End` (any form), the next
// `*** Cell` header, or `*** Abort` (which drops the in-progress
// cell as its body is partial and unsafe to run).
const codeLines: string[] = [];
let cellAborted = false;
while (i < lines.length) {
@@ -210,7 +367,7 @@ export function parseEvalInput(input: string): ParsedEvalInput {
i++;
break;
}
if (BEGIN_RE.test(line)) break;
if (CELL_RE.test(line)) break;
codeLines.push(line);
i++;
}
@@ -224,17 +381,17 @@ export function parseEvalInput(input: string): ParsedEvalInput {
}
const code = codeLines.join("\n");
const language = explicitLanguage ?? sniffEvalLanguage(code) ?? DEFAULT_LANGUAGE;
const languageOrigin: EvalLanguageOrigin = explicitLanguage ? "header" : "default";
const language = header.language ?? sniffEvalLanguage(code) ?? DEFAULT_LANGUAGE;
const languageOrigin: EvalLanguageOrigin = header.language ? "header" : "default";
cells.push({
index: cells.length,
title,
title: header.title,
code,
language,
languageOrigin,
timeoutMs: timeoutMs ?? DEFAULT_TIMEOUT_MS,
reset,
timeoutMs: header.timeoutMs ?? DEFAULT_TIMEOUT_MS,
reset: header.reset,
});
// Skip blank separator lines between cells; an `*** Abort` here
File diff suppressed because one or more lines are too long
@@ -1262,12 +1262,14 @@
return html;
}
// Parse `*** Begin <LANG>` cell headers (canonical) and the legacy
// `===== <info> =====` headers used by older transcripts. Cells emitted
// before the format cutover still need to render in HTML exports.
// Parse `*** Cell <attrs>` headers (canonical), plus legacy
// `*** Begin <LANG>` headers and `===== <info> =====` bars used in
// older transcripts. Cells emitted before each format cutover still
// need to render in HTML exports.
function parseEvalCells(input) {
const text = String(input);
if (/^[*]{2,}\s*Begin\b/im.test(text)) return parseEvalCellsNew(text);
if (/^[*]{2,}\s*Cell\b/im.test(text)) return parseEvalCellsCell(text);
if (/^[*]{2,}\s*Begin\b/im.test(text)) return parseEvalCellsBegin(text);
return parseEvalCellsLegacy(text);
}
@@ -1279,7 +1281,93 @@
return null;
}
function parseEvalCellsNew(text) {
// Tokenize a `*** Cell` header attribute list, preserving quoted
// segments. Mirrors `tokenizeCellAttrs` in src/eval/parse.ts.
function tokenizeCellAttrsHtml(input) {
const tokens = [];
let i = 0;
while (i < input.length) {
while (i < input.length && /\s/.test(input[i])) i++;
if (i >= input.length) break;
let tok = '';
while (i < input.length && !/\s/.test(input[i])) {
const ch = input[i];
if (ch === '"' || ch === "'") {
tok += ch; i++;
while (i < input.length && input[i] !== ch) { tok += input[i]; i++; }
if (i < input.length) { tok += input[i]; i++; }
} else { tok += ch; i++; }
}
tokens.push(tok);
}
return tokens;
}
function parseEvalCellsCell(text) {
const STARS = '\\*{2,}';
const CELL = new RegExp('^' + STARS + '\\s*Cell\\b\\s*(.*)$', 'i');
const END = new RegExp('^' + STARS + '\\s*End\\b.*$', 'i');
const ATTR = /^([a-zA-Z][\w-]*)(?::(?:"([^"]*)"|'([^']*)'|(.*)))?$/;
const DUR = /^\d+(?:ms|s|m)?$/;
const ID_KEYS = ['id', 'title', 'name', 'cell', 'file', 'label'];
const T_KEYS = ['t', 'timeout', 'duration', 'time'];
const RST_KEYS = ['rst', 'reset'];
const lines = text.split('\n');
if (lines.length && lines[lines.length - 1] === '') lines.pop();
const cells = [];
let i = 0;
while (i < lines.length && lines[i].trim() === '') i++;
while (i < lines.length) {
const m = CELL.exec(lines[i]);
if (!m) { i++; continue; }
const tokens = tokenizeCellAttrsHtml(m[1] || '');
let lang = null;
let title = '';
const attrs = [];
let bareReset = false;
const titleParts = [];
for (const tok of tokens) {
const lower = tok.toLowerCase();
if (RST_KEYS.indexOf(lower) >= 0) { bareReset = true; continue; }
const am = ATTR.exec(tok);
if (am && tok.indexOf(':') >= 0) {
const key = am[1].toLowerCase();
const value = am[2] !== undefined ? am[2] : am[3] !== undefined ? am[3] : (am[4] || '');
const lc = evalLangAlias(key);
if (lc) {
if (!lang) lang = lc;
if (!title && value) title = value;
continue;
}
if (ID_KEYS.indexOf(key) >= 0) { if (!title) title = value; continue; }
if (T_KEYS.indexOf(key) >= 0) { attrs.push('t=' + value); continue; }
if (RST_KEYS.indexOf(key) >= 0) { attrs.push('rst'); continue; }
continue;
}
const lc = evalLangAlias(tok);
if (lc && !lang) { lang = lc; continue; }
if (DUR.test(tok)) { attrs.push('t=' + tok); continue; }
titleParts.push(tok);
}
if (!title && titleParts.length) title = titleParts.join(' ');
if (bareReset) attrs.push('rst');
lang = lang || 'py';
i++;
const codeLines = [];
while (i < lines.length) {
if (END.test(lines[i])) { i++; break; }
if (CELL.test(lines[i])) break;
codeLines.push(lines[i]);
i++;
}
while (codeLines.length && codeLines[codeLines.length - 1].trim() === '') codeLines.pop();
cells.push({ lang, title, attrs, code: codeLines.join('\n') });
while (i < lines.length && lines[i].trim() === '') i++;
}
return cells;
}
function parseEvalCellsBegin(text) {
const STARS = '\\*{2,}';
const BEGIN = new RegExp('^' + STARS + '\\s*Begin\\b\\s*(\\S+)?\\s*$', 'i');
const END = new RegExp('^' + STARS + '\\s*End\\b.*$', 'i');
+11 -24
View File
@@ -1,23 +1,18 @@
Run code in a persistent kernel using codeblock cells.
<instruction>
Each cell is wrapped between `*** Begin <LANG>` and `*** End <LANG>`:
Each cell starts with a single header line and runs until the next header (or end of input):
```
*** Begin PY
*** Title: optional title
*** Timeout: 10s
*** Reset
*** Cell py:"optional title" t:10s rst
print("hi")
*** End PY
```
- **Language**: {{#if py}}`PY` for Python{{/if}}{{#ifAll py js}}, {{/ifAll}}{{#if js}}`JS` / `TS` for JavaScript{{/if}}. The opening `<LANG>` and closing `<LANG>` **MUST** match.
- **Attributes** (optional, in any order, immediately after `*** Begin`):
- `*** Title: …` — cell title shown in the UI.
- `*** Timeout: <duration>` — per-cell timeout. Digits with optional `ms` / `s` / `m` units (e.g. `500ms`, `15s`, `2m`). Default 30s.
- `*** Reset` — wipe this cell's own language kernel before running.{{#ifAll py js}} Other languages are untouched.{{/ifAll}}
- Anything between the last attribute and `*** End <LANG>` is the cell's code, verbatim.
- **Language + title**: `<lang>:"<title>"` — {{#if py}}`py` for Python{{/if}}{{#ifAll py js}}, {{/ifAll}}{{#if js}}`js` for JavaScript{{/if}}. Title may be empty (`py:""`).
- **Attributes** (optional, in this order, after the language+title):
- `t:<duration>` — per-cell timeout. Digits with optional `ms` / `s` / `m` units (e.g. `500ms`, `15s`, `2m`). Default 30s.
- `rst` — wipe this cell's own language kernel before running.{{#ifAll py js}} Other languages are untouched.{{/ifAll}}
- Anything after the header line, up to the next `*** Cell` header, is the cell's code, verbatim.
- Stack multiple cells back-to-back; blank lines between cells are ignored.
**Work incrementally:**
@@ -60,30 +55,22 @@ Cells render like a Jupyter notebook. `display(value)` renders non-presentable d
</output>
<caution>
- In session mode, use `*** Reset` on a cell to wipe its language's kernel before running.{{#ifAll py js}} Reset is per-language: a python cell's `*** Reset` does not touch the JavaScript kernel and vice versa.{{/ifAll}}
- In session mode, use `rst` on a cell to wipe its language's kernel before running.{{#ifAll py js}} Reset is per-language: a python cell's `rst` does not touch the JavaScript kernel and vice versa.{{/ifAll}}
{{#if js}}- **js**: the VM exposes a selective `process` subset, Web APIs, `Buffer`, `fs/promises`, and the `Bun` global.
{{/if}}</caution>
<example>
{{#if py}}*** Begin PY
*** Title: imports
*** Timeout: 10s
{{#if py}}*** Cell py:"imports" t:10s
import json
from pathlib import Path
*** End PY
*** Begin PY
*** Title: load config
*** Cell py:"load config"
data = json.loads(read('package.json'))
display(data)
*** End PY
{{/if}}{{#ifAll py js}}
{{/ifAll}}{{#if js}}*** Begin JS
*** Title: js summary
*** Reset
{{/ifAll}}{{#if js}}*** Cell js:"summary" rst
const data = JSON.parse(await read('package.json'));
display(data);
return data.name;
*** End JS
{{/if}}
</example>
+1 -1
View File
@@ -26,7 +26,7 @@ export const EVAL_DEFAULT_PREVIEW_LINES = 10;
export const evalSchema = Type.Object({
input: Type.String({
description: "eval input as a sequence of `*** Begin <LANG>` cell headers followed by code",
description: 'eval input as a sequence of `*** Cell <lang>:"title"` cell headers followed by code',
}),
});
export type EvalToolParams = Static<typeof evalSchema>;
+184 -136
View File
@@ -3,191 +3,156 @@ import { parseEvalInput } from "../../src/eval/parse";
describe("parseEvalInput", () => {
it("parses a single cell with title and timeout", () => {
const result = parseEvalInput(`*** Begin PY
*** Title: setup
*** Timeout: 15s
const result = parseEvalInput(`*** Cell py:"setup" t:10s
print("hi")
*** End PY
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0]).toMatchObject({
index: 0,
title: "setup",
code: 'print("hi")',
language: "python",
languageOrigin: "header",
timeoutMs: 15_000,
timeoutMs: 10_000,
reset: false,
code: 'print("hi")',
});
});
it("treats *** Reset as a per-cell kernel wipe", () => {
const result = parseEvalInput(`*** Begin PY
*** Title: bootstrap
*** Reset
it("treats rst as a per-cell kernel wipe", () => {
const result = parseEvalInput(`*** Cell py:"bootstrap"
import json
*** End PY
*** Begin JS
*** Reset
const x = 1;
*** End JS
`);
*** Cell js:"" rst
const x = 1;
`);
expect(result.cells).toHaveLength(2);
expect(result.cells[0]).toMatchObject({ language: "python", title: "bootstrap", reset: true });
expect(result.cells[1]).toMatchObject({ language: "js", reset: true, title: undefined });
expect(result.cells[0].reset).toBe(false);
expect(result.cells[1].reset).toBe(true);
expect(result.cells[1].language).toBe("js");
});
it("accepts JS, TS, and PY language tokens (case-insensitive)", () => {
const result = parseEvalInput(`*** Begin TS
it("accepts case-insensitive language tokens (lenient)", () => {
const result = parseEvalInput(`*** Cell JS:""
const a = 1;
*** End TS
*** Begin py
*** Cell PY:""
print("py")
*** End py
`);
expect(result.cells.map(c => c.language)).toEqual(["js", "python"]);
expect(result.cells).toHaveLength(2);
expect(result.cells[0].language).toBe("js");
expect(result.cells[1].language).toBe("python");
});
it("parses millisecond, second, and minute durations", () => {
const result = parseEvalInput(`*** Begin PY
*** Timeout: 500ms
const result = parseEvalInput(`*** Cell py:"a" t:500ms
a = 1
*** End PY
*** Begin PY
*** Timeout: 5
*** Cell py:"b" t:5
a = 2
*** End PY
*** Begin PY
*** Timeout: 2m
*** Cell py:"c" t:2m
a = 3
*** End PY
`);
expect(result.cells.map(c => c.timeoutMs)).toEqual([500, 5_000, 120_000]);
});
it("attribute order is flexible and only the first wins", () => {
const result = parseEvalInput(`*** Begin PY
*** Timeout: 1s
*** Title: first
*** Title: ignored
*** Timeout: 9s
print(1)
*** End PY
`);
expect(result.cells[0]).toMatchObject({ title: "first", timeoutMs: 1_000 });
expect(result.cells.map(c => c.timeoutMs)).toEqual([500, 5000, 120_000]);
});
it("preserves blank lines inside the cell body", () => {
const result = parseEvalInput(`*** Begin JS
const result = parseEvalInput(`*** Cell js:""
const x = 1;
const y = 2;
*** End JS
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].code).toBe("const x = 1;\n\nconst y = 2;");
});
it("treats blank lines between cells as separators, not code", () => {
const result = parseEvalInput(`*** Begin PY
const result = parseEvalInput(`*** Cell py:""
print("a")
*** End PY
*** Begin PY
*** Cell py:""
print("b")
*** End PY
`);
expect(result.cells).toHaveLength(2);
expect(result.cells[0].code).toBe('print("a")');
expect(result.cells[1].code).toBe('print("b")');
});
it("falls back to language sniffing when the begin marker has no recognized language", () => {
const result = parseEvalInput(`*** Begin RUBY
it("falls back to language sniffing when the header has no recognized language", () => {
// Bare `ruby` doesn't match LANG_TITLE, but the parser is lenient and
// falls back to body sniffing.
const result = parseEvalInput(`*** Cell ruby:"x"
const x = 1;
console.log(x);
*** End
`);
expect(result.cells[0]).toMatchObject({ language: "js", languageOrigin: "default" });
expect(result.cells).toHaveLength(1);
expect(result.cells[0].languageOrigin).toBe("default");
expect(result.cells[0].language).toBe("js");
});
it("accepts `**Begin` (two stars) as well as `***Begin`", () => {
const result = parseEvalInput(`**Begin PY
it("accepts `**Cell` (two stars) as well as `***Cell`", () => {
const result = parseEvalInput(`**Cell py:""
print(1)
**End
`);
expect(result.cells[0]).toMatchObject({ language: "python", code: "print(1)" });
expect(result.cells).toHaveLength(1);
expect(result.cells[0].language).toBe("python");
});
it("implicitly closes a cell when a new *** Begin appears without an *** End", () => {
const result = parseEvalInput(`*** Begin PY
it("implicitly closes a cell when a new *** Cell appears without *** End", () => {
const result = parseEvalInput(`*** Cell py:""
print("a")
*** Begin JS
*** Cell js:""
const x = 1;
*** End JS
`);
expect(result.cells).toHaveLength(2);
expect(result.cells[0]).toMatchObject({ language: "python", code: 'print("a")' });
expect(result.cells[1]).toMatchObject({ language: "js", code: "const x = 1;" });
expect(result.cells[0].code).toBe('print("a")');
expect(result.cells[1].code).toBe("const x = 1;");
});
it("ignores the language token on `*** End` (leniency)", () => {
const result = parseEvalInput(`*** Begin PY
it("tolerates `*** End` as an optional cell terminator (GPT quirk)", () => {
const result = parseEvalInput(`*** Cell py:""
print(1)
*** End JS
*** End
*** Cell js:""
const x = 1;
*** End
`);
expect(result.cells[0]).toMatchObject({ language: "python", code: "print(1)" });
expect(result.cells).toHaveLength(2);
expect(result.cells[0].code).toBe("print(1)");
expect(result.cells[1].code).toBe("const x = 1;");
});
it("ignores anything trailing `*** End` (leniency)", () => {
const result = parseEvalInput(`*** Cell py:""
print(1)
*** End py
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].code).toBe("print(1)");
});
it("accepts long-form language aliases (Python, JavaScript, TypeScript)", () => {
const result = parseEvalInput(`*** Begin Python
const result = parseEvalInput(`*** Cell Python:""
print(1)
*** End
*** begin javascript
const x = 1;
*** End
*** Cell JavaScript:""
const a = 1;
*** Cell TypeScript:""
const b = 2;
`);
expect(result.cells.map(c => c.language)).toEqual(["python", "js"]);
});
it("tolerates whitespace and case variations on directives", () => {
const result = parseEvalInput(`***\tBegin\tPY
***title: tabby
***\tTimeout:\t250ms
***reset
print(1)
***End
`);
expect(result.cells[0]).toMatchObject({
title: "tabby",
timeoutMs: 250,
reset: true,
language: "python",
code: "print(1)",
});
expect(result.cells.map(c => c.language)).toEqual(["python", "js", "js"]);
});
it("implicitly closes the final cell at EOF when *** End is missing", () => {
const result = parseEvalInput(`*** Begin PY
const result = parseEvalInput(`*** Cell py:""
print(1)
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0]).toMatchObject({ language: "python", code: "print(1)" });
expect(result.cells[0].code).toBe("print(1)");
});
it("treats bare code without any *** Begin as a single implicit cell", () => {
it("treats bare code without any *** Cell as a single implicit cell", () => {
const result = parseEvalInput(`def greet():\n print('hi')\ngreet()\n`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0]).toMatchObject({
language: "python",
languageOrigin: "default",
language: "python",
code: "def greet():\n print('hi')\ngreet()",
});
});
@@ -204,22 +169,107 @@ print(1)
it("rejects invalid duration", () => {
expect(() =>
parseEvalInput(`*** Begin PY
*** Timeout: forever
parseEvalInput(`*** Cell py:"" t:forever
print(1)
*** End PY
`),
).toThrow(/invalid duration/);
});
it("supports titles with embedded spaces", () => {
const result = parseEvalInput(`*** Cell py:"load and validate config"
print(1)
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].title).toBe("load and validate config");
});
it('treats empty title (`py:""`) as no title', () => {
const result = parseEvalInput(`*** Cell py:""
print(1)
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].title).toBeUndefined();
expect(result.cells[0].language).toBe("python");
});
it("accepts bare language token without title (lenient form)", () => {
// Parser is more permissive than the lark; bare `py` is accepted
// even though the canonical form is `py:"title"`.
const result = parseEvalInput(`*** Cell py
print(1)
`);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].language).toBe("python");
expect(result.cells[0].title).toBeUndefined();
});
describe("attribute leniency (accepted but not advertised)", () => {
it("accepts id aliases (title/name/cell/file/label)", () => {
const aliases = ["title", "name", "cell", "file", "label"];
for (const alias of aliases) {
const result = parseEvalInput(`*** Cell py ${alias}:"hi"\nprint(1)\n`);
expect(result.cells[0].title).toBe("hi");
}
});
it("accepts t aliases (timeout/duration/time)", () => {
const aliases = ["timeout", "duration", "time"];
for (const alias of aliases) {
const result = parseEvalInput(`*** Cell py ${alias}:5s\nprint(1)\n`);
expect(result.cells[0].timeoutMs).toBe(5000);
}
});
it("accepts `reset` as an alias for `rst`", () => {
const result = parseEvalInput(`*** Cell py reset\nprint(1)\n`);
expect(result.cells[0].reset).toBe(true);
});
it("accepts `rst:true|false|1|0|yes|no|on|off`", () => {
for (const v of ["true", "1", "yes", "on"]) {
const r = parseEvalInput(`*** Cell py rst:${v}\nx\n`);
expect(r.cells[0].reset).toBe(true);
}
for (const v of ["false", "0", "no", "off"]) {
const r = parseEvalInput(`*** Cell py rst:${v}\nx\n`);
expect(r.cells[0].reset).toBe(false);
}
});
it("rejects an invalid rst value", () => {
expect(() => parseEvalInput(`*** Cell py rst:maybe\nx\n`)).toThrow(/invalid rst/);
});
it("accepts single-quoted titles (`id:'hi'`)", () => {
const result = parseEvalInput(`*** Cell py id:'hello world'\nprint(1)\n`);
expect(result.cells[0].title).toBe("hello world");
});
it("accepts a bare positional duration token (e.g. `30s`)", () => {
const result = parseEvalInput(`*** Cell py 2m\nprint(1)\n`);
expect(result.cells[0].timeoutMs).toBe(120_000);
});
it("first occurrence wins for repeated keys (canonical or alias)", () => {
const result = parseEvalInput(`*** Cell py id:"first" name:"second" t:1s timeout:5s\nprint(1)\n`);
expect(result.cells[0].title).toBe("first");
expect(result.cells[0].timeoutMs).toBe(1000);
});
it("unclassified bare tokens accumulate as a positional title", () => {
const result = parseEvalInput(`*** Cell py setup phase\nprint(1)\n`);
expect(result.cells[0].title).toBe("setup phase");
});
});
describe("*** Abort recovery sentinel (harmony-leak mitigation)", () => {
it("drops the in-progress cell and stops parsing", () => {
const result = parseEvalInput(`*** Begin PY
const result = parseEvalInput(`*** Cell py:""
print("a")
*** End PY
*** Begin JS
*** Cell js:""
const partial = 1; /* contamination starts mid-cell */
*** Abort
*** Begin TS
*** Cell js:""
const never_runs = 1;
`);
expect(result.aborted).toBe(true);
@@ -228,16 +278,19 @@ const never_runs = 1;
expect(result.cells[0].code).toBe('print("a")');
});
it("between cells: keeps preceding cells, sets aborted, drops trailing cells", () => {
const result = parseEvalInput(`*** Begin PY
it("`*** End` before `*** Abort` preserves the closed cell", () => {
// Without `*** End`, the parser can't tell whether the cell was
// complete before contamination — by design, since `*** End` is
// optional and undocumented. Explicit `*** End` is the GPT quirk
// that signals "cell is closed, abort is between cells".
const result = parseEvalInput(`*** Cell py:""
print("a")
*** End PY
*** End
*** Abort
*** Begin PY
*** Cell py:""
print("never")
*** End PY
`);
expect(result.aborted).toBe(true);
expect(result.cells).toHaveLength(1);
@@ -252,53 +305,48 @@ print("never")
expect(result.cells).toHaveLength(0);
});
it("appended sentinel from harmony-leak truncation: abort flag set, prior cell preserved", () => {
// Mirrors the exact shape harmony-leak emits: original input truncated
// at the contaminated line, then "\n*** Abort\n" appended.
const truncated = `*** Begin PY\nprint("ok")\n*** End PY\n*** Abort\n`;
it("appended sentinel from harmony-leak truncation: abort flag set, prior cell dropped", () => {
const truncated = `*** Cell py:""\nprint("ok")\n*** Abort\n`;
const result = parseEvalInput(truncated);
expect(result.aborted).toBe(true);
expect(result.cells).toHaveLength(1);
expect(result.cells[0].code).toBe('print("ok")');
expect(result.cells).toHaveLength(0);
});
it("absent sentinel: aborted is undefined (not falsely set)", () => {
const result = parseEvalInput(`*** Begin PY
const result = parseEvalInput(`*** Cell py:""
print(1)
*** End PY
`);
expect(result.aborted).toBeUndefined();
});
});
it("does not crash on stray non-marker lines between cells", () => {
// Regression: prior to fix, the outer parse loop unconditionally ran
// `BEGIN_RE.exec(lines[i])!` on whatever followed a closed cell, so a
// stray line — common in harmony-leak fragments — threw
// "null is not an object (evaluating 'BEGIN_RE.exec(lines[i])[1]')".
const result = parseEvalInput(`*** Begin PY
// Regression for "null is not an object (evaluating
// 'BEGIN_RE.exec(lines[i])[1]')" — stray fragments must not crash.
// Without `*** End`, the stray junk folds into the prior cell's body;
// the contract for this test is just "don't crash".
const result = parseEvalInput(`*** Cell py:""
print("a")
*** End PY
stray junk that is not a marker
*** Begin PY
*** Cell py:""
print("b")
*** End PY
`);
expect(result.aborted).toBeUndefined();
expect(result.cells).toHaveLength(2);
expect(result.cells[0].code).toBe('print("a")');
expect(result.cells[0].code).toContain('print("a")');
expect(result.cells[1].code).toBe('print("b")');
});
it("does not crash on trailing stray content after the final cell", () => {
const result = parseEvalInput(`*** Begin PY
const result = parseEvalInput(`*** Cell py:""
print(1)
*** End PY
leftover model chatter
more junk
`);
expect(result.aborted).toBeUndefined();
expect(result.cells).toHaveLength(1);
expect(result.cells[0].code).toBe("print(1)");
// Stray lines fold into the cell body (no terminator), which is fine —
// the contract is just "don't crash".
expect(result.cells[0].code).toContain("print(1)");
});
});