feat(coding-agent-eval): added oneshot llm helper and __llm__ bridge
- Added one-shot `llm(prompt, opts)` helpers in JS and Python eval runtimes. - Added `__llm__` eval bridge wiring for synthetic LLM tool dispatch and status/event output. - Added `runEvalLlm` with tier-to-model resolution, effort handling, and oneshot completion execution. - Added structured schema output handling via `respond` tool and JSON fallback parsing. - Documented new llm behavior in eval docs/changelog and added tests for tier mapping and error cases.
This commit is contained in:
@@ -131,6 +131,7 @@ Implemented in `packages/coding-agent/src/eval/js/context-manager.ts` and `packa
|
||||
- `display`, `print`
|
||||
- `read`, `write`, `append`, `sort`, `uniq`, `counter`, `diff`, `tree`, `env`, `output`
|
||||
- `tool.<name>(args)` proxy for arbitrary session tool calls
|
||||
- `llm(prompt, opts)` for oneshot, stateless LLM calls (see _Oneshot LLM helper_ below)
|
||||
- JS helpers are async because they cross the VM/tool boundary
|
||||
- `display(value)` behavior:
|
||||
- plain objects/arrays become JSON outputs
|
||||
@@ -163,6 +164,21 @@ Implemented in `packages/coding-agent/src/eval/py/executor.ts`, `packages/coding
|
||||
- `text/html` → HTML converted to markdown with `htmlToBasicMarkdown()`
|
||||
- Interactive stdin is rejected: `input_request` sends an empty reply, marks `stdinRequested`, and the executor returns exit code `1`
|
||||
|
||||
### Oneshot LLM helper (`llm`)
|
||||
|
||||
Both runtimes expose `llm()` — a single stateless completion against a model tier. It is intentionally minimal: no conversation history, no agent-visible tools, pure text in / text (or object) out. Implemented host-side in `packages/coding-agent/src/eval/llm-bridge.ts` and routed through the existing tool bridge under the reserved name `__llm__`.
|
||||
|
||||
- Signatures:
|
||||
- JS: `await llm(prompt, { model?, system?, schema? })`
|
||||
- Python: `llm(prompt, *, model="default", system=None, schema=None)`
|
||||
- `model` selects a tier (default `"default"`):
|
||||
- `"smol"` → `pi/smol` role (fast / cheap)
|
||||
- `"default"` → the session's active model, falling back to the `pi/default` role
|
||||
- `"slow"` → `pi/slow` role; requests high reasoning effort only on reasoning-capable models
|
||||
- `system` (optional) supplies a system prompt.
|
||||
- `schema` (optional) is a plain JSON-Schema object. When present, the model is forced to call a single synthetic `respond` tool with that schema (loose, non-strict), and the helper returns the parsed object. When absent, the helper returns the completion string.
|
||||
- Errors surface as exceptions: unresolved tier, missing API key, an `error`/`aborted` stop reason, or empty output each raise.
|
||||
|
||||
### Multi-language call behavior
|
||||
|
||||
A single tool call can mix Python and JS cells. Persistence is per language runtime:
|
||||
|
||||
Reference in New Issue
Block a user