feat(coding-agent): implemented dynamic tool activation for autoresearch lifecycle management

- Added `defaultInactive` property to ToolDefinition for conditional tool registration and activation control.
- Added dynamic tool activation/deactivation API `setActiveTools()` for managing experiment tools in autoresearch mode.
- Replaced single `command-start.md` workflow with separate `command-initialize.md` and `command-resume.md` prompts for autoresearch initialization and session resumption.
- Added interactive intent dialog for autoresearch optimization goals with automatic session resumption detection based on autoresearch.md presence.
- Refactored autoresearch command handler to distinguish resume vs initialize flows and dynamically activate/deactivate experiment tools based on mode and session state.
This commit is contained in:
can1357
2026-03-22 21:14:17 +01:00
parent 5d2e2cef1f
commit 66a478e6e6
11 changed files with 463 additions and 22 deletions
+11
View File
@@ -1,6 +1,7 @@
# Changelog
## [Unreleased]
### Breaking Changes
- Changed hashline edit operation types from `replace` (with optional `end`) to explicit `replace_line` and `replace_range` operations
@@ -9,6 +10,11 @@
### Added
- Added `defaultInactive` property to `ToolDefinition` to allow tools to be registered but excluded from the initial active set, with extension responsibility for activation/deactivation
- Added dynamic tool activation/deactivation in autoresearch mode via `setActiveTools()` API
- Added separate initialization and resume workflows for autoresearch with `command-initialize.md` and `command-resume.md` prompts
- Added intent dialog to prompt users for autoresearch optimization goals when starting fresh
- Added automatic detection of existing `autoresearch.md` to resume from previous sessions without re-prompting for intent
- Added autoresearch extension with autonomous experiment loop capabilities
- Added `init_experiment` tool to initialize and reset autoresearch sessions with configurable metrics
- Added `log_experiment` tool to record experiment results with metric parsing and confidence tracking
@@ -30,6 +36,10 @@
### Changed
- Changed autoresearch command to use intent-based initialization instead of goal parameter, with user input dialog for new sessions
- Changed autoresearch startup to activate experiment tools (`init_experiment`, `run_experiment`, `log_experiment`) only when autoresearch mode is enabled
- Changed autoresearch shutdown to deactivate experiment tools when mode is disabled or cleared
- Changed autoresearch session rehydration to dynamically manage experiment tool activation based on session state
- Refactored hashline edit validation to enforce stricter anchor requirements per operation type
- Updated edit application logic to handle explicit file-level operations (`append_eof`, `prepend_bof`) separately from anchor-based operations
- Changed `setWidget` API to accept `ExtensionWidgetOptions` parameter for placement control
@@ -47,6 +57,7 @@
### Removed
- Removed `command-start.md` prompt template in favor of separate initialize and resume workflows
- Removed auto-correction of off-by-one range edits that duplicated closing braces or boundary lines
- Removed `shouldAutocorrect` function and related boundary line deduplication logic from hashline editor
- Removed auto-correction of off-by-one range edits that duplicated closing braces or boundary lines
@@ -0,0 +1,13 @@
Set up autoresearch for this intent:
{{intent}}
Explain briefly what autoresearch will do in this repository, then initialize the workspace.
Your first actions:
- write `autoresearch.md`
- define the benchmark entrypoint in `autoresearch.sh`
- optionally add `autoresearch.checks.sh` if correctness or quality needs a hard gate
- run `init_experiment`
- run and log the baseline
- keep iterating until interrupted or until the configured iteration cap is reached
@@ -0,0 +1,9 @@
Resume autoresearch from the attached notes.
@{{autoresearch_md_path}}
Use the notes as the source of truth for the current direction.
- inspect recent git history for context
- inspect `autoresearch.jsonl` if it exists
- continue the most promising unfinished branch
- keep iterating until interrupted or until the configured iteration cap is reached
@@ -1,10 +0,0 @@
Autoresearch mode is active.
Goal:
{{goal}}
Start or resume the autoresearch loop now.
- Read `autoresearch.md` if it already exists.
- Otherwise create the autoresearch workspace, initialize the experiment, run a baseline, and keep iterating.
- Continue until interrupted or until the configured iteration cap is reached.
+52 -12
View File
@@ -3,7 +3,8 @@ import * as path from "node:path";
import type { AutocompleteItem } from "@oh-my-pi/pi-tui";
import { renderPromptTemplate } from "../config/prompt-templates";
import type { ExtensionContext, ExtensionFactory } from "../extensibility/extensions";
import commandStartTemplate from "./command-start.md" with { type: "text" };
import commandInitializeTemplate from "./command-initialize.md" with { type: "text" };
import commandResumeTemplate from "./command-resume.md" with { type: "text" };
import { createDashboardController } from "./dashboard";
import { readMaxExperiments, resolveWorkDir, validateWorkDir } from "./helpers";
import promptTemplate from "./prompt.md" with { type: "text" };
@@ -22,6 +23,7 @@ import type { AutoresearchRuntime } from "./types";
const AUTORESUME_INTERVAL_MS = 5 * 60 * 1000;
const MAX_AUTORESUME_TURNS = 20;
const EXPERIMENT_TOOL_NAMES = ["init_experiment", "run_experiment", "log_experiment"];
export const createAutoresearchExtension: ExtensionFactory = api => {
const runtimeStore = createRuntimeStore();
@@ -30,7 +32,7 @@ export const createAutoresearchExtension: ExtensionFactory = api => {
const getSessionKey = (ctx: ExtensionContext): string => ctx.sessionManager.getSessionId();
const getRuntime = (ctx: ExtensionContext): AutoresearchRuntime => runtimeStore.ensure(getSessionKey(ctx));
const rehydrate = (ctx: ExtensionContext): void => {
const rehydrate = async (ctx: ExtensionContext): Promise<void> => {
const runtime = getRuntime(ctx);
const workDir = resolveWorkDir(ctx.cwd);
const reconstructed = reconstructStateFromJsonl(workDir);
@@ -47,6 +49,17 @@ export const createAutoresearchExtension: ExtensionFactory = api => {
runtime.lastRunAsi = null;
runtime.runningExperiment = null;
dashboard.updateWidget(ctx, runtime);
const activeTools = api.getActiveTools();
const experimentTools = new Set(EXPERIMENT_TOOL_NAMES);
const nextActiveTools = runtime.autoresearchMode
? [...new Set([...activeTools, ...EXPERIMENT_TOOL_NAMES])]
: activeTools.filter(name => !experimentTools.has(name));
const toolsChanged =
nextActiveTools.length !== activeTools.length ||
nextActiveTools.some((name, index) => name !== activeTools[index]);
if (toolsChanged) {
await api.setActiveTools(nextActiveTools);
}
};
const setMode = (
@@ -86,15 +99,13 @@ export const createAutoresearchExtension: ExtensionFactory = api => {
return;
}
if (trimmed.length === 0) {
ctx.ui.notify("Usage: /autoresearch <goal> | off | clear", "info");
return;
}
if (trimmed === "off") {
setMode(ctx, false, runtime.goal, "off");
runtime.experimentsThisSession = 0;
runtime.autoResumeTurns = 0;
dashboard.updateWidget(ctx, runtime);
const experimentTools = new Set(EXPERIMENT_TOOL_NAMES);
await api.setActiveTools(api.getActiveTools().filter(name => !experimentTools.has(name)));
ctx.ui.notify("Autoresearch mode disabled", "info");
return;
}
@@ -109,19 +120,48 @@ export const createAutoresearchExtension: ExtensionFactory = api => {
runtime.goal = null;
setMode(ctx, false, null, "clear");
dashboard.updateWidget(ctx, runtime);
const experimentTools = new Set(EXPERIMENT_TOOL_NAMES);
await api.setActiveTools(api.getActiveTools().filter(name => !experimentTools.has(name)));
ctx.ui.notify("Autoresearch log cleared", "info");
return;
}
setMode(ctx, true, trimmed, "on");
const workDir = resolveWorkDir(ctx.cwd);
const autoresearchMdPath = path.join(workDir, "autoresearch.md");
const hasAutoresearchMd = fs.existsSync(autoresearchMdPath);
if (hasAutoresearchMd) {
setMode(ctx, true, runtime.goal, "on");
runtime.experimentsThisSession = 0;
runtime.autoResumeTurns = 0;
dashboard.updateWidget(ctx, runtime);
await api.setActiveTools([...new Set([...api.getActiveTools(), ...EXPERIMENT_TOOL_NAMES])]);
api.sendUserMessage(
renderPromptTemplate(commandResumeTemplate, {
autoresearch_md_path: autoresearchMdPath,
}),
);
return;
}
const intentInput = await ctx.ui.input(
"Autoresearch Intent",
trimmed || runtime.goal || "what should autoresearch improve?",
);
if (intentInput === undefined) return;
const intent = intentInput.trim();
if (intent.length === 0) {
ctx.ui.notify("Autoresearch intent is required", "info");
return;
}
setMode(ctx, true, intent, "on");
runtime.experimentsThisSession = 0;
runtime.autoResumeTurns = 0;
dashboard.updateWidget(ctx, runtime);
api.sendUserMessage(
renderPromptTemplate(commandStartTemplate, {
goal: trimmed,
}),
);
await api.setActiveTools([...new Set([...api.getActiveTools(), ...EXPERIMENT_TOOL_NAMES])]);
api.sendUserMessage(renderPromptTemplate(commandInitializeTemplate, { intent }));
},
});
@@ -41,6 +41,7 @@ export function createInitExperimentTool(
description:
"Initialize or reset the autoresearch session for the current optimization target before the first logged run of a segment.",
parameters: initExperimentSchema,
defaultInactive: true,
async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
const workDirError = validateWorkDir(ctx.cwd);
if (workDirError) {
@@ -74,6 +74,7 @@ export function createLogExperimentTool(
description:
"Log the experiment result, update dashboard state, persist JSONL history, and apply git keep or revert behavior.",
parameters: logExperimentSchema,
defaultInactive: true,
async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
const workDirError = validateWorkDir(ctx.cwd);
if (workDirError) {
@@ -61,6 +61,7 @@ export function createRunExperimentTool(
description:
"Run an experiment command with timing, tail capture, structured metric parsing, and optional autoresearch.checks.sh validation.",
parameters: runExperimentSchema,
defaultInactive: true,
async execute(_toolCallId, params, signal, onUpdate, ctx) {
const workDirError = validateWorkDir(ctx.cwd);
if (workDirError) {
@@ -307,6 +307,9 @@ export interface ToolDefinition<TParams extends TSchema = TSchema, TDetails = un
parameters: TParams;
/** If true, tool is excluded unless explicitly listed in --tools or agent's tools field */
hidden?: boolean;
/** If true, tool is registered but not auto-included in the initial active set.
* The registering extension is responsible for activating/deactivating it via setActiveTools(). */
defaultInactive?: boolean;
/** If true, tool may stage deferred changes that require explicit resolve/discard. */
deferrable?: boolean;
/** MCP server name for discovery/search metadata when this tool fronts an MCP server. */
@@ -4,7 +4,16 @@ import * as os from "node:os";
import * as path from "node:path";
import { Snowflake } from "@oh-my-pi/pi-utils";
import { isAutoresearchShCommand } from "../src/autoresearch/helpers";
import { createAutoresearchExtension } from "../src/autoresearch/index";
import { reconstructStateFromJsonl } from "../src/autoresearch/state";
import type {
ExtensionAPI,
ExtensionCommandContext,
ExtensionContext,
RegisteredCommand,
SessionStartEvent,
SessionSwitchEvent,
} from "../src/extensibility/extensions";
function makeTempDir(): string {
const dir = path.join(os.tmpdir(), `pi-autoresearch-test-${Snowflake.next()}`);
@@ -105,3 +114,258 @@ describe("autoresearch command guard", () => {
expect(isAutoresearchShCommand("bash -lc 'autoresearch.sh'")).toBe(false);
});
});
interface AutoresearchCommandHarness {
command: RegisteredCommand;
ctx: ExtensionCommandContext;
sentMessages: string[];
inputCalls: Array<{ title: string; placeholder: string | undefined }>;
notifications: Array<{ message: string; type: "info" | "warning" | "error" | undefined }>;
}
function createAutoresearchCommandHarness(cwd: string, inputResult: string | undefined): AutoresearchCommandHarness {
const sentMessages: string[] = [];
const inputCalls: Array<{ title: string; placeholder: string | undefined }> = [];
const notifications: Array<{ message: string; type: "info" | "warning" | "error" | undefined }> = [];
let command: RegisteredCommand | undefined;
const api = {
appendEntry(_customType: string, _data?: unknown): void {},
on(): void {},
registerCommand(name: string, options: Omit<RegisteredCommand, "name">): void {
command = { name, ...options };
},
registerShortcut(): void {},
registerTool(): void {},
getActiveTools(): string[] {
return [];
},
setActiveTools: async (_toolNames: string[]): Promise<void> => {},
sendUserMessage(content: string | unknown[]): void {
if (typeof content !== "string") {
throw new Error("Expected autoresearch command to send plain text");
}
sentMessages.push(content);
},
} as unknown as ExtensionAPI;
createAutoresearchExtension(api);
if (!command) throw new Error("Expected autoresearch command to register");
const ctx = {
abort(): void {},
branch: async () => ({ cancelled: false }),
compact: async () => {},
cwd,
getContextUsage: () => undefined,
hasUI: false,
isIdle: () => true,
model: undefined,
modelRegistry: {},
newSession: async () => ({ cancelled: false }),
reload: async () => {},
sessionManager: {
getEntries: () => [],
getSessionId: () => "session-1",
},
switchSession: async () => ({ cancelled: false }),
navigateTree: async () => ({ cancelled: false }),
ui: {
confirm: async () => false,
custom: async () => undefined,
input: async (title: string, placeholder?: string) => {
inputCalls.push({ title, placeholder });
return inputResult;
},
notify(message: string, type?: "info" | "warning" | "error"): void {
notifications.push({ message, type });
},
onTerminalInput: () => () => {},
select: async () => undefined,
setFooter(): void {},
setHeader(): void {},
setStatus(): void {},
setTitle(): void {},
setWidget(): void {},
setWorkingMessage(): void {},
},
waitForIdle: async () => {},
} as unknown as ExtensionCommandContext;
return { command, ctx, sentMessages, inputCalls, notifications };
}
interface AutoresearchLifecycleHarness {
sessionStartHandler: ((event: SessionStartEvent, ctx: ExtensionContext) => Promise<void> | void) | undefined;
sessionSwitchHandler: ((event: SessionSwitchEvent, ctx: ExtensionContext) => Promise<void> | void) | undefined;
ctx: ExtensionContext;
setActiveToolsCalls: string[][];
}
function createAutoresearchLifecycleHarness(options: {
activeTools: string[];
controlEntries?: Array<{ type: "custom"; customType: string; data?: unknown }>;
}): AutoresearchLifecycleHarness {
const handlers = new Map<string, (...args: unknown[]) => Promise<void> | void>();
const activeTools = [...options.activeTools];
const setActiveToolsCalls: string[][] = [];
const api = {
appendEntry(_customType: string, _data?: unknown): void {},
on(event: string, handler: (...args: unknown[]) => Promise<void> | void): void {
handlers.set(event, handler);
},
registerCommand(): void {},
registerShortcut(): void {},
registerTool(): void {},
getActiveTools(): string[] {
return [...activeTools];
},
async setActiveTools(toolNames: string[]): Promise<void> {
setActiveToolsCalls.push([...toolNames]);
activeTools.splice(0, activeTools.length, ...toolNames);
},
sendUserMessage(): void {},
} as unknown as ExtensionAPI;
createAutoresearchExtension(api);
const ctx = {
abort(): void {},
compact: async () => {},
cwd: makeTempDir(),
getContextUsage: () => undefined,
hasUI: false,
hasPendingMessages: () => false,
isIdle: () => true,
model: undefined,
modelRegistry: {},
sessionManager: {
getEntries: () => options.controlEntries ?? [],
getSessionId: () => "session-1",
},
shutdown: async () => {},
ui: {
confirm: async () => false,
custom: async () => undefined,
editor: async () => undefined,
getEditorText: () => "",
input: async () => undefined,
notify(): void {},
onTerminalInput: () => () => {},
select: async () => undefined,
setEditorComponent(): void {},
setEditorText(): void {},
setFooter(): void {},
setHeader(): void {},
setStatus(): void {},
setTheme: async () => false,
setTitle(): void {},
setToolsExpanded(): void {},
setWidget(): void {},
setWorkingMessage(): void {},
},
} as unknown as ExtensionContext;
return {
sessionStartHandler: handlers.get("session_start") as
| ((event: SessionStartEvent, ctx: ExtensionContext) => Promise<void> | void)
| undefined,
sessionSwitchHandler: handlers.get("session_switch") as
| ((event: SessionSwitchEvent, ctx: ExtensionContext) => Promise<void> | void)
| undefined,
ctx,
setActiveToolsCalls,
};
}
describe("autoresearch command startup", () => {
const tempDirs: string[] = [];
afterEach(() => {
for (const dir of tempDirs.splice(0)) {
fs.rmSync(dir, { recursive: true, force: true });
}
});
it("asks for intent and sends an initialization prompt when no autoresearch.md exists", async () => {
const dir = makeTempDir();
tempDirs.push(dir);
const harness = createAutoresearchCommandHarness(dir, "reduce edit benchmark runtime variance");
await harness.command.handler("", harness.ctx);
expect(harness.inputCalls).toEqual([
{ title: "Autoresearch Intent", placeholder: "what should autoresearch improve?" },
]);
expect(harness.sentMessages).toHaveLength(1);
expect(harness.sentMessages[0]).toContain("Set up autoresearch for this intent:");
expect(harness.sentMessages[0]).toContain("reduce edit benchmark runtime variance");
expect(harness.sentMessages[0]).toContain("Explain briefly what autoresearch will do in this repository");
expect(harness.notifications).toEqual([]);
});
it("resumes from autoresearch.md without asking for intent when notes already exist", async () => {
const dir = makeTempDir();
tempDirs.push(dir);
const autoresearchMdPath = path.join(dir, "autoresearch.md");
fs.writeFileSync(autoresearchMdPath, "# Autoresearch\n\nExisting notes\n");
const harness = createAutoresearchCommandHarness(dir, "ignored");
await harness.command.handler("", harness.ctx);
expect(harness.inputCalls).toEqual([]);
expect(harness.sentMessages).toEqual([
[
"Resume autoresearch from the attached notes.",
"",
`@${autoresearchMdPath}`,
"",
"Use the notes as the source of truth for the current direction.",
"- inspect recent git history for context",
"- inspect `autoresearch.jsonl` if it exists",
"- continue the most promising unfinished branch",
"- keep iterating until interrupted or until the configured iteration cap is reached",
].join("\n"),
]);
});
it("does not start autoresearch when the intent dialog returns blank input", async () => {
const dir = makeTempDir();
tempDirs.push(dir);
const harness = createAutoresearchCommandHarness(dir, " ");
await harness.command.handler("", harness.ctx);
expect(harness.sentMessages).toEqual([]);
expect(harness.notifications).toEqual([{ message: "Autoresearch intent is required", type: "info" }]);
});
});
describe("autoresearch lifecycle tool activation", () => {
it("activates experiment tools when rehydrating an autoresearch session", async () => {
const harness = createAutoresearchLifecycleHarness({
activeTools: ["read", "write"],
controlEntries: [{ type: "custom", customType: "autoresearch-control", data: { mode: "on", goal: "speed" } }],
});
if (!harness.sessionStartHandler) throw new Error("Expected session_start handler");
await harness.sessionStartHandler({ type: "session_start" }, harness.ctx);
expect(harness.setActiveToolsCalls).toEqual([
["read", "write", "init_experiment", "run_experiment", "log_experiment"],
]);
});
it("removes experiment tools when rehydrating a non-autoresearch session", async () => {
const harness = createAutoresearchLifecycleHarness({
activeTools: ["read", "init_experiment", "run_experiment", "log_experiment"],
});
if (!harness.sessionSwitchHandler) throw new Error("Expected session_switch handler");
await harness.sessionSwitchHandler(
{ type: "session_switch", reason: "resume", previousSessionFile: "/tmp/previous.jsonl" },
harness.ctx,
);
expect(harness.setActiveToolsCalls).toEqual([["read"]]);
});
});
@@ -0,0 +1,108 @@
import { afterEach, describe, expect, it } from "bun:test";
import * as fs from "node:fs";
import * as os from "node:os";
import * as path from "node:path";
import { getBundledModel } from "@oh-my-pi/pi-ai";
import { Settings } from "@oh-my-pi/pi-coding-agent/config/settings";
import { createAgentSession, type ExtensionFactory } from "@oh-my-pi/pi-coding-agent/sdk";
import { SessionManager } from "@oh-my-pi/pi-coding-agent/session/session-manager";
import { Snowflake } from "@oh-my-pi/pi-utils";
import { Type } from "@sinclair/typebox";
const toolActivationExtension: ExtensionFactory = pi => {
pi.registerTool({
name: "default_inactive_tool",
label: "Default Inactive Tool",
description: "Tool hidden from the initial active set unless explicitly requested.",
parameters: Type.Object({}),
defaultInactive: true,
async execute() {
return { content: [{ type: "text", text: "inactive" }] };
},
});
pi.registerTool({
name: "default_active_tool",
label: "Default Active Tool",
description: "Tool included in the initial active set.",
parameters: Type.Object({}),
async execute() {
return { content: [{ type: "text", text: "active" }] };
},
});
};
describe("createAgentSession defaultInactive tool activation", () => {
const tempDirs: string[] = [];
afterEach(() => {
for (const tempDir of tempDirs.splice(0)) {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it("excludes defaultInactive extension tools from the initial active set unless explicitly requested", async () => {
const tempDir = path.join(os.tmpdir(), `pi-sdk-tool-activation-${Snowflake.next()}`);
tempDirs.push(tempDir);
fs.mkdirSync(tempDir, { recursive: true });
const { session } = await createAgentSession({
cwd: tempDir,
agentDir: tempDir,
sessionManager: SessionManager.inMemory(),
settings: Settings.isolated(),
model: getBundledModel("openai", "gpt-4o-mini"),
disableExtensionDiscovery: true,
extensions: [toolActivationExtension],
skills: [],
contextFiles: [],
promptTemplates: [],
slashCommands: [],
enableMCP: false,
enableLsp: false,
});
try {
expect(session.getAllToolNames()).toEqual(
expect.arrayContaining(["default_active_tool", "default_inactive_tool"]),
);
expect(session.getActiveToolNames()).toContain("default_active_tool");
expect(session.getActiveToolNames()).not.toContain("default_inactive_tool");
expect(session.systemPrompt).toContain("default_active_tool");
expect(session.systemPrompt).not.toContain("default_inactive_tool");
} finally {
await session.dispose();
}
});
it("allows explicitly requested defaultInactive extension tools into the initial active set", async () => {
const tempDir = path.join(os.tmpdir(), `pi-sdk-tool-activation-${Snowflake.next()}`);
tempDirs.push(tempDir);
fs.mkdirSync(tempDir, { recursive: true });
const { session } = await createAgentSession({
cwd: tempDir,
agentDir: tempDir,
sessionManager: SessionManager.inMemory(),
settings: Settings.isolated(),
model: getBundledModel("openai", "gpt-4o-mini"),
disableExtensionDiscovery: true,
extensions: [toolActivationExtension],
skills: [],
contextFiles: [],
promptTemplates: [],
slashCommands: [],
enableMCP: false,
enableLsp: false,
toolNames: ["read", "default_inactive_tool"],
});
try {
expect(session.getActiveToolNames()).toEqual(
expect.arrayContaining(["read", "default_active_tool", "default_inactive_tool"]),
);
expect(session.systemPrompt).toContain("default_inactive_tool");
} finally {
await session.dispose();
}
});
});