feat(coding-agent): added ARIA snapshot support for browser tools

- Implemented `tab.ariaSnapshot()` to capture and represent page structures as ARIA-tree YAML.
- Introduced `tab.ref()` and ref-based selector parsing to enable precise element interaction via unique ARIA identifiers.
- Integrated automated script bundling for cross-environment evaluation of ARIA snapshot logic.
- Updated browser action methods to resolve and target elements using ARIA-ref handles.

docs/provider-streaming-internals.md

@@ -104,7 +104,7 @@ Tool-call argument streaming:
 Shared behavior for Anthropic/OpenAI Responses uses `parseStreamingJson()` / `parseStreamingJsonThrottled()` (`packages/ai/src/utils/json-parse.ts`):
 
 1. try `JSON.parse`
-2. fallback to `repairJson()` + the `partial-json` parser for incomplete fragments
+2. fallback to the in-house `RelaxedJson` parser (relaxed/repairing) for incomplete fragments
 3. if both fail, return `{}`
 
 Implications:

docs/tools/browser.md

@@ -14,6 +14,9 @@
   - `packages/coding-agent/src/tools/browser/attach.ts` — CDP attach/reuse, target picking, spawned-app process handling.
   - `packages/coding-agent/src/tools/browser/tab-protocol.ts` — worker init/run/result message schema.
   - `packages/coding-agent/src/tools/browser/readable.ts` — `tab.extract()` readability extraction.
+  - `packages/coding-agent/src/tools/browser/aria/aria-snapshot.ts` — `captureAriaSnapshot()` (puppeteer/CDP path) and `buildAriaSnapshotScript()` (cmux path); imports the committed `aria-snapshot.bundle.txt`.
+  - `packages/coding-agent/src/tools/browser/aria/aria-snapshot.bundle.txt` — generated, committed artifact: Playwright's injected ARIA-snapshot sources (Apache-2.0, (c) Microsoft; ARIA tree + W3C accessible-name computation) bundled to a CJS module. Upstream sources are not vendored into the repo.
+  - `packages/coding-agent/scripts/generate-aria-snapshot.ts` — fetches the pinned Playwright sources to a temp dir and bundles them into `aria-snapshot.bundle.txt` (CJS, browser target). Dev-time, network-bound; only the bundle is committed.
   - `packages/coding-agent/src/tools/browser/cmux/rpc.ts` — cmux browser-kind resolution plus snapshot/eval/wait-state helpers for the cmux backend.
   - `packages/coding-agent/src/tools/browser/cmux/socket-client.ts` — `CmuxSocketClient`: JSON-RPC over the cmux unix socket.
   - `packages/coding-agent/src/tools/browser/cmux/cmux-tab.ts` — `CmuxTab` surface helper API and `runCmuxCode()` execution path.
@@ -120,6 +123,8 @@ The tool returns one result per call; no streaming partial output is emitted fro
    - `tab.title(): Promise<string>`
    - `tab.goto(url, { waitUntil? })`
    - `tab.observe({ includeAll?, viewportOnly? })`
+   - `tab.ariaSnapshot(selector?, { depth?, boxes? })`
+   - `tab.ref(id)`
    - `tab.screenshot({ selector?, fullPage?, save?, silent? })`
    - `tab.extract(format = "markdown")`
    - `tab.click(selector)`
@@ -136,9 +141,12 @@ The tool returns one result per call; no streaming partial output is emitted fro
    - `tab.waitForUrl(pattern, { timeout? })`
    - `tab.waitForResponse(pattern, { timeout? })`
    - `tab.id(n)`
+   - `tab.ref(id)`
 14. Selector handling in `normalizeSelector()` accepts plain CSS and Puppeteer query handlers, and rewrites legacy Playwright-style prefixes `p-text/`, `p-xpath/`, `p-pierce/`, `p-aria/`; other `p-*` prefixes throw a `ToolError`.
 15. `tab.observe()` clears the element cache, takes a Puppeteer accessibility snapshot, filters to interactive nodes unless `includeAll`, optionally filters to viewport-visible nodes, assigns numeric ids, caches `ElementHandle`s, and returns URL/title/viewport/scroll metadata plus `elements`.
+15a. `tab.ariaSnapshot()` resolves the optional `selector` (via `normalizeSelector()` → `page.$`, defaulting to the whole document) and runs the generated Playwright ARIA-snapshot bundle (`src/tools/browser/aria/aria-snapshot.bundle.txt`) via `captureAriaSnapshot()`. The bundle is wrapped in a `new Function` built worker-side (so page CSP never applies) and serialized to a CDP `page.evaluate` in the page's **main world**, returning Playwright-format YAML. It always runs in `ai` mode: every node gets a `[ref=eN]` id, clickables get `[cursor=pointer]`, and matched DOM nodes are tagged with an `_ariaRef` expando. Existing `_ariaRef` expandos are cleared before each snapshot so ids renumber deterministically from e1 (the fresh module's counter resets each call); refs stay valid until the next snapshot. The cmux backend uses `buildAriaSnapshotScript()` over `browser.eval` instead (no `ElementHandle`; CSS selectors only for the root).
 16. `tab.id(n)` resolves the cached `ElementHandle`, verifies `el.isConnected`, and throws a stale-id error after cache invalidation if the DOM changed or the cache was cleared.
+16a. `tab.ref(id)` resolves a `[ref=eN]` id from the latest `ariaSnapshot()` to a live `ElementHandle` via `resolveAriaRefHandle()` (`page.evaluateHandle` in the main world, walking the document + shadow roots for the matching `_ariaRef`), throwing if no element matches; it accepts a bare `eN` or a prefixed form. For inline selector use, `parseAriaRefSelector()` recognizes only the explicit `aria-ref=eN` / `aria-ref/eN` / `ariaref/eN` forms inside `tab.click/type/fill/waitFor/scrollIntoView` — a bare `eN` is intentionally rejected there so it does not collide with cmux's native observe ids. The cmux backend resolves the same explicit forms through its `aria-ref` `SelectorSpec` kind in `findElement`.
 17. `tab.goto()` clears the cached element ids before navigating. Any new `tab.observe()` also clears and rebuilds the cache.
 18. `tab.click()` uses a custom retry loop for `text/...` selectors to find an actionable visible match; other selectors use `page.locator(...).click()` with the run timeout.
 19. `tab.screenshot()` captures either the whole page or a selector PNG, downsizes a copy for model output, chooses a persistence path, writes the image to disk, records metadata, and optionally emits text + image display entries.

packages/coding-agent/CHANGELOG.md

@@ -1,6 +1,10 @@
 # Changelog
 
 ## [Unreleased]
+### Added
+
+- Added `tab.ariaSnapshot(selector?)` to the browser tool for Playwright-format ARIA-tree YAML
+- Added `tab.ref("e5")` and support for `aria-ref=e5` selectors in all `tab` action methods
 
 ## [16.1.9] - 2026-06-21
 
@@ -12232,4 +12236,4 @@ Initial public release.
 
 ## [0.7.6] - 2025-11-13
 
-Previous releases did not maintain a changelog.
+Previous releases did not maintain a changelog.

packages/coding-agent/scripts/generate-aria-snapshot.ts

@@ -0,0 +1,134 @@
+#!/usr/bin/env bun
+/**
+ * Regenerates the committed browser asset
+ *
+ *   src/tools/browser/aria/aria-snapshot.bundle.txt   ← bundled CJS module
+ *
+ * by fetching Playwright's injected ARIA-snapshot sources (pinned to
+ * PLAYWRIGHT_TAG), wrapping them with a small entry, and bundling — all in a
+ * throwaway temp dir. Only the bundle is committed; the upstream sources are NOT
+ * vendored into the repo (no shipping both source + generated copies). This is a
+ * dev-time, network-bound step, exactly like `generate-models`.
+ *
+ * The tab worker imports the `.txt` with `{ type: "text" }`, wraps it in a
+ * `new Function` worker-side, and runs it via puppeteer's CDP evaluate (it
+ * installs nothing on `window`). The committed output means binary and source
+ * installs need no network or build step at runtime.
+ *
+ * Usage: bun scripts/generate-aria-snapshot.ts
+ */
+import * as fs from "node:fs/promises";
+import * as os from "node:os";
+import * as path from "node:path";
+
+const PLAYWRIGHT_TAG = "v1.61.0";
+const RAW_BASE = `https://raw.githubusercontent.com/microsoft/playwright/${PLAYWRIGHT_TAG}/packages`;
+
+const OUTPUT = path.join(import.meta.dir, "..", "src", "tools", "browser", "aria", "aria-snapshot.bundle.txt");
+
+// Upstream source path -> temp path (relative to the temp root).
+const VENDOR_FILES: Array<[string, string]> = [
+	["injected/src/ariaSnapshot.ts", "injected/ariaSnapshot.ts"],
+	["injected/src/roleUtils.ts", "injected/roleUtils.ts"],
+	["injected/src/domUtils.ts", "injected/domUtils.ts"],
+	["isomorphic/ariaSnapshot.ts", "isomorphic/ariaSnapshot.ts"],
+	["isomorphic/stringUtils.ts", "isomorphic/stringUtils.ts"],
+	["isomorphic/cssTokenizer.ts", "isomorphic/cssTokenizer.ts"],
+	["isomorphic/yaml.ts", "isomorphic/yaml.ts"],
+];
+
+// Entry wrapping the upstream modules. Always runs Playwright's `ai` mode so every
+// node carries a `[ref=eN]` id; matched nodes get an `_ariaRef` expando. Existing
+// expandos are cleared first so the fresh module's counter renumbers from e1
+// deterministically (refs are valid until the next snapshot). Installs nothing on
+// `window`.
+const ENTRY_SOURCE = `
+import { generateAriaTree, renderAriaTree } from "./injected/ariaSnapshot";
+
+export interface AriaSnapshotRequest {
+	depth?: number;
+	boxes?: boolean;
+}
+
+function walkElements(fn: (el: Element) => void): void {
+	const walk = (root: { querySelectorAll(s: string): ArrayLike<Element> }): void => {
+		for (const el of Array.from(root.querySelectorAll("*"))) {
+			fn(el);
+			const shadow = (el as Element & { shadowRoot?: { querySelectorAll(s: string): ArrayLike<Element> } | null }).shadowRoot;
+			if (shadow) walk(shadow);
+		}
+	};
+	walk(document as unknown as { querySelectorAll(s: string): ArrayLike<Element> });
+}
+type RefElement = Element & { _ariaRef?: { role: string; name: string; ref: string } };
+
+export function ariaSnapshot(root: Element | null, request: AriaSnapshotRequest = {}): string {
+	walkElements(el => {
+		if ((el as RefElement)._ariaRef) delete (el as RefElement)._ariaRef;
+	});
+	const target = root ?? document.body ?? document.documentElement;
+	const options = { mode: "ai", depth: request.depth, boxes: request.boxes } as const;
+	const tree = generateAriaTree(target, options);
+	return renderAriaTree(tree, options).text;
+}
+
+export function resolveAriaRef(ref: string): Element | null {
+	let found: Element | null = null;
+	walkElements(el => {
+		if (!found && (el as RefElement)._ariaRef?.ref === ref) found = el;
+	});
+	return found;
+}
+`;
+
+async function main(): Promise<void> {
+	const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "omp-aria-"));
+	try {
+		// Fetch pinned upstream sources into the temp dir.
+		for (const [src, dst] of VENDOR_FILES) {
+			const url = `${RAW_BASE}/${src}`;
+			const res = await fetch(url);
+			if (!res.ok) throw new Error(`Failed to fetch ${url}: ${res.status}`);
+			await Bun.write(path.join(tmp, dst), await res.text());
+		}
+		const entry = path.join(tmp, "entry.ts");
+		await Bun.write(entry, ENTRY_SOURCE);
+
+		// The injected sources import isomorphic modules via the `@isomorphic/*`
+		// alias and the `yaml` package (type-only). Resolve the alias to the fetched
+		// copies and stub `yaml` (only referenced from erased `import type`).
+		const aliasPlugin: Bun.BunPlugin = {
+			name: "aria-vendor-alias",
+			setup(build) {
+				build.onResolve({ filter: /^@isomorphic\// }, args => ({
+					path: path.join(tmp, "isomorphic", `${args.path.slice("@isomorphic/".length)}.ts`),
+				}));
+				build.onResolve({ filter: /^yaml$/ }, () => ({ path: "yaml", namespace: "aria-yaml-stub" }));
+				build.onLoad({ filter: /.*/, namespace: "aria-yaml-stub" }, () => ({
+					contents: "export {};",
+					loader: "ts",
+				}));
+			},
+		};
+
+		const result = await Bun.build({
+			entrypoints: [entry],
+			target: "browser",
+			format: "cjs",
+			minify: true,
+			plugins: [aliasPlugin],
+		});
+		if (!result.success) {
+			for (const log of result.logs) console.error(log);
+			throw new Error("aria snapshot bundle failed");
+		}
+		const code = await result.outputs[0].text();
+		const header = `// @generated by scripts/generate-aria-snapshot.ts from Playwright ${PLAYWRIGHT_TAG}\n// Bundled from Playwright's injected ARIA-snapshot sources (Apache-2.0, (c) Microsoft).\n// Do not edit by hand. Regenerate with: bun scripts/generate-aria-snapshot.ts\n`;
+		await Bun.write(OUTPUT, header + code);
+		console.log(`bundled ${path.relative(process.cwd(), OUTPUT)} (${code.length}b)`);
+	} finally {
+		await fs.rm(tmp, { recursive: true, force: true });
+	}
+}
+
+await main();

packages/coding-agent/src/prompts/tools/browser.md

@@ -15,6 +15,8 @@ Drives real Chromium tab; full puppeteer access via JS.
 - `tab` helpers; drop to raw puppeteer `page` for anything uncovered:
   - `tab.goto(url, { waitUntil? })` — navigate.
   - `tab.observe({ includeAll?, viewportOnly? })` — accessibility snapshot: `{ url, title, viewport, scroll, elements: [{ id, role, name, value, states, … }] }`. Ids stable until next observe/goto.
+  - `tab.ariaSnapshot(selector?, { depth?, boxes? })` — Playwright-format ARIA-tree YAML (nested roles + accessible names + `/url`/`/placeholder`), scoped to `selector` or the whole document. Every node carries a `[ref=eN]` id; `[cursor=pointer]` flags clickables. Captures dense, hierarchical structure/text that `observe()`'s flat list flattens away. Refs renumber from e1 each call and stay valid until the next `ariaSnapshot()`.
+  - `tab.ref("e5")` — `[ref=eN]` from the last ariaSnapshot → element handle with the common action methods (`.click()`, `.type()`, `.fill()`, `.hover()`, `.evaluate()`, …); the primary way to act on a ref. For convenience `aria-ref=e5` also works inline in `tab.click`/`type`/`fill`/`waitFor`/`scrollIntoView` (e.g. `tab.click("aria-ref=e5")`).
   - `tab.id(n)` — id from last observe → `ElementHandle` (`.click()`, `.type()`, …).
   - `tab.click(selector)` / `tab.type(selector, text)` / `tab.fill(selector, value)` / `tab.press(key, { selector? })` / `tab.scroll(dx, dy)`.
   - `tab.waitFor(selector)` — wait until attached; returns `ElementHandle`.

packages/coding-agent/src/tools/browser.ts

@@ -16,6 +16,11 @@ import { ToolAbortError, ToolError, throwIfAborted } from "./tool-errors";
 import { toolResult } from "./tool-result";
 import { clampTimeout } from "./tool-timeouts";
 
+export {
+	type AriaSnapshotOptions,
+	buildAriaSnapshotScript,
+	parseAriaRefSelector,
+} from "./browser/aria/aria-snapshot";
 export { cmuxSnapshotToObservation, mapWaitUntil, resolveCmuxKind, serializeEval } from "./browser/cmux/rpc";
 export { CmuxSocketClient } from "./browser/cmux/socket-client";
 export { extractReadableFromHtml, type ReadableFormat, type ReadableResult } from "./browser/readable";

packages/coding-agent/src/tools/browser/aria/aria-snapshot.ts

@@ -0,0 +1,103 @@
+import type { ElementHandle, JSHandle, Page } from "puppeteer-core";
+import ariaBundle from "./aria-snapshot.bundle.txt" with { type: "text" };
+// `aria-snapshot.bundle.txt` is a generated, committed artifact: Playwright's
+// injected ARIA-snapshot sources (pinned, Apache-2.0) bundled to a CJS module.
+// The upstream sources are NOT vendored — regenerate the bundle with:
+//   bun scripts/generate-aria-snapshot.ts
+// (fetches the pinned tag, bundles in a temp dir, rewrites the .txt artifact.)
+
+export interface AriaSnapshotOptions {
+	/** Maximum tree depth to render. */
+	depth?: number;
+	/** Append `[box=x,y,w,h]` bounding boxes to each node. */
+	boxes?: boolean;
+}
+
+/**
+ * Page-side evaluators built ONCE here in the worker — never inside the page, so
+ * page CSP never applies. They run the generated Playwright ARIA-snapshot bundle
+ * (CJS, see scripts/generate-aria-snapshot.ts) in a throwaway module scope.
+ *
+ * Puppeteer serializes these functions to a CDP `Runtime.evaluate` in the page's
+ * MAIN world (the only world where the bundle's `_ariaRef` ref expandos live —
+ * isolated-world locators/query-handlers cannot see them). Nothing is installed
+ * on `window`; the only footprint is the `_ariaRef` markers the snapshot writes,
+ * which are the price of actionable `[ref=eN]` ids.
+ */
+function buildEvaluator(params: string, call: string): (...args: unknown[]) => unknown {
+	return new Function(
+		...params.split(",").map(p => p.trim()),
+		`var module = { exports: {} };\n${ariaBundle}\nreturn module.exports.${call};`,
+	) as unknown as (...args: unknown[]) => unknown;
+}
+
+// Handles (root) must stay top-level args: Puppeteer only unwraps JSHandles
+// passed positionally to page.evaluate, never ones nested inside an object.
+const evaluateAriaSnapshot = buildEvaluator("root, request", "ariaSnapshot(root, request)");
+const evaluateResolveRef = buildEvaluator("ref", "resolveAriaRef(ref)");
+
+/**
+ * Capture a Playwright-format ARIA snapshot of `root` (or the whole document when
+ * null). Always runs in `ai` mode so every node carries a `[ref=eN]` id; resolve
+ * those to elements with {@link resolveAriaRefHandle}. Ids are renumbered from e1
+ * on each call and remain valid until the next snapshot.
+ */
+export async function captureAriaSnapshot(
+	page: Page,
+	root: ElementHandle | null,
+	options: AriaSnapshotOptions = {},
+): Promise<string> {
+	const request = { depth: options.depth, boxes: options.boxes };
+	return (await page.evaluate(evaluateAriaSnapshot as never, root as never, request as never)) as string;
+}
+
+/**
+ * Resolve a `[ref=eN]` id from the latest snapshot to a live `ElementHandle`, or
+ * null when the ref no longer matches any element. Runs in the main world so it
+ * sees the `_ariaRef` expandos the snapshot wrote.
+ */
+export async function resolveAriaRefHandle(page: Page, ref: string): Promise<ElementHandle | null> {
+	const handle = (await page.evaluateHandle(evaluateResolveRef as never, ref as never)) as JSHandle;
+	const element = handle.asElement();
+	if (!element) {
+		await handle.dispose().catch(() => undefined);
+		return null;
+	}
+	return element as ElementHandle;
+}
+
+const ARIA_REF_PREFIXES = ["aria-ref=", "aria-ref/", "ariaref/"];
+
+/**
+ * Recognize the explicit `[ref=eN]` selector forms and return the bare ref id,
+ * else null. Accepts `aria-ref=e5` (Playwright-MCP style), `aria-ref/e5`, and
+ * `ariaref/e5` — lets `tab.click("aria-ref=e5")` etc. act on snapshot refs. A
+ * bare `e5` is intentionally NOT a ref selector: the cmux backend already uses
+ * bare `eN`/`@eN` for its own observe ids, so requiring the prefix keeps action
+ * selectors meaning the same thing on both backends. (`tab.ref("e5")` still
+ * accepts a bare id directly.)
+ */
+export function parseAriaRefSelector(selector: string): string | null {
+	const trimmed = selector.trim();
+	for (const prefix of ARIA_REF_PREFIXES) {
+		if (trimmed.startsWith(prefix)) {
+			const id = trimmed.slice(prefix.length).trim();
+			return /^e\d+$/.test(id) ? id : null;
+		}
+	}
+	return null;
+}
+
+/**
+ * Build a self-contained expression script that runs the vendored bundle in the
+ * page and returns the ARIA snapshot YAML. Used by the cmux backend, whose
+ * `browser.eval` RPC takes a script string and returns the completion value (it
+ * has no ElementHandle to pass in). The script resolves `selector` via
+ * `document.querySelector` in-page (CSS selectors only) or falls back to the
+ * whole document. Like the puppeteer path it installs nothing on `window`.
+ */
+export function buildAriaSnapshotScript(selector: string | undefined, options: AriaSnapshotOptions = {}): string {
+	const request = { depth: options.depth, boxes: options.boxes };
+	const sel = selector ? JSON.stringify(selector) : "null";
+	return `(function(){var module={exports:{}};\n${ariaBundle}\nvar __sel=${sel};var __root=__sel?document.querySelector(__sel):null;if(__sel&&!__root)throw new Error("tab.ariaSnapshot: selector "+__sel+" matched no element");return module.exports.ariaSnapshot(__root,${JSON.stringify(request)});})()`;
+}

packages/coding-agent/src/tools/browser/cmux/cmux-tab.ts

@@ -10,6 +10,7 @@ import { resizeImage } from "../../../utils/image-resize";
 import { resolveToCwd } from "../../path-utils";
 import { formatScreenshot } from "../../render-utils";
 import { ToolAbortError, ToolError } from "../../tool-errors";
+import { type AriaSnapshotOptions, buildAriaSnapshotScript } from "../aria/aria-snapshot";
 import { DEFAULT_VIEWPORT } from "../launch";
 import { extractReadableFromHtml, type ReadableFormat } from "../readable";
 import type { Observation, ReadyInfo, RunResultOk, ScreenshotResult, SessionSnapshot } from "../tab-protocol";
@@ -49,7 +50,7 @@ interface RunContext {
 
 type WaitUntil = "load" | "domcontentloaded" | "networkidle0" | "networkidle2";
 type DragTarget = string | { readonly x: number; readonly y: number };
-type SelectorKind = "css" | "ref" | "text" | "aria" | "xpath" | "pierce" | "ax";
+type SelectorKind = "css" | "ref" | "aria-ref" | "text" | "aria" | "xpath" | "pierce" | "ax";
 
 interface SelectorSpec {
 	kind: SelectorKind;
@@ -124,6 +125,20 @@ const accessibleName = element =>
 const findElement = spec => {
 	if (spec.kind === "css") return document.querySelector(spec.value);
 	if (spec.kind === "pierce") return pierceQuery(document, spec.value);
+	if (spec.kind === "aria-ref") {
+		const wanted = spec.value;
+		const scan = root => {
+			for (const el of Array.from(root.querySelectorAll("*"))) {
+				if (el._ariaRef && el._ariaRef.ref === wanted) return el;
+				if (el.shadowRoot) {
+					const found = scan(el.shadowRoot);
+					if (found) return found;
+				}
+			}
+			return null;
+		};
+		return scan(document);
+	}
 	if (spec.kind === "xpath") {
 		const result = document.evaluate(spec.value, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null);
 		return result.singleNodeValue instanceof Element ? result.singleNodeValue : null;
@@ -352,6 +367,24 @@ export class CmuxTab {
 		return observation;
 	}
 
+	async ariaSnapshot(selector?: string, opts?: AriaSnapshotOptions): Promise<string> {
+		const timeoutMs = Math.min(this.#runContext?.timeoutMs ?? 30_000, 30_000);
+		const result = (await this.#request(
+			"browser.eval",
+			{ script: buildAriaSnapshotScript(selector, opts) },
+			timeoutMs,
+		)) as CmuxEvalResult;
+		return result.value as string;
+	}
+
+	async ref(id: string): Promise<CmuxElementHandle> {
+		const refId = /^e\d+$/.test(id.trim()) ? id.trim() : id.trim().replace(/^(?:aria-ref=|aria-ref\/|ariaref\/)/, "");
+		const selector = `aria-ref=${refId}`;
+		const timeoutMs = this.#runContext?.timeoutMs ?? 30_000;
+		await this.#waitForSelector(selector, timeoutMs);
+		return new CmuxElementHandle(this, selector);
+	}
+
 	async click(selector: string): Promise<void> {
 		await this.#selectorAction(selector, "click");
 	}
@@ -893,6 +926,8 @@ export class CmuxTab {
 		else if (normalized.startsWith("p-aria/")) normalized = `aria/${normalized.slice("p-aria/".length)}`;
 		else if (normalized.startsWith("p-xpath/")) normalized = `xpath/${normalized.slice("p-xpath/".length)}`;
 		else if (normalized.startsWith("p-pierce/")) normalized = `pierce/${normalized.slice("p-pierce/".length)}`;
+		const ariaRef = /^(?:aria-ref=|aria-ref\/|ariaref\/)(e\d+)$/.exec(normalized);
+		if (ariaRef) return { kind: "aria-ref", value: ariaRef[1]!, raw };
 		const ref = /^@?e(\d+)$/.exec(normalized);
 		if (ref) return { kind: "ref", value: ref[1]!, raw, ref: `@e${ref[1]}` };
 		const slash = normalized.indexOf("/");

packages/coding-agent/src/tools/browser/tab-worker.ts

@@ -22,6 +22,12 @@ import { resizeImage } from "../../utils/image-resize";
 import { resolveToCwd } from "../path-utils";
 import { formatScreenshot } from "../render-utils";
 import { ToolAbortError, ToolError, throwIfAborted } from "../tool-errors";
+import {
+	type AriaSnapshotOptions,
+	captureAriaSnapshot,
+	parseAriaRefSelector,
+	resolveAriaRefHandle,
+} from "./aria/aria-snapshot";
 import {
 	applyStealthPatches,
 	applyViewport,
@@ -106,6 +112,7 @@ interface TabApi {
 		opts?: { waitUntil?: "load" | "domcontentloaded" | "networkidle0" | "networkidle2" },
 	): Promise<void>;
 	observe(opts?: { includeAll?: boolean; viewportOnly?: boolean }): Promise<Observation>;
+	ariaSnapshot(selector?: string, opts?: AriaSnapshotOptions): Promise<string>;
 	screenshot(opts?: ScreenshotOptions): Promise<ScreenshotResult>;
 	extract(format?: ReadableFormat): Promise<string>;
 	click(selector: string): Promise<void>;
@@ -128,6 +135,7 @@ interface TabApi {
 		opts?: { timeout?: number },
 	): Promise<HTTPResponse>;
 	id(n: number): Promise<ElementHandle>;
+	ref(id: string): Promise<ElementHandle>;
 }
 
 function normalizeSelector(selector: string): string {
@@ -792,6 +800,28 @@ export class WorkerCore {
 					);
 				}),
 			observe: opts => op("tab.observe()", quickOpMs, sig => this.#collectObservation({ ...opts, signal: sig })),
+			ariaSnapshot: (selector, opts) =>
+				op(
+					selector ? `tab.ariaSnapshot(${JSON.stringify(selector)})` : "tab.ariaSnapshot()",
+					quickOpMs,
+					async sig => {
+						let root: ElementHandle | null = null;
+						if (selector) {
+							root = (await untilAborted(sig, () =>
+								page.$(normalizeSelector(selector)),
+							)) as ElementHandle | null;
+							if (!root)
+								throw new ToolError(
+									`tab.ariaSnapshot: selector ${JSON.stringify(selector)} matched no element`,
+								);
+						}
+						try {
+							return await untilAborted(sig, () => captureAriaSnapshot(page, root, opts));
+						} finally {
+							await root?.dispose().catch(() => undefined);
+						}
+					},
+				),
 			screenshot: opts =>
 				op(describeScreenshot(opts), quickOpMs, sig =>
 					this.#captureScreenshot(session, displays, screenshots, sig, opts),
@@ -815,25 +845,50 @@ export class WorkerCore {
 				}),
 			click: selector =>
 				op(`tab.click(${JSON.stringify(selector)})`, INF, async sig => {
+					if (parseAriaRefSelector(selector) !== null) {
+						const handle = await this.#resolveAriaRef(selector);
+						try {
+							await untilAborted(sig, () => handle.click());
+						} finally {
+							await handle.dispose().catch(() => undefined);
+						}
+						return;
+					}
 					const resolved = normalizeSelector(selector);
 					if (resolved.startsWith("text/")) await clickQueryHandlerText(page, resolved, timeoutMs, sig);
 					else await untilAborted(sig, () => page.locator(resolved).setTimeout(timeoutMs).click());
 				}),
 			type: (selector, text) =>
 				op(`tab.type(${JSON.stringify(selector)})`, INF, async sig => {
-					const handle = (await untilAborted(sig, () =>
-						page.locator(normalizeSelector(selector)).setTimeout(timeoutMs).waitHandle(),
-					)) as ElementHandle;
+					const handle = await this.#resolveActionHandle(selector, timeoutMs, sig);
 					try {
 						await untilAborted(sig, () => handle.type(text, { delay: 0 }));
 					} finally {
-						await handle.dispose();
+						await handle.dispose().catch(() => undefined);
 					}
 				}),
 			fill: (selector, value) =>
-				op(`tab.fill(${JSON.stringify(selector)})`, INF, sig =>
-					untilAborted(sig, () => page.locator(normalizeSelector(selector)).setTimeout(timeoutMs).fill(value)),
-				),
+				op(`tab.fill(${JSON.stringify(selector)})`, INF, async sig => {
+					if (parseAriaRefSelector(selector) !== null) {
+						const handle = await this.#resolveAriaRef(selector);
+						try {
+							await untilAborted(sig, () =>
+								handle.evaluate(el => {
+									const node = el as unknown as { value?: string; focus?: () => void };
+									node.focus?.();
+									if ("value" in node) node.value = "";
+								}),
+							);
+							await untilAborted(sig, () => handle.type(value, { delay: 0 }));
+						} finally {
+							await handle.dispose().catch(() => undefined);
+						}
+						return;
+					}
+					await untilAborted(sig, () =>
+						page.locator(normalizeSelector(selector)).setTimeout(timeoutMs).fill(value),
+					);
+				}),
 			press: (key, opts) =>
 				op(`tab.press(${JSON.stringify(key)})`, INF, async sig => {
 					const selector = opts?.selector;
@@ -844,13 +899,8 @@ export class WorkerCore {
 				op("tab.scroll()", INF, sig => untilAborted(sig, () => page.mouse.wheel({ deltaX, deltaY }))),
 			drag: (from, to) => op("tab.drag()", INF, sig => this.#drag(from, to, sig)),
 			waitFor: selector =>
-				op(
-					`tab.waitFor(${JSON.stringify(selector)})`,
-					INF,
-					async sig =>
-						(await untilAborted(sig, () =>
-							page.locator(normalizeSelector(selector)).setTimeout(timeoutMs).waitHandle(),
-						)) as ElementHandle,
+				op(`tab.waitFor(${JSON.stringify(selector)})`, INF, sig =>
+					this.#resolveActionHandle(selector, timeoutMs, sig),
 				),
 			evaluate: (fn, ...args) =>
 				op("tab.evaluate()", INF, sig =>
@@ -862,9 +912,7 @@ export class WorkerCore {
 				) as never,
 			scrollIntoView: selector =>
 				op(`tab.scrollIntoView(${JSON.stringify(selector)})`, INF, async sig => {
-					const handle = (await untilAborted(sig, () =>
-						page.locator(normalizeSelector(selector)).setTimeout(timeoutMs).waitHandle(),
-					)) as ElementHandle;
+					const handle = await this.#resolveActionHandle(selector, timeoutMs, sig);
 					try {
 						await untilAborted(sig, () =>
 							handle.evaluate(el => {
@@ -889,6 +937,7 @@ export class WorkerCore {
 			waitForResponse: (pattern, opts) =>
 				op("tab.waitForResponse()", INF, sig => this.#waitForResponse(pattern, opts?.timeout ?? timeoutMs, sig)),
 			id: id => this.#resolveCachedHandle(id),
+			ref: id => this.#resolveAriaRef(id),
 		};
 	}
 
@@ -1188,6 +1237,29 @@ export class WorkerCore {
 		}
 		return handle;
 	}
+
+	async #resolveAriaRef(id: string): Promise<ElementHandle> {
+		const ref = parseAriaRefSelector(id) ?? id.trim();
+		const handle = await resolveAriaRefHandle(this.#requirePage(), ref);
+		if (!handle) {
+			throw new ToolError(
+				`Unknown ARIA ref ${JSON.stringify(ref)}. Run tab.ariaSnapshot() to refresh refs (they renumber each snapshot).`,
+			);
+		}
+		return handle;
+	}
+
+	/**
+	 * Resolve a selector to an ElementHandle for handle-based actions. An
+	 * `aria-ref=eN` selector resolves against the latest ariaSnapshot's refs
+	 * (main world); anything else goes through the normal locator wait.
+	 */
+	async #resolveActionHandle(selector: string, timeoutMs: number, sig: AbortSignal): Promise<ElementHandle> {
+		if (parseAriaRefSelector(selector) !== null) return this.#resolveAriaRef(selector);
+		return (await untilAborted(sig, () =>
+			this.#requirePage().locator(normalizeSelector(selector)).setTimeout(timeoutMs).waitHandle(),
+		)) as ElementHandle;
+	}
 	#clearElementCache(): void {
 		if (this.#elementCache.size === 0) {
 			this.#elementCounter = 0;

packages/coding-agent/test/tools/browser-aria-snapshot.test.ts

@@ -0,0 +1,48 @@
+import { describe, expect, it } from "bun:test";
+import { buildAriaSnapshotScript, parseAriaRefSelector } from "@oh-my-pi/pi-coding-agent/tools/browser";
+
+describe("parseAriaRefSelector", () => {
+	it("accepts the explicit aria-ref prefixes and returns the bare id", () => {
+		expect(parseAriaRefSelector("aria-ref=e5")).toBe("e5");
+		expect(parseAriaRefSelector("aria-ref/e12")).toBe("e12");
+		expect(parseAriaRefSelector("ariaref/e0")).toBe("e0");
+		expect(parseAriaRefSelector("  aria-ref=e7  ")).toBe("e7");
+	});
+
+	it("rejects a bare eN id so action selectors mean the same on both backends", () => {
+		// cmux already uses bare `eN`/`@eN` for its native observe refs; requiring
+		// the prefix keeps `tab.click("e5")` from meaning different things per backend.
+		expect(parseAriaRefSelector("e5")).toBeNull();
+		expect(parseAriaRefSelector("@e5")).toBeNull();
+	});
+
+	it("rejects css and other selectors", () => {
+		expect(parseAriaRefSelector("button#go")).toBeNull();
+		expect(parseAriaRefSelector("text/Submit")).toBeNull();
+		expect(parseAriaRefSelector("aria-ref=button")).toBeNull(); // not an eN id
+		expect(parseAriaRefSelector("aria-ref=")).toBeNull();
+	});
+});
+
+describe("buildAriaSnapshotScript", () => {
+	it("resolves a CSS root selector in-page and throws on miss", () => {
+		const script = buildAriaSnapshotScript("main .post");
+		expect(script).toContain('var __sel="main .post"');
+		expect(script).toContain("document.querySelector(__sel)");
+		expect(script).toContain("matched no element");
+		// The vendored bundle's entry is invoked against the resolved root.
+		expect(script).toContain("module.exports.ariaSnapshot(__root,");
+	});
+
+	it("defaults the root to the whole document when no selector is given", () => {
+		const script = buildAriaSnapshotScript(undefined);
+		expect(script).toContain("var __sel=null");
+		expect(script).toContain("module.exports.ariaSnapshot(__root,");
+	});
+
+	it("threads depth and boxes options into the request payload", () => {
+		const script = buildAriaSnapshotScript(undefined, { depth: 3, boxes: true });
+		expect(script).toContain('"depth":3');
+		expect(script).toContain('"boxes":true');
+	});
+});