From 6b7e2ccc400d64d4d66ea8cb5f5a61d9bb22372b Mon Sep 17 00:00:00 2001 From: usr-bin-roygbiv Date: Fri, 24 Jul 2026 17:05:31 +0000 Subject: [PATCH] docs(computer-use): document routing diagnostics --- docs/computer-use.md | 3 ++- docs/tools/computer.md | 23 ++++++++++++----------- packages/coding-agent/CHANGELOG.md | 2 ++ 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/computer-use.md b/docs/computer-use.md index 35a6ac94a..c9deee219 100644 --- a/docs/computer-use.md +++ b/docs/computer-use.md @@ -46,7 +46,7 @@ omp config set computer.enabled true omp config get computer.enabled ``` -Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings still snapshot when the session's desktop controller is created, so change those in config and start a new session. +Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. `/computer status` reports the effective enabled/active state, backend, display and capture limits, active model, and whether that model receives native or function exposure. Explicit enablement and the desktop controller stay active across model switches; exposure is recomputed for the new model. Backend, display, and image-size settings still snapshot when the session's desktop controller is created, so change those in config and start a new session. ### Settings @@ -81,6 +81,7 @@ Codex subscription endpoints and custom or proxy routes do not infer native supp Natively capable OpenAI Responses routes may receive a forced `{ "type": "computer" }` choice. Function-tool fallback forcing is provider-specific: OpenAI/Ollama use a named function, Anthropic/Bedrock use a named tool, Google uses required-tool mode, and adapters without a forcing form keep provider-default selection. Responses Lite moves tools into `additional_tools`; for an explicitly forced computer declaration it sends only that declaration and uses `tool_choice: "required"`, preserving both selection and forcing without an invalid object choice that refers to removed top-level tools. When a session switches from a native-capable API route to a subscription or proxy route, prior native computer history is converted to a representation the target accepts. Codex subscription requests replay it as named `computer` function calls and results, then declare the next computer call as the same named function. Other non-native OpenAI Responses-family targets may use stable assistant text notes; other provider adapters use their ordinary tool format. +While the tool is active, the system prompt makes host-desktop routing explicit even for compact native-tool inventories: desktop requests must use `computer`, and every successful action must be followed by inspection of its fresh screenshot before the next action. This does not auto-enable the tool, bypass approval, or prevent a user-requested alternative after a computer error. If the tool never appears: diff --git a/docs/tools/computer.md b/docs/tools/computer.md index f44094e3f..a3c686979 100644 --- a/docs/tools/computer.md +++ b/docs/tools/computer.md @@ -21,7 +21,7 @@ User setup, safety guidance, platform permissions, and verified limitations: [Na ## Availability and declaration -- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings. +- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings. `/computer status` reports enabled/active state, controller settings, active model, and effective native/function exposure. Explicit enablement and the controller survive model switches while exposure is recomputed for the selected model. - Enabled tool load mode: `essential`. - Concurrency: `exclusive`. - Native descriptor: `{ type: "computer" }`. @@ -132,15 +132,16 @@ OMP native execution never creates a provider Files upload. The provider contrac 1. Tool registration checks `computer.enabled`. 2. `ComputerTool` constructs a `ComputerSupervisor` with session settings but does not start a worker. 3. Provider adapter exposes the native declaration only for capable models. -4. Provider `action`/`actions` and pending safety checks become typed tool-call metadata. -5. Extension wrapper resolves tool approval and mandatory provider safety approval. -6. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval. -7. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker. -8. Worker constructs one native `DesktopSession` and reports capabilities. -9. Worker rejects coordinate input until it has returned a screenshot to the provider. -10. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch. -11. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call. -12. Tool returns image content, display/capability details, and exact GA result metadata. +4. While active, the system prompt routes host-desktop requests through `computer` and requires inspection of each fresh returned screenshot before the next action. +5. Provider `action`/`actions` and pending safety checks become typed tool-call metadata. +6. Extension wrapper resolves tool approval and mandatory provider safety approval. +7. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval. +8. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker. +9. Worker constructs one native `DesktopSession` and reports capabilities. +10. Worker rejects coordinate input until it has returned a screenshot to the provider. +11. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch. +12. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call. +13. Tool returns image content, display/capability details, and exact GA result metadata. ## Capture and coordinate mapping @@ -234,4 +235,4 @@ Key platform failures and remedies are listed in [Native computer use: Troublesh - Linux coordinate input rejects negative global display origins; X11/XTest also rejects global positions above 32767. - Windows backend implemented but not remotely exercised for this feature. - Real remote macOS proof used `ComputerSupervisor` → worker → native session on a real macOS host, controlling TextEdit with global hotkey, double-click, click, type, and 1920×1080 Quartz capture after permissions were granted. -- That proof did not include a live OpenAI native provider round trip. GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI. +- Live subscription-provider round trips exercised function-tool exposure and real screenshot execution for GPT-5.3 Codex Spark plus GPT-5.6 Luna, Terra, and Sol. Live native OpenAI GA transport was not exercised; GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI. diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 7c9fcf50c..db13cbb5c 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -28,10 +28,12 @@ - Large pastes saved via the large-paste menu now insert `local://paste-N.md` references (previously `local://attachment-N`), so the saved paste carries a markdown extension and a clearer name. - Raw SSE debug capture now trims over-budget events smartly instead of chopping off the tail: tool definitions inside `data:` payloads are compacted first (name kept, schema/description elided — often enough to keep the whole payload as valid JSON), and anything still over the 64k cap keeps its head and tail with a `: omp-debug-elided chars=N` comment marking the removed middle, so trailing fields like `usage` stay visible. - The `web_search` tool prompt now tells the model to never search for content that is programmatically accessible or has a known URL (GitHub, known arXiv papers, Wikipedia pages, official docs) and to `read` the URL directly instead. +- Enabled Computer Use sessions now state the desktop-routing contract in the compact system prompt, retain their controller across model switches, expose effective native/function routing through `/computer status`, and emit structured lifecycle diagnostics without logging captured content. ### Fixed - Fixed `todo` calls that omit `op` hard-failing validation ("op must be operation to apply (was missing)"): the tool now validates leniently and infers the op for unambiguous payloads (`list` → `init`, `phase`+`items` → `append`, bare `items` on an empty list → `init`); `op` stays required in the schema, and ambiguous op-less calls surface the schema error as a retryable tool error. +- Fixed Codex subscription and proxy models being sent the unsupported native `{ type: "computer" }` declaration based only on model ID. They now receive the callable function-tool fallback, including after switching from native OpenAI Responses history, while explicit endpoint metadata can still opt into the GA contract. - Fixed credential-free web search engines (SearXNG, DuckDuckGo, Google, Startpage, Ecosia, Mojeek, and the Public Web fan-out) returning zero results for queries with `site:` paths (e.g. `site:github.com/owner/repo`) or `inurl:` operators: scraper engines only match `site:` against a bare domain and DuckDuckGo ignores `inurl:` entirely, so such queries silently emptied the result set and fell through to the next provider in the chain. A shared `formatScraperQuery` formatter now structurally demotes path-carrying `site:` and all `inurl:` values to plain search terms (covering OR-grouped and quoted directives) while preserving bare-domain `site:` filters, negated operators, and each engine's supported syntax; the pipeline post-filter still enforces the demoted constraints on returned sources. - Fixed `ast_edit` previews reading like applied edits to the model: the `⟨proposed⟩` badge was TUI-only, so the model-visible result (hashline header + `-`/`+` rows, identical to applied edit output) carried no staged-proposal signal. The preview result now leads with a "Staged as a proposal — files NOT modified yet" notice naming `xd://resolve`/`xd://reject`, the injected resolve reminder names the source tool, and the `ast_edit` tool prompt documents the two-phase flow. - Fixed the `hub` launch `ps`/`list` response burying the active process behind every exited one and growing without bound in long-lived projects: the broker now lists non-terminal daemons first (oldest to newest) and caps exited/failed history at the 10 most recently exited, so the active launch is immediately visible and the response stays bounded. Broker recovery also preserves each already-terminal daemon's real exit time instead of overwriting it with the restart timestamp, so the history cap keeps the genuinely most-recently-exited processes after an idle-broker restart ([#6517](https://github.com/can1357/oh-my-pi/issues/6517)).