docs(computer-use): document routing diagnostics
This commit is contained in:
@@ -46,7 +46,7 @@ omp config set computer.enabled true
|
||||
omp config get computer.enabled
|
||||
```
|
||||
|
||||
Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings still snapshot when the session's desktop controller is created, so change those in config and start a new session.
|
||||
Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. `/computer status` reports the effective enabled/active state, backend, display and capture limits, active model, and whether that model receives native or function exposure. Explicit enablement and the desktop controller stay active across model switches; exposure is recomputed for the new model. Backend, display, and image-size settings still snapshot when the session's desktop controller is created, so change those in config and start a new session.
|
||||
|
||||
### Settings
|
||||
|
||||
@@ -81,6 +81,7 @@ Codex subscription endpoints and custom or proxy routes do not infer native supp
|
||||
Natively capable OpenAI Responses routes may receive a forced `{ "type": "computer" }` choice. Function-tool fallback forcing is provider-specific: OpenAI/Ollama use a named function, Anthropic/Bedrock use a named tool, Google uses required-tool mode, and adapters without a forcing form keep provider-default selection. Responses Lite moves tools into `additional_tools`; for an explicitly forced computer declaration it sends only that declaration and uses `tool_choice: "required"`, preserving both selection and forcing without an invalid object choice that refers to removed top-level tools.
|
||||
|
||||
When a session switches from a native-capable API route to a subscription or proxy route, prior native computer history is converted to a representation the target accepts. Codex subscription requests replay it as named `computer` function calls and results, then declare the next computer call as the same named function. Other non-native OpenAI Responses-family targets may use stable assistant text notes; other provider adapters use their ordinary tool format.
|
||||
While the tool is active, the system prompt makes host-desktop routing explicit even for compact native-tool inventories: desktop requests must use `computer`, and every successful action must be followed by inspection of its fresh screenshot before the next action. This does not auto-enable the tool, bypass approval, or prevent a user-requested alternative after a computer error.
|
||||
|
||||
If the tool never appears:
|
||||
|
||||
|
||||
+12
-11
@@ -21,7 +21,7 @@ User setup, safety guidance, platform permissions, and verified limitations: [Na
|
||||
|
||||
## Availability and declaration
|
||||
|
||||
- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings.
|
||||
- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings. `/computer status` reports enabled/active state, controller settings, active model, and effective native/function exposure. Explicit enablement and the controller survive model switches while exposure is recomputed for the selected model.
|
||||
- Enabled tool load mode: `essential`.
|
||||
- Concurrency: `exclusive`.
|
||||
- Native descriptor: `{ type: "computer" }`.
|
||||
@@ -132,15 +132,16 @@ OMP native execution never creates a provider Files upload. The provider contrac
|
||||
1. Tool registration checks `computer.enabled`.
|
||||
2. `ComputerTool` constructs a `ComputerSupervisor` with session settings but does not start a worker.
|
||||
3. Provider adapter exposes the native declaration only for capable models.
|
||||
4. Provider `action`/`actions` and pending safety checks become typed tool-call metadata.
|
||||
5. Extension wrapper resolves tool approval and mandatory provider safety approval.
|
||||
6. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval.
|
||||
7. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker.
|
||||
8. Worker constructs one native `DesktopSession` and reports capabilities.
|
||||
9. Worker rejects coordinate input until it has returned a screenshot to the provider.
|
||||
10. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch.
|
||||
11. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call.
|
||||
12. Tool returns image content, display/capability details, and exact GA result metadata.
|
||||
4. While active, the system prompt routes host-desktop requests through `computer` and requires inspection of each fresh returned screenshot before the next action.
|
||||
5. Provider `action`/`actions` and pending safety checks become typed tool-call metadata.
|
||||
6. Extension wrapper resolves tool approval and mandatory provider safety approval.
|
||||
7. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval.
|
||||
8. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker.
|
||||
9. Worker constructs one native `DesktopSession` and reports capabilities.
|
||||
10. Worker rejects coordinate input until it has returned a screenshot to the provider.
|
||||
11. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch.
|
||||
12. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call.
|
||||
13. Tool returns image content, display/capability details, and exact GA result metadata.
|
||||
|
||||
## Capture and coordinate mapping
|
||||
|
||||
@@ -234,4 +235,4 @@ Key platform failures and remedies are listed in [Native computer use: Troublesh
|
||||
- Linux coordinate input rejects negative global display origins; X11/XTest also rejects global positions above 32767.
|
||||
- Windows backend implemented but not remotely exercised for this feature.
|
||||
- Real remote macOS proof used `ComputerSupervisor` → worker → native session on a real macOS host, controlling TextEdit with global hotkey, double-click, click, type, and 1920×1080 Quartz capture after permissions were granted.
|
||||
- That proof did not include a live OpenAI native provider round trip. GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI.
|
||||
- Live subscription-provider round trips exercised function-tool exposure and real screenshot execution for GPT-5.3 Codex Spark plus GPT-5.6 Luna, Terra, and Sol. Live native OpenAI GA transport was not exercised; GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI.
|
||||
|
||||
@@ -28,10 +28,12 @@
|
||||
- Large pastes saved via the large-paste menu now insert `local://paste-N.md` references (previously `local://attachment-N`), so the saved paste carries a markdown extension and a clearer name.
|
||||
- Raw SSE debug capture now trims over-budget events smartly instead of chopping off the tail: tool definitions inside `data:` payloads are compacted first (name kept, schema/description elided — often enough to keep the whole payload as valid JSON), and anything still over the 64k cap keeps its head and tail with a `: omp-debug-elided chars=N` comment marking the removed middle, so trailing fields like `usage` stay visible.
|
||||
- The `web_search` tool prompt now tells the model to never search for content that is programmatically accessible or has a known URL (GitHub, known arXiv papers, Wikipedia pages, official docs) and to `read` the URL directly instead.
|
||||
- Enabled Computer Use sessions now state the desktop-routing contract in the compact system prompt, retain their controller across model switches, expose effective native/function routing through `/computer status`, and emit structured lifecycle diagnostics without logging captured content.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed `todo` calls that omit `op` hard-failing validation ("op must be operation to apply (was missing)"): the tool now validates leniently and infers the op for unambiguous payloads (`list` → `init`, `phase`+`items` → `append`, bare `items` on an empty list → `init`); `op` stays required in the schema, and ambiguous op-less calls surface the schema error as a retryable tool error.
|
||||
- Fixed Codex subscription and proxy models being sent the unsupported native `{ type: "computer" }` declaration based only on model ID. They now receive the callable function-tool fallback, including after switching from native OpenAI Responses history, while explicit endpoint metadata can still opt into the GA contract.
|
||||
- Fixed credential-free web search engines (SearXNG, DuckDuckGo, Google, Startpage, Ecosia, Mojeek, and the Public Web fan-out) returning zero results for queries with `site:` paths (e.g. `site:github.com/owner/repo`) or `inurl:` operators: scraper engines only match `site:` against a bare domain and DuckDuckGo ignores `inurl:` entirely, so such queries silently emptied the result set and fell through to the next provider in the chain. A shared `formatScraperQuery` formatter now structurally demotes path-carrying `site:` and all `inurl:` values to plain search terms (covering OR-grouped and quoted directives) while preserving bare-domain `site:` filters, negated operators, and each engine's supported syntax; the pipeline post-filter still enforces the demoted constraints on returned sources.
|
||||
- Fixed `ast_edit` previews reading like applied edits to the model: the `⟨proposed⟩` badge was TUI-only, so the model-visible result (hashline header + `-`/`+` rows, identical to applied edit output) carried no staged-proposal signal. The preview result now leads with a "Staged as a proposal — files NOT modified yet" notice naming `xd://resolve`/`xd://reject`, the injected resolve reminder names the source tool, and the `ast_edit` tool prompt documents the two-phase flow.
|
||||
- Fixed the `hub` launch `ps`/`list` response burying the active process behind every exited one and growing without bound in long-lived projects: the broker now lists non-terminal daemons first (oldest to newest) and caps exited/failed history at the 10 most recently exited, so the active launch is immediately visible and the response stays bounded. Broker recovery also preserves each already-terminal daemon's real exit time instead of overwriting it with the restart timestamp, so the history cap keeps the genuinely most-recently-exited processes after an idle-broker restart ([#6517](https://github.com/can1357/oh-my-pi/issues/6517)).
|
||||
|
||||
Reference in New Issue
Block a user