smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(computer-use): document routing diagnostics

Browse Source
This commit is contained in:
usr-bin-roygbiv
2026-07-24 17:05:31 +00:00
parent 8e2d654880
commit 6b7e2ccc40
3 changed files with 16 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tools/computer.md
+12 -11
View File
@@ -21,7 +21,7 @@ User setup, safety guidance, platform permissions, and verified limitations: [Na
## Availability and declaration
- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings.
- `computer.enabled` gates registration and defaults to `false`. The `/computer` slash command toggles it for the current session without persisting settings. `/computer status` reports enabled/active state, controller settings, active model, and effective native/function exposure. Explicit enablement and the controller survive model switches while exposure is recomputed for the selected model.
- Enabled tool load mode: `essential`.
- Concurrency: `exclusive`.
- Native descriptor: `{ type: "computer" }`.
@@ -132,15 +132,16 @@ OMP native execution never creates a provider Files upload. The provider contrac
1. Tool registration checks `computer.enabled`.
2. `ComputerTool` constructs a `ComputerSupervisor` with session settings but does not start a worker.
3. Provider adapter exposes the native declaration only for capable models.
4. Provider `action`/`actions` and pending safety checks become typed tool-call metadata.
5. Extension wrapper resolves tool approval and mandatory provider safety approval.
6. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval.
7. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker.
8. Worker constructs one native `DesktopSession` and reports capabilities.
9. Worker rejects coordinate input until it has returned a screenshot to the provider.
10. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch.
11. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call.
12. Tool returns image content, display/capability details, and exact GA result metadata.
4. While active, the system prompt routes host-desktop requests through `computer` and requires inspection of each fresh returned screenshot before the next action.
5. Provider `action`/`actions` and pending safety checks become typed tool-call metadata.
6. Extension wrapper resolves tool approval and mandatory provider safety approval.
7. `ComputerTool.execute()` chooses metadata actions, validates the batch, and rechecks safety approval.
8. Supervisor serializes execution behind a promise tail and lazily starts one Bun worker.
9. Worker constructs one native `DesktopSession` and reports capabilities.
10. Worker rejects coordinate input until it has returned a screenshot to the provider.
11. Native session validates all actions, executes them in order, defers any `screenshot` markers, and captures one fresh PNG after the entire successful batch.
12. Worker transfers the PNG buffer to the parent and preserves session/frame state for the next call.
13. Tool returns image content, display/capability details, and exact GA result metadata.
## Capture and coordinate mapping
@@ -234,4 +235,4 @@ Key platform failures and remedies are listed in [Native computer use: Troublesh
- Linux coordinate input rejects negative global display origins; X11/XTest also rejects global positions above 32767.
- Windows backend implemented but not remotely exercised for this feature.
- Real remote macOS proof used `ComputerSupervisor` → worker → native session on a real macOS host, controlling TextEdit with global hotkey, double-click, click, type, and 1920×1080 Quartz capture after permissions were granted.
- That proof did not include a live OpenAI native provider round trip. GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI.
- Live subscription-provider round trips exercised function-tool exposure and real screenshot execution for GPT-5.3 Codex Spark plus GPT-5.6 Luna, Terra, and Sol. Live native OpenAI GA transport was not exercised; GA transport and replay are contract-tested locally. The pure-Rust Linux backend is exercised by unit tests (pixel conversion, keysym mapping, deadline enforcement), not by a live X session in CI.