smashimo/oh-my-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(computer): describe coordinate-safe capture sizing

Browse Source
This commit is contained in:
Wolfgang Schoenberger
2026-07-26 02:55:20 -07:00
parent 0c5fa60c77
commit 153e1b1f5c
4 changed files with 12 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/computer-use.md
+5 -5
View File
@@ -46,7 +46,7 @@ omp config set computer.enabled true
omp config get computer.enabled
```
Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings snapshot when the desktop controller is created. A model switch that crosses the Claude-family sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change.
Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings snapshot when the desktop controller is created. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change.
### Settings
@@ -55,8 +55,8 @@ Inside a running session, the `/computer` slash command (`/computer`, `/computer
| `computer.enabled` | `false` | Register the essential `computer` tool. |
| `computer.backend` | `auto` | `auto` or `native`. Both require a native backend; neither falls back to browser or software automation. |
| `computer.display` | `all` | Composite every active display, or select one numeric native display ID. |
| `computer.maxWidth` | `1920` | Maximum composite screenshot width in pixels. Claude-family image transports cap the effective width at `1280`; other models retain the configured limit. |
| `computer.maxHeight` | `1200` | Maximum composite screenshot height in pixels. Claude-family image transports cap the effective height at `896`; other models retain the configured limit. |
| `computer.maxWidth` | `1920` | Maximum composite screenshot width in pixels. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. |
| `computer.maxHeight` | `1200` | Maximum composite screenshot height in pixels. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. |
The first successful result lists each display ID, name, logical rectangle, screenshot-pixel rectangle, scale, and primary status. Use one of those IDs as a string when you want a single display:
@@ -107,14 +107,14 @@ A batch containing only `screenshot` and `wait` is observation-only. Any click,
## Screenshot coordinates and image mapping
Always choose coordinates from the immediately preceding successful computer result returned by the current desktop controller. Every coordinate action in one batch maps through that same prior frame. A model switch that crosses the Claude-family sizing boundary recreates the controller and invalidates the prior frame, so capture a fresh screenshot before the next coordinate action. Do not use OS logical coordinates, CSS pixels, terminal cell positions, coordinates copied from another screenshot, or an in-batch `screenshot` marker as a new frame.
Always choose coordinates from the immediately preceding successful computer result returned by the current desktop controller. Every coordinate action in one batch maps through that same prior frame. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and invalidates the prior frame, so capture a fresh screenshot before the next coordinate action. Do not use OS logical coordinates, CSS pixels, terminal cell positions, coordinates copied from another screenshot, or an image resized after capture.
For each capture, OMP:
1. Enumerates the selected native displays and their global logical rectangles.
2. Captures every selected display at native pixel density.
3. Builds one logical bounding rectangle, including negative monitor origins.
4. Chooses one render scale that preserves the desktop layout and stays within the configured `maxWidth` and `maxHeight` limits. Claude-family image transports additionally cap the effective frame at `1280×896`, below their verified resize threshold; other providers retain the configured limits.
4. Chooses one render scale that preserves the desktop layout and stays within the configured `maxWidth` and `maxHeight` limits. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, additionally cap the effective frame at `1280×896`; Claude-family models use the same cap as a compatibility fallback, and other providers retain the configured limits.
5. Places each resized display image into the composite and returns a PNG.
Each result's `displays` metadata maps both spaces:
docs/settings.md
+3 -3
View File
@@ -496,10 +496,10 @@ computer:
| `computer.enabled` | boolean | `false` | Enable the native computer tool. Natively capable OpenAI GA models use the `{ "type": "computer" }` wire form; every other function-calling model gets `computer` as a regular function tool. The `/computer` slash command toggles this for the current session only. |
| `computer.backend` | enum | `auto` | `auto` or `native`; both require native capture/input and never fall back to browser automation. |
| `computer.display` | string | `all` | Composite all active displays, or use a numeric display ID reported by a successful computer result. |
| `computer.maxWidth` | number | `1920` | Maximum composite screenshot width in pixels. Claude-family image transports cap the effective width at `1280` to avoid an unreported provider resize. |
| `computer.maxHeight` | number | `1200` | Maximum composite screenshot height in pixels. Claude-family image transports cap the effective height at `896` to keep the frame below the verified image budget. |
| `computer.maxWidth` | number | `1920` | Maximum composite screenshot width in pixels. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. |
| `computer.maxHeight` | number | `1200` | Maximum composite screenshot height in pixels. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. |
Computer settings are captured when the desktop controller is created. A model switch that crosses the Claude-family sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. The recreated controller has no prior coordinate frame, so capture a fresh screenshot before the next pointer action. Before enabling input, configure `tools.approvalMode` or `tools.approval.computer` and grant platform permissions. See [Native computer use](./computer-use.md) for supported providers, actions, coordinate mapping, displays, platform setup, safety, Files behavior, troubleshooting, and verified limitations.
Computer settings are captured when the desktop controller is created. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. The recreated controller has no prior coordinate frame, so capture a fresh screenshot before the next pointer action. Before enabling input, configure `tools.approvalMode` or `tools.approval.computer` and grant platform permissions. See [Native computer use](computer-use.md).
### Shell, eval, and LSP
docs/tools/computer.md
+3 -3
View File
@@ -38,10 +38,10 @@ Unlike `browser`, `computer` operates the entire visible host session. It can ac
| `computer.enabled` | boolean | `false` | Register tool. |
| `computer.backend` | `auto \| native` | `auto` | Both prohibit non-native fallback. |
| `computer.display` | string | `all` | `all` or numeric native monitor ID. |
| `computer.maxWidth` | number | `1920` | Maximum composite PNG width. Claude-family image transports cap the effective width at `1280`. |
| `computer.maxHeight` | number | `1200` | Maximum composite PNG height. Claude-family image transports cap the effective height at `896`. |
| `computer.maxWidth` | number | `1920` | Maximum composite PNG width. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. |
| `computer.maxHeight` | number | `1200` | Maximum composite PNG height. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. |
The controller snapshots these settings into one `DesktopSessionOptions`. Crossing the Claude-family sizing boundary during a model switch recreates the controller, resnapshots the options, and invalidates the prior coordinate frame; the next pointer action requires a fresh screenshot.
The controller snapshots these settings into one `DesktopSessionOptions`. Crossing the coordinate-safe sizing boundary during a model switch recreates the controller, resnapshots the options, and invalidates the prior coordinate frame; the next pointer action requires a fresh screenshot.
## Inputs
packages/coding-agent/CHANGELOG.md
+1 -1
View File
@@ -14,7 +14,7 @@
- `omp usage` revalidates the broker credential snapshot before rendering: live usage reports were previously paired with a disk-cached account list up to an hour old, so a just-completed re-login (org-less row upserted to org-scoped) rendered as a phantom duplicate until the cache expired.
### Fixed
- Fixed screenshot-relative pointer actions missing their visible targets when Claude-family image transports silently downscaled a large computer screenshot; those sessions now establish the native coordinate frame below the verified image-resize threshold without changing the public capture defaults for other models ([#6596](https://github.com/can1357/oh-my-pi/pull/6596) by [@wolfiesch](https://github.com/wolfiesch)).
- Fixed screenshot-relative pointer actions missing their visible targets when image transports that cannot preserve original detail silently downscaled a large computer screenshot; affected transports now establish the native coordinate frame below the verified image-resize threshold without changing the public capture defaults for other models ([#6596](https://github.com/can1357/oh-my-pi/pull/6596) by [@wolfiesch](https://github.com/wolfiesch)).
## [17.1.3] - 2026-07-24