From 153e1b1f5cb58ad0704f18daddd558223681c10e Mon Sep 17 00:00:00 2001 From: Wolfgang Schoenberger <221313372+wolfiesch@users.noreply.github.com> Date: Sun, 26 Jul 2026 02:55:20 -0700 Subject: [PATCH] docs(computer): describe coordinate-safe capture sizing --- docs/computer-use.md | 10 +++++----- docs/settings.md | 6 +++--- docs/tools/computer.md | 6 +++--- packages/coding-agent/CHANGELOG.md | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/computer-use.md b/docs/computer-use.md index 5c5e5c9fd..4357185f9 100644 --- a/docs/computer-use.md +++ b/docs/computer-use.md @@ -46,7 +46,7 @@ omp config set computer.enabled true omp config get computer.enabled ``` -Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings snapshot when the desktop controller is created. A model switch that crosses the Claude-family sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. +Inside a running session, the `/computer` slash command (`/computer`, `/computer on|off|status`) toggles the tool for that session only; it never writes settings files. Backend, display, and image-size settings snapshot when the desktop controller is created. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. ### Settings @@ -55,8 +55,8 @@ Inside a running session, the `/computer` slash command (`/computer`, `/computer | `computer.enabled` | `false` | Register the essential `computer` tool. | | `computer.backend` | `auto` | `auto` or `native`. Both require a native backend; neither falls back to browser or software automation. | | `computer.display` | `all` | Composite every active display, or select one numeric native display ID. | -| `computer.maxWidth` | `1920` | Maximum composite screenshot width in pixels. Claude-family image transports cap the effective width at `1280`; other models retain the configured limit. | -| `computer.maxHeight` | `1200` | Maximum composite screenshot height in pixels. Claude-family image transports cap the effective height at `896`; other models retain the configured limit. | +| `computer.maxWidth` | `1920` | Maximum composite screenshot width in pixels. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. | +| `computer.maxHeight` | `1200` | Maximum composite screenshot height in pixels. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. | The first successful result lists each display ID, name, logical rectangle, screenshot-pixel rectangle, scale, and primary status. Use one of those IDs as a string when you want a single display: @@ -107,14 +107,14 @@ A batch containing only `screenshot` and `wait` is observation-only. Any click, ## Screenshot coordinates and image mapping -Always choose coordinates from the immediately preceding successful computer result returned by the current desktop controller. Every coordinate action in one batch maps through that same prior frame. A model switch that crosses the Claude-family sizing boundary recreates the controller and invalidates the prior frame, so capture a fresh screenshot before the next coordinate action. Do not use OS logical coordinates, CSS pixels, terminal cell positions, coordinates copied from another screenshot, or an in-batch `screenshot` marker as a new frame. +Always choose coordinates from the immediately preceding successful computer result returned by the current desktop controller. Every coordinate action in one batch maps through that same prior frame. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and invalidates the prior frame, so capture a fresh screenshot before the next coordinate action. Do not use OS logical coordinates, CSS pixels, terminal cell positions, coordinates copied from another screenshot, or an image resized after capture. For each capture, OMP: 1. Enumerates the selected native displays and their global logical rectangles. 2. Captures every selected display at native pixel density. 3. Builds one logical bounding rectangle, including negative monitor origins. -4. Chooses one render scale that preserves the desktop layout and stays within the configured `maxWidth` and `maxHeight` limits. Claude-family image transports additionally cap the effective frame at `1280×896`, below their verified resize threshold; other providers retain the configured limits. +4. Chooses one render scale that preserves the desktop layout and stays within the configured `maxWidth` and `maxHeight` limits. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, additionally cap the effective frame at `1280×896`; Claude-family models use the same cap as a compatibility fallback, and other providers retain the configured limits. 5. Places each resized display image into the composite and returns a PNG. Each result's `displays` metadata maps both spaces: diff --git a/docs/settings.md b/docs/settings.md index 296b96b2b..b0262a2fe 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -496,10 +496,10 @@ computer: | `computer.enabled` | boolean | `false` | Enable the native computer tool. Natively capable OpenAI GA models use the `{ "type": "computer" }` wire form; every other function-calling model gets `computer` as a regular function tool. The `/computer` slash command toggles this for the current session only. | | `computer.backend` | enum | `auto` | `auto` or `native`; both require native capture/input and never fall back to browser automation. | | `computer.display` | string | `all` | Composite all active displays, or use a numeric display ID reported by a successful computer result. | -| `computer.maxWidth` | number | `1920` | Maximum composite screenshot width in pixels. Claude-family image transports cap the effective width at `1280` to avoid an unreported provider resize. | -| `computer.maxHeight` | number | `1200` | Maximum composite screenshot height in pixels. Claude-family image transports cap the effective height at `896` to keep the frame below the verified image budget. | +| `computer.maxWidth` | number | `1920` | Maximum composite screenshot width in pixels. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. | +| `computer.maxHeight` | number | `1200` | Maximum composite screenshot height in pixels. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. | -Computer settings are captured when the desktop controller is created. A model switch that crosses the Claude-family sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. The recreated controller has no prior coordinate frame, so capture a fresh screenshot before the next pointer action. Before enabling input, configure `tools.approvalMode` or `tools.approval.computer` and grant platform permissions. See [Native computer use](./computer-use.md) for supported providers, actions, coordinate mapping, displays, platform setup, safety, Files behavior, troubleshooting, and verified limitations. +Computer settings are captured when the desktop controller is created. A model switch that crosses the coordinate-safe sizing boundary recreates the controller and resnapshots those settings; changing config alone does not, so start a new session after a settings change. The recreated controller has no prior coordinate frame, so capture a fresh screenshot before the next pointer action. Before enabling input, configure `tools.approvalMode` or `tools.approval.computer` and grant platform permissions. See [Native computer use](computer-use.md). ### Shell, eval, and LSP diff --git a/docs/tools/computer.md b/docs/tools/computer.md index 33cc8ded2..3d3e33301 100644 --- a/docs/tools/computer.md +++ b/docs/tools/computer.md @@ -38,10 +38,10 @@ Unlike `browser`, `computer` operates the entire visible host session. It can ac | `computer.enabled` | boolean | `false` | Register tool. | | `computer.backend` | `auto \| native` | `auto` | Both prohibit non-native fallback. | | `computer.display` | string | `all` | `all` or numeric native monitor ID. | -| `computer.maxWidth` | number | `1920` | Maximum composite PNG width. Claude-family image transports cap the effective width at `1280`. | -| `computer.maxHeight` | number | `1200` | Maximum composite PNG height. Claude-family image transports cap the effective height at `896`. | +| `computer.maxWidth` | number | `1920` | Maximum composite PNG width. Image transports that cannot preserve original detail, including GitHub Copilot Responses and xAI OAuth, cap the effective width at `1280`; Claude-family models use the same cap as a compatibility fallback. | +| `computer.maxHeight` | number | `1200` | Maximum composite PNG height. Those coordinate-safe transports cap the effective height at `896`; other models retain the configured limit. | -The controller snapshots these settings into one `DesktopSessionOptions`. Crossing the Claude-family sizing boundary during a model switch recreates the controller, resnapshots the options, and invalidates the prior coordinate frame; the next pointer action requires a fresh screenshot. +The controller snapshots these settings into one `DesktopSessionOptions`. Crossing the coordinate-safe sizing boundary during a model switch recreates the controller, resnapshots the options, and invalidates the prior coordinate frame; the next pointer action requires a fresh screenshot. ## Inputs diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 1eac59436..78807e0d7 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -14,7 +14,7 @@ - `omp usage` revalidates the broker credential snapshot before rendering: live usage reports were previously paired with a disk-cached account list up to an hour old, so a just-completed re-login (org-less row upserted to org-scoped) rendered as a phantom duplicate until the cache expired. ### Fixed -- Fixed screenshot-relative pointer actions missing their visible targets when Claude-family image transports silently downscaled a large computer screenshot; those sessions now establish the native coordinate frame below the verified image-resize threshold without changing the public capture defaults for other models ([#6596](https://github.com/can1357/oh-my-pi/pull/6596) by [@wolfiesch](https://github.com/wolfiesch)). +- Fixed screenshot-relative pointer actions missing their visible targets when image transports that cannot preserve original detail silently downscaled a large computer screenshot; affected transports now establish the native coordinate frame below the verified image-resize threshold without changing the public capture defaults for other models ([#6596](https://github.com/can1357/oh-my-pi/pull/6596) by [@wolfiesch](https://github.com/wolfiesch)). ## [17.1.3] - 2026-07-24