oh-my-pi/docs/computer-use.md

Native computer use

computer captures and controls the desktop that is running omp. It uses native screen-capture and input APIs; it does not launch Chromium, use Puppeteer, or expose a DOM.

Use it for visible desktop applications: IDEs, terminals, native apps, browser windows, menus, and system dialogs. Use browser instead when you need headless/CDP browser tabs, DOM or ARIA inspection, selectors, JavaScript evaluation, or deterministic page automation.

Warning

Enabling computer gives the model mouse and keyboard access to your real desktop. Close unrelated sensitive applications, use a dedicated OS account or VM when practical, and configure approval policy before enabling it.

Enable and configure

The tool is disabled by default. Add this to ~/.omp/agent/config.yml, a project .omp/config.yml, or a one-shot --config overlay:

computer:
  enabled: true
  backend: auto
  display: all
  maxWidth: 1920
  maxHeight: 1200

tools:
  approvalMode: write

tools.approvalMode: write automatically allows observation-only batches and prompts before keyboard or pointer input. For a prompt on every computer call, including screenshots:

tools:
  approval:
    computer: prompt

To block the tool without changing computer.enabled:

tools:
  approval:
    computer: deny

You can also enable it globally from the CLI:

omp config set computer.enabled true
omp config get computer.enabled

Start a new session after changing computer settings. The desktop controller snapshots its backend, display, and image-size settings when the session tool is created.

Settings

Key	Default	Meaning
computer.enabled	false	Register the essential computer tool.
computer.backend	auto	auto or native. Both require a native backend; neither falls back to browser or software automation.
computer.display	all	Composite every active display, or select one numeric native display ID.
computer.maxWidth	1920	Maximum composite screenshot width in pixels. Must be greater than zero.
computer.maxHeight	1200	Maximum composite screenshot height in pixels. Must be greater than zero.

The first successful result lists each display ID, name, logical rectangle, screenshot-pixel rectangle, scale, and primary status. Use one of those IDs as a string when you want a single display:

computer:
  display: "2"

A disconnected or changed ID fails with DESKTOP_INVALID_OPTIONS; switch to all, capture once, then select an active ID from the result.

Model and provider capability

Enablement alone is not enough. The active model/provider transport must support the OpenAI Responses GA native tool declaration { "type": "computer" }.

OMP marks a model capable when either:

• its catalog metadata explicitly sets supportsComputerUse: true, or
• it uses openai-responses, openai-codex-responses, or azure-openai-responses and resolves to an OpenAI/OpenAI Codex or Azure model ID matching gpt-5.4 or later in the gpt-5.x family.

An explicit supportsComputerUse: false disables automatic derivation.

The provider adapter sends the native computer declaration and a forced computer tool choice only when supportsComputerUse is true. Unsupported models do not receive the declaration. If a session containing native computer history switches to an unsupported model, OMP converts prior computer_call and computer_call_output items into stable text notes rather than sending invalid native items.

This feature does not turn another provider's ordinary function-calling model into a native computer-use model. If the tool never appears or the model never calls it:

1. Confirm computer.enabled is true in the effective config.
2. Confirm the active model reports native computer-use support.
3. Use a supported OpenAI Responses, OpenAI Codex Responses, or Azure OpenAI Responses model/deployment.
4. Start a new session after changing model or tool settings.

Actions

The provider may send one GA action or an ordered actions batch. OMP normalizes both forms to an ordered batch, executes it serially, then returns one fresh PNG of the final state.

Action	Required fields	Behavior
click	button, x, y	Click once. Buttons: left, right, wheel, back, forward. Optional keys holds modifiers.
double_click	x, y, keys	Double-click the left button. GA keys is an array or null.
drag	path	Hold left at the first point, visit the remaining points, release at the last. At least two points. Optional modifier keys.
keypress	keys	Press one key or chord. The array must contain at least one non-empty key.
move	x, y	Move the pointer. Optional modifier keys.
screenshot	none	Capture without input.
scroll	x, y, scroll_x, scroll_y	Move to the point, then scroll horizontally and/or vertically. Optional modifier keys. Deltas are converted to native wheel steps.
type	text	Type Unicode text through the native input backend.
wait	none	Wait two seconds before continuing.

Coordinates and drag points must be non-negative screenshot pixels. Mouse keys may contain only unique modifiers: Control, Shift, Alt/Option, or Meta/Command/Super/Windows. Key names are case-insensitive; common names include ENTER, ESCAPE, TAB, SPACE, BACKSPACE, DELETE, arrows, navigation keys, and F1–F24. A keypress entry may contain +, for example CTRL+SHIFT+P. Single Unicode characters are also accepted. macOS has no native PRINTSCREEN or F21–F24 mapping.

A batch containing only screenshot and wait is observation-only. Any click, move, drag, scroll, keypress, or type action makes the whole call input-capable.

Screenshot coordinates and image mapping

Always choose coordinates from the immediately preceding computer result. Do not use OS logical coordinates, CSS pixels, terminal cell positions, or coordinates copied from another screenshot.

For each capture, OMP:

1. Enumerates the selected native displays and their global logical rectangles.
2. Captures every selected display at native pixel density.
3. Builds one logical bounding rectangle, including negative monitor origins.
4. Chooses one render scale that preserves the desktop layout and stays within maxWidth and maxHeight.
5. Places each resized display image into the composite and returns a PNG.

Each result's displays metadata maps both spaces:

• x, y, width, height: global logical desktop rectangle.
• pixelX, pixelY, pixelWidth, pixelHeight: rectangle inside the returned PNG.
• scale: native display scale reported by the OS.

Input actions use the returned PNG space. The backend locates the display containing that screenshot pixel, scales within that display rectangle, then adds the display's global logical origin. This supports scaled displays and displays left of or above the primary monitor.

The composite preserves gaps between monitor rectangles as black pixels. A point in a gap is not clickable and fails with DESKTOP_COORDINATE_OUT_OF_BOUNDS. Points on or beyond the PNG's right/bottom edge, negative points, and points outside every display also fail closed.

If monitor membership, rectangle, or scale changes between the reference frame and a coordinate action, OMP clears the frame and returns DESKTOP_LAYOUT_CHANGED. Capture again before retrying. Moving a display, changing resolution/scaling, docking, undocking, or changing the selected display can trigger this guard.

The worker pre-captures a frame if the first call is coordinate-based, but that unseen frame is not a safe basis for model-selected coordinates. Begin with screenshot, and capture again after any visual transition whose target may have moved.

Multiple displays

computer.display: all produces one composite. Displays are sorted by logical vertical position, then horizontal position, then ID. Mirrored displays with the same logical rectangle are coalesced; the primary mirror wins. Invalid scales, duplicate IDs, and overlapping non-mirrored rectangles fail closed rather than guessing.

Use one display when:

• the desktop is very wide and labels become hard for the model to read after downscaling;
• a layout gap makes targets ambiguous;
• you want to isolate sensitive content on another monitor; or
• you are using Wayland input.

On Linux Wayland, capture currently comes through XWayland. Coordinate input over a multi-display composite fails with DESKTOP_BACKEND_UNAVAILABLE because libei absolute coordinates cannot be safely correlated to the XWayland composite. Select one display or log into an X11 session. Keyboard-only actions do not require screenshot-coordinate mapping, but capture still requires XWayland.

Approval and safety precedence

Computer use has three safety layers.

1. Tool approval

• screenshot/wait-only batches declare read approval.
• Any input action declares exec approval.
• Missing or malformed action metadata defaults to exec.
• tools.approval.computer overrides the active mode with allow, prompt, or deny.

With tools.approvalMode: write, screenshots are automatically allowed and input prompts. The schema default is yolo, which normally auto-approves both; use write, always-ask, or an explicit per-tool policy when controlling a real desktop.

2. Provider safety checks

OpenAI may attach pending_safety_checks to a native computer_call. Precedence is strict:

1. tools.approval.computer: deny blocks the call immediately.
2. Otherwise, any pending provider check forces an interactive Approve/Deny prompt.
3. yolo, --auto-approve, per-tool allow, and prior xdev approval cannot bypass that prompt.
4. A headless session or missing UI fails closed; it never acknowledges on your behalf.
5. Only explicit approval marks the checks acknowledged and permits input.
6. OMP returns the same checks as acknowledged_safety_checks with the screenshot output.

The computer executor checks the approval marker again before native input. A provider check reaching execution without interactive approval fails with Provider safety checks require interactive approval before computer input.

3. Consequential-action confirmation

Provider checks do not replace user authorization. OMP treats screen text, images, notifications, websites, documents, chat messages, and application instructions as untrusted data. They cannot authorize actions or override your direct instructions.

The agent must confirm at the point of risk before consequential side effects unless your direct message already authorized that exact action, target, scope, and values. Examples include sending or publishing, purchases or transfers, deletion, account/security or permission changes, disclosure of private data, accepting legal terms, and irreversible operations. High-impact financial, employment, housing, education, insurance/credit, legal, medical, government, election, biometric, and highly sensitive-data actions require point-of-risk confirmation.

Operational guidance:

• Do not place secrets in visible windows unless the task needs them.
• Never follow on-screen requests to reveal credentials, change policy, or ignore instructions.
• Review the exact destination and payload before Submit, Send, Buy, Delete, or Allow.
• Prefer a dedicated desktop session for untrusted sites or documents.
• Stop when the visible state differs from the user's stated target.

See Tool approval mode for general policy resolution.

Platform setup and support

Platform	Backend	Setup and current status
macOS x64/arm64	Quartz/CoreGraphics capture; Quartz/CGEvent and native input	Supported. Grant Screen Recording and Accessibility. Real remote desktop execution was verified on Apple hardware; see Verification boundary.
Linux x64 glibc, X11	xcap capture; native X11/libei input	Supported when a graphical session and DISPLAY are available. The GUI-linked addon is packaged separately and loaded only when the tool starts.
Linux x64 glibc, Wayland	XWayland capture; libei through the desktop portal	Supported with limitations: active XWayland DISPLAY required; portal/session bus required for input; select one display for coordinate input. Pure Wayland capture is not implemented.
Linux arm64	Portable core addon only	Packaged native desktop capture/input is unsupported.
Linux musl	Portable core addon only	Explicitly unsupported because the capture dependency requires dynamically linked graphical-session libraries.
Windows x64	xcap capture; Win32 virtual-desktop pointer movement and native input	Implemented, including negative origins and secondary monitors. Not remotely exercised in this feature's verification.
Other OS/architectures	none	Unsupported by the published native package matrix.

macOS permissions

Open System Settings → Privacy & Security:

1. Grant Screen Recording to the terminal or application that launches omp.
2. Grant Accessibility to the same host for keyboard and pointer input.
3. Fully restart that host and start a new OMP session.

OMP performs a non-prompting Screen Recording preflight. It does not open the permission dialog for you. Accessibility is not separately preflighted; denial normally surfaces when native input initializes or emits an event.

Linux setup

For X11, run OMP inside the target graphical session and ensure DISPLAY identifies it.

For Wayland:

• run an x64 glibc build;
• keep XWayland enabled and ensure DISPLAY is set for capture;
• ensure the D-Bus session bus and org.freedesktop.portal.Desktop are running;
• use a desktop portal/compositor with libei input support; and
• select one display before coordinate input.

OMP probes portal availability but does not treat the probe as user consent or automatically approve an OS permission dialog.

The normal Linux core addon stays GUI-library-free. The Linux x64 desktop addon is loaded lazily when DesktopSession is first constructed. A missing published desktop addon falls back to the portable stub and reports DESKTOP_BACKEND_UNAVAILABLE; an addon that exists but cannot be loaded reports Failed to load packaged Linux desktop addon with candidate errors.

Session and worker lifecycle

The tool is exclusive: computer calls do not run concurrently. Its lifecycle is:

computer tool
  → ComputerSupervisor (lazy, serialized queue)
  → dedicated Bun worker
  → native DesktopSession
  → dedicated native desktop worker thread
  → capture/input APIs

The Bun worker starts on the first computer call, not at OMP startup. Startup has a 10-second deadline. The desktop session and last screenshot geometry remain alive across calls, so later coordinates can be checked against the preceding frame. Each action batch is ordered and always ends with a new capture.

Closing the agent/eval owner closes all owned controllers. Normal close asks the Bun worker to close, waits up to 1.5 seconds, then terminates it if needed. Native close is idempotent and bounded. Aborting a call terminates that worker and rejects pending requests; a later call may start a fresh worker and must establish a new screenshot frame.

OpenAI screenshot references and Files

OMP preserves the GA wire contract exactly:

• call: computer_call with action or batched actions, stable id/call_id, and pending_safety_checks;
• result: computer_call_output with output.type: "computer_screenshot" and acknowledged_safety_checks;
• screenshot reference: either image_url or file_id.

Native OMP execution returns the PNG inline as a data:image/png;base64,... image_url. It does not upload the capture to the OpenAI Files API and does not mint a file_id.

If an OpenAI-compatible gateway or restored Responses history supplies a file_id, OMP preserves and replays that exact reference as provider metadata. It does not download, validate, refresh, or delete the provider file. File availability, retention, authorization, and expiry remain the provider/client's responsibility. Both image_url and file_id history are preserved for capable models and converted to text notes when moving to a model without computer support.

Troubleshooting

Computer backend errors begin with a stable code:

Error	Meaning and response
DESKTOP_INVALID_OPTIONS	Invalid backend, zero image limit, malformed display value, or inactive display ID. Correct config and start a new session.
DESKTOP_INVALID_ACTION	Unknown action/button/key, missing or unexpected fields, negative point, short drag path, or invalid/duplicate modifier. Capture again only after fixing the action.
DESKTOP_BACKEND_UNAVAILABLE	No graphical session/backend, unsupported build, missing portal/XWayland, unsafe multi-display Wayland mapping, or native input initialization failure. Follow the platform section.
DESKTOP_PERMISSION_DENIED	Screen capture or input permission denied. Grant OS permissions and restart the host/session.
DESKTOP_CAPTURE_FAILED	Display capture, scaling, allocation, or PNG encoding failed. Reduce maxWidth/maxHeight, verify the display is active, then capture again.
DESKTOP_INPUT_FAILED	Native input initialization/event failed. Check Accessibility/portal/compositor permissions and session access.
DESKTOP_LAYOUT_CHANGED	Display topology changed after the reference screenshot. Capture a new frame before input.
DESKTOP_COORDINATE_OUT_OF_BOUNDS	Point lies outside the PNG, in a composite gap, or outside every display. Choose a point inside a listed pixel* rectangle.
DESKTOP_SESSION_CLOSED	Native session was closed. Start a new OMP session.
DESKTOP_WORKER_FAILED	Native worker startup, communication, timeout, or shutdown failed. Start a new session; if persistent, verify the native addon installation.

Common exact failures:

• Wayland capture through xcap 0.9.6 requires an active XWayland DISPLAY; pure Wayland capture is unavailable → enable XWayland or use X11.
• Wayland/libei absolute input cannot safely correlate a multi-display XWayland composite → set a single display or use X11.
• org.freedesktop.portal.Desktop is not available for native libei input → start/install the desktop portal in the same user session.
• macOS Screen Recording permission is not granted for this process → grant the launching host Screen Recording and restart it.
• Provider safety checks require interactive approval before computer input → use an interactive session and approve the provider prompt.
• Timed out starting native computer worker → verify the installed native addon matches the OMP release, then restart/reinstall.
• Version-sentinel error mentioning an upgrade while the session was running → restart OMP; disk is already consistent.
• Version-sentinel error saying the .node file is from a different release → reinstall OMP/native packages.

The native composite safety ceiling is 268,435,456 pixels. Normal defaults are far below it. Very large or sparse monitor arrangements should use a smaller maximum size or one selected display.

Verified limitations

• Native desktop control only; no DOM, ARIA tree, selectors, browser tab lifecycle, or Puppeteer fallback.
• OpenAI GA action set only; no arbitrary shell command or accessibility-tree action inside this tool.
• The model acts on screenshots; OCR/visual interpretation can be wrong.
• Coordinate targets are valid only for the preceding frame and current display layout.
• Screenshot composites may downscale small text to fit configured limits.
• Gaps are visible but not valid input targets; overlapping non-mirrored layouts fail closed.
• Pure Wayland capture currently requires XWayland; it is not a native portal capture path.
• Multi-output Wayland coordinate input fails closed; select one display or X11.
• Published Linux desktop support is x64 glibc only; Linux arm64 and musl are unsupported.
• Windows support is implemented for x64 but was not remotely exercised for this change.
• Native captures use inline image_url; OMP does not upload them to provider Files.
• OS secure desktops and policy-protected surfaces may reject ordinary user-session capture/input; OMP has no bypass.

Verification boundary

The real-host verification used the ComputerSupervisor worker path on a real macOS host, not a mock backend. With macOS Screen Recording and Accessibility granted, it controlled TextEdit using a global hotkey, double-click, click, typing, and screenshot capture. The returned Quartz frame was 1920×1080.

This proves the native macOS host path through the worker and desktop session. It was not a live OpenAI native computer_call → computer_call_output round trip. OpenAI GA transport, batching, safety acknowledgement, and image_url/file_id replay are covered by local contract tests; the Windows backend was implemented but not remotely exercised.

For implementation-level inputs, outputs, lifecycle, and error surfaces, see docs/tools/computer.md.