Deferred libei and RemoteDesktop setup until the first native input operation. Read-only capability, window, display, and AX calls no longer request keyboard or pointer access.
Used non-persistent portal grants and retained the portal session so backend teardown closes it explicitly.
Fixes#7884
Reported compositor-limited per-window input before reaching the AT-SPI focus path and stopped advertising foreground delivery on Wayland.
Removed the obsolete AT-SPI window-raise helper and updated public recovery guidance.
Fixes#7702
WaylandBackend::capabilities() hardcoded capture:true, but the PipeWire
screencast path is compiled only under the wayland-pipewire feature, which
is off by default and excluded from shipped Bazel addons (crate_features=[]).
Released builds therefore advertised capture the binary could never do:
every capture() call returned CaptureFailed, and callers trusting
capabilities() retried into a guaranteed failure.
Gate the capture flag and capture_permission on cfg!(feature =
"wayland-pipewire") so the report matches the compiled-in path, and align
docs/computer-use.md with what shipped builds actually support.
Fixes#7700
- Added window discovery, targeting, and frame validation for macOS, Windows, and Linux platforms.
- Updated computer tool and protocol definitions to support listing and targeting specific application windows.
- Refactored native computer tool exposure to function-only representation across models.
- Updated documentation and test suites to reflect window-scoped computer control capabilities.
Under a rootless XWayland session (the GNOME/KDE/sway default) the X11
root window has no backing pixmap, so core `GetImage` on the root returns
`BadMatch`. The computer tool advertised Wayland support yet failed every
screenshot with a raw X11 protocol dump, and coordinate actions stayed
gated behind a capture that could never succeed.
- `Monitor::all` now probes a 1x1 root `GetImage` at initialization and,
on a `Match`/`Drawable` error, fails fast with an actionable
`DESKTOP_BACKEND_UNAVAILABLE` message naming the rootless-XWayland
constraint via the new `root_capture_error` classifier.
- `capture_image` routes its `GetImage` failure through the same
classifier so any surviving path yields the actionable message rather
than a raw protocol dump; unrelated errors stay verbatim.
- Corrected the module doc premise and `docs/computer-use.md` to list
rootless XWayland as unsupported (capture needs a rooted/rootful X
server, which only exposes X11 clients).
Fixes#7085
- Replaced the separate GUI-linked pi_natives.desktop.linux-x64 addon with
a pure-Rust X11 backend (x11rb RustConnection capture via RandR/GetImage,
XTest input with keysym mapping) compiled into the core addon on every
published target; Linux arm64 and musl are now supported and headless
hosts load the addon unaffected.
- Removed the native-desktop-linux cargo feature, desktop_unsupported.rs,
lazy desktop loader, second napi build, desktop packaging/CI steps, GUI
build dependencies, and the now-unreferenced vendored libspa crate;
reverted setup-system-deps to main.
- Preserved the desktop input hardening semantics on the unified backend:
XTest layouts reject negative origins and coordinates beyond 0..=32767,
batch coordinates stay bound to the frame last returned to JS with
intermediate screenshots deferred, coordinate input requires a
previously returned frame, and failed chord releases still release
every held key.
- Enforced a 60s worker-side execute deadline (DESKTOP_DEADLINE_EXCEEDED):
no input is emitted after expiry and wait-heavy batches are rejected
upfront.
- Added int32 fail-closed validation for coordinates, drag points, and
scroll deltas at the JS ingress and gateway schema.
- Exposed computer to models without native OpenAI computer-use support as
a regular function tool with a typed GA action schema across OpenAI,
Azure, and Codex Responses providers, including named forced choice.
- Added the /computer slash command (on/off/status/toggle) for
session-only enablement via runtime tool registration in SessionTools.
- Updated docs, changelogs, and contract tests accordingly.
roboomp