feat(computer): unified native addon, /computer toggle, function tool

- Replaced the separate GUI-linked pi_natives.desktop.linux-x64 addon with
  a pure-Rust X11 backend (x11rb RustConnection capture via RandR/GetImage,
  XTest input with keysym mapping) compiled into the core addon on every
  published target; Linux arm64 and musl are now supported and headless
  hosts load the addon unaffected.
- Removed the native-desktop-linux cargo feature, desktop_unsupported.rs,
  lazy desktop loader, second napi build, desktop packaging/CI steps, GUI
  build dependencies, and the now-unreferenced vendored libspa crate;
  reverted setup-system-deps to main.
- Preserved the desktop input hardening semantics on the unified backend:
  XTest layouts reject negative origins and coordinates beyond 0..=32767,
  batch coordinates stay bound to the frame last returned to JS with
  intermediate screenshots deferred, coordinate input requires a
  previously returned frame, and failed chord releases still release
  every held key.
- Enforced a 60s worker-side execute deadline (DESKTOP_DEADLINE_EXCEEDED):
  no input is emitted after expiry and wait-heavy batches are rejected
  upfront.
- Added int32 fail-closed validation for coordinates, drag points, and
  scroll deltas at the JS ingress and gateway schema.
- Exposed computer to models without native OpenAI computer-use support as
  a regular function tool with a typed GA action schema across OpenAI,
  Azure, and Codex Responses providers, including named forced choice.
- Added the /computer slash command (on/off/status/toggle) for
  session-only enablement via runtime tool registration in SessionTools.
- Updated docs, changelogs, and contract tests accordingly.
This commit is contained in:
can1357
2026-07-24 01:40:05 +00:00
committed by usr-bin-roygbiv
parent 4fe03e25cb
commit 681d7daf65
72 changed files with 1894 additions and 12477 deletions
@@ -12,8 +12,7 @@ export function buildNamedToolChoice(toolName: string, model?: Model<Api>): Tool
return { type: "tool", name: toolName };
}
if (toolName === "computer") {
if (model.supportsComputerUse !== true) return undefined;
if (toolName === "computer" && model.supportsComputerUse === true) {
if (
model.api === "openai-codex-responses" ||
model.api === "openai-responses" ||
@@ -23,6 +22,8 @@ export function buildNamedToolChoice(toolName: string, model?: Model<Api>): Tool
}
return undefined;
}
// Models without native computer support receive the tool as a regular
// function tool, so the generic named-function forcing below applies.
if (
model.api === "openai-codex-responses" ||