# computer

Drive a full desktop on the iii bus. Where the
[browser](https://github.com/iii-hq/workers/tree/main/browser) worker gives an
agent a Chromium tab, `computer` gives it a whole screen: it hands the model a
screenshot, clicks and scrolls at the coordinates it reads off that image, and
types wherever the desktop's focus is. The harness discovers
`computer::*` as functions automatically, so a model sees the screen and acts on
it with no glue, and the console gets a live viewport of whatever the desktop is
doing.

The worker does the one thing no other worker can: capture the screen and move
the cursor. Three drivers sit behind one surface, picked per session:

- **native** (no `endpoint`, no `image`): drive the local machine this worker
  runs on, capture and input injection built in.
- **sandbox** (an `image`): boot a fresh desktop inside an
  [iii-sandbox](https://github.com/iii-hq/workers/tree/main/iii-sandbox) microVM
  and drive it through iii primitives alone (`sandbox::exec` / `sandbox::fs`),
  with no socket into the guest. A fixed virtual display means 1:1 coordinates,
  no HiDPI or multi-monitor ambiguity, and nothing to grant on the host.
- **remote** (an `endpoint`): drive an already-running desktop through the
  executor inside it, over a WebSocket. `computer` connects; that desktop boots
  out of band.

Everything else composes with workers that already exist: run commands and touch
files with the [shell](https://github.com/iii-hq/workers/tree/main/ide)
worker, persist with `state`, schedule with `cron`.

## Install

```bash
iii trigger compose::add worker=computer
```

To drive a throwaway desktop instead of your own machine, add the sandbox worker
too — it boots the microVM the `image` driver runs in:

```bash
iii trigger compose::add worker=iii-sandbox
```

**macOS, native driver only.** Capture and input are permission-gated and macOS
degrades both silently, so the worker checks and fails loud. Grant the app that
runs the worker **Screen Recording** (without it every screenshot is the
wallpaper with all windows stripped) and **Accessibility** (without it clicks and
typing are dropped) in System Settings > Privacy & Security, then restart it. The
sandbox driver needs neither.

**Linux, native driver.** On an X11 session (Xorg or Xvfb) a native session
captures and acts by coordinate like macOS/Windows; coordinates are physical
root pixels. Run a window manager: on a bare X server a click gives no keyboard
focus, so toolkits that wait for focus (egui, Bevy, Slint) ignore typed keys.
On a Wayland desktop (GNOME and KDE default to it) the session is
semantic-only: `sessions::start` reports `coordinates: { supported: false }`,
`screen` is 0x0, screenshots and coordinate actions are refused with
`unsupported_session`, and you read and act by ref. Ref actions need the
AT-SPI bus; when accessibility is off, `semantic.reason` carries a warning
with the command that turns it on for the login session (`gdbus call … IsEnabled
'<true>'`). The worker never runs it itself: it also persists
`toolkit-accessibility=true` in dconf. Chromium/Electron apps also need
`--force-renderer-accessibility`.

## Quickstart

Start a session, look at the screen, click what you see:

```bash
# drive this machine: a bare start picks the native driver when neither
# sandbox_image nor default_endpoint is configured
iii trigger computer::sessions::start --json '{}'
# -> { "session_id": "c1", "endpoint": "native", "os": "macos", "screen": { "width": 1512, "height": 945 }, "semantic": { "supported": true } }

# see the screen: an image block the vision model renders inline
iii trigger computer::screenshot --json '{"session_id":"c1"}'

# click a pixel read off that screenshot (top-left origin)
iii trigger computer::act --json '{"session_id":"c1","action":"click","x":640,"y":360}'

iii trigger computer::sessions::stop --json '{"session_id":"c1"}'
```

Swap the first call for `{"image":"desktop"}` to boot a sandboxed Linux desktop
instead (see [Sandbox desktop image](#sandbox-desktop-image)), or
`{"endpoint":"ws://host:8000"}` to drive a remote one. Everything after that call
is identical: the session id is the only handle.

The same from a sibling worker:

```rust
use iii_sdk::protocol::TriggerRequest;
use iii_sdk::{register_worker, InitOptions};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let started = iii
        .trigger(TriggerRequest {
            function_id: "computer::sessions::start".into(),
            payload: json!({}),
            action: None,
            timeout_ms: Some(30_000),
        })
        .await?;
    let session_id = started["session_id"].as_str().unwrap();

    iii.trigger(TriggerRequest {
        function_id: "computer::act".into(),
        payload: json!({ "session_id": session_id, "action": "click", "x": 640, "y": 360 }),
        action: None,
        timeout_ms: Some(10_000),
    })
    .await?;

    Ok(())
}
```

The rest of the surface: `computer::act` — pointer actions addressed by
coordinate (click, right_click, double_click, move, drag, scroll) and keyboard
actions that land wherever the desktop's focus is (`type` with `text`,
`press` / `hotkey` with `keys`, e.g. `["cmd","c"]`) —
`computer::observe` (a screenshot and, with `include_a11y`, an app's
accessibility tree), `computer::find`, `computer::displays`,
`computer::sessions::list` and `computer::sessions::stop`.

For commands and files, mind which machine you mean: on a native session the
desktop IS this host, so the
[shell](https://github.com/iii-hq/workers/tree/main/ide) worker is the right
tool. A sandboxed desktop is reached with `sandbox::exec` / `sandbox::fs`, and
a remote one through its own guest executor — `shell` would run on the host
instead, which is a different machine entirely.

On a native session the worker also acts **by element**:
`computer::observe { include_a11y: true, include_image: false }` or
`computer::find { selector: "button[name='Save']" }` return lines carrying
`[ref=…]`, and `computer::act { action: "invoke" | "set_value" | "toggle" |
"select", ref }` acts on that element through the OS accessibility API — the
worker moves no mouse, sends no keys and sets no focus. The effect is not
verified; confirm with another `find`. Refs live until an `observe` with
`include_a11y` or a `find` with matches replaces them, any coordinate or
keyboard action (`type`, `press`, `hotkey`, console input included) runs on
any native session (they share the desktop), a timed-out, cancelled or
crashed accessibility call resets the session's executor, or the session
stops or the worker restarts. macOS needs Accessibility **and** Screen
Recording for the process running the worker. On Windows a password field's
value is hidden only when its name looks like a password (or the app itself
masks it): xa11y does not expose UIA's `IsPassword`. On Linux a password field
is recognised by its AT-SPI role, whatever its name. Remote and sandbox
sessions stay coordinate-only.

`computer::run { app_name, goal, inputs }` drives one app toward a goal in a
single call: each step, the [judge](https://github.com/iii-hq/workers/tree/main/judge)
worker picks the next element and operation from the tree, the worker picks
the verb and acts by ref, and the run returns a status, its steps and the
tree with live refs. Text comes only from `inputs`, keyed by field label.
Without a judge worker it returns `judge_unavailable` and the tree.

## Console

The worker ships its own console page (page `computer`): a session rail, a live
viewport fed by the live screencast, and click / type / scroll forwarding
straight into the desktop. It is injected into any running console at
registration time — nothing to install, nothing to rebuild. Every `computer::*`
call in chat and traces renders through the same asset.

## Configuration

Stored in the `configuration` worker under the `computer` key; every field is
editable live from the console. The screencast rate applies to running
sessions; everything a driver is built with — endpoint, OS label, timeouts,
capture limits, and the sandbox display and network settings — is read at
`sessions::start`, so a change reaches sessions started after it.

```yaml
computer:
  default_endpoint: ''      # guest-executor endpoint for a remote desktop; empty = drive the local machine
  os: linux                 # guest OS label recorded on sessions
  max_sessions: 2           # concurrent desktop connections
  idle_stop_ms: 300000      # stop sessions idle this long; 0 disables
  screencast_fps: 15        # live-view frame rate cap
  max_screenshot_dimension: 1280 # downscale screenshots/frames to this longest edge (native)
  screenshot_quality: 70    # JPEG quality 1-100 (native)
  command_timeout_ms: 120000 # timeout for each driver action (fixed at connect)
  connect_timeout_ms: 15000  # driver connect timeout at session start
  sandbox_image: ''          # iii-sandbox image for a sandbox session; empty = name it per call
  sandbox_width: 1280        # sandbox virtual display width
  sandbox_height: 800        # sandbox virtual display height
  sandbox_network: true      # give the sandboxed desktop network access; false keeps it offline
  sandbox_idle_timeout_secs: 86400 # idle_timeout_secs for sandbox::create; kept high, the worker owns teardown
  screen_capture_preflight: true # macOS: ask for Screen Recording at native start, fail loud if denied
```

## Sandbox desktop image

The sandbox driver needs a desktop inside the guest (Xvfb, xdotool, imagemagick,
openbox). A prebaked image lives in [`images/desktop`](images/desktop): build it,
push it, register it as a `custom_images` entry on the iii-sandbox worker, then
start a session with `{ "image": "desktop" }`. iii-sandbox boots on macOS Apple
Silicon (libkrun) and Linux (`/dev/kvm`).

## Custom trigger types

Sibling workers (and the console page) subscribe to session activity instead of
polling. Both bindings accept an optional `{ "session_id": "..." }` filter.

| Trigger type | Fires when | Payload to subscribers |
|---|---|---|
| `computer::session-started` | A session connected and is ready | `{ session_id, endpoint, os, screen, timestamp }` |
| `computer::session-stopped` | A session ended | `{ session_id, reason: "stopped" \| "idle", timestamp }` |

## Live screen

`computer::screencast::start` / `stop`, `computer::frame` and the
`computer::frame-changed` trigger type drive the live viewport. The worker
captures at `screencast_fps` and keeps only the newest frame per session in
memory (no history). After each frame it fires `computer::frame-changed` with a
small, image-free notification; the viewer reads the frame with
`computer::frame` (notify, then fetch), so the console and any number of
watchers follow the desktop without polling, and a slow watcher skips frames
instead of queueing them.

| Trigger type | Fires when | Payload to subscribers |
|---|---|---|
| `computer::frame-changed` | A new frame was stored (`change: "updated"`), or the frame was dropped because the screencast or session stopped (`change: "cleared"`) | `{ session_id, epoch, frame_seq, change, width, height, mime?, bytes?, timestamp }` |

The binding config `{ "session_id": "..." }` is **required** (a viewer follows
one desktop). Order frames by `(epoch, frame_seq)`; `epoch` changes when the
worker restarts or restores the session. Bind first, then read
`computer::frame` once for the current frame; read it again after each
notification (pass `since_frame` to skip an unchanged image) and after a
reconnect. These are internal console plumbing rather than agent surface, and
stay out of agent function lists.