# computer

Drive a full desktop on the iii bus. Where the
[browser](https://github.com/iii-hq/workers/tree/main/browser) worker gives an
agent a Chromium tab, `computer` gives it a whole screen: it hands the model a
screenshot, clicks and scrolls at the coordinates it reads off that image, and
types wherever the desktop's focus is. The harness discovers
`computer::*` as functions automatically, so a model sees the screen and acts on
it with no glue, and the console gets a live viewport of whatever the desktop is
doing.

The worker does the one thing no other worker can: capture the screen and move
the cursor. Three drivers sit behind one surface, picked per session:

- **native** (no `endpoint`, no `image`): drive the local machine this worker
  runs on, capture and input injection built in.
- **sandbox** (an `image`): boot a fresh desktop inside an
  [iii-sandbox](https://iii.dev/registry/workers/iii-sandbox) microVM
  and drive it through iii primitives alone (`sandbox::exec` / `sandbox::fs`),
  with no socket into the guest. A fixed virtual display means 1:1 coordinates,
  no HiDPI or multi-monitor ambiguity, and nothing to grant on the host.
- **remote** (an `endpoint`): drive an already-running desktop through the
  executor inside it, over a WebSocket. `computer` connects; that desktop boots
  out of band.

Everything else composes with workers that already exist: run commands and touch
files with the [shell](https://github.com/iii-hq/workers/tree/main/ide)
worker, persist with `state`, schedule with `cron`.

## Install

```bash
iii call compose::add worker=computer
```

To drive a throwaway desktop instead of your own machine, run the sandbox
adapter: the `sandbox-libkrun` worker serves the engine's `sandbox::*` calls in
the microVM the `image` driver runs in. `iii call compose::add worker=iii-sandbox`
installs it. To also name the desktop image, declare it yourself in
`worker-compose.yaml`; an unmarked container is yours, so Compose never removes
it:

```yaml
containers:
  sandbox-libkrun:
    worker: package://sandbox-libkrun
    version: '^0.3'
    config_override:
      auto_install: true
      image_allowlist: [desktop, anthropic]
      custom_images:
        desktop: ghcr.io/trycua/linux:24.04-slim-20261013-36bc520
        # Firefox and LibreOffice; give it computer's sandbox_memory_mb >= 2048.
        anthropic: ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest
```

`sandbox::catalog::list` shows what it can boot. For the image, see
[Sandbox desktop image](#sandbox-desktop-image).

**macOS, native driver only.** Capture and input are permission-gated and macOS
degrades both silently, so the worker checks and fails loud. Grant the app that
runs the worker **Screen Recording** (without it every screenshot is the
wallpaper with all windows stripped) and **Accessibility** (without it clicks and
typing are dropped) in System Settings > Privacy & Security, then restart it. The
sandbox driver needs neither.

**Linux, native driver.** On an X11 session (Xorg or Xvfb) a native session
captures and acts by coordinate like macOS/Windows; coordinates are physical
root pixels. Run a window manager: on a bare X server a click gives no keyboard
focus, so toolkits that wait for focus (egui, Bevy, Slint) ignore typed keys.
On a Wayland desktop (GNOME and KDE default to it) the session is
semantic-only: `sessions::start` reports `coordinates: { supported: false }`,
`screen` is 0x0, screenshots and coordinate actions are refused with
`unsupported_session`, and you read and act by ref. Ref actions need the
AT-SPI bus; when accessibility is off, `semantic.reason` carries a warning
with the command that turns it on for the login session (`gdbus call … IsEnabled
'<true>'`). The worker never runs it itself: it also persists
`toolkit-accessibility=true` in dconf. Chromium/Electron apps also need
`--force-renderer-accessibility`.

## Quickstart

Start a session, look at the screen, click what you see:

```bash
# drive this machine: a bare start picks the native driver when neither
# sandbox_image nor default_endpoint is configured; '{"native": true}' always does
iii call computer::sessions::start --json '{}'
# -> { "session_id": "c1", "endpoint": "native", "os": "macos", "screen": { "width": 1512, "height": 945 }, "semantic": { "supported": true } }

# see the screen: an image block the vision model renders inline
iii call computer::screenshot --json '{"session_id":"c1"}'

# click a pixel read off that screenshot (top-left origin)
iii call computer::act --json '{"session_id":"c1","action":"click","x":640,"y":360}'

iii call computer::sessions::stop --json '{"session_id":"c1"}'
```

Swap the first call for `{"image":"desktop"}` to boot a sandboxed Linux desktop
instead (see [Sandbox desktop image](#sandbox-desktop-image)), or
`{"endpoint":"ws://host:8000"}` to drive a remote one. Everything after that call
is identical: the session id is the only handle.

The same from a sibling worker:

```rust
use iii_sdk::{register_worker, CallRequest, InitOptions};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let started = iii
        .call(CallRequest {
            function_id: "computer::sessions::start".into(),
            payload: json!({}),
            action: None,
            timeout_ms: Some(30_000),
        })
        .await?;
    let session_id = started["session_id"].as_str().unwrap();

    iii.call(CallRequest {
        function_id: "computer::act".into(),
        payload: json!({ "session_id": session_id, "action": "click", "x": 640, "y": 360 }),
        action: None,
        timeout_ms: Some(10_000),
    })
    .await?;

    Ok(())
}
```

The rest of the surface: `computer::act` — pointer actions addressed by
coordinate (click, right_click, double_click, move, drag, scroll) and keyboard
actions that land wherever the desktop's focus is (`type` with `text`,
`press` / `hotkey` with `keys`, e.g. `["cmd","c"]`) —
`computer::observe` (a screenshot and, with `include_a11y`, an app's
accessibility tree), `computer::find`, `computer::displays`,
`computer::sessions::list` and `computer::sessions::stop`.

For commands and files, mind which machine you mean: on a native session the
desktop IS this host, so the
[shell](https://github.com/iii-hq/workers/tree/main/ide) worker is the right
tool. A sandboxed desktop is reached with `sandbox::exec` / `sandbox::fs`, and
a remote one through its own guest executor — `shell` would run on the host
instead, which is a different machine entirely.

On a native session the worker also acts **by element**:
`computer::observe { include_a11y: true, include_image: false }` or
`computer::find { selector: "button[name='Save']" }` return lines carrying
`[ref=…]`, and `computer::act { action: "invoke" | "set_value" | "toggle" |
"select", ref }` acts on that element through the OS accessibility API — the
worker moves no mouse, sends no keys and sets no focus. The effect is not
verified; confirm with another `find`. Refs live until an `observe` with
`include_a11y` or a `find` with matches replaces them, any coordinate or
keyboard action (`type`, `press`, `hotkey`, console input included) runs on
any native session (they share the desktop), a timed-out, cancelled or
crashed accessibility call resets the session's executor, or the session
stops or the worker restarts. macOS needs Accessibility **and** Screen
Recording for the process running the worker. On Windows a password field's
value is hidden only when its name looks like a password (or the app itself
masks it): xa11y does not expose UIA's `IsPassword`. On Linux a password field
is recognised by its AT-SPI role, whatever its name. Remote and sandbox
sessions stay coordinate-only.

`computer::run { app_name, goal, inputs }` drives one app toward a goal in a
single call: each step, the [judge](https://github.com/iii-hq/workers/tree/main/judge)
worker picks the next element and operation from the tree, the worker picks
the verb and acts by ref, and the run returns a status, its steps and the
tree with live refs. Text comes only from `inputs`, keyed by field label.
Without a judge worker it returns `judge_unavailable` and the tree; with a
dialog already open it returns `blocked` and touches nothing. The judge
it asks is `judge_provider` in this worker's configuration when set, else the
calling session's pick, else the judge hub's default. `computer::run` cannot
be observed (`NOT_OBSERVABLE`): its `inputs` carry what is typed into fields,
passwords included.

## Console

The worker ships its own console page (page `computer`): a session rail, a live
viewport fed by the live screencast, and click / type / scroll forwarding
straight into the desktop. With no desktop open (or after + in the rail) the
page shows a launcher: this machine, a sandbox picked from iii-sandbox's
catalog, or a remote endpoint. It is injected into any running console at
registration time — nothing to install, nothing to rebuild. Every `computer::*`
call in chat and traces renders through the same asset.

When an agent drives a session (`computer::act`, `computer::run`), a live
preview of it floats in a corner of the console, like the browser's and the
iOS simulator's: drag it, pinch or double-click to resize, tap to fan several
out, expand to open the page. A hidden card stays hidden while the agent keeps
acting, until two quiet minutes pass or a new `computer::run` starts. Nothing
pops for a session the page already shows in that window, or one without a
picture (Wayland).

## Configuration

Stored in the `configuration` worker under the `computer` key; every field is
editable live from the console. The screencast rate applies to running
sessions; everything a driver is built with — endpoint, OS label, timeouts,
capture limits, and the sandbox display and network settings — is read at
`sessions::start`, so a change reaches sessions started after it.

```yaml
computer:
  default_endpoint: ''      # guest-executor endpoint for a remote desktop; empty = drive the local machine
  os: linux                 # guest OS label recorded on sessions
  max_sessions: 2           # concurrent desktop connections
  idle_stop_ms: 300000      # stop sessions idle this long; 0 disables
  screencast_fps: 15        # live-view frame rate cap
  max_screenshot_dimension: 1280 # downscale screenshots/frames to this longest edge (native, remote; a sandbox is its display size)
  screenshot_quality: 70    # JPEG quality 1-100 (native, sandbox, remote)
  command_timeout_ms: 120000 # timeout for each driver action (fixed at connect)
  connect_timeout_ms: 15000  # driver connect timeout at session start
  sandbox_image: ''          # a name in iii-sandbox's catalog (custom_images) a bare sessions::start boots; empty = name it per call
  sandbox_width: 1280        # sandbox virtual display width
  sandbox_height: 800        # sandbox virtual display height
  sandbox_network: true      # give the sandboxed desktop network access; false keeps it offline
  sandbox_memory_mb: 2048    # RAM of the sandbox VM; 0 = iii-sandbox's default (512); a browser needs ~2048
  sandbox_cpus: 2            # vCPUs of the sandbox VM; 0 = iii-sandbox's default (1)
  sandbox_idle_timeout_secs: 86400 # idle_timeout_secs for sandbox::create; kept high, the worker owns teardown
  screen_capture_preflight: true # macOS: ask for Screen Recording at native start, fail loud if denied
  judge_provider: ''         # judge-<provider> computer::run asks (clef, decider, typesafe…); empty = the session's pick, then the judge default
```

## Sandbox desktop image

The sandbox driver runs `Xvfb`, `xdpyinfo`, `setsid`, `pgrep`, `xdotool`,
ImageMagick's `import` and `base64` inside the guest, then the image's own
desktop, the first it has: XFCE (`startxfce4`), mutter with a `tint2` panel, or
`openbox`. An image with none shows a black screen and frameless windows. Any
image with those tools, for arm64 (Apple Silicon) and amd64, works as is:

- **Ready to use, nothing to build:** [`ghcr.io/trycua/linux:24.04-slim-20261013-36bc520`](https://github.com/trycua/cua)
  (MIT, about 490 MB, both architectures; pin a dated tag, `24.04-slim` moves).
  It boots into XFCE (panel, dock with a terminal, files and a browser).
  Verified with a sandbox session that opens a window and closes it by a click.
- **With a browser:** [`ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest`](https://github.com/anthropics/claude-quickstarts/tree/main/computer-use-demo)
  (MIT, about 2.9 GB unpacked) adds Firefox, LibreOffice and xcalc, and boots
  into mutter with its own tint2 launcher panel. Firefox is OOM-killed at iii-sandbox's 512 MB: keep
  `sandbox_memory_mb` at 2048 or more. Verified on Apple Silicon with xcalc
  driven by clicks and Firefox navigated by typing a URL; the first boot pulls
  the image (about 90 s), later ones take under a second.

Register the image as the `desktop` entry of `custom_images` (see
[Install](#install)), then start a session with `{ "image": "desktop" }`.
In the console, the computer worker's settings (Sandbox desktop) pick the
default image among iii-sandbox's `custom_images`, shown as `name — OCI ref`,
and size the VM (memory, CPUs); a new OCI image goes into `custom_images`
first, since iii-sandbox only boots names from its catalog.
iii-sandbox boots on macOS Apple Silicon (libkrun) and Linux (`/dev/kvm`).

## Observable functions

Sibling workers (and the console page) observe session activity instead of
polling. Each event is a call of a private function whose handler returns the
event: only this worker calls them, anyone may observe them. A function
nobody observes costs nothing: the event is never built.

| Function | Args | Delivery | Event |
|---|---|---|---|
| `computer::on-session-started` | none; filter on `input.session_id` | iii's default | `{ session_id, endpoint, os, screen, timestamp }` when a session connected and is ready |
| `computer::on-session-stopped` | none; filter on `input.session_id` | iii's default | `{ session_id, reason: "stopped" \| "idle", timestamp }` when a session ended |
| `computer::on-activity` | none; filter on `input.session_id` | iii's default | `{ session_id, function, timestamp }` when a `computer::act` call reached a session, or a `computer::run` started on it |
| `computer::on-frame-changed` | `{ session_id }` (required) | the worker's own, below | `{ session_id, epoch, frame_seq, change, width, height, mime?, bytes?, timestamp }` |

```ts
// each target receives the Observed envelope; the event is its `output`

// every session that ends
iii.observe(
  { function_id: 'computer::on-session-stopped', capture: ['output'] },
  'my-worker::on-desktop-gone',
)

// what happens on one session
iii.observe(
  {
    function_id: 'computer::on-activity',
    filters: { 'input.session_id': { $eq: 'c1' } },
    capture: ['output'],
  },
  'my-worker::on-desktop-activity',
)
```

## Live screen

`computer::screencast::start` / `stop`, `computer::frame` and the observable
function `computer::on-frame-changed` drive the live viewport. The worker
captures at `screencast_fps` and keeps only the newest frame per session in
memory (no history). After each frame it calls `computer::on-frame-changed`
with a small, image-free notification; the viewer reads the frame with
`computer::frame` (notify, then fetch), so the console and any number of
watchers follow the desktop without polling, and a slow watcher skips frames
instead of queueing them.

Each viewer (the page, a corner preview, in any console window) calls `start`
when it opens and `stop` when it closes; the capture runs while one remains,
and one nobody follows any more (a window closed without a `stop`) stops by
itself after 10 seconds.

`computer::on-frame-changed` is called when a new frame was stored
(`change: "updated"`), or when the frame was dropped because the screencast or
session stopped (`change: "cleared"`). Its args `{ "session_id": "..." }` are
**required** (a viewer follows one desktop), and the worker delivers each
change itself:

- a change goes only to the observers of its session;
- each observer has one delivery slot: one call through the observer in
  flight (bounded by 5 s) and one pending notification, which a newer one
  replaces, so a slow viewer skips frames and never queues them;
- an observer's filters apply as usual; its target receives the notification
  itself, and the caller gets `{ offered }`, the number of observers it was
  offered to;
- at most 64 observers; the 65th is rejected with `OBSERVER_LIMIT`, and a
  blank `session_id` with `INVALID_ARGS`.

```ts
iii.observe(
  { function_id: 'computer::on-frame-changed', args: { session_id: 'c1' } },
  'my-worker::on-frame',
)
```

Order frames by `(epoch, frame_seq)`; `epoch` changes when the worker restarts
or restores the session. Observe first, then read `computer::frame` once for
the current frame; read it again after each notification (pass `since_frame`
to skip an unchanged image) and after a reconnect. These are internal console
plumbing rather than agent surface, and stay out of agent function lists.
