# llm-router

One front door for every LLM provider. The router owns routing, the provider
registry, credential resolution, the model catalog, streaming relay, retries,
and a single failure contract — consumers call one chat surface and never talk
to a provider directly.

llm-router is a standalone iii worker. Providers plug in as separate workers
at runtime through a self-registration protocol
(`iii trigger compose::add worker=provider-<x>`); the router never compiles against a provider, and removing a
provider worker removes the provider.

## Install

```bash
iii trigger compose::add worker=llm-router
```

## Quickstart

A consumer streams a turn by creating an iii channel, handing the router the
channel's **write** endpoint, and reading frames from the **read** endpoint
while `router::chat` runs. Any SDK works; Node shown:

```ts
import { createChannel } from 'iii-sdk';

const { reader, writerRef } = await createChannel(iii);
reader.onMessage((frame) => {
  const event = JSON.parse(frame); // AssistantMessageEvent
  if (event.type === 'text_delta') process.stdout.write(event.delta);
});

const res = await iii.trigger('router::chat', {
  writer_ref: writerRef, // direction "write"
  model: 'claude-sonnet-4',
  messages: [{ role: 'user', content: [{ type: 'text', text: 'Hello' }], timestamp: Date.now() }],
}, { timeout_ms: 620_000 }); // outer timeout ≥ the router's 600s stream budget
// res: { ok, provider, model, stop_reason, usage }
```

The streaming contract: every stream ends with exactly one terminal frame
(`done` or `error`). When the router has to kill a stream itself (idle
timeout, provider crash), it synthesizes the terminal frame and attaches the
partial content, so consumers never hang on a half-open stream.

## Functions

### Consumer surface

| Function | Purpose |
|---|---|
| `router::chat` | Stream a turn into the caller's channel; returns the turn summary. |
| `router::complete` | Non-streaming convenience over the same pipeline; returns the final message. |
| `router::abort` | Cancel an in-flight turn by `request_id`. |
| `router::route` | Read-only routing preview: `{model, provider?}` → `{provider, candidates}`, same rules and error codes as `router::chat`. Pin the result as the explicit `provider` on the chat call when you need the provider before streaming. |
| `router::models::list` | List catalog models, filterable by `provider` / `capability` / `modality` (`chat` by default, `stt`, `tts`, `any`). |
| `router::models::get` | Fetch one model record (`null` when unknown). |
| `router::models::supports` | Check one capability flag for one model. |
| `router::embed` | Batch text embeddings through the first embed-capable provider in the registry (`{model?, provider?, input[]}` → `{provider, model, embeddings[][]}`); providers without an embed surface are skipped. |
| `router::transcribe` | Speech to text: `{model?, provider?, audio_base64, mime?, language?, prompt?}` → `{provider, model, text, segments[]?, language?, duration_secs?}`. The provider comes from the named `provider`, else from the catalog owner of the `stt` model, else the first provider that declared one. |
| `router::speak` | Text to speech: `{model?, provider?, text, voice?, format?, language?, speed?}` → `{provider, model, audio_base64, mime, voice?, duration_secs?}`. Same resolution over `tts` models. |
| `router::count_tokens` | Count prompt tokens: `{model, provider?, system_prompt?, tools?, messages}` → `{provider, model, tokens, estimator}`, resolved with the same routing rules as `router::chat` and forwarded to `provider::<id>::count_tokens`. Never runs the model and costs nothing; `estimator` is `provider` (metering API) or `tiktoken` (local tokenizer). A provider without the surface is a typed `router/no_token_counter` error, so callers can fall back to their own estimate. |
| `router::provider::list` | Registered providers with `configured` / `available` status. |

Only the read surface is agent-callable (`router::models::list` / `get` /
`supports`, `router::provider::list`); everything else is denied to in-run
agents — see [Security model](#security-model).

### Error conventions

Errors use stable `router/<code>` identifiers. Streaming failures return an
`error` object with a user-facing `message`, the shared `kind`, `retryable`,
and an optional diagnostic `detail`. User interfaces should show `message`
and keep `code` / `detail` behind technical details; automation should branch
on `code` and `retryable`, never parse the prose.

The main recovery codes are `router/provider_unavailable`,
`router/capacity_exceeded`, `router/stream_idle_timeout`,
`router/stream_incomplete`, `router/provider_auth_expired`,
`router/provider_rate_limited`, `router/context_overflow`, and
`router/provider_rejected`.

### Provider protocol

Token-gated after the first declare: the response to `register` carries a
registration token, and every later protocol call must present it.

| Function | Purpose |
|---|---|
| `router::provider::register` | Self-declaration at attach time; idempotent re-declare with the token. |
| `router::provider::resolve` | Per-request credential + endpoint resolution (config > env > none). |
| `router::provider::update_credential` | Persist a refreshed credential (OAuth write-back). |
| `router::models::reconcile` | Replace the provider's catalog slice in one write. |

The provider worker itself exposes `provider::<id>::stream` and, when
capable, `provider::<id>::refresh_models` (model discovery),
`provider::<id>::count_tokens` (prompt token counting),
`provider::<id>::embed` (embeddings), and the speech pair
`provider::<id>::transcribe` / `provider::<id>::speak`.

## Configuration

All operator configuration lives in the engine's `llm-router` configuration
entry — no env vars, no config file. The entry schema is composed at runtime
from each registered provider's declaration. The router fetches the complete
entry at boot and keeps an in-memory snapshot synchronized by the
`configuration:updated` trigger; request handlers never poll
`configuration::get`.

```json
{
  "default_provider": "anthropic",
  "providers": {
    "anthropic": {
      "api_key": "sk-…",
      "api_url": "https://api.anthropic.com/v1/messages",
      "max_tokens": 8192
    }
  },
  "routing_heuristics": [{ "pattern": "^gpt-", "provider": "openai" }],
  "settings": {
    "stream_timeout_ms": 600000,
    "idle_timeout_ms": 120000,
    "retry_max": 2,
    "output_token_max": 32000
  }
}
```

| Setting | Default | Meaning |
|---|---|---|
| `stream_timeout_ms` | `600000` | Hard budget for one streamed turn. |
| `idle_timeout_ms` | `120000` | Max silence between provider frames before the attempt is cut. |
| `retry_max` | `2` | Retries per turn for retryable failures before the first forwarded frame (`0`–`10`). |
| `output_token_max` | unset | Optional ceiling on `max_output_tokens`. Unset forwards each model's own output ceiling, so providers run without an artificial limit; set it only to cap spend. |

Pasting a key into a provider's slice is the whole onboarding flow: the
router diffs the changed slice, debounces ~2 s, and kicks that provider's
`provider::<id>::refresh_models` discovery; discovered models land in the
catalog via `router::models::reconcile` and show up in `router::models::list`
within seconds — no restart.

### Operational notes

- **Env-var credential fallback resolves in the router's process.** A
  provider's `credential_env_var` (e.g. `ANTHROPIC_API_KEY`) is read by the
  llm-router binary, not by the provider worker — launch the router with
  those variables set, or put keys in the entry. A key present only in
  another worker's environment shows up as `configured: false`.
- **Registration-token recovery.** Re-registering a provider id without its
  original token is rejected (anti-takeover). If a provider durably lost its
  token, delete the router's registry state (state scope `llm-router`,
  key `registry`) and restart the affected providers to re-bind; pasted
  credentials in the configuration entry are unaffected.

## Security model

The agent-callable surface is governed by the engine-wide permission policy
(repo-root [`iii-permissions.yaml`](https://github.com/iii-hq/workers/blob/main/iii-permissions.yaml)), not a per-worker
file. In-run agents may only **read** the catalog and provider list:

- **Allowed:** `router::models::list`, `router::models::get`,
  `router::models::supports`, `router::provider::list`.
- **Denied to agents:** the chat/spend surface (`router::chat`,
  `router::complete`, `router::abort`, `router::route`), the whole
  provider/credential protocol (`router::provider::resolve` / `register` /
  `update_credential`, `router::models::reconcile`), and direct `provider::*`
  calls.

Worker-to-worker calls bypass the agent gate, so the harness, context-manager,
and provider workers reach the full surface — only in-run agents are
restricted. Provider credentials live in the configuration entry and are never
readable back through any allowed function.

## Events

The router registers three custom trigger types and fans out to every bound
handler. Bind with the standard two-step pattern; the handler receives the
payload verbatim (no envelope).

| Trigger type | Fires when | Payload |
|---|---|---|
| `router::models::changed` | a provider reconciles its catalog slice | `{ "provider": "<id>", "count": <n> }` |
| `router::provider::changed` | the registry changes (declare / availability flip) | `{ "provider": "<id>", "op": "register" \| "available" \| "unavailable" }` |
| `router::ready` | the router finishes booting; providers re-declare on it | `{}` |

```ts
iii.registerFunction('my-worker::on-models-changed', async (payload) => {
  console.log('catalog changed:', payload); // { provider, count }
  return {};
});

iii.registerTrigger({
  type: 'router::models::changed',
  function_id: 'my-worker::on-models-changed',
  config: {},
});
```

## Writing a provider worker

A provider worker must:

1. Register `provider::<id>::stream` honouring the channel-writer contract:
   forward upstream output as `AssistantMessageEvent` frames into the
   `writer_ref` it receives, ending with one terminal frame.
2. Declare itself at startup via `router::provider::register` — retrying with
   backoff until acknowledged (covers provider-before-router boot order) —
   and re-declare on the `router::ready` event after a router restart. The
   declaration may carry `icon_svg`: one self-contained, monochrome `<svg>`
   document (square `viewBox`, no fixed size, at most 32 KiB). The router
   echoes it in `router::provider::list` and the console paints it as a
   `currentColor` mask beside the provider's models; a missing or malformed
   mark falls back to the provider's initial.
3. Resolve credentials per request via `router::provider::resolve`; never
   read keys directly.
4. Treat closure of its stream channel as cancellation: abort the upstream
   request and stop writing frames.
5. Map upstream failures to the shared `ErrorKind` taxonomy on its `error`
   frames. Transport retries (429 / 5xx / connect) are the router's job, not
   the provider's.

A provider may also register `provider::<id>::count_tokens` (`{model,
system_prompt?, tools?, messages}` → `{model, tokens, estimator}`) to serve
`router::count_tokens`: an exact count from a provider metering API
(`estimator: "provider"`) or a local tokenizer estimate (`estimator:
"tiktoken"`). Providers without it simply make `router::count_tokens` return
a typed `router/no_token_counter` error for that provider.

### Speech providers

A speech provider is the same kind of worker with no chat stream: it
declares its models with a `speech` block and serves one or both speech
surfaces. `router::models::list` hides speech models from chat pickers
unless asked (`modality: "stt" | "tts" | "any"`), and `router::chat`
refuses them with a message that names the right door.

```json
{ "id": "scribe-v1", "provider": "elevenlabs", "context_window": 0, "max_output_tokens": 0,
  "speech": { "modality": "stt", "languages": ["en", "hi"], "streaming": true } }
```

| Function | Request | Response |
|---|---|---|
| `provider::<id>::transcribe` | `{model?, audio_base64, mime, language?, prompt?}` | `{model, text, segments[{text, start_secs?, end_secs?}]?, language?, duration_secs?}` |
| `provider::<id>::speak` | `{model?, text, voice?, format, language?, speed?}` | `{model, audio_base64, mime, voice?, duration_secs?}` |

`model` arrives as the id the caller named, or `null` for the provider's
default. `format` is what the caller asked for (`mp3`, `wav`, `pcm16`,
`opus`); answer with the container you produced and say so in `mime`.
Credentials resolve through `router::provider::resolve` exactly as for
chat providers.

The first real provider implementing this protocol is
[`provider-anthropic/`](https://github.com/iii-hq/workers/tree/main/provider-anthropic) — useful as a reference
implementation alongside the scripted provider in the integration tests.
[`provider-openai/`](https://github.com/iii-hq/workers/tree/main/provider-openai) follows the same structure for the
OpenAI Chat Completions API (native structured output, reasoning_effort).
[`provider-opencode-go/`](https://github.com/iii-hq/workers/tree/main/provider-opencode-go) follows the same structure for the
OpenCode Go Chat Completions API (curated model metadata, reasoning_effort).

## Local development & testing

```bash
cargo test                       # unit suite, no engine needed
cargo test --test integration    # engine-backed suite; self-skips without an engine
```

The integration suite spawns a throwaway engine per test when `iii` is on
`PATH` (or `III_ENGINE_BIN` points at a binary) and covers the chat relay,
cancellation, abort, restart recovery, registration token gating, paste-a-key
discovery, and event delivery end to end.

To run the worker locally against an engine:

```bash
cargo run -- --url ws://127.0.0.1:49134
```

`--url` defaults to `ws://127.0.0.1:49134` and honours the `III_WS_URL`
environment variable when the flag is not set. `--config` is accepted per
the standard worker CLI but ignored with a warning — operator config lives
in the engine's `llm-router` configuration entry (see Configuration above).