# llm-router

One front door for every LLM provider. The router owns routing, the provider
registry, credential resolution, the model catalog, streaming relay, retries,
and a single failure contract — consumers call one chat surface and never talk
to a provider directly.

llm-router is a standalone iii worker. Providers plug in as separate workers
at runtime through a self-registration protocol
(`iii trigger compose::add worker=provider-<x>`); the router never compiles against a provider, and removing a
provider worker removes the provider.

## Install

```bash
iii trigger compose::add worker=llm-router
```

## Quickstart

A consumer streams a turn by creating an iii channel, handing the router the
channel's **write** endpoint, and reading frames from the **read** endpoint
while `router::chat` runs. Any SDK works; Node shown:

```ts
import { createChannel } from 'iii-sdk';

const { reader, writerRef } = await createChannel(iii);
reader.onMessage((frame) => {
  const event = JSON.parse(frame); // AssistantMessageEvent
  if (event.type === 'text_delta') process.stdout.write(event.delta);
});

const res = await iii.trigger('router::chat', {
  writer_ref: writerRef, // direction "write"
  model: 'claude-sonnet-4',
  messages: [{ role: 'user', content: [{ type: 'text', text: 'Hello' }], timestamp: Date.now() }],
}, { timeout_ms: 620_000 }); // outer timeout ≥ the router's 600s stream budget
// res: { ok, provider, model, stop_reason, usage }
```

The streaming contract: every stream ends with exactly one terminal frame
(`done` or `error`). When the router has to kill a stream itself (idle
timeout, provider crash), it synthesizes the terminal frame and attaches the
partial content, so consumers never hang on a half-open stream.

## Functions

### Consumer surface

| Function | Purpose |
|---|---|
| `router::chat` | Stream a turn into the caller's channel; returns the turn summary. |
| `router::complete` | Non-streaming convenience over the same pipeline; returns the final message. |
| `router::abort` | Cancel an in-flight turn by `request_id`. |
| `router::route` | Read-only routing preview: `{model, provider?}` → `{provider, candidates}`, same rules and error codes as `router::chat`. Pin the result as the explicit `provider` on the chat call when you need the provider before streaming. |
| `router::models::list` | List catalog models, filterable by `provider` / `capability` / `modality` (`chat` by default, `stt`, `tts`, `any`). |
| `router::models::get` | Fetch one model record (`null` when unknown). |
| `router::models::supports` | Check one capability flag for one model. |
| `router::embed` | Batch text embeddings through the first embed-capable provider in the registry (`{model?, provider?, input[]}` → `{provider, model, embeddings[][]}`); providers without an embed surface are skipped. |
| `router::transcribe` | Speech to text: `{model?, provider?, audio_base64, mime?, language?, prompt?}` → `{provider, model, text, segments[]?, language?, duration_secs?}`. The provider comes from the named `provider`, else from the catalog owner of the `stt` model, else the first provider that declared one. |
| `router::speak` | Text to speech: `{model?, provider?, text, voice?, format?, language?, speed?}` → `{provider, model, audio_base64, mime, voice?, duration_secs?}`. Same resolution over `tts` models. |
| `router::count_tokens` | Count prompt tokens: `{model, provider?, system_prompt?, tools?, messages}` → `{provider, model, tokens, estimator}`, resolved with the same routing rules as `router::chat` and forwarded to `provider::<id>::count_tokens`. Never runs the model and costs nothing; `estimator` is `provider` (metering API) or `tiktoken` (local tokenizer). A provider without the surface is a typed `router/no_token_counter` error, so callers can fall back to their own estimate. |
| `router::provider::list` | Registered providers with `configured` / `available` status, where each credential comes from (`credential_source`, `credential_ref`, `credential_error` — never a value), and each provider's starting point: `default_model` and `default_thinking_level` (see [Default model](#default-model)). |

Only the read surface is agent-callable (`router::models::list` / `get` /
`supports`, `router::provider::list`); everything else is denied to in-run
agents — see [Security model](#security-model).

### Error conventions

Errors use stable `router/<code>` identifiers. Streaming failures return an
`error` object with a user-facing `message`, the shared `kind`, `retryable`,
and an optional diagnostic `detail`. User interfaces should show `message`
and keep `code` / `detail` behind technical details; automation should branch
on `code` and `retryable`, never parse the prose.

The main recovery codes are `router/provider_unavailable`,
`router/capacity_exceeded`, `router/stream_idle_timeout`,
`router/stream_incomplete`, `router/provider_auth_expired`,
`router/provider_rate_limited`, `router/context_overflow`, and
`router/provider_rejected`.

### Provider protocol

Token-gated after the first declare: the response to `register` carries a
registration token, and every later protocol call must present it.

| Function | Purpose |
|---|---|
| `router::provider::register` | Self-declaration at attach time; idempotent re-declare with the token. |
| `router::provider::resolve` | Per-request credential + endpoint resolution (config > `secret://` / `env://` reference > env > none, see [Credentials](#credentials-and-secret-references)). |
| `router::provider::update_credential` | Persist a refreshed credential (OAuth write-back). |
| `router::models::reconcile` | Replace the provider's catalog slice in one write. |

The provider worker itself exposes `provider::<id>::stream` and, when
capable, `provider::<id>::refresh_models` (model discovery),
`provider::<id>::count_tokens` (prompt token counting),
`provider::<id>::embed` (embeddings), and the speech pair
`provider::<id>::transcribe` / `provider::<id>::speak`.

`router::chat` accepts an optional structured prompt next to the flat
`system_prompt`: `system_sections` (ordered `{text, cache_boundary}` blocks)
and `cache_intent` (`{surface_digest}`, the caller's `sha256:` identity for
the text before the boundary). A sections-only request is flattened into
`system_prompt` (sections joined with `\n\n`); when both are sent they must
agree byte for byte or the request is rejected as `invalid_request`. Both are
forwarded verbatim to `provider::<id>::stream` alongside the flat string, so a
provider that knows nothing about sections keeps reading `system_prompt`.
Cache-aware providers put their prefix marker on the boundary block
(Anthropic, Claude Code) or derive a shared `prompt_cache_key` from the digest
(OpenAI, Codex) so every session on the same frozen prefix lands on one cache
entry. The digest is a routing hint, never evidence of a hit — read
`usage.cache_read` for that.

## Configuration

All operator configuration lives in the engine's `llm-router` configuration
entry — no env vars, no config file. The entry schema is composed at runtime
from each registered provider's declaration. The router fetches the complete
entry at boot and keeps an in-memory snapshot synchronized by the
`configuration:updated` trigger; request handlers never poll
`configuration::get`.

```json
{
  "default_provider": "anthropic",
  "providers": {
    "anthropic": {
      "api_key": "secret://ANTHROPIC_API_KEY",
      "api_url": "https://api.anthropic.com/v1/messages",
      "max_tokens": 8192
    }
  },
  "routing_heuristics": [{ "pattern": "^gpt-", "provider": "openai" }],
  "settings": {
    "stream_timeout_ms": 600000,
    "idle_timeout_ms": 120000,
    "retry_max": 2,
    "output_token_max": 32000
  }
}
```

| Setting | Default | Meaning |
|---|---|---|
| `stream_timeout_ms` | `600000` | Hard budget for one streamed turn. |
| `idle_timeout_ms` | `120000` | Max silence between provider frames before the attempt is cut. |
| `retry_max` | `2` | Retries per turn for retryable failures before the first forwarded frame (`0`–`10`). |
| `output_token_max` | unset | Optional ceiling on `max_output_tokens`. Unset forwards each model's own output ceiling, so providers run without an artificial limit; set it only to cap spend. |

Pasting a key into a provider's slice is the whole onboarding flow: the
router diffs the changed slice, debounces ~2 s, and kicks that provider's
`provider::<id>::refresh_models` discovery; discovered models land in the
catalog via `router::models::reconcile` and show up in `router::models::list`
within seconds — no restart.

### Credentials and secret references

The configuration service's default filesystem adapter persists this entry at
`./config/<entry-id>.yaml` (`llm-router` unless `III_CONFIG_NAME` overrides it),
a folder meant to be committed, so keep keys out of it: keep the key in the
[`secrets`](https://github.com/iii-hq/workers/tree/main/secrets) worker, which `llm-router` depends on, and put a versionable
reference in the slice:

- `"api_key": "secret://ANTHROPIC_API_KEY"` — encrypted in the secrets
  worker's vault;
- `"api_key": "env://ANTHROPIC_API_KEY"` — the variable in the project's
  env file (`.env`, or the secrets worker's `env_file` for this namespace;
  else the secrets worker's environment), read by the secrets worker each
  time, so an edit to the file applies without restarting the router.

Provider keys reuse their env var name; `NAME` matches
`^[A-Za-z_][A-Za-z0-9_.-]{0,127}$`. Either way the secret's `consumers` must
list `llm-router`; the Console's key field does this for you.

A provider's credential is the first of:

1. the slice's `credential` object (OAuth write-back via `update_credential`);
2. a literal `api_key`;
3. an `api_key` reference (`secret://` or `env://`), resolved through
   `secrets::resolve`;
4. the provider's `credential_env_var`, read in the router's process;
5. none.

A reference is never forwarded to a provider as a key. When it does not
resolve, the provider is **not configured**, and the error says why: `secret
NAME not found in the secrets worker`, `environment variable NAME is not set
in the project's .env or the secrets worker's environment`, `llm-router is not
allowed to read secret NAME; add llm-router to the secret's consumers`, or
`secrets worker is not running`. The env var is deliberately **not** used as a fallback past an
explicit reference: you chose the stored secret, so a missing one is reported
rather than silently replaced by a different key. Providers built on the
shared scaffold skip their own env fallback in that case too, and
`router::chat` fails before dispatch with `router/not_configured` and the same
reason.

`router::provider::list` entries and the `router::provider::resolve` response
carry `credential_source` (`config` | `env` | `secret` | `none`),
`credential_ref` (the `secret://` or `env://` reference, when the slice uses
one; `credential_source` is `secret` for both, as the secrets worker resolved
it) and
`credential_error`. All three are optional on the wire. The older `source`
field keeps its `config` | `env` | `none` values: a resolved reference reports
`config` there.

Resolved values live only in the router's memory: never in the entry,
logs or traces, and a cached value is wiped when replaced. They are re-read:

- on `secrets::changed` (the router binds it at boot; without a secrets worker
  the engine holds the binding until the type registers). A rotated, deleted
  or revoked secret then nudges every provider whose slice references it
  through `provider::<id>::on_router_ready`, so the provider drops its cached
  resolve, re-declares and refreshes its models;
- when a slice that references it changes, before the debounced
  `refresh_models` fires, so a pasted reference is resolvable by the time
  discovery asks for it;
- when the secrets worker registers after the router, so a router that booted
  first recovers without a restart;
- on demand after 5 minutes (10 seconds for a failure), bounding staleness if
  an event is ever missed. While the secrets worker is unreachable, a value
  already resolved keeps being served: secrets cannot rotate or be revoked
  while it is down.

The reference itself is not secret: it is committed, listed and logged. Never
paste a key after `secret://` or `env://`.

### Operational notes

- **Secret references resolve as `llm-router`.** The secrets worker checks
  the router's engine-stamped worker identity against the secret's
  `consumers`, not the provider's.
- **Env-var credential fallback resolves in the router's process.** Without
  a reference, a provider's `credential_env_var` (e.g. `ANTHROPIC_API_KEY`)
  is read by the llm-router binary, not by the provider worker — launch the
  router with those variables set, or put a reference in the entry. A key
  present only in another worker's environment shows up as
  `configured: false`. `env://NAME` is different: the secrets worker reads it
  from the project's `.env`, so the router needs no `env_file` for it.
- **Registration-token recovery.** Re-registering a provider id without its
  original token is rejected (anti-takeover). If a provider durably lost its
  token, delete the router's registry state (state scope `llm-router`,
  key `registry`) and restart the affected providers to re-bind; pasted
  credentials in the configuration entry are unaffected.

## Security model

The agent-callable surface is governed by the engine-wide permission policy
(repo-root [`iii-permissions.yaml`](https://github.com/iii-hq/workers/blob/main/iii-permissions.yaml)), not a per-worker
file. In-run agents may only **read** the catalog and provider list:

- **Allowed:** `router::models::list`, `router::models::get`,
  `router::models::supports`, `router::provider::list`.
- **Denied to agents:** the chat/spend surface (`router::chat`,
  `router::complete`, `router::abort`, `router::route`), the whole
  provider/credential protocol (`router::provider::resolve` / `register` /
  `update_credential`, `router::models::reconcile`), and direct `provider::*`
  calls.

Worker-to-worker calls bypass the agent gate, so the harness, context-manager,
and provider workers reach the full surface — only in-run agents are
restricted. Provider credentials live in the configuration entry, or behind a
`secret://` or `env://` reference in the `secrets` worker, and are never readable back
through any allowed function: `router::provider::list` reports where a
credential comes from, never its value.

## Events

The router registers three custom trigger types and fans out to every bound
handler. Bind with the standard two-step pattern; the handler receives the
payload verbatim (no envelope).

| Trigger type | Fires when | Payload |
|---|---|---|
| `router::models::changed` | a provider reconciles its catalog slice | `{ "provider": "<id>", "count": <n> }` |
| `router::provider::changed` | the registry changes (declare / availability flip) | `{ "provider": "<id>", "op": "register" \| "available" \| "unavailable" }` |
| `router::ready` | the router finishes booting; providers re-declare on it | `{}` |

The router itself binds the engine's `configuration` trigger (its entry) and,
when the optional `secrets` worker is installed, `secrets::changed`
(`router::on_secret_changed`).

```ts
iii.registerFunction('my-worker::on-models-changed', async (payload) => {
  console.log('catalog changed:', payload); // { provider, count }
  return {};
});

iii.registerTrigger({
  type: 'router::models::changed',
  function_id: 'my-worker::on-models-changed',
  config: {},
});
```

## Writing a provider worker

A provider worker must:

1. Register `provider::<id>::stream` honouring the channel-writer contract:
   forward upstream output as `AssistantMessageEvent` frames into the
   `writer_ref` it receives, ending with one terminal frame.
2. Declare itself at startup via `router::provider::register` — retrying with
   backoff until acknowledged (covers provider-before-router boot order) —
   and re-declare on the `router::ready` event after a router restart. The
   declaration may carry `icon_svg`: one self-contained, monochrome `<svg>`
   document (square `viewBox`, no fixed size, at most 32 KiB). The router
   echoes it in `router::provider::list` and the console paints it as a
   `currentColor` mask beside the provider's models; a missing or malformed
   mark falls back to the provider's initial.
   The declaration may also carry `default_models`, the provider's
   recommended starting models in preference order (the current mid-range
   model first, then its predecessors), and `default_thinking_level`, the
   level to pair with it; see [Default model](#default-model).
3. Resolve credentials per request via `router::provider::resolve`; never
   read keys directly. A response with `credential_source: "secret"` and no
   credential is an unresolvable `secret://` or `env://` reference: report its
   `credential_error`, do not substitute a key of your own.
4. Treat closure of its stream channel as cancellation: abort the upstream
   request and stop writing frames.
5. Map upstream failures to the shared `ErrorKind` taxonomy on its `error`
   frames. Transport retries (429 / 5xx / connect) are the router's job, not
   the provider's.

A provider may also register `provider::<id>::count_tokens` (`{model,
system_prompt?, tools?, messages}` → `{model, tokens, estimator}`) to serve
`router::count_tokens`: an exact count from a provider metering API
(`estimator: "provider"`) or a local tokenizer estimate (`estimator:
"tiktoken"`). Providers without it simply make `router::count_tokens` return
a typed `router/no_token_counter` error for that provider.

### Default model

`router::provider::list` reports one `default_model` per provider so that a
caller with no model choice of its own (a fresh `harness::send` naming only
a provider, a console opening its first chat) starts on something sensible.
It is resolved at read time against the provider's current catalog slice:

1. the model variant first. A variant is the first all-letter id segment
   after the family (`terra` in `gpt-6.1-terra`, `sonnet` in
   `claude-sonnet-5-5`), ranked in the order `default_models` first names
   it. Within a ranked variant the newest version in the slice wins,
   listed or not, so a newer Luna never beats a Terra when Terra is listed
   first. A model with a ranked variant counts even outside the first
   preference's family, as long as it has the same `vendor/` prefix, so
   Copilot ranks GPT and Claude models in one list;
2. ids with no ranked variant come last: the first id in the declared
   `default_models` the slice holds, else the slice model sharing the
   longest id prefix with the first preference, as long as they share the
   family (the id up to its first `-`: `claude`, `gpt`, `codex/gpt`). On a
   tie the highest version wins (`gpt-4.1` over `gpt-4o`), then the plain
   id over a dated snapshot or a `-mini` variant;
3. otherwise absent, and the caller keeps its previous behaviour.

A provider that declares no `default_models` (local model servers, speech
providers) never reports one. `default_thinking_level` is copied from the
declaration; absent means the caller should omit the level and let the
provider apply its own default.

A declaration may also carry `context_overflow_hint`: one or two plain
sentences a console shows under a context-overflow failure on one of the
provider's models, for causes compacting cannot fix (a self-hosted server
started with a small window, say), optionally with a link to the provider's
docs. `router::provider::list` reports it verbatim; absent means the console
shows only its generic steps.

A provider that works without a key (a local server started without one)
declares `credential_optional: true`. The router then reports it
`configured` when no credential resolves; a key that is set is still
resolved and sent. Absent or `false` means a key is required.

### Thinking levels

`router::chat` takes `thinking_level`: `off`, `minimal`, `low`, `medium`,
`high` or `xhigh`. Each provider maps a level onto its own knob and warns
when a model cannot honour it. `off` asks for no reasoning at all and is
honoured only where the model's descriptor says `supports_thinking_off:
true` (a native switch exists: OpenAI `reasoning_effort: none` on the
models that take it, DeepSeek and GLM `thinking.type: disabled`, Anthropic
Sonnet 5.5 `between_tools`). `false` means the model always reasons or the
provider has no switch; absent means the provider does not know. Consoles
offer, hide or disable the Off choice accordingly. A provider that cannot
switch reasoning off sends its lowest effort instead and reports a warning.

### Speech providers

A speech provider is the same kind of worker with no chat stream: it
declares its models with a `speech` block and serves one or both speech
surfaces. `router::models::list` hides speech models from chat pickers
unless asked (`modality: "stt" | "tts" | "any"`), and `router::chat`
refuses them with a message that names the right door.

```json
{ "id": "scribe-v1", "provider": "elevenlabs", "context_window": 0, "max_output_tokens": 0,
  "speech": { "modality": "stt", "languages": ["en", "hi"], "streaming": true } }
```

| Function | Request | Response |
|---|---|---|
| `provider::<id>::transcribe` | `{model?, audio_base64, mime, language?, prompt?}` | `{model, text, segments[{text, start_secs?, end_secs?}]?, language?, duration_secs?}` |
| `provider::<id>::speak` | `{model?, text, voice?, format, language?, speed?}` | `{model, audio_base64, mime, voice?, duration_secs?}` |

`model` arrives as the id the caller named, or `null` for the provider's
default. `format` is what the caller asked for (`mp3`, `wav`, `pcm16`,
`opus`); answer with the container you produced and say so in `mime`.
Credentials resolve through `router::provider::resolve` exactly as for
chat providers.

The first real provider implementing this protocol is
[`provider-anthropic/`](https://github.com/iii-hq/workers/tree/main/provider-anthropic) — useful as a reference
implementation alongside the scripted provider in the integration tests.
[`provider-openai/`](https://github.com/iii-hq/workers/tree/main/provider-openai) follows the same structure for the
OpenAI Chat Completions API (native structured output, reasoning_effort).
[`provider-opencode-go/`](https://github.com/iii-hq/workers/tree/main/provider-opencode-go) follows the same structure for the
OpenCode Go Chat Completions API (curated model metadata, reasoning_effort).

## Local development & testing

```bash
cargo test                       # unit suite, no engine needed
cargo test --test integration    # engine-backed suite; self-skips without an engine
```

The integration suite spawns a throwaway engine per test when `iii` is on
`PATH` (or `III_ENGINE_BIN` points at a binary) and covers the chat relay,
cancellation, abort, restart recovery, registration token gating, paste-a-key
discovery (literal keys, `secret://` and `env://` references, against a fake
`secrets` worker), and event delivery end to end.

To run the worker locally against an engine:

```bash
cargo run -- --url ws://127.0.0.1:49134
```

`--url` defaults to `ws://127.0.0.1:49134` and honours the `III_WS_URL`
environment variable when the flag is not set. `--config` is accepted per
the standard worker CLI but ignored with a warning — operator config lives
in the engine's `llm-router` configuration entry (see Configuration above).