# judge-decider

[decider](https://github.com/Mapika/decider) provider for the
[`judge`](../judge/) hub, running **inside the worker**: no API key, no Python,
no llama-server. decider-4b v2 is a Qwen3.5-4B-Base fine-tune trained for
exactly this readout (JevBench v1.4.2 rank 1, 64.1): one forward pass per
decision, a softmax over the next-token logits of the option labels at the
model's fitted temperature, no decoding. This worker reads it in decider's own
prompt layout through llama.cpp (`crates/llama-native`, the engine every local
judge provider shares) on the CPU, on Metal (macOS) or on Vulkan (Linux
x86_64: AMD, NVIDIA and Intel GPUs), picked automatically at start.

On an RX 6900 XT (Vulkan), against the other judge providers on the same
machine:

| provider | labelled tickets: department / urgency / churn / refund | per ticket | `directory::search_functions` (22 capabilities): hit / top-1 / precision |
|---|---|---|---|
| `decider` (decider-4b-v2) | 1.00 / 0.79 / 0.96 / 0.96 | 0.30 s | 22 / 22 / 0.94 |
| `semif` (Qwen3.5-4B) | 0.88 / 0.62 / 1.00 / 1.00 | 0.29 s | 21 / 18 / 0.81 |
| `typesafe` (hosted Jev) | 0.96 / 0.75 / 0.96 / 1.00 | 0.25 s | 22 / 22 / 0.99 |

## Hardware selection

- **macOS**: Metal is linked in; Apple Silicon GPUs run the model.
- **Linux x86_64**: llama.cpp's backends are modules loaded at start from the
  binary's directory (`GGML_BACKEND_DL`). The Vulkan module loads wherever a
  Vulkan loader and driver exist (`libvulkan.so.1`); without them it is
  skipped and the CPU runs the model, so the same package works on GPU
  desktops, servers and containers. The CPU module is picked for the host
  (AVX2, AVX-512, AMX variants). The package ships `libllama`, `libggml`,
  `libggml-base`, the CPU variants and the Vulkan module (≈71 MB) beside the
  binary, which finds them through its `$ORIGIN` runpath.
- **Linux aarch64**: CPU, statically linked.

`gpu_layers: 0` keeps the model on the CPU even when a GPU is present. The
chosen device is logged at start as `selected inference device`.

## Install

```bash
iii trigger compose::add worker=judge-decider
```

The functions register at start; the model loads on demand. While decider is
the judge hub's **default provider** (`provider: decider` under **Settings →
Workers → judge**) it loads at once and stays loaded. Otherwise the first call
that names `"provider": "decider"` (or comes from a session that picked it)
loads it, and it is released (VRAM included, about 6 GB) after 10 minutes
without calls. A call that cannot wait for the load answers `deadline` while
the load goes on for the next one. The first load downloads the pinned GGUF
(`mindchain/decider-4b-v2-GGUF` Q4_K_M, 2.7 GB, quantized by llama.cpp from
`Mapika/decider-4b` at tag `v2`) into the hf-hub cache (`$HF_HOME`, default
`~/.cache/huggingface`). To keep it loaded while another provider is the
default, turn on **Keep every local provider loaded** (`preload_all`) in the
judge settings. A hub build that does not expose `judge::configuration-id`
keeps the model loaded from the start. Air-gapped installs set
`III_DECIDER_GGUF` to a local GGUF file.

The repository's `main` is v2.1, which trades some of v2's accuracy on hard
decisions for sampled play and has no published GGUF; v2 is the JevBench
rank-1 revision.

Make it the hub's default, then call the hub:

```bash
iii trigger judge::evaluate --timeout-ms 65000 --json '{
  "timeout_ms": 60000, "provider": "decider",
  "evaluations": [{
    "id": "ticket-4411",
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel."},
    "questions": {
      "department": {"type": "choice", "instructions": "Which team should handle it?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
      "urgency": {"type": "score", "instructions": "How urgent is it?", "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}
    }
  }]
}'
```

## How questions map to decider

The worker builds decider's plain state-first prompt as decider-ai 1.5.0's
`/v1/systemone` does (`decider/systemone.py`, `decider/prompt_fast.py`), one
independent prompt per decision:

```
Context:
<state>

Question: <instructions>
Options:
(A) <option>
(B) <option>
Answer: (
```

The state is a string as-is, or Python JSON with `_index` written into every
element of an 8+ element array. `Context:` and each question block are
tokenized separately, as decider does; up to ten options render as one
string, more use one label token each (`A`–`Z`, then the two-letter labels
that are single tokens), up to 255 options. The label logits go through a
softmax at v2's temperature 1.935.

| Contract | decider options | Answer |
|---|---|---|
| `noul` | `no` then `yes`, each followed by its `criteria` description after `: ` | `noul` = P(yes) |
| `choice` | one per criterion, `key: description` | argmax, probabilities, confidence |
| `score` | one yes/no prompt per level: `<instructions>\nProposed answer: <level>\nDoes the proposed answer fit?` | the levels' P(yes) normalized; Σ i·p_i |

Score levels are judged in isolation (`isolated_levels`), without their number
or their neighbours, as decider-4b v2 was trained. `confidence` is decider's
(TypeSafe's definitions): for a choice `(n·p_max − 1) / (n − 1)`, 0 for a
uniform distribution and 1 for all mass on one option; for a score
1 − the expected distance from the likeliest level over the mean distance of
the levels from the middle of the scale. A question without instructions asks
"Which answer fits the context?"; a noul needs instructions or a `true` /
`false` description. A prompt longer than `context_tokens` answers
`payload_too_large`: the state is never truncated.

Every prompt of one evaluation starts with the same `Context:` tokens. The
worker prefills them once, snapshots the sequence, restores it into up to
`parallel_questions` sequences and decodes the question blocks together in
one batch (`iii_llama_runtime::scorer`, shared with judge-semif). The
snapshot stays in the model's device memory (VRAM on a GPU), so restoring it
never goes through the host; the latest one is held until the next evaluation
replaces it, about 300 MB for a 7.6k-token context.
`usage.input_tokens` counts the decoded tokens (the shared prefix once).

`tests/decider.rs` checks the prompts and token ids against decider's own
rows and the answers (probabilities and confidence) against the bf16 weights
computed in float32 (`tests/fixtures/make_decider_prompts.py`): an f16
conversion agrees to 0.0025, Q8_0 to 0.0074, and the shipped Q4_K_M to 0.12
(0.14 on the CPU; 0.03 elsewhere) on a near-flat 12-option question. There
llama.cpp b11379 puts `area0` (0.17) just above the reference's answer `auth`
(0.16 against 0.28), so that question's choice check fails. For the closer Q8_0
(4.5 GB), convert `Mapika/decider-4b` at `v2` with llama.cpp's
`convert_hf_to_gguf.py --outtype q8_0 --no-mtp` and point `III_DECIDER_GGUF`
at it.

## Configuration

**Settings → Workers → judge-decider**:

| Field | Default | Applied |
|---|---|---|
| `model` | `decider-4b-v2` | next start |
| `threads` | min(8, logical cores) | next start |
| `gpu_layers` | all layers when a GPU backend and device exist | next start (`0` = CPU) |
| `context_tokens` | 16384 | next start |
| `parallel_questions` | 8 | next start (`1` = one question per decode) |
| `max_request_bytes` | 8388608 | new calls |
| `max_timeout_ms` | 300000 | new calls |

## Building

llama.cpp b11379 is compiled from source through `crates/llama-native`: its
`build.rs` downloads GitHub's source archive (checked against a pinned
sha256), builds it with cmake and compiles a small C shim against it, with no
bindgen. It needs `curl`, `tar`, `patch`, `cmake` and a C++17 compiler; an
offline build points `III_LLAMA_CPP_TARBALL` at a copy of the archive, checked
the same way. Linux x86_64 builds also need the Vulkan loader headers, the
SPIR-V headers and `glslc` (Ubuntu: `libvulkan-dev spirv-headers glslc`) to
compile the Vulkan module (only the module links `libvulkan`, the binary does
not). The build copies the modules and libraries beside the binary, so
`target/release` has the published layout; the release catalog ships them as
the artifact's `companions`. Windows is not published yet.

For the full API, read the hub's [reference](../judge/reference.md).