# judge-semif

[SemIf](https://github.com/TheoLeeCJ/SemIf) provider for the [`judge`](../judge/)
hub, running **inside the worker**: no API key, no Python, no llama-server.
SemIf reads runtime-defined decisions from a frozen open LLM: one forward pass,
a softmax over the next-token logits of the option letters `A`–`P`, no decoding.
Its published Qwen3.5-4B baseline agrees with Jev's own references on 0.845 of
TypeSafe's public subset (Jev 0.883). This worker runs the same prompt through
llama.cpp (`crates/llama-native`, the engine every local judge provider
shares) on the CPU, on Metal (macOS) or on Vulkan (Linux x86_64: AMD, NVIDIA
and Intel GPUs), picked automatically at start.

## Hardware selection

- **macOS**: Metal is linked in; Apple Silicon GPUs run the model.
- **Linux x86_64**: llama.cpp's backends are modules loaded at start from the
  binary's directory (`GGML_BACKEND_DL`). The Vulkan module loads wherever a
  Vulkan loader and driver exist (`libvulkan.so.1`); without them it is
  skipped and the CPU runs the model, so the same package works on GPU
  desktops, servers and containers. The CPU module is picked for the host
  (AVX2, AVX-512, AMX variants). The package ships `libllama`, `libggml`,
  `libggml-base`, the CPU variants and the Vulkan module (≈71 MB) beside the
  binary, which finds them through its `$ORIGIN` runpath.
- **Linux aarch64**: CPU, statically linked.

`gpu_layers: 0` keeps the model on the CPU even when a GPU is present. The
chosen device is logged at start as `selected inference device`.

## Install

```bash
iii trigger compose::add worker=judge-semif
```

The functions register at start; the model loads on demand. While SemIf is
the judge hub's **default provider** (`provider: semif` under **Settings →
Workers → judge**) it loads at once and stays loaded. Otherwise the first call
that names `"provider": "semif"` (or comes from a session that picked it)
loads it, and it is released (VRAM included) after 10 minutes without calls.
A call that cannot wait for the load answers `deadline` while the load goes on
for the next one. The first load downloads the pinned GGUF
(`qwen3.5-4b`: `bartowski/Qwen_Qwen3.5-4B-GGUF` Q4_K_M, 3.0 GB) into the
hf-hub cache (`$HF_HOME`, default `~/.cache/huggingface`). To keep it loaded
while another provider is the default, turn on **Keep every local provider
loaded** (`preload_all`) in the judge settings. A hub build that does not
expose `judge::configuration-id` keeps the model loaded from the start.
Air-gapped installs set `III_SEMIF_GGUF` to a local GGUF file.

Make it the hub's default, then call the hub:

```bash
iii trigger judge::evaluate --timeout-ms 65000 --json '{
  "timeout_ms": 60000, "provider": "semif",
  "evaluations": [{
    "id": "ticket-4411",
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel."},
    "questions": {
      "department": {"type": "choice", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
      "urgency": {"type": "score", "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}
    }
  }]
}'
```

## How questions map to SemIf

Each question becomes one SemIf decision: `{"evidence": state, "criterion":
instructions, "options": [...]}` in SemIf's frozen system prompt and Qwen chat
template (thinking disabled), tokenized by the GGUF vocabulary. Token ids match
SemIf's reference tokenizer (`tests/prompt.rs`), and on the CPU the option
probabilities match SemIf's own llama.cpp backend to four decimals.

| Contract | SemIf options | Answer |
|---|---|---|
| `noul` | A = `criteria.true` (or "Yes, the statement holds."), B = `criteria.false` | `noul` = P(A) |
| `choice` | one per criterion, `key: description` | argmax, probabilities, confidence |
| `score` | `level i: <level>` | Σ i·p_i, probabilities, confidence, legend |

`confidence` is TypeSafe's (`judge_contract::confidence`, shared by the local
providers): `(n·p_max − 1) / (n − 1)` for a choice, 1 − the expected distance
from the likeliest level over the levels' mean distance from the middle for a
score. SemIf's probabilities are conditional on the offered options and
uncalibrated. At most
16 options per question. A prompt longer than `context_tokens` answers
`payload_too_large`: SemIf never truncates evidence.

Every question of one evaluation shares its state, and SemIf puts the evidence
first. The worker prefills the prompts' longest common token run once,
snapshots it, restores it into up to `parallel_questions` sequences and decodes
their suffixes together in one batch (SemIf's parallel suffixes). The snapshot
stays in the model's device memory (VRAM on a GPU; about 270 MB for a
6.7k-token prefix, held until the next evaluation replaces it). The prefix is
found on tokens, not re-tokenized text: a state ending in `{}` merges with the
following `}` two tokens back, which SemIf's own `_state_prefix` rejects.
`usage.input_tokens` counts the decoded tokens (the shared prefix once).

## Speed (Qwen3.5-4B Q4_K_M, i9-14900K, RX 6900 XT)

Per request, 16 questions about one state; `parallel_questions` 8 unless noted:

| | incident (541-token state) | directory block (4.6k-token state, 64 functions) |
|---|---|---|
| CPU, 8 threads | 3.8 s (6.9 s with parallel 1) | 6.6 s |
| Vulkan | 0.79 s (1.18 s with parallel 1) | 3.4 s |

Before the token-level prefix, the directory block decoded every question in
full: 75k tokens and 35.8 s on Vulkan, against 6k tokens and 3.4 s now.
Parallel suffixes pay off on short states; long states are dominated by the
one-time prefill. On the same card llama.cpp's ROCm backend measured within
10% of Vulkan either way, so the worker ships Vulkan alone. A single short
decision takes 67–140 ms on the GPU and 1.1–1.4 s on the CPU.

These figures predate llama.cpp b11379 (`crates/llama-native`). Against the
previous build, b11379 takes 0.85 of the time on the CPU (a 1.6k-token request
of 9 questions and a 7.6k-token one of 10, 9 runs each) and the same time on
Vulkan (within 2%).

GPU, reuse and batching change probabilities by up to 0.02 against fresh CPU
scoring, as SemIf documents for its own fast paths.

## Configuration

**Settings → Workers → judge-semif**:

| Field | Default | Applied |
|---|---|---|
| `model` | `qwen3.5-4b` | next start |
| `threads` | min(8, logical cores) | next start |
| `gpu_layers` | all layers when a GPU backend and device exist | next start (`0` = CPU) |
| `context_tokens` | 16384 | next start |
| `parallel_questions` | 8 | next start (`1` = one question per decode) |
| `max_request_bytes` | 8388608 | new calls |
| `max_timeout_ms` | 300000 | new calls |

## Building

llama.cpp b11379 is compiled from source through `crates/llama-native`: its
`build.rs` downloads GitHub's source archive (checked against a pinned
sha256), builds it with cmake and compiles a small C shim against it, with no
bindgen. It needs `curl`, `tar`, `patch`, `cmake` and a C++17 compiler; an
offline build points `III_LLAMA_CPP_TARBALL` at a copy of the archive, checked
the same way. Linux x86_64 builds also need the Vulkan loader headers, the
SPIR-V headers and `glslc` (Ubuntu: `libvulkan-dev spirv-headers glslc`) to
compile the Vulkan module (only the module links `libvulkan`, the binary does
not). The build copies the modules and libraries beside the binary, so
`target/release` has the published layout; the release catalog ships them as
the artifact's `companions`. Windows is not published yet.

For the full API, read the hub's [reference](../judge/reference.md).