# memory

Durable cross-session agent memory. Named **banks** hold two kinds of content: **rules** (markdown documents injected whole into the system prompt) and **memories** (auto-extracted records recalled on demand). Everything is a plain file you can open, a function you can call, and an event you can watch: memory that acts visibly, not magically.

## Install

```
iii trigger compose::add worker=memory
```

The default bank `main` materializes on first use. No configuration required; without `llm-router` extraction degrades to explicit `memory::save` calls, and without the harness the worker still serves its full RPC surface.

## Quickstart

Save something, then see exactly what a turn on that topic would be given:

```bash
iii trigger memory::save text="User publishes blog posts on Tuesday mornings" pinned=true tags=blog
iii trigger memory::recall query="when do I publish articles"
```

Give the bank a standing instruction — injected into the system prompt of every session using it:

```bash
iii trigger memory::rule::set name=style content="# Style\nFormal register. No em-dashes."
```

With the harness running there is nothing else to wire: chat normally and memory captures in the background. Say "never start replies with the word Certainly" in a conversation and it lands in the bank's auto-managed `learned` rule; state something durable and it appears as a memory after the turn. If memory seems absent, `iii trigger memory::doctor` runs a real save→recall roundtrip and names what is degraded.

## Configuration

All fields hot-reload through the `configuration` worker and are editable in the worker's explicit form in the Console's global Settings modal; `data_dir` reopens the store on the fly. Fields: `data_dir`, `default_bank`, `inject_rules`, `inject_memories`, `recall_limit`, `recall_budget_tokens`, `extraction_enabled`, `extraction_model`, `extraction_window`, `extraction_timeout_ms`, `max_memories_per_turn`, `rule_learning_enabled`, `max_rule_chars`, `decay_half_life_days` (default 60; installs that predate MOT-5385 keep their stored 30 — set 60 in Settings to adopt the tuned recall curve), `embeddings_enabled`, `embedding_model`.

## Why this shape

- **Files are the source of truth.** `rules/*.md` and `memories.jsonl` under the data dir. Edit them in any editor; `memory::reload` (or a restart) picks the edits up. The search index is a RAM-only cache rebuilt from the files at boot, so store and index can never diverge across restarts.
- **Crash-safe by construction.** Every mutation appends one fsynced JSONL line before touching RAM. There is no shutdown flush to get wrong. An unwritable data dir is boot-fatal: this worker never silently runs in RAM.
- **Supersede, never delete.** Updates append revisions; deletes append tombstones; trashed banks move to `.trash/`. Any state is recoverable.
- **Pinning.** A pinned memory ranks higher in recall and is untouchable by every automatic path.
- **Tags.** Topical labels WITHIN a bank ("iii", "billing") for filtering — organization, not ranking. Set them on save/update, extraction suggests them automatically, and `memory::list`/`memory::recall` take a `tag` filter.
- **One LLM call per turn, zero at query time.** Extraction runs in the background after a turn completes (ADD-only, content-fingerprinted so redelivery reinforces instead of duplicating). Recall is BM25 over the query's content words (English stopwords dropped) + entity match + corroboration + recency, plus a semantic signal when `router::embed` is available: sub-millisecond at this scale.
- **Rules learn from corrections.** Extraction classifies standing instructions (style directives, workflow corrections) separately from memories and appends them to the bank's auto-managed `learned` rule — correcting the agent in chat reaches every later turn (new sessions in the system prompt, running ones as an appended rules update). Hand-authored rules are never touched; dedup by content fingerprint; `rule_learning_enabled` turns it off.
- **Secrets are redacted, deterministically.** Known key and token shapes (provider API keys, bearer tokens, JWTs, private key blocks, credentialed URLs, `password=`-style assignments) become `[REDACTED:<label>]` before anything is stored — memory text, entities, tags, and rules, on every write path — and before the transcript excerpt reaches the extraction model. The `item-changed` event names what was redacted (`reason`), never the value. A denylist of shapes: an unknown format passes, so the extraction prompt still forbids secrets.
- **Time-aware.** Extraction reads each message with its own date and writes dated events ("tomorrow", "next Friday") as absolute dates; every injected memory carries its last-updated date, and the block names the day it was recalled.
- **Honest health.** `memory::doctor` runs a real save→recall→trash roundtrip and reports sibling reachability. `memory::recall` names the retrieval mode it ran. Degradation is explicit, never silent.

## How it hooks into the harness

| Seam | What happens |
|---|---|
| `harness::hook::pre-generate` (fail-open, priority 100) | Injects the session bank's rules into the system prompt once per session (sent verbatim after that; a later rule change arrives as one appended update message, so the prefix stays append-only) and, on each turn's first step, up to `recall_limit` recalled memories as one appended message. Whether an update is due is read from the window the step sends (its newest `<memory update="rules">` message, else the frozen section), so a lost or compacted-away update is sent again and a delivered one never is |
| `harness::turn-completed` | Spawns one background `router::complete` extraction pass over the last `extraction_window` user/assistant messages |

Bank selection order: turn metadata `memory_bank` → session metadata `memory_bank` (`session::set-meta`) → configured `default_bank`. A session-lookup failure injects nothing new (the section the session already has stays) rather than falling back across banks. An unknown bank gets the ambient header while extraction is on (extraction creates the bank after a turn) and nothing while it is off.

Switching a running session's bank does not rewrite what it already saw: the new bank's section arrives as an appended update that overrides the frozen one, while the earlier bank's section and recalled memories stay visible in the transcript. Start a new session when the two contexts must not mix. Turning `inject_rules` and `inject_memories` both off works the same way: a running session keeps its section and is told once that no memory section applies.

All seam bindings are one-shot at startup and rely on the engine's recoverable
triggers (iii #1962, engine ≥ 0.21.8): bound before the owning sibling (harness,
session-manager, queue) is up, they park as pending intents and activate when
the trigger type registers. On older engines the binds are silently dropped.

## Functions

| Function | Purpose |
|---|---|
| `memory::bank::create / list / delete` | Banks as first-class named scopes ("blog" vs "coding" vs "personal"); delete moves to `.trash/` |
| `memory::save` | Explicit save ("remember this"): fingerprinted, reinforces on repeat |
| `memory::get / list / update / delete / pin` | Memory CRUD; delete tombstones; update bumps a revision in the log |
| `memory::recall` | The exact scorer the hook uses: preview what a turn would be given; `tag` narrows to one label; `mode: lexical \| semantic` returns one raw arm (plain BM25 over the content words, or cosine) for callers that fuse arms themselves |
| `memory::tags` | Distinct tags across a bank's live memories with counts (the filter source) |
| `memory::preview` | Full injection dry-run for a hypothetical chat message: rules section (budgets applied), memories post ambient floor and token budget, appended message verbatim — the hook's exact code path |
| `memory::rule::list / set` | The always-injected markdown rules; empty content removes |
| `memory::supersede` | Retire one memory in favor of another (tombstone + pointer) — the consolidation seam; `reinforce` makes it a merge (a non-pinned survivor absorbs the retired record's observations), `reason` rides the events |
| `memory::doctor` | End-to-end self-test (roundtrip + sibling reachability) |
| `memory::reload` | Reload every bank from disk after hand-editing files |

## Custom trigger types

Bind live views here instead of polling; both are filterable by `bank` in the binding config, delivery is fire-and-forget, at-least-once, unordered.

- `memory::item-changed` — a memory was created / updated / superseded / deleted. Payload: `{event_type, bank, memory, reason?}` (`reason` on consolidation changes).
- `memory::bank-changed` — a bank was created / trashed, or its rules changed. Payload: `{event_type, bank}`.

## Storage layout

Compose stores this tree under `III_COMPOSE_DIR`. A standalone worker uses its
process current directory for the same relative default.

```text
<III_COMPOSE_DIR>/data/memory/
  main/
    bank.yaml          # description
    rules/style.md     # always-injected markdown
    memories.jsonl     # append-only memory log (full records, last-wins by id)
    vectors.jsonl      # derived embedding sidecar (rebuildable)
  .trash/              # trashed banks, timestamped
```

## Recall quality eval

`tests/recall_quality.rs` scores the production `Bank::recall` over a synthetic golden bank (`tests/recall_quality/golden.json`: one persona, 76 memories including 4 paraphrase duplicates, 62 English/Portuguese queries across lexical, paraphrase, identity, temporal-update, dated-event and durable-old categories). It runs in three modes:

- `lexical`;
- `hybrid`, with cached `text-embedding-3-small` vectors;
- `hybrid-consolidated`, which first replays the semantic consolidation tier's recorded decisions (`tests/recall_quality/consolidation.json`, from memory-consolidate's judge eval) and scores expectations carried to the survivors.

It reports hit@1/3, MRR@10, nDCG@10 and recall@10 per category and language. It also counts **traps**, where a superseded memory outranks the current one, and **duplicate pollution**, a query whose top 10 holds both members of a `dup_of` pair. The committed snapshot `tests/golden/recall_quality.json` makes it a PR gate. The metric conventions follow Hindsight's retrieval eval.

```bash
cargo test --test recall_quality -- --nocapture                # report
UPDATE_GOLDENS=1 cargo test --test recall_quality              # re-baseline after an intended change
III_NAMESPACE=<ns> cargo test --test recall_quality -- --ignored embed_golden_set   # re-embed after editing golden.json
```

## Permissions

Suggested defaults ship in [`iii-permissions.yaml`](iii-permissions.yaml): agents get `memory::recall`, `memory::save`, `memory::get`, `memory::list`, `memory::update`, `memory::delete`, `memory::pin`; the hook/pipeline internals are denied, and bank/rule writes plus `memory::reload` are human-owned surfaces (console, REST, CLI) by default.

## The memory family

Like `llm-router` and its provider workers, memory is a family of narrow siblings rather than one grown monolith — this worker owns the store, per-turn injection, capture, and recall, and stays deliberately small:

| Worker | Role | Status |
|---|---|---|
| `memory` (this) | Banks, rules, memories: storage, harness injection, background capture, hybrid recall, live trigger types | shipped |
| [`memory-consolidate`](https://github.com/iii-hq/workers/tree/main/memory-consolidate) | Background hygiene on a schedule: deterministic dedup of near-duplicate memories, catch-up-on-boot scheduling. Supersede-only through `memory::supersede`, never touches pinned records; every change lands as a `memory::item-changed` event | shipped |

The split is the point: consolidation can be installed, stopped, or removed without touching stored memory, and this worker never grows a scheduler.

## Boundaries

Context-manager compresses one history for one turn; this worker is the durable cross-session sibling its spec reserves (see [tech-specs/2026-06-agentic/memory.md](https://github.com/iii-hq/workers/blob/main/tech-specs/2026-06-agentic/memory.md)). Recall fuses BM25 with a semantic signal when `router::embed` has an embed-capable provider; `memory::recall.retrieval` names the mode either way. Consolidation belongs to the `memory-consolidate` sibling, not here.