# fp

Lodash-style value transforms and `fp::pipe` — worker-side pipelines that
move big values function→function over the iii bus without routing them
through the model.

## Install

    iii call compose::add worker=fp

The worker is a `deploy: binary` Rust worker with no configuration.

## Why

An agent that fetches a document and re-types it into the next call's
arguments burns its context window and stalls the provider stream mid-call.
`fp::pipe` runs the whole move in one call — each step calls a function
and its result lands in the next step's payload — so the value flows
worker→worker and the chat only ever sees per-step sizes and a preview.

While connected (and unless turned off) the worker also injects a usage
section into the agent system prompt: `fp::inject-guidance` observes the
harness's `harness::hook::on-pre-generate` with args
`{ on_error: "fail_open", inject_prompt: <the guidance> }`. The knob lives in the builtin `configuration`
worker under the `fp` entry: `inject_guidance` is ON by default; turn it
off in global Settings (or via `configuration::set`) to save
the ~750 tokens per generation (the harness's `# Granted functions` catalog
still advertises the `fp::*` ids) — it hot-applies, the worker observes the
hook or disposes the observer on the spot, no restart. The Console renders the
entry with its explicit worker settings form. The guidance stays
presence-gated either way (no fp worker, no prompt text). When enabled before
the harness is up, the observer stays `pending` and becomes active when the
harness registers the hook function. The worker also observes
`configuration::on-updated` for its own entry (`fp::on-config-change`).

## Functions

| function | lodash | request |
|---|---|---|
| `fp::pipe` | — | `{ through: [{function, payload?, into?}], preview_chars? }` |
| `fp::batch` | — | `{ calls: [{id?, function, payload?, namespace?, timeout_ms?}], concurrency?, stop_on_error?, timeout_ms?, max_entry_bytes?, max_calls? }` — independent calls, every answer in place (experimental) |
| `fp::get` | `_.get` | `{ value, path }` — JSON pointer; a miss names the available keys |
| `fp::pick` | `_.pick` | `{ value, paths }` — subset an object by top-level keys |
| `fp::omit` | `_.omit` | `{ value, paths }` — drop top-level keys |
| `fp::take` | `_.take` | `{ value, n }` — first n array elements / string chars |
| `fp::drop` | `_.drop` | `{ value, n }` — skip the first n |
| `fp::map` | `_.map` | `{ value, path }` — pluck a pointer from each element; misses → null |
| `fp::filter` | `_.filter` | `{ value, matches }` — keep elements matching a partial object |
| `fp::split` | `_.split` | `{ value, separator }` |
| `fp::join` | `_.join` | `{ value, separator? }` (default `,`) |
| `fp::uniq` | `_.uniq` | `{ value }` — dedupe, first occurrences win |
| `fp::size` | `_.size` | `{ value }` — array length / string chars / object key count |
| `fp::compact` | `_.compact` | `{ value }` — remove `null` elements (`0`/`false`/`""` are kept) |
| `fp::nth` | `_.nth` | `{ value, n }` — element at index; negative counts from the end |
| `fp::getOr` | `fp.getOr` | `{ value, path, default }` — pointer value, or `default` on a miss |
| `fp::flatten` | `_.flatten` | `{ value }` — unnest one level |
| `fp::sortBy` | `_.sortBy` | `{ value, path }` — stable ascending sort by a plucked pointer (`""` = the element itself) |
| `fp::reverse` | `_.reverse` | `{ value }` — reversed copy (immutable, fp-style) |
| `fp::sum` | `_.sum` / `_.sumBy` | `{ value, path? }` — total; `path` plucks the addend from each element. Empty array totals `0` |
| `fp::mean` | `_.mean` / `_.meanBy` | `{ value, path? }` — arithmetic mean; an empty array errors |
| `fp::min` | `_.min` / `_.minBy` | `{ value, path? }` — smallest NUMBER (not the element holding it); an empty array errors |
| `fp::max` | `_.max` / `_.maxBy` | `{ value, path? }` — largest NUMBER (not the element holding it); an empty array errors |
| `fp::groupBy` | `_.groupBy` | `{ value, path }` — `{ key: [elements] }` bucketed by a plucked key (`""` = the element itself) |
| `fp::countBy` | `_.countBy` | `{ value, path }` — `{ key: count }` with the same key rules |
| `fp::when` | — | `{ value, path?, op, to? }` — `{ passed, value }`; as a pipe step a failing comparison stops the pipe (`short_circuited: true`) |
| `fp::condition` | — | `{ event, condition_config: { path?, op, to?, negate? } }` — `{ decision: "allow" \| "skip", reason }`: the `fp::when` comparison as a guard in a harness observer's `conditions` (`harness::observe`) |

The reductions are the worker's arithmetic: all-integer inputs fold to an
integer (so the result compares cleanly in a `fp::when` guard), a non-numeric
element errors instead of being skipped — a skipped row is a total that
silently covers less than the caller counted — and `min`/`max` return the
number rather than lodash's `…By` element so a guard can compare it directly.

`groupBy`/`countBy` add the per-key axis the reductions lack: `countBy` gives
counts in one step, and `groupBy`'s buckets each feed `fp::sum` for a per-key
total. Group keys must be a string, number, or boolean — lodash coerces a null
or object key to `"null"`/`"[object Object]"`, silently merging distinct groups
into one bucket, so those error here instead.

Transforms take their input at `value` and return `{ value }`. They deviate
from lodash where silence would thread garbage through a pipe: a type
mismatch is an error (not a silent `{}`); a `map` path matching NO element
errors naming what was available (pointers pluck stored fields — no computed
properties like `/length`; `fp::size` counts); `compact` removes only `null`
(lodash's full-falsey removal would eat legitimate `0`/`false`/`""`);
`nth` out of bounds and `size` on a non-collection error instead of returning
`undefined`/`0`; `sortBy` requires every element to carry the key with one
comparable type. Sporadic `map` misses still become `null`, like lodash —
`fp::compact` drops them.

## The pipe

```jsonc
fp::pipe { through: [
  { function: "browser::fetch",
    payload:  { url: "https://…", format: "markdown", main_content_only: true } },
  { function: "fp::get",  payload: { path: "/content" } },
  { function: "fp::take", payload: { n: 20000 } },
  { function: "state::set",   payload: { scope: "research", key: "article" } }
]}
```

- Step N's result lands in step N+1's payload at `into` (a JSON pointer,
  default `/value` — which is exactly where the transforms and `state::set`
  read their input, so most pipes need no `into` at all).
- A transform step threads the transformed value itself — the `{ value }`
  wrapper in the function table is only its direct-call response shape, so
  there is never a `{value}` layer to unwrap between steps.
- The FIRST step receives no threaded value: start with a producing function
  (a fetch, a `state::get`) or seed a leading transform via `payload.value`.
  An unseeded leading transform fails validation before anything runs.
- `fp::*` transform steps run inline in this worker; every other step is
  one bus call with a 120 s budget. The whole pipe must also fit the
  caller's own dispatch timeout on `fp::pipe`.
- 1–12 steps. The response is receipts only: `{ steps: [{function, chars}],
  value_preview }` (`preview_chars` sizes the preview, default 400, capped at
  8000 — the receipt must never become the bulk channel it replaces).
- A failing step stops the pipe; the error carries the completed-step trail.

## The batch (experimental)

```jsonc
fp::batch { calls: [
  { id: "info",   function: "engine::workers::info", payload: { name: "todo" } },
  { id: "http",   function: "http::status" },
  { id: "logs",   function: "compose::logs", payload: { container: "todo", tail: 40 } }
]}
```

- Independent calls, nothing threaded. They run at once, or `concurrency`
  at a time; `stop_on_error: true` runs them in order and stops after the
  first error (the rest answer `SKIPPED`).
- One result per call, in order: `{ id, function, status: "ok", result,
  chars, truncated? }` or `{ id, function, status: "error", code, message }`.
  `id` defaults to the position. `ok` is true only when every call answered.
- Defaults: 10 s per call (`timeout_ms`, per call or per request), 16 KB per
  result (`max_entry_bytes`; a bigger result comes back as the first bytes
  of its JSON text with `truncated: true` and the full `chars`), 12 calls
  per request (`max_calls`; more still runs, with a warning that the
  response may be too large to deliver; hard ceiling 64).
- Same refusals as pipe steps, plus no nested batch and no pipe as a call. A batch is not a pipe step, and a pipe is not a batch call.
- Every refusal lists all the problems of the request, not only the first; fix them all in one resend.
- `completed` and `failed` count the outcomes; `ok` is true only when `failed` is 0.
  Same approval surface: the batch runs with this worker's authority.

### Boundaries

- Steps run with THIS worker's authority, not the calling agent's per-step
  dispatch policy — the `fp::pipe` call itself is the policy/approval
  surface (an approver sees the full step list). For that reason the pipe is
  not agent-callable without approval by default; the pure transforms are.
  See `iii-permissions.yaml`.
- `shell::*`/`coder::*` steps run under the harness-forwarded filesystem
  scope: the harness stamps the trusted `fs_scope` onto the `fp::pipe` call
  itself (`harness/src/filesystem_scope.rs`) and fp re-stamps it onto each
  scoped step as the last write before dispatch, overwriting anything
  authored or threaded at `/fs_scope`. Without a stamp (no session working
  directory, or a non-harness caller — cron, worker-to-worker) scoped steps
  are refused, fail-closed. A path-access rejection inside a pipe just fails
  the step; only a direct call offers the access-grant ladder.
- Statically refused as steps: `fp::batch` and `fp::pipe` (no nesting), `engine::observers::dispose` (observer
  control; agents observe through `harness::observe`), nested pipes, and every class the
  agent policy hard-denies — session writes and `session::store::*`,
  `approval::*`, `configuration::*`/`oauth::*` (credentials),
  `router::*`/`provider::*` (model spend), `harness::*`/`run::*` (turn
  control), `workflow::start` (the harness hook stamps the caller's session
  and folder on a direct call; `workflow::status`/`stop` are allowed), `stream::*` (the deprecated iii-stream surface) and bus internals, `*::on-config-change` — because
  steps run with worker authority and must not ride past those denies.
  Session READS (`session::get`/`list`/`messages*`/`get-message`/
  `get-attachment`/`list-attachments`) are allowed, like `database::query`:
  the policy only gates them behind approval, and the pipe call is that
  surface — pulling an attachment out of the session and threading it on is
  exactly what a pipe is for. `state::*` is the deliberate exception on the
  write side: persisting the threaded value is the pipe's purpose.
  `state::claim-namespace` and `provider-openai-codex::state::*` remain
  operator-only and cannot be pipe steps, even when the pipe itself is approved.
