# E91 Evaluation Protocol (session 1b slice)

**Status:** local, deterministic, same-project. Not an external benchmark.
**Scope:** the six verticals written by session `ec` in `/root/cain-e91-worktree`, loaded read-only. This slice builds the arm harness, the 120-task held-out manifest, the learning/transfer measurements and the honest report.
**Base:** E90 final (`2d444e70746447f3b8c70ff089e9ce0eb5f22109`), plus a vendored read-only snapshot of the six verticals in `_vertical_snapshot/`.

## 1. Task suite

120 held-out tasks, 20 per vertical:

| Vertical | Families | Held-out |
|---|---|---:|
| Software | `fault_localization` | 20 |
| Security | `vuln_detect` | 20 |
| Science | `hypothesis_test` (7), `underpowered` (7), `claim_check` (6) | 20 |
| Business | `allocation` (5), `replan` (5), `scenario_test` (5), `logged_ab` (5) | 20 |
| Engineering | `chain_diagnosis` (7), `skewed_prior_diagnosis` (7), `beam_sizing` (6) | 20 |
| Education | `misconception_diagnosis` (10), `learning_plan` (10) | 20 |

Splits: dev seeds 1000–1999, val 2000–2999, held-out 9000–9999. The manifest contains the blinded task id, vertical, family, checker name, transfer role, allowed tools, a digest and the public view. **The secret (`Task.meta`) is excluded.**

## 2. Arms

| Arm | Policy |
|---|---|
| A | no memory, family-specific default guess |
| B | last verified dev output for the same family, else default |
| C | lexical retrieval of the most similar verified dev example, copy its output |
| D | governed single probe (one tool/strategy, no persistent learning) |
| E | E90-style memory: per-family strategy selected from verified dev outcomes |
| F | planning: the family's deterministic expert policy |
| G0 | learning arm before experience (default) |
| G | learning arm after experience: strategy selected by **macro signature** (probe budget / sample budget / static) |
| H | adaptive: learned signature if known, else expert when tools exist, else probe |
| I | specialist + independent verifier: expert and probe, verifier checks output shape; disagreement falls back/abstains |
| J | full: learned signature + expert fallback + verifier |

The expert policies are **hand-written deterministic solvers**, not a model. This is disclosed. The experiment measures the memory/planning/governance architecture and the strategy-selection loop, not LLM reasoning.

## 3. Learning and transfer

- `learn(dev)` runs `default`, `probe` and `expert` on 84 dev tasks, checks each with the task's own deterministic checker, and selects the best strategy per family and per macro signature.
- G uses the signature map; E uses the family map; G0 is the same arm before learning.
- Transfer: fit only on the **source** verticals (software + science), apply to the **target** verticals (engineering + business). The macro signature is budget-type, not family/tool name, so the source strategy can be applied to an unseen family.
- Negative controls: `skewed_prior_diagnosis` (probe budget 2; bisection cannot finish) and `logged_ab` (sample budget 0; a new experiment is impossible). A transferred habit that ignores these constraints is a negative transfer.

## 4. Acceptance

Only the vertical's checker sets `accepted`. The arm never sees the checker, the secret or the held-out label. An exception in the checker is a failure, never a pass.

## 5. Metrics

Accepted rate with Wilson 95% intervals, by arm and by vertical; exact McNemar for paired arms; tool calls; latency; false accepts (self-verified but checker-rejected); abstentions; per-arm learning and transfer rates.

## 6. Reproduce

```bash
cd /root/cain-e91-eval-worktree
python3 -m benchmarks.e91.eval.harness --out artifacts/e91/eval
python3 scripts/evolution91/eval/generate_manifest.py
```

No model call, no network, no paid service.
