# E91 Evaluation Slice — Vertical Task Harness (session 1b)

**Status: PARTIAL.** The harness, six-vertical 120-task held-out suite, ten arms, learning loop, transfer study and paired statistics ran. The suite is **saturated by a hand-written expert policy**, so it cannot attribute the gain to learning or collaboration. That is the main finding, not a success to hide.

**Scope:** this session's slice only. It does not claim the other sessions' causal/adaptive/team/mission/security/evidence work. It imports their six verticals read-only (vendored snapshot in `_vertical_snapshot/`).
**Base:** E90 final `2d444e70746447f3b8c70ff089e9ce0eb5f22109`.
**Cost:** no model calls, no network, no paid service. $0.00.

## 1. What ran

- 120 held-out tasks, 20 per vertical: software, security, science, business, engineering, education.
- 10 arms A–J plus `G0` (learning arm before experience).
- 84 dev tasks used to fit strategy selection.
- 40 target tasks for the source→target transfer study (engineering + business).
- Exact deterministic checkers; the arm never sees the secret or the checker.
- One run on the final code; raw rows in `artifacts/e91/eval/rows.jsonl`, summary in `results.json`.

## 2. Headline results (120 held-out tasks)

| Arm | Accepted | Rate | 95% Wilson | Tool calls | False accepts |
|---|---:|---:|---|---:|---:|
| A no memory | 26/120 | 0.217 | 0.152–0.299 | 0 | 0 |
| B context | 26/120 | 0.217 | 0.152–0.299 | 0 | 0 |
| C lexical retrieval | 22/120 | 0.183 | 0.123–0.264 | 0 | 0 |
| D governed probe | 96/120 | 0.800 | 0.719–0.862 | 78 | 0 |
| E E90-style memory | 120/120 | 1.000 | 0.969–1.000 | 145 | 0 |
| F planning | 120/120 | 1.000 | 0.969–1.000 | 155 | 0 |
| G0 before learning | 26/120 | 0.217 | 0.152–0.299 | 0 | 0 |
| G learned strategy | 120/120 | 1.000 | 0.969–1.000 | 155 | 0 |
| H adaptive | 120/120 | 1.000 | 0.969–1.000 | 155 | 0 |
| I specialist + verifier | 120/120 | 1.000 | 0.969–1.000 | 233 | 2 |
| J full | 120/120 | 1.000 | 0.969–1.000 | 155 | 2 |

Paired exact McNemar:

| Comparison | Δ | p |
|---|---:|---:|
| A → D (tool use) | +58.33 pp | <1e-8 |
| A → E (memory) | +78.33 pp | <1e-8 |
| G0 → G (learning) | +78.33 pp | <1e-8 |
| D → F (planning) | +20.00 pp | 1.2e-7 |
| E → G | 0.00 pp | 1.0 |
| F → I (collaboration) | 0.00 pp | 1.0 |
| F → J (full) | 0.00 pp | 1.0 |

## 3. Per-vertical accepted rates

| Vertical | A | C | D | E | F | G0 | G | I |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Software | 1/20 | 2/20 | 4/20 | 20/20 | 20/20 | 1/20 | 20/20 | 20/20 |
| Security | 3/20 | 0/20 | 20/20 | 20/20 | 20/20 | 3/20 | 20/20 | 20/20 |
| Science | 11/20 | 13/20 | 18/20 | 20/20 | 20/20 | 11/20 | 20/20 | 20/20 |
| Business | 2/20 | 2/20 | 20/20 | 20/20 | 20/20 | 2/20 | 20/20 | 20/20 |
| Engineering | 2/20 | 0/20 | 14/20 | 20/20 | 20/20 | 2/20 | 20/20 | 20/20 |
| Education | 7/20 | 5/20 | 20/20 | 20/20 | 20/20 | 7/20 | 20/20 | 20/20 |

## 4. Learning result — real gain, confounded by the expert policy

- G0: 26/120 (0.217) → G: 120/120 (1.000), **+78.33 pp**, exact McNemar p < 1e-8.
- The learning loop selected a strategy from verified dev outcomes; the selected strategy was the **expert** for every macro signature (probe, sample, static).
- E (memory) also reached 120/120, and F (planning) also reached 120/120. Therefore the gain demonstrates that the loop can **select a competent policy from a portfolio**, not that it discovered the policy or that memory is the cause. The expert policies are hand-written.
- No claim of model learning, model improvement, or autonomous discovery is made.

## 5. Transfer result — positive on this suite, but not attributable to abstraction

- Trained on source verticals only (software + science), applied to target verticals (engineering + business).
- G_source: 40/40; A: 4/40; D: 32/40; F: 40/40. Paired A→G_source +90 pp, p < 1e-8.
- F (expert planning) also solved 40/40, so the transfer number cannot be attributed to the signature abstraction; the expert policy is applicable to the target families.
- Negative controls: `skewed_prior_diagnosis` 7/7 and `logged_ab` 4/5 accepted. No negative transfer was observed, but the controls also do not discriminate G from F.

## 6. Collaboration result — no measurable benefit

- I (specialist + independent verifier) = F = 120/120. **No accuracy gain.**
- I made 233 tool calls vs F 155 (+50%) and had **2 false accepts** (self-verified output rejected by the checker).
- This is a negative result for collaboration on this suite: there is no headroom for a verifier to add value when a single expert already solves the task.

## 7. What this does and does not demonstrate

**Demonstrated (same-project, deterministic):**
- The six-vertical harness can load 120 blinded held-out tasks, run ten arms under identical conditions, and score with deterministic checkers.
- A verified-experience loop can select a working strategy from a portfolio and raise accepted tasks from 21.7% to 100% on this suite.
- Tool use (D) beats no tools (A) by 58.3 pp.
- The paired statistics, task manifest, raw rows and replay are reproducible.

**Not demonstrated:**
- That CAIN becomes more capable through **model** learning. No LLM is in the loop.
- That memory or causal memory is the cause of the gain; E = F = G on this suite.
- That collaboration helps; I = F and adds cost/false accepts.
- That cross-domain abstraction transfers; F solves the targets without it.
- That the system works on real codebases, real security targets, real science, real businesses, or physical systems.
- Any long-horizon, persistence, kill-switch or ClawX result (delegated to the other E91 sessions; not claimed here).

## 8. Local acceptance

| Gate | Status | Why |
|---|---|---|
| Six verticals imported and 120 held-out tasks built | PASS | manifest 120, 20/vertical |
| Ten arms + G0 run identically | PASS | 1,440 held-out decisions + transfer rows |
| Deterministic checker is the only acceptance authority | PASS | arms never see `Task.meta` |
| Learning improves held-out performance | PARTIAL | G0→G +78 pp, but E/F also 100%; expert is hand-written |
| Cross-domain transfer measured | PARTIAL | G_source 40/40, but F also 40/40 |
| Collaboration benefit | FAIL | I = F; 2 false accepts; +50% tool calls |
| Real professional environments | FAIL | synthetic generated tasks, not real codebases/targets |
| Independent validation | BLOCKED | same-project only; no external audit |
| ClawX publication | BLOCKED | not in this slice; owner approval and other session required |

## 9. Why the suite saturates, and what E92 must change

The verticals expose deterministic tools that make each family solvable by a compact expert policy. Once an arm can call the expert, the task is solved; there is no residual uncertainty for memory, planning, adaptation or collaboration to improve. That is an instrument-design failure, not an intelligence result.

E92 should:
1. Remove the hand-written expert from the arm portfolio; make the policy **learned** from tool feedback, so strategy selection is the only way to improve.
2. Add held-out variants where the tool itself is noisy or partially reliable, so verification and collaboration have something to correct.
3. Use real repositories and real isolated security targets, not generated handlers, for at least the software and security verticals.
4. Keep the same 120-task discipline and the same deterministic checkers, but pre-register a **harder** held-out split that the expert policy does not already solve.
5. Report the ceiling effect explicitly; do not present E=F=G=J as an intelligence gain.

## 10. Reproduction

```bash
cd /root/cain-e91-eval-worktree
python3 -m benchmarks.e91.eval.harness --out artifacts/e91/eval
python3 scripts/evolution91/eval/generate_manifest.py
```

Files: `benchmarks/e91/eval/{EXPERIMENT_PROTOCOL.md,HELD_OUT_TASK_MANIFEST.json,BASELINE_RESULTS.json}`, `artifacts/e91/eval/{results.json,rows.jsonl}`.
