# E91 Experiment Protocol (pre-registered)

Committed before the held-out phase of the core run started. The held-out task digests are in
`HELD_OUT_TASK_MANIFEST.json`; any later change to tasks, checkers, arms or hypotheses after the
first held-out row is written must be recorded as a deviation in the final report.

Two sessions build E91 together (see `EVOLUTION91_COORDINATION.md`):

| Session | Owns |
|---|---|
| `ec` | tasks, harness, arms A–G, learning, transfer, research and taxonomy (Parts I–V, IX) |
| `1b` | causal and adaptive arms (H), team arms (I), long-horizon runs, security lab, evidence bundle (Parts VI–VIII, XI, XII) |

## 1. Tasks

- Six verticals, 16 families, procedurally generated (`cain/cain2027/e91/verticals/`).
- Splits come from disjoint seed ranges:
  - dev = 1000s, the experience pool;
  - val = 2000s, used for skill promotion only;
  - held-out = 9000s, used for evaluation only.
- Ten tasks per family per split, giving 160 held-out tasks:

  | Vertical | Held-out tasks |
  |---|---|
  | software | 20 |
  | security | 20 |
  | science | 30 |
  | business | 40 |
  | engineering | 30 |
  | education | 20 |

- Task IDs are blinded hashes. Arms receive `Task.public()` only. Ground truth lives in `Task.meta["secret"]` and never reaches an arm, a memory or a verifier prompt.
- Acceptance is decided only by the deterministic checker (`verticals.check`). A checker exception is a failure.
- These are constructed tasks written by the same project that evaluates them. They are not an official external benchmark. The external benchmark arm is E89's CL-Bench harness, which is reported separately.

## 2. Arms

All arms use the same local model, Qwen2.5-0.5B-Instruct with greedy decoding, and the same task text.

| Arm | Definition | Model calls per task (max) |
|---|---|---|
| A | model only, no tools | 1 |
| B | A plus the 6 most recent dev experiences of the same vertical (prompt excerpt, answer given, verified outcome) | 1 |
| C | A plus the top-3 dev experiences retrieved by BM25 over prompt text | 1 |
| D | governed agent loop with the task's tools (every call goes through an E88 single-use grant and FencedResource) | 3 |
| E | D plus E90 causal-memory recommendation as a hint. E90 is UNVERIFIED (not committed by its owner) | 3 |
| F | D with an explicit plan step (the plan counts toward the 3 calls) | 3 |
| G | verified experience-based learning: promoted skill (strategy chosen from evidence) when its preconditions hold, else D | 0–3 |
| H | adaptive routing (session 1b) | ≤ 3 |
| I | specialists plus independent verifier (session 1b) | ≤ 3 per agent |
| J | composition of G, E, H and I | as composed |

Disclosed asymmetries:
- A, B and C cannot call tools; D, F and G can.
- G's procedures come from a designer-supplied repertoire. What G learns is which procedure fits which kind of task, and under what verified conditions it works.
- When G defaults to the plain model, it reuses D's held-out row for the same task. The procedure and greedy decoding are identical, so the output is the same.
- The experience pool for B, C and G is D's dev run. G additionally tried its repertoire strategies on the dev tasks.

## 3. Pre-registered hypotheses

- **H1, learning:** G > D on held-out acceptance (paired, exact McNemar, α = 0.05). Expected yes, through tool-using procedures in families with a fitting strategy.
- **H2, memory as text:** B and C ≤ A + 5 pp on held-out. Expected: recent or retrieved experience as context does not help a 0.5B model, consistent with E88 phase 2 and E89 CL-Bench, where ICL is the bar and memory helps mainly on cost.
- **H3, attribution:** G's gain over D comes almost entirely from families where a promoted skill applies. In the families with no promoted skill (`bug_fix`, `vuln_detect`, `remediation`, `claim_check`, `misconception_diagnosis`), G − D is within ±5 pp.
- **H4, transfer:**
  - T1: a skill learned only on `software.fault_localization` raises held-out `engineering.chain_diagnosis` above D.
  - T2: a skill learned only on `science.hypothesis_test` raises `business.scenario_test` above D.
- **H5, negative controls:**
  - With the condition gate on, transfer to `engineering.skewed_prior_diagnosis` and `business.logged_ab` is refused, and G equals D there.
  - With the gate off, the transferred skill scores at or below D (negative transfer).
- **H6, planning:** F − D is within ±5 pp. No expected gain at this model scale.
- **H7, retention:** promoted skills survive a fresh process. They are reloaded from the signed SQLite store, and held-out decisions are identical.
- **H8, poisoning:** forged experience (unknown signer, revoked signer, bad signature) is rejected on read and does not change any promotion.

## 4. Metrics

- **Per arm, vertical and family:** accepted/n, Wilson 95% CI, model calls, tokens in/out, wall seconds, governed tool calls (receipts) and denials.
- **Paired contrasts:** exact McNemar on discordant pairs, reported with raw counts. Effect size is the difference in percentage points.
- **Learning curve for G:** held-out acceptance after the first m dev experiences per signature, m ∈ {0, 1, 2, 5, 10, 20} (signatures shared by two families hold 20 dev tasks; added before held-out results existed). Deterministic strategies are re-scored; model fallback reuses D.
- **Transfer:** target-domain baseline (D), transfer (G trained on source only, gate on), negative transfer (gate off) and in-domain (G trained on target).

## 5. Limits declared in advance

- One small CPU model and one seed.
- Constructed tasks.
- No frontier model: the paid budget is $0, so frontier comparisons are BLOCKED.
- No third-party scorer.
- Governance is E88 Authority/FencedResource in-process, not the live MCPGate deployment.

## Deviations

- **D1, 2026-11-18, committed before any Haiku held-out row existed.** The local 0.5B model measured about 24.5 s per call on this host, which puts the full run at 12 h or more.
  - The owner approved Claude Haiku 5.5 through the Anthropic API with a hard $5 ceiling (`artifacts/e91/paid/APPROVAL.json`).
  - Arms A–G are run a second time on Haiku 5.5, with the same tasks, arms, hypotheses and budgets. Settings: thinking disabled, effort low, max_tokens 320.
  - The Haiku results go to `artifacts/e91/results/haiku/`.
  - The 0.5B run continues as the small-model baseline and is reported for whatever completes.
  - Hypotheses H1–H8 apply to both models. H2's "≤ A + 5 pp" is tested per model.
- **D2, 2026-11-18.** Session 1b started its arm-H Haiku runner (systemd unit `e91-adaptive-haiku`, proxy `:18891` only) a few minutes before session ec's hold message and before the owner's approval amendment.
  - Per session 1b's check, H had written no rows and made no Haiku call before the amendment. At that point it was still on its first task's local 0.5B route.
  - The amendment is recorded in `artifacts/e91/paid/APPROVAL.json`.
  - The paid ledger, `artifacts/e91/paid/ledger.jsonl`, holds token counts only and does not identify the caller. A second check against the ledger's timestamps will be made in the forensic audit.
