# E89 Baseline Forensic Audit (E88 at `f2d12e8b6`)

Date: 2026-11-17. Auditor: E89 session (same project; not independent).
Raw matrix: `research/E89_E88_REPRODUCTION_MATRIX.json`, produced by
`scripts/evolution89/reproduce_e88.py`, which recomputes every figure from E88's raw rows and never
reads E88's own summaries as truth.

## 1. Repository state

| Check | Result |
|---|---|
| E88 head | `f2d12e8b6` "E88: acceptance aggregate corrected to 14 PASS / 4 PARTIAL" (parent `e7be67376`) |
| E88 worktree | `/root/cain-e88-worktree`, clean, on `evolution-88` |
| E89 isolation | `/root/cain-e89-worktree` on `evolution-89` at the same commit; E88 untouched |
| Earlier partial E89 work | uncommitted files from a session 20:50–21:43Z (internal 200-task benchmark) preserved, not used as evidence here |
| Production | not touched (no DNS, Stripe, gateway, customer data) |

## 2. Tests and bundle

| Check | Result |
|---|---|
| `pytest tests/test_e88_*.py` (E88 worktree) | **94 passed** |
| E86/E87 regression (`artifacts/e89/repro/pytest_regression.txt`) | **47 passed** |
| `verify_e88_offline.py artifacts/e88/bundle` | **236/236 checks, ok** |
| `--tamper-test` | tampered `RELEASE_STATUS.json` detected (1 check failed, as intended) |

## 3. Results recomputed from raw rows

11 of 11 headline numbers **REPRODUCED**; 1 diagnosis and 1 inspection recorded.

| Claim | Reported | Recomputed |
|---|---|---|
| Phase-1 correct per arm (n=15) | A3 B4 C4 D5 E4 F7 G4 H5 | identical |
| Phase-2 (n=15) | C4 H0 | identical |
| Planning F vs C | +20 pp, p=0.375 | identical (McNemar exact) |
| Full H vs C, phase 1 | +6.7 pp, p=1.0 | identical |
| Full H did not beat planning F | F7 vs H5 | identical; F vs H paired p=0.625 |
| Phase 2 H vs C | −26.7 pp, p=0.125 | identical |
| Model scale | 135M 0/0, 0.5B 1→0, 1.5B 6→5 of 10 | identical |
| Teams (correct / false accept / false reject, n=10) | single 2/8/0 … heterogeneous 1/0/4, team_memory 1/9/0 | identical |
| Fence, 10/50/100/250 workers | 0 effects after fence | identical from every raw effect timestamp; denial counters match |
| One-hour endurance | 71,697 tasks, 0 duplicates | identical from the four raw effect files |

## 4. Denominator and reporting ambiguities

1. **"30 held-out tasks"** is the suite size. Every per-arm table uses **n = 15 per phase**: 15
   phase-1 tasks and 15 different phase-2 tasks, 0 overlap. Only arms C and H were run on all 30.
   Arms A, B, D, E, F, G were run on 15. Any sentence implying a 30-task comparison for those arms
   is wrong.
2. **"−10 pp model-scale disadvantage"** is **one task out of ten** at each of two scales. It is a
   direction, not a measured disadvantage. p = 1.0 at both scales.
3. **"Full stack did not outperform planning"** reproduces as a direction (7 vs 5). It is not
   significant (p = 0.625). It is not evidence that the full stack is worse.
4. **Endurance JSON keeps only the last 12 of about 60 checkpoints**, and PSS was not captured. The raw
   effect files are complete, so the task count reproduces.

## 5. Phase-2 memory regression: diagnosis from raw rows

- The full stack H used a stored procedure ("skill") on **13 of 15** phase-2 tasks and got **0 of 15**
  right. Control C got 4 of 15 right.
- 9 of 15 H tasks hit `VERIFY_MISMATCH_RETRY`. H produced 858 output tokens against C's 337.
- The families C solved and H lost were `date_diff`, `ratio_scale` and `weighted_mean`.
- Mechanism: a procedure learned in phase 1 was applied to new instances whose answer format
  differed (for example "11%" for `percent_change`, "2^28 - 2" for `subnet_hosts`). The verification
  retry then spent the remaining budget. This is **negative transfer from reused procedures**, not
  noise.

## 6. Construction bias and contamination

- **Seeded experience maps task families, not instances.** The memory arm E reuses a procedure for
  the *same family* on a new instance. That is in-distribution procedure reuse. It is not
  cross-family learning and not continual learning in CL-Bench's sense.
- The benchmark, tasks, oracle and arms are all authored by the same project. No external task
  author is involved, so E88's own "+67.5 pp" latent-rule result says nothing about external
  validity.
- The teams experiment uses one 0.5B model for every agent except the 1.5B reviewer. Agreement is
  therefore weak evidence. E88 already says so, and the raw rows confirm it (agreement 1.00 with
  8/10 false accepts).

## 7. Consequences for E89

- External validity must come from benchmarks CAIN did not write (CL-Bench, MemoryArena,
  AMA-Bench, AgentLAB).
- Memory must be judged on future-task outcomes, not recall. The phase-2 result shows memory can
  hurt.
- Collaboration claims need independent evidence channels, not same-model agreement.
- Every E89 figure is recomputed from raw rows by a script shipped with the evidence.
