# E91 Vertical Benchmark Report

Generated by `scripts/evolution91/build_reports_ec.py` from `artifacts/e91/analysis/core_analysis_*.json`, which `scripts/evolution91/analyze.py` recomputes from the raw rows. These are 160 held-out tasks (`benchmarks/e91/HELD_OUT_TASK_MANIFEST.json`) built by this project. They are **not an external benchmark**, and acceptance comes only from the deterministic checkers.

## Model: claude-haiku-5-5

Rows per arm: {'A': 160, 'B': 160, 'C': 160, 'D': 160, 'E': 160, 'F': 160, 'G': 160}. An arm with fewer than 160 rows did not finish, and its rate covers only the tasks it completed. The paired contrasts use only tasks both arms finished.

| Arm | Configuration | Accepted | Rate | 95% CI | Model calls | Governed tool calls |
|---|---|---|---|---|---|---|
| A | model only | 82/160 | 51.2% | 43.6%–58.9% | 160 | 0 |
| B | + recent experience (context) | 85/160 | 53.1% | 45.4%–60.7% | 160 | 0 |
| C | + BM25 retrieval | 90/160 | 56.2% | 48.5%–63.7% | 160 | 0 |
| D | governed agent + tools | 114/160 | 71.2% | 63.8%–77.7% | 304 | 112 |
| E | D + E90 memory advice | 114/160 | 71.2% | 63.8%–77.7% | 343 | 178 |
| F | D + plan step | 100/160 | 62.5% | 54.8%–69.6% | 415 | 89 |
| G | verified learning (skills) | 153/160 | 95.6% | 91.2%–97.9% | 80 | 240 |

### By vertical (accepted/n)

| Vertical | A | B | C | D | E | F | G |
|---|---|---|---|---|---|---|---|
| business | 7/40 | 4/40 | 12/40 | 22/40 | 22/40 | 11/40 | 39/40 |
| education | 19/20 | 20/20 | 20/20 | 19/20 | 19/20 | 19/20 | 19/20 |
| engineering | 12/30 | 13/30 | 13/30 | 19/30 | 18/30 | 18/30 | 30/30 |
| science | 16/30 | 19/30 | 18/30 | 24/30 | 20/30 | 20/30 | 27/30 |
| security | 18/20 | 18/20 | 17/20 | 18/20 | 20/20 | 19/20 | 18/20 |
| software | 10/20 | 11/20 | 10/20 | 12/20 | 15/20 | 13/20 | 20/20 |

### By family

| Family | A | B | C | D | E | F | G |
|---|---|---|---|---|---|---|---|
| allocation | 1/10 | 0/10 | 0/10 | 5/10 | 3/10 | 2/10 | 10/10 |
| beam_sizing | 0/10 | 1/10 | 1/10 | 4/10 | 2/10 | 7/10 | 10/10 |
| bug_fix | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 |
| chain_diagnosis | 2/10 | 2/10 | 2/10 | 5/10 | 6/10 | 1/10 | 10/10 |
| claim_check | 6/10 | 9/10 | 8/10 | 7/10 | 6/10 | 10/10 | 7/10 |
| fault_localization | 0/10 | 1/10 | 0/10 | 2/10 | 5/10 | 3/10 | 10/10 |
| hypothesis_test | 0/10 | 0/10 | 0/10 | 10/10 | 4/10 | 0/10 | 10/10 |
| learning_plan | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 |
| logged_ab | 3/10 | 0/10 | 5/10 | 3/10 | 4/10 | 2/10 | 10/10 |
| misconception_diagnosis | 9/10 | 10/10 | 10/10 | 9/10 | 9/10 | 9/10 | 9/10 |
| remediation | 8/10 | 9/10 | 8/10 | 8/10 | 10/10 | 10/10 | 8/10 |
| replan | 0/10 | 4/10 | 3/10 | 7/10 | 5/10 | 4/10 | 10/10 |
| scenario_test | 3/10 | 0/10 | 4/10 | 7/10 | 10/10 | 3/10 | 9/10 |
| skewed_prior_diagnosis | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 |
| underpowered | 10/10 | 10/10 | 10/10 | 7/10 | 10/10 | 10/10 | 10/10 |
| vuln_detect | 10/10 | 9/10 | 9/10 | 10/10 | 10/10 | 9/10 | 10/10 |

### Paired contrasts

| Contrast | Pairs | Control | Treatment | Δ pp | Control-only | Treatment-only | exact McNemar p |
|---|---|---|---|---|---|---|---|
| B vs A | 160 | 82 | 85 | 1.88 | 10 | 13 | 0.677639 |
| C vs A | 160 | 82 | 90 | 5.0 | 4 | 12 | 0.076813 |
| D vs A | 160 | 82 | 114 | 20.0 | 11 | 43 | 1.4e-05 |
| G vs B | 160 | 85 | 153 | 42.5 | 5 | 73 | 0.0 |
| G vs C | 160 | 90 | 153 | 39.38 | 4 | 67 | 0.0 |
| E vs D | 160 | 114 | 114 | 0.0 | 17 | 17 | 1.0 |
| F vs D | 160 | 114 | 100 | -8.75 | 32 | 18 | 0.064909 |
| G vs D | 160 | 114 | 153 | 24.38 | 0 | 39 | 0.0 |

### H3: where G's gain comes from

- Families with a skill candidate: D 70 → G 109 of 110 (p = 0.0).
- Families with no fitting strategy (bug_fix, vuln_detect, remediation, claim_check, misconception): D 44 → G 44 of 50 (p = 1.0).
- G's choices: {'EXACT_SKILL': 80, 'NO_SKILL': 60, 'TRANSFERRED_SKILL': 10, 'TRANSFER_REFUSED_OUTSIDE_VERIFIED_CONDITIONS': 10}.

## local-qwen0.5b

No held-out rows yet (run incomplete).
