# E72.1 — Held-out benchmark and measured routing

`scripts/evolution72/benchmark.py` evaluates three real local models and a tool-assisted system under identical
prompts with deterministic checkers. Calibration and test items are disjoint; items are capped at 4 calibration
and 8 test per family to keep CPU time bounded.

## Families

| Family | Source | Checker |
|---|---|---|
| arithmetic | constructed | exact numeric |
| unit conversion | constructed | numeric, 1e-6 |
| logic | constructed transitivity | exact name |
| planning | constructed dependency graph | any valid topological order |
| memory recall | constructed supersession text | current value |
| extraction | constructed advisory text | exact CVE |
| code symbol | **real** `cain_workforce/*.py` files | exact top-level name (AST ground truth) |
| data statistics | **real** public Iris CSV | numeric 1 dp |
| research | **real** arXiv Atom API metadata | exact arXiv id |

## Systems

- `HuggingFaceTB/SmolLM2-135M-Instruct`
- `HuggingFaceTB/SmolLM2-360M-Instruct`
- `Qwen/Qwen2.5-0.5B-Instruct`
- **tool-assisted**: deterministic tools for code, data, extraction, planning and memory; measured model choice
  for arithmetic, conversion and logic.

## Metrics

Per model and family: `n`, `correct`, `accuracy`, Wilson 95% interval, latency, output tokens, completion rate.
Cost is $0 because all inference is local CPU.

## Interpretation

Tiny CPU models are weak at exact answer formats. The benchmark makes that visible instead of hiding it behind a
keyword score. The tool-assisted system is stronger because deterministic subtasks should be solved by tools, not
by a 135M-parameter model. This is an engineering finding, not evidence of general intelligence.

## Limits

Constructed families (arithmetic, unit, logic, planning, memory, extraction) are not a standardised external
benchmark. Code and data items use real sources, but 8 items per family gives wide confidence intervals. No remote
model was called. Results are in `artifacts/e72/benchmark/benchmark.json`.