# Evolution 80 — Intelligence Evaluation

> Generated by `scripts/evolution80/docs_e80.py` from `artifacts/e80/` (run 3). FIRME AI CORPORATION — a three-person team (founder: Vincent Noriega).
> Development branch `evolution-80`. Not deployed. No cloud model was called. No third-party audit.

## The question

Does a routing policy fitted on a training split beat the best single fixed execution path on unseen tasks,
and does any multi-agent architecture beat one agent at this model scale? The hypotheses were written to
`preregistration.json` before any test call was made
(written before first test call: **True**).

## Test split (48 unseen tasks, 7 task families)

| arm | what it is | correct | accuracy | Wilson 95% | model calls | tokens |
|---|---|---|---|---|---|---|
| det_exec | deterministic execution (no model) | 6/6 | 12.5% | [5.9%, 24.7%] | 0 | 0 |
| m05_direct | 0.5B direct answer | 11/48 | 22.9% | [13.3%, 36.5%] | 48 | 5282 |
| m05_pot | 0.5B writes expression, calculator executes | 5/29 | 10.4% | [4.5%, 22.2%] | 29 | 4294 |
| m15_direct | 1.5B direct answer | 19/48 | 39.6% | [27.0%, 53.7%] | 48 | 5558 |
| m15_pot | 1.5B writes expression, calculator executes | 13/29 | 27.1% | [16.6%, 41.0%] | 29 | 4318 |
| fixed_pot05_else_direct05 | fixed composite 0.5B (exploratory) | 16/48 | 33.3% | [21.7%, 47.5%] | 48 | 6599 |
| fixed_pot15_else_direct15 | fixed composite 1.5B (exploratory) | 27/48 | 56.2% | [42.3%, 69.3%] | 48 | 6733 |
| router | adaptive router (frozen on train) | 22/48 | 45.8% | [32.6%, 59.7%] | 42 | 5475 |
| oracle_any_path | oracle: any path correct (upper bound, not achievable) | 32/48 | 66.7% | [52.5%, 78.3%] | 42 | 5111 |

## Verdicts

| hypothesis | result | numbers |
|---|---|---|
| H1 router beats best fixed path chosen on train | NOT SUPPORTED | router 22/48 vs m15_direct 19/48; diff +0.0625; exact McNemar p = 0.647606; bootstrap 95% [-0.1042, 0.25] |
| H2 some multi-agent arm beats one agent | NOT SUPPORTED | best arm c08_evidence_first delta +0.1042; significant arms: none; worse than one agent: ['c07_specialist_committee'] |
| H3 direction of H1 repeats on replication | SUPPORTED | router 10/24 vs m15_direct 8/24; diff 0.0833; same direction: True |

"Supported" for H1 and H2 requires p < 0.05 on the paired exact test. H3 asks only whether the direction repeats;
a repeated direction with a non-significant difference is weak evidence, not confirmation.

## Was each routing decision right?

Oracle accuracy (any path correct) is 66.7%; the router's regret against it is
20.8%. The decision was optimal in hindsight for
2 of 7 families.

| family | tasks | path chosen | chosen correct | best path in hindsight | best correct | optimal |
|---|---|---|---|---|---|---|
| code_trace | 6 | det_exec | 6 | det_exec | 6 | yes |
| label_code_review | 6 | m05_direct | 3 | m15_direct | 5 | no |
| label_logs | 6 | m05_direct | 6 | m05_direct | 6 | yes |
| label_notes | 7 | m05_direct | 2 | m15_direct | 4 | no |
| schedule | 6 | m05_direct | 0 | m15_direct | 1 | no |
| table | 3 | m05_direct | 0 | m15_direct | 1 | no |
| word_math | 14 | m05_pot | 5 | m15_pot | 7 | no |

## What this does and does not show

- It shows measured accuracy of small local CPU models on constructed closed-form tasks, with every number recomputable from raw rows.
- It does not show general intelligence, and the tasks are unseen instances of known templates, not new task types.
- 48 test tasks detect only large effects. A non-significant difference is not evidence of no difference.
