# Evolution 85 — Intelligence Evaluation Framework

**FIRME AI CORPORATION · CAIN-42 · Evolution 85** — branch `evolution-85`, baseline Evolution 84 (`2c419a042`). Development result. Not deployed. Not released. Built by the three-person team.

A completed task is not evidence of intelligence, and a benchmark score is not proof of AGI. This framework exists to
tell activity apart from capability, and to report results that do not flatter the system.

## Disclosure (required for every report)

| Item | Value |
|---|---|
| Suite | `e85-xdomain-v1`: 10 constructed task families, code-computed answers |
| Primary model | `Qwen/Qwen2.5-0.5B-Instruct`, local CPU weights, temperature 0 unless stated |
| Comparison models | `Qwen/Qwen2.5-1.5B-Instruct`, `HuggingFaceTB/SmolLM2-135M-Instruct` |
| Tools | deterministic solvers for six of ten families (arm `E_routed` only) |
| System prompt | "You are a careful analyst. Follow the requested answer format exactly." |
| Held-out sample | 100 tasks (10 per family), seeds 9000+, never shown to the learning loop |
| Validation sample | 72 tasks, seeds 2000+ |
| Scoring | exact match after a fixed parser; graded by code, never by a model |
| Statistics | Wilson 95% interval; exact two-sided McNemar on paired tasks |
| Pre-registration | written once before the first model call; hash of the held-out set `b45eb4fa31340b8d…` |
| Real model calls | 2,122 local, 0 remote |
| Not this | SWE-bench, GAIA, OSWorld, WebArena or any public benchmark |

## What each part measures

| Area | How it is measured here | Status |
|---|---|---|
| Multi-step reasoning, arithmetic | `arith_finance`, `error_detect` | measured |
| Constraint satisfaction | `constraint_schedule` | measured |
| Planning | `plan_order`, arm `C_hierarchical` | measured (narrow) |
| Causal reasoning | `causal`, `dependency_impact` | measured |
| Contradiction, rule application | `contradiction`, `access_decision` | measured |
| Uncertainty / incomplete information | `insufficient` (must answer UNKNOWN) | measured |
| Generalization to an unseen domain | `transfer_supply`, held-out only | measured |
| Learning from verified experience | arm `B_learned` vs `A_single` on held-out | measured |
| Collaboration | arms `C_specialist_verifier`, `C_hierarchical` | measured with cost |
| Consistency, pass-at-k | arm `D_selfconsistency3` | measured |
| Confidence-aware routing | arm `E_routed` | measured (tool-assisted) |
| Long-running autonomous missions | endurance run (no model calls) | measured separately |
| Uncertainty calibration, adapting to changing constraints | — | not built |

## Results (held-out, 100 tasks per arm)

| Arm | Correct | Accuracy | Wilson 95% | Calls | Tokens | ms/task | Δ vs single | McNemar p | Execution |
|---|---|---|---|---|---|---|---|---|---|
| `A_single` | 24/100 | 24.0% | 0.17–0.33 | 100 | 10,240 | 1,496 | — | — | REAL_LOCAL_MODEL |
| `A_single_135m` | 10/100 | 10.0% | 0.06–0.17 | 100 | 11,686 | 1,430 | -0.140 | 0.001312 | REAL_LOCAL_MODEL |
| `A_single_1p5b` | 40/100 | 40.0% | 0.31–0.50 | 100 | 10,212 | 5,628 | +0.160 | 0.013853 | REAL_LOCAL_MODEL |
| `B_learned` | 25/100 | 25.0% | 0.18–0.34 | 100 | 12,270 | 1,840 | +0.010 | 1.0 | REAL_LOCAL_MODEL |
| `C_hierarchical` | 26/100 | 26.0% | 0.18–0.35 | 300 | 42,205 | 14,370 | +0.020 | 0.831812 | REAL_LOCAL_MODEL |
| `C_specialist_verifier` | 24/100 | 24.0% | 0.17–0.33 | 229 | 26,421 | 10,170 | +0.000 | 1.0 | REAL_LOCAL_MODEL |
| `D_selfconsistency3` | 26/100 | 26.0% | 0.18–0.35 | 300 | 31,014 | 7,422 | +0.020 | 0.774414 | REAL_LOCAL_MODEL |
| `E_routed` | 64/100 | 64.0% | 0.54–0.73 | 100 | 4,794 | 764 | +0.400 | 0.0 | DETERMINISTIC_TOOL, REAL_LOCAL_MODEL |
| `F_floor` | 30/100 | 30.0% | 0.22–0.40 | 0 | 0 | 0 | +0.060 | 0.377086 | none |

`F_floor` answers with the most common training answer. `E_routed` uses deterministic tools where validation showed
they beat the model; its score is tool plumbing, not model intelligence.

Self-consistency (three samples at temperature 0.7): pass-at-3 38.0%, all three
agree 34.0%, mean single-sample accuracy
24.7%.

Specialist + verifier: the two agreed on 71 tasks and were both wrong on
53 of those (false accepts). The verifier shares the specialist's model,
so agreement is weak evidence.

### By family

| Family | Domain | Skill | Floor | Single | Learned | Spec+verifier | Routed |
|---|---|---|---|---|---|---|---|
| `arith_finance` | finance and accounting | multi-step arithmetic | 0/10 | 0/10 | 0/10 | 0/10 | 10/10 |
| `constraint_schedule` | business operations | constraint satisfaction | 3/10 | 1/10 | 1/10 | 1/10 | 10/10 |
| `dependency_impact` | infrastructure | transitive reasoning | 2/10 | 2/10 | 2/10 | 1/10 | 10/10 |
| `contradiction` | compliance and auditing | contradiction detection | 3/10 | 3/10 | 3/10 | 3/10 | 3/10 |
| `access_decision` | cybersecurity | rule application | 4/10 | 5/10 | 5/10 | 6/10 | 10/10 |
| `plan_order` | software engineering | planning | 2/10 | 1/10 | 2/10 | 3/10 | 1/10 |
| `insufficient` | logistics | uncertainty / incomplete information | 5/10 | 0/10 | 0/10 | 0/10 | 0/10 |
| `error_detect` | mathematics | error detection | 3/10 | 2/10 | 2/10 | 2/10 | 10/10 |
| `causal` | scientific reasoning | causal reasoning | 6/10 | 8/10 | 8/10 | 8/10 | 8/10 |
| `transfer_supply` | supply chains (unseen) | cross-domain transfer | 2/10 | 2/10 | 2/10 | 0/10 | 2/10 |

## What the results say

- **Single agent:** 24/100 (24.0%); the no-intelligence floor is 30/100 (30.0%).
- **Learning:** 25/100 (25.0%) against 24/100 (24.0%) — no statistically significant held-out improvement (Δ +0.010, p = 1.0).
- **Collaboration:** best collaborative arm `C_hierarchical` 26/100 (26.0%), Δ +0.020, p = 0.831812, at 4.12× the tokens. Not a significant gain over one agent; the extra agents cost more and did not earn it at this model scale.
- **Model size:** 0.5B 24/100 (24.0%), 1.5B 40/100 (40.0%), 135M 10/100 (10.0%). Measured, not assumed.
- **Transfer to an unseen domain:** single 2/10, with the strategy learned in the sibling domain 2/10.

## Integrity controls

Held-out seeds are disjoint from train and validation; the run aborts if a held-out prompt appears in either. The
learning ledger refuses any item whose task id is in the held-out set. Answers are re-derived from the task
specification by a second implementation in the repository and a third in the offline verifier. Every graded row is
kept, including every wrong one.

## Reproduce

    nice -n 10 python3 scripts/evolution85/run_eval.py --out artifacts/e85/eval --heldout 10 --val 8 --train 4
