# Evolution 81 — Collaboration Benchmark

## Pre-registration

The criteria below are fixed in code. The run was resumed and the registration file was rewritten at the resume,
so its timestamp postdates the first arms; "written before the run" cannot be shown from the artifact
(takeover review F-04; the writer is now write-once). The hypothesis was rejected, so nothing was gained by it.

- Task set: 24 frozen tasks, SHA-256 `4246f22a9c0c602578f7c9a3847fb69ae35174a43db1025ab96e21028962d61b`.
- **H1:** the best collaborative arm beats `A_single` by ≥ 0.10 absolute exact-match accuracy,
  McNemar exact p < 0.05, on the same tasks.
- **H0:** no collaborative arm achieves that; collaboration is reported as not superior.
- Model: `qwen2.5-0.5b-instruct` (`REAL_LOCAL_MODEL`), 0 remote calls.

## Results

| arm | description | correct | accuracy | Wilson 95% | delta | p | tokens | messages | mean latency |
|---|---|---|---|---|---|---|---|---|---|
| A_single | one agent, one answer | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | — | 1304 | 0 | 4495.126 ms |
| B_hierarchical | planner then executor | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 3838 | 48 | 14666.723 ms |
| C_parallel_research | three independent samples, majority | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 3931 | 72 | 14567.127 ms |
| H_dynamic_routing | route by task family to the best measured model | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 1304 | 0 | 2366.824 ms |
| I_fixed_routing | route every task to one fixed model | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 1304 | 0 | 3954.864 ms |
| K_model_specialist | match model family to discipline | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 1304 | 0 | 3133.542 ms |
| M_independent_vote | three independent answers, majority | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 3947 | 72 | 10996.818 ms |
| O_cross_domain | two disciplines then synthesis | 10/24 | 41.7% | [0.245, 0.612] | 0.0 | 1.0 | 4779 | 72 | 11020.162 ms |
| L_adaptive_comm | communicate only on disagreement | 9/24 | 37.5% | [0.212, 0.573] | -0.041667 | 1.0 | 3442 | 59 | 9363.196 ms |
| E_adversarial_review | producer then independent critic | 8/24 | 33.3% | [0.180, 0.533] | -0.083333 | 0.625 | 3218 | 48 | 5335.487 ms |
| F_debate_synthesis | two debaters then synthesis | 8/24 | 33.3% | [0.180, 0.533] | -0.083333 | 0.5 | 4702 | 72 | 9464.121 ms |
| G_shared_workspace | producer writes intermediate, second agent reads it | 8/24 | 33.3% | [0.180, 0.533] | -0.083333 | 0.625 | 3349 | 48 | 8803.209 ms |
| N_multistage_verify | answer then independent verification | 8/24 | 33.3% | [0.180, 0.533] | -0.083333 | 0.625 | 3218 | 48 | 7334.646 ms |
| D_evidence_first | answer with an explicit rule/evidence step | 7/24 | 29.2% | [0.149, 0.492] | -0.125 | 0.581055 | 2162 | 0 | 8165.147 ms |
| J_confidence_aware | second sample only when confidence is low | 0/24 | 0.0% | [0.000, 0.138] | -0.416667 | 0.001953 | 2666 | 17 | 7357.129 ms |

## Decision

- Best collaborative arm: **B_hierarchical** at
  41.7%.
- Single-agent baseline: 41.7%.
- Delta: **0.0**, McNemar p = **1.0**.
- **Verdict: COLLABORATION_NOT_SUPERIOR_HONESTLY_REJECTED**.
- Worst arm: `J_confidence_aware` at 0.0%.
- Recommendation: use the single agent; collaboration cost more without a measured benefit.

## Failure analysis

The small model rarely benefits from more turns: every additional agent re-reads a short prompt and
often copies or truncates the previous answer. Communication volume rises while accuracy does not.
Where collaboration helps, the gain is concentrated in tasks where a second sample disagrees and a
majority vote can recover the correct value — a variance effect, not reasoning.

## Supplementary routing measurement (separate from the primary comparison)

In the primary run only one model was viable for every capability, so `H_dynamic_routing`,
`I_fixed_routing` and `K_model_specialist` collapsed to the same policy (identical rows, 10/24).
To measure routing honestly, a small frozen train split was scored with both Qwen sizes and three
policies were then run on the same 24 tasks:

| policy | correct | accuracy | models | mean latency |
|---|---|---|---|---|
| H_dynamic_measured | 10/24 | 41.7% | qwen2.5-0.5b-instruct | 1870.83 ms |
| I_fixed_large | 13/24 | 54.2% | qwen2.5-1.5b-instruct | 6733.44 ms |
| K_model_specialist_fixed | 9/24 | 37.5% | qwen2.5-0.5b-instruct, smollm2-135m-instruct | 1646.67 ms |

The measured dynamic router did **not** beat a fixed larger model: the 8-item train split was too
easy to discriminate the two models (they tied on 7 of 8 families), so the router defaulted to the
0.5B model. The fixed-specialist mapping without measurement was worst. The lesson is that routing
quality is bounded by the representativeness of its calibration set — not by the router mechanism.

## Reproducibility

A separate process re-ran the baseline arm with the same seed: identical rows
`True` (24 rows, SHA-256
`a9fa4054afa4ce9e…`).

## Limits

24 constructed tasks, one 0.5B CPU model, small sample. Wilson intervals and McNemar are reported;
no claim is made outside this task set. A null result here is a **useful** result: it argues against
adding agents for their own sake.
