PROJECT CAIN-2027 · EVOLUTION 79 · evidence bundle
Branch evolution-79, from E78 e2851428d2cd.
Generated 2026-11-15 15:25 UTC.
Nothing here is deployed; nobody was billed; no credits were purchased.
Qwen/Qwen2.5-0.5B-Instruct
(0.5B, weights SHA-256 fdf756fa7fcbe740…)),
167,172 tokens, $0.00 at the point of use.104 tasks, 13 in each of 8 domains, ground truth computed by code (the snippet is executed, the table summed, the critical path run). Exact-match grading. A/B/T run on all 104 tasks; every collaborative arm runs on the same 32 tasks and the single agent is re-scored on that subset.
| ARM | TASKS | CORRECT | ACCURACY | WILSON 95% | CALLS | FALSE ACCEPTS |
|---|---|---|---|---|---|---|
| deterministic heuristic (no model) | 32 | 2 | 6.2% | 0.02–0.20 | 0 | 0 |
| one specialist | 32 | 11 | 34.4% | 0.20–0.52 | 32 | 10 |
| one specialist + authorized tool | 32 | 5 | 15.6% | 0.07–0.32 | 32 | 5 |
| one specialist, 2 samples | 32 | 11 | 34.4% | 0.20–0.52 | 64 | 0 |
| specialist + critical reviewer | 32 | 12 | 37.5% | 0.23–0.55 | 76 | 0 |
| four specialists | 32 | 8 | 25.0% | 0.13–0.42 | 201 | 13 |
| eight specialists | 32 | 10 | 31.2% | 0.18–0.49 | 356 | 13 |
“False accepts” = an independent verifier agreed with an answer that was wrong. With a 0.5B model, verifier agreement is weak evidence, and the table says so instead of hiding it.
| domain | single | +tool | four | eight |
|---|---|---|---|---|
| cybersecurity | 75.0% | 0.0% | 50.0% | 50.0% |
| data_analysis | 0.0% | 0.0% | 25.0% | 25.0% |
| infrastructure | 75.0% | 0.0% | 25.0% | 50.0% |
| knowledge_synthesis | 25.0% | 50.0% | 0.0% | 0.0% |
| mathematics | 50.0% | 25.0% | 50.0% | 50.0% |
| planning | 0.0% | 0.0% | 0.0% | 0.0% |
| scientific_research | 25.0% | 50.0% | 0.0% | 25.0% |
| software_engineering | 25.0% | 0.0% | 50.0% | 50.0% |
The two-step tool protocol (write CALC: then ANSWER:) helped arithmetic marginally and hurt
label reasoning badly, because the model wrote expressions for questions with nothing to compute. A follow-up
“tool-first” iteration (the specialist writes only the expression; the calculator produces the answer) measured
25.0% over 16 numeric tasks versus 37.5% for the
same tasks without a tool. Same-task model-scale run with Qwen2.5-1.5B: 37.5%.
Question: Does an authorized calculator tool, and does multi-specialist collaboration, change exact-match accuracy on a paired closed-form benchmark compared with a single specialist?
Main run: single 0.3438, tool 0.1562, four 0.25, self-consistency 0.3438. Replication (seed 791, 32 tasks, separate process): single 0.3438, tool 0.0938.
Replicated: True · scientific discovery claimed: NOT_CLAIMED · the model's interpretation is labelled MODEL_INFERENCE and never promoted to a fact.
| arm | exact match | questions |
|---|---|---|
| A_no_memory | 10.0% | 10 |
| B_naive_context | 70.0% | 10 |
| C_verified_retrieval | 70.0% | 10 |
| D_verified_temporal | 70.0% | 10 |
Corpus built from this evolution's own verified evidence, with 1 poisoned record and a
superseded fact as traps. A memory hit is context, never permission (grants_authority: false on every hit).
57/57 scenarios blocked, 0 bypassed. 4 worker processes were SIGKILLed mid-task; 39 completions were independently recomputed and 0 duplicate external effects were applied (fencing token + idempotency key).
| layer | blocked |
|---|---|
| delegation | 2/2 |
| evidence | 5/5 |
| execution | 6/6 |
| governance | 3/3 |
| identity | 3/3 |
| memory | 7/7 |
| message_bus | 6/6 |
| prompt_injection | 2/2 |
| resource_limits | 4/4 |
| tenant_isolation | 3/3 |
| tool_authorization | 5/5 |
| tool_sandbox | 11/11 |
Weakness detection → isolated candidate → regression and adversarial suites → held-out A/B on a new seed → independent review. Measured delta -0.125. Status: AWAITING_HUMAN_APPROVAL. No production file was modified and no agent can promote itself.
75 catalogued · 75 importable · 0 model-backed · 0 mission-integrated in this evolution. The deterministic engines are not relabelled as model inference to make the matrix look complete.
Grounded synthesis mission: 7/7 mechanical checks passed; accepted = True. The plan is checked against the artifact — no invented numbers, no authority actions.
Package evidence_v2.json — 22 fields, Ed25519 signed, Merkle root
d42ff7744cfb9ab66149f8d891cac771bcfa950dd53b924f0e48133a18aadcc8, bundle SHA-256 b24b9322e57ea87ba76a95208dc2941f7ab6dec79f041a8275f9ca6cf7e966f1.
Offline verification: 144/144 checks
(verified = True).
The verifier imports no CAIN code. It recomputes every arm's correct count, accuracy, Wilson interval, token total, call count, the McNemar tests, tool denials, memory exact match, the self-improvement delta, the security counts and the durability invariants from the raw rows, then compares them to the signed values.
python3 -I verify_e79_v2_offline.py .
Open the bundle · download the ZIP · public_summary.json · cainstudio.online page
Scorecard 5.78/10 (C-);
acceptance ACCEPTED_WITH_BLOCKERS. Blockers: 15_cloud_model, 16_transparency_log, 17_endurance_6h_plus, 18_third_party_audit, 19_production_release.