FIRME AI CORPORATION, a three-person team. Baseline: Evolution 80. Every number on this page is read from the artifacts below; the headline figures are re-derived from raw rows by a script that imports no CAIN code.
Download the evidence bundle (ZIP) How to reproduce Review findings Implementation report
B_hierarchical,
gained +0.0000. The threshold fixed in the benchmark code (+0.10 with exact McNemar p < 0.05) was not met; the registration file's timestamp cannot show it was written first (finding F-04).
All inference ran on small local CPU models; no remote model was called. A later review of this evidence produced
5 findings and lowered two gates. These results are published as measured.26 PASS · 12 PARTIAL · 2 BLOCKED · 0 FAIL of 40 gates. Every gate's evidence class is LOCAL. Decision: NOT RELEASED.
| Gate | Status | Requirement | Basis |
|---|---|---|---|
| G01 | PASS | Evolution 80 baseline reproduced | E80 tests passed=30; E80 verifier ok=True |
| G02 | PASS | Latest repository ancestry verified | branch=evolution-81 E81 head=31e8ef8ad38c parent=ca043f22cd64 |
| G03 | PASS | Durable workforce runtime implemented | lease, fencing, idempotency, checkpoints, dead letters, lineage |
| G04 | PASS | Persistent agent identity verified | 99 durable identities; same key after reload; rotation invalidates old signatures |
| G05 | PASS | Durable mission state verified | mission state=COMPLETED_WITH_UNVERIFIED_ITEMS, checkpoints chain ok=True |
| G06 | PASS | Process restart recovery verified | killed pid 3742682, leases recovered 1, late commit STALE |
| G07 | PARTIAL | Host restart recovery verified | process-level restart only; no host reboot drill was performed in E81. The host lost power unplanned on 2026-11-16 13:43Z with no mission running: durable stores verified intact afterwards (takeover_review.json), which is state survival, not a mission resumed across a reboot |
| G08 | PASS | Replay-safe execution verified | single-use grants; stale fencing refused; replay refused |
| G09 | PASS | Duplicate-side-effect prevention verified | recovery duplicate effects=0; cross-host duplicate suppressed=True |
| G10 | PASS | Specialist registry implemented | 30 disciplines, 90 profiles, 8 commanders |
| G11 | PARTIAL | Specialist competency measured | measured on a 3-item discipline-neutral probe (reliability floor), not domain expertise |
| G12 | PARTIAL | Agentic75 capability composition verified | composition attenuates tools/scopes/risk and carries no authority; 75 engines executed as E80 evidence, not re-run here |
| G13 | BLOCKED | Real provider inference verified | no credential for OpenAI/Anthropic/Gemini/DeepSeek; adapter implemented, 0 remote calls |
| G14 | PASS | Multiple model families evaluated | families=qwen,smollm2 |
| G15 | PASS | Multi-agent collaboration benchmarked | 15 arms, 24 frozen tasks, 700 real calls in the scored rows; run was resumed and the pre-registration file was rewritten at resume, so its timestamp postdates the first arms (review F-03/F-04); criteria are fixed in code and the hypothesis was rejected |
| G16 | PASS | Collaboration superiority demonstrated or honestly rejected | COLLABORATION_NOT_SUPERIOR_HONESTLY_REJECTED (best delta 0.0, p=1.0) |
| G17 | PASS | Temporal memory implemented | encrypted at rest, valid-from/to, expiry, supersession, contradiction, ACL, audit |
| G18 | BLOCKED | Public memory benchmarks reproduced | dataset files are not present on this host and downloading them would require an explicit data-acquisition decision |
| G19 | PASS | Default-deny authorization verified | 60/60 blocked |
| G20 | PASS | Authority attenuation verified | child grant cannot exceed parent units/tool/expiry; composition intersects authority |
| G21 | PASS | Sandbox isolation verified | bubblewrap: no network, no host secret, read-only system, private workspace; gVisor/Kata/Firecracker absent |
| G22 | PASS | Network egress restrictions verified | network namespace with no route |
| G23 | PARTIAL | Cross-host recovery verified | drill status=PASS: a second host executed governed tasks and a lost acknowledgement was recovered without a duplicate effect. Mission state stayed on the first host; no mission was resumed on another host after its coordinator host failed, which is what this gate asks for. same administrative domain; not an independent witness |
| G24 | PASS | Leases and fencing verified | stale commit refused; fencing monotonic |
| G25 | PASS | Resource budgets enforced | mission/agent/model budgets, concurrency, retry cap |
| G26 | PASS | Kill switch verified across processes | a separate OS process stops after the switch is engaged |
| G27 | PASS | Signed mission evidence verified | files=8 merkle=d744ab90f9e6141c; the bundle covers the reference mission and the model-call ledger. The gate audit log is outside the bundle and is forked (linear=False, review F-02; writer fixed, historical file preserved) |
| G28 | PARTIAL | Independent evidence verification verified | two independent implementations (different sessions, no shared code) agree on the same bundle and Merkle root; still same project, no third party and no out-of-band key pinning |
| G29 | PARTIAL | Off-host evidence anchoring verified | second host in the same administrative domain; not an independent witness, no transparency log |
| G30 | PARTIAL | Authenticated ClawX integration verified | a local service returns real evidence only to an authenticated caller; it is not wired into or deployed on clawx.click, so integration with ClawX itself is not shown |
| G31 | PARTIAL | Real-time mission timelines verified | timeline is built from the durable event log; no live streaming/SSE and no deployment |
| G32 | PARTIAL | Endurance milestones measured | 1 h target reached=True, elapsed=3600.0s; 6/24/72/168/720/1000 h NOT run |
| G33 | PASS | Benchmark reproducibility verified | separate process identical=True |
| G34 | PASS | Cost accounting verified | model cost=$0.0, host estimate=$0.557705 over all ledger tokens; the report's call count 1156 omits early rows under an older label: ledger holds 1475 real local calls plus 114 in the endurance ledger (review F-01) |
| G35 | PARTIAL | Capacity planning completed | 10/50/100/250/500 real workers measured (0 failures, 0 duplicates); 1,000 attempted and blocked by SQLite write contention; 5,000/10,000 are logical identities with 32-way execution |
| G36 | PASS | Production deployment safety reviewed | review completed; no production change was made or attempted |
| G37 | PARTIAL | Rollback procedure verified | development rollback rehearsed by reverting the branch; production rollback not exercised |
| G38 | PASS | Security regression suite passed | 60/60 |
| G39 | PASS | Historical evidence preserved | no historical artifact was rewritten |
| G40 | PARTIAL | Independent release-readiness review completed | internal review only; no external reviewer, no third-party audit |
| Arm | Correct | Accuracy | Wilson 95% | Wins / losses vs single | McNemar p | Calls |
|---|---|---|---|---|---|---|
A_single | 10/24 | 41.7% | 0.245–0.612 | 0 / 0 | 1.0 | 24 |
B_hierarchical | 10/24 | 41.7% | 0.245–0.612 | 3 / 3 | 1.0 | 48 |
C_parallel_research | 10/24 | 41.7% | 0.245–0.612 | 0 / 0 | 1.0 | 72 |
D_evidence_first | 7/24 | 29.2% | 0.149–0.492 | 5 / 8 | 0.581055 | 24 |
E_adversarial_review | 8/24 | 33.3% | 0.180–0.533 | 1 / 3 | 0.625 | 48 |
F_debate_synthesis | 8/24 | 33.3% | 0.180–0.533 | 0 / 2 | 0.5 | 72 |
G_shared_workspace | 8/24 | 33.3% | 0.180–0.533 | 1 / 3 | 0.625 | 48 |
H_dynamic_routing | 10/24 | 41.7% | 0.245–0.612 | 0 / 0 | 1.0 | 24 |
I_fixed_routing | 10/24 | 41.7% | 0.245–0.612 | 0 / 0 | 1.0 | 24 |
J_confidence_aware | 0/24 | 0.0% | 0.000–0.138 | 0 / 10 | 0.001953 | 41 |
K_model_specialist | 10/24 | 41.7% | 0.245–0.612 | 0 / 0 | 1.0 | 24 |
L_adaptive_comm | 9/24 | 37.5% | 0.212–0.573 | 0 / 1 | 1.0 | 59 |
M_independent_vote | 10/24 | 41.7% | 0.245–0.612 | 1 / 1 | 1.0 | 72 |
N_multistage_verify | 8/24 | 33.3% | 0.180–0.533 | 1 / 3 | 0.625 | 48 |
O_cross_domain | 10/24 | 41.7% | 0.245–0.612 | 3 / 3 | 1.0 | 72 |
Constructed tasks with code-computed answers, not a public benchmark. 700 model calls in the scored rows.
| Model | Size | Correct | Mean latency |
|---|---|---|---|
Qwen/Qwen2.5-0.5B-Instruct | 0.5B | 6/12 | 1.6 s |
Qwen/Qwen2.5-1.5B-Instruct | 1.5B | 4/12 | 7.5 s |
HuggingFaceTB/SmolLM2-135M-Instruct | 135M | 0/12 | 1.4 s |
1475 real local calls in the main ledger and 114 in the endurance ledger; 70743 input and 15818 output tokens; 0 remote calls (no provider credential); model cost $0.
90 specialist profiles in 30 disciplines, 8 commanders, 99 durable identities. These are logical identities, not running agents.
| Worker processes | Tasks done | Failed | Duplicate effects | p95 latency |
|---|---|---|---|---|
| 10 | 40 | 0 | 0 | 10 ms |
| 50 | 200 | 0 | 0 | 131 ms |
| 100 | 400 | 0 | 0 | 435 ms |
| 250 | 1000 | 0 | 0 | 2146 ms |
| 500 | 2000 | 0 | 0 | 6083 ms |
Capacity stages ran deterministic governed tasks with no concurrent model inference. Largest measured stage: 500 worker processes. The 1,000-worker stage did not complete; 5,000 and 10,000 are identity records only.
Endurance: 3600 s of wall clock, 114 model calls, 0 failures. elapsed time is real wall clock; the loop is duty-cycled, not CPU-saturated. Campaigns of 6 hours and longer were not run.
Security lab: 60/60 scenarios blocked, 0 bypassed. same-project adversarial lab on one isolated host; not a third-party penetration test.
Re-derivation from raw rows: 17/17 checks pass. takeover review session, 2026-11-16; same project, not a third party.
| ID | Severity | Finding | Effect |
|---|---|---|---|
| F-01 | LOW | Reported call total undercounts the ledger. cost_report/docs say 1156 real local calls; the main ledger holds 1475 (319 early rows carry the older label LOCAL_MODEL_INFERENCE) and the endurance ledger a further 114. Total 1589. | An undercount, not an overclaim. Token totals and inference seconds in the cost report already cover every ledger row, so the cost estimate is unaffected; only the call count is wrong. |
| F-02 | MEDIUM | Gate audit log is forked, and the summary reports chain_ok over a tail only. 1 fork point(s) at seq [1987]: two pipeline processes appended to one file concurrently, each continuing from the tail it read at start-up. Every row hashes correctly and links to an existing row, so this is a concurrency defect, not tampering. gate_audit.json says chain_ok=True but checked only 12 in-process rows of 2094. The gate audit log is also not part of the signed bundle. | Writer fixed (file lock + on-disk tail) in cain/cain2027/e80/core.py with a regression test; the historical file is preserved unmodified and must not be described as an unbroken chain. |
| F-03 | LOW | Benchmark call count reflects only the final resumed process. rows account for 700 calls across 15 arms; the report says 412 (the budget of the process that finished the run after a resume); the ledger holds 855 rows for the mission, including calls from an interrupted first attempt whose arms were re-run. | Accuracy figures are unaffected. Quote 700 calls for the scored rows, not 412. |
| F-04 | MEDIUM | Pre-registration timestamp postdates the first benchmark calls. preregistration.json created_utc=1794828568.809831 but the first M-bench call is at 1794825050.862719 (59 min earlier). The file is rewritten at every process start and the run was resumed, so the original timestamp was overwritten. | The criteria are fixed in code and the hypothesis was REJECTED, so there is no selective-reporting gain, but 'written before the run' cannot be shown from the artifact. Writer fixed to never overwrite. |
| F-05 | MEDIUM | Durable stores and the branch itself existed on one host only. `*.db` is gitignored, so the SQLite mission/endurance stores were never in the repository, and evolution-81 had not been pushed to the development replica when the host lost power (the replica held evolution-80 only). The stores survived on disk; a disk loss would have left only JSON exports of them. | Stores force-added to the branch as evidence and the branch pushed to the replica by the takeover review. Replicate after every milestone commit, not at the end. |
| Shipped offline verifier | 18 / 18 checks on the contents of the published ZIP |
|---|---|
| Second implementation | 29 / 29 checks on the same ZIP. Written by a different session, imports no CAIN code. Same project, not a third party. |
| Second host | ok = True on 45.76.169.191. second host in the same administrative domain; not an independent witness. |
| Merkle root | d744ab90f9e6141c37097a0b8c4863ffe9f56cead46be786557de80724b4faa8 over 8 files of the reference-mission bundle |
| Signing key | Ed25519, self-generated, not pinned out of band: public_key.txt |
| Bundle ZIP SHA-256 | eff101d4761581cc867d722da56c64abd80bece0346756d9d59a89f298aad795 |
The signed bundle covers one reference mission and the model-call ledger. The benchmark, capacity, endurance and security figures are outside the signature; they are checked by the headline recompute only.
acceptance_matrix.json · takeover_review.json · cross_host.json · recovery_drill.json · model_evaluation.json · memory_evaluation.json · specialist_competency.json · sandbox.json · second_host_verification.json · independent_verification_session_b.json · replication.json · capacity_plan.json · workforce.json · SECRETS_REMOVED.md · public_key.txt · model_calls.jsonl · cost_report.json · gate-audit.jsonl · gate_audit.json · collaboration_rows.jsonl · collaboration_benchmark.json · preregistration.json · endurance.json · security_lab.json · capacity.json · endurance/model_calls.jsonl · verify_e81_offline.py · verify_independent.py · recompute_headlines.py
The two SQLite stores are inside the ZIP only.