# E89 CL-Bench Results

Official Continual Learning Bench harness (github.com/pgasawa/continual-learning-bench), task `blind_spectrum_monitoring`, default schedule (90 scans, three stages), corpus `mixed_grid_lifecycle`, official scorer, unmodified task code. Model for every arm: `claude-opus-5-5`. Every arm's stateless baseline is the harness's own reset-each-instance run.

| Arm | Runs | Stateless baseline / scan | Rollout / scan | Gain / scan (95% CI) | Scans better | Spend |
|---|---|---|---|---|---|---|
| CAIN structured memory (E88 store + consolidation) | 3 | 0.2196 | 0.4386 | +0.2190 (+0.2086, +0.2281) | 90/90 | $7.41, 881,973 in / 193,870 out |
| CAIN structured memory, replication by us (fresh process) | 2 | 0.2196 | 0.4010 | +0.1814 (+0.1724, +0.1893) | 89/90 | $5.27, 636,954 in / 135,912 out |
| CAIN basic retrieval (E88 store, top-3 lexical) | 3 | 0.2196 | 0.3661 | +0.1465 (+0.1389, +0.1539) | 90/90 | $7.70, 979,349 in / 189,346 out |
| Plain in-context learning (upstream icl) | 1 | 0.2196 | 0.6371 | +0.4175 (+0.3849, +0.4478) | 89/90 | $17.90, 4,109,758 in / 72,981 out |
| haiku_cain_retrieval | 3 | 0.2196 | 0.2198 | +0.0002 (-0.0025, +0.0030) | 44/90 | $0.00, 0 in / 0 out |
| haiku_icl | 3 | 0.2195 | 0.4590 | +0.2395 (+0.2211, +0.2570) | 90/90 | $1.60, 13,943,174 in / 415,397 out |
| haiku_cain_structured | 3 | 0.2196 | 0.4199 | +0.2003 (+0.1860, +0.2133) | 90/90 | $0.29, 988,621 in / 387,514 out |
| haiku_cain_hybrid | 3 | 0.2196 | 0.4300 | +0.2105 (+0.2001, +0.2192) | 90/90 | $0.46, 2,843,947 in / 356,659 out |
| cain_hybrid | 3 | 0.2196 | 0.4782 | +0.2587 (+0.2438, +0.2720) | 90/90 | $11.44, 2,137,527 in / 144,461 out |
| poker_haiku_cain_hybrid | 1 | 0.3458 | 0.2450 | -0.1008 (-0.5850, +0.4125) | 34/90 | $0.46, 3,493,111 in / 226,894 out |
| poker_haiku_icl | 1 | 0.2058 | 0.0500 | -0.1558 (-0.7133, +0.4383) | 31/90 | $2.09, 19,792,181 in / 229,603 out |
| poker_r2_haiku_cain_hybrid | 1 | 0.3117 | 0.2000 | -0.1117 (-0.5192, +0.3183) | 35/90 | $0.44, 3,288,162 in / 223,510 out |
| poker_r2_haiku_icl | 1 | 0.7592 | 0.0208 | -0.7383 (-1.5158, +0.0142) | 38/90 | $1.31, 12,039,897 in / 204,479 out |

## Paired comparisons (per-scan mean reward)

| Comparison | Mean difference | 95% CI | Sign test p | Scans won |
|---|---|---|---|---|
| cain_structured − cain_structured_rep | +0.0376 | (+0.0348, +0.0402) | 1.5e-25 | cain_structured 89 / cain_structured_rep 1 |
| cain_structured − cain_retrieval | +0.0725 | (+0.0662, +0.0786) | 5.8e-25 | cain_structured 87 / cain_retrieval 1 |
| cain_structured − icl | -0.1985 | (-0.2236, -0.1727) | 1.4e-16 | cain_structured 8 / icl 82 |
| cain_structured − cain_hybrid | -0.0397 | (-0.0474, -0.0315) | 8.1e-09 | cain_structured 18 / cain_hybrid 72 |
| cain_structured_rep − cain_retrieval | +0.0349 | (+0.0293, +0.0403) | 5.2e-13 | cain_structured_rep 78 / cain_retrieval 12 |
| cain_structured_rep − icl | -0.2361 | (-0.2615, -0.2098) | 1.5e-25 | cain_structured_rep 1 / icl 89 |
| cain_structured_rep − cain_hybrid | -0.0773 | (-0.0853, -0.0689) | 2e-22 | cain_structured_rep 3 / cain_hybrid 87 |
| cain_retrieval − icl | -0.2710 | (-0.2970, -0.2437) | 1.5e-25 | cain_retrieval 1 / icl 89 |
| cain_retrieval − cain_hybrid | -0.1122 | (-0.1214, -0.1026) | 1.5e-25 | cain_retrieval 1 / cain_hybrid 89 |
| icl − cain_hybrid | +0.1588 | (+0.1403, +0.1768) | 2.5e-23 | icl 86 / cain_hybrid 2 |
| haiku_cain_retrieval − haiku_icl | -0.2392 | (-0.2567, -0.2209) | 3.2e-27 | haiku_cain_retrieval 0 / haiku_icl 89 |
| haiku_cain_retrieval − haiku_cain_structured | -0.2001 | (-0.2128, -0.1860) | 3.2e-27 | haiku_cain_retrieval 0 / haiku_cain_structured 89 |
| haiku_cain_retrieval − haiku_cain_hybrid | -0.2102 | (-0.2185, -0.2004) | 3.2e-27 | haiku_cain_retrieval 0 / haiku_cain_hybrid 89 |
| haiku_icl − haiku_cain_structured | +0.0391 | (+0.0254, +0.0526) | 1.9e-06 | haiku_icl 67 / haiku_cain_structured 22 |
| haiku_icl − haiku_cain_hybrid | +0.0290 | (+0.0157, +0.0423) | 0.29 | haiku_icl 50 / haiku_cain_hybrid 39 |
| haiku_cain_structured − haiku_cain_hybrid | -0.0101 | (-0.0179, -0.0031) | 0.2 | haiku_cain_structured 51 / haiku_cain_hybrid 38 |
| poker_haiku_cain_hybrid − poker_haiku_icl | +0.1950 | (-0.1208, +0.5092) | 0.13 | poker_haiku_cain_hybrid 32 / poker_haiku_icl 20 |
| poker_haiku_cain_hybrid − poker_r2_haiku_cain_hybrid | +0.0450 | (-0.1883, +0.2908) | 0.87 | poker_haiku_cain_hybrid 20 / poker_r2_haiku_cain_hybrid 18 |
| poker_haiku_cain_hybrid − poker_r2_haiku_icl | +0.2242 | (-0.2183, +0.7208) | 1 | poker_haiku_cain_hybrid 25 / poker_r2_haiku_icl 24 |
| poker_haiku_icl − poker_r2_haiku_cain_hybrid | -0.1500 | (-0.4125, +0.0958) | 0.14 | poker_haiku_icl 17 / poker_r2_haiku_cain_hybrid 28 |
| poker_haiku_icl − poker_r2_haiku_icl | +0.0292 | (-0.3667, +0.4292) | 0.47 | poker_haiku_icl 21 / poker_r2_haiku_icl 27 |
| poker_r2_haiku_cain_hybrid − poker_r2_haiku_icl | +0.1792 | (-0.1833, +0.5583) | 0.75 | poker_r2_haiku_cain_hybrid 18 / poker_r2_haiku_icl 21 |

## Context: published leaderboard (other model versions, 5 runs each; not a controlled comparison)

| System | Gain / scan |
|---|---|
| icl-gpt-5.4 | 0.294 |
| claude-code-sonnet-4.6 | 0.272 |
| icl-claude-sonnet-4.6 | 0.187 |
| icl-notepad-claude-sonnet-4-6 | 0.180 |
| mem0-gpt-5.4 | 0.156 |
| icl-claude-opus-4.7 | 0.153 |
| icl-gemini-3-flash | 0.148 |
| icl-gemini-3.1-pro-preview | 0.147 |
| codex-gpt-5.4 | 0.145 |
| icl-notepad-gpt-5.4 | 0.135 |
| icl-notepad-gemini-3.1-pro-preview | 0.104 |
| ace-gpt-5.4 | 0.000 |

Total stage-1 spend across all attempts: $99.36 (includes one ICL attempt whose billing-only prompt caching never hit and which produced no complete run).

## What this shows and does not show

- Shows: on an externally authored continual-learning task, CAIN's structured memory produces a large, consistent gain over the same frontier model's stateless baseline, and beats basic retrieval.
- Does not show: a result on CL-Bench's other five tasks; a result with other models; independent replication by another party; anything about AGI or VAGI.
- Known caveat: the structured consolidation merges list items from the model's own earlier reports. On a task whose action is a list of persistent entities (BSM) that is well matched; on tasks without list-valued actions it reduces to episodic memory. It was fixed before any real-model BSM run (k=3, rel_tol=0.02, max_rows=24).
- Skill promotion and causal retrieval are NOT EVALUABLE on BSM: the per-scan reward is hidden from the system.
- Deviation: a sitecustomize shim routes every arm to the harness's prompt-based structured output, because claude-opus-5-5 rejects the forced tool_choice that this litellm version uses for response_format.

