CAIN-42 AGAINST RECOGNIZED AGI BENCHMARKS · 2026-11-19
What CAIN-42 has and has not been measured on
CAIN-42 is governed AGI infrastructure; it is not yet AGI. No recognized AGI benchmark has been passed or scored at a competitive level. This table separates what was run from what was not, and why.
| Benchmark | Status | CAIN-42 run | Limit / blocker |
|---|---|---|---|
| AMA-Bench (long-horizon agent memory, ICML 2026) | IN PROGRESS | E115 Track A: official token-F1 on the real-world open-ended set (2,496 QA; dev 636 / sealed 1,860), memory arms M0-M4 | the paper's headline accuracy uses an LLM judge (Qwen3-32B or GPT-5.2) that cannot run here: judge accuracy BLOCKED; F1 only |
| ARC-AGI-2 (abstraction and reasoning) | PLANNED | none yet; planned with a governed local model after the E113 sealed run frees the CPU | local 0.5B-1.5B CPU models; expected near-zero; the public set is not the private leaderboard |
| ARC-AGI-1 | NOT TESTED | none | |
| Humanity's Last Exam | BLOCKED | none | needs a frontier model; the configured API key has no credit; local 0.5B-1.5B models are not meaningful here |
| GPQA Diamond | NOT TESTED | none | |
| GAIA (general assistant) | BLOCKED | none | needs web/tool access plus a frontier model; no API credit |
| SWE-bench Verified | BLOCKED | none | needs container-isolated repository execution and a frontier model; no API credit |
| OSWorld / WebArena (computer and web agents) | NOT TESTED | none | |
| LongMemEval / LoCoMo (conversational memory) | BLOCKED | none | official scoring uses an LLM judge that cannot run here |
| GSM8K (grade-school math) | RAN | E72 small sample: 1/10 with a local model | 10 items; tiny local model; not a benchmark result |
Internal constructed suites (not AGI benchmarks): E75 single-function repair 8/8; E93 procedure synthesis 369/510; E95 composition 79/90; E107/E108 hidden-rule discovery; E110 transfer 4/5 pathways; E113 collective vs individual (running); E114 research lab.