CAIN-42 AGAINST RECOGNIZED AGI BENCHMARKS · 2026-11-19

What CAIN-42 has and has not been measured on

CAIN-42 is governed AGI infrastructure; it is not yet AGI. No recognized AGI benchmark has been passed or scored at a competitive level. This table separates what was run from what was not, and why.

BenchmarkStatusCAIN-42 runLimit / blocker
AMA-Bench (long-horizon agent memory, ICML 2026)IN PROGRESSE115 Track A: official token-F1 on the real-world open-ended set (2,496 QA; dev 636 / sealed 1,860), memory arms M0-M4the paper's headline accuracy uses an LLM judge (Qwen3-32B or GPT-5.2) that cannot run here: judge accuracy BLOCKED; F1 only
ARC-AGI-2 (abstraction and reasoning)PLANNEDnone yet; planned with a governed local model after the E113 sealed run frees the CPUlocal 0.5B-1.5B CPU models; expected near-zero; the public set is not the private leaderboard
ARC-AGI-1NOT TESTEDnone
Humanity's Last ExamBLOCKEDnoneneeds a frontier model; the configured API key has no credit; local 0.5B-1.5B models are not meaningful here
GPQA DiamondNOT TESTEDnone
GAIA (general assistant)BLOCKEDnoneneeds web/tool access plus a frontier model; no API credit
SWE-bench VerifiedBLOCKEDnoneneeds container-isolated repository execution and a frontier model; no API credit
OSWorld / WebArena (computer and web agents)NOT TESTEDnone
LongMemEval / LoCoMo (conversational memory)BLOCKEDnoneofficial scoring uses an LLM judge that cannot run here
GSM8K (grade-school math)RANE72 small sample: 1/10 with a local model10 items; tiny local model; not a benchmark result

Internal constructed suites (not AGI benchmarks): E75 single-function repair 8/8; E93 procedure synthesis 369/510; E95 composition 79/90; E107/E108 hidden-rule discovery; E110 transfer 4/5 pathways; E113 collective vs individual (running); E114 research lab.

status.json · record