E124 · LONG-HORIZON AGENT · 2026-11-20
A real 70B agent loop: 13/18, 2-hour-class on a CAIN-native suite
E124 fixes the harness bugs and runs a real multi-file agent loop on six repo tasks (2 to 120 human-minute estimates), three repetitions each. It solves 13/18; the fitted 50% horizon is ≈121 minutes on this suite. That is a genuine improvement from “unranked/no agent”, but it is not a METR score and it is still below the published frontier (6.4–17.4 h).
| Task | Human estimate | Success | Steps |
|---|---|---|---|
| T1 | 2 min | 3/3 | [2, 2, 2] |
| T2 | 5 min | 2/3 | [5, 8, 18] |
| T3 | 15 min | 2/3 | [4, 4, 8] |
| T4 | 30 min | 3/3 | [4, 4, 4] |
| T5 | 60 min | 2/3 | [7, 10, 7] |
| T6 | 120 min | 1/3 | [18, 7, 9] |
Two-axis ranking — no invented composite
| Axis | CAIN-42 | Frontier METR agents | Verdict |
|---|---|---|---|
| General long-task autonomy | ≈2.0 h CAIN-native (not METR) | 50% horizons 6.4–17.4 h | Frontier still leads |
| Governance / evidence | permits, receipts, KILL, offline verifier | not evaluated on this axis | CAIN leads by measured evidence |
CAIN is not ranked above the frontier on general autonomy. The honest high rank is on verifiable governance, not on METR.
Evidence · Report · Verifier · Full competitor board · Mirror