# Frontier-model measurement — E85 held-out suite

**Model:** `claude-sonnet-5-5` (Anthropic, `REAL_REMOTE_MODEL`) · **Cost:** $0.5138 (cap $8.00) · **Calls:** 416
**Suite:** same pre-registered 100 held-out tasks / 10 families as the local E85 run; 0 local calls in this run.

## Results

| Arm | Correct | Accuracy | Wilson 95% CI |
|---|---:|---:|---|
| A_single (16-token answer) | 36/100 | 36.0% | [0.2727, 0.4576] |
| B_learned (validated strategies) | 65/100 | 65.0% | [0.5525, 0.7364] |

Paired comparison on the same 100 tasks: **+0.29** for B_learned,
exact McNemar **p = 1.3e-07** (31 tasks gained, 2 lost).

## Local baselines (same tasks)

| Arm | Correct |
|---|---:|
| Deterministic floor | 30/100 |
| Local 0.5B single | 24/100 |
| Local 1.5B single | 40/100 |
| Local learned | 25/100 |

## What this shows

- A frontier model **with validated strategies** scores **65/100** on this constructed suite,
  significantly above its own single-agent arm and above every local-model arm.
- The single-agent frontier arm (**36/100**) is limited by the 16-token answer budget
  and the strict exact-match parser; it is not evidence that the model is weaker than a 1.5B local model.
- Learned-strategy transfer is the informative result: the same validation-selected strategies that produced
  no significant local gain (25/100, p=1.0) produced a significant frontier gain (p≈0).

## What this does NOT show

- not AGI, not general intelligence, not human-level performance.
- One model, 100 constructed tasks, one run; no independent witness, reviewer or audit.
- No production integration; no customer outcome; no claim of an "official" designation.
- Cost is estimated from reported tokens and list prices, not an invoice.
