# Evolution 81 — Model Evaluation

## Execution classes

`REAL_LOCAL_MODEL`, `REAL_REMOTE_MODEL`, `CACHED_RESPONSE`, `MOCK`, `DETERMINISTIC_FIXTURE`,
`SIMULATED`. Mocks and caches are never counted as inference.

## Registered models

| key | provider | execution class | available | blocker |
|---|---|---|---|---|
| qwen2.5-0.5b-instruct | self_hosted_hf | REAL_LOCAL_MODEL | HEALTHY | None |
| qwen2.5-1.5b-instruct | self_hosted_hf | REAL_LOCAL_MODEL | HEALTHY | None |
| smollm2-135m-instruct | self_hosted_hf | REAL_LOCAL_MODEL | HEALTHY | None |
| openai-frontier | openai | REAL_REMOTE_MODEL | BLOCKED | NO_CREDENTIAL |
| anthropic-frontier | anthropic | REAL_REMOTE_MODEL | BLOCKED | NO_CREDENTIAL |
| gemini-frontier | google | REAL_REMOTE_MODEL | BLOCKED | NO_CREDENTIAL |
| deepseek-reasoner | deepseek | REAL_REMOTE_MODEL | BLOCKED | NO_CREDENTIAL |

## Measured (12 frozen tasks, exact match, greedy decoding)

| model | family | params | correct | accuracy | mean latency |
|---|---|---|---|---|---|
| qwen2.5-0.5b-instruct | qwen | 0.5B | 6/12 | 50.0% | 1596.46 ms |
| qwen2.5-1.5b-instruct | qwen | 1.5B | 4/12 | 33.3% | 7502.31 ms |
| smollm2-135m-instruct | smollm2 | 135M | 0/12 | 0.0% | 1439.69 ms |

Model cost: **$0.00** (local weights, CPU). Remote calls: **0**.

## Findings

- The 0.5B model is the strongest of the three on this task set.
- The 1.5B model is slower on CPU and did not outperform the 0.5B model at this decoding budget;
  its longer answers are truncated more often. This is a measured pipeline limitation, not a
  claim about the model family.
- SmolLM2-135M is not usable for exact arithmetic at this scale.
- No model may authorize itself: every call passed through a signed single-use Gate grant.

## Blocked

OpenAI, Anthropic, Gemini and DeepSeek adapters exist (`RemoteEngine`), but no credential is
present. They are registered so the router can name them and report them **BLOCKED**; they were
never called and no credit was purchased. G13 is **BLOCKED**.

## Limits

Twelve constructed tasks are not a benchmark standing. Larger models, other families, tool use and
long-context behaviour are **not evaluated**.
