# Evolution 80 — Research Review and Research-to-Implementation Matrix

Reviewed 2026-11-15. Sources were found by web search; abstracts and summaries were read, **full
papers were not**. Every number below is quoted from an abstract or a secondary summary and is
marked accordingly. Nothing here is a CAIN result. No framework was imported.

## 1. Papers reviewed

| # | Work | Authors · date | Dataset / method | Reported result (source level) | Known limits | Compute / license | Reproduced here? |
|---|---|---|---|---|---|---|---|
| 1 | **MultiAgentBench / MARBLE** — arXiv 2503.01935 | Zhu, You et al. (UIUC) · 2025-03 | Collaboration and competition scenarios; star, chain, tree and graph topologies; milestone KPIs | Graph topology best in the research scenario; cognitive planning +3% milestone rate; gpt-4o-mini highest average score (abstract) | Needs API-grade models; tables not read | Cloud LLMs; code public on GitHub | No — NOT EVALUATED |
| 2 | **Why Do Multi-Agent LLM Systems Fail? (MAST)** — arXiv 2503.13657 | Cemri et al. (UC Berkeley) · 2025-03, NeurIPS 2025 | 7 frameworks, 200+ tasks, 6 annotators | 14 failure modes in 3 groups (specification, inter-agent misalignment, task verification); κ = 0.88; gains over single agents "often minimal" (abstract) | Taxonomy, not a fix | Annotation study; dataset public | Used as the lens for E80's failure analysis |
| 3 | **Towards a Science of Scaling Agent Systems** — arXiv 2512.08296 | Kim et al. (Google Research, MIT) · 2025-12 | 180 configurations, 5 architectures, 3 model families, 4 benchmarks | Multi-agent helps parallelizable tasks; **all multi-agent variants degraded sequential reasoning by 39–70%**; independent agents amplify errors 17.2×, centralized 4.4×; a predictor picks the best strategy for 87% of unseen configurations (abstract / Google blog) | Per-task numbers not read | Cloud LLMs | Direction tested by E80's ten-architecture comparison |
| 4 | **MemoryAgentBench** — arXiv 2507.05257 | Hu, Wang, McAuley (UCSD) · 2025-07 | Four memory competencies incl. test-time learning and conflict resolution | Benchmark paper (abstract) | Long contexts | Dataset public | NOT EVALUATED |
| 5 | **LongMemEval** | Wu et al. · 2024–2025 | Long synthetic multi-session chat; five abilities (extraction, multi-session, knowledge update, temporal, abstention) | Benchmark paper | Contexts far beyond a 0.5B CPU model's useful range | Dataset public | NOT EVALUATED — its five-ability structure is copied in E80's constructed set |
| 6 | **LoCoMo** | Maharana et al. · 2024 | Long conversations (~9K tokens average, per a secondary source) | Benchmark paper | — | Dataset public | NOT EVALUATED in E80 (E79 ran a 24-item sample: exact match 0.0) |
| 7 | **AMA-Bench** — arXiv 2602.22769 | 2026-02 | Agent–environment trajectories | Authors' AMA-Agent 57.22% average, +11.16 over the strongest baseline (abstract) | — | — | NOT EVALUATED |
| 8 | **FrugalGPT** — arXiv 2305.05176 | Chen, Zaharia, Zou (Stanford) · 2023-05 | LLM cascade with a learned stopping rule | "Up to 98% cost reduction" matching GPT-4, or +4% accuracy at equal cost (abstract; "up to") | Single-turn QA; dataset-dependent | API models | Idea used: cheapest path that meets a quality bar |
| 9 | **RouteLLM** — arXiv 2406.18665 | Ong et al. (Berkeley / LMSYS) · 2024, ICLR 2025 | Learned router between a strong and a weak model | ≈85% cost reduction on MT-Bench at ≈95% of GPT-4 quality (secondary source) | Two-model routing; benchmark-specific | Open source | Idea used: route on features of the query |
| 10 | **Small Language Models are the Future of Agentic AI** — arXiv 2506.02153 | Belcak et al. (NVIDIA) · 2025-06 | Position paper | Argues SLMs are sufficient and cheaper for many agent calls; heterogeneous systems are the natural design | A position, not a measurement | — | Motivates measuring small models honestly instead of assuming |
| 11 | **Program of Thoughts** — arXiv 2211.12588 | Chen, Ma, Wang, Cohen · 2022-11, TMLR 2023 | Model writes a program; an interpreter computes | ≈12% average gain over chain-of-thought on numeric datasets (abstract) | Codex-scale models | — | Implemented as the `*_pot` paths |

Not found by the searches run for this review, and therefore not cited for results: the
multi-agent debate paper (Du et al., 2023) and the self-consistency paper (Wang et al., 2022).
Both techniques are implemented in E80 as arms (`c03`, `c02`) on the strength of being standard
methods, without quoting their numbers.

## 2. Engineering systems reviewed

| System | What it is | Decision for E80 |
|---|---|---|
| Hugging Face Transformers | Model loading and generation | **Used** (already the E79 runtime) |
| PEFT / LoRA / QLoRA / TRL | Parameter-efficient fine-tuning | `peft` is not installed. LoRA is ~40 lines; **implemented by hand** in `skill80.py` to avoid installing a package on a production host. QLoRA needs GPU bitsandbytes: NOT EVALUATED |
| llama.cpp / Ollama | Quantized CPU inference | An Ollama 0.12.3 container (`cain-cat-ollama`) exists with `qwen2.5-coder:3b` Q4_K_M. It is a production catalog dependency capped at 2 CPU / 4 GiB; four probe calls took 10–40 s each. **Not used for the benchmark** without owner approval |
| vLLM | GPU serving | No GPU: BLOCKED |
| LangGraph, Microsoft Agent Framework, OpenAI Agents SDK | Agent orchestration | Not imported. E80's coordination is ~250 lines over its own gate so every call is authorized and logged |
| MCP, A2A | Tool and agent protocols | Existing CAIN surfaces unchanged; not extended in E80 |
| Temporal | Durable workflow engine | Not installed. E80 uses an SQLite queue with leases, fencing and an idempotent ledger and tests it with process kills |
| bubblewrap | Namespace sandbox | **Used** for every code execution |
| gVisor, Kata, Firecracker, Kubernetes Agent Sandbox | Stronger isolation | Not installed: NOT EVALUATED |

## 3. Research-to-implementation matrix

| Finding | Source | What E80 does | Where |
|---|---|---|---|
| Multi-agent gains are often minimal and failures are structural | MAST (#2) | Ten architectures on identical tasks, with calls and tokens counted per arm; "single agent is better" is an allowed conclusion | `bench80.py`, `COLLABORATION_BENCHMARK.md` |
| Coordination hurts sequential reasoning; independent agents amplify errors | Scaling Agent Systems (#3) | Agents talk only on disagreement (debate), discussion is bounded to one round, and a controller can decline to collaborate | `c03`, `c10` |
| Pick the cheapest model that meets the quality bar | FrugalGPT, RouteLLM (#8, #9) | Per-family routing policy fitted on train, frozen and hashed before test | `fit_policy`, `routing_policy.json` |
| Let an interpreter do the arithmetic | Program of Thoughts (#11) | `*_pot` paths, applied only where there is something to compute | `Bench.pot` |
| Small models suffice for narrow calls | SLM position (#10) | A 135M model gets a LoRA skill for one family; measured against the untouched base | `skill80.py` |
| Memory needs update, temporal and abstention abilities, not just recall | LongMemEval (#5), MemoryAgentBench (#4) | Constructed five-ability set; retrieval hit-rate reported separately from answer accuracy | `memory80.py` |

## 4. What this review does not establish

It does not show that CAIN matches any published system. No public benchmark was run in
Evolution 80. The comparison that matters here is internal: Evolution 80 against Evolution 79 on
unseen instances of the same task families.

## Sources

- https://arxiv.org/abs/2503.01935 · https://huggingface.co/papers/2503.01935
- https://arxiv.org/abs/2503.13657
- https://arxiv.org/abs/2512.08296 · https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/
- https://arxiv.org/abs/2507.05257
- https://arxiv.org/abs/2602.22769
- https://arxiv.org/abs/2305.05176
- https://sky.cs.berkeley.edu/project/routellm/
- https://arxiv.org/abs/2506.02153
- https://arxiv.org/abs/2211.12588
