# E93 Procedure-Synthesis Experiment — Pre-registration

Written 2026-11-18. This file is committed together with the language, the search and the supplied
library, before any E93 task-family code exists. The commit hash is the timestamp of record.
Author caveat: the same session wrote the search, the library and (next) the task families. Task
authors were not separated from implementers. Commit order is the mitigation; it is not independence.

## Question
Can a family-agnostic program search generate procedures, not supplied by designers, that beat
(a) a governed agent with no procedures (D) and (b) the best fixed strategy (F) on held-out tasks,
and does it beat selection from a designer-supplied repertoire (G)?

## Arms (identical per-task tool budget, allow-list and task information on held-out tasks)
- **D**: `library.d_policy`, one fixed generic action rule, no procedures, no model.
- **F**: one template from `library.TEMPLATES`, the single best by pooled development accuracy across
  all families, frozen before held-out evaluation; first instantiation; no per-family choice.
- **G**: per family, the best instantiation of the supplied repertoire (E92's 4 templates + 4
  generic) chosen on development tasks. This is selection.
- **G2**: `search.synthesize` (bottom-up typed enumeration + CEGIS) on the development pool, then
  fresh validation, then a disjoint-seed promotion check, then approval by a distinct approver
  identity. If no procedure is promoted for a family, G2 falls back to D for that family (counted).
- **G2-matched** (secondary): G2 with its development-time tool calls capped at G's
  development-time tool calls for the same family.
- **Model-proposed** (secondary source): bounded local-model proposals (Qwen2.5-1.5B-Instruct, CPU),
  parsed and run through the same validation. Reported separately; not part of G2's primary result.

## Families
1. The three E92 families (reconcile, noisy_select, staged_repair), method hints removed from prompts,
   checker made state-aware (staged_repair must leave the system repaired; E92's output-only checker
   accepted a constant `True`).
2. **12 generated families** from `families.generate(seed=9301)`: random tool tables per task and a
   hidden random target expression of size 5–9 over the language; trivial targets (constant, or equal
   on samples to any expression of size ≤ 3) are rejected by the generator.
3. **Two out-of-language families** (median of five reads; sum of the first p reads, with p varying):
   expected to defeat bounded search. They stay in the pooled result.
Development of the search and harness uses only generator seed **9300**. Seed 9301 is run once.
Replication: **new families** from seed **9302**, plus the five hand families on a new task seed.

## Splits per family
dev 20 + counterexample pool 40 (search only) · validation 10 · promotion check 20 (disjoint seed) ·
**held-out 30** (disjoint seed, never given to any oracle). Held-out tasks: 17 × 30 = 510.

## Outcomes and analysis
- Primary: pooled held-out acceptance. Tests: exact two-sided McNemar, G2 vs D and G2 vs F;
  Holm correction over {G2-D, G2-F, G2-G}; α = 0.05. Effect: difference in proportions with
  Newcombe-paired 95% CI; Wilson CI per arm.
- Gate 4 (beats selection): G2 vs G, same test.
- Gate 5 (beats baselines): G2 > D **and** G2 > F, both significant after Holm, **and** the
  same direction significant in the seed-9302 replication.
- Secondary: per-family table, G2-matched result, number of procedures promoted and their novelty
  (not syntactically equal to any library instantiation), development-time tool calls and CPU seconds
  per arm, model-proposal parse/validity rates.
- Power: with n = 510 pairs, an exact McNemar test at α = 0.0167 has > 0.95 power for a
  discordance of ≥ 10% split 75/25. The sample size is fixed here and is not increased afterwards.

## Exclusions and stopping
No task or family is excluded after the run. Search limits are fixed in `search.Limits` defaults
(max size 11, 40 s per round, 150 s per family, 12 CEGIS rounds). One run per seed; no reruns to
improve a result. If the code crashes, it is fixed and the whole seed is rerun, and the crash is recorded.

## Kill criterion
If G2 does not beat D on held-out tasks under this protocol: **PROCEDURE GENERATION NOT DEMONSTRATED.**
