# Follow-ups from the 2026-11-16 restore, fixed the same day

Two problems were flagged as open when the cluster was restored. Both are fixed here.

## 1. The soak could report a pass for a run that lost a replica

`scripts/cain42_bft/multi_region_soak.py` ended with:

```python
return 0 if counts["divergences"] == 0 else 2
```

A divergence is only computed from a *successful* check. When a replica is gone, the check raises,
`check_errors` increments, and `divergences` stays 0 — so a run with a replica down for 30 minutes
still exits 0 and is published as a 48-hour pass. The stopped 18-hour run had **179** check errors
and would have exited 0.

`check_errors` alone cannot fail a run: the fault injector kills a replica every 20 minutes by
design, and some checks fail while it restarts. The fix separates the two cases:

- **Expected** — the check failed inside the recovery grace period after an injected restart
  (`RECOVERY_GRACE_S = 240`; a restarted replica took ~140 s to answer on 2026-11-16). Counted in
  `check_errors_expected`.
- **Unexpected** — the check failed with no fault in progress. Counted in
  `unexpected_check_errors`, with a running `max_unexpected_check_streak`.

A run now **fails** (`exit 2`, `verdict.result = FAIL`) when any of these holds:

1. `divergences > 0`;
2. more than `UNEXPECTED_CHECK_STREAK_LIMIT = 5` consecutive unreadable checks with no injected
   fault (~30 s at the default 6 s interval);
3. any replica is not `HEALTHY` at the end of the run;
4. the replicas end at different heights;
5. the final cluster status cannot be read.

The verdict is computed by the pure function `verdict()` and written into the final checkpoint as
`verdict: {result, problems, unexpected_check_streak_limit, recovery_grace_s}`.
Tests: `tests/test_soak_verdict.py` (**8 passed**, `soak-verdict-tests.txt`).

## 2. A host could fill up mid-run

The historical incident (Miami, 2026-10-15) was a host filling *during* a run: `docker start`
failed 1,991 times over ~5 hours and the run kept going. `preflight()` only checks disk at the
start.

The soak now also checks disk at every publish:

- new `--min-free-gb-run` (default **5.0**);
- every checkpoint records `disk_free_gb` for all four hosts;
- if any host is below the floor, the run logs `DISK LOW`, stops at the next tick, and fails with
  the host and figure in `verdict.problems`.

## 3. The lax host was reclaimed, not the quarantined evidence

`45.76.169.191` (lax) had **9.23 GB** free. Four clean verification clones from 2026-11-14 were
removed (`e74-verify-9c850de`, `e74-verify-f0aa135`, `e75-verify-1d313be`, `e75-verify-3a8ff86`):
each was a clean checkout (`git status --porcelain --ignored` showed only `.pytest_cache/`) at a
commit that exists in `/root/cain-repo.git`, so all four are reproducible from the mirror.

| | free on `/` |
|---|---|
| before | 9.23 GB (9,674,112 KB, 88 % used) |
| after | 14.94 GB (15,671,212 KB, 80 % used) |
| reclaimed | **5.73 GB** |

**Deliberately not touched:** `/var/lib/cain42-mr02` (20.5 GB) is the quarantined `cain-mr-02`
replica — the evidence of the 2026-11-15 fork. It stays. `/var/lib/cain42-mr-backups` (4.2 GB)
and `/var/lib/cain42-mr` (4.2 GB) are the live `cain-mr-01` data and its backup; both stay.
Docker images report 0 B reclaimable. `journalctl --vacuum-size=200M` freed 0 B (already under the
cap).

Growth measured: `cain-mr-03` replica data on lax was **116 MB** after 18.5 h of running, so a
48-hour run needs a few hundred MB, not gigabytes. 14.94 GB is ample headroom.

## 4. New soak run under the fixed code

Run `soak-multiregion-48h-2026-11-16-mr03c` (started 11:48:09Z) was stopped at **0.28 h**
(133 requests, 0 check errors) because the running process still had the old verdict logic. It is
marked `SUPERSEDED-RUN.md` in its bundle. Its `checkpoint-0000` is VALID.

Run `soak-multiregion-48h-2026-11-16-mr03d` started **12:05:15Z** under the fixed code:

- preflight passed with `--min-free-gb 10 --min-free-gb-run 5`;
- `checkpoint-0000` **VALID, 48 certificates verified** (`soak-d-checkpoint-0000-verify.txt`);
- the checkpoint records `disk_free_gb` per host and the split check-error counters
  (`soak-d-checkpoint-0000.json`).

A live fault window was observed at 12:24:58Z: the injector killed and restarted `cain-mr3-node-1`,
and the run continued with **0 unexpected check errors** and a 0 streak
(`soak-live-fault-window.txt`). The expected-error counter did not need to increment on that fault
because the replica answered before a check failed.

**It is 0.01 of 48 h. This is not a passed soak.**

## What is still not proven

- The verdict function is unit-tested; the new end-of-run path itself has not yet run to completion
  (that happens at hour 48, or on the next failure).
- The disk guard has not fired in a real run.
- The `wal_write_failure` root cause on `cain-mr-01/node-4` is still unexplained.
