# Cluster restore — 2026-11-16

Two PBFT clusters were degraded. Both are back to 4 of 4 healthy replicas with quorum-valid
certificates. This bundle is the evidence of what was wrong, what was changed, and what is
still not proven.

- **Live cluster:** `cain-mr-03` (N=4, f=1, Q=3), four regions, one replica each.
- **Second cluster:** `cain-mr-01` (N=4, f=1, Q=3), also monitored and served by the gateway.
- **Retired cluster:** `cain-mr-02`, quarantined 2026-11-15 for equivocation
  (`docs/audits/PBFT-SAFETY-INCIDENT-2026-11-15.md`). It stays stopped.

Nothing here is a third-party audit or a certification. The 48-hour soak is at **0.01 of 48 h**,
not passed. Evidence class: self-observed, same-admin-domain.

---

## 1. What was found

### 1.1 `cain-mr-03` had lost a replica (root cause: the guardian revived a retired node)

The Moat #7 Sentinel (`cain-sentinel-guardian.service`) still listed the **retired, quarantined**
container `cain-mr2-node-1` as a required container. At `2026-11-16T10:54:29Z` it found that
container stopped and auto-started it.

`cain-mr-02` and `cain-mr-03` were built on the **same overlay addresses and ports**
(`10.88.0.1:29811` … `10.88.0.4:29814`). The revived `cain-mr2-node-1` took `10.88.0.1:29811`
from the live replica, and at `11:04:17Z` `cain-mr3-node-1` exited with code 3:

```
ERROR: [Errno 98] error while attempting to bind on address ('10.88.0.1', 29811): address already in use
```

The guardian had degraded the cluster it was guarding. The live cluster ran on 3 of 4 replicas
(quorum was still met, so it did not halt) while the 48-hour soak kept logging
`CHECK ERROR (state comparison skipped)`.

The retired container was revived once more by the **old** guardian at `11:35:26Z`, seconds after it
was stopped and about a minute before the guardian was restarted with the fix. It could not bind and
died at `11:48:06Z`. It has not been started since; the retired-container guard is exercised
directly in `post-fix/retired-guard-proof.txt`.

### 1.2 `cain-mr-01` was stalled and committing without quorum

A separate, more serious fault. Replica heights were frozen at different values:

| Replica | commit_index | health |
|---|---|---|
| node-1 | 143271 | HEALTHY |
| node-2 | 143230 | HEALTHY |
| node-3 | 143230 | HEALTHY |
| node-4 | 143035 | **SAFE_MODE** |

`node-4` carried `wal_write_failure: "sequence 143035: OperationalError: disk I/O error"` and had
been in SAFE_MODE since that write, with no restart since 2026-11-03.

A write sent to `node-1` returned:

```json
{"status": "COMMITTED", "decision": "ALLOW",
 "quorum_certificate_status": "INVALID: INSUFFICIENT_QUORUM",
 "quorum_certificate_accepted_signers": 1, "sequence": 143273}
```

So the node reported **COMMITTED** while only **one** replica had signed, below the quorum of 3.
The local commit is real (the height advanced); the outcome was not quorum-certified. The response
made that visible only in an adjacent string field — a caller reading `status` alone would treat an
uncertified outcome as final. That is the fail-open reporting defect recorded in section 3.2.

After the repair the same write returns `"quorum_certificate_status": "VALID"` with
**4 accepted signers** (`mr01/post-fix-write-probes.txt`).

---

## 2. What was done

| # | Change | File |
|---|---|---|
| 1 | Stopped the invalid soak run | `systemctl stop cain-mr03-soak-48h-2026-11-15b` |
| 2 | Stopped the revived retired container (container and data preserved) | `docker stop cain-mr2-node-1` |
| 3 | Required container `cain-mr2-node-1` → `cain-mr3-node-1` | `deploy/scripts/cain_sentinel_guardian.py` |
| 4 | The guardian now refuses to start any retired `cain-mr2-node-*` container | same |
| 5 | Cluster monitor `cain-mr-02` → `cain-mr-03` | same |
| 6 | Gateway knows `cain-mr-03` | `platform-gateway/routers/live_cluster.py` |
| 7 | Gateway refuses to count a replica whose reported `node_id`/`cluster_id` does not match the requested cluster | same |
| 8 | `outcome_certified` / `outcome_final` added to the commit response **(staged; not live until the node image is rebuilt)** | `platform-gateway/cain_pbft_engine_33.py` |
| 9 | Restarted `cain-mr3-node-1`; it caught up from its peers | `docker start cain-mr3-node-1` |
| 10 | Restarted the `SAFE_MODE` replica `cain-mr-01/node-4`; it caught up 143035 → 143275 | `docker restart cain-mr-node-4` |
| 11 | Started a new, clean 48-hour soak with a new name | `cain-mr03-soak-48h-2026-11-16c` |

Change 7 matters for honesty: before it, `GET /api/v1/live-cluster/health?cluster=cain-mr-02`
reported **3 of 4 replicas reachable at height 11468** for a cluster that no longer exists — it was
counting `cain-mr-03`'s replicas, which answer on the same overlay addresses. After the change the
retired cluster reports `HALTED, 0/4` and the live cluster reports `HEALTHY, 4/4`.

---

## 3. Verification after the repair

### 3.1 Live cluster `cain-mr-03`

All four replicas: view 59→, identical `commit_index`, identical `application_state_hash`,
`health_state: HEALTHY`. A write returns `VALID` with 4 accepted signers.

### 3.2 Second cluster `cain-mr-01`

All four replicas at commit 143275 and advancing; three consecutive writes return `VALID` with
4 accepted signers. Host free disk at repair time: atl 241 GB, lax 9.2 GB, mia 26 GB, sjc 62 GB.

### 3.3 Soak

New run `soak-multiregion-48h-2026-11-16-mr03c` started `2026-11-16T11:48:09Z`. Its preflight
passed (all four containers running, all HEALTHY at one height and state, 12-sequence decision
window readable from at least two replicas with no disagreement). `checkpoint-0000.json` is
**VALID, 48 certificates verified**; verdict `RUNNING (0.009 of 48.0 h)`.

The previous run `soak-multiregion-48h-2026-11-15-mr03b` is marked **INVALID** in its own bundle
(`INVALID-RUN.md`): it is 18 h of a degraded window, not a 48-hour run, and must not be cited.

---

## 4. Follow-ups (fixed the same day)

Two open items from this restore were fixed on 2026-11-16 and are recorded in `followups/`:

1. **The soak verdict is no longer `divergences == 0` alone.** It now fails on a sustained streak of
   unreadable checks with no injected fault, on a cluster that is not healthy at one height when the
   run ends, and on a host below the new `--min-free-gb-run` floor. `verdict()` is a pure function;
   `tests/test_soak_verdict.py` is 8/8.
2. **The lax host was reclaimed from 9.23 GB to 14.94 GB** by removing four clean, reproducible
   verification clones. The quarantined `cain-mr-02` data (20.5 GB) was **not** touched.

The run that enforces the new verdict is `soak-multiregion-48h-2026-11-16-mr03d`
(started 12:05:15Z, `checkpoint-0000` VALID). It is 0.01 of 48 h.

---

## 5. What this does not prove

- Not a third-party audit, not a certification, not a SOC 2 or ISO claim.
- The 48-hour soak has **not** passed. It is 0.01 h in and will restart from zero if it is
  interrupted. Do not publish a 48h or 72h claim from this bundle.
- The `wal_write_failure` on `cain-mr-01/node-4` was cleared by a restart. **The underlying I/O
  error was not root-caused.** Miami has 26 GB free now; the error is consistent with the host
  having been full earlier (the same host filled up once before), but that is not established here.
- The `outcome_certified` / `outcome_final` fields are in the host source only. The running node
  image (`cain-cluster-node:mr-214970f`) still has the old response shape until it is rebuilt and
  redeployed. The invalid-quorum state is documented, not prevented.
- `cain-mr-01` and `cain-mr-03` are both operated from this same administrative domain. They are
  not independent witnesses of each other.
- `cain-mr-02` remains quarantined. Its evidence is untouched; it was not repaired and must not be
  used as a trust anchor.

---

## 6. Reproduce this check

```bash
python3 verify_restore_2026_11_16.py            # verifies MANIFEST.json and the recorded facts
curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-03'
curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-01'
systemctl is-active cain-mr03-soak-48h-2026-11-16c.service
python3 /root/cain-studio-platform/scripts/cain42_bft/verify_soak.py \
  /root/cain-studio-platform/platform-gateway/frontend/proof/bundle/soak-multiregion-48h-2026-11-16-mr03c
```

## 7. Files

| File | Contents |
|---|---|
| `RESTORE.json` | machine-readable summary: findings, changes, before/after states |
| `pre-fix/` | docker state, exit log, sentinel auto-heal log, soak tail, per-replica status **before** |
| `mr01/` | `cain-mr-01` before-fix status, the invalid-quorum response, after-fix write probes |
| `followups/` | soak-verdict fix and tests, disk reclaim before/after, the new run's checkpoint-0000 |
| `post-fix/` | per-replica status, gateway health for both clusters, sentinel state, soak verification |
| `RECOVERY-COMMANDS.txt` | the exact commands run, in order |
| `verify_restore_2026_11_16.py` | standalone verifier, imports no CAIN code |
| `MANIFEST.json` | SHA-256 of every file in this bundle |
