# cain-mr-03 view-184 orphan — homepage DEGRADED

**Date:** 2026-11-18 (UTC)
**Cluster:** `cain-mr-03` (N=4, f=1, Q=3; atl, lax, mia, sjc)
**Symptom:** the homepage status bar (pinned to `cain-mr-03`) showed **DEGRADED**, while
`GET /api/v1/live-cluster/health?cluster=cain-mr-03` returned:

```
DEGRADED — "replicas at different heights (catching up)", "replicas in different views"
4/4 reachable, same application_state_hash
```

The homepage was telling the truth. The different widget on `now.html` / `verify.html` /
`cainstudio-homepage/index.html` fetches the API default (`cain-mr-01`) and said HEALTHY; the bar
tracks `cain-mr-03`.

## Root cause

1. **00:06:27Z** `cain-mr-autoheal.service` restarted `cain-mr3-node-1` with
   `status=running health=unhealthy`.
   The autoheal is supposed to leave a replica alone when `pbft_alive()` can show it is answering
   PBFT and keeping up. `RECORDS` listed only `cain_mr_deploy_record.json` (mr-01) and
   `cain_mr02_deploy_record.json` (mr-02), so **`pbft_alive("cain-mr3-node-1")` always returned
   `None`** and the guard never applied to cain-mr-03.
2. **00:06:56Z** `cain-sentinel-guardian.service` started the same container again, four seconds
   after the autoheal had started it, because `docker ps` briefly did not show it.
3. On recovery node-1 installed **view 184** while nodes 2/3/4 stayed in **view 183** with node-4
   as primary. Node-1 then rejected the live PRE_PREPAREs:
   `PRE-PREPARE rejected: stale view 183 < current 184`.
4. `anti_entropy_tick` re-broadcast node-1's stored NEW_VIEW, but that NEW_VIEW covers sequence
   **132613** while the cluster had committed **143k+**. Its proposals were already below the
   commit point, so peers could never install it. Node-1 stayed orphaned for hours, trailing by
   8–20 commits. The health monitor logged `VIEW_SPREAD: views [183, 184]` every 15 minutes from
   00:45Z.

## Safety assessment

No fork. All four replicas reported the same `application_state_hash`, the decision chain was
`trusted`, there were no quarantines, and the 3-replica quorum kept committing. This was a
liveness/view anomaly, not a safety failure. It matters because the view spread alone keeps the
classifier at DEGRADED even when heights converge.

## Recovery (performed)

1. Captured node-1 status, logs, WAL rows and container state under
   `/tmp/opencode/mr03-recovery-20261118T031700Z/`.
2. Paused `cain-mr-autoheal.timer` and `cain-sentinel-guardian.service` so neither could race the
   restart.
3. Verified the latest checkpoint had advanced to **sequence 143680/view 183**, past the stale
   view-184 `NEW_VIEW` row (seq 132613), so recovery would land on view 183.
4. `docker restart -t 60 cain-mr3-node-1`. Quorum stayed at 3/4 (nodes 2, 3, 4) throughout.
5. Node-1 recovered to **view 183**, replayed, and caught up from its peers.
6. Resumed the autoheal timer and the sentinel; both report `ok` for node-1.

Verified at **2026-11-18T03:41:31Z**: 4/4 in lockstep, view 183, height 144846, identical state
hash, `/health` = **HEALTHY**, node-1 `restarts=0`.

## Fixes

| # | File | Change | Status |
|---|---|---|---|
| 1 | `scripts/ops/cain_mr_autoheal.py` | `RECORDS` now globs **all** `cain_mr*_deploy_record.json`, so `pbft_alive()` recognizes every live cluster. Deployed to atl, lax, mia and sjc (old copy backed up as `*.bak-20261118`). | **live** |
| 2 | `deploy/scripts/cain_sentinel_guardian.py` | `_docker_state()` guard: never start a container that Docker reports `restarting` / `created` / `paused`, or whose health is `starting`. Closes the 00:06:56 double-start race. Sentinel restarted to load it. | **live** |
| 3 | `platform-gateway/cain_pbft_engine_33.py` | `anti_entropy_tick`: if the stored NEW_VIEW's proposals are below `self.commit_index`, stop re-broadcasting it and **step down** to the cluster's view. f+1 peers in a lower view prove no decision was committed in the higher view (a commit needs 2f+1 replicas at that view), so no committed history is abandoned. | **live** — `mr-2478e9c`, rolled 2026-11-18 04:40Z |

Regression tests:

- `tests/test_cain_mr_autoheal.py` — **12/12** (adds record-coverage and mr-3 recognition tests).
- `tests/test_cain42_pbft_view_divergence.py` + anti-entropy + evolution1 — **46/46** (adds the
  stale-NEW_VIEW step-down test).
- All `tests/test_cain42_pbft*.py` + `tests/test_live_cluster_health.py` — **345/345**.

## Deployment of the engine fix (2026-11-18 04:31–04:41Z)

- Hotfix commit `2478e9cbf32e69d48f750270b9bdab47dd7ecfdc` (`hotfix/mr03-view184`), parent
  `e8df8998f`; diff = **one file**, `platform-gateway/cain_pbft_engine_33.py` (+30/−6).
- Why a minimal commit: branch HEAD was 45 commits / 1,660 files ahead of the deployed image, so a
  normal HEAD build would have shipped unrelated code to consensus.
- Image `cain-cluster-node:mr-2478e9c` was derived from the running `mr-e8df8998f` image plus the
  one patched file (`docker create` + `docker cp` + `docker commit`). `/app/main.py` and
  `requirements.cluster.lock` hashes are identical to the running image; the engine hash matches the
  hotfix tree (`9ee8607c…`); `import cain_pbft_engine_33` succeeds inside the image.
- Rolling upgrade with `deploy_multi_region.py --topology mr03 --rolling`: node-1 139 s, node-3
  43 s, node-4 32 s, primary node-2 last 42 s. Old containers are kept as
  `cain-mr3-node-N-was-mr-e8df8998f-20261118T043614Z` (and later stamps) for rollback.
- Verified at **04:41Z**: 4/4 on `mr-2478e9c`, view 190, height 147937, identical state,
  `/health` HEALTHY; deploy record `commit=2478e9c…`, `upgraded_at=2026-11-18T04:40:33Z`.
- **Provenance caveat:** the deployed image is a commit derivative, not a clean reproducible
  rebuild. A verifier can inspect `/app/cain_pbft_engine_33.py`, but cannot rebuild the image from
  the commit alone.

## 48-hour soak (started 2026-11-18 04:44:01Z)

- Run `soak-multiregion-48h-2026-11-18`, id `1794977002-b4fc4e82`; systemd unit
  `cain-mr03-soak-48h-2026-11-18.service` (enabled, `Restart=on-failure`).
- Cluster `cain-mr-03`; 48 h, write every 6 s, fault every 1200 s, checkpoint every 3600 s,
  min free 12 GB at start / 5 GB during.
- Checkpoint-0000 published at height 148069; **VALID** (48 certificates verified;
  `verify_soak.py`: `VERDICT: RUNNING (0.027 of 48.0 h, all checkpoints valid so far)`).
- Published at `cainstudio.online/proof/bundle/soak-multiregion-48h-2026-11-18/` and
  `clawx.click/evidence/soak-multiregion-48h-2026-11-18/`.
- The live homepage soak card now points at this bundle (it previously showed the completed 24 h
  cain-mr-01 run), and `cain-mr03-soak-48h-watch.timer` re-runs the published verifier hourly
  (first run: `VERDICT: RUNNING`, all checkpoints valid, exit 0).
- Expected final checkpoint ~**2026-11-20T04:43Z**. Until it lands and verifies, the 48 h soak is
  **NOT PASSED**.

## Limits / what is not fixed

- The engine step-down is live, but it mitigates rather than eliminates the startup pacemaker that
  can put a replica one view ahead. A crash/reboot can still create a split view; the fix makes the
  replica recover instead of staying orphaned.
- The autoheal fix prevents the **trigger** (a needless restart on Docker health alone), not every
  possible view split.
- The sentinel guard has no unit-test harness; it is covered by syntax check and the live restart.
- Same-admin-domain, self-observed. No third-party audit or certification is claimed.
- The 48-hour soak is **RUNNING**, not passed. The previous attempt stopped at 1.636 h on
  2026-11-16; no 48 h run has ever completed.
