# cain-mr-03 view-184 orphan — 2026-11-18

**Evidence class:** `SELF_OBSERVED_SAME_ADMIN_DOMAIN` — this is not a third-party audit or a
certification. The 48-hour soak is **RUNNING, not passed**.

This bundle records one operational incident on the CAIN-42 four-replica PBFT cluster
`cain-mr-03`, the fix, the rolling deployment, and the start of a fresh 48-hour soak.

## What happened

The homepage status bar (pinned to `cain-mr-03`) reported **DEGRADED** from about 00:45Z. All four
replicas were reachable and held the same application state, but one replica was in view 184 while
the other three committed in view 183.

Root cause chain:

1. The cluster autoheal's PBFT-alive guard only read the mr-01 and mr-02 deploy records. It could
   not recognise a `cain-mr3` node, so `pbft_alive("cain-mr3-node-1")` always returned None.
2. At **00:06:27Z** the autoheal restarted `cain-mr3-node-1` on Docker health alone; the sentinel
   started the same container again four seconds later.
3. On recovery the replica installed **view 184** while the other three stayed in **183**. It
   re-broadcast a NEW_VIEW covering sequence **132613** while the cluster had committed **143k+**,
   so the peers could never install it. It stayed orphaned and lagged.
4. Safety held: identical `application_state_hash` on all four, no fork, no quarantine, quorum
   3/4 kept committing. The defect was liveness/view churn, not a safety breach.

## What was fixed

| Fix | File | Status |
|---|---|---|
| Autoheal reads every `cain_mr*_deploy_record.json`, so `pbft_alive()` protects mr-3 (and future clusters) | `scripts/ops/cain_mr_autoheal.py` | live on atl, lax, mia, sjc |
| Sentinel never starts a container Docker reports `restarting`/`created`/`paused` or health `starting` | `deploy/scripts/cain_sentinel_guardian.py` | live |
| PBFT engine steps down from an unadopted stale view instead of re-broadcasting it forever | `platform-gateway/cain_pbft_engine_33.py` | live in image `mr-2478e9c` |
| Deploy tool can roll `cain-mr-03` (`mr03` topology) | `scripts/cain42_bft/deploy_multi_region.py` | committed |
| Tests: autoheal record coverage/mr-3 recognition; stale-NEW_VIEW step-down | `tests/` | 12/12, 46/46, 345/345 |

The engine fix was deployed as hotfix commit `2478e9cbf` (parent `e8df8998f`), one file,
`+30/−6`. The image `cain-cluster-node:mr-2478e9c` was derived from the running `mr-e8df8998f`
image plus that one patched file. **The image is a commit derivative, not a clean reproducible
rebuild**; a verifier can inspect `/app/cain_pbft_engine_33.py` but cannot rebuild the image from
the commit alone.

## Timeline

See [`timeline.md`](timeline.md). Key points:

- **03:23Z** node-1 restarted and rejoined view 183; **03:41Z** 4/4 HEALTHY.
- **04:31–04:40Z** rolling deploy of `mr-2478e9c` (node-1 139 s, node-3 43 s, node-4 32 s,
  primary node-2 last 42 s); 4/4 HEALTHY at view 190.
- **04:44Z** soak `soak-multiregion-48h-2026-11-18` started; checkpoint-0000 **VALID** (48
  certificates verified).
- **05:03:37Z** first injected fault: node-1 killed, restarted 05:04:34Z; **5 expected check
  errors, 0 divergences**; cluster returned to HEALTHY. A later view-change window
  (05:09–05:10Z) produced **4 consecutive ReadTimeout check errors** (max streak 4, limit 5);
  the cluster returned to HEALTHY at view 207. See `status/post-first-fault-observation.md`.

## Verify this bundle

```bash
python3 verify_bundle.py          # checks every file against MANIFEST.json
```

The soak evidence is also published separately and continuously:

- https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/
- https://cainstudio.online/proof/bundle/soak-multiregion-48h-2026-11-18/

```bash
curl -so verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/verify_soak.py.txt
python3 verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/
```

The published verifier prints `VERDICT: RUNNING` until the final 48-hour checkpoint lands. A
`RUNNING` verdict is **not** a passed soak.

## Files

| Path | Contents |
|---|---|
| `incident.md` | Full incident/fix/deploy/soak note |
| `runbook.md` | The exact deploy and soak runbook used |
| `timeline.md` | Dated timeline from the recovery, deploy and soak logs |
| `tests.md` | Test commands and results |
| `deployment.json` | Deploy record (commit, image, image id, nodes) |
| `status/` | Pre-recovery, post-recovery, pre-deploy, post-deploy, post-first-fault status |
| `logs/` | node-1 pre-restart status/log, WAL rows, recovery actions, rolling deploy log, stabilization (renice/ionice) |
| `soak/` | Snapshot of checkpoint-0000, heartbeat, index, membership, verifier, first-fault log |

## Honest limits

- No third-party audit, no certification, no independent witness.
- The previous 48-hour attempt stopped at 1.636 h on 2026-11-16; no 48-hour run has completed.
- This run is **RUNNING**; the pass decision comes only from the final signed checkpoint and the
  published verifier (~2026-11-20T04:43Z).
- The cluster experienced view churn after the deploy (host load ~38 on 8 CPUs); the monitor
  caught a transient DEGRADED at 05:00Z, and the cluster returned to HEALTHY at view 207. Four
  consecutive ReadTimeout check errors at 05:09–05:10Z (max streak 4, limit 5) are disclosed, not
  hidden; view changes are not a pass criterion by themselves.
- The autoheal can shorten a deliberately killed replica; the first fault lasted ~57 s and the
  soak counted the five expected check errors. If strict 60 s fault windows are required, pause
  the autoheal during a soak or add a soak marker.
