# PBFT safety incident — conflicting quorum decisions on cain-mr-02

**Date:** 2026-11-15
**Status:** **CRITICAL — open, unresolved**
**Found by:** the 48-hour soak, checkpoint 0
**Cluster:** `cain-mr-02` (4 replicas: atl, lax, mia, sjc; N=4, f=1, Q=3)
**Evidence:** `platform-gateway/frontend/proof/bundle/soak-multiregion-48h-2026-11-15c/checkpoint-0000.json`

---

## 1. What happened

At 14:40:53 UTC I started a real 48-hour soak against `cain-mr-02`:

```bash
python3 scripts/cain42_bft/multi_region_soak.py \
  --hours 48 --interval 6 --fault-every 1200 --publish-every 3600 \
  --min-free-gb 12 --record deploy/multi-region/cain_mr02_deploy_record.json \
  --name soak-multiregion-48h-2026-11-15c --work /root/cain-mr-soak-48h-2026-11-15c
```

The first signed checkpoint was published at 14:42:29 UTC, height 1872350. It was then verified offline with the clean-room `verify_soak.py`.

**Result: INVALID / FAIL.** The verifier found that two replicas report different decisions for the same sequences. I stopped the run instead of letting an invalid 48-hour record accumulate.

---

## 2. The finding

For sequence **1872339**, view **3437**, the checkpoint contains two different COMMIT quorum certificates:

| Replica reporting it | Decision hash | Parent decision hash | Signers |
|---|---|---|---|
| `cain-mr2-node-1` | `3acc9d70ac322824bfdc2db8f3a3d77088d6a7c5fbfbe1a10491b28e70d07ce8` | `48bdd004cb1209ab…` | `node-1, node-2, node-4` |
| `cain-mr2-node-4` | `7e8b44f5d81db83896528ee3e0067c1e9bf179d34ebb0039a328e71fa4f279f2` | `396c59b26b9afbc4…` | `node-2, node-3, node-4` |

Both certificates claim:

- `cluster_id = cain-mr-02`
- `view = 3437`
- `quorum = 3`, policy `{f: 1, n: 4, quorum: 3}`
- certificate type `COMMIT_QC`
- three valid Ed25519 signatures

**`node-2` and `node-4` signed both conflicting decisions.** That is equivocation by two of the four replicas — beyond the `f = 1` fault assumption the quorum is designed for.

All 12 sampled sequences (1872339–1872350) disagree between node-1 and node-4, with separate parent chains. This is not a one-block lag.

`node-2` and `node-3` could not serve the checkpoint window at all:

```
HTTP 404 committed history is not contiguous: expected sequence 45232, got 455…
```

Their local decision history is truncated/reset relative to the committed height, so they are not independent witnesses to the same history.

---

## 3. Why this matters

- The cluster has **two conflicting committed histories** at the same sequence and view.
- A quorum certificate is supposed to make that impossible. Here, both histories have a valid 3-of-4 certificate.
- Any evidence anchored on `cain-mr-02` after the fork is **ambiguous**: a verifier can find a valid quorum for either decision.
- The 48-hour soak is **not established**. No 48h claim may be published.
- Until this is fixed, `cain-mr-02` must not be described as a single Byzantine trust anchor, and should not be used to anchor customer evidence.

This is a self-found failure. It is not a third-party finding, and it is not a certification.

---

## 4. Immediate containment (done)

1. Stopped the 48-hour run at checkpoint 0. It did not continue.
2. Preserved the signed checkpoint and the failed verifier result as evidence. Do not delete or rewrite them.
3. Added a **fail-closed preflight guard** to `scripts/cain42_bft/multi_region_soak.py`:
   - it samples the last 12 sequences;
   - it refuses to start if fewer than 12 have a decision from at least two replicas;
   - it refuses to start if two replicas report different decisions.
   - Verified: the guard now returns `decision window incomplete: only 0/12 sequences…` on this cluster.
4. Stopped preflight test runs; removed their empty bundle directories. The failed 48h checkpoint is kept.

---

## 5. Root-cause questions (open)

1. **Signature binding.** Is each COMMIT signature bound to the exact `(cluster_id, view, sequence, decision_hash, parent_decision_hash)`? If not, a signature over a proposal can be re-used for a different decision.
2. **Double-signing.** Why did `node-2` and `node-4` sign two different decisions at the same sequence and view? Look for a view-change path that re-signs without checking the local commit record, or a leader that proposes two values.
3. **Quorum construction.** Does `COMMIT_QC` verify that all signers signed the **same** decision hash, or only that 3 signatures exist? The checkpoint shows a valid-looking certificate for each branch; verify the verifier itself is binding the signatures to the decision.
4. **History reset.** Why do `node-2` and `node-3` report `expected sequence 45232` while the committed height is 1,872,339? A reset QC store or a state-transfer path that does not restore the decision chain.
5. **Key/domain separation.** Confirm the four node keys and the evidence-root key are distinct, and that COMMIT signatures cannot be replayed across clusters or views.

---

## 6. Recovery plan (not executed)

Do **not** wipe or restart the cluster blindly; that would destroy the evidence of the fork.

1. Snapshot all four replicas: application state, QC store, WAL, logs, config, keys.
2. Compare the two decisions at 1872339 and the 12-sequence window. Determine which branch has the earliest valid quorum and which nodes signed both.
3. Choose the canonical history; rebuild the other branch from a known-good snapshot or state transfer.
4. Fix the commit/QC binding so a node cannot sign two decisions at the same `(view, sequence)`.
5. Add a regression test that constructs two conflicting QCs and asserts that verification refuses both (or refuses the second) — the current verifier accepted both as certificates and only flagged the disagreement at the node-comparison layer.
6. Re-run a short soak; only after the preflight passes should a 48-hour run start.

---

## 7. Honest status

- **48-hour soak: FAILED at checkpoint 0. Not established.**
- **Cluster: unsafe for trust anchoring until repaired.**
- **No third-party audit, no certification, no independent witness.**
- The finding is real, reproducible from the preserved checkpoint, and is exactly the class of failure CAIN-42 exists to detect.

The one-sentence version: **the soak did its job — it found a PBFT safety failure before a customer did.**

---

## 2b. Forensic update — 2026-11-15 17:00 UTC

The first checkpoint's two branches did **not** execute different operations:

- `COMMITTED_EXEC` payloads at 1872337–1872342 are identical (same digest, same state root, same snapshot).
- The first execution difference is at **sequence 1,069,076**: node-1 recorded `snapshot/f258975f…` at 1,069,076 and repeated it at 1,069,077; node-4 recorded `snapshot/73a6f36f…` at 1,069,076 and then `snapshot/f258975f…` at 1,069,077.
- The checkpoint QCs have the same `proposal_digest` and `state_root_claim`; their `parent_decision_hash` values differ. This is a **decision-chain/sequence-skew divergence**, not two different executed effects.
- Rebuilding the chain from the complete 1,873,411-record execution WAL matched **node-1 exactly**. Node-1 was the only non-equivocating signer with a complete chain.

Separately, the cluster produced a **valid `EQUIVOCATION_PROOF`** against `cain-mr2-node-4` at **view 3439, sequence 1874688**: two signed `PRE_PREPARE` messages from node-4 with different `request_digest`s (`cc1682…` vs `63bb2c…`) and different snapshots. That is a real primary equivocation. Node-4 was quarantined and was **not** unquarantined.

**Remediation:** the old `cain-mr-02` cluster is retired/quarantined. A clean four-node cluster `cain-mr-03` was built with new Ed25519 identities, fresh state, and a primary self-guard in `cain_pbft_engine_33._propose_locked`. The 48-hour soak is running; checkpoint 0 verified `VALID` (48 certificates). Evidence: `clawx.click/evidence/cluster-evolution-2026-11-15/`.
