# CAIN-42 cluster evolution — 2026-11-15

**Status:** the old cluster `cain-mr-02` is quarantined after a **valid equivocation proof** against node-4. A clean four-node cluster `cain-mr-03` is running with new identities, fresh state, and the missing primary self-guard. The 48-hour soak is **RUNNING**, not passed.

## What was found

The first 48-hour soak attempt on `cain-mr-02` failed at **checkpoint 0**.

1. The application execution records on the two branches were **identical** (same digest, same state root, same snapshot).
2. The decision chains diverged at sequence **1,069,076**: node-1 had `snapshot/f258975f…`; node-4 had an extra `snapshot/73a6f36f…` and then repeated node-1's operation at 1,069,077.
3. Rebuilding the decision chain from the complete 1,873,411-record execution WAL matched **node-1 exactly**; node-1 was the only non-equivocating signer with a complete chain.
4. The cluster then produced a **valid `EQUIVOCATION_PROOF`** against `cain-mr2-node-4` at **view 3439, sequence 1874688**:
   - two signed `PRE_PREPARE` messages from node-4,
   - different `request_digest`s (`cc1682…` vs `63bb2c…`),
   - different snapshots (`286a3b…`, capability epoch 1 vs `2b81a2…`, epoch 2),
   - same view and sequence.
5. Receivers quarantined node-4. It was **not** unquarantined. A proven equivocator cannot be repaired by restoring data; its identity must be replaced.

## What was changed

- `platform-gateway/cain_pbft_engine_33.py`: the primary now refuses to sign a second, different `PRE_PREPARE` for the same `(view, sequence)`. It re-drives the same signed proposal for the same digest.
- `scripts/cain42_bft/multi_region_soak.py`:
  - the preflight reads the QC endpoint correctly (the endpoint returns the certificate directly, not under `commit_qc`);
  - the decision-window guard requires `min(12, height)` readable decisions, not always 12;
  - the SJC SSH banner is stripped;
  - preflight waits for a common height instead of failing on an in-flight commit.
- 63 PBFT tests pass, including the new `tests/test_pbft_leader_equivocation.py`.

## The new cluster

- **Cluster:** `cain-mr-03`, four nodes in four regions.
- **New identities:** four new Ed25519 keys; the old node-4 key is excluded.
- **Fresh state:** new data directories; no inherited divergent WAL.
- **Patched engine:** the primary guard is copied into all four containers.
- **Health at publication:** 4/4 `HEALTHY`, same commit index (224), same application state hash, zero quarantines.

## The 48-hour soak

- **Started:** 2026-11-15 17:27:44 UTC.
- **Planned end:** 2026-11-17 17:27:44 UTC.
- **Checkpoint 0:** **VALID**, 48 certificates verified, verdict `RUNNING`.
- **Live counters at publication:** 147 requests, 146 ALLOW, 1 DENY, 0 divergences, 0 faults, 0 view changes.
- **Honest status:** running. This page will only say PASS after the final checkpoint at 48 hours verifies.

## Evidence

- Soak bundle: https://clawx.click/evidence/soak-multiregion-48h-2026-11-15-mr03b/
- Checkpoint 0: https://clawx.click/evidence/soak-multiregion-48h-2026-11-15-mr03b/checkpoint-0000.json
- Equivocation proof: `EQUIVOCATION-PROOF.json`
- Incident report: `INCIDENT.md`
- Snapshot manifest: `SNAPSHOT-MANIFEST.json`
- Cluster status: `CLUSTER-STATUS.json`
- Soak status: `SOAK-STATUS.json`
- Code changes: `CHANGES.md`
- Test results: `TEST-RESULTS.txt`

## What this does not prove

- No third-party audit, SOC 2, or certification.
- The old cluster's historical evidence is preserved, not repaired.
- The 48-hour soak is not complete.
- The quorum is vendor-operated; this is not an independent witness network.

## Prior 24-hour run

The 2026-11-02 24-hour run is independently verifiable and returns **PASS**:

- 25 checkpoints (0–24), 24.00 h elapsed.
- 67 fault/restart cycles, 0 divergences.
- 48 quorum certificates verified per checkpoint.
- Bundle: https://clawx.click/evidence/soak-multiregion-24h-2026-11-02/
- Local verifier output: `LEGACY-24H-PASS.txt`

The 48-hour run on `cain-mr-03` is the successor. At publication it has passed checkpoint 0, survived its first injected kill (node-4, 60 s down, restarted HEALTHY), and is still `RUNNING`.
