# E72.1 — Cross-host shared state, failover and reboot recovery

## What was added

`cain_workforce/e72/crosshost.py` is a PostgreSQL-backed shared mission state for two physically separate
development hosts. It extends CAIN-42 without adding a new authority path: every dispatched operation carries an
Ed25519-signed grant (`cain66.child.grant.v1`) and a single-use operation ticket (`cain68.op.ticket.v1`), using the
same canonical serialisation (`cain_workforce/canon.py`) as E68.

| Component | File | Responsibility |
|---|---|---|
| Shared state | `cain_workforce/e72/crosshost.py` (`SharedState`) | missions, tasks, leases, fencing, tickets, idempotent results, hash-chained events |
| Coordinator | `cain_workforce/e72/coordinator_daemon.py` | HTTP node protocol, leader lease, task dispatch |
| Node | `cain_workforce/e72/node_daemon.py` | verifies ticket/grant locally, executes an allowlisted operation, signs the result |
| Deployment | `scripts/evolution72/crosshost_setup.py` | dedicated PostgreSQL container, systemd units, SSH tunnels |
| Demonstration | `scripts/evolution72/crosshost_demo.py` | M1 cross-verification, failover, reboot, expired grant, probes |

## Hosts

- **Host A** — the build host. Runs coordinator A, node A and the PostgreSQL primary (Docker container
  `cain72-pg-local`, loopback only).
- **Host B** — the established Vultr development replica `45.76.169.191`. Runs coordinator B and node B under
  systemd. It reaches the database through an authenticated SSH reverse tunnel (`-R 15432`), and coordinator A
  through `-R 7480`.
- Inbound TCP is blocked by the provider firewall, so transport is SSH. That is a real network boundary, not a
  same-host process.

## Database invariants

- `tasks.state` moves `READY → LEASED → VERIFY_READY → VERIFY_LEASED → VERIFIED`; a lease expiry returns the task
  to the previous ready state.
- Every claim increments `tasks.fencing`. A result with a lower fencing token is rejected with `STALE_FENCING`.
- `results` has a primary key `(task_id, attempt, kind)`, so a duplicate execute/verify row cannot be inserted.
- `tickets.used_at` is set on first use; a second submission is refused.
- Grants are checked *inside* the claim transaction: expiry, capability, operation budget. An expired grant moves
  the task to `BLOCKED` with `GRANT_REFUSED:...`; it is never silently restored after recovery.
- The verification stage excludes the producer's host in SQL (`NOT EXISTS (... r.host_id = verifier_host)`), so a
  producer can never verify its own result.
- Events are hash-chained with `prev_hash`; `SharedState.event_chain()` recomputes the chain from raw rows.

## Leader election and fencing

A single `leaders` row holds `instance_id`, `endpoint`, `lease_expires` and a monotonic `fencing` counter. A
coordinator renews every 1.5 s with a 6 s lease. When the holder dies, the next coordinator acquires the row under
an advisory lock and increments `fencing`. Only the holder serves `poll`; a follower returns the current leader
endpoint so nodes can switch.

**This is not Byzantine fault tolerance.** PostgreSQL is a single primary; a compromised primary or a malicious
coordinator is outside the threat model. Consensus is not claimed.

## Measured results

See `artifacts/e72/crosshost/crosshost.json` for the machine-readable record:

- M1: cross-host independent verification on 12 tasks, all produced on one host and recomputed on the other.
- Failover: coordinator A killed; coordinator B acquired leadership with a higher fencing token and completed a
  120-task mission; zero duplicate results.
- Reboot: host B rebooted. `boot_id` changed and uptime reset; node B re-registered; leases were reclaimed; M4
  (100 tasks) completed.
- Expired authority: a 12 s grant expired across the reboot; remaining tasks became `BLOCKED` with
  `GRANT_REFUSED:GRANT_EXPIRED`.
- Stale fencing: a signed result with `fencing-1` was rejected.

## Honest limitations

- Single PostgreSQL primary; no network-partition test against the database.
- One reboot of one host; not a disaster-recovery proof for a production system.
- Two hosts only; no third-party witness or independent trust anchor.
- The workload is deterministic hash/text work, not model reasoning.