Federation Activation

How independent Xudanu servers connect, trust each other, replicate content, reach consensus, recover from failures, and scale — explained with diagrams for operators and developers. Updated for the hardened protocol: signed certificates, view change, key epochs, and 4-node BFT deployment.

What a federation is

A Xudanu federation is a set of independent server peers, each a full node (it serves the web UI, the client WebSocket /xudanu, and the peer WebSocket /federation). Peers connect to each other, replicate content (chunks verified by BLAKE3), maintain a shared membership via a web-of-trust of Ed25519 endorsements, and make cluster-wide decisions with PBFT consensus. Users connect to any peer's web UI; the peers federate behind the scenes.

The activation layer (FR-3) — now implemented

Five pieces of runtime wiring turn the finished components into a live federation. Each was previously missing; all five are now implemented and merged.

PBFT: Making decisions together

Xudanu uses Practical Byzantine Fault Tolerance (PBFT) for cluster-wide decisions — things like admitting a new member, expelling a bad actor, or recording a royalty payment. PBFT is a consensus protocol that lets a group of nodes agree on a decision even if some nodes are malicious or offline.

What PBFT is — in plain language

Imagine a committee of N people who must vote on a proposal. PBFT guarantees that:

• The group can reach a decision as long as at least 2/3 of the members are honest and reachable (formally: tolerates f faulty nodes out of 3f+1 total).
• No two honest members will ever disagree on the outcome.
• Once a decision is sealed, it cannot be undone.

The quorum rule, simply

For a cluster of N nodes, PBFT tolerates up to f faulty/offline nodes, where f = (N - 1) / 3. To make a decision, you need at least 2f+1 nodes to vote yes:

Broadcasting & synchronization

Once peers are connected, they need to stay in sync. Xudanu uses two complementary mechanisms: periodic sync (every 60 seconds) and event-driven broadcast (immediate, for governance).

How the periodic sync loop works

Every 60 seconds, each outbound connection sends a batch of sync frames to the peer on the other end. The peer processes them and replies with its own state. This ensures both sides converge even if they were temporarily disconnected.

How Xudanu shares content

Xudanu's document model is content-addressed: every piece of content (a chunk of text, an edition, a blob) is identified by its BLAKE3 hash, not by a server-assigned ID. This makes replication straightforward — you can verify any piece of data by recomputing its hash.

The chunk replication flow

When a user edits a document on Peer A, the change flows through the system and reaches all other peers within one sync interval (60 seconds):

What gets replicated

How a donated machine joins

Identity is the server's existing Ed25519 keypair (data/server.key); trust is a web-of-trust of signed endorsements (phase-19a, already built). A donated host proves who it is by its key, and gets vouched for by an existing member.

When things go wrong: failure & recovery

Federations run over real networks, and real networks fail. Xudanu is designed to self-heal — when a failed or partitioned node comes back, it catches up automatically. Here are the common scenarios and how the system handles them.

Scenario 1: Network partition (node temporarily unreachable)

A peer's network goes down (VPN drops, host reboots, ISP hiccup). The other peers continue operating. When the partitioned node returns, it reconnects and syncs.

Scenario 2: Node crash (data dir survives)

If a peer process crashes but its /data directory survives (common case — process killed, OOM, panic), the node restarts, restores from its chunk store, reconnects, and syncs. Content catches up via CRDT sync; governance catches up via state transfer — the restarted node requests the sealed-batch log tail from a peer and ingests it certificate-verified, in order. Data written before the crash is durable (checkpointed every 5 seconds). No manual intervention needed.

Scenario 3: Byzantine (malicious) node

A compromised node tries to inject tampered data or forge endorsements. The system rejects these automatically:

• Tampered blobs/chunks: BLAKE3 hash mismatch on import → rejected.
• Forged endorsements: Ed25519 signature verification fails → rejected.
• False membership claims: MembershipJoinRequest with invalid proofs → rejected.
• Cluster-wide removal: A PBFT Expel transaction removes the bad node's membership from all peers.
• Forged governance: votes and sealed batches without 2f+1 valid Ed25519 signatures are rejected on arrival; a byzantine leader can stall a round (recovered by view change) but cannot forge an outcome.
• Stolen validator key: votes stop at key-epoch expiry; the thief cannot rotate to a new key (rotation needs the operator's offline recovery key or a member quorum) and cannot survive the expiry.

Scenario 3b: Leader failure — automatic view change

The current leader crashes or stalls mid-round. After a 10-second timeout, the remaining replicas emit signed VIEW-CHANGE messages for the next view; when a quorum accumulates, the incoming leader broadcasts a signed NEW-VIEW and re-proposes. Governance continues in the new view within seconds. One caveat: a request whose round never reached prepare-quorum is discarded — the client retries with the new leader (values that did reach prepare-quorum carry over and cannot be changed).

Scenario 4: Split-brain (two partitions, both active)

If the network splits into two groups (e.g., A-B and C-D), both partitions continue accepting edits independently. PBFT cannot reach quorum in either partition (each has fewer than 2f+1 votes for a 4-node cluster with f=1). When the partition heals, CRDTs merge the divergent states — both sets of edits are preserved as alternatives, and the reconcile store's LWW (Last-Writer-Wins) registers pick the most recent timestamp.

Scenario 5: Slow or degraded node

A peer that is reachable but very slow (high latency, low bandwidth, overloaded CPU). Heartbeats detect this (30s timeout). The outbound dialer doesn't block on slow peers — each connection has its own task. Other peers continue syncing among themselves at full speed. The slow node catches up when it can.

Scaling: what to expect as you grow

Adding more machines increases resilience but also increases network traffic, because each peer connects to every other peer (full mesh). Here's what to expect at different scales.

Connection mesh: O(n²)

Each peer maintains an outbound connection to every other peer. With N peers, there are N × (N-1) directional connections (or N(N-1)/2 bidirectional pairs):

Bandwidth per sync cycle

Each 60-second sync cycle sends all published works, membership, states, and endorsements to each peer. For a fresh/small federation this is kilobytes. As content grows, bandwidth scales with total published content × number of peers.

Memory and CPU per peer

• RAM: Each peer holds its own chunk store, the membership OR-Set, reconcile states, and CRDT buffers. A peer with 100 works uses ~50–100 MB RAM. Federation adds ~2–5 MB per connected peer (channel buffers + sync state).
• CPU: Dominated by BLAKE3 hashing (on import) and Ed25519 signature verification (on handshakes/endorsements). Both are fast (< 1ms per operation). A 4-core VPS handles 10+ peers comfortably.
• Disk: Content-addressed storage grows with total published works. BLAKE3 deduplication means the same content stored on multiple peers takes space only once per peer.

Recommended configurations

Problem checklist by scale

A diagnostic reference for operators. If something seems wrong, check these items.

2–3 nodes (personal / small team)

• Common Keys not cross-registered: Peers must know each other's verifying keys — via --pin-members (preferred, strict) or --trusted-peer-key flags. If you see "peer not in trusted peers list," check that the hex keys match.
• Common Wrong peer address format: --peer expects ws://host:port/federation or host:port. Check that the port matches the listening port of the remote peer.
• Common Firewall blocks /federation: The WebSocket upgrade on /federation must be reachable. If using Docker, ensure the port is published.
• Tip First diagnostic stop: ./scripts/federation.sh diagnose — checks node health, cross-node validator-set consistency, margin alerts, peer-trust rejections, view-change activity, and disk headroom in one command.
• Watch Bootstrap not called: If membership is empty, check that --peer was specified (which enables federation and triggers membership_bootstrap_init).

5–10 nodes (organization)

• Watch Key management: Maintain the pinned member set as a versioned file; rotations need the offline recovery keys. Watch /health → governance_lifecycle — margin: 0 means one loss from halting; negative means governance is already halted (safely — nothing forks).
• Watch Sync bandwidth: With 10 peers and significant content, each sync cycle sends data to 9 peers. Monitor outbound bandwidth.
• Watch Stale membership: If a peer leaves without sending MembershipLeave, its entry persists until explicitly removed. Use the admin API to clean up.
• Tip Use 4+ nodes for PBFT: With 4 nodes (f=1), you can lose 1 node and still reach quorum. With 3 nodes (f=0), losing any node blocks governance.

10–20 nodes (large federation)

• Plan Connection overhead: Each peer holds 9–19 WebSocket connections. Monitor file descriptor limits (ulimit -n).
• Plan Startup thundering herd: If all peers restart simultaneously, they all try to dial each other at once. The exponential backoff helps, but consider staggered restarts.
• Watch Endorsement chain growth: The membership OR-Set grows with each endorsement. At 20 nodes with cross-endorsements, the set can be several hundred KB. Not a problem, but watch for sync latency.
• Shipped Leader failover is automatic: a stalled leader triggers view change after a 10s timeout — signed VIEW-CHANGE quorum, NEW-VIEW, re-proposal. At 20 nodes, vote traffic is O(n²) per round, which is the real cost of scale.

20–50+ nodes (research / max scale)

• Challenge Full mesh doesn't scale: 50 nodes = 2,450 connections. Each peer dials 49 others. This works but is wasteful. A gossip/relay topology is needed for sustainable scaling.
• Challenge Sync payload size: With 50 peers and GBs of content, the 60-second full sync is impractical. Incremental sync (using known_fingerprints) is essential.
• Challenge PBFT message complexity: O(n²) vote messages per round. At 50 nodes, each round generates ~2,500 vote messages. Practical but slow.
• Challenge Membership convergence latency: With 50 peers, a new endorsement takes several sync cycles to reach all nodes. Plan for eventual consistency, not immediate.
• Watch NAT / reachability: Donated hosts behind NAT cannot be dialed. Each peer must be directly reachable (public IP or port-forward). A relay/TURN server is a future feature.

Testing it: 4-node test bed

The docker-compose.yml runs four full-node peers on a private bridge network (host ports 8081–8084), each with --peer wiring to the other three — the BFT minimum. Admin passphrases go in via XUDANU_ADMIN_PASSPHRASE from an uncommitted .env file (never on the CLI — arguments are visible in ps and docker inspect).

docker compose up --build
# then browse:
open http://localhost:8081   # peer A (Alice)
open http://localhost:8082   # peer B (Bob)
open http://localhost:8083   # peer C (Carol)
open http://localhost:8084   # peer D (Dave)
curl http://localhost:8081/health | grep -A8 governance_lifecycle

For local testing without Docker, one command runs the whole lifecycle — it inits the nodes, extracts their keys, writes pinned-members.json (strict admission + unified validators), and starts all peers with per-node logs:

./scripts/federation.sh start 4 /tmp/fed   # also: status | diagnose | logs N | stop

The full operations guide — recovery playbooks (crash, leader failure, expired keys, state-transfer gaps), log locations, and the diagnostic reference — lives in FEDERATION-OPS.md in the source repository.

Sequencing (implementation order)