Federation Activation

How independent Xudanu servers connect, trust each other, replicate content, reach consensus, recover from failures, and scale — explained with diagrams for operators and developers.

What a federation is

A Xudanu federation is a set of independent server peers, each a full node (it serves the web UI, the client WebSocket /xudanu, and the peer WebSocket /federation). Peers connect to each other, replicate content (chunks verified by BLAKE3), maintain a shared membership via a web-of-trust of Ed25519 endorsements, and make cluster-wide decisions with PBFT consensus. Users connect to any peer's web UI; the peers federate behind the scenes.

The activation layer (FR-3) — now implemented

Five pieces of runtime wiring turn the finished components into a live federation. Each was previously missing; all five are now implemented and merged.

PBFT: Making decisions together

Xudanu uses Practical Byzantine Fault Tolerance (PBFT) for cluster-wide decisions — things like admitting a new member, expelling a bad actor, or recording a royalty payment. PBFT is a consensus protocol that lets a group of nodes agree on a decision even if some nodes are malicious or offline.

What PBFT is — in plain language

Imagine a committee of N people who must vote on a proposal. PBFT guarantees that:

• The group can reach a decision as long as at least 2/3 of the members are honest and reachable (formally: tolerates f faulty nodes out of 3f+1 total).
• No two honest members will ever disagree on the outcome.
• Once a decision is sealed, it cannot be undone.

The quorum rule, simply

For a cluster of N nodes, PBFT tolerates up to f faulty/offline nodes, where f = (N - 1) / 3. To make a decision, you need at least 2f+1 nodes to vote yes:

Broadcasting & synchronization

Once peers are connected, they need to stay in sync. Xudanu uses two complementary mechanisms: periodic sync (every 60 seconds) and event-driven broadcast (immediate, for governance).

How the periodic sync loop works

Every 60 seconds, each outbound connection sends a batch of sync frames to the peer on the other end. The peer processes them and replies with its own state. This ensures both sides converge even if they were temporarily disconnected.

How Xudanu shares content

Xudanu's document model is content-addressed: every piece of content (a chunk of text, an edition, a blob) is identified by its BLAKE3 hash, not by a server-assigned ID. This makes replication straightforward — you can verify any piece of data by recomputing its hash.

The chunk replication flow

When a user edits a document on Peer A, the change flows through the system and reaches all other peers within one sync interval (60 seconds):

What gets replicated

How a donated machine joins

Identity is the server's existing Ed25519 keypair (data/server.key); trust is a web-of-trust of signed endorsements (phase-19a, already built). A donated host proves who it is by its key, and gets vouched for by an existing member.

When things go wrong: failure & recovery

Federations run over real networks, and real networks fail. Xudanu is designed to self-heal — when a failed or partitioned node comes back, it catches up automatically. Here are the common scenarios and how the system handles them.

Scenario 1: Network partition (node temporarily unreachable)

A peer's network goes down (VPN drops, host reboots, ISP hiccup). The other peers continue operating. When the partitioned node returns, it reconnects and syncs.

Scenario 2: Node crash (data dir survives)

If a peer process crashes but its /data directory survives (common case — process killed, OOM, panic), the node restarts, restores from its chunk store, reconnects, and syncs. Data written before the crash is durable (checkpointed every 5 seconds). No manual intervention needed.

Scenario 3: Byzantine (malicious) node

A compromised node tries to inject tampered data or forge endorsements. The system rejects these automatically:

• Tampered blobs/chunks: BLAKE3 hash mismatch on import → rejected.
• Forged endorsements: Ed25519 signature verification fails → rejected.
• False membership claims: MembershipJoinRequest with invalid proofs → rejected.
• Cluster-wide removal: A PBFT Expel transaction removes the bad node's membership from all peers.

Scenario 4: Split-brain (two partitions, both active)

If the network splits into two groups (e.g., A-B and C-D), both partitions continue accepting edits independently. PBFT cannot reach quorum in either partition (each has fewer than 2f+1 votes for a 4-node cluster with f=1). When the partition heals, CRDTs merge the divergent states — both sets of edits are preserved as alternatives, and the reconcile store's LWW (Last-Writer-Wins) registers pick the most recent timestamp.

Scenario 5: Slow or degraded node

A peer that is reachable but very slow (high latency, low bandwidth, overloaded CPU). Heartbeats detect this (30s timeout). The outbound dialer doesn't block on slow peers — each connection has its own task. Other peers continue syncing among themselves at full speed. The slow node catches up when it can.

Scaling: what to expect as you grow

Adding more machines increases resilience but also increases network traffic, because each peer connects to every other peer (full mesh). Here's what to expect at different scales.

Connection mesh: O(n²)

Each peer maintains an outbound connection to every other peer. With N peers, there are N × (N-1) directional connections (or N(N-1)/2 bidirectional pairs):

Bandwidth per sync cycle

Each 60-second sync cycle sends all published works, membership, states, and endorsements to each peer. For a fresh/small federation this is kilobytes. As content grows, bandwidth scales with total published content × number of peers.

Memory and CPU per peer

• RAM: Each peer holds its own chunk store, the membership OR-Set, reconcile states, and CRDT buffers. A peer with 100 works uses ~50–100 MB RAM. Federation adds ~2–5 MB per connected peer (channel buffers + sync state).
• CPU: Dominated by BLAKE3 hashing (on import) and Ed25519 signature verification (on handshakes/endorsements). Both are fast (< 1ms per operation). A 4-core VPS handles 10+ peers comfortably.
• Disk: Content-addressed storage grows with total published works. BLAKE3 deduplication means the same content stored on multiple peers takes space only once per peer.

Recommended configurations

Problem checklist by scale

A diagnostic reference for operators. If something seems wrong, check these items.

2–3 nodes (personal / small team)

• Common Keys not cross-registered: Both peers must register each other's verifying key via --trusted-peer-key. If you see "peer not in trusted peers list," check that the hex keys match.
• Common Wrong peer address format: --peer expects ws://host:port/federation or host:port. Check that the port matches the listening port of the remote peer.
• Common Firewall blocks /federation: The WebSocket upgrade on /federation must be reachable. If using Docker, ensure the port is published.
• Watch Both nodes offline: With 2 nodes and f=0, if one goes down, PBFT cannot function. Content sync still works when it returns (CRDTs), but governance is blocked until both are up.
• Watch Bootstrap not called: If membership is empty, check that --peer was specified (which enables federation and triggers membership_bootstrap_init).

5–10 nodes (organization)

• Watch Key management: Each new peer's verifying key must be registered on all existing peers. Consider maintaining a shared trusted-peers.txt file.
• Watch Sync bandwidth: With 10 peers and significant content, each sync cycle sends data to 9 peers. Monitor outbound bandwidth.
• Watch Stale membership: If a peer leaves without sending MembershipLeave, its entry persists until explicitly removed. Use the admin API to clean up.
• Tip Use 4+ nodes for PBFT: With 4 nodes (f=1), you can lose 1 node and still reach quorum. With 3 nodes (f=0), losing any node blocks governance.

10–20 nodes (large federation)

• Plan Connection overhead: Each peer holds 9–19 WebSocket connections. Monitor file descriptor limits (ulimit -n).
• Plan Startup thundering herd: If all peers restart simultaneously, they all try to dial each other at once. The exponential backoff helps, but consider staggered restarts.
• Watch Endorsement chain growth: The membership OR-Set grows with each endorsement. At 20 nodes with cross-endorsements, the set can be several hundred KB. Not a problem, but watch for sync latency.
• Watch PBFT leader bottleneck: The current leader proposes all transactions. At 20 nodes, if the leader is slow, governance stalls. Leader rotation exists but view-change (automatic leader failover) is a future feature.

20–50+ nodes (research / max scale)

• Challenge Full mesh doesn't scale: 50 nodes = 2,450 connections. Each peer dials 49 others. This works but is wasteful. A gossip/relay topology is needed for sustainable scaling.
• Challenge Sync payload size: With 50 peers and GBs of content, the 60-second full sync is impractical. Incremental sync (using known_fingerprints) is essential.
• Challenge PBFT message complexity: O(n²) vote messages per round. At 50 nodes, each round generates ~2,500 vote messages. Practical but slow.
• Challenge Membership convergence latency: With 50 peers, a new endorsement takes several sync cycles to reach all nodes. Plan for eventual consistency, not immediate.
• Watch NAT / reachability: Donated hosts behind NAT cannot be dialed. Each peer must be directly reachable (public IP or port-forward). A relay/TURN server is a future feature.

Testing it: Docker test bed

The docker-compose.yml runs three full-node peers on a private bridge network (host ports 8081/8082/8083), each with --peer pointing at the other two. On a 32 GB machine this is trivial for small/fresh data dirs — you could run 5–10+ peers; 3–4 is the sweet spot for exercising PBFT quorum and membership.

docker compose up --build
# then browse:
open http://localhost:8081   # peer A
open http://localhost:8082   # peer B
open http://localhost:8083   # peer C

For manual testing with two peers (no Docker):

# Terminal 1: Peer A
cargo run --release --features server --bin xudanu-server -- \
  run 127.0.0.1:8081 data-a \
  --peer 127.0.0.1:8082 \
  --trusted-peer-key <B's verifying key>

# Terminal 2: Peer B
cargo run --release --features server --bin xudanu-server -- \
  run 127.0.0.1:8082 data-b \
  --peer 127.0.0.1:8081 \
  --trusted-peer-key <A's verifying key>

Each server prints its verifying key at startup: Federation: this server's verifying key: <hex>. Copy it to the other peer's --trusted-peer-key flag.

Sequencing (implementation order)