How Xudanu reclaims disk space without losing data
Every 30 seconds, Xudanu runs a checkpoint that serializes all in-memory state to disk. After the checkpoint completes, the garbage collector (GC) scans the chunk store and deletes any chunk that is no longer referenced by live data.
Chunks are content-addressed blocks of serialized data. Over time, as works are edited, new chunks are written and old chunks become unreachable. The GC reclaims this space.
The GC marks each chunk as referenced (green) or orphaned (red dashed), then deletes the orphans.
For each work in self.works, walk the edition's chunk reference tree and add all reachable chunk hashes to the referenced set.
Read manifest.json from disk and add 6 hash references: links_hash, annotations_hash, historical_authors_hash, blob_metas_hash, content_address_hash, fossil_snapshots_hash.
Walk chunk reference trees for all clubs (self.clubs) and standalone editions (self.standalone_editions).
Scan for manifest_v*.json backup files and collect their chunk references. This prevents GC from deleting chunks that a rollback might need.
Call chunk_store.all_chunk_hashes() to enumerate every chunk file in the chunks/ directory.
For each chunk on disk that is not in the referenced set, call chunk_store.delete_chunk().
Ok(0) immediately — it does not proceed.
This prevents deleting valid chunks that the GC failed to protect.
The GC runs as part of the checkpoint cycle. Understanding the full lifecycle is critical for understanding how data survives crashes.
The full lifecycle: user operation → WAL → checkpoint → GC → WAL truncate. On crash, both manifest and WAL provide recovery.
Chunk writes use the gold-standard durable write pattern. This is why actual data corruption is vanishingly rare:
On the AWS production instance, transclusion links were disappearing after server restarts. The root cause was NOT data corruption — it was the GC deleting valid chunks.
// GC reads manifest to protect chunks let manifest = read_manifest(path); if let Ok(m) = manifest { // add hashes to referenced set... } // GC proceeds even if manifest // read failed! Chunks that should // be protected are seen as orphans. // → links_hash chunk gets DELETED
// GC reads manifest to protect chunks let manifest = read_manifest(path); match manifest { Ok(m) => m, Err(e) => { // GC does NOT run. Stale chunks // may accumulate temporarily, // but valid data is never deleted. return Ok(0); } }
The fix is conservative: if the GC can't verify what's safe to delete, it doesn't delete anything. The cost is temporarily accumulated stale chunks, which will be cleaned up on the next successful GC cycle.
Transclusion links now have three independent persistence mechanisms:
Three independent layers ensure link data survives crashes, corruption, and GC.
| Component | File:Line | Function |
|---|---|---|
| GC | server.rs:10193 |
gc_orphaned_chunks() |
| Checkpoint | server.rs:9731 |
checkpoint_to_store() |
| Auto-checkpoint | server.rs:5157 |
auto_checkpoint() — 30s timer |
| Chunk write | chunk_store.rs:265 |
write_chunk_durable() |
| Chunk read | chunk_store.rs:299 |
read_chunk() — verifies hash |
| Manifest write | manifest.rs |
write_manifest() — dual-slot |
| Manifest read | manifest.rs |
read_manifest_dual() — tries both slots |
| WAL | wal.rs |
WalLog, replay_entries() |
| Restore | server.rs:4373 |
restore_from_data_dir() |
| Link WAL append | server.rs:5338 |
in create_link() |
| Link WAL replay | server.rs:3338 |
wal_replay_create_link() |