Manifest & Chunk Architecture

How manifest.json uses hash pointers to reference document content stored in the chunk store — the indirection chain from metadata to text.

1. Architecture Overview

Xudanu separates metadata (the manifest) from content (the chunk store). The manifest is a single JSON file that acts as an index — it knows about every work, club, link, and edition, but it doesn't hold the actual text. Instead, it stores hash pointers that reference content blocks in the chunk store.

This is the same principle as a database index pointing to data pages, or a Git tree object pointing to blobs — content-addressed storage where the SHA-256 hash of the data IS its address.

Figure 1: The manifest stores metadata and hash pointers. The chunk store holds the actual document content, addressed by SHA-256 hash.

2. Two-Layer Storage Model

Because chunks are content-addressed, identical content across multiple works is automatically deduplicated. If two works contain the same paragraph, there's only one copy in the chunk store.

3. The Indirection Chain

When the server renders a document, it follows a chain of pointers from the manifest down to the actual text characters. Each arrow is a hash lookup in the chunk store:

Figure 2: The indirection chain from manifest to text. Each blue box lives in the manifest; each green box is a chunk in the content-addressed store.

4. Manifest Fields

The manifest struct (src/persist/manifest.rs:156) contains these top-level fields, organized by the version that introduced them:

Envelope Fields

v1 Core: Identity & Content

v2: Transclusion Links

v3: Federation

v4: Metadata & Social

5. Work → Edition Tree

WorkEntry

Each work in the manifest has a WorkEntry that holds metadata about the work and a pointer (WorkChunkRef) to its content tree:

pub struct WorkEntry {
    be_id: BeId,                              // unique ID for this work
    work_ref: WorkChunkRef,                   // → pointer into chunk store
    is_source: bool,                          // imported from external source?
    source_author_id: Option<BeId>,           // original author
    source_edition_info: Option<String>,     // edition label
    content_start_line: Option<u64>,          // for partial imports
    content_end_line: Option<u64>,
    source_fingerprint: Option<Vec<u64>>,    // recomputed on restore
}

WorkChunkRef

The WorkChunkRef is the pointer. It lives only in the manifest — it's never serialized as a chunk itself. It tells the server where to find this work's content:

pub struct WorkChunkRef {
    be_id: BeId,                              // same as WorkEntry.be_id
    owner: Option<BeId>,                     // owning club
    revision_count: u64,                     // how many editions exist
    current_root: EditionChunkRef,            // → LIVE edition (the one users see)
    history: BTreeMap<u64, EditionChunkRef>, // → OLD editions (revision → ref)
    read_club: Option<BeId>,                 // who can read
    edit_club: Option<BeId>,                 // who can edit
    sponsors: Vec<BeId>,                     // sponsoring clubs
    endorsements: Vec<(u64, u64)>,           // (club_id, token_id) pairs
}

EditionChunkRef

The smallest pointer unit. Just a hash and a count:

pub struct EditionChunkRef {
    root_hash: [u8; 32],      // SHA-256 of the EditionRootChunk
    entry_count: u32,         // number of entries (for sanity check)
}

The root_hash is looked up in the chunk store to load the EditionRootChunk.

6. Chunk Types

Three chunk types live inside the chunk store. They form a tree:

struct EditionRootChunk {
  default: Option<RangeElement>,
  domain_start: Option<i64>,
  domain_infinite_above: bool,
  entry_count: u32,
  entry_chunk_hashes: Vec<[u8;32]>,
  provenance_hash: Option<[u8;32]>,
}

struct EntryChunk {
  entries: Vec<(i64, RangeElement)>,
  provenances: Vec<Option<
    ElementProvenance>>,
}

ProvenanceChunk

struct ProvenanceChunk {
  spans: Vec<SpanProvenance>,
}

Tracks who wrote what — which club/author created each span of text. This is the attribution data that powers the "via work:XXXX" labels and endorsement chains. Optional: small documents may have no provenance chunk.

7. Three-Layer Link Persistence

Transclusion links are the most safety-critical data — losing them means losing the connections between documents. Xudanu stores links in three redundant layers:

Figure 3: Three-layer link persistence. Links survive crashes through WAL journaling, inline manifest storage, and content-addressed backup chunks.

8. What Happens on Restart

The restore_from_data_dir() function follows this sequence:

9. Data Directory Layout

my-data/
│
├── manifest.json                    # The index file (metadata + pointers)
├── manifest_v{N}.json               # Sequence-numbered backups (keep last 3)
├── manifest_a.json                   # Dual-slot A backup
├── manifest_b.json                   # Dual-slot B backup
│
├── wal.journal                      # Write-ahead log (append-only)
│
├── chunks/                          # Content-addressed chunk store
│   ├── a1/b2/a1b2c3d4e5f6...        # EntryChunk (text content)
│   ├── d4/e5/d4e5f6a7b8c9...        # EditionRootChunk
│   ├── g7/h8/g7h8i9j0k1l2...        # ProvenanceChunk
│   └── ...                          # (sharded by first 2 bytes of hash)
│
├── blobs/                           # Binary blob data (images, files)
│   └── {hash}.bin                   # Metadata in manifest.blob_metas
│
├── server.key                       # Ed25519 signing key (encrypted)
│
├── attribution.log                   # Tamper-evident provenance chain
│
└── federation/                       # Federation state (if enabled)
    └── peers.json

10. Versioning & Migration

Manifest Version

The format_version field tracks schema changes. The migration chain lives in src/persist/migrations.rs:

pub const CURRENT_MANIFEST_VERSION: u32 = 4;

// Version history:
// v1 — baseline: works, clubs, editions, admin, key_history
// v2 — added: links, links_hash, link_counter
// v3 — added: federation, reconcile_store
// v4 — added: content_address, blob_metas, historical_authors,
//           annotations, fossil_snapshots, social features (starred, trails)
// v5+ — (future, not yet used)

Chunk Format Tags

Separate from the manifest version, the chunk store has its own format versioning. Since v0.7.6, every chunk is prefixed with a format tag byte:

pub const CHUNK_FORMAT_JSON: u8     = 0x4A;     // 'J' = JSON
pub const CHUNK_FORMAT_POSTCARD: u8 = 0x50;     // 'P' = Postcard (binary)

The WAL journal also has a version header since v0.7.6:

pub const WAL_VERSION: u32 = 1;

// WAL file format:
// [4 bytes: WAL_VERSION = 1] [entries...] [entries...]

Versioning Coverage Summary

Quick Reference: Pointer Types

Source Files