Xudanu Storage & Rendering Architecture

1 System Layers

Five layers, each with a single responsibility. Data flows top to bottom on write, bottom to top on read.

👤 User Interface

What the user sees and interacts with — document text, images, graphs, panels

React 19 · Vite 8 · CollaborativeEditor (contentEditable)

TS Frontend State (in memory)

React state + hooks. Text in useCrdtSync, spans in useCompoundEdition, images in imageEntries. Lost on page refresh.

WorkspaceShell.tsx · useCrdtSync.ts · useCompoundEdition.ts

⚡ Wire Protocol

WebSocket (JSON) for real-time CRDT ops + links + annotations. HTTP POST for binary uploads (images). HTTP GET for blob retrieval.

WebSocket /xudanu · HTTP /api/blob/upload · HTTP /blobs/{hash}

RS Server (in memory)

All works, editions, links, annotations, blob metadata loaded on startup. CRDT ops modify this in real time. Checkpointed to disk periodically.

Server struct · HashMaps for works, links, annotations · O-tree CRDT

💾 Disk Storage (permanent)

Three storage systems: chunk store (editions), blob store (images), manifest (metadata). All content-addressed (BLAKE3). Survives restarts.

data/chunks/ · data/blobs/ · data/manifest.json · data/wal.log

2 Disk Layout

What's actually on disk. Three separate storage systems, each optimized for their data type.

data/ ├── chunks/ # O-tree edition chunks (postcard format) │ ├── a3/ # First 2 hex chars = directory │ │ └── a3f2b1c4....xchunk # One edition (BLAKE3-addressed) │ ├── b7/ │ │ └── b7e9d1a2....xchunk │ └── ... │ ├── blobs/ # Binary Large Objects (images, future: audio/video) │ ├── 0a/ │ │ └── 0a1b2c3d... # Optimized image (BLAKE3-addressed) │ │ └── 0a1b2c3d..._preview # 400px WebP thumbnail │ └── ... │ ├── attribution/ # Per-character attribution log │ └── security.log.2026-07-20 # Tamper-evident, Ed25519-signed │ ├── manifest.json # Server metadata + work entries + link entries ├── manifest_v212465.json # Previous manifest version (dual-slot A/B) ├── wal.log # Write-Ahead Log (truncated on checkpoint) ├── server.key # Ed25519 signing key pair ├── key_history.json # Key rotation history └── security.log.seed # Initial seed for attribution chain

Content-addressed = deduplication for free. If two works share the same edition chunk or the same image is uploaded twice, only one copy is stored. The BLAKE3 hash is the filename — if the hash matches, the content is identical.

3 Document & Revision Storage

How text gets from your keyboard to permanent disk, and how it comes back.

Figure 1: Document write flow (top) and read flow (bottom). Yellow = WAL, purple = server RAM, teal = disk, blue = on-demand revision loading.

4 Revision Storage — Three Levels

Revisions exist at three levels of persistence. Each survives longer but costs more to access.

Level	What's stored	Speed	Survives
1. In-memory	Full Edition objects (revision_history BTreeMap)	Instant (HashMap lookup)	Until server restart
2. Chunk store	Chunk references (WorkChunkRef.history map)	~10ms (read + deserialize)	Forever (content-addressed)
3. Manifest	Revision metadata (RevisionMeta: timestamps, descriptions)	Instant (loaded on startup)	Forever

Recent fix: prev_chunk_history preserves old revision chunk references when mark_dirty() clears chunk_ref. Without this, editing a work after restart would lose all pre-restart revisions from the chunk store.

// WorkChunkRef — stored in manifest per work pub struct WorkChunkRef { current_root: EditionChunkRef, // → points to current edition's chunks history: BTreeMap<u64, EditionChunkRef>, // → revision 0, 1, 2, ... each pointing to chunks revision_count: u64, // how many revisions exist } // RevisionMeta — stored in manifest (FR-23) pub struct RevisionMeta { revision_id: u64, parent: Option<u64>, created_at: u64, // Unix timestamp created_by: Option<BeId>, // which identity made this revision description: Option<String>, // "Added section on transclusion" is_notable: bool, // auto-detected or manual flag change_summary: String, // "+450 chars, 15% changed" }

5 Image Storage & Rendering Flow

From drag-drop to rendered pixel. The image passes through 7 steps.

1

User clicks 📷 / drag-drops image

File read into ArrayBuffer in the browser. MIME type detected from file extension.

const buf = await file.arrayBuffer(); // buf = raw PNG/JPEG bytes (e.g., 500KB)

2

HTTP POST upload

Raw bytes sent to /api/blob/upload with Content-Type header and session ID for auth. Not WebSocket — HTTP is 10× more efficient for binary.

POST /api/blob/upload Content-Type: image/png X-Xudanu-Session: 14372... [500KB raw bytes]

3

Server: optimize + thumbnail

Server decodes the image (image crate), then:
• PNG → oxipng lossless compression (40-70% smaller)
• JPEG → re-encode at quality 80 (50% smaller)
• WebP → kept as-is
• Generates 400px WebP thumbnail (always)
• Extracts width × height

4

BlobStore: content-addressed storage

BLAKE3 hash of image bytes = filename. Two files written: optimized image + thumbnail. Dedup is free (same hash = same file).

data/blobs/a3/a3f2b1c4... ← optimized image data/blobs/a3/a3f2b1c4..._preview ← 400px thumbnail

5

O-tree: insert blob element

A RangeElement::Blob is inserted into the document's O-tree at the cursor position. This is how the image "knows" it belongs to this document at this position.

elementInsert(workId, position, { type: "blob", content_hash: 0xa3f2b1c4..., mime_type: "image/png", width: 800, height: 600 })

6

Client: fetch preview (progressive loading)

Frontend fetches the thumbnail first (~15KB) via HTTP GET. Creates a temporary Blob + blob: URL in the browser. Image appears instantly.

GET /blobs/a3f2b1c4.../preview → 15KB WebP Blob → URL.createObjectURL() → <img src="blob:...">

7

Client: fetch full image (on click)

When user clicks the thumbnail, the full-resolution image is fetched. Opens in a new tab or lightbox.

GET /blobs/a3f2b1c4... → 250KB optimized PNG Opens full-resolution in new tab

6 Image Data Model

How image data is structured at each layer.

// ── Disk: BlobStore files ── // data/blobs/{hash}/ ← directory per blob (BLAKE3 hash) // ├── {hash} ← optimized image bytes // └── {hash}_preview ← 400px WebP thumbnail // ── Server (Rust): BlobMeta ── pub struct BlobMeta { content_hash: [u8; 32], // BLAKE3 hash (permanent identity) byte_size: u64, // after optimization mime_type: String, // "image/png", "image/jpeg" preview_hash: Option<[u8; 32]>, // hash of thumbnail width: Option<u32>, // original width in pixels height: Option<u32>, // original height } // ── O-tree: RangeElement::Blob ── // Lives in the edition alongside text and transclusion elements pub enum RangeElement { Text { text: String }, Transclusion { source_work_id, char_start, char_end }, Blob { content_hash: u64, // hash_u64 (first 8 bytes of BLAKE3) mime_type: String, byte_size: u64, width: Option<u32>, height: Option<u32>, }, Overlay { overlay: ImageOverlay }, // transforms: crop, resize, rotate } // ── Client (TypeScript): imageEntries ── // Frontend state in WorkspaceShell, cached in localStorage interface ImageEntry { hash: number; // blobHashToU64(content_hash) mime: string; width?: number; height?: number; url?: string; // temporary blob: URL (not persisted) loading: boolean; }

7 What's Stored Where — Summary

The complete map of data in Xudanu.

Data type	RAM (lost on restart)	Disk (permanent)	Access path
Document text	Works HashMap → Work → Edition	data/chunks/ (postcard format)	crdt_sync_open → text
Revisions (old editions)	Work.revision_history (current session only)	data/chunks/ (via WorkChunkRef.history)	work_text_at_revision()
Revision metadata	Server.revisions HashMap	manifest.json → revisions map	work_revisions_list()
Typed links	Server.links HashMap	manifest.json → links_hash → chunk	link_list_for_work()
Annotations (bold, italic, notes)	O-tree elements (part of edition)	data/chunks/ (part of edition)	annotation_list()
Images	— (not cached in RAM)	data/blobs/{hash} + data/blobs/{hash}_preview	GET /blobs/{hash}
Image reference in document	RangeElement::Blob (part of edition)	data/chunks/ (part of edition)	edition.all_entries()
Attribution (who wrote what)	— (queried on demand)	data/attribution/security.log.*	attribution_query_resolved()
Trails	Server.trails HashMap	manifest.json → social_chunk → chunk	trail_list()
Starred works	Server.starred_works HashMap	manifest.json → social_chunk → chunk	work_is_starred()

8 Planned Storage Improvements

Image position persistence TODO

Currently images are tracked in localStorage (frontend only). Need server-side query: "what blob elements exist in this edition?" A new work_blob_list wire op would scan the O-tree for RangeElement::Blob and return positions.

server.rs::work_blob_list()

Image overlays (crop, resize) Scaffolded

RangeElement::Overlay + ImageOverlay structs already exist. Would allow server-side crop/resize/rotate without duplicating the original image. The overlay references the base hash + list of operations.

range_element.rs::ImageOp

Chunk-level structural sharing Deferred (FR-17)

Currently each revision stores a full edition snapshot. Enfilade-style structural sharing would split editions into paragraph-level chunks, sharing unchanged chunks across revisions. Crossover at ~80 revisions.

docs/dev/FR-17.md

Revision eviction Not needed yet

Old editions load from disk on demand. For works with 1000+ revisions, we'd add eviction: keep recent N revisions in RAM, evict older ones. Loaded from chunk store when requested.

work_load_revision() fallback