Migration Notes: Udanax Gold C++ → Xudanu Rust
Phase 2: Edition / XnRegion / RangeElement
What was implemented
Three new modules under src/edition/:
| Module | C++ Original | Rust Implementation |
|---|---|---|
xn_region.rs |
IntegerRegion (integerx.hxx) |
XnRegion — transition-array encoding of integer sets |
range_element.rs |
FeRangeElement hierarchy (nkernelx.hxx) |
RangeElement enum with Data, Text, Edition, Label, PlaceHolder, IDHolder, Work |
edition.rs |
FeEdition / BeEdition (nkernelx.hxx, brange3x.hxx) |
Edition — wraps OrglRoot O-tree |
Design decisions
-
XnRegion uses transition arrays (same as Gold's
IntegerRegion):(starts_inside: bool, transitions: Vec<i64>). This gives O(log n)contains()via binary search and clean set operations via sorted-merge. -
RangeElement is an enum. Gold uses a deep class hierarchy (FeDataHolder, FeEdition, FePlaceHolder, FeLabel, FeIDHolder, FeWork). Rust's enum is more natural and avoids heap allocation.
-
Carrier pairs element + optional label. Mirrors Gold's
BeCarrierwhich pairs aBeRangeElementwith an optionalBeLabel.
Foundation Hardening: O-tree Implementation
What was implemented
A fourth module under src/edition/:
| Module | C++ Original | Rust Implementation |
|---|---|---|
orgl.rs |
OrglRoot / Loaf / InnerLoaf / OExpandingLoaf (orootx.hxx, loavesx.hxx) |
OrglRoot (Empty/Actual) + Loaf enum (Leaf/Split) with splay algorithm |
O-tree architecture
The O-tree is Gold's persistent splay tree for Edition content. Our Rust port:
- Loaf is a Rust enum:
Leaf { region, entries }orSplit { split, in_child, out_child } Leafstores sorted(i64, Arc<Carrier>)pairs with a domainXnRegionSplitpartitions space by aXnRegioninto in/out children- Leaves auto-split when they exceed
MAX_LEAF_SIZE(16384, matching Gold'stableSegmentMaxSize) - OrglRoot wraps a
Loaf(or is Empty), providing the Edition-level API - Splay algorithm (9-case transformation table from SplitLoaf::actualSplay) is fully implemented:
splay(region)restructures the tree so the region is at/near the root- Returns
SplayResult::Outside / Partial / FullyContained - Handles rotate-right, rotate-left, interleave, and swap operations
Edition now uses OrglRoot internally
Edition was rewritten to wrap OrglRoot instead of BTreeMap:
fetch()now returnsOption<RangeElement>(owned) instead ofOption<&RangeElement>(borrowed)- All mutation methods (
with,without,replace) use structural sharing via the O-tree - Auto-splits leaves when they exceed 16384 entries
- Stress test: 50,000 positions in ~38 seconds including splay operations
Simplifications from Gold's O-tree
| Gold Feature | Rust Status |
|---|---|
| DspLoaf (transform wrapper) | Done — Loaf::Dsp variant, transformed_by returns O(1) Dsp |
| OPartialLoaf (placeholder with TrailBlazer) | Simplified: Leaf stores explicit entries |
| OVirtualLoaf (backed by SharedData) | Simplified: Leaf stores explicit entries |
| RegionLoaf (points to BeRangeElement) | Simplified: Leaf stores explicit entries |
| H-tree (history/version tracking) | Not yet; needed for transclusion backfollow |
| Sensor crum / Bert crum (canopy indices) | Not yet; needed for transclusion queries |
| In-place mutation (placement new) | Rust enum variant replacement instead |
Portability gaps (Gold features not yet in Rust)
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| Infinite-domain Editions | Editions can map infinite regions (e.g., above(5) → constant) |
Done — Leaf default + tombstone entries | Complete |
| H-tree / history tracking | Version tracking parallel to O-tree for backfollow | Done — HUpperCrumData with canopy integration |
Complete |
| Sensor crum / Bert crum (canopy indices) | Canopy trees for filtering transclusion queries | Done — BertCanopy/SensorCanopy with flag propagation |
Complete |
| CoordinateSpace abstraction | Generic Position/Region across IntegerSpace, RealSpace, SequenceSpace, CrossSpace | Only integer positions (i64) | Add when needed; integer is the dominant case |
| Stepper / retrieve | edition->stepper(region, order) for filtered iteration |
iter() only, no region filter |
Add iter_in_region() method |
| Bundle retrieval | retrieve() returns Array/Element/PlaceHolder bundles |
Not implemented | Add when needed for bulk reads |
| Fe/Be split | FrontEnd (session) / BackEnd (persistent) object split | Done — BeRangeElement trait + InMemoryBeStorage |
Disk backing in Phase 6 |
| Admin API | FeAdminer — acceptConnections, shutdown, grants, activeSessions |
Done — AdminState + wire ops |
GateLockSmith in Phase 11 |
| Handshake | SetCommProtocol / BootPlan — version negotiation |
Done — HandshakeResponse at WS start |
Sufficient for v1 |
| Detector wire | CommXxxDetector + DetectorEvent — push events to client |
Partial — subscription tracking + event push | Per-type filtering + queuing |
| Work (mutable container) | FeWork holds current edition + revision history |
Done — Work with revise/history/clubs/sponsors |
Complete |
| GrandMap (ID registry) | BeGrandMap bidirectional ID ↔ BeRangeElement |
Done — GrandMap with IdSpace, assign_id, fetch |
Complete |
| Content Pool | Content-addressed storage for RangeElements | Done — ContentPool with hash-based store/retrieve/find |
Complete |
| Transclusion queries | transcluders(), works(), rangeTranscluders() |
Done — BackfollowEngine orchestrates full query pipeline |
Complete |
| Permissions / endorsements | BertProp, SensorProp, endorsement/permission spaces |
Done — flag-based props with endorsement bit allocation | Complete |
| Canopy tree (Bert/Sensor) | CanopyCrum, BertCrum, SensorCrum |
Done — Rc<RefCell<>> balanced binary tree with flag propagation | Complete |
| H-tree (version tracking) | HistoryCrum, HUpperCrum, HBottomCrum |
Done — HUpperCrumData with delayed_store_backfollow walk | Complete |
| Backfollow engine | RecorderFossil, ResultRecorder, Matcher, TrailBlazer |
Done — synchronous in-memory BackfollowEngine | Complete |
| Label propagation | positionsLabelled(), rebind(), label identity tracking |
Label exists on Carrier but no propagation API | Add in future phase |
| DspLoaf (transform wrapper) | Lazy displacement without rebuilding tree | Done — Loaf::Dsp variant |
Complete |
Enhancement ideas for future phases
-
yrs/CRDT as transport layer: The Edition model maps naturally to yrs
DocwithTextsequences. An Edition could be materialized into a yrs document for real-time sync, while maintaining the Gold partial ordering for conflict preservation. -
Content-addressed storage: Done —
ContentPoolimplements hash-based store/retrieve/find_by_content. -
Compressed transition arrays: For very large regions, run-length encoding with 32-bit deltas could reduce memory usage.
-
DspLoaf for lazy transforms: Done —
Loaf::Dspwraps child with offset. Splay materializes back to concrete nodes. -
Parallel region operations:
merge_transitionscould be parallelized for large regions using rayon. -
Fe/Be trait boundary: Done —
BeRangeElementtrait with identity, owner, clone_boxed.InMemoryBeStorage(HashMap-backed) withClonefor GrandMap integration. -
Identity-based shared_region: Done —
Edition::identity_shared_region(other, id_eq)compares by identity (be_id) instead of value (PartialEq).foundation for transclusion.
Gold test cases ported
- Region: 6 canonical example regions × 10 unary checks + 15 pairs × 8 binary checks = all Gold RegionTester checks pass
- Edition: 14 test cases from
makeEditionTestOn,editionTestOn, andcompareTestOnin nkernelt.cxx - O-tree: 22 tests for Loaf/OrglRoot (splay, split, combine, copy, domain, fetch, etc.)
- Stress: 50K position edition, 10K splay operation
- Total: 212 tests (114 ent + 98 edition), all passing
Test history
| Phase | Tests | Notes |
|---|---|---|
| Phase 2 initial | 189 | BTreeMap-based Edition |
| Foundation hardening | 212 | O-tree based Edition, all Gold tests preserved |
| DspLoaf + Infinite + Fe/Be | 245 | DspLoaf lazy transforms, infinite domains, backend traits |
| Phase 3: GrandMap/Work/Pool | 285 | GrandMap, Work, ContentPool, identity-based shared_region |
| Phase 4: Transclusion | 367 | Props, Canopy, H-tree, TransclusionIndex, BackfollowEngine, EditionMeta |
| Phase 5: Links | 401 | HyperLink, HyperRef (Single/Multi), Path, link-aware transclusion queries |
| Phase 6: Persistence | 477 | Abraham/Shepherd → Persistent trait, SnarfStorage, Counter, Transaction, PersistentWork/Edition round-trip |
Phase 6: Persistence Layer
What was implemented
A complete persistence layer faithful to Gold's architecture, ported to Rust idioms:
| Module | C++ Original | Rust Implementation |
|---|---|---|
persist/persistent.rs |
Abraham/FlockInfo/FlockLocation (shephx.hxx, flkinfox.hxx) | FlockId, FlockInfo (bitflags), FlockLocation, FlockState enum |
persist/traits.rs |
Abraham base class + Heaper hierarchy | Persistent trait (flock_id, flock_info, type_tag, to_bytes), PersistentRef<T>, PersistentRegistry, TypeRegistry (category/recipe dispatch), encode_flock/decode_flock |
persist/engine.rs |
DiskManager abstract interface | StorageEngine trait (store_new, disk_update, remember, forget, destroy, dismantle, begin/end_transaction, commit, rollback) |
persist/memory.rs |
In-memory stub for testing | InMemoryStorage — full StorageEngine impl with transaction support and rollback |
persist/snarf.rs |
SnarfHandler/SnarfInfoHandler (snfinfox.hxx) | Snarf (faithful on-disk layout: header + map table + flock data), SnarfStore (multi-snarf management with forwarding) |
persist/packer.rs |
SnarfPacker (packerx.hxx) | SnarfStorage — full StorageEngine with type-tagged serialization, forwarding on resize, destroy processing |
persist/counter.rs |
Counter/BatchCounter/SingleCounter (counterx.hxx) | Counter, BatchCounter (batch pre-allocation), SingleCounter |
persist/transaction.rs |
BEGIN_CONSISTENT/END_CONSISTENT macros | Transaction guard (RAII, auto-rollback on drop) |
edition/persistent.rs |
BeWork/BeEdition serialization (brange2x, brange3x) | PersistentWork, PersistentEdition with serde-based snapshot serialization |
Architecture decisions
-
Persistenttrait replaces Abraham — No inheritance. Rust trait withtype_tag()+to_bytes()for serialization,as_any()/as_any_mut()for downcasting. -
TypeRegistry replaces Cookbook/Recipe — Gold's category/recipe system maps to a
HashMap<&'static str, DeserializerFn>. Each concrete type registers its deserializer at startup. On-disk format:[tag_len: u16][tag: utf8][payload: bytes]. -
Edition snapshots, not O-tree serialization —
EditionSnapshotcaptures the flat entry list + default + domain. The O-tree is reconstructed on deserialization. This mirrors Gold's approach (the tree is an in-memory optimization; persistent storage is the logical content). -
Snarf layout is faithful — Same header + map table + flock data layout as Gold. Map cells use bit 25 for forwarded/forgotten flags (matching
Flag = 1 << 25). Flock data grows from the end of the snarf toward the front. -
Transaction is RAII —
Transaction::begin(engine)returns a guard.commit()ends the transaction. Drop without commit triggers rollback. New objects registered during a rolled-back transaction are unregistered. -
Rollback unregisters new objects — Both
InMemoryStorageandSnarfStoragetracknew_in_transaction/new_flocksand unregister them on rollback. Gold uses exception-based bomb cleanup; Rust uses Drop. -
Serde for serialization, but gated — The
Persistenttrait'sto_bytes()usesserde_jsoninternally. Theedition/persistentmodule is behind#[cfg(feature = "serde")]. The corepersistmodule works without serde (InMemoryStorage path).
Portability gaps from Gold
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| Urdi (raw disk I/O) | Memory-mapped file with atomic commit | SnarfStore operates on in-memory Vec |
Add file-backed Urdi (mmap or buffered I/O) |
| Turtle bootstrap | SimpleTurtle writes boot heaper + protocol info to first data snarf | Not yet; SnarfStorage bootstraps from empty | Add for production use |
| Purger (memory management) | Converts clean shepherds to stubs, evicts from RAM | Not needed yet; everything in-memory | Add when disk-backed |
| Stub system (stubble codegen) | Placement new replaces shepherd with lightweight stub | Rust enum variants instead; no codegen needed | Not applicable |
| Agenda (persistent work queue) | Deferred work processed at endConsistent | Not yet | Add for production |
| Cross-server ID export/import | IDs reference objects on other servers | Single-server IDs only | Add for federation |
| Full BeRangeElement hierarchy | BeDataHolder, BeIDHolder, BeLabel, BePlaceHolder with detector hooks | Simplified BeRangeElement in backend.rs | Expand in Phase 7 |
Enhancement ideas
-
File-backed Urdi: Replace
Vec<u8>snarfs with memory-mapped files. Usememmap2crate. Atomic commit via write-ahead log or shadow paging. -
Columnar flock format: For large Editions, store entries in a columnar format (sorted position array + sorted element array) instead of JSON. More compact, faster to reconstruct.
-
Parallel commit: Flocks in different snarfs can be written in parallel using rayon.
-
Compression: Snarf-level LZ4/Zstd compression for flocks that haven't changed recently.
Phase 7: Server API Surface
What was implemented
A new src/server/ module with 7 files:
| Module | C++ Original | Rust Implementation |
|---|---|---|
error.rs |
ExceptionRecord + problem codes | ServerError enum (NotAuthorized, NotFound, NotGrabbed, AlreadyGrabbed, etc.) |
detector.rs |
FeFillDetector, FeStatusDetector, FeRevisionDetector | Event enum + Detector trait + FnDetector callback wrapper |
club.rs |
FeClub (extends FeWork) | Club — wraps Work with signature_club and name |
keymaster.rs |
FeKeyMaster (nkernelx.hxx:307-434) | KeyMaster — holds login/actual authority (HashSet |
lock.rs |
Lock hierarchy + LockSmith hierarchy (nadminx.hxx) | Lock trait + BooLock, WallLock, ChallengeLock, MatchLock, MultiLock + LockSmith trait + smith implementations |
session.rs |
FeSession (nadminx.hxx:666-746) | Session — connection context with KeyMaster, connect time, login state |
server.rs |
FeServer (nkernelx.hxx:2161-2457) | Server — owns GrandMap, works, clubs, sessions, detectors; all operations are permission-checked methods on Server |
Key design decisions
-
All operations on Server — Instead of Gold's Fe* wrapper objects (FeWork, FeEdition), all operations are methods on the
Serverstruct. The Server takes aSessionIdparameter for authentication context. This avoids Rust's borrow checker issues with split borrows across wrapper objects. -
Club wraps Work — In Gold, Club IS a Work (C++ inheritance). In Rust,
Clubcontains aWorkinternally. The Club addssignature_club,name, and identity-group semantics. -
Lock trait + LockCredential enum — Gold uses virtual dispatch on Lock subclasses. Rust uses
Locktrait withtry_open(&LockCredential). TheLockCredentialenum dispatches:Boo,ChallengeResponse(Vec<u8>),Password(Vec<u8>),Named { name, credential }for MultiLock. -
Detector is callback-based —
Detectortrait withon_event(&Event).FnDetector<F>wraps closures. Events are an enum: WorkGrabbed, WorkReleased, WorkRevised, RangeFilled, ElementFilled, Done. -
System clubs created at startup —
Server::new()creates 4 system clubs (public, admin, access, empty) with appropriate ownership. Public club is boo-lockable; admin/access are more restricted. -
Grab/release is per-Work —
WorkStatetracks grabber (SessionId), status detectors, and revision detectors. Grab requires edit authority; release requires being the grabber. Disconnect auto-releases all grabs. -
Permission checks —
check_read_permission()andcheck_edit_permission()compare the session's KeyMaster authority against the Work's read/edit club. Public club grants open access.
Portability gaps from Gold
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| Wire protocol | Binary2 over TCP with PromiseManager | In-process only; no network | Add in Phase 8 |
| Promise/Future system | XuPromise with lazy evaluation | Synchronous Result |
Add async runtime |
| Fluid variables | Thread-local CurrentServer, CurrentAuthor, etc. | Explicit session_id parameters | Sufficient for single-threaded |
| Shepherd/Stub paging | In-place replacement with stub when memory pressure | All objects always in-memory | Add when disk-backed |
| Detectors over wire | CommDetector sends events to client | In-process only; fires locally | Add in wire protocol phase |
| XuServer static methods | Connect, login, get, assignID are class methods | Instance methods on Server | Close enough |
| Category-based RTTI | XuCategory hierarchy for wire type checking | Rust's trait + enum dispatch | No need without wire protocol |
| Request dispatch table | 470+ request handlers indexed by number | Direct method calls | Add in wire protocol phase |
Enhancement ideas
-
Async runtime: Make Server methods async. Use tokio channels for detector event dispatch. This would enable serving multiple concurrent sessions.
-
Arc
> : For multi-session access, wrap Server in Arc<Mutex<>> or use per-Work locks (RwLock per work) for finer-grained concurrency. -
Observer pattern with channels: Replace Detector trait with tokio::sync::broadcast channels. Clients subscribe to events they care about.
-
Permission caching: Cache permission check results and invalidate when clubs change. Currently checks are O(1) HashSet lookups but could be optimized for complex transitive authority.
-
Rate limiting: Add rate limiting per session to prevent abuse. Not needed in Gold (trusted local connections).
-
Audit log: Record all mutations (create, revise, grab, release) with session and timestamp. Gold doesn't have this explicitly but the revision history captures some of it.
Phase 8: WebSocket Transport Layer
What was implemented
A new src/server/transport/ module with 7 files, gated behind the server feature:
| Module | Purpose |
|---|---|
varint.rs |
LEB128 varint encoding/decoding (replaces Gold's humber encoding) |
protocol.rs |
Wire types: WireRequest, ResponseValue, WireEvent, OperationCode, ErrorCode, EditionPayload, WireFrame, SubscribeRequest, DetectorType, EventPayload |
codec.rs |
WireCodec trait + BinaryCodec (postcard) + JsonCodec (serde_json) — dual format support |
shared.rs |
AppState, ServerHandle (ArcSharedState |
channel.rs |
ChannelDetector — bridges sync Detector trait to async mpsc channel |
dispatch.rs |
dispatch() — maps WireRequest variants to Server method calls |
handler.rs |
axum WS upgrade handler, read/write loops, subscription management |
Plus src/bin/xudanu-server.rs — the standalone server binary entry point.
Key design decisions
-
Dual-format WireCodec: The
WireCodectrait abstracts serialization.BinaryCodecuses a 4-byte header + LEB128 varint + postcard binary encoding (compact).JsonCodecuses human-readable JSON text frames (easy for third-party integrations). Selection happens at WebSocket upgrade time via?format=jsonquery parameter. -
Arc
> : The synchronous Server is wrapped instd::sync::Mutex(not tokio's async Mutex). Since all Server methods are synchronous with no await points, std Mutex prevents lock-held-across-yield bugs and has lower overhead. -
Single writer task: All outgoing messages (responses, events, heartbeats) flow through a single
mpsc::unbounded_channelto a dedicated writer task that owns the WS sender. This avoids split-borrow issues and ensures ordered message delivery. -
ChannelDetector bridge: The sync
Detectortrait sends events through anmpsc::UnboundedSender<EventMessage>. The writer task receives and encodes them for the wire. -
Operation codes: Numeric for binary (
0x0303= WorkRevise), string for JSON ("work_revise"). Both map to the sameOperationCodeenum which drives the dispatch. -
EditionPayload: Editions are serialized as either
Text(String),Entries(Vec<(i64, RangeElement)>), orEmpty. This avoids serializing the O-tree directly — only the logical content crosses the wire. -
All new deps optional: tokio, axum, postcard, futures-util, tracing, tracing-subscriber are all behind the
serverfeature flag. The core crate remains WASM-friendly.
Binary frame layout
[1B version][1B msg_type][2B request_id BE][payload...]
REQUEST: [varint op_code][varint payload_len][postcard payload]
RESPONSE: [varint payload_len][postcard ResponseValue]
ERROR: [1B error_code][varint msg_len][UTF-8 message]
EVENT: [varint payload_len][postcard WireEvent]
SUBSCRIBE:[varint payload_len][postcard SubscribeRequest]
JSON frame layout
{"v":1,"type":"request","id":42,"op":"work_revise","payload":{"work_id":123,"edition":{...}}}
{"v":1,"type":"response","id":42,"value":{"type":"humber","value":2}}
{"v":1,"type":"error","id":42,"code":"not_grabbed","message":"work 123 not grabbed"}
{"v":1,"type":"event","id":7,"event":{"type":"work_revised","payload":{...}}}
Running the server
cargo run --features server --bin xudanu-server
# listens on 127.0.0.1:8080
cargo run --features server --bin xudanu-server 0.0.0.0:3000
# custom address
Connect with any WebSocket client:
- ws://127.0.0.1:8080/xudanu — binary protocol
- ws://127.0.0.1:8080/xudanu?format=json — JSON protocol
Test counts
- Without
serverfeature: 513 pass, 0 fail - With
serverfeature: 553 pass, 0 fail (+40 new transport tests)
Portability gaps from Gold
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| Custom binary protocol | Binary2 over raw TCP with humber encoding | WebSocket + postcard or JSON | More accessible; same semantic coverage |
| Promise/future system | XuPromise with lazy evaluation | Synchronous dispatch; Result |
Add async client SDK |
| Wire request numbering | 470+ numbered message handlers | ~40 OperationCode variants | Expand as needed |
| MultiLock auth over wire | Full auth protocol | Simplified; BooLock/MatchLock/ChallengeLock supported | Add complete auth flow |
| Subprotocol negotiation | WS subprotocol header | Query parameter ?format=json |
Add subprotocol support |
| TLS | Not in original (trusted network) | Not yet | Add with axum-server + rustls |
| Connection multiplexing | Single TCP connection for all ops | Single WS per client | Sufficient for Phase 8 |
Enhancement ideas
-
Integration tests with real WS client: Add
tokio-tungsteniteas a dev-dependency for end-to-end tests that spin up the server and exercise the full JSON and binary protocol. -
REST API: Add HTTP routes on the same axum Router for read-only queries (
GET /works/{id},GET /clubs). Useful for monitoring and tooling. -
TLS: Add
axum-serverwith rustls forwss://support. -
Subprotocol negotiation: Use
Sec-WebSocket-Protocolheader instead of query parameter for format selection. -
Auth tokens: After login, issue a session token that can be passed in WS upgrade headers for reconnection.
-
Heartbeat timeout: Close connections that don't send heartbeats within a configurable interval.
Phase 9: File-Backed Persistence
What was implemented
Four new modules under src/persist/ and server integration:
| Module | C++ Original | Rust Implementation |
|---|---|---|
persist/urdi.rs |
Urdi / UrdiView / SnarfHandle (urdix.hxx) |
UrdiFile — file-backed snarf read/write with guard records |
persist/file_storage.rs |
SnarfPacker + Turtle (packerx.hxx, turtlex.hxx) |
FileBackedStorage — combines SnarfStorage + UrdiFile with meta record |
snarf.rs extensions |
SnarfHandler::commitWrite() (snfinfox.hxx) |
Dirty tracking + flush_to_urdi/load_from_urdi on SnarfStore |
server/server.rs additions |
FromDiskPlan::connection() + DiskConnection (diskmanx.hxx) |
Server::to_snapshot()/from_snapshot() + checkpoint_to_file()/restore_from_file() |
bin/xudanu-server.rs |
BuildUrdiFile + server main (buildx.cxx) |
CLI with init/run/verify subcommands + graceful shutdown |
On-disk format (.xu file)
┌──────────────────────────────────────┐
│ File Header (32 bytes) │
│ magic: "XUD1" | version: 1 │
│ snarf_size, snarf_count │
│ stage_count, data_start │
│ header_crc (FNV-1a) │
├──────────────────────────────────────┤
│ Meta Snarf 0 (guard + data) │ MetaRecord: counters + flock index
│ Reserved Snarf 1 │ Future: Turtle root object
├──────────────────────────────────────┤
│ Data Snarf 0 (guard + data) │ Flock data
│ Data Snarf 1 │
│ ... │
├──────────────────────────────────────┤
│ Stage Area (stage_count snarfs) │ Future: double-buffer
└──────────────────────────────────────┘
Each snarf has a 32-byte guard record: magic, snarf_id, cycle, data_len, data_hash (FNV-1a), flags, guard_crc. Guards detect partial writes and corruption.
Server snapshot format
The server state is serialized as JSON to <data-dir>/server.json:
{
"grand_map_id_counter": 1005,
"session_counter": 2,
"operation_counter": 42,
"system_clubs": { "public_club": 1000, "admin_club": 1001, ... },
"works": [
{ "work": { "be_id": 1004, "current": { "entries": [...], ... }, ... }, "grabber": null, ... }
],
"clubs": [...],
"standalone_editions": [...]
}
Works and editions use the existing WorkSnapshot/EditionSnapshot types from edition/persistent.rs.
Key design decisions
-
Buffered I/O, not mmap —
UrdiFileusesstd::fs::Filewith seek+read+write. Simpler, more portable, no page fault surprises. mmap can be added later as an optimization. -
Guard records per snarf — 32-byte header with FNV-1a hash, magic
XNSR, snarf ID, cycle counter. Detects partial writes and corruption on each individual snarf. -
SnarfStore offset from UrdiFile — The first
DATA_SNARF_OFFSET(2) UrdiFile slots are reserved for system use (meta, turtle). SnarfStore snarfs are offset by this amount. This prevents the SnarfStore's info snarfs from colliding with the meta record. -
Meta record replaces Turtle — A
MetaRecordstruct stores the hash/token counters and flock index (FlockId → FlockLocation + FlockFlags). This serves the same role as Gold's Turtle for v1. A full Turtle object graph can be added later. -
Lazy loading in FileBackedStorage —
fetch()first checks the in-memory registry, then falls through to reading from the snarf store. Objects are deserialized on first access, not all at startup. This avoids requiring type registration before opening. -
Server snapshot is JSON, not flock-based — The server state is serialized as a single JSON file, not as individual flocks in the storage engine. This is simpler for v1. A future phase can migrate to per-object persistence using FileBackedStorage.
-
Grabs not preserved across restart — Sessions are ephemeral.
WorkState.grabberis always set toNoneduring restore. Any work that was grabbed at checkpoint time is released on restart. -
Rollback wipes snarf cells — Fixed a gap where
SnarfStorage::rollback()previously only removed metadata for new flocks. Now also wipes the actual snarf cell data, preventing leaked space. -
CLI subcommands —
xudanu-server init <dir>creates a new data directory,xudanu-server run [addr] [dir>starts the server with optional persistence,xudanu-server verify <dir>checks snapshot integrity. SIGINT triggers a checkpoint before exit.
Portability gaps from Gold
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| Staging area double-buffer | Write to staging snarfs, then atomically swap | Guard records provide crash detection but no double-buffer | Add staging area for stronger atomicity |
| Full Turtle object graph | Turtle holds boot category + cookbook + protocol | MetaRecord stores counters + flock index | Add Turtle for complex boot scenarios |
| Purger | Converts clean shepherds to stubs, evicts from RAM | Everything in-memory, no stubs | Add when memory-constrained |
| Agenda | Persistent work queue processed at endConsistent | Not yet | Add for production |
| Per-object disk persistence | Each Work/Edition is a separate flock on disk | Single JSON snapshot for entire server | Migrate to per-object persistence |
| Write-ahead log | Ongoing operations logged for crash recovery | Checkpoint-based (write+sync) | Add WAL for sub-second recovery |
| Urdi device partition | Raw disk device access | Regular file I/O | Not needed for modern systems |
| Endian detection | Multi-endian guard records | Little-endian only | Add if cross-platform needed |
Enhancement ideas for future phases
-
Per-object persistence: Store each Work/Edition as a separate flock in FileBackedStorage instead of a single JSON blob. Enables incremental checkpointing and lower memory usage on load.
-
Write-ahead log: Log each mutation before applying it. On crash, replay the WAL to recover. Sub-second recovery instead of losing the last checkpoint interval.
-
mmap optimization: Replace
std::fs::Filewithmemmap2for snarf access. Avoids explicit read/write syscalls for frequently accessed snarfs. -
Incremental checkpoint: Only write changed objects since last checkpoint. Reduces I/O for large stores with frequent checkpoints.
-
Compression: Snarf-level LZ4/Zstd compression for flocks that haven't changed recently.
-
File locking: Use
flock(2)orfcntl(F_SETLK)to prevent multiple server instances from opening the same data directory.
Test counts
| Phase | Tests | Notes |
|---|---|---|
| Phase 8 | 567 unit + 37 integration | WebSocket transport + audit |
| Phase 9 | 603 unit + 37 integration | File-backed persistence |
| Phase 10 | 612 unit + 47 integration | Admin API, handshake |
| Phase 11 | 612 unit + 53 integration | Client, links, transclusion, work listing |
| Total | 665 | All passing, 14 ignored |
Phase 10: Wire-Protocol Detector Events & Admin API
What was implemented
Five new modules and extensions across the server and transport layers:
| Module | C++ Original | Rust Implementation |
|---|---|---|
server/admin.rs |
FeAdminer (sysadmx.hxx) |
AdminState — server state management, accept/reject connections, shutdown, ID grants |
transport/handler.rs additions |
BootPlan/SetCommProtocol (bootplnx.hxx, negoci8x.hxx) |
HandshakeResponse — version negotiation at WebSocket connection start |
transport/handler.rs subscribe tracking |
PromiseManager subscription tracking (promanx.hxx) |
subscriptions HashMap + SUBSCRIPTION_COUNTER — proper subscription ID tracking |
transport/channel.rs additions |
CommXxxDetector + DetectorEvent hierarchy (comdtctr.hxx) |
subscription_id on EventMessage — events carry subscription IDs |
transport/protocol.rs additions |
handlrsx.hxx dispatch table |
OperationCode admin ops, ResponseValue admin types, HandshakeRequest/Response, ServerInfoPayload, SessionInfoPayload, GrantPayload |
New protocol operations
| Op Code | Operation | C++ Equivalent |
|---|---|---|
| 0x0501 | AdminAcceptConnections |
FeAdminer::acceptConnections |
| 0x0502 | AdminIsAcceptingConnections |
FeAdminer::isAcceptingConnections |
| 0x0503 | AdminActiveSessions |
FeAdminer::activeSessions |
| 0x0504 | AdminShutdown |
FeAdminer::shutdown |
| 0x0505 | AdminGrant |
FeAdminer::grant |
| 0x0506 | AdminRevokeGrant |
New (Gold had grants() only) |
| 0x0507 | AdminGrants |
FeAdminer::grants |
| 0x0508 | AdminServerInfo |
New (combines stats + state) |
| 0x0601 | ServerStats |
New (non-admin version of server info) |
Handshake protocol
At connection start, the server sends a HandshakeResponse:
{
"type": "handshake",
"v": 2,
"payload": {
"server_version": 2,
"negotiated_version": 2,
"server_id": "xudanu-0.1.0",
"server_capabilities": ["json", "binary", "detector_events", "admin"]
}
}
The client's requested version is passed as ?version=N query parameter. If the version is outside [MIN_SUPPORTED_VERSION, PROTOCOL_VERSION], an error is sent and the connection is closed.
Admin authority model
Admin operations require the session to hold authority for either the admin_club or access_club. The admin club's read_club is set to the public club, so any logged-in session can escalate to admin authority by:
- Looking up the admin club ID via
club_id_by_name("admin") - Logging in via
session_loginwith that club ID (returns a BooLock since read_club is public) - Authenticating via
session_authenticatewithBoocredential
This mirrors Gold's FeAdminer::make() which requires a KeyMaster with System Admin authority.
Connection gating
When AdminState.accepting_connections is false or shutdown_requested is true:
- New WebSocket connections are immediately closed with not_accepting_connections error
- Existing connections check is_shutdown_requested() on each request and close if true
ID grants
The AdminState tracks IdGrant records mapping (club_id, XnRegion). These grants give clubs authority to assign global IDs in specific regions. Multiple grants per club are supported (additive, not replacing).
Subscription tracking
The handler now tracks subscriptions in a HashMap<u16, (DetectorType, BeId)>. Each subscribe operation returns a unique subscription ID via an AtomicU16 counter. Events carry the subscription ID so clients can correlate them.
Protocol version bump
PROTOCOL_VERSION is now 0x02 (was 0x01). MIN_SUPPORTED_VERSION is 0x01. The handshake negotiates the version.
Portability gaps from Gold
| Gap | Gold Feature | Rust Status | Plan |
|---|---|---|---|
| FeArchiver | Archive/restore Works to secondary storage | Not yet | Add in Phase 13 |
| GateLockSmith | Configurable lock for invalid logins | Not yet | Add in Phase 11 |
| Full detector wire protocol | CommFillDetector, CommRevisionDetector, etc. | Subscription tracking + event push | Add per-type filtering |
| Protocol negotiation | SetCommProtocol/SetDiskProtocol at boot | Version query param + handshake | Sufficient for v1 |
| Execute commands | FeAdminer::execute(PrimIntArray) | Not yet | Add for production |
| Detector event queuing | PromiseManager queues events between requests | Events pushed immediately | Add queuing for ordered delivery |
Enhancement ideas for future phases
-
Per-type detector subscriptions: Allow subscribing to specific event types (grab, release, revise) on a work, not just status/revision/fill.
-
Detector event queuing: Queue events between requests (like Gold's PromiseManager) to ensure ordered delivery.
-
Admin command execution: Add
admin_executefor batch configuration commands. -
Gate lock configuration: Allow setting a custom LockSmith for invalid login attempts via the wire protocol.
-
WebSocket subprotocol negotiation: Use axum's subprotocol support instead of query parameter for format selection.
Phase 11: Make It Usable — Client, Links, Transclusion, Work Listing
What was implemented
Six work items completed, making the system demonstrable end-to-end:
| Item | Description |
|---|---|
| Work listing | WorkList, WorkListByOwner operations + list_works() server methods |
| Link storage & wire ops | LinkCreate, LinkGet, LinkUpdate, LinkDelete, LinkListForWork |
| Transclusion queries | FindTranscluders, FindWorksForContent — search for content across works |
| CLI client | xudanu-cli binary with REPL and subcommand modes |
| Web UI | Served at / — browser-based client for interactive use |
| Demo script | examples/demo.sh — automated walkthrough |
New protocol operations
| Op Code | Operation | Notes |
|---|---|---|
| 0x030E | WorkList |
List all works with owner, revision count, grab status |
| 0x030F | WorkListByOwner |
Filter works by owner BeId |
| 0x0701 | LinkCreate |
Create bidirectional link between two works |
| 0x0702 | LinkGet |
Get link details (origin, destination, refs) |
| 0x0703 | LinkUpdate |
Update link endpoint references |
| 0x0704 | LinkDelete |
Remove a link |
| 0x0705 | LinkListForWork |
List all links involving a work |
| 0x0801 | FindTranscluders |
Find editions/works that contain given content |
| 0x0802 | FindWorksForContent |
Find works containing given content by BeId |
CLI client (xudanu-cli)
Two modes:
- Subcommand: xudanu-cli <url> create-work "hello" — one-shot, for scripting
- REPL: xudanu-cli <url> repl — interactive session
Commands: login, login-admin, create-work, get-work, list-works, grab, revise, release, history, fetch-revision, create-link, get-link, list-links, delete-link, find-content, club-create, club-list, info, quit
Web UI
Served at the root URL (/). Dark-themed browser client that auto-connects via WebSocket, logs in, and provides a command input. Same commands as the CLI.
Link model
Links are stored in Server.links: HashMap<BeId, LinkState> where LinkState holds:
- origin: BeId — source work
- destination: BeId — target work
- link: HyperLink — with LeftEnd and RightEnd HyperRefs
Links are persisted in server snapshots as LinkSnapshot { link_id, origin, destination }.
Transclusion queries
The server currently implements content search by scanning all work editions linearly. For each work/edition, it checks if any entry's element matches the target BeId. This is O(n) but sufficient for v1.
Future phases will integrate the full BackfollowEngine with TransclusionIndex for O(log n) lookups via the canopy/HTree structures.
Demo
cd src-rust
cargo build --features server --bins
./examples/demo.sh
This creates documents, edits them, creates links, lists works, creates clubs, and shows server info.
Test counts
| Phase | Tests | Notes |
|---|---|---|
| Phase 10 | 612 unit + 47 integration | Admin API, handshake, detector subscription |
| Phase 11 | 612 unit + 53 integration | +6 integration (work-list, links, transclusion) |
Phase 14: Space/CoordinateSpace Types
What was implemented
A new src/space/ module with 8 files implementing Gold's CoordinateSpace hierarchy:
| Module | C++ Original | Rust Implementation |
|---|---|---|
traits.rs |
Space/Position/XnRegion/Dsp/OrderSpec (spacex.hxx) |
Space/Position/Region/Dsp/OrderSpec traits with circular associated types |
integer.rs |
IntegerSpace/IntegerPos/IntegerRegion (integerx.hxx) |
IntegerSpace, IntegerPos, IntegerRegion (wraps XnRegion), IntegerDsp, IntegerAscending/IntegerDescending |
real.rs |
RealSpace/RealPos/RealRegion (realx.hxx) |
RealSpace, RealPos, RealRegion, RealDsp, RealAscending/RealDescending |
sequence.rs |
SequenceSpace/Sequence/SequenceRegion/SequenceMapping (sequencex.hxx) |
SequenceSpace, Sequence (VecSequencePos, SequenceRegion, SequenceDsp, SequenceAscending/SequenceDescending |
cross.rs |
CrossSpace/Tuple/CrossRegion/CrossMapping (crossx.hxx) |
CrossSpace2<A,B>, Tuple2<A,B>, CrossRegion2<R1,R2>, CrossDsp2<D1,D2>, CrossOrder2 |
filter.rs |
FilterSpace/Filter/FilterPosition/Joint/RegionDelta (filterx.hxx) |
FilterSpace, FilterPosition, Filter enum (Full/Empty/Subset/Superset/Intersection/NotSubset/NotSuperset/And/Or), Joint, RegionDelta |
mapping.rs |
SimpleMapping/CompositeMapping/ConstantMapping/EmptyMapping (spacep.hxx) |
SimpleMapping<S>, CompositeMapping<S>, ConstantMapping<S>, EmptyMapping<S> with MappingSpace trait |
order.rs |
ReverseOrder/ChainedOrder (spacep.hxx) |
ReverseOrder<P>, ChainedOrder<P> |
Key design decisions
-
Circular associated types:
Position<Region=R>andRegion<Position=P>whereP::Region=RandR::Position=P. This preserves the type-level relationship between positions, regions, and displacements without inheritance. -
IntegerRegion wraps XnRegion: The existing
XnRegiontransition-array implementation is reused as the storage forIntegerRegion. This avoids duplication and maintains compatibility with the Edition layer. -
CrossSpace2 is generic over two axes:
CrossSpace2<A: Space, B: Space>avoids heterogeneous type erasure for the common 2D case. Higher dimensions can be supported by nesting (e.g.,CrossSpace2<CrossSpace2<X, Y>, Z>). -
FilterSpace is standalone (does NOT implement the Space trait): Filter positions are regions from another space, which can't be captured by the circular associated type pattern. Uses tag-based
Filterenum with u64 tags instead of actual region comparison. -
Mapping types use a separate
MappingSpacetrait: Because the mainRegion/Dsptraits' method names (contains,is_empty) conflict with the mapping layer's own traits when both are in scope. Tests use fully-qualifiedRegion::contains(...)calls. -
ReverseOrder::compare() reverses ordering by swapping args AND reversing the result via
.reverse(). Manual Debug/PartialEq/Eq/Hash implementations sinceBox<dyn OrderSpec>doesn't impl Clone/Hash/Eq.
RealRegion bug: edge-flag model vs simple flip model
Root cause: The original implementation used an edge-flag model where each transition stored both a value: f64 and an included: bool flag. The included flag indicated whether the point at exactly that value was inside the region. This works for simple (non-merged) intervals but breaks when the merge algorithm produces same-value transition pairs.
How the bug manifested: The merge_real_regions function produces same-value pairs like [(3.0, false), (3.0, true)] to represent the transition "not inside at 3.0, but inside just past 3.0". The contains_value method used flip counting: walk transitions with value <= v, flipping the inside state for each one. Two transitions at the same value flip twice (net zero), so contains_value(4.0) would see: flip at 3.0 -> true, flip at 3.0 -> false (back to false), then 5.0 > 4.0 so return false. But 4.0 IS in the region (3.0, 5.0).
The fundamental tension: The edge-flag model conflates two pieces of information:
- The "at" state (the included flag at exact boundary values)
- The "between" state (flipped by each transition)
For simple intervals these are aligned. For merged intervals they conflict: same-value pairs represent a real state change ("not at 3.0, but yes past 3.0") that flip counting treats as zero change (two flips cancel).
Failed attempts:
1. Counting distinct-value groups instead of individual transitions -- works for point regions but breaks for merged intervals
2. Using even/odd group sizes -- ambiguous: a group of 2 could mean 1 flip (false->true) or 0 flips (true->false->true->false)
3. Linear scan with per-edge flips -- same double-flip problem
4. normalize_edges that removes even-count groups -- removes the pair entirely, losing the real state change
Solution adopted: Drop the included flag entirely. Use a simple flip model where transitions are just Vec<f64> values (like XnRegion/IntegerRegion). All intervals are internally half-open [start, stop). Open/closed boundary distinctions from the API are mapped to half-open at construction time:
- interval(1.0, 5.0, false, false) -> effective [1.0+e, 5.0) where e = 1.0.next_up() - 1.0
- interval(1.0, 5.0, true, true) -> effective [1.0, 5.0+e)
- above(10.0, false) -> transition at 10.0.next_up() instead of 10.0
- below(10.0, true) -> transition at 10.0.next_up() instead of 10.0
This makes contains_value a simple flip count: count transitions with value <= v. The merge becomes a standard sorted-merge of transition arrays with no edge-flag logic, no normalize_edges, and no same-value ambiguity.
Trade-off: We lose the ability to distinguish [3.0, 5.0] from [3.0, 5.0) at exact float values. For f64, v.next_up() - v is on the order of 10^-16, so this distinction is negligible for practical use. All existing tests pass with the half-open mapping.
Impact on the codebase:
- RealEdge struct removed entirely (was only used internally)
- RealRegion.transitions changed from Vec<RealEdge> to Vec<f64>
- contains_value simplified from 17 lines to 7 lines
- merge_real_regions simplified from 60 lines to 30 lines
- complement simplified from 8 lines to 4 lines (just flip starts_inside)
- normalize_edges, count_distinct_flips, count_net_flips helper functions all removed
- Manual PartialEq/Eq/Hash impls for RealRegion (since Vec<f64> doesn't impl them)
SequenceDsp inverse bug: shift not adjusted on leading-zero trim
Root cause: Sequence::from_numbers_with_shift trimmed leading zeros from the numbers array but didn't adjust the shift field, so the first non-zero value ended up at the wrong index.
How the bug manifested: SequenceDsp::inverse() computed inv_translation = Sequence::zero().minus(&self.translation).shifted(-self.shift). The minus operation produced [0, 0, 5] at shift=-2, meaning 5 is at index 0. But from_numbers_with_shift([0, 0, 5], -2) trimmed the leading zeros to [5] while keeping shift=-2, incorrectly placing 5 at index -2 instead of index 0. This caused inv.of(d.of(x)) to produce { shift: -2, numbers: [5] } instead of { shift: 0, numbers: [5] }.
The fix was one line: increment shift for each leading zero removed:
while numbers.first() == Some(&0) {
numbers.remove(0);
shift += 1; // was missing
}
Additional change: Sequence::from_numbers() was decoupled from from_numbers_with_shift because they have different semantics:
- from_numbers([0, 0, 3, 7]) -> trim and reindex to start at 0 (user-facing, padding removal)
- from_numbers_with_shift(result, start) -> trim and adjust shift to preserve actual indices (internal, for plus/minus)
The inverse formula itself was correct: inverse_transform(seq) = seq.minus(&self.translation).shifted(-self.shift) undoes the forward transform(seq) = seq.shifted(self.shift).plus(&self.translation) correctly. The bug was entirely in the index bookkeeping of from_numbers_with_shift.
Enhancement ideas for future phases
-
yrs/CRDT as transport layer: The Edition model maps naturally to yrs
DocwithTextsequences. An Edition could be materialized into a yrs document for real-time sync, while maintaining the Gold partial ordering for conflict preservation. -
Compressed transition arrays: For very large regions, run-length encoding with 32-bit deltas could reduce memory usage.
-
Parallel region operations:
merge_transitionscould be parallelized for large regions using rayon. -
RealRegion epsilon-aware API: Expose the half-open mapping to callers who need exact boundary control. Could add
interval_half_open(start, stop)constructor. -
SequenceRegion: drop edge-flag model: SequenceRegion uses the same edge-flag pattern as the old RealRegion and may have the same merge/contains bug. Consider adopting the same simple flip model.
Test counts
| Phase | Tests | Notes |
|---|---|---|
| Phase 13 | 612 unit + 54 integration | Arc |
| Phase 14 (space types) | 686 unit + 54 integration | +74 new space tests |
| Total | 740 | All passing, 11 ignored |