Transclusion Engine

How Xudanu detects, indexes, and queries content reuse across documents — the core intellectual property inherited from Udanax-Gold, enhanced with BLAKE3 fingerprinting, the Bert Canopy tree, Recorder pub/sub, and the Hoist propagation algorithm.

1. Overview

Transclusion is Ted Nelson's foundational concept: content is linked, not copied. When the same text appears in multiple documents, the system should know they are the same content, not independent copies.

Xudanu's transclusion engine is a five-layer pipeline:

1. TransclusionIndex — maps BLAKE3 content fingerprints to the editions containing them
2. Bert Canopy — a hierarchical tree that indexes content properties for efficient querying
3. Recorder System — persistent queries ("recordings") that accumulate results over time
4. RecorderHoister — propagates recorder state incrementally up the canopy tree
5. BackfollowEngine — given an edition, finds all other editions/works sharing content

Figure 1: The five-layer transclusion pipeline. Content fingerprints flow left-to-right; queries flow top-to-bottom.

2. TransclusionIndex — BLAKE3 Content Registry

The TransclusionIndex is the foundation. Every RangeElement in every Edition has a 256-bit BLAKE3 content fingerprint. The index is a HashMap<[u8;32], Vec<EditionId>> that maps each fingerprint to the editions containing it.

TrailBlazer — content-based queries

The TrailBlazer struct wraps the index and provides query methods:

• find_editions_with_content(fingerprint) — all editions containing a specific element
• find_works_with_content(fingerprint) — all works (across all editions) containing the element
• find_shared_content(edition_a, edition_b) — the XnRegion of content shared between two editions

TransclusionQuery / TransclusionResult

A TransclusionQuery specifies what to search for (a specific edition, or all works). The TransclusionResult returns matched works with their fossil_id (from the Recorder), edition_be_id, and whether the match is direct or transitive.

3. Bert Canopy — Hierarchical Property Tree

The Bert Canopy is a binary tree structure where each node ("crum") holds computed property flags for the editions beneath it. This is the key data structure that makes transclusion queries efficient across large corpora.

Figure 2: The Bert Canopy tree. Leaf nodes hold per-edition property flags. Interior nodes hold the union of their children's flags. The PropFinder can prune entire subtrees when flags don't match.

CanopyCrumData

Each crum contains:

BertProp — property flags

BertProp is a bitmask computed from an Edition's content. Each bit indicates the presence of a particular content type (text elements, edition references, work references, etc.). The canopy tree propagates these flags upward via compute_join() so a parent crum's flags are the bitwise OR of all descendants.

During a transclusion query, the PropFinder checks a crum's flags before descending. If the sought property bit is not set, the entire subtree is pruned — turning a full-corpus scan into a logarithmic search.

Two canopy types

4. Recorder System — Publish/Subscribe for Content

Recorders are the Udanax-Gold equivalent of persistent database queries. A recorder is "planted" on specific content, and when matching content appears in new editions, the recorder "fires" and accumulates a result (a Fossil).

Key types

Recorder lifecycle

1. Plant: A RecorderQuery is created and installed at a canopy crum. A new Fossil is allocated.
2. Initial match: A Matcher agenda item runs the query against the current transclusion index, populating the fossil with existing matches.
3. Watch: The recorder sits in the canopy tree, listening for new content via the sensor canopy.
4. Trigger: When a new edition is registered and its content matches, a RecorderTrigger fires, adding a new RecordedResult to the fossil.
5. Extinguish: When no longer needed, the fossil is marked extinct and its recorders are removed from the canopy.

5. Recorder Hoist — Incremental Propagation

When recorders are installed at leaf nodes of the canopy, they need to be visible to ancestors for efficient querying. The RecorderHoister is an AgendaItem that incrementally propagates recorders upward through the tree.

Two phases

Why incremental?

A naive approach would recompute the entire canopy on every edition registration. The hoister instead does the minimum work: it only walks the path from the affected leaf to the root, touching O(log n) nodes. For a corpus of 10,000 editions, this means ~13 nodes instead of 10,000.

6. BackfollowEngine — Cross-Document Queries

The BackfollowEngine is the top-level query engine. Given an edition or work, it finds all other works that share content with it.

How it works

1. Walk the edition's elements and collect their BLAKE3 fingerprints
2. For each fingerprint, look up the TransclusionIndex to find other editions containing the same content
3. Use the PropFinder to walk the Bert Canopy, pruning subtrees whose flags don't match the query
4. For matching canopy leaves, check installed recorders and their fossils
5. Aggregate results into TransclusionResult structs with work IDs, edition IDs, and match types (direct or transitive)

Query types

7. End-to-End Flow

Here's what happens when a user pastes a transclusion (copies content from Document A into Document B):

Figure 3: End-to-end transclusion flow when content is pasted between documents.

8. Modern Enhancements Over Udanax-Gold

9. Performance Characteristics