Peers: documents between nodes

crdt::peer — pairwise document exchange with no hub, and per-document access the library checks.

5 min read

crdt::sync is hub-and-spoke: replicas attach to one canonical document on a host. That is half of local-first. The other half (ADR 0006, “every node is a peer”) is two nodes that each hold a DocRegistry moving a document between themselves, with the always-on host stopped or never involved. crdt::peer (features/crdt/crdt/src/peer.rs) is that half. It was lifted from Task’s vault-doc::peer and doc_peer.rs, with the vault taken out.

The pieces#

Item	What it is
`DocPeer`	The `#[vox::service]` lane a node serves to the nodes it admits. `list` returns every document the node holds, each with an opaque digest; `sync` is the ordinary `DocSync` session, addressed by a `DocAt { doc_id, name }` so the serving node can resolve (and register) a document it has never been asked for.
`DocPeerHost<C>`	Serves `DocPeer` over a `DocRegistry` and a `DocCatalog`. `DocPeerHost::new(registry, catalog, admit_peer)`: `admit_peer` receives the caller’s iroh endpoint id.
`DocCatalog`	The app’s index of documents: `held()` and `resolve(&DocAt)`. `OpenDocs(registry)` is the stock one (whatever the registry has open, digested by `vv_digest`), enough for tests and for apps whose doc ids are the address.
`SyncLane`	Something `exchange` can run a sync session through. `PeerLane { client, name }` is a dialled peer; `SelfLane::open(registry, principal)` is this node’s own registry behind a gate that presents a fixed principal, so the registry’s authorizer sees who is pulling.
`exchange(doc_id, local, remote)`	One document, both directions, through an in-memory replica. Local history goes up to the peer and the peer’s comes down to the local registry, each through a normal client session, so each registry persists, relays to whoever has the document open, and applies its own access rules exactly as for a device’s editor. Nothing about the pipe is remembered. Returns `Exchanged` or a `SessionEnd`.
`pull(local, peer, ours, per_doc)`	One sweep against one peer: `list`, skip documents whose digest matches `ours`, `exchange` the rest, each bounded by `per_doc`. Returns a `PullReport { listed, exchanged, pulled, foreign, failed }`.

Both nodes sweep each other, so either pull alone converges two documents. A pull carries both directions anyway, which is what lets an edit reach a peer that cannot dial back (behind a NAT, asleep). Relayed through a third node, an edit reaches a node it was never directly exchanged with: A to B, then B to C, converges C with A while A is offline.

Who may exchange#

Two checks, both against the principal the router’s PermissionsGate resolved for the call (architect::permissions_gate::caller()):

The node. DocPeerHost serves only Principal::Host callers that its admit_peer check accepts (an org’s own devices, by endpoint id). Everyone else, a signed-in user included, is refused on both list and sync. The lane is for nodes.
The document. DocPeer::sync attaches through the registry’s DocRegistry::with_authorizer, which sees the same caller and decides per document. A refusal is SyncError::UnknownDoc, the same answer as a document that does not exist, so it leaks nothing about which ids are real.

Access is therefore checked by every node, in the library, per document. Nothing trusts a machine because it answered.

A history that shares nothing with the served document (two nodes that each created “the same” document separately) is refused with FOREIGN_LINEAGE instead of merged, which would put every line in twice (is_foreign_lineage, check_lineage).

Wiring a node#

The registry decides who may open what; the host decides which nodes may dial; the gate in front of the router supplies the caller:

use crdt::peer::{DocPeerHost, OpenDocs, doc_peer_service_descriptor, DocPeerDispatcher};

let registry = DocRegistry::new(open_or_create)
    .with_authorizer(|who, doc_id| match who {
        Some(Principal::Host { endpoint }) => my_devices.contains(endpoint) && shared(doc_id),
        Some(Principal::Service { name }) => name == "self",
        _ => false,
    });

let host = DocPeerHost::new(registry.clone(), OpenDocs(registry.clone()), |endpoint| {
    my_devices.contains(endpoint)
});
let router = LayerRouter::new()
    .with(doc_peer_service_descriptor(), DocPeerDispatcher::new(host));
// Mount `router` behind a PermissionsGate (permit the DocPeer methods) on
// the node's EngineHost; peers dial it over iroh by endpoint id.

And the sweep a node runs against each peer it knows:

let local = SelfLane::open(registry.clone(), Principal::Service { name: "self".into() }).await?;
let ours = digests(&registry).await;            // doc id -> vv_digest, from OpenDocs::held
let report = crdt::peer::pull(&local, &peer_client, &ours, Duration::from_secs(10)).await?;

Serving a node: devices and links#

crdt::p2p (features/crdt/crdt/src/p2p.rs) is the wiring above, done once: who a node serves, and the router each connection gets.

Item	What it is
`Admission`	The devices the node lets in, by endpoint id (`architect_devices`, Devices): the owner’s other devices (kept current by the account pass, `architect_devices::account::pass`) open any document and sweep; a paired device opens the documents its scopes name. Read on every call, so a revoked device is refused on its next call, and its live sessions end.
`Shares`	The live links the node has handed out. `share(endpoint, doc_id)` mints a key and returns a `DocLink`; `revoke` and `revoke_doc` close them.
`NodeAccess`	Both, answering the three questions: who the caller is (`caller`: the link’s `Principal::Guest` when the call presents a live key, otherwise `Principal::Host` of the proved endpoint), whether they may call at all (it is the gate’s `PermissionEngine`: admitted devices and live guests only, and anyone asking to pair when it takes pairings), and which documents they may open (`admits`, for `DocRegistry::with_authorizer`: an account device any, a paired device its scopes’, a guest the link’s). `NodeAccess::pairing(pairings)` also serves `DevicePairing`.
`node_router(registry, catalog, access, remote)`	`DocSync` and `DocPeer` over the registry, behind that gate, for the connection from `remote`.

architect_devices::serve (iroh_link::serve_peers_until) builds one per connection from the endpoint id the QUIC handshake proved, which is what makes the id an identity rather than a claim (serve_router, one router for every caller, cannot), and closes the connection when its device is revoked:

let access = NodeAccess::new(Admission::new(), Shares::new());
let registry = DocRegistry::new(open_or_create)
    .with_authorizer({ let access = access.clone(); move |who, id| access.admits(who, id) });
architect_devices::serve(&endpoint, access.devices.clone(), move |remote| {
    node_router(registry.clone(), OpenDocs(registry.clone()), access.clone(), &remote.to_string())
})
.await;

A link (crdt::link, features/crdt/crdt/src/link.rs) is fts-doc:<endpoint id>/<doc id>?key=<key>: the node to dial, the document, and the credential. The joiner parses it (DocLink, wasm-clean), dials the endpoint, and runs an ordinary SyncedDoc over a DocSyncClient carrying LinkKey::new(&link), which presents Bearer link:<key> on each call. The guest edits that one document live, until the node revokes the link; it never reaches DocPeer.

Devices nearby. Endpoints bound with PeerLookup::Lan or N0AndLan advertise as FTS nodes, <endpoint id>._fts._udp.local. (iroh_link::LAN_SERVICE; iroh’s QUIC is UDP, hence _udp), with the name iroh_link::advertise_name gives them. iroh_link::LanBrowser lists them (LanEvent::Found { endpoint, name }, Lost); a found id is dialled bare and its address comes from the network.

Peers find each other through iroh: n0 relays and DNS by default, or PeerLookup::N0AndLan / PeerLookup::Lan (feature iroh-lan) to dial by bare endpoint id on a LAN with the internet down (HTTP and iroh). A directory such as fts-auth must never be the only way to find a peer.

Tests#

features/crdt/crdt/tests/peer.rs is the conformance suite and the pattern to copy for an app’s own peer tests (three in-process nodes, each behind its own gate, dialled over vox::memory_link_pair):

Test	Proves
`three_replicas_converge_without_a_hub`	A to B, B to C with A offline and no hub: C converges with A
`an_unadmitted_host_is_refused`	a `Principal::Host` that `admit_peer` rejects gets nothing
`the_registry_authorizer_sees_the_gates_caller`	per-document access sees the real caller, not `None`
`a_foreign_lineage_is_refused`	two unrelated documents with one id are not merged
`any_edit_order_converges`	proptest: random edit orders across replicas converge

features/crdt/crdt/tests/p2p.rs runs the same over real iroh endpoints, each node served by serve_peers + node_router, with no server:

Test	Proves
`a_link_joins_a_live_document_and_edits_flow_both_ways`	a guest holding only the link’s text joins, and edits flow both ways live
`a_link_opens_its_document_only_and_not_once_revoked`	a link opens nothing else, not `DocPeer`, and nothing once revoked
`registered_devices_edit_live_with_no_server`	two devices from one `DeviceDirectory` edit live, and a sweep brings a document not open
`an_unregistered_device_is_refused`	an endpoint off the account gets nothing; a device removed from it is refused on its next call
`devices_on_one_network_find_each_other`	the LAN browser finds the other device by name, and a bare-id dial syncs (needs multicast)

Limits#

Fan-out beyond pairs is sweeps (or a host); there is no gossip.
The digest is a version-vector digest: it skips unchanged documents, it is not an integrity check.
crdt::sync (the hub) and crdt::peer (pairs) share one registry, so a node can be a host for its devices and a peer to another node at once.