The R5N Distributed Hash Table

Introduction FIXME: Here we should also cite and discuss RELOAD (https://datatracker.ietf.org/doc/html/rfc6940) and establish why we need this spec and are not a "Topology plugin" in RELOAD. The argumentation revolves around the trust model (openness) and security aspects (path signatures). Distributed Hash Tables (DHTs) are a key data structure for the construction of completely decentralized applications. DHTs are important because they generally provide a robust and efficient means to distribute the storage and retrieval of key-value pairs. While already provides a peer-to-peer (P2P) signaling protocol with extensible routing and topology mechanisms, it also relies on strict admission control through the use of either centralized enrollment servers or pre-shared keys. Modern decentralized applications require a more open system that enables ad-hoc participation and other means to prevent common attacks on P2P overlays. This document contains the technical specification of the R5N DHT , a secure DHT routing algorithm and data structure for decentralized applications. R5N is an open P2P overlay routing mechanism which supports ad-hoc participation and security properties including support for topologies in restricted-route environments and path signatures. This document defines the normative wire format of peer-to-peer messages, routing algorithms, cryptographic routines and security considerations for use by implementors.

Requirements Notation The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Structure of This Document

Section X defines...

Terminology

Peer:: A host that is participating in the overlay. Peers are responsible for holding some portion of the data that has been stored in the overlay, and they are responsible for routing messages on behalf of other hosts as needed by the Routing Algorithm.
Peer Key:: The Peer Key is the identifier used on the Overlay to address a peer.
Peer ID:: The Peer ID is the identity which is used to authenticate a peer in the underlay. The Peer Address is derived from the Peer ID.
Neighbour:: A neighbour is a node which is directly connected to our node.
Block:: An object or group of objects stored in the DHT.
Block-Type:: A unique 32-bit value identifying a block type. Block-Types are either private or allocated by GANA (see ).
Block Storage: The Block Storage component is used to persist and manage data by nodes. It includes logic for quotas, caching stragegies and data validation.
Responsible Peer:: The peer N that is responsible for a specific resource K, as defined by the SelectClosestNode(K, N) algorithm (see .
Applications: Applications are components which directly use the DHT overlay interfaces. Possible applications include the GNU Name System or the CADET transport system .
Overlay Interface: The Overlay Interface exposes the core operations of the DHT overlay to applications. This includes querying and retrieving data from the DHT.
Message Processing: The Message Processing component processes requests from and responses to applications as well as messages from the underlay network.
Routing: The Routing component includes the routing table as well as routing and node selection logic. It facilitates the R5N routing algorithm with required data structures and algorithms.
Underlay Interface: The DHT Underlay Interface is an abstraction layer on top of the supported links of a node. Nodes may be linked by a variety of different transports, including "classical" protocols such as TCP, UDP and TLS or advanced protocols such as GNUnet, L2P or Tor.

Architecture R5N is an overlay network with a pluggable transport layer. The following figure shows the R5N architecture.

Overlay In the DHT overlay, a node is addressable by its Node Address. The Node Address is a SHA-512 hash of the Node ID. The Node ID is the public key of the corresponding Ed25519 node private key. An implementation of this specification commonly exposes the two API procedures "GET" and "PUT". The following are non-normative examples of such APIs and their behaviour are detailed in order to give implementers a fuller picture of the protocol.

The GET procedure A basic GET procedure may be exposed as: GET(Query-Key) -> Results as List The procedure requires at least a Query-Key to initiate a lookup:

QueryKey:: the key to look for in the DHT.

The procedure may allow a set of optional parameters in order to control or modify the query:

Block-Type:: the type of block to look for.
Replication-Level:: An integer which controls how many nearest peers the request should reach.
Route-Options:: Flags that are used in order to indicate certain processing requirements for messages. Any combination of options as defined in may be specified.
Extended-Query:: is extended query medatadata which may be required depending on the respective Block-Type. A Block-Type must define if the XQuery can or must be used and what the specific format of its contents should be. See also .
Result-Filter:: allows to indicate results which are not relevant anymore to the caller (see ).

If the procedure is implemented synchronuously, it may return a list of results. If it is implemented asynchronuously, it may return individual results. A single result commonly consists of:

Block-Type:: the type of block in the result.
Block-Data:: the block payload. Contents are defined by the Block-Type.
Expiration:: the duration of validity of the result.
Key:: the key of the result. This may be different from the Query-Key, for example if a flag for approximate matches was set.
GET-Path:: is a signed path the query took through the network.
PUT-Path:: is a signed path the PUT-Request of this data took through the network.

The PUT procedure A PUT procedure may be exposed as: PUT(Key, Block) The procedure takes at least two parameters:

Key:: the key under which to store the block.
Block:: the block to store.

The procedure may allow a set of optional parameters in order to control or modify the query:

Block-Type:: the type of the block to store.
Replication-Level:: An integer which controls how many nearest peers the request should reach.
Route-Options:: Flags that are used in order to indicate certain processing requirements for messages. Any combination of options as defined in may be specified.
Block-Expiration: is the requested expiration date for the block payload.

The procedure does not necessarily output any information.

Underlay In the network underlay, a node is addressable by traditional means out of scope of this document. For example, the node may have a TCP/IP address, or a HTTPS endpoint. While the specific addressing options and mechanisms are out of scope for this document, it is necessary to define a universal addressing format in order to facilitate the distribution of connectivity information to other nodes in the DHT overlay. This format is the "HELLO" message. It is expected that there are basic mechanisms available to manage node connectivity and addressing. The required functionality are abstracted through the following procedures:

TRY_CONNECT(N, A): A function which allows the local node to attempt the establishment of a connection to another node N using an address A. When the connection attempt is successful, information on the new peer is offered through the PEER_CONNECTED signal.
HOLD(P): A function which tells the underlay to keep a hold on the connection to a peer P. FIXME what is this needed for?
DROP(P): A function which tells the underlay to drop the connection to a peer P. FIXME what is this needed for?
M = RECEIVE(P): A function or event that allows the local node to receive a protocol message M as defined in this document from a peer P.
SEND(P, M): A function that allows the local node to send a protocol message M as defined in this document to a peer P. If call to SEND fails, the message has not been sent.
S = ESTIMATE_NETWORK_SIZE(): A procedure that provides estimates on the network size S for use in the DHT routing algorithms. FIXME: What is S and give an example.

In addition to the above procedures, which are meant to be actively executed by the implementation as part of the peer-to-peer protocol, the following callbacks or signals drive updates of the routing table:

PEER_CONNECTED -> P: is a signal that allows the DHT to react to a newly connected peer P. Such an event triggers, for example, updates in the routing table.
PEER_DISCONNECTED -> P: is a signal that allows the DHT to react to a recently disconnected peer. Such an event triggers, for example, updates in the routing table.
ADDRESS_ADDED -> A: The underlay signals us that an address A was added for our local peer. This information is used to advertise connectivity information to the local peer. A is a string suitable for inclusion in a HELLO payload .
ADDRESS_DELETED -> A: The underlay signals us that an address A was removed. This information is used, for example, to no longer advertise this address.

Bootstrapping Initially, the implementation depends upon the Underlay to provide at least one initial connection to a peer signalled through PEER_CONNECTED. The initial set of peers is stored in the routing table component . Further, the Underlay must provide the implementation with one or more addresses signalled through ADDRESS_ADDED. The implementation then proceeds to periodically advertise all active addresses in a HELLO block . In order to find more close nodes in the network, an implementation MUST now periodically messages. In both cases the frequency of advertisements and peer discovery MAY be adapted according to network conditions, connected peers, workload of the system and other factors which are at the discretion of the developer. Any implementation encountering a HELLO GET request initially sends its own node address if it.

Routing

Peer Storage A R5N implementation must store the information on connected nodes and update changes accordingly in a local persistance component such as a database. Upon receiving a connection notification from the Underlay through NODE_CONNECTED, information on the new peer is added to the local peer storage. When disconnect is indicated by the Underlay through NODE_DISCONNECTED the peer MUST be removed from the local peer storage. In order to achieve O(log n) routing performance, the data structure for managing connected nodes and their metadata MUST be implemented using the k-buckets concept of . In order to select nodes which are suitable destinations for routing messages, R5N uses a hybrid approach: Given an estimated network size N, the node selection for the first N hops is random. After the initial N hops, node selection follows an XOR-based node distance calculation.

Peer Discovery FIXME: Elaborate on FindPeer here. Why is this a route option? In the code we only answer with HELLO anyway and ignore the type of block requested.

Routing Table The routing table consists of an array of lists of connected nodes. The i-th list stores neighbours whose identifiers are between distance 2^i 2^(i+1) from the local node. An implementation MAY choose an upper limit on the length of the lists, but SHOULD try to keep at least 5 entries per bucket. For example, in case of system constraints with respect to active connections, an implementation SHOULD evict nodes in large buckets rather than nodes from shallow ones. As the message traverses a random path through the network for the first N hops, it is essential that routing loops are avoided. In R5N, a bloomfilter is used as part of the routing metadata in messages. The bloomfilter is updates at each hop with the hops node identity. For the next hop selection in both the random and the deterministic case, any node which is in the bloomfilter for the respective message is not included in the node selection process. The R5N routing component MUST implement the following functions:

GetDistance(A, B) -> Distance as Integer: this function calculates the binary XOR between A and B. The resulting distance is interpreted as an integer where the leftmost bit is the most significant bit.
SelectClosestNode(K, B) -> N: This function selects the connected node N from our routing table with the shortest XOR-distance to the key K. This means that for all other nodes N' in the routing table GetDistance(N, K) < GetDistance(N',K). Nodes in the bloomfilter B are not considered.
SelectRandomNode(B) -> N: This function selects a random node N from all connected nodes. Nodes in the bloomfilter B are not considered.
SelectNode(K, H, B) -> N: This function selects a node N depending on the number of hops H parameter. If H < NETWORK_SIZE_ESTIMATE this function MUST return SelectRandomNode() and SelectClosestnode(K) otherwise. Nodes in the bloomfilter B are not considered.
IsClosestNode(N, K, B) -> true | false: checks if N is the closest node for K (cf. SelectClosestNode(K)). Nodes in the bloomfilter B are not considered.

Message Processing The implementation MUST listen for RECEIVE(P, M) signals from the Underlay and respond to the respective messages sent by the peer P. In the following, the wire formats of the messages and the required processing are detailed. The local node address is referred to as N.

Route Options The RouteOptions consist of the following flags which are represented in an options field in the messages. Each flag is represented by a bit in the field starting from 0 as the rightmost bit to 15 as the leftmost bit. FIXME: Actually, we set those bits and then store the resulting value in NBO...

0: Demultiplex-Everywhere: indicates that each node along the way should process the request.
1: Record-Route: indicates to keep track of the route that the message takes in the P2P network.
2: Allow-Approximate: This is a special flag which modifies the message processing to allow approximate results.
3-15: Reserved: For future use.

Bloomfilter In order to prevent circular routes, GET and PUT messages contain a 128-bit Bloom filter (m=128). The Bloom filter is used to detect duplicate node addresses along the route. A Bloom filter "bf" is initially empty, consisting only of zeroes. There are two functions which can be invoked on the Bloom filter: BF-SET(bf, e) and BF-TEST(bf, e) where "e" is an element which is to be added to the Bloom filter or queried against the set. Any bloom filter uses k=16 different hash functions each of which is defined as follows:

Extended query TODO: Talk about XQuery in the context of messages.

PutMessage

Wire Format

where:

MSIZE: denotes the size of this message in network byte order.
MTYPE: is the 16-bit message type. This type can be one of the DHT message types but for put messages it must be set to the value 146 in network byte order.
BTYPE: is a 32-bit block type field. The block type indicates the content type of the payload. In network byte order.
OPTIONS: is a 16-bit options field (see below).
HOPCOUNT: is a 16-bit number indicating how many hops this message has traversed to far. In network byte order.
REPL_LVL: is a 16-bit number indicating the desired replication level of the data. In network byte order.
PATH_LEN: is a 16-bit number indicating the length of the PUT path recorded in PUTPATH. As PUTPATH is optional, this value may be zero. In network byte order.
EXPIRATION: denotes the absolute 64-bit expiration date of the content. In microseconds since midnight (0 hour), January 1, 1970 in network byte order.
BLOOMFILTER: A bloomfilter (for node addresses) to stop circular routes.
KEY: The key under which the PUT request wants to store content under.
PUTPATH: the variable-length PUT path. The path consists of a list of PATH_LEN node addresses.
BLOCK: the variable-length block payload. The contents are determined by the BTYPE field.

Processing Upon receiving a PutMessage from a peer P. An implementation MUST process it step by step as follows:

The EXPIRATION field is evaluated. If the message is expired, it MUST be discarded.
If the BTYPE is not supported by the implementation, no validation of the block payload is performed and processing continues at (4). Else, the block MUST be validated as defined in (3).
The block payload of the message is evaluated using according to the BTYPE using the respective ValidateBlockStoreRequest procedure. If the block payload is invalid or does not match the key, it MUST be discarded.
The node address of the sender peer P SHOULD be in BLOOMFILTER. If not, the implementation MAY log an error, but MUST continue.
If the RecordRoute flag is set in OPTIONS, the local node address MUST be appended to the PUTPATH of the message.
If the local node is the closest node (cf. IsClosestNode(N, Key)) or the DemultiplexEverywhere options flag ist set, the message MUST be stored locally in the block storage.
Given the value in REPL_LVL, the number of peers to forward to MUST be calculated. If there is at least one peers to forward to, the implementation SHOULD select up to this number of peers to forward the message to. The implementation MAY forward to fewer or no peers in order to handle resource constraints such as bandwidth. Finally, the local node address MUST be added to the BLOOMFILTER of the forwarded message. For all peers with node address P chosen to forward the message to, SEND(P, PutMessage) is called.

GetMessage

Wire Format

where:

MSIZE: denotes the size of this message in network byte order.
MTYPE: is the 16-bit message type. It must be set to the value 147 in network byte order.
BTYPE: is a 32-bit block type field. The block type indicates the content type of the payload. In network byte order.
OPTIONS: is a 16-bit options field (see below).
HOPCOUNT: is a 16-bit number indicating how many hops this message has traversed to far. In network byte order.
REPL_LVL: is a 16-bit number indicating the desired replication level of the data. In network byte order.
XQ_SIZE: is a 32-bit number indicating the length of the optional extended query XQUERY. In network byte order.
BLOOMFILTER: A bloomfilter (for node identities) to stop circular routes.
KEY: The key under which the PUT request wants to store content under.
XQUERY: the variable-length extended query. Optional.
BF_MUTATOR: The 32-bit bloomfilter mutator for the result bloomfilter.
RESULT_BF: the variable-length result bloomfilter.

Processing Upon receiving a GetMmessage from a peer an implementation MUST process it step by step as follows:

The KEY and XQUERY is validated against the requested BTYPE as defined by its respective ValidateBlockQuery procedure. If the BTYPE is not supported, or if the block key does not match or if the XQUERY is malformed, the message MUST be discarded.
The node address of the sender peer P SHOULD be in the BLOOMFILTER. If not, the implementation MAY log an error, but MUST continue.
If the local node is the closest node (cf. IsClosestNode (N, Key)) or the DemultiplexEverywhere options flag is set, a reply MUST be produced:
1. If OPTIONS indicate a FindNode request, FIXME the node selection foo from buckets that probably needs fixing. Take into account REPLY_BF
2. Else, if there is a block in the local Block Storage which is not already in the RESULT_BF, a ResultMessage MUST be sent. FIXME link to how the result is sent?
FIXME: We only handle if not GNUNET_BLOCK_EVALUATION_OK_LAST. This means that we must evaluate the Reply produced in the previous step using ValidateBlockReply for this BTYPE
Given the value in REPL_LVL, the number of nodes to forward to MUST be calculated (NUM-FORWARD-nodeS). If there is at least one node to forward to, the implementation SHOULD select up to this number of nodes to forward the message to. The implementation MAY forward to fewer or no nodes in order to handle resource constraints such as bandwidth. The message BLOOMFILTER MUST be updated with the local node address N. For all peers with node address P' chosen to forward the message to, SEND(P', PutMessage) is called.

ResultMessage

Wire Format

where:

MSIZE: denotes the size of this message in network byte order.
MTYPE: is the 16-bit message type. This type can be one of the DHT message types but for put messages it must be set to the value 148 in network byte order.
OPTIONS: is a 16-bit options field (see below).
BTYPE: is a 32-bit block type field. The block type indicates the content type of the payload. In network byte order.
PUTPATH_L: is a 16-bit number indicating the length of the PUT path recorded in PUTPATH. As PUTPATH is optiona, this value may be zero. In network byte order.
GET_PATH_LEN: is a 16-bit number indicating the length of the GET path recorded in GETPATH. As PUTPATH is optiona, this value may be zero. In network byte order.
EXPIRATION: denotes the absolute 64-bit expiration date of the content. In microseconds since midnight (0 hour), January 1, 1970 in network byte order.
KEY: The key under which the PUT request wants to store content under.
PUTPATH: the variable-length PUT path. The path consists of a list of PATH_LEN node addresses.
GETPATH: the variable-length PUT path. The path consists of a list of PATH_LEN node addresses.
BLOCK: the variable-length resource record data payload. The contents are defined by the respective type of the resource record.

Processing Upon receiving a ResultMessage from a connected node. An implementation MUST process it step by step as follows:

The EXPIRATION field is evaluated. If the message is expired, it MUST be discarded.
If the MTYPE of the message indicates a HELLO block, it must be validated according to . The payload MUST be considered for the local routing table by trying to establish a connection to the node using the information from the HELLO block. If a connection can be established, the node is added to the KBuckets routing table.
If the sender node of the message is already found in the GETPATH, the path MUST be truncated at this position. The implementation MAY log a warning in such a case.
If the KEY of this ResultMessage is found in the list of pending local queries, the KEY and XQUERY are validated against the requested BTYPE using the respective block type implementation of ValidateBlockReply. If the BTYPE is not supported, or if the block key does not match the BTYPE or if the XQUERY is malformed, the message MUST be discarded.
The implementation MAY cache RESULT messages.
If requests by other nodes for this KEY or BTYPE are known, the result block is validated against each request using the respective ValidateBlockReply function. If the request options include FindNode and the result message block type is HELLO the block validation must use the key derived using DeriveBlockKey as the key included in the request is only approximate. (FIXME: So we extract the key to then check it again against the block? This does not make sense...) If the result message block cannot be verified against the KEY of the result message or if BLOCK is malformed, the message MUST be discarded. For each pending request the reply is routed to the requesting node N'. FIXME routed to node or forwarded to peer?

Block Storage

Block Processing RequestEvaluationResult

REQUEST_VALID: Query is valid, no reply given.
REQUEST_INVALID: Query format does not match block type. For example, XQuery not given or of size of XQuery is not appropriate for type.

ReplyEvaluationResult

OK_MORE: Valid result, and there may be more.
OK_LAST: Last possible valid result.
OK_DUPLICATE: Valid result, but duplicate.
RESULT_INVALID: Invalid result. Block does not match query. Value = 4.
RESULT_IRRELEVANT: Block does not match xquery. Valid result, but not relevant for the request.

Block Functions Any block type implementation MUST implement the following functions.

ValidateBlockQuery(Key, XQuery) -> RequestEvaluationResult: is used to evaluate the request for a block. It is used as part of GetMessage processing, where the block payload is still unkown, but the block XQuery (FIXME: Undefined here) and Key can and MUST be verified, if possible.
ValidateBlockStoreRequest(Block, Key) -> RequestEvaluationResult: is used to evaluate a block including its key and payload. It is used as part of PutMessage processing. The validation MUST include a check of the block payload against the Key under which it is requested to be stored.
ValidateBlockReply(Block, XQuery, Key) -> ReplyEvaluationResult: is used to evaluate a block including its Key and payload. It is used as part ResultMessage processing. The validation of the respective Block requires a pending local query or a previously routed request of another node and its associated XQuery data and Key. The validation MUST include a check of the block payload against the key under which it is requested to be stored.
DeriveBlockKey(Block) -> Key: is used to synthesize the block key from the block payload and metadata. It is used as part of FIND-node message processing.
FilterResult(Block, XQuery, Key) -> ReplyEvaluationResult: is used to filter results stored in the local block storage for local queries. Locally stored blocks from previously observed ResultMessages and PutMessages MAY use this function instead of ValidateBlockReply in order to avoid revalidation of the block and only perform filtering based on request parameters.

Block Types Applications can and should define their own block types. The block type determines the format and handling of the block payload by nodes in PUT and RESULT messages. Block types MUST be registered with GANA . For bootstrapping and node discovery, the DHT implementation uses its own block type called "HELLO". A block with this block type contains the NodeID of the node initiating the GET request.

HELLO The HELLO block type wire format is illustrated in . A query for block of type HELLO MUST NOT include extended query data (XQuery). Any implementation encountering a HELLO block with XQuery data MUST consider the block invalid and ignore it.

PEER-ID: is the Peer-ID of the node which has generated this HELLO.
SIGNATURE: is the signature of the HELLO.
EXPIRATION: denotes the absolute 64-bit expiration date of the HELLO. In microseconds since midnight (0 hour), January 1, 1970 in network byte order.
ADDRESSES: is a list of UTF-8 URIs which can be used as addresses to contact the peer. The strings MUST be 0-terminated.

The SIGNATURE covers a 64-bit pseudo header conceptually prefixed to the block. The pseudo header includes the expiration, signature purpose and a hash over the addresses. The wire format is illustrated in .

The Wire Format of the HELLO for Signing.

SIZE: A 32-bit value containing the length of the signed data in bytes in network byte order. The length of the signed data MUST be 80 bytes.
PURPOSE: A 32-bit signature purpose flag. This field MUST be 40 (in network byte order).
EXPIRATION: denotes the absolute 64-bit expiration date of the HELLO. In microseconds since midnight (0 hour), January 1, 1970 in network byte order.
H_ADDRS: a hash over the addresses in the HELLO.

H_ADDRS is generated over the ADDRESSES field as provided in the HELLO block using SHA-512 . A HELLO reply block MAY be empty. Otherwise, it contains the HELLO of a node. The ADDRESSES part of the HELLO indicate endpoints which can be used by the Underlay in order to establish a connection with the node identified by Peer-ID. An example of an addressing scheme used throughout this document is "ip+tcp", which refers to a standard TCP/IP socket connection. The "hier"-part of the URI must provide a suitable address for the given addressing scheme. The following is a non-normative example of address strings:

Security Considerations An implementation MAY limit the number the number of neighbours it stores is any k-bucket of the routing table. However, the lower bound MUST be adhered to. If there is a limit to the maximum number of neighbours in a k-bucket, the implementation must be careful when choosing the which nodes to replace: For example, a simple optimization of the peer selection algorithm may be to oder all nodes within the k-bucket by distance and evict the nodes which are the farthest away. However, this is not a good idea from a resilience perspective. It is much more important to preserve older connections than closer ones. Preferring long-term connections over close connections makes it more difficult for an attacker to execute a Sibyl attack as more resources need to be invested over time.

IANA Considerations TODO: URI handler for "common" URI handler that Underlays may want to use as part of HELLOs.

GANA Considerations GANA is requested to create a "DHT Block Types" registry. The registry shall record for each entry:

Name: The name of the block type (case-insensitive ASCII string, restricted to alphanumeric characters
Number: 32-bit
Comment: Optionally, a brief English text describing the purpose of the block type (in UTF-8)
Contact: Optionally, the contact information of a person to contact for further information
References: Optionally, references describing the record type (such as an RFC)

The registration policy for this sub-registry is "First Come First Served", as described in . GANA is requested to populate this registry as follows:

GANA is requested to amend the "GNUnet Signature Purpose" registry as follows:

Local Storage

Test Vectors