lsd0004

LSD0004: R5N Distributed Hash Table
Log | Files | Refs

commit 82163ff25bfb6da30609991c4066dd969a49a202
parent 9d78bfa2413fcdb1b42ae4bbaee26e91edd47ac2
Author: Christian Grothoff <christian@grothoff.org>
Date:   Sat, 13 Jul 2024 17:02:05 +0200

work on Block section

Diffstat:
Mdraft-schanzen-r5n.xml | 235+++++++++++++++++++++++++++++++++++++++++++------------------------------------
1 file changed, 127 insertions(+), 108 deletions(-)

diff --git a/draft-schanzen-r5n.xml b/draft-schanzen-r5n.xml @@ -2568,12 +2568,12 @@ BEGIN <dt>SetupResultFilter(FilterSize, Mutator) -&gt; RF</dt> <dd> is used to setup an empty result filter. The arguments - are the size of the set of results that must be filtered - at the initiator, and a <tt>Mutator</tt> value which - <bcp14>MAY</bcp14> be used to deterministically - re-randomize probabilistic data structures. <tt>RF</tt> - <bcp14>MUST</bcp14> be a byte sequence suitable for - transmission over the network. + are typically the size of the set of results that must + be filtered at the initiator, and a <tt>Mutator</tt> + value which <bcp14>MAY</bcp14> be used to + deterministically re-randomize probabilistic data + structures. <tt>RF</tt> <bcp14>MUST</bcp14> be a byte + sequence suitable for transmission over the network. </dd> <dt>FilterResult(Block, Key, RF, XQuery) -&gt; (FilterEvaluationResult, RF')</dt> <dd> @@ -2581,7 +2581,7 @@ BEGIN is used to filter results against specific queries. This function does not check the validity of <tt>Block</tt> itself or that it matches the given key, - as this must have been checked earlier. Thus, locally + as this must have been checked earlier. Locally stored blocks from previously observed <tt>ResultMessages</tt> and <tt>PutMessages</tt> use this function to perform filtering based on the request @@ -2632,20 +2632,25 @@ BEGIN <section anchor="hello_block"> <name>HELLO Blocks</name> <t> - For bootstrapping and peer discovery, the DHT implementation uses - its own block type called "HELLO". <tt>HELLO</tt> blocks are the only type - of block that <bcp14>MUST</bcp14> be supported by every - R<sup>5</sup>N implementation. A block with this block type - contains the peer public key of the peer that published the <tt>HELLO</tt> together - with a set of addresses of this peer. The key of a <tt>HELLO</tt> block - is the SHA-512 of the peer public key and thus the peer's identity in the DHT. + For bootstrapping and peer discovery, the DHT + implementation uses its own block type called "HELLO". + <tt>HELLO</tt> blocks are the only type of block that + <bcp14>MUST</bcp14> be supported by every R<sup>5</sup>N + implementation. A block with this block type contains the + peer public key of the peer that published the + <tt>HELLO</tt> together with a set of addresses of this + peer. The key of a <tt>HELLO</tt> block is the SHA-512 + hash <xref target="RFC4634"/> of the peer public key and + thus the peer's identity in the DHT. </t> <t> - The <tt>HELLO</tt> block type wire format is illustrated in - <xref target="figure_hello"/>. A query for block of type <tt>HELLO</tt> <bcp14>MUST NOT</bcp14> - include extended query data (XQuery). Any implementation - encountering a request for a <tt>HELLO</tt> with non-empty XQuery - data <bcp14>MUST</bcp14> consider the request invalid and ignore it. + The <tt>HELLO</tt> block type wire format is illustrated + in <xref target="figure_hello"/>. A query for block of + type <tt>HELLO</tt> <bcp14>MUST NOT</bcp14> include + extended query data (XQuery). Any implementation + encountering a request for a <tt>HELLO</tt> with non-empty + XQuery data <bcp14>MUST</bcp14> consider the request + invalid and ignore it. </t> <figure anchor="figure_hello" title="The HELLO Block Format."> <artwork name="" type="" align="left" alt=""><![CDATA[ @@ -2675,35 +2680,38 @@ BEGIN <dl> <dt>PEER PUBLIC KEY</dt> <dd> - is the public key of the peer which has generated this HELLO. + is the public key of the peer to which the + <tt>ADDRESSES</tt> belong. This is also the public key + needed to verify the <tt>SIGNATURE</tt>. </dd> <dt>EXPIRATION</dt> <dd> - denotes the absolute 64-bit expiration date of the content. - The value specified is microseconds since midnight (0 hour), - January 1, 1970, but must be a multiple of one million (so that it + denotes the absolute 64-bit expiration date of the + content. The value specified is microseconds since + midnight (0 hour), January 1, 1970 in network byte + order, but must be a multiple of one million (so that it can be represented in seconds in a <tt>HELLO</tt> URL). - Stored in network byte order. </dd> <dt>ADDRESSES</dt> <dd> is a list of UTF-8 addresses (<xref target="terminology"/>) which can be used to contact the peer. Each address <bcp14>MUST</bcp14> be 0-terminated. - The set of addresses MAY be empty. + The set of addresses <bcp14>MAY</bcp14> be empty. </dd> <dt>SIGNATURE</dt> <dd> <t> - is the signature of the HELLO. - It covers a 64-bit pseudo header - derived from the information in the <tt>HELLO</tt> block. - The pseudo header includes - the expiration time, a constant that uniquely - identifies the purpose of the signature, - and a hash over the addresses. - The wire format is illustrated - in <xref target="figure_hellowithpseudo"/>. + is the EdDSA signature <xref target="ed25519"/> of the + <tt>HELLO</tt> block. The signature covers various + information derived from the actual data in the + <tt>HELLO</tt> block. The data signed over includes the + block expiration time, a constant that uniquely + identifies the purpose of the signature, and a hash of + the addresses with 0-terminators in the same order as + they are present in the <tt>HELLO</tt> block. The + format is illustrated in <xref + target="figure_hellowithpseudo"/>. </t> <figure anchor="figure_hellowithpseudo" title="The Wire Format of the HELLO for Signing."> <artwork name="" type="" align="left" alt=""><![CDATA[ @@ -2727,88 +2735,97 @@ BEGIN <dl> <dt>SIZE</dt> <dd> - A 32-bit value containing the length of the signed data in bytes - in network byte order. - The length of the signed data <bcp14>MUST</bcp14> be 80 bytes. + A 32-bit value containing the length of the signed data + in bytes in network byte order. The length of the + signed data <bcp14>MUST</bcp14> be 80 bytes. </dd> <dt>PURPOSE</dt> <dd> - A 32-bit signature purpose flag. This field <bcp14>MUST</bcp14> be 7 (in network - byte order). + A 32-bit signature purpose flag. This field + <bcp14>MUST</bcp14> be 7 in network byte order. </dd> <dt>EXPIRATION</dt> <dd> - denotes the absolute 64-bit expiration date of the HELLO - in microseconds since midnight (0 hour), January 1, 1970 - in network byte order. + denotes the absolute 64-bit expiration date of the + <tt>HELLO</tt> in microseconds since midnight (0 hour), + January 1, 1970 in network byte order. </dd> <dt>H_ADDRS</dt> <dd> - a SHA-512 hash over the addresses in the HELLO. - H_ADDRS is generated over the ADDRESSES field - as provided in the <tt>HELLO</tt> block using SHA-512 <xref target="RFC4634"/>. + a SHA-512 hash over the addresses in the <tt>HELLO</tt>. + <tt>H_ADDRS</tt> is generated over the + <tt>ADDRESSES</tt> field as provided in the + <tt>HELLO</tt> block using SHA-512 <xref + target="RFC4634"/>. </dd> </dl> </dd> </dl> <t> - The <tt>HELLO</tt> block functions <bcp14>MUST</bcp14> be implemented - as follows: + The <tt>HELLO</tt> block operations <bcp14>MUST</bcp14> be + implemented as follows: </t> <dl> <dt>ValidateBlockQuery(Key, XQuery) -&gt; RequestEvaluationResult</dt> <dd> To validate a block query for a <tt>HELLO</tt> is to - simply check that the XQuery is empty. If it is empty, + simply check that the <tt>XQuery</tt> is empty. If it is empty, <tt>REQUEST_VALID</tt> ist returned. Otherwise, <tt>REQUEST_INVALID</tt> is returned. </dd> <dt>DeriveBlockKey(Block) -&gt; Key | NONE</dt> <dd> To derive a block key for a <tt>HELLO</tt> is to simply - hash the peer public key from the HELLO. The result of this function - is always the SHA-512 hash over the PEER PUBLIC KEY. + hash the <tt>PEER PUBLIC KEY</tt> from the <tt>HELLO</tt>. The + result of this function is thus always the SHA-512 hash over + the <tt>PEER PUBLIC KEY</tt>. </dd> <dt>ValidateBlockStoreRequest(Block) -&gt; BlockEvaluationResult</dt> <dd> - To validate a block store request is to verify - the EdDSA <tt>SIGNATURE</tt> over the hashed <tt>ADDRESSES</tt> - against the public key from the PEER PUBLIC KEY field. - If the signature is valid BLOCK_VALID is returned. - Otherwise BLOCK_INVALID. + To validate a block store request is to verify the EdDSA + <tt>SIGNATURE</tt> over the hashed <tt>ADDRESSES</tt> + against the public key from the <tt>PEER PUBLIC KEY</tt> + field. If the signature is valid <tt>BLOCK_VALID</tt> is + returned. Otherwise <tt>BLOCK_INVALID</tt> is returned. </dd> <dt>SetupResultFilter(FilterSize, Mutator) -&gt; RF</dt> <dd> <t> - The RESULT_FILTER for <tt>HELLO</tt> blocks is implemented using a - Bloom filter following the definition from <xref target="bloom_filters"/> - and consists of a variable number of bits <tt>L</tt>. - <tt>L</tt> depends on the number of connected peers <tt>|E|</tt> known to - the peer creating a <tt>HELLO</tt> block from its own addresses: - <tt>L</tt> is set to the minimum of - 2<sup>18</sup> bits (2<sup>15</sup> bytes) and the lowest power - of 2 that is strictly larger than <tt>2*K*|E|</tt> bits (<tt>K*|E|/4</tt> bytes). + The <tt>RF</tt> for <tt>HELLO</tt> blocks is implemented + using a Bloom filter following the definition from <xref + target="bloom_filters"/> and consists of a variable number + of bits <tt>L</tt>. <tt>L</tt> depends on the + <tt>FilterSize</tt> which will be the number of + connected peers <tt>|E|</tt> known to the peer creating a + <tt>HELLO</tt> block from its own addresses: <tt>L</tt> is + set to the minimum of 2<sup>18</sup> bits (2<sup>15</sup> + bytes) and the lowest power of 2 that is strictly larger + than <tt>2*K*|E|</tt> bits (<tt>K*|E|/4</tt> bytes). </t> <t> - The <tt>k</tt>-value for the Bloom filter is 16. - The elements used in the Bloom filter - consist of an XOR between the <tt>H_ADDRS</tt> field (as computed using - SHA-512 over the <tt>ADDRESSES</tt>) and the SHA-512 - hash of the <tt>MUTATOR</tt> field from a given <tt>HELLO</tt> block. - The mapping function M(<tt>H_ADDRS XOR MUTATOR</tt>) is defined as follows: + The <tt>k</tt>-value for the Bloom filter + <bcp14>MUST</bcp14> be 16. The elements used in the Bloom + filter consist of an XOR between the <tt>H_ADDRS</tt> + field (as computed using SHA-512 over the + <tt>ADDRESSES</tt>) and the SHA-512 hash of the + <tt>MUTATOR</tt> field from a given <tt>HELLO</tt> block. + The mapping function M(<tt>H_ADDRS XOR MUTATOR</tt>) is + defined as follows: </t> <t> <tt>M(e = H_ADDR XOR MUTATOR) -> e as uint32[]</tt> </t> <t> - <tt>M</tt> is an identity function and returns the 512-bit XOR result unmodified. - This resulting byte string is interpreted as k=16 32-bit - integers in network byte order which are used to set and check the bits in - <tt>B</tt> using <tt>BF-SET()</tt> and <tt>BF-TEST()</tt>. - The 32-bit Mutator is prepended to the L-bit Bloom filter field <tt>HELLO_BF</tt> containing <tt>B</tt> - to create the result filter for a <tt>HELLO</tt> block: + <tt>M</tt> is an identity function and returns the 512-bit + XOR result unmodified. This resulting byte string is + interpreted as k=16 32-bit integers in network byte order + which are used to set and check the bits in <tt>B</tt> + using <tt>BF-SET()</tt> and <tt>BF-TEST()</tt>. The + 32-bit <tt>MUTATOR</tt> is prepended to the L-bit Bloom filter + field <tt>HELLO_BF</tt> containing <tt>B</tt> to create + the result filter for a <tt>HELLO</tt> block: </t> <figure anchor="hello_rf" title="The HELLO Block Result Filter."> <artwork name="" type="" align="left" alt=""><![CDATA[ @@ -2832,54 +2849,56 @@ BEGIN </dd> </dl> <t> - The <tt>MUTATOR</tt> value is used - to additionally "randomize" the computation of the Bloom filter while - remaining deterministic across peers. - It is only ever set by the peer initiating the GET - request, and changed every time the GET request is repeated. - Peers forwarding GET requests <bcp14>MUST</bcp14> not change the - mutator value included in the <tt>RESULT_FILTER</tt> as they might not - be able to recalculate the result filter with a different <tt>MUTATOR</tt> - value. + The <tt>MUTATOR</tt> value is used to additionally + "randomize" the computation of the Bloom filter while + remaining deterministic across peers. It is only ever set + by the peer initiating the GET request, and changed every + time the GET request is repeated. Peers forwarding GET + requests <bcp14>MUST</bcp14> not change the mutator value + included in the <tt>RESULT_FILTER</tt> as they might not + be able to recalculate the result filter with a different + <tt>MUTATOR</tt> value. </t> <t> - Consequently, repeated - requests have statistically independent probabilities of creating - false-positives in a result filter. Thus, even if for one request - a result filter may exclude a result as a false-positive - match, subsequent requests are likely to not have the same + Consequently, repeated requests have statistically + independent probabilities of creating false-positives in a + result filter. Thus, even if for one request a result + filter may exclude a result as a false-positive match, + subsequent requests are likely to not have the same false-positives. </t> <t> - <tt>HELLO</tt> result filters can be merged if the - Bloom filters have the same size and - <tt>MUTATOR</tt> by setting all bits to 1 that are - set in either Bloom filter. This is done whenever - a peer receives a query with the same <tt>MUTATOR</tt>, - predecessor and Bloom filter size. + <tt>HELLO</tt> result filters can be merged if the Bloom + filters have the same size and <tt>MUTATOR</tt> by setting + all bits to 1 that are set in either Bloom filter. This + is done whenever a peer receives a query with the same + <tt>MUTATOR</tt>, predecessor and Bloom filter size. </t> </dd> <dt>FilterResult(Block, Key, RF, XQuery) -&gt; (FilterEvaluationResult, RF')</dt> <dd> - The <tt>H_ADDRS</tt> field is XORed with the SHA-512 - hash of the <tt>MUTATOR</tt> field from the <tt>HELLO</tt> block and the resulting - value is checked against the Bloom filter in RF. - Consequently, HELLOs with completely identical sets of - addresses will be filtered and FILTER_DUPLICATE is returned. - Any small variation in the set of addresses will cause the block - to no longer be filtered (with high probability) and - FILTER_MORE is returned. + The <tt>H_ADDRS</tt> field is XORed with the SHA-512 hash + of the <tt>MUTATOR</tt> field from the <tt>HELLO</tt> + block and the resulting value is checked against the + Bloom filter in <tt>RF</tt>. Consequently, + <tt>HELLOs</tt> with completely identical sets of + addresses will be filtered and <tt>FILTER_DUPLICATE</tt> + is returned. Any small variation in the set of addresses + will cause the block to no longer be filtered (with high + probability) and <tt>FILTER_MORE</tt> is returned. </dd> </dl> </section> <section> <name>Persistence</name> <t> - An implementation <bcp14>SHOULD</bcp14> provide a local persistence mechanism for - blocks. Embedded systems that lack storage capability <bcp14>MAY</bcp14> use - connection-level signalling to indicate that they are merely a client utilizing a - DHT and are not able to participate with storage. - The local storage <bcp14>MUST</bcp14> provide the following functionality: + An implementation <bcp14>SHOULD</bcp14> provide a local + persistence mechanism for blocks. Embedded systems that + lack storage capability <bcp14>MAY</bcp14> use + connection-level signalling to indicate that they are merely + a client utilizing a DHT and are not able to participate + with storage. The local storage <bcp14>MUST</bcp14> provide + the following functionality: </t> <dl> <dt>Store(Key, Block)</dt>