summaryrefslogtreecommitdiff
path: root/draft-schanzen-r5n.xml
diff options
context:
space:
mode:
Diffstat (limited to 'draft-schanzen-r5n.xml')
-rw-r--r--draft-schanzen-r5n.xml2272
1 files changed, 1136 insertions, 1136 deletions
diff --git a/draft-schanzen-r5n.xml b/draft-schanzen-r5n.xml
index 5dd9bd2..9b5aece 100644
--- a/draft-schanzen-r5n.xml
+++ b/draft-schanzen-r5n.xml
@@ -43,106 +43,106 @@
43 <country>DE</country> 43 <country>DE</country>
44 </postal> 44 </postal>
45 <email>schanzen@gnunet.org</email> 45 <email>schanzen@gnunet.org</email>
46 </address> 46 </address>
47 </author> 47 </author>
48 <author fullname="Christian Grothoff" initials="C." surname="Grothoff"> 48 <author fullname="Christian Grothoff" initials="C." surname="Grothoff">
49 <organization>Berner Fachhochschule</organization> 49 <organization>Berner Fachhochschule</organization>
50 <address> 50 <address>
51 <postal> 51 <postal>
52 <street>Hoeheweg 80</street> 52 <street>Hoeheweg 80</street>
53 <city>Biel/Bienne</city> 53 <city>Biel/Bienne</city>
54 <code>2501</code> 54 <code>2501</code>
55 <country>CH</country> 55 <country>CH</country>
56 </postal> 56 </postal>
57 <email>grothoff@gnunet.org</email> 57 <email>grothoff@gnunet.org</email>
58 </address> 58 </address>
59 </author> 59 </author>
60 <author fullname="Bernd Fix" initials="B." surname="Fix"> 60 <author fullname="Bernd Fix" initials="B." surname="Fix">
61 <organization>GNUnet e.V.</organization> 61 <organization>GNUnet e.V.</organization>
62 <address> 62 <address>
63 <postal> 63 <postal>
64 <street>Boltzmannstrasse 3</street> 64 <street>Boltzmannstrasse 3</street>
65 <city>Garching</city> 65 <city>Garching</city>
66 <code>85748</code> 66 <code>85748</code>
67 <country>DE</country> 67 <country>DE</country>
68 </postal> 68 </postal>
69 <email>fix@gnunet.org</email> 69 <email>fix@gnunet.org</email>
70 </address> 70 </address>
71 </author> 71 </author>
72 <!-- Meta-data Declarations --> 72 <!-- Meta-data Declarations -->
73 <area>General</area> 73 <area>General</area>
74 <workgroup>Independent Stream</workgroup> 74 <workgroup>Independent Stream</workgroup>
75 <keyword>distributed hash tables</keyword> 75 <keyword>distributed hash tables</keyword>
76 <abstract> 76 <abstract>
77 <t>This document contains the R5N DHT technical specification.</t> 77 <t>This document contains the R5N DHT technical specification.</t>
78 <t> 78 <t>
79 This document defines the normative wire format of resource records, 79 This document defines the normative wire format of resource records,
80 resolution processes, cryptographic routines and security considerations for 80 resolution processes, cryptographic routines and security considerations for
81 use by implementers. 81 use by implementers.
82 </t> 82 </t>
83 <t> 83 <t>
84 This specification was developed outside the IETF and does not have IETF 84 This specification was developed outside the IETF and does not have IETF
85 consensus. It is published here to guide implementation of R5N and to 85 consensus. It is published here to guide implementation of R5N and to
86 ensure interoperability among implementations. 86 ensure interoperability among implementations.
87 </t> 87 </t>
88 </abstract> 88 </abstract>
89 </front> 89 </front>
90 <middle> 90 <middle>
91 <section anchor="introduction" numbered="true" toc="default"> 91 <section anchor="introduction" numbered="true" toc="default">
92 <name>Introduction</name> 92 <name>Introduction</name>
93<!-- FIXME: Here we should also cite and discuss RELOAD (https://datatracker.ietf.org/doc/html/rfc6940) 93<!-- FIXME: Here we should also cite and discuss RELOAD (https://datatracker.ietf.org/doc/html/rfc6940)
94and establish why we need this spec and are not a "Topology plugin" 94and establish why we need this spec and are not a "Topology plugin"
95in RELOAD. The argumentation revolves around the trust model (openness) and 95in RELOAD. The argumentation revolves around the trust model (openness) and
96security aspects (path signatures). 96security aspects (path signatures).
97--> 97-->
98 <t>
99 Distributed Hash Tables (DHTs) are a key data structure for the
100 construction of completely decentralized applications.
101 DHTs are important because they generally provide a robust and
102 efficient means to distribute the storage and retrieval of
103 key-value pairs.
104 </t>
105 <t>
106 While <xref target="RFC6940"/> already provides a peer-to-peer (P2P)
107 signaling protocol with extensible routing and topology mechanisms,
108 it also relies on strict admission control through the use of either
109 centralized enrollment servers or pre-shared keys.
110 Modern decentralized applications require a more open system that
111 enables ad-hoc participation and other means to prevent common attacks
112 on P2P overlays.
113 </t>
114 <t>
115 This document contains the technical specification
116 of the R5N DHT <xref target="R5N"/>, a secure DHT routing algorithm
117 and data structure for decentralized applications.
118 R5N is an open P2P overlay routing mechanism which supports ad-hoc
119 participation and security properties including support for
120 topologies in restricted-route environments and path signatures.
121 </t>
122 <t>
123 This document defines the normative wire format of peer-to-peer
124 messages, routing algorithms, cryptographic routines and security
125 considerations for use by implementors.
126 </t>
127 <section numbered="true" toc="default">
128 <name>Requirements Notation</name>
98 <t> 129 <t>
99 Distributed Hash Tables (DHTs) are a key data structure for the 130 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
100 construction of completely decentralized applications. 131 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
101 DHTs are important because they generally provide a robust and 132 "OPTIONAL" in this document are to be interpreted as described in
102 efficient means to distribute the storage and retrieval of 133 BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only
103 key-value pairs. 134 when, they appear in all capitals, as shown here.
104 </t>
105 <t>
106 While <xref target="RFC6940"/> already provides a peer-to-peer (P2P)
107 signaling protocol with extensible routing and topology mechanisms,
108 it also relies on strict admission control through the use of either
109 centralized enrollment servers or pre-shared keys.
110 Modern decentralized applications require a more open system that
111 enables ad-hoc participation and other means to prevent common attacks
112 on P2P overlays.
113 </t>
114 <t>
115 This document contains the technical specification
116 of the R5N DHT <xref target="R5N"/>, a secure DHT routing algorithm
117 and data structure for decentralized applications.
118 R5N is an open P2P overlay routing mechanism which supports ad-hoc
119 participation and security properties including support for
120 topologies in restricted-route environments and path signatures.
121 </t> 135 </t>
122 <t>
123 This document defines the normative wire format of peer-to-peer
124 messages, routing algorithms, cryptographic routines and security
125 considerations for use by implementors.
126 </t>
127 <section numbered="true" toc="default">
128 <name>Requirements Notation</name>
129 <t>
130 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
131 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
132 "OPTIONAL" in this document are to be interpreted as described in
133 BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only
134 when, they appear in all capitals, as shown here.
135 </t>
136 </section>
137 </section> 136 </section>
138 <section anchor="architecture" numbered="true" toc="default"> 137 </section>
139 <name>Architecture</name> 138 <section anchor="architecture" numbered="true" toc="default">
140 <t> 139 <name>Architecture</name>
141 R5N is an overlay network with a pluggable transport layer. 140 <t>
142 The following figure shows the R5N architecture. 141 R5N is an overlay network with a pluggable transport layer.
143 </t> 142 The following figure shows the R5N architecture.
144 <figure> 143 </t>
145 <artwork><![CDATA[ 144 <figure>
145 <artwork><![CDATA[
146 | +-----------------+ +-------+ 146 | +-----------------+ +-------+
147Applications | | GNU Name System | | CADET | ... 147Applications | | GNU Name System | | CADET | ...
148 | +-----------------+ +-------+ 148 | +-----------------+ +-------+
@@ -165,466 +165,466 @@ Connectivity | |Underlay| |Underlay|
165 | |Link | |Link | 165 | |Link | |Link |
166 | +--------+ +--------+ 166 | +--------+ +--------+
167]]> 167]]>
168 </artwork> 168 </artwork>
169 </figure> 169 </figure>
170 <dl>
171 <dt>Applications</dt>
172 <dd>
173 Applications are components which directly use the DHT overlay
174 interfaces. Possible applications include the GNU Name System
175 <xref target="I-D.draft-schanzen-gns"/> or the CADET transport system
176 <xref target="cadet"/>.
177 </dd>
178 <dt>Overlay Interface</dt>
179 <dd>
180 The Overlay Interface exposes the core operations of the DHT overlay
181 to applications.
182 This includes querying and retrieving data from the DHT.
183 </dd>
184 <dt>Block Storage</dt>
185 <dd>
186 The Block Storage component is used to persist and manage data
187 by nodes. It includes logic for quotas, caching stragegies and
188 data validation.
189 </dd>
190 <dt>Message Processing</dt>
191 <dd>
192 The Message Processing component processes requests from and responses
193 to applications as well as messages from the underlay network.
194 </dd>
195 <dt>Routing</dt>
196 <dd>
197 The Routing component includes the routing table as well as
198 routing and node selection logic. It facilitates the R5N routing
199 algorithm with required data structures and algorithms.
200 </dd>
201 <dt>Underlay Interface</dt>
202 <dd>
203 The DHT Underlay Interface is an abstraction layer on top of the
204 supported links of a node. Nodes may be linked by a variety of
205 different transports, including "classical" protocols such as
206 TCP, UDP and TLS or advanced protocols such as GNUnet, L2P or Tor.
207 </dd>
208 </dl>
209 <t>
210 Other glossary
211 </t>
212 <dl>
213 <dt>Node</dt>
214 <dd>
215 A node is participant in the network and addressable through the
216 network overlay.
217 </dd>
218 <dt>Node Address</dt>
219 <dd>
220 The <tt>Node Address</tt> is the identifier used on the Overlay
221 to address a node.
222 </dd>
223 <dt>Node ID</dt>
224 <dd>
225 The <tt>Node ID</tt> is the identity which is used to authenticate
226 a node in the underlay.
227 The <tt>Node Address</tt> is derived from the <tt>Node ID</tt>.
228 </dd>
229 <dt>Peer</dt>
230 <dd>
231 A peer is a node which is directly connected to our peer.
232 </dd>
233 </dl>
234 </section>
235 <section anchor="overlay" numbered="true" toc="default">
236 <name>Overlay</name>
237 <t>
238 In the DHT overlay, a node is addressable by its
239 <tt>Node Address</tt>.
240 The <tt>Node Address</tt> is a SHA-512 hash <xref target="RFC4634"/>
241 of the <tt>Node ID</tt>.
242 <!-- FIXME should the node ID be agile? Should the signature then
243 also be agile?-->
244 <!--The node public key is the public key of the corresponding
245 Ed25519<xref target="ed25519" /> node private key.-->
246 </t>
247 <t>
248 Any implementation of this specification MUST expose the two API
249 procedures "GET" and "PUT".
250 </t>
251 <section>
252 <name>The GET procedure</name>
253 <t>
254 The GET procedure is defined as follows:
255 </t>
256 <t>
257 <tt>GET(Key[, GetParams]) -> Results as List</tt>
258 </t>
259 <t>
260 The procedure takes a single additional <tt>QueryParams</tt>
261 argument in order to specify detailed query parameters.
262 The <tt>GetParams</tt> consist of the following parameters:
263 </t>
170 <dl> 264 <dl>
171 <dt>Applications</dt> 265 <dt>BlockType</dt>
172 <dd>
173 Applications are components which directly use the DHT overlay
174 interfaces. Possible applications include the GNU Name System
175 <xref target="I-D.draft-schanzen-gns"/> or the CADET transport system
176 <xref target="cadet"/>.
177 </dd>
178 <dt>Overlay Interface</dt>
179 <dd>
180 The Overlay Interface exposes the core operations of the DHT overlay
181 to applications.
182 This includes querying and retrieving data from the DHT.
183 </dd>
184 <dt>Block Storage</dt>
185 <dd> 266 <dd>
186 The Block Storage component is used to persist and manage data 267 The default setting of this parameter is to request any type of
187 by nodes. It includes logic for quotas, caching stragegies and 268 block types under the key.
188 data validation.
189 </dd> 269 </dd>
190 <dt>Message Processing</dt> 270 <dt>ReplicationLevel</dt>
271 <dd>The default setting of this parameter is X (FIXME).</dd>
272 <dt>RouteOptions</dt>
191 <dd> 273 <dd>
192 The Message Processing component processes requests from and responses 274 are used in order to indicate certain
193 to applications as well as messages from the underlay network. 275 processing requirements for messages.
276 Any combination of options as defined in <xref target="route_options"/>
277 may be specificied.
278 The default setting of this parameter is that no option is set.
194 </dd> 279 </dd>
195 <dt>Routing</dt> 280 <dt>XQuery</dt>
196 <dd> 281 <dd>
197 The Routing component includes the routing table as well as 282 is extended query medatadata which may be
198 routing and node selection logic. It facilitates the R5N routing 283 required depending on the respective <tt>BlockType</tt>.
199 algorithm with required data structures and algorithms. 284 A <tt>BlockType</tt> must define if the <tt>XQuery</tt> can or must
200 </dd> 285 be used and what the specific format of its contents should be.
201 <dt>Underlay Interface</dt> 286 See also <xref target="block_types"/>.
202 <dd> 287 The default setting of this parameter is empty.
203 The DHT Underlay Interface is an abstraction layer on top of the
204 supported links of a node. Nodes may be linked by a variety of
205 different transports, including "classical" protocols such as
206 TCP, UDP and TLS or advanced protocols such as GNUnet, L2P or Tor.
207 </dd> 288 </dd>
208 </dl> 289 </dl>
209 <t> 290 <t>
210 Other glossary 291 The <tt>RouteOptions</tt> consist of the following flags:
211 </t> 292 </t>
212 <dl> 293 <dl anchor="route_options">
213 <dt>Node</dt> 294 <dt>DemultiplexEverywhere</dt>
214 <dd>
215 A node is participant in the network and addressable through the
216 network overlay.
217 </dd>
218 <dt>Node Address</dt>
219 <dd> 295 <dd>
220 The <tt>Node Address</tt> is the identifier used on the Overlay 296 indicates that each node along the way should process the request.
221 to address a node.
222 </dd> 297 </dd>
223 <dt>Node ID</dt> 298 <dt>RecordRoute</dt>
224 <dd> 299 <dd>
225 The <tt>Node ID</tt> is the identity which is used to authenticate 300 indicates to keep track of the route that the message takes
226 a node in the underlay. 301 in the P2P network.
227 The <tt>Node Address</tt> is derived from the <tt>Node ID</tt>.
228 </dd> 302 </dd>
229 <dt>Peer</dt> 303 <dt>FindNode</dt>
230 <dd> 304 <dd>
231 A peer is a node which is directly connected to our peer. 305 indicates that this is a request used to find additional nodes.
306 This is a special flag which modifies the message processing to
307 allow approximate results.
232 </dd> 308 </dd>
233 </dl> 309 </dl>
310 <t>
311 As the time it takes for results to arrive may vary an implementation
312 may either return a list of results after a timeout or allow the
313 caller to provide a callback function which is called for any result
314 received from the DHT until the procedure is cancelled.
315 </t>
234 </section> 316 </section>
235 <section anchor="overlay" numbered="true" toc="default"> 317 <section>
236 <name>Overlay</name> 318 <name>The PUT procedure</name>
237 <t> 319 <t>
238 In the DHT overlay, a node is addressable by its 320 The PUT procedure is defined as follows:
239 <tt>Node Address</tt>.
240 The <tt>Node Address</tt> is a SHA-512 hash <xref target="RFC4634"/>
241 of the <tt>Node ID</tt>.
242 <!-- FIXME should the node ID be agile? Should the signature then
243 also be agile?-->
244 <!--The node public key is the public key of the corresponding
245 Ed25519<xref target="ed25519" /> node private key.-->
246 </t> 321 </t>
247 <t> 322 <t>
248 Any implementation of this specification MUST expose the two API 323 <tt>PUT(Key, Block, BlockType[, PutParams])</tt>
249 procedures "GET" and "PUT".
250 </t> 324 </t>
251 <section>
252 <name>The GET procedure</name>
253 <t>
254 The GET procedure is defined as follows:
255 </t>
256 <t>
257 <tt>GET(Key[, GetParams]) -> Results as List</tt>
258 </t>
259 <t>
260 The procedure takes a single additional <tt>QueryParams</tt>
261 argument in order to specify detailed query parameters.
262 The <tt>GetParams</tt> consist of the following parameters:
263 </t>
264 <dl>
265 <dt>BlockType</dt>
266 <dd>
267 The default setting of this parameter is to request any type of
268 block types under the key.
269 </dd>
270 <dt>ReplicationLevel</dt>
271 <dd>The default setting of this parameter is X (FIXME).</dd>
272 <dt>RouteOptions</dt>
273 <dd>
274 are used in order to indicate certain
275 processing requirements for messages.
276 Any combination of options as defined in <xref target="route_options"/>
277 may be specificied.
278 The default setting of this parameter is that no option is set.
279 </dd>
280 <dt>XQuery</dt>
281 <dd>
282 is extended query medatadata which may be
283 required depending on the respective <tt>BlockType</tt>.
284 A <tt>BlockType</tt> must define if the <tt>XQuery</tt> can or must
285 be used and what the specific format of its contents should be.
286 See also <xref target="block_types"/>.
287 The default setting of this parameter is empty.
288 </dd>
289 </dl>
290 <t>
291 The <tt>RouteOptions</tt> consist of the following flags:
292 </t>
293 <dl anchor="route_options">
294 <dt>DemultiplexEverywhere</dt>
295 <dd>
296 indicates that each node along the way should process the request.
297 </dd>
298 <dt>RecordRoute</dt>
299 <dd>
300 indicates to keep track of the route that the message takes
301 in the P2P network.
302 </dd>
303 <dt>FindNode</dt>
304 <dd>
305 indicates that this is a request used to find additional nodes.
306 This is a special flag which modifies the message processing to
307 allow approximate results.
308 </dd>
309 </dl>
310 <t>
311 As the time it takes for results to arrive may vary an implementation
312 may either return a list of results after a timeout or allow the
313 caller to provide a callback function which is called for any result
314 received from the DHT until the procedure is cancelled.
315 </t>
316 </section>
317 <section>
318 <name>The PUT procedure</name>
319 <t>
320 The PUT procedure is defined as follows:
321 </t>
322 <t>
323 <tt>PUT(Key, Block, BlockType[, PutParams])</tt>
324 </t>
325 <t>
326 The procedure takes three mandatory arguments.
327 The first argument is the query
328 key under which the block is to be stored.
329 The second parameter is the block payload.
330 The third parameter is the type of the block payload.
331 Block types are defined in <xref target="block_types"/>.
332 The PUT procedure also allows an optional <tt>PutParams</tt>
333 parameter.
334 </t>
335 <dl>
336 <dt>ReplicationLevel</dt>
337 <dd>The default setting of this parameter is X (FIXME).</dd>
338 <dt>RouteOptions</dt>
339 <dd>
340 are used in order to indicate certain
341 processing requirements for messages.
342 Any combination of options as defined in <xref target="route_options"/>
343 may be specificied.
344 The default setting of this parameter is that no option is set.
345 </dd>
346 <dt>Expiration</dt>
347 <dd>
348 is the requested expiration date for the block payload.
349 </dd>
350 </dl>
351 </section>
352 </section>
353 <section anchor="underlay" numbered="true" toc="default">
354 <name>Underlay</name>
355 <t> 325 <t>
356 In the network underlay, a node is addressable by traditional 326 The procedure takes three mandatory arguments.
357 means out of scope of this document. For example, the node may 327 The first argument is the query
358 have a TCP/IP address, or a HTTPS endpoint. 328 key under which the block is to be stored.
359 While the specific addressing options and mechanisms are out of scope 329 The second parameter is the block payload.
360 for this document, it is necessary to define a universal addressing 330 The third parameter is the type of the block payload.
361 format in order to facilitate the distribution of connectivity 331 Block types are defined in <xref target="block_types"/>.
362 information to other nodes in the DHT overlay. 332 The PUT procedure also allows an optional <tt>PutParams</tt>
363 This format is the "HELLO" message. 333 parameter.
364 </t> 334 </t>
365 <!-- 335 <dl>
366 1) The current API is always fire+forget, it doesn't allow for flow 336 <dt>ReplicationLevel</dt>
367 control. I think we need to add that, possibly for sending and receiving. 337 <dd>The default setting of this parameter is X (FIXME).</dd>
338 <dt>RouteOptions</dt>
339 <dd>
340 are used in order to indicate certain
341 processing requirements for messages.
342 Any combination of options as defined in <xref target="route_options"/>
343 may be specificied.
344 The default setting of this parameter is that no option is set.
345 </dd>
346 <dt>Expiration</dt>
347 <dd>
348 is the requested expiration date for the block payload.
349 </dd>
350 </dl>
351 </section>
352 </section>
353 <section anchor="underlay" numbered="true" toc="default">
354 <name>Underlay</name>
355 <t>
356 In the network underlay, a node is addressable by traditional
357 means out of scope of this document. For example, the node may
358 have a TCP/IP address, or a HTTPS endpoint.
359 While the specific addressing options and mechanisms are out of scope
360 for this document, it is necessary to define a universal addressing
361 format in order to facilitate the distribution of connectivity
362 information to other nodes in the DHT overlay.
363 This format is the "HELLO" message.
364 </t>
365 <!--
366 1) The current API is always fire+forget, it doesn't allow for flow
367 control. I think we need to add that, possibly for sending and receiving.
368 368
369 IDK. 369 IDK.
370 370
371 2) I'm not sure what to do with the crypto: mandate EdDSA or allow the 371 2) I'm not sure what to do with the crypto: mandate EdDSA or allow the
372 underlay to do whatever public keys it likes. 372 underlay to do whatever public keys it likes.
373 373
374 We need keys in the overlay. (Path signatures). Do they need to 374 We need keys in the overlay. (Path signatures). Do they need to
375 be the same keys??? 375 be the same keys???
376 376
377 3) I think we may want to mandate that the lower layer at least 377 3) I think we may want to mandate that the lower layer at least
378 authenticate the other node (i.e. every UDP message could be in 378 authenticate the other node (i.e. every UDP message could be in
379 cleartext, but would need to come with an EdDSA signature, alas 92 byte 379 cleartext, but would need to come with an EdDSA signature, alas 92 byte
380 overhead and a signature verification _required_). Otherwise, I don't 380 overhead and a signature verification _required_). Otherwise, I don't
381 see how we can offer even the most minimal protections against node 381 see how we can offer even the most minimal protections against node
382 impersonation attacks. WDYT? 382 impersonation attacks. WDYT?
383 383
384 Security considerations? Prerequisites? 384 Security considerations? Prerequisites?
385 --> 385 -->
386 <t>
387 It is expected that there are basic mechanisms available to
388 manage node connectivity and addressing.
389 The required functionality are abstracted through the following
390 procedures and events:
391 </t>
392 <dl>
393 <dt>
394 <tt>PEER_CONNECTED(P)</tt>
395 </dt>
396 <dd>
397 is a signal that allows the DHT to react to a newly connected peer
398 <tt>N</tt>.
399 Such an event triggers, for example, updates in the
400 routing table.
401 </dd>
402 <dt>
403 <tt>PEER_DISCONNECTED(P)</tt>
404 </dt>
405 <dd>
406 is a signal that allows the DHT to react to a recently disconnected
407 peer.
408 Such an event triggers, for example, updates in the
409 routing table.
410 </dd>
411 <dt>
412 <tt>TRY_CONNECT(N, A)</tt>
413 </dt>
414 <dd>
415 A function which allows the local node to attempt the establishment of
416 a connection to another node <tt>N</tt> using an address <tt>A</tt>.
417 When the connection attempt is successful, information on the new
418 peer is offered through the <tt>PEER_CONNECTED</tt> signal.
419 </dd>
420 <dt>
421 <tt>HOLD(P)</tt>
422 </dt>
423 <dd>
424 A function which tells the underlay to keep a hold on the connection
425 to a peer <tt>P</tt>. FIXME what is this needed for?
426 </dd>
427 <dt>
428 <tt>DROP(P)</tt>
429 </dt>
430 <dd>
431 A function which tells the underlay to drop the connection to a
432 peer <tt>P</tt>. FIXME what is this needed for?
433 </dd>
434 <dt>
435 <tt>RECEIVE(P, M)</tt>
436 </dt>
437 <dd>
438 A function or event that allows the local node to receive a protocol
439 message <tt>M</tt> as defined in this document from a peer <tt>P</tt>.
440 </dd>
441 <dt>
442 <tt>SEND(P, M)</tt>
443 </dt>
444 <dd>
445 A function that allows the local node to send a protocol message
446 <tt>M</tt> as defined in this document to a peer <tt>P</tt>.
447 If call to SEND fails, the message has not been sent.
448 </dd>
449 <dt>
450 <tt>NETWORK_SIZE_ESTIMATE(S)</tt>
451 </dt>
452 <dd>
453 A function or event that provides estimates on the network size
454 <tt>S</tt> for use in the DHT routing algorithms.
455 FIXME: What is S and give an example.
456 </dd>
457 <dt>
458 <tt>ADDRESS_ADDED(A)</tt>
459 </dt>
460 <dd>
461 The underlay signals us that an address <tt>A</tt> was added for our
462 local node.
463 This information is used to advertise
464 connectivity information to the local node.
465 <tt>A</tt> is a string suitable for inclusion in a HELLO payload
466 <xref target="hello_block"/>.
467 </dd>
468 <dt>
469 <tt>ADDRESS_DELETED(A)</tt>
470 </dt>
471 <dd>
472 The underlay signals us that an address <tt>A</tt> was removed.
473 This information is used, for example, to no longer advertise
474 this address.
475 </dd>
476 <dt>
477 <tt>VERIFY(blob)</tt>
478 </dt>
479 <dd>
480 Signature verification by underlay. FIXME unclear. Required?
481 </dd>
482 </dl>
483 </section>
484 <section>
485 <name>Bootstrapping</name>
486 <t>
487 It is assumed that the node is already connected to at least
488 one other node.
489 First, those initial nodes are sorted into their respective buckets.
490 </t>
491 <t>
492 In order to find the closest nodes in the network to itself, an
493 implementation MUST now periodically send HELLO GET queries for its own
494 node address.
495 Both the "record route" and "find node" message options are set in the
496 GET queries in order to learn nodes and network topology from the
497 message route and in order to receive approximate replies to the
498 query key (the node address).
499 </t>
500 <t>FIXME: Periodically -&gt; more specific? No. Frequency may be adapted depending on network conditions, known nodes, busy/idle etc.</t>
501 <t>
502 Any implementation encountering a HELLO GET request initially
503 sends its own node address if it.
504 </t>
505 </section>
506 <section anchor="routing" numbered="true" toc="default">
507 <name>Routing</name>
508 <section anchor="peer_storage">
509 <name>Peer Storage</name>
386 <t> 510 <t>
387 It is expected that there are basic mechanisms available to 511 A R5N implementation must store the information on connected nodes
388 manage node connectivity and addressing. 512 and update changes accordingly in a local persistance component such
389 The required functionality are abstracted through the following 513 as a database.
390 procedures and events: 514 Upon receiving a connection notification from the Underlay through
515 <tt>NODE_CONNECTED</tt>, information on the new peer is added to the
516 local peer storage.
517 When disconnect is indicated by the Underlay through
518 <tt>NODE_DISCONNECTED</tt> the peer MUST be removed from the local
519 peer storage.
520 The data structure for managing connected nodes and their
521 metadata may be implemented using the k-buckets concept of
522 <xref target="Kademlia"/>.
523 In order to select nodes which are suitable destinations for
524 routing messages, R5N uses a hybrid approach:
525 Given an estimated network size N, the node selection for the
526 first N hops is random. After the initial N hops, node selection
527 follows an XOR-based node distance calculation.
528 </t>
529 </section>
530 <section anchor="routing_table">
531 <name>Routing Table</name>
532 <t>
533 As the message traverses a random path through the network for the
534 first N hops, it is essential that routing loops are avoided.
535 In R5N, a bloomfilter is used as part of the routing metadata in
536 messages. The bloomfilter is updates at each hop with the hops
537 node identity.
538 For the next hop selection in both the random and the deterministic
539 case, any node which is in the bloomfilter for the respective message
540 is not included in the node selection process.
541 </t>
542 <t>
543 The R5N routing component MUST implement the following functions:
391 </t> 544 </t>
392 <dl> 545 <dl>
393 <dt> 546 <dt>
394 <tt>PEER_CONNECTED(P)</tt> 547 <tt>GetDistance(A, B) -&gt; Distance as Integer</tt>
395 </dt>
396 <dd>
397 is a signal that allows the DHT to react to a newly connected peer
398 <tt>N</tt>.
399 Such an event triggers, for example, updates in the
400 routing table.
401 </dd>
402 <dt>
403 <tt>PEER_DISCONNECTED(P)</tt>
404 </dt>
405 <dd>
406 is a signal that allows the DHT to react to a recently disconnected
407 peer.
408 Such an event triggers, for example, updates in the
409 routing table.
410 </dd>
411 <dt>
412 <tt>TRY_CONNECT(N, A)</tt>
413 </dt>
414 <dd>
415 A function which allows the local node to attempt the establishment of
416 a connection to another node <tt>N</tt> using an address <tt>A</tt>.
417 When the connection attempt is successful, information on the new
418 peer is offered through the <tt>PEER_CONNECTED</tt> signal.
419 </dd>
420 <dt>
421 <tt>HOLD(P)</tt>
422 </dt> 548 </dt>
423 <dd> 549 <dd>
424 A function which tells the underlay to keep a hold on the connection 550 this function calculates the binary XOR between A and B.
425 to a peer <tt>P</tt>. FIXME what is this needed for? 551 The resulting distance is interpreted as an integer where
552 the leftmost bit is the most significant bit.
426 </dd> 553 </dd>
427 <dt> 554 <dt>
428 <tt>DROP(P)</tt> 555 <tt>SelectClosestNode(K, B) -&gt; N</tt>
429 </dt> 556 </dt>
430 <dd> 557 <dd>
431 A function which tells the underlay to drop the connection to a 558 This function selects the connected node <tt>N</tt> from our
432 peer <tt>P</tt>. FIXME what is this needed for? 559 routing table with the shortest XOR-distance to the key <tt>K</tt>.
560 This means that for all other nodes <tt>N'</tt> in the routing table
561 <tt>GetDistance(N, K) &lt; GetDistance(N',K)</tt>.
562 Nodes in the bloomfilter <tt>B</tt> are not considered.
433 </dd> 563 </dd>
434 <dt> 564 <dt>
435 <tt>RECEIVE(P, M)</tt> 565 <tt>SelectRandomNode(B) -&gt; N</tt>
436 </dt> 566 </dt>
437 <dd> 567 <dd>
438 A function or event that allows the local node to receive a protocol 568 This function selects a random node <tt>N</tt> from all connected
439 message <tt>M</tt> as defined in this document from a peer <tt>P</tt>. 569 nodes.
570 Nodes in the bloomfilter <tt>B</tt> are not considered.
440 </dd> 571 </dd>
441 <dt> 572 <dt>
442 <tt>SEND(P, M)</tt> 573 <tt>SelectNode(K, H, B) -&gt; N</tt>
443 </dt> 574 </dt>
444 <dd> 575 <dd>
445 A function that allows the local node to send a protocol message 576 This function selects a node <tt>N</tt> depending on the
446 <tt>M</tt> as defined in this document to a peer <tt>P</tt>. 577 number of hops <tt>H</tt> parameter.
447 If call to SEND fails, the message has not been sent. 578 If <tt>H &lt; NETWORK_SIZE_ESTIMATE</tt>
579 this function MUST return <tt>SelectRandomNode()</tt> and
580 <tt>SelectClosestnode(K)</tt> otherwise.
581 Nodes in the bloomfilter <tt>B</tt> are not considered.
448 </dd> 582 </dd>
449 <dt> 583 <dt>
450 <tt>NETWORK_SIZE_ESTIMATE(S)</tt> 584 <tt>IsClosestNode(N, K, B) -&gt; true | false</tt>
451 </dt> 585 </dt>
452 <dd> 586 <dd>
453 A function or event that provides estimates on the network size 587 checks if <tt>N</tt> is the closest node for <tt>K</tt>
454 <tt>S</tt> for use in the DHT routing algorithms. 588 (cf. <tt>SelectClosestNode(K)</tt>).
455 FIXME: What is S and give an example. 589 Nodes in the bloomfilter <tt>B</tt> are not considered.
456 </dd>
457 <dt>
458 <tt>ADDRESS_ADDED(A)</tt>
459 </dt>
460 <dd>
461 The underlay signals us that an address <tt>A</tt> was added for our
462 local node.
463 This information is used to advertise
464 connectivity information to the local node.
465 <tt>A</tt> is a string suitable for inclusion in a HELLO payload
466 <xref target="hello_block"/>.
467 </dd>
468 <dt>
469 <tt>ADDRESS_DELETED(A)</tt>
470 </dt>
471 <dd>
472 The underlay signals us that an address <tt>A</tt> was removed.
473 This information is used, for example, to no longer advertise
474 this address.
475 </dd>
476 <dt>
477 <tt>VERIFY(blob)</tt>
478 </dt>
479 <dd>
480 Signature verification by underlay. FIXME unclear. Required?
481 </dd> 590 </dd>
482 </dl> 591 </dl>
483 </section> 592 </section>
484 <section> 593 </section>
485 <name>Bootstrapping</name> 594 <section anchor="p2p_messages" numbered="true" toc="default">
595 <name>Message Processing</name>
596 <t>
597 The implementation MUST listen for <tt>RECEIVE(P, M)</tt> signals
598 from the Underlay and respond to the respective messages sent by
599 the peer <tt>P</tt>.
600 In the following, the wire formats of the messages and the required
601 processing are detailed.
602 The local node address is referred to as <tt>N</tt>.
603 </t>
604 <section anchor="p2p_bf" numbered="true" toc="default">
605 <name>Bloomfilter</name>
486 <t> 606 <t>
487 It is assumed that the node is already connected to at least 607 In order to prevent circular routes, GET and PUT messages contain
488 one other node. 608 a 128-bit Bloom filter (m=128). The Bloom filter is used to detect duplicate
489 First, those initial nodes are sorted into their respective buckets. 609 node addresses along the route.
490 </t> 610 A Bloom filter "bf" is initially empty, consisting only of zeroes.
491 <t> 611 There are two functions which can be invoked on the Bloom filter:
492 In order to find the closest nodes in the network to itself, an 612 BF-SET(bf, e) and BF-TEST(bf, e) where "e" is an element which is to
493 implementation MUST now periodically send HELLO GET queries for its own 613 be added to the Bloom filter or queried against the set.
494 node address. 614 Any bloom filter uses k=16 different hash functions each of which is
495 Both the "record route" and "find node" message options are set in the 615 defined as follows:
496 GET queries in order to learn nodes and network topology from the
497 message route and in order to receive approximate replies to the
498 query key (the node address).
499 </t>
500 <t>FIXME: Periodically -&gt; more specific? No. Frequency may be adapted depending on network conditions, known nodes, busy/idle etc.</t>
501 <t>
502 Any implementation encountering a HELLO GET request initially
503 sends its own node address if it.
504 </t> 616 </t>
505 </section> 617 </section>
506 <section anchor="routing" numbered="true" toc="default"> 618 <section anchor="p2p_xq" numbered="true" toc="default">
507 <name>Routing</name> 619 <name>Extended query</name>
508 <section anchor="peer_storage"> 620 <t>TODO: Talk about XQuery in the context of messages.</t>
509 <name>Peer Storage</name>
510 <t>
511 A R5N implementation must store the information on connected nodes
512 and update changes accordingly in a local persistance component such
513 as a database.
514 Upon receiving a connection notification from the Underlay through
515 <tt>NODE_CONNECTED</tt>, information on the new peer is added to the
516 local peer storage.
517 When disconnect is indicated by the Underlay through
518 <tt>NODE_DISCONNECTED</tt> the peer MUST be removed from the local
519 peer storage.
520 The data structure for managing connected nodes and their
521 metadata may be implemented using the k-buckets concept of
522 <xref target="Kademlia"/>.
523 In order to select nodes which are suitable destinations for
524 routing messages, R5N uses a hybrid approach:
525 Given an estimated network size N, the node selection for the
526 first N hops is random. After the initial N hops, node selection
527 follows an XOR-based node distance calculation.
528 </t>
529 </section>
530 <section anchor="routing_table">
531 <name>Routing Table</name>
532 <t>
533 As the message traverses a random path through the network for the
534 first N hops, it is essential that routing loops are avoided.
535 In R5N, a bloomfilter is used as part of the routing metadata in
536 messages. The bloomfilter is updates at each hop with the hops
537 node identity.
538 For the next hop selection in both the random and the deterministic
539 case, any node which is in the bloomfilter for the respective message
540 is not included in the node selection process.
541 </t>
542 <t>
543 The R5N routing component MUST implement the following functions:
544 </t>
545 <dl>
546 <dt>
547 <tt>GetDistance(A, B) -&gt; Distance as Integer</tt>
548 </dt>
549 <dd>
550 this function calculates the binary XOR between A and B.
551 The resulting distance is interpreted as an integer where
552 the leftmost bit is the most significant bit.
553 </dd>
554 <dt>
555 <tt>SelectClosestNode(K, B) -&gt; N</tt>
556 </dt>
557 <dd>
558 This function selects the connected node <tt>N</tt> from our
559 routing table with the shortest XOR-distance to the key <tt>K</tt>.
560 This means that for all other nodes <tt>N'</tt> in the routing table
561 <tt>GetDistance(N, K) &lt; GetDistance(N',K)</tt>.
562 Nodes in the bloomfilter <tt>B</tt> are not considered.
563 </dd>
564 <dt>
565 <tt>SelectRandomNode(B) -&gt; N</tt>
566 </dt>
567 <dd>
568 This function selects a random node <tt>N</tt> from all connected
569 nodes.
570 Nodes in the bloomfilter <tt>B</tt> are not considered.
571 </dd>
572 <dt>
573 <tt>SelectNode(K, H, B) -&gt; N</tt>
574 </dt>
575 <dd>
576 This function selects a node <tt>N</tt> depending on the
577 number of hops <tt>H</tt> parameter.
578 If <tt>H &lt; NETWORK_SIZE_ESTIMATE</tt>
579 this function MUST return <tt>SelectRandomNode()</tt> and
580 <tt>SelectClosestnode(K)</tt> otherwise.
581 Nodes in the bloomfilter <tt>B</tt> are not considered.
582 </dd>
583 <dt>
584 <tt>IsClosestNode(N, K, B) -&gt; true | false</tt>
585 </dt>
586 <dd>
587 checks if <tt>N</tt> is the closest node for <tt>K</tt>
588 (cf. <tt>SelectClosestNode(K)</tt>).
589 Nodes in the bloomfilter <tt>B</tt> are not considered.
590 </dd>
591 </dl>
592 </section>
593 </section> 621 </section>
594 <section anchor="p2p_messages" numbered="true" toc="default"> 622 <section anchor="p2p_put" numbered="true" toc="default">
595 <name>Message Processing</name> 623 <name>PutMessage</name>
596 <t> 624 <section anchor="p2p_put_wire">
597 The implementation MUST listen for <tt>RECEIVE(P, M)</tt> signals 625 <name>Wire Format</name>
598 from the Underlay and respond to the respective messages sent by 626 <figure anchor="figure_putmsg">
599 the peer <tt>P</tt>. 627 <artwork name="" type="" align="left" alt=""><![CDATA[
600 In the following, the wire formats of the messages and the required
601 processing are detailed.
602 The local node address is referred to as <tt>N</tt>.
603 </t>
604 <section anchor="p2p_bf" numbered="true" toc="default">
605 <name>Bloomfilter</name>
606 <t>
607 In order to prevent circular routes, GET and PUT messages contain
608 a 128-bit Bloom filter (m=128). The Bloom filter is used to detect duplicate
609 node addresses along the route.
610 A Bloom filter "bf" is initially empty, consisting only of zeroes.
611 There are two functions which can be invoked on the Bloom filter:
612 BF-SET(bf, e) and BF-TEST(bf, e) where "e" is an element which is to
613 be added to the Bloom filter or queried against the set.
614 Any bloom filter uses k=16 different hash functions each of which is
615 defined as follows:
616 </t>
617 </section>
618 <section anchor="p2p_xq" numbered="true" toc="default">
619 <name>Extended query</name>
620 <t>TODO: Talk about XQuery in the context of messages.</t>
621 </section>
622 <section anchor="p2p_put" numbered="true" toc="default">
623 <name>PutMessage</name>
624 <section anchor="p2p_put_wire">
625 <name>Wire Format</name>
626 <figure anchor="figure_putmsg">
627 <artwork name="" type="" align="left" alt=""><![CDATA[
6280 8 16 24 32 40 48 56 6280 8 16 24 32 40 48 56
629+-----+-----+-----+-----+-----+-----+-----+-----+ 629+-----+-----+-----+-----+-----+-----+-----+-----+
630| MSIZE | MTYPE | BTYPE | 630| MSIZE | MTYPE | BTYPE |
@@ -644,132 +644,132 @@ Connectivity | |Underlay| |Underlay|
644/ BLOCK (variable length) / 644/ BLOCK (variable length) /
645+-----+-----+-----+-----+-----+-----+-----+-----+ 645+-----+-----+-----+-----+-----+-----+-----+-----+
646]]></artwork> 646]]></artwork>
647 </figure> 647 </figure>
648 <t>where:</t> 648 <t>where:</t>
649 <dl> 649 <dl>
650 <dt>MSIZE</dt> 650 <dt>MSIZE</dt>
651 <dd> 651 <dd>
652 denotes the size of this message in network byte order. 652 denotes the size of this message in network byte order.
653 </dd> 653 </dd>
654 <dt>MTYPE</dt> 654 <dt>MTYPE</dt>
655 <dd> 655 <dd>
656 is the 16-bit message type. This type can be one of the DHT message 656 is the 16-bit message type. This type can be one of the DHT message
657 types but for put messages it must be set to 657 types but for put messages it must be set to
658 the value 146 in network byte order. 658 the value 146 in network byte order.
659 </dd> 659 </dd>
660 <dt>BTYPE</dt> 660 <dt>BTYPE</dt>
661 <dd> 661 <dd>
662 is a 32-bit block type field. The block type indicates the content 662 is a 32-bit block type field. The block type indicates the content
663 type of the payload. In network byte order. 663 type of the payload. In network byte order.
664 </dd> 664 </dd>
665 <dt>OPTIONS</dt> 665 <dt>OPTIONS</dt>
666 <dd> 666 <dd>
667 is a 16-bit options field (see below). 667 is a 16-bit options field (see below).
668 </dd> 668 </dd>
669 <dt>HOPCOUNT</dt> 669 <dt>HOPCOUNT</dt>
670 <dd> 670 <dd>
671 is a 16-bit number indicating how many hops this message has 671 is a 16-bit number indicating how many hops this message has
672 traversed to far. In network byte order. 672 traversed to far. In network byte order.
673 </dd> 673 </dd>
674 <dt>REPL_LVL</dt> 674 <dt>REPL_LVL</dt>
675 <dd> 675 <dd>
676 is a 16-bit number indicating the desired replication level of 676 is a 16-bit number indicating the desired replication level of
677 the data. In network byte order. 677 the data. In network byte order.
678 </dd> 678 </dd>
679 <dt>PATH_LEN</dt> 679 <dt>PATH_LEN</dt>
680 <dd> 680 <dd>
681 is a 16-bit number indicating the length of the PUT path recorded 681 is a 16-bit number indicating the length of the PUT path recorded
682 in PUTPATH. 682 in PUTPATH.
683 As PUTPATH is optional, this value may be zero. 683 As PUTPATH is optional, this value may be zero.
684 In network byte order. 684 In network byte order.
685 </dd> 685 </dd>
686 <dt>EXPIRATION</dt> 686 <dt>EXPIRATION</dt>
687 <dd> 687 <dd>
688 denotes the absolute 64-bit expiration date of the content. 688 denotes the absolute 64-bit expiration date of the content.
689 In microseconds since midnight (0 hour), January 1, 1970 in network 689 In microseconds since midnight (0 hour), January 1, 1970 in network
690 byte order. 690 byte order.
691 </dd> 691 </dd>
692 <dt>BLOOMFILTER</dt> 692 <dt>BLOOMFILTER</dt>
693 <dd> 693 <dd>
694 A bloomfilter (for node addresses) to stop circular routes. 694 A bloomfilter (for node addresses) to stop circular routes.
695 </dd> 695 </dd>
696 <dt>KEY</dt> 696 <dt>KEY</dt>
697 <dd> 697 <dd>
698 The key under which the PUT request wants to store content 698 The key under which the PUT request wants to store content
699 under. 699 under.
700 </dd> 700 </dd>
701 <dt>PUTPATH</dt> 701 <dt>PUTPATH</dt>
702 <dd> 702 <dd>
703 the variable-length PUT path. 703 the variable-length PUT path.
704 The path consists of a list of PATH_LEN node addresses. 704 The path consists of a list of PATH_LEN node addresses.
705 </dd> 705 </dd>
706 <dt>BLOCK</dt> 706 <dt>BLOCK</dt>
707 <dd> 707 <dd>
708 the variable-length block payload. The contents are determined 708 the variable-length block payload. The contents are determined
709 by the BTYPE field. 709 by the BTYPE field.
710 </dd> 710 </dd>
711 </dl> 711 </dl>
712 </section> 712 </section>
713 <section anchor="p2p_put_processing"> 713 <section anchor="p2p_put_processing">
714 <name>Processing</name> 714 <name>Processing</name>
715 <t> 715 <t>
716 Upon receiving a <tt>PutMessage</tt> from a peer <tt>P</tt>. 716 Upon receiving a <tt>PutMessage</tt> from a peer <tt>P</tt>.
717 An implementation MUST process it step by step as follows: 717 An implementation MUST process it step by step as follows:
718 </t> 718 </t>
719 <ol> 719 <ol>
720 <li> 720 <li>
721 The <tt>EXPIRATION</tt> field is evaluated. 721 The <tt>EXPIRATION</tt> field is evaluated.
722 If the message is expired, it MUST be discarded. 722 If the message is expired, it MUST be discarded.
723 </li> 723 </li>
724 <li> 724 <li>
725 If the <tt>BTYPE</tt> is not supported by the implementation, 725 If the <tt>BTYPE</tt> is not supported by the implementation,
726 no validation of the block payload is performed and processing 726 no validation of the block payload is performed and processing
727 continues at (4). 727 continues at (4).
728 Else, the block MUST be validated as defined in (3). 728 Else, the block MUST be validated as defined in (3).
729 </li> 729 </li>
730 <li> 730 <li>
731 The block payload of the message is evaluated using according 731 The block payload of the message is evaluated using according
732 to the <tt>BTYPE</tt> using the respective 732 to the <tt>BTYPE</tt> using the respective
733 <tt>ValidateBlockStoreRequest</tt> procedure. 733 <tt>ValidateBlockStoreRequest</tt> procedure.
734 If the block payload is invalid or does not match the key, 734 If the block payload is invalid or does not match the key,
735 it MUST be discarded. 735 it MUST be discarded.
736 </li> 736 </li>
737 <li> 737 <li>
738 The node address of the sender peer <tt>P</tt> SHOULD be in <tt>BLOOMFILTER</tt>. 738 The node address of the sender peer <tt>P</tt> SHOULD be in <tt>BLOOMFILTER</tt>.
739 If not, the implementation MAY log an error, but MUST continue. 739 If not, the implementation MAY log an error, but MUST continue.
740 </li> 740 </li>
741 <li> 741 <li>
742 If the <tt>RecordRoute</tt> flag is set in OPTIONS, 742 If the <tt>RecordRoute</tt> flag is set in OPTIONS,
743 the local node address MUST be appended to the <tt>PUTPATH</tt> 743 the local node address MUST be appended to the <tt>PUTPATH</tt>
744 of the message. 744 of the message.
745 </li> 745 </li>
746 <li> 746 <li>
747 If the local node is the closest node 747 If the local node is the closest node
748 (cf. <tt>IsClosestNode(N, Key)</tt>) or the <tt>DemultiplexEverywhere</tt> 748 (cf. <tt>IsClosestNode(N, Key)</tt>) or the <tt>DemultiplexEverywhere</tt>
749 options flag ist set, the message MUST 749 options flag ist set, the message MUST
750 be stored locally in the block storage. 750 be stored locally in the block storage.
751 </li> 751 </li>
752 <li> 752 <li>
753 Given the value in <tt>REPL_LVL</tt>, the number of peers to 753 Given the value in <tt>REPL_LVL</tt>, the number of peers to
754 forward to MUST be calculated. If there is at least one 754 forward to MUST be calculated. If there is at least one
755 peers to forward to, the implementation SHOULD select up to this 755 peers to forward to, the implementation SHOULD select up to this
756 number of peers to forward the message to. The implementation MAY 756 number of peers to forward the message to. The implementation MAY
757 forward to fewer or no peers in order to handle resource constraints 757 forward to fewer or no peers in order to handle resource constraints
758 such as bandwidth. 758 such as bandwidth.
759 Finally, the local node address MUST be added to the 759 Finally, the local node address MUST be added to the
760 <tt>BLOOMFILTER</tt> of the forwarded message. 760 <tt>BLOOMFILTER</tt> of the forwarded message.
761 For all peers with node address <tt>P</tt> chosen to forward the message 761 For all peers with node address <tt>P</tt> chosen to forward the message
762 to, <tt>SEND(P, PutMessage)</tt> is called. 762 to, <tt>SEND(P, PutMessage)</tt> is called.
763 </li> 763 </li>
764 </ol> 764 </ol>
765 </section>
766 </section> 765 </section>
767 <section anchor="p2p_get" numbered="true" toc="default"> 766 </section>
768 <name>GetMessage</name> 767 <section anchor="p2p_get" numbered="true" toc="default">
769 <section anchor="p2p_get_wire"> 768 <name>GetMessage</name>
770 <name>Wire Format</name> 769 <section anchor="p2p_get_wire">
771 <figure anchor="figure_getmsg"> 770 <name>Wire Format</name>
772 <artwork name="" type="" align="left" alt=""><![CDATA[ 771 <figure anchor="figure_getmsg">
772 <artwork name="" type="" align="left" alt=""><![CDATA[
7730 8 16 24 32 40 48 56 7730 8 16 24 32 40 48 56
774+-----+-----+-----+-----+-----+-----+-----+-----+ 774+-----+-----+-----+-----+-----+-----+-----+-----+
775| MSIZE | MTYPE | BTYPE | 775| MSIZE | MTYPE | BTYPE |
@@ -789,134 +789,134 @@ Connectivity | |Underlay| |Underlay|
789/ BF_RESULT (variable length) / 789/ BF_RESULT (variable length) /
790+-----+-----+-----+-----+-----+-----+-----+-----+ 790+-----+-----+-----+-----+-----+-----+-----+-----+
791]]></artwork> 791]]></artwork>
792 </figure> 792 </figure>
793 <t>where:</t> 793 <t>where:</t>
794 <dl> 794 <dl>
795 <dt>MSIZE</dt> 795 <dt>MSIZE</dt>
796 <dd> 796 <dd>
797 denotes the size of this message in network byte order. 797 denotes the size of this message in network byte order.
798 </dd> 798 </dd>
799 <dt>MTYPE</dt> 799 <dt>MTYPE</dt>
800 <dd> 800 <dd>
801 is the 16-bit message type. It must be set to 801 is the 16-bit message type. It must be set to
802 the value 147 in network byte order. 802 the value 147 in network byte order.
803 </dd> 803 </dd>
804 <dt>BTYPE</dt> 804 <dt>BTYPE</dt>
805 <dd> 805 <dd>
806 is a 32-bit block type field. The block type indicates the content 806 is a 32-bit block type field. The block type indicates the content
807 type of the payload. In network byte order. 807 type of the payload. In network byte order.
808 </dd> 808 </dd>
809 <dt>OPTIONS</dt> 809 <dt>OPTIONS</dt>
810 <dd> 810 <dd>
811 is a 16-bit options field (see below). 811 is a 16-bit options field (see below).
812 </dd> 812 </dd>
813 <dt>HOPCOUNT</dt> 813 <dt>HOPCOUNT</dt>
814 <dd> 814 <dd>
815 is a 16-bit number indicating how many hops this message has 815 is a 16-bit number indicating how many hops this message has
816 traversed to far. In network byte order. 816 traversed to far. In network byte order.
817 </dd> 817 </dd>
818 <dt>REPL_LVL</dt> 818 <dt>REPL_LVL</dt>
819 <dd> 819 <dd>
820 is a 16-bit number indicating the desired replication level of 820 is a 16-bit number indicating the desired replication level of
821 the data. In network byte order. 821 the data. In network byte order.
822 </dd> 822 </dd>
823 <dt>XQ_SIZE</dt> 823 <dt>XQ_SIZE</dt>
824 <dd> 824 <dd>
825 is a 32-bit number indicating the length of the optional 825 is a 32-bit number indicating the length of the optional
826 extended query XQUERY. In network byte order. 826 extended query XQUERY. In network byte order.
827 </dd> 827 </dd>
828 <dt>BLOOMFILTER</dt> 828 <dt>BLOOMFILTER</dt>
829 <dd> 829 <dd>
830 A bloomfilter (for node identities) to stop circular routes. 830 A bloomfilter (for node identities) to stop circular routes.
831 </dd> 831 </dd>
832 <dt>KEY</dt> 832 <dt>KEY</dt>
833 <dd> 833 <dd>
834 The key under which the PUT request wants to store content 834 The key under which the PUT request wants to store content
835 under. 835 under.
836 </dd> 836 </dd>
837 <dt>XQUERY</dt> 837 <dt>XQUERY</dt>
838 <dd> 838 <dd>
839 the variable-length extended query. Optional. 839 the variable-length extended query. Optional.
840 </dd> 840 </dd>
841 <dt>BF_MUTATOR</dt> 841 <dt>BF_MUTATOR</dt>
842 <dd> 842 <dd>
843 The 32-bit bloomfilter mutator for the result bloomfilter. 843 The 32-bit bloomfilter mutator for the result bloomfilter.
844 </dd> 844 </dd>
845 <dt>RESULT_BF</dt> 845 <dt>RESULT_BF</dt>
846 <dd> 846 <dd>
847 the variable-length result bloomfilter. 847 the variable-length result bloomfilter.
848 </dd> 848 </dd>
849 </dl> 849 </dl>
850 </section>
851 <section anchor="p2p_get_processing">
852 <name>Processing</name>
853 <t>
854 Upon receiving a <tt>GetMmessage</tt> from a peer an
855 implementation MUST process it step by step as follows:
856 </t>
857 <ol>
858 <li>
859 The <tt>KEY</tt> and <tt>XQUERY</tt> is validated against the
860 requested <tt>BTYPE</tt> as defined by its respective
861 <tt>ValidateBlockQuery</tt> procedure.
862 If the <tt>BTYPE</tt> is not supported, or if the block key
863 does not match or if the <tt>XQUERY</tt> is malformed,
864 the message MUST be discarded.
865 </li>
866 <li>
867 The node address of the sender peer <tt>P</tt> SHOULD be in the
868 BLOOMFILTER. If not, the
869 implementation MAY log an error, but MUST continue.
870 </li>
871 <li>
872 <t>
873 If the local node is the closest node
874 (cf. <tt>IsClosestNode (N, Key)</tt>) or the
875 <tt>DemultiplexEverywhere</tt> options flag is set, a reply
876 MUST be produced:
877 </t>
878 <ol>
879 <li>
880 If <tt>OPTIONS</tt> indicate a <tt>FindNode</tt> request,
881 FIXME the node selection
882 foo from buckets that probably needs fixing. Take into account
883 <tt>REPLY_BF</tt>
884 </li>
885 <li>
886 Else, if there is a block in the local Block Storage which is
887 not already in the <tt>RESULT_BF</tt>,
888 a ResultMessage MUST be sent.
889 FIXME link to how the result is sent?
890 </li>
891 </ol>
892 </li>
893 <li>
894 FIXME: We only handle if not GNUNET_BLOCK_EVALUATION_OK_LAST.
895 This means that we must evaluate the Reply produced in the
896 previous step using <tt>ValidateBlockReply</tt> for this
897 <tt>BTYPE</tt>
898 </li>
899 <li>
900 Given the value in REPL_LVL, the number of nodes to forward to
901 MUST be calculated (NUM-FORWARD-nodeS). If there is at least one
902 node to forward to, the implementation SHOULD select up to this
903 number of nodes to forward the message to. The implementation MAY
904 forward to fewer or no nodes in order to handle resource constraints
905 such as bandwidth.
906 The message BLOOMFILTER MUST be updated with the local node
907 address <tt>N</tt>.
908 For all peers with node address <tt>P'</tt> chosen to forward the message
909 to, <tt>SEND(P', PutMessage)</tt> is called.
910 </li>
911 </ol>
912 </section>
913 </section> 850 </section>
914 <section anchor="p2p_result" numbered="true" toc="default"> 851 <section anchor="p2p_get_processing">
915 <name>ResultMessage</name> 852 <name>Processing</name>
916 <section anchor="p2p_result_wire"> 853 <t>
917 <name>Wire Format</name> 854 Upon receiving a <tt>GetMmessage</tt> from a peer an
918 <figure anchor="figure_resmsg"> 855 implementation MUST process it step by step as follows:
919 <artwork name="" type="" align="left" alt=""><![CDATA[ 856 </t>
857 <ol>
858 <li>
859 The <tt>KEY</tt> and <tt>XQUERY</tt> is validated against the
860 requested <tt>BTYPE</tt> as defined by its respective
861 <tt>ValidateBlockQuery</tt> procedure.
862 If the <tt>BTYPE</tt> is not supported, or if the block key
863 does not match or if the <tt>XQUERY</tt> is malformed,
864 the message MUST be discarded.
865 </li>
866 <li>
867 The node address of the sender peer <tt>P</tt> SHOULD be in the
868 BLOOMFILTER. If not, the
869 implementation MAY log an error, but MUST continue.
870 </li>
871 <li>
872 <t>
873 If the local node is the closest node
874 (cf. <tt>IsClosestNode (N, Key)</tt>) or the
875 <tt>DemultiplexEverywhere</tt> options flag is set, a reply
876 MUST be produced:
877 </t>
878 <ol>
879 <li>
880 If <tt>OPTIONS</tt> indicate a <tt>FindNode</tt> request,
881 FIXME the node selection
882 foo from buckets that probably needs fixing. Take into account
883 <tt>REPLY_BF</tt>
884 </li>
885 <li>
886 Else, if there is a block in the local Block Storage which is
887 not already in the <tt>RESULT_BF</tt>,
888 a ResultMessage MUST be sent.
889 FIXME link to how the result is sent?
890 </li>
891 </ol>
892 </li>
893 <li>
894 FIXME: We only handle if not GNUNET_BLOCK_EVALUATION_OK_LAST.
895 This means that we must evaluate the Reply produced in the
896 previous step using <tt>ValidateBlockReply</tt> for this
897 <tt>BTYPE</tt>
898 </li>
899 <li>
900 Given the value in REPL_LVL, the number of nodes to forward to
901 MUST be calculated (NUM-FORWARD-nodeS). If there is at least one
902 node to forward to, the implementation SHOULD select up to this
903 number of nodes to forward the message to. The implementation MAY
904 forward to fewer or no nodes in order to handle resource constraints
905 such as bandwidth.
906 The message BLOOMFILTER MUST be updated with the local node
907 address <tt>N</tt>.
908 For all peers with node address <tt>P'</tt> chosen to forward the message
909 to, <tt>SEND(P', PutMessage)</tt> is called.
910 </li>
911 </ol>
912 </section>
913 </section>
914 <section anchor="p2p_result" numbered="true" toc="default">
915 <name>ResultMessage</name>
916 <section anchor="p2p_result_wire">
917 <name>Wire Format</name>
918 <figure anchor="figure_resmsg">
919 <artwork name="" type="" align="left" alt=""><![CDATA[
9200 8 16 24 32 40 48 56 9200 8 16 24 32 40 48 56
921+-----+-----+-----+-----+-----+-----+-----+-----+ 921+-----+-----+-----+-----+-----+-----+-----+-----+
922| MSIZE | MTYPE | BTYPE | 922| MSIZE | MTYPE | BTYPE |
@@ -938,229 +938,229 @@ Connectivity | |Underlay| |Underlay|
938/ (variable length) / 938/ (variable length) /
939+-----+-----+-----+-----+-----+-----+-----+-----+ 939+-----+-----+-----+-----+-----+-----+-----+-----+
940]]></artwork> 940]]></artwork>
941 </figure> 941 </figure>
942 <t>where:</t> 942 <t>where:</t>
943 <dl>
944 <dt>MSIZE</dt>
945 <dd>
946 denotes the size of this message in network byte order.
947 </dd>
948 <dt>MTYPE</dt>
949 <dd>
950 is the 16-bit message type. This type can be one of the DHT message
951 types but for put messages it must be set to
952 the value 148 in network byte order.
953 </dd>
954 <dt>OPTIONS</dt>
955 <dd>
956 is a 16-bit options field (see below).
957 </dd>
958 <dt>BTYPE</dt>
959 <dd>
960 is a 32-bit block type field. The block type indicates the content
961 type of the payload. In network byte order.
962 </dd>
963 <dt>PUTPATH_L</dt>
964 <dd>
965 is a 16-bit number indicating the length of the PUT path recorded
966 in PUTPATH. As PUTPATH is optiona, this value may be zero.
967 In network byte order.
968 </dd>
969 <dt>GET_PATH_LEN</dt>
970 <dd>
971 is a 16-bit number indicating the length of the GET path recorded
972 in GETPATH. As PUTPATH is optiona, this value may be zero.
973 In network byte order.
974 </dd>
975 <dt>EXPIRATION</dt>
976 <dd>
977 denotes the absolute 64-bit expiration date of the content.
978 In microseconds since midnight (0 hour), January 1, 1970 in network
979 byte order.
980 </dd>
981 <dt>KEY</dt>
982 <dd>
983 The key under which the PUT request wants to store content
984 under.
985 </dd>
986 <dt>PUTPATH</dt>
987 <dd>
988 the variable-length PUT path.
989 The path consists of a list of PATH_LEN node addresses.
990 </dd>
991 <dt>GETPATH</dt>
992 <dd>
993 the variable-length PUT path.
994 The path consists of a list of PATH_LEN node addresses.
995 </dd>
996 <dt>BLOCK</dt>
997 <dd>
998 the variable-length resource record data payload.
999 The contents are defined by the respective type of the resource record.
1000 </dd>
1001 </dl>
1002 </section>
1003 <section anchor="p2p_result_processing">
1004 <name>Processing</name>
1005 <t>
1006 Upon receiving a <tt>ResultMessage</tt> from a connected node.
1007 An implementation MUST process it step by step as follows:
1008 </t>
1009 <ol>
1010 <li>
1011 The <tt>EXPIRATION</tt> field is evaluated.
1012 If the message is expired, it MUST be discarded.
1013 </li>
1014 <li>
1015 If the MTYPE of the message indicates a HELLO block, it
1016 must be validated according to <xref target="hello_block"/>.
1017 The payload MUST be considered for the local routing table by
1018 trying to establish a connection to the node using the information
1019 from the HELLO block. If a connection can be established,
1020 the node is added to the <tt>KBuckets</tt> routing table.
1021 </li>
1022 <li>
1023 If the sender node of the message is already found in the
1024 GETPATH, the path MUST be truncated at this position.
1025 The implementation MAY log a warning in such a case.
1026 </li>
1027 <li>
1028 If the <tt>KEY</tt> of this <tt>ResultMessage</tt> is found in the
1029 list of pending local queries, the <tt>KEY</tt> and <tt>XQUERY</tt>
1030 are validated against the requested BTYPE using the respective
1031 block type implementation of <tt>ValidateBlockReply</tt>.
1032 If the BTYPE is not supported, or if the block key
1033 does not match the BTYPE or if the XQUERY is malformed,
1034 the message MUST be discarded.
1035 </li>
1036 <li>
1037 The implementation MAY cache RESULT messages.
1038 </li>
1039 <li>
1040 <t>
1041 If requests by other nodes for this KEY or BTYPE are known,
1042 the result block is validated against each request using
1043 the respective <tt>ValidateBlockReply</tt> function.
1044 </t>
1045 <t>
1046 If the request options include <tt>FindNode</tt> and the result
1047 message block type is HELLO the block validation must use the
1048 key derived using <tt>DeriveBlockKey</tt> as the key included in
1049 the request is only approximate. (FIXME: So we extract the key
1050 to then check it again against the block? This does not make sense...)
1051 </t>
1052 <t>
1053 If the result message block cannot be verified against the
1054 <tt>KEY</tt> of the result message or if <tt>BLOCK</tt> is
1055 malformed, the message MUST be discarded.
1056 </t>
1057 <t>
1058 For each pending request the reply is routed to the requesting
1059 node <tt>N'</tt>. FIXME routed to node or forwarded to peer?
1060 </t>
1061 </li>
1062 </ol>
1063 </section>
1064 </section>
1065 </section>
1066 <section anchor="blockstorage" numbered="true" toc="default">
1067 <name>Block Storage</name>
1068 <section>
1069 <name>Block Processing</name>
1070 <t>RequestEvaluationResult</t>
1071 <dl> 943 <dl>
1072 <dt>REQUEST_VALID</dt> 944 <dt>MSIZE</dt>
1073 <dd>Query is valid, no reply given.</dd>
1074 <dt>REQUEST_INVALID</dt>
1075 <dd> 945 <dd>
1076 Query format does not match block type. For example, XQuery not 946 denotes the size of this message in network byte order.
1077 given or of size of XQuery is not appropriate for type.
1078 </dd> 947 </dd>
1079 </dl> 948 <dt>MTYPE</dt>
1080 <t>ReplyEvaluationResult</t>
1081 <dl>
1082 <dt>OK_MORE</dt>
1083 <dd>Valid result, and there may be more.</dd>
1084 <dt>OK_LAST</dt>
1085 <dd>Last possible valid result.</dd>
1086 <dt>OK_DUPLICATE</dt>
1087 <dd>Valid result, but duplicate.</dd>
1088 <dt>RESULT_INVALID</dt>
1089 <dd>Invalid result. Block does not match query. Value = 4.</dd>
1090 <dt>RESULT_IRRELEVANT</dt>
1091 <dd>Block does not match xquery. Valid result, but not relevant for the request.</dd>
1092 </dl>
1093 </section>
1094 <section anchor="block_functions">
1095 <name>Block Functions</name>
1096 <t>
1097 Any block type implementation MUST implement the following functions.
1098 </t>
1099 <dl>
1100 <dt>ValidateBlockQuery(Key, XQuery) -&gt; RequestEvaluationResult</dt>
1101 <dd> 949 <dd>
1102 is used to evaluate the request for a block. It is used as part of 950 is the 16-bit message type. This type can be one of the DHT message
1103 <tt>GetMessage</tt> processing, where the block payload is still unkown, 951 types but for put messages it must be set to
1104 but the block <tt>XQuery</tt> (FIXME: Undefined here) and <tt>Key</tt> can and 952 the value 148 in network byte order.
1105 MUST be verified, if possible.
1106 </dd> 953 </dd>
1107 <dt>ValidateBlockStoreRequest(Block, Key) -&gt; RequestEvaluationResult</dt> 954 <dt>OPTIONS</dt>
1108 <dd> 955 <dd>
1109 is used to evaluate a block including its key and payload. 956 is a 16-bit options field (see below).
1110 It is used as part of <tt>PutMessage</tt> processing.
1111 The validation MUST include a check of the block payload against the
1112 <tt>Key</tt> under which it is requested to be stored.
1113 </dd> 957 </dd>
1114 <dt>ValidateBlockReply(Block, XQuery, Key) -&gt; ReplyEvaluationResult</dt> 958 <dt>BTYPE</dt>
1115 <dd> 959 <dd>
1116 is used to evaluate a block including its <tt>Key</tt> and payload. 960 is a 32-bit block type field. The block type indicates the content
1117 It is used as part <tt>ResultMessage</tt> processing. 961 type of the payload. In network byte order.
1118 The validation of the respective <tt>Block</tt> requires a pending local query
1119 or a previously routed request of another node and its associated
1120 XQuery data and Key.
1121 The validation MUST include a check of the block payload against the
1122 key under which it is requested to be stored.
1123 </dd> 962 </dd>
1124 <dt>DeriveBlockKey(Block) -&gt; Key</dt> 963 <dt>PUTPATH_L</dt>
1125 <dd> 964 <dd>
1126 is used to synthesize the block key from the block payload 965 is a 16-bit number indicating the length of the PUT path recorded
1127 and metadata. It is used as part of FIND-node message processing. 966 in PUTPATH. As PUTPATH is optiona, this value may be zero.
967 In network byte order.
1128 </dd> 968 </dd>
1129 <dt>FilterResult(Block, XQuery, Key) -&gt; ReplyEvaluationResult</dt> 969 <dt>GET_PATH_LEN</dt>
1130 <dd> 970 <dd>
1131 is used to filter results stored in the local block storage for local 971 is a 16-bit number indicating the length of the GET path recorded
1132 queries. Locally stored blocks from previously observed 972 in GETPATH. As PUTPATH is optiona, this value may be zero.
1133 <tt>ResultMessages</tt> and <tt>PutMessages</tt> MAY use this 973 In network byte order.
1134 function instead of <tt>ValidateBlockReply</tt> in order to 974 </dd>
1135 avoid revalidation of the block and only perform filtering based on 975 <dt>EXPIRATION</dt>
1136 request parameters. 976 <dd>
977 denotes the absolute 64-bit expiration date of the content.
978 In microseconds since midnight (0 hour), January 1, 1970 in network
979 byte order.
980 </dd>
981 <dt>KEY</dt>
982 <dd>
983 The key under which the PUT request wants to store content
984 under.
985 </dd>
986 <dt>PUTPATH</dt>
987 <dd>
988 the variable-length PUT path.
989 The path consists of a list of PATH_LEN node addresses.
990 </dd>
991 <dt>GETPATH</dt>
992 <dd>
993 the variable-length PUT path.
994 The path consists of a list of PATH_LEN node addresses.
995 </dd>
996 <dt>BLOCK</dt>
997 <dd>
998 the variable-length resource record data payload.
999 The contents are defined by the respective type of the resource record.
1137 </dd> 1000 </dd>
1138 </dl> 1001 </dl>
1139 </section> 1002 </section>
1140 <section anchor="block_types"> 1003 <section anchor="p2p_result_processing">
1141 <name>Block Types</name> 1004 <name>Processing</name>
1142 <t> 1005 <t>
1143 Applications can and should define their own block types. 1006 Upon receiving a <tt>ResultMessage</tt> from a connected node.
1144 The block type determines the format and handling of the block 1007 An implementation MUST process it step by step as follows:
1145 payload by nodes in PUT and RESULT messages.
1146 Block types MUST be registered with GANA <xref target="gana"/>.
1147 </t> 1008 </t>
1009 <ol>
1010 <li>
1011 The <tt>EXPIRATION</tt> field is evaluated.
1012 If the message is expired, it MUST be discarded.
1013 </li>
1014 <li>
1015 If the MTYPE of the message indicates a HELLO block, it
1016 must be validated according to <xref target="hello_block"/>.
1017 The payload MUST be considered for the local routing table by
1018 trying to establish a connection to the node using the information
1019 from the HELLO block. If a connection can be established,
1020 the node is added to the <tt>KBuckets</tt> routing table.
1021 </li>
1022 <li>
1023 If the sender node of the message is already found in the
1024 GETPATH, the path MUST be truncated at this position.
1025 The implementation MAY log a warning in such a case.
1026 </li>
1027 <li>
1028 If the <tt>KEY</tt> of this <tt>ResultMessage</tt> is found in the
1029 list of pending local queries, the <tt>KEY</tt> and <tt>XQUERY</tt>
1030 are validated against the requested BTYPE using the respective
1031 block type implementation of <tt>ValidateBlockReply</tt>.
1032 If the BTYPE is not supported, or if the block key
1033 does not match the BTYPE or if the XQUERY is malformed,
1034 the message MUST be discarded.
1035 </li>
1036 <li>
1037 The implementation MAY cache RESULT messages.
1038 </li>
1039 <li>
1040 <t>
1041 If requests by other nodes for this KEY or BTYPE are known,
1042 the result block is validated against each request using
1043 the respective <tt>ValidateBlockReply</tt> function.
1044 </t>
1045 <t>
1046 If the request options include <tt>FindNode</tt> and the result
1047 message block type is HELLO the block validation must use the
1048 key derived using <tt>DeriveBlockKey</tt> as the key included in
1049 the request is only approximate. (FIXME: So we extract the key
1050 to then check it again against the block? This does not make sense...)
1051 </t>
1052 <t>
1053 If the result message block cannot be verified against the
1054 <tt>KEY</tt> of the result message or if <tt>BLOCK</tt> is
1055 malformed, the message MUST be discarded.
1056 </t>
1057 <t>
1058 For each pending request the reply is routed to the requesting
1059 node <tt>N'</tt>. FIXME routed to node or forwarded to peer?
1060 </t>
1061 </li>
1062 </ol>
1063 </section>
1064 </section>
1065 </section>
1066 <section anchor="blockstorage" numbered="true" toc="default">
1067 <name>Block Storage</name>
1068 <section>
1069 <name>Block Processing</name>
1070 <t>RequestEvaluationResult</t>
1071 <dl>
1072 <dt>REQUEST_VALID</dt>
1073 <dd>Query is valid, no reply given.</dd>
1074 <dt>REQUEST_INVALID</dt>
1075 <dd>
1076 Query format does not match block type. For example, XQuery not
1077 given or of size of XQuery is not appropriate for type.
1078 </dd>
1079 </dl>
1080 <t>ReplyEvaluationResult</t>
1081 <dl>
1082 <dt>OK_MORE</dt>
1083 <dd>Valid result, and there may be more.</dd>
1084 <dt>OK_LAST</dt>
1085 <dd>Last possible valid result.</dd>
1086 <dt>OK_DUPLICATE</dt>
1087 <dd>Valid result, but duplicate.</dd>
1088 <dt>RESULT_INVALID</dt>
1089 <dd>Invalid result. Block does not match query. Value = 4.</dd>
1090 <dt>RESULT_IRRELEVANT</dt>
1091 <dd>Block does not match xquery. Valid result, but not relevant for the request.</dd>
1092 </dl>
1093 </section>
1094 <section anchor="block_functions">
1095 <name>Block Functions</name>
1096 <t>
1097 Any block type implementation MUST implement the following functions.
1098 </t>
1099 <dl>
1100 <dt>ValidateBlockQuery(Key, XQuery) -&gt; RequestEvaluationResult</dt>
1101 <dd>
1102 is used to evaluate the request for a block. It is used as part of
1103 <tt>GetMessage</tt> processing, where the block payload is still unkown,
1104 but the block <tt>XQuery</tt> (FIXME: Undefined here) and <tt>Key</tt> can and
1105 MUST be verified, if possible.
1106 </dd>
1107 <dt>ValidateBlockStoreRequest(Block, Key) -&gt; RequestEvaluationResult</dt>
1108 <dd>
1109 is used to evaluate a block including its key and payload.
1110 It is used as part of <tt>PutMessage</tt> processing.
1111 The validation MUST include a check of the block payload against the
1112 <tt>Key</tt> under which it is requested to be stored.
1113 </dd>
1114 <dt>ValidateBlockReply(Block, XQuery, Key) -&gt; ReplyEvaluationResult</dt>
1115 <dd>
1116 is used to evaluate a block including its <tt>Key</tt> and payload.
1117 It is used as part <tt>ResultMessage</tt> processing.
1118 The validation of the respective <tt>Block</tt> requires a pending local query
1119 or a previously routed request of another node and its associated
1120 XQuery data and Key.
1121 The validation MUST include a check of the block payload against the
1122 key under which it is requested to be stored.
1123 </dd>
1124 <dt>DeriveBlockKey(Block) -&gt; Key</dt>
1125 <dd>
1126 is used to synthesize the block key from the block payload
1127 and metadata. It is used as part of FIND-node message processing.
1128 </dd>
1129 <dt>FilterResult(Block, XQuery, Key) -&gt; ReplyEvaluationResult</dt>
1130 <dd>
1131 is used to filter results stored in the local block storage for local
1132 queries. Locally stored blocks from previously observed
1133 <tt>ResultMessages</tt> and <tt>PutMessages</tt> MAY use this
1134 function instead of <tt>ValidateBlockReply</tt> in order to
1135 avoid revalidation of the block and only perform filtering based on
1136 request parameters.
1137 </dd>
1138 </dl>
1139 </section>
1140 <section anchor="block_types">
1141 <name>Block Types</name>
1142 <t>
1143 Applications can and should define their own block types.
1144 The block type determines the format and handling of the block
1145 payload by nodes in PUT and RESULT messages.
1146 Block types MUST be registered with GANA <xref target="gana"/>.
1147 </t>
1148 <t>
1149 For bootstrapping and node discovery, the DHT implementation uses
1150 its own block type called "HELLO". A block with this block type
1151 contains the NodeID of the node initiating the GET request.
1152 </t>
1153 <section anchor="hello_block">
1154 <name>HELLO</name>
1148 <t> 1155 <t>
1149 For bootstrapping and node discovery, the DHT implementation uses 1156 The HELLO block type wire format is illustrated in
1150 its own block type called "HELLO". A block with this block type 1157 <xref target="figure_hello"/>. A query for block of type HELLO MUST
1151 contains the NodeID of the node initiating the GET request. 1158 NOT include extended query data (XQuery). Any implementation
1159 encountering a HELLO block with XQuery data MUST consider the
1160 block invalid and ignore it.
1152 </t> 1161 </t>
1153 <section anchor="hello_block"> 1162 <figure anchor="figure_hello">
1154 <name>HELLO</name> 1163 <artwork name="" type="" align="left" alt=""><![CDATA[
1155 <t>
1156 The HELLO block type wire format is illustrated in
1157 <xref target="figure_hello"/>. A query for block of type HELLO MUST
1158 NOT include extended query data (XQuery). Any implementation
1159 encountering a HELLO block with XQuery data MUST consider the
1160 block invalid and ignore it.
1161 </t>
1162 <figure anchor="figure_hello">
1163 <artwork name="" type="" align="left" alt=""><![CDATA[
11640 8 16 24 32 40 48 56 11640 8 16 24 32 40 48 56
1165+---+-----+-----+-----+-----+-----+-----+-----+ 1165+---+-----+-----+-----+-----+-----+-----+-----+
1166| TYPE | SIZE | NODEID / 1166| TYPE | SIZE | NODEID /
@@ -1170,198 +1170,198 @@ Connectivity | |Underlay| |Underlay|
1170| ADDRESSES / 1170| ADDRESSES /
1171/ (variable length) | 1171/ (variable length) |
1172+---+-----+-----+-----+-----+-----+-----+-----+ 1172+---+-----+-----+-----+-----+-----+-----+-----+
1173]]/artwork> 1173 ]]></artwork>
1174 </figure> 1174 </figure>
1175 <dl> 1175 <dl>
1176 <dt>TYPE</dt> 1176 <dt>TYPE</dt>
1177 <dd> 1177 <dd>
1178 is the type of HELLO. A 16-bit number in network byte order. 1178 is the type of HELLO. A 16-bit number in network byte order.
1179 This value determines the type of the NODEID field. 1179 This value determines the type of the NODEID field.
1180 </dd> 1180 </dd>
1181 <dt>SIZE</dt> 1181 <dt>SIZE</dt>
1182 <dd> 1182 <dd>
1183 is the SIZE of the following fields NODEID and ADDRESSES in bytes. 1183 is the SIZE of the following fields NODEID and ADDRESSES in bytes.
1184 In network byte order. 1184 In network byte order.
1185 </dd> 1185 </dd>
1186 <dt>NODEID</dt> 1186 <dt>NODEID</dt>
1187 <dd> 1187 <dd>
1188 is the Node ID of the node which has generated this HELLO. 1188 is the Node ID of the node which has generated this HELLO.
1189 The length content of this field is determined by the TYPE. 1189 The length content of this field is determined by the TYPE.
1190 Usually, this is a cryptographic public key which allows the 1190 Usually, this is a cryptographic public key which allows the
1191 Underlay to uniquely identify and authenticate the node. 1191 Underlay to uniquely identify and authenticate the node.
1192 </dd> 1192 </dd>
1193 <dt>ADDRESSES</dt> 1193 <dt>ADDRESSES</dt>
1194 <dd> 1194 <dd>
1195 is a list of UTF-8 strings <xref target="RFC3629"/> which can be 1195 is a list of UTF-8 strings <xref target="RFC3629"/> which can be
1196 used as addresses to contact the node. 1196 used as addresses to contact the node.
1197 The strings MUST be 0-terminated. 1197 The strings MUST be 0-terminated.
1198 FIXME: Examples? Format determined? 1198 FIXME: Examples? Format determined?
1199 </dd> 1199 </dd>
1200 </dl> 1200 </dl>
1201 <t> 1201 <t>
1202 A HELLO reply block MAY be empty. Otherwise, it contains the 1202 A HELLO reply block MAY be empty. Otherwise, it contains the
1203 HELLO of a node. 1203 HELLO of a node.
1204 </t> 1204 </t>
1205 <t> 1205 <t>
1206 For the string representation of the node public key, 1206 For the string representation of the node public key,
1207 the base-32 encoding "StringEncode" is used. 1207 the base-32 encoding "StringEncode" is used.
1208 However, instead of following <xref target="RFC4648"/> the 1208 However, instead of following <xref target="RFC4648"/> the
1209 character map is based on the optical character recognition friendly 1209 character map is based on the optical character recognition friendly
1210 proposal of Crockford <xref target="CrockfordB32"/>. 1210 proposal of Crockford <xref target="CrockfordB32"/>.
1211 The only difference to Crockford is that the letter 1211 The only difference to Crockford is that the letter
1212 "U" decodes to the same base-32 value as the letter "V" (27). 1212 "U" decodes to the same base-32 value as the letter "V" (27).
1213 </t> 1213 </t>
1214 <t> 1214 <t>
1215 The <tt>ADDRESSES</tt> part of the <tt>HELLO</tt> indicate endpoints 1215 The <tt>ADDRESSES</tt> part of the <tt>HELLO</tt> indicate endpoints
1216 which can be used by the Underlay in order to establish a connection 1216 which can be used by the Underlay in order to establish a connection
1217 with the node identified by <tt>NODEKEY</tt>. 1217 with the node identified by <tt>NODEKEY</tt>.
1218 An example of an addressing scheme used throughout 1218 An example of an addressing scheme used throughout
1219 this document is "ip+tcp", which refers to a standard TCP/IP socket 1219 this document is "ip+tcp", which refers to a standard TCP/IP socket
1220 connection. The "hier"-part of the URI must provide a suitable 1220 connection. The "hier"-part of the URI must provide a suitable
1221 address for the given addressing scheme. 1221 address for the given addressing scheme.
1222 The following is a non-normative example of address strings: 1222 The following is a non-normative example of address strings:
1223 </t> 1223 </t>
1224 <figure> 1224 <figure>
1225 <artwork name="" type="" align="left" alt=""><![CDATA[ 1225 <artwork name="" type="" align="left" alt=""><![CDATA[
1226ipcp://1.2.3.4:6789 \ 1226ip+tcp://1.2.3.4:6789 \
1227gnet+tcp://12.3.4.5/ \ 1227gnunet+tcp://12.3.4.5/ \
1228i2udp://1.2.4.5:424/ \ 1228i2p+udp://1.2.4.5:424/ \
1229toonionv3://rasdflkjasdfliasduf.onion/ 1229tor+onionv3://rasdflkjasdfliasduf.onion/
1230]]/artwork> 1230 ]]></artwork>
1231 </figure> 1231 </figure>
1232 </section>
1233 </section> 1232 </section>
1234 </section> 1233 </section>
1235 <section anchor="security" numbered="true" toc="default"> 1234 </section>
1236 <name>Security Considerations</name> 1235 <section anchor="security" numbered="true" toc="default">
1237 <!-- FIXME: Here we should (again) discuss how the system is open and 1236 <name>Security Considerations</name>
1238 does not have/require a trust anchor a priori. This is (again) in contrast 1237 <!-- FIXME: Here we should (again) discuss how the system is open and
1239 to RELOAD --> 1238 does not have/require a trust anchor a priori. This is (again) in contrast
1240 </section> 1239 to RELOAD -->
1241 <section anchor="gana" numbered="true" toc="default"> 1240 </section>
1242 <name>GANA Considerations</name> 1241 <section anchor="gana" numbered="true" toc="default">
1243 <t> 1242 <name>GANA Considerations</name>
1244 GANA <xref target="GANA"/> 1243 <t>
1245 is requested to create a "DHT Block Types" registry. 1244 GANA <xref target="GANA"/>
1246 The registry shall record for each entry: 1245 is requested to create a "DHT Block Types" registry.
1247 </t> 1246 The registry shall record for each entry:
1248 <ul> 1247 </t>
1249 <li>Name: The name of the block type (case-insensitive ASCII 1248 <ul>
1250 string, restricted to alphanumeric characters</li> 1249 <li>Name: The name of the block type (case-insensitive ASCII
1251 <li>Number: 32-bit</li> 1250 string, restricted to alphanumeric characters</li>
1252 <li>Comment: Optionally, a brief English text describing the purpose of 1251 <li>Number: 32-bit</li>
1253 the block type (in UTF-8)</li> 1252 <li>Comment: Optionally, a brief English text describing the purpose of
1254 <li>Contact: Optionally, the contact information of a person to contact for 1253 the block type (in UTF-8)</li>
1255 further information</li> 1254 <li>Contact: Optionally, the contact information of a person to contact for
1256 <li>References: Optionally, references describing the record type 1255 further information</li>
1257 (such as an RFC)</li> 1256 <li>References: Optionally, references describing the record type
1258 </ul> 1257 (such as an RFC)</li>
1259 <t> 1258 </ul>
1260 The registration policy for this sub-registry is "First Come First 1259 <t>
1261 Served", as described in <xref target="RFC8126"/>. 1260 The registration policy for this sub-registry is "First Come First
1262 GANA is requested to populate this registry as follows: 1261 Served", as described in <xref target="RFC8126"/>.
1263 </t> 1262 GANA is requested to populate this registry as follows:
1264 <figure anchor="figure_btypenums"> 1263 </t>
1265 <artwork name="" type="" align="left" alt=""><![CDATA[ 1264 <figure anchor="figure_btypenums">
1266Nuer | Name | Contact | References | Description 1265 <artwork name="" type="" align="left" alt=""><![CDATA[
1267-----+--------+---------+------------+------------------------- 1266Number | Name | Contact | References | Description
12680 ANY N/A [This.I-D] Reserved 1267-------+--------+---------+------------+-------------------------
12697 HELLO N/A [This.I-D] Type of a block that contains 12680 ANY N/A [This.I-D] Reserved
1270a LLO for a node 12697 HELLO N/A [This.I-D] Type of a block that contains a HELLO for a node
127111 GNS N/A GNS Block for storing record data 127011 GNS N/A GNS Block for storing record data
1272]]/artwork> 1271]]></artwork>
1273 </figure> 1272 </figure>
1274 <t> 1273 <t>
1275 GANA is requested to amend the "GNUnet Signature Purpose" registry 1274 GANA is requested to amend the "GNUnet Signature Purpose" registry
1276 as follows: 1275 as follows:
1277 </t> 1276 </t>
1278 <figure anchor="figure_purposenums"> 1277 <figure anchor="figure_purposenums">
1279 <artwork name="" type="" align="left" alt=""><![CDATA[ 1278 <artwork name="" type="" align="left" alt=""><![CDATA[
1280Puose | Name | References | Description 1279Purpose | Name | References | Description
1281------+-----------------+------------+-------------------------- 1280--------+-----------------+------------+--------------------------
1282]]/artwork> 1281]]></artwork>
1283 </figure> 1282 </figure>
1284 </section> 1283 </section>
1285 <!-- gana --> 1284 <!-- gana -->
1286 <section> 1285 <section>
1287 <name>Test Vectors</name> 1286 <name>Test Vectors</name>
1288 </section> 1287 </section>
1289 </middle> 1288 </middle>
1290 <back> 1289 <back>
1291 <references><name>Normative References</name> 1290 <references>
1291 <name>Normative References</name>
1292 1292
1293 &RFC2119; 1293 &RFC2119;
1294 &RFC3629; 1294 &RFC3629;
1295 &RFC4634; 1295 &RFC4634;
1296 &RFC4648; 1296 &RFC4648;
1297 &RFC6940; 1297 &RFC6940;
1298 &RFC8126; 1298 &RFC8126;
1299 &RFC8174; 1299 &RFC8174;
1300 1300
1301 <reference anchor="ed25519" target="http://link.springer.com/chapter/10.1007/978-3-642-23951-9_9"><front><title>High-Speed High-Security Signatures</title><author initials="D." surname="Bernstein" fullname="Daniel Bernstein"><organization>University of Illinois at Chicago</organization></author><author initials="N." surname="Duif" fullname="Niels Duif"><organization>Technische Universiteit Eindhoven</organization></author><author initials="T." surname="Lange" fullname="Tanja Lange"><organization>Technische Universiteit Eindhoven</organization></author><author initials="P." surname="Schwabe" fullname="Peter Schwabe"><organization>National Taiwan University</organization></author><author initials="B." surname="Yang" fullname="Bo-Yin Yang"><organization>Academia Sinica</organization></author><date year="2011"/></front></reference> 1301 <reference anchor="ed25519" target="http://link.springer.com/chapter/10.1007/978-3-642-23951-9_9"><front><title>High-Speed High-Security Signatures</title><author initials="D." surname="Bernstein" fullname="Daniel Bernstein"><organization>University of Illinois at Chicago</organization></author><author initials="N." surname="Duif" fullname="Niels Duif"><organization>Technische Universiteit Eindhoven</organization></author><author initials="T." surname="Lange" fullname="Tanja Lange"><organization>Technische Universiteit Eindhoven</organization></author><author initials="P." surname="Schwabe" fullname="Peter Schwabe"><organization>National Taiwan University</organization></author><author initials="B." surname="Yang" fullname="Bo-Yin Yang"><organization>Academia Sinica</organization></author><date year="2011"/></front></reference>
1302 1302
1303 <reference anchor="CrockfordB32" target="https://www.crockford.com/base32.html"><front><title>Base32</title><author initials="D." surname="Douglas" fullname="Crockford"> 1303 <reference anchor="CrockfordB32" target="https://www.crockford.com/base32.html"><front><title>Base32</title><author initials="D." surname="Douglas" fullname="Crockford">
1304 </author><date year="2019" month="March"/></front></reference> 1304 </author><date year="2019" month="March"/></front></reference>
1305 1305
1306 <reference anchor="GANA" target="https://gana.gnunet.org/"><front><title>GNUnet Assigned Numbers Authority (GANA)</title><author><organization>GNUnet e.V.</organization></author><date month="April" year="2020"/></front></reference> 1306 <reference anchor="GANA" target="https://gana.gnunet.org/"><front><title>GNUnet Assigned Numbers Authority (GANA)</title><author><organization>GNUnet e.V.</organization></author><date month="April" year="2020"/></front></reference>
1307 1307
1308 1308
1309 1309
1310 </references> 1310 </references>
1311 <references> 1311 <references>
1312 <name>Informative References</name> 1312 <name>Informative References</name>
1313 <reference anchor="R5N" target="https://doi.org/10.1109/ICNSS.2011.6060022"> 1313 <reference anchor="R5N" target="https://doi.org/10.1109/ICNSS.2011.6060022">
1314 <front> 1314 <front>
1315 <title>R5N: Randomized recursive routing for restricted-route networks</title> 1315 <title>R5N: Randomized recursive routing for restricted-route networks</title>
1316 <author initials="N. S." surname="Evans" fullname="Nathan S. Evans"> 1316 <author initials="N. S." surname="Evans" fullname="Nathan S. Evans">
1317 <organization>Technische Universität München</organization> 1317 <organization>Technische Universität München</organization>
1318 </author> 1318 </author>
1319 <author initials="C." surname="Grothoff" fullname="Christian Grothoff"> 1319 <author initials="C." surname="Grothoff" fullname="Christian Grothoff">
1320 <organization>Technische Universität München</organization> 1320 <organization>Technische Universität München</organization>
1321 </author> 1321 </author>
1322 <date year="2011"/> 1322 <date year="2011"/>
1323 </front> 1323 </front>
1324 </reference> 1324 </reference>
1325 <reference anchor="Kademlia" target="http://css.csail.mit.edu/6.824/2014/papers/kademlia.pdf"> 1325 <reference anchor="Kademlia" target="http://css.csail.mit.edu/6.824/2014/papers/kademlia.pdf">
1326 <front> 1326 <front>
1327 <title>Kademlia: A peer-to-peer information system based on the xor metric.</title> 1327 <title>Kademlia: A peer-to-peer information system based on the xor metric.</title>
1328 <author initials="P." surname="Maymounkov" fullname="Petar Maymounkov"> 1328 <author initials="P." surname="Maymounkov" fullname="Petar Maymounkov">
1329 </author> 1329 </author>
1330 <author initials="D." surname="Mazieres" fullname="David Mazieres"> 1330 <author initials="D." surname="Mazieres" fullname="David Mazieres">
1331 </author> 1331 </author>
1332 <date year="2002"/> 1332 <date year="2002"/>
1333 </front> 1333 </front>
1334 </reference> 1334 </reference>
1335 <reference anchor="cadet" target="https://doi.org/10.1109/MedHocNet.2014.6849107"> 1335 <reference anchor="cadet" target="https://doi.org/10.1109/MedHocNet.2014.6849107">
1336 <front> 1336 <front>
1337 <title>CADET: Confidential ad-hoc decentralized end-to-end transport</title> 1337 <title>CADET: Confidential ad-hoc decentralized end-to-end transport</title>
1338 <author initials="B." surname="Polot" fullname="Bartlomiej Polot"> 1338 <author initials="B." surname="Polot" fullname="Bartlomiej Polot">
1339 <organization>Technische Universität München</organization> 1339 <organization>Technische Universität München</organization>
1340 </author> 1340 </author>
1341 <author initials="C." surname="Grothoff" fullname="Christian Grothoff"> 1341 <author initials="C." surname="Grothoff" fullname="Christian Grothoff">
1342 <organization>Technische Universität München</organization> 1342 <organization>Technische Universität München</organization>
1343 </author> 1343 </author>
1344 <date year="2014"/> 1344 <date year="2014"/>
1345 </front> 1345 </front>
1346 </reference> 1346 </reference>
1347 <reference anchor="I-D.draft-schanzen-gns" target="https://datatracker.ietf.org/doc/draft-schanzen-gns/"> 1347 <reference anchor="I-D.draft-schanzen-gns" target="https://datatracker.ietf.org/doc/draft-schanzen-gns/">
1348 <front> 1348 <front>
1349 <title>The GNU Name System</title> 1349 <title>The GNU Name System</title>
1350 <author initials="M." surname="Schanzenbach" fullname="Martin Schanzenbach"> 1350 <author initials="M." surname="Schanzenbach" fullname="Martin Schanzenbach">
1351 <organization>GNUnet e.V.</organization> 1351 <organization>GNUnet e.V.</organization>
1352 </author> 1352 </author>
1353 <author initials="C." surname="Grothoff" fullname="Christian Grothoff"> 1353 <author initials="C." surname="Grothoff" fullname="Christian Grothoff">
1354 <organization>GNUnet e.V.</organization> 1354 <organization>GNUnet e.V.</organization>
1355 </author> 1355 </author>
1356 <author initials="B." surname="Fix" fullname="Bernd Fix"> 1356 <author initials="B." surname="Fix" fullname="Bernd Fix">
1357 <organization>GNUnet e.V.</organization> 1357 <organization>GNUnet e.V.</organization>
1358 </author> 1358 </author>
1359 <date year="2021"/> 1359 <date year="2021"/>
1360 </front> 1360 </front>
1361 </reference> 1361 </reference>
1362 </references> 1362 </references>
1363 <!-- Change Log 1363 <!-- Change Log
1364 v00 2017-07-23 MS Initial version 1364 v00 2017-07-23 MS Initial version
1365 --> 1365 -->
1366 </back> 1366 </back>
1367 </rfc> 1367</rfc>