diff options
author | ng0 <ng0@infotropique.org> | 2017-10-20 13:43:46 +0000 |
---|---|---|
committer | ng0 <ng0@infotropique.org> | 2017-10-20 13:43:46 +0000 |
commit | eb75dd7cae216e99505772a9b32ffca596924819 (patch) | |
tree | f3544d9b7cba4dc6232261080c6fa9d598ed73fc /doc/chapters | |
parent | 60e3eb315a994ac38778ceb9d493e3624cb4cbe0 (diff) | |
download | gnunet-eb75dd7cae216e99505772a9b32ffca596924819.tar.gz gnunet-eb75dd7cae216e99505772a9b32ffca596924819.zip |
fixes and additions in chapters/developer.texi
Diffstat (limited to 'doc/chapters')
-rw-r--r-- | doc/chapters/developer.texi | 1621 |
1 files changed, 889 insertions, 732 deletions
diff --git a/doc/chapters/developer.texi b/doc/chapters/developer.texi index c858980d0..e690e5f5b 100644 --- a/doc/chapters/developer.texi +++ b/doc/chapters/developer.texi | |||
@@ -110,7 +110,8 @@ following links: | |||
110 | @c ** FIXME: Link to files in source, not online. | 110 | @c ** FIXME: Link to files in source, not online. |
111 | @c ** FIXME: Where is the Java tutorial? | 111 | @c ** FIXME: Where is the Java tutorial? |
112 | @itemize @bullet | 112 | @itemize @bullet |
113 | @item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutorial.pdf, GNUnet C tutorial} | 113 | @item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutoria |
114 | l.pdf, GNUnet C tutorial} | ||
114 | @item GNUnet Java tutorial | 115 | @item GNUnet Java tutorial |
115 | @end itemize | 116 | @end itemize |
116 | 117 | ||
@@ -128,7 +129,8 @@ The public subsystems on the GNUnet server that help developers are: | |||
128 | @item The Version control system keeps our code and enables distributed | 129 | @item The Version control system keeps our code and enables distributed |
129 | development. Only developers with write access can commit code, everyone | 130 | development. Only developers with write access can commit code, everyone |
130 | else is encouraged to submit patches to the | 131 | else is encouraged to submit patches to the |
131 | @uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, GNUnet-developers mailinglist}. | 132 | @uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, |
133 | GNUnet-developers mailinglist}. | ||
132 | @item The GNUnet bugtracking system is used to track feature requests, | 134 | @item The GNUnet bugtracking system is used to track feature requests, |
133 | open bug reports and their resolutions. Anyone can report bugs, only | 135 | open bug reports and their resolutions. Anyone can report bugs, only |
134 | developers can claim to have fixed them. | 136 | developers can claim to have fixed them. |
@@ -4108,6 +4110,7 @@ maturity, and it is still unclear if any particular plugin is generally | |||
4108 | superior. | 4110 | superior. |
4109 | 4111 | ||
4110 | @cindex core subsystem | 4112 | @cindex core subsystem |
4113 | @cindex CORE subsystem | ||
4111 | @node GNUnet's CORE Subsystem | 4114 | @node GNUnet's CORE Subsystem |
4112 | @section GNUnet's CORE Subsystem | 4115 | @section GNUnet's CORE Subsystem |
4113 | @c %**end of header | 4116 | @c %**end of header |
@@ -4120,8 +4123,8 @@ then adds fundamental security to the connections: | |||
4120 | 4123 | ||
4121 | @itemize @bullet | 4124 | @itemize @bullet |
4122 | @item confidentiality with so-called perfect forward secrecy; we use | 4125 | @item confidentiality with so-called perfect forward secrecy; we use |
4123 | ECDHE@footnote{Elliptic-curve Diffie—Hellman | 4126 | ECDHE@footnote{@uref{http://en.wikipedia.org/wiki/Elliptic_curve_ |
4124 | @uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman}} | 4127 | Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}} |
4125 | powered by Curve25519 | 4128 | powered by Curve25519 |
4126 | @footnote{@uref{http://cr.yp.to/ecdh.html, Curve25519}} for the key | 4129 | @footnote{@uref{http://cr.yp.to/ecdh.html, Curve25519}} for the key |
4127 | exchange and then use symmetric encryption, encrypting with both AES-256 | 4130 | exchange and then use symmetric encryption, encrypting with both AES-256 |
@@ -4279,11 +4282,12 @@ using (as theoretically the application may be using a different | |||
4279 | configuration file with a different private key, which would result in | 4282 | configuration file with a different private key, which would result in |
4280 | hard to find bugs). | 4283 | hard to find bugs). |
4281 | 4284 | ||
4282 | As with most service APIs, the CORE API isolates applications from crashes of | 4285 | As with most service APIs, the CORE API isolates applications from crashes |
4283 | the CORE service. If the CORE service crashes, the application will see | 4286 | of the CORE service. If the CORE service crashes, the application will see |
4284 | disconnect events for all existing connections. Once the connections are | 4287 | disconnect events for all existing connections. Once the connections are |
4285 | re-established, the applications will be receive matching connect events. | 4288 | re-established, the applications will be receive matching connect events. |
4286 | 4289 | ||
4290 | @cindex core clinet-service protocol | ||
4287 | @node The CORE Client-Service Protocol | 4291 | @node The CORE Client-Service Protocol |
4288 | @subsection The CORE Client-Service Protocol | 4292 | @subsection The CORE Client-Service Protocol |
4289 | @c %**end of header | 4293 | @c %**end of header |
@@ -4305,59 +4309,61 @@ service (the client) and the CORE service process itself. | |||
4305 | When a client connects to the CORE service, it first sends a | 4309 | When a client connects to the CORE service, it first sends a |
4306 | @code{InitMessage} which specifies options for the connection and a set of | 4310 | @code{InitMessage} which specifies options for the connection and a set of |
4307 | message type values which are supported by the application. The options | 4311 | message type values which are supported by the application. The options |
4308 | bitmask specifies which events the client would like to be notified about. The | 4312 | bitmask specifies which events the client would like to be notified about. |
4309 | options include: | 4313 | The options include: |
4310 | 4314 | ||
4311 | @table @asis | 4315 | @table @asis |
4312 | @item GNUNET_CORE_OPTION_NOTHING No notifications | 4316 | @item GNUNET_CORE_OPTION_NOTHING No notifications |
4313 | @item GNUNET_CORE_OPTION_STATUS_CHANGE Peers connecting and disconnecting | 4317 | @item GNUNET_CORE_OPTION_STATUS_CHANGE Peers connecting and disconnecting |
4314 | @item GNUNET_CORE_OPTION_FULL_INBOUND All inbound messages (after decryption) with | 4318 | @item GNUNET_CORE_OPTION_FULL_INBOUND All inbound messages (after |
4315 | full payload | 4319 | decryption) with full payload |
4316 | @item GNUNET_CORE_OPTION_HDR_INBOUND Just the @code{MessageHeader} | 4320 | @item GNUNET_CORE_OPTION_HDR_INBOUND Just the @code{MessageHeader} |
4317 | of all inbound messages | 4321 | of all inbound messages |
4318 | @item GNUNET_CORE_OPTION_FULL_OUTBOUND All outbound | 4322 | @item GNUNET_CORE_OPTION_FULL_OUTBOUND All outbound |
4319 | messages (prior to encryption) with full payload | 4323 | messages (prior to encryption) with full payload |
4320 | @item GNUNET_CORE_OPTION_HDR_OUTBOUND Just the @code{MessageHeader} of all outbound | 4324 | @item GNUNET_CORE_OPTION_HDR_OUTBOUND Just the @code{MessageHeader} of all |
4321 | messages | 4325 | outbound messages |
4322 | @end table | 4326 | @end table |
4323 | 4327 | ||
4324 | Typical applications will only monitor for connection status changes. | 4328 | Typical applications will only monitor for connection status changes. |
4325 | 4329 | ||
4326 | The CORE service responds to the @code{InitMessage} with an | 4330 | The CORE service responds to the @code{InitMessage} with an |
4327 | @code{InitReplyMessage} which contains the peer's identity. Afterwards, both | 4331 | @code{InitReplyMessage} which contains the peer's identity. Afterwards, |
4328 | CORE and the client can send messages. | 4332 | both CORE and the client can send messages. |
4329 | 4333 | ||
4330 | @node Notifications | 4334 | @node Notifications |
4331 | @subsubsection Notifications | 4335 | @subsubsection Notifications |
4332 | @c %**end of header | 4336 | @c %**end of header |
4333 | 4337 | ||
4334 | The CORE will send @code{ConnectNotifyMessage}s and | 4338 | The CORE will send @code{ConnectNotifyMessage}s and |
4335 | @code{DisconnectNotifyMessage}s whenever peers connect or disconnect from the | 4339 | @code{DisconnectNotifyMessage}s whenever peers connect or disconnect from |
4336 | CORE (assuming their type maps overlap with the message types registered by | 4340 | the CORE (assuming their type maps overlap with the message types |
4337 | the client). When the CORE receives a message that matches the set of message | 4341 | registered by the client). When the CORE receives a message that matches |
4338 | types specified during the @code{InitMessage} (or if monitoring is enabled in | 4342 | the set of message types specified during the @code{InitMessage} (or if |
4339 | for inbound messages in the options), it sends a @code{NotifyTrafficMessage} | 4343 | monitoring is enabled in for inbound messages in the options), it sends a |
4340 | with the peer identity of the sender and the decrypted payload. The same | 4344 | @code{NotifyTrafficMessage} with the peer identity of the sender and the |
4341 | message format (except with @code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} | 4345 | decrypted payload. The same message format (except with |
4342 | for the message type) is used to notify clients monitoring outbound messages; | 4346 | @code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} for the message type) is |
4343 | here, the peer identity given is that of the receiver. | 4347 | used to notify clients monitoring outbound messages; here, the peer |
4348 | identity given is that of the receiver. | ||
4344 | 4349 | ||
4345 | @node Sending | 4350 | @node Sending |
4346 | @subsubsection Sending | 4351 | @subsubsection Sending |
4347 | @c %**end of header | 4352 | @c %**end of header |
4348 | 4353 | ||
4349 | When a client wants to transmit a message, it first requests a transmission | 4354 | When a client wants to transmit a message, it first requests a |
4350 | slot by sending a @code{SendMessageRequest} which specifies the priority, | 4355 | transmission slot by sending a @code{SendMessageRequest} which specifies |
4351 | deadline and size of the message. Note that these values may be ignored by | 4356 | the priority, deadline and size of the message. Note that these values |
4352 | CORE. When CORE is ready for the message, it answers with a | 4357 | may be ignored by CORE. When CORE is ready for the message, it answers |
4353 | @code{SendMessageReady} response. The client can then transmit the payload | 4358 | with a @code{SendMessageReady} response. The client can then transmit the |
4354 | with a @code{SendMessage} message. Note that the actual message size in the | 4359 | payload with a @code{SendMessage} message. Note that the actual message |
4355 | @code{SendMessage} is allowed to be smaller than the size in the original | 4360 | size in the @code{SendMessage} is allowed to be smaller than the size in |
4356 | request. A client may at any time send a fresh @code{SendMessageRequest}, | 4361 | the original request. A client may at any time send a fresh |
4357 | which then superceeds the previous @code{SendMessageRequest}, which is then no | 4362 | @code{SendMessageRequest}, which then superceeds the previous |
4358 | longer valid. The client can tell which @code{SendMessageRequest} the CORE | 4363 | @code{SendMessageRequest}, which is then no longer valid. The client can |
4359 | service's @code{SendMessageReady} message is for as all of these messages | 4364 | tell which @code{SendMessageRequest} the CORE service's |
4360 | contain a "unique" request ID (based on a counter incremented by the client | 4365 | @code{SendMessageReady} message is for as all of these messages contain a |
4366 | "unique" request ID (based on a counter incremented by the client | ||
4361 | for each request). | 4367 | for each request). |
4362 | 4368 | ||
4363 | @node The CORE Peer-to-Peer Protocol | 4369 | @node The CORE Peer-to-Peer Protocol |
@@ -4372,60 +4378,65 @@ for each request). | |||
4372 | * Type maps:: | 4378 | * Type maps:: |
4373 | @end menu | 4379 | @end menu |
4374 | 4380 | ||
4381 | @cindex EphemeralKeyMessage creation | ||
4375 | @node Creating the EphemeralKeyMessage | 4382 | @node Creating the EphemeralKeyMessage |
4376 | @subsubsection Creating the EphemeralKeyMessage | 4383 | @subsubsection Creating the EphemeralKeyMessage |
4377 | @c %**end of header | 4384 | @c %**end of header |
4378 | 4385 | ||
4379 | When the CORE service starts, each peer creates a fresh ephemeral (ECC) | 4386 | When the CORE service starts, each peer creates a fresh ephemeral (ECC) |
4380 | public-private key pair and signs the corresponding @code{EphemeralKeyMessage} | 4387 | public-private key pair and signs the corresponding |
4381 | with its long-term key (which we usually call the peer's identity; the hash of | 4388 | @code{EphemeralKeyMessage} with its long-term key (which we usually call |
4382 | the public long term key is what results in a @code{struct | 4389 | the peer's identity; the hash of the public long term key is what results |
4383 | GNUNET_PeerIdentity} in all GNUnet APIs. The ephemeral key is ONLY used for an | 4390 | in a @code{struct GNUNET_PeerIdentity} in all GNUnet APIs. The ephemeral |
4384 | @uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman, | 4391 | key is ONLY used for an ECDHE@footnote{@uref{http://en.wikipedia.org/wiki/ |
4385 | ECDHE} exchange by the CORE service to establish symmetric session keys. A | 4392 | Elliptic_curve_Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}} |
4386 | peer will use the same @code{EphemeralKeyMessage} for all peers for | 4393 | exchange by the CORE service to establish symmetric session keys. A peer |
4387 | @code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it will | 4394 | will use the same @code{EphemeralKeyMessage} for all peers for |
4388 | create a fresh ephemeral key (forgetting the old one) and broadcast the new | 4395 | @code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it |
4389 | @code{EphemeralKeyMessage} to all connected peers, resulting in fresh | 4396 | will create a fresh ephemeral key (forgetting the old one) and broadcast |
4390 | symmetric session keys. Note that peers independently decide on when to | 4397 | the new @code{EphemeralKeyMessage} to all connected peers, resulting in |
4391 | discard ephemeral keys; it is not a protocol violation to discard keys more | 4398 | fresh symmetric session keys. Note that peers independently decide on |
4392 | often. Ephemeral keys are also never stored to disk; restarting a peer will | 4399 | when to discard ephemeral keys; it is not a protocol violation to discard |
4393 | thus always create a fresh ephemeral key. The use of ephemeral keys is what | 4400 | keys more often. Ephemeral keys are also never stored to disk; restarting |
4394 | provides @uref{http://en.wikipedia.org/wiki/Forward_secrecy, forward secrecy}. | 4401 | a peer will thus always create a fresh ephemeral key. The use of ephemeral |
4395 | 4402 | keys is what provides @uref{http://en.wikipedia.org/wiki/Forward_secrecy, | |
4396 | Just before transmission, the @code{EphemeralKeyMessage} is patched to reflect | 4403 | forward secrecy}. |
4397 | the current sender_status, which specifies the current state of the connection | 4404 | |
4398 | from the point of view of the sender. The possible values are: | 4405 | Just before transmission, the @code{EphemeralKeyMessage} is patched to |
4406 | reflect the current sender_status, which specifies the current state of | ||
4407 | the connection from the point of view of the sender. The possible values | ||
4408 | are: | ||
4399 | 4409 | ||
4400 | @table @asis | 4410 | @itemize @bullet |
4401 | @item KX_STATE_DOWN Initial value, never used on the network | 4411 | @item @code{KX_STATE_DOWN} Initial value, never used on the network |
4402 | @item KX_STATE_KEY_SENT We sent our ephemeral key, do not know the key of the other | 4412 | @item @code{KX_STATE_KEY_SENT} We sent our ephemeral key, do not know the |
4403 | peer | 4413 | key of the other peer |
4404 | @item KX_STATE_KEY_RECEIVED This peer has received a valid ephemeral key | 4414 | @item @code{KX_STATE_KEY_RECEIVED} This peer has received a valid |
4405 | of the other peer, but we are waiting for the other peer to confirm it's | 4415 | ephemeral key of the other peer, but we are waiting for the other peer to |
4406 | authenticity (ability to decode) via challenge-response. | 4416 | confirm it's authenticity (ability to decode) via challenge-response. |
4407 | @item KX_STATE_UP The | 4417 | @item @code{KX_STATE_UP} The connection is fully up from the point of |
4408 | connection is fully up from the point of view of the sender (now performing | 4418 | view of the sender (now performing keep-alives) |
4409 | keep-alives) | 4419 | @item @code{KX_STATE_REKEY_SENT} The sender has initiated a rekeying |
4410 | @item KX_STATE_REKEY_SENT The sender has initiated a rekeying | 4420 | operation; the other peer has so far failed to confirm a working |
4411 | operation; the other peer has so far failed to confirm a working connection | 4421 | connection using the new ephemeral key |
4412 | using the new ephemeral key | 4422 | @end itemize |
4413 | @end table | ||
4414 | 4423 | ||
4415 | @node Establishing a connection | 4424 | @node Establishing a connection |
4416 | @subsubsection Establishing a connection | 4425 | @subsubsection Establishing a connection |
4417 | @c %**end of header | 4426 | @c %**end of header |
4418 | 4427 | ||
4419 | Peers begin their interaction by sending a @code{EphemeralKeyMessage} to the | 4428 | Peers begin their interaction by sending a @code{EphemeralKeyMessage} to |
4420 | other peer once the TRANSPORT service notifies the CORE service about the | 4429 | the other peer once the TRANSPORT service notifies the CORE service about |
4421 | connection. A peer receiving an @code{EphemeralKeyMessage} with a status | 4430 | the connection. |
4431 | A peer receiving an @code{EphemeralKeyMessage} with a status | ||
4422 | indicating that the sender does not have the receiver's ephemeral key, the | 4432 | indicating that the sender does not have the receiver's ephemeral key, the |
4423 | receiver's @code{EphemeralKeyMessage} is sent in response.@ Additionally, if | 4433 | receiver's @code{EphemeralKeyMessage} is sent in response. |
4424 | the receiver has not yet confirmed the authenticity of the sender, it also | 4434 | Additionally, if the receiver has not yet confirmed the authenticity of |
4425 | sends an (encrypted)@code{PingMessage} with a challenge (and the identity of | 4435 | the sender, it also sends an (encrypted)@code{PingMessage} with a |
4426 | the target) to the other peer. Peers receiving a @code{PingMessage} respond | 4436 | challenge (and the identity of the target) to the other peer. Peers |
4427 | with an (encrypted) @code{PongMessage} which includes the challenge. Peers | 4437 | receiving a @code{PingMessage} respond with an (encrypted) |
4428 | receiving a @code{PongMessage} check the challenge, and if it matches set the | 4438 | @code{PongMessage} which includes the challenge. Peers receiving a |
4439 | @code{PongMessage} check the challenge, and if it matches set the | ||
4429 | connection to @code{KX_STATE_UP}. | 4440 | connection to @code{KX_STATE_UP}. |
4430 | 4441 | ||
4431 | @node Encryption and Decryption | 4442 | @node Encryption and Decryption |
@@ -4433,26 +4444,27 @@ connection to @code{KX_STATE_UP}. | |||
4433 | @c %**end of header | 4444 | @c %**end of header |
4434 | 4445 | ||
4435 | All functions related to the key exchange and encryption/decryption of | 4446 | All functions related to the key exchange and encryption/decryption of |
4436 | messages can be found in @code{gnunet-service-core_kx.c} (except for the | 4447 | messages can be found in @file{gnunet-service-core_kx.c} (except for the |
4437 | cryptographic primitives, which are in @code{util/crypto*.c}).@ Given the key | 4448 | cryptographic primitives, which are in @file{util/crypto*.c}). |
4438 | material from ECDHE, a | 4449 | Given the key material from ECDHE, a Key derivation function |
4439 | @uref{http://en.wikipedia.org/wiki/Key_derivation_function, Key derivation | 4450 | @footnote{@uref{https://en.wikipedia.org/wiki/Key_derivation_function, Key |
4440 | function} is used to derive two pairs of encryption and decryption keys for | 4451 | derivation function}} is used to derive two pairs of encryption and |
4441 | AES-256 and TwoFish, as well as initialization vectors and authentication keys | 4452 | decryption keys for AES-256 and TwoFish, as well as initialization vectors |
4442 | (for @uref{http://en.wikipedia.org/wiki/HMAC, HMAC}). The HMAC is computed | 4453 | and authentication keys (for HMAC@footnote{@uref{https://en.wikipedia.org/ |
4443 | over the encrypted payload. Encrypted messages include an iv_seed and the HMAC | 4454 | wiki/HMAC, HMAC}}). The HMAC is computed over the encrypted payload. |
4444 | in the header. | 4455 | Encrypted messages include an iv_seed and the HMAC in the header. |
4445 | 4456 | ||
4446 | Each encrypted message in the CORE service includes a sequence number and a | 4457 | Each encrypted message in the CORE service includes a sequence number and |
4447 | timestamp in the encrypted payload. The CORE service remembers the largest | 4458 | a timestamp in the encrypted payload. The CORE service remembers the |
4448 | observed sequence number and a bit-mask which represents which of the previous | 4459 | largest observed sequence number and a bit-mask which represents which of |
4449 | 32 sequence numbers were already used. Messages with sequence numbers lower | 4460 | the previous 32 sequence numbers were already used. |
4450 | than the largest observed sequence number minus 32 are discarded. Messages | 4461 | Messages with sequence numbers lower than the largest observed sequence |
4451 | with a timestamp that is less than @code{REKEY_TOLERANCE} off (5 minutes) are | 4462 | number minus 32 are discarded. Messages with a timestamp that is less |
4452 | also discarded. This of course means that system clocks need to be reasonably | 4463 | than @code{REKEY_TOLERANCE} off (5 minutes) are also discarded. This of |
4453 | synchronized for peers to be able to communicate. Additionally, as the | 4464 | course means that system clocks need to be reasonably synchronized for |
4454 | ephemeral key changes every 12h, a peer would not even be able to decrypt | 4465 | peers to be able to communicate. Additionally, as the ephemeral key |
4455 | messages older than 12h. | 4466 | changes every 12 hours, a peer would not even be able to decrypt messages |
4467 | older than 12 hours. | ||
4456 | 4468 | ||
4457 | @node Type maps | 4469 | @node Type maps |
4458 | @subsubsection Type maps | 4470 | @subsubsection Type maps |
@@ -4460,103 +4472,111 @@ messages older than 12h. | |||
4460 | 4472 | ||
4461 | Once an encrypted connection has been established, peers begin to exchange | 4473 | Once an encrypted connection has been established, peers begin to exchange |
4462 | type maps. Type maps are used to allow the CORE service to determine which | 4474 | type maps. Type maps are used to allow the CORE service to determine which |
4463 | (encrypted) connections should be shown to which applications. A type map is | 4475 | (encrypted) connections should be shown to which applications. A type map |
4464 | an array of 65536 bits representing the different types of messages understood | 4476 | is an array of 65536 bits representing the different types of messages |
4465 | by applications using the CORE service. Each CORE service maintains this map, | 4477 | understood by applications using the CORE service. Each CORE service |
4466 | simply by setting the respective bit for each message type supported by any of | 4478 | maintains this map, simply by setting the respective bit for each message |
4467 | the applications using the CORE service. Note that bits for message types | 4479 | type supported by any of the applications using the CORE service. Note |
4468 | embedded in higher-level protocols (such as MESH) will not be included in | 4480 | that bits for message types embedded in higher-level protocols (such as |
4469 | these type maps. | 4481 | MESH) will not be included in these type maps. |
4470 | 4482 | ||
4471 | Typically, the type map of a peer will be sparse. Thus, the CORE service | 4483 | Typically, the type map of a peer will be sparse. Thus, the CORE service |
4472 | attempts to compress its type map using @code{gzip}-style compression | 4484 | attempts to compress its type map using @code{gzip}-style compression |
4473 | ("deflate") prior to transmission. However, if the compression fails to | 4485 | ("deflate") prior to transmission. However, if the compression fails to |
4474 | compact the map, the map may also be transmitted without compression | 4486 | compact the map, the map may also be transmitted without compression |
4475 | (resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or | 4487 | (resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or |
4476 | @code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively). Upon | 4488 | @code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively). |
4477 | receiving a type map, the respective CORE service notifies applications about | 4489 | Upon receiving a type map, the respective CORE service notifies |
4478 | the connection to the other peer if they support any message type indicated in | 4490 | applications about the connection to the other peer if they support any |
4479 | the type map (or no message type at all). If the CORE service experience a | 4491 | message type indicated in the type map (or no message type at all). |
4480 | connect or disconnect event from an application, it updates its type map | 4492 | If the CORE service experience a connect or disconnect event from an |
4481 | (setting or unsetting the respective bits) and notifies its neighbours about | 4493 | application, it updates its type map (setting or unsetting the respective |
4482 | the change. The CORE services of the neighbours then in turn generate connect | 4494 | bits) and notifies its neighbours about the change. |
4483 | and disconnect events for the peer that sent the type map for their respective | 4495 | The CORE services of the neighbours then in turn generate connect and |
4496 | disconnect events for the peer that sent the type map for their respective | ||
4484 | applications. As CORE messages may be lost, the CORE service confirms | 4497 | applications. As CORE messages may be lost, the CORE service confirms |
4485 | receiving a type map by sending back a | 4498 | receiving a type map by sending back a |
4486 | @code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation (with | 4499 | @code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation |
4487 | the correct hash of the type map) is not received, the sender will retransmit | 4500 | (with the correct hash of the type map) is not received, the sender will |
4488 | the type map (with exponential back-off). | 4501 | retransmit the type map (with exponential back-off). |
4489 | 4502 | ||
4503 | @cindex cadet subsystem | ||
4504 | @cindex CADET | ||
4490 | @node GNUnet's CADET subsystem | 4505 | @node GNUnet's CADET subsystem |
4491 | @section GNUnet's CADET subsystem | 4506 | @section GNUnet's CADET subsystem |
4492 | 4507 | ||
4493 | The CADET subsystem in GNUnet is responsible for secure end-to-end | 4508 | The CADET subsystem in GNUnet is responsible for secure end-to-end |
4494 | communications between nodes in the GNUnet overlay network. CADET builds on the | 4509 | communications between nodes in the GNUnet overlay network. CADET builds |
4495 | CORE subsystem which provides for the link-layer communication and then adds | 4510 | on the CORE subsystem which provides for the link-layer communication and |
4496 | routing, forwarding and additional security to the connections. CADET offers | 4511 | then adds routing, forwarding and additional security to the connections. |
4497 | the same cryptographic services as CORE, but on an end-to-end level. This is | 4512 | CADET offers the same cryptographic services as CORE, but on an |
4498 | done so peers retransmitting traffic on behalf of other peers cannot access the | 4513 | end-to-end level. This is done so peers retransmitting traffic on behalf |
4499 | payload data. | 4514 | of other peers cannot access the payload data. |
4500 | 4515 | ||
4501 | @itemize @bullet | 4516 | @itemize @bullet |
4502 | @item CADET provides confidentiality with so-called perfect forward secrecy; we | 4517 | @item CADET provides confidentiality with so-called perfect forward |
4503 | use ECDHE powered by Curve25519 for the key exchange and then use symmetric | 4518 | secrecy; we use ECDHE powered by Curve25519 for the key exchange and then |
4504 | encryption, encrypting with both AES-256 and Twofish | 4519 | use symmetric encryption, encrypting with both AES-256 and Twofish |
4505 | @item authentication is achieved by signing the ephemeral keys using Ed25519, a | 4520 | @item authentication is achieved by signing the ephemeral keys using |
4506 | deterministic variant of ECDSA | 4521 | Ed25519, a deterministic variant of ECDSA |
4507 | @item integrity protection (using SHA-512 to do encrypt-then-MAC, although only | 4522 | @item integrity protection (using SHA-512 to do encrypt-then-MAC, although |
4508 | 256 bits are sent to reduce overhead) | 4523 | only 256 bits are sent to reduce overhead) |
4509 | @item replay protection (using nonces, timestamps, challenge-response, message | 4524 | @item replay protection (using nonces, timestamps, challenge-response, |
4510 | counters and ephemeral keys) | 4525 | message counters and ephemeral keys) |
4511 | @item liveness (keep-alive messages, timeout) | 4526 | @item liveness (keep-alive messages, timeout) |
4512 | @end itemize | 4527 | @end itemize |
4513 | 4528 | ||
4514 | Additional to the CORE-like security benefits, CADET offers other properties | 4529 | Additional to the CORE-like security benefits, CADET offers other |
4515 | that make it a more universal service than CORE. | 4530 | properties that make it a more universal service than CORE. |
4516 | 4531 | ||
4517 | @itemize @bullet | 4532 | @itemize @bullet |
4518 | @item CADET can establish channels to arbitrary peers in GNUnet. If a peer is | 4533 | @item CADET can establish channels to arbitrary peers in GNUnet. If a |
4519 | not immediately reachable, CADET will find a path through the network and ask | 4534 | peer is not immediately reachable, CADET will find a path through the |
4520 | other peers to retransmit the traffic on its behalf. | 4535 | network and ask other peers to retransmit the traffic on its behalf. |
4521 | @item CADET offers (optional) reliability mechanisms. In a reliable channel | 4536 | @item CADET offers (optional) reliability mechanisms. In a reliable |
4522 | traffic is guaranteed to arrive complete, unchanged and in-order. | 4537 | channel traffic is guaranteed to arrive complete, unchanged and in-order. |
4523 | @item CADET takes care of flow and congestion control mechanisms, not allowing | 4538 | @item CADET takes care of flow and congestion control mechanisms, not |
4524 | the sender to send more traffic than the receiver or the network are able to | 4539 | allowing the sender to send more traffic than the receiver or the network |
4525 | process. | 4540 | are able to process. |
4526 | @end itemize | 4541 | @end itemize |
4527 | 4542 | ||
4528 | @menu | 4543 | @menu |
4529 | * libgnunetcadet:: | 4544 | * libgnunetcadet:: |
4530 | @end menu | 4545 | @end menu |
4531 | 4546 | ||
4547 | @cindex libgnunetcadet | ||
4532 | @node libgnunetcadet | 4548 | @node libgnunetcadet |
4533 | @subsection libgnunetcadet | 4549 | @subsection libgnunetcadet |
4534 | 4550 | ||
4535 | 4551 | ||
4536 | The CADET API (defined in gnunet_cadet_service.h) is the messaging API used by | 4552 | The CADET API (defined in @file{gnunet_cadet_service.h}) is the |
4537 | P2P applications built using GNUnet. It provides applications the ability to | 4553 | messaging API used by P2P applications built using GNUnet. |
4538 | send and receive encrypted messages to any peer participating in GNUnet. The | 4554 | It provides applications the ability to send and receive encrypted |
4539 | API is heavily base on the CORE API. | 4555 | messages to any peer participating in GNUnet. |
4556 | The API is heavily base on the CORE API. | ||
4540 | 4557 | ||
4541 | CADET delivers messages to other peers in "channels". A channel is a permanent | 4558 | CADET delivers messages to other peers in "channels". |
4542 | connection defined by a destination peer (identified by its public key) and a | 4559 | A channel is a permanent connection defined by a destination peer |
4543 | port number. Internally, CADET tunnels all channels towards a destiantion peer | 4560 | (identified by its public key) and a port number. |
4561 | Internally, CADET tunnels all channels towards a destiantion peer | ||
4544 | using one session key and relays the data on multiple "connections", | 4562 | using one session key and relays the data on multiple "connections", |
4545 | independent from the channels. | 4563 | independent from the channels. |
4546 | 4564 | ||
4547 | Each channel has optional paramenters, the most important being the reliability | 4565 | Each channel has optional paramenters, the most important being the |
4548 | flag. Should a message get lost on TRANSPORT/CORE level, if a channel is | 4566 | reliability flag. |
4549 | created with as reliable, CADET will retransmit the lost message and deliver it | 4567 | Should a message get lost on TRANSPORT/CORE level, if a channel is |
4550 | in order to the destination application. | 4568 | created with as reliable, CADET will retransmit the lost message and |
4551 | 4569 | deliver it in order to the destination application. | |
4552 | To communicate with other peers using CADET, it is necessary to first connect | 4570 | |
4553 | to the service using @code{GNUNET_CADET_connect}. This function takes several | 4571 | To communicate with other peers using CADET, it is necessary to first |
4554 | parameters in form of callbacks, to allow the client to react to various | 4572 | connect to the service using @code{GNUNET_CADET_connect}. |
4555 | events, like incoming channels or channels that terminate, as well as specify a | 4573 | This function takes several parameters in form of callbacks, to allow the |
4556 | list of ports the client wishes to listen to (at the moment it is not possible | 4574 | client to react to various events, like incoming channels or channels that |
4557 | to start listening on further ports once connected, but nothing prevents a | 4575 | terminate, as well as specify a list of ports the client wishes to listen |
4558 | client to connect several times to CADET, even do one connection per listening | 4576 | to (at the moment it is not possible to start listening on further ports |
4559 | port). The function returns a handle which has to be used for any further | 4577 | once connected, but nothing prevents a client to connect several times to |
4578 | CADET, even do one connection per listening port). | ||
4579 | The function returns a handle which has to be used for any further | ||
4560 | interaction with the service. | 4580 | interaction with the service. |
4561 | 4581 | ||
4562 | To connect to a remote peer a client has to call the | 4582 | To connect to a remote peer a client has to call the |
@@ -4564,62 +4584,69 @@ To connect to a remote peer a client has to call the | |||
4564 | given are the remote peer's identity (it public key) and a port, which | 4584 | given are the remote peer's identity (it public key) and a port, which |
4565 | specifies which application on the remote peer to connect to, similar to | 4585 | specifies which application on the remote peer to connect to, similar to |
4566 | TCP/UDP ports. CADET will then find the peer in the GNUnet network and | 4586 | TCP/UDP ports. CADET will then find the peer in the GNUnet network and |
4567 | establish the proper low-level connections and do the necessary key exchanges | 4587 | establish the proper low-level connections and do the necessary key |
4568 | to assure and authenticated, secure and verified communication. Similar to | 4588 | exchanges to assure and authenticated, secure and verified communication. |
4569 | @code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel} returns a handle | 4589 | Similar to @code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel} |
4570 | to interact with the created channel. | 4590 | returns a handle to interact with the created channel. |
4571 | 4591 | ||
4572 | For every message the client wants to send to the remote application, | 4592 | For every message the client wants to send to the remote application, |
4573 | @code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the | 4593 | @code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the |
4574 | channel on which the message should be sent and the size of the message (but | 4594 | channel on which the message should be sent and the size of the message |
4575 | not the message itself!). Once CADET is ready to send the message, the provided | 4595 | (but not the message itself!). Once CADET is ready to send the message, |
4576 | callback will fire, and the message contents are provided to this callback. | 4596 | the provided callback will fire, and the message contents are provided to |
4597 | this callback. | ||
4577 | 4598 | ||
4578 | Please note the CADET does not provide an explicit notification of when a | 4599 | Please note the CADET does not provide an explicit notification of when a |
4579 | channel is connected. In loosely connected networks, like big wireless mesh | 4600 | channel is connected. In loosely connected networks, like big wireless |
4580 | networks, this can take several seconds, even minutes in the worst case. To be | 4601 | mesh networks, this can take several seconds, even minutes in the worst |
4581 | alerted when a channel is online, a client can call | 4602 | case. To be alerted when a channel is online, a client can call |
4582 | @code{GNUNET_CADET_notify_transmit_ready} immediately after | 4603 | @code{GNUNET_CADET_notify_transmit_ready} immediately after |
4583 | @code{GNUNET_CADET_create_channel}. When the callback is activated, it means | 4604 | @code{GNUNET_CADET_create_channel}. When the callback is activated, it |
4584 | that the channel is online. The callback can give 0 bytes to CADET if no | 4605 | means that the channel is online. The callback can give 0 bytes to CADET |
4585 | message is to be sent, this is ok. | 4606 | if no message is to be sent, this is ok. |
4586 | 4607 | ||
4587 | If a transmission was requested but before the callback fires it is no longer | 4608 | If a transmission was requested but before the callback fires it is no |
4588 | needed, it can be cancelled with | 4609 | longer needed, it can be cancelled with |
4589 | @code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle given | 4610 | @code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle |
4590 | back by @code{GNUNET_CADET_notify_transmit_ready}. As in the case of CORE, only | 4611 | given back by @code{GNUNET_CADET_notify_transmit_ready}. |
4591 | one message can be requested at a time: a client must not call | 4612 | As in the case of CORE, only one message can be requested at a time: a |
4592 | @code{GNUNET_CADET_notify_transmit_ready} again until the callback is called or | 4613 | client must not call @code{GNUNET_CADET_notify_transmit_ready} again until |
4593 | the request is cancelled. | 4614 | the callback is called or the request is cancelled. |
4594 | 4615 | ||
4595 | When a channel is no longer needed, a client can call | 4616 | When a channel is no longer needed, a client can call |
4596 | @code{GNUNET_CADET_channel_destroy} to get rid of it. Note that CADET will try | 4617 | @code{GNUNET_CADET_channel_destroy} to get rid of it. |
4597 | to transmit all pending traffic before notifying the remote peer of the | 4618 | Note that CADET will try to transmit all pending traffic before notifying |
4598 | destruction of the channel, including retransmitting lost messages if the | 4619 | the remote peer of the destruction of the channel, including |
4599 | channel was reliable. | 4620 | retransmitting lost messages if the channel was reliable. |
4600 | 4621 | ||
4601 | Incoming channels, channels being closed by the remote peer, and traffic on any | 4622 | Incoming channels, channels being closed by the remote peer, and traffic |
4602 | incoming or outgoing channels are given to the client when CADET executes the | 4623 | on any incoming or outgoing channels are given to the client when CADET |
4603 | callbacks given to it at the time of @code{GNUNET_CADET_connect}. | 4624 | executes the callbacks given to it at the time of |
4625 | @code{GNUNET_CADET_connect}. | ||
4604 | 4626 | ||
4605 | Finally, when an application no longer wants to use CADET, it should call | 4627 | Finally, when an application no longer wants to use CADET, it should call |
4606 | @code{GNUNET_CADET_disconnect}, but first all channels and pending | 4628 | @code{GNUNET_CADET_disconnect}, but first all channels and pending |
4607 | transmissions must be closed (otherwise CADET will complain). | 4629 | transmissions must be closed (otherwise CADET will complain). |
4608 | 4630 | ||
4631 | @cindex nse subsystem | ||
4632 | @cindex NSE | ||
4609 | @node GNUnet's NSE subsystem | 4633 | @node GNUnet's NSE subsystem |
4610 | @section GNUnet's NSE subsystem | 4634 | @section GNUnet's NSE subsystem |
4611 | 4635 | ||
4612 | 4636 | ||
4613 | NSE stands for Network Size Estimation. The NSE subsystem provides other | 4637 | NSE stands for @dfn{Network Size Estimation}. The NSE subsystem provides |
4614 | subsystems and users with a rough estimate of the number of peers currently | 4638 | other subsystems and users with a rough estimate of the number of peers |
4615 | participating in the GNUnet overlay. The computed value is not a precise number | 4639 | currently participating in the GNUnet overlay. |
4616 | as producing a precise number in a decentralized, efficient and secure way is | 4640 | The computed value is not a precise number as producing a precise number |
4617 | impossible. While NSE's estimate is inherently imprecise, NSE also gives the | 4641 | in a decentralized, efficient and secure way is impossible. |
4618 | expected range. For a peer that has been running in a stable network for a | 4642 | While NSE's estimate is inherently imprecise, NSE also gives the expected |
4619 | while, the real network size will typically (99.7% of the time) be in the range | 4643 | range. For a peer that has been running in a stable network for a |
4620 | of [2/3 estimate, 3/2 estimate]. We will now give an overview of the algorithm | 4644 | while, the real network size will typically (99.7% of the time) be in the |
4621 | used to calcualte the estimate; all of the details can be found in this | 4645 | range of [2/3 estimate, 3/2 estimate]. We will now give an overview of the |
4622 | technical report. | 4646 | algorithm used to calculate the estimate; |
4647 | all of the details can be found in this technical report. | ||
4648 | |||
4649 | @c FIXME: link to the report. | ||
4623 | 4650 | ||
4624 | @menu | 4651 | @menu |
4625 | * Motivation:: | 4652 | * Motivation:: |
@@ -4634,13 +4661,14 @@ technical report. | |||
4634 | 4661 | ||
4635 | 4662 | ||
4636 | Some subsytems, like DHT, need to know the size of the GNUnet network to | 4663 | Some subsytems, like DHT, need to know the size of the GNUnet network to |
4637 | optimize some parameters of their own protocol. The decentralized nature of | 4664 | optimize some parameters of their own protocol. The decentralized nature |
4638 | GNUnet makes efficient and securely counting the exact number of peers | 4665 | of GNUnet makes efficient and securely counting the exact number of peers |
4639 | infeasable. Although there are several decentralized algorithms to count the | 4666 | infeasable. Although there are several decentralized algorithms to count |
4640 | number of peers in a system, so far there is none to do so securely. Other | 4667 | the number of peers in a system, so far there is none to do so securely. |
4641 | protocols may allow any malicious peer to manipulate the final result or to | 4668 | Other protocols may allow any malicious peer to manipulate the final |
4642 | take advantage of the system to perform DoS (Denial of Service) attacks against | 4669 | result or to take advantage of the system to perform |
4643 | the network. GNUnet's NSE protocol avoids these drawbacks. | 4670 | @dfn{Denial of Service} (DoS) attacks against the network. |
4671 | GNUnet's NSE protocol avoids these drawbacks. | ||
4644 | 4672 | ||
4645 | 4673 | ||
4646 | 4674 | ||
@@ -4648,28 +4676,34 @@ the network. GNUnet's NSE protocol avoids these drawbacks. | |||
4648 | * Security:: | 4676 | * Security:: |
4649 | @end menu | 4677 | @end menu |
4650 | 4678 | ||
4679 | @cindex NSE security | ||
4680 | @cindex nse security | ||
4651 | @node Security | 4681 | @node Security |
4652 | @subsubsection Security | 4682 | @subsubsection Security |
4653 | 4683 | ||
4654 | 4684 | ||
4655 | The NSE subsystem is designed to be resilient against these attacks. It uses | 4685 | The NSE subsystem is designed to be resilient against these attacks. |
4656 | @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs of work} to | 4686 | It uses @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs |
4657 | prevent one peer from impersonating a large number of participants, which would | 4687 | of work} to prevent one peer from impersonating a large number of |
4658 | otherwise allow an adversary to artifically inflate the estimate. The DoS | 4688 | participants, which would otherwise allow an adversary to artifically |
4659 | protection comes from the time-based nature of the protocol: the estimates are | 4689 | inflate the estimate. |
4660 | calculated periodically and out-of-time traffic is either ignored or stored for | 4690 | The DoS protection comes from the time-based nature of the protocol: |
4661 | later retransmission by benign peers. In particular, peers cannot trigger | 4691 | the estimates are calculated periodically and out-of-time traffic is |
4662 | global network communication at will. | 4692 | either ignored or stored for later retransmission by benign peers. |
4693 | In particular, peers cannot trigger global network communication at will. | ||
4663 | 4694 | ||
4695 | @cindex NSE principle | ||
4696 | @cindex nse principle | ||
4664 | @node Principle | 4697 | @node Principle |
4665 | @subsection Principle | 4698 | @subsection Principle |
4666 | 4699 | ||
4667 | 4700 | ||
4668 | The algorithm calculates the estimate by finding the globally closest peer ID | 4701 | The algorithm calculates the estimate by finding the globally closest |
4669 | to a random, time-based value. | 4702 | peer ID to a random, time-based value. |
4670 | 4703 | ||
4671 | The idea is that the closer the ID is to the random value, the more "densely | 4704 | The idea is that the closer the ID is to the random value, the more |
4672 | packed" the ID space is, and therefore, more peers are in the network. | 4705 | "densely packed" the ID space is, and therefore, more peers are in the |
4706 | network. | ||
4673 | 4707 | ||
4674 | 4708 | ||
4675 | 4709 | ||
@@ -4686,48 +4720,51 @@ packed" the ID space is, and therefore, more peers are in the network. | |||
4686 | @subsubsection Example | 4720 | @subsubsection Example |
4687 | 4721 | ||
4688 | 4722 | ||
4689 | Suppose all peers have IDs between 0 and 100 (our ID space), and the random | 4723 | Suppose all peers have IDs between 0 and 100 (our ID space), and the |
4690 | value is 42. If the closest peer has the ID 70 we can imagine that the average | 4724 | random value is 42. |
4691 | "distance" between peers is around 30 and therefore the are around 3 peers in | 4725 | If the closest peer has the ID 70 we can imagine that the average |
4692 | the whole ID space. On the other hand, if the closest peer has the ID 44, we | 4726 | "distance" between peers is around 30 and therefore the are around 3 |
4693 | can imagine that the space is rather packed with peers, maybe as much as 50 of | 4727 | peers in the whole ID space. On the other hand, if the closest peer has |
4694 | them. Naturally, we could have been rather unlucky, and there is only one peer | 4728 | the ID 44, we can imagine that the space is rather packed with peers, |
4695 | and happens to have the ID 44. Thus, the current estimate is calculated as the | 4729 | maybe as much as 50 of them. |
4696 | average over multiple rounds, and not just a single sample. | 4730 | Naturally, we could have been rather unlucky, and there is only one peer |
4731 | and happens to have the ID 44. Thus, the current estimate is calculated | ||
4732 | as the average over multiple rounds, and not just a single sample. | ||
4697 | 4733 | ||
4698 | @node Algorithm | 4734 | @node Algorithm |
4699 | @subsubsection Algorithm | 4735 | @subsubsection Algorithm |
4700 | 4736 | ||
4701 | 4737 | ||
4702 | Given that example, one can imagine that the job of the subsystem is to | 4738 | Given that example, one can imagine that the job of the subsystem is to |
4703 | efficiently communicate the ID of the closest peer to the target value to all | 4739 | efficiently communicate the ID of the closest peer to the target value |
4704 | the other peers, who will calculate the estimate from it. | 4740 | to all the other peers, who will calculate the estimate from it. |
4705 | 4741 | ||
4706 | @node Target value | 4742 | @node Target value |
4707 | @subsubsection Target value | 4743 | @subsubsection Target value |
4708 | 4744 | ||
4709 | @c %**end of header | 4745 | @c %**end of header |
4710 | 4746 | ||
4711 | The target value itself is generated by hashing the current time, rounded down | 4747 | The target value itself is generated by hashing the current time, rounded |
4712 | to an agreed value. If the rounding amount is 1h (default) and the time is | 4748 | down to an agreed value. If the rounding amount is 1h (default) and the |
4713 | 12:34:56, the time to hash would be 12:00:00. The process is repeated each | 4749 | time is 12:34:56, the time to hash would be 12:00:00. The process is |
4714 | rouning amount (in this example would be every hour). Every repetition is | 4750 | repeated each rouning amount (in this example would be every hour). |
4715 | called a round. | 4751 | Every repetition is called a round. |
4716 | 4752 | ||
4717 | @node Timing | 4753 | @node Timing |
4718 | @subsubsection Timing | 4754 | @subsubsection Timing |
4719 | @c %**end of header | 4755 | @c %**end of header |
4720 | 4756 | ||
4721 | The NSE subsystem has some timing control to avoid everybody broadcasting its | 4757 | The NSE subsystem has some timing control to avoid everybody broadcasting |
4722 | ID all at one. Once each peer has the target random value, it compares its own | 4758 | its ID all at one. Once each peer has the target random value, it |
4723 | ID to the target and calculates the hypothetical size of the network if that | 4759 | compares its own ID to the target and calculates the hypothetical size of |
4724 | peer were to be the closest. Then it compares the hypothetical size with the | 4760 | the network if that peer were to be the closest. |
4725 | estimate from the previous rounds. For each value there is an assiciated point | 4761 | Then it compares the hypothetical size with the estimate from the previous |
4726 | in the period, let's call it "broadcast time". If its own hypothetical estimate | 4762 | rounds. For each value there is an assiciated point in the period, |
4727 | is the same as the previous global estimate, its "broadcast time" will be in | 4763 | let's call it "broadcast time". If its own hypothetical estimate |
4728 | the middle of the round. If its bigger it will be earlier and if its smaler | 4764 | is the same as the previous global estimate, its "broadcast time" will be |
4729 | (the most likely case) it will be later. This ensures that the peers closests | 4765 | in the middle of the round. If its bigger it will be earlier and if its |
4730 | to the target value start broadcasting their ID the first. | 4766 | smaller (the most likely case) it will be later. This ensures that the |
4767 | peers closests to the target value start broadcasting their ID the first. | ||
4731 | 4768 | ||
4732 | @node Controlled Flooding | 4769 | @node Controlled Flooding |
4733 | @subsubsection Controlled Flooding | 4770 | @subsubsection Controlled Flooding |
@@ -4735,52 +4772,56 @@ to the target value start broadcasting their ID the first. | |||
4735 | @c %**end of header | 4772 | @c %**end of header |
4736 | 4773 | ||
4737 | When a peer receives a value, first it verifies that it is closer than the | 4774 | When a peer receives a value, first it verifies that it is closer than the |
4738 | closest value it had so far, otherwise it answers the incoming message with a | 4775 | closest value it had so far, otherwise it answers the incoming message |
4739 | message containing the better value. Then it checks a proof of work that must | 4776 | with a message containing the better value. Then it checks a proof of |
4740 | be included in the incoming message, to ensure that the other peer's ID is not | 4777 | work that must be included in the incoming message, to ensure that the |
4741 | made up (otherwise a malicious peer could claim to have an ID of exactly the | 4778 | other peer's ID is not made up (otherwise a malicious peer could claim to |
4742 | target value every round). Once validated, it compares the brodcast time of the | 4779 | have an ID of exactly the target value every round). Once validated, it |
4743 | received value with the current time and if it's not too early, sends the | 4780 | compares the brodcast time of the received value with the current time |
4744 | received value to its neighbors. Otherwise it stores the value until the | 4781 | and if it's not too early, sends the received value to its neighbors. |
4745 | correct broadcast time comes. This prevents unnecessary traffic of sub-optimal | 4782 | Otherwise it stores the value until the correct broadcast time comes. |
4746 | values, since a better value can come before the broadcast time, rendering the | 4783 | This prevents unnecessary traffic of sub-optimal values, since a better |
4747 | previous one obsolete and saving the traffic that would have been used to | 4784 | value can come before the broadcast time, rendering the previous one |
4748 | broadcast it to the neighbors. | 4785 | obsolete and saving the traffic that would have been used to broadcast it |
4786 | to the neighbors. | ||
4749 | 4787 | ||
4750 | @node Calculating the estimate | 4788 | @node Calculating the estimate |
4751 | @subsubsection Calculating the estimate | 4789 | @subsubsection Calculating the estimate |
4752 | 4790 | ||
4753 | @c %**end of header | 4791 | @c %**end of header |
4754 | 4792 | ||
4755 | Once the closest ID has been spread across the network each peer gets the exact | 4793 | Once the closest ID has been spread across the network each peer gets the |
4756 | distance betweed this ID and the target value of the round and calculates the | 4794 | exact distance betweed this ID and the target value of the round and |
4757 | estimate with a mathematical formula described in the tech report. The estimate | 4795 | calculates the estimate with a mathematical formula described in the tech |
4758 | generated with this method for a single round is not very precise. Remember the | 4796 | report. The estimate generated with this method for a single round is not |
4759 | case of the example, where the only peer is the ID 44 and we happen to generate | 4797 | very precise. Remember the case of the example, where the only peer is the |
4760 | the target value 42, thinking there are 50 peers in the network. Therefore, the | 4798 | ID 44 and we happen to generate the target value 42, thinking there are |
4761 | NSE subsystem remembers the last 64 estimates and calculates an average over | 4799 | 50 peers in the network. Therefore, the NSE subsystem remembers the last |
4762 | them, giving a result of which usually has one bit of uncertainty (the real | 4800 | 64 estimates and calculates an average over them, giving a result of which |
4763 | size could be half of the estimate or twice as much). Note that the actual | 4801 | usually has one bit of uncertainty (the real size could be half of the |
4764 | network size is calculated in powers of two of the raw input, thus one bit of | 4802 | estimate or twice as much). Note that the actual network size is |
4765 | uncertainty means a factor of two in the size estimate. | 4803 | calculated in powers of two of the raw input, thus one bit of uncertainty |
4804 | means a factor of two in the size estimate. | ||
4766 | 4805 | ||
4806 | @cindex libgnunetnse | ||
4767 | @node libgnunetnse | 4807 | @node libgnunetnse |
4768 | @subsection libgnunetnse | 4808 | @subsection libgnunetnse |
4769 | 4809 | ||
4770 | @c %**end of header | 4810 | @c %**end of header |
4771 | 4811 | ||
4772 | The NSE subsystem has the simplest API of all services, with only two calls: | 4812 | The NSE subsystem has the simplest API of all services, with only two |
4773 | @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}. | 4813 | calls: @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}. |
4774 | 4814 | ||
4775 | The connect call gets a callback function as a parameter and this function is | 4815 | The connect call gets a callback function as a parameter and this function |
4776 | called each time the network agrees on an estimate. This usually is once per | 4816 | is called each time the network agrees on an estimate. This usually is |
4777 | round, with some exceptions: if the closest peer has a late local clock and | 4817 | once per round, with some exceptions: if the closest peer has a late |
4778 | starts spreading his ID after everyone else agreed on a value, the callback | 4818 | local clock and starts spreading his ID after everyone else agreed on a |
4779 | might be activated twice in a round, the second value being always bigger than | 4819 | value, the callback might be activated twice in a round, the second value |
4780 | the first. The default round time is set to 1 hour. | 4820 | being always bigger than the first. The default round time is set to |
4821 | 1 hour. | ||
4781 | 4822 | ||
4782 | The disconnect call disconnects from the NSE subsystem and the callback is no | 4823 | The disconnect call disconnects from the NSE subsystem and the callback |
4783 | longer called with new estimates. | 4824 | is no longer called with new estimates. |
4784 | 4825 | ||
4785 | 4826 | ||
4786 | 4827 | ||
@@ -4795,11 +4836,11 @@ longer called with new estimates. | |||
4795 | @c %**end of header | 4836 | @c %**end of header |
4796 | 4837 | ||
4797 | The callback provides two values: the average and the | 4838 | The callback provides two values: the average and the |
4798 | @uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation} of | 4839 | @uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation} |
4799 | the last 64 rounds. The values provided by the callback function are | 4840 | of the last 64 rounds. The values provided by the callback function are |
4800 | logarithmic, this means that the real estimate numbers can be obtained by | 4841 | logarithmic, this means that the real estimate numbers can be obtained by |
4801 | calculating 2 to the power of the given value (2average). From a statistics | 4842 | calculating 2 to the power of the given value (2average). From a |
4802 | point of view this means that: | 4843 | statistics point of view this means that: |
4803 | 4844 | ||
4804 | @itemize @bullet | 4845 | @itemize @bullet |
4805 | @item 68% of the time the real size is included in the interval | 4846 | @item 68% of the time the real size is included in the interval |
@@ -4810,8 +4851,8 @@ point of view this means that: | |||
4810 | [(2average-3*stddev, 2average+3*stddev] | 4851 | [(2average-3*stddev, 2average+3*stddev] |
4811 | @end itemize | 4852 | @end itemize |
4812 | 4853 | ||
4813 | The expected standard variation for 64 rounds in a network of stable size is | 4854 | The expected standard variation for 64 rounds in a network of stable size |
4814 | 0.2. Thus, we can say that normally: | 4855 | is 0.2. Thus, we can say that normally: |
4815 | 4856 | ||
4816 | @itemize @bullet | 4857 | @itemize @bullet |
4817 | @item 68% of the time the real size is in the range [-13%, +15%] | 4858 | @item 68% of the time the real size is in the range [-13%, +15%] |
@@ -4819,10 +4860,11 @@ The expected standard variation for 64 rounds in a network of stable size is | |||
4819 | @item 99.7% of the time the real size is in the range [-34%, +52%] | 4860 | @item 99.7% of the time the real size is in the range [-34%, +52%] |
4820 | @end itemize | 4861 | @end itemize |
4821 | 4862 | ||
4822 | As said in the introduction, we can be quite sure that usually the real size is | 4863 | As said in the introduction, we can be quite sure that usually the real |
4823 | between one third and three times the estimate. This can of course vary with | 4864 | size is between one third and three times the estimate. This can of |
4824 | network conditions. Thus, applications may want to also consider the provided | 4865 | course vary with network conditions. |
4825 | standard deviation value, not only the average (in particular, if the standard | 4866 | Thus, applications may want to also consider the provided standard |
4867 | deviation value, not only the average (in particular, if the standard | ||
4826 | veriation is very high, the average maybe meaningless: the network size is | 4868 | veriation is very high, the average maybe meaningless: the network size is |
4827 | changing rapidly). | 4869 | changing rapidly). |
4828 | 4870 | ||
@@ -4835,15 +4877,23 @@ Let's close with a couple examples. | |||
4835 | 4877 | ||
4836 | @table @asis | 4878 | @table @asis |
4837 | 4879 | ||
4838 | @item Average: 10, std dev: 1 Here the estimate would be 2^10 = 1024 peers. @footnote{The range in which we can be 95% sure is: [2^8, 2^12] = [256, 4096]. We can be very (>99.7%) sure that the network is not a hundred peers and absolutely sure that it is not a million peers, but somewhere around a thousand.} | 4880 | @item Average: 10, std dev: 1 Here the estimate would be |
4881 | 2^10 = 1024 peers. @footnote{The range in which we can be 95% sure is: | ||
4882 | [2^8, 2^12] = [256, 4096]. We can be very (>99.7%) sure that the network | ||
4883 | is not a hundred peers and absolutely sure that it is not a million peers, | ||
4884 | but somewhere around a thousand.} | ||
4839 | 4885 | ||
4840 | @item Average 22, std dev: 0.2 Here the estimate would be 2^22 = 4 Million peers. @footnote{The range in which we can be 99.7% sure is: [2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size is around four million, with absolutely way of it being 1 million.} | 4886 | @item Average 22, std dev: 0.2 Here the estimate would be |
4887 | 2^22 = 4 Million peers. @footnote{The range in which we can be 99.7% sure | ||
4888 | is: [2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size | ||
4889 | is around four million, with absolutely way of it being 1 million.} | ||
4841 | 4890 | ||
4842 | @end table | 4891 | @end table |
4843 | 4892 | ||
4844 | To put this in perspective, if someone remembers the LHC Higgs boson results, | 4893 | To put this in perspective, if someone remembers the LHC Higgs boson |
4845 | were announced with "5 sigma" and "6 sigma" certainties. In this case a 5 sigma | 4894 | results, were announced with "5 sigma" and "6 sigma" certainties. In this |
4846 | minimum would be 2 million and a 6 sigma minimum, 1.8 million. | 4895 | case a 5 sigma minimum would be 2 million and a 6 sigma minimum, |
4896 | 1.8 million. | ||
4847 | 4897 | ||
4848 | @node The NSE Client-Service Protocol | 4898 | @node The NSE Client-Service Protocol |
4849 | @subsection The NSE Client-Service Protocol | 4899 | @subsection The NSE Client-Service Protocol |
@@ -4854,16 +4904,16 @@ As with the API, the client-service protocol is very simple, only has 2 | |||
4854 | different messages, defined in @code{src/nse/nse.h}: | 4904 | different messages, defined in @code{src/nse/nse.h}: |
4855 | 4905 | ||
4856 | @itemize @bullet | 4906 | @itemize @bullet |
4857 | @item @code{GNUNET_MESSAGE_TYPE_NSE_START}@ This message has no parameters and | 4907 | @item @code{GNUNET_MESSAGE_TYPE_NSE_START}@ This message has no parameters |
4858 | is sent from the client to the service upon connection. | 4908 | and is sent from the client to the service upon connection. |
4859 | @item @code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE}@ This message is sent from the | 4909 | @item @code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE}@ This message is sent from |
4860 | service to the client for every new estimate and upon connection. Contains a | 4910 | the service to the client for every new estimate and upon connection. |
4861 | timestamp for the estimate, the average and the standard deviation for the | 4911 | Contains a timestamp for the estimate, the average and the standard |
4862 | respective round. | 4912 | deviation for the respective round. |
4863 | @end itemize | 4913 | @end itemize |
4864 | 4914 | ||
4865 | When the @code{GNUNET_NSE_disconnect} API call is executed, the client simply | 4915 | When the @code{GNUNET_NSE_disconnect} API call is executed, the client |
4866 | disconnects from the service, with no message involved. | 4916 | simply disconnects from the service, with no message involved. |
4867 | 4917 | ||
4868 | @node The NSE Peer-to-Peer Protocol | 4918 | @node The NSE Peer-to-Peer Protocol |
4869 | @subsection The NSE Peer-to-Peer Protocol | 4919 | @subsection The NSE Peer-to-Peer Protocol |
@@ -4873,77 +4923,80 @@ disconnects from the service, with no message involved. | |||
4873 | The NSE subsystem only has one message in the P2P protocol, the | 4923 | The NSE subsystem only has one message in the P2P protocol, the |
4874 | @code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message. | 4924 | @code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message. |
4875 | 4925 | ||
4876 | This message key contents are the timestamp to identify the round (differences | 4926 | This message key contents are the timestamp to identify the round |
4877 | in system clocks may cause some peers to send messages way too early or way too | 4927 | (differences in system clocks may cause some peers to send messages way |
4878 | late, so the timestamp allows other peers to identify such messages easily), | 4928 | too early or way too late, so the timestamp allows other peers to |
4879 | the @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proof of work} | 4929 | identify such messages easily), the |
4930 | @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proof of work} | ||
4880 | used to make it difficult to mount a | 4931 | used to make it difficult to mount a |
4881 | @uref{http://en.wikipedia.org/wiki/Sybil_attack, Sybil attack}, and the public | 4932 | @uref{http://en.wikipedia.org/wiki/Sybil_attack, Sybil attack}, and the |
4882 | key, which is used to verify the signature on the message. | 4933 | public key, which is used to verify the signature on the message. |
4883 | 4934 | ||
4884 | Every peer stores a message for the previous, current and next round. The | 4935 | Every peer stores a message for the previous, current and next round. The |
4885 | messages for the previous and current round are given to peers that connect to | 4936 | messages for the previous and current round are given to peers that |
4886 | us. The message for the next round is simply stored until our system clock | 4937 | connect to us. The message for the next round is simply stored until our |
4887 | advances to the next round. The message for the current round is what we are | 4938 | system clock advances to the next round. The message for the current round |
4888 | flooding the network with right now. At the beginning of each round the peer | 4939 | is what we are flooding the network with right now. |
4889 | does the following: | 4940 | At the beginning of each round the peer does the following: |
4890 | 4941 | ||
4891 | @itemize @bullet | 4942 | @itemize @bullet |
4892 | @item calculates his own distance to the target value | 4943 | @item calculates his own distance to the target value |
4893 | @item creates, signs and stores the message for the current round (unless it | 4944 | @item creates, signs and stores the message for the current round (unless |
4894 | has a better message in the "next round" slot which came early in the previous | 4945 | it has a better message in the "next round" slot which came early in the |
4895 | round) | 4946 | previous round) |
4896 | @item calculates, based on the stored round message (own or received) when to | 4947 | @item calculates, based on the stored round message (own or received) when |
4897 | stard flooding it to its neighbors | 4948 | to stard flooding it to its neighbors |
4898 | @end itemize | 4949 | @end itemize |
4899 | 4950 | ||
4900 | Upon receiving a message the peer checks the validity of the message (round, | 4951 | Upon receiving a message the peer checks the validity of the message |
4901 | proof of work, signature). The next action depends on the contents of the | 4952 | (round, proof of work, signature). The next action depends on the |
4902 | incoming message: | 4953 | contents of the incoming message: |
4903 | 4954 | ||
4904 | @itemize @bullet | 4955 | @itemize @bullet |
4905 | @item if the message is worse than the current stored message, the peer sends | 4956 | @item if the message is worse than the current stored message, the peer |
4906 | the current message back immediately, to stop the other peer from spreading | 4957 | sends the current message back immediately, to stop the other peer from |
4907 | suboptimal results | 4958 | spreading suboptimal results |
4908 | @item if the message is better than the current stored message, the peer stores | 4959 | @item if the message is better than the current stored message, the peer |
4909 | the new message and calculates the new target time to start spreading it to its | 4960 | stores the new message and calculates the new target time to start |
4910 | neighbors (excluding the one the message came from) | 4961 | spreading it to its neighbors (excluding the one the message came from) |
4911 | @item if the message is for the previous round, it is compared to the message | 4962 | @item if the message is for the previous round, it is compared to the |
4912 | stored in the "previous round slot", which may then be updated | 4963 | message stored in the "previous round slot", which may then be updated |
4913 | @item if the message is for the next round, it is compared to the message | 4964 | @item if the message is for the next round, it is compared to the message |
4914 | stored in the "next round slot", which again may then be updated | 4965 | stored in the "next round slot", which again may then be updated |
4915 | @end itemize | 4966 | @end itemize |
4916 | 4967 | ||
4917 | Finally, when it comes to send the stored message for the current round to the | 4968 | Finally, when it comes to send the stored message for the current round to |
4918 | neighbors there is a random delay added for each neighbor, to avoid traffic | 4969 | the neighbors there is a random delay added for each neighbor, to avoid |
4919 | spikes and minimize cross-messages. | 4970 | traffic spikes and minimize cross-messages. |
4920 | 4971 | ||
4972 | @cindex HOSTLIST subsystem | ||
4973 | @cindex hostlist subsystem | ||
4921 | @node GNUnet's HOSTLIST subsystem | 4974 | @node GNUnet's HOSTLIST subsystem |
4922 | @section GNUnet's HOSTLIST subsystem | 4975 | @section GNUnet's HOSTLIST subsystem |
4923 | 4976 | ||
4924 | @c %**end of header | 4977 | @c %**end of header |
4925 | 4978 | ||
4926 | Peers in the GNUnet overlay network need address information so that they can | 4979 | Peers in the GNUnet overlay network need address information so that they |
4927 | connect with other peers. GNUnet uses so called HELLO messages to store and | 4980 | can connect with other peers. GNUnet uses so called HELLO messages to |
4928 | exchange peer addresses. GNUnet provides several methods for peers to obtain | 4981 | store and exchange peer addresses. |
4929 | this information: | 4982 | GNUnet provides several methods for peers to obtain this information: |
4930 | 4983 | ||
4931 | @itemize @bullet | 4984 | @itemize @bullet |
4932 | @item out-of-band exchange of HELLO messages (manually, using for example | 4985 | @item out-of-band exchange of HELLO messages (manually, using for example |
4933 | gnunet-peerinfo) | 4986 | gnunet-peerinfo) |
4934 | @item HELLO messages shipped with GNUnet (automatic with distribution) | 4987 | @item HELLO messages shipped with GNUnet (automatic with distribution) |
4935 | @item UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast) | 4988 | @item UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast) |
4936 | @item topology gossiping (learning from other peers we already connected to), | 4989 | @item topology gossiping (learning from other peers we already connected |
4937 | and | 4990 | to), and |
4938 | @item the HOSTLIST daemon covered in this section, which is particularly | 4991 | @item the HOSTLIST daemon covered in this section, which is particularly |
4939 | relevant for bootstrapping new peers. | 4992 | relevant for bootstrapping new peers. |
4940 | @end itemize | 4993 | @end itemize |
4941 | 4994 | ||
4942 | New peers have no existing connections (and thus cannot learn from gossip among | 4995 | New peers have no existing connections (and thus cannot learn from gossip |
4943 | peers), may not have other peers in their LAN and might be started with an | 4996 | among peers), may not have other peers in their LAN and might be started |
4944 | outdated set of HELLO messages from the distribution. In this case, getting new | 4997 | with an outdated set of HELLO messages from the distribution. |
4945 | peers to connect to the network requires either manual effort or the use of a | 4998 | In this case, getting new peers to connect to the network requires either |
4946 | HOSTLIST to obtain HELLOs. | 4999 | manual effort or the use of a HOSTLIST to obtain HELLOs. |
4947 | 5000 | ||
4948 | @menu | 5001 | @menu |
4949 | * HELLOs:: | 5002 | * HELLOs:: |
@@ -4961,11 +5014,12 @@ HOSTLIST to obtain HELLOs. | |||
4961 | 5014 | ||
4962 | @c %**end of header | 5015 | @c %**end of header |
4963 | 5016 | ||
4964 | The basic information peers require to connect to other peers are contained in | 5017 | The basic information peers require to connect to other peers are |
4965 | so called HELLO messages you can think of as a business card. Besides the | 5018 | contained in so called HELLO messages you can think of as a business card. |
4966 | identity of the peer (based on the cryptographic public key) a HELLO message | 5019 | Besides the identity of the peer (based on the cryptographic public key) a |
4967 | may contain address information that specifies ways to contact a peer. By | 5020 | HELLO message may contain address information that specifies ways to |
4968 | obtaining HELLO messages, a peer can learn how to contact other peers. | 5021 | contact a peer. By obtaining HELLO messages, a peer can learn how to |
5022 | contact other peers. | ||
4969 | 5023 | ||
4970 | @node Overview for the HOSTLIST subsystem | 5024 | @node Overview for the HOSTLIST subsystem |
4971 | @subsection Overview for the HOSTLIST subsystem | 5025 | @subsection Overview for the HOSTLIST subsystem |
@@ -4973,17 +5027,19 @@ obtaining HELLO messages, a peer can learn how to contact other peers. | |||
4973 | @c %**end of header | 5027 | @c %**end of header |
4974 | 5028 | ||
4975 | The HOSTLIST subsystem provides a way to distribute and obtain contact | 5029 | The HOSTLIST subsystem provides a way to distribute and obtain contact |
4976 | information to connect to other peers using a simple HTTP GET request. It's | 5030 | information to connect to other peers using a simple HTTP GET request. |
4977 | implementation is split in three parts, the main file for the daemon itself | 5031 | It's implementation is split in three parts, the main file for the daemon |
4978 | (gnunet-daemon-hostlist.c), the HTTP client used to download peer information | 5032 | itself (@file{gnunet-daemon-hostlist.c}), the HTTP client used to download |
4979 | (hostlist-client.c) and the server component used to provide this information | 5033 | peer information (@file{hostlist-client.c}) and the server component used |
4980 | to other peers (hostlist-server.c). The server is basically a small HTTP web | 5034 | to provide this information to other peers (@file{hostlist-server.c}). |
4981 | server (based on GNU libmicrohttpd) which provides a list of HELLOs known to | 5035 | The server is basically a small HTTP web server (based on GNU |
4982 | the local peer for download. The client component is basically a HTTP client | 5036 | libmicrohttpd) which provides a list of HELLOs known to the local peer for |
4983 | (based on libcurl) which can download hostlists from one or more websites. The | 5037 | download. The client component is basically a HTTP client |
4984 | hostlist format is a binary blob containing a sequence of HELLO messages. Note | 5038 | (based on libcurl) which can download hostlists from one or more websites. |
4985 | that any HTTP server can theoretically serve a hostlist, the build-in hostlist | 5039 | The hostlist format is a binary blob containing a sequence of HELLO |
4986 | server makes it simply convenient to offer this service. | 5040 | messages. Note that any HTTP server can theoretically serve a hostlist, |
5041 | the build-in hostlist server makes it simply convenient to offer this | ||
5042 | service. | ||
4987 | 5043 | ||
4988 | 5044 | ||
4989 | @menu | 5045 | @menu |
@@ -4999,13 +5055,14 @@ server makes it simply convenient to offer this service. | |||
4999 | The HOSTLIST daemon can: | 5055 | The HOSTLIST daemon can: |
5000 | 5056 | ||
5001 | @itemize @bullet | 5057 | @itemize @bullet |
5002 | @item provide HELLO messages with validated addresses obtained from PEERINFO to | 5058 | @item provide HELLO messages with validated addresses obtained from |
5003 | download for other peers | 5059 | PEERINFO to download for other peers |
5004 | @item download HELLO messages and forward these message to the TRANSPORT | 5060 | @item download HELLO messages and forward these message to the TRANSPORT |
5005 | subsystem for validation | 5061 | subsystem for validation |
5006 | @item advertises the URL of this peer's hostlist address to other peers via | 5062 | @item advertises the URL of this peer's hostlist address to other peers |
5007 | gossip | 5063 | via gossip |
5008 | @item automatically learn about hostlist servers from the gossip of other peers | 5064 | @item automatically learn about hostlist servers from the gossip of other |
5065 | peers | ||
5009 | @end itemize | 5066 | @end itemize |
5010 | 5067 | ||
5011 | @node Limitations2 | 5068 | @node Limitations2 |
@@ -5025,25 +5082,28 @@ The HOSTLIST daemon does not: | |||
5025 | 5082 | ||
5026 | @c %**end of header | 5083 | @c %**end of header |
5027 | 5084 | ||
5028 | The HOSTLIST subsystem is currently implemented as a daemon, so there is no | 5085 | The HOSTLIST subsystem is currently implemented as a daemon, so there is |
5029 | need for the user to interact with it and therefore there is no command line | 5086 | no need for the user to interact with it and therefore there is no |
5030 | tool and no API to communicate with the daemon. In the future, we can envision | 5087 | command line tool and no API to communicate with the daemon. In the |
5031 | changing this to allow users to manually trigger the download of a hostlist. | 5088 | future, we can envision changing this to allow users to manually trigger |
5089 | the download of a hostlist. | ||
5090 | |||
5091 | Since there is no command line interface to interact with HOSTLIST, the | ||
5092 | only way to interact with the hostlist is to use STATISTICS to obtain or | ||
5093 | modify information about the status of HOSTLIST: | ||
5032 | 5094 | ||
5033 | Since there is no command line interface to interact with HOSTLIST, the only | ||
5034 | way to interact with the hostlist is to use STATISTICS to obtain or modify | ||
5035 | information about the status of HOSTLIST: | ||
5036 | @example | 5095 | @example |
5037 | $ gnunet-statistics -s hostlist | 5096 | $ gnunet-statistics -s hostlist |
5038 | @end example | 5097 | @end example |
5039 | 5098 | ||
5040 | In particular, HOSTLIST includes a @strong{persistent} value in statistics that | 5099 | @noindent |
5041 | specifies when the hostlist server might be queried next. As this value is | 5100 | In particular, HOSTLIST includes a @strong{persistent} value in statistics |
5042 | exponentially increasing during runtime, developers may want to reset or | 5101 | that specifies when the hostlist server might be queried next. As this |
5043 | manually adjust it. Note that HOSTLIST (but not STATISTICS) needs to be | 5102 | value is exponentially increasing during runtime, developers may want to |
5044 | shutdown if changes to this value are to have any effect on the daemon (as | 5103 | reset or manually adjust it. Note that HOSTLIST (but not STATISTICS) needs |
5045 | HOSTLIST does not monitor STATISTICS for changes to the download | 5104 | to be shutdown if changes to this value are to have any effect on the |
5046 | frequency). | 5105 | daemon (as HOSTLIST does not monitor STATISTICS for changes to the |
5106 | download frequency). | ||
5047 | 5107 | ||
5048 | @node Hostlist security address validation | 5108 | @node Hostlist security address validation |
5049 | @subsection Hostlist security address validation | 5109 | @subsection Hostlist security address validation |
@@ -5051,18 +5111,19 @@ frequency). | |||
5051 | @c %**end of header | 5111 | @c %**end of header |
5052 | 5112 | ||
5053 | Since information obtained from other parties cannot be trusted without | 5113 | Since information obtained from other parties cannot be trusted without |
5054 | validation, we have to distinguish between @emph{validated} and @emph{not | 5114 | validation, we have to distinguish between @emph{validated} and |
5055 | validated} addresses. Before using (and so trusting) information from other | 5115 | @emph{not validated} addresses. Before using (and so trusting) |
5056 | parties, this information has to be double-checked (validated). Address | 5116 | information from other parties, this information has to be double-checked |
5057 | validation is not done by HOSTLIST but by the TRANSPORT service. | 5117 | (validated). Address validation is not done by HOSTLIST but by the |
5058 | 5118 | TRANSPORT service. | |
5059 | The HOSTLIST component is functionally located between the PEERINFO and the | 5119 | |
5060 | TRANSPORT subsystem. When acting as a server, the daemon obtains valid | 5120 | The HOSTLIST component is functionally located between the PEERINFO and |
5061 | (@emph{validated}) peer information (HELLO messages) from the PEERINFO service | 5121 | the TRANSPORT subsystem. When acting as a server, the daemon obtains valid |
5062 | and provides it to other peers. When acting as a client, it contacts the | 5122 | (@emph{validated}) peer information (HELLO messages) from the PEERINFO |
5063 | HOSTLIST servers specified in the configuration, downloads the (unvalidated) | 5123 | service and provides it to other peers. When acting as a client, it |
5064 | list of HELLO messages and forwards these information to the TRANSPORT server | 5124 | contacts the HOSTLIST servers specified in the configuration, downloads |
5065 | to validate the addresses. | 5125 | the (unvalidated) list of HELLO messages and forwards these information |
5126 | to the TRANSPORT server to validate the addresses. | ||
5066 | 5127 | ||
5067 | @node The HOSTLIST daemon | 5128 | @node The HOSTLIST daemon |
5068 | @subsection The HOSTLIST daemon | 5129 | @subsection The HOSTLIST daemon |
@@ -5070,24 +5131,25 @@ to validate the addresses. | |||
5070 | @c %**end of header | 5131 | @c %**end of header |
5071 | 5132 | ||
5072 | The hostlist daemon is the main component of the HOSTLIST subsystem. It is | 5133 | The hostlist daemon is the main component of the HOSTLIST subsystem. It is |
5073 | started by the ARM service and (if configured) starts the HOSTLIST client and | 5134 | started by the ARM service and (if configured) starts the HOSTLIST client |
5074 | server components. | 5135 | and server components. |
5075 | 5136 | ||
5076 | If the daemon provides a hostlist itself it can advertise it's own hostlist to | 5137 | If the daemon provides a hostlist itself it can advertise it's own |
5077 | other peers. To do so it sends a GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT | 5138 | hostlist to other peers. To do so it sends a |
5078 | message to other peers when they connect to this peer on the CORE level. This | 5139 | @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to other peers |
5079 | hostlist advertisement message contains the URL to access the HOSTLIST HTTP | 5140 | when they connect to this peer on the CORE level. This hostlist |
5080 | server of the sender. The daemon may also subscribe to this type of message | 5141 | advertisement message contains the URL to access the HOSTLIST HTTP |
5081 | from CORE service, and then forward these kind of message to the HOSTLIST | 5142 | server of the sender. The daemon may also subscribe to this type of |
5082 | client. The client then uses all available URLs to download peer information | 5143 | message from CORE service, and then forward these kind of message to the |
5083 | when necessary. | 5144 | HOSTLIST client. The client then uses all available URLs to download peer |
5084 | 5145 | information when necessary. | |
5085 | When starting, the HOSTLIST daemon first connects to the CORE subsystem and if | 5146 | |
5086 | hostlist learning is enabled, registers a CORE handler to receive this kind of | 5147 | When starting, the HOSTLIST daemon first connects to the CORE subsystem |
5087 | messages. Next it starts (if configured) the client and server. It passes | 5148 | and if hostlist learning is enabled, registers a CORE handler to receive |
5088 | pointers to CORE connect and disconnect and receive handlers where the client | 5149 | this kind of messages. Next it starts (if configured) the client and |
5089 | and server store their functions, so the daemon can notify them about CORE | 5150 | server. It passes pointers to CORE connect and disconnect and receive |
5090 | events. | 5151 | handlers where the client and server store their functions, so the daemon |
5152 | can notify them about CORE events. | ||
5091 | 5153 | ||
5092 | To clean up on shutdown, the daemon has a cleaning task, shutting down all | 5154 | To clean up on shutdown, the daemon has a cleaning task, shutting down all |
5093 | subsystems and disconnecting from CORE. | 5155 | subsystems and disconnecting from CORE. |
@@ -5097,10 +5159,10 @@ subsystems and disconnecting from CORE. | |||
5097 | 5159 | ||
5098 | @c %**end of header | 5160 | @c %**end of header |
5099 | 5161 | ||
5100 | The server provides a way for other peers to obtain HELLOs. Basically it is a | 5162 | The server provides a way for other peers to obtain HELLOs. Basically it |
5101 | small web server other peers can connect to and download a list of HELLOs using | 5163 | is a small web server other peers can connect to and download a list of |
5102 | standard HTTP; it may also advertise the URL of the hostlist to other peers | 5164 | HELLOs using standard HTTP; it may also advertise the URL of the hostlist |
5103 | connecting on CORE level. | 5165 | to other peers connecting on CORE level. |
5104 | 5166 | ||
5105 | 5167 | ||
5106 | @menu | 5168 | @menu |
@@ -5113,49 +5175,56 @@ connecting on CORE level. | |||
5113 | 5175 | ||
5114 | @c %**end of header | 5176 | @c %**end of header |
5115 | 5177 | ||
5116 | During startup, the server starts a web server listening on the port specified | 5178 | During startup, the server starts a web server listening on the port |
5117 | with the HTTPPORT value (default 8080). In addition it connects to the PEERINFO | 5179 | specified with the HTTPPORT value (default 8080). In addition it connects |
5118 | service to obtain peer information. The HOSTLIST server uses the | 5180 | to the PEERINFO service to obtain peer information. The HOSTLIST server |
5119 | GNUNET_PEERINFO_iterate function to request HELLO information for all peers and | 5181 | uses the GNUNET_PEERINFO_iterate function to request HELLO information for |
5120 | adds their information to a new hostlist if they are suitable (expired | 5182 | all peers and adds their information to a new hostlist if they are |
5121 | addresses and HELLOs without addresses are both not suitable) and the maximum | 5183 | suitable (expired addresses and HELLOs without addresses are both not |
5122 | size for a hostlist is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When | 5184 | suitable) and the maximum size for a hostlist is not exceeded |
5123 | PEERINFO finishes (with a last NULL callback), the server destroys the previous | 5185 | (MAX_BYTES_PER_HOSTLISTS = 500000). |
5124 | hostlist response available for download on the web server and replaces it with | 5186 | When PEERINFO finishes (with a last NULL callback), the server destroys |
5125 | the updated hostlist. The hostlist format is basically a sequence of HELLO | 5187 | the previous hostlist response available for download on the web server |
5126 | messages (as obtained from PEERINFO) without any special tokenization. Since | 5188 | and replaces it with the updated hostlist. The hostlist format is |
5127 | each HELLO message contains a size field, the response can easily be split into | 5189 | basically a sequence of HELLO messages (as obtained from PEERINFO) without |
5128 | separate HELLO messages by the client. | 5190 | any special tokenization. Since each HELLO message contains a size field, |
5129 | 5191 | the response can easily be split into separate HELLO messages by the | |
5130 | A HOSTLIST client connecting to the HOSTLIST server will receive the hostlist | 5192 | client. |
5131 | as a HTTP response and the the server will terminate the connection with the | 5193 | |
5132 | result code HTTP 200 OK. The connection will be closed immediately if no | 5194 | A HOSTLIST client connecting to the HOSTLIST server will receive the |
5133 | hostlist is available. | 5195 | hostlist as a HTTP response and the the server will terminate the |
5196 | connection with the result code @code{HTTP 200 OK}. | ||
5197 | The connection will be closed immediately if no hostlist is available. | ||
5134 | 5198 | ||
5135 | @node Advertising the URL | 5199 | @node Advertising the URL |
5136 | @subsubsection Advertising the URL | 5200 | @subsubsection Advertising the URL |
5137 | 5201 | ||
5138 | @c %**end of header | 5202 | @c %**end of header |
5139 | 5203 | ||
5140 | The server also advertises the URL to download the hostlist to other peers if | 5204 | The server also advertises the URL to download the hostlist to other peers |
5141 | hostlist advertisement is enabled. When a new peer connects and has hostlist | 5205 | if hostlist advertisement is enabled. |
5142 | learning enabled, the server sends a GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT | 5206 | When a new peer connects and has hostlist learning enabled, the server |
5143 | message to this peer using the CORE service. | 5207 | sends a @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this |
5208 | peer using the CORE service. | ||
5144 | 5209 | ||
5145 | @node The HOSTLIST client | 5210 | @node The HOSTLIST client |
5146 | @subsection The HOSTLIST client | 5211 | @subsection The HOSTLIST client |
5147 | 5212 | ||
5148 | @c %**end of header | 5213 | @c %**end of header |
5149 | 5214 | ||
5150 | The client provides the functionality to download the list of HELLOs from a set | 5215 | The client provides the functionality to download the list of HELLOs from |
5151 | of URLs. It performs a standard HTTP request to the URLs configured and learned | 5216 | a set of URLs. |
5217 | It performs a standard HTTP request to the URLs configured and learned | ||
5152 | from advertisement messages received from other peers. When a HELLO is | 5218 | from advertisement messages received from other peers. When a HELLO is |
5153 | downloaded, the HOSTLIST client forwards the HELLO to the TRANSPORT service for | 5219 | downloaded, the HOSTLIST client forwards the HELLO to the TRANSPORT |
5154 | validation. | 5220 | service for validation. |
5155 | 5221 | ||
5156 | The client supports two modes of operation: download of HELLOs (bootstrapping) | 5222 | The client supports two modes of operation: |
5157 | and learning of URLs. | ||
5158 | 5223 | ||
5224 | @itemize @bullet | ||
5225 | @item download of HELLOs (bootstrapping) | ||
5226 | @item learning of URLs | ||
5227 | @end itemize | ||
5159 | 5228 | ||
5160 | @menu | 5229 | @menu |
5161 | * Bootstrapping:: | 5230 | * Bootstrapping:: |
@@ -5167,102 +5236,121 @@ and learning of URLs. | |||
5167 | 5236 | ||
5168 | @c %**end of header | 5237 | @c %**end of header |
5169 | 5238 | ||
5170 | For bootstrapping, it schedules a task to download the hostlist from the set of | 5239 | For bootstrapping, it schedules a task to download the hostlist from the |
5171 | known URLs. The downloads are only performed if the number of current | 5240 | set of known URLs. |
5172 | connections is smaller than a minimum number of connections (at the moment 4). | 5241 | The downloads are only performed if the number of current |
5242 | connections is smaller than a minimum number of connections | ||
5243 | (at the moment 4). | ||
5173 | The interval between downloads increases exponentially; however, the | 5244 | The interval between downloads increases exponentially; however, the |
5174 | exponential growth is limited if it becomes longer than an hour. At that point, | 5245 | exponential growth is limited if it becomes longer than an hour. |
5175 | the frequency growth is capped at (#number of connections * 1h). | 5246 | At that point, the frequency growth is capped at |
5247 | (#number of connections * 1h). | ||
5176 | 5248 | ||
5177 | Once the decision has been taken to download HELLOs, the daemon chooses a | 5249 | Once the decision has been taken to download HELLOs, the daemon chooses a |
5178 | random URL from the list of known URLs. URLs can be configured in the | 5250 | random URL from the list of known URLs. URLs can be configured in the |
5179 | configuration or be learned from advertisement messages. The client uses a HTTP | 5251 | configuration or be learned from advertisement messages. |
5180 | client library (libcurl) to initiate the download using the libcurl multi | 5252 | The client uses a HTTP client library (libcurl) to initiate the download |
5181 | interface. Libcurl passes the data to the callback_download function which | 5253 | using the libcurl multi interface. |
5182 | stores the data in a buffer if space is available and the maximum size for a | 5254 | Libcurl passes the data to the callback_download function which |
5183 | hostlist download is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When a | 5255 | stores the data in a buffer if space is available and the maximum size for |
5184 | full HELLO was downloaded, the HOSTLIST client offers this HELLO message to the | 5256 | a hostlist download is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). |
5185 | TRANSPORT service for validation. When the download is finished or failed, | 5257 | When a full HELLO was downloaded, the HOSTLIST client offers this |
5186 | statistical information about the quality of this URL is updated. | 5258 | HELLO message to the TRANSPORT service for validation. |
5187 | 5259 | When the download is finished or failed, statistical information about the | |
5260 | quality of this URL is updated. | ||
5261 | |||
5262 | @cindex HOSTLIST learning | ||
5188 | @node Learning | 5263 | @node Learning |
5189 | @subsubsection Learning | 5264 | @subsubsection Learning |
5190 | 5265 | ||
5191 | @c %**end of header | 5266 | @c %**end of header |
5192 | 5267 | ||
5193 | The client also manages hostlist advertisements from other peers. The HOSTLIST | 5268 | The client also manages hostlist advertisements from other peers. The |
5194 | daemon forwards GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT messages to the | 5269 | HOSTLIST daemon forwards @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} |
5195 | client subsystem, which extracts the URL from the message. Next, a test of the | 5270 | messages to the client subsystem, which extracts the URL from the message. |
5196 | newly obtained URL is performed by triggering a download from the new URL. If | 5271 | Next, a test of the newly obtained URL is performed by triggering a |
5197 | the URL works correctly, it is added to the list of working URLs. | 5272 | download from the new URL. If the URL works correctly, it is added to the |
5273 | list of working URLs. | ||
5198 | 5274 | ||
5199 | The size of the list of URLs is restricted, so if an additional server is added | 5275 | The size of the list of URLs is restricted, so if an additional server is |
5200 | and the list is full, the URL with the worst quality ranking (determined | 5276 | added and the list is full, the URL with the worst quality ranking |
5201 | through successful downloads and number of HELLOs e.g.) is discarded. During | 5277 | (determined through successful downloads and number of HELLOs e.g.) is |
5202 | shutdown the list of URLs is saved to a file for persistance and loaded on | 5278 | discarded. During shutdown the list of URLs is saved to a file for |
5203 | startup. URLs from the configuration file are never discarded. | 5279 | persistance and loaded on startup. URLs from the configuration file are |
5280 | never discarded. | ||
5204 | 5281 | ||
5205 | @node Usage | 5282 | @node Usage |
5206 | @subsection Usage | 5283 | @subsection Usage |
5207 | 5284 | ||
5208 | @c %**end of header | 5285 | @c %**end of header |
5209 | 5286 | ||
5210 | To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES section | 5287 | To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES |
5211 | for the ARM services. This is done in the default configuration. | 5288 | section for the ARM services. This is done in the default configuration. |
5212 | 5289 | ||
5213 | For more information on how to configure the HOSTLIST subsystem see the | 5290 | For more information on how to configure the HOSTLIST subsystem see the |
5214 | installation handbook:@ Configuring the hostlist to bootstrap@ Configuring your | 5291 | installation handbook:@ |
5215 | peer to provide a hostlist | 5292 | Configuring the hostlist to bootstrap@ |
5293 | Configuring your peer to provide a hostlist | ||
5216 | 5294 | ||
5295 | @cindex IDENTITY | ||
5296 | @cindex identity subsystem | ||
5217 | @node GNUnet's IDENTITY subsystem | 5297 | @node GNUnet's IDENTITY subsystem |
5218 | @section GNUnet's IDENTITY subsystem | 5298 | @section GNUnet's IDENTITY subsystem |
5219 | 5299 | ||
5220 | @c %**end of header | 5300 | @c %**end of header |
5221 | 5301 | ||
5222 | Identities of "users" in GNUnet are called egos. Egos can be used as pseudonyms | 5302 | Identities of "users" in GNUnet are called egos. |
5223 | (fake names) or be tied to an organization (for example, GNU) or even the | 5303 | Egos can be used as pseudonyms ("fake names") or be tied to an |
5224 | actual identity of a human. GNUnet users are expected to have many egos. They | 5304 | organization (for example, "GNU") or even the actual identity of a human. |
5225 | might have one tied to their real identity, some for organizations they manage, | 5305 | GNUnet users are expected to have many egos. They might have one tied to |
5226 | and more for different domains where they want to operate under a pseudonym. | 5306 | their real identity, some for organizations they manage, and more for |
5227 | 5307 | different domains where they want to operate under a pseudonym. | |
5228 | The IDENTITY service allows users to manage their egos. The identity service | 5308 | |
5229 | manages the private keys egos of the local user; it does not manage identities | 5309 | The IDENTITY service allows users to manage their egos. The identity |
5230 | of other users (public keys). Public keys for other users need names to become | 5310 | service manages the private keys egos of the local user; it does not |
5231 | manageable. GNUnet uses the GNU Name System (GNS) to give names to other users | 5311 | manage identities of other users (public keys). Public keys for other |
5232 | and manage their public keys securely. This chapter is about the IDENTITY | 5312 | users need names to become manageable. GNUnet uses the |
5233 | service, which is about the management of private keys. | 5313 | @dfn{GNU Name System} (GNS) to give names to other users and manage their |
5234 | 5314 | public keys securely. This chapter is about the IDENTITY service, | |
5235 | On the network, an ego corresponds to an ECDSA key (over Curve25519, using RFC | 5315 | which is about the management of private keys. |
5236 | 6979, as required by GNS). Thus, users can perform actions under a particular | 5316 | |
5237 | ego by using (signing with) a particular private key. Other users can then | 5317 | On the network, an ego corresponds to an ECDSA key (over Curve25519, |
5238 | confirm that the action was really performed by that ego by checking the | 5318 | using RFC 6979, as required by GNS). Thus, users can perform actions |
5239 | signature against the respective public key. | 5319 | under a particular ego by using (signing with) a particular private key. |
5240 | 5320 | Other users can then confirm that the action was really performed by that | |
5241 | The IDENTITY service allows users to associate a human-readable name with each | 5321 | ego by checking the signature against the respective public key. |
5242 | ego. This way, users can use names that will remind them of the purpose of a | 5322 | |
5243 | particular ego. The IDENTITY service will store the respective private keys and | 5323 | The IDENTITY service allows users to associate a human-readable name with |
5244 | allows applications to access key information by name. Users can change the | 5324 | each ego. This way, users can use names that will remind them of the |
5245 | name that is locally (!) associated with an ego. Egos can also be deleted, | 5325 | purpose of a particular ego. |
5246 | which means that the private key will be removed and it thus will not be | 5326 | The IDENTITY service will store the respective private keys and |
5247 | possible to perform actions with that ego in the future. | 5327 | allows applications to access key information by name. |
5248 | 5328 | Users can change the name that is locally (!) associated with an ego. | |
5249 | Additionally, the IDENTITY subsystem can associate service functions with egos. | 5329 | Egos can also be deleted, which means that the private key will be removed |
5250 | For example, GNS requires the ego that should be used for the shorten zone. GNS | 5330 | and it thus will not be possible to perform actions with that ego in the |
5251 | will ask IDENTITY for an ego for the "gns-short" service. The IDENTITY service | 5331 | future. |
5252 | has a mapping of such service strings to the name of the ego that the user | 5332 | |
5253 | wants to use for this service, for example "my-short-zone-ego". | 5333 | Additionally, the IDENTITY subsystem can associate service functions with |
5254 | 5334 | egos. | |
5255 | Finally, the IDENTITY API provides access to a special ego, the anonymous ego. | 5335 | For example, GNS requires the ego that should be used for the shorten |
5256 | The anonymous ego is special in that its private key is not really private, but | 5336 | zone. GNS will ask IDENTITY for an ego for the "gns-short" service. |
5257 | fixed and known to everyone. Thus, anyone can perform actions as anonymous. | 5337 | The IDENTITY service has a mapping of such service strings to the name of |
5258 | This can be useful as with this trick, code does not have to contain a special | 5338 | the ego that the user wants to use for this service, for example |
5259 | case to distinguish between anonymous and pseudonymous egos. | 5339 | "my-short-zone-ego". |
5340 | |||
5341 | Finally, the IDENTITY API provides access to a special ego, the | ||
5342 | anonymous ego. The anonymous ego is special in that its private key is not | ||
5343 | really private, but fixed and known to everyone. | ||
5344 | Thus, anyone can perform actions as anonymous. This can be useful as with | ||
5345 | this trick, code does not have to contain a special case to distinguish | ||
5346 | between anonymous and pseudonymous egos. | ||
5260 | 5347 | ||
5261 | @menu | 5348 | @menu |
5262 | * libgnunetidentity:: | 5349 | * libgnunetidentity:: |
5263 | * The IDENTITY Client-Service Protocol:: | 5350 | * The IDENTITY Client-Service Protocol:: |
5264 | @end menu | 5351 | @end menu |
5265 | 5352 | ||
5353 | @cindex libgnunetidentity | ||
5266 | @node libgnunetidentity | 5354 | @node libgnunetidentity |
5267 | @subsection libgnunetidentity | 5355 | @subsection libgnunetidentity |
5268 | @c %**end of header | 5356 | @c %**end of header |
@@ -5282,55 +5370,58 @@ case to distinguish between anonymous and pseudonymous egos. | |||
5282 | @c %**end of header | 5370 | @c %**end of header |
5283 | 5371 | ||
5284 | First, typical clients connect to the identity service using | 5372 | First, typical clients connect to the identity service using |
5285 | @code{GNUNET_IDENTITY_connect}. This function takes a callback as a parameter. | 5373 | @code{GNUNET_IDENTITY_connect}. This function takes a callback as a |
5286 | If the given callback parameter is non-null, it will be invoked to notify the | 5374 | parameter. |
5287 | application about the current state of the identities in the system. | 5375 | If the given callback parameter is non-null, it will be invoked to notify |
5376 | the application about the current state of the identities in the system. | ||
5288 | 5377 | ||
5289 | @itemize @bullet | 5378 | @itemize @bullet |
5290 | @item First, it will be invoked on all known egos at the time of the | 5379 | @item First, it will be invoked on all known egos at the time of the |
5291 | connection. For each ego, a handle to the ego and the user's name for the ego | 5380 | connection. For each ego, a handle to the ego and the user's name for the |
5292 | will be passed to the callback. Furthermore, a @code{void **} context argument | 5381 | ego will be passed to the callback. Furthermore, a @code{void **} context |
5293 | will be provided which gives the client the opportunity to associate some state | 5382 | argument will be provided which gives the client the opportunity to |
5294 | with the ego. | 5383 | associate some state with the ego. |
5295 | @item Second, the callback will be invoked with NULL for the ego, the name and | 5384 | @item Second, the callback will be invoked with NULL for the ego, the name |
5296 | the context. This signals that the (initial) iteration over all egos has | 5385 | and the context. This signals that the (initial) iteration over all egos |
5297 | completed. | 5386 | has completed. |
5298 | @item Then, the callback will be invoked whenever something changes about an | 5387 | @item Then, the callback will be invoked whenever something changes about |
5299 | ego. If an ego is renamed, the callback is invoked with the ego handle of the | 5388 | an ego. |
5300 | ego that was renamed, and the new name. If an ego is deleted, the callback is | 5389 | If an ego is renamed, the callback is invoked with the ego handle of the |
5301 | invoked with the ego handle and a name of NULL. In the deletion case, the | 5390 | ego that was renamed, and the new name. If an ego is deleted, the callback |
5302 | application should also release resources stored in the context. | 5391 | is invoked with the ego handle and a name of NULL. In the deletion case, |
5392 | the application should also release resources stored in the context. | ||
5303 | @item When the application destroys the connection to the identity service | 5393 | @item When the application destroys the connection to the identity service |
5304 | using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked with the | 5394 | using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked |
5305 | ego and a name of NULL (equivalent to deletion of the egos). This should again | 5395 | with the ego and a name of NULL (equivalent to deletion of the egos). |
5306 | be used to clean up the per-ego context. | 5396 | This should again be used to clean up the per-ego context. |
5307 | @end itemize | 5397 | @end itemize |
5308 | 5398 | ||
5309 | The ego handle passed to the callback remains valid until the callback is | 5399 | The ego handle passed to the callback remains valid until the callback is |
5310 | invoked with a name of NULL, so it is safe to store a reference to the ego's | 5400 | invoked with a name of NULL, so it is safe to store a reference to the |
5311 | handle. | 5401 | ego's handle. |
5312 | 5402 | ||
5313 | @node Operations on Egos | 5403 | @node Operations on Egos |
5314 | @subsubsection Operations on Egos | 5404 | @subsubsection Operations on Egos |
5315 | 5405 | ||
5316 | @c %**end of header | 5406 | @c %**end of header |
5317 | 5407 | ||
5318 | Given an ego handle, the main operations are to get its associated private key | 5408 | Given an ego handle, the main operations are to get its associated private |
5319 | using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated public key | 5409 | key using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated |
5320 | using @code{GNUNET_IDENTITY_ego_get_public_key}. | 5410 | public key using @code{GNUNET_IDENTITY_ego_get_public_key}. |
5321 | 5411 | ||
5322 | The other operations on egos are pretty straightforward. Using | 5412 | The other operations on egos are pretty straightforward. |
5323 | @code{GNUNET_IDENTITY_create}, an application can request the creation of an | 5413 | Using @code{GNUNET_IDENTITY_create}, an application can request the |
5324 | ego by specifying the desired name. The operation will fail if that name is | 5414 | creation of an ego by specifying the desired name. |
5325 | already in use. Using @code{GNUNET_IDENTITY_rename} the name of an existing ego | 5415 | The operation will fail if that name is |
5326 | can be changed. Finally, egos can be deleted using | 5416 | already in use. Using @code{GNUNET_IDENTITY_rename} the name of an |
5327 | @code{GNUNET_IDENTITY_delete}. All of these operations will trigger updates to | 5417 | existing ego can be changed. Finally, egos can be deleted using |
5328 | the callback given to the @code{GNUNET_IDENTITY_connect} function of all | 5418 | @code{GNUNET_IDENTITY_delete}. All of these operations will trigger |
5329 | applications that are connected with the identity service at the time. | 5419 | updates to the callback given to the @code{GNUNET_IDENTITY_connect} |
5330 | @code{GNUNET_IDENTITY_cancel} can be used to cancel the operations before the | 5420 | function of all applications that are connected with the identity service |
5331 | respective continuations would be called. It is not guaranteed that the | 5421 | at the time. @code{GNUNET_IDENTITY_cancel} can be used to cancel the |
5332 | operation will not be completed anyway, only the continuation will no longer be | 5422 | operations before the respective continuations would be called. |
5333 | called. | 5423 | It is not guaranteed that the operation will not be completed anyway, |
5424 | only the continuation will no longer be called. | ||
5334 | 5425 | ||
5335 | @node The anonymous Ego | 5426 | @node The anonymous Ego |
5336 | @subsubsection The anonymous Ego | 5427 | @subsubsection The anonymous Ego |
@@ -5339,11 +5430,11 @@ called. | |||
5339 | 5430 | ||
5340 | A special way to obtain an ego handle is to call | 5431 | A special way to obtain an ego handle is to call |
5341 | @code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the | 5432 | @code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the |
5342 | "anonymous" user --- anyone knows and can get the private key for this user, so | 5433 | "anonymous" user --- anyone knows and can get the private key for this |
5343 | it is suitable for operations that are supposed to be anonymous but require | 5434 | user, so it is suitable for operations that are supposed to be anonymous |
5344 | signatures (for example, to avoid a special path in the code). The anonymous | 5435 | but require signatures (for example, to avoid a special path in the code). |
5345 | ego is always valid and accessing it does not require a connection to the | 5436 | The anonymous ego is always valid and accessing it does not require a |
5346 | identity service. | 5437 | connection to the identity service. |
5347 | 5438 | ||
5348 | @node Convenience API to lookup a single ego | 5439 | @node Convenience API to lookup a single ego |
5349 | @subsubsection Convenience API to lookup a single ego | 5440 | @subsubsection Convenience API to lookup a single ego |
@@ -5351,98 +5442,106 @@ identity service. | |||
5351 | 5442 | ||
5352 | As applications commonly simply have to lookup a single ego, there is a | 5443 | As applications commonly simply have to lookup a single ego, there is a |
5353 | convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to | 5444 | convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to |
5354 | lookup a single ego by name. Note that this is the user's name for the ego, not | 5445 | lookup a single ego by name. Note that this is the user's name for the |
5355 | the service function. The resulting ego will be returned via a callback and | 5446 | ego, not the service function. The resulting ego will be returned via a |
5356 | will only be valid during that callback. The operation can be cancelled via | 5447 | callback and will only be valid during that callback. The operation can |
5357 | @code{GNUNET_IDENTITY_ego_lookup_cancel} (cancellation is only legal before the | 5448 | be cancelled via @code{GNUNET_IDENTITY_ego_lookup_cancel} |
5358 | callback is invoked). | 5449 | (cancellation is only legal before the callback is invoked). |
5359 | 5450 | ||
5360 | @node Associating egos with service functions | 5451 | @node Associating egos with service functions |
5361 | @subsubsection Associating egos with service functions | 5452 | @subsubsection Associating egos with service functions |
5362 | 5453 | ||
5363 | 5454 | ||
5364 | The @code{GNUNET_IDENTITY_set} function is used to associate a particular ego | 5455 | The @code{GNUNET_IDENTITY_set} function is used to associate a particular |
5365 | with a service function. The name used by the service and the ego are given as | 5456 | ego with a service function. The name used by the service and the ego are |
5366 | arguments. Afterwards, the service can use its name to lookup the associated | 5457 | given as arguments. |
5367 | ego using @code{GNUNET_IDENTITY_get}. | 5458 | Afterwards, the service can use its name to lookup the associated ego |
5459 | using @code{GNUNET_IDENTITY_get}. | ||
5368 | 5460 | ||
5369 | @node The IDENTITY Client-Service Protocol | 5461 | @node The IDENTITY Client-Service Protocol |
5370 | @subsection The IDENTITY Client-Service Protocol | 5462 | @subsection The IDENTITY Client-Service Protocol |
5371 | 5463 | ||
5372 | @c %**end of header | 5464 | @c %**end of header |
5373 | 5465 | ||
5374 | A client connecting to the identity service first sends a message with type | 5466 | A client connecting to the identity service first sends a message with |
5467 | type | ||
5375 | @code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that, the | 5468 | @code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that, the |
5376 | client will receive information about changes to the egos by receiving messages | 5469 | client will receive information about changes to the egos by receiving |
5377 | of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}. Those messages contain the | 5470 | messages of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}. |
5378 | private key of the ego and the user's name of the ego (or zero bytes for the | 5471 | Those messages contain the private key of the ego and the user's name of |
5379 | name to indicate that the ego was deleted). A special bit @code{end_of_list} is | 5472 | the ego (or zero bytes for the name to indicate that the ego was deleted). |
5380 | used to indicate the end of the initial iteration over the identity service's | 5473 | A special bit @code{end_of_list} is used to indicate the end of the |
5381 | egos. | 5474 | initial iteration over the identity service's egos. |
5382 | 5475 | ||
5383 | The client can trigger changes to the egos by sending CREATE, RENAME or DELETE | 5476 | The client can trigger changes to the egos by sending @code{CREATE}, |
5384 | messages. The CREATE message contains the private key and the desired name. The | 5477 | @code{RENAME} or @code{DELETE} messages. |
5385 | RENAME message contains the old name and the new name. The DELETE message only | 5478 | The CREATE message contains the private key and the desired name.@ |
5386 | needs to include the name of the ego to delete. The service responds to each of | 5479 | The RENAME message contains the old name and the new name.@ |
5387 | these messages with a RESULT_CODE message which indicates success or error of | 5480 | The DELETE message only needs to include the name of the ego to delete.@ |
5388 | the operation, and possibly a human-readable error message. | 5481 | The service responds to each of these messages with a @code{RESULT_CODE} |
5482 | message which indicates success or error of the operation, and possibly | ||
5483 | a human-readable error message. | ||
5389 | 5484 | ||
5390 | Finally, the client can bind the name of a service function to an ego by | 5485 | Finally, the client can bind the name of a service function to an ego by |
5391 | sending a SET_DEFAULT message with the name of the service function and the | 5486 | sending a @code{SET_DEFAULT} message with the name of the service function |
5392 | private key of the ego. Such bindings can then be resolved using a GET_DEFAULT | 5487 | and the private key of the ego. |
5393 | message, which includes the name of the service function. The identity service | 5488 | Such bindings can then be resolved using a @code{GET_DEFAULT} message, |
5394 | will respond to a GET_DEFAULT request with a SET_DEFAULT message containing the | 5489 | which includes the name of the service function. The identity service |
5395 | respective information, or with a RESULT_CODE to indicate an error. | 5490 | will respond to a GET_DEFAULT request with a SET_DEFAULT message |
5396 | 5491 | containing the respective information, or with a RESULT_CODE to | |
5492 | indicate an error. | ||
5493 | |||
5494 | @cindex NAMESTORE | ||
5495 | @cindex namestore subsystem | ||
5397 | @node GNUnet's NAMESTORE Subsystem | 5496 | @node GNUnet's NAMESTORE Subsystem |
5398 | @section GNUnet's NAMESTORE Subsystem | 5497 | @section GNUnet's NAMESTORE Subsystem |
5399 | 5498 | ||
5400 | @c %**end of header | ||
5401 | |||
5402 | The NAMESTORE subsystem provides persistent storage for local GNS zone | 5499 | The NAMESTORE subsystem provides persistent storage for local GNS zone |
5403 | information. All local GNS zone information are managed by NAMESTORE. It | 5500 | information. All local GNS zone information are managed by NAMESTORE. It |
5404 | provides both the functionality to administer local GNS information (e.g. | 5501 | provides both the functionality to administer local GNS information (e.g. |
5405 | delete and add records) as well as to retrieve GNS information (e.g to list | 5502 | delete and add records) as well as to retrieve GNS information (e.g to |
5406 | name information in a client). NAMESTORE does only manage the persistent | 5503 | list name information in a client). |
5407 | storage of zone information belonging to the user running the service: GNS | 5504 | NAMESTORE does only manage the persistent storage of zone information |
5408 | information from other users obtained from the DHT are stored by the NAMECACHE | 5505 | belonging to the user running the service: GNS information from other |
5409 | subsystem. | 5506 | users obtained from the DHT are stored by the NAMECACHE subsystem. |
5410 | 5507 | ||
5411 | NAMESTORE uses a plugin-based database backend to store GNS information with | 5508 | NAMESTORE uses a plugin-based database backend to store GNS information |
5412 | good performance. Here sqlite, MySQL and PostgreSQL are supported database | 5509 | with good performance. Here sqlite, MySQL and PostgreSQL are supported |
5413 | backends. NAMESTORE clients interact with the IDENTITY subsystem to obtain | 5510 | database backends. |
5511 | NAMESTORE clients interact with the IDENTITY subsystem to obtain | ||
5414 | cryptographic information about zones based on egos as described with the | 5512 | cryptographic information about zones based on egos as described with the |
5415 | IDENTITY subsystem., but internally NAMESTORE refers to zones using the ECDSA | 5513 | IDENTITY subsystem, but internally NAMESTORE refers to zones using the |
5416 | private key. In addition, it collaborates with the NAMECACHE subsystem and | 5514 | ECDSA private key. |
5417 | stores zone information when local information are modified in the GNS cache to | 5515 | In addition, it collaborates with the NAMECACHE subsystem and |
5418 | increase look-up performance for local information. | 5516 | stores zone information when local information are modified in the |
5517 | GNS cache to increase look-up performance for local information. | ||
5419 | 5518 | ||
5420 | NAMESTORE provides functionality to look-up and store records, to iterate over | 5519 | NAMESTORE provides functionality to look-up and store records, to iterate |
5421 | a specific or all zones and to monitor zones for changes. NAMESTORE | 5520 | over a specific or all zones and to monitor zones for changes. NAMESTORE |
5422 | functionality can be accessed using the NAMESTORE api or the NAMESTORE command | 5521 | functionality can be accessed using the NAMESTORE api or the NAMESTORE |
5423 | line tool. | 5522 | command line tool. |
5424 | 5523 | ||
5425 | @menu | 5524 | @menu |
5426 | * libgnunetnamestore:: | 5525 | * libgnunetnamestore:: |
5427 | @end menu | 5526 | @end menu |
5428 | 5527 | ||
5528 | @cindex libgnunetnamestore | ||
5429 | @node libgnunetnamestore | 5529 | @node libgnunetnamestore |
5430 | @subsection libgnunetnamestore | 5530 | @subsection libgnunetnamestore |
5431 | 5531 | ||
5432 | @c %**end of header | 5532 | To interact with NAMESTORE clients first connect to the NAMESTORE service |
5533 | using the @code{GNUNET_NAMESTORE_connect} passing a configuration handle. | ||
5534 | As a result they obtain a NAMESTORE handle, they can use for operations, | ||
5535 | or NULL is returned if the connection failed. | ||
5433 | 5536 | ||
5434 | To interact with NAMESTORE clients first connect to the NAMESTORE service using | 5537 | To disconnect from NAMESTORE, clients use |
5435 | the @code{GNUNET_NAMESTORE_connect} passing a configuration handle. As a result | 5538 | @code{GNUNET_NAMESTORE_disconnect} and specify the handle to disconnect. |
5436 | they obtain a NAMESTORE handle, they can use for operations, or NULL is | ||
5437 | returned if the connection failed. | ||
5438 | |||
5439 | To disconnect from NAMESTORE, clients use @code{GNUNET_NAMESTORE_disconnect} | ||
5440 | and specify the handle to disconnect. | ||
5441 | 5539 | ||
5442 | NAMESTORE internally uses the ECDSA private key to refer to zones. These | 5540 | NAMESTORE internally uses the ECDSA private key to refer to zones. These |
5443 | private keys can be obtained from the IDENTITY subsytem. Here @emph{egos@emph{ | 5541 | private keys can be obtained from the IDENTITY subsytem. |
5444 | can be used to refer to zones or the default ego assigned to the GNS subsystem | 5542 | Here @emph{egos} @emph{can be used to refer to zones or the default ego |
5445 | can be used to obtained the master zone's private key.}} | 5543 | assigned to the GNS subsystem can be used to obtained the master zone's |
5544 | private key.} | ||
5446 | 5545 | ||
5447 | 5546 | ||
5448 | @menu | 5547 | @menu |
@@ -5456,92 +5555,104 @@ can be used to obtained the master zone's private key.}} | |||
5456 | 5555 | ||
5457 | @c %**end of header | 5556 | @c %**end of header |
5458 | 5557 | ||
5459 | NAMESTORE provides functions to lookup records stored under a label in a zone | 5558 | NAMESTORE provides functions to lookup records stored under a label in a |
5460 | and to store records under a label in a zone. | 5559 | zone and to store records under a label in a zone. |
5461 | 5560 | ||
5462 | To store (and delete) records, the client uses the | 5561 | To store (and delete) records, the client uses the |
5463 | @code{GNUNET_NAMESTORE_records_store} function and has to provide namestore | 5562 | @code{GNUNET_NAMESTORE_records_store} function and has to provide |
5464 | handle to use, the private key of the zone, the label to store the records | 5563 | namestore handle to use, the private key of the zone, the label to store |
5465 | under, the records and number of records plus an callback function. After the | 5564 | the records under, the records and number of records plus an callback |
5466 | operation is performed NAMESTORE will call the provided callback function with | 5565 | function. |
5467 | the result GNUNET_SYSERR on failure (including timeout/queue drop/failure to | 5566 | After the operation is performed NAMESTORE will call the provided |
5468 | validate), GNUNET_NO if content was already there or not found GNUNET_YES (or | 5567 | callback function with the result GNUNET_SYSERR on failure |
5469 | other positive value) on success plus an additional error message. | 5568 | (including timeout/queue drop/failure to validate), GNUNET_NO if content |
5470 | 5569 | was already there or not found GNUNET_YES (or other positive value) on | |
5471 | Records are deleted by using the store command with 0 records to store. It is | 5570 | success plus an additional error message. |
5472 | important to note, that records are not merged when records exist with the | 5571 | |
5473 | label. So a client has first to retrieve records, merge with existing records | 5572 | Records are deleted by using the store command with 0 records to store. |
5573 | It is important to note, that records are not merged when records exist | ||
5574 | with the label. | ||
5575 | So a client has first to retrieve records, merge with existing records | ||
5474 | and then store the result. | 5576 | and then store the result. |
5475 | 5577 | ||
5476 | To perform a lookup operation, the client uses the | 5578 | To perform a lookup operation, the client uses the |
5477 | @code{GNUNET_NAMESTORE_records_store} function. Here he has to pass the | 5579 | @code{GNUNET_NAMESTORE_records_store} function. Here he has to pass the |
5478 | namestore handle, the private key of the zone and the label. He also has to | 5580 | namestore handle, the private key of the zone and the label. He also has |
5479 | provide a callback function which will be called with the result of the lookup | 5581 | to provide a callback function which will be called with the result of |
5480 | operation: the zone for the records, the label, and the records including the | 5582 | the lookup operation: |
5583 | the zone for the records, the label, and the records including the | ||
5481 | number of records included. | 5584 | number of records included. |
5482 | 5585 | ||
5483 | A special operation is used to set the preferred nickname for a zone. This | 5586 | A special operation is used to set the preferred nickname for a zone. |
5484 | nickname is stored with the zone and is automatically merged with all labels | 5587 | This nickname is stored with the zone and is automatically merged with |
5485 | and records stored in a zone. Here the client uses the | 5588 | all labels and records stored in a zone. Here the client uses the |
5486 | @code{GNUNET_NAMESTORE_set_nick} function and passes the private key of the | 5589 | @code{GNUNET_NAMESTORE_set_nick} function and passes the private key of |
5487 | zone, the nickname as string plus a the callback with the result of the | 5590 | the zone, the nickname as string plus a the callback with the result of |
5488 | operation. | 5591 | the operation. |
5489 | 5592 | ||
5490 | @node Iterating Zone Information | 5593 | @node Iterating Zone Information |
5491 | @subsubsection Iterating Zone Information | 5594 | @subsubsection Iterating Zone Information |
5492 | 5595 | ||
5493 | @c %**end of header | 5596 | @c %**end of header |
5494 | 5597 | ||
5495 | A client can iterate over all information in a zone or all zones managed by | 5598 | A client can iterate over all information in a zone or all zones managed |
5496 | NAMESTORE. Here a client uses the @code{GNUNET_NAMESTORE_zone_iteration_start} | 5599 | by NAMESTORE. |
5600 | Here a client uses the @code{GNUNET_NAMESTORE_zone_iteration_start} | ||
5497 | function and passes the namestore handle, the zone to iterate over and a | 5601 | function and passes the namestore handle, the zone to iterate over and a |
5498 | callback function to call with the result. If the client wants to iterate over | 5602 | callback function to call with the result. |
5499 | all the, he passes NULL for the zone. A @code{GNUNET_NAMESTORE_ZoneIterator} | 5603 | If the client wants to iterate over all the, he passes NULL for the zone. |
5500 | handle is returned to be used to continue iteration. | 5604 | A @code{GNUNET_NAMESTORE_ZoneIterator} handle is returned to be used to |
5605 | continue iteration. | ||
5501 | 5606 | ||
5502 | NAMESTORE calls the callback for every result and expects the client to call@ | 5607 | NAMESTORE calls the callback for every result and expects the client to |
5503 | @code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or | 5608 | call @code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or |
5504 | @code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration. When | 5609 | @code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration. |
5505 | NAMESTORE reached the last item it will call the callback with a NULL value to | 5610 | When NAMESTORE reached the last item it will call the callback with a |
5506 | indicate. | 5611 | NULL value to indicate. |
5507 | 5612 | ||
5508 | @node Monitoring Zone Information | 5613 | @node Monitoring Zone Information |
5509 | @subsubsection Monitoring Zone Information | 5614 | @subsubsection Monitoring Zone Information |
5510 | 5615 | ||
5511 | @c %**end of header | 5616 | @c %**end of header |
5512 | 5617 | ||
5513 | Clients can also monitor zones to be notified about changes. Here the clients | 5618 | Clients can also monitor zones to be notified about changes. Here the |
5514 | uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and passes the | 5619 | clients uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and |
5515 | private key of the zone and and a callback function to call with updates for a | 5620 | passes the private key of the zone and and a callback function to call |
5516 | zone. The client can specify to obtain zone information first by iterating over | 5621 | with updates for a zone. |
5517 | the zone and specify a synchronization callback to be called when the client | 5622 | The client can specify to obtain zone information first by iterating over |
5518 | and the namestore are synced. | 5623 | the zone and specify a synchronization callback to be called when the |
5624 | client and the namestore are synced. | ||
5519 | 5625 | ||
5520 | On an update, NAMESTORE will call the callback with the private key of the | 5626 | On an update, NAMESTORE will call the callback with the private key of the |
5521 | zone, the label and the records and their number. | 5627 | zone, the label and the records and their number. |
5522 | 5628 | ||
5523 | To stop monitoring, the client call @code{GNUNET_NAMESTORE_zone_monitor_stop} | 5629 | To stop monitoring, the client calls |
5524 | and passes the handle obtained from the function to start the monitoring. | 5630 | @code{GNUNET_NAMESTORE_zone_monitor_stop} and passes the handle obtained |
5631 | from the function to start the monitoring. | ||
5525 | 5632 | ||
5633 | @cindex PEERINFO | ||
5634 | @cindex peerinfo subsystem | ||
5526 | @node GNUnet's PEERINFO subsystem | 5635 | @node GNUnet's PEERINFO subsystem |
5527 | @section GNUnet's PEERINFO subsystem | 5636 | @section GNUnet's PEERINFO subsystem |
5528 | 5637 | ||
5529 | @c %**end of header | 5638 | @c %**end of header |
5530 | 5639 | ||
5531 | The PEERINFO subsystem is used to store verified (validated) information about | 5640 | The PEERINFO subsystem is used to store verified (validated) information |
5532 | known peers in a persistent way. It obtains these addresses for example from | 5641 | about known peers in a persistent way. It obtains these addresses for |
5533 | TRANSPORT service which is in charge of address validation. Validation means | 5642 | example from TRANSPORT service which is in charge of address validation. |
5534 | that the information in the HELLO message are checked by connecting to the | 5643 | Validation means that the information in the HELLO message are checked by |
5535 | addresses and performing a cryptographic handshake to authenticate the peer | 5644 | connecting to the addresses and performing a cryptographic handshake to |
5536 | instance stating to be reachable with these addresses. Peerinfo does not | 5645 | authenticate the peer instance stating to be reachable with these |
5537 | validate the HELLO messages itself but only stores them and gives them to | 5646 | addresses. |
5538 | interested clients. | 5647 | Peerinfo does not validate the HELLO messages itself but only stores them |
5648 | and gives them to interested clients. | ||
5539 | 5649 | ||
5540 | As future work, we think about moving from storing just HELLO messages to | 5650 | As future work, we think about moving from storing just HELLO messages to |
5541 | providing a generic persistent per-peer information store. More and more | 5651 | providing a generic persistent per-peer information store. |
5542 | subsystems tend to need to store per-peer information in persistent way. To not | 5652 | More and more subsystems tend to need to store per-peer information in |
5543 | duplicate this functionality we plan to provide a PEERSTORE service providing | 5653 | persistent way. |
5544 | this functionality | 5654 | To not duplicate this functionality we plan to provide a PEERSTORE |
5655 | service providing this functionality. | ||
5545 | 5656 | ||
5546 | @menu | 5657 | @menu |
5547 | * Features2:: | 5658 | * Features2:: |
@@ -7636,37 +7747,47 @@ The names for the master directories follow the names of the operations: | |||
7636 | @end itemize | 7747 | @end itemize |
7637 | 7748 | ||
7638 | Each of the master directories contains names (chosen at random) for each | 7749 | Each of the master directories contains names (chosen at random) for each |
7639 | active top-level (master) operation. Note that a download that is associated | 7750 | active top-level (master) operation. |
7640 | with a search result is not a top-level operation. | 7751 | Note that a download that is associated with a search result is not a |
7641 | 7752 | top-level operation. | |
7642 | In contrast to the master directories, the child directories are only consulted | 7753 | |
7643 | when another operation refers to them. For each search, a subdirectory (named | 7754 | In contrast to the master directories, the child directories are only |
7644 | after the master search synchronization file) contains the search results. | 7755 | consulted when another operation refers to them. |
7645 | Search results can have an associated download, which is then stored in the | 7756 | For each search, a subdirectory (named after the master search |
7646 | general "download-child" directory. Downloads can be recursive, in which case | 7757 | synchronization file) contains the search results. |
7647 | children are stored in subdirectories mirroring the structure of the recursive | 7758 | Search results can have an associated download, which is then stored in |
7648 | download (either starting in the master "download" directory or in the | 7759 | the general "download-child" directory. |
7649 | "download-child" directory depending on how the download was initiated). For | 7760 | Downloads can be recursive, in which case children are stored in |
7650 | publishing operations, the "publish-file" directory contains information about | 7761 | subdirectories mirroring the structure of the recursive download |
7651 | the individual files and directories that are part of the publication. However, | 7762 | (either starting in the master "download" directory or in the |
7652 | this directory structure is flat and does not mirror the structure of the | 7763 | "download-child" directory depending on how the download was initiated). |
7653 | publishing operation. Note that unindex operations cannot have associated child | 7764 | For publishing operations, the "publish-file" directory contains |
7654 | operations. | 7765 | information about the individual files and directories that are part of |
7655 | 7766 | the publication. | |
7767 | However, this directory structure is flat and does not mirror the | ||
7768 | structure of the publishing operation. | ||
7769 | Note that unindex operations cannot have associated child operations. | ||
7770 | |||
7771 | @cindex REGEX subsystem | ||
7772 | @cindex regex subsystem | ||
7656 | @node GNUnet's REGEX Subsystem | 7773 | @node GNUnet's REGEX Subsystem |
7657 | @section GNUnet's REGEX Subsystem | 7774 | @section GNUnet's REGEX Subsystem |
7658 | 7775 | ||
7659 | @c %**end of header | 7776 | @c %**end of header |
7660 | 7777 | ||
7661 | Using the REGEX subsystem, you can discover peers that offer a particular | 7778 | Using the REGEX subsystem, you can discover peers that offer a particular |
7662 | service using regular expressions. The peers that offer a service specify it | 7779 | service using regular expressions. |
7663 | using a regular expressions. Peers that want to patronize a service search | 7780 | The peers that offer a service specify it using a regular expressions. |
7664 | using a string. The REGEX subsystem will then use the DHT to return a set of | 7781 | Peers that want to patronize a service search using a string. |
7665 | matching offerers to the patrons. | 7782 | The REGEX subsystem will then use the DHT to return a set of matching |
7783 | offerers to the patrons. | ||
7784 | |||
7785 | For the technical details, we have Max's defense talk and Max's Master's | ||
7786 | thesis. | ||
7666 | 7787 | ||
7667 | For the technical details, we have "Max's defense talk and Max's Master's | 7788 | @c An additional publication is under preparation and available to |
7668 | thesis. An additional publication is under preparation and available to team | 7789 | @c team members (in Git). |
7669 | members (in Git). | 7790 | @c FIXME: Where is the file? Point to it. Assuming that it's szengel2012ms |
7670 | 7791 | ||
7671 | @menu | 7792 | @menu |
7672 | * How to run the regex profiler:: | 7793 | * How to run the regex profiler:: |
@@ -7677,32 +7798,38 @@ members (in Git). | |||
7677 | 7798 | ||
7678 | @c %**end of header | 7799 | @c %**end of header |
7679 | 7800 | ||
7680 | The gnunet-regex-profiler can be used to profile the usage of mesh/regex for a | 7801 | The gnunet-regex-profiler can be used to profile the usage of mesh/regex |
7681 | given set of regular expressions and strings. Mesh/regex allows you to announce | 7802 | for a given set of regular expressions and strings. |
7682 | your peer ID under a certain regex and search for peers matching a particular | 7803 | Mesh/regex allows you to announce your peer ID under a certain regex and |
7683 | regex using a string. See https://gnunet.org/szengel2012ms for a full | 7804 | search for peers matching a particular regex using a string. |
7805 | See @uref{https://gnunet.org/szengel2012ms, szengel2012ms} for a full | ||
7684 | introduction. | 7806 | introduction. |
7685 | 7807 | ||
7686 | First of all, the regex profiler uses GNUnet testbed, thus all the implications | 7808 | First of all, the regex profiler uses GNUnet testbed, thus all the |
7687 | for testbed also apply to the regex profiler (for example you need | 7809 | implications for testbed also apply to the regex profiler |
7688 | password-less ssh login to the machines listed in your hosts file). | 7810 | (for example you need password-less ssh login to the machines listed in |
7811 | your hosts file). | ||
7689 | 7812 | ||
7690 | @strong{Configuration} | 7813 | @strong{Configuration} |
7691 | 7814 | ||
7692 | Moreover, an appropriate configuration file is needed. Generally you can refer | 7815 | Moreover, an appropriate configuration file is needed. |
7693 | to SVN HEAD: contrib/regex_profiler_infiniband.conf for an example | 7816 | Generally you can refer to the |
7694 | configuration. In the following paragraph the important details are | 7817 | @file{contrib/regex_profiler_infiniband.conf} file in the sourcecode |
7695 | highlighted. | 7818 | of GNUnet for an example configuration. |
7819 | In the following paragraph the important details are highlighted. | ||
7696 | 7820 | ||
7697 | Announcing of the regular expressions is done by the | 7821 | Announcing of the regular expressions is done by the |
7698 | gnunet-daemon-regexprofiler, therefore you have to make sure it is started, by | 7822 | gnunet-daemon-regexprofiler, therefore you have to make sure it is |
7699 | adding it to the AUTOSTART set of ARM:@ | 7823 | started, by adding it to the AUTOSTART set of ARM: |
7700 | @code{ | ||
7701 | [regexprofiler]@ | ||
7702 | AUTOSTART = YES@ | ||
7703 | } | ||
7704 | 7824 | ||
7825 | @example | ||
7826 | [regexprofiler] | ||
7827 | AUTOSTART = YES | ||
7828 | @end example | ||
7829 | |||
7830 | @noindent | ||
7705 | Furthermore you have to specify the location of the binary: | 7831 | Furthermore you have to specify the location of the binary: |
7832 | |||
7706 | @example | 7833 | @example |
7707 | [regexprofiler] | 7834 | [regexprofiler] |
7708 | # Location of the gnunet-daemon-regexprofiler binary. | 7835 | # Location of the gnunet-daemon-regexprofiler binary. |
@@ -7712,58 +7839,88 @@ BINARY = /home/szengel/gnunet/src/mesh/.libs/gnunet-daemon-regexprofiler | |||
7712 | REGEX_PREFIX = "GNVPN-0001-PAD" | 7839 | REGEX_PREFIX = "GNVPN-0001-PAD" |
7713 | @end example | 7840 | @end example |
7714 | 7841 | ||
7715 | When running the profiler with a large scale deployment, you probably want to | 7842 | @noindent |
7716 | reduce the workload of each peer. Use the following options to do this.@ | 7843 | When running the profiler with a large scale deployment, you probably |
7844 | want to reduce the workload of each peer. | ||
7845 | Use the following options to do this. | ||
7846 | |||
7717 | @example | 7847 | @example |
7718 | [dht]@ | 7848 | [dht] |
7719 | # Force network size estimation@ | 7849 | # Force network size estimation |
7720 | FORCE_NSE = 1 | 7850 | FORCE_NSE = 1 |
7721 | 7851 | ||
7722 | [dhtcache] | 7852 | [dhtcache] |
7723 | DATABASE = heap@ | 7853 | DATABASE = heap |
7724 | # Disable RC-file for Bloom filter? (for benchmarking with limited IO | 7854 | # Disable RC-file for Bloom filter? (for benchmarking with limited IO |
7725 | # availability)@ | 7855 | # availability) |
7726 | DISABLE_BF_RC = YES@ | 7856 | DISABLE_BF_RC = YES |
7727 | # Disable Bloom filter entirely@ | 7857 | # Disable Bloom filter entirely |
7728 | DISABLE_BF = YES | 7858 | DISABLE_BF = YES |
7729 | 7859 | ||
7730 | [nse]@ | 7860 | [nse] |
7731 | # Minimize proof-of-work CPU consumption by NSE@ | 7861 | # Minimize proof-of-work CPU consumption by NSE |
7732 | WORKBITS = 1 | 7862 | WORKBITS = 1 |
7733 | @end example | 7863 | @end example |
7734 | 7864 | ||
7735 | 7865 | @noindent | |
7736 | @strong{Options} | 7866 | @strong{Options} |
7737 | 7867 | ||
7738 | To finally run the profiler some options and the input data need to be | 7868 | To finally run the profiler some options and the input data need to be |
7739 | specified on the command line. | 7869 | specified on the command line. |
7740 | @code{@ gnunet-regex-profiler -c config-file -d | 7870 | |
7741 | log-file -n num-links -p@ path-compression-length -s search-delay -t | 7871 | @example |
7742 | matching-timeout -a num-search-strings hosts-file policy-dir | 7872 | gnunet-regex-profiler -c config-file -d log-file -n num-links \ |
7743 | search-strings-file@ } | 7873 | -p path-compression-length -s search-delay -t matching-timeout \ |
7744 | 7874 | -a num-search-strings hosts-file policy-dir search-strings-file | |
7745 | @code{config-file} the configuration file created earlier.@ @code{log-file} | 7875 | @end example |
7746 | file where to write statistics output.@ @code{num-links} number of random links | 7876 | |
7747 | between started peers.@ @code{path-compression-length} maximum path compression | 7877 | @noindent |
7748 | length in the DFA.@ @code{search-delay} time to wait between peers finished | 7878 | Where... |
7749 | linking and@ starting to match strings.@ @code{matching-timeout} timeout after | 7879 | |
7750 | witch to cancel the searching.@ @code{num-search-strings} number of strings in | 7880 | @itemize @bullet |
7751 | the search-strings-file. | 7881 | @item ... @code{config-file} means the configuration file created earlier. |
7752 | 7882 | @item ... @code{log-file} is the file where to write statistics output. | |
7753 | The @code{hosts-file} should contain a list of hosts for the testbed, one per | 7883 | @item ... @code{num-links} indicates the number of random links between |
7754 | line in the following format. @code{user@@host_ip:port}. | 7884 | started peers. |
7755 | 7885 | @item ... @code{path-compression-length} is the maximum path compression | |
7756 | The @code{policy-dir} is a folder containing text files containing one or more | 7886 | length in the DFA. |
7757 | regular expressions. A peer is started for each file in that folder and the | 7887 | @item ... @code{search-delay} time to wait between peers finished linking |
7758 | regular expressions in the corresponding file are announced by this peer. | 7888 | and starting to match strings. |
7759 | 7889 | @item ... @code{matching-timeout} timeout after which to cancel the | |
7760 | The @code{search-strings-file} is a text file containing search strings, one in | 7890 | searching. |
7761 | each line. | 7891 | @item ... @code{num-search-strings} number of strings in the |
7762 | 7892 | search-strings-file. | |
7763 | You can create regular expressions and search strings for every AS in the@ | 7893 | @item ... the @code{hosts-file} should contain a list of hosts for the |
7894 | testbed, one per line in the following format: | ||
7895 | |||
7896 | @itemize @bullet | ||
7897 | @item @code{user@@host_ip:port} | ||
7898 | @end itemize | ||
7899 | @item ... the @code{policy-dir} is a folder containing text files | ||
7900 | containing one or more regular expressions. A peer is started for each | ||
7901 | file in that folder and the regular expressions in the corresponding file | ||
7902 | are announced by this peer. | ||
7903 | @item ... the @code{search-strings-file} is a text file containing search | ||
7904 | strings, one in each line. | ||
7905 | @end itemize | ||
7906 | |||
7907 | @noindent | ||
7908 | You can create regular expressions and search strings for every AS in the | ||
7764 | Internet using the attached scripts. You need one of the | 7909 | Internet using the attached scripts. You need one of the |
7765 | @uref{http://data.caida.org/datasets/routing/routeviews-prefix2as/, CAIDA | 7910 | @uref{http://data.caida.org/datasets/routing/routeviews-prefix2as/, CAIDA |
7766 | routeviews prefix2as} data files for this. Run @code{create_regex.py <filename> | 7911 | routeviews prefix2as} data files for this. Run |
7767 | <output path>} to create the regular expressions and @code{create_strings.py | 7912 | |
7768 | <input path> <outfile>} to create a search strings file from the previously | 7913 | @example |
7769 | created regular expressions. | 7914 | create_regex.py <filename> <output path> |
7915 | @end example | ||
7916 | |||
7917 | @noindent | ||
7918 | to create the regular expressions and | ||
7919 | |||
7920 | @example | ||
7921 | create_strings.py <input path> <outfile> | ||
7922 | @end example | ||
7923 | |||
7924 | @noindent | ||
7925 | to create a search strings file from the previously created | ||
7926 | regular expressions. | ||