aboutsummaryrefslogtreecommitdiff
path: root/doc/chapters
diff options
context:
space:
mode:
authorng0 <ng0@infotropique.org>2017-10-20 13:43:46 +0000
committerng0 <ng0@infotropique.org>2017-10-20 13:43:46 +0000
commiteb75dd7cae216e99505772a9b32ffca596924819 (patch)
treef3544d9b7cba4dc6232261080c6fa9d598ed73fc /doc/chapters
parent60e3eb315a994ac38778ceb9d493e3624cb4cbe0 (diff)
downloadgnunet-eb75dd7cae216e99505772a9b32ffca596924819.tar.gz
gnunet-eb75dd7cae216e99505772a9b32ffca596924819.zip
fixes and additions in chapters/developer.texi
Diffstat (limited to 'doc/chapters')
-rw-r--r--doc/chapters/developer.texi1621
1 files changed, 889 insertions, 732 deletions
diff --git a/doc/chapters/developer.texi b/doc/chapters/developer.texi
index c858980d0..e690e5f5b 100644
--- a/doc/chapters/developer.texi
+++ b/doc/chapters/developer.texi
@@ -110,7 +110,8 @@ following links:
110@c ** FIXME: Link to files in source, not online. 110@c ** FIXME: Link to files in source, not online.
111@c ** FIXME: Where is the Java tutorial? 111@c ** FIXME: Where is the Java tutorial?
112@itemize @bullet 112@itemize @bullet
113@item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutorial.pdf, GNUnet C tutorial} 113@item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutoria
114l.pdf, GNUnet C tutorial}
114@item GNUnet Java tutorial 115@item GNUnet Java tutorial
115@end itemize 116@end itemize
116 117
@@ -128,7 +129,8 @@ The public subsystems on the GNUnet server that help developers are:
128@item The Version control system keeps our code and enables distributed 129@item The Version control system keeps our code and enables distributed
129development. Only developers with write access can commit code, everyone 130development. Only developers with write access can commit code, everyone
130else is encouraged to submit patches to the 131else is encouraged to submit patches to the
131@uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, GNUnet-developers mailinglist}. 132@uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers,
133GNUnet-developers mailinglist}.
132@item The GNUnet bugtracking system is used to track feature requests, 134@item The GNUnet bugtracking system is used to track feature requests,
133open bug reports and their resolutions. Anyone can report bugs, only 135open bug reports and their resolutions. Anyone can report bugs, only
134developers can claim to have fixed them. 136developers can claim to have fixed them.
@@ -4108,6 +4110,7 @@ maturity, and it is still unclear if any particular plugin is generally
4108superior. 4110superior.
4109 4111
4110@cindex core subsystem 4112@cindex core subsystem
4113@cindex CORE subsystem
4111@node GNUnet's CORE Subsystem 4114@node GNUnet's CORE Subsystem
4112@section GNUnet's CORE Subsystem 4115@section GNUnet's CORE Subsystem
4113@c %**end of header 4116@c %**end of header
@@ -4120,8 +4123,8 @@ then adds fundamental security to the connections:
4120 4123
4121@itemize @bullet 4124@itemize @bullet
4122@item confidentiality with so-called perfect forward secrecy; we use 4125@item confidentiality with so-called perfect forward secrecy; we use
4123ECDHE@footnote{Elliptic-curve Diffie—Hellman 4126ECDHE@footnote{@uref{http://en.wikipedia.org/wiki/Elliptic_curve_
4124@uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman}} 4127Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}}
4125powered by Curve25519 4128powered by Curve25519
4126@footnote{@uref{http://cr.yp.to/ecdh.html, Curve25519}} for the key 4129@footnote{@uref{http://cr.yp.to/ecdh.html, Curve25519}} for the key
4127exchange and then use symmetric encryption, encrypting with both AES-256 4130exchange and then use symmetric encryption, encrypting with both AES-256
@@ -4279,11 +4282,12 @@ using (as theoretically the application may be using a different
4279configuration file with a different private key, which would result in 4282configuration file with a different private key, which would result in
4280hard to find bugs). 4283hard to find bugs).
4281 4284
4282As with most service APIs, the CORE API isolates applications from crashes of 4285As with most service APIs, the CORE API isolates applications from crashes
4283the CORE service. If the CORE service crashes, the application will see 4286of the CORE service. If the CORE service crashes, the application will see
4284disconnect events for all existing connections. Once the connections are 4287disconnect events for all existing connections. Once the connections are
4285re-established, the applications will be receive matching connect events. 4288re-established, the applications will be receive matching connect events.
4286 4289
4290@cindex core clinet-service protocol
4287@node The CORE Client-Service Protocol 4291@node The CORE Client-Service Protocol
4288@subsection The CORE Client-Service Protocol 4292@subsection The CORE Client-Service Protocol
4289@c %**end of header 4293@c %**end of header
@@ -4305,59 +4309,61 @@ service (the client) and the CORE service process itself.
4305When a client connects to the CORE service, it first sends a 4309When a client connects to the CORE service, it first sends a
4306@code{InitMessage} which specifies options for the connection and a set of 4310@code{InitMessage} which specifies options for the connection and a set of
4307message type values which are supported by the application. The options 4311message type values which are supported by the application. The options
4308bitmask specifies which events the client would like to be notified about. The 4312bitmask specifies which events the client would like to be notified about.
4309options include: 4313The options include:
4310 4314
4311@table @asis 4315@table @asis
4312@item GNUNET_CORE_OPTION_NOTHING No notifications 4316@item GNUNET_CORE_OPTION_NOTHING No notifications
4313@item GNUNET_CORE_OPTION_STATUS_CHANGE Peers connecting and disconnecting 4317@item GNUNET_CORE_OPTION_STATUS_CHANGE Peers connecting and disconnecting
4314@item GNUNET_CORE_OPTION_FULL_INBOUND All inbound messages (after decryption) with 4318@item GNUNET_CORE_OPTION_FULL_INBOUND All inbound messages (after
4315full payload 4319decryption) with full payload
4316@item GNUNET_CORE_OPTION_HDR_INBOUND Just the @code{MessageHeader} 4320@item GNUNET_CORE_OPTION_HDR_INBOUND Just the @code{MessageHeader}
4317of all inbound messages 4321of all inbound messages
4318@item GNUNET_CORE_OPTION_FULL_OUTBOUND All outbound 4322@item GNUNET_CORE_OPTION_FULL_OUTBOUND All outbound
4319messages (prior to encryption) with full payload 4323messages (prior to encryption) with full payload
4320@item GNUNET_CORE_OPTION_HDR_OUTBOUND Just the @code{MessageHeader} of all outbound 4324@item GNUNET_CORE_OPTION_HDR_OUTBOUND Just the @code{MessageHeader} of all
4321messages 4325outbound messages
4322@end table 4326@end table
4323 4327
4324Typical applications will only monitor for connection status changes. 4328Typical applications will only monitor for connection status changes.
4325 4329
4326The CORE service responds to the @code{InitMessage} with an 4330The CORE service responds to the @code{InitMessage} with an
4327@code{InitReplyMessage} which contains the peer's identity. Afterwards, both 4331@code{InitReplyMessage} which contains the peer's identity. Afterwards,
4328CORE and the client can send messages. 4332both CORE and the client can send messages.
4329 4333
4330@node Notifications 4334@node Notifications
4331@subsubsection Notifications 4335@subsubsection Notifications
4332@c %**end of header 4336@c %**end of header
4333 4337
4334The CORE will send @code{ConnectNotifyMessage}s and 4338The CORE will send @code{ConnectNotifyMessage}s and
4335@code{DisconnectNotifyMessage}s whenever peers connect or disconnect from the 4339@code{DisconnectNotifyMessage}s whenever peers connect or disconnect from
4336CORE (assuming their type maps overlap with the message types registered by 4340the CORE (assuming their type maps overlap with the message types
4337the client). When the CORE receives a message that matches the set of message 4341registered by the client). When the CORE receives a message that matches
4338types specified during the @code{InitMessage} (or if monitoring is enabled in 4342the set of message types specified during the @code{InitMessage} (or if
4339for inbound messages in the options), it sends a @code{NotifyTrafficMessage} 4343monitoring is enabled in for inbound messages in the options), it sends a
4340with the peer identity of the sender and the decrypted payload. The same 4344@code{NotifyTrafficMessage} with the peer identity of the sender and the
4341message format (except with @code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} 4345decrypted payload. The same message format (except with
4342for the message type) is used to notify clients monitoring outbound messages; 4346@code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} for the message type) is
4343here, the peer identity given is that of the receiver. 4347used to notify clients monitoring outbound messages; here, the peer
4348identity given is that of the receiver.
4344 4349
4345@node Sending 4350@node Sending
4346@subsubsection Sending 4351@subsubsection Sending
4347@c %**end of header 4352@c %**end of header
4348 4353
4349When a client wants to transmit a message, it first requests a transmission 4354When a client wants to transmit a message, it first requests a
4350slot by sending a @code{SendMessageRequest} which specifies the priority, 4355transmission slot by sending a @code{SendMessageRequest} which specifies
4351deadline and size of the message. Note that these values may be ignored by 4356the priority, deadline and size of the message. Note that these values
4352CORE. When CORE is ready for the message, it answers with a 4357may be ignored by CORE. When CORE is ready for the message, it answers
4353@code{SendMessageReady} response. The client can then transmit the payload 4358with a @code{SendMessageReady} response. The client can then transmit the
4354with a @code{SendMessage} message. Note that the actual message size in the 4359payload with a @code{SendMessage} message. Note that the actual message
4355@code{SendMessage} is allowed to be smaller than the size in the original 4360size in the @code{SendMessage} is allowed to be smaller than the size in
4356request. A client may at any time send a fresh @code{SendMessageRequest}, 4361the original request. A client may at any time send a fresh
4357which then superceeds the previous @code{SendMessageRequest}, which is then no 4362@code{SendMessageRequest}, which then superceeds the previous
4358longer valid. The client can tell which @code{SendMessageRequest} the CORE 4363@code{SendMessageRequest}, which is then no longer valid. The client can
4359service's @code{SendMessageReady} message is for as all of these messages 4364tell which @code{SendMessageRequest} the CORE service's
4360contain a "unique" request ID (based on a counter incremented by the client 4365@code{SendMessageReady} message is for as all of these messages contain a
4366"unique" request ID (based on a counter incremented by the client
4361for each request). 4367for each request).
4362 4368
4363@node The CORE Peer-to-Peer Protocol 4369@node The CORE Peer-to-Peer Protocol
@@ -4372,60 +4378,65 @@ for each request).
4372* Type maps:: 4378* Type maps::
4373@end menu 4379@end menu
4374 4380
4381@cindex EphemeralKeyMessage creation
4375@node Creating the EphemeralKeyMessage 4382@node Creating the EphemeralKeyMessage
4376@subsubsection Creating the EphemeralKeyMessage 4383@subsubsection Creating the EphemeralKeyMessage
4377@c %**end of header 4384@c %**end of header
4378 4385
4379When the CORE service starts, each peer creates a fresh ephemeral (ECC) 4386When the CORE service starts, each peer creates a fresh ephemeral (ECC)
4380public-private key pair and signs the corresponding @code{EphemeralKeyMessage} 4387public-private key pair and signs the corresponding
4381with its long-term key (which we usually call the peer's identity; the hash of 4388@code{EphemeralKeyMessage} with its long-term key (which we usually call
4382the public long term key is what results in a @code{struct 4389the peer's identity; the hash of the public long term key is what results
4383GNUNET_PeerIdentity} in all GNUnet APIs. The ephemeral key is ONLY used for an 4390in a @code{struct GNUNET_PeerIdentity} in all GNUnet APIs. The ephemeral
4384@uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman, 4391key is ONLY used for an ECDHE@footnote{@uref{http://en.wikipedia.org/wiki/
4385ECDHE} exchange by the CORE service to establish symmetric session keys. A 4392Elliptic_curve_Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}}
4386peer will use the same @code{EphemeralKeyMessage} for all peers for 4393exchange by the CORE service to establish symmetric session keys. A peer
4387@code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it will 4394will use the same @code{EphemeralKeyMessage} for all peers for
4388create a fresh ephemeral key (forgetting the old one) and broadcast the new 4395@code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it
4389@code{EphemeralKeyMessage} to all connected peers, resulting in fresh 4396will create a fresh ephemeral key (forgetting the old one) and broadcast
4390symmetric session keys. Note that peers independently decide on when to 4397the new @code{EphemeralKeyMessage} to all connected peers, resulting in
4391discard ephemeral keys; it is not a protocol violation to discard keys more 4398fresh symmetric session keys. Note that peers independently decide on
4392often. Ephemeral keys are also never stored to disk; restarting a peer will 4399when to discard ephemeral keys; it is not a protocol violation to discard
4393thus always create a fresh ephemeral key. The use of ephemeral keys is what 4400keys more often. Ephemeral keys are also never stored to disk; restarting
4394provides @uref{http://en.wikipedia.org/wiki/Forward_secrecy, forward secrecy}. 4401a peer will thus always create a fresh ephemeral key. The use of ephemeral
4395 4402keys is what provides @uref{http://en.wikipedia.org/wiki/Forward_secrecy,
4396Just before transmission, the @code{EphemeralKeyMessage} is patched to reflect 4403forward secrecy}.
4397the current sender_status, which specifies the current state of the connection 4404
4398from the point of view of the sender. The possible values are: 4405Just before transmission, the @code{EphemeralKeyMessage} is patched to
4406reflect the current sender_status, which specifies the current state of
4407the connection from the point of view of the sender. The possible values
4408are:
4399 4409
4400@table @asis 4410@itemize @bullet
4401@item KX_STATE_DOWN Initial value, never used on the network 4411@item @code{KX_STATE_DOWN} Initial value, never used on the network
4402@item KX_STATE_KEY_SENT We sent our ephemeral key, do not know the key of the other 4412@item @code{KX_STATE_KEY_SENT} We sent our ephemeral key, do not know the
4403peer 4413key of the other peer
4404@item KX_STATE_KEY_RECEIVED This peer has received a valid ephemeral key 4414@item @code{KX_STATE_KEY_RECEIVED} This peer has received a valid
4405of the other peer, but we are waiting for the other peer to confirm it's 4415ephemeral key of the other peer, but we are waiting for the other peer to
4406authenticity (ability to decode) via challenge-response. 4416confirm it's authenticity (ability to decode) via challenge-response.
4407@item KX_STATE_UP The 4417@item @code{KX_STATE_UP} The connection is fully up from the point of
4408connection is fully up from the point of view of the sender (now performing 4418view of the sender (now performing keep-alives)
4409keep-alives) 4419@item @code{KX_STATE_REKEY_SENT} The sender has initiated a rekeying
4410@item KX_STATE_REKEY_SENT The sender has initiated a rekeying 4420operation; the other peer has so far failed to confirm a working
4411operation; the other peer has so far failed to confirm a working connection 4421connection using the new ephemeral key
4412using the new ephemeral key 4422@end itemize
4413@end table
4414 4423
4415@node Establishing a connection 4424@node Establishing a connection
4416@subsubsection Establishing a connection 4425@subsubsection Establishing a connection
4417@c %**end of header 4426@c %**end of header
4418 4427
4419Peers begin their interaction by sending a @code{EphemeralKeyMessage} to the 4428Peers begin their interaction by sending a @code{EphemeralKeyMessage} to
4420other peer once the TRANSPORT service notifies the CORE service about the 4429the other peer once the TRANSPORT service notifies the CORE service about
4421connection. A peer receiving an @code{EphemeralKeyMessage} with a status 4430the connection.
4431A peer receiving an @code{EphemeralKeyMessage} with a status
4422indicating that the sender does not have the receiver's ephemeral key, the 4432indicating that the sender does not have the receiver's ephemeral key, the
4423receiver's @code{EphemeralKeyMessage} is sent in response.@ Additionally, if 4433receiver's @code{EphemeralKeyMessage} is sent in response.
4424the receiver has not yet confirmed the authenticity of the sender, it also 4434Additionally, if the receiver has not yet confirmed the authenticity of
4425sends an (encrypted)@code{PingMessage} with a challenge (and the identity of 4435the sender, it also sends an (encrypted)@code{PingMessage} with a
4426the target) to the other peer. Peers receiving a @code{PingMessage} respond 4436challenge (and the identity of the target) to the other peer. Peers
4427with an (encrypted) @code{PongMessage} which includes the challenge. Peers 4437receiving a @code{PingMessage} respond with an (encrypted)
4428receiving a @code{PongMessage} check the challenge, and if it matches set the 4438@code{PongMessage} which includes the challenge. Peers receiving a
4439@code{PongMessage} check the challenge, and if it matches set the
4429connection to @code{KX_STATE_UP}. 4440connection to @code{KX_STATE_UP}.
4430 4441
4431@node Encryption and Decryption 4442@node Encryption and Decryption
@@ -4433,26 +4444,27 @@ connection to @code{KX_STATE_UP}.
4433@c %**end of header 4444@c %**end of header
4434 4445
4435All functions related to the key exchange and encryption/decryption of 4446All functions related to the key exchange and encryption/decryption of
4436messages can be found in @code{gnunet-service-core_kx.c} (except for the 4447messages can be found in @file{gnunet-service-core_kx.c} (except for the
4437cryptographic primitives, which are in @code{util/crypto*.c}).@ Given the key 4448cryptographic primitives, which are in @file{util/crypto*.c}).
4438material from ECDHE, a 4449Given the key material from ECDHE, a Key derivation function
4439@uref{http://en.wikipedia.org/wiki/Key_derivation_function, Key derivation 4450@footnote{@uref{https://en.wikipedia.org/wiki/Key_derivation_function, Key
4440function} is used to derive two pairs of encryption and decryption keys for 4451derivation function}} is used to derive two pairs of encryption and
4441AES-256 and TwoFish, as well as initialization vectors and authentication keys 4452decryption keys for AES-256 and TwoFish, as well as initialization vectors
4442(for @uref{http://en.wikipedia.org/wiki/HMAC, HMAC}). The HMAC is computed 4453and authentication keys (for HMAC@footnote{@uref{https://en.wikipedia.org/
4443over the encrypted payload. Encrypted messages include an iv_seed and the HMAC 4454wiki/HMAC, HMAC}}). The HMAC is computed over the encrypted payload.
4444in the header. 4455Encrypted messages include an iv_seed and the HMAC in the header.
4445 4456
4446Each encrypted message in the CORE service includes a sequence number and a 4457Each encrypted message in the CORE service includes a sequence number and
4447timestamp in the encrypted payload. The CORE service remembers the largest 4458a timestamp in the encrypted payload. The CORE service remembers the
4448observed sequence number and a bit-mask which represents which of the previous 4459largest observed sequence number and a bit-mask which represents which of
444932 sequence numbers were already used. Messages with sequence numbers lower 4460the previous 32 sequence numbers were already used.
4450than the largest observed sequence number minus 32 are discarded. Messages 4461Messages with sequence numbers lower than the largest observed sequence
4451with a timestamp that is less than @code{REKEY_TOLERANCE} off (5 minutes) are 4462number minus 32 are discarded. Messages with a timestamp that is less
4452also discarded. This of course means that system clocks need to be reasonably 4463than @code{REKEY_TOLERANCE} off (5 minutes) are also discarded. This of
4453synchronized for peers to be able to communicate. Additionally, as the 4464course means that system clocks need to be reasonably synchronized for
4454ephemeral key changes every 12h, a peer would not even be able to decrypt 4465peers to be able to communicate. Additionally, as the ephemeral key
4455messages older than 12h. 4466changes every 12 hours, a peer would not even be able to decrypt messages
4467older than 12 hours.
4456 4468
4457@node Type maps 4469@node Type maps
4458@subsubsection Type maps 4470@subsubsection Type maps
@@ -4460,103 +4472,111 @@ messages older than 12h.
4460 4472
4461Once an encrypted connection has been established, peers begin to exchange 4473Once an encrypted connection has been established, peers begin to exchange
4462type maps. Type maps are used to allow the CORE service to determine which 4474type maps. Type maps are used to allow the CORE service to determine which
4463(encrypted) connections should be shown to which applications. A type map is 4475(encrypted) connections should be shown to which applications. A type map
4464an array of 65536 bits representing the different types of messages understood 4476is an array of 65536 bits representing the different types of messages
4465by applications using the CORE service. Each CORE service maintains this map, 4477understood by applications using the CORE service. Each CORE service
4466simply by setting the respective bit for each message type supported by any of 4478maintains this map, simply by setting the respective bit for each message
4467the applications using the CORE service. Note that bits for message types 4479type supported by any of the applications using the CORE service. Note
4468embedded in higher-level protocols (such as MESH) will not be included in 4480that bits for message types embedded in higher-level protocols (such as
4469these type maps. 4481MESH) will not be included in these type maps.
4470 4482
4471Typically, the type map of a peer will be sparse. Thus, the CORE service 4483Typically, the type map of a peer will be sparse. Thus, the CORE service
4472attempts to compress its type map using @code{gzip}-style compression 4484attempts to compress its type map using @code{gzip}-style compression
4473("deflate") prior to transmission. However, if the compression fails to 4485("deflate") prior to transmission. However, if the compression fails to
4474compact the map, the map may also be transmitted without compression 4486compact the map, the map may also be transmitted without compression
4475(resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or 4487(resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or
4476@code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively). Upon 4488@code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively).
4477receiving a type map, the respective CORE service notifies applications about 4489Upon receiving a type map, the respective CORE service notifies
4478the connection to the other peer if they support any message type indicated in 4490applications about the connection to the other peer if they support any
4479the type map (or no message type at all). If the CORE service experience a 4491message type indicated in the type map (or no message type at all).
4480connect or disconnect event from an application, it updates its type map 4492If the CORE service experience a connect or disconnect event from an
4481(setting or unsetting the respective bits) and notifies its neighbours about 4493application, it updates its type map (setting or unsetting the respective
4482the change. The CORE services of the neighbours then in turn generate connect 4494bits) and notifies its neighbours about the change.
4483and disconnect events for the peer that sent the type map for their respective 4495The CORE services of the neighbours then in turn generate connect and
4496disconnect events for the peer that sent the type map for their respective
4484applications. As CORE messages may be lost, the CORE service confirms 4497applications. As CORE messages may be lost, the CORE service confirms
4485receiving a type map by sending back a 4498receiving a type map by sending back a
4486@code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation (with 4499@code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation
4487the correct hash of the type map) is not received, the sender will retransmit 4500(with the correct hash of the type map) is not received, the sender will
4488the type map (with exponential back-off). 4501retransmit the type map (with exponential back-off).
4489 4502
4503@cindex cadet subsystem
4504@cindex CADET
4490@node GNUnet's CADET subsystem 4505@node GNUnet's CADET subsystem
4491@section GNUnet's CADET subsystem 4506@section GNUnet's CADET subsystem
4492 4507
4493The CADET subsystem in GNUnet is responsible for secure end-to-end 4508The CADET subsystem in GNUnet is responsible for secure end-to-end
4494communications between nodes in the GNUnet overlay network. CADET builds on the 4509communications between nodes in the GNUnet overlay network. CADET builds
4495CORE subsystem which provides for the link-layer communication and then adds 4510on the CORE subsystem which provides for the link-layer communication and
4496routing, forwarding and additional security to the connections. CADET offers 4511then adds routing, forwarding and additional security to the connections.
4497the same cryptographic services as CORE, but on an end-to-end level. This is 4512CADET offers the same cryptographic services as CORE, but on an
4498done so peers retransmitting traffic on behalf of other peers cannot access the 4513end-to-end level. This is done so peers retransmitting traffic on behalf
4499payload data. 4514of other peers cannot access the payload data.
4500 4515
4501@itemize @bullet 4516@itemize @bullet
4502@item CADET provides confidentiality with so-called perfect forward secrecy; we 4517@item CADET provides confidentiality with so-called perfect forward
4503use ECDHE powered by Curve25519 for the key exchange and then use symmetric 4518secrecy; we use ECDHE powered by Curve25519 for the key exchange and then
4504encryption, encrypting with both AES-256 and Twofish 4519use symmetric encryption, encrypting with both AES-256 and Twofish
4505@item authentication is achieved by signing the ephemeral keys using Ed25519, a 4520@item authentication is achieved by signing the ephemeral keys using
4506deterministic variant of ECDSA 4521Ed25519, a deterministic variant of ECDSA
4507@item integrity protection (using SHA-512 to do encrypt-then-MAC, although only 4522@item integrity protection (using SHA-512 to do encrypt-then-MAC, although
4508256 bits are sent to reduce overhead) 4523only 256 bits are sent to reduce overhead)
4509@item replay protection (using nonces, timestamps, challenge-response, message 4524@item replay protection (using nonces, timestamps, challenge-response,
4510counters and ephemeral keys) 4525message counters and ephemeral keys)
4511@item liveness (keep-alive messages, timeout) 4526@item liveness (keep-alive messages, timeout)
4512@end itemize 4527@end itemize
4513 4528
4514Additional to the CORE-like security benefits, CADET offers other properties 4529Additional to the CORE-like security benefits, CADET offers other
4515that make it a more universal service than CORE. 4530properties that make it a more universal service than CORE.
4516 4531
4517@itemize @bullet 4532@itemize @bullet
4518@item CADET can establish channels to arbitrary peers in GNUnet. If a peer is 4533@item CADET can establish channels to arbitrary peers in GNUnet. If a
4519not immediately reachable, CADET will find a path through the network and ask 4534peer is not immediately reachable, CADET will find a path through the
4520other peers to retransmit the traffic on its behalf. 4535network and ask other peers to retransmit the traffic on its behalf.
4521@item CADET offers (optional) reliability mechanisms. In a reliable channel 4536@item CADET offers (optional) reliability mechanisms. In a reliable
4522traffic is guaranteed to arrive complete, unchanged and in-order. 4537channel traffic is guaranteed to arrive complete, unchanged and in-order.
4523@item CADET takes care of flow and congestion control mechanisms, not allowing 4538@item CADET takes care of flow and congestion control mechanisms, not
4524the sender to send more traffic than the receiver or the network are able to 4539allowing the sender to send more traffic than the receiver or the network
4525process. 4540are able to process.
4526@end itemize 4541@end itemize
4527 4542
4528@menu 4543@menu
4529* libgnunetcadet:: 4544* libgnunetcadet::
4530@end menu 4545@end menu
4531 4546
4547@cindex libgnunetcadet
4532@node libgnunetcadet 4548@node libgnunetcadet
4533@subsection libgnunetcadet 4549@subsection libgnunetcadet
4534 4550
4535 4551
4536The CADET API (defined in gnunet_cadet_service.h) is the messaging API used by 4552The CADET API (defined in @file{gnunet_cadet_service.h}) is the
4537P2P applications built using GNUnet. It provides applications the ability to 4553messaging API used by P2P applications built using GNUnet.
4538send and receive encrypted messages to any peer participating in GNUnet. The 4554It provides applications the ability to send and receive encrypted
4539API is heavily base on the CORE API. 4555messages to any peer participating in GNUnet.
4556The API is heavily base on the CORE API.
4540 4557
4541CADET delivers messages to other peers in "channels". A channel is a permanent 4558CADET delivers messages to other peers in "channels".
4542connection defined by a destination peer (identified by its public key) and a 4559A channel is a permanent connection defined by a destination peer
4543port number. Internally, CADET tunnels all channels towards a destiantion peer 4560(identified by its public key) and a port number.
4561Internally, CADET tunnels all channels towards a destiantion peer
4544using one session key and relays the data on multiple "connections", 4562using one session key and relays the data on multiple "connections",
4545independent from the channels. 4563independent from the channels.
4546 4564
4547Each channel has optional paramenters, the most important being the reliability 4565Each channel has optional paramenters, the most important being the
4548flag. Should a message get lost on TRANSPORT/CORE level, if a channel is 4566reliability flag.
4549created with as reliable, CADET will retransmit the lost message and deliver it 4567Should a message get lost on TRANSPORT/CORE level, if a channel is
4550in order to the destination application. 4568created with as reliable, CADET will retransmit the lost message and
4551 4569deliver it in order to the destination application.
4552To communicate with other peers using CADET, it is necessary to first connect 4570
4553to the service using @code{GNUNET_CADET_connect}. This function takes several 4571To communicate with other peers using CADET, it is necessary to first
4554parameters in form of callbacks, to allow the client to react to various 4572connect to the service using @code{GNUNET_CADET_connect}.
4555events, like incoming channels or channels that terminate, as well as specify a 4573This function takes several parameters in form of callbacks, to allow the
4556list of ports the client wishes to listen to (at the moment it is not possible 4574client to react to various events, like incoming channels or channels that
4557to start listening on further ports once connected, but nothing prevents a 4575terminate, as well as specify a list of ports the client wishes to listen
4558client to connect several times to CADET, even do one connection per listening 4576to (at the moment it is not possible to start listening on further ports
4559port). The function returns a handle which has to be used for any further 4577once connected, but nothing prevents a client to connect several times to
4578CADET, even do one connection per listening port).
4579The function returns a handle which has to be used for any further
4560interaction with the service. 4580interaction with the service.
4561 4581
4562To connect to a remote peer a client has to call the 4582To connect to a remote peer a client has to call the
@@ -4564,62 +4584,69 @@ To connect to a remote peer a client has to call the
4564given are the remote peer's identity (it public key) and a port, which 4584given are the remote peer's identity (it public key) and a port, which
4565specifies which application on the remote peer to connect to, similar to 4585specifies which application on the remote peer to connect to, similar to
4566TCP/UDP ports. CADET will then find the peer in the GNUnet network and 4586TCP/UDP ports. CADET will then find the peer in the GNUnet network and
4567establish the proper low-level connections and do the necessary key exchanges 4587establish the proper low-level connections and do the necessary key
4568to assure and authenticated, secure and verified communication. Similar to 4588exchanges to assure and authenticated, secure and verified communication.
4569@code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel} returns a handle 4589Similar to @code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel}
4570to interact with the created channel. 4590returns a handle to interact with the created channel.
4571 4591
4572For every message the client wants to send to the remote application, 4592For every message the client wants to send to the remote application,
4573@code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the 4593@code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the
4574channel on which the message should be sent and the size of the message (but 4594channel on which the message should be sent and the size of the message
4575not the message itself!). Once CADET is ready to send the message, the provided 4595(but not the message itself!). Once CADET is ready to send the message,
4576callback will fire, and the message contents are provided to this callback. 4596the provided callback will fire, and the message contents are provided to
4597this callback.
4577 4598
4578Please note the CADET does not provide an explicit notification of when a 4599Please note the CADET does not provide an explicit notification of when a
4579channel is connected. In loosely connected networks, like big wireless mesh 4600channel is connected. In loosely connected networks, like big wireless
4580networks, this can take several seconds, even minutes in the worst case. To be 4601mesh networks, this can take several seconds, even minutes in the worst
4581alerted when a channel is online, a client can call 4602case. To be alerted when a channel is online, a client can call
4582@code{GNUNET_CADET_notify_transmit_ready} immediately after 4603@code{GNUNET_CADET_notify_transmit_ready} immediately after
4583@code{GNUNET_CADET_create_channel}. When the callback is activated, it means 4604@code{GNUNET_CADET_create_channel}. When the callback is activated, it
4584that the channel is online. The callback can give 0 bytes to CADET if no 4605means that the channel is online. The callback can give 0 bytes to CADET
4585message is to be sent, this is ok. 4606if no message is to be sent, this is ok.
4586 4607
4587If a transmission was requested but before the callback fires it is no longer 4608If a transmission was requested but before the callback fires it is no
4588needed, it can be cancelled with 4609longer needed, it can be cancelled with
4589@code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle given 4610@code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle
4590back by @code{GNUNET_CADET_notify_transmit_ready}. As in the case of CORE, only 4611given back by @code{GNUNET_CADET_notify_transmit_ready}.
4591one message can be requested at a time: a client must not call 4612As in the case of CORE, only one message can be requested at a time: a
4592@code{GNUNET_CADET_notify_transmit_ready} again until the callback is called or 4613client must not call @code{GNUNET_CADET_notify_transmit_ready} again until
4593the request is cancelled. 4614the callback is called or the request is cancelled.
4594 4615
4595When a channel is no longer needed, a client can call 4616When a channel is no longer needed, a client can call
4596@code{GNUNET_CADET_channel_destroy} to get rid of it. Note that CADET will try 4617@code{GNUNET_CADET_channel_destroy} to get rid of it.
4597to transmit all pending traffic before notifying the remote peer of the 4618Note that CADET will try to transmit all pending traffic before notifying
4598destruction of the channel, including retransmitting lost messages if the 4619the remote peer of the destruction of the channel, including
4599channel was reliable. 4620retransmitting lost messages if the channel was reliable.
4600 4621
4601Incoming channels, channels being closed by the remote peer, and traffic on any 4622Incoming channels, channels being closed by the remote peer, and traffic
4602incoming or outgoing channels are given to the client when CADET executes the 4623on any incoming or outgoing channels are given to the client when CADET
4603callbacks given to it at the time of @code{GNUNET_CADET_connect}. 4624executes the callbacks given to it at the time of
4625@code{GNUNET_CADET_connect}.
4604 4626
4605Finally, when an application no longer wants to use CADET, it should call 4627Finally, when an application no longer wants to use CADET, it should call
4606@code{GNUNET_CADET_disconnect}, but first all channels and pending 4628@code{GNUNET_CADET_disconnect}, but first all channels and pending
4607transmissions must be closed (otherwise CADET will complain). 4629transmissions must be closed (otherwise CADET will complain).
4608 4630
4631@cindex nse subsystem
4632@cindex NSE
4609@node GNUnet's NSE subsystem 4633@node GNUnet's NSE subsystem
4610@section GNUnet's NSE subsystem 4634@section GNUnet's NSE subsystem
4611 4635
4612 4636
4613NSE stands for Network Size Estimation. The NSE subsystem provides other 4637NSE stands for @dfn{Network Size Estimation}. The NSE subsystem provides
4614subsystems and users with a rough estimate of the number of peers currently 4638other subsystems and users with a rough estimate of the number of peers
4615participating in the GNUnet overlay. The computed value is not a precise number 4639currently participating in the GNUnet overlay.
4616as producing a precise number in a decentralized, efficient and secure way is 4640The computed value is not a precise number as producing a precise number
4617impossible. While NSE's estimate is inherently imprecise, NSE also gives the 4641in a decentralized, efficient and secure way is impossible.
4618expected range. For a peer that has been running in a stable network for a 4642While NSE's estimate is inherently imprecise, NSE also gives the expected
4619while, the real network size will typically (99.7% of the time) be in the range 4643range. For a peer that has been running in a stable network for a
4620of [2/3 estimate, 3/2 estimate]. We will now give an overview of the algorithm 4644while, the real network size will typically (99.7% of the time) be in the
4621used to calcualte the estimate; all of the details can be found in this 4645range of [2/3 estimate, 3/2 estimate]. We will now give an overview of the
4622technical report. 4646algorithm used to calculate the estimate;
4647all of the details can be found in this technical report.
4648
4649@c FIXME: link to the report.
4623 4650
4624@menu 4651@menu
4625* Motivation:: 4652* Motivation::
@@ -4634,13 +4661,14 @@ technical report.
4634 4661
4635 4662
4636Some subsytems, like DHT, need to know the size of the GNUnet network to 4663Some subsytems, like DHT, need to know the size of the GNUnet network to
4637optimize some parameters of their own protocol. The decentralized nature of 4664optimize some parameters of their own protocol. The decentralized nature
4638GNUnet makes efficient and securely counting the exact number of peers 4665of GNUnet makes efficient and securely counting the exact number of peers
4639infeasable. Although there are several decentralized algorithms to count the 4666infeasable. Although there are several decentralized algorithms to count
4640number of peers in a system, so far there is none to do so securely. Other 4667the number of peers in a system, so far there is none to do so securely.
4641protocols may allow any malicious peer to manipulate the final result or to 4668Other protocols may allow any malicious peer to manipulate the final
4642take advantage of the system to perform DoS (Denial of Service) attacks against 4669result or to take advantage of the system to perform
4643the network. GNUnet's NSE protocol avoids these drawbacks. 4670@dfn{Denial of Service} (DoS) attacks against the network.
4671GNUnet's NSE protocol avoids these drawbacks.
4644 4672
4645 4673
4646 4674
@@ -4648,28 +4676,34 @@ the network. GNUnet's NSE protocol avoids these drawbacks.
4648* Security:: 4676* Security::
4649@end menu 4677@end menu
4650 4678
4679@cindex NSE security
4680@cindex nse security
4651@node Security 4681@node Security
4652@subsubsection Security 4682@subsubsection Security
4653 4683
4654 4684
4655The NSE subsystem is designed to be resilient against these attacks. It uses 4685The NSE subsystem is designed to be resilient against these attacks.
4656@uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs of work} to 4686It uses @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs
4657prevent one peer from impersonating a large number of participants, which would 4687of work} to prevent one peer from impersonating a large number of
4658otherwise allow an adversary to artifically inflate the estimate. The DoS 4688participants, which would otherwise allow an adversary to artifically
4659protection comes from the time-based nature of the protocol: the estimates are 4689inflate the estimate.
4660calculated periodically and out-of-time traffic is either ignored or stored for 4690The DoS protection comes from the time-based nature of the protocol:
4661later retransmission by benign peers. In particular, peers cannot trigger 4691the estimates are calculated periodically and out-of-time traffic is
4662global network communication at will. 4692either ignored or stored for later retransmission by benign peers.
4693In particular, peers cannot trigger global network communication at will.
4663 4694
4695@cindex NSE principle
4696@cindex nse principle
4664@node Principle 4697@node Principle
4665@subsection Principle 4698@subsection Principle
4666 4699
4667 4700
4668The algorithm calculates the estimate by finding the globally closest peer ID 4701The algorithm calculates the estimate by finding the globally closest
4669to a random, time-based value. 4702peer ID to a random, time-based value.
4670 4703
4671The idea is that the closer the ID is to the random value, the more "densely 4704The idea is that the closer the ID is to the random value, the more
4672packed" the ID space is, and therefore, more peers are in the network. 4705"densely packed" the ID space is, and therefore, more peers are in the
4706network.
4673 4707
4674 4708
4675 4709
@@ -4686,48 +4720,51 @@ packed" the ID space is, and therefore, more peers are in the network.
4686@subsubsection Example 4720@subsubsection Example
4687 4721
4688 4722
4689Suppose all peers have IDs between 0 and 100 (our ID space), and the random 4723Suppose all peers have IDs between 0 and 100 (our ID space), and the
4690value is 42. If the closest peer has the ID 70 we can imagine that the average 4724random value is 42.
4691"distance" between peers is around 30 and therefore the are around 3 peers in 4725If the closest peer has the ID 70 we can imagine that the average
4692the whole ID space. On the other hand, if the closest peer has the ID 44, we 4726"distance" between peers is around 30 and therefore the are around 3
4693can imagine that the space is rather packed with peers, maybe as much as 50 of 4727peers in the whole ID space. On the other hand, if the closest peer has
4694them. Naturally, we could have been rather unlucky, and there is only one peer 4728the ID 44, we can imagine that the space is rather packed with peers,
4695and happens to have the ID 44. Thus, the current estimate is calculated as the 4729maybe as much as 50 of them.
4696average over multiple rounds, and not just a single sample. 4730Naturally, we could have been rather unlucky, and there is only one peer
4731and happens to have the ID 44. Thus, the current estimate is calculated
4732as the average over multiple rounds, and not just a single sample.
4697 4733
4698@node Algorithm 4734@node Algorithm
4699@subsubsection Algorithm 4735@subsubsection Algorithm
4700 4736
4701 4737
4702Given that example, one can imagine that the job of the subsystem is to 4738Given that example, one can imagine that the job of the subsystem is to
4703efficiently communicate the ID of the closest peer to the target value to all 4739efficiently communicate the ID of the closest peer to the target value
4704the other peers, who will calculate the estimate from it. 4740to all the other peers, who will calculate the estimate from it.
4705 4741
4706@node Target value 4742@node Target value
4707@subsubsection Target value 4743@subsubsection Target value
4708 4744
4709@c %**end of header 4745@c %**end of header
4710 4746
4711The target value itself is generated by hashing the current time, rounded down 4747The target value itself is generated by hashing the current time, rounded
4712to an agreed value. If the rounding amount is 1h (default) and the time is 4748down to an agreed value. If the rounding amount is 1h (default) and the
471312:34:56, the time to hash would be 12:00:00. The process is repeated each 4749time is 12:34:56, the time to hash would be 12:00:00. The process is
4714rouning amount (in this example would be every hour). Every repetition is 4750repeated each rouning amount (in this example would be every hour).
4715called a round. 4751Every repetition is called a round.
4716 4752
4717@node Timing 4753@node Timing
4718@subsubsection Timing 4754@subsubsection Timing
4719@c %**end of header 4755@c %**end of header
4720 4756
4721The NSE subsystem has some timing control to avoid everybody broadcasting its 4757The NSE subsystem has some timing control to avoid everybody broadcasting
4722ID all at one. Once each peer has the target random value, it compares its own 4758its ID all at one. Once each peer has the target random value, it
4723ID to the target and calculates the hypothetical size of the network if that 4759compares its own ID to the target and calculates the hypothetical size of
4724peer were to be the closest. Then it compares the hypothetical size with the 4760the network if that peer were to be the closest.
4725estimate from the previous rounds. For each value there is an assiciated point 4761Then it compares the hypothetical size with the estimate from the previous
4726in the period, let's call it "broadcast time". If its own hypothetical estimate 4762rounds. For each value there is an assiciated point in the period,
4727is the same as the previous global estimate, its "broadcast time" will be in 4763let's call it "broadcast time". If its own hypothetical estimate
4728the middle of the round. If its bigger it will be earlier and if its smaler 4764is the same as the previous global estimate, its "broadcast time" will be
4729(the most likely case) it will be later. This ensures that the peers closests 4765in the middle of the round. If its bigger it will be earlier and if its
4730to the target value start broadcasting their ID the first. 4766smaller (the most likely case) it will be later. This ensures that the
4767peers closests to the target value start broadcasting their ID the first.
4731 4768
4732@node Controlled Flooding 4769@node Controlled Flooding
4733@subsubsection Controlled Flooding 4770@subsubsection Controlled Flooding
@@ -4735,52 +4772,56 @@ to the target value start broadcasting their ID the first.
4735@c %**end of header 4772@c %**end of header
4736 4773
4737When a peer receives a value, first it verifies that it is closer than the 4774When a peer receives a value, first it verifies that it is closer than the
4738closest value it had so far, otherwise it answers the incoming message with a 4775closest value it had so far, otherwise it answers the incoming message
4739message containing the better value. Then it checks a proof of work that must 4776with a message containing the better value. Then it checks a proof of
4740be included in the incoming message, to ensure that the other peer's ID is not 4777work that must be included in the incoming message, to ensure that the
4741made up (otherwise a malicious peer could claim to have an ID of exactly the 4778other peer's ID is not made up (otherwise a malicious peer could claim to
4742target value every round). Once validated, it compares the brodcast time of the 4779have an ID of exactly the target value every round). Once validated, it
4743received value with the current time and if it's not too early, sends the 4780compares the brodcast time of the received value with the current time
4744received value to its neighbors. Otherwise it stores the value until the 4781and if it's not too early, sends the received value to its neighbors.
4745correct broadcast time comes. This prevents unnecessary traffic of sub-optimal 4782Otherwise it stores the value until the correct broadcast time comes.
4746values, since a better value can come before the broadcast time, rendering the 4783This prevents unnecessary traffic of sub-optimal values, since a better
4747previous one obsolete and saving the traffic that would have been used to 4784value can come before the broadcast time, rendering the previous one
4748broadcast it to the neighbors. 4785obsolete and saving the traffic that would have been used to broadcast it
4786to the neighbors.
4749 4787
4750@node Calculating the estimate 4788@node Calculating the estimate
4751@subsubsection Calculating the estimate 4789@subsubsection Calculating the estimate
4752 4790
4753@c %**end of header 4791@c %**end of header
4754 4792
4755Once the closest ID has been spread across the network each peer gets the exact 4793Once the closest ID has been spread across the network each peer gets the
4756distance betweed this ID and the target value of the round and calculates the 4794exact distance betweed this ID and the target value of the round and
4757estimate with a mathematical formula described in the tech report. The estimate 4795calculates the estimate with a mathematical formula described in the tech
4758generated with this method for a single round is not very precise. Remember the 4796report. The estimate generated with this method for a single round is not
4759case of the example, where the only peer is the ID 44 and we happen to generate 4797very precise. Remember the case of the example, where the only peer is the
4760the target value 42, thinking there are 50 peers in the network. Therefore, the 4798ID 44 and we happen to generate the target value 42, thinking there are
4761NSE subsystem remembers the last 64 estimates and calculates an average over 479950 peers in the network. Therefore, the NSE subsystem remembers the last
4762them, giving a result of which usually has one bit of uncertainty (the real 480064 estimates and calculates an average over them, giving a result of which
4763size could be half of the estimate or twice as much). Note that the actual 4801usually has one bit of uncertainty (the real size could be half of the
4764network size is calculated in powers of two of the raw input, thus one bit of 4802estimate or twice as much). Note that the actual network size is
4765uncertainty means a factor of two in the size estimate. 4803calculated in powers of two of the raw input, thus one bit of uncertainty
4804means a factor of two in the size estimate.
4766 4805
4806@cindex libgnunetnse
4767@node libgnunetnse 4807@node libgnunetnse
4768@subsection libgnunetnse 4808@subsection libgnunetnse
4769 4809
4770@c %**end of header 4810@c %**end of header
4771 4811
4772The NSE subsystem has the simplest API of all services, with only two calls: 4812The NSE subsystem has the simplest API of all services, with only two
4773@code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}. 4813calls: @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}.
4774 4814
4775The connect call gets a callback function as a parameter and this function is 4815The connect call gets a callback function as a parameter and this function
4776called each time the network agrees on an estimate. This usually is once per 4816is called each time the network agrees on an estimate. This usually is
4777round, with some exceptions: if the closest peer has a late local clock and 4817once per round, with some exceptions: if the closest peer has a late
4778starts spreading his ID after everyone else agreed on a value, the callback 4818local clock and starts spreading his ID after everyone else agreed on a
4779might be activated twice in a round, the second value being always bigger than 4819value, the callback might be activated twice in a round, the second value
4780the first. The default round time is set to 1 hour. 4820being always bigger than the first. The default round time is set to
48211 hour.
4781 4822
4782The disconnect call disconnects from the NSE subsystem and the callback is no 4823The disconnect call disconnects from the NSE subsystem and the callback
4783longer called with new estimates. 4824is no longer called with new estimates.
4784 4825
4785 4826
4786 4827
@@ -4795,11 +4836,11 @@ longer called with new estimates.
4795@c %**end of header 4836@c %**end of header
4796 4837
4797The callback provides two values: the average and the 4838The callback provides two values: the average and the
4798@uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation} of 4839@uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation}
4799the last 64 rounds. The values provided by the callback function are 4840of the last 64 rounds. The values provided by the callback function are
4800logarithmic, this means that the real estimate numbers can be obtained by 4841logarithmic, this means that the real estimate numbers can be obtained by
4801calculating 2 to the power of the given value (2average). From a statistics 4842calculating 2 to the power of the given value (2average). From a
4802point of view this means that: 4843statistics point of view this means that:
4803 4844
4804@itemize @bullet 4845@itemize @bullet
4805@item 68% of the time the real size is included in the interval 4846@item 68% of the time the real size is included in the interval
@@ -4810,8 +4851,8 @@ point of view this means that:
4810[(2average-3*stddev, 2average+3*stddev] 4851[(2average-3*stddev, 2average+3*stddev]
4811@end itemize 4852@end itemize
4812 4853
4813The expected standard variation for 64 rounds in a network of stable size is 4854The expected standard variation for 64 rounds in a network of stable size
48140.2. Thus, we can say that normally: 4855is 0.2. Thus, we can say that normally:
4815 4856
4816@itemize @bullet 4857@itemize @bullet
4817@item 68% of the time the real size is in the range [-13%, +15%] 4858@item 68% of the time the real size is in the range [-13%, +15%]
@@ -4819,10 +4860,11 @@ The expected standard variation for 64 rounds in a network of stable size is
4819@item 99.7% of the time the real size is in the range [-34%, +52%] 4860@item 99.7% of the time the real size is in the range [-34%, +52%]
4820@end itemize 4861@end itemize
4821 4862
4822As said in the introduction, we can be quite sure that usually the real size is 4863As said in the introduction, we can be quite sure that usually the real
4823between one third and three times the estimate. This can of course vary with 4864size is between one third and three times the estimate. This can of
4824network conditions. Thus, applications may want to also consider the provided 4865course vary with network conditions.
4825standard deviation value, not only the average (in particular, if the standard 4866Thus, applications may want to also consider the provided standard
4867deviation value, not only the average (in particular, if the standard
4826veriation is very high, the average maybe meaningless: the network size is 4868veriation is very high, the average maybe meaningless: the network size is
4827changing rapidly). 4869changing rapidly).
4828 4870
@@ -4835,15 +4877,23 @@ Let's close with a couple examples.
4835 4877
4836@table @asis 4878@table @asis
4837 4879
4838@item Average: 10, std dev: 1 Here the estimate would be 2^10 = 1024 peers. @footnote{The range in which we can be 95% sure is: [2^8, 2^12] = [256, 4096]. We can be very (>99.7%) sure that the network is not a hundred peers and absolutely sure that it is not a million peers, but somewhere around a thousand.} 4880@item Average: 10, std dev: 1 Here the estimate would be
48812^10 = 1024 peers. @footnote{The range in which we can be 95% sure is:
4882[2^8, 2^12] = [256, 4096]. We can be very (>99.7%) sure that the network
4883is not a hundred peers and absolutely sure that it is not a million peers,
4884but somewhere around a thousand.}
4839 4885
4840@item Average 22, std dev: 0.2 Here the estimate would be 2^22 = 4 Million peers. @footnote{The range in which we can be 99.7% sure is: [2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size is around four million, with absolutely way of it being 1 million.} 4886@item Average 22, std dev: 0.2 Here the estimate would be
48872^22 = 4 Million peers. @footnote{The range in which we can be 99.7% sure
4888is: [2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size
4889is around four million, with absolutely way of it being 1 million.}
4841 4890
4842@end table 4891@end table
4843 4892
4844To put this in perspective, if someone remembers the LHC Higgs boson results, 4893To put this in perspective, if someone remembers the LHC Higgs boson
4845were announced with "5 sigma" and "6 sigma" certainties. In this case a 5 sigma 4894results, were announced with "5 sigma" and "6 sigma" certainties. In this
4846minimum would be 2 million and a 6 sigma minimum, 1.8 million. 4895case a 5 sigma minimum would be 2 million and a 6 sigma minimum,
48961.8 million.
4847 4897
4848@node The NSE Client-Service Protocol 4898@node The NSE Client-Service Protocol
4849@subsection The NSE Client-Service Protocol 4899@subsection The NSE Client-Service Protocol
@@ -4854,16 +4904,16 @@ As with the API, the client-service protocol is very simple, only has 2
4854different messages, defined in @code{src/nse/nse.h}: 4904different messages, defined in @code{src/nse/nse.h}:
4855 4905
4856@itemize @bullet 4906@itemize @bullet
4857@item @code{GNUNET_MESSAGE_TYPE_NSE_START}@ This message has no parameters and 4907@item @code{GNUNET_MESSAGE_TYPE_NSE_START}@ This message has no parameters
4858is sent from the client to the service upon connection. 4908and is sent from the client to the service upon connection.
4859@item @code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE}@ This message is sent from the 4909@item @code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE}@ This message is sent from
4860service to the client for every new estimate and upon connection. Contains a 4910the service to the client for every new estimate and upon connection.
4861timestamp for the estimate, the average and the standard deviation for the 4911Contains a timestamp for the estimate, the average and the standard
4862respective round. 4912deviation for the respective round.
4863@end itemize 4913@end itemize
4864 4914
4865When the @code{GNUNET_NSE_disconnect} API call is executed, the client simply 4915When the @code{GNUNET_NSE_disconnect} API call is executed, the client
4866disconnects from the service, with no message involved. 4916simply disconnects from the service, with no message involved.
4867 4917
4868@node The NSE Peer-to-Peer Protocol 4918@node The NSE Peer-to-Peer Protocol
4869@subsection The NSE Peer-to-Peer Protocol 4919@subsection The NSE Peer-to-Peer Protocol
@@ -4873,77 +4923,80 @@ disconnects from the service, with no message involved.
4873The NSE subsystem only has one message in the P2P protocol, the 4923The NSE subsystem only has one message in the P2P protocol, the
4874@code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message. 4924@code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message.
4875 4925
4876This message key contents are the timestamp to identify the round (differences 4926This message key contents are the timestamp to identify the round
4877in system clocks may cause some peers to send messages way too early or way too 4927(differences in system clocks may cause some peers to send messages way
4878late, so the timestamp allows other peers to identify such messages easily), 4928too early or way too late, so the timestamp allows other peers to
4879the @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proof of work} 4929identify such messages easily), the
4930@uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proof of work}
4880used to make it difficult to mount a 4931used to make it difficult to mount a
4881@uref{http://en.wikipedia.org/wiki/Sybil_attack, Sybil attack}, and the public 4932@uref{http://en.wikipedia.org/wiki/Sybil_attack, Sybil attack}, and the
4882key, which is used to verify the signature on the message. 4933public key, which is used to verify the signature on the message.
4883 4934
4884Every peer stores a message for the previous, current and next round. The 4935Every peer stores a message for the previous, current and next round. The
4885messages for the previous and current round are given to peers that connect to 4936messages for the previous and current round are given to peers that
4886us. The message for the next round is simply stored until our system clock 4937connect to us. The message for the next round is simply stored until our
4887advances to the next round. The message for the current round is what we are 4938system clock advances to the next round. The message for the current round
4888flooding the network with right now. At the beginning of each round the peer 4939is what we are flooding the network with right now.
4889does the following: 4940At the beginning of each round the peer does the following:
4890 4941
4891@itemize @bullet 4942@itemize @bullet
4892@item calculates his own distance to the target value 4943@item calculates his own distance to the target value
4893@item creates, signs and stores the message for the current round (unless it 4944@item creates, signs and stores the message for the current round (unless
4894has a better message in the "next round" slot which came early in the previous 4945it has a better message in the "next round" slot which came early in the
4895round) 4946previous round)
4896@item calculates, based on the stored round message (own or received) when to 4947@item calculates, based on the stored round message (own or received) when
4897stard flooding it to its neighbors 4948to stard flooding it to its neighbors
4898@end itemize 4949@end itemize
4899 4950
4900Upon receiving a message the peer checks the validity of the message (round, 4951Upon receiving a message the peer checks the validity of the message
4901proof of work, signature). The next action depends on the contents of the 4952(round, proof of work, signature). The next action depends on the
4902incoming message: 4953contents of the incoming message:
4903 4954
4904@itemize @bullet 4955@itemize @bullet
4905@item if the message is worse than the current stored message, the peer sends 4956@item if the message is worse than the current stored message, the peer
4906the current message back immediately, to stop the other peer from spreading 4957sends the current message back immediately, to stop the other peer from
4907suboptimal results 4958spreading suboptimal results
4908@item if the message is better than the current stored message, the peer stores 4959@item if the message is better than the current stored message, the peer
4909the new message and calculates the new target time to start spreading it to its 4960stores the new message and calculates the new target time to start
4910neighbors (excluding the one the message came from) 4961spreading it to its neighbors (excluding the one the message came from)
4911@item if the message is for the previous round, it is compared to the message 4962@item if the message is for the previous round, it is compared to the
4912stored in the "previous round slot", which may then be updated 4963message stored in the "previous round slot", which may then be updated
4913@item if the message is for the next round, it is compared to the message 4964@item if the message is for the next round, it is compared to the message
4914stored in the "next round slot", which again may then be updated 4965stored in the "next round slot", which again may then be updated
4915@end itemize 4966@end itemize
4916 4967
4917Finally, when it comes to send the stored message for the current round to the 4968Finally, when it comes to send the stored message for the current round to
4918neighbors there is a random delay added for each neighbor, to avoid traffic 4969the neighbors there is a random delay added for each neighbor, to avoid
4919spikes and minimize cross-messages. 4970traffic spikes and minimize cross-messages.
4920 4971
4972@cindex HOSTLIST subsystem
4973@cindex hostlist subsystem
4921@node GNUnet's HOSTLIST subsystem 4974@node GNUnet's HOSTLIST subsystem
4922@section GNUnet's HOSTLIST subsystem 4975@section GNUnet's HOSTLIST subsystem
4923 4976
4924@c %**end of header 4977@c %**end of header
4925 4978
4926Peers in the GNUnet overlay network need address information so that they can 4979Peers in the GNUnet overlay network need address information so that they
4927connect with other peers. GNUnet uses so called HELLO messages to store and 4980can connect with other peers. GNUnet uses so called HELLO messages to
4928exchange peer addresses. GNUnet provides several methods for peers to obtain 4981store and exchange peer addresses.
4929this information: 4982GNUnet provides several methods for peers to obtain this information:
4930 4983
4931@itemize @bullet 4984@itemize @bullet
4932@item out-of-band exchange of HELLO messages (manually, using for example 4985@item out-of-band exchange of HELLO messages (manually, using for example
4933gnunet-peerinfo) 4986gnunet-peerinfo)
4934@item HELLO messages shipped with GNUnet (automatic with distribution) 4987@item HELLO messages shipped with GNUnet (automatic with distribution)
4935@item UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast) 4988@item UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast)
4936@item topology gossiping (learning from other peers we already connected to), 4989@item topology gossiping (learning from other peers we already connected
4937and 4990to), and
4938@item the HOSTLIST daemon covered in this section, which is particularly 4991@item the HOSTLIST daemon covered in this section, which is particularly
4939relevant for bootstrapping new peers. 4992relevant for bootstrapping new peers.
4940@end itemize 4993@end itemize
4941 4994
4942New peers have no existing connections (and thus cannot learn from gossip among 4995New peers have no existing connections (and thus cannot learn from gossip
4943peers), may not have other peers in their LAN and might be started with an 4996among peers), may not have other peers in their LAN and might be started
4944outdated set of HELLO messages from the distribution. In this case, getting new 4997with an outdated set of HELLO messages from the distribution.
4945peers to connect to the network requires either manual effort or the use of a 4998In this case, getting new peers to connect to the network requires either
4946HOSTLIST to obtain HELLOs. 4999manual effort or the use of a HOSTLIST to obtain HELLOs.
4947 5000
4948@menu 5001@menu
4949* HELLOs:: 5002* HELLOs::
@@ -4961,11 +5014,12 @@ HOSTLIST to obtain HELLOs.
4961 5014
4962@c %**end of header 5015@c %**end of header
4963 5016
4964The basic information peers require to connect to other peers are contained in 5017The basic information peers require to connect to other peers are
4965so called HELLO messages you can think of as a business card. Besides the 5018contained in so called HELLO messages you can think of as a business card.
4966identity of the peer (based on the cryptographic public key) a HELLO message 5019Besides the identity of the peer (based on the cryptographic public key) a
4967may contain address information that specifies ways to contact a peer. By 5020HELLO message may contain address information that specifies ways to
4968obtaining HELLO messages, a peer can learn how to contact other peers. 5021contact a peer. By obtaining HELLO messages, a peer can learn how to
5022contact other peers.
4969 5023
4970@node Overview for the HOSTLIST subsystem 5024@node Overview for the HOSTLIST subsystem
4971@subsection Overview for the HOSTLIST subsystem 5025@subsection Overview for the HOSTLIST subsystem
@@ -4973,17 +5027,19 @@ obtaining HELLO messages, a peer can learn how to contact other peers.
4973@c %**end of header 5027@c %**end of header
4974 5028
4975The HOSTLIST subsystem provides a way to distribute and obtain contact 5029The HOSTLIST subsystem provides a way to distribute and obtain contact
4976information to connect to other peers using a simple HTTP GET request. It's 5030information to connect to other peers using a simple HTTP GET request.
4977implementation is split in three parts, the main file for the daemon itself 5031It's implementation is split in three parts, the main file for the daemon
4978(gnunet-daemon-hostlist.c), the HTTP client used to download peer information 5032itself (@file{gnunet-daemon-hostlist.c}), the HTTP client used to download
4979(hostlist-client.c) and the server component used to provide this information 5033peer information (@file{hostlist-client.c}) and the server component used
4980to other peers (hostlist-server.c). The server is basically a small HTTP web 5034to provide this information to other peers (@file{hostlist-server.c}).
4981server (based on GNU libmicrohttpd) which provides a list of HELLOs known to 5035The server is basically a small HTTP web server (based on GNU
4982the local peer for download. The client component is basically a HTTP client 5036libmicrohttpd) which provides a list of HELLOs known to the local peer for
4983(based on libcurl) which can download hostlists from one or more websites. The 5037download. The client component is basically a HTTP client
4984hostlist format is a binary blob containing a sequence of HELLO messages. Note 5038(based on libcurl) which can download hostlists from one or more websites.
4985that any HTTP server can theoretically serve a hostlist, the build-in hostlist 5039The hostlist format is a binary blob containing a sequence of HELLO
4986server makes it simply convenient to offer this service. 5040messages. Note that any HTTP server can theoretically serve a hostlist,
5041the build-in hostlist server makes it simply convenient to offer this
5042service.
4987 5043
4988 5044
4989@menu 5045@menu
@@ -4999,13 +5055,14 @@ server makes it simply convenient to offer this service.
4999The HOSTLIST daemon can: 5055The HOSTLIST daemon can:
5000 5056
5001@itemize @bullet 5057@itemize @bullet
5002@item provide HELLO messages with validated addresses obtained from PEERINFO to 5058@item provide HELLO messages with validated addresses obtained from
5003download for other peers 5059PEERINFO to download for other peers
5004@item download HELLO messages and forward these message to the TRANSPORT 5060@item download HELLO messages and forward these message to the TRANSPORT
5005subsystem for validation 5061subsystem for validation
5006@item advertises the URL of this peer's hostlist address to other peers via 5062@item advertises the URL of this peer's hostlist address to other peers
5007gossip 5063via gossip
5008@item automatically learn about hostlist servers from the gossip of other peers 5064@item automatically learn about hostlist servers from the gossip of other
5065peers
5009@end itemize 5066@end itemize
5010 5067
5011@node Limitations2 5068@node Limitations2
@@ -5025,25 +5082,28 @@ The HOSTLIST daemon does not:
5025 5082
5026@c %**end of header 5083@c %**end of header
5027 5084
5028The HOSTLIST subsystem is currently implemented as a daemon, so there is no 5085The HOSTLIST subsystem is currently implemented as a daemon, so there is
5029need for the user to interact with it and therefore there is no command line 5086no need for the user to interact with it and therefore there is no
5030tool and no API to communicate with the daemon. In the future, we can envision 5087command line tool and no API to communicate with the daemon. In the
5031changing this to allow users to manually trigger the download of a hostlist. 5088future, we can envision changing this to allow users to manually trigger
5089the download of a hostlist.
5090
5091Since there is no command line interface to interact with HOSTLIST, the
5092only way to interact with the hostlist is to use STATISTICS to obtain or
5093modify information about the status of HOSTLIST:
5032 5094
5033Since there is no command line interface to interact with HOSTLIST, the only
5034way to interact with the hostlist is to use STATISTICS to obtain or modify
5035information about the status of HOSTLIST:
5036@example 5095@example
5037$ gnunet-statistics -s hostlist 5096$ gnunet-statistics -s hostlist
5038@end example 5097@end example
5039 5098
5040In particular, HOSTLIST includes a @strong{persistent} value in statistics that 5099@noindent
5041specifies when the hostlist server might be queried next. As this value is 5100In particular, HOSTLIST includes a @strong{persistent} value in statistics
5042exponentially increasing during runtime, developers may want to reset or 5101that specifies when the hostlist server might be queried next. As this
5043manually adjust it. Note that HOSTLIST (but not STATISTICS) needs to be 5102value is exponentially increasing during runtime, developers may want to
5044shutdown if changes to this value are to have any effect on the daemon (as 5103reset or manually adjust it. Note that HOSTLIST (but not STATISTICS) needs
5045HOSTLIST does not monitor STATISTICS for changes to the download 5104to be shutdown if changes to this value are to have any effect on the
5046frequency). 5105daemon (as HOSTLIST does not monitor STATISTICS for changes to the
5106download frequency).
5047 5107
5048@node Hostlist security address validation 5108@node Hostlist security address validation
5049@subsection Hostlist security address validation 5109@subsection Hostlist security address validation
@@ -5051,18 +5111,19 @@ frequency).
5051@c %**end of header 5111@c %**end of header
5052 5112
5053Since information obtained from other parties cannot be trusted without 5113Since information obtained from other parties cannot be trusted without
5054validation, we have to distinguish between @emph{validated} and @emph{not 5114validation, we have to distinguish between @emph{validated} and
5055validated} addresses. Before using (and so trusting) information from other 5115@emph{not validated} addresses. Before using (and so trusting)
5056parties, this information has to be double-checked (validated). Address 5116information from other parties, this information has to be double-checked
5057validation is not done by HOSTLIST but by the TRANSPORT service. 5117(validated). Address validation is not done by HOSTLIST but by the
5058 5118TRANSPORT service.
5059The HOSTLIST component is functionally located between the PEERINFO and the 5119
5060TRANSPORT subsystem. When acting as a server, the daemon obtains valid 5120The HOSTLIST component is functionally located between the PEERINFO and
5061(@emph{validated}) peer information (HELLO messages) from the PEERINFO service 5121the TRANSPORT subsystem. When acting as a server, the daemon obtains valid
5062and provides it to other peers. When acting as a client, it contacts the 5122(@emph{validated}) peer information (HELLO messages) from the PEERINFO
5063HOSTLIST servers specified in the configuration, downloads the (unvalidated) 5123service and provides it to other peers. When acting as a client, it
5064list of HELLO messages and forwards these information to the TRANSPORT server 5124contacts the HOSTLIST servers specified in the configuration, downloads
5065to validate the addresses. 5125the (unvalidated) list of HELLO messages and forwards these information
5126to the TRANSPORT server to validate the addresses.
5066 5127
5067@node The HOSTLIST daemon 5128@node The HOSTLIST daemon
5068@subsection The HOSTLIST daemon 5129@subsection The HOSTLIST daemon
@@ -5070,24 +5131,25 @@ to validate the addresses.
5070@c %**end of header 5131@c %**end of header
5071 5132
5072The hostlist daemon is the main component of the HOSTLIST subsystem. It is 5133The hostlist daemon is the main component of the HOSTLIST subsystem. It is
5073started by the ARM service and (if configured) starts the HOSTLIST client and 5134started by the ARM service and (if configured) starts the HOSTLIST client
5074server components. 5135and server components.
5075 5136
5076If the daemon provides a hostlist itself it can advertise it's own hostlist to 5137If the daemon provides a hostlist itself it can advertise it's own
5077other peers. To do so it sends a GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT 5138hostlist to other peers. To do so it sends a
5078message to other peers when they connect to this peer on the CORE level. This 5139@code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to other peers
5079hostlist advertisement message contains the URL to access the HOSTLIST HTTP 5140when they connect to this peer on the CORE level. This hostlist
5080server of the sender. The daemon may also subscribe to this type of message 5141advertisement message contains the URL to access the HOSTLIST HTTP
5081from CORE service, and then forward these kind of message to the HOSTLIST 5142server of the sender. The daemon may also subscribe to this type of
5082client. The client then uses all available URLs to download peer information 5143message from CORE service, and then forward these kind of message to the
5083when necessary. 5144HOSTLIST client. The client then uses all available URLs to download peer
5084 5145information when necessary.
5085When starting, the HOSTLIST daemon first connects to the CORE subsystem and if 5146
5086hostlist learning is enabled, registers a CORE handler to receive this kind of 5147When starting, the HOSTLIST daemon first connects to the CORE subsystem
5087messages. Next it starts (if configured) the client and server. It passes 5148and if hostlist learning is enabled, registers a CORE handler to receive
5088pointers to CORE connect and disconnect and receive handlers where the client 5149this kind of messages. Next it starts (if configured) the client and
5089and server store their functions, so the daemon can notify them about CORE 5150server. It passes pointers to CORE connect and disconnect and receive
5090events. 5151handlers where the client and server store their functions, so the daemon
5152can notify them about CORE events.
5091 5153
5092To clean up on shutdown, the daemon has a cleaning task, shutting down all 5154To clean up on shutdown, the daemon has a cleaning task, shutting down all
5093subsystems and disconnecting from CORE. 5155subsystems and disconnecting from CORE.
@@ -5097,10 +5159,10 @@ subsystems and disconnecting from CORE.
5097 5159
5098@c %**end of header 5160@c %**end of header
5099 5161
5100The server provides a way for other peers to obtain HELLOs. Basically it is a 5162The server provides a way for other peers to obtain HELLOs. Basically it
5101small web server other peers can connect to and download a list of HELLOs using 5163is a small web server other peers can connect to and download a list of
5102standard HTTP; it may also advertise the URL of the hostlist to other peers 5164HELLOs using standard HTTP; it may also advertise the URL of the hostlist
5103connecting on CORE level. 5165to other peers connecting on CORE level.
5104 5166
5105 5167
5106@menu 5168@menu
@@ -5113,49 +5175,56 @@ connecting on CORE level.
5113 5175
5114@c %**end of header 5176@c %**end of header
5115 5177
5116During startup, the server starts a web server listening on the port specified 5178During startup, the server starts a web server listening on the port
5117with the HTTPPORT value (default 8080). In addition it connects to the PEERINFO 5179specified with the HTTPPORT value (default 8080). In addition it connects
5118service to obtain peer information. The HOSTLIST server uses the 5180to the PEERINFO service to obtain peer information. The HOSTLIST server
5119GNUNET_PEERINFO_iterate function to request HELLO information for all peers and 5181uses the GNUNET_PEERINFO_iterate function to request HELLO information for
5120adds their information to a new hostlist if they are suitable (expired 5182all peers and adds their information to a new hostlist if they are
5121addresses and HELLOs without addresses are both not suitable) and the maximum 5183suitable (expired addresses and HELLOs without addresses are both not
5122size for a hostlist is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When 5184suitable) and the maximum size for a hostlist is not exceeded
5123PEERINFO finishes (with a last NULL callback), the server destroys the previous 5185(MAX_BYTES_PER_HOSTLISTS = 500000).
5124hostlist response available for download on the web server and replaces it with 5186When PEERINFO finishes (with a last NULL callback), the server destroys
5125the updated hostlist. The hostlist format is basically a sequence of HELLO 5187the previous hostlist response available for download on the web server
5126messages (as obtained from PEERINFO) without any special tokenization. Since 5188and replaces it with the updated hostlist. The hostlist format is
5127each HELLO message contains a size field, the response can easily be split into 5189basically a sequence of HELLO messages (as obtained from PEERINFO) without
5128separate HELLO messages by the client. 5190any special tokenization. Since each HELLO message contains a size field,
5129 5191the response can easily be split into separate HELLO messages by the
5130A HOSTLIST client connecting to the HOSTLIST server will receive the hostlist 5192client.
5131as a HTTP response and the the server will terminate the connection with the 5193
5132result code HTTP 200 OK. The connection will be closed immediately if no 5194A HOSTLIST client connecting to the HOSTLIST server will receive the
5133hostlist is available. 5195hostlist as a HTTP response and the the server will terminate the
5196connection with the result code @code{HTTP 200 OK}.
5197The connection will be closed immediately if no hostlist is available.
5134 5198
5135@node Advertising the URL 5199@node Advertising the URL
5136@subsubsection Advertising the URL 5200@subsubsection Advertising the URL
5137 5201
5138@c %**end of header 5202@c %**end of header
5139 5203
5140The server also advertises the URL to download the hostlist to other peers if 5204The server also advertises the URL to download the hostlist to other peers
5141hostlist advertisement is enabled. When a new peer connects and has hostlist 5205if hostlist advertisement is enabled.
5142learning enabled, the server sends a GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT 5206When a new peer connects and has hostlist learning enabled, the server
5143message to this peer using the CORE service. 5207sends a @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this
5208peer using the CORE service.
5144 5209
5145@node The HOSTLIST client 5210@node The HOSTLIST client
5146@subsection The HOSTLIST client 5211@subsection The HOSTLIST client
5147 5212
5148@c %**end of header 5213@c %**end of header
5149 5214
5150The client provides the functionality to download the list of HELLOs from a set 5215The client provides the functionality to download the list of HELLOs from
5151of URLs. It performs a standard HTTP request to the URLs configured and learned 5216a set of URLs.
5217It performs a standard HTTP request to the URLs configured and learned
5152from advertisement messages received from other peers. When a HELLO is 5218from advertisement messages received from other peers. When a HELLO is
5153downloaded, the HOSTLIST client forwards the HELLO to the TRANSPORT service for 5219downloaded, the HOSTLIST client forwards the HELLO to the TRANSPORT
5154validation. 5220service for validation.
5155 5221
5156The client supports two modes of operation: download of HELLOs (bootstrapping) 5222The client supports two modes of operation:
5157and learning of URLs.
5158 5223
5224@itemize @bullet
5225@item download of HELLOs (bootstrapping)
5226@item learning of URLs
5227@end itemize
5159 5228
5160@menu 5229@menu
5161* Bootstrapping:: 5230* Bootstrapping::
@@ -5167,102 +5236,121 @@ and learning of URLs.
5167 5236
5168@c %**end of header 5237@c %**end of header
5169 5238
5170For bootstrapping, it schedules a task to download the hostlist from the set of 5239For bootstrapping, it schedules a task to download the hostlist from the
5171known URLs. The downloads are only performed if the number of current 5240set of known URLs.
5172connections is smaller than a minimum number of connections (at the moment 4). 5241The downloads are only performed if the number of current
5242connections is smaller than a minimum number of connections
5243(at the moment 4).
5173The interval between downloads increases exponentially; however, the 5244The interval between downloads increases exponentially; however, the
5174exponential growth is limited if it becomes longer than an hour. At that point, 5245exponential growth is limited if it becomes longer than an hour.
5175the frequency growth is capped at (#number of connections * 1h). 5246At that point, the frequency growth is capped at
5247(#number of connections * 1h).
5176 5248
5177Once the decision has been taken to download HELLOs, the daemon chooses a 5249Once the decision has been taken to download HELLOs, the daemon chooses a
5178random URL from the list of known URLs. URLs can be configured in the 5250random URL from the list of known URLs. URLs can be configured in the
5179configuration or be learned from advertisement messages. The client uses a HTTP 5251configuration or be learned from advertisement messages.
5180client library (libcurl) to initiate the download using the libcurl multi 5252The client uses a HTTP client library (libcurl) to initiate the download
5181interface. Libcurl passes the data to the callback_download function which 5253using the libcurl multi interface.
5182stores the data in a buffer if space is available and the maximum size for a 5254Libcurl passes the data to the callback_download function which
5183hostlist download is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When a 5255stores the data in a buffer if space is available and the maximum size for
5184full HELLO was downloaded, the HOSTLIST client offers this HELLO message to the 5256a hostlist download is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000).
5185TRANSPORT service for validation. When the download is finished or failed, 5257When a full HELLO was downloaded, the HOSTLIST client offers this
5186statistical information about the quality of this URL is updated. 5258HELLO message to the TRANSPORT service for validation.
5187 5259When the download is finished or failed, statistical information about the
5260quality of this URL is updated.
5261
5262@cindex HOSTLIST learning
5188@node Learning 5263@node Learning
5189@subsubsection Learning 5264@subsubsection Learning
5190 5265
5191@c %**end of header 5266@c %**end of header
5192 5267
5193The client also manages hostlist advertisements from other peers. The HOSTLIST 5268The client also manages hostlist advertisements from other peers. The
5194daemon forwards GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT messages to the 5269HOSTLIST daemon forwards @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT}
5195client subsystem, which extracts the URL from the message. Next, a test of the 5270messages to the client subsystem, which extracts the URL from the message.
5196newly obtained URL is performed by triggering a download from the new URL. If 5271Next, a test of the newly obtained URL is performed by triggering a
5197the URL works correctly, it is added to the list of working URLs. 5272download from the new URL. If the URL works correctly, it is added to the
5273list of working URLs.
5198 5274
5199The size of the list of URLs is restricted, so if an additional server is added 5275The size of the list of URLs is restricted, so if an additional server is
5200and the list is full, the URL with the worst quality ranking (determined 5276added and the list is full, the URL with the worst quality ranking
5201through successful downloads and number of HELLOs e.g.) is discarded. During 5277(determined through successful downloads and number of HELLOs e.g.) is
5202shutdown the list of URLs is saved to a file for persistance and loaded on 5278discarded. During shutdown the list of URLs is saved to a file for
5203startup. URLs from the configuration file are never discarded. 5279persistance and loaded on startup. URLs from the configuration file are
5280never discarded.
5204 5281
5205@node Usage 5282@node Usage
5206@subsection Usage 5283@subsection Usage
5207 5284
5208@c %**end of header 5285@c %**end of header
5209 5286
5210To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES section 5287To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES
5211for the ARM services. This is done in the default configuration. 5288section for the ARM services. This is done in the default configuration.
5212 5289
5213For more information on how to configure the HOSTLIST subsystem see the 5290For more information on how to configure the HOSTLIST subsystem see the
5214installation handbook:@ Configuring the hostlist to bootstrap@ Configuring your 5291installation handbook:@
5215peer to provide a hostlist 5292Configuring the hostlist to bootstrap@
5293Configuring your peer to provide a hostlist
5216 5294
5295@cindex IDENTITY
5296@cindex identity subsystem
5217@node GNUnet's IDENTITY subsystem 5297@node GNUnet's IDENTITY subsystem
5218@section GNUnet's IDENTITY subsystem 5298@section GNUnet's IDENTITY subsystem
5219 5299
5220@c %**end of header 5300@c %**end of header
5221 5301
5222Identities of "users" in GNUnet are called egos. Egos can be used as pseudonyms 5302Identities of "users" in GNUnet are called egos.
5223(fake names) or be tied to an organization (for example, GNU) or even the 5303Egos can be used as pseudonyms ("fake names") or be tied to an
5224actual identity of a human. GNUnet users are expected to have many egos. They 5304organization (for example, "GNU") or even the actual identity of a human.
5225might have one tied to their real identity, some for organizations they manage, 5305GNUnet users are expected to have many egos. They might have one tied to
5226and more for different domains where they want to operate under a pseudonym. 5306their real identity, some for organizations they manage, and more for
5227 5307different domains where they want to operate under a pseudonym.
5228The IDENTITY service allows users to manage their egos. The identity service 5308
5229manages the private keys egos of the local user; it does not manage identities 5309The IDENTITY service allows users to manage their egos. The identity
5230of other users (public keys). Public keys for other users need names to become 5310service manages the private keys egos of the local user; it does not
5231manageable. GNUnet uses the GNU Name System (GNS) to give names to other users 5311manage identities of other users (public keys). Public keys for other
5232and manage their public keys securely. This chapter is about the IDENTITY 5312users need names to become manageable. GNUnet uses the
5233service, which is about the management of private keys. 5313@dfn{GNU Name System} (GNS) to give names to other users and manage their
5234 5314public keys securely. This chapter is about the IDENTITY service,
5235On the network, an ego corresponds to an ECDSA key (over Curve25519, using RFC 5315which is about the management of private keys.
52366979, as required by GNS). Thus, users can perform actions under a particular 5316
5237ego by using (signing with) a particular private key. Other users can then 5317On the network, an ego corresponds to an ECDSA key (over Curve25519,
5238confirm that the action was really performed by that ego by checking the 5318using RFC 6979, as required by GNS). Thus, users can perform actions
5239signature against the respective public key. 5319under a particular ego by using (signing with) a particular private key.
5240 5320Other users can then confirm that the action was really performed by that
5241The IDENTITY service allows users to associate a human-readable name with each 5321ego by checking the signature against the respective public key.
5242ego. This way, users can use names that will remind them of the purpose of a 5322
5243particular ego. The IDENTITY service will store the respective private keys and 5323The IDENTITY service allows users to associate a human-readable name with
5244allows applications to access key information by name. Users can change the 5324each ego. This way, users can use names that will remind them of the
5245name that is locally (!) associated with an ego. Egos can also be deleted, 5325purpose of a particular ego.
5246which means that the private key will be removed and it thus will not be 5326The IDENTITY service will store the respective private keys and
5247possible to perform actions with that ego in the future. 5327allows applications to access key information by name.
5248 5328Users can change the name that is locally (!) associated with an ego.
5249Additionally, the IDENTITY subsystem can associate service functions with egos. 5329Egos can also be deleted, which means that the private key will be removed
5250For example, GNS requires the ego that should be used for the shorten zone. GNS 5330and it thus will not be possible to perform actions with that ego in the
5251will ask IDENTITY for an ego for the "gns-short" service. The IDENTITY service 5331future.
5252has a mapping of such service strings to the name of the ego that the user 5332
5253wants to use for this service, for example "my-short-zone-ego". 5333Additionally, the IDENTITY subsystem can associate service functions with
5254 5334egos.
5255Finally, the IDENTITY API provides access to a special ego, the anonymous ego. 5335For example, GNS requires the ego that should be used for the shorten
5256The anonymous ego is special in that its private key is not really private, but 5336zone. GNS will ask IDENTITY for an ego for the "gns-short" service.
5257fixed and known to everyone. Thus, anyone can perform actions as anonymous. 5337The IDENTITY service has a mapping of such service strings to the name of
5258This can be useful as with this trick, code does not have to contain a special 5338the ego that the user wants to use for this service, for example
5259case to distinguish between anonymous and pseudonymous egos. 5339"my-short-zone-ego".
5340
5341Finally, the IDENTITY API provides access to a special ego, the
5342anonymous ego. The anonymous ego is special in that its private key is not
5343really private, but fixed and known to everyone.
5344Thus, anyone can perform actions as anonymous. This can be useful as with
5345this trick, code does not have to contain a special case to distinguish
5346between anonymous and pseudonymous egos.
5260 5347
5261@menu 5348@menu
5262* libgnunetidentity:: 5349* libgnunetidentity::
5263* The IDENTITY Client-Service Protocol:: 5350* The IDENTITY Client-Service Protocol::
5264@end menu 5351@end menu
5265 5352
5353@cindex libgnunetidentity
5266@node libgnunetidentity 5354@node libgnunetidentity
5267@subsection libgnunetidentity 5355@subsection libgnunetidentity
5268@c %**end of header 5356@c %**end of header
@@ -5282,55 +5370,58 @@ case to distinguish between anonymous and pseudonymous egos.
5282@c %**end of header 5370@c %**end of header
5283 5371
5284First, typical clients connect to the identity service using 5372First, typical clients connect to the identity service using
5285@code{GNUNET_IDENTITY_connect}. This function takes a callback as a parameter. 5373@code{GNUNET_IDENTITY_connect}. This function takes a callback as a
5286If the given callback parameter is non-null, it will be invoked to notify the 5374parameter.
5287application about the current state of the identities in the system. 5375If the given callback parameter is non-null, it will be invoked to notify
5376the application about the current state of the identities in the system.
5288 5377
5289@itemize @bullet 5378@itemize @bullet
5290@item First, it will be invoked on all known egos at the time of the 5379@item First, it will be invoked on all known egos at the time of the
5291connection. For each ego, a handle to the ego and the user's name for the ego 5380connection. For each ego, a handle to the ego and the user's name for the
5292will be passed to the callback. Furthermore, a @code{void **} context argument 5381ego will be passed to the callback. Furthermore, a @code{void **} context
5293will be provided which gives the client the opportunity to associate some state 5382argument will be provided which gives the client the opportunity to
5294with the ego. 5383associate some state with the ego.
5295@item Second, the callback will be invoked with NULL for the ego, the name and 5384@item Second, the callback will be invoked with NULL for the ego, the name
5296the context. This signals that the (initial) iteration over all egos has 5385and the context. This signals that the (initial) iteration over all egos
5297completed. 5386has completed.
5298@item Then, the callback will be invoked whenever something changes about an 5387@item Then, the callback will be invoked whenever something changes about
5299ego. If an ego is renamed, the callback is invoked with the ego handle of the 5388an ego.
5300ego that was renamed, and the new name. If an ego is deleted, the callback is 5389If an ego is renamed, the callback is invoked with the ego handle of the
5301invoked with the ego handle and a name of NULL. In the deletion case, the 5390ego that was renamed, and the new name. If an ego is deleted, the callback
5302application should also release resources stored in the context. 5391is invoked with the ego handle and a name of NULL. In the deletion case,
5392the application should also release resources stored in the context.
5303@item When the application destroys the connection to the identity service 5393@item When the application destroys the connection to the identity service
5304using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked with the 5394using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked
5305ego and a name of NULL (equivalent to deletion of the egos). This should again 5395with the ego and a name of NULL (equivalent to deletion of the egos).
5306be used to clean up the per-ego context. 5396This should again be used to clean up the per-ego context.
5307@end itemize 5397@end itemize
5308 5398
5309The ego handle passed to the callback remains valid until the callback is 5399The ego handle passed to the callback remains valid until the callback is
5310invoked with a name of NULL, so it is safe to store a reference to the ego's 5400invoked with a name of NULL, so it is safe to store a reference to the
5311handle. 5401ego's handle.
5312 5402
5313@node Operations on Egos 5403@node Operations on Egos
5314@subsubsection Operations on Egos 5404@subsubsection Operations on Egos
5315 5405
5316@c %**end of header 5406@c %**end of header
5317 5407
5318Given an ego handle, the main operations are to get its associated private key 5408Given an ego handle, the main operations are to get its associated private
5319using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated public key 5409key using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated
5320using @code{GNUNET_IDENTITY_ego_get_public_key}. 5410public key using @code{GNUNET_IDENTITY_ego_get_public_key}.
5321 5411
5322The other operations on egos are pretty straightforward. Using 5412The other operations on egos are pretty straightforward.
5323@code{GNUNET_IDENTITY_create}, an application can request the creation of an 5413Using @code{GNUNET_IDENTITY_create}, an application can request the
5324ego by specifying the desired name. The operation will fail if that name is 5414creation of an ego by specifying the desired name.
5325already in use. Using @code{GNUNET_IDENTITY_rename} the name of an existing ego 5415The operation will fail if that name is
5326can be changed. Finally, egos can be deleted using 5416already in use. Using @code{GNUNET_IDENTITY_rename} the name of an
5327@code{GNUNET_IDENTITY_delete}. All of these operations will trigger updates to 5417existing ego can be changed. Finally, egos can be deleted using
5328the callback given to the @code{GNUNET_IDENTITY_connect} function of all 5418@code{GNUNET_IDENTITY_delete}. All of these operations will trigger
5329applications that are connected with the identity service at the time. 5419updates to the callback given to the @code{GNUNET_IDENTITY_connect}
5330@code{GNUNET_IDENTITY_cancel} can be used to cancel the operations before the 5420function of all applications that are connected with the identity service
5331respective continuations would be called. It is not guaranteed that the 5421at the time. @code{GNUNET_IDENTITY_cancel} can be used to cancel the
5332operation will not be completed anyway, only the continuation will no longer be 5422operations before the respective continuations would be called.
5333called. 5423It is not guaranteed that the operation will not be completed anyway,
5424only the continuation will no longer be called.
5334 5425
5335@node The anonymous Ego 5426@node The anonymous Ego
5336@subsubsection The anonymous Ego 5427@subsubsection The anonymous Ego
@@ -5339,11 +5430,11 @@ called.
5339 5430
5340A special way to obtain an ego handle is to call 5431A special way to obtain an ego handle is to call
5341@code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the 5432@code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the
5342"anonymous" user --- anyone knows and can get the private key for this user, so 5433"anonymous" user --- anyone knows and can get the private key for this
5343it is suitable for operations that are supposed to be anonymous but require 5434user, so it is suitable for operations that are supposed to be anonymous
5344signatures (for example, to avoid a special path in the code). The anonymous 5435but require signatures (for example, to avoid a special path in the code).
5345ego is always valid and accessing it does not require a connection to the 5436The anonymous ego is always valid and accessing it does not require a
5346identity service. 5437connection to the identity service.
5347 5438
5348@node Convenience API to lookup a single ego 5439@node Convenience API to lookup a single ego
5349@subsubsection Convenience API to lookup a single ego 5440@subsubsection Convenience API to lookup a single ego
@@ -5351,98 +5442,106 @@ identity service.
5351 5442
5352As applications commonly simply have to lookup a single ego, there is a 5443As applications commonly simply have to lookup a single ego, there is a
5353convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to 5444convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to
5354lookup a single ego by name. Note that this is the user's name for the ego, not 5445lookup a single ego by name. Note that this is the user's name for the
5355the service function. The resulting ego will be returned via a callback and 5446ego, not the service function. The resulting ego will be returned via a
5356will only be valid during that callback. The operation can be cancelled via 5447callback and will only be valid during that callback. The operation can
5357@code{GNUNET_IDENTITY_ego_lookup_cancel} (cancellation is only legal before the 5448be cancelled via @code{GNUNET_IDENTITY_ego_lookup_cancel}
5358callback is invoked). 5449(cancellation is only legal before the callback is invoked).
5359 5450
5360@node Associating egos with service functions 5451@node Associating egos with service functions
5361@subsubsection Associating egos with service functions 5452@subsubsection Associating egos with service functions
5362 5453
5363 5454
5364The @code{GNUNET_IDENTITY_set} function is used to associate a particular ego 5455The @code{GNUNET_IDENTITY_set} function is used to associate a particular
5365with a service function. The name used by the service and the ego are given as 5456ego with a service function. The name used by the service and the ego are
5366arguments. Afterwards, the service can use its name to lookup the associated 5457given as arguments.
5367ego using @code{GNUNET_IDENTITY_get}. 5458Afterwards, the service can use its name to lookup the associated ego
5459using @code{GNUNET_IDENTITY_get}.
5368 5460
5369@node The IDENTITY Client-Service Protocol 5461@node The IDENTITY Client-Service Protocol
5370@subsection The IDENTITY Client-Service Protocol 5462@subsection The IDENTITY Client-Service Protocol
5371 5463
5372@c %**end of header 5464@c %**end of header
5373 5465
5374A client connecting to the identity service first sends a message with type 5466A client connecting to the identity service first sends a message with
5467type
5375@code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that, the 5468@code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that, the
5376client will receive information about changes to the egos by receiving messages 5469client will receive information about changes to the egos by receiving
5377of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}. Those messages contain the 5470messages of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}.
5378private key of the ego and the user's name of the ego (or zero bytes for the 5471Those messages contain the private key of the ego and the user's name of
5379name to indicate that the ego was deleted). A special bit @code{end_of_list} is 5472the ego (or zero bytes for the name to indicate that the ego was deleted).
5380used to indicate the end of the initial iteration over the identity service's 5473A special bit @code{end_of_list} is used to indicate the end of the
5381egos. 5474initial iteration over the identity service's egos.
5382 5475
5383The client can trigger changes to the egos by sending CREATE, RENAME or DELETE 5476The client can trigger changes to the egos by sending @code{CREATE},
5384messages. The CREATE message contains the private key and the desired name. The 5477@code{RENAME} or @code{DELETE} messages.
5385RENAME message contains the old name and the new name. The DELETE message only 5478The CREATE message contains the private key and the desired name.@
5386needs to include the name of the ego to delete. The service responds to each of 5479The RENAME message contains the old name and the new name.@
5387these messages with a RESULT_CODE message which indicates success or error of 5480The DELETE message only needs to include the name of the ego to delete.@
5388the operation, and possibly a human-readable error message. 5481The service responds to each of these messages with a @code{RESULT_CODE}
5482message which indicates success or error of the operation, and possibly
5483a human-readable error message.
5389 5484
5390Finally, the client can bind the name of a service function to an ego by 5485Finally, the client can bind the name of a service function to an ego by
5391sending a SET_DEFAULT message with the name of the service function and the 5486sending a @code{SET_DEFAULT} message with the name of the service function
5392private key of the ego. Such bindings can then be resolved using a GET_DEFAULT 5487and the private key of the ego.
5393message, which includes the name of the service function. The identity service 5488Such bindings can then be resolved using a @code{GET_DEFAULT} message,
5394will respond to a GET_DEFAULT request with a SET_DEFAULT message containing the 5489which includes the name of the service function. The identity service
5395respective information, or with a RESULT_CODE to indicate an error. 5490will respond to a GET_DEFAULT request with a SET_DEFAULT message
5396 5491containing the respective information, or with a RESULT_CODE to
5492indicate an error.
5493
5494@cindex NAMESTORE
5495@cindex namestore subsystem
5397@node GNUnet's NAMESTORE Subsystem 5496@node GNUnet's NAMESTORE Subsystem
5398@section GNUnet's NAMESTORE Subsystem 5497@section GNUnet's NAMESTORE Subsystem
5399 5498
5400@c %**end of header
5401
5402The NAMESTORE subsystem provides persistent storage for local GNS zone 5499The NAMESTORE subsystem provides persistent storage for local GNS zone
5403information. All local GNS zone information are managed by NAMESTORE. It 5500information. All local GNS zone information are managed by NAMESTORE. It
5404provides both the functionality to administer local GNS information (e.g. 5501provides both the functionality to administer local GNS information (e.g.
5405delete and add records) as well as to retrieve GNS information (e.g to list 5502delete and add records) as well as to retrieve GNS information (e.g to
5406name information in a client). NAMESTORE does only manage the persistent 5503list name information in a client).
5407storage of zone information belonging to the user running the service: GNS 5504NAMESTORE does only manage the persistent storage of zone information
5408information from other users obtained from the DHT are stored by the NAMECACHE 5505belonging to the user running the service: GNS information from other
5409subsystem. 5506users obtained from the DHT are stored by the NAMECACHE subsystem.
5410 5507
5411NAMESTORE uses a plugin-based database backend to store GNS information with 5508NAMESTORE uses a plugin-based database backend to store GNS information
5412good performance. Here sqlite, MySQL and PostgreSQL are supported database 5509with good performance. Here sqlite, MySQL and PostgreSQL are supported
5413backends. NAMESTORE clients interact with the IDENTITY subsystem to obtain 5510database backends.
5511NAMESTORE clients interact with the IDENTITY subsystem to obtain
5414cryptographic information about zones based on egos as described with the 5512cryptographic information about zones based on egos as described with the
5415IDENTITY subsystem., but internally NAMESTORE refers to zones using the ECDSA 5513IDENTITY subsystem, but internally NAMESTORE refers to zones using the
5416private key. In addition, it collaborates with the NAMECACHE subsystem and 5514ECDSA private key.
5417stores zone information when local information are modified in the GNS cache to 5515In addition, it collaborates with the NAMECACHE subsystem and
5418increase look-up performance for local information. 5516stores zone information when local information are modified in the
5517GNS cache to increase look-up performance for local information.
5419 5518
5420NAMESTORE provides functionality to look-up and store records, to iterate over 5519NAMESTORE provides functionality to look-up and store records, to iterate
5421a specific or all zones and to monitor zones for changes. NAMESTORE 5520over a specific or all zones and to monitor zones for changes. NAMESTORE
5422functionality can be accessed using the NAMESTORE api or the NAMESTORE command 5521functionality can be accessed using the NAMESTORE api or the NAMESTORE
5423line tool. 5522command line tool.
5424 5523
5425@menu 5524@menu
5426* libgnunetnamestore:: 5525* libgnunetnamestore::
5427@end menu 5526@end menu
5428 5527
5528@cindex libgnunetnamestore
5429@node libgnunetnamestore 5529@node libgnunetnamestore
5430@subsection libgnunetnamestore 5530@subsection libgnunetnamestore
5431 5531
5432@c %**end of header 5532To interact with NAMESTORE clients first connect to the NAMESTORE service
5533using the @code{GNUNET_NAMESTORE_connect} passing a configuration handle.
5534As a result they obtain a NAMESTORE handle, they can use for operations,
5535or NULL is returned if the connection failed.
5433 5536
5434To interact with NAMESTORE clients first connect to the NAMESTORE service using 5537To disconnect from NAMESTORE, clients use
5435the @code{GNUNET_NAMESTORE_connect} passing a configuration handle. As a result 5538@code{GNUNET_NAMESTORE_disconnect} and specify the handle to disconnect.
5436they obtain a NAMESTORE handle, they can use for operations, or NULL is
5437returned if the connection failed.
5438
5439To disconnect from NAMESTORE, clients use @code{GNUNET_NAMESTORE_disconnect}
5440and specify the handle to disconnect.
5441 5539
5442NAMESTORE internally uses the ECDSA private key to refer to zones. These 5540NAMESTORE internally uses the ECDSA private key to refer to zones. These
5443private keys can be obtained from the IDENTITY subsytem. Here @emph{egos@emph{ 5541private keys can be obtained from the IDENTITY subsytem.
5444can be used to refer to zones or the default ego assigned to the GNS subsystem 5542Here @emph{egos} @emph{can be used to refer to zones or the default ego
5445can be used to obtained the master zone's private key.}} 5543assigned to the GNS subsystem can be used to obtained the master zone's
5544private key.}
5446 5545
5447 5546
5448@menu 5547@menu
@@ -5456,92 +5555,104 @@ can be used to obtained the master zone's private key.}}
5456 5555
5457@c %**end of header 5556@c %**end of header
5458 5557
5459NAMESTORE provides functions to lookup records stored under a label in a zone 5558NAMESTORE provides functions to lookup records stored under a label in a
5460and to store records under a label in a zone. 5559zone and to store records under a label in a zone.
5461 5560
5462To store (and delete) records, the client uses the 5561To store (and delete) records, the client uses the
5463@code{GNUNET_NAMESTORE_records_store} function and has to provide namestore 5562@code{GNUNET_NAMESTORE_records_store} function and has to provide
5464handle to use, the private key of the zone, the label to store the records 5563namestore handle to use, the private key of the zone, the label to store
5465under, the records and number of records plus an callback function. After the 5564the records under, the records and number of records plus an callback
5466operation is performed NAMESTORE will call the provided callback function with 5565function.
5467the result GNUNET_SYSERR on failure (including timeout/queue drop/failure to 5566After the operation is performed NAMESTORE will call the provided
5468validate), GNUNET_NO if content was already there or not found GNUNET_YES (or 5567callback function with the result GNUNET_SYSERR on failure
5469other positive value) on success plus an additional error message. 5568(including timeout/queue drop/failure to validate), GNUNET_NO if content
5470 5569was already there or not found GNUNET_YES (or other positive value) on
5471Records are deleted by using the store command with 0 records to store. It is 5570success plus an additional error message.
5472important to note, that records are not merged when records exist with the 5571
5473label. So a client has first to retrieve records, merge with existing records 5572Records are deleted by using the store command with 0 records to store.
5573It is important to note, that records are not merged when records exist
5574with the label.
5575So a client has first to retrieve records, merge with existing records
5474and then store the result. 5576and then store the result.
5475 5577
5476To perform a lookup operation, the client uses the 5578To perform a lookup operation, the client uses the
5477@code{GNUNET_NAMESTORE_records_store} function. Here he has to pass the 5579@code{GNUNET_NAMESTORE_records_store} function. Here he has to pass the
5478namestore handle, the private key of the zone and the label. He also has to 5580namestore handle, the private key of the zone and the label. He also has
5479provide a callback function which will be called with the result of the lookup 5581to provide a callback function which will be called with the result of
5480operation: the zone for the records, the label, and the records including the 5582the lookup operation:
5583the zone for the records, the label, and the records including the
5481number of records included. 5584number of records included.
5482 5585
5483A special operation is used to set the preferred nickname for a zone. This 5586A special operation is used to set the preferred nickname for a zone.
5484nickname is stored with the zone and is automatically merged with all labels 5587This nickname is stored with the zone and is automatically merged with
5485and records stored in a zone. Here the client uses the 5588all labels and records stored in a zone. Here the client uses the
5486@code{GNUNET_NAMESTORE_set_nick} function and passes the private key of the 5589@code{GNUNET_NAMESTORE_set_nick} function and passes the private key of
5487zone, the nickname as string plus a the callback with the result of the 5590the zone, the nickname as string plus a the callback with the result of
5488operation. 5591the operation.
5489 5592
5490@node Iterating Zone Information 5593@node Iterating Zone Information
5491@subsubsection Iterating Zone Information 5594@subsubsection Iterating Zone Information
5492 5595
5493@c %**end of header 5596@c %**end of header
5494 5597
5495A client can iterate over all information in a zone or all zones managed by 5598A client can iterate over all information in a zone or all zones managed
5496NAMESTORE. Here a client uses the @code{GNUNET_NAMESTORE_zone_iteration_start} 5599by NAMESTORE.
5600Here a client uses the @code{GNUNET_NAMESTORE_zone_iteration_start}
5497function and passes the namestore handle, the zone to iterate over and a 5601function and passes the namestore handle, the zone to iterate over and a
5498callback function to call with the result. If the client wants to iterate over 5602callback function to call with the result.
5499all the, he passes NULL for the zone. A @code{GNUNET_NAMESTORE_ZoneIterator} 5603If the client wants to iterate over all the, he passes NULL for the zone.
5500handle is returned to be used to continue iteration. 5604A @code{GNUNET_NAMESTORE_ZoneIterator} handle is returned to be used to
5605continue iteration.
5501 5606
5502NAMESTORE calls the callback for every result and expects the client to call@ 5607NAMESTORE calls the callback for every result and expects the client to
5503@code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or 5608call @code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or
5504@code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration. When 5609@code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration.
5505NAMESTORE reached the last item it will call the callback with a NULL value to 5610When NAMESTORE reached the last item it will call the callback with a
5506indicate. 5611NULL value to indicate.
5507 5612
5508@node Monitoring Zone Information 5613@node Monitoring Zone Information
5509@subsubsection Monitoring Zone Information 5614@subsubsection Monitoring Zone Information
5510 5615
5511@c %**end of header 5616@c %**end of header
5512 5617
5513Clients can also monitor zones to be notified about changes. Here the clients 5618Clients can also monitor zones to be notified about changes. Here the
5514uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and passes the 5619clients uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and
5515private key of the zone and and a callback function to call with updates for a 5620passes the private key of the zone and and a callback function to call
5516zone. The client can specify to obtain zone information first by iterating over 5621with updates for a zone.
5517the zone and specify a synchronization callback to be called when the client 5622The client can specify to obtain zone information first by iterating over
5518and the namestore are synced. 5623the zone and specify a synchronization callback to be called when the
5624client and the namestore are synced.
5519 5625
5520On an update, NAMESTORE will call the callback with the private key of the 5626On an update, NAMESTORE will call the callback with the private key of the
5521zone, the label and the records and their number. 5627zone, the label and the records and their number.
5522 5628
5523To stop monitoring, the client call @code{GNUNET_NAMESTORE_zone_monitor_stop} 5629To stop monitoring, the client calls
5524and passes the handle obtained from the function to start the monitoring. 5630@code{GNUNET_NAMESTORE_zone_monitor_stop} and passes the handle obtained
5631from the function to start the monitoring.
5525 5632
5633@cindex PEERINFO
5634@cindex peerinfo subsystem
5526@node GNUnet's PEERINFO subsystem 5635@node GNUnet's PEERINFO subsystem
5527@section GNUnet's PEERINFO subsystem 5636@section GNUnet's PEERINFO subsystem
5528 5637
5529@c %**end of header 5638@c %**end of header
5530 5639
5531The PEERINFO subsystem is used to store verified (validated) information about 5640The PEERINFO subsystem is used to store verified (validated) information
5532known peers in a persistent way. It obtains these addresses for example from 5641about known peers in a persistent way. It obtains these addresses for
5533TRANSPORT service which is in charge of address validation. Validation means 5642example from TRANSPORT service which is in charge of address validation.
5534that the information in the HELLO message are checked by connecting to the 5643Validation means that the information in the HELLO message are checked by
5535addresses and performing a cryptographic handshake to authenticate the peer 5644connecting to the addresses and performing a cryptographic handshake to
5536instance stating to be reachable with these addresses. Peerinfo does not 5645authenticate the peer instance stating to be reachable with these
5537validate the HELLO messages itself but only stores them and gives them to 5646addresses.
5538interested clients. 5647Peerinfo does not validate the HELLO messages itself but only stores them
5648and gives them to interested clients.
5539 5649
5540As future work, we think about moving from storing just HELLO messages to 5650As future work, we think about moving from storing just HELLO messages to
5541providing a generic persistent per-peer information store. More and more 5651providing a generic persistent per-peer information store.
5542subsystems tend to need to store per-peer information in persistent way. To not 5652More and more subsystems tend to need to store per-peer information in
5543duplicate this functionality we plan to provide a PEERSTORE service providing 5653persistent way.
5544this functionality 5654To not duplicate this functionality we plan to provide a PEERSTORE
5655service providing this functionality.
5545 5656
5546@menu 5657@menu
5547* Features2:: 5658* Features2::
@@ -7636,37 +7747,47 @@ The names for the master directories follow the names of the operations:
7636@end itemize 7747@end itemize
7637 7748
7638Each of the master directories contains names (chosen at random) for each 7749Each of the master directories contains names (chosen at random) for each
7639active top-level (master) operation. Note that a download that is associated 7750active top-level (master) operation.
7640with a search result is not a top-level operation. 7751Note that a download that is associated with a search result is not a
7641 7752top-level operation.
7642In contrast to the master directories, the child directories are only consulted 7753
7643when another operation refers to them. For each search, a subdirectory (named 7754In contrast to the master directories, the child directories are only
7644after the master search synchronization file) contains the search results. 7755consulted when another operation refers to them.
7645Search results can have an associated download, which is then stored in the 7756For each search, a subdirectory (named after the master search
7646general "download-child" directory. Downloads can be recursive, in which case 7757synchronization file) contains the search results.
7647children are stored in subdirectories mirroring the structure of the recursive 7758Search results can have an associated download, which is then stored in
7648download (either starting in the master "download" directory or in the 7759the general "download-child" directory.
7649"download-child" directory depending on how the download was initiated). For 7760Downloads can be recursive, in which case children are stored in
7650publishing operations, the "publish-file" directory contains information about 7761subdirectories mirroring the structure of the recursive download
7651the individual files and directories that are part of the publication. However, 7762(either starting in the master "download" directory or in the
7652this directory structure is flat and does not mirror the structure of the 7763"download-child" directory depending on how the download was initiated).
7653publishing operation. Note that unindex operations cannot have associated child 7764For publishing operations, the "publish-file" directory contains
7654operations. 7765information about the individual files and directories that are part of
7655 7766the publication.
7767However, this directory structure is flat and does not mirror the
7768structure of the publishing operation.
7769Note that unindex operations cannot have associated child operations.
7770
7771@cindex REGEX subsystem
7772@cindex regex subsystem
7656@node GNUnet's REGEX Subsystem 7773@node GNUnet's REGEX Subsystem
7657@section GNUnet's REGEX Subsystem 7774@section GNUnet's REGEX Subsystem
7658 7775
7659@c %**end of header 7776@c %**end of header
7660 7777
7661Using the REGEX subsystem, you can discover peers that offer a particular 7778Using the REGEX subsystem, you can discover peers that offer a particular
7662service using regular expressions. The peers that offer a service specify it 7779service using regular expressions.
7663using a regular expressions. Peers that want to patronize a service search 7780The peers that offer a service specify it using a regular expressions.
7664using a string. The REGEX subsystem will then use the DHT to return a set of 7781Peers that want to patronize a service search using a string.
7665matching offerers to the patrons. 7782The REGEX subsystem will then use the DHT to return a set of matching
7783offerers to the patrons.
7784
7785For the technical details, we have Max's defense talk and Max's Master's
7786thesis.
7666 7787
7667For the technical details, we have "Max's defense talk and Max's Master's 7788@c An additional publication is under preparation and available to
7668thesis. An additional publication is under preparation and available to team 7789@c team members (in Git).
7669members (in Git). 7790@c FIXME: Where is the file? Point to it. Assuming that it's szengel2012ms
7670 7791
7671@menu 7792@menu
7672* How to run the regex profiler:: 7793* How to run the regex profiler::
@@ -7677,32 +7798,38 @@ members (in Git).
7677 7798
7678@c %**end of header 7799@c %**end of header
7679 7800
7680The gnunet-regex-profiler can be used to profile the usage of mesh/regex for a 7801The gnunet-regex-profiler can be used to profile the usage of mesh/regex
7681given set of regular expressions and strings. Mesh/regex allows you to announce 7802for a given set of regular expressions and strings.
7682your peer ID under a certain regex and search for peers matching a particular 7803Mesh/regex allows you to announce your peer ID under a certain regex and
7683regex using a string. See https://gnunet.org/szengel2012ms for a full 7804search for peers matching a particular regex using a string.
7805See @uref{https://gnunet.org/szengel2012ms, szengel2012ms} for a full
7684introduction. 7806introduction.
7685 7807
7686First of all, the regex profiler uses GNUnet testbed, thus all the implications 7808First of all, the regex profiler uses GNUnet testbed, thus all the
7687for testbed also apply to the regex profiler (for example you need 7809implications for testbed also apply to the regex profiler
7688password-less ssh login to the machines listed in your hosts file). 7810(for example you need password-less ssh login to the machines listed in
7811your hosts file).
7689 7812
7690@strong{Configuration} 7813@strong{Configuration}
7691 7814
7692Moreover, an appropriate configuration file is needed. Generally you can refer 7815Moreover, an appropriate configuration file is needed.
7693to SVN HEAD: contrib/regex_profiler_infiniband.conf for an example 7816Generally you can refer to the
7694configuration. In the following paragraph the important details are 7817@file{contrib/regex_profiler_infiniband.conf} file in the sourcecode
7695highlighted. 7818of GNUnet for an example configuration.
7819In the following paragraph the important details are highlighted.
7696 7820
7697Announcing of the regular expressions is done by the 7821Announcing of the regular expressions is done by the
7698gnunet-daemon-regexprofiler, therefore you have to make sure it is started, by 7822gnunet-daemon-regexprofiler, therefore you have to make sure it is
7699adding it to the AUTOSTART set of ARM:@ 7823started, by adding it to the AUTOSTART set of ARM:
7700@code{
7701[regexprofiler]@
7702AUTOSTART = YES@
7703}
7704 7824
7825@example
7826[regexprofiler]
7827AUTOSTART = YES
7828@end example
7829
7830@noindent
7705Furthermore you have to specify the location of the binary: 7831Furthermore you have to specify the location of the binary:
7832
7706@example 7833@example
7707[regexprofiler] 7834[regexprofiler]
7708# Location of the gnunet-daemon-regexprofiler binary. 7835# Location of the gnunet-daemon-regexprofiler binary.
@@ -7712,58 +7839,88 @@ BINARY = /home/szengel/gnunet/src/mesh/.libs/gnunet-daemon-regexprofiler
7712REGEX_PREFIX = "GNVPN-0001-PAD" 7839REGEX_PREFIX = "GNVPN-0001-PAD"
7713@end example 7840@end example
7714 7841
7715When running the profiler with a large scale deployment, you probably want to 7842@noindent
7716reduce the workload of each peer. Use the following options to do this.@ 7843When running the profiler with a large scale deployment, you probably
7844want to reduce the workload of each peer.
7845Use the following options to do this.
7846
7717@example 7847@example
7718[dht]@ 7848[dht]
7719# Force network size estimation@ 7849# Force network size estimation
7720FORCE_NSE = 1 7850FORCE_NSE = 1
7721 7851
7722[dhtcache] 7852[dhtcache]
7723DATABASE = heap@ 7853DATABASE = heap
7724# Disable RC-file for Bloom filter? (for benchmarking with limited IO 7854# Disable RC-file for Bloom filter? (for benchmarking with limited IO
7725# availability)@ 7855# availability)
7726DISABLE_BF_RC = YES@ 7856DISABLE_BF_RC = YES
7727# Disable Bloom filter entirely@ 7857# Disable Bloom filter entirely
7728DISABLE_BF = YES 7858DISABLE_BF = YES
7729 7859
7730[nse]@ 7860[nse]
7731# Minimize proof-of-work CPU consumption by NSE@ 7861# Minimize proof-of-work CPU consumption by NSE
7732WORKBITS = 1 7862WORKBITS = 1
7733@end example 7863@end example
7734 7864
7735 7865@noindent
7736@strong{Options} 7866@strong{Options}
7737 7867
7738To finally run the profiler some options and the input data need to be 7868To finally run the profiler some options and the input data need to be
7739specified on the command line. 7869specified on the command line.
7740@code{@ gnunet-regex-profiler -c config-file -d 7870
7741log-file -n num-links -p@ path-compression-length -s search-delay -t 7871@example
7742matching-timeout -a num-search-strings hosts-file policy-dir 7872gnunet-regex-profiler -c config-file -d log-file -n num-links \
7743search-strings-file@ } 7873-p path-compression-length -s search-delay -t matching-timeout \
7744 7874-a num-search-strings hosts-file policy-dir search-strings-file
7745@code{config-file} the configuration file created earlier.@ @code{log-file} 7875@end example
7746file where to write statistics output.@ @code{num-links} number of random links 7876
7747between started peers.@ @code{path-compression-length} maximum path compression 7877@noindent
7748length in the DFA.@ @code{search-delay} time to wait between peers finished 7878Where...
7749linking and@ starting to match strings.@ @code{matching-timeout} timeout after 7879
7750witch to cancel the searching.@ @code{num-search-strings} number of strings in 7880@itemize @bullet
7751the search-strings-file. 7881@item ... @code{config-file} means the configuration file created earlier.
7752 7882@item ... @code{log-file} is the file where to write statistics output.
7753The @code{hosts-file} should contain a list of hosts for the testbed, one per 7883@item ... @code{num-links} indicates the number of random links between
7754line in the following format. @code{user@@host_ip:port}. 7884started peers.
7755 7885@item ... @code{path-compression-length} is the maximum path compression
7756The @code{policy-dir} is a folder containing text files containing one or more 7886length in the DFA.
7757regular expressions. A peer is started for each file in that folder and the 7887@item ... @code{search-delay} time to wait between peers finished linking
7758regular expressions in the corresponding file are announced by this peer. 7888and starting to match strings.
7759 7889@item ... @code{matching-timeout} timeout after which to cancel the
7760The @code{search-strings-file} is a text file containing search strings, one in 7890searching.
7761each line. 7891@item ... @code{num-search-strings} number of strings in the
7762 7892search-strings-file.
7763You can create regular expressions and search strings for every AS in the@ 7893@item ... the @code{hosts-file} should contain a list of hosts for the
7894testbed, one per line in the following format:
7895
7896@itemize @bullet
7897@item @code{user@@host_ip:port}
7898@end itemize
7899@item ... the @code{policy-dir} is a folder containing text files
7900containing one or more regular expressions. A peer is started for each
7901file in that folder and the regular expressions in the corresponding file
7902are announced by this peer.
7903@item ... the @code{search-strings-file} is a text file containing search
7904strings, one in each line.
7905@end itemize
7906
7907@noindent
7908You can create regular expressions and search strings for every AS in the
7764Internet using the attached scripts. You need one of the 7909Internet using the attached scripts. You need one of the
7765@uref{http://data.caida.org/datasets/routing/routeviews-prefix2as/, CAIDA 7910@uref{http://data.caida.org/datasets/routing/routeviews-prefix2as/, CAIDA
7766routeviews prefix2as} data files for this. Run @code{create_regex.py <filename> 7911routeviews prefix2as} data files for this. Run
7767<output path>} to create the regular expressions and @code{create_strings.py 7912
7768<input path> <outfile>} to create a search strings file from the previously 7913@example
7769created regular expressions. 7914create_regex.py <filename> <output path>
7915@end example
7916
7917@noindent
7918to create the regular expressions and
7919
7920@example
7921create_strings.py <input path> <outfile>
7922@end example
7923
7924@noindent
7925to create a search strings file from the previously created
7926regular expressions.