commit e7d0874d9b3107577b8014865d06de0a4534ab9e parent b6a02e6e7742b575d96ba4f665bec00613467d70 Author: Martin Schanzenbach <schanzen@gnunet.org> Date: Fri, 6 Oct 2023 22:32:58 +0200 start separating service API from subsystem description Diffstat:
60 files changed, 1681 insertions(+), 1638 deletions(-)
diff --git a/developers/cadet/cadet.rst b/developers/apis/cadet.rst diff --git a/developers/apis/core.rst b/developers/apis/core.rst @@ -0,0 +1,283 @@ +.. _CORE-Subsystem-Dev: + +.. index:: + double: CORE; subsystem + +CORE +==== + +The CORE API (defined in ``gnunet_core_service.h``) is the basic +messaging API used by P2P applications built using GNUnet. It provides +applications the ability to send and receive encrypted messages to the +peer's \"directly\" connected neighbours. + +As CORE connections are generally \"direct\" connections, applications +must not assume that they can connect to arbitrary peers this way, as +\"direct\" connections may not always be possible. Applications using +CORE are notified about which peers are connected. Creating new +\"direct\" connections must be done using the TRANSPORT API. + +The CORE API provides unreliable, out-of-order delivery. While the +implementation tries to ensure timely, in-order delivery, both message +losses and reordering are not detected and must be tolerated by the +application. Most important, the core will NOT perform retransmission if +messages could not be delivered. + +Note that CORE allows applications to queue one message per connected +peer. The rate at which each connection operates is influenced by the +preferences expressed by local application as well as restrictions +imposed by the other peer. Local applications can express their +preferences for particular connections using the \"performance\" API of +the ATS service. + +Applications that require more sophisticated transmission capabilities +such as TCP-like behavior, or if you intend to send messages to +arbitrary remote peers, should use the CADET API. + +The typical use of the CORE API is to connect to the CORE service using +``GNUNET_CORE_connect``, process events from the CORE service (such as +peers connecting, peers disconnecting and incoming messages) and send +messages to connected peers using ``GNUNET_CORE_notify_transmit_ready``. +Note that applications must cancel pending transmission requests if they +receive a disconnect event for a peer that had a transmission pending; +furthermore, queuing more than one transmission request per peer per +application using the service is not permitted. + +The CORE API also allows applications to monitor all communications of +the peer prior to encryption (for outgoing messages) or after decryption +(for incoming messages). This can be useful for debugging, diagnostics +or to establish the presence of cover traffic (for anonymity). As +monitoring applications are often not interested in the payload, the +monitoring callbacks can be configured to only provide the message +headers (including the message type and size) instead of copying the +full data stream to the monitoring client. + +The init callback of the ``GNUNET_CORE_connect`` function is called with +the hash of the public key of the peer. This public key is used to +identify the peer globally in the GNUnet network. Applications are +encouraged to check that the provided hash matches the hash that they +are using (as theoretically the application may be using a different +configuration file with a different private key, which would result in +hard to find bugs). + +As with most service APIs, the CORE API isolates applications from +crashes of the CORE service. If the CORE service crashes, the +application will see disconnect events for all existing connections. +Once the connections are re-established, the applications will be +receive matching connect events. + +core client-service protocol +.. _The-CORE-Client_002dService-Protocol: + +The CORE Client-Service Protocol +-------------------------------- + +This section describes the protocol between an application using the +CORE service (the client) and the CORE service process itself. + +.. _Setup2: + +Setup2 +^^^^^^ + +When a client connects to the CORE service, it first sends a +``InitMessage`` which specifies options for the connection and a set of +message type values which are supported by the application. The options +bitmask specifies which events the client would like to be notified +about. The options include: + +**GNUNET_CORE_OPTION_NOTHING** + No notifications + +**GNUNET_CORE_OPTION_STATUS_CHANGE** + Peers connecting and disconnecting + +**GNUNET_CORE_OPTION_FULL_INBOUND** + All inbound messages (after decryption) with full payload + +**GNUNET_CORE_OPTION_HDR_INBOUND** + Just the ``MessageHeader`` of all inbound messages + +**GNUNET_CORE_OPTION_FULL_OUTBOUND** + All outbound messages (prior to encryption) with full payload + +**GNUNET_CORE_OPTION_HDR_OUTBOUND** + Just the ``MessageHeader`` of all outbound messages + +Typical applications will only monitor for connection status changes. + +The CORE service responds to the ``InitMessage`` with an +``InitReplyMessage`` which contains the peer's identity. Afterwards, +both CORE and the client can send messages. + +.. _Notifications: + +Notifications +^^^^^^^^^^^^^ + +The CORE will send ``ConnectNotifyMessage``\ s and +``DisconnectNotifyMessage``\ s whenever peers connect or disconnect from +the CORE (assuming their type maps overlap with the message types +registered by the client). When the CORE receives a message that matches +the set of message types specified during the ``InitMessage`` (or if +monitoring is enabled in for inbound messages in the options), it sends +a ``NotifyTrafficMessage`` with the peer identity of the sender and the +decrypted payload. The same message format (except with +``GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND`` for the message type) is +used to notify clients monitoring outbound messages; here, the peer +identity given is that of the receiver. + +.. _Sending: + +Sending +^^^^^^^ + +When a client wants to transmit a message, it first requests a +transmission slot by sending a ``SendMessageRequest`` which specifies +the priority, deadline and size of the message. Note that these values +may be ignored by CORE. When CORE is ready for the message, it answers +with a ``SendMessageReady`` response. The client can then transmit the +payload with a ``SendMessage`` message. Note that the actual message +size in the ``SendMessage`` is allowed to be smaller than the size in +the original request. A client may at any time send a fresh +``SendMessageRequest``, which then superceeds the previous +``SendMessageRequest``, which is then no longer valid. The client can +tell which ``SendMessageRequest`` the CORE service's +``SendMessageReady`` message is for as all of these messages contain a +\"unique\" request ID (based on a counter incremented by the client for +each request). + +CORE Peer-to-Peer Protocol +.. _The-CORE-Peer_002dto_002dPeer-Protocol: + +The CORE Peer-to-Peer Protocol +------------------------------ + +EphemeralKeyMessage creation +.. _Creating-the-EphemeralKeyMessage: + +Creating the EphemeralKeyMessage +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When the CORE service starts, each peer creates a fresh ephemeral (ECC) +public-private key pair and signs the corresponding +``EphemeralKeyMessage`` with its long-term key (which we usually call +the peer's identity; the hash of the public long term key is what +results in a ``struct GNUNET_PeerIdentity`` in all GNUnet APIs. The +ephemeral key is ONLY used for an ECDHE (`Elliptic-curve +Diffie---Hellman <http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman>`__) +exchange by the CORE service to establish symmetric session keys. A peer +will use the same ``EphemeralKeyMessage`` for all peers for +``REKEY_FREQUENCY``, which is usually 12 hours. After that time, it will +create a fresh ephemeral key (forgetting the old one) and broadcast the +new ``EphemeralKeyMessage`` to all connected peers, resulting in fresh +symmetric session keys. Note that peers independently decide on when to +discard ephemeral keys; it is not a protocol violation to discard keys +more often. Ephemeral keys are also never stored to disk; restarting a +peer will thus always create a fresh ephemeral key. The use of ephemeral +keys is what provides `forward +secrecy <http://en.wikipedia.org/wiki/Forward_secrecy>`__. + +Just before transmission, the ``EphemeralKeyMessage`` is patched to +reflect the current sender_status, which specifies the current state of +the connection from the point of view of the sender. The possible values +are: + +- ``KX_STATE_DOWN`` Initial value, never used on the network + +- ``KX_STATE_KEY_SENT`` We sent our ephemeral key, do not know the key + of the other peer + +- ``KX_STATE_KEY_RECEIVED`` This peer has received a valid ephemeral + key of the other peer, but we are waiting for the other peer to + confirm it's authenticity (ability to decode) via challenge-response. + +- ``KX_STATE_UP`` The connection is fully up from the point of view of + the sender (now performing keep-alive) + +- ``KX_STATE_REKEY_SENT`` The sender has initiated a rekeying + operation; the other peer has so far failed to confirm a working + connection using the new ephemeral key + +.. _Establishing-a-connection: + +Establishing a connection +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Peers begin their interaction by sending a ``EphemeralKeyMessage`` to +the other peer once the TRANSPORT service notifies the CORE service +about the connection. A peer receiving an ``EphemeralKeyMessage`` with a +status indicating that the sender does not have the receiver's ephemeral +key, the receiver's ``EphemeralKeyMessage`` is sent in response. +Additionally, if the receiver has not yet confirmed the authenticity of +the sender, it also sends an (encrypted)\ ``PingMessage`` with a +challenge (and the identity of the target) to the other peer. Peers +receiving a ``PingMessage`` respond with an (encrypted) ``PongMessage`` +which includes the challenge. Peers receiving a ``PongMessage`` check +the challenge, and if it matches set the connection to ``KX_STATE_UP``. + +.. _Encryption-and-Decryption: + +Encryption and Decryption +^^^^^^^^^^^^^^^^^^^^^^^^^ + +All functions related to the key exchange and encryption/decryption of +messages can be found in ``gnunet-service-core_kx.c`` (except for the +cryptographic primitives, which are in ``util/crypto*.c``). Given the +key material from ECDHE, a Key derivation function (`Key derivation +function <https://en.wikipedia.org/wiki/Key_derivation_function>`__) is +used to derive two pairs of encryption and decryption keys for AES-256 +and TwoFish, as well as initialization vectors and authentication keys +(for HMAC (`HMAC <https://en.wikipedia.org/wiki/HMAC>`__)). The HMAC is +computed over the encrypted payload. Encrypted messages include an +iv_seed and the HMAC in the header. + +Each encrypted message in the CORE service includes a sequence number +and a timestamp in the encrypted payload. The CORE service remembers the +largest observed sequence number and a bit-mask which represents which +of the previous 32 sequence numbers were already used. Messages with +sequence numbers lower than the largest observed sequence number minus +32 are discarded. Messages with a timestamp that is less than +``REKEY_TOLERANCE`` off (5 minutes) are also discarded. This of course +means that system clocks need to be reasonably synchronized for peers to +be able to communicate. Additionally, as the ephemeral key changes every +12 hours, a peer would not even be able to decrypt messages older than +12 hours. + +.. _Type-maps: + +Type maps +^^^^^^^^^ + +Once an encrypted connection has been established, peers begin to +exchange type maps. Type maps are used to allow the CORE service to +determine which (encrypted) connections should be shown to which +applications. A type map is an array of 65536 bits representing the +different types of messages understood by applications using the CORE +service. Each CORE service maintains this map, simply by setting the +respective bit for each message type supported by any of the +applications using the CORE service. Note that bits for message types +embedded in higher-level protocols (such as MESH) will not be included +in these type maps. + +Typically, the type map of a peer will be sparse. Thus, the CORE service +attempts to compress its type map using ``gzip``-style compression +(\"deflate\") prior to transmission. However, if the compression fails +to compact the map, the map may also be transmitted without compression +(resulting in ``GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP`` or +``GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP`` messages respectively). +Upon receiving a type map, the respective CORE service notifies +applications about the connection to the other peer if they support any +message type indicated in the type map (or no message type at all). If +the CORE service experience a connect or disconnect event from an +application, it updates its type map (setting or unsetting the +respective bits) and notifies its neighbours about the change. The CORE +services of the neighbours then in turn generate connect and disconnect +events for the peer that sent the type map for their respective +applications. As CORE messages may be lost, the CORE service confirms +receiving a type map by sending back a +``GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP``. If such a confirmation +(with the correct hash of the type map) is not received, the sender will +retransmit the type map (with exponential back-off). + + diff --git a/developers/apis/dht.rst b/developers/apis/dht.rst @@ -0,0 +1,354 @@ +.. _Distributed-Hash-Table-Dev: + +.. index:: + double: Distributed hash table; subsystem + see: DHT; Distributed hash table + +DHT +=== + +The API of libgnunetblock +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The block library requires for each (family of) block type(s) a block +plugin (implementing ``gnunet_block_plugin.h``) that provides basic +functions that are needed by the DHT (and possibly other subsystems) to +manage the block. These block plugins are typically implemented within +their respective subsystems. The main block library is then used to +locate, load and query the appropriate block plugin. Which plugin is +appropriate is determined by the block type (which is just a 32-bit +integer). Block plugins contain code that specifies which block types +are supported by a given plugin. The block library loads all block +plugins that are installed at the local peer and forwards the +application request to the respective plugin. + +The central functions of the block APIs (plugin and main library) are to +allow the mapping of blocks to their respective key (if possible) and +the ability to check that a block is well-formed and matches a given +request (again, if possible). This way, GNUnet can avoid storing invalid +blocks, storing blocks under the wrong key and forwarding blocks in +response to a query that they do not answer. + +One key function of block plugins is that it allows GNUnet to detect +duplicate replies (via the Bloom filter). All plugins MUST support +detecting duplicate replies (by adding the current response to the Bloom +filter and rejecting it if it is encountered again). If a plugin fails +to do this, responses may loop in the network. + +.. _Queries: + +Queries +^^^^^^^ + +The query format for any block in GNUnet consists of four main +components. First, the type of the desired block must be specified. +Second, the query must contain a hash code. The hash code is used for +lookups in hash tables and databases and must not be unique for the +block (however, if possible a unique hash should be used as this would +be best for performance). Third, an optional Bloom filter can be +specified to exclude known results; replies that hash to the bits set in +the Bloom filter are considered invalid. False-positives can be +eliminated by sending the same query again with a different Bloom filter +mutator value, which parametrizes the hash function that is used. +Finally, an optional application-specific \"eXtended query\" (xquery) +can be specified to further constrain the results. It is entirely up to +the type-specific plugin to determine whether or not a given block +matches a query (type, hash, Bloom filter, and xquery). Naturally, not +all xquery's are valid and some types of blocks may not support Bloom +filters either, so the plugin also needs to check if the query is valid +in the first place. + +Depending on the results from the plugin, the DHT will then discard the +(invalid) query, forward the query, discard the (invalid) reply, cache +the (valid) reply, and/or forward the (valid and non-duplicate) reply. + +.. _Sample-Code: + +Sample Code +^^^^^^^^^^^ + +The source code in **plugin_block_test.c** is a good starting point for +new block plugins --- it does the minimal work by implementing a plugin +that performs no validation at all. The respective **Makefile.am** shows +how to build and install a block plugin. + +.. _Conclusion2: + +Conclusion2 +^^^^^^^^^^^ + +In conclusion, GNUnet subsystems that want to use the DHT need to define +a block format and write a plugin to match queries and replies. For +testing, the ``GNUNET_BLOCK_TYPE_TEST`` block type can be used; it +accepts any query as valid and any reply as matching any query. This +type is also used for the DHT command line tools. However, it should NOT +be used for normal applications due to the lack of error checking that +results from this primitive implementation. + +libgnunetdht +:index:`libgnunetdht <single: libgnunet; dht>` +libgnunetdht +------------ + +The DHT API itself is pretty simple and offers the usual GET and PUT +functions that work as expected. The specified block type refers to the +block library which allows the DHT to run application-specific logic for +data stored in the network. + +.. _GET: + +GET +^^^ + +When using GET, the main consideration for developers (other than the +block library) should be that after issuing a GET, the DHT will +continuously cause (small amounts of) network traffic until the +operation is explicitly canceled. So GET does not simply send out a +single network request once; instead, the DHT will continue to search +for data. This is needed to achieve good success rates and also handles +the case where the respective PUT operation happens after the GET +operation was started. Developers should not cancel an existing GET +operation and then explicitly re-start it to trigger a new round of +network requests; this is simply inefficient, especially as the internal +automated version can be more efficient, for example by filtering +results in the network that have already been returned. + +If an application that performs a GET request has a set of replies that +it already knows and would like to filter, it can +call ``GNUNET_DHT_get_filter_known_results`` with an array of hashes +over the respective blocks to tell the DHT that these results are not +desired (any more). This way, the DHT will filter the respective blocks +using the block library in the network, which may result in a +significant reduction in bandwidth consumption. + +.. _PUT: + +PUT +^^^ + +.. todo:: inconsistent use of "must" above it's written "MUST" + +In contrast to GET operations, developers **must** manually re-run PUT +operations periodically (if they intend the content to continue to be +available). Content stored in the DHT expires or might be lost due to +churn. Furthermore, GNUnet's DHT typically requires multiple rounds of +PUT operations before a key-value pair is consistently available to all +peers (the DHT randomizes paths and thus storage locations, and only +after multiple rounds of PUTs there will be a sufficient number of +replicas in large DHTs). An explicit PUT operation using the DHT API +will only cause network traffic once, so in order to ensure basic +availability and resistance to churn (and adversaries), PUTs must be +repeated. While the exact frequency depends on the application, a rule +of thumb is that there should be at least a dozen PUT operations within +the content lifetime. Content in the DHT typically expires after one +day, so DHT PUT operations should be repeated at least every 1-2 hours. + +.. _MONITOR: + +MONITOR +^^^^^^^ + +The DHT API also allows applications to monitor messages crossing the +local DHT service. The types of messages used by the DHT are GET, PUT +and RESULT messages. Using the monitoring API, applications can choose +to monitor these requests, possibly limiting themselves to requests for +a particular block type. + +The monitoring API is not only useful for diagnostics, it can also be +used to trigger application operations based on PUT operations. For +example, an application may use PUTs to distribute work requests to +other peers. The workers would then monitor for PUTs that give them +work, instead of looking for work using GET operations. This can be +beneficial, especially if the workers have no good way to guess the keys +under which work would be stored. Naturally, additional protocols might +be needed to ensure that the desired number of workers will process the +distributed workload. + +.. _DHT-Routing-Options: + +DHT Routing Options +^^^^^^^^^^^^^^^^^^^ + +There are two important options for GET and PUT requests: + +GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all + peers should process the request, even if their peer ID is not + closest to the key. For a PUT request, this means that all peers that + a request traverses may make a copy of the data. Similarly for a GET + request, all peers will check their local database for a result. + Setting this option can thus significantly improve caching and reduce + bandwidth consumption --- at the expense of a larger DHT database. If + in doubt, we recommend that this option should be used. + +GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record + the path that a GET or a PUT request is taking through the overlay + network. The resulting paths are then returned to the application + with the respective result. This allows the receiver of a result to + construct a path to the originator of the data, which might then be + used for routing. Naturally, setting this option requires additional + bandwidth and disk space, so applications should only set this if the + paths are needed by the application logic. + +GNUNET_DHT_RO_FIND_PEER This option is an internal option used by + the DHT's peer discovery mechanism and should not be used by + applications. + +GNUNET_DHT_RO_BART This option is currently not implemented. It may + in the future offer performance improvements for clique topologies. + +.. _The-DHT-Client_002dService-Protocol: + +The DHT Client-Service Protocol +------------------------------- + +.. _PUTting-data-into-the-DHT: + +PUTting data into the DHT +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To store (PUT) data into the DHT, the client sends a +``struct GNUNET_DHT_ClientPutMessage`` to the service. This message +specifies the block type, routing options, the desired replication +level, the expiration time, key, value and a 64-bit unique ID for the +operation. The service responds with a +``struct GNUNET_DHT_ClientPutConfirmationMessage`` with the same 64-bit +unique ID. Note that the service sends the confirmation as soon as it +has locally processed the PUT request. The PUT may still be propagating +through the network at this time. + +In the future, we may want to change this to provide (limited) feedback +to the client, for example if we detect that the PUT operation had no +effect because the same key-value pair was already stored in the DHT. +However, changing this would also require additional state and messages +in the P2P interaction. + +.. _GETting-data-from-the-DHT: + +GETting data from the DHT +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To retrieve (GET) data from the DHT, the client sends a +``struct GNUNET_DHT_ClientGetMessage`` to the service. The message +specifies routing options, a replication level (for replicating the GET, +not the content), the desired block type, the key, the (optional) +extended query and unique 64-bit request ID. + +Additionally, the client may send any number of +``struct GNUNET_DHT_ClientGetResultSeenMessage``\ s to notify the +service about results that the client is already aware of. These +messages consist of the key, the unique 64-bit ID of the request, and an +arbitrary number of hash codes over the blocks that the client is +already aware of. As messages are restricted to 64k, a client that +already knows more than about a thousand blocks may need to send several +of these messages. Naturally, the client should transmit these messages +as quickly as possible after the original GET request such that the DHT +can filter those results in the network early on. Naturally, as these +messages are sent after the original request, it is conceivable that the +DHT service may return blocks that match those already known to the +client anyway. + +In response to a GET request, the service will send ``struct +GNUNET_DHT_ClientResultMessage``\ s to the client. These messages +contain the block type, expiration, key, unique ID of the request and of +course the value (a block). Depending on the options set for the +respective operations, the replies may also contain the path the GET +and/or the PUT took through the network. + +A client can stop receiving replies either by disconnecting or by +sending a ``struct GNUNET_DHT_ClientGetStopMessage`` which must contain +the key and the 64-bit unique ID of the original request. Using an +explicit \"stop\" message is more common as this allows a client to run +many concurrent GET operations over the same connection with the DHT +service --- and to stop them individually. + +.. _Monitoring-DHT: + +Monitoring the DHT +^^^^^^^^^^^^^^^^^^ + +To begin monitoring, the client sends a +``struct GNUNET_DHT_MonitorStartStop`` message to the DHT service. In +this message, flags can be set to enable (or disable) monitoring of GET, +PUT and RESULT messages that pass through a peer. The message can also +restrict monitoring to a particular block type or a particular key. Once +monitoring is enabled, the DHT service will notify the client about any +matching event using ``struct GNUNET_DHT_MonitorGetMessage``\ s for GET +events, ``struct GNUNET_DHT_MonitorPutMessage`` for PUT events and +``struct GNUNET_DHT_MonitorGetRespMessage`` for RESULTs. Each of these +messages contains all of the information about the event. + +.. _The-DHT-Peer_002dto_002dPeer-Protocol: + +The DHT Peer-to-Peer Protocol +----------------------------- + +.. _Routing-GETs-or-PUTs: + +Routing GETs or PUTs +^^^^^^^^^^^^^^^^^^^^ + +When routing GETs or PUTs, the DHT service selects a suitable subset of +neighbours for forwarding. The exact number of neighbours can be zero or +more and depends on the hop counter of the query (initially zero) in +relation to the (log of) the network size estimate, the desired +replication level and the peer's connectivity. Depending on the hop +counter and our network size estimate, the selection of the peers maybe +randomized or by proximity to the key. Furthermore, requests include a +set of peers that a request has already traversed; those peers are also +excluded from the selection. + +.. _PUTting-data-into-the-DHT2: + +PUTting data into the DHT +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To PUT data into the DHT, the service sends a ``struct PeerPutMessage`` +of type ``GNUNET_MESSAGE_TYPE_DHT_P2P_PUT`` to the respective neighbour. +In addition to the usual information about the content (type, routing +options, desired replication level for the content, expiration time, key +and value), the message contains a fixed-size Bloom filter with +information about which peers (may) have already seen this request. This +Bloom filter is used to ensure that DHT messages never loop back to a +peer that has already processed the request. Additionally, the message +includes the current hop counter and, depending on the routing options, +the message may include the full path that the message has taken so far. +The Bloom filter should already contain the identity of the previous +hop; however, the path should not include the identity of the previous +hop and the receiver should append the identity of the sender to the +path, not its own identity (this is done to reduce bandwidth). + +.. _GETting-data-from-the-DHT2: + +GETting data from the DHT +^^^^^^^^^^^^^^^^^^^^^^^^^ + +A peer can search the DHT by sending ``struct PeerGetMessage``\ s of +type ``GNUNET_MESSAGE_TYPE_DHT_P2P_GET`` to other peers. In addition to +the usual information about the request (type, routing options, desired +replication level for the request, the key and the extended query), a +GET request also contains a hop counter, a Bloom filter over the peers +that have processed the request already and depending on the routing +options the full path traversed by the GET. Finally, a GET request +includes a variable-size second Bloom filter and a so-called Bloom +filter mutator value which together indicate which replies the sender +has already seen. During the lookup, each block that matches they block +type, key and extended query is additionally subjected to a test against +this Bloom filter. The block plugin is expected to take the hash of the +block and combine it with the mutator value and check if the result is +not yet in the Bloom filter. The originator of the query will from time +to time modify the mutator to (eventually) allow false-positives +filtered by the Bloom filter to be returned. + +Peers that receive a GET request perform a local lookup (depending on +their proximity to the key and the query options) and forward the +request to other peers. They then remember the request (including the +Bloom filter for blocking duplicate results) and when they obtain a +matching, non-filtered response a ``struct PeerResultMessage`` of type +``GNUNET_MESSAGE_TYPE_DHT_P2P_RESULT`` is forwarded to the previous hop. +Whenever a result is forwarded, the block plugin is used to update the +Bloom filter accordingly, to ensure that the same result is never +forwarded more than once. The DHT service may also cache forwarded +results locally if the \"CACHE_RESULTS\" option is set to \"YES\" in the +configuration. + +.. [R5N2011] https://bib.gnunet.org/date.html#R5N diff --git a/developers/apis/fs.rst b/developers/apis/fs.rst @@ -0,0 +1,129 @@ +.. index:: + double: File sharing; subsystem + see: FS; File sharing + +.. _FS-Subsystem-Dev: + +FS +== + +.. _Namespace-Advertisements: + +Namespace Advertisements +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. todo:: FIXME: all zeroses -> ? + +An ``SBlock`` with identifier all zeros is a signed advertisement for a +namespace. This special ``SBlock`` contains metadata describing the +content of the namespace. Instead of the name of the identifier for a +potential update, it contains the identifier for the root of the +namespace. The URI should always be empty. The ``SBlock`` is signed with +the content provider's RSA private key (just like any other SBlock). +Peers can search for ``SBlock``\ s in order to find out more about a +namespace. + +.. _KSBlocks: + +KSBlocks +^^^^^^^^ + +GNUnet implements ``KSBlocks`` which are ``KBlocks`` that, instead of +encrypting a CHK and metadata, encrypt an ``SBlock`` instead. In other +words, ``KSBlocks`` enable GNUnet to find ``SBlocks`` using the global +keyword search. Usually the encrypted ``SBlock`` is a namespace +advertisement. The rationale behind ``KSBlock``\ s and ``SBlock``\ s is +to enable peers to discover namespaces via keyword searches, and, to +associate useful information with namespaces. When GNUnet finds +``KSBlocks`` during a normal keyword search, it adds the information to +an internal list of discovered namespaces. Users looking for interesting +namespaces can then inspect this list, reducing the need for out-of-band +discovery of namespaces. Naturally, namespaces (or more specifically, +namespace advertisements) can also be referenced from directories, but +``KSBlock``\ s should make it easier to advertise namespaces for the +owner of the pseudonym since they eliminate the need to first create a +directory. + +Collections are also advertised using ``KSBlock``\ s. + +.. https://old.gnunet.org/sites/default/files/ecrs.pdf +.. What is this? - WGL + +.. _File_002dsharing-persistence-directory-structure: + +File-sharing persistence directory structure +-------------------------------------------- + +This section documents how the file-sharing library implements +persistence of file-sharing operations and specifically the resulting +directory structure. This code is only active if the +``GNUNET_FS_FLAGS_PERSISTENCE`` flag was set when calling +``GNUNET_FS_start``. In this case, the file-sharing library will try +hard to ensure that all major operations (searching, downloading, +publishing, unindexing) are persistent, that is, can live longer than +the process itself. More specifically, an operation is supposed to live +until it is explicitly stopped. + +If ``GNUNET_FS_stop`` is called before an operation has been stopped, a +``SUSPEND`` event is generated and then when the process calls +``GNUNET_FS_start`` next time, a ``RESUME`` event is generated. +Additionally, even if an application crashes (segfault, SIGKILL, system +crash) and hence ``GNUNET_FS_stop`` is never called and no ``SUSPEND`` +events are generated, operations are still resumed (with ``RESUME`` +events). This is implemented by constantly writing the current state of +the file-sharing operations to disk. Specifically, the current state is +always written to disk whenever anything significant changes (the +exception are block-wise progress in publishing and unindexing, since +those operations would be slowed down significantly and can be resumed +cheaply even without detailed accounting). Note that if the process +crashes (or is killed) during a serialization operation, FS does not +guarantee that this specific operation is recoverable (no strict +transactional semantics, again for performance reasons). However, all +other unrelated operations should resume nicely. + +Since we need to serialize the state continuously and want to recover as +much as possible even after crashing during a serialization operation, +we do not use one large file for serialization. Instead, several +directories are used for the various operations. When +``GNUNET_FS_start`` executes, the master directories are scanned for +files describing operations to resume. Sometimes, these operations can +refer to related operations in child directories which may also be +resumed at this point. Note that corrupted files are cleaned up +automatically. However, dangling files in child directories (those that +are not referenced by files from the master directories) are not +automatically removed. + +Persistence data is kept in a directory that begins with the +\"STATE_DIR\" prefix from the configuration file (by default, +\"$SERVICEHOME/persistence/\") followed by the name of the client as +given to ``GNUNET_FS_start`` (for example, \"gnunet-gtk\") followed by +the actual name of the master or child directory. + +The names for the master directories follow the names of the operations: + +- \"search\" + +- \"download\" + +- \"publish\" + +- \"unindex\" + +Each of the master directories contains names (chosen at random) for +each active top-level (master) operation. Note that a download that is +associated with a search result is not a top-level operation. + +In contrast to the master directories, the child directories are only +consulted when another operation refers to them. For each search, a +subdirectory (named after the master search synchronization file) +contains the search results. Search results can have an associated +download, which is then stored in the general \"download-child\" +directory. Downloads can be recursive, in which case children are stored +in subdirectories mirroring the structure of the recursive download +(either starting in the master \"download\" directory or in the +\"download-child\" directory depending on how the download was +initiated). For publishing operations, the \"publish-file\" directory +contains information about the individual files and directories that are +part of the publication. However, this directory structure is flat and +does not mirror the structure of the publishing operation. Note that +unindex operations cannot have associated child operations. diff --git a/developers/apis/gns.rst b/developers/apis/gns.rst @@ -0,0 +1,576 @@ +.. index:: + double: GNU Name System; subsystem + see: GNS; GNU Name System + +.. _GNU-Name-System-Dev: + +GNS — the GNU Name System +========================= + +libgnunetgns +------------ + +The GNS API itself is extremely simple. Clients first connect to the GNS +service using ``GNUNET_GNS_connect``. They can then perform lookups +using ``GNUNET_GNS_lookup`` or cancel pending lookups using +``GNUNET_GNS_lookup_cancel``. Once finished, clients disconnect using +``GNUNET_GNS_disconnect``. + +.. _Looking-up-records: + +Looking up records +^^^^^^^^^^^^^^^^^^ + +``GNUNET_GNS_lookup`` takes a number of arguments: + +handle This is simply the GNS connection handle from + ``GNUNET_GNS_connect``. + +name The client needs to specify the name to + be resolved. This can be any valid DNS or GNS hostname. + +zone The client + needs to specify the public key of the GNS zone against which the + resolution should be done. Note that a key must be provided, the + client should look up plausible values using its configuration, the + identity service and by attempting to interpret the TLD as a + base32-encoded public key. + +type This is the desired GNS or DNS record type + to look for. While all records for the given name will be returned, + this can be important if the client wants to resolve record types + that themselves delegate resolution, such as CNAME, PKEY or GNS2DNS. + Resolving a record of any of these types will only work if the + respective record type is specified in the request, as the GNS + resolver will otherwise follow the delegation and return the records + from the respective destination, instead of the delegating record. + +only_cached This argument should typically be set to + ``GNUNET_NO``. Setting it to ``GNUNET_YES`` disables resolution via + the overlay network. + +shorten_zone_key If GNS encounters new names during resolution, + their respective zones can automatically be learned and added to the + \"shorten zone\". If this is desired, clients must pass the private + key of the shorten zone. If NULL is passed, shortening is disabled. + +proc This argument identifies + the function to call with the result. It is given proc_cls, the + number of records found (possibly zero) and the array of the records + as arguments. proc will only be called once. After proc,> has been + called, the lookup must no longer be canceled. + +proc_cls The closure for proc. + +.. _Accessing-the-records: + +Accessing the records +^^^^^^^^^^^^^^^^^^^^^ + +The ``libgnunetgnsrecord`` library provides an API to manipulate the GNS +record array that is given to proc. In particular, it offers functions +such as converting record values to human-readable strings (and back). +However, most ``libgnunetgnsrecord`` functions are not interesting to +GNS client applications. + +For DNS records, the ``libgnunetdnsparser`` library provides functions +for parsing (and serializing) common types of DNS records. + +.. _Creating-records: + +Creating records +^^^^^^^^^^^^^^^^ + +Creating GNS records is typically done by building the respective record +information (possibly with the help of ``libgnunetgnsrecord`` and +``libgnunetdnsparser``) and then using the ``libgnunetnamestore`` to +publish the information. The GNS API is not involved in this operation. + +.. _Future-work: + +Future work +^^^^^^^^^^^ + +In the future, we want to expand ``libgnunetgns`` to allow applications +to observe shortening operations performed during GNS resolution, for +example so that users can receive visual feedback when this happens. + +libgnunetgnsrecord +~~~~~~~~~~~~~~~~~~ + +The ``libgnunetgnsrecord`` library is used to manipulate GNS records (in +plaintext or in their encrypted format). Applications mostly interact +with ``libgnunetgnsrecord`` by using the functions to convert GNS record +values to strings or vice-versa, or to lookup a GNS record type number +by name (or vice-versa). The library also provides various other +functions that are mostly used internally within GNS, such as converting +keys to names, checking for expiration, encrypting GNS records to GNS +blocks, verifying GNS block signatures and decrypting GNS records from +GNS blocks. + +We will now discuss the four commonly used functions of the +API. ``libgnunetgnsrecord`` does not perform these operations itself, +but instead uses plugins to perform the operation. GNUnet includes +plugins to support common DNS record types as well as standard GNS +record types. + +.. _Value-handling: + +Value handling +^^^^^^^^^^^^^^ + +``GNUNET_GNSRECORD_value_to_string`` can be used to convert the (binary) +representation of a GNS record value to a human readable, 0-terminated +UTF-8 string. NULL is returned if the specified record type is not +supported by any available plugin. + +``GNUNET_GNSRECORD_string_to_value`` can be used to try to convert a +human readable string to the respective (binary) representation of a GNS +record value. + +.. _Type-handling: + +Type handling +^^^^^^^^^^^^^ + +``GNUNET_GNSRECORD_typename_to_number`` can be used to obtain the +numeric value associated with a given typename. For example, given the +typename \"A\" (for DNS A reocrds), the function will return the number +1. A list of common DNS record types is +`here <http://en.wikipedia.org/wiki/List_of_DNS_record_types>`__. Note +that not all DNS record types are supported by GNUnet GNSRECORD plugins +at this time. + +``GNUNET_GNSRECORD_number_to_typename`` can be used to obtain the +typename associated with a given numeric value. For example, given the +type number 1, the function will return the typename \"A\". + +.. _GNS-plugins: + +GNS plugins +~~~~~~~~~~~ + +Adding a new GNS record type typically involves writing (or extending) a +GNSRECORD plugin. The plugin needs to implement the +``gnunet_gnsrecord_plugin.h`` API which provides basic functions that +are needed by GNSRECORD to convert typenames and values of the +respective record type to strings (and back). These gnsrecord plugins +are typically implemented within their respective subsystems. Examples +for such plugins can be found in the GNSRECORD, GNS and CONVERSATION +subsystems. + +The ``libgnunetgnsrecord`` library is then used to locate, load and +query the appropriate gnsrecord plugin. Which plugin is appropriate is +determined by the record type (which is just a 32-bit integer). The +``libgnunetgnsrecord`` library loads all block plugins that are +installed at the local peer and forwards the application request to the +plugins. If the record type is not supported by the plugin, it should +simply return an error code. + +The central functions of the block APIs (plugin and main library) are +the same four functions for converting between values and strings, and +typenames and numbers documented in the previous subsection. + +.. _The-GNS-Client_002dService-Protocol: + +The GNS Client-Service Protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The GNS client-service protocol consists of two simple messages, the +``LOOKUP`` message and the ``LOOKUP_RESULT``. Each ``LOOKUP`` message +contains a unique 32-bit identifier, which will be included in the +corresponding response. Thus, clients can send many lookup requests in +parallel and receive responses out-of-order. A ``LOOKUP`` request also +includes the public key of the GNS zone, the desired record type and +fields specifying whether shortening is enabled or networking is +disabled. Finally, the ``LOOKUP`` message includes the name to be +resolved. + +The response includes the number of records and the records themselves +in the format created by ``GNUNET_GNSRECORD_records_serialize``. They +can thus be deserialized using ``GNUNET_GNSRECORD_records_deserialize``. + +.. _Hijacking-the-DNS_002dTraffic-using-gnunet_002dservice_002ddns: + +Hijacking the DNS-Traffic using gnunet-service-dns +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section documents how the gnunet-service-dns (and the +gnunet-helper-dns) intercepts DNS queries from the local system. This is +merely one method for how we can obtain GNS queries. It is also possible +to change ``resolv.conf`` to point to a machine running +``gnunet-dns2gns`` or to modify libc's name system switch (NSS) +configuration to include a GNS resolution plugin. The method described +in this chapter is more of a last-ditch catch-all approach. + +``gnunet-service-dns`` enables intercepting DNS traffic using policy +based routing. We MARK every outgoing DNS-packet if it was not sent by +our application. Using a second routing table in the Linux kernel these +marked packets are then routed through our virtual network interface and +can thus be captured unchanged. + +Our application then reads the query and decides how to handle it. If +the query can be addressed via GNS, it is passed to +``gnunet-service-gns`` and resolved internally using GNS. In the future, +a reverse query for an address of the configured virtual network could +be answered with records kept about previous forward queries. Queries +that are not hijacked by some application using the DNS service will be +sent to the original recipient. The answer to the query will always be +sent back through the virtual interface with the original nameserver as +source address. + +.. _Network-Setup-Details: + +Network Setup Details +^^^^^^^^^^^^^^^^^^^^^ + +The DNS interceptor adds the following rules to the Linux kernel: + +:: + + iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \ + -j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \ + --set-mark 3 ip rule add fwmark 3 table2 ip route add default via \ + $VIRTUALDNS table2 + +.. todo:: + FIXME: Rewrite to reflect display which is no longer content + by line due to the < 74 characters limit. + +Line 1 makes sure that all packets coming from a port our application +opened beforehand (``$LOCALPORT``) will be routed normally. Line 2 marks +every other packet to a DNS-Server with mark 3 (chosen arbitrarily). The +third line adds a routing policy based on this mark 3 via the routing +table. + +.. _Importing-DNS-Zones-into-GNS: + +Importing DNS Zones into GNS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section discusses the challenges and problems faced when writing +the Ascension tool. It also takes a look at possible improvements in the +future. + +Consider the following diagram that shows the workflow of Ascension: + +|ascension| + +Further the interaction between components of GNUnet are shown in the +diagram below: + +DNS Conversion +.. _Conversions-between-DNS-and-GNS: + +Conversions between DNS and GNS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The differences between the two name systems lies in the details and is +not always transparent. For instance an SRV record is converted to a BOX +record which is unique to GNS. + +This is done by converting to a BOX record from an existing SRV record: + +:: + + # SRV + # _service._proto.name. TTL class SRV priority weight port target + _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com. + # BOX + # TTL BOX flags port protocol recordtype priority weight port target + 14000 BOX n 5060 6 33 0 0 5060 www.example.com + +Other records that need to undergo such transformation is the MX record +type, as well as the SOA record type. + +Transformation of a SOA record into GNS works as described in the +following example. Very important to note are the rname and mname keys. + +:: + + # BIND syntax for a clean SOA record + IN SOA master.example.com. hostmaster.example.com. ( + 2017030300 ; serial + 3600 ; refresh + 1800 ; retry + 604800 ; expire + 600 ) ; ttl + # Recordline for adding the record + $ gnunet-namestore -z example.com -a -n -t SOA -V \ + rname=master.example.com mname=hostmaster.example.com \ + 2017030300,3600,1800,604800,600 -e 7200s + +The transformation of MX records is done in a simple way. + +:: + + # mail.example.com. 3600 IN MX 10 mail.example.com. + $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail + +Finally, one of the biggest struggling points were the NS records that +are found in top level domain zones. The intended behaviour for those is +to add GNS2DNS records for those so that gnunet-gns can resolve records +for those domains on its own. Those require the values from DNS GLUE +records, provided they are within the same zone. + +The following two examples show one record with a GLUE record and the +other one does not have a GLUE record. This takes place in the 'com' +TLD. + +.. code-block:: shell + + # ns1.example.com 86400 IN A 127.0.0.1 + # example.com 86400 IN NS ns1.example.com. + $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ + example.com@127.0.0.1 + + # example.com 86400 IN NS ns1.example.org. + $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ + example.com@ns1.example.org + +As you can see, one of the GNS2DNS records has an IP address listed and +the other one a DNS name. For the first one there is a GLUE record to do +the translation directly and the second one will issue another DNS query +to figure out the IP of ns1.example.org. + +A solution was found by creating a hierarchical zone structure in GNS +and linking the zones using PKEY records to one another. This allows the +resolution of the name servers to work within GNS while not taking +control over unwanted zones. + +Currently the following record types are supported: + +- A + +- AAAA + +- CNAME + +- MX + +- NS + +- SRV + +- TXT + +This is not due to technical limitations but rather a practical ones. +The problem occurs with DNSSEC enabled DNS zones. As records within +those zones are signed periodically, and every new signature is an +update to the zone, there are many revisions of zones. This results in a +problem with bigger zones as there are lots of records that have been +signed again but no major changes. Also trying to add records that are +unknown that require a different format take time as they cause a CLI +call of the namestore. Furthermore certain record types need +transformation into a GNS compatible format which, depending on the +record type, takes more time. + +Further a blacklist was added to drop for instance DNSSEC related +records. Also if a record type is neither in the white list nor the +blacklist it is considered as a loss of data and a message is shown to +the user. This helps with transparency and also with contributing, as +the not supported record types can then be added accordingly. + +.. _DNS-Zone-Size: + +DNS Zone Size +^^^^^^^^^^^^^ + +Another very big problem exists with very large zones. When migrating a +small zone the delay between adding of records and their expiry is +negligible. However when working with big zones that easily have more +than a few million records this delay becomes a problem. + +Records will start to expire well before the zone has finished +migrating. This is usually not a problem but can cause a high CPU load +when a peer is restarted and the records have expired. + +A good solution has not been found yet. One of the idea that floated +around was that the records should be added with the s (shadow) flag to +keep the records resolvable even if they expired. However this would +introduce the problem of how to detect if a record has been removed from +the zone and would require deletion of said record(s). + +Another problem that still persists is how to refresh records. Expired +records are still displayed when calling gnunet-namestore but do not +resolve with gnunet-gns. Zonemaster will sign the expired records again +and make sure that the records are still valid. With a recent change +this was fixed as gnunet-gns to improve the suffix lookup which allows +for a fast lookup even with thousands of local egos. + +Currently the pace of adding records in general is around 10 records per +second. Crypto is the upper limit for adding of records. The performance +of your machine can be tested with the perf_crypto\_\* tools. There is +still a big discrepancy between the pace of Ascension and the +theoretical limit. + +A performance metric for measuring improvements has not yet been +implemented in Ascension. + +.. _Performance: + +Performance +^^^^^^^^^^^ + +The performance when migrating a zone using the Ascension tool is +limited by a handful of factors. First of all ascension is written in +Python3 and calls the CLI tools of GNUnet. This is comparable to a fork +and exec call which costs a few CPU cycles. Furthermore all the records +that are added to the same label are signed using the zones private key. +This signing operation is very resource heavy and was optimized during +development by adding the '-R' (Recordline) option to gnunet-namestore +which allows to specify multiple records using the CLI tool. Assuming +that in a TLD zone every domain has at least two name servers this +halves the amount of signatures needed. + +Another improvement that could be made is with the addition of multiple +threads or using asynchronous subprocesses when opening the GNUnet CLI +tools. This could be implemented by simply creating more workers in the +program but performance improvements were not tested. + +Ascension was tested using different hardware and database backends. +Performance differences between SQLite and postgresql are marginal and +almost non existent. What did make a huge impact on record adding +performance was the storage medium. On a traditional mechanical hard +drive adding of records were slow compared to a solid state disk. + +In conclusion there are many bottlenecks still around in the program, +namely the single threaded implementation and inefficient, sequential +calls of gnunet-namestore. In the future a solution that uses the C API +would be cleaner and better. + +.. _Registering-names-using-the-FCFS-daemon: + +Registering names using the FCFS daemon +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section describes FCFSD, a daemon used to associate names with PKEY +records following a "First Come, First Served" policy. This policy means +that a certain name can not be registered again if someone registered it +already. + +The daemon can be started by using ``gnunet-namestore-fcfsd``, which +will start a simple HTTP server on localhost, using a port specified by +the ``HTTPORT`` value in its configuration. + +Communication is performed by sending GET or POST requests to specific +paths ("endpoints"), as described in the following sections. + +The daemon will always respond with data structured using the JSON +format. The fields to be expected will be listed for each endpoint. + +The only exceptions are for the "root" endpoint (i.e. ``/``) which will +return a HTML document, and two other HTML documents which will be +served when certain errors are encountered, like when requesting an +unknown endpoint. + +FCFSD GET requests +.. _Obtaining-information-from-the-daemon: + +Obtaining information from the daemon +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To query the daemon, a GET request must be sent to these endpoints, +placing parameters in the address as per the HTTP specification, like +so: + +:: + + GET /endpoint?param1=value¶m2=value + +Each endpoint will be described using its name (``/endpoint`` in the +example above), followed by the name of each parameter (like ``param1`` +and ``param2``.) + + +**Endpoint: /search** :bolditalic:`name` + This endpoint is used to query about the state of <name>, that is, + whether it is available for registration or not. + + The response JSON will contain two fields: + + - error + + - free + + ``error`` can be either the string ``"true"`` or the string + ``"false"``: when ``"true"``, it means there was an error within the + daemon and the name could not be searched at all. + + ``free`` can be either the string ``"true"`` or the string + ``"false"``: when ``"true"``, the requested name can be registered. + +FCFSD POST requests +.. _Submitting-data-to-the-daemon: + +Submitting data to the daemon +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To send data to the daemon, a POST request must be sent to these +endpoints, placing the data to submit in the body of the request, +structured using the JSON format, like so: + +:: + + POST /endpoint + Content-Type: application/json + ... + + {"param1": value1, "param2": value2, ...} + +Each endpoint will be described using its name (``/endpoint`` in the +example above), followed by the name of each JSON field (like ``param1`` +and ``param2``.) + +**Endpoint: /register** :bolditalic:`name key` + This endpoint is used to register a new association between <name> + and <key>. + + For this operation to succeed, both <NAME> and <KEY> **must not** be + registered already. + + The response JSON will contain two fields: + + - error + + - message + + ``error`` can be either the string ``"true"`` or the string + ``"false"``: when ``"true"``, it means the name could not be + registered. Clients can get the reason of the failure from the HTTP + response code or from the ``message`` field. + + ``message`` is a string which can be used by clients to let users + know the result of the operation. It might be localized to the daemon + operator's locale. + +.. _Customizing-the-HTML-output: + +Customizing the HTML output +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In some situations, the daemon will serve HTML documents instead of JSON +values. It is possible to configure the daemon to serve custom documents +instead of the ones provided with GNUnet, by setting the ``HTMLDIR`` +value in its configuration to a directory path. + +Within the provided path, the daemon will search for these three files: + +- fcfsd-index.html + +- fcfsd-notfound.html + +- fcfsd-forbidden.html + +The ``fcfsd-index.html`` file is the daemon's "homepage": operators +might want to provide information about the service here, or provide a +form with which it is possible to register a name. + +The ``fcfsd-notfound.html`` file is used primarily to let users know +they tried to access an unknown endpoint. + +The ``fcfsd-forbidden.html`` file is served to users when they try to +access an endpoint they should not access. For example, sending an +invalid request might result in this page being served. + +.. |ascension| image:: /images/ascension_ssd.png diff --git a/developers/identity/identity.rst b/developers/apis/identity.rst diff --git a/developers/apis/index.rst b/developers/apis/index.rst @@ -0,0 +1,30 @@ +Service APIs +============ + +These services comprise a backbone of core services for +peer-to-peer applications to use. + +.. toctree:: + cadet.rst + core.rst + dht.rst + fs.rst + gns.rst + identity.rst + messenger.rst + namecache.rst + namestore.rst + nse.rst + peerinfo.rst + peerstore.rst + regex.rst + rest.rst + revocation.rst + rps.rst + setops.rst + statistics.rst + transport-ng.rst + transport.rst + vpnstack.rst + + diff --git a/developers/messenger/messenger.rst b/developers/apis/messenger.rst diff --git a/developers/namecache/namecache.rst b/developers/apis/namecache.rst diff --git a/developers/namestore/namestore.rst b/developers/apis/namestore.rst diff --git a/developers/nse/nse.rst b/developers/apis/nse.rst diff --git a/developers/peerinfo/peerinfo.rst b/developers/apis/peerinfo.rst diff --git a/developers/peerstore/peerstore.rst b/developers/apis/peerstore.rst diff --git a/developers/regex/regex.rst b/developers/apis/regex.rst diff --git a/developers/rest/rest.rst b/developers/apis/rest.rst diff --git a/developers/revocation/revocation.rst b/developers/apis/revocation.rst diff --git a/developers/rps/rps.rst b/developers/apis/rps.rst diff --git a/developers/set/set.rst b/developers/apis/set/set.rst diff --git a/developers/seti/seti.rst b/developers/apis/seti/seti.rst diff --git a/developers/setops.rst b/developers/apis/setops.rst diff --git a/developers/setu/setu.rst b/developers/apis/setu/setu.rst diff --git a/developers/statistics/statistics.rst b/developers/apis/statistics.rst diff --git a/developers/transport-ng/transport-ng.rst b/developers/apis/transport-ng.rst diff --git a/developers/transport/transport.rst b/developers/apis/transport.rst diff --git a/developers/vpnstack.rst b/developers/apis/vpnstack.rst diff --git a/developers/core/core.rst b/developers/core/core.rst @@ -1,387 +0,0 @@ -.. _CORE-Subsystem: - -.. index:: - double: CORE; subsystem - -CORE — GNUnet link layer -======================== - -The CORE subsystem in GNUnet is responsible for securing link-layer -communications between nodes in the GNUnet overlay network. CORE builds -on the TRANSPORT subsystem which provides for the actual, insecure, -unreliable link-layer communication (for example, via UDP or WLAN), and -then adds fundamental security to the connections: - -- confidentiality with so-called perfect forward secrecy; we use ECDHE - (`Elliptic-curve - Diffie—Hellman <http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman>`__) - powered by Curve25519 (`Curve25519 <http://cr.yp.to/ecdh.html>`__) - for the key exchange and then use symmetric encryption, encrypting - with both AES-256 - (`AES-256 <http://en.wikipedia.org/wiki/Rijndael>`__) and Twofish - (`Twofish <http://en.wikipedia.org/wiki/Twofish>`__) - -- `authentication <http://en.wikipedia.org/wiki/Authentication>`__ is - achieved by signing the ephemeral keys using Ed25519 - (`Ed25519 <http://ed25519.cr.yp.to/>`__), a deterministic variant of - ECDSA (`ECDSA <http://en.wikipedia.org/wiki/ECDSA>`__) - -- integrity protection (using SHA-512 - (`SHA-512 <http://en.wikipedia.org/wiki/SHA-2>`__) to do - encrypt-then-MAC - (`encrypt-then-MAC <http://en.wikipedia.org/wiki/Authenticated_encryption>`__)) - -- Replay (`replay <http://en.wikipedia.org/wiki/Replay_attack>`__) - protection (using nonces, timestamps, challenge-response, message - counters and ephemeral keys) - -- liveness (keep-alive messages, timeout) - -.. _Limitations: - -:index:`Limitations <CORE; limitations>` -Limitations ------------ - -CORE does not perform -`routing <http://en.wikipedia.org/wiki/Routing>`__; using CORE it is -only possible to communicate with peers that happen to already be -\"directly\" connected with each other. CORE also does not have an API -to allow applications to establish such \"direct\" connections --- for -this, applications can ask TRANSPORT, but TRANSPORT might not be able to -establish a \"direct\" connection. The TOPOLOGY subsystem is responsible -for trying to keep a few \"direct\" connections open at all times. -Applications that need to talk to particular peers should use the CADET -subsystem, as it can establish arbitrary \"indirect\" connections. - -Because CORE does not perform routing, CORE must only be used directly -by applications that either perform their own routing logic (such as -anonymous file-sharing) or that do not require routing, for example -because they are based on flooding the network. CORE communication is -unreliable and delivery is possibly out-of-order. Applications that -require reliable communication should use the CADET service. Each -application can only queue one message per target peer with the CORE -service at any time; messages cannot be larger than approximately 63 -kilobytes. If messages are small, CORE may group multiple messages -(possibly from different applications) prior to encryption. If permitted -by the application (using the `cork <http://baus.net/on-tcp_cork/>`__ -option), CORE may delay transmissions to facilitate grouping of multiple -small messages. If cork is not enabled, CORE will transmit the message -as soon as TRANSPORT allows it (TRANSPORT is responsible for limiting -bandwidth and congestion control). CORE does not allow flow control; -applications are expected to process messages at line-speed. If flow -control is needed, applications should use the CADET service. - -.. when is a peer connected -.. _When-is-a-peer-_0022connected_0022_003f: - -When is a peer \"connected\"? ------------------------------ - -In addition to the security features mentioned above, CORE also provides -one additional key feature to applications using it, and that is a -limited form of protocol-compatibility checking. CORE distinguishes -between TRANSPORT-level connections (which enable communication with -other peers) and application-level connections. Applications using the -CORE API will (typically) learn about application-level connections from -CORE, and not about TRANSPORT-level connections. When a typical -application uses CORE, it will specify a set of message types (from -``gnunet_protocols.h``) that it understands. CORE will then notify the -application about connections it has with other peers if and only if -those applications registered an intersecting set of message types with -their CORE service. Thus, it is quite possible that CORE only exposes a -subset of the established direct connections to a particular application ---- and different applications running above CORE might see different -sets of connections at the same time. - -A special case are applications that do not register a handler for any -message type. CORE assumes that these applications merely want to -monitor connections (or \"all\" messages via other callbacks) and will -notify those applications about all connections. This is used, for -example, by the ``gnunet-core`` command-line tool to display the active -connections. Note that it is also possible that the TRANSPORT service -has more active connections than the CORE service, as the CORE service -first has to perform a key exchange with connecting peers before -exchanging information about supported message types and notifying -applications about the new connection. - - -:index:`libgnunetcore <libgnunet; core>` -libgnunetcore -------------- - -The CORE API (defined in ``gnunet_core_service.h``) is the basic -messaging API used by P2P applications built using GNUnet. It provides -applications the ability to send and receive encrypted messages to the -peer's \"directly\" connected neighbours. - -As CORE connections are generally \"direct\" connections, applications -must not assume that they can connect to arbitrary peers this way, as -\"direct\" connections may not always be possible. Applications using -CORE are notified about which peers are connected. Creating new -\"direct\" connections must be done using the TRANSPORT API. - -The CORE API provides unreliable, out-of-order delivery. While the -implementation tries to ensure timely, in-order delivery, both message -losses and reordering are not detected and must be tolerated by the -application. Most important, the core will NOT perform retransmission if -messages could not be delivered. - -Note that CORE allows applications to queue one message per connected -peer. The rate at which each connection operates is influenced by the -preferences expressed by local application as well as restrictions -imposed by the other peer. Local applications can express their -preferences for particular connections using the \"performance\" API of -the ATS service. - -Applications that require more sophisticated transmission capabilities -such as TCP-like behavior, or if you intend to send messages to -arbitrary remote peers, should use the CADET API. - -The typical use of the CORE API is to connect to the CORE service using -``GNUNET_CORE_connect``, process events from the CORE service (such as -peers connecting, peers disconnecting and incoming messages) and send -messages to connected peers using ``GNUNET_CORE_notify_transmit_ready``. -Note that applications must cancel pending transmission requests if they -receive a disconnect event for a peer that had a transmission pending; -furthermore, queuing more than one transmission request per peer per -application using the service is not permitted. - -The CORE API also allows applications to monitor all communications of -the peer prior to encryption (for outgoing messages) or after decryption -(for incoming messages). This can be useful for debugging, diagnostics -or to establish the presence of cover traffic (for anonymity). As -monitoring applications are often not interested in the payload, the -monitoring callbacks can be configured to only provide the message -headers (including the message type and size) instead of copying the -full data stream to the monitoring client. - -The init callback of the ``GNUNET_CORE_connect`` function is called with -the hash of the public key of the peer. This public key is used to -identify the peer globally in the GNUnet network. Applications are -encouraged to check that the provided hash matches the hash that they -are using (as theoretically the application may be using a different -configuration file with a different private key, which would result in -hard to find bugs). - -As with most service APIs, the CORE API isolates applications from -crashes of the CORE service. If the CORE service crashes, the -application will see disconnect events for all existing connections. -Once the connections are re-established, the applications will be -receive matching connect events. - -core client-service protocol -.. _The-CORE-Client_002dService-Protocol: - -The CORE Client-Service Protocol --------------------------------- - -This section describes the protocol between an application using the -CORE service (the client) and the CORE service process itself. - -.. _Setup2: - -Setup2 -^^^^^^ - -When a client connects to the CORE service, it first sends a -``InitMessage`` which specifies options for the connection and a set of -message type values which are supported by the application. The options -bitmask specifies which events the client would like to be notified -about. The options include: - -**GNUNET_CORE_OPTION_NOTHING** - No notifications - -**GNUNET_CORE_OPTION_STATUS_CHANGE** - Peers connecting and disconnecting - -**GNUNET_CORE_OPTION_FULL_INBOUND** - All inbound messages (after decryption) with full payload - -**GNUNET_CORE_OPTION_HDR_INBOUND** - Just the ``MessageHeader`` of all inbound messages - -**GNUNET_CORE_OPTION_FULL_OUTBOUND** - All outbound messages (prior to encryption) with full payload - -**GNUNET_CORE_OPTION_HDR_OUTBOUND** - Just the ``MessageHeader`` of all outbound messages - -Typical applications will only monitor for connection status changes. - -The CORE service responds to the ``InitMessage`` with an -``InitReplyMessage`` which contains the peer's identity. Afterwards, -both CORE and the client can send messages. - -.. _Notifications: - -Notifications -^^^^^^^^^^^^^ - -The CORE will send ``ConnectNotifyMessage``\ s and -``DisconnectNotifyMessage``\ s whenever peers connect or disconnect from -the CORE (assuming their type maps overlap with the message types -registered by the client). When the CORE receives a message that matches -the set of message types specified during the ``InitMessage`` (or if -monitoring is enabled in for inbound messages in the options), it sends -a ``NotifyTrafficMessage`` with the peer identity of the sender and the -decrypted payload. The same message format (except with -``GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND`` for the message type) is -used to notify clients monitoring outbound messages; here, the peer -identity given is that of the receiver. - -.. _Sending: - -Sending -^^^^^^^ - -When a client wants to transmit a message, it first requests a -transmission slot by sending a ``SendMessageRequest`` which specifies -the priority, deadline and size of the message. Note that these values -may be ignored by CORE. When CORE is ready for the message, it answers -with a ``SendMessageReady`` response. The client can then transmit the -payload with a ``SendMessage`` message. Note that the actual message -size in the ``SendMessage`` is allowed to be smaller than the size in -the original request. A client may at any time send a fresh -``SendMessageRequest``, which then superceeds the previous -``SendMessageRequest``, which is then no longer valid. The client can -tell which ``SendMessageRequest`` the CORE service's -``SendMessageReady`` message is for as all of these messages contain a -\"unique\" request ID (based on a counter incremented by the client for -each request). - -CORE Peer-to-Peer Protocol -.. _The-CORE-Peer_002dto_002dPeer-Protocol: - -The CORE Peer-to-Peer Protocol ------------------------------- - -EphemeralKeyMessage creation -.. _Creating-the-EphemeralKeyMessage: - -Creating the EphemeralKeyMessage -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When the CORE service starts, each peer creates a fresh ephemeral (ECC) -public-private key pair and signs the corresponding -``EphemeralKeyMessage`` with its long-term key (which we usually call -the peer's identity; the hash of the public long term key is what -results in a ``struct GNUNET_PeerIdentity`` in all GNUnet APIs. The -ephemeral key is ONLY used for an ECDHE (`Elliptic-curve -Diffie---Hellman <http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman>`__) -exchange by the CORE service to establish symmetric session keys. A peer -will use the same ``EphemeralKeyMessage`` for all peers for -``REKEY_FREQUENCY``, which is usually 12 hours. After that time, it will -create a fresh ephemeral key (forgetting the old one) and broadcast the -new ``EphemeralKeyMessage`` to all connected peers, resulting in fresh -symmetric session keys. Note that peers independently decide on when to -discard ephemeral keys; it is not a protocol violation to discard keys -more often. Ephemeral keys are also never stored to disk; restarting a -peer will thus always create a fresh ephemeral key. The use of ephemeral -keys is what provides `forward -secrecy <http://en.wikipedia.org/wiki/Forward_secrecy>`__. - -Just before transmission, the ``EphemeralKeyMessage`` is patched to -reflect the current sender_status, which specifies the current state of -the connection from the point of view of the sender. The possible values -are: - -- ``KX_STATE_DOWN`` Initial value, never used on the network - -- ``KX_STATE_KEY_SENT`` We sent our ephemeral key, do not know the key - of the other peer - -- ``KX_STATE_KEY_RECEIVED`` This peer has received a valid ephemeral - key of the other peer, but we are waiting for the other peer to - confirm it's authenticity (ability to decode) via challenge-response. - -- ``KX_STATE_UP`` The connection is fully up from the point of view of - the sender (now performing keep-alive) - -- ``KX_STATE_REKEY_SENT`` The sender has initiated a rekeying - operation; the other peer has so far failed to confirm a working - connection using the new ephemeral key - -.. _Establishing-a-connection: - -Establishing a connection -^^^^^^^^^^^^^^^^^^^^^^^^^ - -Peers begin their interaction by sending a ``EphemeralKeyMessage`` to -the other peer once the TRANSPORT service notifies the CORE service -about the connection. A peer receiving an ``EphemeralKeyMessage`` with a -status indicating that the sender does not have the receiver's ephemeral -key, the receiver's ``EphemeralKeyMessage`` is sent in response. -Additionally, if the receiver has not yet confirmed the authenticity of -the sender, it also sends an (encrypted)\ ``PingMessage`` with a -challenge (and the identity of the target) to the other peer. Peers -receiving a ``PingMessage`` respond with an (encrypted) ``PongMessage`` -which includes the challenge. Peers receiving a ``PongMessage`` check -the challenge, and if it matches set the connection to ``KX_STATE_UP``. - -.. _Encryption-and-Decryption: - -Encryption and Decryption -^^^^^^^^^^^^^^^^^^^^^^^^^ - -All functions related to the key exchange and encryption/decryption of -messages can be found in ``gnunet-service-core_kx.c`` (except for the -cryptographic primitives, which are in ``util/crypto*.c``). Given the -key material from ECDHE, a Key derivation function (`Key derivation -function <https://en.wikipedia.org/wiki/Key_derivation_function>`__) is -used to derive two pairs of encryption and decryption keys for AES-256 -and TwoFish, as well as initialization vectors and authentication keys -(for HMAC (`HMAC <https://en.wikipedia.org/wiki/HMAC>`__)). The HMAC is -computed over the encrypted payload. Encrypted messages include an -iv_seed and the HMAC in the header. - -Each encrypted message in the CORE service includes a sequence number -and a timestamp in the encrypted payload. The CORE service remembers the -largest observed sequence number and a bit-mask which represents which -of the previous 32 sequence numbers were already used. Messages with -sequence numbers lower than the largest observed sequence number minus -32 are discarded. Messages with a timestamp that is less than -``REKEY_TOLERANCE`` off (5 minutes) are also discarded. This of course -means that system clocks need to be reasonably synchronized for peers to -be able to communicate. Additionally, as the ephemeral key changes every -12 hours, a peer would not even be able to decrypt messages older than -12 hours. - -.. _Type-maps: - -Type maps -^^^^^^^^^ - -Once an encrypted connection has been established, peers begin to -exchange type maps. Type maps are used to allow the CORE service to -determine which (encrypted) connections should be shown to which -applications. A type map is an array of 65536 bits representing the -different types of messages understood by applications using the CORE -service. Each CORE service maintains this map, simply by setting the -respective bit for each message type supported by any of the -applications using the CORE service. Note that bits for message types -embedded in higher-level protocols (such as MESH) will not be included -in these type maps. - -Typically, the type map of a peer will be sparse. Thus, the CORE service -attempts to compress its type map using ``gzip``-style compression -(\"deflate\") prior to transmission. However, if the compression fails -to compact the map, the map may also be transmitted without compression -(resulting in ``GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP`` or -``GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP`` messages respectively). -Upon receiving a type map, the respective CORE service notifies -applications about the connection to the other peer if they support any -message type indicated in the type map (or no message type at all). If -the CORE service experience a connect or disconnect event from an -application, it updates its type map (setting or unsetting the -respective bits) and notifies its neighbours about the change. The CORE -services of the neighbours then in turn generate connect and disconnect -events for the peer that sent the type map for their respective -applications. As CORE messages may be lost, the CORE service confirms -receiving a type map by sending back a -``GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP``. If such a confirmation -(with the correct hash of the type map) is not received, the sender will -retransmit the type map (with exponential back-off). - - diff --git a/developers/corestack.rst b/developers/corestack.rst @@ -1,30 +0,0 @@ - -Services -======== - -These services comprise a backbone of core services for -peer-to-peer applications to use. - -.. toctree:: - cadet/cadet.rst - core/core.rst - dht/dht.rst - fs/fs.rst - gnsstack.rst - hostlist/hostlist.rst - identity/identity.rst - messenger/messenger.rst - nse/nse.rst - peerinfo/peerinfo.rst - peerstore/peerstore.rst - regex/regex.rst - rest/rest.rst - revocation/revocation.rst - rps/rps.rst - setops.rst - statistics/statistics.rst - transport-ng/transport-ng.rst - transport/transport.rst - vpnstack.rst - - diff --git a/developers/dht/dht.rst b/developers/dht/dht.rst @@ -1,415 +0,0 @@ -.. _Distributed-Hash-Table-_0028DHT_0029: - -.. index:: - double: Distributed hash table; subsystem - see: DHT; Distributed hash table - -DHT — Distributed Hash Table -============================ - -GNUnet includes a generic distributed hash table that can be used by -developers building P2P applications in the framework. This section -documents high-level features and how developers are expected to use the -DHT. We have a research paper detailing how the DHT works. Also, Nate's -thesis includes a detailed description and performance analysis (in -chapter 6). [R5N2011]_ - -.. todo:: Confirm: Are "Nate's thesis" and the "research paper" separate - entities? - -Key features of GNUnet's DHT include: - -- stores key-value pairs with values up to (approximately) 63k in size - -- works with many underlay network topologies (small-world, random - graph), underlay does not need to be a full mesh / clique - -- support for extended queries (more than just a simple 'key'), - filtering duplicate replies within the network (bloomfilter) and - content validation (for details, please read the subsection on the - block library) - -- can (optionally) return paths taken by the PUT and GET operations to - the application - -- provides content replication to handle churn - -GNUnet's DHT is randomized and unreliable. Unreliable means that there -is no strict guarantee that a value stored in the DHT is always found -— values are only found with high probability. While this is somewhat -true in all P2P DHTs, GNUnet developers should be particularly wary of -this fact (this will help you write secure, fault-tolerant code). Thus, -when writing any application using the DHT, you should always consider -the possibility that a value stored in the DHT by you or some other peer -might simply not be returned, or returned with a significant delay. Your -application logic must be written to tolerate this (naturally, some loss -of performance or quality of service is expected in this case). - -.. _Block-library-and-plugins: - -Block library and plugins -------------------------- - -.. _What-is-a-Block_003f: - -What is a Block? -^^^^^^^^^^^^^^^^ - -Blocks are small (< 63k) pieces of data stored under a key (struct -GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which -defines their data format. Blocks are used in GNUnet as units of static -data exchanged between peers and stored (or cached) locally. Uses of -blocks include file-sharing (the files are broken up into blocks), the -VPN (DNS information is stored in blocks) and the DHT (all information -in the DHT and meta-information for the maintenance of the DHT are both -stored using blocks). The block subsystem provides a few common -functions that must be available for any type of block. - -libgnunetblock API -.. _The-API-of-libgnunetblock: - -The API of libgnunetblock -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The block library requires for each (family of) block type(s) a block -plugin (implementing ``gnunet_block_plugin.h``) that provides basic -functions that are needed by the DHT (and possibly other subsystems) to -manage the block. These block plugins are typically implemented within -their respective subsystems. The main block library is then used to -locate, load and query the appropriate block plugin. Which plugin is -appropriate is determined by the block type (which is just a 32-bit -integer). Block plugins contain code that specifies which block types -are supported by a given plugin. The block library loads all block -plugins that are installed at the local peer and forwards the -application request to the respective plugin. - -The central functions of the block APIs (plugin and main library) are to -allow the mapping of blocks to their respective key (if possible) and -the ability to check that a block is well-formed and matches a given -request (again, if possible). This way, GNUnet can avoid storing invalid -blocks, storing blocks under the wrong key and forwarding blocks in -response to a query that they do not answer. - -One key function of block plugins is that it allows GNUnet to detect -duplicate replies (via the Bloom filter). All plugins MUST support -detecting duplicate replies (by adding the current response to the Bloom -filter and rejecting it if it is encountered again). If a plugin fails -to do this, responses may loop in the network. - -.. _Queries: - -Queries -^^^^^^^ - -The query format for any block in GNUnet consists of four main -components. First, the type of the desired block must be specified. -Second, the query must contain a hash code. The hash code is used for -lookups in hash tables and databases and must not be unique for the -block (however, if possible a unique hash should be used as this would -be best for performance). Third, an optional Bloom filter can be -specified to exclude known results; replies that hash to the bits set in -the Bloom filter are considered invalid. False-positives can be -eliminated by sending the same query again with a different Bloom filter -mutator value, which parametrizes the hash function that is used. -Finally, an optional application-specific \"eXtended query\" (xquery) -can be specified to further constrain the results. It is entirely up to -the type-specific plugin to determine whether or not a given block -matches a query (type, hash, Bloom filter, and xquery). Naturally, not -all xquery's are valid and some types of blocks may not support Bloom -filters either, so the plugin also needs to check if the query is valid -in the first place. - -Depending on the results from the plugin, the DHT will then discard the -(invalid) query, forward the query, discard the (invalid) reply, cache -the (valid) reply, and/or forward the (valid and non-duplicate) reply. - -.. _Sample-Code: - -Sample Code -^^^^^^^^^^^ - -The source code in **plugin_block_test.c** is a good starting point for -new block plugins --- it does the minimal work by implementing a plugin -that performs no validation at all. The respective **Makefile.am** shows -how to build and install a block plugin. - -.. _Conclusion2: - -Conclusion2 -^^^^^^^^^^^ - -In conclusion, GNUnet subsystems that want to use the DHT need to define -a block format and write a plugin to match queries and replies. For -testing, the ``GNUNET_BLOCK_TYPE_TEST`` block type can be used; it -accepts any query as valid and any reply as matching any query. This -type is also used for the DHT command line tools. However, it should NOT -be used for normal applications due to the lack of error checking that -results from this primitive implementation. - -libgnunetdht -:index:`libgnunetdht <single: libgnunet; dht>` -libgnunetdht ------------- - -The DHT API itself is pretty simple and offers the usual GET and PUT -functions that work as expected. The specified block type refers to the -block library which allows the DHT to run application-specific logic for -data stored in the network. - -.. _GET: - -GET -^^^ - -When using GET, the main consideration for developers (other than the -block library) should be that after issuing a GET, the DHT will -continuously cause (small amounts of) network traffic until the -operation is explicitly canceled. So GET does not simply send out a -single network request once; instead, the DHT will continue to search -for data. This is needed to achieve good success rates and also handles -the case where the respective PUT operation happens after the GET -operation was started. Developers should not cancel an existing GET -operation and then explicitly re-start it to trigger a new round of -network requests; this is simply inefficient, especially as the internal -automated version can be more efficient, for example by filtering -results in the network that have already been returned. - -If an application that performs a GET request has a set of replies that -it already knows and would like to filter, it can -call ``GNUNET_DHT_get_filter_known_results`` with an array of hashes -over the respective blocks to tell the DHT that these results are not -desired (any more). This way, the DHT will filter the respective blocks -using the block library in the network, which may result in a -significant reduction in bandwidth consumption. - -.. _PUT: - -PUT -^^^ - -.. todo:: inconsistent use of "must" above it's written "MUST" - -In contrast to GET operations, developers **must** manually re-run PUT -operations periodically (if they intend the content to continue to be -available). Content stored in the DHT expires or might be lost due to -churn. Furthermore, GNUnet's DHT typically requires multiple rounds of -PUT operations before a key-value pair is consistently available to all -peers (the DHT randomizes paths and thus storage locations, and only -after multiple rounds of PUTs there will be a sufficient number of -replicas in large DHTs). An explicit PUT operation using the DHT API -will only cause network traffic once, so in order to ensure basic -availability and resistance to churn (and adversaries), PUTs must be -repeated. While the exact frequency depends on the application, a rule -of thumb is that there should be at least a dozen PUT operations within -the content lifetime. Content in the DHT typically expires after one -day, so DHT PUT operations should be repeated at least every 1-2 hours. - -.. _MONITOR: - -MONITOR -^^^^^^^ - -The DHT API also allows applications to monitor messages crossing the -local DHT service. The types of messages used by the DHT are GET, PUT -and RESULT messages. Using the monitoring API, applications can choose -to monitor these requests, possibly limiting themselves to requests for -a particular block type. - -The monitoring API is not only useful for diagnostics, it can also be -used to trigger application operations based on PUT operations. For -example, an application may use PUTs to distribute work requests to -other peers. The workers would then monitor for PUTs that give them -work, instead of looking for work using GET operations. This can be -beneficial, especially if the workers have no good way to guess the keys -under which work would be stored. Naturally, additional protocols might -be needed to ensure that the desired number of workers will process the -distributed workload. - -.. _DHT-Routing-Options: - -DHT Routing Options -^^^^^^^^^^^^^^^^^^^ - -There are two important options for GET and PUT requests: - -GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all - peers should process the request, even if their peer ID is not - closest to the key. For a PUT request, this means that all peers that - a request traverses may make a copy of the data. Similarly for a GET - request, all peers will check their local database for a result. - Setting this option can thus significantly improve caching and reduce - bandwidth consumption --- at the expense of a larger DHT database. If - in doubt, we recommend that this option should be used. - -GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record - the path that a GET or a PUT request is taking through the overlay - network. The resulting paths are then returned to the application - with the respective result. This allows the receiver of a result to - construct a path to the originator of the data, which might then be - used for routing. Naturally, setting this option requires additional - bandwidth and disk space, so applications should only set this if the - paths are needed by the application logic. - -GNUNET_DHT_RO_FIND_PEER This option is an internal option used by - the DHT's peer discovery mechanism and should not be used by - applications. - -GNUNET_DHT_RO_BART This option is currently not implemented. It may - in the future offer performance improvements for clique topologies. - -.. _The-DHT-Client_002dService-Protocol: - -The DHT Client-Service Protocol -------------------------------- - -.. _PUTting-data-into-the-DHT: - -PUTting data into the DHT -^^^^^^^^^^^^^^^^^^^^^^^^^ - -To store (PUT) data into the DHT, the client sends a -``struct GNUNET_DHT_ClientPutMessage`` to the service. This message -specifies the block type, routing options, the desired replication -level, the expiration time, key, value and a 64-bit unique ID for the -operation. The service responds with a -``struct GNUNET_DHT_ClientPutConfirmationMessage`` with the same 64-bit -unique ID. Note that the service sends the confirmation as soon as it -has locally processed the PUT request. The PUT may still be propagating -through the network at this time. - -In the future, we may want to change this to provide (limited) feedback -to the client, for example if we detect that the PUT operation had no -effect because the same key-value pair was already stored in the DHT. -However, changing this would also require additional state and messages -in the P2P interaction. - -.. _GETting-data-from-the-DHT: - -GETting data from the DHT -^^^^^^^^^^^^^^^^^^^^^^^^^ - -To retrieve (GET) data from the DHT, the client sends a -``struct GNUNET_DHT_ClientGetMessage`` to the service. The message -specifies routing options, a replication level (for replicating the GET, -not the content), the desired block type, the key, the (optional) -extended query and unique 64-bit request ID. - -Additionally, the client may send any number of -``struct GNUNET_DHT_ClientGetResultSeenMessage``\ s to notify the -service about results that the client is already aware of. These -messages consist of the key, the unique 64-bit ID of the request, and an -arbitrary number of hash codes over the blocks that the client is -already aware of. As messages are restricted to 64k, a client that -already knows more than about a thousand blocks may need to send several -of these messages. Naturally, the client should transmit these messages -as quickly as possible after the original GET request such that the DHT -can filter those results in the network early on. Naturally, as these -messages are sent after the original request, it is conceivable that the -DHT service may return blocks that match those already known to the -client anyway. - -In response to a GET request, the service will send ``struct -GNUNET_DHT_ClientResultMessage``\ s to the client. These messages -contain the block type, expiration, key, unique ID of the request and of -course the value (a block). Depending on the options set for the -respective operations, the replies may also contain the path the GET -and/or the PUT took through the network. - -A client can stop receiving replies either by disconnecting or by -sending a ``struct GNUNET_DHT_ClientGetStopMessage`` which must contain -the key and the 64-bit unique ID of the original request. Using an -explicit \"stop\" message is more common as this allows a client to run -many concurrent GET operations over the same connection with the DHT -service --- and to stop them individually. - -.. _Monitoring-DHT: - -Monitoring the DHT -^^^^^^^^^^^^^^^^^^ - -To begin monitoring, the client sends a -``struct GNUNET_DHT_MonitorStartStop`` message to the DHT service. In -this message, flags can be set to enable (or disable) monitoring of GET, -PUT and RESULT messages that pass through a peer. The message can also -restrict monitoring to a particular block type or a particular key. Once -monitoring is enabled, the DHT service will notify the client about any -matching event using ``struct GNUNET_DHT_MonitorGetMessage``\ s for GET -events, ``struct GNUNET_DHT_MonitorPutMessage`` for PUT events and -``struct GNUNET_DHT_MonitorGetRespMessage`` for RESULTs. Each of these -messages contains all of the information about the event. - -.. _The-DHT-Peer_002dto_002dPeer-Protocol: - -The DHT Peer-to-Peer Protocol ------------------------------ - -.. _Routing-GETs-or-PUTs: - -Routing GETs or PUTs -^^^^^^^^^^^^^^^^^^^^ - -When routing GETs or PUTs, the DHT service selects a suitable subset of -neighbours for forwarding. The exact number of neighbours can be zero or -more and depends on the hop counter of the query (initially zero) in -relation to the (log of) the network size estimate, the desired -replication level and the peer's connectivity. Depending on the hop -counter and our network size estimate, the selection of the peers maybe -randomized or by proximity to the key. Furthermore, requests include a -set of peers that a request has already traversed; those peers are also -excluded from the selection. - -.. _PUTting-data-into-the-DHT2: - -PUTting data into the DHT -^^^^^^^^^^^^^^^^^^^^^^^^^ - -To PUT data into the DHT, the service sends a ``struct PeerPutMessage`` -of type ``GNUNET_MESSAGE_TYPE_DHT_P2P_PUT`` to the respective neighbour. -In addition to the usual information about the content (type, routing -options, desired replication level for the content, expiration time, key -and value), the message contains a fixed-size Bloom filter with -information about which peers (may) have already seen this request. This -Bloom filter is used to ensure that DHT messages never loop back to a -peer that has already processed the request. Additionally, the message -includes the current hop counter and, depending on the routing options, -the message may include the full path that the message has taken so far. -The Bloom filter should already contain the identity of the previous -hop; however, the path should not include the identity of the previous -hop and the receiver should append the identity of the sender to the -path, not its own identity (this is done to reduce bandwidth). - -.. _GETting-data-from-the-DHT2: - -GETting data from the DHT -^^^^^^^^^^^^^^^^^^^^^^^^^ - -A peer can search the DHT by sending ``struct PeerGetMessage``\ s of -type ``GNUNET_MESSAGE_TYPE_DHT_P2P_GET`` to other peers. In addition to -the usual information about the request (type, routing options, desired -replication level for the request, the key and the extended query), a -GET request also contains a hop counter, a Bloom filter over the peers -that have processed the request already and depending on the routing -options the full path traversed by the GET. Finally, a GET request -includes a variable-size second Bloom filter and a so-called Bloom -filter mutator value which together indicate which replies the sender -has already seen. During the lookup, each block that matches they block -type, key and extended query is additionally subjected to a test against -this Bloom filter. The block plugin is expected to take the hash of the -block and combine it with the mutator value and check if the result is -not yet in the Bloom filter. The originator of the query will from time -to time modify the mutator to (eventually) allow false-positives -filtered by the Bloom filter to be returned. - -Peers that receive a GET request perform a local lookup (depending on -their proximity to the key and the query options) and forward the -request to other peers. They then remember the request (including the -Bloom filter for blocking duplicate results) and when they obtain a -matching, non-filtered response a ``struct PeerResultMessage`` of type -``GNUNET_MESSAGE_TYPE_DHT_P2P_RESULT`` is forwarded to the previous hop. -Whenever a result is forwarded, the block plugin is used to update the -Bloom filter accordingly, to ensure that the same result is never -forwarded more than once. The DHT service may also cache forwarded -results locally if the \"CACHE_RESULTS\" option is set to \"YES\" in the -configuration. - -.. [R5N2011] https://bib.gnunet.org/date.html#R5N diff --git a/developers/fs/fs.rst b/developers/fs/fs.rst @@ -1,175 +0,0 @@ -.. index:: - double: File sharing; subsystem - see: FS; File sharing - -.. _File_002dsharing-_0028FS_0029-Subsystem: - -FS — File sharing over GNUnet -============================= - -This chapter describes the details of how the file-sharing service -works. As with all services, it is split into an API (libgnunetfs), the -service process (gnunet-service-fs) and user interface(s). The -file-sharing service uses the datastore service to store blocks and the -DHT (and indirectly datacache) for lookups for non-anonymous -file-sharing. Furthermore, the file-sharing service uses the block -library (and the block fs plugin) for validation of DHT operations. - -In contrast to many other services, libgnunetfs is rather complex since -the client library includes a large number of high-level abstractions; -this is necessary since the FS service itself largely only operates on -the block level. The FS library is responsible for providing a -file-based abstraction to applications, including directories, meta -data, keyword search, verification, and so on. - -The method used by GNUnet to break large files into blocks and to use -keyword search is called the \"Encoding for Censorship Resistant -Sharing\" (ECRS). ECRS is largely implemented in the fs library; block -validation is also reflected in the block FS plugin and the FS service. -ECRS on-demand encoding is implemented in the FS service. - -.. note:: The documentation in this chapter is quite incomplete. - -.. _Encoding-for-Censorship_002dResistant-Sharing-_0028ECRS_0029: - -.. index:: - see: Encoding for Censorship-Resistant Sharing; ECRS - -:index:`ECRS — Encoding for Censorship-Resistant Sharing <single: ECRS>` -ECRS — Encoding for Censorship-Resistant Sharing ------------------------------------------------- - -When GNUnet shares files, it uses a content encoding that is called -ECRS, the Encoding for Censorship-Resistant Sharing. Most of ECRS is -described in the (so far unpublished) research paper attached to this -page. ECRS obsoletes the previous ESED and ESED II encodings which were -used in GNUnet before version 0.7.0. The rest of this page assumes that -the reader is familiar with the attached paper. What follows is a -description of some minor extensions that GNUnet makes over what is -described in the paper. The reason why these extensions are not in the -paper is that we felt that they were obvious or trivial extensions to -the original scheme and thus did not warrant space in the research -report. - -.. todo:: Find missing link to file system paper. - -.. _Namespace-Advertisements: - -Namespace Advertisements -^^^^^^^^^^^^^^^^^^^^^^^^ - -.. todo:: FIXME: all zeroses -> ? - -An ``SBlock`` with identifier all zeros is a signed advertisement for a -namespace. This special ``SBlock`` contains metadata describing the -content of the namespace. Instead of the name of the identifier for a -potential update, it contains the identifier for the root of the -namespace. The URI should always be empty. The ``SBlock`` is signed with -the content provider's RSA private key (just like any other SBlock). -Peers can search for ``SBlock``\ s in order to find out more about a -namespace. - -.. _KSBlocks: - -KSBlocks -^^^^^^^^ - -GNUnet implements ``KSBlocks`` which are ``KBlocks`` that, instead of -encrypting a CHK and metadata, encrypt an ``SBlock`` instead. In other -words, ``KSBlocks`` enable GNUnet to find ``SBlocks`` using the global -keyword search. Usually the encrypted ``SBlock`` is a namespace -advertisement. The rationale behind ``KSBlock``\ s and ``SBlock``\ s is -to enable peers to discover namespaces via keyword searches, and, to -associate useful information with namespaces. When GNUnet finds -``KSBlocks`` during a normal keyword search, it adds the information to -an internal list of discovered namespaces. Users looking for interesting -namespaces can then inspect this list, reducing the need for out-of-band -discovery of namespaces. Naturally, namespaces (or more specifically, -namespace advertisements) can also be referenced from directories, but -``KSBlock``\ s should make it easier to advertise namespaces for the -owner of the pseudonym since they eliminate the need to first create a -directory. - -Collections are also advertised using ``KSBlock``\ s. - -.. https://old.gnunet.org/sites/default/files/ecrs.pdf -.. What is this? - WGL - -.. _File_002dsharing-persistence-directory-structure: - -File-sharing persistence directory structure --------------------------------------------- - -This section documents how the file-sharing library implements -persistence of file-sharing operations and specifically the resulting -directory structure. This code is only active if the -``GNUNET_FS_FLAGS_PERSISTENCE`` flag was set when calling -``GNUNET_FS_start``. In this case, the file-sharing library will try -hard to ensure that all major operations (searching, downloading, -publishing, unindexing) are persistent, that is, can live longer than -the process itself. More specifically, an operation is supposed to live -until it is explicitly stopped. - -If ``GNUNET_FS_stop`` is called before an operation has been stopped, a -``SUSPEND`` event is generated and then when the process calls -``GNUNET_FS_start`` next time, a ``RESUME`` event is generated. -Additionally, even if an application crashes (segfault, SIGKILL, system -crash) and hence ``GNUNET_FS_stop`` is never called and no ``SUSPEND`` -events are generated, operations are still resumed (with ``RESUME`` -events). This is implemented by constantly writing the current state of -the file-sharing operations to disk. Specifically, the current state is -always written to disk whenever anything significant changes (the -exception are block-wise progress in publishing and unindexing, since -those operations would be slowed down significantly and can be resumed -cheaply even without detailed accounting). Note that if the process -crashes (or is killed) during a serialization operation, FS does not -guarantee that this specific operation is recoverable (no strict -transactional semantics, again for performance reasons). However, all -other unrelated operations should resume nicely. - -Since we need to serialize the state continuously and want to recover as -much as possible even after crashing during a serialization operation, -we do not use one large file for serialization. Instead, several -directories are used for the various operations. When -``GNUNET_FS_start`` executes, the master directories are scanned for -files describing operations to resume. Sometimes, these operations can -refer to related operations in child directories which may also be -resumed at this point. Note that corrupted files are cleaned up -automatically. However, dangling files in child directories (those that -are not referenced by files from the master directories) are not -automatically removed. - -Persistence data is kept in a directory that begins with the -\"STATE_DIR\" prefix from the configuration file (by default, -\"$SERVICEHOME/persistence/\") followed by the name of the client as -given to ``GNUNET_FS_start`` (for example, \"gnunet-gtk\") followed by -the actual name of the master or child directory. - -The names for the master directories follow the names of the operations: - -- \"search\" - -- \"download\" - -- \"publish\" - -- \"unindex\" - -Each of the master directories contains names (chosen at random) for -each active top-level (master) operation. Note that a download that is -associated with a search result is not a top-level operation. - -In contrast to the master directories, the child directories are only -consulted when another operation refers to them. For each search, a -subdirectory (named after the master search synchronization file) -contains the search results. Search results can have an associated -download, which is then stored in the general \"download-child\" -directory. Downloads can be recursive, in which case children are stored -in subdirectories mirroring the structure of the recursive download -(either starting in the master \"download\" directory or in the -\"download-child\" directory depending on how the download was -initiated). For publishing operations, the \"publish-file\" directory -contains information about the individual files and directories that are -part of the publication. However, this directory structure is flat and -does not mirror the structure of the publishing operation. Note that -unindex operations cannot have associated child operations. diff --git a/developers/gns/gns.rst b/developers/gns/gns.rst @@ -1,617 +0,0 @@ -.. index:: - double: GNU Name System; subsystem - see: GNS; GNU Name System - -.. _GNU-Name-System-_0028GNS_0029: - -GNS — the GNU Name System -========================= - -The GNU Name System (GNS) is a decentralized database that enables users -to securely resolve names to values. Names can be used to identify other -users (for example, in social networking), or network services (for -example, VPN services running at a peer in GNUnet, or purely IP-based -services on the Internet). Users interact with GNS by typing in a -hostname that ends in a top-level domain that is configured in the "GNS" -section, matches an identity of the user or ends in a Base32-encoded -public key. - -Videos giving an overview of most of the GNS and the motivations behind -it is available here and here. The remainder of this chapter targets -developers that are familiar with high level concepts of GNS as -presented in these talks. - -.. todo:: Link to videos and GNS talks? - -GNS-aware applications should use the GNS resolver to obtain the -respective records that are stored under that name in GNS. Each record -consists of a type, value, expiration time and flags. - -The type specifies the format of the value. Types below 65536 correspond -to DNS record types, larger values are used for GNS-specific records. -Applications can define new GNS record types by reserving a number and -implementing a plugin (which mostly needs to convert the binary value -representation to a human-readable text format and vice-versa). The -expiration time specifies how long the record is to be valid. The GNS -API ensures that applications are only given non-expired values. The -flags are typically irrelevant for applications, as GNS uses them -internally to control visibility and validity of records. - -Records are stored along with a signature. The signature is generated -using the private key of the authoritative zone. This allows any GNS -resolver to verify the correctness of a name-value mapping. - -Internally, GNS uses the NAMECACHE to cache information obtained from -other users, the NAMESTORE to store information specific to the local -users, and the DHT to exchange data between users. A plugin API is used -to enable applications to define new GNS record types. - - -:index:`libgnunetgns <single: libgnunet; gns>` -libgnunetgns ------------- - -The GNS API itself is extremely simple. Clients first connect to the GNS -service using ``GNUNET_GNS_connect``. They can then perform lookups -using ``GNUNET_GNS_lookup`` or cancel pending lookups using -``GNUNET_GNS_lookup_cancel``. Once finished, clients disconnect using -``GNUNET_GNS_disconnect``. - -.. _Looking-up-records: - -Looking up records -^^^^^^^^^^^^^^^^^^ - -``GNUNET_GNS_lookup`` takes a number of arguments: - -handle This is simply the GNS connection handle from - ``GNUNET_GNS_connect``. - -name The client needs to specify the name to - be resolved. This can be any valid DNS or GNS hostname. - -zone The client - needs to specify the public key of the GNS zone against which the - resolution should be done. Note that a key must be provided, the - client should look up plausible values using its configuration, the - identity service and by attempting to interpret the TLD as a - base32-encoded public key. - -type This is the desired GNS or DNS record type - to look for. While all records for the given name will be returned, - this can be important if the client wants to resolve record types - that themselves delegate resolution, such as CNAME, PKEY or GNS2DNS. - Resolving a record of any of these types will only work if the - respective record type is specified in the request, as the GNS - resolver will otherwise follow the delegation and return the records - from the respective destination, instead of the delegating record. - -only_cached This argument should typically be set to - ``GNUNET_NO``. Setting it to ``GNUNET_YES`` disables resolution via - the overlay network. - -shorten_zone_key If GNS encounters new names during resolution, - their respective zones can automatically be learned and added to the - \"shorten zone\". If this is desired, clients must pass the private - key of the shorten zone. If NULL is passed, shortening is disabled. - -proc This argument identifies - the function to call with the result. It is given proc_cls, the - number of records found (possibly zero) and the array of the records - as arguments. proc will only be called once. After proc,> has been - called, the lookup must no longer be canceled. - -proc_cls The closure for proc. - -.. _Accessing-the-records: - -Accessing the records -^^^^^^^^^^^^^^^^^^^^^ - -The ``libgnunetgnsrecord`` library provides an API to manipulate the GNS -record array that is given to proc. In particular, it offers functions -such as converting record values to human-readable strings (and back). -However, most ``libgnunetgnsrecord`` functions are not interesting to -GNS client applications. - -For DNS records, the ``libgnunetdnsparser`` library provides functions -for parsing (and serializing) common types of DNS records. - -.. _Creating-records: - -Creating records -^^^^^^^^^^^^^^^^ - -Creating GNS records is typically done by building the respective record -information (possibly with the help of ``libgnunetgnsrecord`` and -``libgnunetdnsparser``) and then using the ``libgnunetnamestore`` to -publish the information. The GNS API is not involved in this operation. - -.. _Future-work: - -Future work -^^^^^^^^^^^ - -In the future, we want to expand ``libgnunetgns`` to allow applications -to observe shortening operations performed during GNS resolution, for -example so that users can receive visual feedback when this happens. - -libgnunetgnsrecord -~~~~~~~~~~~~~~~~~~ - -The ``libgnunetgnsrecord`` library is used to manipulate GNS records (in -plaintext or in their encrypted format). Applications mostly interact -with ``libgnunetgnsrecord`` by using the functions to convert GNS record -values to strings or vice-versa, or to lookup a GNS record type number -by name (or vice-versa). The library also provides various other -functions that are mostly used internally within GNS, such as converting -keys to names, checking for expiration, encrypting GNS records to GNS -blocks, verifying GNS block signatures and decrypting GNS records from -GNS blocks. - -We will now discuss the four commonly used functions of the -API. ``libgnunetgnsrecord`` does not perform these operations itself, -but instead uses plugins to perform the operation. GNUnet includes -plugins to support common DNS record types as well as standard GNS -record types. - -.. _Value-handling: - -Value handling -^^^^^^^^^^^^^^ - -``GNUNET_GNSRECORD_value_to_string`` can be used to convert the (binary) -representation of a GNS record value to a human readable, 0-terminated -UTF-8 string. NULL is returned if the specified record type is not -supported by any available plugin. - -``GNUNET_GNSRECORD_string_to_value`` can be used to try to convert a -human readable string to the respective (binary) representation of a GNS -record value. - -.. _Type-handling: - -Type handling -^^^^^^^^^^^^^ - -``GNUNET_GNSRECORD_typename_to_number`` can be used to obtain the -numeric value associated with a given typename. For example, given the -typename \"A\" (for DNS A reocrds), the function will return the number -1. A list of common DNS record types is -`here <http://en.wikipedia.org/wiki/List_of_DNS_record_types>`__. Note -that not all DNS record types are supported by GNUnet GNSRECORD plugins -at this time. - -``GNUNET_GNSRECORD_number_to_typename`` can be used to obtain the -typename associated with a given numeric value. For example, given the -type number 1, the function will return the typename \"A\". - -.. _GNS-plugins: - -GNS plugins -~~~~~~~~~~~ - -Adding a new GNS record type typically involves writing (or extending) a -GNSRECORD plugin. The plugin needs to implement the -``gnunet_gnsrecord_plugin.h`` API which provides basic functions that -are needed by GNSRECORD to convert typenames and values of the -respective record type to strings (and back). These gnsrecord plugins -are typically implemented within their respective subsystems. Examples -for such plugins can be found in the GNSRECORD, GNS and CONVERSATION -subsystems. - -The ``libgnunetgnsrecord`` library is then used to locate, load and -query the appropriate gnsrecord plugin. Which plugin is appropriate is -determined by the record type (which is just a 32-bit integer). The -``libgnunetgnsrecord`` library loads all block plugins that are -installed at the local peer and forwards the application request to the -plugins. If the record type is not supported by the plugin, it should -simply return an error code. - -The central functions of the block APIs (plugin and main library) are -the same four functions for converting between values and strings, and -typenames and numbers documented in the previous subsection. - -.. _The-GNS-Client_002dService-Protocol: - -The GNS Client-Service Protocol -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The GNS client-service protocol consists of two simple messages, the -``LOOKUP`` message and the ``LOOKUP_RESULT``. Each ``LOOKUP`` message -contains a unique 32-bit identifier, which will be included in the -corresponding response. Thus, clients can send many lookup requests in -parallel and receive responses out-of-order. A ``LOOKUP`` request also -includes the public key of the GNS zone, the desired record type and -fields specifying whether shortening is enabled or networking is -disabled. Finally, the ``LOOKUP`` message includes the name to be -resolved. - -The response includes the number of records and the records themselves -in the format created by ``GNUNET_GNSRECORD_records_serialize``. They -can thus be deserialized using ``GNUNET_GNSRECORD_records_deserialize``. - -.. _Hijacking-the-DNS_002dTraffic-using-gnunet_002dservice_002ddns: - -Hijacking the DNS-Traffic using gnunet-service-dns -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section documents how the gnunet-service-dns (and the -gnunet-helper-dns) intercepts DNS queries from the local system. This is -merely one method for how we can obtain GNS queries. It is also possible -to change ``resolv.conf`` to point to a machine running -``gnunet-dns2gns`` or to modify libc's name system switch (NSS) -configuration to include a GNS resolution plugin. The method described -in this chapter is more of a last-ditch catch-all approach. - -``gnunet-service-dns`` enables intercepting DNS traffic using policy -based routing. We MARK every outgoing DNS-packet if it was not sent by -our application. Using a second routing table in the Linux kernel these -marked packets are then routed through our virtual network interface and -can thus be captured unchanged. - -Our application then reads the query and decides how to handle it. If -the query can be addressed via GNS, it is passed to -``gnunet-service-gns`` and resolved internally using GNS. In the future, -a reverse query for an address of the configured virtual network could -be answered with records kept about previous forward queries. Queries -that are not hijacked by some application using the DNS service will be -sent to the original recipient. The answer to the query will always be -sent back through the virtual interface with the original nameserver as -source address. - -.. _Network-Setup-Details: - -Network Setup Details -^^^^^^^^^^^^^^^^^^^^^ - -The DNS interceptor adds the following rules to the Linux kernel: - -:: - - iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \ - -j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \ - --set-mark 3 ip rule add fwmark 3 table2 ip route add default via \ - $VIRTUALDNS table2 - -.. todo:: - FIXME: Rewrite to reflect display which is no longer content - by line due to the < 74 characters limit. - -Line 1 makes sure that all packets coming from a port our application -opened beforehand (``$LOCALPORT``) will be routed normally. Line 2 marks -every other packet to a DNS-Server with mark 3 (chosen arbitrarily). The -third line adds a routing policy based on this mark 3 via the routing -table. - -.. _Importing-DNS-Zones-into-GNS: - -Importing DNS Zones into GNS -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section discusses the challenges and problems faced when writing -the Ascension tool. It also takes a look at possible improvements in the -future. - -Consider the following diagram that shows the workflow of Ascension: - -|ascension| - -Further the interaction between components of GNUnet are shown in the -diagram below: - -DNS Conversion -.. _Conversions-between-DNS-and-GNS: - -Conversions between DNS and GNS -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The differences between the two name systems lies in the details and is -not always transparent. For instance an SRV record is converted to a BOX -record which is unique to GNS. - -This is done by converting to a BOX record from an existing SRV record: - -:: - - # SRV - # _service._proto.name. TTL class SRV priority weight port target - _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com. - # BOX - # TTL BOX flags port protocol recordtype priority weight port target - 14000 BOX n 5060 6 33 0 0 5060 www.example.com - -Other records that need to undergo such transformation is the MX record -type, as well as the SOA record type. - -Transformation of a SOA record into GNS works as described in the -following example. Very important to note are the rname and mname keys. - -:: - - # BIND syntax for a clean SOA record - IN SOA master.example.com. hostmaster.example.com. ( - 2017030300 ; serial - 3600 ; refresh - 1800 ; retry - 604800 ; expire - 600 ) ; ttl - # Recordline for adding the record - $ gnunet-namestore -z example.com -a -n -t SOA -V \ - rname=master.example.com mname=hostmaster.example.com \ - 2017030300,3600,1800,604800,600 -e 7200s - -The transformation of MX records is done in a simple way. - -:: - - # mail.example.com. 3600 IN MX 10 mail.example.com. - $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail - -Finally, one of the biggest struggling points were the NS records that -are found in top level domain zones. The intended behaviour for those is -to add GNS2DNS records for those so that gnunet-gns can resolve records -for those domains on its own. Those require the values from DNS GLUE -records, provided they are within the same zone. - -The following two examples show one record with a GLUE record and the -other one does not have a GLUE record. This takes place in the 'com' -TLD. - -.. code-block:: shell - - # ns1.example.com 86400 IN A 127.0.0.1 - # example.com 86400 IN NS ns1.example.com. - $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ - example.com@127.0.0.1 - - # example.com 86400 IN NS ns1.example.org. - $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ - example.com@ns1.example.org - -As you can see, one of the GNS2DNS records has an IP address listed and -the other one a DNS name. For the first one there is a GLUE record to do -the translation directly and the second one will issue another DNS query -to figure out the IP of ns1.example.org. - -A solution was found by creating a hierarchical zone structure in GNS -and linking the zones using PKEY records to one another. This allows the -resolution of the name servers to work within GNS while not taking -control over unwanted zones. - -Currently the following record types are supported: - -- A - -- AAAA - -- CNAME - -- MX - -- NS - -- SRV - -- TXT - -This is not due to technical limitations but rather a practical ones. -The problem occurs with DNSSEC enabled DNS zones. As records within -those zones are signed periodically, and every new signature is an -update to the zone, there are many revisions of zones. This results in a -problem with bigger zones as there are lots of records that have been -signed again but no major changes. Also trying to add records that are -unknown that require a different format take time as they cause a CLI -call of the namestore. Furthermore certain record types need -transformation into a GNS compatible format which, depending on the -record type, takes more time. - -Further a blacklist was added to drop for instance DNSSEC related -records. Also if a record type is neither in the white list nor the -blacklist it is considered as a loss of data and a message is shown to -the user. This helps with transparency and also with contributing, as -the not supported record types can then be added accordingly. - -.. _DNS-Zone-Size: - -DNS Zone Size -^^^^^^^^^^^^^ - -Another very big problem exists with very large zones. When migrating a -small zone the delay between adding of records and their expiry is -negligible. However when working with big zones that easily have more -than a few million records this delay becomes a problem. - -Records will start to expire well before the zone has finished -migrating. This is usually not a problem but can cause a high CPU load -when a peer is restarted and the records have expired. - -A good solution has not been found yet. One of the idea that floated -around was that the records should be added with the s (shadow) flag to -keep the records resolvable even if they expired. However this would -introduce the problem of how to detect if a record has been removed from -the zone and would require deletion of said record(s). - -Another problem that still persists is how to refresh records. Expired -records are still displayed when calling gnunet-namestore but do not -resolve with gnunet-gns. Zonemaster will sign the expired records again -and make sure that the records are still valid. With a recent change -this was fixed as gnunet-gns to improve the suffix lookup which allows -for a fast lookup even with thousands of local egos. - -Currently the pace of adding records in general is around 10 records per -second. Crypto is the upper limit for adding of records. The performance -of your machine can be tested with the perf_crypto\_\* tools. There is -still a big discrepancy between the pace of Ascension and the -theoretical limit. - -A performance metric for measuring improvements has not yet been -implemented in Ascension. - -.. _Performance: - -Performance -^^^^^^^^^^^ - -The performance when migrating a zone using the Ascension tool is -limited by a handful of factors. First of all ascension is written in -Python3 and calls the CLI tools of GNUnet. This is comparable to a fork -and exec call which costs a few CPU cycles. Furthermore all the records -that are added to the same label are signed using the zones private key. -This signing operation is very resource heavy and was optimized during -development by adding the '-R' (Recordline) option to gnunet-namestore -which allows to specify multiple records using the CLI tool. Assuming -that in a TLD zone every domain has at least two name servers this -halves the amount of signatures needed. - -Another improvement that could be made is with the addition of multiple -threads or using asynchronous subprocesses when opening the GNUnet CLI -tools. This could be implemented by simply creating more workers in the -program but performance improvements were not tested. - -Ascension was tested using different hardware and database backends. -Performance differences between SQLite and postgresql are marginal and -almost non existent. What did make a huge impact on record adding -performance was the storage medium. On a traditional mechanical hard -drive adding of records were slow compared to a solid state disk. - -In conclusion there are many bottlenecks still around in the program, -namely the single threaded implementation and inefficient, sequential -calls of gnunet-namestore. In the future a solution that uses the C API -would be cleaner and better. - -.. _Registering-names-using-the-FCFS-daemon: - -Registering names using the FCFS daemon -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section describes FCFSD, a daemon used to associate names with PKEY -records following a "First Come, First Served" policy. This policy means -that a certain name can not be registered again if someone registered it -already. - -The daemon can be started by using ``gnunet-namestore-fcfsd``, which -will start a simple HTTP server on localhost, using a port specified by -the ``HTTPORT`` value in its configuration. - -Communication is performed by sending GET or POST requests to specific -paths ("endpoints"), as described in the following sections. - -The daemon will always respond with data structured using the JSON -format. The fields to be expected will be listed for each endpoint. - -The only exceptions are for the "root" endpoint (i.e. ``/``) which will -return a HTML document, and two other HTML documents which will be -served when certain errors are encountered, like when requesting an -unknown endpoint. - -FCFSD GET requests -.. _Obtaining-information-from-the-daemon: - -Obtaining information from the daemon -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To query the daemon, a GET request must be sent to these endpoints, -placing parameters in the address as per the HTTP specification, like -so: - -:: - - GET /endpoint?param1=value¶m2=value - -Each endpoint will be described using its name (``/endpoint`` in the -example above), followed by the name of each parameter (like ``param1`` -and ``param2``.) - - -**Endpoint: /search** :bolditalic:`name` - This endpoint is used to query about the state of <name>, that is, - whether it is available for registration or not. - - The response JSON will contain two fields: - - - error - - - free - - ``error`` can be either the string ``"true"`` or the string - ``"false"``: when ``"true"``, it means there was an error within the - daemon and the name could not be searched at all. - - ``free`` can be either the string ``"true"`` or the string - ``"false"``: when ``"true"``, the requested name can be registered. - -FCFSD POST requests -.. _Submitting-data-to-the-daemon: - -Submitting data to the daemon -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To send data to the daemon, a POST request must be sent to these -endpoints, placing the data to submit in the body of the request, -structured using the JSON format, like so: - -:: - - POST /endpoint - Content-Type: application/json - ... - - {"param1": value1, "param2": value2, ...} - -Each endpoint will be described using its name (``/endpoint`` in the -example above), followed by the name of each JSON field (like ``param1`` -and ``param2``.) - -**Endpoint: /register** :bolditalic:`name key` - This endpoint is used to register a new association between <name> - and <key>. - - For this operation to succeed, both <NAME> and <KEY> **must not** be - registered already. - - The response JSON will contain two fields: - - - error - - - message - - ``error`` can be either the string ``"true"`` or the string - ``"false"``: when ``"true"``, it means the name could not be - registered. Clients can get the reason of the failure from the HTTP - response code or from the ``message`` field. - - ``message`` is a string which can be used by clients to let users - know the result of the operation. It might be localized to the daemon - operator's locale. - -.. _Customizing-the-HTML-output: - -Customizing the HTML output -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In some situations, the daemon will serve HTML documents instead of JSON -values. It is possible to configure the daemon to serve custom documents -instead of the ones provided with GNUnet, by setting the ``HTMLDIR`` -value in its configuration to a directory path. - -Within the provided path, the daemon will search for these three files: - -- fcfsd-index.html - -- fcfsd-notfound.html - -- fcfsd-forbidden.html - -The ``fcfsd-index.html`` file is the daemon's "homepage": operators -might want to provide information about the service here, or provide a -form with which it is possible to register a name. - -The ``fcfsd-notfound.html`` file is used primarily to let users know -they tried to access an unknown endpoint. - -The ``fcfsd-forbidden.html`` file is served to users when they try to -access an endpoint they should not access. For example, sending an -invalid request might result in this page being served. - -.. |ascension| image:: /images/ascension_ssd.png diff --git a/developers/gnsstack.rst b/developers/gnsstack.rst @@ -1,12 +0,0 @@ - -GNS and GNS Support -=================== - -The GNU Name System is a secure and censorship-resistant -alternative to the Domain Name System (DNS) in common use for -resolving domain names. - -.. toctree:: - gns/gns.rst - namecache/namecache.rst - namestore/namestore.rst diff --git a/developers/index.rst b/developers/index.rst @@ -48,7 +48,7 @@ new chapters, sections or insightful comments. util.rst architecture.rst stability.rst - corestack.rst + apis/index.rst rest-api/index.rst tutorial livingstandards diff --git a/index.rst b/index.rst @@ -7,6 +7,7 @@ Welcome to GNUnet’s documentation! about installing + subsystems/index users/index developers/index guis/index diff --git a/users/cadet.rst b/subsystems/cadet.rst diff --git a/subsystems/core.rst b/subsystems/core.rst @@ -0,0 +1,106 @@ +.. _CORE-Subsystem: + +.. index:: + double: CORE; subsystem + +CORE — GNUnet link layer +======================== + +The CORE subsystem in GNUnet is responsible for securing link-layer +communications between nodes in the GNUnet overlay network. CORE builds +on the TRANSPORT subsystem which provides for the actual, insecure, +unreliable link-layer communication (for example, via UDP or WLAN), and +then adds fundamental security to the connections: + +- confidentiality with so-called perfect forward secrecy; we use ECDHE + (`Elliptic-curve + Diffie—Hellman <http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman>`__) + powered by Curve25519 (`Curve25519 <http://cr.yp.to/ecdh.html>`__) + for the key exchange and then use symmetric encryption, encrypting + with both AES-256 + (`AES-256 <http://en.wikipedia.org/wiki/Rijndael>`__) and Twofish + (`Twofish <http://en.wikipedia.org/wiki/Twofish>`__) + +- `authentication <http://en.wikipedia.org/wiki/Authentication>`__ is + achieved by signing the ephemeral keys using Ed25519 + (`Ed25519 <http://ed25519.cr.yp.to/>`__), a deterministic variant of + ECDSA (`ECDSA <http://en.wikipedia.org/wiki/ECDSA>`__) + +- integrity protection (using SHA-512 + (`SHA-512 <http://en.wikipedia.org/wiki/SHA-2>`__) to do + encrypt-then-MAC + (`encrypt-then-MAC <http://en.wikipedia.org/wiki/Authenticated_encryption>`__)) + +- Replay (`replay <http://en.wikipedia.org/wiki/Replay_attack>`__) + protection (using nonces, timestamps, challenge-response, message + counters and ephemeral keys) + +- liveness (keep-alive messages, timeout) + +.. _Limitations: + +:index:`Limitations <CORE; limitations>` +Limitations +----------- + +CORE does not perform +`routing <http://en.wikipedia.org/wiki/Routing>`__; using CORE it is +only possible to communicate with peers that happen to already be +\"directly\" connected with each other. CORE also does not have an API +to allow applications to establish such \"direct\" connections --- for +this, applications can ask TRANSPORT, but TRANSPORT might not be able to +establish a \"direct\" connection. The TOPOLOGY subsystem is responsible +for trying to keep a few \"direct\" connections open at all times. +Applications that need to talk to particular peers should use the CADET +subsystem, as it can establish arbitrary \"indirect\" connections. + +Because CORE does not perform routing, CORE must only be used directly +by applications that either perform their own routing logic (such as +anonymous file-sharing) or that do not require routing, for example +because they are based on flooding the network. CORE communication is +unreliable and delivery is possibly out-of-order. Applications that +require reliable communication should use the CADET service. Each +application can only queue one message per target peer with the CORE +service at any time; messages cannot be larger than approximately 63 +kilobytes. If messages are small, CORE may group multiple messages +(possibly from different applications) prior to encryption. If permitted +by the application (using the `cork <http://baus.net/on-tcp_cork/>`__ +option), CORE may delay transmissions to facilitate grouping of multiple +small messages. If cork is not enabled, CORE will transmit the message +as soon as TRANSPORT allows it (TRANSPORT is responsible for limiting +bandwidth and congestion control). CORE does not allow flow control; +applications are expected to process messages at line-speed. If flow +control is needed, applications should use the CADET service. + +.. when is a peer connected +.. _When-is-a-peer-_0022connected_0022_003f: + +When is a peer \"connected\"? +----------------------------- + +In addition to the security features mentioned above, CORE also provides +one additional key feature to applications using it, and that is a +limited form of protocol-compatibility checking. CORE distinguishes +between TRANSPORT-level connections (which enable communication with +other peers) and application-level connections. Applications using the +CORE API will (typically) learn about application-level connections from +CORE, and not about TRANSPORT-level connections. When a typical +application uses CORE, it will specify a set of message types (from +``gnunet_protocols.h``) that it understands. CORE will then notify the +application about connections it has with other peers if and only if +those applications registered an intersecting set of message types with +their CORE service. Thus, it is quite possible that CORE only exposes a +subset of the established direct connections to a particular application +--- and different applications running above CORE might see different +sets of connections at the same time. + +A special case are applications that do not register a handler for any +message type. CORE assumes that these applications merely want to +monitor connections (or \"all\" messages via other callbacks) and will +notify those applications about all connections. This is used, for +example, by the ``gnunet-core`` command-line tool to display the active +connections. Note that it is also possible that the TRANSPORT service +has more active connections than the CORE service, as the CORE service +first has to perform a key exchange with connecting peers before +exchanging information about supported message types and notifying +applications about the new connection. diff --git a/subsystems/dht.rst b/subsystems/dht.rst @@ -0,0 +1,69 @@ +.. _Distributed-Hash-Table-_0028DHT_0029: + +.. index:: + double: Distributed hash table; subsystem + see: DHT; Distributed hash table + +DHT — Distributed Hash Table +============================ + +GNUnet includes a generic distributed hash table that can be used by +developers building P2P applications in the framework. This section +documents high-level features and how developers are expected to use the +DHT. We have a research paper detailing how the DHT works. Also, Nate's +thesis includes a detailed description and performance analysis (in +chapter 6). [R5N2011]_ + +.. todo:: Confirm: Are "Nate's thesis" and the "research paper" separate + entities? + +Key features of GNUnet's DHT include: + +- stores key-value pairs with values up to (approximately) 63k in size + +- works with many underlay network topologies (small-world, random + graph), underlay does not need to be a full mesh / clique + +- support for extended queries (more than just a simple 'key'), + filtering duplicate replies within the network (bloomfilter) and + content validation (for details, please read the subsection on the + block library) + +- can (optionally) return paths taken by the PUT and GET operations to + the application + +- provides content replication to handle churn + +GNUnet's DHT is randomized and unreliable. Unreliable means that there +is no strict guarantee that a value stored in the DHT is always found +— values are only found with high probability. While this is somewhat +true in all P2P DHTs, GNUnet developers should be particularly wary of +this fact (this will help you write secure, fault-tolerant code). Thus, +when writing any application using the DHT, you should always consider +the possibility that a value stored in the DHT by you or some other peer +might simply not be returned, or returned with a significant delay. Your +application logic must be written to tolerate this (naturally, some loss +of performance or quality of service is expected in this case). + +.. _Block-library-and-plugins: + +Block library and plugins +------------------------- + +.. _What-is-a-Block_003f: + +What is a Block? +^^^^^^^^^^^^^^^^ + +Blocks are small (< 63k) pieces of data stored under a key (struct +GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which +defines their data format. Blocks are used in GNUnet as units of static +data exchanged between peers and stored (or cached) locally. Uses of +blocks include file-sharing (the files are broken up into blocks), the +VPN (DNS information is stored in blocks) and the DHT (all information +in the DHT and meta-information for the maintenance of the DHT are both +stored using blocks). The block subsystem provides a few common +functions that must be available for any type of block. + + +.. [R5N2011] https://bib.gnunet.org/date.html#R5N diff --git a/subsystems/fs.rst b/subsystems/fs.rst @@ -0,0 +1,54 @@ +.. index:: + double: File sharing; subsystem + see: FS; File sharing + +.. _File_002dsharing-_0028FS_0029-Subsystem: + +FS — File sharing over GNUnet +============================= + +This chapter describes the details of how the file-sharing service +works. As with all services, it is split into an API (libgnunetfs), the +service process (gnunet-service-fs) and user interface(s). The +file-sharing service uses the datastore service to store blocks and the +DHT (and indirectly datacache) for lookups for non-anonymous +file-sharing. Furthermore, the file-sharing service uses the block +library (and the block fs plugin) for validation of DHT operations. + +In contrast to many other services, libgnunetfs is rather complex since +the client library includes a large number of high-level abstractions; +this is necessary since the FS service itself largely only operates on +the block level. The FS library is responsible for providing a +file-based abstraction to applications, including directories, meta +data, keyword search, verification, and so on. + +The method used by GNUnet to break large files into blocks and to use +keyword search is called the \"Encoding for Censorship Resistant +Sharing\" (ECRS). ECRS is largely implemented in the fs library; block +validation is also reflected in the block FS plugin and the FS service. +ECRS on-demand encoding is implemented in the FS service. + +.. note:: The documentation in this chapter is quite incomplete. + +.. _Encoding-for-Censorship_002dResistant-Sharing-_0028ECRS_0029: + +.. index:: + see: Encoding for Censorship-Resistant Sharing; ECRS + +:index:`ECRS — Encoding for Censorship-Resistant Sharing <single: ECRS>` +ECRS — Encoding for Censorship-Resistant Sharing +------------------------------------------------ + +When GNUnet shares files, it uses a content encoding that is called +ECRS, the Encoding for Censorship-Resistant Sharing. Most of ECRS is +described in the (so far unpublished) research paper attached to this +page. ECRS obsoletes the previous ESED and ESED II encodings which were +used in GNUnet before version 0.7.0. The rest of this page assumes that +the reader is familiar with the attached paper. What follows is a +description of some minor extensions that GNUnet makes over what is +described in the paper. The reason why these extensions are not in the +paper is that we felt that they were obvious or trivial extensions to +the original scheme and thus did not warrant space in the research +report. + +.. todo:: Find missing link to file system paper. diff --git a/subsystems/gns.rst b/subsystems/gns.rst @@ -0,0 +1,47 @@ +.. index:: + double: GNU Name System; subsystem + see: GNS; GNU Name System + +.. _GNU-Name-System-_0028GNS_0029: + +GNS — the GNU Name System +========================= + +The GNU Name System (GNS) is a decentralized database that enables users +to securely resolve names to values. Names can be used to identify other +users (for example, in social networking), or network services (for +example, VPN services running at a peer in GNUnet, or purely IP-based +services on the Internet). Users interact with GNS by typing in a +hostname that ends in a top-level domain that is configured in the "GNS" +section, matches an identity of the user or ends in a Base32-encoded +public key. + +Videos giving an overview of most of the GNS and the motivations behind +it is available here and here. The remainder of this chapter targets +developers that are familiar with high level concepts of GNS as +presented in these talks. + +.. todo:: Link to videos and GNS talks? + +GNS-aware applications should use the GNS resolver to obtain the +respective records that are stored under that name in GNS. Each record +consists of a type, value, expiration time and flags. + +The type specifies the format of the value. Types below 65536 correspond +to DNS record types, larger values are used for GNS-specific records. +Applications can define new GNS record types by reserving a number and +implementing a plugin (which mostly needs to convert the binary value +representation to a human-readable text format and vice-versa). The +expiration time specifies how long the record is to be valid. The GNS +API ensures that applications are only given non-expired values. The +flags are typically irrelevant for applications, as GNS uses them +internally to control visibility and validity of records. + +Records are stored along with a signature. The signature is generated +using the private key of the authoritative zone. This allows any GNS +resolver to verify the correctness of a name-value mapping. + +Internally, GNS uses the NAMECACHE to cache information obtained from +other users, the NAMESTORE to store information specific to the local +users, and the DHT to exchange data between users. A plugin API is used +to enable applications to define new GNS record types. diff --git a/developers/hostlist/hostlist.rst b/subsystems/hostlist.rst diff --git a/developers/identity/identity.rst b/subsystems/identity.rst diff --git a/subsystems/index.rst b/subsystems/index.rst @@ -0,0 +1,31 @@ +Subsystems +========== + +These services comprise a backbone of core services for +peer-to-peer applications to use. + +.. toctree:: + cadet.rst + core.rst + dht.rst + fs.rst + gns.rst + hostlist.rst + identity.rst + messenger.rst + namecache.rst + namestore.rst + nse.rst + peerinfo.rst + peerstore.rst + regex.rst + rest.rst + revocation.rst + rps.rst + setops.rst + statistics.rst + transport-ng.rst + transport.rst + vpnstack.rst + + diff --git a/developers/messenger/messenger.rst b/subsystems/messenger.rst diff --git a/developers/namecache/namecache.rst b/subsystems/namecache.rst diff --git a/developers/namestore/namestore.rst b/subsystems/namestore.rst diff --git a/developers/nse/nse.rst b/subsystems/nse.rst diff --git a/developers/peerinfo/peerinfo.rst b/subsystems/peerinfo.rst diff --git a/developers/peerstore/peerstore.rst b/subsystems/peerstore.rst diff --git a/developers/regex/regex.rst b/subsystems/regex.rst diff --git a/developers/rest/rest.rst b/subsystems/rest.rst diff --git a/developers/revocation/revocation.rst b/subsystems/revocation.rst diff --git a/developers/rps/rps.rst b/subsystems/rps.rst diff --git a/developers/set/set.rst b/subsystems/set/set.rst diff --git a/developers/seti/seti.rst b/subsystems/seti/seti.rst diff --git a/developers/setops.rst b/subsystems/setops.rst diff --git a/developers/setu/setu.rst b/subsystems/setu/setu.rst diff --git a/developers/statistics/statistics.rst b/subsystems/statistics.rst diff --git a/developers/transport-ng/transport-ng.rst b/subsystems/transport-ng.rst diff --git a/developers/transport/transport.rst b/subsystems/transport.rst diff --git a/developers/vpnstack.rst b/subsystems/vpnstack.rst diff --git a/users/index.rst b/users/index.rst @@ -26,6 +26,5 @@ welcome. fs vpn messenger - cadet configuration