aboutsummaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
authorMartin Schanzenbach <mschanzenbach@posteo.de>2021-02-23 10:42:57 +0100
committerMartin Schanzenbach <mschanzenbach@posteo.de>2021-02-23 10:42:57 +0100
commit74a234f0bb3f2ffa560805ead98e62029e4e9bbd (patch)
treea766653eeedc7ab676f7acc54a6ce97462d2224a /doc
parent03b5746e4ba53a1538eb3367b01560f9d70e45b8 (diff)
downloadgnunet-74a234f0bb3f2ffa560805ead98e62029e4e9bbd.tar.gz
gnunet-74a234f0bb3f2ffa560805ead98e62029e4e9bbd.zip
HANDBOOK: Add some info on TNG
Diffstat (limited to 'doc')
-rw-r--r--doc/handbook/chapters/developer.texi245
1 files changed, 245 insertions, 0 deletions
diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi
index 369e5327c..0d3b4739b 100644
--- a/doc/handbook/chapters/developer.texi
+++ b/doc/handbook/chapters/developer.texi
@@ -82,6 +82,7 @@ new chapters, sections or insightful comments.
82* REGEX Subsystem:: 82* REGEX Subsystem::
83* REST Subsystem:: 83* REST Subsystem::
84* RPS Subsystem:: 84* RPS Subsystem::
85* TRANSPORT-NG Subsystem::
85@end menu 86@end menu
86 87
87@node Developer Introduction 88@node Developer Introduction
@@ -9474,3 +9475,247 @@ view are sampled through the sampler from the random stream of peer IDs.
9474 9475
9475According to the theoretical analysis of Bortnikov et al. this suffices 9476According to the theoretical analysis of Bortnikov et al. this suffices
9476to keep the network connected and having random peers in the view. 9477to keep the network connected and having random peers in the view.
9478
9479@cindex TRANSPORT-NG Subsystem
9480@node TRANSPORT-NG Subsystem
9481@section TRANSPORT-NG Subsystem
9482
9483The current GNUnet TRANSPORT architecture is rooted in the GNUnet 0.4 design
9484of using plugins for the actual transmission operations and the ATS subsystem
9485to select a plugin and allocate bandwidth. The following key issues have been
9486identified with this design:
9487
9488@menu
9489* Issues with TRANSPORT::
9490* Design goals of TNG::
9491* HELLO-NG::
9492* Priorities and preferences::
9493* Communicators::
9494@end menu
9495
9496@node Issues with TRANSPORT
9497@subsection Issues with TRANSPORT
9498
9499
9500@itemize @bullet
9501@item Bugs in one plugin can affect the TRANSPORT service and other plugins.
9502 There is at least one open bug that affects sockets, where the origin is
9503 difficult to pinpoint due to the large code base.
9504@item Relevant operating system default configurations often impose a limit of
9505 1024 file descriptors per process. Thus, one plugin may impact other
9506 plugin's connectivity choices.
9507@item Plugins are required to offer bi-directional connectivity. However,
9508 firewalls (incl. NAT boxes) and physical environments sometimes only
9509 allow uni-directional connectivity, which then currently cannot be
9510 utilized at all.
9511@item Distance vector routing was implemented in 209 but shortly afterwards
9512 broken and due to the complexity of implementing it as a plugin and
9513 dealing with the resource allocation consequences was never useful.
9514@item Most existing plugins communicate completely using cleartext, exposing
9515 metad data (message size) and making it easy to fingerprint and
9516 possibly block GNUnet traffic.
9517@item Various NAT traversal methods are not supported.
9518@item The service logic is cluttered with "manipulation" support code for
9519 TESTBED to enable faking network characteristics like lossy connections
9520 or firewewalls.
9521@item Bandwidth allocation is done in ATS, requiring the duplication of state
9522 and resulting in much delayed allocation decisions. As a result,
9523 often available bandwidth goes unused. Users are expected to manually
9524 configure bandwidth limits, instead of TRANSPORT using congestion control
9525 to adapt automatically.
9526@item TRANSPORT is difficult to test and has bad test coverage.
9527@item HELLOs include an absolute expiration time. Nodes with unsynchronized
9528 clocks cannot connect.
9529@item Displaying the contents of a HELLO requires the respective plugin as the
9530 plugin-specific data is encoded in binary. This also complicates logging.
9531@end itemize
9532
9533
9534@node Design goals of TNG
9535@subsection Design goals of TNG
9536
9537In order to address the above issues, we want to:
9538
9539@itemize @bullet
9540@item Move plugins into separate processes which we shall call
9541 @emph{communicators}. Communicators connect as clients to the transport
9542 service.
9543@item TRANSPORT should be able to utilize any number of communcators to the
9544 same peer at the same time.
9545@item TRANSPORT should be responsible for fragmentation, retransmission,
9546 flow- and congestion-control. Users should no longer have to configure
9547 bandwidth limits: TRANSPORT should detect what is available and use it.
9548@item Commnunicators should be allowed to be uni-directional and unreliable.
9549 TRANSPORT shall create bi-directional channels from this whenever
9550 possible.
9551@item DV should no longer be a plugin, but part of TRANSPORT.
9552@item TRANSPORT should provide communicators help communicating (i.e. in the
9553 case of uni-directional communicators or the need for out-of-band
9554 signalling for NAT traversal). We call this functionality
9555 @emph{backchannels}.
9556@item Transport manipulation should be signalled to CORE on a per-message basis
9557 instead of an approximate bandwidth.
9558@item CORE should signal performance requirements (reliability, latency, etc.)
9559 on a per-message basis to TRANSPORT. If possible, TRANSPORT should
9560 consider those options when scheduling messages for transmission.
9561@item HELLOs should be in a humman-readable format with monotonic time
9562 expirations.
9563@end itemize
9564
9565The new architecture is planned as follows:
9566
9567@verbatim
9568
9569 APPLICATIONS
9570 |
9571 |
9572CORE -- TRANSPORT -- COMMUNICATOR(S) -- NAT
9573 \ /
9574 \ /
9575 PEERSTORE
9576
9577@end verbatim
9578
9579TRANSPORT's main objective is to establish bi-directional virtual links using a
9580variety of possibly uni-directional communicators. Links undergo the following
9581steps:
9582
9583@enumerate
9584@item Communicator informs TRANSPORT A that a queue (direct neighbour) is
9585 available, or equivalently TRANSPORT A discovers a (DV) path to a target
9586 B.
9587@item TRANSPORT A sends a challenge to the target peer, trying to confirm that
9588 the peer can receive. FIXME: This is not implemented properly for DV.
9589 Here we should really take a validated DVH and send a challenge exactly
9590 down that path!
9591@item The other TRANSPORT, TRANSPORT B, receives the challenge, and sends back
9592 a response, possibly using a dierent path. If TRANSPORT B does not yet
9593 have a virtual link to A, it must try to establish a virtual link.
9594@item Upon receiving the response, TRANSPORT A creates the virtual link. If the
9595 response included a challenge, TRANSPORT A must respond to this challenge
9596 as well, eectively re-creating the TCP 3-way handshake (just with longer
9597 challenge values).
9598@end enumerate
9599
9600@node HELLO-NG
9601@subsection HELLO-NG
9602
9603HELLOs change in three ways. First of all, communicators encode the respective
9604addresses in a human-readable URL-like string. This way, we do no longer
9605require the communicator to print the contents of a HELLO.
9606Second, HELLOs no longer contain an expiration time, only a creation time.
9607The receiver must only compare the respective absolute values. So given a HELLO
9608from the same sender with a larger creation time, then the old one is no longer
9609valid. This also obsoletes the need for the gnunet-hello binary to set HELLO
9610expiration times to never.
9611Third, a peer no longer generates one big HELLO that always contains all of the
9612addresses. Instead, each address is signed individually and shared only over
9613the address scopes where it makes sense to share the address. In particular,
9614care should be taken to not share MACs across the Internet and confine their
9615use to the LAN.
9616As each address is signed separately, having multiple addresses valid at the
9617same time (given the new creation time expiration logic) requires that those
9618addresses must have exactly the same creation time.
9619Whenever that monotonic time is increased, all addresses must be re-signed and
9620re-distributed.
9621
9622@node Priorities and preferences
9623@subsection Priorities and preferences
9624
9625In the new design, TRANSPORT adopts a feature (which was previously already
9626available in CORE) of the MQ API to allow applications to specify priorities and
9627preferences per message (or rather, per MQ envelope).
9628The (updated) MQ API allows applications to specify one of four priority levels
9629as well as desired preferences for transmission by setting options on an
9630envelope. These preferences currently are:
9631
9632@itemize @bullet
9633@item GNUNET_MQ_PREF_UNRELIABLE: Disables TRANSPORT waiting for ACKS on
9634 unreliable channels like UDP. Now it is fire and forget. These messages
9635 then cannot be used for RTT estimates either.
9636@item GNUNET_MQ_PREF_LOW_LATENCY: Directs TRANSPORT to select the
9637 lowest-latency transmission choices possible.
9638@item GNUNET_MQ_PREF_CORK_ALLOWED: Allows TRANSPORT to delay transmission to
9639 group the message with other messages into a larger batch to reduce the
9640 number of packets sent.
9641@item GNUNET_MQ_PREF_GOODPUT: Directs TRANSPORT to select the highest goodput
9642 channel available.
9643@item GNUNET_MQ_PREF_OUT_OF_ORDER: Allows TRANSPORT to reorder the messages as
9644 it sees fit, otherwise TRANSPORT should attempt to preserve transmission
9645 order.
9646@end itemize
9647
9648Each MQ envelope is always able to store those options (and the priority), and
9649in the future this uniform API will be used by TRANSPORT, CORE, CADET and
9650possibly other subsystems that send messages (like LAKE).
9651When CORE sets preferences and priorities, it is supposed to respect the
9652preferences and priorities it is given from higher layers. Similarly, CADET also
9653simply passes on the preferences and priorities of the layer above CADET. When a
9654layer combines multiple smaller messages into one larger transmission, the
9655@code{GNUNET_MQ_env_combine_options()} should be used to calculate options for
9656the combined message. We note that the exact semantics of the options may differ
9657by layer. For example, CADET will always strictly implement reliable and
9658in-order delivery of messages, while the same options are only advisory for
9659TRANSPORT and CORE: they should try (using ACKs on unreliable communicators,
9660not changing the message order themselves), but if messages are lost anyway
9661(i.e. because a TCP is dropped in the middle), or if messages are reordered
9662(i.e. because they took dierent paths over the network and arrived in a
9663different order) TRANSPORT and CORE do not have to correct this. Whether a
9664preference is strict or loose is thus dened by the respective layer.
9665
9666@node Communicators
9667@subsection Communicators
9668
9669The API for communicators is defined in
9670@code{gnunet_transport_communication_service.h}.
9671Each communicator must specify its (global) communication characteristics, which
9672for now only say whether the communication is reliable (e.g. TCP, HTTPS) or
9673unreliable (e.g. UDP, WLAN). Each communicator must specify a unique address
9674prex, or NULL if the communicator cannot establish outgoing connections (i.e.
9675is only acting as a TCP server).
9676A communicator must tell TRANSPORT which addresses it is reachable under.
9677Addresses may be added or removed at any time. A communicator may have zero
9678addresses (transmission only).
9679Addresses do not have to match the address prefix.
9680
9681TRANSPORT may ask a communicator to try to connect to another address.
9682TRANSPORT will only ask for connections where the address matches the
9683communicator's address prefix that was provided when the connection was
9684established. Communicators should then attempt to establish a connection.
9685No response is provided to TRANSPORT service on failure. The TRANSPORT service
9686has to ask the communicator explicitly to retry.
9687
9688If a communicator succeeds in establishing an outgoing connection for
9689transmission, or if a communicator receives an incoming bi-directional
9690connection, the communicator must inform the TRANSPORT service that a message
9691queue (MQ) for transmission is now available. For that MQ, the communicator must
9692provide the peer identity claimed by the other end, a human-readable address
9693(for debugging) and a maximum transfer unit (MTU). A MTU of zero means sending
9694is not supported, SIZE_MAX should be used for no MTU. The communicator should
9695also tell TRANSPORT what network type is used for the queue. The communicator
9696may tell TRANSPORT anytime that the queue was deleted and is no longer
9697available.
9698
9699The communicator API also provides for flow control. First, communicators
9700exhibit back-pressure on TRANSPORT: the number of messages TRANSPORT may add to
9701a queue for transmission will be limited. So by not draining the transmission
9702queue, back-pressure is provided to TRANSPORT. In the other direction,
9703communicators may allow TRANSPORT to give back-pressure towards the
9704communicator by providing a non-NULL
9705@code{GNUNET_TRANSPORT_MessageCompletedCallback}
9706argument to the @code{GNUNET_TRANSPORT_communicator_receive} function. In this
9707case, TRANSPORT will only invoke this function once it has processed the message
9708and is ready to receive more. Communicators should then limit how much traffic
9709they receive based on this backpressure. Note that communicators do not have to
9710provide a @code{GNUNET_TRANSPORT_MessageCompletedCallback};
9711for example, UDP cannot support back-pressure due to the nature of the UDP
9712protocol. In this case, TRANSPORT will implement its own TRANSPORT-to-TRANSPORT
9713flow control to reduce the sender's data rate to acceptable levels.
9714
9715TRANSPORT may notify a communicator about backchannel messages TRANSPORT
9716received from other peers for this communicator. Similarly, communicators can
9717ask TRANSPORT to try to send a backchannel message to other communicators of
9718other peers. The semantics of the backchannel message are up to the
9719communicators which use them.
9720TRANSPORT may fail transmitting backchannel messages, and TRANSPORT will not
9721attempt to retransmit them.