11 files changed, 792 insertions, 76 deletions

diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi
index d9c92247b..2da262b34 100644
--- a/doc/handbook/chapters/developer.texi
+++ b/doc/handbook/chapters/developer.texi
@@ -7743,6 +7743,7 @@ record types.
 * The GNS Client-Service Protocol::
 * Hijacking the DNS-Traffic using gnunet-service-dns::
 * Serving DNS lookups via GNS on W32::
+* Importing DNS Zones into GNS::
 @end menu
 
 @node libgnunetgns
@@ -8073,6 +8074,119 @@ applications that use alternative means of resolving names (such as
 sending queries to a DNS server directly by themselves).
 This includes some of well known utilities, like "ping" and "nslookup".
 
+@node Importing DNS Zones into GNS
+@subsection Importing DNS Zones into GNS
+
+@c %**end of header
+
+This section discusses the challenges and problems faced when writing the
+ascension tool. It also takes a look at possible improvements in the future.
+
+@menu
+* Conversions between DNS and GNS::
+* DNS Zone Size::
+* Performance::
+@end menu
+
+@node Conversions between DNS and GNS
+@subsubsection Conversions between DNS and GNS
+
+The differences between the two name systems lies in the details
+and is not always transparent. For instance an SRV record is converted to a
+gnunet only BOX record.
+
+This is done by converting to a BOX record from an existing SRV record:
+
+@example
+# SRV
+# _service._proto.name. TTL class SRV priority weight port target
+_sip._tcp.example.com. 14000 IN SRV     0 0 5060 www.example.com.
+# BOX
+# TTL BOX flags port protocol recordtype priority weight port target
+14000 BOX n 5060 6 33 0 0 5060 www.example.com
+@end example
+
+Other records that have such a transformation is the MX record type, as well as
+the SOA record type.
+
+Transformation of a SOA record into GNS works as described in the following
+example. Very important to note are the rname and mname keys.
+@example
+# BIND syntax for a clean SOA record
+@   IN SOA master.example.com. hostmaster.example.com. (
+    2017030300 ; serial
+    3600       ; refresh
+    1800       ; retry
+    604800     ; expire
+    600 )      ; ttl
+# Recordline for adding the record
+gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \
+    mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s
+@end example
+
+The transformation of MX records is done in a simple way.
+@example
+# mail.example.com. 3600 IN MX 10 mail.example.com.
+gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
+@end example
+
+Finally, one of the biggest struggling points were the NS records that are found
+in top level domain zones. The intended behaviour for those is to add GNS2DNS
+records for those so that gnunet-gns can resolve records for those domains on
+its own. This requires migration of the DNS GLUE records as well, provided that
+they are within the same zone.
+
+A solution was found by creating a hierarchical zone structure in GNS and linking
+the zones using PKEY records to one another. This allows the resolution of the
+nameservers to work within GNS while not taking control over unwanted zones.
+
+@node DNS Zone Size
+@subsubsection DNS Zone Size
+
+Another very big problem exists with very large zones. When migrating a small
+zone the delay between adding of records and their expiry is negligible. However
+when working with a TLD zone that has more that 1 million records this delay
+becomes a problem.
+
+Records will start to expire well before the zone has finished migrating. This
+causes unwanted anomalies when trying to resolve records.
+
+A good solution has not been found yet. One of the idea that floated around was
+that the records should be added with the s (shadow) flag to keep the records
+resolvable even if they expired. However this would introduce the problem of how
+to detect if a record has been removed from the zone and would require deletion
+of said record(s).
+
+Another problem that still persists is how to refresh records. Expired records
+are still displayed when calling gnunet-namestore but do not resolve with
+gnunet-gns. When doing incremental zone transfers this becomes especially
+apparent.
+
+@node Performance
+@subsubsection Performance
+The performance when migrating a zone using the ascension tool is limited by a
+handful of factors. First of all ascension is written in python3 and calls the
+CLI tools of gnunet. Furthermore all the records that are added to the same
+label are signed using the zones private key. This signing operation is very
+resource heavy and was optimized during development by adding the '-R'
+(Recordline) option to gnunet-namestore. This allows to add multiple records
+at once using the CLI.
+
+The result of this was a much faster migration of TLD zones, as most records
+with the same label have two name servers.
+
+Another improvement that could be made is with the addition of multiple threads
+when opening the gnunet CLI tools. This could be implemented by simply creating
+more workers in the program but performance improvements were not tested.
+
+During the entire development of the ascension tool sqlite was used as a
+database backend. Other backends need to be tested in the future.
+
+In conclusion there are many bottlenecks still around in the program, namely the
+signing process and the single threaded implementation. In the future a solution
+that uses the c api would be cleaner and better.
+
+
 @cindex GNS Namecache
 @node GNS Namecache
 @section GNS Namecache
@@ -8768,14 +8882,14 @@ regular expressions.
 
 Using the REST subsystem, you can expose REST-based APIs or services.
 The REST service is designed as a pluggable architecture.
-To create a new REST endpoint, simply add a library in the form 
+To create a new REST endpoint, simply add a library in the form
 ``plugin_rest_*''.
 The REST service will automatically load all REST plugins on startup.
 
 @strong{Configuration}
 
 The REST service can be configured in various ways.
-The reference config file can be found in 
+The reference config file can be found in
 @file{src/rest/rest.conf}:
 @example
 [rest]
diff --git a/doc/handbook/chapters/installation.texi b/doc/handbook/chapters/installation.texi
index c05f776f2..bdff20802 100644
--- a/doc/handbook/chapters/installation.texi
+++ b/doc/handbook/chapters/installation.texi
@@ -35,7 +35,7 @@ The mandatory libraries and applications are
 @item libextractor
 @item libidn
 @item libmicrohttpd @geq{}0.9.52
-@item libnss 
+@item libnss
 @item libunistring
 @item gettext
 @item glibc
@@ -1427,6 +1427,7 @@ and @code{[transport-https_client]} section of the configuration:
 * GNS Proxy Setup::
 * Setup of the GNS CA::
 * Testing the GNS setup::
+* Migrating existing DNS zones into GNS::
 @end menu
 
 
@@ -1693,6 +1694,34 @@ configured proxy) should give you a valid SSL certificate for
 @c FIXME: Image does not exist, create it or save it from Drupal?
 @c @image{images/gnunethpgns.png,5in,, picture of homepage.gnu in Webbrowser}
 
+@node Migrating existing DNS zones into GNS
+@subsubsection Migrating existing DNS zones into GNS
+
+To migrate an existing zone into GNS use the ascension tool.
+
+Ascension transfers entire zones into GNS by doing incremental zone transfers
+and then adding the records to GNS.
+
+You can find the source code here: @code{https://gnunet.org/git/ascension.git/}
+
+The software can be installed into a python virtual environment like this:
+@example
+$ python3 -m venv .venv
+$ source .venv/bin/activate
+$ python3 setup.py install
+@end example
+
+Or installed globally like this (not recommended):
+@example
+$ sudo python3 setup.py install
+@end example
+
+The advantage of using a virtual environment is, that all the dependencies can
+be installed separately in different versions without touching your existing
+python installation and its dependencies.
+
+Using the tool is discussed in the user section of the documentation.
+
 
 @node Configuring the GNUnet VPN
 @subsection Configuring the GNUnet VPN
diff --git a/doc/handbook/chapters/user.texi b/doc/handbook/chapters/user.texi
index ea41bbb6c..0703adafc 100644
--- a/doc/handbook/chapters/user.texi
+++ b/doc/handbook/chapters/user.texi
@@ -1453,6 +1453,7 @@ the user. This results in non-unique name-value mappings as
 * Using Public Keys as Top Level Domains::
 * Resource Records in GNS::
 * Synchronizing with legacy DNS::
+* Migrating an existing DNS zone into GNS::
 @end menu
 
 
@@ -1891,6 +1892,43 @@ DNS zone and that do not have a local gns service in use, it
 is thus advisable to disable the namecache by setting the
 option ``DISABLE'' to ``YES'' in section ``[namecache]''.
 
+@node Migrating an existing DNS zone into GNS
+@subsection Migrating an existing DNS zone into GNS
+
+After installing the tool according to the README file you have the following
+options:
+@example
+Ascension
+
+Usage:
+    ascension.py <domain> [-d]
+    ascension.py <domain> -p <port> [-d]
+    ascension.py <domain> -ns <transferns> [-d]
+    ascension.py <domain> -ns <transferns> -p <port> [-d]
+    ascension.py -h | --help
+    ascension.py -v | --version
+
+Options:
+    <port>          Port for zone transfer
+    <domain>        Domain to migrate
+    <transferns>    DNS Server that does the zone transfer
+    -d --debug      Enable debugging
+    -h --help       Show this screen.
+    -v --version    Show version.
+@end example
+
+To migrate the Syrian top level domain - one of the few top level domains that
+still supports zone transfers - use the following command:
+
+@example
+$ ascension sy. -ns ns1.tld.sy.
+@end example
+
+The program will continue to run as a daemon and update once the refresh time
+specified in the zones SOA record has elapsed.
+
+At this point you might want to write for example a systemd unit file to start
+and enable the service, so that your zone is migrated automatically.
 
 @node re@:claim Identity Provider
 @section re@:claim Identity Provider
@@ -1962,7 +2000,7 @@ Further, the "ticket" can be re-used later to retrieve up-to-date attributes in
 
 To list all given authorizations (tickets) you can execute:
 @example
-$ gnunet-reclaim -e "friend" -T (TODO there is only a C and REST API for this at this time) 
+$ gnunet-reclaim -e "friend" -T (TODO there is only a C and REST API for this at this time)
 @end example
 
 
@@ -2292,3 +2330,4 @@ protocol (--tcp or --udp) and you will again receive an IP address
 that will terminate at the respective peer's service.
 
 
+
diff --git a/src/Makefile.am b/src/Makefile.am
index ffb39260c..5fd65141b 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -94,11 +94,11 @@ SUBDIRS = \
   datacache \
   datastore \
   template \
+  peerstore \
   ats \
   nat \
   nat-auto \
   fragmentation \
-  peerstore \
   transport \
   ats-tool \
   peerinfo-tool \
diff --git a/src/ats/Makefile.am b/src/ats/Makefile.am
index f113d96e7..4f2a93227 100644
--- a/src/ats/Makefile.am
+++ b/src/ats/Makefile.am
@@ -82,6 +82,8 @@ libgnunet_plugin_ats2_simple_la_SOURCES = \
   plugin_ats2_simple.c
 libgnunet_plugin_ats2_simple_la_LIBADD = \
   $(top_builddir)/src/hello/libgnunethello.la \
+  $(top_builddir)/src/peerstore/libgnunetpeerstore.la \
+  $(top_builddir)/src/nt/libgnunetnt.la \
   $(top_builddir)/src/statistics/libgnunetstatistics.la \
   $(top_builddir)/src/util/libgnunetutil.la \
   $(LTLIBINTL)
diff --git a/src/ats/test_ats2_lib.conf b/src/ats/test_ats2_lib.conf
new file mode 100644
index 000000000..e24f5ea68
--- /dev/null
+++ b/src/ats/test_ats2_lib.conf
@@ -0,0 +1,13 @@
+@INLINE@ ../../contrib/conf/gnunet/no_forcestart.conf
+@INLINE@ ../../contrib/conf/gnunet/no_autostart_above_core.conf
+
+[ats]
+BINARY = gnunet-service-ats-new
+
+[core]
+START_ON_DEMAND = NO
+IMMEDIATE_START = NO
+
+[transport]
+START_ON_DEMAND = NO
+IMMEDIATE_START = NO
diff --git a/src/include/gnunet_container_lib.h b/src/include/gnunet_container_lib.h
index e7bd4113d..fee851e1c 100644
--- a/src/include/gnunet_container_lib.h
+++ b/src/include/gnunet_container_lib.h
@@ -2011,7 +2011,7 @@ GNUNET_CONTAINER_multihashmap32_iterator_destroy (struct GNUNET_CONTAINER_MultiH
   if (NULL == (element)->next_##mdll) \
     (tail) = (element); \
   else \
-    (element)->next->prev_##mdll = (element); } while (0)
+    (element)->next_##mdll->prev_##mdll = (element); } while (0)
 
 
 /**
diff --git a/src/include/gnunet_mq_lib.h b/src/include/gnunet_mq_lib.h
index 3f67dc365..2a459636a 100644
--- a/src/include/gnunet_mq_lib.h
+++ b/src/include/gnunet_mq_lib.h
@@ -528,6 +528,51 @@ struct GNUNET_MQ_MessageHandler
 
 
 /**
+ * Insert code for a "check_" function that verifies that
+ * a given variable-length message received over the network
+ * is followed by another variable-length message that fits
+ * exactly with the given size.  If the message @a m
+ * is not followed by another `struct GNUNET_MessageHeader`
+ * with a size that adds up to the total size, an error is logged
+ * and the function is returned with #GNUNET_NO.
+ *
+ * @param an IPC message with proper type to determine
+ *  the size, starting with a `struct GNUNET_MessageHeader`
+ */
+#define GNUNET_MQ_check_boxed_message(m)                \
+  {                                                     \
+    const struct GNUNET_MessageHeader *inbox =          \
+      (const struct GNUNET_MessageHeader *) &m[1];      \
+    const struct GNUNET_MessageHeader *hdr =            \
+      (const struct GNUNET_MessageHeader *) m;          \
+    uint16_t slen = ntohs (hdr->size) - sizeof (*m);    \
+    if ( (slen < sizeof (struct GNUNET_MessageHeader))||\
+         (slen != ntohs (inbox->size)) )                \
+    {                                                   \
+      GNUNET_break (0);                                 \
+      return GNUNET_NO;                                 \
+    }                                                   \
+  }
+
+
+/**
+ * Call the message message handler that was registered
+ * for the type of the given message in the given @a handlers list.
+ *
+ * This function is indended to be used for the implementation
+ * of message queues.
+ *
+ * @param handlers a set of handlers
+ * @param mh message to dispatch
+ * @return #GNUNET_OK on success, #GNUNET_NO if no handler matched,
+ *         #GNUNET_SYSERR if message was rejected by check function
+ */
+int
+GNUNET_MQ_handle_message (const struct GNUNET_MQ_MessageHandler *handlers,
+                          const struct GNUNET_MessageHeader *mh);
+
+
+/**
  * Create a new envelope.
  *
  * @param mhp message header to store the allocated message header in, can be NULL
diff --git a/src/include/gnunet_protocols.h b/src/include/gnunet_protocols.h
index 8593005d7..a8d716b3f 100644
--- a/src/include/gnunet_protocols.h
+++ b/src/include/gnunet_protocols.h
@@ -3125,6 +3125,11 @@ extern "C"
 #define GNUNET_MESSAGE_TYPE_TRANSPORT_DV_BOX 1219
 
 /**
+ * Transport affirming receipt of an ephemeral key.
+ */ 
+#define GNUNET_MESSAGE_TYPE_TRANSPORT_EPHEMERAL_CONFIRMATION 1220
+
+/**
  * Message sent to indicate to the transport that a monitor
  * wants to observe certain events.
  */
diff --git a/src/transport/gnunet-service-tng.c b/src/transport/gnunet-service-tng.c
index 8febbdfff..3cccf5173 100644
--- a/src/transport/gnunet-service-tng.c
+++ b/src/transport/gnunet-service-tng.c
@@ -33,7 +33,7 @@
  *       transport-to-transport traffic)
  *
  * Implement:
- * - manage fragmentation/defragmentation, retransmission, track RTT, loss, etc.
+ * - manage defragmentation, retransmission, track RTT, loss, etc.
  *
  * Easy:
  * - use ATS bandwidth allocation callback and schedule transmissions!
@@ -165,8 +165,8 @@ struct TransportBackchannelEncapsulationMessage
 
 
 /**
- * Message by which a peer confirms that it is using an
- * ephemeral key.
+ * Body by which a peqer confirms that it is using an ephemeral
+ * key.
  */
 struct EphemeralConfirmation
 {
@@ -192,6 +192,37 @@ struct EphemeralConfirmation
 
 
 /**
+ * Message by which a peqer confirms that it is using an ephemeral
+ * key.
+ */
+struct EphemeralConfirmationMessage
+{
+
+  /**
+   * Message header, type is #GNUNET_MESSAGE_TYPE_TRANSPORT_EPHEMERAL_CONFIRMATION
+   */
+  struct GNUNET_MessageHeader header;
+
+  /**
+   * Must be zero.
+   */  
+  uint32_t reserved;
+  
+  /**
+   * How long is this signature over the ephemeral key
+   * valid?
+   */
+  struct GNUNET_TIME_AbsoluteNBO ephemeral_validity;
+
+  /**
+   * Ephemeral key setup by the sender for @e target, used
+   * to encrypt the payload.
+   */
+  struct GNUNET_CRYPTO_EcdhePublicKey ephemeral_key;
+};
+
+
+/**
  * Plaintext of the variable-size payload that is encrypted
  * within a `struct TransportBackchannelEncapsulationMessage`
  */
@@ -863,7 +894,12 @@ struct PendingMessage
    * Kept in a MDLL of messages from this @a cpm  (if @e pmt is #PMT_FRAGMENT_BOX)
    */
   struct PendingMessage *prev_frag;
-    
+
+  /**
+   * This message, reliability boxed. Only possibly available if @e pmt is #PMT_CORE.
+   */ 
+  struct PendingMessage *bpm;
+  
   /**
    * Target of the request.
    */
@@ -1798,6 +1834,36 @@ free_fragment_tree (struct PendingMessage *root)
 
 
 /**
+ * Release memory associated with @a pm and remove @a pm from associated
+ * data structures.  @a pm must be a top-level pending message and not
+ * a fragment in the tree.  The entire tree is freed (if applicable).
+ *
+ * @param pm the pending message to free
+ */
+static void
+free_pending_message (struct PendingMessage *pm)
+{
+  struct TransportClient *tc = pm->client;
+  struct Neighbour *target = pm->target;
+
+  if (NULL != tc)
+  {
+    GNUNET_CONTAINER_MDLL_remove (client,
+                                  tc->details.core.pending_msg_head,
+                                  tc->details.core.pending_msg_tail,
+                                  pm);
+  }
+  GNUNET_CONTAINER_MDLL_remove (neighbour,
+                                target->pending_msg_head,
+                                target->pending_msg_tail,
+                                pm);
+  free_fragment_tree (pm);
+  GNUNET_free_non_null (pm->bpm);
+  GNUNET_free (pm);
+}
+
+
+/**
  * Send a response to the @a pm that we have processed a
  * "send" request with status @a success. We
  * transmitted @a bytes_physical on the actual wire.
@@ -1829,17 +1895,8 @@ client_send_response (struct PendingMessage *pm,
     som->peer = target->pid;
     GNUNET_MQ_send (tc->mq,
                     env);
-    GNUNET_CONTAINER_MDLL_remove (client,
-                                  tc->details.core.pending_msg_head,
-                                  tc->details.core.pending_msg_tail,
-                                  pm);
   }
-  GNUNET_CONTAINER_MDLL_remove (neighbour,
-                                target->pending_msg_head,
-                                target->pending_msg_tail,
-                                pm);
-  free_fragment_tree (pm);
-  GNUNET_free (pm);
+  free_pending_message (pm);
 }
 
 
@@ -2176,36 +2233,291 @@ handle_del_address (void *cls,
 
 
 /**
+ * Context from #handle_incoming_msg().  Closure for many
+ * message handlers below.
+ */
+struct CommunicatorMessageContext
+{
+  /**
+   * Which communicator provided us with the message.
+   */
+  struct TransportClient *tc;
+
+  /**
+   * Additional information for flow control and about the sender.
+   */
+  struct GNUNET_TRANSPORT_IncomingMessage im;
+};
+
+
+/**
+ * Send ACK to communicator (if requested) and free @a cmc.
+ *
+ * @param cmc context for which we are done handling the message
+ */
+static void
+finish_cmc_handling (struct CommunicatorMessageContext *cmc)
+{
+  // FIXME: if (0 != ntohl (im->fc_on)) => send ACK when done to communicator for flow control!
+  GNUNET_SERVICE_client_continue (cmc->tc->client);
+
+  GNUNET_free (cmc);
+}
+
+
+/**
+ * Communicator gave us an unencapsulated message to pass
+ * as-is to CORE.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param mh the message that was received
+ */
+static void
+handle_raw_message (void *cls,
+                    const struct GNUNET_MessageHeader *mh)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a fragment box.  Check the message.
+ *
+ * @param cls a `struct CommunicatorMessageContext`
+ * @param fb the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
+ */
+static int
+check_fragment_box (void *cls,
+                    const struct TransportFragmentBox *fb)
+{
+  // FIXME! check that off + size-of-payload <= total-length!
+  return GNUNET_YES;
+}
+
+
+/**
+ * Communicator gave us a fragment.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param fb the message that was received
+ */
+static void
+handle_fragment_box (void *cls,
+                     const struct TransportFragmentBox *fb)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a fragment acknowledgement.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param fa the message that was received
+ */
+static void
+handle_fragment_ack (void *cls,
+                     const struct TransportFragmentAckMessage *fa)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a reliability box.  Check the message.
+ *
+ * @param cls a `struct CommunicatorMessageContext`
+ * @param rb the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
+ */
+static int
+check_reliability_box (void *cls,
+                       const struct TransportReliabilityBox *rb)
+{
+  GNUNET_MQ_check_boxed_message (rb);
+  return GNUNET_YES;
+}
+
+
+/**
+ * Communicator gave us a reliability box.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param rb the message that was received
+ */
+static void
+handle_reliability_box (void *cls,
+                        const struct TransportReliabilityBox *rb)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a reliability ack.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param ra the message that was received
+ */
+static void
+handle_reliability_ack (void *cls,
+                        const struct TransportReliabilityAckMessage *ra)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a backchannel encapsulation.  Check the message.
+ *
+ * @param cls a `struct CommunicatorMessageContext`
+ * @param be the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
+ */
+static int
+check_backchannel_encapsulation (void *cls,
+                                 const struct TransportBackchannelEncapsulationMessage *be)
+{
+  // FIXME: do work!
+  return GNUNET_YES;
+}
+
+
+/**
+ * Communicator gave us a backchannel encapsulation.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param be the message that was received
+ */
+static void
+handle_backchannel_encapsulation (void *cls,
+                                  const struct TransportBackchannelEncapsulationMessage *be)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us an ephemeral confirmation.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param ec the message that was received
+ */
+static void
+handle_ephemeral_confirmation (void *cls,
+                               const struct EphemeralConfirmationMessage *ec)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a DV learn message.  Check the message.
+ *
+ * @param cls a `struct CommunicatorMessageContext`
+ * @param dvl the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
+ */
+static int
+check_dv_learn (void *cls,
+                const struct TransportDVLearn *dvl)
+{
+  // FIXME: do work!
+  return GNUNET_YES;
+}
+
+
+/**
+ * Communicator gave us a DV learn message.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param dvl the message that was received
+ */
+static void
+handle_dv_learn (void *cls,
+                 const struct TransportDVLearn *dvl)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
+ * Communicator gave us a DV box.  Check the message.
+ *
+ * @param cls a `struct CommunicatorMessageContext`
+ * @param dvb the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
+ */
+static int
+check_dv_box (void *cls,
+              const struct TransportDVBox *dvb)
+{
+  // FIXME: do work!
+  return GNUNET_YES;
+}
+
+
+/**
+ * Communicator gave us a DV box.  Process the request.
+ *
+ * @param cls a `struct CommunicatorMessageContext` (must call #finish_cmc_handling() when done)
+ * @param dvb the message that was received
+ */
+static void
+handle_dv_box (void *cls,
+               const struct TransportDVBox *dvb)
+{
+  struct CommunicatorMessageContext *cmc = cls;
+  
+  // FIXME: do work!
+  finish_cmc_handling (cmc);
+}
+
+
+/**
  * Client notified us about transmission from a peer.  Process the request.
  *
- * @param cls the client
+ * @param cls a `struct TransportClient` which sent us the message
  * @param obm the send message that was sent
+ * @return #GNUNET_YES if message is well-formed
  */
 static int
 check_incoming_msg (void *cls,
                     const struct GNUNET_TRANSPORT_IncomingMessage *im)
 {
   struct TransportClient *tc = cls;
-  uint16_t size;
-  const struct GNUNET_MessageHeader *obmm;
 
   if (CT_COMMUNICATOR != tc->type)
   {
     GNUNET_break (0);
     return GNUNET_SYSERR;
   }
-  size = ntohs (im->header.size) - sizeof (*im);
-  if (size < sizeof (struct GNUNET_MessageHeader))
-  {
-    GNUNET_break (0);
-    return GNUNET_SYSERR;
-  }
-  obmm = (const struct GNUNET_MessageHeader *) &im[1];
-  if (size != ntohs (obmm->size))
-  {
-    GNUNET_break (0);
-    return GNUNET_SYSERR;
-  }
+  GNUNET_MQ_check_boxed_message (im);
   return GNUNET_OK;
 }
 
@@ -2213,7 +2525,6 @@ check_incoming_msg (void *cls,
 /**
  * Incoming meessage.  Process the request.
  *
- * @param cls the client
  * @param im the send message that was received
  */
 static void
@@ -2221,8 +2532,61 @@ handle_incoming_msg (void *cls,
                      const struct GNUNET_TRANSPORT_IncomingMessage *im)
 {
   struct TransportClient *tc = cls;
+  struct CommunicatorMessageContext *cmc = GNUNET_new (struct CommunicatorMessageContext);
+  struct GNUNET_MQ_MessageHandler handlers[] = {
+    GNUNET_MQ_hd_var_size (fragment_box,
+                           GNUNET_MESSAGE_TYPE_TRANSPORT_FRAGMENT,
+                           struct TransportFragmentBox,
+                           &cmc),
+    GNUNET_MQ_hd_fixed_size (fragment_ack,
+                             GNUNET_MESSAGE_TYPE_TRANSPORT_FRAGMENT_ACK,
+                             struct TransportFragmentAckMessage,
+                             &cmc),
+    GNUNET_MQ_hd_var_size (reliability_box,
+                           GNUNET_MESSAGE_TYPE_TRANSPORT_RELIABILITY_BOX,
+                           struct TransportReliabilityBox,
+                           &cmc),
+    GNUNET_MQ_hd_fixed_size (reliability_ack,
+                             GNUNET_MESSAGE_TYPE_TRANSPORT_RELIABILITY_ACK,
+                             struct TransportReliabilityAckMessage,
+                             &cmc),
+    GNUNET_MQ_hd_var_size (backchannel_encapsulation,
+                           GNUNET_MESSAGE_TYPE_TRANSPORT_BACKCHANNEL_ENCAPSULATION,
+                           struct TransportBackchannelEncapsulationMessage,
+                           &cmc),
+    GNUNET_MQ_hd_fixed_size (ephemeral_confirmation,
+                             GNUNET_MESSAGE_TYPE_TRANSPORT_EPHEMERAL_CONFIRMATION,
+                             struct EphemeralConfirmationMessage,
+                             &cmc),
+    GNUNET_MQ_hd_var_size (dv_learn,
+                           GNUNET_MESSAGE_TYPE_TRANSPORT_DV_LEARN,
+                           struct TransportDVLearn,
+                           &cmc),
+    GNUNET_MQ_hd_var_size (dv_box,
+                           GNUNET_MESSAGE_TYPE_TRANSPORT_DV_BOX,
+                           struct TransportDVBox,
+                           &cmc),
+    GNUNET_MQ_handler_end()
+  };
+  int ret;
 
-  GNUNET_SERVICE_client_continue (tc->client);
+  cmc->tc = tc;
+  cmc->im = *im;
+  ret = GNUNET_MQ_handle_message (handlers,
+                                  (const struct GNUNET_MessageHeader *) &im[1]);
+  if (GNUNET_SYSERR == ret)
+  {
+    GNUNET_break (0);
+    GNUNET_SERVICE_client_drop (tc->client);
+    GNUNET_free (cmc);
+    return;
+  }
+  if (GNUNET_NO == ret)
+  {
+    /* unencapsulated 'raw' message */
+    handle_raw_message (&cmc,
+                        (const struct GNUNET_MessageHeader *) &im[1]);
+  }
 }
 
 
@@ -2269,6 +2633,23 @@ tracker_update_in_cb (void *cls)
 
 
 /**
+ * If necessary, generates the UUID for a @a pm
+ *
+ * @param pm pending message to generate UUID for.
+ */
+static void
+set_pending_message_uuid (struct PendingMessage *pm)
+{
+  if (pm->msg_uuid_set)
+    return;
+  GNUNET_CRYPTO_random_block (GNUNET_CRYPTO_QUALITY_NONCE,
+                              &pm->msg_uuid,
+                              sizeof (pm->msg_uuid));
+  pm->msg_uuid_set = GNUNET_YES;
+}
+
+
+/**
  * Fragment the given @a pm to the given @a mtu.  Adds 
  * additional fragments to the neighbour as well. If the
  * @a mtu is too small, generates and error for the @a pm
@@ -2284,13 +2665,7 @@ fragment_message (struct PendingMessage *pm,
 {
   struct PendingMessage *ff;
 
-  if (GNUNET_NO == pm->msg_uuid_set)
-  {
-    GNUNET_CRYPTO_random_block (GNUNET_CRYPTO_QUALITY_NONCE,
-                                &pm->msg_uuid,
-                                sizeof (pm->msg_uuid));
-    pm->msg_uuid_set = GNUNET_YES;
-  }
+  set_pending_message_uuid (pm);
   
   /* This invariant is established in #handle_add_queue_message() */
   GNUNET_assert (mtu > sizeof (struct TransportFragmentBox));
@@ -2390,24 +2765,50 @@ fragment_message (struct PendingMessage *pm,
 static struct PendingMessage *
 reliability_box_message (struct PendingMessage *pm)
 {
-  if (PMT_CORE != pm->pmt) 
-  {
-    /* already fragmented or reliability boxed, or control message: do nothing */
-    return pm;
-  }
-  
-  if (0) // FIXME
+  struct TransportReliabilityBox rbox;
+  struct PendingMessage *bpm;
+  char *msg;
+
+  if (PMT_CORE != pm->pmt)
+    return pm;  /* already fragmented or reliability boxed, or control message: do nothing */
+  if (NULL != pm->bpm)
+    return pm->bpm; /* already computed earlier: do nothing */
+  GNUNET_assert (NULL == pm->head_frag);
+  if (pm->bytes_msg + sizeof (rbox) > UINT16_MAX) 
   {
     /* failed hard */
-    // FIMXE: bitch
+    GNUNET_break (0);
     client_send_response (pm,
                           GNUNET_NO,
                           0);
     return NULL;
   }
-
-  /* FIXME: return boxed PM here! */
-  return NULL;
+  bpm = GNUNET_malloc (sizeof (struct PendingMessage) +
+                       sizeof (rbox) + 
+                       pm->bytes_msg);
+  bpm->target = pm->target;
+  bpm->frag_parent = pm;
+  GNUNET_CONTAINER_MDLL_insert (frag,
+                                pm->head_frag,
+                                pm->tail_frag,
+                                bpm);
+  bpm->timeout = pm->timeout;
+  bpm->pmt = PMT_RELIABILITY_BOX;
+  bpm->bytes_msg = pm->bytes_msg + sizeof (rbox);
+  set_pending_message_uuid (bpm);
+  rbox.header.type = htons (GNUNET_MESSAGE_TYPE_TRANSPORT_RELIABILITY_BOX);
+  rbox.header.size = htons (sizeof (rbox) + pm->bytes_msg);
+  rbox.ack_countdown = htonl (0); // FIXME: implement ACK countdown support
+  rbox.msg_uuid = pm->msg_uuid;
+  msg = (char *) &bpm[1];
+  memcpy (msg,
+          &rbox,
+          sizeof (rbox));
+  memcpy (&msg[sizeof (rbox)],
+          &pm[1],
+          pm->bytes_msg);
+  pm->bpm = bpm;
+  return bpm;
 }
 
 
@@ -2542,26 +2943,64 @@ transmit_on_queue (void *cls)
   else if (PMT_CORE != pm->pmt)
   {
     /* This was an acknowledgement of some type, always free */
-
-    struct Neighbour *neighbour = pm->target;
-    GNUNET_CONTAINER_MDLL_remove (neighbour,
-                                  neighbour->pending_msg_head,
-                                  neighbour->pending_msg_tail,
-                                  pm);
-    GNUNET_free (pm);
+    free_pending_message (pm);
   }
   else
   {
     /* message not finished, waiting for acknowledgement */
-    // FIXME: update time by which we might retransmit 's' based on
-    // queue characteristics (i.e. RTT)
-    
-    // FIXME: move 'pm' back in the transmission queue (simplistic: to
-    // the end, better: with position depending on type, timeout,
-    // etc.)
+    struct Neighbour *neighbour = pm->target;
+    /* Update time by which we might retransmit 's' based on queue
+       characteristics (i.e. RTT); it takes one RTT for the message to
+       arrive and the ACK to come back in the best case; but the other
+       side is allowed to delay ACKs by 2 RTTs, so we use 4 RTT before
+       retransmitting.  Note that in the future this heuristic should
+       likely be improved further (measure RTT stability, consider
+       message urgency and size when delaying ACKs, etc.) */
+    s->next_attempt = GNUNET_TIME_relative_to_absolute
+      (GNUNET_TIME_relative_multiply (queue->rtt,
+                                      4));
+    if (s == pm)
+    {
+      struct PendingMessage *pos;
+
+      /* re-insert sort in neighbour list */
+      GNUNET_CONTAINER_MDLL_remove (neighbour,
+                                    neighbour->pending_msg_head,
+                                    neighbour->pending_msg_tail,
+                                    pm);
+      pos = neighbour->pending_msg_tail;
+      while ( (NULL != pos) &&
+              (pm->next_attempt.abs_value_us > pos->next_attempt.abs_value_us) )
+        pos = pos->prev_neighbour;
+      GNUNET_CONTAINER_MDLL_insert_after (neighbour,
+                                          neighbour->pending_msg_head,
+                                          neighbour->pending_msg_tail,
+                                          pos,
+                                          pm);
+    }
+    else
+    {
+      /* re-insert sort in fragment list */
+      struct PendingMessage *fp = s->frag_parent;
+      struct PendingMessage *pos;
+
+      GNUNET_CONTAINER_MDLL_remove (frag,
+                                    fp->head_frag,
+                                    fp->tail_frag,
+                                    s);
+      pos = fp->tail_frag;
+      while ( (NULL != pos) &&
+              (s->next_attempt.abs_value_us > pos->next_attempt.abs_value_us) )
+        pos = pos->prev_frag;
+      GNUNET_CONTAINER_MDLL_insert_after (frag,
+                                          fp->head_frag,
+                                          fp->tail_frag,
+                                          pos,
+                                          s);
+    }
   }
   
-  /* finally, re-schedule self */
+  /* finally, re-schedule queue transmission task itself */
   schedule_transmit_on_queue (queue);
 }
 
diff --git a/src/util/mq.c b/src/util/mq.c
index 4dfcb72be..d2f5add19 100644
--- a/src/util/mq.c
+++ b/src/util/mq.c
@@ -215,6 +215,35 @@ void
 GNUNET_MQ_inject_message (struct GNUNET_MQ_Handle *mq,
                           const struct GNUNET_MessageHeader *mh)
 {
+  int ret;
+
+  ret = GNUNET_MQ_handle_message (mq->handlers,
+                                  mh);
+  if (GNUNET_SYSERR == ret)
+  {
+    GNUNET_MQ_inject_error (mq,
+                            GNUNET_MQ_ERROR_MALFORMED);
+    return;
+  }
+}
+
+
+/**
+ * Call the message message handler that was registered
+ * for the type of the given message in the given @a handlers list.
+ *
+ * This function is indended to be used for the implementation
+ * of message queues.
+ *
+ * @param handlers a set of handlers
+ * @param mh message to dispatch
+ * @return #GNUNET_OK on success, #GNUNET_NO if no handler matched,
+ *         #GNUNET_SYSERR if message was rejected by check function
+ */
+int
+GNUNET_MQ_handle_message (const struct GNUNET_MQ_MessageHandler *handlers,
+                          const struct GNUNET_MessageHeader *mh)
+{
   const struct GNUNET_MQ_MessageHandler *handler;
   int handled = GNUNET_NO;
   uint16_t msize = ntohs (mh->size);
@@ -224,9 +253,9 @@ GNUNET_MQ_inject_message (struct GNUNET_MQ_Handle *mq,
        "Received message of type %u and size %u\n",
        mtype, msize);
 
-  if (NULL == mq->handlers)
+  if (NULL == handlers)
     goto done;
-  for (handler = mq->handlers; NULL != handler->cb; handler++)
+  for (handler = handlers; NULL != handler->cb; handler++)
   {
     if (handler->type == mtype)
     {
@@ -240,9 +269,7 @@ GNUNET_MQ_inject_message (struct GNUNET_MQ_Handle *mq,
         LOG (GNUNET_ERROR_TYPE_ERROR,
              "Received malformed message of type %u\n",
              (unsigned int) handler->type);
-        GNUNET_MQ_inject_error (mq,
-                                GNUNET_MQ_ERROR_MALFORMED);
-        break;
+        return GNUNET_SYSERR;
       }
       if ( (NULL == handler->mv) ||
            (GNUNET_OK ==
@@ -257,17 +284,20 @@ GNUNET_MQ_inject_message (struct GNUNET_MQ_Handle *mq,
         LOG (GNUNET_ERROR_TYPE_ERROR,
              "Received malformed message of type %u\n",
              (unsigned int) handler->type);
-        GNUNET_MQ_inject_error (mq,
-                                GNUNET_MQ_ERROR_MALFORMED);
+        return GNUNET_SYSERR;
       }
       break;
     }
   }
  done:
   if (GNUNET_NO == handled)
+  {
     LOG (GNUNET_ERROR_TYPE_INFO,
          "No handler for message of type %u and size %u\n",
          mtype, msize);
+    return GNUNET_NO;
+  }
+  return GNUNET_OK;
 }