updated the Ascension documentation

3 files changed, 108 insertions, 67 deletions

diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi
index b725f4111..64ebec46e 100644
--- a/doc/handbook/chapters/developer.texi
+++ b/doc/handbook/chapters/developer.texi
@@ -7967,9 +7967,9 @@ future.
 @node Conversions between DNS and GNS
 @subsubsection Conversions between DNS and GNS
 
-The differences between the two name systems lies in the details
-and is not always transparent.
-For instance an SRV record is converted to a GNS only BOX record.
+The differences between the two name systems lies in the details and is not
+always transparent.  For instance an SRV record is converted to a BOX record
+which is unique to GNS.
 
 This is done by converting to a BOX record from an existing SRV record:
 
@@ -7982,7 +7982,7 @@ _sip._tcp.example.com. 14000 IN SRV     0 0 5060 www.example.com.
 14000 BOX n 5060 6 33 0 0 5060 www.example.com
 @end example
 
-Other records that have such a transformation is the MX record type,
+Other records that need to undergo such transformation is the MX record type,
 as well as the SOA record type.
 
 Transformation of a SOA record into GNS works as described in the
@@ -7997,8 +7997,9 @@ following example. Very important to note are the rname and mname keys.
     604800     ; expire
     600 )      ; ttl
 # Recordline for adding the record
-$ gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \
-  mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s
+$ gnunet-namestore -z example.com -a -n @ -t SOA -V \
+    rname=master.example.com mname=hostmaster.example.com  \
+    2017030300,3600,1800,604800,600 -e 7200s
 @end example
 
 The transformation of MX records is done in a simple way.
@@ -8007,10 +8008,10 @@ The transformation of MX records is done in a simple way.
 $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
 @end example
 
-Finally, one of the biggest struggling points were the NS records that are found
-in top level domain zones. The intended behaviour for those is to add GNS2DNS
-records for those so that gnunet-gns can resolve records for those domains on
-its own. This requires migration of the DNS GLUE records as well, provided that
+Finally, one of the biggest struggling points were the NS records that are
+found in top level domain zones. The intended behaviour for those is to add
+GNS2DNS records for those so that gnunet-gns can resolve records for those
+domains on its own. Those require the values from DNS GLUE records, provided
 they are within the same zone.
 
 The following two examples show one record with a GLUE record and the other one
@@ -8019,10 +8020,12 @@ does not have a GLUE record. This takes place in the 'com' TLD.
 @example
 # ns1.example.com 86400 IN A 127.0.0.1
 # example.com 86400 IN NS ns1.example.com.
-$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@127.0.0.1
+$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
+    example.com@@127.0.0.1
 
 # example.com 86400 IN NS ns1.example.org.
-$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@ns1.example.org
+$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
+    example.com@@ns1.example.org
 @end example
 
 As you can see, one of the GNS2DNS records has an IP address listed and the
@@ -8045,7 +8048,7 @@ Currently the following record types are supported:
 @item TXT
 @end itemize
 
-This is not due to a technical limitation but rather a practical one. The
+This is not due to technical limitations but rather a practical ones. The
 problem occurs with DNSSEC enabled DNS zones. As records within those zones are
 signed periodically, and every new signature is an update to the zone, there are
 many revisions of zones. This results in a problem with bigger zones as there
@@ -8055,16 +8058,22 @@ as they cause a CLI call of the namestore.  Furthermore certain record types
 need transformation into a GNS compatible format which, depending on the record
 type, takes more time.
 
+Further a blacklist was added to drop for instance DNSSEC related records. Also
+if a record type is neither in the white list nor the blacklist it is considered
+as a loss of data and a message is shown to the user. This helps with
+transparency and also with contributing, as the not supported record types can
+then be added accordingly.
+
 @node DNS Zone Size
 @subsubsection DNS Zone Size
-
 Another very big problem exists with very large zones. When migrating a small
 zone the delay between adding of records and their expiry is negligible. However
-when working with a TLD zone that has more that 1 million records this delay
-becomes a problem.
+when working with big zones that easily have more than a few million records
+this delay becomes a problem.
 
 Records will start to expire well before the zone has finished migrating. This
-causes unwanted anomalies when trying to resolve records.
+is usually not a problem but can cause a high CPU load when a peer is restarted
+and the records have expired.
 
 A good solution has not been found yet. One of the idea that floated around was
 that the records should be added with the s (shadow) flag to keep the records
@@ -8074,45 +8083,51 @@ of said record(s).
 
 Another problem that still persists is how to refresh records. Expired records
 are still displayed when calling gnunet-namestore but do not resolve with
-gnunet-gns. When doing incremental zone transfers this becomes especially
-apparent.
+gnunet-gns. Zonemaster will sign the expired records again and make sure that
+the records are still valid. With a recent change this was fixed as gnunet-gns
+to improve the suffix lookup which allows for a fast lookup even with thousands
+of local egos.
+
+Currently the pace of adding records in general is around 10 records per second.
+Crypto is the upper limit for adding of records. The performance of your machine
+can be tested with the perf_crypto_* tools. There is still a big discrepancy
+between the pace of Ascension and the theoretical limit.
 
-I estimate that the limit lies at about 200'000 records in a zone as this is
-the limit that my machine is capable of adding within one hour.  This was
-calculated by running cProfile on the application with a zone of 5000 records
-and calculating what abouts a much bigger zones with 8 million records would
-take. This results in a nice metric of records migrated per hour.
+A performance metric for measuring improvements has not yet been implemented in
+Ascension.
 
 @node Performance
 @subsubsection Performance
 The performance when migrating a zone using the Ascension tool is limited by a
 handful of factors. First of all ascension is written in Python3 and calls the
-CLI tools of GNUnet. Furthermore all the records that are added to the same
+CLI tools of GNUnet. This is comparable to a fork and exec call which costs a
+few CPU cycles. Furthermore all the records that are added to the same
 label are signed using the zones private key. This signing operation is very
 resource heavy and was optimized during development by adding the '-R'
-(Recordline) option to gnunet-namestore. This allows to add multiple records
-at once using the CLI.
-
-The result of this was a much faster migration of TLD zones, as most records
-with the same label have two name servers.
+(Recordline) option to gnunet-namestore which allows to specify multiple records
+using the CLI tool. Assuming that in a TLD zone every domain has at least two
+name servers this halves the amount of signatures needed.
 
 Another improvement that could be made is with the addition of multiple threads
-when opening the GNUnet CLI tools. This could be implemented by simply creating
-more workers in the program but performance improvements were not tested.
+or using asynchronous subprocesses when opening the GNUnet CLI tools. This could
+be implemented by simply creating more workers in the program but performance
+improvements were not tested.
 
-During the entire development of Ascension sqlite was used as a database
-backend for GNUnet. Other backends have not been tested yet.
+Ascension was tested using different hardware and database backends. Performance
+differences between SQLite and postgresql are marginal and almost non existent.
+What did make a huge impact on record adding performance was the storage medium.
+On a traditional mechanical hard drive adding of records were slow compared to a
+solid state disk.
 
 In conclusion there are many bottlenecks still around in the program, namely the
-signing process and the single threaded implementation. In the future a solution
-that uses the C API would be cleaner and better.
+single threaded implementation and inefficient, sequential calls of
+gnunet-namestore. In the future a solution that uses the C API would be cleaner
+and better.
 
 @cindex GNS Namecache
 @node GNS Namecache
 @section GNS Namecache
 
-
-
 The NAMECACHE subsystem is responsible for caching (encrypted) resolution
 results of the GNU Name System (GNS). GNS makes zone information available
 to other users via the DHT. However, as accessing the DHT for every
diff --git a/doc/handbook/chapters/installation.texi b/doc/handbook/chapters/installation.texi
index d02fc7f82..a508feb6a 100644
--- a/doc/handbook/chapters/installation.texi
+++ b/doc/handbook/chapters/installation.texi
@@ -1786,10 +1786,21 @@ Keeping a virtual environment helps with keeping things tidy and prevents
 breaking of Ascension through a future Python update.
 
 The advantage of using a virtual environment is, that all the dependencies can
-be installed separately in different versions without touching your system
+be installed separately in different versions without touching your systems
 Python installation and its dependencies.
 
-@xref{Migrating an existing DNS zone into GNS}, for usage manual of the tool.
+Another way to install Ascension on Debian is to install the python3-ascension
+package. It can be found within the above mentioned Ascension git repository.
+This also adds a system user ascension and runs a GNUnet peer in the
+background. Attention: This only works if a recent version of GNUnet is
+installed on your system. The version number of Ascension is chosen according
+to the required feature level of GNUnet. I.e. Ascension 0.11.5 is only
+compatible with GNUnet 0.11.5 and upwards. As Debian's packages for GNUnet are
+outdated even in experimental, you will need to install GNUnet manually.
+@xref{Installing GNUnet}
+
+Please check @xref{Migrating an existing DNS zone into GNS}, for usage manual
+of the tool.
 
 @node Configuring the GNUnet VPN
 @subsection Configuring the GNUnet VPN
diff --git a/doc/handbook/chapters/user.texi b/doc/handbook/chapters/user.texi
index 42f37c2ea..7fb63b9a7 100644
--- a/doc/handbook/chapters/user.texi
+++ b/doc/handbook/chapters/user.texi
@@ -1842,33 +1842,42 @@ options:
 
 @example
 Ascension
-
 Usage:
-    ascension <domain> [-d] [-p]
-    ascension <domain> <port> [-d] [-p]
-    ascension <domain> -n <transferns> [-d] [-p]
-    ascension <domain> -n <transferns> <port> [-d] [-p]
+    ascension <domain> [-d] [-p] [-s] [--minimum-ttl=<ttl>] \
+        [--dry-run]
+    ascension <domain> <port> [-d] [-p] [-s] \
+        [--minimum-ttl=<ttl>] [--dry-run]
+    ascension <domain> -n <transferns> [-d] [-p] \
+        [-s] [--minimum-ttl=<ttl>] [--dry-run]
+    ascension <domain> -n <transferns> <port> [-d] \
+        [-p] [-s] [--minimum-ttl=<ttl>] [--dry-run]
     ascension -p | --public
+    ascension -d | --debug
+    ascension -s | --standalone
     ascension -h | --help
     ascension -v | --version
 
 Options:
-    <domain>        Domain to migrate
-    <port>          Port for zone transfer
-    <transferns>    DNS Server that does the zone transfer
-    -p --public     Make records public on the DHT
-    -d --debug      Enable debugging
-    -h --help       Show this screen.
-    -v --version    Show version.
+    <domain>              Domain to migrate
+    <port>                Port for zone transfer
+    <transferns>          DNS Server that does the zone transfer
+    --minimum-ttl=<ttl>   Minimum TTL for records to migrate \
+        [default: 3600]
+    --dry-run             Only try if a zone transfer is allowed
+    -p --public           Make records public on the DHT
+    -s --standalone       Run ascension once
+    -d --debug            Enable debugging
+    -h --help         Show this screen.
+    -v --version      Show version.
 @end example
 
-Before you can migrate any zone though, you need to start the GNUnet peer:
+Before you can migrate any zone though, you need to start a local GNUnet peer:
 @example
 $ gnunet-arm -s
 @end example
 
 To migrate the Syrian top level domain - one of the few top level domains that
-still supports zone transfers - into GNS use the following command:
+support zone transfers - into GNS use the following command:
 
 @example
 $ ascension sy. -n ns1.tld.sy. -p
@@ -1881,33 +1890,39 @@ Once the zone is migrated, Ascension will output a message telling you, that it
 will refresh the zone after the time has elapsed.  You can resolve the names in
 the zone directly using GNS or if you want to use it with your browser, check
 out the GNS manual section. @ref{Configuring the GNU Name System}. To resolve
-the records from another system you need the zone PKEY.  To get the zone key,
-you can run the following command:
+the records from another system you need the respective zones PKEY. To get the
+zones public key, you can run the following command:
 
 @example
-$ gnunet-identity -d | grep ^sy | cut -d " " -f3
+$ gnunet-identity -dqe sy
 @end example
 
 Where "sy" is the name of the zone you want to migrate.
 
-As soon as the public flag is implemented, you can share the PKEY of the zone
-with your friends. They can then resolve records in the zone by doing a lookup
-replacing the zone label with your PKEY:
+You can share the PKEY of the zone with your friends. They can then resolve
+records in the zone by doing a lookup replacing the zone label with your PKEY:
 
 @example
-$ gnunet-gns -t SOA -u "@.$PKEY"
+$ gnunet-gns -t SOA -u "$PKEY"
 @end example
 
 The program will continue to run as a daemon and update once the refresh time
 specified in the zones SOA record has elapsed.
 
-The next step would be to add the PKEY record as a DNScurve style NS record
-into the existing DNS zone to enable clients to detect that this zone has
-already been migrated to GNS and to also have a means of distributing the PKEY
-seamlessly.
+DNSCurve style records are supported in the latest release and they are added
+as a PKEY record to be referred to the respective GNS public key. Key
+distribution is still a problem but provided someone else has a public key
+under a given label it can be looked up.
+
+There is an unofficial Debian package called python3-ascension that adds a
+system user ascension and runs a GNUnet peer in the background.
 
-At this point you might want to write for example a systemd unit file to start
-and enable the service, so that your zone is migrated automatically.
+Ascension-bind is also an unofficial Debian package that on installation checks
+for running DNS zones and whether or not they are transferable using DNS zone
+transfer (AXFR). It asks the administrator which zones to migrate into GNS and
+installs a systemd unit file to keep the zone up to date.  If you want to
+migrate different zones you might want to check the unit file from the package
+as a guide.
 
 @node reclaimID Identity Provider
 @section reclaimID Identity Provider