aboutsummaryrefslogtreecommitdiff
path: root/doc/handbook/chapters/developer.texi
diff options
context:
space:
mode:
authorrexxnor <rexxnor+gnunet@brief.li>2019-06-21 23:23:56 +0200
committerrexxnor <rexxnor+gnunet@brief.li>2019-06-21 23:24:11 +0200
commit3973413e583568350785f3ffc608ed2972f086e6 (patch)
tree3d7c3d28d5332d4463be5d48d2bbf1da663b5ec0 /doc/handbook/chapters/developer.texi
parent67b161e86e95152906c80c4fa8cf067614e8653e (diff)
downloadgnunet-3973413e583568350785f3ffc608ed2972f086e6.tar.gz
gnunet-3973413e583568350785f3ffc608ed2972f086e6.zip
updated the Ascension documentation
Diffstat (limited to 'doc/handbook/chapters/developer.texi')
-rw-r--r--doc/handbook/chapters/developer.texi91
1 files changed, 53 insertions, 38 deletions
diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi
index b725f4111..64ebec46e 100644
--- a/doc/handbook/chapters/developer.texi
+++ b/doc/handbook/chapters/developer.texi
@@ -7967,9 +7967,9 @@ future.
7967@node Conversions between DNS and GNS 7967@node Conversions between DNS and GNS
7968@subsubsection Conversions between DNS and GNS 7968@subsubsection Conversions between DNS and GNS
7969 7969
7970The differences between the two name systems lies in the details 7970The differences between the two name systems lies in the details and is not
7971and is not always transparent. 7971always transparent. For instance an SRV record is converted to a BOX record
7972For instance an SRV record is converted to a GNS only BOX record. 7972which is unique to GNS.
7973 7973
7974This is done by converting to a BOX record from an existing SRV record: 7974This is done by converting to a BOX record from an existing SRV record:
7975 7975
@@ -7982,7 +7982,7 @@ _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com.
798214000 BOX n 5060 6 33 0 0 5060 www.example.com 798214000 BOX n 5060 6 33 0 0 5060 www.example.com
7983@end example 7983@end example
7984 7984
7985Other records that have such a transformation is the MX record type, 7985Other records that need to undergo such transformation is the MX record type,
7986as well as the SOA record type. 7986as well as the SOA record type.
7987 7987
7988Transformation of a SOA record into GNS works as described in the 7988Transformation of a SOA record into GNS works as described in the
@@ -7997,8 +7997,9 @@ following example. Very important to note are the rname and mname keys.
7997 604800 ; expire 7997 604800 ; expire
7998 600 ) ; ttl 7998 600 ) ; ttl
7999# Recordline for adding the record 7999# Recordline for adding the record
8000$ gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \ 8000$ gnunet-namestore -z example.com -a -n @ -t SOA -V \
8001 mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s 8001 rname=master.example.com mname=hostmaster.example.com \
8002 2017030300,3600,1800,604800,600 -e 7200s
8002@end example 8003@end example
8003 8004
8004The transformation of MX records is done in a simple way. 8005The transformation of MX records is done in a simple way.
@@ -8007,10 +8008,10 @@ The transformation of MX records is done in a simple way.
8007$ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail 8008$ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
8008@end example 8009@end example
8009 8010
8010Finally, one of the biggest struggling points were the NS records that are found 8011Finally, one of the biggest struggling points were the NS records that are
8011in top level domain zones. The intended behaviour for those is to add GNS2DNS 8012found in top level domain zones. The intended behaviour for those is to add
8012records for those so that gnunet-gns can resolve records for those domains on 8013GNS2DNS records for those so that gnunet-gns can resolve records for those
8013its own. This requires migration of the DNS GLUE records as well, provided that 8014domains on its own. Those require the values from DNS GLUE records, provided
8014they are within the same zone. 8015they are within the same zone.
8015 8016
8016The following two examples show one record with a GLUE record and the other one 8017The following two examples show one record with a GLUE record and the other one
@@ -8019,10 +8020,12 @@ does not have a GLUE record. This takes place in the 'com' TLD.
8019@example 8020@example
8020# ns1.example.com 86400 IN A 127.0.0.1 8021# ns1.example.com 86400 IN A 127.0.0.1
8021# example.com 86400 IN NS ns1.example.com. 8022# example.com 86400 IN NS ns1.example.com.
8022$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@127.0.0.1 8023$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
8024 example.com@@127.0.0.1
8023 8025
8024# example.com 86400 IN NS ns1.example.org. 8026# example.com 86400 IN NS ns1.example.org.
8025$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@ns1.example.org 8027$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
8028 example.com@@ns1.example.org
8026@end example 8029@end example
8027 8030
8028As you can see, one of the GNS2DNS records has an IP address listed and the 8031As you can see, one of the GNS2DNS records has an IP address listed and the
@@ -8045,7 +8048,7 @@ Currently the following record types are supported:
8045@item TXT 8048@item TXT
8046@end itemize 8049@end itemize
8047 8050
8048This is not due to a technical limitation but rather a practical one. The 8051This is not due to technical limitations but rather a practical ones. The
8049problem occurs with DNSSEC enabled DNS zones. As records within those zones are 8052problem occurs with DNSSEC enabled DNS zones. As records within those zones are
8050signed periodically, and every new signature is an update to the zone, there are 8053signed periodically, and every new signature is an update to the zone, there are
8051many revisions of zones. This results in a problem with bigger zones as there 8054many revisions of zones. This results in a problem with bigger zones as there
@@ -8055,16 +8058,22 @@ as they cause a CLI call of the namestore. Furthermore certain record types
8055need transformation into a GNS compatible format which, depending on the record 8058need transformation into a GNS compatible format which, depending on the record
8056type, takes more time. 8059type, takes more time.
8057 8060
8061Further a blacklist was added to drop for instance DNSSEC related records. Also
8062if a record type is neither in the white list nor the blacklist it is considered
8063as a loss of data and a message is shown to the user. This helps with
8064transparency and also with contributing, as the not supported record types can
8065then be added accordingly.
8066
8058@node DNS Zone Size 8067@node DNS Zone Size
8059@subsubsection DNS Zone Size 8068@subsubsection DNS Zone Size
8060
8061Another very big problem exists with very large zones. When migrating a small 8069Another very big problem exists with very large zones. When migrating a small
8062zone the delay between adding of records and their expiry is negligible. However 8070zone the delay between adding of records and their expiry is negligible. However
8063when working with a TLD zone that has more that 1 million records this delay 8071when working with big zones that easily have more than a few million records
8064becomes a problem. 8072this delay becomes a problem.
8065 8073
8066Records will start to expire well before the zone has finished migrating. This 8074Records will start to expire well before the zone has finished migrating. This
8067causes unwanted anomalies when trying to resolve records. 8075is usually not a problem but can cause a high CPU load when a peer is restarted
8076and the records have expired.
8068 8077
8069A good solution has not been found yet. One of the idea that floated around was 8078A good solution has not been found yet. One of the idea that floated around was
8070that the records should be added with the s (shadow) flag to keep the records 8079that the records should be added with the s (shadow) flag to keep the records
@@ -8074,45 +8083,51 @@ of said record(s).
8074 8083
8075Another problem that still persists is how to refresh records. Expired records 8084Another problem that still persists is how to refresh records. Expired records
8076are still displayed when calling gnunet-namestore but do not resolve with 8085are still displayed when calling gnunet-namestore but do not resolve with
8077gnunet-gns. When doing incremental zone transfers this becomes especially 8086gnunet-gns. Zonemaster will sign the expired records again and make sure that
8078apparent. 8087the records are still valid. With a recent change this was fixed as gnunet-gns
8088to improve the suffix lookup which allows for a fast lookup even with thousands
8089of local egos.
8090
8091Currently the pace of adding records in general is around 10 records per second.
8092Crypto is the upper limit for adding of records. The performance of your machine
8093can be tested with the perf_crypto_* tools. There is still a big discrepancy
8094between the pace of Ascension and the theoretical limit.
8079 8095
8080I estimate that the limit lies at about 200'000 records in a zone as this is 8096A performance metric for measuring improvements has not yet been implemented in
8081the limit that my machine is capable of adding within one hour. This was 8097Ascension.
8082calculated by running cProfile on the application with a zone of 5000 records
8083and calculating what abouts a much bigger zones with 8 million records would
8084take. This results in a nice metric of records migrated per hour.
8085 8098
8086@node Performance 8099@node Performance
8087@subsubsection Performance 8100@subsubsection Performance
8088The performance when migrating a zone using the Ascension tool is limited by a 8101The performance when migrating a zone using the Ascension tool is limited by a
8089handful of factors. First of all ascension is written in Python3 and calls the 8102handful of factors. First of all ascension is written in Python3 and calls the
8090CLI tools of GNUnet. Furthermore all the records that are added to the same 8103CLI tools of GNUnet. This is comparable to a fork and exec call which costs a
8104few CPU cycles. Furthermore all the records that are added to the same
8091label are signed using the zones private key. This signing operation is very 8105label are signed using the zones private key. This signing operation is very
8092resource heavy and was optimized during development by adding the '-R' 8106resource heavy and was optimized during development by adding the '-R'
8093(Recordline) option to gnunet-namestore. This allows to add multiple records 8107(Recordline) option to gnunet-namestore which allows to specify multiple records
8094at once using the CLI. 8108using the CLI tool. Assuming that in a TLD zone every domain has at least two
8095 8109name servers this halves the amount of signatures needed.
8096The result of this was a much faster migration of TLD zones, as most records
8097with the same label have two name servers.
8098 8110
8099Another improvement that could be made is with the addition of multiple threads 8111Another improvement that could be made is with the addition of multiple threads
8100when opening the GNUnet CLI tools. This could be implemented by simply creating 8112or using asynchronous subprocesses when opening the GNUnet CLI tools. This could
8101more workers in the program but performance improvements were not tested. 8113be implemented by simply creating more workers in the program but performance
8114improvements were not tested.
8102 8115
8103During the entire development of Ascension sqlite was used as a database 8116Ascension was tested using different hardware and database backends. Performance
8104backend for GNUnet. Other backends have not been tested yet. 8117differences between SQLite and postgresql are marginal and almost non existent.
8118What did make a huge impact on record adding performance was the storage medium.
8119On a traditional mechanical hard drive adding of records were slow compared to a
8120solid state disk.
8105 8121
8106In conclusion there are many bottlenecks still around in the program, namely the 8122In conclusion there are many bottlenecks still around in the program, namely the
8107signing process and the single threaded implementation. In the future a solution 8123single threaded implementation and inefficient, sequential calls of
8108that uses the C API would be cleaner and better. 8124gnunet-namestore. In the future a solution that uses the C API would be cleaner
8125and better.
8109 8126
8110@cindex GNS Namecache 8127@cindex GNS Namecache
8111@node GNS Namecache 8128@node GNS Namecache
8112@section GNS Namecache 8129@section GNS Namecache
8113 8130
8114
8115
8116The NAMECACHE subsystem is responsible for caching (encrypted) resolution 8131The NAMECACHE subsystem is responsible for caching (encrypted) resolution
8117results of the GNU Name System (GNS). GNS makes zone information available 8132results of the GNU Name System (GNS). GNS makes zone information available
8118to other users via the DHT. However, as accessing the DHT for every 8133to other users via the DHT. However, as accessing the DHT for every