diff options
author | rexxnor <rexxnor+gnunet@brief.li> | 2019-06-21 23:23:56 +0200 |
---|---|---|
committer | rexxnor <rexxnor+gnunet@brief.li> | 2019-06-21 23:24:11 +0200 |
commit | 3973413e583568350785f3ffc608ed2972f086e6 (patch) | |
tree | 3d7c3d28d5332d4463be5d48d2bbf1da663b5ec0 /doc/handbook/chapters/developer.texi | |
parent | 67b161e86e95152906c80c4fa8cf067614e8653e (diff) | |
download | gnunet-3973413e583568350785f3ffc608ed2972f086e6.tar.gz gnunet-3973413e583568350785f3ffc608ed2972f086e6.zip |
updated the Ascension documentation
Diffstat (limited to 'doc/handbook/chapters/developer.texi')
-rw-r--r-- | doc/handbook/chapters/developer.texi | 91 |
1 files changed, 53 insertions, 38 deletions
diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi index b725f4111..64ebec46e 100644 --- a/doc/handbook/chapters/developer.texi +++ b/doc/handbook/chapters/developer.texi | |||
@@ -7967,9 +7967,9 @@ future. | |||
7967 | @node Conversions between DNS and GNS | 7967 | @node Conversions between DNS and GNS |
7968 | @subsubsection Conversions between DNS and GNS | 7968 | @subsubsection Conversions between DNS and GNS |
7969 | 7969 | ||
7970 | The differences between the two name systems lies in the details | 7970 | The differences between the two name systems lies in the details and is not |
7971 | and is not always transparent. | 7971 | always transparent. For instance an SRV record is converted to a BOX record |
7972 | For instance an SRV record is converted to a GNS only BOX record. | 7972 | which is unique to GNS. |
7973 | 7973 | ||
7974 | This is done by converting to a BOX record from an existing SRV record: | 7974 | This is done by converting to a BOX record from an existing SRV record: |
7975 | 7975 | ||
@@ -7982,7 +7982,7 @@ _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com. | |||
7982 | 14000 BOX n 5060 6 33 0 0 5060 www.example.com | 7982 | 14000 BOX n 5060 6 33 0 0 5060 www.example.com |
7983 | @end example | 7983 | @end example |
7984 | 7984 | ||
7985 | Other records that have such a transformation is the MX record type, | 7985 | Other records that need to undergo such transformation is the MX record type, |
7986 | as well as the SOA record type. | 7986 | as well as the SOA record type. |
7987 | 7987 | ||
7988 | Transformation of a SOA record into GNS works as described in the | 7988 | Transformation of a SOA record into GNS works as described in the |
@@ -7997,8 +7997,9 @@ following example. Very important to note are the rname and mname keys. | |||
7997 | 604800 ; expire | 7997 | 604800 ; expire |
7998 | 600 ) ; ttl | 7998 | 600 ) ; ttl |
7999 | # Recordline for adding the record | 7999 | # Recordline for adding the record |
8000 | $ gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \ | 8000 | $ gnunet-namestore -z example.com -a -n @ -t SOA -V \ |
8001 | mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s | 8001 | rname=master.example.com mname=hostmaster.example.com \ |
8002 | 2017030300,3600,1800,604800,600 -e 7200s | ||
8002 | @end example | 8003 | @end example |
8003 | 8004 | ||
8004 | The transformation of MX records is done in a simple way. | 8005 | The transformation of MX records is done in a simple way. |
@@ -8007,10 +8008,10 @@ The transformation of MX records is done in a simple way. | |||
8007 | $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail | 8008 | $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail |
8008 | @end example | 8009 | @end example |
8009 | 8010 | ||
8010 | Finally, one of the biggest struggling points were the NS records that are found | 8011 | Finally, one of the biggest struggling points were the NS records that are |
8011 | in top level domain zones. The intended behaviour for those is to add GNS2DNS | 8012 | found in top level domain zones. The intended behaviour for those is to add |
8012 | records for those so that gnunet-gns can resolve records for those domains on | 8013 | GNS2DNS records for those so that gnunet-gns can resolve records for those |
8013 | its own. This requires migration of the DNS GLUE records as well, provided that | 8014 | domains on its own. Those require the values from DNS GLUE records, provided |
8014 | they are within the same zone. | 8015 | they are within the same zone. |
8015 | 8016 | ||
8016 | The following two examples show one record with a GLUE record and the other one | 8017 | The following two examples show one record with a GLUE record and the other one |
@@ -8019,10 +8020,12 @@ does not have a GLUE record. This takes place in the 'com' TLD. | |||
8019 | @example | 8020 | @example |
8020 | # ns1.example.com 86400 IN A 127.0.0.1 | 8021 | # ns1.example.com 86400 IN A 127.0.0.1 |
8021 | # example.com 86400 IN NS ns1.example.com. | 8022 | # example.com 86400 IN NS ns1.example.com. |
8022 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@127.0.0.1 | 8023 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ |
8024 | example.com@@127.0.0.1 | ||
8023 | 8025 | ||
8024 | # example.com 86400 IN NS ns1.example.org. | 8026 | # example.com 86400 IN NS ns1.example.org. |
8025 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@ns1.example.org | 8027 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ |
8028 | example.com@@ns1.example.org | ||
8026 | @end example | 8029 | @end example |
8027 | 8030 | ||
8028 | As you can see, one of the GNS2DNS records has an IP address listed and the | 8031 | As you can see, one of the GNS2DNS records has an IP address listed and the |
@@ -8045,7 +8048,7 @@ Currently the following record types are supported: | |||
8045 | @item TXT | 8048 | @item TXT |
8046 | @end itemize | 8049 | @end itemize |
8047 | 8050 | ||
8048 | This is not due to a technical limitation but rather a practical one. The | 8051 | This is not due to technical limitations but rather a practical ones. The |
8049 | problem occurs with DNSSEC enabled DNS zones. As records within those zones are | 8052 | problem occurs with DNSSEC enabled DNS zones. As records within those zones are |
8050 | signed periodically, and every new signature is an update to the zone, there are | 8053 | signed periodically, and every new signature is an update to the zone, there are |
8051 | many revisions of zones. This results in a problem with bigger zones as there | 8054 | many revisions of zones. This results in a problem with bigger zones as there |
@@ -8055,16 +8058,22 @@ as they cause a CLI call of the namestore. Furthermore certain record types | |||
8055 | need transformation into a GNS compatible format which, depending on the record | 8058 | need transformation into a GNS compatible format which, depending on the record |
8056 | type, takes more time. | 8059 | type, takes more time. |
8057 | 8060 | ||
8061 | Further a blacklist was added to drop for instance DNSSEC related records. Also | ||
8062 | if a record type is neither in the white list nor the blacklist it is considered | ||
8063 | as a loss of data and a message is shown to the user. This helps with | ||
8064 | transparency and also with contributing, as the not supported record types can | ||
8065 | then be added accordingly. | ||
8066 | |||
8058 | @node DNS Zone Size | 8067 | @node DNS Zone Size |
8059 | @subsubsection DNS Zone Size | 8068 | @subsubsection DNS Zone Size |
8060 | |||
8061 | Another very big problem exists with very large zones. When migrating a small | 8069 | Another very big problem exists with very large zones. When migrating a small |
8062 | zone the delay between adding of records and their expiry is negligible. However | 8070 | zone the delay between adding of records and their expiry is negligible. However |
8063 | when working with a TLD zone that has more that 1 million records this delay | 8071 | when working with big zones that easily have more than a few million records |
8064 | becomes a problem. | 8072 | this delay becomes a problem. |
8065 | 8073 | ||
8066 | Records will start to expire well before the zone has finished migrating. This | 8074 | Records will start to expire well before the zone has finished migrating. This |
8067 | causes unwanted anomalies when trying to resolve records. | 8075 | is usually not a problem but can cause a high CPU load when a peer is restarted |
8076 | and the records have expired. | ||
8068 | 8077 | ||
8069 | A good solution has not been found yet. One of the idea that floated around was | 8078 | A good solution has not been found yet. One of the idea that floated around was |
8070 | that the records should be added with the s (shadow) flag to keep the records | 8079 | that the records should be added with the s (shadow) flag to keep the records |
@@ -8074,45 +8083,51 @@ of said record(s). | |||
8074 | 8083 | ||
8075 | Another problem that still persists is how to refresh records. Expired records | 8084 | Another problem that still persists is how to refresh records. Expired records |
8076 | are still displayed when calling gnunet-namestore but do not resolve with | 8085 | are still displayed when calling gnunet-namestore but do not resolve with |
8077 | gnunet-gns. When doing incremental zone transfers this becomes especially | 8086 | gnunet-gns. Zonemaster will sign the expired records again and make sure that |
8078 | apparent. | 8087 | the records are still valid. With a recent change this was fixed as gnunet-gns |
8088 | to improve the suffix lookup which allows for a fast lookup even with thousands | ||
8089 | of local egos. | ||
8090 | |||
8091 | Currently the pace of adding records in general is around 10 records per second. | ||
8092 | Crypto is the upper limit for adding of records. The performance of your machine | ||
8093 | can be tested with the perf_crypto_* tools. There is still a big discrepancy | ||
8094 | between the pace of Ascension and the theoretical limit. | ||
8079 | 8095 | ||
8080 | I estimate that the limit lies at about 200'000 records in a zone as this is | 8096 | A performance metric for measuring improvements has not yet been implemented in |
8081 | the limit that my machine is capable of adding within one hour. This was | 8097 | Ascension. |
8082 | calculated by running cProfile on the application with a zone of 5000 records | ||
8083 | and calculating what abouts a much bigger zones with 8 million records would | ||
8084 | take. This results in a nice metric of records migrated per hour. | ||
8085 | 8098 | ||
8086 | @node Performance | 8099 | @node Performance |
8087 | @subsubsection Performance | 8100 | @subsubsection Performance |
8088 | The performance when migrating a zone using the Ascension tool is limited by a | 8101 | The performance when migrating a zone using the Ascension tool is limited by a |
8089 | handful of factors. First of all ascension is written in Python3 and calls the | 8102 | handful of factors. First of all ascension is written in Python3 and calls the |
8090 | CLI tools of GNUnet. Furthermore all the records that are added to the same | 8103 | CLI tools of GNUnet. This is comparable to a fork and exec call which costs a |
8104 | few CPU cycles. Furthermore all the records that are added to the same | ||
8091 | label are signed using the zones private key. This signing operation is very | 8105 | label are signed using the zones private key. This signing operation is very |
8092 | resource heavy and was optimized during development by adding the '-R' | 8106 | resource heavy and was optimized during development by adding the '-R' |
8093 | (Recordline) option to gnunet-namestore. This allows to add multiple records | 8107 | (Recordline) option to gnunet-namestore which allows to specify multiple records |
8094 | at once using the CLI. | 8108 | using the CLI tool. Assuming that in a TLD zone every domain has at least two |
8095 | 8109 | name servers this halves the amount of signatures needed. | |
8096 | The result of this was a much faster migration of TLD zones, as most records | ||
8097 | with the same label have two name servers. | ||
8098 | 8110 | ||
8099 | Another improvement that could be made is with the addition of multiple threads | 8111 | Another improvement that could be made is with the addition of multiple threads |
8100 | when opening the GNUnet CLI tools. This could be implemented by simply creating | 8112 | or using asynchronous subprocesses when opening the GNUnet CLI tools. This could |
8101 | more workers in the program but performance improvements were not tested. | 8113 | be implemented by simply creating more workers in the program but performance |
8114 | improvements were not tested. | ||
8102 | 8115 | ||
8103 | During the entire development of Ascension sqlite was used as a database | 8116 | Ascension was tested using different hardware and database backends. Performance |
8104 | backend for GNUnet. Other backends have not been tested yet. | 8117 | differences between SQLite and postgresql are marginal and almost non existent. |
8118 | What did make a huge impact on record adding performance was the storage medium. | ||
8119 | On a traditional mechanical hard drive adding of records were slow compared to a | ||
8120 | solid state disk. | ||
8105 | 8121 | ||
8106 | In conclusion there are many bottlenecks still around in the program, namely the | 8122 | In conclusion there are many bottlenecks still around in the program, namely the |
8107 | signing process and the single threaded implementation. In the future a solution | 8123 | single threaded implementation and inefficient, sequential calls of |
8108 | that uses the C API would be cleaner and better. | 8124 | gnunet-namestore. In the future a solution that uses the C API would be cleaner |
8125 | and better. | ||
8109 | 8126 | ||
8110 | @cindex GNS Namecache | 8127 | @cindex GNS Namecache |
8111 | @node GNS Namecache | 8128 | @node GNS Namecache |
8112 | @section GNS Namecache | 8129 | @section GNS Namecache |
8113 | 8130 | ||
8114 | |||
8115 | |||
8116 | The NAMECACHE subsystem is responsible for caching (encrypted) resolution | 8131 | The NAMECACHE subsystem is responsible for caching (encrypted) resolution |
8117 | results of the GNU Name System (GNS). GNS makes zone information available | 8132 | results of the GNU Name System (GNS). GNS makes zone information available |
8118 | to other users via the DHT. However, as accessing the DHT for every | 8133 | to other users via the DHT. However, as accessing the DHT for every |