diff options
Diffstat (limited to 'doc/handbook/chapters/developer.texi')
-rw-r--r-- | doc/handbook/chapters/developer.texi | 338 |
1 files changed, 136 insertions, 202 deletions
diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi index 228603cda..830a345a8 100644 --- a/doc/handbook/chapters/developer.texi +++ b/doc/handbook/chapters/developer.texi | |||
@@ -79,6 +79,7 @@ new chapters, sections or insightful comments. | |||
79 | * File-sharing (FS) Subsystem:: | 79 | * File-sharing (FS) Subsystem:: |
80 | * REGEX Subsystem:: | 80 | * REGEX Subsystem:: |
81 | * REST Subsystem:: | 81 | * REST Subsystem:: |
82 | * RPS Subsystem:: | ||
82 | @end menu | 83 | @end menu |
83 | 84 | ||
84 | @node Developer Introduction | 85 | @node Developer Introduction |
@@ -556,7 +557,7 @@ stacked together to construct complex buildings and it is generally easy | |||
556 | to swap one block for a different one that has the same shape. GNUnet's | 557 | to swap one block for a different one that has the same shape. GNUnet's |
557 | architecture is based on LEGOs: | 558 | architecture is based on LEGOs: |
558 | 559 | ||
559 | @c @image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs as connectors upon Network Protocol on top of a Service} | 560 | @image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs upon IPC/network protocol provided by a service} |
560 | 561 | ||
561 | This chapter documents the GNUnet LEGO system, also known as GNUnet's | 562 | This chapter documents the GNUnet LEGO system, also known as GNUnet's |
562 | system architecture. | 563 | system architecture. |
@@ -573,10 +574,14 @@ Like services, they have holes to be filled by APIs of other services. | |||
573 | Unlike services, daemons do not implement their own network protocol and | 574 | Unlike services, daemons do not implement their own network protocol and |
574 | they have no API: | 575 | they have no API: |
575 | 576 | ||
577 | @image{images/daemon_lego_block,5in,,A daemon in GNUnet is a component that does not offer an API for others to build upon} | ||
578 | |||
576 | The GNUnet system provides a range of services, daemons and user | 579 | The GNUnet system provides a range of services, daemons and user |
577 | interfaces, which are then combined into a layered GNUnet instance (also | 580 | interfaces, which are then combined into a layered GNUnet instance (also |
578 | known as a peer). | 581 | known as a peer). |
579 | 582 | ||
583 | @image{images/service_stack,5in,,A GNUnet peer consists of many layers of services} | ||
584 | |||
580 | Note that while it is generally possible to swap one service for another | 585 | Note that while it is generally possible to swap one service for another |
581 | compatible service, there is often only one implementation. However, | 586 | compatible service, there is often only one implementation. However, |
582 | during development we often have a "new" version of a service in parallel | 587 | during development we often have a "new" version of a service in parallel |
@@ -1876,7 +1881,6 @@ Testbed API can accessed by including the | |||
1876 | * Hosts file format:: | 1881 | * Hosts file format:: |
1877 | * Topology file format:: | 1882 | * Topology file format:: |
1878 | * Testbed Barriers:: | 1883 | * Testbed Barriers:: |
1879 | * Automatic large-scale deployment in the PlanetLab testbed:: | ||
1880 | * TESTBED Caveats:: | 1884 | * TESTBED Caveats:: |
1881 | @end menu | 1885 | @end menu |
1882 | 1886 | ||
@@ -2140,168 +2144,6 @@ message from its upward propagation --- the upward propagation is needed | |||
2140 | for ensuring that the barrier is reached by all the controllers and the | 2144 | for ensuring that the barrier is reached by all the controllers and the |
2141 | downward propagation is for triggering that the barrier is crossed. | 2145 | downward propagation is for triggering that the barrier is crossed. |
2142 | 2146 | ||
2143 | @cindex PlanetLab testbed | ||
2144 | @node Automatic large-scale deployment in the PlanetLab testbed | ||
2145 | @subsection Automatic large-scale deployment in the PlanetLab testbed | ||
2146 | |||
2147 | PlanetLab is a testbed for computer networking and distributed systems | ||
2148 | research. It was established in 2002 and as of June 2010 was composed of | ||
2149 | 1090 nodes at 507 sites worldwide. | ||
2150 | |||
2151 | To automate the GNUnet we created a set of automation tools to simplify | ||
2152 | the large-scale deployment. We provide you a set of scripts you can use | ||
2153 | to deploy GNUnet on a set of nodes and manage your installation. | ||
2154 | |||
2155 | Please also check @uref{https://old.gnunet.org/installation-fedora8-svn} and | ||
2156 | @uref{https://old.gnunet.org/installation-fedora12-svn} to find detailed | ||
2157 | instructions how to install GNUnet on a PlanetLab node. | ||
2158 | |||
2159 | |||
2160 | @c *********************************************************************** | ||
2161 | @menu | ||
2162 | * PlanetLab Automation for Fedora8 nodes:: | ||
2163 | * Install buildslave on PlanetLab nodes running fedora core 8:: | ||
2164 | * Setup a new PlanetLab testbed using GPLMT:: | ||
2165 | * Why do i get an ssh error when using the regex profiler?:: | ||
2166 | @end menu | ||
2167 | |||
2168 | @node PlanetLab Automation for Fedora8 nodes | ||
2169 | @subsubsection PlanetLab Automation for Fedora8 nodes | ||
2170 | |||
2171 | @c *********************************************************************** | ||
2172 | @node Install buildslave on PlanetLab nodes running fedora core 8 | ||
2173 | @subsubsection Install buildslave on PlanetLab nodes running fedora core 8 | ||
2174 | @c ** Actually this is a subsubsubsection, but must be fixed differently | ||
2175 | @c ** as subsubsection is the lowest. | ||
2176 | |||
2177 | Since most of the PlanetLab nodes are running the very old Fedora core 8 | ||
2178 | image, installing the buildslave software is quite some pain. For our | ||
2179 | PlanetLab testbed we figured out how to install the buildslave software | ||
2180 | best. | ||
2181 | |||
2182 | @c This is a very terrible way to suggest installing software. | ||
2183 | @c FIXME: Is there an official, safer way instead of blind-piping a | ||
2184 | @c script? | ||
2185 | @c FIXME: Use newer pypi URLs below. | ||
2186 | Install Distribute for Python: | ||
2187 | |||
2188 | @example | ||
2189 | curl http://python-distribute.org/distribute_setup.py | sudo python | ||
2190 | @end example | ||
2191 | |||
2192 | Install Distribute for zope.interface <= 3.8.0 (4.0 and 4.0.1 will not | ||
2193 | work): | ||
2194 | |||
2195 | @example | ||
2196 | export PYPI=@value{PYPI-URL} | ||
2197 | wget $PYPI/z/zope.interface/zope.interface-3.8.0.tar.gz | ||
2198 | tar xzvf zope.interface-3.8.0.tar.gz | ||
2199 | cd zope.interface-3.8.0 | ||
2200 | sudo python setup.py install | ||
2201 | @end example | ||
2202 | |||
2203 | Install the buildslave software (0.8.6 was the latest version): | ||
2204 | |||
2205 | @example | ||
2206 | export GCODE="http://buildbot.googlecode.com/files" | ||
2207 | wget $GCODE/buildbot-slave-0.8.6p1.tar.gz | ||
2208 | tar xvfz buildbot-slave-0.8.6p1.tar.gz | ||
2209 | cd buildslave-0.8.6p1 | ||
2210 | sudo python setup.py install | ||
2211 | @end example | ||
2212 | |||
2213 | The setup will download the matching twisted package and install it. | ||
2214 | It will also try to install the latest version of zope.interface which | ||
2215 | will fail to install. Buildslave will work anyway since version 3.8.0 | ||
2216 | was installed before! | ||
2217 | |||
2218 | @c *********************************************************************** | ||
2219 | @node Setup a new PlanetLab testbed using GPLMT | ||
2220 | @subsubsection Setup a new PlanetLab testbed using GPLMT | ||
2221 | |||
2222 | @itemize @bullet | ||
2223 | @item Get a new slice and assign nodes | ||
2224 | Ask your PlanetLab PI to give you a new slice and assign the nodes you | ||
2225 | need | ||
2226 | @item Install a buildmaster | ||
2227 | You can stick to the buildbot documentation:@ | ||
2228 | @uref{http://buildbot.net/buildbot/docs/current/manual/installation.html} | ||
2229 | @item Install the buildslave software on all nodes | ||
2230 | To install the buildslave on all nodes assigned to your slice you can use | ||
2231 | the tasklist @code{install_buildslave_fc8.xml} provided with GPLMT: | ||
2232 | |||
2233 | @example | ||
2234 | ./gplmt.py -c contrib/tumple_gnunet.conf -t \ | ||
2235 | contrib/tasklists/install_buildslave_fc8.xml -a -p <planetlab password> | ||
2236 | @end example | ||
2237 | |||
2238 | @item Create the buildmaster configuration and the slave setup commands | ||
2239 | |||
2240 | The master and the and the slaves have need to have credentials and the | ||
2241 | master has to have all nodes configured. This can be done with the | ||
2242 | @file{create_buildbot_configuration.py} script in the @file{scripts} | ||
2243 | directory. | ||
2244 | |||
2245 | This scripts takes a list of nodes retrieved directly from PlanetLab or | ||
2246 | read from a file and a configuration template and creates: | ||
2247 | |||
2248 | @itemize @bullet | ||
2249 | @item a tasklist which can be executed with gplmt to setup the slaves | ||
2250 | @item a master.cfg file containing a PlanetLab nodes | ||
2251 | @end itemize | ||
2252 | |||
2253 | A configuration template is included in the <contrib>, most important is | ||
2254 | that the script replaces the following tags in the template: | ||
2255 | |||
2256 | %GPLMT_BUILDER_DEFINITION :@ GPLMT_BUILDER_SUMMARY@ GPLMT_SLAVES@ | ||
2257 | %GPLMT_SCHEDULER_BUILDERS | ||
2258 | |||
2259 | Create configuration for all nodes assigned to a slice: | ||
2260 | |||
2261 | @example | ||
2262 | ./create_buildbot_configuration.py -u <planetlab username> \ | ||
2263 | -p <planetlab password> -s <slice> -m <buildmaster+port> \ | ||
2264 | -t <template> | ||
2265 | @end example | ||
2266 | |||
2267 | Create configuration for some nodes in a file: | ||
2268 | |||
2269 | @example | ||
2270 | ./create_buildbot_configuration.p -f <node_file> \ | ||
2271 | -m <buildmaster+port> -t <template> | ||
2272 | @end example | ||
2273 | |||
2274 | @item Copy the @file{master.cfg} to the buildmaster and start it | ||
2275 | Use @code{buildbot start <basedir>} to start the server | ||
2276 | @item Setup the buildslaves | ||
2277 | @end itemize | ||
2278 | |||
2279 | @c *********************************************************************** | ||
2280 | @node Why do i get an ssh error when using the regex profiler? | ||
2281 | @subsubsection Why do i get an ssh error when using the regex profiler? | ||
2282 | |||
2283 | Why do i get an ssh error "Permission denied (publickey,password)." when | ||
2284 | using the regex profiler although passwordless ssh to localhost works | ||
2285 | using publickey and ssh-agent? | ||
2286 | |||
2287 | You have to generate a public/private-key pair with no password:@ | ||
2288 | @code{ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_localhost}@ | ||
2289 | and then add the following to your ~/.ssh/config file: | ||
2290 | |||
2291 | @code{Host 127.0.0.1@ IdentityFile ~/.ssh/id_localhost} | ||
2292 | |||
2293 | now make sure your hostsfile looks like | ||
2294 | |||
2295 | @example | ||
2296 | [USERNAME]@@127.0.0.1:22@ | ||
2297 | [USERNAME]@@127.0.0.1:22 | ||
2298 | @end example | ||
2299 | |||
2300 | You can test your setup by running @code{ssh 127.0.0.1} in a | ||
2301 | terminal and then in the opened session run it again. | ||
2302 | If you were not asked for a password on either login, | ||
2303 | then you should be good to go. | ||
2304 | |||
2305 | @cindex TESTBED Caveats | 2147 | @cindex TESTBED Caveats |
2306 | @node TESTBED Caveats | 2148 | @node TESTBED Caveats |
2307 | @subsection TESTBED Caveats | 2149 | @subsection TESTBED Caveats |
@@ -8116,6 +7958,14 @@ This section discusses the challenges and problems faced when writing the | |||
8116 | Ascension tool. It also takes a look at possible improvements in the | 7958 | Ascension tool. It also takes a look at possible improvements in the |
8117 | future. | 7959 | future. |
8118 | 7960 | ||
7961 | Consider the following diagram that shows the workflow of Ascension: | ||
7962 | |||
7963 | @image{images/ascension_ssd,6in,,Ascensions workflow} | ||
7964 | |||
7965 | Further the interaction between components of GNUnet are shown in the diagram | ||
7966 | below: | ||
7967 | @center @image{images/ascension_interaction,,6in,Ascensions workflow} | ||
7968 | |||
8119 | @menu | 7969 | @menu |
8120 | * Conversions between DNS and GNS:: | 7970 | * Conversions between DNS and GNS:: |
8121 | * DNS Zone Size:: | 7971 | * DNS Zone Size:: |
@@ -8126,9 +7976,9 @@ future. | |||
8126 | @node Conversions between DNS and GNS | 7976 | @node Conversions between DNS and GNS |
8127 | @subsubsection Conversions between DNS and GNS | 7977 | @subsubsection Conversions between DNS and GNS |
8128 | 7978 | ||
8129 | The differences between the two name systems lies in the details | 7979 | The differences between the two name systems lies in the details and is not |
8130 | and is not always transparent. | 7980 | always transparent. For instance an SRV record is converted to a BOX record |
8131 | For instance an SRV record is converted to a GNS only BOX record. | 7981 | which is unique to GNS. |
8132 | 7982 | ||
8133 | This is done by converting to a BOX record from an existing SRV record: | 7983 | This is done by converting to a BOX record from an existing SRV record: |
8134 | 7984 | ||
@@ -8141,7 +7991,7 @@ _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com. | |||
8141 | 14000 BOX n 5060 6 33 0 0 5060 www.example.com | 7991 | 14000 BOX n 5060 6 33 0 0 5060 www.example.com |
8142 | @end example | 7992 | @end example |
8143 | 7993 | ||
8144 | Other records that have such a transformation is the MX record type, | 7994 | Other records that need to undergo such transformation is the MX record type, |
8145 | as well as the SOA record type. | 7995 | as well as the SOA record type. |
8146 | 7996 | ||
8147 | Transformation of a SOA record into GNS works as described in the | 7997 | Transformation of a SOA record into GNS works as described in the |
@@ -8156,8 +8006,9 @@ following example. Very important to note are the rname and mname keys. | |||
8156 | 604800 ; expire | 8006 | 604800 ; expire |
8157 | 600 ) ; ttl | 8007 | 600 ) ; ttl |
8158 | # Recordline for adding the record | 8008 | # Recordline for adding the record |
8159 | $ gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \ | 8009 | $ gnunet-namestore -z example.com -a -n @ -t SOA -V \ |
8160 | mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s | 8010 | rname=master.example.com mname=hostmaster.example.com \ |
8011 | 2017030300,3600,1800,604800,600 -e 7200s | ||
8161 | @end example | 8012 | @end example |
8162 | 8013 | ||
8163 | The transformation of MX records is done in a simple way. | 8014 | The transformation of MX records is done in a simple way. |
@@ -8166,10 +8017,10 @@ The transformation of MX records is done in a simple way. | |||
8166 | $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail | 8017 | $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail |
8167 | @end example | 8018 | @end example |
8168 | 8019 | ||
8169 | Finally, one of the biggest struggling points were the NS records that are found | 8020 | Finally, one of the biggest struggling points were the NS records that are |
8170 | in top level domain zones. The intended behaviour for those is to add GNS2DNS | 8021 | found in top level domain zones. The intended behaviour for those is to add |
8171 | records for those so that gnunet-gns can resolve records for those domains on | 8022 | GNS2DNS records for those so that gnunet-gns can resolve records for those |
8172 | its own. This requires migration of the DNS GLUE records as well, provided that | 8023 | domains on its own. Those require the values from DNS GLUE records, provided |
8173 | they are within the same zone. | 8024 | they are within the same zone. |
8174 | 8025 | ||
8175 | The following two examples show one record with a GLUE record and the other one | 8026 | The following two examples show one record with a GLUE record and the other one |
@@ -8178,10 +8029,12 @@ does not have a GLUE record. This takes place in the 'com' TLD. | |||
8178 | @example | 8029 | @example |
8179 | # ns1.example.com 86400 IN A 127.0.0.1 | 8030 | # ns1.example.com 86400 IN A 127.0.0.1 |
8180 | # example.com 86400 IN NS ns1.example.com. | 8031 | # example.com 86400 IN NS ns1.example.com. |
8181 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@127.0.0.1 | 8032 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ |
8033 | example.com@@127.0.0.1 | ||
8182 | 8034 | ||
8183 | # example.com 86400 IN NS ns1.example.org. | 8035 | # example.com 86400 IN NS ns1.example.org. |
8184 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n example.com@@ns1.example.org | 8036 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ |
8037 | example.com@@ns1.example.org | ||
8185 | @end example | 8038 | @end example |
8186 | 8039 | ||
8187 | As you can see, one of the GNS2DNS records has an IP address listed and the | 8040 | As you can see, one of the GNS2DNS records has an IP address listed and the |
@@ -8204,7 +8057,7 @@ Currently the following record types are supported: | |||
8204 | @item TXT | 8057 | @item TXT |
8205 | @end itemize | 8058 | @end itemize |
8206 | 8059 | ||
8207 | This is not due to a technical limitation but rather a practical one. The | 8060 | This is not due to technical limitations but rather a practical ones. The |
8208 | problem occurs with DNSSEC enabled DNS zones. As records within those zones are | 8061 | problem occurs with DNSSEC enabled DNS zones. As records within those zones are |
8209 | signed periodically, and every new signature is an update to the zone, there are | 8062 | signed periodically, and every new signature is an update to the zone, there are |
8210 | many revisions of zones. This results in a problem with bigger zones as there | 8063 | many revisions of zones. This results in a problem with bigger zones as there |
@@ -8214,16 +8067,22 @@ as they cause a CLI call of the namestore. Furthermore certain record types | |||
8214 | need transformation into a GNS compatible format which, depending on the record | 8067 | need transformation into a GNS compatible format which, depending on the record |
8215 | type, takes more time. | 8068 | type, takes more time. |
8216 | 8069 | ||
8070 | Further a blacklist was added to drop for instance DNSSEC related records. Also | ||
8071 | if a record type is neither in the white list nor the blacklist it is considered | ||
8072 | as a loss of data and a message is shown to the user. This helps with | ||
8073 | transparency and also with contributing, as the not supported record types can | ||
8074 | then be added accordingly. | ||
8075 | |||
8217 | @node DNS Zone Size | 8076 | @node DNS Zone Size |
8218 | @subsubsection DNS Zone Size | 8077 | @subsubsection DNS Zone Size |
8219 | |||
8220 | Another very big problem exists with very large zones. When migrating a small | 8078 | Another very big problem exists with very large zones. When migrating a small |
8221 | zone the delay between adding of records and their expiry is negligible. However | 8079 | zone the delay between adding of records and their expiry is negligible. However |
8222 | when working with a TLD zone that has more that 1 million records this delay | 8080 | when working with big zones that easily have more than a few million records |
8223 | becomes a problem. | 8081 | this delay becomes a problem. |
8224 | 8082 | ||
8225 | Records will start to expire well before the zone has finished migrating. This | 8083 | Records will start to expire well before the zone has finished migrating. This |
8226 | causes unwanted anomalies when trying to resolve records. | 8084 | is usually not a problem but can cause a high CPU load when a peer is restarted |
8085 | and the records have expired. | ||
8227 | 8086 | ||
8228 | A good solution has not been found yet. One of the idea that floated around was | 8087 | A good solution has not been found yet. One of the idea that floated around was |
8229 | that the records should be added with the s (shadow) flag to keep the records | 8088 | that the records should be added with the s (shadow) flag to keep the records |
@@ -8233,45 +8092,51 @@ of said record(s). | |||
8233 | 8092 | ||
8234 | Another problem that still persists is how to refresh records. Expired records | 8093 | Another problem that still persists is how to refresh records. Expired records |
8235 | are still displayed when calling gnunet-namestore but do not resolve with | 8094 | are still displayed when calling gnunet-namestore but do not resolve with |
8236 | gnunet-gns. When doing incremental zone transfers this becomes especially | 8095 | gnunet-gns. Zonemaster will sign the expired records again and make sure that |
8237 | apparent. | 8096 | the records are still valid. With a recent change this was fixed as gnunet-gns |
8097 | to improve the suffix lookup which allows for a fast lookup even with thousands | ||
8098 | of local egos. | ||
8099 | |||
8100 | Currently the pace of adding records in general is around 10 records per second. | ||
8101 | Crypto is the upper limit for adding of records. The performance of your machine | ||
8102 | can be tested with the perf_crypto_* tools. There is still a big discrepancy | ||
8103 | between the pace of Ascension and the theoretical limit. | ||
8238 | 8104 | ||
8239 | I estimate that the limit lies at about 200'000 records in a zone as this is | 8105 | A performance metric for measuring improvements has not yet been implemented in |
8240 | the limit that my machine is capable of adding within one hour. This was | 8106 | Ascension. |
8241 | calculated by running cProfile on the application with a zone of 5000 records | ||
8242 | and calculating what abouts a much bigger zones with 8 million records would | ||
8243 | take. This results in a nice metric of records migrated per hour. | ||
8244 | 8107 | ||
8245 | @node Performance | 8108 | @node Performance |
8246 | @subsubsection Performance | 8109 | @subsubsection Performance |
8247 | The performance when migrating a zone using the Ascension tool is limited by a | 8110 | The performance when migrating a zone using the Ascension tool is limited by a |
8248 | handful of factors. First of all ascension is written in Python3 and calls the | 8111 | handful of factors. First of all ascension is written in Python3 and calls the |
8249 | CLI tools of GNUnet. Furthermore all the records that are added to the same | 8112 | CLI tools of GNUnet. This is comparable to a fork and exec call which costs a |
8113 | few CPU cycles. Furthermore all the records that are added to the same | ||
8250 | label are signed using the zones private key. This signing operation is very | 8114 | label are signed using the zones private key. This signing operation is very |
8251 | resource heavy and was optimized during development by adding the '-R' | 8115 | resource heavy and was optimized during development by adding the '-R' |
8252 | (Recordline) option to gnunet-namestore. This allows to add multiple records | 8116 | (Recordline) option to gnunet-namestore which allows to specify multiple records |
8253 | at once using the CLI. | 8117 | using the CLI tool. Assuming that in a TLD zone every domain has at least two |
8254 | 8118 | name servers this halves the amount of signatures needed. | |
8255 | The result of this was a much faster migration of TLD zones, as most records | ||
8256 | with the same label have two name servers. | ||
8257 | 8119 | ||
8258 | Another improvement that could be made is with the addition of multiple threads | 8120 | Another improvement that could be made is with the addition of multiple threads |
8259 | when opening the GNUnet CLI tools. This could be implemented by simply creating | 8121 | or using asynchronous subprocesses when opening the GNUnet CLI tools. This could |
8260 | more workers in the program but performance improvements were not tested. | 8122 | be implemented by simply creating more workers in the program but performance |
8123 | improvements were not tested. | ||
8261 | 8124 | ||
8262 | During the entire development of Ascension sqlite was used as a database | 8125 | Ascension was tested using different hardware and database backends. Performance |
8263 | backend for GNUnet. Other backends have not been tested yet. | 8126 | differences between SQLite and postgresql are marginal and almost non existent. |
8127 | What did make a huge impact on record adding performance was the storage medium. | ||
8128 | On a traditional mechanical hard drive adding of records were slow compared to a | ||
8129 | solid state disk. | ||
8264 | 8130 | ||
8265 | In conclusion there are many bottlenecks still around in the program, namely the | 8131 | In conclusion there are many bottlenecks still around in the program, namely the |
8266 | signing process and the single threaded implementation. In the future a solution | 8132 | single threaded implementation and inefficient, sequential calls of |
8267 | that uses the C API would be cleaner and better. | 8133 | gnunet-namestore. In the future a solution that uses the C API would be cleaner |
8134 | and better. | ||
8268 | 8135 | ||
8269 | @cindex GNS Namecache | 8136 | @cindex GNS Namecache |
8270 | @node GNS Namecache | 8137 | @node GNS Namecache |
8271 | @section GNS Namecache | 8138 | @section GNS Namecache |
8272 | 8139 | ||
8273 | |||
8274 | |||
8275 | The NAMECACHE subsystem is responsible for caching (encrypted) resolution | 8140 | The NAMECACHE subsystem is responsible for caching (encrypted) resolution |
8276 | results of the GNU Name System (GNS). GNS makes zone information available | 8141 | results of the GNU Name System (GNS). GNS makes zone information available |
8277 | to other users via the DHT. However, as accessing the DHT for every | 8142 | to other users via the DHT. However, as accessing the DHT for every |
@@ -9007,3 +8872,72 @@ so please make sure that endpoints are unambiguous. | |||
9007 | 8872 | ||
9008 | This is WIP. Endpoints should be documented appropriately. | 8873 | This is WIP. Endpoints should be documented appropriately. |
9009 | Preferably using annotations. | 8874 | Preferably using annotations. |
8875 | |||
8876 | |||
8877 | @cindex RPS Subsystem | ||
8878 | @node RPS Subsystem | ||
8879 | @section RPS Subsystem | ||
8880 | |||
8881 | In literature, Random Peer Sampling (RPS) refers to the problem of | ||
8882 | reliably drawing random samples from an unstructured p2p network. | ||
8883 | |||
8884 | Doing so in a reliable manner is not only hard because of inherent | ||
8885 | problems but also because of possible malicious peers that could try to | ||
8886 | bias the selection. | ||
8887 | |||
8888 | It is useful for all kind of gossip protocols that require the selection | ||
8889 | of random peers in the whole network like gathering statistics, | ||
8890 | spreading and aggregating information in the network, load balancing and | ||
8891 | overlay topology management. | ||
8892 | |||
8893 | The approach chosen in the rps implementation in GNUnet follows the | ||
8894 | Brahms@uref{https://bib.gnunet.org/full/date.html\#2009_5f0} design. | ||
8895 | |||
8896 | The current state is "work in progress". There are a lot of things that | ||
8897 | need to be done, primarily finishing the experimental evaluation and a | ||
8898 | re-design of the API. | ||
8899 | |||
8900 | The abstract idea is to subscribe to connect to/start the rps service | ||
8901 | and request random peers that will be returned when they represent a | ||
8902 | random selection from the whole network with high probability. | ||
8903 | |||
8904 | An additional feature to the original Brahms-design is the selection of | ||
8905 | sub-groups: The GNUnet implementation of rps enables clients to ask for | ||
8906 | random peers from a group that is defined by a common shared secret. | ||
8907 | (The secret could of course also be public, depending on the use-case.) | ||
8908 | |||
8909 | Another addition to the original protocol was made: The sampler | ||
8910 | mechanism that was introduced in Brahms was slightly adapted and used to | ||
8911 | actually sample the peers and returned to the client. | ||
8912 | This is necessary as the original design only keeps peers connected to | ||
8913 | random other peers in the network. In order to return random peers to | ||
8914 | client requests independently random, they cannot be drawn from the | ||
8915 | connected peers. | ||
8916 | The adapted sampler makes sure that each request for random peers is | ||
8917 | independent from the others. | ||
8918 | |||
8919 | @node Brahms | ||
8920 | @subsection Brahms | ||
8921 | The high-level concept of Brahms is two-fold: Combining push-pull gossip | ||
8922 | with locally fixing a assumed bias using cryptographic min-wise | ||
8923 | permutations. | ||
8924 | The central data structure is the view - a peer's current local sample. | ||
8925 | This view is used to select peers to push to and pull from. | ||
8926 | This simple mechanism can be biased easily. For this reason Brahms | ||
8927 | 'fixes' the bias by using the so-called sampler. A data structure that | ||
8928 | takes a list of elements as input and outputs a random one of them | ||
8929 | independently of the frequency in the input set. Both an element that | ||
8930 | was put into the sampler a single time and an element that was put into | ||
8931 | it a million times have the same probability of being the output. | ||
8932 | This is achieved this is achieved with exploiting min-wise independent | ||
8933 | permutations. In rps we use HMACs: On the initialisation of a sampler | ||
8934 | element, a key is chosen at random. On each input the HMAC with the | ||
8935 | random key is computed. The sampler element keeps the element with the | ||
8936 | minimal HMAC. | ||
8937 | |||
8938 | In order to fix the bias in the view, a fraction of the elements in the | ||
8939 | view are sampled through the sampler from the random stream of peer IDs. | ||
8940 | |||
8941 | According to the theoretical analysis of Bortnikov et al. this suffices | ||
8942 | to keep the network connected and having random peers in the view. | ||
8943 | |||