aboutsummaryrefslogtreecommitdiff
path: root/doc/documentation/chapters/developer.texi
diff options
context:
space:
mode:
Diffstat (limited to 'doc/documentation/chapters/developer.texi')
-rw-r--r--doc/documentation/chapters/developer.texi1389
1 files changed, 762 insertions, 627 deletions
diff --git a/doc/documentation/chapters/developer.texi b/doc/documentation/chapters/developer.texi
index 81a9a6c1f..f92257292 100644
--- a/doc/documentation/chapters/developer.texi
+++ b/doc/documentation/chapters/developer.texi
@@ -771,24 +771,27 @@ if (NULL == (value = lookup_function())) @{ error(); return; @}
771deep(er) nesting. Thus, we would write: 771deep(er) nesting. Thus, we would write:
772 772
773@example 773@example
774next = head; while (NULL != (pos = next)) @{ next = pos->next; if (! 774next = head; while (NULL != (pos = next)) @{
775should_free (pos)) continue; GNUNET_CONTAINER_DLL_remove (head, tail, pos); 775 next = pos->next; if (! should_free (pos))
776GNUNET_free (pos); @} 776 continue;
777 GNUNET_CONTAINER_DLL_remove (head, tail, pos);
778 GNUNET_free (pos); @}
777@end example 779@end example
778 780
779
780instead of 781instead of
782
781@example 783@example
782next = head; while (NULL != (pos = next)) @{ next = 784next = head; while (NULL != (pos = next)) @{
783pos->next; if (should_free (pos)) @{ 785 next = pos->next; if (should_free (pos)) @{
784 /* unnecessary nesting! */ 786 /* unnecessary nesting! */
785 GNUNET_CONTAINER_DLL_remove (head, tail, pos); GNUNET_free (pos); @} @} 787 GNUNET_CONTAINER_DLL_remove (head, tail, pos);
788 GNUNET_free (pos); @} @}
786@end example 789@end example
787 790
788 791
789@item We primarily use @code{for} and @code{while} loops. A @code{while} 792@item We primarily use @code{for} and @code{while} loops.
790loop is used if the method for advancing in the loop is not a 793A @code{while} loop is used if the method for advancing in the loop is
791straightforward increment operation. In particular, we use: 794not a straightforward increment operation. In particular, we use:
792 795
793@example 796@example
794next = head; 797next = head;
@@ -802,7 +805,6 @@ while (NULL != (pos = next))
802@} 805@}
803@end example 806@end example
804 807
805
806to free entries in a list (as the iteration changes the structure of the 808to free entries in a list (as the iteration changes the structure of the
807list due to the free; the equivalent @code{for} loop does no longer 809list due to the free; the equivalent @code{for} loop does no longer
808follow the simple @code{for} paradigm of @code{for(INIT;TEST;INC)}). 810follow the simple @code{for} paradigm of @code{for(INIT;TEST;INC)}).
@@ -889,8 +891,8 @@ If you want to compile and run benchmarks, run configure with the
889@code{--enable-benchmarks} option. 891@code{--enable-benchmarks} option.
890 892
891If you want to obtain code coverage results, run configure with the 893If you want to obtain code coverage results, run configure with the
892@code{--enable-coverage} option and run the coverage.sh script in 894@code{--enable-coverage} option and run the @file{coverage.sh} script in
893@file{contrib/}. 895the @file{contrib/} directory.
894 896
895@c *********************************************************************** 897@c ***********************************************************************
896@node Developing extensions for GNUnet using the gnunet-ext template 898@node Developing extensions for GNUnet using the gnunet-ext template
@@ -905,7 +907,9 @@ development of GNUnet services, command line tools, APIs and tests.
905 907
906First of all you have to obtain gnunet-ext from git: 908First of all you have to obtain gnunet-ext from git:
907 909
908@code{git clone https://gnunet.org/git/gnunet-ext.git} 910@example
911git clone https://gnunet.org/git/gnunet-ext.git
912@end example
909 913
910The next step is to bootstrap and configure it. For configure you have to 914The next step is to bootstrap and configure it. For configure you have to
911provide the path containing GNUnet with 915provide the path containing GNUnet with
@@ -922,7 +926,9 @@ path, you have to add @code{/path/to/gnunet} to the file
922@file{/etc/ld.so.conf} and run @code{ldconfig} or your add it to the 926@file{/etc/ld.so.conf} and run @code{ldconfig} or your add it to the
923environmental variable @code{LD_LIBRARY_PATH} by using 927environmental variable @code{LD_LIBRARY_PATH} by using
924 928
925@code{export LD_LIBRARY_PATH=/path/to/gnunet/lib} 929@example
930export LD_LIBRARY_PATH=/path/to/gnunet/lib
931@end example
926 932
927@c *********************************************************************** 933@c ***********************************************************************
928@node Writing testcases 934@node Writing testcases
@@ -942,9 +948,9 @@ progress information and not display debug messages by default. The
942success or failure of a testcase must be indicated by returning zero 948success or failure of a testcase must be indicated by returning zero
943(success) or non-zero (failure) from the main method of the testcase. The 949(success) or non-zero (failure) from the main method of the testcase. The
944integration with the autotools is relatively straightforward and only 950integration with the autotools is relatively straightforward and only
945requires modifications to the @code{Makefile.am} in the directory 951requires modifications to the @file{Makefile.am} in the directory
946containing the testcase. For a testcase testing the code in @code{foo.c} 952containing the testcase. For a testcase testing the code in @file{foo.c}
947the @code{Makefile.am} would contain the following lines: 953the @file{Makefile.am} would contain the following lines:
948 954
949@example 955@example
950check_PROGRAMS = test_foo TESTS = $(check_PROGRAMS) test_foo_SOURCES = 956check_PROGRAMS = test_foo TESTS = $(check_PROGRAMS) test_foo_SOURCES =
@@ -1122,16 +1128,28 @@ The following code illustrates spawning and killing an ARM process from a
1122testcase: 1128testcase:
1123 1129
1124@example 1130@example
1125static void run (void *cls, char *const *args, const char 1131static void run (void *cls,
1126*cfgfile, const struct GNUNET_CONFIGURATION_Handle *cfg) @{ struct 1132 char *const *args,
1127GNUNET_OS_Process *arm_pid; arm_pid = GNUNET_OS_start_process (NULL, NULL, 1133 const char *cfgfile,
1128"gnunet-service-arm", "gnunet-service-arm", "-c", cfgname, NULL); 1134 const struct GNUNET_CONFIGURATION_Handle *cfg) @{
1135 struct GNUNET_OS_Process *arm_pid;
1136 arm_pid = GNUNET_OS_start_process (NULL,
1137 NULL,
1138 "gnunet-service-arm",
1139 "gnunet-service-arm",
1140 "-c",
1141 cfgname,
1142 NULL);
1129 /* do real test work here */ 1143 /* do real test work here */
1130 if (0 != GNUNET_OS_process_kill (arm_pid, SIGTERM)) GNUNET_log_strerror 1144 if (0 != GNUNET_OS_process_kill (arm_pid, SIGTERM))
1131 (GNUNET_ERROR_TYPE_WARNING, "kill"); GNUNET_assert (GNUNET_OK == 1145 GNUNET_log_strerror
1132 GNUNET_OS_process_wait (arm_pid)); GNUNET_OS_process_close (arm_pid); @} 1146 (GNUNET_ERROR_TYPE_WARNING, "kill");
1133 1147 GNUNET_assert (GNUNET_OK == GNUNET_OS_process_wait (arm_pid));
1134GNUNET_PROGRAM_run (argc, argv, "NAME-OF-TEST", "nohelp", options, &run, cls); 1148 GNUNET_OS_process_close (arm_pid); @}
1149
1150GNUNET_PROGRAM_run (argc, argv,
1151 "NAME-OF-TEST",
1152 "nohelp", options, &run, cls);
1135@end example 1153@end example
1136 1154
1137 1155
@@ -1183,8 +1201,11 @@ The code in the test should look like this:
1183 1201
1184int main (int argc, char *argv[]) @{ 1202int main (int argc, char *argv[]) @{
1185 1203
1186 [run test, generate data] GAUGER("YOUR_MODULE", "METRIC_NAME", (float)value, 1204 [run test, generate data]
1187 "UNIT"); @} 1205 GAUGER("YOUR_MODULE",
1206 "METRIC_NAME",
1207 (float)value,
1208 "UNIT"); @}
1188@end example 1209@end example
1189 1210
1190 1211
@@ -1310,10 +1331,8 @@ found in the standard path. This can be addressed by setting the variable
1310`HELPER_BINARY_PATH' to the path of the testbed helper. Testbed API will 1331`HELPER_BINARY_PATH' to the path of the testbed helper. Testbed API will
1311then use this path to start helper binaries both locally and remotely. 1332then use this path to start helper binaries both locally and remotely.
1312 1333
1313Testbed API can accessed by including "gnunet_testbed_service.h" file and 1334Testbed API can accessed by including the
1314linking with -lgnunettestbed. 1335"@file{gnunet_testbed_service.h}" file and linking with -lgnunettestbed.
1315
1316
1317 1336
1318@c *********************************************************************** 1337@c ***********************************************************************
1319@menu 1338@menu
@@ -1628,16 +1647,20 @@ Install Distribute for zope.interface <= 3.8.0 (4.0 and 4.0.1 will not
1628work): 1647work):
1629 1648
1630@example 1649@example
1631wget https://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.8.0.tar.gz 1650export PYPI="https://pypi.python.org/packages/source"
1632tar zvfz zope.interface-3.8.0.tar.gz@ cd zope.interface-3.8.0 1651wget $PYPI/z/zope.interface/zope.interface-3.8.0.tar.gz
1652tar zvfz zope.interface-3.8.0.tar.gz
1653cd zope.interface-3.8.0
1633sudo python setup.py install 1654sudo python setup.py install
1634@end example 1655@end example
1635 1656
1636Install the buildslave software (0.8.6 was the latest version): 1657Install the buildslave software (0.8.6 was the latest version):
1637 1658
1638@example 1659@example
1639wget http://buildbot.googlecode.com/files/buildbot-slave-0.8.6p1.tar.gz 1660export GCODE="http://buildbot.googlecode.com/files"
1640tar xvfz buildbot-slave-0.8.6p1.tar.gz@ cd buildslave-0.8.6p1 1661wget $GCODE/buildbot-slave-0.8.6p1.tar.gz
1662tar xvfz buildbot-slave-0.8.6p1.tar.gz
1663cd buildslave-0.8.6p1
1641sudo python setup.py install 1664sudo python setup.py install
1642@end example 1665@end example
1643 1666
@@ -1670,8 +1693,8 @@ contrib/tasklists/install_buildslave_fc8.xml -a -p <planetlab password>
1670 1693
1671The master and the and the slaves have need to have credentials and the 1694The master and the and the slaves have need to have credentials and the
1672master has to have all nodes configured. This can be done with the 1695master has to have all nodes configured. This can be done with the
1673@code{create_buildbot_configuration.py} script in the @code{scripts} 1696@file{create_buildbot_configuration.py} script in the @file{scripts}
1674directory 1697directory.
1675 1698
1676This scripts takes a list of nodes retrieved directly from PlanetLab or 1699This scripts takes a list of nodes retrieved directly from PlanetLab or
1677read from a file and a configuration template and creates: 1700read from a file and a configuration template and creates:
@@ -1687,14 +1710,22 @@ that the script replaces the following tags in the template:
1687%GPLMT_BUILDER_DEFINITION :@ GPLMT_BUILDER_SUMMARY@ GPLMT_SLAVES@ 1710%GPLMT_BUILDER_DEFINITION :@ GPLMT_BUILDER_SUMMARY@ GPLMT_SLAVES@
1688%GPLMT_SCHEDULER_BUILDERS 1711%GPLMT_SCHEDULER_BUILDERS
1689 1712
1690Create configuration for all nodes assigned to a slice:@ @code{@ 1713Create configuration for all nodes assigned to a slice:
1691./create_buildbot_configuration.py -u <planetlab username> -p <planetlab 1714
1692password> -s <slice> -m <buildmaster+port> -t <template>@ }@ Create 1715@example
1693configuration for some nodes in a file:@ @code{@ 1716./create_buildbot_configuration.py -u <planetlab username> \
1694./create_buildbot_configuration.p -f <node_file> -m <buildmaster+port> -t 1717-p <planetlab password> -s <slice> -m <buildmaster+port> \
1695<template>@ } 1718-t <template>
1719@end example
1720
1721Create configuration for some nodes in a file:
1722
1723@example
1724./create_buildbot_configuration.p -f <node_file> \
1725-m <buildmaster+port> -t <template>
1726@end example
1696 1727
1697@item Copy the @code{master.cfg} to the buildmaster and start it 1728@item Copy the @file{master.cfg} to the buildmaster and start it
1698Use @code{buildbot start <basedir>} to start the server 1729Use @code{buildbot start <basedir>} to start the server
1699@item Setup the buildslaves 1730@item Setup the buildslaves
1700@end itemize 1731@end itemize
@@ -1713,14 +1744,17 @@ and then add the following to your ~/.ssh/config file:
1713 1744
1714@code{Host 127.0.0.1@ IdentityFile ~/.ssh/id_localhost} 1745@code{Host 127.0.0.1@ IdentityFile ~/.ssh/id_localhost}
1715 1746
1716now make sure your hostsfile looks like@ 1747now make sure your hostsfile looks like
1717 1748
1749@example
1718[USERNAME]@@127.0.0.1:22@ 1750[USERNAME]@@127.0.0.1:22@
1719[USERNAME]@@127.0.0.1:22 1751[USERNAME]@@127.0.0.1:22
1752@end example
1720 1753
1721You can test your setup by running `ssh 127.0.0.1` in a terminal and then 1754You can test your setup by running @code{ssh 127.0.0.1} in a
1722in the opened session run it again. If you were not asked for a password 1755terminal and then in the opened session run it again.
1723on either login, then you should be good to go. 1756If you were not asked for a password on either login,
1757then you should be good to go.
1724 1758
1725@c *********************************************************************** 1759@c ***********************************************************************
1726@node TESTBED Caveats 1760@node TESTBED Caveats
@@ -1729,7 +1763,6 @@ on either login, then you should be good to go.
1729This section documents a few caveats when using the GNUnet testbed 1763This section documents a few caveats when using the GNUnet testbed
1730subsystem. 1764subsystem.
1731 1765
1732
1733@c *********************************************************************** 1766@c ***********************************************************************
1734@menu 1767@menu
1735* CORE must be started:: 1768* CORE must be started::
@@ -1767,7 +1800,7 @@ configure TESTBED to tolerate a certain number of connection failures
1767for dense overlay topologies, especially if you try to create cliques 1800for dense overlay topologies, especially if you try to create cliques
1768with more than 20 peers. 1801with more than 20 peers.
1769 1802
1770@c *********************************************************************** 1803@cindex libgnunetutil
1771@node libgnunetutil 1804@node libgnunetutil
1772@section libgnunetutil 1805@section libgnunetutil
1773 1806
@@ -1820,7 +1853,8 @@ way to make porting of GNUnet easier.
1820* The CONTAINER_MDLL API:: 1853* The CONTAINER_MDLL API::
1821@end menu 1854@end menu
1822 1855
1823@c *********************************************************************** 1856@cindex Logging
1857@cindex log levels
1824@node Logging 1858@node Logging
1825@subsection Logging 1859@subsection Logging
1826 1860
@@ -2062,7 +2096,7 @@ Another limitation, on Windows, GNUNET_FORCE_LOGFILE @strong{MUST} be set
2062in order to GNUNET_FORCE_LOG to work. 2096in order to GNUNET_FORCE_LOG to work.
2063 2097
2064 2098
2065@c *********************************************************************** 2099@cindex Log files
2066@node Log files 2100@node Log files
2067@subsubsection Log files 2101@subsubsection Log files
2068 2102
@@ -2183,7 +2217,8 @@ Which will behave almost like enabling DEBUG in that subsytem before the
2183change. Of course you can adapt it to your particular needs, this is only 2217change. Of course you can adapt it to your particular needs, this is only
2184a quick example. 2218a quick example.
2185 2219
2186@c *********************************************************************** 2220@cindex Interprocess communication API
2221@cindex ICP
2187@node Interprocess communication API (IPC) 2222@node Interprocess communication API (IPC)
2188@subsection Interprocess communication API (IPC) 2223@subsection Interprocess communication API (IPC)
2189 2224
@@ -2525,8 +2560,7 @@ order.
2525byte order. 2560byte order.
2526@end table 2561@end table
2527 2562
2528@c *********************************************************************** 2563@cindex Cryptography API
2529
2530@node Cryptography API 2564@node Cryptography API
2531@subsection Cryptography API 2565@subsection Cryptography API
2532@c %**end of header 2566@c %**end of header
@@ -2563,7 +2597,7 @@ be used by most applications; most importantly,
2563GNUNET_CRYPTO_rsa_key_create_from_hash does not create an RSA-key that 2597GNUNET_CRYPTO_rsa_key_create_from_hash does not create an RSA-key that
2564should be considered secure for traditional applications of RSA. 2598should be considered secure for traditional applications of RSA.
2565 2599
2566@c *********************************************************************** 2600@cindex Message Queue API
2567@node Message Queue API 2601@node Message Queue API
2568@subsection Message Queue API 2602@subsection Message Queue API
2569@c %**end of header 2603@c %**end of header
@@ -2643,8 +2677,10 @@ An envelope containing an instance of the NumberMessage can be
2643constructed like this: 2677constructed like this:
2644 2678
2645@example 2679@example
2646struct GNUNET_MQ_Envelope *ev; struct NumberMessage *msg; ev = 2680struct GNUNET_MQ_Envelope *ev;
2647GNUNET_MQ_msg (msg, GNUNET_MESSAGE_TYPE_EXAMPLE_1); msg->number = htonl (42); 2681struct NumberMessage *msg;
2682ev = GNUNET_MQ_msg (msg, GNUNET_MESSAGE_TYPE_EXAMPLE_1);
2683msg->number = htonl (42);
2648@end example 2684@end example
2649 2685
2650In the above code, @code{GNUNET_MQ_msg} is a macro. The return value is 2686In the above code, @code{GNUNET_MQ_msg} is a macro. The return value is
@@ -2703,7 +2739,7 @@ callback. When canceling an envelope, it is not necessary@ to call
2703@strong{ Implementing Queues }@ 2739@strong{ Implementing Queues }@
2704@code{TODO} 2740@code{TODO}
2705 2741
2706@c *********************************************************************** 2742@cindex Service API
2707@node Service API 2743@node Service API
2708@subsection Service API 2744@subsection Service API
2709@c %**end of header 2745@c %**end of header
@@ -2933,7 +2969,8 @@ previously audited and modified to take advantage of the new capability.
2933In particular, memory consumption of the file-sharing service is expected 2969In particular, memory consumption of the file-sharing service is expected
2934to drop by 20-30% due to this change. 2970to drop by 20-30% due to this change.
2935 2971
2936@c *********************************************************************** 2972
2973@cindex CONTAINER_MDLL API
2937@node The CONTAINER_MDLL API 2974@node The CONTAINER_MDLL API
2938@subsection The CONTAINER_MDLL API 2975@subsection The CONTAINER_MDLL API
2939@c %**end of header 2976@c %**end of header
@@ -2998,7 +3035,8 @@ at tail, after a given element) and removing elements from the list.
2998Iterating over the list should be done by directly accessing the 3035Iterating over the list should be done by directly accessing the
2999"next_XX" and/or "prev_XX" members. 3036"next_XX" and/or "prev_XX" members.
3000 3037
3001@c *********************************************************************** 3038@cindex Automatic Restart Manager
3039@cindex ARM
3002@node The Automatic Restart Manager (ARM) 3040@node The Automatic Restart Manager (ARM)
3003@section The Automatic Restart Manager (ARM) 3041@section The Automatic Restart Manager (ARM)
3004@c %**end of header 3042@c %**end of header
@@ -3015,7 +3053,7 @@ about how ARM works and how to interact with it.
3015@menu 3053@menu
3016* Basic functionality:: 3054* Basic functionality::
3017* Key configuration options:: 3055* Key configuration options::
3018* Availability2:: 3056* ARM - Availability::
3019* Reliability:: 3057* Reliability::
3020@end menu 3058@end menu
3021 3059
@@ -3109,8 +3147,8 @@ services that are going to run.
3109@end table 3147@end table
3110 3148
3111@c *********************************************************************** 3149@c ***********************************************************************
3112@node Availability2 3150@node ARM - Availability
3113@subsection Availability2 3151@subsection ARM - Availability
3114@c %**end of header 3152@c %**end of header
3115 3153
3116As mentioned before, one of the features provided by ARM is starting 3154As mentioned before, one of the features provided by ARM is starting
@@ -3195,7 +3233,7 @@ backoff(S) will remain half an hour, hence ARM won't be busy for a lot of
3195time trying to restart a problematic service. 3233time trying to restart a problematic service.
3196@end itemize 3234@end itemize
3197 3235
3198@c *********************************************************************** 3236@cindex TRANSPORT Subsystem
3199@node GNUnet's TRANSPORT Subsystem 3237@node GNUnet's TRANSPORT Subsystem
3200@section GNUnet's TRANSPORT Subsystem 3238@section GNUnet's TRANSPORT Subsystem
3201@c %**end of header 3239@c %**end of header
@@ -3313,6 +3351,7 @@ The PONG message is protected with a nonce/challenge against replay
3313attacks and uses an expiration time for the signature (but those are 3351attacks and uses an expiration time for the signature (but those are
3314almost implementation details). 3352almost implementation details).
3315 3353
3354@cindex NAT library
3316@node NAT library 3355@node NAT library
3317@section NAT library 3356@section NAT library
3318@c %**end of header 3357@c %**end of header
@@ -3422,6 +3461,7 @@ the message in a new DV message for Carol. The DV transport at Carol
3422receives this message, unwraps the original message, and delivers it to 3461receives this message, unwraps the original message, and delivers it to
3423Carol as though it came directly from Alice. 3462Carol as though it came directly from Alice.
3424 3463
3464@cindex SMTP plugin
3425@node SMTP plugin 3465@node SMTP plugin
3426@section SMTP plugin 3466@section SMTP plugin
3427@c %**end of header 3467@c %**end of header
@@ -3631,6 +3671,7 @@ MTU of 12,000 octets improves the throughput to 13 kbps as figure
3631smtp-MTUs shows. Our research paper) has some more details on the 3671smtp-MTUs shows. Our research paper) has some more details on the
3632benchmarking results. 3672benchmarking results.
3633 3673
3674@cindex Bluetooth plugin
3634@node Bluetooth plugin 3675@node Bluetooth plugin
3635@section Bluetooth plugin 3676@section Bluetooth plugin
3636@c %**end of header 3677@c %**end of header
@@ -3648,8 +3689,6 @@ ask on the IRC channel.
3648@item How can I test it? 3689@item How can I test it?
3649@end itemize 3690@end itemize
3650 3691
3651
3652
3653@menu 3692@menu
3654* What do I need to use the Bluetooth plugin transport?:: 3693* What do I need to use the Bluetooth plugin transport?::
3655* How does it work2?:: 3694* How does it work2?::
@@ -4366,6 +4405,7 @@ tell which @code{SendMessageRequest} the CORE service's
4366"unique" request ID (based on a counter incremented by the client 4405"unique" request ID (based on a counter incremented by the client
4367for each request). 4406for each request).
4368 4407
4408@cindex CORE Peer-to-Peer Protocol
4369@node The CORE Peer-to-Peer Protocol 4409@node The CORE Peer-to-Peer Protocol
4370@subsection The CORE Peer-to-Peer Protocol 4410@subsection The CORE Peer-to-Peer Protocol
4371@c %**end of header 4411@c %**end of header
@@ -4827,7 +4867,7 @@ is no longer called with new estimates.
4827 4867
4828@menu 4868@menu
4829* Results:: 4869* Results::
4830* Examples2:: 4870* libgnunetnse - Examples::
4831@end menu 4871@end menu
4832 4872
4833@node Results 4873@node Results
@@ -4868,8 +4908,8 @@ deviation value, not only the average (in particular, if the standard
4868veriation is very high, the average maybe meaningless: the network size is 4908veriation is very high, the average maybe meaningless: the network size is
4869changing rapidly). 4909changing rapidly).
4870 4910
4871@node Examples2 4911@node libgnunetnse - Examples
4872@subsubsection Examples2 4912@subsubsection libgnunetnse -Examples
4873 4913
4874@c %**end of header 4914@c %**end of header
4875 4915
@@ -4915,6 +4955,7 @@ deviation for the respective round.
4915When the @code{GNUNET_NSE_disconnect} API call is executed, the client 4955When the @code{GNUNET_NSE_disconnect} API call is executed, the client
4916simply disconnects from the service, with no message involved. 4956simply disconnects from the service, with no message involved.
4917 4957
4958@cindex NSE Peer-to-Peer Protocol
4918@node The NSE Peer-to-Peer Protocol 4959@node The NSE Peer-to-Peer Protocol
4919@subsection The NSE Peer-to-Peer Protocol 4960@subsection The NSE Peer-to-Peer Protocol
4920 4961
@@ -5044,7 +5085,7 @@ service.
5044 5085
5045@menu 5086@menu
5046* Features:: 5087* Features::
5047* Limitations2:: 5088* HOSTLIST - Limitations::
5048@end menu 5089@end menu
5049 5090
5050@node Features 5091@node Features
@@ -5065,8 +5106,8 @@ via gossip
5065peers 5106peers
5066@end itemize 5107@end itemize
5067 5108
5068@node Limitations2 5109@node HOSTLIST - Limitations
5069@subsubsection Limitations2 5110@subsubsection HOSTLIST - Limitations
5070 5111
5071@c %**end of header 5112@c %**end of header
5072 5113
@@ -5125,6 +5166,7 @@ contacts the HOSTLIST servers specified in the configuration, downloads
5125the (unvalidated) list of HELLO messages and forwards these information 5166the (unvalidated) list of HELLO messages and forwards these information
5126to the TRANSPORT server to validate the addresses. 5167to the TRANSPORT server to validate the addresses.
5127 5168
5169@cindex HOSTLIST daemon
5128@node The HOSTLIST daemon 5170@node The HOSTLIST daemon
5129@subsection The HOSTLIST daemon 5171@subsection The HOSTLIST daemon
5130 5172
@@ -5154,6 +5196,7 @@ can notify them about CORE events.
5154To clean up on shutdown, the daemon has a cleaning task, shutting down all 5196To clean up on shutdown, the daemon has a cleaning task, shutting down all
5155subsystems and disconnecting from CORE. 5197subsystems and disconnecting from CORE.
5156 5198
5199@cindex HOSTLIST server
5157@node The HOSTLIST server 5200@node The HOSTLIST server
5158@subsection The HOSTLIST server 5201@subsection The HOSTLIST server
5159 5202
@@ -5207,6 +5250,7 @@ When a new peer connects and has hostlist learning enabled, the server
5207sends a @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this 5250sends a @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this
5208peer using the CORE service. 5251peer using the CORE service.
5209 5252
5253@cindex HOSTLIST client
5210@node The HOSTLIST client 5254@node The HOSTLIST client
5211@subsection The HOSTLIST client 5255@subsection The HOSTLIST client
5212 5256
@@ -5655,8 +5699,8 @@ To not duplicate this functionality we plan to provide a PEERSTORE
5655service providing this functionality. 5699service providing this functionality.
5656 5700
5657@menu 5701@menu
5658* Features2:: 5702* PEERINFO - Features::
5659* Limitations3:: 5703* PEERINFO - Limitations::
5660* DeveloperPeer Information:: 5704* DeveloperPeer Information::
5661* Startup:: 5705* Startup::
5662* Managing Information:: 5706* Managing Information::
@@ -5665,8 +5709,8 @@ service providing this functionality.
5665* libgnunetpeerinfo:: 5709* libgnunetpeerinfo::
5666@end menu 5710@end menu
5667 5711
5668@node Features2 5712@node PEERINFO - Features
5669@subsection Features2 5713@subsection PEERINFO - Features
5670 5714
5671@c %**end of header 5715@c %**end of header
5672 5716
@@ -5677,8 +5721,8 @@ service providing this functionality.
5677@item Differentiation between public and friend-only HELLO 5721@item Differentiation between public and friend-only HELLO
5678@end itemize 5722@end itemize
5679 5723
5680@node Limitations3 5724@node PEERINFO - Limitations
5681@subsection Limitations3 5725@subsection PEERINFO - Limitations
5682 5726
5683 5727
5684@itemize @bullet 5728@itemize @bullet
@@ -5690,42 +5734,48 @@ service providing this functionality.
5690 5734
5691@c %**end of header 5735@c %**end of header
5692 5736
5693The PEERINFO subsystem stores these information in the form of HELLO messages 5737The PEERINFO subsystem stores these information in the form of HELLO
5694you can think of as business cards. These HELLO messages contain the public key 5738messages you can think of as business cards.
5695of a peer and the addresses a peer can be reached under. The addresses include 5739These HELLO messages contain the public key of a peer and the addresses
5696an expiration date describing how long they are valid. This information is 5740a peer can be reached under.
5697updated regularly by the TRANSPORT service by revalidating the address. If an 5741The addresses include an expiration date describing how long they are
5698address is expired and not renewed, it can be removed from the HELLO message. 5742valid. This information is updated regularly by the TRANSPORT service by
5699 5743revalidating the address.
5700Some peer do not want to have their HELLO messages distributed to other peers , 5744If an address is expired and not renewed, it can be removed from the
5701especially when GNUnet's friend-to-friend modus is enabled. To prevent this 5745HELLO message.
5702undesired distribution. PEERINFO distinguishes between @emph{public} and 5746
5703@emph{friend-only} HELLO messages. Public HELLO messages can be freely 5747Some peer do not want to have their HELLO messages distributed to other
5704distributed to other (possibly unknown) peers (for example using the hostlist, 5748peers, especially when GNUnet's friend-to-friend modus is enabled.
5705gossiping, broadcasting), whereas friend-only HELLO messages may not be 5749To prevent this undesired distribution. PEERINFO distinguishes between
5706distributed to other peers. Friend-only HELLO messages have an additional flag 5750@emph{public} and @emph{friend-only} HELLO messages.
5707@code{friend_only} set internally. For public HELLO message this flag is not 5751Public HELLO messages can be freely distributed to other (possibly
5708set. PEERINFO does and cannot not check if a client is allowed to obtain a 5752unknown) peers (for example using the hostlist, gossiping, broadcasting),
5753whereas friend-only HELLO messages may not be distributed to other peers.
5754Friend-only HELLO messages have an additional flag @code{friend_only} set
5755internally. For public HELLO message this flag is not set.
5756PEERINFO does and cannot not check if a client is allowed to obtain a
5709specific HELLO type. 5757specific HELLO type.
5710 5758
5711The HELLO messages can be managed using the GNUnet HELLO library. Other GNUnet 5759The HELLO messages can be managed using the GNUnet HELLO library.
5712systems can obtain these information from PEERINFO and use it for their 5760Other GNUnet systems can obtain these information from PEERINFO and use
5713purposes. Clients are for example the HOSTLIST component providing these 5761it for their purposes.
5714information to other peers in form of a hostlist or the TRANSPORT subsystem 5762Clients are for example the HOSTLIST component providing these
5715using these information to maintain connections to other peers. 5763information to other peers in form of a hostlist or the TRANSPORT
5764subsystem using these information to maintain connections to other peers.
5716 5765
5717@node Startup 5766@node Startup
5718@subsection Startup 5767@subsection Startup
5719 5768
5720@c %**end of header 5769@c %**end of header
5721 5770
5722During startup the PEERINFO services loads persistent HELLOs from disk. First 5771During startup the PEERINFO services loads persistent HELLOs from disk.
5723PEERINFO parses the directory configured in the HOSTS value of the 5772First PEERINFO parses the directory configured in the HOSTS value of the
5724@code{PEERINFO} configuration section to store PEERINFO information.@ For all 5773@code{PEERINFO} configuration section to store PEERINFO information.
5725files found in this directory valid HELLO messages are extracted. In addition 5774For all files found in this directory valid HELLO messages are extracted.
5726it loads HELLO messages shipped with the GNUnet distribution. These HELLOs are 5775In addition it loads HELLO messages shipped with the GNUnet distribution.
5727used to simplify network bootstrapping by providing valid peer information with 5776These HELLOs are used to simplify network bootstrapping by providing
5728the distribution. The use of these HELLOs can be prevented by setting the 5777valid peer information with the distribution.
5778The use of these HELLOs can be prevented by setting the
5729@code{USE_INCLUDED_HELLOS} in the @code{PEERINFO} configuration section to 5779@code{USE_INCLUDED_HELLOS} in the @code{PEERINFO} configuration section to
5730@code{NO}. Files containing invalid information are removed. 5780@code{NO}. Files containing invalid information are removed.
5731 5781
@@ -5734,66 +5784,71 @@ the distribution. The use of these HELLOs can be prevented by setting the
5734 5784
5735@c %**end of header 5785@c %**end of header
5736 5786
5737The PEERINFO services stores information about known PEERS and a single HELLO 5787The PEERINFO services stores information about known PEERS and a single
5738message for every peer. A peer does not need to have a HELLO if no information 5788HELLO message for every peer.
5739are available. HELLO information from different sources, for example a HELLO 5789A peer does not need to have a HELLO if no information are available.
5740obtained from a remote HOSTLIST and a second HELLO stored on disk, are combined 5790HELLO information from different sources, for example a HELLO obtained
5791from a remote HOSTLIST and a second HELLO stored on disk, are combined
5741and merged into one single HELLO message per peer which will be given to 5792and merged into one single HELLO message per peer which will be given to
5742clients. During this merge process the HELLO is immediately written to disk to 5793clients. During this merge process the HELLO is immediately written to
5743ensure persistence. 5794disk to ensure persistence.
5744 5795
5745PEERINFO in addition periodically scans the directory where information are 5796PEERINFO in addition periodically scans the directory where information
5746stored for empty HELLO messages with expired TRANSPORT addresses.@ This 5797are stored for empty HELLO messages with expired TRANSPORT addresses.
5747periodic task scans all files in the directory and recreates the HELLO messages 5798This periodic task scans all files in the directory and recreates the
5748it finds. Expired TRANSPORT addresses are removed from the HELLO and if the 5799HELLO messages it finds.
5749HELLO does not contain any valid addresses, it is discarded and removed from 5800Expired TRANSPORT addresses are removed from the HELLO and if the
5750disk. 5801HELLO does not contain any valid addresses, it is discarded and removed
5802from the disk.
5751 5803
5752@node Obtaining Information 5804@node Obtaining Information
5753@subsection Obtaining Information 5805@subsection Obtaining Information
5754 5806
5755@c %**end of header 5807@c %**end of header
5756 5808
5757When a client requests information from PEERINFO, PEERINFO performs a lookup 5809When a client requests information from PEERINFO, PEERINFO performs a
5758for the respective peer or all peers if desired and transmits this information 5810lookup for the respective peer or all peers if desired and transmits this
5759to the client. The client can specify if friend-only HELLOs have to be included 5811information to the client.
5760or not and PEERINFO filters the respective HELLO messages before transmitting 5812The client can specify if friend-only HELLOs have to be included or not
5813and PEERINFO filters the respective HELLO messages before transmitting
5761information. 5814information.
5762 5815
5763To notify clients about changes to PEERINFO information, PEERINFO maintains a 5816To notify clients about changes to PEERINFO information, PEERINFO
5764list of clients interested in this notifications. Such a notification occurs if 5817maintains a list of clients interested in this notifications.
5765a HELLO for a peer was updated (due to a merge for example) or a new peer was 5818Such a notification occurs if a HELLO for a peer was updated (due to a
5766added. 5819merge for example) or a new peer was added.
5767 5820
5768@node The PEERINFO Client-Service Protocol 5821@node The PEERINFO Client-Service Protocol
5769@subsection The PEERINFO Client-Service Protocol 5822@subsection The PEERINFO Client-Service Protocol
5770 5823
5771@c %**end of header 5824@c %**end of header
5772 5825
5773To connect and disconnect to and from the PEERINFO Service PEERINFO utilizes 5826To connect and disconnect to and from the PEERINFO Service PEERINFO
5774the util client/server infrastructure, so no special messages types are used 5827utilizes the util client/server infrastructure, so no special messages
5775here. 5828types are used here.
5776 5829
5777To add information for a peer, the plain HELLO message is transmitted to the 5830To add information for a peer, the plain HELLO message is transmitted to
5778service without any wrapping. Alle information required are stored within the 5831the service without any wrapping. All pieces of information required are
5779HELLO message. The PEERINFO service provides a message handler accepting and 5832stored within the HELLO message.
5780processing these HELLO messages. 5833The PEERINFO service provides a message handler accepting and processing
5834these HELLO messages.
5781 5835
5782When obtaining PEERINFO information using the iterate functionality specific 5836When obtaining PEERINFO information using the iterate functionality
5783messages are used. To obtain information for all peers, a @code{struct 5837specific messages are used. To obtain information for all peers, a
5784ListAllPeersMessage} with message type 5838@code{struct ListAllPeersMessage} with message type
5785@code{GNUNET_MESSAGE_TYPE_PEERINFO_GET_ALL} and a flag include_friend_only to 5839@code{GNUNET_MESSAGE_TYPE_PEERINFO_GET_ALL} and a flag
5786indicate if friend-only HELLO messages should be included are transmitted. If 5840include_friend_only to indicate if friend-only HELLO messages should be
5787information for a specific peer is required a @code{struct ListAllPeersMessage} 5841included are transmitted. If information for a specific peer is required
5788with @code{GNUNET_MESSAGE_TYPE_PEERINFO_GET} containing the peer identity is 5842a @code{struct ListAllPeersMessage} with
5843@code{GNUNET_MESSAGE_TYPE_PEERINFO_GET} containing the peer identity is
5789used. 5844used.
5790 5845
5791For both variants the PEERINFO service replies for each HELLO message he wants 5846For both variants the PEERINFO service replies for each HELLO message it
5792to transmit with a @code{struct ListAllPeersMessage} with type 5847wants to transmit with a @code{struct ListAllPeersMessage} with type
5793@code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO} containing the plain HELLO. The final 5848@code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO} containing the plain HELLO.
5794message is @code{struct GNUNET_MessageHeader} with type 5849The final message is @code{struct GNUNET_MessageHeader} with type
5795@code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO}. If the client receives this message, 5850@code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO}. If the client receives this
5796he can proceed with the next request if any is pending 5851message, it can proceed with the next request if any is pending.
5797 5852
5798@node libgnunetpeerinfo 5853@node libgnunetpeerinfo
5799@subsection libgnunetpeerinfo 5854@subsection libgnunetpeerinfo
@@ -5819,9 +5874,10 @@ The PEERINFO API consists mainly of three different functionalities:
5819 5874
5820@c %**end of header 5875@c %**end of header
5821 5876
5822To connect to the PEERINFO service the function @code{GNUNET_PEERINFO_connect} 5877To connect to the PEERINFO service the function
5823is used, taking a configuration handle as an argument, and to disconnect from 5878@code{GNUNET_PEERINFO_connect} is used, taking a configuration handle as
5824PEERINFO the function @code{GNUNET_PEERINFO_disconnect}, taking the PEERINFO 5879an argument, and to disconnect from PEERINFO the function
5880@code{GNUNET_PEERINFO_disconnect}, taking the PEERINFO
5825handle returned from the connect function has to be called. 5881handle returned from the connect function has to be called.
5826 5882
5827@node Adding Information to the PEERINFO Service 5883@node Adding Information to the PEERINFO Service
@@ -5831,44 +5887,48 @@ handle returned from the connect function has to be called.
5831 5887
5832@code{GNUNET_PEERINFO_add_peer} adds a new peer to the PEERINFO subsystem 5888@code{GNUNET_PEERINFO_add_peer} adds a new peer to the PEERINFO subsystem
5833storage. This function takes the PEERINFO handle as an argument, the HELLO 5889storage. This function takes the PEERINFO handle as an argument, the HELLO
5834message to store and a continuation with a closure to be called with the result 5890message to store and a continuation with a closure to be called with the
5835of the operation. The @code{GNUNET_PEERINFO_add_peer} returns a handle to this 5891result of the operation.
5836operation allowing to cancel the operation with the respective cancel function 5892The @code{GNUNET_PEERINFO_add_peer} returns a handle to this operation
5837@code{GNUNET_PEERINFO_add_peer_cancel}. To retrieve information from PEERINFO 5893allowing to cancel the operation with the respective cancel function
5838you can iterate over all information stored with PEERINFO or you can tell 5894@code{GNUNET_PEERINFO_add_peer_cancel}. To retrieve information from
5839PEERINFO to notify if new peer information are available. 5895PEERINFO you can iterate over all information stored with PEERINFO or you
5896can tell PEERINFO to notify if new peer information are available.
5840 5897
5841@node Obtaining Information from the PEERINFO Service 5898@node Obtaining Information from the PEERINFO Service
5842@subsubsection Obtaining Information from the PEERINFO Service 5899@subsubsection Obtaining Information from the PEERINFO Service
5843 5900
5844@c %**end of header 5901@c %**end of header
5845 5902
5846To iterate over information in PEERINFO you use @code{GNUNET_PEERINFO_iterate}. 5903To iterate over information in PEERINFO you use
5847This function expects the PEERINFO handle, a flag if HELLO messages intended 5904@code{GNUNET_PEERINFO_iterate}.
5848for friend only mode should be included, a timeout how long the operation 5905This function expects the PEERINFO handle, a flag if HELLO messages
5849should take and a callback with a callback closure to be called for the 5906intended for friend only mode should be included, a timeout how long the
5850results. If you want to obtain information for a specific peer, you can specify 5907operation should take and a callback with a callback closure to be called
5908for the results.
5909If you want to obtain information for a specific peer, you can specify
5851the peer identity, if this identity is NULL, information for all peers are 5910the peer identity, if this identity is NULL, information for all peers are
5852returned. The function returns a handle to allow to cancel the operation using 5911returned. The function returns a handle to allow to cancel the operation
5853@code{GNUNET_PEERINFO_iterate_cancel}. 5912using @code{GNUNET_PEERINFO_iterate_cancel}.
5854 5913
5855To get notified when peer information changes, you can use 5914To get notified when peer information changes, you can use
5856@code{GNUNET_PEERINFO_notify}. This function expects a configuration handle and 5915@code{GNUNET_PEERINFO_notify}.
5857a flag if friend-only HELLO messages should be included. The PEERINFO service 5916This function expects a configuration handle and a flag if friend-only
5858will notify you about every change and the callback function will be called to 5917HELLO messages should be included. The PEERINFO service will notify you
5859notify you about changes. The function returns a handle to cancel notifications 5918about every change and the callback function will be called to notify you
5919about changes. The function returns a handle to cancel notifications
5860with @code{GNUNET_PEERINFO_notify_cancel}. 5920with @code{GNUNET_PEERINFO_notify_cancel}.
5861 5921
5862 5922@cindex PEERSTORE subsystem
5863@node GNUnet's PEERSTORE subsystem 5923@node GNUnet's PEERSTORE subsystem
5864@section GNUnet's PEERSTORE subsystem 5924@section GNUnet's PEERSTORE subsystem
5865 5925
5866@c %**end of header 5926@c %**end of header
5867 5927
5868GNUnet's PEERSTORE subsystem offers persistent per-peer storage for other 5928GNUnet's PEERSTORE subsystem offers persistent per-peer storage for other
5869GNUnet subsystems. GNUnet subsystems can use PEERSTORE to persistently store 5929GNUnet subsystems. GNUnet subsystems can use PEERSTORE to persistently
5870and retrieve arbitrary data. Each data record stored with PEERSTORE contains 5930store and retrieve arbitrary data.
5871the following fields: 5931Each data record stored with PEERSTORE contains the following fields:
5872 5932
5873@itemize @bullet 5933@itemize @bullet
5874@item subsystem: Name of the subsystem responsible for the record. 5934@item subsystem: Name of the subsystem responsible for the record.
@@ -5890,10 +5950,11 @@ the following fields:
5890@c %**end of header 5950@c %**end of header
5891 5951
5892Subsystems can store any type of value under a (subsystem, peerid, key) 5952Subsystems can store any type of value under a (subsystem, peerid, key)
5893combination. A "replace" flag set during store operations forces the PEERSTORE 5953combination. A "replace" flag set during store operations forces the
5894to replace any old values stored under the same (subsystem, peerid, key) 5954PEERSTORE to replace any old values stored under the same
5895combination with the new value. Additionally, an expiry date is set after which 5955(subsystem, peerid, key) combination with the new value.
5896the record is *possibly* deleted by PEERSTORE. 5956Additionally, an expiry date is set after which the record is *possibly*
5957deleted by PEERSTORE.
5897 5958
5898Subsystems can iterate over all values stored under any of the following 5959Subsystems can iterate over all values stored under any of the following
5899combination of fields: 5960combination of fields:
@@ -5905,9 +5966,9 @@ combination of fields:
5905@item (subsystem, peerid, key) 5966@item (subsystem, peerid, key)
5906@end itemize 5967@end itemize
5907 5968
5908Subsystems can also request to be notified about any new values stored under a 5969Subsystems can also request to be notified about any new values stored
5909(subsystem, peerid, key) combination by sending a "watch" request to 5970under a (subsystem, peerid, key) combination by sending a "watch"
5910PEERSTORE. 5971request to PEERSTORE.
5911 5972
5912@node Architecture 5973@node Architecture
5913@subsection Architecture 5974@subsection Architecture
@@ -5920,10 +5981,11 @@ PEERSTORE implements the following components:
5920@item PEERSTORE service: Handles store, iterate and watch operations. 5981@item PEERSTORE service: Handles store, iterate and watch operations.
5921@item PEERSTORE API: API to be used by other subsystems to communicate and 5982@item PEERSTORE API: API to be used by other subsystems to communicate and
5922issue commands to the PEERSTORE service. 5983issue commands to the PEERSTORE service.
5923@item PEERSTORE plugins: Handles the persistent storage. At the moment, only an 5984@item PEERSTORE plugins: Handles the persistent storage. At the moment,
5924"sqlite" plugin is implemented. 5985only an "sqlite" plugin is implemented.
5925@end itemize 5986@end itemize
5926 5987
5988@cindex libgnunetpeerstore
5927@node libgnunetpeerstore 5989@node libgnunetpeerstore
5928@subsection libgnunetpeerstore 5990@subsection libgnunetpeerstore
5929 5991
@@ -5932,49 +5994,57 @@ issue commands to the PEERSTORE service.
5932libgnunetpeerstore is the library containing the PEERSTORE API. Subsystems 5994libgnunetpeerstore is the library containing the PEERSTORE API. Subsystems
5933wishing to communicate with the PEERSTORE service use this API to open a 5995wishing to communicate with the PEERSTORE service use this API to open a
5934connection to PEERSTORE. This is done by calling 5996connection to PEERSTORE. This is done by calling
5935@code{GNUNET_PEERSTORE_connect} which returns a handle to the newly created 5997@code{GNUNET_PEERSTORE_connect} which returns a handle to the newly
5936connection. This handle has to be used with any further calls to the API. 5998created connection.
5937 5999This handle has to be used with any further calls to the API.
5938To store a new record, the function @code{GNUNET_PEERSTORE_store} is to be used 6000
5939which requires the record fields and a continuation function that will be 6001To store a new record, the function @code{GNUNET_PEERSTORE_store} is to
5940called by the API after the STORE request is sent to the PEERSTORE service. 6002be used which requires the record fields and a continuation function that
5941Note that calling the continuation function does not mean that the record is 6003will be called by the API after the STORE request is sent to the
5942successfully stored, only that the STORE request has been successfully sent to 6004PEERSTORE service.
5943the PEERSTORE service. @code{GNUNET_PEERSTORE_store_cancel} can be called to 6005Note that calling the continuation function does not mean that the record
5944cancel the STORE request only before the continuation function has been called. 6006is successfully stored, only that the STORE request has been successfully
5945 6007sent to the PEERSTORE service.
5946To iterate over stored records, the function @code{GNUNET_PEERSTORE_iterate} is 6008@code{GNUNET_PEERSTORE_store_cancel} can be called to cancel the STORE
6009request only before the continuation function has been called.
6010
6011To iterate over stored records, the function
6012@code{GNUNET_PEERSTORE_iterate} is
5947to be used. @emph{peerid} and @emph{key} can be set to NULL. An iterator 6013to be used. @emph{peerid} and @emph{key} can be set to NULL. An iterator
5948callback function will be called with each matching record found and a NULL 6014callback function will be called with each matching record found and a
5949record at the end to signal the end of result set. 6015NULL record at the end to signal the end of result set.
5950@code{GNUNET_PEERSTORE_iterate_cancel} can be used to cancel the ITERATE 6016@code{GNUNET_PEERSTORE_iterate_cancel} can be used to cancel the ITERATE
5951request before the iterator callback is called with a NULL record. 6017request before the iterator callback is called with a NULL record.
5952 6018
5953To be notified with new values stored under a (subsystem, peerid, key) 6019To be notified with new values stored under a (subsystem, peerid, key)
5954combination, the function @code{GNUNET_PEERSTORE_watch} is to be used. This 6020combination, the function @code{GNUNET_PEERSTORE_watch} is to be used.
5955will register the watcher with the PEERSTORE service, any new records matching 6021This will register the watcher with the PEERSTORE service, any new
5956the given combination will trigger the callback function passed to 6022records matching the given combination will trigger the callback
5957@code{GNUNET_PEERSTORE_watch}. This continues until 6023function passed to @code{GNUNET_PEERSTORE_watch}. This continues until
5958@code{GNUNET_PEERSTORE_watch_cancel} is called or the connection to the service 6024@code{GNUNET_PEERSTORE_watch_cancel} is called or the connection to the
5959is destroyed. 6025service is destroyed.
5960 6026
5961After the connection is no longer needed, the function 6027After the connection is no longer needed, the function
5962@code{GNUNET_PEERSTORE_disconnect} can be called to disconnect from the 6028@code{GNUNET_PEERSTORE_disconnect} can be called to disconnect from the
5963PEERSTORE service. Any pending ITERATE or WATCH requests will be destroyed. If 6029PEERSTORE service.
5964the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will delay the 6030Any pending ITERATE or WATCH requests will be destroyed.
5965disconnection until all pending STORE requests are sent to the PEERSTORE 6031If the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will
5966service, otherwise, the pending STORE requests will be destroyed as well. 6032delay the disconnection until all pending STORE requests are sent to
5967 6033the PEERSTORE service, otherwise, the pending STORE requests will be
6034destroyed as well.
6035
6036@cindex SET Subsystem
5968@node GNUnet's SET Subsystem 6037@node GNUnet's SET Subsystem
5969@section GNUnet's SET Subsystem 6038@section GNUnet's SET Subsystem
5970 6039
5971@c %**end of header 6040@c %**end of header
5972 6041
5973The SET service implements efficient set operations between two peers over a 6042The SET service implements efficient set operations between two peers
5974mesh tunnel. Currently, set union and set intersection are the only supported 6043over a mesh tunnel.
5975operations. Elements of a set consist of an @emph{element type} and arbitrary 6044Currently, set union and set intersection are the only supported
5976binary @emph{data}. The size of an element's data is limited to around 62 6045operations. Elements of a set consist of an @emph{element type} and
5977KB. 6046arbitrary binary @emph{data}.
6047The size of an element's data is limited to around 62 KB.
5978 6048
5979@menu 6049@menu
5980* Local Sets:: 6050* Local Sets::
@@ -5995,62 +6065,67 @@ KB.
5995Sets created by a local client can be modified and reused for multiple 6065Sets created by a local client can be modified and reused for multiple
5996operations. As each set operation requires potentially expensive special 6066operations. As each set operation requires potentially expensive special
5997auxilliary data to be computed for each element of a set, a set can only 6067auxilliary data to be computed for each element of a set, a set can only
5998participate in one type of set operation (i.e. union or intersection). The type 6068participate in one type of set operation (i.e. union or intersection).
5999of a set is determined upon its creation. If a the elements of a set are needed 6069The type of a set is determined upon its creation.
6000for an operation of a different type, all of the set's element must be copied 6070If a the elements of a set are needed for an operation of a different
6001to a new set of appropriate type. 6071type, all of the set's element must be copied to a new set of appropriate
6072type.
6002 6073
6003@node Set Modifications 6074@node Set Modifications
6004@subsection Set Modifications 6075@subsection Set Modifications
6005 6076
6006@c %**end of header 6077@c %**end of header
6007 6078
6008Even when set operations are active, one can add to and remove elements from a 6079Even when set operations are active, one can add to and remove elements
6009set. However, these changes will only be visible to operations that have been 6080from a set.
6010created after the changes have taken place. That is, every set operation only 6081However, these changes will only be visible to operations that have been
6011sees a snapshot of the set from the time the operation was started. This 6082created after the changes have taken place. That is, every set operation
6012mechanism is @emph{not} implemented by copying the whole set, but by attaching 6083only sees a snapshot of the set from the time the operation was started.
6013@emph{generation information} to each element and operation. 6084This mechanism is @emph{not} implemented by copying the whole set, but by
6085attaching @emph{generation information} to each element and operation.
6014 6086
6015@node Set Operations 6087@node Set Operations
6016@subsection Set Operations 6088@subsection Set Operations
6017 6089
6018@c %**end of header 6090@c %**end of header
6019 6091
6020Set operations can be started in two ways: Either by accepting an operation 6092Set operations can be started in two ways: Either by accepting an
6021request from a remote peer, or by requesting a set operation from a remote 6093operation request from a remote peer, or by requesting a set operation
6022peer. Set operations are uniquely identified by the involved @emph{peers}, an 6094from a remote peer.
6095Set operations are uniquely identified by the involved @emph{peers}, an
6023@emph{application id} and the @emph{operation type}. 6096@emph{application id} and the @emph{operation type}.
6024 6097
6025The client is notified of incoming set operations by @emph{set listeners}. A 6098The client is notified of incoming set operations by @emph{set listeners}.
6026set listener listens for incoming operations of a specific operation type and 6099A set listener listens for incoming operations of a specific operation
6027application id. Once notified of an incoming set request, the client can 6100type and application id.
6028accept the set request (providing a local set for the operation) or reject 6101Once notified of an incoming set request, the client can accept the set
6029it. 6102request (providing a local set for the operation) or reject it.
6030 6103
6031@node Result Elements 6104@node Result Elements
6032@subsection Result Elements 6105@subsection Result Elements
6033 6106
6034@c %**end of header 6107@c %**end of header
6035 6108
6036The SET service has three @emph{result modes} that determine how an operation's 6109The SET service has three @emph{result modes} that determine how an
6037result set is delivered to the client: 6110operation's result set is delivered to the client:
6038 6111
6039@itemize @bullet 6112@itemize @bullet
6040@item @strong{Full Result Set.} All elements of set resulting from the set 6113@item @strong{Full Result Set.} All elements of set resulting from the set
6041operation are returned to the client. 6114operation are returned to the client.
6042@item @strong{Added Elements.} Only elements that result from the operation and 6115@item @strong{Added Elements.} Only elements that result from the
6043are not already in the local peer's set are returned. Note that for some 6116operation and are not already in the local peer's set are returned.
6044operations (like set intersection) this result mode will never return any 6117Note that for some operations (like set intersection) this result mode
6045elements. This can be useful if only the remove peer is actually interested in 6118will never return any elements.
6046the result of the set operation. 6119This can be useful if only the remove peer is actually interested in
6047@item @strong{Removed Elements.} Only elements that are in the local peer's
6048initial set but not in the operation's result set are returned. Note that for
6049some operations (like set union) this result mode will never return any
6050elements. This can be useful if only the remove peer is actually interested in
6051the result of the set operation. 6120the result of the set operation.
6121@item @strong{Removed Elements.} Only elements that are in the local
6122peer's initial set but not in the operation's result set are returned.
6123Note that for some operations (like set union) this result mode will
6124never return any elements. This can be useful if only the remove peer is
6125actually interested in the result of the set operation.
6052@end itemize 6126@end itemize
6053 6127
6128@cindex libgnunetset
6054@node libgnunetset 6129@node libgnunetset
6055@subsection libgnunetset 6130@subsection libgnunetset
6056 6131
@@ -6070,10 +6145,11 @@ the result of the set operation.
6070@c %**end of header 6145@c %**end of header
6071 6146
6072New sets are created with @code{GNUNET_SET_create}. Both the local peer's 6147New sets are created with @code{GNUNET_SET_create}. Both the local peer's
6073configuration (as each set has its own client connection) and the operation 6148configuration (as each set has its own client connection) and the
6074type must be specified. The set exists until either the client calls 6149operation type must be specified.
6075@code{GNUNET_SET_destroy} or the client's connection to the service is 6150The set exists until either the client calls @code{GNUNET_SET_destroy} or
6076disrupted. In the latter case, the client is notified by the return value of 6151the client's connection to the service is disrupted.
6152In the latter case, the client is notified by the return value of
6077functions dealing with sets. This return value must always be checked. 6153functions dealing with sets. This return value must always be checked.
6078 6154
6079Elements are added and removed with @code{GNUNET_SET_add_element} and 6155Elements are added and removed with @code{GNUNET_SET_add_element} and
@@ -6084,12 +6160,13 @@ Elements are added and removed with @code{GNUNET_SET_add_element} and
6084 6160
6085@c %**end of header 6161@c %**end of header
6086 6162
6087Listeners are created with @code{GNUNET_SET_listen}. Each time time a remote 6163Listeners are created with @code{GNUNET_SET_listen}. Each time time a
6088peer suggests a set operation with an application id and operation type 6164remote peer suggests a set operation with an application id and operation
6089matching a listener, the listener's callack is invoked. The client then must 6165type matching a listener, the listener's callback is invoked.
6090synchronously call either @code{GNUNET_SET_accept} or @code{GNUNET_SET_reject}. 6166The client then must synchronously call either @code{GNUNET_SET_accept}
6091Note that the operation will not be started until the client calls 6167or @code{GNUNET_SET_reject}. Note that the operation will not be started
6092@code{GNUNET_SET_commit} (see Section "Supplying a Set"). 6168until the client calls @code{GNUNET_SET_commit}
6169(see Section "Supplying a Set").
6093 6170
6094@node Operations 6171@node Operations
6095@subsubsection Operations 6172@subsubsection Operations
@@ -6097,22 +6174,22 @@ Note that the operation will not be started until the client calls
6097@c %**end of header 6174@c %**end of header
6098 6175
6099Operations to be initiated by the local peer are created with 6176Operations to be initiated by the local peer are created with
6100@code{GNUNET_SET_prepare}. Note that the operation will not be started until 6177@code{GNUNET_SET_prepare}. Note that the operation will not be started
6101the client calls @code{GNUNET_SET_commit} (see Section "Supplying a 6178until the client calls @code{GNUNET_SET_commit}
6102Set"). 6179(see Section "Supplying a Set").
6103 6180
6104@node Supplying a Set 6181@node Supplying a Set
6105@subsubsection Supplying a Set 6182@subsubsection Supplying a Set
6106 6183
6107@c %**end of header 6184@c %**end of header
6108 6185
6109To create symmetry between the two ways of starting a set operation (accepting 6186To create symmetry between the two ways of starting a set operation
6110and nitiating it), the operation handles returned by @code{GNUNET_SET_accept} 6187(accepting and nitiating it), the operation handles returned by
6111and @code{GNUNET_SET_prepare} do not yet have a set to operate on, thus they 6188@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare} do not yet have a
6112can not do any work yet. 6189set to operate on, thus they can not do any work yet.
6113 6190
6114The client must call @code{GNUNET_SET_commit} to specify a set to use for an 6191The client must call @code{GNUNET_SET_commit} to specify a set to use for
6115operation. @code{GNUNET_SET_commit} may only be called once per set 6192an operation. @code{GNUNET_SET_commit} may only be called once per set
6116operation. 6193operation.
6117 6194
6118@node The Result Callback 6195@node The Result Callback
@@ -6121,12 +6198,13 @@ operation.
6121@c %**end of header 6198@c %**end of header
6122 6199
6123Clients must specify both a result mode and a result callback with 6200Clients must specify both a result mode and a result callback with
6124@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result callback 6201@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result
6125with a status indicating either that an element was received, or the operation 6202callback with a status indicating either that an element was received, or
6126failed or succeeded. The interpretation of the received element depends on the 6203the operation failed or succeeded.
6127result mode. The callback needs to know which result mode it is used in, as the 6204The interpretation of the received element depends on the result mode.
6128arguments do not indicate if an element is part of the full result set, or if 6205The callback needs to know which result mode it is used in, as the
6129it is in the difference between the original set and the final set. 6206arguments do not indicate if an element is part of the full result set,
6207or if it is in the difference between the original set and the final set.
6130 6208
6131@node The SET Client-Service Protocol 6209@node The SET Client-Service Protocol
6132@subsection The SET Client-Service Protocol 6210@subsection The SET Client-Service Protocol
@@ -6147,10 +6225,11 @@ it is in the difference between the original set and the final set.
6147 6225
6148@c %**end of header 6226@c %**end of header
6149 6227
6150For each set of a client, there exists a client connection to the service. Sets 6228For each set of a client, there exists a client connection to the service.
6151are created by sending the @code{GNUNET_SERVICE_SET_CREATE} message over a new 6229Sets are created by sending the @code{GNUNET_SERVICE_SET_CREATE} message
6152client connection. Multiple operations for one set are multiplexed over one 6230over a new client connection. Multiple operations for one set are
6153client connection, using a request id supplied by the client. 6231multiplexed over one client connection, using a request id supplied by
6232the client.
6154 6233
6155@node Listeners2 6234@node Listeners2
6156@subsubsection Listeners2 6235@subsubsection Listeners2
@@ -6158,12 +6237,13 @@ client connection, using a request id supplied by the client.
6158@c %**end of header 6237@c %**end of header
6159 6238
6160Each listener also requires a seperate client connection. By sending the 6239Each listener also requires a seperate client connection. By sending the
6161@code{GNUNET_SERVICE_SET_LISTEN} message, the client notifies the service of 6240@code{GNUNET_SERVICE_SET_LISTEN} message, the client notifies the service
6162the application id and operation type it is interested in. A client rejects an 6241of the application id and operation type it is interested in. A client
6163incoming request by sending @code{GNUNET_SERVICE_SET_REJECT} on the listener's 6242rejects an incoming request by sending @code{GNUNET_SERVICE_SET_REJECT}
6164client connection. In contrast, when accepting an incoming request, a a 6243on the listener's client connection.
6165@code{GNUNET_SERVICE_SET_ACCEPT} message must be sent over the@ set that is 6244In contrast, when accepting an incoming request, a
6166supplied for the set operation. 6245@code{GNUNET_SERVICE_SET_ACCEPT} message must be sent over the@ set that
6246is supplied for the set operation.
6167 6247
6168@node Initiating Operations 6248@node Initiating Operations
6169@subsubsection Initiating Operations 6249@subsubsection Initiating Operations
@@ -6192,8 +6272,8 @@ Sets are modified with the @code{GNUNET_SERVICE_SET_ADD} and
6192@subsubsection Results and Operation Status 6272@subsubsection Results and Operation Status
6193@c %**end of header 6273@c %**end of header
6194 6274
6195The service notifies the client of result elements and success/failure of a set 6275The service notifies the client of result elements and success/failure of
6196operation with the @code{GNUNET_SERVICE_SET_RESULT} message. 6276a set operation with the @code{GNUNET_SERVICE_SET_RESULT} message.
6197 6277
6198@node Iterating Sets 6278@node Iterating Sets
6199@subsubsection Iterating Sets 6279@subsubsection Iterating Sets
@@ -6202,9 +6282,10 @@ operation with the @code{GNUNET_SERVICE_SET_RESULT} message.
6202 6282
6203All elements of a set can be requested by sending 6283All elements of a set can be requested by sending
6204@code{GNUNET_SERVICE_SET_ITER_REQUEST}. The server responds with 6284@code{GNUNET_SERVICE_SET_ITER_REQUEST}. The server responds with
6205@code{GNUNET_SERVICE_SET_ITER_ELEMENT} and eventually terminates the iteration 6285@code{GNUNET_SERVICE_SET_ITER_ELEMENT} and eventually terminates the
6206with @code{GNUNET_SERVICE_SET_ITER_DONE}. After each received element, the 6286iteration with @code{GNUNET_SERVICE_SET_ITER_DONE}.
6207client@ must send @code{GNUNET_SERVICE_SET_ITER_ACK}. Note that only one set 6287After each received element, the client
6288must send @code{GNUNET_SERVICE_SET_ITER_ACK}. Note that only one set
6208iteration may be active for a set at any given time. 6289iteration may be active for a set at any given time.
6209 6290
6210@node The SET Intersection Peer-to-Peer Protocol 6291@node The SET Intersection Peer-to-Peer Protocol
@@ -6213,24 +6294,25 @@ iteration may be active for a set at any given time.
6213@c %**end of header 6294@c %**end of header
6214 6295
6215The intersection protocol operates over CADET and starts with a 6296The intersection protocol operates over CADET and starts with a
6216GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer initiating 6297GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer
6217the operation to the peer listening for inbound requests. It includes the 6298initiating the operation to the peer listening for inbound requests.
6218number of elements of the initiating peer, which is used to decide which side 6299It includes the number of elements of the initiating peer, which is used
6219will send a Bloom filter first. 6300to decide which side will send a Bloom filter first.
6220 6301
6221The listening peer checks if the operation type and application identifier are 6302The listening peer checks if the operation type and application
6222acceptable for its current state. If not, it responds with a 6303identifier are acceptable for its current state.
6223GNUNET_MESSAGE_TYPE_SET_RESULT and a status of GNUNET_SET_STATUS_FAILURE (and 6304If not, it responds with a GNUNET_MESSAGE_TYPE_SET_RESULT and a status of
6224terminates the CADET channel). 6305GNUNET_SET_STATUS_FAILURE (and terminates the CADET channel).
6225 6306
6226If the application accepts the request, the listener sends back a@ 6307If the application accepts the request, the listener sends back a
6227GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO if it has more elements 6308@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} if it has
6228in the set than the client. Otherwise, it immediately starts with the Bloom 6309more elements in the set than the client.
6229filter exchange. If the initiator receives a 6310Otherwise, it immediately starts with the Bloom filter exchange.
6230GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO response, it beings the 6311If the initiator receives a
6231Bloom filter exchange, unless the set size is indicated to be zero, in which 6312@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} response,
6232case the intersection is considered finished after just the initial 6313it beings the Bloom filter exchange, unless the set size is indicated to
6233handshake. 6314be zero, in which case the intersection is considered finished after
6315just the initial handshake.
6234 6316
6235 6317
6236@menu 6318@menu
@@ -6243,42 +6325,47 @@ handshake.
6243 6325
6244@c %**end of header 6326@c %**end of header
6245 6327
6246In this phase, each peer transmits a Bloom filter over the remaining keys of 6328In this phase, each peer transmits a Bloom filter over the remaining
6247the local set to the other peer using a 6329keys of the local set to the other peer using a
6248GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_BF message. This message additionally 6330@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_BF} message. This
6249includes the number of elements left in the sender's set, as well as the XOR 6331message additionally includes the number of elements left in the sender's
6250over all of the keys in that set. 6332set, as well as the XOR over all of the keys in that set.
6251 6333
6252The number of bits 'k' set per element in the Bloom filter is calculated based 6334The number of bits 'k' set per element in the Bloom filter is calculated
6253on the relative size of the two sets. Furthermore, the size of the Bloom filter 6335based on the relative size of the two sets.
6254is calculated based on 'k' and the number of elements in the set to maximize 6336Furthermore, the size of the Bloom filter is calculated based on 'k' and
6255the amount of data filtered per byte transmitted on the wire (while avoiding an 6337the number of elements in the set to maximize the amount of data filtered
6256excessively high number of iterations). 6338per byte transmitted on the wire (while avoiding an excessively high
6339number of iterations).
6257 6340
6258The receiver of the message removes all elements from its local set that do not 6341The receiver of the message removes all elements from its local set that
6259pass the Bloom filter test. It then checks if the set size of the sender and 6342do not pass the Bloom filter test.
6260the XOR over the keys match what is left of his own set. If they do, he sends 6343It then checks if the set size of the sender and the XOR over the keys
6261a@ GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_DONE back to indicate that the 6344match what is left of his own set. If they do, he sends a
6262latest set is the final result. Otherwise, the receiver starts another Bloom 6345@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_DONE} back to indicate
6263fitler exchange, except this time as the sender. 6346that the latest set is the final result.
6347Otherwise, the receiver starts another Bloom fitler exchange, except
6348this time as the sender.
6264 6349
6265@node Salt 6350@node Salt
6266@subsubsection Salt 6351@subsubsection Salt
6267 6352
6268@c %**end of header 6353@c %**end of header
6269 6354
6270Bloomfilter operations are probablistic: With some non-zero probability the 6355Bloomfilter operations are probablistic: With some non-zero probability
6271test may incorrectly say an element is in the set, even though it is not. 6356the test may incorrectly say an element is in the set, even though it is
6357not.
6272 6358
6273To mitigate this problem, the intersection protocol iterates exchanging Bloom 6359To mitigate this problem, the intersection protocol iterates exchanging
6274filters using a different random 32-bit salt in each iteration (the salt is 6360Bloom filters using a different random 32-bit salt in each iteration (the
6275also included in the message). With different salts, set operations may fail 6361salt is also included in the message).
6276for different elements. Merging the results from the executions, the 6362With different salts, set operations may fail for different elements.
6277probability of failure drops to zero. 6363Merging the results from the executions, the probability of failure drops
6364to zero.
6278 6365
6279The iterations terminate once both peers have established that they have sets 6366The iterations terminate once both peers have established that they have
6280of the same size, and where the XOR over all keys computes the same 512-bit 6367sets of the same size, and where the XOR over all keys computes the same
6281value (leaving a failure probability of 2-511). 6368512-bit value (leaving a failure probability of 2-511).
6282 6369
6283@node The SET Union Peer-to-Peer Protocol 6370@node The SET Union Peer-to-Peer Protocol
6284@subsection The SET Union Peer-to-Peer Protocol 6371@subsection The SET Union Peer-to-Peer Protocol
@@ -6290,105 +6377,114 @@ without prior context. You should read this paper first if you want to
6290understand the protocol. 6377understand the protocol.
6291 6378
6292The union protocol operates over CADET and starts with a 6379The union protocol operates over CADET and starts with a
6293GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer initiating 6380GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer
6294the operation to the peer listening for inbound requests. It includes the 6381initiating the operation to the peer listening for inbound requests.
6295number of elements of the initiating peer, which is currently not used. 6382It includes the number of elements of the initiating peer, which is
6383currently not used.
6296 6384
6297The listening peer checks if the operation type and application identifier are 6385The listening peer checks if the operation type and application
6298acceptable for its current state. If not, it responds with a 6386identifier are acceptable for its current state. If not, it responds with
6299GNUNET_MESSAGE_TYPE_SET_RESULT and a status of GNUNET_SET_STATUS_FAILURE (and 6387a @code{GNUNET_MESSAGE_TYPE_SET_RESULT} and a status of
6300terminates the CADET channel). 6388@code{GNUNET_SET_STATUS_FAILURE} (and terminates the CADET channel).
6301 6389
6302If the application accepts the request, it sends back a strata estimator using 6390If the application accepts the request, it sends back a strata estimator
6303a message of type GNUNET_MESSAGE_TYPE_SET_UNION_P2P_SE. The initiator evaluates 6391using a message of type GNUNET_MESSAGE_TYPE_SET_UNION_P2P_SE. The
6304the strata estimator and initiates the exchange of invertible Bloom filters, 6392initiator evaluates the strata estimator and initiates the exchange of
6305sending a GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF. 6393invertible Bloom filters, sending a GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF.
6306 6394
6307During the IBF exchange, if the receiver cannot invert the Bloom filter or 6395During the IBF exchange, if the receiver cannot invert the Bloom filter or
6308detects a cycle, it sends a larger IBF in response (up to a defined maximum 6396detects a cycle, it sends a larger IBF in response (up to a defined
6309limit; if that limit is reached, the operation fails). Elements decoded while 6397maximum limit; if that limit is reached, the operation fails).
6310processing the IBF are transmitted to the other peer using 6398Elements decoded while processing the IBF are transmitted to the other
6311GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS, or requested from the other peer using 6399peer using GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS, or requested from the
6312GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS messages, depending on the sign 6400other peer using GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS messages,
6313observed during decoding of the IBF. Peers respond to a 6401depending on the sign observed during decoding of the IBF.
6314GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS message with the respective 6402Peers respond to a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS message
6315element in a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS message. If the IBF fully 6403with the respective element in a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS
6316decodes, the peer responds with a GNUNET_MESSAGE_TYPE_SET_UNION_P2P_DONE 6404message. If the IBF fully decodes, the peer responds with a
6317message instead of another GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF. 6405GNUNET_MESSAGE_TYPE_SET_UNION_P2P_DONE message instead of another
6318 6406GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF.
6319All Bloom filter operations use a salt to mingle keys before hasing them into 6407
6320buckets, such that future iterations have a fresh chance of succeeding if they 6408All Bloom filter operations use a salt to mingle keys before hasing them
6321failed due to collisions before. 6409into buckets, such that future iterations have a fresh chance of
6322 6410succeeding if they failed due to collisions before.
6411
6412@cindex STATISTICS subsystem
6323@node GNUnet's STATISTICS subsystem 6413@node GNUnet's STATISTICS subsystem
6324@section GNUnet's STATISTICS subsystem 6414@section GNUnet's STATISTICS subsystem
6325 6415
6326@c %**end of header 6416@c %**end of header
6327 6417
6328In GNUnet, the STATISTICS subsystem offers a central place for all subsystems 6418In GNUnet, the STATISTICS subsystem offers a central place for all
6329to publish unsigned 64-bit integer run-time statistics. Keeping this 6419subsystems to publish unsigned 64-bit integer run-time statistics.
6330information centrally means that there is a unified way for the user to obtain 6420Keeping this information centrally means that there is a unified way for
6331data on all subsystems, and individual subsystems do not have to always include 6421the user to obtain data on all subsystems, and individual subsystems do
6332a custom data export method for performance metrics and other statistics. For 6422not have to always include a custom data export method for performance
6333example, the TRANSPORT system uses STATISTICS to update information about the 6423metrics and other statistics. For example, the TRANSPORT system uses
6334number of directly connected peers and the bandwidth that has been consumed by 6424STATISTICS to update information about the number of directly connected
6335the various plugins. This information is valuable for diagnosing connectivity 6425peers and the bandwidth that has been consumed by the various plugins.
6336and performance issues. 6426This information is valuable for diagnosing connectivity and performance
6427issues.
6337 6428
6338Following the GNUnet service architecture, the STATISTICS subsystem is divided 6429Following the GNUnet service architecture, the STATISTICS subsystem is
6339into an API which is exposed through the header 6430divided into an API which is exposed through the header
6340@strong{gnunet_statistics_service.h} and the STATISTICS service 6431@strong{gnunet_statistics_service.h} and the STATISTICS service
6341@strong{gnunet-service-statistics}. The @strong{gnunet-statistics} command-line 6432@strong{gnunet-service-statistics}. The @strong{gnunet-statistics}
6342tool can be used to obtain (and change) information about the values stored by 6433command-line tool can be used to obtain (and change) information about
6343the STATISTICS service. The STATISTICS service does not communicate with other 6434the values stored by the STATISTICS service. The STATISTICS service does
6344peers. 6435not communicate with other peers.
6345 6436
6346Data is stored in the STATISTICS service in the form of tuples 6437Data is stored in the STATISTICS service in the form of tuples
6347@strong{(subsystem, name, value, persistence)}. The subsystem determines to 6438@strong{(subsystem, name, value, persistence)}. The subsystem determines
6348which other GNUnet's subsystem the data belongs. name is the name through which 6439to which other GNUnet's subsystem the data belongs. name is the name
6349value is associated. It uniquely identifies the record from among other records 6440through which value is associated. It uniquely identifies the record
6350belonging to the same subsystem. In some parts of the code, the pair 6441from among other records belonging to the same subsystem.
6351@strong{(subsystem, name)} is called a @strong{statistic} as it identifies the 6442In some parts of the code, the pair @strong{(subsystem, name)} is called
6352values stored in the STATISTCS service.The persistence flag determines if the 6443a @strong{statistic} as it identifies the values stored in the STATISTCS
6353record has to be preserved across service restarts. A record is said to be 6444service.The persistence flag determines if the record has to be preserved
6354persistent if this flag is set for it; if not, the record is treated as a 6445across service restarts. A record is said to be persistent if this flag
6355non-persistent record and it is lost after service restart. Persistent records 6446is set for it; if not, the record is treated as a non-persistent record
6356are written to and read from the file @strong{statistics.data} before shutdown 6447and it is lost after service restart. Persistent records are written to
6448and read from the file @strong{statistics.data} before shutdown
6357and upon startup. The file is located in the HOME directory of the peer. 6449and upon startup. The file is located in the HOME directory of the peer.
6358 6450
6359An anomaly of the STATISTICS service is that it does not terminate immediately 6451An anomaly of the STATISTICS service is that it does not terminate
6360upon receiving a shutdown signal if it has any clients connected to it. It 6452immediately upon receiving a shutdown signal if it has any clients
6361waits for all the clients that are not monitors to close their connections 6453connected to it. It waits for all the clients that are not monitors to
6362before terminating itself. This is to prevent the loss of data during peer 6454close their connections before terminating itself.
6363shutdown --- delaying the STATISTICS service shutdown helps other services to 6455This is to prevent the loss of data during peer shutdown --- delaying the
6364store important data to STATISTICS during shutdown. 6456STATISTICS service shutdown helps other services to store important data
6457to STATISTICS during shutdown.
6365 6458
6366@menu 6459@menu
6367* libgnunetstatistics:: 6460* libgnunetstatistics::
6368* The STATISTICS Client-Service Protocol:: 6461* The STATISTICS Client-Service Protocol::
6369@end menu 6462@end menu
6370 6463
6464@cindex libgnunetstatistics
6371@node libgnunetstatistics 6465@node libgnunetstatistics
6372@subsection libgnunetstatistics 6466@subsection libgnunetstatistics
6373 6467
6374@c %**end of header 6468@c %**end of header
6375 6469
6376@strong{libgnunetstatistics} is the library containing the API for the 6470@strong{libgnunetstatistics} is the library containing the API for the
6377STATISTICS subsystem. Any process requiring to use STATISTICS should use this 6471STATISTICS subsystem. Any process requiring to use STATISTICS should use
6378API by to open a connection to the STATISTICS service. This is done by calling 6472this API by to open a connection to the STATISTICS service.
6379the function @code{GNUNET_STATISTICS_create()}. This function takes the 6473This is done by calling the function @code{GNUNET_STATISTICS_create()}.
6380subsystem's name which is trying to use STATISTICS and a configuration. All 6474This function takes the subsystem's name which is trying to use STATISTICS
6381values written to STATISTICS with this connection will be placed in the section 6475and a configuration.
6382corresponding to the given subsystem's name. The connection to STATISTICS can 6476All values written to STATISTICS with this connection will be placed in
6383be destroyed with the function GNUNET_STATISTICS_destroy(). This function 6477the section corresponding to the given subsystem's name.
6384allows for the connection to be destroyed immediately or upon transferring all 6478The connection to STATISTICS can be destroyed with the function
6479@code{GNUNET_STATISTICS_destroy()}. This function allows for the
6480connection to be destroyed immediately or upon transferring all
6385pending write requests to the service. 6481pending write requests to the service.
6386 6482
6387Note: STATISTICS subsystem can be disabled by setting @code{DISABLE = YES} 6483Note: STATISTICS subsystem can be disabled by setting @code{DISABLE = YES}
6388under the @code{[STATISTICS]} section in the configuration. With such a 6484under the @code{[STATISTICS]} section in the configuration. With such a
6389configuration all calls to @code{GNUNET_STATISTICS_create()} return @code{NULL} 6485configuration all calls to @code{GNUNET_STATISTICS_create()} return
6390as the STATISTICS subsystem is unavailable and no other functions from the API 6486@code{NULL} as the STATISTICS subsystem is unavailable and no other
6391can be used. 6487functions from the API can be used.
6392 6488
6393 6489
6394@menu 6490@menu
@@ -6402,66 +6498,74 @@ can be used.
6402 6498
6403@c %**end of header 6499@c %**end of header
6404 6500
6405Once a connection to the statistics service is obtained, information about any 6501Once a connection to the statistics service is obtained, information
6406other system which uses statistics can be retrieved with the function 6502about any other system which uses statistics can be retrieved with the
6407GNUNET_STATISTICS_get(). This function takes the connection handle, the name of 6503function GNUNET_STATISTICS_get().
6408the subsystem whose information we are interested in (a @code{NULL} value will 6504This function takes the connection handle, the name of the subsystem
6409retrieve information of all available subsystems using STATISTICS), the name of 6505whose information we are interested in (a @code{NULL} value will
6410the statistic we are interested in (a @code{NULL} value will retrieve all 6506retrieve information of all available subsystems using STATISTICS), the
6411available statistics), a continuation callback which is called when all of 6507name of the statistic we are interested in (a @code{NULL} value will
6412requested information is retrieved, an iterator callback which is called for 6508retrieve all available statistics), a continuation callback which is
6413each parameter in the retrieved information and a closure for the 6509called when all of requested information is retrieved, an iterator
6414aforementioned callbacks. The library then invokes the iterator callback for 6510callback which is called for each parameter in the retrieved information
6415each value matching the request. 6511and a closure for the aforementioned callbacks. The library then invokes
6512the iterator callback for each value matching the request.
6416 6513
6417Call to @code{GNUNET_STATISTICS_get()} is asynchronous and can be canceled with 6514Call to @code{GNUNET_STATISTICS_get()} is asynchronous and can be
6418the function @code{GNUNET_STATISTICS_get_cancel()}. This is helpful when 6515canceled with the function @code{GNUNET_STATISTICS_get_cancel()}.
6419retrieving statistics takes too long and especially when we want to shutdown 6516This is helpful when retrieving statistics takes too long and especially
6420and cleanup everything. 6517when we want to shutdown and cleanup everything.
6421 6518
6422@node Setting statistics and updating them 6519@node Setting statistics and updating them
6423@subsubsection Setting statistics and updating them 6520@subsubsection Setting statistics and updating them
6424 6521
6425@c %**end of header 6522@c %**end of header
6426 6523
6427So far we have seen how to retrieve statistics, here we will learn how we can 6524So far we have seen how to retrieve statistics, here we will learn how we
6428set statistics and update them so that other subsystems can retrieve them. 6525can set statistics and update them so that other subsystems can retrieve
6526them.
6429 6527
6430A new statistic can be set using the function @code{GNUNET_STATISTICS_set()}. 6528A new statistic can be set using the function
6431This function takes the name of the statistic and its value and a flag to make 6529@code{GNUNET_STATISTICS_set()}.
6432the statistic persistent. The value of the statistic should be of the type 6530This function takes the name of the statistic and its value and a flag to
6433@code{uint64_t}. The function does not take the name of the subsystem; it is 6531make the statistic persistent.
6434determined from the previous @code{GNUNET_STATISTICS_create()} invocation. If 6532The value of the statistic should be of the type @code{uint64_t}.
6533The function does not take the name of the subsystem; it is determined
6534from the previous @code{GNUNET_STATISTICS_create()} invocation. If
6435the given statistic is already present, its value is overwritten. 6535the given statistic is already present, its value is overwritten.
6436 6536
6437An existing statistics can be updated, i.e its value can be increased or 6537An existing statistics can be updated, i.e its value can be increased or
6438decreased by an amount with the function @code{GNUNET_STATISTICS_update()}. The 6538decreased by an amount with the function
6439parameters to this function are similar to @code{GNUNET_STATISTICS_set()}, 6539@code{GNUNET_STATISTICS_update()}.
6440except that it takes the amount to be changed as a type @code{int64_t} instead 6540The parameters to this function are similar to
6441of the value. 6541@code{GNUNET_STATISTICS_set()}, except that it takes the amount to be
6542changed as a type @code{int64_t} instead of the value.
6442 6543
6443The library will combine multiple set or update operations into one message if 6544The library will combine multiple set or update operations into one
6444the client performs requests at a rate that is faster than the available IPC 6545message if the client performs requests at a rate that is faster than the
6445with the STATISTICS service. Thus, the client does not have to worry about 6546available IPC with the STATISTICS service. Thus, the client does not have
6446sending requests too quickly. 6547to worry about sending requests too quickly.
6447 6548
6448@node Watches 6549@node Watches
6449@subsubsection Watches 6550@subsubsection Watches
6450 6551
6451@c %**end of header 6552@c %**end of header
6452 6553
6453As interesting feature of STATISTICS lies in serving notifications whenever a 6554As interesting feature of STATISTICS lies in serving notifications
6454statistic of our interest is modified. This is achieved by registering a watch 6555whenever a statistic of our interest is modified.
6455through the function @code{GNUNET_STATISTICS_watch()}. The parameters of this 6556This is achieved by registering a watch through the function
6456function are similar to those of @code{GNUNET_STATISTICS_get()}. Changes to the
6457respective statistic's value will then cause the given iterator callback to be
6458called. Note: A watch can only be registered for a specific statistic. Hence
6459the subsystem name and the parameter name cannot be @code{NULL} in a call to
6460@code{GNUNET_STATISTICS_watch()}. 6557@code{GNUNET_STATISTICS_watch()}.
6558The parameters of this function are similar to those of
6559@code{GNUNET_STATISTICS_get()}.
6560Changes to the respective statistic's value will then cause the given
6561iterator callback to be called.
6562Note: A watch can only be registered for a specific statistic. Hence
6563the subsystem name and the parameter name cannot be @code{NULL} in a
6564call to @code{GNUNET_STATISTICS_watch()}.
6461 6565
6462A registered watch will keep notifying any value changes until 6566A registered watch will keep notifying any value changes until
6463@code{GNUNET_STATISTICS_watch_cancel()} is called with the same parameters that 6567@code{GNUNET_STATISTICS_watch_cancel()} is called with the same
6464are used for registering the watch. 6568parameters that are used for registering the watch.
6465 6569
6466@node The STATISTICS Client-Service Protocol 6570@node The STATISTICS Client-Service Protocol
6467@subsection The STATISTICS Client-Service Protocol 6571@subsection The STATISTICS Client-Service Protocol
@@ -6480,12 +6584,13 @@ are used for registering the watch.
6480@c %**end of header 6584@c %**end of header
6481 6585
6482To retrieve statistics, the client transmits a message of type 6586To retrieve statistics, the client transmits a message of type
6483@code{GNUNET_MESSAGE_TYPE_STATISTICS_GET} containing the given subsystem name 6587@code{GNUNET_MESSAGE_TYPE_STATISTICS_GET} containing the given subsystem
6484and statistic parameter to the STATISTICS service. The service responds with a 6588name and statistic parameter to the STATISTICS service.
6485message of type @code{GNUNET_MESSAGE_TYPE_STATISTICS_VALUE} for each of the 6589The service responds with a message of type
6486statistics parameters that match the client request for the client. The end of 6590@code{GNUNET_MESSAGE_TYPE_STATISTICS_VALUE} for each of the statistics
6487information retrieved is signaled by the service by sending a message of type 6591parameters that match the client request for the client. The end of
6488@code{GNUNET_MESSAGE_TYPE_STATISTICS_END}. 6592information retrieved is signaled by the service by sending a message of
6593type @code{GNUNET_MESSAGE_TYPE_STATISTICS_END}.
6489 6594
6490@node Setting and updating statistics 6595@node Setting and updating statistics
6491@subsubsection Setting and updating statistics 6596@subsubsection Setting and updating statistics
@@ -6497,64 +6602,70 @@ communicated to the service through the message
6497@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}. 6602@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}.
6498 6603
6499When the service receives a message of type 6604When the service receives a message of type
6500@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}, it retrieves the subsystem name and 6605@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}, it retrieves the subsystem
6501checks for a statistic parameter with matching the name given in the message. 6606name and checks for a statistic parameter with matching the name given in
6502If a statistic parameter is found, the value is overwritten by the new value 6607the message.
6503from the message; if not found then a new statistic parameter is created with 6608If a statistic parameter is found, the value is overwritten by the new
6504the given name and value. 6609value from the message; if not found then a new statistic parameter is
6610created with the given name and value.
6505 6611
6506In addition to just setting an absolute value, it is possible to perform a 6612In addition to just setting an absolute value, it is possible to perform a
6507relative update by sending a message of type 6613relative update by sending a message of type
6508@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET} with an update flag 6614@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET} with an update flag
6509(@code{GNUNET_STATISTICS_SETFLAG_RELATIVE}) signifying that the value in the 6615(@code{GNUNET_STATISTICS_SETFLAG_RELATIVE}) signifying that the value in
6510message should be treated as an update value. 6616the message should be treated as an update value.
6511 6617
6512@node Watching for updates 6618@node Watching for updates
6513@subsubsection Watching for updates 6619@subsubsection Watching for updates
6514 6620
6515@c %**end of header 6621@c %**end of header
6516 6622
6517The function registers the watch at the service by sending a message of type 6623The function registers the watch at the service by sending a message of
6518@code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH}. The service then sends 6624type @code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH}. The service then sends
6519notifications through messages of type 6625notifications through messages of type
6520@code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH_VALUE} whenever the statistic 6626@code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH_VALUE} whenever the statistic
6521parameter's value is changed. 6627parameter's value is changed.
6522 6628
6629@cindex DHT
6630@cindex Distributed Hash Table
6523@node GNUnet's Distributed Hash Table (DHT) 6631@node GNUnet's Distributed Hash Table (DHT)
6524@section GNUnet's Distributed Hash Table (DHT) 6632@section GNUnet's Distributed Hash Table (DHT)
6525 6633
6526@c %**end of header 6634@c %**end of header
6527 6635
6528GNUnet includes a generic distributed hash table that can be used by developers 6636GNUnet includes a generic distributed hash table that can be used by
6529building P2P applications in the framework. This section documents high-level 6637developers building P2P applications in the framework.
6530features and how developers are expected to use the DHT. We have a research 6638This section documents high-level features and how developers are
6531paper detailing how the DHT works. Also, Nate's thesis includes a detailed 6639expected to use the DHT.
6532description and performance analysis (in chapter 6). 6640We have a research paper detailing how the DHT works.
6641Also, Nate's thesis includes a detailed description and performance
6642analysis (in chapter 6).
6533 6643
6534Key features of GNUnet's DHT include: 6644Key features of GNUnet's DHT include:
6535 6645
6536@itemize @bullet 6646@itemize @bullet
6537@item stores key-value pairs with values up to (approximately) 63k in size 6647@item stores key-value pairs with values up to (approximately) 63k in size
6538@item works with many underlay network topologies (small-world, random graph), 6648@item works with many underlay network topologies (small-world, random
6539underlay does not need to be a full mesh / clique 6649graph), underlay does not need to be a full mesh / clique
6540@item support for extended queries (more than just a simple 'key'), filtering 6650@item support for extended queries (more than just a simple 'key'),
6541duplicate replies within the network (bloomfilter) and content validation (for 6651filtering duplicate replies within the network (bloomfilter) and content
6542details, please read the subsection on the block library) 6652validation (for details, please read the subsection on the block library)
6543@item can (optionally) return paths taken by the PUT and GET operations to the 6653@item can (optionally) return paths taken by the PUT and GET operations
6544application 6654to the application
6545@item provides content replication to handle churn 6655@item provides content replication to handle churn
6546@end itemize 6656@end itemize
6547 6657
6548GNUnet's DHT is randomized and unreliable. Unreliable means that there is no 6658GNUnet's DHT is randomized and unreliable. Unreliable means that there is
6549strict guarantee that a value stored in the DHT is always found --- values are 6659no strict guarantee that a value stored in the DHT is always
6550only found with high probability. While this is somewhat true in all P2P DHTs, 6660found --- values are only found with high probability.
6551GNUnet developers should be particularly wary of this fact (this will help you 6661While this is somewhat true in all P2P DHTs, GNUnet developers should be
6552write secure, fault-tolerant code). Thus, when writing any application using 6662particularly wary of this fact (this will help you write secure,
6553the DHT, you should always consider the possibility that a value stored in the 6663fault-tolerant code). Thus, when writing any application using the DHT,
6554DHT by you or some other peer might simply not be returned, or returned with a 6664you should always consider the possibility that a value stored in the
6555significant delay. Your application logic must be written to tolerate this 6665DHT by you or some other peer might simply not be returned, or returned
6556(naturally, some loss of performance or quality of service is expected in this 6666with a significant delay.
6557case). 6667Your application logic must be written to tolerate this (naturally, some
6668loss of performance or quality of service is expected in this case).
6558 6669
6559@menu 6670@menu
6560* Block library and plugins:: 6671* Block library and plugins::
@@ -6583,62 +6694,71 @@ case).
6583 6694
6584Blocks are small (< 63k) pieces of data stored under a key (struct 6695Blocks are small (< 63k) pieces of data stored under a key (struct
6585GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which defines 6696GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which defines
6586their data format. Blocks are used in GNUnet as units of static data exchanged 6697their data format. Blocks are used in GNUnet as units of static data
6587between peers and stored (or cached) locally. Uses of blocks include 6698exchanged between peers and stored (or cached) locally.
6588file-sharing (the files are broken up into blocks), the VPN (DNS information is 6699Uses of blocks include file-sharing (the files are broken up into blocks),
6589stored in blocks) and the DHT (all information in the DHT and meta-information 6700the VPN (DNS information is stored in blocks) and the DHT (all
6590for the maintenance of the DHT are both stored using blocks). The block 6701information in the DHT and meta-information for the maintenance of the
6591subsystem provides a few common functions that must be available for any type 6702DHT are both stored using blocks).
6592of block. 6703The block subsystem provides a few common functions that must be
6593 6704available for any type of block.
6705
6706@cindex libgnunetblock API
6594@node The API of libgnunetblock 6707@node The API of libgnunetblock
6595@subsubsection The API of libgnunetblock 6708@subsubsection The API of libgnunetblock
6596 6709
6597@c %**end of header 6710@c %**end of header
6598 6711
6599The block library requires for each (family of) block type(s) a block plugin 6712The block library requires for each (family of) block type(s) a block
6600(implementing gnunet_block_plugin.h) that provides basic functions that are 6713plugin (implementing @file{gnunet_block_plugin.h}) that provides basic
6601needed by the DHT (and possibly other subsystems) to manage the block. These 6714functions that are needed by the DHT (and possibly other subsystems) to
6602block plugins are typically implemented within their respective subsystems.@ 6715manage the block.
6603The main block library is then used to locate, load and query the appropriate 6716These block plugins are typically implemented within their respective
6604block plugin. Which plugin is appropriate is determined by the block type 6717subsystems.
6605(which is just a 32-bit integer). Block plugins contain code that specifies 6718The main block library is then used to locate, load and query the
6606which block types are supported by a given plugin. The block library loads all 6719appropriate block plugin.
6607block plugins that are installed at the local peer and forwards the application 6720Which plugin is appropriate is determined by the block type (which is
6608request to the respective plugin. 6721just a 32-bit integer). Block plugins contain code that specifies which
6609 6722block types are supported by a given plugin. The block library loads all
6610The central functions of the block APIs (plugin and main library) are to allow 6723block plugins that are installed at the local peer and forwards the
6611the mapping of blocks to their respective key (if possible) and the ability to 6724application request to the respective plugin.
6612check that a block is well-formed and matches a given request (again, if 6725
6613possible). This way, GNUnet can avoid storing invalid blocks, storing blocks 6726The central functions of the block APIs (plugin and main library) are to
6614under the wrong key and forwarding blocks in response to a query that they do 6727allow the mapping of blocks to their respective key (if possible) and the
6728ability to check that a block is well-formed and matches a given
6729request (again, if possible).
6730This way, GNUnet can avoid storing invalid blocks, storing blocks under
6731the wrong key and forwarding blocks in response to a query that they do
6615not answer. 6732not answer.
6616 6733
6617One key function of block plugins is that it allows GNUnet to detect duplicate 6734One key function of block plugins is that it allows GNUnet to detect
6618replies (via the Bloom filter). All plugins MUST support detecting duplicate 6735duplicate replies (via the Bloom filter). All plugins MUST support
6619replies (by adding the current response to the Bloom filter and rejecting it if 6736detecting duplicate replies (by adding the current response to the
6620it is encountered again). If a plugin fails to do this, responses may loop in 6737Bloom filter and rejecting it if it is encountered again).
6621the network. 6738If a plugin fails to do this, responses may loop in the network.
6622 6739
6623@node Queries 6740@node Queries
6624@subsubsection Queries 6741@subsubsection Queries
6625@c %**end of header 6742@c %**end of header
6626 6743
6627The query format for any block in GNUnet consists of four main components. 6744The query format for any block in GNUnet consists of four main components.
6628First, the type of the desired block must be specified. Second, the query must 6745First, the type of the desired block must be specified. Second, the query
6629contain a hash code. The hash code is used for lookups in hash tables and 6746must contain a hash code. The hash code is used for lookups in hash
6630databases and must not be unique for the block (however, if possible a unique 6747tables and databases and must not be unique for the block (however, if
6631hash should be used as this would be best for performance). Third, an optional 6748possible a unique hash should be used as this would be best for
6632Bloom filter can be specified to exclude known results; replies that hash to 6749performance).
6633the bits set in the Bloom filter are considered invalid. False-positives can be 6750Third, an optional Bloom filter can be specified to exclude known results;
6634eliminated by sending the same query again with a different Bloom filter 6751replies that hash to the bits set in the Bloom filter are considered
6635mutator value, which parameterizes the hash function that is used. Finally, an 6752invalid. False-positives can be eliminated by sending the same query
6636optional application-specific "eXtended query" (xquery) can be specified to 6753again with a different Bloom filter mutator value, which parameterizes
6637further constrain the results. It is entirely up to the type-specific plugin to 6754the hash function that is used.
6638determine whether or not a given block matches a query (type, hash, Bloom 6755Finally, an optional application-specific "eXtended query" (xquery) can
6639filter, and xquery). Naturally, not all xquery's are valid and some types of 6756be specified to further constrain the results. It is entirely up to
6640blocks may not support Bloom filters either, so the plugin also needs to check 6757the type-specific plugin to determine whether or not a given block
6641if the query is valid in the first place. 6758matches a query (type, hash, Bloom filter, and xquery).
6759Naturally, not all xquery's are valid and some types of blocks may not
6760support Bloom filters either, so the plugin also needs to check if the
6761query is valid in the first place.
6642 6762
6643Depending on the results from the plugin, the DHT will then discard the 6763Depending on the results from the plugin, the DHT will then discard the
6644(invalid) query, forward the query, discard the (invalid) reply, cache the 6764(invalid) query, forward the query, discard the (invalid) reply, cache the
@@ -6649,10 +6769,11 @@ Depending on the results from the plugin, the DHT will then discard the
6649 6769
6650@c %**end of header 6770@c %**end of header
6651 6771
6652The source code in @strong{plugin_block_test.c} is a good starting point for 6772The source code in @strong{plugin_block_test.c} is a good starting point
6653new block plugins --- it does the minimal work by implementing a plugin that 6773for new block plugins --- it does the minimal work by implementing a
6654performs no validation at all. The respective @strong{Makefile.am} shows how to 6774plugin that performs no validation at all.
6655build and install a block plugin. 6775The respective @strong{Makefile.am} shows how to build and install a
6776block plugin.
6656 6777
6657@node Conclusion2 6778@node Conclusion2
6658@subsubsection Conclusion2 6779@subsubsection Conclusion2
@@ -6660,21 +6781,23 @@ build and install a block plugin.
6660@c %**end of header 6781@c %**end of header
6661 6782
6662In conclusion, GNUnet subsystems that want to use the DHT need to define a 6783In conclusion, GNUnet subsystems that want to use the DHT need to define a
6663block format and write a plugin to match queries and replies. For testing, the 6784block format and write a plugin to match queries and replies. For testing,
6664"GNUNET_BLOCK_TYPE_TEST" block type can be used; it accepts any query as valid 6785the "@code{GNUNET_BLOCK_TYPE_TEST}" block type can be used; it accepts
6665and any reply as matching any query. This type is also used for the DHT command 6786any query as valid and any reply as matching any query.
6666line tools. However, it should NOT be used for normal applications due to the 6787This type is also used for the DHT command line tools.
6667lack of error checking that results from this primitive implementation. 6788However, it should NOT be used for normal applications due to the lack
6668 6789of error checking that results from this primitive implementation.
6790
6791@cindex libgnunetdht
6669@node libgnunetdht 6792@node libgnunetdht
6670@subsection libgnunetdht 6793@subsection libgnunetdht
6671 6794
6672@c %**end of header 6795@c %**end of header
6673 6796
6674The DHT API itself is pretty simple and offers the usual GET and PUT functions 6797The DHT API itself is pretty simple and offers the usual GET and PUT
6675that work as expected. The specified block type refers to the block library 6798functions that work as expected. The specified block type refers to the
6676which allows the DHT to run application-specific logic for data stored in the 6799block library which allows the DHT to run application-specific logic for
6677network. 6800data stored in the network.
6678 6801
6679 6802
6680@menu 6803@menu
@@ -6689,23 +6812,26 @@ network.
6689 6812
6690@c %**end of header 6813@c %**end of header
6691 6814
6692When using GET, the main consideration for developers (other than the block 6815When using GET, the main consideration for developers (other than the
6693library) should be that after issuing a GET, the DHT will continuously cause 6816block library) should be that after issuing a GET, the DHT will
6694(small amounts of) network traffic until the operation is explicitly canceled. 6817continuously cause (small amounts of) network traffic until the operation
6695So GET does not simply send out a single network request once; instead, the 6818is explicitly canceled.
6696DHT will continue to search for data. This is needed to achieve good success 6819So GET does not simply send out a single network request once; instead,
6697rates and also handles the case where the respective PUT operation happens 6820the DHT will continue to search for data. This is needed to achieve good
6698after the GET operation was started. Developers should not cancel an existing 6821success rates and also handles the case where the respective PUT
6699GET operation and then explicitly re-start it to trigger a new round of 6822operation happens after the GET operation was started.
6700network requests; this is simply inefficient, especially as the internal 6823Developers should not cancel an existing GET operation and then
6701automated version can be more efficient, for example by filtering results in 6824explicitly re-start it to trigger a new round of network requests;
6702the network that have already been returned. 6825this is simply inefficient, especially as the internal automated version
6826can be more efficient, for example by filtering results in the network
6827that have already been returned.
6703 6828
6704If an application that performs a GET request has a set of replies that it 6829If an application that performs a GET request has a set of replies that it
6705already knows and would like to filter, it can call@ 6830already knows and would like to filter, it can call@
6706@code{GNUNET_DHT_get_filter_known_results} with an array of hashes over the 6831@code{GNUNET_DHT_get_filter_known_results} with an array of hashes over
6707respective blocks to tell the DHT that these results are not desired (any 6832the respective blocks to tell the DHT that these results are not
6708more). This way, the DHT will filter the respective blocks using the block 6833desired (any more).
6834This way, the DHT will filter the respective blocks using the block
6709library in the network, which may result in a significant reduction in 6835library in the network, which may result in a significant reduction in
6710bandwidth consumption. 6836bandwidth consumption.
6711 6837
@@ -6714,18 +6840,20 @@ bandwidth consumption.
6714 6840
6715@c %**end of header 6841@c %**end of header
6716 6842
6717In contrast to GET operations, developers @strong{must} manually re-run PUT 6843In contrast to GET operations, developers @strong{must} manually re-run
6718operations periodically (if they intend the content to continue to be 6844PUT operations periodically (if they intend the content to continue to be
6719available). Content stored in the DHT expires or might be lost due to churn. 6845available). Content stored in the DHT expires or might be lost due to
6720Furthermore, GNUnet's DHT typically requires multiple rounds of PUT operations 6846churn.
6721before a key-value pair is consistently available to all peers (the DHT 6847Furthermore, GNUnet's DHT typically requires multiple rounds of PUT
6722randomizes paths and thus storage locations, and only after multiple rounds of 6848operations before a key-value pair is consistently available to all
6723PUTs there will be a sufficient number of replicas in large DHTs). An explicit 6849peers (the DHT randomizes paths and thus storage locations, and only
6724PUT operation using the DHT API will only cause network traffic once, so in 6850after multiple rounds of PUTs there will be a sufficient number of
6725order to ensure basic availability and resistance to churn (and adversaries), 6851replicas in large DHTs). An explicit PUT operation using the DHT API will
6726PUTs must be repeated. While the exact frequency depends on the application, a 6852only cause network traffic once, so in order to ensure basic availability
6727rule of thumb is that there should be at least a dozen PUT operations within 6853and resistance to churn (and adversaries), PUTs must be repeated.
6728the content lifetime. Content in the DHT typically expires after one day, so 6854While the exact frequency depends on the application, a rule of thumb is
6855that there should be at least a dozen PUT operations within the content
6856lifetime. Content in the DHT typically expires after one day, so
6729DHT PUT operations should be repeated at least every 1-2 hours. 6857DHT PUT operations should be repeated at least every 1-2 hours.
6730 6858
6731@node MONITOR 6859@node MONITOR
@@ -6733,18 +6861,21 @@ DHT PUT operations should be repeated at least every 1-2 hours.
6733 6861
6734@c %**end of header 6862@c %**end of header
6735 6863
6736The DHT API also allows applications to monitor messages crossing the local 6864The DHT API also allows applications to monitor messages crossing the
6737DHT service. The types of messages used by the DHT are GET, PUT and RESULT 6865local DHT service.
6738messages. Using the monitoring API, applications can choose to monitor these 6866The types of messages used by the DHT are GET, PUT and RESULT messages.
6867Using the monitoring API, applications can choose to monitor these
6739requests, possibly limiting themselves to requests for a particular block 6868requests, possibly limiting themselves to requests for a particular block
6740type. 6869type.
6741 6870
6742The monitoring API is not only usefu only for diagnostics, it can also be used 6871The monitoring API is not only usefu only for diagnostics, it can also be
6743to trigger application operations based on PUT operations. For example, an 6872used to trigger application operations based on PUT operations.
6744application may use PUTs to distribute work requests to other peers. The 6873For example, an application may use PUTs to distribute work requests to
6745workers would then monitor for PUTs that give them work, instead of looking 6874other peers.
6746for work using GET operations. This can be beneficial, especially if the 6875The workers would then monitor for PUTs that give them work, instead of
6747workers have no good way to guess the keys under which work would be stored. 6876looking for work using GET operations.
6877This can be beneficial, especially if the workers have no good way to
6878guess the keys under which work would be stored.
6748Naturally, additional protocols might be needed to ensure that the desired 6879Naturally, additional protocols might be needed to ensure that the desired
6749number of workers will process the distributed workload. 6880number of workers will process the distributed workload.
6750 6881
@@ -6756,25 +6887,26 @@ number of workers will process the distributed workload.
6756There are two important options for GET and PUT requests: 6887There are two important options for GET and PUT requests:
6757 6888
6758@table @asis 6889@table @asis
6759@item GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all peers 6890@item GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all
6760should process the request, even if their peer ID is not closest to the key. 6891peers should process the request, even if their peer ID is not closest to
6761For a PUT request, this means that all peers that a request traverses may make 6892the key. For a PUT request, this means that all peers that a request
6762a copy of the data. Similarly for a GET request, all peers will check their 6893traverses may make a copy of the data.
6763local database for a result. Setting this option can thus significantly improve 6894Similarly for a GET request, all peers will check their local database
6764caching and reduce bandwidth consumption --- at the expense of a larger DHT 6895for a result. Setting this option can thus significantly improve caching
6896and reduce bandwidth consumption --- at the expense of a larger DHT
6765database. If in doubt, we recommend that this option should be used. 6897database. If in doubt, we recommend that this option should be used.
6766@item GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record the path 6898@item GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record
6767that a GET or a PUT request is taking through the overlay network. The 6899the path that a GET or a PUT request is taking through the overlay
6768resulting paths are then returned to the application with the respective 6900network. The resulting paths are then returned to the application with
6769result. This allows the receiver of a result to construct a path to the 6901the respective result. This allows the receiver of a result to construct
6770originator of the data, which might then be used for routing. Naturally, 6902a path to the originator of the data, which might then be used for
6771setting this option requires additional bandwidth and disk space, so 6903routing. Naturally, setting this option requires additional bandwidth
6772applications should only set this if the paths are needed by the application 6904and disk space, so applications should only set this if the paths are
6773logic. 6905needed by the application logic.
6774@item GNUNET_DHT_RO_FIND_PEER This option is an internal option used by 6906@item GNUNET_DHT_RO_FIND_PEER This option is an internal option used by
6775the DHT's peer discovery mechanism and should not be used by applications. 6907the DHT's peer discovery mechanism and should not be used by applications.
6776@item GNUNET_DHT_RO_BART This option is currently not implemented. It may in 6908@item GNUNET_DHT_RO_BART This option is currently not implemented. It may
6777the future offer performance improvements for clique topologies. 6909in the future offer performance improvements for clique topologies.
6778@end table 6910@end table
6779 6911
6780@node The DHT Client-Service Protocol 6912@node The DHT Client-Service Protocol
@@ -6793,73 +6925,76 @@ the future offer performance improvements for clique topologies.
6793 6925
6794@c %**end of header 6926@c %**end of header
6795 6927
6796To store (PUT) data into the DHT, the client sends a@ @code{struct 6928To store (PUT) data into the DHT, the client sends a
6797GNUNET_DHT_ClientPutMessage} to the service. This message specifies the block 6929@code{struct GNUNET_DHT_ClientPutMessage} to the service.
6798type, routing options, the desired replication level, the expiration time, key, 6930This message specifies the block type, routing options, the desired
6799value and a 64-bit unique ID for the operation. The service responds with a@ 6931replication level, the expiration time, key,
6800@code{struct GNUNET_DHT_ClientPutConfirmationMessage} with the same 64-bit 6932value and a 64-bit unique ID for the operation. The service responds with
6801unique ID. Note that the service sends the confirmation as soon as it has 6933a @code{struct GNUNET_DHT_ClientPutConfirmationMessage} with the same
6802locally processed the PUT request. The PUT may still be propagating through the 693464-bit unique ID. Note that the service sends the confirmation as soon as
6803network at this time. 6935it has locally processed the PUT request. The PUT may still be
6936propagating through the network at this time.
6804 6937
6805In the future, we may want to change this to provide (limited) feedback to the 6938In the future, we may want to change this to provide (limited) feedback
6806client, for example if we detect that the PUT operation had no effect because 6939to the client, for example if we detect that the PUT operation had no
6807the same key-value pair was already stored in the DHT. However, changing this 6940effect because the same key-value pair was already stored in the DHT.
6808would also require additional state and messages in the P2P 6941However, changing this would also require additional state and messages
6809interaction. 6942in the P2P interaction.
6810 6943
6811@node GETting data from the DHT 6944@node GETting data from the DHT
6812@subsubsection GETting data from the DHT 6945@subsubsection GETting data from the DHT
6813 6946
6814@c %**end of header 6947@c %**end of header
6815 6948
6816To retrieve (GET) data from the DHT, the client sends a@ @code{struct 6949To retrieve (GET) data from the DHT, the client sends a
6817GNUNET_DHT_ClientGetMessage} to the service. The message specifies routing 6950@code{struct GNUNET_DHT_ClientGetMessage} to the service. The message
6818options, a replication level (for replicating the GET, not the content), the 6951specifies routing options, a replication level (for replicating the GET,
6819desired block type, the key, the (optional) extended query and unique 64-bit 6952not the content), the desired block type, the key, the (optional)
6820request ID. 6953extended query and unique 64-bit request ID.
6821 6954
6822Additionally, the client may send any number of@ @code{struct 6955Additionally, the client may send any number of
6823GNUNET_DHT_ClientGetResultSeenMessage}s to notify the service about results 6956@code{struct GNUNET_DHT_ClientGetResultSeenMessage}s to notify the
6824that the client is already aware of. These messages consist of the key, the 6957service about results that the client is already aware of.
6825unique 64-bit ID of the request, and an arbitrary number of hash codes over the 6958These messages consist of the key, the unique 64-bit ID of the request,
6826blocks that the client is already aware of. As messages are restricted to 64k, 6959and an arbitrary number of hash codes over the blocks that the client is
6827a client that already knows more than about a thousand blocks may need to send 6960already aware of. As messages are restricted to 64k, a client that
6828several of these messages. Naturally, the client should transmit these messages 6961already knows more than about a thousand blocks may need to send
6829as quickly as possible after the original GET request such that the DHT can 6962several of these messages. Naturally, the client should transmit these
6830filter those results in the network early on. Naturally, as these messages are 6963messages as quickly as possible after the original GET request such that
6831send after the original request, it is conceivalbe that the DHT service may 6964the DHT can filter those results in the network early on. Naturally, as
6832return blocks that match those already known to the client anyway. 6965these messages are send after the original request, it is conceivalbe
6966that the DHT service may return blocks that match those already known
6967to the client anyway.
6833 6968
6834In response to a GET request, the service will send @code{struct 6969In response to a GET request, the service will send @code{struct
6835GNUNET_DHT_ClientResultMessage}s to the client. These messages contain the 6970GNUNET_DHT_ClientResultMessage}s to the client. These messages contain the
6836block type, expiration, key, unique ID of the request and of course the value 6971block type, expiration, key, unique ID of the request and of course the
6837(a block). Depending on the options set for the respective operations, the 6972value (a block). Depending on the options set for the respective
6838replies may also contain the path the GET and/or the PUT took through the 6973operations, the replies may also contain the path the GET and/or the PUT
6839network. 6974took through the network.
6840 6975
6841A client can stop receiving replies either by disconnecting or by sending a 6976A client can stop receiving replies either by disconnecting or by sending
6842@code{struct GNUNET_DHT_ClientGetStopMessage} which must contain the key and 6977a @code{struct GNUNET_DHT_ClientGetStopMessage} which must contain the
6843the 64-bit unique ID of the original request. Using an explicit "stop" message 6978key and the 64-bit unique ID of the original request. Using an
6844is more common as this allows a client to run many concurrent GET operations 6979explicit "stop" message is more common as this allows a client to run
6845over the same connection with the DHT service --- and to stop them 6980many concurrent GET operations over the same connection with the DHT
6846individually. 6981service --- and to stop them individually.
6847 6982
6848@node Monitoring the DHT 6983@node Monitoring the DHT
6849@subsubsection Monitoring the DHT 6984@subsubsection Monitoring the DHT
6850 6985
6851@c %**end of header 6986@c %**end of header
6852 6987
6853To begin monitoring, the client sends a @code{struct 6988To begin monitoring, the client sends a
6854GNUNET_DHT_MonitorStartStop} message to the DHT service. In this message, flags 6989@code{struct GNUNET_DHT_MonitorStartStop} message to the DHT service.
6855can be set to enable (or disable) monitoring of GET, PUT and RESULT messages 6990In this message, flags can be set to enable (or disable) monitoring of
6856that pass through a peer. The message can also restrict monitoring to a 6991GET, PUT and RESULT messages that pass through a peer. The message can
6857particular block type or a particular key. Once monitoring is enabled, the DHT 6992also restrict monitoring to a particular block type or a particular key.
6858service will notify the client about any matching event using @code{struct 6993Once monitoring is enabled, the DHT service will notify the client about
6859GNUNET_DHT_MonitorGetMessage}s for GET events, @code{struct 6994any matching event using @code{struct GNUNET_DHT_MonitorGetMessage}s for
6860GNUNET_DHT_MonitorPutMessage} for PUT events and@ @code{struct 6995GET events, @code{struct GNUNET_DHT_MonitorPutMessage} for PUT events
6861GNUNET_DHT_MonitorGetRespMessage} for RESULTs. Each of these messages contains 6996and @code{struct GNUNET_DHT_MonitorGetRespMessage} for RESULTs. Each of
6862all of the information about the event. 6997these messages contains all of the information about the event.
6863 6998
6864@node The DHT Peer-to-Peer Protocol 6999@node The DHT Peer-to-Peer Protocol
6865@subsection The DHT Peer-to-Peer Protocol 7000@subsection The DHT Peer-to-Peer Protocol