diff options
author | Martin Schanzenbach <schanzen@gnunet.org> | 2022-08-02 17:25:41 +0200 |
---|---|---|
committer | Martin Schanzenbach <schanzen@gnunet.org> | 2022-08-02 17:25:41 +0200 |
commit | 4fe567d9d94b9159254a2f2cce64855a794d9699 (patch) | |
tree | b286f5a454472ec92ff88376b297e411e64c5843 /doc/handbook/chapters | |
parent | e8b7707f833739851227d4865e6c6064865f19ec (diff) | |
download | gnunet-4fe567d9d94b9159254a2f2cce64855a794d9699.tar.gz gnunet-4fe567d9d94b9159254a2f2cce64855a794d9699.zip |
DOC: Move from texinfo to sphinx
Diffstat (limited to 'doc/handbook/chapters')
-rw-r--r-- | doc/handbook/chapters/configuration.texi | 77 | ||||
-rw-r--r-- | doc/handbook/chapters/contributing.texi | 117 | ||||
-rw-r--r-- | doc/handbook/chapters/developer.texi | 10154 | ||||
-rw-r--r-- | doc/handbook/chapters/installation.texi | 2497 | ||||
-rw-r--r-- | doc/handbook/chapters/keyconcepts.texi | 350 | ||||
-rw-r--r-- | doc/handbook/chapters/philosophy.texi | 82 | ||||
-rw-r--r-- | doc/handbook/chapters/preface.texi | 212 | ||||
-rw-r--r-- | doc/handbook/chapters/user.texi | 2603 | ||||
-rw-r--r-- | doc/handbook/chapters/vocabulary.texi | 72 |
9 files changed, 0 insertions, 16164 deletions
diff --git a/doc/handbook/chapters/configuration.texi b/doc/handbook/chapters/configuration.texi deleted file mode 100644 index 27efc82e2..000000000 --- a/doc/handbook/chapters/configuration.texi +++ /dev/null | |||
@@ -1,77 +0,0 @@ | |||
1 | @node Configuration Handbook | ||
2 | @chapter Configuration Handbook | ||
3 | |||
4 | This chapter has yet to be fully written. It is intended to be about in-depth | ||
5 | configuration of GNUnet. | ||
6 | |||
7 | |||
8 | @menu | ||
9 | * Config file format:: | ||
10 | @end menu | ||
11 | |||
12 | |||
13 | @node Config file format | ||
14 | @section Config file format | ||
15 | |||
16 | In GNUnet realm, all components obey the same pattern to get configuration | ||
17 | values. According to this pattern, once the component has been installed, the | ||
18 | installation deploys default values in @file{$prefix/share/gnunet/config.d/}, | ||
19 | in @file{.conf} files. In order to override these defaults, the user can | ||
20 | write a custom @file{.conf} file and either pass it to the component at | ||
21 | execution time, or name it @file{gnunet.conf} and place it under | ||
22 | @file{$HOME/.config/}. | ||
23 | |||
24 | A config file is a text file containing sections, and each section | ||
25 | contains its values. The right format follows: | ||
26 | |||
27 | @example | ||
28 | [section1] | ||
29 | value1 = string | ||
30 | value2 = 23 | ||
31 | |||
32 | [section2] | ||
33 | value21 = string | ||
34 | value22 = /path22 | ||
35 | @end example | ||
36 | |||
37 | Throughout any configuration file, it is possible to use @code{$}-prefixed | ||
38 | variables, like @code{$VAR}, especially when they represent filenames in in | ||
39 | the filesystem. It is also possible to provide defaults values for those | ||
40 | variables that are unset, by using the following syntax: | ||
41 | |||
42 | @example | ||
43 | $@{VAR:-default@} | ||
44 | @end example | ||
45 | |||
46 | @noindent | ||
47 | However, there are two ways a user can set @code{$}-prefixable variables: | ||
48 | (a) by defining them under a @code{[paths]} section | ||
49 | |||
50 | @example | ||
51 | [paths] | ||
52 | GNUNET_DEPLOYMENT_SHARED = $@{HOME@}/shared-data | ||
53 | .. | ||
54 | [section-x] | ||
55 | path-x = $@{GNUNET_DEPLOYMENT_SHARED@}/x | ||
56 | @end example | ||
57 | |||
58 | @noindent | ||
59 | or (b) by setting them in the environment | ||
60 | |||
61 | @example | ||
62 | $ export VAR=/x | ||
63 | @end example | ||
64 | |||
65 | @noindent | ||
66 | The configuration loader will give precedence to variables set under | ||
67 | @code{[path]}, though. | ||
68 | |||
69 | The utility @samp{gnunet-config}, which gets installed along with GNUnet, | ||
70 | serves to get and set configuration values without directly editing the | ||
71 | @file{.conf} file. The option @samp{-f} is particularly useful to resolve | ||
72 | filenames, when they use several levels of @code{$}-expanded variables. | ||
73 | See @samp{gnunet-config --help}. | ||
74 | |||
75 | Note that, in this stage of development, the file | ||
76 | @file{$HOME/.config/gnunet.conf} can contain sections for @strong{all} the | ||
77 | components. | ||
diff --git a/doc/handbook/chapters/contributing.texi b/doc/handbook/chapters/contributing.texi deleted file mode 100644 index f3beeef76..000000000 --- a/doc/handbook/chapters/contributing.texi +++ /dev/null | |||
@@ -1,117 +0,0 @@ | |||
1 | @node GNUnet Contributors Handbook | ||
2 | @chapter GNUnet Contributors Handbook | ||
3 | |||
4 | @menu | ||
5 | * Contributing to GNUnet:: | ||
6 | * Licenses of contributions:: | ||
7 | * Copyright Assignment:: | ||
8 | * Contributing to the Reference Manual:: | ||
9 | * Contributing testcases:: | ||
10 | @end menu | ||
11 | |||
12 | @node Contributing to GNUnet | ||
13 | @section Contributing to GNUnet | ||
14 | |||
15 | @cindex licenses | ||
16 | @cindex licenses of contributions | ||
17 | @node Licenses of contributions | ||
18 | @section Licenses of contributions | ||
19 | |||
20 | GNUnet is a @uref{https://www.gnu.org/, GNU} package. | ||
21 | All code contributions must thus be put under the | ||
22 | @uref{https://www.gnu.org/licenses/agpl.html, GNU Affero Public License (AGPL)}. | ||
23 | All documentation should be put under FSF approved licenses | ||
24 | (see @uref{https://www.gnu.org/copyleft/fdl.html, fdl}). | ||
25 | |||
26 | By submitting documentation, translations, and other content to GNUnet | ||
27 | you automatically grant the right to publish code under the | ||
28 | GNU Public License and documentation under either or both the | ||
29 | GNU Public License or the GNU Free Documentation License. | ||
30 | When contributing to the GNUnet project, GNU standards and the | ||
31 | @uref{https://www.gnu.org/philosophy/philosophy.html, GNU philosophy} | ||
32 | should be adhered to. | ||
33 | |||
34 | @cindex copyright assignment | ||
35 | @node Copyright Assignment | ||
36 | @section Copyright Assignment | ||
37 | We require a formal copyright assignment for GNUnet contributors | ||
38 | to GNUnet e.V.; nevertheless, we do allow pseudonymous contributions. | ||
39 | By signing the copyright agreement and submitting your code (or | ||
40 | documentation) to us, you agree to share the rights to your code | ||
41 | with GNUnet e.V.; GNUnet e.V. receives non-exclusive ownership | ||
42 | rights, and in particular is allowed to dual-license the code. You | ||
43 | retain non-exclusive rights to your contributions, so you can also | ||
44 | share your contributions freely with other projects. | ||
45 | |||
46 | GNUnet e.V. will publish all accepted contributions under the AGPLv3 | ||
47 | or any later version. The association may decide to publish | ||
48 | contributions under additional licenses (dual-licensing). | ||
49 | |||
50 | We do not intentionally remove your name from your contributions; | ||
51 | however, due to extensive editing it is not always trivial to | ||
52 | attribute contributors properly. If you find that you significantly | ||
53 | contributed to a file (or the project as a whole) and are not listed | ||
54 | in the respective authors file or section, please do let us know. | ||
55 | |||
56 | @node Contributing to the Reference Manual | ||
57 | @section Contributing to the Reference Manual | ||
58 | |||
59 | @itemize @bullet | ||
60 | |||
61 | @item When writing documentation, please use | ||
62 | @uref{https://en.wikipedia.org/wiki/Singular_they, gender-neutral wording} | ||
63 | when referring to people, such as singular “they”, “their”, “them”, and so | ||
64 | forth. | ||
65 | |||
66 | @item Keep line length below 74 characters, except for URLs. | ||
67 | URLs break in the PDF output when they contain linebreaks. | ||
68 | |||
69 | @item Do not use tab characters (see chapter 2.1 texinfo manual) | ||
70 | |||
71 | @item Write texts in the third person perspective. | ||
72 | |||
73 | @c FIXME: This is questionable, it feels like bike shed painging to do | ||
74 | @c this for several k lines. It only helps to jump between sentences in | ||
75 | @c editors afaik. | ||
76 | @c @item Use 2 spaces between sentences, so instead of: | ||
77 | |||
78 | @c @example | ||
79 | @c We do this and the other thing. This is done by foo. | ||
80 | @c @end example | ||
81 | |||
82 | @c Write: | ||
83 | |||
84 | @c @example | ||
85 | @c We do this and the other thing. This is done by foo. | ||
86 | @c @end example | ||
87 | |||
88 | @end itemize | ||
89 | |||
90 | @node Contributing testcases | ||
91 | @section Contributing testcases | ||
92 | |||
93 | In the core of gnunet, we restrict new testcases to a small subset | ||
94 | of languages, in order of preference: | ||
95 | @enumerate | ||
96 | @item C | ||
97 | @item Portable Shell Scripts | ||
98 | @item Python (3.7 or later) | ||
99 | @end enumerate | ||
100 | |||
101 | We welcome efforts to remove our existing python-2.7 scripts to | ||
102 | replace them either with portable shell scripts or, | ||
103 | at your choice, python-3.7+. | ||
104 | |||
105 | If you contribute new python based testcases, we advise you to | ||
106 | not repeat our past misfortunes and write the tests in a standard | ||
107 | test framework like for example pytest. | ||
108 | |||
109 | For writing portable shell scripts, these tools are useful: | ||
110 | @uref{https://github.com/koalaman/shellcheck, Shellcheck}, | ||
111 | @uref{https://salsa.debian.org/debian/devscripts/blob/master/scripts/checkbashisms.pl, checkbashisms}, | ||
112 | @uref{http://www.etalabs.net/sh_tricks.html}, | ||
113 | @uref{https://wiki.ubuntu.com/DashAsBinSh}, | ||
114 | and @uref{https://mywiki.wooledge.org/Bashism} | ||
115 | |||
116 | @c You could also run "bin/check_shell_script" (which we still have | ||
117 | @c to write). | ||
diff --git a/doc/handbook/chapters/developer.texi b/doc/handbook/chapters/developer.texi deleted file mode 100644 index c8905f2c1..000000000 --- a/doc/handbook/chapters/developer.texi +++ /dev/null | |||
@@ -1,10154 +0,0 @@ | |||
1 | @c *********************************************************************** | ||
2 | @node GNUnet Developer Handbook | ||
3 | @chapter GNUnet Developer Handbook | ||
4 | |||
5 | This book is intended to be an introduction for programmers that want to | ||
6 | extend the GNUnet framework. GNUnet is more than a simple peer-to-peer | ||
7 | application. | ||
8 | |||
9 | For developers, GNUnet is: | ||
10 | |||
11 | @itemize @bullet | ||
12 | @item developed by a community that believes in the GNU philosophy | ||
13 | @item Free Software (Free as in Freedom), licensed under the | ||
14 | GNU Affero General Public License | ||
15 | (@uref{https://www.gnu.org/licenses/licenses.html#AGPL}) | ||
16 | @item A set of standards, including coding conventions and | ||
17 | architectural rules | ||
18 | @item A set of layered protocols, both specifying the communication | ||
19 | between peers as well as the communication between components | ||
20 | of a single peer | ||
21 | @item A set of libraries with well-defined APIs suitable for | ||
22 | writing extensions | ||
23 | @end itemize | ||
24 | |||
25 | In particular, the architecture specifies that a peer consists of many | ||
26 | processes communicating via protocols. Processes can be written in almost | ||
27 | any language. | ||
28 | @code{C}, @code{Java} and @code{Guile} APIs exist for accessing existing | ||
29 | services and for writing extensions. | ||
30 | It is possible to write extensions in other languages by | ||
31 | implementing the necessary IPC protocols. | ||
32 | |||
33 | GNUnet can be extended and improved along many possible dimensions, and | ||
34 | anyone interested in Free Software and Freedom-enhancing Networking is | ||
35 | welcome to join the effort. This Developer Handbook attempts to provide | ||
36 | an initial introduction to some of the key design choices and central | ||
37 | components of the system. | ||
38 | This part of the GNUnet documentation is far from complete, | ||
39 | and we welcome informed contributions, be it in the form of | ||
40 | new chapters, sections or insightful comments. | ||
41 | |||
42 | @menu | ||
43 | * Developer Introduction:: | ||
44 | * Internal dependencies:: | ||
45 | * Code overview:: | ||
46 | * System Architecture:: | ||
47 | * Subsystem stability:: | ||
48 | * Naming conventions and coding style guide:: | ||
49 | * Build-system:: | ||
50 | * Developing extensions for GNUnet using the gnunet-ext template:: | ||
51 | * Writing testcases:: | ||
52 | * Building GNUnet and its dependencies:: | ||
53 | * TESTING library:: | ||
54 | * Performance regression analysis with Gauger:: | ||
55 | * TESTBED Subsystem:: | ||
56 | * libgnunetutil:: | ||
57 | * Automatic Restart Manager (ARM):: | ||
58 | * TRANSPORT Subsystem:: | ||
59 | * NAT library:: | ||
60 | * Distance-Vector plugin:: | ||
61 | * SMTP plugin:: | ||
62 | * Bluetooth plugin:: | ||
63 | * WLAN plugin:: | ||
64 | * ATS Subsystem:: | ||
65 | * CORE Subsystem:: | ||
66 | * CADET Subsystem:: | ||
67 | * NSE Subsystem:: | ||
68 | * HOSTLIST Subsystem:: | ||
69 | * IDENTITY Subsystem:: | ||
70 | * NAMESTORE Subsystem:: | ||
71 | * PEERINFO Subsystem:: | ||
72 | * PEERSTORE Subsystem:: | ||
73 | * SET Subsystem:: | ||
74 | * SETI Subsystem:: | ||
75 | * SETU Subsystem:: | ||
76 | * STATISTICS Subsystem:: | ||
77 | * Distributed Hash Table (DHT):: | ||
78 | * GNU Name System (GNS):: | ||
79 | * GNS Namecache:: | ||
80 | * REVOCATION Subsystem:: | ||
81 | * File-sharing (FS) Subsystem:: | ||
82 | * REGEX Subsystem:: | ||
83 | * REST Subsystem:: | ||
84 | * RPS Subsystem:: | ||
85 | * TRANSPORT-NG Subsystem:: | ||
86 | * MESSENGER Subsystem:: | ||
87 | @end menu | ||
88 | |||
89 | @node Developer Introduction | ||
90 | @section Developer Introduction | ||
91 | |||
92 | This Developer Handbook is intended as first introduction to GNUnet for | ||
93 | new developers that want to extend the GNUnet framework. After the | ||
94 | introduction, each of the GNUnet subsystems (directories in the | ||
95 | @file{src/} tree) is (supposed to be) covered in its own chapter. In | ||
96 | addition to this documentation, GNUnet developers should be aware of the | ||
97 | services available on the GNUnet server to them. | ||
98 | |||
99 | New developers can have a look a the @uref{https://docs.gnunet.org/tutorial/tutorial.html, GNUnet C tutorial}. | ||
100 | |||
101 | @c ** FIXME: Link to files in source, not online. | ||
102 | @c ** FIXME: Where is the Java tutorial? | ||
103 | |||
104 | In addition to the GNUnet Reference Documentation you are reading, | ||
105 | the GNUnet server at @uref{https://gnunet.org} contains | ||
106 | various resources for GNUnet developers and those | ||
107 | who aspire to become regular contributors. | ||
108 | They are all conveniently reachable via the "Developer" | ||
109 | entry in the navigation menu. Some additional tools (such as continuous | ||
110 | integration) require a special developer access to perform certain | ||
111 | operations. If you want (or require) access, you should contact | ||
112 | GNUnet's maintainers. | ||
113 | |||
114 | @c FIXME: A good part of this belongs on the website or should be | ||
115 | @c extended in subsections explaining usage of this. A simple list | ||
116 | @c is just taking space people have to read. | ||
117 | The developer services on the GNUnet project infrastructure are: | ||
118 | |||
119 | @itemize @bullet | ||
120 | |||
121 | @item The version control system (git) keeps our code and enables | ||
122 | distributed development. | ||
123 | It is publicly accessible at @uref{https://git.gnunet.org/}. | ||
124 | Only developers with write access can commit code, everyone else is | ||
125 | encouraged to submit patches to the GNUnet-developers mailinglist: | ||
126 | @uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, https://lists.gnu.org/mailman/listinfo/gnunet-developers} | ||
127 | |||
128 | @item The bugtracking system (Mantis). | ||
129 | We use it to track feature requests, open bug reports and their | ||
130 | resolutions. | ||
131 | It can be accessed at | ||
132 | @uref{https://bugs.gnunet.org/, https://bugs.gnunet.org/}. | ||
133 | Anyone can report bugs. | ||
134 | |||
135 | @item Continuous integration (Buildbot). | ||
136 | Used to build gnunet and its websites upon new commits. | ||
137 | It can be accessed at | ||
138 | @uref{https://buildbot.gnunet.org/, https://buildbot.gnunet.org/}. | ||
139 | Anyone can see the builds. | ||
140 | |||
141 | @item Regularly we make use of static analysis tools. | ||
142 | Note that not everything that is flagged by the | ||
143 | analysis is a bug, sometimes even good code can be marked as possibly | ||
144 | problematic. Nevertheless, developers are encouraged to at least be | ||
145 | aware of all issues in their code that are listed. | ||
146 | |||
147 | @c @item We use Gauger for automatic performance regression visualization. | ||
148 | @c FIXME: LINK! | ||
149 | @c Details on how to use Gauger are here. | ||
150 | |||
151 | @end itemize | ||
152 | |||
153 | |||
154 | |||
155 | @c *********************************************************************** | ||
156 | @menu | ||
157 | * Project overview:: | ||
158 | @end menu | ||
159 | |||
160 | @node Project overview | ||
161 | @subsection Project overview | ||
162 | |||
163 | The GNUnet project consists at this point of several sub-projects. This | ||
164 | section is supposed to give an initial overview about the various | ||
165 | sub-projects. Note that this description also lists projects that are far | ||
166 | from complete, including even those that have literally not a single line | ||
167 | of code in them yet. | ||
168 | |||
169 | GNUnet sub-projects in order of likely relevance are currently: | ||
170 | |||
171 | @table @asis | ||
172 | |||
173 | @item @command{gnunet} | ||
174 | Core of the P2P framework, including file-sharing, VPN and | ||
175 | chat applications; this is what the Developer Handbook covers mostly | ||
176 | @item @command{gnunet-gtk} | ||
177 | Gtk+-based user interfaces, including: | ||
178 | |||
179 | @itemize @bullet | ||
180 | @item @command{gnunet-fs-gtk} (file-sharing), | ||
181 | @item @command{gnunet-statistics-gtk} (statistics over time), | ||
182 | @item @command{gnunet-peerinfo-gtk} | ||
183 | (information about current connections and known peers), | ||
184 | @item @command{gnunet-namestore-gtk} (GNS record editor), | ||
185 | @item @command{gnunet-conversation-gtk} (voice chat GUI) and | ||
186 | @item @command{gnunet-setup} (setup tool for "everything") | ||
187 | @end itemize | ||
188 | |||
189 | @item @command{gnunet-fuse} | ||
190 | Mounting directories shared via GNUnet's file-sharing | ||
191 | on GNU/Linux distributions | ||
192 | @item @command{gnunet-update} | ||
193 | Installation and update tool | ||
194 | @item @command{gnunet-ext} | ||
195 | Template for starting 'external' GNUnet projects | ||
196 | @item @command{gnunet-java} | ||
197 | Java APIs for writing GNUnet services and applications | ||
198 | @item @command{gnunet-java-ext} | ||
199 | @item @command{eclectic} | ||
200 | Code to run GNUnet nodes on testbeds for research, development, | ||
201 | testing and evaluation | ||
202 | @c ** FIXME: Solve the status and location of gnunet-qt | ||
203 | @item @command{gnunet-qt} | ||
204 | Qt-based GNUnet GUI (is it deprecated?) | ||
205 | @item @command{gnunet-cocoa} | ||
206 | cocoa-based GNUnet GUI (is it deprecated?) | ||
207 | @item @command{gnunet-guile} | ||
208 | Guile bindings for GNUnet | ||
209 | @item @command{gnunet-python} | ||
210 | Python bindings for GNUnet | ||
211 | |||
212 | @end table | ||
213 | |||
214 | We are also working on various supporting libraries and tools: | ||
215 | @c ** FIXME: What about gauger, and what about libmwmodem? | ||
216 | |||
217 | @table @asis | ||
218 | @item @command{libextractor} | ||
219 | GNU libextractor (meta data extraction) | ||
220 | @item @command{libmicrohttpd} | ||
221 | GNU libmicrohttpd (embedded HTTP(S) server library) | ||
222 | @item @command{gauger} | ||
223 | Tool for performance regression analysis | ||
224 | @item @command{monkey} | ||
225 | Tool for automated debugging of distributed systems | ||
226 | @item @command{libmwmodem} | ||
227 | Library for accessing satellite connection quality reports | ||
228 | @item @command{libgnurl} | ||
229 | gnURL (feature-restricted variant of cURL/libcurl) | ||
230 | @item @command{www} | ||
231 | the gnunet.org website (Jinja2 based) | ||
232 | @item @command{bibliography} | ||
233 | Our collected bibliography, papers, references, and so forth | ||
234 | @item @command{gnunet-videos-} | ||
235 | Videos about and around GNUnet activities | ||
236 | @end table | ||
237 | |||
238 | Finally, there are various external projects (see links for a list of | ||
239 | those that have a public website) which build on top of the GNUnet | ||
240 | framework. | ||
241 | |||
242 | @c *********************************************************************** | ||
243 | @node Internal dependencies | ||
244 | @section Internal dependencies | ||
245 | |||
246 | This section tries to give an overview of what processes a typical GNUnet | ||
247 | peer running a particular application would consist of. All of the | ||
248 | processes listed here should be automatically started by | ||
249 | @command{gnunet-arm -s}. | ||
250 | The list is given as a rough first guide to users for failure diagnostics. | ||
251 | Ideally, end-users should never have to worry about these internal | ||
252 | dependencies. | ||
253 | |||
254 | In terms of internal dependencies, a minimum file-sharing system consists | ||
255 | of the following GNUnet processes (in order of dependency): | ||
256 | |||
257 | @itemize @bullet | ||
258 | @item gnunet-service-arm | ||
259 | @item gnunet-service-resolver (required by all) | ||
260 | @item gnunet-service-statistics (required by all) | ||
261 | @item gnunet-service-peerinfo | ||
262 | @item gnunet-service-transport (requires peerinfo) | ||
263 | @item gnunet-service-core (requires transport) | ||
264 | @item gnunet-daemon-hostlist (requires core) | ||
265 | @item gnunet-daemon-topology (requires hostlist, peerinfo) | ||
266 | @item gnunet-service-datastore | ||
267 | @item gnunet-service-dht (requires core) | ||
268 | @item gnunet-service-identity | ||
269 | @item gnunet-service-fs (requires identity, mesh, dht, datastore, core) | ||
270 | @end itemize | ||
271 | |||
272 | @noindent | ||
273 | A minimum VPN system consists of the following GNUnet processes (in | ||
274 | order of dependency): | ||
275 | |||
276 | @itemize @bullet | ||
277 | @item gnunet-service-arm | ||
278 | @item gnunet-service-resolver (required by all) | ||
279 | @item gnunet-service-statistics (required by all) | ||
280 | @item gnunet-service-peerinfo | ||
281 | @item gnunet-service-transport (requires peerinfo) | ||
282 | @item gnunet-service-core (requires transport) | ||
283 | @item gnunet-daemon-hostlist (requires core) | ||
284 | @item gnunet-service-dht (requires core) | ||
285 | @item gnunet-service-mesh (requires dht, core) | ||
286 | @item gnunet-service-dns (requires dht) | ||
287 | @item gnunet-service-regex (requires dht) | ||
288 | @item gnunet-service-vpn (requires regex, dns, mesh, dht) | ||
289 | @end itemize | ||
290 | |||
291 | @noindent | ||
292 | A minimum GNS system consists of the following GNUnet processes (in | ||
293 | order of dependency): | ||
294 | |||
295 | @itemize @bullet | ||
296 | @item gnunet-service-arm | ||
297 | @item gnunet-service-resolver (required by all) | ||
298 | @item gnunet-service-statistics (required by all) | ||
299 | @item gnunet-service-peerinfo | ||
300 | @item gnunet-service-transport (requires peerinfo) | ||
301 | @item gnunet-service-core (requires transport) | ||
302 | @item gnunet-daemon-hostlist (requires core) | ||
303 | @item gnunet-service-dht (requires core) | ||
304 | @item gnunet-service-mesh (requires dht, core) | ||
305 | @item gnunet-service-dns (requires dht) | ||
306 | @item gnunet-service-regex (requires dht) | ||
307 | @item gnunet-service-vpn (requires regex, dns, mesh, dht) | ||
308 | @item gnunet-service-identity | ||
309 | @item gnunet-service-namestore (requires identity) | ||
310 | @item gnunet-service-gns (requires vpn, dns, dht, namestore, identity) | ||
311 | @end itemize | ||
312 | |||
313 | @c *********************************************************************** | ||
314 | @node Code overview | ||
315 | @section Code overview | ||
316 | |||
317 | This section gives a brief overview of the GNUnet source code. | ||
318 | Specifically, we sketch the function of each of the subdirectories in | ||
319 | the @file{gnunet/src/} directory. The order given is roughly bottom-up | ||
320 | (in terms of the layers of the system). | ||
321 | |||
322 | @table @asis | ||
323 | @item @file{util/} --- libgnunetutil | ||
324 | Library with general utility functions, all | ||
325 | GNUnet binaries link against this library. Anything from memory | ||
326 | allocation and data structures to cryptography and inter-process | ||
327 | communication. The goal is to provide an OS-independent interface and | ||
328 | more 'secure' or convenient implementations of commonly used primitives. | ||
329 | The API is spread over more than a dozen headers, developers should study | ||
330 | those closely to avoid duplicating existing functions. | ||
331 | @pxref{libgnunetutil}. | ||
332 | @item @file{hello/} --- libgnunethello | ||
333 | HELLO messages are used to | ||
334 | describe under which addresses a peer can be reached (for example, | ||
335 | protocol, IP, port). This library manages parsing and generating of HELLO | ||
336 | messages. | ||
337 | @item @file{block/} --- libgnunetblock | ||
338 | The DHT and other components of GNUnet | ||
339 | store information in units called 'blocks'. Each block has a type and the | ||
340 | type defines a particular format and how that binary format is to be | ||
341 | linked to a hash code (the key for the DHT and for databases). The block | ||
342 | library is a wrapper around block plugins which provide the necessary | ||
343 | functions for each block type. | ||
344 | @item @file{statistics/} --- statistics service | ||
345 | The statistics service enables associating | ||
346 | values (of type uint64_t) with a component name and a string. The main | ||
347 | uses is debugging (counting events), performance tracking and user | ||
348 | entertainment (what did my peer do today?). | ||
349 | @item @file{arm/} --- Automatic Restart Manager (ARM) | ||
350 | The automatic-restart-manager (ARM) service | ||
351 | is the GNUnet master service. Its role is to start gnunet-services, to | ||
352 | re-start them when they crashed and finally to shut down the system when | ||
353 | requested. | ||
354 | @item @file{peerinfo/} --- peerinfo service | ||
355 | The peerinfo service keeps track of which peers are known | ||
356 | to the local peer and also tracks the validated addresses for each peer | ||
357 | (in the form of a HELLO message) for each of those peers. The peer is not | ||
358 | necessarily connected to all peers known to the peerinfo service. | ||
359 | Peerinfo provides persistent storage for peer identities --- peers are | ||
360 | not forgotten just because of a system restart. | ||
361 | @item @file{datacache/} --- libgnunetdatacache | ||
362 | The datacache library provides (temporary) block storage for the DHT. | ||
363 | Existing plugins can store blocks in Sqlite, Postgres or MySQL databases. | ||
364 | All data stored in the cache is lost when the peer is stopped or | ||
365 | restarted (datacache uses temporary tables). | ||
366 | @item @file{datastore/} --- datastore service | ||
367 | The datastore service stores file-sharing blocks in | ||
368 | databases for extended periods of time. In contrast to the datacache, data | ||
369 | is not lost when peers restart. However, quota restrictions may still | ||
370 | cause old, expired or low-priority data to be eventually discarded. | ||
371 | Existing plugins can store blocks in Sqlite, Postgres or MySQL databases. | ||
372 | @item @file{template/} --- service template | ||
373 | Template for writing a new service. Does nothing. | ||
374 | @item @file{ats/} --- Automatic Transport Selection | ||
375 | The automatic transport selection (ATS) service | ||
376 | is responsible for deciding which address (i.e. | ||
377 | which transport plugin) should be used for communication with other peers, | ||
378 | and at what bandwidth. | ||
379 | @item @file{nat/} --- libgnunetnat | ||
380 | Library that provides basic functions for NAT traversal. | ||
381 | The library supports NAT traversal with | ||
382 | manual hole-punching by the user, UPnP and ICMP-based autonomous NAT | ||
383 | traversal. The library also includes an API for testing if the current | ||
384 | configuration works and the @code{gnunet-nat-server} which provides an | ||
385 | external service to test the local configuration. | ||
386 | @item @file{fragmentation/} --- libgnunetfragmentation | ||
387 | Some transports (UDP and WLAN, mostly) have restrictions on the maximum | ||
388 | transfer unit (MTU) for packets. The fragmentation library can be used to | ||
389 | break larger packets into chunks of at most 1k and transmit the resulting | ||
390 | fragments reliably (with acknowledgment, retransmission, timeouts, | ||
391 | etc.). | ||
392 | @item @file{transport/} --- transport service | ||
393 | The transport service is responsible for managing the | ||
394 | basic P2P communication. It uses plugins to support P2P communication | ||
395 | over TCP, UDP, HTTP, HTTPS and other protocols. The transport service | ||
396 | validates peer addresses, enforces bandwidth restrictions, limits the | ||
397 | total number of connections and enforces connectivity restrictions (e.g. | ||
398 | friends-only). | ||
399 | @item @file{peerinfo-tool/} --- gnunet-peerinfo | ||
400 | This directory contains the gnunet-peerinfo binary which can be used to | ||
401 | inspect the peers and HELLOs known to the peerinfo service. | ||
402 | @item @file{core/} | ||
403 | The core service is responsible for establishing encrypted, authenticated | ||
404 | connections with other peers, encrypting and decrypting messages and | ||
405 | forwarding messages to higher-level services that are interested in them. | ||
406 | @item @file{testing/} --- libgnunettesting | ||
407 | The testing library allows starting (and stopping) peers | ||
408 | for writing testcases. | ||
409 | It also supports automatic generation of configurations for peers | ||
410 | ensuring that the ports and paths are disjoint. libgnunettesting is also | ||
411 | the foundation for the testbed service | ||
412 | @item @file{testbed/} --- testbed service | ||
413 | The testbed service is used for creating small or large scale deployments | ||
414 | of GNUnet peers for evaluation of protocols. | ||
415 | It facilitates peer deployments on multiple | ||
416 | hosts (for example, in a cluster) and establishing various network | ||
417 | topologies (both underlay and overlay). | ||
418 | @item @file{nse/} --- Network Size Estimation | ||
419 | The network size estimation (NSE) service | ||
420 | implements a protocol for (securely) estimating the current size of the | ||
421 | P2P network. | ||
422 | @item @file{dht/} --- distributed hash table | ||
423 | The distributed hash table (DHT) service provides a | ||
424 | distributed implementation of a hash table to store blocks under hash | ||
425 | keys in the P2P network. | ||
426 | @item @file{hostlist/} --- hostlist service | ||
427 | The hostlist service allows learning about | ||
428 | other peers in the network by downloading HELLO messages from an HTTP | ||
429 | server, can be configured to run such an HTTP server and also implements | ||
430 | a P2P protocol to advertise and automatically learn about other peers | ||
431 | that offer a public hostlist server. | ||
432 | @item @file{topology/} --- topology service | ||
433 | The topology service is responsible for | ||
434 | maintaining the mesh topology. It tries to maintain connections to friends | ||
435 | (depending on the configuration) and also tries to ensure that the peer | ||
436 | has a decent number of active connections at all times. If necessary, new | ||
437 | connections are added. All peers should run the topology service, | ||
438 | otherwise they may end up not being connected to any other peer (unless | ||
439 | some other service ensures that core establishes the required | ||
440 | connections). The topology service also tells the transport service which | ||
441 | connections are permitted (for friend-to-friend networking) | ||
442 | @item @file{fs/} --- file-sharing | ||
443 | The file-sharing (FS) service implements GNUnet's | ||
444 | file-sharing application. Both anonymous file-sharing (using gap) and | ||
445 | non-anonymous file-sharing (using dht) are supported. | ||
446 | @item @file{cadet/} --- cadet service | ||
447 | The CADET service provides a general-purpose routing abstraction to create | ||
448 | end-to-end encrypted tunnels in mesh networks. We wrote a paper | ||
449 | documenting key aspects of the design. | ||
450 | @item @file{tun/} --- libgnunettun | ||
451 | Library for building IPv4, IPv6 packets and creating | ||
452 | checksums for UDP, TCP and ICMP packets. The header | ||
453 | defines C structs for common Internet packet formats and in particular | ||
454 | structs for interacting with TUN (virtual network) interfaces. | ||
455 | @item @file{mysql/} --- libgnunetmysql | ||
456 | Library for creating and executing prepared MySQL | ||
457 | statements and to manage the connection to the MySQL database. | ||
458 | Essentially a lightweight wrapper for the interaction between GNUnet | ||
459 | components and libmysqlclient. | ||
460 | @item @file{dns/} | ||
461 | Service that allows intercepting and modifying DNS requests of | ||
462 | the local machine. Currently used for IPv4-IPv6 protocol translation | ||
463 | (DNS-ALG) as implemented by "pt/" and for the GNUnet naming system. The | ||
464 | service can also be configured to offer an exit service for DNS traffic. | ||
465 | @item @file{vpn/} --- VPN service | ||
466 | The virtual public network (VPN) service provides a virtual | ||
467 | tunnel interface (VTUN) for IP routing over GNUnet. | ||
468 | Needs some other peers to run an "exit" service to work. | ||
469 | Can be activated using the "gnunet-vpn" tool or integrated with DNS using | ||
470 | the "pt" daemon. | ||
471 | @item @file{exit/} | ||
472 | Daemon to allow traffic from the VPN to exit this | ||
473 | peer to the Internet or to specific IP-based services of the local peer. | ||
474 | Currently, an exit service can only be restricted to IPv4 or IPv6, not to | ||
475 | specific ports and or IP address ranges. If this is not acceptable, | ||
476 | additional firewall rules must be added manually. exit currently only | ||
477 | works for normal UDP, TCP and ICMP traffic; DNS queries need to leave the | ||
478 | system via a DNS service. | ||
479 | @item @file{pt/} | ||
480 | protocol translation daemon. This daemon enables 4-to-6, | ||
481 | 6-to-4, 4-over-6 or 6-over-4 transitions for the local system. It | ||
482 | essentially uses "DNS" to intercept DNS replies and then maps results to | ||
483 | those offered by the VPN, which then sends them using mesh to some daemon | ||
484 | offering an appropriate exit service. | ||
485 | @item @file{identity/} | ||
486 | Management of egos (alter egos) of a user; identities are | ||
487 | essentially named ECC private keys and used for zones in the GNU name | ||
488 | system and for namespaces in file-sharing, but might find other uses later | ||
489 | @item @file{revocation/} | ||
490 | Key revocation service, can be used to revoke the | ||
491 | private key of an identity if it has been compromised | ||
492 | @item @file{namecache/} | ||
493 | Cache for resolution results for the GNU name system; | ||
494 | data is encrypted and can be shared among users, | ||
495 | loss of the data should ideally only result in a | ||
496 | performance degradation (persistence not required) | ||
497 | @item @file{namestore/} | ||
498 | Database for the GNU name system with per-user private information, | ||
499 | persistence required | ||
500 | @item @file{gns/} | ||
501 | GNU name system, a GNU approach to DNS and PKI. | ||
502 | @item @file{dv/} | ||
503 | A plugin for distance-vector (DV)-based routing. | ||
504 | DV consists of a service and a transport plugin to provide peers | ||
505 | with the illusion of a direct P2P connection for connections | ||
506 | that use multiple (typically up to 3) hops in the actual underlay network. | ||
507 | @item @file{regex/} | ||
508 | Service for the (distributed) evaluation of regular expressions. | ||
509 | @item @file{scalarproduct/} | ||
510 | The scalar product service offers an API to perform a secure multiparty | ||
511 | computation which calculates a scalar product between two peers | ||
512 | without exposing the private input vectors of the peers to each other. | ||
513 | @item @file{consensus/} | ||
514 | The consensus service will allow a set of peers to agree | ||
515 | on a set of values via a distributed set union computation. | ||
516 | @item @file{reclaim/} | ||
517 | A decentralized personal data sharing service used to realize a decentralized | ||
518 | identity provider. Supports OpenID Connect. See also @uref{https://reclaim.gnunet.org}. | ||
519 | @item @file{rest/} | ||
520 | The rest API allows access to GNUnet services using RESTful interaction. | ||
521 | The services provide plugins that can exposed by the rest server. | ||
522 | @c FIXME: Where did this disappear to? | ||
523 | @c @item @file{experimentation/} | ||
524 | @c The experimentation daemon coordinates distributed | ||
525 | @c experimentation to evaluate transport and ATS properties. | ||
526 | @end table | ||
527 | |||
528 | @c *********************************************************************** | ||
529 | @node System Architecture | ||
530 | @section System Architecture | ||
531 | |||
532 | @c FIXME: For those irritated by the textflow, we are missing images here, | ||
533 | @c in the short term we should add them back, in the long term this should | ||
534 | @c work without images or have images with alt-text. | ||
535 | |||
536 | GNUnet developers like LEGOs. The blocks are indestructible, can be | ||
537 | stacked together to construct complex buildings and it is generally easy | ||
538 | to swap one block for a different one that has the same shape. GNUnet's | ||
539 | architecture is based on LEGOs: | ||
540 | |||
541 | @image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs upon IPC/network protocol provided by a service} | ||
542 | |||
543 | This chapter documents the GNUnet LEGO system, also known as GNUnet's | ||
544 | system architecture. | ||
545 | |||
546 | The most common GNUnet component is a service. Services offer an API (or | ||
547 | several, depending on what you count as "an API") which is implemented as | ||
548 | a library. The library communicates with the main process of the service | ||
549 | using a service-specific network protocol. The main process of the service | ||
550 | typically doesn't fully provide everything that is needed --- it has holes | ||
551 | to be filled by APIs to other services. | ||
552 | |||
553 | A special kind of component in GNUnet are user interfaces and daemons. | ||
554 | Like services, they have holes to be filled by APIs of other services. | ||
555 | Unlike services, daemons do not implement their own network protocol and | ||
556 | they have no API: | ||
557 | |||
558 | @image{images/daemon_lego_block,5in,,A daemon in GNUnet is a component that does not offer an API for others to build upon} | ||
559 | |||
560 | The GNUnet system provides a range of services, daemons and user | ||
561 | interfaces, which are then combined into a layered GNUnet instance (also | ||
562 | known as a peer). | ||
563 | |||
564 | @image{images/service_stack,5in,,A GNUnet peer consists of many layers of services} | ||
565 | |||
566 | Note that while it is generally possible to swap one service for another | ||
567 | compatible service, there is often only one implementation. However, | ||
568 | during development we often have a "new" version of a service in parallel | ||
569 | with an "old" version. While the "new" version is not working, developers | ||
570 | working on other parts of the service can continue their development by | ||
571 | simply using the "old" service. Alternative design ideas can also be | ||
572 | easily investigated by swapping out individual components. This is | ||
573 | typically achieved by simply changing the name of the "BINARY" in the | ||
574 | respective configuration section. | ||
575 | |||
576 | Key properties of GNUnet services are that they must be separate | ||
577 | processes and that they must protect themselves by applying tight error | ||
578 | checking against the network protocol they implement (thereby achieving a | ||
579 | certain degree of robustness). | ||
580 | |||
581 | On the other hand, the APIs are implemented to tolerate failures of the | ||
582 | service, isolating their host process from errors by the service. If the | ||
583 | service process crashes, other services and daemons around it should not | ||
584 | also fail, but instead wait for the service process to be restarted by | ||
585 | ARM. | ||
586 | |||
587 | |||
588 | @c *********************************************************************** | ||
589 | @node Subsystem stability | ||
590 | @section Subsystem stability | ||
591 | |||
592 | This section documents the current stability of the various GNUnet | ||
593 | subsystems. Stability here describes the expected degree of compatibility | ||
594 | with future versions of GNUnet. For each subsystem we distinguish between | ||
595 | compatibility on the P2P network level (communication protocol between | ||
596 | peers), the IPC level (communication between the service and the service | ||
597 | library) and the API level (stability of the API). P2P compatibility is | ||
598 | relevant in terms of which applications are likely going to be able to | ||
599 | communicate with future versions of the network. IPC communication is | ||
600 | relevant for the implementation of language bindings that re-implement the | ||
601 | IPC messages. Finally, API compatibility is relevant to developers that | ||
602 | hope to be able to avoid changes to applications build on top of the APIs | ||
603 | of the framework. | ||
604 | |||
605 | The following table summarizes our current view of the stability of the | ||
606 | respective protocols or APIs: | ||
607 | |||
608 | @multitable @columnfractions .20 .20 .20 .20 | ||
609 | @headitem Subsystem @tab P2P @tab IPC @tab C API | ||
610 | @item util @tab n/a @tab n/a @tab stable | ||
611 | @item arm @tab n/a @tab stable @tab stable | ||
612 | @item ats @tab n/a @tab unstable @tab testing | ||
613 | @item block @tab n/a @tab n/a @tab stable | ||
614 | @item cadet @tab testing @tab testing @tab testing | ||
615 | @item consensus @tab experimental @tab experimental @tab experimental | ||
616 | @item core @tab stable @tab stable @tab stable | ||
617 | @item datacache @tab n/a @tab n/a @tab stable | ||
618 | @item datastore @tab n/a @tab stable @tab stable | ||
619 | @item dht @tab stable @tab stable @tab stable | ||
620 | @item dns @tab stable @tab stable @tab stable | ||
621 | @item dv @tab testing @tab testing @tab n/a | ||
622 | @item exit @tab testing @tab n/a @tab n/a | ||
623 | @item fragmentation @tab stable @tab n/a @tab stable | ||
624 | @item fs @tab stable @tab stable @tab stable | ||
625 | @item gns @tab stable @tab stable @tab stable | ||
626 | @item hello @tab n/a @tab n/a @tab testing | ||
627 | @item hostlist @tab stable @tab stable @tab n/a | ||
628 | @item identity @tab stable @tab stable @tab n/a | ||
629 | @item multicast @tab experimental @tab experimental @tab experimental | ||
630 | @item mysql @tab stable @tab n/a @tab stable | ||
631 | @item namestore @tab n/a @tab stable @tab stable | ||
632 | @item nat @tab n/a @tab n/a @tab stable | ||
633 | @item nse @tab stable @tab stable @tab stable | ||
634 | @item peerinfo @tab n/a @tab stable @tab stable | ||
635 | @item psyc @tab experimental @tab experimental @tab experimental | ||
636 | @item pt @tab n/a @tab n/a @tab n/a | ||
637 | @item regex @tab stable @tab stable @tab stable | ||
638 | @item revocation @tab stable @tab stable @tab stable | ||
639 | @item social @tab experimental @tab experimental @tab experimental | ||
640 | @item statistics @tab n/a @tab stable @tab stable | ||
641 | @item testbed @tab n/a @tab testing @tab testing | ||
642 | @item testing @tab n/a @tab n/a @tab testing | ||
643 | @item topology @tab n/a @tab n/a @tab n/a | ||
644 | @item transport @tab experimental @tab experimental @tab experimental | ||
645 | @item tun @tab n/a @tab n/a @tab stable | ||
646 | @item vpn @tab testing @tab n/a @tab n/a | ||
647 | @end multitable | ||
648 | |||
649 | Here is a rough explanation of the values: | ||
650 | |||
651 | @table @samp | ||
652 | @item stable | ||
653 | No incompatible changes are planned at this time; for IPC/APIs, if | ||
654 | there are incompatible changes, they will be minor and might only require | ||
655 | minimal changes to existing code; for P2P, changes will be avoided if at | ||
656 | all possible for the 0.10.x-series | ||
657 | |||
658 | @item testing | ||
659 | No incompatible changes are | ||
660 | planned at this time, but the code is still known to be in flux; so while | ||
661 | we have no concrete plans, our expectation is that there will still be | ||
662 | minor modifications; for P2P, changes will likely be extensions that | ||
663 | should not break existing code | ||
664 | |||
665 | @item unstable | ||
666 | Changes are planned and will happen; however, they | ||
667 | will not be totally radical and the result should still resemble what is | ||
668 | there now; nevertheless, anticipated changes will break protocol/API | ||
669 | compatibility | ||
670 | |||
671 | @item experimental | ||
672 | Changes are planned and the result may look nothing like | ||
673 | what the API/protocol looks like today | ||
674 | |||
675 | @item unknown | ||
676 | Someone should think about where this subsystem headed | ||
677 | |||
678 | @item n/a | ||
679 | This subsystem does not have an API/IPC-protocol/P2P-protocol | ||
680 | @end table | ||
681 | |||
682 | @c *********************************************************************** | ||
683 | @node Naming conventions and coding style guide | ||
684 | @section Naming conventions and coding style guide | ||
685 | |||
686 | Here you can find some rules to help you write code for GNUnet. | ||
687 | |||
688 | @c *********************************************************************** | ||
689 | @menu | ||
690 | * Naming conventions:: | ||
691 | * Coding style:: | ||
692 | * Continuous integration:: | ||
693 | * Commit messages and developer branches:: | ||
694 | @end menu | ||
695 | |||
696 | @node Naming conventions | ||
697 | @subsection Naming conventions | ||
698 | |||
699 | |||
700 | @c *********************************************************************** | ||
701 | @menu | ||
702 | * include files:: | ||
703 | * binaries:: | ||
704 | * logging:: | ||
705 | * configuration:: | ||
706 | * exported symbols:: | ||
707 | * private (library-internal) symbols (including structs and macros):: | ||
708 | * testcases:: | ||
709 | * performance tests:: | ||
710 | * src/ directories:: | ||
711 | @end menu | ||
712 | |||
713 | @node include files | ||
714 | @subsubsection include files | ||
715 | |||
716 | @itemize @bullet | ||
717 | @item _lib: library without need for a process | ||
718 | @item _service: library that needs a service process | ||
719 | @item _plugin: plugin definition | ||
720 | @item _protocol: structs used in network protocol | ||
721 | @item exceptions: | ||
722 | @itemize @bullet | ||
723 | @item gnunet_config.h --- generated | ||
724 | @item platform.h --- first included | ||
725 | @item gnunet_common.h --- fundamental routines | ||
726 | @item gnunet_directories.h --- generated | ||
727 | @item gettext.h --- external library | ||
728 | @end itemize | ||
729 | @end itemize | ||
730 | |||
731 | @c *********************************************************************** | ||
732 | @node binaries | ||
733 | @subsubsection binaries | ||
734 | |||
735 | @itemize @bullet | ||
736 | @item gnunet-service-xxx: service process (has listen socket) | ||
737 | @item gnunet-daemon-xxx: daemon process (no listen socket) | ||
738 | @item gnunet-helper-xxx[-yyy]: SUID helper for module xxx | ||
739 | @item gnunet-yyy: command-line tool for end-users | ||
740 | @item libgnunet_plugin_xxx_yyy.so: plugin for API xxx | ||
741 | @item libgnunetxxx.so: library for API xxx | ||
742 | @end itemize | ||
743 | |||
744 | @c *********************************************************************** | ||
745 | @node logging | ||
746 | @subsubsection logging | ||
747 | |||
748 | @itemize @bullet | ||
749 | @item services and daemons use their directory name in | ||
750 | @code{GNUNET_log_setup} (e.g. 'core') and log using | ||
751 | plain 'GNUNET_log'. | ||
752 | @item command-line tools use their full name in | ||
753 | @code{GNUNET_log_setup} (e.g. 'gnunet-publish') and log using | ||
754 | plain 'GNUNET_log'. | ||
755 | @item service access libraries log using | ||
756 | '@code{GNUNET_log_from}' and use '@code{DIRNAME-api}' for the | ||
757 | component (e.g. 'core-api') | ||
758 | @item pure libraries (without associated service) use | ||
759 | '@code{GNUNET_log_from}' with the component set to their | ||
760 | library name (without lib or '@file{.so}'), | ||
761 | which should also be their directory name (e.g. '@file{nat}') | ||
762 | @item plugins should use '@code{GNUNET_log_from}' | ||
763 | with the directory name and the plugin name combined to produce | ||
764 | the component name (e.g. 'transport-tcp'). | ||
765 | @item logging should be unified per-file by defining a | ||
766 | @code{LOG} macro with the appropriate arguments, | ||
767 | along these lines: | ||
768 | |||
769 | @example | ||
770 | #define LOG(kind,...) | ||
771 | GNUNET_log_from (kind, "example-api",__VA_ARGS__) | ||
772 | @end example | ||
773 | |||
774 | @end itemize | ||
775 | |||
776 | @c *********************************************************************** | ||
777 | @node configuration | ||
778 | @subsubsection configuration | ||
779 | |||
780 | @itemize @bullet | ||
781 | @item paths (that are substituted in all filenames) are in PATHS | ||
782 | (have as few as possible) | ||
783 | @item all options for a particular module (@file{src/MODULE}) | ||
784 | are under @code{[MODULE]} | ||
785 | @item options for a plugin of a module | ||
786 | are under @code{[MODULE-PLUGINNAME]} | ||
787 | @end itemize | ||
788 | |||
789 | @c *********************************************************************** | ||
790 | @node exported symbols | ||
791 | @subsubsection exported symbols | ||
792 | |||
793 | @itemize @bullet | ||
794 | @item must start with @code{GNUNET_modulename_} and be defined in | ||
795 | @file{modulename.c} | ||
796 | @item exceptions: those defined in @file{gnunet_common.h} | ||
797 | @end itemize | ||
798 | |||
799 | @c *********************************************************************** | ||
800 | @node private (library-internal) symbols (including structs and macros) | ||
801 | @subsubsection private (library-internal) symbols (including structs and macros) | ||
802 | |||
803 | @itemize @bullet | ||
804 | @item must NOT start with any prefix | ||
805 | @item must not be exported in a way that linkers could use them or@ other | ||
806 | libraries might see them via headers; they must be either | ||
807 | declared/defined in C source files or in headers that are in the | ||
808 | respective directory under @file{src/modulename/} and NEVER be declared | ||
809 | in @file{src/include/}. | ||
810 | @end itemize | ||
811 | |||
812 | @node testcases | ||
813 | @subsubsection testcases | ||
814 | |||
815 | @itemize @bullet | ||
816 | @item must be called @file{test_module-under-test_case-description.c} | ||
817 | @item "case-description" maybe omitted if there is only one test | ||
818 | @end itemize | ||
819 | |||
820 | @c *********************************************************************** | ||
821 | @node performance tests | ||
822 | @subsubsection performance tests | ||
823 | |||
824 | @itemize @bullet | ||
825 | @item must be called @file{perf_module-under-test_case-description.c} | ||
826 | @item "case-description" maybe omitted if there is only one performance | ||
827 | test | ||
828 | @item Must only be run if @code{HAVE_BENCHMARKS} is satisfied | ||
829 | @end itemize | ||
830 | |||
831 | @c *********************************************************************** | ||
832 | @node src/ directories | ||
833 | @subsubsection src/ directories | ||
834 | |||
835 | @itemize @bullet | ||
836 | @item gnunet-NAME: end-user applications (like gnunet-search or gnunet-arm) | ||
837 | @item gnunet-service-NAME: service processes with accessor library (e.g. | ||
838 | gnunet-service-arm) | ||
839 | @item libgnunetNAME: accessor library (_service.h-header) or standalone | ||
840 | library (_lib.h-header) | ||
841 | @item gnunet-daemon-NAME: daemon process without accessor library (e.g. | ||
842 | gnunet-daemon-hostlist) and no GNUnet management port | ||
843 | @item libgnunet_plugin_DIR_NAME: loadable plugins (e.g. | ||
844 | libgnunet_plugin_transport_tcp) | ||
845 | @end itemize | ||
846 | |||
847 | @cindex Coding style | ||
848 | @node Coding style | ||
849 | @subsection Coding style | ||
850 | |||
851 | @c XXX: Adjust examples to GNU Standards! | ||
852 | @itemize @bullet | ||
853 | @item We follow the GNU Coding Standards (@pxref{Top, The GNU Coding Standards,, standards, The GNU Coding Standards}); | ||
854 | @item Indentation is done with spaces, two per level, no tabs; specific (incomplete!) indentation rules are provided in an @code{uncrustify} configuration file (in ``contrib/``) and enforced by Git hooks; | ||
855 | @item C99 struct initialization is fine and generally encouraged (but not required); | ||
856 | @item As in all good C code, we care about symbol space pollution and thus use @code{static} to limit the scope where possible, even in the compilation unit that contains @code{main}; | ||
857 | @item declare only one variable per line, for example: | ||
858 | |||
859 | @noindent | ||
860 | instead of | ||
861 | |||
862 | @example | ||
863 | int i,j; | ||
864 | @end example | ||
865 | |||
866 | @noindent | ||
867 | write: | ||
868 | |||
869 | @example | ||
870 | int i; | ||
871 | int j; | ||
872 | @end example | ||
873 | |||
874 | @c TODO: include actual example from a file in source | ||
875 | |||
876 | @noindent | ||
877 | This helps keep diffs small and forces developers to think precisely about | ||
878 | the type of every variable. | ||
879 | Note that @code{char *} is different from @code{const char*} and | ||
880 | @code{int} is different from @code{unsigned int} or @code{uint32_t}. | ||
881 | Each variable type should be chosen with care. | ||
882 | |||
883 | @item While @code{goto} should generally be avoided, having a | ||
884 | @code{goto} to the end of a function to a block of clean up | ||
885 | statements (free, close, etc.) can be acceptable. | ||
886 | |||
887 | @item Conditions should be written with constants on the left (to avoid | ||
888 | accidental assignment) and with the @code{true} target being either the | ||
889 | @code{error} case or the significantly simpler continuation. For example: | ||
890 | |||
891 | @example | ||
892 | if (0 != stat ("filename," | ||
893 | &sbuf)) | ||
894 | @{ | ||
895 | error(); | ||
896 | @} | ||
897 | else | ||
898 | @{ | ||
899 | /* handle normal case here */ | ||
900 | @} | ||
901 | @end example | ||
902 | |||
903 | @noindent | ||
904 | instead of | ||
905 | |||
906 | @example | ||
907 | if (stat ("filename," &sbuf) == 0) @{ | ||
908 | /* handle normal case here */ | ||
909 | @} else @{ | ||
910 | error(); | ||
911 | @} | ||
912 | @end example | ||
913 | |||
914 | @noindent | ||
915 | If possible, the error clause should be terminated with a @code{return} (or | ||
916 | @code{goto} to some cleanup routine) and in this case, the @code{else} clause | ||
917 | should be omitted: | ||
918 | |||
919 | @example | ||
920 | if (0 != stat ("filename", | ||
921 | &sbuf)) | ||
922 | @{ | ||
923 | error(); | ||
924 | return; | ||
925 | @} | ||
926 | /* handle normal case here */ | ||
927 | @end example | ||
928 | |||
929 | This serves to avoid deep nesting. The 'constants on the left' rule | ||
930 | applies to all constants (including. @code{GNUNET_SCHEDULER_NO_TASK}), | ||
931 | NULL, and enums). With the two above rules (constants on left, errors in | ||
932 | 'true' branch), there is only one way to write most branches correctly. | ||
933 | |||
934 | @item Combined assignments and tests are allowed if they do not hinder | ||
935 | code clarity. For example, one can write: | ||
936 | |||
937 | @example | ||
938 | if (NULL == (value = lookup_function())) | ||
939 | @{ | ||
940 | error(); | ||
941 | return; | ||
942 | @} | ||
943 | @end example | ||
944 | |||
945 | @item Use @code{break} and @code{continue} wherever possible to avoid | ||
946 | deep(er) nesting. Thus, we would write: | ||
947 | |||
948 | @example | ||
949 | next = head; | ||
950 | while (NULL != (pos = next)) | ||
951 | @{ | ||
952 | next = pos->next; | ||
953 | if (! should_free (pos)) | ||
954 | continue; | ||
955 | GNUNET_CONTAINER_DLL_remove (head, | ||
956 | tail, | ||
957 | pos); | ||
958 | GNUNET_free (pos); | ||
959 | @} | ||
960 | @end example | ||
961 | |||
962 | instead of | ||
963 | |||
964 | @example | ||
965 | next = head; while (NULL != (pos = next)) @{ | ||
966 | next = pos->next; | ||
967 | if (should_free (pos)) @{ | ||
968 | /* unnecessary nesting! */ | ||
969 | GNUNET_CONTAINER_DLL_remove (head, tail, pos); | ||
970 | GNUNET_free (pos); | ||
971 | @} | ||
972 | @} | ||
973 | @end example | ||
974 | |||
975 | @item We primarily use @code{for} and @code{while} loops. | ||
976 | A @code{while} loop is used if the method for advancing in the loop is | ||
977 | not a straightforward increment operation. In particular, we use: | ||
978 | |||
979 | @example | ||
980 | next = head; | ||
981 | while (NULL != (pos = next)) | ||
982 | @{ | ||
983 | next = pos->next; | ||
984 | if (! should_free (pos)) | ||
985 | continue; | ||
986 | GNUNET_CONTAINER_DLL_remove (head, | ||
987 | tail, | ||
988 | pos); | ||
989 | GNUNET_free (pos); | ||
990 | @} | ||
991 | @end example | ||
992 | |||
993 | to free entries in a list (as the iteration changes the structure of the | ||
994 | list due to the free; the equivalent @code{for} loop does no longer | ||
995 | follow the simple @code{for} paradigm of @code{for(INIT;TEST;INC)}). | ||
996 | However, for loops that do follow the simple @code{for} paradigm we do | ||
997 | use @code{for}, even if it involves linked lists: | ||
998 | |||
999 | @example | ||
1000 | /* simple iteration over a linked list */ | ||
1001 | for (pos = head; | ||
1002 | NULL != pos; | ||
1003 | pos = pos->next) | ||
1004 | @{ | ||
1005 | use (pos); | ||
1006 | @} | ||
1007 | @end example | ||
1008 | |||
1009 | |||
1010 | @item The first argument to all higher-order functions in GNUnet must be | ||
1011 | declared to be of type @code{void *} and is reserved for a closure. We do | ||
1012 | not use inner functions, as trampolines would conflict with setups that | ||
1013 | use non-executable stacks. | ||
1014 | The first statement in a higher-order function, which unusually should | ||
1015 | be part of the variable declarations, should assign the | ||
1016 | @code{cls} argument to the precise expected type. For example: | ||
1017 | |||
1018 | @example | ||
1019 | int | ||
1020 | callback (void *cls, | ||
1021 | char *args) | ||
1022 | @{ | ||
1023 | struct Foo *foo = cls; | ||
1024 | int other_variables; | ||
1025 | |||
1026 | /* rest of function */ | ||
1027 | @} | ||
1028 | @end example | ||
1029 | |||
1030 | @item As shown in the example above, after the return type of a | ||
1031 | function there should be a break. Each parameter should | ||
1032 | be on a new line. | ||
1033 | |||
1034 | @item It is good practice to write complex @code{if} expressions instead | ||
1035 | of using deeply nested @code{if} statements. However, except for addition | ||
1036 | and multiplication, all operators should use parens. This is fine: | ||
1037 | |||
1038 | @example | ||
1039 | if ( (1 == foo) || | ||
1040 | ( (0 == bar) && | ||
1041 | (x != y) ) ) | ||
1042 | return x; | ||
1043 | @end example | ||
1044 | |||
1045 | |||
1046 | However, this is not: | ||
1047 | |||
1048 | @example | ||
1049 | if (1 == foo) | ||
1050 | return x; | ||
1051 | if (0 == bar && x != y) | ||
1052 | return x; | ||
1053 | @end example | ||
1054 | |||
1055 | @noindent | ||
1056 | Note that splitting the @code{if} statement above is debatable as the | ||
1057 | @code{return x} is a very trivial statement. However, once the logic after | ||
1058 | the branch becomes more complicated (and is still identical), the "or" | ||
1059 | formulation should be used for sure. | ||
1060 | |||
1061 | @item There should be two empty lines between the end of the function and | ||
1062 | the comments describing the following function. There should be a single | ||
1063 | empty line after the initial variable declarations of a function. If a | ||
1064 | function has no local variables, there should be no initial empty line. If | ||
1065 | a long function consists of several complex steps, those steps might be | ||
1066 | separated by an empty line (possibly followed by a comment describing the | ||
1067 | following step). The code should not contain empty lines in arbitrary | ||
1068 | places; if in doubt, it is likely better to NOT have an empty line (this | ||
1069 | way, more code will fit on the screen). | ||
1070 | |||
1071 | |||
1072 | @item When command-line arguments become too long (and would result in | ||
1073 | some particularly ugly @code{uncrustify} wrapping), we start all arguments | ||
1074 | on a new line. As a result, there must never be a new line within an | ||
1075 | argument declaration (i.e. between @code{struct} and the struct's name) or | ||
1076 | between the type and the variable). Example: | ||
1077 | |||
1078 | @example | ||
1079 | struct GNUNET_TRANSPORT_CommunicatorHandle * | ||
1080 | GNUNET_TRANSPORT_communicator_connect ( | ||
1081 | const struct GNUNET_CONFIGURATION_Handle *cfg, | ||
1082 | const char *config_section_name, | ||
1083 | const char *addr_prefix, | ||
1084 | ...); | ||
1085 | @end example | ||
1086 | |||
1087 | Note that for short function names and arguments, the first argument | ||
1088 | does remain on the same line. Example: | ||
1089 | |||
1090 | @example | ||
1091 | void | ||
1092 | fun (short i, | ||
1093 | short j); | ||
1094 | @end example | ||
1095 | |||
1096 | @end itemize | ||
1097 | |||
1098 | @cindex Continuous integration | ||
1099 | @node Continuous integration | ||
1100 | @subsection Continuous integration | ||
1101 | |||
1102 | The continuous integration buildbot can be found at @uref{https://buildbot.gnunet.org}. | ||
1103 | Repositories need to be enabled by a buildbot admin in order to participate | ||
1104 | in the builds. | ||
1105 | |||
1106 | The buildbot can be configured to process scripts in your repository root under @code{.buildbot/}: | ||
1107 | |||
1108 | The files @code{build.sh}, @code{install.sh} and @code{test.sh} are executed | ||
1109 | in order if present. If you want a specific worker to behave differently, | ||
1110 | you can provide a worker specific script, e.g. @code{myworker_build.sh}. | ||
1111 | In this case, the generic step will not be executed. | ||
1112 | |||
1113 | For the @code{gnunet.git} repository, you may use "!tarball" or "!coverity" in | ||
1114 | your commit messages. | ||
1115 | "!tarball" will trigger a @code{make dist} of the gnunet source and verify that it | ||
1116 | can be compiled. The artifact will then be published to @uref{https://buildbot.gnunet.org/artifacts}. | ||
1117 | This is a good way to create a tarball for a release as it verifies the build | ||
1118 | on another machine. | ||
1119 | |||
1120 | The "!coverity" trigger will trigger a coverity build and submit the results | ||
1121 | for analysis to coverity: @uref{https://scan.coverity.com/}. | ||
1122 | Only developers with accounts for the GNUnet project on coverity.com are able to | ||
1123 | see the analysis results. | ||
1124 | |||
1125 | @cindex Commit messages and developer branches | ||
1126 | @node Commit messages and developer branches | ||
1127 | @subsection Commit messages and developer branches | ||
1128 | |||
1129 | You can find the GNUnet project repositories at @uref{https://git.gnunet.org}. | ||
1130 | For each release, the ChangeLog file is generated from the commit history. | ||
1131 | Hence, commit messages are required to convey what changes were made in | ||
1132 | a self-contained fashion. Commit messages such as "fix" or "cleanup" are | ||
1133 | not acceptable. | ||
1134 | You commit message should ideally start with the subsystem name and be followed | ||
1135 | by a succinct description of the change. Where applicable a reference to | ||
1136 | a bug number in the bugtracker may also be included. | ||
1137 | Example: | ||
1138 | |||
1139 | @example | ||
1140 | # <subsystem>: <description>. (#XXXX) | ||
1141 | IDENTITY: Fix wrong key construction for anonymous ECDSA identity. (Fixes #12344) | ||
1142 | @end example | ||
1143 | |||
1144 | If you need to commit a minor fix you may prefix the commit message with a | ||
1145 | dash. It will then be ignored when generating the ChangeLog entries: | ||
1146 | |||
1147 | @example | ||
1148 | # -<text> | ||
1149 | -fix | ||
1150 | @end example | ||
1151 | |||
1152 | If you need to modify a commit you can do so using: | ||
1153 | |||
1154 | @example | ||
1155 | $ git commit --amend | ||
1156 | @end example | ||
1157 | |||
1158 | If you need to modify a number of successive commits you will have to rebase | ||
1159 | and squash. | ||
1160 | Note that most branches are protected. This means that you can only fix commits | ||
1161 | as long as they are not pushed. You can only modify pushed commits in your own | ||
1162 | developer branches. | ||
1163 | |||
1164 | A developer branch is a branch which matches a developer-specific prefix. | ||
1165 | As a developer with git access, you should have a git username. If you do not | ||
1166 | know it, please ask an admin. | ||
1167 | A developer branch has the format: | ||
1168 | |||
1169 | @example | ||
1170 | dev/<username>/<branchname> | ||
1171 | @end example | ||
1172 | |||
1173 | |||
1174 | Assuming the developer with username "jdoe" wants to create a new branch for an | ||
1175 | i18n fix, the branch name could be: | ||
1176 | |||
1177 | @example | ||
1178 | dev/jdoe/i18n_fx | ||
1179 | @end example | ||
1180 | |||
1181 | The developer will be able to force push to and delete branches under his prefix. | ||
1182 | It is highly recommended to work in your own developer branches. | ||
1183 | Code which conforms to the commit message guidelines and coding style, is tested | ||
1184 | and builds may be merged to the master branch. | ||
1185 | Preferably, you would then... | ||
1186 | |||
1187 | @itemize | ||
1188 | @item ...squash your commits, | ||
1189 | @item rebase to master and then | ||
1190 | @item merge your branch. | ||
1191 | @item (optional) Delete the branch. | ||
1192 | @end itemize | ||
1193 | |||
1194 | In general, you may want to follow the rule "commit often, push tidy": | ||
1195 | You can create smaller, succinct commits with limited meaning on the commit | ||
1196 | messages. In the end and before you push or merge your branch, you | ||
1197 | can then squash the commits or rename them. | ||
1198 | |||
1199 | @c *********************************************************************** | ||
1200 | @node Build-system | ||
1201 | @section Build-system | ||
1202 | |||
1203 | If you have code that is likely not to compile or build rules you might | ||
1204 | want to not trigger for most developers, use @code{if HAVE_EXPERIMENTAL} | ||
1205 | in your @file{Makefile.am}. | ||
1206 | Then it is OK to (temporarily) add non-compiling (or known-to-not-port) | ||
1207 | code. | ||
1208 | |||
1209 | If you want to compile all testcases but NOT run them, run configure with | ||
1210 | the @code{--enable-test-suppression} option. | ||
1211 | |||
1212 | If you want to run all testcases, including those that take a while, run | ||
1213 | configure with the @code{--enable-expensive-testcases} option. | ||
1214 | |||
1215 | If you want to compile and run benchmarks, run configure with the | ||
1216 | @code{--enable-benchmarks} option. | ||
1217 | |||
1218 | If you want to obtain code coverage results, run configure with the | ||
1219 | @code{--enable-coverage} option and run the @file{coverage.sh} script in | ||
1220 | the @file{contrib/} directory. | ||
1221 | |||
1222 | @cindex gnunet-ext | ||
1223 | @node Developing extensions for GNUnet using the gnunet-ext template | ||
1224 | @section Developing extensions for GNUnet using the gnunet-ext template | ||
1225 | |||
1226 | For developers who want to write extensions for GNUnet we provide the | ||
1227 | gnunet-ext template to provide an easy to use skeleton. | ||
1228 | |||
1229 | gnunet-ext contains the build environment and template files for the | ||
1230 | development of GNUnet services, command line tools, APIs and tests. | ||
1231 | |||
1232 | First of all you have to obtain gnunet-ext from git: | ||
1233 | |||
1234 | @example | ||
1235 | git clone https://git.gnunet.org/gnunet-ext.git | ||
1236 | @end example | ||
1237 | |||
1238 | The next step is to bootstrap and configure it. For configure you have to | ||
1239 | provide the path containing GNUnet with | ||
1240 | @code{--with-gnunet=/path/to/gnunet} and the prefix where you want the | ||
1241 | install the extension using @code{--prefix=/path/to/install}: | ||
1242 | |||
1243 | @example | ||
1244 | ./bootstrap | ||
1245 | ./configure --prefix=/path/to/install --with-gnunet=/path/to/gnunet | ||
1246 | @end example | ||
1247 | |||
1248 | When your GNUnet installation is not included in the default linker search | ||
1249 | path, you have to add @code{/path/to/gnunet} to the file | ||
1250 | @file{/etc/ld.so.conf} and run @code{ldconfig} or your add it to the | ||
1251 | environmental variable @code{LD_LIBRARY_PATH} by using | ||
1252 | |||
1253 | @example | ||
1254 | export LD_LIBRARY_PATH=/path/to/gnunet/lib | ||
1255 | @end example | ||
1256 | |||
1257 | @cindex writing testcases | ||
1258 | @node Writing testcases | ||
1259 | @section Writing testcases | ||
1260 | |||
1261 | Ideally, any non-trivial GNUnet code should be covered by automated | ||
1262 | testcases. Testcases should reside in the same place as the code that is | ||
1263 | being tested. The name of source files implementing tests should begin | ||
1264 | with @code{test_} followed by the name of the file that contains | ||
1265 | the code that is being tested. | ||
1266 | |||
1267 | Testcases in GNUnet should be integrated with the autotools build system. | ||
1268 | This way, developers and anyone building binary packages will be able to | ||
1269 | run all testcases simply by running @code{make check}. The final | ||
1270 | testcases shipped with the distribution should output at most some brief | ||
1271 | progress information and not display debug messages by default. The | ||
1272 | success or failure of a testcase must be indicated by returning zero | ||
1273 | (success) or non-zero (failure) from the main method of the testcase. | ||
1274 | The integration with the autotools is relatively straightforward and only | ||
1275 | requires modifications to the @file{Makefile.am} in the directory | ||
1276 | containing the testcase. For a testcase testing the code in @file{foo.c} | ||
1277 | the @file{Makefile.am} would contain the following lines: | ||
1278 | |||
1279 | @example | ||
1280 | check_PROGRAMS = test_foo | ||
1281 | TESTS = $(check_PROGRAMS) | ||
1282 | test_foo_SOURCES = test_foo.c | ||
1283 | test_foo_LDADD = $(top_builddir)/src/util/libgnunetutil.la | ||
1284 | @end example | ||
1285 | |||
1286 | Naturally, other libraries used by the testcase may be specified in the | ||
1287 | @code{LDADD} directive as necessary. | ||
1288 | |||
1289 | Often testcases depend on additional input files, such as a configuration | ||
1290 | file. These support files have to be listed using the @code{EXTRA_DIST} | ||
1291 | directive in order to ensure that they are included in the distribution. | ||
1292 | |||
1293 | Example: | ||
1294 | |||
1295 | @example | ||
1296 | EXTRA_DIST = test_foo_data.conf | ||
1297 | @end example | ||
1298 | |||
1299 | Executing @code{make check} will run all testcases in the current | ||
1300 | directory and all subdirectories. Testcases can be compiled individually | ||
1301 | by running @code{make test_foo} and then invoked directly using | ||
1302 | @code{./test_foo}. Note that due to the use of plugins in GNUnet, it is | ||
1303 | typically necessary to run @code{make install} before running any | ||
1304 | testcases. Thus the canonical command @code{make check install} has to be | ||
1305 | changed to @code{make install check} for GNUnet. | ||
1306 | |||
1307 | @c *********************************************************************** | ||
1308 | @cindex Building GNUnet | ||
1309 | @node Building GNUnet and its dependencies | ||
1310 | @section Building GNUnet and its dependencies | ||
1311 | |||
1312 | In the following section we will outline how to build GNUnet and | ||
1313 | some of its dependencies. We will assume a fair amount of knowledge | ||
1314 | for building applications under UNIX-like systems. Furthermore we | ||
1315 | assume that the build environment is sane and that you are aware of | ||
1316 | any implications actions in this process could have. | ||
1317 | Instructions here can be seen as notes for developers (an extension to | ||
1318 | the 'HACKING' section in README) as well as package maintainers. | ||
1319 | @b{Users should rely on the available binary packages.} | ||
1320 | We will use Debian as an example Operating System environment. Substitute | ||
1321 | accordingly with your own Operating System environment. | ||
1322 | |||
1323 | For the full list of dependencies, consult the appropriate, up-to-date | ||
1324 | section in the @file{README} file. | ||
1325 | |||
1326 | @example | ||
1327 | libgpgerror, libgcrypt, libnettle, libunbound, GnuTLS (with libunbound | ||
1328 | support) | ||
1329 | @end example | ||
1330 | |||
1331 | After we have build and installed those packages, we continue with | ||
1332 | packages closer to GNUnet in this step: libgnurl (our libcurl fork), | ||
1333 | GNU libmicrohttpd, and GNU libextractor. Again, if your package manager | ||
1334 | provides one of these packages, use the packages provided from it | ||
1335 | unless you have good reasons (package version too old, conflicts, etc). | ||
1336 | We advise against compiling widely used packages such as GnuTLS | ||
1337 | yourself if your OS provides a variant already unless you take care | ||
1338 | of maintenance of the packages then. | ||
1339 | |||
1340 | In the optimistic case, this command will give you all the dependencies | ||
1341 | on Debian, Debian derived systems or any Linux Operating System using | ||
1342 | the apt package manager: | ||
1343 | |||
1344 | @example | ||
1345 | sudo apt-get install libgnurl libmicrohttpd libextractor | ||
1346 | @end example | ||
1347 | |||
1348 | From experience we know that at the very least libgnurl is not | ||
1349 | available in some environments. You could substitute libgnurl | ||
1350 | with libcurl, but we recommend to install libgnurl, as it gives | ||
1351 | you a predefined libcurl with the small set GNUnet requires. | ||
1352 | libgnurl has been developed to co-exist with libcurl installations, | ||
1353 | installing it will cause no filename or namespace collisions. | ||
1354 | |||
1355 | @cindex libgnurl | ||
1356 | @cindex compiling libgnurl | ||
1357 | GNUnet and some of its function depend on a limited subset of cURL/libcurl. | ||
1358 | Rather than trying to enforce a certain configuration on the world, we | ||
1359 | opted to maintain a microfork of it that ensures that we can link | ||
1360 | against the right set of features. | ||
1361 | We called this specialized set of libcurl "libgnurl". | ||
1362 | It is fully ABI compatible with libcurl and currently used by | ||
1363 | GNUnet and some of its dependencies. | ||
1364 | |||
1365 | We download libgnurl and its digital signature from the GNU fileserver, | ||
1366 | assuming @env{TMPDIR} exists. | ||
1367 | |||
1368 | @quotation | ||
1369 | Note: TMPDIR might be @file{/tmp}, @env{TMPDIR}, @env{TMP} or any other | ||
1370 | location. For consistency we assume @env{TMPDIR} points to @file{/tmp} | ||
1371 | for the remainder of this section. | ||
1372 | @end quotation | ||
1373 | |||
1374 | @example | ||
1375 | cd \$TMPDIR | ||
1376 | wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.65.3.tar.Z | ||
1377 | wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.65.3.tar.Z.sig | ||
1378 | @end example | ||
1379 | |||
1380 | Next, verify the digital signature of the file: | ||
1381 | |||
1382 | @example | ||
1383 | gpg --verify gnurl-7.65.3.tar.Z.sig | ||
1384 | @end example | ||
1385 | |||
1386 | If gpg fails, you might try with @command{gpg2} on your OS. If the error | ||
1387 | states that ``the key can not be found'' or it is unknown, you have to | ||
1388 | retrieve the key (A88C8ADD129828D7EAC02E52E22F9BBFEE348588) from a | ||
1389 | keyserver first: | ||
1390 | |||
1391 | @example | ||
1392 | gpg --keyserver pgp.mit.edu --recv-keys A88C8ADD129828D7EAC02E52E22F9BBFEE348588 | ||
1393 | @end example | ||
1394 | |||
1395 | or | ||
1396 | |||
1397 | @example | ||
1398 | gpg --keyserver hkps://keys.openpgp.org --recv-keys A88C8ADD129828D7EAC02E52E22F9BBFEE348588 | ||
1399 | @end example | ||
1400 | |||
1401 | and rerun the verification command. | ||
1402 | |||
1403 | libgnurl will require the following packages to be present at runtime: | ||
1404 | GnuTLS (with DANE support / libunbound), libidn, zlib and at compile time: | ||
1405 | libtool, perl, pkg-config, and (for tests) python (2.7, or | ||
1406 | any version of python 3). | ||
1407 | |||
1408 | Once you have verified that all the required packages are present on your | ||
1409 | system, we can proceed to compile libgnurl. This assumes you will install | ||
1410 | gnurl in the default location as prefix. To change this, pass --prefix= to | ||
1411 | the configure-gnurl script (which is a simple wrapper around configure). | ||
1412 | |||
1413 | @example | ||
1414 | tar -xvf gnurl-7.65.3.tar.Z | ||
1415 | cd gnurl-7.65.3 | ||
1416 | sh ./configure-gnurl | ||
1417 | make | ||
1418 | make -C tests test | ||
1419 | sudo make install | ||
1420 | @end example | ||
1421 | |||
1422 | After you've compiled and installed libgnurl, we can proceed to building | ||
1423 | GNUnet. | ||
1424 | |||
1425 | |||
1426 | |||
1427 | |||
1428 | First, in addition to the GNUnet sources you might require downloading the | ||
1429 | latest version of various dependencies, depending on how recent the | ||
1430 | software versions in your distribution of GNU/Linux are. | ||
1431 | Most distributions do not include sufficiently recent versions of these | ||
1432 | dependencies. | ||
1433 | Thus, a typically installation on a "modern" GNU/Linux distribution | ||
1434 | requires you to install the following dependencies (ideally in this | ||
1435 | order): | ||
1436 | |||
1437 | @itemize @bullet | ||
1438 | @item libgpgerror and libgcrypt | ||
1439 | @item libnettle and libunbound (possibly from distribution), GnuTLS | ||
1440 | @item libgnurl (read the README) | ||
1441 | @item GNU libmicrohttpd | ||
1442 | @item GNU libextractor | ||
1443 | @end itemize | ||
1444 | |||
1445 | Make sure to first install the various mandatory and optional | ||
1446 | dependencies including development headers from your distribution. | ||
1447 | |||
1448 | Other dependencies that you should strongly consider to install is a | ||
1449 | database (MySQL, SQLite3 or Postgres). | ||
1450 | The following instructions will assume that you installed at least | ||
1451 | SQLite3 (commonly distributed as ``sqlite'' or ``sqlite3''). | ||
1452 | For most distributions you should be able to find pre-build packages for | ||
1453 | the database. Again, make sure to install the client libraries @b{and} the | ||
1454 | respective development headers (if they are packaged separately) as well. | ||
1455 | |||
1456 | @c TODO: Do these platform specific descriptions still exist? If not, | ||
1457 | @c we should find a way to sync website parts with this texinfo. | ||
1458 | You can find specific, detailed instructions for installing of the | ||
1459 | dependencies (and possibly the rest of the GNUnet installation) in the | ||
1460 | platform-specific descriptions, which can be found in the Index. | ||
1461 | Please consult them now. | ||
1462 | If your distribution is not listed, please study the build | ||
1463 | instructions for Debian stable, carefully as you try to install the | ||
1464 | dependencies for your own distribution. | ||
1465 | Contributing additional instructions for further platforms is always | ||
1466 | appreciated. | ||
1467 | Please take in mind that operating system development tends to move at | ||
1468 | a rather fast speed. Due to this you should be aware that some of | ||
1469 | the instructions could be outdated by the time you are reading this. | ||
1470 | If you find a mistake, please tell us about it (or even better: send | ||
1471 | a patch to the documentation to fix it!). | ||
1472 | |||
1473 | Before proceeding further, please double-check the dependency list. | ||
1474 | Note that in addition to satisfying the dependencies, you might have to | ||
1475 | make sure that development headers for the various libraries are also | ||
1476 | installed. | ||
1477 | There maybe files for other distributions, or you might be able to find | ||
1478 | equivalent packages for your distribution. | ||
1479 | |||
1480 | While it is possible to build and install GNUnet without having root | ||
1481 | access, we will assume that you have full control over your system in | ||
1482 | these instructions. | ||
1483 | First, you should create a system user @emph{gnunet} and an additional | ||
1484 | group @emph{gnunetdns}. On the GNU/Linux distributions Debian and Ubuntu, | ||
1485 | type: | ||
1486 | |||
1487 | @example | ||
1488 | sudo adduser --system --home /var/lib/gnunet --group \ | ||
1489 | --disabled-password gnunet | ||
1490 | sudo addgroup --system gnunetdns | ||
1491 | @end example | ||
1492 | |||
1493 | @noindent | ||
1494 | On other Unix-like systems, this should have the same effect: | ||
1495 | |||
1496 | @example | ||
1497 | sudo useradd --system --groups gnunet --home-dir /var/lib/gnunet | ||
1498 | sudo addgroup --system gnunetdns | ||
1499 | @end example | ||
1500 | |||
1501 | Now compile and install GNUnet using: | ||
1502 | |||
1503 | @example | ||
1504 | tar xvf gnunet-@value{VERSION}.tar.gz | ||
1505 | cd gnunet-@value{VERSION} | ||
1506 | ./configure --with-sudo=sudo --with-nssdir=/lib | ||
1507 | make | ||
1508 | sudo make install | ||
1509 | @end example | ||
1510 | |||
1511 | If you want to be able to enable DEBUG-level log messages, add | ||
1512 | @code{--enable-logging=verbose} to the end of the | ||
1513 | @command{./configure} command. | ||
1514 | @code{DEBUG}-level log messages are in English only and | ||
1515 | should only be useful for developers (or for filing | ||
1516 | really detailed bug reports). | ||
1517 | |||
1518 | @noindent | ||
1519 | Next, edit the file @file{/etc/gnunet.conf} to contain the following: | ||
1520 | |||
1521 | @example | ||
1522 | [arm] | ||
1523 | START_SYSTEM_SERVICES = YES | ||
1524 | START_USER_SERVICES = NO | ||
1525 | @end example | ||
1526 | |||
1527 | @noindent | ||
1528 | You may need to update your @code{ld.so} cache to include | ||
1529 | files installed in @file{/usr/local/lib}: | ||
1530 | |||
1531 | @example | ||
1532 | # ldconfig | ||
1533 | @end example | ||
1534 | |||
1535 | @noindent | ||
1536 | Then, switch from user @code{root} to user @code{gnunet} to start | ||
1537 | the peer: | ||
1538 | |||
1539 | @example | ||
1540 | # su -s /bin/sh - gnunet | ||
1541 | $ gnunet-arm -c /etc/gnunet.conf -s | ||
1542 | @end example | ||
1543 | |||
1544 | You may also want to add the last line in the gnunet user's @file{crontab} | ||
1545 | prefixed with @code{@@reboot} so that it is executed whenever the system | ||
1546 | is booted: | ||
1547 | |||
1548 | @example | ||
1549 | @@reboot /usr/local/bin/gnunet-arm -c /etc/gnunet.conf -s | ||
1550 | @end example | ||
1551 | |||
1552 | @noindent | ||
1553 | This will only start the system-wide GNUnet services. | ||
1554 | Type @command{exit} to get back your root shell. | ||
1555 | Now, you need to configure the per-user part. For each | ||
1556 | user that should get access to GNUnet on the system, run | ||
1557 | (replace alice with your username): | ||
1558 | |||
1559 | @example | ||
1560 | sudo adduser alice gnunet | ||
1561 | @end example | ||
1562 | |||
1563 | @noindent | ||
1564 | to allow them to access the system-wide GNUnet services. Then, each | ||
1565 | user should create a configuration file @file{~/.config/gnunet.conf} | ||
1566 | with the lines: | ||
1567 | |||
1568 | @example | ||
1569 | [arm] | ||
1570 | START_SYSTEM_SERVICES = NO | ||
1571 | START_USER_SERVICES = YES | ||
1572 | DEFAULTSERVICES = gns | ||
1573 | @end example | ||
1574 | |||
1575 | @noindent | ||
1576 | and start the per-user services using | ||
1577 | |||
1578 | @example | ||
1579 | $ gnunet-arm -c ~/.config/gnunet.conf -s | ||
1580 | @end example | ||
1581 | |||
1582 | @noindent | ||
1583 | Again, adding a @code{crontab} entry to autostart the peer is advised: | ||
1584 | |||
1585 | @example | ||
1586 | @@reboot /usr/local/bin/gnunet-arm -c $HOME/.config/gnunet.conf -s | ||
1587 | @end example | ||
1588 | |||
1589 | @noindent | ||
1590 | Note that some GNUnet services (such as socks5 proxies) may need a | ||
1591 | system-wide TCP port for each user. | ||
1592 | For those services, systems with more than one user may require each user | ||
1593 | to specify a different port number in their personal configuration file. | ||
1594 | |||
1595 | Finally, the user should perform the basic initial setup for the GNU Name | ||
1596 | System (GNS) certificate authority. This is done by running: | ||
1597 | |||
1598 | @example | ||
1599 | $ gnunet-gns-proxy-setup-ca | ||
1600 | @end example | ||
1601 | |||
1602 | @noindent | ||
1603 | The first generates the default zones, whereas the second setups the GNS | ||
1604 | Certificate Authority with the user's browser. Now, to activate GNS in the | ||
1605 | normal DNS resolution process, you need to edit your | ||
1606 | @file{/etc/nsswitch.conf} where you should find a line like this: | ||
1607 | |||
1608 | @example | ||
1609 | hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 | ||
1610 | @end example | ||
1611 | |||
1612 | @noindent | ||
1613 | The exact details may differ a bit, which is fine. Add the text | ||
1614 | @emph{"gns [NOTFOUND=return]"} after @emph{"files"}. | ||
1615 | Keep in mind that we included a backslash ("\") here just for | ||
1616 | markup reasons. You should write the text below on @b{one line} | ||
1617 | and @b{without} the "\": | ||
1618 | |||
1619 | @example | ||
1620 | hosts: files gns [NOTFOUND=return] mdns4_minimal \ | ||
1621 | [NOTFOUND=return] dns mdns4 | ||
1622 | @end example | ||
1623 | |||
1624 | @c FIXME: Document new behavior. | ||
1625 | You might want to make sure that @file{/lib/libnss_gns.so.2} exists on | ||
1626 | your system, it should have been created during the installation. | ||
1627 | |||
1628 | |||
1629 | @c ********************************************************************** | ||
1630 | @cindex TESTING library | ||
1631 | @node TESTING library | ||
1632 | @section TESTING library | ||
1633 | |||
1634 | The TESTING library is used for writing testcases which involve starting a | ||
1635 | single or multiple peers. While peers can also be started by testcases | ||
1636 | using the ARM subsystem, using TESTING library provides an elegant way to | ||
1637 | do this. The configurations of the peers are auto-generated from a given | ||
1638 | template to have non-conflicting port numbers ensuring that peers' | ||
1639 | services do not run into bind errors. This is achieved by testing ports' | ||
1640 | availability by binding a listening socket to them before allocating them | ||
1641 | to services in the generated configurations. | ||
1642 | |||
1643 | An another advantage while using TESTING is that it shortens the testcase | ||
1644 | startup time as the hostkeys for peers are copied from a pre-computed set | ||
1645 | of hostkeys instead of generating them at peer startup which may take a | ||
1646 | considerable amount of time when starting multiple peers or on an embedded | ||
1647 | processor. | ||
1648 | |||
1649 | TESTING also allows for certain services to be shared among peers. This | ||
1650 | feature is invaluable when testing with multiple peers as it helps to | ||
1651 | reduce the number of services run per each peer and hence the total | ||
1652 | number of processes run per testcase. | ||
1653 | |||
1654 | TESTING library only handles creating, starting and stopping peers. | ||
1655 | Features useful for testcases such as connecting peers in a topology are | ||
1656 | not available in TESTING but are available in the TESTBED subsystem. | ||
1657 | Furthermore, TESTING only creates peers on the localhost, however by | ||
1658 | using TESTBED testcases can benefit from creating peers across multiple | ||
1659 | hosts. | ||
1660 | |||
1661 | @menu | ||
1662 | * API:: | ||
1663 | * Finer control over peer stop:: | ||
1664 | * Helper functions:: | ||
1665 | * Testing with multiple processes:: | ||
1666 | @end menu | ||
1667 | |||
1668 | @cindex TESTING API | ||
1669 | @node API | ||
1670 | @subsection API | ||
1671 | |||
1672 | TESTING abstracts a group of peers as a TESTING system. All peers in a | ||
1673 | system have common hostname and no two services of these peers have a | ||
1674 | same port or a UNIX domain socket path. | ||
1675 | |||
1676 | TESTING system can be created with the function | ||
1677 | @code{GNUNET_TESTING_system_create()} which returns a handle to the | ||
1678 | system. This function takes a directory path which is used for generating | ||
1679 | the configurations of peers, an IP address from which connections to the | ||
1680 | peers' services should be allowed, the hostname to be used in peers' | ||
1681 | configuration, and an array of shared service specifications of type | ||
1682 | @code{struct GNUNET_TESTING_SharedService}. | ||
1683 | |||
1684 | The shared service specification must specify the name of the service to | ||
1685 | share, the configuration pertaining to that shared service and the | ||
1686 | maximum number of peers that are allowed to share a single instance of | ||
1687 | the shared service. | ||
1688 | |||
1689 | TESTING system created with @code{GNUNET_TESTING_system_create()} chooses | ||
1690 | ports from the default range @code{12000} - @code{56000} while | ||
1691 | auto-generating configurations for peers. | ||
1692 | This range can be customised with the function | ||
1693 | @code{GNUNET_TESTING_system_create_with_portrange()}. This function is | ||
1694 | similar to @code{GNUNET_TESTING_system_create()} except that it take 2 | ||
1695 | additional parameters --- the start and end of the port range to use. | ||
1696 | |||
1697 | A TESTING system is destroyed with the function | ||
1698 | @code{GNUNET_TESTING_system_destory()}. This function takes the handle of | ||
1699 | the system and a flag to remove the files created in the directory used | ||
1700 | to generate configurations. | ||
1701 | |||
1702 | A peer is created with the function | ||
1703 | @code{GNUNET_TESTING_peer_configure()}. This functions takes the system | ||
1704 | handle, a configuration template from which the configuration for the peer | ||
1705 | is auto-generated and the index from where the hostkey for the peer has to | ||
1706 | be copied from. When successful, this function returns a handle to the | ||
1707 | peer which can be used to start and stop it and to obtain the identity of | ||
1708 | the peer. If unsuccessful, a NULL pointer is returned with an error | ||
1709 | message. This function handles the generated configuration to have | ||
1710 | non-conflicting ports and paths. | ||
1711 | |||
1712 | Peers can be started and stopped by calling the functions | ||
1713 | @code{GNUNET_TESTING_peer_start()} and @code{GNUNET_TESTING_peer_stop()} | ||
1714 | respectively. A peer can be destroyed by calling the function | ||
1715 | @code{GNUNET_TESTING_peer_destroy}. When a peer is destroyed, the ports | ||
1716 | and paths in allocated in its configuration are reclaimed for usage in new | ||
1717 | peers. | ||
1718 | |||
1719 | @c *********************************************************************** | ||
1720 | @node Finer control over peer stop | ||
1721 | @subsection Finer control over peer stop | ||
1722 | |||
1723 | Using @code{GNUNET_TESTING_peer_stop()} is normally fine for testcases. | ||
1724 | However, calling this function for each peer is inefficient when trying to | ||
1725 | shutdown multiple peers as this function sends the termination signal to | ||
1726 | the given peer process and waits for it to terminate. It would be faster | ||
1727 | in this case to send the termination signals to the peers first and then | ||
1728 | wait on them. This is accomplished by the functions | ||
1729 | @code{GNUNET_TESTING_peer_kill()} which sends a termination signal to the | ||
1730 | peer, and the function @code{GNUNET_TESTING_peer_wait()} which waits on | ||
1731 | the peer. | ||
1732 | |||
1733 | Further finer control can be achieved by choosing to stop a peer | ||
1734 | asynchronously with the function @code{GNUNET_TESTING_peer_stop_async()}. | ||
1735 | This function takes a callback parameter and a closure for it in addition | ||
1736 | to the handle to the peer to stop. The callback function is called with | ||
1737 | the given closure when the peer is stopped. Using this function | ||
1738 | eliminates blocking while waiting for the peer to terminate. | ||
1739 | |||
1740 | An asynchronous peer stop can be canceled by calling the function | ||
1741 | @code{GNUNET_TESTING_peer_stop_async_cancel()}. Note that calling this | ||
1742 | function does not prevent the peer from terminating if the termination | ||
1743 | signal has already been sent to it. It does, however, cancels the | ||
1744 | callback to be called when the peer is stopped. | ||
1745 | |||
1746 | @c *********************************************************************** | ||
1747 | @node Helper functions | ||
1748 | @subsection Helper functions | ||
1749 | |||
1750 | Most of the testcases can benefit from an abstraction which configures a | ||
1751 | peer and starts it. This is provided by the function | ||
1752 | @code{GNUNET_TESTING_peer_run()}. This function takes the testing | ||
1753 | directory pathname, a configuration template, a callback and its closure. | ||
1754 | This function creates a peer in the given testing directory by using the | ||
1755 | configuration template, starts the peer and calls the given callback with | ||
1756 | the given closure. | ||
1757 | |||
1758 | The function @code{GNUNET_TESTING_peer_run()} starts the ARM service of | ||
1759 | the peer which starts the rest of the configured services. A similar | ||
1760 | function @code{GNUNET_TESTING_service_run} can be used to just start a | ||
1761 | single service of a peer. In this case, the peer's ARM service is not | ||
1762 | started; instead, only the given service is run. | ||
1763 | |||
1764 | @c *********************************************************************** | ||
1765 | @node Testing with multiple processes | ||
1766 | @subsection Testing with multiple processes | ||
1767 | |||
1768 | When testing GNUnet, the splitting of the code into a services and clients | ||
1769 | often complicates testing. The solution to this is to have the testcase | ||
1770 | fork @code{gnunet-service-arm}, ask it to start the required server and | ||
1771 | daemon processes and then execute appropriate client actions (to test the | ||
1772 | client APIs or the core module or both). If necessary, multiple ARM | ||
1773 | services can be forked using different ports (!) to simulate a network. | ||
1774 | However, most of the time only one ARM process is needed. Note that on | ||
1775 | exit, the testcase should shutdown ARM with a @code{TERM} signal (to give | ||
1776 | it the chance to cleanly stop its child processes). | ||
1777 | |||
1778 | @c TODO: Is this still compiling and working as intended? | ||
1779 | The following code illustrates spawning and killing an ARM process from a | ||
1780 | testcase: | ||
1781 | |||
1782 | @example | ||
1783 | static void run (void *cls, | ||
1784 | char *const *args, | ||
1785 | const char *cfgfile, | ||
1786 | const struct GNUNET_CONFIGURATION_Handle *cfg) @{ | ||
1787 | struct GNUNET_OS_Process *arm_pid; | ||
1788 | arm_pid = GNUNET_OS_start_process (NULL, | ||
1789 | NULL, | ||
1790 | "gnunet-service-arm", | ||
1791 | "gnunet-service-arm", | ||
1792 | "-c", | ||
1793 | cfgname, | ||
1794 | NULL); | ||
1795 | /* do real test work here */ | ||
1796 | if (0 != GNUNET_OS_process_kill (arm_pid, SIGTERM)) | ||
1797 | GNUNET_log_strerror | ||
1798 | (GNUNET_ERROR_TYPE_WARNING, "kill"); | ||
1799 | GNUNET_assert (GNUNET_OK == GNUNET_OS_process_wait (arm_pid)); | ||
1800 | GNUNET_OS_process_close (arm_pid); @} | ||
1801 | |||
1802 | GNUNET_PROGRAM_run (argc, argv, | ||
1803 | "NAME-OF-TEST", | ||
1804 | "nohelp", | ||
1805 | options, | ||
1806 | &run, | ||
1807 | cls); | ||
1808 | @end example | ||
1809 | |||
1810 | |||
1811 | An alternative way that works well to test plugins is to implement a | ||
1812 | mock-version of the environment that the plugin expects and then to | ||
1813 | simply load the plugin directly. | ||
1814 | |||
1815 | @c *********************************************************************** | ||
1816 | @node Performance regression analysis with Gauger | ||
1817 | @section Performance regression analysis with Gauger | ||
1818 | |||
1819 | To help avoid performance regressions, GNUnet uses Gauger. Gauger is a | ||
1820 | simple logging tool that allows remote hosts to send performance data to | ||
1821 | a central server, where this data can be analyzed and visualized. Gauger | ||
1822 | shows graphs of the repository revisions and the performance data recorded | ||
1823 | for each revision, so sudden performance peaks or drops can be identified | ||
1824 | and linked to a specific revision number. | ||
1825 | |||
1826 | In the case of GNUnet, the buildbots log the performance data obtained | ||
1827 | during the tests after each build. The data can be accessed on GNUnet's | ||
1828 | Gauger page. | ||
1829 | |||
1830 | The menu on the left allows to select either the results of just one | ||
1831 | build bot (under "Hosts") or review the data from all hosts for a given | ||
1832 | test result (under "Metrics"). In case of very different absolute value | ||
1833 | of the results, for instance arm vs. amd64 machines, the option | ||
1834 | "Normalize" on a metric view can help to get an idea about the | ||
1835 | performance evolution across all hosts. | ||
1836 | |||
1837 | Using Gauger in GNUnet and having the performance of a module tracked over | ||
1838 | time is very easy. First of course, the testcase must generate some | ||
1839 | consistent metric, which makes sense to have logged. Highly volatile or | ||
1840 | random dependent metrics probably are not ideal candidates for meaningful | ||
1841 | regression detection. | ||
1842 | |||
1843 | To start logging any value, just include @code{gauger.h} in your testcase | ||
1844 | code. Then, use the macro @code{GAUGER()} to make the Buildbots log | ||
1845 | whatever value is of interest for you to @code{gnunet.org}'s Gauger | ||
1846 | server. No setup is necessary as most Buildbots have already everything | ||
1847 | in place and new metrics are created on demand. To delete a metric, you | ||
1848 | need to contact a member of the GNUnet development team (a file will need | ||
1849 | to be removed manually from the respective directory). | ||
1850 | |||
1851 | The code in the test should look like this: | ||
1852 | |||
1853 | @example | ||
1854 | [other includes] | ||
1855 | #include <gauger.h> | ||
1856 | |||
1857 | int main (int argc, char *argv[]) @{ | ||
1858 | |||
1859 | [run test, generate data] | ||
1860 | GAUGER("YOUR_MODULE", | ||
1861 | "METRIC_NAME", | ||
1862 | (float)value, | ||
1863 | "UNIT"); @} | ||
1864 | @end example | ||
1865 | |||
1866 | Where: | ||
1867 | |||
1868 | @table @asis | ||
1869 | |||
1870 | @item @strong{YOUR_MODULE} is a category in the gauger page and should be | ||
1871 | the name of the module or subsystem like "Core" or "DHT" | ||
1872 | @item @strong{METRIC} is | ||
1873 | the name of the metric being collected and should be concise and | ||
1874 | descriptive, like "PUT operations in sqlite-datastore". | ||
1875 | @item @strong{value} is the value | ||
1876 | of the metric that is logged for this run. | ||
1877 | @item @strong{UNIT} is the unit in | ||
1878 | which the value is measured, for instance "kb/s" or "kb of RAM/node". | ||
1879 | @end table | ||
1880 | |||
1881 | If you wish to use Gauger for your own project, you can grab a copy of the | ||
1882 | latest stable release or check out Gauger's Subversion repository. | ||
1883 | |||
1884 | @cindex TESTBED Subsystem | ||
1885 | @node TESTBED Subsystem | ||
1886 | @section TESTBED Subsystem | ||
1887 | |||
1888 | The TESTBED subsystem facilitates testing and measuring of multi-peer | ||
1889 | deployments on a single host or over multiple hosts. | ||
1890 | |||
1891 | The architecture of the testbed module is divided into the following: | ||
1892 | @itemize @bullet | ||
1893 | |||
1894 | @item Testbed API: An API which is used by the testing driver programs. It | ||
1895 | provides with functions for creating, destroying, starting, stopping | ||
1896 | peers, etc. | ||
1897 | |||
1898 | @item Testbed service (controller): A service which is started through the | ||
1899 | Testbed API. This service handles operations to create, destroy, start, | ||
1900 | stop peers, connect them, modify their configurations. | ||
1901 | |||
1902 | @item Testbed helper: When a controller has to be started on a host, the | ||
1903 | testbed API starts the testbed helper on that host which in turn starts | ||
1904 | the controller. The testbed helper receives a configuration for the | ||
1905 | controller through its stdin and changes it to ensure the controller | ||
1906 | doesn't run into any port conflict on that host. | ||
1907 | @end itemize | ||
1908 | |||
1909 | |||
1910 | The testbed service (controller) is different from the other GNUnet | ||
1911 | services in that it is not started by ARM and is not supposed to be run | ||
1912 | as a daemon. It is started by the testbed API through a testbed helper. | ||
1913 | In a typical scenario involving multiple hosts, a controller is started | ||
1914 | on each host. Controllers take up the actual task of creating peers, | ||
1915 | starting and stopping them on the hosts they run. | ||
1916 | |||
1917 | While running deployments on a single localhost the testbed API starts the | ||
1918 | testbed helper directly as a child process. When running deployments on | ||
1919 | remote hosts the testbed API starts Testbed Helpers on each remote host | ||
1920 | through remote shell. By default testbed API uses SSH as a remote shell. | ||
1921 | This can be changed by setting the environmental variable | ||
1922 | GNUNET_TESTBED_RSH_CMD to the required remote shell program. This | ||
1923 | variable can also contain parameters which are to be passed to the remote | ||
1924 | shell program. For e.g: | ||
1925 | |||
1926 | @example | ||
1927 | export GNUNET_TESTBED_RSH_CMD="ssh -o BatchMode=yes \ | ||
1928 | -o NoHostAuthenticationForLocalhost=yes %h" | ||
1929 | @end example | ||
1930 | |||
1931 | Substitutions are allowed in the command string above, | ||
1932 | this allows for substitutions through placemarks which begin with a `%'. | ||
1933 | At present the following substitutions are supported | ||
1934 | |||
1935 | @itemize @bullet | ||
1936 | @item %h: hostname | ||
1937 | @item %u: username | ||
1938 | @item %p: port | ||
1939 | @end itemize | ||
1940 | |||
1941 | Note that the substitution placemark is replaced only when the | ||
1942 | corresponding field is available and only once. Specifying | ||
1943 | |||
1944 | @example | ||
1945 | %u@@%h | ||
1946 | @end example | ||
1947 | |||
1948 | doesn't work either. If you want to user username substitutions for | ||
1949 | @command{SSH}, use the argument @code{-l} before the | ||
1950 | username substitution. | ||
1951 | |||
1952 | For example: | ||
1953 | @example | ||
1954 | ssh -l %u -p %p %h | ||
1955 | @end example | ||
1956 | |||
1957 | The testbed API and the helper communicate through the helpers stdin and | ||
1958 | stdout. As the helper is started through a remote shell on remote hosts | ||
1959 | any output messages from the remote shell interfere with the communication | ||
1960 | and results in a failure while starting the helper. For this reason, it is | ||
1961 | suggested to use flags to make the remote shells produce no output | ||
1962 | messages and to have password-less logins. The default remote shell, SSH, | ||
1963 | the default options are: | ||
1964 | |||
1965 | @example | ||
1966 | -o BatchMode=yes -o NoHostBasedAuthenticationForLocalhost=yes" | ||
1967 | @end example | ||
1968 | |||
1969 | Password-less logins should be ensured by using SSH keys. | ||
1970 | |||
1971 | Since the testbed API executes the remote shell as a non-interactive | ||
1972 | shell, certain scripts like .bashrc, .profiler may not be executed. If | ||
1973 | this is the case testbed API can be forced to execute an interactive | ||
1974 | shell by setting up the environmental variable | ||
1975 | @code{GNUNET_TESTBED_RSH_CMD_SUFFIX} to a shell program. | ||
1976 | |||
1977 | An example could be: | ||
1978 | |||
1979 | @example | ||
1980 | export GNUNET_TESTBED_RSH_CMD_SUFFIX="sh -lc" | ||
1981 | @end example | ||
1982 | |||
1983 | The testbed API will then execute the remote shell program as: | ||
1984 | |||
1985 | @example | ||
1986 | $GNUNET_TESTBED_RSH_CMD -p $port $dest $GNUNET_TESTBED_RSH_CMD_SUFFIX \ | ||
1987 | gnunet-helper-testbed | ||
1988 | @end example | ||
1989 | |||
1990 | On some systems, problems may arise while starting testbed helpers if | ||
1991 | GNUnet is installed into a custom location since the helper may not be | ||
1992 | found in the standard path. This can be addressed by setting the variable | ||
1993 | `@code{HELPER_BINARY_PATH}' to the path of the testbed helper. | ||
1994 | Testbed API will then use this path to start helper binaries both | ||
1995 | locally and remotely. | ||
1996 | |||
1997 | Testbed API can accessed by including the | ||
1998 | @file{gnunet_testbed_service.h} file and linking with | ||
1999 | @code{-lgnunettestbed}. | ||
2000 | |||
2001 | @c *********************************************************************** | ||
2002 | @menu | ||
2003 | * Supported Topologies:: | ||
2004 | * Hosts file format:: | ||
2005 | * Topology file format:: | ||
2006 | * Testbed Barriers:: | ||
2007 | * TESTBED Caveats:: | ||
2008 | @end menu | ||
2009 | |||
2010 | @node Supported Topologies | ||
2011 | @subsection Supported Topologies | ||
2012 | |||
2013 | While testing multi-peer deployments, it is often needed that the peers | ||
2014 | are connected in some topology. This requirement is addressed by the | ||
2015 | function @code{GNUNET_TESTBED_overlay_connect()} which connects any given | ||
2016 | two peers in the testbed. | ||
2017 | |||
2018 | The API also provides a helper function | ||
2019 | @code{GNUNET_TESTBED_overlay_configure_topology()} to connect a given set | ||
2020 | of peers in any of the following supported topologies: | ||
2021 | |||
2022 | @itemize @bullet | ||
2023 | |||
2024 | @item @code{GNUNET_TESTBED_TOPOLOGY_CLIQUE}: All peers are connected with | ||
2025 | each other | ||
2026 | |||
2027 | @item @code{GNUNET_TESTBED_TOPOLOGY_LINE}: Peers are connected to form a | ||
2028 | line | ||
2029 | |||
2030 | @item @code{GNUNET_TESTBED_TOPOLOGY_RING}: Peers are connected to form a | ||
2031 | ring topology | ||
2032 | |||
2033 | @item @code{GNUNET_TESTBED_TOPOLOGY_2D_TORUS}: Peers are connected to | ||
2034 | form a 2 dimensional torus topology. The number of peers may not be a | ||
2035 | perfect square, in that case the resulting torus may not have the uniform | ||
2036 | poloidal and toroidal lengths | ||
2037 | |||
2038 | @item @code{GNUNET_TESTBED_TOPOLOGY_ERDOS_RENYI}: Topology is generated | ||
2039 | to form a random graph. The number of links to be present should be given | ||
2040 | |||
2041 | @item @code{GNUNET_TESTBED_TOPOLOGY_SMALL_WORLD}: Peers are connected to | ||
2042 | form a 2D Torus with some random links among them. The number of random | ||
2043 | links are to be given | ||
2044 | |||
2045 | @item @code{GNUNET_TESTBED_TOPOLOGY_SMALL_WORLD_RING}: Peers are | ||
2046 | connected to form a ring with some random links among them. The number of | ||
2047 | random links are to be given | ||
2048 | |||
2049 | @item @code{GNUNET_TESTBED_TOPOLOGY_SCALE_FREE}: Connects peers in a | ||
2050 | topology where peer connectivity follows power law - new peers are | ||
2051 | connected with high probability to well connected peers. | ||
2052 | (See Emergence of Scaling in Random Networks. Science 286, | ||
2053 | 509-512, 1999 | ||
2054 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/emergence_of_scaling_in_random_networks__barabasi_albert_science_286__1999.pdf, pdf})) | ||
2055 | |||
2056 | @item @code{GNUNET_TESTBED_TOPOLOGY_FROM_FILE}: The topology information | ||
2057 | is loaded from a file. The path to the file has to be given. | ||
2058 | @xref{Topology file format}, for the format of this file. | ||
2059 | |||
2060 | @item @code{GNUNET_TESTBED_TOPOLOGY_NONE}: No topology | ||
2061 | @end itemize | ||
2062 | |||
2063 | |||
2064 | The above supported topologies can be specified respectively by setting | ||
2065 | the variable @code{OVERLAY_TOPOLOGY} to the following values in the | ||
2066 | configuration passed to Testbed API functions | ||
2067 | @code{GNUNET_TESTBED_test_run()} and | ||
2068 | @code{GNUNET_TESTBED_run()}: | ||
2069 | |||
2070 | @itemize @bullet | ||
2071 | @item @code{CLIQUE} | ||
2072 | @item @code{RING} | ||
2073 | @item @code{LINE} | ||
2074 | @item @code{2D_TORUS} | ||
2075 | @item @code{RANDOM} | ||
2076 | @item @code{SMALL_WORLD} | ||
2077 | @item @code{SMALL_WORLD_RING} | ||
2078 | @item @code{SCALE_FREE} | ||
2079 | @item @code{FROM_FILE} | ||
2080 | @item @code{NONE} | ||
2081 | @end itemize | ||
2082 | |||
2083 | |||
2084 | Topologies @code{RANDOM}, @code{SMALL_WORLD} and @code{SMALL_WORLD_RING} | ||
2085 | require the option @code{OVERLAY_RANDOM_LINKS} to be set to the number of | ||
2086 | random links to be generated in the configuration. The option will be | ||
2087 | ignored for the rest of the topologies. | ||
2088 | |||
2089 | Topology @code{SCALE_FREE} requires the options | ||
2090 | @code{SCALE_FREE_TOPOLOGY_CAP} to be set to the maximum number of peers | ||
2091 | which can connect to a peer and @code{SCALE_FREE_TOPOLOGY_M} to be set to | ||
2092 | how many peers a peer should be at least connected to. | ||
2093 | |||
2094 | Similarly, the topology @code{FROM_FILE} requires the option | ||
2095 | @code{OVERLAY_TOPOLOGY_FILE} to contain the path of the file containing | ||
2096 | the topology information. This option is ignored for the rest of the | ||
2097 | topologies. @xref{Topology file format}, for the format of this file. | ||
2098 | |||
2099 | @c *********************************************************************** | ||
2100 | @node Hosts file format | ||
2101 | @subsection Hosts file format | ||
2102 | |||
2103 | The testbed API offers the function | ||
2104 | @code{GNUNET_TESTBED_hosts_load_from_file()} to load from a given file | ||
2105 | details about the hosts which testbed can use for deploying peers. | ||
2106 | This function is useful to keep the data about hosts | ||
2107 | separate instead of hard coding them in code. | ||
2108 | |||
2109 | Another helper function from testbed API, @code{GNUNET_TESTBED_run()} | ||
2110 | also takes a hosts file name as its parameter. It uses the above | ||
2111 | function to populate the hosts data structures and start controllers to | ||
2112 | deploy peers. | ||
2113 | |||
2114 | These functions require the hosts file to be of the following format: | ||
2115 | @itemize @bullet | ||
2116 | @item Each line is interpreted to have details about a host | ||
2117 | @item Host details should include the username to use for logging into the | ||
2118 | host, the hostname of the host and the port number to use for the remote | ||
2119 | shell program. All thee values should be given. | ||
2120 | @item These details should be given in the following format: | ||
2121 | @example | ||
2122 | <username>@@<hostname>:<port> | ||
2123 | @end example | ||
2124 | @end itemize | ||
2125 | |||
2126 | Note that having canonical hostnames may cause problems while resolving | ||
2127 | the IP addresses (See this bug). Hence it is advised to provide the hosts' | ||
2128 | IP numerical addresses as hostnames whenever possible. | ||
2129 | |||
2130 | @c *********************************************************************** | ||
2131 | @node Topology file format | ||
2132 | @subsection Topology file format | ||
2133 | |||
2134 | A topology file describes how peers are to be connected. It should adhere | ||
2135 | to the following format for testbed to parse it correctly. | ||
2136 | |||
2137 | Each line should begin with the target peer id. This should be followed by | ||
2138 | a colon(`:') and origin peer ids separated by `|'. All spaces except for | ||
2139 | newline characters are ignored. The API will then try to connect each | ||
2140 | origin peer to the target peer. | ||
2141 | |||
2142 | For example, the following file will result in 5 overlay connections: | ||
2143 | [2->1], [3->1],[4->3], [0->3], [2->0]@ | ||
2144 | @code{@ 1:2|3@ 3:4| 0@ 0: 2@ } | ||
2145 | |||
2146 | @c *********************************************************************** | ||
2147 | @node Testbed Barriers | ||
2148 | @subsection Testbed Barriers | ||
2149 | |||
2150 | The testbed subsystem's barriers API facilitates coordination among the | ||
2151 | peers run by the testbed and the experiment driver. The concept is | ||
2152 | similar to the barrier synchronisation mechanism found in parallel | ||
2153 | programming or multi-threading paradigms - a peer waits at a barrier upon | ||
2154 | reaching it until the barrier is reached by a predefined number of peers. | ||
2155 | This predefined number of peers required to cross a barrier is also called | ||
2156 | quorum. We say a peer has reached a barrier if the peer is waiting for the | ||
2157 | barrier to be crossed. Similarly a barrier is said to be reached if the | ||
2158 | required quorum of peers reach the barrier. A barrier which is reached is | ||
2159 | deemed as crossed after all the peers waiting on it are notified. | ||
2160 | |||
2161 | The barriers API provides the following functions: | ||
2162 | @itemize @bullet | ||
2163 | @item @strong{@code{GNUNET_TESTBED_barrier_init()}:} function to | ||
2164 | initialize a barrier in the experiment | ||
2165 | @item @strong{@code{GNUNET_TESTBED_barrier_cancel()}:} function to cancel | ||
2166 | a barrier which has been initialized before | ||
2167 | @item @strong{@code{GNUNET_TESTBED_barrier_wait()}:} function to signal | ||
2168 | barrier service that the caller has reached a barrier and is waiting for | ||
2169 | it to be crossed | ||
2170 | @item @strong{@code{GNUNET_TESTBED_barrier_wait_cancel()}:} function to | ||
2171 | stop waiting for a barrier to be crossed | ||
2172 | @end itemize | ||
2173 | |||
2174 | |||
2175 | Among the above functions, the first two, namely | ||
2176 | @code{GNUNET_TESTBED_barrier_init()} and | ||
2177 | @code{GNUNET_TESTBED_barrier_cancel()} are used by experiment drivers. All | ||
2178 | barriers should be initialised by the experiment driver by calling | ||
2179 | @code{GNUNET_TESTBED_barrier_init()}. This function takes a name to | ||
2180 | identify the barrier, the quorum required for the barrier to be crossed | ||
2181 | and a notification callback for notifying the experiment driver when the | ||
2182 | barrier is crossed. @code{GNUNET_TESTBED_barrier_cancel()} cancels an | ||
2183 | initialised barrier and frees the resources allocated for it. This | ||
2184 | function can be called upon a initialised barrier before it is crossed. | ||
2185 | |||
2186 | The remaining two functions @code{GNUNET_TESTBED_barrier_wait()} and | ||
2187 | @code{GNUNET_TESTBED_barrier_wait_cancel()} are used in the peer's | ||
2188 | processes. @code{GNUNET_TESTBED_barrier_wait()} connects to the local | ||
2189 | barrier service running on the same host the peer is running on and | ||
2190 | registers that the caller has reached the barrier and is waiting for the | ||
2191 | barrier to be crossed. Note that this function can only be used by peers | ||
2192 | which are started by testbed as this function tries to access the local | ||
2193 | barrier service which is part of the testbed controller service. Calling | ||
2194 | @code{GNUNET_TESTBED_barrier_wait()} on an uninitialised barrier results | ||
2195 | in failure. @code{GNUNET_TESTBED_barrier_wait_cancel()} cancels the | ||
2196 | notification registered by @code{GNUNET_TESTBED_barrier_wait()}. | ||
2197 | |||
2198 | |||
2199 | @c *********************************************************************** | ||
2200 | @menu | ||
2201 | * Implementation:: | ||
2202 | @end menu | ||
2203 | |||
2204 | @node Implementation | ||
2205 | @subsubsection Implementation | ||
2206 | |||
2207 | Since barriers involve coordination between experiment driver and peers, | ||
2208 | the barrier service in the testbed controller is split into two | ||
2209 | components. The first component responds to the message generated by the | ||
2210 | barrier API used by the experiment driver (functions | ||
2211 | @code{GNUNET_TESTBED_barrier_init()} and | ||
2212 | @code{GNUNET_TESTBED_barrier_cancel()}) and the second component to the | ||
2213 | messages generated by barrier API used by peers (functions | ||
2214 | @code{GNUNET_TESTBED_barrier_wait()} and | ||
2215 | @code{GNUNET_TESTBED_barrier_wait_cancel()}). | ||
2216 | |||
2217 | Calling @code{GNUNET_TESTBED_barrier_init()} sends a | ||
2218 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_INIT} message to the master | ||
2219 | controller. The master controller then registers a barrier and calls | ||
2220 | @code{GNUNET_TESTBED_barrier_init()} for each its subcontrollers. In this | ||
2221 | way barrier initialisation is propagated to the controller hierarchy. | ||
2222 | While propagating initialisation, any errors at a subcontroller such as | ||
2223 | timeout during further propagation are reported up the hierarchy back to | ||
2224 | the experiment driver. | ||
2225 | |||
2226 | Similar to @code{GNUNET_TESTBED_barrier_init()}, | ||
2227 | @code{GNUNET_TESTBED_barrier_cancel()} propagates | ||
2228 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_CANCEL} message which causes | ||
2229 | controllers to remove an initialised barrier. | ||
2230 | |||
2231 | The second component is implemented as a separate service in the binary | ||
2232 | `gnunet-service-testbed' which already has the testbed controller service. | ||
2233 | Although this deviates from the gnunet process architecture of having one | ||
2234 | service per binary, it is needed in this case as this component needs | ||
2235 | access to barrier data created by the first component. This component | ||
2236 | responds to @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} messages from | ||
2237 | local peers when they call @code{GNUNET_TESTBED_barrier_wait()}. Upon | ||
2238 | receiving @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} message, the | ||
2239 | service checks if the requested barrier has been initialised before and | ||
2240 | if it was not initialised, an error status is sent through | ||
2241 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message to the local | ||
2242 | peer and the connection from the peer is terminated. If the barrier is | ||
2243 | initialised before, the barrier's counter for reached peers is incremented | ||
2244 | and a notification is registered to notify the peer when the barrier is | ||
2245 | reached. The connection from the peer is left open. | ||
2246 | |||
2247 | When enough peers required to attain the quorum send | ||
2248 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} messages, the controller | ||
2249 | sends a @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message to its | ||
2250 | parent informing that the barrier is crossed. If the controller has | ||
2251 | started further subcontrollers, it delays this message until it receives | ||
2252 | a similar notification from each of those subcontrollers. Finally, the | ||
2253 | barriers API at the experiment driver receives the | ||
2254 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} when the barrier is | ||
2255 | reached at all the controllers. | ||
2256 | |||
2257 | The barriers API at the experiment driver responds to the | ||
2258 | @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message by echoing it | ||
2259 | back to the master controller and notifying the experiment controller | ||
2260 | through the notification callback that a barrier has been crossed. The | ||
2261 | echoed @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message is | ||
2262 | propagated by the master controller to the controller hierarchy. This | ||
2263 | propagation triggers the notifications registered by peers at each of the | ||
2264 | controllers in the hierarchy. Note the difference between this downward | ||
2265 | propagation of the @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} | ||
2266 | message from its upward propagation --- the upward propagation is needed | ||
2267 | for ensuring that the barrier is reached by all the controllers and the | ||
2268 | downward propagation is for triggering that the barrier is crossed. | ||
2269 | |||
2270 | @cindex TESTBED Caveats | ||
2271 | @node TESTBED Caveats | ||
2272 | @subsection TESTBED Caveats | ||
2273 | |||
2274 | This section documents a few caveats when using the GNUnet testbed | ||
2275 | subsystem. | ||
2276 | |||
2277 | @c *********************************************************************** | ||
2278 | @menu | ||
2279 | * CORE must be started:: | ||
2280 | * ATS must want the connections:: | ||
2281 | @end menu | ||
2282 | |||
2283 | @node CORE must be started | ||
2284 | @subsubsection CORE must be started | ||
2285 | |||
2286 | A uncomplicated issue is bug #3993 | ||
2287 | (@uref{https://bugs.gnunet.org/view.php?id=3993, https://bugs.gnunet.org/view.php?id=3993}): | ||
2288 | Your configuration MUST somehow ensure that for each peer the | ||
2289 | @code{CORE} service is started when the peer is setup, otherwise | ||
2290 | @code{TESTBED} may fail to connect peers when the topology is initialized, | ||
2291 | as @code{TESTBED} will start some @code{CORE} services but not | ||
2292 | necessarily all (but it relies on all of them running). The easiest way | ||
2293 | is to set | ||
2294 | |||
2295 | @example | ||
2296 | [core] | ||
2297 | IMMEDIATE_START = YES | ||
2298 | @end example | ||
2299 | |||
2300 | @noindent | ||
2301 | in the configuration file. | ||
2302 | Alternatively, having any service that directly or indirectly depends on | ||
2303 | @code{CORE} being started with @code{IMMEDIATE_START} will also do. | ||
2304 | This issue largely arises if users try to over-optimize by not | ||
2305 | starting any services with @code{IMMEDIATE_START}. | ||
2306 | |||
2307 | @c *********************************************************************** | ||
2308 | @node ATS must want the connections | ||
2309 | @subsubsection ATS must want the connections | ||
2310 | |||
2311 | When TESTBED sets up connections, it only offers the respective HELLO | ||
2312 | information to the TRANSPORT service. It is then up to the ATS service to | ||
2313 | @strong{decide} to use the connection. The ATS service will typically | ||
2314 | eagerly establish any connection if the number of total connections is | ||
2315 | low (relative to bandwidth). Details may further depend on the | ||
2316 | specific ATS backend that was configured. If ATS decides to NOT establish | ||
2317 | a connection (even though TESTBED provided the required information), then | ||
2318 | that connection will count as failed for TESTBED. Note that you can | ||
2319 | configure TESTBED to tolerate a certain number of connection failures | ||
2320 | (see '-e' option of gnunet-testbed-profiler). This issue largely arises | ||
2321 | for dense overlay topologies, especially if you try to create cliques | ||
2322 | with more than 20 peers. | ||
2323 | |||
2324 | @cindex libgnunetutil | ||
2325 | @node libgnunetutil | ||
2326 | @section libgnunetutil | ||
2327 | |||
2328 | libgnunetutil is the fundamental library that all GNUnet code builds upon. | ||
2329 | Ideally, this library should contain most of the platform dependent code | ||
2330 | (except for user interfaces and really special needs that only few | ||
2331 | applications have). It is also supposed to offer basic services that most | ||
2332 | if not all GNUnet binaries require. The code of libgnunetutil is in the | ||
2333 | @file{src/util/} directory. The public interface to the library is in the | ||
2334 | gnunet_util.h header. The functions provided by libgnunetutil fall | ||
2335 | roughly into the following categories (in roughly the order of importance | ||
2336 | for new developers): | ||
2337 | |||
2338 | @itemize @bullet | ||
2339 | @item logging (common_logging.c) | ||
2340 | @item memory allocation (common_allocation.c) | ||
2341 | @item endianness conversion (common_endian.c) | ||
2342 | @item internationalization (common_gettext.c) | ||
2343 | @item String manipulation (string.c) | ||
2344 | @item file access (disk.c) | ||
2345 | @item buffered disk IO (bio.c) | ||
2346 | @item time manipulation (time.c) | ||
2347 | @item configuration parsing (configuration.c) | ||
2348 | @item command-line handling (getopt*.c) | ||
2349 | @item cryptography (crypto_*.c) | ||
2350 | @item data structures (container_*.c) | ||
2351 | @item CPS-style scheduling (scheduler.c) | ||
2352 | @item Program initialization (program.c) | ||
2353 | @item Networking (network.c, client.c, server*.c, service.c) | ||
2354 | @item message queuing (mq.c) | ||
2355 | @item bandwidth calculations (bandwidth.c) | ||
2356 | @item Other OS-related (os*.c, plugin.c, signal.c) | ||
2357 | @item Pseudonym management (pseudonym.c) | ||
2358 | @end itemize | ||
2359 | |||
2360 | It should be noted that only developers that fully understand this entire | ||
2361 | API will be able to write good GNUnet code. | ||
2362 | |||
2363 | Ideally, porting GNUnet should only require porting the gnunetutil | ||
2364 | library. More testcases for the gnunetutil APIs are therefore a great | ||
2365 | way to make porting of GNUnet easier. | ||
2366 | |||
2367 | @menu | ||
2368 | * Logging:: | ||
2369 | * Interprocess communication API (IPC):: | ||
2370 | * Cryptography API:: | ||
2371 | * Message Queue API:: | ||
2372 | * Service API:: | ||
2373 | * Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps:: | ||
2374 | * CONTAINER_MDLL API:: | ||
2375 | @end menu | ||
2376 | |||
2377 | @cindex Logging | ||
2378 | @cindex log levels | ||
2379 | @node Logging | ||
2380 | @subsection Logging | ||
2381 | |||
2382 | GNUnet is able to log its activity, mostly for the purposes of debugging | ||
2383 | the program at various levels. | ||
2384 | |||
2385 | @file{gnunet_common.h} defines several @strong{log levels}: | ||
2386 | @table @asis | ||
2387 | |||
2388 | @item ERROR for errors | ||
2389 | (really problematic situations, often leading to crashes) | ||
2390 | @item WARNING for warnings | ||
2391 | (troubling situations that might have negative consequences, although | ||
2392 | not fatal) | ||
2393 | @item INFO for various information. | ||
2394 | Used somewhat rarely, as GNUnet statistics is used to hold and display | ||
2395 | most of the information that users might find interesting. | ||
2396 | @item DEBUG for debugging. | ||
2397 | Does not produce much output on normal builds, but when extra logging is | ||
2398 | enabled at compile time, a staggering amount of data is outputted under | ||
2399 | this log level. | ||
2400 | @end table | ||
2401 | |||
2402 | |||
2403 | Normal builds of GNUnet (configured with @code{--enable-logging[=yes]}) | ||
2404 | are supposed to log nothing under DEBUG level. The | ||
2405 | @code{--enable-logging=verbose} configure option can be used to create a | ||
2406 | build with all logging enabled. However, such build will produce large | ||
2407 | amounts of log data, which is inconvenient when one tries to hunt down a | ||
2408 | specific problem. | ||
2409 | |||
2410 | To mitigate this problem, GNUnet provides facilities to apply a filter to | ||
2411 | reduce the logs: | ||
2412 | @table @asis | ||
2413 | |||
2414 | @item Logging by default When no log levels are configured in any other | ||
2415 | way (see below), GNUnet will default to the WARNING log level. This | ||
2416 | mostly applies to GNUnet command line utilities, services and daemons; | ||
2417 | tests will always set log level to WARNING or, if | ||
2418 | @code{--enable-logging=verbose} was passed to configure, to DEBUG. The | ||
2419 | default level is suggested for normal operation. | ||
2420 | @item The -L option Most GNUnet executables accept an "-L loglevel" or | ||
2421 | "--log=loglevel" option. If used, it makes the process set a global log | ||
2422 | level to "loglevel". Thus it is possible to run some processes | ||
2423 | with -L DEBUG, for example, and others with -L ERROR to enable specific | ||
2424 | settings to diagnose problems with a particular process. | ||
2425 | @item Configuration files. Because GNUnet | ||
2426 | service and daemon processes are usually launched by gnunet-arm, it is not | ||
2427 | possible to pass different custom command line options directly to every | ||
2428 | one of them. The options passed to @code{gnunet-arm} only affect | ||
2429 | gnunet-arm and not the rest of GNUnet. However, one can specify a | ||
2430 | configuration key "OPTIONS" in the section that corresponds to a service | ||
2431 | or a daemon, and put a value of "-L loglevel" there. This will make the | ||
2432 | respective service or daemon set its log level to "loglevel" (as the | ||
2433 | value of OPTIONS will be passed as a command-line argument). | ||
2434 | |||
2435 | To specify the same log level for all services without creating separate | ||
2436 | "OPTIONS" entries in the configuration for each one, the user can specify | ||
2437 | a config key "GLOBAL_POSTFIX" in the [arm] section of the configuration | ||
2438 | file. The value of GLOBAL_POSTFIX will be appended to all command lines | ||
2439 | used by the ARM service to run other services. It can contain any option | ||
2440 | valid for all GNUnet commands, thus in particular the "-L loglevel" | ||
2441 | option. The ARM service itself is, however, unaffected by GLOBAL_POSTFIX; | ||
2442 | to set log level for it, one has to specify "OPTIONS" key in the [arm] | ||
2443 | section. | ||
2444 | @item Environment variables. | ||
2445 | Setting global per-process log levels with "-L loglevel" does not offer | ||
2446 | sufficient log filtering granularity, as one service will call interface | ||
2447 | libraries and supporting libraries of other GNUnet services, potentially | ||
2448 | producing lots of debug log messages from these libraries. Also, changing | ||
2449 | the config file is not always convenient (especially when running the | ||
2450 | GNUnet test suite).@ To fix that, and to allow GNUnet to use different | ||
2451 | log filtering at runtime without re-compiling the whole source tree, the | ||
2452 | log calls were changed to be configurable at run time. To configure them | ||
2453 | one has to define environment variables "GNUNET_FORCE_LOGFILE", | ||
2454 | "GNUNET_LOG" and/or "GNUNET_FORCE_LOG": | ||
2455 | @itemize @bullet | ||
2456 | |||
2457 | @item "GNUNET_LOG" only affects the logging when no global log level is | ||
2458 | configured by any other means (that is, the process does not explicitly | ||
2459 | set its own log level, there are no "-L loglevel" options on command line | ||
2460 | or in configuration files), and can be used to override the default | ||
2461 | WARNING log level. | ||
2462 | |||
2463 | @item "GNUNET_FORCE_LOG" will completely override any other log | ||
2464 | configuration options given. | ||
2465 | |||
2466 | @item "GNUNET_FORCE_LOGFILE" will completely override the location of the | ||
2467 | file to log messages to. It should contain a relative or absolute file | ||
2468 | name. Setting GNUNET_FORCE_LOGFILE is equivalent to passing | ||
2469 | "--log-file=logfile" or "-l logfile" option (see below). It supports "[]" | ||
2470 | format in file names, but not "@{@}" (see below). | ||
2471 | @end itemize | ||
2472 | |||
2473 | |||
2474 | Because environment variables are inherited by child processes when they | ||
2475 | are launched, starting or re-starting the ARM service with these | ||
2476 | variables will propagate them to all other services. | ||
2477 | |||
2478 | "GNUNET_LOG" and "GNUNET_FORCE_LOG" variables must contain a specially | ||
2479 | formatted @strong{logging definition} string, which looks like this:@ | ||
2480 | |||
2481 | @c FIXME: Can we close this with [/component] instead? | ||
2482 | @example | ||
2483 | [component];[file];[function];[from_line[-to_line]];loglevel[/component...] | ||
2484 | @end example | ||
2485 | |||
2486 | That is, a logging definition consists of definition entries, separated by | ||
2487 | slashes ('/'). If only one entry is present, there is no need to add a | ||
2488 | slash to its end (although it is not forbidden either).@ All definition | ||
2489 | fields (component, file, function, lines and loglevel) are mandatory, but | ||
2490 | (except for the loglevel) they can be empty. An empty field means | ||
2491 | "match anything". Note that even if fields are empty, the semicolon (';') | ||
2492 | separators must be present.@ The loglevel field is mandatory, and must | ||
2493 | contain one of the log level names (ERROR, WARNING, INFO or DEBUG).@ | ||
2494 | The lines field might contain one non-negative number, in which case it | ||
2495 | matches only one line, or a range "from_line-to_line", in which case it | ||
2496 | matches any line in the interval [from_line;to_line] (that is, including | ||
2497 | both start and end line).@ GNUnet mostly defaults component name to the | ||
2498 | name of the service that is implemented in a process ('transport', | ||
2499 | 'core', 'peerinfo', etc), but logging calls can specify custom component | ||
2500 | names using @code{GNUNET_log_from}.@ File name and function name are | ||
2501 | provided by the compiler (__FILE__ and __FUNCTION__ built-ins). | ||
2502 | |||
2503 | Component, file and function fields are interpreted as non-extended | ||
2504 | regular expressions (GNU libc regex functions are used). Matching is | ||
2505 | case-sensitive, "^" and "$" will match the beginning and the end of the | ||
2506 | text. If a field is empty, its contents are automatically replaced with | ||
2507 | a ".*" regular expression, which matches anything. Matching is done in | ||
2508 | the default way, which means that the expression matches as long as it's | ||
2509 | contained anywhere in the string. Thus "GNUNET_" will match both | ||
2510 | "GNUNET_foo" and "BAR_GNUNET_BAZ". Use '^' and/or '$' to make sure that | ||
2511 | the expression matches at the start and/or at the end of the string. | ||
2512 | The semicolon (';') can't be escaped, and GNUnet will not use it in | ||
2513 | component names (it can't be used in function names and file names | ||
2514 | anyway). | ||
2515 | |||
2516 | @end table | ||
2517 | |||
2518 | |||
2519 | Every logging call in GNUnet code will be (at run time) matched against | ||
2520 | the log definitions passed to the process. If a log definition fields are | ||
2521 | matching the call arguments, then the call log level is compared to the | ||
2522 | log level of that definition. If the call log level is less or equal to | ||
2523 | the definition log level, the call is allowed to proceed. Otherwise the | ||
2524 | logging call is forbidden, and nothing is logged. If no definitions | ||
2525 | matched at all, GNUnet will use the global log level or (if a global log | ||
2526 | level is not specified) will default to WARNING (that is, it will allow | ||
2527 | the call to proceed, if its level is less or equal to the global log | ||
2528 | level or to WARNING). | ||
2529 | |||
2530 | That is, definitions are evaluated from left to right, and the first | ||
2531 | matching definition is used to allow or deny the logging call. Thus it is | ||
2532 | advised to place narrow definitions at the beginning of the logdef | ||
2533 | string, and generic definitions - at the end. | ||
2534 | |||
2535 | Whether a call is allowed or not is only decided the first time this | ||
2536 | particular call is made. The evaluation result is then cached, so that | ||
2537 | any attempts to make the same call later will be allowed or disallowed | ||
2538 | right away. Because of that runtime log level evaluation should not | ||
2539 | significantly affect the process performance. | ||
2540 | Log definition parsing is only done once, at the first call to | ||
2541 | @code{GNUNET_log_setup ()} made by the process (which is usually | ||
2542 | done soon after it starts). | ||
2543 | |||
2544 | At the moment of writing there is no way to specify logging definitions | ||
2545 | from configuration files, only via environment variables. | ||
2546 | |||
2547 | At the moment GNUnet will stop processing a log definition when it | ||
2548 | encounters an error in definition formatting or an error in regular | ||
2549 | expression syntax, and will not report the failure in any way. | ||
2550 | |||
2551 | |||
2552 | @c *********************************************************************** | ||
2553 | @menu | ||
2554 | * Examples:: | ||
2555 | * Log files:: | ||
2556 | * Updated behavior of GNUNET_log:: | ||
2557 | @end menu | ||
2558 | |||
2559 | @node Examples | ||
2560 | @subsubsection Examples | ||
2561 | |||
2562 | @table @asis | ||
2563 | |||
2564 | @item @code{GNUNET_FORCE_LOG=";;;;DEBUG" gnunet-arm -s} Start GNUnet | ||
2565 | process tree, running all processes with DEBUG level (one should be | ||
2566 | careful with it, as log files will grow at alarming rate!) | ||
2567 | @item @code{GNUNET_FORCE_LOG="core;;;;DEBUG" gnunet-arm -s} Start GNUnet | ||
2568 | process tree, running the core service under DEBUG level (everything else | ||
2569 | will use configured or default level). | ||
2570 | |||
2571 | @item Start GNUnet process tree, allowing any logging calls from | ||
2572 | gnunet-service-transport_validation.c (everything else will use | ||
2573 | configured or default level). | ||
2574 | |||
2575 | @example | ||
2576 | GNUNET_FORCE_LOG=";gnunet-service-transport_validation.c;;; DEBUG" \ | ||
2577 | gnunet-arm -s | ||
2578 | @end example | ||
2579 | |||
2580 | @item Start GNUnet process tree, allowing any logging calls from | ||
2581 | gnunet-gnunet-service-fs_push.c (everything else will use configured or | ||
2582 | default level). | ||
2583 | |||
2584 | @example | ||
2585 | GNUNET_FORCE_LOG="fs;gnunet-service-fs_push.c;;;DEBUG" gnunet-arm -s | ||
2586 | @end example | ||
2587 | |||
2588 | @item Start GNUnet process tree, allowing any logging calls from the | ||
2589 | GNUNET_NETWORK_socket_select function (everything else will use | ||
2590 | configured or default level). | ||
2591 | |||
2592 | @example | ||
2593 | GNUNET_FORCE_LOG=";;GNUNET_NETWORK_socket_select;;DEBUG" gnunet-arm -s | ||
2594 | @end example | ||
2595 | |||
2596 | @item Start GNUnet process tree, allowing any logging calls from the | ||
2597 | components that have "transport" in their names, and are made from | ||
2598 | function that have "send" in their names. Everything else will be allowed | ||
2599 | to be logged only if it has WARNING level. | ||
2600 | |||
2601 | @example | ||
2602 | GNUNET_FORCE_LOG="transport.*;;.*send.*;;DEBUG/;;;;WARNING" gnunet-arm -s | ||
2603 | @end example | ||
2604 | |||
2605 | @end table | ||
2606 | |||
2607 | |||
2608 | On Windows, one can use batch files to run GNUnet processes with special | ||
2609 | environment variables, without affecting the whole system. Such batch | ||
2610 | file will look like this: | ||
2611 | |||
2612 | @example | ||
2613 | set GNUNET_FORCE_LOG=;;do_transmit;;DEBUG@ gnunet-arm -s | ||
2614 | @end example | ||
2615 | |||
2616 | (note the absence of double quotes in the environment variable definition, | ||
2617 | as opposed to earlier examples, which use the shell). | ||
2618 | Another limitation, on Windows, GNUNET_FORCE_LOGFILE @strong{MUST} be set | ||
2619 | in order to GNUNET_FORCE_LOG to work. | ||
2620 | |||
2621 | |||
2622 | @cindex Log files | ||
2623 | @node Log files | ||
2624 | @subsubsection Log files | ||
2625 | |||
2626 | GNUnet can be told to log everything into a file instead of stderr (which | ||
2627 | is the default) using the "--log-file=logfile" or "-l logfile" option. | ||
2628 | This option can also be passed via command line, or from the "OPTION" and | ||
2629 | "GLOBAL_POSTFIX" configuration keys (see above). The file name passed | ||
2630 | with this option is subject to GNUnet filename expansion. If specified in | ||
2631 | "GLOBAL_POSTFIX", it is also subject to ARM service filename expansion, | ||
2632 | in particular, it may contain "@{@}" (left and right curly brace) | ||
2633 | sequence, which will be replaced by ARM with the name of the service. | ||
2634 | This is used to keep logs from more than one service separate, while only | ||
2635 | specifying one template containing "@{@}" in GLOBAL_POSTFIX. | ||
2636 | |||
2637 | As part of a secondary file name expansion, the first occurrence of "[]" | ||
2638 | sequence ("left square brace" followed by "right square brace") in the | ||
2639 | file name will be replaced with a process identifier or the process when | ||
2640 | it initializes its logging subsystem. As a result, all processes will log | ||
2641 | into different files. This is convenient for isolating messages of a | ||
2642 | particular process, and prevents I/O races when multiple processes try to | ||
2643 | write into the file at the same time. This expansion is done | ||
2644 | independently of "@{@}" expansion that ARM service does (see above). | ||
2645 | |||
2646 | The log file name that is specified via "-l" can contain format characters | ||
2647 | from the 'strftime' function family. For example, "%Y" will be replaced | ||
2648 | with the current year. Using "basename-%Y-%m-%d.log" would include the | ||
2649 | current year, month and day in the log file. If a GNUnet process runs for | ||
2650 | long enough to need more than one log file, it will eventually clean up | ||
2651 | old log files. Currently, only the last three log files (plus the current | ||
2652 | log file) are preserved. So once the fifth log file goes into use (so | ||
2653 | after 4 days if you use "%Y-%m-%d" as above), the first log file will be | ||
2654 | automatically deleted. Note that if your log file name only contains "%Y", | ||
2655 | then log files would be kept for 4 years and the logs from the first year | ||
2656 | would be deleted once year 5 begins. If you do not use any date-related | ||
2657 | string format codes, logs would never be automatically deleted by GNUnet. | ||
2658 | |||
2659 | |||
2660 | @c *********************************************************************** | ||
2661 | |||
2662 | @node Updated behavior of GNUNET_log | ||
2663 | @subsubsection Updated behavior of GNUNET_log | ||
2664 | |||
2665 | It's currently quite common to see constructions like this all over the | ||
2666 | code: | ||
2667 | |||
2668 | @example | ||
2669 | #if MESH_DEBUG | ||
2670 | GNUNET_log (GNUNET_ERROR_TYPE_DEBUG, "MESH: client disconnected\n"); | ||
2671 | #endif | ||
2672 | @end example | ||
2673 | |||
2674 | The reason for the #if is not to avoid displaying the message when | ||
2675 | disabled (GNUNET_ERROR_TYPE takes care of that), but to avoid the | ||
2676 | compiler including it in the binary at all, when compiling GNUnet for | ||
2677 | platforms with restricted storage space / memory (MIPS routers, | ||
2678 | ARM plug computers / dev boards, etc). | ||
2679 | |||
2680 | This presents several problems: the code gets ugly, hard to write and it | ||
2681 | is very easy to forget to include the #if guards, creating non-consistent | ||
2682 | code. A new change in GNUNET_log aims to solve these problems. | ||
2683 | |||
2684 | @strong{This change requires to @file{./configure} with at least | ||
2685 | @code{--enable-logging=verbose} to see debug messages.} | ||
2686 | |||
2687 | Here is an example of code with dense debug statements: | ||
2688 | |||
2689 | @example | ||
2690 | switch (restrict_topology) @{ | ||
2691 | case GNUNET_TESTING_TOPOLOGY_CLIQUE:#if VERBOSE_TESTING | ||
2692 | GNUNET_log (GNUNET_ERROR_TYPE_DEBUG, _("Blacklisting all but clique | ||
2693 | topology\n")); #endif unblacklisted_connections = create_clique (pg, | ||
2694 | &remove_connections, BLACKLIST, GNUNET_NO); break; case | ||
2695 | GNUNET_TESTING_TOPOLOGY_SMALL_WORLD_RING: #if VERBOSE_TESTING GNUNET_log | ||
2696 | (GNUNET_ERROR_TYPE_DEBUG, _("Blacklisting all but small world (ring) | ||
2697 | topology\n")); #endif unblacklisted_connections = create_small_world_ring | ||
2698 | (pg,&remove_connections, BLACKLIST); break; | ||
2699 | @end example | ||
2700 | |||
2701 | |||
2702 | Pretty hard to follow, huh? | ||
2703 | |||
2704 | From now on, it is not necessary to include the #if / #endif statements to | ||
2705 | achieve the same behavior. The @code{GNUNET_log} and @code{GNUNET_log_from} | ||
2706 | macros take care of it for you, depending on the configure option: | ||
2707 | |||
2708 | @itemize @bullet | ||
2709 | @item If @code{--enable-logging} is set to @code{no}, the binary will | ||
2710 | contain no log messages at all. | ||
2711 | @item If @code{--enable-logging} is set to @code{yes}, the binary will | ||
2712 | contain no DEBUG messages, and therefore running with @command{-L DEBUG} | ||
2713 | will have | ||
2714 | no effect. Other messages (ERROR, WARNING, INFO, etc) will be included. | ||
2715 | @item If @code{--enable-logging} is set to @code{verbose}, or | ||
2716 | @code{veryverbose} the binary will contain DEBUG messages (still, it will | ||
2717 | be necessary to run with @command{-L DEBUG} or set the DEBUG config option | ||
2718 | to show them). | ||
2719 | @end itemize | ||
2720 | |||
2721 | |||
2722 | If you are a developer: | ||
2723 | @itemize @bullet | ||
2724 | @item please make sure that you @code{./configure | ||
2725 | --enable-logging=@{verbose,veryverbose@}}, so you can see DEBUG messages. | ||
2726 | @item please remove the @code{#if} statements around @code{GNUNET_log | ||
2727 | (GNUNET_ERROR_TYPE_DEBUG, ...)} lines, to improve the readability of your | ||
2728 | code. | ||
2729 | @end itemize | ||
2730 | |||
2731 | Since now activating DEBUG automatically makes it VERBOSE and activates | ||
2732 | @strong{all} debug messages by default, you probably want to use the | ||
2733 | @uref{https://docs.gnunet.org/#Logging, https://docs.gnunet.org/#Logging} | ||
2734 | functionality to filter only relevant messages. | ||
2735 | A suitable configuration could be: | ||
2736 | |||
2737 | @example | ||
2738 | $ export GNUNET_FORCE_LOG="^YOUR_SUBSYSTEM$;;;;DEBUG/;;;;WARNING" | ||
2739 | @end example | ||
2740 | |||
2741 | Which will behave almost like enabling DEBUG in that subsystem before the | ||
2742 | change. Of course you can adapt it to your particular needs, this is only | ||
2743 | a quick example. | ||
2744 | |||
2745 | @cindex Interprocess communication API | ||
2746 | @cindex ICP | ||
2747 | @node Interprocess communication API (IPC) | ||
2748 | @subsection Interprocess communication API (IPC) | ||
2749 | |||
2750 | In GNUnet a variety of new message types might be defined and used in | ||
2751 | interprocess communication, in this tutorial we use the | ||
2752 | @code{struct AddressLookupMessage} as a example to introduce how to | ||
2753 | construct our own message type in GNUnet and how to implement the message | ||
2754 | communication between service and client. | ||
2755 | (Here, a client uses the @code{struct AddressLookupMessage} as a request | ||
2756 | to ask the server to return the address of any other peer connecting to | ||
2757 | the service.) | ||
2758 | |||
2759 | |||
2760 | @c *********************************************************************** | ||
2761 | @menu | ||
2762 | * Define new message types:: | ||
2763 | * Define message struct:: | ||
2764 | * Client - Establish connection:: | ||
2765 | * Client - Initialize request message:: | ||
2766 | * Client - Send request and receive response:: | ||
2767 | * Server - Startup service:: | ||
2768 | * Server - Add new handles for specified messages:: | ||
2769 | * Server - Process request message:: | ||
2770 | * Server - Response to client:: | ||
2771 | * Server - Notification of clients:: | ||
2772 | * Conversion between Network Byte Order (Big Endian) and Host Byte Order:: | ||
2773 | @end menu | ||
2774 | |||
2775 | @node Define new message types | ||
2776 | @subsubsection Define new message types | ||
2777 | |||
2778 | First of all, you should define the new message type in | ||
2779 | @file{gnunet_protocols.h}: | ||
2780 | |||
2781 | @example | ||
2782 | // Request to look addresses of peers in server. | ||
2783 | #define GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_LOOKUP 29 | ||
2784 | // Response to the address lookup request. | ||
2785 | #define GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_REPLY 30 | ||
2786 | @end example | ||
2787 | |||
2788 | @c *********************************************************************** | ||
2789 | @node Define message struct | ||
2790 | @subsubsection Define message struct | ||
2791 | |||
2792 | After the type definition, the specified message structure should also be | ||
2793 | described in the header file, e.g. transport.h in our case. | ||
2794 | |||
2795 | @example | ||
2796 | struct AddressLookupMessage @{ | ||
2797 | struct GNUNET_MessageHeader header; | ||
2798 | int32_t numeric_only GNUNET_PACKED; | ||
2799 | struct GNUNET_TIME_AbsoluteNBO timeout; | ||
2800 | uint32_t addrlen GNUNET_PACKED; | ||
2801 | /* followed by 'addrlen' bytes of the actual address, then | ||
2802 | followed by the 0-terminated name of the transport */ @}; | ||
2803 | GNUNET_NETWORK_STRUCT_END | ||
2804 | @end example | ||
2805 | |||
2806 | |||
2807 | Please note @code{GNUNET_NETWORK_STRUCT_BEGIN} and @code{GNUNET_PACKED} | ||
2808 | which both ensure correct alignment when sending structs over the network. | ||
2809 | |||
2810 | @menu | ||
2811 | @end menu | ||
2812 | |||
2813 | @c *********************************************************************** | ||
2814 | @node Client - Establish connection | ||
2815 | @subsubsection Client - Establish connection | ||
2816 | |||
2817 | |||
2818 | |||
2819 | At first, on the client side, the underlying API is employed to create a | ||
2820 | new connection to a service, in our example the transport service would be | ||
2821 | connected. | ||
2822 | |||
2823 | @example | ||
2824 | struct GNUNET_CLIENT_Connection *client; | ||
2825 | client = GNUNET_CLIENT_connect ("transport", cfg); | ||
2826 | @end example | ||
2827 | |||
2828 | @c *********************************************************************** | ||
2829 | @node Client - Initialize request message | ||
2830 | @subsubsection Client - Initialize request message | ||
2831 | |||
2832 | |||
2833 | When the connection is ready, we initialize the message. In this step, | ||
2834 | all the fields of the message should be properly initialized, namely the | ||
2835 | size, type, and some extra user-defined data, such as timeout, name of | ||
2836 | transport, address and name of transport. | ||
2837 | |||
2838 | @example | ||
2839 | struct AddressLookupMessage *msg; | ||
2840 | size_t len = sizeof (struct AddressLookupMessage) | ||
2841 | + addressLen | ||
2842 | + strlen (nameTrans) | ||
2843 | + 1; | ||
2844 | msg->header->size = htons (len); | ||
2845 | msg->header->type = htons | ||
2846 | (GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_LOOKUP); | ||
2847 | msg->timeout = GNUNET_TIME_absolute_hton (abs_timeout); | ||
2848 | msg->addrlen = htonl (addressLen); | ||
2849 | char *addrbuf = (char *) &msg[1]; | ||
2850 | memcpy (addrbuf, address, addressLen); | ||
2851 | char *tbuf = &addrbuf[addressLen]; | ||
2852 | memcpy (tbuf, nameTrans, strlen (nameTrans) + 1); | ||
2853 | @end example | ||
2854 | |||
2855 | Note that, here the functions @code{htonl}, @code{htons} and | ||
2856 | @code{GNUNET_TIME_absolute_hton} are applied to convert little endian | ||
2857 | into big endian, about the usage of the big/small endian order and the | ||
2858 | corresponding conversion function please refer to Introduction of | ||
2859 | Big Endian and Little Endian. | ||
2860 | |||
2861 | @c *********************************************************************** | ||
2862 | @node Client - Send request and receive response | ||
2863 | @subsubsection Client - Send request and receive response | ||
2864 | |||
2865 | |||
2866 | @b{FIXME: This is very outdated, see the tutorial for the current API!} | ||
2867 | |||
2868 | Next, the client would send the constructed message as a request to the | ||
2869 | service and wait for the response from the service. To accomplish this | ||
2870 | goal, there are a number of API calls that can be used. In this example, | ||
2871 | @code{GNUNET_CLIENT_transmit_and_get_response} is chosen as the most | ||
2872 | appropriate function to use. | ||
2873 | |||
2874 | @example | ||
2875 | GNUNET_CLIENT_transmit_and_get_response | ||
2876 | (client, msg->header, timeout, GNUNET_YES, &address_response_processor, | ||
2877 | arp_ctx); | ||
2878 | @end example | ||
2879 | |||
2880 | the argument @code{address_response_processor} is a function with | ||
2881 | @code{GNUNET_CLIENT_MessageHandler} type, which is used to process the | ||
2882 | reply message from the service. | ||
2883 | |||
2884 | @node Server - Startup service | ||
2885 | @subsubsection Server - Startup service | ||
2886 | |||
2887 | After receiving the request message, we run a standard GNUnet service | ||
2888 | startup sequence using @code{GNUNET_SERVICE_run}, as follows, | ||
2889 | |||
2890 | @example | ||
2891 | int main(int argc, char**argv) @{ | ||
2892 | GNUNET_SERVICE_run(argc, argv, "transport" | ||
2893 | GNUNET_SERVICE_OPTION_NONE, &run, NULL)); @} | ||
2894 | @end example | ||
2895 | |||
2896 | @c *********************************************************************** | ||
2897 | @node Server - Add new handles for specified messages | ||
2898 | @subsubsection Server - Add new handles for specified messages | ||
2899 | |||
2900 | |||
2901 | in the function above the argument @code{run} is used to initiate | ||
2902 | transport service,and defined like this: | ||
2903 | |||
2904 | @example | ||
2905 | static void run (void *cls, | ||
2906 | struct GNUNET_SERVER_Handle *serv, | ||
2907 | const struct GNUNET_CONFIGURATION_Handle *cfg) @{ | ||
2908 | GNUNET_SERVER_add_handlers (serv, handlers); @} | ||
2909 | @end example | ||
2910 | |||
2911 | |||
2912 | Here, @code{GNUNET_SERVER_add_handlers} must be called in the run | ||
2913 | function to add new handlers in the service. The parameter | ||
2914 | @code{handlers} is a list of @code{struct GNUNET_SERVER_MessageHandler} | ||
2915 | to tell the service which function should be called when a particular | ||
2916 | type of message is received, and should be defined in this way: | ||
2917 | |||
2918 | @example | ||
2919 | static struct GNUNET_SERVER_MessageHandler handlers[] = @{ | ||
2920 | @{&handle_start, | ||
2921 | NULL, | ||
2922 | GNUNET_MESSAGE_TYPE_TRANSPORT_START, | ||
2923 | 0@}, | ||
2924 | @{&handle_send, | ||
2925 | NULL, | ||
2926 | GNUNET_MESSAGE_TYPE_TRANSPORT_SEND, | ||
2927 | 0@}, | ||
2928 | @{&handle_try_connect, | ||
2929 | NULL, | ||
2930 | GNUNET_MESSAGE_TYPE_TRANSPORT_TRY_CONNECT, | ||
2931 | sizeof (struct TryConnectMessage) | ||
2932 | @}, | ||
2933 | @{&handle_address_lookup, | ||
2934 | NULL, | ||
2935 | GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_LOOKUP, | ||
2936 | 0@}, | ||
2937 | @{NULL, | ||
2938 | NULL, | ||
2939 | 0, | ||
2940 | 0@} | ||
2941 | @}; | ||
2942 | @end example | ||
2943 | |||
2944 | |||
2945 | As shown, the first member of the struct in the first area is a callback | ||
2946 | function, which is called to process the specified message types, given | ||
2947 | as the third member. The second parameter is the closure for the callback | ||
2948 | function, which is set to @code{NULL} in most cases, and the last | ||
2949 | parameter is the expected size of the message of this type, usually we | ||
2950 | set it to 0 to accept variable size, for special cases the exact size of | ||
2951 | the specified message also can be set. In addition, the terminator sign | ||
2952 | depicted as @code{@{NULL, NULL, 0, 0@}} is set in the last area. | ||
2953 | |||
2954 | @c *********************************************************************** | ||
2955 | @node Server - Process request message | ||
2956 | @subsubsection Server - Process request message | ||
2957 | |||
2958 | |||
2959 | After the initialization of transport service, the request message would | ||
2960 | be processed. Before handling the main message data, the validity of this | ||
2961 | message should be checked out, e.g., to check whether the size of message | ||
2962 | is correct. | ||
2963 | |||
2964 | @example | ||
2965 | size = ntohs (message->size); | ||
2966 | if (size < sizeof (struct AddressLookupMessage)) @{ | ||
2967 | GNUNET_break_op (0); | ||
2968 | GNUNET_SERVER_receive_done (client, GNUNET_SYSERR); | ||
2969 | return; @} | ||
2970 | @end example | ||
2971 | |||
2972 | |||
2973 | Note that, opposite to the construction method of the request message in | ||
2974 | the client, in the server the function @code{nothl} and @code{ntohs} | ||
2975 | should be employed during the extraction of the data from the message, so | ||
2976 | that the data in big endian order can be converted back into little | ||
2977 | endian order. See more in detail please refer to Introduction of | ||
2978 | Big Endian and Little Endian. | ||
2979 | |||
2980 | Moreover in this example, the name of the transport stored in the message | ||
2981 | is a 0-terminated string, so we should also check whether the name of the | ||
2982 | transport in the received message is 0-terminated: | ||
2983 | |||
2984 | @example | ||
2985 | nameTransport = (const char *) &address[addressLen]; | ||
2986 | if (nameTransport[size - sizeof | ||
2987 | (struct AddressLookupMessage) | ||
2988 | - addressLen - 1] != '\0') @{ | ||
2989 | GNUNET_break_op (0); | ||
2990 | GNUNET_SERVER_receive_done (client, | ||
2991 | GNUNET_SYSERR); | ||
2992 | return; @} | ||
2993 | @end example | ||
2994 | |||
2995 | Here, @code{GNUNET_SERVER_receive_done} should be called to tell the | ||
2996 | service that the request is done and can receive the next message. The | ||
2997 | argument @code{GNUNET_SYSERR} here indicates that the service didn't | ||
2998 | understand the request message, and the processing of this request would | ||
2999 | be terminated. | ||
3000 | |||
3001 | In comparison to the aforementioned situation, when the argument is equal | ||
3002 | to @code{GNUNET_OK}, the service would continue to process the request | ||
3003 | message. | ||
3004 | |||
3005 | @c *********************************************************************** | ||
3006 | @node Server - Response to client | ||
3007 | @subsubsection Server - Response to client | ||
3008 | |||
3009 | |||
3010 | Once the processing of current request is done, the server should give the | ||
3011 | response to the client. A new @code{struct AddressLookupMessage} would be | ||
3012 | produced by the server in a similar way as the client did and sent to the | ||
3013 | client, but here the type should be | ||
3014 | @code{GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_REPLY} rather than | ||
3015 | @code{GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_LOOKUP} in client. | ||
3016 | @example | ||
3017 | struct AddressLookupMessage *msg; | ||
3018 | size_t len = sizeof (struct AddressLookupMessage) | ||
3019 | + addressLen | ||
3020 | + strlen (nameTrans) + 1; | ||
3021 | msg->header->size = htons (len); | ||
3022 | msg->header->type = htons | ||
3023 | (GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_REPLY); | ||
3024 | |||
3025 | // ... | ||
3026 | |||
3027 | struct GNUNET_SERVER_TransmitContext *tc; | ||
3028 | tc = GNUNET_SERVER_transmit_context_create (client); | ||
3029 | GNUNET_SERVER_transmit_context_append_data | ||
3030 | (tc, | ||
3031 | NULL, | ||
3032 | 0, | ||
3033 | GNUNET_MESSAGE_TYPE_TRANSPORT_ADDRESS_REPLY); | ||
3034 | GNUNET_SERVER_transmit_context_run (tc, rtimeout); | ||
3035 | @end example | ||
3036 | |||
3037 | |||
3038 | Note that, there are also a number of other APIs provided to the service | ||
3039 | to send the message. | ||
3040 | |||
3041 | @c *********************************************************************** | ||
3042 | @node Server - Notification of clients | ||
3043 | @subsubsection Server - Notification of clients | ||
3044 | |||
3045 | |||
3046 | Often a service needs to (repeatedly) transmit notifications to a client | ||
3047 | or a group of clients. In these cases, the client typically has once | ||
3048 | registered for a set of events and then needs to receive a message | ||
3049 | whenever such an event happens (until the client disconnects). The use of | ||
3050 | a notification context can help manage message queues to clients and | ||
3051 | handle disconnects. Notification contexts can be used to send | ||
3052 | individualized messages to a particular client or to broadcast messages | ||
3053 | to a group of clients. An individualized notification might look like | ||
3054 | this: | ||
3055 | |||
3056 | @example | ||
3057 | GNUNET_SERVER_notification_context_unicast(nc, | ||
3058 | client, | ||
3059 | msg, | ||
3060 | GNUNET_YES); | ||
3061 | @end example | ||
3062 | |||
3063 | |||
3064 | Note that after processing the original registration message for | ||
3065 | notifications, the server code still typically needs to call | ||
3066 | @code{GNUNET_SERVER_receive_done} so that the client can transmit further | ||
3067 | messages to the server. | ||
3068 | |||
3069 | @c *********************************************************************** | ||
3070 | @node Conversion between Network Byte Order (Big Endian) and Host Byte Order | ||
3071 | @subsubsection Conversion between Network Byte Order (Big Endian) and Host Byte Order | ||
3072 | @c %** subsub? it's a referenced page on the ipc document. | ||
3073 | |||
3074 | |||
3075 | Here we can simply comprehend big endian and little endian as Network Byte | ||
3076 | Order and Host Byte Order respectively. What is the difference between | ||
3077 | both two? | ||
3078 | |||
3079 | Usually in our host computer we store the data byte as Host Byte Order, | ||
3080 | for example, we store a integer in the RAM which might occupies 4 Byte, | ||
3081 | as Host Byte Order the higher Byte would be stored at the lower address | ||
3082 | of RAM, and the lower Byte would be stored at the higher address of RAM. | ||
3083 | However, contrast to this, Network Byte Order just take the totally | ||
3084 | opposite way to store the data, says, it will store the lower Byte at the | ||
3085 | lower address, and the higher Byte will stay at higher address. | ||
3086 | |||
3087 | For the current communication of network, we normally exchange the | ||
3088 | information by surveying the data package, every two host wants to | ||
3089 | communicate with each other must send and receive data package through | ||
3090 | network. In order to maintain the identity of data through the | ||
3091 | transmission in the network, the order of the Byte storage must changed | ||
3092 | before sending and after receiving the data. | ||
3093 | |||
3094 | There ten convenient functions to realize the conversion of Byte Order in | ||
3095 | GNUnet, as following: | ||
3096 | |||
3097 | @table @asis | ||
3098 | |||
3099 | @item uint16_t htons(uint16_t hostshort) Convert host byte order to net | ||
3100 | byte order with short int | ||
3101 | @item uint32_t htonl(uint32_t hostlong) Convert host byte | ||
3102 | order to net byte order with long int | ||
3103 | @item uint16_t ntohs(uint16_t netshort) | ||
3104 | Convert net byte order to host byte order with short int | ||
3105 | @item uint32_t | ||
3106 | ntohl(uint32_t netlong) Convert net byte order to host byte order with | ||
3107 | long int | ||
3108 | @item unsigned long long GNUNET_ntohll (unsigned long long netlonglong) | ||
3109 | Convert net byte order to host byte order with long long int | ||
3110 | @item unsigned long long GNUNET_htonll (unsigned long long hostlonglong) | ||
3111 | Convert host byte order to net byte order with long long int | ||
3112 | @item struct GNUNET_TIME_RelativeNBO GNUNET_TIME_relative_hton | ||
3113 | (struct GNUNET_TIME_Relative a) Convert relative time to network byte | ||
3114 | order. | ||
3115 | @item struct GNUNET_TIME_Relative GNUNET_TIME_relative_ntoh | ||
3116 | (struct GNUNET_TIME_RelativeNBO a) Convert relative time from network | ||
3117 | byte order. | ||
3118 | @item struct GNUNET_TIME_AbsoluteNBO GNUNET_TIME_absolute_hton | ||
3119 | (struct GNUNET_TIME_Absolute a) Convert relative time to network byte | ||
3120 | order. | ||
3121 | @item struct GNUNET_TIME_Absolute GNUNET_TIME_absolute_ntoh | ||
3122 | (struct GNUNET_TIME_AbsoluteNBO a) Convert relative time from network | ||
3123 | byte order. | ||
3124 | @end table | ||
3125 | |||
3126 | @cindex Cryptography API | ||
3127 | @node Cryptography API | ||
3128 | @subsection Cryptography API | ||
3129 | |||
3130 | |||
3131 | The gnunetutil APIs provides the cryptographic primitives used in GNUnet. | ||
3132 | GNUnet uses 2048 bit RSA keys for the session key exchange and for signing | ||
3133 | messages by peers and most other public-key operations. Most researchers | ||
3134 | in cryptography consider 2048 bit RSA keys as secure and practically | ||
3135 | unbreakable for a long time. The API provides functions to create a fresh | ||
3136 | key pair, read a private key from a file (or create a new file if the | ||
3137 | file does not exist), encrypt, decrypt, sign, verify and extraction of | ||
3138 | the public key into a format suitable for network transmission. | ||
3139 | |||
3140 | For the encryption of files and the actual data exchanged between peers | ||
3141 | GNUnet uses 256-bit AES encryption. Fresh, session keys are negotiated | ||
3142 | for every new connection.@ Again, there is no published technique to | ||
3143 | break this cipher in any realistic amount of time. The API provides | ||
3144 | functions for generation of keys, validation of keys (important for | ||
3145 | checking that decryptions using RSA succeeded), encryption and decryption. | ||
3146 | |||
3147 | GNUnet uses SHA-512 for computing one-way hash codes. The API provides | ||
3148 | functions to compute a hash over a block in memory or over a file on disk. | ||
3149 | |||
3150 | The crypto API also provides functions for randomizing a block of memory, | ||
3151 | obtaining a single random number and for generating a permutation of the | ||
3152 | numbers 0 to n-1. Random number generation distinguishes between WEAK and | ||
3153 | STRONG random number quality; WEAK random numbers are pseudo-random | ||
3154 | whereas STRONG random numbers use entropy gathered from the operating | ||
3155 | system. | ||
3156 | |||
3157 | Finally, the crypto API provides a means to deterministically generate a | ||
3158 | 1024-bit RSA key from a hash code. These functions should most likely not | ||
3159 | be used by most applications; most importantly, | ||
3160 | GNUNET_CRYPTO_rsa_key_create_from_hash does not create an RSA-key that | ||
3161 | should be considered secure for traditional applications of RSA. | ||
3162 | |||
3163 | @cindex Message Queue API | ||
3164 | @node Message Queue API | ||
3165 | @subsection Message Queue API | ||
3166 | |||
3167 | |||
3168 | @strong{ Introduction }@ | ||
3169 | Often, applications need to queue messages that | ||
3170 | are to be sent to other GNUnet peers, clients or services. As all of | ||
3171 | GNUnet's message-based communication APIs, by design, do not allow | ||
3172 | messages to be queued, it is common to implement custom message queues | ||
3173 | manually when they are needed. However, writing very similar code in | ||
3174 | multiple places is tedious and leads to code duplication. | ||
3175 | |||
3176 | MQ (for Message Queue) is an API that provides the functionality to | ||
3177 | implement and use message queues. We intend to eventually replace all of | ||
3178 | the custom message queue implementations in GNUnet with MQ. | ||
3179 | |||
3180 | @strong{ Basic Concepts }@ | ||
3181 | The two most important entities in MQ are queues and envelopes. | ||
3182 | |||
3183 | Every queue is backed by a specific implementation (e.g. for mesh, stream, | ||
3184 | connection, server client, etc.) that will actually deliver the queued | ||
3185 | messages. For convenience,@ some queues also allow to specify a list of | ||
3186 | message handlers. The message queue will then also wait for incoming | ||
3187 | messages and dispatch them appropriately. | ||
3188 | |||
3189 | An envelope holds the memory for a message, as well as metadata | ||
3190 | (Where is the envelope queued? What should happen after it has been | ||
3191 | sent?). Any envelope can only be queued in one message queue. | ||
3192 | |||
3193 | @strong{ Creating Queues }@ | ||
3194 | The following is a list of currently available message queues. Note that | ||
3195 | to avoid layering issues, message queues for higher level APIs are not | ||
3196 | part of @code{libgnunetutil}, but@ the respective API itself provides the | ||
3197 | queue implementation. | ||
3198 | |||
3199 | @table @asis | ||
3200 | |||
3201 | @item @code{GNUNET_MQ_queue_for_connection_client} | ||
3202 | Transmits queued messages over a @code{GNUNET_CLIENT_Connection} handle. | ||
3203 | Also supports receiving with message handlers. | ||
3204 | |||
3205 | @item @code{GNUNET_MQ_queue_for_server_client} | ||
3206 | Transmits queued messages over a @code{GNUNET_SERVER_Client} handle. Does | ||
3207 | not support incoming message handlers. | ||
3208 | |||
3209 | @item @code{GNUNET_MESH_mq_create} Transmits queued messages over a | ||
3210 | @code{GNUNET_MESH_Tunnel} handle. Does not support incoming message | ||
3211 | handlers. | ||
3212 | |||
3213 | @item @code{GNUNET_MQ_queue_for_callbacks} This is the most general | ||
3214 | implementation. Instead of delivering and receiving messages with one of | ||
3215 | GNUnet's communication APIs, implementation callbacks are called. Refer to | ||
3216 | "Implementing Queues" for a more detailed explanation. | ||
3217 | @end table | ||
3218 | |||
3219 | |||
3220 | @strong{ Allocating Envelopes }@ | ||
3221 | A GNUnet message (as defined by the GNUNET_MessageHeader) has three | ||
3222 | parts: The size, the type, and the body. | ||
3223 | |||
3224 | MQ provides macros to allocate an envelope containing a message | ||
3225 | conveniently, automatically setting the size and type fields of the | ||
3226 | message. | ||
3227 | |||
3228 | Consider the following simple message, with the body consisting of a | ||
3229 | single number value. | ||
3230 | @c why the empty code function? | ||
3231 | @code{} | ||
3232 | |||
3233 | @example | ||
3234 | struct NumberMessage @{ | ||
3235 | /** Type: GNUNET_MESSAGE_TYPE_EXAMPLE_1 */ | ||
3236 | struct GNUNET_MessageHeader header; | ||
3237 | uint32_t number GNUNET_PACKED; | ||
3238 | @}; | ||
3239 | @end example | ||
3240 | |||
3241 | An envelope containing an instance of the NumberMessage can be | ||
3242 | constructed like this: | ||
3243 | |||
3244 | @example | ||
3245 | struct GNUNET_MQ_Envelope *ev; | ||
3246 | struct NumberMessage *msg; | ||
3247 | ev = GNUNET_MQ_msg (msg, GNUNET_MESSAGE_TYPE_EXAMPLE_1); | ||
3248 | msg->number = htonl (42); | ||
3249 | @end example | ||
3250 | |||
3251 | In the above code, @code{GNUNET_MQ_msg} is a macro. The return value is | ||
3252 | the newly allocated envelope. The first argument must be a pointer to some | ||
3253 | @code{struct} containing a @code{struct GNUNET_MessageHeader header} | ||
3254 | field, while the second argument is the desired message type, in host | ||
3255 | byte order. | ||
3256 | |||
3257 | The @code{msg} pointer now points to an allocated message, where the | ||
3258 | message type and the message size are already set. The message's size is | ||
3259 | inferred from the type of the @code{msg} pointer: It will be set to | ||
3260 | 'sizeof(*msg)', properly converted to network byte order. | ||
3261 | |||
3262 | If the message body's size is dynamic, then the macro | ||
3263 | @code{GNUNET_MQ_msg_extra} can be used to allocate an envelope whose | ||
3264 | message has additional space allocated after the @code{msg} structure. | ||
3265 | |||
3266 | If no structure has been defined for the message, | ||
3267 | @code{GNUNET_MQ_msg_header_extra} can be used to allocate additional space | ||
3268 | after the message header. The first argument then must be a pointer to a | ||
3269 | @code{GNUNET_MessageHeader}. | ||
3270 | |||
3271 | @strong{Envelope Properties}@ | ||
3272 | A few functions in MQ allow to set additional properties on envelopes: | ||
3273 | |||
3274 | @table @asis | ||
3275 | |||
3276 | @item @code{GNUNET_MQ_notify_sent} Allows to specify a function that will | ||
3277 | be called once the envelope's message has been sent irrevocably. | ||
3278 | An envelope can be canceled precisely up to the@ point where the notify | ||
3279 | sent callback has been called. | ||
3280 | |||
3281 | @item @code{GNUNET_MQ_disable_corking} No corking will be used when | ||
3282 | sending the message. Not every@ queue supports this flag, per default, | ||
3283 | envelopes are sent with corking.@ | ||
3284 | |||
3285 | @end table | ||
3286 | |||
3287 | |||
3288 | @strong{Sending Envelopes}@ | ||
3289 | Once an envelope has been constructed, it can be queued for sending with | ||
3290 | @code{GNUNET_MQ_send}. | ||
3291 | |||
3292 | Note that in order to avoid memory leaks, an envelope must either be sent | ||
3293 | (the queue will free it) or destroyed explicitly with | ||
3294 | @code{GNUNET_MQ_discard}. | ||
3295 | |||
3296 | @strong{Canceling Envelopes}@ | ||
3297 | An envelope queued with @code{GNUNET_MQ_send} can be canceled with | ||
3298 | @code{GNUNET_MQ_cancel}. Note that after the notify sent callback has | ||
3299 | been called, canceling a message results in undefined behavior. | ||
3300 | Thus it is unsafe to cancel an envelope that does not have a notify sent | ||
3301 | callback. When canceling an envelope, it is not necessary@ to call | ||
3302 | @code{GNUNET_MQ_discard}, and the envelope can't be sent again. | ||
3303 | |||
3304 | @strong{ Implementing Queues }@ | ||
3305 | @code{TODO} | ||
3306 | |||
3307 | @cindex Service API | ||
3308 | @node Service API | ||
3309 | @subsection Service API | ||
3310 | |||
3311 | |||
3312 | Most GNUnet code lives in the form of services. Services are processes | ||
3313 | that offer an API for other components of the system to build on. Those | ||
3314 | other components can be command-line tools for users, graphical user | ||
3315 | interfaces or other services. Services provide their API using an IPC | ||
3316 | protocol. For this, each service must listen on either a TCP port or a | ||
3317 | UNIX domain socket; for this, the service implementation uses the server | ||
3318 | API. This use of server is exposed directly to the users of the service | ||
3319 | API. Thus, when using the service API, one is usually also often using | ||
3320 | large parts of the server API. The service API provides various | ||
3321 | convenience functions, such as parsing command-line arguments and the | ||
3322 | configuration file, which are not found in the server API. | ||
3323 | The dual to the service/server API is the client API, which can be used to | ||
3324 | access services. | ||
3325 | |||
3326 | The most common way to start a service is to use the | ||
3327 | @code{GNUNET_SERVICE_run} function from the program's main function. | ||
3328 | @code{GNUNET_SERVICE_run} will then parse the command line and | ||
3329 | configuration files and, based on the options found there, | ||
3330 | start the server. It will then give back control to the main | ||
3331 | program, passing the server and the configuration to the | ||
3332 | @code{GNUNET_SERVICE_Main} callback. @code{GNUNET_SERVICE_run} | ||
3333 | will also take care of starting the scheduler loop. | ||
3334 | If this is inappropriate (for example, because the scheduler loop | ||
3335 | is already running), @code{GNUNET_SERVICE_start} and | ||
3336 | related functions provide an alternative to @code{GNUNET_SERVICE_run}. | ||
3337 | |||
3338 | When starting a service, the service_name option is used to determine | ||
3339 | which sections in the configuration file should be used to configure the | ||
3340 | service. A typical value here is the name of the @file{src/} | ||
3341 | sub-directory, for example @file{statistics}. | ||
3342 | The same string would also be given to | ||
3343 | @code{GNUNET_CLIENT_connect} to access the service. | ||
3344 | |||
3345 | Once a service has been initialized, the program should use the | ||
3346 | @code{GNUNET_SERVICE_Main} callback to register message handlers | ||
3347 | using @code{GNUNET_SERVER_add_handlers}. | ||
3348 | The service will already have registered a handler for the | ||
3349 | "TEST" message. | ||
3350 | |||
3351 | @findex GNUNET_SERVICE_Options | ||
3352 | The option bitfield (@code{enum GNUNET_SERVICE_Options}) | ||
3353 | determines how a service should behave during shutdown. | ||
3354 | There are three key strategies: | ||
3355 | |||
3356 | @table @asis | ||
3357 | |||
3358 | @item instant (@code{GNUNET_SERVICE_OPTION_NONE}) | ||
3359 | Upon receiving the shutdown | ||
3360 | signal from the scheduler, the service immediately terminates the server, | ||
3361 | closing all existing connections with clients. | ||
3362 | @item manual (@code{GNUNET_SERVICE_OPTION_MANUAL_SHUTDOWN}) | ||
3363 | The service does nothing by itself | ||
3364 | during shutdown. The main program will need to take the appropriate | ||
3365 | action by calling GNUNET_SERVER_destroy or GNUNET_SERVICE_stop (depending | ||
3366 | on how the service was initialized) to terminate the service. This method | ||
3367 | is used by gnunet-service-arm and rather uncommon. | ||
3368 | @item soft (@code{GNUNET_SERVICE_OPTION_SOFT_SHUTDOWN}) | ||
3369 | Upon receiving the shutdown signal from the scheduler, | ||
3370 | the service immediately tells the server to stop | ||
3371 | listening for incoming clients. Requests from normal existing clients are | ||
3372 | still processed and the server/service terminates once all normal clients | ||
3373 | have disconnected. Clients that are not expected to ever disconnect (such | ||
3374 | as clients that monitor performance values) can be marked as 'monitor' | ||
3375 | clients using GNUNET_SERVER_client_mark_monitor. Those clients will | ||
3376 | continue to be processed until all 'normal' clients have disconnected. | ||
3377 | Then, the server will terminate, closing the monitor connections. | ||
3378 | This mode is for example used by 'statistics', allowing existing 'normal' | ||
3379 | clients to set (possibly persistent) statistic values before terminating. | ||
3380 | |||
3381 | @end table | ||
3382 | |||
3383 | @c *********************************************************************** | ||
3384 | @node Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps | ||
3385 | @subsection Optimizing Memory Consumption of GNUnet's (Multi-) Hash Maps | ||
3386 | |||
3387 | |||
3388 | A commonly used data structure in GNUnet is a (multi-)hash map. It is most | ||
3389 | often used to map a peer identity to some data structure, but also to map | ||
3390 | arbitrary keys to values (for example to track requests in the distributed | ||
3391 | hash table or in file-sharing). As it is commonly used, the DHT is | ||
3392 | actually sometimes responsible for a large share of GNUnet's overall | ||
3393 | memory consumption (for some processes, 30% is not uncommon). The | ||
3394 | following text documents some API quirks (and their implications for | ||
3395 | applications) that were recently introduced to minimize the footprint of | ||
3396 | the hash map. | ||
3397 | |||
3398 | |||
3399 | @c *********************************************************************** | ||
3400 | @menu | ||
3401 | * Analysis:: | ||
3402 | * Solution:: | ||
3403 | * Migration:: | ||
3404 | * Conclusion:: | ||
3405 | * Availability:: | ||
3406 | @end menu | ||
3407 | |||
3408 | @node Analysis | ||
3409 | @subsubsection Analysis | ||
3410 | |||
3411 | |||
3412 | The main reason for the "excessive" memory consumption by the hash map is | ||
3413 | that GNUnet uses 512-bit cryptographic hash codes --- and the | ||
3414 | (multi-)hash map also uses the same 512-bit 'struct GNUNET_HashCode'. As | ||
3415 | a result, storing just the keys requires 64 bytes of memory for each key. | ||
3416 | As some applications like to keep a large number of entries in the hash | ||
3417 | map (after all, that's what maps are good for), 64 bytes per hash is | ||
3418 | significant: keeping a pointer to the value and having a linked list for | ||
3419 | collisions consume between 8 and 16 bytes, and 'malloc' may add about the | ||
3420 | same overhead per allocation, putting us in the 16 to 32 byte per entry | ||
3421 | ballpark. Adding a 64-byte key then triples the overall memory | ||
3422 | requirement for the hash map. | ||
3423 | |||
3424 | To make things "worse", most of the time storing the key in the hash map | ||
3425 | is not required: it is typically already in memory elsewhere! In most | ||
3426 | cases, the values stored in the hash map are some application-specific | ||
3427 | struct that _also_ contains the hash. Here is a simplified example: | ||
3428 | |||
3429 | @example | ||
3430 | struct MyValue @{ | ||
3431 | struct GNUNET_HashCode key; | ||
3432 | unsigned int my_data; @}; | ||
3433 | |||
3434 | // ... | ||
3435 | val = GNUNET_malloc (sizeof (struct MyValue)); | ||
3436 | val->key = key; | ||
3437 | val->my_data = 42; | ||
3438 | GNUNET_CONTAINER_multihashmap_put (map, &key, val, ...); | ||
3439 | @end example | ||
3440 | |||
3441 | This is a common pattern as later the entries might need to be removed, | ||
3442 | and at that time it is convenient to have the key immediately at hand: | ||
3443 | |||
3444 | @example | ||
3445 | GNUNET_CONTAINER_multihashmap_remove (map, &val->key, val); | ||
3446 | @end example | ||
3447 | |||
3448 | |||
3449 | Note that here we end up with two times 64 bytes for the key, plus maybe | ||
3450 | 64 bytes total for the rest of the 'struct MyValue' and the map entry in | ||
3451 | the hash map. The resulting redundant storage of the key increases | ||
3452 | overall memory consumption per entry from the "optimal" 128 bytes to 192 | ||
3453 | bytes. This is not just an extreme example: overheads in practice are | ||
3454 | actually sometimes close to those highlighted in this example. This is | ||
3455 | especially true for maps with a significant number of entries, as there | ||
3456 | we tend to really try to keep the entries small. | ||
3457 | |||
3458 | @c *********************************************************************** | ||
3459 | @node Solution | ||
3460 | @subsubsection Solution | ||
3461 | |||
3462 | |||
3463 | The solution that has now been implemented is to @strong{optionally} | ||
3464 | allow the hash map to not make a (deep) copy of the hash but instead have | ||
3465 | a pointer to the hash/key in the entry. This reduces the memory | ||
3466 | consumption for the key from 64 bytes to 4 to 8 bytes. However, it can | ||
3467 | also only work if the key is actually stored in the entry (which is the | ||
3468 | case most of the time) and if the entry does not modify the key (which in | ||
3469 | all of the code I'm aware of has been always the case if there key is | ||
3470 | stored in the entry). Finally, when the client stores an entry in the | ||
3471 | hash map, it @strong{must} provide a pointer to the key within the entry, | ||
3472 | not just a pointer to a transient location of the key. If | ||
3473 | the client code does not meet these requirements, the result is a dangling | ||
3474 | pointer and undefined behavior of the (multi-)hash map API. | ||
3475 | |||
3476 | @c *********************************************************************** | ||
3477 | @node Migration | ||
3478 | @subsubsection Migration | ||
3479 | |||
3480 | |||
3481 | To use the new feature, first check that the values contain the respective | ||
3482 | key (and never modify it). Then, all calls to | ||
3483 | @code{GNUNET_CONTAINER_multihashmap_put} on the respective map must be | ||
3484 | audited and most likely changed to pass a pointer into the value's struct. | ||
3485 | For the initial example, the new code would look like this: | ||
3486 | |||
3487 | @example | ||
3488 | struct MyValue @{ | ||
3489 | struct GNUNET_HashCode key; | ||
3490 | unsigned int my_data; @}; | ||
3491 | |||
3492 | // ... | ||
3493 | val = GNUNET_malloc (sizeof (struct MyValue)); | ||
3494 | val->key = key; val->my_data = 42; | ||
3495 | GNUNET_CONTAINER_multihashmap_put (map, &val->key, val, ...); | ||
3496 | @end example | ||
3497 | |||
3498 | |||
3499 | Note that @code{&val} was changed to @code{&val->key} in the argument to | ||
3500 | the @code{put} call. This is critical as often @code{key} is on the stack | ||
3501 | or in some other transient data structure and thus having the hash map | ||
3502 | keep a pointer to @code{key} would not work. Only the key inside of | ||
3503 | @code{val} has the same lifetime as the entry in the map (this must of | ||
3504 | course be checked as well). Naturally, @code{val->key} must be | ||
3505 | initialized before the @code{put} call. Once all @code{put} calls have | ||
3506 | been converted and double-checked, you can change the call to create the | ||
3507 | hash map from | ||
3508 | |||
3509 | @example | ||
3510 | map = | ||
3511 | GNUNET_CONTAINER_multihashmap_create (SIZE, GNUNET_NO); | ||
3512 | @end example | ||
3513 | |||
3514 | to | ||
3515 | |||
3516 | @example | ||
3517 | map = GNUNET_CONTAINER_multihashmap_create (SIZE, GNUNET_YES); | ||
3518 | @end example | ||
3519 | |||
3520 | If everything was done correctly, you now use about 60 bytes less memory | ||
3521 | per entry in @code{map}. However, if now (or in the future) any call to | ||
3522 | @code{put} does not ensure that the given key is valid until the entry is | ||
3523 | removed from the map, undefined behavior is likely to be observed. | ||
3524 | |||
3525 | @c *********************************************************************** | ||
3526 | @node Conclusion | ||
3527 | @subsubsection Conclusion | ||
3528 | |||
3529 | |||
3530 | The new optimization can is often applicable and can result in a | ||
3531 | reduction in memory consumption of up to 30% in practice. However, it | ||
3532 | makes the code less robust as additional invariants are imposed on the | ||
3533 | multi hash map client. Thus applications should refrain from enabling the | ||
3534 | new mode unless the resulting performance increase is deemed significant | ||
3535 | enough. In particular, it should generally not be used in new code (wait | ||
3536 | at least until benchmarks exist). | ||
3537 | |||
3538 | @c *********************************************************************** | ||
3539 | @node Availability | ||
3540 | @subsubsection Availability | ||
3541 | |||
3542 | |||
3543 | The new multi hash map code was committed in SVN 24319 (which made its | ||
3544 | way into GNUnet version 0.9.4). | ||
3545 | Various subsystems (transport, core, dht, file-sharing) were | ||
3546 | previously audited and modified to take advantage of the new capability. | ||
3547 | In particular, memory consumption of the file-sharing service is expected | ||
3548 | to drop by 20-30% due to this change. | ||
3549 | |||
3550 | |||
3551 | @cindex CONTAINER_MDLL API | ||
3552 | @node CONTAINER_MDLL API | ||
3553 | @subsection CONTAINER_MDLL API | ||
3554 | |||
3555 | |||
3556 | This text documents the GNUNET_CONTAINER_MDLL API. The | ||
3557 | GNUNET_CONTAINER_MDLL API is similar to the GNUNET_CONTAINER_DLL API in | ||
3558 | that it provides operations for the construction and manipulation of | ||
3559 | doubly-linked lists. The key difference to the (simpler) DLL-API is that | ||
3560 | the MDLL-version allows a single element (instance of a "struct") to be | ||
3561 | in multiple linked lists at the same time. | ||
3562 | |||
3563 | Like the DLL API, the MDLL API stores (most of) the data structures for | ||
3564 | the doubly-linked list with the respective elements; only the 'head' and | ||
3565 | 'tail' pointers are stored "elsewhere" --- and the application needs to | ||
3566 | provide the locations of head and tail to each of the calls in the | ||
3567 | MDLL API. The key difference for the MDLL API is that the "next" and | ||
3568 | "previous" pointers in the struct can no longer be simply called "next" | ||
3569 | and "prev" --- after all, the element may be in multiple doubly-linked | ||
3570 | lists, so we cannot just have one "next" and one "prev" pointer! | ||
3571 | |||
3572 | The solution is to have multiple fields that must have a name of the | ||
3573 | format "next_XX" and "prev_XX" where "XX" is the name of one of the | ||
3574 | doubly-linked lists. Here is a simple example: | ||
3575 | |||
3576 | @example | ||
3577 | struct MyMultiListElement @{ | ||
3578 | struct MyMultiListElement *next_ALIST; | ||
3579 | struct MyMultiListElement *prev_ALIST; | ||
3580 | struct MyMultiListElement *next_BLIST; | ||
3581 | struct MyMultiListElement *prev_BLIST; | ||
3582 | void | ||
3583 | *data; | ||
3584 | @}; | ||
3585 | @end example | ||
3586 | |||
3587 | |||
3588 | Note that by convention, we use all-uppercase letters for the list names. | ||
3589 | In addition, the program needs to have a location for the head and tail | ||
3590 | pointers for both lists, for example: | ||
3591 | |||
3592 | @example | ||
3593 | static struct MyMultiListElement *head_ALIST; | ||
3594 | static struct MyMultiListElement *tail_ALIST; | ||
3595 | static struct MyMultiListElement *head_BLIST; | ||
3596 | static struct MyMultiListElement *tail_BLIST; | ||
3597 | @end example | ||
3598 | |||
3599 | |||
3600 | Using the MDLL-macros, we can now insert an element into the ALIST: | ||
3601 | |||
3602 | @example | ||
3603 | GNUNET_CONTAINER_MDLL_insert (ALIST, head_ALIST, tail_ALIST, element); | ||
3604 | @end example | ||
3605 | |||
3606 | |||
3607 | Passing "ALIST" as the first argument to MDLL specifies which of the | ||
3608 | next/prev fields in the 'struct MyMultiListElement' should be used. The | ||
3609 | extra "ALIST" argument and the "_ALIST" in the names of the | ||
3610 | next/prev-members are the only differences between the MDDL and DLL-API. | ||
3611 | Like the DLL-API, the MDLL-API offers functions for inserting (at head, | ||
3612 | at tail, after a given element) and removing elements from the list. | ||
3613 | Iterating over the list should be done by directly accessing the | ||
3614 | "next_XX" and/or "prev_XX" members. | ||
3615 | |||
3616 | @cindex Automatic Restart Manager | ||
3617 | @cindex ARM | ||
3618 | @node Automatic Restart Manager (ARM) | ||
3619 | @section Automatic Restart Manager (ARM) | ||
3620 | |||
3621 | |||
3622 | GNUnet's Automated Restart Manager (ARM) is the GNUnet service responsible | ||
3623 | for system initialization and service babysitting. ARM starts and halts | ||
3624 | services, detects configuration changes and restarts services impacted by | ||
3625 | the changes as needed. It's also responsible for restarting services in | ||
3626 | case of crashes and is planned to incorporate automatic debugging for | ||
3627 | diagnosing service crashes providing developers insights about crash | ||
3628 | reasons. The purpose of this document is to give GNUnet developer an idea | ||
3629 | about how ARM works and how to interact with it. | ||
3630 | |||
3631 | @menu | ||
3632 | * Basic functionality:: | ||
3633 | * Key configuration options:: | ||
3634 | * ARM - Availability:: | ||
3635 | * Reliability:: | ||
3636 | @end menu | ||
3637 | |||
3638 | @c *********************************************************************** | ||
3639 | @node Basic functionality | ||
3640 | @subsection Basic functionality | ||
3641 | |||
3642 | |||
3643 | @itemize @bullet | ||
3644 | @item ARM source code can be found under "src/arm".@ Service processes are | ||
3645 | managed by the functions in "gnunet-service-arm.c" which is controlled | ||
3646 | with "gnunet-arm.c" (main function in that file is ARM's entry point). | ||
3647 | |||
3648 | @item The functions responsible for communicating with ARM , starting and | ||
3649 | stopping services -including ARM service itself- are provided by the | ||
3650 | ARM API "arm_api.c".@ Function: GNUNET_ARM_connect() returns to the caller | ||
3651 | an ARM handle after setting it to the caller's context (configuration and | ||
3652 | scheduler in use). This handle can be used afterwards by the caller to | ||
3653 | communicate with ARM. Functions GNUNET_ARM_start_service() and | ||
3654 | GNUNET_ARM_stop_service() are used for starting and stopping services | ||
3655 | respectively. | ||
3656 | |||
3657 | @item A typical example of using these basic ARM services can be found in | ||
3658 | file test_arm_api.c. The test case connects to ARM, starts it, then uses | ||
3659 | it to start a service "resolver", stops the "resolver" then stops "ARM". | ||
3660 | @end itemize | ||
3661 | |||
3662 | @c *********************************************************************** | ||
3663 | @node Key configuration options | ||
3664 | @subsection Key configuration options | ||
3665 | |||
3666 | |||
3667 | Configurations for ARM and services should be available in a .conf file | ||
3668 | (As an example, see test_arm_api_data.conf). When running ARM, the | ||
3669 | configuration file to use should be passed to the command: | ||
3670 | |||
3671 | @example | ||
3672 | $ gnunet-arm -s -c configuration_to_use.conf | ||
3673 | @end example | ||
3674 | |||
3675 | If no configuration is passed, the default configuration file will be used | ||
3676 | (see GNUNET_PREFIX/share/gnunet/defaults.conf which is created from | ||
3677 | contrib/defaults.conf).@ Each of the services is having a section starting | ||
3678 | by the service name between square brackets, for example: "[arm]". | ||
3679 | The following options configure how ARM configures or interacts with the | ||
3680 | various services: | ||
3681 | |||
3682 | @table @asis | ||
3683 | |||
3684 | @item PORT Port number on which the service is listening for incoming TCP | ||
3685 | connections. ARM will start the services should it notice a request at | ||
3686 | this port. | ||
3687 | |||
3688 | @item HOSTNAME Specifies on which host the service is deployed. Note | ||
3689 | that ARM can only start services that are running on the local system | ||
3690 | (but will not check that the hostname matches the local machine name). | ||
3691 | This option is used by the @code{gnunet_client_lib.h} implementation to | ||
3692 | determine which system to connect to. The default is "localhost". | ||
3693 | |||
3694 | @item BINARY The name of the service binary file. | ||
3695 | |||
3696 | @item OPTIONS To be passed to the service. | ||
3697 | |||
3698 | @item PREFIX A command to pre-pend to the actual command, for example, | ||
3699 | running a service with "valgrind" or "gdb" | ||
3700 | |||
3701 | @item DEBUG Run in debug mode (much verbosity). | ||
3702 | |||
3703 | @item START_ON_DEMAND ARM will listen to UNIX domain socket and/or TCP port of | ||
3704 | the service and start the service on-demand. | ||
3705 | |||
3706 | @item IMMEDIATE_START ARM will always start this service when the peer | ||
3707 | is started. | ||
3708 | |||
3709 | @item ACCEPT_FROM IPv4 addresses the service accepts connections from. | ||
3710 | |||
3711 | @item ACCEPT_FROM6 IPv6 addresses the service accepts connections from. | ||
3712 | |||
3713 | @end table | ||
3714 | |||
3715 | |||
3716 | Options that impact the operation of ARM overall are in the "[arm]" | ||
3717 | section. ARM is a normal service and has (except for START_ON_DEMAND) all of the | ||
3718 | options that other services do. In addition, ARM has the | ||
3719 | following options: | ||
3720 | |||
3721 | @table @asis | ||
3722 | |||
3723 | @item GLOBAL_PREFIX Command to be pre-pended to all services that are | ||
3724 | going to run. | ||
3725 | |||
3726 | @item GLOBAL_POSTFIX Global option that will be supplied to all the | ||
3727 | services that are going to run. | ||
3728 | |||
3729 | @end table | ||
3730 | |||
3731 | @c *********************************************************************** | ||
3732 | @node ARM - Availability | ||
3733 | @subsection ARM - Availability | ||
3734 | |||
3735 | |||
3736 | As mentioned before, one of the features provided by ARM is starting | ||
3737 | services on demand. Consider the example of one service "client" that | ||
3738 | wants to connect to another service a "server". The "client" will ask ARM | ||
3739 | to run the "server". ARM starts the "server". The "server" starts | ||
3740 | listening to incoming connections. The "client" will establish a | ||
3741 | connection with the "server". And then, they will start to communicate | ||
3742 | together.@ One problem with that scheme is that it's slow!@ | ||
3743 | The "client" service wants to communicate with the "server" service at | ||
3744 | once and is not willing wait for it to be started and listening to | ||
3745 | incoming connections before serving its request.@ One solution for that | ||
3746 | problem will be that ARM starts all services as default services. That | ||
3747 | solution will solve the problem, yet, it's not quite practical, for some | ||
3748 | services that are going to be started can never be used or are going to | ||
3749 | be used after a relatively long time.@ | ||
3750 | The approach followed by ARM to solve this problem is as follows: | ||
3751 | |||
3752 | @itemize @bullet | ||
3753 | |||
3754 | @item For each service having a PORT field in the configuration file and | ||
3755 | that is not one of the default services ( a service that accepts incoming | ||
3756 | connections from clients), ARM creates listening sockets for all addresses | ||
3757 | associated with that service. | ||
3758 | |||
3759 | @item The "client" will immediately establish a connection with | ||
3760 | the "server". | ||
3761 | |||
3762 | @item ARM --- pretending to be the "server" --- will listen on the | ||
3763 | respective port and notice the incoming connection from the "client" | ||
3764 | (but not accept it), instead | ||
3765 | |||
3766 | @item Once there is an incoming connection, ARM will start the "server", | ||
3767 | passing on the listen sockets (now, the service is started and can do its | ||
3768 | work). | ||
3769 | |||
3770 | @item Other client services now can directly connect directly to the | ||
3771 | "server". | ||
3772 | |||
3773 | @end itemize | ||
3774 | |||
3775 | @c *********************************************************************** | ||
3776 | @node Reliability | ||
3777 | @subsection Reliability | ||
3778 | |||
3779 | One of the features provided by ARM, is the automatic restart of crashed | ||
3780 | services.@ ARM needs to know which of the running services died. Function | ||
3781 | "gnunet-service-arm.c/maint_child_death()" is responsible for that. The | ||
3782 | function is scheduled to run upon receiving a SIGCHLD signal. The | ||
3783 | function, then, iterates ARM's list of services running and monitors | ||
3784 | which service has died (crashed). For all crashing services, ARM restarts | ||
3785 | them.@ | ||
3786 | Now, considering the case of a service having a serious problem causing it | ||
3787 | to crash each time it's started by ARM. If ARM keeps blindly restarting | ||
3788 | such a service, we are going to have the pattern: | ||
3789 | start-crash-restart-crash-restart-crash and so forth!! Which is of course | ||
3790 | not practical.@ | ||
3791 | For that reason, ARM schedules the service to be restarted after waiting | ||
3792 | for some delay that grows exponentially with each crash/restart of that | ||
3793 | service.@ To clarify the idea, considering the following example: | ||
3794 | |||
3795 | @itemize @bullet | ||
3796 | |||
3797 | @item Service S crashed. | ||
3798 | |||
3799 | @item ARM receives the SIGCHLD and inspects its list of services to find | ||
3800 | the dead one(s). | ||
3801 | |||
3802 | @item ARM finds S dead and schedules it for restarting after "backoff" | ||
3803 | time which is initially set to 1ms. ARM will double the backoff time | ||
3804 | correspondent to S (now backoff(S) = 2ms) | ||
3805 | |||
3806 | @item Because there is a severe problem with S, it crashed again. | ||
3807 | |||
3808 | @item Again ARM receives the SIGCHLD and detects that it's S again that's | ||
3809 | crashed. ARM schedules it for restarting but after its new backoff time | ||
3810 | (which became 2ms), and doubles its backoff time (now backoff(S) = 4). | ||
3811 | |||
3812 | @item and so on, until backoff(S) reaches a certain threshold | ||
3813 | (@code{EXPONENTIAL_BACKOFF_THRESHOLD} is set to half an hour), | ||
3814 | after reaching it, backoff(S) will remain half an hour, | ||
3815 | hence ARM won't be busy for a lot of time trying to restart a | ||
3816 | problematic service. | ||
3817 | @end itemize | ||
3818 | |||
3819 | @cindex TRANSPORT Subsystem | ||
3820 | @node TRANSPORT Subsystem | ||
3821 | @section TRANSPORT Subsystem | ||
3822 | |||
3823 | |||
3824 | This chapter documents how the GNUnet transport subsystem works. The | ||
3825 | GNUnet transport subsystem consists of three main components: the | ||
3826 | transport API (the interface used by the rest of the system to access the | ||
3827 | transport service), the transport service itself (most of the interesting | ||
3828 | functions, such as choosing transports, happens here) and the transport | ||
3829 | plugins. A transport plugin is a concrete implementation for how two | ||
3830 | GNUnet peers communicate; many plugins exist, for example for | ||
3831 | communication via TCP, UDP, HTTP, HTTPS and others. Finally, the | ||
3832 | transport subsystem uses supporting code, especially the NAT/UPnP | ||
3833 | library to help with tasks such as NAT traversal. | ||
3834 | |||
3835 | Key tasks of the transport service include: | ||
3836 | |||
3837 | @itemize @bullet | ||
3838 | |||
3839 | @item Create our HELLO message, notify clients and neighbours if our HELLO | ||
3840 | changes (using NAT library as necessary) | ||
3841 | |||
3842 | @item Validate HELLOs from other peers (send PING), allow other peers to | ||
3843 | validate our HELLO's addresses (send PONG) | ||
3844 | |||
3845 | @item Upon request, establish connections to other peers (using address | ||
3846 | selection from ATS subsystem) and maintain them (again using PINGs and | ||
3847 | PONGs) as long as desired | ||
3848 | |||
3849 | @item Accept incoming connections, give ATS service the opportunity to | ||
3850 | switch communication channels | ||
3851 | |||
3852 | @item Notify clients about peers that have connected to us or that have | ||
3853 | been disconnected from us | ||
3854 | |||
3855 | @item If a (stateful) connection goes down unexpectedly (without explicit | ||
3856 | DISCONNECT), quickly attempt to recover (without notifying clients) but do | ||
3857 | notify clients quickly if reconnecting fails | ||
3858 | |||
3859 | @item Send (payload) messages arriving from clients to other peers via | ||
3860 | transport plugins and receive messages from other peers, forwarding | ||
3861 | those to clients | ||
3862 | |||
3863 | @item Enforce inbound traffic limits (using flow-control if it is | ||
3864 | applicable); outbound traffic limits are enforced by CORE, not by us (!) | ||
3865 | |||
3866 | @item Enforce restrictions on P2P connection as specified by the blacklist | ||
3867 | configuration and blacklisting clients | ||
3868 | @end itemize | ||
3869 | |||
3870 | Note that the term "clients" in the list above really refers to the | ||
3871 | GNUnet-CORE service, as CORE is typically the only client of the | ||
3872 | transport service. | ||
3873 | |||
3874 | @menu | ||
3875 | * Address validation protocol:: | ||
3876 | @end menu | ||
3877 | |||
3878 | @node Address validation protocol | ||
3879 | @subsection Address validation protocol | ||
3880 | |||
3881 | |||
3882 | This section documents how the GNUnet transport service validates | ||
3883 | connections with other peers. It is a high-level description of the | ||
3884 | protocol necessary to understand the details of the implementation. It | ||
3885 | should be noted that when we talk about PING and PONG messages in this | ||
3886 | section, we refer to transport-level PING and PONG messages, which are | ||
3887 | different from core-level PING and PONG messages (both in implementation | ||
3888 | and function). | ||
3889 | |||
3890 | The goal of transport-level address validation is to minimize the chances | ||
3891 | of a successful man-in-the-middle attack against GNUnet peers on the | ||
3892 | transport level. Such an attack would not allow the adversary to decrypt | ||
3893 | the P2P transmissions, but a successful attacker could at least measure | ||
3894 | traffic volumes and latencies (raising the adversaries capabilities by | ||
3895 | those of a global passive adversary in the worst case). The scenarios we | ||
3896 | are concerned about is an attacker, Mallory, giving a @code{HELLO} to | ||
3897 | Alice that claims to be for Bob, but contains Mallory's IP address | ||
3898 | instead of Bobs (for some transport). | ||
3899 | Mallory would then forward the traffic to Bob (by initiating a | ||
3900 | connection to Bob and claiming to be Alice). As a further | ||
3901 | complication, the scheme has to work even if say Alice is behind a NAT | ||
3902 | without traversal support and hence has no address of her own (and thus | ||
3903 | Alice must always initiate the connection to Bob). | ||
3904 | |||
3905 | An additional constraint is that @code{HELLO} messages do not contain a | ||
3906 | cryptographic signature since other peers must be able to edit | ||
3907 | (i.e. remove) addresses from the @code{HELLO} at any time (this was | ||
3908 | not true in GNUnet 0.8.x). A basic @strong{assumption} is that each peer | ||
3909 | knows the set of possible network addresses that it @strong{might} | ||
3910 | be reachable under (so for example, the external IP address of the | ||
3911 | NAT plus the LAN address(es) with the respective ports). | ||
3912 | |||
3913 | The solution is the following. If Alice wants to validate that a given | ||
3914 | address for Bob is valid (i.e. is actually established @strong{directly} | ||
3915 | with the intended target), she sends a PING message over that connection | ||
3916 | to Bob. Note that in this case, Alice initiated the connection so only | ||
3917 | Alice knows which address was used for sure (Alice may be behind NAT, so | ||
3918 | whatever address Bob sees may not be an address Alice knows she has). | ||
3919 | Bob checks that the address given in the @code{PING} is actually one | ||
3920 | of Bob's addresses (ie: does not belong to Mallory), and if it is, | ||
3921 | sends back a @code{PONG} (with a signature that says that Bob | ||
3922 | owns/uses the address from the @code{PING}). | ||
3923 | Alice checks the signature and is happy if it is valid and the address | ||
3924 | in the @code{PONG} is the address Alice used. | ||
3925 | This is similar to the 0.8.x protocol where the @code{HELLO} contained a | ||
3926 | signature from Bob for each address used by Bob. | ||
3927 | Here, the purpose code for the signature is | ||
3928 | @code{GNUNET_SIGNATURE_PURPOSE_TRANSPORT_PONG_OWN}. After this, Alice will | ||
3929 | remember Bob's address and consider the address valid for a while (12h in | ||
3930 | the current implementation). Note that after this exchange, Alice only | ||
3931 | considers Bob's address to be valid, the connection itself is not | ||
3932 | considered 'established'. In particular, Alice may have many addresses | ||
3933 | for Bob that Alice considers valid. | ||
3934 | |||
3935 | The @code{PONG} message is protected with a nonce/challenge against replay | ||
3936 | attacks (@uref{http://en.wikipedia.org/wiki/Replay_attack, replay}) | ||
3937 | and uses an expiration time for the signature (but those are almost | ||
3938 | implementation details). | ||
3939 | |||
3940 | @cindex NAT library | ||
3941 | @node NAT library | ||
3942 | @section NAT library | ||
3943 | |||
3944 | |||
3945 | The goal of the GNUnet NAT library is to provide a general-purpose API for | ||
3946 | NAT traversal @strong{without} third-party support. So protocols that | ||
3947 | involve contacting a third peer to help establish a connection between | ||
3948 | two peers are outside of the scope of this API. That does not mean that | ||
3949 | GNUnet doesn't support involving a third peer (we can do this with the | ||
3950 | distance-vector transport or using application-level protocols), it just | ||
3951 | means that the NAT API is not concerned with this possibility. The API is | ||
3952 | written so that it will work for IPv6-NAT in the future as well as | ||
3953 | current IPv4-NAT. Furthermore, the NAT API is always used, even for peers | ||
3954 | that are not behind NAT --- in that case, the mapping provided is simply | ||
3955 | the identity. | ||
3956 | |||
3957 | NAT traversal is initiated by calling @code{GNUNET_NAT_register}. Given a | ||
3958 | set of addresses that the peer has locally bound to (TCP or UDP), the NAT | ||
3959 | library will return (via callback) a (possibly longer) list of addresses | ||
3960 | the peer @strong{might} be reachable under. Internally, depending on the | ||
3961 | configuration, the NAT library will try to punch a hole (using UPnP) or | ||
3962 | just "know" that the NAT was manually punched and generate the respective | ||
3963 | external IP address (the one that should be globally visible) based on | ||
3964 | the given information. | ||
3965 | |||
3966 | The NAT library also supports ICMP-based NAT traversal. Here, the other | ||
3967 | peer can request connection-reversal by this peer (in this special case, | ||
3968 | the peer is even allowed to configure a port number of zero). If the NAT | ||
3969 | library detects a connection-reversal request, it returns the respective | ||
3970 | target address to the client as well. It should be noted that | ||
3971 | connection-reversal is currently only intended for TCP, so other plugins | ||
3972 | @strong{must} pass @code{NULL} for the reversal callback. Naturally, the | ||
3973 | NAT library also supports requesting connection reversal from a remote | ||
3974 | peer (@code{GNUNET_NAT_run_client}). | ||
3975 | |||
3976 | Once initialized, the NAT handle can be used to test if a given address is | ||
3977 | possibly a valid address for this peer (@code{GNUNET_NAT_test_address}). | ||
3978 | This is used for validating our addresses when generating PONGs. | ||
3979 | |||
3980 | Finally, the NAT library contains an API to test if our NAT configuration | ||
3981 | is correct. Using @code{GNUNET_NAT_test_start} @strong{before} binding to | ||
3982 | the respective port, the NAT library can be used to test if the | ||
3983 | configuration works. The test function act as a local client, initialize | ||
3984 | the NAT traversal and then contact a @code{gnunet-nat-server} (running by | ||
3985 | default on @code{gnunet.org}) and ask for a connection to be established. | ||
3986 | This way, it is easy to test if the current NAT configuration is valid. | ||
3987 | |||
3988 | @node Distance-Vector plugin | ||
3989 | @section Distance-Vector plugin | ||
3990 | |||
3991 | |||
3992 | The Distance Vector (DV) transport is a transport mechanism that allows | ||
3993 | peers to act as relays for each other, thereby connecting peers that would | ||
3994 | otherwise be unable to connect. This gives a larger connection set to | ||
3995 | applications that may work better with more peers to choose from (for | ||
3996 | example, File Sharing and/or DHT). | ||
3997 | |||
3998 | The Distance Vector transport essentially has two functions. The first is | ||
3999 | "gossiping" connection information about more distant peers to directly | ||
4000 | connected peers. The second is taking messages intended for non-directly | ||
4001 | connected peers and encapsulating them in a DV wrapper that contains the | ||
4002 | required information for routing the message through forwarding peers. Via | ||
4003 | gossiping, optimal routes through the known DV neighborhood are discovered | ||
4004 | and utilized and the message encapsulation provides some benefits in | ||
4005 | addition to simply getting the message from the correct source to the | ||
4006 | proper destination. | ||
4007 | |||
4008 | The gossiping function of DV provides an up to date routing table of | ||
4009 | peers that are available up to some number of hops. We call this a | ||
4010 | fisheye view of the network (like a fish, nearby objects are known while | ||
4011 | more distant ones unknown). Gossip messages are sent only to directly | ||
4012 | connected peers, but they are sent about other knowns peers within the | ||
4013 | "fisheye distance". Whenever two peers connect, they immediately gossip | ||
4014 | to each other about their appropriate other neighbors. They also gossip | ||
4015 | about the newly connected peer to previously | ||
4016 | connected neighbors. In order to keep the routing tables up to date, | ||
4017 | disconnect notifications are propagated as gossip as well (because | ||
4018 | disconnects may not be sent/received, timeouts are also used remove | ||
4019 | stagnant routing table entries). | ||
4020 | |||
4021 | Routing of messages via DV is straightforward. When the DV transport is | ||
4022 | notified of a message destined for a non-direct neighbor, the appropriate | ||
4023 | forwarding peer is selected, and the base message is encapsulated in a DV | ||
4024 | message which contains information about the initial peer and the intended | ||
4025 | recipient. At each forwarding hop, the initial peer is validated (the | ||
4026 | forwarding peer ensures that it has the initial peer in its neighborhood, | ||
4027 | otherwise the message is dropped). Next the base message is | ||
4028 | re-encapsulated in a new DV message for the next hop in the forwarding | ||
4029 | chain (or delivered to the current peer, if it has arrived at the | ||
4030 | destination). | ||
4031 | |||
4032 | Assume a three peer network with peers Alice, Bob and Carol. Assume that | ||
4033 | |||
4034 | @example | ||
4035 | Alice <-> Bob and Bob <-> Carol | ||
4036 | @end example | ||
4037 | |||
4038 | @noindent | ||
4039 | are direct (e.g. over TCP or UDP transports) connections, but that | ||
4040 | Alice cannot directly connect to Carol. | ||
4041 | This may be the case due to NAT or firewall restrictions, or perhaps | ||
4042 | based on one of the peers respective configurations. If the Distance | ||
4043 | Vector transport is enabled on all three peers, it will automatically | ||
4044 | discover (from the gossip protocol) that Alice and Carol can connect via | ||
4045 | Bob and provide a "virtual" Alice <-> Carol connection. Routing between | ||
4046 | Alice and Carol happens as follows; Alice creates a message destined for | ||
4047 | Carol and notifies the DV transport about it. The DV transport at Alice | ||
4048 | looks up Carol in the routing table and finds that the message must be | ||
4049 | sent through Bob for Carol. The message is encapsulated setting Alice as | ||
4050 | the initiator and Carol as the destination and sent to Bob. Bob receives | ||
4051 | the messages, verifies that both Alice and Carol are known to Bob, and | ||
4052 | re-wraps the message in a new DV message for Carol. | ||
4053 | The DV transport at Carol receives this message, unwraps the original | ||
4054 | message, and delivers it to Carol as though it came directly from Alice. | ||
4055 | |||
4056 | @cindex SMTP plugin | ||
4057 | @node SMTP plugin | ||
4058 | @section SMTP plugin | ||
4059 | |||
4060 | @c TODO: Update! | ||
4061 | |||
4062 | This section describes the new SMTP transport plugin for GNUnet as it | ||
4063 | exists in the 0.7.x and 0.8.x branch. SMTP support is currently not | ||
4064 | available in GNUnet 0.9.x. This page also describes the transport layer | ||
4065 | abstraction (as it existed in 0.7.x and 0.8.x) in more detail and gives | ||
4066 | some benchmarking results. The performance results presented are quite | ||
4067 | old and maybe outdated at this point. | ||
4068 | For the readers in the year 2019, you will notice by the mention of | ||
4069 | version 0.7, 0.8, and 0.9 that this section has to be taken with your | ||
4070 | usual grain of salt and be updated eventually. | ||
4071 | |||
4072 | @itemize @bullet | ||
4073 | @item Why use SMTP for a peer-to-peer transport? | ||
4074 | @item SMTPHow does it work? | ||
4075 | @item How do I configure my peer? | ||
4076 | @item How do I test if it works? | ||
4077 | @item How fast is it? | ||
4078 | @item Is there any additional documentation? | ||
4079 | @end itemize | ||
4080 | |||
4081 | |||
4082 | @menu | ||
4083 | * Why use SMTP for a peer-to-peer transport?:: | ||
4084 | * How does it work?:: | ||
4085 | * How do I configure my peer?:: | ||
4086 | * How do I test if it works?:: | ||
4087 | * How fast is it?:: | ||
4088 | @end menu | ||
4089 | |||
4090 | @node Why use SMTP for a peer-to-peer transport? | ||
4091 | @subsection Why use SMTP for a peer-to-peer transport? | ||
4092 | |||
4093 | |||
4094 | There are many reasons why one would not want to use SMTP: | ||
4095 | |||
4096 | @itemize @bullet | ||
4097 | @item SMTP is using more bandwidth than TCP, UDP or HTTP | ||
4098 | @item SMTP has a much higher latency. | ||
4099 | @item SMTP requires significantly more computation (encoding and decoding | ||
4100 | time) for the peers. | ||
4101 | @item SMTP is significantly more complicated to configure. | ||
4102 | @item SMTP may be abused by tricking GNUnet into sending mail to@ | ||
4103 | non-participating third parties. | ||
4104 | @end itemize | ||
4105 | |||
4106 | So why would anybody want to use SMTP? | ||
4107 | @itemize @bullet | ||
4108 | @item SMTP can be used to contact peers behind NAT boxes (in virtual | ||
4109 | private networks). | ||
4110 | @item SMTP can be used to circumvent policies that limit or prohibit | ||
4111 | peer-to-peer traffic by masking as "legitimate" traffic. | ||
4112 | @item SMTP uses E-mail addresses which are independent of a specific IP, | ||
4113 | which can be useful to address peers that use dynamic IP addresses. | ||
4114 | @item SMTP can be used to initiate a connection (e.g. initial address | ||
4115 | exchange) and peers can then negotiate the use of a more efficient | ||
4116 | protocol (e.g. TCP) for the actual communication. | ||
4117 | @end itemize | ||
4118 | |||
4119 | In summary, SMTP can for example be used to send a message to a peer | ||
4120 | behind a NAT box that has a dynamic IP to tell the peer to establish a | ||
4121 | TCP connection to a peer outside of the private network. Even an | ||
4122 | extraordinary overhead for this first message would be irrelevant in this | ||
4123 | type of situation. | ||
4124 | |||
4125 | @node How does it work? | ||
4126 | @subsection How does it work? | ||
4127 | |||
4128 | |||
4129 | When a GNUnet peer needs to send a message to another GNUnet peer that has | ||
4130 | advertised (only) an SMTP transport address, GNUnet base64-encodes the | ||
4131 | message and sends it in an E-mail to the advertised address. The | ||
4132 | advertisement contains a filter which is placed in the E-mail header, | ||
4133 | such that the receiving host can filter the tagged E-mails and forward it | ||
4134 | to the GNUnet peer process. The filter can be specified individually by | ||
4135 | each peer and be changed over time. This makes it impossible to censor | ||
4136 | GNUnet E-mail messages by searching for a generic filter. | ||
4137 | |||
4138 | @node How do I configure my peer? | ||
4139 | @subsection How do I configure my peer? | ||
4140 | |||
4141 | |||
4142 | First, you need to configure @code{procmail} to filter your inbound E-mail | ||
4143 | for GNUnet traffic. The GNUnet messages must be delivered into a pipe, for | ||
4144 | example @code{/tmp/gnunet.smtp}. You also need to define a filter that is | ||
4145 | used by @command{procmail} to detect GNUnet messages. You are free to | ||
4146 | choose whichever filter you like, but you should make sure that it does | ||
4147 | not occur in your other E-mail. In our example, we will use | ||
4148 | @code{X-mailer: GNUnet}. The @code{~/.procmailrc} configuration file then | ||
4149 | looks like this: | ||
4150 | |||
4151 | @example | ||
4152 | :0: | ||
4153 | * ^X-mailer: GNUnet | ||
4154 | /tmp/gnunet.smtp | ||
4155 | # where do you want your other e-mail delivered to | ||
4156 | # (default: /var/spool/mail/) | ||
4157 | :0: /var/spool/mail/ | ||
4158 | @end example | ||
4159 | |||
4160 | After adding this file, first make sure that your regular E-mail still | ||
4161 | works (e.g. by sending an E-mail to yourself). Then edit the GNUnet | ||
4162 | configuration. In the section @code{SMTP} you need to specify your E-mail | ||
4163 | address under @code{EMAIL}, your mail server (for outgoing mail) under | ||
4164 | @code{SERVER}, the filter (X-mailer: GNUnet in the example) under | ||
4165 | @code{FILTER} and the name of the pipe under @code{PIPE}.@ The completed | ||
4166 | section could then look like this: | ||
4167 | |||
4168 | @example | ||
4169 | EMAIL = me@@mail.gnu.org MTU = 65000 SERVER = mail.gnu.org:25 FILTER = | ||
4170 | "X-mailer: GNUnet" PIPE = /tmp/gnunet.smtp | ||
4171 | @end example | ||
4172 | |||
4173 | Finally, you need to add @code{smtp} to the list of @code{TRANSPORTS} in | ||
4174 | the @code{GNUNETD} section. GNUnet peers will use the E-mail address that | ||
4175 | you specified to contact your peer until the advertisement times out. | ||
4176 | Thus, if you are not sure if everything works properly or if you are not | ||
4177 | planning to be online for a long time, you may want to configure this | ||
4178 | timeout to be short, e.g. just one hour. For this, set | ||
4179 | @code{HELLOEXPIRES} to @code{1} in the @code{GNUNETD} section. | ||
4180 | |||
4181 | This should be it, but you may probably want to test it first. | ||
4182 | |||
4183 | @node How do I test if it works? | ||
4184 | @subsection How do I test if it works? | ||
4185 | |||
4186 | |||
4187 | Any transport can be subjected to some rudimentary tests using the | ||
4188 | @code{gnunet-transport-check} tool. The tool sends a message to the local | ||
4189 | node via the transport and checks that a valid message is received. While | ||
4190 | this test does not involve other peers and can not check if firewalls or | ||
4191 | other network obstacles prohibit proper operation, this is a great | ||
4192 | testcase for the SMTP transport since it tests pretty much nearly all of | ||
4193 | the functionality. | ||
4194 | |||
4195 | @code{gnunet-transport-check} should only be used without running | ||
4196 | @code{gnunetd} at the same time. By default, @code{gnunet-transport-check} | ||
4197 | tests all transports that are specified in the configuration file. But | ||
4198 | you can specifically test SMTP by giving the option | ||
4199 | @code{--transport=smtp}. | ||
4200 | |||
4201 | Note that this test always checks if a transport can receive and send. | ||
4202 | While you can configure most transports to only receive or only send | ||
4203 | messages, this test will only work if you have configured the transport | ||
4204 | to send and receive messages. | ||
4205 | |||
4206 | @node How fast is it? | ||
4207 | @subsection How fast is it? | ||
4208 | |||
4209 | |||
4210 | We have measured the performance of the UDP, TCP and SMTP transport layer | ||
4211 | directly and when used from an application using the GNUnet core. | ||
4212 | Measuring just the transport layer gives the better view of the actual | ||
4213 | overhead of the protocol, whereas evaluating the transport from the | ||
4214 | application puts the overhead into perspective from a practical point of | ||
4215 | view. | ||
4216 | |||
4217 | The loopback measurements of the SMTP transport were performed on three | ||
4218 | different machines spanning a range of modern SMTP configurations. We | ||
4219 | used a PIII-800 running RedHat 7.3 with the Purdue Computer Science | ||
4220 | configuration which includes filters for spam. We also used a Xenon 2 GHZ | ||
4221 | with a vanilla RedHat 8.0 sendmail configuration. Furthermore, we used | ||
4222 | qmail on a PIII-1000 running Sorcerer GNU Linux (SGL). The numbers for | ||
4223 | UDP and TCP are provided using the SGL configuration. The qmail benchmark | ||
4224 | uses qmail's internal filtering whereas the sendmail benchmarks relies on | ||
4225 | procmail to filter and deliver the mail. We used the transport layer to | ||
4226 | send a message of b bytes (excluding transport protocol headers) directly | ||
4227 | to the local machine. This way, network latency and packet loss on the | ||
4228 | wire have no impact on the timings. n messages were sent sequentially over | ||
4229 | the transport layer, sending message i+1 after the i-th message was | ||
4230 | received. All messages were sent over the same connection and the time to | ||
4231 | establish the connection was not taken into account since this overhead is | ||
4232 | minuscule in practice --- as long as a connection is used for a | ||
4233 | significant number of messages. | ||
4234 | |||
4235 | @multitable @columnfractions .20 .15 .15 .15 .15 .15 | ||
4236 | @headitem Transport @tab UDP @tab TCP @tab SMTP (Purdue sendmail) | ||
4237 | @tab SMTP (RH 8.0) @tab SMTP (SGL qmail) | ||
4238 | @item 11 bytes @tab 31 ms @tab 55 ms @tab 781 s @tab 77 s @tab 24 s | ||
4239 | @item 407 bytes @tab 37 ms @tab 62 ms @tab 789 s @tab 78 s @tab 25 s | ||
4240 | @item 1,221 bytes @tab 46 ms @tab 73 ms @tab 804 s @tab 78 s @tab 25 s | ||
4241 | @end multitable | ||
4242 | |||
4243 | The benchmarks show that UDP and TCP are, as expected, both significantly | ||
4244 | faster compared with any of the SMTP services. Among the SMTP | ||
4245 | implementations, there can be significant differences depending on the | ||
4246 | SMTP configuration. Filtering with an external tool like procmail that | ||
4247 | needs to re-parse its configuration for each mail can be very expensive. | ||
4248 | Applying spam filters can also significantly impact the performance of | ||
4249 | the underlying SMTP implementation. The microbenchmark shows that SMTP | ||
4250 | can be a viable solution for initiating peer-to-peer sessions: a couple of | ||
4251 | seconds to connect to a peer are probably not even going to be noticed by | ||
4252 | users. The next benchmark measures the possible throughput for a | ||
4253 | transport. Throughput can be measured by sending multiple messages in | ||
4254 | parallel and measuring packet loss. Note that not only UDP but also the | ||
4255 | TCP transport can actually loose messages since the TCP implementation | ||
4256 | drops messages if the @code{write} to the socket would block. While the | ||
4257 | SMTP protocol never drops messages itself, it is often so | ||
4258 | slow that only a fraction of the messages can be sent and received in the | ||
4259 | given time-bounds. For this benchmark we report the message loss after | ||
4260 | allowing t time for sending m messages. If messages were not sent (or | ||
4261 | received) after an overall timeout of t, they were considered lost. The | ||
4262 | benchmark was performed using two Xeon 2 GHZ machines running RedHat 8.0 | ||
4263 | with sendmail. The machines were connected with a direct 100 MBit Ethernet | ||
4264 | connection.@ Figures udp1200, tcp1200 and smtp-MTUs show that the | ||
4265 | throughput for messages of size 1,200 octets is 2,343 kbps, 3,310 kbps | ||
4266 | and 6 kbps for UDP, TCP and SMTP respectively. The high per-message | ||
4267 | overhead of SMTP can be improved by increasing the MTU, for example, an | ||
4268 | MTU of 12,000 octets improves the throughput to 13 kbps as figure | ||
4269 | smtp-MTUs shows. Our research paper) has some more details on the | ||
4270 | benchmarking results. | ||
4271 | |||
4272 | @cindex Bluetooth plugin | ||
4273 | @node Bluetooth plugin | ||
4274 | @section Bluetooth plugin | ||
4275 | |||
4276 | |||
4277 | This page describes the new Bluetooth transport plugin for GNUnet. The | ||
4278 | plugin is still in the testing stage so don't expect it to work | ||
4279 | perfectly. If you have any questions or problems just post them here or | ||
4280 | ask on the IRC channel. | ||
4281 | |||
4282 | @itemize @bullet | ||
4283 | @item What do I need to use the Bluetooth plugin transport? | ||
4284 | @item BluetoothHow does it work? | ||
4285 | @item What possible errors should I be aware of? | ||
4286 | @item How do I configure my peer? | ||
4287 | @item How can I test it? | ||
4288 | @end itemize | ||
4289 | |||
4290 | @menu | ||
4291 | * What do I need to use the Bluetooth plugin transport?:: | ||
4292 | * How does it work2?:: | ||
4293 | * What possible errors should I be aware of?:: | ||
4294 | * How do I configure my peer2?:: | ||
4295 | * How can I test it?:: | ||
4296 | * The implementation of the Bluetooth transport plugin:: | ||
4297 | @end menu | ||
4298 | |||
4299 | @node What do I need to use the Bluetooth plugin transport? | ||
4300 | @subsection What do I need to use the Bluetooth plugin transport? | ||
4301 | |||
4302 | |||
4303 | If you are a GNU/Linux user and you want to use the Bluetooth | ||
4304 | transport plugin you should install the | ||
4305 | @command{BlueZ} development libraries (if they aren't already | ||
4306 | installed). | ||
4307 | For instructions about how to install the libraries you should | ||
4308 | check out the BlueZ site | ||
4309 | (@uref{http://www.bluez.org/, http://www.bluez.org}). If you don't know if | ||
4310 | you have the necessary libraries, don't worry, just run the GNUnet | ||
4311 | configure script and you will be able to see a notification at the end | ||
4312 | which will warn you if you don't have the necessary libraries. | ||
4313 | |||
4314 | @c If you are a Windows user you should have installed the | ||
4315 | @c @emph{MinGW}/@emph{MSys2} with the latest updates (especially the | ||
4316 | @c @emph{ws2bth} header). If this is your first build of GNUnet on Windows | ||
4317 | @c you should check out the SBuild repository. It will semi-automatically | ||
4318 | @c assembles a @emph{MinGW}/@emph{MSys2} installation with a lot of extra | ||
4319 | @c packages which are needed for the GNUnet build. So this will ease your | ||
4320 | @c work!@ Finally you just have to be sure that you have the correct drivers | ||
4321 | @c for your Bluetooth device installed and that your device is on and in a | ||
4322 | @c discoverable mode. The Windows Bluetooth Stack supports only the RFCOMM | ||
4323 | @c protocol so we cannot turn on your device programmatically! | ||
4324 | |||
4325 | @c FIXME: Change to unique title | ||
4326 | @node How does it work2? | ||
4327 | @subsection How does it work2? | ||
4328 | |||
4329 | |||
4330 | The Bluetooth transport plugin uses virtually the same code as the WLAN | ||
4331 | plugin and only the helper binary is different. The helper takes a single | ||
4332 | argument, which represents the interface name and is specified in the | ||
4333 | configuration file. Here are the basic steps that are followed by the | ||
4334 | helper binary used on GNU/Linux: | ||
4335 | |||
4336 | @itemize @bullet | ||
4337 | @item it verifies if the name corresponds to a Bluetooth interface name | ||
4338 | @item it verifies if the interface is up (if it is not, it tries to bring | ||
4339 | it up) | ||
4340 | @item it tries to enable the page and inquiry scan in order to make the | ||
4341 | device discoverable and to accept incoming connection requests | ||
4342 | @emph{The above operations require root access so you should start the | ||
4343 | transport plugin with root privileges.} | ||
4344 | @item it finds an available port number and registers a SDP service which | ||
4345 | will be used to find out on which port number is the server listening on | ||
4346 | and switch the socket in listening mode | ||
4347 | @item it sends a HELLO message with its address | ||
4348 | @item finally it forwards traffic from the reading sockets to the STDOUT | ||
4349 | and from the STDIN to the writing socket | ||
4350 | @end itemize | ||
4351 | |||
4352 | Once in a while the device will make an inquiry scan to discover the | ||
4353 | nearby devices and it will send them randomly HELLO messages for peer | ||
4354 | discovery. | ||
4355 | |||
4356 | @node What possible errors should I be aware of? | ||
4357 | @subsection What possible errors should I be aware of? | ||
4358 | |||
4359 | |||
4360 | @emph{This section is dedicated for GNU/Linux users} | ||
4361 | |||
4362 | Well there are many ways in which things could go wrong but I will try to | ||
4363 | present some tools that you could use to debug and some scenarios. | ||
4364 | |||
4365 | @itemize @bullet | ||
4366 | |||
4367 | @item @code{bluetoothd -n -d} : use this command to enable logging in the | ||
4368 | foreground and to print the logging messages | ||
4369 | |||
4370 | @item @code{hciconfig}: can be used to configure the Bluetooth devices. | ||
4371 | If you run it without any arguments it will print information about the | ||
4372 | state of the interfaces. So if you receive an error that the device | ||
4373 | couldn't be brought up you should try to bring it manually and to see if | ||
4374 | it works (use @code{hciconfig -a hciX up}). If you can't and the | ||
4375 | Bluetooth address has the form 00:00:00:00:00:00 it means that there is | ||
4376 | something wrong with the D-Bus daemon or with the Bluetooth daemon. Use | ||
4377 | @code{bluetoothd} tool to see the logs | ||
4378 | |||
4379 | @item @code{sdptool} can be used to control and interrogate SDP servers. | ||
4380 | If you encounter problems regarding the SDP server (like the SDP server is | ||
4381 | down) you should check out if the D-Bus daemon is running correctly and to | ||
4382 | see if the Bluetooth daemon started correctly(use @code{bluetoothd} tool). | ||
4383 | Also, sometimes the SDP service could work but somehow the device couldn't | ||
4384 | register its service. Use @code{sdptool browse [dev-address]} to see if | ||
4385 | the service is registered. There should be a service with the name of the | ||
4386 | interface and GNUnet as provider. | ||
4387 | |||
4388 | @item @code{hcitool} : another useful tool which can be used to configure | ||
4389 | the device and to send some particular commands to it. | ||
4390 | |||
4391 | @item @code{hcidump} : could be used for low level debugging | ||
4392 | @end itemize | ||
4393 | |||
4394 | @c FIXME: A more unique name | ||
4395 | @node How do I configure my peer2? | ||
4396 | @subsection How do I configure my peer2? | ||
4397 | |||
4398 | |||
4399 | On GNU/Linux, you just have to be sure that the interface name | ||
4400 | corresponds to the one that you want to use. | ||
4401 | Use the @code{hciconfig} tool to check that. | ||
4402 | By default it is set to hci0 but you can change it. | ||
4403 | |||
4404 | A basic configuration looks like this: | ||
4405 | |||
4406 | @example | ||
4407 | [transport-bluetooth] | ||
4408 | # Name of the interface (typically hciX) | ||
4409 | INTERFACE = hci0 | ||
4410 | # Real hardware, no testing | ||
4411 | TESTMODE = 0 TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
4412 | @end example | ||
4413 | |||
4414 | In order to use the Bluetooth transport plugin when the transport service | ||
4415 | is started, you must add the plugin name to the default transport service | ||
4416 | plugins list. For example: | ||
4417 | |||
4418 | @example | ||
4419 | [transport] ... PLUGINS = dns bluetooth ... | ||
4420 | @end example | ||
4421 | |||
4422 | If you want to use only the Bluetooth plugin set | ||
4423 | @emph{PLUGINS = bluetooth} | ||
4424 | |||
4425 | On Windows, you cannot specify which device to use. The only thing that | ||
4426 | you should do is to add @emph{bluetooth} on the plugins list of the | ||
4427 | transport service. | ||
4428 | |||
4429 | @node How can I test it? | ||
4430 | @subsection How can I test it? | ||
4431 | |||
4432 | |||
4433 | If you have two Bluetooth devices on the same machine and you are using | ||
4434 | GNU/Linux you must: | ||
4435 | |||
4436 | @itemize @bullet | ||
4437 | |||
4438 | @item create two different file configuration (one which will use the | ||
4439 | first interface (@emph{hci0}) and the other which will use the second | ||
4440 | interface (@emph{hci1})). Let's name them @emph{peer1.conf} and | ||
4441 | @emph{peer2.conf}. | ||
4442 | |||
4443 | @item run @emph{gnunet-peerinfo -c peerX.conf -s} in order to generate the | ||
4444 | peers private keys. The @strong{X} must be replace with 1 or 2. | ||
4445 | |||
4446 | @item run @emph{gnunet-arm -c peerX.conf -s -i=transport} in order to | ||
4447 | start the transport service. (Make sure that you have "bluetooth" on the | ||
4448 | transport plugins list if the Bluetooth transport service doesn't start.) | ||
4449 | |||
4450 | @item run @emph{gnunet-peerinfo -c peer1.conf -s} to get the first peer's | ||
4451 | ID. If you already know your peer ID (you saved it from the first | ||
4452 | command), this can be skipped. | ||
4453 | |||
4454 | @item run @emph{gnunet-transport -c peer2.conf -p=PEER1_ID -s} to start | ||
4455 | sending data for benchmarking to the other peer. | ||
4456 | |||
4457 | @end itemize | ||
4458 | |||
4459 | |||
4460 | This scenario will try to connect the second peer to the first one and | ||
4461 | then start sending data for benchmarking. | ||
4462 | |||
4463 | @c On Windows you cannot test the plugin functionality using two Bluetooth | ||
4464 | @c devices from the same machine because after you install the drivers there | ||
4465 | @c will occur some conflicts between the Bluetooth stacks. (At least that is | ||
4466 | @c what happened on my machine : I wasn't able to use the Bluesoleil stack and | ||
4467 | @c the WINDCOMM one in the same time). | ||
4468 | |||
4469 | If you have two different machines and your configuration files are good | ||
4470 | you can use the same scenario presented on the beginning of this section. | ||
4471 | |||
4472 | Another way to test the plugin functionality is to create your own | ||
4473 | application which will use the GNUnet framework with the Bluetooth | ||
4474 | transport service. | ||
4475 | |||
4476 | @node The implementation of the Bluetooth transport plugin | ||
4477 | @subsection The implementation of the Bluetooth transport plugin | ||
4478 | |||
4479 | |||
4480 | This page describes the implementation of the Bluetooth transport plugin. | ||
4481 | |||
4482 | First I want to remind you that the Bluetooth transport plugin uses | ||
4483 | virtually the same code as the WLAN plugin and only the helper binary is | ||
4484 | different. Also the scope of the helper binary from the Bluetooth | ||
4485 | transport plugin is the same as the one used for the WLAN transport | ||
4486 | plugin: it accesses the interface and then it forwards traffic in both | ||
4487 | directions between the Bluetooth interface and stdin/stdout of the | ||
4488 | process involved. | ||
4489 | |||
4490 | The Bluetooth plugin transport could be used both on GNU/Linux and Windows | ||
4491 | platforms. | ||
4492 | |||
4493 | @itemize @bullet | ||
4494 | @item Linux functionality | ||
4495 | @c @item Windows functionality | ||
4496 | @item Pending Features | ||
4497 | @end itemize | ||
4498 | |||
4499 | |||
4500 | |||
4501 | @menu | ||
4502 | * Linux functionality:: | ||
4503 | * THE INITIALIZATION:: | ||
4504 | * THE LOOP:: | ||
4505 | * Details about the broadcast implementation:: | ||
4506 | @c * Windows functionality:: | ||
4507 | * Pending features:: | ||
4508 | @end menu | ||
4509 | |||
4510 | @node Linux functionality | ||
4511 | @subsubsection Linux functionality | ||
4512 | |||
4513 | |||
4514 | In order to implement the plugin functionality on GNU/Linux I | ||
4515 | used the BlueZ stack. | ||
4516 | For the communication with the other devices I used the RFCOMM | ||
4517 | protocol. Also I used the HCI protocol to gain some control over the | ||
4518 | device. The helper binary takes a single argument (the name of the | ||
4519 | Bluetooth interface) and is separated in two stages: | ||
4520 | |||
4521 | @c %** 'THE INITIALIZATION' should be in bigger letters or stand out, not | ||
4522 | @c %** starting a new section? | ||
4523 | @node THE INITIALIZATION | ||
4524 | @subsubsection THE INITIALIZATION | ||
4525 | |||
4526 | @itemize @bullet | ||
4527 | @item first, it checks if we have root privileges | ||
4528 | (@emph{Remember that we need to have root privileges in order to be able | ||
4529 | to bring the interface up if it is down or to change its state.}). | ||
4530 | |||
4531 | @item second, it verifies if the interface with the given name exists. | ||
4532 | |||
4533 | @strong{If the interface with that name exists and it is a Bluetooth | ||
4534 | interface:} | ||
4535 | |||
4536 | @item it creates a RFCOMM socket which will be used for listening and call | ||
4537 | the @emph{open_device} method | ||
4538 | |||
4539 | On the @emph{open_device} method: | ||
4540 | @itemize @bullet | ||
4541 | @item creates a HCI socket used to send control events to the device | ||
4542 | @item searches for the device ID using the interface name | ||
4543 | @item saves the device MAC address | ||
4544 | @item checks if the interface is down and tries to bring it UP | ||
4545 | @item checks if the interface is in discoverable mode and tries to make it | ||
4546 | discoverable | ||
4547 | @item closes the HCI socket and binds the RFCOMM one | ||
4548 | @item switches the RFCOMM socket in listening mode | ||
4549 | @item registers the SDP service (the service will be used by the other | ||
4550 | devices to get the port on which this device is listening on) | ||
4551 | @end itemize | ||
4552 | |||
4553 | @item drops the root privileges | ||
4554 | |||
4555 | @strong{If the interface is not a Bluetooth interface the helper exits | ||
4556 | with a suitable error} | ||
4557 | @end itemize | ||
4558 | |||
4559 | @c %** Same as for @node entry above | ||
4560 | @node THE LOOP | ||
4561 | @subsubsection THE LOOP | ||
4562 | |||
4563 | The helper binary uses a list where it saves all the connected neighbour | ||
4564 | devices (@emph{neighbours.devices}) and two buffers (@emph{write_pout} and | ||
4565 | @emph{write_std}). The first message which is send is a control message | ||
4566 | with the device's MAC address in order to announce the peer presence to | ||
4567 | the neighbours. Here are a short description of what happens in the main | ||
4568 | loop: | ||
4569 | |||
4570 | @itemize @bullet | ||
4571 | @item Every time when it receives something from the STDIN it processes | ||
4572 | the data and saves the message in the first buffer (@emph{write_pout}). | ||
4573 | When it has something in the buffer, it gets the destination address from | ||
4574 | the buffer, searches the destination address in the list (if there is no | ||
4575 | connection with that device, it creates a new one and saves it to the | ||
4576 | list) and sends the message. | ||
4577 | @item Every time when it receives something on the listening socket it | ||
4578 | accepts the connection and saves the socket on a list with the reading | ||
4579 | sockets. @item Every time when it receives something from a reading | ||
4580 | socket it parses the message, verifies the CRC and saves it in the | ||
4581 | @emph{write_std} buffer in order to be sent later to the STDOUT. | ||
4582 | @end itemize | ||
4583 | |||
4584 | So in the main loop we use the select function to wait until one of the | ||
4585 | file descriptor saved in one of the two file descriptors sets used is | ||
4586 | ready to use. The first set (@emph{rfds}) represents the reading set and | ||
4587 | it could contain the list with the reading sockets, the STDIN file | ||
4588 | descriptor or the listening socket. The second set (@emph{wfds}) is the | ||
4589 | writing set and it could contain the sending socket or the STDOUT file | ||
4590 | descriptor. After the select function returns, we check which file | ||
4591 | descriptor is ready to use and we do what is supposed to do on that kind | ||
4592 | of event. @emph{For example:} if it is the listening socket then we | ||
4593 | accept a new connection and save the socket in the reading list; if it is | ||
4594 | the STDOUT file descriptor, then we write to STDOUT the message from the | ||
4595 | @emph{write_std} buffer. | ||
4596 | |||
4597 | To find out on which port a device is listening on we connect to the local | ||
4598 | SDP server and search the registered service for that device. | ||
4599 | |||
4600 | @emph{You should be aware of the fact that if the device fails to connect | ||
4601 | to another one when trying to send a message it will attempt one more | ||
4602 | time. If it fails again, then it skips the message.} | ||
4603 | @emph{Also you should know that the transport Bluetooth plugin has | ||
4604 | support for @strong{broadcast messages}.} | ||
4605 | |||
4606 | @node Details about the broadcast implementation | ||
4607 | @subsubsection Details about the broadcast implementation | ||
4608 | |||
4609 | |||
4610 | First I want to point out that the broadcast functionality for the CONTROL | ||
4611 | messages is not implemented in a conventional way. Since the inquiry scan | ||
4612 | time is too big and it will take some time to send a message to all the | ||
4613 | discoverable devices I decided to tackle the problem in a different way. | ||
4614 | Here is how I did it: | ||
4615 | |||
4616 | @itemize @bullet | ||
4617 | @item If it is the first time when I have to broadcast a message I make an | ||
4618 | inquiry scan and save all the devices' addresses to a vector. | ||
4619 | @item After the inquiry scan ends I take the first address from the list | ||
4620 | and I try to connect to it. If it fails, I try to connect to the next one. | ||
4621 | If it succeeds, I save the socket to a list and send the message to the | ||
4622 | device. | ||
4623 | @item When I have to broadcast another message, first I search on the list | ||
4624 | for a new device which I'm not connected to. If there is no new device on | ||
4625 | the list I go to the beginning of the list and send the message to the | ||
4626 | old devices. After 5 cycles I make a new inquiry scan to check out if | ||
4627 | there are new discoverable devices and save them to the list. If there | ||
4628 | are no new discoverable devices I reset the cycling counter and go again | ||
4629 | through the old list and send messages to the devices saved in it. | ||
4630 | @end itemize | ||
4631 | |||
4632 | @strong{Therefore}: | ||
4633 | |||
4634 | @itemize @bullet | ||
4635 | @item every time when I have a broadcast message I look up on the list | ||
4636 | for a new device and send the message to it | ||
4637 | @item if I reached the end of the list for 5 times and I'm connected to | ||
4638 | all the devices from the list I make a new inquiry scan. | ||
4639 | @emph{The number of the list's cycles after an inquiry scan could be | ||
4640 | increased by redefining the MAX_LOOPS variable} | ||
4641 | @item when there are no new devices I send messages to the old ones. | ||
4642 | @end itemize | ||
4643 | |||
4644 | Doing so, the broadcast control messages will reach the devices but with | ||
4645 | delay. | ||
4646 | |||
4647 | @emph{NOTICE:} When I have to send a message to a certain device first I | ||
4648 | check on the broadcast list to see if we are connected to that device. If | ||
4649 | not we try to connect to it and in case of success we save the address and | ||
4650 | the socket on the list. If we are already connected to that device we | ||
4651 | simply use the socket. | ||
4652 | |||
4653 | @c @node Windows functionality | ||
4654 | @c @subsubsection Windows functionality | ||
4655 | |||
4656 | |||
4657 | @c For Windows I decided to use the Microsoft Bluetooth stack which has the | ||
4658 | @c advantage of coming standard from Windows XP SP2. The main disadvantage is | ||
4659 | @c that it only supports the RFCOMM protocol so we will not be able to have | ||
4660 | @c a low level control over the Bluetooth device. Therefore it is the user | ||
4661 | @c responsibility to check if the device is up and in the discoverable mode. | ||
4662 | @c Also there are no tools which could be used for debugging in order to read | ||
4663 | @c the data coming from and going to a Bluetooth device, which obviously | ||
4664 | @c hindered my work. Another thing that slowed down the implementation of the | ||
4665 | @c plugin (besides that I wasn't too accommodated with the win32 API) was that | ||
4666 | @c there were some bugs on MinGW regarding the Bluetooth. Now they are solved | ||
4667 | @c but you should keep in mind that you should have the latest updates | ||
4668 | @c (especially the @emph{ws2bth} header). | ||
4669 | |||
4670 | @c Besides the fact that it uses the Windows Sockets, the Windows | ||
4671 | @c implementation follows the same principles as the GNU/Linux one: | ||
4672 | |||
4673 | @c @itemize @bullet | ||
4674 | @c @item It has a initialization part where it initializes the | ||
4675 | @c Windows Sockets, creates a RFCOMM socket which will be binded and switched | ||
4676 | @c to the listening mode and registers a SDP service. In the Microsoft | ||
4677 | @c Bluetooth API there are two ways to work with the SDP: | ||
4678 | @c @itemize @bullet | ||
4679 | @c @item an easy way which works with very simple service records | ||
4680 | @c @item a hard way which is useful when you need to update or to delete the | ||
4681 | @c record | ||
4682 | @c @end itemize | ||
4683 | @c @end itemize | ||
4684 | |||
4685 | @c Since I only needed the SDP service to find out on which port the device | ||
4686 | @c is listening on and that did not change, I decided to use the easy way. | ||
4687 | @c In order to register the service I used the @emph{WSASetService} function | ||
4688 | @c and I generated the @emph{Universally Unique Identifier} with the | ||
4689 | @c @emph{guidgen.exe} Windows's tool. | ||
4690 | |||
4691 | @c In the loop section the only difference from the GNU/Linux implementation | ||
4692 | @c is that I used the @code{GNUNET_NETWORK} library for | ||
4693 | @c functions like @emph{accept}, @emph{bind}, @emph{connect} or | ||
4694 | @c @emph{select}. I decided to use the | ||
4695 | @c @code{GNUNET_NETWORK} library because I also needed to interact | ||
4696 | @c with the STDIN and STDOUT handles and on Windows | ||
4697 | @c the select function is only defined for sockets, | ||
4698 | @c and it will not work for arbitrary file handles. | ||
4699 | |||
4700 | @c Another difference between GNU/Linux and Windows implementation is that in | ||
4701 | @c GNU/Linux, the Bluetooth address is represented in 48 bits | ||
4702 | @c while in Windows is represented in 64 bits. | ||
4703 | @c Therefore I had to do some changes on @emph{plugin_transport_wlan} header. | ||
4704 | |||
4705 | @c Also, currently on Windows the Bluetooth plugin doesn't have support for | ||
4706 | @c broadcast messages. When it receives a broadcast message it will skip it. | ||
4707 | |||
4708 | @node Pending features | ||
4709 | @subsubsection Pending features | ||
4710 | |||
4711 | |||
4712 | @itemize @bullet | ||
4713 | @c @item Implement the broadcast functionality on Windows @emph{(currently | ||
4714 | @c working on)} | ||
4715 | @item Implement a testcase for the helper :@ @emph{The testcase | ||
4716 | consists of a program which emulates the plugin and uses the helper. It | ||
4717 | will simulate connections, disconnections and data transfers.} | ||
4718 | @end itemize | ||
4719 | |||
4720 | If you have a new idea about a feature of the plugin or suggestions about | ||
4721 | how I could improve the implementation you are welcome to comment or to | ||
4722 | contact me. | ||
4723 | |||
4724 | @node WLAN plugin | ||
4725 | @section WLAN plugin | ||
4726 | |||
4727 | |||
4728 | This section documents how the wlan transport plugin works. Parts which | ||
4729 | are not implemented yet or could be better implemented are described at | ||
4730 | the end. | ||
4731 | |||
4732 | @cindex ATS Subsystem | ||
4733 | @node ATS Subsystem | ||
4734 | @section ATS Subsystem | ||
4735 | |||
4736 | |||
4737 | ATS stands for "automatic transport selection", and the function of ATS in | ||
4738 | GNUnet is to decide on which address (and thus transport plugin) should | ||
4739 | be used for two peers to communicate, and what bandwidth limits should be | ||
4740 | imposed on such an individual connection. To help ATS make an informed | ||
4741 | decision, higher-level services inform the ATS service about their | ||
4742 | requirements and the quality of the service rendered. The ATS service | ||
4743 | also interacts with the transport service to be appraised of working | ||
4744 | addresses and to communicate its resource allocation decisions. Finally, | ||
4745 | the ATS service's operation can be observed using a monitoring API. | ||
4746 | |||
4747 | The main logic of the ATS service only collects the available addresses, | ||
4748 | their performance characteristics and the applications requirements, but | ||
4749 | does not make the actual allocation decision. This last critical step is | ||
4750 | left to an ATS plugin, as we have implemented (currently three) different | ||
4751 | allocation strategies which differ significantly in their performance and | ||
4752 | maturity, and it is still unclear if any particular plugin is generally | ||
4753 | superior. | ||
4754 | |||
4755 | @cindex CORE Subsystem | ||
4756 | @node CORE Subsystem | ||
4757 | @section CORE Subsystem | ||
4758 | |||
4759 | |||
4760 | The CORE subsystem in GNUnet is responsible for securing link-layer | ||
4761 | communications between nodes in the GNUnet overlay network. CORE builds | ||
4762 | on the TRANSPORT subsystem which provides for the actual, insecure, | ||
4763 | unreliable link-layer communication (for example, via UDP or WLAN), and | ||
4764 | then adds fundamental security to the connections: | ||
4765 | |||
4766 | @itemize @bullet | ||
4767 | @item confidentiality with so-called perfect forward secrecy; we use | ||
4768 | ECDHE | ||
4769 | (@uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}) | ||
4770 | powered by Curve25519 | ||
4771 | (@uref{http://cr.yp.to/ecdh.html, Curve25519}) for the key | ||
4772 | exchange and then use symmetric encryption, encrypting with both AES-256 | ||
4773 | (@uref{http://en.wikipedia.org/wiki/Rijndael, AES-256}) and | ||
4774 | Twofish (@uref{http://en.wikipedia.org/wiki/Twofish, Twofish}) | ||
4775 | @item @uref{http://en.wikipedia.org/wiki/Authentication, authentication} | ||
4776 | is achieved by signing the ephemeral keys using Ed25519 | ||
4777 | (@uref{http://ed25519.cr.yp.to/, Ed25519}), a deterministic | ||
4778 | variant of ECDSA | ||
4779 | (@uref{http://en.wikipedia.org/wiki/ECDSA, ECDSA}) | ||
4780 | @item integrity protection (using SHA-512 | ||
4781 | (@uref{http://en.wikipedia.org/wiki/SHA-2, SHA-512}) to do | ||
4782 | encrypt-then-MAC | ||
4783 | (@uref{http://en.wikipedia.org/wiki/Authenticated_encryption, encrypt-then-MAC})) | ||
4784 | @item Replay | ||
4785 | (@uref{http://en.wikipedia.org/wiki/Replay_attack, replay}) | ||
4786 | protection (using nonces, timestamps, challenge-response, | ||
4787 | message counters and ephemeral keys) | ||
4788 | @item liveness (keep-alive messages, timeout) | ||
4789 | @end itemize | ||
4790 | |||
4791 | @menu | ||
4792 | * Limitations:: | ||
4793 | * When is a peer "connected"?:: | ||
4794 | * libgnunetcore:: | ||
4795 | * The CORE Client-Service Protocol:: | ||
4796 | * The CORE Peer-to-Peer Protocol:: | ||
4797 | @end menu | ||
4798 | |||
4799 | @cindex core subsystem limitations | ||
4800 | @node Limitations | ||
4801 | @subsection Limitations | ||
4802 | |||
4803 | |||
4804 | CORE does not perform | ||
4805 | @uref{http://en.wikipedia.org/wiki/Routing, routing}; using CORE it is | ||
4806 | only possible to communicate with peers that happen to already be | ||
4807 | "directly" connected with each other. CORE also does not have an | ||
4808 | API to allow applications to establish such "direct" connections --- for | ||
4809 | this, applications can ask TRANSPORT, but TRANSPORT might not be able to | ||
4810 | establish a "direct" connection. The TOPOLOGY subsystem is responsible for | ||
4811 | trying to keep a few "direct" connections open at all times. Applications | ||
4812 | that need to talk to particular peers should use the CADET subsystem, as | ||
4813 | it can establish arbitrary "indirect" connections. | ||
4814 | |||
4815 | Because CORE does not perform routing, CORE must only be used directly by | ||
4816 | applications that either perform their own routing logic (such as | ||
4817 | anonymous file-sharing) or that do not require routing, for example | ||
4818 | because they are based on flooding the network. CORE communication is | ||
4819 | unreliable and delivery is possibly out-of-order. Applications that | ||
4820 | require reliable communication should use the CADET service. Each | ||
4821 | application can only queue one message per target peer with the CORE | ||
4822 | service at any time; messages cannot be larger than approximately | ||
4823 | 63 kilobytes. If messages are small, CORE may group multiple messages | ||
4824 | (possibly from different applications) prior to encryption. If permitted | ||
4825 | by the application (using the @uref{http://baus.net/on-tcp_cork/, cork} | ||
4826 | option), CORE may delay transmissions to facilitate grouping of multiple | ||
4827 | small messages. If cork is not enabled, CORE will transmit the message as | ||
4828 | soon as TRANSPORT allows it (TRANSPORT is responsible for limiting | ||
4829 | bandwidth and congestion control). CORE does not allow flow control; | ||
4830 | applications are expected to process messages at line-speed. If flow | ||
4831 | control is needed, applications should use the CADET service. | ||
4832 | |||
4833 | @cindex when is a peer connected | ||
4834 | @node When is a peer "connected"? | ||
4835 | @subsection When is a peer "connected"? | ||
4836 | |||
4837 | |||
4838 | In addition to the security features mentioned above, CORE also provides | ||
4839 | one additional key feature to applications using it, and that is a | ||
4840 | limited form of protocol-compatibility checking. CORE distinguishes | ||
4841 | between TRANSPORT-level connections (which enable communication with other | ||
4842 | peers) and application-level connections. Applications using the CORE API | ||
4843 | will (typically) learn about application-level connections from CORE, and | ||
4844 | not about TRANSPORT-level connections. When a typical application uses | ||
4845 | CORE, it will specify a set of message types | ||
4846 | (from @code{gnunet_protocols.h}) that it understands. CORE will then | ||
4847 | notify the application about connections it has with other peers if and | ||
4848 | only if those applications registered an intersecting set of message | ||
4849 | types with their CORE service. Thus, it is quite possible that CORE only | ||
4850 | exposes a subset of the established direct connections to a particular | ||
4851 | application --- and different applications running above CORE might see | ||
4852 | different sets of connections at the same time. | ||
4853 | |||
4854 | A special case are applications that do not register a handler for any | ||
4855 | message type. | ||
4856 | CORE assumes that these applications merely want to monitor connections | ||
4857 | (or "all" messages via other callbacks) and will notify those applications | ||
4858 | about all connections. This is used, for example, by the | ||
4859 | @code{gnunet-core} command-line tool to display the active connections. | ||
4860 | Note that it is also possible that the TRANSPORT service has more active | ||
4861 | connections than the CORE service, as the CORE service first has to | ||
4862 | perform a key exchange with connecting peers before exchanging information | ||
4863 | about supported message types and notifying applications about the new | ||
4864 | connection. | ||
4865 | |||
4866 | @cindex libgnunetcore | ||
4867 | @node libgnunetcore | ||
4868 | @subsection libgnunetcore | ||
4869 | |||
4870 | |||
4871 | The CORE API (defined in @file{gnunet_core_service.h}) is the basic | ||
4872 | messaging API used by P2P applications built using GNUnet. It provides | ||
4873 | applications the ability to send and receive encrypted messages to the | ||
4874 | peer's "directly" connected neighbours. | ||
4875 | |||
4876 | As CORE connections are generally "direct" connections,@ applications must | ||
4877 | not assume that they can connect to arbitrary peers this way, as "direct" | ||
4878 | connections may not always be possible. Applications using CORE are | ||
4879 | notified about which peers are connected. Creating new "direct" | ||
4880 | connections must be done using the TRANSPORT API. | ||
4881 | |||
4882 | The CORE API provides unreliable, out-of-order delivery. While the | ||
4883 | implementation tries to ensure timely, in-order delivery, both message | ||
4884 | losses and reordering are not detected and must be tolerated by the | ||
4885 | application. Most important, the core will NOT perform retransmission if | ||
4886 | messages could not be delivered. | ||
4887 | |||
4888 | Note that CORE allows applications to queue one message per connected | ||
4889 | peer. The rate at which each connection operates is influenced by the | ||
4890 | preferences expressed by local application as well as restrictions | ||
4891 | imposed by the other peer. Local applications can express their | ||
4892 | preferences for particular connections using the "performance" API of the | ||
4893 | ATS service. | ||
4894 | |||
4895 | Applications that require more sophisticated transmission capabilities | ||
4896 | such as TCP-like behavior, or if you intend to send messages to arbitrary | ||
4897 | remote peers, should use the CADET API. | ||
4898 | |||
4899 | The typical use of the CORE API is to connect to the CORE service using | ||
4900 | @code{GNUNET_CORE_connect}, process events from the CORE service (such as | ||
4901 | peers connecting, peers disconnecting and incoming messages) and send | ||
4902 | messages to connected peers using | ||
4903 | @code{GNUNET_CORE_notify_transmit_ready}. Note that applications must | ||
4904 | cancel pending transmission requests if they receive a disconnect event | ||
4905 | for a peer that had a transmission pending; furthermore, queuing more | ||
4906 | than one transmission request per peer per application using the | ||
4907 | service is not permitted. | ||
4908 | |||
4909 | The CORE API also allows applications to monitor all communications of the | ||
4910 | peer prior to encryption (for outgoing messages) or after decryption (for | ||
4911 | incoming messages). This can be useful for debugging, diagnostics or to | ||
4912 | establish the presence of cover traffic (for anonymity). As monitoring | ||
4913 | applications are often not interested in the payload, the monitoring | ||
4914 | callbacks can be configured to only provide the message headers (including | ||
4915 | the message type and size) instead of copying the full data stream to the | ||
4916 | monitoring client. | ||
4917 | |||
4918 | The init callback of the @code{GNUNET_CORE_connect} function is called | ||
4919 | with the hash of the public key of the peer. This public key is used to | ||
4920 | identify the peer globally in the GNUnet network. Applications are | ||
4921 | encouraged to check that the provided hash matches the hash that they are | ||
4922 | using (as theoretically the application may be using a different | ||
4923 | configuration file with a different private key, which would result in | ||
4924 | hard to find bugs). | ||
4925 | |||
4926 | As with most service APIs, the CORE API isolates applications from crashes | ||
4927 | of the CORE service. If the CORE service crashes, the application will see | ||
4928 | disconnect events for all existing connections. Once the connections are | ||
4929 | re-established, the applications will be receive matching connect events. | ||
4930 | |||
4931 | @cindex core clinet-service protocol | ||
4932 | @node The CORE Client-Service Protocol | ||
4933 | @subsection The CORE Client-Service Protocol | ||
4934 | |||
4935 | |||
4936 | This section describes the protocol between an application using the CORE | ||
4937 | service (the client) and the CORE service process itself. | ||
4938 | |||
4939 | |||
4940 | @menu | ||
4941 | * Setup2:: | ||
4942 | * Notifications:: | ||
4943 | * Sending:: | ||
4944 | @end menu | ||
4945 | |||
4946 | @node Setup2 | ||
4947 | @subsubsection Setup2 | ||
4948 | |||
4949 | |||
4950 | When a client connects to the CORE service, it first sends a | ||
4951 | @code{InitMessage} which specifies options for the connection and a set of | ||
4952 | message type values which are supported by the application. The options | ||
4953 | bitmask specifies which events the client would like to be notified about. | ||
4954 | The options include: | ||
4955 | |||
4956 | @table @asis | ||
4957 | @item GNUNET_CORE_OPTION_NOTHING No notifications | ||
4958 | @item GNUNET_CORE_OPTION_STATUS_CHANGE Peers connecting and disconnecting | ||
4959 | @item GNUNET_CORE_OPTION_FULL_INBOUND All inbound messages (after | ||
4960 | decryption) with full payload | ||
4961 | @item GNUNET_CORE_OPTION_HDR_INBOUND Just the @code{MessageHeader} | ||
4962 | of all inbound messages | ||
4963 | @item GNUNET_CORE_OPTION_FULL_OUTBOUND All outbound | ||
4964 | messages (prior to encryption) with full payload | ||
4965 | @item GNUNET_CORE_OPTION_HDR_OUTBOUND Just the @code{MessageHeader} of all | ||
4966 | outbound messages | ||
4967 | @end table | ||
4968 | |||
4969 | Typical applications will only monitor for connection status changes. | ||
4970 | |||
4971 | The CORE service responds to the @code{InitMessage} with an | ||
4972 | @code{InitReplyMessage} which contains the peer's identity. Afterwards, | ||
4973 | both CORE and the client can send messages. | ||
4974 | |||
4975 | @node Notifications | ||
4976 | @subsubsection Notifications | ||
4977 | |||
4978 | |||
4979 | The CORE will send @code{ConnectNotifyMessage}s and | ||
4980 | @code{DisconnectNotifyMessage}s whenever peers connect or disconnect from | ||
4981 | the CORE (assuming their type maps overlap with the message types | ||
4982 | registered by the client). When the CORE receives a message that matches | ||
4983 | the set of message types specified during the @code{InitMessage} (or if | ||
4984 | monitoring is enabled in for inbound messages in the options), it sends a | ||
4985 | @code{NotifyTrafficMessage} with the peer identity of the sender and the | ||
4986 | decrypted payload. The same message format (except with | ||
4987 | @code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} for the message type) is | ||
4988 | used to notify clients monitoring outbound messages; here, the peer | ||
4989 | identity given is that of the receiver. | ||
4990 | |||
4991 | @node Sending | ||
4992 | @subsubsection Sending | ||
4993 | |||
4994 | |||
4995 | When a client wants to transmit a message, it first requests a | ||
4996 | transmission slot by sending a @code{SendMessageRequest} which specifies | ||
4997 | the priority, deadline and size of the message. Note that these values | ||
4998 | may be ignored by CORE. When CORE is ready for the message, it answers | ||
4999 | with a @code{SendMessageReady} response. The client can then transmit the | ||
5000 | payload with a @code{SendMessage} message. Note that the actual message | ||
5001 | size in the @code{SendMessage} is allowed to be smaller than the size in | ||
5002 | the original request. A client may at any time send a fresh | ||
5003 | @code{SendMessageRequest}, which then superceeds the previous | ||
5004 | @code{SendMessageRequest}, which is then no longer valid. The client can | ||
5005 | tell which @code{SendMessageRequest} the CORE service's | ||
5006 | @code{SendMessageReady} message is for as all of these messages contain a | ||
5007 | "unique" request ID (based on a counter incremented by the client | ||
5008 | for each request). | ||
5009 | |||
5010 | @cindex CORE Peer-to-Peer Protocol | ||
5011 | @node The CORE Peer-to-Peer Protocol | ||
5012 | @subsection The CORE Peer-to-Peer Protocol | ||
5013 | |||
5014 | |||
5015 | |||
5016 | @menu | ||
5017 | * Creating the EphemeralKeyMessage:: | ||
5018 | * Establishing a connection:: | ||
5019 | * Encryption and Decryption:: | ||
5020 | * Type maps:: | ||
5021 | @end menu | ||
5022 | |||
5023 | @cindex EphemeralKeyMessage creation | ||
5024 | @node Creating the EphemeralKeyMessage | ||
5025 | @subsubsection Creating the EphemeralKeyMessage | ||
5026 | |||
5027 | |||
5028 | When the CORE service starts, each peer creates a fresh ephemeral (ECC) | ||
5029 | public-private key pair and signs the corresponding | ||
5030 | @code{EphemeralKeyMessage} with its long-term key (which we usually call | ||
5031 | the peer's identity; the hash of the public long term key is what results | ||
5032 | in a @code{struct GNUNET_PeerIdentity} in all GNUnet APIs. The ephemeral | ||
5033 | key is ONLY used for an ECDHE | ||
5034 | (@uref{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman, Elliptic-curve Diffie---Hellman}) | ||
5035 | exchange by the CORE service to establish symmetric session keys. A peer | ||
5036 | will use the same @code{EphemeralKeyMessage} for all peers for | ||
5037 | @code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it | ||
5038 | will create a fresh ephemeral key (forgetting the old one) and broadcast | ||
5039 | the new @code{EphemeralKeyMessage} to all connected peers, resulting in | ||
5040 | fresh symmetric session keys. Note that peers independently decide on | ||
5041 | when to discard ephemeral keys; it is not a protocol violation to discard | ||
5042 | keys more often. Ephemeral keys are also never stored to disk; restarting | ||
5043 | a peer will thus always create a fresh ephemeral key. The use of ephemeral | ||
5044 | keys is what provides @uref{http://en.wikipedia.org/wiki/Forward_secrecy, forward secrecy}. | ||
5045 | |||
5046 | Just before transmission, the @code{EphemeralKeyMessage} is patched to | ||
5047 | reflect the current sender_status, which specifies the current state of | ||
5048 | the connection from the point of view of the sender. The possible values | ||
5049 | are: | ||
5050 | |||
5051 | @itemize @bullet | ||
5052 | @item @code{KX_STATE_DOWN} Initial value, never used on the network | ||
5053 | @item @code{KX_STATE_KEY_SENT} We sent our ephemeral key, do not know the | ||
5054 | key of the other peer | ||
5055 | @item @code{KX_STATE_KEY_RECEIVED} This peer has received a valid | ||
5056 | ephemeral key of the other peer, but we are waiting for the other peer to | ||
5057 | confirm it's authenticity (ability to decode) via challenge-response. | ||
5058 | @item @code{KX_STATE_UP} The connection is fully up from the point of | ||
5059 | view of the sender (now performing keep-alive) | ||
5060 | @item @code{KX_STATE_REKEY_SENT} The sender has initiated a rekeying | ||
5061 | operation; the other peer has so far failed to confirm a working | ||
5062 | connection using the new ephemeral key | ||
5063 | @end itemize | ||
5064 | |||
5065 | @node Establishing a connection | ||
5066 | @subsubsection Establishing a connection | ||
5067 | |||
5068 | |||
5069 | Peers begin their interaction by sending a @code{EphemeralKeyMessage} to | ||
5070 | the other peer once the TRANSPORT service notifies the CORE service about | ||
5071 | the connection. | ||
5072 | A peer receiving an @code{EphemeralKeyMessage} with a status | ||
5073 | indicating that the sender does not have the receiver's ephemeral key, the | ||
5074 | receiver's @code{EphemeralKeyMessage} is sent in response. | ||
5075 | Additionally, if the receiver has not yet confirmed the authenticity of | ||
5076 | the sender, it also sends an (encrypted)@code{PingMessage} with a | ||
5077 | challenge (and the identity of the target) to the other peer. Peers | ||
5078 | receiving a @code{PingMessage} respond with an (encrypted) | ||
5079 | @code{PongMessage} which includes the challenge. Peers receiving a | ||
5080 | @code{PongMessage} check the challenge, and if it matches set the | ||
5081 | connection to @code{KX_STATE_UP}. | ||
5082 | |||
5083 | @node Encryption and Decryption | ||
5084 | @subsubsection Encryption and Decryption | ||
5085 | |||
5086 | |||
5087 | All functions related to the key exchange and encryption/decryption of | ||
5088 | messages can be found in @file{gnunet-service-core_kx.c} (except for the | ||
5089 | cryptographic primitives, which are in @file{util/crypto*.c}). | ||
5090 | Given the key material from ECDHE, a Key derivation function | ||
5091 | (@uref{https://en.wikipedia.org/wiki/Key_derivation_function, Key derivation function}) | ||
5092 | is used to derive two pairs of encryption and decryption keys for AES-256 | ||
5093 | and TwoFish, as well as initialization vectors and authentication keys | ||
5094 | (for HMAC | ||
5095 | (@uref{https://en.wikipedia.org/wiki/HMAC, HMAC})). | ||
5096 | The HMAC is computed over the encrypted payload. | ||
5097 | Encrypted messages include an iv_seed and the HMAC in the header. | ||
5098 | |||
5099 | Each encrypted message in the CORE service includes a sequence number and | ||
5100 | a timestamp in the encrypted payload. The CORE service remembers the | ||
5101 | largest observed sequence number and a bit-mask which represents which of | ||
5102 | the previous 32 sequence numbers were already used. | ||
5103 | Messages with sequence numbers lower than the largest observed sequence | ||
5104 | number minus 32 are discarded. Messages with a timestamp that is less | ||
5105 | than @code{REKEY_TOLERANCE} off (5 minutes) are also discarded. This of | ||
5106 | course means that system clocks need to be reasonably synchronized for | ||
5107 | peers to be able to communicate. Additionally, as the ephemeral key | ||
5108 | changes every 12 hours, a peer would not even be able to decrypt messages | ||
5109 | older than 12 hours. | ||
5110 | |||
5111 | @node Type maps | ||
5112 | @subsubsection Type maps | ||
5113 | |||
5114 | |||
5115 | Once an encrypted connection has been established, peers begin to exchange | ||
5116 | type maps. Type maps are used to allow the CORE service to determine which | ||
5117 | (encrypted) connections should be shown to which applications. A type map | ||
5118 | is an array of 65536 bits representing the different types of messages | ||
5119 | understood by applications using the CORE service. Each CORE service | ||
5120 | maintains this map, simply by setting the respective bit for each message | ||
5121 | type supported by any of the applications using the CORE service. Note | ||
5122 | that bits for message types embedded in higher-level protocols (such as | ||
5123 | MESH) will not be included in these type maps. | ||
5124 | |||
5125 | Typically, the type map of a peer will be sparse. Thus, the CORE service | ||
5126 | attempts to compress its type map using @code{gzip}-style compression | ||
5127 | ("deflate") prior to transmission. However, if the compression fails to | ||
5128 | compact the map, the map may also be transmitted without compression | ||
5129 | (resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or | ||
5130 | @code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively). | ||
5131 | Upon receiving a type map, the respective CORE service notifies | ||
5132 | applications about the connection to the other peer if they support any | ||
5133 | message type indicated in the type map (or no message type at all). | ||
5134 | If the CORE service experience a connect or disconnect event from an | ||
5135 | application, it updates its type map (setting or unsetting the respective | ||
5136 | bits) and notifies its neighbours about the change. | ||
5137 | The CORE services of the neighbours then in turn generate connect and | ||
5138 | disconnect events for the peer that sent the type map for their respective | ||
5139 | applications. As CORE messages may be lost, the CORE service confirms | ||
5140 | receiving a type map by sending back a | ||
5141 | @code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation | ||
5142 | (with the correct hash of the type map) is not received, the sender will | ||
5143 | retransmit the type map (with exponential back-off). | ||
5144 | |||
5145 | @cindex CADET Subsystem | ||
5146 | @cindex CADET | ||
5147 | @cindex cadet | ||
5148 | @node CADET Subsystem | ||
5149 | @section CADET Subsystem | ||
5150 | |||
5151 | The CADET subsystem in GNUnet is responsible for secure end-to-end | ||
5152 | communications between nodes in the GNUnet overlay network. CADET builds | ||
5153 | on the CORE subsystem which provides for the link-layer communication and | ||
5154 | then adds routing, forwarding and additional security to the connections. | ||
5155 | CADET offers the same cryptographic services as CORE, but on an | ||
5156 | end-to-end level. This is done so peers retransmitting traffic on behalf | ||
5157 | of other peers cannot access the payload data. | ||
5158 | |||
5159 | @itemize @bullet | ||
5160 | @item CADET provides confidentiality with so-called perfect forward | ||
5161 | secrecy; we use ECDHE powered by Curve25519 for the key exchange and then | ||
5162 | use symmetric encryption, encrypting with both AES-256 and Twofish | ||
5163 | @item authentication is achieved by signing the ephemeral keys using | ||
5164 | Ed25519, a deterministic variant of ECDSA | ||
5165 | @item integrity protection (using SHA-512 to do encrypt-then-MAC, although | ||
5166 | only 256 bits are sent to reduce overhead) | ||
5167 | @item replay protection (using nonces, timestamps, challenge-response, | ||
5168 | message counters and ephemeral keys) | ||
5169 | @item liveness (keep-alive messages, timeout) | ||
5170 | @end itemize | ||
5171 | |||
5172 | Additional to the CORE-like security benefits, CADET offers other | ||
5173 | properties that make it a more universal service than CORE. | ||
5174 | |||
5175 | @itemize @bullet | ||
5176 | @item CADET can establish channels to arbitrary peers in GNUnet. If a | ||
5177 | peer is not immediately reachable, CADET will find a path through the | ||
5178 | network and ask other peers to retransmit the traffic on its behalf. | ||
5179 | @item CADET offers (optional) reliability mechanisms. In a reliable | ||
5180 | channel traffic is guaranteed to arrive complete, unchanged and in-order. | ||
5181 | @item CADET takes care of flow and congestion control mechanisms, not | ||
5182 | allowing the sender to send more traffic than the receiver or the network | ||
5183 | are able to process. | ||
5184 | @end itemize | ||
5185 | |||
5186 | @menu | ||
5187 | * libgnunetcadet:: | ||
5188 | @end menu | ||
5189 | |||
5190 | @cindex libgnunetcadet | ||
5191 | @node libgnunetcadet | ||
5192 | @subsection libgnunetcadet | ||
5193 | |||
5194 | |||
5195 | The CADET API (defined in @file{gnunet_cadet_service.h}) is the | ||
5196 | messaging API used by P2P applications built using GNUnet. | ||
5197 | It provides applications the ability to send and receive encrypted | ||
5198 | messages to any peer participating in GNUnet. | ||
5199 | The API is heavily base on the CORE API. | ||
5200 | |||
5201 | CADET delivers messages to other peers in "channels". | ||
5202 | A channel is a permanent connection defined by a destination peer | ||
5203 | (identified by its public key) and a port number. | ||
5204 | Internally, CADET tunnels all channels towards a destination peer | ||
5205 | using one session key and relays the data on multiple "connections", | ||
5206 | independent from the channels. | ||
5207 | |||
5208 | Each channel has optional parameters, the most important being the | ||
5209 | reliability flag. | ||
5210 | Should a message get lost on TRANSPORT/CORE level, if a channel is | ||
5211 | created with as reliable, CADET will retransmit the lost message and | ||
5212 | deliver it in order to the destination application. | ||
5213 | |||
5214 | @pindex GNUNET_CADET_connect | ||
5215 | To communicate with other peers using CADET, it is necessary to first | ||
5216 | connect to the service using @code{GNUNET_CADET_connect}. | ||
5217 | This function takes several parameters in form of callbacks, to allow the | ||
5218 | client to react to various events, like incoming channels or channels that | ||
5219 | terminate, as well as specify a list of ports the client wishes to listen | ||
5220 | to (at the moment it is not possible to start listening on further ports | ||
5221 | once connected, but nothing prevents a client to connect several times to | ||
5222 | CADET, even do one connection per listening port). | ||
5223 | The function returns a handle which has to be used for any further | ||
5224 | interaction with the service. | ||
5225 | |||
5226 | @pindex GNUNET_CADET_channel_create | ||
5227 | To connect to a remote peer, a client has to call the | ||
5228 | @code{GNUNET_CADET_channel_create} function. The most important parameters | ||
5229 | given are the remote peer's identity (it public key) and a port, which | ||
5230 | specifies which application on the remote peer to connect to, similar to | ||
5231 | TCP/UDP ports. CADET will then find the peer in the GNUnet network and | ||
5232 | establish the proper low-level connections and do the necessary key | ||
5233 | exchanges to assure and authenticated, secure and verified communication. | ||
5234 | Similar to @code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel} | ||
5235 | returns a handle to interact with the created channel. | ||
5236 | |||
5237 | @pindex GNUNET_CADET_notify_transmit_ready | ||
5238 | For every message the client wants to send to the remote application, | ||
5239 | @code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the | ||
5240 | channel on which the message should be sent and the size of the message | ||
5241 | (but not the message itself!). Once CADET is ready to send the message, | ||
5242 | the provided callback will fire, and the message contents are provided to | ||
5243 | this callback. | ||
5244 | |||
5245 | Please note the CADET does not provide an explicit notification of when a | ||
5246 | channel is connected. In loosely connected networks, like big wireless | ||
5247 | mesh networks, this can take several seconds, even minutes in the worst | ||
5248 | case. To be alerted when a channel is online, a client can call | ||
5249 | @code{GNUNET_CADET_notify_transmit_ready} immediately after | ||
5250 | @code{GNUNET_CADET_create_channel}. When the callback is activated, it | ||
5251 | means that the channel is online. The callback can give 0 bytes to CADET | ||
5252 | if no message is to be sent, this is OK. | ||
5253 | |||
5254 | @pindex GNUNET_CADET_notify_transmit_cancel | ||
5255 | If a transmission was requested but before the callback fires it is no | ||
5256 | longer needed, it can be canceled with | ||
5257 | @code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle | ||
5258 | given back by @code{GNUNET_CADET_notify_transmit_ready}. | ||
5259 | As in the case of CORE, only one message can be requested at a time: a | ||
5260 | client must not call @code{GNUNET_CADET_notify_transmit_ready} again until | ||
5261 | the callback is called or the request is canceled. | ||
5262 | |||
5263 | @pindex GNUNET_CADET_channel_destroy | ||
5264 | When a channel is no longer needed, a client can call | ||
5265 | @code{GNUNET_CADET_channel_destroy} to get rid of it. | ||
5266 | Note that CADET will try to transmit all pending traffic before notifying | ||
5267 | the remote peer of the destruction of the channel, including | ||
5268 | retransmitting lost messages if the channel was reliable. | ||
5269 | |||
5270 | Incoming channels, channels being closed by the remote peer, and traffic | ||
5271 | on any incoming or outgoing channels are given to the client when CADET | ||
5272 | executes the callbacks given to it at the time of | ||
5273 | @code{GNUNET_CADET_connect}. | ||
5274 | |||
5275 | @pindex GNUNET_CADET_disconnect | ||
5276 | Finally, when an application no longer wants to use CADET, it should call | ||
5277 | @code{GNUNET_CADET_disconnect}, but first all channels and pending | ||
5278 | transmissions must be closed (otherwise CADET will complain). | ||
5279 | |||
5280 | @cindex NSE Subsystem | ||
5281 | @node NSE Subsystem | ||
5282 | @section NSE Subsystem | ||
5283 | |||
5284 | |||
5285 | NSE stands for @dfn{Network Size Estimation}. The NSE subsystem provides | ||
5286 | other subsystems and users with a rough estimate of the number of peers | ||
5287 | currently participating in the GNUnet overlay. | ||
5288 | The computed value is not a precise number as producing a precise number | ||
5289 | in a decentralized, efficient and secure way is impossible. | ||
5290 | While NSE's estimate is inherently imprecise, NSE also gives the expected | ||
5291 | range. For a peer that has been running in a stable network for a | ||
5292 | while, the real network size will typically (99.7% of the time) be in the | ||
5293 | range of [2/3 estimate, 3/2 estimate]. We will now give an overview of the | ||
5294 | algorithm used to calculate the estimate; | ||
5295 | all of the details can be found in this technical report. | ||
5296 | |||
5297 | @c FIXME: link to the report. | ||
5298 | |||
5299 | @menu | ||
5300 | * Motivation:: | ||
5301 | * Principle:: | ||
5302 | * libgnunetnse:: | ||
5303 | * The NSE Client-Service Protocol:: | ||
5304 | * The NSE Peer-to-Peer Protocol:: | ||
5305 | @end menu | ||
5306 | |||
5307 | @node Motivation | ||
5308 | @subsection Motivation | ||
5309 | |||
5310 | |||
5311 | Some subsystems, like DHT, need to know the size of the GNUnet network to | ||
5312 | optimize some parameters of their own protocol. The decentralized nature | ||
5313 | of GNUnet makes efficient and securely counting the exact number of peers | ||
5314 | infeasible. Although there are several decentralized algorithms to count | ||
5315 | the number of peers in a system, so far there is none to do so securely. | ||
5316 | Other protocols may allow any malicious peer to manipulate the final | ||
5317 | result or to take advantage of the system to perform | ||
5318 | @dfn{Denial of Service} (DoS) attacks against the network. | ||
5319 | GNUnet's NSE protocol avoids these drawbacks. | ||
5320 | |||
5321 | |||
5322 | |||
5323 | @menu | ||
5324 | * Security:: | ||
5325 | @end menu | ||
5326 | |||
5327 | @cindex NSE security | ||
5328 | @cindex nse security | ||
5329 | @node Security | ||
5330 | @subsubsection Security | ||
5331 | |||
5332 | |||
5333 | The NSE subsystem is designed to be resilient against these attacks. | ||
5334 | It uses @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proofs of work} | ||
5335 | to prevent one peer from impersonating a large number of participants, | ||
5336 | which would otherwise allow an adversary to artificially inflate the | ||
5337 | estimate. | ||
5338 | The DoS protection comes from the time-based nature of the protocol: | ||
5339 | the estimates are calculated periodically and out-of-time traffic is | ||
5340 | either ignored or stored for later retransmission by benign peers. | ||
5341 | In particular, peers cannot trigger global network communication at will. | ||
5342 | |||
5343 | @cindex NSE principle | ||
5344 | @cindex nse principle | ||
5345 | @node Principle | ||
5346 | @subsection Principle | ||
5347 | |||
5348 | |||
5349 | The algorithm calculates the estimate by finding the globally closest | ||
5350 | peer ID to a random, time-based value. | ||
5351 | |||
5352 | The idea is that the closer the ID is to the random value, the more | ||
5353 | "densely packed" the ID space is, and therefore, more peers are in the | ||
5354 | network. | ||
5355 | |||
5356 | |||
5357 | |||
5358 | @menu | ||
5359 | * Example:: | ||
5360 | * Algorithm:: | ||
5361 | * Target value:: | ||
5362 | * Timing:: | ||
5363 | * Controlled Flooding:: | ||
5364 | * Calculating the estimate:: | ||
5365 | @end menu | ||
5366 | |||
5367 | @node Example | ||
5368 | @subsubsection Example | ||
5369 | |||
5370 | |||
5371 | Suppose all peers have IDs between 0 and 100 (our ID space), and the | ||
5372 | random value is 42. | ||
5373 | If the closest peer has the ID 70 we can imagine that the average | ||
5374 | "distance" between peers is around 30 and therefore the are around 3 | ||
5375 | peers in the whole ID space. On the other hand, if the closest peer has | ||
5376 | the ID 44, we can imagine that the space is rather packed with peers, | ||
5377 | maybe as much as 50 of them. | ||
5378 | Naturally, we could have been rather unlucky, and there is only one peer | ||
5379 | and happens to have the ID 44. Thus, the current estimate is calculated | ||
5380 | as the average over multiple rounds, and not just a single sample. | ||
5381 | |||
5382 | @node Algorithm | ||
5383 | @subsubsection Algorithm | ||
5384 | |||
5385 | |||
5386 | Given that example, one can imagine that the job of the subsystem is to | ||
5387 | efficiently communicate the ID of the closest peer to the target value | ||
5388 | to all the other peers, who will calculate the estimate from it. | ||
5389 | |||
5390 | @node Target value | ||
5391 | @subsubsection Target value | ||
5392 | |||
5393 | |||
5394 | |||
5395 | The target value itself is generated by hashing the current time, rounded | ||
5396 | down to an agreed value. If the rounding amount is 1h (default) and the | ||
5397 | time is 12:34:56, the time to hash would be 12:00:00. The process is | ||
5398 | repeated each rounding amount (in this example would be every hour). | ||
5399 | Every repetition is called a round. | ||
5400 | |||
5401 | @node Timing | ||
5402 | @subsubsection Timing | ||
5403 | |||
5404 | |||
5405 | The NSE subsystem has some timing control to avoid everybody broadcasting | ||
5406 | its ID all at one. Once each peer has the target random value, it | ||
5407 | compares its own ID to the target and calculates the hypothetical size of | ||
5408 | the network if that peer were to be the closest. | ||
5409 | Then it compares the hypothetical size with the estimate from the previous | ||
5410 | rounds. For each value there is an associated point in the period, | ||
5411 | let's call it "broadcast time". If its own hypothetical estimate | ||
5412 | is the same as the previous global estimate, its "broadcast time" will be | ||
5413 | in the middle of the round. If its bigger it will be earlier and if its | ||
5414 | smaller (the most likely case) it will be later. This ensures that the | ||
5415 | peers closest to the target value start broadcasting their ID the first. | ||
5416 | |||
5417 | @node Controlled Flooding | ||
5418 | @subsubsection Controlled Flooding | ||
5419 | |||
5420 | |||
5421 | |||
5422 | When a peer receives a value, first it verifies that it is closer than the | ||
5423 | closest value it had so far, otherwise it answers the incoming message | ||
5424 | with a message containing the better value. Then it checks a proof of | ||
5425 | work that must be included in the incoming message, to ensure that the | ||
5426 | other peer's ID is not made up (otherwise a malicious peer could claim to | ||
5427 | have an ID of exactly the target value every round). Once validated, it | ||
5428 | compares the broadcast time of the received value with the current time | ||
5429 | and if it's not too early, sends the received value to its neighbors. | ||
5430 | Otherwise it stores the value until the correct broadcast time comes. | ||
5431 | This prevents unnecessary traffic of sub-optimal values, since a better | ||
5432 | value can come before the broadcast time, rendering the previous one | ||
5433 | obsolete and saving the traffic that would have been used to broadcast it | ||
5434 | to the neighbors. | ||
5435 | |||
5436 | @node Calculating the estimate | ||
5437 | @subsubsection Calculating the estimate | ||
5438 | |||
5439 | |||
5440 | |||
5441 | Once the closest ID has been spread across the network each peer gets the | ||
5442 | exact distance between this ID and the target value of the round and | ||
5443 | calculates the estimate with a mathematical formula described in the tech | ||
5444 | report. The estimate generated with this method for a single round is not | ||
5445 | very precise. Remember the case of the example, where the only peer is the | ||
5446 | ID 44 and we happen to generate the target value 42, thinking there are | ||
5447 | 50 peers in the network. Therefore, the NSE subsystem remembers the last | ||
5448 | 64 estimates and calculates an average over them, giving a result of which | ||
5449 | usually has one bit of uncertainty (the real size could be half of the | ||
5450 | estimate or twice as much). Note that the actual network size is | ||
5451 | calculated in powers of two of the raw input, thus one bit of uncertainty | ||
5452 | means a factor of two in the size estimate. | ||
5453 | |||
5454 | @cindex libgnunetnse | ||
5455 | @node libgnunetnse | ||
5456 | @subsection libgnunetnse | ||
5457 | |||
5458 | |||
5459 | |||
5460 | The NSE subsystem has the simplest API of all services, with only two | ||
5461 | calls: @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}. | ||
5462 | |||
5463 | The connect call gets a callback function as a parameter and this function | ||
5464 | is called each time the network agrees on an estimate. This usually is | ||
5465 | once per round, with some exceptions: if the closest peer has a late | ||
5466 | local clock and starts spreading its ID after everyone else agreed on a | ||
5467 | value, the callback might be activated twice in a round, the second value | ||
5468 | being always bigger than the first. The default round time is set to | ||
5469 | 1 hour. | ||
5470 | |||
5471 | The disconnect call disconnects from the NSE subsystem and the callback | ||
5472 | is no longer called with new estimates. | ||
5473 | |||
5474 | |||
5475 | |||
5476 | @menu | ||
5477 | * Results:: | ||
5478 | * libgnunetnse - Examples:: | ||
5479 | @end menu | ||
5480 | |||
5481 | @node Results | ||
5482 | @subsubsection Results | ||
5483 | |||
5484 | |||
5485 | |||
5486 | The callback provides two values: the average and the | ||
5487 | @uref{http://en.wikipedia.org/wiki/Standard_deviation, standard deviation} | ||
5488 | of the last 64 rounds. The values provided by the callback function are | ||
5489 | logarithmic, this means that the real estimate numbers can be obtained by | ||
5490 | calculating 2 to the power of the given value (2average). From a | ||
5491 | statistics point of view this means that: | ||
5492 | |||
5493 | @itemize @bullet | ||
5494 | @item 68% of the time the real size is included in the interval | ||
5495 | [(2average-stddev), 2] | ||
5496 | @item 95% of the time the real size is included in the interval | ||
5497 | [(2average-2*stddev, 2^average+2*stddev] | ||
5498 | @item 99.7% of the time the real size is included in the interval | ||
5499 | [(2average-3*stddev, 2average+3*stddev] | ||
5500 | @end itemize | ||
5501 | |||
5502 | The expected standard variation for 64 rounds in a network of stable size | ||
5503 | is 0.2. Thus, we can say that normally: | ||
5504 | |||
5505 | @itemize @bullet | ||
5506 | @item 68% of the time the real size is in the range [-13%, +15%] | ||
5507 | @item 95% of the time the real size is in the range [-24%, +32%] | ||
5508 | @item 99.7% of the time the real size is in the range [-34%, +52%] | ||
5509 | @end itemize | ||
5510 | |||
5511 | As said in the introduction, we can be quite sure that usually the real | ||
5512 | size is between one third and three times the estimate. This can of | ||
5513 | course vary with network conditions. | ||
5514 | Thus, applications may want to also consider the provided standard | ||
5515 | deviation value, not only the average (in particular, if the standard | ||
5516 | variation is very high, the average maybe meaningless: the network size is | ||
5517 | changing rapidly). | ||
5518 | |||
5519 | @node libgnunetnse - Examples | ||
5520 | @subsubsection libgnunetnse -Examples | ||
5521 | |||
5522 | |||
5523 | |||
5524 | Let's close with a couple examples. | ||
5525 | |||
5526 | @table @asis | ||
5527 | |||
5528 | @item Average: 10, std dev: 1 Here the estimate would be | ||
5529 | 2^10 = 1024 peers. (The range in which we can be 95% sure is: | ||
5530 | [2^8, 2^12] = [256, 4096]. We can be very (>99.7%) sure that the network | ||
5531 | is not a hundred peers and absolutely sure that it is not a million peers, | ||
5532 | but somewhere around a thousand.) | ||
5533 | |||
5534 | @item Average 22, std dev: 0.2 Here the estimate would be | ||
5535 | 2^22 = 4 Million peers. (The range in which we can be 99.7% sure | ||
5536 | is: [2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size | ||
5537 | is around four million, with absolutely way of it being 1 million.) | ||
5538 | |||
5539 | @end table | ||
5540 | |||
5541 | To put this in perspective, if someone remembers the LHC Higgs boson | ||
5542 | results, were announced with "5 sigma" and "6 sigma" certainties. In this | ||
5543 | case a 5 sigma minimum would be 2 million and a 6 sigma minimum, | ||
5544 | 1.8 million. | ||
5545 | |||
5546 | @node The NSE Client-Service Protocol | ||
5547 | @subsection The NSE Client-Service Protocol | ||
5548 | |||
5549 | |||
5550 | |||
5551 | As with the API, the client-service protocol is very simple, only has 2 | ||
5552 | different messages, defined in @code{src/nse/nse.h}: | ||
5553 | |||
5554 | @itemize @bullet | ||
5555 | @item @code{GNUNET_MESSAGE_TYPE_NSE_START}@ This message has no parameters | ||
5556 | and is sent from the client to the service upon connection. | ||
5557 | @item @code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE}@ This message is sent from | ||
5558 | the service to the client for every new estimate and upon connection. | ||
5559 | Contains a timestamp for the estimate, the average and the standard | ||
5560 | deviation for the respective round. | ||
5561 | @end itemize | ||
5562 | |||
5563 | When the @code{GNUNET_NSE_disconnect} API call is executed, the client | ||
5564 | simply disconnects from the service, with no message involved. | ||
5565 | |||
5566 | @cindex NSE Peer-to-Peer Protocol | ||
5567 | @node The NSE Peer-to-Peer Protocol | ||
5568 | @subsection The NSE Peer-to-Peer Protocol | ||
5569 | |||
5570 | |||
5571 | @pindex GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD | ||
5572 | The NSE subsystem only has one message in the P2P protocol, the | ||
5573 | @code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message. | ||
5574 | |||
5575 | This message key contents are the timestamp to identify the round | ||
5576 | (differences in system clocks may cause some peers to send messages way | ||
5577 | too early or way too late, so the timestamp allows other peers to | ||
5578 | identify such messages easily), the | ||
5579 | @uref{http://en.wikipedia.org/wiki/Proof-of-work_system, proof of work} | ||
5580 | used to make it difficult to mount a | ||
5581 | @uref{http://en.wikipedia.org/wiki/Sybil_attack, Sybil attack}, and the | ||
5582 | public key, which is used to verify the signature on the message. | ||
5583 | |||
5584 | Every peer stores a message for the previous, current and next round. The | ||
5585 | messages for the previous and current round are given to peers that | ||
5586 | connect to us. The message for the next round is simply stored until our | ||
5587 | system clock advances to the next round. The message for the current round | ||
5588 | is what we are flooding the network with right now. | ||
5589 | At the beginning of each round the peer does the following: | ||
5590 | |||
5591 | @itemize @bullet | ||
5592 | @item calculates its own distance to the target value | ||
5593 | @item creates, signs and stores the message for the current round (unless | ||
5594 | it has a better message in the "next round" slot which came early in the | ||
5595 | previous round) | ||
5596 | @item calculates, based on the stored round message (own or received) when | ||
5597 | to start flooding it to its neighbors | ||
5598 | @end itemize | ||
5599 | |||
5600 | Upon receiving a message the peer checks the validity of the message | ||
5601 | (round, proof of work, signature). The next action depends on the | ||
5602 | contents of the incoming message: | ||
5603 | |||
5604 | @itemize @bullet | ||
5605 | @item if the message is worse than the current stored message, the peer | ||
5606 | sends the current message back immediately, to stop the other peer from | ||
5607 | spreading suboptimal results | ||
5608 | @item if the message is better than the current stored message, the peer | ||
5609 | stores the new message and calculates the new target time to start | ||
5610 | spreading it to its neighbors (excluding the one the message came from) | ||
5611 | @item if the message is for the previous round, it is compared to the | ||
5612 | message stored in the "previous round slot", which may then be updated | ||
5613 | @item if the message is for the next round, it is compared to the message | ||
5614 | stored in the "next round slot", which again may then be updated | ||
5615 | @end itemize | ||
5616 | |||
5617 | Finally, when it comes to send the stored message for the current round to | ||
5618 | the neighbors there is a random delay added for each neighbor, to avoid | ||
5619 | traffic spikes and minimize cross-messages. | ||
5620 | |||
5621 | @cindex HOSTLIST Subsystem | ||
5622 | @node HOSTLIST Subsystem | ||
5623 | @section HOSTLIST Subsystem | ||
5624 | |||
5625 | |||
5626 | |||
5627 | Peers in the GNUnet overlay network need address information so that they | ||
5628 | can connect with other peers. GNUnet uses so called HELLO messages to | ||
5629 | store and exchange peer addresses. | ||
5630 | GNUnet provides several methods for peers to obtain this information: | ||
5631 | |||
5632 | @itemize @bullet | ||
5633 | @item out-of-band exchange of HELLO messages (manually, using for example | ||
5634 | gnunet-peerinfo) | ||
5635 | @item HELLO messages shipped with GNUnet (automatic with distribution) | ||
5636 | @item UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast) | ||
5637 | @item topology gossiping (learning from other peers we already connected | ||
5638 | to), and | ||
5639 | @item the HOSTLIST daemon covered in this section, which is particularly | ||
5640 | relevant for bootstrapping new peers. | ||
5641 | @end itemize | ||
5642 | |||
5643 | New peers have no existing connections (and thus cannot learn from gossip | ||
5644 | among peers), may not have other peers in their LAN and might be started | ||
5645 | with an outdated set of HELLO messages from the distribution. | ||
5646 | In this case, getting new peers to connect to the network requires either | ||
5647 | manual effort or the use of a HOSTLIST to obtain HELLOs. | ||
5648 | |||
5649 | @menu | ||
5650 | * HELLOs:: | ||
5651 | * Overview for the HOSTLIST subsystem:: | ||
5652 | * Interacting with the HOSTLIST daemon:: | ||
5653 | * Hostlist security address validation:: | ||
5654 | * The HOSTLIST daemon:: | ||
5655 | * The HOSTLIST server:: | ||
5656 | * The HOSTLIST client:: | ||
5657 | * Usage:: | ||
5658 | @end menu | ||
5659 | |||
5660 | @node HELLOs | ||
5661 | @subsection HELLOs | ||
5662 | |||
5663 | |||
5664 | |||
5665 | The basic information peers require to connect to other peers are | ||
5666 | contained in so called HELLO messages you can think of as a business card. | ||
5667 | Besides the identity of the peer (based on the cryptographic public key) a | ||
5668 | HELLO message may contain address information that specifies ways to | ||
5669 | contact a peer. By obtaining HELLO messages, a peer can learn how to | ||
5670 | contact other peers. | ||
5671 | |||
5672 | @node Overview for the HOSTLIST subsystem | ||
5673 | @subsection Overview for the HOSTLIST subsystem | ||
5674 | |||
5675 | |||
5676 | |||
5677 | The HOSTLIST subsystem provides a way to distribute and obtain contact | ||
5678 | information to connect to other peers using a simple HTTP GET request. | ||
5679 | It's implementation is split in three parts, the main file for the daemon | ||
5680 | itself (@file{gnunet-daemon-hostlist.c}), the HTTP client used to download | ||
5681 | peer information (@file{hostlist-client.c}) and the server component used | ||
5682 | to provide this information to other peers (@file{hostlist-server.c}). | ||
5683 | The server is basically a small HTTP web server (based on GNU | ||
5684 | libmicrohttpd) which provides a list of HELLOs known to the local peer for | ||
5685 | download. The client component is basically a HTTP client | ||
5686 | (based on libcurl) which can download hostlists from one or more websites. | ||
5687 | The hostlist format is a binary blob containing a sequence of HELLO | ||
5688 | messages. Note that any HTTP server can theoretically serve a hostlist, | ||
5689 | the built-in hostlist server makes it simply convenient to offer this | ||
5690 | service. | ||
5691 | |||
5692 | |||
5693 | @menu | ||
5694 | * Features:: | ||
5695 | * HOSTLIST - Limitations:: | ||
5696 | @end menu | ||
5697 | |||
5698 | @node Features | ||
5699 | @subsubsection Features | ||
5700 | |||
5701 | |||
5702 | |||
5703 | The HOSTLIST daemon can: | ||
5704 | |||
5705 | @itemize @bullet | ||
5706 | @item provide HELLO messages with validated addresses obtained from | ||
5707 | PEERINFO to download for other peers | ||
5708 | @item download HELLO messages and forward these message to the TRANSPORT | ||
5709 | subsystem for validation | ||
5710 | @item advertises the URL of this peer's hostlist address to other peers | ||
5711 | via gossip | ||
5712 | @item automatically learn about hostlist servers from the gossip of other | ||
5713 | peers | ||
5714 | @end itemize | ||
5715 | |||
5716 | @node HOSTLIST - Limitations | ||
5717 | @subsubsection HOSTLIST - Limitations | ||
5718 | |||
5719 | |||
5720 | |||
5721 | The HOSTLIST daemon does not: | ||
5722 | |||
5723 | @itemize @bullet | ||
5724 | @item verify the cryptographic information in the HELLO messages | ||
5725 | @item verify the address information in the HELLO messages | ||
5726 | @end itemize | ||
5727 | |||
5728 | @node Interacting with the HOSTLIST daemon | ||
5729 | @subsection Interacting with the HOSTLIST daemon | ||
5730 | |||
5731 | |||
5732 | |||
5733 | The HOSTLIST subsystem is currently implemented as a daemon, so there is | ||
5734 | no need for the user to interact with it and therefore there is no | ||
5735 | command line tool and no API to communicate with the daemon. In the | ||
5736 | future, we can envision changing this to allow users to manually trigger | ||
5737 | the download of a hostlist. | ||
5738 | |||
5739 | Since there is no command line interface to interact with HOSTLIST, the | ||
5740 | only way to interact with the hostlist is to use STATISTICS to obtain or | ||
5741 | modify information about the status of HOSTLIST: | ||
5742 | |||
5743 | @example | ||
5744 | $ gnunet-statistics -s hostlist | ||
5745 | @end example | ||
5746 | |||
5747 | @noindent | ||
5748 | In particular, HOSTLIST includes a @strong{persistent} value in statistics | ||
5749 | that specifies when the hostlist server might be queried next. As this | ||
5750 | value is exponentially increasing during runtime, developers may want to | ||
5751 | reset or manually adjust it. Note that HOSTLIST (but not STATISTICS) needs | ||
5752 | to be shutdown if changes to this value are to have any effect on the | ||
5753 | daemon (as HOSTLIST does not monitor STATISTICS for changes to the | ||
5754 | download frequency). | ||
5755 | |||
5756 | @node Hostlist security address validation | ||
5757 | @subsection Hostlist security address validation | ||
5758 | |||
5759 | |||
5760 | |||
5761 | Since information obtained from other parties cannot be trusted without | ||
5762 | validation, we have to distinguish between @emph{validated} and | ||
5763 | @emph{not validated} addresses. Before using (and so trusting) | ||
5764 | information from other parties, this information has to be double-checked | ||
5765 | (validated). Address validation is not done by HOSTLIST but by the | ||
5766 | TRANSPORT service. | ||
5767 | |||
5768 | The HOSTLIST component is functionally located between the PEERINFO and | ||
5769 | the TRANSPORT subsystem. When acting as a server, the daemon obtains valid | ||
5770 | (@emph{validated}) peer information (HELLO messages) from the PEERINFO | ||
5771 | service and provides it to other peers. When acting as a client, it | ||
5772 | contacts the HOSTLIST servers specified in the configuration, downloads | ||
5773 | the (unvalidated) list of HELLO messages and forwards these information | ||
5774 | to the TRANSPORT server to validate the addresses. | ||
5775 | |||
5776 | @cindex HOSTLIST daemon | ||
5777 | @node The HOSTLIST daemon | ||
5778 | @subsection The HOSTLIST daemon | ||
5779 | |||
5780 | |||
5781 | |||
5782 | The hostlist daemon is the main component of the HOSTLIST subsystem. It is | ||
5783 | started by the ARM service and (if configured) starts the HOSTLIST client | ||
5784 | and server components. | ||
5785 | |||
5786 | @pindex GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT | ||
5787 | If the daemon provides a hostlist itself it can advertise it's own | ||
5788 | hostlist to other peers. To do so it sends a | ||
5789 | @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to other peers | ||
5790 | when they connect to this peer on the CORE level. This hostlist | ||
5791 | advertisement message contains the URL to access the HOSTLIST HTTP | ||
5792 | server of the sender. The daemon may also subscribe to this type of | ||
5793 | message from CORE service, and then forward these kind of message to the | ||
5794 | HOSTLIST client. The client then uses all available URLs to download peer | ||
5795 | information when necessary. | ||
5796 | |||
5797 | When starting, the HOSTLIST daemon first connects to the CORE subsystem | ||
5798 | and if hostlist learning is enabled, registers a CORE handler to receive | ||
5799 | this kind of messages. Next it starts (if configured) the client and | ||
5800 | server. It passes pointers to CORE connect and disconnect and receive | ||
5801 | handlers where the client and server store their functions, so the daemon | ||
5802 | can notify them about CORE events. | ||
5803 | |||
5804 | To clean up on shutdown, the daemon has a cleaning task, shutting down all | ||
5805 | subsystems and disconnecting from CORE. | ||
5806 | |||
5807 | @cindex HOSTLIST server | ||
5808 | @node The HOSTLIST server | ||
5809 | @subsection The HOSTLIST server | ||
5810 | |||
5811 | |||
5812 | |||
5813 | The server provides a way for other peers to obtain HELLOs. Basically it | ||
5814 | is a small web server other peers can connect to and download a list of | ||
5815 | HELLOs using standard HTTP; it may also advertise the URL of the hostlist | ||
5816 | to other peers connecting on CORE level. | ||
5817 | |||
5818 | |||
5819 | @menu | ||
5820 | * The HTTP Server:: | ||
5821 | * Advertising the URL:: | ||
5822 | @end menu | ||
5823 | |||
5824 | @node The HTTP Server | ||
5825 | @subsubsection The HTTP Server | ||
5826 | |||
5827 | |||
5828 | |||
5829 | During startup, the server starts a web server listening on the port | ||
5830 | specified with the HTTPPORT value (default 8080). In addition it connects | ||
5831 | to the PEERINFO service to obtain peer information. The HOSTLIST server | ||
5832 | uses the GNUNET_PEERINFO_iterate function to request HELLO information for | ||
5833 | all peers and adds their information to a new hostlist if they are | ||
5834 | suitable (expired addresses and HELLOs without addresses are both not | ||
5835 | suitable) and the maximum size for a hostlist is not exceeded | ||
5836 | (MAX_BYTES_PER_HOSTLISTS = 500000). | ||
5837 | When PEERINFO finishes (with a last NULL callback), the server destroys | ||
5838 | the previous hostlist response available for download on the web server | ||
5839 | and replaces it with the updated hostlist. The hostlist format is | ||
5840 | basically a sequence of HELLO messages (as obtained from PEERINFO) without | ||
5841 | any special tokenization. Since each HELLO message contains a size field, | ||
5842 | the response can easily be split into separate HELLO messages by the | ||
5843 | client. | ||
5844 | |||
5845 | A HOSTLIST client connecting to the HOSTLIST server will receive the | ||
5846 | hostlist as an HTTP response and the server will terminate the | ||
5847 | connection with the result code @code{HTTP 200 OK}. | ||
5848 | The connection will be closed immediately if no hostlist is available. | ||
5849 | |||
5850 | @node Advertising the URL | ||
5851 | @subsubsection Advertising the URL | ||
5852 | |||
5853 | |||
5854 | |||
5855 | The server also advertises the URL to download the hostlist to other peers | ||
5856 | if hostlist advertisement is enabled. | ||
5857 | When a new peer connects and has hostlist learning enabled, the server | ||
5858 | sends a @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this | ||
5859 | peer using the CORE service. | ||
5860 | |||
5861 | @cindex HOSTLIST client | ||
5862 | @node The HOSTLIST client | ||
5863 | @subsection The HOSTLIST client | ||
5864 | |||
5865 | |||
5866 | |||
5867 | The client provides the functionality to download the list of HELLOs from | ||
5868 | a set of URLs. | ||
5869 | It performs a standard HTTP request to the URLs configured and learned | ||
5870 | from advertisement messages received from other peers. When a HELLO is | ||
5871 | downloaded, the HOSTLIST client forwards the HELLO to the TRANSPORT | ||
5872 | service for validation. | ||
5873 | |||
5874 | The client supports two modes of operation: | ||
5875 | |||
5876 | @itemize @bullet | ||
5877 | @item download of HELLOs (bootstrapping) | ||
5878 | @item learning of URLs | ||
5879 | @end itemize | ||
5880 | |||
5881 | @menu | ||
5882 | * Bootstrapping:: | ||
5883 | * Learning:: | ||
5884 | @end menu | ||
5885 | |||
5886 | @node Bootstrapping | ||
5887 | @subsubsection Bootstrapping | ||
5888 | |||
5889 | |||
5890 | |||
5891 | For bootstrapping, it schedules a task to download the hostlist from the | ||
5892 | set of known URLs. | ||
5893 | The downloads are only performed if the number of current | ||
5894 | connections is smaller than a minimum number of connections | ||
5895 | (at the moment 4). | ||
5896 | The interval between downloads increases exponentially; however, the | ||
5897 | exponential growth is limited if it becomes longer than an hour. | ||
5898 | At that point, the frequency growth is capped at | ||
5899 | (#number of connections * 1h). | ||
5900 | |||
5901 | Once the decision has been taken to download HELLOs, the daemon chooses a | ||
5902 | random URL from the list of known URLs. URLs can be configured in the | ||
5903 | configuration or be learned from advertisement messages. | ||
5904 | The client uses a HTTP client library (libcurl) to initiate the download | ||
5905 | using the libcurl multi interface. | ||
5906 | Libcurl passes the data to the callback_download function which | ||
5907 | stores the data in a buffer if space is available and the maximum size for | ||
5908 | a hostlist download is not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). | ||
5909 | When a full HELLO was downloaded, the HOSTLIST client offers this | ||
5910 | HELLO message to the TRANSPORT service for validation. | ||
5911 | When the download is finished or failed, statistical information about the | ||
5912 | quality of this URL is updated. | ||
5913 | |||
5914 | @cindex HOSTLIST learning | ||
5915 | @node Learning | ||
5916 | @subsubsection Learning | ||
5917 | |||
5918 | |||
5919 | |||
5920 | The client also manages hostlist advertisements from other peers. The | ||
5921 | HOSTLIST daemon forwards @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} | ||
5922 | messages to the client subsystem, which extracts the URL from the message. | ||
5923 | Next, a test of the newly obtained URL is performed by triggering a | ||
5924 | download from the new URL. If the URL works correctly, it is added to the | ||
5925 | list of working URLs. | ||
5926 | |||
5927 | The size of the list of URLs is restricted, so if an additional server is | ||
5928 | added and the list is full, the URL with the worst quality ranking | ||
5929 | (determined through successful downloads and number of HELLOs e.g.) is | ||
5930 | discarded. During shutdown the list of URLs is saved to a file for | ||
5931 | persistence and loaded on startup. URLs from the configuration file are | ||
5932 | never discarded. | ||
5933 | |||
5934 | @node Usage | ||
5935 | @subsection Usage | ||
5936 | |||
5937 | |||
5938 | |||
5939 | To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES | ||
5940 | section for the ARM services. This is done in the default configuration. | ||
5941 | |||
5942 | For more information on how to configure the HOSTLIST subsystem see the | ||
5943 | installation handbook:@ | ||
5944 | Configuring the hostlist to bootstrap@ | ||
5945 | Configuring your peer to provide a hostlist | ||
5946 | |||
5947 | @cindex IDENTITY Subsystem | ||
5948 | @node IDENTITY Subsystem | ||
5949 | @section IDENTITY Subsystem | ||
5950 | |||
5951 | |||
5952 | |||
5953 | Identities of "users" in GNUnet are called egos. | ||
5954 | Egos can be used as pseudonyms ("fake names") or be tied to an | ||
5955 | organization (for example, "GNU") or even the actual identity of a human. | ||
5956 | GNUnet users are expected to have many egos. They might have one tied to | ||
5957 | their real identity, some for organizations they manage, and more for | ||
5958 | different domains where they want to operate under a pseudonym. | ||
5959 | |||
5960 | The IDENTITY service allows users to manage their egos. The identity | ||
5961 | service manages the private keys egos of the local user; it does not | ||
5962 | manage identities of other users (public keys). Public keys for other | ||
5963 | users need names to become manageable. GNUnet uses the | ||
5964 | @dfn{GNU Name System} (GNS) to give names to other users and manage their | ||
5965 | public keys securely. This chapter is about the IDENTITY service, | ||
5966 | which is about the management of private keys. | ||
5967 | |||
5968 | On the network, an ego corresponds to an ECDSA key (over Curve25519, | ||
5969 | using RFC 6979, as required by GNS). Thus, users can perform actions | ||
5970 | under a particular ego by using (signing with) a particular private key. | ||
5971 | Other users can then confirm that the action was really performed by that | ||
5972 | ego by checking the signature against the respective public key. | ||
5973 | |||
5974 | The IDENTITY service allows users to associate a human-readable name with | ||
5975 | each ego. This way, users can use names that will remind them of the | ||
5976 | purpose of a particular ego. | ||
5977 | The IDENTITY service will store the respective private keys and | ||
5978 | allows applications to access key information by name. | ||
5979 | Users can change the name that is locally (!) associated with an ego. | ||
5980 | Egos can also be deleted, which means that the private key will be removed | ||
5981 | and it thus will not be possible to perform actions with that ego in the | ||
5982 | future. | ||
5983 | |||
5984 | Additionally, the IDENTITY subsystem can associate service functions with | ||
5985 | egos. | ||
5986 | For example, GNS requires the ego that should be used for the shorten | ||
5987 | zone. GNS will ask IDENTITY for an ego for the "gns-short" service. | ||
5988 | The IDENTITY service has a mapping of such service strings to the name of | ||
5989 | the ego that the user wants to use for this service, for example | ||
5990 | "my-short-zone-ego". | ||
5991 | |||
5992 | Finally, the IDENTITY API provides access to a special ego, the | ||
5993 | anonymous ego. The anonymous ego is special in that its private key is not | ||
5994 | really private, but fixed and known to everyone. | ||
5995 | Thus, anyone can perform actions as anonymous. This can be useful as with | ||
5996 | this trick, code does not have to contain a special case to distinguish | ||
5997 | between anonymous and pseudonymous egos. | ||
5998 | |||
5999 | @menu | ||
6000 | * libgnunetidentity:: | ||
6001 | * The IDENTITY Client-Service Protocol:: | ||
6002 | @end menu | ||
6003 | |||
6004 | @cindex libgnunetidentity | ||
6005 | @node libgnunetidentity | ||
6006 | @subsection libgnunetidentity | ||
6007 | |||
6008 | |||
6009 | |||
6010 | @menu | ||
6011 | * Connecting to the service:: | ||
6012 | * Operations on Egos:: | ||
6013 | * The anonymous Ego:: | ||
6014 | * Convenience API to lookup a single ego:: | ||
6015 | * Associating egos with service functions:: | ||
6016 | @end menu | ||
6017 | |||
6018 | @node Connecting to the service | ||
6019 | @subsubsection Connecting to the service | ||
6020 | |||
6021 | |||
6022 | |||
6023 | First, typical clients connect to the identity service using | ||
6024 | @code{GNUNET_IDENTITY_connect}. This function takes a callback as a | ||
6025 | parameter. | ||
6026 | If the given callback parameter is non-null, it will be invoked to notify | ||
6027 | the application about the current state of the identities in the system. | ||
6028 | |||
6029 | @itemize @bullet | ||
6030 | @item First, it will be invoked on all known egos at the time of the | ||
6031 | connection. For each ego, a handle to the ego and the user's name for the | ||
6032 | ego will be passed to the callback. Furthermore, a @code{void **} context | ||
6033 | argument will be provided which gives the client the opportunity to | ||
6034 | associate some state with the ego. | ||
6035 | @item Second, the callback will be invoked with NULL for the ego, the name | ||
6036 | and the context. This signals that the (initial) iteration over all egos | ||
6037 | has completed. | ||
6038 | @item Then, the callback will be invoked whenever something changes about | ||
6039 | an ego. | ||
6040 | If an ego is renamed, the callback is invoked with the ego handle of the | ||
6041 | ego that was renamed, and the new name. If an ego is deleted, the callback | ||
6042 | is invoked with the ego handle and a name of NULL. In the deletion case, | ||
6043 | the application should also release resources stored in the context. | ||
6044 | @item When the application destroys the connection to the identity service | ||
6045 | using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked | ||
6046 | with the ego and a name of NULL (equivalent to deletion of the egos). | ||
6047 | This should again be used to clean up the per-ego context. | ||
6048 | @end itemize | ||
6049 | |||
6050 | The ego handle passed to the callback remains valid until the callback is | ||
6051 | invoked with a name of NULL, so it is safe to store a reference to the | ||
6052 | ego's handle. | ||
6053 | |||
6054 | @node Operations on Egos | ||
6055 | @subsubsection Operations on Egos | ||
6056 | |||
6057 | |||
6058 | |||
6059 | Given an ego handle, the main operations are to get its associated private | ||
6060 | key using @code{GNUNET_IDENTITY_ego_get_private_key} or its associated | ||
6061 | public key using @code{GNUNET_IDENTITY_ego_get_public_key}. | ||
6062 | |||
6063 | The other operations on egos are pretty straightforward. | ||
6064 | Using @code{GNUNET_IDENTITY_create}, an application can request the | ||
6065 | creation of an ego by specifying the desired name. | ||
6066 | The operation will fail if that name is | ||
6067 | already in use. Using @code{GNUNET_IDENTITY_rename} the name of an | ||
6068 | existing ego can be changed. Finally, egos can be deleted using | ||
6069 | @code{GNUNET_IDENTITY_delete}. All of these operations will trigger | ||
6070 | updates to the callback given to the @code{GNUNET_IDENTITY_connect} | ||
6071 | function of all applications that are connected with the identity service | ||
6072 | at the time. @code{GNUNET_IDENTITY_cancel} can be used to cancel the | ||
6073 | operations before the respective continuations would be called. | ||
6074 | It is not guaranteed that the operation will not be completed anyway, | ||
6075 | only the continuation will no longer be called. | ||
6076 | |||
6077 | @node The anonymous Ego | ||
6078 | @subsubsection The anonymous Ego | ||
6079 | |||
6080 | |||
6081 | |||
6082 | A special way to obtain an ego handle is to call | ||
6083 | @code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the | ||
6084 | "anonymous" user --- anyone knows and can get the private key for this | ||
6085 | user, so it is suitable for operations that are supposed to be anonymous | ||
6086 | but require signatures (for example, to avoid a special path in the code). | ||
6087 | The anonymous ego is always valid and accessing it does not require a | ||
6088 | connection to the identity service. | ||
6089 | |||
6090 | @node Convenience API to lookup a single ego | ||
6091 | @subsubsection Convenience API to lookup a single ego | ||
6092 | |||
6093 | |||
6094 | As applications commonly simply have to lookup a single ego, there is a | ||
6095 | convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to | ||
6096 | lookup a single ego by name. Note that this is the user's name for the | ||
6097 | ego, not the service function. The resulting ego will be returned via a | ||
6098 | callback and will only be valid during that callback. The operation can | ||
6099 | be canceled via @code{GNUNET_IDENTITY_ego_lookup_cancel} | ||
6100 | (cancellation is only legal before the callback is invoked). | ||
6101 | |||
6102 | @node Associating egos with service functions | ||
6103 | @subsubsection Associating egos with service functions | ||
6104 | |||
6105 | |||
6106 | The @code{GNUNET_IDENTITY_set} function is used to associate a particular | ||
6107 | ego with a service function. The name used by the service and the ego are | ||
6108 | given as arguments. | ||
6109 | Afterwards, the service can use its name to lookup the associated ego | ||
6110 | using @code{GNUNET_IDENTITY_get}. | ||
6111 | |||
6112 | @node The IDENTITY Client-Service Protocol | ||
6113 | @subsection The IDENTITY Client-Service Protocol | ||
6114 | |||
6115 | |||
6116 | |||
6117 | A client connecting to the identity service first sends a message with | ||
6118 | type | ||
6119 | @code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that, the | ||
6120 | client will receive information about changes to the egos by receiving | ||
6121 | messages of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}. | ||
6122 | Those messages contain the private key of the ego and the user's name of | ||
6123 | the ego (or zero bytes for the name to indicate that the ego was deleted). | ||
6124 | A special bit @code{end_of_list} is used to indicate the end of the | ||
6125 | initial iteration over the identity service's egos. | ||
6126 | |||
6127 | The client can trigger changes to the egos by sending @code{CREATE}, | ||
6128 | @code{RENAME} or @code{DELETE} messages. | ||
6129 | The CREATE message contains the private key and the desired name.@ | ||
6130 | The RENAME message contains the old name and the new name.@ | ||
6131 | The DELETE message only needs to include the name of the ego to delete.@ | ||
6132 | The service responds to each of these messages with a @code{RESULT_CODE} | ||
6133 | message which indicates success or error of the operation, and possibly | ||
6134 | a human-readable error message. | ||
6135 | |||
6136 | Finally, the client can bind the name of a service function to an ego by | ||
6137 | sending a @code{SET_DEFAULT} message with the name of the service function | ||
6138 | and the private key of the ego. | ||
6139 | Such bindings can then be resolved using a @code{GET_DEFAULT} message, | ||
6140 | which includes the name of the service function. The identity service | ||
6141 | will respond to a GET_DEFAULT request with a SET_DEFAULT message | ||
6142 | containing the respective information, or with a RESULT_CODE to | ||
6143 | indicate an error. | ||
6144 | |||
6145 | @cindex NAMESTORE Subsystem | ||
6146 | @node NAMESTORE Subsystem | ||
6147 | @section NAMESTORE Subsystem | ||
6148 | |||
6149 | The NAMESTORE subsystem provides persistent storage for local GNS zone | ||
6150 | information. All local GNS zone information are managed by NAMESTORE. It | ||
6151 | provides both the functionality to administer local GNS information (e.g. | ||
6152 | delete and add records) as well as to retrieve GNS information (e.g to | ||
6153 | list name information in a client). | ||
6154 | NAMESTORE does only manage the persistent storage of zone information | ||
6155 | belonging to the user running the service: GNS information from other | ||
6156 | users obtained from the DHT are stored by the NAMECACHE subsystem. | ||
6157 | |||
6158 | NAMESTORE uses a plugin-based database backend to store GNS information | ||
6159 | with good performance. Here sqlite, MySQL and PostgreSQL are supported | ||
6160 | database backends. | ||
6161 | NAMESTORE clients interact with the IDENTITY subsystem to obtain | ||
6162 | cryptographic information about zones based on egos as described with the | ||
6163 | IDENTITY subsystem, but internally NAMESTORE refers to zones using the | ||
6164 | ECDSA private key. | ||
6165 | In addition, it collaborates with the NAMECACHE subsystem and | ||
6166 | stores zone information when local information are modified in the | ||
6167 | GNS cache to increase look-up performance for local information. | ||
6168 | |||
6169 | NAMESTORE provides functionality to look-up and store records, to iterate | ||
6170 | over a specific or all zones and to monitor zones for changes. NAMESTORE | ||
6171 | functionality can be accessed using the NAMESTORE api or the NAMESTORE | ||
6172 | command line tool. | ||
6173 | |||
6174 | @menu | ||
6175 | * libgnunetnamestore:: | ||
6176 | @end menu | ||
6177 | |||
6178 | @cindex libgnunetnamestore | ||
6179 | @node libgnunetnamestore | ||
6180 | @subsection libgnunetnamestore | ||
6181 | |||
6182 | To interact with NAMESTORE clients first connect to the NAMESTORE service | ||
6183 | using the @code{GNUNET_NAMESTORE_connect} passing a configuration handle. | ||
6184 | As a result they obtain a NAMESTORE handle, they can use for operations, | ||
6185 | or NULL is returned if the connection failed. | ||
6186 | |||
6187 | To disconnect from NAMESTORE, clients use | ||
6188 | @code{GNUNET_NAMESTORE_disconnect} and specify the handle to disconnect. | ||
6189 | |||
6190 | NAMESTORE internally uses the ECDSA private key to refer to zones. These | ||
6191 | private keys can be obtained from the IDENTITY subsystem. | ||
6192 | Here @emph{egos} @emph{can be used to refer to zones or the default ego | ||
6193 | assigned to the GNS subsystem can be used to obtained the master zone's | ||
6194 | private key.} | ||
6195 | |||
6196 | |||
6197 | @menu | ||
6198 | * Editing Zone Information:: | ||
6199 | * Iterating Zone Information:: | ||
6200 | * Monitoring Zone Information:: | ||
6201 | @end menu | ||
6202 | |||
6203 | @node Editing Zone Information | ||
6204 | @subsubsection Editing Zone Information | ||
6205 | |||
6206 | |||
6207 | |||
6208 | NAMESTORE provides functions to lookup records stored under a label in a | ||
6209 | zone and to store records under a label in a zone. | ||
6210 | |||
6211 | To store (and delete) records, the client uses the | ||
6212 | @code{GNUNET_NAMESTORE_records_store} function and has to provide | ||
6213 | namestore handle to use, the private key of the zone, the label to store | ||
6214 | the records under, the records and number of records plus an callback | ||
6215 | function. | ||
6216 | After the operation is performed NAMESTORE will call the provided | ||
6217 | callback function with the result GNUNET_SYSERR on failure | ||
6218 | (including timeout/queue drop/failure to validate), GNUNET_NO if content | ||
6219 | was already there or not found GNUNET_YES (or other positive value) on | ||
6220 | success plus an additional error message. | ||
6221 | |||
6222 | Records are deleted by using the store command with 0 records to store. | ||
6223 | It is important to note, that records are not merged when records exist | ||
6224 | with the label. | ||
6225 | So a client has first to retrieve records, merge with existing records | ||
6226 | and then store the result. | ||
6227 | |||
6228 | To perform a lookup operation, the client uses the | ||
6229 | @code{GNUNET_NAMESTORE_records_store} function. Here it has to pass the | ||
6230 | namestore handle, the private key of the zone and the label. It also has | ||
6231 | to provide a callback function which will be called with the result of | ||
6232 | the lookup operation: | ||
6233 | the zone for the records, the label, and the records including the | ||
6234 | number of records included. | ||
6235 | |||
6236 | A special operation is used to set the preferred nickname for a zone. | ||
6237 | This nickname is stored with the zone and is automatically merged with | ||
6238 | all labels and records stored in a zone. Here the client uses the | ||
6239 | @code{GNUNET_NAMESTORE_set_nick} function and passes the private key of | ||
6240 | the zone, the nickname as string plus a the callback with the result of | ||
6241 | the operation. | ||
6242 | |||
6243 | @node Iterating Zone Information | ||
6244 | @subsubsection Iterating Zone Information | ||
6245 | |||
6246 | |||
6247 | |||
6248 | A client can iterate over all information in a zone or all zones managed | ||
6249 | by NAMESTORE. | ||
6250 | Here a client uses the @code{GNUNET_NAMESTORE_zone_iteration_start} | ||
6251 | function and passes the namestore handle, the zone to iterate over and a | ||
6252 | callback function to call with the result. | ||
6253 | To iterate over all the zones, it is possible to pass NULL for the zone. | ||
6254 | A @code{GNUNET_NAMESTORE_ZoneIterator} handle is returned to be used to | ||
6255 | continue iteration. | ||
6256 | |||
6257 | NAMESTORE calls the callback for every result and expects the client to | ||
6258 | call @code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or | ||
6259 | @code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration. | ||
6260 | When NAMESTORE reached the last item it will call the callback with a | ||
6261 | NULL value to indicate. | ||
6262 | |||
6263 | @node Monitoring Zone Information | ||
6264 | @subsubsection Monitoring Zone Information | ||
6265 | |||
6266 | |||
6267 | |||
6268 | Clients can also monitor zones to be notified about changes. Here the | ||
6269 | clients uses the @code{GNUNET_NAMESTORE_zone_monitor_start} function and | ||
6270 | passes the private key of the zone and and a callback function to call | ||
6271 | with updates for a zone. | ||
6272 | The client can specify to obtain zone information first by iterating over | ||
6273 | the zone and specify a synchronization callback to be called when the | ||
6274 | client and the namestore are synced. | ||
6275 | |||
6276 | On an update, NAMESTORE will call the callback with the private key of the | ||
6277 | zone, the label and the records and their number. | ||
6278 | |||
6279 | To stop monitoring, the client calls | ||
6280 | @code{GNUNET_NAMESTORE_zone_monitor_stop} and passes the handle obtained | ||
6281 | from the function to start the monitoring. | ||
6282 | |||
6283 | @cindex PEERINFO Subsystem | ||
6284 | @node PEERINFO Subsystem | ||
6285 | @section PEERINFO Subsystem | ||
6286 | |||
6287 | |||
6288 | |||
6289 | The PEERINFO subsystem is used to store verified (validated) information | ||
6290 | about known peers in a persistent way. It obtains these addresses for | ||
6291 | example from TRANSPORT service which is in charge of address validation. | ||
6292 | Validation means that the information in the HELLO message are checked by | ||
6293 | connecting to the addresses and performing a cryptographic handshake to | ||
6294 | authenticate the peer instance stating to be reachable with these | ||
6295 | addresses. | ||
6296 | Peerinfo does not validate the HELLO messages itself but only stores them | ||
6297 | and gives them to interested clients. | ||
6298 | |||
6299 | As future work, we think about moving from storing just HELLO messages to | ||
6300 | providing a generic persistent per-peer information store. | ||
6301 | More and more subsystems tend to need to store per-peer information in | ||
6302 | persistent way. | ||
6303 | To not duplicate this functionality we plan to provide a PEERSTORE | ||
6304 | service providing this functionality. | ||
6305 | |||
6306 | @menu | ||
6307 | * PEERINFO - Features:: | ||
6308 | * PEERINFO - Limitations:: | ||
6309 | * DeveloperPeer Information:: | ||
6310 | * Startup:: | ||
6311 | * Managing Information:: | ||
6312 | * Obtaining Information:: | ||
6313 | * The PEERINFO Client-Service Protocol:: | ||
6314 | * libgnunetpeerinfo:: | ||
6315 | @end menu | ||
6316 | |||
6317 | @node PEERINFO - Features | ||
6318 | @subsection PEERINFO - Features | ||
6319 | |||
6320 | |||
6321 | |||
6322 | @itemize @bullet | ||
6323 | @item Persistent storage | ||
6324 | @item Client notification mechanism on update | ||
6325 | @item Periodic clean up for expired information | ||
6326 | @item Differentiation between public and friend-only HELLO | ||
6327 | @end itemize | ||
6328 | |||
6329 | @node PEERINFO - Limitations | ||
6330 | @subsection PEERINFO - Limitations | ||
6331 | |||
6332 | |||
6333 | @itemize @bullet | ||
6334 | @item Does not perform HELLO validation | ||
6335 | @end itemize | ||
6336 | |||
6337 | @node DeveloperPeer Information | ||
6338 | @subsection DeveloperPeer Information | ||
6339 | |||
6340 | |||
6341 | |||
6342 | The PEERINFO subsystem stores these information in the form of HELLO | ||
6343 | messages you can think of as business cards. | ||
6344 | These HELLO messages contain the public key of a peer and the addresses | ||
6345 | a peer can be reached under. | ||
6346 | The addresses include an expiration date describing how long they are | ||
6347 | valid. This information is updated regularly by the TRANSPORT service by | ||
6348 | revalidating the address. | ||
6349 | If an address is expired and not renewed, it can be removed from the | ||
6350 | HELLO message. | ||
6351 | |||
6352 | Some peer do not want to have their HELLO messages distributed to other | ||
6353 | peers, especially when GNUnet's friend-to-friend modus is enabled. | ||
6354 | To prevent this undesired distribution. PEERINFO distinguishes between | ||
6355 | @emph{public} and @emph{friend-only} HELLO messages. | ||
6356 | Public HELLO messages can be freely distributed to other (possibly | ||
6357 | unknown) peers (for example using the hostlist, gossiping, broadcasting), | ||
6358 | whereas friend-only HELLO messages may not be distributed to other peers. | ||
6359 | Friend-only HELLO messages have an additional flag @code{friend_only} set | ||
6360 | internally. For public HELLO message this flag is not set. | ||
6361 | PEERINFO does and cannot not check if a client is allowed to obtain a | ||
6362 | specific HELLO type. | ||
6363 | |||
6364 | The HELLO messages can be managed using the GNUnet HELLO library. | ||
6365 | Other GNUnet systems can obtain these information from PEERINFO and use | ||
6366 | it for their purposes. | ||
6367 | Clients are for example the HOSTLIST component providing these | ||
6368 | information to other peers in form of a hostlist or the TRANSPORT | ||
6369 | subsystem using these information to maintain connections to other peers. | ||
6370 | |||
6371 | @node Startup | ||
6372 | @subsection Startup | ||
6373 | |||
6374 | |||
6375 | |||
6376 | During startup the PEERINFO services loads persistent HELLOs from disk. | ||
6377 | First PEERINFO parses the directory configured in the HOSTS value of the | ||
6378 | @code{PEERINFO} configuration section to store PEERINFO information. | ||
6379 | For all files found in this directory valid HELLO messages are extracted. | ||
6380 | In addition it loads HELLO messages shipped with the GNUnet distribution. | ||
6381 | These HELLOs are used to simplify network bootstrapping by providing | ||
6382 | valid peer information with the distribution. | ||
6383 | The use of these HELLOs can be prevented by setting the | ||
6384 | @code{USE_INCLUDED_HELLOS} in the @code{PEERINFO} configuration section to | ||
6385 | @code{NO}. Files containing invalid information are removed. | ||
6386 | |||
6387 | @node Managing Information | ||
6388 | @subsection Managing Information | ||
6389 | |||
6390 | |||
6391 | |||
6392 | The PEERINFO services stores information about known PEERS and a single | ||
6393 | HELLO message for every peer. | ||
6394 | A peer does not need to have a HELLO if no information are available. | ||
6395 | HELLO information from different sources, for example a HELLO obtained | ||
6396 | from a remote HOSTLIST and a second HELLO stored on disk, are combined | ||
6397 | and merged into one single HELLO message per peer which will be given to | ||
6398 | clients. During this merge process the HELLO is immediately written to | ||
6399 | disk to ensure persistence. | ||
6400 | |||
6401 | PEERINFO in addition periodically scans the directory where information | ||
6402 | are stored for empty HELLO messages with expired TRANSPORT addresses. | ||
6403 | This periodic task scans all files in the directory and recreates the | ||
6404 | HELLO messages it finds. | ||
6405 | Expired TRANSPORT addresses are removed from the HELLO and if the | ||
6406 | HELLO does not contain any valid addresses, it is discarded and removed | ||
6407 | from the disk. | ||
6408 | |||
6409 | @node Obtaining Information | ||
6410 | @subsection Obtaining Information | ||
6411 | |||
6412 | |||
6413 | |||
6414 | When a client requests information from PEERINFO, PEERINFO performs a | ||
6415 | lookup for the respective peer or all peers if desired and transmits this | ||
6416 | information to the client. | ||
6417 | The client can specify if friend-only HELLOs have to be included or not | ||
6418 | and PEERINFO filters the respective HELLO messages before transmitting | ||
6419 | information. | ||
6420 | |||
6421 | To notify clients about changes to PEERINFO information, PEERINFO | ||
6422 | maintains a list of clients interested in this notifications. | ||
6423 | Such a notification occurs if a HELLO for a peer was updated (due to a | ||
6424 | merge for example) or a new peer was added. | ||
6425 | |||
6426 | @node The PEERINFO Client-Service Protocol | ||
6427 | @subsection The PEERINFO Client-Service Protocol | ||
6428 | |||
6429 | |||
6430 | |||
6431 | To connect and disconnect to and from the PEERINFO Service PEERINFO | ||
6432 | utilizes the util client/server infrastructure, so no special messages | ||
6433 | types are used here. | ||
6434 | |||
6435 | To add information for a peer, the plain HELLO message is transmitted to | ||
6436 | the service without any wrapping. All pieces of information required are | ||
6437 | stored within the HELLO message. | ||
6438 | The PEERINFO service provides a message handler accepting and processing | ||
6439 | these HELLO messages. | ||
6440 | |||
6441 | When obtaining PEERINFO information using the iterate functionality | ||
6442 | specific messages are used. To obtain information for all peers, a | ||
6443 | @code{struct ListAllPeersMessage} with message type | ||
6444 | @code{GNUNET_MESSAGE_TYPE_PEERINFO_GET_ALL} and a flag | ||
6445 | include_friend_only to indicate if friend-only HELLO messages should be | ||
6446 | included are transmitted. If information for a specific peer is required | ||
6447 | a @code{struct ListAllPeersMessage} with | ||
6448 | @code{GNUNET_MESSAGE_TYPE_PEERINFO_GET} containing the peer identity is | ||
6449 | used. | ||
6450 | |||
6451 | For both variants the PEERINFO service replies for each HELLO message it | ||
6452 | wants to transmit with a @code{struct ListAllPeersMessage} with type | ||
6453 | @code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO} containing the plain HELLO. | ||
6454 | The final message is @code{struct GNUNET_MessageHeader} with type | ||
6455 | @code{GNUNET_MESSAGE_TYPE_PEERINFO_INFO}. If the client receives this | ||
6456 | message, it can proceed with the next request if any is pending. | ||
6457 | |||
6458 | @node libgnunetpeerinfo | ||
6459 | @subsection libgnunetpeerinfo | ||
6460 | |||
6461 | |||
6462 | |||
6463 | The PEERINFO API consists mainly of three different functionalities: | ||
6464 | |||
6465 | @itemize @bullet | ||
6466 | @item maintaining a connection to the service | ||
6467 | @item adding new information to the PEERINFO service | ||
6468 | @item retrieving information from the PEERINFO service | ||
6469 | @end itemize | ||
6470 | |||
6471 | @menu | ||
6472 | * Connecting to the PEERINFO Service:: | ||
6473 | * Adding Information to the PEERINFO Service:: | ||
6474 | * Obtaining Information from the PEERINFO Service:: | ||
6475 | @end menu | ||
6476 | |||
6477 | @node Connecting to the PEERINFO Service | ||
6478 | @subsubsection Connecting to the PEERINFO Service | ||
6479 | |||
6480 | |||
6481 | |||
6482 | To connect to the PEERINFO service the function | ||
6483 | @code{GNUNET_PEERINFO_connect} is used, taking a configuration handle as | ||
6484 | an argument, and to disconnect from PEERINFO the function | ||
6485 | @code{GNUNET_PEERINFO_disconnect}, taking the PEERINFO | ||
6486 | handle returned from the connect function has to be called. | ||
6487 | |||
6488 | @node Adding Information to the PEERINFO Service | ||
6489 | @subsubsection Adding Information to the PEERINFO Service | ||
6490 | |||
6491 | |||
6492 | |||
6493 | @code{GNUNET_PEERINFO_add_peer} adds a new peer to the PEERINFO subsystem | ||
6494 | storage. This function takes the PEERINFO handle as an argument, the HELLO | ||
6495 | message to store and a continuation with a closure to be called with the | ||
6496 | result of the operation. | ||
6497 | The @code{GNUNET_PEERINFO_add_peer} returns a handle to this operation | ||
6498 | allowing to cancel the operation with the respective cancel function | ||
6499 | @code{GNUNET_PEERINFO_add_peer_cancel}. To retrieve information from | ||
6500 | PEERINFO you can iterate over all information stored with PEERINFO or you | ||
6501 | can tell PEERINFO to notify if new peer information are available. | ||
6502 | |||
6503 | @node Obtaining Information from the PEERINFO Service | ||
6504 | @subsubsection Obtaining Information from the PEERINFO Service | ||
6505 | |||
6506 | |||
6507 | |||
6508 | To iterate over information in PEERINFO you use | ||
6509 | @code{GNUNET_PEERINFO_iterate}. | ||
6510 | This function expects the PEERINFO handle, a flag if HELLO messages | ||
6511 | intended for friend only mode should be included, a timeout how long the | ||
6512 | operation should take and a callback with a callback closure to be called | ||
6513 | for the results. | ||
6514 | If you want to obtain information for a specific peer, you can specify | ||
6515 | the peer identity, if this identity is NULL, information for all peers are | ||
6516 | returned. The function returns a handle to allow to cancel the operation | ||
6517 | using @code{GNUNET_PEERINFO_iterate_cancel}. | ||
6518 | |||
6519 | To get notified when peer information changes, you can use | ||
6520 | @code{GNUNET_PEERINFO_notify}. | ||
6521 | This function expects a configuration handle and a flag if friend-only | ||
6522 | HELLO messages should be included. The PEERINFO service will notify you | ||
6523 | about every change and the callback function will be called to notify you | ||
6524 | about changes. The function returns a handle to cancel notifications | ||
6525 | with @code{GNUNET_PEERINFO_notify_cancel}. | ||
6526 | |||
6527 | @cindex PEERSTORE Subsystem | ||
6528 | @node PEERSTORE Subsystem | ||
6529 | @section PEERSTORE Subsystem | ||
6530 | |||
6531 | |||
6532 | |||
6533 | GNUnet's PEERSTORE subsystem offers persistent per-peer storage for other | ||
6534 | GNUnet subsystems. GNUnet subsystems can use PEERSTORE to persistently | ||
6535 | store and retrieve arbitrary data. | ||
6536 | Each data record stored with PEERSTORE contains the following fields: | ||
6537 | |||
6538 | @itemize @bullet | ||
6539 | @item subsystem: Name of the subsystem responsible for the record. | ||
6540 | @item peerid: Identity of the peer this record is related to. | ||
6541 | @item key: a key string identifying the record. | ||
6542 | @item value: binary record value. | ||
6543 | @item expiry: record expiry date. | ||
6544 | @end itemize | ||
6545 | |||
6546 | @menu | ||
6547 | * Functionality:: | ||
6548 | * Architecture:: | ||
6549 | * libgnunetpeerstore:: | ||
6550 | @end menu | ||
6551 | |||
6552 | @node Functionality | ||
6553 | @subsection Functionality | ||
6554 | |||
6555 | |||
6556 | |||
6557 | Subsystems can store any type of value under a (subsystem, peerid, key) | ||
6558 | combination. A "replace" flag set during store operations forces the | ||
6559 | PEERSTORE to replace any old values stored under the same | ||
6560 | (subsystem, peerid, key) combination with the new value. | ||
6561 | Additionally, an expiry date is set after which the record is *possibly* | ||
6562 | deleted by PEERSTORE. | ||
6563 | |||
6564 | Subsystems can iterate over all values stored under any of the following | ||
6565 | combination of fields: | ||
6566 | |||
6567 | @itemize @bullet | ||
6568 | @item (subsystem) | ||
6569 | @item (subsystem, peerid) | ||
6570 | @item (subsystem, key) | ||
6571 | @item (subsystem, peerid, key) | ||
6572 | @end itemize | ||
6573 | |||
6574 | Subsystems can also request to be notified about any new values stored | ||
6575 | under a (subsystem, peerid, key) combination by sending a "watch" | ||
6576 | request to PEERSTORE. | ||
6577 | |||
6578 | @node Architecture | ||
6579 | @subsection Architecture | ||
6580 | |||
6581 | |||
6582 | |||
6583 | PEERSTORE implements the following components: | ||
6584 | |||
6585 | @itemize @bullet | ||
6586 | @item PEERSTORE service: Handles store, iterate and watch operations. | ||
6587 | @item PEERSTORE API: API to be used by other subsystems to communicate and | ||
6588 | issue commands to the PEERSTORE service. | ||
6589 | @item PEERSTORE plugins: Handles the persistent storage. At the moment, | ||
6590 | only an "sqlite" plugin is implemented. | ||
6591 | @end itemize | ||
6592 | |||
6593 | @cindex libgnunetpeerstore | ||
6594 | @node libgnunetpeerstore | ||
6595 | @subsection libgnunetpeerstore | ||
6596 | |||
6597 | |||
6598 | |||
6599 | libgnunetpeerstore is the library containing the PEERSTORE API. Subsystems | ||
6600 | wishing to communicate with the PEERSTORE service use this API to open a | ||
6601 | connection to PEERSTORE. This is done by calling | ||
6602 | @code{GNUNET_PEERSTORE_connect} which returns a handle to the newly | ||
6603 | created connection. | ||
6604 | This handle has to be used with any further calls to the API. | ||
6605 | |||
6606 | To store a new record, the function @code{GNUNET_PEERSTORE_store} is to | ||
6607 | be used which requires the record fields and a continuation function that | ||
6608 | will be called by the API after the STORE request is sent to the | ||
6609 | PEERSTORE service. | ||
6610 | Note that calling the continuation function does not mean that the record | ||
6611 | is successfully stored, only that the STORE request has been successfully | ||
6612 | sent to the PEERSTORE service. | ||
6613 | @code{GNUNET_PEERSTORE_store_cancel} can be called to cancel the STORE | ||
6614 | request only before the continuation function has been called. | ||
6615 | |||
6616 | To iterate over stored records, the function | ||
6617 | @code{GNUNET_PEERSTORE_iterate} is | ||
6618 | to be used. @emph{peerid} and @emph{key} can be set to NULL. An iterator | ||
6619 | callback function will be called with each matching record found and a | ||
6620 | NULL record at the end to signal the end of result set. | ||
6621 | @code{GNUNET_PEERSTORE_iterate_cancel} can be used to cancel the ITERATE | ||
6622 | request before the iterator callback is called with a NULL record. | ||
6623 | |||
6624 | To be notified with new values stored under a (subsystem, peerid, key) | ||
6625 | combination, the function @code{GNUNET_PEERSTORE_watch} is to be used. | ||
6626 | This will register the watcher with the PEERSTORE service, any new | ||
6627 | records matching the given combination will trigger the callback | ||
6628 | function passed to @code{GNUNET_PEERSTORE_watch}. This continues until | ||
6629 | @code{GNUNET_PEERSTORE_watch_cancel} is called or the connection to the | ||
6630 | service is destroyed. | ||
6631 | |||
6632 | After the connection is no longer needed, the function | ||
6633 | @code{GNUNET_PEERSTORE_disconnect} can be called to disconnect from the | ||
6634 | PEERSTORE service. | ||
6635 | Any pending ITERATE or WATCH requests will be destroyed. | ||
6636 | If the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will | ||
6637 | delay the disconnection until all pending STORE requests are sent to | ||
6638 | the PEERSTORE service, otherwise, the pending STORE requests will be | ||
6639 | destroyed as well. | ||
6640 | |||
6641 | @cindex SET Subsystem | ||
6642 | @node SET Subsystem | ||
6643 | @section SET Subsystem | ||
6644 | |||
6645 | The SET subsystem is in process of being replaced by the SETU and | ||
6646 | SETI subsystems, which provide basically the same functionality, | ||
6647 | just using two different subsystems. SETI and SETU should be used | ||
6648 | for new code. | ||
6649 | |||
6650 | The SET service implements efficient set operations between two peers | ||
6651 | over a CADET tunnel. | ||
6652 | Currently, set union and set intersection are the only supported | ||
6653 | operations. Elements of a set consist of an @emph{element type} and | ||
6654 | arbitrary binary @emph{data}. | ||
6655 | The size of an element's data is limited to around 62 KB. | ||
6656 | |||
6657 | @menu | ||
6658 | * Local Sets:: | ||
6659 | * Set Modifications:: | ||
6660 | * Set Operations:: | ||
6661 | * Result Elements:: | ||
6662 | * libgnunetset:: | ||
6663 | * The SET Client-Service Protocol:: | ||
6664 | * The SET Intersection Peer-to-Peer Protocol:: | ||
6665 | * The SET Union Peer-to-Peer Protocol:: | ||
6666 | @end menu | ||
6667 | |||
6668 | @node Local Sets | ||
6669 | @subsection Local Sets | ||
6670 | |||
6671 | |||
6672 | |||
6673 | Sets created by a local client can be modified and reused for multiple | ||
6674 | operations. As each set operation requires potentially expensive special | ||
6675 | auxiliary data to be computed for each element of a set, a set can only | ||
6676 | participate in one type of set operation (either union or intersection). | ||
6677 | The type of a set is determined upon its creation. | ||
6678 | If a the elements of a set are needed for an operation of a different | ||
6679 | type, all of the set's element must be copied to a new set of appropriate | ||
6680 | type. | ||
6681 | |||
6682 | @node Set Modifications | ||
6683 | @subsection Set Modifications | ||
6684 | |||
6685 | |||
6686 | |||
6687 | Even when set operations are active, one can add to and remove elements | ||
6688 | from a set. | ||
6689 | However, these changes will only be visible to operations that have been | ||
6690 | created after the changes have taken place. That is, every set operation | ||
6691 | only sees a snapshot of the set from the time the operation was started. | ||
6692 | This mechanism is @emph{not} implemented by copying the whole set, but by | ||
6693 | attaching @emph{generation information} to each element and operation. | ||
6694 | |||
6695 | @node Set Operations | ||
6696 | @subsection Set Operations | ||
6697 | |||
6698 | |||
6699 | |||
6700 | Set operations can be started in two ways: Either by accepting an | ||
6701 | operation request from a remote peer, or by requesting a set operation | ||
6702 | from a remote peer. | ||
6703 | Set operations are uniquely identified by the involved @emph{peers}, an | ||
6704 | @emph{application id} and the @emph{operation type}. | ||
6705 | |||
6706 | The client is notified of incoming set operations by @emph{set listeners}. | ||
6707 | A set listener listens for incoming operations of a specific operation | ||
6708 | type and application id. | ||
6709 | Once notified of an incoming set request, the client can accept the set | ||
6710 | request (providing a local set for the operation) or reject it. | ||
6711 | |||
6712 | @node Result Elements | ||
6713 | @subsection Result Elements | ||
6714 | |||
6715 | |||
6716 | |||
6717 | The SET service has three @emph{result modes} that determine how an | ||
6718 | operation's result set is delivered to the client: | ||
6719 | |||
6720 | @itemize @bullet | ||
6721 | @item @strong{Full Result Set.} All elements of set resulting from the set | ||
6722 | operation are returned to the client. | ||
6723 | @item @strong{Added Elements.} Only elements that result from the | ||
6724 | operation and are not already in the local peer's set are returned. | ||
6725 | Note that for some operations (like set intersection) this result mode | ||
6726 | will never return any elements. | ||
6727 | This can be useful if only the remove peer is actually interested in | ||
6728 | the result of the set operation. | ||
6729 | @item @strong{Removed Elements.} Only elements that are in the local | ||
6730 | peer's initial set but not in the operation's result set are returned. | ||
6731 | Note that for some operations (like set union) this result mode will | ||
6732 | never return any elements. This can be useful if only the remove peer is | ||
6733 | actually interested in the result of the set operation. | ||
6734 | @end itemize | ||
6735 | |||
6736 | @cindex libgnunetset | ||
6737 | @node libgnunetset | ||
6738 | @subsection libgnunetset | ||
6739 | |||
6740 | |||
6741 | |||
6742 | @menu | ||
6743 | * Sets:: | ||
6744 | * Listeners:: | ||
6745 | * Operations:: | ||
6746 | * Supplying a Set:: | ||
6747 | * The Result Callback:: | ||
6748 | @end menu | ||
6749 | |||
6750 | @node Sets | ||
6751 | @subsubsection Sets | ||
6752 | |||
6753 | |||
6754 | |||
6755 | New sets are created with @code{GNUNET_SET_create}. Both the local peer's | ||
6756 | configuration (as each set has its own client connection) and the | ||
6757 | operation type must be specified. | ||
6758 | The set exists until either the client calls @code{GNUNET_SET_destroy} or | ||
6759 | the client's connection to the service is disrupted. | ||
6760 | In the latter case, the client is notified by the return value of | ||
6761 | functions dealing with sets. This return value must always be checked. | ||
6762 | |||
6763 | Elements are added and removed with @code{GNUNET_SET_add_element} and | ||
6764 | @code{GNUNET_SET_remove_element}. | ||
6765 | |||
6766 | @node Listeners | ||
6767 | @subsubsection Listeners | ||
6768 | |||
6769 | |||
6770 | |||
6771 | Listeners are created with @code{GNUNET_SET_listen}. Each time time a | ||
6772 | remote peer suggests a set operation with an application id and operation | ||
6773 | type matching a listener, the listener's callback is invoked. | ||
6774 | The client then must synchronously call either @code{GNUNET_SET_accept} | ||
6775 | or @code{GNUNET_SET_reject}. Note that the operation will not be started | ||
6776 | until the client calls @code{GNUNET_SET_commit} | ||
6777 | (see Section "Supplying a Set"). | ||
6778 | |||
6779 | @node Operations | ||
6780 | @subsubsection Operations | ||
6781 | |||
6782 | |||
6783 | |||
6784 | Operations to be initiated by the local peer are created with | ||
6785 | @code{GNUNET_SET_prepare}. Note that the operation will not be started | ||
6786 | until the client calls @code{GNUNET_SET_commit} | ||
6787 | (see Section "Supplying a Set"). | ||
6788 | |||
6789 | @node Supplying a Set | ||
6790 | @subsubsection Supplying a Set | ||
6791 | |||
6792 | |||
6793 | |||
6794 | To create symmetry between the two ways of starting a set operation | ||
6795 | (accepting and initiating it), the operation handles returned by | ||
6796 | @code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare} do not yet have a | ||
6797 | set to operate on, thus they can not do any work yet. | ||
6798 | |||
6799 | The client must call @code{GNUNET_SET_commit} to specify a set to use for | ||
6800 | an operation. @code{GNUNET_SET_commit} may only be called once per set | ||
6801 | operation. | ||
6802 | |||
6803 | @node The Result Callback | ||
6804 | @subsubsection The Result Callback | ||
6805 | |||
6806 | |||
6807 | |||
6808 | Clients must specify both a result mode and a result callback with | ||
6809 | @code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result | ||
6810 | callback with a status indicating either that an element was received, or | ||
6811 | the operation failed or succeeded. | ||
6812 | The interpretation of the received element depends on the result mode. | ||
6813 | The callback needs to know which result mode it is used in, as the | ||
6814 | arguments do not indicate if an element is part of the full result set, | ||
6815 | or if it is in the difference between the original set and the final set. | ||
6816 | |||
6817 | @node The SET Client-Service Protocol | ||
6818 | @subsection The SET Client-Service Protocol | ||
6819 | |||
6820 | |||
6821 | |||
6822 | @menu | ||
6823 | * Creating Sets:: | ||
6824 | * Listeners2:: | ||
6825 | * Initiating Operations:: | ||
6826 | * Modifying Sets:: | ||
6827 | * Results and Operation Status:: | ||
6828 | * Iterating Sets:: | ||
6829 | @end menu | ||
6830 | |||
6831 | @node Creating Sets | ||
6832 | @subsubsection Creating Sets | ||
6833 | |||
6834 | |||
6835 | |||
6836 | For each set of a client, there exists a client connection to the service. | ||
6837 | Sets are created by sending the @code{GNUNET_SERVICE_SET_CREATE} message | ||
6838 | over a new client connection. Multiple operations for one set are | ||
6839 | multiplexed over one client connection, using a request id supplied by | ||
6840 | the client. | ||
6841 | |||
6842 | @node Listeners2 | ||
6843 | @subsubsection Listeners2 | ||
6844 | |||
6845 | |||
6846 | |||
6847 | Each listener also requires a separate client connection. By sending the | ||
6848 | @code{GNUNET_SERVICE_SET_LISTEN} message, the client notifies the service | ||
6849 | of the application id and operation type it is interested in. A client | ||
6850 | rejects an incoming request by sending @code{GNUNET_SERVICE_SET_REJECT} | ||
6851 | on the listener's client connection. | ||
6852 | In contrast, when accepting an incoming request, a | ||
6853 | @code{GNUNET_SERVICE_SET_ACCEPT} message must be sent over the@ set that | ||
6854 | is supplied for the set operation. | ||
6855 | |||
6856 | @node Initiating Operations | ||
6857 | @subsubsection Initiating Operations | ||
6858 | |||
6859 | |||
6860 | |||
6861 | Operations with remote peers are initiated by sending a | ||
6862 | @code{GNUNET_SERVICE_SET_EVALUATE} message to the service. The@ client | ||
6863 | connection that this message is sent by determines the set to use. | ||
6864 | |||
6865 | @node Modifying Sets | ||
6866 | @subsubsection Modifying Sets | ||
6867 | |||
6868 | |||
6869 | |||
6870 | Sets are modified with the @code{GNUNET_SERVICE_SET_ADD} and | ||
6871 | @code{GNUNET_SERVICE_SET_REMOVE} messages. | ||
6872 | |||
6873 | |||
6874 | @c %@menu | ||
6875 | @c %* Results and Operation Status:: | ||
6876 | @c %* Iterating Sets:: | ||
6877 | @c %@end menu | ||
6878 | |||
6879 | @node Results and Operation Status | ||
6880 | @subsubsection Results and Operation Status | ||
6881 | |||
6882 | |||
6883 | The service notifies the client of result elements and success/failure of | ||
6884 | a set operation with the @code{GNUNET_SERVICE_SET_RESULT} message. | ||
6885 | |||
6886 | @node Iterating Sets | ||
6887 | @subsubsection Iterating Sets | ||
6888 | |||
6889 | |||
6890 | |||
6891 | All elements of a set can be requested by sending | ||
6892 | @code{GNUNET_SERVICE_SET_ITER_REQUEST}. The server responds with | ||
6893 | @code{GNUNET_SERVICE_SET_ITER_ELEMENT} and eventually terminates the | ||
6894 | iteration with @code{GNUNET_SERVICE_SET_ITER_DONE}. | ||
6895 | After each received element, the client | ||
6896 | must send @code{GNUNET_SERVICE_SET_ITER_ACK}. Note that only one set | ||
6897 | iteration may be active for a set at any given time. | ||
6898 | |||
6899 | @node The SET Intersection Peer-to-Peer Protocol | ||
6900 | @subsection The SET Intersection Peer-to-Peer Protocol | ||
6901 | |||
6902 | |||
6903 | |||
6904 | The intersection protocol operates over CADET and starts with a | ||
6905 | GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer | ||
6906 | initiating the operation to the peer listening for inbound requests. | ||
6907 | It includes the number of elements of the initiating peer, which is used | ||
6908 | to decide which side will send a Bloom filter first. | ||
6909 | |||
6910 | The listening peer checks if the operation type and application | ||
6911 | identifier are acceptable for its current state. | ||
6912 | If not, it responds with a GNUNET_MESSAGE_TYPE_SET_RESULT and a status of | ||
6913 | GNUNET_SET_STATUS_FAILURE (and terminates the CADET channel). | ||
6914 | |||
6915 | If the application accepts the request, the listener sends back a | ||
6916 | @code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} if it has | ||
6917 | more elements in the set than the client. | ||
6918 | Otherwise, it immediately starts with the Bloom filter exchange. | ||
6919 | If the initiator receives a | ||
6920 | @code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} response, | ||
6921 | it beings the Bloom filter exchange, unless the set size is indicated to | ||
6922 | be zero, in which case the intersection is considered finished after | ||
6923 | just the initial handshake. | ||
6924 | |||
6925 | |||
6926 | @menu | ||
6927 | * The Bloom filter exchange:: | ||
6928 | * Salt:: | ||
6929 | @end menu | ||
6930 | |||
6931 | @node The Bloom filter exchange | ||
6932 | @subsubsection The Bloom filter exchange | ||
6933 | |||
6934 | |||
6935 | |||
6936 | In this phase, each peer transmits a Bloom filter over the remaining | ||
6937 | keys of the local set to the other peer using a | ||
6938 | @code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_BF} message. This | ||
6939 | message additionally includes the number of elements left in the sender's | ||
6940 | set, as well as the XOR over all of the keys in that set. | ||
6941 | |||
6942 | The number of bits 'k' set per element in the Bloom filter is calculated | ||
6943 | based on the relative size of the two sets. | ||
6944 | Furthermore, the size of the Bloom filter is calculated based on 'k' and | ||
6945 | the number of elements in the set to maximize the amount of data filtered | ||
6946 | per byte transmitted on the wire (while avoiding an excessively high | ||
6947 | number of iterations). | ||
6948 | |||
6949 | The receiver of the message removes all elements from its local set that | ||
6950 | do not pass the Bloom filter test. | ||
6951 | It then checks if the set size of the sender and the XOR over the keys | ||
6952 | match what is left of its own set. If they do, it sends a | ||
6953 | @code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_DONE} back to indicate | ||
6954 | that the latest set is the final result. | ||
6955 | Otherwise, the receiver starts another Bloom filter exchange, except | ||
6956 | this time as the sender. | ||
6957 | |||
6958 | @node Salt | ||
6959 | @subsubsection Salt | ||
6960 | |||
6961 | |||
6962 | |||
6963 | Bloomfilter operations are probabilistic: With some non-zero probability | ||
6964 | the test may incorrectly say an element is in the set, even though it is | ||
6965 | not. | ||
6966 | |||
6967 | To mitigate this problem, the intersection protocol iterates exchanging | ||
6968 | Bloom filters using a different random 32-bit salt in each iteration (the | ||
6969 | salt is also included in the message). | ||
6970 | With different salts, set operations may fail for different elements. | ||
6971 | Merging the results from the executions, the probability of failure drops | ||
6972 | to zero. | ||
6973 | |||
6974 | The iterations terminate once both peers have established that they have | ||
6975 | sets of the same size, and where the XOR over all keys computes the same | ||
6976 | 512-bit value (leaving a failure probability of 2-511). | ||
6977 | |||
6978 | @node The SET Union Peer-to-Peer Protocol | ||
6979 | @subsection The SET Union Peer-to-Peer Protocol | ||
6980 | |||
6981 | |||
6982 | |||
6983 | The SET union protocol is based on Eppstein's efficient set reconciliation | ||
6984 | without prior context. You should read this paper first if you want to | ||
6985 | understand the protocol. | ||
6986 | |||
6987 | The union protocol operates over CADET and starts with a | ||
6988 | GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer | ||
6989 | initiating the operation to the peer listening for inbound requests. | ||
6990 | It includes the number of elements of the initiating peer, which is | ||
6991 | currently not used. | ||
6992 | |||
6993 | The listening peer checks if the operation type and application | ||
6994 | identifier are acceptable for its current state. If not, it responds with | ||
6995 | a @code{GNUNET_MESSAGE_TYPE_SET_RESULT} and a status of | ||
6996 | @code{GNUNET_SET_STATUS_FAILURE} (and terminates the CADET channel). | ||
6997 | |||
6998 | If the application accepts the request, it sends back a strata estimator | ||
6999 | using a message of type GNUNET_MESSAGE_TYPE_SET_UNION_P2P_SE. The | ||
7000 | initiator evaluates the strata estimator and initiates the exchange of | ||
7001 | invertible Bloom filters, sending a GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF. | ||
7002 | |||
7003 | During the IBF exchange, if the receiver cannot invert the Bloom filter or | ||
7004 | detects a cycle, it sends a larger IBF in response (up to a defined | ||
7005 | maximum limit; if that limit is reached, the operation fails). | ||
7006 | Elements decoded while processing the IBF are transmitted to the other | ||
7007 | peer using GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS, or requested from the | ||
7008 | other peer using GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS messages, | ||
7009 | depending on the sign observed during decoding of the IBF. | ||
7010 | Peers respond to a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS message | ||
7011 | with the respective element in a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS | ||
7012 | message. If the IBF fully decodes, the peer responds with a | ||
7013 | GNUNET_MESSAGE_TYPE_SET_UNION_P2P_DONE message instead of another | ||
7014 | GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF. | ||
7015 | |||
7016 | All Bloom filter operations use a salt to mingle keys before hashing them | ||
7017 | into buckets, such that future iterations have a fresh chance of | ||
7018 | succeeding if they failed due to collisions before. | ||
7019 | |||
7020 | |||
7021 | |||
7022 | |||
7023 | |||
7024 | |||
7025 | |||
7026 | |||
7027 | @cindex SETI Subsystem | ||
7028 | @node SETI Subsystem | ||
7029 | @section SETI Subsystem | ||
7030 | |||
7031 | The SET service implements efficient set intersection between two peers | ||
7032 | over a CADET tunnel. | ||
7033 | Elements of a set consist of an @emph{element type} and | ||
7034 | arbitrary binary @emph{data}. | ||
7035 | The size of an element's data is limited to around 62 KB. | ||
7036 | |||
7037 | @menu | ||
7038 | * Intersection Sets:: | ||
7039 | * Set Intersection Modifications:: | ||
7040 | * Set Intersection Operations:: | ||
7041 | * Intersection Result Elements:: | ||
7042 | * libgnunetseti:: | ||
7043 | * The SETI Client-Service Protocol:: | ||
7044 | * The SETI Intersection Peer-to-Peer Protocol:: | ||
7045 | @end menu | ||
7046 | |||
7047 | @node Intersection Sets | ||
7048 | @subsection Intersection Sets | ||
7049 | |||
7050 | Sets created by a local client can be modified (by adding additional elements) | ||
7051 | and reused for multiple operations. If elements are to be removed, a fresh | ||
7052 | set must be created by the client. | ||
7053 | |||
7054 | @node Set Intersection Modifications | ||
7055 | @subsection Set Intersection Modifications | ||
7056 | |||
7057 | Even when set operations are active, one can add elements | ||
7058 | to a set. | ||
7059 | However, these changes will only be visible to operations that have been | ||
7060 | created after the changes have taken place. That is, every set operation | ||
7061 | only sees a snapshot of the set from the time the operation was started. | ||
7062 | This mechanism is @emph{not} implemented by copying the whole set, but by | ||
7063 | attaching @emph{generation information} to each element and operation. | ||
7064 | |||
7065 | @node Set Intersection Operations | ||
7066 | @subsection Set Intersection Operations | ||
7067 | |||
7068 | Set operations can be started in two ways: Either by accepting an | ||
7069 | operation request from a remote peer, or by requesting a set operation | ||
7070 | from a remote peer. | ||
7071 | Set operations are uniquely identified by the involved @emph{peers}, an | ||
7072 | @emph{application id} and the @emph{operation type}. | ||
7073 | |||
7074 | The client is notified of incoming set operations by @emph{set listeners}. | ||
7075 | A set listener listens for incoming operations of a specific operation | ||
7076 | type and application id. | ||
7077 | Once notified of an incoming set request, the client can accept the set | ||
7078 | request (providing a local set for the operation) or reject it. | ||
7079 | |||
7080 | @node Intersection Result Elements | ||
7081 | @subsection Intersection Result Elements | ||
7082 | |||
7083 | The SET service has two @emph{result modes} that determine how an | ||
7084 | operation's result set is delivered to the client: | ||
7085 | |||
7086 | @itemize @bullet | ||
7087 | @item @strong{Return intersection.} All elements of set resulting from the set | ||
7088 | intersection are returned to the client. | ||
7089 | @item @strong{Removed Elements.} Only elements that are in the local | ||
7090 | peer's initial set but not in the intersection are returned. | ||
7091 | @end itemize | ||
7092 | |||
7093 | @cindex libgnunetseti | ||
7094 | @node libgnunetseti | ||
7095 | @subsection libgnunetseti | ||
7096 | |||
7097 | @menu | ||
7098 | * Intersection Set API:: | ||
7099 | * Intersection Listeners:: | ||
7100 | * Intersection Operations:: | ||
7101 | * Supplying a Set for Intersection:: | ||
7102 | * The Intersection Result Callback:: | ||
7103 | @end menu | ||
7104 | |||
7105 | @node Intersection Set API | ||
7106 | @subsubsection Intersection Set API | ||
7107 | |||
7108 | New sets are created with @code{GNUNET_SETI_create}. Only the local peer's | ||
7109 | configuration (as each set has its own client connection) must be provided. | ||
7110 | The set exists until either the client calls @code{GNUNET_SET_destroy} or | ||
7111 | the client's connection to the service is disrupted. | ||
7112 | In the latter case, the client is notified by the return value of | ||
7113 | functions dealing with sets. This return value must always be checked. | ||
7114 | |||
7115 | Elements are added with @code{GNUNET_SET_add_element}. | ||
7116 | |||
7117 | @node Intersection Listeners | ||
7118 | @subsubsection Intersection Listeners | ||
7119 | |||
7120 | Listeners are created with @code{GNUNET_SET_listen}. Each time time a | ||
7121 | remote peer suggests a set operation with an application id and operation | ||
7122 | type matching a listener, the listener's callback is invoked. | ||
7123 | The client then must synchronously call either @code{GNUNET_SET_accept} | ||
7124 | or @code{GNUNET_SET_reject}. Note that the operation will not be started | ||
7125 | until the client calls @code{GNUNET_SET_commit} | ||
7126 | (see Section "Supplying a Set"). | ||
7127 | |||
7128 | @node Intersection Operations | ||
7129 | @subsubsection Intersection Operations | ||
7130 | |||
7131 | Operations to be initiated by the local peer are created with | ||
7132 | @code{GNUNET_SET_prepare}. Note that the operation will not be started | ||
7133 | until the client calls @code{GNUNET_SET_commit} | ||
7134 | (see Section "Supplying a Set"). | ||
7135 | |||
7136 | @node Supplying a Set for Intersection | ||
7137 | @subsubsection Supplying a Set for Intersection | ||
7138 | |||
7139 | To create symmetry between the two ways of starting a set operation | ||
7140 | (accepting and initiating it), the operation handles returned by | ||
7141 | @code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare} do not yet have a | ||
7142 | set to operate on, thus they can not do any work yet. | ||
7143 | |||
7144 | The client must call @code{GNUNET_SET_commit} to specify a set to use for | ||
7145 | an operation. @code{GNUNET_SET_commit} may only be called once per set | ||
7146 | operation. | ||
7147 | |||
7148 | @node The Intersection Result Callback | ||
7149 | @subsubsection The Intersection Result Callback | ||
7150 | |||
7151 | Clients must specify both a result mode and a result callback with | ||
7152 | @code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result | ||
7153 | callback with a status indicating either that an element was received, or | ||
7154 | the operation failed or succeeded. | ||
7155 | The interpretation of the received element depends on the result mode. | ||
7156 | The callback needs to know which result mode it is used in, as the | ||
7157 | arguments do not indicate if an element is part of the full result set, | ||
7158 | or if it is in the difference between the original set and the final set. | ||
7159 | |||
7160 | @node The SETI Client-Service Protocol | ||
7161 | @subsection The SETI Client-Service Protocol | ||
7162 | |||
7163 | @menu | ||
7164 | * Creating Intersection Sets:: | ||
7165 | * Listeners for Intersection:: | ||
7166 | * Initiating Intersection Operations:: | ||
7167 | * Modifying Intersection Sets:: | ||
7168 | * Intersection Results and Operation Status:: | ||
7169 | @end menu | ||
7170 | |||
7171 | @node Creating Intersection Sets | ||
7172 | @subsubsection Creating Intersection Sets | ||
7173 | |||
7174 | For each set of a client, there exists a client connection to the service. | ||
7175 | Sets are created by sending the @code{GNUNET_SERVICE_SETI_CREATE} message | ||
7176 | over a new client connection. Multiple operations for one set are | ||
7177 | multiplexed over one client connection, using a request id supplied by | ||
7178 | the client. | ||
7179 | |||
7180 | @node Listeners for Intersection | ||
7181 | @subsubsection Listeners for Intersection | ||
7182 | |||
7183 | Each listener also requires a separate client connection. By sending the | ||
7184 | @code{GNUNET_SERVICE_SETI_LISTEN} message, the client notifies the service | ||
7185 | of the application id and operation type it is interested in. A client | ||
7186 | rejects an incoming request by sending @code{GNUNET_SERVICE_SETI_REJECT} | ||
7187 | on the listener's client connection. | ||
7188 | In contrast, when accepting an incoming request, a | ||
7189 | @code{GNUNET_SERVICE_SETI_ACCEPT} message must be sent over the@ set that | ||
7190 | is supplied for the set operation. | ||
7191 | |||
7192 | @node Initiating Intersection Operations | ||
7193 | @subsubsection Initiating Intersection Operations | ||
7194 | |||
7195 | Operations with remote peers are initiated by sending a | ||
7196 | @code{GNUNET_SERVICE_SETI_EVALUATE} message to the service. The@ client | ||
7197 | connection that this message is sent by determines the set to use. | ||
7198 | |||
7199 | @node Modifying Intersection Sets | ||
7200 | @subsubsection Modifying Intersection Sets | ||
7201 | |||
7202 | Sets are modified with the @code{GNUNET_SERVICE_SETI_ADD} message. | ||
7203 | |||
7204 | |||
7205 | @c %@menu | ||
7206 | @c %* Results and Operation Status:: | ||
7207 | @c %* Iterating Sets:: | ||
7208 | @c %@end menu | ||
7209 | |||
7210 | @node Intersection Results and Operation Status | ||
7211 | @subsubsection Intersection Results and Operation Status | ||
7212 | |||
7213 | The service notifies the client of result elements and success/failure of | ||
7214 | a set operation with the @code{GNUNET_SERVICE_SETI_RESULT} message. | ||
7215 | |||
7216 | @node The SETI Intersection Peer-to-Peer Protocol | ||
7217 | @subsection The SETI Intersection Peer-to-Peer Protocol | ||
7218 | |||
7219 | The intersection protocol operates over CADET and starts with a | ||
7220 | GNUNET_MESSAGE_TYPE_SETI_P2P_OPERATION_REQUEST being sent by the peer | ||
7221 | initiating the operation to the peer listening for inbound requests. | ||
7222 | It includes the number of elements of the initiating peer, which is used | ||
7223 | to decide which side will send a Bloom filter first. | ||
7224 | |||
7225 | The listening peer checks if the operation type and application | ||
7226 | identifier are acceptable for its current state. | ||
7227 | If not, it responds with a GNUNET_MESSAGE_TYPE_SETI_RESULT and a status of | ||
7228 | GNUNET_SETI_STATUS_FAILURE (and terminates the CADET channel). | ||
7229 | |||
7230 | If the application accepts the request, the listener sends back a | ||
7231 | @code{GNUNET_MESSAGE_TYPE_SETI_P2P_ELEMENT_INFO} if it has | ||
7232 | more elements in the set than the client. | ||
7233 | Otherwise, it immediately starts with the Bloom filter exchange. | ||
7234 | If the initiator receives a | ||
7235 | @code{GNUNET_MESSAGE_TYPE_SETI_P2P_ELEMENT_INFO} response, | ||
7236 | it beings the Bloom filter exchange, unless the set size is indicated to | ||
7237 | be zero, in which case the intersection is considered finished after | ||
7238 | just the initial handshake. | ||
7239 | |||
7240 | |||
7241 | @menu | ||
7242 | * The Bloom filter exchange in SETI:: | ||
7243 | * Intersection Salt:: | ||
7244 | @end menu | ||
7245 | |||
7246 | @node The Bloom filter exchange in SETI | ||
7247 | @subsubsection The Bloom filter exchange in SETI | ||
7248 | |||
7249 | In this phase, each peer transmits a Bloom filter over the remaining | ||
7250 | keys of the local set to the other peer using a | ||
7251 | @code{GNUNET_MESSAGE_TYPE_SETI_P2P_BF} message. This | ||
7252 | message additionally includes the number of elements left in the sender's | ||
7253 | set, as well as the XOR over all of the keys in that set. | ||
7254 | |||
7255 | The number of bits 'k' set per element in the Bloom filter is calculated | ||
7256 | based on the relative size of the two sets. | ||
7257 | Furthermore, the size of the Bloom filter is calculated based on 'k' and | ||
7258 | the number of elements in the set to maximize the amount of data filtered | ||
7259 | per byte transmitted on the wire (while avoiding an excessively high | ||
7260 | number of iterations). | ||
7261 | |||
7262 | The receiver of the message removes all elements from its local set that | ||
7263 | do not pass the Bloom filter test. | ||
7264 | It then checks if the set size of the sender and the XOR over the keys | ||
7265 | match what is left of its own set. If they do, it sends a | ||
7266 | @code{GNUNET_MESSAGE_TYPE_SETI_P2P_DONE} back to indicate | ||
7267 | that the latest set is the final result. | ||
7268 | Otherwise, the receiver starts another Bloom filter exchange, except | ||
7269 | this time as the sender. | ||
7270 | |||
7271 | @node Intersection Salt | ||
7272 | @subsubsection Intersection Salt | ||
7273 | |||
7274 | Bloom filter operations are probabilistic: With some non-zero probability | ||
7275 | the test may incorrectly say an element is in the set, even though it is | ||
7276 | not. | ||
7277 | |||
7278 | To mitigate this problem, the intersection protocol iterates exchanging | ||
7279 | Bloom filters using a different random 32-bit salt in each iteration (the | ||
7280 | salt is also included in the message). | ||
7281 | With different salts, set operations may fail for different elements. | ||
7282 | Merging the results from the executions, the probability of failure drops | ||
7283 | to zero. | ||
7284 | |||
7285 | The iterations terminate once both peers have established that they have | ||
7286 | sets of the same size, and where the XOR over all keys computes the same | ||
7287 | 512-bit value (leaving a failure probability of 2-511). | ||
7288 | |||
7289 | |||
7290 | @cindex SETU Subsystem | ||
7291 | @node SETU Subsystem | ||
7292 | @section SETU Subsystem | ||
7293 | |||
7294 | The SETU service implements efficient set union operations between two peers | ||
7295 | over a CADET tunnel. Elements of a set consist of an @emph{element type} and | ||
7296 | arbitrary binary @emph{data}. The size of an element's data is limited to | ||
7297 | around 62 KB. | ||
7298 | |||
7299 | @menu | ||
7300 | * Union Sets:: | ||
7301 | * Set Union Modifications:: | ||
7302 | * Set Union Operations:: | ||
7303 | * Union Result Elements:: | ||
7304 | * libgnunetsetu:: | ||
7305 | * The SETU Client-Service Protocol:: | ||
7306 | * The SETU Union Peer-to-Peer Protocol:: | ||
7307 | @end menu | ||
7308 | |||
7309 | @node Union Sets | ||
7310 | @subsection Union Sets | ||
7311 | |||
7312 | Sets created by a local client can be modified (by adding additional elements) | ||
7313 | and reused for multiple operations. If elements are to be removed, a fresh | ||
7314 | set must be created by the client. | ||
7315 | |||
7316 | @node Set Union Modifications | ||
7317 | @subsection Set Union Modifications | ||
7318 | |||
7319 | Even when set operations are active, one can add elements | ||
7320 | to a set. | ||
7321 | However, these changes will only be visible to operations that have been | ||
7322 | created after the changes have taken place. That is, every set operation | ||
7323 | only sees a snapshot of the set from the time the operation was started. | ||
7324 | This mechanism is @emph{not} implemented by copying the whole set, but by | ||
7325 | attaching @emph{generation information} to each element and operation. | ||
7326 | |||
7327 | @node Set Union Operations | ||
7328 | @subsection Set Union Operations | ||
7329 | |||
7330 | Set operations can be started in two ways: Either by accepting an | ||
7331 | operation request from a remote peer, or by requesting a set operation | ||
7332 | from a remote peer. | ||
7333 | Set operations are uniquely identified by the involved @emph{peers}, an | ||
7334 | @emph{application id} and the @emph{operation type}. | ||
7335 | |||
7336 | The client is notified of incoming set operations by @emph{set listeners}. | ||
7337 | A set listener listens for incoming operations of a specific operation | ||
7338 | type and application id. | ||
7339 | Once notified of an incoming set request, the client can accept the set | ||
7340 | request (providing a local set for the operation) or reject it. | ||
7341 | |||
7342 | @node Union Result Elements | ||
7343 | @subsection Union Result Elements | ||
7344 | |||
7345 | The SET service has three @emph{result modes} that determine how an | ||
7346 | operation's result set is delivered to the client: | ||
7347 | |||
7348 | @itemize @bullet | ||
7349 | @item @strong{Locally added Elements.} Elements that are in the union | ||
7350 | but not already in the local peer's set are returned. | ||
7351 | @item @strong{Remote added Elements.} Additionally, notify the client | ||
7352 | if the remote peer lacked some elements and thus also return to the | ||
7353 | local client those elements that we are sending to the remote peer to | ||
7354 | be added to its union. Obtaining these elements requires setting | ||
7355 | the @code{GNUNET_SETU_OPTION_SYMMETRIC} option. | ||
7356 | @end itemize | ||
7357 | |||
7358 | @cindex libgnunetsetu | ||
7359 | @node libgnunetsetu | ||
7360 | @subsection libgnunetsetu | ||
7361 | |||
7362 | @menu | ||
7363 | * Union Set API:: | ||
7364 | * Union Listeners:: | ||
7365 | * Union Operations:: | ||
7366 | * Supplying a Set for Union:: | ||
7367 | * The Union Result Callback:: | ||
7368 | @end menu | ||
7369 | |||
7370 | @node Union Set API | ||
7371 | @subsubsection Union Set API | ||
7372 | |||
7373 | New sets are created with @code{GNUNET_SETU_create}. Only the local peer's | ||
7374 | configuration (as each set has its own client connection) must be provided. | ||
7375 | The set exists until either the client calls @code{GNUNET_SETU_destroy} or | ||
7376 | the client's connection to the service is disrupted. | ||
7377 | In the latter case, the client is notified by the return value of | ||
7378 | functions dealing with sets. This return value must always be checked. | ||
7379 | |||
7380 | Elements are added with @code{GNUNET_SETU_add_element}. | ||
7381 | |||
7382 | @node Union Listeners | ||
7383 | @subsubsection Union Listeners | ||
7384 | |||
7385 | Listeners are created with @code{GNUNET_SETU_listen}. Each time time a | ||
7386 | remote peer suggests a set operation with an application id and operation | ||
7387 | type matching a listener, the listener's callback is invoked. | ||
7388 | The client then must synchronously call either @code{GNUNET_SETU_accept} | ||
7389 | or @code{GNUNET_SETU_reject}. Note that the operation will not be started | ||
7390 | until the client calls @code{GNUNET_SETU_commit} | ||
7391 | (see Section "Supplying a Set"). | ||
7392 | |||
7393 | @node Union Operations | ||
7394 | @subsubsection Union Operations | ||
7395 | |||
7396 | Operations to be initiated by the local peer are created with | ||
7397 | @code{GNUNET_SETU_prepare}. Note that the operation will not be started | ||
7398 | until the client calls @code{GNUNET_SETU_commit} | ||
7399 | (see Section "Supplying a Set"). | ||
7400 | |||
7401 | @node Supplying a Set for Union | ||
7402 | @subsubsection Supplying a Set for Union | ||
7403 | |||
7404 | To create symmetry between the two ways of starting a set operation | ||
7405 | (accepting and initiating it), the operation handles returned by | ||
7406 | @code{GNUNET_SETU_accept} and @code{GNUNET_SETU_prepare} do not yet have a | ||
7407 | set to operate on, thus they can not do any work yet. | ||
7408 | |||
7409 | The client must call @code{GNUNET_SETU_commit} to specify a set to use for | ||
7410 | an operation. @code{GNUNET_SETU_commit} may only be called once per set | ||
7411 | operation. | ||
7412 | |||
7413 | @node The Union Result Callback | ||
7414 | @subsubsection The Union Result Callback | ||
7415 | |||
7416 | Clients must specify both a result mode and a result callback with | ||
7417 | @code{GNUNET_SETU_accept} and @code{GNUNET_SETU_prepare}. The result | ||
7418 | callback with a status indicating either that an element was received, | ||
7419 | transmitted to the other peer (if this information was requested), or | ||
7420 | if the operation failed or ultimately succeeded. | ||
7421 | |||
7422 | @node The SETU Client-Service Protocol | ||
7423 | @subsection The SETU Client-Service Protocol | ||
7424 | |||
7425 | @menu | ||
7426 | * Creating Union Sets:: | ||
7427 | * Listeners for Union:: | ||
7428 | * Initiating Union Operations:: | ||
7429 | * Modifying Union Sets:: | ||
7430 | * Union Results and Operation Status:: | ||
7431 | @end menu | ||
7432 | |||
7433 | @node Creating Union Sets | ||
7434 | @subsubsection Creating Union Sets | ||
7435 | |||
7436 | For each set of a client, there exists a client connection to the service. | ||
7437 | Sets are created by sending the @code{GNUNET_SERVICE_SETU_CREATE} message | ||
7438 | over a new client connection. Multiple operations for one set are | ||
7439 | multiplexed over one client connection, using a request id supplied by | ||
7440 | the client. | ||
7441 | |||
7442 | @node Listeners for Union | ||
7443 | @subsubsection Listeners for Union | ||
7444 | |||
7445 | Each listener also requires a separate client connection. By sending the | ||
7446 | @code{GNUNET_SERVICE_SETU_LISTEN} message, the client notifies the service | ||
7447 | of the application id and operation type it is interested in. A client | ||
7448 | rejects an incoming request by sending @code{GNUNET_SERVICE_SETU_REJECT} | ||
7449 | on the listener's client connection. | ||
7450 | In contrast, when accepting an incoming request, a | ||
7451 | @code{GNUNET_SERVICE_SETU_ACCEPT} message must be sent over the@ set that | ||
7452 | is supplied for the set operation. | ||
7453 | |||
7454 | @node Initiating Union Operations | ||
7455 | @subsubsection Initiating Union Operations | ||
7456 | |||
7457 | |||
7458 | |||
7459 | Operations with remote peers are initiated by sending a | ||
7460 | @code{GNUNET_SERVICE_SETU_EVALUATE} message to the service. The@ client | ||
7461 | connection that this message is sent by determines the set to use. | ||
7462 | |||
7463 | @node Modifying Union Sets | ||
7464 | @subsubsection Modifying Union Sets | ||
7465 | |||
7466 | Sets are modified with the @code{GNUNET_SERVICE_SETU_ADD} message. | ||
7467 | |||
7468 | |||
7469 | @c %@menu | ||
7470 | @c %* Results and Operation Status:: | ||
7471 | @c %* Iterating Sets:: | ||
7472 | @c %@end menu | ||
7473 | |||
7474 | @node Union Results and Operation Status | ||
7475 | @subsubsection Union Results and Operation Status | ||
7476 | |||
7477 | The service notifies the client of result elements and success/failure of | ||
7478 | a set operation with the @code{GNUNET_SERVICE_SETU_RESULT} message. | ||
7479 | |||
7480 | |||
7481 | @node The SETU Union Peer-to-Peer Protocol | ||
7482 | @subsection The SETU Union Peer-to-Peer Protocol | ||
7483 | |||
7484 | |||
7485 | The SET union protocol is based on Eppstein's efficient set reconciliation | ||
7486 | without prior context. You should read this paper first if you want to | ||
7487 | understand the protocol. | ||
7488 | |||
7489 | The union protocol operates over CADET and starts with a | ||
7490 | GNUNET_MESSAGE_TYPE_SETU_P2P_OPERATION_REQUEST being sent by the peer | ||
7491 | initiating the operation to the peer listening for inbound requests. | ||
7492 | It includes the number of elements of the initiating peer, which is | ||
7493 | currently not used. | ||
7494 | |||
7495 | The listening peer checks if the operation type and application | ||
7496 | identifier are acceptable for its current state. If not, it responds with | ||
7497 | a @code{GNUNET_MESSAGE_TYPE_SETU_RESULT} and a status of | ||
7498 | @code{GNUNET_SETU_STATUS_FAILURE} (and terminates the CADET channel). | ||
7499 | |||
7500 | If the application accepts the request, it sends back a strata estimator | ||
7501 | using a message of type GNUNET_MESSAGE_TYPE_SETU_P2P_SE. The | ||
7502 | initiator evaluates the strata estimator and initiates the exchange of | ||
7503 | invertible Bloom filters, sending a GNUNET_MESSAGE_TYPE_SETU_P2P_IBF. | ||
7504 | |||
7505 | During the IBF exchange, if the receiver cannot invert the Bloom filter or | ||
7506 | detects a cycle, it sends a larger IBF in response (up to a defined | ||
7507 | maximum limit; if that limit is reached, the operation fails). | ||
7508 | Elements decoded while processing the IBF are transmitted to the other | ||
7509 | peer using GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENTS, or requested from the | ||
7510 | other peer using GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENT_REQUESTS messages, | ||
7511 | depending on the sign observed during decoding of the IBF. | ||
7512 | Peers respond to a GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENT_REQUESTS message | ||
7513 | with the respective element in a GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENTS | ||
7514 | message. If the IBF fully decodes, the peer responds with a | ||
7515 | GNUNET_MESSAGE_TYPE_SETU_P2P_DONE message instead of another | ||
7516 | GNUNET_MESSAGE_TYPE_SETU_P2P_IBF. | ||
7517 | |||
7518 | All Bloom filter operations use a salt to mingle keys before hashing them | ||
7519 | into buckets, such that future iterations have a fresh chance of | ||
7520 | succeeding if they failed due to collisions before. | ||
7521 | |||
7522 | |||
7523 | |||
7524 | |||
7525 | |||
7526 | |||
7527 | @cindex STATISTICS Subsystem | ||
7528 | @node STATISTICS Subsystem | ||
7529 | @section STATISTICS Subsystem | ||
7530 | |||
7531 | |||
7532 | |||
7533 | In GNUnet, the STATISTICS subsystem offers a central place for all | ||
7534 | subsystems to publish unsigned 64-bit integer run-time statistics. | ||
7535 | Keeping this information centrally means that there is a unified way for | ||
7536 | the user to obtain data on all subsystems, and individual subsystems do | ||
7537 | not have to always include a custom data export method for performance | ||
7538 | metrics and other statistics. For example, the TRANSPORT system uses | ||
7539 | STATISTICS to update information about the number of directly connected | ||
7540 | peers and the bandwidth that has been consumed by the various plugins. | ||
7541 | This information is valuable for diagnosing connectivity and performance | ||
7542 | issues. | ||
7543 | |||
7544 | Following the GNUnet service architecture, the STATISTICS subsystem is | ||
7545 | divided into an API which is exposed through the header | ||
7546 | @strong{gnunet_statistics_service.h} and the STATISTICS service | ||
7547 | @strong{gnunet-service-statistics}. The @strong{gnunet-statistics} | ||
7548 | command-line tool can be used to obtain (and change) information about | ||
7549 | the values stored by the STATISTICS service. The STATISTICS service does | ||
7550 | not communicate with other peers. | ||
7551 | |||
7552 | Data is stored in the STATISTICS service in the form of tuples | ||
7553 | @strong{(subsystem, name, value, persistence)}. The subsystem determines | ||
7554 | to which other GNUnet's subsystem the data belongs. name is the name | ||
7555 | through which value is associated. It uniquely identifies the record | ||
7556 | from among other records belonging to the same subsystem. | ||
7557 | In some parts of the code, the pair @strong{(subsystem, name)} is called | ||
7558 | a @strong{statistic} as it identifies the values stored in the STATISTCS | ||
7559 | service.The persistence flag determines if the record has to be preserved | ||
7560 | across service restarts. A record is said to be persistent if this flag | ||
7561 | is set for it; if not, the record is treated as a non-persistent record | ||
7562 | and it is lost after service restart. Persistent records are written to | ||
7563 | and read from the file @strong{statistics.data} before shutdown | ||
7564 | and upon startup. The file is located in the HOME directory of the peer. | ||
7565 | |||
7566 | An anomaly of the STATISTICS service is that it does not terminate | ||
7567 | immediately upon receiving a shutdown signal if it has any clients | ||
7568 | connected to it. It waits for all the clients that are not monitors to | ||
7569 | close their connections before terminating itself. | ||
7570 | This is to prevent the loss of data during peer shutdown --- delaying the | ||
7571 | STATISTICS service shutdown helps other services to store important data | ||
7572 | to STATISTICS during shutdown. | ||
7573 | |||
7574 | @menu | ||
7575 | * libgnunetstatistics:: | ||
7576 | * The STATISTICS Client-Service Protocol:: | ||
7577 | @end menu | ||
7578 | |||
7579 | @cindex libgnunetstatistics | ||
7580 | @node libgnunetstatistics | ||
7581 | @subsection libgnunetstatistics | ||
7582 | |||
7583 | |||
7584 | |||
7585 | @strong{libgnunetstatistics} is the library containing the API for the | ||
7586 | STATISTICS subsystem. Any process requiring to use STATISTICS should use | ||
7587 | this API by to open a connection to the STATISTICS service. | ||
7588 | This is done by calling the function @code{GNUNET_STATISTICS_create()}. | ||
7589 | This function takes the subsystem's name which is trying to use STATISTICS | ||
7590 | and a configuration. | ||
7591 | All values written to STATISTICS with this connection will be placed in | ||
7592 | the section corresponding to the given subsystem's name. | ||
7593 | The connection to STATISTICS can be destroyed with the function | ||
7594 | @code{GNUNET_STATISTICS_destroy()}. This function allows for the | ||
7595 | connection to be destroyed immediately or upon transferring all | ||
7596 | pending write requests to the service. | ||
7597 | |||
7598 | Note: STATISTICS subsystem can be disabled by setting @code{DISABLE = YES} | ||
7599 | under the @code{[STATISTICS]} section in the configuration. With such a | ||
7600 | configuration all calls to @code{GNUNET_STATISTICS_create()} return | ||
7601 | @code{NULL} as the STATISTICS subsystem is unavailable and no other | ||
7602 | functions from the API can be used. | ||
7603 | |||
7604 | |||
7605 | @menu | ||
7606 | * Statistics retrieval:: | ||
7607 | * Setting statistics and updating them:: | ||
7608 | * Watches:: | ||
7609 | @end menu | ||
7610 | |||
7611 | @node Statistics retrieval | ||
7612 | @subsubsection Statistics retrieval | ||
7613 | |||
7614 | |||
7615 | |||
7616 | Once a connection to the statistics service is obtained, information | ||
7617 | about any other system which uses statistics can be retrieved with the | ||
7618 | function GNUNET_STATISTICS_get(). | ||
7619 | This function takes the connection handle, the name of the subsystem | ||
7620 | whose information we are interested in (a @code{NULL} value will | ||
7621 | retrieve information of all available subsystems using STATISTICS), the | ||
7622 | name of the statistic we are interested in (a @code{NULL} value will | ||
7623 | retrieve all available statistics), a continuation callback which is | ||
7624 | called when all of requested information is retrieved, an iterator | ||
7625 | callback which is called for each parameter in the retrieved information | ||
7626 | and a closure for the aforementioned callbacks. The library then invokes | ||
7627 | the iterator callback for each value matching the request. | ||
7628 | |||
7629 | Call to @code{GNUNET_STATISTICS_get()} is asynchronous and can be | ||
7630 | canceled with the function @code{GNUNET_STATISTICS_get_cancel()}. | ||
7631 | This is helpful when retrieving statistics takes too long and especially | ||
7632 | when we want to shutdown and cleanup everything. | ||
7633 | |||
7634 | @node Setting statistics and updating them | ||
7635 | @subsubsection Setting statistics and updating them | ||
7636 | |||
7637 | |||
7638 | |||
7639 | So far we have seen how to retrieve statistics, here we will learn how we | ||
7640 | can set statistics and update them so that other subsystems can retrieve | ||
7641 | them. | ||
7642 | |||
7643 | A new statistic can be set using the function | ||
7644 | @code{GNUNET_STATISTICS_set()}. | ||
7645 | This function takes the name of the statistic and its value and a flag to | ||
7646 | make the statistic persistent. | ||
7647 | The value of the statistic should be of the type @code{uint64_t}. | ||
7648 | The function does not take the name of the subsystem; it is determined | ||
7649 | from the previous @code{GNUNET_STATISTICS_create()} invocation. If | ||
7650 | the given statistic is already present, its value is overwritten. | ||
7651 | |||
7652 | An existing statistics can be updated, i.e its value can be increased or | ||
7653 | decreased by an amount with the function | ||
7654 | @code{GNUNET_STATISTICS_update()}. | ||
7655 | The parameters to this function are similar to | ||
7656 | @code{GNUNET_STATISTICS_set()}, except that it takes the amount to be | ||
7657 | changed as a type @code{int64_t} instead of the value. | ||
7658 | |||
7659 | The library will combine multiple set or update operations into one | ||
7660 | message if the client performs requests at a rate that is faster than the | ||
7661 | available IPC with the STATISTICS service. Thus, the client does not have | ||
7662 | to worry about sending requests too quickly. | ||
7663 | |||
7664 | @node Watches | ||
7665 | @subsubsection Watches | ||
7666 | |||
7667 | |||
7668 | |||
7669 | As interesting feature of STATISTICS lies in serving notifications | ||
7670 | whenever a statistic of our interest is modified. | ||
7671 | This is achieved by registering a watch through the function | ||
7672 | @code{GNUNET_STATISTICS_watch()}. | ||
7673 | The parameters of this function are similar to those of | ||
7674 | @code{GNUNET_STATISTICS_get()}. | ||
7675 | Changes to the respective statistic's value will then cause the given | ||
7676 | iterator callback to be called. | ||
7677 | Note: A watch can only be registered for a specific statistic. Hence | ||
7678 | the subsystem name and the parameter name cannot be @code{NULL} in a | ||
7679 | call to @code{GNUNET_STATISTICS_watch()}. | ||
7680 | |||
7681 | A registered watch will keep notifying any value changes until | ||
7682 | @code{GNUNET_STATISTICS_watch_cancel()} is called with the same | ||
7683 | parameters that are used for registering the watch. | ||
7684 | |||
7685 | @node The STATISTICS Client-Service Protocol | ||
7686 | @subsection The STATISTICS Client-Service Protocol | ||
7687 | |||
7688 | |||
7689 | |||
7690 | @menu | ||
7691 | * Statistics retrieval2:: | ||
7692 | * Setting and updating statistics:: | ||
7693 | * Watching for updates:: | ||
7694 | @end menu | ||
7695 | |||
7696 | @node Statistics retrieval2 | ||
7697 | @subsubsection Statistics retrieval2 | ||
7698 | |||
7699 | |||
7700 | |||
7701 | To retrieve statistics, the client transmits a message of type | ||
7702 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_GET} containing the given subsystem | ||
7703 | name and statistic parameter to the STATISTICS service. | ||
7704 | The service responds with a message of type | ||
7705 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_VALUE} for each of the statistics | ||
7706 | parameters that match the client request for the client. The end of | ||
7707 | information retrieved is signaled by the service by sending a message of | ||
7708 | type @code{GNUNET_MESSAGE_TYPE_STATISTICS_END}. | ||
7709 | |||
7710 | @node Setting and updating statistics | ||
7711 | @subsubsection Setting and updating statistics | ||
7712 | |||
7713 | |||
7714 | |||
7715 | The subsystem name, parameter name, its value and the persistence flag are | ||
7716 | communicated to the service through the message | ||
7717 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}. | ||
7718 | |||
7719 | When the service receives a message of type | ||
7720 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}, it retrieves the subsystem | ||
7721 | name and checks for a statistic parameter with matching the name given in | ||
7722 | the message. | ||
7723 | If a statistic parameter is found, the value is overwritten by the new | ||
7724 | value from the message; if not found then a new statistic parameter is | ||
7725 | created with the given name and value. | ||
7726 | |||
7727 | In addition to just setting an absolute value, it is possible to perform a | ||
7728 | relative update by sending a message of type | ||
7729 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_SET} with an update flag | ||
7730 | (@code{GNUNET_STATISTICS_SETFLAG_RELATIVE}) signifying that the value in | ||
7731 | the message should be treated as an update value. | ||
7732 | |||
7733 | @node Watching for updates | ||
7734 | @subsubsection Watching for updates | ||
7735 | |||
7736 | |||
7737 | |||
7738 | The function registers the watch at the service by sending a message of | ||
7739 | type @code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH}. The service then sends | ||
7740 | notifications through messages of type | ||
7741 | @code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH_VALUE} whenever the statistic | ||
7742 | parameter's value is changed. | ||
7743 | |||
7744 | @cindex DHT | ||
7745 | @cindex Distributed Hash Table | ||
7746 | @node Distributed Hash Table (DHT) | ||
7747 | @section Distributed Hash Table (DHT) | ||
7748 | |||
7749 | |||
7750 | |||
7751 | GNUnet includes a generic distributed hash table that can be used by | ||
7752 | developers building P2P applications in the framework. | ||
7753 | This section documents high-level features and how developers are | ||
7754 | expected to use the DHT. | ||
7755 | We have a research paper detailing how the DHT works. | ||
7756 | Also, Nate's thesis includes a detailed description and performance | ||
7757 | analysis (in chapter 6). | ||
7758 | |||
7759 | Key features of GNUnet's DHT include: | ||
7760 | |||
7761 | @itemize @bullet | ||
7762 | @item stores key-value pairs with values up to (approximately) 63k in size | ||
7763 | @item works with many underlay network topologies (small-world, random | ||
7764 | graph), underlay does not need to be a full mesh / clique | ||
7765 | @item support for extended queries (more than just a simple 'key'), | ||
7766 | filtering duplicate replies within the network (bloomfilter) and content | ||
7767 | validation (for details, please read the subsection on the block library) | ||
7768 | @item can (optionally) return paths taken by the PUT and GET operations | ||
7769 | to the application | ||
7770 | @item provides content replication to handle churn | ||
7771 | @end itemize | ||
7772 | |||
7773 | GNUnet's DHT is randomized and unreliable. Unreliable means that there is | ||
7774 | no strict guarantee that a value stored in the DHT is always | ||
7775 | found --- values are only found with high probability. | ||
7776 | While this is somewhat true in all P2P DHTs, GNUnet developers should be | ||
7777 | particularly wary of this fact (this will help you write secure, | ||
7778 | fault-tolerant code). Thus, when writing any application using the DHT, | ||
7779 | you should always consider the possibility that a value stored in the | ||
7780 | DHT by you or some other peer might simply not be returned, or returned | ||
7781 | with a significant delay. | ||
7782 | Your application logic must be written to tolerate this (naturally, some | ||
7783 | loss of performance or quality of service is expected in this case). | ||
7784 | |||
7785 | @menu | ||
7786 | * Block library and plugins:: | ||
7787 | * libgnunetdht:: | ||
7788 | * The DHT Client-Service Protocol:: | ||
7789 | * The DHT Peer-to-Peer Protocol:: | ||
7790 | @end menu | ||
7791 | |||
7792 | @node Block library and plugins | ||
7793 | @subsection Block library and plugins | ||
7794 | |||
7795 | |||
7796 | |||
7797 | @menu | ||
7798 | * What is a Block?:: | ||
7799 | * The API of libgnunetblock:: | ||
7800 | * Queries:: | ||
7801 | * Sample Code:: | ||
7802 | * Conclusion2:: | ||
7803 | @end menu | ||
7804 | |||
7805 | @node What is a Block? | ||
7806 | @subsubsection What is a Block? | ||
7807 | |||
7808 | |||
7809 | |||
7810 | Blocks are small (< 63k) pieces of data stored under a key (struct | ||
7811 | GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which defines | ||
7812 | their data format. Blocks are used in GNUnet as units of static data | ||
7813 | exchanged between peers and stored (or cached) locally. | ||
7814 | Uses of blocks include file-sharing (the files are broken up into blocks), | ||
7815 | the VPN (DNS information is stored in blocks) and the DHT (all | ||
7816 | information in the DHT and meta-information for the maintenance of the | ||
7817 | DHT are both stored using blocks). | ||
7818 | The block subsystem provides a few common functions that must be | ||
7819 | available for any type of block. | ||
7820 | |||
7821 | @cindex libgnunetblock API | ||
7822 | @node The API of libgnunetblock | ||
7823 | @subsubsection The API of libgnunetblock | ||
7824 | |||
7825 | |||
7826 | |||
7827 | The block library requires for each (family of) block type(s) a block | ||
7828 | plugin (implementing @file{gnunet_block_plugin.h}) that provides basic | ||
7829 | functions that are needed by the DHT (and possibly other subsystems) to | ||
7830 | manage the block. | ||
7831 | These block plugins are typically implemented within their respective | ||
7832 | subsystems. | ||
7833 | The main block library is then used to locate, load and query the | ||
7834 | appropriate block plugin. | ||
7835 | Which plugin is appropriate is determined by the block type (which is | ||
7836 | just a 32-bit integer). Block plugins contain code that specifies which | ||
7837 | block types are supported by a given plugin. The block library loads all | ||
7838 | block plugins that are installed at the local peer and forwards the | ||
7839 | application request to the respective plugin. | ||
7840 | |||
7841 | The central functions of the block APIs (plugin and main library) are to | ||
7842 | allow the mapping of blocks to their respective key (if possible) and the | ||
7843 | ability to check that a block is well-formed and matches a given | ||
7844 | request (again, if possible). | ||
7845 | This way, GNUnet can avoid storing invalid blocks, storing blocks under | ||
7846 | the wrong key and forwarding blocks in response to a query that they do | ||
7847 | not answer. | ||
7848 | |||
7849 | One key function of block plugins is that it allows GNUnet to detect | ||
7850 | duplicate replies (via the Bloom filter). All plugins MUST support | ||
7851 | detecting duplicate replies (by adding the current response to the | ||
7852 | Bloom filter and rejecting it if it is encountered again). | ||
7853 | If a plugin fails to do this, responses may loop in the network. | ||
7854 | |||
7855 | @node Queries | ||
7856 | @subsubsection Queries | ||
7857 | |||
7858 | |||
7859 | The query format for any block in GNUnet consists of four main components. | ||
7860 | First, the type of the desired block must be specified. Second, the query | ||
7861 | must contain a hash code. The hash code is used for lookups in hash | ||
7862 | tables and databases and must not be unique for the block (however, if | ||
7863 | possible a unique hash should be used as this would be best for | ||
7864 | performance). | ||
7865 | Third, an optional Bloom filter can be specified to exclude known results; | ||
7866 | replies that hash to the bits set in the Bloom filter are considered | ||
7867 | invalid. False-positives can be eliminated by sending the same query | ||
7868 | again with a different Bloom filter mutator value, which parametrizes | ||
7869 | the hash function that is used. | ||
7870 | Finally, an optional application-specific "eXtended query" (xquery) can | ||
7871 | be specified to further constrain the results. It is entirely up to | ||
7872 | the type-specific plugin to determine whether or not a given block | ||
7873 | matches a query (type, hash, Bloom filter, and xquery). | ||
7874 | Naturally, not all xquery's are valid and some types of blocks may not | ||
7875 | support Bloom filters either, so the plugin also needs to check if the | ||
7876 | query is valid in the first place. | ||
7877 | |||
7878 | Depending on the results from the plugin, the DHT will then discard the | ||
7879 | (invalid) query, forward the query, discard the (invalid) reply, cache the | ||
7880 | (valid) reply, and/or forward the (valid and non-duplicate) reply. | ||
7881 | |||
7882 | @node Sample Code | ||
7883 | @subsubsection Sample Code | ||
7884 | |||
7885 | |||
7886 | |||
7887 | The source code in @strong{plugin_block_test.c} is a good starting point | ||
7888 | for new block plugins --- it does the minimal work by implementing a | ||
7889 | plugin that performs no validation at all. | ||
7890 | The respective @strong{Makefile.am} shows how to build and install a | ||
7891 | block plugin. | ||
7892 | |||
7893 | @node Conclusion2 | ||
7894 | @subsubsection Conclusion2 | ||
7895 | |||
7896 | |||
7897 | |||
7898 | In conclusion, GNUnet subsystems that want to use the DHT need to define a | ||
7899 | block format and write a plugin to match queries and replies. For testing, | ||
7900 | the @code{GNUNET_BLOCK_TYPE_TEST} block type can be used; it accepts | ||
7901 | any query as valid and any reply as matching any query. | ||
7902 | This type is also used for the DHT command line tools. | ||
7903 | However, it should NOT be used for normal applications due to the lack | ||
7904 | of error checking that results from this primitive implementation. | ||
7905 | |||
7906 | @cindex libgnunetdht | ||
7907 | @node libgnunetdht | ||
7908 | @subsection libgnunetdht | ||
7909 | |||
7910 | |||
7911 | |||
7912 | The DHT API itself is pretty simple and offers the usual GET and PUT | ||
7913 | functions that work as expected. The specified block type refers to the | ||
7914 | block library which allows the DHT to run application-specific logic for | ||
7915 | data stored in the network. | ||
7916 | |||
7917 | |||
7918 | @menu | ||
7919 | * GET:: | ||
7920 | * PUT:: | ||
7921 | * MONITOR:: | ||
7922 | * DHT Routing Options:: | ||
7923 | @end menu | ||
7924 | |||
7925 | @node GET | ||
7926 | @subsubsection GET | ||
7927 | |||
7928 | |||
7929 | |||
7930 | When using GET, the main consideration for developers (other than the | ||
7931 | block library) should be that after issuing a GET, the DHT will | ||
7932 | continuously cause (small amounts of) network traffic until the operation | ||
7933 | is explicitly canceled. | ||
7934 | So GET does not simply send out a single network request once; instead, | ||
7935 | the DHT will continue to search for data. This is needed to achieve good | ||
7936 | success rates and also handles the case where the respective PUT | ||
7937 | operation happens after the GET operation was started. | ||
7938 | Developers should not cancel an existing GET operation and then | ||
7939 | explicitly re-start it to trigger a new round of network requests; | ||
7940 | this is simply inefficient, especially as the internal automated version | ||
7941 | can be more efficient, for example by filtering results in the network | ||
7942 | that have already been returned. | ||
7943 | |||
7944 | If an application that performs a GET request has a set of replies that it | ||
7945 | already knows and would like to filter, it can call@ | ||
7946 | @code{GNUNET_DHT_get_filter_known_results} with an array of hashes over | ||
7947 | the respective blocks to tell the DHT that these results are not | ||
7948 | desired (any more). | ||
7949 | This way, the DHT will filter the respective blocks using the block | ||
7950 | library in the network, which may result in a significant reduction in | ||
7951 | bandwidth consumption. | ||
7952 | |||
7953 | @node PUT | ||
7954 | @subsubsection PUT | ||
7955 | |||
7956 | |||
7957 | |||
7958 | @c inconsistent use of ``must'' above it's written ``MUST'' | ||
7959 | In contrast to GET operations, developers @strong{must} manually re-run | ||
7960 | PUT operations periodically (if they intend the content to continue to be | ||
7961 | available). Content stored in the DHT expires or might be lost due to | ||
7962 | churn. | ||
7963 | Furthermore, GNUnet's DHT typically requires multiple rounds of PUT | ||
7964 | operations before a key-value pair is consistently available to all | ||
7965 | peers (the DHT randomizes paths and thus storage locations, and only | ||
7966 | after multiple rounds of PUTs there will be a sufficient number of | ||
7967 | replicas in large DHTs). An explicit PUT operation using the DHT API will | ||
7968 | only cause network traffic once, so in order to ensure basic availability | ||
7969 | and resistance to churn (and adversaries), PUTs must be repeated. | ||
7970 | While the exact frequency depends on the application, a rule of thumb is | ||
7971 | that there should be at least a dozen PUT operations within the content | ||
7972 | lifetime. Content in the DHT typically expires after one day, so | ||
7973 | DHT PUT operations should be repeated at least every 1-2 hours. | ||
7974 | |||
7975 | @node MONITOR | ||
7976 | @subsubsection MONITOR | ||
7977 | |||
7978 | |||
7979 | |||
7980 | The DHT API also allows applications to monitor messages crossing the | ||
7981 | local DHT service. | ||
7982 | The types of messages used by the DHT are GET, PUT and RESULT messages. | ||
7983 | Using the monitoring API, applications can choose to monitor these | ||
7984 | requests, possibly limiting themselves to requests for a particular block | ||
7985 | type. | ||
7986 | |||
7987 | The monitoring API is not only useful for diagnostics, it can also be | ||
7988 | used to trigger application operations based on PUT operations. | ||
7989 | For example, an application may use PUTs to distribute work requests to | ||
7990 | other peers. | ||
7991 | The workers would then monitor for PUTs that give them work, instead of | ||
7992 | looking for work using GET operations. | ||
7993 | This can be beneficial, especially if the workers have no good way to | ||
7994 | guess the keys under which work would be stored. | ||
7995 | Naturally, additional protocols might be needed to ensure that the desired | ||
7996 | number of workers will process the distributed workload. | ||
7997 | |||
7998 | @node DHT Routing Options | ||
7999 | @subsubsection DHT Routing Options | ||
8000 | |||
8001 | |||
8002 | |||
8003 | There are two important options for GET and PUT requests: | ||
8004 | |||
8005 | @table @asis | ||
8006 | @item GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all | ||
8007 | peers should process the request, even if their peer ID is not closest to | ||
8008 | the key. For a PUT request, this means that all peers that a request | ||
8009 | traverses may make a copy of the data. | ||
8010 | Similarly for a GET request, all peers will check their local database | ||
8011 | for a result. Setting this option can thus significantly improve caching | ||
8012 | and reduce bandwidth consumption --- at the expense of a larger DHT | ||
8013 | database. If in doubt, we recommend that this option should be used. | ||
8014 | @item GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record | ||
8015 | the path that a GET or a PUT request is taking through the overlay | ||
8016 | network. The resulting paths are then returned to the application with | ||
8017 | the respective result. This allows the receiver of a result to construct | ||
8018 | a path to the originator of the data, which might then be used for | ||
8019 | routing. Naturally, setting this option requires additional bandwidth | ||
8020 | and disk space, so applications should only set this if the paths are | ||
8021 | needed by the application logic. | ||
8022 | @item GNUNET_DHT_RO_FIND_PEER This option is an internal option used by | ||
8023 | the DHT's peer discovery mechanism and should not be used by applications. | ||
8024 | @item GNUNET_DHT_RO_BART This option is currently not implemented. It may | ||
8025 | in the future offer performance improvements for clique topologies. | ||
8026 | @end table | ||
8027 | |||
8028 | @node The DHT Client-Service Protocol | ||
8029 | @subsection The DHT Client-Service Protocol | ||
8030 | |||
8031 | |||
8032 | |||
8033 | @menu | ||
8034 | * PUTting data into the DHT:: | ||
8035 | * GETting data from the DHT:: | ||
8036 | * Monitoring the DHT:: | ||
8037 | @end menu | ||
8038 | |||
8039 | @node PUTting data into the DHT | ||
8040 | @subsubsection PUTting data into the DHT | ||
8041 | |||
8042 | |||
8043 | |||
8044 | To store (PUT) data into the DHT, the client sends a | ||
8045 | @code{struct GNUNET_DHT_ClientPutMessage} to the service. | ||
8046 | This message specifies the block type, routing options, the desired | ||
8047 | replication level, the expiration time, key, | ||
8048 | value and a 64-bit unique ID for the operation. The service responds with | ||
8049 | a @code{struct GNUNET_DHT_ClientPutConfirmationMessage} with the same | ||
8050 | 64-bit unique ID. Note that the service sends the confirmation as soon as | ||
8051 | it has locally processed the PUT request. The PUT may still be | ||
8052 | propagating through the network at this time. | ||
8053 | |||
8054 | In the future, we may want to change this to provide (limited) feedback | ||
8055 | to the client, for example if we detect that the PUT operation had no | ||
8056 | effect because the same key-value pair was already stored in the DHT. | ||
8057 | However, changing this would also require additional state and messages | ||
8058 | in the P2P interaction. | ||
8059 | |||
8060 | @node GETting data from the DHT | ||
8061 | @subsubsection GETting data from the DHT | ||
8062 | |||
8063 | |||
8064 | |||
8065 | To retrieve (GET) data from the DHT, the client sends a | ||
8066 | @code{struct GNUNET_DHT_ClientGetMessage} to the service. The message | ||
8067 | specifies routing options, a replication level (for replicating the GET, | ||
8068 | not the content), the desired block type, the key, the (optional) | ||
8069 | extended query and unique 64-bit request ID. | ||
8070 | |||
8071 | Additionally, the client may send any number of | ||
8072 | @code{struct GNUNET_DHT_ClientGetResultSeenMessage}s to notify the | ||
8073 | service about results that the client is already aware of. | ||
8074 | These messages consist of the key, the unique 64-bit ID of the request, | ||
8075 | and an arbitrary number of hash codes over the blocks that the client is | ||
8076 | already aware of. As messages are restricted to 64k, a client that | ||
8077 | already knows more than about a thousand blocks may need to send | ||
8078 | several of these messages. Naturally, the client should transmit these | ||
8079 | messages as quickly as possible after the original GET request such that | ||
8080 | the DHT can filter those results in the network early on. Naturally, as | ||
8081 | these messages are sent after the original request, it is conceivable | ||
8082 | that the DHT service may return blocks that match those already known | ||
8083 | to the client anyway. | ||
8084 | |||
8085 | In response to a GET request, the service will send @code{struct | ||
8086 | GNUNET_DHT_ClientResultMessage}s to the client. These messages contain the | ||
8087 | block type, expiration, key, unique ID of the request and of course the | ||
8088 | value (a block). Depending on the options set for the respective | ||
8089 | operations, the replies may also contain the path the GET and/or the PUT | ||
8090 | took through the network. | ||
8091 | |||
8092 | A client can stop receiving replies either by disconnecting or by sending | ||
8093 | a @code{struct GNUNET_DHT_ClientGetStopMessage} which must contain the | ||
8094 | key and the 64-bit unique ID of the original request. Using an | ||
8095 | explicit "stop" message is more common as this allows a client to run | ||
8096 | many concurrent GET operations over the same connection with the DHT | ||
8097 | service --- and to stop them individually. | ||
8098 | |||
8099 | @node Monitoring the DHT | ||
8100 | @subsubsection Monitoring the DHT | ||
8101 | |||
8102 | |||
8103 | |||
8104 | To begin monitoring, the client sends a | ||
8105 | @code{struct GNUNET_DHT_MonitorStartStop} message to the DHT service. | ||
8106 | In this message, flags can be set to enable (or disable) monitoring of | ||
8107 | GET, PUT and RESULT messages that pass through a peer. The message can | ||
8108 | also restrict monitoring to a particular block type or a particular key. | ||
8109 | Once monitoring is enabled, the DHT service will notify the client about | ||
8110 | any matching event using @code{struct GNUNET_DHT_MonitorGetMessage}s for | ||
8111 | GET events, @code{struct GNUNET_DHT_MonitorPutMessage} for PUT events | ||
8112 | and @code{struct GNUNET_DHT_MonitorGetRespMessage} for RESULTs. Each of | ||
8113 | these messages contains all of the information about the event. | ||
8114 | |||
8115 | @node The DHT Peer-to-Peer Protocol | ||
8116 | @subsection The DHT Peer-to-Peer Protocol | ||
8117 | |||
8118 | |||
8119 | |||
8120 | @menu | ||
8121 | * Routing GETs or PUTs:: | ||
8122 | * PUTting data into the DHT2:: | ||
8123 | * GETting data from the DHT2:: | ||
8124 | @end menu | ||
8125 | |||
8126 | @node Routing GETs or PUTs | ||
8127 | @subsubsection Routing GETs or PUTs | ||
8128 | |||
8129 | |||
8130 | |||
8131 | When routing GETs or PUTs, the DHT service selects a suitable subset of | ||
8132 | neighbours for forwarding. The exact number of neighbours can be zero or | ||
8133 | more and depends on the hop counter of the query (initially zero) in | ||
8134 | relation to the (log of) the network size estimate, the desired | ||
8135 | replication level and the peer's connectivity. | ||
8136 | Depending on the hop counter and our network size estimate, the selection | ||
8137 | of the peers maybe randomized or by proximity to the key. | ||
8138 | Furthermore, requests include a set of peers that a request has already | ||
8139 | traversed; those peers are also excluded from the selection. | ||
8140 | |||
8141 | @node PUTting data into the DHT2 | ||
8142 | @subsubsection PUTting data into the DHT2 | ||
8143 | |||
8144 | |||
8145 | |||
8146 | To PUT data into the DHT, the service sends a @code{struct PeerPutMessage} | ||
8147 | of type @code{GNUNET_MESSAGE_TYPE_DHT_P2P_PUT} to the respective | ||
8148 | neighbour. | ||
8149 | In addition to the usual information about the content (type, routing | ||
8150 | options, desired replication level for the content, expiration time, key | ||
8151 | and value), the message contains a fixed-size Bloom filter with | ||
8152 | information about which peers (may) have already seen this request. | ||
8153 | This Bloom filter is used to ensure that DHT messages never loop back to | ||
8154 | a peer that has already processed the request. | ||
8155 | Additionally, the message includes the current hop counter and, depending | ||
8156 | on the routing options, the message may include the full path that the | ||
8157 | message has taken so far. | ||
8158 | The Bloom filter should already contain the identity of the previous hop; | ||
8159 | however, the path should not include the identity of the previous hop and | ||
8160 | the receiver should append the identity of the sender to the path, not | ||
8161 | its own identity (this is done to reduce bandwidth). | ||
8162 | |||
8163 | @node GETting data from the DHT2 | ||
8164 | @subsubsection GETting data from the DHT2 | ||
8165 | |||
8166 | |||
8167 | |||
8168 | A peer can search the DHT by sending @code{struct PeerGetMessage}s of type | ||
8169 | @code{GNUNET_MESSAGE_TYPE_DHT_P2P_GET} to other peers. In addition to the | ||
8170 | usual information about the request (type, routing options, desired | ||
8171 | replication level for the request, the key and the extended query), a GET | ||
8172 | request also contains a hop counter, a Bloom filter over the peers | ||
8173 | that have processed the request already and depending on the routing | ||
8174 | options the full path traversed by the GET. | ||
8175 | Finally, a GET request includes a variable-size second Bloom filter and a | ||
8176 | so-called Bloom filter mutator value which together indicate which | ||
8177 | replies the sender has already seen. During the lookup, each block that | ||
8178 | matches they block type, key and extended query is additionally subjected | ||
8179 | to a test against this Bloom filter. | ||
8180 | The block plugin is expected to take the hash of the block and combine it | ||
8181 | with the mutator value and check if the result is not yet in the Bloom | ||
8182 | filter. The originator of the query will from time to time modify the | ||
8183 | mutator to (eventually) allow false-positives filtered by the Bloom filter | ||
8184 | to be returned. | ||
8185 | |||
8186 | Peers that receive a GET request perform a local lookup (depending on | ||
8187 | their proximity to the key and the query options) and forward the request | ||
8188 | to other peers. | ||
8189 | They then remember the request (including the Bloom filter for blocking | ||
8190 | duplicate results) and when they obtain a matching, non-filtered response | ||
8191 | a @code{struct PeerResultMessage} of type | ||
8192 | @code{GNUNET_MESSAGE_TYPE_DHT_P2P_RESULT} is forwarded to the previous | ||
8193 | hop. | ||
8194 | Whenever a result is forwarded, the block plugin is used to update the | ||
8195 | Bloom filter accordingly, to ensure that the same result is never | ||
8196 | forwarded more than once. | ||
8197 | The DHT service may also cache forwarded results locally if the | ||
8198 | "CACHE_RESULTS" option is set to "YES" in the configuration. | ||
8199 | |||
8200 | @cindex GNS | ||
8201 | @cindex GNU Name System | ||
8202 | @node GNU Name System (GNS) | ||
8203 | @section GNU Name System (GNS) | ||
8204 | |||
8205 | |||
8206 | |||
8207 | The GNU Name System (GNS) is a decentralized database that enables users | ||
8208 | to securely resolve names to values. | ||
8209 | Names can be used to identify other users (for example, in social | ||
8210 | networking), or network services (for example, VPN services running at a | ||
8211 | peer in GNUnet, or purely IP-based services on the Internet). | ||
8212 | Users interact with GNS by typing in a hostname that ends in a | ||
8213 | top-level domain that is configured in the ``GNS'' section, matches | ||
8214 | an identity of the user or ends in a Base32-encoded public key. | ||
8215 | |||
8216 | Videos giving an overview of most of the GNS and the motivations behind | ||
8217 | it is available here and here. | ||
8218 | The remainder of this chapter targets developers that are familiar with | ||
8219 | high level concepts of GNS as presented in these talks. | ||
8220 | @c TODO: Add links to here and here and to these. | ||
8221 | |||
8222 | GNS-aware applications should use the GNS resolver to obtain the | ||
8223 | respective records that are stored under that name in GNS. | ||
8224 | Each record consists of a type, value, expiration time and flags. | ||
8225 | |||
8226 | The type specifies the format of the value. Types below 65536 correspond | ||
8227 | to DNS record types, larger values are used for GNS-specific records. | ||
8228 | Applications can define new GNS record types by reserving a number and | ||
8229 | implementing a plugin (which mostly needs to convert the binary value | ||
8230 | representation to a human-readable text format and vice-versa). | ||
8231 | The expiration time specifies how long the record is to be valid. | ||
8232 | The GNS API ensures that applications are only given non-expired values. | ||
8233 | The flags are typically irrelevant for applications, as GNS uses them | ||
8234 | internally to control visibility and validity of records. | ||
8235 | |||
8236 | Records are stored along with a signature. | ||
8237 | The signature is generated using the private key of the authoritative | ||
8238 | zone. This allows any GNS resolver to verify the correctness of a | ||
8239 | name-value mapping. | ||
8240 | |||
8241 | Internally, GNS uses the NAMECACHE to cache information obtained from | ||
8242 | other users, the NAMESTORE to store information specific to the local | ||
8243 | users, and the DHT to exchange data between users. | ||
8244 | A plugin API is used to enable applications to define new GNS | ||
8245 | record types. | ||
8246 | |||
8247 | @menu | ||
8248 | * libgnunetgns:: | ||
8249 | * libgnunetgnsrecord:: | ||
8250 | * GNS plugins:: | ||
8251 | * The GNS Client-Service Protocol:: | ||
8252 | * Hijacking the DNS-Traffic using gnunet-service-dns:: | ||
8253 | @c * Serving DNS lookups via GNS on W32:: | ||
8254 | * Importing DNS Zones into GNS:: | ||
8255 | * Registering names using the FCFS daemon:: | ||
8256 | @end menu | ||
8257 | |||
8258 | @node libgnunetgns | ||
8259 | @subsection libgnunetgns | ||
8260 | |||
8261 | |||
8262 | |||
8263 | The GNS API itself is extremely simple. Clients first connect to the | ||
8264 | GNS service using @code{GNUNET_GNS_connect}. | ||
8265 | They can then perform lookups using @code{GNUNET_GNS_lookup} or cancel | ||
8266 | pending lookups using @code{GNUNET_GNS_lookup_cancel}. | ||
8267 | Once finished, clients disconnect using @code{GNUNET_GNS_disconnect}. | ||
8268 | |||
8269 | @menu | ||
8270 | * Looking up records:: | ||
8271 | * Accessing the records:: | ||
8272 | * Creating records:: | ||
8273 | * Future work:: | ||
8274 | @end menu | ||
8275 | |||
8276 | @node Looking up records | ||
8277 | @subsubsection Looking up records | ||
8278 | |||
8279 | |||
8280 | |||
8281 | @code{GNUNET_GNS_lookup} takes a number of arguments: | ||
8282 | |||
8283 | @table @asis | ||
8284 | @item handle This is simply the GNS connection handle from | ||
8285 | @code{GNUNET_GNS_connect}. | ||
8286 | @item name The client needs to specify the name to | ||
8287 | be resolved. This can be any valid DNS or GNS hostname. | ||
8288 | @item zone The client | ||
8289 | needs to specify the public key of the GNS zone against which the | ||
8290 | resolution should be done. | ||
8291 | Note that a key must be provided, the client should | ||
8292 | look up plausible values using its configuration, | ||
8293 | the identity service and by attempting to interpret the | ||
8294 | TLD as a base32-encoded public key. | ||
8295 | @item type This is the desired GNS or DNS record type | ||
8296 | to look for. While all records for the given name will be returned, this | ||
8297 | can be important if the client wants to resolve record types that | ||
8298 | themselves delegate resolution, such as CNAME, PKEY or GNS2DNS. | ||
8299 | Resolving a record of any of these types will only work if the respective | ||
8300 | record type is specified in the request, as the GNS resolver will | ||
8301 | otherwise follow the delegation and return the records from the | ||
8302 | respective destination, instead of the delegating record. | ||
8303 | @item only_cached This argument should typically be set to | ||
8304 | @code{GNUNET_NO}. Setting it to @code{GNUNET_YES} disables resolution via | ||
8305 | the overlay network. | ||
8306 | @item shorten_zone_key If GNS encounters new names during resolution, | ||
8307 | their respective zones can automatically be learned and added to the | ||
8308 | "shorten zone". If this is desired, clients must pass the private key of | ||
8309 | the shorten zone. If NULL is passed, shortening is disabled. | ||
8310 | @item proc This argument identifies | ||
8311 | the function to call with the result. It is given proc_cls, the number of | ||
8312 | records found (possibly zero) and the array of the records as arguments. | ||
8313 | proc will only be called once. After proc,> has been called, the lookup | ||
8314 | must no longer be canceled. | ||
8315 | @item proc_cls The closure for proc. | ||
8316 | @end table | ||
8317 | |||
8318 | @node Accessing the records | ||
8319 | @subsubsection Accessing the records | ||
8320 | |||
8321 | |||
8322 | |||
8323 | The @code{libgnunetgnsrecord} library provides an API to manipulate the | ||
8324 | GNS record array that is given to proc. In particular, it offers | ||
8325 | functions such as converting record values to human-readable | ||
8326 | strings (and back). However, most @code{libgnunetgnsrecord} functions are | ||
8327 | not interesting to GNS client applications. | ||
8328 | |||
8329 | For DNS records, the @code{libgnunetdnsparser} library provides | ||
8330 | functions for parsing (and serializing) common types of DNS records. | ||
8331 | |||
8332 | @node Creating records | ||
8333 | @subsubsection Creating records | ||
8334 | |||
8335 | |||
8336 | |||
8337 | Creating GNS records is typically done by building the respective record | ||
8338 | information (possibly with the help of @code{libgnunetgnsrecord} and | ||
8339 | @code{libgnunetdnsparser}) and then using the @code{libgnunetnamestore} to | ||
8340 | publish the information. The GNS API is not involved in this | ||
8341 | operation. | ||
8342 | |||
8343 | @node Future work | ||
8344 | @subsubsection Future work | ||
8345 | |||
8346 | |||
8347 | |||
8348 | In the future, we want to expand @code{libgnunetgns} to allow | ||
8349 | applications to observe shortening operations performed during GNS | ||
8350 | resolution, for example so that users can receive visual feedback when | ||
8351 | this happens. | ||
8352 | |||
8353 | @node libgnunetgnsrecord | ||
8354 | @subsection libgnunetgnsrecord | ||
8355 | |||
8356 | |||
8357 | |||
8358 | The @code{libgnunetgnsrecord} library is used to manipulate GNS | ||
8359 | records (in plaintext or in their encrypted format). | ||
8360 | Applications mostly interact with @code{libgnunetgnsrecord} by using the | ||
8361 | functions to convert GNS record values to strings or vice-versa, or to | ||
8362 | lookup a GNS record type number by name (or vice-versa). | ||
8363 | The library also provides various other functions that are mostly | ||
8364 | used internally within GNS, such as converting keys to names, checking for | ||
8365 | expiration, encrypting GNS records to GNS blocks, verifying GNS block | ||
8366 | signatures and decrypting GNS records from GNS blocks. | ||
8367 | |||
8368 | We will now discuss the four commonly used functions of the API.@ | ||
8369 | @code{libgnunetgnsrecord} does not perform these operations itself, | ||
8370 | but instead uses plugins to perform the operation. | ||
8371 | GNUnet includes plugins to support common DNS record types as well as | ||
8372 | standard GNS record types. | ||
8373 | |||
8374 | @menu | ||
8375 | * Value handling:: | ||
8376 | * Type handling:: | ||
8377 | @end menu | ||
8378 | |||
8379 | @node Value handling | ||
8380 | @subsubsection Value handling | ||
8381 | |||
8382 | |||
8383 | |||
8384 | @code{GNUNET_GNSRECORD_value_to_string} can be used to convert | ||
8385 | the (binary) representation of a GNS record value to a human readable, | ||
8386 | 0-terminated UTF-8 string. | ||
8387 | NULL is returned if the specified record type is not supported by any | ||
8388 | available plugin. | ||
8389 | |||
8390 | @code{GNUNET_GNSRECORD_string_to_value} can be used to try to convert a | ||
8391 | human readable string to the respective (binary) representation of | ||
8392 | a GNS record value. | ||
8393 | |||
8394 | @node Type handling | ||
8395 | @subsubsection Type handling | ||
8396 | |||
8397 | |||
8398 | |||
8399 | @code{GNUNET_GNSRECORD_typename_to_number} can be used to obtain the | ||
8400 | numeric value associated with a given typename. For example, given the | ||
8401 | typename "A" (for DNS A reocrds), the function will return the number 1. | ||
8402 | A list of common DNS record types is | ||
8403 | @uref{http://en.wikipedia.org/wiki/List_of_DNS_record_types, here}. | ||
8404 | Note that not all DNS record types are supported by GNUnet GNSRECORD | ||
8405 | plugins at this time. | ||
8406 | |||
8407 | @code{GNUNET_GNSRECORD_number_to_typename} can be used to obtain the | ||
8408 | typename associated with a given numeric value. | ||
8409 | For example, given the type number 1, the function will return the | ||
8410 | typename "A". | ||
8411 | |||
8412 | @node GNS plugins | ||
8413 | @subsection GNS plugins | ||
8414 | |||
8415 | |||
8416 | |||
8417 | Adding a new GNS record type typically involves writing (or extending) a | ||
8418 | GNSRECORD plugin. The plugin needs to implement the | ||
8419 | @code{gnunet_gnsrecord_plugin.h} API which provides basic functions that | ||
8420 | are needed by GNSRECORD to convert typenames and values of the respective | ||
8421 | record type to strings (and back). | ||
8422 | These gnsrecord plugins are typically implemented within their respective | ||
8423 | subsystems. | ||
8424 | Examples for such plugins can be found in the GNSRECORD, GNS and | ||
8425 | CONVERSATION subsystems. | ||
8426 | |||
8427 | The @code{libgnunetgnsrecord} library is then used to locate, load and | ||
8428 | query the appropriate gnsrecord plugin. | ||
8429 | Which plugin is appropriate is determined by the record type (which is | ||
8430 | just a 32-bit integer). The @code{libgnunetgnsrecord} library loads all | ||
8431 | block plugins that are installed at the local peer and forwards the | ||
8432 | application request to the plugins. If the record type is not | ||
8433 | supported by the plugin, it should simply return an error code. | ||
8434 | |||
8435 | The central functions of the block APIs (plugin and main library) are the | ||
8436 | same four functions for converting between values and strings, and | ||
8437 | typenames and numbers documented in the previous subsection. | ||
8438 | |||
8439 | @node The GNS Client-Service Protocol | ||
8440 | @subsection The GNS Client-Service Protocol | ||
8441 | |||
8442 | |||
8443 | The GNS client-service protocol consists of two simple messages, the | ||
8444 | @code{LOOKUP} message and the @code{LOOKUP_RESULT}. Each @code{LOOKUP} | ||
8445 | message contains a unique 32-bit identifier, which will be included in the | ||
8446 | corresponding response. Thus, clients can send many lookup requests in | ||
8447 | parallel and receive responses out-of-order. | ||
8448 | A @code{LOOKUP} request also includes the public key of the GNS zone, | ||
8449 | the desired record type and fields specifying whether shortening is | ||
8450 | enabled or networking is disabled. Finally, the @code{LOOKUP} message | ||
8451 | includes the name to be resolved. | ||
8452 | |||
8453 | The response includes the number of records and the records themselves | ||
8454 | in the format created by @code{GNUNET_GNSRECORD_records_serialize}. | ||
8455 | They can thus be deserialized using | ||
8456 | @code{GNUNET_GNSRECORD_records_deserialize}. | ||
8457 | |||
8458 | @node Hijacking the DNS-Traffic using gnunet-service-dns | ||
8459 | @subsection Hijacking the DNS-Traffic using gnunet-service-dns | ||
8460 | |||
8461 | |||
8462 | |||
8463 | This section documents how the gnunet-service-dns (and the | ||
8464 | gnunet-helper-dns) intercepts DNS queries from the local system. | ||
8465 | This is merely one method for how we can obtain GNS queries. | ||
8466 | It is also possible to change @code{resolv.conf} to point to a machine | ||
8467 | running @code{gnunet-dns2gns} or to modify libc's name system switch | ||
8468 | (NSS) configuration to include a GNS resolution plugin. | ||
8469 | The method described in this chapter is more of a last-ditch catch-all | ||
8470 | approach. | ||
8471 | |||
8472 | @code{gnunet-service-dns} enables intercepting DNS traffic using policy | ||
8473 | based routing. | ||
8474 | We MARK every outgoing DNS-packet if it was not sent by our application. | ||
8475 | Using a second routing table in the Linux kernel these marked packets are | ||
8476 | then routed through our virtual network interface and can thus be | ||
8477 | captured unchanged. | ||
8478 | |||
8479 | Our application then reads the query and decides how to handle it. | ||
8480 | If the query can be addressed via GNS, it is passed to | ||
8481 | @code{gnunet-service-gns} and resolved internally using GNS. | ||
8482 | In the future, a reverse query for an address of the configured virtual | ||
8483 | network could be answered with records kept about previous forward | ||
8484 | queries. | ||
8485 | Queries that are not hijacked by some application using the DNS service | ||
8486 | will be sent to the original recipient. | ||
8487 | The answer to the query will always be sent back through the virtual | ||
8488 | interface with the original nameserver as source address. | ||
8489 | |||
8490 | |||
8491 | @menu | ||
8492 | * Network Setup Details:: | ||
8493 | @end menu | ||
8494 | |||
8495 | @node Network Setup Details | ||
8496 | @subsubsection Network Setup Details | ||
8497 | |||
8498 | |||
8499 | |||
8500 | The DNS interceptor adds the following rules to the Linux kernel: | ||
8501 | @example | ||
8502 | iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \ | ||
8503 | -j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \ | ||
8504 | --set-mark 3 ip rule add fwmark 3 table2 ip route add default via \ | ||
8505 | $VIRTUALDNS table2 | ||
8506 | @end example | ||
8507 | |||
8508 | @c FIXME: Rewrite to reflect display which is no longer content by line | ||
8509 | @c FIXME: due to the < 74 characters limit. | ||
8510 | Line 1 makes sure that all packets coming from a port our application | ||
8511 | opened beforehand (@code{$LOCALPORT}) will be routed normally. | ||
8512 | Line 2 marks every other packet to a DNS-Server with mark 3 (chosen | ||
8513 | arbitrarily). The third line adds a routing policy based on this mark | ||
8514 | 3 via the routing table. | ||
8515 | |||
8516 | @c @node Serving DNS lookups via GNS on W32 | ||
8517 | @c @subsection Serving DNS lookups via GNS on W32 | ||
8518 | |||
8519 | |||
8520 | |||
8521 | @c This section documents how the libw32nsp (and | ||
8522 | @c gnunet-gns-helper-service-w32) do DNS resolutions of DNS queries on the | ||
8523 | @c local system. This only applies to GNUnet running on W32. | ||
8524 | |||
8525 | @c W32 has a concept of "Namespaces" and "Namespace providers". | ||
8526 | @c These are used to present various name systems to applications in a | ||
8527 | @c generic way. | ||
8528 | @c Namespaces include DNS, mDNS, NLA and others. For each namespace any | ||
8529 | @c number of providers could be registered, and they are queried in an order | ||
8530 | @c of priority (which is adjustable). | ||
8531 | |||
8532 | @c Applications can resolve names by using WSALookupService*() family of | ||
8533 | @c functions. | ||
8534 | |||
8535 | @c However, these are WSA-only facilities. Common BSD socket functions for | ||
8536 | @c namespace resolutions are gethostbyname and getaddrinfo (among others). | ||
8537 | @c These functions are implemented internally (by default - by mswsock, | ||
8538 | @c which also implements the default DNS provider) as wrappers around | ||
8539 | @c WSALookupService*() functions (see "Sample Code for a Service Provider" | ||
8540 | @c on MSDN). | ||
8541 | |||
8542 | @c On W32 GNUnet builds a libw32nsp - a namespace provider, which can then be | ||
8543 | @c installed into the system by using w32nsp-install (and uninstalled by | ||
8544 | @c w32nsp-uninstall), as described in "Installation Handbook". | ||
8545 | |||
8546 | @c libw32nsp is very simple and has almost no dependencies. As a response to | ||
8547 | @c NSPLookupServiceBegin(), it only checks that the provider GUID passed to | ||
8548 | @c it by the caller matches GNUnet DNS Provider GUID, | ||
8549 | @c then connects to | ||
8550 | @c gnunet-gns-helper-service-w32 at 127.0.0.1:5353 (hardcoded) and sends the | ||
8551 | @c name resolution request there, returning the connected socket to the | ||
8552 | @c caller. | ||
8553 | |||
8554 | @c When the caller invokes NSPLookupServiceNext(), libw32nsp reads a | ||
8555 | @c completely formed reply from that socket, unmarshalls it, then gives | ||
8556 | @c it back to the caller. | ||
8557 | |||
8558 | @c At the moment gnunet-gns-helper-service-w32 is implemented to ever give | ||
8559 | @c only one reply, and subsequent calls to NSPLookupServiceNext() will fail | ||
8560 | @c with WSA_NODATA (first call to NSPLookupServiceNext() might also fail if | ||
8561 | @c GNS failed to find the name, or there was an error connecting to it). | ||
8562 | |||
8563 | @c gnunet-gns-helper-service-w32 does most of the processing: | ||
8564 | |||
8565 | @c @itemize @bullet | ||
8566 | @c @item Maintains a connection to GNS. | ||
8567 | @c @item Reads GNS config and loads appropriate keys. | ||
8568 | @c @item Checks service GUID and decides on the type of record to look up, | ||
8569 | @c refusing to make a lookup outright when unsupported service GUID is | ||
8570 | @c passed. | ||
8571 | @c @item Launches the lookup | ||
8572 | @c @end itemize | ||
8573 | |||
8574 | @c When lookup result arrives, gnunet-gns-helper-service-w32 forms a complete | ||
8575 | @c reply (including filling a WSAQUERYSETW structure and, possibly, a binary | ||
8576 | @c blob with a hostent structure for gethostbyname() client), marshalls it, | ||
8577 | @c and sends it back to libw32nsp. If no records were found, it sends an | ||
8578 | @c empty header. | ||
8579 | |||
8580 | @c This works for most normal applications that use gethostbyname() or | ||
8581 | @c getaddrinfo() to resolve names, but fails to do anything with | ||
8582 | @c applications that use alternative means of resolving names (such as | ||
8583 | @c sending queries to a DNS server directly by themselves). | ||
8584 | @c This includes some of well known utilities, like "ping" and "nslookup". | ||
8585 | |||
8586 | @node Importing DNS Zones into GNS | ||
8587 | @subsection Importing DNS Zones into GNS | ||
8588 | |||
8589 | This section discusses the challenges and problems faced when writing the | ||
8590 | Ascension tool. It also takes a look at possible improvements in the | ||
8591 | future. | ||
8592 | |||
8593 | Consider the following diagram that shows the workflow of Ascension: | ||
8594 | |||
8595 | @image{images/ascension_ssd,6in,,Ascensions workflow} | ||
8596 | |||
8597 | Further the interaction between components of GNUnet are shown in the diagram | ||
8598 | below: | ||
8599 | @center @image{images/ascension_interaction,,6in,Ascensions workflow} | ||
8600 | |||
8601 | @menu | ||
8602 | * Conversions between DNS and GNS:: | ||
8603 | * DNS Zone Size:: | ||
8604 | * Performance:: | ||
8605 | @end menu | ||
8606 | |||
8607 | @cindex DNS Conversion | ||
8608 | @node Conversions between DNS and GNS | ||
8609 | @subsubsection Conversions between DNS and GNS | ||
8610 | |||
8611 | The differences between the two name systems lies in the details and is not | ||
8612 | always transparent. For instance an SRV record is converted to a BOX record | ||
8613 | which is unique to GNS. | ||
8614 | |||
8615 | This is done by converting to a BOX record from an existing SRV record: | ||
8616 | |||
8617 | @example | ||
8618 | # SRV | ||
8619 | # _service._proto.name. TTL class SRV priority weight port target | ||
8620 | _sip._tcp.example.com. 14000 IN SRV 0 0 5060 www.example.com. | ||
8621 | # BOX | ||
8622 | # TTL BOX flags port protocol recordtype priority weight port target | ||
8623 | 14000 BOX n 5060 6 33 0 0 5060 www.example.com | ||
8624 | @end example | ||
8625 | |||
8626 | Other records that need to undergo such transformation is the MX record type, | ||
8627 | as well as the SOA record type. | ||
8628 | |||
8629 | Transformation of a SOA record into GNS works as described in the | ||
8630 | following example. Very important to note are the rname and mname keys. | ||
8631 | |||
8632 | @example | ||
8633 | # BIND syntax for a clean SOA record | ||
8634 | @ IN SOA master.example.com. hostmaster.example.com. ( | ||
8635 | 2017030300 ; serial | ||
8636 | 3600 ; refresh | ||
8637 | 1800 ; retry | ||
8638 | 604800 ; expire | ||
8639 | 600 ) ; ttl | ||
8640 | # Recordline for adding the record | ||
8641 | $ gnunet-namestore -z example.com -a -n @ -t SOA -V \ | ||
8642 | rname=master.example.com mname=hostmaster.example.com \ | ||
8643 | 2017030300,3600,1800,604800,600 -e 7200s | ||
8644 | @end example | ||
8645 | |||
8646 | The transformation of MX records is done in a simple way. | ||
8647 | @example | ||
8648 | # mail.example.com. 3600 IN MX 10 mail.example.com. | ||
8649 | $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail | ||
8650 | @end example | ||
8651 | |||
8652 | Finally, one of the biggest struggling points were the NS records that are | ||
8653 | found in top level domain zones. The intended behaviour for those is to add | ||
8654 | GNS2DNS records for those so that gnunet-gns can resolve records for those | ||
8655 | domains on its own. Those require the values from DNS GLUE records, provided | ||
8656 | they are within the same zone. | ||
8657 | |||
8658 | The following two examples show one record with a GLUE record and the other one | ||
8659 | does not have a GLUE record. This takes place in the 'com' TLD. | ||
8660 | |||
8661 | @example | ||
8662 | # ns1.example.com 86400 IN A 127.0.0.1 | ||
8663 | # example.com 86400 IN NS ns1.example.com. | ||
8664 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ | ||
8665 | example.com@@127.0.0.1 | ||
8666 | |||
8667 | # example.com 86400 IN NS ns1.example.org. | ||
8668 | $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \ | ||
8669 | example.com@@ns1.example.org | ||
8670 | @end example | ||
8671 | |||
8672 | As you can see, one of the GNS2DNS records has an IP address listed and the | ||
8673 | other one a DNS name. For the first one there is a GLUE record to do the | ||
8674 | translation directly and the second one will issue another DNS query to figure | ||
8675 | out the IP of ns1.example.org. | ||
8676 | |||
8677 | A solution was found by creating a hierarchical zone structure in GNS and linking | ||
8678 | the zones using PKEY records to one another. This allows the resolution of the | ||
8679 | name servers to work within GNS while not taking control over unwanted zones. | ||
8680 | |||
8681 | Currently the following record types are supported: | ||
8682 | @itemize @bullet | ||
8683 | @item A | ||
8684 | @item AAAA | ||
8685 | @item CNAME | ||
8686 | @item MX | ||
8687 | @item NS | ||
8688 | @item SRV | ||
8689 | @item TXT | ||
8690 | @end itemize | ||
8691 | |||
8692 | This is not due to technical limitations but rather a practical ones. The | ||
8693 | problem occurs with DNSSEC enabled DNS zones. As records within those zones are | ||
8694 | signed periodically, and every new signature is an update to the zone, there are | ||
8695 | many revisions of zones. This results in a problem with bigger zones as there | ||
8696 | are lots of records that have been signed again but no major changes. Also | ||
8697 | trying to add records that are unknown that require a different format take time | ||
8698 | as they cause a CLI call of the namestore. Furthermore certain record types | ||
8699 | need transformation into a GNS compatible format which, depending on the record | ||
8700 | type, takes more time. | ||
8701 | |||
8702 | Further a blacklist was added to drop for instance DNSSEC related records. Also | ||
8703 | if a record type is neither in the white list nor the blacklist it is considered | ||
8704 | as a loss of data and a message is shown to the user. This helps with | ||
8705 | transparency and also with contributing, as the not supported record types can | ||
8706 | then be added accordingly. | ||
8707 | |||
8708 | @node DNS Zone Size | ||
8709 | @subsubsection DNS Zone Size | ||
8710 | Another very big problem exists with very large zones. When migrating a small | ||
8711 | zone the delay between adding of records and their expiry is negligible. However | ||
8712 | when working with big zones that easily have more than a few million records | ||
8713 | this delay becomes a problem. | ||
8714 | |||
8715 | Records will start to expire well before the zone has finished migrating. This | ||
8716 | is usually not a problem but can cause a high CPU load when a peer is restarted | ||
8717 | and the records have expired. | ||
8718 | |||
8719 | A good solution has not been found yet. One of the idea that floated around was | ||
8720 | that the records should be added with the s (shadow) flag to keep the records | ||
8721 | resolvable even if they expired. However this would introduce the problem of how | ||
8722 | to detect if a record has been removed from the zone and would require deletion | ||
8723 | of said record(s). | ||
8724 | |||
8725 | Another problem that still persists is how to refresh records. Expired records | ||
8726 | are still displayed when calling gnunet-namestore but do not resolve with | ||
8727 | gnunet-gns. Zonemaster will sign the expired records again and make sure that | ||
8728 | the records are still valid. With a recent change this was fixed as gnunet-gns | ||
8729 | to improve the suffix lookup which allows for a fast lookup even with thousands | ||
8730 | of local egos. | ||
8731 | |||
8732 | Currently the pace of adding records in general is around 10 records per second. | ||
8733 | Crypto is the upper limit for adding of records. The performance of your machine | ||
8734 | can be tested with the perf_crypto_* tools. There is still a big discrepancy | ||
8735 | between the pace of Ascension and the theoretical limit. | ||
8736 | |||
8737 | A performance metric for measuring improvements has not yet been implemented in | ||
8738 | Ascension. | ||
8739 | |||
8740 | @node Performance | ||
8741 | @subsubsection Performance | ||
8742 | The performance when migrating a zone using the Ascension tool is limited by a | ||
8743 | handful of factors. First of all ascension is written in Python3 and calls the | ||
8744 | CLI tools of GNUnet. This is comparable to a fork and exec call which costs a | ||
8745 | few CPU cycles. Furthermore all the records that are added to the same | ||
8746 | label are signed using the zones private key. This signing operation is very | ||
8747 | resource heavy and was optimized during development by adding the '-R' | ||
8748 | (Recordline) option to gnunet-namestore which allows to specify multiple records | ||
8749 | using the CLI tool. Assuming that in a TLD zone every domain has at least two | ||
8750 | name servers this halves the amount of signatures needed. | ||
8751 | |||
8752 | Another improvement that could be made is with the addition of multiple threads | ||
8753 | or using asynchronous subprocesses when opening the GNUnet CLI tools. This could | ||
8754 | be implemented by simply creating more workers in the program but performance | ||
8755 | improvements were not tested. | ||
8756 | |||
8757 | Ascension was tested using different hardware and database backends. Performance | ||
8758 | differences between SQLite and postgresql are marginal and almost non existent. | ||
8759 | What did make a huge impact on record adding performance was the storage medium. | ||
8760 | On a traditional mechanical hard drive adding of records were slow compared to a | ||
8761 | solid state disk. | ||
8762 | |||
8763 | In conclusion there are many bottlenecks still around in the program, namely the | ||
8764 | single threaded implementation and inefficient, sequential calls of | ||
8765 | gnunet-namestore. In the future a solution that uses the C API would be cleaner | ||
8766 | and better. | ||
8767 | |||
8768 | @node Registering names using the FCFS daemon | ||
8769 | @subsection Registering names using the FCFS daemon | ||
8770 | |||
8771 | This section describes FCFSD, a daemon used to associate names with PKEY | ||
8772 | records following a ``First Come, First Served'' policy. This policy means | ||
8773 | that a certain name can not be registered again if someone registered it | ||
8774 | already. | ||
8775 | |||
8776 | The daemon can be started by using @code{gnunet-namestore-fcfsd}, which will | ||
8777 | start a simple HTTP server on localhost, using a port specified by the | ||
8778 | @code{HTTPORT} value in its configuration. | ||
8779 | |||
8780 | Communication is performed by sending GET or POST requests to specific paths | ||
8781 | (``endpoints''), as described in the following sections. | ||
8782 | |||
8783 | The daemon will always respond with data structured using the JSON format. | ||
8784 | The fields to be expected will be listed for each endpoint. | ||
8785 | |||
8786 | The only exceptions are for the ``root'' endpoint (i.e. @code{/}) which will | ||
8787 | return a HTML document, and two other HTML documents which will be served when | ||
8788 | certain errors are encountered, like when requesting an unknown endpoint. | ||
8789 | |||
8790 | @menu | ||
8791 | * Obtaining information from the daemon:: | ||
8792 | * Submitting data to the daemon:: | ||
8793 | * Customizing the HTML output:: | ||
8794 | @end menu | ||
8795 | |||
8796 | @cindex FCFSD GET requests | ||
8797 | @node Obtaining information from the daemon | ||
8798 | @subsubsection Obtaining information from the daemon | ||
8799 | |||
8800 | To query the daemon, a GET request must be sent to these endpoints, placing | ||
8801 | parameters in the address as per the HTTP specification, like so: | ||
8802 | |||
8803 | @example | ||
8804 | GET /endpoint?param1=value¶m2=value | ||
8805 | @end example | ||
8806 | |||
8807 | Each endpoint will be described using its name (@code{/endpoint} in the | ||
8808 | example above), followed by the name of each parameter (like @code{param1} and | ||
8809 | @code{param2}.) | ||
8810 | |||
8811 | @deffn Endpoint /search name | ||
8812 | This endpoint is used to query about the state of @var{name}, that is, whether | ||
8813 | it is available for registration or not. | ||
8814 | |||
8815 | The response JSON will contain two fields: | ||
8816 | |||
8817 | @itemize @bullet | ||
8818 | @item error | ||
8819 | @item free | ||
8820 | @end itemize | ||
8821 | |||
8822 | @code{error} can be either the string @code{"true"} or the string | ||
8823 | @code{"false"}: when @code{"true"}, it means there was an error within the | ||
8824 | daemon and the name could not be searched at all. | ||
8825 | |||
8826 | @code{free} can be either the string @code{"true"} or the string | ||
8827 | @code{"false"}: when @code{"true"}, the requested name can be registered. | ||
8828 | @end deffn | ||
8829 | |||
8830 | @cindex FCFSD POST requests | ||
8831 | @node Submitting data to the daemon | ||
8832 | @subsubsection Submitting data to the daemon | ||
8833 | |||
8834 | To send data to the daemon, a POST request must be sent to these endpoints, | ||
8835 | placing the data to submit in the body of the request, structured using the | ||
8836 | JSON format, like so: | ||
8837 | |||
8838 | @example | ||
8839 | POST /endpoint | ||
8840 | Content-Type: application/json | ||
8841 | ... | ||
8842 | |||
8843 | @{"param1": value1, "param2": value2, ...@} | ||
8844 | @end example | ||
8845 | |||
8846 | Each endpoint will be described using its name (@code{/endpoint} in the | ||
8847 | example above), followed by the name of each JSON field (like @code{param1} | ||
8848 | and @code{param2}.) | ||
8849 | |||
8850 | @deffn Endpoint /register name key | ||
8851 | This endpoint is used to register a new association between @var{name} and | ||
8852 | @var{key}. | ||
8853 | |||
8854 | For this operation to succeed, both @var{NAME} and @var{KEY} @strong{must not} | ||
8855 | be registered already. | ||
8856 | |||
8857 | The response JSON will contain two fields: | ||
8858 | |||
8859 | @itemize @bullet | ||
8860 | @item error | ||
8861 | @item message | ||
8862 | @end itemize | ||
8863 | |||
8864 | @code{error} can be either the string @code{"true"} or the string | ||
8865 | @code{"false"}: when @code{"true"}, it means the name could not be registered. | ||
8866 | Clients can get the reason of the failure from the HTTP response code or from | ||
8867 | the @code{message} field. | ||
8868 | |||
8869 | @code{message} is a string which can be used by clients to let users know the | ||
8870 | result of the operation. It might be localized to the daemon operator's | ||
8871 | locale. | ||
8872 | @end deffn | ||
8873 | |||
8874 | @node Customizing the HTML output | ||
8875 | @subsubsection Customizing the HTML output | ||
8876 | |||
8877 | In some situations, the daemon will serve HTML documents instead of JSON | ||
8878 | values. It is possible to configure the daemon to serve custom documents | ||
8879 | instead of the ones provided with GNUnet, by setting the @code{HTMLDIR} value | ||
8880 | in its configuration to a directory path. | ||
8881 | |||
8882 | Within the provided path, the daemon will search for these three files: | ||
8883 | |||
8884 | @itemize @bullet | ||
8885 | @item fcfsd-index.html | ||
8886 | @item fcfsd-notfound.html | ||
8887 | @item fcfsd-forbidden.html | ||
8888 | @end itemize | ||
8889 | |||
8890 | The @file{fcfsd-index.html} file is the daemon's ``homepage'': operators might | ||
8891 | want to provide information about the service here, or provide a form with | ||
8892 | which it is possible to register a name. | ||
8893 | |||
8894 | The @file{fcfsd-notfound.html} file is used primarily to let users know they | ||
8895 | tried to access an unknown endpoint. | ||
8896 | |||
8897 | The @file{fcfsd-forbidden.html} file is served to users when they try to | ||
8898 | access an endpoint they should not access. For example, sending an invalid | ||
8899 | request might result in this page being served. | ||
8900 | |||
8901 | @cindex GNS Namecache | ||
8902 | @node GNS Namecache | ||
8903 | @section GNS Namecache | ||
8904 | |||
8905 | The NAMECACHE subsystem is responsible for caching (encrypted) resolution | ||
8906 | results of the GNU Name System (GNS). GNS makes zone information available | ||
8907 | to other users via the DHT. However, as accessing the DHT for every | ||
8908 | lookup is expensive (and as the DHT's local cache is lost whenever the | ||
8909 | peer is restarted), GNS uses the NAMECACHE as a more persistent cache for | ||
8910 | DHT lookups. | ||
8911 | Thus, instead of always looking up every name in the DHT, GNS first | ||
8912 | checks if the result is already available locally in the NAMECACHE. | ||
8913 | Only if there is no result in the NAMECACHE, GNS queries the DHT. | ||
8914 | The NAMECACHE stores data in the same (encrypted) format as the DHT. | ||
8915 | It thus makes no sense to iterate over all items in the | ||
8916 | NAMECACHE --- the NAMECACHE does not have a way to provide the keys | ||
8917 | required to decrypt the entries. | ||
8918 | |||
8919 | Blocks in the NAMECACHE share the same expiration mechanism as blocks in | ||
8920 | the DHT --- the block expires wheneever any of the records in | ||
8921 | the (encrypted) block expires. | ||
8922 | The expiration time of the block is the only information stored in | ||
8923 | plaintext. The NAMECACHE service internally performs all of the required | ||
8924 | work to expire blocks, clients do not have to worry about this. | ||
8925 | Also, given that NAMECACHE stores only GNS blocks that local users | ||
8926 | requested, there is no configuration option to limit the size of the | ||
8927 | NAMECACHE. It is assumed to be always small enough (a few MB) to fit on | ||
8928 | the drive. | ||
8929 | |||
8930 | The NAMECACHE supports the use of different database backends via a | ||
8931 | plugin API. | ||
8932 | |||
8933 | @menu | ||
8934 | * libgnunetnamecache:: | ||
8935 | * The NAMECACHE Client-Service Protocol:: | ||
8936 | * The NAMECACHE Plugin API:: | ||
8937 | @end menu | ||
8938 | |||
8939 | @node libgnunetnamecache | ||
8940 | @subsection libgnunetnamecache | ||
8941 | |||
8942 | |||
8943 | |||
8944 | The NAMECACHE API consists of five simple functions. First, there is | ||
8945 | @code{GNUNET_NAMECACHE_connect} to connect to the NAMECACHE service. | ||
8946 | This returns the handle required for all other operations on the | ||
8947 | NAMECACHE. Using @code{GNUNET_NAMECACHE_block_cache} clients can insert a | ||
8948 | block into the cache. | ||
8949 | @code{GNUNET_NAMECACHE_lookup_block} can be used to lookup blocks that | ||
8950 | were stored in the NAMECACHE. Both operations can be canceled using | ||
8951 | @code{GNUNET_NAMECACHE_cancel}. Note that canceling a | ||
8952 | @code{GNUNET_NAMECACHE_block_cache} operation can result in the block | ||
8953 | being stored in the NAMECACHE --- or not. Cancellation primarily ensures | ||
8954 | that the continuation function with the result of the operation will no | ||
8955 | longer be invoked. | ||
8956 | Finally, @code{GNUNET_NAMECACHE_disconnect} closes the connection to the | ||
8957 | NAMECACHE. | ||
8958 | |||
8959 | The maximum size of a block that can be stored in the NAMECACHE is | ||
8960 | @code{GNUNET_NAMECACHE_MAX_VALUE_SIZE}, which is defined to be 63 kB. | ||
8961 | |||
8962 | @node The NAMECACHE Client-Service Protocol | ||
8963 | @subsection The NAMECACHE Client-Service Protocol | ||
8964 | |||
8965 | |||
8966 | |||
8967 | All messages in the NAMECACHE IPC protocol start with the | ||
8968 | @code{struct GNUNET_NAMECACHE_Header} which adds a request | ||
8969 | ID (32-bit integer) to the standard message header. | ||
8970 | The request ID is used to match requests with the | ||
8971 | respective responses from the NAMECACHE, as they are allowed to happen | ||
8972 | out-of-order. | ||
8973 | |||
8974 | |||
8975 | @menu | ||
8976 | * Lookup:: | ||
8977 | * Store:: | ||
8978 | @end menu | ||
8979 | |||
8980 | @node Lookup | ||
8981 | @subsubsection Lookup | ||
8982 | |||
8983 | |||
8984 | |||
8985 | The @code{struct LookupBlockMessage} is used to lookup a block stored in | ||
8986 | the cache. | ||
8987 | It contains the query hash. The NAMECACHE always responds with a | ||
8988 | @code{struct LookupBlockResponseMessage}. If the NAMECACHE has no | ||
8989 | response, it sets the expiration time in the response to zero. | ||
8990 | Otherwise, the response is expected to contain the expiration time, the | ||
8991 | ECDSA signature, the derived key and the (variable-size) encrypted data | ||
8992 | of the block. | ||
8993 | |||
8994 | @node Store | ||
8995 | @subsubsection Store | ||
8996 | |||
8997 | |||
8998 | |||
8999 | The @code{struct BlockCacheMessage} is used to cache a block in the | ||
9000 | NAMECACHE. | ||
9001 | It has the same structure as the @code{struct LookupBlockResponseMessage}. | ||
9002 | The service responds with a @code{struct BlockCacheResponseMessage} which | ||
9003 | contains the result of the operation (success or failure). | ||
9004 | In the future, we might want to make it possible to provide an error | ||
9005 | message as well. | ||
9006 | |||
9007 | @node The NAMECACHE Plugin API | ||
9008 | @subsection The NAMECACHE Plugin API | ||
9009 | |||
9010 | |||
9011 | The NAMECACHE plugin API consists of two functions, @code{cache_block} to | ||
9012 | store a block in the database, and @code{lookup_block} to lookup a block | ||
9013 | in the database. | ||
9014 | |||
9015 | |||
9016 | @menu | ||
9017 | * Lookup2:: | ||
9018 | * Store2:: | ||
9019 | @end menu | ||
9020 | |||
9021 | @node Lookup2 | ||
9022 | @subsubsection Lookup2 | ||
9023 | |||
9024 | |||
9025 | |||
9026 | The @code{lookup_block} function is expected to return at most one block | ||
9027 | to the iterator, and return @code{GNUNET_NO} if there were no non-expired | ||
9028 | results. | ||
9029 | If there are multiple non-expired results in the cache, the lookup is | ||
9030 | supposed to return the result with the largest expiration time. | ||
9031 | |||
9032 | @node Store2 | ||
9033 | @subsubsection Store2 | ||
9034 | |||
9035 | |||
9036 | |||
9037 | The @code{cache_block} function is expected to try to store the block in | ||
9038 | the database, and return @code{GNUNET_SYSERR} if this was not possible | ||
9039 | for any reason. | ||
9040 | Furthermore, @code{cache_block} is expected to implicitly perform cache | ||
9041 | maintenance and purge blocks from the cache that have expired. Note that | ||
9042 | @code{cache_block} might encounter the case where the database already has | ||
9043 | another block stored under the same key. In this case, the plugin must | ||
9044 | ensure that the block with the larger expiration time is preserved. | ||
9045 | Obviously, this can done either by simply adding new blocks and selecting | ||
9046 | for the most recent expiration time during lookup, or by checking which | ||
9047 | block is more recent during the store operation. | ||
9048 | |||
9049 | @cindex REVOCATION Subsystem | ||
9050 | @node REVOCATION Subsystem | ||
9051 | @section REVOCATION Subsystem | ||
9052 | |||
9053 | |||
9054 | The REVOCATION subsystem is responsible for key revocation of Egos. | ||
9055 | If a user learns that their private key has been compromised or has lost | ||
9056 | it, they can use the REVOCATION system to inform all of the other users | ||
9057 | that their private key is no longer valid. | ||
9058 | The subsystem thus includes ways to query for the validity of keys and to | ||
9059 | propagate revocation messages. | ||
9060 | |||
9061 | @menu | ||
9062 | * Dissemination:: | ||
9063 | * Revocation Message Design Requirements:: | ||
9064 | * libgnunetrevocation:: | ||
9065 | * The REVOCATION Client-Service Protocol:: | ||
9066 | * The REVOCATION Peer-to-Peer Protocol:: | ||
9067 | @end menu | ||
9068 | |||
9069 | @node Dissemination | ||
9070 | @subsection Dissemination | ||
9071 | |||
9072 | |||
9073 | |||
9074 | When a revocation is performed, the revocation is first of all | ||
9075 | disseminated by flooding the overlay network. | ||
9076 | The goal is to reach every peer, so that when a peer needs to check if a | ||
9077 | key has been revoked, this will be purely a local operation where the | ||
9078 | peer looks at its local revocation list. Flooding the network is also the | ||
9079 | most robust form of key revocation --- an adversary would have to control | ||
9080 | a separator of the overlay graph to restrict the propagation of the | ||
9081 | revocation message. Flooding is also very easy to implement --- peers that | ||
9082 | receive a revocation message for a key that they have never seen before | ||
9083 | simply pass the message to all of their neighbours. | ||
9084 | |||
9085 | Flooding can only distribute the revocation message to peers that are | ||
9086 | online. | ||
9087 | In order to notify peers that join the network later, the revocation | ||
9088 | service performs efficient set reconciliation over the sets of known | ||
9089 | revocation messages whenever two peers (that both support REVOCATION | ||
9090 | dissemination) connect. | ||
9091 | The SET service is used to perform this operation efficiently. | ||
9092 | |||
9093 | @node Revocation Message Design Requirements | ||
9094 | @subsection Revocation Message Design Requirements | ||
9095 | |||
9096 | |||
9097 | |||
9098 | However, flooding is also quite costly, creating O(|E|) messages on a | ||
9099 | network with |E| edges. | ||
9100 | Thus, revocation messages are required to contain a proof-of-work, the | ||
9101 | result of an expensive computation (which, however, is cheap to verify). | ||
9102 | Only peers that have expended the CPU time necessary to provide | ||
9103 | this proof will be able to flood the network with the revocation message. | ||
9104 | This ensures that an attacker cannot simply flood the network with | ||
9105 | millions of revocation messages. The proof-of-work required by GNUnet is | ||
9106 | set to take days on a typical PC to compute; if the ability to quickly | ||
9107 | revoke a key is needed, users have the option to pre-compute revocation | ||
9108 | messages to store off-line and use instantly after their key has expired. | ||
9109 | |||
9110 | Revocation messages must also be signed by the private key that is being | ||
9111 | revoked. Thus, they can only be created while the private key is in the | ||
9112 | possession of the respective user. This is another reason to create a | ||
9113 | revocation message ahead of time and store it in a secure location. | ||
9114 | |||
9115 | @node libgnunetrevocation | ||
9116 | @subsection libgnunetrevocation | ||
9117 | |||
9118 | |||
9119 | |||
9120 | The REVOCATION API consists of two parts, to query and to issue | ||
9121 | revocations. | ||
9122 | |||
9123 | |||
9124 | @menu | ||
9125 | * Querying for revoked keys:: | ||
9126 | * Preparing revocations:: | ||
9127 | * Issuing revocations:: | ||
9128 | @end menu | ||
9129 | |||
9130 | @node Querying for revoked keys | ||
9131 | @subsubsection Querying for revoked keys | ||
9132 | |||
9133 | |||
9134 | |||
9135 | @code{GNUNET_REVOCATION_query} is used to check if a given ECDSA public | ||
9136 | key has been revoked. | ||
9137 | The given callback will be invoked with the result of the check. | ||
9138 | The query can be canceled using @code{GNUNET_REVOCATION_query_cancel} on | ||
9139 | the return value. | ||
9140 | |||
9141 | @node Preparing revocations | ||
9142 | @subsubsection Preparing revocations | ||
9143 | |||
9144 | |||
9145 | |||
9146 | It is often desirable to create a revocation record ahead-of-time and | ||
9147 | store it in an off-line location to be used later in an emergency. | ||
9148 | This is particularly true for GNUnet revocations, where performing the | ||
9149 | revocation operation itself is computationally expensive and thus is | ||
9150 | likely to take some time. | ||
9151 | Thus, if users want the ability to perform revocations quickly in an | ||
9152 | emergency, they must pre-compute the revocation message. | ||
9153 | The revocation API enables this with two functions that are used to | ||
9154 | compute the revocation message, but not trigger the actual revocation | ||
9155 | operation. | ||
9156 | |||
9157 | @code{GNUNET_REVOCATION_check_pow} should be used to calculate the | ||
9158 | proof-of-work required in the revocation message. This function takes the | ||
9159 | public key, the required number of bits for the proof of work (which in | ||
9160 | GNUnet is a network-wide constant) and finally a proof-of-work number as | ||
9161 | arguments. | ||
9162 | The function then checks if the given proof-of-work number is a valid | ||
9163 | proof of work for the given public key. Clients preparing a revocation | ||
9164 | are expected to call this function repeatedly (typically with a | ||
9165 | monotonically increasing sequence of numbers of the proof-of-work number) | ||
9166 | until a given number satisfies the check. | ||
9167 | That number should then be saved for later use in the revocation | ||
9168 | operation. | ||
9169 | |||
9170 | @code{GNUNET_REVOCATION_sign_revocation} is used to generate the | ||
9171 | signature that is required in a revocation message. | ||
9172 | It takes the private key that (possibly in the future) is to be revoked | ||
9173 | and returns the signature. | ||
9174 | The signature can again be saved to disk for later use, which will then | ||
9175 | allow performing a revocation even without access to the private key. | ||
9176 | |||
9177 | @node Issuing revocations | ||
9178 | @subsubsection Issuing revocations | ||
9179 | |||
9180 | |||
9181 | Given a ECDSA public key, the signature from @code{GNUNET_REVOCATION_sign} | ||
9182 | and the proof-of-work, | ||
9183 | @code{GNUNET_REVOCATION_revoke} can be used to perform the | ||
9184 | actual revocation. The given callback is called upon completion of the | ||
9185 | operation. @code{GNUNET_REVOCATION_revoke_cancel} can be used to stop the | ||
9186 | library from calling the continuation; however, in that case it is | ||
9187 | undefined whether or not the revocation operation will be executed. | ||
9188 | |||
9189 | @node The REVOCATION Client-Service Protocol | ||
9190 | @subsection The REVOCATION Client-Service Protocol | ||
9191 | |||
9192 | |||
9193 | The REVOCATION protocol consists of four simple messages. | ||
9194 | |||
9195 | A @code{QueryMessage} containing a public ECDSA key is used to check if a | ||
9196 | particular key has been revoked. The service responds with a | ||
9197 | @code{QueryResponseMessage} which simply contains a bit that says if the | ||
9198 | given public key is still valid, or if it has been revoked. | ||
9199 | |||
9200 | The second possible interaction is for a client to revoke a key by passing a | ||
9201 | @code{RevokeMessage} to the service. The @code{RevokeMessage} contains the | ||
9202 | ECDSA public key to be revoked, a signature by the corresponding private key | ||
9203 | and the proof-of-work. The service responds with a | ||
9204 | @code{RevocationResponseMessage} which can be used to indicate that the | ||
9205 | @code{RevokeMessage} was invalid (e.g. the proof of work is incorrect), or | ||
9206 | otherwise to indicate that the revocation has been processed successfully. | ||
9207 | |||
9208 | @node The REVOCATION Peer-to-Peer Protocol | ||
9209 | @subsection The REVOCATION Peer-to-Peer Protocol | ||
9210 | |||
9211 | |||
9212 | |||
9213 | Revocation uses two disjoint ways to spread revocation information among | ||
9214 | peers. | ||
9215 | First of all, P2P gossip exchanged via CORE-level neighbours is used to | ||
9216 | quickly spread revocations to all connected peers. | ||
9217 | Second, whenever two peers (that both support revocations) connect, | ||
9218 | the SET service is used to compute the union of the respective revocation | ||
9219 | sets. | ||
9220 | |||
9221 | In both cases, the exchanged messages are @code{RevokeMessage}s which | ||
9222 | contain the public key that is being revoked, a matching ECDSA signature, | ||
9223 | and a proof-of-work. | ||
9224 | Whenever a peer learns about a new revocation this way, it first | ||
9225 | validates the signature and the proof-of-work, then stores it to disk | ||
9226 | (typically to a file $GNUNET_DATA_HOME/revocation.dat) and finally | ||
9227 | spreads the information to all directly connected neighbours. | ||
9228 | |||
9229 | For computing the union using the SET service, the peer with the smaller | ||
9230 | hashed peer identity will connect (as a "client" in the two-party set | ||
9231 | protocol) to the other peer after one second (to reduce traffic spikes | ||
9232 | on connect) and initiate the computation of the set union. | ||
9233 | All revocation services use a common hash to identify the SET operation | ||
9234 | over revocation sets. | ||
9235 | |||
9236 | The current implementation accepts revocation set union operations from | ||
9237 | all peers at any time; however, well-behaved peers should only initiate | ||
9238 | this operation once after establishing a connection to a peer with a | ||
9239 | larger hashed peer identity. | ||
9240 | |||
9241 | @cindex FS | ||
9242 | @cindex FS Subsystem | ||
9243 | @node File-sharing (FS) Subsystem | ||
9244 | @section File-sharing (FS) Subsystem | ||
9245 | |||
9246 | |||
9247 | |||
9248 | This chapter describes the details of how the file-sharing service works. | ||
9249 | As with all services, it is split into an API (libgnunetfs), the service | ||
9250 | process (gnunet-service-fs) and user interface(s). | ||
9251 | The file-sharing service uses the datastore service to store blocks and | ||
9252 | the DHT (and indirectly datacache) for lookups for non-anonymous | ||
9253 | file-sharing. | ||
9254 | Furthermore, the file-sharing service uses the block library (and the | ||
9255 | block fs plugin) for validation of DHT operations. | ||
9256 | |||
9257 | In contrast to many other services, libgnunetfs is rather complex since | ||
9258 | the client library includes a large number of high-level abstractions; | ||
9259 | this is necessary since the Fs service itself largely only operates on | ||
9260 | the block level. | ||
9261 | The FS library is responsible for providing a file-based abstraction to | ||
9262 | applications, including directories, meta data, keyword search, | ||
9263 | verification, and so on. | ||
9264 | |||
9265 | The method used by GNUnet to break large files into blocks and to use | ||
9266 | keyword search is called the | ||
9267 | "Encoding for Censorship Resistant Sharing" (ECRS). | ||
9268 | ECRS is largely implemented in the fs library; block validation is also | ||
9269 | reflected in the block FS plugin and the FS service. | ||
9270 | ECRS on-demand encoding is implemented in the FS service. | ||
9271 | |||
9272 | NOTE: The documentation in this chapter is quite incomplete. | ||
9273 | |||
9274 | @menu | ||
9275 | * Encoding for Censorship-Resistant Sharing (ECRS):: | ||
9276 | * File-sharing persistence directory structure:: | ||
9277 | @end menu | ||
9278 | |||
9279 | @cindex ECRS | ||
9280 | @cindex Encoding for Censorship-Resistant Sharing | ||
9281 | @node Encoding for Censorship-Resistant Sharing (ECRS) | ||
9282 | @subsection Encoding for Censorship-Resistant Sharing (ECRS) | ||
9283 | |||
9284 | |||
9285 | |||
9286 | When GNUnet shares files, it uses a content encoding that is called ECRS, | ||
9287 | the Encoding for Censorship-Resistant Sharing. | ||
9288 | Most of ECRS is described in the (so far unpublished) research paper | ||
9289 | attached to this page. ECRS obsoletes the previous ESED and ESED II | ||
9290 | encodings which were used in GNUnet before version 0.7.0. | ||
9291 | The rest of this page assumes that the reader is familiar with the | ||
9292 | attached paper. What follows is a description of some minor extensions | ||
9293 | that GNUnet makes over what is described in the paper. | ||
9294 | The reason why these extensions are not in the paper is that we felt | ||
9295 | that they were obvious or trivial extensions to the original scheme and | ||
9296 | thus did not warrant space in the research report. | ||
9297 | |||
9298 | @menu | ||
9299 | * Namespace Advertisements:: | ||
9300 | * KSBlocks:: | ||
9301 | @end menu | ||
9302 | |||
9303 | @node Namespace Advertisements | ||
9304 | @subsubsection Namespace Advertisements | ||
9305 | |||
9306 | |||
9307 | @c %**FIXME: all zeroses -> ? | ||
9308 | |||
9309 | An @code{SBlock} with identifier all zeros is a signed | ||
9310 | advertisement for a namespace. This special @code{SBlock} contains | ||
9311 | metadata describing the content of the namespace. | ||
9312 | Instead of the name of the identifier for a potential update, it contains | ||
9313 | the identifier for the root of the namespace. | ||
9314 | The URI should always be empty. The @code{SBlock} is signed with the | ||
9315 | content provider's RSA private key (just like any other SBlock). Peers | ||
9316 | can search for @code{SBlock}s in order to find out more about a namespace. | ||
9317 | |||
9318 | @node KSBlocks | ||
9319 | @subsubsection KSBlocks | ||
9320 | |||
9321 | |||
9322 | |||
9323 | GNUnet implements @code{KSBlocks} which are @code{KBlocks} that, instead | ||
9324 | of encrypting a CHK and metadata, encrypt an @code{SBlock} instead. | ||
9325 | In other words, @code{KSBlocks} enable GNUnet to find @code{SBlocks} | ||
9326 | using the global keyword search. | ||
9327 | Usually the encrypted @code{SBlock} is a namespace advertisement. | ||
9328 | The rationale behind @code{KSBlock}s and @code{SBlock}s is to enable | ||
9329 | peers to discover namespaces via keyword searches, and, to associate | ||
9330 | useful information with namespaces. When GNUnet finds @code{KSBlocks} | ||
9331 | during a normal keyword search, it adds the information to an internal | ||
9332 | list of discovered namespaces. Users looking for interesting namespaces | ||
9333 | can then inspect this list, reducing the need for out-of-band discovery | ||
9334 | of namespaces. | ||
9335 | Naturally, namespaces (or more specifically, namespace advertisements) can | ||
9336 | also be referenced from directories, but @code{KSBlock}s should make it | ||
9337 | easier to advertise namespaces for the owner of the pseudonym since they | ||
9338 | eliminate the need to first create a directory. | ||
9339 | |||
9340 | Collections are also advertised using @code{KSBlock}s. | ||
9341 | |||
9342 | @c https://old.gnunet.org/sites/default/files/ecrs.pdf | ||
9343 | |||
9344 | @node File-sharing persistence directory structure | ||
9345 | @subsection File-sharing persistence directory structure | ||
9346 | |||
9347 | |||
9348 | |||
9349 | This section documents how the file-sharing library implements | ||
9350 | persistence of file-sharing operations and specifically the resulting | ||
9351 | directory structure. | ||
9352 | This code is only active if the @code{GNUNET_FS_FLAGS_PERSISTENCE} flag | ||
9353 | was set when calling @code{GNUNET_FS_start}. | ||
9354 | In this case, the file-sharing library will try hard to ensure that all | ||
9355 | major operations (searching, downloading, publishing, unindexing) are | ||
9356 | persistent, that is, can live longer than the process itself. | ||
9357 | More specifically, an operation is supposed to live until it is | ||
9358 | explicitly stopped. | ||
9359 | |||
9360 | If @code{GNUNET_FS_stop} is called before an operation has been stopped, a | ||
9361 | @code{SUSPEND} event is generated and then when the process calls | ||
9362 | @code{GNUNET_FS_start} next time, a @code{RESUME} event is generated. | ||
9363 | Additionally, even if an application crashes (segfault, SIGKILL, system | ||
9364 | crash) and hence @code{GNUNET_FS_stop} is never called and no | ||
9365 | @code{SUSPEND} events are generated, operations are still resumed (with | ||
9366 | @code{RESUME} events). | ||
9367 | This is implemented by constantly writing the current state of the | ||
9368 | file-sharing operations to disk. | ||
9369 | Specifically, the current state is always written to disk whenever | ||
9370 | anything significant changes (the exception are block-wise progress in | ||
9371 | publishing and unindexing, since those operations would be slowed down | ||
9372 | significantly and can be resumed cheaply even without detailed | ||
9373 | accounting). | ||
9374 | Note that if the process crashes (or is killed) during a serialization | ||
9375 | operation, FS does not guarantee that this specific operation is | ||
9376 | recoverable (no strict transactional semantics, again for performance | ||
9377 | reasons). However, all other unrelated operations should resume nicely. | ||
9378 | |||
9379 | Since we need to serialize the state continuously and want to recover as | ||
9380 | much as possible even after crashing during a serialization operation, | ||
9381 | we do not use one large file for serialization. | ||
9382 | Instead, several directories are used for the various operations. | ||
9383 | When @code{GNUNET_FS_start} executes, the master directories are scanned | ||
9384 | for files describing operations to resume. | ||
9385 | Sometimes, these operations can refer to related operations in child | ||
9386 | directories which may also be resumed at this point. | ||
9387 | Note that corrupted files are cleaned up automatically. | ||
9388 | However, dangling files in child directories (those that are not | ||
9389 | referenced by files from the master directories) are not automatically | ||
9390 | removed. | ||
9391 | |||
9392 | Persistence data is kept in a directory that begins with the "STATE_DIR" | ||
9393 | prefix from the configuration file | ||
9394 | (by default, "$SERVICEHOME/persistence/") followed by the name of the | ||
9395 | client as given to @code{GNUNET_FS_start} (for example, "gnunet-gtk") | ||
9396 | followed by the actual name of the master or child directory. | ||
9397 | |||
9398 | The names for the master directories follow the names of the operations: | ||
9399 | |||
9400 | @itemize @bullet | ||
9401 | @item "search" | ||
9402 | @item "download" | ||
9403 | @item "publish" | ||
9404 | @item "unindex" | ||
9405 | @end itemize | ||
9406 | |||
9407 | Each of the master directories contains names (chosen at random) for each | ||
9408 | active top-level (master) operation. | ||
9409 | Note that a download that is associated with a search result is not a | ||
9410 | top-level operation. | ||
9411 | |||
9412 | In contrast to the master directories, the child directories are only | ||
9413 | consulted when another operation refers to them. | ||
9414 | For each search, a subdirectory (named after the master search | ||
9415 | synchronization file) contains the search results. | ||
9416 | Search results can have an associated download, which is then stored in | ||
9417 | the general "download-child" directory. | ||
9418 | Downloads can be recursive, in which case children are stored in | ||
9419 | subdirectories mirroring the structure of the recursive download | ||
9420 | (either starting in the master "download" directory or in the | ||
9421 | "download-child" directory depending on how the download was initiated). | ||
9422 | For publishing operations, the "publish-file" directory contains | ||
9423 | information about the individual files and directories that are part of | ||
9424 | the publication. | ||
9425 | However, this directory structure is flat and does not mirror the | ||
9426 | structure of the publishing operation. | ||
9427 | Note that unindex operations cannot have associated child operations. | ||
9428 | |||
9429 | @cindex REGEX subsystem | ||
9430 | @node REGEX Subsystem | ||
9431 | @section REGEX Subsystem | ||
9432 | |||
9433 | |||
9434 | |||
9435 | Using the REGEX subsystem, you can discover peers that offer a particular | ||
9436 | service using regular expressions. | ||
9437 | The peers that offer a service specify it using a regular expressions. | ||
9438 | Peers that want to patronize a service search using a string. | ||
9439 | The REGEX subsystem will then use the DHT to return a set of matching | ||
9440 | offerers to the patrons. | ||
9441 | |||
9442 | For the technical details, we have Max's defense talk and Max's Master's | ||
9443 | thesis. | ||
9444 | |||
9445 | @c An additional publication is under preparation and available to | ||
9446 | @c team members (in Git). | ||
9447 | @c FIXME: Where is the file? Point to it. Assuming that it's szengel2012ms | ||
9448 | |||
9449 | @menu | ||
9450 | * How to run the regex profiler:: | ||
9451 | @end menu | ||
9452 | |||
9453 | @node How to run the regex profiler | ||
9454 | @subsection How to run the regex profiler | ||
9455 | |||
9456 | |||
9457 | |||
9458 | The gnunet-regex-profiler can be used to profile the usage of mesh/regex | ||
9459 | for a given set of regular expressions and strings. | ||
9460 | Mesh/regex allows you to announce your peer ID under a certain regex and | ||
9461 | search for peers matching a particular regex using a string. | ||
9462 | See @uref{https://bib.gnunet.org/full/date.html#2012_5f2, szengel2012ms} for a full | ||
9463 | introduction. | ||
9464 | |||
9465 | First of all, the regex profiler uses GNUnet testbed, thus all the | ||
9466 | implications for testbed also apply to the regex profiler | ||
9467 | (for example you need password-less ssh login to the machines listed in | ||
9468 | your hosts file). | ||
9469 | |||
9470 | @strong{Configuration} | ||
9471 | |||
9472 | Moreover, an appropriate configuration file is needed. | ||
9473 | In the following paragraph the important details are highlighted. | ||
9474 | |||
9475 | Announcing of the regular expressions is done by the | ||
9476 | gnunet-daemon-regexprofiler, therefore you have to make sure it is | ||
9477 | started, by adding it to the START_ON_DEMAND set of ARM: | ||
9478 | |||
9479 | @example | ||
9480 | [regexprofiler] | ||
9481 | START_ON_DEMAND = YES | ||
9482 | @end example | ||
9483 | |||
9484 | @noindent | ||
9485 | Furthermore you have to specify the location of the binary: | ||
9486 | |||
9487 | @example | ||
9488 | [regexprofiler] | ||
9489 | # Location of the gnunet-daemon-regexprofiler binary. | ||
9490 | BINARY = /home/szengel/gnunet/src/mesh/.libs/gnunet-daemon-regexprofiler | ||
9491 | # Regex prefix that will be applied to all regular expressions and | ||
9492 | # search string. | ||
9493 | REGEX_PREFIX = "GNVPN-0001-PAD" | ||
9494 | @end example | ||
9495 | |||
9496 | @noindent | ||
9497 | When running the profiler with a large scale deployment, you probably | ||
9498 | want to reduce the workload of each peer. | ||
9499 | Use the following options to do this. | ||
9500 | |||
9501 | @example | ||
9502 | [dht] | ||
9503 | # Force network size estimation | ||
9504 | FORCE_NSE = 1 | ||
9505 | |||
9506 | [dhtcache] | ||
9507 | DATABASE = heap | ||
9508 | # Disable RC-file for Bloom filter? (for benchmarking with limited IO | ||
9509 | # availability) | ||
9510 | DISABLE_BF_RC = YES | ||
9511 | # Disable Bloom filter entirely | ||
9512 | DISABLE_BF = YES | ||
9513 | |||
9514 | [nse] | ||
9515 | # Minimize proof-of-work CPU consumption by NSE | ||
9516 | WORKBITS = 1 | ||
9517 | @end example | ||
9518 | |||
9519 | @noindent | ||
9520 | @strong{Options} | ||
9521 | |||
9522 | To finally run the profiler some options and the input data need to be | ||
9523 | specified on the command line. | ||
9524 | |||
9525 | @example | ||
9526 | gnunet-regex-profiler -c config-file -d log-file -n num-links \ | ||
9527 | -p path-compression-length -s search-delay -t matching-timeout \ | ||
9528 | -a num-search-strings hosts-file policy-dir search-strings-file | ||
9529 | @end example | ||
9530 | |||
9531 | @noindent | ||
9532 | Where... | ||
9533 | |||
9534 | @itemize @bullet | ||
9535 | @item ... @code{config-file} means the configuration file created earlier. | ||
9536 | @item ... @code{log-file} is the file where to write statistics output. | ||
9537 | @item ... @code{num-links} indicates the number of random links between | ||
9538 | started peers. | ||
9539 | @item ... @code{path-compression-length} is the maximum path compression | ||
9540 | length in the DFA. | ||
9541 | @item ... @code{search-delay} time to wait between peers finished linking | ||
9542 | and starting to match strings. | ||
9543 | @item ... @code{matching-timeout} timeout after which to cancel the | ||
9544 | searching. | ||
9545 | @item ... @code{num-search-strings} number of strings in the | ||
9546 | search-strings-file. | ||
9547 | @item ... the @code{hosts-file} should contain a list of hosts for the | ||
9548 | testbed, one per line in the following format: | ||
9549 | |||
9550 | @itemize @bullet | ||
9551 | @item @code{user@@host_ip:port} | ||
9552 | @end itemize | ||
9553 | @item ... the @code{policy-dir} is a folder containing text files | ||
9554 | containing one or more regular expressions. A peer is started for each | ||
9555 | file in that folder and the regular expressions in the corresponding file | ||
9556 | are announced by this peer. | ||
9557 | @item ... the @code{search-strings-file} is a text file containing search | ||
9558 | strings, one in each line. | ||
9559 | @end itemize | ||
9560 | |||
9561 | @noindent | ||
9562 | You can create regular expressions and search strings for every AS in the | ||
9563 | Internet using the attached scripts. You need one of the | ||
9564 | @uref{http://data.caida.org/datasets/routing/routeviews-prefix2as/, CAIDA routeviews prefix2as} | ||
9565 | data files for this. Run | ||
9566 | |||
9567 | @example | ||
9568 | create_regex.py <filename> <output path> | ||
9569 | @end example | ||
9570 | |||
9571 | @noindent | ||
9572 | to create the regular expressions and | ||
9573 | |||
9574 | @example | ||
9575 | create_strings.py <input path> <outfile> | ||
9576 | @end example | ||
9577 | |||
9578 | @noindent | ||
9579 | to create a search strings file from the previously created | ||
9580 | regular expressions. | ||
9581 | |||
9582 | @cindex REST subsystem | ||
9583 | @node REST Subsystem | ||
9584 | @section REST Subsystem | ||
9585 | |||
9586 | |||
9587 | |||
9588 | Using the REST subsystem, you can expose REST-based APIs or services. | ||
9589 | The REST service is designed as a pluggable architecture. | ||
9590 | To create a new REST endpoint, simply add a library in the form | ||
9591 | ``plugin_rest_*''. | ||
9592 | The REST service will automatically load all REST plugins on startup. | ||
9593 | |||
9594 | @strong{Configuration} | ||
9595 | |||
9596 | The REST service can be configured in various ways. | ||
9597 | The reference config file can be found in | ||
9598 | @file{src/rest/rest.conf}: | ||
9599 | @example | ||
9600 | [rest] | ||
9601 | REST_PORT=7776 | ||
9602 | REST_ALLOW_HEADERS=Authorization,Accept,Content-Type | ||
9603 | REST_ALLOW_ORIGIN=* | ||
9604 | REST_ALLOW_CREDENTIALS=true | ||
9605 | @end example | ||
9606 | |||
9607 | The port as well as | ||
9608 | @deffn{cross-origin resource sharing} (CORS) | ||
9609 | @end deffn | ||
9610 | headers that are supposed to be advertised by the rest service are | ||
9611 | configurable. | ||
9612 | |||
9613 | @menu | ||
9614 | * Namespace considerations:: | ||
9615 | * Endpoint documentation:: | ||
9616 | @end menu | ||
9617 | |||
9618 | @node Namespace considerations | ||
9619 | @subsection Namespace considerations | ||
9620 | |||
9621 | The @command{gnunet-rest-service} will load all plugins that are installed. | ||
9622 | As such it is important that the endpoint namespaces do not clash. | ||
9623 | |||
9624 | For example, plugin X might expose the endpoint ``/xxx'' while plugin Y | ||
9625 | exposes endpoint ``/xxx/yyy''. | ||
9626 | This is a problem if plugin X is also supposed to handle a call | ||
9627 | to ``/xxx/yyy''. | ||
9628 | Currently the REST service will not complain or warn about such clashes, | ||
9629 | so please make sure that endpoints are unambiguous. | ||
9630 | |||
9631 | @node Endpoint documentation | ||
9632 | @subsection Endpoint documentation | ||
9633 | |||
9634 | This is WIP. Endpoints should be documented appropriately. | ||
9635 | Preferably using annotations. | ||
9636 | |||
9637 | |||
9638 | @cindex RPS Subsystem | ||
9639 | @node RPS Subsystem | ||
9640 | @section RPS Subsystem | ||
9641 | |||
9642 | In literature, Random Peer Sampling (RPS) refers to the problem of | ||
9643 | reliably@footnote{"Reliable" in this context means having no bias, | ||
9644 | neither spatial, nor temporal, nor through malicious activity.} drawing | ||
9645 | random samples from an unstructured p2p network. | ||
9646 | |||
9647 | Doing so in a reliable manner is not only hard because of inherent | ||
9648 | problems but also because of possible malicious peers that could try to | ||
9649 | bias the selection. | ||
9650 | |||
9651 | It is useful for all kind of gossip protocols that require the selection | ||
9652 | of random peers in the whole network like gathering statistics, | ||
9653 | spreading and aggregating information in the network, load balancing and | ||
9654 | overlay topology management. | ||
9655 | |||
9656 | The approach chosen in the RPS service implementation in GNUnet follows | ||
9657 | the @uref{https://bib.gnunet.org/full/date.html\#2009_5f0, Brahms} | ||
9658 | design. | ||
9659 | |||
9660 | The current state is "work in progress". There are a lot of things that | ||
9661 | need to be done, primarily finishing the experimental evaluation and a | ||
9662 | re-design of the API. | ||
9663 | |||
9664 | The abstract idea is to subscribe to connect to/start the RPS service | ||
9665 | and request random peers that will be returned when they represent a | ||
9666 | random selection from the whole network with high probability. | ||
9667 | |||
9668 | An additional feature to the original Brahms-design is the selection of | ||
9669 | sub-groups: The GNUnet implementation of RPS enables clients to ask for | ||
9670 | random peers from a group that is defined by a common shared secret. | ||
9671 | (The secret could of course also be public, depending on the use-case.) | ||
9672 | |||
9673 | Another addition to the original protocol was made: The sampler | ||
9674 | mechanism that was introduced in Brahms was slightly adapted and used to | ||
9675 | actually sample the peers and returned to the client. | ||
9676 | This is necessary as the original design only keeps peers connected to | ||
9677 | random other peers in the network. In order to return random peers to | ||
9678 | client requests independently random, they cannot be drawn from the | ||
9679 | connected peers. | ||
9680 | The adapted sampler makes sure that each request for random peers is | ||
9681 | independent from the others. | ||
9682 | |||
9683 | @menu | ||
9684 | * Brahms:: | ||
9685 | @end menu | ||
9686 | |||
9687 | @node Brahms | ||
9688 | @subsection Brahms | ||
9689 | The high-level concept of Brahms is two-fold: Combining push-pull gossip | ||
9690 | with locally fixing a assumed bias using cryptographic min-wise | ||
9691 | permutations. | ||
9692 | The central data structure is the view - a peer's current local sample. | ||
9693 | This view is used to select peers to push to and pull from. | ||
9694 | This simple mechanism can be biased easily. For this reason Brahms | ||
9695 | 'fixes' the bias by using the so-called sampler. A data structure that | ||
9696 | takes a list of elements as input and outputs a random one of them | ||
9697 | independently of the frequency in the input set. Both an element that | ||
9698 | was put into the sampler a single time and an element that was put into | ||
9699 | it a million times have the same probability of being the output. | ||
9700 | This is achieved with exploiting min-wise independent | ||
9701 | permutations. | ||
9702 | In the RPS service we use HMACs: On the initialisation of a sampler | ||
9703 | element, a key is chosen at random. On each input the HMAC with the | ||
9704 | random key is computed. The sampler element keeps the element with the | ||
9705 | minimal HMAC. | ||
9706 | |||
9707 | In order to fix the bias in the view, a fraction of the elements in the | ||
9708 | view are sampled through the sampler from the random stream of peer IDs. | ||
9709 | |||
9710 | According to the theoretical analysis of Bortnikov et al. this suffices | ||
9711 | to keep the network connected and having random peers in the view. | ||
9712 | |||
9713 | @cindex TRANSPORT-NG Subsystem | ||
9714 | @node TRANSPORT-NG Subsystem | ||
9715 | @section TRANSPORT-NG Subsystem | ||
9716 | |||
9717 | The current GNUnet TRANSPORT architecture is rooted in the GNUnet 0.4 design | ||
9718 | of using plugins for the actual transmission operations and the ATS subsystem | ||
9719 | to select a plugin and allocate bandwidth. The following key issues have been | ||
9720 | identified with this design: | ||
9721 | |||
9722 | @itemize @bullet | ||
9723 | @item Bugs in one plugin can affect the TRANSPORT service and other plugins. | ||
9724 | There is at least one open bug that affects sockets, where the origin is | ||
9725 | difficult to pinpoint due to the large code base. | ||
9726 | @item Relevant operating system default configurations often impose a limit of | ||
9727 | 1024 file descriptors per process. Thus, one plugin may impact other | ||
9728 | plugin's connectivity choices. | ||
9729 | @item Plugins are required to offer bi-directional connectivity. However, | ||
9730 | firewalls (incl. NAT boxes) and physical environments sometimes only | ||
9731 | allow uni-directional connectivity, which then currently cannot be | ||
9732 | utilized at all. | ||
9733 | @item Distance vector routing was implemented in 209 but shortly afterwards | ||
9734 | broken and due to the complexity of implementing it as a plugin and | ||
9735 | dealing with the resource allocation consequences was never useful. | ||
9736 | @item Most existing plugins communicate completely using cleartext, exposing | ||
9737 | metad data (message size) and making it easy to fingerprint and | ||
9738 | possibly block GNUnet traffic. | ||
9739 | @item Various NAT traversal methods are not supported. | ||
9740 | @item The service logic is cluttered with "manipulation" support code for | ||
9741 | TESTBED to enable faking network characteristics like lossy connections | ||
9742 | or firewewalls. | ||
9743 | @item Bandwidth allocation is done in ATS, requiring the duplication of state | ||
9744 | and resulting in much delayed allocation decisions. As a result, | ||
9745 | often available bandwidth goes unused. Users are expected to manually | ||
9746 | configure bandwidth limits, instead of TRANSPORT using congestion control | ||
9747 | to adapt automatically. | ||
9748 | @item TRANSPORT is difficult to test and has bad test coverage. | ||
9749 | @item HELLOs include an absolute expiration time. Nodes with unsynchronized | ||
9750 | clocks cannot connect. | ||
9751 | @item Displaying the contents of a HELLO requires the respective plugin as the | ||
9752 | plugin-specific data is encoded in binary. This also complicates logging. | ||
9753 | @end itemize | ||
9754 | |||
9755 | |||
9756 | @menu | ||
9757 | * Design goals of TNG:: | ||
9758 | * HELLO-NG:: | ||
9759 | * Priorities and preferences:: | ||
9760 | * Communicators:: | ||
9761 | @end menu | ||
9762 | |||
9763 | @node Design goals of TNG | ||
9764 | @subsection Design goals of TNG | ||
9765 | |||
9766 | In order to address the above issues, we want to: | ||
9767 | |||
9768 | @itemize @bullet | ||
9769 | @item Move plugins into separate processes which we shall call | ||
9770 | @emph{communicators}. Communicators connect as clients to the transport | ||
9771 | service. | ||
9772 | @item TRANSPORT should be able to utilize any number of communcators to the | ||
9773 | same peer at the same time. | ||
9774 | @item TRANSPORT should be responsible for fragmentation, retransmission, | ||
9775 | flow- and congestion-control. Users should no longer have to configure | ||
9776 | bandwidth limits: TRANSPORT should detect what is available and use it. | ||
9777 | @item Commnunicators should be allowed to be uni-directional and unreliable. | ||
9778 | TRANSPORT shall create bi-directional channels from this whenever | ||
9779 | possible. | ||
9780 | @item DV should no longer be a plugin, but part of TRANSPORT. | ||
9781 | @item TRANSPORT should provide communicators help communicating, for example | ||
9782 | in the case of uni-directional communicators or the need for out-of-band | ||
9783 | signalling for NAT traversal. We call this functionality | ||
9784 | @emph{backchannels}. | ||
9785 | @item Transport manipulation should be signalled to CORE on a per-message basis | ||
9786 | instead of an approximate bandwidth. | ||
9787 | @item CORE should signal performance requirements (reliability, latency, etc.) | ||
9788 | on a per-message basis to TRANSPORT. If possible, TRANSPORT should | ||
9789 | consider those options when scheduling messages for transmission. | ||
9790 | @item HELLOs should be in a humman-readable format with monotonic time | ||
9791 | expirations. | ||
9792 | @end itemize | ||
9793 | |||
9794 | The new architecture is planned as follows: | ||
9795 | |||
9796 | |||
9797 | @image{images/tng,5in,,TNG architecture.} | ||
9798 | |||
9799 | TRANSPORT's main objective is to establish bi-directional virtual links using a | ||
9800 | variety of possibly uni-directional communicators. Links undergo the following | ||
9801 | steps: | ||
9802 | |||
9803 | @enumerate | ||
9804 | @item Communicator informs TRANSPORT A that a queue (direct neighbour) is | ||
9805 | available, or equivalently TRANSPORT A discovers a (DV) path to a target | ||
9806 | B. | ||
9807 | @item TRANSPORT A sends a challenge to the target peer, trying to confirm that | ||
9808 | the peer can receive. FIXME: This is not implemented properly for DV. | ||
9809 | Here we should really take a validated DVH and send a challenge exactly | ||
9810 | down that path! | ||
9811 | @item The other TRANSPORT, TRANSPORT B, receives the challenge, and sends back | ||
9812 | a response, possibly using a dierent path. If TRANSPORT B does not yet | ||
9813 | have a virtual link to A, it must try to establish a virtual link. | ||
9814 | @item Upon receiving the response, TRANSPORT A creates the virtual link. If the | ||
9815 | response included a challenge, TRANSPORT A must respond to this challenge | ||
9816 | as well, eectively re-creating the TCP 3-way handshake (just with longer | ||
9817 | challenge values). | ||
9818 | @end enumerate | ||
9819 | |||
9820 | @node HELLO-NG | ||
9821 | @subsection HELLO-NG | ||
9822 | |||
9823 | HELLOs change in three ways. First of all, communicators encode the respective | ||
9824 | addresses in a human-readable URL-like string. This way, we do no longer | ||
9825 | require the communicator to print the contents of a HELLO. | ||
9826 | Second, HELLOs no longer contain an expiration time, only a creation time. | ||
9827 | The receiver must only compare the respective absolute values. So given a HELLO | ||
9828 | from the same sender with a larger creation time, then the old one is no longer | ||
9829 | valid. This also obsoletes the need for the gnunet-hello binary to set HELLO | ||
9830 | expiration times to never. | ||
9831 | Third, a peer no longer generates one big HELLO that always contains all of the | ||
9832 | addresses. Instead, each address is signed individually and shared only over | ||
9833 | the address scopes where it makes sense to share the address. In particular, | ||
9834 | care should be taken to not share MACs across the Internet and confine their | ||
9835 | use to the LAN. | ||
9836 | As each address is signed separately, having multiple addresses valid at the | ||
9837 | same time (given the new creation time expiration logic) requires that those | ||
9838 | addresses must have exactly the same creation time. | ||
9839 | Whenever that monotonic time is increased, all addresses must be re-signed and | ||
9840 | re-distributed. | ||
9841 | |||
9842 | @node Priorities and preferences | ||
9843 | @subsection Priorities and preferences | ||
9844 | |||
9845 | In the new design, TRANSPORT adopts a feature (which was previously already | ||
9846 | available in CORE) of the MQ API to allow applications to specify priorities and | ||
9847 | preferences per message (or rather, per MQ envelope). | ||
9848 | The (updated) MQ API allows applications to specify one of four priority levels | ||
9849 | as well as desired preferences for transmission by setting options on an | ||
9850 | envelope. These preferences currently are: | ||
9851 | |||
9852 | @itemize @bullet | ||
9853 | @item GNUNET_MQ_PREF_UNRELIABLE: Disables TRANSPORT waiting for ACKS on | ||
9854 | unreliable channels like UDP. Now it is fire and forget. These messages | ||
9855 | then cannot be used for RTT estimates either. | ||
9856 | @item GNUNET_MQ_PREF_LOW_LATENCY: Directs TRANSPORT to select the | ||
9857 | lowest-latency transmission choices possible. | ||
9858 | @item GNUNET_MQ_PREF_CORK_ALLOWED: Allows TRANSPORT to delay transmission to | ||
9859 | group the message with other messages into a larger batch to reduce the | ||
9860 | number of packets sent. | ||
9861 | @item GNUNET_MQ_PREF_GOODPUT: Directs TRANSPORT to select the highest goodput | ||
9862 | channel available. | ||
9863 | @item GNUNET_MQ_PREF_OUT_OF_ORDER: Allows TRANSPORT to reorder the messages as | ||
9864 | it sees fit, otherwise TRANSPORT should attempt to preserve transmission | ||
9865 | order. | ||
9866 | @end itemize | ||
9867 | |||
9868 | Each MQ envelope is always able to store those options (and the priority), and | ||
9869 | in the future this uniform API will be used by TRANSPORT, CORE, CADET and | ||
9870 | possibly other subsystems that send messages (like LAKE). | ||
9871 | When CORE sets preferences and priorities, it is supposed to respect the | ||
9872 | preferences and priorities it is given from higher layers. Similarly, CADET also | ||
9873 | simply passes on the preferences and priorities of the layer above CADET. When a | ||
9874 | layer combines multiple smaller messages into one larger transmission, the | ||
9875 | @code{GNUNET_MQ_env_combine_options()} should be used to calculate options for | ||
9876 | the combined message. We note that the exact semantics of the options may differ | ||
9877 | by layer. For example, CADET will always strictly implement reliable and | ||
9878 | in-order delivery of messages, while the same options are only advisory for | ||
9879 | TRANSPORT and CORE: they should try (using ACKs on unreliable communicators, | ||
9880 | not changing the message order themselves), but if messages are lost anyway | ||
9881 | (e.g. because a TCP is dropped in the middle), or if messages are reordered | ||
9882 | (e.g. because they took different paths over the network and arrived in a | ||
9883 | different order) TRANSPORT and CORE do not have to correct this. Whether a | ||
9884 | preference is strict or loose is thus dened by the respective layer. | ||
9885 | |||
9886 | @node Communicators | ||
9887 | @subsection Communicators | ||
9888 | |||
9889 | The API for communicators is defined in | ||
9890 | @code{gnunet_transport_communication_service.h}. | ||
9891 | Each communicator must specify its (global) communication characteristics, which | ||
9892 | for now only say whether the communication is reliable (e.g. TCP, HTTPS) or | ||
9893 | unreliable (e.g. UDP, WLAN). Each communicator must specify a unique address | ||
9894 | prex, or NULL if the communicator cannot establish outgoing connections | ||
9895 | (for example because it is only acting as a TCP server). | ||
9896 | A communicator must tell TRANSPORT which addresses it is reachable under. | ||
9897 | Addresses may be added or removed at any time. A communicator may have zero | ||
9898 | addresses (transmission only). | ||
9899 | Addresses do not have to match the address prefix. | ||
9900 | |||
9901 | TRANSPORT may ask a communicator to try to connect to another address. | ||
9902 | TRANSPORT will only ask for connections where the address matches the | ||
9903 | communicator's address prefix that was provided when the connection was | ||
9904 | established. Communicators should then attempt to establish a connection. | ||
9905 | No response is provided to TRANSPORT service on failure. The TRANSPORT service | ||
9906 | has to ask the communicator explicitly to retry. | ||
9907 | |||
9908 | If a communicator succeeds in establishing an outgoing connection for | ||
9909 | transmission, or if a communicator receives an incoming bi-directional | ||
9910 | connection, the communicator must inform the TRANSPORT service that a message | ||
9911 | queue (MQ) for transmission is now available. For that MQ, the communicator must | ||
9912 | provide the peer identity claimed by the other end, a human-readable address | ||
9913 | (for debugging) and a maximum transfer unit (MTU). A MTU of zero means sending | ||
9914 | is not supported, SIZE_MAX should be used for no MTU. The communicator should | ||
9915 | also tell TRANSPORT what network type is used for the queue. The communicator | ||
9916 | may tell TRANSPORT anytime that the queue was deleted and is no longer | ||
9917 | available. | ||
9918 | |||
9919 | The communicator API also provides for flow control. First, communicators | ||
9920 | exhibit back-pressure on TRANSPORT: the number of messages TRANSPORT may add to | ||
9921 | a queue for transmission will be limited. So by not draining the transmission | ||
9922 | queue, back-pressure is provided to TRANSPORT. In the other direction, | ||
9923 | communicators may allow TRANSPORT to give back-pressure towards the | ||
9924 | communicator by providing a non-NULL | ||
9925 | @code{GNUNET_TRANSPORT_MessageCompletedCallback} | ||
9926 | argument to the @code{GNUNET_TRANSPORT_communicator_receive} function. In this | ||
9927 | case, TRANSPORT will only invoke this function once it has processed the message | ||
9928 | and is ready to receive more. Communicators should then limit how much traffic | ||
9929 | they receive based on this backpressure. Note that communicators do not have to | ||
9930 | provide a @code{GNUNET_TRANSPORT_MessageCompletedCallback}; | ||
9931 | for example, UDP cannot support back-pressure due to the nature of the UDP | ||
9932 | protocol. In this case, TRANSPORT will implement its own TRANSPORT-to-TRANSPORT | ||
9933 | flow control to reduce the sender's data rate to acceptable levels. | ||
9934 | |||
9935 | TRANSPORT may notify a communicator about backchannel messages TRANSPORT | ||
9936 | received from other peers for this communicator. Similarly, communicators can | ||
9937 | ask TRANSPORT to try to send a backchannel message to other communicators of | ||
9938 | other peers. The semantics of the backchannel message are up to the | ||
9939 | communicators which use them. | ||
9940 | TRANSPORT may fail transmitting backchannel messages, and TRANSPORT will not | ||
9941 | attempt to retransmit them. | ||
9942 | |||
9943 | @cindex MESSENGER Subsystem | ||
9944 | @cindex MESSENGER | ||
9945 | @cindex messenger | ||
9946 | @node MESSENGER Subsystem | ||
9947 | @section MESSENGER Subsystem | ||
9948 | |||
9949 | The MESSENGER subsystem is responsible for secure end-to-end communication in | ||
9950 | groups of nodes in the GNUnet overlay network. MESSENGER builds on the CADET | ||
9951 | subsystem which provides a reliable and secure end-to-end communication between | ||
9952 | the nodes inside of these groups. | ||
9953 | |||
9954 | Additionally to the CADET security benefits, MESSENGER provides following | ||
9955 | properties designed for application level usage: | ||
9956 | |||
9957 | @itemize @bullet | ||
9958 | @item MESSENGER provides integrity by signing the messages with the users | ||
9959 | provided ego | ||
9960 | @item MESSENGER adds (optional) forward secrecy by replacing the key pair of the | ||
9961 | used ego and signing the propagation of the new one with old one (chaining | ||
9962 | egos) | ||
9963 | @item MESSENGER provides verification of a original sender by checking against | ||
9964 | all used egos from a member which are currently in active use (active use | ||
9965 | depends on the state of a member session) | ||
9966 | @item MESSENGER offsers (optional) decentralized message forwarding between all | ||
9967 | nodes in a group to improve availability and prevent MITM-attacks | ||
9968 | @item MESSENGER handles new connections and disconnections from nodes in the | ||
9969 | group by reconnecting them preserving an efficient structure for message | ||
9970 | distribution (ensuring availability and accountablity) | ||
9971 | @item MESSENGER provides replay protection (messages can be uniquely identified | ||
9972 | via SHA-512, include a timestamp and the hash of the last message) | ||
9973 | @item MESSENGER allows detection for dropped messages by chaining them (messages | ||
9974 | refer to the last message by their hash) improving accountability | ||
9975 | @item MESSENGER allows requesting messages from other peers explicitly to ensure | ||
9976 | availability | ||
9977 | @item MESSENGER provides confidentiality by padding messages to few different | ||
9978 | sizes (512 bytes, 4096 bytes, 32768 bytes and maximal message size from | ||
9979 | CADET) | ||
9980 | @item MESSENGER adds (optional) confidentiality with ECDHE to exchange and use | ||
9981 | symmetric encryption, encrypting with both AES-256 and Twofish but | ||
9982 | allowing only selected members to decrypt (using the receivers ego for | ||
9983 | ECDHE) | ||
9984 | @end itemize | ||
9985 | |||
9986 | Also MESSENGER provides multiple features with privacy in mind: | ||
9987 | |||
9988 | @itemize @bullet | ||
9989 | @item MESSENGER allows deleting messages from all peers in the group by the | ||
9990 | original sender (uses the MESSENGER provided verification) | ||
9991 | @item MESSENGER allows using the publicly known anonymous ego instead of any | ||
9992 | unique identifying ego | ||
9993 | @item MESSENGER allows your node to decide between acting as relay of the used | ||
9994 | messaging room (sharing your peer's identity with all nodes in the group) | ||
9995 | or acting as guest (sharing your peer's identity only with the nodes you | ||
9996 | explicitly open a connection to) | ||
9997 | @item MESSENGER handles members independently of the peer's identity making | ||
9998 | forwarded messages indistinguishable from directly received ones ( | ||
9999 | complicating the tracking of messages and identifying its origin) | ||
10000 | @item MESSENGER allows names of members being not unique (also names are | ||
10001 | optional) | ||
10002 | @item MESSENGER does not include information about the selected receiver of an | ||
10003 | explicitly encrypted message in its header, complicating it for other | ||
10004 | members to draw conclusions from communication partners | ||
10005 | @end itemize | ||
10006 | |||
10007 | @menu | ||
10008 | * libgnunetmessenger:: | ||
10009 | * Member sessions:: | ||
10010 | @end menu | ||
10011 | |||
10012 | @node libgnunetmessenger | ||
10013 | @subsection libgnunetmessenger | ||
10014 | |||
10015 | The MESSENGER API (defined in @file{gnunet_messenger_service.h}) allows P2P | ||
10016 | applications built using GNUnet to communicate with specified kinds of messages | ||
10017 | in a group. It provides applications the ability to send and receive encrypted | ||
10018 | messages to any group of peers participating in GNUnet in a decentralized way ( | ||
10019 | without even knowing all peers's identities). | ||
10020 | |||
10021 | MESSENGER delivers messages to other peers in "rooms". A room uses a variable | ||
10022 | amount of CADET "channels" which will all be used for message distribution. Each | ||
10023 | channel can represent an outgoing connection opened by entering a room with | ||
10024 | @code{GNUNET_MESSENGER_enter_room} or an incoming connection if the room was | ||
10025 | opened before via @code{GNUNET_MESSENGER_open_room}. | ||
10026 | |||
10027 | @image{images/messenger_room,6in,,Room structure} | ||
10028 | |||
10029 | To enter a room you have to specify the "door" (peer's identity of a peer which | ||
10030 | has opened the room) and the key of the room (which is identical to a CADET | ||
10031 | "port"). To open a room you have to specify only the key to use. When opening a | ||
10032 | room you automatically distribute a PEER-message sharing your peer's identity in | ||
10033 | the room. | ||
10034 | |||
10035 | Entering or opening a room can also be combined in any order. In any case you | ||
10036 | will automatically get a unique member ID and send a JOIN-message notifying | ||
10037 | others about your entry and your public key from your selected ego. | ||
10038 | |||
10039 | The ego can be selected by name with the initial @code{GNUNET_MESSENGER_connect} | ||
10040 | besides setting a (identity-)callback for each change/confirmation of the used | ||
10041 | ego and a (message-)callback which gets called every time a message gets sent or | ||
10042 | received in the room. Once the identity-callback got called you can check your | ||
10043 | used ego with @code{GNUNET_MESSENGER_get_key} providing only its public key. The | ||
10044 | function returns NULL if the anonymous ego is used. If the ego should be | ||
10045 | replaced with a newly generated one, you can use @code{GNUNET_MESSENGER_update} | ||
10046 | to ensure proper chaining of used egos. | ||
10047 | |||
10048 | Also once the identity-callback got called you can check your used name with | ||
10049 | @code{GNUNET_MESSENGER_get_name} and potentially change or set a name via | ||
10050 | @code{GNUNET_MESSENGER_set_name}. A name is for example required to create a new | ||
10051 | ego with @code{GNUNET_MESSENGER_update}. Also any change in ego or name will | ||
10052 | automatically be distributed in the room with a NAME- or KEY-message | ||
10053 | respectively. | ||
10054 | |||
10055 | To send a message a message inside of a room you can use | ||
10056 | @code{GNUNET_MESSENGER_send_message}. If you specify a selected contact as | ||
10057 | receiver, the message gets encrypted automatically and will be sent as PRIVATE- | ||
10058 | message instead. | ||
10059 | |||
10060 | To request a potentially missed message or to get a specific message after its | ||
10061 | original call of the message-callback, you can use | ||
10062 | @code{GNUNET_MESSENGER_get_message}. Additionally once a message was distributed | ||
10063 | to application level and the message-callback got called, you can get the | ||
10064 | contact respresenting a message's sender respectively with | ||
10065 | @code{GNUNET_MESSENGER_get_sender}. This allows getting name and the public key | ||
10066 | of any sender currently in use with @code{GNUNET_MESSENGER_contact_get_name} | ||
10067 | and @code{GNUNET_MESSENGER_contact_get_key}. It is also possible to iterate | ||
10068 | through all current members of a room with | ||
10069 | @code{GNUNET_MESSENGER_iterate_members} using a callback. | ||
10070 | |||
10071 | To leave a room you can use @code{GNUNET_MESSENGER_close_room} which will also | ||
10072 | close the rooms connections once all applications on the same peer have left | ||
10073 | the room. Leaving a room will also send a LEAVE-message closing a member session | ||
10074 | on all connected peers before any connection will be closed. Leaving a room is | ||
10075 | however not required for any application to keep your member session open | ||
10076 | between multiple sessions of the actual application. | ||
10077 | |||
10078 | Finally, when an application no longer wants to use CADET, it should call | ||
10079 | @code{GNUNET_MESSENGER_disconnect}. You don't have to explicitly close the used | ||
10080 | rooms or leave them. | ||
10081 | |||
10082 | Here is a little summary to the kinds of messages you can send manually: | ||
10083 | |||
10084 | @menu | ||
10085 | * MERGE-message:: | ||
10086 | * INVITE-message:: | ||
10087 | * TEXT-message:: | ||
10088 | * FILE-message:: | ||
10089 | * DELETE-message:: | ||
10090 | @end menu | ||
10091 | |||
10092 | @node MERGE-message | ||
10093 | @subsubsection MERGE-message | ||
10094 | |||
10095 | MERGE-messages will generally be sent automatically to reduce the amount of | ||
10096 | parallel chained messages. This is necessary to close a member session for | ||
10097 | example. You can also send MERGE-messages manually if required to merge two | ||
10098 | chains of messages. | ||
10099 | |||
10100 | @node INVITE-message | ||
10101 | @subsubsection INVITE-message | ||
10102 | |||
10103 | INVITE-messages can be used to invite other members in a room to a different | ||
10104 | room, sharing one potential door and the required key to enter the room. This | ||
10105 | kind of message is typically sent as encrypted PRIVATE-message to selected | ||
10106 | members because it doesn't make much sense to invite all members from one room | ||
10107 | to another considering a rooms key doesn't specify its usage. | ||
10108 | |||
10109 | @node TEXT-message | ||
10110 | @subsubsection TEXT-message | ||
10111 | |||
10112 | TEXT-messages can be used to send simple text-based messages and should be | ||
10113 | considered as being in readable form without complex decoding. The text has to | ||
10114 | end with a NULL-terminator character and should be in UTF-8 encoding for most | ||
10115 | compatibility. | ||
10116 | |||
10117 | @node FILE-message | ||
10118 | @subsubsection FILE-message | ||
10119 | |||
10120 | FILE-messages can be used to share files inside of a room. They do not contain | ||
10121 | the actual file being shared but its original hash, filename, URI to download | ||
10122 | the file and a symmetric key to decrypt the downloaded file. | ||
10123 | |||
10124 | It is recommended to use the FS subsystem and the FILE-messages in combination. | ||
10125 | |||
10126 | @node DELETE-message | ||
10127 | @subsubsection DELETE-message | ||
10128 | |||
10129 | DELETE-messages can be used to delete messages selected with its hash. You can | ||
10130 | also select any custom delay relative to the time of sending the DELETE-message. | ||
10131 | Deletion will only be processed on each peer in a room if the sender is | ||
10132 | authorized. | ||
10133 | |||
10134 | The only information of a deleted message which being kept will be the chained | ||
10135 | hashes connecting the message graph for potential traversion. For example the | ||
10136 | check for completion of a member session requires this information. | ||
10137 | |||
10138 | @node Member sessions | ||
10139 | @subsection Member sessions | ||
10140 | |||
10141 | A member session is a triple of the room key, the member ID and the public key | ||
10142 | of the member's ego. Member sessions allow that a member can change their ID or | ||
10143 | their ego once at a time without losing the ability to delete old messages or | ||
10144 | identifying the original sender of a message. On every change of ID or EGO a | ||
10145 | session will be marked as closed. So every session chain will only contain one | ||
10146 | open session with the current ID and public key. | ||
10147 | |||
10148 | If a session is marked as closed the MESSENGER service will check from the first | ||
10149 | message opening a session to its last one closing the session for completion. If | ||
10150 | a the service can confirm that there is no message still missing which was sent | ||
10151 | from the closed member session, it will be marked as completed. | ||
10152 | |||
10153 | A completed member session is not able to verify any incoming message to ensure | ||
10154 | forward secrecy preventing others from using old stolen egos. | ||
diff --git a/doc/handbook/chapters/installation.texi b/doc/handbook/chapters/installation.texi deleted file mode 100644 index f93645114..000000000 --- a/doc/handbook/chapters/installation.texi +++ /dev/null | |||
@@ -1,2497 +0,0 @@ | |||
1 | @node Installing GNUnet | ||
2 | @chapter Installing GNUnet | ||
3 | |||
4 | This guide is intended for those who want to install Gnunet from | ||
5 | source. For instructions on how to install GNUnet as a binary package | ||
6 | please refer to the official documentation of your operating system or | ||
7 | package manager. | ||
8 | |||
9 | For understanding this guide properly it is important to know that | ||
10 | there are two different ways of running GNUnet: | ||
11 | |||
12 | @itemize @bullet | ||
13 | @item the @emph{single-user setup} | ||
14 | @item the @emph{multi-user setup} | ||
15 | @end itemize | ||
16 | |||
17 | The latter variant has a better security model and requires extra | ||
18 | preparation before running @code{make install} and a different | ||
19 | configuration. Beginners who want to quickly try out GNUnet can | ||
20 | use the @emph{single-user setup}. | ||
21 | |||
22 | @menu | ||
23 | * Installing dependencies:: | ||
24 | * Getting the Source Code:: | ||
25 | * Create user and groups for the system services:: | ||
26 | * Preparing and Compiling the Source Code:: | ||
27 | * Installation:: | ||
28 | * Minimal configuration:: | ||
29 | * Checking the Installation:: | ||
30 | * The graphical configuration interface:: | ||
31 | * Config Leftovers:: | ||
32 | @end menu | ||
33 | |||
34 | @c ----------------------------------------------------------------------- | ||
35 | @node Installing dependencies | ||
36 | @section Installing dependencies | ||
37 | |||
38 | GNUnet needs few libraries and applications for being able to run and | ||
39 | another few optional ones for using certain features. Preferably they | ||
40 | should be installed with a package manager. | ||
41 | |||
42 | The mandatory libraries and applications are | ||
43 | @itemize @bullet | ||
44 | @item autoconf 2.59 or above (when building from git) | ||
45 | @item automake 1.11.1 or above (when building from git) | ||
46 | @item recutils 1.0 or above (when building from git) | ||
47 | @item gettext | ||
48 | @item glibc (read below, other libcs work) | ||
49 | @item GnuTLS 3.2.12 or above, recommended to be linked against libunbound | ||
50 | @item GNU make 4.0 or higher (other make implementations do work) | ||
51 | @item iptables (on Linux systems) | ||
52 | @item libtool 2.2 or above | ||
53 | @item libltdl (part of libtool) | ||
54 | @item libgcrypt 1.6 or above | ||
55 | @item libidn2 or libidn | ||
56 | @item libmicrohttpd 0.9.63 or above | ||
57 | @item libunistring | ||
58 | @item libjansson | ||
59 | @item libgmp | ||
60 | @item libgnurl or libcurl (libcurl has to be linked to GnuTLS) 7.35.0 or above | ||
61 | @item Texinfo 5.2 or above (for building the documentation) | ||
62 | @item Texlive 2012 or above (for building the documentation, and for gnunet-bcd) | ||
63 | @item makeinfo 4.8 or above | ||
64 | @item pkgconf (or pkg-config) | ||
65 | @item zlib | ||
66 | @end itemize | ||
67 | |||
68 | Glibc is required for certain NSS features: | ||
69 | |||
70 | @example | ||
71 | One mechanism of integrating GNS with legacy applications via NSS is | ||
72 | not available if this is disabled. But applications that don't use the | ||
73 | glibc for NS resolution won't work anyway with this, so little is lost | ||
74 | on *BSD systems. | ||
75 | GNS via direct use or via the HTTP or DNS proxies is unaffected. | ||
76 | @end example | ||
77 | |||
78 | Other libcs should work, the resulting builds just don't include the | ||
79 | glibc NSS specific code. One example is the build against NetBSD's libc | ||
80 | as detailed in @uref{https://bugs.gnunet.org/view.php?id=5605}. | ||
81 | |||
82 | In addition GNUnet needs at least one of these three databases | ||
83 | (at the minimum sqlite3) | ||
84 | @itemize @bullet | ||
85 | @item sqlite + libsqlite 3.8 or above (the default, requires no further configuration) | ||
86 | @item postgres + libpq | ||
87 | @item mysql + libmysqlclient | ||
88 | @end itemize | ||
89 | |||
90 | These are the dependencies only required for certain features | ||
91 | @itemize @bullet | ||
92 | @item miniupnpc (for traversing NAT boxes more reliably) | ||
93 | @item libnss | ||
94 | @item libopus (for running the GNUnet conversation telephony application) | ||
95 | @item libogg (for running the GNUnet conversation telephony application) | ||
96 | @item gstreamer OR libpulse (for running the GNUnet conversation telephony application) | ||
97 | @item bluez (for bluetooth support) | ||
98 | @item libextractor (optional but highly recommended, read below) | ||
99 | @item libpbc | ||
100 | (for attribute-based encryption and the identity provider subsystem) | ||
101 | @item libgabe | ||
102 | (for attribute-based encryption and the identity provider subsystem) | ||
103 | @item texi2mdoc (for automatic mdoc generation) | ||
104 | @item perl5 for some utilities (which are not installed) | ||
105 | @end itemize | ||
106 | |||
107 | About libextractor being optional: | ||
108 | @example | ||
109 | While libextractor ("LE") is optional, it is recommended to build gnunet | ||
110 | against it. If you install it later, you won't benefit from libextractor. | ||
111 | If you are a distributor, we recommend to split LE into basis + plugins | ||
112 | rather than making LE an option as an afterthought by the user. LE | ||
113 | itself is very small, but its dependency chain on first, second, third | ||
114 | etc level can be big. There is a small effect on privacy if your LE | ||
115 | build differs from one which includes all plugins (plugins are build as | ||
116 | shared objects): if users publish a directory with a mixture of file | ||
117 | types (for example mpeg, jpeg, png, gif) the configuration of LE could | ||
118 | leak which plugins are installed for which filetypes are not providing | ||
119 | more details. However, this leak is just a minor concern. | ||
120 | @end example | ||
121 | |||
122 | These are the test-suite requirements: | ||
123 | @itemize @bullet | ||
124 | @item python3.6 or higher | ||
125 | @item gnunet (installation first) | ||
126 | @item some core-utils: which(1), bc(1), curl(1), sed(1), awk(1), etc. | ||
127 | @item a shell (very few Bash scripts, the majority are POSIX sh scripts) | ||
128 | @end itemize | ||
129 | |||
130 | These are runtime requirements: | ||
131 | @itemize @bullet | ||
132 | @item nss (the certutil binary, for gnunet-gns-proxy-setup-ca) | ||
133 | @item openssl (openssl binary, for gnunet-gns-proxy-setup-ca) | ||
134 | @end itemize | ||
135 | |||
136 | @c ----------------------------------------------------------------------- | ||
137 | @node Getting the Source Code | ||
138 | @section Getting the Source Code | ||
139 | You can either download the source code using git (you obviously need | ||
140 | git installed) or as an archive. | ||
141 | |||
142 | Using git type | ||
143 | @example | ||
144 | git clone https://git.gnunet.org/gnunet.git | ||
145 | @end example | ||
146 | |||
147 | The archive can be found at | ||
148 | @uref{https://ftpmirror.gnu.org/gnu/gnunet/}. Extract it using a graphical | ||
149 | archive tool or @code{tar}: | ||
150 | @example | ||
151 | tar xzvf gnunet-@value{VERSION}.tar.gz | ||
152 | @end example | ||
153 | |||
154 | In the next chapter we will assume that the source code is available | ||
155 | in the home directory at @code{~/gnunet}. | ||
156 | |||
157 | @c ----------------------------------------------------------------------- | ||
158 | @node Create user and groups for the system services | ||
159 | @section Create user and groups for the system services | ||
160 | |||
161 | @cartouche | ||
162 | For the single-user setup this section can be skipped. | ||
163 | @end cartouche | ||
164 | |||
165 | The multi-user setup means that there are @emph{system services}, which are | ||
166 | run once per machine as a dedicated system user (called @code{gnunet}) and | ||
167 | @emph{user services} which can be started by every user who wants to use | ||
168 | GNUnet applications. The user services communicate with the system services | ||
169 | over unix domain sockets. To gain permissions to read and write those sockets | ||
170 | the users running GNUnet applications will need to be in the @code{gnunet} | ||
171 | group. In addition the group @code{gnunetdns} may be needed (see below). | ||
172 | |||
173 | Create user @code{gnunet} who is member of the group @code{gnunet} | ||
174 | (automatically created) and specify a home directory where the GNUnet | ||
175 | services will store persistent data such as information about peers. | ||
176 | @example | ||
177 | $ sudo useradd --system --home-dir /var/lib/gnunet --create-home gnunet | ||
178 | @end example | ||
179 | |||
180 | Now add your own user to the @code{gnunet} group. | ||
181 | |||
182 | @example | ||
183 | $ sudo usermod -aG gnunet alice | ||
184 | @end example | ||
185 | |||
186 | Create a group @code{gnunetdns}. This allows using @code{setgid} in a way | ||
187 | that only the DNS service can run the @code{gnunet-helper-dns} binary. This | ||
188 | is only needed if @emph{system-wide DNS interception} will be used. For more | ||
189 | information see @xref{Configuring system-wide DNS interception}. | ||
190 | |||
191 | @example | ||
192 | $ sudo groupadd gnunetdns | ||
193 | @end example | ||
194 | |||
195 | @c ----------------------------------------------------------------------- | ||
196 | @node Preparing and Compiling the Source Code | ||
197 | @section Preparing and Compiling the Source Code | ||
198 | For preparing the source code for compilation a bootstrap script and | ||
199 | @code{configure} has to be run from the source code directory. When | ||
200 | running @code{configure} the following options can be specified to | ||
201 | customize the compilation and installation process: | ||
202 | |||
203 | @itemize @bullet | ||
204 | @item @code{--disable-documentation} - don't build the documentation | ||
205 | @item @code{--enable-logging=[LOGLEVEL]} - choose a loglevel (@code{debug}, @code{info}, @code{warning} or @code{error}) | ||
206 | @item @code{--prefix=[PATH]} - the directory where the GNUnet libraries and binaries will be installed | ||
207 | @item @code{--with-extractor=[PATH]} - the path to libextractor | ||
208 | @item @code{--with-libidn=[PATH]} - the path to libidn | ||
209 | @item @code{--with-libidn2=[PATH]} - the path to libidn2 (takes priority over libidn if both are found) | ||
210 | @item @code{--with-microhttpd=[PATH]} - the path to libmicrohttpd | ||
211 | @item @code{--with-sqlite=[PATH]} - the path to libsqlite | ||
212 | @item @code{--with-zlib=[PATH]} - the path to zlib | ||
213 | @end itemize | ||
214 | |||
215 | Note that the list above is not always up to date and you | ||
216 | should check the output of @code{./configure --help}, read | ||
217 | the @file{configure.ac} or send an email asking for assistance | ||
218 | if you are in doubt of any configure options or require fixes | ||
219 | for your operating system. | ||
220 | |||
221 | The following example configures the installation prefix | ||
222 | @code{/usr/local} and disables building the documentation | ||
223 | @example | ||
224 | $ cd ~/gnunet | ||
225 | $ ./bootstrap | ||
226 | $ configure --prefix=/usr/local --disable-documentation | ||
227 | @end example | ||
228 | |||
229 | After running the bootstrap script and @code{configure} successfully | ||
230 | the source code can be compiled with make. Here @code{-j5} specifies | ||
231 | that 5 threads should be used. | ||
232 | @example | ||
233 | $ make -j5 | ||
234 | @end example | ||
235 | |||
236 | @c ----------------------------------------------------------------------- | ||
237 | @node Installation | ||
238 | @section Installation | ||
239 | The compiled binaries can be installed using @code{make install}. It | ||
240 | needs to be run as root (or with sudo) because some binaries need the | ||
241 | @code{suid} bit set. Without that some features (e.g. the VPN service, | ||
242 | system-wide DNS interception, NAT traversal using ICMP) will not work. | ||
243 | |||
244 | @example | ||
245 | $ sudo make install | ||
246 | @end example | ||
247 | |||
248 | @menu | ||
249 | * NSS plugin (Optional):: | ||
250 | * Installing the GNS Certificate Authority (Optional):: | ||
251 | @end menu | ||
252 | |||
253 | @node NSS plugin (Optional) | ||
254 | @subsection NSS plugin (Optional) | ||
255 | |||
256 | @cartouche | ||
257 | The installation of the NSS plugin is only necessary if GNS | ||
258 | resolution shall be used with legacy applications (that only | ||
259 | support DNS). | ||
260 | @end cartouche | ||
261 | |||
262 | One important library is the GNS plugin for NSS (the name services | ||
263 | switch) which allows using GNS (the GNU name system) in the normal DNS | ||
264 | resolution process. Unfortunately NSS expects it in a specific | ||
265 | location (probably @code{/lib}) which may differ from the installation | ||
266 | prefix (see @code{--prefix} option in the previous section). This is | ||
267 | why the plugin has to be installed manually. | ||
268 | |||
269 | Find the directory where nss plugins are installed on your system, e.g. | ||
270 | |||
271 | @example | ||
272 | $ ls -l /lib/libnss_* | ||
273 | /lib/libnss_mymachines.so.2 | ||
274 | /lib/libnss_resolve.so.2 | ||
275 | /lib/libnss_myhostname.so.2 | ||
276 | /lib/libnss_systemd.so.2 | ||
277 | @end example | ||
278 | |||
279 | Copy the GNS NSS plugin to that directory: | ||
280 | |||
281 | @example | ||
282 | cp ~/gnunet/src/gns/nss/.libs/libnss_gns.so.2 /lib | ||
283 | @end example | ||
284 | |||
285 | Now, to activate the plugin, you need to edit your | ||
286 | @code{/etc/nsswitch.conf} where you should find a line like this: | ||
287 | |||
288 | @example | ||
289 | hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 | ||
290 | @end example | ||
291 | |||
292 | The exact details may differ a bit, which is fine. Add the text | ||
293 | @code{"gns [NOTFOUND=return]"} after @code{"files"}. | ||
294 | |||
295 | @example | ||
296 | hosts: files gns [NOTFOUND=return] mdns4_minimal [NOTFOUND=return] dns mdns4 | ||
297 | @end example | ||
298 | |||
299 | @node Installing the GNS Certificate Authority (Optional) | ||
300 | @subsection Installing the GNS Certificate Authority (Optional) | ||
301 | |||
302 | @cartouche | ||
303 | Installing the GNS certificate authority is only necessary if GNS shall | ||
304 | be used in a browser. | ||
305 | @end cartouche | ||
306 | |||
307 | The GNS Certificate authority can provide TLS certificates for GNS names while | ||
308 | downloading webpages from legacy webservers. This allows browsers to use HTTPS | ||
309 | in combinations with GNS name resolution. | ||
310 | |||
311 | To install it execute the GNS CA-setup script. So far Firefox and Chromium are | ||
312 | supported. | ||
313 | |||
314 | @example | ||
315 | $ gnunet-gns-proxy-setup-ca | ||
316 | @end example | ||
317 | |||
318 | A local proxy server, that takes care of the name resolution and provides | ||
319 | certificates on-the-fly needs to be started: | ||
320 | |||
321 | @example | ||
322 | $ /usr/lib/gnunet/libexec/gnunet-gns-proxy | ||
323 | @end example | ||
324 | |||
325 | Now GNS should work in browsers that are configured to use a SOCKS proxy on | ||
326 | @code{localhost:7777}. | ||
327 | |||
328 | |||
329 | @node Minimal configuration | ||
330 | @section Minimal configuration | ||
331 | GNUnet needs a configuration file to start (@pxref{Config file format}). | ||
332 | For the @emph{single-user setup} an empty file is sufficient: | ||
333 | |||
334 | @example | ||
335 | $ touch ~/.config/gnunet.conf | ||
336 | @end example | ||
337 | |||
338 | For the @emph{multi-user setup} we need an extra config file for the system | ||
339 | services. The default location is @code{/etc/gnunet.conf}. The minimal | ||
340 | content of that file which activates the system services roll is: | ||
341 | |||
342 | @example | ||
343 | [arm] | ||
344 | START_SYSTEM_SERVICES = YES | ||
345 | START_USER_SERVICES = NO | ||
346 | @end example | ||
347 | |||
348 | The config file for the user services (@code{~/.config/gnunet.conf}) needs | ||
349 | the opposite configuration to activate the user services roll: | ||
350 | |||
351 | @example | ||
352 | [arm] | ||
353 | START_SYSTEM_SERVICES = NO | ||
354 | START_USER_SERVICES = YES | ||
355 | @end example | ||
356 | |||
357 | |||
358 | @node Checking the Installation | ||
359 | @section Checking the Installation | ||
360 | |||
361 | |||
362 | This section describes a quick, casual way to check if your GNUnet | ||
363 | installation works. However, if it does not, we do not cover | ||
364 | steps for recovery --- for this, please study the instructions | ||
365 | provided in the developer handbook as well as the system-specific | ||
366 | instruction in the source code repository. | ||
367 | Please note that the system specific instructions are not provided | ||
368 | as part of this handbook! | ||
369 | |||
370 | |||
371 | @menu | ||
372 | * Starting GNUnet:: | ||
373 | * gnunet-gtk:: | ||
374 | * Statistics:: | ||
375 | * Peer Information:: | ||
376 | @end menu | ||
377 | |||
378 | @cindex Starting GNUnet | ||
379 | @cindex GNUnet GTK | ||
380 | @cindex GTK | ||
381 | @cindex GTK user interface | ||
382 | |||
383 | @node Starting GNUnet | ||
384 | @subsection Starting GNUnet | ||
385 | The GNUnet services are started and stopped by the ARM service (Automatic | ||
386 | Restart Manager). For the @emph{single-user setup} a simple | ||
387 | |||
388 | @example | ||
389 | $ gnunet-arm -s | ||
390 | @end example | ||
391 | |||
392 | starts a default set of services. Later GNUnet applications can request more | ||
393 | services to start without additional user interaction. GNUnet can be stopped | ||
394 | again using the @code{-e} option: | ||
395 | |||
396 | @example | ||
397 | $ gnunet-arm -e | ||
398 | @end example | ||
399 | |||
400 | The list of running services can be displayed using the @code{-I} option. | ||
401 | It should look similar to this example: | ||
402 | |||
403 | @example | ||
404 | $ gnunet-arm -I | ||
405 | Running services: | ||
406 | topology (gnunet-daemon-topology) | ||
407 | nat (gnunet-service-nat) | ||
408 | vpn (gnunet-service-vpn) | ||
409 | gns (gnunet-service-gns) | ||
410 | cadet (gnunet-service-cadet) | ||
411 | namecache (gnunet-service-namecache) | ||
412 | hostlist (gnunet-daemon-hostlist) | ||
413 | revocation (gnunet-service-revocation) | ||
414 | ats (gnunet-service-ats) | ||
415 | peerinfo (gnunet-service-peerinfo) | ||
416 | zonemaster (gnunet-service-zonemaster) | ||
417 | zonemaster-monitor (gnunet-service-zonemaster-monitor) | ||
418 | dht (gnunet-service-dht) | ||
419 | namestore (gnunet-service-namestore) | ||
420 | set (gnunet-service-set) | ||
421 | statistics (gnunet-service-statistics) | ||
422 | nse (gnunet-service-nse) | ||
423 | fs (gnunet-service-fs) | ||
424 | peerstore (gnunet-service-peerstore) | ||
425 | core (gnunet-service-core) | ||
426 | rest (gnunet-rest-server) | ||
427 | transport (gnunet-service-transport) | ||
428 | datastore (gnunet-service-datastore) | ||
429 | @end example | ||
430 | |||
431 | For the @emph{multi-user setup} first the system services need to be started | ||
432 | as the system user, i.e. the user @code{gnunet} needs to execute | ||
433 | @code{gnunet-arm -s}. This should be done by the system's init system. | ||
434 | Then the user who wants to start GNUnet applications has to run | ||
435 | @code{gnunet-arm -s} too. It is recommended to automate this, e.g. using | ||
436 | the user's crontab. | ||
437 | |||
438 | @node gnunet-gtk | ||
439 | @subsection gnunet-gtk | ||
440 | |||
441 | |||
442 | The @command{gnunet-gtk} package contains several graphical | ||
443 | user interfaces for the respective GNUnet applications. | ||
444 | Currently these interfaces cover: | ||
445 | |||
446 | @itemize @bullet | ||
447 | @item Statistics | ||
448 | @item Peer Information | ||
449 | @item GNU Name System | ||
450 | @item File Sharing | ||
451 | @item Conversation | ||
452 | @item Setup | ||
453 | @end itemize | ||
454 | |||
455 | Previously, many of these interfaces were combined into one application | ||
456 | called @command{gnunet-gtk}, with different tabs for each interface. This | ||
457 | combined application has been removed in version 0.11.0, but each of the | ||
458 | interfaces is still available as a standalone application | ||
459 | (@command{gnunet-statistics-gtk} for statistics, @command{gnunet-fs-gtk} | ||
460 | for filesharing, etc). | ||
461 | |||
462 | @node Statistics | ||
463 | @subsection Statistics | ||
464 | |||
465 | |||
466 | We assume that you have started gnunet via @code{gnunet-arm} or via your | ||
467 | system-provided method for starting services. | ||
468 | First, you should launch GNUnet's graphical statistics interface. | ||
469 | You can do this from the command-line by typing | ||
470 | |||
471 | @example | ||
472 | gnunet-statistics-gtk | ||
473 | @end example | ||
474 | |||
475 | If your peer is running correctly, you should see a bunch | ||
476 | of lines, all of which should be ``significantly'' above zero (at | ||
477 | least if your peer has been running for more than a few seconds). The | ||
478 | lines indicate how many other peers your peer is connected to (via | ||
479 | different mechanisms) and how large the entire overlay network is | ||
480 | currently estimated to be. The X-axis represents time (in seconds | ||
481 | since the start of @command{gnunet-statistics-gtk}). | ||
482 | |||
483 | You can click on "Traffic" to see information about the amount of | ||
484 | bandwidth your peer has consumed, and on "Storage" to check the amount | ||
485 | of storage available and used by your peer. Note that "Traffic" is | ||
486 | plotted cumulatively, so you should see a strict upwards trend in the | ||
487 | traffic. | ||
488 | |||
489 | The term ``peer'' is a common word used in | ||
490 | federated and distributed networks to describe a participating device | ||
491 | which is connected to the network. Thus, your Personal Computer or | ||
492 | whatever it is you are looking at the Gtk+ interface describes a | ||
493 | ``Peer'' or a ``Node''. | ||
494 | |||
495 | @node Peer Information | ||
496 | @subsection Peer Information | ||
497 | |||
498 | |||
499 | First, you should launch the peer information graphical user interface. | ||
500 | You can do this from the command-line by typing | ||
501 | |||
502 | @example | ||
503 | $ gnunet-peerinfo-gtk | ||
504 | @end example | ||
505 | |||
506 | Once you have done this, you will see a list of known peers (by the | ||
507 | first four characters of their public key), their friend status (all | ||
508 | should be marked as not-friends initially), their connectivity (green | ||
509 | is connected, red is disconnected), assigned bandwidth, country of | ||
510 | origin (if determined) and address information. If hardly any peers | ||
511 | are listed and/or if there are very few peers with a green light for | ||
512 | connectivity, there is likely a problem with your network | ||
513 | configuration. | ||
514 | |||
515 | @c NOTE: Inserted from Installation Handbook in original ``order'': | ||
516 | @c FIXME: Move this to User Handbook. | ||
517 | @node The graphical configuration interface | ||
518 | @section The graphical configuration interface | ||
519 | |||
520 | If you also would like to use @command{gnunet-gtk} and | ||
521 | @command{gnunet-setup} (highly recommended for beginners), do: | ||
522 | |||
523 | @menu | ||
524 | * Configuring your peer:: | ||
525 | * Configuring the Friend-to-Friend (F2F) mode:: | ||
526 | * Configuring the hostlist to bootstrap:: | ||
527 | * Configuration of the HOSTLIST proxy settings:: | ||
528 | * Configuring your peer to provide a hostlist :: | ||
529 | * Configuring the datastore:: | ||
530 | * Configuring the MySQL database:: | ||
531 | * Reasons for using MySQL:: | ||
532 | * Reasons for not using MySQL:: | ||
533 | * Setup Instructions:: | ||
534 | * Testing:: | ||
535 | * Performance Tuning:: | ||
536 | * Setup for running Testcases:: | ||
537 | * Configuring the Postgres database:: | ||
538 | * Reasons to use Postgres:: | ||
539 | * Reasons not to use Postgres:: | ||
540 | * Manual setup instructions:: | ||
541 | * Testing the setup manually:: | ||
542 | * Configuring the datacache:: | ||
543 | * Configuring the file-sharing service:: | ||
544 | * Configuring logging:: | ||
545 | * Configuring the transport service and plugins:: | ||
546 | * Configuring the WLAN transport plugin:: | ||
547 | * Configuring HTTP(S) reverse proxy functionality using Apache or nginx:: | ||
548 | * Blacklisting peers:: | ||
549 | * Configuration of the HTTP and HTTPS transport plugins:: | ||
550 | * Configuring the GNU Name System:: | ||
551 | * Configuring the GNUnet VPN:: | ||
552 | * Bandwidth Configuration:: | ||
553 | * Configuring NAT:: | ||
554 | * Peer configuration for distributors (e.g. Operating Systems):: | ||
555 | @end menu | ||
556 | |||
557 | @node Configuring your peer | ||
558 | @subsection Configuring your peer | ||
559 | |||
560 | This chapter will describe the various configuration options in GNUnet. | ||
561 | |||
562 | The easiest way to configure your peer is to use the | ||
563 | @command{gnunet-setup} tool. | ||
564 | @command{gnunet-setup} is part of the @command{gnunet-gtk} | ||
565 | package. You might have to install it separately. | ||
566 | |||
567 | Many of the specific sections from this chapter actually are linked from | ||
568 | within @command{gnunet-setup} to help you while using the setup tool. | ||
569 | |||
570 | While you can also configure your peer by editing the configuration | ||
571 | file by hand, this is not recommended for anyone except for developers | ||
572 | as it requires a more in-depth understanding of the configuration files | ||
573 | and internal dependencies of GNUnet. | ||
574 | |||
575 | @node Configuring the Friend-to-Friend (F2F) mode | ||
576 | @subsection Configuring the Friend-to-Friend (F2F) mode | ||
577 | |||
578 | GNUnet knows three basic modes of operation: | ||
579 | @itemize @bullet | ||
580 | @item In standard "peer-to-peer" mode, | ||
581 | your peer will connect to any peer. | ||
582 | @item In the pure "friend-to-friend" | ||
583 | mode, your peer will ONLY connect to peers from a list of friends | ||
584 | specified in the configuration. | ||
585 | @item Finally, in mixed mode, | ||
586 | GNUnet will only connect to arbitrary peers if it | ||
587 | has at least a specified number of connections to friends. | ||
588 | @end itemize | ||
589 | |||
590 | When configuring any of the F2F ("friend-to-friend") modes, | ||
591 | you first need to create a file with the peer identities | ||
592 | of your friends. Ask your friends to run | ||
593 | |||
594 | @example | ||
595 | $ gnunet-peerinfo -sq | ||
596 | @end example | ||
597 | |||
598 | @noindent | ||
599 | The resulting output of this command needs to be added to your | ||
600 | @file{friends} file, which is simply a plain text file with one line | ||
601 | per friend with the output from the above command. | ||
602 | |||
603 | You then specify the location of your @file{friends} file in the | ||
604 | @code{FRIENDS} option of the "topology" section. | ||
605 | |||
606 | Once you have created the @file{friends} file, you can tell GNUnet to only | ||
607 | connect to your friends by setting the @code{FRIENDS-ONLY} option | ||
608 | (again in the "topology" section) to YES. | ||
609 | |||
610 | If you want to run in mixed-mode, set "FRIENDS-ONLY" to NO and configure a | ||
611 | minimum number of friends to have (before connecting to arbitrary peers) | ||
612 | under the "MINIMUM-FRIENDS" option. | ||
613 | |||
614 | If you want to operate in normal P2P-only mode, simply set | ||
615 | @code{MINIMUM-FRIENDS} to zero and @code{FRIENDS_ONLY} to NO. | ||
616 | This is the default. | ||
617 | |||
618 | @node Configuring the hostlist to bootstrap | ||
619 | @subsection Configuring the hostlist to bootstrap | ||
620 | |||
621 | After installing the software you need to get connected to the GNUnet | ||
622 | network. The configuration file included in your download is already | ||
623 | configured to connect you to the GNUnet network. | ||
624 | In this section the relevant configuration settings are explained. | ||
625 | |||
626 | To get an initial connection to the GNUnet network and to get to know | ||
627 | peers already connected to the network you can use the so called | ||
628 | "bootstrap servers". | ||
629 | These servers can give you a list of peers connected to the network. | ||
630 | To use these bootstrap servers you have to configure the hostlist daemon | ||
631 | to activate bootstrapping. | ||
632 | |||
633 | To activate bootstrapping, edit the @code{[hostlist]}-section in your | ||
634 | configuration file. You have to set the argument @command{-b} in the | ||
635 | options line: | ||
636 | |||
637 | @example | ||
638 | [hostlist] | ||
639 | OPTIONS = -b | ||
640 | @end example | ||
641 | |||
642 | Additionally you have to specify which server you want to use. | ||
643 | The default bootstrapping server is | ||
644 | "@uref{http://v10.gnunet.org/hostlist, http://v10.gnunet.org/hostlist}". | ||
645 | [^] To set the server you have to edit the line "SERVERS" in the hostlist | ||
646 | section. To use the default server you should set the lines to | ||
647 | |||
648 | @example | ||
649 | SERVERS = http://v10.gnunet.org/hostlist [^] | ||
650 | @end example | ||
651 | |||
652 | @noindent | ||
653 | To use bootstrapping your configuration file should include these lines: | ||
654 | |||
655 | @example | ||
656 | [hostlist] | ||
657 | OPTIONS = -b | ||
658 | SERVERS = http://v10.gnunet.org/hostlist [^] | ||
659 | @end example | ||
660 | |||
661 | @noindent | ||
662 | Besides using bootstrap servers you can configure your GNUnet peer to | ||
663 | receive hostlist advertisements. | ||
664 | Peers offering hostlists to other peers can send advertisement messages | ||
665 | to peers that connect to them. If you configure your peer to receive these | ||
666 | messages, your peer can download these lists and connect to the peers | ||
667 | included. These lists are persistent, which means that they are saved to | ||
668 | your hard disk regularly and are loaded during startup. | ||
669 | |||
670 | To activate hostlist learning you have to add the @command{-e} | ||
671 | switch to the @code{OPTIONS} line in the hostlist section: | ||
672 | |||
673 | @example | ||
674 | [hostlist] | ||
675 | OPTIONS = -b -e | ||
676 | @end example | ||
677 | |||
678 | @noindent | ||
679 | Furthermore you can specify in which file the lists are saved. | ||
680 | To save the lists in the file @file{hostlists.file} just add the line: | ||
681 | |||
682 | @example | ||
683 | HOSTLISTFILE = hostlists.file | ||
684 | @end example | ||
685 | |||
686 | @noindent | ||
687 | Best practice is to activate both bootstrapping and hostlist learning. | ||
688 | So your configuration file should include these lines: | ||
689 | |||
690 | @example | ||
691 | [hostlist] | ||
692 | OPTIONS = -b -e | ||
693 | HTTPPORT = 8080 | ||
694 | SERVERS = http://v10.gnunet.org/hostlist [^] | ||
695 | HOSTLISTFILE = $SERVICEHOME/hostlists.file | ||
696 | @end example | ||
697 | |||
698 | @node Configuration of the HOSTLIST proxy settings | ||
699 | @subsection Configuration of the HOSTLIST proxy settings | ||
700 | |||
701 | The hostlist client can be configured to use a proxy to connect to the | ||
702 | hostlist server. | ||
703 | This functionality can be configured in the configuration file directly | ||
704 | or using the @command{gnunet-setup} tool. | ||
705 | |||
706 | The hostlist client supports the following proxy types at the moment: | ||
707 | |||
708 | @itemize @bullet | ||
709 | @item HTTP and HTTP 1.0 only proxy | ||
710 | @item SOCKS 4/4a/5/5 with hostname | ||
711 | @end itemize | ||
712 | |||
713 | In addition authentication at the proxy with username and password can be | ||
714 | configured. | ||
715 | |||
716 | To configure proxy support for the hostlist client in the | ||
717 | @command{gnunet-setup} tool, select the "hostlist" tab and select | ||
718 | the appropriate proxy type. | ||
719 | The hostname or IP address (including port if required) has to be entered | ||
720 | in the "Proxy hostname" textbox. If required, enter username and password | ||
721 | in the "Proxy username" and "Proxy password" boxes. | ||
722 | Be aware that this information will be stored in the configuration in | ||
723 | plain text (TODO: Add explanation and generalize the part in Chapter 3.6 | ||
724 | about the encrypted home). | ||
725 | |||
726 | To provide these options directly in the configuration, you can | ||
727 | enter the following settings in the @code{[hostlist]} section of | ||
728 | the configuration: | ||
729 | |||
730 | @example | ||
731 | # Type of proxy server, | ||
732 | # Valid values: HTTP, HTTP_1_0, SOCKS4, SOCKS5, SOCKS4A, SOCKS5_HOSTNAME | ||
733 | # Default: HTTP | ||
734 | # PROXY_TYPE = HTTP | ||
735 | |||
736 | # Hostname or IP of proxy server | ||
737 | # PROXY = | ||
738 | # User name for proxy server | ||
739 | # PROXY_USERNAME = | ||
740 | # User password for proxy server | ||
741 | # PROXY_PASSWORD = | ||
742 | @end example | ||
743 | |||
744 | @node Configuring your peer to provide a hostlist | ||
745 | @subsection Configuring your peer to provide a hostlist | ||
746 | |||
747 | If you operate a peer permanently connected to GNUnet you can configure | ||
748 | your peer to act as a hostlist server, providing other peers the list of | ||
749 | peers known to him. | ||
750 | |||
751 | Your server can act as a bootstrap server and peers needing to obtain a | ||
752 | list of peers can contact it to download this list. | ||
753 | To download this hostlist the peer uses HTTP. | ||
754 | For this reason you have to build your peer with libgnurl (or libcurl) | ||
755 | and microhttpd support. | ||
756 | |||
757 | To configure your peer to act as a bootstrap server you have to add the | ||
758 | @command{-p} option to @code{OPTIONS} in the @code{[hostlist]} section | ||
759 | of your configuration file. | ||
760 | Besides that you have to specify a port number for the http server. | ||
761 | In conclusion you have to add the following lines: | ||
762 | |||
763 | @example | ||
764 | [hostlist] | ||
765 | HTTPPORT = 12980 | ||
766 | OPTIONS = -p | ||
767 | @end example | ||
768 | |||
769 | @noindent | ||
770 | If your peer acts as a bootstrap server other peers should know about | ||
771 | that. You can advertise the hostlist your are providing to other peers. | ||
772 | Peers connecting to your peer will get a message containing an | ||
773 | advertisement for your hostlist and the URL where it can be downloaded. | ||
774 | If this peer is in learning mode, it will test the hostlist and, in the | ||
775 | case it can obtain the list successfully, it will save it for | ||
776 | bootstrapping. | ||
777 | |||
778 | To activate hostlist advertisement on your peer, you have to set the | ||
779 | following lines in your configuration file: | ||
780 | |||
781 | @example | ||
782 | [hostlist] | ||
783 | EXTERNAL_DNS_NAME = example.org | ||
784 | HTTPPORT = 12981 | ||
785 | OPTIONS = -p -a | ||
786 | @end example | ||
787 | |||
788 | @noindent | ||
789 | With this configuration your peer will a act as a bootstrap server and | ||
790 | advertise this hostlist to other peers connecting to it. | ||
791 | The URL used to download the list will be | ||
792 | @code{@uref{http://example.org:12981/, http://example.org:12981/}}. | ||
793 | |||
794 | Please notice: | ||
795 | |||
796 | @itemize @bullet | ||
797 | @item The hostlist is @b{not} human readable, so you should not try to | ||
798 | download it using your webbrowser. Just point your GNUnet peer to the | ||
799 | address! | ||
800 | @item Advertising without providing a hostlist does not make sense and | ||
801 | will not work. | ||
802 | @end itemize | ||
803 | |||
804 | @node Configuring the datastore | ||
805 | @subsection Configuring the datastore | ||
806 | |||
807 | The datastore is what GNUnet uses for long-term storage of file-sharing | ||
808 | data. Note that long-term does not mean 'forever' since content does have | ||
809 | an expiration date, and of course storage space is finite (and hence | ||
810 | sometimes content may have to be discarded). | ||
811 | |||
812 | Use the @code{QUOTA} option to specify how many bytes of storage space | ||
813 | you are willing to dedicate to GNUnet. | ||
814 | |||
815 | In addition to specifying the maximum space GNUnet is allowed to use for | ||
816 | the datastore, you need to specify which database GNUnet should use to do | ||
817 | so. Currently, you have the choice between sqLite, MySQL and Postgres. | ||
818 | |||
819 | @node Configuring the MySQL database | ||
820 | @subsection Configuring the MySQL database | ||
821 | |||
822 | This section describes how to setup the MySQL database for GNUnet. | ||
823 | |||
824 | Note that the mysql plugin does NOT work with mysql before 4.1 since we | ||
825 | need prepared statements. | ||
826 | We are generally testing the code against MySQL 5.1 at this point. | ||
827 | |||
828 | @node Reasons for using MySQL | ||
829 | @subsection Reasons for using MySQL | ||
830 | |||
831 | @itemize @bullet | ||
832 | |||
833 | @item On up-to-date hardware where | ||
834 | mysql can be used comfortably, this module | ||
835 | will have better performance than the other database choices (according | ||
836 | to our tests). | ||
837 | |||
838 | @item Its often possible to recover the mysql database from internal | ||
839 | inconsistencies. Some of the other databases do not support repair. | ||
840 | @end itemize | ||
841 | |||
842 | @node Reasons for not using MySQL | ||
843 | @subsection Reasons for not using MySQL | ||
844 | |||
845 | @itemize @bullet | ||
846 | @item Memory usage (likely not an issue if you have more than 1 GB) | ||
847 | @item Complex manual setup | ||
848 | @end itemize | ||
849 | |||
850 | @node Setup Instructions | ||
851 | @subsection Setup Instructions | ||
852 | |||
853 | @itemize @bullet | ||
854 | |||
855 | @item In @file{gnunet.conf} set in section @code{DATASTORE} the value for | ||
856 | @code{DATABASE} to @code{mysql}. | ||
857 | |||
858 | @item Access mysql as root: | ||
859 | |||
860 | @example | ||
861 | $ mysql -u root -p | ||
862 | @end example | ||
863 | |||
864 | @noindent | ||
865 | and issue the following commands, replacing $USER with the username | ||
866 | that will be running @command{gnunet-arm} (so typically "gnunet"): | ||
867 | |||
868 | @example | ||
869 | CREATE DATABASE gnunet; | ||
870 | GRANT select,insert,update,delete,create,alter,drop,create \ | ||
871 | temporary tables ON gnunet.* TO $USER@@localhost; | ||
872 | SET PASSWORD FOR $USER@@localhost=PASSWORD('$the_password_you_like'); | ||
873 | FLUSH PRIVILEGES; | ||
874 | @end example | ||
875 | |||
876 | @item | ||
877 | In the $HOME directory of $USER, create a @file{.my.cnf} file with the | ||
878 | following lines | ||
879 | |||
880 | @example | ||
881 | [client] | ||
882 | user=$USER | ||
883 | password=$the_password_you_like | ||
884 | @end example | ||
885 | |||
886 | @end itemize | ||
887 | |||
888 | That's it. Note that @file{.my.cnf} file is a slight security risk unless | ||
889 | its on a safe partition. The @file{$HOME/.my.cnf} can of course be | ||
890 | a symbolic link. | ||
891 | Luckily $USER has only privileges to mess up GNUnet's tables, | ||
892 | which should be pretty harmless. | ||
893 | |||
894 | @node Testing | ||
895 | @subsection Testing | ||
896 | |||
897 | You should briefly try if the database connection works. First, login | ||
898 | as $USER. Then use: | ||
899 | |||
900 | @example | ||
901 | $ mysql -u $USER | ||
902 | mysql> use gnunet; | ||
903 | @end example | ||
904 | |||
905 | @noindent | ||
906 | If you get the message | ||
907 | |||
908 | @example | ||
909 | Database changed | ||
910 | @end example | ||
911 | |||
912 | @noindent | ||
913 | it probably works. | ||
914 | |||
915 | If you get | ||
916 | |||
917 | @example | ||
918 | ERROR 2002: Can't connect to local MySQL server | ||
919 | through socket '/tmp/mysql.sock' (2) | ||
920 | @end example | ||
921 | |||
922 | @noindent | ||
923 | it may be resolvable by | ||
924 | |||
925 | @example | ||
926 | ln -s /var/run/mysqld/mysqld.sock /tmp/mysql.sock | ||
927 | @end example | ||
928 | |||
929 | @noindent | ||
930 | so there may be some additional trouble depending on your mysql setup. | ||
931 | |||
932 | @node Performance Tuning | ||
933 | @subsection Performance Tuning | ||
934 | |||
935 | For GNUnet, you probably want to set the option | ||
936 | |||
937 | @example | ||
938 | innodb_flush_log_at_trx_commit = 0 | ||
939 | @end example | ||
940 | |||
941 | @noindent | ||
942 | for a rather dramatic boost in MySQL performance. However, this reduces | ||
943 | the "safety" of your database as with this options you may loose | ||
944 | transactions during a power outage. | ||
945 | While this is totally harmless for GNUnet, the option applies to all | ||
946 | applications using MySQL. So you should set it if (and only if) GNUnet is | ||
947 | the only application on your system using MySQL. | ||
948 | |||
949 | @node Setup for running Testcases | ||
950 | @subsection Setup for running Testcases | ||
951 | |||
952 | If you want to run the testcases, you must create a second database | ||
953 | "gnunetcheck" with the same username and password. This database will | ||
954 | then be used for testing (@command{make check}). | ||
955 | |||
956 | @node Configuring the Postgres database | ||
957 | @subsection Configuring the Postgres database | ||
958 | |||
959 | This text describes how to setup the Postgres database for GNUnet. | ||
960 | |||
961 | This Postgres plugin was developed for Postgres 8.3 but might work for | ||
962 | earlier versions as well. | ||
963 | |||
964 | @node Reasons to use Postgres | ||
965 | @subsection Reasons to use Postgres | ||
966 | |||
967 | @itemize @bullet | ||
968 | @item Easier to setup than MySQL | ||
969 | @item Real database | ||
970 | @end itemize | ||
971 | |||
972 | @node Reasons not to use Postgres | ||
973 | @subsection Reasons not to use Postgres | ||
974 | |||
975 | @itemize @bullet | ||
976 | @item Quite slow | ||
977 | @item Still some manual setup required | ||
978 | @end itemize | ||
979 | |||
980 | @node Manual setup instructions | ||
981 | @subsection Manual setup instructions | ||
982 | |||
983 | @itemize @bullet | ||
984 | @item In @file{gnunet.conf} set in section @code{DATASTORE} the value for | ||
985 | @code{DATABASE} to @code{postgres}. | ||
986 | @item Access Postgres to create a user: | ||
987 | |||
988 | @table @asis | ||
989 | @item with Postgres 8.x, use: | ||
990 | |||
991 | @example | ||
992 | # su - postgres | ||
993 | $ createuser | ||
994 | @end example | ||
995 | |||
996 | @noindent | ||
997 | and enter the name of the user running GNUnet for the role interactively. | ||
998 | Then, when prompted, do not set it to superuser, allow the creation of | ||
999 | databases, and do not allow the creation of new roles. | ||
1000 | |||
1001 | @item with Postgres 9.x, use: | ||
1002 | |||
1003 | @example | ||
1004 | # su - postgres | ||
1005 | $ createuser -d $GNUNET_USER | ||
1006 | @end example | ||
1007 | |||
1008 | @noindent | ||
1009 | where $GNUNET_USER is the name of the user running GNUnet. | ||
1010 | |||
1011 | @end table | ||
1012 | |||
1013 | |||
1014 | @item | ||
1015 | As that user (so typically as user "gnunet"), create a database (or two): | ||
1016 | |||
1017 | @example | ||
1018 | $ createdb gnunet | ||
1019 | # this way you can run "make check" | ||
1020 | $ createdb gnunetcheck | ||
1021 | @end example | ||
1022 | |||
1023 | @end itemize | ||
1024 | |||
1025 | Now you should be able to start @code{gnunet-arm}. | ||
1026 | |||
1027 | @node Testing the setup manually | ||
1028 | @subsection Testing the setup manually | ||
1029 | |||
1030 | You may want to try if the database connection works. First, again login | ||
1031 | as the user who will run @command{gnunet-arm}. Then use: | ||
1032 | |||
1033 | @example | ||
1034 | $ psql gnunet # or gnunetcheck | ||
1035 | gnunet=> \dt | ||
1036 | @end example | ||
1037 | |||
1038 | @noindent | ||
1039 | If, after you have started @command{gnunet-arm} at least once, you get | ||
1040 | a @code{gn090} table here, it probably works. | ||
1041 | |||
1042 | @node Configuring the datacache | ||
1043 | @subsection Configuring the datacache | ||
1044 | |||
1045 | |||
1046 | The datacache is what GNUnet uses for storing temporary data. This data is | ||
1047 | expected to be wiped completely each time GNUnet is restarted (or the | ||
1048 | system is rebooted). | ||
1049 | |||
1050 | You need to specify how many bytes GNUnet is allowed to use for the | ||
1051 | datacache using the @code{QUOTA} option in the section @code{[dhtcache]}. | ||
1052 | Furthermore, you need to specify which database backend should be used to | ||
1053 | store the data. Currently, you have the choice between | ||
1054 | sqLite, MySQL and Postgres. | ||
1055 | |||
1056 | @node Configuring the file-sharing service | ||
1057 | @subsection Configuring the file-sharing service | ||
1058 | |||
1059 | In order to use GNUnet for file-sharing, you first need to make sure | ||
1060 | that the file-sharing service is loaded. | ||
1061 | This is done by setting the @code{START_ON_DEMAND} option in | ||
1062 | section @code{[fs]} to "YES". Alternatively, you can run | ||
1063 | |||
1064 | @example | ||
1065 | $ gnunet-arm -i fs | ||
1066 | @end example | ||
1067 | |||
1068 | @noindent | ||
1069 | to start the file-sharing service by hand. | ||
1070 | |||
1071 | Except for configuring the database and the datacache the only important | ||
1072 | option for file-sharing is content migration. | ||
1073 | |||
1074 | Content migration allows your peer to cache content from other peers as | ||
1075 | well as send out content stored on your system without explicit requests. | ||
1076 | This content replication has positive and negative impacts on both system | ||
1077 | performance and privacy. | ||
1078 | |||
1079 | FIXME: discuss the trade-offs. Here is some older text about it... | ||
1080 | |||
1081 | Setting this option to YES allows gnunetd to migrate data to the local | ||
1082 | machine. Setting this option to YES is highly recommended for efficiency. | ||
1083 | Its also the default. If you set this value to YES, GNUnet will store | ||
1084 | content on your machine that you cannot decrypt. | ||
1085 | While this may protect you from liability if the judge is sane, it may | ||
1086 | not (IANAL). If you put illegal content on your machine yourself, setting | ||
1087 | this option to YES will probably increase your chances to get away with it | ||
1088 | since you can plausibly deny that you inserted the content. | ||
1089 | Note that in either case, your anonymity would have to be broken first | ||
1090 | (which may be possible depending on the size of the GNUnet network and the | ||
1091 | strength of the adversary). | ||
1092 | |||
1093 | @node Configuring logging | ||
1094 | @subsection Configuring logging | ||
1095 | |||
1096 | Since version 0.9.0, logging in GNUnet is controlled via the | ||
1097 | @code{-L} and @code{-l} options. | ||
1098 | Using @code{-L}, a log level can be specified. With log level | ||
1099 | @code{ERROR} only serious errors are logged. | ||
1100 | The default log level is @code{WARNING} which causes anything of | ||
1101 | concern to be logged. | ||
1102 | Log level @code{INFO} can be used to log anything that might be | ||
1103 | interesting information whereas | ||
1104 | @code{DEBUG} can be used by developers to log debugging messages | ||
1105 | (but you need to run @code{./configure} with | ||
1106 | @code{--enable-logging=verbose} to get them compiled). | ||
1107 | The @code{-l} option is used to specify the log file. | ||
1108 | |||
1109 | Since most GNUnet services are managed by @code{gnunet-arm}, using the | ||
1110 | @code{-l} or @code{-L} options directly is not possible. | ||
1111 | Instead, they can be specified using the @code{OPTIONS} configuration | ||
1112 | value in the respective section for the respective service. | ||
1113 | In order to enable logging globally without editing the @code{OPTIONS} | ||
1114 | values for each service, @command{gnunet-arm} supports a | ||
1115 | @code{GLOBAL_POSTFIX} option. | ||
1116 | The value specified here is given as an extra option to all services for | ||
1117 | which the configuration does contain a service-specific @code{OPTIONS} | ||
1118 | field. | ||
1119 | |||
1120 | @code{GLOBAL_POSTFIX} can contain the special sequence "@{@}" which | ||
1121 | is replaced by the name of the service that is being started. | ||
1122 | Furthermore, @code{GLOBAL_POSTFIX} is special in that sequences | ||
1123 | starting with "$" anywhere in the string are expanded (according | ||
1124 | to options in @code{PATHS}); this expansion otherwise is | ||
1125 | only happening for filenames and then the "$" must be the | ||
1126 | first character in the option. Both of these restrictions do | ||
1127 | not apply to @code{GLOBAL_POSTFIX}. | ||
1128 | Note that specifying @code{%} anywhere in the @code{GLOBAL_POSTFIX} | ||
1129 | disables both of these features. | ||
1130 | |||
1131 | In summary, in order to get all services to log at level | ||
1132 | @code{INFO} to log-files called @code{SERVICENAME-logs}, the | ||
1133 | following global prefix should be used: | ||
1134 | |||
1135 | @example | ||
1136 | GLOBAL_POSTFIX = -l $SERVICEHOME/@{@}-logs -L INFO | ||
1137 | @end example | ||
1138 | |||
1139 | @node Configuring the transport service and plugins | ||
1140 | @subsection Configuring the transport service and plugins | ||
1141 | |||
1142 | The transport service in GNUnet is responsible to maintain basic | ||
1143 | connectivity to other peers. | ||
1144 | Besides initiating and keeping connections alive it is also responsible | ||
1145 | for address validation. | ||
1146 | |||
1147 | The GNUnet transport supports more than one transport protocol. | ||
1148 | These protocols are configured together with the transport service. | ||
1149 | |||
1150 | The configuration section for the transport service itself is quite | ||
1151 | similar to all the other services | ||
1152 | |||
1153 | @example | ||
1154 | START_ON_DEMAND = YES | ||
1155 | @@UNIXONLY@@ PORT = 2091 | ||
1156 | HOSTNAME = localhost | ||
1157 | HOME = $SERVICEHOME | ||
1158 | CONFIG = $DEFAULTCONFIG | ||
1159 | BINARY = gnunet-service-transport | ||
1160 | #PREFIX = valgrind | ||
1161 | NEIGHBOUR_LIMIT = 50 | ||
1162 | ACCEPT_FROM = 127.0.0.1; | ||
1163 | ACCEPT_FROM6 = ::1; | ||
1164 | PLUGINS = tcp udp | ||
1165 | UNIXPATH = /tmp/gnunet-service-transport.sock | ||
1166 | @end example | ||
1167 | |||
1168 | Different are the settings for the plugins to load @code{PLUGINS}. | ||
1169 | The first setting specifies which transport plugins to load. | ||
1170 | |||
1171 | @itemize @bullet | ||
1172 | @item transport-unix | ||
1173 | A plugin for local only communication with UNIX domain sockets. Used for | ||
1174 | testing and available on unix systems only. Just set the port | ||
1175 | |||
1176 | @example | ||
1177 | [transport-unix] | ||
1178 | PORT = 22086 | ||
1179 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1180 | @end example | ||
1181 | |||
1182 | @item transport-tcp | ||
1183 | A plugin for communication with TCP. Set port to 0 for client mode with | ||
1184 | outbound only connections | ||
1185 | |||
1186 | @example | ||
1187 | [transport-tcp] | ||
1188 | # Use 0 to ONLY advertise as a peer behind NAT (no port binding) | ||
1189 | PORT = 2086 | ||
1190 | ADVERTISED_PORT = 2086 | ||
1191 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1192 | # Maximum number of open TCP connections allowed | ||
1193 | MAX_CONNECTIONS = 128 | ||
1194 | @end example | ||
1195 | |||
1196 | @item transport-udp | ||
1197 | A plugin for communication with UDP. Supports peer discovery using | ||
1198 | broadcasts. | ||
1199 | |||
1200 | @example | ||
1201 | [transport-udp] | ||
1202 | PORT = 2086 | ||
1203 | BROADCAST = YES | ||
1204 | BROADCAST_INTERVAL = 30 s | ||
1205 | MAX_BPS = 1000000 | ||
1206 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1207 | @end example | ||
1208 | |||
1209 | @item transport-http | ||
1210 | HTTP and HTTPS support is split in two part: a client plugin initiating | ||
1211 | outbound connections and a server part accepting connections from the | ||
1212 | client. The client plugin just takes the maximum number of connections as | ||
1213 | an argument. | ||
1214 | |||
1215 | @example | ||
1216 | [transport-http_client] | ||
1217 | MAX_CONNECTIONS = 128 | ||
1218 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1219 | @end example | ||
1220 | |||
1221 | @example | ||
1222 | [transport-https_client] | ||
1223 | MAX_CONNECTIONS = 128 | ||
1224 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1225 | @end example | ||
1226 | |||
1227 | @noindent | ||
1228 | The server has a port configured and the maximum number of connections. | ||
1229 | The HTTPS part has two files with the certificate key and the certificate | ||
1230 | file. | ||
1231 | |||
1232 | The server plugin supports reverse proxies, so a external hostname can be | ||
1233 | set using the @code{EXTERNAL_HOSTNAME} setting. | ||
1234 | The webserver under this address should forward the request to the peer | ||
1235 | and the configure port. | ||
1236 | |||
1237 | @example | ||
1238 | [transport-http_server] | ||
1239 | EXTERNAL_HOSTNAME = fulcrum.net.in.tum.de/gnunet | ||
1240 | PORT = 1080 | ||
1241 | MAX_CONNECTIONS = 128 | ||
1242 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1243 | @end example | ||
1244 | |||
1245 | @example | ||
1246 | [transport-https_server] | ||
1247 | PORT = 4433 | ||
1248 | CRYPTO_INIT = NORMAL | ||
1249 | KEY_FILE = https.key | ||
1250 | CERT_FILE = https.cert | ||
1251 | MAX_CONNECTIONS = 128 | ||
1252 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1253 | @end example | ||
1254 | |||
1255 | @item transport-wlan | ||
1256 | |||
1257 | The next section describes how to setup the WLAN plugin, | ||
1258 | so here only the settings. Just specify the interface to use: | ||
1259 | |||
1260 | @example | ||
1261 | [transport-wlan] | ||
1262 | # Name of the interface in monitor mode (typically monX) | ||
1263 | INTERFACE = mon0 | ||
1264 | # Real hardware, no testing | ||
1265 | TESTMODE = 0 | ||
1266 | TESTING_IGNORE_KEYS = ACCEPT_FROM; | ||
1267 | @end example | ||
1268 | @end itemize | ||
1269 | |||
1270 | @node Configuring the WLAN transport plugin | ||
1271 | @subsection Configuring the WLAN transport plugin | ||
1272 | |||
1273 | The wlan transport plugin enables GNUnet to send and to receive data on a | ||
1274 | wlan interface. | ||
1275 | It has not to be connected to a wlan network as long as sender and | ||
1276 | receiver are on the same channel. This enables you to get connection to | ||
1277 | GNUnet where no internet access is possible, for example during | ||
1278 | catastrophes or when censorship cuts you off from the internet. | ||
1279 | |||
1280 | |||
1281 | @menu | ||
1282 | * Requirements for the WLAN plugin:: | ||
1283 | * Configuration:: | ||
1284 | * Before starting GNUnet:: | ||
1285 | * Limitations and known bugs:: | ||
1286 | @end menu | ||
1287 | |||
1288 | |||
1289 | @node Requirements for the WLAN plugin | ||
1290 | @subsubsection Requirements for the WLAN plugin | ||
1291 | |||
1292 | @itemize @bullet | ||
1293 | |||
1294 | @item wlan network card with monitor support and packet injection | ||
1295 | (see @uref{http://www.aircrack-ng.org/, aircrack-ng.org}) | ||
1296 | |||
1297 | @item Linux kernel with mac80211 stack, introduced in 2.6.22, tested with | ||
1298 | 2.6.35 and 2.6.38 | ||
1299 | |||
1300 | @item Wlantools to create the a monitor interface, tested with airmon-ng | ||
1301 | of the aircrack-ng package | ||
1302 | @end itemize | ||
1303 | |||
1304 | @node Configuration | ||
1305 | @subsubsection Configuration | ||
1306 | |||
1307 | There are the following options for the wlan plugin (they should be like | ||
1308 | this in your default config file, you only need to adjust them if the | ||
1309 | values are incorrect for your system) | ||
1310 | |||
1311 | @example | ||
1312 | # section for the wlan transport plugin | ||
1313 | [transport-wlan] | ||
1314 | # interface to use, more information in the | ||
1315 | # "Before starting GNUnet" section of the handbook. | ||
1316 | INTERFACE = mon0 | ||
1317 | # testmode for developers: | ||
1318 | # 0 use wlan interface, | ||
1319 | #1 or 2 use loopback driver for tests 1 = server, 2 = client | ||
1320 | TESTMODE = 0 | ||
1321 | @end example | ||
1322 | |||
1323 | @node Before starting GNUnet | ||
1324 | @subsubsection Before starting GNUnet | ||
1325 | |||
1326 | Before starting GNUnet, you have to make sure that your wlan interface is | ||
1327 | in monitor mode. | ||
1328 | One way to put the wlan interface into monitor mode (if your interface | ||
1329 | name is wlan0) is by executing: | ||
1330 | |||
1331 | @example | ||
1332 | sudo airmon-ng start wlan0 | ||
1333 | @end example | ||
1334 | |||
1335 | @noindent | ||
1336 | Here is an example what the result should look like: | ||
1337 | |||
1338 | @example | ||
1339 | Interface Chipset Driver | ||
1340 | wlan0 Intel 4965 a/b/g/n iwl4965 - [phy0] | ||
1341 | (monitor mode enabled on mon0) | ||
1342 | @end example | ||
1343 | |||
1344 | @noindent | ||
1345 | The monitor interface is mon0 is the one that you have to put into the | ||
1346 | configuration file. | ||
1347 | |||
1348 | @node Limitations and known bugs | ||
1349 | @subsubsection Limitations and known bugs | ||
1350 | |||
1351 | Wlan speed is at the maximum of 1 Mbit/s because support for choosing the | ||
1352 | wlan speed with packet injection was removed in newer kernels. | ||
1353 | Please pester the kernel developers about fixing this. | ||
1354 | |||
1355 | The interface channel depends on the wlan network that the card is | ||
1356 | connected to. If no connection has been made since the start of the | ||
1357 | computer, it is usually the first channel of the card. | ||
1358 | Peers will only find each other and communicate if they are on the same | ||
1359 | channel. Channels must be set manually, e.g. by using: | ||
1360 | |||
1361 | @example | ||
1362 | iwconfig wlan0 channel 1 | ||
1363 | @end example | ||
1364 | |||
1365 | @node Configuring HTTP(S) reverse proxy functionality using Apache or nginx | ||
1366 | @subsection Configuring HTTP(S) reverse proxy functionality using Apache or nginx | ||
1367 | |||
1368 | The HTTP plugin supports data transfer using reverse proxies. A reverse | ||
1369 | proxy forwards the HTTP request he receives with a certain URL to another | ||
1370 | webserver, here a GNUnet peer. | ||
1371 | |||
1372 | So if you have a running Apache or nginx webserver you can configure it to | ||
1373 | be a GNUnet reverse proxy. Especially if you have a well-known website | ||
1374 | this improves censorship resistance since it looks as normal surfing | ||
1375 | behaviour. | ||
1376 | |||
1377 | To do so, you have to do two things: | ||
1378 | |||
1379 | @itemize @bullet | ||
1380 | @item Configure your webserver to forward the GNUnet HTTP traffic | ||
1381 | @item Configure your GNUnet peer to announce the respective address | ||
1382 | @end itemize | ||
1383 | |||
1384 | As an example we want to use GNUnet peer running: | ||
1385 | |||
1386 | @itemize @bullet | ||
1387 | |||
1388 | @item HTTP server plugin on @code{gnunet.foo.org:1080} | ||
1389 | |||
1390 | @item HTTPS server plugin on @code{gnunet.foo.org:4433} | ||
1391 | |||
1392 | @item A apache or nginx webserver on | ||
1393 | @uref{http://www.foo.org/, http://www.foo.org:80/} | ||
1394 | |||
1395 | @item A apache or nginx webserver on https://www.foo.org:443/ | ||
1396 | @end itemize | ||
1397 | |||
1398 | And we want the webserver to accept GNUnet traffic under | ||
1399 | @code{http://www.foo.org/bar/}. The required steps are described here: | ||
1400 | |||
1401 | @menu | ||
1402 | * Reverse Proxy - Configure your Apache2 HTTP webserver:: | ||
1403 | * Reverse Proxy - Configure your Apache2 HTTPS webserver:: | ||
1404 | * Reverse Proxy - Configure your nginx HTTPS webserver:: | ||
1405 | * Reverse Proxy - Configure your nginx HTTP webserver:: | ||
1406 | * Reverse Proxy - Configure your GNUnet peer:: | ||
1407 | @end menu | ||
1408 | |||
1409 | @node Reverse Proxy - Configure your Apache2 HTTP webserver | ||
1410 | @subsubsection Reverse Proxy - Configure your Apache2 HTTP webserver | ||
1411 | |||
1412 | First of all you need mod_proxy installed. | ||
1413 | |||
1414 | Edit your webserver configuration. Edit | ||
1415 | @code{/etc/apache2/apache2.conf} or the site-specific configuration file. | ||
1416 | |||
1417 | In the respective @code{server config},@code{virtual host} or | ||
1418 | @code{directory} section add the following lines: | ||
1419 | |||
1420 | @example | ||
1421 | ProxyTimeout 300 | ||
1422 | ProxyRequests Off | ||
1423 | <Location /bar/ > | ||
1424 | ProxyPass http://gnunet.foo.org:1080/ | ||
1425 | ProxyPassReverse http://gnunet.foo.org:1080/ | ||
1426 | </Location> | ||
1427 | @end example | ||
1428 | |||
1429 | @node Reverse Proxy - Configure your Apache2 HTTPS webserver | ||
1430 | @subsubsection Reverse Proxy - Configure your Apache2 HTTPS webserver | ||
1431 | |||
1432 | We assume that you already have an HTTPS server running, if not please | ||
1433 | check how to configure a HTTPS host. An uncomplicated to use example | ||
1434 | is the example configuration file for Apache2/HTTPD provided in | ||
1435 | @file{apache2/sites-available/default-ssl}. | ||
1436 | |||
1437 | In the respective HTTPS @code{server config},@code{virtual host} or | ||
1438 | @code{directory} section add the following lines: | ||
1439 | |||
1440 | @example | ||
1441 | SSLProxyEngine On | ||
1442 | ProxyTimeout 300 | ||
1443 | ProxyRequests Off | ||
1444 | <Location /bar/ > | ||
1445 | ProxyPass https://gnunet.foo.org:4433/ | ||
1446 | ProxyPassReverse https://gnunet.foo.org:4433/ | ||
1447 | </Location> | ||
1448 | @end example | ||
1449 | |||
1450 | @noindent | ||
1451 | More information about the apache mod_proxy configuration can be found | ||
1452 | in the | ||
1453 | @uref{http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass, Apache documentation}. | ||
1454 | |||
1455 | @node Reverse Proxy - Configure your nginx HTTPS webserver | ||
1456 | @subsubsection Reverse Proxy - Configure your nginx HTTPS webserver | ||
1457 | |||
1458 | Since nginx does not support chunked encoding, you first of all have to | ||
1459 | install the @code{chunkin} | ||
1460 | @uref{http://wiki.nginx.org/HttpChunkinModule, module}. | ||
1461 | |||
1462 | To enable chunkin add: | ||
1463 | |||
1464 | @example | ||
1465 | chunkin on; | ||
1466 | error_page 411 = @@my_411_error; | ||
1467 | location @@my_411_error @{ | ||
1468 | chunkin_resume; | ||
1469 | @} | ||
1470 | @end example | ||
1471 | |||
1472 | @noindent | ||
1473 | Edit your webserver configuration. Edit @file{/etc/nginx/nginx.conf} or | ||
1474 | the site-specific configuration file. | ||
1475 | |||
1476 | In the @code{server} section add: | ||
1477 | |||
1478 | @example | ||
1479 | location /bar/ @{ | ||
1480 | proxy_pass http://gnunet.foo.org:1080/; | ||
1481 | proxy_buffering off; | ||
1482 | proxy_connect_timeout 5; # more than http_server | ||
1483 | proxy_read_timeout 350; # 60 default, 300s is GNUnet's idle timeout | ||
1484 | proxy_http_version 1.1; # 1.0 default | ||
1485 | proxy_next_upstream error timeout invalid_header http_500 http_503 http_502 http_504; | ||
1486 | @} | ||
1487 | @end example | ||
1488 | |||
1489 | @node Reverse Proxy - Configure your nginx HTTP webserver | ||
1490 | @subsubsection Reverse Proxy - Configure your nginx HTTP webserver | ||
1491 | |||
1492 | Edit your webserver configuration. Edit @file{/etc/nginx/nginx.conf} or | ||
1493 | the site-specific configuration file. | ||
1494 | |||
1495 | In the @code{server} section add: | ||
1496 | |||
1497 | @example | ||
1498 | ssl_session_timeout 6m; | ||
1499 | location /bar/ | ||
1500 | @{ | ||
1501 | proxy_pass https://gnunet.foo.org:4433/; | ||
1502 | proxy_buffering off; | ||
1503 | proxy_connect_timeout 5; # more than http_server | ||
1504 | proxy_read_timeout 350; # 60 default, 300s is GNUnet's idle timeout | ||
1505 | proxy_http_version 1.1; # 1.0 default | ||
1506 | proxy_next_upstream error timeout invalid_header http_500 http_503 http_502 http_504; | ||
1507 | @} | ||
1508 | @end example | ||
1509 | |||
1510 | @node Reverse Proxy - Configure your GNUnet peer | ||
1511 | @subsubsection Reverse Proxy - Configure your GNUnet peer | ||
1512 | |||
1513 | To have your GNUnet peer announce the address, you have to specify the | ||
1514 | @code{EXTERNAL_HOSTNAME} option in the @code{[transport-http_server]} | ||
1515 | section: | ||
1516 | |||
1517 | @example | ||
1518 | [transport-http_server] | ||
1519 | EXTERNAL_HOSTNAME = http://www.foo.org/bar/ | ||
1520 | @end example | ||
1521 | |||
1522 | @noindent | ||
1523 | and/or @code{[transport-https_server]} section: | ||
1524 | |||
1525 | @example | ||
1526 | [transport-https_server] | ||
1527 | EXTERNAL_HOSTNAME = https://www.foo.org/bar/ | ||
1528 | @end example | ||
1529 | |||
1530 | @noindent | ||
1531 | Now restart your webserver and your peer... | ||
1532 | |||
1533 | @node Blacklisting peers | ||
1534 | @subsection Blacklisting peers | ||
1535 | |||
1536 | Transport service supports to deny connecting to a specific peer of to a | ||
1537 | specific peer with a specific transport plugin using the blacklisting | ||
1538 | component of transport service. With@ blacklisting it is possible to deny | ||
1539 | connections to specific peers of@ to use a specific plugin to a specific | ||
1540 | peer. Peers can be blacklisted using@ the configuration or a blacklist | ||
1541 | client can be asked. | ||
1542 | |||
1543 | To blacklist peers using the configuration you have to add a section to | ||
1544 | your configuration containing the peer id of the peer to blacklist and | ||
1545 | the plugin@ if required. | ||
1546 | |||
1547 | Examples: | ||
1548 | |||
1549 | To blacklist connections to P565... on peer AG2P... using tcp add: | ||
1550 | |||
1551 | @c FIXME: This is too long and produces errors in the pdf. | ||
1552 | @example | ||
1553 | [transport-blacklist AG2PHES1BARB9IJCPAMJTFPVJ5V3A72S3F2A8SBUB8DAQ2V0O3V8G6G2JU56FHGFOHMQVKBSQFV98TCGTC3RJ1NINP82G0RC00N1520] | ||
1554 | P565723JO1C2HSN6J29TAQ22MN6CI8HTMUU55T0FUQG4CMDGGEQ8UCNBKUMB94GC8R9G4FB2SF9LDOBAJ6AMINBP4JHHDD6L7VD801G = tcp | ||
1555 | @end example | ||
1556 | |||
1557 | To blacklist connections to P565... on peer AG2P... using all plugins add: | ||
1558 | |||
1559 | @example | ||
1560 | [transport-blacklist-AG2PHES1BARB9IJCPAMJTFPVJ5V3A72S3F2A8SBUB8DAQ2V0O3V8G6G2JU56FHGFOHMQVKBSQFV98TCGTC3RJ1NINP82G0RC00N1520] | ||
1561 | P565723JO1C2HSN6J29TAQ22MN6CI8HTMUU55T0FUQG4CMDGGEQ8UCNBKUMB94GC8R9G4FB2SF9LDOBAJ6AMINBP4JHHDD6L7VD801G = | ||
1562 | @end example | ||
1563 | |||
1564 | You can also add a blacklist client using the blacklist API. On a | ||
1565 | blacklist check, blacklisting first checks internally if the peer is | ||
1566 | blacklisted and if not, it asks the blacklisting clients. Clients are | ||
1567 | asked if it is OK to connect to a peer ID, the plugin is omitted. | ||
1568 | |||
1569 | On blacklist check for (peer, plugin) | ||
1570 | |||
1571 | @itemize @bullet | ||
1572 | @item Do we have a local blacklist entry for this peer and this plugin? | ||
1573 | @item YES: disallow connection | ||
1574 | @item Do we have a local blacklist entry for this peer and all plugins? | ||
1575 | @item YES: disallow connection | ||
1576 | @item Does one of the clients disallow? | ||
1577 | @item YES: disallow connection | ||
1578 | @end itemize | ||
1579 | |||
1580 | @node Configuration of the HTTP and HTTPS transport plugins | ||
1581 | @subsection Configuration of the HTTP and HTTPS transport plugins | ||
1582 | |||
1583 | The client parts of the http and https transport plugins can be configured | ||
1584 | to use a proxy to connect to the hostlist server. This functionality can | ||
1585 | be configured in the configuration file directly or using the | ||
1586 | gnunet-setup tool. | ||
1587 | |||
1588 | Both the HTTP and HTTPS clients support the following proxy types at | ||
1589 | the moment: | ||
1590 | |||
1591 | @itemize @bullet | ||
1592 | @item HTTP 1.1 proxy | ||
1593 | @item SOCKS 4/4a/5/5 with hostname | ||
1594 | @end itemize | ||
1595 | |||
1596 | In addition authentication at the proxy with username and password can be | ||
1597 | configured. | ||
1598 | |||
1599 | To configure proxy support for the clients in the gnunet-setup tool, | ||
1600 | select the "transport" tab and activate the respective plugin. Now you | ||
1601 | can select the appropriate proxy type. The hostname or IP address | ||
1602 | (including port if required) has to be entered in the "Proxy hostname" | ||
1603 | textbox. If required, enter username and password in the "Proxy username" | ||
1604 | and "Proxy password" boxes. Be aware that these information will be stored | ||
1605 | in the configuration in plain text. | ||
1606 | |||
1607 | To configure these options directly in the configuration, you can | ||
1608 | configure the following settings in the @code{[transport-http_client]} | ||
1609 | and @code{[transport-https_client]} section of the configuration: | ||
1610 | |||
1611 | @example | ||
1612 | # Type of proxy server, | ||
1613 | # Valid values: HTTP, SOCKS4, SOCKS5, SOCKS4A, SOCKS5_HOSTNAME | ||
1614 | # Default: HTTP | ||
1615 | # PROXY_TYPE = HTTP | ||
1616 | |||
1617 | # Hostname or IP of proxy server | ||
1618 | # PROXY = | ||
1619 | # User name for proxy server | ||
1620 | # PROXY_USERNAME = | ||
1621 | # User password for proxy server | ||
1622 | # PROXY_PASSWORD = | ||
1623 | @end example | ||
1624 | |||
1625 | @node Configuring the GNU Name System | ||
1626 | @subsection Configuring the GNU Name System | ||
1627 | |||
1628 | @menu | ||
1629 | * Configuring system-wide DNS interception:: | ||
1630 | * Configuring the GNS nsswitch plugin:: | ||
1631 | @c * Configuring GNS on W32:: | ||
1632 | * GNS Proxy Setup:: | ||
1633 | * Setup of the GNS CA:: | ||
1634 | * Testing the GNS setup:: | ||
1635 | * Migrating existing DNS zones into GNS:: | ||
1636 | @end menu | ||
1637 | |||
1638 | |||
1639 | @node Configuring system-wide DNS interception | ||
1640 | @subsubsection Configuring system-wide DNS interception | ||
1641 | |||
1642 | Before you install GNUnet, make sure you have a user and group 'gnunet' | ||
1643 | as well as an empty group 'gnunetdns'. | ||
1644 | |||
1645 | When using GNUnet with system-wide DNS interception, it is absolutely | ||
1646 | necessary for all GNUnet service processes to be started by | ||
1647 | @code{gnunet-service-arm} as user and group 'gnunet'. You also need to be | ||
1648 | sure to run @code{make install} as root (or use the @code{sudo} option to | ||
1649 | configure) to grant GNUnet sufficient privileges. | ||
1650 | |||
1651 | With this setup, all that is required for enabling system-wide DNS | ||
1652 | interception is for some GNUnet component (VPN or GNS) to request it. | ||
1653 | The @code{gnunet-service-dns} will then start helper programs that will | ||
1654 | make the necessary changes to your firewall (@code{iptables}) rules. | ||
1655 | |||
1656 | Note that this will NOT work if your system sends out DNS traffic to a | ||
1657 | link-local IPv6 address, as in this case GNUnet can intercept the traffic, | ||
1658 | but not inject the responses from the link-local IPv6 address. Hence you | ||
1659 | cannot use system-wide DNS interception in conjunction with link-local | ||
1660 | IPv6-based DNS servers. If such a DNS server is used, it will bypass | ||
1661 | GNUnet's DNS traffic interception. | ||
1662 | |||
1663 | Using the GNU Name System (GNS) requires two different configuration | ||
1664 | steps. | ||
1665 | First of all, GNS needs to be integrated with the operating system. Most | ||
1666 | of this section is about the operating system level integration. | ||
1667 | |||
1668 | The remainder of this chapter will detail the various methods for | ||
1669 | configuring the use of GNS with your operating system. | ||
1670 | |||
1671 | At this point in time you have different options depending on your OS: | ||
1672 | |||
1673 | @itemize @bullet | ||
1674 | @item Use the gnunet-gns-proxy@* | ||
1675 | This approach works for all operating systems and is likely the | ||
1676 | easiest. However, it enables GNS only for browsers, not for other | ||
1677 | applications that might be using DNS, such as SSH. Still, using the | ||
1678 | proxy is required for using HTTP with GNS and is thus recommended for | ||
1679 | all users. To do this, you simply have to run the | ||
1680 | @code{gnunet-gns-proxy-setup-ca} script as the user who will run the | ||
1681 | browser (this will create a GNS certificate authority (CA) on your | ||
1682 | system and import its key into your browser), then start | ||
1683 | @code{gnunet-gns-proxy} and inform your browser to use the Socks5 | ||
1684 | proxy which @code{gnunet-gns-proxy} makes available by default on port | ||
1685 | 7777. | ||
1686 | @item Use a nsswitch plugin (recommended on GNU systems)@* | ||
1687 | This approach has the advantage of offering fully personalized | ||
1688 | resolution even on multi-user systems. A potential disadvantage is | ||
1689 | that some applications might be able to bypass GNS. | ||
1690 | @item Use a W32 resolver plugin (recommended on W32)@* | ||
1691 | This is currently the only option on W32 systems. | ||
1692 | @item Use system-wide DNS packet interception@* | ||
1693 | This approach is recommended for the GNUnet VPN. It can be used to | ||
1694 | handle GNS at the same time; however, if you only use this method, you | ||
1695 | will only get one root zone per machine (not so great for multi-user | ||
1696 | systems). | ||
1697 | @end itemize | ||
1698 | |||
1699 | You can combine system-wide DNS packet interception with the nsswitch | ||
1700 | plugin. | ||
1701 | The setup of the system-wide DNS interception is described here. All of | ||
1702 | the other GNS-specific configuration steps are described in the following | ||
1703 | sections. | ||
1704 | |||
1705 | @node Configuring the GNS nsswitch plugin | ||
1706 | @subsubsection Configuring the GNS nsswitch plugin | ||
1707 | |||
1708 | The Name Service Switch (NSS) is a facility in Unix-like operating systems | ||
1709 | (in most cases provided by the GNU C Library) | ||
1710 | that provides a variety of sources for common configuration databases and | ||
1711 | name resolution mechanisms. | ||
1712 | A superuser (system administrator) usually configures the | ||
1713 | operating system's name services using the file | ||
1714 | @file{/etc/nsswitch.conf}. | ||
1715 | |||
1716 | GNS provides a NSS plugin to integrate GNS name resolution with the | ||
1717 | operating system's name resolution process. | ||
1718 | To use the GNS NSS plugin you have to either | ||
1719 | |||
1720 | @itemize @bullet | ||
1721 | @item install GNUnet as root or | ||
1722 | @item compile GNUnet with the @code{--with-sudo=yes} switch. | ||
1723 | @end itemize | ||
1724 | |||
1725 | Name resolution is controlled by the @emph{hosts} section in the NSS | ||
1726 | configuration. By default this section first performs a lookup in the | ||
1727 | @file{/etc/hosts} file and then in DNS. | ||
1728 | The nsswitch file should contain a line similar to: | ||
1729 | |||
1730 | @example | ||
1731 | hosts: files dns [NOTFOUND=return] mdns4_minimal mdns4 | ||
1732 | @end example | ||
1733 | |||
1734 | @noindent | ||
1735 | Here the GNS NSS plugin can be added to perform a GNS lookup before | ||
1736 | performing a DNS lookup. | ||
1737 | The GNS NSS plugin has to be added to the "hosts" section in | ||
1738 | @file{/etc/nsswitch.conf} file before DNS related plugins: | ||
1739 | |||
1740 | @example | ||
1741 | ... | ||
1742 | hosts: files gns [NOTFOUND=return] dns mdns4_minimal mdns4 | ||
1743 | ... | ||
1744 | @end example | ||
1745 | |||
1746 | @noindent | ||
1747 | The @code{NOTFOUND=return} will ensure that if a @code{.gnu} name is not | ||
1748 | found in GNS it will not be queried in DNS. | ||
1749 | |||
1750 | @c @node Configuring GNS on W32 | ||
1751 | @c @subsubsection Configuring GNS on W32 | ||
1752 | |||
1753 | @c This document is a guide to configuring GNU Name System on W32-compatible | ||
1754 | @c platforms. | ||
1755 | |||
1756 | @c After GNUnet is installed, run the w32nsp-install tool: | ||
1757 | |||
1758 | @c @example | ||
1759 | @c w32nsp-install.exe libw32nsp-0.dll | ||
1760 | @c @end example | ||
1761 | |||
1762 | @c @noindent | ||
1763 | @c ('0' is the library version of W32 NSP; it might increase in the future, | ||
1764 | @c change the invocation accordingly). | ||
1765 | |||
1766 | @c This will install GNS namespace provider into the system and allow other | ||
1767 | @c applications to resolve names that end in '@strong{gnu}' | ||
1768 | @c and '@strong{zkey}'. Note that namespace provider requires | ||
1769 | @c gnunet-gns-helper-service-w32 to be running, as well as gns service | ||
1770 | @c itself (and its usual dependencies). | ||
1771 | |||
1772 | @c Namespace provider is hardcoded to connect to @strong{127.0.0.1:5353}, | ||
1773 | @c and this is where gnunet-gns-helper-service-w32 should be listening to | ||
1774 | @c (and is configured to listen to by default). | ||
1775 | |||
1776 | @c To uninstall the provider, run: | ||
1777 | |||
1778 | @c @example | ||
1779 | @c w32nsp-uninstall.exe | ||
1780 | @c @end example | ||
1781 | |||
1782 | @c @noindent | ||
1783 | @c (uses provider GUID to uninstall it, does not need a dll name). | ||
1784 | |||
1785 | @c Note that while MSDN claims that other applications will only be able to | ||
1786 | @c use the new namespace provider after re-starting, in reality they might | ||
1787 | @c stat to use it without that. Conversely, they might stop using the | ||
1788 | @c provider after it's been uninstalled, even if they were not re-started. | ||
1789 | @c W32 will not permit namespace provider library to be deleted or | ||
1790 | @c overwritten while the provider is installed, and while there is at least | ||
1791 | @c one process still using it (even after it was uninstalled). | ||
1792 | |||
1793 | @node GNS Proxy Setup | ||
1794 | @subsubsection GNS Proxy Setup | ||
1795 | |||
1796 | When using the GNU Name System (GNS) to browse the WWW, there are several | ||
1797 | issues that can be solved by adding the GNS Proxy to your setup: | ||
1798 | |||
1799 | @itemize @bullet | ||
1800 | |||
1801 | @item If the target website does not support GNS, it might assume that it | ||
1802 | is operating under some name in the legacy DNS system (such as | ||
1803 | example.com). It may then attempt to set cookies for that domain, and the | ||
1804 | web server might expect a @code{Host: example.com} header in the request | ||
1805 | from your browser. | ||
1806 | However, your browser might be using @code{example.gnu} for the | ||
1807 | @code{Host} header and might only accept (and send) cookies for | ||
1808 | @code{example.gnu}. The GNS Proxy will perform the necessary translations | ||
1809 | of the hostnames for cookies and HTTP headers (using the LEHO record for | ||
1810 | the target domain as the desired substitute). | ||
1811 | |||
1812 | @item If using HTTPS, the target site might include an SSL certificate | ||
1813 | which is either only valid for the LEHO domain or might match a TLSA | ||
1814 | record in GNS. However, your browser would expect a valid certificate for | ||
1815 | @code{example.gnu}, not for some legacy domain name. The proxy will | ||
1816 | validate the certificate (either against LEHO or TLSA) and then | ||
1817 | on-the-fly produce a valid certificate for the exchange, signed by your | ||
1818 | own CA. Assuming you installed the CA of your proxy in your browser's | ||
1819 | certificate authority list, your browser will then trust the | ||
1820 | HTTPS/SSL/TLS connection, as the hostname mismatch is hidden by the proxy. | ||
1821 | |||
1822 | @item Finally, the proxy will in the future indicate to the server that it | ||
1823 | speaks GNS, which will enable server operators to deliver GNS-enabled web | ||
1824 | sites to your browser (and continue to deliver legacy links to legacy | ||
1825 | browsers) | ||
1826 | @end itemize | ||
1827 | |||
1828 | @node Setup of the GNS CA | ||
1829 | @subsubsection Setup of the GNS CA | ||
1830 | |||
1831 | First you need to create a CA certificate that the proxy can use. | ||
1832 | To do so use the provided script gnunet-gns-proxy-ca: | ||
1833 | |||
1834 | @example | ||
1835 | $ gnunet-gns-proxy-setup-ca | ||
1836 | @end example | ||
1837 | |||
1838 | @noindent | ||
1839 | This will create a personal certification authority for you and add this | ||
1840 | authority to the firefox and chrome database. The proxy will use the this | ||
1841 | CA certificate to generate @code{*.gnu} client certificates on the fly. | ||
1842 | |||
1843 | Note that the proxy uses libcurl. Make sure your version of libcurl uses | ||
1844 | GnuTLS and NOT OpenSSL. The proxy will @b{not} work with libcurl compiled | ||
1845 | against OpenSSL. | ||
1846 | |||
1847 | You can check the configuration your libcurl was build with by | ||
1848 | running: | ||
1849 | |||
1850 | @example | ||
1851 | curl --version | ||
1852 | @end example | ||
1853 | |||
1854 | the output will look like this (without the linebreaks): | ||
1855 | |||
1856 | @example | ||
1857 | gnurl --version | ||
1858 | curl 7.56.0 (x86_64-unknown-linux-gnu) libcurl/7.56.0 \ | ||
1859 | GnuTLS/3.5.13 zlib/1.2.11 libidn2/2.0.4 | ||
1860 | Release-Date: 2017-10-08 | ||
1861 | Protocols: http https | ||
1862 | Features: AsynchDNS IDN IPv6 Largefile NTLM SSL libz \ | ||
1863 | TLS-SRP UnixSockets HTTPS-proxy | ||
1864 | @end example | ||
1865 | |||
1866 | @node Testing the GNS setup | ||
1867 | @subsubsection Testing the GNS setup | ||
1868 | |||
1869 | Now for testing purposes we can create some records in our zone to test | ||
1870 | the SSL functionality of the proxy: | ||
1871 | |||
1872 | @example | ||
1873 | $ gnunet-identity -C test | ||
1874 | $ gnunet-namestore -a -e "1 d" -n "homepage" \ | ||
1875 | -t A -V 131.159.74.67 -z test | ||
1876 | $ gnunet-namestore -a -e "1 d" -n "homepage" \ | ||
1877 | -t LEHO -V "gnunet.org" -z test | ||
1878 | @end example | ||
1879 | |||
1880 | @noindent | ||
1881 | At this point we can start the proxy. Simply execute | ||
1882 | |||
1883 | @example | ||
1884 | $ gnunet-arm -i gns-proxy | ||
1885 | @end example | ||
1886 | |||
1887 | To run the proxy at all times in the future, you should | ||
1888 | change your configuration as follows: | ||
1889 | |||
1890 | @example | ||
1891 | $ gnunet-config -s gns-proxy -o AUTOSTART -V YES | ||
1892 | @end example | ||
1893 | |||
1894 | @noindent | ||
1895 | Configure your browser to use this SOCKSv5 proxy using | ||
1896 | @code{localhost} on port 7777. | ||
1897 | If you use @command{Firefox} (or one of its derivatives/forks such as | ||
1898 | Icecat) you also have to go to @code{about:config} and set the key | ||
1899 | @code{network.proxy.socks_remote_dns} to @code{true}. | ||
1900 | |||
1901 | When you visit @code{https://homepage.test/}, you should get to the | ||
1902 | @code{https://gnunet.org/} frontpage and the browser (with the correctly | ||
1903 | configured proxy) should give you a valid SSL certificate for | ||
1904 | @code{homepage.gnu} and no warnings. It should look like this: | ||
1905 | |||
1906 | @c FIXME: Image does not exist, create it or save it from Drupal? | ||
1907 | @c @image{images/gnunethpgns.png,5in,, picture of homepage.gnu in Webbrowser} | ||
1908 | |||
1909 | @node Migrating existing DNS zones into GNS | ||
1910 | @subsubsection Migrating existing DNS zones into GNS | ||
1911 | |||
1912 | To migrate an existing zone into GNS use the Ascension tool. | ||
1913 | |||
1914 | Ascension transfers entire zones into GNS by doing incremental zone transfers | ||
1915 | and then adding the records to GNS. | ||
1916 | |||
1917 | Compared to the gnunet-zoneimport tool it strictly uses AXFR or IXFR depending | ||
1918 | on whether or not there exists a SOA record for the zone. If that is the case it | ||
1919 | will take the serial as a reference point and request the zone. The server will | ||
1920 | either answer the IXFR request with a correct incremental zone or with the | ||
1921 | entire zone, which depends on the server configuration. | ||
1922 | |||
1923 | You can find the source code here: @code{https://git.gnunet.org/ascension.git/} | ||
1924 | |||
1925 | The software can be installed into a Python virtual environment like this: | ||
1926 | @example | ||
1927 | $ python3 -m venv .venv | ||
1928 | $ source .venv/bin/activate | ||
1929 | $ python3 setup.py install | ||
1930 | @end example | ||
1931 | |||
1932 | Or installed globally like this: | ||
1933 | @example | ||
1934 | $ sudo python3 setup.py install | ||
1935 | @end example | ||
1936 | |||
1937 | Pip will then install all the necessary requirements that are needed to run | ||
1938 | Ascension. For development purposes a virtual environment should suffice. | ||
1939 | Keeping a virtual environment helps with keeping things tidy and prevents | ||
1940 | breaking of Ascension through a future Python update. | ||
1941 | |||
1942 | The advantage of using a virtual environment is, that all the dependencies can | ||
1943 | be installed separately in different versions without touching your systems | ||
1944 | Python installation and its dependencies. | ||
1945 | |||
1946 | Another way to install Ascension on Debian is to install the python3-ascension | ||
1947 | package. It can be found within the above mentioned Ascension git repository. | ||
1948 | This also adds a system user called ascension and runs a GNUnet peer in the | ||
1949 | background. Please note: This only works if a recent version of GNUnet is | ||
1950 | installed on your system. The version number of Ascension is chosen according | ||
1951 | to the required feature level of GNUnet: Ascension 0.11.5 is only | ||
1952 | compatible with GNUnet 0.11.5 or later and so on. | ||
1953 | As Debian's packages for GNUnet are outdated even in experimental, | ||
1954 | you will need to install GNUnet manually | ||
1955 | @xref{Installing GNUnet}. | ||
1956 | |||
1957 | Please check @xref{Migrating an existing DNS zone into GNS}, for usage manual | ||
1958 | of the tool. | ||
1959 | |||
1960 | @node Configuring the GNUnet VPN | ||
1961 | @subsection Configuring the GNUnet VPN | ||
1962 | |||
1963 | @menu | ||
1964 | * IPv4 address for interface:: | ||
1965 | * IPv6 address for interface:: | ||
1966 | * Configuring the GNUnet VPN DNS:: | ||
1967 | * Configuring the GNUnet VPN Exit Service:: | ||
1968 | * IP Address of external DNS resolver:: | ||
1969 | * IPv4 address for Exit interface:: | ||
1970 | * IPv6 address for Exit interface:: | ||
1971 | @end menu | ||
1972 | |||
1973 | Before configuring the GNUnet VPN, please make sure that system-wide DNS | ||
1974 | interception is configured properly as described in the section on the | ||
1975 | GNUnet DNS setup. @pxref{Configuring the GNU Name System}, | ||
1976 | if you haven't done so already. | ||
1977 | |||
1978 | The default options for the GNUnet VPN are usually sufficient to use | ||
1979 | GNUnet as a Layer 2 for your Internet connection. | ||
1980 | However, what you always have to specify is which IP protocol you want | ||
1981 | to tunnel: IPv4, IPv6 or both. | ||
1982 | Furthermore, if you tunnel both, you most likely should also tunnel | ||
1983 | all of your DNS requests. | ||
1984 | You theoretically can tunnel "only" your DNS traffic, but that usually | ||
1985 | makes little sense. | ||
1986 | |||
1987 | The other options as shown on the gnunet-setup tool are: | ||
1988 | |||
1989 | @node IPv4 address for interface | ||
1990 | @subsubsection IPv4 address for interface | ||
1991 | |||
1992 | This is the IPv4 address the VPN interface will get. You should pick a | ||
1993 | 'private' IPv4 network that is not yet in use for you system. For example, | ||
1994 | if you use @code{10.0.0.1/255.255.0.0} already, you might use | ||
1995 | @code{10.1.0.1/255.255.0.0}. | ||
1996 | If you use @code{10.0.0.1/255.0.0.0} already, then you might use | ||
1997 | @code{192.168.0.1/255.255.0.0}. | ||
1998 | If your system is not in a private IP-network, using any of the above will | ||
1999 | work fine. | ||
2000 | You should try to make the mask of the address big enough | ||
2001 | (@code{255.255.0.0} or, even better, @code{255.0.0.0}) to allow more | ||
2002 | mappings of remote IP Addresses into this range. | ||
2003 | However, even a @code{255.255.255.0} mask will suffice for most users. | ||
2004 | |||
2005 | @node IPv6 address for interface | ||
2006 | @subsubsection IPv6 address for interface | ||
2007 | |||
2008 | The IPv6 address the VPN interface will get. Here you can specify any | ||
2009 | non-link-local address (the address should not begin with @code{fe80:}). | ||
2010 | A subnet Unique Local Unicast (@code{fd00::/8} prefix) that you are | ||
2011 | currently not using would be a good choice. | ||
2012 | |||
2013 | @node Configuring the GNUnet VPN DNS | ||
2014 | @subsubsection Configuring the GNUnet VPN DNS | ||
2015 | |||
2016 | To resolve names for remote nodes, activate the DNS exit option. | ||
2017 | |||
2018 | @node Configuring the GNUnet VPN Exit Service | ||
2019 | @subsubsection Configuring the GNUnet VPN Exit Service | ||
2020 | |||
2021 | If you want to allow other users to share your Internet connection (yes, | ||
2022 | this may be dangerous, just as running a Tor exit node) or want to | ||
2023 | provide access to services on your host (this should be less dangerous, | ||
2024 | as long as those services are secure), you have to enable the GNUnet exit | ||
2025 | daemon. | ||
2026 | |||
2027 | You then get to specify which exit functions you want to provide. By | ||
2028 | enabling the exit daemon, you will always automatically provide exit | ||
2029 | functions for manually configured local services (this component of the | ||
2030 | system is under | ||
2031 | development and not documented further at this time). As for those | ||
2032 | services you explicitly specify the target IP address and port, there is | ||
2033 | no significant security risk in doing so. | ||
2034 | |||
2035 | Furthermore, you can serve as a DNS, IPv4 or IPv6 exit to the Internet. | ||
2036 | Being a DNS exit is usually pretty harmless. However, enabling IPv4 or | ||
2037 | IPv6-exit without further precautions may enable adversaries to access | ||
2038 | your local network, send spam, attack other systems from your Internet | ||
2039 | connection and do other mischiefs that will appear to come from your | ||
2040 | machine. This may or may not get you into legal trouble. | ||
2041 | If you want to allow IPv4 or IPv6-exit functionality, you should strongly | ||
2042 | consider adding additional firewall rules manually to protect your local | ||
2043 | network and to restrict outgoing TCP traffic (e.g. by not allowing access | ||
2044 | to port 25). While we plan to improve exit-filtering in the future, | ||
2045 | you're currently on your own here. | ||
2046 | Essentially, be prepared for any kind of IP-traffic to exit the respective | ||
2047 | TUN interface (and GNUnet will enable IP-forwarding and NAT for the | ||
2048 | interface automatically). | ||
2049 | |||
2050 | Additional configuration options of the exit as shown by the gnunet-setup | ||
2051 | tool are: | ||
2052 | |||
2053 | @node IP Address of external DNS resolver | ||
2054 | @subsubsection IP Address of external DNS resolver | ||
2055 | |||
2056 | If DNS traffic is to exit your machine, it will be send to this DNS | ||
2057 | resolver. You can specify an IPv4 or IPv6 address. | ||
2058 | |||
2059 | @node IPv4 address for Exit interface | ||
2060 | @subsubsection IPv4 address for Exit interface | ||
2061 | |||
2062 | This is the IPv4 address the Interface will get. Make the mask of the | ||
2063 | address big enough (255.255.0.0 or, even better, 255.0.0.0) to allow more | ||
2064 | mappings of IP addresses into this range. As for the VPN interface, any | ||
2065 | unused, private IPv4 address range will do. | ||
2066 | |||
2067 | @node IPv6 address for Exit interface | ||
2068 | @subsubsection IPv6 address for Exit interface | ||
2069 | |||
2070 | The public IPv6 address the interface will get. If your kernel is not a | ||
2071 | very recent kernel and you are willing to manually enable IPv6-NAT, the | ||
2072 | IPv6 address you specify here must be a globally routed IPv6 address of | ||
2073 | your host. | ||
2074 | |||
2075 | Suppose your host has the address @code{2001:4ca0::1234/64}, then | ||
2076 | using @code{2001:4ca0::1:0/112} would be fine (keep the first 64 bits, | ||
2077 | then change at least one bit in the range before the bitmask, in the | ||
2078 | example above we changed bit 111 from 0 to 1). | ||
2079 | |||
2080 | You may also have to configure your router to route traffic for the entire | ||
2081 | subnet (@code{2001:4ca0::1:0/112} for example) through your computer (this | ||
2082 | should be automatic with IPv6, but obviously anything can be | ||
2083 | disabled). | ||
2084 | |||
2085 | @node Bandwidth Configuration | ||
2086 | @subsection Bandwidth Configuration | ||
2087 | |||
2088 | You can specify how many bandwidth GNUnet is allowed to use to receive | ||
2089 | and send data. This is important for users with limited bandwidth or | ||
2090 | traffic volume. | ||
2091 | |||
2092 | @node Configuring NAT | ||
2093 | @subsection Configuring NAT | ||
2094 | |||
2095 | Most hosts today do not have a normal global IP address but instead are | ||
2096 | behind a router performing Network Address Translation (NAT) which assigns | ||
2097 | each host in the local network a private IP address. | ||
2098 | As a result, these machines cannot trivially receive inbound connections | ||
2099 | from the Internet. GNUnet supports NAT traversal to enable these machines | ||
2100 | to receive incoming connections from other peers despite their | ||
2101 | limitations. | ||
2102 | |||
2103 | In an ideal world, you can press the "Attempt automatic configuration" | ||
2104 | button in gnunet-setup to automatically configure your peer correctly. | ||
2105 | Alternatively, your distribution might have already triggered this | ||
2106 | automatic configuration during the installation process. | ||
2107 | However, automatic configuration can fail to determine the optimal | ||
2108 | settings, resulting in your peer either not receiving as many connections | ||
2109 | as possible, or in the worst case it not connecting to the network at all. | ||
2110 | |||
2111 | To manually configure the peer, you need to know a few things about your | ||
2112 | network setup. First, determine if you are behind a NAT in the first | ||
2113 | place. | ||
2114 | This is always the case if your IP address starts with "10.*" or | ||
2115 | "192.168.*". Next, if you have control over your NAT router, you may | ||
2116 | choose to manually configure it to allow GNUnet traffic to your host. | ||
2117 | If you have configured your NAT to forward traffic on ports 2086 (and | ||
2118 | possibly 1080) to your host, you can check the "NAT ports have been opened | ||
2119 | manually" option, which corresponds to the "PUNCHED_NAT" option in the | ||
2120 | configuration file. If you did not punch your NAT box, it may still be | ||
2121 | configured to support UPnP, which allows GNUnet to automatically | ||
2122 | configure it. In that case, you need to install the "upnpc" command, | ||
2123 | enable UPnP (or PMP) on your NAT box and set the "Enable NAT traversal | ||
2124 | via UPnP or PMP" option (corresponding to "ENABLE_UPNP" in the | ||
2125 | configuration file). | ||
2126 | |||
2127 | Some NAT boxes can be traversed using the autonomous NAT traversal method. | ||
2128 | This requires certain GNUnet components to be installed with "SUID" | ||
2129 | privileges on your system (so if you're installing on a system you do | ||
2130 | not have administrative rights to, this will not work). | ||
2131 | If you installed as 'root', you can enable autonomous NAT traversal by | ||
2132 | checking the "Enable NAT traversal using ICMP method". | ||
2133 | The ICMP method requires a way to determine your NAT's external (global) | ||
2134 | IP address. This can be done using either UPnP, DynDNS, or by manual | ||
2135 | configuration. If you have a DynDNS name or know your external IP address, | ||
2136 | you should enter that name under "External (public) IPv4 address" (which | ||
2137 | corresponds to the "EXTERNAL_ADDRESS" option in the configuration file). | ||
2138 | If you leave the option empty, GNUnet will try to determine your external | ||
2139 | IP address automatically (which may fail, in which case autonomous | ||
2140 | NAT traversal will then not work). | ||
2141 | |||
2142 | Finally, if you yourself are not behind NAT but want to be able to | ||
2143 | connect to NATed peers using autonomous NAT traversal, you need to check | ||
2144 | the "Enable connecting to NATed peers using ICMP method" box. | ||
2145 | |||
2146 | |||
2147 | @node Peer configuration for distributors (e.g. Operating Systems) | ||
2148 | @subsection Peer configuration for distributors (e.g. Operating Systems) | ||
2149 | |||
2150 | The "GNUNET_DATA_HOME" in "[PATHS]" in @file{/etc/gnunet.conf} should be | ||
2151 | manually set to "/var/lib/gnunet/data/" as the default | ||
2152 | "~/.local/share/gnunet/" is probably not that appropriate in this case. | ||
2153 | Similarly, distributors may consider pointing "GNUNET_RUNTIME_DIR" to | ||
2154 | "/var/run/gnunet/" and "GNUNET_HOME" to "/var/lib/gnunet/". Also, should a | ||
2155 | distributor decide to override system defaults, all of these changes | ||
2156 | should be done in a custom @file{/etc/gnunet.conf} and not in the files | ||
2157 | in the @file{config.d/} directory. | ||
2158 | |||
2159 | Given the proposed access permissions, the "gnunet-setup" tool must be | ||
2160 | run as use "gnunet" (and with option "-c /etc/gnunet.conf" so that it | ||
2161 | modifies the system configuration). As always, gnunet-setup should be run | ||
2162 | after the GNUnet peer was stopped using "gnunet-arm -e". Distributors | ||
2163 | might want to include a wrapper for gnunet-setup that allows the | ||
2164 | desktop-user to "sudo" (e.g. using gtksudo) to the "gnunet" user account | ||
2165 | and then runs "gnunet-arm -e", "gnunet-setup" and "gnunet-arm -s" in | ||
2166 | sequence. | ||
2167 | |||
2168 | @node Config Leftovers | ||
2169 | @section Config Leftovers | ||
2170 | |||
2171 | This section describes how to start a GNUnet peer. It assumes that you | ||
2172 | have already compiled and installed GNUnet and its' dependencies. | ||
2173 | Before you start a GNUnet peer, you may want to create a configuration | ||
2174 | file using gnunet-setup (but you do not have to). | ||
2175 | Sane defaults should exist in your | ||
2176 | @file{$GNUNET_PREFIX/share/gnunet/config.d/} directory, so in practice | ||
2177 | you could simply start without any configuration. If you want to | ||
2178 | configure your peer later, you need to stop it before invoking the | ||
2179 | @code{gnunet-setup} tool to customize further and to test your | ||
2180 | configuration (@code{gnunet-setup} has built-in test functions). | ||
2181 | |||
2182 | The most important option you might have to still set by hand is in | ||
2183 | [PATHS]. Here, you use the option "GNUNET_HOME" to specify the path where | ||
2184 | GNUnet should store its data. | ||
2185 | It defaults to @code{$HOME/}, which again should work for most users. | ||
2186 | Make sure that the directory specified as GNUNET_HOME is writable to | ||
2187 | the user that you will use to run GNUnet (note that you can run frontends | ||
2188 | using other users, GNUNET_HOME must only be accessible to the user used to | ||
2189 | run the background processes). | ||
2190 | |||
2191 | You will also need to make one central decision: should all of GNUnet be | ||
2192 | run under your normal UID, or do you want distinguish between system-wide | ||
2193 | (user-independent) GNUnet services and personal GNUnet services. The | ||
2194 | multi-user setup is slightly more complicated, but also more secure and | ||
2195 | generally recommended. | ||
2196 | |||
2197 | @menu | ||
2198 | * The Single-User Setup:: | ||
2199 | * The Multi-User Setup:: | ||
2200 | * Killing GNUnet services:: | ||
2201 | * Access Control for GNUnet:: | ||
2202 | @end menu | ||
2203 | |||
2204 | @node The Single-User Setup | ||
2205 | @subsection The Single-User Setup | ||
2206 | |||
2207 | For the single-user setup, you do not need to do anything special and can | ||
2208 | just start the GNUnet background processes using @code{gnunet-arm}. | ||
2209 | By default, GNUnet looks in @file{~/.config/gnunet.conf} for a | ||
2210 | configuration (or @code{$XDG_CONFIG_HOME/gnunet.conf} if@ | ||
2211 | @code{$XDG_CONFIG_HOME} is defined). If your configuration lives | ||
2212 | elsewhere, you need to pass the @code{-c FILENAME} option to all GNUnet | ||
2213 | commands. | ||
2214 | |||
2215 | Assuming the configuration file is called @file{~/.config/gnunet.conf}, | ||
2216 | you start your peer using the @code{gnunet-arm} command (say as user | ||
2217 | @code{gnunet}) using: | ||
2218 | |||
2219 | @example | ||
2220 | gnunet-arm -c ~/.config/gnunet.conf -s | ||
2221 | @end example | ||
2222 | |||
2223 | @noindent | ||
2224 | The "-s" option here is for "start". The command should return almost | ||
2225 | instantly. If you want to stop GNUnet, you can use: | ||
2226 | |||
2227 | @example | ||
2228 | gnunet-arm -c ~/.config/gnunet.conf -e | ||
2229 | @end example | ||
2230 | |||
2231 | @noindent | ||
2232 | The "-e" option here is for "end". | ||
2233 | |||
2234 | Note that this will only start the basic peer, no actual applications | ||
2235 | will be available. | ||
2236 | If you want to start the file-sharing service, use (after starting | ||
2237 | GNUnet): | ||
2238 | |||
2239 | @example | ||
2240 | gnunet-arm -c ~/.config/gnunet.conf -i fs | ||
2241 | @end example | ||
2242 | |||
2243 | @noindent | ||
2244 | The "-i fs" option here is for "initialize" the "fs" (file-sharing) | ||
2245 | application. You can also selectively kill only file-sharing support using | ||
2246 | |||
2247 | @example | ||
2248 | gnunet-arm -c ~/.config/gnunet.conf -k fs | ||
2249 | @end example | ||
2250 | |||
2251 | @noindent | ||
2252 | Assuming that you want certain services (like file-sharing) to be always | ||
2253 | automatically started whenever you start GNUnet, you can activate them by | ||
2254 | setting "IMMEDIATE_START=YES" in the respective section of the configuration | ||
2255 | file (for example, "[fs]"). Then GNUnet with file-sharing support would | ||
2256 | be started whenever you@ enter: | ||
2257 | |||
2258 | @example | ||
2259 | gnunet-arm -c ~/.config/gnunet.conf -s | ||
2260 | @end example | ||
2261 | |||
2262 | @noindent | ||
2263 | Alternatively, you can combine the two options: | ||
2264 | |||
2265 | @example | ||
2266 | gnunet-arm -c ~/.config/gnunet.conf -s -i fs | ||
2267 | @end example | ||
2268 | |||
2269 | @noindent | ||
2270 | Using @code{gnunet-arm} is also the preferred method for initializing | ||
2271 | GNUnet from @code{init}. | ||
2272 | |||
2273 | Finally, you should edit your @code{crontab} (using the @code{crontab} | ||
2274 | command) and insert a line@ | ||
2275 | |||
2276 | @example | ||
2277 | @@reboot gnunet-arm -c ~/.config/gnunet.conf -s | ||
2278 | @end example | ||
2279 | |||
2280 | to automatically start your peer whenever your system boots. | ||
2281 | |||
2282 | @node The Multi-User Setup | ||
2283 | @subsection The Multi-User Setup | ||
2284 | |||
2285 | This requires you to create a user @code{gnunet} and an additional group | ||
2286 | @code{gnunetdns}, prior to running @code{make install} during | ||
2287 | installation. | ||
2288 | Then, you create a configuration file @file{/etc/gnunet.conf} which should | ||
2289 | contain the lines:@ | ||
2290 | |||
2291 | @example | ||
2292 | [arm] | ||
2293 | START_SYSTEM_SERVICES = YES | ||
2294 | START_USER_SERVICES = NO | ||
2295 | @end example | ||
2296 | |||
2297 | @noindent | ||
2298 | Then, perform the same steps to run GNUnet as in the per-user | ||
2299 | configuration, except as user @code{gnunet} (including the | ||
2300 | @code{crontab} installation). | ||
2301 | You may also want to run @code{gnunet-setup} to configure your peer | ||
2302 | (databases, etc.). | ||
2303 | Make sure to pass @code{-c /etc/gnunet.conf} to all commands. If you | ||
2304 | run @code{gnunet-setup} as user @code{gnunet}, you might need to change | ||
2305 | permissions on @file{/etc/gnunet.conf} so that the @code{gnunet} user can | ||
2306 | write to the file (during setup). | ||
2307 | |||
2308 | Afterwards, you need to perform another setup step for each normal user | ||
2309 | account from which you want to access GNUnet. First, grant the normal user | ||
2310 | (@code{$USER}) permission to the group gnunet: | ||
2311 | |||
2312 | @example | ||
2313 | # adduser $USER gnunet | ||
2314 | @end example | ||
2315 | |||
2316 | @noindent | ||
2317 | Then, create a configuration file in @file{~/.config/gnunet.conf} for the | ||
2318 | $USER with the lines: | ||
2319 | |||
2320 | @example | ||
2321 | [arm] | ||
2322 | START_SYSTEM_SERVICES = NO | ||
2323 | START_USER_SERVICES = YES | ||
2324 | @end example | ||
2325 | |||
2326 | @noindent | ||
2327 | This will ensure that @code{gnunet-arm} when started by the normal user | ||
2328 | will only run services that are per-user, and otherwise rely on the | ||
2329 | system-wide services. | ||
2330 | Note that the normal user may run gnunet-setup, but the | ||
2331 | configuration would be ineffective as the system-wide services will use | ||
2332 | @file{/etc/gnunet.conf} and ignore options set by individual users. | ||
2333 | |||
2334 | Again, each user should then start the peer using | ||
2335 | @file{gnunet-arm -s} --- and strongly consider adding logic to start | ||
2336 | the peer automatically to their crontab. | ||
2337 | |||
2338 | Afterwards, you should see two (or more, if you have more than one USER) | ||
2339 | @code{gnunet-service-arm} processes running in your system. | ||
2340 | |||
2341 | @node Killing GNUnet services | ||
2342 | @subsection Killing GNUnet services | ||
2343 | |||
2344 | It is not necessary to stop GNUnet services explicitly when shutting | ||
2345 | down your computer. | ||
2346 | |||
2347 | It should be noted that manually killing "most" of the | ||
2348 | @code{gnunet-service} processes is generally not a successful method for | ||
2349 | stopping a peer (since @code{gnunet-service-arm} will instantly restart | ||
2350 | them). The best way to explicitly stop a peer is using | ||
2351 | @code{gnunet-arm -e}; note that the per-user services may need to be | ||
2352 | terminated before the system-wide services will terminate normally. | ||
2353 | |||
2354 | @node Access Control for GNUnet | ||
2355 | @subsection Access Control for GNUnet | ||
2356 | |||
2357 | This chapter documents how we plan to make access control work within the | ||
2358 | GNUnet system for a typical peer. It should be read as a best-practice | ||
2359 | installation guide for advanced users and builders of binary | ||
2360 | distributions. The recommendations in this guide apply to POSIX-systems | ||
2361 | with full support for UNIX domain sockets only. | ||
2362 | |||
2363 | Note that this is an advanced topic. The discussion presumes a very good | ||
2364 | understanding of users, groups and file permissions. Normal users on | ||
2365 | hosts with just a single user can just install GNUnet under their own | ||
2366 | account (and possibly allow the installer to use SUDO to grant additional | ||
2367 | permissions for special GNUnet tools that need additional rights). | ||
2368 | The discussion below largely applies to installations where multiple users | ||
2369 | share a system and to installations where the best possible security is | ||
2370 | paramount. | ||
2371 | |||
2372 | A typical GNUnet system consists of components that fall into four | ||
2373 | categories: | ||
2374 | |||
2375 | @table @asis | ||
2376 | |||
2377 | @item User interfaces | ||
2378 | User interfaces are not security sensitive and are supposed to be run and | ||
2379 | used by normal system users. | ||
2380 | The GTK GUIs and most command-line programs fall into this category. | ||
2381 | Some command-line tools (like gnunet-transport) should be excluded as they | ||
2382 | offer low-level access that normal users should not need. | ||
2383 | @item System services and support tools | ||
2384 | System services should always run and offer services that can then be | ||
2385 | accessed by the normal users. | ||
2386 | System services do not require special permissions, but as they are not | ||
2387 | specific to a particular user, they probably should not run as a | ||
2388 | particular user. Also, there should typically only be one GNUnet peer per | ||
2389 | host. System services include the gnunet-service and gnunet-daemon | ||
2390 | programs; support tools include command-line programs such as gnunet-arm. | ||
2391 | @item Privileged helpers | ||
2392 | Some GNUnet components require root rights to open raw sockets or perform | ||
2393 | other special operations. These gnunet-helper binaries are typically | ||
2394 | installed SUID and run from services or daemons. | ||
2395 | @item Critical services | ||
2396 | Some GNUnet services (such as the DNS service) can manipulate the service | ||
2397 | in deep and possibly highly security sensitive ways. For example, the DNS | ||
2398 | service can be used to intercept and alter any DNS query originating from | ||
2399 | the local machine. Access to the APIs of these critical services and their | ||
2400 | privileged helpers must be tightly controlled. | ||
2401 | @end table | ||
2402 | |||
2403 | @c FIXME: The titles of these chapters are too long in the index. | ||
2404 | |||
2405 | @menu | ||
2406 | * Recommendation - Disable access to services via TCP:: | ||
2407 | * Recommendation - Run most services as system user "gnunet":: | ||
2408 | * Recommendation - Control access to services using group "gnunet":: | ||
2409 | * Recommendation - Limit access to certain SUID binaries by group "gnunet":: | ||
2410 | * Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns":: | ||
2411 | * Differences between "make install" and these recommendations:: | ||
2412 | @end menu | ||
2413 | |||
2414 | @node Recommendation - Disable access to services via TCP | ||
2415 | @subsubsection Recommendation - Disable access to services via TCP | ||
2416 | |||
2417 | GNUnet services allow two types of access: via TCP socket or via UNIX | ||
2418 | domain socket. | ||
2419 | If the service is available via TCP, access control can only be | ||
2420 | implemented by restricting connections to a particular range of IP | ||
2421 | addresses. | ||
2422 | This is acceptable for non-critical services that are supposed to be | ||
2423 | available to all users on the local system or local network. | ||
2424 | However, as TCP is generally less efficient and it is rarely the case | ||
2425 | that a single GNUnet peer is supposed to serve an entire local network, | ||
2426 | the default configuration should disable TCP access to all GNUnet | ||
2427 | services on systems with support for UNIX domain sockets. | ||
2428 | Since GNUnet 0.9.2, configuration files with TCP access disabled should be | ||
2429 | generated by default. Users can re-enable TCP access to particular | ||
2430 | services simply by specifying a non-zero port number in the section of | ||
2431 | the respective service. | ||
2432 | |||
2433 | |||
2434 | @node Recommendation - Run most services as system user "gnunet" | ||
2435 | @subsubsection Recommendation - Run most services as system user "gnunet" | ||
2436 | |||
2437 | GNUnet's main services should be run as a separate user "gnunet" in a | ||
2438 | special group "gnunet". | ||
2439 | The user "gnunet" should start the peer using "gnunet-arm -s" during | ||
2440 | system startup. The home directory for this user should be | ||
2441 | @file{/var/lib/gnunet} and the configuration file should be | ||
2442 | @file{/etc/gnunet.conf}. | ||
2443 | Only the @code{gnunet} user should have the right to access | ||
2444 | @file{/var/lib/gnunet} (@emph{mode: 700}). | ||
2445 | |||
2446 | @node Recommendation - Control access to services using group "gnunet" | ||
2447 | @subsubsection Recommendation - Control access to services using group "gnunet" | ||
2448 | |||
2449 | Users that should be allowed to use the GNUnet peer should be added to the | ||
2450 | group "gnunet". Using GNUnet's access control mechanism for UNIX domain | ||
2451 | sockets, those services that are considered useful to ordinary users | ||
2452 | should be made available by setting "UNIX_MATCH_GID=YES" for those | ||
2453 | services. | ||
2454 | Again, as shipped, GNUnet provides reasonable defaults. | ||
2455 | Permissions to access the transport and core subsystems might additionally | ||
2456 | be granted without necessarily causing security concerns. | ||
2457 | Some services, such as DNS, must NOT be made accessible to the "gnunet" | ||
2458 | group (and should thus only be accessible to the "gnunet" user and | ||
2459 | services running with this UID). | ||
2460 | |||
2461 | @node Recommendation - Limit access to certain SUID binaries by group "gnunet" | ||
2462 | @subsubsection Recommendation - Limit access to certain SUID binaries by group "gnunet" | ||
2463 | |||
2464 | Most of GNUnet's SUID binaries should be safe even if executed by normal | ||
2465 | users. However, it is possible to reduce the risk a little bit more by | ||
2466 | making these binaries owned by the group "gnunet" and restricting their | ||
2467 | execution to user of the group "gnunet" as well (4750). | ||
2468 | |||
2469 | @node Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns" | ||
2470 | @subsubsection Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns" | ||
2471 | |||
2472 | A special group "gnunetdns" should be created for controlling access to | ||
2473 | the "gnunet-helper-dns". | ||
2474 | The binary should then be owned by root and be in group "gnunetdns" and | ||
2475 | be installed SUID and only be group-executable (2750). | ||
2476 | @b{Note that the group "gnunetdns" should have no users in it at all, | ||
2477 | ever.} | ||
2478 | The "gnunet-service-dns" program should be executed by user "gnunet" (via | ||
2479 | gnunet-service-arm) with the binary owned by the user "root" and the group | ||
2480 | "gnunetdns" and be SGID (2700). This way, @strong{only} | ||
2481 | "gnunet-service-dns" can change its group to "gnunetdns" and execute the | ||
2482 | helper, and the helper can then run as root (as per SUID). | ||
2483 | Access to the API offered by "gnunet-service-dns" is in turn restricted | ||
2484 | to the user "gnunet" (not the group!), which means that only | ||
2485 | "benign" services can manipulate DNS queries using "gnunet-service-dns". | ||
2486 | |||
2487 | @node Differences between "make install" and these recommendations | ||
2488 | @subsubsection Differences between "make install" and these recommendations | ||
2489 | |||
2490 | The current build system does not set all permissions automatically based | ||
2491 | on the recommendations above. In particular, it does not use the group | ||
2492 | "gnunet" at all (so setting gnunet-helpers other than the | ||
2493 | gnunet-helper-dns to be owned by group "gnunet" must be done manually). | ||
2494 | Furthermore, 'make install' will silently fail to set the DNS binaries to | ||
2495 | be owned by group "gnunetdns" unless that group already exists (!). | ||
2496 | An alternative name for the "gnunetdns" group can be specified using the | ||
2497 | @code{--with-gnunetdns=GRPNAME} configure option. | ||
diff --git a/doc/handbook/chapters/keyconcepts.texi b/doc/handbook/chapters/keyconcepts.texi deleted file mode 100644 index c8dd1599b..000000000 --- a/doc/handbook/chapters/keyconcepts.texi +++ /dev/null | |||
@@ -1,350 +0,0 @@ | |||
1 | |||
2 | @cindex Key Concepts | ||
3 | @node Key Concepts | ||
4 | @chapter Key Concepts | ||
5 | |||
6 | In this section, the fundamental concepts of GNUnet are explained. | ||
7 | @c FIXME: Use @uref{https://docs.gnunet.org/bib/, research papers} | ||
8 | @c once we have the new bibliography + subdomain setup. | ||
9 | Most of them are also described in our research papers. | ||
10 | First, some of the concepts used in the GNUnet framework are detailed. | ||
11 | The second part describes concepts specific to anonymous file-sharing. | ||
12 | |||
13 | @menu | ||
14 | * Authentication:: | ||
15 | * Accounting to Encourage Resource Sharing:: | ||
16 | * Confidentiality:: | ||
17 | * Anonymity:: | ||
18 | * Deniability:: | ||
19 | * Peer Identities:: | ||
20 | * Zones in the GNU Name System (GNS Zones):: | ||
21 | * Egos:: | ||
22 | @end menu | ||
23 | |||
24 | @cindex Authentication | ||
25 | @node Authentication | ||
26 | @section Authentication | ||
27 | |||
28 | Almost all peer-to-peer communications in GNUnet are between mutually | ||
29 | authenticated peers. The authentication works by using ECDHE, that is a | ||
30 | DH (Diffie---Hellman) key exchange using ephemeral elliptic curve | ||
31 | cryptography. The ephemeral ECC (Elliptic Curve Cryptography) keys are | ||
32 | signed using ECDSA (@uref{http://en.wikipedia.org/wiki/ECDSA, ECDSA}). | ||
33 | The shared secret from ECDHE is used to create a pair of session keys | ||
34 | @c FIXME: Long word for HKDF. More FIXMEs: Explain MITM etc. | ||
35 | (using HKDF) which are then used to encrypt the communication between the | ||
36 | two peers using both 256-bit AES (Advanced Encryption Standard) | ||
37 | and 256-bit Twofish (with independently derived secret keys). | ||
38 | As only the two participating hosts know the shared secret, this | ||
39 | authenticates each packet | ||
40 | without requiring signatures each time. GNUnet uses SHA-512 | ||
41 | (Secure Hash Algorithm) hash codes to verify the integrity of messages. | ||
42 | |||
43 | @c FIXME: A while back I got the feedback that I should try and integrate | ||
44 | @c explanation boxes in the long-run. So we could explain | ||
45 | @c "man-in-the-middle" and "man-in-the-middle attacks" and other words | ||
46 | @c which are not common knowledge. MITM is not common knowledge. To be | ||
47 | @c selfcontained, we should be able to explain words and concepts used in | ||
48 | @c a chapter or paragraph without hinting at Wikipedia and other online | ||
49 | @c sources which might not be available or accessible to everyone. | ||
50 | @c On the other hand we could write an introductionary chapter or book | ||
51 | @c that we could then reference in each chapter, which sound like it | ||
52 | @c could be more reusable. | ||
53 | In GNUnet, the identity of a host is its public key. For that reason, | ||
54 | man-in-the-middle attacks will not break the authentication or accounting | ||
55 | goals. Essentially, for GNUnet, the IP of the host has nothing to do with | ||
56 | the identity of the host. As the public key is the only thing that truly | ||
57 | matters, faking an IP, a port or any other property of the underlying | ||
58 | transport protocol is irrelevant. In fact, GNUnet peers can use | ||
59 | multiple IPs (IPv4 and IPv6) on multiple ports --- or even not use the | ||
60 | IP protocol at all (by running directly on layer 2). | ||
61 | @c FIXME: "IP protocol" feels wrong, but could be what people expect, as | ||
62 | @c IP is "the number" and "IP protocol" the protocol itself in general | ||
63 | @c knowledge? | ||
64 | |||
65 | @c NOTE: For consistency we will use @code{HELLO}s throughout this Manual. | ||
66 | GNUnet uses a special type of message to communicate a binding between | ||
67 | public (ECC) keys to their current network address. These messages are | ||
68 | commonly called @code{HELLO}s or @code{peer advertisements}. | ||
69 | They contain the public key of the peer and its current network | ||
70 | addresses for various transport services. | ||
71 | A transport service is a special kind of shared library that | ||
72 | provides (possibly unreliable, out-of-order) message delivery between | ||
73 | peers. | ||
74 | For the UDP and TCP transport services, a network address is an IP and a | ||
75 | port. | ||
76 | GNUnet can also use other transports (HTTP, HTTPS, WLAN, etc.) which use | ||
77 | various other forms of addresses. Note that any node can have many | ||
78 | different active transport services at the same time, | ||
79 | and each of these can have a different addresses. | ||
80 | Binding messages expire after at most a week (the timeout can be | ||
81 | shorter if the user configures the node appropriately). | ||
82 | This expiration ensures that the network will eventually get rid of | ||
83 | outdated advertisements. | ||
84 | |||
85 | For more information, refer to the following paper: | ||
86 | |||
87 | Ronaldo A. Ferreira, Christian Grothoff, and Paul Ruth. | ||
88 | A Transport Layer Abstraction for Peer-to-Peer Networks | ||
89 | Proceedings of the 3rd International Symposium on Cluster Computing | ||
90 | and the Grid (GRID 2003), 2003. | ||
91 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/transport.pdf, https://git.gnunet.org/bibliography.git/plain/docs/transport.pdf}) | ||
92 | |||
93 | @cindex Accounting to Encourage Resource Sharing | ||
94 | @node Accounting to Encourage Resource Sharing | ||
95 | @section Accounting to Encourage Resource Sharing | ||
96 | |||
97 | Most distributed P2P networks suffer from a lack of defenses or | ||
98 | precautions against attacks in the form of freeloading. | ||
99 | While the intentions of an attacker and a freeloader are different, their | ||
100 | effect on the network is the same; they both render it useless. | ||
101 | Most simple attacks on networks such as @command{Gnutella} | ||
102 | involve flooding the network with traffic, particularly | ||
103 | with queries that are, in the worst case, multiplied by the network. | ||
104 | |||
105 | In order to ensure that freeloaders or attackers have a minimal impact | ||
106 | on the network, GNUnet's file-sharing implementation (@code{FS}) tries | ||
107 | to distinguish good (contributing) nodes from malicious (freeloading) | ||
108 | nodes. In GNUnet, every file-sharing node keeps track of the behavior | ||
109 | of every other node it has been in contact with. Many requests | ||
110 | (depending on the application) are transmitted with a priority (or | ||
111 | importance) level. That priority is used to establish how important | ||
112 | the sender believes this request is. If a peer responds to an | ||
113 | important request, the recipient will increase its trust in the | ||
114 | responder: the responder contributed resources. If a peer is too busy | ||
115 | to answer all requests, it needs to prioritize. For that, peers do | ||
116 | not take the priorities of the requests received at face value. | ||
117 | First, they check how much they trust the sender, and depending on | ||
118 | that amount of trust they assign the request a (possibly lower) | ||
119 | effective priority. Then, they drop the requests with the lowest | ||
120 | effective priority to satisfy their resource constraints. This way, | ||
121 | GNUnet's economic model ensures that nodes that are not currently | ||
122 | considered to have a surplus in contributions will not be served if | ||
123 | the network load is high. | ||
124 | |||
125 | For more information, refer to the following paper: | ||
126 | Christian Grothoff. An Excess-Based Economic Model for Resource | ||
127 | Allocation in Peer-to-Peer Networks. Wirtschaftsinformatik, June 2003. | ||
128 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/ebe.pdf, https://git.gnunet.org/bibliography.git/plain/docs/ebe.pdf}) | ||
129 | |||
130 | @cindex Confidentiality | ||
131 | @node Confidentiality | ||
132 | @section Confidentiality | ||
133 | |||
134 | Adversaries (malicious, bad actors) outside of GNUnet are not supposed | ||
135 | to know what kind of actions a peer is involved in. Only the specific | ||
136 | neighbor of a peer that is the corresponding sender or recipient of a | ||
137 | message may know its contents, and even then application protocols may | ||
138 | place further restrictions on that knowledge. In order to ensure | ||
139 | confidentiality, GNUnet uses link encryption, that is each message | ||
140 | exchanged between two peers is encrypted using a pair of keys only | ||
141 | known to these two peers. Encrypting traffic like this makes any kind | ||
142 | of traffic analysis much harder. Naturally, for some applications, it | ||
143 | may still be desirable if even neighbors cannot determine the concrete | ||
144 | contents of a message. In GNUnet, this problem is addressed by the | ||
145 | specific application-level protocols. See for example the following | ||
146 | sections @pxref{Anonymity}, @pxref{How file-sharing achieves Anonymity}, | ||
147 | and @pxref{Deniability}. | ||
148 | |||
149 | @cindex Anonymity | ||
150 | @node Anonymity | ||
151 | @section Anonymity | ||
152 | |||
153 | @menu | ||
154 | * How file-sharing achieves Anonymity:: | ||
155 | * How messaging provides Anonymity:: | ||
156 | @end menu | ||
157 | |||
158 | Providing anonymity for users is the central goal for the anonymous | ||
159 | file-sharing application. Many other design decisions follow in the | ||
160 | footsteps of this requirement. | ||
161 | Anonymity is never absolute. While there are various | ||
162 | scientific metrics | ||
163 | (Claudia Díaz, Stefaan Seys, Joris Claessens, | ||
164 | and Bart Preneel. Towards measuring anonymity. | ||
165 | 2002. | ||
166 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/article-89.pdf, https://git.gnunet.org/bibliography.git/plain/docs/article-89.pdf})) | ||
167 | that can help quantify the level of anonymity that a given mechanism | ||
168 | provides, there is no such thing as "complete anonymity". | ||
169 | |||
170 | GNUnet's file-sharing implementation allows users to select for each | ||
171 | operation (publish, search, download) the desired level of anonymity. | ||
172 | The metric used is based on the amount of cover traffic needed to hide | ||
173 | the request. | ||
174 | |||
175 | While there is no clear way to relate the amount of available cover | ||
176 | traffic to traditional scientific metrics such as the anonymity set or | ||
177 | information leakage, it is probably the best metric available to a | ||
178 | peer with a purely local view of the world, in that it does not rely | ||
179 | on unreliable external information or a particular adversary model. | ||
180 | |||
181 | The default anonymity level is @code{1}, which uses anonymous routing | ||
182 | but imposes no minimal requirements on cover traffic. It is possible | ||
183 | to forego anonymity when this is not required. The anonymity level of | ||
184 | @code{0} allows GNUnet to use more efficient, non-anonymous routing. | ||
185 | |||
186 | @cindex How file-sharing achieves Anonymity | ||
187 | @node How file-sharing achieves Anonymity | ||
188 | @subsection How file-sharing achieves Anonymity | ||
189 | |||
190 | Contrary to other designs, we do not believe that users achieve strong | ||
191 | anonymity just because their requests are obfuscated by a couple of | ||
192 | indirections. This is not sufficient if the adversary uses traffic | ||
193 | analysis. | ||
194 | The threat model used for anonymous file sharing in GNUnet assumes that | ||
195 | the adversary is quite powerful. | ||
196 | In particular, we assume that the adversary can see all the traffic on | ||
197 | the Internet. And while we assume that the adversary | ||
198 | can not break our encryption, we assume that the adversary has many | ||
199 | participating nodes in the network and that it can thus see many of the | ||
200 | node-to-node interactions since it controls some of the nodes. | ||
201 | |||
202 | The system tries to achieve anonymity based on the idea that users can be | ||
203 | anonymous if they can hide their actions in the traffic created by other | ||
204 | users. | ||
205 | Hiding actions in the traffic of other users requires participating in the | ||
206 | traffic, bringing back the traditional technique of using indirection and | ||
207 | source rewriting. Source rewriting is required to gain anonymity since | ||
208 | otherwise an adversary could tell if a message originated from a host by | ||
209 | looking at the source address. If all packets look like they originate | ||
210 | from one node, the adversary can not tell which ones originate from that | ||
211 | node and which ones were routed. | ||
212 | Note that in this mindset, any node can decide to break the | ||
213 | source-rewriting paradigm without violating the protocol, as this | ||
214 | only reduces the amount of traffic that a node can hide its own traffic | ||
215 | in. | ||
216 | |||
217 | If we want to hide our actions in the traffic of other nodes, we must make | ||
218 | our traffic indistinguishable from the traffic that we route for others. | ||
219 | As our queries must have us as the receiver of the reply | ||
220 | (otherwise they would be useless), we must put ourselves as the receiver | ||
221 | of replies that actually go to other hosts; in other words, we must | ||
222 | indirect replies. | ||
223 | Unlike other systems, in anonymous file-sharing as implemented on top of | ||
224 | GNUnet we do not have to indirect the replies if we don't think we need | ||
225 | more traffic to hide our own actions. | ||
226 | |||
227 | This increases the efficiency of the network as we can indirect less under | ||
228 | higher load. | ||
229 | Refer to the following paper for more: | ||
230 | Krista Bennett and Christian Grothoff. | ||
231 | GAP --- practical anonymous networking. In Proceedings of | ||
232 | Designing Privacy Enhancing Technologies, 2003. | ||
233 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/aff.pdf, https://git.gnunet.org/bibliography.git/plain/docs/aff.pdf}) | ||
234 | |||
235 | @cindex How messaging provides Anonymity | ||
236 | @node How messaging provides Anonymity | ||
237 | @subsection How messaging provides Anonymity | ||
238 | |||
239 | While the file-sharing tries to achieve anonymity through hiding actions in | ||
240 | other traffic, the messaging service provides a weaker form of protection | ||
241 | against identification. | ||
242 | |||
243 | The messaging service allows the use of an anonymous ego for the signing and | ||
244 | verification process of messages instead of a unique ego. This anonymous ego is | ||
245 | a publicly known key pair which is shared between all peers in GNUnet. | ||
246 | |||
247 | Using this ego only ensures that individual messages alone can't identify its | ||
248 | sender inside of a messenger room. It should be clarified that the route of | ||
249 | the traffic for each message can still be tracked to identify the senders peer | ||
250 | inside of a messenger room if the threat agent controls certain peers hosting | ||
251 | the room. | ||
252 | |||
253 | Also opening a room in the messenger service will potentially match your peer | ||
254 | identity with the internal member identity from the messenger service. So | ||
255 | despite using the anonymous ego you can reveal your peer identity. This means | ||
256 | to decrease the chance of being identified, it is recommended to enter rooms but | ||
257 | you should not open them for others. | ||
258 | |||
259 | @cindex Deniability | ||
260 | @node Deniability | ||
261 | @section Deniability | ||
262 | |||
263 | Even if the user that downloads data and the server that provides data are | ||
264 | anonymous, the intermediaries may still be targets. In particular, if the | ||
265 | intermediaries can find out which queries or which content they are | ||
266 | processing, a strong adversary could try to force them to censor | ||
267 | certain materials. | ||
268 | |||
269 | With the file-encoding used by GNUnet's anonymous file-sharing, this | ||
270 | problem does not arise. | ||
271 | The reason is that queries and replies are transmitted in | ||
272 | an encrypted format such that intermediaries cannot tell what the query | ||
273 | is for or what the content is about. Mind that this is not the same | ||
274 | encryption as the link-encryption between the nodes. GNUnet has | ||
275 | encryption on the network layer (link encryption, confidentiality, | ||
276 | authentication) and again on the application layer (provided | ||
277 | by @command{gnunet-publish}, @command{gnunet-download}, | ||
278 | @command{gnunet-search} and @command{gnunet-fs-gtk}). | ||
279 | |||
280 | Refer to the following paper for more: | ||
281 | Christian Grothoff, Krista Grothoff, Tzvetan Horozov, | ||
282 | and Jussi T. Lindgren. | ||
283 | An Encoding for Censorship-Resistant Sharing. | ||
284 | 2009. | ||
285 | (@uref{https://git.gnunet.org/bibliography.git/plain/docs/ecrs.pdf, https://git.gnunet.org/bibliography.git/plain/docs/ecrs.pdf}) | ||
286 | |||
287 | @cindex Peer Identities | ||
288 | @node Peer Identities | ||
289 | @section Peer Identities | ||
290 | |||
291 | Peer identities are used to identify peers in the network and are unique | ||
292 | for each peer. The identity for a peer is simply its public key, which is | ||
293 | generated along with a private key when the peer is started for the first | ||
294 | time. While the identity is binary data, it is often expressed as an ASCII | ||
295 | string. For example, the following is a peer identity as you might see it | ||
296 | in various places: | ||
297 | |||
298 | @example | ||
299 | UAT1S6PMPITLBKSJ2DGV341JI6KF7B66AC4JVCN9811NNEGQLUN0 | ||
300 | @end example | ||
301 | |||
302 | @noindent | ||
303 | You can find your peer identity by running @command{gnunet-peerinfo -s}. | ||
304 | |||
305 | @cindex Zones in the GNU Name System (GNS Zones) | ||
306 | @node Zones in the GNU Name System (GNS Zones) | ||
307 | @section Zones in the GNU Name System (GNS Zones) | ||
308 | |||
309 | @c FIXME: Explain or link to an explanation of the concept of public keys | ||
310 | @c and private keys. | ||
311 | @c FIXME: Rewrite for the latest GNS changes. | ||
312 | GNS zones are similar to those of DNS zones, but instead of a hierarchy of | ||
313 | authorities to governing their use, GNS zones are controlled by a private | ||
314 | key. | ||
315 | When you create a record in a DNS zone, that information is stored in your | ||
316 | nameserver. Anyone trying to resolve your domain then gets pointed | ||
317 | (hopefully) by the centralised authority to your nameserver. | ||
318 | Whereas GNS, being fully decentralized by design, stores that information | ||
319 | in DHT. The validity of the records is assured cryptographically, by | ||
320 | signing them with the private key of the respective zone. | ||
321 | |||
322 | Anyone trying to resolve records in a zone of your domain can then verify | ||
323 | the signature of the records they get from the DHT and be assured that | ||
324 | they are indeed from the respective zone. | ||
325 | To make this work, there is a 1:1 correspondence between zones and | ||
326 | their public-private key pairs. | ||
327 | So when we talk about the owner of a GNS zone, that's really the owner of | ||
328 | the private key. | ||
329 | And a user accessing a zone needs to somehow specify the corresponding | ||
330 | public key first. | ||
331 | |||
332 | For more information, refer to the following paper: | ||
333 | |||
334 | Matthias Wachs, Martin Schanzenbach, and Christian Grothoff. | ||
335 | A Censorship-Resistant, Privacy-Enhancing and Fully Decentralized Name | ||
336 | System. In proceedings of 13th International Conference on Cryptology and | ||
337 | Network Security (CANS 2014). 2014. | ||
338 | @uref{https://git.gnunet.org/bibliography.git/plain/docs/gns2014wachs.pdf, https://git.gnunet.org/bibliography.git/plain/docs/gns2014wachs.pdf} | ||
339 | |||
340 | @cindex Egos | ||
341 | @node Egos | ||
342 | @section Egos | ||
343 | |||
344 | @c what is the difference between peer identity and egos? It seems | ||
345 | @c like both are linked to public-private key pair. | ||
346 | Egos are your "identities" in GNUnet. Any user can assume multiple | ||
347 | identities, for example to separate their activities online. Egos can | ||
348 | correspond to "pseudonyms" or "real-world identities". Technically an | ||
349 | ego is first of all a key pair of a public- and private-key. | ||
350 | |||
diff --git a/doc/handbook/chapters/philosophy.texi b/doc/handbook/chapters/philosophy.texi deleted file mode 100644 index 180589829..000000000 --- a/doc/handbook/chapters/philosophy.texi +++ /dev/null | |||
@@ -1,82 +0,0 @@ | |||
1 | @cindex Philosophy | ||
2 | @node Philosophy | ||
3 | @chapter Philosophy | ||
4 | |||
5 | @c NOTE: We should probably re-use some of the images lynX created | ||
6 | @c for secushare, showing some of the relations and functionalities | ||
7 | @c of GNUnet. | ||
8 | The primary goal of the GNUnet project is to provide a reliable, open, | ||
9 | non-discriminating and censorship-resistant system for information | ||
10 | exchange. | ||
11 | We value free speech above state interests and intellectual | ||
12 | monopoly. GNUnet's long-term goal is to serve as a development | ||
13 | platform for the next generation of Internet protocols. | ||
14 | |||
15 | Participants are encouraged to | ||
16 | contribute at least as much resources (storage, bandwidth) to the network | ||
17 | as they consume, so that their participation does not have a negative | ||
18 | impact on other users. | ||
19 | |||
20 | @menu | ||
21 | * Design Principles:: | ||
22 | * Privacy and Anonymity:: | ||
23 | * Practicality:: | ||
24 | @end menu | ||
25 | |||
26 | @cindex Design Principles | ||
27 | @node Design Principles | ||
28 | @section Design Principles | ||
29 | |||
30 | These are the GNUnet design principles, in order of importance: | ||
31 | |||
32 | @itemize | ||
33 | @item GNUnet must be implemented as | ||
34 | @uref{https://www.gnu.org/philosophy/free-sw.html, Free Software} --- | ||
35 | This means that you have the four essential freedoms: to run | ||
36 | the program, to study and change the program in source code form, | ||
37 | to redistribute exact copies, and to distribute modified versions. | ||
38 | (@uref{https://www.gnu.org/philosophy/free-sw.html}). | ||
39 | @item GNUnet must minimize the amount of personally identifiable information exposed. | ||
40 | @item GNUnet must be fully distributed and resilient to external attacks and rogue participants. | ||
41 | @item GNUnet must be self-organizing and not depend on administrators or centralized infrastructure. | ||
42 | @item GNUnet must inform the user which other participants have to be trusted when establishing private communications. | ||
43 | @item GNUnet must be open and permit new peers to join. | ||
44 | @item GNUnet must support a diverse range of applications and devices. | ||
45 | @item GNUnet must use compartmentalization to protect sensitive information. | ||
46 | @item The GNUnet architecture must be resource efficient. | ||
47 | @item GNUnet must provide incentives for peers to contribute more resources than they consume. | ||
48 | @end itemize | ||
49 | |||
50 | |||
51 | @cindex Privacy and Anonymity | ||
52 | @node Privacy and Anonymity | ||
53 | @section Privacy and Anonymity | ||
54 | |||
55 | The GNUnet protocols minimize the leakage of personally identifiable | ||
56 | information of participants and do not allow adversaries to control, | ||
57 | track, monitor or censor users activities. The GNUnet protocols also | ||
58 | make it as hard as possible to disrupt operations by participating in | ||
59 | the network with malicious intent. | ||
60 | |||
61 | Analyzing participant's activities becomes more difficult as the | ||
62 | number of peers and applications that generate traffic on the network | ||
63 | grows, even if the additional traffic generated is not related to | ||
64 | anonymous communication. This is one of the reasons why GNUnet is | ||
65 | developed as a peer-to-peer framework where many applications share | ||
66 | the lower layers of an increasingly complex protocol stack. The GNUnet | ||
67 | architecture encourages many different forms of peer-to-peer | ||
68 | applications. | ||
69 | |||
70 | @cindex Practicality | ||
71 | @node Practicality | ||
72 | @section Practicality | ||
73 | |||
74 | Wherever possible GNUnet allows the peer to adjust its operations and | ||
75 | functionalities to specific use cases. A GNUnet peer running on a | ||
76 | mobile device with limited battery for example might choose not to | ||
77 | relay traffic for other participants. | ||
78 | |||
79 | For certain applications like file-sharing GNUnet allows participants | ||
80 | to trade degrees of anonymity in exchange for increased | ||
81 | efficiency. However, it is not possible for any user's efficiency | ||
82 | requirements to compromise the anonymity of any other user. | ||
diff --git a/doc/handbook/chapters/preface.texi b/doc/handbook/chapters/preface.texi deleted file mode 100644 index 961ec0737..000000000 --- a/doc/handbook/chapters/preface.texi +++ /dev/null | |||
@@ -1,212 +0,0 @@ | |||
1 | @node Preface | ||
2 | @chapter Preface | ||
3 | |||
4 | @c FIXME: Do we have to mention that this is Free Software? | ||
5 | @c FIXME: where did 'Free Software' in this sentence fit before | ||
6 | @c FIXME: we changed it? | ||
7 | This collection of manuals describes GNUnet, a framework | ||
8 | for secure peer-to-peer networking with the high-level goal to provide | ||
9 | a strong foundation for a global, distributed network | ||
10 | that provides security and privacy. | ||
11 | GNUnet in that sense aims to replace the current Internet protocol stack. | ||
12 | Along with an application for secure publication of files, it has grown to | ||
13 | include all kinds of basic applications for the foundation of a new | ||
14 | Internet. | ||
15 | |||
16 | @menu | ||
17 | * About this book:: | ||
18 | * Contributing to this book:: | ||
19 | * Introduction:: | ||
20 | * Project governance:: | ||
21 | * Typography:: | ||
22 | @end menu | ||
23 | |||
24 | @node About this book | ||
25 | @section About this book | ||
26 | |||
27 | The books (described as ``book'' or ``books'' in the following) | ||
28 | bundled as the ``GNUnet Reference Manual'' are based on the historic | ||
29 | work of all contributors to previous documentation of GNUnet. | ||
30 | It is our hope | ||
31 | that the content is described in a way that does not require any | ||
32 | academic background, although some concepts will require further | ||
33 | reading. | ||
34 | |||
35 | Our (long-term) goal with these books is to keep them | ||
36 | self-contained. If you see references to Wikipedia and other external | ||
37 | sources (except for our academic papers) it means that we are working | ||
38 | on a solution to describe the explanations found there which fits our | ||
39 | use-case and licensing. | ||
40 | |||
41 | Previously the documentation was contained in Drupal books, on the | ||
42 | old website. This format was considered unmaintainable for the future, so | ||
43 | Texinfo was chosen. You might find old and very old sections in | ||
44 | here in addition to more recent content. It took a long time to | ||
45 | finish the move to Texinfo (from Drupal to LaTeX to wrong Texinfo | ||
46 | output dump to good Texinfo) and only recently (late 2018, early | ||
47 | 2019) content checking started. We apologize to the reader for | ||
48 | any inconvenience and hope you apply logic where bad advice from | ||
49 | 10 years ago can be found (pipe to sudo to install software is | ||
50 | one example). Patches (contributions) to this documentation are more | ||
51 | than welcome! | ||
52 | |||
53 | The first chapter (``Preface'') as well as the second chapter | ||
54 | (``Philosophy'') give an introduction to GNUnet as a project, what | ||
55 | GNUnet tries to achieve. ``Key Concepts'' explains the key concepts | ||
56 | in GNUnet. | ||
57 | These three chapters are the most complete in the documentation. | ||
58 | They are followed by chapters which explain specific parts of | ||
59 | GNUnet (and need more work): | ||
60 | ``Installing GNUnet'', ``GNUnet Contributors Handbook'' and | ||
61 | ``GNUnet Developer Handbook''. | ||
62 | |||
63 | @node Contributing to this book | ||
64 | @section Contributing to this book | ||
65 | |||
66 | @c FIXME: There's a good amount of repetition here, we should | ||
67 | @c FIXME: fix this. | ||
68 | The GNUnet Reference Manual is a collective work produced by various | ||
69 | people throughout the years. | ||
70 | |||
71 | The version you are reading is derived | ||
72 | from many individual efforts hosted on one of our old websites. | ||
73 | In the end it was considered to be impractical to read by | ||
74 | those who required the information. | ||
75 | With the conversion to Texinfo --- the version you are reading | ||
76 | right now --- we hope to address this in the longterm. | ||
77 | Texinfo is the documentation language of the GNU project. | ||
78 | |||
79 | While it can be intimidating at first and look scary or complicated, | ||
80 | it is just another way to express text format instructions. | ||
81 | |||
82 | We encourage you to take this opportunity and learn about Texinfo, | ||
83 | learn about GNUnet, and one word at a time we will arrive at a | ||
84 | book which explains GNUnet in the least complicated way to you. | ||
85 | |||
86 | Even when you don't want to or can't learn Texinfo, you can contribute. | ||
87 | Send us an Email or join our IRC chat room on freenode and talk with | ||
88 | us about the documentation (the preferred way to reach out is the | ||
89 | mailinglist, since you can communicate with us without waiting on | ||
90 | someone in the chatroom). | ||
91 | One way or another you can help shape the understanding of GNUnet | ||
92 | without the ability to read and understand its sourcecode. | ||
93 | |||
94 | @node Introduction | ||
95 | @section Introduction | ||
96 | |||
97 | GNUnet in its current version is the result of almost 20 years of work | ||
98 | from many contributors. So far, most contributions were made by | ||
99 | volunteers or people paid to do fundamental research. At this stage, | ||
100 | GNUnet remains an experimental system where | ||
101 | significant parts of the software lack a reasonable degree of | ||
102 | professionalism in its implementation. Furthermore, we are aware of a | ||
103 | significant number of existing bugs and critical design flaws, as some | ||
104 | unfortunate early design decisions remain to be rectified. There are | ||
105 | still known open problems; GNUnet remains an active research project. | ||
106 | |||
107 | The project was started in 2001 when some initial ideas for improving | ||
108 | Freenet's file-sharing turned out to be too radical to be easily | ||
109 | realized within the scope of the existing Freenet project. We lost | ||
110 | our first contributor on 11.9.2001 as the contributor realized that | ||
111 | privacy may help terrorists. The rest of the team concluded that it | ||
112 | was now even more important to fight for civil liberties. The first | ||
113 | release was called ``GNet'' -- already with the name GNUnet in mind, | ||
114 | but without the blessing of GNU we did not dare to call it GNUnet | ||
115 | immediately. A few months after the first release we contacted the | ||
116 | GNU project, happily agreed to their governance model and became an | ||
117 | official GNU package. | ||
118 | |||
119 | Within the first year, we created | ||
120 | @uref{https://gnu.org/s/libextractor, GNU libextractor}, a helper library | ||
121 | for meta data extraction which has been used by a few other projects | ||
122 | as well. 2003 saw the emergence of pluggable transports, the ability | ||
123 | for GNUnet to use different mechanisms for communication, starting | ||
124 | with TCP, UDP and SMTP (support for the latter was later dropped due | ||
125 | to a lack of maintenance). In 2005, the project first started to | ||
126 | evolve beyond the original file-sharing application with a first | ||
127 | simple P2P chat. In 2007, we created | ||
128 | @uref{https://gnu.org/s/libmicrohttpd, GNU libmicrohttpd} | ||
129 | to support a pluggable transport based on HTTP. In 2009, the | ||
130 | architecture was radically modularized into the multi-process system | ||
131 | that exists today. Coincidentally, the first version of the ARM | ||
132 | service (ARM: Automatic Restart Manager) | ||
133 | was implemented a day before systemd was announced. From 2009 | ||
134 | to 2014 work progressed rapidly thanks to a significant research grant | ||
135 | from the Deutsche Forschungsgesellschaft. This resulted in particular | ||
136 | in the creation of the R5N DHT, CADET, ATS and the GNU Name System. | ||
137 | In 2010, GNUnet was selected as the basis for the | ||
138 | @uref{https://secushare.org, secushare} online | ||
139 | social network, resulting in a significant growth of the core team. | ||
140 | In 2013, we launched @uref{https://taler.net, GNU Taler} to address | ||
141 | the challenge of convenient | ||
142 | and privacy-preserving online payments. In 2015, the | ||
143 | @c XXX: It is not correct to refer to pEp as pEp stylistic, | ||
144 | @c XXX: but the correct version would lead to problems with | ||
145 | @c XXX: some of our outputs and/or older versions of texinfo | ||
146 | @c XXX: and devices that display versions on consoles etc. | ||
147 | @c XXX: This is why we keep the pEp until proven that p(triple bar)p | ||
148 | @c XXX: does not create broken outputs. | ||
149 | @uref{https://pep.foundation/, pretty Easy privacy} (pEp) project | ||
150 | announced that they will use GNUnet as the technology for their | ||
151 | meta-data protection layer, ultimately resulting in GNUnet e.V. | ||
152 | entering into a formal long-term collaboration with the pEp | ||
153 | Foundation. In 2016, Taler Systems SA, a first startup using GNUnet | ||
154 | technology, was founded with support from the community. | ||
155 | |||
156 | GNUnet is not merely a technical project, but also a political | ||
157 | mission: like the GNU project as a whole, we are writing software to | ||
158 | achieve political goals with a focus on the human right of | ||
159 | informational self-determination. Putting users in control of their | ||
160 | computing has been the core driver of the GNU project. With GNUnet we | ||
161 | are focusing on informational self-determination for collaborative | ||
162 | computing and communication over networks. | ||
163 | |||
164 | The Internet is shaped as much by code and protocols as it is by its | ||
165 | associated political processes (IETF, ICANN, IEEE, etc.). | ||
166 | Similarly its flaws are not limited to the protocol design. Thus, | ||
167 | technical excellence by itself will not suffice to create a better | ||
168 | network. We also need to build a community that is wise, humble and | ||
169 | has a sense of humor to achieve our goal to create a technical | ||
170 | foundation for a society we would like to live in. | ||
171 | |||
172 | |||
173 | @node Project governance | ||
174 | @section Project governance | ||
175 | |||
176 | GNUnet, like the GNU project and many other free software projects, | ||
177 | follows the governance model of a benevolent dictator. This means | ||
178 | that ultimately, the GNU project appoints the GNU maintainer and can | ||
179 | overrule decisions made by the GNUnet maintainer. Similarly, the | ||
180 | GNUnet maintainer can overrule any decisions made by individual | ||
181 | developers. Still, in practice neither has happened in the last 20 | ||
182 | years for GNUnet, and we hope to keep it that way. | ||
183 | |||
184 | The current maintainers of GNUnet are: | ||
185 | |||
186 | @itemize @bullet | ||
187 | |||
188 | @item @uref{https://grothoff.org/christian/, Christian Grothoff} | ||
189 | @item @uref{https://schanzen.eu, Martin Schanzenbach} | ||
190 | |||
191 | @end itemize | ||
192 | |||
193 | The GNUnet project is supported by GNUnet e.V., a German association | ||
194 | where any developer can become a member. GNUnet e.V. serves as a | ||
195 | legal entity to hold the copyrights to GNUnet. GNUnet e.V. may also | ||
196 | choose to pay for project resources, and can collect donations as | ||
197 | well as choose to adjust the license of the | ||
198 | software (with the constraint that it has to remain free software). | ||
199 | In 2018 we switched from GPL3 to AGPL3, in practice these changes do | ||
200 | not happen very often. | ||
201 | |||
202 | |||
203 | @node Typography | ||
204 | @section Typography | ||
205 | |||
206 | When giving examples for commands, shell prompts are used to show if the | ||
207 | command should/can be issued as root, or if "normal" user privileges are | ||
208 | sufficient. We use a @code{#} for root's shell prompt, a | ||
209 | @code{%} for users' shell prompt, assuming they use the C-shell or tcsh | ||
210 | and a @code{$} for bourne shell and derivatives. | ||
211 | @c TODO: Really? Why the different prompts? Do we already have c-shell | ||
212 | @c TODO: examples? | ||
diff --git a/doc/handbook/chapters/user.texi b/doc/handbook/chapters/user.texi deleted file mode 100644 index 714336228..000000000 --- a/doc/handbook/chapters/user.texi +++ /dev/null | |||
@@ -1,2603 +0,0 @@ | |||
1 | @node Using GNUnet | ||
2 | @chapter Using GNUnet | ||
3 | |||
4 | |||
5 | This tutorial is supposed to give a first introduction for users | ||
6 | trying to do something real with GNUnet. Installation and | ||
7 | configuration are specifically outside of the scope of this tutorial. | ||
8 | Instead, we start by briefly checking that the installation works, and | ||
9 | then dive into uncomplicated, concrete practical things that can be done | ||
10 | with the framework provided by GNUnet. | ||
11 | |||
12 | In short, this chapter of the ``GNUnet Reference Documentation'' will | ||
13 | show you how to use the various peer-to-peer applications of the | ||
14 | GNUnet system. | ||
15 | As GNUnet evolves, we will add new sections for the various | ||
16 | applications that are being created. | ||
17 | |||
18 | Comments on the content of this chapter, and extensions of it are | ||
19 | always welcome. | ||
20 | |||
21 | |||
22 | @menu | ||
23 | * Start and stop GNUnet:: | ||
24 | * First steps - Using the GNU Name System:: | ||
25 | * First steps - Using GNUnet Conversation:: | ||
26 | * First steps - Using the GNUnet VPN:: | ||
27 | * File-sharing:: | ||
28 | * The GNU Name System:: | ||
29 | * reclaimID Identity Provider:: | ||
30 | * Using the Virtual Public Network:: | ||
31 | * Using the GNUnet Messenger:: | ||
32 | @end menu | ||
33 | |||
34 | @node Start and stop GNUnet | ||
35 | @section Start and stop GNUnet | ||
36 | |||
37 | Prior to using any GNUnet-based application, one has to start a node: | ||
38 | |||
39 | @example | ||
40 | $ gnunet-arm -s -l gnunet.log | ||
41 | @end example | ||
42 | |||
43 | To stop GNUnet: | ||
44 | |||
45 | @example | ||
46 | $ gnunet-arm -e | ||
47 | @end example | ||
48 | |||
49 | @node First steps - Using the GNU Name System | ||
50 | @section First steps - Using the GNU Name System | ||
51 | |||
52 | |||
53 | @menu | ||
54 | * Preliminaries:: | ||
55 | * Managing Egos:: | ||
56 | * The GNS Tab:: | ||
57 | * Creating a Record:: | ||
58 | * Resolving GNS records:: | ||
59 | * Integration with Browsers (DNS2GNS service):: | ||
60 | * Integration with Browsers (SOCKS proxy):: | ||
61 | * Creating a Business Card:: | ||
62 | * Be Social:: | ||
63 | * Backup of Identities and Egos:: | ||
64 | * Revocation:: | ||
65 | * What's Next?:: | ||
66 | @end menu | ||
67 | |||
68 | @node Preliminaries | ||
69 | @subsection Preliminaries | ||
70 | |||
71 | In the default configuration, there are two zones defined and shipped with | ||
72 | GNUnet: | ||
73 | |||
74 | The first is ``gnunet.org'', which points to the authoritate zone of the | ||
75 | GNUnet project. It can be used to resolve, for example, ``www.gnunet.org''. | ||
76 | |||
77 | ``.pin'' is another default zone which points to a special zone also managed | ||
78 | by gnunet.org. Users may register submodomains on a first-come first-served-basis | ||
79 | at @url{https://fcfs.gnunet.org}. | ||
80 | |||
81 | Use @code{gnunet-config -s gns} to view the GNS configuration, including | ||
82 | all configured zones that are operated by other users. The respective | ||
83 | configuration entry names start with a ``.'', e.g. ``.pin''. | ||
84 | |||
85 | You can configure any number of top-level domains, and point them to | ||
86 | the respective zones of your friends! For this, simply obtain the | ||
87 | respective public key (you will learn how below) and extend the | ||
88 | configuration: | ||
89 | |||
90 | @example | ||
91 | $ gnunet-config -s gns -o .myfriend -V PUBLIC_KEY | ||
92 | @end example | ||
93 | |||
94 | @node Managing Egos | ||
95 | @subsection Managing Egos | ||
96 | |||
97 | In GNUnet, identity management is about managing egos. Egos can | ||
98 | correspond to pseudonyms or real-world identities. If you value your | ||
99 | privacy, you are encouraged to use separate egos for separate | ||
100 | activities. | ||
101 | |||
102 | Technically, an ego is first of all a public-private key pair, and | ||
103 | thus egos also always correspond to a GNS zone. Egos are managed by | ||
104 | the IDENTITY service. Note that this service has nothing to do with | ||
105 | the peer identity. The IDENTITY service essentially stores the | ||
106 | private keys under human-readable names, and keeps a mapping of which | ||
107 | private key should be used for particular important system functions. | ||
108 | The existing identities can be listed using the command | ||
109 | @command{gnunet-identity --display} | ||
110 | |||
111 | @example | ||
112 | gnu - JTDVJC69NHU6GQS4B5721MV8VM7J6G2DVRGJV0ONIT6QH7OI6D50 | ||
113 | rules - GO0T87F9BPMF8NKD5A54L2AH1T0GRML539TPFSRMCEA98182QD30 | ||
114 | @end example | ||
115 | |||
116 | |||
117 | @node The GNS Tab | ||
118 | @subsection The GNS Tab | ||
119 | |||
120 | |||
121 | Maintaining your zones is through the NAMESTORE service and is discussed | ||
122 | here. You can manage your zone using @command{gnunet-identity} and | ||
123 | @command{gnunet-namestore}, or most conveniently using | ||
124 | @command{gnunet-namestore-gtk}. | ||
125 | |||
126 | We will use the GTK+ interface in this introduction. Please start | ||
127 | @command{gnunet-gkt} and switch to the GNS tab, which is the tab in | ||
128 | the middle with the letters "GNS" connected by a graph. | ||
129 | |||
130 | Next to the ``Add'' button there is a field where you can enter the | ||
131 | label (pseudonym in IDENTITY subsystem speak) of a zone you would like | ||
132 | to create. Pushing the ``Add'' button will create the zone. | ||
133 | Afterwards, you can change the label in the combo box below at any | ||
134 | time. The label will be the top-level domain that the GNU Name System | ||
135 | will resolve using your zone. For the label, you should pick | ||
136 | a name by which you would like to | ||
137 | be known by your friends (or colleagues). You should pick a label that | ||
138 | is reasonably unique within your social group. Be aware that | ||
139 | the label will be published together with every record in that zone. | ||
140 | |||
141 | Once you have created a first zone, you should see a QR code for the | ||
142 | zone on the right. Next to it is a "Copy" button to copy the public | ||
143 | key string to the clipboard. You can also save the QR code image to | ||
144 | disk. | ||
145 | |||
146 | Furthermore, you now can see the bottom part of the dialog. The | ||
147 | bottom of the window contains the existing entries in the selected zone. | ||
148 | |||
149 | @node Creating a Record | ||
150 | @subsection Creating a Record | ||
151 | |||
152 | |||
153 | We will begin by creating a simple record in your master zone. | ||
154 | To do this, click on the text "<new name>" in the table. The field is | ||
155 | editable, allowing you to enter a fresh label. Labels are restricted | ||
156 | to 63 characters and must not contain dots. For now, simply enter | ||
157 | "test", then press ENTER to confirm. This will create a new (empty) | ||
158 | record group under the label "test". Now click on "<new record>" next | ||
159 | to the new label "test". In the drop-down menu, select "A" and push | ||
160 | ENTER to confirm. Afterwards, a new dialog will pop up, asking to enter | ||
161 | details for the "A" record. | ||
162 | |||
163 | "A" records are used in the @dfn{Domain Name System} (DNS) to specify | ||
164 | IPv4 addresses. An IPv4 address is a number that is used to identify | ||
165 | and address a computer on the Internet (version 4). Please enter | ||
166 | "217.92.15.146" in the dialog below "Destination IPv4 Address" and | ||
167 | select "Record is public". Do not change any of the other options. | ||
168 | Note that as you enter a (well-formed) IPv4 address, the "Save" | ||
169 | button in the bottom right corner becomes sensitive. In general, buttons | ||
170 | in dialogs are often insensitive as long as the contents of the dialog | ||
171 | are incorrect. | ||
172 | |||
173 | Once finished, press the "Save" button. Back in the main dialog, select | ||
174 | the tiny triangle left of the "test" label. By doing so, you get to see | ||
175 | all of the records under "test". Note that you can right-click a record | ||
176 | to edit it later. | ||
177 | |||
178 | |||
179 | @node Resolving GNS records | ||
180 | @subsection Resolving GNS records | ||
181 | |||
182 | |||
183 | Next, you should try resolving your own GNS records. The method we | ||
184 | found to be the most uncomplicated is to do this by explicitly | ||
185 | resolving using @code{gnunet-gns}. For this exercise, we will assume | ||
186 | that you used the string ``gnu'' for the pseudonym (or label) of your | ||
187 | GNS zone. If you used something else, replace ``.gnu'' with your real | ||
188 | pseudonym in the examples below. | ||
189 | |||
190 | In the shell, type: | ||
191 | |||
192 | @example | ||
193 | $ gnunet-gns -u test.gnu # what follows is the reply | ||
194 | test.gnu: | ||
195 | Got `A' record: 217.92.15.146 | ||
196 | @end example | ||
197 | |||
198 | @noindent | ||
199 | That shows that resolution works, once GNS is integrated with | ||
200 | the application. | ||
201 | |||
202 | @node Integration with Browsers (DNS2GNS service) | ||
203 | @subsection Integration with Browsers (DNS2GNS service) | ||
204 | |||
205 | Most OSes allow you to either modify your @code{/etc/resolv.conf} directly or | ||
206 | through @code{resolvectl}. | ||
207 | We are going to configure the @code{dns2gns} service in order to translate DNS name | ||
208 | queries by applications to GNS name queries where applicable and else fall back | ||
209 | to DNS. | ||
210 | |||
211 | Optionally, you may want to configure your @code{dns2gns} service to run on a | ||
212 | non-priviledged port like 5353. | ||
213 | But, in case you are going to edit @code{/etc/resolv.conf} directly, the | ||
214 | @code{dns2gns} service MUST run on port 53 as you cannot specify the port number. | ||
215 | A $FALLBACK_DNS variable should be a DNS server you trust such as your local router: | ||
216 | |||
217 | @example | ||
218 | $ gnunet-config -s dns2gns -o OPTIONS -V "-d $FALLBACK_DNS -p 5252" | ||
219 | $ gnunet-arm -i dns2gns # Make sure the service is started | ||
220 | @end example | ||
221 | |||
222 | If you edit your resolv.conf directly, it should contain and entry like this: | ||
223 | |||
224 | @example | ||
225 | nameserver 127.0.0.1 | ||
226 | @end example | ||
227 | |||
228 | In any case, it is very likely that the method of modification of your | ||
229 | resolver is OS specific. | ||
230 | Recently, the combination of NetworkManager and systemd-resolved is becoming | ||
231 | increasingly popular. | ||
232 | |||
233 | If you use resolvectl and systemd-resolved you can temporarily | ||
234 | set the nameserver like this: | ||
235 | |||
236 | @example | ||
237 | $ resolvectl $INTERFACE 127.0.0.1:5353 | ||
238 | @end example | ||
239 | |||
240 | Where @code{$INTERFACE} is your network interface such as ``eth0''. | ||
241 | |||
242 | In order to automatically set the DNS2GNS server if it is running already you | ||
243 | can use NetworkManager-dispatcher. First, enable it: | ||
244 | |||
245 | @example | ||
246 | $ sudo systemctl enable NetworkManager-dispatcher.service | ||
247 | $ sudo systemctl start NetworkManager-dispatcher.service | ||
248 | @end example | ||
249 | |||
250 | Then, create a script /etc/NetworkManager/dispatch.h/10-dns2-gns.sh: | ||
251 | |||
252 | @example | ||
253 | #!/bin/sh | ||
254 | interface=$1 | ||
255 | status=$2 | ||
256 | |||
257 | if [ "$interface" = "eth0" ]; then | ||
258 | case $status in | ||
259 | up) | ||
260 | if nc -u -z 127.0.0.1 5353; then | ||
261 | resolvectl dns $interface 127.0.0.1:5353 | ||
262 | fi | ||
263 | ;; | ||
264 | down) | ||
265 | ;; | ||
266 | esac | ||
267 | fi | ||
268 | @end example | ||
269 | |||
270 | Make sure the script is owned by root and executable: | ||
271 | |||
272 | @example | ||
273 | $ sudo root:root /etc/NetworkManager/dispatch.d/10-dns2gns.sh | ||
274 | $ sudo +x /etc/NetworkManager/dispatch.d/10-dns2gns.sh | ||
275 | @end example | ||
276 | |||
277 | You can test accessing this website using your browser or curl: | ||
278 | |||
279 | @example | ||
280 | $ curl www.gnunet.org | ||
281 | @end example | ||
282 | |||
283 | Note that ``gnunet.org'' is a domain that also exists in DNS and for which the | ||
284 | GNUnet project webservers can provide trusted TLS certificates. | ||
285 | When using non-DNS names with GNS or aliases, this may result in issues | ||
286 | when accessing HTTPS websites with browsers. | ||
287 | In order learn how to provide relief for this issue, read on. | ||
288 | |||
289 | @node Integration with Browsers (SOCKS proxy) | ||
290 | @subsection Integration with Browsers (SOCKS proxy) | ||
291 | |||
292 | While we recommend integrating GNS using the DNS2GNS service or the | ||
293 | NSSwitch plugin, you can also | ||
294 | integrate GNS directly with your browser via the @code{gnunet-gns-proxy}. | ||
295 | This method can have the advantage that the proxy can validate | ||
296 | TLS/X.509 records and thus strengthen web security; however, the proxy | ||
297 | is still a bit brittle, so expect subtle failures. We have had reasonable | ||
298 | success with Chromium, and various frustrations with Firefox in this area | ||
299 | recently. | ||
300 | |||
301 | The first step is to start the proxy. As the proxy is (usually) | ||
302 | not started by default, this is done as a unprivileged user | ||
303 | using @command{gnunet-arm -i gns-proxy}. Use @command{gnunet-arm -I} | ||
304 | as a unprivileged user to check that the proxy was actually | ||
305 | started. (The most common error for why the proxy may fail to start | ||
306 | is that you did not run @command{gnunet-gns-proxy-setup-ca} during | ||
307 | installation.) The proxy is a SOCKS5 proxy running (by default) | ||
308 | on port 7777. Thus, you need to now configure your browser to use | ||
309 | this proxy. With Chromium, you can do this by starting the browser | ||
310 | as a unprivileged user using | ||
311 | @command{chromium --proxy-server="socks5://localhost:7777"} | ||
312 | For @command{Firefox} (or @command{Icecat}), select "Edit-Preferences" | ||
313 | in the menu, and then select the "Advanced" tab in the dialog | ||
314 | and then "Network": | ||
315 | |||
316 | Here, select "Settings..." to open the proxy settings dialog. | ||
317 | Select "Manual proxy configuration" and enter @code{localhost} | ||
318 | with port 7777 under SOCKS Host. Furthermore, set the | ||
319 | checkbox ``Proxy DNS when using SOCKS v5'' at the bottom of | ||
320 | the dialog. Finally, push "OK". | ||
321 | |||
322 | You must also go to about:config and change the | ||
323 | @code{browser.fixup.alternate.enabled} option to @code{false}, | ||
324 | otherwise the browser will autoblunder an address like | ||
325 | @code{@uref{http://www.gnu/, www.gnu}} to | ||
326 | @code{@uref{http://www.gnu.com/, www.gnu.com}}. If you want | ||
327 | to resolve @@ in your own TLDs, you must additionally | ||
328 | set @code{browser.fixup.dns_first_use_for_single_words} to @code{true}. | ||
329 | |||
330 | After configuring your browser, you might want to first confirm that it | ||
331 | continues to work as before. (The proxy is still experimental and if you | ||
332 | experience "odd" failures with some webpages, you might want to disable | ||
333 | it again temporarily.) Next, test if things work by typing | ||
334 | "@uref{http://test.gnu/}" into the URL bar of your browser. | ||
335 | This currently fails with (my version of) Firefox as Firefox is | ||
336 | super-smart and tries to resolve "@uref{http://www.test.gnu/}" instead of | ||
337 | "@uref{test.gnu}". Chromium can be convinced to comply if you explicitly | ||
338 | include the "http://" prefix --- otherwise a Google search might be | ||
339 | attempted, which is not what you want. If successful, you should | ||
340 | see a simple website. | ||
341 | |||
342 | Note that while you can use GNS to access ordinary websites, this is | ||
343 | more an experimental feature and not really our primary goal at this | ||
344 | time. Still, it is a possible use-case and we welcome help with testing | ||
345 | and development. | ||
346 | |||
347 | @pindex gnunet-bcd | ||
348 | @node Creating a Business Card | ||
349 | @subsection Creating a Business Card | ||
350 | @c FIXME: Which parts of texlive are needed? Some systems offer a modular | ||
351 | @c texlive (smaller size). | ||
352 | Before we can really use GNS, you should create a business card. | ||
353 | Note that this requires having @command{LaTeX} installed on your system. | ||
354 | If you are using a Debian GNU/Linux based operating system, the | ||
355 | following command should install the required components. | ||
356 | Keep in mind that this @b{requires 3GB} of downloaded data and possibly | ||
357 | @b{even more} when unpacked. On a GNU Guix based system texlive 2017 has | ||
358 | returns a DAG size of 5032.4 MiB. | ||
359 | The packages which are confirmed to be required are: | ||
360 | |||
361 | @itemize @bullet | ||
362 | @item texlive-units | ||
363 | @item texlive-labels | ||
364 | @item texlive-pst-barcode | ||
365 | @item texlive-luatex85 | ||
366 | @item texlive-preview | ||
367 | @item texlive-pdfcrop | ||
368 | @item texlive-koma-script | ||
369 | @end itemize | ||
370 | |||
371 | |||
372 | @b{We welcome any help in identifying the required components of the | ||
373 | TexLive Distribution. This way we could just state the required components | ||
374 | without pulling in the full distribution of TexLive.} | ||
375 | |||
376 | @example | ||
377 | apt-get install texlive-full | ||
378 | @end example | ||
379 | |||
380 | @noindent | ||
381 | Start creating a business card by clicking the "Copy" button | ||
382 | in @command{gnunet-namestore-gtk}. Next, you should start | ||
383 | the @command{gnunet-bcd} program (in the terminal, on the command-line). | ||
384 | You do not need to pass any options, and please be not surprised if | ||
385 | there is no output: | ||
386 | |||
387 | @example | ||
388 | $ gnunet-bcd # seems to hang... | ||
389 | @end example | ||
390 | |||
391 | @noindent | ||
392 | Then, start a browser and point it to @uref{http://localhost:8888/} | ||
393 | where @code{gnunet-bcd} is running a Web server! | ||
394 | |||
395 | First, you might want to fill in the "GNS Public Key" field by | ||
396 | right-clicking and selecting "Paste", filling in the public key | ||
397 | from the copy you made in @command{gnunet-namestore-gtk}. | ||
398 | Then, fill in all of the other fields, including your @b{GNS NICKname}. | ||
399 | Adding a GPG fingerprint is optional. | ||
400 | Once finished, click "Submit Query". | ||
401 | If your @code{LaTeX} installation is incomplete, the result will be | ||
402 | disappointing. | ||
403 | Otherwise, you should get a PDF containing fancy 5x2 double-sided | ||
404 | translated business cards with a QR code containing your public key | ||
405 | and a GNUnet logo. | ||
406 | We'll explain how to use those a bit later. | ||
407 | You can now go back to the shell running @code{gnunet-bcd} and press | ||
408 | @b{CTRL-C} to shut down the Web server. | ||
409 | |||
410 | |||
411 | @node Be Social | ||
412 | @subsection Be Social | ||
413 | |||
414 | |||
415 | Next, you should print out your business card and be social. | ||
416 | Find a friend, help them install GNUnet and exchange business cards with | ||
417 | them. Or, if you're a desperate loner, you might try the next step with | ||
418 | your own card. Still, it'll be hard to have a conversation with | ||
419 | yourself later, so it would be better if you could find a friend. | ||
420 | You might also want a camera attached to your computer, so | ||
421 | you might need a trip to the store together. | ||
422 | |||
423 | Before we get started, we need to tell @code{gnunet-qr} which zone | ||
424 | it should import new records into. For this, run: | ||
425 | |||
426 | @pindex gnunet-identity | ||
427 | @example | ||
428 | $ gnunet-identity -s namestore -e NAME | ||
429 | @end example | ||
430 | where NAME is the name of the zone you want to import records | ||
431 | into. In our running example, this would be ``gnu''. | ||
432 | |||
433 | @pindex gnunet-qr | ||
434 | Henceforth, for every business card you collect, simply run: | ||
435 | @example | ||
436 | $ gnunet-qr | ||
437 | @end example | ||
438 | |||
439 | @noindent | ||
440 | to open a window showing whatever your camera points at. | ||
441 | Hold up your friend's business card and tilt it until | ||
442 | the QR code is recognized. At that point, the window should | ||
443 | automatically close. At that point, your friend's NICKname and their | ||
444 | public key should have been automatically imported into your zone. | ||
445 | |||
446 | Assuming both of your peers are properly integrated in the | ||
447 | GNUnet network at this time, you should thus be able to | ||
448 | resolve your friends names. Suppose your friend's nickname | ||
449 | is "Bob". Then, type | ||
450 | |||
451 | @pindex gnunet-gns | ||
452 | @example | ||
453 | $ gnunet-gns -u test.bob.gnu | ||
454 | @end example | ||
455 | |||
456 | @noindent | ||
457 | to check if your friend was as good at following instructions | ||
458 | as you were. | ||
459 | |||
460 | |||
461 | @node Backup of Identities and Egos | ||
462 | @subsection Backup of Identities and Egos | ||
463 | |||
464 | |||
465 | One should always backup their files, especially in these SSD days (our | ||
466 | team has suffered 3 SSD crashes over a span of 2 weeks). Backing up peer | ||
467 | identity and zones is achieved by copying the following files: | ||
468 | |||
469 | The peer identity file can be found | ||
470 | in @file{~/.local/share/gnunet/private_key.ecc} | ||
471 | |||
472 | The private keys of your egos are stored in the | ||
473 | directory @file{~/.local/share/gnunet/identity/egos/}. | ||
474 | They are stored in files whose filenames correspond to the zones' | ||
475 | ego names. These are probably the most important files you want | ||
476 | to backup from a GNUnet installation. | ||
477 | |||
478 | Note: All these files contain cryptographic keys and they are | ||
479 | stored without any encryption. So it is advisable to backup | ||
480 | encrypted copies of them. | ||
481 | |||
482 | |||
483 | @node Revocation | ||
484 | @subsection Revocation | ||
485 | |||
486 | Now, in the situation of an attacker gaining access to the private key of | ||
487 | one of your egos, the attacker can create records in the respective | ||
488 | GNS zone | ||
489 | and publish them as if you published them. Anyone resolving your | ||
490 | domain will get these new records and when they verify they seem | ||
491 | authentic because the attacker has signed them with your key. | ||
492 | |||
493 | To address this potential security issue, you can pre-compute | ||
494 | a revocation certificate corresponding to your ego. This certificate, | ||
495 | when published on the P2P network, flags your private key as invalid, | ||
496 | and all further resolutions or other checks involving the key will fail. | ||
497 | |||
498 | @pindex gnunet-revocation | ||
499 | A revocation certificate is thus a useful tool when things go out of | ||
500 | control, but at the same time it should be stored securely. | ||
501 | Generation of the revocation certificate for a zone can be done through | ||
502 | @command{gnunet-revocation}. For example, the following command (as | ||
503 | unprivileged user) generates a revocation file | ||
504 | @file{revocation.dat} for the zone @code{zone1}: | ||
505 | @command{gnunet-revocation -f revocation.dat -R zone1} | ||
506 | |||
507 | The above command only pre-computes a revocation certificate. It does | ||
508 | not revoke the given zone. Pre-computing a revocation certificate | ||
509 | involves computing a proof-of-work and hence may take up to 4 to 5 days | ||
510 | on a modern processor. Note that you can abort and resume the | ||
511 | calculation at any time. Also, even if you did not finish the | ||
512 | calculation, the resulting file will contain the signature, which is | ||
513 | sufficient to complete the revocation process even without access to | ||
514 | the private key. So instead of waiting for a few days, you can just | ||
515 | abort with CTRL-C, backup the revocation certificate and run the | ||
516 | calculation only if your key actually was compromised. This has the | ||
517 | disadvantage of revocation taking longer after the incident, but | ||
518 | the advantage of saving a significant amount of energy. So unless | ||
519 | you believe that a key compromise will need a rapid response, we | ||
520 | urge you to wait with generating the revocation certificate. | ||
521 | Also, the calculation is deliberately expensive, to deter people from | ||
522 | doing this just for fun (as the actual revocation operation is expensive | ||
523 | for the network, not for the peer performing the revocation). | ||
524 | |||
525 | |||
526 | @c FIXME: The Manual should give away the command using an example that is | ||
527 | @c very likely to never exist. | ||
528 | To avoid TL;DR ones from accidentally revocating their zones, we are not | ||
529 | giving away the command, but it is uncomplicated: the actual revocation is | ||
530 | performed by using the @command{-p} option of @command{gnunet-revocation}. | ||
531 | |||
532 | |||
533 | @node What's Next? | ||
534 | @subsection What's Next? | ||
535 | |||
536 | |||
537 | This may seem not like much of an application yet, but you have | ||
538 | just been one of the first to perform a decentralized secure name | ||
539 | lookup (where nobody could have altered the value supplied by your | ||
540 | friend) in a privacy-preserving manner (your query on the network | ||
541 | and the corresponding response were always encrypted). So what | ||
542 | can you really do with this? Well, to start with, you can publish your | ||
543 | GnuPG fingerprint in GNS as a "CERT" record and replace the public | ||
544 | web-of-trust with its complicated trust model with explicit names | ||
545 | and privacy-preserving resolution. Also, you should read the next | ||
546 | chapter of the tutorial and learn how to use GNS to have a | ||
547 | private conversation with your friend. Finally, help us | ||
548 | with the next GNUnet release for even more applications | ||
549 | using this new public key infrastructure. | ||
550 | |||
551 | @pindex gnunet-conservation-gtk | ||
552 | @node First steps - Using GNUnet Conversation | ||
553 | @section First steps - Using GNUnet Conversation | ||
554 | |||
555 | |||
556 | First, you should launch the graphical user interface. You can do | ||
557 | this from the command-line by typing | ||
558 | |||
559 | @example | ||
560 | $ gnunet-conversation-gtk | ||
561 | @end example | ||
562 | |||
563 | @menu | ||
564 | * Testing your Audio Equipment:: | ||
565 | * GNS Zones:: | ||
566 | @end menu | ||
567 | |||
568 | @node Testing your Audio Equipment | ||
569 | @subsection Testing your Audio Equipment | ||
570 | |||
571 | |||
572 | First, you should use @code{gnunet-conversation-test} to check that your | ||
573 | microphone and speaker are working correctly. You will be prompted to | ||
574 | speak for 5 seconds, and then those 5 seconds will be replayed to you. | ||
575 | The network is not involved in this test. If it fails, you should run | ||
576 | your pulse audio configuration tool to check that microphone and | ||
577 | speaker are not muted and, if you have multiple input/output devices, | ||
578 | that the correct device is being associated with GNUnet's audio tools. | ||
579 | |||
580 | @node GNS Zones | ||
581 | @subsection GNS Zones | ||
582 | |||
583 | |||
584 | @code{gnunet-conversation} uses GNS for addressing. This means that | ||
585 | you need to have a GNS zone created before using it. Information | ||
586 | about how to create GNS zones can be found here. | ||
587 | |||
588 | |||
589 | @menu | ||
590 | * Picking an Identity:: | ||
591 | * Calling somebody:: | ||
592 | @end menu | ||
593 | |||
594 | @node Picking an Identity | ||
595 | @subsubsection Picking an Identity | ||
596 | |||
597 | |||
598 | To make a call with @code{gnunet-conversation}, you first | ||
599 | need to choose an identity. This identity is both the caller ID | ||
600 | that will show up when you call somebody else, as well as the | ||
601 | GNS zone that will be used to resolve names of users that you | ||
602 | are calling. Run | ||
603 | |||
604 | @pindex gnunet-conversation | ||
605 | @example | ||
606 | gnunet-conversation -e zone-name | ||
607 | @end example | ||
608 | |||
609 | @noindent | ||
610 | to start the command-line tool. You will see a message saying | ||
611 | that your phone is now "active on line 0". You can connect | ||
612 | multiple phones on different lines at the same peer. For the | ||
613 | first phone, the line zero is of course a fine choice. | ||
614 | |||
615 | Next, you should type in @command{/help} for a list of | ||
616 | available commands. We will explain the important ones | ||
617 | during this tutorial. First, you will need to type in | ||
618 | @command{/address} to determine the address of your | ||
619 | phone. The result should look something like this: | ||
620 | |||
621 | @example | ||
622 | /address | ||
623 | 0-PD67SGHF3E0447TU9HADIVU9OM7V4QHTOG0EBU69TFRI2LG63DR0 | ||
624 | @end example | ||
625 | |||
626 | @noindent | ||
627 | Here, the "0" is your phone line, and what follows | ||
628 | after the hyphen is your peer's identity. This information will | ||
629 | need to be placed in a PHONE record of | ||
630 | your GNS master-zone so that other users can call you. | ||
631 | |||
632 | Start @code{gnunet-namestore-gtk} now (possibly from another | ||
633 | shell) and create an entry home-phone in your master zone. | ||
634 | For the record type, select PHONE. You should then see the | ||
635 | PHONE dialog: | ||
636 | |||
637 | @image{images/gnunet-namestore-gtk-phone,5in,,Dialog to publish a PHONE record} | ||
638 | |||
639 | Note: Do not choose the expiry time to be 'Never'. If you | ||
640 | do that, you assert that this record will never change and | ||
641 | can be cached indefinitely by the DHT and the peers which | ||
642 | resolve this record. A reasonable period is 1 year. | ||
643 | |||
644 | Enter your peer identity under Peer and leave the line | ||
645 | at zero. Select the first option to make the record public. | ||
646 | If you entered your peer identity incorrectly, | ||
647 | the "Save" button will not work; you might want to use | ||
648 | copy-and-paste instead of typing in the peer identity | ||
649 | manually. Save the record. | ||
650 | |||
651 | @node Calling somebody | ||
652 | @subsubsection Calling somebody | ||
653 | |||
654 | |||
655 | Now you can call a buddy. Obviously, your buddy will have to have GNUnet | ||
656 | installed and must have performed the same steps. Also, you must have | ||
657 | your buddy in your GNS master zone, for example by having imported | ||
658 | your buddy's public key using @code{gnunet-qr}. Suppose your buddy | ||
659 | is in your zone as @code{buddy.mytld} and they also created their | ||
660 | phone using a label "home-phone". Then you can initiate a call using: | ||
661 | |||
662 | @example | ||
663 | /call home-phone.buddy.mytld | ||
664 | @end example | ||
665 | |||
666 | It may take some time for GNUnet to resolve the name and to establish | ||
667 | a link. If your buddy has your public key in their master zone, they | ||
668 | should see an incoming call with your name. If your public key is not | ||
669 | in their master zone, they will just see the public key as the caller ID. | ||
670 | |||
671 | Your buddy then can answer the call using the "/accept" command. After | ||
672 | that, (encrypted) voice data should be relayed between your two peers. | ||
673 | Either of you can end the call using @command{/cancel}. You can exit | ||
674 | @code{gnunet-conversation} using @command{/quit}. | ||
675 | |||
676 | |||
677 | @node First steps - Using the GNUnet VPN | ||
678 | @section First steps - Using the GNUnet VPN | ||
679 | |||
680 | |||
681 | |||
682 | @menu | ||
683 | * VPN Preliminaries:: | ||
684 | * GNUnet-Exit configuration:: | ||
685 | * GNS configuration:: | ||
686 | * Accessing the service:: | ||
687 | * Using a Browser:: | ||
688 | @end menu | ||
689 | |||
690 | @node VPN Preliminaries | ||
691 | @subsection VPN Preliminaries | ||
692 | |||
693 | |||
694 | To test the GNUnet VPN, we should first run a web server. | ||
695 | The easiest way to do this is to just start @code{gnunet-bcd}, | ||
696 | which will run a webserver on port @code{8888} by default. | ||
697 | Naturally, you can run some other HTTP server for our little tutorial. | ||
698 | |||
699 | If you have not done this, you should also configure your | ||
700 | Name System Service switch to use GNS. In your @code{/etc/nsswitch.conf} | ||
701 | you should fine a line like this: | ||
702 | |||
703 | @example | ||
704 | hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 | ||
705 | @end example | ||
706 | |||
707 | @noindent | ||
708 | The exact details may differ a bit, which is fine. Add the text | ||
709 | @code{gns [NOTFOUND=return]} after @code{files}: | ||
710 | |||
711 | @example | ||
712 | hosts: files gns [NOTFOUND=return] mdns4_minimal [NOTFOUND=return] dns mdns4 | ||
713 | @end example | ||
714 | |||
715 | @c TODO: outdated section, we no longer install this as part of the | ||
716 | @c TODO: standard installation procedure and should point out the manual | ||
717 | @c TODO: steps required to make it useful. | ||
718 | @noindent | ||
719 | You might want to make sure that @code{/lib/libnss_gns.so.2} exists on | ||
720 | your system, it should have been created during the installation. | ||
721 | If not, re-run | ||
722 | |||
723 | @example | ||
724 | $ configure --with-nssdir=/lib | ||
725 | $ cd src/gns/nss; sudo make install | ||
726 | @end example | ||
727 | |||
728 | @noindent | ||
729 | to install the NSS plugins in the proper location. | ||
730 | |||
731 | @node GNUnet-Exit configuration | ||
732 | @subsection GNUnet-Exit configuration | ||
733 | |||
734 | |||
735 | Stop your peer (as user @code{gnunet}, run @command{gnunet-arm -e}) and | ||
736 | run @command{gnunet-setup}. In @command{gnunet-setup}, make sure to | ||
737 | activate the @strong{EXIT} and @strong{GNS} services in the General tab. | ||
738 | Then select the Exit tab. Most of the defaults should be fine (but | ||
739 | you should check against the screenshot that they have not been modified). | ||
740 | In the bottom area, enter @code{bcd} under Identifier and change the | ||
741 | Destination to @code{169.254.86.1:8888} (if your server runs on a port | ||
742 | other than 8888, change the 8888 port accordingly). | ||
743 | |||
744 | Now exit @command{gnunet-setup} and restart your peer | ||
745 | (@command{gnunet-arm -s}). | ||
746 | |||
747 | @node GNS configuration | ||
748 | @subsection GNS configuration | ||
749 | |||
750 | |||
751 | Now, using your normal user (not the @code{gnunet} system user), run | ||
752 | @command{gnunet-namestore-gtk}. Add a new label www in your | ||
753 | master zone. For the record type, select @code{VPN}. You should then | ||
754 | see the VPN dialog: | ||
755 | |||
756 | @image{images/gnunet-namestore-gtk-vpn,5in,,Dialog to publish a VPN record} | ||
757 | |||
758 | Under peer, you need to supply the peer identity of your own peer. You can | ||
759 | obtain the respective string by running @command{gnunet-peerinfo -sq} | ||
760 | as the @code{gnunet} user. For the Identifier, you need to supply the same | ||
761 | identifier that we used in the Exit setup earlier, so here supply "bcd". | ||
762 | If you want others to be able to use the service, you should probably make | ||
763 | the record public. For non-public services, you should use a passphrase | ||
764 | instead of the string "bcd". Save the record and | ||
765 | exit @command{gnunet-namestore-gtk}. | ||
766 | |||
767 | @node Accessing the service | ||
768 | @subsection Accessing the service | ||
769 | |||
770 | |||
771 | You should now be able to access your webserver. Type in: | ||
772 | |||
773 | @example | ||
774 | $ wget http://www.gnu/ | ||
775 | @end example | ||
776 | |||
777 | @noindent | ||
778 | The request will resolve to the VPN record, telling the GNS resolver | ||
779 | to route it via the GNUnet VPN. The GNS resolver will ask the | ||
780 | GNUnet VPN for an IPv4 address to return to the application. The | ||
781 | VPN service will use the VPN information supplied by GNS to create | ||
782 | a tunnel (via GNUnet's MESH service) to the EXIT peer. | ||
783 | At the EXIT, the name "bcd" and destination port (80) will be mapped | ||
784 | to the specified destination IP and port. While all this is currently | ||
785 | happening on just the local machine, it should also work with other | ||
786 | peers --- naturally, they will need a way to access your GNS zone | ||
787 | first, for example by learning your public key from a QR code on | ||
788 | your business card. | ||
789 | |||
790 | @node Using a Browser | ||
791 | @subsection Using a Browser | ||
792 | |||
793 | |||
794 | Sadly, modern browsers tend to bypass the Name Services Switch and | ||
795 | attempt DNS resolution directly. You can either run | ||
796 | a @code{gnunet-dns2gns} DNS proxy, or point the browsers to an | ||
797 | HTTP proxy. When we tried it, Iceweasel did not like to connect to | ||
798 | the socks proxy for @code{.gnu} TLDs, even if we disabled its | ||
799 | autoblunder of changing @code{.gnu} to ".gnu.com". Still, | ||
800 | using the HTTP proxy with Chrome does work. | ||
801 | |||
802 | @node File-sharing | ||
803 | @section File-sharing | ||
804 | |||
805 | |||
806 | This chapter documents the GNUnet file-sharing application. The original | ||
807 | file-sharing implementation for GNUnet was designed to provide | ||
808 | @strong{anonymous} file-sharing. However, over time, we have also added | ||
809 | support for non-anonymous file-sharing (which can provide better | ||
810 | performance). Anonymous and non-anonymous file-sharing are quite | ||
811 | integrated in GNUnet and, except for routing, share most of the concepts | ||
812 | and implementation. There are three primary file-sharing operations: | ||
813 | publishing, searching and downloading. For each of these operations, | ||
814 | the user specifies an @strong{anonymity level}. If both the publisher and | ||
815 | the searcher/downloader specify "no anonymity", non-anonymous | ||
816 | file-sharing is used. If either user specifies some desired degree | ||
817 | of anonymity, anonymous file-sharing will be used. | ||
818 | |||
819 | After a short introduction, we will first look at the various concepts | ||
820 | in GNUnet's file-sharing implementation. Then, we will discuss | ||
821 | specifics as to how they impact users that publish, search or download | ||
822 | files. | ||
823 | |||
824 | |||
825 | @menu | ||
826 | * fs-Searching:: | ||
827 | * fs-Downloading:: | ||
828 | * fs-Publishing:: | ||
829 | * fs-Concepts:: | ||
830 | * Namespace Management:: | ||
831 | * File-Sharing URIs:: | ||
832 | * GTK User Interface:: | ||
833 | @end menu | ||
834 | |||
835 | @node fs-Searching | ||
836 | @subsection Searching | ||
837 | |||
838 | |||
839 | The command @command{gnunet-search} can be used to search | ||
840 | for content on GNUnet. The format is: | ||
841 | |||
842 | @example | ||
843 | $ gnunet-search [-t TIMEOUT] KEYWORD | ||
844 | @end example | ||
845 | |||
846 | @noindent | ||
847 | The @command{-t} option specifies that the query should timeout after | ||
848 | approximately TIMEOUT seconds. A value of zero (``0'') is interpreted | ||
849 | as @emph{no timeout}, which is the default. In this case, | ||
850 | @command{gnunet-search} will never terminate (unless you press | ||
851 | @command{CTRL-C}). | ||
852 | |||
853 | If multiple words are passed as keywords, they will all be | ||
854 | considered optional. Prefix keywords with a "+" to make them mandatory. | ||
855 | |||
856 | Note that searching using | ||
857 | |||
858 | @example | ||
859 | $ gnunet-search Das Kapital | ||
860 | @end example | ||
861 | |||
862 | @noindent | ||
863 | is not the same as searching for | ||
864 | |||
865 | @example | ||
866 | $ gnunet-search "Das Kapital" | ||
867 | @end example | ||
868 | |||
869 | @noindent | ||
870 | as the first will match files shared under the keywords | ||
871 | "Das" or "Kapital" whereas the second will match files | ||
872 | shared under the keyword "Das Kapital". | ||
873 | |||
874 | Search results are printed by @command{gnunet-search} like this: | ||
875 | |||
876 | @c it will be better the avoid the ellipsis altogether because I don't | ||
877 | @c understand the explanation below that | ||
878 | @c ng0: who is ``I'' and what was the complete sentence? | ||
879 | @example | ||
880 | #15: | ||
881 | gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446 | ||
882 | |||
883 | @end example | ||
884 | |||
885 | @noindent | ||
886 | The whole line is the command you would have to enter to download | ||
887 | the file. The first argument passed to @code{-o} is the suggested | ||
888 | filename (you may change it to whatever you like). | ||
889 | It is followed by the key for decrypting the file, the query for | ||
890 | searching the file, a checksum (in hexadecimal) finally the size of | ||
891 | the file in bytes. | ||
892 | |||
893 | @node fs-Downloading | ||
894 | @subsection Downloading | ||
895 | |||
896 | |||
897 | In order to download a file, you need the whole line returned by | ||
898 | @command{gnunet-search}. | ||
899 | You can then use the tool @command{gnunet-download} to obtain the file: | ||
900 | |||
901 | @example | ||
902 | $ gnunet-download -o <FILENAME> <GNUNET-URL> | ||
903 | @end example | ||
904 | |||
905 | @noindent | ||
906 | FILENAME specifies the name of the file where GNUnet is supposed | ||
907 | to write the result. Existing files are overwritten. If the | ||
908 | existing file contains blocks that are identical to the | ||
909 | desired download, those blocks will not be downloaded again | ||
910 | (automatic resume). | ||
911 | |||
912 | If you want to download the GPL from the previous example, | ||
913 | you do the following: | ||
914 | |||
915 | @example | ||
916 | $ gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446 | ||
917 | @end example | ||
918 | |||
919 | @noindent | ||
920 | If you ever have to abort a download, you can continue it at any time by | ||
921 | re-issuing @command{gnunet-download} with the same filename. | ||
922 | In that case, GNUnet will @strong{not} download blocks again that are | ||
923 | already present. | ||
924 | |||
925 | GNUnet's file-encoding mechanism will ensure file integrity, even if the | ||
926 | existing file was not downloaded from GNUnet in the first place. | ||
927 | |||
928 | You may want to use the @command{-V} switch to turn on verbose | ||
929 | reporting. In this case, @command{gnunet-download} will print the | ||
930 | current number of bytes downloaded whenever new data was received. | ||
931 | |||
932 | @node fs-Publishing | ||
933 | @subsection Publishing | ||
934 | |||
935 | |||
936 | The command @command{gnunet-publish} can be used to add content | ||
937 | to the network. The basic format of the command is | ||
938 | |||
939 | @example | ||
940 | $ gnunet-publish [-n] [-k KEYWORDS]* [-m TYPE:VALUE] FILENAME | ||
941 | @end example | ||
942 | |||
943 | For example | ||
944 | @example | ||
945 | $ gnunet-publish -m "description:GNU License" -k gpl -k test -m "mimetype:text/plain" COPYING | ||
946 | @end example | ||
947 | |||
948 | @menu | ||
949 | * Important command-line options:: | ||
950 | * Indexing vs. Inserting:: | ||
951 | @end menu | ||
952 | |||
953 | @node Important command-line options | ||
954 | @subsubsection Important command-line options | ||
955 | |||
956 | |||
957 | The option @code{-k} is used to specify keywords for the file that | ||
958 | should be inserted. You can supply any number of keywords, | ||
959 | and each of the keywords will be sufficient to locate and | ||
960 | retrieve the file. Please note that you must use the @code{-k} option | ||
961 | more than once -- one for each expression you use as a keyword for | ||
962 | the filename. | ||
963 | |||
964 | The -m option is used to specify meta-data, such as descriptions. | ||
965 | You can use -m multiple times. The TYPE passed must be from the | ||
966 | list of meta-data types known to libextractor. You can obtain this | ||
967 | list by running @command{extract -L}. Use quotes around the entire | ||
968 | meta-data argument if the value contains spaces. The meta-data | ||
969 | is displayed to other users when they select which files to | ||
970 | download. The meta-data and the keywords are optional and | ||
971 | may be inferred using @code{GNU libextractor}. | ||
972 | |||
973 | @command{gnunet-publish} has a few additional options to handle | ||
974 | namespaces and directories. Refer to the man-page for details: | ||
975 | |||
976 | @example | ||
977 | man gnunet-publish | ||
978 | @end example | ||
979 | |||
980 | @node Indexing vs. Inserting | ||
981 | @subsubsection Indexing vs Inserting | ||
982 | |||
983 | |||
984 | By default, GNUnet indexes a file instead of making a full copy. | ||
985 | This is much more efficient, but requires the file to stay unaltered | ||
986 | at the location where it was when it was indexed. If you intend to move, | ||
987 | delete or alter a file, consider using the option @code{-n} which will | ||
988 | force GNUnet to make a copy of the file in the database. | ||
989 | |||
990 | Since it is much less efficient, this is strongly discouraged for large | ||
991 | files. When GNUnet indexes a file (default), GNUnet does @strong{not} | ||
992 | create an additional encrypted copy of the file but just computes a | ||
993 | summary (or index) of the file. That summary is approximately two percent | ||
994 | of the size of the original file and is stored in GNUnet's database. | ||
995 | Whenever a request for a part of an indexed file reaches GNUnet, | ||
996 | this part is encrypted on-demand and send out. This way, there is no | ||
997 | need for an additional encrypted copy of the file to stay anywhere | ||
998 | on the drive. This is different from other systems, such as Freenet, | ||
999 | where each file that is put online must be in Freenet's database in | ||
1000 | encrypted format, doubling the space requirements if the user wants | ||
1001 | to preserve a directly accessible copy in plaintext. | ||
1002 | |||
1003 | Thus indexing should be used for all files where the user will keep | ||
1004 | using this file (at the location given to gnunet-publish) and does | ||
1005 | not want to retrieve it back from GNUnet each time. If you want to | ||
1006 | remove a file that you have indexed from the local peer, use the tool | ||
1007 | @command{gnunet-unindex} to un-index the file. | ||
1008 | |||
1009 | The option @code{-n} may be used if the user fears that the file might | ||
1010 | be found on their drive (assuming the computer comes under the control | ||
1011 | of an adversary). When used with the @code{-n} flag, the user has a | ||
1012 | much better chance of denying knowledge of the existence of the file, | ||
1013 | even if it is still (encrypted) on the drive and the adversary is | ||
1014 | able to crack the encryption (e.g. by guessing the keyword. | ||
1015 | |||
1016 | @node fs-Concepts | ||
1017 | @subsection Concepts | ||
1018 | |||
1019 | |||
1020 | For better results with filesharing it is useful to understand the | ||
1021 | following concepts. | ||
1022 | In addition to anonymous routing GNUnet attempts to give users a better | ||
1023 | experience in searching for content. GNUnet uses cryptography to safely | ||
1024 | break content into smaller pieces that can be obtained from different | ||
1025 | sources without allowing participants to corrupt files. GNUnet makes it | ||
1026 | difficult for an adversary to send back bogus search results. GNUnet | ||
1027 | enables content providers to group related content and to establish a | ||
1028 | reputation. Furthermore, GNUnet allows updates to certain content to be | ||
1029 | made available. This section is supposed to introduce users to the | ||
1030 | concepts that are used to achieve these goals. | ||
1031 | |||
1032 | |||
1033 | @menu | ||
1034 | * Files:: | ||
1035 | * Keywords:: | ||
1036 | * Directories:: | ||
1037 | * Egos and File-Sharing:: | ||
1038 | * Namespaces:: | ||
1039 | * Advertisements:: | ||
1040 | * Anonymity level:: | ||
1041 | * Content Priority:: | ||
1042 | * Replication:: | ||
1043 | @end menu | ||
1044 | |||
1045 | @node Files | ||
1046 | @subsubsection Files | ||
1047 | |||
1048 | |||
1049 | A file in GNUnet is just a sequence of bytes. Any file-format is allowed | ||
1050 | and the maximum file size is theoretically @math{2^64 - 1} bytes, except | ||
1051 | that it would take an impractical amount of time to share such a file. | ||
1052 | GNUnet itself never interprets the contents of shared files, except when | ||
1053 | using GNU libextractor to obtain keywords. | ||
1054 | |||
1055 | @node Keywords | ||
1056 | @subsubsection Keywords | ||
1057 | |||
1058 | |||
1059 | Keywords are the most simple mechanism to find files on GNUnet. | ||
1060 | Keywords are @strong{case-sensitive} and the search string | ||
1061 | must always match @strong{exactly} the keyword used by the | ||
1062 | person providing the file. Keywords are never transmitted in | ||
1063 | plaintext. The only way for an adversary to determine the keyword | ||
1064 | that you used to search is to guess it (which then allows the | ||
1065 | adversary to produce the same search request). Since providing | ||
1066 | keywords by hand for each shared file is tedious, GNUnet uses | ||
1067 | GNU libextractor to help automate this process. Starting a | ||
1068 | keyword search on a slow machine can take a little while since | ||
1069 | the keyword search involves computing a fresh RSA key to formulate the | ||
1070 | request. | ||
1071 | |||
1072 | @node Directories | ||
1073 | @subsubsection Directories | ||
1074 | |||
1075 | |||
1076 | A directory in GNUnet is a list of file identifiers with meta data. | ||
1077 | The file identifiers provide sufficient information about the files | ||
1078 | to allow downloading the contents. Once a directory has been created, | ||
1079 | it cannot be changed since it is treated just like an ordinary file | ||
1080 | by the network. Small files (of a few kilobytes) can be inlined in | ||
1081 | the directory, so that a separate download becomes unnecessary. | ||
1082 | |||
1083 | Directories are shared just like ordinary files. If you download a | ||
1084 | directory with @command{gnunet-download}, you can use | ||
1085 | @command{gnunet-directory} to list its contents. The canonical | ||
1086 | extension for GNUnet directories when stored as files in your | ||
1087 | local file-system is ".gnd". The contents of a directory are URIs and | ||
1088 | meta data. | ||
1089 | The URIs contain all the information required by | ||
1090 | @command{gnunet-download} to retrieve the file. The meta data | ||
1091 | typically includes the mime-type, description, a filename and | ||
1092 | other meta information, and possibly even the full original file | ||
1093 | (if it was small). | ||
1094 | |||
1095 | @node Egos and File-Sharing | ||
1096 | @subsubsection Egos and File-Sharing | ||
1097 | |||
1098 | When sharing files, it is sometimes desirable to build a reputation as | ||
1099 | a source for quality information. With egos, publishers can | ||
1100 | (cryptographically) sign files, thereby demonstrating that various | ||
1101 | files were published by the same entity. An ego thus allows users to | ||
1102 | link different publication events, thereby deliberately reducing | ||
1103 | anonymity to pseudonymity. | ||
1104 | |||
1105 | Egos used in GNUnet's file-sharing for such pseudonymous publishing | ||
1106 | also correspond to the egos used to identify and sign zones in the | ||
1107 | GNU Name System. However, if the same ego is used for file-sharing | ||
1108 | and for a GNS zone, this will weaken the privacy assurances provided | ||
1109 | by the anonymous file-sharing protocol. | ||
1110 | |||
1111 | Note that an ego is NOT bound to a GNUnet peer. There can be multiple | ||
1112 | egos for a single user, and users could (theoretically) share | ||
1113 | the private keys of an ego by copying the respective private keys. | ||
1114 | |||
1115 | |||
1116 | @node Namespaces | ||
1117 | @subsubsection Namespaces | ||
1118 | |||
1119 | A namespace is a set of files that were signed by the same ego. | ||
1120 | Today, namespaces are implemented independently of GNS zones, but | ||
1121 | in the future we plan to merge the two such that a GNS zone can | ||
1122 | basically contain files using a file-sharing specific record type. | ||
1123 | |||
1124 | Files (or directories) that have been signed and placed into a | ||
1125 | namespace can be updated. Updates are identified as authentic if the | ||
1126 | same secret key was used to sign the update. | ||
1127 | |||
1128 | @node Advertisements | ||
1129 | @subsubsection Advertisements | ||
1130 | |||
1131 | Advertisements are used to notify other users about the existence of a | ||
1132 | namespace. Advertisements are propagated using the normal keyword | ||
1133 | search. When an advertisement is received (in response to a search), | ||
1134 | the namespace is added to the list of namespaces available in the | ||
1135 | namespace-search dialogs of gnunet-fs-gtk and printed by | ||
1136 | @code{gnunet-identity}. Whenever a namespace is created, an | ||
1137 | appropriate advertisement can be generated. The default keyword for | ||
1138 | the advertising of namespaces is "namespace". | ||
1139 | |||
1140 | |||
1141 | @node Anonymity level | ||
1142 | @subsubsection Anonymity level | ||
1143 | |||
1144 | The anonymity level determines how hard it should be for an adversary to | ||
1145 | determine the identity of the publisher or the searcher/downloader. An | ||
1146 | anonymity level of zero means that anonymity is not required. The default | ||
1147 | anonymity level of "1" means that anonymous routing is desired, but no | ||
1148 | particular amount of cover traffic is necessary. A powerful adversary | ||
1149 | might thus still be able to deduce the origin of the traffic using | ||
1150 | traffic analysis. Specifying higher anonymity levels increases the | ||
1151 | amount of cover traffic required. | ||
1152 | |||
1153 | The specific numeric value (for anonymity levels above 1) is simple: | ||
1154 | Given an anonymity level L (above 1), each request FS makes on your | ||
1155 | behalf must be hidden in L-1 equivalent requests of cover traffic | ||
1156 | (traffic your peer routes for others) in the same time-period. The | ||
1157 | time-period is twice the average delay by which GNUnet artificially | ||
1158 | delays traffic. | ||
1159 | |||
1160 | While higher anonymity levels may offer better privacy, they can also | ||
1161 | significantly hurt performance. | ||
1162 | |||
1163 | |||
1164 | @node Content Priority | ||
1165 | @subsubsection Content Priority | ||
1166 | |||
1167 | Depending on the peer's configuration, GNUnet peers migrate content | ||
1168 | between peers. Content in this sense are individual blocks of a file, | ||
1169 | not necessarily entire files. When peers run out of space (due to | ||
1170 | local publishing operations or due to migration of content from other | ||
1171 | peers), blocks sometimes need to be discarded. GNUnet first always | ||
1172 | discards expired blocks (typically, blocks are published with an | ||
1173 | expiration of about two years in the future; this is another option). | ||
1174 | If there is still not enough space, GNUnet discards the blocks with the | ||
1175 | lowest priority. The priority of a block is decided by its popularity | ||
1176 | (in terms of requests from peers we trust) and, in case of blocks | ||
1177 | published locally, the base-priority that was specified by the user | ||
1178 | when the block was published initially. | ||
1179 | |||
1180 | |||
1181 | @node Replication | ||
1182 | @subsubsection Replication | ||
1183 | |||
1184 | When peers migrate content to other systems, the replication level | ||
1185 | of a block is used to decide which blocks need to be migrated most | ||
1186 | urgently. GNUnet will always push the block with the highest | ||
1187 | replication level into the network, and then decrement the replication | ||
1188 | level by one. If all blocks reach replication level zero, the | ||
1189 | selection is simply random. | ||
1190 | |||
1191 | |||
1192 | @node Namespace Management | ||
1193 | @subsection Namespace Management | ||
1194 | |||
1195 | The @code{gnunet-identity} tool can be used to create egos. | ||
1196 | By default, @code{gnunet-identity --display} simply | ||
1197 | lists all locally available egos. | ||
1198 | |||
1199 | |||
1200 | @menu | ||
1201 | * Creating Egos:: | ||
1202 | * Deleting Egos:: | ||
1203 | @end menu | ||
1204 | |||
1205 | @node Creating Egos | ||
1206 | @subsubsection Creating Egos | ||
1207 | |||
1208 | With the @command{--create=NICK} option it can also be used to create a new | ||
1209 | ego. An ego is the virtual identity of the entity in control of a | ||
1210 | namespace or GNS zone. Anyone can create any number of egos. The | ||
1211 | provided NICK name automatically corresponds to a GNU Name System | ||
1212 | domain name. Thus, henceforth name resolution for any name ending in | ||
1213 | ``.NICK'' will use the NICK's zone. You should avoid using NICKs that | ||
1214 | collide with well-known DNS names. | ||
1215 | |||
1216 | Currently, the IDENTITY subsystem supports two types of identity keys: | ||
1217 | ECDSA and EdDSA. By default, ECDSA identities are creates with ECDSA keys. | ||
1218 | In order to create an identity with EdDSA keys, you can use the | ||
1219 | @command{--eddsa} flag. | ||
1220 | |||
1221 | @node Deleting Egos | ||
1222 | @subsubsection Deleting Egos | ||
1223 | |||
1224 | With the @command{-D NICK} option egos can be deleted. Once the ego | ||
1225 | has been deleted it is impossible to add content to the corresponding | ||
1226 | namespace or zone. However, the existing GNS zone data is currently | ||
1227 | not dropped. This may change in the future. | ||
1228 | |||
1229 | Deleting the pseudonym does not make the namespace or any content in | ||
1230 | it unavailable. | ||
1231 | |||
1232 | @node File-Sharing URIs | ||
1233 | @subsection File-Sharing URIs | ||
1234 | |||
1235 | |||
1236 | GNUnet (currently) uses four different types of URIs for | ||
1237 | file-sharing. They all begin with "gnunet://fs/". | ||
1238 | This section describes the four different URI types in detail. | ||
1239 | |||
1240 | For FS URIs empty KEYWORDs are not allowed. Quotes are allowed to | ||
1241 | denote whitespace between words. Keywords must contain a balanced | ||
1242 | number of double quotes. Doubles quotes can not be used in the actual | ||
1243 | keywords. This means that the string '""foo bar""' will be turned | ||
1244 | into two OR-ed keywords 'foo' and 'bar', not into '"foo bar"'. | ||
1245 | |||
1246 | @menu | ||
1247 | * Encoding of hash values in URIs:: | ||
1248 | * Content Hash Key (chk):: | ||
1249 | * Location identifiers (loc):: | ||
1250 | * Keyword queries (ksk):: | ||
1251 | * Namespace content (sks):: | ||
1252 | @end menu | ||
1253 | |||
1254 | @node Encoding of hash values in URIs | ||
1255 | @subsubsection Encoding of hash values in URIs | ||
1256 | |||
1257 | |||
1258 | Most URIs include some hash values. Hashes are encoded using | ||
1259 | base32hex (RFC 2938). | ||
1260 | |||
1261 | @cindex chk-uri | ||
1262 | @node Content Hash Key (chk) | ||
1263 | @subsubsection Content Hash Key (chk) | ||
1264 | |||
1265 | |||
1266 | A chk-URI is used to (uniquely) identify a file or directory | ||
1267 | and to allow peers to download the file. Files are stored in | ||
1268 | GNUnet as a tree of encrypted blocks. | ||
1269 | The chk-URI thus contains the information to download and decrypt | ||
1270 | those blocks. A chk-URI has the format | ||
1271 | "gnunet://fs/chk/KEYHASH.QUERYHASH.SIZE". Here, "SIZE" | ||
1272 | is the size of the file (which allows a peer to determine the | ||
1273 | shape of the tree), KEYHASH is the key used to decrypt the file | ||
1274 | (also the hash of the plaintext of the top block) and QUERYHASH | ||
1275 | is the query used to request the top-level block (also the hash | ||
1276 | of the encrypted block). | ||
1277 | |||
1278 | @cindex loc-uri | ||
1279 | @node Location identifiers (loc) | ||
1280 | @subsubsection Location identifiers (loc) | ||
1281 | |||
1282 | |||
1283 | For non-anonymous file-sharing, loc-URIs are used to specify which | ||
1284 | peer is offering the data (in addition to specifying all of the | ||
1285 | data from a chk-URI). Location identifiers include a digital | ||
1286 | signature of the peer to affirm that the peer is truly the | ||
1287 | origin of the data. The format is | ||
1288 | "gnunet://fs/loc/KEYHASH.QUERYHASH.SIZE.PEER.SIG.EXPTIME". | ||
1289 | Here, "PEER" is the public key of the peer (in GNUnet format in | ||
1290 | base32hex), SIG is the RSA signature (in GNUnet format in | ||
1291 | base32hex) and EXPTIME specifies when the signature expires | ||
1292 | (in milliseconds after 1970). | ||
1293 | |||
1294 | @cindex ksk-uri | ||
1295 | @node Keyword queries (ksk) | ||
1296 | @subsubsection Keyword queries (ksk) | ||
1297 | |||
1298 | |||
1299 | A keyword-URI is used to specify that the desired operation | ||
1300 | is the search using a particular keyword. The format is simply | ||
1301 | "gnunet://fs/ksk/KEYWORD". Non-ASCII characters can be specified | ||
1302 | using the typical URI-encoding (using hex values) from HTTP. | ||
1303 | "+" can be used to specify multiple keywords (which are then | ||
1304 | logically "OR"-ed in the search, results matching both keywords | ||
1305 | are given a higher rank): "gnunet://fs/ksk/KEYWORD1+KEYWORD2". | ||
1306 | ksk-URIs must not begin or end with the plus ('+') character. | ||
1307 | Furthermore they must not contain '++'. | ||
1308 | |||
1309 | @cindex sks-uri | ||
1310 | @node Namespace content (sks) | ||
1311 | @subsubsection Namespace content (sks) | ||
1312 | |||
1313 | |||
1314 | @b{Please note that the text in this subsection is outdated and needs} | ||
1315 | @b{to be rewritten for version 0.10!} | ||
1316 | @b{This especially concerns the terminology of Pseudonym/Ego/Identity.} | ||
1317 | |||
1318 | Namespaces are sets of files that have been approved by some (usually | ||
1319 | pseudonymous) user --- typically by that user publishing all of the | ||
1320 | files together. A file can be in many namespaces. A file is in a | ||
1321 | namespace if the owner of the ego (aka the namespace's private key) | ||
1322 | signs the CHK of the file cryptographically. An SKS-URI is used to | ||
1323 | search a namespace. The result is a block containing meta data, | ||
1324 | the CHK and the namespace owner's signature. The format of a sks-URI | ||
1325 | is "gnunet://fs/sks/NAMESPACE/IDENTIFIER". Here, "NAMESPACE" | ||
1326 | is the public key for the namespace. "IDENTIFIER" is a freely | ||
1327 | chosen keyword (or password!). A commonly used identifier is | ||
1328 | "root" which by convention refers to some kind of index or other | ||
1329 | entry point into the namespace. | ||
1330 | |||
1331 | @node GTK User Interface | ||
1332 | @subsection GTK User Interface | ||
1333 | This chapter describes first steps for file-sharing with GNUnet. | ||
1334 | To start, you should launch @command{gnunet-fs-gtk}. | ||
1335 | |||
1336 | As we want to be sure that the network contains the data that we are | ||
1337 | looking for for testing, we need to begin by publishing a file. | ||
1338 | |||
1339 | @menu | ||
1340 | * gtk-Publishing:: | ||
1341 | * gtk-Searching:: | ||
1342 | * gtk-Downloading:: | ||
1343 | @end menu | ||
1344 | |||
1345 | @node gtk-Publishing | ||
1346 | @subsubsection Publishing | ||
1347 | |||
1348 | |||
1349 | To publish a file, select "File Sharing" in the menu bar just below the | ||
1350 | "Statistics" icon, and then select "Publish" from the menu. | ||
1351 | |||
1352 | Afterwards, the following publishing dialog will appear: | ||
1353 | |||
1354 | @image{images/gnunet-gtk-0-10-fs-publish,5in,,The gnunet-fs-gtk publishing dialog} | ||
1355 | |||
1356 | In this dialog, select the "Add File" button. This will open a | ||
1357 | file selection dialog: | ||
1358 | |||
1359 | @image{images/gnunet-gtk-0-10-fs-publish-select,5in,,Dialog to select the file to publish (looks may differ for other Gtk+ versions)} | ||
1360 | |||
1361 | Now, you should select a file from your computer to be published on | ||
1362 | GNUnet. To see more of GNUnet's features later, you should pick a | ||
1363 | PNG or JPEG file this time. You can leave all of the other options | ||
1364 | in the dialog unchanged. Confirm your selection by pressing the "OK" | ||
1365 | button in the bottom right corner. Now, you will briefly see a | ||
1366 | "Messages..." dialog pop up, but most likely it will be too short for | ||
1367 | you to really read anything. That dialog is showing you progress | ||
1368 | information as GNUnet takes a first look at the selected file(s). | ||
1369 | For a normal image, this is virtually instant, but if you later | ||
1370 | import a larger directory you might be interested in the progress dialog | ||
1371 | and potential errors that might be encountered during processing. | ||
1372 | After the progress dialog automatically disappears, your file | ||
1373 | should now appear in the publishing dialog: | ||
1374 | |||
1375 | @image{images/gnunet-gtk-0-10-fs-publish-with-file,5in,,Publishing dialog with file added} | ||
1376 | |||
1377 | Now, select the file (by clicking on the file name) and then click | ||
1378 | the "Edit" button. This will open the editing dialog: | ||
1379 | |||
1380 | @image{images/gnunet-gtk-0-10-fs-publish-editing,5in,,Editing meta data of a file to be published} | ||
1381 | |||
1382 | In this dialog, you can see many details about your file. In the | ||
1383 | top left area, you can see meta data extracted about the file, | ||
1384 | such as the original filename, the mimetype and the size of the image. | ||
1385 | In the top right, you should see a preview for the image | ||
1386 | (if GNU libextractor was installed correctly with the | ||
1387 | respective plugins). Note that if you do not see a preview, this | ||
1388 | is not a disaster, but you might still want to install more of | ||
1389 | GNU libextractor in the future. In the bottom left, the dialog contains | ||
1390 | a list of keywords. These are the keywords under which the file will be | ||
1391 | made available. The initial list will be based on the extracted meta data. | ||
1392 | Additional publishing options are in the right bottom corner. We will | ||
1393 | now add an additional keyword to the list of keywords. This is done by | ||
1394 | entering the keyword above the keyword list between the label "Keyword" | ||
1395 | and the "Add keyword" button. Enter "test" and select "Add keyword". | ||
1396 | Note that the keyword will appear at the bottom of the existing keyword | ||
1397 | list, so you might have to scroll down to see it. Afterwards, push the | ||
1398 | "OK" button at the bottom right of the dialog. | ||
1399 | |||
1400 | You should now be back at the "Publish content on GNUnet" dialog. Select | ||
1401 | "Execute" in the bottom right to close the dialog and publish your file | ||
1402 | on GNUnet! Afterwards, you should see the main dialog with a new area | ||
1403 | showing the list of published files (or ongoing publishing operations | ||
1404 | with progress indicators). | ||
1405 | |||
1406 | @node gtk-Searching | ||
1407 | @subsubsection Searching | ||
1408 | |||
1409 | |||
1410 | Below the menu bar, there are four entry widges labeled "Namespace", | ||
1411 | "Keywords", "Anonymity" and "Mime-type" (from left to right). These | ||
1412 | widgets are used to control searching for files in GNUnet. Between the | ||
1413 | "Keywords" and "Anonymity" widgets, there is also a big "Search" button, | ||
1414 | which is used to initiate the search. We will ignore the "Namespace", | ||
1415 | "Anonymity" and "Mime-type" options in this tutorial, please leave them | ||
1416 | empty. Instead, simply enter "test" under "Keywords" and press "Search". | ||
1417 | Afterwards, you should immediately see a new tab labeled after your | ||
1418 | search term, followed by the (current) number of search | ||
1419 | results --- "(15)" in our screenshot. Note that your results may | ||
1420 | vary depending on what other users may have shared and how your | ||
1421 | peer is connected. | ||
1422 | |||
1423 | You can now select one of the search results. Once you do this, | ||
1424 | additional information about the result should be displayed on the | ||
1425 | right. If available, a preview image should appear on the top right. | ||
1426 | Meta data describing the file will be listed at the bottom right. | ||
1427 | |||
1428 | Once a file is selected, at the bottom of the search result list | ||
1429 | a little area for downloading appears. | ||
1430 | |||
1431 | @node gtk-Downloading | ||
1432 | @subsubsection Downloading | ||
1433 | |||
1434 | |||
1435 | In the downloading area, you can select the target directory (default is | ||
1436 | "Downloads") and specify the desired filename (by default the filename it | ||
1437 | taken from the meta data of the published file). Additionally, you can | ||
1438 | specify if the download should be anonymous and (for directories) if | ||
1439 | the download should be recursive. In most cases, you can simply start | ||
1440 | the download with the "Download!" button. | ||
1441 | |||
1442 | Once you selected download, the progress of the download will be | ||
1443 | displayed with the search result. You may need to resize the result | ||
1444 | list or scroll to the right. The "Status" column shows the current | ||
1445 | status of the download, and "Progress" how much has been completed. | ||
1446 | When you close the search tab (by clicking on the "X" button next to | ||
1447 | the "test" label), ongoing and completed downloads are not aborted | ||
1448 | but moved to a special "*" tab. | ||
1449 | |||
1450 | You can remove completed downloads from the "*" tab by clicking the | ||
1451 | cleanup button next to the "*". You can also abort downloads by right | ||
1452 | clicking on the respective download and selecting "Abort download" | ||
1453 | from the menu. | ||
1454 | |||
1455 | That's it, you now know the basics for file-sharing with GNUnet! | ||
1456 | |||
1457 | |||
1458 | @node The GNU Name System | ||
1459 | @section The GNU Name System | ||
1460 | |||
1461 | |||
1462 | |||
1463 | The GNU Name System (GNS) is secure and decentralized naming system. | ||
1464 | It allows its users to register names as @dfn{top-level domains} (TLDs) and | ||
1465 | resolve other namespaces within their TLDs. | ||
1466 | |||
1467 | GNS is designed to provide: | ||
1468 | @itemize @bullet | ||
1469 | @item Censorship resistance | ||
1470 | @item Query privacy | ||
1471 | @item Secure name resolution | ||
1472 | @item Compatibility with DNS | ||
1473 | @end itemize | ||
1474 | |||
1475 | For the initial configuration and population of your | ||
1476 | GNS installation, please follow the GNS setup instructions. | ||
1477 | The remainder of this chapter will provide some background on GNS | ||
1478 | and then describe how to use GNS in more detail. | ||
1479 | |||
1480 | Unlike DNS, GNS does not rely on central root zones or authorities. | ||
1481 | Instead any user administers their own root and can can create arbitrary | ||
1482 | name value mappings. Furthermore users can delegate resolution to other | ||
1483 | users' zones just like DNS NS records do. Zones are uniquely identified | ||
1484 | via public keys and resource records are signed using the corresponding | ||
1485 | public key. Delegation to another user's zone is done using special PKEY | ||
1486 | records and petnames. A petname is a name that can be freely chosen by | ||
1487 | the user. This results in non-unique name-value mappings as | ||
1488 | @code{@uref{http://www.bob.gnu/, www.bob.gnu}} to one user might be | ||
1489 | @code{@uref{http://www.friend.gnu/, www.friend.gnu}} for someone else. | ||
1490 | |||
1491 | |||
1492 | @menu | ||
1493 | * Creating a Zone:: | ||
1494 | * Maintaining your own Zones:: | ||
1495 | * Obtaining your Zone Key:: | ||
1496 | * Adding Links to Other Zones:: | ||
1497 | * Using Public Keys as Top Level Domains:: | ||
1498 | * Resource Records in GNS:: | ||
1499 | * Synchronizing with legacy DNS:: | ||
1500 | * Migrating an existing DNS zone into GNS:: | ||
1501 | @end menu | ||
1502 | |||
1503 | |||
1504 | @node Creating a Zone | ||
1505 | @subsection Creating a Zone | ||
1506 | |||
1507 | To use GNS, you probably should create at least one zone of your own. | ||
1508 | You can create any number of zones using the gnunet-identity tool | ||
1509 | using: | ||
1510 | |||
1511 | @example | ||
1512 | $ gnunet-identity --create="myzone" | ||
1513 | @end example | ||
1514 | |||
1515 | Henceforth, on your system you control the TLD ``myzone''. | ||
1516 | |||
1517 | All of your zones can be listed (displayed) using the | ||
1518 | @command{gnunet-identity} command line tool as well: | ||
1519 | |||
1520 | @example | ||
1521 | $ gnunet-identity --display | ||
1522 | @end example | ||
1523 | |||
1524 | @node Maintaining your own Zones | ||
1525 | @subsection Maintaining your own Zones | ||
1526 | |||
1527 | @noindent | ||
1528 | Now you can add (or edit, or remove) records in your GNS zone using the | ||
1529 | @command{gnunet-namestore-gtk} GUI or using the @command{gnunet-namestore} | ||
1530 | command-line tool. | ||
1531 | In either case, your records will be stored in an SQL database under | ||
1532 | control of the @command{gnunet-service-namestore}. | ||
1533 | Note that if multiple users use one peer, the namestore database will | ||
1534 | include the combined records of all users. | ||
1535 | However, users will not be able to see each other's records | ||
1536 | if they are marked as private. | ||
1537 | |||
1538 | To provide a short example for editing your own zone, suppose you | ||
1539 | have your own web server with the IP @code{1.2.3.4}. Then you can put an | ||
1540 | @code{A} record (@code{A} records in DNS are for IPv4 IP addresses) | ||
1541 | into your local zone ``myzone'' using the command: | ||
1542 | |||
1543 | @example | ||
1544 | $ gnunet-namestore -z myzone -a -n www -t A -V 1.2.3.4 -e never | ||
1545 | @end example | ||
1546 | |||
1547 | @noindent | ||
1548 | Afterwards, you will be able to access your webpage under "www.myzone" | ||
1549 | (assuming your webserver does not use virtual hosting, if it does, | ||
1550 | please read up on setting up the GNS proxy). | ||
1551 | |||
1552 | Similar commands will work for other types of DNS and GNS records, | ||
1553 | the syntax largely depending on the type of the record. | ||
1554 | Naturally, most users may find editing the zones using the | ||
1555 | @command{gnunet-namestore-gtk} GUI to be easier. | ||
1556 | |||
1557 | @node Obtaining your Zone Key | ||
1558 | @subsection Obtaining your Zone Key | ||
1559 | |||
1560 | Each zone in GNS has a public-private key. Usually, gnunet-namestore and | ||
1561 | gnunet-setup will access your private key as necessary, so you do not | ||
1562 | have to worry about those. What is important is your public key | ||
1563 | (or rather, the hash of your public key), as you will likely want to | ||
1564 | give it to others so that they can securely link to you. | ||
1565 | |||
1566 | You can usually get the hash of your public key using | ||
1567 | |||
1568 | @example | ||
1569 | $ gnunet-identity -d $options | grep myzone | awk '@{print $3@}' | ||
1570 | @end example | ||
1571 | |||
1572 | @noindent | ||
1573 | For example, the output might be something like: | ||
1574 | |||
1575 | @example | ||
1576 | DC3SEECJORPHQNVRH965A6N74B1M37S721IG4RBQ15PJLLPJKUE0 | ||
1577 | @end example | ||
1578 | |||
1579 | @noindent | ||
1580 | Alternatively, you can obtain a QR code with your zone key AND your | ||
1581 | pseudonym from gnunet-namestore-gtk. The QR code is displayed in the | ||
1582 | main window and can be stored to disk using the ``Save as'' button | ||
1583 | next to the image. | ||
1584 | |||
1585 | @node Adding Links to Other Zones | ||
1586 | @subsection Adding Links to Other Zones | ||
1587 | |||
1588 | |||
1589 | A central operation in GNS is the ability to securely delegate to | ||
1590 | other zones. Basically, by adding a delegation you make all of the | ||
1591 | names from the other zone available to yourself. This section | ||
1592 | describes how to create delegations. | ||
1593 | |||
1594 | Suppose you have a friend who you call 'bob' who also uses GNS. | ||
1595 | You can then delegate resolution of names to Bob's zone by adding | ||
1596 | a PKEY record to their local zone: | ||
1597 | |||
1598 | @example | ||
1599 | $ gnunet-namestore -a -n bob --type PKEY -V XXXX -e never -Z myzone | ||
1600 | @end example | ||
1601 | |||
1602 | @noindent | ||
1603 | Note that ``XXXX'' in the command above must be replaced with the hash | ||
1604 | of Bob's public key (the output your friend obtained using the | ||
1605 | @command{gnunet-identity} command from the previous section and told | ||
1606 | you, for example by giving you a business card containing this | ||
1607 | information as a QR code). | ||
1608 | |||
1609 | Assuming Bob has an ``A'' record for their website under the name of | ||
1610 | ``www'' in his zone, you can then access Bob's website under | ||
1611 | ``www.bob.myzone'' --- as well as any (public) GNS record that Bob has | ||
1612 | in their zone by replacing www with the respective name of the | ||
1613 | record in Bob's zone. | ||
1614 | |||
1615 | @c themselves? themself? | ||
1616 | Furthermore, if Bob has themselves a (public) delegation to Carol's | ||
1617 | zone under "carol", you can access Carol's records under | ||
1618 | ``NAME.carol.bob.myzone'' (where ``NAME'' is the name of Carol's | ||
1619 | record you want to access). | ||
1620 | |||
1621 | |||
1622 | @node Using Public Keys as Top Level Domains | ||
1623 | @subsection Using Public Keys as Top Level Domains | ||
1624 | |||
1625 | |||
1626 | GNS also assumes responsibility for any name that uses in a | ||
1627 | well-formed public key for the TLD. Names ending this way are then | ||
1628 | resolved by querying the respective zone. Such public key TLDs are | ||
1629 | expected to be used under rare circumstances where globally unique | ||
1630 | names are required, and for integration with legacy systems. | ||
1631 | |||
1632 | @node Resource Records in GNS | ||
1633 | @subsection Resource Records in GNS | ||
1634 | |||
1635 | |||
1636 | GNS supports the majority of the DNS records as defined in | ||
1637 | @uref{http://www.ietf.org/rfc/rfc1035.txt, RFC 1035}. Additionally, | ||
1638 | GNS defines some new record types the are unique to the GNS system. | ||
1639 | For example, GNS-specific resource records are used to give petnames | ||
1640 | for zone delegation, revoke zone keys and provide some compatibility | ||
1641 | features. | ||
1642 | |||
1643 | For some DNS records, GNS does extended processing to increase their | ||
1644 | usefulness in GNS. In particular, GNS introduces special names | ||
1645 | referred to as "zone relative names". Zone relative names are allowed | ||
1646 | in some resource record types (for example, in NS and CNAME records) | ||
1647 | and can also be used in links on webpages. Zone relative names end | ||
1648 | in ".+" which indicates that the name needs to be resolved relative | ||
1649 | to the current authoritative zone. The extended processing of those | ||
1650 | names will expand the ".+" with the correct delegation chain to the | ||
1651 | authoritative zone (replacing ".+" with the name of the location | ||
1652 | where the name was encountered) and hence generate a | ||
1653 | valid GNS name. | ||
1654 | |||
1655 | The GNS currently supports the record types as defined in | ||
1656 | @uref{https://git.gnunet.org/gana.git/tree/gnu-name-system-record-types/registry.rec, GANA}. | ||
1657 | In addition, GNS supports DNS record types, such as A, AAAA or TXT. | ||
1658 | |||
1659 | For a complete description of the records, please refer to the specification | ||
1660 | at @uref{https://lsd.gnunet.org/lsd0001, LSD0001}. | ||
1661 | |||
1662 | In the following, we discuss GNS records with specific behaviour or special | ||
1663 | handling of DNS records. | ||
1664 | |||
1665 | @menu | ||
1666 | * NICK:: | ||
1667 | * PKEY:: | ||
1668 | * BOX:: | ||
1669 | * LEHO:: | ||
1670 | * VPN:: | ||
1671 | * REDIRECT:: | ||
1672 | * GNS2DNS:: | ||
1673 | * TOMBSTONE:: | ||
1674 | * SOA SRV PTR and MX:: | ||
1675 | @end menu | ||
1676 | |||
1677 | @node NICK | ||
1678 | @subsubsection NICK | ||
1679 | |||
1680 | A NICK record is used to give a zone a name. With a NICK record, you | ||
1681 | can essentially specify how you would like to be called. GNS expects | ||
1682 | this record under the empty label ``@@'' in the zone's database | ||
1683 | (NAMESTORE); however, it will then automatically be copied into each | ||
1684 | record set, so that clients never need to do a separate lookup to | ||
1685 | discover the NICK record. Also, users do not usually have to worry | ||
1686 | about setting the NICK record: it is automatically set to the local | ||
1687 | name of the TLD. | ||
1688 | |||
1689 | @b{Example}@ | ||
1690 | |||
1691 | @example | ||
1692 | Name: @@; RRType: NICK; Value: bob | ||
1693 | @end example | ||
1694 | |||
1695 | @noindent | ||
1696 | This record in Bob's zone will tell other users that this zone wants | ||
1697 | to be referred to as 'bob'. Note that nobody is obliged to call Bob's | ||
1698 | zone 'bob' in their own zones. It can be seen as a | ||
1699 | recommendation ("Please call this zone 'bob'"). | ||
1700 | |||
1701 | @node PKEY | ||
1702 | @subsubsection PKEY | ||
1703 | |||
1704 | PKEY records are used to add delegation to other users' zones and | ||
1705 | give those zones a petname. | ||
1706 | |||
1707 | @b{Example}@ | ||
1708 | |||
1709 | Let Bob's zone be identified by the hash "ABC012". Bob is your friend | ||
1710 | so you want to give them the petname "friend". Then you add the | ||
1711 | following record to your zone: | ||
1712 | |||
1713 | @example | ||
1714 | Name: friend; RRType: PKEY; Value: ABC012; | ||
1715 | @end example | ||
1716 | |||
1717 | @noindent | ||
1718 | This will allow you to resolve records in bob's zone | ||
1719 | under "*.friend.gnu". | ||
1720 | |||
1721 | @node BOX | ||
1722 | @subsubsection BOX | ||
1723 | |||
1724 | BOX records are there to integrate information from TLSA or | ||
1725 | SRV records under the main label. In DNS, TLSA and SRV records | ||
1726 | use special names of the form @code{_port._proto.(label.)*tld} to | ||
1727 | indicate the port number and protocol (like TCP or UDP) for which | ||
1728 | the TLSA or SRV record is valid. This causes various problems, and | ||
1729 | is elegantly solved in GNS by integrating the protocol and port | ||
1730 | numbers together with the respective value into a "BOX" record. | ||
1731 | Note that in the GUI, you do not get to edit BOX records directly | ||
1732 | right now --- the GUI will provide the illusion of directly | ||
1733 | editing the TLSA and SRV records, even though they internally | ||
1734 | are BOXed up. | ||
1735 | |||
1736 | @node LEHO | ||
1737 | @subsubsection LEHO | ||
1738 | |||
1739 | The LEgacy HOstname of a server. Some webservers expect a specific | ||
1740 | hostname to provide a service (virtual hosting). Also SSL | ||
1741 | certificates usually contain DNS names. To provide the expected | ||
1742 | legacy DNS name for a server, the LEHO record can be used. | ||
1743 | To mitigate the just mentioned issues the GNS proxy has to be used. | ||
1744 | The GNS proxy will use the LEHO information to apply the necessary | ||
1745 | transformations. | ||
1746 | |||
1747 | @node VPN | ||
1748 | @subsubsection VPN | ||
1749 | |||
1750 | GNS allows easy access to services provided by the GNUnet Virtual Public | ||
1751 | Network. When the GNS resolver encounters a VPN record it will contact | ||
1752 | the VPN service to try and allocate an IPv4/v6 address (if the queries | ||
1753 | record type is an IP address) that can be used to contact the service. | ||
1754 | |||
1755 | @b{Example}@ | ||
1756 | |||
1757 | I want to provide access to the VPN service "web.gnu." on port 80 on peer | ||
1758 | ABC012:@ | ||
1759 | Name: www; RRType: VPN; Value: 80 ABC012 web.gnu. | ||
1760 | |||
1761 | The peer ABC012 is configured to provide an exit point for the service | ||
1762 | "web.gnu." on port 80 to it's server running locally on port 8080 by | ||
1763 | having the following lines in the @file{gnunet.conf} configuration file: | ||
1764 | |||
1765 | @example | ||
1766 | [web.gnunet.] | ||
1767 | TCP_REDIRECTS = 80:localhost4:8080 | ||
1768 | @end example | ||
1769 | |||
1770 | @node REDIRECT | ||
1771 | @subsubsection REDIRECT | ||
1772 | |||
1773 | As specified in LSD0001 whenever a REDIRECT is encountered the query | ||
1774 | needs to be restarted with the specified name. A REDIRECT | ||
1775 | can either be: | ||
1776 | |||
1777 | @itemize @bullet | ||
1778 | @item A zone relative name, | ||
1779 | @item A zkey name or | ||
1780 | @item A DNS name (in which case resolution will continue outside | ||
1781 | of GNS with the systems DNS resolver) | ||
1782 | @end itemize | ||
1783 | |||
1784 | @node GNS2DNS | ||
1785 | @subsubsection GNS2DNS | ||
1786 | |||
1787 | GNS can delegate authority to a legacy DNS zone. For this, the | ||
1788 | name of the DNS nameserver and the name of the DNS zone are | ||
1789 | specified in a GNS2DNS record. | ||
1790 | |||
1791 | @b{Example} | ||
1792 | |||
1793 | @example | ||
1794 | Name: pet; RRType: GNS2DNS; Value: gnunet.org@@a.ns.joker.com | ||
1795 | @end example | ||
1796 | |||
1797 | @noindent | ||
1798 | Any query to @code{pet.gnu} will then be delegated to the DNS server at | ||
1799 | @code{a.ns.joker.com}. For example, | ||
1800 | @code{@uref{http://www.pet.gnu/, www.pet.gnu}} will result in a DNS query | ||
1801 | for @code{@uref{http://www.gnunet.org/, www.gnunet.org}} to the server | ||
1802 | at @code{a.ns.joker.com}. Delegation to DNS via NS records in GNS can | ||
1803 | be useful if you do not want to start resolution in the DNS root zone | ||
1804 | (due to issues such as censorship or availability). | ||
1805 | |||
1806 | Note that you would typically want to use a relative name for the | ||
1807 | nameserver, like so: | ||
1808 | |||
1809 | @example | ||
1810 | Name: pet; RRType: GNS2DNS; Value: gnunet.org@@ns-joker.+@ | ||
1811 | |||
1812 | Name: ns-joker; RRType: A; Value: 184.172.157.218 | ||
1813 | @end example | ||
1814 | |||
1815 | @noindent | ||
1816 | This way, you can avoid involving the DNS hierarchy in the resolution of | ||
1817 | @code{a.ns.joker.com}. In the example above, the problem may not be | ||
1818 | obvious as the nameserver for "gnunet.org" is in the ".com" zone. | ||
1819 | However, imagine the nameserver was "ns.gnunet.org". In this case, | ||
1820 | delegating to "ns.gnunet.org" would mean that despite using GNS, | ||
1821 | censorship in the DNS ".org" zone would still be effective. | ||
1822 | |||
1823 | @node TOMBSTONE | ||
1824 | @subsubsection TOMBSTONE | ||
1825 | |||
1826 | The GNUnet GNS implementation uses the TOMBSTONE record to ensure | ||
1827 | ciphertext indistinguishability for published records. | ||
1828 | It must be ensured that when relative expiration times are decreased, the | ||
1829 | expiration time of the next record block MUST be after the last published block. | ||
1830 | A similar issue arises if the record set under a label is deleted and reused | ||
1831 | later. | ||
1832 | |||
1833 | The creation and maintenance of the TOMBSTONE record is done automatically. | ||
1834 | You do not need to mind it yourself and can safely ignore any TOMBSTONE | ||
1835 | blocks you may see when investigating your zone(s). | ||
1836 | TOMBSTONE records are always private and will never be published. | ||
1837 | |||
1838 | @node SOA SRV PTR and MX | ||
1839 | @subsubsection SOA SRV PTR and MX | ||
1840 | |||
1841 | The domain names in those records can, again, be either | ||
1842 | |||
1843 | @itemize @bullet | ||
1844 | @item A zone relative name, | ||
1845 | @item A zkey name or | ||
1846 | @item A DNS name | ||
1847 | @end itemize | ||
1848 | |||
1849 | The resolver will expand the zone relative name if possible. | ||
1850 | Note that when using MX records within GNS, the target mail | ||
1851 | server might still refuse to accept e-mails to the resulting | ||
1852 | domain as the name might not match. GNS-enabled mail clients | ||
1853 | should use the ZKEY zone as the destination hostname and | ||
1854 | GNS-enabled mail servers should be configured to accept | ||
1855 | e-mails to the ZKEY-zones of all local users. | ||
1856 | |||
1857 | To add a SOA record via the gnunet-namestore command line | ||
1858 | tool use the following syntax for the value option. Choose | ||
1859 | the other options according to your preference, however in | ||
1860 | this example we will use a relative expiry, add the record | ||
1861 | under the label @ and add the records to the zone bar | ||
1862 | which already exists: | ||
1863 | |||
1864 | @example | ||
1865 | $ gnunet-namestore -a -n @ -t SOA -z bar -e 3600s -V \ | ||
1866 | > "rname=$PRIMARY_NS \ | ||
1867 | > mname=$CONTACT_MAIL \ | ||
1868 | > $SERIAL,$REFRESH,$RETRY,$EXPIRY,$MINIMUM_TTL" | ||
1869 | @end example | ||
1870 | |||
1871 | The above command filled in with values looks like this: | ||
1872 | @example | ||
1873 | $ gnunet-namestore -a -n @ -t SOA -z bar -e 3600s -V \ | ||
1874 | > "rname=ns1.bar \ | ||
1875 | > mname=root.bar \ | ||
1876 | > 2019081701,3600,1800,86400,7200" | ||
1877 | @end example | ||
1878 | |||
1879 | MX records use a similar syntax which is outlined in the | ||
1880 | example below. $SERVER is a domain name as mentioned above. | ||
1881 | @example | ||
1882 | $ gnunet-namestore -a -n mail -t MX -z bar -e 3600s -V \ | ||
1883 | > "$PRIORITY,$SERVER" | ||
1884 | @end example | ||
1885 | |||
1886 | With the values substituted this is an example of a working | ||
1887 | command: | ||
1888 | @example | ||
1889 | $ gnunet-namestore -a -n mail -t MX -z bar -e 3600s -V \ | ||
1890 | > "10,mail.bar" | ||
1891 | @end example | ||
1892 | |||
1893 | @node Synchronizing with legacy DNS | ||
1894 | @subsection Synchronizing with legacy DNS | ||
1895 | |||
1896 | If you want to support GNS but the master database for a zone | ||
1897 | is only available and maintained in DNS, GNUnet includes the | ||
1898 | @command{gnunet-zoneimport} tool to monitor a DNS zone and | ||
1899 | automatically import records into GNS. Today, the tool does | ||
1900 | not yet support DNS AF(X)R, as we initially used it on the | ||
1901 | ``.fr'' zone which does not allow us to perform a DNS zone | ||
1902 | transfer. Instead, @command{gnunet-zoneimport} reads a list | ||
1903 | of DNS domain names from @code{stdin}, issues DNS queries for | ||
1904 | each, converts the obtained records (if possible) and stores | ||
1905 | the result in the namestore. | ||
1906 | |||
1907 | @image{images/gns,6in,, picture of DNS-GNS data flow} | ||
1908 | |||
1909 | The zonemaster service then takes the records from the namestore, | ||
1910 | publishes them into the DHT which makes the result available to the | ||
1911 | GNS resolver. In the GNS configuration, non-local zones can be | ||
1912 | configured to be intercepted by specifying ``.tld = PUBLICKEY'' in the | ||
1913 | configuration file in the ``[gns]'' section. | ||
1914 | |||
1915 | Note that the namestore by default also populates the namecache. | ||
1916 | This pre-population is cryptographically expensive. Thus, on | ||
1917 | systems that only serve to import a large (millions of records) | ||
1918 | DNS zone and that do not have a local gns service in use, it | ||
1919 | is thus advisable to disable the namecache by setting the | ||
1920 | option ``DISABLE'' to ``YES'' in section ``[namecache]''. | ||
1921 | |||
1922 | @node Migrating an existing DNS zone into GNS | ||
1923 | @subsection Migrating an existing DNS zone into GNS | ||
1924 | |||
1925 | Ascension is a tool to migrate existing DNS zones into GNS. | ||
1926 | |||
1927 | @xref{Migrating existing DNS zones into GNS}, for installation instructions and | ||
1928 | further information about Ascension. | ||
1929 | |||
1930 | Compared to the gnunet-zoneimport tool it strictly uses AXFR or IXFR depending | ||
1931 | on whether or not there exists a SOA record for the zone. If that is the case it | ||
1932 | will take the serial as a reference point and request the zone. The server will | ||
1933 | either answer the IXFR request with a correct incremental zone or with the | ||
1934 | entire zone, which depends on the server configuration. | ||
1935 | |||
1936 | After installing the tool according to the README file you have the following | ||
1937 | options: | ||
1938 | |||
1939 | @example | ||
1940 | Ascension | ||
1941 | Usage: | ||
1942 | ascension <domain> [-d] [-p] [-s] [--minimum-ttl=<ttl>] \ | ||
1943 | [--dry-run] | ||
1944 | ascension <domain> <port> [-d] [-p] [-s] \ | ||
1945 | [--minimum-ttl=<ttl>] [--dry-run] | ||
1946 | ascension <domain> -n <transferns> [-d] [-p] \ | ||
1947 | [-s] [--minimum-ttl=<ttl>] [--dry-run] | ||
1948 | ascension <domain> -n <transferns> <port> [-d] \ | ||
1949 | [-p] [-s] [--minimum-ttl=<ttl>] [--dry-run] | ||
1950 | ascension -p | --public | ||
1951 | ascension -d | --debug | ||
1952 | ascension -s | --standalone | ||
1953 | ascension -h | --help | ||
1954 | ascension -v | --version | ||
1955 | |||
1956 | Options: | ||
1957 | <domain> Domain to migrate | ||
1958 | <port> Port for zone transfer | ||
1959 | <transferns> DNS Server that does the zone transfer | ||
1960 | --minimum-ttl=<ttl> Minimum TTL for records to migrate \ | ||
1961 | [default: 3600] | ||
1962 | --dry-run Only try if a zone transfer is allowed | ||
1963 | -p --public Make records public on the DHT | ||
1964 | -s --standalone Run ascension once | ||
1965 | -d --debug Enable debugging | ||
1966 | -h --help Show this screen. | ||
1967 | -v --version Show version. | ||
1968 | @end example | ||
1969 | |||
1970 | Before you can migrate any zone though, you need to start a local GNUnet peer: | ||
1971 | @example | ||
1972 | $ gnunet-arm -s | ||
1973 | @end example | ||
1974 | |||
1975 | To migrate the Syrian top level domain - one of the few top level domains that | ||
1976 | support zone transfers - into GNS use the following command: | ||
1977 | |||
1978 | @example | ||
1979 | $ ascension sy. -n ns1.tld.sy. -p | ||
1980 | @end example | ||
1981 | |||
1982 | The -p flag will tell GNS to put these records on the DHT so that other users | ||
1983 | may resolve these records by using the public key of the zone. | ||
1984 | |||
1985 | Once the zone is migrated, Ascension will output a message telling you, that it | ||
1986 | will refresh the zone after the time has elapsed. You can resolve the names in | ||
1987 | the zone directly using GNS or if you want to use it with your browser, check | ||
1988 | out the GNS manual section. @ref{Configuring the GNU Name System}. To resolve | ||
1989 | the records from another system you need the respective zones PKEY. To get the | ||
1990 | zones public key, you can run the following command: | ||
1991 | |||
1992 | @example | ||
1993 | $ gnunet-identity -dqe sy | ||
1994 | @end example | ||
1995 | |||
1996 | Where "sy" is the name of the zone you want to migrate. | ||
1997 | |||
1998 | You can share the PKEY of the zone with your friends. They can then resolve | ||
1999 | records in the zone by doing a lookup replacing the zone label with your PKEY: | ||
2000 | |||
2001 | @example | ||
2002 | $ gnunet-gns -t SOA -u "$PKEY" | ||
2003 | @end example | ||
2004 | |||
2005 | The program will continue to run as a daemon and update once the refresh time | ||
2006 | specified in the zones SOA record has elapsed. | ||
2007 | |||
2008 | DNSCurve style records are supported in the latest release and they are added | ||
2009 | as a PKEY record to be referred to the respective GNS public key. Key | ||
2010 | distribution is still a problem but provided someone else has a public key | ||
2011 | under a given label it can be looked up. | ||
2012 | |||
2013 | There is an unofficial Debian package called python3-ascension that adds a | ||
2014 | system user ascension and runs a GNUnet peer in the background. | ||
2015 | |||
2016 | Ascension-bind is also an unofficial Debian package that on installation checks | ||
2017 | for running DNS zones and whether or not they are transferable using DNS zone | ||
2018 | transfer (AXFR). It asks the administrator which zones to migrate into GNS and | ||
2019 | installs a systemd unit file to keep the zone up to date. If you want to | ||
2020 | migrate different zones you might want to check the unit file from the package | ||
2021 | as a guide. | ||
2022 | |||
2023 | @node reclaimID Identity Provider | ||
2024 | @section reclaimID Identity Provider | ||
2025 | |||
2026 | The re:claimID Identity Provider (IdP) is a decentralized IdP service. | ||
2027 | It allows its users to manage and authorize third parties to access | ||
2028 | their identity attributes such as email or shipping addresses. | ||
2029 | |||
2030 | It basically mimics the concepts of centralized IdPs, such as those | ||
2031 | offered by Google or Facebook. | ||
2032 | Like other IdPs, reclaimID features an (optional) OpenID Connect | ||
2033 | 1.0-compliant protocol layer that can be used for websites to | ||
2034 | integrate reclaimID as an Identity Provider with little effort. | ||
2035 | |||
2036 | @menu | ||
2037 | * Managing Attributes:: | ||
2038 | * Managing Credentials:: | ||
2039 | * Sharing Attributes with Third Parties:: | ||
2040 | * Revoking Authorizations of Third Parties:: | ||
2041 | * OpenID Connect:: | ||
2042 | * Providing Third Party Attestation:: | ||
2043 | @end menu | ||
2044 | |||
2045 | @node Managing Attributes | ||
2046 | @subsection Managing Attributes | ||
2047 | |||
2048 | Before adding attributes to an identity, you must first create an ego: | ||
2049 | |||
2050 | @example | ||
2051 | $ gnunet-identity --create="user" | ||
2052 | @end example | ||
2053 | |||
2054 | Henceforth, you can manage a new user profile of the user ``user''. | ||
2055 | |||
2056 | To add an email address to your user profile, simply use the @command{gnunet-reclaim} command line tool:: | ||
2057 | |||
2058 | @example | ||
2059 | $ gnunet-reclaim -e "user" -a "email" -V "username@@example.gnunet" | ||
2060 | @end example | ||
2061 | |||
2062 | All of your attributes can be listed using the @command{gnunet-reclaim} | ||
2063 | command line tool as well: | ||
2064 | |||
2065 | @example | ||
2066 | $ gnunet-reclaim -e "user" -D | ||
2067 | @end example | ||
2068 | |||
2069 | Currently, and by default, attribute values are interpreted as plain text. | ||
2070 | In the future there might be more value types such as X.509 certificate credentials. | ||
2071 | |||
2072 | @node Managing Credentials | ||
2073 | @subsection Managing Credentials | ||
2074 | |||
2075 | Attribute values may reference a claim in a third party attested credential. | ||
2076 | Such a credential can have a variety of formats such as JSON-Web-Tokens or | ||
2077 | X.509 certificates. | ||
2078 | Currently, reclaimID only supports JSON-Web-Token credentials. | ||
2079 | |||
2080 | To add a credential to your user profile, invoke the @command{gnunet-reclaim} command line tool as follows: | ||
2081 | |||
2082 | @example | ||
2083 | $ gnunet-reclaim -e "user"\ | ||
2084 | --credential-name="email"\ | ||
2085 | --credential-type="JWT"\ | ||
2086 | --value="ey..." | ||
2087 | @end example | ||
2088 | |||
2089 | All of your credentials can be listed using the @command{gnunet-reclaim} | ||
2090 | command line tool as well: | ||
2091 | |||
2092 | @example | ||
2093 | $ gnunet-reclaim -e "user" --credentials | ||
2094 | @end example | ||
2095 | |||
2096 | In order to add an attribe backed by a credential, specify the attribute | ||
2097 | value as the claim name in the credential to reference along with the credential | ||
2098 | ID: | ||
2099 | |||
2100 | @example | ||
2101 | $ gnunet-reclaim -e "user"\ | ||
2102 | --add="email"\ | ||
2103 | --value="verified_email"\ | ||
2104 | --credential-id="<CREDENTIAL_ID>" | ||
2105 | @end example | ||
2106 | |||
2107 | |||
2108 | @node Sharing Attributes with Third Parties | ||
2109 | @subsection Sharing Attributes with Third Parties | ||
2110 | |||
2111 | If you want to allow a third party such as a website or friend to access to your attributes (or a subset thereof) execute: | ||
2112 | |||
2113 | @example | ||
2114 | $ TICKET=$(gnunet-reclaim -e "user"\ | ||
2115 | -r "$RP_KEY"\ | ||
2116 | -i "attribute1,attribute2,...") | ||
2117 | @end example | ||
2118 | |||
2119 | The command will return a "ticket" string. | ||
2120 | You must give $TICKET to the requesting third party. | ||
2121 | |||
2122 | $RP_KEY is the public key of the third party and "attribute1,attribute2,..." is a comma-separated list of attribute names, such as "email,name,...", that you want to share. | ||
2123 | |||
2124 | The third party may retrieve the key in string format for use in the above | ||
2125 | call using "gnunet-identity": | ||
2126 | |||
2127 | @example | ||
2128 | $ RP_KEY=$(gnunet-identity -d | grep "relyingparty" | awk '@{print $3@}') | ||
2129 | @end example | ||
2130 | |||
2131 | The third party can then retrieve your shared identity attributes using: | ||
2132 | |||
2133 | @example | ||
2134 | $ gnunet-reclaim -e "relyingparty" -C "ticket" | ||
2135 | @end example | ||
2136 | |||
2137 | Where "relyingparty" is the name for the identity behind $RP_KEY that the | ||
2138 | requesting party is using. | ||
2139 | This will retrieve and list the shared identity attributes. | ||
2140 | The above command will also work if the user is currently offline since the attributes are retrieved from GNS. | ||
2141 | Further, $TICKET can be re-used later to retrieve up-to-date attributes in case "friend" has changed the value(s). For instance, because his email address changed. | ||
2142 | |||
2143 | To list all given authorizations (tickets) you can execute: | ||
2144 | @example | ||
2145 | $ gnunet-reclaim -e "user" -T | ||
2146 | @end example | ||
2147 | |||
2148 | @node Revoking Authorizations of Third Parties | ||
2149 | @subsection Revoking Authorizations of Third Parties | ||
2150 | |||
2151 | If you want to revoke the access of a third party to your attributes you can execute: | ||
2152 | |||
2153 | @example | ||
2154 | $ gnunet-reclaim -e "user" -R $TICKET | ||
2155 | @end example | ||
2156 | |||
2157 | This will prevent the third party from accessing the attribute in the future. | ||
2158 | Please note that if the third party has previously accessed the attribute, there is not way in which the system could have prevented the thiry party from storing the data. | ||
2159 | As such, only access to updated data in the future can be revoked. | ||
2160 | This behaviour is _exactly the same_ as with other IdPs. | ||
2161 | |||
2162 | @node OpenID Connect | ||
2163 | @subsection OpenID Connect | ||
2164 | |||
2165 | There is an @uref{OpenID Connect, https://openid.net/specs/openid-connect-core-1_0.html} API for use with re:claimID. | ||
2166 | However, its use is quite complicated to setup. | ||
2167 | |||
2168 | @example | ||
2169 | https://api.reclaim/openid/authorize | ||
2170 | http://localhost:7776/openid/token | ||
2171 | http://localhost:7776/openid/userinfo | ||
2172 | http://localhost:7776/openid/login | ||
2173 | @end example | ||
2174 | |||
2175 | The token endpoint is protected using HTTP basic authentication. | ||
2176 | You can authenticate using any username and the password configured under: | ||
2177 | |||
2178 | @example | ||
2179 | $ gnunet-config -s reclaim-rest-plugin -o OIDC_CLIENT_SECRET | ||
2180 | @end example | ||
2181 | |||
2182 | The authorize endpoint is protected using a Cookie which can be obtained through | ||
2183 | a request against the login endpoint. | ||
2184 | This functionality is meant to be used in the context of the OpenID Connect authorization | ||
2185 | flow to collect user consent interactively. | ||
2186 | Without a Cookie, the authorize endpoint redirects to a URI configured under: | ||
2187 | |||
2188 | @example | ||
2189 | $ gnunet-config -s reclaim-rest-plugin -o ADDRESS | ||
2190 | @end example | ||
2191 | |||
2192 | The token endpoint is protected using OAuth2 and expects the grant | ||
2193 | which is retrieved from the authorization endpoint according to the standard. | ||
2194 | |||
2195 | The userinfo endpoint is protected using OAuth2 and expects a bearer access | ||
2196 | token which is retrieved from a token request. | ||
2197 | |||
2198 | In order to make use of OpenID Connect flows as a user, you need to install | ||
2199 | the browser plugin: | ||
2200 | |||
2201 | @itemize @bullet | ||
2202 | @item @uref{https://addons.mozilla.org/addon/reclaimid/, Firefox Add-on} | ||
2203 | @item @uref{https://chrome.google.com/webstore/detail/reclaimid/jiogompmdejcnacmlnjhnaicgkefcfll, Chrome Web Store} | ||
2204 | @end itemize | ||
2205 | |||
2206 | In order to create and register an OpenID Connect client as a relying party, | ||
2207 | you need to execute the following steps: | ||
2208 | |||
2209 | @example | ||
2210 | $ gnunet-identity -C <client_name> | ||
2211 | $ gnunet-namestore -z <client_name> -a -n "@@" -t RECLAIM_OIDC_REDIRECT -V <redirect_uri> -e 1d -p | ||
2212 | $ gnunet-namestore -z <client_name> -a -n "@@" -t RECLAIM_OIDC_CLIENT -V "My OIDC Client" -e 1d -p | ||
2213 | @end example | ||
2214 | |||
2215 | The "client_id" for use in OpenID Connect is the public key of the client as | ||
2216 | displayed using: | ||
2217 | @example | ||
2218 | $ gnunet-identity -d grep "relyingparty" | awk '@{print $3@}' | ||
2219 | @end example | ||
2220 | |||
2221 | The RECLAIM_OIDC_REDIRECT record contains your website redirect URI. | ||
2222 | You may use any globally unique DNS or GNS URI. | ||
2223 | The RECLAIM_OIDC_CLIENT record represents the client description which whill | ||
2224 | be displayed to users in an authorization request. | ||
2225 | |||
2226 | Any website or relying party must use the authorization endpoint | ||
2227 | @uref{https://api.reclaim/openid/authorize} in its authorization redirects, e.g. | ||
2228 | |||
2229 | @example | ||
2230 | <a href="https://api.reclaim/openid/authorize?client_id=<PKEY>\ | ||
2231 | &scope=openid email\ | ||
2232 | &redirect_uri=<redirect_uri>\ | ||
2233 | &nonce=<random>">Login</a> | ||
2234 | @end example | ||
2235 | |||
2236 | This will direct the user's browser onto his local reclaimID instance. | ||
2237 | After giving consent, you will be provided with the OpenID Connect authorization | ||
2238 | code according to the specifications at your provided redirect URI. | ||
2239 | |||
2240 | The ID Tokens issues by the token endpoints are signed using HS512 with the | ||
2241 | shared secret configured under: | ||
2242 | |||
2243 | @example | ||
2244 | $ gnunet-config -s reclaim-rest-plugin -o JWT_SECRET | ||
2245 | @end example | ||
2246 | |||
2247 | The authorization code flow optionally supports @uref{https://tools.ietf.org/html/rfc7636, Proof Key for Code Exchange}. | ||
2248 | If PKCE is used, the client does not need to authenticate against the token | ||
2249 | endpoint. | ||
2250 | |||
2251 | @node Providing Third Party Attestation | ||
2252 | @subsection Providing Third Party Attestation | ||
2253 | |||
2254 | If you are running an identity provider (IdP) service you may be able to | ||
2255 | support providing credentials for re:claimID users. | ||
2256 | IdPs can issue JWT credentials as long as they support OpenID Connect and | ||
2257 | @uref{https://openid.net/specs/openid-connect-discovery-1_0.html,OpenID Connect Discovery}. | ||
2258 | |||
2259 | In order to allow users to import attributes through the re:claimID user interface, | ||
2260 | you need to register the following public OAuth2/OIDC client: | ||
2261 | |||
2262 | @itemize @bullet | ||
2263 | @item client_id: reclaimid | ||
2264 | @item client_secret: none | ||
2265 | @item redirect_uri: https://ui.reclaim (The URI of the re:claimID webextension) | ||
2266 | @item grant_type: authorization_code with PKCE (@uref{https://tools.ietf.org/html/rfc7636, RFC7636}) | ||
2267 | @item scopes: all you want to offer. | ||
2268 | @item id_token: JWT | ||
2269 | @end itemize | ||
2270 | |||
2271 | When your users add an attribute with name "email" which supports webfinger | ||
2272 | discovery they will be prompted with the option to retrieve the OpenID Connect | ||
2273 | ID Token through the user interface. | ||
2274 | |||
2275 | @node Using the Virtual Public Network | ||
2276 | @section Using the Virtual Public Network | ||
2277 | |||
2278 | @menu | ||
2279 | * Setting up an Exit node:: | ||
2280 | * Fedora and the Firewall:: | ||
2281 | * Setting up VPN node for protocol translation and tunneling:: | ||
2282 | @end menu | ||
2283 | |||
2284 | Using the GNUnet Virtual Public Network (VPN) application you can | ||
2285 | tunnel IP traffic over GNUnet. Moreover, the VPN comes | ||
2286 | with built-in protocol translation and DNS-ALG support, enabling | ||
2287 | IPv4-to-IPv6 protocol translation (in both directions). | ||
2288 | This chapter documents how to use the GNUnet VPN. | ||
2289 | |||
2290 | The first thing to note about the GNUnet VPN is that it is a public | ||
2291 | network. All participating peers can participate and there is no | ||
2292 | secret key to control access. So unlike common virtual private | ||
2293 | networks, the GNUnet VPN is not useful as a means to provide a | ||
2294 | "private" network abstraction over the Internet. The GNUnet VPN | ||
2295 | is a virtual network in the sense that it is an overlay over the | ||
2296 | Internet, using its own routing mechanisms and can also use an | ||
2297 | internal addressing scheme. The GNUnet VPN is an Internet | ||
2298 | underlay --- TCP/IP applications run on top of it. | ||
2299 | |||
2300 | The VPN is currently only supported on GNU/Linux systems. | ||
2301 | Support for operating systems that support TUN (such as FreeBSD) | ||
2302 | should be easy to add (or might not even require any coding at | ||
2303 | all --- we just did not test this so far). Support for other | ||
2304 | operating systems would require re-writing the code to create virtual | ||
2305 | network interfaces and to intercept DNS requests. | ||
2306 | |||
2307 | The VPN does not provide good anonymity. While requests are routed | ||
2308 | over the GNUnet network, other peers can directly see the source | ||
2309 | and destination of each (encapsulated) IP packet. Finally, if you | ||
2310 | use the VPN to access Internet services, the peer sending the | ||
2311 | request to the Internet will be able to observe and even alter | ||
2312 | the IP traffic. We will discuss additional security implications | ||
2313 | of using the VPN later in this chapter. | ||
2314 | |||
2315 | @node Setting up an Exit node | ||
2316 | @subsection Setting up an Exit node | ||
2317 | |||
2318 | Any useful operation with the VPN requires the existence of an exit | ||
2319 | node in the GNUnet Peer-to-Peer network. Exit functionality can only | ||
2320 | be enabled on peers that have regular Internet access. If you want | ||
2321 | to play around with the VPN or support the network, we encourage | ||
2322 | you to setup exit nodes. This chapter documents how to setup an | ||
2323 | exit node. | ||
2324 | |||
2325 | There are four types of exit functions an exit node can provide, | ||
2326 | and using the GNUnet VPN to access the Internet will only work | ||
2327 | nicely if the first three types are provided somewhere in | ||
2328 | the network. The four exit functions are: | ||
2329 | |||
2330 | @itemize @bullet | ||
2331 | @item DNS: allow other peers to use your DNS resolver | ||
2332 | @item IPv4: allow other peers to access your IPv4 Internet connection | ||
2333 | @item IPv6: allow other peers to access your IPv6 Internet connection | ||
2334 | @item Local service: allow other peers to access a specific TCP or | ||
2335 | UDP service your peer is providing | ||
2336 | @end itemize | ||
2337 | |||
2338 | By enabling "exit" in gnunet-setup and checking the respective boxes | ||
2339 | in the "exit" tab, you can easily choose which of the above exit | ||
2340 | functions you want to support. | ||
2341 | |||
2342 | Note, however, that by supporting the first three functions you will | ||
2343 | allow arbitrary other GNUnet users to access the Internet via your | ||
2344 | system. This is somewhat similar to running a Tor exit node. The | ||
2345 | Torproject has a nice article about what to consider if you want | ||
2346 | to do this here. We believe that generally running a DNS exit node | ||
2347 | is completely harmless. | ||
2348 | |||
2349 | The exit node configuration does currently not allow you to restrict the | ||
2350 | Internet traffic that leaves your system. In particular, you cannot | ||
2351 | exclude SMTP traffic (or block port 25) or limit to HTTP traffic using | ||
2352 | the GNUnet configuration. However, you can use your host firewall to | ||
2353 | restrict outbound connections from the virtual tunnel interface. This | ||
2354 | is highly recommended. In the future, we plan to offer a wider range | ||
2355 | of configuration options for exit nodes. | ||
2356 | |||
2357 | Note that by running an exit node GNUnet will configure your kernel | ||
2358 | to perform IP-forwarding (for IPv6) and NAT (for IPv4) so that the | ||
2359 | traffic from the virtual interface can be routed to the Internet. | ||
2360 | In order to provide an IPv6-exit, you need to have a subnet routed | ||
2361 | to your host's external network interface and assign a subrange of | ||
2362 | that subnet to the GNUnet exit's TUN interface. | ||
2363 | |||
2364 | When running a local service, you should make sure that the local | ||
2365 | service is (also) bound to the IP address of your EXIT interface | ||
2366 | (e.g. 169.254.86.1). It will NOT work if your local service is | ||
2367 | just bound to loopback. You may also want to create a "VPN" record | ||
2368 | in your zone of the GNU Name System to make it easy for others to | ||
2369 | access your service via a name instead of just the full service | ||
2370 | descriptor. Note that the identifier you assign the service can | ||
2371 | serve as a passphrase or shared secret, clients connecting to the | ||
2372 | service must somehow learn the service's name. VPN records in the | ||
2373 | GNU Name System can make this easier. | ||
2374 | |||
2375 | @node Fedora and the Firewall | ||
2376 | @subsection Fedora and the Firewall | ||
2377 | |||
2378 | |||
2379 | When using an exit node on Fedora 15, the standard firewall can | ||
2380 | create trouble even when not really exiting the local system! | ||
2381 | For IPv4, the standard rules seem fine. However, for IPv6 the | ||
2382 | standard rules prohibit traffic from the network range of the | ||
2383 | virtual interface created by the exit daemon to the local IPv6 | ||
2384 | address of the same interface (which is essentially loopback | ||
2385 | traffic, so you might suspect that a standard firewall would | ||
2386 | leave this traffic alone). However, as somehow for IPv6 the | ||
2387 | traffic is not recognized as originating from the local | ||
2388 | system (and as the connection is not already "established"), | ||
2389 | the firewall drops the traffic. You should still get ICMPv6 | ||
2390 | packets back, but that's obviously not very useful. | ||
2391 | |||
2392 | Possible ways to fix this include disabling the firewall (do you | ||
2393 | have a good reason for having it on?) or disabling the firewall | ||
2394 | at least for the GNUnet exit interface (or the respective | ||
2395 | IPv4/IPv6 address range). The best way to diagnose these kinds | ||
2396 | of problems in general involves setting the firewall to REJECT | ||
2397 | instead of DROP and to watch the traffic using wireshark | ||
2398 | (or tcpdump) to see if ICMP messages are generated when running | ||
2399 | some tests that should work. | ||
2400 | |||
2401 | @node Setting up VPN node for protocol translation and tunneling | ||
2402 | @subsection Setting up VPN node for protocol translation and tunneling | ||
2403 | |||
2404 | |||
2405 | The GNUnet VPN/PT subsystem enables you to tunnel IP traffic over the | ||
2406 | VPN to an exit node, from where it can then be forwarded to the | ||
2407 | Internet. This section documents how to setup VPN/PT on a node. | ||
2408 | Note that you can enable both the VPN and an exit on the same peer. | ||
2409 | In this case, IP traffic from your system may enter your peer's VPN | ||
2410 | and leave your peer's exit. This can be useful as a means to do | ||
2411 | protocol translation. For example, you might have an application that | ||
2412 | supports only IPv4 but needs to access an IPv6-only site. In this case, | ||
2413 | GNUnet would perform 4to6 protocol translation between the VPN (IPv4) | ||
2414 | and the Exit (IPv6). Similarly, 6to4 protocol translation is also | ||
2415 | possible. However, the primary use for GNUnet would be to access | ||
2416 | an Internet service running with an IP version that is not supported | ||
2417 | by your ISP. In this case, your IP traffic would be routed via GNUnet | ||
2418 | to a peer that has access to the Internet with the desired IP version. | ||
2419 | |||
2420 | Setting up an entry node into the GNUnet VPN primarily requires you | ||
2421 | to enable the "VPN/PT" option in "gnunet-setup". This will launch the | ||
2422 | "gnunet-service-vpn", "gnunet-service-dns" and "gnunet-daemon-pt" | ||
2423 | processes. The "gnunet-service-vpn" will create a virtual interface | ||
2424 | which will be used as the target for your IP traffic that enters the | ||
2425 | VPN. Additionally, a second virtual interface will be created by | ||
2426 | the "gnunet-service-dns" for your DNS traffic. You will then need to | ||
2427 | specify which traffic you want to tunnel over GNUnet. If your ISP only | ||
2428 | provides you with IPv4 or IPv6-access, you may choose to tunnel the | ||
2429 | other IP protocol over the GNUnet VPN. If you do not have an ISP | ||
2430 | (and are connected to other GNUnet peers via WLAN), you can also | ||
2431 | choose to tunnel all IP traffic over GNUnet. This might also provide | ||
2432 | you with some anonymity. After you enable the respective options | ||
2433 | and restart your peer, your Internet traffic should be tunneled | ||
2434 | over the GNUnet VPN. | ||
2435 | |||
2436 | The GNUnet VPN uses DNS-ALG to hijack your IP traffic. Whenever an | ||
2437 | application resolves a hostname (like 'gnunet.org'), the | ||
2438 | "gnunet-daemon-pt" will instruct the "gnunet-service-dns" to intercept | ||
2439 | the request (possibly route it over GNUnet as well) and replace the | ||
2440 | normal answer with an IP in the range of the VPN's interface. | ||
2441 | "gnunet-daemon-pt" will then tell "gnunet-service-vpn" to forward all | ||
2442 | traffic it receives on the TUN interface via the VPN to the original | ||
2443 | destination. | ||
2444 | |||
2445 | For applications that do not use DNS, you can also manually create | ||
2446 | such a mapping using the gnunet-vpn command-line tool. Here, you | ||
2447 | specify the desired address family of the result (e.g. "-4"), and the | ||
2448 | intended target IP on the Internet (e.g. "-i 131.159.74.67") and | ||
2449 | "gnunet-vpn" will tell you which IP address in the range of your | ||
2450 | VPN tunnel was mapped. | ||
2451 | |||
2452 | @command{gnunet-vpn} can also be used to access "internal" services | ||
2453 | offered by GNUnet nodes. So if you happen to know a peer and a | ||
2454 | service offered by that peer, you can create an IP tunnel to | ||
2455 | that peer by specifying the peer's identity, service name and | ||
2456 | protocol (--tcp or --udp) and you will again receive an IP address | ||
2457 | that will terminate at the respective peer's service. | ||
2458 | |||
2459 | @node Using the GNUnet Messenger | ||
2460 | @section Using the GNUnet Messenger | ||
2461 | |||
2462 | The GNUnet Messenger subsystem allows decentralized message-based | ||
2463 | communication inside of so called rooms. Each room can be relayed by | ||
2464 | a variable amount of peers. Every member of a room has the possibility | ||
2465 | to relay the room on its own peer. A peer allows any amount of members | ||
2466 | to join a room. The amount of members in a room is not restricted. | ||
2467 | |||
2468 | Messages in a room will be distributed between all peers relaying the | ||
2469 | room or being internally (in context of the messenger service) connected | ||
2470 | to a relaying peer. All received or sent messages will be stored on any | ||
2471 | peer locally which is relaying the respective room or is internally | ||
2472 | connected to such a relaying peer. | ||
2473 | |||
2474 | The Messenger service is built on the CADET subsystem to make internal | ||
2475 | connections between peers using a reliable and encrypted transmission. | ||
2476 | Additionally the service uses a discrete padding to few different sizes. | ||
2477 | So kinds of messages and potential content can't be identified by the | ||
2478 | size of traffic from any attacker being unable to break the encryption | ||
2479 | of the transmission layer. | ||
2480 | |||
2481 | Another feature is additional end-to-end encryption for selected messages | ||
2482 | which uses the public key of another member (the receiver) to encrypt | ||
2483 | the message. Therefore it is ensured that only the selected member can | ||
2484 | read its content. This will also use additional padding. | ||
2485 | |||
2486 | @menu | ||
2487 | * Current state:: | ||
2488 | * Entering a room:: | ||
2489 | * Opening a room:: | ||
2490 | * Messaging in a room:: | ||
2491 | * Private messaging:: | ||
2492 | @end menu | ||
2493 | |||
2494 | @node Current state | ||
2495 | @subsection Current state | ||
2496 | |||
2497 | Currently there is only a simplistic CLI application available to use the | ||
2498 | messenger service. You can use this application with the | ||
2499 | @command{gnunet-messenger} command. | ||
2500 | |||
2501 | This application was designed for testing purposes and it does not provide | ||
2502 | full functionality in the current state. It is planned to replace this CLI | ||
2503 | application in later stages with a fully featured one using a client-side | ||
2504 | library designed for messenger applications. | ||
2505 | |||
2506 | @node Entering a room | ||
2507 | @subsection Entering a room | ||
2508 | |||
2509 | You can enter any room by its ROOMKEY and any PEERIDENTITY of a relaying | ||
2510 | peer. Optionally you can provide any IDENTITY which can represent a local | ||
2511 | ego by its name. | ||
2512 | |||
2513 | @example | ||
2514 | $ gnunet-messenger [-e IDENTITY] -d PEERIDENTITY -r ROOMKEY | ||
2515 | @end example | ||
2516 | |||
2517 | A PEERIDENTITY gets entered in encoded form. You can get your own peer ID by | ||
2518 | using the @command{gnunet-peerinfo} command: | ||
2519 | |||
2520 | @example | ||
2521 | $ gnunet-peerinfo -s | ||
2522 | @end example | ||
2523 | |||
2524 | A ROOMKEY gets entered in readable text form. The service will then hash the | ||
2525 | entered ROOMKEY and use the result as shared secret for transmission through | ||
2526 | the CADET submodule. You can also optionally leave out the '-r' parameter and | ||
2527 | the ROOMKEY to use the zeroed hash instead. | ||
2528 | |||
2529 | If no IDENTITY is provided you will not send any name to others, you will be | ||
2530 | referred as "anonymous" instead and use the anonymous ego. If you provide any | ||
2531 | IDENTITY a matching ego will be used to sign your messages. If there is no | ||
2532 | matching ego you will use the anonymous ego instead. The provided IDENTITY will | ||
2533 | be distributed as your name for the service in any case. | ||
2534 | |||
2535 | @node Opening a room | ||
2536 | @subsection Opening a room | ||
2537 | |||
2538 | You can open any room in a similar way to entering it. You just have to leave | ||
2539 | out the '-d' parameter and the PEERIDENTITY of the relaying peer. | ||
2540 | |||
2541 | @example | ||
2542 | $ gnunet-messenger [-e IDENTITY] -r ROOMKEY | ||
2543 | @end example | ||
2544 | |||
2545 | Providing ROOMKEY and IDENTITY is identical to entering a room. Opening a room | ||
2546 | will also make your peer to a relay of this room. So others can enter the room | ||
2547 | through your peer if they have the required ROOMKEY and your peer ID. | ||
2548 | |||
2549 | If you want to use the zeroed hash as shared secret key for the room you can | ||
2550 | also leave it out as well: | ||
2551 | |||
2552 | @example | ||
2553 | $ gnunet-messenger | ||
2554 | @end example | ||
2555 | |||
2556 | @node Messaging in a room | ||
2557 | @subsection Messaging in a room | ||
2558 | |||
2559 | Once joined a room by entering it or opening it you can write text-based | ||
2560 | messages which will be distributed between all internally conntected peers. All | ||
2561 | sent messages will be displayed in the same way as received messages. | ||
2562 | |||
2563 | This relates to the internal handling of sent and received messages being mostly | ||
2564 | identical on application layer. Every handled message will be represented | ||
2565 | visually depending on its kind, content and sender. A sender can usually be | ||
2566 | identified by the encoded member ID or their name. | ||
2567 | |||
2568 | @example | ||
2569 | [17X37K] * 'anonymous' says: "hey" | ||
2570 | @end example | ||
2571 | |||
2572 | @node Private messaging | ||
2573 | @subsection Private messaging | ||
2574 | |||
2575 | As referred in the introduction the service allows sending private messages with | ||
2576 | additional end-to-end encryption. These messages will be visually represented | ||
2577 | by messages of the kind 'PRIVATE' in case they can't be decrypted with your used | ||
2578 | ego. Members who can't decrypt the message can potentially only identify its | ||
2579 | sender but they can't identify its receiver. | ||
2580 | |||
2581 | @example | ||
2582 | [17X37K] ~ message: PRIVATE | ||
2583 | @end example | ||
2584 | |||
2585 | If they can be decrypted they will appear as their secret message instead | ||
2586 | but marked visually. | ||
2587 | |||
2588 | @example | ||
2589 | [17X37K] ** 'anonymous' says: "hey" | ||
2590 | @end example | ||
2591 | |||
2592 | Currently you can only activate sending such encrypted text messages instead of | ||
2593 | usual text messages by adding the '-p' parameter: | ||
2594 | |||
2595 | @example | ||
2596 | $ gnunet-messenger [-e IDENTITY] -d PEERIDENTITY -r ROOMKEY -p | ||
2597 | @end example | ||
2598 | |||
2599 | Notice that you can only send such encrypted messages to members who use an ego | ||
2600 | which is not publicly known as the anonymous ego to ensure transparency. If | ||
2601 | any user could decrypt these messages they would not be private. So as receiver | ||
2602 | of such messages the IDENTITY is required and it has to match a local ego. | ||
2603 | |||
diff --git a/doc/handbook/chapters/vocabulary.texi b/doc/handbook/chapters/vocabulary.texi deleted file mode 100644 index 0ee472b95..000000000 --- a/doc/handbook/chapters/vocabulary.texi +++ /dev/null | |||
@@ -1,72 +0,0 @@ | |||
1 | @node Vocabulary | ||
2 | @chapter Vocabulary | ||
3 | |||
4 | @menu | ||
5 | * Definitions abbreviations and acronyms:: | ||
6 | * Words and characters:: | ||
7 | * Technical Assumptions:: | ||
8 | @end menu | ||
9 | |||
10 | Throughout this Reference Manual we will use certain words and characters | ||
11 | which are listed in this introductionary chapter. | ||
12 | |||
13 | @node Definitions abbreviations and acronyms | ||
14 | @section Definitions abbreviations and acronyms | ||
15 | |||
16 | @menu | ||
17 | * Definitions:: | ||
18 | @end menu | ||
19 | |||
20 | @node Definitions | ||
21 | @subsection Definitions | ||
22 | |||
23 | Throughout this Reference Manual, the following terms and definitions | ||
24 | apply. | ||
25 | |||
26 | @node Words and characters | ||
27 | @section Words and characters | ||
28 | |||
29 | @enumerate | ||
30 | @item | ||
31 | In chapter Installation Handbook, | ||
32 | ``@command{#}'' in example code blocks describes commands executed as root | ||
33 | |||
34 | @example | ||
35 | # echo "I am root" | ||
36 | I am root | ||
37 | @end example | ||
38 | |||
39 | @item | ||
40 | However, in the chapter GNUnet C Tutorial | ||
41 | ``@command{#}'' in example code blocks describes commands, ie comments. | ||
42 | |||
43 | @example | ||
44 | # Do the foobar thing: | ||
45 | $ make foobar | ||
46 | @end example | ||
47 | |||
48 | @item | ||
49 | Dollarsign ``@command{$}'' in example code blocks describes commands you | ||
50 | execute as unprivileged users. | ||
51 | |||
52 | @example | ||
53 | $ cd foo; ./configure --example-switch | ||
54 | @end example | ||
55 | |||
56 | @item | ||
57 | Backslash ``@command{\}'' describes linebreaks. | ||
58 | |||
59 | @example | ||
60 | ./configure --foo --bar --baz \ | ||
61 | --short-loop | ||
62 | @end example | ||
63 | |||
64 | ...expands to @code{./configure --foo --bar --baz --short-loop} | ||
65 | |||
66 | @end enumerate | ||
67 | |||
68 | @node Technical Assumptions | ||
69 | @section Technical Assumptions | ||
70 | |||
71 | @c Is it really assuming Bash (ie Bash extensions of POSIX being used)? | ||
72 | The shell on GNU systems is assumed to be Bash. | ||