aboutsummaryrefslogtreecommitdiff
path: root/doc/documentation/chapters/developer.texi
diff options
context:
space:
mode:
Diffstat (limited to 'doc/documentation/chapters/developer.texi')
-rw-r--r--doc/documentation/chapters/developer.texi384
1 files changed, 382 insertions, 2 deletions
diff --git a/doc/documentation/chapters/developer.texi b/doc/documentation/chapters/developer.texi
index 6a895ed11..d33072057 100644
--- a/doc/documentation/chapters/developer.texi
+++ b/doc/documentation/chapters/developer.texi
@@ -2,7 +2,7 @@
2@node GNUnet Developer Handbook 2@node GNUnet Developer Handbook
3@chapter GNUnet Developer Handbook 3@chapter GNUnet Developer Handbook
4 4
5This book is intended to be an introduction for programmers that want to 50This book is intended to be an introduction for programmers that want to
6extend the GNUnet framework. GNUnet is more than a simple peer-to-peer 6extend the GNUnet framework. GNUnet is more than a simple peer-to-peer
7application. 7application.
8 8
@@ -11,7 +11,7 @@ For developers, GNUnet is:
11@itemize @bullet 11@itemize @bullet
12@item developed by a community that believes in the GNU philosophy 12@item developed by a community that believes in the GNU philosophy
13@item Free Software (Free as in Freedom), licensed under the 13@item Free Software (Free as in Freedom), licensed under the
14GNU General Public License@footnote{@uref{https://www.gnu.org/licenses/licenses.html#GPL, https://www.gnu.org/licenses/licenses.html#GPL}} 14GNU Affero General Public License@footnote{@uref{https://www.gnu.org/licenses/licenses.html#AGPL, https://www.gnu.org/licenses/licenses.html#AGPL}}
15@item A set of standards, including coding conventions and 15@item A set of standards, including coding conventions and
16architectural rules 16architectural rules
17@item A set of layered protocols, both specifying the communication 17@item A set of layered protocols, both specifying the communication
@@ -40,6 +40,7 @@ new chapters, sections or insightful comments.
40 40
41@menu 41@menu
42* Developer Introduction:: 42* Developer Introduction::
43* Internal dependencies::
43* Code overview:: 44* Code overview::
44* System Architecture:: 45* System Architecture::
45* Subsystem stability:: 46* Subsystem stability::
@@ -47,6 +48,7 @@ new chapters, sections or insightful comments.
47* Build-system:: 48* Build-system::
48* Developing extensions for GNUnet using the gnunet-ext template:: 49* Developing extensions for GNUnet using the gnunet-ext template::
49* Writing testcases:: 50* Writing testcases::
51* Building GNUnet and its dependencies::
50* TESTING library:: 52* TESTING library::
51* Performance regression analysis with Gauger:: 53* Performance regression analysis with Gauger::
52* TESTBED Subsystem:: 54* TESTBED Subsystem::
@@ -251,6 +253,77 @@ those that have a public website) which build on top of the GNUnet
251framework. 253framework.
252 254
253@c *********************************************************************** 255@c ***********************************************************************
256@node Internal dependencies
257@section Internal dependencies
258
259This section tries to give an overview of what processes a typical GNUnet
260peer running a particular application would consist of. All of the
261processes listed here should be automatically started by
262@command{gnunet-arm -s}.
263The list is given as a rough first guide to users for failure diagnostics.
264Ideally, end-users should never have to worry about these internal
265dependencies.
266
267In terms of internal dependencies, a minimum file-sharing system consists
268of the following GNUnet processes (in order of dependency):
269
270@itemize @bullet
271@item gnunet-service-arm
272@item gnunet-service-resolver (required by all)
273@item gnunet-service-statistics (required by all)
274@item gnunet-service-peerinfo
275@item gnunet-service-transport (requires peerinfo)
276@item gnunet-service-core (requires transport)
277@item gnunet-daemon-hostlist (requires core)
278@item gnunet-daemon-topology (requires hostlist, peerinfo)
279@item gnunet-service-datastore
280@item gnunet-service-dht (requires core)
281@item gnunet-service-identity
282@item gnunet-service-fs (requires identity, mesh, dht, datastore, core)
283@end itemize
284
285@noindent
286A minimum VPN system consists of the following GNUnet processes (in
287order of dependency):
288
289@itemize @bullet
290@item gnunet-service-arm
291@item gnunet-service-resolver (required by all)
292@item gnunet-service-statistics (required by all)
293@item gnunet-service-peerinfo
294@item gnunet-service-transport (requires peerinfo)
295@item gnunet-service-core (requires transport)
296@item gnunet-daemon-hostlist (requires core)
297@item gnunet-service-dht (requires core)
298@item gnunet-service-mesh (requires dht, core)
299@item gnunet-service-dns (requires dht)
300@item gnunet-service-regex (requires dht)
301@item gnunet-service-vpn (requires regex, dns, mesh, dht)
302@end itemize
303
304@noindent
305A minimum GNS system consists of the following GNUnet processes (in
306order of dependency):
307
308@itemize @bullet
309@item gnunet-service-arm
310@item gnunet-service-resolver (required by all)
311@item gnunet-service-statistics (required by all)
312@item gnunet-service-peerinfo
313@item gnunet-service-transport (requires peerinfo)
314@item gnunet-service-core (requires transport)
315@item gnunet-daemon-hostlist (requires core)
316@item gnunet-service-dht (requires core)
317@item gnunet-service-mesh (requires dht, core)
318@item gnunet-service-dns (requires dht)
319@item gnunet-service-regex (requires dht)
320@item gnunet-service-vpn (requires regex, dns, mesh, dht)
321@item gnunet-service-identity
322@item gnunet-service-namestore (requires identity)
323@item gnunet-service-gns (requires vpn, dns, dht, namestore, identity)
324@end itemize
325
326@c ***********************************************************************
254@node Code overview 327@node Code overview
255@section Code overview 328@section Code overview
256 329
@@ -1089,6 +1162,313 @@ typically necessary to run @code{make install} before running any
1089testcases. Thus the canonical command @code{make check install} has to be 1162testcases. Thus the canonical command @code{make check install} has to be
1090changed to @code{make install check} for GNUnet. 1163changed to @code{make install check} for GNUnet.
1091 1164
1165@c ***********************************************************************
1166@cindex Building GNUnet
1167@node Building GNUnet and its dependencies
1168@section Building GNUnet and its dependencies
1169
1170In the following section we will outline how to build GNUnet and
1171some of its dependencies. We will assume a fair amount of knowledge
1172for building applications under UNIX-like systems. Furthermore we
1173assume that the build environment is sane and that you are aware of
1174any implications actions in this process could have.
1175Instructions here can be seen as notes for developers (an extension to
1176the 'HACKING' section in README) as well as package mantainers.
1177@b{Users should rely on the available binary packages.}
1178We will use Debian as an example Operating System environment. Substitute
1179accordingly with your own Ooperating System environment.
1180
1181For the full list of depenendencies, consult the appropriate, up-to-date
1182section in the @file{README} file.
1183
1184First, we need to build or install (depending on your OS) the following
1185packages. If you build them from source, build them in this exact order:
1186
1187@example
1188libgpgerror, libgcrypt, libnettle, libunbound, GnuTLS (with libunbound
1189support)
1190@end example
1191
1192After we have build and installed those packages, we continue with
1193packages closer to GNUnet in this step: libgnurl (our libcurl fork),
1194GNU libmicrohttpd, and GNU libextractor. Again, if your package manager
1195provides one of these packages, use the packages provided from it
1196unless you have good reasons (package version too old, conflicts, etc).
1197We advise against compiling widely used packages such as GnuTLS
1198yourself if your OS provides a variant already unless you take care
1199of maintenance of the packages then.
1200
1201In the optimistic case, this command will give you all the dependencies:
1202
1203@example
1204sudo apt-get install libgnurl libmicrohttpd libextractor
1205@end example
1206
1207From experience we know that at the very least libgnurl is not
1208available in some environments. You could substitute libgnurl
1209with libcurl, but we recommend to install libgnurl, as it gives
1210you a predefined libcurl with the small set GNUnet requires. In
1211the past namespaces of libcurl and libgnurl were shared, which
1212caused problems when you wanted to integrate both of them in one
1213Operating System. This has been resolved, and they can be installed
1214side by side now.
1215
1216@cindex libgnurl
1217@cindex compiling libgnurl
1218GNUnet and some of its function depend on a limited subset of cURL/libcurl.
1219Rather than trying to enforce a certain configuration on the world, we
1220opted to maintain a microfork of it that ensures we can link against the
1221right set of features. We called this specialized set of libcurl
1222``libgnurl''. It is fully ABI compatible with libcurl and currently used
1223by GNUnet and some of its dependencies.
1224
1225We download libgnurl and its digital signature from the GNU fileserver,
1226assuming @env{TMPDIR} exists@footnote{It might be @file{/tmp}, @env{TMPDIR}, @env{TMP} or any other location. For consistency we assume @env{TMPDIR} points to @file{/tmp} for the remainder of this section.}
1227
1228@example
1229cd \$TMPDIR
1230wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.60.0.tar.Z
1231wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.60.0.tar.Z.sig
1232@end example
1233
1234Next, verify the digital signature of the file:
1235
1236@example
1237gpg --verify gnurl-7.60.0.tar.Z.sig
1238@end example
1239
1240If gpg fails, you might try with @command{gpg2} on your OS. If the error
1241states that ``the key can not be found'' or it is unknown, you have to
1242retrieve the key (A88C8ADD129828D7EAC02E52E22F9BBFEE348588) from a
1243keyserver first:
1244
1245@example
1246gpg --keyserver pgp.mit.edu --recv-keys A88C8ADD129828D7EAC02E52E22F9BBFEE348588
1247@end example
1248
1249and rerun the verification command.
1250
1251libgnurl will require the following packages to be present at runtime:
1252gnutls (with DANE support / libunbound), libidn, zlib and at compile time:
1253libtool, groff, perl, pkg-config, and python 2.7.
1254
1255Once you have verified that all the required packages are present on your
1256system, we can proceed to compile libgnurl:
1257
1258@example
1259tar -xvf gnurl-7.60.0.tar.Z
1260cd gnurl-7.60.0
1261sh configure --disable-ntlm-wb
1262make
1263make -C tests test
1264sudo make install
1265@end example
1266
1267After you've compiled and installed libgnurl, we can proceed to building
1268GNUnet.
1269
1270
1271
1272
1273First, in addition to the GNUnet sources you might require downloading the
1274latest version of various dependencies, depending on how recent the
1275software versions in your distribution of GNU/Linux are.
1276Most distributions do not include sufficiently recent versions of these
1277dependencies.
1278Thus, a typically installation on a "modern" GNU/Linux distribution
1279requires you to install the following dependencies (ideally in this
1280order):
1281
1282@itemize @bullet
1283@item libgpgerror and libgcrypt
1284@item libnettle and libunbound (possibly from distribution), GnuTLS
1285@item libgnurl (read the README)
1286@item GNU libmicrohttpd
1287@item GNU libextractor
1288@end itemize
1289
1290Make sure to first install the various mandatory and optional
1291dependencies including development headers from your distribution.
1292
1293Other dependencies that you should strongly consider to install is a
1294database (MySQL, sqlite or Postgres).
1295The following instructions will assume that you installed at least sqlite.
1296For most distributions you should be able to find pre-build packages for
1297the database. Again, make sure to install the client libraries @b{and} the
1298respective development headers (if they are packaged separately) as well.
1299
1300You can find specific, detailed instructions for installing of the
1301dependencies (and possibly the rest of the GNUnet installation) in the
1302platform-specific descriptions, which can be found in the Index.
1303Please consult them now.
1304If your distribution is not listed, please study the build
1305instructions for Debian stable, carefully as you try to install the
1306dependencies for your own distribution.
1307Contributing additional instructions for further platforms is always
1308appreciated.
1309Please take in mind that operating system development tends to move at
1310a rather fast speed. Due to this you should be aware that some of
1311the instructions could be outdated by the time you are reading this.
1312If you find a mistake, please tell us about it (or even better: send
1313a patch to the documentation to fix it!).
1314
1315Before proceeding further, please double-check the dependency list.
1316Note that in addition to satisfying the dependencies, you might have to
1317make sure that development headers for the various libraries are also
1318installed.
1319There maybe files for other distributions, or you might be able to find
1320equivalent packages for your distribution.
1321
1322While it is possible to build and install GNUnet without having root
1323access, we will assume that you have full control over your system in
1324these instructions.
1325First, you should create a system user @emph{gnunet} and an additional
1326group @emph{gnunetdns}. On the GNU/Linux distributions Debian and Ubuntu,
1327type:
1328
1329@example
1330sudo adduser --system --home /var/lib/gnunet --group \
1331--disabled-password gnunet
1332sudo addgroup --system gnunetdns
1333@end example
1334
1335@noindent
1336On other Unixes and GNU systems, this should have the same effect:
1337
1338@example
1339sudo useradd --system --groups gnunet --home-dir /var/lib/gnunet
1340sudo addgroup --system gnunetdns
1341@end example
1342
1343Now compile and install GNUnet using:
1344
1345@example
1346tar xvf gnunet-@value{VERSION}.tar.gz
1347cd gnunet-@value{VERSION}
1348./configure --with-sudo=sudo --with-nssdir=/lib
1349make
1350sudo make install
1351@end example
1352
1353If you want to be able to enable DEBUG-level log messages, add
1354@code{--enable-logging=verbose} to the end of the
1355@command{./configure} command.
1356@code{DEBUG}-level log messages are in English only and
1357should only be useful for developers (or for filing
1358really detailed bug reports).
1359
1360@noindent
1361Next, edit the file @file{/etc/gnunet.conf} to contain the following:
1362
1363@example
1364[arm]
1365SYSTEM_ONLY = YES
1366USER_ONLY = NO
1367@end example
1368
1369@noindent
1370You may need to update your @code{ld.so} cache to include
1371files installed in @file{/usr/local/lib}:
1372
1373@example
1374# ldconfig
1375@end example
1376
1377@noindent
1378Then, switch from user @code{root} to user @code{gnunet} to start
1379the peer:
1380
1381@example
1382# su -s /bin/sh - gnunet
1383$ gnunet-arm -c /etc/gnunet.conf -s
1384@end example
1385
1386You may also want to add the last line in the gnunet user's @file{crontab}
1387prefixed with @code{@@reboot} so that it is executed whenever the system
1388is booted:
1389
1390@example
1391@@reboot /usr/local/bin/gnunet-arm -c /etc/gnunet.conf -s
1392@end example
1393
1394@noindent
1395This will only start the system-wide GNUnet services.
1396Type @command{exit} to get back your root shell.
1397Now, you need to configure the per-user part. For each
1398user that should get access to GNUnet on the system, run
1399(replace alice with your username):
1400
1401@example
1402sudo adduser alice gnunet
1403@end example
1404
1405@noindent
1406to allow them to access the system-wide GNUnet services. Then, each
1407user should create a configuration file @file{~/.config/gnunet.conf}
1408with the lines:
1409
1410@example
1411[arm]
1412SYSTEM_ONLY = NO
1413USER_ONLY = YES
1414DEFAULTSERVICES = gns
1415@end example
1416
1417@noindent
1418and start the per-user services using
1419
1420@example
1421$ gnunet-arm -c ~/.config/gnunet.conf -s
1422@end example
1423
1424@noindent
1425Again, adding a @code{crontab} entry to autostart the peer is advised:
1426
1427@example
1428@@reboot /usr/local/bin/gnunet-arm -c $HOME/.config/gnunet.conf -s
1429@end example
1430
1431@noindent
1432Note that some GNUnet services (such as SOCKS5 proxies) may need a
1433system-wide TCP port for each user.
1434For those services, systems with more than one user may require each user
1435to specify a different port number in their personal configuration file.
1436
1437Finally, the user should perform the basic initial setup for the GNU Name
1438System (GNS) certificate authority. This is done by running:
1439
1440@example
1441$ gnunet-gns-proxy-setup-ca
1442@end example
1443
1444@noindent
1445The first generates the default zones, wheras the second setups the GNS
1446Certificate Authority with the user's browser. Now, to activate GNS in the
1447normal DNS resolution process, you need to edit your
1448@file{/etc/nsswitch.conf} where you should find a line like this:
1449
1450@example
1451hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
1452@end example
1453
1454@noindent
1455The exact details may differ a bit, which is fine. Add the text
1456@emph{"gns [NOTFOUND=return]"} after @emph{"files"}.
1457Keep in mind that we included a backslash ("\") here just for
1458markup reasons. You should write the text below on @b{one line}
1459and @b{without} the "\":
1460
1461@example
1462hosts: files gns [NOTFOUND=return] mdns4_minimal \
1463[NOTFOUND=return] dns mdns4
1464@end example
1465
1466@c FIXME: Document new behavior.
1467You might want to make sure that @file{/lib/libnss_gns.so.2} exists on
1468your system, it should have been created during the installation.
1469
1470
1471@c **********************************************************************
1092@cindex TESTING library 1472@cindex TESTING library
1093@node TESTING library 1473@node TESTING library
1094@section TESTING library 1474@section TESTING library