From 7101471b5fc9ad10a0a0c06fb2aaeb5a568dbf56 Mon Sep 17 00:00:00 2001 From: Christian Grothoff Date: Thu, 25 Oct 2018 15:46:45 +0200 Subject: doc/documentation split into doc/tutorial and doc/handbook for clarity and to avoid automake freakout --- doc/tutorial/gnunet-tutorial.texi | 1568 +++++++++++++++++++++++++++++++++++++ 1 file changed, 1568 insertions(+) create mode 100644 doc/tutorial/gnunet-tutorial.texi (limited to 'doc/tutorial/gnunet-tutorial.texi') diff --git a/doc/tutorial/gnunet-tutorial.texi b/doc/tutorial/gnunet-tutorial.texi new file mode 100644 index 000000000..4f685668d --- /dev/null +++ b/doc/tutorial/gnunet-tutorial.texi @@ -0,0 +1,1568 @@ +\input texinfo +@c %**start of header +@setfilename gnunet-tutorial.info +@documentencoding UTF-8 +@settitle GNUnet Tutorial +@c @exampleindent 2 +@c %**end of header + +@include version.texi + +@copying +Copyright @copyright{} 2001-2018 GNUnet e.V. + +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with no +Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A +copy of the license is included in the section entitled ``GNU Free +Documentation License''. + +A copy of the license is also available from the Free Software +Foundation Web site at @url{http://www.gnu.org/licenses/fdl.html}. + +Alternately, this document is also available under the General +Public License, version 3 or later, as published by the Free Software +Foundation. A copy of the license is included in the section entitled +``GNU General Public License''. + +A copy of the license is also available from the Free Software +Foundation Web site at @url{http://www.gnu.org/licenses/gpl.html}. +@end copying + +@dircategory Tutorial +@direntry +* GNUnet-C-Tutorial: (gnunet-c-tutorial). C Tutorial for GNunet +@end direntry + + +@titlepage +@title GNUnet C Tutorial +@subtitle A Tutorial for GNUnet @value{VERSION} (C version) +@author The GNUnet Developers + +@page +@vskip 0pt plus 1filll + +@insertcopying +@end titlepage + +@contents + +@c **** TODO +@c 1. Update content? +@c 2. Either reference main documentation or +@c 3. Merge this into main documentation + +@node Top +@top Introduction + +This tutorials explains how to install GNUnet on a +GNU/Linux system and gives an introduction on how +GNUnet can be used to develop a Peer-to-Peer application. +Detailed installation instructions for +various operating systems and a detailed list of all +dependencies can be found on our website at +@uref{https://gnunet.org/installation} and in our +Reference Documentation (GNUnet Handbook). + +Please read this tutorial carefully since every single step is +important, and do not hesitate to contact the GNUnet team if you have +any questions or problems! Visit this link in your webbrowser to learn +how to contact the GNUnet team: +@uref{https://gnunet.org/contact_information} + +@menu + +* Installing GNUnet:: Installing GNUnet +* Introduction to GNUnet Architecture:: Introduction to GNUnet Architecture +* First Steps with GNUnet:: First Steps with GNUnet +* Developing Applications:: Developing Applications +* GNU Free Documentation License:: The license of this manual + +@detailmenu + --- The Detailed Node Listing --- + +Installing GNUnet + +* Obtaining a stable version:: +* Installing Build Tool Chain and Dependencies:: +* Obtaining the latest version from Git:: +* Compiling and Installing GNUnet:: +* Common Issues - Check your GNUnet installation:: + +Introduction to GNUnet Architecture + +First Steps with GNUnet + +* Configure your peer:: +* Start a peer:: +* Monitor a peer:: +* Starting Two Peers by Hand:: +* Starting Peers Using the Testbed Service:: + +Developing Applications + +* gnunet-ext:: +* Adapting the Template:: +* Writing a Client Application:: +* Writing a Service:: +* Interacting directly with other Peers using the CORE Service:: +* Storing peer-specific data using the PEERSTORE service:: +* Using the DHT:: +* Debugging with gnunet-arm:: + +@end detailmenu +@end menu + +@node Installing GNUnet +@chapter Installing GNUnet + +First of all you have to install a current version of GNUnet. +You can download a tarball of a stable version from GNU FTP mirrors +or obtain the latest development version from our Git repository. + +Most of the time you should prefer to download the stable version +since with the latest development version things can be broken, +functionality can be changed or tests can fail. You should only use +the development version if you know that you require a certain +feature or a certain issue has been fixed since the last release. + +@menu +* Obtaining a stable version:: +* Installing Build Tool Chain and Dependencies:: +* Obtaining the latest version from Git:: +* Compiling and Installing GNUnet:: +* Common Issues - Check your GNUnet installation:: +@end menu + +@node Obtaining a stable version +@section Obtaining a stable version + +Download the tarball from +@indicateurl{https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz}. + +Make sure to download the associated @file{.sig} file and to verify the +authenticity of the tarball against it, like this: + +@example +$ wget https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz.sig +$ gpg --verify-files gnunet-@value{VERSION}.tar.gz.sig +@end example + +@noindent +If this command fails because you do not have the required public key, +then you need to run the following command to import it: + +@example +$ gpg --keyserver keys.gnupg.net --recv-keys 48426C7E +@end example + +@noindent +and rerun the @code{gpg --verify-files} command. + +@b{Note:}@ +@b{The pub key to sign the 0.10.1 release has been +revoked}. You will get an error message stating that +@b{there is no known public key or that it has been revoked}. +The next release of GNUnet will have a valid signature +again. We are sorry for the inconvenience this causes. +Another possible source you could use is our +"gnunet" git repository which, since the change from SVN to git in 2016, +has mandatory signed commits by every developer. + +After verifying the signature you can extract the tarball. +The resulting directory will be renamed to @file{gnunet}, which we will +be using in the remainder of this document to refer to the +root of the source directory. + +@example +$ tar xvzf gnunet-@value{VERSION}.tar.gz +$ mv gnunet-@value{VERSION} gnunet +@end example + +@c FIXME: This can be irritating for the reader - First we say git should +@c be avoid unless it is really required, and then we write this +@c paragraph: +@noindent +However, please note that stable versions can be very outdated. +As a developer you are @b{strongly} encouraged to use the version +from @uref{https://gnunet.org/git/, git}. + +@node Installing Build Tool Chain and Dependencies +@section Installing Build Tool Chain and Dependencies + +To successfully compile GNUnet, you need the tools to build GNUnet and +the required dependencies. Please take a look at the +GNUnet Reference Documentation +(@pxref{Dependencies, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) +for a list of required dependencies and +(@pxref{Generic installation instructions, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) +read its Installation chapter for specific instructions for +your Operating System. +Please check the notes at the end of the configure process about +required dependencies. + +For GNUnet bootstrapping support and the HTTP(S) plugin you should +install @uref{https://gnunet.org/gnurl, libgnurl}. +For the filesharing service you should install at least one of the +datastore backends (MySQL, SQlite and PostgreSQL are supported). + +@node Obtaining the latest version from Git +@section Obtaining the latest version from Git + +The latest development version can be obtained from our Git repository. +To get the code you need to have @code{Git} installed. Usually your +Operating System package manager should provide a suitable distribution +of git (otherwise check out Guix or Nix). If you are using an Operating +System based on Debian's apt: + +@example +$ sudo apt-get install git +@end example + +This is required for obtaining the repository, which is achieved with +the following command: + +@example +$ git clone https://gnunet.org/git/gnunet +@end example + +@noindent +After cloning the repository, you have to execute the @file{bootstrap} +script in the new directory: + +@example +$ cd gnunet +$ ./bootstrap +@end example + +@noindent +The remainder of this tutorial will assume that you have the +Git branch ``master'' checked out. + +@node Compiling and Installing GNUnet +@section Compiling and Installing GNUnet + +Note: This section is a duplication of the more in depth +@pxref{GNUnet Installation Handbook, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}. + +First, you need to install libgnupgerror @geq{} 1.27 and +libgcrypt @geq{} 1.7.6: + +@example +$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" +$ wget $GNUPGFTP/libgpg-error/libgpg-error-1.27.tar.bz2 +$ tar xf libgpg-error-1.27.tar.bz2 +$ cd libgpg-error-1.27 +$ ./configure +$ make +$ sudo make install +$ cd .. +@end example + +@example +$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" +$ wget $GNUPGFTP/libgcrypt/libgcrypt-1.7.6.tar.bz2 +$ tar xf libgcrypt-1.7.6.tar.bz2 +$ cd libgcrypt-1.7.6 +$ ./configure +$ make +$ sudo make install +$ cd .. +@end example + +@menu +* Installation:: +@end menu + +@node Installation +@subsection Installation +Assuming all dependencies are installed, the following commands will +compile and install GNUnet in your home directory. You can specify the +directory where GNUnet will be installed by changing the +@code{--prefix} value when calling @command{./configure}. If +you do not specifiy a prefix, GNUnet is installed in the directory +@file{/usr/local}. When developing new applications you may want +to enable verbose logging by adding @code{--enable-logging=verbose}: + +@example +$ export PREFIX=$HOME +$ ./configure --prefix=$PREFIX --enable-logging +$ make +$ make install +@end example + +@noindent +After installing GNUnet you have to add your GNUnet installation +to your path environmental variable. In addition you have to +create the @file{.config} directory in your home directory +(unless it already exists) where GNUnet stores its data and an +empty GNUnet configuration file: + +@example +$ export PATH=$PATH:$PREFIX/bin +$ echo export PATH=$PREFIX/bin:\\$PATH >> ~/.bashrc +$ mkdir ~/.config/ +$ touch ~/.config/gnunet.conf +@end example + +@node Common Issues - Check your GNUnet installation +@section Common Issues - Check your GNUnet installation + +You should check your installation to ensure that installing GNUnet +was successful up to this point. You should be able to access GNUnet's +binaries and run GNUnet's self check. + +@example +$ which gnunet-arm +$PREFIX/bin/gnunet-arm +@end example + +@noindent +should return $PREFIX/bin/gnunet-arm (where $PREFIX is the location +you have set earlier). It should be located in your +GNUnet installation and the output should not be empty. + +If you see an output like: + +@example +$ which gnunet-arm +@end example + +@noindent +check your PATH variable to ensure GNUnet's @file{bin} directory is +included. + +GNUnet provides tests for all of its subcomponents. Assuming you have +successfully built GNUnet, run + +@example +$ cd gnunet +$ make check +@end example + +@noindent +to execute tests for all components. @command{make check} traverses all +subdirectories in @file{src}. For every subdirectory you should +get a message like this: + +@example +make[2]: Entering directory `/home/$USER/gnunet/contrib' +PASS: test_gnunet_prefix +============= +1 test passed +============= +@end example + +@node Introduction to GNUnet Architecture +@chapter Introduction to GNUnet Architecture + +GNUnet is organized in layers and services. Each service is composed of a +main service implementation and a client library for other programs to use +the service's functionality, described by an API. +@c This approach is shown in +@c FIXME: enable this once the commented block below works: +@c figure~\ref fig:service. +Some services provide an additional command line tool to enable the user +to interact with the service. + +Very often it is other GNUnet services that will use these APIs to build +the higher layers of GNUnet on top of the lower ones. Each layer expands +or extends the functionality of the service below (for instance, to build +a mesh on top of a DHT). +@c FXIME: See comment above. +@c See figure ~\ref fig:interaction for an illustration of this approach. + +@c ** @image filename[, width[, height[, alttext[, extension]]]] +@c FIXME: Texlive (?) 20112 makes the assumption that this means +@c 'images/OBJECTNAME.txt' but later versions of it (2017) use this +@c syntax as described below. +@c TODO: Checkout the makedoc script Guile uses. + +@c FIXME!!! +@c @image{images/gnunet-tutorial-service,,5in,Service with API and network protocol,.png} +@c @image{images/gnunet-tutorial-system,,5in,The layered system architecture of GNUnet,.png} + +@c \begin{figure}[!h] +@c \begin{center} +@c % \begin{subfigure} +@c \begin{subfigure}[b]{0.3\textwidth} +@c \centering +@c \includegraphics[width=\textwidth]{figs/Service.pdf} +@c \caption{Service with API and network protocol} +@c \label{fig:service} +@c \end{subfigure} +@c ~~~~~~~~~~ +@c \begin{subfigure}[b]{0.3\textwidth} +@c \centering +@c \includegraphics[width=\textwidth]{figs/System.pdf} +@c \caption{Service interaction} +@c \label{fig:interaction} +@c \end{subfigure} +@c \end{center} +@c \caption{GNUnet's layered system architecture} +@c \end{figure} + +The main service implementation runs as a standalone process in the +Operating System and the client code runs as part of the client program, +so crashes of a client do not affect the service process or other clients. +The service and the clients communicate via a message protocol to be +defined and implemented by the programmer. + +@node First Steps with GNUnet +@chapter First Steps with GNUnet + +@menu +* Configure your peer:: +* Start a peer:: +* Monitor a peer:: +* Starting Two Peers by Hand:: +* Starting Peers Using the Testbed Service:: +@end menu + +@node Configure your peer +@section Configure your peer + +First of all we need to configure your peer. Each peer is started with +a configuration containing settings for GNUnet itself and its services. +This configuration is based on the default configuration shipped with +GNUnet and can be modified. The default configuration is located in the +@file{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you +can specify a customized configuration using the the @command{-c} command +line switch when starting the ARM service and all other services. When +using a modified configuration the default values are loaded and only +values specified in the configuration file will replace the default +values. + +Since we want to start additional peers later, we need some modifications +from the default configuration. We need to create a separate service +home and a file containing our modifications for this peer: + +@example +$ mkdir ~/gnunet1/ +$ touch peer1.conf +@end example + +@noindent +Now add the following lines to @file{peer1.conf} to use this directory. +For simplified usage we want to prevent the peer to connect to the GNUnet +network since this could lead to confusing output. This modifications +will replace the default settings: + +@example +[PATHS] +# Use this directory to store GNUnet data +GNUNET_HOME = ~/gnunet1/ +[hostlist] +# prevent bootstrapping +SERVERS = +@end example + +@node Start a peer +@section Start a peer +Each GNUnet instance (called peer) has an identity (peer ID) based on a +cryptographic public private key pair. The peer ID is the printable hash +of the public key. + +GNUnet services are controlled by a master service, the so called +@dfn{Automatic Restart Manager} (ARM). ARM starts, stops and even +restarts services automatically or on demand when a client connects. +You interact with the ARM service using the @command{gnunet-arm} tool. +GNUnet can then be started with @command{gnunet-arm -s} and stopped with +@command{gnunet-arm -e}. An additional service not automatically started +can be started using @command{gnunet-arm -i } and stopped +using @command{gnunet-arm -k }. + +Once you have started your peer, you can use many other GNUnet commands +to interact with it. For example, you can run: + +@example +$ gnunet-peerinfo -s +@end example + +@noindent +to obtain the public key of your peer. + +You should see an output containing the peer ID similar to: + +@example +I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'. +@end example + +@node Monitor a peer +@section Monitor a peer + +In this section, we will monitor the behaviour of our peer's DHT +service with respect to a specific key. First we will start +GNUnet and then start the DHT service and use the DHT monitor tool +to monitor the PUT and GET commands we issue ussing the +@command{gnunet-dht-put} and @command{gnunet-dht-get} commands. +Using the ``monitor'' line given below, you can observe the behavior +of your own peer's DHT with respect to the specified KEY: + +@example +# start gnunet with all default services: +$ gnunet-arm -c ~/peer1.conf -s +# start DHT service: +$ gnunet-arm -c ~/peer1.conf -i dht +$ cd ~/gnunet/src/dht; +$ ./gnunet-dht-monitor -c ~/peer1.conf -k KEY +@end example + +@noindent +Now open a separate terminal and change again to +the @file{gnunet/src/dht} directory: + +@example +$ cd ~/gnunet/src/dht +# put VALUE under KEY in the DHT: +$ ./gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE +# get key KEY from the DHT: +$ ./gnunet/src/dht/gnunet-dht-get -c ~/peer1.conf -k KEY +# print statistics about current GNUnet state: +$ gnunet-statistics -c ~/peer1.conf +# print statistics about DHT service: +$ gnunet-statistics -c ~/peer1.conf -s dht +@end example + +@node Starting Two Peers by Hand +@section Starting Two Peers by Hand + +This section describes how to start two peers on the same machine by hand. +The process is rather painful, but the description is somewhat +instructive. In practice, you might prefer the automated method +(@pxref{Starting Peers Using the Testbed Service}). + +@menu +* Setup a second peer:: +* Start the second peer and connect the peers:: +* How to connect manually:: +@end menu + +@node Setup a second peer +@subsection Setup a second peer +We will now start a second peer on your machine. +For the second peer, you will need to manually create a modified +configuration file to avoid conflicts with ports and directories. +A peers configuration file is by default located +in @file{~/.gnunet/gnunet.conf}. This file is typically very short +or even empty as only the differences to the defaults need to be +specified. The defaults are located in many files in the +@file{$PREFIX/share/gnunet/config.d} directory. + +To configure the second peer, use the files +@file{$PREFIX/share/gnunet/config.d} as a template for your main +configuration file: + +@example +$ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf +@end example + +@noindent +Now you have to edit @file{peer2.conf} and change: + +@itemize +@item @code{GNUNET\_TEST\_HOME} under @code{PATHS} +@item Every (uncommented) value for ``@code{PORT}'' (add 10000) in any +section (the option may be commented out if @code{PORT} is +prefixed by "\#", in this case, UNIX domain sockets are used +and the PORT option does not need to be touched) +@item Every value for ``@code{UNIXPATH}'' in any section +(e.g. by adding a "-p2" suffix) +@end itemize + +to a fresh, unique value. Make sure that the PORT numbers stay +below 65536. From now on, whenever you interact with the second peer, +you need to specify @command{-c peer2.conf} as an additional +command line argument. + +Now, generate the 2nd peer's private key: + +@example +$ gnunet-peerinfo -s -c peer2.conf +@end example + +@noindent +This may take a while, generate entropy using your keyboard or mouse +as needed. Also, make sure the output is different from the +gnunet-peerinfo output for the first peer (otherwise you made an +error in the configuration). + +@node Start the second peer and connect the peers +@subsection Start the second peer and connect the peers + +Then, you can start a second peer using: + +@example +$ gnunet-arm -c peer2.conf -s +$ gnunet-arm -c peer2.conf -i dht +$ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE +$ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY +@end example + +If you want the two peers to connect, you have multiple options: + +@itemize +@item UDP neighbour discovery (automatic) +@item Setup a bootstrap server +@item Connect manually +@end itemize + +To setup peer 1 as bootstrapping server change the configuration of +the first one to be a hostlist server by adding the following lines to +@file{peer1.conf} to enable bootstrapping server: + +@example +[hostlist] +OPTIONS = -p +@end example + +@noindent +Then change @file{peer2.conf} and replace the ``@code{SERVERS}'' +line in the ``@code{[hostlist]}'' section with +``@code{http://localhost:8080/}''. Restart both peers using: + +@example +# stop first peer +$ gnunet-arm -c peer1.conf -e +# start first peer +$ gnunet-arm -c peer1.conf -s +# start second peer +$ gnunet-arm -c peer2.conf -s +@end example + +@noindent +Note that if you start your peers without changing these settings, they +will use the ``global'' hostlist servers of the GNUnet P2P network and +likely connect to those peers. At that point, debugging might become +tricky as you're going to be connected to many more peers and would +likely observe traffic and behaviors that are not explicitly controlled +by you. + +@node How to connect manually +@subsection How to connect manually + +If you want to use the @code{peerinfo} tool to connect your +peers, you should: + +@itemize +@item Set @code{IMMEDIATE_START = NO} in section @code{hostlist} +(to not connect to the global GNUnet) +@item Start both peers running @command{gnunet-arm -c peer1.conf -s} +and @command{gnunet-arm -c peer2.conf -s} +@item Get @code{HELLO} message of the first peer running +@command{gnunet-peerinfo -c peer1.conf -g} +@item Give the output to the second peer by running +@command{gnunet-peerinfo -c peer2.conf -p ''} +@end itemize + +Check that they are connected using @command{gnunet-core -c peer1.conf}, +which should give you the other peer's peer identity: + +@example +$ gnunet-core -c peer1.conf +Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG' +@end example + +@node Starting Peers Using the Testbed Service +@section Starting Peers Using the Testbed Service +@c \label{sec:testbed} + +GNUnet's testbed service is used for testing scenarios where +a number of peers are to be started. The testbed can manage peers +on a single host or on multiple hosts in a distributed fashion. +On a single affordable computer, it should be possible to run +around tens of peers without drastically increasing the load on the +system. + +The testbed service can be access through its API +@file{include/gnunet\_testbed\_service.h}. The API provides many +routines for managing a group of peers. It also provides a helper +function @code{GNUNET\_TESTBED\_test\_run()} to quickly setup a +minimalistic testing environment on a single host. + +This function takes a configuration file which will be used as a +template configuration for the peers. The testbed takes care of +modifying relevant options in the peers' configuration such as +@code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values +so that peers run without running into conflicts. It also checks +and assigns the ports in configurations only if they are free. + +Additionally, the testbed service also reads its options from the +same configuration file. Various available options and details +about them can be found in the testbed default configuration file +@file{src/testbed/testbed.conf}. + +With the testbed API, a sample test case can be structured as follows: + +@example +@verbatiminclude examples/testbed_test.c +@end example + +@noindent +The source code for the above listing can be found at +@uref{https://gnunet.org/git/gnunet.git/tree/doc/ +documentation/testbed_test.c} +or in the @file{doc/documentation/} folder of your repository check-out. +After installing GNUnet, the above source code can be compiled as: + +@example +$ export CPPFLAGS="-I/path/to/gnunet/headers" +$ export LDFLAGS="-L/path/to/gnunet/libraries" +$ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \ + -lgnunettestbed -lgnunetdht -lgnunetutil +# Generate (empty) configuration +$ touch template.conf +# run it (press CTRL-C to stop) +$ ./testbed-test +@end example + +@noindent +The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet +is installed into a different directory other than @file{/usr/local}. + +All of testbed API's peer management functions treat management +actions as operations and return operation handles. It is expected +that the operations begin immediately, but they may get delayed (to +balance out load on the system). The program using the API then has +to take care of marking the operation as ``done'' so that its +associated resources can be freed immediately and other waiting +operations can be executed. Operations will be canceled if they are +marked as ``done'' before their completion. + +An operation is treated as completed when it succeeds or fails. +Completion of an operation is either conveyed as events through +@dfn{controller event callback} or through respective +@dfn{operation completion callbacks}. +In functions which support completion notification +through both controller event callback and operation +completion callback, first the controller event callback will be +called. If the operation is not marked as done in that callback +or if the callback is given as NULL when creating the operation, +the operation completion callback will be called. The API +documentation shows which event are to be expected in the +controller event notifications. It also documents any exceptional +behaviour. + +Once the peers are started, test cases often need to connect +some of the peers' services. Normally, opening a connect to +a peer's service requires the peer's configuration. While using +testbed, the testbed automatically generates per-peer configuration. +Accessing those configurations directly through file system is +discouraged as their locations are dynamically created and will be +different among various runs of testbed. To make access to these +configurations easy, testbed API provides the function +@code{GNUNET\_TESTBED\_service\_connect()}. This function fetches +the configuration of a given peer and calls the @dfn{Connect Adapter}. +In the example code, it is the @code{dht\_ca}. A connect adapter is +expected to open the connection to the needed service by using the +provided configuration and return the created service connection handle. +Successful connection to the needed service is signaled through +@code{service\_connect\_comp\_cb}. + +A dual to connect adapter is the @dfn{Disconnect Adapter}. This callback +is called after the connect adapter has been called when the operation +from @code{GNUNET\_TESTBED\_service\_connect()} is marked as ``done''. +It has to disconnect from the service with the provided service +handle (@code{op\_result}). + +Exercise: Find out how many peers you can run on your system. + +Exercise: Find out how to create a 2D torus topology by changing the +options in the configuration file. +@xref{Supported Topologies, The GNUnet Reference Documentation ,, gnunet, The GNUnet Reference Documentation}, +then use the DHT API to store and retrieve values in the network. + +@node Developing Applications +@chapter Developing Applications + +@menu +* gnunet-ext:: +* Adapting the Template:: +* Writing a Client Application:: +* Writing a Service:: +* Interacting directly with other Peers using the CORE Service:: +* Storing peer-specific data using the PEERSTORE service:: +* Using the DHT:: +* Debugging with gnunet-arm:: +@end menu + +@node gnunet-ext +@section gnunet-ext +To develop a new peer-to-peer application or to extend GNUnet we provide +a template build system for writing GNUnet extensions in C. It can be +obtained as follows: + +@example +$ git clone https://gnunet.org/git/gnunet-ext +$ cd gnunet-ext/ +$ ./bootstrap +$ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX +$ make +$ make install +$ make check +@end example + +@noindent +The GNUnet ext template includes examples and a working buildsystem +for a new GNUnet service. A common GNUnet service consists of the +following parts which will be discussed in detail in the remainder +of this document. The functionality of a GNUnet service is implemented in: + +@itemize +@item the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c) +@item the client API (gnunet-ext/src/ext/ext_api.c) +@item the client application using the service API +(gnunet-ext/src/ext/gnunet-ext.c) +@end itemize + +The interfaces for these entities are defined in: + +@itemize +@item client API interface (gnunet-ext/src/ext/ext.h) +@item the service interface (gnunet-ext/src/include/gnunet_service_SERVICE.h) +@item the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h) +@end itemize + + +In addition the ext systems provides: + +@itemize +@item a test testing the API (gnunet-ext/src/ext/test_ext_api.c) +@item a configuration template for the service +(gnunet-ext/src/ext/ext.conf.in) +@end itemize + +@node Adapting the Template +@section Adapting the Template + +The first step for writing any extension with a new service is to +ensure that the @file{ext.conf.in} file contains entries for the +@code{UNIXPATH}, @code{PORT} and @code{BINARY} for the service in a +section named after the service. + +If you want to adapt the template rename the @file{ext.conf.in} to +match your services name, you have to modify the @code{AC\_OUTPUT} +section in @file{configure.ac} in the @file{gnunet-ext} root. + +@node Writing a Client Application +@section Writing a Client Application + +When writing any client application (for example, a command-line +tool), the basic structure is to start with the +@code{GNUNET\_PROGRAM\_run} function. This function will parse +command-line options, setup the scheduler and then invoke the +@code{run} function (with the remaining non-option arguments) +and a handle to the parsed configuration (and the configuration +file name that was used, which is typically not needed): + +@example +@verbatiminclude examples/001.c +@end example + +@menu +* Handling command-line options:: +* Writing a Client Library:: +* Writing a user interface:: +@end menu + +@node Handling command-line options +@subsection Handling command-line options + +Options can then be added easily by adding global variables and +expanding the @code{options} array. For example, the following would +add a string-option and a binary flag (defaulting to @code{NULL} and +@code{GNUNET\_NO} respectively): + +@example +@verbatiminclude examples/002.c +@end example + +Issues such as displaying some helpful text describing options using +the @code{--help} argument and error handling are taken care of when +using this approach. Other @code{GNUNET\_GETOPT\_}-functions can be used +to obtain integer value options, increment counters, etc. You can +even write custom option parsers for special circumstances not covered +by the available handlers. To check if an argument was specified by the +user you initialize the variable with a specific value (e.g. NULL for +a string and GNUNET\_SYSERR for a integer) and check after parsing +happened if the values were modified. + +Inside the @code{run} method, the program would perform the +application-specific logic, which typically involves initializing and +using some client library to interact with the service. The client +library is supposed to implement the IPC whereas the service provides +more persistent P2P functions. + +Exercise: Add a few command-line options and print them inside +of @code{run}. What happens if the user gives invalid arguments? + +@node Writing a Client Library +@subsection Writing a Client Library + +The first and most important step in writing a client library is to +decide on an API for the library. Typical API calls include +connecting to the service, performing application-specific requests +and cleaning up. Many examples for such service APIs can be found +in the @file{gnunet/src/include/gnunet\_*\_service.h} files. + +Then, a client-service protocol needs to be designed. This typically +involves defining various message formats in a header that will be +included by both the service and the client library (but is otherwise +not shared and hence located within the service's directory and not +installed by @command{make install}). Each message must start with a +@code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By +convention, all fields in IPC (and P2P) messages must be in big-endian +format (and thus should be read using @code{ntohl} and similar +functions and written using @code{htonl} and similar functions). +Unique message types must be defined for each message struct in the +@file{gnunet\_protocols.h} header (or an extension-specific include +file). + +@menu +* Connecting to the Service:: +* Sending messages:: +* Receiving Replies from the Service:: +@end menu + +@node Connecting to the Service +@subsubsection Connecting to the Service + +Before a client library can implement the application-specific protocol +with the service, a connection must be created: + +@example +@verbatiminclude examples/003.c +@end example + +@noindent +As a result a @code{GNUNET\_MQ\_Handle} is returned +which can to used henceforth to transmit messages to the service. +The complete MQ API can be found in @file{gnunet\_mq\_lib.h}. +The @code{hanlders} array in the example above is incomplete. +Here is where you will define which messages you expect to +receive from the service, and which functions handle them. +The @code{error\_cb} is a function that is to be called whenever +there are errors communicating with the service. + +@node Sending messages +@subsubsection Sending messages + +In GNUnet, messages are always sent beginning with a +@code{struct GNUNET\_MessageHeader} in big endian format. +This header defines the size and the type of the +message, the payload follows after this header. + +@example +@verbatiminclude examples/004.c +@end example + +@noindent +Existing message types are defined in @file{gnunet\_protocols.h}. +A common way to create a message is with an envelope: + +@example +@verbatiminclude examples/005.c +@end example + +@noindent +Exercise: Define a message struct that includes a 32-bit +unsigned integer in addition to the standard GNUnet MessageHeader. +Add a C struct and define a fresh protocol number for your message. +Protocol numbers in gnunet-ext are defined +in @file{gnunet-ext/src/include/gnunet_protocols_ext.h} + +Exercise: Find out how you can determine the number of messages +in a message queue. + +Exercise: Find out how you can determine when a message you +have queued was actually transmitted. + +Exercise: Define a helper function to transmit a 32-bit +unsigned integer (as payload) to a service using some given client +handle. + +@node Receiving Replies from the Service +@subsubsection Receiving Replies from the Service + +Clients can receive messages from the service using the handlers +specified in the @code{handlers} array we specified when connecting +to the service. Entries in the the array are usually created using +one of two macros, depending on whether the message is fixed size +or variable size. Variable size messages are managed using two +callbacks, one to check that the message is well-formed, the other +to actually process the message. Fixed size messages are fully +checked by the MQ-logic, and thus only need to provide the handler +to process the message. Note that the prefixes @code{check\_} +and @code{handle\_} are mandatory. + +@example +@verbatiminclude examples/006.c +@end example + +@noindent +Exercise: Expand your helper function to receive a response message +(for example, containing just the @code{struct GNUnet MessageHeader} +without any payload). Upon receiving the service's response, you +should call a callback provided to your helper function's API. + +Exercise: Figure out where you can pass values to the +closures (@code{cls}). + +@node Writing a user interface +@subsection Writing a user interface + +Given a client library, all it takes to access a service now is to +combine calls to the client library with parsing command-line +options. + +Exercise: Call your client API from your @code{run()} method in your +client application to send a request to the service. For example, +send a 32-bit integer value based on a number given at the +command-line to the service. + +@node Writing a Service +@section Writing a Service + +Before you can test the client you've written so far, you'll +need to also implement the corresponding service. + +@menu +* Code Placement:: +* Starting a Service:: +@end menu + +@node Code Placement +@subsection Code Placement + +New services are placed in their own subdirectory under +@file{gnunet/src}. This subdirectory should contain the API +implementation file @file{SERVICE\_api.c}, the description of +the client-service protocol @file{SERVICE.h} and P2P protocol +@file{SERVICE\_protocol.h}, the implementation of the service itself +@file{gnunet-service-SERVICE.h} and several files for tests, +including test code and configuration files. + +@node Starting a Service +@subsection Starting a Service + +The key API definition for creating a service is the +@code{GNUNET\_SERVICE\_MAIN} macro: + +@example +@verbatiminclude examples/007.c +@end example + +@noindent +In addition to the service name and flags, the macro takes three +functions, typically called @code{run}, @code{client\_connect\_cb} and +@code{client\_disconnect\_cb} as well as an array of message handlers +that will be called for incoming messages from clients. + +A minimal version of the three central service funtions would look +like this: + +@example +@verbatiminclude examples/008.c +@end example + +@noindent +Exercise: Write a stub service that processes no messages at all +in your code. Create a default configuration for it, integrate it +with the build system and start the service from +@command{gnunet-service-arm} using @command{gnunet-arm -i NAME}. + +Exercise: Figure out how to set the closure (@code{cls}) for handlers +of a service. + +Exercise: Figure out how to send messages from the service back to the +client. + +Each handler function in the service @b{must} eventually (possibly in some +asynchronous continuation) call +@code{GNUNET\_SERVICE\_client\_continue()}. Only after this call +additional messages from the same client may +be processed. This way, the service can throttle processing messages +from the same client. + +Exercise: Change the service to ``handle'' the message from your +client (for now, by printing a message). What happens if you +forget to call @code{GNUNET\_SERVICE\_client\_continue()}? + +@node Interacting directly with other Peers using the CORE Service +@section Interacting directly with other Peers using the CORE Service + +FIXME: This section still needs to be updated to the lastest API! + +One of the most important services in GNUnet is the @code{CORE} service +managing connections between peers and handling encryption between peers. + +One of the first things any service that extends the P2P protocol +typically does is connect to the @code{CORE} service using: + +@example +@verbatiminclude examples/009.c +@end example + +@menu +* New P2P connections:: +* Receiving P2P Messages:: +* Sending P2P Messages:: +* End of P2P connections:: +@end menu + +@node New P2P connections +@subsection New P2P connections + +Before any traffic with a different peer can be exchanged, the peer must +be known to the service. This is notified by the @code{CORE} +@code{connects} callback, which communicates the identity of the new +peer to the service: + +@example +@verbatiminclude examples/010.c +@end example + +@noindent +Note that whatever you return from @code{connects} is given as the +@code{cls} argument to the message handlers for messages from +the respective peer. + +Exercise: Create a service that connects to the @code{CORE}. Then +start (and connect) two peers and print a message once your connect +callback is invoked. + +@node Receiving P2P Messages +@subsection Receiving P2P Messages + +To receive messages from @code{CORE}, you pass the desired +@code{handlers} to the @code{GNUNET\_CORE\_connect()} function, +just as we showed for services. + +It is your responsibility to process messages fast enough or +to implement flow control. If an application does not process +CORE messages fast enough, CORE will randomly drop messages +to not keep a very long queue in memory. + +Exercise: Start one peer with a new service that has a message +handler and start a second peer that only has your ``old'' service +without message handlers. Which ``connect'' handlers are invoked when +the two peers are connected? Why? + +@node Sending P2P Messages +@subsection Sending P2P Messages + +You can transmit messages to other peers using the @code{mq} you were +given during the @code{connect} callback. Note that the @code{mq} +automatically is released upon @code{disconnect} and that you must +not use it afterwards. + +It is your responsibility to not over-fill the message queue, GNUnet +will send the messages roughly in the order given as soon as possible. + +Exercise: Write a service that upon connect sends messages as +fast as possible to the other peer (the other peer should run a +service that ``processes'' those messages). How fast is the +transmission? Count using the STATISTICS service on both ends. Are +messages lost? How can you transmit messages faster? What happens if +you stop the peer that is receiving your messages? + +@node End of P2P connections +@subsection End of P2P connections + +If a message handler returns @code{GNUNET\_SYSERR}, the remote +peer shuts down or there is an unrecoverable network +disconnection, CORE notifies the service that the peer disconnected. +After this notification no more messages will be received from the +peer and the service is no longer allowed to send messages to the peer. +The disconnect callback looks like the following: + +@example +@verbatiminclude examples/011.c +@end example + +@noindent +Exercise: Fix your service to handle peer disconnects. + +@node Storing peer-specific data using the PEERSTORE service +@section Storing peer-specific data using the PEERSTORE service + +GNUnet's PEERSTORE service offers a persistorage for arbitrary +peer-specific data. Other GNUnet services can use the PEERSTORE +to store, retrieve and monitor data records. Each data record +stored with PEERSTORE contains the following fields: + +@itemize +@item subsystem: Name of the subsystem responsible for the record. +@item peerid: Identity of the peer this record is related to. +@item key: a key string identifying the record. +@item value: binary record value. +@item expiry: record expiry date. +@end itemize + +The first step is to start a connection to the PEERSTORE service: +@example +@verbatiminclude examples/012.c +@end example + +The service handle @code{peerstore_handle} will be needed for +all subsequent PEERSTORE operations. + +@menu +* Storing records:: +* Retrieving records:: +* Monitoring records:: +* Disconnecting from PEERSTORE:: +@end menu + +@node Storing records +@subsection Storing records + +To store a new record, use the following function: + +@example +@verbatiminclude examples/013.c +@end example + +@noindent +The @code{options} parameter can either be +@code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple +values can be stored under the same key combination +(subsystem, peerid, key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE} +which means that PEERSTORE will replace any existing values under the +given key combination (subsystem, peerid, key) with the new given value. + +The continuation function @code{cont} will be called after the store +request is successfully sent to the PEERSTORE service. This does not +guarantee that the record is successfully stored, only that it was +received by the service. + +The @code{GNUNET_PEERSTORE_store} function returns a handle to the store +operation. This handle can be used to cancel the store operation only +before the continuation function is called: + +@example +@verbatiminclude examples/013.1.c +@end example + +@node Retrieving records +@subsection Retrieving records + +To retrieve stored records, use the following function: + +@example +@verbatiminclude examples/014.c +@end example + +@noindent +The values of @code{peer} and @code{key} can be @code{NULL}. This +allows the iteration over values stored under any of the following +key combinations: + +@itemize +@item (subsystem) +@item (subsystem, peerid) +@item (subsystem, key) +@item (subsystem, peerid, key) +@end itemize + +The @code{callback} function will be called once with each retrieved +record and once more with a @code{NULL} record to signal the end of +results. + +The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the +iterate operation. This handle can be used to cancel the iterate +operation only before the callback function is called with a +@code{NULL} record. + +@node Monitoring records +@subsection Monitoring records + +PEERSTORE offers the functionality of monitoring for new records +stored under a specific key combination (subsystem, peerid, key). +To start the monitoring, use the following function: + +@example +@verbatiminclude examples/015.c +@end example + +@noindent +Whenever a new record is stored under the given key combination, +the @code{callback} function will be called with this new +record. This will continue until the connection to the PEERSTORE +service is broken or the watch operation is canceled: + +@example +@verbatiminclude examples/016.c +@end example + +@node Disconnecting from PEERSTORE +@subsection Disconnecting from PEERSTORE + +When the connection to the PEERSTORE service is no longer needed, +disconnect using the following function: + +@example +@verbatiminclude examples/017.c +@end example + +@noindent +If the @code{sync_first} flag is set to @code{GNUNET_YES}, +the API will delay the disconnection until all store requests +are received by the PEERSTORE service. Otherwise, it will +disconnect immediately. + +@node Using the DHT +@section Using the DHT + +The DHT allows to store data so other peers in the P2P network can +access it and retrieve data stored by any peers in the network. +This section will explain how to use the DHT. Of course, the first +thing to do is to connect to the DHT service: + +@example +@verbatiminclude examples/018.c +@end example + +@noindent +The second parameter indicates how many requests in parallel to expect. +It is not a hard limit, but a good approximation will make the DHT more +efficient. + +@menu +* Storing data in the DHT:: +* Obtaining data from the DHT:: +* Implementing a block plugin:: +* Monitoring the DHT:: +@end menu + +@node Storing data in the DHT +@subsection Storing data in the DHT +Since the DHT is a dynamic environment (peers join and leave frequently) +the data that we put in the DHT does not stay there indefinitely. It is +important to ``refresh'' the data periodically by simply storing it +again, in order to make sure other peers can access it. + +The put API call offers a callback to signal that the PUT request has been +sent. This does not guarantee that the data is accessible to others peers, +or even that is has been stored, only that the service has requested to +a neighboring peer the retransmission of the PUT request towards its final +destination. Currently there is no feedback about whether or not the data +has been sucessfully stored or where it has been stored. In order to +improve the availablilty of the data and to compensate for possible +errors, peers leaving and other unfavorable events, just make several +PUT requests! + +@example +@verbatiminclude examples/019.c +@end example + +@noindent +Exercise: Store a value in the DHT periodically to make sure it +is available over time. You might consider using the function +@code{GNUNET\_SCHEDULER\_add\_delayed} and call +@code{GNUNET\_DHT\_put} from inside a helper function. + +@node Obtaining data from the DHT +@subsection Obtaining data from the DHT + +As we saw in the previous example, the DHT works in an asynchronous mode. +Each request to the DHT is executed ``in the background'' and the API +calls return immediately. In order to receive results from the DHT, the +API provides a callback. Once started, the request runs in the service, +the service will try to get as many results as possible (filtering out +duplicates) until the timeout expires or we explicitly stop the request. +It is possible to give a ``forever'' timeout with +@code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}. + +If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE} +the callback will get a list of all the peers the data has travelled, +both on the PUT path and on the GET path. + +@example +@verbatiminclude examples/020.c +@end example + +@noindent +Exercise: Store a value in the DHT and after a while retrieve it. +Show the IDs of all the peers the requests have gone through. +In order to convert a peer ID to a string, use the function +@code{GNUNET\_i2s}. Pay attention to the route option parameters +in both calls! + +@node Implementing a block plugin +@subsection Implementing a block plugin + +In order to store data in the DHT, it is necessary to provide a block +plugin. The DHT uses the block plugin to ensure that only well-formed +requests and replies are transmitted over the network. + +The block plugin should be put in a file @file{plugin\_block\_SERVICE.c} +in the service's respective directory. The +mandatory functions that need to be implemented for a block plugin are +described in the following sections. + +@menu +* Validating requests and replies:: +* Deriving a key from a reply:: +* Initialization of the plugin:: +* Shutdown of the plugin:: +* Integration of the plugin with the build system:: +@end menu + +@node Validating requests and replies +@subsubsection Validating requests and replies + +The evaluate function should validate a reply or a request. It returns +a @code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All +possible answers are in @file{gnunet\_block\_lib.h}. The function will +be called with a @code{reply\_block} argument of @code{NULL} for +requests. Note that depending on how @code{evaluate} is called, only +some of the possible return values are valid. The specific meaning of +the @code{xquery} argument is application-specific. Applications that +do not use an extended query should check that the @code{xquery\_size} +is zero. The block group is typically used to filter duplicate +replies. + +@example +@verbatiminclude examples/021.c +@end example + +@noindent +Note that it is mandatory to detect duplicate replies in this function +and return the respective status code. Duplicate detection is +typically done using the Bloom filter block group provided by +@file{libgnunetblockgroup.so}. Failure to do so may cause replies to +circle in the network. + +@node Deriving a key from a reply +@subsubsection Deriving a key from a reply + +The DHT can operate more efficiently if it is possible to derive a key +from the value of the corresponding block. The @code{get\_key} +function is used to obtain the key of a block --- for example, by +means of hashing. If deriving the key is not possible, the function +should simply return @code{GNUNET\_SYSERR} (the DHT will still work +just fine with such blocks). + +@example +@verbatiminclude examples/022.c +@end example + +@node Initialization of the plugin +@subsubsection Initialization of the plugin + +The plugin is realized as a shared C library. The library must export +an initialization function which should initialize the plugin. The +initialization function specifies what block types the plugin cares +about and returns a struct with the functions that are to be used for +validation and obtaining keys (the ones just defined above). + +@example +@verbatiminclude examples/023.c +@end example + +@node Shutdown of the plugin +@subsubsection Shutdown of the plugin + +Following GNUnet's general plugin API concept, the plugin must +export a second function for cleaning up. It usually does very +little. + +@example +@verbatiminclude examples/024.c +@end example + +@node Integration of the plugin with the build system +@subsubsection Integration of the plugin with the build system + +In order to compile the plugin, the @file{Makefile.am} file for the +service SERVICE should contain a rule similar to this: +@c Actually this is a Makefile not C. But the whole structure of examples +@c must be improved. + +@example +@verbatiminclude examples/025.Makefile.am +@end example + +@noindent +Exercise: Write a block plugin that accepts all queries +and all replies but prints information about queries and replies +when the respective validation hooks are called. + +@node Monitoring the DHT +@subsection Monitoring the DHT + +It is possible to monitor the functioning of the local +DHT service. When monitoring the DHT, the service will +alert the monitoring program of any events, both started +locally or received for routing from another peer. +The are three different types of events possible: a +GET request, a PUT request or a response (a reply to a GET). + +Since the different events have different associated data, +the API gets 3 different callbacks (one for each message type) +and optional type and key parameters, to allow for filtering of +messages. When an event happens, the appropiate callback is +called with all the information about the event. + +@example +@verbatiminclude examples/026.c +@end example + +@node Debugging with gnunet-arm +@section Debugging with gnunet-arm + +Even if services are managed by @command{gnunet-arm}, you can +start them with @command{gdb} or @command{valgrind}. For +example, you could add the following lines to your +configuration file to start the DHT service in a @command{gdb} +session in a fresh @command{xterm}: + +@example +[dht] +PREFIX=xterm -e gdb --args +@end example + +@noindent +Alternatively, you can stop a service that was started via +ARM and run it manually: + +@example +$ gnunet-arm -k dht +$ gdb --args gnunet-service-dht -L DEBUG +$ valgrind gnunet-service-dht -L DEBUG +@end example + +@noindent +Assuming other services are well-written, they will automatically +re-integrate the restarted service with the peer. + +GNUnet provides a powerful logging mechanism providing log +levels @code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}. +The current log level is configured using the @code{$GNUNET_FORCE_LOG} +environmental variable. The @code{DEBUG} level is only available if +@command{--enable-logging=verbose} was used when running +@command{configure}. More details about logging can be found under +@uref{https://gnunet.org/logging}. + +You should also probably enable the creation of core files, by setting +@code{ulimit}, and echo'ing @code{1} into +@file{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the +core dumps with @command{gdb}, which is often the fastest method to +find simple errors. + +Exercise: Add a memory leak to your service and obtain a trace +pointing to the leak using @command{valgrind} while running the service +from @command{gnunet-service-arm}. + + +@c ********************************************************************* +@node GNU Free Documentation License +@appendix GNU Free Documentation License +@cindex license, GNU Free Documentation License +@include fdl-1.3.texi + + +@bye -- cgit v1.2.3