From 5131514b29473fb93d4a9e0521090d326b0efa38 Mon Sep 17 00:00:00 2001 From: ng0 Date: Sat, 9 Nov 2019 17:19:45 +0000 Subject: rename texi source gnunet-tutorial to gnunet_tutorial --- doc/tutorial/gnunet-tutorial.texi | 1568 ------------------------------------- doc/tutorial/gnunet_tutorial.texi | 1568 +++++++++++++++++++++++++++++++++++++ 2 files changed, 1568 insertions(+), 1568 deletions(-) delete mode 100644 doc/tutorial/gnunet-tutorial.texi create mode 100644 doc/tutorial/gnunet_tutorial.texi (limited to 'doc/tutorial') diff --git a/doc/tutorial/gnunet-tutorial.texi b/doc/tutorial/gnunet-tutorial.texi deleted file mode 100644 index 6982b4200..000000000 --- a/doc/tutorial/gnunet-tutorial.texi +++ /dev/null @@ -1,1568 +0,0 @@ -\input texinfo - -@setfilename gnunet-tutorial.info -@documentencoding UTF-8 -@settitle GNUnet Tutorial -@c @exampleindent 2 - - -@include version.texi - -@copying -Copyright @copyright{} 2001-2018 GNUnet e.V. - -Permission is granted to copy, distribute and/or modify this document -under the terms of the GNU Free Documentation License, Version 1.3 or -any later version published by the Free Software Foundation; with no -Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A -copy of the license is included in the section entitled ``GNU Free -Documentation License''. - -A copy of the license is also available from the Free Software -Foundation Web site at @url{http://www.gnu.org/licenses/fdl.html}. - -Alternately, this document is also available under the General -Public License, version 3 or later, as published by the Free Software -Foundation. A copy of the license is included in the section entitled -``GNU General Public License''. - -A copy of the license is also available from the Free Software -Foundation Web site at @url{http://www.gnu.org/licenses/gpl.html}. -@end copying - -@dircategory Tutorial -@direntry -* GNUnet-C-Tutorial: (gnunet-c-tutorial). C Tutorial for GNunet -@end direntry - - -@titlepage -@title GNUnet C Tutorial -@subtitle A Tutorial for GNUnet @value{VERSION} (C version) -@author The GNUnet Developers - -@page -@vskip 0pt plus 1filll - -@insertcopying -@end titlepage - -@contents - -@c **** TODO -@c 1. Update content? -@c 2. Either reference main documentation or -@c 3. Merge this into main documentation - -@node Top -@top Introduction - -This tutorials explains how to install GNUnet on a -GNU/Linux system and gives an introduction on how -GNUnet can be used to develop a Peer-to-Peer application. -Detailed installation instructions for -various operating systems and a detailed list of all -dependencies can be found on our website at -@uref{https://docs.gnunet.org/#Installation} and in our -Reference Documentation (GNUnet Handbook). - -Please read this tutorial carefully since every single step is -important, and do not hesitate to contact the GNUnet team if you have -any questions or problems! Visit this link in your webbrowser to learn -how to contact the GNUnet team: -@uref{https://gnunet.org/en/contact.html} - -@menu - -* Installing GNUnet:: Installing GNUnet -* Introduction to GNUnet Architecture:: Introduction to GNUnet Architecture -* First Steps with GNUnet:: First Steps with GNUnet -* Developing Applications:: Developing Applications -* GNU Free Documentation License:: The license of this manual - -@detailmenu - --- The Detailed Node Listing --- - -Installing GNUnet - -* Obtaining a stable version:: -* Installing Build Tool Chain and Dependencies:: -* Obtaining the latest version from Git:: -* Compiling and Installing GNUnet:: -* Common Issues - Check your GNUnet installation:: - -Introduction to GNUnet Architecture - -First Steps with GNUnet - -* Configure your peer:: -* Start a peer:: -* Monitor a peer:: -* Starting Two Peers by Hand:: -* Starting Peers Using the Testbed Service:: - -Developing Applications - -* gnunet-ext:: -* Adapting the Template:: -* Writing a Client Application:: -* Writing a Service:: -* Interacting directly with other Peers using the CORE Service:: -* Storing peer-specific data using the PEERSTORE service:: -* Using the DHT:: -* Debugging with gnunet-arm:: - -@end detailmenu -@end menu - -@node Installing GNUnet -@chapter Installing GNUnet - -First of all you have to install a current version of GNUnet. -You can download a tarball of a stable version from GNU FTP mirrors -or obtain the latest development version from our Git repository. - -Most of the time you should prefer to download the stable version -since with the latest development version things can be broken, -functionality can be changed or tests can fail. You should only use -the development version if you know that you require a certain -feature or a certain issue has been fixed since the last release. - -@menu -* Obtaining a stable version:: -* Installing Build Tool Chain and Dependencies:: -* Obtaining the latest version from Git:: -* Compiling and Installing GNUnet:: -* Common Issues - Check your GNUnet installation:: -@end menu - -@node Obtaining a stable version -@section Obtaining a stable version - -Download the tarball from -@indicateurl{https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz}. - -Make sure to download the associated @file{.sig} file and to verify the -authenticity of the tarball against it, like this: - -@example -$ wget https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz.sig -$ gpg --verify-files gnunet-@value{VERSION}.tar.gz.sig -@end example - -@noindent -If this command fails because you do not have the required public key, -then you need to run the following command to import it: - -@example -$ gpg --keyserver keys.gnupg.net --recv-keys 48426C7E -@end example - -@noindent -and rerun the @code{gpg --verify-files} command. - -@b{Note:}@ -@b{The pub key to sign the 0.10.1 release has been -revoked}. You will get an error message stating that -@b{there is no known public key or that it has been revoked}. -The next release of GNUnet will have a valid signature -again. We are sorry for the inconvenience this causes. -Another possible source you could use is our -"gnunet" git repository which, since the change from SVN to git in 2016, -has mandatory signed commits by every developer. - -After verifying the signature you can extract the tarball. -The resulting directory will be renamed to @file{gnunet}, which we will -be using in the remainder of this document to refer to the -root of the source directory. - -@example -$ tar xvzf gnunet-@value{VERSION}.tar.gz -$ mv gnunet-@value{VERSION} gnunet -@end example - -@c FIXME: This can be irritating for the reader - First we say git should -@c be avoid unless it is really required, and then we write this -@c paragraph: -@noindent -However, please note that stable versions can be very outdated. -As a developer you are @b{strongly} encouraged to use the version -from @uref{https://git.gnunet.org/, the git server}. - -@node Installing Build Tool Chain and Dependencies -@section Installing Build Tool Chain and Dependencies - -To successfully compile GNUnet, you need the tools to build GNUnet and -the required dependencies. Please take a look at the -GNUnet Reference Documentation -(@pxref{Dependencies, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) -for a list of required dependencies and -(@pxref{Generic installation instructions, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) -read its Installation chapter for specific instructions for -your Operating System. -Please check the notes at the end of the configure process about -required dependencies. - -For GNUnet bootstrapping support and the HTTP(S) plugin you should -install @uref{https://gnunet.org/en/gnurl.html, libgnurl}. -For the filesharing service you should install at least one of the -datastore backends (MySQL, SQlite and PostgreSQL are supported). - -@node Obtaining the latest version from Git -@section Obtaining the latest version from Git - -The latest development version can be obtained from our Git repository. -To get the code you need to have @code{Git} installed. Usually your -Operating System package manager should provide a suitable distribution -of git (otherwise check out Guix or Nix). If you are using an Operating -System based on Debian's apt: - -@example -$ sudo apt-get install git -@end example - -This is required for obtaining the repository, which is achieved with -the following command: - -@example -$ git clone https://git.gnunet.org/gnunet -@end example - -@noindent -After cloning the repository, you have to execute the @file{bootstrap} -script in the new directory: - -@example -$ cd gnunet -$ ./bootstrap -@end example - -@noindent -The remainder of this tutorial will assume that you have the -Git branch ``master'' checked out. - -@node Compiling and Installing GNUnet -@section Compiling and Installing GNUnet - -Note: This section is a duplication of the more in depth -@pxref{GNUnet Installation Handbook, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}. - -First, you need to install libgnupgerror 1.27 (or later) and -libgcrypt 1.7.6 (or later): - -@example -$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" -$ wget $GNUPGFTP/libgpg-error/libgpg-error-1.27.tar.bz2 -$ tar xf libgpg-error-1.27.tar.bz2 -$ cd libgpg-error-1.27 -$ ./configure -$ make -$ sudo make install -$ cd .. -@end example - -@example -$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" -$ wget $GNUPGFTP/libgcrypt/libgcrypt-1.7.6.tar.bz2 -$ tar xf libgcrypt-1.7.6.tar.bz2 -$ cd libgcrypt-1.7.6 -$ ./configure -$ make -$ sudo make install -$ cd .. -@end example - -@menu -* Installation:: -@end menu - -@node Installation -@subsection Installation -Assuming all dependencies are installed, the following commands will -compile and install GNUnet in your home directory. You can specify the -directory where GNUnet will be installed by changing the -@code{--prefix} value when calling @command{./configure}. If -you do not specifiy a prefix, GNUnet is installed in the directory -@file{/usr/local}. When developing new applications you may want -to enable verbose logging by adding @code{--enable-logging=verbose}: - -@example -$ export PREFIX=$HOME -$ ./configure --prefix=$PREFIX --enable-logging -$ make -$ make install -@end example - -@noindent -After installing GNUnet you have to add your GNUnet installation -to your path environmental variable. In addition you have to -create the @file{.config} directory in your home directory -(unless it already exists) where GNUnet stores its data and an -empty GNUnet configuration file: - -@example -$ export PATH=$PATH:$PREFIX/bin -$ echo export PATH=$PREFIX/bin:\\$PATH >> ~/.bashrc -$ mkdir ~/.config/ -$ touch ~/.config/gnunet.conf -@end example - -@node Common Issues - Check your GNUnet installation -@section Common Issues - Check your GNUnet installation - -You should check your installation to ensure that installing GNUnet -was successful up to this point. You should be able to access GNUnet's -binaries and run GNUnet's self check. - -@example -$ which gnunet-arm -$PREFIX/bin/gnunet-arm -@end example - -@noindent -should return $PREFIX/bin/gnunet-arm (where $PREFIX is the location -you have set earlier). It should be located in your -GNUnet installation and the output should not be empty. - -If you see an output like: - -@example -$ which gnunet-arm -@end example - -@noindent -check your PATH variable to ensure GNUnet's @file{bin} directory is -included. - -GNUnet provides tests for all of its subcomponents. Assuming you have -successfully built GNUnet, run - -@example -$ cd gnunet -$ make check -@end example - -@noindent -to execute tests for all components. @command{make check} traverses all -subdirectories in @file{src}. For every subdirectory you should -get a message like this: - -@example -make[2]: Entering directory `/home/$USER/gnunet/contrib' -PASS: test_gnunet_prefix -============= -1 test passed -============= -@end example - -@node Introduction to GNUnet Architecture -@chapter Introduction to GNUnet Architecture - -GNUnet is organized in layers and services. Each service is composed of a -main service implementation and a client library for other programs to use -the service's functionality, described by an API. -@c This approach is shown in -@c FIXME: enable this once the commented block below works: -@c figure~\ref fig:service. -Some services provide an additional command line tool to enable the user -to interact with the service. - -Very often it is other GNUnet services that will use these APIs to build -the higher layers of GNUnet on top of the lower ones. Each layer expands -or extends the functionality of the service below (for instance, to build -a mesh on top of a DHT). -@c FXIME: See comment above. -@c See figure ~\ref fig:interaction for an illustration of this approach. - -@c ** @image filename[, width[, height[, alttext[, extension]]]] -@c FIXME: Texlive (?) 20112 makes the assumption that this means -@c 'images/OBJECTNAME.txt' but later versions of it (2017) use this -@c syntax as described below. -@c TODO: Checkout the makedoc script Guile uses. - -@c FIXME!!! -@c @image{images/gnunet-tutorial-service,,5in,Service with API and network protocol,.png} -@c @image{images/gnunet-tutorial-system,,5in,The layered system architecture of GNUnet,.png} - -@c \begin{figure}[!h] -@c \begin{center} -@c % \begin{subfigure} -@c \begin{subfigure}[b]{0.3\textwidth} -@c \centering -@c \includegraphics[width=\textwidth]{figs/Service.pdf} -@c \caption{Service with API and network protocol} -@c \label{fig:service} -@c \end{subfigure} -@c ~~~~~~~~~~ -@c \begin{subfigure}[b]{0.3\textwidth} -@c \centering -@c \includegraphics[width=\textwidth]{figs/System.pdf} -@c \caption{Service interaction} -@c \label{fig:interaction} -@c \end{subfigure} -@c \end{center} -@c \caption{GNUnet's layered system architecture} -@c \end{figure} - -The main service implementation runs as a standalone process in the -Operating System and the client code runs as part of the client program, -so crashes of a client do not affect the service process or other clients. -The service and the clients communicate via a message protocol to be -defined and implemented by the programmer. - -@node First Steps with GNUnet -@chapter First Steps with GNUnet - -@menu -* Configure your peer:: -* Start a peer:: -* Monitor a peer:: -* Starting Two Peers by Hand:: -* Starting Peers Using the Testbed Service:: -@end menu - -@node Configure your peer -@section Configure your peer - -First of all we need to configure your peer. Each peer is started with -a configuration containing settings for GNUnet itself and its services. -This configuration is based on the default configuration shipped with -GNUnet and can be modified. The default configuration is located in the -@file{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you -can specify a customized configuration using the the @command{-c} command -line switch when starting the ARM service and all other services. When -using a modified configuration the default values are loaded and only -values specified in the configuration file will replace the default -values. - -Since we want to start additional peers later, we need some modifications -from the default configuration. We need to create a separate service -home and a file containing our modifications for this peer: - -@example -$ mkdir ~/gnunet1/ -$ touch peer1.conf -@end example - -@noindent -Now add the following lines to @file{peer1.conf} to use this directory. -For simplified usage we want to prevent the peer to connect to the GNUnet -network since this could lead to confusing output. This modifications -will replace the default settings: - -@example -[PATHS] -# Use this directory to store GNUnet data -GNUNET_HOME = ~/gnunet1/ -[hostlist] -# prevent bootstrapping -SERVERS = -@end example - -@node Start a peer -@section Start a peer -Each GNUnet instance (called peer) has an identity (peer ID) based on a -cryptographic public private key pair. The peer ID is the printable hash -of the public key. - -GNUnet services are controlled by a master service, the so called -@dfn{Automatic Restart Manager} (ARM). ARM starts, stops and even -restarts services automatically or on demand when a client connects. -You interact with the ARM service using the @command{gnunet-arm} tool. -GNUnet can then be started with @command{gnunet-arm -s} and stopped with -@command{gnunet-arm -e}. An additional service not automatically started -can be started using @command{gnunet-arm -i } and stopped -using @command{gnunet-arm -k }. - -Once you have started your peer, you can use many other GNUnet commands -to interact with it. For example, you can run: - -@example -$ gnunet-peerinfo -s -@end example - -@noindent -to obtain the public key of your peer. - -You should see an output containing the peer ID similar to: - -@example -I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'. -@end example - -@node Monitor a peer -@section Monitor a peer - -In this section, we will monitor the behaviour of our peer's DHT -service with respect to a specific key. First we will start -GNUnet and then start the DHT service and use the DHT monitor tool -to monitor the PUT and GET commands we issue ussing the -@command{gnunet-dht-put} and @command{gnunet-dht-get} commands. -Using the ``monitor'' line given below, you can observe the behavior -of your own peer's DHT with respect to the specified KEY: - -@example -# start gnunet with all default services: -$ gnunet-arm -c ~/peer1.conf -s -# start DHT service: -$ gnunet-arm -c ~/peer1.conf -i dht -$ cd ~/gnunet/src/dht; -$ ./gnunet-dht-monitor -c ~/peer1.conf -k KEY -@end example - -@noindent -Now open a separate terminal and change again to -the @file{gnunet/src/dht} directory: - -@example -$ cd ~/gnunet/src/dht -# put VALUE under KEY in the DHT: -$ ./gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE -# get key KEY from the DHT: -$ ./gnunet/src/dht/gnunet-dht-get -c ~/peer1.conf -k KEY -# print statistics about current GNUnet state: -$ gnunet-statistics -c ~/peer1.conf -# print statistics about DHT service: -$ gnunet-statistics -c ~/peer1.conf -s dht -@end example - -@node Starting Two Peers by Hand -@section Starting Two Peers by Hand - -This section describes how to start two peers on the same machine by hand. -The process is rather painful, but the description is somewhat -instructive. In practice, you might prefer the automated method -(@pxref{Starting Peers Using the Testbed Service}). - -@menu -* Setup a second peer:: -* Start the second peer and connect the peers:: -* How to connect manually:: -@end menu - -@node Setup a second peer -@subsection Setup a second peer -We will now start a second peer on your machine. -For the second peer, you will need to manually create a modified -configuration file to avoid conflicts with ports and directories. -A peers configuration file is by default located -in @file{~/.gnunet/gnunet.conf}. This file is typically very short -or even empty as only the differences to the defaults need to be -specified. The defaults are located in many files in the -@file{$PREFIX/share/gnunet/config.d} directory. - -To configure the second peer, use the files -@file{$PREFIX/share/gnunet/config.d} as a template for your main -configuration file: - -@example -$ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf -@end example - -@noindent -Now you have to edit @file{peer2.conf} and change: - -@itemize -@item @code{GNUNET\_TEST\_HOME} under @code{PATHS} -@item Every (uncommented) value for ``@code{PORT}'' (add 10000) in any -section (the option may be commented out if @code{PORT} is -prefixed by "\#", in this case, UNIX domain sockets are used -and the PORT option does not need to be touched) -@item Every value for ``@code{UNIXPATH}'' in any section -(e.g. by adding a "-p2" suffix) -@end itemize - -to a fresh, unique value. Make sure that the PORT numbers stay -below 65536. From now on, whenever you interact with the second peer, -you need to specify @command{-c peer2.conf} as an additional -command line argument. - -Now, generate the 2nd peer's private key: - -@example -$ gnunet-peerinfo -s -c peer2.conf -@end example - -@noindent -This may take a while, generate entropy using your keyboard or mouse -as needed. Also, make sure the output is different from the -gnunet-peerinfo output for the first peer (otherwise you made an -error in the configuration). - -@node Start the second peer and connect the peers -@subsection Start the second peer and connect the peers - -Then, you can start a second peer using: - -@example -$ gnunet-arm -c peer2.conf -s -$ gnunet-arm -c peer2.conf -i dht -$ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE -$ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY -@end example - -If you want the two peers to connect, you have multiple options: - -@itemize -@item UDP neighbour discovery (automatic) -@item Setup a bootstrap server -@item Connect manually -@end itemize - -To setup peer 1 as bootstrapping server change the configuration of -the first one to be a hostlist server by adding the following lines to -@file{peer1.conf} to enable bootstrapping server: - -@example -[hostlist] -OPTIONS = -p -@end example - -@noindent -Then change @file{peer2.conf} and replace the ``@code{SERVERS}'' -line in the ``@code{[hostlist]}'' section with -``@code{http://localhost:8080/}''. Restart both peers using: - -@example -# stop first peer -$ gnunet-arm -c peer1.conf -e -# start first peer -$ gnunet-arm -c peer1.conf -s -# start second peer -$ gnunet-arm -c peer2.conf -s -@end example - -@noindent -Note that if you start your peers without changing these settings, they -will use the ``global'' hostlist servers of the GNUnet P2P network and -likely connect to those peers. At that point, debugging might become -tricky as you're going to be connected to many more peers and would -likely observe traffic and behaviors that are not explicitly controlled -by you. - -@node How to connect manually -@subsection How to connect manually - -If you want to use the @code{peerinfo} tool to connect your -peers, you should: - -@itemize -@item Set @code{IMMEDIATE_START = NO} in section @code{hostlist} -(to not connect to the global GNUnet) -@item Start both peers running @command{gnunet-arm -c peer1.conf -s} -and @command{gnunet-arm -c peer2.conf -s} -@item Get @code{HELLO} message of the first peer running -@command{gnunet-peerinfo -c peer1.conf -g} -@item Give the output to the second peer by running -@command{gnunet-peerinfo -c peer2.conf -p ''} -@end itemize - -Check that they are connected using @command{gnunet-core -c peer1.conf}, -which should give you the other peer's peer identity: - -@example -$ gnunet-core -c peer1.conf -Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG' -@end example - -@node Starting Peers Using the Testbed Service -@section Starting Peers Using the Testbed Service -@c \label{sec:testbed} - -GNUnet's testbed service is used for testing scenarios where -a number of peers are to be started. The testbed can manage peers -on a single host or on multiple hosts in a distributed fashion. -On a single affordable computer, it should be possible to run -around tens of peers without drastically increasing the load on the -system. - -The testbed service can be access through its API -@file{include/gnunet\_testbed\_service.h}. The API provides many -routines for managing a group of peers. It also provides a helper -function @code{GNUNET\_TESTBED\_test\_run()} to quickly setup a -minimalistic testing environment on a single host. - -This function takes a configuration file which will be used as a -template configuration for the peers. The testbed takes care of -modifying relevant options in the peers' configuration such as -@code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values -so that peers run without running into conflicts. It also checks -and assigns the ports in configurations only if they are free. - -Additionally, the testbed service also reads its options from the -same configuration file. Various available options and details -about them can be found in the testbed default configuration file -@file{src/testbed/testbed.conf}. - -With the testbed API, a sample test case can be structured as follows: - -@example -@verbatiminclude examples/testbed_test.c -@end example - -@noindent -The source code for the above listing can be found at -@c FIXME: This is not the correct URL. Where is the file? -@uref{https://git.gnunet.org/gnunet.git/tree/doc/documentation/testbed_test.c} -or in the @file{doc/} folder of your repository check-out. -After installing GNUnet, the above source code can be compiled as: - -@example -$ export CPPFLAGS="-I/path/to/gnunet/headers" -$ export LDFLAGS="-L/path/to/gnunet/libraries" -$ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \ - -lgnunettestbed -lgnunetdht -lgnunetutil -# Generate (empty) configuration -$ touch template.conf -# run it (press CTRL-C to stop) -$ ./testbed-test -@end example - -@noindent -The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet -is installed into a different directory other than @file{/usr/local}. - -All of testbed API's peer management functions treat management -actions as operations and return operation handles. It is expected -that the operations begin immediately, but they may get delayed (to -balance out load on the system). The program using the API then has -to take care of marking the operation as ``done'' so that its -associated resources can be freed immediately and other waiting -operations can be executed. Operations will be canceled if they are -marked as ``done'' before their completion. - -An operation is treated as completed when it succeeds or fails. -Completion of an operation is either conveyed as events through -@dfn{controller event callback} or through respective -@dfn{operation completion callbacks}. -In functions which support completion notification -through both controller event callback and operation -completion callback, first the controller event callback will be -called. If the operation is not marked as done in that callback -or if the callback is given as NULL when creating the operation, -the operation completion callback will be called. The API -documentation shows which event are to be expected in the -controller event notifications. It also documents any exceptional -behaviour. - -Once the peers are started, test cases often need to connect -some of the peers' services. Normally, opening a connect to -a peer's service requires the peer's configuration. While using -testbed, the testbed automatically generates per-peer configuration. -Accessing those configurations directly through file system is -discouraged as their locations are dynamically created and will be -different among various runs of testbed. To make access to these -configurations easy, testbed API provides the function -@code{GNUNET\_TESTBED\_service\_connect()}. This function fetches -the configuration of a given peer and calls the @dfn{Connect Adapter}. -In the example code, it is the @code{dht\_ca}. A connect adapter is -expected to open the connection to the needed service by using the -provided configuration and return the created service connection handle. -Successful connection to the needed service is signaled through -@code{service\_connect\_comp\_cb}. - -A dual to connect adapter is the @dfn{Disconnect Adapter}. This callback -is called after the connect adapter has been called when the operation -from @code{GNUNET\_TESTBED\_service\_connect()} is marked as ``done''. -It has to disconnect from the service with the provided service -handle (@code{op\_result}). - -Exercise: Find out how many peers you can run on your system. - -Exercise: Find out how to create a 2D torus topology by changing the -options in the configuration file. -@xref{Supported Topologies, The GNUnet Reference Documentation ,, gnunet, The GNUnet Reference Documentation}, -then use the DHT API to store and retrieve values in the network. - -@node Developing Applications -@chapter Developing Applications - -@menu -* gnunet-ext:: -* Adapting the Template:: -* Writing a Client Application:: -* Writing a Service:: -* Interacting directly with other Peers using the CORE Service:: -* Storing peer-specific data using the PEERSTORE service:: -* Using the DHT:: -* Debugging with gnunet-arm:: -@end menu - -@node gnunet-ext -@section gnunet-ext -To develop a new peer-to-peer application or to extend GNUnet we provide -a template build system for writing GNUnet extensions in C. It can be -obtained as follows: - -@example -$ git clone https://git.gnunet.org/gnunet-ext -$ cd gnunet-ext/ -$ ./bootstrap -$ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX -$ make -$ make install -$ make check -@end example - -@noindent -The GNUnet ext template includes examples and a working buildsystem -for a new GNUnet service. A common GNUnet service consists of the -following parts which will be discussed in detail in the remainder -of this document. The functionality of a GNUnet service is implemented in: - -@itemize -@item the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c) -@item the client API (gnunet-ext/src/ext/ext_api.c) -@item the client application using the service API -(gnunet-ext/src/ext/gnunet-ext.c) -@end itemize - -The interfaces for these entities are defined in: - -@itemize -@item client API interface (gnunet-ext/src/ext/ext.h) -@item the service interface (gnunet-ext/src/include/gnunet_service_SERVICE.h) -@item the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h) -@end itemize - - -In addition the ext systems provides: - -@itemize -@item a test testing the API (gnunet-ext/src/ext/test_ext_api.c) -@item a configuration template for the service -(gnunet-ext/src/ext/ext.conf.in) -@end itemize - -@node Adapting the Template -@section Adapting the Template - -The first step for writing any extension with a new service is to -ensure that the @file{ext.conf.in} file contains entries for the -@code{UNIXPATH}, @code{PORT} and @code{BINARY} for the service in a -section named after the service. - -If you want to adapt the template rename the @file{ext.conf.in} to -match your services name, you have to modify the @code{AC\_OUTPUT} -section in @file{configure.ac} in the @file{gnunet-ext} root. - -@node Writing a Client Application -@section Writing a Client Application - -When writing any client application (for example, a command-line -tool), the basic structure is to start with the -@code{GNUNET\_PROGRAM\_run} function. This function will parse -command-line options, setup the scheduler and then invoke the -@code{run} function (with the remaining non-option arguments) -and a handle to the parsed configuration (and the configuration -file name that was used, which is typically not needed): - -@example -@verbatiminclude examples/001.c -@end example - -@menu -* Handling command-line options:: -* Writing a Client Library:: -* Writing a user interface:: -@end menu - -@node Handling command-line options -@subsection Handling command-line options - -Options can then be added easily by adding global variables and -expanding the @code{options} array. For example, the following would -add a string-option and a binary flag (defaulting to @code{NULL} and -@code{GNUNET\_NO} respectively): - -@example -@verbatiminclude examples/002.c -@end example - -Issues such as displaying some helpful text describing options using -the @code{--help} argument and error handling are taken care of when -using this approach. Other @code{GNUNET\_GETOPT\_}-functions can be used -to obtain integer value options, increment counters, etc. You can -even write custom option parsers for special circumstances not covered -by the available handlers. To check if an argument was specified by the -user you initialize the variable with a specific value (e.g. NULL for -a string and GNUNET\_SYSERR for a integer) and check after parsing -happened if the values were modified. - -Inside the @code{run} method, the program would perform the -application-specific logic, which typically involves initializing and -using some client library to interact with the service. The client -library is supposed to implement the IPC whereas the service provides -more persistent P2P functions. - -Exercise: Add a few command-line options and print them inside -of @code{run}. What happens if the user gives invalid arguments? - -@node Writing a Client Library -@subsection Writing a Client Library - -The first and most important step in writing a client library is to -decide on an API for the library. Typical API calls include -connecting to the service, performing application-specific requests -and cleaning up. Many examples for such service APIs can be found -in the @file{gnunet/src/include/gnunet\_*\_service.h} files. - -Then, a client-service protocol needs to be designed. This typically -involves defining various message formats in a header that will be -included by both the service and the client library (but is otherwise -not shared and hence located within the service's directory and not -installed by @command{make install}). Each message must start with a -@code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By -convention, all fields in IPC (and P2P) messages must be in big-endian -format (and thus should be read using @code{ntohl} and similar -functions and written using @code{htonl} and similar functions). -Unique message types must be defined for each message struct in the -@file{gnunet\_protocols.h} header (or an extension-specific include -file). - -@menu -* Connecting to the Service:: -* Sending messages:: -* Receiving Replies from the Service:: -@end menu - -@node Connecting to the Service -@subsubsection Connecting to the Service - -Before a client library can implement the application-specific protocol -with the service, a connection must be created: - -@example -@verbatiminclude examples/003.c -@end example - -@noindent -As a result a @code{GNUNET\_MQ\_Handle} is returned -which can to used henceforth to transmit messages to the service. -The complete MQ API can be found in @file{gnunet\_mq\_lib.h}. -The @code{hanlders} array in the example above is incomplete. -Here is where you will define which messages you expect to -receive from the service, and which functions handle them. -The @code{error\_cb} is a function that is to be called whenever -there are errors communicating with the service. - -@node Sending messages -@subsubsection Sending messages - -In GNUnet, messages are always sent beginning with a -@code{struct GNUNET\_MessageHeader} in big endian format. -This header defines the size and the type of the -message, the payload follows after this header. - -@example -@verbatiminclude examples/004.c -@end example - -@noindent -Existing message types are defined in @file{gnunet\_protocols.h}. -A common way to create a message is with an envelope: - -@example -@verbatiminclude examples/005.c -@end example - -@noindent -Exercise: Define a message struct that includes a 32-bit -unsigned integer in addition to the standard GNUnet MessageHeader. -Add a C struct and define a fresh protocol number for your message. -Protocol numbers in gnunet-ext are defined -in @file{gnunet-ext/src/include/gnunet_protocols_ext.h} - -Exercise: Find out how you can determine the number of messages -in a message queue. - -Exercise: Find out how you can determine when a message you -have queued was actually transmitted. - -Exercise: Define a helper function to transmit a 32-bit -unsigned integer (as payload) to a service using some given client -handle. - -@node Receiving Replies from the Service -@subsubsection Receiving Replies from the Service - -Clients can receive messages from the service using the handlers -specified in the @code{handlers} array we specified when connecting -to the service. Entries in the the array are usually created using -one of two macros, depending on whether the message is fixed size -or variable size. Variable size messages are managed using two -callbacks, one to check that the message is well-formed, the other -to actually process the message. Fixed size messages are fully -checked by the MQ-logic, and thus only need to provide the handler -to process the message. Note that the prefixes @code{check\_} -and @code{handle\_} are mandatory. - -@example -@verbatiminclude examples/006.c -@end example - -@noindent -Exercise: Expand your helper function to receive a response message -(for example, containing just the @code{struct GNUnet MessageHeader} -without any payload). Upon receiving the service's response, you -should call a callback provided to your helper function's API. - -Exercise: Figure out where you can pass values to the -closures (@code{cls}). - -@node Writing a user interface -@subsection Writing a user interface - -Given a client library, all it takes to access a service now is to -combine calls to the client library with parsing command-line -options. - -Exercise: Call your client API from your @code{run()} method in your -client application to send a request to the service. For example, -send a 32-bit integer value based on a number given at the -command-line to the service. - -@node Writing a Service -@section Writing a Service - -Before you can test the client you've written so far, you'll -need to also implement the corresponding service. - -@menu -* Code Placement:: -* Starting a Service:: -@end menu - -@node Code Placement -@subsection Code Placement - -New services are placed in their own subdirectory under -@file{gnunet/src}. This subdirectory should contain the API -implementation file @file{SERVICE\_api.c}, the description of -the client-service protocol @file{SERVICE.h} and P2P protocol -@file{SERVICE\_protocol.h}, the implementation of the service itself -@file{gnunet-service-SERVICE.h} and several files for tests, -including test code and configuration files. - -@node Starting a Service -@subsection Starting a Service - -The key API definition for creating a service is the -@code{GNUNET\_SERVICE\_MAIN} macro: - -@example -@verbatiminclude examples/007.c -@end example - -@noindent -In addition to the service name and flags, the macro takes three -functions, typically called @code{run}, @code{client\_connect\_cb} and -@code{client\_disconnect\_cb} as well as an array of message handlers -that will be called for incoming messages from clients. - -A minimal version of the three central service funtions would look -like this: - -@example -@verbatiminclude examples/008.c -@end example - -@noindent -Exercise: Write a stub service that processes no messages at all -in your code. Create a default configuration for it, integrate it -with the build system and start the service from -@command{gnunet-service-arm} using @command{gnunet-arm -i NAME}. - -Exercise: Figure out how to set the closure (@code{cls}) for handlers -of a service. - -Exercise: Figure out how to send messages from the service back to the -client. - -Each handler function in the service @b{must} eventually (possibly in some -asynchronous continuation) call -@code{GNUNET\_SERVICE\_client\_continue()}. Only after this call -additional messages from the same client may -be processed. This way, the service can throttle processing messages -from the same client. - -Exercise: Change the service to ``handle'' the message from your -client (for now, by printing a message). What happens if you -forget to call @code{GNUNET\_SERVICE\_client\_continue()}? - -@node Interacting directly with other Peers using the CORE Service -@section Interacting directly with other Peers using the CORE Service - -FIXME: This section still needs to be updated to the lastest API! - -One of the most important services in GNUnet is the @code{CORE} service -managing connections between peers and handling encryption between peers. - -One of the first things any service that extends the P2P protocol -typically does is connect to the @code{CORE} service using: - -@example -@verbatiminclude examples/009.c -@end example - -@menu -* New P2P connections:: -* Receiving P2P Messages:: -* Sending P2P Messages:: -* End of P2P connections:: -@end menu - -@node New P2P connections -@subsection New P2P connections - -Before any traffic with a different peer can be exchanged, the peer must -be known to the service. This is notified by the @code{CORE} -@code{connects} callback, which communicates the identity of the new -peer to the service: - -@example -@verbatiminclude examples/010.c -@end example - -@noindent -Note that whatever you return from @code{connects} is given as the -@code{cls} argument to the message handlers for messages from -the respective peer. - -Exercise: Create a service that connects to the @code{CORE}. Then -start (and connect) two peers and print a message once your connect -callback is invoked. - -@node Receiving P2P Messages -@subsection Receiving P2P Messages - -To receive messages from @code{CORE}, you pass the desired -@code{handlers} to the @code{GNUNET\_CORE\_connect()} function, -just as we showed for services. - -It is your responsibility to process messages fast enough or -to implement flow control. If an application does not process -CORE messages fast enough, CORE will randomly drop messages -to not keep a very long queue in memory. - -Exercise: Start one peer with a new service that has a message -handler and start a second peer that only has your ``old'' service -without message handlers. Which ``connect'' handlers are invoked when -the two peers are connected? Why? - -@node Sending P2P Messages -@subsection Sending P2P Messages - -You can transmit messages to other peers using the @code{mq} you were -given during the @code{connect} callback. Note that the @code{mq} -automatically is released upon @code{disconnect} and that you must -not use it afterwards. - -It is your responsibility to not over-fill the message queue, GNUnet -will send the messages roughly in the order given as soon as possible. - -Exercise: Write a service that upon connect sends messages as -fast as possible to the other peer (the other peer should run a -service that ``processes'' those messages). How fast is the -transmission? Count using the STATISTICS service on both ends. Are -messages lost? How can you transmit messages faster? What happens if -you stop the peer that is receiving your messages? - -@node End of P2P connections -@subsection End of P2P connections - -If a message handler returns @code{GNUNET\_SYSERR}, the remote -peer shuts down or there is an unrecoverable network -disconnection, CORE notifies the service that the peer disconnected. -After this notification no more messages will be received from the -peer and the service is no longer allowed to send messages to the peer. -The disconnect callback looks like the following: - -@example -@verbatiminclude examples/011.c -@end example - -@noindent -Exercise: Fix your service to handle peer disconnects. - -@node Storing peer-specific data using the PEERSTORE service -@section Storing peer-specific data using the PEERSTORE service - -GNUnet's PEERSTORE service offers a persistorage for arbitrary -peer-specific data. Other GNUnet services can use the PEERSTORE -to store, retrieve and monitor data records. Each data record -stored with PEERSTORE contains the following fields: - -@itemize -@item subsystem: Name of the subsystem responsible for the record. -@item peerid: Identity of the peer this record is related to. -@item key: a key string identifying the record. -@item value: binary record value. -@item expiry: record expiry date. -@end itemize - -The first step is to start a connection to the PEERSTORE service: -@example -@verbatiminclude examples/012.c -@end example - -The service handle @code{peerstore_handle} will be needed for -all subsequent PEERSTORE operations. - -@menu -* Storing records:: -* Retrieving records:: -* Monitoring records:: -* Disconnecting from PEERSTORE:: -@end menu - -@node Storing records -@subsection Storing records - -To store a new record, use the following function: - -@example -@verbatiminclude examples/013.c -@end example - -@noindent -The @code{options} parameter can either be -@code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple -values can be stored under the same key combination -(subsystem, peerid, key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE} -which means that PEERSTORE will replace any existing values under the -given key combination (subsystem, peerid, key) with the new given value. - -The continuation function @code{cont} will be called after the store -request is successfully sent to the PEERSTORE service. This does not -guarantee that the record is successfully stored, only that it was -received by the service. - -The @code{GNUNET_PEERSTORE_store} function returns a handle to the store -operation. This handle can be used to cancel the store operation only -before the continuation function is called: - -@example -@verbatiminclude examples/013.1.c -@end example - -@node Retrieving records -@subsection Retrieving records - -To retrieve stored records, use the following function: - -@example -@verbatiminclude examples/014.c -@end example - -@noindent -The values of @code{peer} and @code{key} can be @code{NULL}. This -allows the iteration over values stored under any of the following -key combinations: - -@itemize -@item (subsystem) -@item (subsystem, peerid) -@item (subsystem, key) -@item (subsystem, peerid, key) -@end itemize - -The @code{callback} function will be called once with each retrieved -record and once more with a @code{NULL} record to signal the end of -results. - -The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the -iterate operation. This handle can be used to cancel the iterate -operation only before the callback function is called with a -@code{NULL} record. - -@node Monitoring records -@subsection Monitoring records - -PEERSTORE offers the functionality of monitoring for new records -stored under a specific key combination (subsystem, peerid, key). -To start the monitoring, use the following function: - -@example -@verbatiminclude examples/015.c -@end example - -@noindent -Whenever a new record is stored under the given key combination, -the @code{callback} function will be called with this new -record. This will continue until the connection to the PEERSTORE -service is broken or the watch operation is canceled: - -@example -@verbatiminclude examples/016.c -@end example - -@node Disconnecting from PEERSTORE -@subsection Disconnecting from PEERSTORE - -When the connection to the PEERSTORE service is no longer needed, -disconnect using the following function: - -@example -@verbatiminclude examples/017.c -@end example - -@noindent -If the @code{sync_first} flag is set to @code{GNUNET_YES}, -the API will delay the disconnection until all store requests -are received by the PEERSTORE service. Otherwise, it will -disconnect immediately. - -@node Using the DHT -@section Using the DHT - -The DHT allows to store data so other peers in the P2P network can -access it and retrieve data stored by any peers in the network. -This section will explain how to use the DHT. Of course, the first -thing to do is to connect to the DHT service: - -@example -@verbatiminclude examples/018.c -@end example - -@noindent -The second parameter indicates how many requests in parallel to expect. -It is not a hard limit, but a good approximation will make the DHT more -efficient. - -@menu -* Storing data in the DHT:: -* Obtaining data from the DHT:: -* Implementing a block plugin:: -* Monitoring the DHT:: -@end menu - -@node Storing data in the DHT -@subsection Storing data in the DHT -Since the DHT is a dynamic environment (peers join and leave frequently) -the data that we put in the DHT does not stay there indefinitely. It is -important to ``refresh'' the data periodically by simply storing it -again, in order to make sure other peers can access it. - -The put API call offers a callback to signal that the PUT request has been -sent. This does not guarantee that the data is accessible to others peers, -or even that is has been stored, only that the service has requested to -a neighboring peer the retransmission of the PUT request towards its final -destination. Currently there is no feedback about whether or not the data -has been sucessfully stored or where it has been stored. In order to -improve the availablilty of the data and to compensate for possible -errors, peers leaving and other unfavorable events, just make several -PUT requests! - -@example -@verbatiminclude examples/019.c -@end example - -@noindent -Exercise: Store a value in the DHT periodically to make sure it -is available over time. You might consider using the function -@code{GNUNET\_SCHEDULER\_add\_delayed} and call -@code{GNUNET\_DHT\_put} from inside a helper function. - -@node Obtaining data from the DHT -@subsection Obtaining data from the DHT - -As we saw in the previous example, the DHT works in an asynchronous mode. -Each request to the DHT is executed ``in the background'' and the API -calls return immediately. In order to receive results from the DHT, the -API provides a callback. Once started, the request runs in the service, -the service will try to get as many results as possible (filtering out -duplicates) until the timeout expires or we explicitly stop the request. -It is possible to give a ``forever'' timeout with -@code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}. - -If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE} -the callback will get a list of all the peers the data has travelled, -both on the PUT path and on the GET path. - -@example -@verbatiminclude examples/020.c -@end example - -@noindent -Exercise: Store a value in the DHT and after a while retrieve it. -Show the IDs of all the peers the requests have gone through. -In order to convert a peer ID to a string, use the function -@code{GNUNET\_i2s}. Pay attention to the route option parameters -in both calls! - -@node Implementing a block plugin -@subsection Implementing a block plugin - -In order to store data in the DHT, it is necessary to provide a block -plugin. The DHT uses the block plugin to ensure that only well-formed -requests and replies are transmitted over the network. - -The block plugin should be put in a file @file{plugin\_block\_SERVICE.c} -in the service's respective directory. The -mandatory functions that need to be implemented for a block plugin are -described in the following sections. - -@menu -* Validating requests and replies:: -* Deriving a key from a reply:: -* Initialization of the plugin:: -* Shutdown of the plugin:: -* Integration of the plugin with the build system:: -@end menu - -@node Validating requests and replies -@subsubsection Validating requests and replies - -The evaluate function should validate a reply or a request. It returns -a @code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All -possible answers are in @file{gnunet\_block\_lib.h}. The function will -be called with a @code{reply\_block} argument of @code{NULL} for -requests. Note that depending on how @code{evaluate} is called, only -some of the possible return values are valid. The specific meaning of -the @code{xquery} argument is application-specific. Applications that -do not use an extended query should check that the @code{xquery\_size} -is zero. The block group is typically used to filter duplicate -replies. - -@example -@verbatiminclude examples/021.c -@end example - -@noindent -Note that it is mandatory to detect duplicate replies in this function -and return the respective status code. Duplicate detection is -typically done using the Bloom filter block group provided by -@file{libgnunetblockgroup.so}. Failure to do so may cause replies to -circle in the network. - -@node Deriving a key from a reply -@subsubsection Deriving a key from a reply - -The DHT can operate more efficiently if it is possible to derive a key -from the value of the corresponding block. The @code{get\_key} -function is used to obtain the key of a block --- for example, by -means of hashing. If deriving the key is not possible, the function -should simply return @code{GNUNET\_SYSERR} (the DHT will still work -just fine with such blocks). - -@example -@verbatiminclude examples/022.c -@end example - -@node Initialization of the plugin -@subsubsection Initialization of the plugin - -The plugin is realized as a shared C library. The library must export -an initialization function which should initialize the plugin. The -initialization function specifies what block types the plugin cares -about and returns a struct with the functions that are to be used for -validation and obtaining keys (the ones just defined above). - -@example -@verbatiminclude examples/023.c -@end example - -@node Shutdown of the plugin -@subsubsection Shutdown of the plugin - -Following GNUnet's general plugin API concept, the plugin must -export a second function for cleaning up. It usually does very -little. - -@example -@verbatiminclude examples/024.c -@end example - -@node Integration of the plugin with the build system -@subsubsection Integration of the plugin with the build system - -In order to compile the plugin, the @file{Makefile.am} file for the -service SERVICE should contain a rule similar to this: -@c Actually this is a Makefile not C. But the whole structure of examples -@c must be improved. - -@example -@verbatiminclude examples/025.Makefile.am -@end example - -@noindent -Exercise: Write a block plugin that accepts all queries -and all replies but prints information about queries and replies -when the respective validation hooks are called. - -@node Monitoring the DHT -@subsection Monitoring the DHT - -It is possible to monitor the functioning of the local -DHT service. When monitoring the DHT, the service will -alert the monitoring program of any events, both started -locally or received for routing from another peer. -The are three different types of events possible: a -GET request, a PUT request or a response (a reply to a GET). - -Since the different events have different associated data, -the API gets 3 different callbacks (one for each message type) -and optional type and key parameters, to allow for filtering of -messages. When an event happens, the appropiate callback is -called with all the information about the event. - -@example -@verbatiminclude examples/026.c -@end example - -@node Debugging with gnunet-arm -@section Debugging with gnunet-arm - -Even if services are managed by @command{gnunet-arm}, you can -start them with @command{gdb} or @command{valgrind}. For -example, you could add the following lines to your -configuration file to start the DHT service in a @command{gdb} -session in a fresh @command{xterm}: - -@example -[dht] -PREFIX=xterm -e gdb --args -@end example - -@noindent -Alternatively, you can stop a service that was started via -ARM and run it manually: - -@example -$ gnunet-arm -k dht -$ gdb --args gnunet-service-dht -L DEBUG -$ valgrind gnunet-service-dht -L DEBUG -@end example - -@noindent -Assuming other services are well-written, they will automatically -re-integrate the restarted service with the peer. - -GNUnet provides a powerful logging mechanism providing log -levels @code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}. -The current log level is configured using the @code{$GNUNET_FORCE_LOG} -environmental variable. The @code{DEBUG} level is only available if -@command{--enable-logging=verbose} was used when running -@command{configure}. More details about logging can be found under -@uref{https://docs.gnunet.org/#Logging}. - -You should also probably enable the creation of core files, by setting -@code{ulimit}, and echo'ing @code{1} into -@file{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the -core dumps with @command{gdb}, which is often the fastest method to -find simple errors. - -Exercise: Add a memory leak to your service and obtain a trace -pointing to the leak using @command{valgrind} while running the service -from @command{gnunet-service-arm}. - - -@c ********************************************************************* -@node GNU Free Documentation License -@appendix GNU Free Documentation License -@cindex license, GNU Free Documentation License -@include fdl-1.3.texi - - -@bye diff --git a/doc/tutorial/gnunet_tutorial.texi b/doc/tutorial/gnunet_tutorial.texi new file mode 100644 index 000000000..6982b4200 --- /dev/null +++ b/doc/tutorial/gnunet_tutorial.texi @@ -0,0 +1,1568 @@ +\input texinfo + +@setfilename gnunet-tutorial.info +@documentencoding UTF-8 +@settitle GNUnet Tutorial +@c @exampleindent 2 + + +@include version.texi + +@copying +Copyright @copyright{} 2001-2018 GNUnet e.V. + +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with no +Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A +copy of the license is included in the section entitled ``GNU Free +Documentation License''. + +A copy of the license is also available from the Free Software +Foundation Web site at @url{http://www.gnu.org/licenses/fdl.html}. + +Alternately, this document is also available under the General +Public License, version 3 or later, as published by the Free Software +Foundation. A copy of the license is included in the section entitled +``GNU General Public License''. + +A copy of the license is also available from the Free Software +Foundation Web site at @url{http://www.gnu.org/licenses/gpl.html}. +@end copying + +@dircategory Tutorial +@direntry +* GNUnet-C-Tutorial: (gnunet-c-tutorial). C Tutorial for GNunet +@end direntry + + +@titlepage +@title GNUnet C Tutorial +@subtitle A Tutorial for GNUnet @value{VERSION} (C version) +@author The GNUnet Developers + +@page +@vskip 0pt plus 1filll + +@insertcopying +@end titlepage + +@contents + +@c **** TODO +@c 1. Update content? +@c 2. Either reference main documentation or +@c 3. Merge this into main documentation + +@node Top +@top Introduction + +This tutorials explains how to install GNUnet on a +GNU/Linux system and gives an introduction on how +GNUnet can be used to develop a Peer-to-Peer application. +Detailed installation instructions for +various operating systems and a detailed list of all +dependencies can be found on our website at +@uref{https://docs.gnunet.org/#Installation} and in our +Reference Documentation (GNUnet Handbook). + +Please read this tutorial carefully since every single step is +important, and do not hesitate to contact the GNUnet team if you have +any questions or problems! Visit this link in your webbrowser to learn +how to contact the GNUnet team: +@uref{https://gnunet.org/en/contact.html} + +@menu + +* Installing GNUnet:: Installing GNUnet +* Introduction to GNUnet Architecture:: Introduction to GNUnet Architecture +* First Steps with GNUnet:: First Steps with GNUnet +* Developing Applications:: Developing Applications +* GNU Free Documentation License:: The license of this manual + +@detailmenu + --- The Detailed Node Listing --- + +Installing GNUnet + +* Obtaining a stable version:: +* Installing Build Tool Chain and Dependencies:: +* Obtaining the latest version from Git:: +* Compiling and Installing GNUnet:: +* Common Issues - Check your GNUnet installation:: + +Introduction to GNUnet Architecture + +First Steps with GNUnet + +* Configure your peer:: +* Start a peer:: +* Monitor a peer:: +* Starting Two Peers by Hand:: +* Starting Peers Using the Testbed Service:: + +Developing Applications + +* gnunet-ext:: +* Adapting the Template:: +* Writing a Client Application:: +* Writing a Service:: +* Interacting directly with other Peers using the CORE Service:: +* Storing peer-specific data using the PEERSTORE service:: +* Using the DHT:: +* Debugging with gnunet-arm:: + +@end detailmenu +@end menu + +@node Installing GNUnet +@chapter Installing GNUnet + +First of all you have to install a current version of GNUnet. +You can download a tarball of a stable version from GNU FTP mirrors +or obtain the latest development version from our Git repository. + +Most of the time you should prefer to download the stable version +since with the latest development version things can be broken, +functionality can be changed or tests can fail. You should only use +the development version if you know that you require a certain +feature or a certain issue has been fixed since the last release. + +@menu +* Obtaining a stable version:: +* Installing Build Tool Chain and Dependencies:: +* Obtaining the latest version from Git:: +* Compiling and Installing GNUnet:: +* Common Issues - Check your GNUnet installation:: +@end menu + +@node Obtaining a stable version +@section Obtaining a stable version + +Download the tarball from +@indicateurl{https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz}. + +Make sure to download the associated @file{.sig} file and to verify the +authenticity of the tarball against it, like this: + +@example +$ wget https://ftp.gnu.org/gnu/gnunet/gnunet-@value{VERSION}.tar.gz.sig +$ gpg --verify-files gnunet-@value{VERSION}.tar.gz.sig +@end example + +@noindent +If this command fails because you do not have the required public key, +then you need to run the following command to import it: + +@example +$ gpg --keyserver keys.gnupg.net --recv-keys 48426C7E +@end example + +@noindent +and rerun the @code{gpg --verify-files} command. + +@b{Note:}@ +@b{The pub key to sign the 0.10.1 release has been +revoked}. You will get an error message stating that +@b{there is no known public key or that it has been revoked}. +The next release of GNUnet will have a valid signature +again. We are sorry for the inconvenience this causes. +Another possible source you could use is our +"gnunet" git repository which, since the change from SVN to git in 2016, +has mandatory signed commits by every developer. + +After verifying the signature you can extract the tarball. +The resulting directory will be renamed to @file{gnunet}, which we will +be using in the remainder of this document to refer to the +root of the source directory. + +@example +$ tar xvzf gnunet-@value{VERSION}.tar.gz +$ mv gnunet-@value{VERSION} gnunet +@end example + +@c FIXME: This can be irritating for the reader - First we say git should +@c be avoid unless it is really required, and then we write this +@c paragraph: +@noindent +However, please note that stable versions can be very outdated. +As a developer you are @b{strongly} encouraged to use the version +from @uref{https://git.gnunet.org/, the git server}. + +@node Installing Build Tool Chain and Dependencies +@section Installing Build Tool Chain and Dependencies + +To successfully compile GNUnet, you need the tools to build GNUnet and +the required dependencies. Please take a look at the +GNUnet Reference Documentation +(@pxref{Dependencies, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) +for a list of required dependencies and +(@pxref{Generic installation instructions, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}) +read its Installation chapter for specific instructions for +your Operating System. +Please check the notes at the end of the configure process about +required dependencies. + +For GNUnet bootstrapping support and the HTTP(S) plugin you should +install @uref{https://gnunet.org/en/gnurl.html, libgnurl}. +For the filesharing service you should install at least one of the +datastore backends (MySQL, SQlite and PostgreSQL are supported). + +@node Obtaining the latest version from Git +@section Obtaining the latest version from Git + +The latest development version can be obtained from our Git repository. +To get the code you need to have @code{Git} installed. Usually your +Operating System package manager should provide a suitable distribution +of git (otherwise check out Guix or Nix). If you are using an Operating +System based on Debian's apt: + +@example +$ sudo apt-get install git +@end example + +This is required for obtaining the repository, which is achieved with +the following command: + +@example +$ git clone https://git.gnunet.org/gnunet +@end example + +@noindent +After cloning the repository, you have to execute the @file{bootstrap} +script in the new directory: + +@example +$ cd gnunet +$ ./bootstrap +@end example + +@noindent +The remainder of this tutorial will assume that you have the +Git branch ``master'' checked out. + +@node Compiling and Installing GNUnet +@section Compiling and Installing GNUnet + +Note: This section is a duplication of the more in depth +@pxref{GNUnet Installation Handbook, The GNUnet Reference Documentation,, gnunet, The GNUnet Reference Documentation}. + +First, you need to install libgnupgerror 1.27 (or later) and +libgcrypt 1.7.6 (or later): + +@example +$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" +$ wget $GNUPGFTP/libgpg-error/libgpg-error-1.27.tar.bz2 +$ tar xf libgpg-error-1.27.tar.bz2 +$ cd libgpg-error-1.27 +$ ./configure +$ make +$ sudo make install +$ cd .. +@end example + +@example +$ export GNUPGFTP="https://www.gnupg.org/ftp/gcrypt" +$ wget $GNUPGFTP/libgcrypt/libgcrypt-1.7.6.tar.bz2 +$ tar xf libgcrypt-1.7.6.tar.bz2 +$ cd libgcrypt-1.7.6 +$ ./configure +$ make +$ sudo make install +$ cd .. +@end example + +@menu +* Installation:: +@end menu + +@node Installation +@subsection Installation +Assuming all dependencies are installed, the following commands will +compile and install GNUnet in your home directory. You can specify the +directory where GNUnet will be installed by changing the +@code{--prefix} value when calling @command{./configure}. If +you do not specifiy a prefix, GNUnet is installed in the directory +@file{/usr/local}. When developing new applications you may want +to enable verbose logging by adding @code{--enable-logging=verbose}: + +@example +$ export PREFIX=$HOME +$ ./configure --prefix=$PREFIX --enable-logging +$ make +$ make install +@end example + +@noindent +After installing GNUnet you have to add your GNUnet installation +to your path environmental variable. In addition you have to +create the @file{.config} directory in your home directory +(unless it already exists) where GNUnet stores its data and an +empty GNUnet configuration file: + +@example +$ export PATH=$PATH:$PREFIX/bin +$ echo export PATH=$PREFIX/bin:\\$PATH >> ~/.bashrc +$ mkdir ~/.config/ +$ touch ~/.config/gnunet.conf +@end example + +@node Common Issues - Check your GNUnet installation +@section Common Issues - Check your GNUnet installation + +You should check your installation to ensure that installing GNUnet +was successful up to this point. You should be able to access GNUnet's +binaries and run GNUnet's self check. + +@example +$ which gnunet-arm +$PREFIX/bin/gnunet-arm +@end example + +@noindent +should return $PREFIX/bin/gnunet-arm (where $PREFIX is the location +you have set earlier). It should be located in your +GNUnet installation and the output should not be empty. + +If you see an output like: + +@example +$ which gnunet-arm +@end example + +@noindent +check your PATH variable to ensure GNUnet's @file{bin} directory is +included. + +GNUnet provides tests for all of its subcomponents. Assuming you have +successfully built GNUnet, run + +@example +$ cd gnunet +$ make check +@end example + +@noindent +to execute tests for all components. @command{make check} traverses all +subdirectories in @file{src}. For every subdirectory you should +get a message like this: + +@example +make[2]: Entering directory `/home/$USER/gnunet/contrib' +PASS: test_gnunet_prefix +============= +1 test passed +============= +@end example + +@node Introduction to GNUnet Architecture +@chapter Introduction to GNUnet Architecture + +GNUnet is organized in layers and services. Each service is composed of a +main service implementation and a client library for other programs to use +the service's functionality, described by an API. +@c This approach is shown in +@c FIXME: enable this once the commented block below works: +@c figure~\ref fig:service. +Some services provide an additional command line tool to enable the user +to interact with the service. + +Very often it is other GNUnet services that will use these APIs to build +the higher layers of GNUnet on top of the lower ones. Each layer expands +or extends the functionality of the service below (for instance, to build +a mesh on top of a DHT). +@c FXIME: See comment above. +@c See figure ~\ref fig:interaction for an illustration of this approach. + +@c ** @image filename[, width[, height[, alttext[, extension]]]] +@c FIXME: Texlive (?) 20112 makes the assumption that this means +@c 'images/OBJECTNAME.txt' but later versions of it (2017) use this +@c syntax as described below. +@c TODO: Checkout the makedoc script Guile uses. + +@c FIXME!!! +@c @image{images/gnunet-tutorial-service,,5in,Service with API and network protocol,.png} +@c @image{images/gnunet-tutorial-system,,5in,The layered system architecture of GNUnet,.png} + +@c \begin{figure}[!h] +@c \begin{center} +@c % \begin{subfigure} +@c \begin{subfigure}[b]{0.3\textwidth} +@c \centering +@c \includegraphics[width=\textwidth]{figs/Service.pdf} +@c \caption{Service with API and network protocol} +@c \label{fig:service} +@c \end{subfigure} +@c ~~~~~~~~~~ +@c \begin{subfigure}[b]{0.3\textwidth} +@c \centering +@c \includegraphics[width=\textwidth]{figs/System.pdf} +@c \caption{Service interaction} +@c \label{fig:interaction} +@c \end{subfigure} +@c \end{center} +@c \caption{GNUnet's layered system architecture} +@c \end{figure} + +The main service implementation runs as a standalone process in the +Operating System and the client code runs as part of the client program, +so crashes of a client do not affect the service process or other clients. +The service and the clients communicate via a message protocol to be +defined and implemented by the programmer. + +@node First Steps with GNUnet +@chapter First Steps with GNUnet + +@menu +* Configure your peer:: +* Start a peer:: +* Monitor a peer:: +* Starting Two Peers by Hand:: +* Starting Peers Using the Testbed Service:: +@end menu + +@node Configure your peer +@section Configure your peer + +First of all we need to configure your peer. Each peer is started with +a configuration containing settings for GNUnet itself and its services. +This configuration is based on the default configuration shipped with +GNUnet and can be modified. The default configuration is located in the +@file{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you +can specify a customized configuration using the the @command{-c} command +line switch when starting the ARM service and all other services. When +using a modified configuration the default values are loaded and only +values specified in the configuration file will replace the default +values. + +Since we want to start additional peers later, we need some modifications +from the default configuration. We need to create a separate service +home and a file containing our modifications for this peer: + +@example +$ mkdir ~/gnunet1/ +$ touch peer1.conf +@end example + +@noindent +Now add the following lines to @file{peer1.conf} to use this directory. +For simplified usage we want to prevent the peer to connect to the GNUnet +network since this could lead to confusing output. This modifications +will replace the default settings: + +@example +[PATHS] +# Use this directory to store GNUnet data +GNUNET_HOME = ~/gnunet1/ +[hostlist] +# prevent bootstrapping +SERVERS = +@end example + +@node Start a peer +@section Start a peer +Each GNUnet instance (called peer) has an identity (peer ID) based on a +cryptographic public private key pair. The peer ID is the printable hash +of the public key. + +GNUnet services are controlled by a master service, the so called +@dfn{Automatic Restart Manager} (ARM). ARM starts, stops and even +restarts services automatically or on demand when a client connects. +You interact with the ARM service using the @command{gnunet-arm} tool. +GNUnet can then be started with @command{gnunet-arm -s} and stopped with +@command{gnunet-arm -e}. An additional service not automatically started +can be started using @command{gnunet-arm -i } and stopped +using @command{gnunet-arm -k }. + +Once you have started your peer, you can use many other GNUnet commands +to interact with it. For example, you can run: + +@example +$ gnunet-peerinfo -s +@end example + +@noindent +to obtain the public key of your peer. + +You should see an output containing the peer ID similar to: + +@example +I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'. +@end example + +@node Monitor a peer +@section Monitor a peer + +In this section, we will monitor the behaviour of our peer's DHT +service with respect to a specific key. First we will start +GNUnet and then start the DHT service and use the DHT monitor tool +to monitor the PUT and GET commands we issue ussing the +@command{gnunet-dht-put} and @command{gnunet-dht-get} commands. +Using the ``monitor'' line given below, you can observe the behavior +of your own peer's DHT with respect to the specified KEY: + +@example +# start gnunet with all default services: +$ gnunet-arm -c ~/peer1.conf -s +# start DHT service: +$ gnunet-arm -c ~/peer1.conf -i dht +$ cd ~/gnunet/src/dht; +$ ./gnunet-dht-monitor -c ~/peer1.conf -k KEY +@end example + +@noindent +Now open a separate terminal and change again to +the @file{gnunet/src/dht} directory: + +@example +$ cd ~/gnunet/src/dht +# put VALUE under KEY in the DHT: +$ ./gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE +# get key KEY from the DHT: +$ ./gnunet/src/dht/gnunet-dht-get -c ~/peer1.conf -k KEY +# print statistics about current GNUnet state: +$ gnunet-statistics -c ~/peer1.conf +# print statistics about DHT service: +$ gnunet-statistics -c ~/peer1.conf -s dht +@end example + +@node Starting Two Peers by Hand +@section Starting Two Peers by Hand + +This section describes how to start two peers on the same machine by hand. +The process is rather painful, but the description is somewhat +instructive. In practice, you might prefer the automated method +(@pxref{Starting Peers Using the Testbed Service}). + +@menu +* Setup a second peer:: +* Start the second peer and connect the peers:: +* How to connect manually:: +@end menu + +@node Setup a second peer +@subsection Setup a second peer +We will now start a second peer on your machine. +For the second peer, you will need to manually create a modified +configuration file to avoid conflicts with ports and directories. +A peers configuration file is by default located +in @file{~/.gnunet/gnunet.conf}. This file is typically very short +or even empty as only the differences to the defaults need to be +specified. The defaults are located in many files in the +@file{$PREFIX/share/gnunet/config.d} directory. + +To configure the second peer, use the files +@file{$PREFIX/share/gnunet/config.d} as a template for your main +configuration file: + +@example +$ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf +@end example + +@noindent +Now you have to edit @file{peer2.conf} and change: + +@itemize +@item @code{GNUNET\_TEST\_HOME} under @code{PATHS} +@item Every (uncommented) value for ``@code{PORT}'' (add 10000) in any +section (the option may be commented out if @code{PORT} is +prefixed by "\#", in this case, UNIX domain sockets are used +and the PORT option does not need to be touched) +@item Every value for ``@code{UNIXPATH}'' in any section +(e.g. by adding a "-p2" suffix) +@end itemize + +to a fresh, unique value. Make sure that the PORT numbers stay +below 65536. From now on, whenever you interact with the second peer, +you need to specify @command{-c peer2.conf} as an additional +command line argument. + +Now, generate the 2nd peer's private key: + +@example +$ gnunet-peerinfo -s -c peer2.conf +@end example + +@noindent +This may take a while, generate entropy using your keyboard or mouse +as needed. Also, make sure the output is different from the +gnunet-peerinfo output for the first peer (otherwise you made an +error in the configuration). + +@node Start the second peer and connect the peers +@subsection Start the second peer and connect the peers + +Then, you can start a second peer using: + +@example +$ gnunet-arm -c peer2.conf -s +$ gnunet-arm -c peer2.conf -i dht +$ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE +$ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY +@end example + +If you want the two peers to connect, you have multiple options: + +@itemize +@item UDP neighbour discovery (automatic) +@item Setup a bootstrap server +@item Connect manually +@end itemize + +To setup peer 1 as bootstrapping server change the configuration of +the first one to be a hostlist server by adding the following lines to +@file{peer1.conf} to enable bootstrapping server: + +@example +[hostlist] +OPTIONS = -p +@end example + +@noindent +Then change @file{peer2.conf} and replace the ``@code{SERVERS}'' +line in the ``@code{[hostlist]}'' section with +``@code{http://localhost:8080/}''. Restart both peers using: + +@example +# stop first peer +$ gnunet-arm -c peer1.conf -e +# start first peer +$ gnunet-arm -c peer1.conf -s +# start second peer +$ gnunet-arm -c peer2.conf -s +@end example + +@noindent +Note that if you start your peers without changing these settings, they +will use the ``global'' hostlist servers of the GNUnet P2P network and +likely connect to those peers. At that point, debugging might become +tricky as you're going to be connected to many more peers and would +likely observe traffic and behaviors that are not explicitly controlled +by you. + +@node How to connect manually +@subsection How to connect manually + +If you want to use the @code{peerinfo} tool to connect your +peers, you should: + +@itemize +@item Set @code{IMMEDIATE_START = NO} in section @code{hostlist} +(to not connect to the global GNUnet) +@item Start both peers running @command{gnunet-arm -c peer1.conf -s} +and @command{gnunet-arm -c peer2.conf -s} +@item Get @code{HELLO} message of the first peer running +@command{gnunet-peerinfo -c peer1.conf -g} +@item Give the output to the second peer by running +@command{gnunet-peerinfo -c peer2.conf -p ''} +@end itemize + +Check that they are connected using @command{gnunet-core -c peer1.conf}, +which should give you the other peer's peer identity: + +@example +$ gnunet-core -c peer1.conf +Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG' +@end example + +@node Starting Peers Using the Testbed Service +@section Starting Peers Using the Testbed Service +@c \label{sec:testbed} + +GNUnet's testbed service is used for testing scenarios where +a number of peers are to be started. The testbed can manage peers +on a single host or on multiple hosts in a distributed fashion. +On a single affordable computer, it should be possible to run +around tens of peers without drastically increasing the load on the +system. + +The testbed service can be access through its API +@file{include/gnunet\_testbed\_service.h}. The API provides many +routines for managing a group of peers. It also provides a helper +function @code{GNUNET\_TESTBED\_test\_run()} to quickly setup a +minimalistic testing environment on a single host. + +This function takes a configuration file which will be used as a +template configuration for the peers. The testbed takes care of +modifying relevant options in the peers' configuration such as +@code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values +so that peers run without running into conflicts. It also checks +and assigns the ports in configurations only if they are free. + +Additionally, the testbed service also reads its options from the +same configuration file. Various available options and details +about them can be found in the testbed default configuration file +@file{src/testbed/testbed.conf}. + +With the testbed API, a sample test case can be structured as follows: + +@example +@verbatiminclude examples/testbed_test.c +@end example + +@noindent +The source code for the above listing can be found at +@c FIXME: This is not the correct URL. Where is the file? +@uref{https://git.gnunet.org/gnunet.git/tree/doc/documentation/testbed_test.c} +or in the @file{doc/} folder of your repository check-out. +After installing GNUnet, the above source code can be compiled as: + +@example +$ export CPPFLAGS="-I/path/to/gnunet/headers" +$ export LDFLAGS="-L/path/to/gnunet/libraries" +$ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \ + -lgnunettestbed -lgnunetdht -lgnunetutil +# Generate (empty) configuration +$ touch template.conf +# run it (press CTRL-C to stop) +$ ./testbed-test +@end example + +@noindent +The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet +is installed into a different directory other than @file{/usr/local}. + +All of testbed API's peer management functions treat management +actions as operations and return operation handles. It is expected +that the operations begin immediately, but they may get delayed (to +balance out load on the system). The program using the API then has +to take care of marking the operation as ``done'' so that its +associated resources can be freed immediately and other waiting +operations can be executed. Operations will be canceled if they are +marked as ``done'' before their completion. + +An operation is treated as completed when it succeeds or fails. +Completion of an operation is either conveyed as events through +@dfn{controller event callback} or through respective +@dfn{operation completion callbacks}. +In functions which support completion notification +through both controller event callback and operation +completion callback, first the controller event callback will be +called. If the operation is not marked as done in that callback +or if the callback is given as NULL when creating the operation, +the operation completion callback will be called. The API +documentation shows which event are to be expected in the +controller event notifications. It also documents any exceptional +behaviour. + +Once the peers are started, test cases often need to connect +some of the peers' services. Normally, opening a connect to +a peer's service requires the peer's configuration. While using +testbed, the testbed automatically generates per-peer configuration. +Accessing those configurations directly through file system is +discouraged as their locations are dynamically created and will be +different among various runs of testbed. To make access to these +configurations easy, testbed API provides the function +@code{GNUNET\_TESTBED\_service\_connect()}. This function fetches +the configuration of a given peer and calls the @dfn{Connect Adapter}. +In the example code, it is the @code{dht\_ca}. A connect adapter is +expected to open the connection to the needed service by using the +provided configuration and return the created service connection handle. +Successful connection to the needed service is signaled through +@code{service\_connect\_comp\_cb}. + +A dual to connect adapter is the @dfn{Disconnect Adapter}. This callback +is called after the connect adapter has been called when the operation +from @code{GNUNET\_TESTBED\_service\_connect()} is marked as ``done''. +It has to disconnect from the service with the provided service +handle (@code{op\_result}). + +Exercise: Find out how many peers you can run on your system. + +Exercise: Find out how to create a 2D torus topology by changing the +options in the configuration file. +@xref{Supported Topologies, The GNUnet Reference Documentation ,, gnunet, The GNUnet Reference Documentation}, +then use the DHT API to store and retrieve values in the network. + +@node Developing Applications +@chapter Developing Applications + +@menu +* gnunet-ext:: +* Adapting the Template:: +* Writing a Client Application:: +* Writing a Service:: +* Interacting directly with other Peers using the CORE Service:: +* Storing peer-specific data using the PEERSTORE service:: +* Using the DHT:: +* Debugging with gnunet-arm:: +@end menu + +@node gnunet-ext +@section gnunet-ext +To develop a new peer-to-peer application or to extend GNUnet we provide +a template build system for writing GNUnet extensions in C. It can be +obtained as follows: + +@example +$ git clone https://git.gnunet.org/gnunet-ext +$ cd gnunet-ext/ +$ ./bootstrap +$ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX +$ make +$ make install +$ make check +@end example + +@noindent +The GNUnet ext template includes examples and a working buildsystem +for a new GNUnet service. A common GNUnet service consists of the +following parts which will be discussed in detail in the remainder +of this document. The functionality of a GNUnet service is implemented in: + +@itemize +@item the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c) +@item the client API (gnunet-ext/src/ext/ext_api.c) +@item the client application using the service API +(gnunet-ext/src/ext/gnunet-ext.c) +@end itemize + +The interfaces for these entities are defined in: + +@itemize +@item client API interface (gnunet-ext/src/ext/ext.h) +@item the service interface (gnunet-ext/src/include/gnunet_service_SERVICE.h) +@item the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h) +@end itemize + + +In addition the ext systems provides: + +@itemize +@item a test testing the API (gnunet-ext/src/ext/test_ext_api.c) +@item a configuration template for the service +(gnunet-ext/src/ext/ext.conf.in) +@end itemize + +@node Adapting the Template +@section Adapting the Template + +The first step for writing any extension with a new service is to +ensure that the @file{ext.conf.in} file contains entries for the +@code{UNIXPATH}, @code{PORT} and @code{BINARY} for the service in a +section named after the service. + +If you want to adapt the template rename the @file{ext.conf.in} to +match your services name, you have to modify the @code{AC\_OUTPUT} +section in @file{configure.ac} in the @file{gnunet-ext} root. + +@node Writing a Client Application +@section Writing a Client Application + +When writing any client application (for example, a command-line +tool), the basic structure is to start with the +@code{GNUNET\_PROGRAM\_run} function. This function will parse +command-line options, setup the scheduler and then invoke the +@code{run} function (with the remaining non-option arguments) +and a handle to the parsed configuration (and the configuration +file name that was used, which is typically not needed): + +@example +@verbatiminclude examples/001.c +@end example + +@menu +* Handling command-line options:: +* Writing a Client Library:: +* Writing a user interface:: +@end menu + +@node Handling command-line options +@subsection Handling command-line options + +Options can then be added easily by adding global variables and +expanding the @code{options} array. For example, the following would +add a string-option and a binary flag (defaulting to @code{NULL} and +@code{GNUNET\_NO} respectively): + +@example +@verbatiminclude examples/002.c +@end example + +Issues such as displaying some helpful text describing options using +the @code{--help} argument and error handling are taken care of when +using this approach. Other @code{GNUNET\_GETOPT\_}-functions can be used +to obtain integer value options, increment counters, etc. You can +even write custom option parsers for special circumstances not covered +by the available handlers. To check if an argument was specified by the +user you initialize the variable with a specific value (e.g. NULL for +a string and GNUNET\_SYSERR for a integer) and check after parsing +happened if the values were modified. + +Inside the @code{run} method, the program would perform the +application-specific logic, which typically involves initializing and +using some client library to interact with the service. The client +library is supposed to implement the IPC whereas the service provides +more persistent P2P functions. + +Exercise: Add a few command-line options and print them inside +of @code{run}. What happens if the user gives invalid arguments? + +@node Writing a Client Library +@subsection Writing a Client Library + +The first and most important step in writing a client library is to +decide on an API for the library. Typical API calls include +connecting to the service, performing application-specific requests +and cleaning up. Many examples for such service APIs can be found +in the @file{gnunet/src/include/gnunet\_*\_service.h} files. + +Then, a client-service protocol needs to be designed. This typically +involves defining various message formats in a header that will be +included by both the service and the client library (but is otherwise +not shared and hence located within the service's directory and not +installed by @command{make install}). Each message must start with a +@code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By +convention, all fields in IPC (and P2P) messages must be in big-endian +format (and thus should be read using @code{ntohl} and similar +functions and written using @code{htonl} and similar functions). +Unique message types must be defined for each message struct in the +@file{gnunet\_protocols.h} header (or an extension-specific include +file). + +@menu +* Connecting to the Service:: +* Sending messages:: +* Receiving Replies from the Service:: +@end menu + +@node Connecting to the Service +@subsubsection Connecting to the Service + +Before a client library can implement the application-specific protocol +with the service, a connection must be created: + +@example +@verbatiminclude examples/003.c +@end example + +@noindent +As a result a @code{GNUNET\_MQ\_Handle} is returned +which can to used henceforth to transmit messages to the service. +The complete MQ API can be found in @file{gnunet\_mq\_lib.h}. +The @code{hanlders} array in the example above is incomplete. +Here is where you will define which messages you expect to +receive from the service, and which functions handle them. +The @code{error\_cb} is a function that is to be called whenever +there are errors communicating with the service. + +@node Sending messages +@subsubsection Sending messages + +In GNUnet, messages are always sent beginning with a +@code{struct GNUNET\_MessageHeader} in big endian format. +This header defines the size and the type of the +message, the payload follows after this header. + +@example +@verbatiminclude examples/004.c +@end example + +@noindent +Existing message types are defined in @file{gnunet\_protocols.h}. +A common way to create a message is with an envelope: + +@example +@verbatiminclude examples/005.c +@end example + +@noindent +Exercise: Define a message struct that includes a 32-bit +unsigned integer in addition to the standard GNUnet MessageHeader. +Add a C struct and define a fresh protocol number for your message. +Protocol numbers in gnunet-ext are defined +in @file{gnunet-ext/src/include/gnunet_protocols_ext.h} + +Exercise: Find out how you can determine the number of messages +in a message queue. + +Exercise: Find out how you can determine when a message you +have queued was actually transmitted. + +Exercise: Define a helper function to transmit a 32-bit +unsigned integer (as payload) to a service using some given client +handle. + +@node Receiving Replies from the Service +@subsubsection Receiving Replies from the Service + +Clients can receive messages from the service using the handlers +specified in the @code{handlers} array we specified when connecting +to the service. Entries in the the array are usually created using +one of two macros, depending on whether the message is fixed size +or variable size. Variable size messages are managed using two +callbacks, one to check that the message is well-formed, the other +to actually process the message. Fixed size messages are fully +checked by the MQ-logic, and thus only need to provide the handler +to process the message. Note that the prefixes @code{check\_} +and @code{handle\_} are mandatory. + +@example +@verbatiminclude examples/006.c +@end example + +@noindent +Exercise: Expand your helper function to receive a response message +(for example, containing just the @code{struct GNUnet MessageHeader} +without any payload). Upon receiving the service's response, you +should call a callback provided to your helper function's API. + +Exercise: Figure out where you can pass values to the +closures (@code{cls}). + +@node Writing a user interface +@subsection Writing a user interface + +Given a client library, all it takes to access a service now is to +combine calls to the client library with parsing command-line +options. + +Exercise: Call your client API from your @code{run()} method in your +client application to send a request to the service. For example, +send a 32-bit integer value based on a number given at the +command-line to the service. + +@node Writing a Service +@section Writing a Service + +Before you can test the client you've written so far, you'll +need to also implement the corresponding service. + +@menu +* Code Placement:: +* Starting a Service:: +@end menu + +@node Code Placement +@subsection Code Placement + +New services are placed in their own subdirectory under +@file{gnunet/src}. This subdirectory should contain the API +implementation file @file{SERVICE\_api.c}, the description of +the client-service protocol @file{SERVICE.h} and P2P protocol +@file{SERVICE\_protocol.h}, the implementation of the service itself +@file{gnunet-service-SERVICE.h} and several files for tests, +including test code and configuration files. + +@node Starting a Service +@subsection Starting a Service + +The key API definition for creating a service is the +@code{GNUNET\_SERVICE\_MAIN} macro: + +@example +@verbatiminclude examples/007.c +@end example + +@noindent +In addition to the service name and flags, the macro takes three +functions, typically called @code{run}, @code{client\_connect\_cb} and +@code{client\_disconnect\_cb} as well as an array of message handlers +that will be called for incoming messages from clients. + +A minimal version of the three central service funtions would look +like this: + +@example +@verbatiminclude examples/008.c +@end example + +@noindent +Exercise: Write a stub service that processes no messages at all +in your code. Create a default configuration for it, integrate it +with the build system and start the service from +@command{gnunet-service-arm} using @command{gnunet-arm -i NAME}. + +Exercise: Figure out how to set the closure (@code{cls}) for handlers +of a service. + +Exercise: Figure out how to send messages from the service back to the +client. + +Each handler function in the service @b{must} eventually (possibly in some +asynchronous continuation) call +@code{GNUNET\_SERVICE\_client\_continue()}. Only after this call +additional messages from the same client may +be processed. This way, the service can throttle processing messages +from the same client. + +Exercise: Change the service to ``handle'' the message from your +client (for now, by printing a message). What happens if you +forget to call @code{GNUNET\_SERVICE\_client\_continue()}? + +@node Interacting directly with other Peers using the CORE Service +@section Interacting directly with other Peers using the CORE Service + +FIXME: This section still needs to be updated to the lastest API! + +One of the most important services in GNUnet is the @code{CORE} service +managing connections between peers and handling encryption between peers. + +One of the first things any service that extends the P2P protocol +typically does is connect to the @code{CORE} service using: + +@example +@verbatiminclude examples/009.c +@end example + +@menu +* New P2P connections:: +* Receiving P2P Messages:: +* Sending P2P Messages:: +* End of P2P connections:: +@end menu + +@node New P2P connections +@subsection New P2P connections + +Before any traffic with a different peer can be exchanged, the peer must +be known to the service. This is notified by the @code{CORE} +@code{connects} callback, which communicates the identity of the new +peer to the service: + +@example +@verbatiminclude examples/010.c +@end example + +@noindent +Note that whatever you return from @code{connects} is given as the +@code{cls} argument to the message handlers for messages from +the respective peer. + +Exercise: Create a service that connects to the @code{CORE}. Then +start (and connect) two peers and print a message once your connect +callback is invoked. + +@node Receiving P2P Messages +@subsection Receiving P2P Messages + +To receive messages from @code{CORE}, you pass the desired +@code{handlers} to the @code{GNUNET\_CORE\_connect()} function, +just as we showed for services. + +It is your responsibility to process messages fast enough or +to implement flow control. If an application does not process +CORE messages fast enough, CORE will randomly drop messages +to not keep a very long queue in memory. + +Exercise: Start one peer with a new service that has a message +handler and start a second peer that only has your ``old'' service +without message handlers. Which ``connect'' handlers are invoked when +the two peers are connected? Why? + +@node Sending P2P Messages +@subsection Sending P2P Messages + +You can transmit messages to other peers using the @code{mq} you were +given during the @code{connect} callback. Note that the @code{mq} +automatically is released upon @code{disconnect} and that you must +not use it afterwards. + +It is your responsibility to not over-fill the message queue, GNUnet +will send the messages roughly in the order given as soon as possible. + +Exercise: Write a service that upon connect sends messages as +fast as possible to the other peer (the other peer should run a +service that ``processes'' those messages). How fast is the +transmission? Count using the STATISTICS service on both ends. Are +messages lost? How can you transmit messages faster? What happens if +you stop the peer that is receiving your messages? + +@node End of P2P connections +@subsection End of P2P connections + +If a message handler returns @code{GNUNET\_SYSERR}, the remote +peer shuts down or there is an unrecoverable network +disconnection, CORE notifies the service that the peer disconnected. +After this notification no more messages will be received from the +peer and the service is no longer allowed to send messages to the peer. +The disconnect callback looks like the following: + +@example +@verbatiminclude examples/011.c +@end example + +@noindent +Exercise: Fix your service to handle peer disconnects. + +@node Storing peer-specific data using the PEERSTORE service +@section Storing peer-specific data using the PEERSTORE service + +GNUnet's PEERSTORE service offers a persistorage for arbitrary +peer-specific data. Other GNUnet services can use the PEERSTORE +to store, retrieve and monitor data records. Each data record +stored with PEERSTORE contains the following fields: + +@itemize +@item subsystem: Name of the subsystem responsible for the record. +@item peerid: Identity of the peer this record is related to. +@item key: a key string identifying the record. +@item value: binary record value. +@item expiry: record expiry date. +@end itemize + +The first step is to start a connection to the PEERSTORE service: +@example +@verbatiminclude examples/012.c +@end example + +The service handle @code{peerstore_handle} will be needed for +all subsequent PEERSTORE operations. + +@menu +* Storing records:: +* Retrieving records:: +* Monitoring records:: +* Disconnecting from PEERSTORE:: +@end menu + +@node Storing records +@subsection Storing records + +To store a new record, use the following function: + +@example +@verbatiminclude examples/013.c +@end example + +@noindent +The @code{options} parameter can either be +@code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple +values can be stored under the same key combination +(subsystem, peerid, key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE} +which means that PEERSTORE will replace any existing values under the +given key combination (subsystem, peerid, key) with the new given value. + +The continuation function @code{cont} will be called after the store +request is successfully sent to the PEERSTORE service. This does not +guarantee that the record is successfully stored, only that it was +received by the service. + +The @code{GNUNET_PEERSTORE_store} function returns a handle to the store +operation. This handle can be used to cancel the store operation only +before the continuation function is called: + +@example +@verbatiminclude examples/013.1.c +@end example + +@node Retrieving records +@subsection Retrieving records + +To retrieve stored records, use the following function: + +@example +@verbatiminclude examples/014.c +@end example + +@noindent +The values of @code{peer} and @code{key} can be @code{NULL}. This +allows the iteration over values stored under any of the following +key combinations: + +@itemize +@item (subsystem) +@item (subsystem, peerid) +@item (subsystem, key) +@item (subsystem, peerid, key) +@end itemize + +The @code{callback} function will be called once with each retrieved +record and once more with a @code{NULL} record to signal the end of +results. + +The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the +iterate operation. This handle can be used to cancel the iterate +operation only before the callback function is called with a +@code{NULL} record. + +@node Monitoring records +@subsection Monitoring records + +PEERSTORE offers the functionality of monitoring for new records +stored under a specific key combination (subsystem, peerid, key). +To start the monitoring, use the following function: + +@example +@verbatiminclude examples/015.c +@end example + +@noindent +Whenever a new record is stored under the given key combination, +the @code{callback} function will be called with this new +record. This will continue until the connection to the PEERSTORE +service is broken or the watch operation is canceled: + +@example +@verbatiminclude examples/016.c +@end example + +@node Disconnecting from PEERSTORE +@subsection Disconnecting from PEERSTORE + +When the connection to the PEERSTORE service is no longer needed, +disconnect using the following function: + +@example +@verbatiminclude examples/017.c +@end example + +@noindent +If the @code{sync_first} flag is set to @code{GNUNET_YES}, +the API will delay the disconnection until all store requests +are received by the PEERSTORE service. Otherwise, it will +disconnect immediately. + +@node Using the DHT +@section Using the DHT + +The DHT allows to store data so other peers in the P2P network can +access it and retrieve data stored by any peers in the network. +This section will explain how to use the DHT. Of course, the first +thing to do is to connect to the DHT service: + +@example +@verbatiminclude examples/018.c +@end example + +@noindent +The second parameter indicates how many requests in parallel to expect. +It is not a hard limit, but a good approximation will make the DHT more +efficient. + +@menu +* Storing data in the DHT:: +* Obtaining data from the DHT:: +* Implementing a block plugin:: +* Monitoring the DHT:: +@end menu + +@node Storing data in the DHT +@subsection Storing data in the DHT +Since the DHT is a dynamic environment (peers join and leave frequently) +the data that we put in the DHT does not stay there indefinitely. It is +important to ``refresh'' the data periodically by simply storing it +again, in order to make sure other peers can access it. + +The put API call offers a callback to signal that the PUT request has been +sent. This does not guarantee that the data is accessible to others peers, +or even that is has been stored, only that the service has requested to +a neighboring peer the retransmission of the PUT request towards its final +destination. Currently there is no feedback about whether or not the data +has been sucessfully stored or where it has been stored. In order to +improve the availablilty of the data and to compensate for possible +errors, peers leaving and other unfavorable events, just make several +PUT requests! + +@example +@verbatiminclude examples/019.c +@end example + +@noindent +Exercise: Store a value in the DHT periodically to make sure it +is available over time. You might consider using the function +@code{GNUNET\_SCHEDULER\_add\_delayed} and call +@code{GNUNET\_DHT\_put} from inside a helper function. + +@node Obtaining data from the DHT +@subsection Obtaining data from the DHT + +As we saw in the previous example, the DHT works in an asynchronous mode. +Each request to the DHT is executed ``in the background'' and the API +calls return immediately. In order to receive results from the DHT, the +API provides a callback. Once started, the request runs in the service, +the service will try to get as many results as possible (filtering out +duplicates) until the timeout expires or we explicitly stop the request. +It is possible to give a ``forever'' timeout with +@code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}. + +If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE} +the callback will get a list of all the peers the data has travelled, +both on the PUT path and on the GET path. + +@example +@verbatiminclude examples/020.c +@end example + +@noindent +Exercise: Store a value in the DHT and after a while retrieve it. +Show the IDs of all the peers the requests have gone through. +In order to convert a peer ID to a string, use the function +@code{GNUNET\_i2s}. Pay attention to the route option parameters +in both calls! + +@node Implementing a block plugin +@subsection Implementing a block plugin + +In order to store data in the DHT, it is necessary to provide a block +plugin. The DHT uses the block plugin to ensure that only well-formed +requests and replies are transmitted over the network. + +The block plugin should be put in a file @file{plugin\_block\_SERVICE.c} +in the service's respective directory. The +mandatory functions that need to be implemented for a block plugin are +described in the following sections. + +@menu +* Validating requests and replies:: +* Deriving a key from a reply:: +* Initialization of the plugin:: +* Shutdown of the plugin:: +* Integration of the plugin with the build system:: +@end menu + +@node Validating requests and replies +@subsubsection Validating requests and replies + +The evaluate function should validate a reply or a request. It returns +a @code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All +possible answers are in @file{gnunet\_block\_lib.h}. The function will +be called with a @code{reply\_block} argument of @code{NULL} for +requests. Note that depending on how @code{evaluate} is called, only +some of the possible return values are valid. The specific meaning of +the @code{xquery} argument is application-specific. Applications that +do not use an extended query should check that the @code{xquery\_size} +is zero. The block group is typically used to filter duplicate +replies. + +@example +@verbatiminclude examples/021.c +@end example + +@noindent +Note that it is mandatory to detect duplicate replies in this function +and return the respective status code. Duplicate detection is +typically done using the Bloom filter block group provided by +@file{libgnunetblockgroup.so}. Failure to do so may cause replies to +circle in the network. + +@node Deriving a key from a reply +@subsubsection Deriving a key from a reply + +The DHT can operate more efficiently if it is possible to derive a key +from the value of the corresponding block. The @code{get\_key} +function is used to obtain the key of a block --- for example, by +means of hashing. If deriving the key is not possible, the function +should simply return @code{GNUNET\_SYSERR} (the DHT will still work +just fine with such blocks). + +@example +@verbatiminclude examples/022.c +@end example + +@node Initialization of the plugin +@subsubsection Initialization of the plugin + +The plugin is realized as a shared C library. The library must export +an initialization function which should initialize the plugin. The +initialization function specifies what block types the plugin cares +about and returns a struct with the functions that are to be used for +validation and obtaining keys (the ones just defined above). + +@example +@verbatiminclude examples/023.c +@end example + +@node Shutdown of the plugin +@subsubsection Shutdown of the plugin + +Following GNUnet's general plugin API concept, the plugin must +export a second function for cleaning up. It usually does very +little. + +@example +@verbatiminclude examples/024.c +@end example + +@node Integration of the plugin with the build system +@subsubsection Integration of the plugin with the build system + +In order to compile the plugin, the @file{Makefile.am} file for the +service SERVICE should contain a rule similar to this: +@c Actually this is a Makefile not C. But the whole structure of examples +@c must be improved. + +@example +@verbatiminclude examples/025.Makefile.am +@end example + +@noindent +Exercise: Write a block plugin that accepts all queries +and all replies but prints information about queries and replies +when the respective validation hooks are called. + +@node Monitoring the DHT +@subsection Monitoring the DHT + +It is possible to monitor the functioning of the local +DHT service. When monitoring the DHT, the service will +alert the monitoring program of any events, both started +locally or received for routing from another peer. +The are three different types of events possible: a +GET request, a PUT request or a response (a reply to a GET). + +Since the different events have different associated data, +the API gets 3 different callbacks (one for each message type) +and optional type and key parameters, to allow for filtering of +messages. When an event happens, the appropiate callback is +called with all the information about the event. + +@example +@verbatiminclude examples/026.c +@end example + +@node Debugging with gnunet-arm +@section Debugging with gnunet-arm + +Even if services are managed by @command{gnunet-arm}, you can +start them with @command{gdb} or @command{valgrind}. For +example, you could add the following lines to your +configuration file to start the DHT service in a @command{gdb} +session in a fresh @command{xterm}: + +@example +[dht] +PREFIX=xterm -e gdb --args +@end example + +@noindent +Alternatively, you can stop a service that was started via +ARM and run it manually: + +@example +$ gnunet-arm -k dht +$ gdb --args gnunet-service-dht -L DEBUG +$ valgrind gnunet-service-dht -L DEBUG +@end example + +@noindent +Assuming other services are well-written, they will automatically +re-integrate the restarted service with the peer. + +GNUnet provides a powerful logging mechanism providing log +levels @code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}. +The current log level is configured using the @code{$GNUNET_FORCE_LOG} +environmental variable. The @code{DEBUG} level is only available if +@command{--enable-logging=verbose} was used when running +@command{configure}. More details about logging can be found under +@uref{https://docs.gnunet.org/#Logging}. + +You should also probably enable the creation of core files, by setting +@code{ulimit}, and echo'ing @code{1} into +@file{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the +core dumps with @command{gdb}, which is often the fastest method to +find simple errors. + +Exercise: Add a memory leak to your service and obtain a trace +pointing to the leak using @command{valgrind} while running the service +from @command{gnunet-service-arm}. + + +@c ********************************************************************* +@node GNU Free Documentation License +@appendix GNU Free Documentation License +@cindex license, GNU Free Documentation License +@include fdl-1.3.texi + + +@bye -- cgit v1.2.3