@c *********************************************************************** @node GNUnet Developer Handbook @chapter GNUnet Developer Handbook This book is intended to be an introduction for programmers that want to extend the GNUnet framework. GNUnet is more than a simple peer-to-peer application. For developers, GNUnet is: @itemize @bullet @item developed by a community that believes in the GNU philosophy @item Free Software (Free as in Freedom), licensed under the GNU Affero General Public License (@uref{https://www.gnu.org/licenses/licenses.html#AGPL}) @item A set of standards, including coding conventions and architectural rules @item A set of layered protocols, both specifying the communication between peers as well as the communication between components of a single peer @item A set of libraries with well-defined APIs suitable for writing extensions @end itemize In particular, the architecture specifies that a peer consists of many processes communicating via protocols. Processes can be written in almost any language. @code{C}, @code{Java} and @code{Guile} APIs exist for accessing existing services and for writing extensions. It is possible to write extensions in other languages by implementing the necessary IPC protocols. GNUnet can be extended and improved along many possible dimensions, and anyone interested in Free Software and Freedom-enhancing Networking is welcome to join the effort. This Developer Handbook attempts to provide an initial introduction to some of the key design choices and central components of the system. This part of the GNUNet documentation is far from complete, and we welcome informed contributions, be it in the form of new chapters, sections or insightful comments. @menu * Developer Introduction:: * Internal dependencies:: * Code overview:: * System Architecture:: * Subsystem stability:: * Naming conventions and coding style guide:: * Build-system:: * Developing extensions for GNUnet using the gnunet-ext template:: * Writing testcases:: * Building GNUnet and its dependencies:: * TESTING library:: * Performance regression analysis with Gauger:: * TESTBED Subsystem:: * libgnunetutil:: * Automatic Restart Manager (ARM):: * TRANSPORT Subsystem:: * NAT library:: * Distance-Vector plugin:: * SMTP plugin:: * Bluetooth plugin:: * WLAN plugin:: * ATS Subsystem:: * CORE Subsystem:: * CADET Subsystem:: * NSE Subsystem:: * HOSTLIST Subsystem:: * IDENTITY Subsystem:: * NAMESTORE Subsystem:: * PEERINFO Subsystem:: * PEERSTORE Subsystem:: * SET Subsystem:: * STATISTICS Subsystem:: * Distributed Hash Table (DHT):: * GNU Name System (GNS):: * GNS Namecache:: * REVOCATION Subsystem:: * File-sharing (FS) Subsystem:: * REGEX Subsystem:: * REST Subsystem:: @end menu @node Developer Introduction @section Developer Introduction This Developer Handbook is intended as first introduction to GNUnet for new developers that want to extend the GNUnet framework. After the introduction, each of the GNUnet subsystems (directories in the @file{src/} tree) is (supposed to be) covered in its own chapter. In addition to this documentation, GNUnet developers should be aware of the services available on the GNUnet server to them. New developers can have a look a the GNUnet tutorials for C and java available in the @file{src/} directory of the repository or under the following links: @c ** FIXME: Link to files in source, not online. @c ** FIXME: Where is the Java tutorial? @itemize @bullet @item @xref{Top, Introduction,, gnunet-c-tutorial, The GNUnet C Tutorial}. @c broken link @c @item @uref{https://gnunet.org/git/gnunet.git/plain/doc/gnunet-c-tutorial.pdf, GNUnet C tutorial} @item GNUnet Java tutorial @end itemize In addition to the GNUnet Reference Documentation you are reading, the GNUnet server at @uref{https://gnunet.org} contains various resources for GNUnet developers and those who aspire to become regular contributors. They are all conveniently reachable via the "Developer" entry in the navigation menu. Some additional tools (such as static analysis reports) require a special developer access to perform certain operations. If you want (or require) access, you should contact @uref{http://grothoff.org/christian/, Christian Grothoff}, GNUnet's maintainer. @c FIXME: A good part of this belongs on the website or should be @c extended in subsections explaining usage of this. A simple list @c is just taking space people have to read. The public subsystems on the GNUnet server that help developers are: @itemize @bullet @item The version control system (git) keeps our code and enables distributed development. It is publicly accessible at @uref{https://gnunet.org/git/}. Only developers with write access can commit code, everyone else is encouraged to submit patches to the GNUnet-developers mailinglist: @uref{https://lists.gnu.org/mailman/listinfo/gnunet-developers, https://lists.gnu.org/mailman/listinfo/gnunet-developers} @item The bugtracking system (Mantis). We use it to track feature requests, open bug reports and their resolutions. It can be accessed at @uref{https://gnunet.org/bugs/, https://gnunet.org/bugs/}. Anyone can report bugs. @item Our site installation of the Continuous Integration (CI) system @code{Buildbot} is used to check GNUnet builds automatically on a range of platforms. The web interface of this CI is exposed at @uref{https://gnunet.org/buildbot/, https://gnunet.org/buildbot/}. Builds are triggered automatically 30 minutes after the last commit to our repository was made. @item The current quality of our automated test suite is assessed using Code coverage analysis. This analysis is run daily; however the webpage is only updated if all automated tests pass at that time. Testcases that improve our code coverage are always welcome. @item We try to automatically find bugs using a static analysis scan. This scan is run daily; however the webpage is only updated if all automated tests pass at the time. Note that not everything that is flagged by the analysis is a bug, sometimes even good code can be marked as possibly problematic. Nevertheless, developers are encouraged to at least be aware of all issues in their code that are listed. @item We use Gauger for automatic performance regression visualization. @c FIXME: LINK! Details on how to use Gauger are here. @item We use @uref{http://junit.org/, junit} to automatically test @command{gnunet-java}. Automatically generated, current reports on the test suite are here. @c FIXME: Likewise. @item We use Cobertura to generate test coverage reports for gnunet-java. Current reports on test coverage are here. @c FIXME: Likewise. @end itemize @c *********************************************************************** @menu * Project overview:: @end menu @node Project overview @subsection Project overview The GNUnet project consists at this point of several sub-projects. This section is supposed to give an initial overview about the various sub-projects. Note that this description also lists projects that are far from complete, including even those that have literally not a single line of code in them yet. GNUnet sub-projects in order of likely relevance are currently: @table @asis @item @command{gnunet} Core of the P2P framework, including file-sharing, VPN and chat applications; this is what the Developer Handbook covers mostly @item @command{gnunet-gtk} Gtk+-based user interfaces, including: @itemize @bullet @item @command{gnunet-fs-gtk} (file-sharing), @item @command{gnunet-statistics-gtk} (statistics over time), @item @command{gnunet-peerinfo-gtk} (information about current connections and known peers), @item @command{gnunet-chat-gtk} (chat GUI) and @item @command{gnunet-setup} (setup tool for "everything") @end itemize @item @command{gnunet-fuse} Mounting directories shared via GNUnet's file-sharing on GNU/Linux distributions @item @command{gnunet-update} Installation and update tool @item @command{gnunet-ext} Template for starting 'external' GNUnet projects @item @command{gnunet-java} Java APIs for writing GNUnet services and applications @item @command{gnunet-java-ext} @item @command{eclectic} Code to run GNUnet nodes on testbeds for research, development, testing and evaluation @c ** FIXME: Solve the status and location of gnunet-qt @item @command{gnunet-qt} Qt-based GNUnet GUI (is it deprecated?) @item @command{gnunet-cocoa} cocoa-based GNUnet GUI (is it deprecated?) @item @command{gnunet-guile} Guile bindings for GNUnet @item @command{gnunet-python} Python bindings for GNUnet @end table We are also working on various supporting libraries and tools: @c ** FIXME: What about gauger, and what about libmwmodem? @table @asis @item @command{libextractor} GNU libextractor (meta data extraction) @item @command{libmicrohttpd} GNU libmicrohttpd (embedded HTTP(S) server library) @item @command{gauger} Tool for performance regression analysis @item @command{monkey} Tool for automated debugging of distributed systems @item @command{libmwmodem} Library for accessing satellite connection quality reports @item @command{libgnurl} gnURL (feature-restricted variant of cURL/libcurl) @item @command{www} work in progress of the new gnunet.org website (Jinja2 framework based to replace our current Drupal website) @item @command{bibliography} Our collected bibliography, papers, references, and so forth @item @command{gnunet-videos-} Videos about and around gnunet activities @end table Finally, there are various external projects (see links for a list of those that have a public website) which build on top of the GNUnet framework. @c *********************************************************************** @node Internal dependencies @section Internal dependencies This section tries to give an overview of what processes a typical GNUnet peer running a particular application would consist of. All of the processes listed here should be automatically started by @command{gnunet-arm -s}. The list is given as a rough first guide to users for failure diagnostics. Ideally, end-users should never have to worry about these internal dependencies. In terms of internal dependencies, a minimum file-sharing system consists of the following GNUnet processes (in order of dependency): @itemize @bullet @item gnunet-service-arm @item gnunet-service-resolver (required by all) @item gnunet-service-statistics (required by all) @item gnunet-service-peerinfo @item gnunet-service-transport (requires peerinfo) @item gnunet-service-core (requires transport) @item gnunet-daemon-hostlist (requires core) @item gnunet-daemon-topology (requires hostlist, peerinfo) @item gnunet-service-datastore @item gnunet-service-dht (requires core) @item gnunet-service-identity @item gnunet-service-fs (requires identity, mesh, dht, datastore, core) @end itemize @noindent A minimum VPN system consists of the following GNUnet processes (in order of dependency): @itemize @bullet @item gnunet-service-arm @item gnunet-service-resolver (required by all) @item gnunet-service-statistics (required by all) @item gnunet-service-peerinfo @item gnunet-service-transport (requires peerinfo) @item gnunet-service-core (requires transport) @item gnunet-daemon-hostlist (requires core) @item gnunet-service-dht (requires core) @item gnunet-service-mesh (requires dht, core) @item gnunet-service-dns (requires dht) @item gnunet-service-regex (requires dht) @item gnunet-service-vpn (requires regex, dns, mesh, dht) @end itemize @noindent A minimum GNS system consists of the following GNUnet processes (in order of dependency): @itemize @bullet @item gnunet-service-arm @item gnunet-service-resolver (required by all) @item gnunet-service-statistics (required by all) @item gnunet-service-peerinfo @item gnunet-service-transport (requires peerinfo) @item gnunet-service-core (requires transport) @item gnunet-daemon-hostlist (requires core) @item gnunet-service-dht (requires core) @item gnunet-service-mesh (requires dht, core) @item gnunet-service-dns (requires dht) @item gnunet-service-regex (requires dht) @item gnunet-service-vpn (requires regex, dns, mesh, dht) @item gnunet-service-identity @item gnunet-service-namestore (requires identity) @item gnunet-service-gns (requires vpn, dns, dht, namestore, identity) @end itemize @c *********************************************************************** @node Code overview @section Code overview This section gives a brief overview of the GNUnet source code. Specifically, we sketch the function of each of the subdirectories in the @file{gnunet/src/} directory. The order given is roughly bottom-up (in terms of the layers of the system). @table @asis @item @file{util/} --- libgnunetutil Library with general utility functions, all GNUnet binaries link against this library. Anything from memory allocation and data structures to cryptography and inter-process communication. The goal is to provide an OS-independent interface and more 'secure' or convenient implementations of commonly used primitives. The API is spread over more than a dozen headers, developers should study those closely to avoid duplicating existing functions. @pxref{libgnunetutil}. @item @file{hello/} --- libgnunethello HELLO messages are used to describe under which addresses a peer can be reached (for example, protocol, IP, port). This library manages parsing and generating of HELLO messages. @item @file{block/} --- libgnunetblock The DHT and other components of GNUnet store information in units called 'blocks'. Each block has a type and the type defines a particular format and how that binary format is to be linked to a hash code (the key for the DHT and for databases). The block library is a wrapper around block plugins which provide the necessary functions for each block type. @item @file{statistics/} --- statistics service The statistics service enables associating values (of type uint64_t) with a component name and a string. The main uses is debugging (counting events), performance tracking and user entertainment (what did my peer do today?). @item @file{arm/} --- Automatic Restart Manager (ARM) The automatic-restart-manager (ARM) service is the GNUnet master service. Its role is to start gnunet-services, to re-start them when they crashed and finally to shut down the system when requested. @item @file{peerinfo/} --- peerinfo service The peerinfo service keeps track of which peers are known to the local peer and also tracks the validated addresses for each peer (in the form of a HELLO message) for each of those peers. The peer is not necessarily connected to all peers known to the peerinfo service. Peerinfo provides persistent storage for peer identities --- peers are not forgotten just because of a system restart. @item @file{datacache/} --- libgnunetdatacache The datacache library provides (temporary) block storage for the DHT. Existing plugins can store blocks in Sqlite, Postgres or MySQL databases. All data stored in the cache is lost when the peer is stopped or restarted (datacache uses temporary tables). @item @file{datastore/} --- datastore service The datastore service stores file-sharing blocks in databases for extended periods of time. In contrast to the datacache, data is not lost when peers restart. However, quota restrictions may still cause old, expired or low-priority data to be eventually discarded. Existing plugins can store blocks in Sqlite, Postgres or MySQL databases. @item @file{template/} --- service template Template for writing a new service. Does nothing. @item @file{ats/} --- Automatic Transport Selection The automatic transport selection (ATS) service is responsible for deciding which address (i.e. which transport plugin) should be used for communication with other peers, and at what bandwidth. @item @file{nat/} --- libgnunetnat Library that provides basic functions for NAT traversal. The library supports NAT traversal with manual hole-punching by the user, UPnP and ICMP-based autonomous NAT traversal. The library also includes an API for testing if the current configuration works and the @code{gnunet-nat-server} which provides an external service to test the local configuration. @item @file{fragmentation/} --- libgnunetfragmentation Some transports (UDP and WLAN, mostly) have restrictions on the maximum transfer unit (MTU) for packets. The fragmentation library can be used to break larger packets into chunks of at most 1k and transmit the resulting fragments reliably (with acknowledgment, retransmission, timeouts, etc.). @item @file{transport/} --- transport service The transport service is responsible for managing the basic P2P communication. It uses plugins to support P2P communication over TCP, UDP, HTTP, HTTPS and other protocols.The transport service validates peer addresses, enforces bandwidth restrictions, limits the total number of connections and enforces connectivity restrictions (i.e. friends-only). @item @file{peerinfo-tool/} --- gnunet-peerinfo This directory contains the gnunet-peerinfo binary which can be used to inspect the peers and HELLOs known to the peerinfo service. @item @file{core/} The core service is responsible for establishing encrypted, authenticated connections with other peers, encrypting and decrypting messages and forwarding messages to higher-level services that are interested in them. @item @file{testing/} --- libgnunettesting The testing library allows starting (and stopping) peers for writing testcases. It also supports automatic generation of configurations for peers ensuring that the ports and paths are disjoint. libgnunettesting is also the foundation for the testbed service @item @file{testbed/} --- testbed service The testbed service is used for creating small or large scale deployments of GNUnet peers for evaluation of protocols. It facilitates peer deployments on multiple hosts (for example, in a cluster) and establishing various network topologies (both underlay and overlay). @item @file{nse/} --- Network Size Estimation The network size estimation (NSE) service implements a protocol for (securely) estimating the current size of the P2P network. @item @file{dht/} --- distributed hash table The distributed hash table (DHT) service provides a distributed implementation of a hash table to store blocks under hash keys in the P2P network. @item @file{hostlist/} --- hostlist service The hostlist service allows learning about other peers in the network by downloading HELLO messages from an HTTP server, can be configured to run such an HTTP server and also implements a P2P protocol to advertise and automatically learn about other peers that offer a public hostlist server. @item @file{topology/} --- topology service The topology service is responsible for maintaining the mesh topology. It tries to maintain connections to friends (depending on the configuration) and also tries to ensure that the peer has a decent number of active connections at all times. If necessary, new connections are added. All peers should run the topology service, otherwise they may end up not being connected to any other peer (unless some other service ensures that core establishes the required connections). The topology service also tells the transport service which connections are permitted (for friend-to-friend networking) @item @file{fs/} --- file-sharing The file-sharing (FS) service implements GNUnet's file-sharing application. Both anonymous file-sharing (using gap) and non-anonymous file-sharing (using dht) are supported. @item @file{cadet/} --- cadet service The CADET service provides a general-purpose routing abstraction to create end-to-end encrypted tunnels in mesh networks. We wrote a paper documenting key aspects of the design. @item @file{tun/} --- libgnunettun Library for building IPv4, IPv6 packets and creating checksums for UDP, TCP and ICMP packets. The header defines C structs for common Internet packet formats and in particular structs for interacting with TUN (virtual network) interfaces. @item @file{mysql/} --- libgnunetmysql Library for creating and executing prepared MySQL statements and to manage the connection to the MySQL database. Essentially a lightweight wrapper for the interaction between GNUnet components and libmysqlclient. @item @file{dns/} Service that allows intercepting and modifying DNS requests of the local machine. Currently used for IPv4-IPv6 protocol translation (DNS-ALG) as implemented by "pt/" and for the GNUnet naming system. The service can also be configured to offer an exit service for DNS traffic. @item @file{vpn/} --- VPN service The virtual public network (VPN) service provides a virtual tunnel interface (VTUN) for IP routing over GNUnet. Needs some other peers to run an "exit" service to work. Can be activated using the "gnunet-vpn" tool or integrated with DNS using the "pt" daemon. @item @file{exit/} Daemon to allow traffic from the VPN to exit this peer to the Internet or to specific IP-based services of the local peer. Currently, an exit service can only be restricted to IPv4 or IPv6, not to specific ports and or IP address ranges. If this is not acceptable, additional firewall rules must be added manually. exit currently only works for normal UDP, TCP and ICMP traffic; DNS queries need to leave the system via a DNS service. @item @file{pt/} protocol translation daemon. This daemon enables 4-to-6, 6-to-4, 4-over-6 or 6-over-4 transitions for the local system. It essentially uses "DNS" to intercept DNS replies and then maps results to those offered by the VPN, which then sends them using mesh to some daemon offering an appropriate exit service. @item @file{identity/} Management of egos (alter egos) of a user; identities are essentially named ECC private keys and used for zones in the GNU name system and for namespaces in file-sharing, but might find other uses later @item @file{revocation/} Key revocation service, can be used to revoke the private key of an identity if it has been compromised @item @file{namecache/} Cache for resolution results for the GNU name system; data is encrypted and can be shared among users, loss of the data should ideally only result in a performance degradation (persistence not required) @item @file{namestore/} Database for the GNU name system with per-user private information, persistence required @item @file{gns/} GNU name system, a GNU approach to DNS and PKI. @item @file{dv/} A plugin for distance-vector (DV)-based routing. DV consists of a service and a transport plugin to provide peers with the illusion of a direct P2P connection for connections that use multiple (typically up to 3) hops in the actual underlay network. @item @file{regex/} Service for the (distributed) evaluation of regular expressions. @item @file{scalarproduct/} The scalar product service offers an API to perform a secure multiparty computation which calculates a scalar product between two peers without exposing the private input vectors of the peers to each other. @item @file{consensus/} The consensus service will allow a set of peers to agree on a set of values via a distributed set union computation. @item @file{rest/} The rest API allows access to GNUnet services using RESTful interaction. The services provide plugins that can exposed by the rest server. @c FIXME: Where did this disappear to? @c @item @file{experimentation/} @c The experimentation daemon coordinates distributed @c experimentation to evaluate transport and ATS properties. @end table @c *********************************************************************** @node System Architecture @section System Architecture @c FIXME: For those irritated by the textflow, we are missing images here, @c in the short term we should add them back, in the long term this should @c work without images or have images with alt-text. GNUnet developers like LEGOs. The blocks are indestructible, can be stacked together to construct complex buildings and it is generally easy to swap one block for a different one that has the same shape. GNUnet's architecture is based on LEGOs: @c @image{images/service_lego_block,5in,,picture of a LEGO block stack - 3 APIs as connectors upon Network Protocol on top of a Service} This chapter documents the GNUnet LEGO system, also known as GNUnet's system architecture. The most common GNUnet component is a service. Services offer an API (or several, depending on what you count as "an API") which is implemented as a library. The library communicates with the main process of the service using a service-specific network protocol. The main process of the service typically doesn't fully provide everything that is needed --- it has holes to be filled by APIs to other services. A special kind of component in GNUnet are user interfaces and daemons. Like services, they have holes to be filled by APIs of other services. Unlike services, daemons do not implement their own network protocol and they have no API: The GNUnet system provides a range of services, daemons and user interfaces, which are then combined into a layered GNUnet instance (also known as a peer). Note that while it is generally possible to swap one service for another compatible service, there is often only one implementation. However, during development we often have a "new" version of a service in parallel with an "old" version. While the "new" version is not working, developers working on other parts of the service can continue their development by simply using the "old" service. Alternative design ideas can also be easily investigated by swapping out individual components. This is typically achieved by simply changing the name of the "BINARY" in the respective configuration section. Key properties of GNUnet services are that they must be separate processes and that they must protect themselves by applying tight error checking against the network protocol they implement (thereby achieving a certain degree of robustness). On the other hand, the APIs are implemented to tolerate failures of the service, isolating their host process from errors by the service. If the service process crashes, other services and daemons around it should not also fail, but instead wait for the service process to be restarted by ARM. @c *********************************************************************** @node Subsystem stability @section Subsystem stability This section documents the current stability of the various GNUnet subsystems. Stability here describes the expected degree of compatibility with future versions of GNUnet. For each subsystem we distinguish between compatibility on the P2P network level (communication protocol between peers), the IPC level (communication between the service and the service library) and the API level (stability of the API). P2P compatibility is relevant in terms of which applications are likely going to be able to communicate with future versions of the network. IPC communication is relevant for the implementation of language bindings that re-implement the IPC messages. Finally, API compatibility is relevant to developers that hope to be able to avoid changes to applications build on top of the APIs of the framework. The following table summarizes our current view of the stability of the respective protocols or APIs: @multitable @columnfractions .20 .20 .20 .20 @headitem Subsystem @tab P2P @tab IPC @tab C API @item util @tab n/a @tab n/a @tab stable @item arm @tab n/a @tab stable @tab stable @item ats @tab n/a @tab unstable @tab testing @item block @tab n/a @tab n/a @tab stable @item cadet @tab testing @tab testing @tab testing @item consensus @tab experimental @tab experimental @tab experimental @item core @tab stable @tab stable @tab stable @item datacache @tab n/a @tab n/a @tab stable @item datastore @tab n/a @tab stable @tab stable @item dht @tab stable @tab stable @tab stable @item dns @tab stable @tab stable @tab stable @item dv @tab testing @tab testing @tab n/a @item exit @tab testing @tab n/a @tab n/a @item fragmentation @tab stable @tab n/a @tab stable @item fs @tab stable @tab stable @tab stable @item gns @tab stable @tab stable @tab stable @item hello @tab n/a @tab n/a @tab testing @item hostlist @tab stable @tab stable @tab n/a @item identity @tab stable @tab stable @tab n/a @item multicast @tab experimental @tab experimental @tab experimental @item mysql @tab stable @tab n/a @tab stable @item namestore @tab n/a @tab stable @tab stable @item nat @tab n/a @tab n/a @tab stable @item nse @tab stable @tab stable @tab stable @item peerinfo @tab n/a @tab stable @tab stable @item psyc @tab experimental @tab experimental @tab experimental @item pt @tab n/a @tab n/a @tab n/a @item regex @tab stable @tab stable @tab stable @item revocation @tab stable @tab stable @tab stable @item social @tab experimental @tab experimental @tab experimental @item statistics @tab n/a @tab stable @tab stable @item testbed @tab n/a @tab testing @tab testing @item testing @tab n/a @tab n/a @tab testing @item topology @tab n/a @tab n/a @tab n/a @item transport @tab stable @tab stable @tab stable @item tun @tab n/a @tab n/a @tab stable @item vpn @tab testing @tab n/a @tab n/a @end multitable Here is a rough explanation of the values: @table @samp @item stable No incompatible changes are planned at this time; for IPC/APIs, if there are incompatible changes, they will be minor and might only require minimal changes to existing code; for P2P, changes will be avoided if at all possible for the 0.10.x-series @item testing No incompatible changes are planned at this time, but the code is still known to be in flux; so while we have no concrete plans, our expectation is that there will still be minor modifications; for P2P, changes will likely be extensions that should not break existing code @item unstable Changes are planned and will happen; however, they will not be totally radical and the result should still resemble what is there now; nevertheless, anticipated changes will break protocol/API compatibility @item experimental Changes are planned and the result may look nothing like what the API/protocol looks like today @item unknown Someone should think about where this subsystem headed @item n/a This subsystem does not have an API/IPC-protocol/P2P-protocol @end table @c *********************************************************************** @node Naming conventions and coding style guide @section Naming conventions and coding style guide Here you can find some rules to help you write code for GNUnet. @c *********************************************************************** @menu * Naming conventions:: * Coding style:: @end menu @node Naming conventions @subsection Naming conventions @c *********************************************************************** @menu * include files:: * binaries:: * logging:: * configuration:: * exported symbols:: * private (library-internal) symbols (including structs and macros):: * testcases:: * performance tests:: * src/ directories:: @end menu @node include files @subsubsection include files @itemize @bullet @item _lib: library without need for a process @item _service: library that needs a service process @item _plugin: plugin definition @item _protocol: structs used in network protocol @item exceptions: @itemize @bullet @item gnunet_config.h --- generated @item platform.h --- first included @item plibc.h --- external library @item gnunet_common.h --- fundamental routines @item gnunet_directories.h --- generated @item gettext.h --- external library @end itemize @end itemize @c *********************************************************************** @node binaries @subsubsection binaries @itemize @bullet @item gnunet-service-xxx: service process (has listen socket) @item gnunet-daemon-xxx: daemon process (no listen socket) @item gnunet-helper-xxx[-yyy]: SUID helper for module xxx @item gnunet-yyy: command-line tool for end-users @item libgnunet_plugin_xxx_yyy.so: plugin for API xxx @item libgnunetxxx.so: library for API xxx @end itemize @c *********************************************************************** @node logging @subsubsection logging @itemize @bullet @item services and daemons use their directory name in @code{GNUNET_log_setup} (i.e. 'core') and log using plain 'GNUNET_log'. @item command-line tools use their full name in @code{GNUNET_log_setup} (i.e. 'gnunet-publish') and log using plain 'GNUNET_log'. @item service access libraries log using '@code{GNUNET_log_from}' and use '@code{DIRNAME-api}' for the component (i.e. 'core-api') @item pure libraries (without associated service) use '@code{GNUNET_log_from}' with the component set to their library name (without lib or '@file{.so}'), which should also be their directory name (i.e. '@file{nat}') @item plugins should use '@code{GNUNET_log_from}' with the directory name and the plugin name combined to produce the component name (i.e. 'transport-tcp'). @item logging should be unified per-file by defining a @code{LOG} macro with the appropriate arguments, along these lines: @example #define LOG(kind,...) GNUNET_log_from (kind, "example-api",__VA_ARGS__) @end example @end itemize @c *********************************************************************** @node configuration @subsubsection configuration @itemize @bullet @item paths (that are substituted in all filenames) are in PATHS (have as few as possible) @item all options for a particular module (@file{src/MODULE}) are under @code{[MODULE]} @item options for a plugin of a module are under @code{[MODULE-PLUGINNAME]} @end itemize @c *********************************************************************** @node exported symbols @subsubsection exported symbols @itemize @bullet @item must start with @code{GNUNET_modulename_} and be defined in @file{modulename.c} @item exceptions: those defined in @file{gnunet_common.h} @end itemize @c *********************************************************************** @node private (library-internal) symbols (including structs and macros) @subsubsection private (library-internal) symbols (including structs and macros) @itemize @bullet @item must NOT start with any prefix @item must not be exported in a way that linkers could use them or@ other libraries might see them via headers; they must be either declared/defined in C source files or in headers that are in the respective directory under @file{src/modulename/} and NEVER be declared in @file{src/include/}. @end itemize @node testcases @subsubsection testcases @itemize @bullet @item must be called @file{test_module-under-test_case-description.c} @item "case-description" maybe omitted if there is only one test @end itemize @c *********************************************************************** @node performance tests @subsubsection performance tests @itemize @bullet @item must be called @file{perf_module-under-test_case-description.c} @item "case-description" maybe omitted if there is only one performance test @item Must only be run if @code{HAVE_BENCHMARKS} is satisfied @end itemize @c *********************************************************************** @node src/ directories @subsubsection src/ directories @itemize @bullet @item gnunet-NAME: end-user applications (i.e., gnunet-search, gnunet-arm) @item gnunet-service-NAME: service processes with accessor library (i.e., gnunet-service-arm) @item libgnunetNAME: accessor library (_service.h-header) or standalone library (_lib.h-header) @item gnunet-daemon-NAME: daemon process without accessor library (i.e., gnunet-daemon-hostlist) and no GNUnet management port @item libgnunet_plugin_DIR_NAME: loadable plugins (i.e., libgnunet_plugin_transport_tcp) @end itemize @cindex Coding style @node Coding style @subsection Coding style @c XXX: Adjust examples to GNU Standards! @itemize @bullet @item We follow the GNU Coding Standards (@pxref{Top, The GNU Coding Standards,, standards, The GNU Coding Standards}); @item Indentation is done with spaces, two per level, no tabs; @item C99 struct initialization is fine; @item declare only one variable per line, for example: @noindent instead of @example int i,j; @end example @noindent write: @example int i; int j; @end example @c TODO: include actual example from a file in source @noindent This helps keep diffs small and forces developers to think precisely about the type of every variable. Note that @code{char *} is different from @code{const char*} and @code{int} is different from @code{unsigned int} or @code{uint32_t}. Each variable type should be chosen with care. @item While @code{goto} should generally be avoided, having a @code{goto} to the end of a function to a block of clean up statements (free, close, etc.) can be acceptable. @item Conditions should be written with constants on the left (to avoid accidental assignment) and with the @code{true} target being either the @code{error} case or the significantly simpler continuation. For example: @example if (0 != stat ("filename," &sbuf)) @{ error(); @} else @{ /* handle normal case here */ @} @end example @noindent instead of @example if (stat ("filename," &sbuf) == 0) @{ /* handle normal case here */ @} else @{ error(); @} @end example @noindent If possible, the error clause should be terminated with a @code{return} (or @code{goto} to some cleanup routine) and in this case, the @code{else} clause should be omitted: @example if (0 != stat ("filename," &sbuf)) @{ error(); return; @} /* handle normal case here */ @end example This serves to avoid deep nesting. The 'constants on the left' rule applies to all constants (including. @code{GNUNET_SCHEDULER_NO_TASK}), NULL, and enums). With the two above rules (constants on left, errors in 'true' branch), there is only one way to write most branches correctly. @item Combined assignments and tests are allowed if they do not hinder code clarity. For example, one can write: @example if (NULL == (value = lookup_function())) @{ error(); return; @} @end example @item Use @code{break} and @code{continue} wherever possible to avoid deep(er) nesting. Thus, we would write: @example next = head; while (NULL != (pos = next)) @{ next = pos->next; if (! should_free (pos)) continue; GNUNET_CONTAINER_DLL_remove (head, tail, pos); GNUNET_free (pos); @} @end example instead of @example next = head; while (NULL != (pos = next)) @{ next = pos->next; if (should_free (pos)) @{ /* unnecessary nesting! */ GNUNET_CONTAINER_DLL_remove (head, tail, pos); GNUNET_free (pos); @} @} @end example @item We primarily use @code{for} and @code{while} loops. A @code{while} loop is used if the method for advancing in the loop is not a straightforward increment operation. In particular, we use: @example next = head; while (NULL != (pos = next)) @{ next = pos->next; if (! should_free (pos)) continue; GNUNET_CONTAINER_DLL_remove (head, tail, pos); GNUNET_free (pos); @} @end example to free entries in a list (as the iteration changes the structure of the list due to the free; the equivalent @code{for} loop does no longer follow the simple @code{for} paradigm of @code{for(INIT;TEST;INC)}). However, for loops that do follow the simple @code{for} paradigm we do use @code{for}, even if it involves linked lists: @example /* simple iteration over a linked list */ for (pos = head; NULL != pos; pos = pos->next) @{ use (pos); @} @end example @item The first argument to all higher-order functions in GNUnet must be declared to be of type @code{void *} and is reserved for a closure. We do not use inner functions, as trampolines would conflict with setups that use non-executable stacks. The first statement in a higher-order function, which unusually should be part of the variable declarations, should assign the @code{cls} argument to the precise expected type. For example: @example int callback (void *cls, char *args) @{ struct Foo *foo = cls; int other_variables; /* rest of function */ @} @end example @item It is good practice to write complex @code{if} expressions instead of using deeply nested @code{if} statements. However, except for addition and multiplication, all operators should use parens. This is fine: @example if ( (1 == foo) || ((0 == bar) && (x != y)) ) return x; @end example However, this is not: @example if (1 == foo) return x; if (0 == bar && x != y) return x; @end example @noindent Note that splitting the @code{if} statement above is debatable as the @code{return x} is a very trivial statement. However, once the logic after the branch becomes more complicated (and is still identical), the "or" formulation should be used for sure. @item There should be two empty lines between the end of the function and the comments describing the following function. There should be a single empty line after the initial variable declarations of a function. If a function has no local variables, there should be no initial empty line. If a long function consists of several complex steps, those steps might be separated by an empty line (possibly followed by a comment describing the following step). The code should not contain empty lines in arbitrary places; if in doubt, it is likely better to NOT have an empty line (this way, more code will fit on the screen). @end itemize @c *********************************************************************** @node Build-system @section Build-system If you have code that is likely not to compile or build rules you might want to not trigger for most developers, use @code{if HAVE_EXPERIMENTAL} in your @file{Makefile.am}. Then it is OK to (temporarily) add non-compiling (or known-to-not-port) code. If you want to compile all testcases but NOT run them, run configure with the @code{--enable-test-suppression} option. If you want to run all testcases, including those that take a while, run configure with the @code{--enable-expensive-testcases} option. If you want to compile and run benchmarks, run configure with the @code{--enable-benchmarks} option. If you want to obtain code coverage results, run configure with the @code{--enable-coverage} option and run the @file{coverage.sh} script in the @file{contrib/} directory. @cindex gnunet-ext @node Developing extensions for GNUnet using the gnunet-ext template @section Developing extensions for GNUnet using the gnunet-ext template For developers who want to write extensions for GNUnet we provide the gnunet-ext template to provide an easy to use skeleton. gnunet-ext contains the build environment and template files for the development of GNUnet services, command line tools, APIs and tests. First of all you have to obtain gnunet-ext from git: @example git clone https://gnunet.org/git/gnunet-ext.git @end example The next step is to bootstrap and configure it. For configure you have to provide the path containing GNUnet with @code{--with-gnunet=/path/to/gnunet} and the prefix where you want the install the extension using @code{--prefix=/path/to/install}: @example ./bootstrap ./configure --prefix=/path/to/install --with-gnunet=/path/to/gnunet @end example When your GNUnet installation is not included in the default linker search path, you have to add @code{/path/to/gnunet} to the file @file{/etc/ld.so.conf} and run @code{ldconfig} or your add it to the environmental variable @code{LD_LIBRARY_PATH} by using @example export LD_LIBRARY_PATH=/path/to/gnunet/lib @end example @cindex writing testcases @node Writing testcases @section Writing testcases Ideally, any non-trivial GNUnet code should be covered by automated testcases. Testcases should reside in the same place as the code that is being tested. The name of source files implementing tests should begin with @code{test_} followed by the name of the file that contains the code that is being tested. Testcases in GNUnet should be integrated with the autotools build system. This way, developers and anyone building binary packages will be able to run all testcases simply by running @code{make check}. The final testcases shipped with the distribution should output at most some brief progress information and not display debug messages by default. The success or failure of a testcase must be indicated by returning zero (success) or non-zero (failure) from the main method of the testcase. The integration with the autotools is relatively straightforward and only requires modifications to the @file{Makefile.am} in the directory containing the testcase. For a testcase testing the code in @file{foo.c} the @file{Makefile.am} would contain the following lines: @example check_PROGRAMS = test_foo TESTS = $(check_PROGRAMS) test_foo_SOURCES = test_foo.c test_foo_LDADD = $(top_builddir)/src/util/libgnunetutil.la @end example Naturally, other libraries used by the testcase may be specified in the @code{LDADD} directive as necessary. Often testcases depend on additional input files, such as a configuration file. These support files have to be listed using the @code{EXTRA_DIST} directive in order to ensure that they are included in the distribution. Example: @example EXTRA_DIST = test_foo_data.conf @end example Executing @code{make check} will run all testcases in the current directory and all subdirectories. Testcases can be compiled individually by running @code{make test_foo} and then invoked directly using @code{./test_foo}. Note that due to the use of plugins in GNUnet, it is typically necessary to run @code{make install} before running any testcases. Thus the canonical command @code{make check install} has to be changed to @code{make install check} for GNUnet. @c *********************************************************************** @cindex Building GNUnet @node Building GNUnet and its dependencies @section Building GNUnet and its dependencies In the following section we will outline how to build GNUnet and some of its dependencies. We will assume a fair amount of knowledge for building applications under UNIX-like systems. Furthermore we assume that the build environment is sane and that you are aware of any implications actions in this process could have. Instructions here can be seen as notes for developers (an extension to the 'HACKING' section in README) as well as package maintainers. @b{Users should rely on the available binary packages.} We will use Debian as an example Operating System environment. Substitute accordingly with your own Operating System environment. For the full list of dependencies, consult the appropriate, up-to-date section in the @file{README} file. First, we need to build or install (depending on your OS) the following packages. If you build them from source, build them in this exact order: @example libgpgerror, libgcrypt, libnettle, libunbound, GnuTLS (with libunbound support) @end example After we have build and installed those packages, we continue with packages closer to GNUnet in this step: libgnurl (our libcurl fork), GNU libmicrohttpd, and GNU libextractor. Again, if your package manager provides one of these packages, use the packages provided from it unless you have good reasons (package version too old, conflicts, etc). We advise against compiling widely used packages such as GnuTLS yourself if your OS provides a variant already unless you take care of maintenance of the packages then. In the optimistic case, this command will give you all the dependencies: @example sudo apt-get install libgnurl libmicrohttpd libextractor @end example From experience we know that at the very least libgnurl is not available in some environments. You could substitute libgnurl with libcurl, but we recommend to install libgnurl, as it gives you a predefined libcurl with the small set GNUnet requires. In the past namespaces of libcurl and libgnurl were shared, which caused problems when you wanted to integrate both of them in one Operating System. This has been resolved, and they can be installed side by side now. @cindex libgnurl @cindex compiling libgnurl GNUnet and some of its function depend on a limited subset of cURL/libcurl. Rather than trying to enforce a certain configuration on the world, we opted to maintain a microfork of it that ensures we can link against the right set of features. We called this specialized set of libcurl ``libgnurl''. It is fully ABI compatible with libcurl and currently used by GNUnet and some of its dependencies. We download libgnurl and its digital signature from the GNU fileserver, assuming @env{TMPDIR} exists. Note: TMPDIR might be @file{/tmp}, @env{TMPDIR}, @env{TMP} or any other location. For consistency we assume @env{TMPDIR} points to @file{/tmp} for the remainder of this section. @example cd \$TMPDIR wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.60.0.tar.Z wget https://ftp.gnu.org/gnu/gnunet/gnurl-7.60.0.tar.Z.sig @end example Next, verify the digital signature of the file: @example gpg --verify gnurl-7.60.0.tar.Z.sig @end example If gpg fails, you might try with @command{gpg2} on your OS. If the error states that ``the key can not be found'' or it is unknown, you have to retrieve the key (A88C8ADD129828D7EAC02E52E22F9BBFEE348588) from a keyserver first: @example gpg --keyserver pgp.mit.edu --recv-keys A88C8ADD129828D7EAC02E52E22F9BBFEE348588 @end example and rerun the verification command. libgnurl will require the following packages to be present at runtime: gnutls (with DANE support / libunbound), libidn, zlib and at compile time: libtool, groff, perl, pkg-config, and python 2.7. Once you have verified that all the required packages are present on your system, we can proceed to compile libgnurl: @example tar -xvf gnurl-7.60.0.tar.Z cd gnurl-7.60.0 sh configure --disable-ntlm-wb make make -C tests test sudo make install @end example After you've compiled and installed libgnurl, we can proceed to building GNUnet. First, in addition to the GNUnet sources you might require downloading the latest version of various dependencies, depending on how recent the software versions in your distribution of GNU/Linux are. Most distributions do not include sufficiently recent versions of these dependencies. Thus, a typically installation on a "modern" GNU/Linux distribution requires you to install the following dependencies (ideally in this order): @itemize @bullet @item libgpgerror and libgcrypt @item libnettle and libunbound (possibly from distribution), GnuTLS @item libgnurl (read the README) @item GNU libmicrohttpd @item GNU libextractor @end itemize Make sure to first install the various mandatory and optional dependencies including development headers from your distribution. Other dependencies that you should strongly consider to install is a database (MySQL, sqlite or Postgres). The following instructions will assume that you installed at least sqlite. For most distributions you should be able to find pre-build packages for the database. Again, make sure to install the client libraries @b{and} the respective development headers (if they are packaged separately) as well. You can find specific, detailed instructions for installing of the dependencies (and possibly the rest of the GNUnet installation) in the platform-specific descriptions, which can be found in the Index. Please consult them now. If your distribution is not listed, please study the build instructions for Debian stable, carefully as you try to install the dependencies for your own distribution. Contributing additional instructions for further platforms is always appreciated. Please take in mind that operating system development tends to move at a rather fast speed. Due to this you should be aware that some of the instructions could be outdated by the time you are reading this. If you find a mistake, please tell us about it (or even better: send a patch to the documentation to fix it!). Before proceeding further, please double-check the dependency list. Note that in addition to satisfying the dependencies, you might have to make sure that development headers for the various libraries are also installed. There maybe files for other distributions, or you might be able to find equivalent packages for your distribution. While it is possible to build and install GNUnet without having root access, we will assume that you have full control over your system in these instructions. First, you should create a system user @emph{gnunet} and an additional group @emph{gnunetdns}. On the GNU/Linux distributions Debian and Ubuntu, type: @example sudo adduser --system --home /var/lib/gnunet --group \ --disabled-password gnunet sudo addgroup --system gnunetdns @end example @noindent On other Unixes and GNU systems, this should have the same effect: @example sudo useradd --system --groups gnunet --home-dir /var/lib/gnunet sudo addgroup --system gnunetdns @end example Now compile and install GNUnet using: @example tar xvf gnunet-@value{VERSION}.tar.gz cd gnunet-@value{VERSION} ./configure --with-sudo=sudo --with-nssdir=/lib make sudo make install @end example If you want to be able to enable DEBUG-level log messages, add @code{--enable-logging=verbose} to the end of the @command{./configure} command. @code{DEBUG}-level log messages are in English only and should only be useful for developers (or for filing really detailed bug reports). @noindent Next, edit the file @file{/etc/gnunet.conf} to contain the following: @example [arm] START_SYSTEM_SERVICES = YES START_USER_SERVICES = NO @end example @noindent You may need to update your @code{ld.so} cache to include files installed in @file{/usr/local/lib}: @example # ldconfig @end example @noindent Then, switch from user @code{root} to user @code{gnunet} to start the peer: @example # su -s /bin/sh - gnunet $ gnunet-arm -c /etc/gnunet.conf -s @end example You may also want to add the last line in the gnunet user's @file{crontab} prefixed with @code{@@reboot} so that it is executed whenever the system is booted: @example @@reboot /usr/local/bin/gnunet-arm -c /etc/gnunet.conf -s @end example @noindent This will only start the system-wide GNUnet services. Type @command{exit} to get back your root shell. Now, you need to configure the per-user part. For each user that should get access to GNUnet on the system, run (replace alice with your username): @example sudo adduser alice gnunet @end example @noindent to allow them to access the system-wide GNUnet services. Then, each user should create a configuration file @file{~/.config/gnunet.conf} with the lines: @example [arm] START_SYSTEM_SERVICES = NO START_USER_SERVICES = YES DEFAULTSERVICES = gns @end example @noindent and start the per-user services using @example $ gnunet-arm -c ~/.config/gnunet.conf -s @end example @noindent Again, adding a @code{crontab} entry to autostart the peer is advised: @example @@reboot /usr/local/bin/gnunet-arm -c $HOME/.config/gnunet.conf -s @end example @noindent Note that some GNUnet services (such as SOCKS5 proxies) may need a system-wide TCP port for each user. For those services, systems with more than one user may require each user to specify a different port number in their personal configuration file. Finally, the user should perform the basic initial setup for the GNU Name System (GNS) certificate authority. This is done by running: @example $ gnunet-gns-proxy-setup-ca @end example @noindent The first generates the default zones, whereas the second setups the GNS Certificate Authority with the user's browser. Now, to activate GNS in the normal DNS resolution process, you need to edit your @file{/etc/nsswitch.conf} where you should find a line like this: @example hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 @end example @noindent The exact details may differ a bit, which is fine. Add the text @emph{"gns [NOTFOUND=return]"} after @emph{"files"}. Keep in mind that we included a backslash ("\") here just for markup reasons. You should write the text below on @b{one line} and @b{without} the "\": @example hosts: files gns [NOTFOUND=return] mdns4_minimal \ [NOTFOUND=return] dns mdns4 @end example @c FIXME: Document new behavior. You might want to make sure that @file{/lib/libnss_gns.so.2} exists on your system, it should have been created during the installation. @c ********************************************************************** @cindex TESTING library @node TESTING library @section TESTING library The TESTING library is used for writing testcases which involve starting a single or multiple peers. While peers can also be started by testcases using the ARM subsystem, using TESTING library provides an elegant way to do this. The configurations of the peers are auto-generated from a given template to have non-conflicting port numbers ensuring that peers' services do not run into bind errors. This is achieved by testing ports' availability by binding a listening socket to them before allocating them to services in the generated configurations. An another advantage while using TESTING is that it shortens the testcase startup time as the hostkeys for peers are copied from a pre-computed set of hostkeys instead of generating them at peer startup which may take a considerable amount of time when starting multiple peers or on an embedded processor. TESTING also allows for certain services to be shared among peers. This feature is invaluable when testing with multiple peers as it helps to reduce the number of services run per each peer and hence the total number of processes run per testcase. TESTING library only handles creating, starting and stopping peers. Features useful for testcases such as connecting peers in a topology are not available in TESTING but are available in the TESTBED subsystem. Furthermore, TESTING only creates peers on the localhost, however by using TESTBED testcases can benefit from creating peers across multiple hosts. @menu * API:: * Finer control over peer stop:: * Helper functions:: * Testing with multiple processes:: @end menu @cindex TESTING API @node API @subsection API TESTING abstracts a group of peers as a TESTING system. All peers in a system have common hostname and no two services of these peers have a same port or a UNIX domain socket path. TESTING system can be created with the function @code{GNUNET_TESTING_system_create()} which returns a handle to the system. This function takes a directory path which is used for generating the configurations of peers, an IP address from which connections to the peers' services should be allowed, the hostname to be used in peers' configuration, and an array of shared service specifications of type @code{struct GNUNET_TESTING_SharedService}. The shared service specification must specify the name of the service to share, the configuration pertaining to that shared service and the maximum number of peers that are allowed to share a single instance of the shared service. TESTING system created with @code{GNUNET_TESTING_system_create()} chooses ports from the default range @code{12000} - @code{56000} while auto-generating configurations for peers. This range can be customised with the function @code{GNUNET_TESTING_system_create_with_portrange()}. This function is similar to @code{GNUNET_TESTING_system_create()} except that it take 2 additional parameters --- the start and end of the port range to use. A TESTING system is destroyed with the function @code{GNUNET_TESTING_system_destory()}. This function takes the handle of the system and a flag to remove the files created in the directory used to generate configurations. A peer is created with the function @code{GNUNET_TESTING_peer_configure()}. This functions takes the system handle, a configuration template from which the configuration for the peer is auto-generated and the index from where the hostkey for the peer has to be copied from. When successful, this function returns a handle to the peer which can be used to start and stop it and to obtain the identity of the peer. If unsuccessful, a NULL pointer is returned with an error message. This function handles the generated configuration to have non-conflicting ports and paths. Peers can be started and stopped by calling the functions @code{GNUNET_TESTING_peer_start()} and @code{GNUNET_TESTING_peer_stop()} respectively. A peer can be destroyed by calling the function @code{GNUNET_TESTING_peer_destroy}. When a peer is destroyed, the ports and paths in allocated in its configuration are reclaimed for usage in new peers. @c *********************************************************************** @node Finer control over peer stop @subsection Finer control over peer stop Using @code{GNUNET_TESTING_peer_stop()} is normally fine for testcases. However, calling this function for each peer is inefficient when trying to shutdown multiple peers as this function sends the termination signal to the given peer process and waits for it to terminate. It would be faster in this case to send the termination signals to the peers first and then wait on them. This is accomplished by the functions @code{GNUNET_TESTING_peer_kill()} which sends a termination signal to the peer, and the function @code{GNUNET_TESTING_peer_wait()} which waits on the peer. Further finer control can be achieved by choosing to stop a peer asynchronously with the function @code{GNUNET_TESTING_peer_stop_async()}. This function takes a callback parameter and a closure for it in addition to the handle to the peer to stop. The callback function is called with the given closure when the peer is stopped. Using this function eliminates blocking while waiting for the peer to terminate. An asynchronous peer stop can be canceled by calling the function @code{GNUNET_TESTING_peer_stop_async_cancel()}. Note that calling this function does not prevent the peer from terminating if the termination signal has already been sent to it. It does, however, cancels the callback to be called when the peer is stopped. @c *********************************************************************** @node Helper functions @subsection Helper functions Most of the testcases can benefit from an abstraction which configures a peer and starts it. This is provided by the function @code{GNUNET_TESTING_peer_run()}. This function takes the testing directory pathname, a configuration template, a callback and its closure. This function creates a peer in the given testing directory by using the configuration template, starts the peer and calls the given callback with the given closure. The function @code{GNUNET_TESTING_peer_run()} starts the ARM service of the peer which starts the rest of the configured services. A similar function @code{GNUNET_TESTING_service_run} can be used to just start a single service of a peer. In this case, the peer's ARM service is not started; instead, only the given service is run. @c *********************************************************************** @node Testing with multiple processes @subsection Testing with multiple processes When testing GNUnet, the splitting of the code into a services and clients often complicates testing. The solution to this is to have the testcase fork @code{gnunet-service-arm}, ask it to start the required server and daemon processes and then execute appropriate client actions (to test the client APIs or the core module or both). If necessary, multiple ARM services can be forked using different ports (!) to simulate a network. However, most of the time only one ARM process is needed. Note that on exit, the testcase should shutdown ARM with a @code{TERM} signal (to give it the chance to cleanly stop its child processes). The following code illustrates spawning and killing an ARM process from a testcase: @example static void run (void *cls, char *const *args, const char *cfgfile, const struct GNUNET_CONFIGURATION_Handle *cfg) @{ struct GNUNET_OS_Process *arm_pid; arm_pid = GNUNET_OS_start_process (NULL, NULL, "gnunet-service-arm", "gnunet-service-arm", "-c", cfgname, NULL); /* do real test work here */ if (0 != GNUNET_OS_process_kill (arm_pid, SIGTERM)) GNUNET_log_strerror (GNUNET_ERROR_TYPE_WARNING, "kill"); GNUNET_assert (GNUNET_OK == GNUNET_OS_process_wait (arm_pid)); GNUNET_OS_process_close (arm_pid); @} GNUNET_PROGRAM_run (argc, argv, "NAME-OF-TEST", "nohelp", options, &run, cls); @end example An alternative way that works well to test plugins is to implement a mock-version of the environment that the plugin expects and then to simply load the plugin directly. @c *********************************************************************** @node Performance regression analysis with Gauger @section Performance regression analysis with Gauger To help avoid performance regressions, GNUnet uses Gauger. Gauger is a simple logging tool that allows remote hosts to send performance data to a central server, where this data can be analyzed and visualized. Gauger shows graphs of the repository revisions and the performance data recorded for each revision, so sudden performance peaks or drops can be identified and linked to a specific revision number. In the case of GNUnet, the buildbots log the performance data obtained during the tests after each build. The data can be accessed on GNUnet's Gauger page. The menu on the left allows to select either the results of just one build bot (under "Hosts") or review the data from all hosts for a given test result (under "Metrics"). In case of very different absolute value of the results, for instance arm vs. amd64 machines, the option "Normalize" on a metric view can help to get an idea about the performance evolution across all hosts. Using Gauger in GNUnet and having the performance of a module tracked over time is very easy. First of course, the testcase must generate some consistent metric, which makes sense to have logged. Highly volatile or random dependent metrics probably are not ideal candidates for meaningful regression detection. To start logging any value, just include @code{gauger.h} in your testcase code. Then, use the macro @code{GAUGER()} to make the Buildbots log whatever value is of interest for you to @code{gnunet.org}'s Gauger server. No setup is necessary as most Buildbots have already everything in place and new metrics are created on demand. To delete a metric, you need to contact a member of the GNUnet development team (a file will need to be removed manually from the respective directory). The code in the test should look like this: @example [other includes] #include int main (int argc, char *argv[]) @{ [run test, generate data] GAUGER("YOUR_MODULE", "METRIC_NAME", (float)value, "UNIT"); @} @end example Where: @table @asis @item @strong{YOUR_MODULE} is a category in the gauger page and should be the name of the module or subsystem like "Core" or "DHT" @item @strong{METRIC} is the name of the metric being collected and should be concise and descriptive, like "PUT operations in sqlite-datastore". @item @strong{value} is the value of the metric that is logged for this run. @item @strong{UNIT} is the unit in which the value is measured, for instance "kb/s" or "kb of RAM/node". @end table If you wish to use Gauger for your own project, you can grab a copy of the latest stable release or check out Gauger's Subversion repository. @cindex TESTBED Subsystem @node TESTBED Subsystem @section TESTBED Subsystem The TESTBED subsystem facilitates testing and measuring of multi-peer deployments on a single host or over multiple hosts. The architecture of the testbed module is divided into the following: @itemize @bullet @item Testbed API: An API which is used by the testing driver programs. It provides with functions for creating, destroying, starting, stopping peers, etc. @item Testbed service (controller): A service which is started through the Testbed API. This service handles operations to create, destroy, start, stop peers, connect them, modify their configurations. @item Testbed helper: When a controller has to be started on a host, the testbed API starts the testbed helper on that host which in turn starts the controller. The testbed helper receives a configuration for the controller through its stdin and changes it to ensure the controller doesn't run into any port conflict on that host. @end itemize The testbed service (controller) is different from the other GNUnet services in that it is not started by ARM and is not supposed to be run as a daemon. It is started by the testbed API through a testbed helper. In a typical scenario involving multiple hosts, a controller is started on each host. Controllers take up the actual task of creating peers, starting and stopping them on the hosts they run. While running deployments on a single localhost the testbed API starts the testbed helper directly as a child process. When running deployments on remote hosts the testbed API starts Testbed Helpers on each remote host through remote shell. By default testbed API uses SSH as a remote shell. This can be changed by setting the environmental variable GNUNET_TESTBED_RSH_CMD to the required remote shell program. This variable can also contain parameters which are to be passed to the remote shell program. For e.g: @example export GNUNET_TESTBED_RSH_CMD="ssh -o BatchMode=yes \ -o NoHostAuthenticationForLocalhost=yes %h" @end example Substitutions are allowed in the command string above, this allows for substitutions through placemarks which begin with a `%'. At present the following substitutions are supported @itemize @bullet @item %h: hostname @item %u: username @item %p: port @end itemize Note that the substitution placemark is replaced only when the corresponding field is available and only once. Specifying @example %u@@%h @end example doesn't work either. If you want to user username substitutions for @command{SSH}, use the argument @code{-l} before the username substitution. For example: @example ssh -l %u -p %p %h @end example The testbed API and the helper communicate through the helpers stdin and stdout. As the helper is started through a remote shell on remote hosts any output messages from the remote shell interfere with the communication and results in a failure while starting the helper. For this reason, it is suggested to use flags to make the remote shells produce no output messages and to have password-less logins. The default remote shell, SSH, the default options are: @example -o BatchMode=yes -o NoHostBasedAuthenticationForLocalhost=yes" @end example Password-less logins should be ensured by using SSH keys. Since the testbed API executes the remote shell as a non-interactive shell, certain scripts like .bashrc, .profiler may not be executed. If this is the case testbed API can be forced to execute an interactive shell by setting up the environmental variable @code{GNUNET_TESTBED_RSH_CMD_SUFFIX} to a shell program. An example could be: @example export GNUNET_TESTBED_RSH_CMD_SUFFIX="sh -lc" @end example The testbed API will then execute the remote shell program as: @example $GNUNET_TESTBED_RSH_CMD -p $port $dest $GNUNET_TESTBED_RSH_CMD_SUFFIX \ gnunet-helper-testbed @end example On some systems, problems may arise while starting testbed helpers if GNUnet is installed into a custom location since the helper may not be found in the standard path. This can be addressed by setting the variable `@code{HELPER_BINARY_PATH}' to the path of the testbed helper. Testbed API will then use this path to start helper binaries both locally and remotely. Testbed API can accessed by including the @file{gnunet_testbed_service.h} file and linking with @code{-lgnunettestbed}. @c *********************************************************************** @menu * Supported Topologies:: * Hosts file format:: * Topology file format:: * Testbed Barriers:: * Automatic large-scale deployment in the PlanetLab testbed:: * TESTBED Caveats:: @end menu @node Supported Topologies @subsection Supported Topologies While testing multi-peer deployments, it is often needed that the peers are connected in some topology. This requirement is addressed by the function @code{GNUNET_TESTBED_overlay_connect()} which connects any given two peers in the testbed. The API also provides a helper function @code{GNUNET_TESTBED_overlay_configure_topology()} to connect a given set of peers in any of the following supported topologies: @itemize @bullet @item @code{GNUNET_TESTBED_TOPOLOGY_CLIQUE}: All peers are connected with each other @item @code{GNUNET_TESTBED_TOPOLOGY_LINE}: Peers are connected to form a line @item @code{GNUNET_TESTBED_TOPOLOGY_RING}: Peers are connected to form a ring topology @item @code{GNUNET_TESTBED_TOPOLOGY_2D_TORUS}: Peers are connected to form a 2 dimensional torus topology. The number of peers may not be a perfect square, in that case the resulting torus may not have the uniform poloidal and toroidal lengths @item @code{GNUNET_TESTBED_TOPOLOGY_ERDOS_RENYI}: Topology is generated to form a random graph. The number of links to be present should be given @item @code{GNUNET_TESTBED_TOPOLOGY_SMALL_WORLD}: Peers are connected to form a 2D Torus with some random links among them. The number of random links are to be given @item @code{GNUNET_TESTBED_TOPOLOGY_SMALL_WORLD_RING}: Peers are connected to form a ring with some random links among them. The number of random links are to be given @item @code{GNUNET_TESTBED_TOPOLOGY_SCALE_FREE}: Connects peers in a topology where peer connectivity follows power law - new peers are connected with high probability to well connected peers. (See Emergence of Scaling in Random Networks. Science 286, 509-512, 1999 (@uref{https://gnunet.org/git/bibliography.git/plain/docs/emergence_of_scaling_in_random_networks__barabasi_albert_science_286__1999.pdf, pdf})) @item @code{GNUNET_TESTBED_TOPOLOGY_FROM_FILE}: The topology information is loaded from a file. The path to the file has to be given. @xref{Topology file format}, for the format of this file. @item @code{GNUNET_TESTBED_TOPOLOGY_NONE}: No topology @end itemize The above supported topologies can be specified respectively by setting the variable @code{OVERLAY_TOPOLOGY} to the following values in the configuration passed to Testbed API functions @code{GNUNET_TESTBED_test_run()} and @code{GNUNET_TESTBED_run()}: @itemize @bullet @item @code{CLIQUE} @item @code{RING} @item @code{LINE} @item @code{2D_TORUS} @item @code{RANDOM} @item @code{SMALL_WORLD} @item @code{SMALL_WORLD_RING} @item @code{SCALE_FREE} @item @code{FROM_FILE} @item @code{NONE} @end itemize Topologies @code{RANDOM}, @code{SMALL_WORLD} and @code{SMALL_WORLD_RING} require the option @code{OVERLAY_RANDOM_LINKS} to be set to the number of random links to be generated in the configuration. The option will be ignored for the rest of the topologies. Topology @code{SCALE_FREE} requires the options @code{SCALE_FREE_TOPOLOGY_CAP} to be set to the maximum number of peers which can connect to a peer and @code{SCALE_FREE_TOPOLOGY_M} to be set to how many peers a peer should be at least connected to. Similarly, the topology @code{FROM_FILE} requires the option @code{OVERLAY_TOPOLOGY_FILE} to contain the path of the file containing the topology information. This option is ignored for the rest of the topologies. @xref{Topology file format}, for the format of this file. @c *********************************************************************** @node Hosts file format @subsection Hosts file format The testbed API offers the function @code{GNUNET_TESTBED_hosts_load_from_file()} to load from a given file details about the hosts which testbed can use for deploying peers. This function is useful to keep the data about hosts separate instead of hard coding them in code. Another helper function from testbed API, @code{GNUNET_TESTBED_run()} also takes a hosts file name as its parameter. It uses the above function to populate the hosts data structures and start controllers to deploy peers. These functions require the hosts file to be of the following format: @itemize @bullet @item Each line is interpreted to have details about a host @item Host details should include the username to use for logging into the host, the hostname of the host and the port number to use for the remote shell program. All thee values should be given. @item These details should be given in the following format: @example @@: @end example @end itemize Note that having canonical hostnames may cause problems while resolving the IP addresses (See this bug). Hence it is advised to provide the hosts' IP numerical addresses as hostnames whenever possible. @c *********************************************************************** @node Topology file format @subsection Topology file format A topology file describes how peers are to be connected. It should adhere to the following format for testbed to parse it correctly. Each line should begin with the target peer id. This should be followed by a colon(`:') and origin peer ids separated by `|'. All spaces except for newline characters are ignored. The API will then try to connect each origin peer to the target peer. For example, the following file will result in 5 overlay connections: [2->1], [3->1],[4->3], [0->3], [2->0]@ @code{@ 1:2|3@ 3:4| 0@ 0: 2@ } @c *********************************************************************** @node Testbed Barriers @subsection Testbed Barriers The testbed subsystem's barriers API facilitates coordination among the peers run by the testbed and the experiment driver. The concept is similar to the barrier synchronisation mechanism found in parallel programming or multi-threading paradigms - a peer waits at a barrier upon reaching it until the barrier is reached by a predefined number of peers. This predefined number of peers required to cross a barrier is also called quorum. We say a peer has reached a barrier if the peer is waiting for the barrier to be crossed. Similarly a barrier is said to be reached if the required quorum of peers reach the barrier. A barrier which is reached is deemed as crossed after all the peers waiting on it are notified. The barriers API provides the following functions: @itemize @bullet @item @strong{@code{GNUNET_TESTBED_barrier_init()}:} function to initialize a barrier in the experiment @item @strong{@code{GNUNET_TESTBED_barrier_cancel()}:} function to cancel a barrier which has been initialized before @item @strong{@code{GNUNET_TESTBED_barrier_wait()}:} function to signal barrier service that the caller has reached a barrier and is waiting for it to be crossed @item @strong{@code{GNUNET_TESTBED_barrier_wait_cancel()}:} function to stop waiting for a barrier to be crossed @end itemize Among the above functions, the first two, namely @code{GNUNET_TESTBED_barrier_init()} and @code{GNUNET_TESTBED_barrier_cancel()} are used by experiment drivers. All barriers should be initialised by the experiment driver by calling @code{GNUNET_TESTBED_barrier_init()}. This function takes a name to identify the barrier, the quorum required for the barrier to be crossed and a notification callback for notifying the experiment driver when the barrier is crossed. @code{GNUNET_TESTBED_barrier_cancel()} cancels an initialised barrier and frees the resources allocated for it. This function can be called upon a initialised barrier before it is crossed. The remaining two functions @code{GNUNET_TESTBED_barrier_wait()} and @code{GNUNET_TESTBED_barrier_wait_cancel()} are used in the peer's processes. @code{GNUNET_TESTBED_barrier_wait()} connects to the local barrier service running on the same host the peer is running on and registers that the caller has reached the barrier and is waiting for the barrier to be crossed. Note that this function can only be used by peers which are started by testbed as this function tries to access the local barrier service which is part of the testbed controller service. Calling @code{GNUNET_TESTBED_barrier_wait()} on an uninitialised barrier results in failure. @code{GNUNET_TESTBED_barrier_wait_cancel()} cancels the notification registered by @code{GNUNET_TESTBED_barrier_wait()}. @c *********************************************************************** @menu * Implementation:: @end menu @node Implementation @subsubsection Implementation Since barriers involve coordination between experiment driver and peers, the barrier service in the testbed controller is split into two components. The first component responds to the message generated by the barrier API used by the experiment driver (functions @code{GNUNET_TESTBED_barrier_init()} and @code{GNUNET_TESTBED_barrier_cancel()}) and the second component to the messages generated by barrier API used by peers (functions @code{GNUNET_TESTBED_barrier_wait()} and @code{GNUNET_TESTBED_barrier_wait_cancel()}). Calling @code{GNUNET_TESTBED_barrier_init()} sends a @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_INIT} message to the master controller. The master controller then registers a barrier and calls @code{GNUNET_TESTBED_barrier_init()} for each its subcontrollers. In this way barrier initialisation is propagated to the controller hierarchy. While propagating initialisation, any errors at a subcontroller such as timeout during further propagation are reported up the hierarchy back to the experiment driver. Similar to @code{GNUNET_TESTBED_barrier_init()}, @code{GNUNET_TESTBED_barrier_cancel()} propagates @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_CANCEL} message which causes controllers to remove an initialised barrier. The second component is implemented as a separate service in the binary `gnunet-service-testbed' which already has the testbed controller service. Although this deviates from the gnunet process architecture of having one service per binary, it is needed in this case as this component needs access to barrier data created by the first component. This component responds to @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} messages from local peers when they call @code{GNUNET_TESTBED_barrier_wait()}. Upon receiving @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} message, the service checks if the requested barrier has been initialised before and if it was not initialised, an error status is sent through @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message to the local peer and the connection from the peer is terminated. If the barrier is initialised before, the barrier's counter for reached peers is incremented and a notification is registered to notify the peer when the barrier is reached. The connection from the peer is left open. When enough peers required to attain the quorum send @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_WAIT} messages, the controller sends a @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message to its parent informing that the barrier is crossed. If the controller has started further subcontrollers, it delays this message until it receives a similar notification from each of those subcontrollers. Finally, the barriers API at the experiment driver receives the @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} when the barrier is reached at all the controllers. The barriers API at the experiment driver responds to the @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message by echoing it back to the master controller and notifying the experiment controller through the notification callback that a barrier has been crossed. The echoed @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message is propagated by the master controller to the controller hierarchy. This propagation triggers the notifications registered by peers at each of the controllers in the hierarchy. Note the difference between this downward propagation of the @code{GNUNET_MESSAGE_TYPE_TESTBED_BARRIER_STATUS} message from its upward propagation --- the upward propagation is needed for ensuring that the barrier is reached by all the controllers and the downward propagation is for triggering that the barrier is crossed. @cindex PlanetLab testbed @node Automatic large-scale deployment in the PlanetLab testbed @subsection Automatic large-scale deployment in the PlanetLab testbed PlanetLab is a testbed for computer networking and distributed systems research. It was established in 2002 and as of June 2010 was composed of 1090 nodes at 507 sites worldwide. To automate the GNUnet we created a set of automation tools to simplify the large-scale deployment. We provide you a set of scripts you can use to deploy GNUnet on a set of nodes and manage your installation. Please also check @uref{https://gnunet.org/installation-fedora8-svn} and @uref{https://gnunet.org/installation-fedora12-svn} to find detailed instructions how to install GNUnet on a PlanetLab node. @c *********************************************************************** @menu * PlanetLab Automation for Fedora8 nodes:: * Install buildslave on PlanetLab nodes running fedora core 8:: * Setup a new PlanetLab testbed using GPLMT:: * Why do i get an ssh error when using the regex profiler?:: @end menu @node PlanetLab Automation for Fedora8 nodes @subsubsection PlanetLab Automation for Fedora8 nodes @c *********************************************************************** @node Install buildslave on PlanetLab nodes running fedora core 8 @subsubsection Install buildslave on PlanetLab nodes running fedora core 8 @c ** Actually this is a subsubsubsection, but must be fixed differently @c ** as subsubsection is the lowest. Since most of the PlanetLab nodes are running the very old Fedora core 8 image, installing the buildslave software is quite some pain. For our PlanetLab testbed we figured out how to install the buildslave software best. @c This is a very terrible way to suggest installing software. @c FIXME: Is there an official, safer way instead of blind-piping a @c script? @c FIXME: Use newer pypi URLs below. Install Distribute for Python: @example curl http://python-distribute.org/distribute_setup.py | sudo python @end example Install Distribute for zope.interface <= 3.8.0 (4.0 and 4.0.1 will not work): @example export PYPI=@value{PYPI-URL} wget $PYPI/z/zope.interface/zope.interface-3.8.0.tar.gz tar zvfz zope.interface-3.8.0.tar.gz cd zope.interface-3.8.0 sudo python setup.py install @end example Install the buildslave software (0.8.6 was the latest version): @example export GCODE="http://buildbot.googlecode.com/files" wget $GCODE/buildbot-slave-0.8.6p1.tar.gz tar xvfz buildbot-slave-0.8.6p1.tar.gz cd buildslave-0.8.6p1 sudo python setup.py install @end example The setup will download the matching twisted package and install it. It will also try to install the latest version of zope.interface which will fail to install. Buildslave will work anyway since version 3.8.0 was installed before! @c *********************************************************************** @node Setup a new PlanetLab testbed using GPLMT @subsubsection Setup a new PlanetLab testbed using GPLMT @itemize @bullet @item Get a new slice and assign nodes Ask your PlanetLab PI to give you a new slice and assign the nodes you need @item Install a buildmaster You can stick to the buildbot documentation:@ @uref{http://buildbot.net/buildbot/docs/current/manual/installation.html} @item Install the buildslave software on all nodes To install the buildslave on all nodes assigned to your slice you can use the tasklist @code{install_buildslave_fc8.xml} provided with GPLMT: @example ./gplmt.py -c contrib/tumple_gnunet.conf -t \ contrib/tasklists/install_buildslave_fc8.xml -a -p @end example @item Create the buildmaster configuration and the slave setup commands The master and the and the slaves have need to have credentials and the master has to have all nodes configured. This can be done with the @file{create_buildbot_configuration.py} script in the @file{scripts} directory. This scripts takes a list of nodes retrieved directly from PlanetLab or read from a file and a configuration template and creates: @itemize @bullet @item a tasklist which can be executed with gplmt to setup the slaves @item a master.cfg file containing a PlanetLab nodes @end itemize A configuration template is included in the , most important is that the script replaces the following tags in the template: %GPLMT_BUILDER_DEFINITION :@ GPLMT_BUILDER_SUMMARY@ GPLMT_SLAVES@ %GPLMT_SCHEDULER_BUILDERS Create configuration for all nodes assigned to a slice: @example ./create_buildbot_configuration.py -u \ -p -s -m \ -t