path: root/doc/gnunet-java-tutorial.tex

\documentclass[10pt]{article}
\usepackage[ansinew]{inputenc}
\usepackage{makeidx,amsmath,amssymb,exscale,multicol,epsfig,graphics,verbatim,ulem}
\usepackage{epsfig,geometry,url,listings,subcaption,boxedminipage}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\geometry{headsep=3ex,hscale=0.9}
\usepackage{hyperref}
\hypersetup{
  pdftitle={gnunet-java tutorial},
  pdfsubject={gnunet-java},
  pdfauthor={Florian Dold <dold@in.tum.de>},
  pdfkeywords={gnunet,java,tutorial},
  colorlinks=true, % otherwise the boxes overlap content
  urlcolor=blue
}

\lstset{
basicstyle=\ttfamily,  
upquote=true,
columns=fullflexible,
literate={*}{{\char42}}1
         {-}{{\char45}}1
         {\\\$}{{\$ }}1
}

\newcommand{\exercise}[1]{\vskip5pt\noindent\begin{boxedminipage}{\textwidth}{\bf Exercise:} #1 \end{boxedminipage}}

\title{A Tutorial for GNUnet 0.10.x (Java version)}

\author{Florian Dold}

\begin{document}

\maketitle

Although GNUnet is primarily developed in the C programming language, it is also possible to write peer-to-peer applications
with GNUnet in Java.
The GNUnet-Java project provides bindings to a subset of existing GNUnet components, as well as the infrastructure to
develop new components.  GNUnet-Java is less stable and less complete than the C version. Please report any bugs or feature requests at \url{https://gnunet.org/bugs/}.

\section{Getting Started}
\subsection{Installing GNUnet}
This tutorial assumes that you have GNUnet $\geq$ 0.10.x installed on your
machine.  Instructions on how to do this can be found at
\url{https://gnunet.org/installation}, or in the C version of the GNUnet
tutorial.  
Make sure to run {\tt ./configure} with the option {\tt -{}-enable-javaports}, in order to allow
Java clients to connect to GNUnet services.
Start your GNUnet peer with the command {\tt gnunet-arm -s} and convince yourself that the default GNUnet services are running by typing {\tt gnunet-arm -I}.

\exercise{%
  Read the first three chapters of the GNUnet C tutorial, available at
  \url{https://gnunet.org/svn/gnunet/doc/gnunet-c-tutorial.pdf}.
}

\subsection{Other Dependencies}
Make sure that you have OpenJDK 6 or later installed on your system. 

\section{A simple GNUnet-Java extension}
The simplest way to create a new GNUnet component in Java is to copy
the template extension project,  which already contains a build system
and sample code.

Obtain the extension template from the subversion repository:
\begin{lstlisting}[language=bash,deletekeywords={export}]
\$ svn export https://gnunet.org/svn/gnunet-java-ext/ my-extension
\$ cd my-extension
\end{lstlisting}

The template project uses Gradle\footnote{\url{http://gradle.org}} as a build system.  Simply use the \texttt{./gradlew}
script to run build tasks.  The wrapper script will automatically download the correct version
of Gradle on the first run.  Alternatively, download Gradle$\ge$1.11 yourself, and use the \texttt{gradle} command
directly instead of the wrapper.

Build the template project by running the \texttt{assemble} task:
\begin{lstlisting}[language=bash]
\$ ./gradlew assemble
\end{lstlisting}
This will download all direct and transitive dependencies.  Gradle stores all dependencies in an internal cache.  Run
\begin{lstlisting}[language=bash]
\$ ./gradlew copyDeps
\end{lstlisting}
in order to copy all dependencies into the \texttt{lib/} folder\footnote{This is not strictly necessary---you can also
ask Gradle for the classpath pointing to the internal cache. However, having all dependencies in one folder
is more convenient, especially when using shell wrappers for Java entry points later on.}.

Check if you're on the right track by running the Java client for GNUnet's network size estimation service:
\begin{lstlisting}[language=bash]
\$ java -cp 'lib/*' org.gnunet.nse.NetworkSizeEstimation
\end{lstlisting}
This should print something like \texttt{est: 42.3 dev: 3.14 t: Sun Apr 06 23:40:37 CEST 2014} to
your terminal.  If the program hangs, check if your peer is running and correctly configured.

\subsection{The Basics}

This is the most basic skeleton for a GNUnet-Java application:
\begin{lstlisting}[language=java]
import org.gnunet.util.*;
public class HelloGnuNet {
    public static void main(String[] args) {
            new Program() {
                public void run() {
                    System.out.println("Hello, GNUnet");
                }
            }.start(args);
}
\end{lstlisting}

Calling {\tt start} initializes GNUnet-Java, parses the command line,
loads configuration files and starts the task scheduler, with the code
in the {\tt run} method executed in the initial task.

\exercise{%
  Get the code above to execute by placing it in a \texttt{*.java} file
  in \texttt{src/main/java/}, running the \texttt{assemble} task from Gradle,
  and invoking \texttt{java} with the right parameters.
The source code in the extension template should follow
the the Maven Standard Directory Layout%
\footnote{\url{https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html}}
for application code, tests and resources.
}

\subsection{Adding and using command line arguments}

Command line options are added by annotating members of your
{\tt org.gnunet.util.Program} subclass with the \@Option-annotation.

Here is a simple example:
\begin{lstlisting}[language=java]
new Program(args) {
    @Option(
        shortname = "n",
        longname = "name",
        action = OptionAction.STORE_STRING,
        description = "the name of the person you want to greet")
    String name = "Jane Doe"; // default value if option is missing
[...]
}
\end{lstlisting}

You can now specify a value for the member {\tt name} at the command
line, either by the long name with two dashes ({\tt -{}-name=Foo} / {\tt
-{}-name FOO}) or the short name ({\tt -n FOO}) with one dash.

Before the \texttt{run} method is called, the field {\tt name} will
be set to the given argument.  If the \texttt{name} option is missing,
the field will keep the value specified in the constructor or the field's initializer.

The \texttt{\@Option} annotation can not only be used with Strings, but also
with booleans and numbers. These are a few of the available options:

\begin{itemize}
\item {\tt STORE\_STRING}: Store a string in a {\tt String} variable
\item {\tt STORE\_NUMBER}: Store a number in a member of primitive type
\item {\tt SET}: Set a {\tt boolean} to {\tt true}
\end{itemize}

By default, the following arguments are available on the command line:

\begin{itemize}
  \item {\tt -h} / {\tt -{}-help} shows the help text
  \item {\tt -v} / {\tt -{}-version} shows version information
  \item {\tt -c} / {\tt -{}-config} specify an additional configuration file to load
  \item {\tt -L} / {\tt -{}-log} specify the log level
  \item {\tt -l} / {\tt -{}-logfile} specify a file to write the logs to
\end{itemize}

You can change the about text and the version information by
overriding the {\tt makeHelpText} or {\tt makeVersionDescription} methods in your
{\tt Program} subclass.

\exercise{Add a few different command line options to your program and print them with {\tt System.out}!}

\subsection{Shell Wrappers}
It is usually more convenient to have a shell wrapper for each entry point than
it is to pass the main class and classpath to the JVM manually every time.  The
shell wrappers in the \texttt{bin/} directory of the template set the JVM
classpath classpath relative to the location of the script file.  The shell wrappers
should therefore always be kept in the \texttt{bin/} directory.

\exercise{Copy the wrapper \texttt{bin/my-ext} and modify it to
call your \texttt{HelloGnuNet} class.}

\subsection{More Documentation}
The documentation for \texttt{gnunet-java} generated by Javadoc is available
at \url{https://gnunet.org/javadoc/}.

\section{The statistics API}
The statistics service allows to store numbers under a subsystem and a name.
These values are available to other components, even after your program quits.

\subsection{Connecting to the statistics service}

\begin{lstlisting}[language=java]
Statistics statistics = new Statistics(getConfiguration());
\end{lstlisting}

The Statistics constructor is called with the configuration,
provided by the method {\tt getConfiguration} of the {\tt Program}
class.  The configuration contains the necessary information (port numbers, socket paths, \dots) to establish a connection to the
statistics service.  As with most API calls in GNUnet-Java, this
operation is asynchronous, meaning that the above statement
does not wait for the connection to be established, but returns immediately.

Always remember to explicitly destroy your {\tt
  Statistics} instance by calling its {\tt destroy()}
method. Otherwise there might be pending operations that prevent the
termination of your program.

\subsection{Setting statistics}

You can use the newly created {\tt statistics} handle to, for instance,
set the value named ``\# bytes sent'' to the value 42.
\begin{lstlisting}[language=java]
statistics.set("gnunet-java-hello", "# bytes sent", 42, true);
\end{lstlisting}
The last parameter (\texttt{true})
indicates that the value should be stored persistently (%
persistent values are stored even if the statistics service restarts).


\subsection{Retrieving statistics}

Retrieving a value is slightly more complex. Because of the
asynchronous nature of the GNUnet-Java APIs, the {\tt get} method
does not directly return values, but a handle (implementing the
interface {\tt Cancelable}) to cancel the get request.
The actual values are accessed by passing a callback object to the {\tt get}
method.

This example retrieves the statistics value ``\# Requests Served'' for the subsystem ``gnunet-java-hello''
\begin{lstlisting}[language=java]
Cancelable getCancel = statistics.get(RelativeTime.SECOND, "gnunet-java-hello",
        "# Requests Served", new Statistics.StatisticsReceiver {
    public void onDone() {
        System.out.println("everything done");
    }
    public void onReceive(String subsystem, String name, long val) {
        System.out.println(subsyste + " " + name + " " + val);
    }
    public void onTimeout() {
        // called if the service does not respond after the
        // specified timeout (one second)
        System.out.println("timeout occured");
    }
});
\end{lstlisting}

\exercise{Read the Javadoc of the statistics service.  What other operations can be done on statistics values, other
than reading and writing them?}
\exercise{Write a program that increments a statistics value each second. Check the result with the {\tt gnunet-statistics} command line tool.
\emph{Hint}: Use \texttt{Scheduler.addDelayed} to run a function after a timeout. }

\section{Sending encrypted messages}

The CORE service is one of the most important components of GNUnet, and
allows sending encrypted messages to directly connected peers.  Be aware that,
depending on the used transport protocol, messages sent by CORE arrive with
varying reliability.

\subsection{Defining new Messages}

All GNUnet messages follow a common format.  Every
message consists of a header (with the message size and the message
type) and a body.  The same message format is used both for communication between
GNUnet services and clients, as well as between peers in the network.

You can define a new type of message in GNUnet-Java by annotating a
class with information on how to represent its members in binary format.

Additionally, you have to register your new message type with
GNUnet-Java, giving it a unique message type number.  Here is an example:

\begin{lstlisting}[language=java]
@UnionCase(4242)
public class ExampleMessage implements GnunetMessage.Body {
    @UInt8
    public int age;
    @ZeroTerminatedString;
    public String name;
}
\end{lstlisting}

The {\tt @UnionCase} annotation specifies the message type id of the
message body below (4242 in the example).  GnunetMessage.Body is a
union of messages, and ExampleMessage is one (new) member of the
union.

Every time you add a new type of GNUnet message, you have to run the command
\lstset{language=bash}
\begin{lstlisting}
\$ ./gradle msgtypes
\end{lstlisting}
This generates the file {\tt
  src/main/java/org/gnunet/construct/MsgMap.txt}, which allows the system to
instantiate the right Java class when de-serializing a message from its binary representation.

The above message then contains a value annotated with {\tt @UInt8}:
An {\bf 8}-bit {\bf U}nsigned {\bf int}eger.  There are similar
annotations for integers of other sizes, and {\tt @Int}{\it N}
annotations for signed integers.  The second member is a String, whose
binary representation appends a zero-byte to the string to mark its
end.

Other useful annotations can be found in the package {\tt
  org.gnunet.construct}. Among them are annotations for arrays of
fixed or variable size ({\tt @VariableSizeArray, @FixedSizeArray}),
for embedding other messages in your message ({\tt @NestedMessage} and
for implementing your own message unions.

\exercise{Define a message that contains a 32-bit signed integer.}
\exercise{%
  Look at the class {\tt  org.gnunet.dht.messages.MonitorPutMessage} in the GNUnet-Java source
  code\footnote{\url{https://gnunet.org/svn/gnunet-java/}}.
  This message uses a variety of different annotations, try to understand the purpose
  of each member's annotation.
}

\subsection{Connecting to Core}

After creating a handle to CORE by calling the {\tt Core} constructor,
you have to specify what types of messages you are interested in. The
CORE service will only send messages of these types to you, and only
notify you of connecting peers if they share a subset of the messages
you are interested in.

The {\tt handleMessages} method allows you to specify an object of a
class inheriting {\tt Runabout}.  The Runabout is a mechanism for
single-argument multiple dispatch in Java.  You have to define one
{\tt visit} method for every type of message you are interested in.
Once {\tt Core} receives a message, it is dispatched dynamically to
the {\tt visit} method with the appropriate signature.  Note that
every {\tt visit} method, as well as the receiver's class, has to be
public in order for the dynamic dispatch to work.

Example:

\begin{lstlisting}[language=java]
public class MyMessageReceiver extends Runabout {
    public void visit(MyFooMessage m) {
      // do something 
    }
    public void visit(MyBarMessage m) {
      // do something else 
    }
}
\end{lstlisting}

After specifying your message handler, the {\tt init} method has to be
called with a callback object.  This starts the handshake with the
CORE service, and once done the callback object's {\tt onInit} method will
be called with your peer's identity.

\subsection{Sending a message to another peer}

Before you can actually send a message, you have to wait until the
CORE service is ready to send your message.  This is done by calling
the {\tt notifyTransmitReady} method. You have to provide a callback
object to this method, whose {\tt transmit} method is invoked with a
{\tt MessageSink} object once CORE is ready to transmit your
message.  Call the {\tt transmit} method in the {\tt MessageSink}
with a {\tt GnunetMessage.Body} in order transmit it to CORE.
to finally transmit it. The header of the message is automatically
added to your message body.

Example:
\begin{lstlisting}[language=java]
// arguments: messagePriority, timeout, targetPeer, messageSize, transmitter
core.notifyTransmitReady(0, RelativeTime.FOREVER, myIdentity, 42, new MessageTransmitter() {
    public transmit(Connection.MessageSink sink) {
        sink.transmit(myMessage);
    }
    public handleError() {
        // do something
    }
\end{lstlisting}

You can use {\tt Construct.getSize} to calculate the size of a
message, or compute it manually.

\exercise{Write an echo program for CORE: Send a message to the local peer and receive it!}

\section{Establishing channels to remote peers with MESH}
In contrast to CORE, the MESH
service can send messages reliably (if requested) over channels to distant peers, who must not necessarily be directly connected.
The following code connects to the mesh service, and waits for connections on port 42:
\begin{lstlisting}[language=java]
Mesh m = new Mesh(cfg, inboundChannelHandler, messageHandler, 42);
\end{lstlisting}

The \texttt{inboundChannelHandler}'s \texttt{onInboundChannel} is called whenever another peer wants to establish a connection
to our peer on port 42.  The \texttt{messageHandle} must be a Runabout instance, and implement visit methods analogously to the
CORE message handler in the previous section.

The following snippet establishes a channel to the given peer on port 42, where
the channel should not buffer data (first boolean argument) and be reliable (second boolean argument).
\begin{lstlisting}[language=java]
Channel c = m.createChannel(targetPeer, 42, true, true);
\end{lstlisting}

A channel can be used to send messages, which are first queued and then sent to the mesh service:
\begin{lstlisting}[language=java]
c.send(myMessage);
\end{lstlisting}

Using this way of sending messages may cause the message queue of the channel to fill up quickly.
To prevent this, wrap the message in an \texttt{Envelope}, which can invoke a notification callback
once the message has been sent to the service:
\begin{lstlisting}[language=java]
Envelope ev = new Envelope(myMessage);
ev.notifySent(myNotifySentHandler);
c.send(myMessage);
// use ev.cancel() to abort sending the message
\end{lstlisting}
Note that the notification is called when the local MESH service accepts the message for further transmission,
not when the target peer receives the message.

\exercise{Write a netcat-style tool that allows to interactively send and receive a stream of text on the command line over MESH.}

\section{Managing a peer's egos}
An ego in GNUnet is a name tied to a key pair. Egos can represent the identity of actual users, organisations,
or more abstract entities.  Managed by the \texttt{IDENTITY} service, egos are entirely local to your peer.

For looking up the key of egos, there is a convenient helper function:
\begin{lstlisting}[language=java]
Identity.lookup(getConfiguration(), "my-ego-name", new IdentityCallback() {
    @Override
    public void onEgo(Identity.Ego ego) {
        System.out.println("public key: " + ego.getPublicKey());
        });
    }
    @Override
    public void onError(String errorMessage) {
        System.err.println("lookup failed: " + errorMessage);
    }
});
\end{lstlisting}
Creating, renaming and deleting egos requires a handle to the identity service:
\begin{lstlisting}[language=java]
Identity identity = new Identity();
identity.connect(getConfiguration(), null);
\end{lstlisting}
The second parameter of \texttt{connect}, which is null in the above code, can
be a listener object of type \texttt{IdentityListCallback}, and is called whenever an identity
is added, deleted or changed.

After connecting, identities can be created like this:
\begin{lstlisting}[language=java]
identity.create(myEgoName, new IdentityContinuation() {
    @Override
    public void onError(String errorMessage) {
      System.out.println("create failed: " + error message);
    }
    @Override
    public void onDone() {
      System.out.println("create successful");
    }
});
\end{lstlisting}
Renaming and deleting egos is done by similar means.


\section{Using the GNU Name System}
Resolving and publishing name records in the network can be done with GNS, a secure
and decentralized alternative to the widely used Domain Name System.
Currently, GNUnet-Java only supports resolving names.

Names must be looked up in a zone, which is simply an ego. 
Run \texttt{gnunet-gns-import.sh} (distributed with the main GNUnet package) in order to
set up GNS.

Verify that this created the master-zone ego:
\begin{lstlisting}[language=bash]
\$ gnunet-identity -d # display all egos
\end{lstlisting}

Create an A-record in your your master-zone with:
\begin{lstlisting}[language=bash]
\$ gnunet-namestore -z master-zone -a -n myrecord -t A -V 1.2.3.4 -e never
\end{lstlisting}

The following snippet assumes that \texttt{master} is the ego with the name ``master-zone'',
as retrieved with the \texttt{Identity.lookup} function.
\begin{lstlisting}
final Gns gns = new Gns(getConfiguration());
// look up an A record
long typeId = GnsRecord.getIdFromString("A");
gns.lookup("alice.gnu", master.getPublicKey(), typeId, GNS_LOOKUP_OPTION_DEFAULT,
        null, new LookupResultProcessor() {
    @Override
    public void process(GnsRecord[] records) {
        for (GnsRecord record : records) {
            System.out.println("Record " +
                record.getRecordData().asRecordString());
        }
    }
});
\end{lstlisting}
A \texttt{GnsRecord} contains the record stored in binary form.
Calling \texttt{getRecordData} on the record instantiates an object
whose class is specific to the record type (e.g. \texttt{ARecordData}),
or an object of type \texttt{UnknownRecordData} if GNUnet-Java does not support
the given record type.

\exercise{Read more about GNS at \url{https://gnunet.org/gns-namestore-editing}.}
\exercise{Look at the Javadoc for \texttt{org.gnunet.gns.record}. What types of records are
currently supported, which are missing?}


\section{Other useful APIs}
Some other useful service APIs currently implemented are NSE (in {\tt
  org.gnunet.nse.NetworkSizeEstimation}), a service that gives an
estimation of the current size of the network, DHT (in {\tt
  org.gnunet.dht.DistributedHashTable}), a service that allows
  key/value pairs to be stored distributed across the network, and PEERINFO
  (in {\tt org.gnunet.peerinfo.PeerInfo}), a service for
retrieving information about other known peers.

The API to the TESTBED service, which allows to manage multiple peers
for testing and evaluating GNUnet components, is partially implemented.

Among the most important APIs still missing are FS (filesharing) and NAMESTORE (manages GNS zone entries).

\section{Writing your own client and service}

GNUnet is split up into many components, with every component running in its own
process.  In the previous sections you have used existing APIs to
interface with other services written in C.  GNUnet-Java also provides
the tools necessary to both directly interface with services yourself and
write completely new services.

\subsection{The service configuration}
Each service has its own configuration, specifying basic information like the
executable file of the service (used by ARM), the port or socket used to reach it,
as well as configuration options specific to the service.

\exercise{%
  Look at the configuration file for the example service {\tt config/greeting.conf}
  and try to understand the meaning of each option, by looking at the comments and the source code
  of the example service.
}

\subsection{Writing a client}
The {\tt org.gnunet.util.Client} class allows you to connect to a GNUnet
service and exchange messages with it:
\begin{lstlisting}[language=java]
Client myClient = new Client("myservice", configuration);
\end{lstlisting}
In the above example, the configuration values for the clients are
taken from the configuration section {\tt myservice}.

Keep in mind that all configuration files either have to be in one of the default locations, or
specified on the command line with the {\tt -c CFGFILE} option.

\subsection{Writing a service}

To implement your own service, inherit {\tt org.gnunet.util.Service} instead of
{\tt org.gnunet.util.Program}. The main difference between {\tt Program} and {\tt Service} is that the
{\tt Service} also creates a {\tt Server}, which waits for messages from clients.
You can register a Runabout to receive messages from clients with
{\tt getServer().setHandler(myRunabout)}, in similar fashion to handling messages from {\tt core}.

Example:
\begin{lstlisting}[language=java]
public class MyService {
    public static void main(String... argv) {
        new Service(
            "greeting", // name of the service, for choosing the right configuration
            RelativeTime.MINUTE, // timeout for disconnecting idle clients
            true, // disallow messages of unknown type
            argv) { // command line arguments parsed by Service

            @Override
            public void run() {
                getServer().setHandler(new Server.MessageRunabout() {
                    public void visit(MyMessage m) {
                        [...]
                        getSender().receiveDone();
                    }
                });
            }
        }.start(args);
    }
}
\end{lstlisting}

Always remember to call {\tt getSender().receiveDone()}, as the server does not receive further messages
until {\tt receiveDone} is called, in order to support flow control.
The object returned by {\tt getSender()} has a {\tt notifyTransmitReady} method, which can be used
to send messages to clients in a similar fashion to sending messages with CORE.

\exercise{Look at the example service implemented in {\tt org.gnunet.ext.GreetingService}.
  Run the service ({\tt gnunet-service-greeting}), and connect to it with the client program
  ({\tt gnunet-greeting}).
  Try to understand the code, and modify both the client and the service so that can send and accept
  another message type.}
\exercise{Write an API for a GNUnet service that has not been implemented yet in gnunet-java and contribute it back to the project.}

\end{document}