taler-docs

taler-challenger-manual.rst (23292B)

---

 ..
   This file is part of GNU TALER.
 
   Copyright (C) 2023, 2024 Taler Systems SA
 
   TALER is free software; you can redistribute it and/or modify it under the
   terms of the GNU Affero General Public License as published by the Free Software
   Foundation; either version 3.0, or (at your option) any later version.
 
   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
 
   You should have received a copy of the GNU Affero General Public License along with
   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
 
   @author Christian Grothoff
   @author Florian Dold
 
 Challenger Operator Manual
 ##########################
 
 Introduction
 ============
 
 About Challenger
 ----------------
 
 Challenger is an OAuth 2.0-compatible address validation service.
 By redirecting a user-agent to a Challenger service a client can
 have Challenger validate that the user is able to receive messages
 at a particular address and obtain that address via the ``/info``
 endpoint.
 
 
 About this manual
 -----------------
 
 This manual targets system administrators who want to install,
 operate or integrate a challenger service.  To report issues
 or learn about known limitations, please check our
 `bug tracker <https://bugs.taler.net>`__.
 
 
 Architecture overview
 ---------------------
 
 The following picture gives an overview of the Challenger
 architecture and the main interactions:
 
 .. image:: images/challenger.png
 
 Here, the *resource owner* is a user that is in control
 of some *address* at a messaging service. This could be
 an e-mail account, a mobile phone number (for SMS), or
 a physical mail address (using the post office as the
 messaging service).
 
 The *resource owner* makes some request that requires
 some *client* to be in need of address validation. The
 *client* is registered with the Challenger OAuth 2.0
 service and first authorizes an address validation to
 be initiated. The client then redirects the resource
 owner to the Challenger service. In step (2), the resource
 owner submits the address that they claim to own.
 
 The Challenger service then creates a TAN code and
 submits it to the given address via a configurable
 *helper script* that is specific to the type of address
 being validated. When the resource owner submits the
 correct TAN code in step (6), they are given a token
 that they can provide to the client. Using this token
 the client can then finally obtain the now validated
 address in step (8).
 
 Address data, TAN codes and meta-data such as the number
 of failed attempts to submit a TAN code are recorded
 in a Postgres database by the Challenger service.
 
 .. _ChallengerInstallation:
 
 Installation
 ============
 
 In this guide's shell-session fragments, the command prompt shows two pieces
 of information:
 
 * Who is performing the command
   (``$user`` vs ``root``, and ending character ``$`` vs ``#``).
 
 
 Installing from source
 ----------------------
 
 The following instructions will show how to install libgnunetutil and
 the core GNU Taler libraries from source.
 
 The package sources can be find in our
 `download directory <http://ftpmirror.gnu.org/taler/>`__.
 
 GNU Taler components version numbers follow the ``MAJOR.MINOR.MICRO`` format.
 The general rule for compatibility is that ``MAJOR`` and ``MINOR`` must match.
 Exceptions to this general rule are documented in the release notes.
 For example, Challenger 1.3.0 should be compatible with Taler exchange 1.4.x
 as the MAJOR version matches.  A MAJOR version of 0 indicates experimental
 development, and you are expected to always run all of the *latest* releases
 together (no compatibility guarantees).
 
 First, the following packages need to be installed before we can compile the
 backend:
 
 .. include:: frags/list-of-dependencies.rst
 
 .. include:: frags/installing-gnunet.rst
 
 .. include:: frags/install-before-check.rst
 
 There is no need to actually run a GNUnet peer or a Taler exchange to use
 Challenger -- all Challenger needs from GNUnet and Taler are a number of
 headers and libraries!
 
 .. include:: frags/installing-taler-exchange.rst
 
 
 .. include:: frags/installing-challenger.rst
 
 .. include:: frags/install-before-check.rst
 
 
 
 Installing the Challenger binary packages on Debian
 ---------------------------------------------------
 
 .. include:: frags/installing-debian.rst
 
 To install the Challenger, you can now simply run:
 
 .. code-block:: shell-session
 
    # apt install challenger
 
 Note that the package does not perform any configuration work except for
 setting up the various users and the systemd service scripts. You still must
 configure at least the database, HTTP reverse proxy (typically with TLS
 certificates) and the terms of service.
 
 Installing the GNU Taler binary packages on Trisquel
 ----------------------------------------------------
 
 .. include:: frags/installing-trisquel.rst
 
 Installing the GNU Taler binary packages on Ubuntu
 --------------------------------------------------
 
 .. include:: frags/installing-ubuntu.rst
 
 To install the Taler exchange, you can now simply run:
 
 .. code-block:: shell-session
 
    # apt install challenger
 
 Note that the package does not perform any configuration work except for
 setting up the various users and the systemd service scripts. You still must
 configure at least the database, HTTP reverse proxy (typically with TLS
 certificates), and the terms of service.
 
 
 Services, users, groups and file system hierarchy
 -------------------------------------------------
 
 The *challenger* package will use several system users
 to compartmentalize different parts of the system:
 
 * ``challenger-httpd``: runs the HTTP daemon with the core business logic.
 * ``postgres``: runs the PostgreSQL database (from *postgresql* package).
 * ``www-data``: runs the frontend HTTPS service with the TLS keys (from *nginx* package).
 
 The package will deploy systemd service files in
 ``/usr/lib/systemd/system/`` for Challenger:
 
 * ``challenger-httpd.service``: the Challenger logic with the public REST API.
 
 
 Configuration Fundamentals
 ==========================
 
 This chapter provides fundamental details about the exchange configuration.
 
 The configuration for all Taler components uses a single configuration file
 as entry point: ``/etc/challenger/challenger.conf``.
 
 System defaults are automatically loaded from files in
 ``/usr/share/challenger/config.d``.  These default files should never be modified.
 
 The default configuration ``challenger.conf`` configuration file also includes all
 configuration files in ``/etc/challenger/conf.d``.
 
 To view the entire configuration annotated with the source of each configuration option, you
 can use the ``challenger-config`` helper:
 
 
 .. code-block:: shell-session
 
   [root@exchange-online]# challenger-config --diagnostics
   < ... annotated, full configuration ... >
 
 .. warning::
 
   While ``challenger-config`` also supports rewriting configuration files, we strongly
   recommend to edit configuration files manually, as ``challenger-config`` does not
   preserve comments and, by default, rewrites ``/etc/challenger/challenger.conf``.
 
 .. include:: frags/configuration-format.rst
 
 
 Fundamental Setup: Address validation
 -------------------------------------
 
 Each challenger service is designed to validate one type of address. Possible
 address types include:
 
 * phone numbers (via SMS)
 * e-mail addresses (via SMTP)
 * mail addresses (via postal service)
 
 In principle, additional types of addresses can easily be added by extending
 the respective HTML and programs to send challenges to the new address type.
 
 To make different types of address validations possible, the Challenger
 configuration contains two configuration options.
 
 (1) The ``ADDRESS_TYPE`` configuration option informs Challenger about the
     type of address it is expected to validate. It is returned as part of
     the OAuth 2.0 ``/info`` endpoint to the client, and is typically also
     used when deciding how to render the HTML form for address entry that is
     shown to the user.
 
 (2) The ``AUTH_COMMAND`` configuration option specifies which command
     Challenger should run to send a challenge to an address. The value may
     include arguments separated by spaces; quoting and escaping are not
     supported. The actual address is appended as the final argument, while
     the text with the challenge is passed to standard input. The command
     should terminate with a status code of 0 on success.
 
 .. code-block:: ini
   :caption: /etc/challenger/challenger.conf
 
    [challenger]
    ADDRESS_TYPE = email
    AUTH_COMMAND = challenger-send-email.sh
    # ... rest of file ...
 
 Challenger comes with ``AUTH_COMMAND`` shell scripts for sending e-mail, SMS
 and postal mail. Note that for SMS and postal mail the Challenger scripts use
 third party services to actually send the SMS or print and mail the postal
 mail. These third parties naturally charge money for their services, and thus
 the Challenger administrator will need to obtain an account with one of them.
 The scripts read the credentials from their **environment** rather than from a
 configuration file, so they are configured by setting those variables for the
 ``challenger-httpd`` service and not by editing the scripts; see
 *SMS transmission providers* below. In any case, these scripts should be
 primarily seen as *examples* on how to write authentication commands.
 
 .. note::
 
   We strongly welcome contributions for additional scripts with alternative
   providers or for new types of addresses.
 
 
 SMS transmission providers
 --------------------------
 
 The ``AUTH_COMMAND`` for phone numbers is ``challenger-send-sms``, which is not
 itself an SMS sender: it checks the number, then runs each provider helper in
 turn until one of them accepts the message. The helpers live next to it and are
 named ``challenger-send-sms-``\ *PROVIDER*; which of them are tried, and in
 which order, is the ``PROVIDERS`` variable near the top of the dispatcher
 script. A helper that is not installed or not executable is skipped.
 
 Credentials are **not** configuration options in ``challenger.conf``. They are
 read from the environment of the invoking daemon, so that they do not end up in
 a world-readable configuration file nor in the process table. With systemd they
 belong in a drop-in for the service, ideally through
 ``EnvironmentFile=`` pointing at a file readable only by the service user:
 
 .. code-block:: ini
   :caption: /etc/systemd/system/challenger-httpd.service.d/sms.conf
 
    [Service]
    EnvironmentFile=/etc/challenger/sms-credentials.env
 
 A helper whose credentials are missing or empty exits with status 40, which the
 dispatcher treats as a configuration failure and falls through to the next
 provider.
 
 .. list-table::
   :header-rows: 1
   :widths: 22 18 60
 
   * - Helper
     - Operated from
     - Environment
   * - ``challenger-send-sms-telesign``
     - United States
     - ``TELESIGN_AUTH_TOKEN``
   * - ``challenger-send-sms-clicksend``
     - Australia
     - ``CLICKSEND_USERNAME``, ``CLICKSEND_API_KEY``
   * - ``challenger-send-sms-aspsms``
     - Switzerland
     - ``ASPSMS_USERKEY``, ``ASPSMS_PASSWORD``
   * - ``challenger-send-sms-seven``
     - Germany
     - ``SEVEN_API_KEY``
   * - ``challenger-send-sms-gatewayapi``
     - Denmark
     - ``GATEWAYAPI_TOKEN``, optionally ``GATEWAYAPI_BASE_URL``
   * - ``challenger-send-sms-budgetsms``
     - Netherlands
     - ``BUDGETSMS_USERNAME``, ``BUDGETSMS_USERID``, ``BUDGETSMS_HANDLE``,
       optionally ``BUDGETSMS_SENDER``
 
 Two of the variables above are not credentials:
 
 ``GATEWAYAPI_BASE_URL``
   Which GatewayAPI platform to use. Optional, defaults to
   ``https://gatewayapi.eu``, the platform on which hosting, ownership and
   routing stay inside the European Union. Set it to ``https://gatewayapi.com``
   for an account on the global platform. An API token is only valid on the
   platform it was created for.
 
 ``BUDGETSMS_SENDER``
   Sender identity to show on the message, at most 11 alphanumeric or 16
   numeric characters. Optional, defaults to ``Challenger``. The BudgetSMS API
   requires a sender on every message; the other helpers use the sender
   configured on the provider account. Note that an alphanumeric sender has to
   be registered with the operator in a number of countries before messages
   using it are delivered.
 
 Only the first three helpers are listed in ``PROVIDERS`` as shipped. The three
 below them are operated inside the European Economic Area, which avoids
 transferring the phone number and the TAN — personal data that is at the same
 time an authentication secret — to a third country, and with it the need for a
 transfer instrument under Chapter V of the GDPR. They have been written against
 the published specifications of their APIs and are covered by the offline test
 suite, but have not yet been exercised against a live account, which is why a
 deployment has to enable them explicitly by adding their name to ``PROVIDERS``.
 
 Whether a failing provider causes the next one to be tried depends on the class
 of failure: a number that one provider rejects would be rejected by all of
 them, so that is reported immediately, whereas a provider that is down, out of
 credit, rate limited or misconfigured does not prevent another from succeeding.
 The full table of exit statuses is in challenger-send-sms(1), and each helper's
 manual page documents the subset it can report, its API, and its credentials.
 
 .. warning::
 
   A provider that accepts a message charges for it. Because a submission that
   times out may still have been accepted, a fall-through can send the same TAN
   twice; the dispatcher prefers that over not delivering a TAN at all.
 
 
 Legal conditions for using the service
 --------------------------------------
 
 .. include:: frags/legal.rst
 
 Database Configuration
 ----------------------
 
 The access credentials for the Challenger database are configured in
 ``/etc/challenger/challenger.conf``.  Currently, only PostgreSQL is
 supported as a database backend.
 
 .. note::
 
    The **challenger-dbconfig** tool can be used to automate the database
    setup. When using the Debian/Ubuntu packages, the user should already have
    been created, so you can just run the tool without any arguments and should
    have a working database configuration.  Subsequently, you should still run
    **taler-challenger-dbinit** as the ``challenger-httpd`` user to
    initialize the database schema.
 
 
 To create a database for Challenger on the local system, run:
 
 .. code-block:: shell-session
 
   [root@exchange-online]# su - postgres
   [postgres@exchange-online]# createuser challenger-httpd
   [postgres@exchange-online]# createdb -O challenger-httpd challenger
   [postgres@exchange-online]# exit
 
 This will create a ``challenger`` database owned by the ``taler-httpd`` user.
 We will use that user later to perform database maintenance operations.
 
 Assuming the above database setup, the database credentials to configure
 in the configuration file would simply be:
 
 .. code-block:: ini
   :caption: /etc/challenger/challenger.conf
 
   [challenger]
   DB = postgres
 
   [challenger-postgres]
   CONFIG = postgres:///challenger
 
 If the database is run on a different host, please follow the instructions
 from the PostgreSQL manual for configuring remote access.
 
 After configuring the database credentials, the Challenger database needs
 to be initialized with the following command:
 
 .. code-block:: shell-session
 
   [root@exchange-online]# sudo -u challenger-httpd challenger-dbinit
 
 .. note::
 
   To run this command, the user must have ``CREATE TABLE``, ``CREATE
   INDEX``, ``ALTER TABLE`` and (in the future possibly even) ``DROP TABLE``
   permissions.  Those permissions are only required for this step (which may
   have to be repeated when upgrading a deployment).  Afterwards, during
   normal operation, permissions to ``CREATE`` or ``ALTER`` tables are not
   required by Challenger and thus should not be granted.  For more
   information, see :doc:`manpages/challenger-dbinit.1`.
 
 
 Deployment
 ==========
 
 This chapter describes how to deploy Challenger once the basic installation
 and configuration are completed.
 
 .. _ChallengerServing:
 
 Serving
 -------
 
 The Challenger can serve HTTP over both TCP and UNIX domain socket.
 
 The following options are to be configured in the section ``[challenger]``:
 
 -  ``SERVE``: Must be set to ``tcp`` to serve HTTP over TCP, or ``unix`` to serve
    HTTP over a UNIX domain socket.
 
 -  ``PORT``: Set to the TCP port to listen on if ``SERVE`` is ``tcp``.
 
 -  ``UNIXPATH``: Set to the UNIX domain socket path to listen on if ``SERVE`` is
    ``unix``.
 
 - ``UNIXPATH_MODE``: Number giving the mode with the access permission mask
    for the ``UNIXPATH`` (i.e. 660 = ``rw-rw---``). Make sure to set it in such
    a way that your reverse proxy has permissions to access the UNIX domain
    socket.  The default (660) assumes that the reverse proxy is a member of
    the group under which the exchange HTTP server is running.
 
 .. _ChallengerReverseProxy:
 
 Reverse Proxy Setup
 -------------------
 
 By default, the ``challenger-httpd`` service listens for HTTP connections
 on a UNIX domain socket.  To make the service publicly available, a reverse
 proxy such as nginx should be used.  You must configure the reverse proxy
 to use TLS as this is required by OAuth 2.0.
 
 The ``challenger`` package ships with a sample configuration that can be
 enabled in nginx:
 
 .. code-block:: shell-session
 
   [root@exchange-online]# vim /etc/nginx/sites-available/challenger
   < ... customize configuration ... >
   [root@exchange-online]# ln -s /etc/nginx/sites-available/challenger \
                                 /etc/nginx/sites-enabled/challenger
   [root@exchange-online]# systemctl reload nginx
 
 
 Launching Challenger
 --------------------
 
 A running exchange requires starting the following processes:
 
 -   ``challenger-httpd`` (needs database access)
 
 The processes should be started via a hypervisor like
 ``systemd`` or ``gnunet-arm`` that automatically re-starts them should they
 have terminated unexpectedly.  Furthermore, the hypervisor
 *should* periodically re-start the service (say once per hour)
 to limit Postgres database memory utilization.
 
 .. note::
 
   The ``challenger-httpd`` does not ship with HTTPS enabled by default.
   It must thus be run behind an HTTPS reverse proxy that performs
   TLS termination on the same system.  Thus, it would typically be configured
   to listen on a UNIX domain socket.
 
 Given proper packaging, all of the above are realized via a simple systemd
 target. This enables Challenger to be properly started using a simple command:
 
 .. code-block:: shell-session
 
   # systemctl start challenger-httpd.service
 
 
 Authorizing clients
 -------------------
 
 Before clients can use Challenger, they must be explicitly configured. Each
 client is identified via its OAuth 2.0 REDIRECT URI.  Thus, a client must have
 exactly one REDIRECT URI. Challenger also does not allow multiple clients sharing the same REDIRECT URI.
 
 .. note::
 
   The OAuth 2.0 specification allows for a client to register
   zero or multiple REDIRECT URIs. However, zero is insecure
   as it creates an open redirector, and multiple REDIRECT URIs
   can trivially be implemented with Challenger by adding more
   clients.
 
 You can add or remove clients at any time; the Challenger service does not
 need to be running, but if it is you can still add or remove clients without
 restarting the service.  To add (or remove) a client, you must use the
 ``challenger-admin`` command:
 
 .. code-block:: shell-session
 
   # sudo -u challenger-httpd challenger-admin --add=$CLIENT_SECRET $CLIENT_REDIRECT_URI
 
 Here, ``$CLIENT_SECRET`` is the client secret of OAuth 2.0 which will be used in
 various parts of the protocol to authenticate the client. It must begin with the "secret-token:" prefix of RFC 8959.  The
 ``$CLIENT_REDIRECT_URI`` is the REDIRECT URI where the user-agent will be redirected to upon
 completion of the process.  The ``challenger-admin`` command will
 then output the *CLIENT_ID*, which will be a unique positive number.
 The first time you run the command, you will thus likely see:
 ``Client added. Client ID is: 1``.  This CLIENT_ID, the ``$CLIENT_SECRET``
 and the ``$CLIENT_REDIRECT_URI`` will form the foundation for the OAuth 2.0
 configuration.
 
 
 OAuth 2.0 integration
 ---------------------
 
 When integrating Challenger into an OAuth 2.0 process, you need to provide the
 three options from the previous section, but also the authorization, token and
 info endpoints.  For Challenger, these are ``/authorize``, ``/token`` and
 ``/info``.  However, the ``/authorize`` endpoint is special, as it is actually
 ``/authorize/$NONCE`` where ``$NONCE`` is a nonce that must be first requested
 by the client using the ``/setup/$CLIENT_ID`` endpoint!
 
 .. note::
 
   This extra step prevents user-agents from (ab)using the Challenger service
   to send challenges to addresses even when there is no authorized client
   that desires address validation. This is an important feature as address
   validation could be expensive.
 
 Thus, to generate the authorization URL, a client must first POST to
 ``/setup/$CLIENT_ID`` using their client secret in an ``Authorization: Bearer
 $CLIENT_SECRET`` HTTP header to obtain a fresh ``$NONCE``.  It is (optionally)
 possible to pass an address in the body of the ``/setup`` POST request. In
 this case, Challenger will pre-populate the address of the KYC form with the
 given body. Here, the format of the body is *address type*-specific.
 
 .. note::
 
   By passing a flag ``read_only: true`` editing of the address in the SPA can
   be disabled. However, the backend currently does not enforce this
   (see #9349).
 
 In the GNU Taler exchange configuration, this is indicated by appending
 ``#setup`` to the ``KYC_OAUTH2_AUTHORIZE_URL`` endpoint.  Be careful to quote
 the URL, as ``#`` is otherwise interpreted as the beginning of a comment by
 the configuration file syntax:
 
 .. code-block:: ini
   :caption: /etc/taler-exchange/conf.d/exchange-oauth2.conf
 
   [kyc-provider-example-oauth2]
   LOGIC = oauth2
   # (generic options omitted)
   KYC_OAUTH2_AUTHORIZE_URL = "https://challenger.example.com/authorize#setup"
   KYC_OAUTH2_TOKEN_URL = "https://challenger.example.com/token"
   KYC_OAUTH2_INFO_URL = "https://challenger.example.com/info"
   KYC_OAUTH2_CLIENT_ID = 1
   # Make sure to include the RFC 8959 prefix in "$SECRET"
   KYC_OAUTH2_CLIENT_SECRET = "$SECRET"
 
 
 
 Database management
 -------------------
 
 .. note::
 
   We advise to make good backups before experimenting with
   the database.
 
 To update the Challenger database after upgrading to a newer
 version of Challenger, you should simply re-run ``challenger-dbinit``.
 Without further options, this command is expected to preserve
 all data and only migrate the existing database to the latest
 schema:
 
 .. code-block:: console
 
    $ challenger-dbinit
 
 To delete stale data from the Challenger database, you can use
 garbage collection:
 
 .. code-block:: console
 
    $ challenger-dbinit --garbagecollect
 
 
 The Challenger database can be re-initialized using:
 
 .. code-block:: console
 
    $ challenger-dbinit --reset
 
 However, running this command will result in all data in the database
 being lost.