taler-challenger-manual.rst (24038B)
1 .. 2 This file is part of GNU TALER. 3 4 Copyright (C) 2023, 2024 Taler Systems SA 5 6 TALER is free software; you can redistribute it and/or modify it under the 7 terms of the GNU Affero General Public License as published by the Free Software 8 Foundation; either version 3.0, or (at your option) any later version. 9 10 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 11 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 12 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 13 14 You should have received a copy of the GNU Affero General Public License along with 15 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 16 17 @author Christian Grothoff 18 @author Florian Dold 19 20 Challenger Operator Manual 21 ########################## 22 23 Introduction 24 ============ 25 26 About Challenger 27 ---------------- 28 29 Challenger is an OAuth 2.0-compatible address validation service. 30 By redirecting a user-agent to a Challenger service a client can 31 have Challenger validate that the user is able to receive messages 32 at a particular address and obtain that address via the ``/info`` 33 endpoint. 34 35 36 About this manual 37 ----------------- 38 39 This manual targets system administrators who want to install, 40 operate or integrate a challenger service. To report issues 41 or learn about known limitations, please check our 42 `bug tracker <https://bugs.taler.net>`__. 43 44 45 Architecture overview 46 --------------------- 47 48 The following picture gives an overview of the Challenger 49 architecture and the main interactions: 50 51 .. image:: images/challenger.png 52 53 Here, the *resource owner* is a user that is in control 54 of some *address* at a messaging service. This could be 55 an e-mail account, a mobile phone number (for SMS), or 56 a physical mail address (using the post office as the 57 messaging service). 58 59 The *resource owner* makes some request that requires 60 some *client* to be in need of address validation. The 61 *client* is registered with the Challenger OAuth 2.0 62 service and first authorizes an address validation to 63 be initiated. The client then redirects the resource 64 owner to the Challenger service. In step (2), the resource 65 owner submits the address that they claim to own. 66 67 The Challenger service then creates a TAN code and 68 submits it to the given address via a configurable 69 *helper script* that is specific to the type of address 70 being validated. When the resource owner submits the 71 correct TAN code in step (6), they are given a token 72 that they can provide to the client. Using this token 73 the client can then finally obtain the now validated 74 address in step (8). 75 76 Address data, TAN codes and meta-data such as the number 77 of failed attempts to submit a TAN code are recorded 78 in a Postgres database by the Challenger service. 79 80 .. _ChallengerInstallation: 81 82 Installation 83 ============ 84 85 In this guide's shell-session fragments, the command prompt shows two pieces 86 of information: 87 88 * Who is performing the command 89 (``$user`` vs ``root``, and ending character ``$`` vs ``#``). 90 91 92 Installing from source 93 ---------------------- 94 95 The following instructions will show how to install libgnunetutil and 96 the core GNU Taler libraries from source. 97 98 The package sources can be find in our 99 `download directory <http://ftpmirror.gnu.org/taler/>`__. 100 101 GNU Taler components version numbers follow the ``MAJOR.MINOR.MICRO`` format. 102 The general rule for compatibility is that ``MAJOR`` and ``MINOR`` must match. 103 Exceptions to this general rule are documented in the release notes. 104 For example, Challenger 1.3.0 should be compatible with Taler exchange 1.4.x 105 as the MAJOR version matches. A MAJOR version of 0 indicates experimental 106 development, and you are expected to always run all of the *latest* releases 107 together (no compatibility guarantees). 108 109 First, the following packages need to be installed before we can compile the 110 backend: 111 112 .. include:: frags/list-of-dependencies.rst 113 114 .. include:: frags/installing-gnunet.rst 115 116 .. include:: frags/install-before-check.rst 117 118 There is no need to actually run a GNUnet peer or a Taler exchange to use 119 Challenger -- all Challenger needs from GNUnet and Taler are a number of 120 headers and libraries! 121 122 .. include:: frags/installing-taler-exchange.rst 123 124 125 .. include:: frags/installing-challenger.rst 126 127 .. include:: frags/install-before-check.rst 128 129 130 131 Installing the Challenger binary packages on Debian 132 --------------------------------------------------- 133 134 .. include:: frags/installing-debian.rst 135 136 To install the Challenger, you can now simply run: 137 138 .. code-block:: shell-session 139 140 # apt install challenger 141 142 Note that the package does not perform any configuration work except for 143 setting up the various users and the systemd service scripts. You still must 144 configure at least the database, HTTP reverse proxy (typically with TLS 145 certificates) and the terms of service. 146 147 Installing the GNU Taler binary packages on Trisquel 148 ---------------------------------------------------- 149 150 .. include:: frags/installing-trisquel.rst 151 152 Installing the GNU Taler binary packages on Ubuntu 153 -------------------------------------------------- 154 155 .. include:: frags/installing-ubuntu.rst 156 157 To install the Taler exchange, you can now simply run: 158 159 .. code-block:: shell-session 160 161 # apt install challenger 162 163 Note that the package does not perform any configuration work except for 164 setting up the various users and the systemd service scripts. You still must 165 configure at least the database, HTTP reverse proxy (typically with TLS 166 certificates), and the terms of service. 167 168 169 Services, users, groups and file system hierarchy 170 ------------------------------------------------- 171 172 The *challenger* package will use several system users 173 to compartmentalize different parts of the system: 174 175 * ``challenger-httpd``: runs the HTTP daemon with the core business logic. 176 * ``postgres``: runs the PostgreSQL database (from *postgresql* package). 177 * ``www-data``: runs the frontend HTTPS service with the TLS keys (from *nginx* package). 178 179 The package will deploy systemd service files in 180 ``/usr/lib/systemd/system/`` for Challenger: 181 182 * ``challenger-httpd.service``: the Challenger logic with the public REST API. 183 184 185 Configuration Fundamentals 186 ========================== 187 188 This chapter provides fundamental details about the exchange configuration. 189 190 The configuration for all Taler components uses a single configuration file 191 as entry point: ``/etc/challenger/challenger.conf``. 192 193 System defaults are automatically loaded from files in 194 ``/usr/share/challenger/config.d``. These default files should never be modified. 195 196 The default configuration ``challenger.conf`` configuration file also includes all 197 configuration files in ``/etc/challenger/conf.d``. 198 199 To view the entire configuration annotated with the source of each configuration option, you 200 can use the ``challenger-config`` helper: 201 202 203 .. code-block:: shell-session 204 205 [root@exchange-online]# challenger-config --diagnostics 206 < ... annotated, full configuration ... > 207 208 .. warning:: 209 210 While ``challenger-config`` also supports rewriting configuration files, we strongly 211 recommend to edit configuration files manually, as ``challenger-config`` does not 212 preserve comments and, by default, rewrites ``/etc/challenger/challenger.conf``. 213 214 .. include:: frags/configuration-format.rst 215 216 217 Fundamental Setup: Address validation 218 ------------------------------------- 219 220 Each challenger service is designed to validate one type of address. Possible 221 address types include: 222 223 * phone numbers (via SMS) 224 * e-mail addresses (via SMTP) 225 * mail addresses (via postal service) 226 227 In principle, additional types of addresses can easily be added by extending 228 the respective HTML and programs to send challenges to the new address type. 229 230 To make different types of address validations possible, the Challenger 231 configuration contains two configuration options. 232 233 (1) The ``ADDRESS_TYPE`` configuration option informs Challenger about the 234 type of address it is expected to validate. It is returned as part of 235 the OAuth 2.0 ``/info`` endpoint to the client, and is typically also 236 used when deciding how to render the HTML form for address entry that is 237 shown to the user. 238 239 (2) The ``AUTH_COMMAND`` configuration option specifies which command 240 Challenger should run to send a challenge to an address. The value may 241 include arguments separated by spaces; quoting and escaping are not 242 supported. The actual address is appended as the final argument, while 243 the text with the challenge is passed to standard input. The command 244 should terminate with a status code of 0 on success. 245 246 .. code-block:: ini 247 :caption: /etc/challenger/challenger.conf 248 249 [challenger] 250 ADDRESS_TYPE = email 251 AUTH_COMMAND = challenger-send-email 252 # ... rest of file ... 253 254 Challenger comes with ``AUTH_COMMAND`` shell scripts for sending e-mail, SMS 255 and postal mail. Note that for SMS and postal mail the Challenger scripts use 256 third party services to actually send the SMS or print and mail the postal 257 mail. These third parties naturally charge money for their services, and thus 258 the Challenger administrator will need to obtain an account with one of them. 259 The scripts read the credentials from their **environment** rather than from a 260 configuration file, so they are configured by setting those variables for the 261 ``challenger-httpd`` service and not by editing the scripts; see 262 *SMS transmission providers* below. In any case, these scripts should be 263 primarily seen as *examples* on how to write authentication commands. 264 265 Further variables, all optional and none of them credentials, set the texts 266 around the message. ``EMAIL_SUBJECT`` is the subject of the mail that 267 ``challenger-send-email`` sends (``KYC Challenger`` by default). 268 ``POSTAL_SUBJECT``, ``POSTAL_OPENING``, ``POSTAL_CLOSING`` and 269 ``POSTAL_SIGNATURE`` are the subject line, salutation, closing and signature 270 line of the letter that ``challenger-send-post`` posts; their defaults are 271 English, and the default signature is that of Taler Operations AG, so any other 272 operator should set at least ``POSTAL_SIGNATURE``. See 273 :doc:`manpages/challenger-send-email.1` and 274 :doc:`manpages/challenger-send-post.1`. 275 276 .. note:: 277 278 We strongly welcome contributions for additional scripts with alternative 279 providers or for new types of addresses. 280 281 282 SMS transmission providers 283 -------------------------- 284 285 The ``AUTH_COMMAND`` for phone numbers is ``challenger-send-sms``, which is not 286 itself an SMS sender: it checks the number, then runs each provider helper in 287 turn until one of them accepts the message. The helpers live next to it and are 288 named ``challenger-send-sms-``\ *PROVIDER*; which of them are tried, and in 289 which order, is the ``PROVIDERS`` variable near the top of the dispatcher 290 script. A helper that is not installed or not executable is skipped. 291 292 Credentials are **not** configuration options in ``challenger.conf``. They are 293 read from the environment of the invoking daemon, so that they do not end up in 294 a world-readable configuration file nor in the process table. With systemd they 295 belong in a drop-in for the service, ideally through 296 ``EnvironmentFile=`` pointing at a file readable only by the service user: 297 298 .. code-block:: ini 299 :caption: /etc/systemd/system/challenger-httpd.service.d/sms.conf 300 301 [Service] 302 EnvironmentFile=/etc/challenger/sms-credentials.env 303 304 A helper whose credentials are missing or empty exits with status 40, which the 305 dispatcher treats as a configuration failure and falls through to the next 306 provider. 307 308 .. list-table:: 309 :header-rows: 1 310 :widths: 22 18 60 311 312 * - Helper 313 - Operated from 314 - Environment 315 * - ``challenger-send-sms-telesign`` 316 - United States 317 - ``TELESIGN_AUTH_TOKEN`` 318 * - ``challenger-send-sms-clicksend`` 319 - Australia 320 - ``CLICKSEND_USERNAME``, ``CLICKSEND_API_KEY`` 321 * - ``challenger-send-sms-aspsms`` 322 - Switzerland 323 - ``ASPSMS_USERKEY``, ``ASPSMS_PASSWORD``, optionally ``ASPSMS_SENDER`` 324 * - ``challenger-send-sms-seven`` 325 - Germany 326 - ``SEVEN_API_KEY``, optionally ``SEVEN_SENDER`` 327 * - ``challenger-send-sms-gatewayapi`` 328 - Denmark 329 - ``GATEWAYAPI_TOKEN``, optionally ``GATEWAYAPI_BASE_URL`` 330 * - ``challenger-send-sms-budgetsms`` 331 - Netherlands 332 - ``BUDGETSMS_USERNAME``, ``BUDGETSMS_USERID``, ``BUDGETSMS_HANDLE``, 333 optionally ``BUDGETSMS_SENDER`` 334 335 Four of the variables above are not credentials: 336 337 ``GATEWAYAPI_BASE_URL`` 338 Which GatewayAPI platform to use. Optional, defaults to 339 ``https://gatewayapi.eu``, the platform on which hosting, ownership and 340 routing stay inside the European Union. Set it to ``https://gatewayapi.com`` 341 for an account on the global platform. An API token is only valid on the 342 platform it was created for. 343 344 ``ASPSMS_SENDER``, ``SEVEN_SENDER``, ``BUDGETSMS_SENDER`` 345 Sender identity to show on the message, at most 11 alphanumeric or 16 346 numeric characters. Optional; it defaults to ``Taler Ops`` for seven.io and 347 to ``Challenger`` for the other two. The GatewayAPI, Telesign and ClickSend 348 helpers use the sender configured on the provider account. Note that an 349 alphanumeric sender has to be registered with the operator in a number of 350 countries before messages using it are delivered. 351 352 As shipped, ``PROVIDERS`` tries ``gatewayapi``, ``budgetsms``, ``aspsms``, 353 ``seven``, ``telesign`` and ``clicksend``, in this order. GatewayAPI, BudgetSMS 354 and seven.io are operated inside the European Economic Area, which avoids 355 transferring the phone number and the TAN — personal data that is at the same 356 time an authentication secret — to a third country, and with it the need for a 357 transfer instrument under Chapter V of the GDPR. With the two of them first, a 358 message reaches a provider outside the EEA only when both have failed. To 359 change the order or leave a provider out, edit ``PROVIDERS``. 360 361 Whether a failing provider causes the next one to be tried depends on the class 362 of failure: a number that one provider rejects would be rejected by all of 363 them, so that is reported immediately, whereas a provider that is down, out of 364 credit, rate limited or misconfigured does not prevent another from succeeding. 365 The full table of exit statuses is in challenger-send-sms(1), and each helper's 366 manual page documents the subset it can report, its API, and its credentials. 367 368 .. warning:: 369 370 A provider that accepts a message charges for it. Because a submission that 371 times out may still have been accepted, a fall-through can send the same TAN 372 twice; the dispatcher prefers that over not delivering a TAN at all. 373 374 375 Legal conditions for using the service 376 -------------------------------------- 377 378 .. include:: frags/legal.rst 379 380 Database Configuration 381 ---------------------- 382 383 The access credentials for the Challenger database are configured in 384 ``/etc/challenger/challenger.conf``. Currently, only PostgreSQL is 385 supported as a database backend. 386 387 .. note:: 388 389 The **challenger-dbconfig** tool can be used to automate the database 390 setup. When using the Debian/Ubuntu packages, the user should already have 391 been created, so you can just run the tool without any arguments and should 392 have a working database configuration. Subsequently, you should still run 393 **taler-challenger-dbinit** as the ``challenger-httpd`` user to 394 initialize the database schema. 395 396 397 To create a database for Challenger on the local system, run: 398 399 .. code-block:: shell-session 400 401 [root@exchange-online]# su - postgres 402 [postgres@exchange-online]# createuser challenger-httpd 403 [postgres@exchange-online]# createdb -O challenger-httpd challenger 404 [postgres@exchange-online]# exit 405 406 This will create a ``challenger`` database owned by the ``taler-httpd`` user. 407 We will use that user later to perform database maintenance operations. 408 409 Assuming the above database setup, the database credentials to configure 410 in the configuration file would simply be: 411 412 .. code-block:: ini 413 :caption: /etc/challenger/challenger.conf 414 415 [challenger] 416 DB = postgres 417 418 [challenger-postgres] 419 CONFIG = postgres:///challenger 420 421 If the database is run on a different host, please follow the instructions 422 from the PostgreSQL manual for configuring remote access. 423 424 After configuring the database credentials, the Challenger database needs 425 to be initialized with the following command: 426 427 .. code-block:: shell-session 428 429 [root@exchange-online]# sudo -u challenger-httpd challenger-dbinit 430 431 .. note:: 432 433 To run this command, the user must have ``CREATE TABLE``, ``CREATE 434 INDEX``, ``ALTER TABLE`` and (in the future possibly even) ``DROP TABLE`` 435 permissions. Those permissions are only required for this step (which may 436 have to be repeated when upgrading a deployment). Afterwards, during 437 normal operation, permissions to ``CREATE`` or ``ALTER`` tables are not 438 required by Challenger and thus should not be granted. For more 439 information, see :doc:`manpages/challenger-dbinit.1`. 440 441 442 Deployment 443 ========== 444 445 This chapter describes how to deploy Challenger once the basic installation 446 and configuration are completed. 447 448 .. _ChallengerServing: 449 450 Serving 451 ------- 452 453 The Challenger can serve HTTP over both TCP and UNIX domain socket. 454 455 The following options are to be configured in the section ``[challenger]``: 456 457 - ``SERVE``: Must be set to ``tcp`` to serve HTTP over TCP, or ``unix`` to serve 458 HTTP over a UNIX domain socket. 459 460 - ``PORT``: Set to the TCP port to listen on if ``SERVE`` is ``tcp``. 461 462 - ``UNIXPATH``: Set to the UNIX domain socket path to listen on if ``SERVE`` is 463 ``unix``. 464 465 - ``UNIXPATH_MODE``: Number giving the mode with the access permission mask 466 for the ``UNIXPATH`` (i.e. 660 = ``rw-rw---``). Make sure to set it in such 467 a way that your reverse proxy has permissions to access the UNIX domain 468 socket. The default (660) assumes that the reverse proxy is a member of 469 the group under which the exchange HTTP server is running. 470 471 .. _ChallengerReverseProxy: 472 473 Reverse Proxy Setup 474 ------------------- 475 476 By default, the ``challenger-httpd`` service listens for HTTP connections 477 on a UNIX domain socket. To make the service publicly available, a reverse 478 proxy such as nginx should be used. You must configure the reverse proxy 479 to use TLS as this is required by OAuth 2.0. 480 481 The ``challenger`` package ships with a sample configuration that can be 482 enabled in nginx: 483 484 .. code-block:: shell-session 485 486 [root@exchange-online]# vim /etc/nginx/sites-available/challenger 487 < ... customize configuration ... > 488 [root@exchange-online]# ln -s /etc/nginx/sites-available/challenger \ 489 /etc/nginx/sites-enabled/challenger 490 [root@exchange-online]# systemctl reload nginx 491 492 493 Launching Challenger 494 -------------------- 495 496 A running exchange requires starting the following processes: 497 498 - ``challenger-httpd`` (needs database access) 499 500 The processes should be started via a hypervisor like 501 ``systemd`` or ``gnunet-arm`` that automatically re-starts them should they 502 have terminated unexpectedly. Furthermore, the hypervisor 503 *should* periodically re-start the service (say once per hour) 504 to limit Postgres database memory utilization. 505 506 .. note:: 507 508 The ``challenger-httpd`` does not ship with HTTPS enabled by default. 509 It must thus be run behind an HTTPS reverse proxy that performs 510 TLS termination on the same system. Thus, it would typically be configured 511 to listen on a UNIX domain socket. 512 513 Given proper packaging, all of the above are realized via a simple systemd 514 target. This enables Challenger to be properly started using a simple command: 515 516 .. code-block:: shell-session 517 518 # systemctl start challenger-httpd.service 519 520 521 Authorizing clients 522 ------------------- 523 524 Before clients can use Challenger, they must be explicitly configured. Each 525 client is identified via its OAuth 2.0 REDIRECT URI. Thus, a client must have 526 exactly one REDIRECT URI. Challenger also does not allow multiple clients sharing the same REDIRECT URI. 527 528 .. note:: 529 530 The OAuth 2.0 specification allows for a client to register 531 zero or multiple REDIRECT URIs. However, zero is insecure 532 as it creates an open redirector, and multiple REDIRECT URIs 533 can trivially be implemented with Challenger by adding more 534 clients. 535 536 You can add or remove clients at any time; the Challenger service does not 537 need to be running, but if it is you can still add or remove clients without 538 restarting the service. To add (or remove) a client, you must use the 539 ``challenger-admin`` command: 540 541 .. code-block:: shell-session 542 543 # sudo -u challenger-httpd challenger-admin --add=$CLIENT_SECRET $CLIENT_REDIRECT_URI 544 545 Here, ``$CLIENT_SECRET`` is the client secret of OAuth 2.0 which will be used in 546 various parts of the protocol to authenticate the client. It must begin with the "secret-token:" prefix of RFC 8959. The 547 ``$CLIENT_REDIRECT_URI`` is the REDIRECT URI where the user-agent will be redirected to upon 548 completion of the process. The ``challenger-admin`` command will 549 then output the *CLIENT_ID*, which will be a unique positive number. 550 The first time you run the command, you will thus likely see: 551 ``Client added. Client ID is: 1``. This CLIENT_ID, the ``$CLIENT_SECRET`` 552 and the ``$CLIENT_REDIRECT_URI`` will form the foundation for the OAuth 2.0 553 configuration. 554 555 556 OAuth 2.0 integration 557 --------------------- 558 559 When integrating Challenger into an OAuth 2.0 process, you need to provide the 560 three options from the previous section, but also the authorization, token and 561 info endpoints. For Challenger, these are ``/authorize``, ``/token`` and 562 ``/info``. However, the ``/authorize`` endpoint is special, as it is actually 563 ``/authorize/$NONCE`` where ``$NONCE`` is a nonce that must be first requested 564 by the client using the ``/setup/$CLIENT_ID`` endpoint! 565 566 .. note:: 567 568 This extra step prevents user-agents from (ab)using the Challenger service 569 to send challenges to addresses even when there is no authorized client 570 that desires address validation. This is an important feature as address 571 validation could be expensive. 572 573 Thus, to generate the authorization URL, a client must first POST to 574 ``/setup/$CLIENT_ID`` using their client secret in an ``Authorization: Bearer 575 $CLIENT_SECRET`` HTTP header to obtain a fresh ``$NONCE``. It is (optionally) 576 possible to pass an address in the body of the ``/setup`` POST request. In 577 this case, Challenger will pre-populate the address of the KYC form with the 578 given body. Here, the format of the body is *address type*-specific. 579 580 .. note:: 581 582 By passing a flag ``read_only: true`` editing of the address in the SPA can 583 be disabled. However, the backend currently does not enforce this 584 (see #9349). 585 586 In the GNU Taler exchange configuration, this is indicated by appending 587 ``#setup`` to the ``KYC_OAUTH2_AUTHORIZE_URL`` endpoint. Be careful to quote 588 the URL, as ``#`` is otherwise interpreted as the beginning of a comment by 589 the configuration file syntax: 590 591 .. code-block:: ini 592 :caption: /etc/taler-exchange/conf.d/exchange-oauth2.conf 593 594 [kyc-provider-example-oauth2] 595 LOGIC = oauth2 596 # (generic options omitted) 597 KYC_OAUTH2_AUTHORIZE_URL = "https://challenger.example.com/authorize#setup" 598 KYC_OAUTH2_TOKEN_URL = "https://challenger.example.com/token" 599 KYC_OAUTH2_INFO_URL = "https://challenger.example.com/info" 600 KYC_OAUTH2_CLIENT_ID = 1 601 # Make sure to include the RFC 8959 prefix in "$SECRET" 602 KYC_OAUTH2_CLIENT_SECRET = "$SECRET" 603 604 605 606 Database management 607 ------------------- 608 609 .. note:: 610 611 We advise to make good backups before experimenting with 612 the database. 613 614 To update the Challenger database after upgrading to a newer 615 version of Challenger, you should simply re-run ``challenger-dbinit``. 616 Without further options, this command is expected to preserve 617 all data and only migrate the existing database to the latest 618 schema: 619 620 .. code-block:: console 621 622 $ challenger-dbinit 623 624 To delete stale data from the Challenger database, you can use 625 garbage collection: 626 627 .. code-block:: console 628 629 $ challenger-dbinit --garbagecollect 630 631 632 The Challenger database can be re-initialized using: 633 634 .. code-block:: console 635 636 $ challenger-dbinit --reset 637 638 However, running this command will result in all data in the database 639 being lost.