taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

taler-developer-manual.rst (65522B)


      1 ..
      2   This file is part of GNU TALER.
      3 
      4   Copyright (C) 2014-2025 Taler Systems SA
      5 
      6   TALER is free software; you can redistribute it and/or modify it under the
      7   terms of the GNU Affero General Public License as published by the Free Software
      8   Foundation; either version 2.1, or (at your option) any later version.
      9 
     10   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
     11   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
     12   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
     13 
     14   You should have received a copy of the GNU Affero General Public License along with
     15   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
     16 
     17   @author Christian Grothoff
     18 
     19 General Developer Manual
     20 ########################
     21 
     22 .. note::
     23 
     24   This manual contains information for developers working on GNU Taler
     25   and related components.  It is not intended for a general audience.
     26 
     27 
     28 Project Overview
     29 ================
     30 
     31 GNU Taler consists of a large (and growing) number of components
     32 in various Git repositories.  The following list gives a first
     33 overview:
     34 
     35 * exchange: core payment processing logic with a REST API, plus various
     36   helper processes for interaction with banks and cryptographic
     37   computations. Also includes the logic for the auditor and an
     38   in-memory "bank" API implementation for testing.
     39 
     40 * libeufin: implementation of the "bank" API using the EBICS protocol
     41   used by banks in the EU.  Allows an exchange to interact with
     42   European banks.
     43 
     44 * taler-magnet-bank: implementation of the "bank" API using the Magnet Bank
     45   API.Allows an exchange to interact with Magnet Bank.
     46 
     47 * taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network.
     48 
     49 * depolymerization: implementation of the "bank" API on top of
     50   blockchains, specifically Bitcoin and Ethereum. Allows an exchange
     51   to interact with crypto-currencies.
     52 
     53 * merchant: payment processing backend to be run by merchants,
     54   offering a REST API.
     55 
     56 * wallet-core: platform-independent implementation of a wallet to be run by
     57   normal users. Includes also the WebExtension for various browsers.
     58   Furthermore, includes various single-page apps used by other
     59   components (especially as libeufin and merchant).  Also includes
     60   command-line wallet and tools for testing.
     61 
     62 * taler-android: Android Apps including the Android wallet, the
     63   Android point-of-sale App and the Android casher app.
     64 
     65 * taler-ios: iOS wallet App.
     66 
     67 * sync: backup service, provides a simple REST API to allow users to
     68   make encrypted backups of their wallet state.
     69 
     70 * anastasis: key escrow service, provides a simple REST API to allow
     71   users to distribute encryption keys across multiple providers and
     72   define authorization policies for key recovery.
     73 
     74 * taler-mdb: integration of Taler with the multi-drop-bus (MDB) API
     75   used by vending machines. Allows Taler payments to be integrated
     76   with vending machines.
     77 
     78 * gnu-taler-payment-for-woocommerce: payment plugin for the
     79   woocommerce (wordpress) E-commerce solution.
     80 
     81 * twister: man-in-the-middle proxy for tests that require fuzzing a
     82   REST/JSON protocol.  Used for some of our testing.
     83 
     84 * challenger: implementation of an OAuth 2.0 provider that can be used
     85   to verify that a user can receive SMS or E-mail at particular addresses.
     86   Used as part of KYC processes of the exchange.
     87 
     88 * taler-mailbox: messaging service used to store and forward payment
     89   messages to Taler wallets.
     90 
     91 * taldir: directory service used to lookup Taler wallet addresses for
     92   sending invoices or payments to other wallets.
     93 
     94 * taler-merchant-demos: various demonstration services operated at
     95   'demo.taler.net', including a simple shop and a donation page.
     96 
     97 There are other important repositories without code, including:
     98 
     99 * gana: Hosted on git.gnunet.org, this repository defines various
    100   constants used in the GNU Taler project.
    101 
    102 * docs: documentation, including this very document.
    103 
    104 * marketing: various presentations, papers and other resources for
    105   outreach.
    106 
    107 * large-media: very large data objects, such as videos.
    108 
    109 * www: the taler.net website.
    110 
    111 Fundamentals
    112 ============
    113 
    114 Versioning
    115 ----------
    116 
    117 A central rule is to never break anything for any dependency. To accomplish
    118 this, we use versioning, of the APIs, database schema and the protocol.  The
    119 database versioning approach is described in the :ref:`Database schema
    120 versioning <DatabaseVersioning>` section.  Here, we will focus on API and
    121 protocol versioning.
    122 
    123 The key issue we need to solve with protocols and APIs (and that does not
    124 apply to database versioning) is being able to introduce and remove features
    125 without requiring a flag day where all components must update at the same
    126 time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE
    127 and *not* semantic versioning (SEMVER).  With GNU libtool style versioning,
    128 first the REVISION should be increased on every change to the respective code.
    129 Then, each time a feature is introduced or deprecated, the MAJOR and AGE
    130 numbers are increased. Whenever an API is actually removed the AGE number is
    131 reduced to match the distance since the removed API was deprecated.  Thus, if
    132 some client implements version X of the protocol (including not using any APIs
    133 that have been deprecated), it is compatible for any implementation where
    134 MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X.
    135 REVISION is not used for expected compatibility issues and merely serves to
    136 uniquely identify each version (in combination with MAJOR).
    137 
    138 To evolve any implementation, it is thus critical to first of all never
    139 just break an existing API or endpoint. The only acceptable modifications
    140 are to return additional information (being aware of binary compatibility!)
    141 or to accept additional optional arguments (again, in a way that does not
    142 break existing users). Thus, the most common way to introduce changes will
    143 be the addition of new endpoints. Breaking existing endpoints is only ever
    144 at best acceptable while in the process of introducing it and if you are
    145 absolutely sure that there are zero users in other components.
    146 
    147 When removing endpoints (or fields being returned), you must first deprecate
    148 the existing API (incrementing MAJOR and AGE) and then wait for all clients,
    149 including all clients in operation (e.g. Android and iOS Apps, e-commerce
    150 integrations, etc.) to upgrade to a protocol implementation above the
    151 deprecated MAJOR revision. Only then you should remove the endpoint and reduce
    152 AGE.
    153 
    154 To document these changes, please try to use ``@since`` annotations in the API
    155 specifications to explain the MAJOR revision when a feature became available,
    156 but most importantly use ``@deprecated X`` annotations to indicate that an API
    157 was deprecated and will be removed once MAJOR minus AGE is above X. When using
    158 an API, use the ``/config`` endpoints to check for compatibility and show a
    159 warning if the version(s) you support and the version(s) offered by the server
    160 are incompatible.
    161 
    162 
    163 Tagging and Package Versioning
    164 ------------------------------
    165 
    166 Release tags are of the form ``v${major}.${minor}.${patch}``.  Release tags *should* be
    167 annotated git tags.
    168 
    169 We usually consider Debian packaging files (in ``debian/``) to be part of a release.
    170 When only the Debian packaging files need to be changed, there are two options:
    171 
    172 * Make a new patch release (``v${major}.${minor}.${patch+1}``)
    173 * Make a Debian release:
    174 
    175   * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}``
    176   * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}``
    177 
    178 All source repos *should* include a ``contrib/bump`` script that automates bumping the\
    179 version in all relevant source and packaging files.
    180 In the future, we might add an option to the script to only release a packaging bump.
    181 Right now, that process is manual.
    182 
    183 We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``.
    184 The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``.
    185 
    186 Nightly Debian packages should follow the `Debian conventions <https://wiki.debian.org/Versioning>`__ of ``{upcoming_version}~git{date}.{hash}-{revision}``.
    187 
    188 Testing Tools
    189 -------------
    190 
    191 For full ``make check`` support, install these programs:
    192 
    193 - `jq <https://github.com/stedolan/jq>`__
    194 - `curl <http://curl.haxx.se>`__
    195 - `faketime <https://github.com/wolfcw/libfaketime>`__
    196 
    197 The ``make check`` should be able to function without them, but
    198 their presence permits some tests to run that would otherwise be skipped.
    199 
    200 Manual Testing Database Reset
    201 -----------------------------
    202 
    203 Sometimes ``make check`` will fail with some kind of database (SQL)
    204 error, perhaps with a message like ``OBJECT does not exist`` in the
    205 ``test-suite.log`` file, where ``OBJECT`` is the name of a table or function.
    206 In that case, it may be necessary to reset the ``talercheck`` database
    207 with the commands:
    208 
    209 .. code-block:: console
    210 
    211    $ dropdb talercheck
    212    $ createdb talercheck
    213 
    214 This is because, at the moment, there is no support for
    215 doing these steps automatically in the ``make check`` flow.
    216 
    217 (If ``make check`` still fails after the reset, file a bug report as usual.)
    218 
    219 Bug Tracking
    220 ------------
    221 
    222 Bug tracking is done with Mantis (https://www.mantisbt.org/).  The bug tracker
    223 is available at `<https://bugs.taler.net>`_. A registration on the Web site is
    224 needed in order to use the bug tracker, only read access is granted without a
    225 login.
    226 
    227 We use the following conventions for the bug states:
    228 
    229 * NEW: Incoming bugs are in 'new' so that management (or developers)
    230   can easily identify those that need to be checked (report correct?
    231   something we want to fix?), prioritized and targeted for releases.
    232   "NEW" bugs are never assigned to a developer.
    233 
    234 * FEEDBACK: When blocked on feedback from reporter or other developer.
    235   Assigned to other developer (but cannot be assigned to reporter,
    236   in this case MAY remain associated with the developer who expects
    237   the feedback). If a bug is on feedback, it automatically should be
    238   considered to be high-priority to give the feedback (as it is
    239   blocking someone else!).
    240 
    241 * ACKNOWLEDGED: The bug has been reviewed, but no decision about
    242   what action to take has been made yet. Should not be worked on
    243   until management (or a developer) comes up with a plan.
    244   "ACKNOWLEDGED" bugs should NOT be assigned to a developer.
    245 
    246 * CONFIRMED: This is a real issue that should be worked on, but
    247   is not yet actively worked on. If working on this bug requires
    248   other bugs to be fixed first, they should be added as
    249   child-bugs (via relationships). Developers are always welcome
    250   to self-assign bugs that are "CONFIRMED" if they start to work
    251   on a bug.  "CONFIRMED" bugs should NOT be assigned to a developer.
    252 
    253 * ASSIGNED: The specific developer the bug is assigned to is
    254   **actively** working on the issue. Developers should strive to
    255   not have more than 5-10 bugs assigned to them at any time.
    256   Only having one assigned to you is totally OK!
    257   Developers should aggressively un-assign bugs that they are
    258   blocked on, cannot make progress on, or are no longer actively
    259   working on (but of course, better *resolve* them before
    260   moving on if possible). If the bug remains open, it probably
    261   should go back to "CONFIRMED" or "ACKNOWLEDGED".
    262 
    263 * RESOLVED: The bug has been fixed in Git.
    264 
    265 * CLOSED: An official release was made with the fix in it.
    266 
    267 When developers want to keep an eye on certain bugs, they should
    268 **monitor** them. Multiple developers can be monitoring a bug, but
    269 it can only be assigned to one.  Developers should also keep an
    270 eye on the roadmap (by release), bug categories they care about,
    271 and of course priorities / severities.
    272 
    273 We use **tags** to categorize bugs. Common tags that also imply
    274 some urgency include (in alphabetical order):
    275 
    276 * accounting: issues required for accounting (such as taxes by merchants)
    277 * compliance: issues related to regulatory compliance
    278 * $CUSTOMER: issues requested by a particular customer
    279 * performance: performance problems or ideas for improvement
    280 * security: security issues (including planned improvements to security)
    281 * UX: user experience issues
    282 
    283 These tags **should** be attached to "NEW" bugs if they apply.
    284 
    285 
    286 Code Repositories
    287 -----------------
    288 
    289 Taler code is versioned with Git. For those users without write access, all the
    290 codebases are found at the following URL:
    291 
    292 .. code-block:: none
    293 
    294    git://git.taler.net/<repository>
    295 
    296 A complete list of all the existing repositories is currently found at
    297 `<https://git.taler.net/>`_.
    298 
    299 
    300 Committing code
    301 ---------------
    302 
    303 Before you can obtain Git write access, you must sign the copyright
    304 agreement. As we collaborate closely with GNUnet, we use their
    305 copyright agreement -- with the understanding that your contributions
    306 to GNU Taler are included in the assignment.  You can find the
    307 agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_.
    308 Please sign and mail it to Christian Grothoff as he currently collects
    309 all the documents for GNUnet e.V.
    310 
    311 To obtain Git access, you need to send us your SSH public key. Most core
    312 team members have administrative Git access, so simply contact whoever
    313 is your primary point of contact so far. You can
    314 find instructions on how to generate an SSH key
    315 in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_.
    316 If you have been granted write access, you first of all must change the URL of
    317 the respective repository to:
    318 
    319 .. code-block:: none
    320 
    321    ssh://git@git.taler.net/<repository>
    322 
    323 For an existing checkout, this can be done by editing the ``.git/config`` file.
    324 
    325 The server is configured to reject all commits that have not been signed with
    326 GnuPG. If you do not yet have a GnuPG key, you must create one, as explained
    327 in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_.
    328 You do not need to share the respective public key with us to make commits.
    329 However, we recommend that you upload it to key servers, put it on your
    330 business card and personally meet with other GNU hackers to have it signed
    331 such that others can verify your commits later.
    332 
    333 To sign all commits, you should run
    334 
    335 .. code-block:: console
    336 
    337    $ git config --global commit.gpgsign true
    338 
    339 You can also sign individual commits only by adding the ``-S`` option to the
    340 ``git commit`` command. If you accidentally already made commits but forgot
    341 to sign them, you can retroactively add signatures using:
    342 
    343 .. code-block:: console
    344 
    345    $ git rebase -S
    346 
    347 
    348 Whether you commit to a personal branch (recommended: ``dev/$USER/...``),
    349 a feature branch or to ``master`` should
    350 depend on your level of comfort and the nature of the change.  As a general
    351 rule, the code in ``master`` must always build and tests should always pass, at
    352 least on your own system. However, we all make mistakes and you should expect
    353 to receive friendly reminders if your change did not live up to this simple
    354 standard.  We plan to move to a system where the CI guarantees this invariant
    355 in the future.
    356 
    357 In order to keep a linear and clean commits history, we advise to avoid
    358 merge commits and instead always rebase your changes before pushing to
    359 the ``master`` branch.  If you commit and later find out that new commits were
    360 pushed, the following command will pull the new commits and rebase yours
    361 on top of them.
    362 
    363 .. code-block:: console
    364 
    365    # -S instructs Git to (re)sign your commits
    366    $ git pull --rebase -S
    367 
    368 
    369 
    370 Observing changes
    371 -----------------
    372 
    373 Every commit to the ``master`` branch of any of our public repositories
    374 (and almost all are public) is automatically sent to the
    375 gnunet-svn@gnu.org mailinglist.  That list is for Git commits only,
    376 and must not be used for discussions. It also carries commits from
    377 our main dependencies, namely GNUnet and GNU libmicrohttpd.  While
    378 it can be high volume, the lists is a good way to follow overall
    379 development.
    380 
    381 
    382 Code generator usage policy
    383 ---------------------------
    384 
    385 We do neither encourage nor discourage the use of tools for code generation.
    386 It is up to the individual developer to decide if a tool is acceptable for
    387 a particular task. But of course, we do encourage you to use FLOSS tools
    388 and we MUST NOT become dependend on non-free software! That said, if you
    389 use tools, you must document their use and in particular satisfy the
    390 `NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__.
    391 
    392 Specifically, we ask developers to always put generated code into a *separate*
    393 Git commit and to include the full prompt in the commit message. Naturally,
    394 you may clean up the code generator's output, but then you should do so in
    395 separate Git commits (and of course only merge into master/stable after the
    396 clean up is complete). But do preserve (not squash!) the commit with the
    397 generated code so that it remains documented what the prompts were and which
    398 code is generated.  This will go a long way to keep code auditors sane!
    399 
    400 
    401 Communication
    402 -------------
    403 
    404 For public discussions we use the taler@gnu.org mailinglist.  All developers
    405 should subscribe to the low-volume Taler mailinglist.  There are separate
    406 low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd
    407 (@gnu.org).  For internal discussions we use https://mattermost.taler.net/
    408 (invitation only, but also achieved).
    409 
    410 
    411 What to put in bootstrap
    412 ------------------------
    413 
    414 Each repository has a ``bootstrap`` script, which contains commands for the
    415 developer to run after a repository checkout (i.e., after ``git clone`` or
    416 ``git pull``).
    417 Typically, this updates and initializes submodules, prepares the tool chain,
    418 and runs ``autoreconf``.
    419 The last step generates the ``configure`` script, whether for immediate use or
    420 for inclusion in the distribution tarball.
    421 
    422 One common submodule is ``contrib/gana``, which pulls from the
    423 `GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__.
    424 For example, in the
    425 `Taler exchange repository <https://git.taler.net/exchange.git>`__,
    426 the bootstrap script eventually runs the ``git submodule update --init`` command
    427 early on, and later runs script ``./contrib/gana-generate.sh``, which
    428 generates files such as ``src/include/taler_signatures.h``.
    429 
    430 Thus, to update that file, you need to:
    431 
    432 - (in GANA repo) Find a suitable (unused) name and number for the Signature
    433   Purposes database.
    434 
    435 - Add it to GANA, in ``gnunet-signatures/registry.rec``.
    436   (You can check for uniqueness with the ``recfix`` utility.)
    437 
    438 - Commit the change, and push it to the GANA Git repo.
    439 
    440 - (in Taler Repo) Run the ``contrib/gana-latest.sh`` script.
    441 
    442 - Bootstrap, configure, do ``make install``, ``make check``, etc.
    443   (Basically, make sure the change does not break anything.)
    444 
    445 - Commit the submodule change, and push it to the Taler exchange Git repo.
    446 
    447 A similar procedure is required for other databases in GANA.
    448 See file ``README`` in the various directories for specific instructions.
    449 
    450 
    451 Debian and Ubuntu Repositories
    452 ==============================
    453 
    454 We package our software for Debian and Ubuntu.
    455 
    456 Nightly Repositories
    457 --------------------
    458 
    459 To try the latest, unstable and untested versions of packages,
    460 you can add the nightly package sources.
    461 
    462 .. code-block:: shell-session
    463 
    464    # For Debian (trixie)
    465    $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \
    466      | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources
    467 
    468 
    469 Taler Deployment on gv.taler.net
    470 ================================
    471 
    472 This section describes the GNU Taler deployment on ``gv.taler.net``.  ``gv``
    473 is our server at BFH. It hosts the Git repositories, Web sites, CI and other
    474 services.  Developers can receive an SSH account and e-mail alias for the
    475 system, you should contact Javier, Christian or Florian.  As with Git, ask
    476 your primary team contact for shell access if you think you need it.
    477 
    478 
    479 DNS
    480 ---
    481 
    482 DNS records for taler.net are controlled by the GNU Taler maintainers,
    483 specifically Christian and Florian, and our system administrator, Javier. If
    484 you need a sub-domain to be added, please contact one of them.
    485 
    486 
    487 User Acccounts
    488 --------------
    489 
    490 On ``gv.taler.net``, there are three system users that are set up to
    491 serve Taler on the Internet:
    492 
    493 -  ``head``: serves ``*.head.taler.net`` and gets automatically
    494    built by Buildbot every 2 hours from the ``sandcastle-ng.git``.
    495    Master key may be reset occasionally
    496 
    497 -  ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get
    498    automatically built, and runs more recent tags and/or unreleased
    499    versions of Taler components. Master key may be reset
    500    occasionally.
    501 
    502 - ``demo``: serves ``*.demo.taler.net``.  Never automatically built.
    503   Master key is retained.
    504 
    505 Demo Upgrade Procedure
    506 ======================
    507 
    508 #. Login as the ``demo`` user on ``gv.taler.net``.
    509 #. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``.
    510 #. Run ``systemctl --user restart container-taler-sandcastle-demo.service``
    511 #. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/)
    512    for more info.
    513 
    514 
    515 Upgrading the ``demo`` environment should be done with care, and ideally be
    516 coordinated on the mailing list before.  It is our goal for ``demo`` to always
    517 run a "working version" that is compatible with various published wallets.
    518 Please use the :doc:`demo upgrade checklist <../checklists/checklist-demo-upgrade>` to make
    519 sure everything is working.
    520 Nginx is already configured to reach the services as exported by the user unit.
    521 
    522 
    523 Tagging components
    524 ------------------
    525 
    526 All Taler components must be tagged with git before they are deployed on the
    527 ``demo`` environment, using a tag of the following form:
    528 
    529 .. code-block:: none
    530 
    531   demo-YYYY-MM-DD-SS
    532   YYYY = year
    533   MM = month
    534   DD = day
    535   SS = serial
    536 
    537 Environments and Builders on taler.net
    538 ======================================
    539 
    540 Buildbot implementation
    541 -----------------------
    542 
    543 GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration.  Buildbot documentation is at https://docs.buildbot.net/.
    544 
    545 Here are some highlights:
    546 
    547 - The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed.  The WORKER executes the commands that perform all end-functions of buildbot.
    548 
    549 - The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command.
    550 
    551 - The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost.
    552 
    553 - The FACTORY is tied to the WORKER in master.cfg by a BUILDER.
    554 
    555 - The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed.
    556 
    557 - Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed).  Do not edit this file directly/locally on taler.net, but check changes into Git.
    558 
    559 
    560 Best Practices:
    561 
    562 - When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from.  (At this time, taler.net is the only server used by this implementation, but it's still good practice.)
    563 
    564 - Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>``
    565 
    566 Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")``
    567 
    568 Test builder
    569 ------------
    570 
    571 This builder (``test-builder``) compiles and starts every Taler component.
    572 The associated worker is run by the ``taler-test`` Gv user, via the SystemD
    573 unit ``buildbot-worker-taler``.  The following commands start/stop/restart
    574 the worker:
    575 
    576 .. code-block::
    577 
    578    systemctl --user start buildbot-worker-taler
    579    systemctl --user stop buildbot-worker-taler
    580    systemctl --user restart buildbot-worker-taler
    581 
    582 .. note::
    583   the mentioned unit file can be found at ``deployment.git/systemd-services/``
    584 
    585 Wallet builder
    586 --------------
    587 
    588 This builder (``wallet-builder``) compiles every Taler component
    589 and runs the wallet integration tests.  The associated worker is
    590 run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``.
    591 The following commands start/stop/restart the worker:
    592 
    593 .. code-block::
    594 
    595    systemctl --user start buildbot-worker-wallet
    596    systemctl --user stop buildbot-worker-wallet
    597    systemctl --user restart buildbot-worker-wallet
    598 
    599 .. note::
    600   the mentioned unit file can be found at ``deployment.git/systemd-services/``
    601 
    602 Documentation Builder
    603 ---------------------
    604 
    605 All the Taler documentation is built by the user ``docbuilder`` that
    606 runs a Buildbot worker.  The following commands set the ``docbuilder`` up,
    607 starting with an empty home directory.
    608 
    609 .. code-block:: console
    610 
    611   # Log-in as the 'docbuilder' user.
    612 
    613   $ cd $HOME
    614   $ git clone git://git.taler.net/deployment
    615   $ ./deployment/bootstrap-docbuilder
    616 
    617   # If the previous step worked, the setup is
    618   # complete and the Buildbot worker can be started.
    619 
    620   $ buildbot-worker start worker/
    621 
    622 
    623 Website Builder
    624 ---------------
    625 
    626 
    627 Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the
    628 user ``taler-websites`` by the means of a Buildbot worker.  The following
    629 commands set the ``taler-websites`` up, starting with an empty home directory.
    630 
    631 .. code-block:: console
    632 
    633   # Log-in as the 'taler-websites' user.
    634 
    635   $ cd $HOME
    636   $ git clone git://git.taler.net/deployment
    637   $ ./deployment/bootstrap-sitesbuilder
    638 
    639   # If the previous step worked, the setup is
    640   # complete and the Buildbot worker can be started.
    641 
    642   $ buildbot-worker start worker/
    643 
    644 
    645 Code coverage
    646 -------------
    647 
    648 Code coverage tests are run by the ``lcovworker`` user, and are also driven
    649 by Buildbot.
    650 
    651 .. code-block:: console
    652 
    653   # Log-in as the 'lcovworker' user.
    654 
    655   $ cd $HOME
    656   $ git clone git://git.taler.net/deployment
    657   $ ./deployment/bootstrap-taler lcov
    658 
    659   # If the previous step worked, the setup is
    660   # complete and the Buildbot worker can be started.
    661 
    662   $ buildbot-worker start worker/
    663 
    664 The results are then published at ``https://lcov.taler.net/``.
    665 
    666 Producing auditor reports
    667 -------------------------
    668 
    669 Both 'test' and 'demo' setups get their auditor reports compiled
    670 by a Buildbot worker.  The following steps get the reports compiler
    671 prepared.
    672 
    673 .. code-block:: console
    674 
    675   # Log-in as <env>-auditor, with <env> being either 'test' or 'demo'
    676 
    677   $ git clone git://git.taler.net/deployment
    678   $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env>
    679 
    680   # If the previous steps worked, then it should suffice to start
    681   # the worker, with:
    682 
    683   $ buildbot-worker start worker/
    684 
    685 
    686 .. _DatabaseVersioning:
    687 
    688 Database schema versioning
    689 --------------------------
    690 
    691 The PostgreSQL databases of the exchange and the auditor are versioned.
    692 See the ``versioning.sql`` file in the respective directory for documentation.
    693 
    694 Every set of changes to the database schema must be stored in a new
    695 versioned SQL script. The scripts must have contiguous numbers. After
    696 any release (or version being deployed to a production or staging
    697 environment), existing scripts MUST be immutable.
    698 
    699 Developers and operators MUST NOT make changes to database schema
    700 outside of this versioning.  All tables of a GNU Taler component should live in their own schema.
    701 
    702 
    703 QA Plans
    704 ========
    705 
    706 .. include:: ../checklists/qa-1.0.rst
    707 
    708 
    709 Releases
    710 ========
    711 
    712 .. include:: ../checklists/checklist-release.rst
    713 
    714 Release Process
    715 ---------------
    716 
    717 This document describes the process for releasing a new version of the
    718 various Taler components to the official GNU mirrors.
    719 
    720 The following components are published on the GNU mirrors
    721 
    722 -  taler-exchange (exchange.git)
    723 -  taler-merchant (merchant.git)
    724 -  sync (sync.git)
    725 -  taler-mdb (taler-mdb.git)
    726 -  libeufin (libeufin.git)
    727 -  challenger (challenger.git)
    728 -  wallet-core (wallet-core.git)
    729 
    730 Tagging
    731 -------
    732 
    733 Tag releases with an **annotated** commit, like
    734 
    735 .. code-block:: console
    736 
    737    $ git tag -a v0.1.0 -m "Official release v0.1.0"
    738    $ git push origin v0.1.0
    739 
    740 
    741 Database for tests
    742 ------------------
    743 
    744 For tests in the exchange and merchant to run, make sure that a database
    745 *talercheck* is accessible by *$USER*. Otherwise tests involving the
    746 database logic are skipped.
    747 
    748 .. include:: ../frags/db-stores-sensitive-data.rst
    749 
    750 Exchange, merchant
    751 ------------------
    752 
    753 Set the version in ``configure.ac``. The commit being tagged should be
    754 the change of the version.
    755 
    756 Tag the current GANA version that works with the exchange and merchant and
    757 checkout that tag of gana.git (instead of master). Otherwise, if there are
    758 incompatible changes in GANA (like removed symbols), old builds could break.
    759 
    760 Update the Texinfo documentation using the files from docs.git:
    761 
    762 .. code-block:: console
    763 
    764    # Get the latest documentation repository
    765    $ cd $GIT/docs
    766    $ git pull
    767    $ make texinfo
    768    # The *.texi files are now in _build/texinfo
    769    #
    770    # This checks out the prebuilt branch in the prebuilt directory
    771    $ git worktree add prebuilt prebuilt
    772    $ cd prebuilt
    773    # Copy the pre-built documentation into the prebuilt directory
    774    $ cp -r ../_build/texinfo .
    775    # Push and commit to branch
    776    $ git commit -a -S -m "updating texinfo"
    777    $ git status
    778    # Verify that all files that should be tracked are tracked,
    779    # new files will have to be added to the Makefile.am in
    780    # exchange.git as well!
    781    $ git push
    782    # Remember $REVISION of commit
    783    #
    784    # Go to exchange
    785    $ cd $GIT/exchange/doc/prebuilt
    786    # Update submodule to point to latest commit
    787    $ git checkout $REVISION
    788 
    789 Finally, the Automake ``Makefile.am`` files may have to be adjusted to
    790 include new ``*.texi`` files or images.
    791 
    792 For bootstrap, you will need to install
    793 `GNU Recutils <https://www.gnu.org/software/recutils/>`_.
    794 
    795 For the exchange test cases to pass, ``make install`` must be run first.
    796 Without it, test cases will fail because plugins can't be located.
    797 
    798 .. code-block:: console
    799 
    800    $ ./bootstrap
    801    $ ./configure # add required options for your system
    802    $ make dist
    803    $ tar -xf taler-$COMPONENT-$VERSION.tar.gz
    804    $ cd taler-$COMPONENT-$VERSION
    805    $ make install check
    806 
    807 Wallet WebExtension
    808 -------------------
    809 
    810 The version of the wallet is in *manifest.json*. The ``version_name``
    811 should be adjusted, and *version* should be increased independently on
    812 every upload to the WebStore.
    813 
    814 .. code-block:: console
    815 
    816    $ ./configure
    817    $ make dist
    818 
    819 Upload to GNU mirrors
    820 ---------------------
    821 
    822 See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads
    823 
    824 Directive file:
    825 
    826 .. code-block:: none
    827 
    828    version: 1.2
    829    directory: taler
    830    filename: taler-exchange-0.1.0.tar.gz
    831    symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz
    832 
    833 Upload the files in **binary mode** to the ftp servers.
    834 
    835 
    836 Creating Debian packages
    837 ------------------------
    838 
    839 Our general setup is based on
    840 https://wiki.debian.org/DebianRepository/SetupWithReprepro
    841 
    842 First, update at least the version of the Debian package in
    843 debian/changelog, and then run:
    844 
    845 .. code-block:: bash
    846 
    847   $ dpkg-buildpackage -rfakeroot -b -uc -us
    848 
    849 in the respective source directory (GNUnet, exchange, merchant) to create the
    850 ``.deb`` files. Note that they will be created in the parent directory.  This
    851 can be done on gv.taler.net, or on another (secure) machine.
    852 Actual release builds should be done via the Docker images
    853 that can be found in ``deployment.git`` under packaging.
    854 
    855 On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository.
    856 
    857 Next, the ``*.deb`` files should be copied to gv.taler.net, say to
    858 ``/home/aptbuilder/incoming``.  Then, run
    859 
    860 .. code-block:: bash
    861 
    862   # cd /home/aptbuilder/apt
    863   # reprepro includedeb bullseye ~/incoming/*.deb
    864 
    865 to import all Debian files from ``~/incoming/`` into the ``bullseye``
    866 distribution.  If Debian packages were build against other distributions,
    867 reprepro may need to be first configured for those and the import command
    868 updated accordingly.
    869 
    870 Finally, make sure to clean up ``~/incoming/`` (by deleting the
    871 now imported ``*.deb`` files).
    872 
    873 
    874 
    875 Continuous integration
    876 ======================
    877 
    878 CI is done with Buildbot (https://buildbot.net/), and builds are
    879 triggered by the means of Git hooks. The results are published at
    880 https://buildbot.taler.net/ .
    881 
    882 In order to avoid downtimes, CI uses a "blue/green" deployment
    883 technique. In detail, there are two users building code on the system,
    884 the "green" and the "blue" user; and at any given time, one is running
    885 Taler services and the other one is either building the code or waiting
    886 for that.
    887 
    888 There is also the possibility to trigger builds manually, but this is
    889 only reserved to "admin" users.
    890 
    891 
    892 Internationalisation
    893 ====================
    894 
    895 Internationalisation (a.k.a "translation") is handled with using text-based 
    896 localization files named PO (Portable Object) holding pairs of original and 
    897 translated strings.
    898 
    899 iOS Apps
    900 ========
    901 
    902 .. _Build-iOS-from-source:
    903 
    904 Building Taler Wallet for iOS from source
    905 -----------------------------------------
    906 
    907 The GNU Taler Wallet iOS app is in
    908 `the official Git repository <https://git.taler.net/taler-ios.git>`__.
    909 
    910 Compatibility
    911 ^^^^^^^^^^^^^
    912 
    913 The minimum version of iOS supported is 15.0.
    914 This app runs on all iPhone models at least as new as the iPhone 6S.
    915 
    916 
    917 Building
    918 ^^^^^^^^
    919 
    920 Before building the iOS wallet, you must first checkout the
    921 `quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__
    922 and the
    923 `wallet-core repo <https://git.taler.net/wallet-core.git>`__.
    924 
    925 Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at
    926 the same level (e.g. in a "GNU_Taler" folder)
    927 Taler.xcworkspace expects the QuickJS framework sub-project to be at
    928 ``../quickjs-tart/QuickJS-rt.xcodeproj``.
    929 
    930 Build wallet-core first:
    931 
    932 .. code-block:: shell-session
    933 
    934   $ cd wallet-core
    935   $ make embedded
    936   $ open packages/taler-wallet-embedded/dist
    937 
    938 then drag or move its product "taler-wallet-core-qjs.mjs"
    939 into your quickjs-tart folder right at the top level.
    940 
    941 Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run...
    942 
    943 Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything
    944 there - all needed libraries and frameworks will be built automatically from
    945 Taler.xcworkspace.
    946 
    947 
    948 Android Apps
    949 ============
    950 
    951 Android App Nightly Builds
    952 --------------------------
    953 
    954 There are currently three Android apps in
    955 `the official Git repository <https://git.taler.net/taler-android.git>`__:
    956 
    957 * Wallet
    958   [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__]
    959 * Merchant PoS Terminal
    960   [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__]
    961 * Cashier
    962   [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__]
    963 
    964 Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__
    965 to utilize their CI
    966 and `F-Droid <https://f-droid.org>`_'s Gitlab integration
    967 to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_
    968 for each change on the ``master`` branch.
    969 
    970 All three apps publish their builds to the same F-Droid nightly repository
    971 (which is stored as a git repository):
    972 https://gitlab.com/gnu-taler/fdroid-repo-nightly
    973 
    974 You can download the APK files directly from that repository
    975 or add it to the F-Droid app for automatic updates
    976 by clicking the following link (on the phone that has F-Droid installed).
    977 
    978     `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_
    979 
    980 .. note::
    981     Nightly apps can be installed alongside official releases
    982     and thus are meant **only for testing purposes**.
    983     Use at your own risk!
    984 
    985 .. _Build-apps-from-source:
    986 
    987 Building apps from source
    988 -------------------------
    989 
    990 Note that this guide is different from other guides for building Android apps,
    991 because it does not require you to run non-free software.
    992 It uses the Merchant PoS Terminal as an example, but works as well for the other apps
    993 if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``.
    994 
    995 First, ensure that you have the required dependencies installed:
    996 
    997 * Java Development Kit 8 or higher (default-jdk-headless)
    998 * git
    999 * unzip
   1000 
   1001 Then you can get the app's source code using git:
   1002 
   1003 .. code-block:: console
   1004 
   1005   # Start by cloning the Android git repository
   1006   $ git clone https://git.taler.net/taler-android.git
   1007 
   1008   # Change into the directory of the cloned repository
   1009   $ cd taler-android
   1010 
   1011   # Find out which Android SDK version you will need
   1012   $ grep -i compileSdkVersion merchant-terminal/build.gradle
   1013 
   1014 The last command will return something like ``compileSdkVersion 29``.
   1015 So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project
   1016 and look for that version of the Android SDK there.
   1017 If the SDK version is not yet available as a free rebuild,
   1018 you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file.
   1019 Note that this might break things
   1020 or require you to also lower other versions such as ``targetSdkVersion``.
   1021 
   1022 In our example, the version is ``29`` which is available,
   1023 so download the "SDK Platform" package of "Android 10.0.0 (API 29)"
   1024 and unpack it:
   1025 
   1026 .. code-block:: console
   1027 
   1028   # Change into the directory that contains your downloaded SDK
   1029   $ cd $HOME
   1030 
   1031   # Unpack/extract the Android SDK
   1032   $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip
   1033 
   1034   # Tell the build system where to find the SDK
   1035   $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86"
   1036 
   1037   # Change into the directory of the cloned repository
   1038   $ cd taler-android
   1039 
   1040   # Build the merchant-terminal app
   1041   $ ./gradlew :merchant-terminal:assembleRelease
   1042 
   1043 If you get an error message complaining about build-tools
   1044 
   1045     > Failed to install the following Android SDK packages as some licences have not been accepted.
   1046          build-tools;29.0.3 Android SDK Build-Tools 29.0.3
   1047 
   1048 you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file
   1049 to the latest "Android SDK build tools" version supported by the Android Rebuilds project.
   1050 
   1051 After the build finished successfully,
   1052 you will find your APK in ``merchant-terminal/build/outputs/apk/release/``.
   1053 
   1054 Update translations
   1055 -------------------
   1056 
   1057 Translations are managed with Taler's weblate instance:
   1058 https://weblate.taler.net/projects/gnu-taler/
   1059 
   1060 To update translations, enter the taler-android git repository
   1061 and ensure that the weblate remote exists:
   1062 
   1063 .. code-block:: console
   1064 
   1065   $ git config -l | grep weblate
   1066 
   1067 If it does not yet exist (empty output), you can add it like this:
   1068 
   1069 .. code-block:: console
   1070 
   1071   $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/
   1072 
   1073 Then you can merge in translations commit from the weblate remote:
   1074 
   1075 .. code-block:: console
   1076 
   1077   # ensure you have latest version
   1078   $ git fetch weblate
   1079 
   1080   # merge in translation commits
   1081   $ git merge weblate/master
   1082 
   1083 Afterwards, build the entire project from source and test the UI
   1084 to ensure that no erroneous translations (missing placeholders) are breaking things.
   1085 
   1086 Release process
   1087 ---------------
   1088 
   1089 After extensive testing, the code making up a new release should get a signed git tag.
   1090 The current tag format is:
   1091 
   1092 * cashier-$VERSION
   1093 * pos-$VERSION
   1094 * wallet-$VERSION (where $VERSION has a v prefix)
   1095 
   1096 .. code-block:: console
   1097 
   1098   $ git tag -s $APP-$VERSION
   1099 
   1100 F-Droid
   1101 ^^^^^^^
   1102 Nightly builds get published automatically (see above) after pushing code to the official repo.
   1103 Actual releases get picked up by F-Droid's official repository via git tags.
   1104 So ensure that all releases get tagged properly.
   1105 
   1106 Some information for F-Droid official repository debugging:
   1107 
   1108 * Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__]
   1109 * Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__]
   1110 * PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__]
   1111 
   1112 Google Play
   1113 ^^^^^^^^^^^
   1114 Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__.
   1115 Before proceeding, ensure that this is properly set up
   1116 and that you have access to the Google Play API.
   1117 
   1118 It is important to have access to the signing keys and Google Play access keys
   1119 (JSON) and to ensure that the following environment variables are set
   1120 correctly and made available to Fastlane:
   1121 
   1122 .. code-block:: bash
   1123 
   1124    TALER_KEYSTORE_PATH=
   1125    TALER_KEYSTORE_PASS=
   1126    TALER_KEYSTORE_WALLET_ALIAS=
   1127    TALER_KEYSTORE_WALLET_PASS=
   1128    TALER_KEYSTORE_POS_ALIAS=
   1129    TALER_KEYSTORE_POS_PASS=
   1130    TALER_KEYSTORE_CASHIER_ALIAS=
   1131    TALER_KEYSTORE_CASHIER_PASS=
   1132    TALER_JSON_KEY_FILE=
   1133 
   1134 To release an app, enter into its respective folder and run fastlane:
   1135 
   1136 .. code-block:: console
   1137 
   1138   $ bundle exec fastlane
   1139 
   1140 Then select the deploy option.
   1141 
   1142 All uploads are going to the beta track by default.  These can be promoted to
   1143 production later or immediately after upload if you feel daring. It is also
   1144 important to bump the version and build code with every release.
   1145 
   1146 .. _Code-coverage:
   1147 
   1148 Code Coverage
   1149 =============
   1150 
   1151 Code coverage is done with the Gcov / Lcov
   1152 (http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run
   1153 nightly (once a day) by a Buildbot worker. The coverage results are
   1154 then published at https://lcov.taler.net/ .
   1155 
   1156 
   1157 Coding Conventions
   1158 ==================
   1159 
   1160 GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript.
   1161 
   1162 Components written in C
   1163 -----------------------
   1164 
   1165 These are the general coding style rules for Taler.
   1166 
   1167 * Baseline rules are to follow GNU guidelines, modified or extended
   1168   by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style
   1169 
   1170 Naming conventions
   1171 ^^^^^^^^^^^^^^^^^^
   1172 
   1173 * include files (very similar to GNUnet):
   1174 
   1175   * if installed, must start with "``taler_``" (exception: platform.h),
   1176     and MUST live in src/include/
   1177   * if NOT installed, must NOT start with "``taler_``" and
   1178     MUST NOT live in src/include/ and
   1179     SHOULD NOT be included from outside of their own directory
   1180   * end in "_lib" for "simple" libraries
   1181   * end in "_plugin" for plugins
   1182   * end in "_service" for libraries accessing a service, i.e. the exchange
   1183 
   1184 * binaries:
   1185 
   1186   * taler-exchange-xxx: exchange programs
   1187   * taler-merchant-xxx: merchant programs (demos)
   1188   * taler-wallet-xxx: wallet programs
   1189   * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx
   1190   * libtalerxxx: library for API xxx
   1191 
   1192 * logging
   1193 
   1194   * tools use their full name in GNUNET_log_setup
   1195     (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'.
   1196   * pure libraries (without associated service) use 'GNUNET_log_from'
   1197     with the component set to their library name (without lib or '.so'),
   1198     which should also be their directory name (i.e. 'util')
   1199   * plugin libraries (without associated service) use 'GNUNET_log_from'
   1200     with the component set to their type and plugin name (without lib or '.so'),
   1201     which should also be their directory name (i.e. 'exchangedb-postgres')
   1202   * libraries with associated service) use 'GNUNET_log_from'
   1203     with the name of the service,  which should also be their
   1204     directory name (i.e. 'exchange')
   1205   * for tools with ``-l LOGFILE``, its absence means write logs to stderr
   1206 
   1207 * configuration
   1208 
   1209   * same rules as for GNUnet
   1210 
   1211 * exported symbols
   1212 
   1213   * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME
   1214     MUST match the subdirectory of src/ in which the symbol is defined
   1215   * from libtalerutil start just with ``TALER_``, without subsystemname
   1216   * if scope is ONE binary and symbols are not in a shared library,
   1217     use binary-specific prefix (such as TMH = taler-exchange-httpd) for
   1218     globals, possibly followed by the subsystem (TMH_DB_xxx).
   1219 
   1220 * structs:
   1221 
   1222   * structs that are 'packed' and do not contain pointers and are
   1223     thus suitable for hashing or similar operations are distinguished
   1224     by adding a "P" at the end of the name. (NEW)  Note that this
   1225     convention does not hold for the GNUnet-structs (yet).
   1226   * structs that are used with a purpose for signatures, additionally
   1227     get an "S" at the end of the name.
   1228 
   1229 * private (library-internal) symbols (including structs and macros)
   1230 
   1231   * must not start with ``TALER_`` or any other prefix
   1232 
   1233 * testcases
   1234 
   1235   * must be called "test_module-under-test_case-description.c"
   1236 
   1237 * performance tests
   1238 
   1239   * must be called "perf_module-under-test_case-description.c"
   1240 
   1241 Shell Scripts
   1242 -------------
   1243 
   1244 Shell scripts should be avoided if at all possible.  The only permissible uses of shell scripts
   1245 in GNU Taler are:
   1246 
   1247 * Trivial invocation of other commands.
   1248 * Scripts for compatibility (e.g. ``./configure``) that must run on
   1249   as many systems as possible.
   1250 
   1251 When shell scripts are used, they ``MUST`` begin with the following ``set`` command:
   1252 
   1253 .. code-block:: console
   1254 
   1255   # Make the shell fail on undefined variables and
   1256   # commands with non-zero exit status.
   1257   $ set -eu
   1258 
   1259 Kotlin
   1260 ------
   1261 
   1262 We so far have no specific guidelines, please follow best practices
   1263 for the language.
   1264 
   1265 
   1266 Python
   1267 ------
   1268 
   1269 Supported Python Versions
   1270 ^^^^^^^^^^^^^^^^^^^^^^^^^
   1271 
   1272 Python code should be written and build against version 3.7 of Python.
   1273 
   1274 Style
   1275 ^^^^^
   1276 
   1277 We use `yapf <https://github.com/google/yapf>`_ to reformat the
   1278 code to conform to our style instructions.
   1279 A reusable yapf style file can be found in ``build-common``,
   1280 which is intended to be used as a git submodule.
   1281 
   1282 Python for Scripting
   1283 ^^^^^^^^^^^^^^^^^^^^
   1284 
   1285 When using Python for writing small utilities, the following libraries
   1286 are useful:
   1287 
   1288 * ``click`` for argument parsing (should be preferred over argparse)
   1289 * ``pathlib`` for path manipulation (part of the standard library)
   1290 * ``subprocess`` for "shelling out" to other programs.  Prefer ``subprocess.run``
   1291   over the older APIs.
   1292 
   1293 
   1294 Swift
   1295 -----
   1296 
   1297 Please follow best practices for the language.
   1298 
   1299 
   1300 TypeScript
   1301 ----------
   1302 
   1303 Please follow best practices for the language.
   1304 
   1305 
   1306 Testing library
   1307 ===============
   1308 
   1309 This chapter is a VERY ABSTRACT description of how testing is
   1310 implemented in Taler, and in NO WAY wants to substitute the reading of
   1311 the actual source code by the user.
   1312 
   1313 In Taler, a test case is an array of ``struct TALER_TESTING_Command``,
   1314 informally referred to as ``CMD``, that is iteratively executed by the
   1315 testing interpreter. This latter is transparently initiated by the
   1316 testing library.
   1317 
   1318 However, the developer does not have to defined CMDs manually, but
   1319 rather call the proper constructor provided by the library. For example,
   1320 if a CMD is supposed to test feature ``x``, then the library would
   1321 provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously,
   1322 each constructor has its own particular arguments that make sense to
   1323 test ``x``, and all constructor are thoroughly commented within the
   1324 source code.
   1325 
   1326 Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The
   1327 former contains the main logic to test feature ``x``, whereas the latter
   1328 cleans the memory up after execution.
   1329 
   1330 In a test life, each CMD needs some internal state, made by values it
   1331 keeps in memory. Often, the test has to *share* those values with other
   1332 CMDs: for example, CMD1 may create some key material and CMD2 needs this
   1333 key material to encrypt data.
   1334 
   1335 The offering of internal values from CMD1 to CMD2 is made by *traits*. A
   1336 trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array
   1337 of traits, that it offers via the public trait interface to other
   1338 commands. The definition and filling of such array happens transparently
   1339 to the test developer.
   1340 
   1341 For example, the following example shows how CMD2 takes an amount object
   1342 offered by CMD1 via the trait interface.
   1343 
   1344 Note: the main interpreter and the most part of CMDs and traits are
   1345 hosted inside the exchange codebase, but nothing prevents the developer
   1346 from implementing new CMDs and traits within other codebases.
   1347 
   1348 .. code-block:: c
   1349 
   1350    /* Without loss of generality, let's consider the
   1351     * following logic to exist inside the run() method of CMD1 */
   1352    ...
   1353 
   1354    struct TALER_Amount *a;
   1355    /**
   1356     * the second argument (0) points to the first amount object offered,
   1357     * in case multiple are available.
   1358     */
   1359    if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a))
   1360      return GNUNET_SYSERR;
   1361    ...
   1362 
   1363    use(a); /* 'a' points straight into the internal state of CMD2 */
   1364 
   1365 In the Taler realm, there is also the possibility to alter the behaviour
   1366 of supposedly well-behaved components. This is needed when, for example,
   1367 we want the exchange to return some corrupted signature in order to
   1368 check if the merchant backend detects it.
   1369 
   1370 This alteration is accomplished by another service called *twister*. The
   1371 twister acts as a proxy between service A and B, and can be programmed
   1372 to tamper with the data exchanged by A and B.
   1373 
   1374 Please refer to the Twister codebase (under the ``test`` directory) in
   1375 order to see how to configure it.
   1376 
   1377 
   1378 User-Facing Terminology
   1379 =======================
   1380 
   1381 This section contains terminology that should be used and that should not be
   1382 used in the user interface and help materials.
   1383 
   1384 Terms to Avoid
   1385 --------------
   1386 
   1387 Refreshing
   1388   Refreshing is the internal technical terminology for the protocol to
   1389   give change for partially spent coins
   1390 
   1391   **Use instead**: "Obtaining change"
   1392 
   1393 Charge
   1394   Charge has two opposite meanings (charge to a credit card vs. charge a battery).
   1395   This can confuse users.
   1396 
   1397   **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up"
   1398 
   1399 Coin
   1400   Coins are an internal construct, the user should never be aware that their balance
   1401   is represented by coins of different denominations.
   1402 
   1403   **Use instead**: "(Digital) Cash" or "(Wallet) Balance"
   1404 
   1405 Consumer
   1406   Has bad connotation of consumption.
   1407 
   1408   **Use instead**: Customer or user.
   1409 
   1410 Proposal
   1411   The term used to describe the process of the merchant facilitating the download
   1412   of the signed contract terms for an order.
   1413 
   1414   **Avoid**.  Generally events that relate to proposal downloads
   1415   should not be shown to normal users, only developers.  Instead, use
   1416   "communication with mechant failed" if a proposed order can't be downloaded.
   1417 
   1418 Anonymous E-Cash
   1419   Should be generally avoided, since Taler is only anonymous for
   1420   the customer. Also some people are scared of anonymity (which as
   1421   a term is also way too absolute, as anonymity is hardly ever perfect).
   1422 
   1423   **Use instead**:  "Privacy-preserving", "Privacy-friendly"
   1424 
   1425 Payment Replay
   1426   The process of proving to the merchant that the customer is entitled
   1427   to view a digital product again, as they already paid for it.
   1428 
   1429   **Use instead**:  In the event history, "re-activated digital content purchase"
   1430   could be used. (FIXME: this is still not nice.)
   1431 
   1432 Session ID
   1433   See Payment Replay.
   1434 
   1435 Order
   1436   Too ambiguous in the wallet.
   1437 
   1438   **Use instead**: Purchase
   1439 
   1440 Fulfillment URL
   1441   URL that the serves the digital content that the user purchased
   1442   with their payment.  Can also be something like a donation receipt.
   1443 
   1444 Donau
   1445   Developer-internal name for the tax authority component.
   1446 
   1447   **Use instead**:  Tax authority
   1448 
   1449 Terms to Use
   1450 ------------
   1451 
   1452 Auditor
   1453   Regulatory entity that certifies exchanges and oversees their operation.
   1454 
   1455 Exchange Operator
   1456   The entity/service that gives out digital cash in exchange for some
   1457   other means of payment.
   1458 
   1459   In some contexts, using "Issuer" could also be appropriate.
   1460   When showing a balance breakdown,
   1461   we can say "100 Eur (issued by exchange.euro.taler.net)".
   1462   Sometimes we may also use the more generic term "Payment Service Provider"
   1463   when the concept of an "Exchange" is still unclear to the reader.
   1464 
   1465 Refund
   1466   A refund is given by a merchant to the customer (rather the customer's wallet)
   1467   and "undoes" a previous payment operation.
   1468 
   1469 Payment
   1470   The act of sending digital cash to a merchant to pay for an order.
   1471 
   1472 Purchase
   1473   Used to refer to the "result" of a payment, as in "view purchase".
   1474   Use sparsingly, as the word doesn't fit for all payments, such as donations.
   1475 
   1476 Contract Terms
   1477   Partially machine-readable representation of the merchant's obligation after the
   1478   customer makes a payment.
   1479 
   1480 Merchant
   1481   Party that receives a payment.
   1482 
   1483 Wallet
   1484   Also "Taler Wallet".  Software component that manages the user's digital cash
   1485   and payments.
   1486 
   1487 
   1488 Developer Glossary
   1489 ==================
   1490 
   1491 This glossary is meant for developers.  It contains some terms that we usually do not
   1492 use when talking to end users or even system administrators.
   1493 
   1494 .. glossary::
   1495   :sorted:
   1496 
   1497   absolute time
   1498     method of keeping time in :term:`GNUnet` where the time is represented
   1499     as the number of microseconds since 1.1.1970 (UNIX epoch).  Called
   1500     absolute time in contrast to :term:`relative time`.
   1501 
   1502   aggregate
   1503     the :term:`exchange` combines multiple payments received by the
   1504     same :term:`merchant` into one larger :term:`wire transfer` to
   1505     the respective merchant's :term:`bank` account
   1506 
   1507   auditor
   1508     trusted third party that verifies that the :term:`exchange` is operating correctly
   1509 
   1510   bank
   1511     traditional financial service provider who offers
   1512     :term:`wire transfers <wire transfer>` between accounts
   1513 
   1514   buyer
   1515     individual in control of a Taler :term:`wallet`, usually using it to
   1516     :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`).
   1517 
   1518   close
   1519     operation an :term:`exchange` performs on a :term:`reserve` that has not been
   1520     :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the
   1521     exchange wires the remaining funds back to the customer, minus a :term:`fee`
   1522     for closing
   1523 
   1524   customer
   1525     individual that directs the buyer (perhaps the same individual) to make a purchase
   1526 
   1527   coin
   1528     coins are individual token representing a certain amount of value, also known as the :term:`denomination` of the coin
   1529 
   1530   refresh commitment
   1531     data that the wallet commits to during the :term:`melt` stage of the
   1532     :term:`refresh` protocol where it
   1533     has to prove to the :term:`exchange` that it is deriving the :term:`fresh`
   1534     coins as specified by the Taler protocol.  The commitment is verified
   1535     probabilistically (see: :term:`kappa`) during the :term:`reveal` stage.
   1536 
   1537   contract
   1538     formal agreement between :term:`merchant` and :term:`customer` specifying the
   1539     :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the
   1540     customer
   1541 
   1542   contract terms
   1543     the individual clauses specifying what the buyer is purchasing from the
   1544     :term:`merchant`
   1545 
   1546   denomination
   1547     unit of currency, specifies both the currency and the face value of a :term:`coin`,
   1548     as well as associated fees and validity periods
   1549 
   1550   denomination key
   1551     (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a
   1552     particular :term:`denomination`
   1553 
   1554   deposit
   1555     operation by which a merchant passes coins to an exchange, expecting the
   1556     exchange to credit his bank account in the future using an
   1557     :term:`aggregate` :term:`wire transfer`
   1558 
   1559   drain
   1560     process by which an exchange operator takes the profits
   1561     (from :term:`fees <fee>`) out of the escrow account and moves them into
   1562     their regular business account
   1563 
   1564   dirty
   1565     a :term:`coin` is dirty if its public key may be known to an entity other than
   1566     the customer, thereby creating the danger of some entity being able to
   1567     link multiple transactions of coin's owner if the coin is not refreshed
   1568 
   1569   empty
   1570     a :term:`reserve` is being emptied when a :term:`wallet` is using the
   1571     reserve's private key to :term:`withdraw` coins from it. This reduces
   1572     the balance of the reserve. Once the balance reaches zero, we say that
   1573     the reserve has been (fully) emptied.  Reserves that are not emptied
   1574     (which is the normal process) are :term:`closed <close>` by the exchange.
   1575 
   1576   exchange
   1577     Taler's payment service operator.  Issues electronic coins during
   1578     withdrawal and redeems them when they are deposited by merchants
   1579 
   1580   expired
   1581     Various operations come with time limits. In particular, denomination keys
   1582     come with strict time limits for the various operations involving the
   1583     coin issued under the denomination. The most important limit is the
   1584     deposit expiration, which specifies until when wallets are allowed to
   1585     use the coin in deposit or refreshing operations. There is also a "legal"
   1586     expiration, which specifies how long the exchange keeps records beyond the
   1587     deposit expiration time.  This latter expiration matters for legal disputes
   1588     in courts and also creates an upper limit for refreshing operations on
   1589     special zombie coin
   1590 
   1591   GNUnet
   1592     Codebase of various libraries for a better Internet, some of which
   1593     GNU Taler depends upon.
   1594 
   1595   fakebank
   1596     implementation of the :term:`bank` API in memory to be used only for test
   1597     cases.
   1598 
   1599   fee
   1600     an :term:`exchange` charges various fees for its service. The different
   1601     fees are specified in the protocol. There are fees per coin for
   1602     :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and
   1603     :term:`refunding <refund>`.  Furthermore, there are fees per wire transfer
   1604     when a :term:`reserve` is :term:`closed <close>`
   1605     and for :term:`aggregate` :term:`wire transfers <wire transfer>`
   1606     to the :term:`merchant`.
   1607 
   1608   fresh
   1609     a :term:`coin` is fresh if its public key is only known to the customer
   1610 
   1611   JSON
   1612     JavaScript Object Notation (JSON) is a
   1613     serialization format derived from the JavaScript language which is
   1614     commonly used in the Taler protocol as the payload of HTTP requests
   1615     and responses.
   1616 
   1617   kappa
   1618     security parameter used in the :term:`refresh` protocol. Defined to be 3.
   1619     The probability of successfully evading the income transparency with the
   1620     refresh protocol is 1:kappa.
   1621 
   1622   libeufin
   1623     Kotlin component that implements a regional currency bank and an
   1624     adapter to communicate via EBICS with European core banking systems.
   1625 
   1626   link
   1627     specific step in the :term:`refresh` protocol that an exchange must offer
   1628     to prevent abuse of the :term:`refresh` mechanism.  The link step is
   1629     not needed in normal operation, it just must be offered.
   1630 
   1631   master key
   1632     offline key used by the exchange to certify denomination keys and
   1633     message signing keys
   1634 
   1635   melt
   1636     step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin`
   1637     is invalidated to be reborn :term:`fresh` in a subsequent
   1638     :term:`reveal` step.
   1639 
   1640   merchant
   1641     party receiving payments (usually in return for goods or services)
   1642 
   1643   message signing key
   1644      key used by the exchange to sign online messages, other than coins
   1645 
   1646   order
   1647     offer made by the merchant to a wallet; pre-cursor to
   1648     a contract where the wallet is not yet fixed. Turns
   1649     into a :term:`contract` when a wallet claims the order.
   1650 
   1651   owner
   1652     a coin is owned by the entity that knows the private key of the coin
   1653 
   1654   relative time
   1655     method of keeping time in :term:`GNUnet` where the time is represented
   1656     as a relative number of microseconds.  Thus, a relative time specifies
   1657     an offset or a duration, but not a date.  Called relative time in
   1658     contrast to :term:`absolute time`.
   1659 
   1660   recoup
   1661     Operation by which an exchange returns the value of coins affected
   1662     by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to
   1663     withdraw new coins or wiring funds back to the bank account of the :term:`owner`.
   1664 
   1665   planchet
   1666     precursor data for a :term:`coin`. A planchet includes the coin's internal
   1667     secrets (coin private key, blinding factor), but lacks the RSA signature
   1668     of the :term:`exchange`.  When :term:`withdrawing <withdraw>`, a :term:`wallet`
   1669     creates and persists a planchet before asking the exchange to sign it to
   1670     get the coin.
   1671 
   1672   purchase
   1673     Refers to the overall process of negotiating a :term:`contract` and then
   1674     making a payment with :term:`coins <coin>` to a :term:`merchant`.
   1675 
   1676   privacy policy
   1677     Statement of an operator how they will protect the privacy of users.
   1678 
   1679   proof
   1680     Message that cryptographically demonstrates that a particular claim is correct.
   1681 
   1682   proposal
   1683     a list of :term:`contract terms` that has been completed and signed by the
   1684     merchant backend.
   1685 
   1686   refresh
   1687     operation by which a :term:`dirty` :term:`coin` is converted into one or more
   1688     :term:`fresh` coins.  Involves :term:`melting <melt>` the :term:`dirty` coins and
   1689     then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`.
   1690 
   1691   refund
   1692     operation by which a merchant steps back from the right to funds that he
   1693     obtained from a :term:`deposit` operation, giving the right to the funds back
   1694     to the customer
   1695 
   1696   refund transaction id
   1697     unique number by which a merchant identifies a :term:`refund`. Needed
   1698     as refunds can be partial and thus there could be multiple refunds for
   1699     the same :term:`purchase`.
   1700 
   1701   reserve
   1702     accounting mechanism used by the exchange to track customer funds
   1703     from incoming :term:`wire transfers <wire transfer>`.  A reserve is created whenever
   1704     a customer wires money to the exchange using a well-formed public key
   1705     in the subject.  The exchange then allows the customer's :term:`wallet`
   1706     to :term:`withdraw` up to the amount received in :term:`fresh`
   1707     :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a
   1708     reserve is not emptied, the exchange will eventually :term:`close` it.
   1709 
   1710     Other definition: Funds set aside for future use; either the balance of a customer at the
   1711     exchange ready for withdrawal, or the funds kept in the exchange;s bank
   1712     account to cover obligations from coins in circulation.
   1713 
   1714   reveal
   1715     step in the :term:`refresh` protocol where some of the transfer private
   1716     keys are revealed to prove honest behavior on the part of the wallet.
   1717     In the reveal step, the exchange returns the signed :term:`fresh` coins.
   1718 
   1719   revoke
   1720     exceptional operation by which an exchange withdraws a denomination from
   1721     circulation, either because the signing key was compromised or because
   1722     the exchange is going out of operation; unspent coins of a revoked
   1723     denomination are subjected to recoup.
   1724 
   1725   sharing
   1726     users can share ownership of a :term:`coin` by sharing access to the coin&#39;s
   1727     private key, thereby allowing all co-owners to spend the coin at any
   1728     time.
   1729 
   1730   spend
   1731     operation by which a customer gives a merchant the right to deposit
   1732     coins in return for merchandise
   1733 
   1734   transfer key
   1735     special cryptographic key used in the :term:`refresh` protocol, some of which
   1736     are revealed during the :term:`reveal` step. Note that transfer keys have,
   1737     despite the name, no relationship to :term:`wire transfers <wire transfer>`.  They merely
   1738     help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin
   1739 
   1740   terms
   1741     the general terms of service of an operator, possibly including
   1742     the :term:`privacy policy`.  Not to be confused with the
   1743     :term:`contract terms` which are about the specific purchase.
   1744 
   1745   transaction
   1746     method by which ownership is exclusively transferred from one entity
   1747 
   1748   user
   1749     any individual using the Taler payment system
   1750     (see :term:`customer`, :term:`buyer`, :term:`merchant`).
   1751 
   1752   version
   1753     Taler uses various forms of versioning. There is a database
   1754     schema version (stored itself in the database, see \*-0000.sql) describing
   1755     the state of the table structure in the database of an :term:`exchange`,
   1756     :term:`auditor` or :term:`merchant`. There is a protocol
   1757     version (CURRENT:REVISION:AGE, see GNU libtool) which specifies
   1758     the network protocol spoken by an :term:`exchange` or :term:`merchant`
   1759     including backwards-compatibility. And finally there is the software
   1760     release version (MAJOR.MINOR.PATCH, see https://semver.org/) of
   1761     the respective code base.
   1762 
   1763   wallet
   1764     software running on a customer's computer; withdraws, stores and
   1765     spends coins
   1766 
   1767   WebExtension
   1768     Cross-browser API used to implement the GNU Taler wallet browser extension.
   1769 
   1770   wire gateway
   1771     API used by the exchange to talk with some real-time gross settlement system
   1772     (core banking system, blockchain) to notice inbound credits wire transfers
   1773     (during withdraw) and to trigger outbound debit wire transfers (primarily
   1774     for deposits).
   1775 
   1776   wire transfer
   1777     a wire transfer is a method of sending funds between :term:`bank` accounts
   1778 
   1779   wire transfer identifier
   1780     Subject of a wire transfer from the exchange to a merchant;
   1781     set by the aggregator to a random nonce which uniquely
   1782     identifies the transfer.
   1783 
   1784   withdraw
   1785     operation by which a :term:`wallet` can convert funds from a :term:`reserve` to
   1786     fresh coins
   1787 
   1788   zombie
   1789     :term:`coin` where the respective :term:`denomination key` is past its
   1790     :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid
   1791     for an operation because it was :term:`melted <melt>` while it was still
   1792     valid, and then later again credited during a :term:`recoup` process
   1793 
   1794 
   1795 
   1796 Developer Tools
   1797 ===============
   1798 
   1799 This section describes various internal programs to make life easier for the
   1800 developer.
   1801 
   1802 
   1803 taler-harness
   1804 -------------
   1805 
   1806 **taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation.
   1807 
   1808 
   1809 **taler-harness deployment gen-coin-config**
   1810 [**-min-amount**=**\ ‌\ *VALUE*]
   1811 [**-max-amount**=**\ ‌\ *VALUE*]